ICAROB
2024
PROCEEDINGS OF THE 2024
INTERNATIONAL CONFERENCE
ON ARTIFICIAL LIFE AND ROBOTICS
February 22 to 25, 2024
J:COM HorutoHall, Oita, Japan
29th AROB International Meeting Series
Editor-in-Chief
Masanori Sugisaka
Editors: Yingmin Jia, Takao Ito, Ju-Jang Lee
ISBN 978-4-9913337-0-5

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

Proceedings of The 2024 International Conference on

ARTIFICIAL LIFE AND ROBOTICS(ICAROB2024)

February 22 to 25, J:COM HorutoHall, Oita, Japan, 2024
29th AROB International Meeting Series
Editor-in-Chief
Masanori Sugisaka
Editors: Yingmin Jia, Takao Ito, Ju-Jang Lee
ISBN 978-4-9913337-0-5©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

Contents
1

Organization, etc.

1

2

Messages

11

3Time Table

15

4

Opening Ceremony

19

5Technical paper index

20

6
6-1
6-2
6-3
7

Abstracts
PS abstracts
OS abstracts
GS abstracts
Authors index

50
51
103
114

©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

SPONSERED
ALife Robotics Corporation Ltd.

ORGANIZED BYInternational Steering Committee of International Conference on Artificial Life andRobotics (ICAROB)SUPPORTED BYIEEE Fukuoka Section (Japan)

CO-ORGANIZED BY
Chinese Association for Artificial Intelligence (CAAI, P. R. China)

1©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

ADVISORY COMMITTEE CHAIRS
Kazuo Ishii (Kyushu Institute of Technology, Japan)
Kensuke Harada (Osaka University, Japan)
Takashi Kohno (University of Tokyo, Japan)
Eiji Hayashi (Kyushu Institute of Technology, Japan)
Jeffrey Johnson (The Open University, UK)
Katia Passerini (Seton Hall University, USA)ADVISORY COMMITTEE
Bruce Eisenstein (Drexel University, Former IEEE President, USA)
Hidenori Kimura (RIKEN & Wasada University, Japan)
Joshua M. Epstein (The Johns Hopkins University, USA)
Ken-ichi Tanaka (Executive Fellow, Mitsubishi Electric Corporation, Japan)
Masato Nakagawa (Executive Fellow, DENSO CORPORATION, Visiting Professor of
Hiroshima University, Japan)
Moshe Kam (New Jersey Institute of Technology, Former IEEE President, USA)Paul Kalata (Drexel University, USA)Paul Oh (Drexel University, USA)Peter Herczfeld (Drexel University, USA)
GENERAL CHAIR
Masanori Sugisaka(Alife Robotics Corporation Ltd., Japan)
CO-GENERAL CHAIRS
Yingmin Jia (Beihang Univesty, P. R. China)
Takao Ito (Hiroshima University, Japan)
Ju-Jang Lee (Honorary professor, KAIST, Korea)
VICE GENERAL CHAIR
Ang, Chun Kit (University of UCSI, Malaysia)
Henrik. H. Lund (Technical University of Denmark, Denmark)John. L. Casti (International Institute for Applied Systems Analysis, Austria)
Luigi Pagliarini (Technical University of Denmark, Denmark)(Academy of Fine Arts of Macerata, Italy)
Mohd Rizon bin Juhari (University of UCSI, Malaysia)
Norrima Mokhtar (University of Malaya, Malaysia)

2©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

PROGRAM CHAIRMAN
Makoto Sakamoto (University of Miyazaki, Japan)
CO-PROGRAM CHAIR
Marion Oswald (Technische Universität Wien, Austria)
INTERNATIONAL ORGANIZING COMMITTEE
Akira Nakamura (Saitama Institute of Technology, Japan)
Akinori Abe (Chiba University, Japan)
Caoli Wang (University of Shanghai for Science and Technology, P. R. China)
Chan Gook Park (Seoul National University, Korea)
Evgeni Magid (Kazan Federal University, Russia)
Fuzhong Wang (Henan Polytechnic University, P. R. China)
Hazry Desa (University of Malaysia, Perlis, Malaysia)
Hidehiko Yamamoto (Gifu University, Japan)
Hiroki Tamura (The University of Miyazaki, Japan)
Hiroshi Kage (Mitsubishi Electric Corporation, Japan)
Hiroshi Matsuno (Yamaguchi University, Japan)
Jiwu Wang (Beijing Jiaotong University, P. R. China)
Jovana Jovic (CNRS-AIST JRL, Japan, France)
Junping Du (Beijing University of Posts and Telecommunications, P. R. China)
Katsunori Shimohara (Doshisha University, Japan)
Kazuo Ishii (Kyushu Institute of Technology, Japan)
Kenichi Tanaka (Nagasaki Institute of Applied Science, Japan)
Kenji Hashimoto (Waseda University, Japan)
Kunikazu Kobayashi (Aichi Prefectural University, Japan)
Kuo-Hsien Hsia (Far East University, Taiwan)
Kuo-Lan Su (National Yunlin University of Science and Technology, Taiwan)
Kyungho Park (U.S. ARMY, Japan and USA)
Masao Kubo (National Defense Academy of Japan, Japan)
Maxim Sokolov (Innopolis University, Russia)
Mehta Rajiv (New Jersey Institute of Technology, USA)
Minoru Kumano (University of Miyazaki, Japan)
Peter Sapaty (Ukrainian Academy of Science, Ukraine)
Pierre Parrend (University of Strasbourg, France)
Qiang Cai (Beijing Technology and Business University, P. R. China)
Qu Yanbin (Harbin Institute of Technology, P. R. China)
Singo Mabu (Yamaguchi University, Japan)
Takashi Kohno (The University of Tokyo, Japan)
Takashi Ogata (Iwate Prefectural University)
Tetsuro Hattori (Kagawa University, Japan)
3©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

Thi Thi Zin (University of Miyazaki, Japan)
Thomas S. Ray (University of Oklahoma, USA)
Toru Yamamoto (Hiroshima University, Japan)
Victor Berdonosov (Komsomolsk-on-Amur State University of Technology, Russia)
Yasunari Yoshitomi (Kyoto Prefectural University, Japan)
Yi Chai (Chongqing University, P. R. China)
Yoshifumi Morita (Nagoya Institute of Technology, Japan)
Yoshiro Imai (Kagawa University, Japan)
INTERNATIONAL PROGRAM COMMITTEE
Abdul Rahman bin Dullah (Universiti Teknikal Malaysia Melaka, Malaysia)
Akinori Abe (Chiba University, Japan)
Akihiro Yamaguchi (Fukuoka Institute of Technology, Japan)
Akihito Kanai (Hosei University, Japan)
Ali Selamat (University of Technology of Malaysia, Malaysia)
Aminurafiuddin bin Zulkifli (Multimedia University, Malaysia)
Amornphun Phunopas (King Mongkut’s University of Technology, North Bankok,
Thailand)
Andre Rosendo (ShanghaiTech University, P. R. China)
Anne Jeannin-Girardon (University of Strasbourg, France)
Anton Shiriaev (Norwegian University of Science and Technology, Norway)
Artur Sagitov (Kazan Federal University, Russia)
Ashi Gautam (The University of Tokyo, Japan)
Atsuya Tange (The University of Tokyo, Japan)
Ayumu Tominaga (Kitakyushu National College of Technology, Japan)
Bin Zhang (Beijing University of Posts and Telecommunications, P. R. China)
Cecilia Zanni-Merk (INSA-Rouen, France)
Chaoli Wang (University of Shanghai for Science and Technology, P. R. China)
Chia-Nan Ko (Nan kai University of Technology, Taiwan)
Chung-Wen Hung (National Yunlin University of Science & Technology, Taiwan)
Congdao Han (Shanghai Institute of Technology, P. R. China)
Cui Zhang (Beihang University, P. R. China)
Cynthia Matuszek (University of Maryland, USA)
Donglian Qi (Zhejiang University, P. R. China)
Duangiai Jitkongchuen (Dhurakij Pundit University, Thailand)
Dunwei Gong (China University of Mining and Technology, P. R. China)
Jiao Jia (Beihang University, P. R. China)
Endra Joelianto (Bandung Institute of Technology, Indonesia)
Fabio Guigou (University of Strasbourg, France)
Fei Hao (Beihang University, P. R. China)
Fuzhong Wang (Henan Polytechnic University, P. R. China)
Haibo Li (Royal Institute of Technology, Sweden)
Haisheng Li (Beijing Technology and Business University, P. R. China)
4©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

Haruhisa Okuda (Mitsubishi Electric Corporation, Japan)
Haruka Tsuboi (The University of Miyazaki, Japan)
Haruna Matsushita (Kagawa University, Japan)
Heeje Kim (Pusan National University, Korea)
Hidetsugu Suto (Muroran Institute of Technology, Japan)
Hiroyuki Iizuka (Osaka University, Japan)
Hongbo Li (Tsinghua University, P. R. China)
Hongji Ma (Shandong University of Science and Technology, P. R. China)
Hongjiu Yang (Yanshan University, P. R. China)
Hongpeng Yin (Chongqing University, P. R. China)
Hussein Abbass (University of New South Wales, and ADFA, Australia)
I-Hsien Liu (National Cheng Kung Univrsiity, Taiwan)
Ilya Afanasyev (Innopolis University, Russia)
Istvan Harmati (Budapest Institute of Technology and Economics, Hungary)
Ivan Tanev (Doshisha University, Japan)
Jiandong Zhao (Beijing Jiaotong University, P. R. China)
JJ Merelo (University of Granada, Spain)
Joono Cheong (Korea University, Korea)
Julio Navarro Lara (University of Strasbourg, France)
Jun Kobayashi (Kyushu Institute of Technology, Japan)
Jung-Shian Li (National Cheng Kung University, Taiwan)
Keiji Kamei (Nishinippon Institute of Technology, Japan)
Keisuke Watanabe (Tokai University, Japan)
Keita Honda (Gifu University, Japan)
Ke Zhang (Chongqing University, P. R. China)
Kenichi Tanaka (Nagasaki Institute of Applied Science, Japan)
Kensuke Ogata (Osaka City University Japan)
Khairul Salleh bin Mohamed Sahari (Universiti Tenaga Nasional, Malaysia)
Khoyratee Farad (The University of Tokyo, Japan)
Kouichi Takeuchi (Okayama University, Japan)
Konstantin Yakovlev (Russian Academy of Sciences, Higher School of Economics, Russia)
Kui Xiang (Wuhan University of Technology, P. R. China)
Leonid Freidovich (Umea University, Sweden)
Levi Timothée (The University of Tokyo, Japan)
Liming Chen (Beihang University, P. R. China)
Lin Li (University of Shanghai for Science and Technology, P. R. China)
Lin Zhao (Qingdao University, P. R. China)
Mamoru Yoshimura (The University of Miyazaki, Japan)
Manabu Yamada (Nagoya Institute of Technology, Japan)
Masahide Ito (Aichi Prefectural University, Japan)
Masahiro Ushio (Kyushu Institute of Technology, Japan)
Masahiro Yokomichi (The University of Miyazaki, Japan)
Masamichi Hori (The University of Miyazaki, Japan)
Masanori Takahashi (Tokai University, Japan)
5©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

Masayoshi Kano (Chukyou University, Japan)
Masayoshi Tabuse (Kyoto Prefectural University, Japan)
Masaomi Hatakeyama (university of Zurich, Switzerland)
Max Talanov (Kazan Federal University, Russia)
Meng Duan (Beihang University, P. R. China)
Moeko Tominaga (Kyushu Institute of Technology, Japan)
Mohammad Al-Shabi (University of Sharjah, United Arab Emirates)
Mohammad Biglarbegian (University of Guelph, Canada)
Mou Chen (Nanjing University of Aeronautics and Astronautics, P. R. China)
Nan Xiao (Northeastern University at Qinhuangdao, P. R. China)
Nicolas Monmarché (university of Tours, France)
Noritaka Sato (Nagoya Institute of Technology, Japan)
Norrima Mokhtar (University of Malaya, Malaysia)
Palakorn Tantrakool (King Mongkut’s Institute of Technology, North Bankok, Thailand)
Ping Wang (North China Electric Power University, P. R. China)
Pierre David (University of Strasbourg, France)
Pierre Willaume (University of Strasbourg, France)
Rizauddin bin Ramli (Universiti Kebangsaan Malaysia, Malaysia)
Roman Lavrenov (Kazan Federal University, Russia)
Ruztamreen bin Jenal (Universiti Teknikal Malaysia Melaka, Malaysia)
Ryohei Anshi (Kyushu Institute of Technology, Japan)
R.K.P.S. Ranaweera (University of Moratuwa, Sri Lanka)
Satoshi Ikeda (The University of Miyazaki, Japan)
Sanjay S. Joshi (University of California, USA)
Seong-Ik Han (Pusan National University, Korea)
Shahriman Abu Bakar (Universiti Malaysia Perlis, Malaysia)
Shanbi Wei (Chongqing University, P. R. China)
Shihao Sun (Beihang University, P. R. China)
Shin-ichi Asakawa (Tokyo Woman's Christian University, Japan)
Shin Wakitani (Hiroshima University)
Shumin Fei (Southeast University, P. R. China)
Shyi-Ming Chen (National Taichung University of Education, Taiwan)
Stephen Wilkerson (York College of Pennsylvania, USA)
Takashi Kuremoto (Nippon Institute of Technology, Japan)
Takayoshi Yamada (Gifu University, Japan)
Takuya Fuginaga (Fukuoka University, Japan)
Takuya Nanami (The University of Tokyo. Japan)
Taishiro Kishimoto (Keio University, Japan)
Taisuke Akimoto (Kyushu Institute of Technology, Japan)
Tarik bin Abd Latef (Universiti Malaya, Malaysia)
Taro Shibanoki (Okayama University, Japan)
Tetsuro Katayama (The University of Miyazaki, Japan)
Thomas Noel (University of Strasbourg, France)
Thunyaseth Sethaput (Thammasat University, Thailand)
6©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

Tianping Zhang (Yangzhou University, P. R. China)
Tomohiko Takagi (Kagawa University, Japan)
Toru Hiraoka (University of Nagasaki, Japan)
Toshihiro Inukai (DENSO Wave Incorporated, Japan)
Toshinori Nawata (Kumamoto National College of Technology, Japan)
Tsunehiro Yoshinaga (Tokuyama National College of Technology, Japan)
Ussath Martin (The University of Potsdam, Germany)
Wan Khairunizam Wan Ahmad (Universiti Malaysia Perlis, Malaysia)
Weicun Zhang (University of Science and Technology Beijing, P. R. China)
Wenhao Zheng (Beihang University, P. R. China)
Wenlin Li (Beihang University, P. R. China)
Wisanu Jiviriya (King Mongkut’s University of Technology North Bankok, Thailand)
Xiaocan Wang (Technical University Munich, Germany)
Xiaofeng Su (Beijing Institute of Astronautical Systems Engineering, P. R. China)
Xiaoyan Chen (Tianjin University of Science and Technology, P. R. China)
Xiaoyan Fu (Capital Normal University, P. R. China)
Xuemei Ren (Beijing Institute of Technology, P. R. China)
Xuhui Lu (Beihang University, P. R. China)
Yan Cui (Shanxi Normal University, P. R. China)
Yasunori Takemura (Nishinippon Institute of Technology, Japan)
Yo Horikawa (Kagawa University, Japan)
Yongqiang Qi (China University of Mining and Technology, P. R. China)
Yoshihiro Kita (University of Nagasaki, Japan)
Youji Kawamura (Kindai University, Japan)
Yu-an Zhang (Qinghai University, P. R. China)
Yue Lin (Beijing Institute of Control Engineering, P.R. China)
Yueqian Liang (China Academy of Elecronics and Information Technology, P. R. China)
Yuji Minami (National Institute of Technology, Ube College, Japan)
Yunju Chen ((Shiga University, Japan)
Yunzhong Song (Henan Polytechnic University, P. R. China)
Zacharie Mbaitiga National Institute of Technology, Okinawa College, Japan)
Zakri bin Ghazali (Universiti Malaysia Pahang, Malaysia)
Zengqiang Chen (Nankai University, P. R. China)
Zhao Peng (Huazhong University of Science and Technology, P. R. China)
Zhengquan Yang (Civil Aviation University of China, P. R. China)
Zhongxin Liu (Nankai University, P. R. China)
Zuradzman bin Mohamad Razlan (Universiti Malaysia Perlis, Malaysia)
LOCAL ARRANGEMENT COMMITTEE
Makoto Sakamoto (University of Miyazaki, Japan)
Masanori Sugisaka (ALife Robotics Corporation Ltd., Japan)
Takao Ito (Hiroshima University, Japan)

7©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

HISTORY
The International Conference on Artificial Life and Robotics (ICAROB) resulted from the
AROB-symposium (International Symposium on Artificial Life and Robotics) whose firstedition was held in 1996 and the eighteenth and last edition in 2013. The AROB
symposium was annually organized by Oita University and ALife Robotics Corporation
Ltd., under the sponsorship of the Science and Technology Policy Bureau, the Ministryof Education, Science, Sports, and Culture (Monbusho), presently, the Ministry of
Education, Culture, Sports, Science, and Technology (Monkasho), Japanese Government,
Japan Society for the Promotion of Science (JSPS), the Commemorative Organization for
the Japan World Exposition ('70), Air Force Office of Scientific Research, Asian Office ofAerospace Research and Development (AFOSR/AOARD), USA. I would like to express mysincere thanks to not only Monkasho (annually fund support from 1996 to 2013) but also
JSPS, the Commemorative Organization for the Japan World Exposition ('70), and various
other Japanese companies for their repeated support. The old symposium (thissymposium has been held every year at B-Con Plaza, Beppu, Oita, Japan except in Oita,
Japan (AROB 5th ’00) and in Tokyo, Japan (AROB 6th ’01).) was organized by theInternational Organizing Committee of AROB and was co-operated by the Santa Fe
Institute (USA), RSJ, IEEJ, ICASE (Now ICROS) (Korea), CAAI (P. R. China), ISCIE, IEICE, IEEE
(Japan Council), JARA, and SICE. The old AROB-symposium expanded much by absorbing
much new knowledge and technologies into it. This history and character of the former
AROB symposiums are passed on the current ICAROB conference and to thesejournals, Journal of Robotics, Networking and Artificial Life (JRNAL)(vol1-8) indexed by
SCOPUS & ESCI and Journal of Robotics, Networking and Artificial Life (JRNAL)(vol9-)
indexed by SCOPUS & ESCI and Journal of Advances in Artificial Life Robotics (JAALR).
From now on, ALife Robotics Corporation Ltd. is in charge of management of both theconference and the journals. The future of the ICAROB is brilliant from a point of viewof yielding new technologies to human society in the 21st century. We also expect toestablish an international research institute on Artificial Life and Robotics in the futurewith the help of Japanese Government and ICAROB. This conference invites you all.

8
©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

AIMS AND SCOPE
The objective of this conference is the development of new technologies for artificial lifeand robotics which have been recently born in Japan and are expected to be applied invarious fields. This conference presents original technical papers and authoritative stateof-the-art reviews on the development of new technologies concerning robotics,
networking and artificial life and, especially computer-based simulation and hardware
for the twenty-first century. This conference covers a broad multidisciplinary field,
including areas such as:
Artificial intelligence & complexity
Artificial living
Artificial mind research
Artificial nervous systems for robots
Artificial sciences
Bipedal robot
Brain science and computing
Chaos
Cognitive science
Computational Molecular biology
Computer graphics
Data mining
Disasters robotics
DNA computing
Empirical research on network and MOT
Environment navigation and localization
Evolutionary computations
Facial expression analysis, music recommendation and augmented reality
Foundation of computation and its application
Fuzzy control
Genetic algorithms
Human-welfare robotics
Image processing
Insect-like aero vehicles
Intelligence in biological systems
Intelligent control
Management of technology
Medical surgical robot
Micro-machines
Multi-agent systems
Nano-biology
Nano-robotics
Networking
Neural circuits
Neuro-computer
9©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

Neuromorphic Systems
Neuroscience
Pattern recognition
Quantum computing
Reinforcement learning system & genetic programing
Robotics
Software development support method
System cybernetics
Unmanned underwater vehicles
Unmanned Aerial Systems Technologies
Unmanned Aerial Systems designing, controls and navigation
Unmanned Aero vehicles
Virtual reality
Visualization
Hardware-oriented submissions are particularly welcome. This conference will discuss
new results in the field of artificial life and robotics
COPYRIGHTS
Accepted papers will be published in the proceeding of The 2024 International
Conference on Artificial Life and Robotics (ICAROB2024) by ALife Robotics Corp. Ltd.
Copyright belongs to ALife Robotics Corp. Ltd. Some of high-quality papers in theproceeding will be requested to re-submit their papers for the consideration of
publication in Journal of Robotics, Networking and Artificial Life indexed by SCOPUS &ESCI (JRNAL)(vol.1-8) & Journal of Robotics, Networking and Artificial Life(vol.9-) indexed
by SCOPUS & ESCI and Journal of Advances in Artificial Life Robotics under agreement ofboth Editor-in- Chief Dr. Masanori Sugisaka and 3 reviewers. All correspondence related
to the conference should be addressed to ICAROB Office.
ICAROB Office
ALife Robotics Corporation Ltd.
4-Go, 8-Ban, Higshi 2Cyome, Handadai, Oita 870-1108, JAPANTEL/FAX：+81-97-597-7760
E-MAIL：icarob@alife-robotics.co.jp
Home Page：https://alife-robotics.co.jp/

10©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

MESSAGES

Masanori Sugisaka
General Chair of ICAROB

Masanori Sugisaka
General Chair(President, ALife Robotics Corp.,
Ltd, Japan)

It is my great honor to invite you all to The 2024 International Conferenceon Artificial Life and Robotics (ICAROB 2024) to be held at J:COM
HorutoHall, Oita, Japan, 2024. This Conference is changed as the old
symposium from the first (1996) to the Eighteenth (2013) annually whichwere organized by Oita University and ALife Robotics Corporation Ltd.
under the sponsorship of the Science and Technology Policy Bureau, theMinistry of Education, Science, Sports, and Culture (Monbusho), presently,
the Ministry of Education, Culture, Sports, Science, and Technology(Monkasho), Japanese Government, Japan Society for the Promotion of
Science (JSPS), The Commemorative Organization for the Japan World
Exposition (’70), Air Force Office of Scientific Research, Asian Office ofAerospace Research and Development (AFOSR/AOARD), USA. I would liketo express my sincere thanks to not only Monkasho (annually fund support
from 1996 to 2013) but also JSPS, the Commemorative Organization for theJapan World Exposition (’70), Japanese companies for their repeatedsupport.
The old symposium was organized by International Organizing Committeeof AROB and was co-operated by the Santa Fe Institute (USA), RSJ, IEEJ,ICASE (Now ICROS) (Korea), CAAI (P. R. China), ISCIE, IEICE, IEEE (Japan
Council), JARA, and SICE. The old AROB symposium was growing up by
absorbing many new knowledge and technologies into it. This history andcharacter was inherited also from ICAROB2014(The 2014 International
Conference on Artificial Life and Robotics, included a series of ICAROB
proceedings indexed by SCOPUS and CPCI-Web of Science now. From now
on, ALife Robotics Corporation Ltd. is in charge of management. This year
we have The 2024 International Conference on Artificial Life and Robotics(ICAROB2024) (29th AROB Anniversary). The future of The ICAROB isbrilliant from a point of view of yielding new technologies to human society
in 21st century. I have founded Robot Artificial Life Society in 2017/12/07
together with Professor at Hiroshima University Takao Ito and Professor at
University of Miyazaki Makoto Sakamoto. I hope that fruitful discussions
and exchange of ideas between researchers during Conference(ICAROB2024) will yield new merged technologies for happiness of humanbeings and, hence, will facilitate the establishment of an international joint
research institute on Artificial Life and Robotics in future.

11©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

Yingmin Jia
Co-General Chair of ICAROB

Yingmin Jia
Co-General Chair(Professor, Beihang University,
P.R. China)

It is my great pleasure to invite you to The 2024 International Conferenceon Artificial Life and Robotics (ICAROB 2024), will be held at J:COM Horuto
Hall, Oita, JAPAN, from February 22 to 25, your understanding and support
will be the strongest driving force for us to organize the meeting well.
ICAROB develops from the AROB that was created in 1996 by Prof.
Masanori Sugisaka and will celebrate her 29th birthday in 2024. So far
many important results have been presented at the past meetings andhave a profound impact on artificial life and robotics. Doubtless, it is reallyone of the most famous international conferences in the field of artificial
intelligence and attract wide interests among scientist, researchers, andengineers around the world.
For a successful meeting, many people have contributed their great efforts
to the ICAROB. Here, I would like to express my special thanks to all authors
and speakers, and the meeting organizing team for their excellent works.
Looking forward to seeing you at the ICAROB2024.

12©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

Takao Ito
Co-General Chair of ICAROB

Takao Ito
Co-General Chair(Professor Hiroshima
University, Japan)

It is my great honor and pleasure to invite you all to the 2024 International
Conference on Artificial Life and Robotics (ICAROB 2024).
The ICAROB has its long history. First launched in 1996 as ISAROB, thisformer organization of ICAROB, was developed under the strong leadershipand yeoman efforts of the President—the internationally famous Professor
Masanori Sugisaka, who is widely acknowledged as the father of our AROB
conference. Our conference has brought together many research scholars,
faculty members, and graduate students from all over the world, andpublished numerous manuscripts in high-quality proceedings as well as
highly reputed journals every year.
Over the years, dramatic improvements have been made in the field ofartificial life and its applications. The ICAROB has provided a foundation forunifying the exchange of scientific information on the studies of man-madesystems that exhibit the behavioral characteristics of natural living systems,
including software, hardware, and wetware. Our conference shapes thedevelopment of artificial life, extending our empirical research beyond theterritory circumscribed by life-as-we-know-it and into the domain of lifeas-it-could–be. It will provide us a good place to present our new research
results, innovative ideas, and valuable information about artificial
intelligence, complex systems theories, robotics, and management oftechnology.
The conference site is Horuto Hall, one of the most famous international
convention centers in Oita city, Japan. You can find many fantastic scenic
spots and splendid historical places in Oita city. Enjoy your stay!
I eagerly look forward to personally meeting you during the ICAROB 2024
and to sharing a most pleasant, interesting, and fruitful conference withyou. Do come and make this conference a fruitful, productive as well as
enjoyable event!

13©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

Ju-Jang Lee
Co-General Chair of ICAROB

Ju-Jang Lee
Co-General Chair(Honorary professor, KAIST)The First International Conference on Artificial Life and Robotics (ICAROB)
was held in Oita City, Oita, Japan from Jan. 11th to 13th, 2014. This
year’s Conference will be held amidst the high expectation of theincreasingly important role of the new interdisciplinary paradigm ofscience and engineering represented by the field of artificial life androbotics that continuously attracts wide interests among scientist,
researchers, and engineers around the globe.
Distinguished researchers and technologists from around the world arelooking forward to attending and meeting at ICAROB. ICAROB is becoming
the annual excellent forum that represents a unique opportunity for theacademic and industrial communities to meet and assess the latest
developments in this fast-growing artificial life and robotics field. ICAROB
enables them to address new challenges, share solutions, discussresearch directions for the future, exchange views and ideas, view theresults of applied research, present and discuss the latest developmentof new technologies and relevant applications.
In addition, ICAROB offers the opportunity of hearing the opinions ofwell-known leading experts in the field through the keynote sessions,provides the bases for regional and international collaborative research,and enables to foresee the future evolution of new scientific paradigms
and theories contributed by the field of artificial life and robotics andassociated research area. The twenty-first century will become thecentury of artificial life and intelligent machines in support of humankind
and ICAROB is contributing through wide technical topics of interest that
support this direction.
It is a great honor for me as a Co-General Chair of the 11th ICAROB 2024
to welcome everyone to this important event. Also, I would like to extend
my special thanks to all authors and speakers for contributing theirresearch works, the participants, and the organizing team of the 11th
ICAROB.
I’m looking forward to meeting you at the 11th ICAROB in on line andwishing you all the best.

14©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

GENERAL SESSION TOPICS
GS1 Machine Learning & Neural Network & Artificial
Life (8)
GS3 Image Processing II (3)GS2 Image Processing I (5)GS5 Applications I (4)GS6 Applications II (4)GS4 Robotics (6)GS7 Applications III (6)ORGANIZED SESSION TOPICS
OS1 Intelligent Life and Cybersecurity (6)OS2 Pattern Recognition and Robotics I (5)OS3 Pattern Recognition and Robotics II (9)OS4 Pattern Recognition and Robotics III (10)OS5 Intelligent Life and Robotics (6)OS6 Intelligence and Optimization (5)OS7 Deep Learning and its Applications (4)OS8 Intelligent Control (5)OS9 Software Development Support Method (4)OS10 Intelligent Life and Robotics I (6)OS11 Intelligent Life and Robotics II (6)OS12 Machine Learning and its Applications (4)OS13 Robot Control (10)OS14 Robotic Manipulation (3)OS15 Artificial Intelligence for Embedded Systems and OS16 Industrial Artificial Intelligence Robotics (4)Robotics (6)OS17 Electronics and Kansei Engineering Based on ETT OS18 Computer and Information Engineering (12)theory (5)OS19 Natural Computing (3)OS20 Advances in Field Robotics and Their
Applications (15)OS21 Human-Machine Interface (3)OS22 Mathematical Informatics (8)OS23 Industrial Revolution (5)OS24 Robotics and Intelligent Casting (5)OS25 Research Towards the Sustainable Development
Goals (SDG’s) (10)OS27 Post-narratological Approaches to Cognition in
Humans and Robots (5)OS26 Navigating the Digital Frontier: Innovations in theAge of Industry Revolution 4.0 (11)15©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

TIME TABLE (2/22)

in Japan time

2/22(Thu.) 17:30-19:30

Welcome Party (Oita Century Hotel)

2/25(Sun.) 15:30-16:00

Farewell party (Room 406 ZOOM ID: 823 7614 7664)

TIME TABLE (2/23)
2/23(Fri.)

Room 403
on-siteRoom 404
on-siteRoom 405
on-site, online(ZOOM ID: 851 9389 0058)

OS24 Robotics and Intelligent
Casting (5)Chair: Jiwu Wang

OS9 Software Development
Support Method (4)Chair: Tetsuro Katayama

GS2 Image Processing I (5)Chair: Yui Tanjo

9:4010:00-11:15

Room 406
on-site, online(ZOOM ID: 823 7614 7664)

Room 407
online(ZOOM ID: 883 1147 2552)

OS23 Industrial Revolution (5)Chair: Hazry Desaon-site, online

OS13-1 Robot Control (5)Chair: Yizhun Peng

Registration (407)

online11:15-11:30
Chair: Marion Oswald (Room 302, 303)
Opening ceremony

11:30-12:00

12:00-13:00

OS2 Pattern Recognition andRobotics I (5)Chair: Fengzhi Dai
Will be end at 12:45 online

OS13-2 Robot Control (5)Chair: Yizhun Peng
Will be end at 12:45

Lunch

13:00-14:00

Chair: Takao Ito (Room 302, 303)
Plenary Speech PS1
Tomoaki Ozaki

OS3-1 Pattern Recognition
and Robotics II (5)Chair: Fangyan LiWill be end at 14:15

Chair: Marion Oswald (Room 302, 303)
Plenary Speech PS2
Hazry Desa

OS3-2 Pattern Recognition
and Robotics II (4)Chair: Fangyan Li

14:00-14:20
14:20-15:20

15:20-15:40
15:40-17:10

Coffee break
OS1 Intelligent Life andCybersecurity (6)Chair: I-Hsien Liu

OS15 Artificial Intelligence forEmbedded Systems andRobotics (6)Chair Hakaru Tamukoh

GS4 Robotics (6)Chair: Jiwu Wang
onsite, online

16©ICAROB 2024 ALife Robotics Corp. Ltd.

GS3 Image Processing II (3)Chair: Seiji Ishikawa
online, on-site

OS10 Intelligent Life andRobotics I (6)Chair: Evgeni Magid

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

TIME TEBLE (2/24)
2/24(Sat.)

Room 303
on-siteRoom 403
on-siteRoom 404
on-site9:4010:00-11:15

Room 406
on-site, online(ZOOM ID: 823 7614 7664)

OS19 Natural Computing (3)Chair Marion Oswald
Will be end at 10:45

OS18-1 Computer andInformation Engineering (5)Chair: Norrima Mokhtar
online, onsite

OS22 Mathematical
Informatics (8)from 11:00 to 13:00
Chair: Takao Ito

OS18-2 Computer andInformation Engineering (2)Chair: Norrima Mokhtar
online, on-site

OS5-1 Intelligent Life andRobotics (4)Chair: Kuo-Hsien Hsia
on-site, online

OS18-3 Computer andInformation Engineering (5)Chair: Norrima Mokhtar
online, on-site

OS5-2 Intelligent Life andRobotics (2)Chair: Kuo-Hsien Hsia
on-site, online

OS21 Human-MachineInterface (3)Chair Norrima Mokhtar
online, on-site

OS16 Industrial Artificial
Intelligence Robotics (4)Chair Eiji Hayashi
on-siteGS6 Applications II (4)Chair Masayuki Fujiwara
on-siteRegistration (407)OS20-1 Advances in Field
Robotics and Their
Applications (5)Chair: Kazuo Ishii

OS17 Electronics and Kansei
Engineering Based on ETTtheory (5)Chair: Tetsuo Hattori

GS1-1 Machine Learning &Neural Network & Artificial
Life (5)Chair: Hiroki Tamura

11:15-11:30
11:30-12:00

Room 405
on-site, online(ZOOM ID: 851 9389 0058)

Coffee breakOS20-2 Advances in Field
Robotics and Their
Applications (2)Chair: Kazuo Ishii

GS1-2 Machine Learning &Neural Network & Artificial
Life (3) Chair: Hiroki Tamura
Will be end at 12:15

12:00-13:00

Lunch

13:00-14:00

Chair: Yingmin Jia (Room 302, 303)
Plenary Speech PS3 Haruhisa Okuda

14:00-14:15

Coffee break

14:15-15:30

OS20-3 Advances in Field
Robotics and Their
Applications (5)Chair: Kazuo Ishii

OS27 Post-narratologicalApproaches to Cognition in
Humans and Robots (5)Chair: Jumpei Ono(ZOOM ID: 826 8397 1634)

OS8 Intelligent Control (5)Chair: Yingmin Jia

15:30-15:40
15:40-16:25

Coffee breakOS20-4 Advances in Field
Robotics and Their
Applications (3)Chair: Kazuo Ishii

OS14 Robotic Manipulation(3) Chair: Kensuke Harada
onsite, online(ZOOM ID: 849 1028 4955)
Coffee break

16:25-16:40
16:40-17:40

OS7 Deep Learning and its
Applications (4) online
Chair Mastaneh Mokayef(ZOOM ID: 824 3203 3781)

OS12 Machine Learning and
its Applications (4)Chair: Masato Nagayoshi

GS5 Applications I (4)Chair:
onsite, online(ZOOM ID: 849 1028 4955)
Banquet (Tokiwa Kaikan)

18:30-20:30
17©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

TIME TEBLE (2/25)
Room 405
online(ZOOM ID: 851 9389 0058)

2/25(Sun.)

Room 406
online(ZOOM ID: 823 7614 7664)

9:4010:00-11:30

Registration (407)
OS6 Intelligence andOptimization (5)Chair Mastaneh Mokayef

OS26-1 Navigating the Digital Frontier: Innovationsin the Age of Industry Revolution 4.0 (6)Chair Wei Hong Lim

11:30-12:30
12:30-13:45

OS25-1 Research Towards the Sustainable
Development Goals (SDG’s) (5)Chair Ammar A.M. Al Talib

Lunch
OS4-1 Pattern Recognition and Robotics III (5)Chair Huahao Li

OS26-2 Navigating the Digital Frontier: Innovationsin the Age of Industry Revolution 4.0 (5)Chair Wei Hong Lim

13:45-14:00
14:00-15:30

Room 407
online(ZOOM ID: 883 1147 2552)

OS25-2 Research Towards the Sustainable
Development Goals (SDG’s) (5)Chair Ammar A.M. Al Talib

Coffee breakOS4-2 Pattern Recognition and Robotics III (5)Chair Huahao Li

15:30-16:00

OS11 Intelligent Life and Robotics (6)Chair: Evgeni Magid
Farewell Party (Room 406 ZOOM ID: 823 7614 7664)

18©ICAROB 2024 ALife Robotics Corp. Ltd.

GS7 Applications III (6)Chair: Kasthuri Subaramaniam

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

The 2024 International Conference on
ARTIFICIAL LIFE AND ROBOTICS(ICAROB2024)
February 22 (Thursday)
17:30-19:30

Welcome Party (Oita Century Hotel)

February 23 (Friday)
Room 302, 303
11:30-12:00 Opening Ceremony
Chair: Marion Oswald (Vienna University of Technology, Austria)
Welcome Addresses
1. General Chairman of ICAROB
2. Co-General Chairman of ICAROB
3. Co-General Chairman of ICAROB
4. Vice General Chair of ICAROB
5. Vice General Chair of ICAROB

Masanori Sugisaka (ALife Robotics Co., Ltd., Japan)
Yingmin Jia (Beihang University, China)
TaKao Ito (Hiroshima University, Japan)
Katia Passerini (Seton Hall University, USA)
Norrima Mokhtar (University of Malaya, Malaysia)

February 24 (Sunday)
Banquet: Tokiwa Kaikan
18:30-20:30
Chair: Takao Ito (Hiroshima University, Japan)
Welcome Addresses
Prof. Yingmin Jia ( Beihang University, P.R. China)
Dr. Norrima Mokhtar (University of Malaya, Malaysia)

19©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

TECHNICAL PAPER INDEX
February 23 (Friday)
9:40-Registration
Room 302, 303
11:30-12:00 Opening Ceremony
Chair: Marion Oswald (Vienna University of Technology, Austria)
13:00-14:00
Plenary Speech PS1
Chair: Takao Ito (Hiroshima University, Japan)
PS1 Developing High-Speed Working Motion of the Multi Robot in DENSO
Tomoaki Ozaki (DENSO CORPORATION, Japan)
14:20-15:20
Plenary Speech PS2
Chair: Marion Oswald (Vienna University of Technology, Austria)
PS2 Experimenting with Variable Arm Quadrotors: Realizing Dynamic Configurations forEnhanced Flight Performance
Hazry Desa (Universiti Malaysia Perlis, Malaysia)
Room 403
10:00-11:15 OS24-Robotics and Intelligent Casting (5)Chair: Jiwu Wang (Beijing Jiaotong University, China)
OS24-1

A high-performance motion planning method based on asymptotically optimal RRT
Tianbin Meng, Jiwu Wang (Beijing Jiaotong University, China)

OS24-2

CSM-RRT*: an improved RRT* algorithm based on constrained samplingmechanism
Hang Yang, Jiwu Wang (Beijing Jiaotong University, China)
Xueqiang Shang (Aero Engine Corporation of China, China)

OS24-3

Small Sample Object Detection Based on Improved YOLOv5
Yuxuan Gao (Beijing Jiaotong University, China),
Jiwu Wang (Beijing Jiaotong University, China),
Zixin Li (Aero Engine Corporation of China)

20©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS24-4

Research on Gas Pore Prediction Method Based on Sand Core Characteristic Time
Xiaolong Wang (Beijing Jiaotong University, China), Qihua Wu (Weichai Power Co.,
Ltd. China), Jiwu Wang (Beijing Jiaotong University, China), Jinwu Kang (Tsinghua
University,China),
Na Li (Weichai Power Co., Ltd. China), Yucheng Sun(Weichai Power Co., Ltd. China)

OS24-5

Optimization Analysis of a Deep Learning-Based Model for Predicting Temperature
Fields in the Solidification Process of Castings
Yahui Yang (Beijing Jiaotong University, China), Jiwu Wang (Beijing Jiaotong
University, China),
Jinwu Kang (Tsinghua University,China)

15:40-17:10 OS1 Intelligent Life and Cybersecurity (6)Chair: I-Hsien Liu (National Cheng Kung University, Taiwan)
Co-Chair: Chu-Fen Li (National Formosa University, Taiwan)
Co-Chair: Pang-Wei Tsai (National Cheng Kung University, Taiwan)
OS1-1

Detecting abnormal operations in ICS using finite-state machines
Pei-Wen Chou, Nai-Yu Chen, Jung-Shian Li, I-Hsien Liu(National Cheng Kung University, Taiwan)

OS1-2

Industrial Control System State Monitor Using Blockchain Technology
Yun-Hao Chang, Tzu-En Peng, Jung-Shian Li, I-Hsien Liu(National Cheng Kung University, Taiwan)

OS1-3

Enhancing Dam Security and Water Level Alerting with Blockchain Technology
YingCheng Wu, Jung-Shian Li, Chu-Fen Li, I-Hsien Liu(National Cheng Kung University, Taiwan)

OS1-4

MiniDAM: A Dam Cybersecurity Toolkit
Tzu-En Peng, Meng-Wei Chang, Yun-Hao Chang, Jung-Shian Li, I-Hsien Liu(National Cheng Kung University, Taiwan)

OS1-5

Case Study of Network-Based Intrusion Detection System Deployment in Industrial
Control Systems with Network Isolation
Nai-Yu Chen, Pei-Wen Chou, Jung-Shian Li, I-Hsien Liu(National Cheng Kung University, Taiwan)

OS1-6

The AI integration service innovation model of real estate industry in Taiwan
Li-Min Chuang, Chih-Hung Chen (Chang Jung Christian University, Taiwan)

21©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

Room 404
10:00-11:00 OS9 Software Development Support Method (4)Chair: Tetsuro Katayama (University of Miyazaki, Japan)
Co-Chair: Tomohiko Takagi (Kagawa University, Japan)
OS9-1

An Improved Conversion Technique from EPNAT Models to VDM++ Specifications for
Simulation of Abstract Software Behavior
Sho Matsumoto1, Ryoichi Ishigami1, Tetsuro Katayama2, Tomohiko Takagi1
1
Kagawa University, Japan, 2 University of Miyazaki, Japan

OS9-2

Prototype of RAGESS Which Is a Tool for Automatically Generating SwiftDiagrams
to Support iOS App Development
Haruki Onaga*, Tetsuro Katayama*, Yoshihiro Kita†, Hisaaki Yamaba*, Kentaro
Aburada*, and Naonobu Okazaki**University of Miyazaki, Japan, †University of Nagasaki, Japan

OS9-3

Extension to Support Types and Operation/Function Definitions in BWDM toGenerate Test Case Tool from the VDM++ Specification
Shota Takakura*, Tetsuro Katayama*, Yoshihiro Kita†, Hisaaki Yamaba*, Kentaro
Aburada*, and Naonobu Okazaki**University of Miyazaki, Japan, †University of Nagasaki, Japan

OS9-4

Proposal of ASLA Which Is a Segmentation and Labeling Tool for Document Images
Based on Deep Learning
Kanta Kakinoki*, Tetsuro Katayama*, Yoshihiro Kita†, Hisaaki Yamaba*, Kentaro
Aburada*, and Naonobu Okazaki**University of Miyazaki, Japan, †University of Nagasaki, Japan

15:40-17:10 OS15 Artificial Intelligence for Embedded Systems and Robotics (6)Chair: Hakaru Tamukoh (Kyushu Institute of Technology, Japan)
Co-Chair Yuma Yoshimoto (National Institute of Technology, Kitakyushu College, Japan)
OS15-1YOLO real-time object detection on EV3-Robot using FPGA hardware Accelerator
Dinda Pramanta1, Ninnart Fuengfusin2, Arie Rachmad Syulistyo2, Hakaru Tamukoh2(1Kyushu Institute of Information Sciences, Japan) (2Kyushu Institute of Technology,
Japan)

OS15-2

A Low Computational Cost Hand Waving Action Recognition System with Echo State
Network for Home Service Robots
Hiromasa Yamaguchi1, Akinobu Mizutani1, Arie Rachmad Syulistyo1, Yuichiro
Tanaka1, Hakaru Tamukoh1 (1Kyushu Institute of Technology, Japan)

OS15-3

A Rapidly Adjustable Object Recognition System through Language Based Prompt
Engineering
Naoki Yamaguchi1, Tomoya Shiba1, Kosei Isomoto1, Hakaru Tamukoh1(1Kyushu Institute of Technology, Japan)

22©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS15-4Development of a SayCan-based task planning system capable of handling abstract
nouns
Kosei Yamao1, Daiju Kanaoka1, Kosei Isomoto1, Akinobu Mizutani1, Yuichiro Tanaka1,
Hakaru Tamukoh1, (1Kyushu Institute of Technology, Japan)

OS15-5

RoboCup@Home 2023: Stickler for the Rules Task Solutions
Tomoya Shiba1, Hakaru Tamukoh1, (1Kyushu Institute of Technology, Japan)

OS15-6

Offloading Intellectual Processing from Home Service Robots to Edge Devices
Yuma Yoshimoto1, Mizuki Kawashima1, Shun Yonehara1(1National Institute of Technology, Kitakyushu College, Japan)

Room 405
10:00-11:15 GS2 Image Processing I (5)Chair: Yui Tanjo (Kyushu Institute of Technology, Japan)
GS2-1

A Method for Embedding Multiple Photographic Images in a Photographic Image
Naoki Kouno, Kanya Goto, Toru Hiraoka (University of Nagasaki, Japan)

GS2-2

Generation of Flowing-Line Images Using Vertical and Horizontal Smoothing Filters
Karin Kuroki, Toru Hiraoka (University of Nagasaki, Japan)

GS2-3

Human Motion Recognition from Multiple Directions and Its Gait Cycles Analysis
Miki Ooba, Yui Tanjo (Kyushu Institute of Technology, Japan)

GS2-4

A Method of Improving the QOL of the People with Visual Impairment by MY VISION
Shun Kitazumi, Yui Tanjo (Kyushu Institute of Technology, Japan)

GS2-5

Human Behavior Segmentation and Recognition Using a Single-camera
Jing Cao, Yui Tanjo (Kyushu Institute of Technology, Japan)

15:40-17:10 GS4 Robotics (6)Chair: Jiwu Wang (Beijing Jiaotong University, China)
GS4-1

Development and evaluation of a learning support robot for vector learning
Kosei Machida, Shinichi Imai (Tokyo Gakugei University, Japan)

GS4-2

Feasibility Study on Methods to Measure the Strain on Young Children's Bodies
Sachiko Kido, Praveen Nuwantha Gunaratne, Hiroki Tamura(University of Miyazaki, Japan)

23©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

GS4-3

An Integration of Contact Force Models with Multibody Dynamics Analyses for
Human Joint Mechanisms and Effects of Viscoelastic Ground Contact
Shintaro Kasai1, Dondogjamts Batbaatar2, Hiroaki Wagatsuma1(1Kyushu Institute of Technology, Japan; 2Mongolian University of Science andTechnology, Mongolia)

GS4-4

Haptic Sensation Enhancement via the Stochastic Resonance Effect and Its
Application to Haptic Feedback for Myoelectric Prosthetic Hands
Yoshitaka Mizumoto, Taro Shibanoki (Okayama University, Japan)

GS4-5

PID Parameter Tuning of a Low-Cost DC Motor Speed Control for Mobile Robot
Application
Munkh-Erdene Ayurzana1, Erkhembayar Gankhuyag1, Naranbaatar Erdenesuren1,
Dondogjamts Batbaatar1(1Mongolian University of Science and Technology, Mongolia)

GS4-6

Reinforcement Learning DDPG Algorithm Based Wheeled Mobility Aid Robot Control
Methods
Junkai Li, Mohd Rizon Mohamad Juhari, Tiang Sew Sun (UCSI University, Malaysia)

Room 406
10:00-11:15 OS23 Industrial Revolution (5)Chair: Hazry Desa (Universiti Malaysia Perlis (UniMAP), Malaysia)
OS23-1

Investigating the Engineering Interventions in the Conservation of Malaysia
Heritage Structures: A Review on Preserving Historical Edifices Through Advance
Civil Engineering Techniques
Muhammad Azizi Azizan, Nurfadzillah Ishak, Hazry Desa (UniMAP, Malaysia)

OS23-2

Drones and Data: A Comprehensive Exploration of UAVs in Data Mining
Muhammad Azizi Azizan, Nurfadzillah Ishak, Hazry Desa (UniMAP, Malaysia)

OS23-3Development of Variable Arm to Control the Manoeuvrability of Quadrotor
L. Y. Hong, H. Desa and M. A. Azizan (UniMAP, Malaysia)
M. H. Tanveer (Kennesaw State University, USA)OS23-4Development of IOT-Enabled Smart Water Metering System
S. D. Wen, H. Desa and M. A. Azizan (UniMAP, Malaysia)
A. -S. T. Hussain (Al-Kitab University, Iraq)
M. H. Tanveer and R. Patan (Kennesaw State University, USA)OS23-5

Object Detection and Instance Segmentation with YOLOV8: Progress andLimitations
L. J. Lee, H. Desa and M. A. Azizan (UniMAP, Malaysia)
A. -S. T. Hussain (Al-Kitab University, Iraq)
M. H. Tanveer (Kennesaw State University, USA)

24
©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

11:30-12:45 OS2 Pattern Recognition and Robotics I (5)Chair: Fengzhi Dai (Tianjin University of Science and Technology, China)
Co-Chair: Yunzhong Song (Henan Polytechnic University, China)
OS2-1

A Study on Sales Patterns for Vegetable Products in Retail Stores
Yuhao Zhang 1, Shuangshuang Ma 1, Jiashuai Wang 1, Fengzhi Dai 1, Lijiang Zhang 2(1 Tianjin University of Science and Technology, China, 2 Xinjiang Shenhua
Biotechnology Co., Ltd, China)

OS2-2

Research and Implementation of Cooperative Control for ROS Mobile Robot
Saijie Zhang, Huailin Zhao
(Shanghai Institute of Technology, China)

OS2-3

An OpenCV-based Method for Workpiece Residue Image Processing
Jiaxin Wang, Hao He, Fangyv Liu, Fengzhi Dai(Tianjin University of Science and Technology, China)

OS2-4

On Nonblockingness Verification and Enforcement of Controlled Nondeterministic
Discrete-Event Systems
Xiang Ren, Zipei Wang (Tianjin University of Science and Technology, China)

OS2-5Modeling and Reachability Verification of Controlled Nondeterministic Finite-State
Automata
Zipei Wang, Xiang Ren (Tianjin University of Science and Technology, China)

15:40-16:25 GS3 Image Processing II (3)Chair: Seiji Ishikawa (Kyushu Institute of Technology, Japan)
GS3-1

Online Classroom Student Engagement Analysis using Enhanced YOLOv5
Shuai Wang, Abdul Samad Shibghatullah (UCSI University, Malaysia)

GS3-2

A Method of Recognizing Body Movements Based on a Self-viewpoint Video
Iichirou Moribe, Yui Tanjo (Kyushu Institute of Technology, Japan)

GS3-3

Supporting Safe Walk of a Visually Impaired Person at a Station Platform based on
MY VISION
Shintaro Yamada, Yui Tanjo, Seiji Ishikawa (Kyushu Institute of Technology, Japan)

25©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

Room 407
10:00-11:15 OS13-1 Robot Control (5)Chair: Yizhun Peng (Tianjin University of Science and Technology, China)
OS13-1Intelligent Logistics Handling Robot: Design, Control, and Recognition
Yanchao Bi, Jiale Cheng, Limei Wang, Yizhun Peng
(Tianjin University of Science and Technology, China)

OS13-2

Greenhouse Design Using Visual Recognition and IoT Technology
Yuntian Xia, Yizhun Peng
(Tianjin University of Science and Technology, China)

OS13-3

Design of Modular Photovoltaic Environmentally Friendly Portable Stroller
Suqing Duan, Yizhun Peng
(Tianjin University of Science and Technology, China)

OS13-4"Teenage Mutant Ninja Turtles" - Design of a Bionic Quadrupedal Rescue Robot
Hongpi Zhao, Yingfan Zhu, Zhihan Zhao, Xin Liang, Lei Lv, Yizhun Peng (Tianjin
University of Science and Technology, China)

OS13-5

Design of Grass Lattice Planter for Complex Environment Based on Adaptive
Suspension Technology
Shaokai Tian, Wenqi Fu, Yizhun Peng
(Tianjin University of Science and Technology, China)

11:30-12:45 OS13-2 Robot Control (5)Chair: Yizhun Peng (Tianjin University of Science and Technology, China)
OS13-6

Design of Intelligent Ecological Multifunctional Plant and Animal Breeding System
Suqing Duan, Yuntian Xia, Siyi Wang, Yizhun Peng (Tianjin University of Scienceand Technology, China)

OS13-7

Design of a Fully Automated Logistics Handling Platform
Hongpi Zhao, Jianfeng Qin, Yizhun Peng (Tianjin University of Science andTechnology, China)

OS13-8Design and Application of AI-based Brush Calligraphy and Painting Robot
Haibo Li, Yizhe Sun, Shuxin Wang, Yizhun Peng (Tianjin University of Science andTechnology, China)

OS13-9Intelligent Wheelchair System: Non-contact Heart Rate and Body Temperature
Measurement
Dongpo Ma, Junsheng Zhang, Yizhun Peng (Tianjin University of Science andTechnology, China)

26©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS13-10

Recognition and Localization Method for Automotive Axle Holes in Assembly Robots
Junsheng Zhang, Dongpo Ma, Yizhun Peng (Tianjin University of Science andTechnology, China)

13:00-14:15 OS3-1 Pattern Recognition and Robotics II (5)Chair: Fangyan Li (Tianjin University of Science and Technology, China)
Co-Chair: Haozhe Sun (Tianjin University of Science and Technology, China)
OS3-1

Detection and Identification of Daylily Maturity Based on YOLOv8
Fangyan Li (Tianjin University of Science and Technology, China)

OS3-2

Chaos Synchronization and Circuit Design of Chen System and Lü System withDifferent Structures
Haozhe Sun (Tianjin University of Science and Technology, China)

OS3-3

A Parking Space Recognition Method Based on Digital Image Technology
Hao He, Fangyv Liu, Jiaxin Wang (Tianjin University of Science and Technology,
China)OS3-4

A Design of Intelligent Handling Robot Based on AT89C52
Fangyv Liu, Jiaxin Wang, Hao He (Tianjin University of Science and Technology,
China)

OS3-5

Application and Differences of Robotic Arms, Traditional Machines and Manual
Work in Production
Xue Yang, Ying Su, Yuping Mei, Haiquan Wang (Tianjin University of Science andTechnology, China)

14:20-15:20 OS3-2 Pattern Recognition and Robotics II (4)Chair: Fangyan Li (Tianjin University of Science and Technology, China)
Co-Chair: Haozhe Sun (Tianjin University of Science and Technology, China)
OS3-6

A Deep Exploration of the Mounting Issues Related to Six Rotor UVA
Yuping Mei, Ying Su, Xue Yang (Tianjin University of Science and Technology, China)

OS3-7

The Application of Hexacopter UAV in The Field of Climbing Evasion
Ying Su, Yuping Mei, Xue Yang (Tianjin University of Science and Technology, China)

OS3-8

Deep Learning and Embedded Based Operational Safety System for Special Vehicles
Haoran Gong, Yumei Huang, Jiahao Xie (Tianjin University of Science andTechnology, China)

27©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS3-9

"Green Fruit" - Intelligent Traceable Agricultural Product Production and Marketing
Platform Based on Blockchain Technology
Yumei Huang, Jiahao Xie, Haoran Gong, Ziyue Xiao (Tianjin University of Science andTechnology, China)

15:40-17:10 OS10 Intelligent Life and Robotics (6)Chair: Evgeni Magid (Kazan Federal University, Russia)
Co-Chair: Kuo-Hsien Hsia (National Yunlin University of Science and Technology, Taiwan)
OS10-1A Design of a Modular Mobile Robot for Rescue Operations
Baris Celiker, Shifa Sulaiman, Tatyana Tsoy (Kazan Federal University, Russia)

OS10-2

Implementation of Bug1 and Bug2 Path Planning Algorithms for TurtleBot Using
ROS Noetic
Ilya Spektor1, Aidar Zagirov2, Ramil Safin2, Evgeni Magid1,2(1HSE University, Russia) (2Kazan Federal University, Russia)

OS10-3

Implementation of Alg1 and Alg2 Path Planning Algorithms for Mobile Robots Using
ROS Noetic
Anastasia Yankova1, Timur Gamberov2, Tatyana Tsoy2(1HSE University, Russia) (2Kazan Federal University, Russia)

OS10-4

Implementation of VisBug-21 and VisBug-22 Path Planning Algorithms Using ROS
NoeticViktoriia Mirzoian1, Maxim Mustafin2, Evgeni Magid1,2(1HSE University, Russia) (2Kazan Federal University, Russia)

OS10-5DistBug path planning algorithm package for ROS Noetic
Alexander Pak1, Alexander Eremin2, Tatyana Tsoy2(1HSE University, Russia) (2Kazan Federal University, Russia)

OS10-6On sensor modeling in Gazebo simulator
Niez Yuldashev, Alexandra Dobrokvashina, Roman Lavrenov(Kazan Federal University, Russia)

28©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

February 24 (Saturday)
9:40-Registration
Room 302, 303
13:00-14:00
Plenary Speech PS3
Chair: Yingmin Jia
PS3 Artificial Intelligence and Technologies of Arm-type and Mobile Robots in Industry
Haruhisa Okuda(Mitsubishi Electric Corporation, Japan)
Room 303
10:00-11:15 OS20-1 Advances in Field Robotics and Their Applications (5)Chair: Kazuo Ishii (Kyushu Institute of Technology, Japan)
Co-Chair: Keisuke Watanabe (Tokai University, Japan)
OS20-1

Image Collection Experiments of a Handy AUV for Offshore Structure Inspection
Keisuke Watanabe, Koki Amano, Shingen Urano, Yasutaka Taniguchi, Konosuke
Watanabe (Tokai University, Japan)

OS20-2

Design of Disassembly-reassembly Type USV for Coral Reef Research
Keisuke Watanabe, Koki Amano, Gaku Minato, Yasutaka Taniguchi, Konosuke
Watanabe (Tokai University, Japan)

OS20-3

Optimization method to improve visual SLAM in dynamic environment
Yufei Liu, Kazuo Ishii (Kyushu Institute of Technology, Japan)

OS20-4

Design of flexible mechanism for flexible manipulator
Huang Jiawei, Kazuo Ishii (Kyushu Institute of Technology, Japan)

OS20-5

Driver Drowsiness Detection Method based on Deep Learning
Shi Puwei, Kazuo Ishii (Kyushu Institute of Technology, Japan)

29©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

11:30-12:00 OS20-2 Advances in Field Robotics and Their Applications (2)Chair: Kazuo Ishii (Kyushu Institute of Technology, Japan)
Co-Chair: Keisuke Watanabe (Tokai University, Japan)
OS20-6An Analysis of Translational Motion for a Mobile Robot with Line-Symmetric Rollers
Arrangement
1
Kenji Kimura, 2Kazuo Ishii(1National Institute of Technology, Matsue College, Japan, 2Kyushu Institute ofTechnology, Japan)

OS20-7Development of Teaching Materials for Robot Programming for Junior High SchoolStudents: Student-Based Educational Activities
Kenji Kimura, Youta Takano (National Institute of Technology, Matsue College,
Japan)

14:15-15:30 OS20-3 Advances in Field Robotics and Their Applications (5)Chair: Kazuo Ishii (Kyushu Institute of Technology, Japan)
Co-Chair: Keisuke Watanabe (Tokai University, Japan)
OS20-8Design and Software Production of Robotics Educational Design for Elementary andJunior High School Student
Youta Takano, Kenji Kimura (National Institute of Technology, Matsue College,
Japan)

OS20-9Development of visual inspection system for low-reflective material utilizing of astring shadow
Keiji Kamei1, Tomorou Kawahara1, Yoshiyuki Daimaru2(1Nishinippon Institute of Technology, Japan) (2Nissan Motor Kyushu, Japan)

OS20-10Development of IoT-Based Remote Monitoring Module for Greenhouse Environment
to Facilitate Crop Growth Data Analysis
Moeko Tominaga, Yasunori Takemura, Junya Era, Wataru Kaishita(Nishinippon Institute of Technology, Japan)
Eiji Mizoe, Tomoyasu Furukawa(Kumamoto Fruit and Vegetable Shippers Association Co. Ltd., Japan)

OS20-11Development greenhouse environment prediction system using IoT data
Yasunori Takemura, Naoya Nishida, Moeko Tominga(Nishinippon Institute of Technology, Japan)
Eiji Mizoe, Tomoyasu Furukawa(Kumamoto Fruit and Vegetable Shippers Association Co. Ltd., Japan)

OS20-12

The Development of SaaS for Quantifying the Amount of Drifted Debris on the Coast
Ayumu Tominaga, Ryohei Komori(National Institute of Technology Kitakyushu College, Japan)
Eiji Hayashi (Kyushu Institute of Technology, Japan)

30©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

15:40-16:25 OS20-4 Advances in Field Robotics and Their Applications (3)Chair: Kazuo Ishii (Kyushu Institute of Technology, Japan)
Co-Chair: Keisuke Watanabe (Tokai University, Japan)
OS20-13Development of an antagonistic wire-driven joint mechanism capable of rapid
motion and variable stiffness
Katsuaki Suzuki1, Yuya Nishida2, Kazuo Ishii2(1Kumamoto Industrial Research Institute, Japan, 2Kyushu Institute of Technology,
Japan)

OS20-14

Gakken Hills Interdisciplinary Ekiden Competing with Humans, Animals, and Robots
Takuya Fujinaga1, Moeko Tominaga2, Daigo Katayama3, Kazuo Ishii3(1Fukuoka University, 2Nishinippon Institute of Technology, 3Kyushu Institute ofTechnology, Japan)

OS20-15

Development of a Low-Cost Underwater Robot for Research and Education
Takuya Fujinaga (Fukuoka University Japan)

16:40-17:40 OS7 Deep Learning and its Applications (4)Chair: Mastaneh Mokayef (UCSI University, Malaysia)
Co-Chair: Takao Ito (Hiroshima University, Japan)
OS7-1

Parallel Cross Window Attention Transformer and CNN Model for Segmentation ofInstrument during Surgery
Abdul Qayyum1, M. K. A. Ahamed Khan3, Moona Mazher4, Imran Razzak5, Steven
Niederer1,2, Mastaneh Mokayef3, C.S. Hassan3, Ridzuan, A.3(1Imperial College, London, United Kingdom) (2The Alan Turning Institute, London,UK) (3UCSI University, Malaysia) (4University College London, UK) (5University of
New South Wales, Australia)

OS7-2

Magnetic Resonance Spectroscopy (MRS) Reconstruction using Style Transfer Deep
Depth wise Framework
Abdul Qayyum1, M. K. A. Ahamed Khan3, Moona Mazher4, Imran Razzak5, Steven
Niederer1,2, Mastaneh Mokayef6, C.S. Hassan6, Ridzuan, A.6(1Imperial College, London, UK) (2The Alan Turning Institute, UK)(3,6 UCSI University, Malaysia) (4University College London, UK)(5University of New South Wales, Australia)

OS7-3

Federated Learning on Brain Disease Research: Segmentation of Cerebral Small
Vessel Diseases (CSVD) using Multi-scale Hybrid Spatial Deep Learning Approach
Moona Mazher1, Abdul Qayyum2, M. K. A. Ahamed Khan3, Steven Niederer2,4,
Mastaneh Mokayef3, Ridzuan, A3, C. S. Hassan3 (1University College London, UK)(2National Heart & Lung Institute, Imperial College, London, UK) (3UCSI University,
Malaysia) (4Alan Turning Institute, London, UK)

31©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS7-4

Hybrid Classical and Quantum Deep Learning Models for Medical ImageClassification
Moona Mazher1, Abdul Qayyum2, M. K. A. Ahamed Khan3, Steven Niederer2,4,
Mastaneh Mokayef3, Ridzuan, A3, C. S. Hassan3 (1University College London, UK,) (2National Heart & Lung Institute, Imperial College, UK) (3UCSI University, Malaysia)(4Alan Turning Institute, UK)

Room 403
10:00-11:15 OS17 Electronics and Kansei Engineering Based on ETT theory (5)Chair: Tetsuo Hattori (Kagawa University, Hiroshima Institute of Technology, Japan)
Co-Chair: Yusuke Kawakami (NIT (Kagawa College), Japan)
OS17-1A High-Speed Estimation Method of Parameters in Impulse Response
Toshiki Tanaka, Ivan Tanev (Doshisha University, Japan), Tetsuo Hattori (Kagawa
University, Japan)

OS17-2

A Consideration on Amplification Function in BJT Evers-Moll Model and PTT (I)---- V-I Characteristics ---Shimon Hattori, Osamu Matoba (Kobe University, Japan), Tetsuo Hattori (Kagawa
University, Japan), Toshiki Tanaka (Kinkei System, Japan), Yusuke Kawakami (NIT(Kagawa College), Japan)

OS17-3

A Consideration on Amplification Function in BJT Evers-Moll Model and PTT (II)---- H Parameters in the Small Signal Amplifier Circuit---Shimon Hattori, Osamu Matoba (Kobe University, Japan), Tetsuo Hattori (Kagawa
University, Japan), Toshiki Tanaka (Kinkei System, Japan), Yusuke Kawakami (NIT(Kagawa College), Japan)

OS17-4

Color Image Arrangement Using Histogram Matching
Yusuke Kawakami (NIT (Kagawa College), Japan), Tetsuo Hattori (Kagawa University,
Japan), R.P.C. Janaka Rajapakse (Tainan National University of the Arts, Taiwan(R.O.C.))

OS17-5

Methodology for Creativity Oriented STEM Education Based on ETT Theory
Tetsuo Hattori, Toshihiro Hayashi, Mai Hattori, Yoshiro Imai (Kagawa University,
Japan), Asako Ohno (Osaka Sangyo University, Japan), Takeshi Tanaka (Hiroshima
Institute of Technology, Japan)

32©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

14:15-15:30 OS27 Post-narratological Approaches to Cognition in Humans and Robots(5)
Chair: Jumpei Ono (Aomori University, Japan)
Co-Chair: Hiroki Fxyma (Kobe University, Japan)
OS27-1

A Game Framework Based on the Disinformation Warfare in Russo-Ukrainian War
Jumpei Ono (Aomori University, Japan), Takashi Ogata (Iwate Prefectural University,
Japan)

OS27-2Comparative Analysis of Eye Tracking between Veteran and Novice during
Radiological Interpretation
Yuka Naito (Chuo University, Japan), Jun Nakamura (Chuo University, Japan),
Yoshinobu Ishiwata (Yokohama City University, Japan)

OS27-3Development of Notification System to Prevent Working Productivity from
Declining Caused by Increased Carbon Dioxide Concentration
KAMODA Yohei, NAKAMURA Jun (CHUO University, Japan)

OS27-4Visualization of the Skilled Physician's Gaze Characteristic during Diagnosis
Taiki Sugimoto (Chuo University, Japan), Jun Nakamura (Chuo University, Japan),and Yoshinobu Ishiwata (Yokohama City University, Japan)

OS27-5Comparative Analysis of Methods for Visualizing the Sensory Experience of Foodand Beverages
Hiroki Fxyma (Kobe University, Japan)

33©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

16:40-17:40 OS12 Machine Learning and its Applications (4)Chair: Masato Nagayoshi (Niigata College of Nursing, Japan)
Co-Chair Takashi Kuremoto (Nippon Institute of Technology, Japan)
OS12-1

Restoration of Guqin Music by Deep Learning Methods
Takashi Kuremoto*, Kazuma Fujino*, Hirokazu Takahashi*, Shun Kuremoto**, ***,
Mamiko Koshiba**, Hiroo Hieda***, Shingo Mabu**(*Nippon Institute of Technology, **Yamaguchi University, ***Institute for Future
Engineering, Japan)

OS12-2

Constructive Nurse Scheduling Using Reinforcement Learning Considering
Variations in Nurse Work Patterns
Masato Nagayoshi (Niigata College of Nursing, Japan),Hisashi Tamaki (Kobe University, Japan)

OS12-3

A Basic Study on Indicator of Transfer Learning for Reinforcement Learning
Satoshi Sugikawa, Kenta Takeoka, Naoki Kotani(Osaka Institute of Technology, Japan)

OS12-4

Machine Learning Approach to Predict Cooling Load for Existing Buildings
Makoto Ohara*, Hideo Isozaki**(*International Professional University of Technology in Osaka, Japan,**Kobe University, Japan)

Room 404
10:00-11:15 GS1-1 Machine learning& Neural Network & Artificial Life (5)Chair: Hiroki Tamura (University of Miyazaki, Japan)
GS1-1

Deep Learning Based Prediction of Heat Transfer Coefficient Using Spectrogram
Images from Boiling Sound
Fuga Mitsuyama, Ren Umeno, Tomohide Yabuki, Tohru Kamiya (Kyushu Institute ofTechnology, Japan)

GS1-2

A Study on Classification of Faulty Motor Sound Using Convolutional Neural
Networks
Jamil Md Shafayet, Praveen Nuwantha Gunaratne, Hiroki Tamura(University of Miyazaki, Japan)

GS1-3

Design and Development of a Flexible Active Ankle Joint Orthosis for Locomotion
Assistance
Praveen Nuwantha Gunaratne, Hiroki Tamura (University of Miyazaki, Japan)

34©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

GS1-4

A study on the Real-Time Biomechanical Analysis of Lamber Burden Utilizing
Stereoscope Cameras
Taufik Hidayat Soesilo, Praveen Nuwantha Gunaratne, Hiroki Tamura(University of Miyazaki, Japan)

GS1-5

Verification of Determination Possibility using Convolutional Autoencoder for
Machine Tool Abnormality Detection
Yuta Sumoto, Praveen Nuwantha, Hiroki Tamura (University of Miyazaki, Japan)

11:30-12:15 GS1-2 Machine learning& Neural Network & Artificial Life (3)Chair: Hiroki Tamura (University of Miyazaki, Japan)
GS1-6

Basic Research for High-speed Heart Sound Determination using AI
Riku Nakashima, Praveen Nuwantha Gunaratne, Hiroki Tamura(University of Miyazaki, Japan)

GS1-7

Prediction of High-Energy Electron Flux at Geosynchronous Orbit using a neuralnetwork technique
Ami Iwabu, Kentaro Kitamura (Kyusyu Institute of Technology, Japan)

GS1-8

Image Gradient-based Monocular Visual-Inertial Odometry
Tae Ihn Kim (Hyundai Motor Company, Republic of Korea), Jae Hyung Jung, andChan Gook Park (Seoul National University, Republic of Korea)

14:15-15:30 OS8 Intelligent Control (5)Chair: Yingmin Jia (Beihang University, China)
Co-Chair: Weicun Zhang (University of Science and Technology Beijing, China)
OS8-1

Global Stabilization of A Class of Nonholonomic Integrators via Discontinuous
Control
Lixia Yan, Yingmin Jia (Beihang University (BUAA), China)

OS8-2

Frequency Dependence Performance Limit of Vibration Absorbers
Jiqiang Wang1, Xiaoyu Yin2, Weicun Zhang3(1Chinese Academy of Sciences, China)(2Science & Technology Bureau of Zhenhai District, China)(3University of Science and Technology Beijing, China)

OS8-3

Adaptive Concurrent Learning Algorithm Based on Pontryagin’s Maximum Principle
for Nonlinear System Optimal Tracking Control with State Inequality Constraints
Yuqi Zhang, Bin Zhang (Beijing University of Posts and Telecommunications, China)

35©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS8-4

Privacy preserving Mean-square consensus for discrete-time heterogeneous multiagent systems with Communication Noises
Tongqing Yang1, Lipo Mo1, Yingmin Jia2
(1Beijing Technology and Business University, China)(2Beihang University (BUAA), China)

OS8-5

Event-Triggered Consensus Control for Nonlinear Singular Multi-Agent Systems
under Directed Topology
Lin Li, Tong Yuan, Mei Huang(University of Shanghai for Science and Technology, China)

15:40-16:25 OS14 Robotic Manipulation (3)Chair: Kensuke Harada (Osaka University, Japan)
Co-Chair: Akira Nakamura (Saitama Institute of Technology, Japan)
Co-Chair: Tokuo Tsuji (Kanazawa University, Japan)
OS14-1

Evaluation Standard of Error Recovery Planning Focused on Revival Process from Failures in
Robotic Manufacturing Plants
Akira Nakamura1, Kensuke Harada2(1Saitama Institute of Technology, 2Osaka University, Japan)

OS14-2

Robotic Food Handling Utilizing Temperature Dependent Variable-Stiffness Material
Rozilyn Marco1, Prashant Kumar2, Xinyi Zhang2, Weiwei Wan2 , Kensuke Harada2(1University of Toronto, Canada, 2Osaka University, Japan)

OS14-3

Vegetable maturity evaluation for harvest robots
Reno Muhammad Fadilla, Tokuo Tsuji, Tatsuhiro Hiramitsu, and Hiroaki Seki
(Kanazawa University, Japan)

16:40-17:40 GS5 Applications I (4)Chair: Chair Jiwu Wang (Beijing Jiaotong University, China)
GS5-1

Unsupervised image registration based on Residual-connected DRMINE for diagnostic
metastatic bone tumors
Shogo Baba1, Tohru Kamiya1, Takashi Terasawa2, Takatoshi Aoki2(1Kyushu Institute of Technology, Japan), (2University of Occupational and Environmental Health,Japan)

GS5-2

Developing a smart Belt for Monitoring Elderly Activities Based on Multi-Modal Sensors
Integration
and Internet of Things
Abdul Jalil, Pujianti Wahyuningsih, Najirah Umar, Muhammad Risal, Suwatri Jura, A. Edeth Fuari
Anatasya (Universitas Handayani Makassar, Indonesia)

36©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

GS5-3

A penalized motion detection model for extracting ionospheric echoes from low signal-to-noise
ratio Ionogram video images
Yuu Hiroshige1, Akiko Fujimoto1, Akihiro Ikeda2, Shuji Abe3, Akimasa Yoshikawa3(1Kyushu Institute of Technology, Japan) (2National Institute of Technology, Kagoshima College,
Japan), (3Kyushu University, Japan)

GS5-4

Verification Experiments on the Lower Back Burden caused by Posture and Environment during
Lifting Operations
Tomoka Kimura, Yutaro Fujino, Sachiko Kido, Praveen Nuwantha Gunaratne, Hiroki Tamura(University of Miyazaki, Japan)

Room 405
10:00-10:45 OS19 Natural Computing (3)Chair: Marion Oswald (Vienna University of Technology, Austria)
Co-Chair: Yasuhiro Suzuki (Nagoya University, Japan)
OS19-1A Model of Reaction-diffusion phenomena with Multiset Processing
Yasuhiro Suzuki (Nagoya University Japan)

OS19-2Extract tactile qualities from time series data
Yasuhiro Suzuki (Nagoya University, Japan)

OS19-3

Healthcare applications of vibrotactile stimulation developed by Tactile Score
Yasuhiro Suzuki, Rie Taniguchi (Nagoya University, Japan)

11:00-13:00 OS22 Mathematical Informatics (8)Chair Takao Ito (Hiroshima University, Japan)
Co-Chair Makoto Sakamoto (University of Miyazaki, Japan)
OS22-1

Adsorption Behavior of Arsenic and Selenium using NiZn Hydroxy Double Salts withAcetate, Chloride, Nitrate, and Sulfate Anions
Kaoru Ohe*, Ryosuke Tabuchi, Tatsuya Oshima(University of Miyazaki, Japan)

OS22-2

Parallel Acoustic Analysis Based on the Domain Decomposition Method with HigherOrder Element
Amane Takei1, Makoto Sakamoto1, Akihiro Kudo2(1University of Miyazaki, Japan)(2National Institute of Technology, Tomakomai Collage, Japan)

OS22-3

Sound Field Evaluation on Acoustical Experiment using Non-Steady State Analysis
Akihiro Kudo1, Makoto Sakamoto2, Amane Takei2(1National Institute of Technology, Tomakomai College, Japan)(2 University of Miyazaki, Japan)

37©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS22-4A DeepInsight Method with Morphological Analysis
Toyoaki Tomioka1, Satoshi Ikeda1, Makoto Sakamoto1, Takao Ito2
(1University of Miyazaki, Japan), (2Hiroshima University, Japan)

OS22-5

Support for Museum Exhibition of Small Fungi using AR Technology
Kakeru Takemura1, Ota Hamasuna1, Fumito Hamakawa1, Satoshi Ikeda1, Kaoru Ohe1,
Amane Takei1, Makoto Sakamoto1, Shuichi Kurogi2(1University of Miyazaki, Japan)(2Miyazaki Prefectural Museum of Nature and History, Japan)

OS22-6

Automatic Selection of High-Grade Dried Shiitake Mushrooms using Machine
Learning
Ota Hamasuna1, Kakeru Takemura1, Kodai Hasebe1, Fumito Hamakawa1, Bidesh
Biswas Biki1, Satoshi Ikeda1, Kaoru Ohe1, Amane Takei1, Makoto Sakamoto1,
Kazuhide Sugimoto2(1University of Miyazaki, Japan), (2 SUGIMOTO Co., Ltd. Japan)

OS22-7Predicting High Volatility Cryptocurrency Prices using Deep Learning
Tsutomu Ito1, Kodai Hasebe2, Fumito Hamakawa2, Bidesh Biswas Biki2, Satoshi
Ikeda2, Amane Takei2, Makoto Sakamoto2, Md Riajuliislam3, Sabrina Bari Shital4,
Takao Ito5
(1National Institute of Technology, Ube College, Japan)(2University of Miyazaki, Japan), (3 TU Dortmund, Germany)(4Daffodil International University, Bangladesh), (5Hiroshima University, Japan)

OS22-8Simulation of Weathering Representation using Vertex and UV Information
Tsutomu Ito1, Fumito Hamakawa2, Kodai Hasebe2, Satoshi Ikeda2, Amane Takei2,
Makoto Sakamoto2, Takao Ito3
(1National Institute of Technology, Ube College, Japan),(2University of Miyazaki, Japan), (3Hiroshima University, Japan)

14:15-15:15 OS5-1 Intelligent Life and Robotics (4)Chair Kuo-Hsien Hsia (National Yunlin University of Science and Technology, Taiwan)
Co-Chair Evgeni Magid (Kazan Federal University, Russia)
OS5-1

Research on dynamic obstacle avoidance and complex path planning strategies
based on ROS robots
Yi-Wei Chen, Jr-Hung Guo(National Yunlin University of Science and Technology, Taiwan)

OS5-2

Research on Multi-Robot Formation on Two-Dimensional Plane
Kuo-Hsien Hsia, Chun-Chi Lai, Yi-Ting Liu, Yu-Le Chen(National Yunlin University of Science and Technology, Taiwan)

38©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS5-3

The Development of Utilization Rate and Energy Consumption Monitoring and
Networking System
Chung-Wen HUNG, Chun-Chieh WANG, Heng-En CHANG(National Yunlin University of Science and Technology, Taiwan)

OS5-4

MCU Based Edge Computing Platform for Liquid Level Measurement
Chung-Wen HUNG, Chun-Liang LIU, Tai-Hsuan WU (National Yunlin University of
Science and Technology, Taiwan)

15:40-16:10 OS5-2 Intelligent Life and Robotics (2)Chair Kuo-Hsien Hsia (National Yunlin University of Science and Technology, Taiwan)
Co-Chair Evgeni Magid (Kazan Federal University, Russian Federation)
OS5-5

Potential of genetic algorithms in multi-UAV coverage problem
Ramil Faizullin1, Tatyana Tsoy1, Edgar A. Martínez-García2, Evgeni Magid1,3(1Kazan Federal University, Russian Federation)(2The Autonomous University of Ciudad Juarez, Mexico)(3HSE University, Russia)

OS5-6

Construction of Anthropomorphic Grippers with Adaptive Control
Evgeny Dudorov1, Julia Zhdanova2, Ivan Zhidenko1, Vladimir Moshkin2, Alexander Eryomin3,
Evgeni Magid3, Alexander Permyakov1(1JSC ‘SPA ‘Android technics’, Russia) (2MIREA – Russian Technological University, Russia)(3Kazan Federal University, Russia)

16:40-17:40 OS16 Industrial Artificial Intelligence Robotics (4)Chair: Eiji Hayashi (Kyushu Institute of Technology, Japan)
OS16-1

A Research on Performance Information Editing Support System for Automatic Piano
Yoshiki Hori, Eiji Hayashi (Kyushu Institute of Technology, Japan)

OS16-2Development of Autonomous Mobile Field Robots – Accuracy Verification of Self-Localization
through Simulation Takamasa Hayashi1, Shintaro Ogawa1, Yuto Okawachi1, Tan Chi Jie1, Janthori Titan1, Ayumu
Tominaga2, Eiji Hayashi1, Satoko Seino3(1Kyushu Institute of Technology, Japan), (2National Institute of Technology (Kitakyushu College),
Japan), (3Kyushu University, Japan)

OS16-3Development of AR System for Grasping String Foods on Introduction of Industrial Robot
Yoshihiro Koyama, Eiji Hayashi (Kyushu Institute of Technology, Japan), Akira Kawaguchi (The City
College of New York of The City University of New York, United States of America)

39©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS16-4

An Image Analysis of Coastal Debris Detection -Detection of microplastics using deep learning
Yuto Okawachi1, Ayumu Tominaga2, Shintaro Ogawa1, Takamasa Hayashi1, Tan Chi Jie1,
Janthori Titan1, Eiji Hayashi1, Satoko Seino3(1Kyushu Institute of Technology, Japan), (2National Institute of Technology (Kitakyushu College),
Japan), (3Kyushu University, Japan)

Room 406
10:00-11:15 OS18-1 Computer and Information Engineering (5)Chair: Norrima Mokhtar (University of Malaya, Malaysia)
Co-Chair: Heshalini Rajagopal (UCSI University, Malaysia)
OS18-1

Efficient Campus Shuttle Tracking and Management Mobile Application for College Campus
Andrea Tantay Gonzales1, Kavitha Thamadharan1, Neesha Jothi2(1INTI International College Penang, Malaysia) (2UCSI University, Malaysia)

OS18-2GCN Analysis of Task-Based fMRI Data for Diagnosis of Schizophrenia
Tejaswini Thota1, Reuben Stephen John1, Dr R Dhanush1, Dr Amutha S1(1 Vellore Institute of Technology, India)

OS18-3AR-Based Application for Campus Navigation
Renuka Devi Rajagopal1, Akshay S1, Manoj Rathinam1, Shakthi B1, Heshalini Rajagopal2(1 Vellore Institute of Technology, India), (2UCSI University, Malaysia)

OS18-4Development of Robotic Assistant for Health Care Sector with A Special Focus to Aid the GeriartricPatients
Narayanan Ganesh (Vellore Institute of Technology, India)

OS18-5Development of a Desktop Application Restaurant Management System
Gabriela Maria Ancilla, Heshalini Rajagopal, Ismail Ahmed Al-Qasem Al-Hadi(UCSI University, Malaysia)

11:30-12:00 OS18-2 Computer and Information Engineering (2)Chair: Norrima Mokhtar (University of Malaya, Malaysia)
Co-Chair: Heshalini Rajagopal (UCSI University, Malaysia)
OS18-6

Face Recognition based on Attendance System
Koh Pei Cong1, Heshalini Rajagopal1, Ghassan Saleh1, Norrima Mokthar2(1UCSI University, Malaysia), (2University of Malaya, Malaysia)

OS18-7

U-Reserve: Development of a Facility Reservation System for UCSI University
Esther Chong Jun Lynn1, Neesha Jothi1, Ismail Ahmed1(1UCSI University, Malaysia)

40©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

14:15-15:30 OS18-3 Computer and Information Engineering (5)Chair: Norrima Mokhtar (University of Malaya, Malaysia)
Co-Chair: Heshalini Rajagopal (UCSI University, Malaysia)
OS18-8The Smart Document Processing with Artificial Intelligence
Raenu Kolandaisamy1, Heshalini Rajagopal1, Indraah K2, Glaret Shirley Sinnappan3(1 UCSI University, Malaysia), (2University Utara Malaysia, Kedah)(3Tunku Abdul Rahman University of Management and Technology, Malaysia)

OS18-9

Digital Security Challenges Faced by Business Organizations
Raenu Kolandaisamy1, Heshalini Rajagopal1, Indraah K2, Glaret Shirley Sinnappan3(1UCSI University, Malaysia), (2University Utara Malaysia, Malaysia), (3Tunku Abdul
Rahman University of Management and Technology, Malaysia)

OS18-10The Study on Perception on E-Waste Among the People
Raenu Kolandaisamy1, Heshalini Rajagopal1, Indraah K2(1UCSI University, Malaysia), (2University Utara Malaysia, Malaysia)

OS18-11

Emergence of Cybercrimes in Online Social Networks
Raenu Kolandaisamy1, Heshalini Rajagopal1, Indraah K2(1UCSI University, Malaysia) (2University Utara Malaysia, Malaysia)

OS18-12Development of a music recommendation application by using facial emotion
recognition
Shengke Xie, Raenu Kolandaisamy, Ghassan Saleh, Heshalini Rajagopal(UCSI University, Malaysia)

15:40-16:25 OS21 Human-Machine Interface (3)Chair: Norrima Mokhtar (University of Malaya, Malaysia)
Co-Chair: Heshalini Rajagopal (UCSI University, Malaysia)
OS21-1

Enhancing Reconnaissance Missions Through Multiple Unmanned Systems in ROS
Anees ul Husnain1,2, Norrima Mokhtar1, Takao Ito3, Siti Sendari4, Muhammad
Farris Kyasudeen5, Muhammad Badri M Noor1,6, Heshalini Rajagopal7(1 Universiti Malaya, Malaysia), (2 The Islamia University of Bahawalpur, Pakistan),(3Hiroshima University, Japan), (4 Universitas Negeri Malang, Indonesia), (5University Technology MARA (UiTM), Malaysia)(6 Ifcon Technology Sdn Bhd, Malaysia), (7UCSI University, Malaysia)

OS21-2Illumination Effects on Facial Expression Recognition using Empirical Mode
Decomposition
Hashimah Ali1, Wan Khairunizam Wan Ahmad1, Hariharan Muthusamy2,
Mohamed Elshaikh1(1 Universiti Malaysia Perlis, Malaysia)(2 National Institute of Technology Uttarakhand, India)

41©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS21-3

Supercontinuum Generation Pump By a Molydenum Disulfide Based Soliton ModeLocked Fiber Laser
Aeryn D. Ahmad1, Norrima Mokhtar1, Hamzah Arof1, Sulaiman Wadi harun1,
Ahmad Haziq Aiman Rosol2(1Universiti Malaya, Malaysia), (2MJIIT, UTM, Malaysia)

16:40-17:40 GS6 Applications II (4)Chair: Masayuki Fujiwara (Kyushu Institute of Technology, Japan)
GS6-1

Development of Smartphone Application for Calculating the Low Back Pain Risk
Seigo Imura, Praveen Nuwantha, Hiroki Tamura (University of Miyazaki, Japan)

GS6-2

A Computational Approach for Global Trade Analysis Sensitive to Free Trade
Agreement Circumstances: A Case Study Focusing on the Great Mekong Subregion
Ahmad Altaweel1, Bo-Young Lee2, Masayuki Fujiwara1, Jang-Sok Yoon2, Hiroaki
Wagatsuma1(1Kyushu Institute of Technology, Japan; 2Logistics Revolution Korea, Korea)

GS6-3

Trigger circuit design and system integration for simultaneous measurement of
human EEG, motion, and gaze
Masayuki Fujiwara1, Phan Hoang Huu Duc1, Laurent Bougrain2, Patrick Hénaff2,
Hiroaki Wagatsuma1 (1Kyushu Institute of Technology, Japan; 2Université deLorraine, France)

GS6-4

Terminal Synergetic Controller for Car’s Active Suspension System Using Dragonfly
Algorithm
Tinnakorn Kumsaen1, Sorn Simatrang2, Arsit Boonyaprapasorn3, Thunyaseth
Sethaput4(1Khon Kaen University, Thailand; 2Nacres Co., Ltd, Thailand; 3Chulachomklao Royal
Military Academy, Thailand; 4Thammasat University, Thailand)

42©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

February 25 (Sunday)
9:40-Registration
10:00-11:00
Room 405
10:00-11:15 OS6 Intelligence and Optimization (5)Chair: Mastaneh Mokayef (UCSI University, Malaysia)
Co-Chair Takao Ito (Hiroshima University, Japan)
OS6-1

Simulation-Based Enhancement of SNR in Drone Communication through Uniform
Linear Array Configurations
Gershom Phiri1, Mastaneh Mokayef1, MHD Amen Summakieh1, M.K.A Ahamed
Khan1, Sew Sun Tiang1, Wei Hong Lim1, Abdul Qayyum2(1UCSI University Malaysia)(2National Heart and Lung Institute, Imperial College London, UK)

OS6-2

Empowering Elderly Individuals through the Intelligent Shopping Trolley
Mastaneh Mokayef1, Muzaiyanah Binti Hidayab1, MHD Amen Summakieh1, M.K.A
Ahamed Khan1, Kim Soon Chong1, Chin Hong Wong2, Chua Huang Shen3, Abdul
Qayyum4(1UCSI University Malaysia) (2Fuzhou University, China) (3UOW Malaysia University,
Malaysia) (4National Heart and Lung Institute, Imperial College, UK)OS6-3

Optimized Microstrip Slot UWB Patch Antenna for Medical Imaging
Maxime Duvacher1, Mastaneh Mokayef2, MHD Amen Summakieh2, M.K.A Ahamed
Khan2, Sew Sun Tiang2, Wei Hong Lim2, Abdul Qayyum3(1Polytech Nantes, France) (2UCSI University Malaysia) (3National Heart and Lung
Institute, Imperial College London, UK)

OS6-4

Development of an Innovative Undergraduate Industrial Automation and Robotics
Degree Program
M.K.A. Ahamed Khan1, Mastaneh Mokayef1, Ridzuan, A.1, Irraivan Elamvazuthi,
Badli Shah Yusoff2, Abu Hassan Darusman3 (1UCSI University Malaysia) (2University
Technology Petronos, Malaysia) (3UNIKL Malaysia France Institute, Malaysia)

OS6-5

Smart Assistive Trolley for Elderly Care and Independence
Dina Ashraf1, Mastaneh Mokayef1, MHD Amen Summakieh1, M.K.A Ahamed Khan1,
Abdul Qayyum2, Sivajothi A/L Paramasivam3 (1UCSI University Malaysia) (2National
Heart and Lung Institute, Imperial College London, UK) (3UOW Malaysia University
College, Malaysia)

43©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

12:30-13:45 OS4-1 Pattern Recognition and Robotics III (5)Chair: Huahao Li (Tianjin University of Science and Technology, China)
Co-Chair Hongshuo Zhai (Tianjin University of Science and Technology, China)
OS4-1

A Digital Twin Design Based on Robot Workstation
Huahao Li (Tianjin University of Science and Technology, China)

OS4-2

A Study of Chemical Reactor Simulation System Based on PCS7
Hongshuo Zhai (Tianjin University of Science and Technology, China)

OS4-3

Analysis of Learning Quality Evaluation for University Student Courses with Process
Assessment
Yuhao Zhang, Ying Gong, Xuran Wang
(Tianjin University of Science and Technology, China)

OS4-4Motion Analysis and Transfer Applications Based on Posture Recognition
Yuhao Zhang, Mingyue Li, Jianhao Jiao (Tianjin University of Science and Technology,
China)

OS4-5

Functional Safety Assessment of the Safety Protection System Based on Petri Net
Peng Wang, Mengyuan Hu(Tianjin University of Science and Technology, China)

14:00-15:15 OS4-2 Pattern Recognition and Robotics III (5)Chair: Huahao Li (Tianjin University of Science and Technology, China)
Co-Chair: Hongshuo Zhai (Tianjin University of Science and Technology, China)
OS4-6

Pedestrian Attribute Recognition Based on Deep Learning
Peng Wang, Qikun Wang, Shengfeng Wang
(Tianjin University of Science and Technology, China)

OS4-7

Simulation of office air conditioning air supply based on COMSOLPeng Wang, Mengda Liu, Qikun Wang
(Tianjin University of Science and Technology, China)

OS4-8

Solo Wheel Technology-Self-balancing Wheelbarrow
Ziyue Xiao, Yumei Huang, Zhencheng Chang, Mingxuan Li(Tianjin University of Science and Technology, China)

OS4-9

Second-order self-balancing inverted pendulum
Ziyue Xiao, Zhencheng Chang, Mingxuan Li, Yumei Huang(Tianjin University of Science and Technology, China)

44©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS4-10Design of Nanny's Abnormal Behavior Recognition Bracelet Based on Human
Activity Recognition (HAR) Deep Learning Model
Depeng Wang, Yingfan Zhu, Yande Xiang, Ziyue Xiao
(Tianjin University of Science and Technology, China)

Room 406
10:00-11:30 OS26-1 Navigating the Digital Frontier: Innovations in the Age of Industry
Revolution 4.0(6)Chair: Wei Hong Lim (UCSI University, Malaysia)
Co-Chair: Takao Ito (Hiroshima University, Japan)
OS26-1An Intelligent Cargo/Warehouse Management System
Zhongheng Sun, Zhou Yue, Xun Sun, Wenzhuo Fan, Wenxuan Zhou
(Fuzou University, China)

OS26-2A Comprehensive Approach to Design and Implement an IoT-Enabled Intelligent
Shopping Cart System with Obstacle-Aware Navigation and Enhanced Customer
Engagement for Elevated Consumer Experiences
Yao Chen1, Jiacheng Du1, Bo Peng1, Ningfei Wang1, Zehan Huang1, Wei Hong Lim2,
Sew Sun Tiang2, Mastaneh Mokayef2, Chin Hong Wong1(1Fuzhou University, China) (2UCSI University, Malaysia)

OS26-3

Design of a Four-Port Flexible UWB-MIMO Antenna for Wearable and IoTApplications
Jia Wei Tan, Sew Sun Tiang, Kim Soon Chong, Mohammad Arif Ilyas, Mastaneh
Mokayef, Zhi Ying Yeoh, Wei Kang Lai, Wei Hong Lim(UCSI University, Malaysia)

OS26-4

Investigate Power Efficiency in PLECS and MATLAB Software by Designing USB 5W
ChargerZhi Ying Yeoh, Kim Soon Chong, Sew Sun Tiang, Mohammad Arif Ilyas, Jia Wei Tan,
Wei Kang Lai, Wei Hong Lim (UCSI University, Malaysia)

OS26-5

Design and Simulation and Performance of Grid Connected Photovoltaic System for
Small, Tall Building in Malaysia
Wei Kang Lai, Kim Soon Chong, Sew Sun Tiang, Mohammad Arif Ilyas, Jia Wei Tan,Zhi Ying Yeoh, Wei Hong Lim(UCSI University, Malaysia)

OS26-6

Design of Dual-Band Coplanar Waveguide (CPW) Printed Antenna for 1.9 - 3.6GHz
Applications
Jia Wei Tan, Sew Sun Tiang, Kim Soon Chong, Mohammad Arif Ilyas, Mastaneh
Mokayef, Zhi Ying Yeoh, Wei Kang Lai, Wei Hong Lim(UCSI University, Malaysia)

45©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

12:30-13:45 OS26-2 Navigating the Digital Frontier: Innovations in the Age of Industry
Revolution 4.0 (5)Chair: Wei Hong Lim (UCSI University, Malaysia)
Co-Chair: Takao Ito (Hiroshima University, Japan)
OS26-7

Optimized Convolutional Neural Network Towards Effective Wafer Defects
Classification
Koon Hian Ang1, Koon Meng Ang1, Chin Hong Wong2,3, Abhishek Sharma4, Chun Kit
Ang1, Kim Soon Chong1, Sew Sun Tiang1, Wei Hong Lim1
(1UCSI University, Malaysia) (2Fuzou University, China) (3Maynooth University,
Ireland) (4Graphic Era Deemed to be University, India)

OS26-8

Tackling Photovoltaic (PV) Estimation Challenges: An Innovative AOA Variant for
Improved Accuracy and Robustness
Rayan Mohammed Noor Mohammed Bakhit1, Abhishek Sharma2, Tiong Hoo Lim3,
Chin Hong Wong4,5, Kim Soon Chong1, Li Pan1, Sew Sun Tiang1, Wei Hong Lim1
(1UCSI University, Malaysia) (2Graphic Era Deemed to be University, India)(3Universiti Teknologi Brunei, Brunei Darussalam) (4Fuzou University, China)(5Maynooth University, Ireland)OS26-9Deep Learning in Manufacturing: A Focus on Welding Defect Classification withCNNs
Tin Chang Ting1, Hameedur Rahman2, Tiong Hoo Lim3, Chin Hong Wong4,5, Chun Kit
Ang1, Mohamed Khan Afthab Ahamed Khan1, Sew Sun Tiang1, Wei Hong Lim1
(1UCSI University, Malaysia) (2Air University, Pakistan) (3Universiti Teknologi Brunei,
Brunei Darussalam) (4Fuzou University, China) (5Maynooth University, Ireland)OS26-10Enhancing Global Optimization Performance of Arithmetic Optimization Algorithmwith a Modified Population Initialization Scheme
Tin Chang Ting1, Hameedur Rahman2, Meng Choung Chiong1, Mohamed Khan
Afthab Ahamed Khan1, Cik Suhana Hassan1, Farah Adilah Binti Jamaludin1, Sew Sun
Tiang1, Wei Hong Lim1
(1UCSI University, Malaysia) (2Air University, Pakistan)OS26-11

Enhancing Precision Object Detection and Identification for Autonomous Vehicles
through YOLOv5 Refinement with YOLO-ALPHA
Guandong Li1, Yanzhe Xie1, Yuhao Lu1, Jingzhen Fan1, Yuankui Huang1, Zongyan
Wen1, Wei Hong Lim2, Chin Hong Wong1(1Maynooth University, Ireland) (2UCSI University, Malaysia)

46©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

14:00-15:30 OS11 Intelligent Life and Robotics (6)Chair: Evgeni Magid (Kazan Federal University, Russia)
Co-Chair: Kuo-Hsien Hsia (National Yunlin University of Science and Technology, Taiwan)
OS11-1An Overview of Kinect Based Gesture Recognition Methods
Alexander Alexeev1, Tatyana Tsoy1, Edgar A. Martínez-García2, Evgeni Magid1,3(1Kazan Federal University, Russia)(2The Autonomous University of Ciudad Juarez, Mexico)(3HSE University, Russia)

OS11-2

An investigation on the impact of human-robot interactions during an autonomous
obstacle avoidance task
Riham Salman, Shifa Sulaiman, Renata Islamova, Tatyana Tsoy
(Kazan Federal University, Russia)

OS11-3

A Comparative Analysis of Object Detection Methods for Robotic Grasping
Nikita Kolin, Elvira Chebotareva (Kazan Federal University, Russia)

OS11-4

Vision-based autonomous navigation for medical examination using a UR3emanipulator
Bulat Abbyasov1, Aidar Zagirov1, Timur Gamberov1, Hongbing Li2, Evgeni Magid1(1Kazan Federal University, Russia)(2Shanghai Jiao Tong University, China)

OS11-5

Robot-Assisted Language Learning: Scientific Data Analysis
Karina Sadyikova, Valeriya Zhukova, Roman Lavrenov(Kazan Federal University, Russia)

OS11-6

Monitoring Beehive Sound Levels with Arduino-based System
Kulmukhametov Ramis1, Ramil Safin1, Tatyana Tsoy1, Kuo-Hsien Hsia2, Evgeni Magid1(1Kazan Federal University, Russia)(2National Yunlin University of Science and Technology, Taiwan)

Room 407
10:00-11:15 OS25-1 Research Towards the Sustainable Development Goals (SDG’s) (5)Chair: Ammar A.M. Al Talib (UCSI University, Malaysia)
Co-Chair: Takao Ito (Hiroshima University, Japan)
OS25-1

Portable Green Energy Mobile Laptop Charging Station
Ammar A. M. Al-Talib1, Rodney Tan1, Ang Aun Jie1, Idayu M. Tahir1, Sarah Atifah
Saruchi2, Cik Suhana Bt. Hassan1, Amar Rizwan1(1UCSI University) (2UMPSA, Malaysia)

47©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS25-2

Auto Indoor Hydroponics Plant Growth Chamber
Ammar A.M. Al-Talib1, Tew Hwa Hui1, Sarah Atifah Saruchi2, Idayu M. Tahir1, Nor
Fazilah Binti Abdullah1(1UCSI University) (2UMPSA, Malaysia)

OS25-3

A Design and Fabrication of a Solar Agriculture Water Pumping System
Ammar A.M. Al-Talib1, Idayu M. Tahir1, Ain Atiqa1, Amar Rizwan1, Sarah Atifah
Saruchi2, Yazan Abu Al shaikh1(1UCSI University) (2UMPSA, Malaysia)

OS25-4

Design and Performance of a Power Generating Manual Treadmill
Ammar A. M. Al-Talib1, Sarah Atifah Saruchi2, Cik Suhana Bt. Hassan1, Nor Fazilah
Binti Abdullah1, Ain Atiqa1, Ahmad Jelban1(1UCSI University) (2UMPSA, Malaysia)

OS25-5

Smart Car Jack Using Internet of Things
Idayu M.T. Noor, Ammar A.M. Al- Talib, Mahmoud E.A. Zeiad, Suhana B.H. Cik,(UCSI University, Malaysia)

12:30-13:45 OS25-2 Research Towards the Sustainable Development Goals (SDG’s) (4)Chair: Ammar A.M. Al Talib (UCSI University, Malaysia)
Co-Chair: Takao Ito (Hiroshima University, Japan)
OS25-6

Gas Detection for Biogas System Using Internet Of Things (IoT)
Ammar A.M. Al Talib, I.H.W. Yang, Idayu M.T. Noor, Haslija A.B. Ayu, Afifi. Z. Nur
Muhammad(UCSI University, Malaysia)

OS25-7

IoT- Based Smart Mushroom Growing Kit
Ammar A.M. Al- Talib, C.K.J. Ting, Noor Idayu M. Tahir, Ain Atiqa, T.Y. Hui(UCSI University, Malaysia)

OS25-8Design and Analysis of Artificial Magnetic Conductor for Metal Shielding
Applications in RFID Car Detection Applications
Eryana Hussin1,2, Azman Zakariya2, Md. Ashraful Haque2,3, Nur Izzati Ali4
(1UCSI University) (2UTPSA, Malaysia) (3Daffodil International University,
Bangladesh) (4UMP, Malaysia)

OS25-9Effect of Fibre Orientation on the Mechanical Performance of Natural Fibre Polymer
Composite Bicycle Frame using Finite Element Analysis
Kok Sem Too, Cik Suhana Hassan, Nor Fazilah Abdullah, Ammar Abdulaziz Majeed
Al-Talib,(UCSI University, Malaysia)

48©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS25-10Modelling of short-circuit protection for a residential grid-connected BESS
Kong De Kang, Farah Adilah Jamaludin, Rodney H.G. Tan(UCSI University, Malaysia)

14:00-15:30 GS7 Applications III (6)Chair: Kasthuri Subaramaniam (UCSI University, Malaysia)
GS7-1

Rehabilitating Flood-Damaged Cars for Sustainable Car Rental Services: A WebBased System
Pon Xiao Qi, Abdul Samad Shibghatullah, Kasthuri Subaramaniam(UCSI University, Malaysia)

GS7-2

Optimizing E-Invoicing Rollout: Adaptive E-Invoicing Rollout (AER) Framework for
Navigating Malaysia's Digital Transformation
Koh Chee Hong, Abdul Samad Shibghatullah(UCSI University, Malaysia)

GS7-3

App Alert System for Smart Phones
Chee Kin Hoe, Kasthuri Subaramaniam, Abdul Samad Shibghatullah(UCSI University, Malaysia)

GS7-4

Developing Hand Gesture Recognition System in Interpreting American Sign
Language
Kong Seh Chong, Kasthuri Subaramaniam, Ismail Ahmed Al-Qasem Al-H
(UCSI University, Malaysia)

GS7-5

Miniature Enterprise Resource Planning
Adim Khalid Aldireejah, Kasthuri Subaramaniam, Ghassan Saleh(UCSI University, Malaysia)

GS7-6

Developing Cloud-based Sportswear Website
Lim Wei Yee, Kasthuri Subaramaniam, Raenu Kolandaisamy(UCSI University, Malaysia)

Farewell Party

49©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

Abstract
PS Abstract (3)PS1 Developing High-Speed Working Motion of the Multi Robot in DENSO
Tomoaki Ozaki (DENSO CORPORATION, Japan)
The number of labor force is expected to decrease due to the declining birthrate and aging
population. Although many efforts to solve this social issue using automation technology by
robots around the world have been implemented, most of the current applications of robots arestill repetitive works such as picking and placing in mass production lines in factories, and small
progress of the application of robots for high-mix low-volume production lines where theoperations are frequently changed. In this paper, we discuss the reasons and the potential
solutions for autonomous control technologies including the AI and deep learning. Moreover,the high-speed working motion of the multi robot developed by DENSO will be presented inthis paper.

High-Speed Working
Motion of the MultiRobot

PS2 Experimenting with Variable Arm Quadrotors: Realizing Dynamic Configurations forEnhanced Flight Performance
Hazry Desa, Muhammad Azizi bin Azizan (Universiti Malaysia Perlis, Malaysia)
This paper introduces two innovative variable arm concepts for quadrotors, enhancing precise
movement control by manipulating bending moments through arm length variations. Its key
goal is to identify the optimal arm configuration for smooth and stable quadrotor maneuvers.
Exploring two concept designs tailored for quadrotors, the study focuses on regulating
manoeuvrability using variable arms, enabling bending moment adjustments. Results validatethat the electric actuator with linear guide-type 2 variable arm ensures smooth and stable
quadrotor movement.

PS3 Artificial Intelligence and Technologies of Arm-type and Mobile Robots in Industry
Haruhisa Okuda (Mitsubishi Electric Corporation, Japan)
In recent years, labor shortage has become a serious issue in industrial fields. Various
technologies including robot and information processing system to realize flexible work likehumans are effective solutions to this issue. Artificial intelligence technology of arm-type robots
equipped with 3D sensors and force sensors has been applied in the manufacturing field to copewith different intelligent and highly precise tasks in Mitsubishi Electric. In addition, various
technologies to expand the scope of application to the service field, as well as to realize highly
functional delivery with mobile robots is under development. Furthermore, IoT technology isalso being used for easy and quick on-site implementation and efficient operation. This speech
introduces these initiatives with actual examples.

50©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS Abstract
OS1 Intelligent Life and Cybersecurity (6)Chair I-Hsien Liu (National Cheng Kung University, Taiwan)
Co-Chair Chu-Fen Li (National Formosa University, Taiwan)
Co-Chair Pang-Wei Tsai (National Cheng Kung University, Taiwan)
OS1-1 Detecting abnormal operations in ICS using finite-state machines
Pei-Wen Chou, Nai-Yu Chen, Jung-Shian Li, I-Hsien Liu (National Cheng Kung University, Taiwan)
In 2021, a water treatment facility in Florida, USA, fell victim to an external malicious attack.
In this incident, malicious actors attempted to manipulate the quantities of specific chemicals
to impact water quality and safety. Given the intricacies of abnormal operation detection in
Industrial Control Systems and the advantages of finite-state machine, we endeavored to apply
this approach for the detection of abnormal ICS operations. We conducted a series of tests using
the dam control system cybersecurity testbed established by TWISC@NCKU, Taiwan. The
results indicate that our approach effectively enhances the efficiency of identifying non-standard
operational behaviors, enabling maintenance personnel to promptly identify anomalies.

OS1-2 Industrial Control System State Monitor Using Blockchain Technology
Yun-Hao Chang, Tzu-En Peng, Jung-Shian Li, I-Hsien Liu (National Cheng Kung University, Taiwan)
This paper introduces an innovative approach to enhance data verification and security inintelligent systems through the integration of blockchain technology. The proposed method
amalgamates the transparency and decentralization inherent to blockchain with the commandand oversight functionalities of PLC to ensure the utmost data integrity. The devised approachsynergizes the decentralized attributes of blockchain with the control capabilities of PLCs, thus
establishing robust safeguards for data integrity. Through the utilization of blockchain's tamperresistant ledger, PLCs orchestrate data interactions and enforce real-time monitoring andcontrol. The viability and efficacy of this innovative scheme are substantiated through empiricalevaluations and simulations, conclusively affirming its practicality.

OS1-3 Enhancing Dam Security and Water Level Alerting with Blockchain Technology
YingCheng Wu, Jung-Shian Li, Chu-Fen Li, I-Hsien Liu (National Cheng Kung University, Taiwan)
Ensuring the security, monitoring, and timely alerting of water levels in dams is a major
challenge. We use blockchain technology to enhance the security and monitoring of dam
infrastructure, and also improving the alerting system for water level changes. The use ofblockchain technology in dam infrastructure management provides a decentralized, transparent,
and tamper-resistant platform for storing and managing data. This ensures the integrity andsecurity of critical data related to dam operations and water levels. This research investigates
the enhanced security, monitoring, and alerting capabilities that this integration offers, and aims
to contribute to the improved security and efficiency of dam infrastructure, leading to morereliable operations and better protection against potential disasters.

OS1-4 MiniDAM: A Dam Cybersecurity Toolkit
Tzu-En Peng, Meng-Wei Chang, Yun-Hao Chang, Jung-Shian Li, I-Hsien Liu(National Cheng Kung University, Taiwan)
Testbeds, serving as simulations of real-world scenarios, are of paramount importance forresearch in cybersecurity related to critical infrastructure. In this paper, we aim to offer acomprehensive exploration of the MiniDAM and our testbed, introducing its physical settings
based on real dam operational standards. Furthermore, a comparative analysis between the
Secure Water Treatment (SWaT) testbed, MiniCPS, our testbed, and MiniDAM is presented.
This paper also includes insights into dataset generation and the integration of otherfunctionalities. The exposition of MiniDAM's features and capabilities serves as a foundationfor enhancing resilience and provides valuable support for advancing research within the
broader field of dam-related studies.

51©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS1-5 Case Study of Network-Based Intrusion Detection System Deployment in Industrial
Control Systems with Network Isolation
Nai-Yu Chen, Pei-Wen Chou, Jung-Shian Li, I-Hsien Liu (National Cheng Kung University, Taiwan)
Deploying intrusion detection systems is a common cybersecurity measure, and intrusion
detection systems typically operate at the ports of gateways. In critical infrastructure, industrial
control systems often employ network isolation strategies, lacking the role of gateways. This
research primarily explores the deployment of the Snort intrusion detection system in such anenvironment, combined with specific OT rules. Validation is conducted using the cybersecurity
testbed of the dam control system established by TWISC@NCKU in Taiwan. The results
indicate that by employing our proposed approach, it is possible to effectively detect abnormal
network traffic, addressing the common issue of inadequate monitoring in environments with
network isolation.

OS1-6 The AI integration service innovation model of real estate industry in Taiwan
Li-Min Chuang, Chih-Hung Chen (Chang Jung Christian University, Taiwan)
In real estate transactions, the intermediary role is often played by real estate agents. In recent
years, with the integration of information and AI, the real estate industry can now provide higher
quality services. This study references relevant literature and collects the services currently
offered by real estate agents. The main dimensions and sub-dimensions are extracted using the
Likert scale. Then, the Fuzzy Analytic Hierarchy Process (FAHP) is employed in aquestionnaire study to obtain the relative weights among four main dimensions and twelve subdimensions. This research develops propositions and conclusions, summarizing the key factors
for real estate transactions in Taiwan's real estate industry. These findings serve as importantreference points for the industry.

OS2 Pattern Recognition and Robotics I (5)Chair Fengzhi Dai (Tianjin University of Science and Technology, China)
Co-Chair Yunzhong Song (Henan Polytechnic University, China)
OS2-1 A Study on Sales Patterns for Vegetable Products in Retail Stores
Yuhao Zhang 1, Shuangshuang Ma 1, Jiashuai Wang 1, Fengzhi Dai 1, Lijiang Zhang 2( Tianjin University of Science and Technology, China, 2 Xinjiang Shenhua Biotechnology Co., Ltd, China)
1

In fresh produce supermarkets, the shelf life of vegetable products is typically short,necessitating daily restocking based on historical sales data and the formulation of a rational
pricing strategy to maximize the store's profits. This paper, based on sales data for vegetableproducts in a particular store from July 2020 to June 2023, employs various analytical methods,
including multidimensional analysis, clustering, and regression, to explore the interrelationships
among different types of vegetables. Furthermore, it combines cost-plus pricing and priceelasticity models to establish a pricing framework that optimizes revenue for the supermarket.

OS2-2 Research and Implementation of Cooperative Control for ROS Mobile Robot
Saijie Zhang, Huailin Zhao
(Shanghai Institute of Technology, China)
This paper discusses on the distributed control for the multiple ROS-based robots. A
communication platform is built to execute the data transmission. The SLAM mapping andautonomous navigation of the robots are completed. The simulation tools of both Gazebo andRviz are applied to analyze the multi-point navigation. The proposed distributed control system
is tried and the synchronous control of the robots is realized, which achieve more accurate andmore synchronous robot motion control. At last, the multi-robot following and multi-robot
formatting come true.

52©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS2-3 An OpenCV-based Method for Workpiece Residue Image Processing
Jiaxin Wang, Hao He, Fangyv Liu, Fengzhi Dai (Tianjin University of Science and Technology, China)
Workpiece residue refers to a thin film formed on the surface of the workpiece during themachining process, due to factors such as cutting fluid, chips, oil stains, etc., which affects thequality and performance of the workpiece. This paper proposes an OpenCV-based method forworkpiece residue image processing, aiming to achieve automatic detection and analysis of
workpiece residue. By building a workpiece image acquisition system, workpiece residue
images of different types and degrees are collected, and the proposed method is verified andevaluated. The experimental results show that the proposed method can effectively detect andanalyze workpiece residue, with high accuracy and robustness, providing an effective means
for workpiece quality control.

OS2-4 On Nonblockingness Verification and Enforcement of Controlled Nondeterministic
Discrete-Event Systems
Xiang Ren, Zipei Wang (Tianjin University of Science and Technology, China)
Discrete event systems, as an important kind of cyber-physical systems, have been widely usedin engineering field. In this paper, we first express the dynamics of a controlled nondeterministic
discrete-event system (acronym is DES) as an algebraic state-space representation using thesemi-tensor product (STP) theory. And then, we discuss the problems of state-based
nonblockingness verification and enforcement of nondeterministic DESs. Specifically, we
obtain a criterion of verifying whether a given controlled nondeterministic DES is nonblocking.
Further, we develop an efficient matrix-based approach to enforce state-based nonblockingness.
We illustrate the applications of the proposed theoretical results using an example.

OS2-5 Modeling and Reachability Verification of Controlled Nondeterministic Finite-State
Automata
Zipei Wang, Xiang Ren (Tianjin University of Science and Technology, China)
In this paper, we investigate the modeling and state reachability of controlled nondeterministic
finite-state automata (NFA). The key feature of a controlled NFA is to admit a supervisor tointervene the behavior of original system. We first express the dynamics of a controlled NFA as
an algebraic state-space representation in the framework of the semi-tensor product (STP) of
matrices. Then, the necessary and sufficient condition for verifying state reachability ofcontrolled NFA is presented. An explicit formula for calculating all paths of any two states isderived. Finally, we use an example to illustrate the application of the proposed theoreticalresults.

OS3 Pattern Recognition and Robotics II (9)Chair Fangyan Li (Tianjin University of Science and Technology, China)
Co-Chair Haozhe Sun (Tianjin University of Science and Technology, China)
OS3-1 Detection and Identification of Daylily Maturity Based on YOLOv8
Fangyan Li (Tianjin University of Science and Technology, China)
To better apply object detection and identification techniques from deep learning to the field ofagricultural automation, this paper focuses on the growth process of daylilies. It employs thestate-of-the-art YOLOv8 model to achieve accurate assessment of daylily maturity. The
backbone network of YOLOv8 draws inspiration from the CSPDarkNet network structure to
extract image features. Experimental results demonstrate that the detection accuracy of daylilies
based on YOLOv8 exceeds 95.6%, with a recall rate of 90.5% and a mean average precision(mAP) reaching 0.94. Moreover, the identification speed is significantly improved.

53©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS3-2 Chaos Synchronization and Circuit Design of Chen System and Lü System with Different
Structures
Haozhe Sun (Tianjin University of Science and Technology, China)
In this paper, by using nonlinear feedback control, chaos synchronization is achieved between
the Chen system and the Lü system with different initial values, and the error curves and statesynchronization curves of the corresponding states in the response Lü system and the drive
Chen system are plotted. Finally, the simulation circuit model of the synchronization system of
the drive Chen system and the response Lü system is designed by Multisim circuit simulation
software. Comparing the output curves with the curves obtained by MATLAB simulation
software, it can be found that the two curves achieve a good qualitative agreement. The
synchronization of the drive Chen system and the response Lü system is accomplished.

OS3-3 A Parking Space Recognition Method Based on Digital Image Technology
Hao He, Fangyv Liu, Jiaxin Wang (Tianjin University of Science and Technology, , China)
In recent years, the number of cars in the city has been increasing, leading to an increasingly
prominent issue of urban parking space. Consequently, the automatic identification method forparking spaces has emerged as a crucial research direction. This paper presents a design andimplementation scheme for recognizing the status of parking spaces in urban areas based on
digital image processing and other technologies. Real-time images of multiple parking spaces
are collected and transmitted for splicing and detection purposes to determine their availability.
The experimental results demonstrate the feasibility and effectiveness of the proposed method,which holds practical significance in addressing parking space detection problems throughdigital image processing.

OS3-4 A Design of Intelligent Handling Robot Based on AT89C52
Fangyv Liu, Jiaxin Wang, Hao He (Tianjin University of Science and Technology, China)
Aimed at the shortcomings of the low efficiency and the limitation of the artificial logistics
handling, this paper presents a logistics handling robot based on AT89C52 single chip
microcomputer. The integration of the power module, sensor, and drive motor module enables
automatic obstacle avoidance and information collection.The incorporation of an ultrasonicobstacle avoidance module and an infrared tracking module enhances the capability for obstacle
avoidance and path searching, and can easily cope with different workplace. The Yaskawa MPL
manipulator is highly suitable for high-speed and high-precision palletizing, picking, packaging,and other industries.

OS3-5 Application and Differences of Robotic Arms, Traditional Machines and Manual Workin Production
Xue Yang, Ying Su, Yuping Mei, Haiquan Wang
(Tianjin University of Science and Technology, China)
This article aims to explore the differences between robotic arms and traditional machines andhumans. Firstly, robotic arms are a kind of automated equipment with high flexibility andaccuracy, which can perform a variety of complex tasks. In contrast, traditional machines lackflexibility and accuracy, while humans have problems such as low work efficiency and higherror rates. Secondly, the emergence of robotic arms can solve many problems in traditional
machines and humans, and improve production efficiency and quality. Finally, there is nocontradiction between robotic arms, traditional machines and humans, but they can complement
each other. By using robotic arms and humans reasonably, we can give full play to theirrespective advantages and improve the overall production efficiency and economic benefits.

54©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS3-6 A Deep Exploration of the Mounting Issues Related to Six Rotor UVA
Yuping Mei, Ying Su, Xue Yang (Tianjin University of Science and Technology, China)
In recent years, the hexacopter UAV has developed rapidly, especially the technologicalbreakthrough in the field of automatic driving, which is of great significance in both militaryand civilian fields. The six-rotor UAV uses six rotors as the power source and adjusts the attitude
by changing the rotor speed to further achieve position control. It has excellent hovering ability
and sensitivity, and is equipped with a precise positioning system and advanced sensors.
However, there are still common problems such as weak mounting capacity and single mounting
mode. In view of this problem, this paper will test the fuselage structure, avionics system andpower system one by one from two aspects: changing the fuselage structure and installing the
motor position.

OS3-7 The Application of Hexacopter UAV in The Field of Climbing Evasion
Ying Su, Yuping Mei, Xue Yang (Tianjin University of Science and Technology, China)
With the development of tourism to climb peaks, the safety of high mountain walls plays an
increasingly important role. In the past, the terrain of the mountain wall was too steep and too
fast, and people could not patrol the mountain wall climbers, resulting in the safety of climbers
could not be guaranteed, and the needs of climbers were difficult to solve in time. In order tosolve the above problems, the method of real-time monitoring of UAVs and delivery ofmaterials by UAVs was proposed. Hexacopter UAV is an unmanned small aerial vehicle
equipment that can carry out vertical lifting, which is expected to improve the safety factor of
climbing mountain walls and solve the needs of climbers in a timely manner.

OS3-8 Deep Learning and Embedded Based Operational Safety System for Special Vehicles
Haoran Gong, Yumei Huang, Jiahao Xie (Tianjin University of Science and Technology, China)
Aiming at the characteristics of special operation vehicles, such as complex working
environment, large vehicle weight, long braking distance, and many visual dead angles fordrivers, this project designs and implements a special vehicle operation safety system based on
deep learning and embedded system. First of all, deep learning model is applied to detect thehumanoid target appearing around the vehicle, get the detection frame of the humanoid target
and estimate the distance from the target to the camera. According to different distancedistances, relevant voice announcements are made to remind the driver to make timely actions
such as avoiding or braking the vehicle.

OS3-9 "Green Fruit" - Intelligent Traceable Agricultural Product Production and Marketing
Platform Based on Blockchain Technology
Yumei Huang, Jiahao Xie, Haoran Gong, Ziyue Xiao (Tianjin University of Science and Technology, China)
The production process of agricultural products has been greatly improved and optimized
through the "Qingguo" intelligent agricultural greenhouse system. Using corrosion-resistant,ageing-resistant, solar-spectrum-converting plastic film, combined with Internet of Things
technology, a simple, highly automated and intelligent greenhouse intelligent control system
has been created. This not only improves the yield and quality of agricultural products, but also
effectively reduces production costs and risks.

55©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS4 Pattern Recognition and Robotics III (10)Chair Huahao Li (Tianjin University of Science and Technology, China)
Co-Chair Hongshuo Zhai (Tianjin University of Science and Technology, China)
OS4-1 A Digital Twin Design Based on Robot Workstation
Huahao Li (Tianjin University of Science and Technology, China)
With the introduction of Made in China 2025 and Industry 4.0, digital twins have rapidly
become a trend. By interacting with physical objects and virtual models, mapping iscompleted in the virtual simulation space to reflect the actual operation process of the entire
lifecycle of the corresponding device workstation. This article starts from the researchbackground and significance of digital twin technology, the establishment of Solidworks
physical model, the construction of PDPS workstation, and virtual simulation. We havecompleted the construction of a robot seven color panel assembly workstation model andcombined it with PDPS to simulate the workstation.

OS4-2 A Study of Chemical Reactor Simulation System Based on PCS7
Hongshuo Zhai (Tianjin University of Science and Technology, China)
This paper introduces a simulation method of plant control, and corrects the system throughPID Tuner, and finally gets a reasonable and correct simulation curve. In this paper, PCS7
software is used to build the factory model, and the simulation of four circuits of feed,
pressure, liquid level and temperature is built by CFC block in the software. The automatic
operation of the system is completed by SFC block. The final simulation curve obtained by
PID Tuner tool based on PCS7 software is optimized.

OS4-3 Analysis of Learning Quality Evaluation for University Student Courses with Process
Assessment
Yuhao Zhang, Ying Gong, Xuran Wang (Tianjin University of Science and Technology, China)
With the progressive exploration and application of formative assessment in university
pedagogy, this evaluative method has become widely adopted for appraising students'
everyday learning attitudes and conditions. Drawing upon pertinent data regarding classroom
learning experiences of students at a specific university, this paper employs machine learning,
K-means clustering, the Topsis evaluation model, and the entropy weighting method toinvestigate the relationship between formative assessment and the quality of university student
learning, culminating in the creation of an evaluation model. This model allows us to pinpoint
the key factors influencing student learning attitudes and offers support for formative
assessment in the university context.

OS4-4 Motion Analysis and Transfer Applications Based on Posture Recognition
Yuhao Zhang, Mingyue Li, Jianhao Jiao (Tianjin University of Science and Technology, China)
This paper investigates the posture trajectories in human motion using pose recognition
technology based on the deep learning framework MediaPipe. By detecting key points on thehuman body and plotting and calculating these trajectories in the temporal dimension, we
further conduct comparative analysis of these trajectories with professional sports coaches'motion guidance to assist athletes in correcting their posture. Additionally, this technology has
been deployed on Jetson Nano, enabling its practical application in mobile scenarios andproviding robust tools and methods for fields such as rehabilitation therapy, sports training,
and animal behavior analysis. This study offers insights into the transfer applications of
posture recognition.

56©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS4-5 Functional Safety Assessment of the Safety Protection System Based on Petri Net
Peng Wang, Mengyuan Hu (Tianjin University of Science and Technology, China)
In this paper, the functional safety evaluation of the safety protection system of gasolinehydrogenation unit was carried out using Petri net. Firstly, the principle and framework of thegasoline hydrogen refueling unit was described. Secondly, the safety integrity level was
introduced, and the influencing factors of the safety integrity level were summarized. Thirdly,
the Petri net model and the Markov model are compared and the Petri net model is used toverify its security integrity level. Finally,the calculation result demonstrated that the SIL did
not reach the target level, and then reached the target level after improvement.This analysis
method can provide reference for the safety integrity level evaluation of similar devices.

M3

M9
M10

M5

T9

M1

M7

`
M0

T1
T0

M8

M11
M6

M2

M4

OS4-6 Pedestrian Attribute Recognition Based on Deep Learning
Peng Wang, Qikun Wang, Shengfeng Wang (Tianjin University of Science and Technology, China)
This paper studied pedestrian attribute recognition based on deep learning, for its importancein the fields of smart city construction. Firstly, the research status of pedestrian attributerecognition and common deep learning models was introduced. Secondly, considering theaccuracy decline problem and gradient problem of the neural network, the residual network
was used as the main body of the neural network model. Thirdly, the model was trained toclassify multiple person attributes through two data sets, Market-1501 and DukeMTMC-reID.
Finally, the pedestrian attribute recognition model was tested, and good results were obtained.

OS4-7 Simulation of office air conditioning air supply based on COMSOLPeng Wang*, Mengda Liu,Qikun Wang (Tianjin University of Science and Technology, China)
This paper analyzed the summer thermal environment of an office in Tianjin University of
Science and Technology based on COMSOL software.Firstly,the principle of indoor thermal
environment distribution was introduced. Secondly, according to the actual position of thefurniture in the office, the mathematical model is constructed based on the basic theory ofcomputational fluid dynamics.Thirdly, the COMSOL software was used for simulation andcalculation, combining boundary conditions.Finally, the simulation results are analyzed
through the simulated indoor three-dimensional velocity field and temperature field. The
simulation results show that the air conditioning supply can well achieve indoor occupants'
comfort.

OS4-8 Solo Wheel Technology-Self-balancing Wheelbarrow
Ziyue Xiao, Yumei Huang, Zhencheng Chang, Mingxuan Li(Tianjin University of Science and Technology, China)
As a new type of personal transportation, self-balancing unicycle has attracted wide attention
with its unique design and advanced control system. We aim to study and optimize the selfbalancing performance of the unicycle to improve its stability and maneuverability. Firstly,
we gain an in-depth understanding of its operation principle. On this basis, an advanced
control algorithm is proposed to realize real-time attitude adjustment. Second, for applications
in complex environments, we propose an intelligent sensing system to enhance its
environment sensing capability.

57©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS4-9 Second-order self-balancing inverted pendulum
Ziyue Xiao, Zhencheng Chang, Mingxuan Li, Yumei Huang(Tianjin University of Science and Technology, China)
The team utilizes the second-order inverted pendulum cart based on LQR controller for steadypendulum control with light rods. First, the second-order inverted pendulum is used as a research
object to obtain its set of dynamical equations, then, the set of dynamical equations is written in
the form of state-space expressions, and finally, the second-order inverted pendulum system of abalanced trolley with a light rod is controlled by a stabilized pendulum using the LQR controller.
The inverted pendulum is a typical nonlinear, underdriven and unstable system, which can realize
the all-round wind resistance of the rod and occupy less space with high stability.

OS4-10 Design of Nanny's Abnormal Behavior Recognition Bracelet Based on Human Activity
Recognition (HAR) Deep Learning Model
Depeng Wang, Yingfan Zhu, Yande Xiang, Ziyue Xiao (Tianjin University of Science and Technology, China)
This article designs a nanny abnormal behavior recognition bracelet. The bracelet is equippedwith multiple sensors and a powerful control board, forming a complete nanny abnormal behavior
recognition system, which realizes nanny abnormal behavior recognition and alarm in various
environments. It uses a gyroscope to collect the three-axis acceleration information of the nanny,and uses the HAR model to infer the nanny's real-time behavior. When the abnormal behavior of
the nanny is inferred, STM32 sends the GPS collected positioning information to the employer
through ESP32 for timely alarm.

OS5 Intelligent Life and Robotics (6)Chair Kuo-Hsien Hsia (National Yunlin University of Science and Technology, Taiwan)
Co-Chair Evgeni Magid (Kazan Federal University, Russia)
OS5-1 Research on dynamic obstacle avoidance and complex path planning strategies based on
ROS robots
Yi-Wei Chen, Jr-Hung Guo (National Yunlin University of Science and Technology, Taiwan)
Robot Operating System (ROS) is a software system framework used by many robot systems.
Although ROS provides a good development environment and related frameworks, ROS is notsuitable for public places such as restaurants because of the coming and going of people. Dynamic
obstacle avoidance is often handled by stopping the robot, or when there are frequentenvironmental map changes or when sensors such as optical radar fail, the stop action is also
used. However, this often causes path obstructions or delays in completing tasks. Therefore, thisstudy attempts to use images, auxiliary sensors, and various path avoidance strategies to solve
the problem of the robot stopping and waiting for the obstacles to disappear. The problem of rapidchanges in map paths.

OS5-2 Research on Multi-Robot Formation on Two-Dimensional Plane
Kuo-Hsien Hsia, Chun-Chi Lai, Yi-Ting Liu, Yu-Le Chen(National Yunlin University of Science and Technology, Taiwan)
Mobile robots are playing an increasingly important role in both service and manufacturingindustry. The management of multiple mobile robots is a very important issue on the research ofmobile robotics. From a mathematical perspective, this paper discusses the problem of multiple
robots on a two-dimensional plane reaching the designated positions in the shortest time tocomplete formation transformation. We improved the algorithm proposed by Hsia, Li and Su andproposed a new algorithm using a determinant and the Munkres allocation algorithm. Finally, the
new algorithm is compared with the path distribution obtained by the Monte Carlo method under
different numbers of robots and the excellence of the new algorithm has been verified.

58©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS5-3 The Development of Utilization Rate and Energy Consumption Monitoring and
Networking System
Chung-Wen Hung, Chun-Chieh Wang, Heng-En Chang(National Yunlin University of Science and Technology, Taiwan)
In this paper, we present a system based on microcontroller unit for measuring the utilization
rate of traditional, non-networked machinery. This equipment is designed for use with older
machines equipped with andon lights. It employs optocoupler circuits to capture the status ofthese lights and current transformers to measure their operating and standby currents. Data istransmitted to the server using the Hypertext Transfer Protocol (HTTP) in JavaScript Object
Notation (JSON) format. On the server side, a PHP interprets the data, connects to a
Structured Query Language (SQL) database, and stores the data using SQL commands. Users
can access graphical data through a web-based interface, using it to refine production
processes, reduce production costs, and minimize carbon emissions.

OS5-4 MCU Based Edge Computing Platform for Liquid Level Measurement
Chung-Wen Hung, Chun-Liang Liu, Tai-Hsuan Wu(National Yunlin University of Science and Technology, Taiwan)
An edge computing system based on micro control unit (MCU) for liquid level measurement
is proposed in this paper. The system includes a solenoid electromagnet for bottle hit and a
microphone to capture sound waves. The signals are converted from time domain tofrequency domain by Fast Fourier Transform (FFT), employing an artificial intelligence (AI)model to predict the water level. Artificial Neural Network (ANN) model is applied forclassification on the MCU. When optimizing hyperparameters, the accuracy of eachparameter combinations should be considered. Ensure the model size suits the limited MCU
memory and computing capabilities. Experimental results confirm the system's effectivenesswith 99% accuracy.

OS5-5 Potential of genetic algorithms in multi-UAV coverage problem
Ramil Faizullin1, Tatyana Tsoy1, Edgar A. Martínez-García2, Evgeni Magid1,3(1Kazan Federal University, Russian Federation)
2( The Autonomous University of Ciudad Juarez, Mexico)(3HSE University, Russian Federation)
For a rapid area coverage multiple UAVs are often used simultaneously. However, a path
planning for a UAVs group during an area coverage task is computationally challenging. In
practice, heuristic algorithms are applied to solve this problem. This paper overviews
approaches to an area coverage problem with a group of UAVs using genetic algorithms. The
article explores modifications that may be useful for a genetic algorithm for solving thecoverage problem as well as representation methods for chromosomes that reflect a path ofmulti-UAV. Additionally, UAV group collision avoidance strategies during area coverage areconsidered.

59©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS5-6 Construction of Anthropomorphic Grippers with Adaptive Control
Evgeny Dudorov1, Julia Zhdanova2, Ivan Zhidenko1, Vladimir Moshkin2, Alexander Eryomin3, Evgeni
Magid3, Alexander Permyakov1
1( JSC ‘SPA ‘Android technics’, Russian Federation)(2MIREA – Russian Technological University, Russian Federation)(3Kazan Federal University, Russian Federation)
A functionality of a service robot that physically interacts with a human is provided primarily
by capabilities of its end effector. Rather limited capabilities of industrial robot grippers
determined a gradual transition to anthropomorphic grippers. A number of degrees of freedom(DoF) of an end effector ensuring reliable grasping and holding of an arbitrary shaped object
should be at least nine, preferably twelve. Such design implementation requires to switch
toward underactuated grippers systems. This paper proposes a concept of constructing a groupdrive, which enables a motion of output links of two or more executive groups from a single
motor. The presented technical solutions are based on methods of analyzing complex
mechanical systems using functional circuits.

OS6 Intelligence and Optimization (5)Chair Mastaneh Mokayef (UCSI University, Malaysia)
Co-Chair Takao Ito (Takao Ito, Hiroshima University, Japan)
OS6-1 Simulation-Based Enhancement of SNR in Drone Communication through Uniform
Linear Array Configurations
Gershom Phiri1, Mastaneh Mokayef1, MHD Amen Summakieh1, M.K.A Ahamed Khan1, Sew Sun Tiang1,
Wei Hong Lim1, Abdul Qayyum2
1
2( UCSI University Malaysia, National Heart and Lung Institute, Imperial College London, UK)As drones navigate through shared airspace, they often encounter other drones, wirelessdevices, and communication systems. This coexistence creates potential sources ofinterference that can degrade the signal-to-noise ratio (SNR). To maintain reliablecommunication in drone systems, it is crucial to effectively manage and mitigate interference
from other drones and wireless devices operating on the same frequency bands. By addressingthese challenges, we can ensure a stable and dependable SNR for seamless communication
among drones. This paper sheds light on the history of applications and challenges of utilizing
flying base stations for wireless networks and analyzes different factors that affect signal-tonoise ratio (SNR) to enhance the performance of drone communication.

OS6-2 Empowering Elderly Individuals through the Intelligent Shopping Trolley
Mastaneh Mokayef1, Muzaiyanah Binti Hidayab1, MHD Amen Summakieh1, M.K.A Ahamed Khan1,
Kim Soon Chong1, Chin Hong Wong2, Chua Huang Shen3, Abdul Qayyum4
1( UCSI University Malaysia, 2 Fuzhou University, China, 3 UOW Malaysia University, Malaysia,
4
National Heart and Lung Institute, Imperial College London, UK)
In this research, we have developed a prototype that aims to improve the weekly shopping
experience for senior citizens. Our system tracks the movements of elderly individuals,
eliminating the need for them to exert physical force in pushing or pulling the shopping trolley.
To achieve this, we utilize a combination of sensors, including a gyroscope and magnetometer,
to estimate the user's walking distance and direction. Additionally, we employ WiFi
fingerprinting to accurately determine the user's position. Our experiments have yieldedsatisfactory results in terms of tracking accuracy and the overall functionality of the system.
By addressing the specific challenges faced by senior citizens during the routine and essential
process of grocery shopping, our smart shopping trolley concept seeks to enhance theirexperience. Through the integration of tracking technology and sensor-based solutions, we
aim to make shopping more convenient and comfortable for elderly individuals. The positive
outcomes observed in our experiments validate the effectiveness and feasibility of thisapproach.

60©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS6-3 Optimized Microstrip Slot UWB Patch Antenna for Medical Imaging
Maxime Duvacher1, Mastaneh Mokayef2, MHD Amen Summakieh2, M.K.A Ahamed Khan2, Sew Sun Tiang2,
Wei Hong Lim2, Abdul Qayyum3 (1Polytech Nantes, France, 2UCSI University Malaysia,
3
National Heart and Lung Institute, Imperial College London, UK)
This research work presents the development of an Ultra-Wideband (UWB) microstrip patch
antenna (MPA) with the specific purpose of tissue characterization. The antenna was carefully
designed and simulated to operate within a frequency range of 4.8 to 6.9 GHz, optimized for its
intended application. To ensure the best performance, a series of simulations and comparisonswere conducted using CST Microwave Studio. Various antenna shapes were tested andevaluated to determine the most effective design. The results of these simulations were highly
promising, as they revealed a simulated return loss (S_11) of -33dB. This indicates excellent
performance and demonstrates the suitability and acceptability of the proposed antenna for
medical imaging such as breast imaging, tumor detection, or monitoring physiological changes.

OS6-4 Development of an Innovative Undergraduate Industrial Automation and Robotics
Degree Program
M.K.A. Ahamed Khan1, Mastaneh Mokayef1, Ridzuan, A.1, Irraivan Elamvazuthi, Badli Shah Yusoff 2,
Abu Hassan Darusman3 (1UCSI University Malaysia, 2University Technology Petronos, Malaysia,
3
UNIKL Malaysia France Institute, Malaysia)
In recent years, the need for integrated engineering courses has increased. Due to its
multidisciplinary nature, Industrial Automation and Robotics degree course is an ideal example
of curriculum integration. This paper discusses several issues such as course offerings, topicalcontent, student profile, student performance and other pertinent matters related to the recent
development of an Industrial Automation and Robotics undergraduate degree program at the
University of Kuala Lumpur, Malaysia.

OS6-5 Smart Assistive Trolley for Elderly Care and Independence
Dina Ashraf1, Mastaneh Mokayef1, MHD Amen Summakieh1, M.K.A Ahamed Khan1, Abdul Qayyum2,
Sivajothi A/L Paramasivam (1UCSI University Malaysia, 2National Heart and Lung Institute, Imperial College
London, UK, 3 UOW Malaysia University College, Malaysia)
As people age, shopping can become increasingly challenging, especially when it involves
pushing heavy trolleys and managing items throughout the entire trip. To address these
difficulties, a prototype project was developed with the aim of introducing a robotic trolley. This
innovative trolley is designed to autonomously follow senior citizens during their shopping
journeys, thanks to face detection technology. The system eliminates the need for physicalassistance and provides real-time feedback on the distance covered during the trip. To estimate
the walking distance, the trolley incorporates an ultrasonic sensor that activates when the camera
detects and tracks the user. The implementation of this project utilizes the OpenCV library,specifically tailored for Python programming. The results of this project have shown significant
improvements in the lives of elderly individuals, offering them enhanced comfort and increased
independence during their shopping experiences.

61©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS7 Deep Learning and its Applications (4)Chair Mastaneh Mokayef (UCSI University, Malaysia)
Co-Chair Takao Ito (Hiroshima University, Japan)
OS7-1 Parallel Cross Window Attention Transformer and CNN Model for Segmentation ofInstrument during Surgery
Abdul Qayyum1, M. K. A. Ahamed Khan3, Moona Mazher4, Imran Razzak5, Steven Niederer1,2,
Mastaneh Mokayef3, C.S. Hassan3, Ridzuan, A3(1Imperial College, London, UK) (2The Alan Turning Institute, UK) (3UCSI University, Malaysia)(4University College London, UK) (5University of New South Wales, Australia)
In this work, we present encoder and decoder-based hybrid parallel cross window
attention-based transformer during the feature extraction, which consists of the multiscale channel attention, convolutional layers, and Transformer layers, forming a unified
block. Syn-ISS challenge dataset comprised of two tasks. In first task 1, they need todevelop deep learning-based method for binary instrument segmentation and in second
task multiclass instrument segmentation is required. Experiments conducted on Syn-ISS
dataset achieved 0.993 F-score for task 1 and 0.993, 0.975, and 0.951 F-score for shaft,wrist, and jaw segmentation respectively for Task 2.

OS7-2 Magnetic Resonance Spectroscopy (MRS) Reconstruction using Style Transfer Deep
Depth wise Framework
Abdul Qayyum1, M. K. A. Ahamed Khan3, Moona Mazher4, Imran Razzak5, Steven Niederer1,2,
Mastaneh Mokayef 6, C.S. Hassan6, Ridzuan, A6
1( Imperial College, London, UK) (2The Alan Turning Institute, UK)(3,6UCSI University, Malaysia) (4University College London, UK)(5University of New South Wales, Australia)
To analyze the chemical composition of tissues in brain, in vivo magnetic resonancespectroscopy allows non-invasive measurements of neurochemicals in either single
voxel or multiple voxels. In this work, we present a deep depth-wise channel attention
module (DCAM) based fine-tuned network for magnetic resonance spectroscopy image
reconstruction. Besides, we have used channel-wise convolutions and average poolingwithout dimensionality reduction. We have trained the initial network from scratch on
track-1 simulated dataset, however due to the limited dataset, we finetune the networkon track-2 and track-3. Experiments are conducted on Edited-MRS-Rec-Challenge
dataset1 that showed significantly better performance.

OS7-3 Federated Learning on Brain Disease Research: Segmentation of Cerebral Small Vessel
Diseases (CSVD) using Multi-scale Hybrid Spatial Deep Learning Approach
Moona Mazher1, Abdul Qayyum2, M. K. A. Ahamed Khan3, Steven Niederer2,4, Mastaneh Mokayef3,
Ridzuan, A3, C. S. Hassan3(1University College London, UK) (2National Heart & Lung Institute, Imperial College, London, UK)(3UCSI University, Malaysia) (4Alan Turning Institute, London, UK)
In this paper, we propose a hybrid architecture for medical image segmentation to
produce efficient representations from global and local features and adaptively
aggregate them, aiming to fully exploit their strengths to obtain better segmentation
performance in federated learning. Furthermore, we propose a multi-scale feature
extraction module embedded at the bottom of the proposed model, which can efficiently
extract hidden multi-scale contextual information and aggregate multi-scale features.
Experiments on segmentation over three-dimensional rotational angiography of internal
Carotid Artery with aneurysm (SHINY-ICARUS) challenge dataset show theeffectiveness of the proposed multiscale framework.

62©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS7-4 Hybrid Classical and Quantum Deep Learning Models for Medical Image Classification
Moona Mazher1, Abdul Qayyum2, M. K. A. Ahamed Khan3, Steven Niederer2,4, Mastaneh Mokayef3,
Ridzuan, A3, C. S. Hassan3
12( University College London, UK) ( National Heart & Lung Institute, Imperial College, London, UK)(3UCSI University, Malaysia) (4Alan Turning Institute, UK)
In this paper, we proposed a hybrid classical and quantum convolutional neuralnetwork for Alzheimer's disease (AD) classification. The proposed model was further
validated on the brain tumor classification task. The fundamental concept involves
encoding data into quantum states, facilitating quicker information extraction, andsubsequently utilizing this information to discern the data class. The proposed model
results underscore the reliability and robustness and demonstrated by optimal
performance accuracies across various datasets, the proposed model substantiates its
efficacy in detecting and classifying AD disease and brain tumors.

OS8 Intelligent Control (5)Chair Yingmin Jia (Beihang University, China)
Co-Chair Weicun Zhang (University of Science and Technology Beijing, China)
OS8-1 Global Stabilization of A Class of Nonholonomic Integrators via Discontinuous Control
Lixia Yan, Yingmin Jia (Beihang University (BUAA), China)
This paper investigates the discontinuous state feedback control for stabilizing a classof nonholonomic integrators with drift terms. The control design relies on constraining
state trajectory in an invariant set. To this end, we apply constant controls to drive thestates moving into the invariant set and then switch to a continuous control law withsuitable gain selections. It is proven in the Lyapunov sense that the proposed control
scheme achieves global exponential stabilization of the states, and the control switch
would only occur at most once. Numerical simulations are carried out to validate theproposed control law.

OS8-2 Frequency Dependence Performance Limit of Vibration Absorbers
Jiqiang Wang1, Xiaoyu Yin2, Weicun Zhang3( Chinese Academy of Sciences, China) (2Science & Technology Bureau of Zhenhai District, China)(3University of Science and Technology Beijing, China)
1

Optimal design of vibration absorbers has been extensively investigated. Most of thedesign methods are approached by optimizing certain performance indices, resulting ina set of optimal parameters that are independent of exogenous forcing frequencies. In
practical designs, however, it is often desirable to know the performance limits over afrequency band of interest. This problem is tackled in the present paper where both
lower and upper bounds are obtained. A refined upper bound is also derived that canfurther provide a systematic design methodology. Extensive remarks are also given
exploring different avenues useful for design. Numerical examples are given to validate
the corresponding designs.

OS8-3 Adaptive Concurrent Learning Algorithm Based on Pontryagin’s Maximum Principle for
Nonlinear System Optimal Tracking Control with State Inequality Constraints
Yuqi Zhang, Bin Zhang (Beijing University of Posts and Telecommunications, China)
In this article, an adaptive learning algorithm is derived for finite-horizon optimal
tracking problems of nonlinear systems. Concurrent learning is implemented to identify
the unknown parameters of the system dynamics. Tracking of the desired trajectory andconvergence of the developed policy are guaranteed via stability analysis. Different
from the adaptive dynamic programming (ADP) infinite-horizon method, the proposed
method solves the finite-horizon optimal tracking problems. Simulation results show
the effectiveness of the proposed algorithm.

63©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS8-4 Privacy preserving Mean-square consensus for discrete-time heterogeneous multi-agent
systems with Communication Noises
Tongqing Yang1, Lipo Mo1, Yingmin Jia2
( Beijing Technology and Business University, China) (2Beihang University (BUAA), China)
1This paper studies the privacy preserving consensus of distributed heterogeneous multiagent systems, which consists of first-order agents and second-order agents. A newprotocol is designed for each agent. Then, by applying algebraic graph theory andmatrix theory, it is proved that the closed-loop system could achieve consensus. After
that, the designed protocol is encrypted by a cryptographic cryptosystem to prevent
eavesdropping in the communication link and loss of information privacy between
agents while achieving consensus. Finally, the effectiveness of the proposed consensus
protocol and privacy protection algorithm are verified by numerical simulations.

OS8-5 Event-Triggered Consensus Control for Nonlinear Singular Multi-Agent Systems underDirected Topology
Lin Li, Tong Yuan, Mei Huang (University of Shanghai for Science and Technology, China)
This paper is devoted to the problem of event-triggered consensus for a class ofnonlinear singular multi-agent systems under directed topology. An event-triggered
sampling mechanism is constructed, which naturally avoids Zeno behavior. And then,a distributed event-triggered consensus protocol is designed. By employing the
Lyapunov-Krasovskii functional method and model transformation approach,sufficient conditions that can guarantee the consensus of the considered singular multiagent systems are obtained, while the consensus control gain matrix and the eventtriggered parameter are also given. Finally, a numerical example is included to illustrate
the effectiveness of the proposed method.

OS9 Software Development Support Method (4)Chair Tetsuro Katayama (University of Miyazaki, Japan)
Co-Chair Tomohiko Takagi (Kagawa University, Japan)
OS9-1 An Improved Conversion Technique from EPNAT Models to VDM++ Specifications for
Simulation of Abstract Software Behavior
Sho Matsumoto1, Ryoichi Ishigami1, Tetsuro Katayama2, Tomohiko Takagi1(1Kagawa University, Japan) (2University of Miyazaki, Japan)
Formal software models based on EPNAT (Extended Place/transition Net with
Attributed Tokens) can be converted to VDM++ specifications that enable simulation of
abstract software behavior before implementation processes. However, the conversion
technique has two problems, that is, (1) extracting all properties to be checked from the
VDM++ specifications requires time and effort, and (2) the structure of the VDM++
specifications has less readability and maintainability. In this study, we improve theconversion technique by (1) adding a function to extract an abstract current state ofsoftware, and (2) dividing into classes that correspond to subnets of EPNAT models.
This paper shows a new conversion rule, a new structure of VDM++ specifications, a
simple example, and the discussion about their effectiveness.

64©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS9-2 Prototype of RAGESS Which Is a Tool for Automatically Generating SwiftDiagrams to
Support iOS App Development
Haruki Onaga1, Tetsuro Katayama1, Yoshihiro Kita2, Hisaaki Yamaba1, Kentaro Aburada1, Naonobu Okazaki1(1University of Miyazaki, Japan) (2University of Nagasaki, Japan)
In the development of large-scale and complex mobile applications, it is difficult fordevelopers to continually grasp the overall structure of the app. To support iOS app
development, we proposed SwiftDiagram, a visualization of the static structure of Swiftsource code and confirmed its high usefulness. However, manually drawing
SwiftDiagrams is labor-intensive. This paper implements a prototype of RAGESS (Realtime Automatic Generation of SwiftDiagram System), a tool that automatically generates
SwiftDiagrams by performing static analysis on Swift source code every time an iOS app
build is successful. Compared with other tools, RAGESS is confirmed to enable
developers to visualize the static structure of source code effortlessly.

OS9-3 Extension to Support Types and Operation/Function Definitions in BWDM to Generate
Test Case Tool from the VDM++ Specification
Shota Takakura1, Tetsuro Katayama1, Yoshihiro Kita2, Hisaaki Yamaba1, Kentaro Aburada1, Naonobu
Okazaki1
1( University of Miyazaki, Japan) (2University of Nagasaki, Japan)
Generating test cases from the formal specification description VDM++, which is amethod for disambiguating specifications, is time-consuming and labor intensive.
Therefore, our laboratory has developed BWDM, a tool that automatically generates test
cases from VDM++ specifications. However, existing BWDM has problems that it only
supports integer types and cannot generate test cases for operation and function definitions
including recursive structure. Therefore, in order to improve the usefulness of BWDM,
this study extends BWDM to solve the above problems. Consequently, it has confirmedthat the use of extended BWDM can reduce the test case generation time compared tomanual test case generation.

OS9-4 Proposal of ASLA Which Is a Segmentation and Labeling Tool for Document Images
Based on Deep Learning
Kanta Kakinoki1, Tetsuro Katayama1, Yoshihiro Kita2, Hisaaki Yamaba1, Kentaro Aburada1, Naonobu Okazaki1(1University of Miyazaki, Japan) (2University of Nagasaki, Japan)
The current situation of the electronic documents is only a substitute for paper. As a new
way to utilize electronic documents, we focus on dividing electronic documents into
regions by their elements and generating keywords and sentences as labels from thecontents of the elements. However, these tasks, when performed manually, are timeconsuming and labor-intensive. This study proposes a prototype of ASLA (Automatic
Segmentation and Labeling tool using AI), segmentation and labeling tool for document
images based on deep learning, with the aim of reducing the time required for region
segmentation and label generation. To evaluate the usefulness of ASLA, we havecompared the time required for region segmentation and label generation using ASLA andby hand, and then confirmed the reduction in time.

65©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS10 Intelligent Life and Robotics (6)Chair Evgeni Magid (Kazan Federal University, Russia)
Co-Chair Kuo-Hsien Hsia (National Yunlin University of Science and Technology, Taiwan)
OS10-1 A Design of a Modular Mobile Robot for Rescue Operations
Baris Celiker, Shifa Sulaiman, Tatyana Tsoy (Kazan Federal University, Russia)
Modular robotics is one of the subfields of mobile robotics, which is emerging as a new
trend in various sectors. Modular mobile robots can be reconfigured to perform a wide
variety of tasks. In this paper, applications of modular mobile robots in various sectors
such as industry, space, surgery, rescue and entertainment tasks are discussed. Based on
the study, an improved design of a modular mobile robot for navigating through different
terrains during a rescue operation is presented. Simulation study of the robot is included
to demonstrate a motion capability of the modular mobile robot.

OS10-2 Implementation of Bug1 and Bug2 Path Planning Algorithms for TurtleBot Using ROS
NoeticIlya Spektor1, Aidar Zagirov2, Ramil Safin2, Evgeni Magid1,2(1HSE University, Russia) (2Kazan Federal University, Russia)
Mobile robots typically operate in a dynamically changing unknown environments. Bug
family algorithms were proposed to deal with a path planning of a ground robot in a 2D
configuration space of an unknown environment with local data about obstacles that
could be collected using only a touch sensor. This paper presents an implementation of
Bug1 and Bug2 local path planning algorithms. The implementation brings classical 2D
algorithms into a 3D environment of the Gazebo simulation using the Noetic version of
the Robot Operation System (ROS). A Turtlebot3 Burger model was used as a target
robot and its performance was evaluated in simple convex and maze environments.

OS10-3 Implementation of Alg1 and Alg2 Path Planning Algorithms for Mobile Robots Using
ROS Noetic
Anastasia Yankova1, Timur Gamberov2, Tatyana Tsoy2( HSE University, Russia) (2Kazan Federal University, Russia)
1

Two standard approaches for a robot path planning include a global and a local
navigation. The later does not require to store an environment model in a robot memory.
This paper presents implementations of two local navigation algorithms, Alg1 andAlg2, with a robot having no prior information about an environment and obstacles. It
calculates a path in a real time, continuously changing its states depending on
correspondent conditions. The algorithms were implemented for an existing differential
drive robot Turtlebot3 Burger using Robot Operation System (ROS). Virtual
experiments were performed in the Gazebo simulator employing a simple 3D
environment with only convex obstacles and a small 3D maze.

66©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS10-4 Implementation of VisBug-21 and VisBug-22 Path Planning Algorithms Using ROS
NoeticViktoriia Mirzoian1, Maxim Mustafin2, Evgeni Magid1,2( HSE University, Russia) (2Kazan Federal University, Russia)
1

Local navigation algorithms are crucial for autonomous robots operating in unknown
environments where a presence of obstacles and dynamic changes pose significant
challenges. A focus of these algorithms is to enable a real-time path calculation,
allowing a robot to adapt its states dynamically based on corresponding environmental
conditions, despite an absence of prior knowledge about surroundings. This paper
presents an implementation of the VisBug-21 and VisBug-22 algorithms, designed toaddress challenges of a local navigation. The algorithms were implemented for adifferential drive robot Turtlebot3 Burger using Robot Operation System (ROS).
Virtual experiments were performed in the Gazebo simulator employing a simple 3D
environment that contained only convex obstacles and a small 3D maze.

OS10-5 DistBug path planning algorithm package for ROS Noetic
Alexander Pak1, Alexander Eremin2, Tatyana Tsoy2( HSE University, Russia) (2Kazan Federal University, Russia)
1

Algorithms of path-planning in an unknown environment play an important role inrobotics. They do not require a prior information about obstacles’ locations around arobot and allow calculating a path in a real time. This article presents animplementation of a sensory-based DistBug algorithm, which operates reactively using
range data for immediate decision-making without constructing a world model. The
algorithm was programmed in Python using robot operating system (ROS) andvalidated in the Gazebo simulator. For virtual experiments Turtlebot 3 Burger mobile
robot was employed. The experiments were conducted in two types of environment:
an environment with convex obstacles and a maze. The paper demonstrates analysis ofexperiments using several standard criteria of a path quality estimation.

OS10-6 On sensor modeling in Gazebo simulator
Niez Yuldashev, Alexandra Dobrokvashina, Roman Lavrenov (Kazan Federal University, Russia)
Sensor modeling in the Gazebo simulator is fundamental to robotics advancement. This
review explores sensor modeling intricacies, methodologies, and applications, whileemphasizing a critical role of a precise sensor modeling. Application scenarios
demonstrate a sensor modeling's broad utility across fields including medical
diagnostics, autonomous navigation, and industrial automation. Differences in research
focus, methodology, and implementation underline a varied nature of sensor modelingstudies. Key challenges include a need for more detailed world models. The paper
guides research in sensor modeling and identifies crucial questions.

67©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS11 Intelligent Life and Robotics (6)Chair Evgeni Magid (Kazan Federal University, Russia)
Co-Chair Kuo-Hsien Hsia (National Yunlin University of Science and Technology, Taiwan)
OS11-1 An Overview of Kinect Based Gesture Recognition Methods
Alexander Alexeev1, Tatyana Tsoy1, Edgar A. Martínez-García2, Evgeni Magid1,3( Kazan Federal University, Russia) (2The Autonomous University of Ciudad Juarez, Mexico)(3HSE University, Russia)
1

Visual sensors play an important role in a broad variety of robotic systems applications.
Even though Kinect technology appeared over 10 years ago, Kinect sensors are still
actively employed by researchers around the world. This paper presents an overview ofKinect and Kinect 2 sensors’ applications in a human gesture based control. We analyzed
existing research papers to estimate a popularity of particular feature extraction andgesture recognition methods, recommendations on a distance between an object ofinterest and a sensor, reported accuracy and latency of the sensor. Our analysis is
supposed to facilitate a selection of a suitable combination of methods for a particularapplication of Kinect sensor in gesture recognition while considering its performance.

OS11-2 An investigation on the impact of human-robot interactions during an autonomous
obstacle avoidance task
Riham Salman, Shifa Sulaiman, Renata Islamova, Tatyana Tsoy (Kazan Federal University, Russia)
The purpose of this research is to investigate how an interaction between humans androbots influences a safety of an autonomous obstacle avoidance task. The researchcollected and analyzed data from surveys and interviews using a combination ofquantitative and qualitative methodologies. The findings contributed to ourunderstanding of a complex interplay between a human-robot interaction, a perception,
and a robot navigation safety. Based on these findings, the study proposed a number ofrecommendations for improving both physical and psychological safety aspects of anautonomous robot navigation.

OS11-3 A Comparative Analysis of Object Detection Methods for Robotic Grasping
Nikita Kolin, Elvira Chebotareva (Kazan Federal University, Russia)
The objects grasping is one of the fundamental robotic problems. Accurate and efficientreal-time object detection is crucial for successful grasping in robots equipped withmonocular vision. Deep machine learning has made significant progress in solving
problems of object detection and image segmentation. At the same time, classicalcomputer vision methods do not lose their relevance and can also be used for thesetasks. In this research, we conduct a comparative analysis of the effectiveness the
YOLOv8-seg neural network model versions for solving the image segmentation
problem with classical segmentation methods. The obtained results allowed us toformulate some recommendations on the choice of a particular method for object
detection depending on the surrounding environment conditions.

68©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS11-4 Vision-based autonomous navigation for medical examination using a UR3e manipulator
Bulat Abbyasov1, Aidar Zagirov1, Timur Gamberov1, Hongbing Li2, Evgeni Magid1(1Kazan Federal University, Russia) (2Shanghai Jiao Tong University, China)
Medical robotics is an emerging field of robotics within a healthcare sector. This
interdisciplinary field focuses on prototyping, building and developing advanced
robots for numerous clinical application. Modern robotic technologies have atremendous potential for performing medical examination procedures such as
ultrasonography using robotic manipulators that act in an autonomous manner. A
manipulator navigation plays a key role in a safe and efficient exploration of a humanbody. This paper presents a development of a vision-based autonomous navigation
system for a 6-DOF UR3e robotic arm. The developed system is based on a 3D point
cloud and uses MoveIt for path planning. Gazebo was used as a simulation framework
to validate the navigation system.

OS11-5 Robot-Assisted Language Learning: Scientific Data Analysis
Karina Sadyikova, Valeriya Zhukova, Roman Lavrenov (Kazan Federal University, Russia)
Robot-assisted language learning (RALL) is a direction in education that uses robots
in a foreign languages studying process. In this paper, we present results of our research
work devoted to studying trends in a quantity of publications in the field of using social
robots for learning foreign languages. Additionally, we analyzed some published paperin the field of RALL, which allowed us to identify several popular robot models usedin practical research in recent years. Among these models were the NAO, Pepper and
DARWIN OP-2 robots. We have found that these robots are actively used in an
educational process for learning foreign languages. In particular, these models are used
to improve language and communication skills during implicit learning and interactive
games.

OS11-6 Monitoring Beehive Sound Levels with Arduino-based System
Kulmukhametov Ramis1, Ramil Safin1, Tatyana Tsoy1, Kuo-Hsien Hsia2, Evgeni Magid1( Kazan Federal University, Russia) (2National Yunlin University of Science and Technology, Taiwan)
1

An automated beekeeping is a promising approach to addressing various issues
associated with a beekeeping. Among primary problems, a swarming procedure stands
out as a major concern. An uncontrolled swarming can lead to significant financial
losses. During a swarming period a potential for losing a bee swarm is high, therefore
a noise monitoring at this period gains a significant importance. Our long term researchaims at a development of an intelligent monitoring system for beehive conditions,based on a hive-generated noises analysis. This paper presents an experiment thatcollected data about acoustic characteristics of bees’ state using an Arduino
microcontroller and a MAX9814 microphone module. The obtained data analysis isdiscussed.

69©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS12 Machine Learning and its Applications (4)Chair Masato Nagayoshi (Niigata College of Nursing, Japan)
Co-Chair Takashi Kuremoto (Nippon Institute of Technology, Japan)
OS12-1 Restoration of Guqin Music by Deep Learning Methods
Takashi Kuremoto1, Kazuma Fujino1, Hirokazu Takahashi1, Shun Kuremoto2, 3, Mamiko Koshiba*2,
Hiroo Hieda3, Shingo Mabu2
1( Nippon Institute of Technology, Japan) (2Yamaguchi University, Japan)(3Institute for Future Engineering, Japan)
Guqin music played an important role in the history of Asia cultures. The notation of
Guqin are remained more than 600, however, only 100 music are played in nowadays.
The handwriting Guqin notation named Jianzi Pu is hard to be understood, however,
we challenge to restore the Guqin music by deep learning methods and few Jianzi Pu
images. VGG16 and YOLOv5 were adopted in the recognition experiment for Guqin
music restoration. As a result, YOLOv5 realized an online output of Guqin music as
its output of audio or video forms.

OS12-2 Constructive Nurse Scheduling Using Reinforcement Learning Considering Variations
in Nurse Work Patterns
Masato Nagayoshi (Niigata College of Nursing, Japan), Hisashi Tamaki (Kobe University, Japan)
It is very difficult to create a work schedule that satisfies all the different
requirements in nurse scheduling. For this reason, numerous studies have been
conducted on the nurse scheduling problem. However, the created shift schedule isoften not practical as it is, because adjustments including various constraints andevaluation criteria are required. Therefore, we have proposed a work revision method
using reinforcement learning in a constructive nurse scheduling system. In this paper,
we investigate the feasibility of creating a practical work schedule that considers
different evaluations of nurses' work patterns, i.e., nurses' life stages.

OS12-3 A Basic Study on Indicator of Transfer Learning for Reinforcement Learning
Satoshi Sugikawa, Kenta Takeoka, Naoki Kotani (Osaka Institute of Technology, Japan)
Reinforcement learning requires a lot of learning time for the agent to learn.
Transition learning is a method to reduce this learning time, but it has the problem
that the user does not know which knowledge is effective in which environment until
it is learned. Therefore, it is necessary for the user to consider the relationship between
the source and destination when transferring knowledge. Therefore, this study
proposes Indicator of adaptation criteria that can determine this relationship inadvance. In simulations, we demonstrate the usefulness of the proposed method byusing some example problems.

OS12-4 Machine Learning Approach to Predict Cooling Load for Existing Buildings
Makoto Ohara1, Hideo Isozaki2
1( International Professional University of Technology in Osaka, Japan) ( 2Kobe University, Japan)
The objective of this study is to predict air conditioning loads for existing buildings
using operational data, weather forecasts and visitor forecasts. The proposed prediction
method is based on a neural network approach. However, it is important to note that the
proposed method does not learn the entire loads. Loads are divided into factors which
can be predicted by traditional thermodynamics and factors which are subject tomachine learning. The proposed method has been applied to an example instance using
operational data from an underground mall in Kobe, and its validity has been confirmed.
This result could potentially lead to more efficient use of energy in buildings.

70©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS13 Robot Control (10)Chair Yizhun Peng (Tianjin University of Science and Technology, China)
OS13-1 Intelligent Logistics Handling Robot: Design, Control, and Recognition
Yanchao Bi, Jiale Cheng, Limei Wang, Yizhun Peng (Tianjin University of Science and Technology, China)
This study aims to investigate various aspects of intelligent logistics handling robots,
including mechanical design, automatic control, and image recognition. With thecontinuous development of the logistics industry and advancements in automation
technology, intelligent logistics handling robots play a crucial role in improving
logistics efficiency and reducing costs. Leveraging existing technologies, we havedesigned and developed an omnidirectional mobile intelligent logistics handling robot
equipped with a SCARA-type robotic arm. The robot integrates functions such as task
acquisition, global positioning, material detection, warehouse identification, material
handling, and stacking, achieving a fully automated and streamlined logistics handlingprocess.

OS13-2 Greenhouse Design Using Visual Recognition and IoT Technology
Yuntian Xia, Yizhun Peng (Tianjin University of Science and Technology, China)
This device is to solve the traditional pesticide spraying method on the human body has
a greater impact and other issues, through the STM32-based visualisation of theintelligent greenhouse to achieve automatic spraying of pesticides and remotemonitoring and other functions, the establishment of a visualisation of the intelligent
greenhouse monitoring platform. This equipment through the MQTT protocol, not only
through the Internet of Things platform real-time monitoring of crop growth status in
the greenhouse, but also through the platform to determine whether to spray pesticides,
data transmission, so as to use the cross slide to control the position of the nozzle, andthen through the visual recognition algorithms to improve the accuracy of the visual
recognition part of the accuracy of the spraying of plants affected by insect pests, therealization of the digital intelligent greenhouse.

OS13-3 Design of Modular Photovoltaic Environmentally Friendly Portable Stroller
Suqing Duan, Yizhun Peng (Tianjin University of Science and Technology, China)
The work is centered on ROS and integrates speech recognition and natural language
processing modules to enhance the environmental awareness of the stroller and monitor
the baby's condition to ensure a timely and appropriate response. The Raspberry Pi is
the main control unit of the stroller and connects to a cloud-based IoT platform via the
MQTT protocol. The platform facilitates seamless communication between the
Raspberry Pi and cloud data for efficient data visualization on mobile devices. This
innovative solution solves the challenge of balancing parental responsibilities withcareer advancement, promoting healthier and happier babies while allowing parents tomaintain a harmonious work-life balance.

OS13-4 "Teenage Mutant Ninja Turtles" - Design of a Bionic Quadrupedal Rescue Robot
Hongpi Zhao, Yingfan Zhu, Zhihan Zhao, Xin Liang, Lei Lv, Yizhun Peng
(Tianjin University of Science and Technology, China)"Teenage Mutant Ninja Turtles - Bionic Quadrupedal Rescue Robot is a quadrupedal
robot based on the principle of bionics, inspired by the quadrupedal animals in nature.
The robot has rescue and life detection capabilities and can perform rescue operations
at disaster sites. The design of the robot enables it to operate efficiently and stably incomplex environments, and at the same time it has the qualities of bravery andtoughness, which are in line with the image of the Teenage Mutant Ninja Turtles. The
robot can be used in a wide range of application scenarios, such as earthquake, fire andother disasters, to provide more efficient and safer support for rescue work.

71©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS13-5 Design of Grass Lattice Planter for Complex Environment Based on Adaptive
Suspension Technology
Shaokai Tian, Wenqi Fu, Yizhun Peng (Tianjin University of Science and Technology, China)
The Complex environmental grass grid growers based on adaptive suspension
adopted the high-performance stm32 as the master chip, and use the Bluetoothcommunication module, so that users can easily control the vehicle driving, feeding
and the posture of the pressing wheel through the mobile phone application. In addition,
we have introduced new mechanical structures such as adaptive suspension to ensure
effective planting of grass squares in complex terrain such as sloping land. Our vehicle
uses a Mecanum wheel motion system to squeeze seeds by rotating the friction wheels
at high speed.

OS13-6 Design of Intelligent Ecological Multifunctional Plant and Animal Breeding System
Suqing Duan, Yuntian Xia, Siyi Wang, Yizhun Peng (Tianjin University of Science and Technology, China)
The intelligent ecological multi-functional system for animal and plant cultivation
integrates PLC and MCU automatic control technology. This innovative approach
amalgamates aquaculture with horticulture, fostering a symbiotic environment for the
breeding of animals and plants. The system's operations are categorized into four key
components: mechanical structure, environmental perception, automatic control, andintelligent networking. Its comprehensive functionality encompasses plant monitoring,self-checking of temperature and humidity, group control for fertilizer replenishment,
consistent temperature regulation, full-spectrum illumination, advanced filtration,
fogging supplementation, versatile water management, fertilizer blending, and water
purification through sterilization.

OS13-7 Design of a Fully Automated Logistics Handling Platform
Hongpi Zhao, Jianfeng Qin, Yizhun Peng (Tianjin University of Science and Technology, China)
With the progress of science and technology and the development of society, the world's
countries on industrial production efficiency, intelligent manufacturing transformation
continues to grow. Therefore, we designed a fully automatic logistics handling
platform, which is a multi-modular device based on mechanical design, microcontroller
control, visual positioning, the device can realize autonomous identification,
autonomous transportation, improve production efficiency, the device has a wide range
of applications, can be applied to food packaging, parts processing, intelligent
manufacturing and other automation scenarios.

OS13-8 Design and Application of AI-based Brush Calligraphy and Painting Robot
Haibo Li, Yizhe Sun, Shuxin Wang, Yizhun Peng (Tianjin University of Science and Technology, China)
With the rapid development of digital technology and artificial intelligence, theinnovation and exploration of traditional painting forms in the field of digital art havebecome increasingly captivating. The main focus of this research is to create a brushcalligraphy and painting robot system based on PLT files. It digitizes user creative
instructions to autonomously generate brush calligraphy and landscape paintings. This
technology not only advances the deep integration of digital art and traditional culture
but also opens up new perspectives and vast possibilities for artistic creation.

72©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS13-9 Intelligent Wheelchair System: Non-contact Heart Rate and Body Temperature
Measurement
Dongpo Ma, Junsheng Zhang, Yizhun Peng (Tianjin University of Science and Technology, China)
With the continuous development of society and human civilization, people, especiallythe disabled and the elderly, need more and more to use modern high technology to
improve their lives and improve the comfort of life. Therefore, this paper introduces asingle-chip microcomputer-based non-contact body temperature and facialrecognition-based heart rate measurement method and applies it to the intelligent
wheelchair system. Monitor the vital signs of the elderly, and give early warnings intime when abnormal values occur, so that family members or caregivers can rescue
them in time.

OS13-10 Recognition and Localization Method for Automotive Axle Holes in Assembly Robots
Junsheng Zhang, Dongpo Ma, Yizhun Peng (Tianjin University of Science and Technology, China)
During the production of automotive axle holes, the roundness error at the pipe opening
leads to low detection efficiency due to manual measurements, rendering real-timeinspection unfeasible. This paper proposes a method for detecting and sorting theroundness of automotive axle holes based on visual inspection. Ensuring accuracy ingrasping, it establishes the kinematic model of the robot Aubo_I5. Targeting
automotive axle holes for grasping, it employs adaptive threshold segmentation tohighlight the section features of the axle hole. The Canny algorithm is then used to
extract edge information, and finally, the least squares method is utilized to detectroundness errors for sorting the axle holes based on this error.

OS14 Robotic Manipulation (3)Chair Kensuke Harada (Osaka University, Japan)
Co-Chair Akira Nakamura (Saitama Institute of Technology, Japan)
Co-Chair Tokuo Tsuji (Kanazawa University, Japan)
OS14-1 Evaluation Standard of Error Recovery Planning Focused on Revival Process from
Failures in Robotic Manufacturing Plants
Akira Nakamura1, Kensuke Harada2 (1Saitama Institute of Technology, Japan) (2Osaka University, Japan)
In recent years, many intelligent robots have been used in various fields. In many cases,
these robots need to be able to perform not only repetitive tasks but also non-routine
tasks. As work errors are more likely to occur in such situations, we have proposedseveral methods for error recovery. Our method uses both forward recovery andbackward recovery to restore work. The former is a recovery method that is often used
for minor modifications, and the latter is the recovery from relatively large failures that
we mainly use. This paper proposes a method for selecting the best path from several
possible recovery paths using a new evaluation method.

OS14-2 Robotic Food Handling Utilizing Temperature Dependent Variable-Stiffness Material
Rozilyn Marco1, Prashant Kumar2, Xinyi Zhang2, Weiwei Wan2, Kensuke Harada2(1 University of Toronto, Canada) (2 Osaka University, Japan)
This paper presents a robotic gripper that addresses challenges of automated robotic foodhandling. Depending on the food material, the robotic gripper should adjust its finger
compliance. To cope with this problem, we apply the approach on using a variable
stiffness fabric on the finger surfaces of the gripper, where the finger stiffness changes
depending on its temperature. This gripper design was validated empirically throughforce and object-grasping experiments. By using the gripper, hard objects can be grasped
with the hard mode of the finger while a fragile object can be safely grasped with thesoft mode.

73©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS14-3 Vegetable maturity evaluation for harvest robots
Reno Muhammad Fadilla, Tokuo Tsuji, Tatsuhiro Hiramitsu, Hiroaki Seki (Kanazawa University, Japan)
We propose a method for detecting vegetables in an image and classifing their maturity
by color. Vegetable images are classified according to their maturity and used as training
data. We represent the images in different color systems and investigate the accuracy ofthe classification of maturity. The proposed method can be applied to harvesting robots.

OS15 Artificial Intelligence for Embedded Systems and Robotics (6)Chair Hakaru Tamukoh (Kyushu Institute of Technology, Japan)
Co-Chair Yuma Yoshimoto (National Institute of Technology, Kitakyushu College, Japan)
OS15-1 YOLO real-time object detection on EV3-Robot using FPGA hardware Accelerator
Dinda Pramanta1, Ninnart Fuengfusin2, Arie Rachmad Syulistyo2, Hakaru Tamukoh2( Kyushu Institute of Information Sciences, Japan) (2Kyushu Institute of Technology, Japan)
1

The growing demand for robots necessitates faster and more precise processing.
However, running large Artificial Intelligence (AI) models from cloud data centers tomobile robots via inference models uses considerable computation resources, whichleads to power limitations, particularly for mobile robots. The use of reconfigurable
semiconductor devices at the hardware level is a promising solution to this problem. We
introduce the educational kit EV3-Robot with a co-design methodology utilizing Fieldprogrammable Gate Arrays (FPGA) Kria KV260 as a hardware accelerator specificallyfor object detection. We apply the You Only Look Once (YOLO) model for object
detection, which provides real-time results for practical applications. Additionally, we
analyze the processing times of the local PC and EV3-Robot.

OS15-2 A Low Computational Cost Hand Waving Action Recognition System with Echo State
Network for Home Service Robots
Hiromasa Yamaguchi, Akinobu Mizutani, Arie Rachmad Syulistyo, Yuichiro Tanaka, Hakaru Tamukoh(Kyushu Institute of Technology, Japan)
This study proposes a low computational cost hand-waving action recognition system
for non-verbal communication in home service robots. The system is based on an echo
state network, which requires lower computational costs than that of deep neuralnetworks (DNNs), and processes time-series data of skeletal coordinates of humans torecognize hand-waving actions. Additionally, this study proposes and compares two
types of preprocessing methods of the skeletal coordinates to ensure the robustness ofthe human positions on the screen: one method extracts elbow and arm angles, which
are invariable regardless of the humans’ positions and the other normalizes the skeletal
coordinates. The experimental result shows that the proposed system has competitive
accuracy and is faster than DNN-based methods and robust to varying humanpositions.

pred: waving

OS15-3 A Rapidly Adjustable Object Recognition System through Language Based Prompt
Engineering
Naoki Yamaguchi, Tomoya Shiba, Kosei Isomoto, Hakaru Tamukoh (Kyushu Institute of Technology, Japan)
We propose the use of language-based prompt engineering to achieve rapidly
adjustable object recognition in RoboCup@Home. The proposed prompt engineering
involves humans adding features, such as the color and material of an object, into thetext prompts inputted into Language Segment Anything. In this research, we evaluatedthe effectiveness of our proposed method in three benchmark tests for object
recognition at RoboCup@Home held in France in 2023. The results showed that thehighest scores were obtained in certain tasks, indicating that it could be applied to arange of recognition tasks.

74©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS15-4 Development of a SayCan-based task planning system capable of handling abstract
nouns
Kosei Yamao1, Daiju Kanaoka, Kosei Isomoto, Akinobu Mizutani, Yuichiro Tanaka, Hakaru Tamukoh(Kyushu Institute of Technology, Japan)
The task planning system is required to accomplish various requests from a humanin real-world environments. SayCan, one of the task planning systems, has highaccuracy. However, its accuracy decreases for requests that include abstract nouns of
the ambiguous word/phrase, and the inference time increases as the number of skills
increases. We propose a novel task planning system based on SayCan that introduces
a function for listening back to abstract nouns and a rule-based skill extraction,enhancing accuracy and reducing inference time. The proposed system facilitates theinterpretation of requests and enables appropriate task planning with low inference
time. The effectiveness of the proposed system was demonstrated at
RoboCup@Home, where it achieved high performance.

OS15-5 RoboCup@Home 2023: Stickler for the Rules Task Solutions
Tomoya Shiba, Hakaru Tamukoh (Kyushu Institute of Technology, Japan)
This paper proposes an approach where home service robots use only two recognition
models that can be capable of prompt tuning to detect rule violations in the home.
The robots are increasingly required to perform more advanced and complex
perceptions beyond their traditional roles such as cleaning. Advanced tasks such asserving and security, for example, usually require multiple AI systems, including
person detection, object detection and more. Our approach simplifies this byleveraging the combined capabilities of Grounding DINO and SAM to detect ruleviolations effectively. The success of our method was proven at RoboCup@Home
2023, where it secured the highest score among all participating teams.

drink

shoes

OS15-6 Offloading Intellectual Processing from Home Service Robots to Edge Devices
Yuma Yoshimoto, Mizuki Kawashima, Shun Yonehara(National Institute of Technology, Kitakyushu College, Japan)
In this study, we focus on extending the operational time of home service robots byoffloading intellectual processing to circuit devices such as Field Programmable Gate
Arrays (FPGAs), which in turn reduces power consumption. The core of ourapproach involves developing a method for implementing intellectual processing on
FPGAs, coupled with a dynamic circuit reconfiguration technique. This enables the
FPGA to adaptively respond to frequent task changes. We present: (a) methods fortransitioning circuits from a robot's control computer to FPGA in response to varying
tasks, and (b) an evaluation of the effectiveness of using FPGA to extend operationaltime under these rapidly changing task conditions.

System on a Chip (SoC)

Main Computer

PS/CPU
State Machine⑦① ②

SoC Manager

⑥

③
④

FPGA
Manager

PL/FPGAIntellectual
Processing⑤

OS16 Industrial Artificial Intelligence Robotics (4)Chair Eiji Hayashi (Kyushu Institute of Technology, Japan)
OS16-1 A Research on Performance Information Editing Support System for Automatic Piano
Yoshiki Hori, Eiji Hayashi (Kyushu Institute of Technology, Japan)
In 1996, Hayashi et al. from our laboratory developed an automatic piano playing
device that requires data with nuances for each note to perform in a human-like piano
playing. However, this device lacks the function to infer such nuanced data.
Therefore, prior research focused on developing a system to infer data with nuancesfor each note. Yet, this system required manual inference, consuming a significant
amount of time. In this study, we have constructed a system capable of automatically
performing inference using deep learning. This system not only improves theaccuracy of piano playing inference but also contributes to the efficiency of theinference process.
75©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS16-2 Development of Autonomous Mobile Field Robots – Accuracy Verification of SelfLocalization through Simulation Takamasa Hayashi1, Shintaro Ogawa1, Yuto Okawachi1, Tan Chi Jie1, Janthori Titan1, Ayumu Tominaga2,
Eiji Hayashi1, Satoko Seino3
1( Kyushu Institute of Technology, Japan) (2National Institute of Technology (Kitakyushu College), Japan),(3Kyushu University, Japan)
In recent years, the increase in marine debris has become a significant challenge interms of its collection. Costal debris, a type of marine debris, can be collected by humanhands, but the variety in shapes, types, and sizes presents limitations to human-onlycollection efforts. To address this, I focused on developing an autonomous mobile robot,
establishing a simulation environment was considered crucial for facilitating smooth
progress. This paper focuses on self-localization, an essential aspect for autonomous
movement. We replicated an actual coastal cleaning site within the simulation
environment and evaluated the accuracy of self-localization using an EKF (Extended
Kalman Filter) with multiple sensors.

OS16-3 Development of AR System for Grasping String Foods on Introduction of Industrial
Robot
Yoshihiro Koyama, Eiji Hayashi (Kyushu Institute of Technology, Japan), Akira Kawaguchi (The City
College of New York of The City University of New York, United States of America)
In recent years, the food service industry has been facing a labor shortage. However,
the introduction of industrial robots is not easy due to the high cost of equipment andsystem integration. Therefore, we are developing an Augmented Reality (AR)application for the purpose of introducing robots to small and medium-sized companies.
By using this application to perform tasks necessary for introducing robots, such asteaching, cost reductions can be expected when introducing industrial robots. In aprevious study, an AR-based grasping and serving simulation system was developed forsolidified foods such as fried chicken and rice balls. In this study, we focused on stringshaped food items such as spaghetti, and attempted to develop an AR system forgrasping and serving a string-shaped object by controlling a gripper.

OS16-4 An Image Analysis of Coastal Debris Detection -Detection of microplastics using deeplearning
Yuto Okawachi1, Ayumu Tominaga2, Shintaro Ogawa1, Takamasa Hayashi1, Tan Chi Jie1, Janthori Titan1,
Eiji Hayashi1, Satoko Seino3
1( Kyushu Institute of Technology, Japan) (2National Institute of Technology (Kitakyushu College), Japan)(3Kyushu University, Japan)
To address the issue of litter drifting ashore, this study developed a deep learning-based
microplastic detection system. The system employed yolov7 as its deep learning
network, complemented by SAHI (Slicing Aided Hyper Inference) as an additional
vision library. yolov7 is renowned for its efficacy in real-time object detection. Our
experimental framework involved four tests, utilizing two variations of yolov7 - thestandard model and yolov7-e6e - in conjunction with SAHI. The effectiveness of eachtest was quantified using metrics such as Intersection over Union (IoU), Precision,
Recall, F-measure, and Detection Time in seconds. For our dataset, we gathered images
from actual cleanup locations, such as Hokuto Mizukumi Park. The model's
discriminator underwent 700 training iterations, with a learning rate set at 0.001.

76©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS17 Electronics and Kansei Engineering Based on ETT theory (5)Chair: Tetsuo Hattori (Kagawa University, Hiroshima Institute of Technology, Japan)
Co-Chair: Yusuke Kawakami (NIT (Kagawa College), Japan)
OS17-1 A High-Speed Estimation Method of Parameters in Impulse Response
Toshiki Tanaka, Ivan Tanev (Doshisha University, Japan), Tetsuo Hattori (Kagawa University, Japan)
This paper proposes a high-speed parameters estimation method for compartment
model where output function is described by the convolution between input function
and impulse response, which is like a time-invariant linear system. The proposed
method uses linear regression analysis based on equations that can be obtained fromthe differentiation of convolution (DOC). In this paper, taking the parameters
estimation problem of PET (Positron Emission Tomography) inspection system and
RLC series electrical circuit for examples, we show that the method can estimateparameters in those impulse responses.

OS17-2 A Consideration on Amplification Function in BJT Evers-Moll Model and PTT (I)---- V-I Characteristics ---Shimon Hattori, Osamu Matoba (Kobe University, Japan), Tetsuo Hattori (Kagawa University, Japan),Toshiki Tanaka (Kinkei System, Japan), Yusuke Kawakami (NIT (Kagawa College), Japan)
In 1989, the notion of PTT (Photon Transport Transistor) has been proposed by B.J.
Van Zeghbroeck et al., at IBM Research Center at that time. PTT is an optical coupling
device of light emitting diode (LED) or laser diode and light receiving diode (Photo
Diode, PD) where the carrier of the base layer is light (Photon) only. Later in 1996, W.
N. Cheung and Paul J. Edwards have shown that PTT can be a very low noise amplifier
in a positive feedback circuit, based on theoretical calculations of the noise figure. In
this paper we consider the amplification principle showing the VI characteristics of the
PTT and bipolar junction transistor (BJT).

OS17-3 A Consideration on Amplification Function in BJT Evers-Moll Model and PTT (II)---- H Parameters in the Small Signal Amplifier Circuit---Shimon Hattori, Osamu Matoba (Kobe University, Japan), Tetsuo Hattori (Kagawa University, Japan),Toshiki Tanaka (Kinkei System, Japan), Yusuke Kawakami (NIT (Kagawa College), Japan)
The notion of PTT (Photon Transport Transistor) has been proposed in 1989 as an
optical coupling device of light emitting diode (LED) or laser diode and light receiving
diode (Photo Diode, PD), where the carrier of the base layer is light (Photon) only. In
this paper we consider the amplification principle of the PTT and BJT (bipolar junction
transistor), by illustrating those h parameters of equivalent small signal amplifier
circuit. Moreover, this paper also discusses the similarities and differences between
PTT and BJT, based on the derived h parameters.

OS17-4 Color Image Arrangement Using Histogram Matching
Yusuke Kawakami (NIT (Kagawa College), Japan), Tetsuo Hattori (Kagawa University, Japan),
R.P.C. Janaka Rajapakse (Tainan National University of the Arts, Taiwan (R.O.C.))
For the arrangement processing of image using its histogram, we previously havepresented a Histogram Matching method based on Gaussian Distribution (HMGD).
However, in the case where the brightness histogram of input image has multiple peaks,
the HMGD processing does not always bring good results. In this paper, we present an
improved histogram matching method using the reference histogram that is made byappropriate moving average processing over the histogram of input image. Also in thispaper, we show the experimental results.

77©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS17-5 Methodology for Creativity Oriented STEM Education Based on ETT Theory
Tetsuo Hattori, Toshihiro Hayashi, Mai Hattori, Yoshiro Imai (Kagawa University, Japan),Asako Ohno (Osaka Sangyo University, Japan), Takeshi Tanaka (Hiroshima Institute of Technology, Japan)
Recently, the necessity of integrated and comprehensive methodology for STEM
(Science, Technology, Engineering and Mathematics) education is growing. In thispaper, we propose an educational methodology utilizing the viewpoint of EquivalentTransformation Thinking (ETT) theory which has been proposed by Dr. Kikuya
Ichikawa in 1955 as a principle of creativity. Especially, we show that the viewpoint isvery useful not only for new technology invention but also for STEM Education in thesense that it deepens insights of the contents to be learned, motivates students to study
further, and inspires their creativity.

OS18 Computer and Information Engineering (12)Chair Norrima Mokhtar (University of Malaya, Malaysia)
Co-Chair Heshalini Rajagopal (UCSI University, Malaysia)
OS18-1 Efficient Campus Shuttle Tracking and Management Mobile Application for College
CampusAndrea Tantay Gonzales1, Kavitha Thamadharan1, Neesha Jothi2( INTI International College Penang, Malaysia) (2UCSI University, Malaysia)
1

Shuttle Stalk, a Real-Time Campus Shuttle Booking and Tracking System was a
proposed solution which enables the students to register for the shuttle service throughthe mobile application, reducing the administrative burden and to track its current
location in real-time, allowing them to plan their journeys more effectively. The
application developed serves as a comprehensive solution for both students andadministrators, providing real-time shuttle tracking and streamlined management
capabilities.

OS18-2 GCN Analysis of Task-Based fMRI Data for Diagnosis of Schizophrenia
Tejaswini Thota1, Reuben Stephen John1, Dr R Dhanush1, Dr Amutha S1(1Vellore Institute of Technology, India)
This study focuses on schizophrenia, characterized by distorted reality and delusions.
We propose utilizing a Graph Convolutional Network (GCN) model on a task-based
fMRI data to differentiate schizophrenia instances, with and without auditory
hallucinations, utilizing healthy participants for comparison. Various node embedding
algorithms are compared to extract structural properties, and statistical features fordifferentiation. The study focuses on identifying specific auditory stimuli thatsignificantly differentiate individuals. Model validation deploys k-fold stratified
crossvalidation, with evaluation metrics, including accuracy, precision, recall, F1
macro, and MCC guide the assessment of the GCN model.

OS18-3 AR-Based Application for Campus Navigation
Renuka Devi Rajagopal1, Akshay S1, Manoj Rathinam1, Shakthi B1, Heshalini Rajagopal2(1Vellore Institute of Technology, India) (2UCSI University, Malaysia)
The "Campus Navigator" is an innovative augmented reality (AR) application designed
for seamless campus navigation. It combines geolocation technology, QR code
recognition, and predefined geospatial anchors to transition users from outdoor to indoor
spaces. Augmented reality overlays provide real-time wayfinding guidance, eliminating
the need for traditional maps. The user-centric design and user feedback integration
enhance accessibility and user satisfaction. Real-world testing demonstrates promising
performance, accuracy, and user engagement. The Campus Navigator transforms campus
navigation, with potential for broader applications.

78©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS18-4 Development of Robotic Assistant for Health Care Sector with A Special Focus to Aid
the Geriartric Patients
Narayanan Ganesh (Vellore Institute of Technology, India)
Human manual efforts in their day-to-day tasks are reduced by robotic helpers. This
paper analyzes the robot created to assist the aged people. The created robot can becontrolled by voice commands with its own inbuilt microphone to pick up human speechorders. This robot can do a variety of movements, turns, grab operations, move an object
from one location to another. Personal assistant robot is built using Microcontrollers. The
results reveal that the developed robot will be one of the best companions for the geriatricpatients and will be an alternate solution will make a new mark in the health care sector.

OS18-5 Development of a Desktop Application Restaurant Management System
Gabriela Maria Ancilla, Heshalini Rajagopa, Ismail Ahmed Al-Qasem Al-Hadi (UCSI University, Malaysia)
The number of restaurants has grown rapidly around the world, the awareness ofmanaging it efficiently has increased. This necessity yields the idea to invent a restaurant
management system (RMS). This RMS will bring several benefits such as greater
management of a restaurant, reduced resources cost, raise profit, and time-saving by
allowing the admin to manage the restaurant easily through the functionality provided in
the system. The provided features include adding, updating, and deleting information,
stock calculation and auto re-order stock items, ingredients management, and financecalculation.

OS18-6 Face Recognition based on Attendance System
Koh Pei Cong 1, Heshalini Rajagopal1, Ghassan Saleh1, Norrima Mokthar2(1UCSI University, Malaysia) (2University of Malaya, Malaysia)
The use of face recognition technology for attendance tracking has grown popular inrecent years. Hence, the main goal of this project is to produce an accurate, fast, and robust
face recognition based on an attendance system. The system detects the user’s uniquefeatures, understand the identity of the user through face recognition technology, and thus
records the attendance from the face recognition dataset to the user that matches the user,
in an attempt to help the user automatically check in with real-time attendance date. The
system includes features such as face detection, face recognition, distance estimation, andattendance recording.

OS18-7 U-Reserve: Development of a Facility Reservation System for UCSI University
Esther Chong Jun Lynn1, Neesha Jothi1, Ismail Ahmed1 (1UCSI University, Malaysia)
In today's technology-driven era, there is a growing emphasis on efficient and convenientmethods to enhance user satisfaction. UCSI University, Kuala Lumpur, currently employs
a manual facility reservation system that requires in-person or email submissions. Thisoutdated approach often results in double booking and underutilization of resources.
Therefore, this project aims to develop a web-based facility reservation system named UReserve. This system is developed by HTML, CSS, JS, JSP, Servlet, and MySQL,
focusing on colour tone and font type. These considerations are vital in providing an
improved user experience. With U-Reserve, not only the reservation requests andcancellations being streamlined, but informed decisions regarding facility management
can also be made by a data-driven dashboard.

79©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS18-8 The Smart Document Processing with Artificial Intelligence
Raenu Kolandaisamy1, Heshalini Rajagopal1, Indraah K2, Glaret Shirley Sinnappan3(1UCSI University, Malaysia) (2University Utara Malaysia, Malaysia)(3Tunku Abdul Rahman University, Malaysia)
This study focuses on the challenges and potential for Intelligent Document Processing(IDP) with Artificial Intelligence (AI) to manage unstructured data. A large amount of data
in many different forms, such as information from the IoT, cybersecurity and more areproduced during this modern Digital Age which has been widely distributed through awide range of unformatted formats. IDP utilizes AI technologies like Machine Learning(ML), Natural Language Processing (NLP), and Computer Vision (CV), with the aim ofconverting unstructured data into structured, usable information. The IDP, are exploredin the findings and discussion section that emphasizes its capability to automatically
perform redundant tasks, reduce operating costs as well as improve employee
productivity.

OS18-9 Digital Security Challenges Faced by Business Organizations
Raenu Kolandaisamy1, Heshalini Rajagopal1, Indraah K2, Glaret Shirley Sinnappan3(1UCSI University, Malaysia) (2University Utara Malaysia, Malaysia)(3Tunku Abdul Rahman University, Malaysia)
Over the decades, the forms of cyber-attacks have evolved from disruption level,
cybercriminal, followed by cyber espionage and lastly threatening level. Digital security
has played a significant role in protecting enterprises from any form of cyber-attacks,
especially in today’s era of digitalization. Aligning with the global effort in emerging theconcept of Industrial Revolution 4.0 (IR4.0) in organizations, where sensitive data andconfidential information can be accessed at a fingertip of an employee. This paper
discusses the difficulties of implementing digital security solutions in an enterprise interms of external potential cyber threats, internal cyber security roadblocks within theorganization and how Covid-19 pandemic imposed challenges towards cyber security inthe organizations.

OS18-10 The Study on Perception on E-Waste Among the People
Raenu Kolandaisamy1, Heshalini Rajagopal1, Indraah K2(1UCSI University, Malaysia) (2Utara Malaysia, Malaysia)
E-waste is piling up at an alarming rate, which results in our environment's pollution, while
only a few are being disposed of correctly. This research aims to discover people's
awareness of e-waste and how to handle it. An online survey was done among 75
respondents, who were asked about their general knowledge of e-waste. Most respondents
are aware of the effects of e-waste but lack knowledge on how to tackle them, including
questions concerning e-waste management and its causes. In conclusion, the general publicmust be educated on e-waste to reduce and dispose of e-waste efficiently and effectively.

80©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS18-11 Emergence of Cybercrimes in Online Social Networks
Raenu Kolandaisamy1, Heshalini Rajagopal1, Indraah K2( UCSI University, Malaysia) (2University Utara Malaysia, Malaysia)
1

The rise of social networking websites has been seen in recent years. Everyone will bespending most of their time on social networking websites such as Facebook, Instagramand Whatsapp. The great advantage that this social networking website offers benefited
many of the users. It can help people to promote themselves or their business to gain more
popularity and also customers through these social networking websites. There are manycybercrimes that can be identified which are identity theft, hacking, fraud and so on. The
emergence of cybercrimes has created an awareness so that the users will know what thecommon attacks are and how they can be prevented from being lure and being a victim of
this attack. This research will discuss about the attacks and how these attacks can beprevented by the users.

OS18-12 Development of a music recommendation application by using facial emotion
recognition
Shengke Xie, Raenu Kolandaisamy, Ghassan Saleh, Heshalini Rajagopal (UCSI University, Malaysia)
Music is an important part of human life and culture, and it can affect people’ s emotions
and moods. However, choosing music from a large library can be a challenging and timeconsuming task. In this paper, we propose a facial expression recognition-based music
recommendation system that can recommend suitable music which matches the user’ scurrent mood. The system uses a camera to capture the user’s face and a convolutional
neural network model which trained facial emotion recognition database to recognize
seven basic emotions: anger, disgust, fear, happiness, sadness, surprise and neutral. The
paper contributes to the research of facial emotion recognition and music recommendationand provides a convenient way for people to enjoy music.

OS19 Natural Computing (3)Chair Marion Oswald (Vienna University of Technology, Austria)
Co-Chair Yasuhiro Suzuki (Nagoya University, Japan)
OS19 -1 A Model of Reaction-diffusion phenomena with Multiset Processing
Yasuhiro Suzuki (Nagoya University Japan)
We propose a model of reaction-diffusion phenomena using Abstract Rewriting System on
Multisets ARMS, which is a model of Multiset Processing. Although proposed model issimple, computer simulations confirm that the Turing pattern is generated.

OS19-2 Extract tactile qualities from time series data
Yasuhiro Suzuki (Nagoya University, Japan)
We proposed the Tactile Score to describe time-varying tactile sensation by the time
variation of vertical force. Tactile quality is essential in hardness/softness, roughness, andtemperature. Hardness and softness can be extracted from the shape of the Tactile Score.
Roughness can be extracted from the pattern of hardness and softness. The arbitrary timeseries data can be interpreted as a tactile score by considering the time variation of thevertical force, and the hardness and softness are extracted from the time series datainterpreted as the tactile score. This method can extract different features fromconventional data science methods.

81©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS19-3 Healthcare applications of vibrotactile stimulation developed by Tactile Score
Yasuhiro Suzuki, Rie Taniguchi (Nagoya University, Japan)
We constructed a system to convert the tactile score into vibratory tactile sensation. We
conduct basic biological experiments on electronic tactile stimuli and apply the results tomedical and healthcare applications. We use two types of electronic tactile stimuli. One isdirect vibrating tactile stimulation, and the other is airborne vibrating tactile stimulation.
Verification experiments of this system have confirmed cosmetic effects such as skincollagen aggregation, improvement of hypertension and diabetes, and reduction of pain
and nerve paralysis. In addition, a clinical study was conducted in which patients withdementia were exposed to vibrotactile sensation (low-frequency sound) via air vibration
and an improvement in cognitive function was confirmed.

OS20 Advances in Field Robotics and Their Applications (15)Chair Kazuo Ishii (Kyushu Institute of Technology)
Co-Chair Keisuke Watanabe (Tokai University)
OS20-1 Image Collection Experiments of a Handy AUV for Offshore Structure Inspection
Keisuke Watanabe, Koki Amano, Shingen Urano, Yasutaka Taniguchi, Konosuke Watanabe(Tokai University, Japan)
In recent years, in Japan, with the decline in the working population, there has been a
noticeable shortage of labor at construction sites and offshore operations. On the other
hand, with the aim of realizing a GX society, it is expected that offshore wind power
generation platforms will rapidly increase and be deployed offshore. Since it is not easy
to access offshore platforms, it is necessary to automate equipment inspection work, and
it is expected that the use of AUVs will be particularly desirable for underwaterinspection work. The most basic inspection work is assumed to be to collect images ofunderwater structures. Therefore, in this study, we created an AUV and conducted animage collection experiment using a pier owned by our university as an example toconfirm its functionality.

OS20-2 Design of Disassembly-reassembly Type USV for Coral Reef Research
Keisuke Watanabe, Koki Amano, Gaku Minato, Yasutaka Taniguchi, Konosuke Watanabe(Tokai University, Japan)
Researchers are conducting physical surveys by diving and swimming to study theeffects of microplastics and global warming on coral reefs. The area, time, and water
depth that can be investigated by diving are extremely limited. In addition, to collectmicroplastics, it is necessary to tow the nets, but chartering a ship is expensive.
Therefore, the authors are developing a system that simultaneously operates a USV anda UUV to simultaneously observe the sea surface and underwater. In this paper, we
conducted a conceptual design, carried out fluid force measurements, trajectory
tracking experiments, and image recognition using AI, with the aim of realizing alightweight USV that can be divided during transportation to coral reef areas andreassembled on site for operation.

OS20-3 Optimization method to improve visual SLAM in dynamic environment
Yufei Liu, Kazuo Ishii (Kyushu Institute of Technology, Japan)
Aiming at the problem that the robustness of the classic visual SLAM system isgreatly affected by dynamic feature points in the environment, a method of eliminating
dynamic feature points using a target detection algorithm is proposed. Use the target
detection algorithm YOLOv5 to identify and classify the collected feature points. Thenremove the extracted dynamic feature points, use the remaining static feature points
for map construction and positioning, and finally test on the TUM data set. The results
show that this algorithm effectively improves the positioning accuracy and robustnessof the visual SLAM system.
82©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS20-4 Design of flexible mechanism for flexible manipulator
Huang Jiawei, Kazuo Ishii (Kyushu Institute of Technology, Japan)
Based on the study of the kinematic limitations of the rigid manipulator structure andthe characteristics of the existing flexible manipulator in different categories, this paper
proposes a structure assumption of flexible manipulator based on the advantages ofhigh load of rigid joint and high flexibility of flexible manipulator. This paper designed
a flexible manipulator structure driven by wires. Using the forward kinematics analysis
the sport model of the structure. The motion range of the end of the manipulator arm
and the length changes of the wires were simulated with the movement of the model.

OS20-5 Driver Drowsiness Detection Method based on Deep Learning
Shi Puwei, Kazuo Ishii (Kyushu Institute of Technology, Japan)
Drowsiness driving will pose a serious threat to the lives of drivers and others.
Determining the state of the driver through face recognition has the advantages of low
cost and convenience. Therefore, this study deploys the face recognition model to themobile phone, and finally realizes the recognition of the driver's Drowsiness by thefront camera of the mobile phone. The whole research is divided into three parts. The
first part is to train the face 68-keypoints recognition model based on the YOLO facealgorithm. The second part is to deploy the trained model to the mobile phone using
ONNX and NCNN. The third part is to calculate EAR and MAR using several facial
key points, and finally complete the recognition of the driver's drowsiness state using
EAR and MAR.

OS20-6 An Analysis of Translational Motion for a Mobile Robot with Line-Symmetric Rollers
Arrangement
Kenji Kimura1, Kazuo Ishii2( National Institute of Technology, Matsue College, Japan) (2Kyushu Institute of Technology, Japan)
1

In fields such as logistics, robots are required to have efficient mobility. There arevarious types of rollers used in mobile robots. Among them, omni-rollers haveexcellent omnidirectional mobility and are easy to control. In this study, mechanism
kinematics has been proposed that assumes arbitrary changes in the roller arrangement
position on a circular mechanism and the roller arrangement has been evaluated fromthe viewpoint of speed efficiency. Furthermore, we aim to evaluate the mobility ofmobile robots by focusing on their translational components. Moreover, we haveexamined the behavior of the area of the area generated by the end point of the robot
velocity vector and evaluated the velocity efficiency.

OS20-7 Development of Teaching Materials for Robot Programming for Junior High SchoolStudents:Student-Based Educational Activities
Kenji Kimura, Youta Takano (National Institute of Technology, Matsue College, Japan)
With the extensive use of robots in recent years, an age of co-existence between
humans and robots is expected to arrive in the future. Therefore, providing robot
education in the early stages of elementary and junior high school is necessary tostimulate students’ interest in robots. Furthermore, educational institutions are
becoming more active in robot education as a part of their contribution to the local
community. For elementary school students, a Beauto Racer (Vstone) has been
adopted as a teaching material for robot education, and teaching materials have beendeveloped for teaching line tracing. In this study, we will develop educational
materials for junior high school students on maze exploration using a Beauto Rover(Vstone). The students of NIT, Matsue College took the initiative in this activity.
The purpose is to improve their teaching skills.

OS20-8 Design and Software Production of Robotics Educational Design for Elementary andJunior High School Student
83©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

Youta Takano, Kenji Kimura (National Institute of Technology, Matsue College, Japan)
The Ministry of Education, Culture, Sports, Science and Technology (MEXT) has beenpromoting cross-curricular learning including STEAM education in its educational
policy for 2021. The purpose of this study is to have students experience not only
control engineering but also basic programming techniques, and to have them become
interested in mathematical subjects in general, which are the basis for control
engineering. The educational design and accompanying software were designed using
Beauto Balancer2 (Viston) educational robot and Scratch, a visual programming editor
developed by the Scratch Foundation. In addition, workshops were conducted and thelesson design was evaluated by a questionnaire.

OS20-9 Development of visual inspection system for low-reflective material utilizing of a string
shadow
Keiji Kamei1, Tomorou Kawahara1, Yoshiyuki Daimaru2( Nishinippon Institute of Technology, Japan) (2Nissan Motor Kyushu, Japan)
1

Visual inspection of products in manufacturing plant is usually conducted by workers.
However, that inspection process by workers occasionally happen oversight due tocareless check, so that automatic visual inspection system is needed frommanufacturer. Products made of low-reflective material are difficult to inspect by lightreflection because contours of defect areas are got blurred. To solve this problem, we
propose to inspect products utilizing of a string shadow. Form our research, a string
shadow was different between contours of defect area and its of non-defect. On theother hand, former research did not success to detect defect area due to determination
algorithm for defect. In this paper, we apply linear approximation of a string shadow
to defect detection. From the experimental results, we succeed in detecting defects.

OS20-10 Development of IoT-Based Remote Monitoring Module for Greenhouse Environment
to Facilitate Crop Growth Data Analysis
Moeko Tominaga, Yasunori Takemura, Junya Era, Wataru Kaishita(Nishinippon Institute of Technology, Japan)
Eiji Mizoe, Tomoyasu Furukawa (Kumamoto Fruit and Vegetable Shippers Association Co. Ltd., Japan)
As the shortage of agricultural workforce becomes a concern, IoT proves effective inalleviating labor burdens. Monitoring, storing, and analyzing the growing
environment, especially for new crops, is crucial for subsequent agricultural cycles.
However, small farms often face challenges in data collection due to cost constraints,
particularly with new crops. An agricultural IoT system was developed for small
farmers, featuring a module for collecting, accumulating, and visualizing time-series
data from sensors. To demonstrate feasibility, the IoT system was implemented andvalidated by intermittently monitoring banana and coffee growth parameters in a
Kumamoto greenhouse over two years. Results confirmed that the proposed IoTremote monitoring module empowers farmers to monitor crop growth environments.

84©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS20-11 Development greenhouse environment prediction system using IoT data
Yasunori Takemura, Naoya Nishida, Moeko Tominga (Nishinippon Institute of Technology, Japan)
Eiji Mizoe, Tomoyasu Furukawa (Kumamoto Fruit and Vegetable Shippers Association Co. Ltd., Japan)
Currently, the decrease in the working population is a significant problem in agriculture
in Japan. In addition, the prices of imported vegetables and fruits have been rising due
to international trade problems such as the weak yen. This study has developed anenvironmental data storage system using IoT to assist the labor force and to understandthe cultivation data of crops that have not been cultivated in Japan. Using this big data,
we have constructed and evaluated a forecasting system. In this study, we focus on thedomestic cultivation of bananas grown in the tropics as the subject to obtain data andmake predictions. For learning, a system was constructed to infer the next day's
temperature by time-series analysis, using as input data obtained from the IoT. In thispaper, we report a comparison of two time series analysis algorithms.

OS20-12 The Development of SaaS for Quantifying the Amount of Drifted Debris on the Coast
Ayumu Tominaga, Ryohei Komori (National Institute of Technology Kitakyushu College, Japan)
Eiji Hayashi (Kyushu Institute of Technology, Japan)
In the process of collecting drifted debris along a coastal area, it has been required toquantify the collection results. This study introduced a Software as a Service (SaaS)
equipped with image processing capabilities for the detection and identification ofdebris in photographs. The SaaS was developed to facilitate the automated weighingof the collected debris. In the beach cleanup event in June 2023, 317 images capturing
debris found during the coastal cleanup were gathered on a cloud server from 90
participants via this SaaS. These images were subsequently analyzed by object
recognition AI executed on a cloud server, leading to the detection and identificationof 954 instances of debris. This result represents the actual amount of debris collectedduring this cleanup event, and indicates that the achievements of the event were
quantified via the SaaS.

OS20-13 Development of an antagonistic wire-driven joint mechanism capable of rapid motion
and variable stiffness
Katsuaki Suzuki!, Yuya Nishida2, Kazuo Ishii2( Kumamoto Industrial Research Institute, Japan, 2Kyushu Institute of Technology, Japan)
1

The advancement of digital transformation in manufacturing is expected to demandnovel tasks for industrial machinery unlike before. When adding mechanisms or
actuators to enhance machine functionality, concerns arise about potential system
enlargement and increased complexity. The design concept of integrating multiple
functions into mechanisms using a few actuators and components serves as one means
to address these issues. We aim to elucidate the structural arrangement capable ofachieving three functions, including normal motion, rapid motion, and variable
stiffness, using two electric actuators. In this paper, we introduce a newly devised
antagonistic wire-driven joint mechanism aimed at improving the variable stiffness
function within the articulated mechanism we have developed, enabling threefunctionalities.

85©ICAROB 2024 ALife Robotics Corp. Ltd.

SliderSpringOutput link
Fixed link

Wire
Electric linear actuator

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS20-14 Gakken Hills Interdisciplinary Ekiden Competing with Humans, Animals, andRobotsTakuya Fujinaga1, Moeko Tominaga2, Daigo Katayama3, Kazuo Ishii3( Fukuoka University, 2Nishinippon Institute of Technology, 3Kyushu Institute of Technology, Japan)
1With the advancement of science and technology, the societal implementation of robots
has been expected. In recent years, the number of robots exiting in human living spaces
has increased, and interaction between humans and robots has become an important
issue. The purpose of this study is to propose a future society where humans, animals,
and robots coexist. We organize relay races, or ekiden in Japanese, where they compete
together as runners to discuss the future society. The competition rules have been
established for this ekiden to ensure that each runner can compete fairly. For example,
an advantage is given to the time of humans, animals, and robots, so that the results are
the same. This paper explains the competition rules and reports the competition resultsof the 8th race held in 2023.

OS20-15 Development of a Low-Cost Underwater Robot for Research and Education
Takuya Fujinaga (Fukuoka University Japan)
Underwater robots are frequently utilized for underwater observation. When used as
tools, commercially available underwater robots are sufficient. However, when utilized
for research, development, or education, the required specifications vary based on the
purpose, necessitating custom development. Generally, the development of underwater
robots demands a significant amount of funds, posing a high barrier for research,development, and education. The aim of this study is to reduce these barriers,
facilitating broader implementation of research, development, and education in thefield of underwater robotics. This paper explains the development requirements,
process and cost of a low-cost underwater robot designed for a robotics competitionand discusses key considerations during development.

OS21 Human-Machine Interface (3)Chair Norrima Mokhtar (University of Malaya, Malaysia)
Co-Chair Heshalini Rajagopal (UCSI University, Malaysia)
OS21-1 Enhancing Reconnaissance Missions Through Multiple Unmanned Systems in ROS

Anees ul Husnain1,2, Norrima Mokhtar1, Takao Ito3, Siti Sendari4, Muhammad Farris Kyasudeen5, Muhammad
Badri M Noor1,6, Heshalini Rajagopal7
1( Universiti Malaya, Malaysia), (2 The Islamia University of Bahawalpur, Pakistan)(3Hiroshima University, Japan), (4 Universitas Negeri Malang, Indonesia)(5 University Technology MARA (UiTM), Malaysia)
6( Ifcon Technology Sdn Bhd, Malaysia), (7UCSI University, Malaysia)
The synergistic collaboration between UAVs and UGVs addresses the limitationsof individual platforms, offering a versatile solution for reconnaissance tasks indiverse environments. The proposed system employs ROS as the underlyingarchitecture to facilitate seamless communication and coordination among multiple
UAVs and UGVs. Optimal coverage is ensured through efficient exploration andcoverage of the reconnaissance area, under by comparing raster-scan, expanding
spiral and zig-zag area exploration approaches. The article concludes by discussing
potential extensions, such as the integration of machine learning techniques forenhanced autonomy and the scalability of the system for larger-scale missions bypresenting a ROS-based framework that maximizes the synergy between UAVs and
UGVs.

86
©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS21-2 Illumination Effects on Facial Expression Recognition using Empirical Mode
Decomposition

Hashimah Ali1, Wan Khairunizam Wan Ahmad1, Hariharan Muthusamy2, Mohamed Elshaikh1(1Universiti Malaysia Perlis, Malaysia)
2( National Institute of Technology Uttarakhand, India)

Thus, this paper aims to investigate the illumination effects (brightness variations)on facial expression recognition using empirical mode decomposition
reconstruction techniques. In this framework, firstly, the noisy facial expression
images were simulated with the illumination effects using different brightnesslevels of 30%, 40%, 50%, 60%, and 70%. Then, the EMD will decompose the noisy
facial expression images into a small set of intrinsic mode functions (IMF), namely
IMF1, IMF2, IMF3, and residue. Based on property held by EMD, the signals aredecomposed into several IMF components, each with a different time scale.
Because the last several IMFs represent the majority of illumination effects, various
reconstruction techniques for IMFs have been investigated at various brightnesslevels. Feature reduction techniques Principal component analysis (PCA) and linear
discriminant analysis (LDA) have been employed to reduce the high-dimensional
space of IMF features into low-dimensional IMF features. The reduced IMF
reconstructions were then used as input to the k-nearest neighbour classifier torecognise the seven facial expressions. A series of experiments have been
conducted on the JAFEE database using various reconstruction IMFs together with
PCA plus LDA. Based on the results obtained, the reconstruction of IMF1 + IMF2
+ IMF3 shows the highest accuracy in high illumination conditions, which is
99.06%.

OS21-3 Supercontinuum Generation Pump By a Molydenum Disulfide Based Soliton ModeLocked Fiber Laser
Aeryn D. Ahmad1, Norrima Mokhtar1, Hamzah Arof1, Sulaiman Wadi harun1, Ahmad Haziq Aiman Rosol2(1Universiti Malaya, Malaysia), (2MJIIT, UTM, Malaysia)
In this work, a highly stable soliton mode-locked Erbium-doped fiber laser (EDFL)
is passively obtained using a molybdenum disulfide (MoS2) thin film as a saturable
absorber (SA). The MoS2 thin film obtained via electrochemical deposition
technique is integrated into an EDFL cavity to generate mode-locked pulses
operating at 1.88 MHz with a pulse duration of 3.03 ps. Supercontinuum (SC) lightis generated using the proposed soliton mode pulses operating at 1560.4 nm as they
are injected into a 100 m long highly nonlinear photonic crystal fiber (HN-PCF)
after it is amplified to the output power of 17.8 dBm. The SC light operates in awavelength range starting from 1360 nm to more than 1750 nm with the intensity
above -35 dBm. The proposed supercontinuum laser can be seen as a promisinglight source for metrology and sensing applications.

87©ICAROB 2024 ALife Robotics Corp. Ltd.

980nm LD

Isolator

EDF

WDM

2.4m

SA
MOS2

980/1550

SMF
100m
Coupler80%
20%

PCF 100m
EDFA
oscilloscope

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS22 Mathematical Informatics (8)Chair Takao Ito (Hiroshima University, Japan)
Co-Chair Makoto Sakamoto (University of Miyazaki, Japan)
OS22-1 Adsorption Behavior of Arsenic and Selenium using NiZn Hydroxy Double Salts withAcetate, Chloride, Nitrate, and Sulfate Anions
Kaoru Ohe, Ryosuke Tabuchi, Tatsuya Oshima (University of Miyazaki, Japan)
Arsenic(As) and selenium(Se) has become an increasingly serious water contaminationworldwide, so the development of adsorbents to improve the adsorption performanceof As and Se is desired. In this study, hydroxy double salt NiZn-AcO, -Cl, -NO3, and SO4, with acetate, chloride, nitrate, and sulfate anions between layers and theadsorption behavior of As and Se was investigated. The Langmuir isotherm model,
characteristic of monolayer adsorption, fit the data best with R2 >0.94. The adsorption
capacity(qmax) of As(V) by NiZn-Cl was about twice that of the other adsorbents. The
qmax of Se(IV) by NiZn-AcO, NiZn-Cl, and NiZn-NO3 was almost the same value asthat of Se(VI) and 1.4-1.7 times larger than that of NiZn-SO4. The prepared adsorbent
was confirmed to be useful for the removal of As and Se.

OS22-2 Parallel acoustic analysis based on the domain decomposition method with higher-orderelement
Amane Takei1, Makoto Sakamoto1, Akihiro Kudo2( University of Miyazaki, Japan) (2National Institute of Technology, Tomakomai Collage, Japan)
1

Numerical analysis approaches, such as the finite element method, are widely used forthe estimation of the sound field in architectural spaces. Large-scale analyses, using
numerical models with over 10 trillion elements, are required for the analysis of a large
space such as a concert hall with higher-frequency bands. Large spaces are often
limited to low-frequency analysis. In this study, the number of elements is reduced bywave acoustic analysis using higher-order elements. Based on the results using higherorder elements, it is shown that it is possible to analyze a real environment model suchas a live music club and a concert hall.

(a) Visualization results (X-Y)(b) Visualization results(X-Z)
Fig.1 Numerical result
using concert hall model

OS22-3 Sound field evaluation on the acoustical experiment using non-steady state analysis
Akihiro Kudo1, Makoto Sakamoto2, Amane Takei2(1National Institute of Technology, Tomakomai College, Japan) (2 University of Miyazaki, Japan)
Identifying acoustic effects of sound waves emitted by a sound source on the listeneris important for understanding the localization of sound as perceived by the listener. In
this presentation, we will analyze the effect of the presence of a chair on thepsychoacoustic effect using a practical model simulating a psychoacoustic experimentconsisting of a loudspeaker, a listener, and a chair. Non-steady state analysis is required
to analyze changes in acoustic effects over time. To analyze the interference of soundwaves in the vicinity of the listener, a short Gaussian pulse-like solitary wave is set upon the loudspeaker's vibrating surface. Since sound waves in the audible band are used
for these simulations, a mesh size of less than 25 mm is used. The laboratory is 3m of
depth and width, and 2.5m of height, which means that the computational whole target
consists of over 15 million elements. This simulation requires a large number ofcalculations, so a parallel computing method with ADVENTURE_Sound that is an
open-source software for sound simulation, is used to run the simulation. From thesimulation results of the non-stationary analysis and consider the effect of the presenceof the chair on the sound image localization.

88©ICAROB 2024 ALife Robotics Corp. Ltd.

Fig.1 Example of resultsof non-steady state
analysis with gaussianpulseThe 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS22-4 A DeepInsight method with Morphological Analysis
Toyoaki Tomioka1, Satoshi Ikeda1, Makoto Sakamoto1, Takao Ito2
( Faculty of Engineering, University of Miyazaki, Japan) (2Hiroshima University, Japan)
1

The DeepInsight method is a comprehensive approach aimed at applying convolutional
neural networks (CNN) to non-image data. This study specifically focuses on using the
DeepInsight method to forecast stock prices, which involve time series data. To achievethis, we employ morphological analysis to extract the necessary word feature forpredicting stock prices from news articles. Additionally, t-SNE is utilized to reduce thedimensionality of the high-dimensional data, transforming it into a more manageable,
low-dimensional format suitable for stock price prediction. After reducing
dimensionality in both news text data and stock price time series data using t-SNE, andsubsequently applying CNN for learning from the transformed representations,predictions were made with an accuracy of approximately 60%.

OS22-5 Support for Museum Exhibition of Small Fungi using AR Technology
Kakeru Takemura1, Ota Hamasuna1, Fumito Hamakawa1, Satoshi Ikeda1, Kaoru Ohe1, Amane Takei1,
Makoto Sakamoto1, Shuichi Kurogi2
1( University of Miyazaki, Japan) (2Miyazaki Prefectural Museum of Nature and History, Japan)
There are 25 species of bioluminescent fungi that have been confirmed in Japan, and12 species have been confirmed in Miyazaki Prefecture. Those fungi are very small.
Therefore, it is difficult to observe the structure of fungi exhibited in museums with
the naked eye. The purpose of this research is to display 3DCG models of mushrooms
using AR (Augmented Reality) technology in order to facilitate observation of thesesmall mushrooms exhibited in museums. Two hundred visitors to the museum wereasked to use the application and complete a survey. In order to measure the ease ofobservation of different app functions and mobile devices used in the survey, we
divided the respondents into four groups.

OS22-6 Automatic Selection of High-Grade Dried Shiitake Mushrooms using Machine Learning
Ota Hamasuna1, Kakeru Takemura1, Kodai Hasebe1, Fumito Hamakawa1, Bidesh Biswas Biki1, Satoshi
Ikeda1, Kaoru Ohe1, Amane Takei1, Makoto Sakamoto1, Kazuhide Sugimoto2(1University of Miyazaki, Japan) (2SUGIMOTO Co., Ltd. Japan)
SUGIMOTO Co., Ltd. collects dried shiitake mushrooms directly from approximately
600 producers in Takachiho. Shiitake mushrooms are at their peak in spring and fall,
and at peak times, more than 1 ton of mushrooms can be brought in each day. However,shiitake mushrooms are still sorted manually, and sorting this much requires a lot ofeffort. In this paper, we aim to efficiently determine the quality of shiitake mushrooms
using image processing and deep learning.

OS22-7 Predicting High Volatility Cryptocurrency Prices using Deep Learning
Tsutomu Ito1, Kodai Hasebe2, Fumito Hamakawa2, Bidesh Biswas Biki2, Satoshi Ikeda2, Amane Takei2,
Makoto Sakamoto2, Md Riajuliislam3, Sabrina Bari Shital4, Takao Ito5
(1National Institute of Technology, Ube College, Japan) (2University of Miyazaki, Japan)
3( TU Dortmund, Germany) (4Daffodil International University, Bangladesh) (5Hiroshima University, Japan)
Even if you want to make a profit from cryptocurrency, you are worried that you willlose money, and it is difficult to afford it. There are a vast number of papers that study
such unpredictable price fluctuations of cryptocurrency. Currently, it is mainstream touse deep learning to predict the price of cryptocurrency. The goal of this research is topredict the price of cryptocurrency over the long-term using deep learning. The
algorithms used are LSTM, GRU, and Bi-LSTM. The cryptocurrency targeted are
Bitcoin, Ethereum, Litecoin, and Cardano. Finally, we will compare it with previous
research and verify the performance of our model.

89©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS22-8 Simulation of Weathering Representation using Vertex and UV Information
Tsutomu Ito1, Fumito Hamakawa2, Kodai Hasebe2, Satoshi Ikeda2, Amane Takei2, Makoto Sakamoto2,
Takao Ito3 (1National Institute of Technology, Ube College, Japan) (2University of Miyazaki, Japan),(4Hiroshima University, Japan)
In recent years, three-dimensional computer graphics (3DCG) technology has beendeveloped. In addition, a plethora of research have been published on weathering
representations such as rust and moss for realistic representation. However, when
outputting simulation results on an image, an enormous number of images are required
to display different simulation results in a three-dimensional space. In this paper, asimulation method using vertex information of a 3D model and simple images is
proposed. In this method, when the number of vertices is sufficient, the simulation
result output does not use images, thus reducing the data increase. This figure is one of
the experimental results.

.

OS23 Industrial Revolution (5)Chair Hazry Desa ( Universiti Malaysia Perlis (UniMAP), Malaysia)
OS23-1 Investigating the Engineering Interventions in the Conservation of Malaysia Heritage
Structures: A Review on Preserving Historical Edifices Through Advance Civil Engineering
Techniques
Muhammad Azizi Azizan, Nurfadzillah Ishak, Hazry Desa (COE-UAS, UniMAP, Malaysia)
This review delves into the realm of Malaysia's heritage conservation, spotlighting thetransformative impact of advanced artificial robotic aid in civil engineering techniques.
Through the integration of Fiber-Reinforced Polymers and nanotechnology, historicaledifices are fortified, seamlessly blending modern engineering with architectural
elegance. Non-destructive testing (NDT) methods, including ground-penetrating radarand Finite Element Analysis (FEA), empower conservationists with deep insights into
structural intricacies, guiding targeted interventions. In the digital sphere, 3D laser
scanning captures intricate details, while Virtual Reality (VR) simulations facilitate
immersive exploration and decision-making. This harmonious fusion of cutting-edgeengineering ensures the enduring legacy of Malaysia's architectural treasures.

OS23-2 Drones and Data: A Comprehensive Exploration of UAVs in Data Mining
Muhammad Azizi Azizan, Nurfadzillah Ishak, Hazry Desa (UniMAP, Malaysia)
Encapsulates a comprehensive investigation into the symbiotic relationship between
Unmanned Aerial Vehicles (UAVs) and data mining, as encapsulated in the discourse
titled "Drones and Data." The technological transitions facilitated by UAVs,
emphasizing the integration of machine learning algorithms, cloud-based dataprocessing, and the collaborative synergy between stakeholders. Therefore, this study
a glimpse into the intricate web of applications and technological advancements at theintersection of UAVs and data mining. It serves as a scholarly guide through the
evolving landscape of "Drones and Data," UAVs play a central role in unlocking
unprecedented insights and efficiencies, reshaping the future of data mining.

90©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS23-3 Development of Variable Arm to Control the Manoeuvrability of Quadrotor
L. Y. Hong, H. Desa and M. A. Azizan (UniMAP, Malaysia)
M. H. Tanveer (Kennesaw State University, USA)This paper introduces a variable arm concept for quadrotors, enabling manoeuvrability
by adjusting arm length. Variations in arm's length impact thrust-generated bending
moments, tilting and directing quadrotor movement. The main aim is to create aquadrotor with adaptable arm length to efficiently control maneuvers, reducing theneed for extra thrust during flight. The focus is on designing a quadrotor capable ofextending or retracting its arms. The proposed idea centers on using the variable arm
to alter bending moments and control manoeuvrability. Testing a quadrotor with thisadjustable arm showed successful performance in agile manoeuvres. Results confirm
the variable arm's ability to induce body rotation, effectively regulating quadrotor
manoeuvrability.

OS23-4 Development of IOT-Enabled Smart Water Metering System
S. D. Wen, H. Desa and M. A. Azizan (UniMAP, Malaysia)
A. -S. T. Hussain (Al-Kitab University, Iraq) M. H. Tanveer, R. Patan (Kennesaw State University, USA)This paper presents a smart IoT water meter for automated readings, aiming to create
both a device and a compatible mobile app. Instead of manual reading, the IoTenabled meter, using a camera and CNN, accurately captures data. It utilizes ESP32
CAM for collection, a laptop as a gateway, and MQTT for transfer. Data is stored in
Firebase and analyzed via the app. A functional prototype tested in a housing areacollects data, monitored through the app. The analysis evaluates method suitability,
offering future improvement recommendations.

OS23-5 Object Detection and Instance Segmentation with YOLOV8: Progress and Limitations
L. J. Lee, H. Desa and M. A. Azizan (UniMAP, Malaysia)
A. -S. T. Hussain (Al-Kitab University, Iraq) M. H. Tanveer (Kennesaw State University, USA)This research employs object detection and instance segmentation algorithms to
distinguish between objects & backgrounds and to interpret the detected objects. The
YOLOV8 (You Only Look Once) framework and COCO dataset are utilized for
detecting and interpreting the objects. Additionally, the accuracy of detection,
segmentation, and interpretation is tested by placing objects at various distances fromthe camera. The algorithm's performance was evaluated, and the results were
documented. In the experiments, a sample of 11 objects was tested, and 8 of themwere successfully detected at distances of 45cm, 75cm, 105cm, and 135cm. For
instance, segmentation, segmentation maps appeared clean when detecting a single
object but faced challenges when multiple objects overlapped.

91©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS24-Robotics and Intelligent Casting (5)Jiwu Wang (Beijing Jiaotong University, China)
OS24-1 A high-performance motion planning method based on asymptotically optimal RRT
Tianbin Meng (Beijing Jiaotong University, China), Jiwu Wang (Beijing Jiaotong University, China)
The motion planning algorithm of the robot arm plays an important role in the working
process of the robot arm, especially in the complex environment, an efficient algorithm
is more conducive to the robot arm to complete the corresponding planning task.
Aiming at the problems such as low exploration efficiency and poor planning path in
the current motion planning task of robotic arm, we propose a distance constraint
mechanism. Based on 𝐑𝐑𝐓 ∗ and 𝐈𝐧𝐟𝐨𝐫𝐦𝐞𝐝 − 𝐑𝐑𝐓 ∗ , the algorithm uses haltonsequence to generate random points and introduces the current lowest cost path length.
To avoid useless nodes extension. The simulation results show that the algorithm with
distance constraint mechanism can improve the exploration efficiency and planning
quality to some extent.

OS24-2 CSM-RRT*: an improved RRT* algorithm based on constrained sampling mechanism
Hang Yang (Beijing Jiaotong University, China), Jiwu Wang (Beijing Jiaotong University, China),
Xueqiang Shang (Aero Engine Corporation of China, China)
In this paper,an improved RRT* algorithm(CSM-RRT*) based on constrained samplingmechanism is proposed .The entire planning process is divided into two steps: fast
exploration and optimization of the initial path.Firstly,with the removing of redundantnodes and saving of collision nodes by constrained sampling strategy,the number of
redundant nodes and collision times is reduced which avoids excessive exploration ofrepetitive and collision regions.The target bias strategy is used to reduce unnecessary
expansion while preserving the node tree's ability to explore unknown regions andquickly finding a path connecting the starting and ending points.Subsequently, adynamic sampling region consists of removed redundant nodes and collision nodes isformed around initial path .By prioritizing exploration within this dynamic
region,computational resources can be saved and the asymptotic optimal path can bequickly converged from the initial path.Eventually,simulation results presented invarious obstacle cluttered environments confirm the efficiency of CSM-RRT*.

OS24-3 Small Sample Object Detection Based on Improved YOLOv5
Yuxuan Gao(Beijing Jiaotong University, China),Jiwu Wang(Beijing Jiaotong University, China), Zixin
Li(Aero Engine Corporation of China)
Object detection is widely used in various production and life, such as mask detection
and recognition during the epidemic, face recognition with masks. Object detection
algorithm based on deep learning has always been an important research content andimplementation method in the field of object detection. Due to the large number oflead seals and fuses, their locations are not fixed, the lead seals and fuses havedifficulties such as few sample datasets, complex target background and easy to be
blocked, and strong reflective interference, and the conventional image processing
methods are difficult to solve the problem of effective object recognition. In this study,
by expanding the datasets, using different data enhancement methods, and training in
the improved algorithm, the detection accuracy, detection speed, and adaptability wereeffectively improved.

92©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS24-4 Research on Gas Pore Prediction Method Based on Sand Core Characteristic Time
Xiaolong Wang1, Qihua Wu2, Jiwu Wang1, Jinwu Kang3, Na Li2, Yucheng Sun2
( Beijing Jiaotong University, China) (2Weichai Power Co., Ltd. China) (3Tsinghua University, China)
1

In the production of castings, gas pores are a prevalent defect, particularly incomponents where air-tightness is crucial, such as cylinder blocks and heads. These
defects primarily arise from the intrusion of gases into the casting during the
combustion of resin in the sand core. Due to the complexity of the sand core's gas
evolution and venting process, predicting gas pores using numerical simulation posessignificant computational and time challenges. This paper introduces a rapid prediction
method for gas pores based on the characteristic time of the sand core. By setting theheat transfer boundaries, initial conditions, and termination criteria for computation,
the thermal conductivity of the sand core is adjusted. The termination computation time
is used as the characteristic time of the sand core. This time is then compared with acritical characteristic time to predict the distribution of gas pores. As the analysis of
more sand cores is incorporated, the precision of the critical characteristic time
improves, leading to more accurate predictions of gas pores in castings.

OS24-5 Optimization Analysis of a Deep Learning-Based Model for Predicting Temperature
Fields in the Solidification Process of Castings
Yahui Yang, Jiwu Wang (Beijing Jiaotong University, China) Jinwu Kang (Tsinghua University, China)
In the foundry industry, accurate prediction of the temperature field of castings duringsolidification is essential to ensure high product quality and improve productivity. This
study provides an in-depth investigation of the Unet deep learning model developed
specifically for this purpose, focusing on the impact of key training parameters such as
optimiser selection, batch size, number of iterations and loss function selection on theprediction performance of the Unet model. The findings provide practical insights on
how these parameters can be selected and tuned to improve the accuracy and reliabilityof model predictions. The results of this study not only provide new ideas for the practiceof deep learning applications in the foundry industry, but also help to improve theaccuracy and efficiency of the production process.

OS25 Research Towards the Sustainable Development Goals (SDG’s) (10)Chair Ammar A.M. Al Talib (UCSI University, Malaysia)
Co-Chair Takao Ito (Hiroshima University, Japan)
OS25-1 Portable Green Energy Mobile Laptop Charging Station
Ammar A. M. Al-Talib1, Rodney Tan1, Ang Aun Jie1, Idayu M. Tahir1, Sarah Atifah Saruchi2, Cik Suhana Bt.
Hasan1, Amar Rizwan1
1( UCSI University) (2UMPSA, Malaysia)
Mobile phones and laptop computers require electrical power to recharge
when the battery is down.. As a result, it would be very useful if a portablecharging station derived from renewable energy harvesting could be built, sothat individuals could recharge their phones and laptops whenever needed.
The objective of this project is to design and develop a green energy mobileand laptop charging station that uses wind and solar energy and evaluate
the performance of the designed station under different working conditions.
The efficiency of the power generated to charge the station is achieving 95.6%
for solar charging, which is considered a high efficiency for a renewableenergy charging station. Based on the analysis of the charging station
results, it has been proven that it can provide sufficient power and is safe foruse as a portable mobile laptop charging station.

93©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS25-2 Auto Indoor Hydroponics Plant Growth Chamber
Ammar A.M. Al-Talib1, Tew Hwa Hui1, Sarah Atifah Saruchi2, Idayu M. Tahir1, Nor Fazilah Binti Abdullah1(1UCSI University) (2UMPSA, Malaysia)
The objective of this project is to build an auto indoor hydroponics plantgrowing chamber that has an auto monitoring and controlling system. A
ESP32 based hydroponics electrical system is built with the attachment of
hardware components such as temperature and humidity sensor, lightintensity sensor, water level sensor, and water flow rate sensor. The softwaredevelopment of the system is through Arduino IoT Cloud platform, which
has an overall suitability in terms of features, cost, and user intuitivenessfor starters. Results have shown that ESP32 can ensure stable power supply..
After testing and validation, all of the electrical components are stored in a
power enclosure box to prevent contact with liquid. In short, the developed
auto indoor hydroponics plant growth chamber has effectively demonstrated
the ability in easing the plant cultivation procedure for agriculturalcommunity.

OS25-3 A Design and Fabrication of a Solar Agriculture Water Pumping System
Ammar A.M. Al-Talib1, Idayu M.Tahir1, Ain Atiqa1 , Amar Rizwan1, Sarah Atifah Saruchi2, Yazan Abu Al
shaikh1
1( UCSI University) (2UMPSA, Malaysia)
This study explores the use of a solar driven water pump. PV technology replaces
conventional electricity and diesel pumps by using solar energy to power DC or AC
water pumps. The main objective of this study is to design and construct a solar-powered
agriculture water pumping system and to evaluate its performance. The solar agriculture
water pumping system used in this project consists of a 40-watt monocrystalline solar
cell with an efficiency conversion of between 23% – 24%. This can supply power toa 16.8 W DC Flow Submersible Pump. It could lift the water up to 5m and a flow
range of 700 Liter/Hour. The system also includes a PWM 30A Solar Charge Controller
to regulate the input power to a 12V, 7.2A Sealed Rechargeable Battery. Finally, a 20m
long watering kit with nozzles irrigation system is connected to a 12V DC Submersible
Pump to water the plants.

OS25-4 Design and Performance of a Power Generating Manual Treadmill
Ammar A. M. Al-Talib1, Sarah Atifah Saruchi2, Cik Suhana Bt. Hasan1, Nor Fazilah Binti Abdullah1, Ain
Atiqa1, Ahmad Jelban1
1( UCSI University) (2UMPSA, Malaysia)
Treadmills are one of the most popular training equipment in the gym and at
home. The working principle of treadmills is by moving the belt with thehuman knee bending, which creates mechanical energy to turn the belt. A
gear or pulleyand belt system connects to the generator along the axel lineof the rolling bars. The power generated by the DC generator is stored in abattery pack and could be used to charge phones or other equipment. It has
been found that treadmills can provide an efficiency of 95% when the DC
motor is used and 92% when the AC motor is used. The main objective of thisstudy is to design and fabricate a powder-generating manual treadmill andto analyze the performance of the system under differentoperation conditions.

94©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS25-5 Smart Car Jack Using Internet of Things
Idayu M.T. Noor, Ammar A.M. Al- Talib, Mahmoud E.A. Zeiad, Suhana B.H. Cik,(UCSI University, Malaysia)
This paper presents a modified car jack design that is safe, easy to operate, and reduces
physical effort for lifting and lowering vehicles during automobile repair, especially forroadside situations. The study explores the integration of Internet of Things (IoT)
technology in automotive tools, particularly in the development of smart car jacks. The
proposed smart car jack offers convenience and enhanced control over vehicle
maintenance tasks. Comparative analysis between smart car jacks and traditional jacks
reveals several advantages of the former. Smart jacks demonstrate higher lifting
capacities and efficiency due to their use of hydraulic systems and advanced motorizedmechanisms. The most significant differentiator is the integration of technology in smart
car jacks. Equipped with microcontrollers, sensors, and mobile applications, they offer
remote control and real-time monitoring features. These technological advancements
provide users with enhanced convenience and safety, setting smart jacks apart from theirtraditional counterparts.

OS25-6 Gas Detection for Biogas System Using Internet-of-Things (IoT)
Ammar A.M. Al Talib, I.H.W. Yang, Idayu M.T. Noor, Haslija A.B. Ayu, Afifi. Z. Nur Muhammad(UCSI University, Malaysia)
The melting of the polar icecaps, rising sea levels, increase in allergies and infectious
disease outbreaks, those are only a few effects of global warming. Global warming
remains one of the most detrimental by-products of industrialization. Fossil fuels
contribute to the majority of greenhouse gases emitted but remain a popular option for
the generation of energy. An easy fix for this conundrum is to utilize other forms of fuelfor energy generation which burns cleaner and renewable as opposed to fossil fuels. The
solution would be to use biomethane generated from waste products which burn cleaner
and comes from a renewable source. In this paper, an IoT based biogas monitoring
system for biogas reactors is proposed. An ESP-32 microcontroller system is deployed
and tested to detect the presence of gas production. A dashboard plotting the dataobtained from sensors is designed to help user monitor parameters.

OS25-7 IoT Based Smart Mushroom Growing Kit
Ammar A.M. Al- Talib, C.K.J. Ting, Noor Idayu M. Tahir, Ain Atiqa, T.Y. Hui(UCSI University, Malaysia)
This project introduces an IoT-based smart mushroom growing kit to meet the rising
global demand for high-quality mushrooms. Various species of mushrooms can be
efficiently grown in different environmental conditions with the help of IoT devices thatenable farmers to regulate the climate condition according to the specific needs of eachtype of mushroom. The kit employs sensors and actuators, including temperature,humidity, MQ-135, and ultrasonic sensors, along with an ESP32 camera, controlled bya microcontroller. The collected data is transmitted to an IoT platform via Wi-Fi,
facilitating real-time monitoring and control through a user-friendly dashboard on Blynk
website and Blynk app. This innovative system optimizes mushroom cultivation byadjusting environmental conditions, offering efficiency and profitability. Users canremotely monitor and regulate the growth environment through their smartphones,
enhancing the overall mushroom cultivation experience.

95©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS25-8 Design and Analysis of Artificial Magnetic Conductor for Metal Shielding Applications
in RFID Car Detection Applications
Eryana Hussin1,2, Azman Zakariya2, Md. Ashraful Haque2,3, Nur Izzati Ali4
(1UCSI University) (2UTPSA, Malaysia) (3Daffodil International University, Bangladesh) (4UMP, Malaysia)
This paper presents the design of symmetrical Artificial Magnetic Conductor (AMC) for
metal shielding in RFID applications. It integrates a simple straight dipole as the RFID
tag representation, optimizing signal fidelity within metallic environments. Tailored for
low-frequency RFID use, the multiple-array AMC addresses UHF bandwidth limitations
by incorporating slots, ensuring a compact design and expanded bandwidth. This
proposed structure enhances performance of the dipole which includes return loss,
directivity, and bandwidth.

OS25-9 Effect of Fiber Orientation on the Mechanical Performance of Natural Fiber Polymer
Composite Bicycle Frame using Finite Element Analysis
Kok Sem Too, Cik Suhana Hassan, Nor Fazilah Abdullah, Ammar Abdulaziz Majeed Al-Talib(UCSI University, Malaysia)
In this research, the performance of oil palm empty fruit bunch (OPEFB) fibrereinforced epoxy composite with varying fibre orientation and stacking sequence as thematerial for mountain bike frame was studied utilizing ANSYS software. The choice of
OPEFB fibre was motivated by the fact that the waste by-product of oil palm extractionin Malaysia alone might reach 70-80 million tons per year, with 90% of oil palm biomass
lost as waste. The properties of epoxy OPEFB composite in principal 1, 2, and 3
directions were calculated using Whitney and Riley estimates. 10 stacking sequences
and five loading conditions were taken. The results have shown that the fibre orientationof epoxy OPEFB composite on the bicycle frame had little effect on the performance
contrary to the number of plies in the laminate or number of laminates which had major
effects.

OS25-10 Modelling of Short-Circuit Protection for A Residential Grid-Connected BESS
Kong De Kang, Farah Adilah Jamaludin, Rodney H.G. Tan(UCSI University, Malaysia)
This research paper presents the power protection study on a grid-connected Battery
Energy Storage System (BESS) in a typical Malaysia low-voltage (LV) residential
network. The BESS model and control algorithm is developed in MATLAB/Simulink
environment to provide a platform for short circuit fault analysis of BESS. The BESS
model can charge and discharge its energy with an algorithm-controlled bidirectional
AC/DC converter. The paper also presents two cases for BESS short circuit.The
protection level on BESS is also optimized by introducing a time-delay characteristics
model to coordinate circuit breakers in compliance with standards outlined in IEEE Std
1375-1998, IEC/EN 60898-2, IEC/EN 60947-2, UL 1077, and CSA 22.2. No.235. As aconclusion, BESS was interrupted within stipulated time. In case of internal short circuit
protection failure, a backup protection will act on isolating BESS from the grid provided
that the discharged current exceeds system capacity.

96©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS26 Navigating the Digital Frontier: Innovations in the Age of Industry Revolution 4.0 (11)Chair Wei Hong Lim (UCSI University, Malaysia)
Co-Chair Takao Ito (Hiroshima University, Japan)
OS26-1 An Intelligent Cargo/Warehouse Management System
Zhongheng Sun1, Zhou Yue1, Xun Sun1, Wenzhuo Fan1, Wenxuan Zhou1(1Fuzou University, China)
This paper presents a low-cost and user-friendly warehouse management systemdeveloped using Arduino and ESP8266 hardware. The system accurately monitors
temperature, humidity, and harmful gas concentration, and acts as an intrusion detector ina warehouse environment. It also includes functions for managing and detecting goods
using RFID chips and for automatic delivery using unmanned intelligent vehicles. Data is
uploaded to a cloud-based relational database in real time. The system provides a costeffective and efficient solution for warehouse management.

OS26-2 A Comprehensive Approach to Design and Implement an IoT-Enabled Intelligent
Shopping Cart System with Obstacle-Aware Navigation and Enhanced Customer Engagement
for Elevated Consumer Experiences
Yao Chen1, Jiacheng Du1, Bo Peng1, Ningfei Wang1, Zehan Huang1, Wei Hong Lim2, Sew Sun Tiang2,
Mastaneh Mokayef2, Chin Hong Wong1
1( Fuzhou University, China) (2UCSI University, Malaysia)
Supermarket shopping is a universal experience. This project focuses on enhancing thisexperience by designing an intelligent, user-friendly shopping cart and an interactive
webpage. The cart features include accessing item information, automatic movement,
and obstacle detection. Users can view their itemized costs and control the cart's
movements through the webpage. Utilizing an ultrasonic module, the cart autonomously
navigates and follows specific customers, incorporating automatic barcode scanning for
product identification. Connectivity is achieved through a WiFi module using the MQTT
protocol, facilitating server communication. This development exemplifies Internet ofThings applications, showcasing how internet connectivity can significantly increase
convenience in everyday activities.

OS26-3 Design of a Four-Port Flexible UWB-MIMO Antenna for Wearable and IoTApplications
Jia Wei Tan, Sew Sun Tiang, Kim Soon Chong, Mohammad Arif Ilyas, Mastaneh Mokayef, Zhi Ying Yeoh,
Wei Kang Lai, Wei Hong Lim (UCSI University, Malaysia)
This paper introduces a compact, four-port MIMO antenna for ultrawideband (UWB)
applications, measuring 60×60 mm². Printed on a single-layer flame resistant (FR-4)
substrate (permittivity 4.3, thickness 1.6mm), the antenna features four microstrip cells,
each orthogonal to its neighbor for improved isolation. It includes a rectangular patchwith staircase slits and a stepped feed line. Square stubs at the top center enhanceisolation. The antenna boasts a significant return loss (-43.75dB), wide impedance
bandwidth (1.967-12GHz), and isolation below -15dB. Its envelope correction
coefficient (ECC) is under 0.02 with a moderate 4.4dBi gain. Although its 63% radiation
efficiency could be enhanced, the antenna's ultrawide bandwidth and compactness makeit suitable for UWB wearable IoT applications.

97©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS26-4 Investigate Power Efficiency in PLECS and MATLAB Software by Designing USB 5W
ChargerZhi Ying Yeoh, Kim Soon Chong, Sew Sun Tiang, Mohammad Arif Ilyas, Jia Wei Tan, Wei Kang Lai,
Wei Hong Lim (UCSI University, Malaysia)
In power electronics, accurately assessing simulation tools is key for precise, reliable
electronic system designs. This study compares MATLAB Simulink and PLECS inmodeling a 5W USB charger’s power characteristic. The charger, using an AC-DC fullbridge rectifier and DC-DC flyback topology, delivers a stable 5VDC at 1A. The
analysis focuses on power efficiency and thermal characteristics, incorporating real-life
components for detailed insights. Results show PLECS, specialized in powerelectronics, surpasses MATLAB in accuracy and consistency. This research aids inunderstanding simulation tools' effectiveness, guiding engineers and researchers in
power efficiency evaluations for electronic systems.

OS26-5 Design and Simulation and Performance of Grid Connected Photovoltaic System for
Small, Tall Building in Malaysia
Wei Kang Lai, Kim Soon Chong, Sew Sun Tiang, Mohammad Arif Ilyas, Jia Wei Tan, Zhi Ying Yeoh,
Wei Hong Lim (UCSI University, Malaysia)
In Malaysia's rapidly urbanizing landscape, sustainable energy for small, tall buildings
is increasingly vital. This study addresses this need through the design, simulation, andperformance analysis of grid-connected photovoltaic systems tailored for these uniquestructures. Utilizing AutoCAD and PVSyst for design and simulation, the researchdetails rooftop array dimensions, PV panel wiring, and system components aligned withMS 1837-2018 standards. Results indicate the designed system can save 526.70MWh
annually, costing RM 339,306.98, with a 9.8% return on investment (ROI) and a 13year breakeven point, emphasizing its sustainability and economic viability.

OS26-6 A Design of Dual-Band Coplanar Waveguide (CPW) Printed Antenna for 1.9 - 3.6GHz
Applications
Jia Wei Tan, Sew Sun Tiang, Kim Soon Chong, Mohammad Arif Ilyas, Mastaneh Mokayef, Zhi Ying Yeoh,
Wei Kang Lai, Wei Hong Lim (UCSI University, Malaysia)
A dual-band coplanar waveguide (CPW) printed antenna for IoT application is
proposed, mounted on a low-profile RT/duroid 5880 substrate with dieletric constant of
2.2, loss tangent of 0.0009 and a standard height of 0.787mm. This design aims to cover
the major frequency bands from LTE to Bluetooth/Wi-Fi band/WiMax/Zigbee,
Extended IMT and 5G whereby the bandwidth range between 1.7 GHz to 3.6 GHz. The
antenna is miniaturized through the CPW technique and has a rectangular size of 60 x
30 x 1.187mm3. The design and simulation of the result records a return loss of 25.12dB, peak gain of 4.23dBi, voltage standing wave ratio (VSWR) close to 1,
omnidirectional radiation, and current distribution. Radiation efficiency reaches
approximately 94%, with a total efficiency of 89.4% between 1.9GHz-3.6GHz.

98©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS26-7 Optimized Convolutional Neural Network Towards Effective Wafer Defects
Classification
Koon Hian Ang1, Koon Meng Ang1, Chin Hong Wong2,3, Abhishek Sharma4, Chun Kit Ang1, Kim Soon
Chong1, Sew Sun Tiang1, Wei Hong Lim1
1
( UCSI University, Malaysia) (2Fuzou University, China) (3Maynooth University, Ireland)(4Graphic Era Deemed to be University, India)
Semiconductor defect inspection is crucial for yield improvement but is hindered bymanual inspection's subjectivity and error. This paper employs Convolutional Neural
Networks (CNNs) for automated wafer defect classification, addressing the challengesof time-intensive training and complex hyperparameter tuning. We propose theArithmetic Optimization Algorithm (AOA) to efficiently optimize CNN
hyperparameters like momentum, initial learning rate, maximum epochs, and L2
regularization. Our method reduces the trial-and-error in hyperparameter tuning. Using
the AOA-optimized ResNet-18 model, our simulations show superior performance in
defect classification compared to the unoptimized model, demonstrating its
effectiveness and practical potential.

OS26-8 Tackling Photovoltaic (PV) Estimation Challenges: An Innovative AOA Variant for
Improved Accuracy and Robustness
Rayan Mohammed Noor Mohammed Bakhit1, Abhishek Sharma2, Tiong Hoo Lim3, Chin Hong Wong4,5, Kim
Soon Chong1, Li Pan1, Sew Sun Tiang1, Wei Hong Lim1
1
( UCSI University, Malaysia) (2Graphic Era Deemed to be University, India) (3Universiti Teknologi Brunei,
Brunei Darussalam) (4Fuzou University, China) (5Maynooth University, Ireland)
Optimizing photovoltaic (PV) cell/module modeling is key to advancing solar power
and achieving net zero carbon goals. Challenges in accurate PV parameter estimationarise from environmental variability, aging, and incomplete manufacturer data.
Traditional Arithmetic Optimization Algorithm (AOA) often struggles with premature
convergence due to imbalanced exploration and exploitation. This paper presents anenhanced AOA variant, incorporating chaotic maps and oppositional-based learning tobetter balance the optimization process. Our extensive simulations show that thisimproved AOA variant significantly enhances accuracy and robustness in PV
cell/module parameter estimation compared to the conventional method.

OS26-9 Deep Learning in Manufacturing: A Focus on Welding Defect Classification with CNNs
Tin Chang Ting1, Hameedur Rahman2, Tiong Hoo Lim3, Chin Hong Wong4,5, Chun Kit Ang1, Mohamed Khan
Afthab Ahamed Khan1, Sew Sun Tiang1, Wei Hong Lim1
1
( UCSI University, Malaysia) (2 Air University, Pakistan) (3Universiti Teknologi Brunei, Brunei Darussalam)(4Fuzou University, China) (5Maynooth University, Ireland)
Welding is integral to modern manufacturing, yet the complex process often leads todefects, impacting the quality of the final product. Recent advances in deep learning,particularly Convolutional Neural Networks (CNNs), have shown remarkable results inapplications like defect recognition. This study evaluated AlexNet, ResNet-18, ResNet50, ResNet-101, MobileNet-v2, ShuffleNet, and SqueezeNet for their effectiveness in
identifying welding defects, using accuracy, precision, sensitivity, specificity, and Fscore as metrics. The dataset covered defects like cracks, lack of penetration, porosity,and a no-defect class. Our analysis shows that most of these architectures deliver
promising results in accuracy, sensitivity, specificity, precision, and F1-score,highlighting their potential in defect recognition.

99©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS26-10 Enhancing Global Optimization Performance of Arithmetic Optimization Algorithmwith a Modified Population Initialization Scheme
Tin Chang Ting1, Hameedur Rahman2, Meng Choung Chiong1, Mohamed Khan Afthab Ahamed Khan1, Cik
Suhana Hassan1, Farah Adilah Binti Jamaludin1, Sew Sun Tiang1, Wei Hong Lim1
(1UCSI University, Malaysia) (2Air University, Pakistan)
Arithmetic Optimization Algorithm (AOA) is widely used to solve global optimization
problems. However, it often faces premature convergence challenges in complex
optimization scenarios. A key factor affecting AOA's performance is the solution qualityof the initial population. The conventional initialization scheme, despite its prevalence,lacks reliability in ensuring high-quality solutions due to inherent stochastic processes.
To address this issue, we propose a modified initialization scheme that improves initial
population quality by integrating chaotic maps and oppositional-based learning.
Through extensive simulation studies, we demonstrate that the enhanced AOA,
equipped with this new initialization scheme, exhibits superior performance in solving
a range of benchmark functions with improved accuracy.

OS26-11 Enhancing Precision Object Detection and Identification for Autonomous Vehicles
through YOLOv5 Refinement with YOLO-ALPHA
Guandong Li1, Yanzhe Xie1, Yuhao Lu1, Jingzhen Fan1, Yuankui Huang1, Zongyan Wen1, Wei Hong Lim2,
Chin Hong Wong1
1( Maynooth University, Ireland) (2UCSI University, Malaysia)
Advancing swiftly in contemporary society, the rapid growth of autonomous driving
technology suggests its potential adoption across continents. The realization of fullyautonomous driving relies on proficiently detecting, classifying, and tracking roadobjects such as pedestrians and vehicles. This research employs the YOLOv5 neuralnetwork, enhancing it with YOLO-ALPHA. Modifications, encompassing freeze andattention mechanisms, serve to refine accuracy and expedite training. Furthermore,adjustments to the activation function aim to stabilize precision and recall. The
integration of a FCN based on semantic segmentation theory contributes to improved
accuracy in detecting road conditions during autonomous driving. Consequently, thisenables the successful and highly accurate functionality of automatic identification.

OS27 Post-narratological Approaches to Cognition in Humans and Robots (5)Chair: Jumpei Ono (Aomori University, Japan)
Co-Chair: Hiroki Fxyma (Kobe University, Japan)
OS27-1 A Game Framework Based on the Disinformation Warfare in Russo-Ukrainian War
Jumpei Ono (Aomori University, Japan), Takashi Ogata (Iwate Prefectural University, Japan)
From 2022, when Russia began its invasion of Ukraine, to the present, the number ofspeeches and information directly/indirectly related to Russo-Ukrainian war has
continued to increase in mass communication venues such as SNS and TV. This study
proposes a game that simulates the spread of disinformation by focusing on theinformation published by Ukraine as a list of disinformation speakers, especially on the
topics they talk about. The game is intended to provide players with psychologicalimmunity against disinformation through the simulation of disinformation
dissemination.

100©ICAROB 2024 ALife Robotics Corp. Ltd.

PL

Scenario

Story technique

Story content
knowledge
base

GMThe 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS27-2 Comparative Analysis of Eye Tracking between Veteran and Novice during
Radiological Interpretation
Yuka Naito, Jun Nakamura (Chuo University, Japan),
Yoshinobu Ishiwata (Yokohama City University, Japan)
With the development of AI, AI may substitute for physicians in radiologicalinterpretations in the future. The purpose of this paper is to identify factors that could
form the foundation for algorithms enabling AI to explore pathological findings. This
study focuses on the "angles of gaze trajectory" and conducts a comparative analysis
between veteran and novice. The results indicate commonalities and variances in theangles of gaze trajectory between veteran and novice, particularly around 10, 350, and180 degrees. Both Veteran and Novice have a relatively higher frequency of gazemovement around 10 and 350 degrees compared to other angles. Veteran has a relatively
higher frequency of gaze movement around 180 degrees compared to other angles,
while novice has a relatively lower frequency around 180 degrees.

OS27-3 Development of Notification System to Prevent Working Productivity from Declining
Caused by Increased Carbon Dioxide Concentration
Yohei Kamoda, Jun Nakamura (Chuo University, Japan)
In association with the change of work style to remote, the home environment should
be considered in terms of carbon dioxide concentration (CDC) which has negative
effects on the human body, such as less cognitive abilities. To challenge this problem,the author developed an alert system that supports remote workers to be notified of anincrease in CDC, by combining Raspberry Pi, CDC sensor, and Slack. As a result, thedeveloped alert system was able to support the user in keeping the CDC in the room
below the set threshold. In addition, the system shows that a significant increase in CDC
can be observed in the room when it comes to insufficient ventilation.

OS27-4 Visualization of the Skilled Physician's Gaze Characteristic during Diagnosis
Taiki Sugimoto, Jun Nakamura (Chuo University, Japan)
and Yoshinobu Ishiwata (Yokohama City University, Japan)
This study examines gaze movement differences in diagnosis between skilled andunskilled physicians, aiming to identify factors influencing diagnostic speed andenhance artificial intelligence's diagnostic capabilities. Results reveal that experienced
physicians, on average, spent 61% less diagnostic time than beginners, covering 49%of the gaze distance on the X and Y-axes. (comparison of the cumulative distance ofgaze on the screen). Despite increased movement on the Z-axis (comparison of thescrolling speed of the CT scan results), skilled physicians moved 2-3 times faster,effectively narrowing attention and identifying specific areas.

101©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS27-5 Comparative Analysis of Methods for Visualizing the Sensory Experience of Food andBeverages
Hiroki Fxyma (Kobe University, Japan)
This research explores and compares various methodologies employed in thevisualization of taste experiences associated with food and beverages. The study delves
into sensory analysis techniques, technological innovations, and artistic
representations used to convey the complex and subjective nature of flavor. Through acomprehensive review of existing literature and practical experimentation, the paper
evaluates the efficacy of different approaches in capturing and communicating thenuanced aspects of taste, aroma, and texture. The findings aim to contribute to theadvancement of sensory science, culinary arts, and consumer research by providing
insights into the strengths, limitations, and potential applications of diverse taste
visualization methods. As the interdisciplinary field of gastronomy continues to
evolve, this research serves as a valuable resource for professionals, researchers, andenthusiasts seeking to enhance our understanding and appreciation of the multisensory
aspects of food and beverage consumption.

102©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

GS abstracts
GS1 Machine learning& Neural Network & Artificial Life (7)Chair Hiroki Tamura (University of Miyazaki, Japan)
GS1-1 Deep Learning Based Prediction of Heat Transfer Coefficient Using Spectrogram Images
from Boiling Sound
Fuga Mitsuyama, Ren Umeno, Tomohide Yabuki, Tohru Kamiya (Kyushu Institute of Technology, Japan)
Recently, cooling methods based on water boiling have attracted attention as a thermal
solution for electronic equipment. In this situation, it is necessary to measure Heat
Transfer Coefficient (HTC) to design more efficient cooling systems. In this paper, we
propose a method to predict of the HTC from boiling sound data using deep learningtechniques. Measuring HTC requires a variety of sensors and modifications to theequipment, which can be costly. Predicting HTC by sound alone can therefore reducethese costs. The accuracy improved by 1.12% compared to the conventional method
through the development of Convolutional Neural Network (CNN) incorporate
Convolutional Block Attention Module (CBAM).

GS1-2 A Study on Classification of Faulty Motor Sound Using Convolutional Neural Networks
Jamil Md Shafayet, Praveen Nuwantha Gunaratne, Hiroki Tamura (University of Miyazaki, Japan)
This paper is focused towards determining the use of audio classification methods todetect faulty motors by their sound, both with an abnormal noise and noise-freescenarios, so that the requirement of human inspection can be reduced in factories. Here,the soundwaves of motors, having both normal and faulty mechanical parts wereacquired using an electronic stethoscope. The audio files were recorded in ‘.wav’ format
which later split into chunks. Noise reduction is widely used in audio classification tasks.
Among various available methods, the Autoencoder was implemented for noise
reduction. Our proposal method is to use various soundwave analysis methods forgenerating features such as in Mel-Frequency Cepstral Coefficient, Mel-spectrogram,and Spectrogram, and comparing the resulting accuracy. Convolutional neural networks
are employed for the purpose of classification. Our proposed method resulted in approx.
97% accuracy in classifying 2 classes having 93 samples of motors and 2.23% loss innoise reduction with autoencoder.

GS1-3 Design and Development of a Flexible Active Ankle Joint Orthosis for Locomotion
Assistance
Praveen Nuwantha Gunaratne, Hiroki Tamura (University of Miyazaki, Japan)
Active ankle joint orthosis is an artificial wearable device that is closely fitted to thehuman ankle and combines human intelligence and intentions with the powered robot
joints. However, most of the existing designs have only focused on one degree of
freedom (DOF) of the ankle joint: namely plantarflexion-dorsiflexion. Alternatively, a
modular power-assist flexible active ankle orthosis is proposed, addressing
anthropomorphic architecture of design, and supporting multiple DOFs of ankle forlocomotion assistance of the physically weak individuals. The proposed ankle joint
orthosis was designed with the aid of CAD tools followed by fabrication of a working
model. Electrically powered systems with gear transmissions were adapted to support
the principal motions during normal Gait. In parallel, the design of the control algorithm
was carried out based on the EMG signals. The future advancements have been focusedon developing a novel control method that provides sufficient flexibility to assist a wide
variety of lower-limb motions.

103©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

GS1-4 A study on the Real-Time Biomechanical Analysis of Lamber Burden Utilizing
Stereoscope Cameras
Taufik Hidayat Soesilo, Praveen Nuwantha Gunaratne, Hiroki Tamura (University of Miyazaki, Japan)
Lamber region is susceptible to strain and stress due to various physical activities andoccupational task. To study about the lamber burden, researchers have developed many
tools but most of existing design can only use static imaging or doesn’t provide real-time
update. Our proposed system records and examines the change of body posture in realtime by utilizing the capabilities of stereoscope cameras. By using MediaPipe
algorithms, 2D keypoints representing body joint can be extracted from images.
Afterward, using the Direct Linear Transform (DLT) the corresponding 3D keypoints
can be calculated using obtained 2D keypoints. With the 3D keypoints the body angles
can be computed and used to calculate the wight of the lumbar burden using the digital
human model software JACK-like calculation. Finally, our proposed system reached toits aim to study about lamber burden and adjust the person needs in real-time.

GS1-5 Verification of Determination Possibility using Convolutional Autoencoder for Machine
Tool Abnormality Detection
Yuta Sumoto, Praveen Nuwantha, Hiroki Tamura (University of Miyazaki, Japan)
The purpose of this research is to clarify the cause of failure and to improve the accuracyof abnormality detection by predicting the noise added to the machine tool from theresults of the convolutional autoencoder. Data obtained from an acceleration sensor
mounted on the machine tool are reconstructed using a convolutional autoencoder, andthe average absolute error is calculated. The maximum value of the average absolute
error of the training data is used as the threshold value for abnormality detection.
Multiple simulated data with different amplitude values based on a composite sine wave
with a specific frequency, white noise, and random numbers within a specified amplitude
value were verified. In this paper, the characteristics of each type of noise and theparameters of the optimal model were examined from error rate and error distribution.

GS1-6 Basic Research for High-speed Heart Sound Determination using AI
Riku Nakashima, Praveen Nuwantha Gunaratne, Hiroki Tamura (University of Miyazaki, Japan)
In recent years, many researchers focus on enhancing the current situation of the homehealthcare system. In this paper, we developed a system that uses AI to quickly determine
whether auscultation sounds are normal or not. In our analysis, heart sounds wereimaged, and their abnormalities were identified using machine learning tools. The
proposal approach uses a YOLO_v7 model based anomaly detection and the Wavelet
Transform was employed to analyze the acquired heart sound data. Our proposal system
resulted with an 89% accuracy rate on the basis of 5 seconds of heartbeat data, in various
recording environments.

GS1-7 Prediction of High-Energy Electron Flux at Geosynchronous Orbit using a neuralnetwork technique
Ami Iwabu, Kentaro Kitamura (Kyusyu Institute of Technology, Japan)
The radiation belt, where the high-energy particles are predominant in near earth spacefrom the low earth orbit (LEO) to the geostationary orbit (GEO), sometimes causes
satellite malfunction. Therefore, the objective of this study is to predict the high-energy
electron flux at GEO with the energy above 2 MeV after 24 hours with higher accuracy
for the safety satellite operation in terms of the space weather science. In this study thevarious kinds of solar wind data from satellite observations and ground geomagnetic
observation data in 1999 were used for the Recurrent Neural Network (RNN). Prediction
results were evaluated by the prediction efficiency, which is derived from both predicted
and actual variation data. As a result, the prediction using combined data of solar windand geomagnetic data shows highest prediction efficiency of 0.72.

104©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

GS1-8 Image Gradient-based Monocular Visual-Inertial Odometry
Tae Ihn Kim1. Jae Hyung Jung2, Chan Gook Park2( Hyundai Motor Company, Republic of Korea), (2Seoul National University, Republic of Korea)
1This paper presents an image gradient-based monocular visual-inertial odometry (VIO)
algorithm, using image gradient measurements, robust to illumination change. Our
proposed algorithm follows the multi-state constraint Kalman filter (MSCKF)
framework, a sliding windowed extended Kalman filter (EKF). We expand themeasurements from the reprojected feature locations on the image coordinates to thecorresponding image gradients in the measurement selection. The iterated EKF and lowpass pyramid are adapted to reduce the linearization error in the MSCKF measurement
update process. We verify that our proposed algorithm outperforms both conventional
indirect and direct MSCKF-based VIO algorithms by evaluating the pose estimation
performance using a real-world dataset, including illumination change.

GS2 Image Processing I (5)Chair Yui Tanjo (Kyushu Institute of Technology, Japan)
GS2-1 A Method for Embedding Multiple Photographic Images in a Photographic Image
Naoki Kouno, Kanya Goto, Toru Hiraoka (University of Nagasaki, Japan)
A method has been proposed for embedding another photographic image (image B) in aphotographic image (image A). An image (image A') is generated by embedding
information of image B in image A, and then an image (image B') is restored byextracting information from image A'. The conventional methods can only embed one
photographic image in one photographic image. Therefore, we extend the conventional
method and propose a method for embedding multiple photographic images in one
photographic image. In our method, as more images B are embedding in image A, theimage quality of image A' deteriorates, but the image quality of images B' dose notdeteriorate. To verify the effectiveness of our method, experiments using various
photographic images were performed.

GS2-2 Generation of Flowing-Line Images Using Vertical and Horizontal Smoothing Filters
Karin Kuroki, Toru Hiraoka (University of Nagasaki, Japan)
We propose a non-photorealistic rendering method for automatically generating flowingline images from photographic images. Flowing-line images consist of unidirectionallines with smooth curves. Our method is executed by an iterative calculation using
vertical and horizontal smoothing filters. To verify the effectiveness of our method, we
conducted an experiment using various photographic images to confirm that flowingline patterns can be generated on the entire image. Additionally, we conducted anexperiment to visually examine how flowing-line patterns generated by changing thevalues of the parameters in our method change.

GS2-3 Human Motion Recognition from Multiple Directions and Its Gait Cycles Analysis
Miki Ooba, Yui Tanjo (Kyushu Institute of Technology, Japan)
It is crucial for individuals to keep walking and stay healthy to prevent receiving
nursing care. This paper proposes a method of recognizing walk motions andanalyzing the gait cycle of a human focusing on his/her posture. We use 43 structuralfeatures defined from human joint coordinates obtained using OpenPose and 18
figural features from human domain images and their difference images. The feature
vector containing these 61 features is used for the recognition of walk motion by
Random Forest. In the experiment, we applied the method to recognizing six typesof motions and analyzed the walk gait cycles of five persons, and obtained
satisfactory results.

105©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

GS2-4 A Method of Improving the QOL of the People with Visual Impairment by MY VISION
Shun Kitazumi, Yui Tanjo (Kyushu Institute of Technology, Japan)
Visually impaired people face several difficulties in indoor activities, such as spending
excessive time in locating objects. This paper proposes a method for assisting object
acquisition by detecting desired objects and guiding users to them. The method requests
a user to express the object he/she wants to acquire verbally and utilizes speech
recognition to detect the specified object. Subsequently, the system guides in voice theuser's hand to the location of the desired object. The performance of the method is
experimentally shown. The method contributes to enhancing the comfort of indoor
activities of visually impaired and, in this way, improves their quality of life.

GS2-5 Human Behavior Segmentation and Recognition Using a Single-camera
Jing Cao, Yui Tanjo (Kyushu Institute of Technology, Japan)
In recent years, elderly people living alone account for a large proportion of the elderly
population, and the issue of safety has also been a matter of great concern for the public.
Considering the importance of monitoring the behavior and activities of the elderly anddetecting abnormal movements, this paper proposes a method that can segment humanbehavior into each action and identify the action from the videos taken by a single
camera. It uses features that can represent the shape of the human area in the depth
direction, as well as the features such as motion direction and speed. The performanceand effectiveness of the method are verified by experiments.

GS3 Image Processing II (3)Chair Seiji Ishikawa (Kyushu Institute of Technology, Japan)
GS3-1 Online Classroom Student Engagement Analysis using Enhanced YOLOv5
Shuai Wang, Abdul Samad Shibghatullah (UCSI University, Malaysia)
The rise of online education has highlighted the urgency of addressing cyberbullying invirtual classrooms. This paper presents an innovative method for early cyberbullying
detection through the analysis of students' engagement and emotional responses in online
classrooms. The SFER-YOLOv5 model, a fusion of Student Facial Expression
Recognition and an enhanced YOLOv5 object detection model, introduces key
optimizations, including Soft NMS for Non-Maximum Suppression, integration of a
Channel Attention (CA) module, and use of Enhanced Intersection over Union (EIOU)
as the bounding box regression loss function. This approach proactively identifies
reduced engagement and emotional irregularities, providing a framework to mitigate
cyberbullying in online classrooms.

GS3-2 A Method of Recognizing Body Movements Based on a Self-viewpoint Video
Iichirou Moribe, Yui Tanjo (Kyushu Institute of Technology, Japan)
The most critical human sensory function resides in vision. This study focuses on
utilizing visual information, specifically self-perspective footage, to identify individual
movements. Existing researches require third-party filming to recognize body
movements and states. The proposed method, on the other hand, simply attaches acamera to the human head and enables the recognition of the subject's actions.
Consequently, it becomes easier to monitor daily movements of a human and gather
his/her data on body kinetics. This approach would be beneficial in scenarios involving
individuals engaging in risky behavior or, during a certain emergency, providing valuableassistance.

106©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

GS3-3 Supporting Safe Walk of a Visually Impaired Person at a Station Platform based on MY
VISION
Shintaro Yamada, Yui Tanjo, Seiji Ishikawa (Kyushu Institute of Technology, Japan)
When individuals with visual impairment go out, public transportation such as trains andbuses is commonly used. However, many of them experience accidents, such as fallingfrom train platforms or tripping due to unexpected contact with other passengers. To
solve this problem, we propose method using the MY VISION system which detects
the obstacle that may pose risks to individuals with visual impairment. The proposed
method detects obstacles such as passengers pillars and platform edges at train stations.
We employ an RGB-D camera for capturing frontal view of a user, use depth images todetect the edge of obstacles and level differences, and give warning to the visually
impaired user based on the distance between him/her and the detectedobstacle.
Experimental results show satisfactory performance of the method.

GS4 Robotics (6)Chair Jiwu Wang (Beijing Jiaotong University, China)
GS4-1 Development and evaluation of a learning support robot for vector learning
Kosei Machida, Shinichi Imai (Tokyo Gakugei University, Japan)
Vectors studied in high school are a new concept that differs from those studied inprevious scalars. Therefore, it has been pointed out that difficulties arise in theconceptual formation of vectors. This study aims to develop a learning support robot thatcan visualize vector information by robot movements, and to acquire vector concepts forstudents through classroom practice using the robot. The robot operates on a piece ofimitation paper, marking the start and end points and connecting them with arrows tovisualize the vectors. In a class using the robot, the student predicts the sum of thevectors, and the robot confirms it. A questionnaire was administered before and after theclass. The results revealed significantly higher mean values when comparing the preand post-assessments.

GS4-2 Feasibility Study on Methods to Measure the Strain on Young Children's Bodies.
Sachiko Kido, Praveen Nuwantha Gunaratne, Hiroki Tamura (University of Miyazaki, Japan)
This paper investigates the effective methods of measuring the burden on the infant's
body. Initially, the “the burden on the body and the lumbar spine burden are the same”
was defined. The calculations and comparisons were carried out based on the two
methods: the AnyBody Modeling System and the smartphone application Yo-bukun (alumbar spine burden measurement application). Videos of a five-year-old girl squatting
and balancing were filmed, and the lumbar burden was calculated using the AnyBody
Modelling System. In addition, while the video was being filmed, an iPhone running the
Yo-bukun lumbar burden measurement application was placed close to the heart in order
to measure lumbar burden and the amount of burden was calculated. The results werecompared with the respective average values and validated.

107©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

GS4-3 An Integration of Contact Force Models with Multibody Dynamics Analyses for Human
Joint Mechanisms and Effects of Viscoelastic Ground Contact
Shintaro Kasai1, Dondogjamts Batbaatar2, Hiroaki Wagatsuma1( Kyushu Institute of Technology, Japan) (2Mongolian University of Science and Technology, Mongolia)
1

In human movement and rehabilitation analyses, human joint dynamics is a key toconsider the incorporation of spring-damper components, flexible bodies and contact
forces analytically. In the present study, an analytical method for human gaits wereintroduced to integrate those essential elements, and viscoelastic properties ofmusculoskeletal system were modeled with the absolute nodal coordination formula(ANCF) method representing flexible body motions. A contact force model simulatesinteractions between different body segments and the environment. The proposed system
is applied to a slider crank mechanism, demonstrating its capabilities in human joint
motion analysis using integrated dynamic model within the framework of multibody
dynamics (MBD), which realizes dynamic/inverse dynamics for human biomechanics.

GS4-4 Haptic Sensation Enhancement via the Stochastic Resonance Effect and Its Application
to Haptic Feedback for Myoelectric Prosthetic Hands
Yoshitaka Mizumoto, Taro Shibanoki (Okayama University, Japan)
In this paper, we propose a highly realistic haptic feedback method for myoelectric
prosthetic hands. Although several haptic feedback methods have been studied, thisstudy attempts to create realistic sensory feedback through prosthetic hands by not only
using feedback methods but also by improving the haptic sensation based on thestochastic resonance effect. In the experiment, contact information obtained from a
microphone attached to the fingertip of a prosthetic hand was transferred through avibrotactile stimulator near the elbow socket, and white noise vibration was also applied
near the elbow socket to verify the improvement of tactile sensitivity of the fingertips.
The results demonstrated the possibility of transmitting tactile information for
myoelectric prostheses through sensory enhancement of the user.

GS4-5 PID Parameter Tuning of a Low-Cost DC Motor Speed Control for Mobile Robot
Application
Munkh-Erdene Ayurzana, Erkhembayar Gankhuyag, Naranbaatar Erdenesuren, Dondogjamts Batbaata(Mongolian University of Science and Technology, Mongolia)
Precise control of DC motors is vital for robotics, industrial automation, andmechatronics. Traditional PID (Proportional-Integral-Derivative) control methods, while
widely used, often require offline parameter tuning, which can be time-consuming andsuboptimal for real-time applications. This paper proposes a GUI (Graphical User
Interface) based approach for PID tuning of a DC motor model. The proposed method
utilizes a MATLAB-based parameter estimation model with GUI to continuously
monitor and update PID controller parameters based on real-time data from the Arduinocontrolled DC motor setup.

GS4-6 Reinforcement Learning DDPG Algorithm Based Wheeled Mobility Aid Robot Control
Methods
Junkai Li, Mohd Rizon Mohamad Juhari, Tiang Sew Sun (UCSI University, Malaysia)
Ensuring stability and precise trajectory tracking is crucial when employing wheeled
walker robots to enhance mobility for individuals with limited walking ability. We
propose a novel trajectory tracking method for wheeled walking robots by combining
the Deep Deterministic Policy Gradient (DDPG) algorithm in reinforcement learningwith a Proportional Integral Differential (PID) controller. We verify the effectiveness of
the research scheme and control strategy through joint simulation experiments. The
results demonstrate that the DDPG-based PID controller can automatically adjust
parameters to ensure trajectory accuracy and exhibits a strong anti-interference
capability.
108©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

GS5 Applications I (4)Chair Jiwu Wang (Beijing Jiaotong University, China)
GS5-1 Unsupervised image registration based on Residual-connected DRMINE for diagnostic
metastatic bone tumors
Shogo Baba1, Tohru Kamiya1, Takashi Terasawa2, Takatoshi Aoki2( Kyushu Institute of Technology, Japan), (2University of Occupational and Environmental Health, Japan)
1Computed tomography (CT) scans produce more than 100 images in an examination,
which imposes a significant burden on radiologists and entails a potential risk of
misdiagnosis. In this study, we focus on the preliminary stage of Computer-Aided
Diagnosis (CAD) development specialized in bone metastasis extraction, with aparticular emphasis on rigid registration. We propose a novel rigid registration technique
by augmenting DRMINE, which estimates mutual information using neural networks,
with skip connections and normalization. The proposed method was influenced by thecapture area, but it indicated the potential to provide stable registration as the standard
deviation decreased for all Full Width at Half Maximum (FWHM).

GS5-2 Developing a smart Belt for Monitoring Elderly Activities Based on Multi-Modal Sensors
Integration and Internet of Things
Abdul Jalil, Pujianti Wahyuningsih, Najirah Umar, Muhammad Risal, Suwatri Jura, A. Edeth Fuari Anatasya(Universitas Handayani Makassar, Indonesia)
This study aims to develop a smart belt for monitoring elderly activities at home basedon the integration of multi-modal sensors and the Internet of Things (IoT). The multimodal sensors used in this study to collect the elderly information are the IMU sensor,
vibration sensor, push-button, and ESP32 to process the data. Furthermore, the IoTframework used to transmit the elderly information data from the smart belt to thefamily’s smartphone is a Blynk. In this study, the smart belt can monitor the elderly
activities when walking, sitting down, lying down, and sleeping, after that, can give asecurity warning when the smart belt detects the elderly doing abnormal activities. This
study's results show that the smart belt is effectively used to monitor elderly activities tohelp families when taking care of the elderly at home.

GS5-3 A penalized motion detection model for extracting ionospheric echoes from low signal-tonoise ratio Ionogram video images
Yuu Hiroshige1, Akiko Fujimoto1, Akihiro Ikeda2, Shuji Abe3, Akimasa Yoshikawa3( Kyushu Institute of Technology, Japan) (2National Institute of Technology, Kagoshima College, Japan)(3Kyushu University, Japan)
1

Measuring the altitude distribution of electron density in the upper atmosphere, known
as the ionosphere, using High-Frequency radio wave reflections often causes the low
signal-to-noise ratio of ionospheric echoes due to radio frequency interference. We
propose a model for converting low-signal-to-noise-ratio ionospheric echo video images(Ionogram) into noise-reduced images using image processing techniques, for tracing
the ionospheric echoes from Ionogram. The proposed method consists of three
processing parts: noise removal optimized for individual Ionogram images, extraction ofionospheric echoes by penalized background subtraction technique, and fine-tuning of
ionospheric echo signals using a minimum spanning tree algorithm. The proposed model
successfully reproduces fine Ionograms with 98% recall and 99% precision.

109©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

GS5-4 Verification Experiments on the Lower Back Burden caused by Posture and Environment
during Lifting Operations
Tomoka Kimura, Yutaro Fujino, Sachiko Kido, Praveen Nuwantha Gunaratne, Hiroki Tamura(University of Miyazaki, Japan)
One of the measures to prevent back pain is the use of appropriate posture corrections.
In generally, the Squat method (method of keeping the knees bent and the waist as
straight as possible) is recommended over the Stoop method (method of keeping theknees straight and the waist bent). The results of previous studies using acceleration andmuscle potentials have shown that the lifting with knees in a kneeling position reduces
the amount of load born by the lower back. However, few studies discuss the case where
there is an obstacle between the subject and the object to be lifted and further, thescenarios where the object is away from the body's center of gravity. Therefore, thisresearch focus on analyzing two types of movements using the AnyBody
musculoskeletal mechanics analysis software and the Delsys surface EMG and verifying
the amount of lower back burden when there is an obstacle between the lifting object
and the subject. This paper presents the verification results.

GS6 Applications II (4)Chair Masayuki Fujiwara (Kyushu Institute of Technology, Japan)
GS6-1 Development of Smartphone Application for Calculating the Low Back Pain Risk
Seigo Imura, Praveen Nuwantha, Hiroki Tamura (University of Miyazaki, Japan)
This study primarily relies on a smartphone application, developed within our research
institute known as Yo-bukun, for the real-time estimation of lumbar burden. The Yobukun is capable of estimating the lumbar burden of a subject (the user) by placing anapp. installed smartphone in the subject’s chest pocket, while the subject is lifting/relocating an object. The subject's movements are assessed through sensors embeddedin the smartphone and certain aspects of their physical information initially fed into theapp. are used in the estimation formula for determining lumbar burden. In the current
scenario, Yo-bukun lacks the capability to ascertain whether the user is holding an object
or not; consequently, it can only estimate the lumbar burden for limited cases of a subject
holding an object. To address such limitations, the proposed system integrates voicerecognition to facilitate lumbar burden estimation, considering the presence or absenceof an object. Further, it was incorporated with the capability to recalculate lumbar burden
after measurements, enabling prospective studies.

GS6-2 A Computational Approach for Global Trade Analysis Sensitive to Free Trade Agreement
Circumstances: A Case Study Focusing on the Great Mekong Subregion
Ahmad Altaweel1, Bo-Young Lee2, Masayuki Fujiwara1, Jang-Sok Yoon2, Hiroaki Wagatsuma1(1Kyushu Institute of Technology, Japan) (2Logistics Revolution Korea, Korea)
In the Global Trade Analysis Project (GTAP), GDP and economic statistical indices canbe analyzed for forecasting future trends of them in multiple countries by using the GTAP
database and GEMPACK utilities, which realize the numerical calculation based on theconcept of Computable General Equilibrium (CGE) models. Even by such well-prepared
tools with the official database, an appropriate forecasting is still difficult due to thesensitivity to Free Trade Agreement (FTA) circumstances. FTA scenarios with a uniform
tariff reduction can be assumed in general, while an abrupt and unreasonable change
may occur in the simulation depending on the network structure of trading countries andupper and lower bounds of tariffs in the time course. In the present study, we focused on
the Great Mekong subregion (GMS) and explored possible methods to calculate
substantially.

110©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

GS6-3 Trigger circuit design and system integration for simultaneous measurement of human
EEG, motion, and gaze
Masayuki Fujiwara1, Phan Hoang Huu Duc1, Laurent Bougrain2, Patrick Hénaff2, Hiroaki Wagatsuma1(1Kyushu Institute of Technology, Japan) (2Université de Lorraine, France)
The simultaneous measurement of human EEG, motion, and gaze has the potential tolead to the discovery of new scientific insights. In order to achieve these simultaneous
measurements, it is necessary to manage triggers and time information between
measurement devices, as well as to correct time offsets. However, the management of
accurate triggers and time information requires the design of a dedicated circuit board
and the integration of TTL signal voltage information. In this study, we report on the
fabrication of a trigger circuit and an experimental system using it to solve these
problems. We created a home-made trigger circuit board for voltage integration andcombined it with a commercially available microcomputer to realize an integrated trigger
circuit and measurement system.

GS6-4 Terminal Synergetic Controller for Car’s Active Suspension System Using Dragonfly
Algorithm
Tinnakorn Kumsaen1, Sorn Simatrang2, Arsit Boonyaprapasorn3, Thunyaseth Sethaput4( Khon Kaen University, Thailand) (2Nacres Co., Ltd, Thailand) (3Chulachomklao Royal Military Academy,
Thailand) (4Thammasat University, Thailand)1This research introduces a terminal synergetic controller (TSC) designed for the active
suspension system of automobiles through the implementation of the dragonfly
algorithm (DA). The proposed controller aims to enhance the dynamic performance of acar's suspension using the DA in tuning the system parameters. The stability of thedesigned controller is proved through the application of Lyapunov stability theory.
Through iterative optimization processes, the TSC approach seeks to achieve an optimalbalance between ride comfort and vehicle handling. The simulation results demonstratethat the proposed controller enhances convergence properties and alleviates the presenceof chattering. The results indicate that the proposed approach with the optimal
parameters provided insights into its potential application in improving the overallsuspension system.

GS7 Applications III (6)Chair Kasthuri Subaramaniam (UCSI University, Malaysia)
GS7-1 Rehabilitating Flood-Damaged Cars for Sustainable Car Rental Services: A WebBased System
Pon Xiao Qi, Abdul Samad Shibghatullah, Kasthuri Subaramaniam (UCSI University, Malaysia)
The web-based system aims to redefine transportation norms by unlocking the potentialof flood-damaged cars. It seeks to reutilize these cars, providing affordable andsustainable transportation options while minimizing landfill waste. The systemdevelopment approach that the researcher used for this system is Rapid Application
Development (RAD) Model. The reason that the researcher chose this methodology is
because it enables her to develop the system under limited time while ensuring thequality of the system. There is four stages in Rapid Application Development (RAD)methodology life cycle, which is requirement planning phase, user design phase, rapid
construction phase and cutover phase.

111©ICAROB 2024 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

GS7-2 Optimizing E-Invoicing Rollout: Adaptive E-Invoicing Rollout (AER) Framework for
Navigating Malaysia's Digital Transformation
Koh Chee Hong, Abdul Samad Shibghatulla (UCSI University, Malaysia)
This study presents the Adaptive E-Invoicing Rollout (AER) Framework, developed to
guide Malaysia's impending e-Invoicing mandate. It is crafted through a meticulous
literature review and expert consensus, highlighting crucial variables such as integration
capability, adoption rate, readiness, and compliance. These elements are central to thepredictive model, designed to forecast and facilitate successful e-Invoicing
implementation. The findings from our extensive review and model application confirm
the framework's effectiveness in diverse business environments, demonstrating its
adaptability and strategic value. The AER Framework significantly contributes to digitalfinance strategies, providing an innovative, empirically based tool for not only Malaysia
but also other regions in the globe pursuing digital economic transformations.

GS7-3 App Alert System for Smart Phones
Chee Kin Hoe, Kasthuri Subaramaniam, Abdul Samad Shibghatullah (UCSI University, Malaysia)¥

This development is to develop a porotype mobile app that notifies the user to take theirphones after shutting down the car. This development is being done to gather feedback
from participants who frequently forget to take their phones such as dementia sufferers,
careless youngsters and users who park in a dark space. As for the results of the survey,
it was extensively researched and analyzed to find out the problems and requirements ofusers for the app. For the methodology, after extensive consideration the researcher
decided to use Modified Waterfall model a derivative from the famous waterfall model.
The main factors that influenced the decision is the suitability of the methodology, theconfidence of the researcher to use the methodology and the time constraint to develop
the app. There are 6 phases in the modified waterfall model which are Requirement
Analysis, System Design, Implementation, Testing, Deployment and Maintenance. Afterdeveloping the app, the user acceptance test was done to see whether how the generalpublic would accept the app. For the result, the app was accepted by most of theparticipants and they would like the researcher to do further development to improve andmaintain the app. Based on the results of the development the researcher can conclude
that this app can benefit users in helping them to remember to take their phones fromtheir car. Rather than using the old fashion way which is remember to take the phone.
With this app, user can rest assured as they will be always be reminded to take theirphones from their car.

.

GS7-4 Developing Hand Gesture Recognition System in Interpreting American Sign Language
Kong Seh Chong, Kasthuri Subaramaniam, Ismail Ahmed Al-Qasem Al-H (UCSI University, Malaysia)
This research is going to study and gather more information on the hand gesturerecognition system in order to bridge the communication between non-signers andsigners by implementing the sign language into the system. The system may help people
with physical disabilities such as deaf, mute and etc, to have an efficient and directcommunication with others without the needs of interpreters. Through the studying of
the hand gesture recognition system, there are two main frameworks which are OpenCV
and MediaPipe that hold significant value in the system as it is mostly used in computer
vision related problem. With this system developed, it can help many signers to avoid
the problem of indirect interaction and also enable those who want to learn sign language
by practicing hand gesture through the system. The system development approach thatthe researcher used for this system is Agile Methodology. The reason that the researcher
chooses this methodology is because the development of this system mostly deals withthe datasets as well as the accuracy of the detection and recognition of hand gestures.
Hence, an ability to adapt to changes of agile methodology is needed if there are any
problems happen in the system. There are seven of the stages in Agile Methodology life
cycle, which are planning phase, designing phase, implementation phase, testing phase,
deployment phase, review phase and launch phase. All these seven phases of the Agile
methodology will be having some deliverables as their output.
112©ICAROB 2024 ALife Robotics Corp. Ltd.

.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

GS7-5 Miniature Enterprise Resource Planning
Adim Khalid Aldireejah, Kasthuri Subaramaniam, Ghassan Saleh (UCSI University, Malaysia)
Organizations aim for maximum efficiency, acquiring a competitive edge, and quickly
completing their goals in today's competitive business environment. The achievementof these business objectives depends heavily on enterprise resource planning (ERP).
ERP is a software program that combines several elements needed for a company's
administration and operation, spanning several sectors like supply chain management,
manufacturing, and finance. ERP makes work easier, saves time and money, and gives
managers operational insights by automating corporate activities within a single
platform. As a result, ERP has become essential for businesses of all sizes that want toadvance and maintain their position as industry leaders. This research proposal intends
to analyze the key traits, implementation concerns, and challenges associated with ERP
systems while also outlining potential fixes. By creating a convenient and affordablemini-ERP solution, the initiative further intends to increase the accessibility of ERP
systems to small and medium-sized enterprises (SMEs). The study of current ERP
systems, their development, and their advantages and disadvantages will form the basisof the research. It will also investigate the difficulties SMEs have in using common ERP
solutions.

GS7-6 Developing Cloud-based Sportswear Website
Lim Wei Yee, Kasthuri Subaramaniam, Raenu Kolandaisamy (UCSI University, Malaysia)
This research is going to develop a cloud-based sportswear website that can handle highvolumes of traffic during events or seasonal sales, assist customers in selecting the rightsize for their sportswear, the interface should have options, size charts, fitting
instructions, and recommendations. A study on the existing system will be conducted,
design, develop and evaluate on the sportswear website also will be carried out. The
system development approach used on this website is Rapid Application Development(RAD) model. The reason to choose RAD is because this method is time efficient,flexibility and adaptability. With RAD, functional software is delivered quickly. We canquickly build prototypes, iterate on them, and gather feedback from stakeholders with a
website project. There are four of the steps in Rapid Application Development (RAD)methodology life cycle, which is define the requirements, prototypes, rapid construction
and feedback gathering, and cutover.

113©ICAROB 2024 ALife Robotics Corp. Ltd.

.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

AUTHORS INDEX
Notation of session name
PS: Plenary Session IS: Invited Session, OS: Organized Session, GS: General Session
Note: 33/90 = (page no. in Technical Paper Index) / (page no. in Abstracts)[A]
A.

RidzuanAltaweel

Ahmad

GS6-2

42/110

Amano

Koki

OS20-1

29/82

OS20-2

29/82

OS6-4

43/61

OS7-1

31/62

OS7-2

31/62

AncillaGabriela MariaOS18-5

40/79

OS7-3

31/62

Ang

Aun Jie

OS25-1

47/93

OS7-4

31/63

Ang

Chun Kit

OS26-7

46/99

OS26-9

46/99

A.B.

Ayu Haslija

OS25-6

48/95

A. Edeth Fuari

Anatasya

GS5-2

36/109Ang

Koon Hian

OS26-7

46/99

AbbyasovBulat

OS11-4

47/69

Ang

Koon Meng

OS26-7

46/99

Abdullah

Nor Fazilah

OS25-9

48/96

Aoki

Takatoshi

GS5-1

36/109

Abe

Shuji

GS5-3

37/109

Ahmed

Ismail

OS18-7

40/79

GS5-2

36/109Arof

HamzahOS21-342/87

OS25-2

48/94

Ashra

Dina

OS6-5

43/61

OS25-4

48/94

Ayurzana

Munkh-Erdene

GS4-5

24/108

OS9-2

22/65

AzizanMuhammadOS23-1

24/90

OS9-3

22/65

OS9-4

22/65

OS23-2

24/90

Abdul Jalil
Abdullah

Nor Fazilah

AburadaKentaroAzizi

Aiman Rosol

Ahmad Haziq

OS21-342/87

OS23-3

24/91

Al-HadiIsmail Ahmed

GS7-4

49/112OS23-4

24/91

Al- Ismail Ahmed

OS18-5

40/79

OS23-524/91

Al shaikh

Yazan Abu

OS25-3

48/94

Al-Talib

Ammar A.

OS25-1

47/93

B.

Shakthi

OS18-3

40/78

OS25-2

48/94

Baba

Shogo

GS5-1

36/109OS25-3

48/94

Bari Shital

SabrinaOS22-7

38/89

OS25-448/94

Batbaatar

Dondogjamts

GS4-3

24/108OS25-5

48/95

GS4-5

24/108OS25-6

48/95

Bi

YanchaoOS13-1

26/71

OS25-7

48/95

Biki

Bidesh Biswas

OS22-6

38/89

OS25-948/96

OS22-7

38/89

Al-Qasem
Hadi[B]

Aldireejah

Adim Khalid

GS7-5

49/113

Boonyaprapasorn

Arsit

GS6-4

42/111Ali

HashimahOS21-2

41/87

Bougrain

Laurent

GS6-3

42/111Ali

Nur Izzati

OS25-8

48/96

Alexeev

Alexander

OS11-1

47/68

[C]
114

©ICAROB 2023 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

Cao

Jing

GS2-5

23/106OS23-2

24/90

Celiker

Baris

OS10-1

28/66

OS23-3

24/91

Chang

Heng-En

OS5-3

39/59

OS23-4

24/91

Chang

Meng-Wei

OS1-4

21/52

OS23-5

24/91

Chang

Yun-Hao

OS1-2

21/51

DhanushR.

OS18-2

40/78

OS1-4

21/51

Dobrokvashina

Alexandra

OS10-6

28/67

OS4-8

44/57

Du

Jiacheng

OS26-2

45/97

OS4-9

44/58

Duan

SuqingOS13-3

26/71

OS13-6

26/72

Chang

Zhencheng

Chebotareva

ElviraOS11-3

47/68

Chen

Chih-Hung

OS1-6

21/52

Dudorov

Evgeny

OS5-6

39/60

Chen

Nai-Yu

OS1-1

21/51

DuvacherMaxime

OS6-3

43/61

OS1-5

21/52

Martínez-

OS5-5

39/59

OS11-1

47/68

Chen

Yao

OS26-2

45/97

[E]

Chen

Yi-Wei

OS5-1

38/58

Edgar A.

Chen

Yu-Le

OS5-2

38/58

Cheng

Jiale

OS13-1

26/71

ChiongMeng Choung

OS26-10

46/100Elshaikh

Mohamed

OS21-2

41/87

Chong

Kim Soon

OS6-2

43/60

Elamvazuthi

Irraivan

OS6-4

43/61

OS26-3

45/97

Era

Junya

OS20-1030/84

OS26-445/98

Erdenesuren

Naranbaatar

GS4-5

24/108OS26-545/98

EreminAlexander

OS10-5

28/67

OS26-645/98

Eryomin

Alexander

OS5-6

39/60

OS26-7

46/99

OS26-8

46/99

García[F]

Chong

Kong Seh

GS7-4

49/112

FadillaReno

OS14-336/74

Chong Jun

Esther

OS18-7

40/79

Faizullin

Ramil

OS5-5

39/59

Fan

JingzhenOS26-11

46/100Lynn
Chou

Pei-WenChuangLi-Min

OS1-1

21/51

Fan

WenzhuoOS26-1

45/97

OS1-5

21/52

Fu

Wenqi

OS13-5

26/72

OS1-6

21/52

Fuengfusin

Ninnart

OS15-1

22/74

Fujimoto

Akiko

GS5-3

37/109Fujinaga

TakuyaOS20-14

31/86

OS20-15

31/86[D]
D. AhmadAeryn

OS21-342/87

DaiFengzhi

OS2-1

25/52

FujinoKazumaOS12-1

34/70

OS2-3

25/53

FujinoYutaroGS5-4

37/110Fujiwara

MasayukiGS6-2

42/110

GS6-3

42/111

OS20-1030/84

OS20-1130/85

Daimaru

Yoshiyuki

OS20-930/84

DarusmanAbu Hassan

OS6-4

43/61

Desa

Hazry

PS2

20/50

OS23-1

24/90

Furukawa

115©ICAROB 2023 ALife Robotics Corp. Ltd.

Tomoyasu

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

Fxyma

HirokiOS27-533/102[G]
Gamberov

Hasebe
Timur

Kodai

OS25-948/96

OS26-10

46/100OS22-6

38/89

OS10-328/66

OS22-7

38/89

OS11-4

47/69

OS22-838/90

OS17-2

32/77

OS17-3

32/77

Gankhuyag

Erkhembayar

GS4-5

24/108

Hattori

ShimonGao

YuxuanOS24-3

20/92

Gong

Haoran

OS3-8

27/55

HattoriMai

OS17-5

32/78

OS3-9

28/55

Hattori

TetsuoOS17-2

32/77

Gong

Ying

OS4-3

44/56

OS17-332/77

GonzalesAndrea Tantay

OS18-1

40/78

OS17-4

32/77

Goto

Kanya

GS2-1

23/105OS17-5

32/78

Gunaratne

Praveen

GS1-2

34/103

OS16-1

39/75

GS1-3

34/103

OS16-2

39/76

GS1-4

35/104OS16-3

39/76

GS1-5

35/104OS16-4

40/76

GS1-6

35/104OS20-12

30/85

GS4-2

23/107OS16-2

39/76

GS5-4

37/110OS16-4

40/76

GS6-1

42/110HayashiToshihiro

OS17-5

32/78

OS5-1

38/58

He

Hao

OS2-3

25/53

OS3-3

27/54

OS3-4

27/54

Guo

Jr-Hung

HayashiHayashi

Eiji

Takamasa[H]
Hamakawa

Hamasuna

FumitoOta

OS22-5

37/89

HénaffPatrick

GS6-3

42/111

OS22-6

38/89

Hidayab

Muzaiyanah

OS6-2

43/60

OS22-7

38/89

Hieda

Hiroo

OS12-1

34/70

OS22-838/90

Hiroshige

Yuu

GS5-3

37/109OS22-5

37/89

Hiramitsu

Tatsuhiro

OS14-336/74

OS22-6

38/89

Hiraoka

Toru

GS2-1

23/105

GS2-2

23/105

Haque

Ashraful

OS25-8

48/96

HaradaKensukeOS14-1

36/73

Hoe

Chee Kin

GS7-3

49/112OS14-2

36/73

Hoh

Ian

OS25-6

48/95

OS7-1

31/62

Hong

Koh CheeGS7-2

49/112

OS7-2

31/62

Hong

L. Y.

OS23-3

24/91

OS7-3

31/62

Hori

Yoshiki

OS16-1

39/75

OS7-4

31/63

Hsia

Kuo-Hsien

OS5-2

38/58

OS25-1

47/93

OS11-6

47/69

OS25-4

48/94

Hu

Mengyuan

OS4-5

44/57

OS25-5

48/95

Huang

Mei

OS8-5

36/64

HassanCik Suhana

116©ICAROB 2023 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

Huang

JiaweiOS20-4

29/83

Huang

YuankuiOS26-11

Huang

Yumei

Ito

Takao

OS21-1

41/86

46/100OS22-4

37/89

OS3-8

27/55

OS22-7

38/89

OS3-9

28/55

OS22-8

38/90

OS4-8

44/57

OS22-7

38/89

OS4-9

44/58

OS22-8

38/90

Ami

GS1-7

35/104Farah Adilah

OS25-1049/96

OS26-10

46/100Ito

Tsutomu

Huang

Zehan

OS26-2

45/97

Hung

Chung-Wen

OS5-3

39/59

OS5-4

39/59

[J]

OS23-4

24/91

Jamaludin

OS23-5

24/91

OS25-8

48/96

Hussain

HussinA. -S. T.

Eryana

Iwabu

[I]

Jamil

Md Shafayet

GS1-2

34/103JelbanAhmad

OS25-4

48/94

Jia

Yingmin

OS8-1

35/63

OS8-4

36/64

Ikeda

AkihiroGS5-3

37/109

Ikeda

SatoshiOS22-4

37/87

Jiao

Jianhao

OS4-4

44/56

OS22-5

37/89

Jothi

Neesha

OS18-1

40/78

OS22-6

38/89

OS18-7

40/79

OS22-738/89

Jae HyungGS1-8

35/105OS22-838/90

IndraahOS18-8

41/80

Jung

Imai

ShinichiGS4-1

23/107[K]

Imai

YoshiroOS17-5

32/78

Imura

Seigo

GS6-1

42/110OS18-9

41/80

Ishak

Nurfadzillah

OS23-1

24/90

OS18-1041/80

OS23-2

24/90

OS18-11

41/81

K.

Ishigami

Ryoichi

OS9-1

22/64

Kaishita

WataruOS20-10

30/84

Ishii

Kazuo

OS20-3

29/82

Kakinoki

Kanta

OS9-4

22/65

OS20-429/83

KamodaYohei

OS27-3

33/101

OS20-5

29/83

Kamei

Keiji

OS20-930/84

OS20-630/83

Kamiya

Tohru

GS1-1

34/103OS20-13

31/85

GS5-1

36/109OS20-14

31/86

KanaokaDaiju

OS15-4

23/75

Kang

Jinwu

OS24-421/93

OS24-5

21/93

Ishikawa

Seiji

GS3-3

25/107Ishiwata

Yoshinobu

OS27-2

33/101

OS27-433/101

Kasai

Shintaro

GS4-3

24/108Islamova

RenataOS11-2

47/68

Katayama

Daigo

OS20-1431/86

IsomotoKosei

OS15-3

22/74

Katayama

Tetsuro

OS9-1

22/64

OS15-4

23/75

OS9-2

22/65

OS12-4

34/70

OS9-3

22/65

Isozaki

Hideo

117©ICAROB 2023 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

OS9-4

22/65

Kong

De Kang

OS25-1049/96

Kawaguchi

Akira

OS16-3

39/76

Koshiba

MamikoOS12-1

34/70

Kawahara

Tomorou

OS20-930/84

KotaniNaoki

OS12-3

34/70

Kawakami

YusukeOS17-2

32/77

Kouno

Naoki

GS2-1

23/105OS17-3

32/77

KoyamaYoshihiro

OS16-3

39/76

OS17-4

32/77

Kuan

Chyntia

OS25-7

48/95

Kudo

AkihiroOS22-2

37/88

OS22-3

37/88

Kawashima

MizukiOS15-6

23/75

Khan

M.K.A Ahamed

OS6-1

43/60

OS6-2

43/60

Kumar

Prashant

OS14-2

36/73

OS6-3

43/61

Kumsaen

Tinnakorn

GS6-4

42/111

OS6-4

43/61

Kuremoto

Shun

OS12-1

34/70

OS6-5

43/61

Kuremoto

TakashiOS12-1

34/70

OS7-1

31/62

Kurogi

ShuichiOS22-5

37/89

OS7-2

31/62

Kuroki

Karin

GS2-2

23/105

OS7-3

31/62

Kyasudeen

MuhammadOS21-1

41/86

OS7-4

31/63

OS26-946/99

OS26-10

46/100

GS4-2

23/107

Lai

Chun-Chi

OS5-2

38/58

GS5-4

37/110Lai

Wei Kang

OS26-3

45/97

Kido

Sachiko

Farris[L]

Kim

Tae Ihn

GS1-8

35/105OS26-445/98

KimuraKenji

OS20-630/83

OS26-545/98

OS20-730/83

OS26-645/98

OS20-830/84

OS10-6

28/67

OS11-547/69

Lavrenov

Roman

KimuraTomokaGS5-4

37/110Kita

Yoshihiro

OS9-2

22/65

Lee

Bo-Young

GS6-2

42/110

OS9-3

22/65

Lee

L. J.

OS23-5

24/91

OS9-4

22/65

Li

Chu-Fen

OS1-3

21/51

Kitamura

KentarouGS1-7

35/104

Li

Fangyan

OS3-1

27/53

Kitazumi

Shun

GS2-4

23/106

Li

Guandong

OS26-11

46/100Koh

Pei Cong

OS18-6

40/79

Li

Haibo

OS13-826/72

Kolandaisamy

Raenu

GS7-6

49/113

Li

Hongbing

OS11-427/69

OS18-841/80

LiHuahao

OS4-1

44/56

OS18-941/80

LiJung-Shian

OS1-1

21/51

OS18-10

41/80

OS1-2

21/51

OS18-11

41/81

OS1-3

21/51

OS18-12

41/81

OS1-4

21/51

OS1-5

21/52

GS4-6

24/108Kolin

NikitaOS11-3

47/68

Komori

RyoheiOS20-12

30/85

Li
118

©ICAROB 2023 ALife Robotics Corp. Ltd.

Junkai

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

Li

Lin

OS8-5

36/64

Li

Mingxuan

OS4-8

44/57

OS4-9

44/58

M NoorMuhammadOS21-1

41/86

OS25-1

47/93

Badri
M.Tahir

Noor Idayu

Li

Mingyue

OS4-4

44/56

OS25-2

48/94

Li

Na

OS24-4

21/93

OS25-3

48/94

Li

Zixin

OS24-3

20/92

OS25-4

48/94

Liang

Xin

OS13-426/71

OS25-5

48/95

Lim

Tiong HooOS26-8

46/99

OS25-6

48/95

OS26-946/99

OS25-7

48/95

OS6-1

43/60

OS13-9

26/73

OS6-3

43/61

OS13-1027/73

OS26-245/97

MaShuangshuang

OS2-1

25/52

OS26-3

45/97

Mabu

ShingoOS12-1

34/70

OS26-445/98

Machida

Kosei

GS4-1

23/107OS26-5

45/98

Magid

Evgeni

OS5-5

39/59

OS26-6

45/98

OS5-6

39/60

OS26-7

46/99

OS10-2

28/66

OS26-8

46/99

OS10-428/67

OS26-946/99

OS11-1

47/68

OS26-10

46/100OS11-4

47/69

OS26-11

46/100OS11-647/69

Lim

Wei Hong

Ma

DongpoLim

Wei YeeGS7-6

49/113

Mahmoud E.A.

Zeiad

OS25-5

48/95

Liu

Chun-Liang

OS5-4

39/59

Marco

RozilynOS14-2

36/73

Liu

Fangyv

OS2-3

25/53

MatobaOsamu

OS17-2

32/77

OS3-3

27/54

OS17-3

32/77

OS3-4

27/54

Matsumoto

Sho

OS9-1

22/64

OS1-1

21/51

Mazher

Moona

OS7-1

31/62

OS1-2

21/51

OS7-2

31/62

OS1-3

21/51

OS7-3

31/62

OS1-4

21/51

OS7-4

31/63

OS1-5

21/52

OS3-5

27/54

Liu

I-HsienMei

YupingLiu

Mengda

OS4-7

44/57

OS3-6

27/55

Liu

Yi-Ting

OS5-2

38/58

OS3-7

27/55

Liu

Yufei

OS20-3

29/82

Meng

TianbinOS24-1

20/92

Lu

Yuhao

OS26-11

46/100

MinatoGaku

OS20-2

29/82

Lv

Lei

OS13-4

26/71

Mirzoian

Viktoriia

OS10-4

28/67

Mitsuyama

Fuga

GS1-1

34/103

Mizoe

Eiji

OS20-1030/84

OS20-11

30/85

[M]

119
©ICAROB 2023 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

Mizutani

Akinobu

OS15-2

22/74

Nakamura

Akira

OS14-1

36/73

OS15-423/75

NakamuraJun

OS27-2

33/101

Mizumoto

YoshitakaGS4-4

24/108OS27-3

33/101

Mo

Lipo

OS8-4

36/64

OS27-433/101

Mohamed

Rizon

GS4-6

24/108Nakashima

Riku

GS1-6

35/104

MohammadIlyas

OS26-3

45/97

NarayananGanesh

OS18-440/79

Niederer

Steven

OS7-1

31/62

Arif

Mokayef

MokhtarOS26-4

45/98

OS7-2

31/62

OS26-5

45/98

OS7-3

31/62

OS26-6

45/98

OS7-4

31/63

OS6-1

43/60

NishidaNaoya

OS20-11

30/85

OS6-2

43/60

NishidaYuya

OS20-13

31/85

OS6-3

43/61

Noor

Rayan

OS26-8

46/99

OS6-4

43/61

Mohammed

Mohammed

OS6-5

43/61

Bakhit

OS7-1

31/62

Nur

Afifi

OS25-6

48/95

OS7-2

31/62

Muhammad

OS7-3

31/62

OS7-4

31/63

OS26-2

45/97

Ogata

Takashi

OS27-133/100OS26-3

45/97

Ogawa

Shintaro

OS16-2

39/76

OS26-645/98

OS16-4

40/76

OS18-6

40/79

Ohno

Asako

OS17-5

32/78

OS21-1

41/86

Ohara

MakotoOS12-434/70

OS21-3

42/87

Ohe

Kaoru

OS22-1

37/88

Mastaneh

Norrima[O]

MoribeIichirouGS3-2

25/106OS22-5

37/89

MoshkinVladimir

OS5-6

39/60

OS22-6

38/89

GS5-2

36/109OS16-2

39/76

OS16-4

40/76

OS9-2

22/65

MuhammadOkawachi

Yuto

Risal
Muthusamy

Hariharan

OS21-2

41/87

Mustapha

Ain Atiqa

OS25-3

48/94

OS9-3

22/65

OS25-4

48/94

OS9-4

22/65

OS25-7

48/95

Okuda

Haruhisa

PS3

29/50

OS10-4

28/67

Onaga

Haruki

OS9-2

22/65

Ono

JumpeiOS27-133/100

Ooba

Miki

GS2-3

23/105

Mustafin

MaksimOkazaki[N]

Naonobu

Nagayoshi

MasatoOS12-234/70

OshimaTatsuyaOS22-1

37/88

Naito

Yuka

OS27-2

33/101

Ozaki

Tomoaki

PS1

20/50

GS5-2

36/109Najirah Umar

120
©ICAROB 2023 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

[P]

RahmanHameedurOS26-946/99

OS26-10

46/100OS18-3

40/78

Pak

Alexander

OS10-5

28/67

Pan

Li

OS26-8

46/99

Paramasivam

Sivajothi

OS6-5

43/61

OS18-5

40/79

Park

Chan Gook

GS1-8

35/105

OS18-6

40/79

Patan

R.

OS23-4

24/91

OS18-8

41/80

Peng

Bo

OS26-2

45/97

OS18-9

41/80

Peng

Tzu-En

OS1-2

21/52

OS18-10

41/80

OS1-4

21/51

OS18-11

41/81

OS13-1

26/71

OS18-12

41/81

OS13-2

26/71

OS21-1

41/86

OS13-3

26/71

Rajagopal

Renuka

OS18-3

40/78

OS13-4

26/71

Rajapakse

R.P.C. Janaka

OS17-4

32/77

OS13-5

26/72

Ramis

Kulmukhametov

OS11-6

47/69

OS13-6

26/72

RathinamManoj

OS18-3

40/78

OS13-7

26/72

Razzak

Imran

OS7-1

31/62

OS13-8

26/72

OS7-2

31/62

OS13-9

26/73

OS2-4

25/53

OS13-10

27/73

OS2-5

25/53

Peng

YizhunRajagopal

Ren

Heshalini

Xiang

Permyakov

Alexander

OS5-6

39/60

RiajuliislamMD

OS22-7

38/89

Phan

Duc

GS6-3

42/111

RizwanAmar

OS25-1

47/93

Phiri

Gershom

OS6-1

43/60

OS25-3

48/94

Pramanta

Dinda

OS15-1

22/74

GS5-2

36/109Pujianti[S]

Wahyuningsih[Q]

S

Amutha

OS18-2

40/78

S

AkshayOS18-3

40/78

Sadyikoｃa

KarinaOS11-5

47/69

Safin

Ramil

OS10-2

28/66

OS11-6

47/69

OS22-2

37/88

Qi

Pon XiaoGS7-1

49/111Qin

Jianfeng

OS13-726/72

Qayyum

Abdul

OS6-1

43/60

OS6-2

43/60

OS22-3

37/88

OS6-3

43/61

OS22-4

37/89

OS6-5

43/61

OS22-5

37/89

OS7-1

31/62

OS22-6

38/89

OS7-2

31/62

OS22-7

38/89

OS7-3

31/62

OS22-8

38/90

OS7-4

31/63

Sakamoto[R]
121©ICAROB 2023 ALife Robotics Corp. Ltd.

MakotoSalmanRiham

OS11-2

47/68

Sendari

Siti

OS21-1

41/86

Saleh

GhassanOS18-6

40/79

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

SaruchiSarah Atifah

OS25-1

47/93

Sugimoto

Taiki

OS27-433/101

OS25-2

48/94

Sulaiman

Shifa

OS10-1

28/66

OS25-348/93

OS11-2

47/68

OS6-1

43/60

Saw

D. WenOS23-4

24/91

Seino

SatokoOS16-2

39/76

OS6-2

43/60

OS16-4

40/76

OS6-3

43/61

OS6-5

43/61

Summakieh

MHD Amen

Seki

Hiroaki

OS14-3

36/74

Saleh

GhassanOS18-12

41/81

Sumoto

Yuta

GS1-5

35/104

GS7-5

49/113Sun

Haozhe

OS3-2

27/54

SethaputThunyaseth

GS6-4

42/111Sun

Xun

OS26-1

45/97

Shang

Xueqiang

OS24-2

20/92

Sun

Yizhe

OS13-826/72

SharmaAbhishek

OS26-7

46/99

Sun

Yucheng

OS24-421/93

OS26-8

46/99

SunZhonghengOS26-1

45/97

GS5-2

36/109Shen

Chua Huang

OS6-2

43/60

Suwatri Jura

Shi

Puwei

OS20-5

29/83

Suzuki

KatsuakiOS20-13

31/85

Shiba

TomoyaOS15-3

22/74

Suzuki

YasuhiroOS19-1

37/81

OS15-523/75

OS19-2

37/81

GS3-1

25/106OS19-3

37/82

GS7-1

49/111

OS15-1

22/74

GS7-2

49/112OS15-2

22/74

GS7-3

49/112

Shibghatullah

Abdul

Shibanoki

Taro

GS4-4

24/108

Shirley

GlaretOS18-841/80

SyulistyoArie Rachmad[T]

Sinnappan

T.

Y. HuiOS25-7

48/95

Tabuchi

RyosukeOS22-1

37/88

OS18-941/80

TakagiTomohiko

OS9-1

22/64

Simatrang

Sorn

GS6-4

42/111

Takahashi

Hirokazu

OS12-1

34/70

Stephen John

ReubenOS18-2

40/78

Takakura

Shota

OS9-3

22/65

SoesiloTaufik

GS1-4

35/104

TakanoYouta

OS20-7

30/83

SpektorIlya

OS10-2

28/66

OS20-8

30/84

Su

Ying

OS3-5

27/54

OS22-2

37/88

OS3-6

27/55

OS22-3

37/88

OS3-7

27/55

OS22-5

37/89

GS7-1

49/111

OS22-6

38/89

GS7-3

49/112OS22-7

38/89

GS7-4

49/112OS22-8

38/90

GS7-5

49/113OS22-5

37/89

GS7-6

49/113OS22-6

38/89

OS20-1030/84

OS20-1130/85

Subaramaniam

Kasthuri

Sugikawa

SatoshiOS12-334/70

Sugimoto

KazuhideOS22-6

38/89

Takei

Takemura

Takemura

122©ICAROB 2023 ALife Robotics Corp. Ltd.

Amane

KakeruYasunori

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

TakeokaKenta

OS12-3

34/70

GS3-2

25/106

Tamaki

HisashiOS12-2

34/70

GS3-3

25/107

Tamukoh

HakaruOS15-1

22/74

Terasawa

Takashi

GS5-1

36/109OS15-2

22/74

Tew

Hwa Hui

OS25-2

48/94

OS15-3

22/74

Thamadharan

Kavitha

OS18-1

40/78

OS15-4

23/75

Thota

TejaswiniOS18-2

40/78

OS15-5

23/75

Tian

ShaokaiOS13-5

26/72

GS1-2

34/103Tiang

Sew SunGS4-6

24/108

GS1-3

34/103

OS6-1

43/60

GS1-4

35/104

OS6-3

43/61

GS1-5

35/104OS26-2

45/97

GS1-6

35/104OS26-3

45/97

GS4-2

23/107OS26-4

45/98

GS5-4

37/110OS26-5

45/98

GS6-1

42/110OS26-645/98

OS16-2

39/76

OS26-7

46/99

OS16-440/76

OS26-8

46/99

OS26-345/97

OS26-946/99

OS26-445/98

OS26-10

46/100OS26-545/98

OS26-946/99

OS26-645/98

OS26-10

46/100OS25-1

47/93

OS16-2

39/76

OS25-1049/96

OS16-440/76

OS16-2

39/76

Tamura

Tan

Tan

Tan

HirokiChi Jie

Jia WeiRodney

Ting

Titan

Janthori

TanakaTakeshiOS17-5

32/78

TanakaToshiki

OS17-1

32/77

OS16-4

40/76

OS17-2

32/77

OS20-1230/85

OS17-3

32/77

OS20-1030/84

OS15-2

22/74

OS20-1130/85

OS15-423/75

OS20-14

31/86

TanakaYuichiro

Tominaga

Tin Chang

Tominaga

Ayumu

Moeko

Tanev

Ivan

OS17-1

32/77

Tomioka

Toyoaki

OS22-4

37/89

Taniguchi

Rie

OS19-3

37/82

Too

Kok Sem

OS25-948/96

Tanveer

M. H.

OS23-3

24/91

Tsoy

Tatyana

OS5-5

39/59

OS23-4

24/91

OS10-1

28/66

OS23-5

24/91

OS10-328/66

OS20-1

29/82

OS10-5

28/67

OS20-2

29/82

OS11-1

47/68

GS2-3

23/105OS11-2

47/68

GS2-4

23/106OS11-6

47/69

GS2-5

23/106OS14-3

36/74

Taniguchi

Tanjo

Yasutaka

Yui

Tsuji
123

©ICAROB 2023 ALife Robotics Corp. Ltd.

Tokuo

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

[U]
Umeno

Ren

GS1-1

34/103ul Husnain

Anees

OS21-1

41/86

Urano

Shingen

OS20-1

29/82

Wang

Xiaolong

OS24-4

21/93

Wang

Xuran

OS4-3

44/56

Wang

Zipei

OS2-4

25/53

OS2-5

25/53

OS20-1

29/82

OS20-2

29/82

OS20-1

29/82

OS20-2

29/82

Watanabe

Watanabe

Keisuke

Konosuke[W]
Wadi harun

SulaimanOS21-3

42/87

Wen

ZongyanOS26-11

46/100Wagatsuma

HiroakiGS4-3

24/108Wong

Chin Hong

OS6-2

43/60

GS6-2

42/110OS26-2

45/97

GS6-3

42/111

OS26-7

46/99

Wan

WeiweiOS14-2

36/73

OS26-846/99

Wan Ahmad

Wan

OS21-2

41/87

OS26-946/99

OS26-11

46/100Khairunizam
Wang

Chun-Chieh

OS5-3

39/59

Wu

Qihua

OS24-4

21/93

Wang

DepengOS4-1045/58

Wu

Tai-Hsuan

OS5-4

39/59

Wang

Haiquan

OS3-5

27/54

Wu

YingCheng

OS1-3

21/51

Wang

Jiashuai

OS2-1

25/52

Wang

Jiaxin

OS2-3

25/53

OS3-3

27/54

YuntianOS13-2

26/71

OS3-4

27/54

OS13-6

26/72

[X]
Xia

Wang

Jiqiang

OS8-2

35/63

Xiang

Yande

OS4-10

45/58

Wang

Jiwu

OS24-1

20/92

Xiao

Ziyue

OS3-9

28/55

OS24-2

20/92

OS4-8

44/57

OS24-3

20/92

OS4-9

44/58

OS24-421/93

OS4-1045/58

OS24-5

21/93

OS3-8

27/55

OS3-9

28/55

Xie

Jiahao

Wang

Limei

OS13-1

26/71

Wang

Ningfei

OS26-2

45/97

Xie

Shengke

OS18-12

41/81

Wang

Peng

OS4-5

44/57

Xie

YanzheOS26-11

46/100

OS4-6

44/57

OS4-7

44/57

OS4-6

44/57

YabukiTomohideGS1-1

34/103

OS4-7

44/57

YamabaHisaaki

OS9-2

22/65

Wang

Qikun

[Y]

Wang

Shengfeng

OS4-6

44/57

OS9-3

22/65

Wang

Shuai

GS3-1

25/106

OS9-4

22/65

Wang

ShuxinOS13-8

26/72

YamadaShintaro

GS3-3

25/107Wang

Siyi

OS13-6

26/72

Yamaguchi

Hiromasa

OS15-2

22/74

124©ICAROB 2023 ALife Robotics Corp. Ltd.

The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

Yamaguchi

Naoki

OS15-322/74

OS11-4

47/69

Yamao

Kosei

OS15-4

23/75

ZakariyaAzman

OS25-8

48/96

Yan

Lixia

OS8-1

35/63

Zhai

Hongshuo

OS4-2

44/56

Yang

Hang

OS24-2

20/92

Zhang

Bin

OS8-3

35/63

Yang

Tongqing

OS8-4

36/64

Zhang

Junsheng

OS13-9

26/73

Yang

Xue

OS3-5

27/54

OS13-10

27/73

OS3-6

27/55

Zhang

Lijiang

OS2-1

25/52

OS3-7

27/55

Zhang

Saijie

OS2-2

25/52

Yang

Yahui

OS24-5

21/93

Zhang

Weicun

OS8-2

35/63

Yankova

Anastasia

OS10-3

28/66

Zhang

Xinyi

OS14-2

36/73

Yeoh

Zhi Ying

OS26-3

45/97

Zhang

Yuhao

OS2-1

25/52

OS26-4

45/98

OS4-3

44/56

OS26-5

45/98

OS4-4

44/56

OS26-6

45/98

Zhang

Yuqi

OS8-3

35/63

Yin

Xiaoyu

OS8-2

35/63

Zhao

Huailin

OS2-2

25/52

Yonehara

Shun

OS15-6

23/75

Zhao

HongpiOS13-4

26/71

Yoon

Jang-Sok

GS6-2

42/110OS13-7

26/72

YoshikawaAkimasaGS5-3

37/109

Zhao

ZhihanOS13-4

26/71

YoshimotoYuma

OS15-6

23/75

Zhdanova

Julia

OS5-6

39/60

Yuan

Tong

OS8-5

36/64

Zhidenko

Ivan

OS5-6

39/60

Yuldashev

Niez

OS10-6

28/67

Zhou

WenxuanOS26-1

45/97

YusoffBadli Shah

OS6-4

43/61

Zhou

Yue

OS26-1

45/97

Zhu

YingfanOS4-1045/58

OS13-4

26/71

OS11-547/69[Z]
Zagirov

Aidar

OS10-2

28/66

Zhukova

125©ICAROB 2023 ALife Robotics Corp. Ltd.

Valeriya

Developing a High-Speed Working Motion of the Multi Robot in DENSO
Tomoaki Ozaki

DENSO CORPORATION Advanced Research and Innovation Center,
500-1, Minamiyama, komenoki-cho, Nisshin-shi, Aichi-ken, 470-0111, Japan
Email: tomoaki.ozaki.j4a@jp.denso.com

Abstract
The number of labor force is expected to decrease due to the declining birthrate and aging population. Although manyefforts to solve this social issue using automation technology by robots around the world have been implemented,
most of the current applications of robots are still repetitive works such as picking and placing in mass productionlines in factories, and small progress of the application of robots for high-mix low-volume production lines where theoperations are is frequently changed. In this paper, we discuss the reasons and the potential solutions for autonomous
control technologies including the AI and deep learning. Moreover, the high-speed working motion of the multi robot
developed by DENSO will be presented in this paper.
Keywords: Unmanned Manufacturing, Motion Planning, Deep Neural Networks (DNN), Intelligent robotics

physically operate the robot and refine its movements,
resulting in significant time and effort when introducing
robots. Although the cost of the robot itself is becoming
affordable in the market, it cannot be ignored due to its
motion design cost when frequent motion changes are
made in low-volume, high-mix production. The same
issue arises in scenes such as agriculture field or vehicle
disassembling outside of factory, where it is difficult to
fix the motion of robots due to their variability.

1. Introduction
The decline in the working-age population due to aging
is progressing worldwide, becoming a serious social issue.
In particularly Japan, we have been facing on a severe
decrease in the working-age population, with estimatessuggesting a decrease to 52.75 million people (a 29.2%
decrease from 2021) by 2050 [1]. To address this issue,
DENSO is actively intensifying its research anddevelopment efforts in automation technology utilizing
robots. At DENSO's factories, we aim to achieve fully
unmanned manufacturing and significantly reduce
production lead times through the utilization of AI inprocess design for automation by 2035. We are also
advancing the development of robots for automatedharvesting of crops in agriculture field and automateddisassembling of vehicles in the field of circular economy.
As the utilization of robots advances in various sectors ofsociety, such as production and agriculture, our groupcompany DENSO WAVE has developed the humancollaborative robot COBOTTA PRO. COBOTTA PRO isa human-collaborative robot that combines the speed(2,500mm/s) and precision (repeatability ±0.04mm)
comparable to industrial robots. It maximizes its
operation speed when there are no workers nearby andminimizes the distance required for deceleration andstoppage when a worker approaches, achieving a goodbalance between productivity and safety aspect.

The second issue is related to adapting to changes andvariations in the workpieces. In the teaching playback
method commonly used in repetitive tasks in factories, it
is difficult to manage changes and variations in theworkpieces, making it challenging to accommodate
misalignment or stacking of the workpieces. Efforts toaddress these issues have progressed at a practical levelby utilizing vision systems. The vision system uses 2D or
3D sensor cameras and recognition algorithms torecognize the position and posture of the workpiece andthe surrounding environment, and the robot system as awhole uses this information to generate robot movements
according to the situation. However, the range of
workpieces that can be handled is still limited, and therecognition capabilities for complex shapes, reflective
properties, transparent objects, and even indefinite shapes
are still insufficient. Furthermore, the construction ofthese vision systems requires the actual objects of theworkpieces for adjustment, which involves significant
time and effort.

However, there are still several important issues thatneed to be addressed for further utilization of robots.
Firstly, one of these issues is related to the design of robot
movements. The work of teaching and programming for
robot motion design requires specialized technicians to

Additionally, tasks that involve handling flexible
materials are particularly challenging for robots. The
tasks such as handling wire harnesses and arranging
components that undergo significant shape changes due©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1

Tomoaki Ozaki

to interactions with the robot and the environment are
difficult for robots to predict and respond to, making
them one of the tasks where automation has not yetadvanced in factories.

of training data to achieve high performance, which
presents a challenge for real-world applications.
However, Dactyl addresses this issue by employing atechnique called domain randomization, which
diversifies the physical behavior in the simulation
environment. By training solely in the simulated spacewith varied physics, it is able to acquire movements thatcan be applied Denso’s in the real world. While thecurrent examples in the Dactyl project primarily focus on
manipulating rigid objects, it is anticipated that similar
techniques will be explored to extend the acquisition of
movements for flexible objects in the future.

2. Literature Survey
Efforts to automate teaching are primarily progressing
through the development of motion planning techniques.
Motion planning is the technology that automaticallygenerates a path connecting the start and end points ofrobot motion. Traditional manual teaching requires
significant effort to create robot paths that are collisionfree and have the shortest possible motion time. Motion
planning techniques are predominantly based on thesampling-based methods, with Rapidly-exploring
Random Trees (RRT [2]) being devised and subsequently
improved upon [3], [4], [5]. Currently, these techniques
have reached a practical stage where they are
implemented within robot simulators and used for preverification of motion in simulation environments.
However, there are issues related to the trade-off between
computation time and the quality of generated paths, aswell as the variability in the computation time requiredfor path generation. Further technological advancements
are expected to enable real-time motion generation,
particularly in complex environments with narrow spaces
or multiple robots sharing the workspace, where
computation time significantly increases. One promising
approach for generating high-quality paths with fast andconstant computation time is to utilize Deep Neural
Networks (DNN) as an alternative to traditional
computation methods. For example, Chi, C., et al. have
employed diffusion models to acquire DNN models forgenerating robot motion paths, which have demonstrated
superior performance in terms of discreteness andmultimodality compared to conventional imitation
learning [6]. As improvements in learning stability andenvironmental robustness progress, expectations forpractical implementation are increasing.

3. Research in Denso
In order to replace robots with more advanced tasks inthe future, it is essential to have coordination among
multiple robot arms as a single robot arm has its
limitations. However, when trying to apply RRT-based
motion planning to multiple robots, as mentioned earlier,the search space expands exponentially with the increasein the number of robots, resulting in a significant increasein computation time. To address this challenge, we
incorporated the expertise of skilled teaching engineers
into the exploration algorithm and effectively constrained
the search range, significantly reducing computation time
while maintaining path optimality. Fig. 1 illustrates therelationship between the number of controlled robots andthe computation time required to output the initialsolutions for the generated paths. With the conventional
RRT-based method, the computation time exceeded 31
seconds for two robots, over 32 minutes for three robots,and reached more than 87 hours for four robots (estimatedcomputation time for four robots). However, with themethod we developed, the computation time was reduced
to approximately 0.4 seconds for two robots, 0.7 seconds
for three robots, and 1.6 seconds for four robots.

One example of addressing changes and variations inwork pieces is the utilization of AI in vision systems. The
Dex-Net (The Dexterity Network [7]) project hasconstructed a large-scale database for object grasping anddeveloped models that predict the gropability of given
objects by training DNN using the database as training
data. This project utilizes not only real-world data but
also synthetic point clouds generated from 3D models,significantly reducing the cost of creating training data.
The scalability of both DNN models and training data is
expected to enhance generalization, leading to improved
performance for unknown objects.

Fig. 1. Relationship between computation time andnumber of robots in route generation
Furthermore, by adding post-processing for path
optimization, we reduced the robot motion time. Throughpractical verification in actual pick-and-place operations(In the environment shown in Fig. 2, we conducted ademonstration of placing 75 CDs to three designatedlocations using three robots.), we confirmed a 23.8%
reduction in robot operation time compared to the results

In the Dactyl project by OpenAI, they demonstrate an
example of acquiring dexterous manipulation skills using
deep reinforcement learning, where a robot with fivehuman-like fingers adeptly manipulates a cube [8]. The
Deep Learning techniques often require a large amount©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 20242

Developing a High-Speed Workingof skilled teaching engineers (The results of skilledteaching engineers: 48.9 seconds, Automatic Generation
by Motion Planning: 37.2 seconds).

Table. 1. Comparison of the results of motion
generation by teaching and automatic generation
algorithms
1-1. Comparison of joint movement

Robot 1
Robot 2
Robot 3
Total

Joint movement [rad](a)(b)
Teaching
Proposedmethodmethod(Automatic
generation)
6363.36268.75835.75352.74209.94176.716408.9
15798.1

Reduction rate [%]Reduction ratefrom (a) to (b)

1.49
8.28
0.79
3.72

1-2. Comparison of simultaneous operation rates ofmultiple robots(a)Teaching
method2-1. View of the actual machine environment

Total
operation
time [sec]Maximum
possible
concurrent
operation
time [sec]ActualConcurrent
Operation
Time [sec]Rate ofconcurrent
operation[%]

2-2. Overview of the demonstration environment
Fig. 2. The demonstration of pick-and-place operations
with multiple robots
Table 1 shows a more detailed comparison. In a multirobot operation, the amount of movement (joint
movement) as well as the degree to which the robot isable to operate simultaneously greatly affect its working
time. When we compared these results, we found that
there was no significant difference in joint movement
between the teaching result and the auto-generated result(The auto-generated result showed approximately 3.7%
less movement compared to the teaching result.), but theauto-generated result showed a greater improvement ofover 30% in the simultaneous movement rate comparedto the teaching result.

(b)
Proposedmethod(Automatic
generation)

Improvement
value
from (a) to (b)

31.7

-

46.6

29.3

-

28.2

26.8

-

60.6

91.4

30.8

48.9

4. Conclusion
To expand the applications of robots, the developmentof technology that automatically generates their
movements is desired. Motion planning is an effective
technique for automatically generating optimal paths for
robots to reach their work points. In this paper, we
introduced examples of automatically generating
movements, specifically for cases where multiple robots
operate simultaneously with high motion speed, todemonstrate the effectiveness of motion planning. The
technology for accurately recognizing and manipulating
the position and orientation of target objects is still underactive research. We are confident that the advancementof deep learning-based techniques will contribute tosolving these challenges. Unlike language and images,
the cost of generating large amounts of training data is an
issue in robot motion. However, by utilizing synthetic
point clouds generated from 3D models and physicalsimulations, successful examples of robot motion

In this development case study, we have shown that we
can automate advanced teaching tasks that handle
multiple robots and generate highly optimal paths
compared to skilled teaching engineers. In the future, we
intend to make it possible to automatically generate
placement of robots and peripheral equipment, andoperation procedures, leading to fully automated start-upof equipment.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
3

Powered by TCPDF (www.tcpdf.org)

Tomoaki Ozaki

generation applicable to real space are emerging, andfurther performance improvements can be expected.
References
1.

2.

3.

4.

5.

6.

7.

8.

Cabinet Office. (2022). 2022 edition of the white paper on
aging
society.
Retrievedfromhttps://www8.cao.go.jp/kourei/whitepaper/w2022/zenbun/pdf/1s1s_01.pdf [In Japanese].
LaValle, S. M., & Kuffner, J. J. (2001). Rapidly-exploring
random trees: Progress and prospects. In B. R. Donald, K.
M. Lynch, & D. Rus (Eds.), Algorithmic andcomputational robotics: New directions (pp. 293-308). A.
K. Peters.
Kuffner, J. J., & LaValle, S. M. (2000). RRT-connect: An
efficient approach to single-query path planning. In
Proceedings of the IEEE International Conference on
Robotics and Automation (ICRA '00) (Vol. 2, pp. 9951001). San Francisco, CA.
Kavraki, L. E., Svestka, P., Latombe, J.-C., & Overmars,
M. H. (1996). Probabilistic roadmaps for path planning in
high-dimensional configuration spaces. IEEE Transactions
on Robotics and Automation, 12(4), 566-580.
Harada, K. (2014). ロボットの動作計画における最適化 [Optimization in robotic motion planning]. 日本ロボット学会誌 [Journal of the Robotics Society of Japan],32(6), 508-511. [In Japanese].
Chi, C., Feng, S., Du, Y., Xu, Z., Cousineau, E., Burchfiel,
B., & Song, S. (2023). Diffusion policy: Visuomotor
policy learning via action diffusion. Retrieved fromhttps://arxiv.org/abs/2303.04137
Mahler, J., & Goldberg, K. (n.d.). Dex-Net: Deep learningfor dexterous robotic manipulation. Berkeley AUTOLAB.
Retrieved from https://berkeleyautomation.github.io/dexnet/
Andrychowicz, M., Baker, B., Chociej, M., et al. (2018).
Learning dexterous in-hand manipulation. Retrieved fromhttps://arxiv.org/abs/1808.00177

Authors Introduction
Mr. Tomoaki Ozaki
He is the Chief of Intelligent Robotics
R & I Section in AI R & I Division at
DENSO
CORPORATION.
He
graduated from the Graduate School ofInformation Technology, Kyushu
Institute of Technology in 2001. He iscurrently working on research anddevelopment related to the applicationof AI in factory automation, with a
specific focus on autonomous control of robot arms.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
4

Experimenting with Variable Arm Quadrotors: Realizing Dynamic Configurations forEnhanced Flight Performance
Hazry Desa

Centre of Excellence for Unmanned Aerial System (COE-UAS), Universiti Malaysia Perlis, Block E, Pusat Perniagaan
Pengkalan Jaya, Jalan Kangar – Alor Setar, 01000 Kangar, Perlis, Malaysia
Muhammad Azizi Azizan

Centre of Excellence for Unmanned Aerial System (COE-UAS), Universiti Malaysia Perlis, Block E, Pusat Perniagaan
Pengkalan Jaya, Jalan Kangar – Alor Setar, 01000 Kangar, Perlis, Malaysia
Email: hazry@unimap.edu.my

Abstract
This paper introduces two innovative concepts for variable arms designed for a quadrotor, enabling precise control of
its movement through manipulation of the bending moment via varying arm lengths. The primary objective of thisresearch is to develop and identify the most suitable variable arm configuration that facilitates smooth and stable
quadrotor movement. The study delves into two concept designs that are well-suited for the quadrotor application. By
employing a suitable variable arm, the quadrotor's maneuverability can be effectively regulated based on the bending
moment adjustments made possible by the variable arm. Ultimately, the paper presents the design and performance
testing of two types of variable arms. The obtained results confirm that the variable arm of the electric actuator withlinear guide-Type 2 exhibits smooth and stable movement.
Keywords: Variable Arm, Quadrotor, Dynamic Configuration, Flight Performance

reduced flying time [8]. This conventional design
approach can be both costly and inefficient [9].

1. Introduction
The quadrotor, a type of UAV (unmanned aerial
vehicle), holds great potential for overcoming terrestrial
challenges [1], leading to its widespread use in various
industrial and commercial applications such as
geographic mapping, surveillance, agricultural tasks likefertilization or pesticide application, and aerial
photography [2]. This continuous utilization has spurred
the development of the quadrotor to enhance its stability,
performance, and multimodal capabilities [3]. Typically,quadrotors are designed with four rotors, positioned at thevertices of a square frame [4]. To understand how thequadrotor operates, Newton's Third Law of Motion comes
into play, stating that every action elicits an equal andopposite reaction [5]. When the quadrotor's motor rotates
the propellers, it generates a downward force on the air,as explained by the Bernoulli Principle [6]. In a
conventional quadrotor, adjusting motor speeds in thefour motors allows the control of thrust and facilitates therequired movements [7]. However, as the size and weightof the quadrotor increase, more thrust is needed to lift it,
often necessitating larger propellers or faster motors,
which in turn lead to higher power consumption andConsidering the issues of power consumption and cost,limited research has explored alternative methods togenerate more thrust besides varying motor speed andpropeller size [10]. One such study by Wu in 2018
proposed controlling the quadrotor through changes inboth motor speed and rotor blade pitch angle, affecting the
produced thrust and power consumption [11], [12].
Additionally, a simulation analysis in 'Effects of Variable
Arm Length on UAV Control Systems' in 2020highlighted the impact of varying arm lengths on thequadrotor's bending moment. Increasing the arm length
resulted in higher bending moments and, consequently,
increased thrust production [13]. To address thesechallenges and explore new possibilities, this project aims
to design and test two concept designs for variable arms,
ensuring smooth and stable extension and retraction.
2. Design of variable arm
2.1. Concept design
Fig. 1 shows three concept designs of variable arms inboth the retract and extend positions. Fig. 1(a)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
5

Hazry Desa, Muhammad Azizi Azizan

corresponds to a concept design previously introduced inresearch [14]. However, when a quadrotor's arm
experiences thrust, it may undergo deflection andvibration caused by the motor, impacting the arm's
extension. To counter this issue, the use of a rod shaft andlinear guide method for the variable arm proves effective
in preventing arm deflection and ensuring smooth
movement. The design depicted in Fig. 1(a) presents
challenges in linear translation control due to thepossibility of high deflection and the lack of motion
guidance. On the other hand, the designs shown in Fig.
1(b) and Fig. 1(c) employ different methods for arm
movement control. Fig. 1(b) employs a stepper motor witha screw shaft while Fig. 1(c) utilizes an electric actuatorwith a linear guide for variable arm control.

M = FL

Fig. 2 illustrates the variable arm of type 1, utilizing a
stepper motor with a screw shaft and rod shaft, in both theextend and retract positions. The 3D drawing is used to
calculate the shaft's deflection using Eq. (2) and Fig. 3(a)assists in determining the moment of inertia. The results
display the maximum deflection when the variable arm
moves from its retracted position to a fully extended
position, as depicted in Fig. 3(b).

a.

a.

b.

Retract position

Variable arm concept

b. Extend position
Fig. 2. 3D modeling of variable arm steeper motor withscrew shaft and rod shaft (type 1).

Variable arm of stepper motor with screw shaft androd shaft (type 1)

𝑃𝐿3
3𝐸𝐼

(2)

𝜋𝑟 4
4

(3)

𝛿𝑚𝑎𝑥 =
𝐼𝑥 =

c.

(1)

Variable arm of electric actuator with linear guide(type 2)
Fig. 1. Variable arm quadrotor design concept.

Analyzing Fig. 3(b), it becomes evident that the
maximum deflection increases with an elongating length
under a constant force. The maximum deflection amounts
to approximately 0.001 mm, positioned 66.5 mm awayfrom the support of the bracket design. Given its small
magnitude, this deflection does not significantly impact
the turning of the screw shaft caused by the deflection of
the shaft.

2.2. Variable arm of speed motor with screw shaft
and rod shaft (type 1)
According to Eq. (1), the bending moment is determinedby multiplying the force by the length. In conventional
quadrotors, maneuverability relies on varying forces toinfluence the bending moment and consequently rotate
the quadrotor's body. This means that the same force canproduce different bending moments depending on thelength of the variable arm. By manipulating the bending
moment, the quadrotor can be directed and moved in a
specific direction. For the purpose of testing, variablearms capable of moving within a range of 20 mm to 30
mm have been designed.

a.

Cross section of screws shaft and rod shaft©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
6

Experimenting with Variable Arm

Technical Parameter
Dynamic Load (kgf)
Static Load (kgf)
Moment of R (kgf.m)
Moment of P (kgf.m)
Moment of Y (kgf.m)

b.

Value
140
200
0.78
0.49
0.49

Given the small size of the 2 kg quadrotor, a force of 2
kgf is required to maintain equilibrium, while around 3 to
4 kgf thrust is needed to lift the quadrotor. In the design
configuration, the center of the brushless motor ispositioned 26 mm away from the center of the linear guide,
enabling the calculation of the bending moment resulting
from the thrust force applied to the linear guide, as per Eq.
1. The linear guide proves to be suitable for use,considering a maximum load of 4 kgf and a maximum
bending moment of 0.104 kgf.m, leading to very minimal
deflection of the linear guide.

Maximum deflection of the shafts
Fig. 3. Deflection analysis.

Furthermore, Table 1 presents the specifications of the
stepper motor utilized, indicating an RPM range ofapproximately 15 to 20. With the use of a screw shaft of
0.5 mm pitch, the variable arm is able to move at the speedof 10 mm/min.
Table 1. The planning and control components.
Technical Parameter
Operating Voltage (V)Operating Current (mA)Step Angle
Reduction Ratio
Phase
Frequency (Hz)Friction torque (gf.cm)
Pull in torque (gf.cm)
CoilDecent Torque (mN.m)
Speed (RPM)

Table 3 presents the specifications of the electric actuator
used. As indicated in the table, the speed of the electric
actuator is approximately 8 mm/s. Moreover, considering
the force of 4 kgf, which equates to around 40 N, so thespecification of the electric actuator is suitable for thevariable arm application.

Value
5
240
5.625/64
1/64
4100
600-1200
300
Unipolar 5 lead coil
34.3
15-20

Table 3. Specification of LA-YR type electric actuator.
Technical Parameter
Input Voltage (V)No Load Speed (mm/s)
Load Push Capacity (N)Load Pull Capacity (N)Static Damping (N)Stroke Length (mm)2.3. Variable arm of electric actuator with linear
guide (type 2)
Fig. 4 presents the variable arm of type 2, which utilizes
an electric actuator with a linear guide, in both theretracted position (Fig. 4a) and extended position (Fig. 4b).
Referring to Table 2 reveals that the dynamic load of thelinear guide is approximately 140 kgf. Considering thepull exerted by the brushless motor on the variable arm, it
results in the moment of P acting on the linear guide.

3. Testing on variable arms
Based on the concept designs presented in the previous
section, two types of variable arms have been developed,
as depicted in Fig. 5 and Fig. 6. The first type is variable
arm of stepper motor with screw shaft and rod shaft
referred to as type 1. It comprises a total of 11 components,
including a motor driver to control the stepper motor, abracket to secure the stepper motor, the stepper motor
itself, a coupling to connect the stepper motor's shaft withthe screw shaft, the screw shaft, a brushless motor, thevariable arm, a bracket to hold the rod shaft, the rod shaft,a frame, and a linear bushing that ensures smooth
movement of the variable arm along the rod shaft.

Linear Guide

a.

Value
12
8
90
90
90
30

Retract position

Linear Guide

On the other hand, the second type, variable arm ofelectric actuator with linear guide known as type 2, isintended for an electric actuator with a linear guide. It
consists of 6 parts: an MDD3A driver to control theelectric actuator, the electric actuator, a linear guide, aframe, the variable arm, and a brushless motor.

b. Extend position
Fig. 4. Variable arm of electric actuator with linear guide(type 2).
Table 2. Specification of MGN7H linear guide.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
7

Hazry Desa, Muhammad Azizi Azizana.

Without operation of brushless motor

Fig. 5. Variable arm of stepper motor with screw shaft
and rod shaft (type 1)

b. With operation of brushless motor
Fig. 8. Vibration data for variable arm of electric
actuator with linear guide (type 2)
4. Conclusion
Fig. 6. Variable arm of electric actuator with linear guide(type 2)

Table 4. Comparison between 2 designs – type 1
and type 2.
Designtype 1
type 2
Cost
Low
High
Weight (g)13996Movement speed
Slow
Fast
Stability
High vibration
Low vibration

Through the utilization of a mobile app equipped with avibration meter, the vibration of the variable arm inmotion is assessed. The phone is placed on the variable
arm during the evaluation, and measurements are taken
both with and without the operation of a brushless motor.
The results are illustrated in Fig. 7 and Fig. 8, displaying
the vibration levels with and without the motor's operation
for both variable arms.

Based on the observations made during the motion of
the two variable arms, the results presented in Table 4
indicate that using type 2 would be a more costly option.
This is because electric actuators tend to be expensive
when compared to stepper motors. However, it should be
noted that the electric actuator demonstrated smooth andrapid movement, reaching approximately 8 mm/s, which
made it well-suited for quadrotor control. Additionally,
the electric actuator's advantage lies in its lightweightnature, as it utilizes fewer components, thereby
minimizing the overall weight impact on the quadrotor.

According to the graphs, it was evident that thebrushless motor caused higher vibration, indicating thatthe motor generates significant vibrations. Additionally, acomparison between the two types of variable arms
revealed that type 1 exhibited higher vibration than type2.

a.

Type 2 indeed offers a cost-effective solution. However,it comes with certain drawbacks. One major concern is theincreased weight due to the presence of multiple brackets
and shafts, which can adversely affect the overall
performance of the system. Moreover, the movement of
the variable arm driven by the stepper motor is not stable,
mainly due to the high vibration experienced along thearm and its slow-moving speed. This instability can be
attributed to the variable arm's elevated position, leading
to a higher center of gravity, with the screw and rod shaft
situated at the center of the variable arm. Consequently,
this setup causes vibrations that oscillate the arm to theleft and right. As a result, the thrust force provided by thebrushless motor becomes unstable, impacting the system's
overall performance.

Without operation of brushless motor

b. With operation of brushless motor
Fig. 7. Vibration data for variable arm of stepper
motor with screw shaft and rod shaft (type 1)

Additionally, the pitch of the M3 screw shaft is merely
0.5 mm. As a consequence, with each revolution of the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
8

Powered by TCPDF (www.tcpdf.org)

Experimenting with Variable Arm

8.

stepper motor, the variable arm can only move by 0.5 mm.
This limited displacement capability results in thevariable arm moving at a slow speed of only 10 mm/min,making it less suitable for certain applications that require
faster movement.

9.

In summary, type 2 is suitable to use as it provides fast
motion and stability which is suitable for quadrotors thatare sensitive to motion. Besides type 1, it will be suitable
to use when a high-pitch screw shaft or higher RPM motor
is used so that it can be moved faster. However, there isroom for improvement in the design of type 1. One
approach is to consider replacing the rod shaft with alinear guide or exploring other methods to enhancestability. Such modifications could help address the issues
related to vibration and slow movement, making it a moresuitable option for certain scenarios.

10.

11.

12.

Acknowledgements

13.

This work was funded by the Universiti Malaysia Perlis(UniMAP) under the Commercialization Grant 900100748.

14.

References
1.

2.

3.

4.

5.

6.

7.

Y. H. Tan and B. M. Chen, Underwater Stability of aMorphable Aerial-Aquatic Quadrotor with Variable
Thruster Angles, IEEE International Conference on
Robotics and Automation (ICRA), 2021, pp. 314–320.
H. Shraim, A. Awada and R. Youness R, A Survey on
Quadrotors: Configurations, Modeling and Identification,
Control, Collision Avoidance, Fault Diagnosis andTolerant Control, IEEE Aerospace and Electronic Systems
Magazine, Vol. 33(7), 2018, pp. 14-33.
J. Goslinski, W. Giernacki and S. Gardecki S,
Introduction of the Flying Robots into the Human
Environment: An Adaptive Square-Root Unscented
Kalman Filter for a Fault Tolerant State Estimation ina Quadrotor, IEEE International Conference onIntelligent Environments, 2014, pp. 117-123.
G. E. U. Faelden, J. M. Z. Maningo, R. C. S.
Nakano, A. A. Bandala and E. P. Dadios, A Neural
Network Approach to a Cooperative Balancing
Problem in Quadrotor-Unmanned Aerial Vehicles(QUAVs), International Conference on Humanoid,
Nanotechnology,
Information
Technology,
Communication & Control, Environment andManagement (HNICEM), 2016, pp.1-5.
M. F. Ahmed and Y. S. Narayan, Fabrication andTesting of Quadcopter Prototype for Surveillance,
International Journal of Mechanical & Production
Engineering Research and Development, 2018, pp.
99-105.
S. A. Khan, Z. Mehmood and Z. Afshan, Design,
Analysis and Topology Optimization of a Landing
Gear Strut for a Quadcopter Upon Impact,
International Conference on Applied and Engineering
Mathematics, 2 0 2 1 , pp. 37-42.
Y. R. Tang and Y. Li Y, Realization of the Flight
Control for an Indoor UAV Quadrotor, International
Conference on Information and Automation(ICIA), 2014, pp. 1278-1283.

A. V. Javir, K. Pawar, S. Dhudum, N. Patale and S.
Patil, Design Analysis and Fabrication of Quadrotor,
Journal of The International Association ofAdvance Technology and Science (JIAATS), Vol.
16, 2015.
N. Y. Kamil, D. Hazry, K. Wan and Z. M. Razlan,
Trajectory Tracking Based on Arm’s Length Variation,
Journal of Theoretical and Applied Information
Technology, Vol. 79 (3), 2015, pp. 528-536.
N. Y. Kamil, D. Hazry, K. Wan, M. Z. Razlan and
A. O. Khaldoon, Payload Capability of VAL –Quadrotor Based on PID Controller, International
Journal of Mechanical and Mechatronics
Engineering, Vol. 16(2), 2016, pp. 22-29.
M. Cutler and J. P. How, Analysis and Control of a
Variable-Pitch Quadrotor for Agile Flight, Journal of
Dynamic Systems, Measurement, and Control, Vol.
137(10), 2015.
S. Sheng and C. Sun, Control and Optimization ofa Variable-Pitch Quadrotor with Minimum Power
Consumption, Energies, Vol. 9(4), 2016.
N. Y. Kamil, D. Hazry, K. Wan and M. Z. Razlan,
A Novel VAL: Quadrotor Control Technique forTrajectory Based on Varying the Arm’s Length,
ARPN Journal of Engineering and Applied
Sciences, Vol. 11(5), 2016, pp. 9195-9204.
M. Rizon, C . K . Ang, M . I . Solihin, M. Z.
Razlan, D. Hazry, S. A. Bakar, K. Wan and I .
Zunaidi, Effects of Variable Arm Length on UAV
Control Systems, Journal of Robotics, Networking
and Artificial Life, Vol. 7 (2), 2020, pp. 91-97.

Authors Introduction
Dr. Hazry Desa
He obtained his PhD in Materials
Science and Production Engineering(Robotics) from Oita University andcurrently holds the position as a Headat the Centre of Excellence for
Unmanned Aerial Systems (COEUAS) at Universiti Malaysia Perlis(UniMAP).
Dr. Muhammad Azizi Azizan
He received his PhD in Civil
Engineering from the Universiti
Malaysia Perlis. He is currently a
Senior Lecturer in the same
institution. He is Head of Project
Integration & Management (PIM) at
Centre of Excellence for Unmanned
AerialSystem(COEUAS),
Universiti Malaysia Perlis.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
9

Artificial Intelligence and Technologies of Arm-type and Mobile Robots in Industry
Haruhisa Okuda

Mitsubishi Electric Corporation, 8-1-1 Tsukaguchi-honmachi, Amagasaki, Hyogo, 661-8661, Japan
Email: Okuda.Haruhisa@ct.MitsubishiElectric.co.jp

Abstract
In recent years, labor shortage has become a serious issue in industrial fields. Various technologies including robot
and information processing system to realize flexible work like humans are effective solutions to this issue. Artificial
intelligence technology of arm-type robots equipped with 3D sensors and force sensors has been applied in the
manufacturing field to cope with different intelligent and highly precise tasks in Mitsubishi Electric. In addition,
various technologies to expand the scope of application to the service field, as well as to realize highly functionaldelivery with mobile robots is under development. Furthermore, IoT technology is also being used for easy and quick
on-site implementation and efficient operation. This paper introduces these initiatives with actual examples.
Keywords: Industry, Arm-type robot, Service, Mobile robot, Artificial intelligence, IoT, 3D sensor, Force sensor

service field, as well as to realize highly functionaltransportation with mobile robots is under development.
In addition, as a future technology that will further
expand the scope of robot applications, we are also
developing technologies for remote robot operation byhumans for non-standardized tasks and troubleshooting.
Specifically, we have been developing technologies toconvert force information, into visual information that is
easily recognized by humans, to decompose summary
instructions into robot actions, and to operate multiple
types of robots with a common interface.

1. Introduction
In recent years, the decline in the working populationis accelerating worldwide as the birthrate declines and the
population ages and labor shortage has become a serious
issue in industrial fields. Many intelligent technologies
including robot system and information processing
system to realize flexible work like humans are effective
solutions to this issue, because there is a serious shortageof skilled and experienced workers. Based on these ideas,
Mitsubishi electric corp. has developed the intelligent
technologies of arm-type robots equipped with 3D
sensors and force sensors in the manufacturing field tocope with intelligent and highly precise tasks [1], [2], [3].
By combining artificial intelligent (AI) technology withthese technologies, we have developed technologies suchas automatic parameter adjustment technology to achieve
optimum motion adjustment quickly, and technology forsafe operation by determining abnormal conditions [4].

Furthermore, the increase in the number of target
devices monitored in real time with advancement ofcommunication technology and the increase in computercapacity have led to remarkable progress in 3D
simulation technology. These technologies are also being
used for easy and quick on-site implementation andefficient operation. In this paper, these initiatives withactual examples are described.

To spread various applications with intelligent robots,
we have developed the teaching-less robot system using
intuitive I/F technology to facilitate teaching operations
that previously required a high level of expertise andskilled experience. This system includes technologies togenerate optimum routes and movements, interference
checking algorithm with the surrounding area, and seriesof technologies for bin picking of random stacked objects
using 3D sensor.

2. Teaching-less Robot System
2.1. Overview of the system
The use of machines and robots in mass production isincreasing due to a shrinking labor force caused by anaging population and declining birth rate. However, there
are still challenges in implementing robots in certain
production lines, such as those involved in making“Bento” boxes and ready-made dishes. These lines
require expertise and time-consuming adjustments due tothe soft and irregular shapes of the food products andfrequent menu changes. We have developed a teachingless robot system that utilizes technology and researchdata accumulated over the years [5]. This system includes
arm robots, controllers, 3D vision sensors, force sensors,

There is a growing demand for automation of transport
operations not only in factories, but also in various
facilities such as logistics, public buildings, commercial
facilities, and hospitals. For this reason, multiple
technologies to expand the scope of application to the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
10

Haruhisa Okudaa tablet PC and a master PC for control. Task assignments
can be easily inputted using a tablet PC and 3D sensor
camera. The 3D sensor camera senses the area around thearm robot, allowing for static environment recognition
and accurate measurement of spaces. 2D vision sensors
installed on the production line detect the conditions of
the items moving on the line. Task assignments can be set
using voice and touch inputs, and the movement path of
the arm robot is displayed using augmented reality. This
teaching-less robot system eliminates the need for expert
knowledge and time-consuming programming, as it
automatically generates optimum movements based on
assigned start and endpoints. Autonomous programs
generate various approaches to determine the optimum
and fastest speed for movements with reducing the needfor test runs and saving time. The 3D sensors monitor the
production area and ensure smooth workflow. The
sensors identify slanted “Bento” boxes, allowing therobot to determine their position and add ingredients
accordingly. These technologies not only control the arm
robot but also facilitate the expansion of production lines
with multiple robots.

the robot's motion can be confirmed in advance on atablet PC, thus ensuring the safety of the operator (Fig. 2).
Easy touch and voice UIIndications that suggest instructions

Required informationon 3-D scanned actual
work environment isoverlayed"3 pieces" of "karaage" in"4" of the lunch box.

Fig. 2. Intuitive UI for easy robot teaching
2.4. Automatic Generation of Optimum Path
We have developed the automatic calculation method
which generate optimum path without interfering toperipheral equipment by simply specifying start and end
points. It can derive the avoidance route that minimizes
the operating time by considering the amount of
movement of each joint and allowable torque (Fig. 3).

2.2. Surrounding Environment Modeling
In order to acquire the 3D model of the robot’s
surrounding environment, we have used our original
simultaneous localization and mapping (SLAM) method
which use only an inexpensive camera (Fig. 1). Thiscamera can be used for interference detection during pathgeneration and displayed on the operation teaching
device. We have achieved processing time by simplifying
3D models and reducing data size (approximately 60%
reduction).

Overview of the Path Generation Process
User Input

Layoutinformation(CAD registration,
Capture static
environment
recognition
results, virtual
obstacles)

Images, depth, camera
poses can be measured
or estimated by SLAM (# 1

path generation
execution

Constraint
condition

Fig. 3.
3-dimension
modeling

Visualize
generated path

Outputseries commandsof optimum
moving path(maximum speed,
Allowable torque,
Interference
avoidance
margin, ...)

Srrounding environment modeling and data usage

Scanning ofsurroundings

Path generation package

Start andend points

Automatic Generation of Optimum Path2.5. Irregular Shape Object Recognition
Data size reduction Data transmission

3D point clouds andApproximation to apolygons can besimilar shape and noise
generated from colors,
removaldepth and camera poses

Data for display on
operation teaching
device,interference
detection during pathgeneration

To packing food objects using random bin-picking
method with 3D sensor, each object is distinguishedaccording to segmented region, and each grasping
position is estimated with high accuracy at high-speed
using the deep learning (DL) based AI algorithm (Fig. 4).

Examples of sensors
Intel(R) realsenese D455

Fig. 1. Surrounding Environment ModelingLearning phase2.3. Intuitive UI for easy robot teaching
TargetobjectWe have developed the intuitive user interface (UI)system which use tablet-touching and voice-controlled
operation to realize easy teaching operation of robots.
Various specifications such as the workpiece sizes andpositions can be confirmed beforehand by checking thesimulated movies using the augmented reality (AR)technology. After the motion instruction is completed,

Objectinformation

Modelgeneration

Simulation

Data set
generation

DL
Learning

Estimation phase

Measurement
data

Segmentation

Fig. 4.

Calculation ofgrasping
position candidates

InferenceengineAcquire grasping
position

Irregular Shape Object Recognition©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
11

Artificial Intelligence and Technologies

Prior learning of the DL network for determining
optimum grasping position according to the shape of eachobject is operated. Then this algorithm calculates
segmented region of each object using the trained DL
network.

3.2. Remote Robotics for Dexterous Operation
The technologies for remotely operating robots areuseful for individual operations in non-standardized tasks
and for troubleshooting operations in automated tasks.
Force-haptic information is often important in these tasks,
but feedback of these information to the operator
typically requires a complex operation system. We
proposed the “Visual Haptics” architecture that converts
force-haptics information into visual ones that are easily
precepted by humans (Fig. 7). The cognitive process inthis method utilizes visual and force-haptic crossmodalities. We have showed its effectiveness throughquantitative experiments [8].

3. Robot Technologies in Service Field
3.1. Mobile Robots for Transportation
The demand for the Autonomous Mobile Robot (AMR)
is increasing due to the decrease in the working
population and the increase in demand for various typesof transport (factories, distribution warehouses,
commercial facilities, hospitals etc.). To meet thesedemands, we have been developing the system that canbe redesigned to variation of transport configurations. We
have also developed the series of technologies to enablerobots to run safely and autonomously in response tochanges in facility rules and conditions through thecontrol system that controls multiple robots. In addition
to these technologies, we have also realized autonomous
vertical and horizontal movement in coordination withfacilities such as elevators and access control systems [6],[7] (Fig. 5).

Conventional method

Proposed method

Discomfort due to differencesfrom brain predictions

Elimination of discomfort by complementing
information in the brain
Display with colorand size changes
according to loadand pressure

VisualForce

HapticFig. 7.

We have been also conducting actual experiments withmobile robots in various situations (e.g., transportingbooks in reading rooms, making announcements whilemoving, and transporting items in backyards) as a fieldtest of mobile robots that can coexist with people (Fig. 6).

”

TargetObjectVisual Haptics” architecture

Furthermore, we have showed its effectiveness invarious situations through demonstration systems [9](Fig. 8).

AMR

Fig. 8.
Fig. 5.

Integrated Control System for AMR

Fig. 6.

Operation Test Scene

4. IoT Platforms for Data Integration
Mitsubishi Electric is ready to provide the digitaltransformation (DX) services including multiple IoT
platforms that organically connect data from facilities,
equipment, mobile applications functioning in various
fields such as manufacturing, logistics, business facilities,
hospitals, etc., with the aim of contributing to area valueenhancement and advanced operational management.
The various data collected in these distributed platforms
can be used to accelerate the creation of new value-added
products and seamless indoor/outdoor services (Fig. 9).

Field Test image of AMR©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
12

Red Text : Mitsubishi Electric's Contribution Domain: Products that Mitsubishi Electric can provide■DX System Map
Total
Management
SystemService・Applicati
ons

System DeveloperHigh efficiency and sophisticationof operations and management

QOL／Safe＆Secure
HealthCareLiveliness/
Interaction

Security

Disasterprevention/reduction

Facilitationofmigration

Cost cutLaborsavingEnvironme
ntal
Responses

Completecirculation

Universal
design…

6.

Data integration PF

Data
Platform

Various fields PF

PF

PF

Carrier communications, Wi-Fi, Local 5G, etc.

N/W

QuasiZenith
Satellites

Mobility

Centralmonitoring
systemRobot Human
Drone opera
tions
etc.

7.

etc.

Managementtask

Facilities

EnergyTransportationSecurity

On-siteEnvironment/Situation

Fig. 9.

Image/video
analysis

LocatorAccount
Payment

HealthPersonal Data

Camera、Beacon、Sensor etc.

Local Energy

Edge
RegionData

5.

Management of new areas and functions using advanced technology and digital technology

Personal
Attribute

Powered by TCPDF (www.tcpdf.org)

Haruhisa Okuda

Sensing, measurement, business and other data

DX System including IoT Platforms
8.

5. Conclusion
9.

At the first in this paper, the teaching-less robot system
was described, which is using intuitive I/F, optimum
routes and movements generation, interference checking
algorithm and series technologies for 3D random binpicking. At the second, as the robot technologies inservice field, the system that realize the flexible andsafety control of multiple AMR and the remote robotic
technologies for dexterous operations are showed withactual examples. At the last, the integrated IoT platforms
for connecting data from facilities, equipment and mobile
applications functioning in various fields is described.
Various solutions using these initiatives will be provided
in the near future.

https://www.mitsubishielectric.com/news/2021/pdf/1125.
pdf, 2021-11-25.
Mitsubishi Electric Press Release, “Mitsubishi Electric
Develops Teaching-less Robot System Technology”,
https://www.mitsubishielectric.com/news/2022/0228.html,
2022-2-28.
M. Yamazumi, T. Hirai, S. Kobori, T. Asahina, “A
coordinated control between a robot and a facility forservice robots in a building”, Proc. of JSME Annual Conf.
on Robotics and Mechatronics (ROBOMECH) 2021, 2A1H07, 2021, DOI: 10.1299/jsmermd.2021.2A1-H07.
(in Japanese)
M. Yamazumi, D. Mizuno, T. Koshimoto, T. Yamamoto,“Development of Mobile Robot Management Systemcooperated with Indoor Dynamic Map Management
System”, Proc. of JSME Annual Conf. on Robotics andMechatronics (ROBOMECH)2022, 1P1-Q04. 2022,
DOI:10.1299/jsmermd.2022.1P1-Q04. (in Japanese)
M. Haruna, N. Kawaguchi, M. Ogino, and T. Koike-Akino,“Comparison of Three Feedback Modalities for Haptics
Sensation in Remote Machine Manipulation,” IEEE Robot.
Autom. Lett., 2021, DOI: 10.1109/LRA.2021.3070301.
CEATEC2022, “Evolutionary tele-operation service
platform utilizing human cognitive characteristics”,, 2022.
(Online in Japanese)

Authors Introduction
Dr. Haruhisa Okuda
He received B.E. and M.E. from theDept. of Precision Engineering,
Kyoto University in 1991 and 1993
respectively. In 1993, he joined
Mitsubishi Electric Corp., where hehas been engaged in the researchand development of 2-D and 3D
object recognition system and AI
technology for industrial robots at mainly Advanced
Technology R&D Center. He received the Ph.D. degree in2007. He has been concurrently serving as a visiting
professor at the Graduate School of System Informatics,
Kobe University since 2019.He is a member of the RSJ.

Acknowledgements
These technologies, demonstration system and products
have been researched and developed by many colleagues
in Mitsubishi Electric Corporation and collaborative
research scientists. I owe them a great deal of gratitude.
References
1.

2.

3.

4.

H. Okuda, R. Haraguchi, Y. Domae, and K. Shiratsuchi,“Novel intelligent technologies for industrial robot inmanufacturing – Architectures and applications”, Proc.
International Symposium on Robotics (ISR2016), pp.570575, 2016.
H. Okuda, Y. Domae, Y. Kitaaki, K. Sumi, Y. Kimura, H.
Takauji and S. Kaneko, “Industrial Robot System utilizing
3-D Vision Sensing for Flexible Cable Handling”, 16th
Korea-Japan Joint Workshop on Frontiers of Computer
Vision (FCV2010), O5-2, pp.271-276, 2010.
K. Tanaka, H. Okuda, “Innovation on Manufacturing
Generated by Intelligent Technologies”, Proc. The 2018
International Conference on Artificial Life and Robotics(ICAROB 2018), pp.11-14, 2018.
Mitsubishi Electric Press Release, “Mitsubishi Electric and
AIST Develop AI Technology for Real-time Control of FA
Equipment”,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
13

The Detecting Abnormal Operations in ICS Using Finite-State Machines
Pei-Wen Chou

Department of Electrical Engineering / M.S. Degree Program on Cyber-Security Intelligence,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 701401, Taiwan
Nai-Yu Chen

Department of Electrical Engineering / M.S. Degree Program on Cyber-Security Intelligence,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 701401, Taiwan
Jung-Shian Li

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 701401, Taiwan
I-Hsien Liu*

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 701401, Taiwan
E-mail: pwchou@cans.ee.ncku.edu.tw, nychen@cans.ee.ncku.edu.tw, jsli@ cans.ee.ncku.edu.tw,ihliu@cans.ee.ncku.edu.tw*
www.ncku.edu.tw

Abstract
In 2021, a water treatment facility in Florida, USA, fell victim to an external malicious attack.in this incident,
malicious actors attempted to manipulate the quantities of specific chemicals to impact water quality and safety. Given
the intricacies of abnormal operation detection in Industrial Control Systems and the advantages of finite-statemachine, we endeavored to apply this approach for the detection of abnormal ICS (Industrial Control System)
operations. We conducted a series of tests using the dam control system cybersecurity testbed established by
TWISC@NCKU, Taiwan. The results indicate that our approach effectively enhances the efficiency of identifying
non-standard operational behaviors, enabling maintenance personnel to promptly identify anomalies.
Keywords: FSM, ICS Security, PLC, Dam Gate Testbed

monitor changes in PLC states. This amalgamation ofstate and anomaly detection enables swift identification
and response to potential issues, ensuring the stable
operation of the system.
This research focused on applying FSM methods todetect abnormal operations within ICS, aiming toenhance detection efficiency and accuracy. This researchuses the dam control system cyber-security testbed built
by TWISC@NCKU to conduct relevant research andverification. The results show that the scheme is feasible
and effective, and the method can be effective at a very
low cost. It can identify non-standard operating behaviors
and help personnel identify abnormal situations
immediately.

1. Introduction
In 2021, a water treatment facility in Florida, USA,
faced an external malicious attack, where attackers
attempted to manipulate the quantities of specificchemicals, severely jeopardizing water quality and safety.
[1] How to effectively detect abnormal operation of ICS
has become an important challenge?
The integration of FSM methods involves
comprehensive monitoring and analysis of system states[2], such as utilizing finite state machines. Taking
equipment malfunction as an example, the awareness thatits operation may deviate from normalcy prompted thedesign of a method for detection, employing FSM to©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
14

Pei-Wen Chou, Nai-Yu Chen, Jung-Shian Li, I-Hsien Liu

2. Methodology

information exchange among industrial devices such as
PLCs and sensors [6]. Despite the convenience ofplaintext data transmission provided by Modbus/TCP, it
inherently poses potential security risks, making it
susceptible to unauthorized access and cyber attacks.
In the context of my research, using Modbus/TCP as
an example, although it facilitates communication amongindustrial devices, careful consideration of associatedsecurity risks is crucial. The use of plaintext datatransmission introduces the risk of information exposure,thereby compromising the integrity and confidentiality ofthe system. Therefore, during the implementation of
Modbus/TCP, robust security measures must be
implemented to mitigate potential risks and enhance theresilience of critical infrastructure.
Simultaneously, by utilizing Modbus commands toread the memory addresses of PLCs [7], specific statessuch as DI and DO can be understood, with these states
stored in specific memory variable addresses. ThroughModbus queries, real-time insight into the current statusof the PLC can be obtained. This underscores theimportance, in practical applications, of handling
Modbus communication security cautiously to ensurethat the system's operation remains resilient against
potential risks and cyber threats.

2.1. Programmable Logic Controller

A Programmable Logic Controller (PLC) [3] automates
control by executing instructions stored in its memory.
Crucial in industries, it interfaces with systems like Finite
State Machines (FSM) for more efficient monitoring.
PLC states indicate its operations during automation.
"DI" and "DO" signify Digital Input and Output,
reflecting signal statuses (e.g., switches). "AI" and "AO"
represent Analog Input and Output, detailing continuous
signal conditions. Together, they showcase the PLC's
performance across various inputs and outputs.
Its architecture consists of a CPU, input, and output
modules. The CPU processes logic and manages device
communication. Inputs gather data from sensors, while
outputs control actuators. This robust design ensures
stable operation in demanding industrial environments.
2.2. Finite-State Machine
In recent years, Finite State Machines (FSMs) havebeen widely applied in various fields, including softwaredevelopment [2] and machine learning [4]. The strengthof FSM methodology lies in its intuitive system model,
aiding in the comprehension, design, and testing of
intricate system behaviors. Its simplicity and scalability
render it an effective tool for describing and controllingsystem behaviors.
In the context of anomaly detection, FSMs are
employed for modeling normal system behavior bydefining states and transitions. (Fig.1) This approach
involves monitoring system operations, detecting statetransitions deviating from expected behavior, andproviding real-time alerts. The integration of FSM-based
anomaly detection is widely adopted to enhance theefficiency of system behavior modeling and anomaly
detection [5], highlighting the pivotal role of FSMs incomprehending system behavior and addressing
challenges in anomaly detection.

2.4. Critical infrastructure testbed
The research through a series of tests conducted at
the Dam Control System Cybersecurity Testbed
established by TWISC@NCKU in Taiwan [8]. This
simulation validation ensures that the dam systemaccurately and promptly responds to abnormal conditions
during actual operations. The simulation of gate control
scenarios on the testbed verifies the correct response ofgate operations to simulated abnormal events during
actual operations, ensuring the stable operation of thedam system under abnormal conditions. This integrated
validation on the testbed contributes to ensuring thesafety and reliability of dam systems, enhancing theefficiency of responding to abnormal events during actual
operations.
3. Construct the PLC status set based oncontinuous discovery
3.1. System architecture
In response to cybersecurity and abnormal detection
issues in critical infrastructure, this paper proposes thedesign of an experimental platform for simulating theabnormal detection system of dam gates. (Fig. 2) Eachgate is operated by a Programmable Logic Controller(PLC), which is equipped with registers for accessing
relevant instructions and data. Consequently, thisFig. 1. Finite-State Machine

2.3. Modbus/TCP
Modbus is widely employed as a communication
protocol in industrial environments, facilitating©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
15

The Detecting Abnormal Operations

information can reflect the current state of theenvironment, such as the requirement for the abnormal
indicator light to be in the off state before any operations
can take place.

mode, the remote light is illuminated, indicating normal
system operation. However, if floodgate discharge isrequired, personnel must adhere to regulations andphysically attend the site, prompting a switch to on-sitemode with the power light activated.
Following the commencement of operations, theascending light is activated if gate opening is necessary,
signifying the gate's upward movement. Upon reaching a
specified height while fully opening the gate, theascending light ceases, and both the on-site and power
lights illuminate, denoting complete gate opening. Upon
achieving full gate openness at the base, the fully open
light illuminates, the ascending light extinguishes, andthe descending light activates, indicating an impending
gate descent. Operation Process Diagram in Fig. 3.
However, during operation, anomalies such as shortcircuits or system malfunctions might occur, potentiallycausing gate loosening or jamming during ascent, leading
to an overload condition. In such instances, the systemshould promptly react, for instance, by activating thecorresponding warning light to alert operators to perform
maintenance or emergency procedures.
In summary, this system proficiently communicatesthe diverse statuses and phases of gate operations throughdistinctive light cues. Additionally, it promptly issues
warnings during abnormal situations, ensuring
operational safety and manageability. Experimental
results as shown in Fig .4.

Fig. 2. Anomaly Detection Diagram
3.2.
State
set
construction
process
Virtual Dam Gate Abnormal Detection testbed
The testing platform employs the Modbus
communication protocol to scan the state ofProgrammable Logic Controllers (PLCs). The scanninginterval is set at 0.1 seconds, with a 1-second pause after
each scan to simulate real operating conditions. This
operation is repeated infinitely to observe changes in two
key variables: scan frequency and state variation.
Through an infinite number of scans, we will record theresults of each observation, laying the foundation for
subsequent discussions to thoroughly analyze potential
variations and system behaviors. The objective of thistesting platform is to gain a comprehensive
understanding of the abnormal detection performance ofthe virtual dam gate system.

Fig. 3. Operation Process Diagram

Fig. 4. Experimental results
4. Experiment Results
5. Conclusion
Given the complexity of abnormal operation
detection in industrial control systems (ICS), we opted toThis system effectively communicates the various
states and operational stages of the gate through distinct
light variations. When the gate is in remote monitoring©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
16

Powered by TCPDF (www.tcpdf.org)

Pei-Wen Chou, Nai-Yu Chen, Jung-Shian Li, I-Hsien Liu

employ the Finite State Machine (FSM) method foranomaly detection in ICS operations. Through a series oftests conducted at the Dam Control System Cybersecurity
Testbed established by TWISC@NCKU in Taiwan, ourresults demonstrated the effectiveness of our approach in
improving the efficiency of identifying non-standard
operational behaviors, enabling maintenance personnel topromptly recognize anomalies. This study not only
constitutes a technical exploration but also emphasizes
addressing potential anomalies in unique scenarios. In
summary, this research provides a comprehensive andeffective approach to ICS security, particularly in therealm of abnormal operation detection.

Authors Introduction
Ms. Pei-Wen Chou
She is a postgraduate of Cloud andNetwork Security (CANS) Lab,
InstituteofComputerandCommunication
Engineering,
National Cheng Kung University in
Taiwan. She received her B.B.A.
degree from the Department of
Healthcare
Administration
andMedical Informatics, Kaohsiung
Medical University, Taiwan in 2022. Her interests
encompass network security, blockchain, and industrial
control systems.
Ms. Nai-Yu Chen
She was born in Taichung, Taiwan in
1998. She is acquiring the master’s
degree in Degree Program on CyberSecurity Intelligence, National Cheng
Kung University in Taiwan. She
received her B.B.A. degree from the
Bachelor
of
BioBusiness
Management,
National
ChiayiUniversity, Taiwan in 2021. Her
interests are ICS Security, Network-Based Intrusion and
PLC.

Acknowledgements
This work was supported by the National Scienceand Technology Council (NSTC) in Taiwan undercontract number 112-2634-F-006-001-MBK.
References
1.

2.

3.

4.

5.

6.

7.

8.

CNN, “Someone tried to poison a Florida city by hacking
into the water treatment system, sheriff says,” 2021.
[Online].
Available:
https://www.cnn.com/2021/02/08/us/oldsmar-floridahack-water-poison. [Accessed 30 Oct. 2023].
I. Y. Smolyakov and S. A. Belyaev, “Design of the
Software Architecture for Starcraft Video Game on theBasis of Finite State Machines,” 2019 IEEE Conferenceof Russian Young Researchers in Electrical andElectronic Engineering (EIConRus), Moscow, Russia,
28-30 Jan., 2019.
T. Imanto, and A. Adriansyah. “Performance analysis ofprofinet network in plc-based automation system,” 2nd2020 International Conference on Broadband
Communication, Wireless Sensors and Powering(BCWSP), Yogyakarta, Indonesia, 28-30 Sep. 2020.
S. Wang, J. F. Balarezo, S. Kandeepan, A. Al-Hourani,
K. G. Chavez, and B. Rubinstein, “Machine learning in
network anomaly detection: A survey,” IEEE Access, vol.
9, pp. 152379-152396, 2021.
F. Farahmandi and P. Mishra, “FSM anomaly detection
using formal analysis,” 2017 IEEE 35th International
Conference on Computer Design (ICCD), Boston, USA,
5-8 Nov., 2017.
Y. Wang, Y. Wang, Z. Zhu, and Q. Wang, “Modbus TCP
protocol in industrial control system Research on
anomaly detection method,” 2021 IEEE International
Conference on Electrical Engineering and Mechatronics
Technology (ICEEMT), Qingdao, China, 2-4 Jul., 2021.
X. Li, F. Meng, and X. Zheng, “Automatic Control
System of Sluice Based on PLC, MCGS and MODBUS
Communication,” 2021 7th Annual International
Conference on Network and Information Systems for
Computers (ICNISC), Guiyang, China, 23-25 Jul., 2021.
M.-W. Chang, J.-S. Li, and I-H. Liu, “Cyber-Physical
Security Testbed for Dam Control System”, Journal ofAdvances in Artificial Life Robotics, Vol. 4, No. 2, pp.
63-66, 2023.

Prof. I-Hsien Liu
He is an assistant professor in
Department
ofElectricalEngineering, National Cheng Kung
University, Taiwan. He obtained his
Ph.D. in 2015 in Computer andCommunication Engineering fromthe National Cheng Kung University.
He teaches cybersecurity courses andhis interests are Cyber-Security,
Wireless Network, Group Communication, and Reliable
Transmission. He is the deputy director of Taiwan
Information Security Center @ National Cheng Kung
University(TWISC@NCKU).
Prof. Jung-Shian Li
He is a full Professor in the
Department
ofElectricalEngineering, National Cheng Kung
University, Taiwan. He graduatedfrom the National Taiwan University,
Taiwan, with B.S. in 1990 and M.S.
degrees in 1992 in ElectricalEngineering. He obtained his PhD in
1999 in Computer Science from theTechnical University of Berlin, Germany. He teaches
communication courses and his research interests include
cybersecurity, cloud computing and network management.
He is currently involved in funded research projects
dealing with cybersecurity and critical infrastructure
protection. He is the director of Taiwan Information
Security Center @ National Cheng Kung University.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 202417

Industrial Control System State Monitor Using Blockchain Technology
Yun-Hao Chang

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University,
No.1, University Rd., East Dist., Tainan City 701401, Taiwan
Tzu-En Peng

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University,
No.1, University Rd., East Dist., Tainan City 701401, Taiwan
Jung-Shian Li

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 701401, Taiwan
I-Hsien Liu

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 701401, Taiwan
E-mail: yhchang@cans.ee.ncku.edu.tw, tepeng@cans.ee.ncku.edu.tw, jsli@cans.ee.ncku.edu.tw,ihliu@cans.ee.ncku.edu.tw*
www.ncku.edu.tw

Abstract
This paper introduces an innovative approach to enhance data verification and security in intelligent systems throughthe integration of blockchain technology. The proposed method amalgamates the transparency and decentralization
inherent to blockchain with the command and oversight functionalities of PLC to ensure the utmost data integrity.
The devised approach synergizes the decentralized attributes of blockchain with the control capabilities of PLCs, thus
establishing robust safeguards for data integrity. Through the utilization of blockchain's tamper-resistant ledger, PLCs
orchestrate data interactions and enforce real-time monitoring and control. The viability and efficacy of this innovative
scheme are substantiated through empirical evaluations and simulations, conclusively affirming its practicality.
Keywords: Industrial Control System, Programmable Logic Controller, Blockchain, State Monitor

integrating blockchain into Programmable Logic
Controllers (PLC) ensures the availability and integrity ofthe data.

1. Introduction
In recent years, with the continuous development ofindustrial automation, Industrial Control Systems (ICS)
have played a crucial role in modern production
environments [1]. These systems possess complex
architectures, and their proper functioning is paramountto ensuring the stability and efficiency of the production
process. However, with the ongoing advancement ofdigital technology, industrial environments have become
more susceptible to data tampering and security threats.
Our research is based on the application of blockchain
technology, which, with its decentralized and transparent
nature, introduces a new level of data security. By

2. Related WorkThe related work in our study revolves around theapplication of Industrial Internet of Things (IIoT) andblockchain technology.
2.1. Industrial Control Systems and Blockchain
Researchers have explored the integration ofblockchain technology with Industrial Control Systems(ICS) to enhance security and privacy within the
Industrial Internet of Things (IIoT). Notable studies, such

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 202418

Yun-Hao Chang , Tzu-En Peng, Jung-Shian Li, I-Hsien Liu

as Z.-H. Sun et al. [2] survey of enterprise and literature
reviews to identify specific industrial requirements invarious supply chain scenarios, and G. Puthilibai et al. [3]proposal of a secure wireless solution based onblockchain technology for IIoT, and W. Zhou and J. Jin's
development and evaluation of a distributed access
control system using blockchain and smart contracttechnologies [4], collectively contribute to advancing theunderstanding and application of blockchain in theindustrial domain.

node consensus mechanism where designated nodes are
responsible for generating and validating new blocks.

Fig.1. Blockchain Structure

2.2. Blockchain Consensus

Clients are also participants in the blockchain
network, with their primary responsibility being thesubmission of transactions or information to theblockchain. Clients utilize the Remote Procedure Call
(RPC) protocol to communicate with blockchain nodes.
This mode of communication allows clients to send
remote requests, such as transaction submission requests,
while nodes process these requests, participate in theconsensus mechanism, and provide feedback on the final
outcomes to the clients. (Fig. 2).

Blockchain technology originated from Satoshi
Nakamoto's “Bitcoin: A Peer-to-Peer Electronic Cash
System” [5] and is a decentralized database technology.
Various consensus protocols, including Proof of Work
(PoW), Proof of Stake (PoS), Practical Byzantine Fault
Tolerance (PBFT), are chosen based on blockchain use
cases, performance requirements, and trust levels. The
application of PoA (Table. 1) in the Industrial Internet ofThings (IIoT) is notable, providing features such as asimplified block verification process, resulting in lower
block generation times, and consequently, improved
overall efficiency in industrial environments. X. Chen et
al. measured the latency performance of Internet ofThings (IoT) applications on private Ethereum
blockchains, focusing on two consensus algorithms:
Proof of Work (PoW) and Proof of Authority (PoA) [6].
The Study show that PoA Ethereum network has a lower
Block-Oriented Latency (BOL) than the PoW one due toa simpler block verification process [7].
Table 1. PoA Overview
PoA
consumes
lower
energycompared to other consensuss, making
it suitable for applications thatdemand efficiency.
Transaction
PoA provides higher transaction
Throughput
throughput, supporting a large volumeof real-time transactions, aligningwith the needs of industrial control
systems.
Applicability Particularly well-suited for industrial
control systems, meeting their strict
requirements for efficiency, real-time
capabilities, and security.

Fig.2. PoA Message Sequence Chart.

EnergyEfficiency

3.2. Data Analysis
By conducting tests in various scenarios using thesame Genesis Block configuration and same mining
setting to evaluate our blockchain performance [9].
Firstly, we will explore the multi-node scenario by
establishing multiple nodes to simulate a decentralized
environment, evaluating the system's performance in adecentralized setting. Subsequently, our focus will shift
to the multi-authority scenario, where we increase thenumber of authority nodes to assess system performancein a multi-authority context. We will emphasize themeasurement of Transactions Per Second (TPS) [10],employing different test cases and ensuring experimentrepeatability to guarantee result stability. Ultimately, we
will present the data distribution using box plots for
intuitive comparisons, coupled with statistical analysis togain deeper insights into the system's performance
characteristics under various conditions.

3. Methodology
3.1. System Design
By utilizing Proof of Authority (PoA) blockchain(Fig. 1) [8] as the record framework for the industrial
control system. PoA blockchain employs an authority©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 202419

Industrial Control System State

4. Results and Discussion
4.1. System Simulation
By emulating the communication between the
Human-Machine Interface (HMI) and the Programmable
Logic Controller (PLC). The HMI, serving as an
authoritative node, is responsible for block generation,transaction verification, as well as establishingconnections with the database (DB). On the other hand,
the PLC operates as a regular node, handling thesynchronization and reception of transactions throughclient RPC. (Fig. 3).

Fig. 4 TPS under different number of nodesuFig. 5 TPS under different number of validators
Fig. 3. Integration of blockchain with PLC.
The figures above illustrates that there is minimal
variation in Transactions Per Second (TPS) acrossscenarios with 1-3 nodes and 1-3 validators. We
hypothesize that the performance of Proof of Authority(PoA) may be more closely associated with the speed oftransmission rather than the number of nodes.

4.2. Experiment Result
We measured the service rate variations of a PoA
blockchain under the same conditions for other variables(Table 2). Specifically, we assessed the impact ofdifferent node counts (from 1 to 3) (Fig. 4) and different
validator counts (from 1 to 3) (Fig. 5) on the Transactions
Per Second (TPS). Each scenario underwent 30 test
iterations, and each test iteration included 1000
transactions.

5. Conclusion
In this study, we conducted practical experiments on
the record-keeping functionality of the Proof of Authority(PoA) blockchain in a real-world Cyber-Physical System(CPS) environment. Additionally, we evaluated theperformance of the PoA blockchain under different nodecounts and validator counts.
In the future, our focus will be on developing acomprehensive blockchain solution for CPS, addressing
the undeniable cybersecurity needs of criticalinfrastructure. We will also continue to monitor the
performance assessment of this solution.

Table 2. experiment setting
Genesis block
Period15 secs
Epoch
30000
Difficulty
0x010
Mining Setting
CPUIntel(R) Core(TM) i5-8250U CPU
Thread1
Software
PoA
Geth/v1.10.26-stableblockchain
e5eb32ac/windows-amd64/go1.18.5
ClientNethereum 4.18

Acknowledgements
This work is supported by the project from the Water
Resources Agency, Ministry of Economic Affairs undercontract number MOEAWRA1120307, and the National
Science and Technology Council (NSTC) under contract
number 112-2634-F-006-001-MBK in Taiwan. The
authors sincerely appreciate the assistance provided bythem.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
20

Powered by TCPDF (www.tcpdf.org)

Yun-Hao Chang , Tzu-En Peng, Jung-Shian Li, I-Hsien Liu

References
Mr. Tzu-En Peng
He is acquiring a master’s degree intheDepartment
ofElectricalEngineering/Institute of Computerand Communication Engineering,
National Cheng Kung University,
Taiwan. He obtained his B.S. degreefrom the Department of ElectricalEngineering, National Cheng Kung
University, Taiwan in 2022. Hisinterests are Cyber-Security and Industrial Control
Systems.

1. I-H. Liu, K.-M. Su, and J.-S. Li, “The Security Issue of ICS:The Use of IT Infrastructure,” Journal of Robotics, Networking
and Artificial Life, vol. 8, no. 1, p. pp. 29–32.
2. Z.-H. Sun, Z. Chen, S. Cao and X. Ming, “Potential
Requirements and Opportunities of Blockchain-Based
Industrial IoT in Supply Chain: A Survey,” IEEE Transactions
on Computational Social Systems, vol. 9, no. 5, pp. 1469-1483,
2022.
3. G. Puthilibai, T. Benil, S. Chitradevi, V. Devatarika, D. R.
Ashwin Kumar and U. Padma, “Securing IIoT sensors
communication using blockchain technology,” 2022
International Conference on Power, Energy, Control andTransmission Systems (ICPECTS), Chennai, India ,08 - 09 Dec.,
2022.
4. W. Zhou and J. Jin, “A Blockchain-Based Access Control
Framework for Secured Data Sharing in Industrial Internet,”2020 Eighth International Conference on Advanced Cloud and
Big Data (CBD), Taiyuan, China, 5-6 Dec., 2020.
5. Satoshi Nakamoto, “Bitcoin: A Peer-to-Peer Electronic Cash
System,”2009.
[Online].
Available:
https://bitcoin.org/bitcoin.pdf. [Accessed 2 Nov. 2023].
6. X. Chen, K. Nguyen and H. Sekiya, “On the Latency
Performance in Private Blockchain Networks,” IEEE Internetof Things Journal, vol. 9, no. 19, pp. 19246-19259, 2022.
7. I-H. Liu, Y.-C. Tsai, C.-F. Li and J.-S. Li, “Crossorganizational Non-repudiation Industrial Control Log System
Based on Blockchain,” Journal of Robotics,Networking andArtificial Life, vol. 9, no. 3, pp. 240-244.
8. C.-H. Wu, I-H. Liu, J.-S. Li and C.-F. Li, “Device's Operation
Tracking using Blockchain in Industrial Control System,”
ICAROB 2023, Oita, Japan, 9-12 Feb., 2023.
9. M. Schäffer, M. di Angelo and G. Salzer, “Performance andScalability of Private Ethereum Blockchains,” Business Process
Management: Blockchain and Central and Eastern Europe
Forum 2019., Vienna, Austria, 1–6 ,Sep., 2019.
10. Y.-C. Tsai, I-H. Liu and J.-S. Li, “Blockchain-based
Verification Mechanism for Industrial Control System,”
ICAROB 2022, Oita, Japan. 20–23,Jan. , 2022.

Prof. Jung-Shian Li
He is a full Professor in the
Department
ofElectricalEngineering, National Cheng Kung
University, Taiwan. He graduatedfrom the National Taiwan University,
Taiwan, with B.S. in 1990 and M.S.
degrees in 1992 in ElectricalEngineering. He obtained his PhD in
1999 in Computer Science from theTechnical University of Berlin, Germany. He teaches
communication courses and his research interests include
cybersecurity, cloud computing and network management.
He is currently involved in funded research projects
dealing with cybersecurity and critical infrastructure
protection. He is the director of Taiwan Information
Security Center @ National Cheng Kung University.
Prof. I-Hsien Liu
He is an assistant professor in
Department
ofElectricalEngineering, National Cheng Kung
University, Taiwan. He obtained his
Ph.D. in 2015 in Computer andCommunication Engineering fromthe National Cheng Kung University.
He teaches cybersecurity courses andhis interests are Cyber-Security,
Wireless Network, Group Communication, and Reliable
Transmission. He is the deputy director of Taiwan
Information Security Center @ National Cheng Kung
University(TWISC@NCKU).

Authors Introduction
Mr. Yun Hao Chang
He is acquiring a master’s degree inthe Department of ElectricalEngineering /Institute of Computerand Communication Engineering,
National Cheng Kung University,
Taiwan. His interests are blockchain
technology ,and Industrial Control
Systems.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
21

Enhancing Dam Security and Water Level Alerting with Blockchain Technology
YingCheng Wu

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 701401, Taiwan
Jung-Shian Li

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 701401, Taiwan
Chu-Fen Li

Department of Finance, National Formosa University
No.64, Wunhua Rd., Huwei Township, Yunlin County 632301, Taiwan
I-Hsien Liu*

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 701401, Taiwan
E-mail: ycwu@cans.ee.ncku.edu.tw, jsli@mail.ncku.edu.tw, chufenli@gmail.com, ihliu@cans.ee.ncku.edu.tw*
www.ncku.edu.tw, www.nfu.edu.tw

Abstract
Ensuring the security, monitoring, and timely alerting of water levels in dams is a major challenge. We use blockchain
technology to enhance the security and monitoring of dam infrastructure, and also improving the alerting system forwater level changes. The use of blockchain technology in dam infrastructure management provides a decentralized,
transparent, and tamper-resistant platform for storing and managing data. This ensures the integrity and security ofcritical data related to dam operations and water levels. This research investigates the enhanced security, monitoring,
and alerting capabilities that this integration offers, and aims to contribute to the improved security and efficiency of
dam infrastructure, leading to more reliable operations and better protection against potential disasters.
Keywords: Water Dam, Blockchain, Infrastructure Security

issues of water dams becomes important [3]. The
decentralized and tamper-resistant properties ofblockchain present a possible solution to enhance thesecurity and monitoring of dams to changing conditions.
We propose the network architecture and operational
workflow for the integrated system. The integration ofblockchain technology aims to enhance the security,
monitoring, and alerting systems of dam infrastructure,addressing current shortcomings and leading to morereliable operations and better protection against potential
disasters.

1. Introduction
Water dams play an important role as fundamental
components of critical infrastructure, fulfilling essentialfunctions in water resource management, flood control,and energy production. Recent cyber incidents,
exemplified by the reported data destruction attack on a
Ukrainian power plant by Russian hackers on April 8,2022 [1], underscore the vulnerabilities of criticalinfrastructure. This incident exemplifies the growing
threats faced by critical infrastructure worldwide [2],emphasizing the need for innovative approaches toenhance the security, monitoring, and responsiveness ofsuch infrastructure.

2. Background2.1. Regulatory and Policy

As demands on these dams increase due topopulation growth and climate change, ensuring security,
effective monitoring, and timely response to potential

Strategies for easing security risks must consider
regulatory and policy frameworks. This involves aligningsecurity measures with industry standards, governmental©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
22

YingCheng Wu, Jung-Shian Li, Chu-Fen Li, I-Hsien Liu

regulations, and cybersecurity best practices to ensure acomprehensive and effective approach. Regulations
should contain scenarios involving abnormal water levels.
Policies can ensure the implementation of advanced
monitoring systems capable of detecting sudden or
unexpected changes in water levels. This ensures timely
responses to mitigate potential risks associated withextreme water level variations.

operational data [5]. Manipulating these important data
sets could lead to inaccurate assessments, potentiallyresulting in inadequate responses to changing conditions
and an increased vulnerability to operational failures.
2.3. Blockchain Technology
As technological advancements continue to appear in
critical infrastructure, the integration of blockchain
technology is seen as a possible solution for enhancing
the security and transparency of dam operations.
Blockchain technology enhances the security of dam
infrastructure by providing a transparent and immutable
record of all transactions and data related to the operationof gates, water levels, and other important parameters.

Regulatory frameworks must not only recognize thepotential for abnormal water levels but also involve theimportance of consistency in gate status and water level
readings. These consistencies can arise from various
factors, including technical problems, sensor
inaccuracies, or even intentional manipulations.
Addressing these issues in policies is an important step
toward enhancing the resilience of dam infrastructure.

The decentralized property of blockchain ensures thatdata is distributed across a network of nodes, reducing therisk of a single point of failure or unauthorized access.
The tamper-resistant property of blockchain ensures theintegrity of data generated by sensors and monitoring
systems. This feature is important in relation tointentional data manipulations that could compromise thereliability of water level readings.

In the scenario of gate status and water level
inconsistencies, dam operators face the challenge ofaccurately assessing the true conditions of the dam. A
gate that is reported as closed in the control system, while
physically open, can lead to an underestimation of thewater level, posing risks of flooding downstream. On thecontrary, a reported open gate when it is closed might
result in an overestimation of water levels, potentially
impacting water resource management.

3. DamChain Architecture
The DamChain architecture is designed to enhance
the security and monitoring of dam infrastructure throughthe integration of blockchain technology. Our proposedarchitecture integrates the decentralized and tamperresistant features of blockchain to build an architecture
for managing important data related to dam operations
and water levels.

Regulatory and policy considerations play animportant role in addressing security risks in dam
infrastructure. By incorporating events such as gate
abnormal openings, water level fluctuations, equipment
reading problems, and extreme weather events,
regulatory frameworks contribute to a comprehensive
and adaptive approach to safeguarding criticalinfrastructure.

3.1. Network Architecture
The network architecture of the integrated system is
an important component depending on the design of theblockchain network. Choosing the right consensus
mechanism is important. Moreover, incorporating
blockchain technologies and a consensus mechanism canplay an important role in implementing the blockchain
network to the requirements of dam operations.

2.2. Data Integrity
The potential for unauthorized access andmanipulation of critical operational data poses a
significant security risk. Unauthorized individuals or
external malicious hackers may attempt to gain
unauthorized access to sensitive information related towater levels or operational parameters. This external
threat could lead to misleading dam operators,
compromise decision-making processes, and introducerisks to the safety and functionality of the dam [4].

We designed the network architecture in Fig. 1. The
network architecture demonstrates the application ofblockchain technology in the management of dam
infrastructure. By recording water level data and dam
operations on the blockchain, the system achievestransparent information storage and protection against
tampering. Operators can monitor the dam's status byobserving data on the blockchain.

The risk also extends to internal employees who
may intentionally manipulate operational data. This
insider threat poses unique challenges as it involves
individuals with legitimate access to the system. Internal
data tampering can mislead dam operators, compromisedecision-making, and lead to potential risks in the safety
and functionality of the dam. Implementing measures todetect and prevent both external and internal data
tampering is important for maintaining the integrity of©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
23

Enhancing Dam Security and4. Experiment
The integrated system uses Geth (Go Ethereum) as
the blockchain implementation and is hosted on a systemwith the following specifications in Table 1.

Processor
Operating
System
RAM

Fig. 1 Network Architecture

Table 1. System Specifications
Intel Core i5-7500 CPU @ 3.40GHz
Windows 10 Enterprise 64-bit
16GB

The experiment result is shown in Fig. 3. The result
is in an average transaction throughput of 115.575
transactions per second, and shows the evaluation of thesystem ability to handle and record dam operations within
the specified hardware and software environment.

3.2. Operational Workflow
An essential aspect of the regulatory model andoperational workflow is the continuous monitoring ofcompliance with established policies. This involves
regular audits and reviews to identify any differencesfrom the regulatory framework. The operational
workflow involves the interaction between the operator,
dam gate, water sensors, and the blockchain. The
flowchart in Fig. 2 illustrates the process.

Fig. 3 Transaction CountAn additional experiment was conducted to evaluate theaverage response time of the system to water level alert
events. The result indicates an average response time of
5.06425 seconds in Fig. 4. The experimental evaluationof the system provides reasonable insights into its
performance and capabilities within a real-world dam
management scenario.

Fig. 2 Operational Workflow

In this workflow, the operator operates the dam gate,
and the actions are recorded on the blockchain. Water
sensors also record data, and a smart contract evaluates
whether the water level is normal. If yes, no further action
is required; if no, the operator and all related staff are
notified.

Fig. 4 Response Time

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
24

Powered by TCPDF (www.tcpdf.org)

YingCheng Wu, Jung-Shian Li, Chu-Fen Li, I-Hsien Liu

5. Conclusion

Authors Introduction

The study demonstrates the possible benefits ofintegrating dam infrastructure management withblockchain technology. The integration not only
enhances security and monitoring but also shows thepossibilities for resilient and efficient operations. The
regulatory and policy frameworks underscored theimportance of continuous monitoring to address
unexpected challenges.

Mr. YingCheng Wu
He is acquiring a master’s degree in the
Department
ofElectricalEngineering/Institute of Computer andCommunication Engineering, National
Cheng Kung University, Taiwan. He
obtained his B.S. degree from the
DepartmentofCommunication
Engineering,
National
TaipeiUniversity, Taiwan in 2020. His interests are CyberSecurity and Blockchain.

The integration system introduces how importantdata related to dam operations and water levels ismanaged. The blockchain technology ensures data
integrity, providing an indisputable record of transactions.
This is important in critical infrastructure scenarios toprevent unauthorized access or data tampering. Moreover,the implementation of the system addresses the need forwater level alerting. These features enhance security andmonitoring to dam security and water level alerting.

This work was supported by the National Scienceand Technology Council (NSTC) in Taiwan undercontract number 112-2634-F-006-001-MBK.

Prof. Jung-Shian Li
He is a full Professor in the Department
of Electrical Engineering, National
Cheng Kung University, Taiwan. He
graduated from the National Taiwan
University, Taiwan, with B.S. in 1990
and M.S. degrees in 1992 in ElectricalEngineering. He obtained his PhD in
1999 in Computer Science from theTechnical University of Berlin, Germany. He teaches
communication courses and his research interests include
cybersecurity, cloud computing and network management.
He is currently involved in funded research projects
dealing with cybersecurity and critical infrastructure
protection. He is the director of Taiwan Information
Security Center @ National Cheng Kung University.

References

Prof. Chu-Fen Li

1. ESET Research, “Industroyer2: Industroyer reloaded This
ICS-capable malware targets a Ukrainian energy company,”
2022.
[Online].
Available:
https://www.welivesecurity.com/2022/04/12/industroyer2industroyer-reloaded/. [Accessed 15 Nov 2023].
2. CNN, “Russian military-linked hackers target Ukrainian
power company, investigators say,” 2022 2022. [Online].
Available:
https://edition.cnn.com/2022/04/12/politics/grurussia-hackers-ukraine-power-grid/index.html. [Accessed 15
Nov 2023].
3. I-H. Liu, C.-H. Wu, J.-S. Li, C.-F. Li, “Utilizing Blockchain
to Monitor the Functioning of Devices in Industrial Control
Systems,” Journal of Advances in Artificial Life Robotics, vol.
3, no. 4, pp. 205-208, 2023.
4. A. Parvizimosaed, H. Azad, D. Amyot and J. Mylopoulos,“Protection against Ransomware in Industrial Control Systems
through Decentralization using Blockchain,” 2023 20th Annual
International Conference on Privacy, Security and Trust (PST),
Copenhagen, Denmark, 21-23, Aug., 2023.
5. Y. Shah and S. Sengupta, “A survey on Classification ofCyber-attacks on IoT and IIoT devices,” 2020 11th IEEE
Annual Ubiquitous Computing, Electronics & Mobile
Communication Conference (UEMCON), New York, NY,
USA, 28-31, Oct., 2020.

She is an Associate Professor in the
Department of Finance at the
National
Formosa
University,
Taiwan. She received her PhD in
information management, financeand banking from the EuropaUniversität Viadrina Frankfurt,
Germany. Her current research interests include
intelligence finance, e-commerce security, financial
technology, IoT security management, as well as
financial institutions and markets. Her papers havebeen published in several international refereed
journals such as European J. of Operational
Research, J. of System and Software, International J.
of Information and Management Sciences, Asia J. of
Management and Humanity Sciences, and others.

Acknowledgements

Prof. I-Hsien Liu
He is an assistant professor in
Department of Electrical Engineering,
National Cheng Kung University,
Taiwan. He obtained his Ph.D. in 2015
in Computer and Communication
Engineering from the National Cheng
Kung
University.
Heteaches
cybersecurity courses and his interests
are Cyber-Security, Wireless Network, Group
Communication, and Reliable Transmission. He is thedeputy director of Taiwan Information Security Center @National Cheng Kung University (TWISC@NCKU).

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
25

MiniDAM: A Dam Cybersecurity Toolkit
Tzu-En Peng

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 701401, Taiwan
Meng-Wei Chang

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 701401, Taiwan
Yun-Hao Chang

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 701401, Taiwan
Jung-Shian Li

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 701401, Taiwan
I-Hsien Liu

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 701401, Taiwan
E-mail: tepeng@cans.ee.ncku.edu.tw, mwchang@cans.ee.ncku.edu.tw, yhchang@cans.ee.ncku.edu.tw,jsli@cans.ee.ncku.edu.tw, ihliu@cans.ee.ncku.edu.tw*
www.ncku.edu.tw

Abstract
Testbeds, serving as simulations of real-world scenarios, are of paramount importance for research in cybersecurity
related to critical infrastructure. In this paper, we aim to offer a comprehensive exploration of the MiniDAM and our
testbed, introducing its physical settings based on real dam operational standards. Furthermore, a comparative analysis
between the Secure Water Treatment (SWaT) testbed, MiniCPS, our testbed, and MiniDAM is presented. This paper
also includes insights into dataset generation and the integration of other functionalities. The exposition of
MiniDAM's features and capabilities serves as a foundation for enhancing resilience and provides valuable support
for advancing research within the broader field of dam-related studies.
Keywords: Dam Testbed, Critical Infrastructure, Industrial Control System, Cyber-Physical System, Cybersecurity

physical processes and network traffic within a system,
thereby enhancing overall system performance andresource allocation.
However, despite the advancements, numerous
vulnerabilities persist within the CPS of criticalinfrastructures, posing potential risks to public safety [4].
Dam facilities, in particular, experience failures andsecurity breaches annually. The cyberattack on the
Bowman Avenue Dam in 2013 underscored the potential
crisis posed by malicious actors targeting these systems.
Conversely, Some failures are attributed to anomalies in

1. Introduction
Critical infrastructures [1] have significantly
enhanced our quality of life over years of development.
Since the introduction of the Industry 4.0 [2] concept,
ensuring the security of Cyber-Physical Systems (CPS)[3] has become a primary objective across various
industrial sectors. CPS involves the seamless integrationof computation, networking, and physical processes.
Through CPS, we gain the capability to monitor both the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
26

Tzu-En Peng, Meng-Wei Chang, Yun-Hao Chang, Jung-Shian Li, I-Hsien Liu

2.2.2. MiniCPS

inflow, often triggered by extreme weather conditions.
For instance, the collapse of the Laos Dam and the
Sandford Dam failure in 2018 were both consequences ofheavy rainfall.
The security of dam CPS can no longer be ignored
after the tragedies that happened around the world. As aresult, a testbed with a toolkit that contains both physicaland network aspects of data for further research in a dam
scenario is needed.

MiniCPS is a toolkit built on top of Mininet [9] toprovide an extensible, reproducible research environment
for network communications, control systems, andphysical-layer interactions in CPS. MiniCPS was also
used to model the communication and control aspects of
SWaT while illuminating example applications.
MiniCPS focuses on high-fidelity network
emulation and being a framework for all fields of CPS.
However, the full-fledged physical process simulation
was considered to be out of the scope of MiniCPS since
it does not aim to be a performance simulator.
Furthermore, MiniCPS can only run on Linux operating
systems due to its reliance on Mininet. MiniCPS also put
very little emphasis on Visualization, such as Graphical
User Interface (GUI).
The issues mentioned above could pose challenges
to the reproducibility of research for researchers inspecific domains.

2. Research BackgroundOur research is primarily based on the concept ofCyber-Physical Systems (CPS), focusing on theinvestigation of the widely referenced SWaT Testbed [5]and MiniCPS [6]. Subsequently, we compare theirfeatures with our own testbed and toolkit.
2.1. Cyber-Physical System
CPS represents an innovative paradigm that
integrates computational algorithms and physicalprocesses to create intelligent and interconnected systems.
These systems enable seamless communication andcollaboration between the digital and physical worlds,
allowing for real-time monitoring, control, andoptimization of diverse applications.
R. Alguliyev et al. [7] illuminated the principle of
CPS operation and philosophical issues of CPS raised,and also proposed a tree of attacks on CPS. J. Shi et al.
[8] described and summarized the features of CPSs, thenthree classic cases were given with research challenges
and suggestions for future work.

3. Dam Cybersecurity Toolkit Architecture
For the physical part of our toolkit, we utilize the
PLCs that had been used in a real dam field environmentin Taiwan. We also obtained the dam history log which
consists of related information during the period, such as
water level, opening of the gates, gates inflow, gates
outflow, etc.

2.2. SWaT Testbed & MiniCPS
When it comes to testbeds and toolkits in the field of
CPS, the SWaT Testbed and MiniCPS are frequently
referenced.
2.2.1. SWaT Testbed
Secure Water Treatment (SWaT) is a water
treatment testbed for research in the field of cybersecurity.
SWaT consists of a six-stage process of water treatment
with frequently used industrial components, such as
Programmable Logic Controllers (PLCs) made by AllenBradley, Human Machine Interfaces (HMIs),
Supervisory Control and Data Acquisition (SCADA)
workstation, Historian, etc. The SWaT Dataset
systematically generated from the testbed is also
provided for CPS researchers to do further analysis andother related works.

Fig. 1. Architecture of The Dam Cybersecurity Toolkit

The architecture of the dam cybersecurity toolkit isshown in Fig. 1. We further process the history log withthe script based on the operational standards of the real
dam to generate the operation log which defines how
PLCs should work while facing the situation of thecorresponding history log. As shown in Fig. 2, we also
use our HMI to monitor the states of the PLCs, the overall
operation is recorded in the database, and Modbus/TCP
packets between HMI and PLCs are simultaneously©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
27

MiniDAM: A Dam Cybersecurity

captured using Wireshark for CPS researchers in the dam
field to conduct further analysis and experiments thatneed related dataset, such as machine learning [10].

Fig. 3. Modbus/TCP Packets Dataset

Fig. 2. MiniDAM HMI

4. Applications of The Dam Cybersecurity Toolkit
4.1. Dam Environment Simulation

Fig. 4. Comparison of The History Log and Dataset
regarding The Water Capacity Difference over The
Period of 2017 Summer in a Normal Scenario

In contrast to MiniCPS, MiniDAM places aheightened focus on Cyber-Physical Systems (CPS)
within dam field environments. Leveraging pertinent
Programmable Logic Controllers (PLCs) and historicallogs derived from actual usage in dam fields, we
meticulously construct a tailored Graphical User
Interface (GUI) to facilitate operational tasks. Thisapproach enables us to bridge the gap between theoreticalmodels and real-world dam scenarios. As a result, we
generate a diverse array of datasets corresponding tovarious operational scenarios. These datasets serve as
valuable resources for dam domain researchers, offering
them rich materials for in-depth analysis and explorationof the intricacies within dam environments.

5. Conclusion
In this study, we introduce MiniDAM’s physicalsettings based on the real dam operational standards andhow our toolkit differs from MiniCPS. Furthermore, we
illustrate the features of MiniDAM, including dataset
generation.
In the future, we are heading to build acomprehensive CPS interface for the dam operations toextend the testbed's [11] functions. Thoroughly define
how virtual and physical devices interface with ourtoolkit, taking into account hydrological information forbroader watershed analysis.

4.2. Dataset Generation
Acknowledgment
In addition to recording related information in the
database during operation. Modbus/TCP packets are also
captured using Wireshark. As shown in Fig. 3, HMI and
PLCs communicate with each other using Modbus/TCP
protocol. We can generate datasets for different nonnormal scenarios by modifying the logic within the script.
As illustrated in Fig. 4, due to significant variations
in water flow in Taiwan during the summer season, acomparison of the history log and dataset regarding thewater capacity difference over the period of the summerof 2017 was conducted within a normal scenario definedaccording to operational standards. We observed that thedataset and history log exhibit similar characteristics,
demonstrating the validity of our operation log and thesimilarity between the dataset and the actual field.

This work was supported by the National Scienceand Technology Council (NSTC) in Taiwan undercontract number 112-2634-F-006-001-MBK.
References
1. W. Liu and Z. Song, “Review of studies on the resilience ofurban critical infrastructure networks,” Reliability Engineering& System Safety, vol. 193, 106617, 2020.
2. M. Ghobakhloo, “Industry 4.0, digitization, and opportunities
for sustainability,” Journal of Cleaner Production, vol. 252,
119869, 2020.
3. E. A. Lee, “CPS foundations,” DAC '10: Proceedings of the
47th Design Automation Conference, Anaheim, California,
USA, 12-18 Jun., 2010.
4. J. M. Taylor and H. R. Sharif, “Security challenges andmethods for protecting critical infrastructure cyber-physicalsystems,” 2017 International Conference on Selected Topics in©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
28

Powered by TCPDF (www.tcpdf.org)

Tzu-En Peng, Meng-Wei Chang, Yun-Hao Chang, Jung-Shian Li, I-Hsien Liu

Mobile and Wireless Networking (MoWNeT), Avignon, France,
17-19 May, 2017.
5. A. P. Mathur and N. O. Tippenhauer, “SWaT: a water
treatment testbed for research and training on ICS security,”
2016 International Workshop on Cyber-physical Systems for
Smart Water Networks (CySWater), Vienna, Austria, 11-11
Apr., 2016.
6. D. Antonioli and N. O. Tippenhauer, “MiniCPS: A Toolkit
for Security Research on CPS Networks,” CPS-SPC '15:
Proceedings of the First ACM Workshop on Cyber-Physical
Systems-Security and/or PrivaCy, Denver, Colorado, USA, 1216 Oct., 2015.
7. R. Alguliyev, Y. Imamverdiyev and L. Sukhostat, “Cyberphysical systems and their security issues,” Computers in
Industry, vol. 100, pp. 212 - 223, 2018.
8. J. Shi, J. Wan, H. Yan and H. Suo, “A survey of CyberPhysical Systems,” 2011 International Conference on WirelessCommunications and Signal Processing (WCSP), Nanjing,
China, 09-11 Nov., 2011.
9. Mininet, “Mininet: An Instant Virtual Network on yourLaptop (or other PC),” 2023. [Online]. Available:
http://mininet.org/. [Accessed 10 12 2023].
10. O. Yavanoglu and M. Aydos, “A review on cyber security
datasets for machine learning algorithms,” 2017 IEEE
International Conference on Big Data (Big Data), Boston, MA,
USA, 11-14 Dec., 2017.
11. M.-W. Chang, J.-S. Li, I-H. Liu, 2023, “Cyber-Physical
Security Testbed for Dam Control System”, Journal ofAdvances in Artificial Life Robotics, Vol. 4, No. 2, pp. 63-66.

Mr. Yun-Hao Chang
He is acquiring a master’s degree in the
Department of Electrical Engineering/InstituteofComputerandCommunication Engineering, National
Cheng Kung University, Taiwan. Hisinterests are blockchain technology andIndustrial Control Systems.
Prof. Jung-Shian Li
He is a full Professor in the
Department
ofElectricalEngineering, National Cheng Kung
University, Taiwan. He graduatedfrom the National Taiwan University,
Taiwan, with B.S. in 1990 and M.S.
degrees in 1992 in ElectricalEngineering. He obtained his PhD in
1999 in Computer Science from theTechnical University of Berlin, Germany. He teaches
communication courses and his research interests include
cybersecurity, cloud computing and network management.
He is currently involved in funded research projects
dealing with cybersecurity and critical infrastructure
protection. He is the director of Taiwan Information
Security Center @ National Cheng Kung University.
Prof. I-Hsien Liu
He is an assistant professor in
Department
ofElectricalEngineering, National Cheng Kung
University, Taiwan. He obtained his
Ph.D. in 2015 in Computer andCommunication Engineering fromthe
National
Cheng
Kung
University.
Heteaches
cybersecurity courses and his
interests are Cyber-Security, Wireless Network, Group
Communication, and Reliable Transmission. He is thedeputy director of Taiwan Information Security Center @National Cheng Kung University(TWISC@NCKU).

Authors Introduction
Mr. Tzu-En Peng
He is acquiring a master’s degree inthe Department of ElectricalEngineering/Institute of Computerand Communication Engineering,
National Cheng Kung University,
Taiwan. He obtained his B.S. degreefrom the Department of ElectricalEngineering, National Cheng Kung
University, Taiwan in 2022. Hisinterests are Cyber-Security and Industrial Control
Systems.
Mr. Meng-Wei Chang
He was born in Pingtung, Taiwan in
1997. He is acquiring the master’s
degree in Department of ElectricalEngineering/Institute of Computerand Communication Engineering,
National Cheng Kung University in
Taiwan. He received his B.S. degreefrom the Department of Physics,
National Taiwan Normal University,
Taiwan in 2021. His interests are Cyber-Security and ICS
Security.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
29

A Case Study of Network-Based Intrusion Detection System Deployment in Industrial Control
Systems with Network Isolation
Nai-Yu Chen

Department of Electrical Engineering / Degree Program on Cyber-Security Intelligence,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 701401, Taiwan
Pei-Wen Chou

Department of Electrical Engineering / Degree Program on Cyber-Security Intelligence,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 701401, Taiwan
Jung-Shian Li

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 701401, Taiwan
I-Hsien Liu

Department of Electrical Engineering / Institute of Computer and Communication Engineering,
National Cheng Kung University
No.1, University Rd., East Dist., Tainan City 701401, Taiwan
E- mail: nychen@cans.ee.ncku.edu.tw, pwchou@cans.ee.ncku.edu.tw, jsli@ cans.ee.ncku.edu.tw,ihliu@cans.ee.ncku.edu.tw*
www.ncku.edu.tw

Abstract
Deploying intrusion detection systems is a common cybersecurity measure, and intrusion detection systems typicallyoperate at the ports of gateways. In critical infrastructure, industrial control systems often employ network isolation
strategies, lacking the role of gateways. This research primarily explores the deployment of the Snort intrusion
detection system in such an environment, combined with specific OT rules. Validation is conducted using the
cybersecurity testbed of the dam control system established by TWISC@NCKU in Taiwan. The results indicate thatby employing our proposed approach, it is possible to effectively detect abnormal network traffic, addressing thecommon issue of inadequate monitoring in environments with network isolation.
Keywords: Industrial Control Systems, Critical infrastructure, Network Isolation, Network-Based Intrusion
Detection System

Snort intrusion detection system within network-isolated
ICS [3]. Departing from conventional approaches, it
addresses challenges posed by network isolation, offering
nuanced network security tailored to ICS requirements.
Snort, typically installed at gateway, assumes a novel role
within network-isolated ICS. The research is motivatedby the recognition that existing IDSs may fall short ineffectively monitoring ICS within network isolation,
prompting a reevaluation of intrusion detection strategies.
To validate the approach, this study leverages the
TWISC@NCKU dam control systems cybersecurity
testbed in Taiwan, providing insights into real-world
applicability. The research's significance lies inaddressing insufficient monitoring in network-isolated

1. Introduction
The cybersecurity landscape in critical infrastructure,
particularly Industrial Control Systems (ICS), faces
escalating threats [1], exemplified by the recent CISA
AA22-103A alert. This alert, prompted by internal ICS
intrusion cases, underscores operators' vulnerability totargeted attacks, emphasizing the need for robust
cybersecurity measures [2]. Deploying Intrusion
Detection Systems (IDS) is standard, but challenges arise
in ICS scenarios with network isolation policies.
This study delves into deploying Network-Based
Intrusion Detection System (NIDS), focusing on the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
30

Nai-Yu Chen, Pei-Wen Chou, Jung-Shian Li, I-Hsien Liu

environments. Developing Snort rules to detect "Close
PLC Gate Controller" commands enhances ICS'scapacity to defend against potential threats, fortifying
critical infrastructure against evolving cyber threats. This
introduction establishes the groundwork for exploring
NIDS deployment in network-isolated ICSs, aiming toprovide effective and tailored cybersecurity solutions for
safeguarding ICSs.

Examining NIDS deployment in ICS reveals internal
attackmethodologies.
The
TWISC@NCKU
collaboration establishes a secure gateway control system
testing platform at the third layer, emphasizing the criticalrole of sensor readings on Human-Machine Interface(HMI) in dam operations. Our research introduces crosslayer attack methodologies tailored to dam operations,disrupting sensor readings through Modbus/TCP packet
injections to Programmable Logic Controllers (PLCs).
Focusing on countering malicious shutdowns,
specifically targeting PLC controllers, we develop
security measures and IDSs to fortify ICS against
evolving threats in complex network environments. This
study offers practical solutions to enhance ICSs' security.

2. Background2.1. Isolation of Industrial Control Networks
This study explores ICS cybersecurity through PurdueEnterprise Reference Architecture (PERA) [4], renowned
for its comprehensive approach. PERA's distinctivenesslies in its lack of a gateway component, challenging
conventional norms in critical infrastructure. Thisabsence leads to a nuanced examination of network
isolation strategies in ICS. While considering theIndustrial Demilitarized Zone (IDMZ), its inapplicability
due to the absence of a gateway is acknowledged.
Consequently, our research underscores the need toenhance security by introducing NIDSs between the first
and third layers of the PERA. Integrating PERA into
intrusion detection improves understanding, optimizes
deployment, and strengthens resilience against
inappropriate network behaviors in industrial settings.

3. ICS Network Intrusion Detection Methods
To explore the deployment of NIDS in ICS, wedeveloped a comprehensive framework encompassing
SNORT deployment, PLC, NIDS, and HMI
establishment. Initially, we utilized Twido to construct an
HMI, offering monitoring and control for the PLC(TWDLCAE40DRF). To simulate an attack, wedeliberately shutdown the controllers of PLC andemployed Wireshark to capture Modbus packets.
Subsequently, SNORT was implemented with a focus onconfiguring it as a NIDS.

2.2. Network-Based Intrusion Detection System
NIDS is intended to be deployed at the third layer,
specifically in the operational management segment, withthe database at its core storing information displayed on
the HMI. In this scenario, the objective of NIDS is tomonitor the operational management layer of ICS todetect potential intrusion threats.

Network-Based Intrusion Detection Systems (NIDS)
are indispensable for cybersecurity, providing vigilant
monitoring and timely alerts against potential intrusions
in network traffic. The deployment of intrusion detectionin ICSs, where network isolation is prevalent, poses a
significant challenge. This study investigates the viabilityof integrating Snort, an open-source Intrusion Detection
System, with tailored Operational Technology (OT) rules
for such environments. Our methodology seamlessly
incorporates Snort into ICSs, effectively addressing thecomplexities introduced by network isolation. Operating
on signature and rule-based matching, Snort adeptly
identifies abnormal network traffic and potential threats,solidifying its role as a reliable NIDS tool for ICSs in
network-isolated environments. This integration
significantly enhances cybersecurity by enabling thedetection and response to potential intrusions within theconstraints of network isolation.

Fig. 1. NIDS, PLC, HMI Integration Architecture
2.3. Internal ICS Attack Scenarios
During the integration of PLC and NIDS, we ensured
seamless communication and monitored PLC
communication to identify anomalies, with a particularemphasis on effectively detecting abnormal network
behavior in an isolated network environment (Fig. 1).
This comprehensive experimental procedure aims toassess the effectiveness of SNORT in ICS, contributingThis study explores NIDSs deployment in Industrial
Control Systems (ICS) within network-isolated
environments. Our investigation identifies vulnerabilities,
particularly in instances of command injection and IDE
access attacks [5], posing risks to the confidentiality,
integrity, and availability of ICS.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
31

A Case Study ofto an improved understanding of cybersecurity
enhancement in network-isolated environments.

is a 99.4% probability of detecting attack packets,demonstrating the robustness of the proposed approach.

In the pursuit of validating our proposed approach, weleverage the OT testbed provided by TWISC@NCKU.
This cybersecurity testbed, specifically designed for dam
control systems, serves as the foundation for our
experiments. Its realistic representation of industrial
control scenarios enables a practical examination of
SNORT's deployment within a network-isolated
environment.
Our investigation extends to the formulation of NIDS
rules, particularly focusing on the integration of SNORTwith tailored OT rules. The development of rules aims todetect attack techniques, with a special emphasis on
countering malicious shutdowns, specifically targeting
PLC controllers [6] (Fig. 2). This strategic developmentof rules contributes to fortifying ICS against evolving
threats, thereby enhancing overall system security.

Fig. 3. Background Traffic in the Environment
5. Conclusion
This study delves into the deployment of NIDS, with a
specific focus on Snort, in network-isolated ICS.
Traditional intrusion detection systems often facechallenges in ICS scenarios with network isolation,
where they typically operate at gateway ports.
Leveraging the cybersecurity testbed for dam control
systems at TWISC@NCKU in Taiwan, our researchvalidates Snort's efficacy in detecting abnormal network
traffic, addressing the common issue of insufficientmonitoring in network-isolated environments.

alert tcp any any -> any 502(msg:"Detected Modbus/TCP Packet with
Data '41ff00'"; content:"|00 00 00 00
00 06 ff 90|"; depth: 8;
content:"41ff00"; sid:10000004;)
Fig. 2. Snort Rule for PLC Shutdown Detection
4. Experiment

In the pursuit of enhancing the overall security of
network-isolated ICS, this study emphasizes thenecessity of adapting intrusion detection strategies to theunique challenges posed by critical infrastructure.
Looking ahead, future research aims to broaden the scopeby exploring additional attack scenarios. Due to current
limitations in experimental settings, there are plans toincorporate Engineering Working Station (EWS) andintroduce common IT attack scenarios to assess theireffectiveness. This expansion will provide a morecomprehensive understanding of the capabilities andlimitations of intrusion detection systems in safeguarding
network-isolated ICS against a diverse range of cyberthreats.

In the experimental phase, we meticulously configured
simulation parameters and established the experimental
environment, focusing on realistic attack scenarios
targeting Programmable Logic Controllers (PLCs). We
simulated a shutdown attack on the controllers of PLC,
capturing Modbus packets using Wireshark. Snort was
configured as a NIDS with specific OT rules [7]. All
experiments were conducted in the realistic dam control
system testbed provided by TWISC@NCKU at National
Cheng Kung University in Taiwan, ensuring authentic
and reliable results. The configuration aimed to evaluate
Snort's deployment effectiveness in industrial control
systems with network isolation.

Acknowledgements
Utilizing the established experimental environment, a
total of 30 experiments were conducted, each lasting 20
minutes and involving 0 to 20 instances of attacks. The
attacks followed the methodologies outlined in Section
2.3. Background traffic, consisting of normal HMI [8]and PLC communication, was present in the environment(Fig. 3). The results indicated that, with an average of4560 experimental packets, approximately 6.5 attacks
were detected, resulting in a detection rate of 99.4%. The
lowest detection rate, around 83%, occurred wheninterference devices were activated. In conclusion, there

This work was supported by the National Science andTechnology Council (NSTC) in Taiwan, under contract
number 112-2634-F-006-001-MBK, and the Water
Resources Agency (WRA) under the Ministry of
Economic Affairs (MOEA) in Taiwan.
References
1.

E. Abdulova and A. Kalashnikov, “Categorization andCriticality Assessment of Facilities of Critical
Infrastructure,” 2022 15th International Conference©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
32

Powered by TCPDF (www.tcpdf.org)

Nai-Yu Chen, Pei-Wen Chou, Jung-Shian Li, I-Hsien Liu

2.

3.

4.

5.

6.

7.

8.

Management of large-scale system development (MLSD),
Moscow, Russian, 26-28 Sep. 2022.
P. Hu, B. Yang, D. Wang, Q. Wang, K. Meng, Y. Wang
and Z. Chen, “Research on Cybersecurity Strategy and
Key Technology of the Wind Farms’ Industrial Control
System,” 2021 IEEE International Conference on
Electrical Engineering and Mechatronics Technology(ICEEMT), Qingdao, China, 02-04 July 2021.
D. Zhang and J. Wang, “Research on Security Protection
Method of Industrial Control Boundary Network,” 2021
IEEE Conference on Telecommunications, Optics andComputer Science (TOCS), Shenyang, China, 10-11
December 2021.
D. He, A. Lobov, L. E. G. Moctezumas and J. L. M. Lastra,“An approach to use PERA in Enterprise Modeling forindustrial systems,” IECON 2012 - 38th Annual
Conference on IEEE Industrial Electronics Society,Montreal, QC, Canada, 25-28 October 2012.
I-H. Liu, K.-M. Su and J.-S. Li, 2021, “The Security Issueof ICS: The Use of IT Infrastructure,” Journal of Robotics,
Networking and Artificial Life, Vol. 8, No. 1, pp. 29-32.
E. R. Alphonsus and M. O. Abdullah, “A review on theapplications of programmable logic controllers (PLCs),”
Renewable and Sustainable Rnergy Reviews, Vol. 60, pp.
1185-1205, 2016.
J. Luswata, P. Zavarsky, B. Swar and D. Zvabva,“Analysis of SCADA Security Using Penetration Testing:
A Case Study on Modbus TCP Protocol,” 2018 29th
Biennial Symposium on Communications (BSC), Toronto,
ON, Canada, 06-07 June 2018.
P. Papcun, E. Kajati and J. Koziorek, “Human MachineInterface in Concept of Industry 4.0,” 2018 World
Symposium on Digital Intelligence for Systems andMachines (DISA), 23-25 August 2018.

Prof. Jung-Shian Li
He is a full Professor in the
Department
ofElectricalEngineering, National Cheng Kung
University, Taiwan. He graduatedfrom the National Taiwan University,
Taiwan, with B.S. in 1990 and M.S.
degrees in 1992 in ElectricalEngineering. He obtained his PhD in
1999 in Computer Science from theTechnical University of Berlin, Germany. He teaches
communication courses and his research interests include
cybersecurity, cloud computing and network management.
He is currently involved in funded research projects
dealing with cybersecurity and critical infrastructure
protection. He is the director of Taiwan Information
Security Center @ National Cheng Kung University.
Prof. I-Hsien Liu
He is an assistant professor in
Department of Electrical Engineering,
National Cheng Kung University,
Taiwan. He obtained his Ph.D. in
2015in
ComputerandCommunication Engineering from the
National Cheng Kung University. He
teaches cybersecurity courses and his
interests are Cyber-Security, Wireless
Network, Group Communication, and Reliable
Transmission. He is the deputy director of Taiwan
Information Security Center @ National Cheng Kung
University(TWISC@NCKU).

Authors Introduction
Ms. Nai-Yu Chen
She is a postgraduate of Cloud andNetwork Security (CANS) Lab,
InstituteofComputerandCommunication
Engineering,
National Cheng Kung University in
Taiwan. She received her B.B.A.
degree from the Bachelor of
BioBusiness Management, National
Chiayi University, Taiwan in 2021. Her interests are ICS
Security and Network-Based Intrusion.
Ms. Pei-Wen Chou
She is a postgraduate of Cloud andNetwork Security (CANS) Lab,
InstituteofComputerandCommunication
Engineering,
National Cheng Kung University in
Taiwan. She received her B.B.A.
degree from the Department of
Healthcare
Administration
andMedical Informatics, Kaohsiung
Medical University, Taiwan in 2022. Her interests
encompass network security, blockchain, and industrial
control systems.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
33

The AI integration service innovation model of real estate industry in Taiwan
Li-Min Chuang

The Department of International Business, Chang Jung Christian University,No. 1, Changda Rd., Gueiren District,Tainan City, 711301, Taiwan
Chih-Hung Chen

The Ph.D. Program in Business and Operations Management, College of Management,Chang Jung Christian University,
No. 1, Changda Rd., Gueiren District, Tainan City, 711301, Taiwan
Email: liming@gmail.com, macuro6793@gmail.com

Abstract
This paper aims to explore the innovative model of integrating AI (Artificial Intelligence) services in the Taiwanesereal estate industry. The research employs literature analysis and the Fuzzy Analytic Hierarchy Process (FAHP) as themethodological approach. A FAHP questionnaire survey was conducted among real estate professionals in the Tainan
region of Taiwan. Through the calculation of relative weights among various dimensions, the study identifies key
factors related to the adoption of AI-based innovations by real estate agents in Taiwan. These findings serve as crucial
references for the real estate industry in transactional and operational management.
Keywords: Real Estate Agency, Artificial Intelligence (AI), Service Innovation, Fuzzy Analytic Hierarchy Process (FAHP)

dimensions.
(3)To draw conclusions and summarize the key factors in
the integration of AI-based service innovations in the
Taiwanese real estate industry. These findings serve asvital references for industry operational performance andbusiness development.

1. Introduction
Real estate transactions involve not only the buyers andsellers but also a crucial intermediary—the real estateagent. These agents play pivotal roles in propertytransactions, serving as key figures in advertising,marketing, and negotiation processes.
Quality real estate agents leverage effective
communication and professional marketing strategies tofacilitate successful transactions, guiding both parties
towards the common goal of closing deals.
This paper asserts that real estate agents consistently
improve overall marketing performance through various
innovative services. However, the study suggests that byincorporating an AI-driven integration of innovative
services, it is possible to identify key factors influencing
various sales strategies employed by real estate agents. The
research believes that uncovering these factors cansignificantly contribute to the operational performance ofreal estate agencies.
The objectives of this study are as follows:(1)To examine relevant literature and identify key factors
in service innovation, developing primary and secondary
dimensions.
(2)To integrate AI technology into the reviewed
dimensions and conduct a questionnaire study using the
Fuzzy Analytic Hierarchy Process (FAHP) to obtain therelative weights of four main dimensions and twelve sub-

2. Literature Review
Innovation has emerged as a pivotal force propelling thedevelopment of the service economy [1]. Within the realestate industry reports, there is a growing trend toincorporate elements such as digitization, newtechnologies, the sharing economy, tenant relationships, aswell as new services and business models [2], [3], [4].
AI has the potential to drive market capabilities, utilizing
various AI methods such as customer-oriented, industryoriented, and cross-functional integration approaches tobetter understand consumer demands. These methods arethen applied to the marketing behaviors of companies [5].
To streamline AI-driven innovation, service organizations
need to develop an integrated Dynamic Capability
Framework (DCF) that focuses on the ethical applicationof technology, talent, organizational culture, and structure
to optimize the entire service journey [5].
This paper gathers insights from domestic andinternational scholars and experts to explore literature
related to service innovation and the application of AI in©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
34

Li-Min Chuang, Chih-Hung Chen

the field of real estate. Through this review, the study
synthesizes and constructs the main dimensions of service
innovation, including customer engagement, businessmodels, technology integration, and process innovation.
The subsequent sections provide detailed discussions oneach of these dimensions.
At the "customer engagement" level, customers areparamount in the real estate industry. Real estate agents
must fully comprehend customer needs to seize crucialtransaction opportunities. Therefore, the concept ofcustomer engagement necessitates the integration ofvarious transaction methods, considering potential impacts
on company behavior. To achieve this, companies need tounderstand customer demands, identify the value
customers seek beyond normal transaction processes,
recognize customer roles, and anticipate the roles
customers and the company play in interacting with eachother [6].
At the "business model" level, business models havebecome essential means for the commercialization of
innovative ventures. They are perceived to provide aframework for companies to create and extract value frominnovative ideas or technological developments,emphasizing that innovation concepts or technologicaldevelopments alone do not represent any "single target
value." The commercialization of such concepts occurs
through business models [7], [8].
At the "technology integration" level, the rapid
development and widespread deployment of technology
integration form the foundation for many service
innovations. The significance of technology integration forservice-oriented enterprises and industries, as well as theimportance of service innovation, has long beenrecognized [9]. Understanding the four dimensions ofservice innovation—service concept, customer interface,service delivery system, and technology—many service
innovations involve some combination of these
dimensions [10].
At the "process innovation" level, revolutionary
approaches to enhance business performance must address
how to perceive and build businesses and how to improvethem. Businesses should not be viewed from theperspective of functions, departments, or products, butrather from the standpoint of key processes. Adopting aprocess-oriented approach means adopting the customer's
viewpoint. Processes are the structures for performing
operations required to create value for customers.
Therefore, a crucial measure of processes is customer
satisfaction with the process outputs, as they are theultimate arbiters of process design and ongoing
performance [11].
For real estate sales, the use of technology and digitization
has become a fundamental marketing method. As

information technology continues to advance, anincreasing number of real estate agents are incorporating
AI technology into their sales promotions. This involves
integrating various service innovations and streamlining
operational processes to address the dynamic demands ofthe real estate market, thereby strengthening business
operations and achieving profitability goals.
3. Constructing an AI-Integrated Service Innovation
Model in the Real Estate Industry
3.1. Research Framework
This study synthesizes the results of focus group
discussions to establish a determined research framework
for an AI-integrated service innovation model in the realestate industry. The developed framework comprises fourmain dimensions and twelve evaluation indicators, as
illustrated in (Fig. 1).
Figure 1 Research Framework

3.2. Research Subjects
To investigate the integration of AI technology in service
innovation within the Taiwanese real estate industry, thispaper focuses on real estate professionals in the Tainan
region as the primary study participants. Employing
purposive sampling, 12 scholars and industry experts
specializing in AI and the real estate industry were selectedto participate in the FAHP questionnaire. Prior to
conducting the FAHP survey, the researchers provided an
overview of the study's objectives to these participants andengaged willing real estate professionals in the FAHP
questionnaire survey operations.
4. Empirical Evidence of AI-Integrated Service
Innovation Models©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
35

The AI integration service

5. Conclusions

In this paper, the multiplication of the four evaluation
indicators by their respective dimensions results incomposite weights.
As can be seen from Table 1, the most importantevaluation indicators recognized by the 12 real estateindustry players are "customer acceptance of AI for sales
objects" (the comprehensive weight is 0.223), and "thebenefit of sales process innovation to the enterprise" (thecomprehensive weight is 0.173, ranking No. 2. Ranked
third is "The difference in customer perceptions of objects
using traditional copywriting or the introduction of AI"(the comprehensive weight is 0.101), ranking fourth is"The effectiveness of AI technology introduced into theservice innovation process" (the comprehensive weight is0.084), ranking The fifth is "The benefit of innovativetechnology to real estate agents" (the comprehensive
weight is 0.083), and the evaluation indicators ranked sixth
to twelfth are: "The acceptance of AI technology
innovation business model by marketers" (thecomprehensive weight is 0.067) , "Customer participationin using AI" (the comprehensive weight is 0.059), "Thebenefits generated by the introduction of AI technology
into the real estate business model" (the comprehensive
weight is 0.056), "The organizational application of AI
technology integration" (the comprehensive weight is0.054) , "Innovation completeness of enterprise
introduction process" (the comprehensive weight is 0.041),"The impact of the innovative model integrating AI
marketing on company operations" (the comprehensive
weight is 0.030), and "The participation of real estateagents in AI technology" (the comprehensive weight is0.029 ).
Table 1 The comprehensive weights for the AI-integrated
service innovation model

The four key findings of this study are listed below:
5.1. The most important evaluation indicator forcustomer participation is "Customer acceptance ofsales items imported into AI""Customer participation" is the most important aspectconsidered by real estate industry players, among which"customer acceptance of AI for sales items" is the mostimportant. For customers, no matter what kind of realestate property is needed or what sales method is used, theprimary evaluation factor is whether the customer canaccept and like it. It is also worth noting that no matter
what kind of technology is used to promote the sale ofobjects, the key point is to let customers fully understandthe characteristics, price, regional market and housing
conditions of the object, which is the best way to achieve
the final sale.
5.2. The most important evaluation indicator of process
innovation is "the benefit of sales process
innovation to the enterprise""Process innovation" is the second most important aspectconsidered by real estate industry players, among which"the benefit of sales process innovation to enterprises" isthe most important. Research shows that the vast majorityof real estate industry operators agree that continuous
innovation in sales processes, in addition to the benefits toenterprises, It is obviously helpful, but it will also be agreat help for marketers to implement business promotion.
The possible reason is that with the changes of the times
and in response to different customer needs, only
innovating the sales process in a timely manner caneffectively improve operational efficiency. In other words,
real estate industry players cannot just focus on sales
performance. Grasping the pulse of the market andestablishing a process innovation mechanism areindispensable key factors.
5.3. The most important evaluation indicator fortechnology integration is "the benefit of innovativetechnology to real estate agents""Technology integration" is the third most important aspectconsidered by real estate industry players, among which"the benefit of innovative technology to real estate agents"
is the most important. The main reason is that information
technology is changing with each passing day, and AI is
penetrating into every person. Today, the main purpose ofpromoting technology-based real estate brokerage is tocreate convenience, security, and innovation between
operators and consumers. At this time, it must rely heavily
on the integration and linkage of various technological©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
36

Powered by TCPDF (www.tcpdf.org)

Li-Min Chuang, Chih-Hung Chen

11. TH Davenport (1993). “Process innovation: reengineering
work through information technology”

knowledge and information equipment. For example,
through smart technology-related technologies, people canview houses remotely or use 360-degree panoramic views,
so that they do not need to visit the house in person. Instead,they can use technology to filter suitable objects to savemoney. time and improve sales performance.

Authors Introduction
Dr. Li-Min Chuang
He received the Ph.D. degree from the
National Cheng Kung University. He
is currently an associate professor anddepartment head at the Department ofInternational Business at Chang Jung
Christian University, Taiwan. His
research interests focus on strategic
management,
innovation
management, and service innovation.

5.4. The most important evaluation indicator of thebusiness model is "marketing staff acceptance of AI
technology innovation business model""Business model" is an aspect that real estate industry
players consider to be less important. Among which, "AItechnology innovation business model marketers'
acceptance" is the most important. The main reasonsshould be the age and acceptance of new technology
applications by real estate agents and marketers. There is alarge gap in levels, and not every colleague may be quickor familiar with the application mode of AI technology, andsome consumers have not popularized the operation
method of information technology for house viewing.
There are still most consumers who can only discover thehouse through on-site viewing. The actual problem lies, butwith the increasing popularity of smart systems andequipment, AI technology will gradually become more
popular in the future, and the public will also have a certain
degree of concept. The promotion of this business model
will still be a key point in the future.

Mr. Chih-Hung Chen
He received his Master’s degree fromthe EMBA, Chang Jung Christian
University, Taiwan in 2013. He iscurrently a Doctoral course student inChang Jung Christian University,
Taiwan.His research interests include
the application of information
technology to real estate industry.

References
1. Rust, R. T., & Huang, M. H. (2014). The service revolution andthe transformation of marketing science. Marketing Science,33(2), 206–221.
2. Westergren, U.H., Saarikko, T. and Blomquist, T. (2017), “theIoT guide – a business guide to the internet of things”, Industry
Report, Department of Informatics, Umeå University.
3. Fastighetsägarna Stockholm. (2018), “Lysande utsikter för
fastighetsbranschen”,
Industry
Report,Fastighetsägarna
Stockholm Publications.
4. FIBREE (2021), “Industry Report Blockchain Real Estate”,
industry report, FIBREE.
5. Akter S.et al.(2023), “A framework for AI-powered service
innovation capability:review and agenda for future
research.Technovation“. Technovation 125, pp. 102768.
6. S.C. Ng, J.C. Sweeney, C. Plewa (2020) “Customer
engagement: A systematic review and future research priorities”
Australasian Marketing Journal, 28 (4), pp. 235-252..
7. Chesbrough, H and S Rosenbloom (2002). The role of thebusiness model in capturing value from innovation: Evidence
from Xerox Corporation’s technology spinoff companies.
Industrial and Corporate Change, 11(3), pp. 529–555.
8. Chesbrough, H (2010). Business model innovation:
Opportunities and barriers. Long Range Planning, 43(2/3),
pp. 354–363.
9. Barras, R. (1986). “Towards a Theory of Innovation in
Services,”Research Policy (15), pp. 161-173.
10. Miles, I. (2008). “Patterns of Innovation in Service Industries,”
IBM Systems Journal, 47(1), pp. 115-128.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
37

A Study on Sales Patterns for Vegetable Products in Retail Stores
Yuhao Zhang1*, Shuangshuang Ma2, Jiashuai Wang3, Fengzhi Dai4, Lijiang Zhang5
College of Artificial Intelligence, Tianjin University of Science and Technology,

1300222，China
2

College of Light Industry Science and Engineering, Tianjin University of Science and Technology,
300222，China
College of Artificial Intelligence, Tianjin University of Science and Technology,

3300222，China
College of Electronic Information and Automation, Tianjin University of Science and Technology,
300222，China
5

Xinjiang Shenhua Biotechnology Co., Ltd, China
E-mail: *a15839176622@163.com
www.tust.edu.cn

Abstract
In fresh produce supermarkets, the shelf life of vegetable products is typically short, necessitating daily restocking
based on historical sales data and the formulation of a rational pricing strategy to maximize the store's profits. This
paper, based on sales data for vegetable products in a particular store from July 2020 to June 2023, employs various
analytical methods, including multidimensional analysis, clustering, and regression, to explore the interrelationships
among different types of vegetables. Furthermore, it combines cost-plus pricing and price elasticity models toestablish a pricing framework that optimizes revenue for the supermarket.
Keywords: vegetables for sale, vegetable pairing, cost plus pricing, pricing strategy

1. Introduction

purchase time, season, store space and other factors toreasonably replenish different vegetables. In addition, thestore should also consider the consumption time ofcustomers, the number of types of dishes, market
conditions and other factors to determine the price. The
research on reasonable replenishment and pricing ofvegetables has important guiding significance for the saleof vegetables in real life.
This paper will explore the distribution of different
vegetable sales from different dimensions, and combinedwith the cost plus pricing [1] and price elasticity model [2]to explore and set a set of pricing relationships tomaximize the profits of supermarkets.

Nowadays, with the continuous improvement of people's
living standards, fresh merchants are opening in various
places. In the sale of vegetable commodities, because thefresh-keeping period of vegetable commodities isrelatively short, and the product phase becomes worsewith the increase of sales time, it is of great significancefor merchants to replenish according to the historicalsales and demand of various types of vegetables in thepast.
In the actual operation of the store, the store should
consider the types of various vegetables, origin, shelf life,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
38

Yuhao Zhang, Shuangshuang Ma, Jiashuai Wang, Fengzhi Dai, Lijiang Zhang

The rest of this paper is organized as follows. The
second part studies the law of vegetable sales according
to the data of previous years. Section III introduces theconstruction of the pricing model. In Section IV, theevaluation model and the obtained results are analyzed.
Section 5 summarizes the main contents of this paper.
2. Rules of Vegetable Sales
Fig.2. weekday and non-weekday factors

The data statistics the flow details of vegetable products
sold in a store from July 1, 2020, to June 30, 2023, whichcontains six vegetable categories and the corresponding
single product sales volume and sales unit price. In thefollowing, the sales rules of vegetables will be analyzed
from three dimensions: quarterly, weekday andnon-weekday, and freshness.

It can be seen from Fig.2 that each category sells wellon Saturday and Sunday, and the sales volume is large,
which reflects that the average sales volume of the six
vegetable categories on non-working days is stronger
than that on working days.
2.3. Freshness of vegetables

2.1. The quarterly factor

In a day, from morning to evening, the freshness ofdishes gradually decreases. The average daily sales
volume of the six vegetable categories is divided into
three periods by time, and the sales volume of eachvegetable category in the morning, middle and lateperiods is summed up, to analyze the distribution of the
freshness of dishes in the store on the sales rule, The data
are visualized in Fig.3.

By summarizing the total sales volume of the six
categories of goods in each quarter, the visualized data
are presented in Fig.1.

Fig.1. quarterly factor chart

As can be seen from the Fig.1, the sales volume ofeach vegetable category shows periodic changes overtime. The sales volume of the third quarter of 2020, thethird quarter of 2021 and the third quarter of 2022 to thethird quarter of 2023 show the same trend. The trend
between the third quarter of 2021-the third quarter of2022 and the two cycles may be different because the
COVID-19 pandemic has affected sales.

Fig.3. freshness factor map of vegetables

It can be seen from Fig.3 that the sales volume of alldishes is the largest in the morning, followed by theafternoon, and the least in the morning. This result is
consistent with the actual life, and reminds the merchants
to prepare all the goods in the morning, and carefully
replenish the goods in the afternoon and evening.

2.2. Working days and non-working days
In real life, people often have meals at work on weekdays
and buy fewer vegetables every day. On non-working
days, that is, Saturday and Sunday, people are more
willing to buy more vegetables to eat. The average sales
volume of six types of vegetables on weekdays andnon-working days are shown in Fig.2.

2.4. The distribution law of sales volume of each single
product
Each major category of vegetables contains a variety ofsingle vegetable products. The single vegetable products
are divided into 7 categories by K-means clustering©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
39

A Study on Sales

method [3], and the trend of the average sales volume ofeach category of single product over the quarter is drawn,
which is shown in Fig.4.

maintaining the stability of prices. The calculation
formula of the cost-plus pricing method is as follows.
Price = unit cost + unit cost × cost margin(1)

3.2. Price elastic demand
Price elastic demand refers to the response degree ofquantity demanded to price changes, which is usually
expressed by the ratio of the percentage change inquantity demanded to the percentage change in price, thatis, the price elasticity coefficient of demand.
3.3. Buildup of modelFig.4. regular trend chart of single product sales

The average sales volume and sales volatility (standard
deviation) of each category of single product arecalculated, and combined with the quarterly sales trend of
single product, the sales rule of single product is shownin Fig.5.
Category

Average Sales Volume/KG

Sales Volatility(Standard Deviation)

First Category

About 663.58, at a
medium level

Second Category

About 754.04, also at a About 1211.53, highmedium level
volatility

Third Category

About 54.89,
significantly lower
than other categories

Fourth Category

About 2278.61,
significantly higher
than other categories

Fifth Category
Sixth Category

Seventh Category

About 2347.03, also ahigh sales volume
category
About 739.19, at a
medium level
About 1598.93, highsales volume

About 529.82,
relatively small

About 88.17, lowest
volatility

About 858.75

About 1203.68, highvolatility
About 1029.03, alsohigh volatility
About 2503.23, highest
volatility

The impact of establishing the selling price and costmarkup rate on sales volume for each vegetable category
is as follows.
𝑄𝑖 = 𝛼𝑖 + 𝛽1𝑖 𝑃𝑖 + 𝛽2𝑖 𝑅𝑖 + 𝜖𝑖
(2)Here, α𝑖 is the intercept term. β1𝑖 and β2𝑖 are thecoefficients representing the influence of selling price 𝑃𝑖
and cost markup rate 𝑅𝑖 on sales volume 𝑄𝑖 , and ϵ𝑖 isthe error term.
The price elasticity, denoted as β2𝑖 =𝐸𝑖 , is defined as:

Quarterly Trend
Sales volume is relatively
stable, without obvious
peaks or valleys
Sales volume has somefluctuations, with someseasonal effects
Sales volume is overall
low and stable

𝐸𝑖 =There are several quarters
with very high sales
volume, possibly seasonal
products
Sales volume fluctuates
greatly in different
quarters
Sales volume has somefluctuations
There are several quarters
with very high sales
volume, large seasonal
impact𝛥𝑄𝑖 /𝑄𝑖
𝛥𝑃𝑖 /𝑃𝑖(3)

To maximize the profit for supermarkets, we establish
a constraint equation for profit Π.
𝛱 = (𝑃𝑖 − 𝐶𝑖 × (1 + 𝐿)) × (𝐸 + 1) × 𝐷(4)Here, 𝑃𝑖 is the selling price, 𝐶𝑖 is the cost, 𝐿 is theloss rate, 𝐸𝑖 is the price elasticity's impact on sales
volume, and 𝐷 is the demand quantity.

Fig.5. summary of single product sales rules

3. Building A Pricing Model

4. Interpretation of Result

The purpose of establishing the pricing model is tomaximize the profit obtained by the merchants [1]. Firstly,
the purchase price and sales volume of each category ofvegetables in the future should be predicted according to
the previous data, and then the cost markup rate andelastic demand should be added for modeling.

According to the above model construction, six types ofvegetable commodity data are brought into the model andcalculated with Python to obtain the final answer.
4.1. Evaluation of the model
Advantages:(1) By comprehensively considering the relationshipbetween sales volume, cost markup and price elastic
demand, merchants can maximize profits. This
comprehensive consideration can help merchants morecomprehensively consider the impact of various factors
when formulating pricing strategies.

3.1. Cost-plus pricing
Cost-plus pricing is a method to determine the price of a
product based on the cost of the product, plus a certain
profit margin [2]. This method is simple and easy to
implement, which can ensure that the costs and profits ofenterprises are compensated, and is also conducive to©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
40

Yuhao Zhang, Shuangshuang Ma, Jiashuai Wang, Fengzhi Dai, Lijiang Zhang(2) The use of historical data for forecasting canprovide certain reference basis to help merchants make areasonable estimate of future sales volume and purchaseprice.
(3) The cost-plus pricing method is simple and easy to
implement and can ensure that the costs and profits ofenterprises are compensated, which is conducive to thestability of prices.
(4) Considering price elastic demand, that is, thesensitivity of demand to price, can help merchants better
understand the changes in market demand and makecorresponding adjustments when setting prices.
Disadvantages:(1) The model excludes the influence of special factors
such as weather and source of goods, which may have an
important impact on the sales volume and purchase price.
Therefore, in practical applications, it may be necessary
to further consider the influence of these factors.
(2) The model does not consider the pricing strategies
of competitors and the impact of market changes, which
may limit the applicability of the model in the real market
environment.

Authors Introduction
Mr. Yuhao Zhang
He is an undergraduate majoring in
artificial intelligence at Tianjin
Universityof
ScienceandTechnology. He learns about
electronic information and artificial
intelligence, and currently focuseson digital image processing, deeplearning and pose estimation.

Ms. Shuangshuang Ma
She is an undergraduate in packaging
engineering at Tianjin University of
Science and Technology, and hercurrent
research
directionispackaging machinery and intelligent
packaging technology.

Mr. Xuran Wang
He passed The National College
Entrance Examination in 2022 andstudied applied chemistry at Tianjin
Universityof
ScienceandTechnology. He is studying themolecular design and synthesis ofporous inorganic-organic frameworkmaterials.

5. Conclusion
Through THE ANALYSIS OF THE flow records ofvegetable commodities in a store, the sales relationshipbetween different vegetable categories is explored, andthe basic pricing model is obtained, which makes thestore maximize the profit, which will help the store
replenishment and pricing.

Dr. Fengzhi Dai
He received an M.E. and Doctor ofEngineering (PhD) from the Beijing
Institute of Technology, China in 1998
and Oita University, Japan in 2004
respectively. His main researchinterests are artificial intelligence,pattern recognition and robotics.
He worked in National Institute ofTechnology, Matsue College, Japan
from 2003 to 2009. Since October 2009, he has been thestaff in College of Electronic Information andAutomation, Tianjin University of Science and Technology,
China.

References
1.

2.

3.

Cyril Tomkins, Making sense of cost-plus transfer prices
where there are imperfect intermediate good markets by a'pragmatic-analytical'
perspective,
Management
Accounting Research, 1990, 1(3): 199-216.
Paik J, Baek S-J, Kim J-W, et al., Influence of Social
Overhead Capital Facilities on Housing Prices Using
Machine Learning, Applied Sciences, 2023, 13(19):10732.
Wang Qian, Wang Cheng, Feng Zhenyuan, et al., A
Review of the K-means Clustering Algorithm, Electronic
Design Engineering, 2012, 20(07): 21-24.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
41

Powered by TCPDF (www.tcpdf.org)

A Study on Sales

Mr. Lijiang Zhang
He has been chairman of Xinjiang
Shenhua Biotechnology Co., Ltd,
China and Zhongnong Guoyou
Technolegy (Tianjin) Co., Ltd,
China since 2019. His reaearchinterest is deep processing ofagricultural products.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
42

Research and Implementation of Cooperative Control for ROS Mobile Robot
Saijie Zhang, Huailin Zhao*
School of Electrical and Electronic Engineering, Shanghai Institute of Technology, Shanghai, 201418, China
E-mail: *zhao_huailin@yahoo.com
www.sit.edu.cn

Abstract
In this paper, the control of multiple mobile robots TurtleBot is studied for the robot operating system (ROS). A
distributed control and communication platform is built to solve the problem of data transmission between multiple
robots, and the configuration of robot workstations that can be monitored and remotely controlled is completed. On
this basis, the SLAM mapping and autonomous navigation of the robot are completed. The SLAM mapping algorithmof Gampping is selected. With the help of Gazebo simulation platform and Rviz visualization tool, the local path
planning of TEB is used to realize the autonomous navigation and multi-point navigation of the robot based on the
known environment. On the basis of the constructed distributed control system and the control of a single robot, theresearch on the coordination of multiple robots is further carried out, and the synchronous control of the robot isrealized by using a single node and multi-thread method. By changing the commands issued by the topic, the control
of multiple robots for different attributes is released synchronously to achieve more accurate and more synchronous
robot motion control. With the help of TF tools, multi-machine following and multi-machine formation are completed,and based on the path planning of a single robot, the navigation of multiple robots is realized.
Keywords: ROS, Multi-robot cooperation, Distributed control

1. Introduction
Robot Operating System (ROS) is an open source
standard for the robotics industry. It integrates sensors,
motors, and controllers into reusable modules through a
distributed messaging architecture to achieve mobility,
operation, navigation, and recognition tasks [1]. With thecontinuous development of the Robot Operating System(ROS), ROS has gradually entered various fields tocomplete various task requirements. However, when
dealing with some complex tasks, it still seems to be
somewhat inadequate, and sometimes it is necessary tocooperate with the staff to meet the requirements.
Therefore, the research on multi-robot cooperative control
is particularly important. The realization of multi-robot

collaboration can not only cope with complex work
scenarios and complete complex work requirements, butalso improve work efficiency and enhance the intelligenceof robots. Moreover, with the development of artificial
intelligence, robots have ushered in new development
opportunities. Based on the ROS multi-robot patrol
algorithm simulation and benchmark test framework, thedevelopment of multi-robot systems, virtual reality androbot interaction has become popular. In the future, ROS
will develop rapidly in the fields of navigation andpositioning, 3D object recognition, motion planning,
multi-joint manipulator motion control, machine learning,
human-machine interaction, robot collaboration and so on.
communication, because in the process of collaboration,task allocation, data transmission and information
generalization are involved. Robots need to communicate
frequently to obtain location information and working

2. Cooperative control of ROS mobile robot
The basis of collaboration between robots is two-way©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
43

Saijie Zhang, Huailin Zhao

status of other devices. Therefore, to realize thecooperative control of mobile robots based on ROS, we
first need to build a communication platform, and all
devices are connected to the platform, and use this as abenchmark for communication and wireless connection.
At the same time, in order to facilitate the management andmonitoring of all robot devices, a remote workstation
device needs to be set up. The workstation can remotelyconnect any robot device for management, and can also
monitor the working status and task progress of all robots.

the workstation, and eventually need to run on the client.
It is necessary to upload related directories and files fromthe workstation to the client. In fact, the design of thecontrol system of multiple robots often adopts the mode ofremote management of the workstation. After the networkand function of each robot client are configured for the firsttime, in order to facilitate the update of the robot firmware,
the processing when the operation is wrong and theiterative upgrade of the function package, it is also
convenient to monitor the whole system. A remote
workstation solves these problems. The problem is how torealize remote connection on the workstation and log in to
the client for management. ROS gives the answer, which
provides two remote control tools, ssh and vnc. Here only
introduce ssh this way.

2.1. Design of distributed control system
A distributed communication and control platform isbuilt for robots to provide network support for robot
collaboration.

SSH (Secure Shell) is an encrypted network protocol for
secure network services such as secure data
communication, remote command line login, and remotecommand execution between two networked computers. It
encrypts all data transmitted between two computers andprovides secure and authenticated communication over an
insecure network. It is commonly used by systemadministrators to manage remote servers and by
developers to securely access and transfer files between
machines. The whole process of data transmission istransparent and safe, using openssh tools will enhanceyour system security. The SSH implementationarchitecture is divided into two parts: client and server. The
client is the requester and requests to upload files. The
server is the receiver, receiving and downloading files
from the client. The workstation belongs to the client andsends data to the robot. The robot belongs to the server.
The implementation process is divided into four steps:
install SSH client and server respectively ; the server starts
the SSH service ; client remote login server ; realize datatransmission.

A. Static IP settings and master-slave IP configuration
In order to ensure the normal communication function
between robots and between robots and remote control
terminals (workstations), each device needs to beconnected to the same network, which can be provided by
the wireless signal of the router or maintained by the hot
spot service of the mobile phone. Because the general
wireless network will assign different IP addresses todifferent users, although different robot devices are in thesame network, because the IP address is different, and can
not communicate, so the network provider is required tomanually set up static IP and set up a multi-machinecommunication environment in the. bashrc file of eachdevice. The setting of static IP is very simple. It only needs
to change dynamic acquisition to manual acquisition in
IPV4, and set fixed IP address, default routing, subnet
mask and DNS.
ROS is a distributed computing environment. A running
ROS system can contain multiple nodes distributed on
multiple computers. According to the configuration of therobot operating system, multiple nodes between multiple
computers, different nodes running on the same computer
may need to communicate with each other at any time.
Therefore, ROS requires a complete two-way connection
between all robots on all ports for network configuration,and each computer must announce itself through a namethat all other computers can parse.

C. Optimization of ssh usage
The client needs to enter the password every time it logs
on to the server, and it also needs to enter the password
when uploading and downloading files, which actuallybrings great inconvenience to the transmission of files.
How to optimize the use of SSH, make the server trust theclient, give the client a permission without password
access, simplify the login steps with the help of the key,realize the secret-free login, and improve the efficiency ofthe operation. The implementation idea of optimization isto generate a pair of public key and private key. The private
key is stored locally, and the public key is uploaded to theserver. When each login, the local directly uploads theB. ssh Remote Connection
In the distributed architecture of multi-processors, filetransmission may be frequently involved between different
robot systems. For example, ROS programs are written on©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
44

Cooperative control of mobile robot

private key to the server. The server can find the matching
public key, and it is identified as a legitimate user. The ssh
connection is directly created, and the client does not needto enter the password.
D. Communication between robots
The ROS kernel (roscore) is the basis of ROS operation,
which has a parameter server. A running ROS has and only
has one ROS kernel, and everything on ROS depends on
this kernel [2]. ROS is a distributed framework based on
the development and integration of many Nodes (nodes).
Therefore, how to connect and manage many nodes is aproblem that ROS must solve, which introduces acommunication mechanism. In the communication
mechanism, ROS Master (master node, node manager)
manages and schedules the communication process
between nodes in the network, and provides a service for
configuring global parameters in the network. There arethree basic communication modes of ROS, namely topic
communication (publish-subscribe mode), service
communication (request-response mode) and parameter
server (parameter sharing mode).

Fig. 2 Intermediate nodes assist in publishing
Both schemes allow the robot to obtain the same motion
control. Scheme 1 needs to modify the code of the original
given controller. After modification, it is impossible tocontrol a single robot independently. Scheme 2 retains theoriginal control node and still retains the control
characteristics of a single robot. When multiple robots
need to be controlled, only one more intermediate nodeneeds to be run. Therefore, Scheme 2 is more practical.
2.3. Mobile robot queuing to follow

2.2. Synchronization control of mobile robot

In order to realize the following of multiple robots, thefollowing between two robots is considered first. If robot
A follows robot B, the relationship between the position
and attitude of the two robots must be known. Thenthrough the coordinate transformation, the coordinate
system transformation matrix between the two robots canbe obtained, as well as the displacement vector randomlytransformed to the pilot. The robot follows the simulation
as shown in Fig. 3.

There are two control schemes to realize thesynchronous control of robots. The first scheme is that thecontroller synchronously publishes multiple topics, andeach topic corresponds to the ns attribute of different
robots. The detailed process is shown in Fig. 1.

Fig. 1 The controller synchronously publishes multiple
topics
The second scheme is to create an intermediate node,
which receives the control instructions of the controller
and processes them. After adding the ns attributes ofdifferent robots, it is released to the robot for control. Fig.
2 shows the implementation of scheme 2.

Fig. 3 Multi-robot following simulation
As long as the relationship between the coordinates iscalculated in real time, ROS provides four methods toobtain the coordinate relationship of robot pose, which canbe obtained by SLAM mapping, odometer, global camera,
motion capture device and so on. After obtaining thecoordinates, it can be sent to the TF tool in ROS by
broadcast. Then subscribe to the relationship between the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
45

Saijie Zhang, Huailin Zhao

two robots in the TF tool. The distance and attitude angle
difference between the two rob real-time follow can beachieved through the TF tool. ots can be obtained by thepose relationship between the two robots. The speed andangular velocity instructions are issued by PID control torealize the following control of the robot.
The key to realize multi-machine following is to use TF(Transformations Frames) tool, which is a tool formonitoring and calculating robot pose information,
providing pose monitoring, coordinate transformation andother functions. In the robot operating system, the TF
toolkit includes Broadcaster, Listener, and TF conversion
tools. The broadcaster broadcasts the position relationshipbetween each robot and the world coordinate system to the
TF tool through the topic; listener obtains the relative
position of the two robot coordinate systems by viewing
the TF toolkit.

through the TF tool. The remaining formation aircraft track
the position and attitude of the specified virtual coordinate
system respectively, and the robot formation movement ofvarious formations can be realized. By using the TF tooland the basic communication method to publish the virtual
coordinate system of each specified position, the robot
formation problem is transformed into the tracking
problem of the target point. It is necessary to modify the
callback function in the control of the pilot robot to realize
the release of the specified virtual coordinates. Because thevirtual coordinate is relative to the position of the pilot
robot, it can be adjusted according to the required
formation. In this paper, three formation modes aredemonstrated, including horizontal line, triangle andsquare escort formation, and the position andtransformation of virtual coordinates of triangle formation
are given. They are shown in Fig. 5, 6 and 7.

2.4. Multi-robot formation of mobile robot
The principle of formation and following is that the TF
tool is needed to obtain the relationship between the robot
coordinate systems, but the following needs thetransformation of the coordinate systems of the pilot andthe random. The random moving target point is thecoordinate origin of the pilot. The pilot moves, and thecoordinate origin moves accordingly. TF will broadcast its
pose in real time to control the random speed and angularvelocity to achieve the purpose of real-time following. The
coordinate system and coordinate transformation are
shown in Fig. 4.

Fig. 5. Horizontal formation simulation

Fig. 6 Triangle formation simulation

Fig. 4 Formation coordinate system and coordinatetransformation
Compared with the following, the realization of theformation needs to establish a virtual coordinate system of
the specified position around the leader, and obtain thechange of the position and attitude of the virtual coordinate
system following the leader coordinate system in real time

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
46

Cooperative control of mobile robot

obstacles. Firstly, several rough paths that can reach thetarget point are taken as the original paths. Then, various
constraints are considered on each path, such as time,collision, speed, acceleration, etc., and these constraints
are used to construct the optimization problem. Finally, theoptimization problem of each initial path is solved, and abetter path than initialization is obtained. Then, a better
path is selected from the optimized path, and the result of
local path planning can be obtained.
B. Robot cooperative navigation
ROS multi-robot cooperative navigation is to complete
the cooperative work of multiple robots in the sameenvironment to achieve navigation and exploration of theenvironment. In ROS, multi-robot cooperative navigation
usually uses multiple nodes to achieve. Each robot has its
own navigation nodes, which can exchange information
with each other through ROS communication mechanism.
In cooperative navigation, robots need to understand eachother's position and state in order to coordinate theiractions. When implementing ROS multi-robot cooperative
navigation, the following aspects need to be considered:Communication mechanism: ROS provides a variety ofcommunication mechanisms, including Topic, Service andParameter Server. In multi-robot cooperative navigation,
topics and services are usually used to achievecommunication between robots.
Position estimation: The robot needs to accurately
estimate its position and posture. Laser radar, vision
sensors and other sensors can be used to achieve position
estimation.
Obstacle avoidance algorithm: In the navigation process,the robot needs to avoid obstacles and avoid collisions.
Cooperative planning algorithm: In multi-robot
cooperative navigation, robots need to coordinate theiractions to avoid mutual interference. Collaborativeplanning algorithms can be used to achieve collaborative
action.

Fig. 7 Square escort formation simulation
2.5. Cooperative navigation of mobile robot

Fig. 8 Robot cooperative navigation
Fig. 8 shows the scene of three robots navigating at thesame time in the same environment, and the three robots
will determine their own movement routes throughcooperation to avoid the risk of collision with each other,and move to the established target point in the shortest timeand most efficiently.
A. Robot path planning
Before path planning, it is necessary to build a grid mapof the robot 's working environment through SLAM
mapping [3], because the robot 's navigation process needs
to create target points on the map, and in the path planning
process, the path planned by the robot will be displayed on
the map in real time. Mapping algorithms are also diverse,
including unscented Kalman filter UKF-SLAM, FASTSLAM, Gmapping, etc [4], [5]. In this paper, theGmapping mapping algorithm based on lidar data isselected. The basic idea of Gmapping algorithm is toobtain the information of the surrounding environment
through lidar when the robot moves, and then construct a
map and locate the position of the robot through filteringand matching algorithms.
In this paper, the TEB algorithm [6] is used to avoid

3. Conclusion
Since the birth of ROS in 2007, the research anddevelopment of this distributed framework system withnodes as processing units and functional units hasflourished, and robot distributed control has great
prospects in the development of service-oriented robots
and warehouse logistics robots. As one of the populardevelopment directions of robots in the future, multimachine collaboration not only provides a solution to deal
with complex working environments in the working
process of robots, but also greatly improves the working
efficiency of robots. The working requirements of serviceoriented robots are not only the quality of service, but also
the efficiency of their work. With the fast-paced operationof society, we are mostly in a ' fast-food ' attitude towards©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
47

Powered by TCPDF (www.tcpdf.org)

Saijie Zhang, Huailin Zhao

life. The efficiency of a single robot is limited, and the
management of multiple robots is complex. Therefore, amulti-machine cooperative control system that can beautonomously managed, information shared, divided andautonomous, and autonomously coordinated is developed,that is, swarm intelligence. It meets the requirements ofhigh efficiency and simple management. Warehouselogistics robot is a system that

Authors’ Introduction
Mr. Saijie Zhang
He received the B.S. degree from Shanghai Instituteof Technology, Shanghai, China. His main researchinterests is automation.

pursues time efficiency. How to ensure its work efficiency
and achieve the required accuracy at the same time is moreconvenient for the management of multiple robots in thewhole system. Multi-machine coordination of distributed
control may give its answer. Let the robot coordinate
autonomously, and at the same time share data in real timeand allocate tasks, which is convenient for management
and efficient.
References
1.

2.

3.

4.

5.

6.

Dr. Huailin Zhao
He received his PhD from Oita University, Japan in
2008. He is a professor in the School of Electrical &Electronic Engineering, Shanghai Institute ofTechnology, China. His main research interests arerobotics, multi-agent system and artificial
intelligence. He is the members of both IEEE andSigma Xi.

Vincenzo DiLuoffo, William R Michalson & Berk Sunar.
(2018).Robot Operating System 2. International Journal ofAdvanced
Robotic
Systems(3).
doi:10.1177/1729881418770011.
Yahui Qi, Xiuzhen Wu, Qijie Chen, Shi Yan & Peng Zhang.
(2022). Implementation of multi-UAV cooperative control
based on ROS networking. (eds.) Proceedings of the Fifth
National Conference on Cluster Intelligence andCooperative Control in 2021 ( pp.104-107 ).
Chengkan Huang. (2020). vision-based laser SLAM AGV
initial positioning and path planning research (master 's
thesis,Zhejiang
UniversityofTechnology).
https://kns.cnki.net/KCMS/detail/detail.aspx?dbname=CM
FD202101&filename=1020439617.nh
Xuebing Wang, Qingsheng Guo, Yong Wang, Qizhi Liu&
Zhiwei Wei. (2019). Feature extraction and automatic
recognition methods for map images. Surveying andmapping and spatial geographic information (09), 28-32.
Jos é Carlos Rangel,Miguel Cazorla,Ismael Garc í aVarea,Cristina Romero-Gonz á lez & Jesus Mart í nez-G ómez.(2019). Automatic semantic maps generation fromlexicalannotations. Autonomous
Robots(3).
doi:10.1007/s10514-018-9723-8.
C. Roesmann, W. Feiten, T. Woesch, F. Hoffmann and T.
Bertram, "Trajectory modification considering dynamic
constraints of autonomous robots," ROBOTIK 2012; 7thGerman Conference on Robotics, Munich, Germany, 2012,
pp. 1-6.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
48

An OpenCV-based Method for Workpiece Residue Image Processing
Jiaxin Wang* , Hao He, Fangyv Liu, Fengzhi Dai
Tianjin University of Science and Technology, Tianjin, China
Email: * 13293455078@163.com

Abstract
Workpiece residue refers to a thin film formed on the surface of the workpiece during the machining process, due tofactors such as cutting fluid, chips, oil stains, etc., which affects the quality and performance of the workpiece. This
paper proposes an OpenCV-based method for workpiece residue image processing, aiming to achieve automatic
detection and analysis of workpiece residue. By building a workpiece image acquisition system, workpiece residue
images of different types and degrees are collected, and the proposed method is verified and evaluated. The
experimental results show that the proposed method can effectively detect and analyze workpiece residue, with highaccuracy and robustness, providing an effective means for workpiece quality control.
Keywords: Workpiece residue, OpenCV-based, Image processing, Automatic detection and analysis

residue is likely to exist. Finally, geometric shapemodel and machine learning algorithm are used forpost-processing to determine the location and size ofthe residue. This method can analyze the image in the
production process and automatically identify the
workpiece and residue. The experimental results
show that the method can detect the workpiece residue
effectively and improve the production efficiency andproduct quality.

1. Introduction
With the continuous development of technology,
artificial intelligence and machine learning have achieved
remarkable results in many fields. Especially in the
manufacturing industry, workpiece residue has a great
impact on product quality and production efficiency. In
order tosolve this problem, many researchers haveproposed a method of workpiece residue detection basedon image processing. These methods process and analyze
the image to identify the work parts and residues, so as torealize the automatic detection and classification ofresidues in the production process.
In the manufacturing industry, the strict requirements for
product quality make the detection of workpiece
residue an important task. Whether at home or abroad,many researchers are exploring how to effectively realize
the automatic detection of workpiece residue. However,existing methods often have some problems, such as poor
detection of workpiece and residue with complex shapes
and sizes, or require a large number of known samples
and expensive computational resources for training andlearning.
This paper introduces a method of detecting
workpieceresidue based on OpenCV image processing.
OpenCV (Open Source Computer Vision Library) [1] is a
widely used open source computer vision library. It
contains rich image processing and analysis functions [2],and provides many efficient and practical tools andtechniques, which are widely used in various image
processing fields. Firstly, the image is preprocessed,
including grayscale, binarization, noise removal andsmoothing, to extract the features of the workpiece andresidue. OpenCV's object detection algorithm and deeplearning model are then used to identify areas where

2. Methodology
2.1. Image preprocessing
The image is read and preprocessed using OpenCV
library. The pre-processing includes gray-scale,binarization, noise removal, smoothing, etc.[3], [4], inorder to extract the features better. Grayscale is theconversion of color images to grayscale images in orderto reduce computation and improve processing speed.
Binarization is the conversion of a grayscale image into abinary image to highlight the differences in the object andbackground in the image. Noise removal and smoothing
are designed to reduce noise and detail in images toimprove the accuracy and stability of target
detection. Some of the core code is shown in Fig.1.

Fig. 1 Preprocessing phase code

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
49

Jiaxin Wang, Hao He, Fangyv Liu, Fengzhi Dai

2.2. Feature extraction
In the pre-processed image, the features of theworkpiece and residue are extracted. These features may
include color, shape, texture, etc. OpenCV's feature
extraction algorithms (such as SIFT, SURF, ORB, etc.)[5] are used to extract these features. These algorithms
can automatically identify key points and featuredescriptors in images, thus improving the accuracy androbustness of target detection. At the same time, edge
detection is carried out to extract and draw the contour[4]. Some of the core code is shown in Fig. 2.

Fig.4 Post-processing phase code

2.5. Result display and output
Finally, the detected workpiece and residue are marked
on the image, and the detection result is output. The
qualified workpiece is marked with green "YES", and theimage of the unqualified workpiece with residue ismarked with red "NO".
3. Experimental Results and Analysis
3.1. Related work
In the field of image processing, many researchers haveproposed various methods of workpiece residue
Detection [7]. Among them, the method based on
template matching is a simple and commonly used method.
By matching the image to be detected with the standard
template, the method finds the matching area with thetemplate, and then identifies the workpiece and residue.
However, this method is poor for the detection of
workpiece and residue with complex shapes and sizes. To
solve this problem, some researchers have proposed an
approach based on machine learning [7], [8]. By training
and learning a large number of known samples, aclassifier is obtained, which can be used to classify andrecognize new samples. However, this method requires alarge number of known samples, and the training processof the classifier consumes a lot of time and computational
resources.
In order to verify the validity of the proposed method,
we conducted a series of experiments. The experimental
results show that the method can detect the residue
effectively and improve the production efficiency andproduct quality.

Fig.2 Feature extraction phase code

2.3. Target detection
Use OpenCV's object detection algorithms, such as
Haar cascades [5] or deep learning models [6], toidentify areas where residue may be present. Haar
cascade is an object detection algorithm based on featureclassification, which identifies objects in an image by
combining a series of simple features together. Deeplearning model is a kind of object detection algorithm
based on neural network. By training and learning a large
amount of data, a classifier can be used to classify andidentify new samples. Some of the core code is shown in
Fig. 3.

3.2. Experiment settings
We used a set of real artifact images as theexperimental data set. The images include artifacts ofvarious shapes and sizes as well as residues. The
proposed method is verified by experiments, and theresults are compared with traditional template matching
methods and machine learning methods.

Fig.3 Target detection phase code

2.4. Post-processing

3.3. Experimental result

After possible residue areas are detected, some postprocessing is performed to determine if these areas areindeed residue. For example, a geometric shape model isused to identify areas that do not match the shape of theworkpiece, or a machine learning algorithm is used toclassify and identify areas. Finally, the detected residue ismarked out and the detection result is output. Some of thecore code is shown in Fig. 4.

We use metrics such as accuracy, recall, and F1 scores toevaluate the performance of the approach presented inthis paper. Accuracy refers to the proportion of correctlyclassified samples to the total number of samples; Therecall rate refers to the proportion of the samples recalled©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
50

An Opencv-based Method forin the correctly classified samples to the total number ofreal samples. The F1 score is a harmonic average ofaccuracy and recall. Specific experimental results areshown in the following Table 1.

accurately identified and classified. Compared with thetraditional manual detection method, this method has
higher detection efficiency and accuracy, and caneffectively improve production efficiency and productquality. At the same time, the method has good
universality and expansibility, and can be widely used invarious production fields.

Table 1. Performance comparison of different methods
Methodaccuracy

recall rate

F1 scoreAcknowledgements
Template
Matching

0.82

0.76

MachineLearning

0.85

0.82

Thanks to the tutor and students for their guidance andhelp during the experiment and test. Thanks for theexperimental equipment and site provided by the school,
as well as the cooperation and support of relevant
enterprises. At the same time, we would also like to thankthe contributors and users of the OpenCV open source
community, who have provided rich image processing
functions and machine learning algorithms library tofacilitate and support our research. In the future work, wewill continue to improve the method and make greater
contributions to the production efficiency and productquality improvement.

0.79

3.4. Advantages
1. Real-time: The method can detect the workpiece
residue in theimage in real time, so as to find the
problems in the production process in time, improve
production efficiency and product quality.
2. High efficiency: The method can automatically
identify the workpiece and residue in the image, avoiding
the tedious and time-consuming traditional manual
detection, and improving the detection efficiency.
3. Accuracy: The method adopts advanced image
processing and machine learning technology, which canaccurately detect the workpiece and residue, avoiding thesituation of missing and false detection.
4. Versatility: The method can be applied to different
types and shapes of workpieces and residues, with strong
versatility, can be widely used in various production
fields.
5. Scalability: The method can improve the accuracy
and efficiency of detection through continuous learningand training, and can also expand its function andapplication scope by adding new features and algorithms.
In short, the method of detecting workpiece residue
based on OpenCV image processing can improve
production efficiency and product quality, reduce
production cost and risk, and has important practicalsignificance and application value.

References
1. OpenCV: Introduction
2. Image Processing and Analysis_15_Image Registration ： asurvey of image registration techniques——1992...
3. Getting started: Image processing with OpenCV
4. OpenCV image preprocessing common functions andprocesses
5. OpenCV image feature extraction: corner feature, Harris and
Shi-Tomas algorithm, SIFT/SURF algorithm, Fast and ORB
algorithm
6. How to implement object detection algorithm in OpenCV?
7. Chen Xiao, ZHENG Xiaoyan, Zhao Lei, et al. Research on
Surface defect image processing method of Workpiece basedon OpenCV. Mechanical Engineering & Automation,
2018, (6): 142- 144.
8. Machine Vision based Defect Detection Approach using
Image Processing

4 . Conclusion
This paper presents an OpenCV based image
processing method for detecting workpiece residue.
Through feature extraction and object detection of the
pre-processed images, the workpieces and residues can be©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
51

Powered by TCPDF (www.tcpdf.org)

Jiaxin Wang, Hao He ,Fangyv Liu, Fengzhi Dai

Authors Introduction
Ms. Jiaxin Wang
She is currently an undergraduate in
Tianjin University of Science andTechnology. She is majoring inrobotics engineering.Her research area
is about Image processing based on
OpenCV.

Ms. Hao He
She is currently an undergraduate in
Tianjin University of Science andTechnology. She is majoring inrobotics engineering.Her researcharea is about Image processing
based on OpenCV.

Ms. Fangyv Liu
She is currently an undergraduate in
Tianjin University of Science andTechnology. Her research area isabout intelligent robot and intelligent
control.

Dr. Fengzhi Dai
He received an M.E. and Doctor ofEngineering (PhD) from the Beijing
Institute of Technology, China in 1998
and Oita University, Japan in 2004
respectively. His main researchinterests are artificial intelligence,pattern recognition and robotics. He
worked in National Institute ofTechnology, Matsue College, Japan
from 2003 to 2009. Since October 2009, he has been thestaff in College of Electronic Information andAutomation, Tianjin University of Science and Technology,
China.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
52

On Nonblockingness Verification and Enforcement of Controlled Nondeterministic DiscreteEvent Systems
Xiang Ren*, Zipei Wang

College of Electronic Information and Automation, Tianjin University of Science and Technology, 300222 ，China
Email: *renxtust@163.com

Abstract
Discrete event systems, as an important kind of cyber-physical systems, have been widely used in engineering field.
In this paper, we first express the dynamics of a controlled nondeterministic discrete-event system (acronym is DES)
as an algebraic state-space representation using the semi-tensor product (STP) theory. And then, we discuss the
problems of state-based nonblockingness verification and enforcement of nondeterministic DESs. Specifically, we
obtain a criterion of verifying whether a given controlled nondeterministic DES is nonblocking. Further, we develop
an efficient matrix-based approach to enforce state-based nonblockingness. We illustrate the applications of the
proposed theoretical results using an example.
Keywords: Discrete event systems, nondeterministic, nonblockingness, semi-tensor product theory

The rest of this article is organized as follows. The
second section presents some basic notations, concepts
and a matrix-based expression needed in this paper. In thethird part, we formulate the state-based nonblockingness
property verification problem, and we show how the
proposed approach can be to applied to enforce
nonblockingness. In the fourth section, an example ispresented to illustrate the application of the proposed
approach. The fifth part summarizes the main content of
this paper.

1. Introduction
With the development of information and network
technologies, discrete event systems (DESs) havereceived considerable attention in information physics
systems for many years, synthesizing multidisciplinary
methods such as automatic control and computer science,
see, e.g., [1], [2]. Verification and synthesis are two main
research issues in the field of DESs. We refer the readers
to the recent survey papers [3] for more details.

2. Preliminaries
In the above-mentioned literature, the DESs of interest,in general, are deterministic DESs. The case of
nondeterministic DESs has not been considered widely
so far since they have more complex dynamic evolution
behaviors than deterministic DESs. It should be pointed
out that, however, there are many realistic plants thatneed to be modeled as nondeterministic DESs. Therefore,
how to model and analyze effectively the dynamics ofcontrolled nondeterministic DESs are still an interesting
topic for both computer and control researchers.

2.1. Notations
In this subsection, we introduce some notations, which
will be used in the sequel. n is the set of all vectors ofdimension n ; X is the cardinality of set X ; M mn is theset of m  n matrices; M (i , j ) is the (i, j ) element of matrix

M ; Col j (M ) is the j -th column of matrix M ; Col (M )is the set of all columns of matrix M ; 0n := 0, 0,, 0 ;n

In classical DESs, the nonblockingness is a basic andimportant problem in the study of DESs. Also, someefficient approaches have been proposed. For instance, in[4], the authors studied the problem of blocking detectionof DESs by means of language-based approach. In thispaper, we develop a new methodology to investigate how
to model controlled nondeterministic DESs by using the
STP of matrices. Based the proposed new model, we
further study the property verification and enforcement ofnonblockingness.

,1 ;  n := 0,0,

1n := 1,1,,0 ;  nk := Colk ( I n ) ,
T

0

n



n

1  k  n ; n :=  ,  ,
1
n

2
n

,  nn  ; n :=  n0 ,  n1 , ,  nn  ;L  M mn is a logical matrix (resp., generalised logicalmatrix) if Col (L)  m (resp., Col ( L)  m ). We denotethe set of m  n logical matrices (resp., generalised
logical matrix) by Lmn (resp., Lmn ); If matrix L Lmn ,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
53

Xiang Ren, Zipei Wang

then it can be expressed as L   m1 ,  m2 ,ibrieflyik  0,1,

denotedasL   m i1 , i2 ,ibe partitioned into two disjoint subsets, i.e.,  = cuc ,where c denotes the set of controllable events, uc,  min  and it is, in 

,where

denotes the set of uncontrollable events. We here assume
that all events in  are observable.

, m , 1  k  n .

2.2. Semi-tensor product (STP) of matrices

In general, we wish to adjoin a supervisor or acontroller S to interact with G in a feedback manner.
More precisely, the transition function of G can becontrolled by S in the sense that the controllable eventsof G can be dynamically enabled or disabled by S after
each transition. Formally, a state-feedback supervisor,
denoted by S , is a function S : X → 2c that determines
the set of events S ( x)  c to be disabled at each state

In this subsection, we recall the notions of STP andswap matrix. We refer the readers to [5] and/or [6] formore details on them.
Definition 2.1 ([5]): Let A M mn , B  M pq . The STP
of A and B is defined as
A・ B = ( A  It / n )(B  It / p ),(1)

x  X , while events not belonging to the set S ( x) remain
enabled at state x . The controlled system (or calledsupervised system) consisting of G and S , denoted by
S / G , is another nondeterministic finite automation
given asS / G = ( X , , S , X 0 , X m ),(4)

where t denotes the least common multiple of n and p ,i.e., t = lcm(n, p) ;  is the Kronecker product.
Definition 2.2 ([5]): A swap matrix Wm, n is a mn  mn
logical matrix, which is defined as
Wm,n =  mn [1, m + 1, 2m + 1, ,(n − 1)m + 1,

2, m + 2, 2m + 2, ,(n -1)m + 2,
, m, 2m,3m, , nm].

where X ,  , X 0 and X m are as defined above, S is thepartial transition function of S / G , i.e.,( x, ), if ( x) and  S ( x)(5)
S ( x, ) = undefined, otherwise.

(2)

Lemma 2.1([5]): Let X  m and Y  n be twocolumn vectors. Then
(3)
Wm, n XY = YX , Wn, mYX = XY .

We use the notation H ( x) to denote the set of feasible
events of G at state x . Thus supervisor S is called
permissible if for all x  X , S ( x)  H ( x) c . Note that
it is not difficult to see that nondeterministic DESs can beviewed as special case of controlled nondeterministic
DESs with S ( x) =  for all x  X . The controlled

2.3. System model
In this subsection we recall the formalism used in thepaper. More details on DESs can be found in [2].

nondeterministic DES S / G is depicted in Fig.1.

A DES is modeled as a nondeterministic finite
automaton (NFA) G = ( X , ,  , X 0 , X m ) , where X is thefinite set of states,  is the finite set of events calledalphabet or input symbols, X 0  X is the set of initialstates, X m  X is the set of marked states (or acceptedstates),  : X  → 2X is the partial transition
function( 2 X denotes the power set of X ), which
describes the system dynamics: given states x, y  X and

Fig. 1. The controlled nondeterministic DES S / G

an event    , y  ( x, ) means the execution of 

3. Nonblockingness of Controlled Nondeterministic
DESs

from state x takes the system to state y . Note that  ( x, )is undefined when the event σ cannot be executed fromthe state x .  ( x, )! denotes  ( x, ) is well-defined.
Obviously, the transition function can be extended to : X * → 2X in terms of  ( x, e) :=  ( (  ( ( x, e j ), e j ) ), e jt ) ,1

where e = e j1 e j2

3.1. Matrix-based expression of controlled
nondeterministic DESs
In order to obtain the matrix expression of the
dynamics of controlled nondeterministic DESs, let us
first give an equivalent description of the controlled
system (4). For the nondeterministic DES G , we define acontrol pattern as a Boolean function:  : c →{0,1} and2e jt * ， * denotes the set of finite

strings on the alphabet  , including the empty string ・ .
The objective of this paper is to investigate the controlled
nondeterministic DESs. In this regard, the event set  can©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
54

On Nonblockingness Verification and

we use the notation  = {0,1}c to denote the set of allwhere x(t ) and u (t ) have the same interpretation as in

Boolean functions on c .   is interpreted as follows:

Theorem 3.1; Fc is the TSM of (6), which is representedasFc =  F1c , F2c , , Fmc   Lnmn ,(12)for any c ,  () = 1 means that the control pattern allows  to happen, while  () = 0 means that the control
pattern  refuse  to happen. Note that it is convenient towhere the n  n matrix Fjc is defined as

extend each   to a function  :  → {0,1} by defining

1, if ns f (tn , mj )Fjc( s ,t ) = 
0, otherwise.

 () = 1 for each uncontrollable event uc . Further,
we define a partial function of the form f : X →  , called
the state-feedback control function or state-feedback for
short, that maps each state x in X into control pattern  ,3.2. Nonblockingness verification of controlled
nondeterministic DESs

i.e.  () = f (x)() . Thus the controlled nondeterministic
DES consisting of the state-feedback f , control pattern and nondeterministic DES G is described asGf = ( X , , f , X 0 , X m ),In classical DESs, the problem of nonblockingness
verification is a basic and important topic. A controlled
deterministic DES, also denoted by S / G is said to be
nonblocking if and only if L (S / G) = Lm (S / G) , where(6)

where X ,  , X 0 and X m are as defined above; f denotes

L (S / G) and Lm (S / G) stand for the language generated
and language marked by S / G , respectively. See, e.g., [2]the partial transition function of Gf ,which is defined as

( x, ), if ( x, )! and f ( x)() = 1(7)f ( x, ) =  undefined, otherwise.

for more details. As a generalization of deterministic
model, we now investigate the nonblockingness ofcontrolled nondeterministic DESs. We for simplicity use
a state-based description to present the notion of
nonblockingness. We call it state-based nonblockingness,while the nonblockingness defined in [2] is called thelanguage-based nonblockingness. Actually, these twoconcepts are equivalent.

For event e j (1  j  m) , we define a n  n matrix Fj as

1, if ns f (tn , mj )  mjFj ( s,t ) =  rtj , if ns (tn , mj )  mj
 0, otherwise,e j uce j c(13)(8)

Definition 3.1: Given a controlled nondeterministic
DES (6), a state x  X is called a nonblocking state if it isreachable from initial state set X 0 and it can reach awhere Fj is called transition structure matrix w.r.t. event

e j . Thus, the transition structure matrix (TSM) of
controlled nondeterministic DES (6) is defined as(9)F =  F1 , F2 , , Fm  ,where F is a n  mn symbol matrix.

marked state from X m . (6) is called nonblocking if its
each state is nonblocking.
Remark 3.2: 1) By Definition 3.1, we know that a stateof (6) is a blocking state if and only if it is not reachable
from the set of initial states, or it can not reach any
marked state.2) For the controlled nondeterministic DES(6), there exist two forms of blocking state, i.e., deadlock
state and livelock state. More specifically, x  X is adeadlock state of (6) if it is an unmarked reachable stateand H ( x) =  ; x  X is a livelock state of (6) if it can not

Theorem 3.1: Given a controlled nondeterministic
DES (6), the dynamics of (6) can be equivalently
described asx(t +1) = Fu(t )x(t ),(10)where F is the TSM of system (6), which is defined in(9); x(t ) = ( x1 (t ), x2 (t ), , xn (t ))T is the vector form ofstate at step t , xi (t ) denotes the number of different

reach any marked state and H ( x)   .

paths from the set of initial states to the state xi =  witha feasible event string of length t − 1 ; u(t ) m is vector
form of event at step t .
in

In what follows, we give a matrix-based criterion to
verifythenonblockingnessofcontrolled
nondeterministic DES (6). Consider the controlled
nondeterministic DES (6) with the matrix-based
expression (11), from the definition of the TSM Fc
presented in (13) and the viewpoint of graph theory, we

Proof: We here omit this proof.
Remark 3.1: When the state-feedback f is known, thedynamics of (6) can be equivalently described by thefollowing equationx(t +1) = Fcu(t ) x(t ),(11)mreadily know that the matrix  F jc stands for the
j =1

adjacency matrix of the directed graph representation of
the controlled nondeterministic DES (6). Thus, state©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
55

Xiang Ren, Zipei Wang

xq = qn is reachable from x p = np in t steps if and only ifof controlled nondeterministic DESs, since a controlled
nondeterministic DES, denoted by S/G, has a generated
language and a marked language associated it. In thissubsection, we present a matrix-based approach toenforcingnonblockingness
property
callednonblockingness enforcement hereinafter.

m( Fjc )t( q, p)  0 . In this regard, we define a matrix
j =1

n

m

 =1

j =1

M =:  ( Fjc ) , called reachability matrix ofcontrolled nondeterministic DES (6), where n = X ,Consider the controlled nondeterministic DES (6) withthe matrix-based expression (11), and we assume that,without loss of generality, system (6) is blocking. Using
Lemma 2.1, equation (11) can be rewritten as(17)
x(t + 1) = Fc x(t )u(t ),m =  . Using it, we have the following result.
Proposition 3.1: Given a controlled nondeterministic
DES (6) with the matrix-based expression (11), then
1) The set of states reachable from the set of initialstates X 0 is

where Fc = FcWn, m  M nmn , called the dual TSM of (6).
Partition matrix Fc into Fc = [ F1c , F2c ,R ( X 0 ) = { | M ( k , p)  0,   X 0 ,1  k  n}, (14)
k
n

n

m

 =1

j =1

p
n

, Fnc ] , where

Fi c  M nm , 1  i  n . Also, construct matrix:

where M =  ( Fjc ) is the reachability matrix of (6).

F trim = [F1trim, F trim
2 ,, F trim
n ],

(18)

where

2) The set of states that can reach X m isci
 F , if n  R ( X 0 ) R ( X m )Fi trim =  i 0nm , otherwise.

R ( X m ) = {kn | M ( q, k )  0, qn  X m ,1  k  n}, (15)(19)

where M is as defined above.
Based on the above-constructed matrix (18), we havethe following result.

Proof: We here omit this proof.
Remark 3.3: By convention, we assume that

Theorem 3.3: Given a controlled nondeterministic
DES (6) with the matrix-based expression (11), then thecontrolled nondeterministic DES described by thefollowing equation is nonblocking.
(20)
x(t + 1) = F trimu(t ) x(t ),X 0  R ( X 0 ) and X m  R ( X m ) hereinafter. Using
Definition 3.1 and Proposition 3.1, the following main
result on the verification of nonblockingness for thecontrolled nondeterministic DES (6) is verystraightforward.

where F trim = F trimW[ m, n] , F trim is defined in (18).

Theorem 3.2: Given a controlled nondeterministic
DES (6) with the matrix-based expression (11), then (6)is nonblocking if and only if the following condition
holds:(16)
R ( X0 ) = R ( Xm ) = X ,Proof: we here omit the proof of Theorem 3.3.
Remark 3.4: From Theorem 3.3, we can obtain that,when the DES (6) is blocking, the computational
complexity of enforcing the nonblockingness property byusing the proposed approach is also polynomial time, i.e.,o(n2 m) , where n and m are the numbers of state nodes
and event nodes, respectively. An example concerning
the supervisory control of a submarine guard system ispresented (see Example 4.1 below) to illustrate theeffectiveness and application of the proposed results.

where R ( X 0 ) and R ( X m ) have the same interpretation asin (14) and (15), respectively.
Notice that, from Theorem 3.2, it is not difficult to seethat using our approach to verify nonblockingness ofcontrolled nondeterministic DESs is very efficient andstraightforward in comparison to the existing ones in thesense that it involves only some basic matrix
manipulations that are completed via polynomial time.

4. Illustrative Example

3.3. Nonblockingness enforcement of controlled
nondeterministic DESs

In this section, we give an example to illustrate theproposed results.

We know that, for uncontrolled DESs, the trimness
property is closely related to the concept of
nonblockingness. For instance, when a given original
system does not satisfy the nonblockingness property, we
can to enforce it via trim operation to restrict the system’s
behavior, see, e.g.,[2]. It is also meaningful for the caseExample 4.1: Let us consider the supervisory control
of a submarine guard system. A sea region is divided into
several sections in terms of the shape of the seabed and it
is depicted in Fig. 2. Two submarines, namely, and,cruise and guard the sea region. has its own separate
repair port and home harbor, i.e., section No. 5 and No. 2,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
56

On Nonblockingness Verification andrespectively. has repair and home harbor in section No. 4.
Due to the seabed form and the sea underwater streams,
as well as different size of the submarines, possible andcontrollable transitions of the submarines between these
sections are also shown in Fig. 2.

According to lemma in reference [7], the dynamics ofG1 G2 (called the parallel composition of G1 and G2 ) canbe described by the following algebraic formx(t +1) = Fu(t )x(t ),where F = [F1 , F2 ,(21)

, F9 , F10 ] L15150 , i.e.,F = 15 [0,0,0,1, 2,3,0,0,0,0,0,0,0,0,0, 4,5,6,0,0, 0,0,
0,0,0,0,0,0,0,0,0,0,0,7,8,9, 0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,10,11,12,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,13,14,15,0,0,0,0, 0,0,0,0,0,0,0,0,0,0,
0,10,11,12,0,0,0,0,0,0,0,0,0,0,0,0, 4,5,6, 2,0,
0,5,0,0,8,0,0,11,0,0,14,0,0, 0,3,0,0,6,0,0,9,0,
0,12,0,0,15,0,0,0,1,0,0, 4,0, 0,7,0,0,10,0,0,13].

Using Lemma 2.1, system (21) can be rewritten asx(t + 1) = Fx(t )u(t ),(22)

where F = FW[15,10] L15150 .
, F15 ] , where Fk L1510 ,Partitioning F = [ F1 , F2 ,Fig. 2. Space guarded by the submarines.

1  k  15

.

Since

states900 3533 = 15and33   =  are forbidden states , all events attached
4 2
5 3

At the beginning of inspection, A is in section No. 5
and B in section No. 4. It is required that each submarine
can return its home harbor. Additionally, meeting ofsubmarines A and B in any section of the sea region
must be prohibited for safety reasons. The task of thesupervisory control is how to synthesize the cruise
trajectory of A and B to prevent the violation of therequirements described above.

11
15

2

9
to 15and 1115 should be deleted and each state that can
9reach 15or 1115 in 1 step by an uncontrollable event is also
forbidden state. To this end, by applying Proposition 3.1,
9
the sets of states reachable from 15
and 1115 in 1 step are
7
12
14
{15
, 15
} and {1215 , 15 } , respectively. The sets of states

69
8that can reach 15and 1115 in 1 step are {15 , 15 } and
8
14
{15
, 10
15 , 15 } , respectively. According to the conclusionof reachability analysis in reference[8], we have

Now we use the proposed approach to solve it.
According to Fig. 2, the dynamics of submarines A
and B can be modeled by two finite-state automata
shown in Fig. 3(a) and Fig 3(b), respectively. That is,G1 = ( X11, 1, x01 , X m1 ) with the initial state x10 = 5 , and the

10
10

4
10

3
10

9
10

9
10

 → ,  → ,  → ,  → ;  →12
15 ,
9
15

7
15

5
10

9
15

12
15

4
10

6
15

8
15

9
15

9
15

11
15

6
10

8
10

14
8
11
10
11
14
11
14
11
15 →15 , 15 →15 , 15 →15 , 15 →15 . Thus, state 15
9is also forbidden state. Analogous to 15
and 11
15 , wemarked state X m1 = {2} ,while G2 = ( X 2 2 , 2 , x02 , X m2 ) with

7
10

9
10

8
10

5
14
15
13
14
have 14
15 →15 , 15 →15 , 15 →15 .

the initial state x02 = 4 and the marked state X m2 = {4} .

Based on the above analysis, we construct matrix
Fc1 = [ F1c1 , F2c1 , , F15c1 ] ,
(23)
where

0 , i = 9,11,14
Fi c1 =  1510
Fi , otherwise.
Consequently, system (22) becomesx(t + 1) = Fc1 x(t )u (t ) ,Fig. 3. (a) represents a DFA model of the submarine A
cruise; (b) represents a DFA model of the submarine B
cruise.

0  , 3  , 1  ; a j −1
4
5

5
5

2
3

4
2
3
4
5
6
7
12 13 15R(15) = {115 , 15
, 15
, 15
, 15
, 15
, 15
, 10
15 , 15 , 15 , 15} . We know that

 (1  j  6) . While for G2 ,j10

identifying 4  , 3  , 0  ; bk
1
3

3
3

7+k
10



(25)

The next step to preserve nonblockingness of thecontrolled DES (25). Using Proposition 3.1, we canobtain easily that R (115 ) = {115 , 152 , 153 , 154 , 155 , 156 , 157 , 158 , 1015 , 1315} ,For the automation G1 , identifying 5 15 , 2 52 ,
3
5

(24)

states 158 , 1215 and 1515 are forbidden states. Thus, let(1  k  3) .

Fc2 = [ F1c2 , F2c2 ,, F15c2 ] ,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
57

(26)

Powered by TCPDF (www.tcpdf.org)

Xiang Ren, Zipei Wang

where

Acknowledgements

0 , i = 8,12,15
Fi c2 =  15c101
 Fi , otherwise.
Hence, the controlled system (25) becomesx(t + 1) = Fc2 x(t )u (t ) ,(27)

This work was supported in part by the Natural Science
Foundation of China under Grant 61903274.

(28)

References

Again, using Lemma 2.1, we rewrite system (28) isx(t + 1) = Fc3 u (t ) x(t ) ,

1.

(29)

where Fc3 = Fc2 W[10,15] , specifically,2.

Fc2 = 15[0,0,0,1, 2,3,0,0,0,0,0,0,0,0,0, 4,5,6,0,0, 0,0,
0,0,0,0,0,0,0,0,0,0,0,7,0, 0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,10,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,13,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,
0,10,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 4,0,0, 2,0,
0,5,0,0,0,0,0,0,0,0,0,0,0, 0,3,0,0,6,0,0,0,0,
0,0,0,0,0,0,0,0,1,0,0, 4,0, 0,0,0,0,0,0, 0,0].

3.

4.

5.

It is readily verified that the controlled DES (29)satisfies the requirements described above. Consequently,
the cruise trajectory of submarines A and B, by resortingto the algebraic representation (29), is depicted in Fig. 4.

6.

7.

8.

Y. F. Chen, Z. W. Li, K. Barkaoui, et al., “Compact
supervisory control of discrete event systems by Petri nets
with data inhibitor arcs,” IEEE Trans. Syst., Man, Cybern.,
Syst., vol. 47, no. 2, pp. 364–379, Feb. 2017.
C. G. Cassandras and S. Lafortune, Introduction to
Discrete Event Systems, 2nd Ed. New York: Springer,
2008.
A. Giua and M. Silva “Modeling, analysis and control of
discrete event systems: A Petri net perspective,” in Proc.
the 20th IFAC World Congress, Toulouse, France, Jul.
2017, pp. 1808–1819.
S. Abdelwahed and W. M. Wonham, “Blocking detection
in discrete event systems,” American Control Conference,
Denver, USA, Jun. 2003, pp. 1673–1678.
D. Z. Cheng, H. S. Qi, and Y. Zhao, An Introduction to
Semi-tensor Product of Matrices and Its Applications.
Singapore: World Scientific, 2012.
X. G. Han, Z. Q. Chen, Z. X. Liu, et al., “Calculation of
siphons and minimal siphons in Petri nets based on semitensor product of matrices,” IEEE Trans. Syst., Man,
Cybern., Syst., vol. 47, no. 3, pp. 531–536, Mar. 2017.
X. R. Xu, Y. G. Hong, and H. Lin, “Matrix approach tosimulation and bisimulation analysis of finite automata,”in proc. 10th Word Congress on Intelligent Control andAutomation, Beijing, China, Jul. 2012, pp. 2716–2721.
X. R. Xu and Y. G. Hong, “Matrix expression andreachability analysis of finite automata,” J. Control Theory
Appl., vol. 10, no. 2, pp. 210–215, 2012.

Authors Introduction

Fig. 4. The cruise trajectory of submarines A and B in
the submarine system.

Mr. Xiang Ren
He received his B.S. degree from
College of Electronic Information andAutomation, Tianjin University of
Science and Technology, China in
2021. He is currently a Master course
student in Tianjin University of
Science and Technology.

5. Conclusion
In this paper, we used matrix-based form to model the
dynamics of controlled nondeterministic DESs. Using it,
we investigated the verification of nonblockingness
property of controlled nondeterministic DESs. Also, thecriteria of verifying nonblockingness was presented interms of the methodology described in this paper.
Furthermore, the proposed approach can be applied to
enforce nonblockingness of controlled nondeterministic
DESs.

Ms. Zipei Wang
She received the B.S. degree from
College of Electronic Information
and Automation, Tianjin Universityof Science and Technology, China in
2023. She is currently a Mastercourse student in Tianjin Universityof Science and Technology.

We hope that our work can establish the foundation for
a new theory of supervisory control of controlled
nondeterministic DESs. Future work will investigate theproblem of synthesizing property-enforcing supervisor
for controlled nondeterministic DESs by means of the
proposed theoretical framework.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
58

Modeling and Reachability Verification of Controlled Nondeterministic Finite-State Automata
Zipei Wang*, Xiang Ren
College of Electronic Information and Automation, Tianjin University of Science and Technology, 300222，China
E-mail: *wangzipei0301@163.com

Abstract
In this paper, we investigate the modeling and state reachability of controlled nondeterministic finite-state automata(NFA). The key feature of a controlled NFA is to admit a controller (also called a supervisor) to intervene thebehavior of an original system. We first express the dynamics of a controlled NFA as an algebraic state-space
representation in the framework of the semi-tensor product (STP) of matrices. Then, the necessary and sufficientcondition for verifying state reachability of controlled NFA is presented. An explicit formula for calculating allpaths of any two states is derived. Finally, we use an example to illustrate the application of the proposed theoreticalresults.
Keywords: Discrete event systems, nondeterministic finite-state automata, reachability, semi-tensor product of
matrices

proposed approach. The sixth part summarizes the maincontent of this paper.

1. Introduction
Discrete event systems (DESs), also called plants, havereceived considerable attention within the automatic
control and computer science communities for many
years, see, e.g., [1], [2].

2. Preliminaries
2.1. Notations
In this subsection, we introduce some notations, which
will be used in the sequel. n is the set of all vectors ofdimension n ; X is the cardinality of set X ; M mn is

Nondeterministic plants can be viewed as a
generalization of deterministic plants. It is useful when
designing a system at a higher-level of abstraction so thatlower-level details of system are omitted to obtainhigher-level models that may be nondeterministic. It is
well-known that controlled DESs modeled by controlled
finite automata have more complex structures anddynamics than uncontrolled DESs. Therefore, how tomodel and analyze effectively the dynamics of controlled
nondeterministic DESs are still an interesting topic.

the set of m  n matrices; M (i , j ) is the (i, j) element of
matrix M ; Col j (M ) is the j -th column of matrix

M ; Col (M ) is the set of all columns of matrix M ;0n := 0, 0,, 0 ; 1n := 1,1,

,1 ;
n

n

1 k  n ; := Colk ( I n ) ,kn n0 := 0,0, ,0 ;Tnn :=  n1 ,  n2 , ,  nn  ;n :=  ,  , ,   ; L  M mn is a logical matrix
0n

In classical DESs, the reachability is a basic andimportant problem in the study of DESs. For instance, [3]studied the matrix expression and reachability verificationof finite automata (including deterministic andnondeterministic). The approach in these two papers are
based on algebraic state-space representation. In thispaper, we develop a new methodology to investigate how
to model controlled nondeterministic finite-state automata(NFA) using the STP of matrices.

1
n

n
n

(resp., generalised logical matrix) if Col (L)  m (resp.,

Col ( L)  m ). We denote the set of m  n logicalmatrices (resp., generalised logical matrix) by Lmn (resp.,
Lm n ); If matrix L Lmn , then it can be expressed as

L   mi1 ,  mi2 , ,  min 
L   m i1 , i2 ,The rest of this article is organized as follows. The
second section presents some basic notations andconcepts needed in this paper. In the third part, we
present a matrix-based expression for the dynamics ofcontrolled NFA. In the fourth section, we give amethodology of verifying state reachability of controlled
NFA, and an explicit formula of finding all paths of any
two states is also provided, if exists. In the fifth section,
an example is presented to illustrate the application of theand it is briefly denoted as, in  , where ik  0,1,, m , 1  k  n .

2.2. Semi-tensor product (STP) of matrices
In this subsection, we give some necessary basic
knowledge of the STP of matrices used in the paper.
Definition 2.1 ([4]): Let A  M mn , B  M pq . The
STP of A and B is defined as(1)
A・ B = ( A  It / n )(B  It / p ),©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
59

Zipei Wang, Xiang Ren

where t denotes the least common multiple of n andp , i.e., t = lcm(n, p) ;  is the Kronecker product.

after each transition. Formally, a state-feedback
supervisor, denoted by S , is a function S : X → 2cthat determines the set of events S ( x)  c to be disabled
at each state x  X , while events not belonging to the set
S ( x) remain enabled at state x . The controlled system(or called supervised system) consisting of G and S ,denoted by S / G , is another nondeterministic finite
automation given as(4)
S / G = ( X , , S , X 0 , X m ),where X ,  , X 0 and X m are as defined above, S

Remark 2.1: When n = p ， A・ B = AB . Hence, the
STP is a generalization of the standard matrix product.
Throughout this paper the matrix product is assumed tobe the STP. We mostly omit the symbol “ ・ ”hereinafter.
Definition 2.2 ([4]): A swap matrix Wm, n is amn  mn logical matrix, which is defined as
Wm,n =  mn [1, m + 1, 2m + 1, ,(n − 1)m + 1,

(2)

2, m + 2, 2m + 2, ,(n -1)m + 2,
, m, 2m,3m, , nm].
Lemma 2.1([4]): Let X  m and Y column vectors. Then
Wm, n XY = YX , Wn, mYX = XY .

n

is the partial transition function of S / G , i.e.,( x, ), if ( x) and  S ( x)(5)
S ( x, ) =  undefined, otherwise
We use the notation H ( x) to denote the set of feasible
events of G at state x . Thus supervisor S is called
permissible if for all x  X , S ( x)  H ( x) c . Notethat it is not difficult to see that NFA can be viewed as
special case of controlled NFA with S ( x) =  for allx  X . The controlled NFA S / G is depicted in Fig. 1.

be two(3)

2.3. System model
In this subsection we recall the formalism used in thepaper.
A nondeterministic DES is modeled as a NFA
G = ( X , ,  , X 0 , X m ) , where X is the finite set of states, is the finite set of events called alphabet or input
symbols, X 0  X is the set of initial states, X m  X is theFig. 1. The controlled NFA S / G

set of marked states (or accepted states),  : X  → 2
is the partial transition function( 2 X denotes the power
set of X ), which describes the system dynamics: given
states x, y  X and an event    , y  ( x, ) means
the execution of  from state x takes the system tostate y . Note that  ( x, ) is undefined when the event σcannot be executed from the state x .  ( x, )! denotes
X

3. Matrix-Based Expression of Controlled NFA
In order to obtain the matrix expression of the
dynamics of controlled NFA, let us first give an
equivalent description of the controlled system (4). For
the NFA G , we define a control pattern as a Booleanfunction:  : c →{0,1} and we use the notation ( x, ) is well-defined. Obviously, the transition

 = {0,1}c to denote the set of all Boolean functions on
c .   is interpreted as follows: for any c , () = 1 means that the control pattern  allows σ tohappen, while  () = 0 means that the control pattern
 refuse  to happen. Note that it is convenient toextend each   to a function  :  → {0,1} bydefining  () = 1 for each uncontrollable event uc .
Further, we define a partial function of the formf : X →  , called the state-feedback control function orstate-feedback for short, that maps each state x in X
into control pattern  , i.e.,  () = f ( x)() .Thus thecontrolled NFA consisting of the state-feedback f ,control pattern  and NFA G is described asfunction can be extended to  : X * → 2X in terms of (x, e) :=  ( (  ( (x, ej ), ej ) ), ejt ) ,where e = e j e j e j * ,
1

2

1

2

t

 denotes the set of finite strings on the alphabet  ,including the empty string ・ . The objective of this paper
is to investigate the controlled NFA. In this regard, theevent set  can be partitioned into two disjoint subsets,i.e.,  = c uc , where c denotes the set of controllable*

events, uc denotes the set of uncontrollable events. We
here assume that all events in Σ are observable.
In general, we wish to adjoin a supervisor or acontroller S to interact with G in a feedback manner.
More precisely, the transition function of G can becontrolled by S in the sense that the controllable eventsof G can be dynamically enabled or disabled by S

Gf = ( X , , f , X 0 , X m ),©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
60

(6)Modeling and Reachability Verification

where X ,  , X 0 and X m are as defined above; f denotes

where F is the TSM of system (6), which is defined in (9);x(t ) = ( x1 (t ), x2 (t ), , xn (t ))T is the vector form of state atstep t , xi (t ) denotes the number of different paths fromf

the partial transition function of G ,which is defined as

( x, ), if ( x, )! and f ( x)() = 1
f ( x, ) =  undefined, otherwise.

the set of initial states to the state xi = in with a feasible
event string of length t − 1 ; u(t ) m is vector form of
event at step t .

(7)

Similarly, the state-feedback f is called permissible if
for all x  X ,  H ( x) uc , we have f ( x)() = 1 .
There is a bijective correspondence between thestate-feedback f and the state-feedback supervisor S ,i.e., x  X and  H ( x) c , f ( x)() = 1 (resp.,

Let us assume (10) holds when t = p −1 , i.e.,x( p) = Fu( p −1) x( p −1) . And let u( p) = km for any
1  k  m . Then
Fu ( p) x( p) = Fk ( x1 ( p), , xn ( p))Tf ( x)() = 0 ) if and only if  S ( x) (resp.,  S ( x) ). In
this regard, we will use for simplicity the controlled
system (6) instead of system (4) to present our results.
Our objective, in this section, is to model the controlled
NFA in the framework of the STP of matrices.

n= ( Fk (1,i ) xi ( p),i =1identifying xi

n

F
i =1

of different paths from the initial state x(1) = in0 to statemj (1  j  m) , we call

in

t
n

f

represents the partial transition function of G =1

can reach state 

i
n

and 

i
n

definedf ( xi )(e j ) , i.e.,

By Theorem 3.1, we know readily that verifying
whether or not any two states of controlled NFA (6) arereachable can be determined by both the transition
function  defined in G and the state-feedback f .

For event e j (1  j  m) , we define a n  n matrix Fj as

where Fj

(8)

In particular, when the state-feedback f is known, thematrix F becomes a constant matrix. More concretely,
F is a generalised logical matrix. In this case, we
replace F with Fc . Therefore, by Theorem 3.1, we
have the following result.

is called transition structure matrix w.r.t.

event e j . Thus, the transition structure matrix (TSM) of
controlled NFA (6) is defined as
F =  F1 , F2 , , Fm  ,x j ( p + 1) :=  Fk ( j ,i ) xi ( p) ,which implies x( p +1) = Fu( p) x( p) . By mathematicalinduction, the proof is completed.

xi  X and e j c are the state and controllable eventof controlled NFA(6), respectively.

e j uce j ccan reach nj with the event u( p) = km .
i jrij = f ( xi )(e j ) where 1, if ns f (tn , mj )  mjFj ( s ,t ) =  rtj , if ns (tn , mj )  mj
 0, otherwise，with a feasible string of length p −1 ,Consequently, we definein (7). For brevity, we use the notation rij to denote thestate-feedbackx ( p) represents the sum of the number ofk ( j ,i ) i

l

f
c

j
m

p −1 , thenidifferent paths by which the initial state set x(1) =  n

be expressed equivalently as   (  ) , where fcwith a feasible string of length

F

i j

, mm  . Therefore, xs cf ( xt , e j ) can
s
n

i j

assumption, we know that xi ( p) stands for the number

in and mj the vector forms of xi and e j , respectively.
Thus the state set X and event set  can be identified
with n and m , respectively, where n = 1n , n2 , , nn 
and  m = 1m , m2 ,x ( p)=  Fk ( j ,i ) xi ( p),1  j  n,k ( j ,i ) i

where  j = {i |1  i  n, nj f (in , km )} .From the above, em  .To obtain the dynamics of (6),in (1  i  n) , e ji =1

Using (8), one has

Let us consider the controlled NFA(6), we assume
that the set of states is X = x1 , x2 , , xn  , and the set of
events is  = e1 , e2 ,n,  Fk ( n,i ) xi ( p))T .

(9)
Corollary 3.1: Given a controlled NFA (6) in which
the state-feedback f is known, the dynamics of (6) can bewhere F is a n × mn symbol matrix.
Based on the above representations and the STP ofmatrices, we can obtain the following result on the matrix
expression of dynamics of controlled NFA (6).

equivalently described by the following equationx(t +1) = Fcu(t )x(t ) ,where x(t ) and u(t ) have the same interpretation

Theorem 3.1: Given a controlled NFA (6), thedynamics of (6) can be equivalently described as(10)x(t +1) = Fu(t ) x(t ) ,as in Theorem 3.1; Fc is the TSM of (6), which isrepresented as©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
61

(11)

Zipei Wang, Xiang Ren

Fc =  F1c , F2c ,, Fmc   Lnmn ,Theorem 4.1: Given a controlled NFA (6) with its
dynamics (11), let X 0 = {in1 , in2 , , inl } and x* = qn be
the set of initial states and target state of (6), respectively.
Then1) x* = qn is reachable from X 0 in t steps if and(12)

where the n  n matrix Fjc is defined as

1, if ns f (tn , mj )Fjc( s ,t ) = 
0, otherwise.

(13)

only if there exists a positive integer k (1  k  mt ) such
that
(15)
qn  (Colk (( FcWn, m )t x(1))),

Remark 3.1: It should be pointed out that our approach
are also suitable for the controlled deterministic DESs in
the sense that deterministic DESs can be viewed as
special case of nondeterministic DESs. In particular,when the controlled DES (6) is deterministic, i.e.,
X 0 = 1 and ( x, )  1 for all x  X and    ,where x(1) represents the vector form of the initial stateliset X 0 , i.e., x(1) =  n .

 =1

the state that is reachable from initial state with a feasible2) Assume that Lqt ( X 0 ) denotes the set consisting of
all event strings of length t by which the initial state set
X 0 can reach x* = qn in t steps, thent −1*event string e =  e jk   is unique, which means thatk =1

there exists only one 1  i  n such that xi (t ) = 1 in

Lqt ( X 0 ) = {e = ek1 ek1(11). Namely, x(t ) n . In contrast, if there is no a stateekt  * there is k of satisfying(15) such that kmt = ek1 ek1

t −1*
that is reachable from initial state with e =  e jk   ,ekt },

where the feasible event string of e = ek1 ek1

k =1

easily obtained from the formula then we have x(t ) = 0n .

k
mt

(16)

ekt can be= ek1 ek1

ekt interms of Lemma 4.1.
4. Reachability of Controlled NFA

Proof: we here omit the proof of Lemma 4.1.

4.1. Reachability verification

From (11), we havex(t + 1) = Fc u (t ) x(t )We, in this subsection, verify the reachability property
of controlled NFA (6) by means of equation (11). We first
give the following lemma, which is crucial to find allpaths from the set of initial states to any target state forthe controlled NFA (6).
Lemma 4.1 ([5]): Let ・  = ti =1formulas  = Si ・ jimilm1 m2mtjimi, i = 1,2,l
m1 m2= FcWn, m x(t )u(t )= ( FcWn, m )2 x(t − 1)u (t − 1)u (t )= ( FcWn, m )t x(1)u(1)u(2)mt, then the

(17)

u(t )

= (( FcWn, m )t x(1)) ・ tj =1 u ( j ).

, t hold, where S1 = I m1  1m2  mt , S2 = [ I m2  1m3  mt , , I m2 1m3  mt ],

m2



 St −1 = [ I mt −1  1mt , , I mt −1  1mt ],
m2   mt −1

S
=
[
I
,
,
I
 t
mt
mt ].

m2   mt

Note that the following corollary gives a criterion to
verify whether any two states of a controlled NFA are
reachable or not. Also, an effective approach of finding
all paths of any two reachable states is provided.
(14)

Corollary 4.1: Given a controlled NFA (6) with its
dynamics (11), let x p = np and xq = nq be any two
states of (6). Then
1) xq = nq is reachable from x p = np in t steps if
and only if there exists a positive integer k (1  k  mt )such that(18)qn (Colk (( FcWn, m )t np )) .

To present the main results on reachability of thecontrolled NFA (6), we for brevity need to introduce thefollowing notation: let  = (a1 , a2 , , an )T be anonnegative column vector of dimension n , we define( ) := {kn | ak  0,1  k  n} . For instance, let2) Let Lqt ( p) be the set consisting of all event stringsof length t by which x p = np can reach xq = nq in t
steps, then = (1,2,0,3)T , then ( ) = {14 24 44 } .

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
62Modeling and Reachability Verification

Lqt ( p) = {e = ek1 ek1

ekt  * | there exist k of satisfying(18) such that kmt = ek1 ek1
where the event string e = ek1 ek1from 

kmt= ek1 ek1

ekt },

system, which means that equation (11) in [6] is incorrect.
We refer the reader to [6] for the interpretations of thesenotations. As a part of our work, these errors presented in[6] were corrected and the interesting issues mentioned in[6] were discussed systematically in a more generalized
DES.

(19)

ekt can be obtained

ekt in terms of Lemma 4.1.

Proof: Obviously, here we omit its proof.

5. Illustrative Example

Remark 4.1: From Corollary 4.1, we readily know that
if the controlled NFA (6) is deterministic, then thenumber of different paths from state x p = np to xq = nq

In this section, we use an example to illustrate theproposed results.
Example 5.1 : Let us consider the uncontrolled NFA,in t steps coincide with the number of k satisfying(18). In contrast, when the controlled NFA (6) is
nondeterministic, we for convenience assume that

 = ( 1 ,  2 ,

G = ( X , , , X 0 , X m ) depicted in Fig. 2, wherec = {e2 , e3} , uc = {e1 , e4 } , X 0 = {e1 , e4 } , X m = {x5} .

mt,  n ) := Col j (( FcWn,m )t np ) . Then theTheTstate-feedbackS

supervisor

is

given

as

S ( x3 ) = {e2 , e3} .

j =1

number of different paths from x p = np to xq = nq int steps is equal to  q .
4.2. Comparison with the existing approaches
There are several papers addressing modeling andreachability analysis in DESs in terms of the algebraicstate-space approaches, see, e.g., [3] and [6]. Amongthem, [3] investigated matrix expression and reachability
verification of finite automata in which all events arecontrollable and the cardinality of the set of initial statesequals 1. We know that the controlled DESs have more
complex structures and dynamics than the uncontrolled
DESs as an external control input is allowed to intervene
the behaviors of original systems. Again, they did notconsider to adjoin a supervisor to interact with an original
system. These restrictions significantly simplify the
behavior of the system. Obviously, their approaches aredifferent completely from the approach using in thispaper since we do not make the above-mentioned theserestrictions.

Fig. 2. An uncontrolled NFA.
We now verify the state reachability of the controlled
system S / G consisting of the uncontrolled NFA G
shown in Fig.2 and the state-feedback supervisor S .
By Corollary 3.1, the dynamics of the uncontrolled
system S / G can be expressed asx(t +1) = Fcu(t ) x(t ),(20)
where x(1) =  nj ,j =1,4Fc = 

[6] is most closely related to our work, but there aresome fatal errors therein. Concretely, the authors asserted
that they presented a sufficient and necessary condition to
verify whether or not any two states of a controlled DES
are reachable (see, e.g., Theorem 2 in [6]). Also, analgorithm was designed to find all paths from one state toanother one if they are reachable (see, e.g., Algorithm 1).
These results were obtained from Corollary 1 in [6]. It
should be pointed out that, however, Corollary 1
addressed the problem of which states are reachable froma given state and a given input string of length t for acontrolled DES, which means a contradiction with theprevious assertion. Furthermore, the state-feedback
control specification was defined as f : X → in [6]. In

0
0
0
0
0

0
0
0
1
0

0
0
0
0
0

0
0
0
0
0

0
0
0
0
0

0
0
1
1
0

0
0
0
0
0

0
0
0
0
0

0
0
0
0
1

0
0
0
0
0

0
0
0
1
0

0
0
0
0
0

0
0
0
0
0

0
0
0
0
0

0
0
0
0
0

0
0
0
0
0

0
0
0
0
1

0
0
0
0
1

0
0
0
0
0



.



When t = 1 , we haveCol1 ( FcW[5,4] x(1)) = (0,1,0,0,0)T ,Col4 ( FcW[5,4] x(1)) = (0,0,0,0,1)T ,Col j (FcW[5,4] x(1)) = (0,0,0,0,0)T , j = 2,3.

x2 and x5 arereachable from the set of initial states X 0 in 1 step, andthe corresponding event strings of length 1 are e = e1
By applying Theorem 4.1, statese1and e = e4 , respectively. Specifically, we have x1 → x2 ;this regard, rj = f ( F t 粐in u( j ))( j +1 ) is undefined ine4x4 → x5 .

(11) of [6] since F 粐 u( j) is not necessarily thevector form of a state for a controlled nondeterministic
t

0
1
0
0
0

i
n

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
63

Powered by TCPDF (www.tcpdf.org)

Zipei Wang, Xiang Ren

When t = 2 , we haveCol2 (( FcW[5,4] )2 x(1)) = (0,0,1,1,0)T ,Acknowledgements
This work was supported in part by the Natural Science
Foundation of China under Grant 61903274.

Col3 (( FcW[5,4] )2 x(1)) = (0,0,0,1,0)T ,Col14 ((FcW[5,4] )2 x(1)) = (0,0,0,0,1)T .
with all the other columns equal to 50 . By Theorem 4.1,

References
1. R. Jacob, J.J. Lesage, and J.M. Faure. Overview of discrete
event systems opacity: Models, validation, andquantification. Annual Reviews in Control, 41:135–146,
2016.
2. C.G. Cassandras and S. Lafortune. Introduction to disctete
event systems. Springer, Switzerland AG, 3rd Edition,
2021.
3. X. R. Xu and Y. G. Hong, “Matrix expression andreachability analysis of finite automata,” J. Control Theory
Appl., vol. 10, no. 2, pp. 210–215, 2012.
4. D. Z. Cheng, H. S. Qi, and Y. Zhao, An Introduction to
Semi-tensor Product of Matrices and Its Applications.
Singapore: World Scientific, 2012.
5. J. J. Wang, X. G. Han, Z. Q. Chen, et al, “Model matchingof input-output asynchronous sequential machines based on
the semi-tensor product of matrices,” Future Generation
Computer Systems: The International Journal of eScience,
vol. 83, pp. 468-475, 2018.
6. Y. Y. Yan, Z. Q. Chen, J. Y. Yue, et al, “STP approach tomodel controlled automata with application to reachability
analysis of DEDS,” Asian J. Control, vol. 18, no. 6, pp. 1–
10, Nov. 2016.

states x3 , x4 and x5 are reachable from X 0 in 2
steps, and the corresponding event string of length 2 are
3e = 162 = 1424 , e = 162 = 1424 or e = 16= 1434 , and4 2
e = 1416 = 4 4 , respectively. Specifically, we havee1e2

x1 → x2 → x3 ;
e4

e1

e3

e1

e2

x1 → x2 → x4

orx1 → x2 → x4 ; ande2x4 → x5 → x5 .
When t = 3 , we have
Col5 (( FcW[5,4] )3 x(1)) = (0,0,0,1,0)T ,Col8 (( FcW[5,4] )3 x(1)) = (0,0,0,0, 2)T ,Col12 ((FcW[5,4] )3 x(1)) = (0,0,0,0,1)T ,Col54 (( FcW[5,4] )3 x(1)) = (0,0,0,0,1)Twith all the other columns equal to 50 . Using Theorem
4.1, states x4 and x5 are reachable from X 0 in 3
steps, and the corresponding event string of length 3 are
1 3 4
e = 564 = 142414 , e = 864 = 142444 or e = 12
64 = 4 4 4 ,Authors Introduction

4 2 2
and e = 54
64 = 4 4 4 , respectively. Specifically, we have
e1

e2

e1

e1

e2

e1

e1

e2

e4

e1

e3

e4

e4

e2

e2

Ms. Zipei Wang

x1 → x2 → x3 → x4 ; x1 → x2 → x3 → x5 ;She received the B.S. degree from
College of Electronic Information
and Automation, Tianjin Universityof Science and Technology, China in
2023. She is currently a Mastercourse student in Tianjin Universityof Science and Technology.

x1 → x2 → x4 → x5 , x1 → x2 → x4 → x5 ;
x4 → x5 → x5 → x5 .
The cases of t  4 are similar to t = 1,2,3 , the details
are omitted here for space limitations.
Mr. Xiang Ren

6. Conclusion

He received his B.S. degree from
College of Electronic Information
and Automation, Tianjin Universityof Science and Technology, China in
2021. He is currently a Master course
student in Tianjin University of
Science and Technology.

In this paper, we proposed a new framework that is
matrix-based form to model the dynamics of controlled
NFA. Using it, we investigated the verifications ofreachability property of controlled NFA. Also, the criteriaof verifying this property was presented in terms of themethodology described in this paper.
Future work will concentrate on investigating the
problem of synthesizing property-enforcing supervisor forcontrolled nondeterministic DESs by means of the
proposed theoretical framework.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 202464

Detection and Identification of Daylily Maturity Based on YOLOv8
Fangyan Li

College of Electronic Information and Automation, Tianjin University of Science and Technology,
300222，China
E-mail: 3162129540@qq.com
www.tust.edu.cn

Abstract
To better apply object detection and identification techniques from deep learning to the field of agricultural
automation, this paper focuses on the growth process of daylilies. It employs the state-of-the-art YOLOv8 model toachieve accurate assessment of daylily maturity. The backbone network of YOLOv8 draws inspiration from the
CSPDarkNet network structure to extract image features. The introduction of cross-stage connections in the middle
layers of the network enhances the efficiency of feature propagation. The neck section adopts the PAFPN
bidirectional channel network to further process the features extracted by the backbone network, facilitating thesmooth transmission of information from the bottom to the top layers. The Head section utilizes the Decoupled-Headstructure to generate the output for object detection. The trained model had a mAP which reached up to 94%, an
accuracy rate up to 95.6%, a recall rate up to 90.5%. Moreover, the identification speed is significantly improved.

Keywords: Deep Learning, Object Detection, YOLOv8, Daylily

1. Introduction

CNN model implemented by VGG16, which demonstrated
good detection performance for fruit images under
different time and lighting conditions. Liang [3] proposed
improvements on the SSD network based on VGG16 and
ZFNet for real-time mango detection. The enhanced
algorithm achieved higher accuracy compared to Faster RCNN, although the deep network structure led to slower
computation speed. Zhen [4] introduced a vegetable image
classification method, utilizing the Caffe open-source deeplearning framework. They improved the VGG network byadding a Batch Normalization layer to the VGG-M
network, enhancing the convergence speed and accuracyof the network (VGG-M-BN) with a vegetableclassification recognition accuracy of up to 96.5%. Xue[5] aimed to improve the accuracy of immature mango
detection in orchard scenes. They modified the YOLO V2
algorithm, developing a tiny-yolo network structure by
combining multiple feature layers. This addressed thechallenging issue of detecting overlapping and occluded
parts of mangoes, achieving a detection accuracy ofWith the gradual maturity of AI technology, the focus ofagricultural harvesting is gradually shifting to automation.
The agricultural sector has been continuously exploring
how to leverage modern computer vision technology toenhance crop production efficiency and quality. In the
production and management of agricultural products,
understanding the maturity of plants is crucial forachieving optimal harvesting and resource utilization. As
an important edible plant, the timely monitoring andaccurate assessment of the maturity of daylilies are
essential for agricultural production [1]. Traditional
harvesting of daylilies typically relies on manual
inspection and experiential judgment, which poseschallenges in terms of high labor costs and low efficiency.
With the rapid development of computer vision and deeplearning technologies, object detection techniques have
gradually become an effective means to address this issue.
For instance, Song [2] constructed and trained a Faster R-©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
65

Fangyan Li

97.02%. Zhao [6] proposed an apple localization method
based on the YOLOv3 deep convolutional neural network.
This method ensures the detection of apples in complex
environments while balancing efficiency and accuracy.
The main contributions of this paper include theconstruction of a large-scale daylily image dataset,
detailed annotation of daylily features at different growthstages, and the design of an architecture and training
methodology based on the YOLOv8 model. Throughexperiments, the proposed approach has been
demonstrated to exhibit high accuracy and practicality inthe automatic detection and identification of daylily
maturity.

daylilies, characterized by unopened flower buds, exhibit
deep green or green coloration. These buds are relatively
small and not ready for harvesting. Harvestable daylilies
have bright yellow-green buds, larger in size, and fully
developed, ready for picking. Overly mature daylilies have
either opened or wilted buds, displaying bright yellow or
dark yellow coloration with petal-like structures. The
annotation process is shown in Fig. 1. The marked data is
a JSON file, which is converted to a txt file . The dataset
was split into 80% for training and 20% for testing andvalidation collectively. The testing set was utilized toassess the model's generalization capabilities post-training,
while the validation set was employed for fine-tuning
hyperparameters and conducting preliminary assessmentsof the model's performance.

2. Data Acquisition and Annotation
2.1. Data acquisition
The datasets for this experiment are all derived frommanual field shooting with an f/1.6 wide-angle camera.
Image categories encompass immature daylilies, mature
daylilies, and overly mature daylilies. The dataset includes
single and multiple targets, unobstructed and clustered
daylily samples, captured under different weather
conditions (sunny and cloudy) and at various time intervals.
After acquiring a sufficient amount of video footage, thevideos were decomposed frame by frame and converted
into static image data. To curate the dataset, images withminimal variations in daylily features within the same
second or in close proximity were excluded, as well as
those that lacked specific images of daylily maturity
flowers. The initial dataset was then refined by retaining
images with distinct characteristics and clearly identifiable
daylily buds. In order to expand the diversity of data, thispaper uses the ACE image enhancement algorithm andhistogram equalization algorithm to enhance the daylily
dataset. In the end, we obtained a dataset of 2060 daylily
images.

Fig. 1 Dataset labels

3. Methodology
This study selected YOLOv8n as the foundationalframework for research. YOLOv8, being at the forefrontof object detection technology, is recognized for its
efficient detection speed and accuracy across various
object detection tasks. The algorithm takes the input image
and resizes it to a fixed size, dividing the resized image
into a grid of cells. Each cell is responsible for predicting
objects that fall within its boundaries. For each cell, thealgorithm predicts B bounding boxes and the probability C
for each category, each containing 5 values: (x, y, w, h, c).
These (x, y) values represent the center of the boundingbox relative to the cell boundary, and (w, h) represent thewidth and height of the bounding box. This C value
indicates the confidence of the algorithm in the prediction.
Non-maximal suppression (NMS) is applied to remove
redundant bounding box predictions. NMS compares theoverlap of the predicted bounding box, leaving only the

2.2. Images annotation and dataset production
To ensure the accuracy of training data, prior toexperimentation, the daylily dataset was meticulously
annotated, defining precise labels for optimal training
outcomes. The Labelme annotation software was
uniformly used for annotating daylily images. Based on
characteristics such as color and shape of daylily flowers,
the images were categorized into three classes: immature
daylily, mature daylily, and overmature daylily. Immature©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
66

Detection and Identification of

one with the highest confidence, and IoU evaluates theaccuracy of the bounding box prediction from 0 to 1, withhigher values indicating better overlap between thepredicted bounding box and the true bounding box. The

final output is the detection result, including the categorylabel, bounding box coordinates, and confidence score.
Leveraging its capabilities, the study applies YOLOv8 tothe detection of daylily maturity,
algorithm directly incorporates TOOD's Task Aligned
Assigner. It selects positive samples based on the weighted
scores of classification and regression. Loss computation
includes classification and regression. Binary Cross
Entropy (BCE) Loss is still employed for classification.
For the regression branch, it is bound to the integral
representation introduced in Distribution Focal Loss, thus
utilizing Distribution Focal Loss and CIoU Loss. The threelosses are weighted using a certain proportion. The final
training results are generated accordingly.

offering an innovative and intelligent solution for
agriculture.
The YOLOv8 model comprises three main components:
the backbone, neck, and head network.CSPDarknet53
serves as the backbone network for YOLOv8, primarily
responsible for extracting high-level features from input
images. It extracts features at multiple levels and outputs
feature maps at different scales. These feature maps
possess varying receptive fields and semantic information,aiding the model in better handling targets of different
sizes and complexities. In contrast to earlier versions of
YOLOv5, YOLOv8 eliminates the less user-friendly
Focus module, and the initial network layers are directly
accomplished by straightforward and conventional
convolutions. CSPDarknet53 is an improvement upon
Darknet53, introducing the CSPNet (Cross Stage Partial
Network) structure. This structure involves cross-stage
partial connections, enhancing information propagation
efficiency. It divides feature maps into two parts, with onepart directly passed to the next stage, and the other partsubjected to further processing. This contributes tostrengthening information propagation and feature fusion
within the network.
The neck still employs the PaFPN (Path Aggregation
Feature Pyramid Network) structure to construct the
YOLO feature pyramid, facilitating comprehensive fusionof multi-scale information. In comparison to YOLOv5, theonly difference lies in the reduction of a 1x1 convolutional
layer in the upsampling operation during the top-down
process. Additionally, the C3 module has been replacedwith the C2f module. Notably, the channel numbers
returned from the three scales are kept equal to the channel
numbers output by the backbone for the three scales.
The Head section adopts a decoupled structure,
separating the classification and detection heads. Twoparallel branches independently extract category andposition features, followed by a 1×1 convolution layer foreach to perform classification and localization tasks.
Simultaneously, the use of anchor boxes is discarded.
The loss calculation process comprises two main
components: positive and negative sample assignment
strategy and loss computation. Considering the superior
performance of dynamic assignment strategies, YOLOv8

4. Experiments
4.1. Experimental environment
To ensure the rigor of the experiments, both the training
and testing processes in this paper were conducted on thesame experimental platform. The experimental
environment was built and configured based on Ubuntu
18.04, with a NVIDIA GeForce GTX 1070 graphics card,
Intel Core i5-9400F CPU, and programming platform
Anaconda 2022.05. The CUDA version used was 11.2, andthe deep learning framework employed was PyTorch 1.10
+CUDA11.1 support. The programming language used
was Python 3.7.
4.2. Model training and evaluation
Through multiple experiments in the early stage, themaximum convergence of daylily target detection modeltraining was obtained. The optimal number of iterations
was 400 times, and 400 iterations were used for modeltraining to ensure the consistency of experimental
parameters. The basic parameters of the model are set toepoch=400 and batch size=16, and the initial learning rate
is set to 0.01 using the SGD optimizer.
In this paper, an objective criterion was used to evaluate
the daylily identification line. we use precision (P), recall(R), and mAP (mean average precision) to evaluate themodel trained in this paper. As shown in Equation (1).

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
67

Fangyan Li

TP
TP + FPTPR=
TP + FN
1 N
mAP =  P(k )R ( k )C k =iof values, k is the threshold, P(k) is the accuracy, and R(k)is the recall.

P=

(1)

5. Result
The results of different species of daylily tested on thismodel are shown in Fig. 2. The P-R curve and R curve of
the model are shown in Fig. 3.

where TP is the true number of positive samples and FP
is the false number of positive samples, FN is the numberof spurious negative samples, C is the number ofcategories, and N is the reference threshold. The number(a) P-R Curve(b) R-Curve

Fig. 2 Detection result of the algorithm
Fig. 3 Performance parameters

identifying the maturity of daylilies. The accuracy
achieved is above 95.6%, with a recall rate of 90.5% and amean Average Precision (mAP) of 0.94. Furthermore, ourmodel demonstrates stability under varying lighting
conditions and significant detection performance acrossdaylilies with different shapes and colors.

6. Conclusion
After completing the model training, we conductedvalidation on daylilies at different growth stages.
Experimental results indicate that the proposed method
exhibits high accuracy and robustness in detecting and©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
68

Powered by TCPDF (www.tcpdf.org)

Detection and Identification of

Our model not only provides a reliable solution for theintelligent cultivation of daylilies but also offers effective
support for agricultural decision-making. The stability of
performance in diverse lighting conditions and its ability
to detect daylilies of varying shapes and colors make it avaluable tool for farmers. This research lays the foundation
for agricultural intelligence, enhancing the productivity
and decision-making capabilities of farmers.
References
1. Zhang X, Ma H, Wei S, et al. Design of Day-Lily Robot
Recognition System Based on ZYNQ. International
Conference on Networking, Communications andInformation Technology (NetCIT). IEEE, 2021: 154-157.
2. SONG ZH ZH，FU L SH，WU J ZH，et al． Kiwifruit
detection in field images using Faster Ｒ -CNN with
VGG16． IFAC-PapersOnLine，2019，52 ( 30 ) : 76-81.
3. LIANG Q K，ZHU W，LONG J Y，et al． A real-time
detection framework for on-tree mango based on SSD
network.
International Conference on Intelligent Ｒ
obotics and Applications． Springer， Cham， 2018: 423436.
4. Zhenbo Li, Fei Li, Ling Zhu, et al. Vegetable Recognition
and Classification Based on Improved VGG Deep Learning
Network Model. International Journal of Computational
Intelligence Systems, 2020, 13(1) : 559-564.
5. XUE Y J，HUANG N，TU SH Q，et al． Immature mango
detection based on improved YOLOv2. Transactions of theChinese Society of Agricultural Engineering，2018， 34( 7) :173-179.
6. ZHAO D AN，WU Ｒ D，LIU X Y，et al． Apple
positioning based on YOLO deep convolutional neuralnetwork for picking robot in complex background.
Transactions of the Chinese Society of AgriculturalEngineering，2019，35( 3) : 164-173.

Authors Introduction
Ms. Fangyan Li
She is the scond-year graduate student of
Tianjin University of Science andTechnology. Her research direction ismachine Learning.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
69

Chaos Synchronization and Circuit Design of Chen System and Lü System with Different
Structures
Haozhe Sun

College of Electronic Information and Automation, Tianjin University of Science and Technology,
300222, China
E-mail: 2921407938@qq.com
www.tust.edu.cn

Abstract
In this paper, by using nonlinear feedback control, chaos synchronization is achieved between the Chen system andthe Lü system with different initial values, and the error curves and state synchronization curves of thecorresponding states in the response Lü system and the drive Chen system are plotted. Finally, the simulation circuit
model of the synchronization system of the drive Chen system and the response Lü system is designed by Multisim
circuit simulation software. Comparing the output curves with the curves obtained by MATLAB simulation
software, it can be found that the two curves achieve a good qualitative agreement. It prove that the circuit model ofthe synchronization system is correct and the synchronization of the drive Chen system and the response Lü systemis accomplished.
Keywords: Chaos; Chaos synchronization of different systems; Nonlinear feedback control; Chaotic circuit

system and the Lü system to be successfully applied in
the field of confidential communication.

1. Introduction
In 1999, Professor Guanrong Chen of the University of
Houston discovered a new chaotic attractor, the Chensystem, using engineering feedback control [1]. which issimilar to the Lorenz system but not topologicallyequivalent, with a more complex structure and morediverse dynamic behavior.In 2001, Jinhu Lü andGuanrong Chen discovered a new chaotic system, the Lüsystem, that system links the Lorenz system and the Chensystem and represents a continuous evolution between thetwo systems [2].
In this paper, the above two systems are taken as theresearch objects to analyze their chaotic properties, andbased on this, the two systems are synchronized to chaos
by using the method of nonlinear feedback [3], and thennumerical simulation is carried out by using the
MATLAB software, and the simulated circuits aredesigned, and the circuit simulation is carried out in
Multisim, which is designed to provide sufficient
theoretical support for the future application of theheterostructured chaos synchronization of the Chen

2. Chaos Synchronization and Circuit Design ofChen System and Lü System with Different
Structures
2.1. Modeling of the Drive Chen system
𝑥̇1 = 𝑎1 (𝑦1 − 𝑥1 ){𝑦̇1 = (𝑐1 − 𝑎1 )𝑥1 − 𝑥1 𝑧1 + 𝑐1 𝑦1
𝑧̇1 = 𝑥1 𝑦1 − 𝑏1 𝑧1(1)
Where, x1 ，y1 ，z1 ∈ R is the state variable of thesystem and a1 = 35，b1 = 3，c1 = 28 is the typicalparameter of Lü system.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
70

Haozhe Sun

The error system is then written in the form of 𝑒̇ =𝑓(𝑒, 𝑡). If the equilibrium state of error system at theorigin is uniformly asymptotic and stable in a large range,
it indicates that the chaos synchronization of different
structure has been realized between Chen system and Lü
system.
The value of the synchronization controller parameter𝑘 = [𝑘1 𝑘2 𝑘3 ]𝑇 is determined by the Lyapunov
second method.
According to Lyapunov's second method, firstly takethe positive definite Lyapunov function 𝑉(𝑒̃ ) =1/2[(𝑎1 − 𝑐1 )𝑒12 /𝑎2 + 𝑒22 + 𝑒32 ] , and then take its
derivative:
𝑎1 − 𝑐1
V(𝑒̃̇ ) =𝑒1 𝑒1̇ + 𝑒2 𝑒2̇ + 𝑒3 𝑒3̇
𝑎2= (𝑎1 − 𝑐1 )𝑒1 𝑒2 − (𝑎1 − 𝑐1 )𝑒12

2.2. Modeling of Response Lü Systems

𝑥̇ 2 = 𝑎2 (𝑦2 − 𝑥2 ) + 𝑢𝑐1{𝑦2̇ = −𝑥2 𝑧2 + 𝑐2 𝑦2 + 𝑢𝑐2
𝑧̇2 = 𝑥2 𝑦2 − 𝑏2 𝑧2 + 𝑢𝑐3(2)
Where, 𝑥2 ，𝑦2 ，𝑧2 ∈ 𝑅 is the state variable of thesystem and 𝑎2 = 36，𝑏2 = 3，𝑐2 = 20 is the typicalparameter of Lü system. 𝑢𝑐 = [𝑢𝑐1 𝑢𝑐2 𝑢𝑐3 ]𝑇 is therequired synchronization controller, through which thesynchronization of two chaotic attractors with very
different structures can be realized.
2.3. Design of Synchronization Controller

Subtracting the corresponding terms in the mathematicalmodel Eq. (1) of the driving Chen system from the terms
in the mathematical model Eq. (2) of the response Lü
system, the error system model is shown as Eq. (3)below:

𝑘1 (𝑎1 − 𝑐1 ) 2
𝑒1 + 𝑐2 𝑒22
𝑎2+(𝑐1 − 𝑎1 )𝑒1 𝑒2 − 𝑘2 𝑒22−𝑏2 𝑒32 − 𝑘3 𝑒32
𝑘1 (𝑎1 − 𝑐1 )= −[+ (𝑎1 − 𝑐1 )] 𝑒12𝑎2−(𝑘2 − 𝑐2 )𝑒22 − (𝑘3 + 𝑏2 )𝑒32−

𝑒̇1 = 𝑥̇ 2 − 𝑥̇1= 𝑎2 (𝑦2 − 𝑥2 ) − a1 (𝑦1 − 𝑥1 ) + 𝑢c1
𝑒̇2 = 𝑦̇ 2 − 𝑦̇1= −𝑥2 𝑧2 + c2 𝑦2−[(𝑐1 − a1 )𝑥1 − 𝑥1 𝑧1 + 𝑐1 𝑦1 ] + 𝑢c2
𝑒̇3 = 𝑧̇2 − 𝑧̇1= 𝑥2 𝑦2 − b2 𝑧2{ −(𝑥1 𝑦1 − b1 𝑧1 ) + 𝑢c3

According to Lyapunov's second method, if 𝑉(𝑒̃ ) is
positively definite and 𝑉̇ (𝑒̃ ) is negatively definite, thenthe error system is uniformly asymptotically stable in alarge range at the origin. At this time, the variable
coefficients of the error system should satisfy:(3)

𝑘 (𝑎1 −𝑐1 )Where𝑒 = [𝑒1 𝑒2 𝑒3 ]𝑇 =
𝑇
𝑥
−
𝑥
𝑦
−
𝑦
𝑧
−
𝑧
[ 2
1
2
1
2
1 ] , 𝑒1 ，𝑒2 ，𝑒3 is the statevariable of the error system. The synchronization
controller is designed as:

{

(6)

−[ 1
{

𝑎2

+ (𝑎1 − 𝑐1 )] < 0−(𝑘2 − 𝑐2 ) < 0−(𝑘3 + 𝑏2 ) < 0(7)

𝑢𝑐1 = (𝑎1 − 𝑎2 )(𝑦1 − 𝑥1 ) − 𝑘1 𝑒1
𝑢𝑐2 = 𝑥2 𝑧2 − 𝑥1 𝑧1 + (𝑐1 − 𝑐2 )𝑦1+(𝑐1 − 𝑎1 )𝑥2 − 𝑘2 𝑒2
𝑢𝑐3 = −𝑥2 𝑦2 + 𝑥1 𝑦1 + (𝑏2 − 𝑏1 )𝑧1−𝑘3 𝑒3

Solved:𝑘1 > −𝑎2 = −36{ 𝑘2 > 𝑐2 = 20
𝑘3 > −𝑏2 = −3
(4)

(8)

Where 𝑘1 ，𝑘2 ，𝑘3 ≥ 0, Eq. (4) is substituted into Eq.
(3), and the mathematical model of the error system isobtained as shown in Eq. (5).

If (𝑘1 , 𝑘2 , 𝑘3 ) = (0,25,0), the synchronous controller
𝑢𝑐 is as follows:{𝑒̇1 = 𝑥̇ 2 − 𝑥̇1= 𝑎2 (𝑒2 − 𝑒1 ) − 𝑘1 𝑒1
𝑒̇2 = 𝑦̇ 2 − 𝑦̇1= 𝑐2 𝑒2 + (𝑐1 − 𝑎1 )𝑒1 − 𝑘2 𝑒2
𝑒̇3 = 𝑧̇2 − 𝑧̇1 = −𝑏2 𝑒3 − 𝑘3 𝑒3{

𝑢cl = −(𝑦1 − 𝑥1 )𝑢𝑐2 = 𝑥2 𝑧2 − 𝑥1 𝑧1 + 33𝑦1
−7𝑥2 − 25𝑦2
𝑢𝑐3 = −𝑥2 𝑦2 + 𝑥1 𝑦1(9)

So far the synchronization controller has been
designed.

(5)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
71

Chaos Synchronization and Circuit

The numerical simulation diagram of the chaotic
synchronization controller of Chen system and Lü systemwith different structures built in Simulink is shown in Fig.

Through careful observation of the images, it can beseen that the controller makes the corresponding state
error curve gradually converge to zero with the passage oftime, and the corresponding state synchronization curve
follows the running trajectory of Chen driving system.
This is enough to indicate that all the above derivations of
all equations on the different structure chaos
synchronization of Chen system and Lü system arecorrect, and Chen system and Lü system can achievedifferent structure chaos synchronization.

1.

After setting up the simulation environment, click therun button, and the corresponding state error curve isshown in Fig. 2.
The corresponding state synchronization curve isshown in Fig. 3.

Fig. 1 Numerical simulation of chaos synchronization of Chen system and Lü system with different structure

Fig. 2 State error curve: (a) e1; (b) e2; (c) e3©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
72

Haozhe Sun

Fig. 3 State synchronization curve: （a）x1-x2；（b）y1-y2；（c）z1-z2
Substitute 𝜏0 = 100 into Eq. (10) and take 𝐶4 =2.4. Design and construction of an analog circuit
𝐶5 = 𝐶6 = 10nF, and the calculation is as follows:model for heterostructure chaotic synchronous

R14 = R15 =control

= 1MΩ
1
1
R18 =
=
33𝜏0 𝐶5 33 × 100 × 10 × 10−9
= 30.3kΩ
1
1
R19 =
=
7𝜏0 𝐶5 7 × 100 × 10 × 10−9= 142.8kΩ
1
1
R 20 ==
25𝜏0 𝐶5 25 × 100 × 10 × 10−9
= 40kΩ
1
R16 = 𝑅17 = 𝑅21 = 𝑅22 =100𝜏0 𝐶51=
= 10kΩ
100 × 100 × 10 × 10−9

The circuit model of the synchronous controller isdesigned and constructed, and the circuit of thesynchronous controller is designed by the method of
improved modular circuit design.
The state expression of the control circuit is shown in
Eq. (10).
𝜏0 𝑢c1 = 𝜏0 [−(𝑦1 − 𝑥1 )]= −𝜏0 𝑦1 − 𝜏0 (−𝑥1 )1
1(−𝑥1 )=−𝑦 −R14 C4 1 R15 C4
𝜏0 𝑢𝑐2 = 𝜏0 [10𝑥2 𝑧2 − 10𝑥1 𝑧1+33𝑦1 − 7𝑥2 − 25𝑦2 ]= −10𝜏0 (−𝑥2 𝑧2 ) − 10𝜏0 𝑥1 𝑧1−33𝜏0 (−𝑦1 ) − 7𝜏0 𝑥2 − 25𝜏0 𝑦2
1(−𝑥2 𝑧2 )=−10R16 𝐶5
1
1(−𝑦1 )−𝑥 𝑧 −10R17 𝐶5 1 1 R13 𝐶5
1
1
−
𝑥 −
𝑦
R19 𝐶5 2 R 20 𝐶5 2
𝜏0 𝑢𝑐3 = 𝜏0 (−10𝑥2 𝑦2 + 10𝑥1 𝑦1 )= −10𝜏0 𝑥2 𝑦2 − 10𝜏0 (−𝑥1 𝑦1 )1=−𝑥 𝑦
10R 21 C6 2 21(−𝑥1 𝑦1 )−10R 22 C6

11=
𝜏0 𝐶4 100 × 10 × 10−9(11)The analog circuit for the synchronous control of Chensystem and Lü system is shown in Fig. 4.
Build the analog circuit shown in the above figure in
Multisim software. Wait for the oscilloscope to display
the waveform, observe the corresponding waveform, andobtain the corresponding synchronization curve shown in
Fig. 5 and the corresponding error curve shown in Fig. 6.

(10)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 202473

Chaos Synchronization and Circuit

Fig. 4 Analog Circuit for Synchronous Control of Chen System and Lü System

Fig. 6 Error curve:（a）e1；（b）e2；（c）e3
3. Conclusion

Fig. 5 Synchronization state curve:（a）x1-x2；（b）y1-y2；（c）z1-z2

In this paper, the driving Chen system and the responding
Lü system are first mathematically modeled separately,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
74

Powered by TCPDF (www.tcpdf.org)

Haozhe Sun

and then the synchronous controllers are designed toachieve the purpose of removing all nonlinear terms. The
parameters of the controller are solved according to
Lyapunov's second method. The controller allows thenumerical simulation of the system in MATLAB to showthat the corresponding state error curves graduallyconverge to zero over time and the corresponding statesynchronization curves follow the trajectory of the Chen
drive system.
The simulation circuit model of the Chen chaotic
system and the Lü chaotic system with heterostructure
chaotic synchronization is designed and constructed withthe help of Multisim software using a modified modular
approach. The installation of separate switches for the
structural compensator and the controller to ensure
individual control is a critical step in the design of thesystem, and the observation of the oscilloscope showing
the corresponding error curves reveals that the curves
quickly converge to 0 after the switches are closed, andthe whole experiment shows that the designed
synchronization controllers perform the control role. The
heterostructured chaotic synchronization control of Chensystem with L ü system and its circuit design are
implemented.
References
1.

2.

3.

Guopeng Zhou, Xiaoxin Liao, Bingji Xu, Pei Yu,
Guanrong Chen. Simple algebraic necessary and sufficientconditions for Lyapunov stability of a Chen system andtheir applications. Transactions of the Institute of
Measurement and Control, 2018, 40(7).
Jiakun Zhao, Di Zhou, Yexin Li. A new impulsive
synchronization of Chen hyper-chaotic system and L ühyper-chaotic system. Journal of Vibration and Control,
2013, 19(12).
W. Kinzel, A. Englert, I. Kanter. On chaos synchronizationand secure Communication. Philosophical Transactions Of
The Royal Society A, 2010, 368, 379–389.

Authors Introduction
Mr. Haozhe Sun
In 2023, he received his Bachelor ofEngineering degree from the School ofElectronic Information and Automation,
Tianjin University of Science andTechnology, China. He is pursuing amaster's degree in engineering from
Tianjin University of Science andTechnology.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
75

A Parking Space Recognition Method Based on Digital Image Technology
Hao He*, Fangyv Liu, Jiaxin Wang

Tianjin University of Science and Technology, Tianjin,300222, China
E-Mail: *hehao0310252022@163.com
Abstract
In recent years, the number of cars in the city has been increasing, leading to an increasingly prominent issue of
urban parking space. Consequently, the automatic identification method for parking spaces has emerged as a
crucial research direction. This paper presents a design and implementation scheme for recognizing the status ofparking spaces in urban areas based on digital image processing and other technologies. In this paper, the real-time
pictures of multiple parking Spaces are collected to identify the parking space, so as to determine the use of theparking space. The experimental results demonstrate the feasibility and effectiveness of the proposed method,which holds practical significance in addressing parking space detection problems through digital image
processing.
Keywords: OpenCV, parking spot detection, convolutional neural network, intelligent transportation

1. Introduction
2. Introduction to development environment
In recent years, with the popularity of family cars, urban
road traffic and urban parking space management
problems become increasingly prominent.
In the new intelligent parking management system, themethod of identification of parking space status is also
improved with the progress of technology. At first, allkinds of sensors are used to recognize and judge thestatus of parking Spaces, but these methods havedisadvantages: ultrasonic recognition method makes theerror of the sensor larger because there are multiple
echoes. The method of infrared identification of parking
space status is affected by the change of ambient lightsource and heat source, resulting in high misjudgmentrate. Geomagnetic detection is extremely inconvenient toinstall.
In response to the above problems, experts in related
fields began to look for new solutions. A new research
hotspot has been formed. According to the researchfindings, it has certain advantages to use digital image
technology for real-time monitoring of parking lots and toanalyze and judge the use of parking Spaces within thescope of video surveillance through image algorithms.
Therefore, this paper studies the parking space
recognition method based on digital image technology.
This paper is divided into four parts. The first part is the
background. The second part is an introduction to thedevelopment environment, briefly introducing Python,
OpenCV and CNN. The third part introduces the systemprocess, including pre-processing, parking space division,
CNN model construction and parking space detection.
The fourth part summarizes the main content of thispaper.

2.1. Introduction to Python
Python was designed in the early 1990s by Guido van
Rosum of the National Center for Mathematical andComputer Science Research in the Netherlands as areplacement for a language called ABC. Python provides
efficient high-level data structures, as well as easy andeffective object-oriented programming. Python's syntax
and dynamic typing, as well as the nature of theinterpreted language, make it a programming language
for scripting and rapid application development on mostplatforms, as versions are updated and new features of thelanguage are added, it is gradually used for independent,
large-scale project development [1]. Python is one of themost popular programming languages for beginners, and
the Python interpreter is easy to extend with new
functions and data types in C, C++, or other languages
that can be called through C. Python can also be used as
an extension language in customizable software. Python's
rich standard library provides source code or machinecode for every major system platform.
2.2. Introduction to OpenCV
OpenCV is an open-source library for machine visionand machine learning, based on Apache2.0. It boasts
excellent compatibility and can be deployed acrossmultiple operating systems including Linux, Windows,
Android, and Mac OS. Comprising both C and C++
languages as a whole, it also provides interfaces for
Python, Ruby, MATLAB, and other programming
languages to facilitate development work in various
language preferences. OpenCV incorporates a vast arrayof general algorithms in the fields of image processing©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
76

Hao He, Fangyv Liu, Jiaxin Wangand computer vision. Its functionalities encompass object
recognition, face recognition, action recognition, motion
tracking among others. As such, it stands as a pivotal
software library within the realm of image processing andmachine vision [2], [3].

3.1. pretreatment
Due to the influence of various natural factors, thedifference between the target object in the picture and thebackscene image is not high, the available information is
weak, and the parking space can not be recognizeddirectly. Therefore, a variety of processing of the original
image is required. For example, removing noise andenhancing image detail to distinguish the target object
from the background image.
In order to facilitate post-processing, backgroundfiltering should first be carried out to filter out irrelevant
background information in the image and facilitate
subsequent recognition. The method is to define the mask,
set the maximum and minimum thresholds, and convert
information outside the threshold interval into zero value
information.
The second step is binary processing, which converts
the image into grayscale. This is because the grayscale
image only retains the brightness information and omits a
lot of irrelevant information, which effectively reduces
the amount of calculation in the later stage.
The third step is Canny edge detection, which is divided
into four steps: Gaussian filtering, calculating gradientvalue and gradient direction, filtering non-maximum
value, double threshold detection method. It caneffectively determine real and potential edges, and canbetter detect most of the required boundaries [4].
The original image is shown in Fig. 1. and thepre-processed image is shown in Fig. 2.

2.3. Introduction to CNN
The human visual system is processed hierarchically inthe visual cortex of the brain, and the working process ofthe brain's processing of visual information is acontinuous iterative and abstract process. Starting withthe original image of pupil intake, some cells in the
cerebral cortex find the edge and direction of the object.
Then, the brain determines the general shape of the object
in front of the eyes and abstracts. Then the big brain
further determines the class of the object. Finally, throughthe analysis of the information, the conclusion is drawn.
CNN network, full name Convolutional neural network,
is a deep learning model inspired by the human visual
nervous system. It can be divided into convolution layer,
pooling layer and fully connected layer. Convolution
layer is used to extract local features in image. Poolinglayer is used to greatly reduce the magnitude ofparameters. The fully connected layer is used to output
the result. Convolutional neural network (CNN) is aclassic algorithm of deep learning, which is widely usedin computer vision, natural language processing and otherfields, and can better process data information with stableeffects such as pixels and audio and video without
additional feature engineering requirements.
3. Methodology
The system design process can be roughly divided
into pre-processing, parking space division, CNN
model construction and training, parking space
recognition and other stages. In the pre-processing
stage, the information will be processed and only theeffective information will be retained. Parking Spaces
are extracted and stored as samples in the stage ofparking space division. In the model training stage,
the two-classification model will be trained torecognize whether there is a car in the parking space.
Finally, the images are monitored in real time and thegaps are marked.

Fig. 1 Original image©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
77

A Parking Space Recognitionneural network jointly developed by Oxford University
and Google DeepMind in 2014, which is composed of 13
convolutional layers +3 fully connected layers. The
convolutional layer is the core of the convolutional neuralnetwork. The essence of the convolutional process isbased on a small moment matrix, that is, theconvolutional kernel. The pixel matrix of each layer ofthe image is continuously scanned by step size, and thescanned value is multiplied by the number of the
convolution check position, and then added and summed.
Convolution kernel is equivalent to a filter, used to extract
image features, convolution kernel generally has 3*3 and5*5 sizes, usually used is 3*3 size, training effect is better.
The fully connected layer refers to the connection
between each node of the N-1 layer and the n layer, that is,
the activation function input of the node of the n layer isthe weight of all the nodes of the N-1 layer, which is used
to synthesize the features extracted from the previous side.
It plays the role of "classifier" in the whole convolutional
neural network.
Due to the small size of the data set, in order to get abetter model, the third step is to carry out dataenhancement operations. Data enhancement can bedivided into online enhancement and offline enhancement
according to the use method. Offline enhancement issuitable for small data sets and can effectively increase
their size. The multiplier of enhancement depends on thenumber of images converted, such as rotating all images
to double the data set. There are two types of commonlyused data enhancement, supervised and unsupervised.
The commonly used one is supervised data enhancement,
which is based on the existing data set and expands theexisting data by analyzing the completeness of the data
and adopting certain rules. Supervised data enhancement
can be subdivided into single sample data enhancement
and multi-sample data enhancement. In practice, single
sample data enhancement is more used.
Finally, the data of the training set is imported into themodel for training. The accuracy of the test set obtained
after the completion of all programs is 91.4%, and theaccuracy rate is consistent with the prediction period.
Later, VCC16 can be changed to ResNet50 to further
improve the accuracy, but the training speed of ResNet50
is slower than that of VCC16, so from the perspective ofefficiency, VCC16 is finally chosen.
The results of parking space detection are shown in Fig.
3.

Fig. 2 Pretreatment result
3.2. Parking space division
Through observation, it can be found that the parking
lot is composed of different straight lines, which can befurther divided by detecting these straight lines. The
method is to use Hough transform, a feature detection
algorithm widely used in computer vision, to find out
some features in graphics. Hough transform obtains moreaccurate results by giving the type of shapes to be
identified, and then determining the shape of the object
through voting mechanism in the parameter space, andfiltering out objects with too large or too small intervals.
Some cracks or other line segments in the image may be
mistaken for parking space lines, and the parking spacelines are basically vertical or horizontal. The irregular line
segments can be further eliminated by calculating theslope of the line segments to improve the accuracy.
3.3. CNN network training
After the parking space image is obtained, it is divided
into training set and test set according to the proportion of
70% and 30%, and then divided into occupied set andnon-occupied set according to whether or not the vehicle
occupies the parking space, which is used for subsequentmodel training.
This model training uses VGG16 network, VGGNet is a

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
78

Powered by TCPDF (www.tcpdf.org)

Hao He, Fangyv Liu, Jiaxin Wang

Authors Introduction
Ms. Hao He
She is currently an undergraduate
in Tianjin University of Scienceand Technology. She is majoringinrobotics
engineering.Her
research area is about Image
processing based on OpenCV.

Ms. Fangyv Liu
She is currently an undergraduate
in Tianjin University of Scienceand Technology. Her research area
is about intelligent robot andintelligent control.

Fig. 3 Parking test results
4. Conclusion
The program will process and detect the picture in realtime, and the vacant parking Spaces in the picture aremarked green, and the occupied parking Spaces are notmarked. Some areas in some pictures are incorrectly
marked, such as the guiding line area of the parking lot ismarked as empty parking space, which can be further
improved by modifying the CNN model. The detection
and statistics of the remaining areas are basically in linewith expectations, and the processing and detection of
parking space information is realized.
Through OpenCV and convolutional neural network,
real-time monitoring of parking space occupancy isrealized, which can be applied to urban parking lots forconvenient management. In addition, the system can befurther optimized to increase speed and adapt to more
complex scenarios, thus providing more efficient andtimely data for parking lot management.

Ms. Jiaxin Wang
She is currently an undergraduate
in Tianjin University of Scienceand Technology. She is majoringinrobotics
engineering.Her
research area is about Image
processing based on OpenCV.

References
1.
2.

3.

4.

Haitao Huang, Python 3 Ice-Breaking Artificial
Intelligence from Entry to Actual Combat (in Chinese).
Yaling Zhou, Traffic flow detection system based on
OpenCV. Beijing: Beijing University of Posts andTelecommunications, 2015.
Minglu Zhang, He Xiaodong, Research on Traffic flow
statistics Algorithm based on OpenCV. Computer
Knowledge and Technology, 2016,12 (19) : 169-170.
Mingrui Zhao, Research on parking status detection
method based on machine vision. Shenyang: Shenyang
Jianzhu University, 2021.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
79

A Design of Intelligent Handling Robot Based on AT89C52
Fangyv Liu*, Jiaxin Wang, Hao He

College of Electronic Information and Automation, Tianjin University of Science and Technology, 300222,China
Email:*himawari0216@163.com

Abstract
At present, manual logistics handling is inefficient and limited, manual operation is not only difficult to manage andmaintain, but also has certain security risks in the handling process. This paper presents a logistics handling robot
based on AT89C52 single chip microcomputer. The integration of the power module, sensor, and drive motor module
enables automatic obstacle avoidance and information collection. The incorporation of an ultrasonic obstacle
avoidance module and an infrared tracking module enhances the capability for obstacle avoidance and path searching,and can easily cope with different workplace. Handling robot arm structure reference Yaskawa MPL manipulator,theYaskawa MPL manipulator is highly suitable for high-speed and high-precision palletizing, picking, packaging, andother industries.
Keywords: logistics handling; robot; MCU; modularization; obstacle avoidance

1. Introduction
With the rapid development and application oftechnology in the 21st century, the logistics transportationand handling industry has also ushered in great changes
and development. Manual handling in the logistics
production line has many years of history, however, thetraditional manual handling has many shortcomings, suchas large physical consumption, low degree of automation,
low efficiency, environmental impact and so on.
Nowadays, many domestic logistics enterprises still use a
lot of physical handling and manual operation in the daily
logistics handling operation. This situation in the current
logistics industry can no longer adapt to the development
needs of the market.
Although manual logistics handling still plays animportant role in reality, it also faces some status quo anddrawbacks, including human resource dependence, theinfluence of human factors, low efficiency and risinglabor costs. Therefore, the use of handling robots canimprove the efficiency of logistics handling, reduce costs,
improve safety, and provide greater flexibility andadaptability. With the continuous development oftechnology, handling robots will play an increasingly
important role in the logistics industry.
However, the current handling robot still has someshortcomings and still faces certain difficulties in thecomplex and dynamic environment. For example, objectsof different shapes and sizes, different types of ground,and the presence of people and other robots can have an
impact on the robot's operation and navigation. The
accuracy and flexibility of handling robots still need to be
improved. This paper proposes the design of logistics
handling robot based on AT89C52 single chip
microcomputer, and optimizes the current handling robot
on the system to realize the autonomous tracking of thehandling robot, avoid obstacles and complete theautomatic picking of objects by the robot arm, so as toachieve the purpose of more efficient and safe logistics
handling.
The rest of this article is organized as follows.The
second part of the design of the robot, first introduces thesystem design of the robot, and then introduces the designof the drive module and the robot arm according to thesystem design. The third part is hardware design, which
introduces the main hardware used by the robot. The
fourth part of the module test, mainly ultrasonic obstacle
avoidance module and infrared tracking module fortesting, data analysis. The fifth part summarizes the maincontent of this paper.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
80

Fangyv Liu, Jiaxin Wang, Hao He

2. Design of Handling Robot
2.3. Robot arm design
2.1. Robot system design
Mechanical arm structure reference Yaskawa MPL seriesmechanical arm, Yaskawa MPL series mechanical arm
structure is simple, fewer parts, parts of low failure rate,
reliable performance, simple maintenance andmaintenance, less inventory parts required. The MPL
structure has high control performance and precision, canmeet the complex motion control needs, and can be
flexibly configured and expanded to adapt to different
application scenarios and requirements.
The Yaskawa MPL manipulator has a modular design,
and individual joints and components can be installed andreplaced independently. This design makes repair andmaintenance easier and faster, reducing downtime andrepair costs. The Yaskawa MPL0100 robot arm modeland fixture model is shown in Fig. 3.

The design of this paper is based on the modular designof AT89C52 microcontroller. The functions of the whole
design system are divided into power module, motor
drive module, ultrasonic obstacle avoidance module,
infrared tracking module and robot arm module. The
control of the robot arm is achieved through a vision
sensor, through which information about the desired
object is obtained [1]. The System composition diagram
is shown in Fig. 1.

Fig.1 The system composition diagram
2.2. Drive module design
The whole work flow of the handling robot starts fromthe designated starting work area and gets the routeinformation through the road condition feedback, so as toreach the designated area in accordance with the set route
to complete the corresponding work. Therefore, thedesign of the driving module of the robot in this paper
needs an automatic tracking system to realize. The
automatic tracking system flow is shown in Fig. 2.

Fig.3 Yaskawa MPL0100 robot arm model and fixture
model
3. Hardware Design
3.1. Main control chip
The main controller of the control system uses AT89C52
microcontroller, AT89C52 microcontroller has powerful
functions and flexibility, suitable for a variety ofembedded systems and control applications, belongs to
CMOS8-bit microcontroller. Its advantages are mainlyreflected in low power consumption, high performance,
and flash memory [3], [4]. Receive the input signal
collected by the sensor, and output the control signal tothe driver chip, so as to control the robot to walk along
the specified route and stop to the specified destination.

Fig. 2 The automatic tracking system flow
The infrared tracking module uses the TCRT5000
infrared sensor as the path detection sensor to realize thedetection of the road track route [2].

3.2. Power supply
In the whole system design, AT89C52 MCU is the control©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
81

A Design of Intelligent

center, and it is powered by 2 3.7V power modules.
Information acquisition through the sensor, its operating
voltage requirement is 5V, the use of AMS1117 regulator
to achieve voltage regulation, AMS1117 is acost-effective regulator, relatively low price, stable andreliable performance, At the same time, the AMS1117
regulator has the advantages of simple and easy to use,
stable and reliable, low power consumption, overload
protection, multiple output voltages and high cost
efficiency. The operating voltage of the motor drive
module in the device is 5 V, if the voltage is greater than
5 V, it is directly supplied by the power module.

Table 1. Ultrasonic obstacle avoidance module test data

Ultrasonic ranging(mm)Ruler ranging(mm)Error(%)

266.8
350.6
439.8
658.8

266.0
350.0
439.0
658.0

0.15
0.17
0.18
0.12

772.1

771.0

0.14

The error between the ultrasonic obstacle avoidance
module's ranging value and the actual ruler's ranging
value is less than 2%, which can realize automatic
obstacle avoidance.

3.3. Drive motor
The robot is usually driven by DC reducer motor, which
can output large torque through the deceleration of thegear box [5]. After comparison and selection, the model
JGB37-550 DC gear motor is selected. The JGB37-550
DC gear motor has the advantages of high torque, goodstability, wide speed regulation range, high energy
efficiency, high reliability and easy installation, whichmakes it widely used in many industrial and commercial
applications.

4.2. Infrared tracking module test
Natural light has a certain influence on the receiving
ability of the infrared receiving tube, so the indoor
environment test matching the actual use environment isused. Before the test, connect P3.2~3.5 of the pin J11 andP3.2~3.5 of the AT89C52. Place the robot on the white
test bench, turn on the power, and fine-tune thepotentiometer RW3 and RW4 in a clockwise manner until
LED2 and LED light up. Infrared tracking module test
data is shown in Table 2.

3.4. Controls the robotic arm chip
In order to identify the grasping material workpiece, thecolor and shape of the material workpiece are used to
identify the target. The recognition sensor adopts
OpenMV3 vision module with MCU interface. There is acamera on the OpenMV3 module for taking images ofobjects, and the images taken are processed by AT89C52
microcontroller.

Table 2. Infrared tracking module test data
Transfer robot

Left infrared tube

Right infrared tube

Go forward

Turn left

1
0
1

1
0
0

Turn right0

1

Back up

4. Module Testing
4.1. Ultrasonic obstacle avoidance module test

5. Conclusion

According to the actual working environment of thehandling robot, the indoor environment is selected for
ultrasonic obstacle avoidance test, obstacles are set, andECH0 of the HC-SR04 ultrasonic module is connected to
the needle J9-P2.0, TRIG is connected to J9-2.1, VCC isconnected to J9-VCC, and GND is connected to J9-GND.
The LCD1602 liquid crystal was installed in the
LCD1602 socket of the single chip microcomputer, andthe contrast of the liquid crystal was adjusted. Ultrasonicobstacle avoidance module test data is shown in Table 1.

This paper takes the handling robot as the research object,uses the AT89C52 single chip microcomputer as the main
control chip, uses the infrared tracking module, ultrasonicobstacle avoidance module, power module, motor moduleand robot arm module to design the handling robot,
optimizes the shortcomings of some handling robots in
the general working environment, improves the efficiency
of logistics handling and reduces the risk of manual
logistics handling.
Intelligent handling robot has broad development
prospects. With the continuous progress of technology
and the expansion of application fields, intelligent
handling robots will play an important role in various
industries, improve work efficiency, reduce costs, and©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
82

Powered by TCPDF (www.tcpdf.org)

Fangyv Liu, Jiaxin Wang, Hao He

promote the intelligent transformation of the industry.
References
1.

2.

3.

4.

5.

Luo Xiangxi, Yuan Fengwei, Liu Zhiwei.Design andresearch of manipulator based on vision orientation.
Mechanical Engineer, 2016(12): pp.54-56.
Zhang Meng, Shi Baohua.PC motherboard automatic
assembly based on machine vision system.China
Instrumentation, 2016(9): pp.32-35.
LI Y K，WEI S L.Research on automatic control ofcooperative operation trajectory of unmanned
warehousemultihandling
robot.Computer
Measurement and Control:1-8[2023-01-05DOI]:
10.16526/j.cnki.11-4762/tp.2023.02.018.
WU Y Q,YU T. Design of dual robot coordinatedcontrolsystem. Manufacturing Automation, 2022,
44(10):pp.1-5.
Liu Zhenyu, Li Zhongsheng, Zhao Xue,etal.,Research on sorting technology of industrial robot
based on machine vision.Manufacturing Automation,

2013, 32(9):pp.25-30.
Authors Introduction
Ms. Fangyv Liu
She is currently an undergraduate in
Tianjin University of Science andTechnology. Her research area is about
intelligent robot and intelligent control.

MS. Jiaxin Wang
She is currently an undergraduate in
Tianjin University of Science andTechnology. She is majoring in robotics
engineering.Her research area is about
Image processing based on OpenCV.

MS. Hao He
She is currently an undergraduate in
Tianjin University of Science andTechnology. She is majoring in robotics
engineering.Her research area is about
Image processing based on OpenCV.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
83

Application and Differences of Robotic Arms, Traditional Machines, and Manual Work in
Production
Xue Yang*, Ying Su, Yuping Mei, Haiquan Wang

College of Electronic Information and Automation, Tianjin University of Science and Technology,
300222，China
E-mail: *wybzdwzdbzd@126.com

Abstract
This article aims to explore the differences between robotic arms and traditional machines and humans. Firstly,
robotic arms are a kind of automated equipment with high flexibility and accuracy, which can perform a variety ofcomplex tasks. In contrast, traditional machines lack flexibility and accuracy, while humans have problems such as
low work efficiency and high error rates. Secondly, the emergence of robotic arms can solve many problems in
traditional machines and humans, and improve production efficiency and quality. Finally, there is no contradiction
between robotic arms, traditional machines and humans, but they can complement each other. By using robotic arms
and humans reasonably, we can give full play to their respective advantages and improve the overall production
efficiency and economic benefits.
Keywords: robotic arm, traditional machine, human, production efficiency, economic benefit

1. Introduction

robotic arms also need human supervision andmanagement to ensure their normal operation andproduction safety. In the future, with the continuous
advancement and development of technology, roboticarms will be applied and developed in more fields.
The picture of the robotic arm board is shown in Fig. 1.

Robotic arms are a kind of automated equipment with highflexibility and accuracy, which can perform a variety ofcomplex tasks. In contrast, traditional machines lackflexibility and accuracy, while humans have problems
such as low work efficiency and high error rates.
The emergence of robotic arms can solve many
problems in traditional machines and humans, andimprove production efficiency and quality.
There is no contradiction between robotic arms,
traditional machines and humans, but they cancomplement each other. By using robotic arms andhumans reasonably, we can give full play to theirrespective advantages and improve the overall production
efficiency and economic benefits.
The rest of this article is organized as follows. The
second section introduces the appearance differences
between robotic arms and traditional machines. In the thirdpart, the application of robotic arms in manufacturing cansolve many problems. In the fourth part, robotic arms andhuman labor can complement each other. At the same time,

Fig. 1 The picture of the robotic arm

2. The Different Appearances©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
84

Xue Yang, Ying Su, Yuping Mei, Haiquan Wang

Machine arms and traditional machines have distinct
differences in appearance.

Traditional human production has been unable to keep
up with basic standards, which can easily lead to a decline
in economic efficiency [1]. Therefore, in appearance,machine arms are more flexible, complex, and intelligent
compared to traditional machines

2.1. Traditional machines
Traditional machines often have a rigid mechanicalstructure, consisting of various metal and plastic
components and parts. Their shape and size are relatively
fixed, and they can only be produced and processed on alarge scale in factories.
The picture of the traditional machines is shown in Fig.
2.

3. The Application of Robotic Arms
The application of robotic arms in manufacturing does
provide solutions to many problems. Here is a detailed
explanation of some specific applications and advantagesof robotic arms in manufacturing.
3.1. Substitution for dangerous and heavy work
In the manufacturing industry, there are many dangerous,
heavy, and repetitive tasks that can have negative effects
on workers' physical and mental health. Robotic arms cantake on these tasks, avoiding direct contact with hazardous
environments while reducing their workloads. For
example, in the automotive manufacturing process, roboticarms can be used for heavy lifting, welding, and painting,which are more dangerous tasks, thus protecting workers
from work-related injuries.

Fig. 2 The picture of the traditional machines

3.2. Improving production quality
2.2. Robotic arms

The precise control and repeatability of robotic arms
enables them to perform production tasks more accurately
than humans. Through pre-set programs or precise sensor
technology, robotic arms can ensure consistency andproduct quality during the production process. On
assembly lines, robotic arms can precisely assemble andtighten components, ensuring that each product meetsspecifications and quality requirements.

The design of robotic arms typically adopt a more flexible
mechanical structure, with multiple movable joints andarms, as well as various sensors and actuators at the end.
These joints and arms can move independently, allowing
the machine arm to adapt to various work environments
and tasks. In addition, machine arms are usually equippedwith various sensors, such as visual sensors, tactile sensors,and force sensors, to perceive the surrounding
environment and perform precise operations.
The picture of the is robotic arms shown in Fig. 3.

3.3. 24-hour uninterrupted work
Robotic arms can work continuously without needing torest or sleep. They can work for 24 hours straight, which
improves production efficiency and production cycles. In
contrast, humans need to rest and recover and cannot work
continuously for long periods of time. Therefore, roboticarms are very useful in situations where high production
volume and rapid production are required.
In short, the application of robotic arms in
manufacturing can solve many problems, including the
replacement of dangerous work, improving production
quality, 24-hour uninterrupted work, and increasing
production efficiency and economic efficiency. With theFig. 3 The picture of the robotic arms

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
85

Application and Differences ofcontinuous progress and development of technology, theapplication of robotic arms will become more and more
extensive, bringing more innovation and value to the
manufacturing industry.

or high-intensity work, thereby improving work efficiency
and quality.
The picture of the cooperation between robotic arm andhuman is shown in Fig. 4.

4. The collaboration
4.1. Collaboration between robotic arms and traditional
machines
Robots are highly automated and precise machines that canperform a series of repetitive physical tasks. Traditional
machines are more versatile machines that can perform awider range of tasks. By combining robots with traditional
machines, we can create a more efficient and flexible
production line.
For example, in the manufacturing industry, traditional
machines can complete a series of complex processing
tasks, while robotic arms can be responsible for repetitive
tasks such as handling, assembly, and inspection. This
collaborative approach allows traditional machines androbotic arms to each leverage their strengths, thereby
improving overall production efficiency. At the same time,
this collaboration can also reduce production costs, as
robotic arms can replace humans in performing dangerous
or high-intensity tasks, reducing the reliance on humanlabor.

Fig. 4 The picture of the cooperation between robotic arm andhuman

In general, there is no contradiction between roboticarms, traditional machines, and human labor, but rather acollaborative relationship. By using robotic arms andhuman labor in a reasonable manner, we can fully leverage
their respective strengths and improve overall production
efficiency and economic benefits. In the future, as
technology continues to advance and innovate, thiscollaborative approach will become increasingly common
and important.

4.2. Cooperation between robotic arm and human
5. Conclusion

The collaboration between robotic arms and human beings
can be divided into two aspects: one is human-computerinteraction, and the other is machine-assisted human.
Human-robot interaction refers to the execution of tasks
by a robot arm under the guidance of a human. In this mode,
humans can control the actions of the robot arm byentering commands or operating it. This collaborative
approach is often used for dangerous or high-intensity
work, such as deep-sea operations or space exploration.
Through human-robot interaction, humans can fully utilize
their creativity and flexibility while avoiding somedangerous tasks.
Machine-assisted human labor refers to the use ofrobotic arms as an assistive tool to help humans complete
complex tasks. For example, in the medical field, roboticarms can assist doctors in surgical operations, thereby
improving the accuracy and efficiency of the procedure.
This collaborative approach can free humans from tedious

Through in-depth research, it is found that there is nocontradiction between robotic arms, traditional machines,
and human labor, but rather they can collaborate with eachother to improve overall production efficiency andeconomic benefits.
In summary, robotic arms, traditional machines, andhuman labor each have their own advantages anddisadvantages, and are suitable for different production
scenarios. In the future, with the continuous advancement
and innovation of technology, we can fully leverage theirrespective strengths to improve overall production
efficiency and economic benefits. At the same time, we
also need to focus on how to better achieve humanmachine collaboration, improve production safety, andproduct quality.
The picture of the human-machine collaboration isshown in Fig. 5.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
86

Powered by TCPDF (www.tcpdf.org)

Xue Yang, Ying Su, Yuping Mei, Haiquan Wang

Ms. Yuming Mei
She is studying at Tianjin Universityof Science and Technology, majoringin Robotics Engineering. She is very
interested in drones.

Fig. 5 The picture of the human-machine collaboration

Mr. Haiquan Wang
He is currently an undergraduate
inTianjin University of Science
andTechnology. He is majoring in
Artificial intelligence. His field of
research is deep learning.

Acknowledgements
First of all, I would like to thank the experts and scholars
in the fields of robotic arms, traditional machines, andartificial intelligence for their valuable experience andinsights, which have provided rich research materials andideas for this article. I would like to thank the reviewers
and editors for their hard work and guidance, which haveprovided valuable comments and suggestions for the
improvement and enhancement of this article.
References[1] Qi Li. Application analysis of robotic arms in the fieldof mechanical engineering technology. TechnologicalInnovation and Application, 2021, 14(3): 182-184.

Authors Introduction
Ms. Xue Yang
She is studying in College ofElectronic
Information
andAutomation, Tianjin University of
Science and Technology, China. Her
research interest is Robotics Arm.

Ms. Ying Su
She is studying at Tianjin Universityof Science and Technology, majoringin Robotics Engineering. Passionate
about drone machines.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
87

A Deep Exploration of the Mounting Issues Related to Six Rotor UVA
Yuping Mei*, Ying Su, Xue Yang

Tianjin University of Science and Technology, Tianjin, China
E-mail: *2070909305@qq.com

Abstract
In recent years, the hexacopter UAV has developed rapidly, the technological breakthrough in the field ofautomatic driving, which is of great significance in both military and civilian fields. The six-rotor UAV uses six
rotors as the power source and adjusts the attitude by changing the rotor speed to further achieve position control.
It has excellent hovering ability and sensitivity, and is equipped with a precise positioning system and advanced
sensors. However, there are still common problems such as weak mounting capacity and single mounting mode,in view of this problem, this paper will test the fuselage structure, avionics system and power system one by one
from two aspects: changing the fuselage structure and installing the motor position.
Keywords：the hexacopter UAV, weak mounting capacity, changing the fuselage structure, installing the motor

position

1. Introduction
2. The Development Process and Characteristics ofHexagoning UAVs

With the continuous development of science andtechnology, drones have become an important tool in themodern military and civilian fields.
As a new type of UAV, hexagoning UAV has thecharacteristics of high flight altitude and long endurance,
so it has a wide range of application prospects in militaryreconnaissance, geological exploration, agricultural plant
protection and other fields.
However, in actual use, hexagoning UAVs often
encounter some technical and performance problems in
the process of mounting, such as weak mount capacity
and single mount mode, which limits its wider
application. Therefore, it is of great significance toconduct in-depth discussion and research on themounting problem of hexagoned UAV.

2.1 Development history
The development history of hexagoning UAVs can betraced back to the earliest quadrotors. With thecontinuous innovation of technology and the continuous
expansion of demand, hexarotor UAVs, as a multi-rotor
aircraft, have gradually emerged in various fields. Early
six-rotor UAVs were mainly used in scientific research
and military applications to perform tasks that required
high maneuverability and flexibility. With the rise of
aerial photography technology, hexacopter UAVs havebegun to be widely used in the civilian field for aerial
photography, monitoring and survey work. In recent
years, the application of hexacopter UAVs in emergency
rescue, agriculture, logistics and other fields has also
increased.

2.2 Features
Just as shown in Fig. 1, the hexacopter UAVs have thefollowing characteristics. (1) The multi-rotor design:Compared to quadcopter drones, six-rotors employ more

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
88

Yuping Mei, Ying Su, Xue Yangrotors, making them more stable during vertical take-off,landing and hovering. This design improves mobility andadaptability to complex environments, making it ideal for
performing specific tasks. (2) High maneuverability:
Hexacopter UAVs have excellent maneuverability due tomore rotors and are able to flexibly cope with a variety offlight missions. This mobility makes it particularly
advantageous when performing search and rescue
missions in emergency situations. (3) Load capacity:
Hexacopter drones typically have a higher load capacity
and are able to carry more sensors, cameras, or otherequipment.

determine the drone's position relative to these reference
objects.
3.2. Sensor system
Inertial Measurement Unit (IMU): The IMU includes
accelerometers and gyroscopes to measure the linear
acceleration and angular velocity of the aircraft,
supporting flight control and navigation. Magnetometer:
Magnetometers are used to detect the geomagnetic fieldto help determine the direction and heading of an aircraft.
Barometer: A barometer is used to measureatmospheric pressure, which provides altitude
information. It is often used in combination with othersensors to improve the accuracy of altitude
measurements.
Millimeter-wave radar: Provides high-precision
obstacle detection and ranging capabilities, suitable forflying in complex environments.
Infrared sensor: Used to detect thermal radiation fortarget detection and identification at night or in low-light
conditions. Optical Flow Sensor: Used to measure motion
relative to the ground to help achieve hovering and stableflight.
Temperature sensors: monitor the ambient temperatureand play an important role in some special applications
such as meteorological research, cold chain logistics, etc.

Fig. 1 A hexacopter UAVThis makes it stand out for tasks that require large
equipment, such as scientific research, monitoring, andsurveys. Diversification: Due to its flexibility andstability, the application of hexacopter UAV in various
fields is constantly expanding, including but not limited
to military reconnaissance, aerial photography,
agricultural spraying, cargo transportation, etc., showing
diversified application scenarios.

4. Analysis of the Current Situation of the Mounting
Problem of Hexagophoid UAV
4.1 Mountability issues

3. The Function of the Six-wing UAV is Realized

Load capacity limitations: The structure and powersystem of a hexacopter drone may not be designed towithstand large or heavy equipment, resulting in reduced
performance or unstable flight when loaded.
Reduced flight time: Large loads often cause the drone
to consume more power, resulting in a lower flight time.
The weak payload capacity may limit the durability of the
UAV to perform tasks in the air.
Height impacted: Hexacopter drones with heavy loads
can lead to increased sensitivity to altitude changes anddifficulty in stable flight at different altitudes, especiallyat low altitudes or in complex environments.
Reduced handling performance: Large or heavy loads
may increase the inertia of the drone, making it moresluggish or inflexible when maneuvering, affecting the
maneuverability of the flight.

3.1Positioning system
Global Positioning System (GPS): GPS is one of the mostcommon and widely used positioning systems that
determines the location, speed, and altitude of a drone
through satellite signals. It provides global location
information for flight, but in some environments, such asurban canyons or dense woodlands, there may be signal
occlusion issues.Inertial
Navigation System (INS): An inertial navigation
system uses sensors such as accelerometers andgyroscopes to measure the linear acceleration and angularvelocity of an aircraft to estimate its position andorientation. However, errors that accumulate over time
can lead to navigation drift and are therefore often used inconjunction with other positioning systems.
Visual positioning system: Hexacopter drones canachieve visual positioning through cameras or multiple
cameras. Computer vision algorithms can identify
landmarks, landmarks, or environmental structures to

4.2 Single mount typeThe reason for the single type of hexacopter UAV mount
may involve the following factors:©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
89

A Deep Exploration ofTechnical limitations: When manufacturing hexacopter
drones, technical and engineering constraints may resultin limited payload types. For example, flight controls,
powertrains, or structural design may make it more
complex or impractical to support multifunctional
mounts.
Cost considerations: Versatile mount systems can add
to manufacturing costs and complexity. In order to keep
costs under control or to offer a more competitive price,
manufacturers may choose to adopt a design with a single
mount type.
Regulatory and compliance: Regulatory requirementsin specific industries or regions may impose restrictions
on the design and use of drones, leading manufacturers to
select specific payload types to comply with regulatory
requirements.

Efficient Power Management: Optimize power
management systems to increase battery capacity andperformance to support sustained flight time when
mounting large equipment. The use of advanced energy
storage technology and power management systems is a
key factor.
System integration and collaborative optimization:
Implement comprehensive optimization in the entire
system, including the collaborative work of the mounting
system, flight control system, power system, etc. By
considering the individual components together, theoverall performance can be improved. Technological
Improvements
5.2. Function optimization
Design a universal mount system: Design a system that
supports multi-functional mounts, allowing users to swap
out different types of devices or sensors. Such a system
can increase the flexibility of the drone, making itsuitable for a variety of application scenarios. Simulink
was used to model the UAV with six degrees of freedom,and the motion attitude characteristics of the UAV during
the gliding process, the influence of ailerons on the
motion of the UAV, and the movement attitude
characteristics of the UAV in the environment of windinterference were analyzed [3].
Modular design: The modular design allows the mountsystem to be easily integrated and replaced. This allows
users to quickly adjust the mount to the needs of thespecific task, thus achieving versatility.
Open interface standards: Develop or adopt open
interface standards to ensure that a wide range of devices
and sensors are compatible with the drone's payload
system. This helps drive industry standards and improvedevice interoperability.
Technology Upgrades and Updates: Regular
technology upgrades and updates are carried out tosupport the integration of new types of devices andsensors. This helps to keep the drone's technology
competitive and adapt to the development of emerging
technologies. Combined with the sliding mode observer
to estimate the fault information matrix, a fault-tolerant
controller for a six-rotor actuator based on the switchingsystem was designed by using the handover control
algorithm. The simulation analysis shows that thedesigned fault-tolerant control system has good control
performance and robustness, which can effectively
reduce the impact of actuator monowing damage on flight[2].

5. The Solution to the Mounting Problem of the
Six-wing UAV
In response to the above problems, we put forward somepossible solutions and suggestions:5.1. Performance optimization
Structural and powertrain optimization：The structure andpower system of the drone were redesigned to enhance its
load capacity. This could include the use of lighter but
stronger materials, optimized motor and propeller
configurations, and improved battery technology toprovide greater power output. A fixed-time control
scheme based on integral terminal sliding mode can be
designed to realize the trajectory tracking control of UAV,
and the fixed-time convergence performance of theclosed-loop control system is analyzed through the
Lyapunov theory [1].
Multi-Functional Mounting System: Design a systemthat supports multi-functional mounting, allowing for
quick replacement and integration of different types ofdevices or sensors. Sensors such as gyroscope,
accelerometer and magnetometer can be used to obtainthe current attitude information of the aircraft in real time,
the whole flight control system can be designed as awhole by using the modular idea, and the complementary
filtering algorithm based on the quaternion and thecascade PID control algorithm can be used to realize thecurrent attitude solution and control output of the aircraft.
Intelligent flight control system: The intelligent flightcontrol system is introduced to achieve more accurate
flight control through advanced algorithm and sensor data
fusion, and improve stability and maneuverability whenloading loads. Fault-state controllers can be designed
using a backstepping method that sacrifices yaw control
to achieve hovering and simple trajectory tracking.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
90

Powered by TCPDF (www.tcpdf.org)

Yuping Mei, Ying Su, Xue Yang

Authors Introduction

6. Future Research Directions and Development
TrendsMs.Yuping Mei
She is studying at Tianjin
Universityof
ScienceandTechnology, majoring in Robotics
Engineering. She is very interested
in drones.

With the continuous development of science andtechnology, the research on the mounting problem ofhexapopter UAV will become an important direction in
the future. In future research, we can try to apply
advanced materials and technologies to the mounted
equipment to improve the performance and stability ofthe equipment, and at the same time, we can further study
the co-design method of the aircraft and the mounted
equipment to achieve better coordination and overall
performance. These research results will provide stronger
support for the application of hexagophoid UAVs andpromote the continuous progress of UAV technology.

Ms.Ying Su
She is studying at Tianjin
University of Science andTechnology, majoring in Robotics
Engineering. Passionate about
drone machines.

7. Conclusion
In this paper, the mounting problem of hexacopter UAV
is studied in depth, and the types of mounting equipment
and their influence in different application scenarios are
discussed. Through the analysis of camera mounting,multi-rotor design, high maneuverability, etc., thedevelopment process and characteristics of six-rotor
UAV are comprehensively demonstrated.
The paper mentions the methods to solve the problem
of weak payload capacity, including structural
optimization, multi-functional payload system andintelligent flight control system.
Finally, the importance of improving the adaptability
and flexibility of UAVs to meet the needs of different
missions was emphasized. Through the discussion of thesingle problem of sensor and mount type, it provides
useful theoretical support for the field of UAV researchand points out the direction for future technology
development.

Ms.Xue Yang
She is studying in College ofElectronic
Information
andAutomation, Tianjin University of
Science and Technology, China.
Her research interest is Robotics
Arm.

Reference
1. Xiong Hang, Zhang Haichao, Qin Ke. Integrated control
method of UAV fixed-time position and pose based on
jamming observer. Manufacturing automation, 2023, 45(01):149-155.
2. Wang Siming, Wang Tianyu, Wang Yihu. Fault-tolerant
control of six-rotor UAVs based on switching system. 2020,(06): 35-41+48. DOI:10.13645/j.cnki.f.d.20200528.001.
3. Li Wenqiang, Peng Xuefeng, Zheng Zhiqiang. Six-degreeof-freedom simulation of UAV based on Simulink. Journalof System Simulation, 2007 (19): 4604-4606.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
91

The Application of Hexacopter UAV in The Field of Climbing Evasion
Ying Su*, Yuping Mei, Xue Yang

Tianjin University of Science and Technology, Tianjin, 300222 China
E-Mail: *sy789464@outlook.com

Abstract
With the development of tourism to climb peaks, the safety of high mountain walls plays an increasingly important role.
In the past, the terrain of the mountain wall was too steep and too fast, and people could not patrol the mountain wall
climbers, resulting in the safety of climbers could not be guaranteed, and the needs of climbers were difficult to solve intime. In order to solve the above problems, the method of real-time monitoring of UAVs and delivery of materials by
UAVs was proposed. Hexacopter UAV is an unmanned small aerial vehicle equipment that can carry out vertical lifting,which is expected to improve the safety factor of climbing mountain walls and solve the needs of climbers in a timely
manner.
Keywords: Hexacopter UAV，real-time monitoring，delivery of materials，mountion climbing

vertical lifting, hovering, and loading. The six-rotor
UAV has three groups with a total of six motors placed
coaxially up and down to provide lift, adjust the attitude
by changing the rotor speed, and realize the position
control through the change of attitude.

1. Introduction
In order to ensure the sustainable development ofclimbing tourism, it is urgent to improve the safety ofhigh-altitude operations. This study mainly analyzes thefeasibility and problems of hexacopter UAV in the fieldof climbing. The stability and load-bearing capacity of
the hexacopter UAV determine the bright future of thehexacopter UAV in the field of climbing. Among them,the most significant application of hexacopter UAVs inthe field of climbing is to provide magnificent highaltitude shooting conditions. As climbers conquer steep
peaks and climb steep rock faces, hexacopter drones
with high-definition cameras can record the scene.
Aerial footage can not only be used to monitor the
climbing process, but also can be used as promotional
material to promote climbing projects and attractions,and attract more people to participate in climbing. In thecase of very mature UAV monitoring technology, thereal-time recording and monitoring of high-altitudeconditions by the hexagoter can play a very good role inthe climbing tourism industry.

2.2. Features of the six-rotor UAV
The six propellers of the hexagoning UAV are
distributed in hexagonal vertex positions, with six
propeller speeds as six input forces and six degrees of
freedom as outputs. The rotation direction of the six
propellers is not the same, and when encounteringstrong external interference or part of the rotor is
disturbed, the output of multiple propellers controlling
the rest of the propellers can show good stability [1].
(1) The flight altitude of the hexcaopter
Hexacopter drones typically fly at altitudes between
1,500 and 3,000 meters, but there are some professional
drones that can fly to higher altitudes. The technologies
that determine the flight altitude of hexacopter UAVs
mainly include the following aspects: flight control
systems, aviation power systems, communication andnavigation
systems,
andmeteorologicalandenvironmental sensors. The combined effect of thesetechnologies determines the flight altitude of the drone.

2. Introduction of six-rotor UAV
2.1. The concept of a six-rotor drone
UAV is a small unmanned aerial vehicle with theadvantages of flexible flight performance such as

The flight control system is one of the key factors that

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
92

Ying Su, Yuping Mei, Xue Yang

determine the flight altitude of the drone, which
includes the flight controller, attitude stabilization
system, and altitude control system. These systems
work together to ensure that the drone can fly stably andmaintain a predetermined altitude [2].

an important role in commercial areas such as logistics
and distribution, building inspection, and power
inspection. With the continuous progress of technology,
UAVs are gradually used in emergency medical rescue,disaster monitoring and rescue, providing a new meansof ensuring the safety of people's lives and property. The
development of hexacopter UAVs has not only enriched
the application field, but also provided people with more
efficient and safer solutions, becoming an indispensable
part of modern society. This study introduces theapplication of hexacopter UAV in the field of climbing,and solves the gap that there is no hexacopter UAV
involved in the climbing tourism industry.

The aerodynamic system includes engines, propellers,and power system controls. These systems provide thepower needed to fly and influence the drone's climb
performance and altitude performance.
Communication and navigation systems also play avital role in the flight altitude of the drone. Satellite
navigation systems (such as GPS) and ground console
devices provide precise navigation and positioning
information for six-rotor drones in the air, allowing
them to accurately control altitude and position.

4. Analysis of the advantages of hexacopter UAVs inthe field of climbing
4.1. High-altitude monitoring function

Meteorological and environmental sensors can detect
meteorological factors such as air pressure, temperature,humidity, etc., helping the drone adjust its flight altitude
and attitude according to the current environmental
conditions.

The high-definition high-magnification zoom camera
embedded in the six-rotor drone can clearly see objects
one to two kilometers away from the drone. Picture of ahigh-altitude surveillance drone is shown in Fig. 1.
Equipping the six-rotor drone with a stable gimbal
gimbal stabilization function, coupled with thesmoothness and regularity of the drone's flight, canobtain enough stable images and lenses for surveillanceeven at full zoom. Regardless of the aircraft's movement,
the gimbal maintains its orientation and inclination. The
drones responsible for surveillance can navigate
autonomously. "Waypoints were previously introducedon the map via a desktop computer or tablet, where
waypoints were added by clicking on the map andnavigation was stored in the drone's memory. The
waypoint can be grouped into one or more points ofinterest. Navigation can be easily initiated via radio
control, so there is no need to use a computer or tablet
in the field, and the drone can be in the air in a matterof minutes.

To a large extent, these technologies have been very
mature, especially in the professional field, the flight
altitude control technology of UAVs has been very
sophisticated, and has provided reliable technicalsupport for various application scenarios.
(2) The volume of the hexacopter
The size of drones can generally be divided intominiature, small, medium, and large. The size of small
UAVs is generally between 30 cm and 1 meter, and it issuitable for aerial photography, agricultural plant
protection, environmental monitoring and other fields.
Among them, medium-sized UAVs are between 1 meterand 5 meters in size, which are suitable for use in searchand rescue, logistics and distribution, geological survey
and other fields. The carrying capacity of a drone isusually related to its size. Medium-sized UAVs have alarge carrying capacity and can already carry relatively
heavyweight equipment and materials, such as rescue
equipment, survey and monitoring equipment, etc.
3. The application status of six-rotor UAVs
At present, the application field of hexacopter UAV
is expanding day by day, covering military, civilian andcommercial aspects. In the military field, hexacopter
UAVs can be used for reconnaissance, surveillance,
target positioning and strike, providing a new way ofintelligence acquisition and combat. In the civilian field,
UAVs are widely used in aerial photography, geologicalsurvey, agricultural plant protection, weather
monitoring and forest fire prevention. Drones also play

Fig. 1 A high-altitude surveillance drone
4.2. Navigation features
Hexacopter drones play an important navigationalrole in the field of rock climbing. Rock climbers often
need to face complex terrain and changeable
environments during climbing, and the navigation©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
93

The Application of Hexacopter

function of drones can provide valuable auxiliary
information for climbers.
Equipped with high-precision cameras and navigationequipment, drones can see the climbing route, rock wall
structures, and the terrain faced by climbers from the air.
This bird's-eye view provides climbers with a newperspective and helps them better understand thecomplexity and difficulty of the climbing route so thatthey can develop a targeted climbing strategy. Its realtime monitoring function can help climbers timely
obtain changes in the climbing route, such as rockcollapse, climate change, etc., and provide timely
warnings and risk warnings for climbers. This real-timemonitoring is critical for the safety of climbers,
allowing them to better respond to unexpected situations
and reduce the likelihood of accidents [3].

Fig. 2 Actual transportation of hexarotor drones
In general, the high-altitude teleportation function ofdrones in the field of rock climbing provides convenientmaterial support for climbers, provides more safety andconvenience during climbing, and adds new
possibilities and innovations to rock climbing activities[4].

The navigation role of hexacopter drones in the fieldof rock climbing can provide climbers with a full range
of information support, help them better cope withchallenges, and improve the safety and success rate ofclimbing.

5. Development prospects and challenges ofhexacopter UAVs in the field of climbing

4.3. Aerial teleportation function

The application of hexacopter drones in the field ofclimbing shows great potential, but at the same time, it
is also necessary to pay attention to issues such asprivacy protection and environmental protection. Whenusing drones for high-altitude shooting, it is necessary
to comply with local aviation regulations to ensure thatthey do not interfere with the flight of other aircraft. In
addition, when flying, it is necessary to strictly comply
with local environmental protection laws andregulations to avoid affecting the local ecologicalenvironment.

For the preparation of rock climbing competitions
and climbing events, drones can be used as a fasttransmission tool to transmit competition equipment
and judging equipment to the required locations,improving the work efficiency of organizers andparticipants. Climbers often need to carry a variety ofequipment, food and water during the climb, and drones
can be used as a high-altitude teleportation tool to help
climbers deliver these supplies to the designatedlocation. For climbers, there are often difficult parts ofthe climbing route to pass, such as steep cliff walls,
inaccessible caves, etc. Drones can carry ropes, gear,and food to reach places that climbers can't, delivering
them safely to where they are, providing them with thesupport they need. Climbers can also be provided withemergency supplies during the climb, such as deliveringmuch-needed clothing, food, and medicine to climbers
in case of inclement weather to help them get throughthe storm. Photos of actual transportation of hexarotor
drones is in Fig. 2.

6. Conclusion
The application of hexacopter drones in the field ofclimbing provides more safety and technical support for
climbers, expands the possibilities of climbing activities,
and brings new innovations to climbing sports. With thecontinuous progress of technology and the expansion of
applications, it is believed that the role of UAVs in thefield of climbing will become more prominent, bringing
more convenience and safety to climbing sports.
Acknowledgement
In the process of writing this dissertation, I would
like to thank my supervisor for his careful guidance andunremitting encouragement. At the same time, I would
also like to thank the teachers and students of thelaboratory for their support and help in academic
discussions and research work. In addition, I would liketo thank my family and friends for their understanding©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
94

Powered by TCPDF (www.tcpdf.org)

Ying Su, Yuping Mei, Xue Yangand support in my studies and life. Finally, I would liketo thank all the people who helped me in the literature
research and research to complete this paper. Thank you
from the bottom of my heart!
[1] Xiaohan Liao, Chenchen Xu, Huping Ye, et al. Unmanned
Aerial Vehicle (UAV) Application Development Key
Infrastructureand Low-altitude Public Navigation
Network Planning (in Chinese). Bulletin of the Chinese
Academy of Sciences. [ChinaXiv:202303.10019][2] Xinyao Hui. Design and implementation of composite
UAV flight control system based on STM32. North China
UniversityofTechnology,2022.
DOI:10.26926/d.cnki.gbfgu.2022.000674[3] LIU Huan, HAO Qianqian, LIU Wenhui, et al.
Development trend of UAV intelligent measurement andcontrol technology. China Science and Technology
Information, 2023, (19):67-71.
[4] Jinsheng Guan. Unmanned and emergency logistics
support under smart logistics. China Storage andTransportation,
2021,
(05):
137138.DOI:10.16301/j.cnki.cn12-1204/f.2021.05.052

Authors Introduction
Ms. Yuping Mei
She is studying at Tianjin
Universityof
ScienceandTechnology, majoring in Robotics
Engineering. She is very interested
in drones.

Ms. Xue Yang
She is studying in College ofElectronic
Information
andAutomation, Tianjin University of
Science and Technology,China. Her
research interest is Robotics Arm.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
95

Deep Learning and Embedded Based Operational Safety System for Special Vehicles
Haoran Gong*, Yumei Huang, Jiahao Xie

Tianjin University of Science and Technology, Tianjin, China
E-mail: *1465157817@qq.com
Abstract

Aiming at the characteristics of special operation vehicles, such as complex operating environment, heavy
vehicle weight, long braking distance, and many visual dead angles for drivers, this project designs andimplements a special vehicle operation safety system based on deep learning and embedded system. Firstly,
the deep learning model is applied to detect the humanoid target appearing around the vehicle, get thedetection frame of the humanoid target, and estimate the distance from the target to the camera; using theadvantage of the embedded system which is relatively small can be easily deployed on the heavy vehicle.
According to different distances, the system will issue relevant voice prompts to remind the driver to taketimely measures such as avoiding or braking.
Keywords：special operation vehicles，Deep Learning，Embedded system，Complex working.

1. Introduction
Blind spots in the field of vision of specialised vehicles
during operations are a serious safety challenge that canlead to a wide range of accidents, as the vehicles tend to
be large-bodied. These drivers may not be able to directly
observe the blind spots in their field of vision, increasing
the risk of hazards to workers, other vehicles andobstacles.
Fig. 1 Common Heavy Vehicles
For example, in specialised vehicles such as excavators
or loader trucks (Fig. 1 below). drivers may not be able
to directly see pedestrians or workers around them due tothe special construction of the equipment itself,
increasing the risk of run-over or collision accidents. In
addition, when turning, working or reversing, the blindspot in the field of vision may also result in the driver not
being able to perceive other vehicles behind the side,which may lead to a collision accident. Drivers are alsoless likely to be aware of fixed obstacles such as rocks
and pillars, which can lead to collisions [1].

In order to counteract these risks, specialised vehicles
can take appropriate measures. To this end, we havedesigned a system that helps drivers to better avoid anddetect obstacles in a timely manner, for example byinstalling auxiliary cameras and sensors. The use ofadvanced assistance systems, the strengthening of thedriver's field of vision and the incorporation of a hazard
warning system reduce the probability of safety accidents
caused by blind spots or obstacles in the field of vision of
specialised vehicles during operations [2].
The rest of this paper is organised as follows. Part II
describes the hardware usage scheme. The third partdescribes the innovative points of the design as well as
the leading advantages. The fourth part explains theapplication scenarios using the design. Part Vsummarises the main points of the paper.
2. Hardware usage programme©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
96

Haoran Gong, Yumei Huang, Jiahao Xie

2.1 Jetson nano

Cameras are generally selected industrial cameras, such
as Hikvision cameras or select Dahua cameras. The
advantages of these two cameras are high-definition,clear images, large shooting angle, can be rotated and in
the dim light conditions, has a strong night vision effect.
They provide a more accurate video feed to the central
processor. Fig. 4 below shows the Hikvision camera
shooting debugging screen.

Hardware selection, we take into account the costeffectiveness and durability of the two major aspects ofthe main development board selected NVIDIA series ofdevelopment boards - Jetson nano development board.
Here to introduce this development board type: Jetson
nano is NVIDIA's development of high-performance
GPU equipped with a microcontroller, is to achieve deeplearning
andparallelcomputingnecessary
microcontroller. It is broken down into four different
types of development boards. They are: Jetson Nano,
Jetson TX, Jetson Xavier NX, and Jetson AGX Xavier,where we choose Jetson Nano. Because of its low energy
consumption and low cost it takes the lead among theother three microcontrollers. And its arithmetic powerhas been increased by tens to hundreds of times, evenmore than a small part of the current computer CPU, and
NVIDIA also equipped with a full set of SDK for it. thefollowing Fig. 2 is the picture of Jetson Nanodevelopment board, Fig. 3 is the development board coreboard [3].

Fig. 4 Debug screen
3. Innovative design and advantages
3.1. Deep Learning
The initial idea of our design is to first recognise thehuman figure, and then to warn the driver by triggering
an alarm when the human target walks into a relatively
dangerous area of the vehicle's operation.

Fig. 2 Development Board

Due to the relative complexity of the special vehicle
operating environment, traditional computer vision andimage processing algorithms are relatively fuzzy anddifficult to deal with to identify the humanoid target fromthe environment, and often triggers the alarm system
incorrectly (e.g., a roadblock or reflective clothing),
therefore, deep learning models are used in this project tocomplete the identification of the humanoid target, whichis shown in Fig. 5.

Fig. 3 Core Board

The jetson nano performance metrics (representative)
are shown in Table 1.
Table 1 jetson nano performance metrics
arithmetic power
40TOPSGPURAM

1024-core NVIDIA Ampere
architecture GPU with 32
Tensor cores
6-core Arm® Cortex®A78AEv8.2 64-bit CPU
1.5MB L2+4MB L3
8GB+68GB

Camera Interface

2 CSI interfaces

output

5W-10WCPU

Deep learning is a research direction in the field ofmachine learning. Deep learning uses deep neuralnetworks for modelling and uses large amounts of data to
optimise the parameters in the model to achieve highaccuracy and robustness.
We have designed a deep learning model that canrecognise targets within 50 to 100 metres in terms of
distance, up to 9000 types of objects (yolo-9000) in termsof type, and hundreds or even thousands of targets in asingle frame at the same time.

2.2 surveillance camera©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
97

Deep Learning and Embedded

unnecessary distractions for the driver. In addition, if thevehicle is travelling fast in a complex scene, there may be
a requirement for the real-time performance of the model.
Generally in complex scenes there will be a large
number of staff and supervisors. We want to recogniseand estimate their distance to their own vehicle as muchas possible.

Fig. 5 Initial identification of deep learning models
3.2. Technical difficulties

3.3. Problem-solving responses.

There are many types of special vehicle operating
scenarios such as harbours, construction sites, mining
sites, as in Fig. 6. intensive construction of people, many
obstacles and close vehicle traffic. Sand, gravel, goods,
and materials are piled up. These serious obstacles to ouracquisition of the surrounding image information.

For the human-type and object-type densification
problems, we have prepared datasets for training. A
certain amount of data is collected to form the dataset fortraining and the dataset for testing. The amount of data is
usually 2000 images, which contain at least 5000
humanoid targets. The dataset is labelled using manual
annotation (Fig. 7). At the same time, the surrounding
environment is constructed (Fig. 8).

Fig. 7 Dataset labellingFig. 6 List of construction scenes at mines andconstruction sites

Fig. 8 Environmental scanning
The mainstream deep learning yolov5 and the deepneural network Pedestrian net, which is specialised forhumanoid and vehicle detection, are deployed on jetson
nano [4].

In addition, because it is assembled on the carrier, it isrequired to have low energy consumption; and the spaceinside the vehicle is narrow, which can only
accommodate smaller equipment for normal operation,so we can only compress its size.

The model uses ResNet34 as the basic underlying layer
and combines advanced techniques such as feature fusion,
local zoom, and exact query. The original model is
84.62MB, after pruning 7.85MB, then use Tensor RT for
assisted optimisation, generate the physics engine in

Due to the ever-changing environments in complex
places, we cannot predict what will happen next. So weneed as little false recognition or frame skipping as
possible. Too many misjudgements can cause©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
98

Haoran Gong, Yumei Huang, Jiahao Xie

semi-precision form, and then deploy the physics engine
into deep learning, then after a series of operations can bedone on a development board to deal with four to eight
video signal sources.

tunnel for light reinforcement, passed to the pos switch,
and finally complete the exploration of the environment
around the shield tunnel machine. Fig. 9 below shows acomparison of the colour of the enhanced picture in dark
conditions for the night vision effect.

Finally, the target detection metadata is sent in the formof a request to the thread responsible for estimating the
distance and voice announcement, and multiple
calculations are performed to compare the estimatedrelative distance with a pre-set threshold to minimise theimpact of erroneous judgements. Finally, an alarm istriggered to warn the driver to be careful of thesurroundings.
4. Job Scenario Application.
4.1. yard operation
Fig. 9 Night Vision Enhanced Picture Colour Before andAfter Comparison

Heavy vehicles loading and unloading in port yards
face serious visual challenges, with a large number ofvisual dead zones and blind spots. At the same time, theconstruction site is a noisy environment with a large
number of people, which makes it difficult for the driverof the operating vehicle to detect people in the vicinity ofthe vehicle in a timely manner in complex situations, thus
spawning many safety hazards.

Fig. 10 below shows the final physical model drawing
and the final physical picture.

We use a four-channel video signal source to assist theoperation. The four video sources are processed
simultaneously on a single jetson nano, which canachieve 10 frames per second for each source. The
communication is done through a pos switch and finallya voice alarm system.
4.2. Shield Tunneling
Shield Tunnel Battery Vehicle is an electric vehicle
specially designed for shield tunneling project, and its
main task is to transport materials and transfer personnel
at the tunnel construction site.
However, due to the poor line of sight in the tunnel andthe heavy load of the vehicle, there is a great safety
hazard in the construction. The main reasons include: theexistence of a blind field of vision, the existence of a nosignal section in the tunnel, the absence of reminders
during operation, the possible failure of the brakes, andimproper operation.

Fig. 10 Physical models and final objects
5. Synthesis
This paper describes the various safety hazards ofspecialised vehicles in complex situations and how todeal with them. A machine has been designed to assist
special vehicle drivers to avoid accidents. It can beadapted to various complex sites including yards,
harbours and mines. By coping with various conditions

We made a solution to these situations in the follow-up
processing, first of all, the tunnel front and rear of about
fifty metres distance sweep, the construction workers tocollect data, and then through our are reinforcing thenight vision effect algorithm will be the situation in the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
99

Powered by TCPDF (www.tcpdf.org)

Deep Learning and Embeddedwith ease. It is believed that in the future, it can help more
drivers and escort the safe operation of special vehicles.
6. References[1]. Zhao, J.F., Zhao, J.L., Li, Q., et al, Research onmachine vision-based fast-passing technology forspecial vehicles at intersections, Design andApplication, Vol. 31, No. 2023, pp. 240-246.
[2]. Liu X, Chacoco, An environmental target
recognition method for special vehicle operations at
airports, Computers and Modernisation, Vol. 2023,
No. 8, pp. 18-24.
[3]. Fangxin Xu, Rong Fan, Xiaolu Ma, Improved
YOLOv7 Algorithm for Congested Pedestrian
Detection, Computer Engineering, November 2023,
pp. 1-10.
[4]. Lu Beiyao, Zhang Tiecheng, Song Yang, et al,Control system of a hospital ordering robot basedon JETSON NANO, Intelligent Applications, Vol.
14, 2021, pp. 39-41.

Authors Introduction
Mr. Haoran Gong
He is currently studying in the School ofElectronic Information and Automationof Tianjin University of Science andTechnology, and is proficient inembedded architecture with strong
single-player working ability.

Ms. Yumei Huang
She studied at Tianjin University of
Science and Technology in 2021. She is
now the first major in her faculty and is
expected to receive her Bachelor's
degree in 2025.

Mr. Jiahao Xie
He studied Oceanographic Sciences at
Tianjin University of Science andTechnology in 2022 and is currently abachelor student of science majoring in
Marine Technology. He expects toreceive a bachelor's degree in 2026.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
100

"Green Fruit" - Intelligent Traceable Agricultural Product Production and Marketing
Platform Based on Blockchain Technology
Yumei Huang*, Jiahao Xie, Haoran Gong, Ziyue Xiao

Tianjin University of Science and Technology, Tianjin, China
E-mail: *3033665674@qq.com

Abstract
The production process of agricultural products has been greatly improved and optimised through the intelligent
agricultural greenhouse system. Using intelligent irrigation, measuring CO2, IoT and other technologies, it creates agreenhouse intelligent control system that is simple to operate, highly automated and intelligent. In addition, using
the double combination of front-end architecture and applets, it can record all the growth process, perfectly integrating
the intelligent mode + big data application. Create a tailor-made ID for agricultural products. Under our design, we
break the traditional automation mode, which is more conducive to understanding the whole growth process ofagricultural products.
Keywords: Internet of Things; automation; big data application; intelligent agriculture.

and feedback to the user's mobile phone. The user canobserve the crop growth at any time. So, IoT can monitor
various environmental parameters in the greenhouse inreal time, such as temperature, humidity, light, soil
moisture, and so on. This real-time monitoring enables
farmers to understand the growth of their crops moreaccurately and quickly adjust the conditions of thegreenhouse to optimise the production environment.
When problems arise, such as disease, pest infestation orother production challenges, the traceability system canquickly pinpoint the source of the problem. This helps
farmers take quick action to minimise losses and maintaincontinuity of production [2].
Each crop can be identified with its own identity
card. Smart greenhouses generate large amounts of data
through sensors and monitoring systems, which can beused for data storage. Farmers can make more informed
decisions based on actual environmental and production
data, and buyers can learn about their growth.
The rest of the paper is organised as follows. Section
2 presents the structure and principles of the systemmodel. Part III presents the feasibility study of the systemmodel. Part IV gives the physical pictures of theconstructed model and the verification process to finalise
the correctness of the designed. Part V summarises themain points of the paper [3].

1. Introduction
The aim of the paper is to discuss the many advantages
brought by the traceable smart greenhouse model in termsof agricultural production, food safety and market
competitiveness, as well as the initial system modelling
and related technologies established [1].
From the perspective of food safety and security By
integrating product traceability technologies, it ispossible to trace the entire process of agricultural
products from planting and growing to harvesting,processing and marketing. This helps to monitor potential
sources of contamination, diseases or other risk factors,
and to identify and deal with problems in a timely manner,
thereby improving food safety. Automated control allows
real-time monitoring of environmental parameters in thegreenhouse, such as temperature, humidity and light,
which can precisely control the growing conditions ofcrops. Through meticulous data recording, farmers can
optimise planting strategies to improve the quality andyield of their produce. The increased resource efficiency
enables smart greenhouse technology to enable precise
irrigation and fertiliser application, as well as efficientenergy use. This helps to reduce water wastage, fertiliser
use and energy expenditure, and improve thesustainability of agricultural production. The Internet ofThings (IoT) in this plays an extremely crucial role andoffers many opportunities for innovation and efficiency
gains in agricultural production. It can deliver real-time
metrics of various parameters in the model to the network

2. Introduction to the system model
Intelligent greenhouse control system is a scientificsystem for intelligent management of greenhouses.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
101

Yumei Huang, Jiahao Xie, Haoran Gong, Ziyue Xiao

Environmental data is collected through thecorresponding sensor equipment. Then, through various
commands set by the system, the management work iscompleted automatically.

requirements and optimal CO2 concentration ranges.
Once the measured CO2 concentration exceeds or falls
below the set threshold, the system triggers appropriate
control measures, such as adjusting the ventilation
system or CO2 supply unit.

At the same time, the data collected by the sensors will
be analysed and the results will be fed back to the farmers.
Under the guidance of agricultural experts, the yield isincreased and the quality is improved. In the wholesystem, QCA9531 gateway module is used as the corecomponent, and the core programme is burned into it. At
the same time, all kinds of sensors, rolling shutters, fans,water valve controllers, cameras and other hardware
devices will be accessed. When the power is to be turned
on, it can be run and controlled automatically according
to the programme. The use of sensors to obtainenvironmental data, when detected more than thethreshold value of the fruits and vegetables, will start thecorresponding actuator.

The CO2 sensor monitors the CO2 concentration in
the greenhouse in real time and transmits the data to themonitoring system. These data are usually recorded andarchived for future analysis and reference, as shown in
Fig. 2 below.

2.1 CO2 Sensor
Carbon dioxide (CO2) sensor is composed of LEDPR measuring unit, LED driving circuit, LDO power
supply, photoelectric signal processing circuit,temperature measuring circuit and microprocessor,
which is shown in Fig. 1 below.

Fig. 2 Sensor characteristic curve
The Y-coordinate represents the output signal, with amaximum value close to 330mV; the X-coordinate
represents the concentration of CO2, with a minimum
value close to 210ppm.
2.2 temperature detector

Fig. 1. Sample sensors

Temperature detection is very important forgreenhouse intelligent control system. As shown in Fig.
3, the temperature sensor is used to collect thesurrounding environmental factors, read thetemperature data, and through the signal processing
function, the peripheral devices, as shown in Fig 4,
respond accordingly and transmit the temperature data
wirelessly to the cloud platform. When the temperature
is higher than the preset temperature e, the MCU drives
a relay to control the fan rotation to reduce thetemperature. When the temperature is lower than thepreset temperature, the MCU drives the relay to control
the collector to heat up [14].

Firstly, a suitable location should be chosen to install
the CO2 sensor. The location should be chosen to
represent the average CO2 concentration of the plantgrowing area in the greenhouse. The sensor is usually
installed in a hanging position to ensure that it canaccurately measure atmospheric CO2. before use, theCO2 sensor needs to be calibrated to ensure theaccuracy of its measurement. Calibration can be carriedout by using a standard gas with a known CO2
concentration, according to the model number of thesensor and the guidelines provided by the manufacturer.
Then, connect to the monitoring system. CO 2 sensors
are usually connected to the monitoring or control
system of a smart greenhouse. This can be achieved
through a wired or wireless connection. The sensor
sends real-time measured CO2 concentration data to themonitoring system, enabling the farmer to remotely
monitor and manage CO2 levels in the greenhouse. The
farmer can set thresholds based on plant growthFig. 3 Temperature sensor DHT11©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
102

"Green Fruit" - Intelligent Traceable

Table 1 symbols denote meaning

Q(t)Battery power at time t

I ( )

Current as a function of timeτ

integral variable

Q0

Initial battery charge

The design of the automatic irrigation system is amulti-level, multi-component project as shown in Fig.
6 need to receive the sent data from the soil moisturesensor to determine whether irrigation is required. By
setting a default upper and lower threshold of normal
humidity (±3%) in the software. When the soil moistureis lower than the set lower threshold, the pump is
activated to deliver water to the underground pipeline
and spray it out to drench and moisten the soil toachieve the effect of irrigation. At the same time, thesoil moisture sensor continuously checks whether thehumidity has risen above the middle of the upper andlower thresholds, and if it meets the requirements, it
stops the delivery; if it does not meet the requirements,
it continues to cycle the process until it reaches thenormal humidity range (Fig. 7).

Fig. 4 Typical Circuit Applications
2.3 Soil moisture sensors and automatic irrigation
systems
The soil moisture sensor is chosen to be installed per
1m2 of soil by selecting the LM393 soil moisture sensor,
as shown in Fig. 5, based on the needs of the crop andthe layout of the greenhouse. Typically, the sensor isburied near the root system of the plant to ensure that it
can accurately measure the moisture in the soil. Soil
moisture sensors usually require a power supply (Eq.
(1)). The corresponding values for Equation I are shownin Table 1. Sensors usually use low-power electronic
components that can be powered by batteries or othersuitable power sources. Then connected to themonitoring system: The soil moisture sensor isconnected to the monitoring system of the shed. Realtime soil moisture values are fed back to the platform toprovide numerical feedback for subsequent soil
rehydration. The premise is to set threshold values forsoil moisture based on crop needs and optimal growing
conditions. These thresholds will be used in theirrigation system [4].

Fig. 5 LM393 Soil moisture sensors(Battery power at time t)

Fig. 6 Embedded part of the code

(1)

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
103

Yumei Huang, Jiahao Xie, Haoran Gong, Ziyue Xiao

universality test [7], which are shown in Fig. 8, Fig. 9 andFig. 10.

Fig. 7 Get DHT11 code
3. System feasibility study
The application of IoT technology in smart agriculture
has enabled real-time monitoring and control ofparameters in the greenhouses, providing a high degreeof environmental control and visibility. This helps
farmers to understand the situation in the greenhouse inreal time and take necessary measures to meet the needsof the crop and improve yield and quality.
The traceability system provides a reliable guarantee ofthe quality and safety of agricultural products through QR
codes and database technology [5]. Consumers can easily
access detailed information about the products, whichbuilds trust and improves their market competitiveness.
The high degree of automation of the greenhousesystem makes it very easy and convenient for farmers tooperate. Monitoring of environmental parameters, waterand nutrient supply in the greenhouse can be automatedthrough automatic control systems. This reduces thedependence on a lot of manpower and farmers can focus
more on monitoring and managing the system rather thanhaving to physically work in the greenhouses for longhours.

Fig. 8 on-site demonstrationBy
accurately
monitoring
andcontrolling
environmental conditions, the system is expected tosignificantly improve the yield and quality of produce.
Because the growing environment in the greenhouse canbe finely tuned, crops can be optimised for optimal light,
temperature and humidity conditions at different stagesof growth, leading to increased yields and improved
product quality.
4. Physical pictures and verification process
The model is now completed in terms of technology
development, with hardware models, WeChat small
programmes, traceability systems [6], e-commerce
website development. website, and in the future it willenter the data interaction stage and improve the audienceFig. 9 Cloud Platform Home

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
104

Powered by TCPDF (www.tcpdf.org)"Green Fruit" - Intelligent Traceable
[5].

[6].

[7].

[8].

Yang, Mina, "The design and development of "helpfarmers" sales system based on WeChat applet", Software
Development and Application, 2021, 4: pp. 34-35.
Jinwen Chen, "Research on Key Technology of HighPrivacy Agricultural Traceability Based on Blockchain,"
Master's thesis, 2023, 6
Jian Nian, "Research on the Application of Traceabilityof Agricultural Products in Smart Agriculture," Smart
Agriculture: monographs and reviews, 2021, no. 4: pp.
16-18.
Jingxuan Yang, "Current situation and future
development trend of agricultural greenhouses in China,"
Seed Science and Technology, 2023, no. 14: pp. 142-144.

Authors Introduction
Ms. Yumei Huang
She studied at Tianjin University of
Science and Technology in 2021. She
is now the first major in her faculty andis expected to receive her Bachelor's
degree in 2025.

Fig. 10 Crop "ID"5. Synthesis

Mr. Jiahao Xie
He studied Oceanographic Sciences
at Tianjin University of Science andTechnology in 2022 and is currently abachelor student of science majoringin Marine Technology. He expects toreceive a bachelor's degree in 2026.

The paper mainly describes the growing of crops insmart greenhouses under the fully automated mode.
"Traceable Smart Greenhouse" is a modern agricultural
production system that uses advanced technology andInternet of Things (IoT) to improve the efficiency ofagricultural production and realise precise agricultural
management. Including: intelligent sensor technology,
Internet of Things, automation control, precise fertiliser
application and water management. It improves theefficiency, quality and sustainability of agricultural
production, while ensuring product quality through a
traceability system to provide consumers with safer,
traceable agricultural products [8].

Mr. Haoran Gong
He is currently studying in the Schoolof Electronic Information andAutomation of Tianjin University of
Science and Technology, and is
proficient in embedded architecturewith strong single-player working
ability.

6. References
[1].

[2].

[3].

[4].

He, Y. K., Song, Y. F., Tan, P., et al. "Design ofMultifunctional Potted Plant Farming Device Based on
STM32F4", Computer Applications, 2021, 4: pp. 88-90.
Miao, Junchen, Wang, Rui Hao, You Da, et al. "An
intelligent agricultural greenhouse greenhouse detection
system based on the Internet of Things", Internet ofThings Technology, 2023, 1: pp. 16-18.
Gao Guanlei, Li Changpu, Fu Shuashuai, et al. "Designof intelligent greenhouse greenhouse control system
based on Internet of Things", Shandong Industrial
Technology, 2023, 2: pp. 71-77.
Yang, Hailong, Kou, Jian, Wen, Xiaodong, et al., "Designof Temperature and Humidity Detection System forIntelligent Buildings Based on ESP8266," Journal of
Hebei University of Architecture and Engineering, 2023,
9: pp. 177-181.

Mr..Ziyue Xiao
Hold a positive and serious attitude
towards work, have a strong sense of
responsibility, be sincere, meticulous,
optimistic, and stable, have a goodteam spirit, can quickly adapt to theworkenvironment,
andcancontinuously learn and improveoneself in practical work, and do theirjob well.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
105

A Digital Twin Design Based on Robot Workstation
Huahao Li

College of Electronic Information and Automation, Tianjin University of Science and Technology, 300222 ，China
E-mail: 1983618432@qq.com
www.tust.edu.cn

Abstract
With the rapid development of intelligent manufacturing industry and the proposal of Made in China 2025 andIndustry 4.0, Digital Twins DT has rapidly become a craze. Through the interaction between the physical object andthe virtual model, the mapping is completed in the virtual simulation space, so as to reflect the actual operation
process of the whole life cycle of the corresponding equipment workstation. The main structural content of thisarticle is the research background and significance of digital twin technology, the current research status at home
and abroad, the establishment of Solidworks physical models, the construction of PDPS (Process Designer&Process
Simulate) workstations, and virtual simulation. Completed the construction of the robot seven color board assembly
workstation, established a model of the assembly workstation, and combined this model with PDPS to simulate theworkstation. Finally, the principle of connection communication and the achievement diagram were elaborated.
Keywords: Digital Twin, Robotic Assembly Workstation, Virtual simulation, PDPSFirst of all, the overall design and planning of thedigital twin of the virtual simulation of the robot
assembly workstation should be carried out. The
following five aspects need to be studied and analyzed
respectively:(1) Understand and learn the overall structure andspecific layout of the robot assembly workstation. Be
fully prepared to build a layout in PDPS.
(2) Make and build models of ABB robot 1410
assembling workstation and other robot bases and otherparts in SolidWorks 2019.
(3) Import all the models built in SolidWorks into the
PDPS simulation software. Then build the workstation
and write the robot motion instructions and paths. In the
PDPS simulation software according to the actual layoutof the assembly workstation.
(4) Carry out assembly simulation of all processes androute debugging in PDPS software to achieve the effectof workstation assembly and operation simulation.
(5) Communication steps and principles of virtual andreal equipment of PDPS and robot assembly workstation.

1. Introduction"Made in China 2025" and "Industry 4.0" strategic plan
proposed, so that intelligent manufacturing and artificial
intelligence and a series of intelligent technologies andrelated development strategies continue to be proposed
and implemented, therefore, digital twin technology has
become a basic principle of intelligent manufacturing
direction, and has received the general attention of allparties. Digital twin technology is the basis of intelligent
manufacturing. It is gradually penetrating into
manufacturing, medical care, urban management andother fields. The practical application of digital twin
technology in control, perception, big data, artificial
intelligence, modeling and other fields has also seen acentralized outbreak and growth [1].
As an advanced stage of digital development, digitaltwin will promote the integration of intelligent,
information and digital development processes ofintegrated enterprises [2]. The development of digital
twins has reached an unprecedented height, and now
digital twins are not only an inevitable trend of national
industrial development in the future, but also an
important embodiment of the level of scientific andtechnological development.
The specific research content of this design:

2. General idea of system design
The system is mainly composed of off-line simulation
module, twin module and auxiliary function module.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
106

Huahao Li

Table 1. Model selection and size table (Unit: mm)

The overall process design flow chart of the robot
assembly workstation system structure is shown in Fig. 1.

Fig.1.Process layout design flow chart of the system

The process mainly includes: the ABB1410 modelindustrial robot is placed in the No. 1 station and No. 2
station respectively, so that the robot carries out thehandling and assembly work on the three seven-color
panels and a cover plate on the storage platform in turn,and then transported to the No. 2 station robot operates
the cover screw. After the screw is fixed, a set ofprocesses is completed, and finally the object istransported to the finished product area by the transfer
track.

No.

Name

LengthWidth

Height

one

path

2400

575

800

two

Transport
car

600

400

80

three

Bottom
plate

200

200

30

four

Cover
plate

200

200

20

five

Color
palette
one

150

150

20

six

Seven
color
palettenumber
two

150

75√2

20

seven

Color
palettenumber
three

150

75√2

20

eight

Robot
chassisRadius
350

0

500

nine

Storage
table

130

60

80

4. PDPS workstation construction and program
design and simulation

3. Robot assembly workstation

The construction of the assembly workstation is mainly
based on the layout of the robot training base to simulate
the construction of the layout. Through the actual
working principle and path of the robot, the coordinate
position of the robot and the location of the material
workpiece are first determined. Secondly, the trajectoryand mode of movement of the robot are understood, andthen the layout is carried out through the built model. As
shown in the figure, the basic layout is composed of two
ABB1410 robots, corresponding to station No. 1 andstation No. 2 of the workstation. Station No. 1 is
responsible for the assembly of 7 color board No. 1, 7
Color Board No. 2, 7 color Board No. 3 and the coverplate. It is transported to station No. 2 by the conveyor
rail, and is responsible for fixing the screws on the four
corners of the cover plate at station No. 2. Fig. 2 shows
the preliminary model construction:

The industrial robot used in the robot assembly
workstation is an industrial robot specially designed for
assembly. Compared with other industrial robots, theassembly robot has the advantages of strong flexibility,
high precision, small working range, and can be used in
combination with other operating systems. The main
driving mode of the system is electric driving mode. The
research scene of this paper should be carried out in theassembly of seven-color plates, and the advantages ofindustrial application are to improve production
efficiency and reduce human capital.
For the model built above, the specific parameters are
sorted out as follows. The model selection and size tableof robot assembly workstation are shown in Table 1
below:©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
107

A Digital Twin Design

5. TIA+PLCSIM+NETTOPLCSIM+KEPServerE
X virtual debugging
Open our robot assembly workstation in production line
mode and click Add Zone, where the path and location
can be observed [3]. Click on the motion simulation, andthe robot station OP10 starts to carry and assemble thered triangle plate, yellow triangle plate, purple triangle
plate and transparent cover plate successively. Meanwhile,
the robot station OP20 also synchronously fixed thescrews of the four stations of the cover plate. When theassembly is completed, sensor B receives the signal. The
signal is received by NetToPLCsim. And the channel 4
signal shared by KEPServerEX, the PDPS workstation
receives the virtual map of the OPC signal 1, and theconveyor begins to move. When the sensor A receives the
position signal, which is the same as the signal B, thesignal becomes 0, the conveyor belt stops moving, andthe robot station OP10 and OP20 start handling andassembly. When the stop button is pressed, the signal is
disconnected and the robot assembly workstation stops
moving. The workstation of the robot is shown in Fig. 4.

Fig. 2. Basic layout of robot assembly workstation

The robot assembly workstation is mainly composed oftwo stations, namely P10 station and P20 station. The
robot at P10 station is mainly responsible for the handlingand assembly of the 7-color board and the 7-color board
cover plate, while the robot at P20 station is mainly
responsible for the final assembly of the 7-color board by
fixing the bottom plate and the 7-color board cover plateby screws at four corners.
The overall planning of the coordinate system of therobot assembly workstation, through which thecoordinates of the robot and the robot base are connected,
can be roughly divided into the establishment of thecoordinate system of the seven-color plate and the coverplate, the coordinate system of the transport car and thechassis, and the motion coordinate system of theassembly robot OP10 station and the robot OP20 station.
As shown in Fig. 3 below.

Fig.4. Robot motion diagram

6. Conclusion
In this paper, a simple control system can be realized
through the construction and simulation of the basicmodel of PDPS software based on the working mode andbasic method of robot assembly workstation. Do model
construction, basic operation methods, path debugging.
The
virtualrealitytechnology
canrealizehuman-computer interaction. The digital twin based on
the robot assembly workstation mainly takes theassembly process of the seven-color panel as an example
to simulate and simulate the actual robot assembly
production line in the virtual space. The digital twin
pursues the life cycle of the whole process, and thisFig.3. Coordinate diagram of robot and workstation©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
108

Powered by TCPDF (www.tcpdf.org)

Huahao Li

design only adopts two stations of the robot assembly
workstation for simulation and simulation. I hope it can
provide some help for the research of relevant personnel.
References
1. Xuan Liang, ZHANG Chunfei, Liu Xiaohui, Yang Peijie.
Research on digital twin modeling method of manufacturing
production line under the background of Industry 4.0.
Journal of Jianghan University (Natural Science
Edition),2023, vol. 51 (2): 68-77.
2. S. Mihai et al. Digital Twins: A Survey on Enabling
Technologies, Challenges, Trends and Future Prospects,
Proceedings of IEEE Communications Surveys & Tutorials,
vol. 24, no. 4, pp. 2255-2291.
3. J. Wang, H. Liu, Design and Algorithm of Multi-Robot
Cooperative Welding and Assembling Workstation Based on
RobotStudio, Proceedings of 2019 International Conferenceon Robots & Intelligent System (ICRIS), Haikou, China,
2019, pp. 14-17.

Authors Introduction
Mr. Huahao Li
He is a first-year master candidate in
Tianjin University of Science andTechnology, majoring in Intelligent
multibody robot.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
109

A Study of Chemical Reactor Simulation System Based on PCS7
Hongshuo Zhai

Tianjin University of Science and Technology, Tianjin, China
E-mail: 1936130557@qq.com
www.tust.edu.cn

Abstract
Since 1960, the reason for the rapid development of industrial control and its systems is the rapid development of the
key process control theory, and their application in science and technology is more and more extensive. In general,
the factory site has a series of hazards, and people tend to take a remote, accurate approach to remote control. In thisresearch, PCS7 software is used to simulate the remote control of chemical reactor, and reasonable and correct
simulation can be obtained after PID correction.
Keywords: PCS7, PID correction, process control, chemical reactor

1. Introduction
Normal human life is inextricably linked to a series ofchemical reactions.
The chemical reactor is defined as a vessel for reaction.
The device is widely used in various fields, and theultimate goal is to obtain reactants that meet therequirements while ensuring production efficiency.
However, due to the presence of a large number ofharmful gases in the chemical reaction site that are notconducive to the human body and the environment,
affecting the physical and mental health of the practicaloperator, and because of the high requirements of thechemical reaction parameters, the previous on-site
operation has been transformed into a more accurate andsafe remote control.
This paper mainly adopts PLC-SIM as the controller,
and the controlled object and control method are realized
by PCS7. Firstly, the infrastructure of chemical reactor
PCS7 is introduced, and then the structural model andautomatic operation are built by CFC block and SFC
block. Finally, the loop is designed and PID correction iscarried out to obtain appropriate graphics.

Figure 2-1 PCS7 framework

2.1. Engineer station
The engineer station (ES station) can achieve a series ofoperations such AS new construction, editing project files,
and communication between AS station and OS station. In
general, the design of ES station is completed in thesoftware SIMATIC Mananger, and the configuration
mainly consists of two links : AS station configuration andOS station configuration.
The configuration of automation station involves thecompilation of continuous function diagram and sequentialfunction diagram, factory level design, hardware
configuration, communication network configuration andso on.
In the operator station, the process screen design andoperation function design interface are mainly, and there
are data archiving and protocol design.

2. PCS7 Project Basic Framework
In most cases, a project has ES stations, OS stations, AS
stations and buses, and each station port has its special
function [1], which is shown in Figure 2-1.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
110

Hongshuo Zhai

2.2. Automation station(1) Operator station server
The function of the operator station server is to pass thevalue in the program to another client that it has contact
with, that is, the operator station client, and transfer thecontent of the OS station to the corresponding AS function
block.
(2) Operator station client
The Data connection between the operator station client
and the operator station Server is established through theTerminal Bus, and the data in the operator station server isaccessed through Server Data.
The operator stands on the design, the configuration
design of the above machine screen is the main work,
should have the switch, operation panel CFC and SFC
block writing and other functions, simulate the factory
screen equipment diagram, in addition, there are data toarchive and so on.

The automation station (AS station) is the control stationof the Siemens DCS monitoring system, and the main
modules of the station are: RACK, PS, CPU, I/O module,etc.
The CPU in PCS7 usually uses 400, in RACK, there
are CP modules for the purpose of communication, butsome CPU modules also have the function ofcommunication directly, such as PROFIBUS DP
interface or PROFIBUS DP serial communication
module, through which to achieve device communication.
The automation station operates the system with fastindustrial Ethernet access or the engineer station accesses
the ET200 distributed I/O station using the Profibus-DP
fieldbus. Note: The "block" in PCS7 is very critical, it isrelated to the work of the CPU in the software STEP7, thespecific utility is shown in Table 2-1.
Table 2-1 Utility of blocks
BlockTissue block (OB)

System Function Block
(SFB)
and
SystemFunction Call Block(SFC)
Functional Block (FB)
Function
(FC)

call

block

Background Data block(INSTANCE DB)

Data block (DB)System
(SDB)

Data

Block

2.4. Factory bus and terminal bus

Utility
The function is to store
the user master programand control successively.
They are stored in the
S7 CPU so that they can
implement some of thereserved functions.
It is a block that storesthe user's own design.
A subroutine that does
not take up memory, but
can be used repeatedly.
When a function blockand a system function
block are used, the block
automatically connectedwith the block. No manual
addition is required.
Its purpose is to store
user data.
It stores hardware
configuration data.

Plant Bus and Terminal Bus adopt Industrial Ethernet,
which is consistent with IEC 802.3. This communication
mode is mainly applied in network structure, most ofwhich use optical fiber network as communication
structure. Both the factory bus and the terminal bus will
have good stability, and its role is to achieve reliablecommunication between various industrial communication
buses.
For some medium or large specifications of industrial
actual sites, it will have a high ball, for some industrial sites
require a relatively smooth, fast contact device, PCS7 toprovide it with optical fiber network and equipped withindustrial Ethernet technology to achieve fastcommunication. The system uses advanced optical fiberconnection technology and high-speed opticalinterconnection equipment, making it a new milestone in
the field of communication. Among them, the latest andfastest Ethernet communication technology combines thescalable performance of switching technology with thehigh security of optical fiber ring network, and has a highdata transfer rate and 1Gbit/S transfer rate, industrial
twisted pair (ITP) and optical cable (FOC) can be used.
3. Mathematical Modeling and Automatic
Operation of the Controlled Object

2.3. Operator station
Most of the time operator stations are where we use thecomputer as a medium for control purposes. The OS
configuration occurs at the engineer station, so it isfeasible to consider the OS station project as part of the
ES station project. Operator stations under distributed
operating systems also have some basic two categories,
the following is their introduction.

In PCS7, an engineering application, there are many ways
to construct mathematical models.
3.1. CFC block
Also known as continuous function diagram, it is based on
the function block to establish a numerical model of thecontrolled object, which has the function block of the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
111

A Study of Chemical

configured proportion, integration, differentiation, delay
and operation, such function block is configured on the
PCS7 has been programmed.
CFC block is used to build the object model, andoperation block is used to construct the mathematicalformula, so as to realize the mathematical modelingprocess of the controlled object.
For the establishment of the mathematical model ofchemical reactor, the mathematical modeling of liquid
level object, pressure object, feed object and temperature
object was obtained by referring to the data and using
CFC block.
1. Liquid level objects
Mainly: integral link and inertia link.
G(S)=

3.25
1+5.47S

*

2.13

(3-1)

0.97S

2. Objects of stress
Mainly: inertia link and integral link.
G(S)=

2.78
1+5.29S

*

0.91

(3-2)

5.58S

3. Feed flow
Mainly: the inertial link of its principal and subordinate
reactants.
G(S)=

1.87

(3-3)

1+5.49S

G(S)=

2.39

(3-4）

1+4.68S

4. Temperature object
Mainly: the second stage of the lag time [2].
G(S)=0.6 ∗

1
4.29S

∗

1
3.93S

Figure 3-1 Simulating the SFC starting sequence(3-5）

3.2. SFC block4. Design of Advanced Control Scheme forSimulation System of Chemical Reactor

In the process of running, the simulation system
automatically executes the sequence control program,uses SFC to initialize the initial value of the control
parameters required by the whole control system, andexecutes each control loop in turn, which is shown in
Figure 3-1.
Each control loop enters the automatic control stateaccording to the initial value, and after the
implementation of the END step, the system willautomatically enter the stable operation state, and theoperation switch guide operator mode is fully controlled,that is, the operator can complete the automatic control
according to the initial control target value, so that eachcontrol loop enters the automatic control state after theparameter initialization.

The PID controller is used to control the reactor. The PID
controller here uses the APL library of PCS7.
This topic has the feed part, the pressure part, the liquid
level part and the temperature part.
The feed part is realized by a relatively simple ratio
module, in which there are two modules, the mastermodule and the slave module. The former controls the flowrate of the main reactant, the flow rate of the auxiliary
reactant is controlled by the set value of the main module,
the ratio controller provides two PID controls, and thelatter controls the catalyst part of the reactant
independently.
The pressure part is controlled by a separate PID module.
To achieve this task, an initial setting value is set first. In
actual simulation, if the running value exceeds our initial
setting, the exhaust valve will be started; otherwise, if therunning value is insufficient, the inert gas intake valve will
be started until the process value each the set value. Then

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
112

Hongshuo Zhai

the feedback control of the whole system is carried out by
proportional integral and differential control method. In
the liquid level part, a separate PID module is used torealize it.
In the temperature part, cascade control is used. The
inner ring of cascade control adopts PID module tocontrol the flow rate of hot and cold water valves torealize the control of jacket temperature [3]. The output
MV of a PID module is adopted in the outer ring as thegiven value of the PID controller in the inner ring.
Figure 4-3 Pipes and functional blocks of the

4.1. Liquid level control scheme

liquid level control loop

For the control of liquid level in the loop, a negative
feedback closed-loop loop is used, as shown in Figure 41.

Figure 4-1 Pipes and devices
The value of the set value SP1 can be given by an
external or internal PID controller whose parameters are
tuned by the PID Tuner.
The liquid level control model, as shown in Figure 4-2
is built using the CFC block of PCS7.

Figure 4-4 Connection structure of each CFC block in
the liquid level control loop
In addition, for the feed flow, pressure, temperature
three parts, also to design the corresponding control
scheme. Among them, the feed part is controlled by ratio,
the pressure part is controlled by division, and thetemperature is controlled by cascade.

Figure 4-2 Liquid level simulation objects

4.2. Tuning of controller PID parameters
Using PID module to control the controlled object is very
important for the parameters, and only appropriate
parameters can meet our purpose. This topic only takes 4
control loops for example to study the liquid level control
loop.
The PID Tuner tool is developed based on PCS7
software, and the data is collected and recorded throughthe background data block of PID function block, and theideal data is obtained by simulation and applied in thecontroller. Different adjustment effects under various
control modes can be achieved by setting. The tool

For these four parts, it is usually used in a user-definedclosed-loop control system set by users, integrating the
input values according to the ladder rule, and output theresults.
Figure 4-3 shows the pipe diagram and function blockof the liquid level control loop. A control loop is provided
in the figure. The CFC block in Figure 4-4 is the liquid
level control loop pipe.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
113

Powered by TCPDF (www.tcpdf.org)

A Study of Chemical

provides synchronous real-time data curve display of
initial, median, and final values, and intuitive overall
optimization functions are displayed.
Below, after proper PID tuning, the appropriate tuning
pattern is obtained, which is shown in Figure 4-5.

[2]

[3]

System based on PCS7." Electric Drive 50.05 (2020): 102107. doi:10.19457/j.1001-2095.dqcd20821.
Jiangtao Zhang, et al. "Research Progress on Temperature
Control of Chemical Reactor." Chemical Automation andInstrumentation
49.04(2022):
419-423.
doi:10.20030/j.cnki.1000-3932.202204002.
Hengfei Wang, Yongliang Chen, et al. "the boiler
superheated steam outlet pressure cascade expert PID
control." automation instrument 36. 10 (2015) : 99-102.
The doi: 10.16086 / j.carol carroll nki issn10000380.201510025.

Authors Introduction
Mr. Hongshuo Zhai
He is a first-year master candidate
in Tianjin University of Science andTechnology.

Figure 4-5 Setting results of liquid level, flow rate,pressure, and temperature
References[1]

HongBin Fang, et al. "Simulation Design and
Implementation of Chemical Reaction Process Control©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
114

Analysis of Learning Quality Evaluation for University Student Courses with Process
Assessment
Yuhao Zhang1*, Ying Gong2, Xuran Wang3

College of Artificial Intelligence, Tianjin University of Science and Technology, 300222, China

12

College of Economic and Management, Tianjin University of Science and Technology, 300222, China

College of Chemical Engineering and Materials, Tianjin University of Science and Technology, 300222，
China

3

E-mail: *a15839176622@163.com
www.tust.edu.cn

Abstract
With the progressive exploration and application of formative assessment in university pedagogy, this evaluative
method has become widely adopted for appraising students' everyday learning attitudes and conditions. Drawing
upon pertinent data regarding classroom learning experiences of students at a specific university, this paper
employs machine learning, K-means clustering, the TOPSIS evaluation model, and the entropy weighting method
to investigate the relationship between formative assessment and the quality of university student learning,
culminating in the creation of an evaluation model. This model allows us to pinpoint the key factors influencing
student learning attitudes and offers support for formative assessment in the university context.
Keywords: University Process Assessment, attitude toward learning, machine learning, TOPSIS Evaluation model

1. Introduction

introduces the construction of evaluation model. The
fourth part analyzes the evaluation model and the results
obtained. The fifth part summarizes the main content of
this paper.

With the continuous deepening reform of education andteaching mechanism, the educational method of processassessment has been widely explored and applied bymore and more universities. Compared with the outcome
assessment, the process assessment can better evaluate
the students' learning attitude and state [1].
In the process assessment, students' learning attitude ismainly affected by the objective environment andsubjective initiative [2]. This paper will explore which
factors have a greater impact on students' learning
attitude, build a mathematical model that can reasonably
evaluate the school's learning style and class style, andbuild an evaluation model that can reflect the level ofstudents' learning enthusiasm.
The rest of this article is organized as follows. The
second part analyzes the data and explores the factors thatreflect students' learning attitude. The third part

2. Data Analysis
The data collected students' student number, college,class, course code, class code, teacher code, test name,test time, test full score, student score and normalized
score. It should be noted that students choose different
courses within the scope of core basic courses, somestudents choose multiple courses, some students choose
only one course; Some students may have done work thatwas not posted by the class teacher; Some students don'ttake every test for some reason; Teachers in the same
course don't publish pre-class tests the same often.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
115

Yuhao Zhang, Ying Gong, Xuran Wang

In order to explore which factors have a greater impact
on students' learning attitude, it is necessary to explore
the control variables. The following will explore theinfluence of college factors, the number of courses
selected by students, the attendance rate of students, andthe factors of major, class, course, teacher and classroom
on students' learning attitude.
2.1. College factor
By calculating the average scores of students in different
colleges of the same course, and then summarizing allcolleges, the average and variance of the scores ofstudents in each college are obtained. The data are shownin Fig. 1.

Fig. 2 figure of the number of selected courses
As can be seen from Fig. 2, with the increase in thenumber of courses selected by students, the averagescores of students are basically the same, and the overall
scores of students are higher and more and more stable. It
can be inferred that the more courses a student chooses,
the higher the student's enthusiasm for the course and thebetter the attitude.
2.3. Student attendance
When the total number of tests for each course is taken as
the number of the last measurement for this course, that is,
the total number of tests organized by the teacher for thiscourse, the attendance rate of each student is obtained.
The scatter plot corresponding to the attendance rate ofstudents and the representative score is shown in Fig. 3.

Fig. 1 College factor visualization
The average score of students in different colleges isdifferent, and the stability of grades is also different.
Therefore, the learning atmosphere of different colleges
is different, and students' learning attitude is also
different. Therefore, students' learning attitude is affectedby the factor of college.
2.2. The number of courses taken by students
We define the average of each student's normalized
scores on several tests as the student's representativescore. The number of courses selected by each student is
1, 2, and 1 of 3. The representative scores of students
with 1, 2, and 3 courses selected respectively arecalculated, and the corresponding scatter map is shown in
Fig. 2.

Fig. 3 Scatter plot of attendanceCount the number of people with scores greater than
80 in each attendance range and draw a bar chart, whichis shown in Fig. 4.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
116

Analysis of Learning Quality

3.1. Data standardization
Since attendance, grade point average, and average
number of courses taken are all positive indicators, it is
only necessary to standardize the dataset consisting ofcollege (or class), average test score, average test
participation rate, and number of courses taken.
3.2. Calculate index weight
To ensure the objectivity and scientificity of the index
weights, entropy weight method and analytic hierarchy
process are used to determine the index weights, and theweights are shown in Fig. 5.

Fig. 4 attendance and high score statistics chart
As can be seen from Fig. 3 and Fig. 4, when students'
attendance rate is small (0-0.4), students' high scores arecompared. With the increase of students' attendance rate,
students' scores are generally higher, and with theincrease of students' attendance rate, the number ofstudents who get high scores is increasing. Therefore,
students’ attendance rate has a great impact on students'
scores and can better reflect students' learning attitude.

Fig. 5 index weight
3.3. Calculating the score

2.4. Major, class, curriculum, teacher factor

Define the maximum value 𝑍 + as the maximum value ofeach column in the vector matrix and define theminimum value 𝑍 − as the minimum value of eachcolumn in the vector matrix. Define the distance of the𝑖 (𝑖 = 1,2, … , 𝑛)evaluation object to the maximum valueas:Calculate the average grade and variance of grades foreach major, each class, each course, and each classroom,
as well as the average grade and variance of studentstaught by each teacher.
After analysis, the following conclusions are reached:(1) Students' learning attitude is influenced by theirmajor, but the influence is small. (2) On the whole,
students' grades and the stability of grades are also
different depending on the class they belong to. Therefore,
students' learning attitude is affected by class factors. (3)The overall performance of students taught by someteachers is poor and unstable; The rest of the teacherstaught students different overall grades, but thefluctuation is not large, can reflect the true level.
Therefore, students' learning attitudes are influenced bytheir teachers and courses.

2+
𝐷𝑖+ = √∑𝑚
𝑗=1 ω𝑗 (𝑍𝑗 − 𝑧𝑖𝑗 )(1)

Define the distance of the 𝑖 (𝑖 = 1,2, … , 𝑛) evaluation
object to the minimum value as:2−
𝐷𝑖− = √∑𝑚
𝑗=1 ω𝑗 (𝑍𝑗 − 𝑧𝑖𝑗 )(2)Then, we can calculate the non-normalized score of the𝑖 (𝑖 = 1,2, … , 𝑛) evaluation object as:

𝑆𝑖 =

𝐷𝑖−
+
𝐷𝑖 +𝐷𝑖−(3)

It is evident that 0 ≤ 𝑆𝑖 ≤ 1 , and the larger 𝑆𝑖 thesmaller 𝐷𝑖+ , indicating closer proximity to the maximum
value. By incorporating the obtained indicator weights
into the calculation of sample distances, we obtain thescore for each sample [3].

3. Construction of Evaluation Model
Data for all students in each college and class isaggregated to get an average of average test scores,
average test participation rates, and number of courses
taken by students in each college and class. This will
result in a dataset consisting of the college (or class),
average test score, average test participation rate, andnumber of courses taken. Since there are few indicators,
TOPSIS method is adopted here [3].

4. Introduction of Result
According to the above model construction, MATLAB isused to calculate, and the final answer is obtained.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
117

Powered by TCPDF (www.tcpdf.org)

Yuhao Zhang, Ying Gong, Xuran Wang

4.1. Evaluation hierarchy

Authors Introduction

The hierarchy of evaluation models is shown in Fig. 6.
Mr. Yuhao Zhang
He is an undergraduate majoring in
artificial intelligence at Tianjin
Universityof
ScienceandTechnology. He learns about
electronic information and artificial
intelligence, and currently focuses on
digital image processing, deeplearning and pose estimation.

Fig. 6 evaluation hierarchy diagram
Ms. Ying Gong

4.2. Personal evaluation model

She is currently a third-year student
majoring in financial engineering at
the School of Economics andManagement of Tianjin University of
Science and Technology, focusing ondata mining.

Here, we want to find the best and worst people, and we
first cluster them using K-Means into better and worsegroups. Then, the two groups were evaluated by TOPSIS,
and the evaluation index was the student attendance rate,
the number of courses selected, and the individual
performance [2].
5. Conclusion

Mr. Jiashuai Wang
He is currently studying Computerscience and Technology at Tianjin
Universityof
ScienceandTechnology, where his researchinterests are deep learning andcomputer vision.

Through the students' usual performance to construct theprocess evaluation model, get the factors that affectstudents' learning attitude, and can find the class, college,
and students with better performance through the data.
This research will help colleges and universities tocarry out process assessment better.
References
1.

2.

3.

Li Q, Li Z, Han J, et al., Quality Assurance for Performing
Arts Education: A Multi-Dimensional Analysis Approach,Applied Sciences, 2022, 12(10): 4813.
Zhang Rui, Xu Peng, Fang Mingfeng, a process-based anddiversified curriculum assessment method for local
comprehensive universities, Journal of Chongqing
University of Technology (Social Sciences), 2014, 28(04):129-133.
Yu Xiaofen, Fu Dai, A review of multi-index
comprehensive evaluation methods, Statistics and Decision,
2004(11): 119-121.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
118

Motion Analysis and Transfer Applications Based on Posture Recognition
Yuhao Zhang*, Mingyue Li, Jianhao Jiao

College of Artificial Intelligence, Tianjin University of Science and Technology, 300222, China
E-mail: * a15839176622@163.com
www.tust.edu.cn

Abstract
This paper investigates the posture trajectories in human motion using pose recognition technology based on the
deep learning framework MediaPipe. By detecting key points on the human body and plotting and calculating these
trajectories in the temporal dimension, we further conduct comparative analysis of these trajectories withprofessional sports coaches' motion guidance to assist athletes in correcting their posture. Additionally, thistechnology has been deployed on Jetson Nano, enabling its practical application in mobile scenarios, and providing
robust tools and methods for fields such as rehabilitation therapy, sports training, and animal behavior analysis.
This study offers insights into the transfer applications of posture recognition.
Keywords: deep learning, MediaPipe, posture recognition, Jetson Nano

1. Introduction

outdoors or indoors [2]. In addition, we added the camera
to the steering gear, so that the camera can follow the
movement in real time and accurately analyze the attitude
data.
In addition, the continuous actions in the same periodof time can be transferred to the movement of animals,
such as octopus, snake and other animals, through deeplearning to train a model to identify key points, and thenanalyze the movement behavior of animals, which will
help us better understand animals [3].
The rest of this article is organized as follows. The
second part introduces the hardware. The third part
introduces the software part. The fourth part analyzes thecaptured motion changes. The fifth part summarizes themain content of this paper.

With the continuous development of artificial intelligence,
artificial intelligence technology has played a huge role inthe development of the sports and fitness industry.
Moreover, with the continuous development of economy
and society and the continuous improvement of people's
living standards, people are more and more willing tostrengthen their health through physical exercise. The
application of artificial intelligence technology in sports
will inject vitality into the sports industry.
People in the usual self-training, because there is no
manual coach guidance, which will make the athletes can
not get timely feedback when improving skills. Throughthe use of the MediaPipe framework developed by Goole
company to realize the identification of human bone key
points, through the changes of bone points in continuous
time to draw the human movement trajectory, and thenanalyze the movement Angle of each action [1]. This will
provide a wealth of data for athletes to continuously
optimize their level.
The posture analysis of athletes has been deployed on
the Jetson Nano, so that athletes can be monitored bymoving the jetson nano, whether they are exercising

2. The Hardware Structure
The main hardware used in this research is jetson nano,steering gear and high-definition camera. The camera is
moved through the steering gear to better capture people's
posture information, and the powerful image processing
capability of jetson nano is used for data analysis.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
119

Yuhao Zhang, Mingyue Li, Jianhao Jiao

2.1. Jetson nano2.3. DS3230 digital steering gear

Jetson Nano is an embedded computing platform basedon NVIDIA Maxwell architecture with a 128-core
NVIDIA Maxwell GPU and quad-core ARM Cortex-A57
CPU that delivers high-performance deep learning andcomputer vision capabilities. The board supports a varietyof peripheral connections, including GPIO pins, I2C
buses, UART serial ports, and more, making it an ideal
platform for integrating a wide range of sensors andactuators [2]. Jetson Nano runs Ubuntu Linux with strong
software support for deep learning and computer visionapplications. The jetson nano used in this study is shownin Fig. 1.

Two DS3230 digital steering engines are used in thisstudy. This steering gear provides precise Angle control,
enabling the system to dynamically adjust the direction of
the camera according to the movement of the target
object, achieving a 360-degree rotation. Through thesteering engine, we can realize the full range of the target
object trajectory capture. The DS3230 digital steering
gear used in this study is shown in Fig. 3.

Fig. 3 DS3230 digital steering gear
3. Software Distribution
In software development, this design is based onMediaPipe open-source framework development, andcombined with OpenCV to achieve the complete code.
Fig. 1 Jetson Nano
3.1. Introduction to MediaPipe
2.2. IMX415 HD camera

MediaPipe is an open-source computer vision librarydeveloped by Google to provide developers with aflexible and efficient set of tools to build visually aware
applications. It is designed to simplify the development
process of computer vision tasks by providing pre-trainedmodels and easy-to-use Apis that make it easier for
developers to implement a variety of visual tasks. The
main performance is as follows:(1) Real-time performance: MediaPipe focuses on
real-time computing, enabling low latency and efficientreal-time image processing on multiple hardware
platforms.
(2) Multimodal input: Support a variety of input
sources, including images, videos, camera streams, etc.,
making it suitable for a variety of scenes and devices.
(3) Rich pre-trained models: MediaPipe offers a rangeof pre-trained models covering many areas such as
human pose estimation, hand tracking, face detection, etc.,
which can be used to quickly build various computer
vision applications.

Sony's IMX415 HD camera is our choice. The camerafeatures 8.46M effective pixels, a frame rate of up to
120fps, and a field of view Angle of 130 degrees,
providing the system with clear, high-resolution images.
To provide the system with clear, high-resolution images.
The IMX415 is an advanced image sensor for
applications that require high-quality image capture, suchas our attitude recognition system. The IMX415 HD
camera used in this study is shown in Fig. 2.

Fig. 2 IMX415 HD camera©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
120

Motion Analysis and Transfer

4.1. Effect of display(4) Cross-platform support: MediaPipe supports
multiple operating systems (including Linux, Windows,
Android, iOS) and a variety of hardware accelerators(such as Gpus, Tpus), making it widely portable.
In this design, we chose the MediaPipe framework as
the basis to realize the key function of attitude
recognition. By combining OpenCV, we were able toflexibly process camera input and use the pre-trainedmodel provided by MediaPipe to accurately capture key
pose information of the target object. This combination
allows us to quickly build a powerful computer visionsystem for monitoring and analyzing the movements oftarget objects.

As shown in Fig. 5 below, the trajectory of the key pointsof the arm in a period of time can be drawn, so that the
movement posture of the professional athlete can becompared, and the change of the Angle of the joint can be
accurately viewed to help the athlete find the correct
movement posture.

3.2. Establishment of trajectory analysis model
Let's take the motion analysis of one arm as an example.
Using
the
MediaPipeframework,
call
theposeLandmarkerResult function to generate thethree-dimensional coordinates of the key points. In Fig. 4
below, A, B, and C represent the three nodes of the arm.
The poseLandmarkerResult function can be used toreturn the three-dimensional coordinates of the node, andcan draw the trajectory motion [1].

Fig.5 effect display diagram
Through the object detection algorithm, the camera cancapture objects such as tennis balls and ping-pong balls,
which can better identify the position of people [2]. The
display effect is shown in Fig. 6.

Fig. 6 object identification

Fig. 4 Keypoint graph4.2. Idea transfer application
Then according to the following formula, the angulardistance change of the node relative to the previous state
can be calculated.
⃗⃗⃗⃗⃗
⃗⃗⃗⃗⃗
𝐴𝐵 ∗𝐵𝐶𝜃 = 𝑎𝑟𝑐 𝑐𝑜𝑠 ( ⃗⃗⃗⃗⃗⃗⃗⃗⃗⃗ |
|𝐴𝐵 |⋅|𝐵𝐶)

It has far-reaching and multi-level significance to transfer
human posture recognition technology to animals to
realize key point detection and trajectory tracking. Thisnot only provides new research avenues in the field of
biology, deepening the understanding of animal ecology,
behavior, and physiology, but also brings substantial
contributions to wildlife conservation, livestock
management and scientific education. Through theapplication of this technology, we can monitor the
movement patterns, posture changes and behavior
patterns of animals in real time, providing a scientific(1)

4. Introduction of Result
Running the code on the jetson nano shows the trajectoryon the screen and can calculate the Angle between thenodes.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
121

Powered by TCPDF (www.tcpdf.org)

Yuhao Zhang, Mingyue Li, Jianhao Jiao

Ms. Mingyue Libasis for protecting endangered species, monitoring
wildlife migration, and improving the efficiency oflivestock production [3].

She is an undergraduate majoring in
artificial
intelligenceatTianjin
University of Science and Technology.
She currently focuses on digital image
processing and deep learning.

5. Conclusion
Through the movement posture analysis of human arm as
an example, the trajectory of the athlete is successfully
analyzed, which also reflects the key points of the whole
body, and the movement posture analysis will help theathlete to correct the posture in a more three-dimensional
way. And this model is deployed on the jetson nano,which will facilitate device movement. The transfer of
posture analysis to animal movement behavior will
greatly help the study of animals.

Mr. Jianhao Jiao
He is an undergraduate majoring in
artificial
intelligenceatTianjin
University of Science and Technology.
He learns about electronic information,and currently focuses on deep learningand pose estimation.

Acknowledgements
This paper is supported by the project "Fire Hero - Fire
rescue intelligent robot" of Tianjin University of Scienceand Technology, with the project number of202310057042.
References
1.
2.

3.

Han K, Li X, Research Method of Discontinuous-Gait
Image Recognition Based on Human Skeleton Keypoint
Extraction, Sensors, 2023, 23(16):7274.
Zhang Shang, WANG Hengtao, RAN Xiukang,Lightweight traffic sign detection method based on
YOLOv5, Electronic Measurement Technology, 2002,
45(08): 129-135.
Li Yandong, HAO Zongbo, Lei Hang, Convolutional
neural networks research review, Journal of Computer
Applications, 2016, 36(09): 2508-2515+2565.

Authors Introduction
Mr. Yuhao Zhang
He is an undergraduate majoring in
artificial
intelligenceatTianjin
University of Science and Technology.
He learns about electronic information
and artificial intelligence, and currently
focuses on digital image processing,
deep learning and pose estimation.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
122

Functional Safety Assessment of the Safety Protection System Based on Petri Net
Peng Wang*, Mengyuan Hu

College of Electronic Information and Automation, Tianjin University of Science and Technology,
300222, China
E-mail: *1090465299@qq.com
www.tust.edu.cn

Abstract
In this paper, the functional safety evaluation of the safety protection system of gasoline hydrogenation unit was
carried out using Petri net. Firstly, the principle and framework of the gasoline hydrogen refueling unit was described.
Secondly, the safety integrity level was introduced, and the influencing factors of the safety integrity level weresummarized. Thirdly, the Petri net model and the Markov model are compared and the Petri net model is used toverify its security integrity level. Finally, the calculation result demonstrated that the SIL did not reach the target
level, and then reached the target level after improvement. This analysis method can provide reference for the safety
integrity level evaluation of similar devices.
Keywords: Gasoline hydrogenation, Petri net, Functional safety assessment

gasoline hydrogenation unit based on Petri network is a
topic worth discussing.
The rest of this article is organized as follows. The second
section introduces the research status at home and abroad
about the gasoline hydrogenation technology and thesafety analysis. In the third part, the gasolinehydrogenation unit is presented. In the fourth section, aexample is given to analysis the risk of facility. In the fifth
part verifies and improves the safety integrity level ofgasoline hydrogenation unit. The sixth part summarizes
the main content of this paper.

1. Introduction
With the passage of time, modern science and technology
continue to develop, the number of vehicles climbed. The
sulfides produced by automobile exhaust have a great
impact on air quality. Therefore, the production of clean
gasoline is very important for environmental protection. In
response to this problem, we have proposed thehydrogenation of gasoline. The purpose of gasolinehydrogenation is to desulfurize at a small octane loss.
Gasoline hydrogenation equipment is a device used forfilling the car fuel tank, which includes liquid level sensors,
flowmeters, valves and other equipment. Because it isusually operated in high temperature, high pressure andhydrogen environment, there are safety risks, which
requires us to do a good job of safety protection, equippedwith safety instrument system. Safety instrument system iscomposed of sensor logic controller and actuator, it canperform one or more absolute safety control instrument
functions. Petri networks, which can intuitively describe
the relationship between system states and events, arewidely applied in many fields. Therefore, the functionalsafety assessment of the safety protection system of the

2. Research Status at Home and Abroad
2.1. Research status
technology

ofgasolinehydrogenation

At present, most of the catalytic raw materials selected by
the manufacturers in China have the problem of excessive
metal content, as well as excessive sulfur content and nonhydrocarbon components. These problems will not only
cause the decline of product quality in the gasoline
production process, but also cause the problem ofenvironmental pollution in view of the existence of many

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
123

Peng Wang, Mengyuan Hu

sulfur and other elements in the production process. In the
production process, using diolefin saturation treatment caneffectively reduce the occurrence of olefin coking
phenomenon; in addition, in the relatively mild reaction
environment, the influence of impurities can be further
reduced, so as to reduce the octane value factors and losses,
and better meet the relevant national technical standards
and production requirements.

carried out by means of the hydrodesulfurization reaction.
The heating is stopped after the temperature is repeatedly
raised to a suitable temperature. Next, the diene to olefin
reaction is carried out to meet the requirements. In order toensure the smooth progress of the whole reaction process,we introduce the catalyst under suitable catalytic
conditions for the reaction. After the reaction is completed,
we will use a tandem selective hydrogenation reactor to
reprocess the product. Through the hydrogenation static
desulfurization treatment, and the selective hydrogenation
reaction redox reaction, the heat exchange is finallyrealized through the heat exchanger, and then after thetreatment of the catalyst, the saturated state of the olefin
product is successfully obtained.
During this process, the reactant is reinjected into theheat exchanger for heat exchange. When it reaches acertain extent, the subsequent reaction of the reactants canbe continued. By adjusting the temperature and theinteraction of the catalyst, they enter the second reactor fordesulfurization treatment, and the catalyst can efficiently
achieve the requirement of complete desulfurization. The
product of hydrodesulfurization reaction enters the heatseparation tank after heat exchange, and the gas phase isadded to the product cold separation tank and circulating
cleaning equipment after air cooling and cooling of thedesulfurization reaction. After the treatment ofdesulfurization and hydrogenation reaction, the obtained
product will be transported to the separation tank after gas
phase cooling. In the separation tank, part of the oil phasematerial will be separated. After it is stable, the noncondensable gas will be transported to the liquid separation
tank for further treatment. The separated liquid isdischarged after condensation treatment through thecondenser, and some of the gas is sent to the gas-liquid
separator to mix with water vapor and return to the heatexchanger to continue heating. Then, the product afterpressure stabilization treatment is transported to theabsorption tower, and then processed again by the liquid
separation equipment of the circulating hydrogencompressor. After the mixed hydrogen treatment, it meetsthe needs of desulfurization production.

2.2. Research status of safety analysis
The gasoline hydrogenation equipment of a chemical plantin China uses a set of independent safety instrument
system (SIS) for safety control. Its process is complex, andthe operating environment is high temperature and highpressure, involving flammable and explosive gases, which
requires close monitoring. Since its launch, the applicationof SIS system has significantly improved the stability andreliability of the equipment, realized safe, stable andefficient operation, avoided the occurrence of equipment
shutdown and failure problems, at the same time brought
economic benefits to the enterprise, but also guaranteed
the personal safety of workers.
2.3. Application of petri network in system security
analysis
Petri was first proposed by Kal Adam Peri. Petri network
represents the system model in the form of a meshstructure model. Petri network is a visual mathematicalmodeling tool that contains elements such as place,transition, arc and token, which can represent the static
function and structure of the system. In addition, Petri
networks combine data flow, control flow, and various
logical relationships to support a more rich and complex
system modeling, allowing for a better analysis of the
dynamic behavior of a specific system.
3. Analysis of the Gasoline Hydrogenation Unit
The raw oil is introduced into the filter through an external
device, and the purpose of the overall filtration is toseparate the particles in the raw material. Among them,after the large particles are removed, they enter theselective hydrogenation feed equipment. The feed can beused as a way to select the hydrogenation reactor feedpump to control the flow rate to achieve a reasonable effect,
and then use the supplementary hydrogen mixing mode toform a mixed feed. First, the mixture is sent to the selectivehydrogenation reaction device, and the heat exchange is

4. Device Risk Analysis
4.1. Confirmation of the SIL level
Safety Integrity Level is a measure of the safety of safety
instrument system and enterprise safety instrument systemmanagement level, its value represents the order of risk

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
124

Functional Safety Assessment of

the discontinuation of SIF. Under certain time andconditions, the possibility of safety-related systems
performing their prescribed safety functions can be calledsafety integrity level (SIL), which represents the level of
reducing the risk of safety instrumented systems. It is very
important to select the appropriate SIL level. If theselection is too high, it will cause cost waste, and if it is
too low, it will bring unacceptable risks. According to theIEC61508 standard, SIL4 level is the highest and SIL1
level is the lowest.

reduction level. The three foundations for determining thelevel of security integrity are: structural constraints,
system capabilities, and hardware security integrity. The
safety integrity of hardware mainly depends on thereliability of hardware in the case of dangerous failure. The
influence of these factors mainly includes the failure
model, system structure, functional test cycle andcomponent reliability level. In the IEC61508 standard, thesafety integrity level is divided into four levels, and theoperation mode of the system is divided into low
requirement operation mode and high requirement
operation mode. In the low requirement operation mode,
the SIL level is based on the average failure probability,
while in the high requirement or continuous operationmode, the SIL level is based on the average failure
probability per hour. The requirements for SIL levels fordifferent operating modes, which are shown in Table 1andTable 2.

4.2. Influencing factors of safety integrity level⚫
Failure mode⚫
Redundant structure⚫
Common cause failure⚫
Diagnostic coverage rate⚫
Periodic function test⚫
Maintenance mode
4.3. Safety integrity level verification
Hardware security integrity is a part of the overall security
integrity of SIS, and its influencing factors include failuremode, component failure rate and detection cycle. Whenevaluating an interlocking circuit, it is necessary to
determine its failure probability to determine its SIL level.
Finally, the SIL verification results are compared with thegrading analysis results to verify whether the circuit meetsthe requirements. In order to meet the requirements of SIL
grading analysis, it is necessary to reset the layout structure,component selection, redundant design and maintenanceplan of the interlocking loop. IEC standard provides avariety of methods to verify SIL level, including simple
formula method, reliability block diagram method, fault
tree analysis method and Markov model method.
For the transition of the system state, the Markov modelis represented by a circle and an arrow, which represent thestate of the system and the transition between states,
respectively. The Markov model of the 1oo1 structure isshown in Fig. 1.

Table 1 The SIL level for the low-requirement operationmode
Low-required operation mode
SafetyAverage
Risk reduction
integrity levelprobability offactorfailure on
demand（SIL）（PFDavg）（RRF）

10-4~10-5
10-3~10-4
10-2~10-3
10-1~10-2432110000~100000
1000~10000
100~1000
10~100

Table 2 The SIL level for the high requirement operation
mode
The SIL level for the high requirement operation mode
Safety Integrity Level
Meverage hourly failure
probability（SIL）
PFHavg

4
3
2
1

-8

FS
1
λSD+λSUͷSD-9

10 ~1010-7~10-8
10-6~10-7
10-5~10-6

λDD

OK
0

FDD
2

ͷ0
λDU

The SIS system is more focused on monitoring whether
there are risk conditions in the production process andreducing the possibility of risk occurrence. The scheme isa passive system, which is generally in a non-dynamic
state. Only when necessary, it will play a role and will not
actively participate in the normal operation of the basic
process control system. The safety life cycle covers theentire period from the conceptual design of the project to

FDU
3

Fig.1. 1oo1 redundant structure Markov model
The state transition matrix of 1oo1 redundant structure
is P [1].

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
125

Peng Wang, Mengyuan Hu

1（-  s +  D）  SD +  SU  DD

1− 
0
 SD
SD
P=

0
1− 
0
0

0
0
0


 
DU

0 
0 

1 

5. Verification of safety integrity level of gasolinehydrogenation unit

(1)

In this section, SIL is evaluated for one of the dangerous
scenarios SIF5 of the gasoline hydrogenation unit. The
main fuel gas of the fractionator reboiler F9101 of SIF5 isselected to set low pressure and low PSLL6049 A / B / C(2oo3) interlock shutdown XV6011 as a representative. In
this process, the components to be considered are as
follows.
⚫
SIF serial number:SIF5⚫
Accident scenario：The pressure of F9201 fuel gas in
the hydrodesulfurization heating furnace is too low,which leads to the extinction of the main nozzle. If
the fuel gas continues to enter the furnace, the furnace
explosion accident may occur.
⚫
Consequence description:The pressure of F9201 fuelgas in the hydrodesulfurization heating furnace is too
low, which leads to the extinction of the main nozzle.
If the fuel gas continues to enter the furnace, thefurnace explosion accident may occur.
⚫
SIF function description:F9101 main fuel gas settingpressure is low PSLL6049A / B / C (2oo3) interlock
closed XV6011.
⚫
SIL target: SIL2
In this process, the components to be considered include
as shown in Table 3.

To calculate the mean time between failures, you can
use the system state transition diagram, the system statetransition diagram is shown in Fig. 2.

Fig.2 1oo1 structure Markov model (MTTFS)
The corresponding matrix is: + )

Q = 1 − (And because:

SD



(2)

SU



N = I −Q

−1

(3)

Therefore:N=1
 SD +  SU

(4)
The sum of the elements of the N rows of the matrix isthe result of MTTFS:

MTTF =

1
 SD +  SU

(5)
The formula of system dangerous failure probability is:

PFS = S 0 P

8760

0 1 0 0

T

Table 3 Components to be considered for SIF5

(6)

The formula of system safety failure probability is:

PFD = S 0 P

8760

0 0 1 1

PSLL6049A / B / C (2oo3)T

Sensor Part Pressure Sensor(7)
Markov model is a model that can represent the dynamic
behavior of the system. It can consider multiple reliability
factors, and multiple reliability indexes can be obtained by
one modeling. However, it will face problems such as
space explosion. Especially when the complexity of thesystem increases, the difficulty of this method will
increase greatly.
The stochastic Petri net is used to analyze the hardware
failure probability, which solves the problem of Markov
state explosion, reveals the dynamic characteristics of thesystem, and can evaluate the hardware safety integrity ofthe safety instrumented system.

Safety barrier (1oo2)
Logic controller part

PLC（2oo3）
Relay 1 (1oo1)

Actuator

Emergency Shut-off Valve
XV6011 (1oo1)
Relay 2 (1oo1)

In order to evaluate random hardware failures, complete
failure data of systems and components are required,considering functional test cycles and common cause
failures. These factors will be included in the evaluation
category in order to more accurately determine the random
hardware failure probability of the system, and thenevaluate its hardware security integrity level. The failure©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
126

Functional Safety Assessment ofprobability of the system needs to meet the system
requirements.
PFDavg = PFDavg sensor + PFDavg PLC + PFDavg actuator (8)The relay 1, relay 2, emergency cut-off valve andsolenoid valve are all 1oo1 redundant structure, while thesafety gate is 1oo2 redundant structure to ensure thestability and reliability of the system. Based on the Petrinet model, the average failure probability of theemergency shut-off valve is calculated.
The state transition matrix Q is :− 7.48 10 −6

5.71 10 −5
Q= 0.04167
 0.1252.34 10 −6− 5.71 10 −54.73 10 −6− 0.04167

4.13 10 −6 
− 0.125 (11)Equation solving:(9)

𝑋𝑄 = 0Equation solving:{  Xi = 1,1  i  n

XQ = 0{ Xi = 1,1  i  n

(10)

(12)

i

i

Solution:

Solution:

PFDavg Cut-off valve=1.98×10-2
PFDavg Relay 1=2.455×10-6
PFDavg Solenoid valve=7.454×10-4
PFDavg Safety barrier =7.902×10-4
PFDavg Relay 2= 2.5164×10-6
PLC,the system sensor is a 2oo3 redundant structure,and the reachability diagram of the 2oo3 structure Petri net
is shown in Fig. 3.

PFD avgPLC =1.21×10-6
PFD avg sensor= 2.3×10-4
Then PFDSIS = PFDavg cut-off valve + PFDavg relay1 + PFDavgsolenoid valve + PFDavg relay2 + PFDavg safety barrier + PFD avg PLC +PFD avg sensor = 2.15×10-2, which does not meet the SIL2
requirements and needs to be adjusted. It can be seen from
Table 3-10 that the shut-off valve has the greatest influenceon the total PFD of the equipment is the shut-off
valve,which is shown in Table 4.
Table 4 The contribution of subsystem to system PFD
Systematic nameCut-off valve
Solenoid valve
Pressure sensor

Fig. 3 2oo3 structure reachable graph

Safety barrier

The state matrix is:

Proportion of impact on PFD
90%
3%
4%
3%

Therefore, we can take the transformation of theemergency shut-off valve and increase the emergency
shut-off valve group to realize the transformation from the
1oo1 structure to the 1oo2 redundant structure. After thetransformation, the PFDavg shut-off valve' = 4.36×10-3. After thetransformation, PFDSIS = PFDavg shut-off valve' + PFDavg relay1 +PFDavg solenoid valve + PFDavg relay2 + PFDavg safety barrier + PFDavg©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
127

Powered by TCPDF (www.tcpdf.org)

Peng Wang, Mengyuan Hu
PLC + PFDavg sensor = 6.13×10-3, which meets the SIL2 safety
Authors Introduction

requirements.
Ms. Peng Wang

6. Conclusion

She is a postgraduate tutor of Tianjin
University of Science and Technology.
In 2014, she received a doctorate from
North
China
Electric
Power
University. The research direction is
the functional safety assessment ofsafety instrumented systems.

In recent years, China has paid more and more attention to
production safety, and the government has also issued
relevant policies and regulations for industries with highrisks. In this context, the automation level of China 's
petrochemical enterprises has been continuously improved,
which provides good conditions for ensuring production
safety. Driven by the policy, with the superposition of new
facilities and demand and the increase in the demand forfacilities upgrading, SIS system will be more widely usedand play a vital role in today 's society and the country. The
main conclusion of this paper is that the security level ofthe system depends on many factors, among which
structural constraints, system capability and hardware
security integrity level are crucial factors. In order toensure the safe operation of the system, it is necessary to
make reasonable analysis and judgment on these factors.
Explore a variety of factors that affect the functional safetyof safety instrumented systems, and use Petri net models
to model and optimize them, so as to improve the level ofsafety instrumented systems.

Ms. Mengyuan Hu
In 2023, she received her Bachelor ofEngineering degree from the School ofElectronic
Information
andAutomation, Tianjin University of
Science and Technology, China.She ispursuing a master's degree inengineering from Tianjin University of
Science and Technology.

References
1. Guo H, Yang X. Automatic creation of Markov models forreliability assessment of safety instrumented systems.
Reliability Engineering & System Safety, 2008, 93(6):829837.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
128

Pedestrian Attribute Recognition Based on Deep Learning
Peng Wang*, Qikun Wang, Shengfeng Wang

College of Electronic Information and Automation, Tianjin University of Science and Technology,
300222, China
E-mail: *2324365941@qq.com
www.tust.edu.cn

Abstract

This paper studied pedestrian attribute recognition based on deep learning, for its importance in the fieldsof smart city construction. Firstly, the research status of pedestrian attribute recognition and common deeplearning models was introduced. Secondly, considering the accuracy decline problem and gradientproblem of the neural network, the residual network was used as the main body of the neural networkmodel. Thirdly, the model was trained to classify multiple person attributes through two data sets, Market1501 and DukeMTMC-reID. Finally, the pedestrian attribute recognition model was tested, and goodresults were obtained.
Keywords: Neural networks, Pedestrian attributes, ResNet50

1. Introduction
2. Methods and results of pedestrian attributerecognition

The contribution of attribute recognition technology inmedical, security, intelligent furniture and other fields has
attracted more and more attention. After the introductionof deep learning algorithms in the computer field, thecomputer has realized the processing and application ofmassive information through continuous learning. The
person attribute recognition based on deep learning
extracts the feature information of a known pedestrian
photo through the convolution and pooling network model,
and classifies it to obtain several attributes about theperson. The acquisition of these attributes brings importantpractical applications to the fields of smart city
construction and military security.
In this paper, we first introduce the research backgroundand current research status of pedestrian attributerecognition. In the second chapter, the theoretical basis ofdeep learning will be introduced, and the attributerecognition of pedestrians will be mainly studied andverified by relevant experiments. In the third chapter, thefull text is summarized, discussed and analyzed.

In the field of computer vision, deep learning has becomea basic tool. In this study, a large number of deep learning
concepts will be used, and deep learning neural networks
will be used to identify pedestrian attributes. Finally,
results will be obtained in experiments..
2.1. The emergence of artificial neural networks
T Landahl et al. first proposed artificial neural network,
which is a network model built by imitating the connectionof nerve cells in the nervous system of animals. As shownin Fig. 1, each circle represents neurons and arrows
represent the direction of signal transmission, which issimilar to the synapses in nerve cells to transmit signals.

Fig. 1 Artificial neural network structure diagram©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
129

Peng Wang，Qikun Wang, Shengfeng Wang

As the basic part of the neural network model, thecalculation formula of neurons is as follows:𝑦 = 𝑓𝑎𝑐𝑡𝑖𝑣𝑒 (∑𝑛𝑖=1 𝑤𝑖 ∗ 𝑥𝑖 + 𝑏)
(1)
Where, xi represents the ith input of the neuron, wi is theweight parameter of the input, which can be changed, nrepresents a total of n neuron inputs, and b is the bias value.
Here is called the activation function, a neuron multiplies
the input with a weight and then adds a bias value, which
is output to the next layer through the activation function.
In the deep learning network, the weight value and bias
value are the parameters that need to be learned. Throughcontinuous learning, the gap between the output value andthe real value becomes smaller and smaller.

Fig. 2. Schematic diagram of convolution operation of 3*3
convolution kernel

2.3. Principles and advantages of ResNet network
With the more and more extensive application of deeplearning in computer vision, the depth of neural network isalso deepening. However, it is found that with the
deepening of network depth, the results obtained are notgetting better and better, and the problem of gradientdisappearance is becoming more and more serious.
Therefore, He et al proposed residual convolutional neuralnetwork. The residual block structure is added to the
network structure, and the short circuit design is added
next to the convolutional layer to solve the problem of thegradient disappearing and the accuracy of the training set.
ResNet is divided into 18, 34, 50, 101 and 152 layers,which differ in the number of layers with 4 groups ofconvolutional layers in the middle.

2.2. Principle analysis of convolution layer
As shown in Fig. 2, the function of the convolution layer
is to use multiple convolution operations to extract multichannel eigenvalues from the output of the upper layer,
and then send these eigenvalues as outputs to the next layer.
The convolution layer reduces the parameters to be trainedin the process of layer by layer extraction. In a given image,the local pixel information is converted to thecorresponding information of the output image after
weighted addition. The part of the convolution layer used
for convolution is called the convolution kernel, its size
can be defined, and the parameters in it are the parameters
we want to train.
Other parameters we define in the convolution kernel
determine the effect of image data extraction, including fill,number of output channels, number of input channels, andstep size. The size of the convolution image is:
𝐻𝑜𝑢𝑡 =
𝐻𝑖𝑛 +2×𝑝𝑎𝑑𝑑𝑖𝑛𝑔[0]−𝑑𝑖𝑙𝑎𝑡𝑖𝑜𝑛[0]×(𝑘𝑒𝑟 𝑛𝑒𝑙𝑠𝑖𝑧𝑒[0]−1)−1[

𝑠𝑡𝑟𝑖𝑑𝑒[0]𝑊𝑜𝑢𝑡 =
𝑊 +2×𝑝𝑎𝑑𝑑𝑖𝑛𝑔[1]−𝑑𝑖𝑙𝑎𝑡𝑖𝑜𝑛[1]×(𝑘𝑒𝑟 𝑛𝑒𝑙𝑠𝑖𝑧𝑒[1]−1)−1[ 𝑖𝑛

𝑠𝑡𝑟𝑖𝑑𝑒[1]

+1

(2)

+ 1]

(3)

2.4. Meaning of loss function
The loss function is an operation function used to measure
the difference between the predicted value f(x) of themodel and the real value Y. It is a non-negative real valuefunction, usually expressed by L(Y, f(x)). The smaller theloss function, the better the robustness of the model. We
use is called the two-dimensional cross-entropy loss
function. This function is a binary classification function,
BCELoss is the binary loss function between the target
value and the predicted value, the formula is:𝑙𝑛 = −𝑤𝑛 ∗ [𝑦𝑛 ∗ 𝑙𝑜𝑔 𝑥𝑛 + (1 − 𝑦𝑛 ) ∗ 𝑙𝑜𝑔( 1 − 𝑥𝑛 )] (4)Where Wn represents the matrix of weights, Xn
represents the prediction matrix of function output, and Yn
represents the target matrix.
2.5. Introduction and application of data sets
In recent years, thanks to the exploration of many
researchers on pedestrian attribute recognition, there aremany open source data sets that can be used, and we will
mainly use the following two data sets.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
130

Pedestrian Attribute Recognition BasedFirst, the first data set is Mark-1501 data set.

stored in the form of a pedestrian border box, each frameof which was manually marked by someone. In the video,
images are captured every 120 frames, and the resulting
images make up the data set. There were 1,404 pedestriansin the dataset, most of whom were captured by two or morecameras, and 36,411 images made up the DukeMTMCreID dataset. In this data set, the training set consists ofseven hundred and two images of people in rows randomly
selected, and the remaining images are used as the test set.
It is worth noting that there are 408 people caught by only
one camera, and they are included as interference items inthe data set. Lin et al. labeled a total of 23 attributes in thisdataset, including gender, whether to wear boots, whether
to wear a hat, whether to have a backpack, shoe color,seven lower body clothing colors and eight upper bodyclothing colors.

3-1(a)Picture under camera a3-1(b)Pictures in other cameras
Fig. 3 Markrt-1501 data set
As shown in Fig. 3, the Mark-1501 dataset was originallyused for gender reidentification research, and was filmedon the Tsinghua campus using six cameras with different
viewing angles. Lin et al. [1]. annotated attributes for each
person in this dataset. In this dataset, a total of 1501
pedestrians and 32,668 character rectangles are included,
as shown in Table 1. In so many images, there are two parts:
the training set and the test set.
Table 1 Labeling probabilities of some data attributesfor Mark-1501
label
probability label
probability
Young0.0186Adult
0.2130Old
0.0107Bag
0.2463
Handbag
0.1145
Downblue
0.1638
Downbrown 0.0919
Downgreen 0.0186
Downpink
0.0386
Downwhite 0.0772
Downyellow 0.0133
Upblue0.0613upgreen0.0746
Uppurple
0.0399upred
0.3901
Upyellow
0.3901clothes
0.3901Up
0.9481hair
0.3262gender
0.4261
Then, the second data set is DukeMTMC-reID.
In the training set, there were 12,936 images, including
751 pedestrians, with an average of 17.2 images perpedestrian. In the test set, there were 750 pedestrians with
19,732 images, an average of 26.3 images per person.
As shown in Fig. 4, the DukemtMC-Reid dataset is asubset of the DukeMTMC dataset. On the campus of Duke
University, eight cameras were used to capture the video

4-1(a)Picture under camera a4-2(b)Pictures in other cameras
Fig.4.DukeMTMC-reID

2.6. Evaluation index design
The number of correct identifications for each semantic
attribute and the number of all samples is calculated, andthe quotient of these two numbers is evaluated as theaccuracy of each attribute identification. The average
accuracy of all attributes can indicate the extent of themodel effect. The formula used is as follows:𝑇
𝑎𝑐𝑐𝑖 = 𝑖(5)𝑁

𝑎𝑐𝑐𝑚𝑒𝑎𝑛 =∑𝑐𝑖=1 𝑎𝑐𝑐𝑖𝑐

(6)

In addition, due to the great imbalance of pedestrian
attribute sample data, the negative sample of a certain©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
131

Peng Wang，Qikun Wang, Shengfeng Wang

attribute may occupy more than 90%. Therefore, abalanced indicator is also needed, and the average
accuracy rate (AP) can balance the identification accuracyof positive and negative samples. The mAP of the average
accuracy of each attribute can show the superiority of thealgorithm on unbalanced data sets. For the ith attribute,
TPi represents the number of positive samples correctlypredicted, Pi is the total number of positive samples, TNi
represents the number of negative samples correctlypredicted, and Ni is the total number of negative samples,
then:
𝐴𝑃𝑖 =
𝑚𝐴𝑃 =

𝑇𝑃𝑖 𝑇𝑁𝑖+𝑃𝑖
𝑁𝑖2∑𝑐𝑖=1 𝐴𝑃𝑖𝑐

Fig.5. Accuracy of some attributes

(7)
(8)

For the research of pedestrian attribute recognition, we
should not only pay attention to the accuracy of individual
attributes, but also pay attention to how many attributes ineach image can be successfully recognized. This index
mainly statistics the accuracy (acc.), accuracy (prec.),
recall rate (Recc.) and F1 values of attribute recognition at
the sample level.
Accuracy: In both positive and negative cases, theproportion of the predicted correct number to the total
number is expressed by the formula:
𝑇𝑃+𝑇𝑁
𝐴𝐶𝐶 =
(9)

Fig.6. Precision of some attributes
On the DukeMTMC-reid dataset, we concluded that the
average accuracy was 0.9152 and the average F1 score was
0.5739.The following figure (Fig.7 and Fig.8)shows therecognition results of some attributes.As shown in the
figure, figure 7 shows partial accuracy, and figure 8 shows
partial precision.
Fig. 7 Accuracy of some attribute
Fig. 8 Precision of some attributes

𝑇𝑃+𝐹𝑃+𝐹𝑁+𝑇𝑁

Precision is relative to the prediction results of positive
examples. The accuracy of the predicted positive examples
is evaluated by the proportion of the real positive examples
in the predicted positive examples. The formula is asfollows :𝑇𝑃
𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =
(10)

3.

𝑇𝑃+𝐹𝑃

Conclusion

The recall rate is judged according to the actual sample.
Its main purpose is to judge the proportion of the predicted
positive example in the actual positive example, and canbe expressed by the formula :𝑇𝑃
𝑟𝑒𝑐𝑎𝑙𝑙 =
(11)
𝑇𝑃+𝐹𝑁

The F1 score is the harmonic mean of the correct rate andthe recall rate, defined as :2∗𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛∗𝑟𝑒𝑐𝑎𝑙𝑙
𝐹1 =
(12)
𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛+𝑟𝑒𝑐𝑎𝑙𝑙

In this paper, nonlinear functions are introduced into afour-dimensional conservative chaotic system to generate
multiple scrolls. After the introduction of nonlinear
function, the equilibrium point of the system changes froma fixed point to a set of equilibrium points. The system
carries out basic characteristic analysis, and discusses its
divergence, equilibrium point and whether the energy is

2.7. experimental result analysis
The conclusion on Mark-1501 is the average accuracy is
0.9631; the average F1 score is 0.6492. The following
figures (Fig. 5 and Fig. 6) show the recognition results ofsome attributes.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
132

Powered by TCPDF (www.tcpdf.org)

Pedestrian Attribute Recognition Based

Mr. Qikun Wang

conservative. The equilibrium points obtained byintroducing one-dimensional nonlinear function are
divided into two categories. For its Lyapunov exponent
analysis, after introducing the sine function without
multiple angles, The Lyapunov exponents of the systemequations show similar periodic characteristics to the sinefunction, and the Lyapunov exponents obtained bychanging the initial values are very different. With thechange of initial value, the phase diagrams obtained arealso different, and the number of vortex attractors formed
is also different, which verifies the multi stability.
Then, the nonlinear function is extended, two nonlinear
functions are introduced, and the system with two
nonlinear functions is further analyzed. The obtainedphase diagram changes from one-dimensional to twodimensional scroll attractor, and the relationship between
the number and arrangement of scroll and the threshold
width is obtained. By changing the initial value, the phase
diagram with different internal distribution but the same
number of scroll is obtained.

In 1996, he received his Bachelor ofEngineering degree from the School ofElectronic
Information
andAutomation, Tianjin University of
Science and Technology, China. He isa senior engineer, research direction is
HVAC, building technology.

References
1. LIN Y, ZHENG L, ZHENG Z, et al. Improving Person Reidentification by Attribute and Identity Learning.
arXiv:1703.07220 [cs.CV], 2017.

Authors Introduction
Ms. Peng Wang
She is a postgraduate tutor of Tianjin
University of Science and Technology.
In 2014, she received a doctorate from
North
China
Electric
Power
University. The research direction is
the functional safety assessment ofsafety instrumented systems.
2.
Mr. Shengfeng Wang
In 2023, he received his Bachelor ofEngineering degree from the Schoolof Electronic Information andAutomation, Tianjin University of
Science and Technology, China.He is
pursuing a master's degree inengineering from Tianjin Universityof Science and Technology.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
133

Simulation of office air conditioning air supply based on COMSOLPeng Wang*, Mengda Liu, Qikun Wang

College of Electronic Information and Automation, Tianjin University of Science and Technology,
300222, China
E-mail: *1789025463@qq.com
www.tust.edu.cn

Abstract
This paper analyzed the summer thermal environment of an office in Tianjin University of Science and Technology
based on COMSOL software. Firstly, the principle of indoor thermal environment distribution was introduced.
Secondly, according to the actual position of the furniture in the office, the mathematical model is constructed basedon the basic theory of computational fluid dynamics. Thirdly, the COMSOL software was used for simulation andcalculation, combining boundary conditions. Finally, the simulation results are analyzed through the simulated indoor
three-dimensional velocity field and temperature field. The simulation results show that the air conditioning supply
can well achieve indoor occupants' comfort.
Keywords: Temperature field, Wind speed field, Indoor thermal environment, CFD simulation, COMSOL

1. Introduction
2. Indoor Physical Model Simulation

Indoor temperature and velocity are one of the mostimportant and frequently measured parameters in indoor
environments, affecting personnel thermal comfort,
energy balance and air flow [1]. Because the indoor
temperature field and velocity field is a complex physicalfield, and there are many influencing factors. The change
process of indoor temperature and velocity has thecharacteristics of large inertia, non-linearity, and easy tobe affected by external environmental factors. Therefore,
the indoor temperature distribution of most buildings isuneven, and the temperature data of different areas such as
the corner, center, floor and ground of the room are very
different. Therefore, it is necessary to use COMSOL fluid
dynamics simulation software for indoor temperature andvelocity simulation.
The first chapter introduces the necessity of simulating
indoor temperature and velocity fields. The second chapter
mainly introduces the construction of the indoor model,
including the indoor objects, the heat transfer coefficientof the wall, the heat transfer coefficient of the window, etc.
Meshing of indoor models. In Chapter 3, the simulated
temperature and velocity fields are analyzed. In Chapter 4,
simulation conclusions are drawn.

2.1. Model selection
The research object is an office on the fourth floor facing
south. The north window and the corridor are adjacent.
Due to the different functions of the room, the interior
structure will also be different. In addition, the parametersof the office building walls, indoor personnel activities androom ventilation times and other factors, all greatly affectthe office internal thermal environment. Since there aremany objects in the office, not all objects can beconsidered, so the physical model only considers two typesof objects, one is large objects such as desks, chairs andcabinets, and the other is hot objects such as computers,
notebooks, human bodies and refrigerators. All objects are
simplified to cuboids, which is conducive to grid divisionand calculation convergence.
This paper takes an office of Tianjin University of
Science and Technology as the research object. The
geometric size of the office is 6m×7.m×3.5m, and there isan air conditioner in the southwest corner of the room. The
air supply port of the air conditioner is 1m×0.5m. The
return air port is simulated as the return air port of the air
conditioner, which is located below the air outlet. One of
the Windows is located on the south wall with dimensionsof 4m×2.3m and 1m above the ground. Suppose there is no©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
134

Peng Wang, Mengda Liu, Qikun Wang

turnover in the office，the number of people in the officeis six. The physical model of the office is shown in Fig. 1
and Fig. 2.

boundary conditions include the properties of the wall
structural materials, the outdoor temperature of 30 degrees
Celsius, the corridor temperature of 30 degrees Celsius, theair outlet speed of the air conditioner of 1m/s, and the air
outlet temperature of the air conditioner of 20 degrees
Celsius. Assuming that the indoor and ambient
temperatures are the same at the beginning, the wholeoffice is divided using a conventional grid to calculate
laminar flow, solid and fluid heat transfer, and nonisothermal flows. The state of the office temperature fieldand velocity field within 6000s of air supply is simulated
and calculated. The boundary conditions and heat sources
are shown in Table 1 and Table 2.

Fig. 1 The exterior of the office physical mode

Table 1 Boundary condition
Boundary
Body of wall
Windows
Door

Heat transfer coefficient5(W/(m2*K))
10(W/(m2*K))
4(W/(m2*K))

Fig. 2 The interior of the office physical model
Table 2 Source of heat
2.2. Governing equation
Source of heatcomputerpeople80

Mass conservation equation is the embodiment of massconservation law in fluid mechanics [2], as shown in Eq.
(1), u,v,w--velocity component, unit m/s,ρ--Fluid density
in kg/m3.
∂(ρ)
∂t

+

∂(ρu)
∂x

+

∂(ρv)
∂y

+

∂(ρw)
∂z

=0

(1)

3. Image Analysis

The hysical meaning of the energy conservation
equationp is that the increment of the total energy of thefluid controlled in the body per unit time is equal to thesum of the energy increment generated by the fluid
capacity of the inflow control and the external action,
which includes the energy increment generated by thermalradiation, heat conduction, work done by volume force,work done by pressure and work done by viscous force.asshown in Eq. (2).
𝜕(𝜌𝑇)
𝜕

(

+

𝜕𝑡
𝜇 𝜕𝑇

𝜕𝑥 𝑝𝑟 𝜕𝑥

𝜕(𝜌𝑢𝑇)𝜕𝑥𝜕+𝜕(𝜌𝑣𝑇)

𝜇 𝜕∅

𝜕𝑦

+

𝜕

The temperature cloud image at room x=1, x=3, x=5m (asshown in Fig. 3, Fig. 4, Fig. 5) and velocity cloud image(as shown in Fig. 6, Fig. 7, Fig. 8) were captured
respectively. The scale on the right of the temperaturecloud image is the size of the temperature, corresponding
to 20 to 30 degrees Celsius from blue to red, and the scaleon the right of the velocity cloud image corresponds to thesize of 0 to 2m/s. Fig. 3 shows the temperature cloud map
at x=1m. The upper left corner is close to the air outlet ofthe air conditioner, where cold air blows into the room andthe temperature is low. The temperature is about 20
degrees Celsius according to the scale. Fig. 4 and Fig. 5
are x=3m and x=5m respectively. The reason why thetemperature in the lower left corner is relatively low is thatthe cold air from the air conditioner blows out and falls
down due to gravity, and the cold air gathers to the groundon the left side, where the temperature is about 25 degrees
Celsius. The combination of the three images shows thatthe temperature on the upper floor of the room is evenly
distributed, about 27 degrees Celsius. However, the𝜕(𝜌𝑤𝑇)𝜕𝑧𝜇 𝜕𝜑=) + 𝜕𝑦 (𝑝𝑟 𝜕𝑦) + 𝜕𝑥 (𝑝𝑟 𝜕𝑧 ) +ST

Flux of heat(W)
100
50
80

(2)

2.3. Mesh partition
COMSOL 6.0 software was used to model the office, thetransient solver was used to solve the established model,
and the dynamic simulation of the air conditioning supply
situation of the office was carried out. The computed©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
135

Simulation of office air

position temperature of the air outlet is low, which is notconducive to the comfort of indoor personnel.

Fig. 6 The velocity field at x=1m
Fig. 3 The temperature field at x=1m

Fig. 7 The velocity field at x=3m
Fig. 4 The temperature field at x=3m

Fig. 8 The velocity field at x=3m
Fig. 5 The temperature field at x=5m

The cross sections at y=1m and y=3.5m were intercepted,
respectively, where Fig. 9 and Fig. 10 show thetemperature cloud, and Fig. 11 and Fig. 12 show thevelocity cloud. Fig. 9 shows the position of the air outletof the air conditioner. It can be seen from the figure thatafter the air from the air conditioner blows out, it falls tothe position of the table due to gravity, and the temperature
is low. Fig. 10 shows that the temperature distribution in
the upper and lower parts of the room is relatively uniform
for the location located in the center of the room at y=3.5m.

Fig. 6 shows the indoor velocity cloud map at x=1m.
The reason for the large wind speed in the upper rightcorner is that it is located at the air conditioning outlet, andthe wind speed is 2m/s. The reason for the large windspeed in Fig. 7 and Fig. 8 is that the wind speed in otherplaces is evenly distributed due to the cold air blown by
the air conditioner. The comprehensive wind speed cloud
map shows that the wind speed in other places is evenly
distributed except for the air conditioner outlet, and theindoor personnel are more comfortable.

Fig.9 The temperature field at y=1m

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
136

Peng Wang, Mengda Liu, Qikun Wang

Fig. 10 The temperature field at y=3.5m
Combined with Fig. 11 and Fig. 12, the comfort ofindoor personnel is not high under the air conditioning, andthe comfort of personnel is good in the middle position of
the room.

Fig. 14 The temperature field at z=1.75m

Fig. 11 The velocity field at y=1m

Fig. 15 The velocity field at z=0.2m

Fig. 12 The velocity field at y=3.5m
Screenshots at z=0.2m and z=1.75m were taken,
respectively, where Fig. 13 and Fig. 14 show thetemperature cloud, and Fig. 15 and Fig. 16 show thevelocity cloud. Combined with the above analysis, theupper temperature of the room is higher than the lower
temperature, and with the height increase, the temperature
also increases. The temperature distribution is uniform in
the z plane.

Fig. 16 The velocity field at z=1.75m
Fig. 17 shows the 3D diagram of the room, where thegreen line represents the path of the wind sent by the air
conditioner, and the red arrow represents the direction andmagnitude of the wind speed. The figure shows that the air
provided by the air conditioner can cover the whole room.
In addition to the position of the air outlet of the air
conditioner, the wind speed is also moderate, which canmeet the comfort of indoor personnel.

Fig. 13 The temperature field at z=0.2m©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
137

Powered by TCPDF (www.tcpdf.org)

Simulation of office air

Mr. Qikun Wang
He
graduatedfrom
Tianjin
Universityof
ScienceandTechnology, is a senior engineer,
research direction is HVAC, building
technology.

Fig. 17 3D air supply path diagram
4. Conclusion
Based on the above simulation analysis, the air provided
by the air conditioner can cover the whole room, and theair supply by the air conditioner can meet the comfort ofindoor personnel.
References
1. Jin, X., He, Y., Liu, S., Liu, Z., & Liu, P. (2022). Analysis
of Distribution Characteristics and Influencing Factors ofAmbient Temperature Field in Buildings. E3S Web ofConferences.
2. Khatri R, Singh A P, Khare V R. Identification of Ideal Air
Temperature Distribution using different location for Air
Conditioner in a room integrated with EATHE–A CFD
based approach. Energy Procedia, 2017, 109: 11-17.

Authors Introduction
Ms. Peng Wang
She is a postgraduate tutor of Tianjin
Universityof
ScienceandTechnology. In 2014, she received adoctorate from North China ElectricPower University. The research
direction is the functional safetyassessment of safety instrumented
systems.
Mr. Mengda Liu
In 2023, he received his Bachelor ofEngineering degree from the School ofElectronic
Information
andAutomation, Tianjin University of
Science and Technology, China.He is
pursuing a master's degree inengineering from Tianjin Universityof Science and Technology.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
138

Solo Wheel Technology-Self-balancing Wheelbarrow
Ziyue Xiao*, Yumei Huang, Zhencheng Chang, Mingxuan Li

Tianjin University of Science and Technology, Tianjin, China
E-mail: *1418249500@qq.com

Abstract
As a new type of personal transportation, self-balancing unicycle has attracted wide attention
with its unique design and advanced control system. We aim to study and optimize theself-balancing performance of the unicycle to improve its stability and maneuverability. Firstly,
we gain an in-depth understanding of its operation principle. On this basis, an advanced control
algorithm is proposed to realize real-time attitude adjustment. Second, for applications incomplex environments, we propose an intelligent sensing system to enhance its environment
sensing capability.
Keywords: Cascade PID, Angular feedback, MPU6050, Motor driver IC 33886.

1. Introduction

practice. This requires two conditions: one is that thepalm of the hand holding the wooden stick can be moved;
The other is that the eye can observe the tilt angle andinclination tendency (angular velocity) of the stick.
Counteract the angle and tendency of the stick by moving
the palm of your hand, thus keeping the stick upright.
These two conditions are indispensable, and in fact they
are the negative feedback mechanism in the control [1],see Fig. 1.

Since they all end up controlling the same control
object (the motor of the car model), there is a coupling
between them. For the sake of analysis, one of them isassumed to be stable with the other control objects. For
example, when controlling the speed, the car model needs
to be able to maintain upright control; When controllingthe direction, the car model needs to be able to maintainbalance and constant speed; Similarly, when it comes tobalance control, speed and direction control are also
required to be smooth. Of these three tasks, keeping themodel balanced is key. Since the car model is affected bythree controls at the same time, from the perspective of
the car model balance control, the other two controls
become its interference. Therefore, the control of the
speed and direction of the car model should be kept assmooth as possible to reduce the interference of balance
control. Taking speed adjustment as an example, it isnecessary to change the inclination angle setting value ofthe car model in the balance control of the car model, so
as to change the actual inclination angle of the car model.
In order not to affect the balance control of the model, thechange of the inclination of the model needs to be carriedout very slowly. This will be discussed in more detaillater in Speed Control.
The intuitive experience of controlling the balanceof the car model comes from people's daily life
experience. The average person can keep a straightwooden stick upright on the tip of a finger with simple

Fig. 1 Feedback control to keep the stick upright
Balance control is also achieved through negative
feedback, which is relatively simple compared to keeping
the stick upright above. Because the model has twowheels on the ground, the body will only tilt in the
direction of the wheels. Controlling the rotation of thewheels to counteract the tendency to tilt in one dimension
keeps the car balanced. The gravitational field, whichuses thin lines to suspend heavy objects, is simplified toform an idealized pendulum model. An upright model canbe seen as an upside-down pendulum placed on a

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
139

Ziyue Xiao, Yumei Huang, Zhencheng Chang, Mingxuan Li

platform that can be moved from side to side [2]. This isshown in Fig. 2.

Therefore, in order for each motor to be able to achieve
forward and reverse rotation, two PWM signals are
required. The two motors require a total of 4 PWM
signals. The drive circuit is shown in Fig. 4.

Fig. 4 Motor drive circuits

Fig. 2 Simple pendulum
In the case of a small offset angle, the restoring
force is proportional to the magnitude of the offset angle
and is in opposite directions. Under the action of thisrestoring force, a single pendulum undergoes a periodic
motion. A pendulum that moves in the air will eventuallystop in a vertically balanced position due to the damping
force of the air. The damping force of the air is directly
proportional to the speed of the single pendulum motion
and is in the opposite direction. The greater the damping
force, the faster the pendulum will stabilize in the vertical
position.

2.2 Surveillance camera
Each photocell outputs two pulsed signals, whichhave the same waveform, but are 90° out of phase. If
the motor is forward, the second pulse is 90° behind,
and if the motor is reversed, the second pulse is 90°
ahead. This relationship can be used to determine whether
the motor is rotating forward or reverse. In the actual
circuit, only one pulse signal is detected, and the speed ofthe motor is measured by its frequency. The steering of
the motor is judged by the positive or negative voltage
applied to the motor. Although due to the inertia of the
car model, it is possible that in the case of low motor
speed, the direction of rotation of the motor and thevoltage applied to it may be different, resulting ininaccurate measurement of motor speed. Due to theracing during the race, the motor is generally running athigh speed, and the direction of the motor is consistent
with the direction of the voltage. Experiments show that
this method can effectively determine the rotation
direction of the motor and control the speed.
2.3 Surveillance camera

2. Hardware Usage Programme
2.1 Motor driver IC 33886
The DSC F8013 in Fig. 3 is a 3.3V device, and its
IO output voltage is up to 3.3V, which does not meet therequirement that the 33886 must be greater than 3.5V, so
a 5V power supply is specially designed in the circuit topull the drive signal of the 33886 up to 5V. Since the IO
port of the F8013 can tolerate 5V, the above circuit canmake the 33886 drive signal voltage reach 5V.

In the article on the control principle of the car
model, there are two ways to measure the angle andangular velocity of the car model, the first is through thegyroscope and the acceleration sensor, and the second isthrough two acceleration sensors. The angular velocity is
integrated with feedback to obtain a smooth angle signalthat is consistent with that of the accelerometer. This partof the signal processing can be done by the
microcontroller software, or it can be realized by external
op amp circuits. In order to verify the performance of thiscircuit, the ENC-03 measurement signal mounted on themodel car was also acquired in Fig. 5.

Fig. 3 Dual-motor drive circuit
In order to improve the application efficiency of thepower supply, the PWM waveform of the drive motor
adopts a unipolar drive method. That is, the voltage
applied to the motor is a voltage in a PWM cycle.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
140

Solo Wheel Technology-Self-balancing Wheelbarrowrecommended that the Bootloader download function of
F8013 and serial port monitoring be jointly developed
into a host computer software, so that the development
and debugging of the entire car model program can becompleted through the serial port.
3.3. Analog acquisition
The F8013 has six channels of AD conversion. Fiveof these channels are applied in the reference design.
Write an AD conversion program and send the collectedvalues to the upper computer for display. By observing
the collected values of each channel, the operating statusof each sensor, the dynamic range of the signal, and theinfluence of noise on each channel are confirmed. It isbest to display the values collected by each channel on a
curve, so that you can intuitively observe the changes inthe signal and the amplitude of the noise. The following
is the acquisition waveform of the gyroscope andaccelerometer displayed on the upper microcomputer.
Each analog quantity will have a different
proportion of noise, and low-pass filtering is required foreach acquisition. The simple way to do this is to average
it over multiple acquisitions. The waveform data shownin Fig. 6 is averaged over 20 acquisitions [3].

Fig. 5 Measurement angle waveform plot
From the waveform, it can be seen that the angularvelocity of the car model integrated by the accelerometeris the same as the waveform of the angular velocity
measured by the gyroscope, which verifies thecorrectness of the method.
3. Innovative Design and Advantages
3.1. Vehicle tuning
The debugging of the car model is divided into
various links such as debugging preparation, staticparameter tuning, dynamic parameter tuning, mechanicaladjustment of the car model and competition strategy
formulation. There are many parameters and components
involved in the whole commissioning, and theseparameters are closely influenced by each other. If you
don't have a clear understanding of the physical process
and whether there are correct debugging steps, then all
kinds of errors in the debugging process will cover up thecorrect cause, affect the entire debugging process, andeven shake the confidence of the production.

Fig. 6 AD acquires the signal
4. Job Scenario Application.

3.2. MCU program download and communication

4.1. Personal travel

The DSC 56F8013 has on-the-system programming(ISP) capability. After the MCU came out, there was abootloader program inside, and the user program could be
downloaded through the UART serial port. If the program
itself is bootstrapped for the first time, all future program
downloads can be done through UART. For more
information, please refer to the materials available on theFreescale website. UART can not only improve thedownload of the program, but also display the serial
characters sent during the operation of the program withthe assistance of the upper serial port monitor, which isconvenient for debugging the program. It is

The wheelwheel can be used as a lightweight, portablemeans of transportation for short distances in the city.
The wheels are fast and flexible, they can shuttle between
the flow of people, and they can control the direction andspeed of the wheels by tilting their bodies, which is very
convenient and fast.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
141

Powered by TCPDF (www.tcpdf.org)

Ziyue Xiao, Yumei Huang, Zhencheng Chang, Mingxuan Li

4.2. Industrial applications and military
applications
Authors Introduction

The wheels can also be applied to industrial scenarios,
such as warehouses, factories, etc. The uniwheel cancontrol the direction and speed of travel by tilting thebody, making it suitable for transportation and handlingwork in narrow, crowded environments. The uniwheel isalso used in military fields, such as reconnaissance,
surveying, etc. The single wheel can carry out fast andmaneuverable action and detection in severe
environments through its lightness and flexibility. For
example, on the battlefield, a single wheel can be used forquick patrols, monitoring the situation, gathering
intelligence, etc.

Mr. Ziyue Xiao
He is hold a positive and serious attitude
towards work, have a strong sense of
responsibility, be sincere, meticulous,
optimistic, and stable, and cancontinuously learn and improve oneself
in practical work, and do their job well

Ms. Yumei Huang
She studied at Tianjin University of
Science and Technology in 2021. She is
now the first major in her faculty and is
expected to receive her Bachelor's degreein 2025.

5. Synthesis
The unicycle is a balanced vehicle based on theprinciple of an inverted pendulum, consisting of a bodywith only one wheel and a balance control system. The
movement and control of the uniwheel is realized by theuser's body tilt, which is lightweight, flexible and fast,and is suitable for a variety of application scenarios suchas personal travel, toys and entertainment, industrial
applications, and military fields.

Mr. Zhencheng Chang
He is currently pursuing his undergraduate
degree at the School of Electronic
Information and Automation, Tianjin
University of Science and Technology.

References

Ms. Mingxuan Li[1]. De Xiao Kong, A Yi Wang, Jun Hagi Li, Research on
two-wheeled self-balancing vehicle control system based onquaternion, Electronic Measurement Technology, 2023.6, pp.
75-76.
[2]. T.S. Chen, S.Y. Chen, An engineering solution design
based on STM32 two-wheeled balance bike, China Scienceand Technology Information, Vol. 20, No. 20, 2023, pp.
121-123.
[3]. Yuanwei Chen, Theoretical study and modelling ofembedded-based self-balancing vehicle, Instrumentation
Technology, Vol. 3, No. 2023, pp. 75-76.

Her major is environmental science. She
is currently a undergraduate student in
Tianjin University of Science andTechnology©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
142

Second-order Self-balancing Inverted Pendulum
Ziyue Xiao*, Zhencheng Chang, Mingxuan Li, Yumei Huang

Tianjin University of Science and Technology, Tianjin, China
E-mail: *1418249500@qq.com

Abstract
The team utilizes the second-order inverted pendulum cart based on LQR controller for steady pendulum control
with light rods. First, the second-order inverted pendulum is used as a research object to obtain its set of dynamical
equations, then, the set of dynamical equations is written in the form of state-space expressions, and finally, thesecond-order inverted pendulum system of a balanced trolley with a light rod is controlled by a stabilized pendulum
using the LQR controller. The inverted pendulum is a typical nonlinear, underdriven and unstable system, which canrealize the all-round wind resistance of the rod and occupy less space with high stability.
Keywords：LQR controller，Second-order handstand pendulum，Lagrangian mechanics.

properties, the study of their stability under windforce is more complex and critical. Therefore, inorder to ensure the wind resistance and stabilityof the rod, it is necessary to carry out relevant
research and application.

1. Introduction
In real life, many structural systems need to
be stable in a variety of environments, such as
high-rise buildings, bridges, tower structures, etc.
One of the important factors is the influence ofclimatic conditions, especially wind on thebuilding structure. Due to the uncertainty andvariability of wind forces, buildings andstructural systems need to be designed to bestable enough to resist the effects of wind. In
practical application, in order to ensure thestability of the structural system, it is usuallynecessary to take various resistance measures,such as adding dampers, support devices andtraction devices to the structural system, This isshown in Fig. 1.

To this end, it is possible to study the
dynamic characteristics of the rod under theaction of wind, etc., the morphological andmaterial characteristics of the rod, and thestructural design of the rod to explore how toachieve the stability of the rod. At the same time,some practical application devices can also be
developed to help the rod stabilize under theaction of wind and other forces, such as adding
support, traction and adjustment devices [1].
However, these methods are too traditionaland require the support of manpower andmaterial resources, and when the device isdamaged, it is not easy to repair,time-consuming and labor-intensive. Therefore,
it is urgent to develop a new type of stable andnon-falling rod device.
In view of the above shortcomings, ourgroup proposed a second-order inverted
pendulum system based on LQR controller tostabilize the pendulum control of the light bar.
Our team first took the second-order inverted
pendulum system of the balance trolley withlight bar as the research object, analyzed its
dynamics based on the Lagrangian mechanics

Fig. 1. Telephone poles in kindtheir

For slender structures such as rods, due tospecial morphological and material©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
143

Ziyue Xiao, Zhencheng Chang, Mingxuan Li, Yumei Huang

method, obtained its dynamic equations, andthen wrote the dynamic equations into the formof state space expressions, and finally, after abrief overview of the LQR controller, the LQR
controller was used to stabilize the second-orderinverted pendulum system of the balance trolleywith light bar. The inverted pendulum is atypical nonlinear, underdriven, and unstablesystem. Through the research and optimizationof the inverted pendulum system, our team has
obtained new control algorithms, which haveapplication prospects in robot control, missileinterception control, aviation docking control
and general industry.
CAN bus, multiple serial communicationinterfaces, analog-to-digital converters anddigital-to-analog converters, etc., and has a wide
range of applications in industrial control,
automotive electronics, home appliance control,
medical equipment, smart home and other fields.

2. Hardware Usage Programme
2.1 Mpu6050
MPU6050 is a commonly used six-axis
inertial measurement unit (IMU) consisting of agyroscope and an accelerometer. It is capable ofmeasuring changes in the rotation andacceleration of an object in three axes.

In addition, STM32C8T6 has a wealth ofdevelopment resources, including development
boards, online IDE tools, official documentation,and community support, which facilitateengineers to quickly develop and test embedded
system applications.

MPU6050 has a built-in 3-axis gyroscope
and a 3-axis accelerometer that cancommunicate with the microcontroller via an
I2C or SPI interface. It is widely used in robots,
drones, game controllers, and other applications
that require gesture perception or motion
tracking [2].

3. System programming
3.1LQR control

With MPU6050, you can obtain the angularvelocity and acceleration data of an object,which can be used to calculate the object's poseor perform motion tracking. This is shown in Fig.
2,

After the above qualitative description of
the LQR controller, the simulation calculation
was carried out by the relevant functions ofMatlab (the mass of the body is 0.9 kg, theheight is 0.126 m, the mass of the rod is 0.1 kg,
and the length is 0.390 m [3], which is shown in
Fig. 3.

Fig, 2 MPU6050
2.2 STM32C8T6
Fig. 3 MATLAB simulation results
STM32C8T6 is a 32-bit ARM Cortex-M3
core microcontroller from STMicroelectronics. It
has high performance, low power consumption,a wide range of peripheral interfaces and richdevelopment resources, and is one of thecommonly used MCUs in embedded systemdevelopment.

The device was then simulated with
Simulink, and the results are shown in Fig. 4.

STM32C8T6 has expandable memory andperipheral interfaces, including high-speed USB,
©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
144

Second-order Self-balancing Inverted Pendulum

unchanged, so as to study the influence of thechanged factor on things, study them separately,
and finally solve them comprehensively, thismethod is called the control variable method. It
is an important method of thought in scientificinquiry, and is widely used in various scientificexplorations and scientific experiments.
Fig. 4 Simulink simulation results

4. Introduction of Functional Module

It can be seen from Fig. 2 that under theaction of wind, the rotation angle of the wheel ofthe second-order inverted pendulum system withlight rod reaches the maximum at 0.7s, and therotation angle of the wheel reaches the
maximum at 0.05s, and the second-orderinverted pendulum system with light rod
gradually tends to stabilize at 2s and basicallyachieves the stable effect at 3s [4].

4.1Simulink emulation
For the test of the second-order inverted
pendulum, a combination of simulation andreality is used. First, the system was simulatedusing Simulink to obtain the state data of thedevice. It is then combined with the real thing toverify the correctness of the simulation data. The
controllable FS-75 industrial fan was used toperform wind tests on the second-order inverted
pendulum system with light rods, and the device
data was measured by changing the length,
weight, and material of the rods. Throughcontinuous simulation and testing, the efficiencyof simulation testing is improved and the trial
and error time is reduced. Extensive testing toensure the correctness of the simulation and theability to improve the simulated model in realtime complement each other for continuous
improvement.

3.2Wind simulation
Wind Simulation We use Fluid Dynamics(CFD) simulation, which is a method of
simulating fluid flow and force on objects
throughnumerical
calculations.
When
simulating the action of wind, the flow of thewind field can be simulated by establishing amathematical horizontal shape, and the object tobe tested can be added to the horizontal
simulation, and the wind force on the object canbe calculated through numerical calculation.

4.2 FS-75 Industrial Fan Test

We perform a CFD simulation of asecond-order inverted pendulum model with alight bar, and the results are shown in Fig. 5.

Experiments with a controllable FS-75
industrial fan and anemometer allow for better
control of single variables for experimental
measurements. After the experimental model isestablished, the system can be simulated andmeasured by Simulink rods of different lengths,
weights, and materials, which greatly reduces
the experimental time and improves the accuracyof the experimental data, which is shown in Fig.
6.

Fig, 5 CFD simulation
Wind power is the key to test the stabilityof the second-order inverted pendulum trolleywith light bar, and the control variable method isused to test the wind power of the second-orderinverted pendulum trolley with light bar. In
physics, the method of controlling factors isoften used to control multi-factor problems,
turning multi-factor problems into multi-factor
problems. Only one of the factors is changed at atime, and the remaining factors are controlled.
Fig. 6 FS-75©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
145

Powered by TCPDF (www.tcpdf.org)

Ziyue Xiao, Zhencheng Chang, Mingxuan Li, Yumei Huanglinear two-stage inverted pendulums, Computer
Simulation, 2023.Vol. 40, pp. 314-320[3]. Hui-Wen Shi, Jie Zheng, MATLAB and
ADAMS simulation study of linear inverted pendulum
based on model predictive control, Industrial
Technology and Vocational Education, 2023 XXI, pp.
1-5[4]. Liu, Jiawei, Song, Nan, et, al, Optimisation andreal-time simulation of inverted pendulum control
system, Chemical Automation and Instrumentation,
Vol. 50, 2023, pp. 453-458[5]. Ji Hong, Wang Yingli, Zhao Zhongyi, et, al,Mathematical modelling and system performance
analysis based on a two-stage inverted pendulum,
Mechanical Design, Vol. 2023, No. 40, pp. 137-142

5. Synthesis
In this experiment, the second-order inverted
pendulum system was used to carry out the
non-inverted experiment on the light rod, and thesecond-order inverted pendulum system withlight rod was simulated by Simulink, and thesystem gradually stabilized in 2s. Then, thesecond-order inverted pendulum system withdifferent rod lengths, different system states anddifferent materials was simulated respectively,
and the data state of the optimal second-orderinverted pendulum system and device with lightrod was obtained, and the influence of using
different controllers on the stability of the device
was discussed, and the time error was calculated
to obtain the best controller. Then, combinedwith the real thing, the wind test of thesecond-order inverted pendulum system withlight rod was carried out with the controllable
FS-75 industrial fan and the stability test of thedevice was carried out by changing the length ofthe light rod, and it can be concluded that theexperimental device has high stability, and thetime to reach the stable state at the wind speed of
1 ~ 3m/s is 0.72s, 0.91s and 1.33s respectively,
and the length of the rod is increased within acertain range, and the system can achieve
stability, which meets the requirements ofphysical experiments [5].
In this experimental setup, the second-orderinverted pendulum handstand model was studied,the optimal handstand state was discussed under
different rod lengths and materials, the stabilityof the system was simulated under different
external disturbances (wind speeds), and theinfluence of different controllers on the stabilityof the device and the time error were discussed.
In the future, it is planned to change the lengthof the trolley (lower rod part) in theexperimental device and simulate themeasurement. Multiple fans were used to test thedevice to discuss whether the device could be
more stable in more complex situations.

Authors Introduction
Mr. Ziyue Xiao

He is hold a positive andserious attitude towards work,
have a strong sense of
responsibility, be sincere,
meticulous, optimistic, andstable, and can continuously
learn and improve oneself inpractical wor
Mr. Zhencheng Chang
He is currently pursuing his
undergraduate degree at the
School of Electronic Information
andAutomation,
Tianjin
University of Science andTechnology.

Ms. Mingxuan Li
Her major is environmental
science. She is currently aundergra student in Tianjin
University of Science andTechnology.

Ms. Yumei Huang
She studied at Tianjin Universityof Science and Technology in
2021. She is now the first major inher faculty and is expected toreceive her Bachelor's degree in
2025.

6. References[1]. Hui-Hui Zhang, B.J. Hou, J. Gao, et, al,Research and implementation of stable pendulum
control for linear inverted pendulum based on LQR,
Mechanical Design and Manufacturing, 2023.7, pp.
1-6[2]. Jiang, Lingyun,Xu, Bingji,Zhang, Fenghua, et,al, A study on a stabilisation control algorithm for©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
146

Design of Nanny's Abnormal Behavior Recognition Bracelet Based on Human Activity
Recognition(HAR) Deep Learning Model
Dengpeng Wang*, Yingfan Zhu, Yande Xiang, Ziyue Xaio

Tianjin University of Science and Technology, Tianjin, China
Email: 3264418107@qq.com

Abstract
This article designs a nanny abnormal behavior recognition bracelet. The bracelet is equipped with multiple sensors
and a powerful control board, forming a complete nanny abnormal behavior recognition system, which realizes nanny
abnormal behavior recognition and alarm in various environments. It uses a gyroscope to collect the three-axis
acceleration information of the nanny, and uses the HAR model to infer the nanny's real-time behavior. When theabnormal behavior of the nanny is inferred, STM32 sends the GPS collected positioning information to the employer
through ESP32 for timely alarm.
Keywords: Behavior recognition, Nanny, HAR model, Alarm

of HAR model. The third part introduces the hardware
selection. The fourth part introduces the software designof the system. The fifth part summarizes the main content
of this paper.

1. Introduction
With the increasing aging situation in China and thegradual implementation of the two-child and three-childpolicy, the number of elderly and young people in China
has soared, and more and more elderly and children need
to be taken care of, and the market scale of the nanny
industry has expanded year by year. However, it isdifficult to regulate the behavior of nannies, and theincidents of nannies abusing the caretakers are frequent.
At present, employers can only monitor nannies by
installing surveillance cameras, but there are large deadcorners where nannies can still commit violent acts
against the caretakers.In particular to dynamics analysis
with moving bodies, the selection of the numericalintegration method is crucial for the realization of theactual dynamics occurring in the real world. It cannot be
solved by a simple way to chip the time step of theintegration in the explicit numerical method. In theimplicit numerical integration, it will help to refine thetime step adaptively.

2. Construction and deployment of HAR model
2.1. HAR Model Introduction
Activity recognition has recently gained attention as a
research topic because of the increasing availability ofaccelerometers in consumer products, like cell phones,
and because of the many potential applications [1]. HAR
model is an efficient human behavior recognition model,
which aims to accurately monitor complex humanbehavior with limited hardware resources. The model
uses a convolutional neural network (CNN [2])architecture, a deep learning architecture that has
achieved remarkable success in the field of image
recognition and is particularly suitable for processing
multidimensional time series data.
The reasoning principle of the model is that when thehuman body performs different actions, the three-axis
acceleration curves of Roll, Pitch and Yaw are
significantly different. Fig. 1 and Fig. 2 show the threeaxis acceleration curves of the human body while sitting
and jogging respectively, and it can be seen that the two
curves are significantly different. The model can judge
the specific behavior of the human body by the triaxial
acceleration when performing different actions.

A nanny's abnormal behavior recognition bracelet
based on HAR deep learning model was studied to solve
such pain points. The nanny needs to wear the bracelet
anytime and anywhere, and the employer can check thenanny's behavior in real time through the mobile phone.
When the nanny's behavior to the caregiver is identified,the location of the nanny is directly sent to the employer
so that the employer can timely understand the situation
and report to the police.
The rest of this paper is organized as follows: The
second part introduces the construction and deployment©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
147

Dengpeng Wang, Yingfan Zhu, Yande Xiang, Ziyue Xaio

convolutional nuclei to 90, the size to 3*3, and the step
size to 1, the local pattern of the input data is effectively
extracted, which helps to capture subtle changes in the
behavioral data. The activation function of the
convolutional layer is ReLU [3] to introduce nonlinear
characteristics and enhance the expressiveness of themodel.
In the maximum pooling layer of the second layer, a
2*2 pooling kernel is adopted and its step size is set to 2
to reduce the spatial dimension of the data and thesensitivity of the model to the input data. This step helps
to improve the generalization performance of the modelso that it can better adapt to changes in different samples.

Fig.1 Three axis acceleration curve of human bodyduring sitting still.

The design of the third convolutional layer continues tofollow the above principles, the number of convolutionalcores is 44, the size is 3*3, the step size is 1, the zero
padding is 0, and the activation function is ReLU. The
setting of this layer further deepens the feature extractionof time series data, and provides more informative input
for the subsequent processing of the full connection layer.
The next three fully connected layers play a key role in
the overall structure. The 5632 nodes of the fourth fully
connected layer are responsible for integrating the local
information extracted by the convolutional layer to form
a holistic understanding of different behaviors. Two fully
connected layers with a number of 64 nodes further
improve the abstraction capability of the model, enabling
it to better capture the abstract characteristics of the
behavior.

Fig.2 Triaxial Acceleration Curve of the Human Body
During Jogging.
2.2. Construction of HAR model
In the process of model construction, we fully consider
the advantages of convolutional neural network in timeseries data processing. The convolutional layer caneffectively capture local patterns in the data, while themaximum pooling layer helps to reduce the sensitivity ofthe model to the input data and improve the robustness ofthe model. The fully connected layer is responsible for
integrating these local information to form a holistic
understanding of different behaviors. This hierarchicalstructure allows the model to better learn and understandcomplex behavioral features. The overall structure isshown in Fig. 3.

The final output layer is a fully connected layer with six
nodes, each corresponding to a behavior category(walking, jogging, going up, going down, standing,sitting, etc.). This design ensures that the model canaccurately classify different behaviors, providing a
reliable basis for real-time monitoring.
2.3. Deployment of the HAR model
In order to deploy the model on a small-capacity
embedded device, we first upgraded the HAR model
iteratively for several times, and successfully compressed
the size of the HAR model from the initial 387.43KB to
19.75KB through operations such as pruning,quantization and optimization of the algorithm structure.
The model size optimization effect is shown in Fig. 4.

Fig. 3 Overall Network Structure of HAR ModelIn the design of the network structure, the parameter
configuration of each layer is paid attention to to ensurethat the model can effectively capture the key features inthe timing data. First, by setting the number of©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
148

Design of Nanny's Abnormal

168MHz. With 512KB Flash and 192KB RAM, thepowerful computing power and sufficient memory areenough to meet the real-time requirements of nanny
abnormal behavior recognition bracelet behavior
reasoning. The outline design of the chip is shown in Fig.
6.

a. Before optimizationb. After optimization
Fig. 4 Size comparison of HAR model before and after
optimization
Fig. 6 STM32F407VET6 chip shape design

After lightweight and efficient processing, HAR model
not only has a smaller storage footprint, but also can run
on low-resource devices. In the training and design phaseof the model, we employ a series of efficient algorithms
and neural network structures, including advanced
techniques such as convolutional neural networks (CNN)
and recurrent neural networks (RNN [4]). These choices
greatly reduce the training time and prediction time of themodel to ensure both accuracy and timely reasoning inreal-time applications. The improved model has achieved
remarkable results in more than 100,000 test sets, and theaccuracy rate of the model has increased from 85.19% to89.22%, as shown in Fig. 5.

a.

3.2. WIFI moduleThis design uses ESP32-WROOM-32WIFI module, its
appearance is shown in Fig. 7. The ESP32-WROOM-32
is a powerful universal Wi-Fi+BT+BLE MCU modulewith powerful features and a wide range of application
areas. It can be used in low-power sensor networks andhigh-performance tasks such as speech coding, audio
streaming, and MP3 decoding.

Before optimizationb. After optimization
Fig. 5 Comparison of accuracy before and after HAR
model optimization

Fig. 7 WIFI module appearance diagram

The deployment of the model is realized by
STM32CubeMX tool, and the Cube AI component in thesoftware can compress the model according to different
magnifications and convert it into C language, so as tofacilitate the deployment of the model to the embedded
development equipment.

The module combines high performance computing
and low power consumption, which is very suitable foriot project development application scenarios. It uses
asynchronous serial port to communicate with STM32 toreceive data, acts as a Wi-Fi access point or client,accesses Aliyun iot platform to achieve wireless network
connection, and supports wireless communication
between iot devices.

3. Hardware selection

3.3. Gyroscope

3.1. Main control chip

The gyroscope in this design uses MPU6050 attitude
sensor, and its chip appearance is shown in Fig. 8. The
MPU6050 is a 6-axis attitude sensor that integrates a 3axis MEMS gyroscope, a 3-axis MEMS accelerometer,
and a scalable Digital Motion Processor (DMP). The
range of the accelerometer can reach ±16g, and there is aconfigurable digital low-pass filter inside, and the

The main control chip adopts STM32F407VET6
developed by ST Company. The chip adopts Cortex-M4
core, and the peripherals are mounted with multi-channel
I2C, SPI, CAN and other communication interfaces. It
has rich functions and its main frequency reaches©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
149

Dengpeng Wang, Yingfan Zhu, Yande Xiang, Ziyue Xaio

configurable register can select the output data for lowpass filtering. The MPU6050 communicates with the
STM32 via the IIC bus to transmit the nanny's three-axis
acceleration.

Fig. 10 shows each meaning of the acceleration range
register of the MPU6050 attitude sensor and theconfiguration guide. During configuration, theaccelerometer range can be controlled by writing data to
the register. Fig. 11 is an introduction of the acceleration
read register of the MPU6050 attitude sensor. When the
MPU6050 is used, the raw data of the three-axis
acceleration is obtained through the 16-bit ADC value ofthe acceleration read register after the accelerometer
RANGE is configured. The relationship between thethree-axis acceleration (ACC) and the acceleration range(R) and the raw data (ADC) of the three-axis acceleration
can be expressed as:
𝐴𝐶𝐶 =Fig. 8 MPU6050 chip𝑅 ∗ 𝐴𝐷𝐶32768(1)

3.4. GPS Module
The GPS module uses TAU1201 positioning module,
and its module appearance is shown in Fig. 9. TAU1201
is a high-performance dual-band GNSS positioning
module, equipped with BDA Beidou's CYNOSURE III
GNSS SoC chip, the module supports the new generationof Beidou III signal system, and supports all civil
navigation satellite systems in the world. These include
BDS, GPS, GLONASS, Galileo, IRNSS, QZSS and
SBAS (WAAS, EGNOS, GAGAN and MSAS). The
TAU1201 integrates an efficient power management
architecture to provide a high-precision, high-sensitivity,
low-power solution for GNSS navigation applications.

Fig. 10 MPU6050 aceleration configuration registerFig. 9 Appearance of the TAU1201 module

Fig. 11 MPU6050 acceleration read register

4. System software design

4.2. Nanny location acquisition

4.1. Acquisition of triaxial acceleration

The nanny location is obtained through the TAU1201
module. TAU1201 module can capture the signal of thesatellite to be tested according to a certain height cutoff
Angle of the satellite, track the operation of thesesatellites, transform, amplify and process the received
GPS signal, so as to measure the propagation time of the
GPS signal from the satellite to the receiver antenna, andinterpret the navigation message sent by the GPS satellite.
When the TAU1201 is finished interpreting, it
communicates with STM32 through its own serial port tosend the nanny location information to STM32.

The STM32 reads the contents of the relevant registersof the MPU6050 through the IIC communication
protocol, a serial communication bus using a multimaster-slave architecture developed by Philips in the
1980s to allow motherboards, embedded systems, ormobile phones to connect low-speed peripherals. The IIC
communication rate of STM32F407VET6 can reach up
to 400KHz, which is enough to meet the requirements of
HAR model inference.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
150

Powered by TCPDF (www.tcpdf.org)

Design of Nanny's Abnormal

4.3. Behavioral inference data and location
information upload

Ms. YingFan Zhu
She is currently pursuing her
undergraduate degree at the School ofElectronic Information and Automation,
Tianjin University of Science andTechnology.

Data upload is completed through the ESP32-Wroom32wifi module. When the nanny's behavior is deduced on
STM32, the HAR model inference result will be sent to
ESP32 through STM32's own serial port. After the
ESP32 makes the judgment of dangerous behavior, thejudgment result will be sent to the employer's mobile
phone server through MQTT protocol. When it isdeduced that the nanny is in danger, the nanny's real-time
location information is uploaded to the employer's mobile
phone so that the employer can take the next step.

Mr. YanDe XiangHeiscurrently
pursuing
hisundergraduate degree at the School ofElectronic Information and Automation,
Tianjin University of Science andTechnology.

5. Conclusion
The nanny abnormal behavior recognition bracelet, to alarge extent, solves the problems of traditional
surveillance cameras, such as large monitoring deadAngle and not easy to carry, so that the nanny's behavior
can be better supervised and standardized, has a good rolein promoting the development of the nanny industry, andindirectly protects the life and property safety of thecaretakers.

Mr. ZiYue Xiao
He is currently pursuing his undergraduate
degree at the School of Electronic
Information and Automation, Tianjin
University of Science and Technology.

References
1.

2.

3.

4.

Jennifer R. Kwapisz, Gary M. Weiss, Samuel A. Moore，
Activity Recognition using Cell Phone Accelerometers，
ACM SIGKDD Explorations Newsletter.Volume 12,Issue
2.2011: pp.74-82.
Wang Qinghua;Li Ziwei;Zhang Shuqi;Chi Nan;Dai
Qionghai,Aversatile
Wavelet-Enhanced
CNNTransformer for improved fluorescence microscopy image
restoration,Neural
Networks.Volume
170,Issue.
2024:pp.227-241.
He Zihao;Shu Qianyu;Wang Yinghua;Wen Jinming,A
ReLU-based hard-thresholding algorithm for non-negative
sparse signal recovery,Signal Processing.Volume
215,Issue.2024.
Xu Ziming;Leung Juliana Y,A novel formulation of
RNN-based neural network with real-time updating – Anapplication for dynamic hydraulic fractured shale gas
production
forecasting,Geoenergy
ScienceandEngineering.Volume 233,Issue.2024.

Authors Introduction
Mr. DePeng Wang
He is studying at the School of Electronic
Information and Automation at Tianjin
University of Science and Technology,
China.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
151

Research on dynamic obstacle avoidance and complex path planning strategies based on ROS
robotsYi-Wei Chen

Bachelor's Degree Program in Intelligent Robotics, National Yunlin University of Science & Technology .123 University Road,
Section 3, Douliou, Yunlin 64002, Taiwan, R.O.C
Jr-Hung Guo*

Department of Electrical Engineering, National Yunlin University of Science & Technology,123 University Road,
Section 3, Douliou, Yunlin 64002, Taiwan, R.O.C
Kuo-Hsien Hsia

College of Future National Yunlin University of Science and Technology,
123 University Road, Section 3, Douliou, Yunlin 64002, Taiwan, R.O.C

Email: b10811150@yuntech.edu.tw, jrhung@yuntech.edu.tw, khhsia@yuntech.edu.tw
Abstract
Robot Operating System (ROS) is a software system framework used by many robot systems. Although ROS provides agood development environment and related frameworks, ROS is not suitable for public places such as restaurants becauseof the coming and going of people. Dynamic obstacle avoidance is often handled by stopping the robot, or when there are
frequent environmental map changes or when sensors such as optical radar fail, the stop action is also used. However, thisoften causes path obstructions or delays in completing tasks. Therefore, this study attempts to use images, auxiliary sensors,and various path avoidance strategies to solve the problem of the robot stopping and waiting for the obstacles to disappear.
The problem of rapid changes in map paths.
Keywords:Robot Operating System (ROS), Dynamic obstacle, Dynamic obstacle avoidance.

band (TEB) [5]. Make mobile robot obstacle avoidance more
efficient.
It can be found from previous related research that many
mobile robots mostly use small controllers such as Raspberry
Pi [6] due to cost and volume. And it uses LiDAR Light
Detection and Ranging (LiDAR) as the main sensor.
Although such an architecture is lower in cost, it results ininsufficient computing power. And because when detecting
obstacles, lidar can only detect obstacles and measure
distances on the plane that light or laser can scan. It often
happens that when an obstacle is not in the LiDAR lightscanning area, the obstacle is misjudged or cannot be
detected. Therefore, the development and application ofmobile robots are hindered. Therefore, this research will use
the architecture of x86 computers. In addition to using
LiDAR, it will also use imaging technology. Through thisarchitecture, the mobile robot's computing power andobstacle detection can be faster and more accurate. Allowing
mobile robots to truly overcome dynamic obstacles in theenvironment and enable practical applications.

1. Introduction
Robot Operating System (ROS) [1] is a very commonlyused development platform for developing and researchingrobot systems in recent years. Because this software uses
open-source code and uses resources provided by many
robot developers. Let this ROS system grow rapidly.
Whether mobile robots can actually operate in actual
environments is also a very important research topic formobile robots. Because the mobile robot is in the actual
environment, cannot avoidance dynamic or static obstacles.
Or without the ability to re-plan the correct path, the mobile
robot well cannot actually operate in a real environment. Of
course, this also limits the application of mobile robots.
Therefore, this study uses ROS as a research platform tofocus on how mobile robots dodge dynamic obstacles in theenvironment. And re-chart the path to try to find a solution.
This can reduce the chance of the mobile robot stopping
when encountering obstacles and improve the efficiency ofthe mobile robot. It also enables mobile robots to be appliedin real environments. In previous research, dynamic obstacle
avoidance is an important research topic for many mobile
robots. For example, CHEN, Chin S., et al. [2] use the ROS
architecture to evaluate the path cost of autonomous mobile
robots (AMR) for obstacles in the environment. Then choose
the better path and let the mobile robot run. CHOI, Jaewan.,
et al. [3] use reinforcement learning to improve the reliabilityof dynamic window approach (DWA) [4] and timed elastic

2. System Architecture
The architecture diagram of the mobile robot is shown in
Fig. 1. The robot is equipped with cameras, LiDAR andSTM32 control boards, etc. It also has many various
interfaces for expansion of different hardware. It can be
widely used in actual or simulated various task development
and applications. This robot architecture uses ROS as the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
152

Yi-Wei Chen, Jr-Hung Guo, Kuo-Hsien Hsia

main core and adds modules such as targets following
systems. The system can be adjusted according to different
applications to provide a diverse robot development
environment. At the same time, the development results canalso be quickly transferred to different mobile robot
platforms.

recognition and distance calculation core, the integrated
operation structure diagram is shown in Fig. 2. In thisprocessing block, KCF is mainly used to quickly identify
targets and establish preliminary samples for subsequentidentification. The main purpose is to identify possible
obstacles or targets as quickly as possible, and then use
DSST to further process the identification model andidentification samples. Because KCF has a faster processing
speed, it can first determine which objects may be closer tothe mobile robot. Then, when necessary, it will be handed
over to LiDAR and DSST for confirmation. Once a possible
obstacle or path impact on the mobile robot is confirmed, it
will continue to be tracked with KCF.

Fig.1 ROS mobile robot architecture block diagram.
In the ROS system, the sensors commonly used for object
distance measurement are mainly LiDAR and imaging lenses.
In order to consider the calculation amount and get the object
distance, many robot platforms are based on LiDAR.
However, general LiDAR is only 2D, that is, it can only
measure one plane, and this also causes LiDAR to be unable
to detect objects outside the 2D plane it scans. Therefore, inorder to avoid obstacles that cannot be detected using LiDAR,
this study uses images to detect obstacles and calculatedistances.
There are two algorithms commonly used in ROS for
object detection and distance calculation. One is KCF [7](Kernelized Correlation Filter), which is a method to find theimage features being tracked and then use this feature to
track the object. Therefore, the operation speed of thismethod can be very fast. However, when obtaining object
features and image tracking, this algorithm will lose theimage data, making it impossible to perform accurate
tracking. The other is DSST [8] (Discriminative Scale Space
Tracker). DSST mainly improves the shortcomings of KCF
that only uses gray scale to obtain object features and does
not calculate the object scale. It makes the judgment ofobjects more accurate and enables distance estimation. The
most important thing is that it will not have a big impact on
the overall calculation amount. Although the above twoalgorithms are sufficient for tracking and following generalmobile robots. However, because this research hopes to use
images to process more complex, even more obstacles andmore targets, the TLD [9] (Tracking-Learning-Detection)
algorithm is also added. This algorithm can accurately
identify the target, even when the object reappears after
being obscured, it can still correctly identify the target. And
if the target object rotates, TLD can also correctly identify it.
Of course, the number of calculations required for thisalgorithm will be relatively large.

Fig.2 Block diagram of KCF+DSST target recognition
system.
In the TLD algorithm part, because the preset
requirements are relatively high for the computing power ofthe system, the calculation will only be performed when
KCF+DSST determines that the object is closer to the mobile
robot. Its block diagram is shown in Fig. 3. Moreover, thefrequency of calculation can be set in the system, such as
once per second, 10 times per second, etc. In addition, themaximum number of tracking can also be set. This prevents
the CPU from consuming too much computing power inobject tracking. Next, we will conduct experimental tests on
the above methods.

Fig.3 TLD target identification system block diagram.

2.1 Research methods

3. Experimental Results

In this study, in order to make ROS have a higher object
recognition rate and calculate a more accurate object
distance, we integrated KCF and DSST. As the main object

We use an x86 computer as the main control core, and thespecifications used are: CPU, Intel® Processor N-series©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
153

Research on dynamic obstacle

N100, 8GB DDR4 DRAM, 512GB SSD. And directly install
Ubuntu, and then install Gazebo and ROS in the system. First,
we first confirm the object detection capabilities of the
KCF+DSST and TLD algorithms. Fig. 4 is our test results on
ROS. We use the same obstacles to identify objects andcalculate scale distances using the KCF+DSST and TLD
algorithms. According to the experimental results, bothalgorithms can operate normally. The operation speed of
KCF+DSST is about 20% faster than the TLD algorithm.
The calculation results of the scale distance between the twoare almost the same.

a.

performed when there are many dynamic obstacles in theenvironment. Of course, the current structure still cannot
completely allow mobile robots to pause. However, based on
the current experimental results, more than 90% of
suspension situations can be avoided. The other 10% arealmost all dynamic obstacles that completely block the pathof the mobile robot. We will also try to solve this part in the
future so that mobile robots can overcome the obstacles ofdynamic emergence and movement in the environment andserve people in real environments.

Objects move away from the robot.
a.

Mobile robot left view.

b. Object approaches the robot
Fig.4 Object tracking and recognition results
But when we dynamically generate multiple objects withdifferent sizes, distances, and positions. Although
KCF+DSST can still find objects quickly. However, when
objects are interlaced or have large displacement or rotation,
KCF+DSST will lose the tracked object. As a result, the riskof collision of mobile robots becomes higher. When using
TLD, this algorithm can quickly and correctly handle theinterlacing of objects, as well as rapid displacement, rotation,etc., and can also correctly calculate the distance of objects.
But when more than 150 objects appear, there will be a shortpause based on the hardware we are using. Although it does
not affect the operation of the system, there will be obvious
pauses. KCF+DSST will only pause for a short time when
there are about 300 objects.

b. Mobile robot right viewFig.5 Mobile robot platform

References
1.
2.

3.

4. Conclusions
This paper uses imaging technology to deal with thedetection problem of multi-obstacle objects, and develops amobile robot platform using an x86 CPU computer. The
actual photo of this platform is shown in Fig. 5. We verified
on ROS that KCF+DSST and the TLD method were used toverify the path re-planning that can be used to solve multidynamic obstacles. That is to say, obstacle avoidance can be

4.

5.

ROS Website, “https://www.ros.org/ “,2023/12/02”
CHEN, Chin S., et al. “Velocity estimation and cost map
generation for dynamic obstacle avoidance of ROS based
AMR.”, Machines, 2022, 10.7: 501.
CHOI, Jaewan; LEE, Geonhee; LEE, Chibum.
“Reinforcement learning-based dynamic obstacle avoidance
and integration of path planning. Intelligent Service Robotics.”,
2021, 14: 663-677.
OX, Dieter; BURGARD, Wolfram; THRUN, Sebastian. “Thedynamic window approach to collision avoidance.”, IEEE
Robotics & Automation Magazine, 1997, 4.1: 23-33.
RÖSMANN, Christoph; HOFFMANN, Frank; BERTRAM,
Torsten. “Timed-elastic-bands for time-optimal point-to-point
nonlinear model predictive control.”, In: 2015 european
control conference (ECC). IEEE, 2015. p. 3352-3357.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
154

Powered by TCPDF (www.tcpdf.org)

Yi-Wei Chen, Jr-Hung Guo, Kuo-Hsien Hsia

6.
7.

8.

9.

Raspberry_Pi
Wikipedia
Website,“https://en.wikipedia.org/wiki/Raspberry_Pi “,2023/12/02
HENRIQUES, João F., et al. “High-speed tracking withkernelized correlation filters.”, IEEE transactions on patternanalysis and machine intelligence, 2014, 37.3: 583-596.
DANELLJAN, Martin, et al. “Discriminative scale spacetracking.”, IEEE transactions on pattern analysis and machine
intelligence, 2016, 39.8: 1561-1575.
KALAL, Zdenek; MIKOLAJCZYK, Krystian; MATAS,
Jiri.,“Tracking-learning-detection.”,IEEE transactions on
pattern analysis and machine intelligence, 2011, 34.7: 14091422.

Authors Introduction
Mr. Yi-Wei Chen
Yi-Wei Chen, a graduate of National
Yunlin University of Science andTechnology, Taiwan, holds anengineering degree with a focus on
robotics-based technologies. His
academic pursuits are deeply rooted in
the exploration and advancement ofrobotic applications.
Dr.

Jr. Hung Guo

He is an Assistant Professor at the
Department
ofElectricalEngineering,
National
YunlinUniversity of Science & Technology,
Douliou, Taiwan. He received his
Ph.D. Degree from National Yunlin
University of Science & Technology,
Taiwan in 2012. His researchinterests include sensor network,
intelligent systems, intelligent robot. and technical andvocational educationDr.

Kuo-Hsien Hsia
He received the Ph.D. degree inelectrical engineering from the
National Sun Yat-Sen University,
Taiwan, in 1994. He is currently an
Associate Pofessor of National
Yunlin University of Science andTechnology in Taiwan. His researchinterests are in the area of mobile
robotics, fuzzy control and image
processing.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
155

Research on Multi-Robot Formation on Two-Dimensional Plane
Kuo-Hsien Hsia

National Yunlin University of Science and Technology, 123 University Road, Section 3,Douliou, Yunlin 64002, Taiwan
Chun-Chi Lai

National Yunlin University of Science and Technology, 123 University Road, Section 3,Douliou, Yunlin 64002, Taiwan
Yi-Ting Liu

National Yunlin University of Science and Technology, 123 University Road, Section 3,Douliou, Yunlin 64002, Taiwan
Yu-Le Chen

National Yunlin University of Science and Technology, 123 University Road, Section 3,Douliou, Yunlin 64002, Taiwan
Email: {khhsia, cclai, M11112017}@yuntech.edu.tw, 1048576yc@gmail.com

Abstract
Mobile robots are playing an increasingly important role in both service and manufacturing industry. The managementof multiple mobile robots is a very important issue on the research of mobile robotics. From a mathematicalperspective, this paper discusses the problem of multiple robots on a two-dimensional plane reaching the designatedpositions in the shortest time to complete formation transformation. We improved the algorithm proposed by Hsia, Li
and Su and proposed a new algorithm using a determinant and the Munkres assignment algorithm. Finally, the newalgorithm is compared with the path distribution obtained by the Monte Carlo method under different numbers ofrobots and the excellence of the new algorithm has been verified.
Keywords: Multiple mobile robot, Pattern formation, Two-dimensional plane, Munkres assignment algorithm.

robot system, there is a control center that makes
decisions and assignments, while the other robots executethe received commands. In this paper, the multi-robot
system is a centralized system and an algorithm is used todetermine the destination that each robot is planning tomove to. There are many types of formations in
discussion, such as wild goose formation, front arrowformation, T-shaped formation, etc. The robots are asked
to transfer their formation from one to another. The
robots could only move laterally or vertically on a plane,so the robot is limited to a two-dimensional grid plane. In
order to allow multiple robots to complete the formationin the shortest time, an algorithm can be used to find theassignment way that makes the formation transformation
successfully in the shortest time.

1. Introduction
In this society where time is money, saving time is an
important factor that must be considered on taskcompletion. How to complete tasks in the shortest time iswhat task assignment issues aim to achieve. Assignment
problem [1] refers to the process of assigning specificwork, tasks or responsibilities to specific people or teams.
In an organization or team, assigning issues is a way ofallocating work to ensure that matters are carried out
efficiently and achieve desired goals. How to make anassignment that the tasks could be completed in the
shortest time has always been the most important issue.
The shortest-path planning problem [2] is to find an
assignment way for shortest-time running. Path planningrefers to the movement path generated in order to enable
one or more objects to complete a certain task in adesignated area. In order to complete the task assignment
problem in the shortest time, it is important to find a way
to have the shortest path. The importance of the algorithm
for this problem is that it can have the solution withhigher efficiency, and the obtained result is repeatable.

2. Preliminary
2.1. Problem description
In the multi-robot formation transformation problem,
there are n robots on a two-dimensional grid plane. The
robots move horizontally or vertically from the starting
positions to the target positions in the same speed. Inorder to find the fastest way to complete the formation
change, we have to decide the target position for every
robot and the robot with the longest distance will beassigned with a fastest way. This is a min-max problem.

This paper discusses the assignment problem offormation change of multiple robots. For a centralized©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
156

Kuo-Hsien Hsia, Chun-Chi Lai, Yi-Ting Liu, Yu-Le Chen

2.2. L1-norm between two points

corresponding position in the decision matrix is filledwith prime numbers from small to large, and the rest are
not searched. The reached position is represented by 0 in
the judgment matrix. When the search is completed, findthe determinant of the judgement matrix. If the result is 0,
add 1 to the search value and search again. Repeat theabove steps until the determinant value is not 0, and thenuse the Munkres assignment algorithm to assign tasks.

The L1-norm between two points is defined by the total
distance of the projection of the line segment formed byeach point on the axes on the Cartesian coordinate system,that is, the horizontal distance between the two points
plus the longitudinal distance between them. Suppose thetwo points are at (𝑥1 , 𝑦1 ) and (𝑥2 , 𝑦2 ), their distance is|𝑥1 − 𝑥2 | + |𝑦1 − 𝑦2 |.

3.2. The algorithm
2.3. Determinant
Step 1: Calculate the L1-norm of all starting points totarget points, and form the results into a matrix
named distance matrix. The columns of the matrix
correspond to the starting points, and the rows
correspond to the target points.
Step 2: Initialize the judgement matrix of same dimension
as distance matrix with all entries 0.
Step 3: Set the search value as 0.
Step 4: Fill in prime numbers in sequence to thecorresponding position in the judgment matrix
where the entry of the distance matrix is equal to thesearch value. The selection of prime numbers starts
from 2, and each prime number filled in is thesmallest prime number larger than the previous one.
Step 5: If the determinant of the decision matrix is not 0,
jump to step 7.
Step 6: Increase the search value and back to step 4.
Step 7: Replace all positions of the distance matrix by asufficiently large number, e.g. 999, if thecorresponding position of the judgment matrix is 0.
Step 8: For a distance matrix, subtract the minimum valueof that row from each row and subtract theminimum value of that column from each column.
Thus at least one element in each row and eachcolumn will be zero.
Step 9: Use the least straight line to pass through all 0s,
so that each 0 has at least one line passing throughit.
Step 10: When the number of lines equals the number ofrobots, the position of 0 in the matrix can be used
for task assignment. Starting from the row with thefewest 0s, only one 0 can be circled in each row andcolumn. If the number of circles is equal to thenumber of robots, the assignment of the task iscompleted. If not, go to the next step.
Step 11: Subtract the minimum value in the undrawn row/column, and add the minimum value to theintersection position of the lines. After completion,back to step 9.

To complete the assignment problem of n robots, eachrobot at his starting position will have a corresponding
target point position. When all robots are assigned, theircorresponding relationships will be presented in the
distance matrix, and unassigned positions can be filledwith 0. Under such circumstances, there will be n nonzero values in the distance matrix on different rows andcolumns, forming a square matrix of full rank. The fastest
way to check for full rank is to check whether its
determinant is 0 or not. Therefore, we use the determinant
as a tool for checking the feasibility of formationtransformation planning.
2.4. Munkres assignment algorithm
Munkres assignment algorithm [3] is an optimization
algorithm for solving assignment problems. Given a costor benefit matrix, it finds a set of assignments thatminimize the total cost or maximize the total benefit, inwhich each row and column only can be assigned once.
2.5. Monte Carlo method
The Monte Carlo method is a numerical calculation
method based on random sampling and statisticalsimulation [4]. When assigning tasks using the Monte
Carlo method, one can randomly find an allocation way,
and then find the maximum of the solution. One can get
the number of steps for completing the tasks by using theMonte Carlo method one time. The more times ofsampling, the more assignment ways there will be. By
minimizing the maximum value of these, one can get aassignment way almost the most suitable.
3. Proposed new algorithm
3.1. Deviation of the new algorithm
We refer to the algorithm proposed by Hsia, Li and Su[5], but instead of finding the assignment way from theentries of the distance matrix, we find the minimum pathfrom the positional relationship of the values in the matrix,
and use the determinant to make the decision. The
program searches sequentially from 0, and fills thesearched position into another decision matrix. When avalue is searched in the distance matrix, the

4. Comparison to Monte Carlo method
This comparison was performed on Matlab 2021,
simulating the formation transformation situations of 10,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
157

Research on Multi-Robot Formation

30, 50, 100, 500 and 1000 robots respectively. The
starting and target points of the robots are randomly
generated each time. Then the proposed algorithm and theMonte Carlo method are used for task assignment. The
Monte Carlo method takes the best allocation method outof 10,000 times as the representative value. Aftercomparison, the smaller one is the better assignment in
the simulation. For each number of robots, we have 1000
simulations for both methods. This comparison index is
to count the number of simulations that the proposed
algorithm's result is not greater than the Monte Carlo
method's result in 1000 simulations with different
numbers of robots. The simulation results are
summarized in Table 1. It is clear from Table 1 that the
proposed algorithm can always have the same or better
assignment way. Hence the proposed algorithm is better
than Monte Carlo method for the formationtransformation problem.

6. Conclusion
This paper proposed a new algorithm to solve the task
assignment problem for a multi-robot system on
formation transformation. Each mobile robot is limited tohorizontal and vertical movement and starts at the sametime. We’d like the transformation to be completed in the
shortest time. The proposed algorithm can be used tosystematically find out how to assign the target points ofeach robot so that the robot's formation can betransformed in the shortest time. Unlike previous works,
we use determinant of a matrix to find out the shortest
time required to complete the formation change, and use
the Munkres assignment algorithm to obtain the shortest
time. This work can ensure the best assignment result.

Table 1. Comparison results.
Number of robots
1030
50
100 500 1000
Number of the resultsof proposed algorithm
1000 1000 1000 1000 1000 1000
not worse than Monte
Carlo method

5. Illustrated Examples(a) flying goose formation

This paper takes three formation transformations as
examples, namely the flying goose formation, the front
vector formation and the T-shaped formation, andMATLAB is used for simulation. The robot is located on
a 7x7 grid plane. The configuration of each formation isshown in Fig. 1. For the flying goose formation (V shape),
the robots are at (4, 1), (4, 7), (5, 2), (5, 6), (6, 3), (6, 5),
and (7, 4). For the front vector formation (Q shape), therobots are at (2, 4), (3, 4), (4, 4), (5, 4), (6, 3), (6, 5), and(7, 4). For the T-shaped formation (T shape), the robots
are at (2, 1), (2, 2), (2, 3), (2, 4), (2, 5), (3, 3), and (4, 3).

(b) front vector formation

The 7 robots form the flying goose formation initially.
We would like to make the robots changed to the front
vector formation, and then changed to the T-shaped
formation. The assignment results are summarized inTable 2, and the routes for the robots are shown in Fig. 2.

Table 2. The assignment results.
Pts of V-shape (4, 1) (4, 7)
Pts of Q-shape (2, 4) (4, 4)Distance for #1
5
3
Pts of T-shape (2, 1) (2, 2)
Distance for #2
3
4
Pts of V-shape (4, 1) (5, 2)Distance for #32
3(5, 2) (5, 6)(3, 4) (5, 4)
4
2(2, 3) (2, 4)
2
3(6, 3) (4, 7)
4
5(6, 3) (6, 5)(6, 3) (6, 5)
00(3, 3) (2, 5)
3
4(5, 6) (6, 5)
54(c) T-shaped formation(7, 4)(7, 4)
0(4, 3)
4(7, 4)
4

Fig. 1. Formations considered in this paper©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
158

Powered by TCPDF (www.tcpdf.org)

Kuo-Hsien Hsia, Chun-Chi Lai, Yi-Ting Liu, Yu-Le Chen

References
Authors Introduction
1.

2.

3.

4.

5.

R. Burkard, M. Dell’Aamico and S. Martello, Assignment
Problems, Society for Industrial and Applied Mathematics,
2012.
W. F. Chiang, Realization of the rectilinear shortest path
planning problems, Master Thesis, National Taipei
University, 2022.
F. Bourgeois and J. C. Lassalle, An extension of the
Munkres Algorithm for the assignment problem torectangular matrices, Communications of the ACM,
14(12), 802-804, 1971.
N. Metropolis and S. Ulam, The Monte Carlo Method,
Journal of the American statistical association, 44(247),
335-341, 1949.
K. H. Hsia, B. Y. Li and K. L. Su, Task assignment on gridplane for multiple mobile robots, Tenth Int. Conf. on
Innovative Computing, Information and Control(ICICIC2015) (Dalian, China, 2015), 193, 2015.

Dr. Kuo-Hsien Hsia
He received the Ph.D. degree inelectrical engineering from National
Sun Yat-Sen University, Taiwan. He
is currently an Associate Pofessor of
National Yunlin University of
Science and Technology in Taiwan.
His research interests are in the areaof robotics, fuzzy systems, intelligent
control. He is a member of IEEE.
Dr. Chun-Chi Lai
He received the Ph.D. degree inelectrical engineering from National
Chung Cheng University, Chiayi,
Taiwan, in 2014. He is currently an
Assistant Professor of National
Yunlin University of Science andTechnology in Taiwan. His researchinterests include multisensor fusion
and intelligent robotics.
Mr. Yu-Le Chen
He received his master's degree inelectrical engineering from National
Yunlin University of Science andTechnology, Taiwan in 2023. He iscurrently a student in the
Government AI Career Training
Institute, Taiwan.

(a) flying goose formation -> front vector formation

Miss Yi-Ting Liu
She received her bachelor's degree inelectrical engineering from National
Chin-Yi University of Technology in
Taiwan in 2022. Currently, she is amaster's student at National Yunlin
Universityof
ScienceandTechnology.

(b) front vector formation -> T-shaped formation(c) T-shaped formation -> flying goose formation

Fig. 2 Formation transformation simulation©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
159

The Development of Utilization Rate and Energy Consumption Monitoring and Networking
System for Old Machine
Chung-Wen Hung, Chun-Chieh Wang, Heng-En Chang

Nation Yunlin University of Science and Technology,
123 University Road, Section 3, Douliou, Yunlin 64002, Taiwan, R.O.C.
Chau-Chung SongNational Formosa University,
64 Wunhua Road, Huwei Township, Yunlin County 632, Taiwan, R.O.C.
Email: wenhung@yuntech.edu.tw, jasonccw@yuntech.edu.tw, m11212047@yuntech.edu.tw, ccsong@nfu.edu.tw

Abstract
In this paper, we present a system based on a microcontroller unit (MCU) for measuring the utilization rate of
traditional, non-networked machinery. This equipment is designed for use with older machines equipped with andon
lights. It employs optocoupler circuits to capture the status of these lights and current transformers to measure theiroperating and standby currents. Data is transmitted to the server using the Hypertext Transfer Protocol (HTTP) in
JavaScript Object Notation (JSON) format. On the server side, a Hypertext Preprocessor (PHP) interprets the data,
connects to a Structured Query Language (SQL) database, and stores the data using SQL commands. Users can accessgraphical data through a web-based interface, using it to refine production processes, reduce production costs, andminimize carbon emissions.
Keywords: Utilization, Non-networked machinery, Andon lights, Current transformers.

heightened concern. The determination of product carbon
footprints and the subsequent implementation of carbon
reduction labels are tasks to be addressed in the future.

1. Introduction
With the widespread adoption of Industry 4.0 concepts
and the concurrent advancement of relevant technologies
such as networking and microchips, the Internet ofThings (IoT) has emerged as a globally esteemedparadigm. The internet of machines refers to the abilityof machines and devices to interact with the internet
without human intervention. Information from devices,
including operational progress, work duration,
production history, and more, is transmitted via theInternet to servers for recording and analysis. The data
facilitates analyses such as production process andutilization rate, allowing enterprises to adjust production
processes based on this data. However, these actions are
typically limited to new generations of IoT-enabled
devices, as traditional machinery lacks such capabilities.

This paper aims to assist factories in recording changes
in andon light signals and corresponding time-related
variations in electrical currents without intrusively
altering traditional machinery's existing hardware andsoftware. This is achieved by incorporating IoT devices
to facilitate the documentation of machine operatingstates, thereby reducing downtime and enhancing overall
operational efficiency. In addition, it also provides
relevant information on the product's carbon footprint,aligning with the demands of Industry 4.0 and theimplementation of carbon taxes.
2. Related Work2.1. Utilization

On the other hand, environmental protection has
become a topic of global concern in recent years. Nations
worldwide recognize the severity of greenhouse gas
emissions and the importance of reducing carbon
footprints. Governments have initiated carbon fees andtax policies, with the European Union passing a Carbon
Border Tax set to take effect in 2026 [1]. This policy
targets high carbon-emitting industries, imposing
restrictions and charges. For enterprises, maintaining or
reducing carbon emissions has become a matter of

The production or manufacturing process involves
transforming raw materials or components into finishedproducts. In the industrial context, production lines canbe categorized into three types: automated productionlines, semi-automated production lines, and manual
production lines. The nature of a production line depends
on the complexity of the finished product, production
qunatity, and costs. Enterprises can plan and lay out
production lines based on their requirements [2].

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
160

All Author’s Full Name

As highlighted by both [2], [3], inefficient working
hours on the production line primarily stem from machine
waiting times for personnel handling, waiting for materialchanges, troubleshooting, and maintenance. The
efficiency metric used is referred to as "utilization rate."
The utilization rate indicates the percentage of the total
working hours where the machine is under load during
effective working hours. Effective personnel
management plays a crucial role in improving thefactory's utilization rate.

information provided by the machine. The connectivity
layer deals with how the collected information istransmitted to the server for processing. The data layeremphasizes planning the format of machine data andstoring it in a database for subsequent applications andanalysis.
2.3. Carbon credit issues
With global industrialization and increased energy
consumption, there is a significant increase in carbon
dioxide emissions, contributing to climate challenges.
Addressing these issues requires international
cooperation to reduce and control carbon emissions. The
European Parliament has taken legislative action this year,
entering a transitional phase from October. They are
implementing the European Union Carbon Border
Adjustment Mechanism (CBAM), which restricts theimport of high-carbon products such as steel, cement, andelectricity. It will be mandatory for imported products
falling into supervised categories to purchase CBAM
certificates [1] after 2026.

2.2. Industry 4.0 and the Internet of Things (IoT)
The term Industry 4.0 emerged in Germany in 2011,referring to the technology used in the manufacturing
process to monitor processes and employ data-based
techniques for predicting, correcting, and adaptively
adjusting production strategies [4]. Its objective is tointegrate various data, tools, and processes to analyze andreduce costs, mitigate risks, and enhance efficiency [5],[6], [7]. Unlike previous industrial revolutions, Industry
4.0 is not based on a single technology. Instead, it is aconvergence of modern technologies such as Big Data,
the Internet of Things (IoT), Cloud Computing, andArtificial Intelligence (AI) [5].

3. System Architecture
The Internet of Things (IoT) applications have deeply
penetrated our lives, with interconnected IoT devices
sharing vast amounts of information without humanintervention. According to [8], global IoT devices online
in 2020 totaled 9.76 billion, with application areas
illustrated in Fig. 1. Of these, 22% were applied in theindustrial sector, 15% in transportation, and 14% inenergy. By 2026, the number of online devices is
projected to reach 21.09 billion; by 2030, it is estimatedto approach nearly 30 billion devices.

In this project, the collaborating partners are concerned
about the machinery's operational status and energy
consumption. However, due to the large number ofmachines, totaling 27 units with over ten different models,some of the actual machine appearances are depicted inFig. 2. Embedded systems, coupled with sensors, will beutilized to extract information on machine status andenergy consumption. This data will be wirelesslytransmitted to a server for storage, facilitating subsequent
analysis and processing.

Fig. 1 The top 10 IoT application areas in 2020 [9]This project aims to transform traditional machinery byfurther integrating it into the framework of Industry 4.0
to enhance machine utilization rates and production line
efficiency. The referenced Internet of Machine
Architecture model, as established in [10], [11],comprises five layers: the physical layer, sensor layer,
connectivity layer, data layer, and application layer. The
machinery transformation in line with Industry 4.0
primarily focuses on configuring the sensor, connectivity,and data layers. In the sensor layer, determining required
machine states is crucial, involving establishingcorresponding sensors and utilizing the production

Fig. 2 The actual appearance of the machinery
As illustrated in Fig. 3, the network architecture
primarily consists of an MCU and a server. When thesensor reads values and transmits data to the MCU, theMCU packages the data in a predefined format.
Subsequently, using the built-in Wi-Fi module, the datais transmitted to the server. Specific PHP (Hypertext
Preprocessor) programs perform data analysis within theserver and upload it to the MySQL database. Upon

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
161

Paper Title (First 4 Words)

completion of the upload, users can view all the data asof a webpage or export the data into reports throughWhere 𝑉𝐴𝑅𝐸𝐹 denotes analog reference voltage, CT
TURNS represents the current transformer turns ratio,

Fig. 3 The network architecture diagram

Fig. 5 Current measurement circuit diagram(measurement part) [13]and 𝐼𝑝𝑚 denotes the maximum current on the primaryside. By generating a voltage across the burden resistor
through the secondary side current, the capacitor in Fig.
5 can accurately represent the direction and magnitude of
the current. This information is acquired through the
MCU's Analog-to-Digital Converter (ADC).

additional programming.
4. System Development
4.1. Andon light status detection
The core component of the Andon light status
detection circuit is component U1 in Fig. 4, which is a
optocoupler element. A optocoupler element is a device
that uses light (such as visible light or infrared) as a
medium to transmit electrical signals [12]. As previously
mentioned, different machine models on the production
line may use different electrical configurations forwarning lights, distinguishing between AC and DC.
Therefore, the PC814 bidirectional photocouplingelement manufactured by Sharp, a Japanese electronics
company, has been selected for development for thisproject. The main difference lies in the internal structureof the PC814, which includes a pair of light-emitting
diodes — one forward and one reverse — enabling it tofunction in both directions of current flow. It elegantly
resolved the compatibility issue between alternating
current and direct current.

4.3. Data format and transmission
Upon receiving the status and data from peripheraldevices, the MCU must analyze and upload theinformation to the server. The transmitted data should
include details such as machine ID, data type, andtimestamp. JSON (JavaScript Object Notation) can be
employed as the data packaging format. JSON is alightweight data interchange format supported by mostprogramming languages, often with additional libraries
for convenient packaging and parsing. The specificpackaging format for this project is illustrated in Fig. 6.

Fig. 4 Andon light status detection circuit diagram(detection part)4.2. Current Measurement
Fig. 6 Data format
This project utilized the SCT013 split-core current

The data, transmitted through HTTP request by the
MCU with Wi-Fi functionality, undergoes analysis on theserver side using PHP. PHP also responds to the HTTP
request and uploads the data to the database for storage.
The server-side data transfer path is illustrated in Fig. 7.

transformer manufactured by YHDC to avoid damaging
the original power wiring. The circuit structure employedis illustrated in Fig. 5 with the calculation method for theburden resistor according to Eq. (1).
Burden Resistor(Ω) =𝑉𝐴𝑅𝐸𝐹 ∗ 𝐶𝑇 𝑇𝑈𝑅𝑁𝑆
2√2 ∗ 𝐼𝑝𝑚

(1)

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
162

All Author’s Full Nameinternational carbon policies, helping to minimize
carbon-related costs and reduce unnecessary energy
consumption during idle times.
6. Conclusion
This article utilizes a microcontroller with wirelessconnectivity, sensors, and electronic components tocreate a non-invasive device. Combined with a web app,factory administrators can swiftly gather production
information to analyze and adjust production strategies.

Fig. 7 The server-side data transfer path
5. ResultThis paper uses the ESP32 series MCU, which
Espressif Systems developed as the core for edge-side
development. Its advantages lie in its cost-effectivenessand the integration of network modules with basic
microprocessors. This MCU is commonly employed in
the development of IoT applications.

References[1] European Commission, Carbon Border Adjustment
Mechanism,
TaxationandCustoms
Union,
https://taxation-customs.ec.europa.eu/carbon-borderadjustment-mechanism_en (accessed Nov. 15, 2023).
[2] SKAL Subramaniam, Siti Huzaimah Binti Husin,
Yusmarnita Binti Yusop, and Abdul Hamid Bin Hamidon,
Machine efficiency and man power utilization on
production lines, in WSEAS International Conference.
Proceedings. Mathematics and Computers in Science andEngineering, vol. 7, 2008, World Scientific andEngineering Academy and Society.
[3] Y. Y. Wang, J. Lionel, Case Study of the Machine
Utilization on the Turning Process,” in Industrial Scienceand Management Journal, Mar. 31, 2008.
[4] T. Ainsworth, J. Brake, P. Gonzalez, D. Toma, and A. F.
Browne, A comprehensive survey of Industry 4.0, IIoT,
and areas of implementation, SoutheastCon 2021*, pp. 16, 2021, IEEE.
[5] H. Khujamatov, E. Reypnazarov, D. Khasanov, and N.
Akhmedov, IoT, IIoT, and cyber-physical systems
integration, in Emergence of Cyber Physical System and
IoT in Smart Automation and Robotics: Computer
Engineering in Automation, Springer, 2021, pp. 31-50.
[6] K. Wójcicki, M. Biegańska, B. Paliwoda, and J. Górna,Internet of Things in Industry: Research Profiling,
Application, Challenges and Opportunities—A Review,
Energies, vol. 15, no. 5, p. 1806, 2022, MDPI.
[7] R. Y. Zhong, X. Xu, E. Klotz, and S. T. Newman,
Intelligent manufacturing in the context of Industry 4.0: areview, Engineering, vol. 3, no. 5, pp. 616-630, 2017,
Elsevier.
[8] L. S. Vailshery, IOT connected devices worldwide 20192030,
Statista,
https://www.statista.com/statistics/1183457/iotconnected-devices-worldwide/ (accessed Jun. 15, 2023).)[9] M. H. Alsharif, A. Jahid, A. H. Kelechi, and R.
Kannadasan, Green IoT: A Review and Future Research
Directions, in Symmetry, 2023, vol. 15, p. 757.
[10] D. Jaspert, M. Ebel, A. Eckhardt, and J. Poeppelbuss,
Smart retrofitting in manufacturing: A systematic review,
Journal of Cleaner Production, vol. 312, p. 127555, 2021,
Elsevier.
[11] P. Zheng, Z. Wang, C.-H. Chen, and L. P. Khoo, A surveyof smart product-service systems: Key aspects, challenges
and future perspectives, Advanced Engineering
Informatics, vol. 42, p. 100973, 2019, Elsevier.
[12] Opto-Isolator,
Wikipedia,
https://en.wikipedia.org/wiki/Opto-isolator (accessed Dec.
4, 2023).

After collecting data, present it to users in a graphicalformat for more straightforward interpretation andanalysis. The system is developed as a Web App to avoid
the need for different applications on various devices andthe authentication thresholds of mobile applications. The
advantages of a Web App lie in its accessibility—any
internet-connected device with basic browser
functionality can use it. The web page is coded using
HTML (Hypertext Markup Language) for interface
design. JavaScript is employed for real-time content
updates and user interaction. PHP serves as anintermediary between JavaScript and the database fordata requests.

Fig. 8 Web App
Fig. 8 shows the interface displayed in the web app,
divided into two main sections. Users can choose themachine number and time of interest on the left-hand sidein the blue area. After selection, the data for the chosenmachine will be displayed on the right-hand side in thewhite area. Each row in the chart on the right represents
the data for a single machine on a given day. The various
colored blocks represent the machine's status (indicatorlights) during specific time intervals, while the lines
depict the machine's current consumption. Analyzing thisdata assists factories in power consumption statistics,
emission calculations, and formulating strategies foremission reduction. Aligns with national and©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
163

Powered by TCPDF (www.tcpdf.org)

Paper Title (First 4 Words)[13] CT sensors - interfacing with an arduino, CT Sensors Interfacing with an Arduino - OpenEnergyMonitor 0.0.1
documentation,
https://docs.openenergymonitor.org/electricitymonitoring/ct-sensors/interface-with-arduino.html(accessed Dec. 4, 2023).

Authors Introduction
Prof. Chung-Wen HungHe received the PhD degree inElectrical Engineering from National
Taiwan University in 2006.
Currently, he is a professor at
National Yunlin University of
Science and Technology. His
research interests include IoT, IIoT,
power electronics, motor control,and AI application.
Chun-Chieh Wang
He received the PhD degree in The
graduate school of engineering scienceand technology from Nation Yunlin
University of Science and Technology.
Currently, he is a project professor at
National Yunlin University of Scienceand Technology.

Heng-En Chang
He received the B.S. degrees and now
he is studying for the M.S. degree inelectrical engineering from National
Yunlin University of Science andTechnology.

Chau-Chung SongHe received the Ph.D. degrees inElectrical Engineering from National
chiao tung university. Currently he is
a Distinguished professor and Chairin National Formosa University.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
164

MCU Based Edge Computing Platform for Liquid Level Measurement
Chung-Wen Hung, Chun-Liang Liu, Tai-Hsuan Wu
Nation Yunlin University of Science and Technology, Taiwan

123 University Road, Section 3,Douliou, Yunlin 64002, Taiwan, R.O.C.
Chau-Chung SongNational Formosa University, Taiwan

64 Wenhua Road, Huwei Town, Yunlin County 632301, Taiwan, R.O.C.
Email: wenhung@yuntech.edu.tw, clliu@yuntech.edu.tw, m11112080@ yuntech.edu.tw, ccsong@nfu.edu.tw

Abstract
An edge computing system based on micro control unit (MCU) for liquid level measurement is proposed in this paper.
The system includes a solenoid electromagnet for bottle hit and a microphone to capture sound waves. The signals
are converted from time domain to frequency domain by Fast Fourier Transform (FFT), employing an artificial
intelligence (AI) model to predict the water level. Artificial Neural Network (ANN) model is applied for classificationon the MCU. When optimizing hyperparameters, the accuracy of each parameter combinations should be considered.
Ensure the model size suits the limited MCU memory and computing capabilities. The experimental results can be up
to 99% accurate under multiple tests.
Keywords: Artificial Neural Network, Hyperparameters, Edge Computing, Audio Process

During the training stage, the proposed system offers a data
sampling function, where the gathered information istransmitted to a personal computer (PC). The hyperparametersoptimization and model training processes are conducted on the
PC. Once completed, the trained model will be installed onto
the MCU, enabling edge computing for measurement tasks
performed by this MCU.

1. Introduction
As artificial intelligence (AI) is more popular, there are many
applications to be proposed no matter for the consumer orindustrial market. Moreover, it is crucial to tackle the emerging
challenges associated with edge computing [1], a topic of greatsignificance and current prominence. The utilization of
endpoints for model computation has attracted considerable
attention, given its potential to reduce network latency andlower costs [2], [3]. Furthermore, the exploration of alternative
AI models for measurement becomes pivotal in situations where
sensor availability is constrained or cost prohibitive. When acontainer, such as bottle, experiences physical impact, it
produces distinct tones depending on the liquid's varying levels
inside, owing to vibration frequencies. Determining the waterlevel through sound analysis, as presented in [4], relies on
specific vessel geometries. However, employing an AI modelor sound-based water level measurement becomes a viableoption. Furthermore, given the limited resources of the endpoint
platform, careful consideration of AI model selection andhyperparameter optimization for edge computing is crucial.
Artificial neural networks (ANN) model, is simple yet effective
for classification tasks, represent a viable candidate for thisstudy [5], [6]. Moreover, Bayesian optimization emerges as apopular method for efficient hyperparameter optimization,offering convenient and efficient.

2. Methodology
2.1. SYSTEM STRUCTURE
The proposed system structure is shown in Fig. 1. Two
micro switches are used to trigger procedure, which
includes knocking, sampling, and edge computing.
Moreover, these two switches also could make sure thatthe hitting situation, such as position and distance. MCU
is the center of the system, and controls the full procedure,as mentioned previously, which also provide the edge
computing. The UART (Universal Asynchronous
Receiver/Transmitter) protocol is adopted forcommunication with PC, the sound sampling is sent to
PC for training from MCU.

A micro control unit (MCU) based edge computing system for
liquid level measurement is proposed in this paper, and all
calculation and control including data acquisition are performed
by a consumer MCU, Artery AT32F415 [7]. The proposed AI
model utilizes one-dimensional sound information forclassification, resulting in reduced memory requirements.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
165

Chung-Wen Hung, Chun-Liang Liu, Tai-Hsuan Wu, Chau-Chung SongIn order to prevent the data from being affected byabnormal waveforms and to enhance the features, thesystem adopts Min-Max Normalization for each data.
The Normalization finds the maximum and minimum
values of the sound frequency, and then converts the
maximum value is set to 1, and the minimum value isadjusted to 0. Finally, all audio data are scaled to therange between 0 and 1. The calculation formula is as (1) :𝑋𝑛𝑜𝑚 =

Fig. 1 The system diagram
A capacitive microphone is applied to convert soundto voltage, which amplitude is amplified and offset isshift by operational amplifier (OPA) [8] circuit for the
input range of the Analog to Digital Converter (ADC),
which is a built peripheral function of MCU. The knockcomponent, a solenoid electromagnet, is driven by HBridge [9] to eject and attract.

𝑋 − 𝑋𝑚𝑖𝑛𝑋𝑚𝑎𝑥− 𝑋𝑚𝑖𝑛

(1)Here, 𝑋 is the original data, 𝑋min and 𝑋𝑚𝑎𝑥 are theminimum and maximum value of the frequency domain
data, and 𝑋_𝑛𝑜𝑚 indicates the normalized data infrequency domain. The waveforms after data preprocessing are shown in the Fig. 3.

2.2. DATA ANALYSIS AND REPROPCESSING
Before algorithm development, the data analysis isnecessary. As mentioned previously, the converted
sampling data are sent to PC for analysis and training via
UART. The audio voltage is sampled at 10k Hz, and thenumber of sampling points is 2048 per trigger. The time
domain waveforms of the different water levels areshown in Fig. 2 and there is not any saturation andtruncation, it means that the sampling rate and length aresuitable for this application. To develop the algorithm,
the Fourier transform converts waveform data in the time
domain into the frequency domain as shown in Fig. 3.
Note, the frequency of the first spike, which decreases
with increasing water volume, this phenomenon isconsistent with the sound heard. Frequency information
only from 83Hz to 1464.8Hz is adopted to identify waterlevel. Furthermore, the frequency domain differences
between different water levels are obvious, and theclassification function could be performed by theartificial neural networks (ANN) model, which is simple
and efficient for categorization. Compared to most otherAI models, ANN requires less memory and calculation
resource, in other words, it is suitable to be adopted into
the simple edge-computing platform, MCU. So, Fouriertransform (FFT) is selected for pre-processing, and ANN
is applied to classify the water level.

Fig. 3 Audio signals at different water level
2.3. HYPERPARAMETER OPTIMIZATION
Hyperparameter [10] is an important issue in deeplearning algorithm as it determines the size and accuracyof AI model by selecting the number of hidden layers, thenumber of nodes and activation function in each layer.
Typically, the hyperparameter combination that achieve
the highest accuracy will be chosen to be implemented in
the MCU. However, in this system, the implementationof edge computing is also a crucial consideration.
Therefore, while ensuring the accuracy requirements aremet, it is necessary to select a parameter combination thatis suitable for the resource constraints.
Bayesian optimization [11] is a modelling method basedon surrogate function that is used to optimizing thehyperparameter in the proposed system. It employs
Gaussian process to model and updates the acquisition
function based on the model’s results, continually
exploring the uncertain regions. As the number ofsearching increases, the surrogate function will be closer
to the distribution of real function. Table 1 displays theresults by hyperparameter search, sorted by the modelsize, number of hidden layers, number of nodes in eachlayer, and the corresponding model size and the accuracy
rate Considering the memory limitation of the MCU, thefirst model which satisfies the accuracy requirements

Fig. 2 Audio signals at different water level©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
166

MCU Based Edge Computing

while also relatively small memory requirements isselected as the training parameters for this system.

After installing the model into the MCU, the liquid
level can be predicted. The performance of the AI model
is often assessed using the confusion matrix, which
categorizes predicted results into four classes: true
positive (TP), false positive (FP), true negative (TN), andfalse negative (FN). These metrics are used to calculate
various performance indicators, including accuracy as
defined in formula (5), true positive rate (TPR), false
positive rate (FPR), true negative rate (TNR), and false
negative rate (FNR).
In this paper, a multicategory model is proposed as thebasis of analysis. Fig. 6 illustrates the index calculation
for the '100ml' category when there are five output
categories. If the predicted result matches the true label('100ml'), it is labeled as C_1,1 (𝑇𝑃). Here, the symbol 𝐶
represents a case, and C_(i,j) denotes the case in the i-th
row and j-th column of a confusion matrix. Therefore,
C_1,1 represents the case in the 1st row (true label:'100ml') and 1st column (predicted label: '100ml') of a
confusion matrix. TPR represents the probability that apositive sample is correctly identified as positive, while
FPR is the probability that a negative sample isincorrectly classified as positive. TNR is the probability
that an actual negative sample is correctly identified as
negative, and FNR is the probability that a positive
sample is incorrectly identified as negative. A complete
list of formulas is provided in the following sections (2)to (6).

Table 1. HYPERPARAMETER BY BAYESIANOPTIMIZATION RESULTS
HiddenLayer(s)
2
2
2
3
4
4

Node
1
5
5
5
5
32
32

Node
2
5
7
16
8
32
32

Node
3
0
0
0
8
5
5

Node
4
0
0
0
0
8
16

Size
(kB)
49.75
49.77
50.75
57.53
172.50174.50

acc
0.99
0.98
0.99
0.99
0.99
0.80

After confirming the hyperparameters, the collected
data and hyperparameters are utilized to train the AI
model. The MCU code includes a knocking driver, ADC
sampling, and pre-processing functions. Subsequently,
the execution program is installed on the MCU. The
execution time of the program, as shown in Table 2 isonly 0.71 seconds for the entire process, including data
sampling, pre-processing, and prediction, as illustrated inFig. 4. Regarding MCU memory usage, the proposedsystem requires only 125.8 KB of read-only memory(ROM) and 28.8 KB of random-access memory (RAM).
This system can be implemented in most of the memory
management systems available in the market.
Table 2. PROGRAM EXECUTION TIME
Time(s)Sampling
0.20

Pre-processing
0.22

Prediction
0.014

All
0.71

Fig. 4 Memory usage at MCU
3. Results and Discussion
The edge-computing platform proposed in this paper,
which is include MCU, microphone, and solenoid
electromagnet is shown in Fig. 5. In the picture, the micro
switches on both sides of the solenoid valve ensure thatthe former can knock the middle of the bottle correctly

Fig. 6 Categories of Confusion matrix
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 =

TPR =

𝑇𝑃
𝐶100,100=
𝑇𝑃 + 𝐹𝑁
𝐶100,100 + ∑500
𝑘=200 𝐶100,𝑘

FPR =
=

𝑇𝑃 + 𝑇𝑁𝑇𝑃 + 𝐹𝑁 + 𝐹𝑃 + 𝑇𝑁

𝐹𝑃
𝐹𝑃 + 𝑇𝑁∑500
𝑘=200 𝐶𝑘,100

500
500
∑500
𝑘=200 𝐶𝑘,100 + ∑𝑖=200( ∑𝑗=200 𝐶𝑖,𝑗 )𝑇𝑁
TNR =𝐹𝑃 + 𝑇𝑁500∑500
𝑖=200( ∑𝑗=200 𝐶𝑖,𝑗 )
= 500
500
∑𝑘=200 𝐶𝑘,100 + ∑𝑖=200( ∑500
𝑗=200 𝐶𝑖,𝑗 )Fig. 5 Edge-computing platform

FNR =

∑500
𝐹𝑁
𝑘=200 𝐶100,𝑘=
𝑇𝑃 + 𝐹𝑁
𝐶100,100 + ∑500
𝑘=200 𝐶100,𝑘©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:CO HorutoHall, Oita, Japan, 2024
167

(2)

(3)

(4)

(5)

(6)

Chung-Wen Hung, Chun-Liang Liu, Tai-Hsuan Wu, Chau-Chung Song

Sensing for a Pressurized Cylinder. Journal of Sensors.
5027916 (2016).
5. J. J. Popoola, and R. V. Olst: Development andComparative Study of Effects of Training Algorithms on
Performance of Artificial Neural Network Based Analog
and Digital Automatic Modulation Recognition. Journal ofEngineering Science and Technology Review 8 (4), 135144(2015).
6. J. J. Popoola, and R. V. Olst: Automatic recognition of
analog modulated signals using artificial neural networks.
Journal of Computer Technology and Applications, Vol. 2,
No. 1, pp. 29-35(2011).
7. ARTERY Technology, AT32F415S DATASHEET,
JUNE. 2022[V2.01].
8. Texas
Instruments,:
OPA1612ADATASHEET,March,2015[Rev.c].
9. ASIC, L9110H DATASHEET, L9110H Datasheet.
10. J. Bergstra, D. Yamins, and D. Cox: Making a Science ofModel Search: Hyperparameter Optimization in Hundredsof Dimensions for Vision Architectures. Proceedings ofthe 30th International Conference on Machine Learning,Feb. 2013, pp. 115–123.
11. B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, and N.
de Freitas: Taking the Human Out of the Loop: A Review
of Bayesian Optimization. Proceedings of the IEEE, vol.
104, no. 1, pp. 148–175, Jan. 2016.

The Fig. 7 confusion matrix summarizes the actual andpredicted classifications of online testing, and theconfidence values of each liquid level are in line withexpectations. As a result, 95% accuracy when calculating
the true negative value of each water level, and 99%
accuracy rate when calculating the full water levels.
The results of this study can be used to verify thefeasibility of the proposed system as well as the discussed
artificial intelligence model.

Authors Introduction
Fig. 7 Confusion matrix of the model

Prof. Chung-Wen HungHe received the Ph.D. degrees inElectrical Engineering from National
Taiwan University in 2006. Currently heis a Professor in National Yunlin
University of Science &Technology.
His research interests include the IoT,
IIoT, and AI application.

4. Conclusion
The paper proposes a MCU-based edge computing
platform for liquid level measurement. This includes not
only a knocking block, an audio signal preprocessing
circuit, and MCU. but contains a detailed explanation of
the development process, such as the optimization of
hyperparameters, conversion of the model andinstallation for MCU. The utilization of glass bottles inthe experiment led to more pronounced variations in theenergy of the knocking sound. The experimental results
can be up to 99% accurate under multiple tests.
Experimental results indicate that the system is feasible
and measurement accuracy is acceptable.

Dr. Chun-Liang Liu
He received the Ph.D. degrees inElectrical Engineering from National
Taiwan University of Science andTechnology. Currently he is a AssistantProfessor in National Yunlin Universityof Science &Technology.

References
1.

2.

3.

4.

Mr. Tai-Hsuan Wu
He received the B.S. degrees and now heis studying for the M.S. degree inelectrical engineering from National
Yunlin University of Science andTechnology.

S. Deng, H. Zhao, W. Fang, J. Yin, S. Dustdar, and A.Y.
Zomaya: Edge Intelligence: The Confluence of Edge
Computing and Artificial Intelligence. IEEE Internet ofThings Journal, vol. 7, no. 8, pp. 7457–7469(2020).
Z. Zhou, X. Chen, E. Li, L. Zeng, K. Luo and J. Zhang:
Edge Intelligence: Paving the Last Mile of Artificial
Intelligence with Edge Computing. in Proceedings of theIEEE, vol. 107, no. 8, pp. 1738–1762, 18884046 (2019).
C. W. Hung, S. X. Zeng, C. H. Lee, W. T. Li: End-to-End
Deep Learning by MCU Implementation: An Intelligent
Gripper for Shape Identification. Sensors. 21(3):891(2021).
H. H. P. Wu, C. W. Hung, S. H. Chang, Z. H. Yang:
Analysis and Implementation of Noncontact Level©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
168

Powered by TCPDF (www.tcpdf.org)

MCU Based Edge Computing

Dr. Chau-Chung SongHe received the Ph.D. degrees inElectrical Engineering from National
chiao tung university. Currently he is
a Distinguished professor and Chair in
National Formosa University©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:CO HorutoHall, Oita, Japan, 2024
169

Potential of genetic algorithms in multi-UAV coverage problem
Ramil FaizullinIntelligent Robotics Department, Kazan Federal University, 420008, Kazan, Russian Federation
Tatyana TsoyIntelligent Robotics Department, Kazan Federal University, 420008, Kazan, Russian Federation
Edgar A. Martínez-García

Department of Industrial Engineering and Manufacturing, The Autonomous University of Ciudad Juarez, 32315 Cd
Juárez, Chihuahua, Mexico
Evgeni MagidIntelligent Robotics Department, Kazan Federal University, 420008, Kazan, Russian Federation
HSE University, Moscow, Russian Federation
E-mail: fzllnrml@gmail.com, tt@it.kfu.ru, edmartin@uacj.mx, magid@it.kfu.ru

Abstract
For a rapid area coverage multiple UAVs are often used simultaneously. However, a path planning for a UAVs groupduring an area coverage task is computationally challenging. In practice, heuristic algorithms are applied to solve thisproblem. This paper overviews approaches to an area coverage problem with a group of UAVs using genetic
algorithms. The article explores modifications that may be useful for a genetic algorithm for solving the coverage
problem as well as representation methods for chromosomes that reflect a path of multi-UAV. Additionally, UAV
group collision avoidance strategies during area coverage are considered.
Keywords: Genetic algorithms, Coverage path, multi-UAV coverageelement in solving complex problems such as security[17], mapping [18], forest fire monitoring [19], search
operations [20]. The task of covering an area with a groupof UAVs attempts finding a path for each UAV such that
the UAVs observe all locations of a given area without
colliding with each other [21]. This task may haveconstraints, such as a time limit for an entire mission,
safety constraints related to a minimum distance between
UAVs, flight range constraints due to a battery capacity,and others.

1. Introduction
A current rapid progress in robotics drives robotic
systems into many areas of a human life including
autonomous driving [1], manufacturing [2], search andrescue activities [3], agriculture [4], [5], and medical
services [6]. In modern industrial applications of Industry
4.0 [7] a human–robot interaction in a shared workspaceis emphasized [8], [9]. Service robotics [10] is apromising area for robots’ integration covering multiple
applications from education [11] and medicine [12] toentertainment [13] and advertisement [14]. A task of area
coverage using mobile robots often arises in various
practical applications of mobile robotics. Together with aproper scheduling, a determining of an optimal route fora mobile robot to ensure coverage of all points in a certain
area while avoiding collisions with obstacles plays animportant role [15].

The coverage path problem for a mobile robot is NPhard [22] due to a need of analyzing an entire set of
possible solutions, a number of which grows
exponentially with increasing a complexity and a size ofan area to be covered. In practice, heuristic algorithms areoften employed for this task as they allow to get close to
the optimal solution rather quickly, yet do not find aguaranteed optimal solution.

Unmanned Aerial Vehicles (UAVs) provide an
efficient solution for automating coverage andmonitoring of large areas. UAVs are not constrained by
obstacles on the ground and can quickly navigate throughhard-to-reach areas while reducing human labor costs[16]. The use of UAVs for area coverage can be a key

Genetic algorithms are popular heuristic algorithms thathave a relatively simple implementation and could be
easily parallelized, which is especially important whenworking with large solution spaces. Genetic algorithms
can deal with multiple optimization criteria, which isoften the case in practice. They provide a method for©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
170

Ramil Faizullin, Tatyana Tsoy, Edgar A. Martínez-García, Evgeni Magid

exploring an entire solution space to find a global
optimum, which is useful in problems with many local
extrema. All these turn genetic algorithms into aneffective solution of the multiagent coverage problem.

UAV's assigned area. If parents represent acceptable
chromosomes, the offspring could contain the same genes
and do not represent solutions of the coverage problem.
For example, child A contains genes 2, 4, and 5, andaccording to the chromosome, the UAV trajectory passes
through these regions twice, but it does not visit regions
1, 3, and 8.

2. Genetic algorithms
Genetic algorithms are based on principles of biologicalevolution and often are applied to search problems [23].
They model a process of natural selection to find optimal
solutions. These algorithms interact with a population ofpotential solutions, each called an individual. By
successively evaluating a set of solutions a newgeneration is created. The best solutions have a higher
chance of passing on their characteristics to a next
generation. Over time, solutions within the population
become better adapted to a particular task.

Fig. 1. An example of a crossover operation where an
offspring no longer reflects a proper UAV trajectoryFirst a genetic algorithm creates an initial population,
which in typically contains random chromosomes. Each
chromosome is a possible solution of a problem and afitness function assigns each chromosome a numericalvalue that reflects its effectiveness in solving the problem.
To create a next generation, individuals are selected fromthe population in a fixed manner. Individuals with ahigher fitness for an environment have a better chance ofsurviving and passing their traits on to the next generation.
This leads to an improvement in the population over time.
A crossover operation is performed to create newoffspring chromosomes. This allows new individuals toinherit the best characteristics from their parents, thus
preserving them for future generations. Mutations are
random changes in chromosomes of individuals thatmaintain a genetic diversity and periodically contributeto significant leaps in a population development. After acrossover, there is a certain probability that a mutation
occurs. This mutation slightly alters chromosomes of an
offspring.

One solution is to modify the crossover operation. For
example, using a two-point crossover from a single parent.
With this method of the crossover from a single parent
many children are formed and then two with a highest
value of the fitness function are selected (Fig.2). In thiscase, each offspring reflects a proper UAV trajectory. In[25] the authors reported a four times computation
acceleration of a genetic algorithm simulation using the
proposed crossover method.

3. Features of genetic operators’ application inarea coverage
In the area coverage problem for UAVs, it is necessary
to find an optimal trajectory employed by a UAV to visit
all given points of an area. Each chromosome encodes a
trajectory of a UAV and the population is a set of possible
UAV trajectories. In practice, chromosomes are usually
sequences of integers, yet a user decides on a way of
trajectory encoding within a chromosome [24].

Fig. 2. An example of a crossover from one parent
producing three offspring
4. Single UAV and group of UAV coverage tasks

An important feature of applying a genetic algorithm tothe coverage problem is that after crossover or mutation
operations, a chromosome must encode some trajectory
for a UAV. Therefore, crossover and mutation operations
are subjects to certain constraints.

In practice, using a group of UAVs for an area coverageincreases a speed and a reliability of the procedure. EachUAV acts as a moving obstacle for other UAVs, and its
route cannot be planned in isolation from the rest of thegroup [26]. Therefore, the genetic algorithm must
consider routes of all UAVs together. A multiagent
coverage problem solution is a path for each UAV in

Fig. 1 shows a two-point classical crossover operationthat produces two offspring. A chromosome represents anorder of visiting obstacle-free cells while covering a©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
171

Potential of genetic algorithms

which they cover a given area together and do not collidewith each other, i.e., the solution combines paths of all
UAVs into a single chromosome (Fig. 3) [27].

comparison of a minimum distance between each pair of
UAVs and as a number of UAVs increases, a complexity
increases exponentially.
6. Conclusions
Classical genetic algorithms require a task dependent
adaptation in order to be employed for the problem ofcovering a known territory with a group of UAVs as theuse of standard genetic algorithm operators in this context
may destruct a chromosome that represents a trajectoryof a UAVs’ group. Genetic algorithms should optimize
methods of representation and decomposition of asurveyed area and incorporate dynamic obstacles’
collision avoidance strategies to increase the efficiency of
their usage.

Fig. 3. A chromosome combines paths of four UAVs
5. UAV group collision avoidance strategies
during area coverage
While using a group of UAVs reduces a time tocomplete the task, it also introduces potential risks of
UAV collisions during the mission. Two strategies of
UAVs’ collision avoidance are discussed in this sectionin the context of genetic algorithms: assigning UAVs a
different coverage areas and introducing a penalty for
approaching other UAVs.

Acknowledgement
This paper has been supported by the Kazan Federal
University Strategic Academic Leadership Program("PRIORITY-2030").

In the first approach, to prevent UAVs from collidingin the air, the area could be divided into equal subareas
and each UAV is assigned a different subarea to cover[28]. This method allows a computation paralleling foreach UAV with a genetic algorithm (as their trajectories
are independent, (Fig. 4) and keeps a genetic algorithm
simple. Disadvantages of this strategy include a highdependence of a resulting coverage effectiveness on aquality of an area distribution between UAVs, collision
avoidance considerations while distributing UAVs (thattakeoff in a common area) for the assigned subareas, andcomplicated splitting into subareas for a case of different
survey priorities of the subareas.

References
1.

J. Cheng, L. Zhang, Q. Chen, X. Hu and J. Cai, A review
of visual slam methods for autonomous driving vehicles,
Engineering Applications of Artifcial Intelligence,
vol. 114, 2022, pp. 104992.
2. A. Gribkov, M. Morozkin, V. Kuptsov, P. Pivkin and
A. Zelenskii, Industry 4.0 concepts in the machine-toolindustry, Russian Engineering Research, vol. 41(7), 2021,
pp. 634–635.
3. E. Magid, Simulation Tools for Urban Search and Rescue
Robotics, International Conference on Artificial Life andRobotics (ICAROB 2023), 2023, pp. 4-11.
4. V. Nguyen, Q. Vu, O. Solenaya and A. Ronzhin, Analysis
of main tasks of precision farming solved with the use ofrobotic means, in MATEC Web of Conferences, EDP
Sciences, vol. 113, 2017, pp. 02009.
5. R. Iakovlev, A. Saveliev, Approach to Implementation of
Local Navigation of Mobile Robotic Systems in
Agriculture with the Aid of Radio Modules, Telfor Journal.
vol. 12(2), pp. 92-97. 2020.
6. D. Kolpashchikov, O. Gerget and R. Meshcheryakov,
Robotics in healthcare, in Handbook of Artifcial
Intelligence in Healthcare, Springer, 2022, pp. 281–306.
7. R. Galin and R. Meshcheryakov, Automation and robotics
in the context of Industry 4.0: the shift to collaborative
robots. In IOP Conference Series: Materials Science andEngineering, vol. 537(3), May 2019, pp. 032073.
8. R. Galin and R. Meshcheryakov, Review on human–robot
interaction during collaboration in a shared workspace, inInternational Conference on Interactive Collaborative
Robotics, Springer, 2019, pp. 63–74.
9. R. Sultanov, S. Sulaiman, H. Li, R. Meshcheryakov andE. Magid, A review on collaborative robots used inindustrial and service sectors. Siberian Conference onControl and Communications, (SIBCON 2022), 2022,
pp. 1-7.
10. R. Belk, Ethical issues in service robotics and artifcial
intelligence, The Service Industries Journal, vol. 41(13-14),
2021, pp. 860–876.

In the second approach, a penalty function can be
introduced to reduce a fitness function value if UAVs fly
too close to each other at a coverage time [29]. This
strategy is a flexible way to influence the genetic
algorithm and allows using different penalty functions
depending on task goals. Yet, it requires a pairwise

Fig. 4. Example of division of area to be covered into
separate areas for each UAV©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
172

Powered by TCPDF (www.tcpdf.org)

Ramil Faizullin, Tatyana Tsoy, Edgar A. Martínez-García, Evgeni Magid

11. L. Gavrilova, A. Kotik, T. Tsoy, E. A. Martinez-Garcia, M.
Svinin and E. Magid, Facilitating a preparatory stage ofreal-world experiments in a humanoid robot assisted
English language teaching using gazebo simulator, 2020
13th International Conference on Developments ineSystems Engineering (DeSE), IEEE, 2020, pp. 222–227.
12. A. Sagitov, T. Tsoy, H. Li and E. Magid, Automated open
wound suturing: detection and planning algorithm,
J. Robotics Netw. Artif. Life, vol. 5(2), 2018, pp. 144–148.
13. K. J. Morris, V. Samonin, J. Baltes, J. Anderson and
M. C. Lau, A robust interactive entertainment robot for
robot magic performances, Applied Intelligence,
vol. 49(11), 2019, pp. 3834–3844.
14. P. Bustos, L. J. Manso, A. J. Bandera, J. P. Bandera, I.
Garcia-Varea and J. Martinez-Gomez, The cortex
cognitive robotics architecture: Use cases, Cognitive
Systems Research, vol. 55,2019, pp. 107–123.
15. E. Galceran and M. Carreras, A survey on coverage path
planning for robotics, Robotics and Autonomous systems,
vol. 61(12), 2013, pp. 1258-1276.
16. E. Denisov, A. Sagitov, K. Yakovlev, K. Su, M. Svinin and
E. Magid, Towards Total Coverage in Autonomous
Exploration for UGV in 2.5D Dense Clutter Environment,
In Proceedings of the 16th International Conference on
Informatics in Control, Automation and Robotics,
ICINCO; SciTePress, vol. 2, 2019, pp. 409-416.
17. T. M. Cabreira, L. B. Brisolara and P. R. Ferreira Jr.,
Survey on coverage path planning with unmanned aerial
vehicles, Drones, vol. 3(1), 2019, p. 4.
18. D. Albani and D. Nardi, V. Trianni, Field coverage andweed mapping by UAV swarms, IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS),
Ieee, 2017.
19. K. A. Ghamry and Y. Zhang, Cooperative control ofmultiple UAVs for forest fire monitoring and detection,
12th IEEE/ASME International Conference on
Mechatronic and Embedded Systems and Applications(MESA), IEEE, 2016.
20. I. Maza and A. Ollero, Multiple UAV cooperative
searching operation using polygon area decomposition andefficient coverage algorithms, Distributed Autonomous
Robotic Systems 6. Tokyo, Springer Japan, 2007, pp. 221230.
21. J. Muñoz, B. López, F. Quevedo, C. A. Monje, S. Garrido
and L. E. Moreno, Multi UAV coverage path planning inurban environments. Sensors, vol. 21(21), 2021, 7365.
22. E. M. Arkin, S. P. Fekete and J. S. Mitchell,
Approximation algorithms for lawn mowing and milling,
Computational Geometry, vol. 17(1-2), 2000, pp. 25-50.
23. S. Katoch, S. S. Chauhan and V. Kumar, A review on
genetic algorithm: past, present, and future, Multimedia
tools and applications, vol. 80, 2021, pp. 8091-8126.
24. A. Tuncer and M. Yildirim, Chromosome coding methods
in genetic algorithm for path planning of mobile robots,
Computer and Information Sciences II, 26th International
Symposium on Computer and Information Sciences,
Springer London, 2012.
25. T. H. Pham, Y. Bestaoui and S. Mammar, Aerial robot
coverage path planning approach with concave obstacles
in precision agriculture, Workshop on Research, Education
and Development of Unmanned Aerial Systems (REDUAS), IEEE, 2017.
26. Y. Bai, Y. Wang, M. Svinin, E. Magid and R. Sun,
Adaptive multi-agent coverage control with obstacle
avoidance, IEEE Control Systems Letters 6, 2021, pp. 944949.

27. B. Li, B. Moridian, A. Kamal, S. Patankar and
N. Mahmoudian, Multi-robot mission planning with static
energy replenishment, Journal of Intelligent & Robotic
Systems 95, 2019, pp. 745-759.
28. R. Almadhoun, T. Taha, L. Seneviratne and Y. Zweiri, A
survey on multi-robot coverage path planning for modelreconstruction and mapping, SN Applied Sciences 1, 2019,
pp. 1-24.
29. M. Nazarahari, E. Khanmirza and S. Doostie, Multiobjective multi-robot path planning in continuous
environment using an enhanced genetic algorithm, Expert
Systems with Applications 115, 2019, pp. 106-120.

Authors Introduction
Mr. Ramil Faizullin
He is a Ph.D. student in Computerscience and information processes at
the Institute of Information Technology
and Intelligent Systems (ITIS), Kazan
Federal University (KFU), Russia.

Ms. Tatyana Tsoy
In 2012 she graduated fromInternational Area Studies Master's
Degree Program at the University ofTsukuba. In 2023 completed a PhD in
Computer science and information
processes at ITIS, KFU.

Professor Edgar A. Martínez-García
He is a full Professor at the
Universidad Autónoma de Ciudad
Juárez, Mexico; founder and Head of
the Robotics Laboratory; leader of the
Mechatronics academic body at theInstituteofEngineeringandTechnology, since 2007. He obtained his Ph.D. degree in
Robotics Engineering from the University of Tsukuba, Japan(2005).
Professor Evgeni Magid
A Professor, a Head of Intelligent
Robotics Department and a Head ofLaboratory of Intelligent Robotic
Systems (LIRS) at KFU. Professor at
HSE University, Russia. Senior IEEE
member. Previously he worked at
University of Bristol, UK; Carnegie
Mellon University, USA; Universityof Tsukuba, Japan; National Institute of Advanced Industrial
Science and Technology, Japan. He earned his Ph.D. degreefrom University of Tsukuba, Japan. He authors over 200
publications.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
173

Construction of Anthropomorphic Grippers with Adaptive Control
Evgeny Dudorov

JSC ‘SPA ‘Android technics’, 23 Grayvoronovskaya st., Moscow, 109518, Russia
Julia Zhdanova

Institute of Cybernetics, MIREA – Russian Technological University, 78 Vernadskij prosp., Moscow 119454, Russia
Ivan Zhidenko

JSC ‘SPA ‘Android technics’, 23 Grayvoronovskaya st., Moscow, 109518, Russia
Vladimir Moshkin

Institute of Cybernetics, MIREA – Russian Technological University, 78 Vernadskij prosp., Moscow 119454, Russia
Alexander Eryomin

Intelligent Robotics Department, Kazan Federal University, Kazan, 420111, Russia
Evgeni MagidIntelligent Robotics Department, Kazan Federal University, Kazan, 420111, Russia
Alexander Permyakov

JSC ‘SPA ‘Android technics’, 23 Grayvoronovskaya st., Moscow, 109518, Russia
Email: rusandroid@mail.ru, aneremin@it.kfu.ru, magid@it.kfu.ru

Abstract
The functionality of industrial robots is provided primarily by the capabilities of their end effectors. The limited
capabilities of robotic grippers determined the transition to the creation of anthropomorphic grippers. The number ofdegrees of freedom (DOF) of the end effector in the form of an anthropomorphic hand ensuring reliable grasping andholding the object should be at least nine, preferably twelve. The implementation of such a design is possible only
when switching to the principle of construction of an underactuated grippers system. This paper presents the concept
of constructing a group drive, which ensures the implementation of the movement of the output links of two or more
executive groups from one motor. Technical solutions are based on the development of methods for analyzing
complex mechanical systems using functional circuits.
Keywords: Anthropomorphic gripper, underactuated grippers, end effector, robotic grasp

1. Introductionin the design or/and an inability to adjust the arm to a newtask. One of the solutions is to unify the hand [4].

Today, robotic manipulators are actively used inindustrial production [1]. They improve working
conditions of employees and reduce enterprise costs
ensuring a continuous production. Typical applications
include assembly, casting, stamping, cutting, machineloading/unloading, welding and material handling [2].
Generally, a robotic arm functionality is determined bydesign and properties of an end effector. The mostapplied end tool is a robotic gripper. The choice of a grip
depends on many characteristics, including weight, shapeof an object, motion speed, permissible compression
force, and a point of contact [3]. Values of eachcharacteristics are often taken into account at a design
stage of a robotic arm. Thus, problems arise with aninefficient operation of a robot in case of miscalculations

The most widely used are universal vacuum suction
grippers and multi-fingered hands [5]. The first one is asingle mass of granular material that flows around a target
object and takes its shape upon applying a pressure on theobject. The disadvantage of this approach is a need toreturn to a neutral state after each grasping. Multifingered hands usually possess many independently
actuated joints. This design of the end effector has
sufficient softness and rigidity for a reliable and stablegrip of an object [6]. [7]. MCU Based Edge Computing.
The main difficulty of the approach is related to acomputational complexity of tactile sensing andcomputer vision based algorithms [8]. Our research is
aimed at developing a concept of a multi-fingered handusing both active and passive joints. Such design allows©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
174

Evgeny Dudorov, Ivan Zhidenko, Alexander Permyakov, Julia Zhdanova, Vladimir Moshkin, Alexander Eryomin, Evgeni Magidthe hand to envelop an object without complex
calculations.

a single motor. Thus, inspired by [23] we designed amodifiable structure of an anthropomorphic gripper,
which is presented in this paper.

The most common anthropomorphic grippers are
structure diagrams of grippers with three [9] or fourparallel executive groups of links [10]. Each executive
group includes, as a rule, three output links [11].
Collectively, the gripper has nine or more degrees of
freedom (DOF). There are two typical layout schemes for
installing individual drives for each link. In accordancewith the first one, the motors are placed at the output links[10]. In the second case, the motors are located within asingle link of the manipulator [12], [13].

3. A gripper design
The proposed design is based on an anthropomorphic
gripper with two parallel executive groups of links (Fig.
1). From motor 1 that is installed at the basis of thegripper, the movement is transmitted by two parallel
streams to two executive groups. A flexible twisting
spring is integrated into a kinematic scheme of eachstream, connecting coaxial shafts a and b. Links 2 and 3transmit movement through transmission systems to the
output links using the shafts. The installation of flexible
elements adds a relative rotation of the coaxial shafts to a
motion transmission system.

Typically, dimensions of output links do not allow
internal installation of motors with a sufficient power[14]. Consequently, forces generated at endpoints of theoutput links do not exceed 1.5 N [10]. Installing motors
in a single link involves a significant complication of adesign. For instance, a gripper by DLR [15] has 65
pulleys for laying flexible rods on a manipulator link and38 pulleys directly on the gripper. At the same time, it isalso impossible to provide a significant effort at theoutput link. In a modified DLR version, the maximum
force is 9 N [16]. The use of individual drives complicatesa control system. In this case, a number of controllers fordrives equals to a number of output links.
A group drive solves the abovementioned problems ofcontrollability and complexity of individual drives. In the
ASIMO grippers [17] and JPL‐Nautilus Gripper [18] asingle motor provides a movement of three output links
in each executive group. An object is grasped according
to the principle of kinematically dependent movements of
the links. Thus, the manipulator reliably grips objects of
only a certain size with all three links.

Fig. 1. Structure diagram of an anthropomorphic
gripper with two parallel executive groups of theoutput links.

2. Related WorkA qualitative change in operational characteristics of asystem with a group drive is achieved by using a principleof underactuated grippers. It is based on an
implementation of additional passive elements into a
structure of each executive group. These can becompression springs [19], [20] or tension springs [21],[22]. At the same time, an adaptive motion control isimplemented by output links in the executive group.
Links move sequentially, from a proximal link to a distal
link. A change of a control object (an output link) isachieved when external conditions change, and a moving
link stops after reaching an external object.
A further reduction of anthropomorphic gripper mass ispossible by ensuring a motion of output links of adjacentexecutive groups from a single motor. Using the principleof underactuated grippers, flexible elements should beplaced into a group drive system for an entire gripper.
Similarly, for the executive group, this will ensure
independent motion of the executive groups of links from

Fig. 2. Structure diagram of an anthropomorphic
gripper with two opposite executive groups of theoutput links.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
175

Construction of Anthropomorphic Grippers

Fig. 2 shows a structure diagram of theanthropomorphic gripper with opposite executive groupsof links 4, 5 and a lever system for motion transmitting[24]. The movement is transmitted from motor 1 in twoparallel streams to driving links 2 and 3 involving
twisting springs a and b. Then links 2 and 3 set into
motion lever mechanisms 6 and 7. Executive groups 4
and 5 are constructed using the principle of underactuatedgrippers. Compression springs d and c are installed
between the links of the motion transmission system andthe driving links.

performed using a single motor. A number of
implemented flexible links equals to a number of
executive groups. A shape of an object to be grasped
determines a resulting motion of executive group links
and a sequence of their motion. Experimental validation
confirmed feasibility of the proposed construction of theanthropomorphic gripper.
References
1.

Executive groups 4 and 5 move from initial position
synchronously in a direction of an object located between
them, which should be grasped. A grasp can shift towards
one of the executive groups, depending on a shape of theobject. As the object is gripped by link 8, a rotational
movement of leading link 2 decreases and spring a getspartially deformed. After grasping the object with all
links 8-10, the transmission of rotation through spring astops and the spring gets twisted. The movement ofexecutive group 5 is maintained until its links completely
encircle the object. The rotation of the motor stops after
reaching a preset torque value. Deformed springs a and b
retain a force corresponding to an end of the graspingprocess. This provides a given force effect on the object.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

Fig. 3. The constructed anthropomorphic gripper withthree executive groups of links.

12.

Fig. 3 shows the constructed anthropomorphic gripper
with three executive groups of links. The single motor
provides the movement of the output links. The distal linkpossesses 10.5 N force for the entire length of theexecutive group links of 100 mm.

13.

14.

4. Conclusion
15.

The paper proposed a new design an anthropomorphic
hand with an underactuated gripper. A movement of
output links of several executive groups of the gripper is

L. Birglen and T. Schlicht, A statistical review of industrial
robotic grippers. Robotics and Computer-Integrated
Manufacturing, vol. 49, 2018, pp. 88-97.
H. A. F. Almurib, H. F. Al-Qrimli and N. Kumar, A review
of application industrial robotic design. 2011 Ninth
International Conference on ICT and Knowledge
Engineering, IEEE, 2012, pp. 105-112.
Z. Samadikhoshkho, K. Zareinia and F. Janabi-Sharifi, A
brief review on robotic grippers classifications. 2019 IEEE
Canadian Conference of Electrical and Computer
Engineering (CCECE), IEEE, 2019, pp. 1-4.
M. S. Choi et al. Development of multi-purpose universal
gripper. 2017 56th Annual Conference of the Society of
Instrument and Control Engineers of Japan (SICE), IEEE,
2017, pp. 1421-1424.
P. V. P. Reddy and V. Suresh, A review on importance ofuniversal gripper in industrial robot applications. Int. J.
Mech. Eng. Robot. Res, vol. 2, 2013, pp. 255-264.
K. Khusnutdinov et al. Development and Implementationof Grasp Algorithm for Humanoid Robot AR-601M.
ICINCO, vol. 2, 2019, pp. 379-386.
K. Khusnutdinov et al. Household objects pick and placetask for AR-601M humanoid robot. Interactive
Collaborative Robotics: 4th International Conference, ICR
2019, Proceedings 4, Springer International Publishing,
August 2019, pp. 139-149.
E. Brown et al. Universal robotic gripper based on thejamming of granular material. Proceedings of the National
Academy of Sciences, vol. 107, 2010, pp. 18809-18814.
L. U. Odhner et al. A compliant, underactuated hand forrobust manipulation. The International Journal of Robotics
Research, vol. 33, no. 5, 2014, pp. 736-752.
Y. H. Lee et al. Design of anthropomorphic robot handwith IMC joints. 2016 13th International Conference on
Ubiquitous Robots and Ambient Intelligence (URAI),
IEEE, 2016, pp. 336-337.
N. Elangovan et al. Improving robotic manipulation
without sacrificing grasping efficiency: A multi-modal,
adaptive gripper with reconfigurable finger bases. IEEE
Access, vol. 9, 2021, pp. 83298-83308.
M. Shahmohammadi and M. Liarokapis, A series elastic,compact differential mechanism: On the development ofadaptive, lightweight robotic grippers and hands. 2021
IEEE/RSJ International Conference on Intelligent Robots
and Systems (IROS), IEEE, 2021, pp. 6110-6116.
K. Mizushima et al. Multi-fingered robotic hand based on
hybrid mechanism of tendon-driven and jamming
transition. 2018 IEEE International Conference on Soft
Robotics (RoboSoft), IEEE, 2018, pp. 376-381.
U. Kim et al. Integrated linkage-driven dexterous
anthropomorphic robotic hand. Nature communications,vol. 12, no. 1, 2021, p. 7177.
M. Grebenstein et al. The DLR hand arm system. 2011
IEEE International Conference on Robotics andAutomation, IEEE, 2011, pp. 3175-3182.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
176

Powered by TCPDF (www.tcpdf.org)

Evgeny Dudorov, Ivan Zhidenko, Alexander Permyakov, Julia Zhdanova, Vladimir Moshkin, Alexander Eryomin, Evgeni Magid

16. F. Lange, G. Quere and A. Raffin, Decoupled Control of
Position and/or Force of Tendon Driven Fingers. 2019
International Conference on Robotics and Automation(ICRA), IEEE, 2019, pp. 1176-1182.
17. Y. Sakagami et al. The intelligent ASIMO: System
overview and integration. IEEE/RSJ international
conference on intelligent robots and systems, IEEE, vol. 3,
2002, pp. 2478-2483.
18. S. B. Backus et al. Design and testing of the JPL‐Nautilus
Gripper for deep‐ocean geological sampling. Journal of
Field Robotics, vol. 37, no. 6, 2020, pp. 972-986.
19. D. Hirano, K. Nagaoka and K. Yoshida, Design ofunderactuated hand for caging-based grasping of freeflying object. Proceedings of the 2013 IEEE/SICE
International Symposium on System Integration, IEEE,
2013, pp. 436-442.
20. Z. Ren et al. HERI hand: A quasi dexterous and powerful
hand with asymmetrical finger dimensions and under
actuation. 2017 IEEE/RSJ International Conference onIntelligent Robots and Systems (IROS), IEEE, 2017, pp.
322-328.
21. S. Qiao et al. Self-adaptive grasp process and equilibrium
configuration analysis of a 3-DOF UACT robotic finger.
Mechanism and Machine Theory, vol. 133, 2019, pp. 250266.
22. L. Kang, S. H. Kim and B. J. Yi, Modeling, Design, andImplementation of an Underactuated Gripper with
Capability of Grasping Thin Objects. Machines, vol. 9, no.
12, 2021, p. 347.
23. T. Laliberte, L. Birglen and C. Gosselin, Underactuation in
robotic grasping hands. Machine Intelligence & Robotic
Control, vol. 4, no. 3, 2002, pp. 1-11.
24. A. Bogdanov, A. Permyakov and Y. Zhdanova, Synthesisof structural scheme of drive of adaptive multiple-link
gripper. MATEC Web of Conferences, EDP Sciences, vol.
161, 2018, p. 03009.

Ms. Julia Zhdanova
She is Deputy Director of the Instituteof Cybernetics, MIREA — RussianTechnological University, Russia.

Dr. Vladimir Moshkin

He is Candidate of Technical
Sciences, assistant professor,
MIREA — Russian Technological
University, Russia.

Mr. Alexander Eryomin
He received a BSc degree from
Siberian Federal University in 2022.
Currently, he is a second-year studentof Master degree program inIntelligent Robotics at the Institute ofInformation
Technology
andIntelligent Systems (ITIS) at the
Kazan Federal University (KFU).
Prof. Evgeni Magid
He is Professor, Head of Intelligent
Robotics Department and Head ofLaboratory of Intelligent Robotic
Systems (LIRS) at KFU, Russia.
Professor at HSE University, Russia.
Senior IEEE member. Previously heworked at University of Bristol, UK;
Carnegie Mellon University, USA; University of Tsukuba,
Japan; National Institute of Advanced Industrial Scienceand Technology, Japan. He earned his Ph.D. degree from
University of Tsukuba, Japan. He authors over 200
publications.

Authors Introduction
Dr. Evgeny A. Dudorov
He is Candidate of Technical
Sciences,
Assistant
Professor,
Executive Director of JSC “Scientificproduction association “Android
technics”, Russia.

Mr. Alexander Permyakov
He is General Director of JSC“Scientific production association“Android technics”, Russia.

Mr. Ivan Zhidenko
He is Head of the Advanced Projects
Department, of JSC “Scientificproduction association “Android
technics”, Russia.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
177

Simulation-Based Enhancement of SNR in Drone Communication through Uniform Linear
Array Configurations
Gershom Phiri1, Mastaneh Mokayef1*, MHD Amen Summakieh1, M.K.A Ahamed Khan1, Sew Sun Tiang1, Wei Hong Lim1,
Abdul Qayyum2
1

Faculty of Engineering Technology and Built Environment, UCSI University,
Kuala Lumpur, 56000, Malaysia*
Email: mastaneh@ucsiuniversity.edu.myhttps://www.ucsiuniversity.edu.my/2

National Heart and Lung Institute, Imperial College London, UK

Abstract
As drones navigate through shared airspace, they often encounter other drones, wireless devices, and communication
systems. This coexistence creates potential sources of interference that can degrade the signal-to-noise ratio (SNR).
To maintain reliable communication in drone systems, it is crucial to effectively manage and mitigate interference
from other drones and wireless devices operating on the same frequency bands. By addressing these challenges, we
can ensure a stable and dependable SNR for seamless communication among drones. This paper sheds light on thehistory of applications and challenges of utilizing flying base stations for wireless networks and analyzes different
factors that affect signal-to-noise ratio (SNR) to enhance the performance of drone communication.
Keywords: SNR, UAV, drone, Matlab, IoT

accurate data transmission [1]. However, achieving ahigh signal-to-noise ratio in drone networks presents
several challenges [2]. The use of multiple drones
operating in close proximity can result in interference andsignal degradation, affecting the overall quality ofcommunication. Moreover, the noise generated by thedrones themselves, commonly referred to as ego-noise,
can further impact the signal-to-noise ratio. These
challenges necessitate careful consideration andinnovative solutions to optimize the signal-to-noise ratio
in drone networks, ensuring efficient and reliablecommunication between drones and ground control
stations [3]. The interference experienced by each drone
at a particular location can be measured by its signal-tonoise ratio (SNR). The SNR is influenced by multiple
factors, including the transmission power levels, antenna
properties, drone altitude, path loss, and the power
spectral density of the surrounding noise. These variables
collectively contribute to the observed interference levels
and play a crucial role in determining the quality ofcommunication in drone networks [4]. The altitude atwhich drones fly significantly impacts the signal-to-noise
ratio in their networks. Higher altitudes result in larger
distances between the transmitter and receiver, leading to
a decrease in signal quality. Atmospheric attenuation
further contributes to signal loss as altitude increases.
Transmission power levels and antenna characteristics ofdrones also play a vital role in determining signal strength.
Optimal power levels ensure overcoming noise andinterference, while antenna properties affect signal

1. Introduction
Unmanned aerial vehicles (UAVs) are set to play a
crucial role in the next generation of wireless networks,
offering a promising solution to meet the ever-increasinguser demands. Their mobility, flexibility, improved lineof-sight capabilities, and ability to reach inaccessible
areas make them ideal candidates to act as aerial basestations. Researchers are actively exploring various
aspects of deploying, analyzing performance, managing
resources, optimizing trajectories, and modeling channels
in such networks. This survey article focuses on different
applications and the algorithms involved in implementing
aerial base stations, providing a comprehensive review of
each research area. In summary, this article highlights the
key applications, challenges, and technology employed in
the design and analysis of UAVs as base stations.
In recent years, the widespread adoption of drones
across various industries and applications has beenremarkable. From capturing stunning aerial photographs
to delivering packages, drones have become increasingly
prevalent. One crucial factor that significantly influences
the performance and efficiency of drone networks is thesignal-to-noise ratio. The signal-to-noise ratio represents
the ratio between the desired signal and unwanted noise
in a communication system. A high signal-to-noise ratio
is paramount for clear and reliable communication
between drones and ground control stations, as well as for©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
178

Gershom Phiri, Mastaneh Mokayef, MHD Amen Summakieh, M. K. A Ahamed Khan, Sew Sun Tiang, Wei Hong Lim, Abdul Qayyum

coverage and directionality. Lastly, the noise power
spectral density, originating from various sources,
introduces background noise that affects the overallsignal-to-noise ratio. Careful consideration of thesefactors is crucial for maintaining reliable communicationin drone networks [5], [6], [7]. In [8], they delve into therealm of drone-assisted backscatter communication
within an Internet of Things (IoT) sensor network. Theirwork focuses on developing a framework to evaluate thelikelihood of ground-based sensor nodes being covered.
Raja's study in 2021 explores how communication
propagates between drones and base stations in indoor
environments, taking into account factors like diffraction,
frequency, and atmospheric attenuation [9]. While paper[10] proposes an innovative multi-UAV communication
model that prioritizes security. This model employs awireless mesh network and cryptographic techniques toensure efficient and protected communication among
multiple drones. The study underscores the significanceof establishing reliable data communication security
between drones and servers. Collectively, these papers
offer valuable insights into coverage probability
assessment, propagation mechanisms, multi-UAV
coordination, and security measures within the realm ofdrone communication. To aid in understanding theMicro(weight6100g)

communication requirements of UAVs, Table 1
concisely presents a summary of these requirements. Thisreference table offers a clear and accessible overview of
the key aspects involved in UAV communication,
making it easier for readers to grasp the necessary
information.
Table 1: Essential Communication Requirements for
UAV Systems
Data Type

Data Rate

Synchronization (PSS/SSS)DL

ULRadio control (PDCCH)

N/A

Command and control ( C&C)

60-100 kbps

Command and Control (C &C)

60-100 kbps

Application data

Up to 50Mbps

A Comparative Study of the distinctive features ofdrone types are presented in [9], [10], [11] and thesummary of these results is tabulated in Table 2.

Table 2: Distinctive Features of Drone Types [11], [12]Very Small
Small
Medium(100g<weight<2kg (2kg6weight<25kg)(25kg6weight6150kg))

Large (weight>150kg)

Model

Kogan nanodrone

Parrot Disco

DJI Spreading Wings
S900

Scout B-330 UAV
helicopter

Predator B

Illustration

A
16g N/A
Multi-rotor

B
750g
N/A
Fixed-wing

C
3.3kg
4.9 kgMulti-rotor

D
90kg
50kg
Multi-rotorE
2223kg1700kg1700kgMulti-rotor

WeightPayloadFlyingmechanism

16g
N/A
Multi-rotor

750g
N/A
Fixed-wing

3.3kg
4.9kg
Multi-rotor

90kg
50kg
Multi-rotor

2223gg1700kgMulti-rotor

Range
Altitude
Flight time

50-80 mN/A
6-8 min

2km
N/A
45min

N/A
N/A
18min

N/A
3km
180min1852km
15km
1800min

Speed

N/A

80 km/h57.6 km/h

482km/hPowersupply3.7V/160mAh
Li-battery

2700mAh/25A 3cell
LiPo Battery

LiPo Battery (6S,
10000mAh_15000mAh,
15C(Min))100km/h(horizontal)
Gasoline

2. Methodology
2.1. Performance evaluation Through SNR strength950-shaft-horsepower
Turboprop Engine

analyzing a key metric of the SNR. This metric will be
used to thoroughly assess and scrutinize the effectivenessof drone operations, providing valuable insights into theiroverall efficacy. SNR is commonly expressed in decibels
and represents the ratio of signal power to noise power.
An SNR greater than 1:1 (greater than 0 dB) indicates thatthe signal is stronger than the noise. When information istransmitted wirelessly from one point to another, it isUnmanned aerial vehicles (UAVs) have gained
significant popularity, due to their autonomous features,
versatility, and wide range of applications. In this section,
we will delve into a detailed evaluation of drone
performance. Our main focus will revolve around©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
179

Simulation-Based Enhancement of SNR

referred to as a data link. In the context of unmanned
aviation, this wireless transmission is known as a radio
link or radio modem. The data link enables the exchange
of information between the aircraft's autopilot and theGround Control Station (GCS). This communication
involves two distinct links: the uplink, which transmits
information from the GCS to the aircraft, and the
downlink, which transmits information from the aircraft
to the GCS. Eq. (1) is used to calculate the SNR level.

optimal output quality, it is recommended to have an
SNR higher than -20 dB. The graph illustrates a gradualdecline in signal quality as the range increases, with thehighest signal quality observed in close proximity to thetransmitter. Considering the fixed parameters of afrequency threshold of 2.4 GHz, a power threshold of 1W,
and a specific distance, it becomes evident that a stronger
signal is required at a distance of 1m and a frequency of
10 MHz, with a transmit power of 1W [11]. This is dueto the reduced interference from various sources such as
remote controls, phones, Bluetooth devices, plants, andbuildings, which tend to degrade the signal quality. As
the distance between the transmitter and receiverincreases, the SNR decreases, indicating weaker signal
quality at longer ranges.

𝑆𝑁𝑅 (𝑑𝐵) = 𝑃 𝑠𝑖𝑔𝑛𝑎𝑙 (𝑑𝐵) − 𝑃𝑛𝑜𝑖𝑠𝑒 (𝑑𝐵) (1)Where P is the power. The power levels can be
expressed in decibels per dBm. The range at which adrone can fly in relation to a base station is determined by
the minimum SNR required for the specific application.
It is important to establish the maximum distance that canbe achieved while maintaining the desired SNR. During
flight, the SNR changes as the drone moves throughdifferent distances, adapting to meet the target SNR. To
illustrate this relationship, Fig. 1 depicts a simulationgraph created using MATLAB, showing how the SNR is
influenced by the distance traveled by the drone.

3. SNR enhancement by Array Gain for Line-ofSight Propagation
SNR enhancement by array gain is a technique used to
improve the quality and reliability of wirelesscommunication in line-of sight propagation scenarios.
Modern wireless communication systems often
implement large-scale antenna arrays to take advantageof the benefits offered by array gain [14]. To improve the
SNR level and consequently wireless communication, we
proposed the use of an array antenna instead of the single
antenna into the drone side [15], [16]. The SNR is
calculated and compared for both SISO and SIMO cases
using Eq. (2) and Eq. (3) respectively.
𝑆𝑁𝑅𝑠𝑖𝑠𝑜 =𝑃𝑡 𝐺𝑡 𝐺𝑟 𝜆2
4𝜋𝑑 2 𝑁0

(2)

Where 𝑃𝑡 , 𝐺𝑡 , 𝐺𝑟 , and 𝜆 are transmitted power,
Transmitted gain, Received gain and wavelength oftransmitted signal respectively. The 𝑑 and 𝑁0 also stands
for distance between transmitter-receiver, and the noise
power.

a.

𝑆𝑁𝑅𝑠𝑖𝑠𝑜 =𝑃𝑡 𝐺𝑡 ∑ 𝐺𝑟𝑖 𝜆2
4𝜋𝑑 2 𝑁0

(3)

Where 𝐺𝑟𝑖 is the gain of the i-th receiving antenna.
The result of these calculations has been presented in acomparison way showing the energy per bit to noise
power spectral density ratio. This technique is commonlyused as a metric to quantify the quality of acommunication system in terms of the received signal
power and the noise power affecting the transmission.

b.
Fig. 1 SNR as a function of (a) distance traveled bydrone, and (b) frequency of operation [13]2.2. SNR enhancement methods
The analysis presented in Fig. 1 provides valuable
insights into the significance of the signal-to-noise ratio(SNR) in determining the quality of output. The observed
threshold value for SNR is -10 dB; however, to achieve©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
180

Gershom Phiri, Mastaneh Mokayef, MHD Amen Summakieh, M. K. A Ahamed Khan, Sew Sun Tiang, Wei Hong Lim, Abdul Qayyum

communications and Internet of Things (IoT)
communications. When equipped with cellular links,
drones require secure and low-latency communication
with ground base stations. The coherent nature ofreceived signals through the Single-Input MultipleOutput (SIMO) receive array presents an opportunity todirect the array towards the transmitter, thereby
increasing the Signal-to-Noise Ratio (SNR). Analysis of
the Bit Error Rate (BER) curve reveals a gain of 6 dBachieved through the utilization of the receive array. This
gain indicates that the receiver has knowledge of theincoming signal path. These findings highlight theadvantages of incorporating UAVs into wirelessnetworks, showcasing their potential to enhance coverage,
power, and communication capabilities. The ability tosteer the receive array towards the transmitter, coupledwith the coherent nature of received signals, contributes
to improved SNR and overall system performance.
Future research can focus on optimizing UAV
deployments, refining beamforming techniques, andexploring additional applications where UAVs can beeffectively utilized in wireless networks. By harnessing
the capabilities of UAVs in these contexts, advancements
can be made towards more secure, reliable, and efficient
wireless communication systems.

Fig. 2 Energy per Bit to Noise Power Spectral
Density Ratio (E/N) Comparison for SISO andSIMO Systems
Due to the LOS communications, the transmitter andreceiver are in direct communication. For the ease of
calculation, a four-element Uniform Linear Array (ULA)with half-wavelength spacing, is considered in our
simulations. The SISO and SIMO channel BER and theenergy per bit to noise power spectral density ratio is
depicted in Fig. 2. In a SIMO system, the BER curve
demonstrates a notable improvement of 6 dB when areceive array is utilized. This gain is primarily attributed
to the coherent nature of the received signals that interact
with the components of the receive array. As a result, thereceive array can be strategically oriented towards thetransmitter, leading to an enhanced Signal-to-Noise Ratio(SNR). This phenomenon indicates that the receiver has
knowledge of the incoming signal path, allowing for
improved performance in terms of signal quality anderror rate. The observed gain can be explained by the factthat the received signals around the receive arraycomponents exhibit coherence. This coherence arises dueto the spatial diversity provided by multiple antennas in
the receive array. Each antenna captures multiple
instances of the transmitted signal, with slight variations
in phase and amplitude. When these signals are combined,
they undergo constructive interference, leading to anincrease in the overall received signal power. Thisphenomenon highlights the advantage of utilizing a
receive array with multiple antennas, as it enables thesystem to effectively capture and utilize the coherent
signals, resulting in improved signal strength.

References
1.

2.

3.

4.

5.

4. Conclusion

6.

In summary, this research paper emphasizes thesignificant role of Unmanned Aerial Vehicles (UAVs) inwireless networks, offering a wide range of applications
such as aerial base stations, wireless network user
services, and mobile relays in flying ad-hoc networks.
Deploying UAV base stations can greatly enhance
wireless network coverage and power, enabling
communication in various scenarios, including thosewhere the dissemination of public safety information is
critical. Furthermore, UAVs show promise in supporting
secureandenergy-efficient
millimeter-wave

7.

8.

R. P. Fernandes, J. A. Apolinário Jr, A. L. Ramos, and J.
M. de Seixas, "Applying the majority voting rule inacoustic detection and classification of drones."
M. Mokayef, T. A. Rahman, R. Ngah, and M. Y. Ahmed,"Spectrum sharing model for coexistence between highaltitude platform system and fixed services at 5.8 GHz,"
Int. J. Multimed. Ubiquitous Eng, vol. 8, pp. 265-275,
2013.
M. A. Summakieh and M. Mokayef, "Single wideband
microstrip patch antenna for 5G wireless communication
application," Journal of Electrical andElectronics
Engineering (IJR DO) Volume-1, Issue-4, Paper-2, 2016.
X. Ren, S. Vashisht, G. S. Aujla, and P. Zhang, "Droneedge coalesce for energy-aware and sustainable service
delivery for smart city applications," Sustainable Cities
and Society, vol. 77, p. 103505, 2022.
I. Gómez-Arista, J. A. Dávila-Pintle, N. MontalvoMontalvo, A. A. Rubin-Alvarado, Y. E. Bravo-García, and
E. Reynoso-Lara, "Fourier Coefficients Applied to
Improve Backscattered Signals in A Short-Range LIDAR
System," Electronics, vol. 9, no. 3, p. 390, 2020.
D. Mukhutdinov, A. Alex, A. Cavallaro, and L. Wang,"Deep learning models for single-channel speechenhancement on drones," IEEE Access, vol. 11, pp. 2299323007, 2023.
Z. A. Khan, S. Amjad, F. Ahmed, A. M. Almasoud, M.
Imran, and N. Javaid, "A Blockchain-Based DeepLearning-Driven Architecture for Quality Routing in
Wireless Sensor Networks," IEEE Access, vol. 11, pp.
31036-31051, 2023.
A. C. Eska, "Cellular Communication Propagation at
Drone around Building Environment with Single Knife
Edge at 10 GHz," JURNAL INFOTEL, vol. 13, no. 1, pp.
25-30, 2021.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
181

Powered by TCPDF (www.tcpdf.org)

Simulation-Based Enhancement of SNR

9.

10.

11.

12.

13.

14.

15.

16.

G. Raja, S. Anbalagan, A. Ganapathisubramaniyan, M. S.
Selvakumar, A. K. Bashir, and S. Mumtaz, "Efficient andsecured swarm pattern multi-UAV communication," IEEE
Transactions on Vehicular Technology, vol. 70, no. 7, pp.
7050-7058, 2021.
F. Ronaldo, D. Pramadihanto, and A. Sudarsono, "Securecommunication system of drone service using hybrid
cryptography over 4g/lte network," in 2020 International
Electronics Symposium (IES), 2020: IEEE, pp. 116-122.
B. Vergouw, H. Nagel, G. Bondt, and B. Custers, "Drone
technology: Types, payloads, applications, frequency
spectrum issues and future developments," The Future ofDrone Use: Opportunities and Threats from Ethical andLegal Perspectives, pp. 21-45, 2016.
P. Garg, "Characterisation of Fixed-Wing Versus
Multirotors UAVs/Drones," Journal of Geomatics, vol. 16,
no. 2, pp. 152-159, 2022.
G. Phiri, M. Mokayef, S. S. Tiang, and W. C. Hong,"Drone Performance Analysis Based on SNR Factor,"
2022.
L. N. Ribeiro, A. L. de Almeida, and J. C. Mota, "Lowrank tensor MMSE equalization," in 2019 16th
International Symposium on Wireless Communication
Systems (ISWCS), 2019: IEEE, pp. 511-516.
M. Khalily, M. R. Kamarudin, M. Mokayef, S. Danesh,
and S. E. A. Ghahferokhi, "A new wideband circularly
polarized dielectric resonator antenna," Radioengineering,
vol. 23, no. 1, pp. 175-180, 2014.
A. A. Aldair, M. T. Rashid, A. F. Halihal, and M. Mokayef,"Design of pitch angle controller for wind turbine based onpi neurofuzzy model," Indonesian Journal of ElectricalEngineering and Computer Science, vol. 15, no. 3, pp.
1664-1670, 2019Dr. M. KA. Ahamed Khan
He completed his undergraduate degree
in Electronics and Communication
Engineering and postgraduate degree inElectronics and Control Engineering inIndia. Additionally, he obtained anadvanced diploma in Power Electronics
and Drives from Lucas-Nuelle inGermany in 2002, and a Diploma in Drives and Controls
from Woo Sun in Korea in 2014. He pursued a PhD in
Robotics, Power Electronics, and Controls in the UnitedStates and holds certifications. He is a Senior member of
the IEEE in the USA and a member of MIET in the UK.
Dr. Sew Sun Tiang
She is an Assistant Professor in Facultyof Engineering at UCSI University in
Malaysia. She received her PhD in
Electrical and Electronic Engineeringfrom Universiti Sains Malaysia in 2014.
Her research interests are optimization
and antenna design.
Dr. Wei Hong Lim
He is an Assistant Professor in Faculty ofEngineering at UCSI University in
Malaysia. He received his PhD in
Computational
Intelligence
fromUniversiti Sains Malaysia in 2014. His
research interests are optimization andartificial intelligence.
Dr. Abdul Qayyum
He received his PhD in Electrical andElectronic Engineering from Universiti
Teknologi Petronas, Malaysia, in 2018.
He developed the totype. Previously, hecompleted his bachelor's degree incomputer engineering and master's
degree in electronic engineering from
Pakistan. He is a postdoctoral researcher at the Universityof Burgundy, France, where he studies cardiac MRI images
using deep learning approaches. He is currently affiliatedwith the National Heart and Lung Institute at Imperial
College London, UK.

Authors Introduction
Dr. Mastaneh Mokayef
She has received her PhD from WirelessCommunication Centre Faculty ofElectrical Engineering in University
Technology Malaysia (UTM) in 2014.
She has also obtained her master’s degreefrom the faculty of engineering in 2009
from the University Technology
Malaysia. She has been working in UCSI
University, Malaysia, since 2015 in which she currently
serves as an Assistant Professor in the Faculty ofEngineering and Built Environment. Her research interests
include: Wireless communications, spectrum sharing
method, spectrum management, etc.
Mr. MHD Amen Summakieh
He received the B.Eng. degree (Hons.) in
communication
andelectronics
engineering from UCSI University,
Malaysia, in 2016, and the M.Eng.Sc.
degree from Multimedia University,
Malaysia, in 2020. His research interests
include heterogeneous LTE-advanced
cellular networks, user association,
metaheuristic algorithms, and antennas design.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
182

Empowering Elderly Individuals through the Intelligent Shopping Trolley
Mastaneh Mokayef1*, Muzaiyanah Binti Hidayab1, MHD Amen Summakieh1, M.K.A Ahamed Khan1, Kim Soon Chong1, Chin
Hong Wong2, Chua Huang Shen3, Abdul Qayyum4
1

Department of Electrical and Electronic Engineering, Faculty of Engineering Technology and Built Environment, UCSI
University, Kuala Lumpur, 56000, Malaysia*Email: mastaneh@ucsiuniversity.edu.myhttps://www.ucsiuniversity.edu.my/2

Maynooth International Engineering College, Fuzhou University, Fujian, China

3

Department of Electrical & Electronics Engineering, UOW Malaysia University Utropolis Glenmarie, Shah Alam,
Malaysia
4

National Heart and Lung Institute, Imperial College London, UK

Abstract
In this research, we have developed a prototype that aims to improve the weekly shopping experience for seniorcitizens. Our system tracks the movements of elderly individuals, eliminating the need for them to exert physical force
in pushing or pulling the shopping trolley. To achieve this, we utilize a combination of sensors, including a gyroscopeand magnetometer, to estimate the user's walking distance and direction. Additionally, we employ WiFi fingerprinting
to accurately determine the user's position. Our experiments have yielded satisfactory results in terms of tracking
accuracy and the overall functionality of the system. By addressing the specific challenges faced by senior citizens
during the routine and essential process of grocery shopping, our smart shopping trolley concept seeks to enhancetheir experience. Through the integration of tracking technology and sensor-based solutions, we aim to make shopping
more convenient and comfortable for elderly individuals. The positive outcomes observed in our experiments validate
the effectiveness and feasibility of this approach.
Keywords: smart shopping trolley, elderly citizen, smart phone, UML

communication, the collected data can be transmitted tothe counter, reducing the time spent waiting in queues.
Moreover, the use of RFID technology in smart trolleys
can enhance security by facilitating cashless transactions.

1. Introduction
Senior citizens often encounter challenges when it
comes to shopping, particularly in the case of grocery
shopping. While there are options available such as
helpers or online shopping, many elderly individuals
prefer to do their own shopping but still require assistancein carrying their bags back home. The advancement oftechnology has sparked interest in integrating different
systems, resulting in the rise of the Internet of Things(IoT) and its widespread use in wireless communication.

Additionally, the successful integration of purchased
items, payment systems, and data collection relies on theimplementation of IoT standards. This ensures smoothconnectivity and compatibility among the various
components involved in the shopping process. The
Internet of Things (IoT) field is experiencing rapid
growth in the world of wireless telecommunications [1].
Its purpose is to enable computers to interact with theirsurroundings and access information about devices andobjects without the need for direct human involvement[2]. According to a report [3], IoT is characterized bymoderate data transmission rates, affordability, andoperates within the frequency range of 100MHz to
5.8GHz. This advancement in wireless technology offers
users a flexible and cost-effective means ofcommunication. A recent report [4], [5] suggests that by
the year 2020, around 50 billion devices will be

The term "smart" has become increasingly common invarious electronic devices and technologies, such as
smartphones, smart cars, smartwatches, and even smart
homes. In line with this trend, the concepts of "smart
shopping" and "smart trolleys" have emerged to provide
valuable support to senior citizens. Smart trolleys often
incorporate Radio Frequency Identification (RFID)
technology, enabling products to be scanned as they areplaced in the trolley. By utilizing ZIGBEE©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
183

Mastaneh Mokayef, Muzaiyanah Binti Hidayab, MHD Amen Summakieh, M.K.A Ahamed Khan, Kim Soon Chong, Chin Hong Wong, Chua Huang
Shen, Abdul Qayyum.

connected to the internet, opening up a plethora offunctionalities. There is a rising demand for technology
and a growing need for convenient shopping experiences,which has spurred increased research efforts in thedevelopment of smart shopping concepts. These trolleys
are equipped with core functionalities such as following
the user, navigating through the shopping paths, andavoiding collisions with obstacles. This technology aims
to eliminate the physical strain and difficulties thatelderly individuals often face while shopping, resulting ina more comfortable and enjoyable experience.
Advancements in autonomous navigation have been a
key focus in the development of smart shopping trolleys.
Researchers have been exploring various approaches toenable autonomous navigation for smart shopping
trolleys. For example, a new automated smart cart system
for modern shopping centers was presented, along with
Android application-controlled approaches [1]. Theseadvancements in autonomous navigation provide thetrolleys with the ability to move independently, allowing
elderly individuals to navigate the store without physicalexertion. The implementation of smart shopping trolleys
for elderly care holds several key advancements andbenefits. Other recent reports in [6], [7] describe a smart
trolley system for mega malls that uses a microcontroller.
It allows customers to automatically scan products with abarcode scanner and see running totals on an LCD display,without needing to queue for checkout.

through a coil, they do not have a polarity. Revering theconnection will only reverse the direction of the motor.
2.3. Inertial gyro sensor
In our proposed design, we have included a special
sensor called a gyroscope to help us keep track of how
something is rotating. This sensor is pretty clever because
it also has a built-in accelerometer, which measures how
fast something is moving in a straight line. So, while theaccelerometer tells us about the linear movement of theuser, the gyroscope tells us about the angular rotation. By
combining these two sets of information, we canaccurately detect and understand how the user is moving.
Another great thing about this sensor is that it's small anddoesn't cost too much, which makes it a perfect fit for our
project. We wanted to make sure we chose componentsthat are both effective and affordable.
2.4. H-Bridge dual motor controller L298N
In order to control the rotation direction of the motor,
the current flow through the motor must be inverted. The
most common method of doing that is by using an HBridge circuit. An H-Bridge circuit contains four
switching elements, transistors or MOSFETs, with the
motor at the center forming an H-like configuration. By
activating two switches at the same time, the direction of
the current flow and as a result the rotation direction of
the motor will be changed. In this project, the L298N isused as a dual H-Bridge motor driver which allows speed
and direction control of two DC motors at the same time.

In this paper, the novel smart shopping trolley has been
introduced. The proposed smart trolley is designed to
avoid the pushing the typical shopping trolley. It is also
designed to follow the user to ease of the controlling
process for the user.

3. Design and fabrication
2. Hardware Interfaces
In the world of shopping trolleys, we've seen various
types introduced over the years. However, our projecttakes things a step further with the development of asmart shopping trolley that brings convenience to a whole
new level. Imagine being able to control your shopping
trolley using your smartphone, thanks to Wi-Fi
technology. Through a real-time Wi-Fi connection
between the trolley's micro-controller and yoursmartphone, you can effortlessly guide the trolley to yourdesired destination. All it takes is a few taps on a
dedicated app installed on your smartphone to sendcommands to the micro-controller via the Wi-Fi signal.
It's like having a personal assistant for your shoppingneeds.

The smart shopping trolley designed specifically for
elderly care integrates various essential components toimprove its functionality and user-friendliness. These
components are crucial in creating a smooth andconvenient shopping experience for elderly individuals.
The following are the components utilized in this project:
2.1. Micro Controller Unit (NodeMCU)
The NodeMCU micro-controller is used in this project
as the main controller which has the integrated WiFi
module. NodeMCU is an open source Lua based
firmware for the ESP8266 WiFi SOC from Espressif anduses an on-module flash-based SPIFFS file system.
NodeMCU is implemented in C that ease the use of thiscomponent in IoT based projects.

But convenience isn't the only thing we focused on. We
also wanted to ensure the safety of both the user and thetrolley. That's why we integrated an ultrasonic sensor
right at the front of the trolley. This little hero plays acrucial role in preventing accidents and damage causedby sudden stops or collisions. It works by emitting soundwaves and then listening for the echoes that bounce back.
By analyzing the time it takes for the sound waves totravel and return, the ultrasonic sensor can accurately

2.2. DC motor
A utilized DC motor would have only two terminals.
Since these terminals are connected together only

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
184

Empowering Elderly Individuals throughmeasure the distance between the trolley and any
obstacles in its path. This allows the trolley to adjust its
movement and avoid potential collisions by utilizing thesignals it sends and receives. So, in a nutshell, our smart
shopping trolley combines the convenience ofsmartphone-controlled Wi-Fi technology with the added
peace of mind provided by the ultrasonic sensor. With
this integration, we aim to enhance your shopping
experience by giving you seamless control and ensuringa safe journey through the store, effortlessly avoiding
obstacles along the way. It's like having a reliable
shopping companion right by your side. The operation ofultrasonic sensor has been shown in Fig. 1.
Fig.2 Circuit design for the proposed trolley
The Node MCU, known for its energy efficiency and16-bit RSIC, is utilized as a controller because of its
integrated ESP-12 and convenient WiFi module, which
streamline the control process. The ESP-12 incorporates
a CPU clock speed of 80MHz, with the ability to reach amaximum value of 160MHz. The L298N motor
controller is employed to connect the DC motors to themicrocontroller. Both the microcontroller and L298N
motor controller are powered by a 9V battery in parallel.
The controller has been programmed to define themovements of the trolley, as outlined in Table 1.
Fig.1 Obstacle detection via ultrasonic sensor
Table 1. The motor movement instruction

To establish the necessary connections for theultrasonic sensor in the trolley, we connect the VCC(power supply) and GND (ground) pins to a 5V powersource. Additionally, the trigger input (Trig) pin is linked
to a digital output, while the echo (Echo) pin is connected
to a digital input on the trolley's microcontroller.

Left

Right

Right

Movemen

motor

motor

motor

motor

t

forwarbackwarforwarbackwar

d

d

d

d

To accurately measure the distance between the trolley
and an obstacle, we employ a technique that involves
pulsing the trigger pin to a high level for approximately
10 microseconds. After that, we wait for a high-levelsignal on the echo pin. The duration of this high-levelsignal corresponds to the time it takes for the ultrasonicsound to travel to the obstacle and back. By analyzing the
response time and duration, we can determine theproximity of the trolley to the obstacle. This allows for
precise distance measurement and enables the trolley toeffectively detect and avoid obstacles. The distance canbe calculated as follows:
Distance= Time×Speed=time×0.034cm/µs

Left

Forward

High

Low

High

Off

Backward

Low

High

Low

High

Left

-

-

High

Low

Right

High

Low

-

-

The controlling instructions are sent to themicrocontroller through the app created by MIT app
inventor. The block function of the created app is shownin Fig. 3.

(1)

As sound travels at approximately 340 meters per second.
This corresponds to about 29.41 μs.
The circuit design has been shown in Fig. 2. As shown inthis figure, the MCU, ultrasonic sensor, two motors andmotor controller are connected together as the main
components of our proposed design.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
185

Mastaneh Mokayef, Muzaiyanah Binti Hidayab, MHD Amen Summakieh, M.K.A Ahamed Khan, Kim Soon Chong, Chin Hong Wong, Chua Huang
Shen, Abdul Qayyum.

allowing users to easily navigate and even return thetrolley to its designated location rather than being limited
to indoor use only. This enhancement would offer evengreater convenience and flexibility to the users, making
their shopping trips more efficient and enjoyable. Insummary, our smart shopping trolley for elderly care has
shown promising results, offering a user-friendly andassistance-driven approach to shopping. The inclusion of
GPS technology in future iterations has the potential tofurther enhance the functionality and versatility of thetrolley, providing an improved and seamless shopping
experience for elderly individuals.
References1.

Fig.3 The controlling function block of the app
2.

The fabrication of our proposed prototype is shown in
Fig. 4.
3.

4.

5.
6.

7.

V. Shete, A. Shaikh, R. Sawant, C. Singh and P.
Srideviponmalar. "Smart Shopping Cart for Visually
Impaired Individuals". Aug. 2022.
P. S. Yadav, V. Agrawal, J. C. Mohanta and M. F. Ahmed.
"A Theoretical Review of Mobile Robot Locomotion
based on Mecanum Wheels". Jun. 2022.
H. Bello-Salau, A. J. Onumanyi, M. David, R. Isa, C. O.
Alenoghena and H. Ohize, "A new automated smart cartsystem for modern shopping centres".
Summakieh, MHD Amen, and Mastaneh Mokayef.
"Single wideband microstrip patch antenna for 5G wirelesscommunication application." Journal of Electrical andElectronics Engineering (IJR DO) Volume-1, Issue-4,
Paper-2 (2016).
R. O’Neill. (Jun. 21, 2005). Smart Trolley Shops for You.
Accessed: Jun. 21, 2017.
Aldair, Ammar A., et al. "Design of pitch angle controller
for wind turbine based on pi neuro-fuzzymodel." Indonesian Journal of Electrical Engineering andComputer Science 15.3 (2019): 1664-1670.
Patil, Shishir R., et al. "Smart Trolley with Automatic
Billing System using Arduino." IAES International
Journal of Artificial Intelligence (IJ-AI) 2.2 (2022): 22682273.

Authors Introduction
Dr. Mastaneh Mokayef
She has received her PhD from WirelessCommunication Centre Faculty of ElectricalEngineering in University Technology
Malaysia (UTM) in 2014. She has also
obtained her master’s degree from thefaculty of engineering in 2009 from the
University Technology Malaysia. She has been working in
UCSI University, Malaysia, since 2015 in which she
currently serves as an Assistant Professor in the Faculty ofEngineering and Built Environment. Her research interests
include: Wireless communications, spectrum sharing
method, etc

Fig.4 The fabricated prototype
4. Conclusion and Future Work
In conclusion, we have successfully designed,
fabricated, and tested a smart shopping trolley
specifically tailored for elderly care. The results havebeen highly satisfactory, indicating the effectiveness of
our proposed solution. By implementing advanced
technology, our smart trolley is designed to recognize themovement patterns of elderly users and autonomously
follow them. This eliminates the need for users to exert
physical effort in pushing or pulling the trolley, providing
a more convenient and comfortable shopping experience.
Looking ahead, there is potential for further improvement
and expansion of our smart shopping trolley. One
possible avenue for future work is the integration of GPS
technology. By incorporating GPS, the trolley canaccurately determine its indoor and outdoor location,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
186

Powered by TCPDF (www.tcpdf.org)

Empowering Elderly Individuals throughDr. Abdul Qayyum
He received his PhD in Electrical andElectronic Engineering from Universiti
Teknologi Petronas, Malaysia, in 2018. He
developed the totype. Previously, hecompleted his bachelor's degree in computerengineering and master's degree inelectronic engineering from Pakistan. He isa postdoctoral researcher at the University of Burgundy,France, where he studies cardiac MRI images using deeplearning approaches. He is currently affiliated with the
National Heart and Lung Institute at Imperial College
London, UK.

Mr. MHD Amen Summakieh
He received the B.Eng. degree (Hons.) in
communication and electronics engineeringfrom UCSI University, Malaysia, in 2016,
and the M.Eng.Sc. degree from Multimedia
University, Malaysia, in 2020. His researchinterests include heterogeneous LTEadvanced
cellularnetworks,
user
association, metaheuristic algorithms, andantennas design.
Dr. M. KA. Ahamed Khan
He completed his undergraduate degree inElectronics
andCommunication
Engineering and postgraduate degree inElectronics and Control Engineering inIndia. Additionally, he obtained anadvanced diploma in Power Electronics
and Drives from Lucas-Nuelle in Germany
in 2002, and a Diploma in Drives and Controls from Woo
Sun in Korea in 2014. He pursued a PhD in Robotics, PowerElectronics, and Controls in the United States and holds
certifications. He is a Senior member of the IEEE in the USA
and a member of MIET in the UK.
Dr. Kim Soon Chong
He obtained his B.Eng., M.S., and Ph.D.
degrees in electrical and electronicengineering from Universiti Kebangsaan
Malaysia in 2012, 2015, and 2019,respectively. He currently serves as an
Assistant Professor in the Department ofComputerEngineering(Artificial
Intelligence) within the Faculty of Engineering Technology
and Built Environment at UCSI Kuala Lumpur. His research
focuses on areas such as medical signal processing andinstrumentation.
Dr. Chin Hong Wong
He is a Lecturer in Maynooth International
Engineering College at Fuzhou University
in China. He received his PhD in Electricaland Electronic Engineering from Universiti
Sains Malaysia in 2017. His researchinterests are Energy harvesting and control
system.
Mr. Chua Huang Shen (Mason)
He holds the position of Senior Lecturer at UOW Malaysia
KDU. He received a Master of Science
degree from the University of Belfast in
2004 and a Bachelor of Science degree(Hons) in Electronic Control Systems from
the Moore School of Liverpool University
in 2002. - Earned a bachelor's degree inengineering from Rahman University(UTAR). He is currently working on his Ph.D.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
187

Optimized Microstrip Slot UWB Patch Antenna for Medical Imaging
Maxime DUVACHER2, Mastaneh Mokayef1*, MHD Amen Summakieh1, M.K.A Ahamed Khan1, Sew Sun Tiang1, Wei Hong
Lim1, Abdul Qayyum3
1

Faculty of Engineering Technology and Built Environment, UCSI University,
Kuala Lumpur, 56000, Malaysia*
Email: mastaneh@ucsiuniversity.edu.myhttps://www.ucsiuniversity.edu.my/
2

Polytech Nantes, Rue Christian Pauc, 44300 Nantes, France

Abstract
This research work presents the development of an Ultra-Wideband (UWB) microstrip patch antenna (MPA) with thespecific purpose of tissue characterization. The antenna was carefully designed and simulated to operate within afrequency range of 4.8 to 6.9 GHz, optimized for its intended application. To ensure the best performance, a series ofsimulations and comparisons were conducted using CST Microwave Studio. Various antenna shapes were tested andevaluated to determine the most effective design. The results of these simulations were highly promising, as they
revealed a simulated return loss (𝑆11 ) of -33dB. This indicates excellent performance and demonstrates the suitability
and acceptability of the proposed antenna for medical imaging such as breast imaging, tumor detection, or monitoring
physiological changes.
Keywords: Microstrip antenna, medical imaging, UWB, CST

characteristics are particularly valuable in medical
imaging, as they contribute to clearer and more accurate
imaging results. With these advantages, microstrip
antennas have the potential to revolutionize medical
imaging by providing high-quality images with improved
resolution and accuracy. However, it is important toaddress the limitations associated with microstrip
antennas in order to fully harness their potential in
medical imaging.

1. Introduction
In recent years, the use of microstrip antennas inmedical imaging has attracted significant attention. Theseantennas have demonstrated immense potential indelivering high-resolution images and precise
measurements in various medical imaging techniques
like magnetic resonance imaging and positron emission
tomography. According to Shimu et al., microstrip patch
antennas have found wide-ranging applications in the
medical field, as well as in other industries [1]. Microstrip
antennas offer several advantages that make them wellsuited for medical imaging applications. Firstly,
microstrip antennas have a small size and low profile,which makes them ideal for integration into medical
devices and equipment. Their lightweight nature allows
for easy customization to meet specific requirements.
Additionally, microstrip antennas can be designed withwide bandwidths, enabling the transmission andreception of diverse frequencies. Microstrip patch
antennas also provide high gain and low return loss,
ensuring efficient signal transmission and reception [1, 2].
Another advantage of microstrip patch antennas inmedical imaging lies in their robust design andfabrication. These antennas are known for their durability
and reliability, which are crucial factors in medical
imaging where accuracy and consistency are ofparamount importance. Furthermore, microstrip antennas
exhibit excellent radiation characteristics, including low
cross-polarization and high directivity. These

One limitation of microstrip antennas is their narrow
bandwidth, which restricts the range of frequencies thatcan be transmitted or received. Researchers are actively
exploring different techniques, such as adding additional
resonant elements or using multiband designs, to increase
the bandwidth of microstrip antennas. Additionally,
microstrip antennas often have relatively low gain
compared to other types of antennas [3]. To address this,researchers are investigating techniques such as
designing stacked or array configurations to enhance thegain of microstrip antennas. Impedance matching isanother issue that microstrip antennas face, leading tosignal degradation. Researchers are focusing on
improving the manufacturing process and developing
advanced impedance-matching techniques, such as using
matching networks or adding dielectric layers, tooptimize impedance matching and improve overall
performance. Furthermore, efforts are being made toenhance the polarization characteristics of microstrip
antennas, as certain imaging techniques require specific©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
188

M. Mokayef et al.

polarization orientations for optimal image quality.
Researchers are exploring various polarization
techniques, such as circularly polarized microstrip
antennas or the use of metasurfaces, to control thepolarization of the antenna [4]. The study in [1] presents
the design, fabrication, and analysis of a compact
microstrip patch antenna operating in the Industrial,
Scientific, and Medical (ISM) frequency range of 2.4-2.5
GHz for medical applications. To investigate theperformance of this antenna structure throughsimulations, the high-frequency structure simulator(ANSYS HFSS) and Advanced Design System (ADS)
has been used. Moreover, to measure the level ofelectromagnetic waves absorbed by the human headmodel the specific absorption rate (SAR) is simulated at
different positions. A 2.45 GHz Rectangular Patch
Microstrip Antenna (RMSA) was designed and analyzed
for breast cancer detection in [5], using an insect feedtechnique for impedance matching. Simulations showed
the RMSA had a VSWR<2 bandwidth of 70 MHz (2.8%of operating frequency), resonating at 2.45 GHz. Results
indicate the RMSA outperforms existing designs, making
it a promising option for improved breast tumor detection.
While in [7, 8] a low-cost microstrip patch antenna wasdeveloped specifically for microwave imaging (MWI)applications across a wideband frequency range. The
antenna design incorporates an artificial magnetic
conductor (AMC) to enhance its performance. Computer
simulations using CST demonstrated that the presence of
the AMC significantly improved the frequency
selectivity at 8.6 GHz, resulting in a peak realized gain of
9.90443 dBi (simulated) and 10.61 dBi (measured). To
validate the simulation results, the proposed microstrip
antenna was fabricated and experimentally tested.

Fig.1 Front View of the Proposed Antenna
Table 1. Antenna Dimensions
Parameter
Dimensions (mm)Width
10
Length
13
a
2
PATCH
b
9
(copper)c0.5d5
Thickness
0.035
Width
0.6
FEED LINE(FL)
Length8(copper)Thickness
0.035
Width
2
MAIN LINE(ML)
Length4(copper)Thickness
0.035
Width
28
SUBSTRATE
Length29(FR-4)
Thickness
1.6
Width
28
GROUND
Length10.5(copper)Thickness
0.035
Area

2. Antenna Structure
In this paper, we introduce a newly proposed UltraWideband microstrip antenna and provide an overview of
its structure. The decision to utilize an FR-4 substrate isdriven by its cost-effectiveness and wide accessibility inthe market. The simplicity and convenience of
implementing FR-4 as a substrate make it an ideal choicefor the design of our antenna. In the upcoming sections,
we will delve into the fabrication process and design
specifics of the microstrip antenna, emphasizing its
potential for Ultra-Wideband applications.

3. Results and Discussion
3.1. Accuracy of numerical integration
As defined in [4], the UWB has a bandwidth of greaterthan 500MHz and an operating frequency of between
3.1GHz and 10.6GHz. The return loss threshold of -10dB
has been defined in [5]. A simulation of return loss 𝑆11 isshown in Fig. 2.

As depicted in Fig. 1, the antenna is mounted on the FR4 substrate including the main and sub-fluidline for the
impedance matching purpose. The detailed dimensions of
the design can be found in Table 1.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
189

Optimized Microstrip Slot UWB

Fig. 4. Simulation of VSWR
The acceptable range of Voltage Standing Wave Radio(VSWR) is under 2V. So the acceptable frequency of
between 4.7 and 7 GHz are the acceptable range whichmatches well with the previous results obtained in theoperating frequency from 4.8 to 6.9 GHz.

Fig.2 Return Loss of the Proposed Antenna
The simulation and expected results match well. The
threshold is pointed at -10dB, so the bandwidth is 2.1GHz,in agreement with the 500MHz minimum required by
UWB. Furthermore, the resonance frequency is 5.8GHz.
This one is great, the signal propagation characteristics at
5.8GHz in free space are similar in rainy conditions. Thisissue is important to consider in tropical climates such as
Malaysia [6]. The radiation pattern of the proposed
antenna is illustrated in Fig. 3.

4. Conclusion
In conclusion, this project successfully proposed an
Ultra-wideband (UWB) microstrip patch antenna (MPA)
using an FR-4 substrate. The achieved return loss of 33dB at a resonance frequency of 5.8GHz demonstrates
the effectiveness of the antenna design. With dimensionsof 29×28mm, this antenna offers a cost-effective, userfriendly, and portable solution for various applications,particularly in breast cancer detection. Throughmeticulous design and simulation using CST Microwave
Studio, the antenna was optimized to operate within thedesired frequency range of 4.8 to 6.9 GHz. The extensive
simulations and comparisons performed confirmed thesuperior performance of the proposed design. The
obtained results, including the impressive return loss of 33dB, highlight the antenna's suitability for medical
imaging, such as breast imaging, tumor detection, ormonitoring physiological changes.
References
1.

Fig. 3 Antenna’s radiation pattern
As shown in Fig. 3, the main lobe magnitude is 1.68 dBi
and the main lobe direction of 179 deg has been obtained.
The Angular width (3dB) of 94.8 degrees and the side
lobe level of-1.7 dB has been obtained. Finally, theresults of the VSWR simulation are given in Fig. 4.

2.

3.

4.

5.

Alhuwaidi, Sadiq, and Tanghid Rashid. "A novel compact
wearable microstrip patch antenna for medical
applications." 2020 International Conference on
Communications, Signal Processing, and their
Applications (ICCSPA). IEEE, 2021.
Borkar, Nishant Madhukar and P. K. Parlewar. “Antennas
in Medical Applications.” 2021 IEEE Indian Conference
on Antennas and Propagation (InCAP) (2021): 152-154.
S. Alhuwaidi and T. Rashid, "A Novel Compact Wearable
Microstrip Patch Antenna for Medical Applications," 2020
International Conference on Communications, Signal
Processing, and their Applications (ICCSPA), Sharjah,
United Arab Emirates, 2021, pp. 1-6, doi:
10.1109/ICCSPA49915.2021.9385726.
Khalily, Mohsen, et al. "A new wideband circularly
polarized dielectric resonator antenna." Radioengineering
23.1 (2014): 175-180.
SM, Asha Banu, and Niresh Kumar. "Design of Microstrip
Patch Antenna for the Detection of Malignant Tumor."©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
190

M. Mokayef et al.

6.

7.

8.

2022 4th International Conference on Inventive Researchin Computing Applications (ICIRCA). IEEE, 2022.
Ahmed, M. Firoz, and Md Hasnat Kabir. "Multislot
Rectangular Microstrip Patch Antenna Design to Detect
Breast Cancer." ECS Transactions 107.1 (2022): 10721.
Summakieh, MHD Amen, and Mastaneh Mokayef.
"Single wideband microstrip patch antenna for 5G wirelesscommunication application." Journal of ElectricalandElectronics Engineering (IJR DO) Volume-1, Issue-4,
Paper-2 (2016).
Hamza, Musa N., et al. "Low-Cost Antenna-Array-Based
Metamaterials for Non-Invasive Early-Stage Breast Tumor
Detection in the Human Body." Biosensors 12.10 (2022):
828.

Dr. M. KA. Ahamed Khan
He completed his undergraduate
degreeinElectronics
andCommunication Engineering andpostgraduate degree in Electronics
and Control Engineering in India.
Additionally, he obtained anadvanced diploma in PowerElectronics and Drives from LucasNuelle in Germany in 2002, and a
Diploma in Drives and Controls from Woo Sun in Korea
in 2014. He pursued a PhD in Robotics, Power Electronics,
and Controls in the United States and holds certifications
as a Professional Engineer (PEng) in the USA and aChartered Engineer (CEng) in the UK. He is a Senior
member of the IEEE in the USA and a member of MIETin the UK.

Authors Introduction

Dr. Sew Sun Tiang

Dr. Mastaneh Mokayef
She has received her PhD from
Wireless Communication Centre
Faculty of Electrical Engineering in
University Technology Malaysia(UTM) in 2014. She has also
obtained her master’s degree fromthe faculty of engineering in 2009
from the University Technology
Malaysia. She is a member of Boardof Engineers Malaysia (BEM) since2017, She has been working in UCSI University, Malaysia,
since 2015 in which she currently serves as an AssistantProfessor in the Faculty of Engineering and Built
Environment (FETBE). Her research interests include:
Wireless communications, spectrum sharing method,
spectrum management, cellular communication systems
and Antenna design. To date, he has been awarded with thequalifications of a Chartered Engineer (C.Eng.) from U.K.
Engineering Council.

She is an Assistant Professor in
Faculty of Engineering at UCSI
University in Malaysia. She received
her PhD in Electrical and Electronic
Engineering from Universiti Sains
Malaysia in 2014. Her researchinterests are optimization and antenna
design.

Dr. Wei Hong Lim
He is an Assistant Professor in
Faculty of Engineering at UCSI
University in Malaysia. He received
his
PhD
in
Computational
Intelligence from Universiti Sains
Malaysia in 2014. His researchinterests are optimization andartificial intelligence.

Mr. MHD Amen Summakieh
He received the B.Eng. degree(Hons.) in communication andelectronics engineering from UCSI
University, Malaysia, in 2016, andtheM.Eng.Sc.
degreefromMultimedia University, Malaysia, in
2020. His research interests include
heterogeneous
LTE-advanced
cellular networks, user association,
metaheuristic algorithms, and antennas design.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
191

Powered by TCPDF (www.tcpdf.org)

Optimized Microstrip Slot UWB

Dr. Abdul Qayyum
He did his PhD in Electrical andElectronics
Engineering
fromUniversiti Teknologi PETRONAS,
Malaysia 2018. He developed deeplearning based algorithms for depth
estimation of vegetation, trees near
power lines for Tenaga Nasional
Berhad (TNB) and Sabah Electric
Supply Berhad (SESB) under theministry of Green , Water and Technology (KeTTHA)
Malaysia. He also developed a prototype for Vital signs(heart rate, breathing rate, Sp02) estimation andassessment of stroke and Arterial fibrillation (AF) using
face video analytic based on deep leaning models. Earlier,
he had completed his bachelor’s in computer engineeringand Master in Electronic Engineering from Pakistan.
Besides, he gained one-year industrial experience whileworking as a BSS engineer for Huawei, Pakistan. He had
also taught several courses under electrical, specifically,signal processing domain for 7 years in various public andprivate universities in Pakistan. He was working as a
research scientist in CISIR, UTP for less than one year andwas developed deep learning algorithms for brain signal(EEG) classification and reconstruction, remote sensing
image segmentation and biomedical image analysis. He is
associated with burgundy university France as a Post Doc
researcher and working on the cardiac MRI images using
deep learning approach. He is also working as a consultant
in various projects involving deep learning models in Big
Data, Vital Sign Estimation, IoT and BCI applications.
Currently he is associated with National Heart and Lung
Institute, Imperial College London, UK.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
192

Development of an Innovative Undergraduate Industrial Automation and Robotics Degree
Program
1*

1

2

3

M. K. A. Ahamed Khan , Mastaneh Mokayef , Ridzuan, A1, Irraivan Elamvazuthi , Badli Shah Yusoff , Abu Hassan Darusman

3

1

UCSI University, Faculty of Engineering, Taman Connaught, 56000 Malayisa*Email: Mohamedkhan@ucsiuniversity.edu.my2

University Technology Petronos, Faculty of Engineering, Malaysia
3

NIKL Malaysia France Institute, Malaysia

Abstract
In recent years, the need for integrated engineering courses has increased. Due to its multidisciplinary nature,
Industrial Automation and Robotics degree course is an ideal example of curriculum integration. This paper discusses
several issues such as course offerings, topical content, student profile, student performance and other pertinent
matters related to the recent development of an Industrial Automation and Robotics undergraduate degree programme
at the University of Kuala Lumpur, Malaysia.
Keywords: Industrial Automation, Robotics, Curriculum Development, Undergraduate

multidisciplinary engineering education [4], [5]. It isclear that there is no one correct curricula or approach totreating multidisciplinary engineering education. Rather,
one must take into careful consideration the regional
employment demands, as well as the preparationnecessary for graduate study. It is necessary to practicecareful discernment regarding the rapid technologicalfluctuations that are prevalent today. Specifically,
industry can be expected to make regular demands ofundergraduate preparation to meet their perceived needs[6], [7], [8]. Sometimes these expectations will provide
valuable insights into changing career paths, and at othertimes the demands will be inappropriate, and best
understood as related to the “tools and toys” of the trade.
Engineering students must have a fundamental conceptbase which allows agility and flexibility, for if a programis too specialised, the range of opportunities available forgraduates will be compromised. In the course ofinvestigating the content and structure of a new program
to address the design and development of the ‘industrial
automation and robotics’ programme, it became clear thatclose ties to the fundamental disciplines associated withelectrical engineering was critical. The wide variety of
distributed sensors, their associated electronic interfaces,
the large-scale control systems, and the extreme
environments within which they operate require careful
treatment of analog and digital electronic circuit design,communication and control theory, mathematics, andphysics. These topics traditionally constitute the core of
the electrical engineering discipline [9], [10].

1. Introduction
Industrial Automation and Robotics is amultidisciplinary science that encompasses the areas ofcomputer science, mathematics, physics, mechanicalengineering,industrial
engineering,
electricalengineering, computer engineering, materials science,and manufacturing engineering among others. It provides
an opportunity to break the barriers between all these
disciplines in a single course and it offers an excellent
example of multidisciplinary integration for engineering
students. In recent years, several institutions, following
the lead of the robot-building course at the Massachusetts
Institute of Technology [1], now offer robot buildingclasses and laboratories. Some graduate level courses in
Artificial Intelligence also rely on robot building
techniques to illustrate general concepts of decisionmaking and machine intelligence [2], [3].
In Malaysia colleges, public and private universities
offer many programs which lead to the degree of
Bachelor of Engineering and Bachelor of Engineering
Technology. These programmes range in topics from
Electrical, Mechanical, Chemical, etc. With increasing
frequency, employers are requiring employees in theirengineering or technical staff to have multi-skills.
Naturally, aspiring employees want to take-up interdisciplinary courses to enhance their marketability. To
help meet these desires, a new degree was developed by
University Kuala Lumpur with the first batch of students
enrolled in July 2002.

A curriculum planning process is necessary toinvestigate the individual program topics, theirrelationships, and the pedagogy necessary to present
them in an integrated fashion at an undergraduate level.
We utilised two methods of inquiry to meet these goals.
The first method of inquiry was based on a series of

2. Planning Process
A number of authors have considered the broad issues
associated with current and future trends in©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
193

M. K. A. Ahamed Khan1, Mastaneh Mokayef1, Ridzuan, A1, Irraivan Elamvazuthi, Badli Shah Yusoff, Abu Hassan Darusman

formal meetings (both group and individual) withrepresentatives drawn from our industrial partners andindustrial advisory board. This varied group of people has
been asked to consider their educational needs (industrial
automation and robotics) based on the large and clearly
identified trends in their industries. The second method
involved the exploration of the substantial educational
literature available including such publications as theIEEE Transactions on Education and the Journal ofEngineering Education. At the end of the planning period,
the following outcomes were generated:•curriculum so that appropriate acquisition oflaboratory equipment could be addressed andthe attendant facilities set up; anda time-bound roadmap for introducing the new
curriculum•

3. The Development of Bachelor of Engineering
Technology Programme
The development of Bachelor of Engineering
Technology Programme in Industrial Automation andRobotics Technology (IART) will discussed in thefollowing paras. Once admitted into IART degree
programme, the student takes a minimum of 120 semester

a compilation of the topical content for courses,
particularly for topics that are new in the sense

Table 1: Subject Distribution of IART
NO
1

2

3

4

SUBJECTS

CH

%

LAN

9

7.31

English6

4.88

French2

1.63

Mathematics

8

6.50

Unikl

4

3.25

Mfi

11

8.94

Automation

32

26.02

Electrical

32

26.02

Mechanical

13

10.57

Intra

6

4.88

CATEOGORY
GeneralManagement

Technical

Intra
TOTAL CREDITS HOURS

123

that they address emerging technologies andhence are non-traditional;• a prioritisation of the topics since covering
everything may not be feasible;• a list of courses and their appropriate hierarchyin the curriculum;• an identification of the necessary technologies
that would be needed to offer this new
From Table 1, it can also be seen that the general studies
is composed of 20.32%, management 12.19%, technical62.61% and INTRA 4.88%. The total credits hours is 123.
It is important to note that the department responsible forthe technical concentration areas have specified thecourses for each IART core area. Since concentration
areas are attached to individual academic departments
such department of automation, electrical and mechanical,
this allows the individual departments a great deal offlexibility. There are currently 3 different technicalconcentrations available for the IART student, i.e.,
automation, electrical and mechanical.

TCH

%

25

20.32

15

12.19

77

62.61

06

4.88

123

100%

credit hours of coursework (over 8 semesters) divided
among four general categories of classes. The four areas
are general studies, management, technical and industri
training (INTRA) as shown in Table 1, where CH is thecredit hours and TCH is the total credit hours.

Various subjects are offered in the 8 semesters. As an
example, some of the subjects offered in semester 1 isshown in Table 2.
©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
194

Development of an Innovative

Table 2: Subjects of 1st Semester
No

Code

Subjects

1
2
3
4
5
6
7
8

AUT 3412
FDE 3012
ELE 3112
INM 3012
LAN 1002
LAN 1004
MAT 3112
MAA 3012

Introduction to Automation
Engg. Drawing
Electrical Fundamental
Safety Management
Bahasa Malaysia
Islamic (or Moral) Studies
Mathematics 1
Static and Dynamics

CreditHours

No. of Hrs
Per Week

LEC

2
2
2
2
3
3
2
2
18

3
3
3
2
3
3
3
2.5
22

1
2
1
2
3
3
1
1
14

•

From Table 2, it can seen that the subjects offered insemester 1 are ‘Introduction to Automation’, Engineering
Drawing, Electrical Fundamental, Safety Management,
BahasaMalaysia,
Islamic(Moral)Studies’,‘Mathematics I’ and ‘Statics and Dynamics’. As an
example, looking at ‘Introduction to Automation’, it cannoted that it is 2 credits hours subject where the numberof contact hours between the lecturers and students are 3
hours which is divided as 1 hour for lecture and 2 hours
for practical. In one semester, 54 hours is allocated forthis subject. A total of 396 ‘contact hours’ takes place
between the lecturers and students.

•

To look at the topical contents of the subjects offered,an example of a typical higher level subject is discussed
below. The title of the subject title is known as ‘Robotics
I’. It is offered as a 3 credit hour core subject. Its main
objective is to provide senior undergraduate students withbroad knowledge on robots. The class consists of two anda half hours of lecture and tutorial, and a three-hourlaboratory session weekly. The lecture and tutorial
portion is used to teach fundamentals of robotics while
the laboratory portion teaches ‘hands on’ robot
programming techniques. The course is concerned with“classical” Robotics, which covers geometric models ofrobot manipulators, kinematics, dynamics, control, andpath planning. Many excellent textbooks in this area are
available. The authors have used and particularly
recommends the texts by John J. Craig [6] or P. J.
McKerrow [7].

TD

No of Hrs
Per Sem

P
2

54
54
54
36
54
54
45
45
396

2

1.5
1.5
3

4

Dynamics and Control of a Two-link robot. The
students are introduced to the concepts of
Newton-Euler dynamics only in the case of a
two-link robot. Classical joint-position andvelocity control is also examined without
delving too deeply into this subject.
Robot Manipulator Programming. A six-axis
industrial robot, ABB and Staubli provides
students with a test bed for robot manipulator
programming. As a laboratory exercise, students
program the ABB and Staubli, for a pick andplace task or a simple assembly task.

4. Student Data
This section is divided into three sections, i.e., student
profile in first section which encompasses the student
intake and graduation, whilst the second section covers
the student performance and the third section discusses
about student employment.
4.1
Student Profile
The student profile which encompasses the student
intake and graduation is given in Table 3.

The course contents are listed below:• Review of Mathematical concepts. This part is areview and an introduction to trigonometricfunctions, vectors and matrices, and geometrictransforms such as translations and rotations ofobjects in space. The use of Matlab in solving
numerical problems is examined through lecture
examples and homework assignments.
• Kinematics of robot manipulators. The use of
the Denavit-Hartenberg model for robot
manipulators is introduced and the study of
direct and inverse kinematics is presented.
• Robot path planning and generation. Student are
taught to use cubic splines to determine a pathfor the robot between two end points.

Table 3: Student Profile
IntakeM
Batch July 2002
40
Batch July 2003
19

F
18
4

TOTAL
58
23

Final Semester
Batch July 2002
Batch July 2003

M
31
12

F
13
3

TOTAL
44
15

Graduate
Batch July 2002
Batch July 2003

M
30
12

F
11
3

TOTAL
41
15

Fail to Graduate
Batch July 2002
Batch July 2003

M
1
0

F
2
0

TOTAL
3
0

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
195

M. K. A. Ahamed Khan1, Mastaneh Mokayef1, Ridzuan, A1, Irraivan Elamvazuthi, Badli Shah Yusoff, Abu Hassan Darusman

It can be seen from the Table 3, for the batch July 2002
out of 44 students, 41 manage to get the graduation and 3
fail to graduate. For the batch July 2003 all 15 students
graduated.

CGP
A
Batch
July
2002
Batch
July
2003

The age group of batch July 2002 and batch July 2003
is shown in Table 4.

Table 4: Age Group
Age
Batch
July
2002
Batch
July
2003

28

27

26

25

24

23

22

21

To
tal

1

2

3

1

6

8

11

26

58

4

6

12

23

1

9

1
0

5

3

41

15

The trend of GPA for the batch July 2003 is shown in the
Table 7.
Table 7: Trend of GPA
KB
KP1
GB

The final semester results for the batch July 2002 andJuly 2003 is shown in Table 5.

DL
KB
KP1
KP2
GB
INC

2
6

In batch July 2003, 10 students have got CGPA ranging
from 2.50 to 3.00 and 5 students have got CGPA ranging
from 3.00 to 3.50.Total passed is 15.

Student Performance

Table 5: Final Semester Results
CategoryDL
KB
Batch July 2002
3
38
Batch July 2003
4
11

3

Tota
l

From Table 6, it can be seen that in batch July 2002, 3
students have got final CGPA ranging from 2.00 to 2.50,
26 students have got CGPA ranging from 2.50 to 3.00, 9
students have got CGPA ranging from 3.00 to 3.50 andfinally 3 students rated above 3.5 CGPA. Total passed is41.

From Table 4, it can be seen that the age of the students
enrolled in first semester varies from 21 to 28 years. The
first and second batches had initial enrolment of 58 and
23 students. The higher age group is an evidence ofcandidates enrolement with some Industrial experience.
4.2

Table 6: Final Semester Results<2.0
2.002.503.0>3.
0
2.50
3.00
3.50
5

Total

2

INC

Total

3

22
16

Sem 2003/2
Sem 2004/1

17
13

Sem 2004/2
Sem 2005/1

15
15

15
15

Sem 2005/2
Sem 2006/1

15
15

15
15

3

The trend of GPA is shown in Table 7, it can be seen thatfor the batch July 2003 [Sem 2003/2] out of total 22
students, 17 got KB, 2 got GB and 3 doesn’t complete thesemester. In Sem 2004/1, 13 students come under KB and
3 come under KP1. In Sem 2004/2, 15 come under KB.In
Sem 2005/1, 15 come under KB.In Sem 2005/2, 15 comeunder KB. In Sem 2006/1, 15 come under KB.

41
15

>3.5
2.00-3.49<2.00 1st time
<2.00 2nd time
<1.00 or <2.00 3rd time
Incomplete semester

4.3 Student Employment
The majority of the students enrolled in the IART
degree program are currently employed. In the first batch,
60-70% of students were able to find employment within
6 months of their graduation whereas 50-60% of thestudents were employed from the second batch which
graduated in July 2006. They are employed in
multinationals and as well as locally owned small andmedium enterprises. Amongst the companies that there
are attached with are Texas Instruments, Solectron,
GlaxoSmith Kline, B Braun, Hicom, to name a few. At
the time of writing this paper, it is already four months
since the graduation of the second batch. As mentioned
above, although 60% of them have found employment

From Table 5, it can be seen that in batch July 2002, 3
students fall under the category of DL and 38 students
comes under the category of KB. Total number ofstudents passed is 41.
In batch July 2003, 4 students fall under the category of
DL and 11 students come under KB. Total number ofstudents passed is 15.
The final CGPA results for the batch July 2002 and batch
July 2003 is shown in the Table 6.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
196

Development of an Innovative

robot," 2018 Chinese Control And Decision Conference(CCDC), Shenyang, China, 2018, pp. 3467-3472.

within this period, it is expected that over time, themajority of graduates will find employment.
5.0

Conclusion

Authors Introduction

This paper has discussed the development of an
multidisciplinary course in industrial automation androbotics for undergraduate engineering students. The
IART degree offers a student with a completed diploma
the opportunity to complete a bachelor’s degree in atimely fashion, with little lose in transfer credits. The
average CGPA of IART students are competitive, andemployers recognise the degree and employ the graduates
because of the academic content and the graduates’
ability to perform, both academically and professionally.

Dr M. K. A. Ahamed Khan
He is currently working at UCSI
University, Malaysia. He received his
Ph.D in Robotics and controls from USA.
His area of research is robotics, AI andcontrols. He has published more than100
papers. He is also an IEEE Senior member. He is also thepast chair for IEEE RAS Malaysia chapter.
Dr. Mastaneh Mokayef
She has received her PhD from WirelessCommunication Centre Faculty ofElectrical Engineering in University
Technology Malaysia (UTM). She is amember of Board of Engineers Malaysia(BEM). She has been working in UCSI
University, Malaysia, in which she
currently serves as an Assistant Professor in the Faculty ofEngineering and Built Environment (FETBE). To date, he
has been awarded the qualifications of a Chartered
Engineer (C.Eng.) from the U.K. Engineering Council.

Acknowledgements
The authors are grateful to Universiti Kuala LumpurMalaysia France Institute (UniKL-MFI) and UCSI
University for the support. The authors would also like toextend their thanks to all the lecturers involved inIndustrial Automation and Robotics Technology
programme and Mechatronics Engineering Programme
without whom this work might not be possible.

Ts Amar Ridzuan Bin Abd Hamid
He is a lecturer of Mechanical andMechatronic
programmes
from
Department of Mechanical Engineering,
UCSI University, Malaysia. He hascompleted his Master Degree from
Universiti Putra Malaysia, Postgraduate
Diploma from UCSI, and a Bachelor
Degree with Hounours in Mechanical Engineering(Automotive) from Universiti Teknikal Malaysia Melaka(UTeM), Malaysia.

References
Martin, F., Oberoi, P., Sargent, R., “The 6.270
RobotBuilder’s Guide” , MIT Media Lab, Cambridge, MA,
1992.
2. Turner, C., Ford, K., Dobbs, S., Suri, N., & Hayes, P.,“Robots in the Classroom.” In J. Stewman(Ed.),Proceedings of the Ninth Florida AI Research
Symposium (pp. 497-500). Key West, FL:FLAIRS.
3. Turner, C., Ford, K., Hayes, P., Shamma, D., Manseur, R.,“Robots in Education”, Proceedings of the US-Japan
Graduate Student Forum in Robotics, Osaka, Japan,
November 1996.
4. W. Wolf and J. Madsen, “Embedded Systems Education
for the Future,” Proceeding of the IEEE, January 2000, pp.
23-30.
5. A. McGettrick et al, “Computer Engineering Curriculumin the New Millennium,” IEEE Transactions on Education,
November 2003, pp. 456-462.
6. John J. Craig, “Introduction to Robotics, Mechanics andControl”, 2nd Edition, Addison-Wesley. 1989
7. McKerrow, P. J., “Introduction to Robotics,” AddisonWesley, 1991
8. Sergeyev and Alaraje ., “Promoting Robotics Education:
Curriculum and State-of-the-Art Robotics Laboratory
Development’. The Technology Interface Journal/Spring2010
9. Olga I. Vaganova, Zhanna V. Smirnova., “Development oftraining content for master students in course"mechatronics and robotics" at the University” .Amazonia
Investiga. Volume 8, No 22
10. C. Li, L. Fu and L. Wang, "Innovate engineering educationby using virtual laboratory platform based industrial
1.

Dr. Irraivan Elamvazuthi
He is a senior Member, IEEE. He
received the Ph.D. degree from the
Department of Automatic Control andSystems Engineering, The University of
Sheffield, U.K., in 2002. He is currently
an Associate Professor at the Department
of Electrical and Electronic Engineering,
Universiti Teknologi PETRONAS (UTP), Malaysia. His
research interests include control, robotics, mechatronics,
power systems, and bio-medical applications. He is also
the Chair of the IEEE Robotics and Automation Society(Malaysia Chapter).
Dr Abu Hassan Darusman
He is currently an Associate Professor at
the Department of Industrial automation
at UNIKL MFI, Malaysia. His researchinterests include control, robotics,
mechatronics.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
197

Powered by TCPDF (www.tcpdf.org)

M. K. A. Ahamed Khan1, Mastaneh Mokayef1, Ridzuan, A1, Irraivan Elamvazuthi, Badli Shah Yusoff, Abu Hassan Darusman

Dr Badli Shah Yusoff

He is currently an Associate Professor
at the Department of Industrial
automation at UNIKL MFI, Malaysia.
His research interests include control,
robotics, mechatronics.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
198

Smart Assistive Trolley for Elderly Care and Independence
Dina Koushek1, Mastaneh Mokayef1*, MHD Amen Summakieh1, M.K.A Ahamed Khan1, Abdul Qayyum2, Sivajothi A/LParamasivam3
1

Faculty of Engineering Technology and Built Environment, UCSI University,
Kuala Lumpur, 56000, Malaysia*
Email: mastaneh@ucsiuniversity.edu.myhttps://www.ucsiuniversity.edu.my/2

National Heart and Lung Institute, Imperial College London, UK

3

University of Wollongong Malaysia, Shah Alam, Slenagor, 40150, Malaysia

Abstract
As people get older their shopping experience gets harder, as they have to keep pushing a trolley that is made of steel,as well as they keep on adding items through their shopping journey the trolley gets heavier. As a result, this project
aims to help these senior citizens by providing a robotic trolley that follows them during their journey (through facedetection) without the need of any physical interreference, in addition to calculating the walking distance providing
the property of showing how many meters they have walked. The trolley has been fabricated with the walking distanceestimation feature that has been accomplished by the ultrasonic sensor after the camera detects the user and tracks
him. This strategy is achieved by using an OpenCV library that is particularly could be used in Python programming
language. The results have shown a great improvement in the elderly individuals’ lives, as it supports them by giving
more comfort and extra liberty to the shopping experience.
Keywords: Smart following trolley, OpenCV, Face detection, Walking distance estimation

The brain will keep reading the messages from thesensors the engineer chose to use as ultrasonic sensors
which is very good when it comes to objects detection asdemonstrated by Sanghavi [3], an infrared sensor isefficient to use for people detection even at night because
it follows the temperature mapping technique as analyzed
by Nagarajan [4], as well as the proximity sensor can beused for the reasons as the ultrasonic sensors used for
however with better efficiency, on account of theproximity’s higher efficiency it is used for accidents orcollision prevention, which is why it’s a perfectly suitableoption as elaborated by Lee [5].

1. Introduction
As we live in a time where technology is advancing
daily, people occasionally come up with contemporary
ideas to update or develop specific technologies as
elaborated by Ng [1]. One of these technologies is the
shopping trolleys which are available in all supermarkets
to make it easier for customers to choose and store
products. Customers need to drop the products they wantto purchase and then proceed to checkout. Currently,
when entering the shopping mall or a grocery store thecustomer has to choose a shopping cart (trolley), selectthe items they want to buy, and then put them in the
shopping trolley which is very challenging this to do forsenior citizens.

In order to get the trolley moving automatically, thepresence of a motor is a must, as it will receive a signal
from the brain (microcontroller) to move, however to get
the motors to move in different directions forward,
backward, right, and left motor controllers are needed,
many motors can be used to get a robot to move it
depends on the speed required, the place the robot will be
used in, as DC motors or a servo motor.

To get a robot to move and follow you, a brain of the
prototype is in order, a brain will move everything insequence and give orders to all the components chosen by
the engineer (designer), and change this sequence if
needed to and all of this is accomplished by the algorithm
which is the software used to put everything in to place,the software means the language we people choose tocommunicate with device in this case an automatedtrolley we are creating as explained by Li [2].

Before the microcontroller sends a message to themotors to move in any direction first the microcontroller
should locate the person who is supposed to follow andthen send the message, and that is accomplished byadding a camera to the trolley that would track the person©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
199

Dina Koushek, Mastaneh Mokayef, MHD Amen Summakieh, M. K. A Ahamed Khan, Abdul Qayyum, Sivajothi A/L Paramasivam.

based on his/her features, or even the camera could
follow something that the trolley user is holding on to, oreven by using a transceiver

2.1. Hardware of the trolley

2. Methodology and Experimental Setup

The most recent model of the Raspberry Pi
computer line is the Raspberry Pi 4 Model B. Comparing
it to the Raspberry Pi 3 Model B+ of the previous
generation, it provides revolutionary improvements inprocessing speed, multimedia performance, memory, andconnection while maintaining backward compatibility
and a similar level of power consumption. Performanceon the desktop is comparable to that of entry-level x86
PC systems for the end-user on the Raspberry Pi 4 Model
B as denoted by Raspberry Pi [6].

Raspberry pi 4 Model B

In this chapter, the methodology applied for this projectis described, as well as the mathematical approaches andcorrelations related to the smart follower trolley and thecomponents selection shall be mentioned in Fig. 1.
The Trolley follows
the user.

start

Ultrasonic sensor senses theuser's movement
System Initialization

This fourth-generation raspberry pi model is chosen forin project as the main system core in other words the brainof the robot, due to its high-performance 64-bit quad-coreprocessor, a dual display support at a resolution up to 4K(Kilo) through the micro-HDMI (High Definition
Multimedia Interface) ports as modular compliancecertification for Bluetooth and dual-band wireless LAN(Local Area Network) enables the board to be integrated
into finalized devices with a great reduction incompliance testing, reducing cost and speeding up time
to market, as well as the raspberry pi 4 model B contains
a higher memory (2/4/8GB RAM option), its connectivity
is better than the other raspberry pi computer types, as it
has 2.4 GHz and 5.0 GHz, the data transfer is done at avery high rate as it contains 2 USB 3.0 ports 2 USB 2.0
ports. It has 40 GPIO pins.

The distance calculation
continues
Camera turns on

Is the user stationary?

Yes Ultrasonic sensor
stops calculating

Search for a UserNo
Yes
The ultrasonic sensor
continues calculatingis the user Detected ?

No
The trolley keeps moving

No

Yes
Has the user stopped ?

Motors are initiatedYes
the trolley stops

Ultrasonic sensor is
Initiatedadjust the trolley tothe center.

Arduino UNO

Ultrasonic sensor stop andestimates the final walking
distance

An ATmega328P-based microcontroller board is theArduino UNO. It contains 6 analogue inputs, a 16 MHz
ceramic resonator, a USB port, a power jack, an ICSP(In
Circuit Serial Programming) header, and a reset button. It
also has 14 digital input/output pins, six of which can beused as PWM outputs. It comes with everything required
to support the microcontroller; to get started, just use a
USB cable to connect it to a computer, or an AC-to-DC
adapter or battery to power it. You can experiment withyour UNO without being very concerned about making a
mistake as demonstrated by Arduino memory guide [7].

End

Fig. 1. Project flow chart

In this proposed project the methodology thesis will be
implemented into three parts. To begin with, how will thedetection and tracking of the user’s movements will be
implemented, following on from that, walking distancebesides the user’s directions estimation, finally the wholeproject’s (the smart follower trolley) fabrication
specification in terms of connecting all the components
chosen due to their quality and cost.

An ATmega328P-based microcontroller board is theArduino UNO. It contains 6 analogue inputs, a 16 MHz
ceramic resonator, a USB port, a power jack, an ICSP
header, and a reset button. It also has 14 digital
input/output pins, six of which can be used as PWM
outputs. It comes with everything required to support the
microcontroller; to get started, just use a USB cable toconnect it to a computer, or an AC-to-DC adapter orbattery to power it. You can experiment with your UNO
without being very concerned about making a mistake as
demonstrated by Arduino Memory Guide [7].

The components used to bring this project’s idea to life
are a raspberry pi 4 model B as the main system core(brain of the system), a permanent magnet DC motor, an
Arduino UNO is used to control these motors while being
connected to the Raspberry pi 4 model B through an USB
as a communication interface, the sensors chosen are theultrasonic sensor used for obstacle detection andavoidance, as for the user’s detection the infrared sensor
is used along with a camera settled on the trolley.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
200

Smart Assistive Trolley for

Using an Arduino UNO to control the PMDC motors isvery effective and why is that, due to its simplicity, theArduino UNO is designed in a way to work efficiently
with real-time projects, as well as the Arduino UNO canwork easily with motor drivers as an L293D motor driver,
L298N motor driver, in addition to it way to work
effectively with components that consume low power, as
well as changing the speed of a DC motor is fairly simple.
The speed of a DC motor can change when PWM isapplied to the analogue output pin of an Arduino.
Therefore, it is a straightforward task. Unlike theraspberry pi 4 model B, it works efficiently withcomponents that have high power consumption, it will
work with the PMDC motor, but not as effectively as theArduino UNO as explained by DC motor speed [8].

is chosen in this project to use in the software part for
user’s detection. Processing a video means executing
operations frame-by-frame to the video. Frames arenothing more than a single discrete instance of the video
at a given point in time. Even in one second, there could
be several frames. It is possible to treat frames similarly
to an image. Therefore, we can execute all actions onframes that we can on images precisely explained by [9].
3. Results and Discussion
In order for the brain to do its job which is to control allother components it needs to be connected to a powersource which in this case is the laptop, after theconnection is settled the set-up is made by VNC viewer
which is a way to access and control the raspberry pi (anykind) desktop remotely from another computer, using a
USB C-type cable to activate the brain itself, for thisminicomputer to work it needs internet to function basedon this the raspberry pi 4 model B has the feature toconnect it to the internet without using an ethernet cable.
In this project the raspberry pi has a wireless connection
with internet.

High precision ultrasonic sensor range finder
Three ultrasonic sensors will be used, each one of themwill be directly connected to the raspberry pi 4 model B(brain), they will be used for obstacle avoidance,
ultrasonic sensor ESP32 is chosen as its better than theultrasonic sensor ESP8266 in terms of performance aswell as it provides BLE.

The connection between the Raspberry Pi and theArduino UNO is done using a USB B type cable, this is
only done for energizing it and it’s the way in which theArduino UNO will take the orders from the raspberry pi,
as for the connections between the Arduino UNO and the
PMDC motor, first the motor works at a 250 watts, and it
needs to be stepped down this is the main reason why aconverter is used. The motors are designed in a way thatwhen it works it works only in one direction to make thismotor move in all directions a motor driver is used. The
connections are made as follows starting with connecting
the Arduino UNO to the motor driver ENA pins and IN1,
afterward connecting the motor driver to the PMDC
motor through positive and negative terminals, and thestep-down converter as well in the same way. This
converter will be connected to the power supply. And allof the components are connected to the same ground on
the used breadboard.

PMDC motor
A PMDC motor is chosen for this project dues to its
high torque and low power consumption, the kind of
PMDC motor is chosen based on its torque speed, andthis parameter is chosen after choosing the trolley itself
and deciding which size is preferred to bring this idea tolife.
43A H-Bridge driver
If any motor would work on its own it would work in
only one direction (forward), however in order for us tocontrol how the motor we are using, in this case, a PMDC
motor needs a driver to make the motor move forward or
backward, this driver will be receiving the orders fromthe main system core (raspberry pi 4) to tell what shouldbe the motor’s next move, in addition to what velocity
should it move with, as well as the direction it should beheaded to upon the logic added to this core.

For connecting the ultrasonic sensor with the raspberry
pi to achieve the second objective, first the ultrasonicsensor is set at the front of the trolley to be able to detect
the person and calculate his walked distance, there are
four main wires for this connection, one from the VCC(ultrasonic sensor) to the positive railway, ground wire to
the negative railway on breadboard. As for the raspberrypi the GPIO pin 5 volts which is pin number twoconnected to the positive railway of the breadboard, as
well as pin number six is connected to the negative
railway where I plugged in the ultrasonic sensor. Thenconnecting the trig connection of ultrasonic sensor to
GPIO 23 which is pin number 16 through a blank rail on
the breadboard, and finally connecting ECHO in theultrasonic sensor with GPIO 24 which is pin number
eighteen through blank rail on the breadboard. Fig. 2

LM2596 3A Buck module with display
A converter is in order as its very important when it
comes to prototyping, as different components are chosen
for this project and each work at a very specific level ofvoltage, so the converter’s job is to step down the voltage
in order not to ruin the components.
2.2. Software
The most important library used for this project is
OpenCV python has the option of image processing that

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
201

Dina Koushek, Mastaneh Mokayef, MHD Amen Summakieh, M. K. A Ahamed Khan, Abdul Qayyum, Sivajothi A/L Paramasivam.

shows the primary fabrication of the proposed smart
trolley.

the while loop, the photos (frames) captured will beturned into a grayscale for a better quality formation. The
next step for all the faces detected in each frame is a bluebox will appear marking all of them that will keep
appearing even if the people move around in the camera’s
region the system will wait for the person using thetrolley to press the “ESC” button if they would like to end
the shopping journey. If the person presses this button alldata will be destroyed, and the system will be ready forthe next customer. The result of face detection is
illustrated in Fig. 3.
Supported Sustainable Development Goals
The Sustainable Development Goals ("SDGs") areintended to direct international development efforts
during the ensuing fifteen years, from 2015 to 2030. They
are planned to be endorsed by the United Nations General
Assembly (UNGA) in 2015 as part of its development
agenda. The Millennium Development Goals ("MDGs"),
which were in effect from 2000 to 2015, have been
replaced. The sustainable development means to makethe world a better place without destroying thepossibilities for the next generations keeping three life
aspects in mind which are social progress, economic
development and climate and environment. The smart
follower trolley for elderly care project supports as
explained by Pogge [10].
4. Conclusion
The fabrication of a smart assistive trolley specificallydesigned for elderly care has brought about a remarkableadvancement in enhancing the shopping experience forsenior citizens. The incorporation of face detection
technology and walking distance estimation capabilities
in this robotic trolley has resulted in increased comfort,
convenience, and a greater sense of independence for theelderly. These improvements ultimately contribute toenhancing the overall quality of life for older individuals,
making their shopping experiences more enjoyable andempowering.

Fig. 2 Automated Trolley
3.1. Software Results

References
1.

Fig. 3. Face Detection
2.

The performance or real-time face detection is one of
the goals of this project first the numpy and OpenCV
libraries were added to the code, and then adding the
XML file that contains all of the important knowledge foraccomplishing the face detection for us to enable thecamera to recognize the person in front of it, afterward avideo of size 640 cm x 480 cm starts that its only
functionality is to keep capturing images, then a while
loop is created for the continuous photo capturing. Inside3.

Y. L. Ng, C. S. Lim, K. A. Danapalasingam, M. L. P. Tan,and C. W. Tan, ‘Automatic human guided shopping trolleywith smart shopping system’, J Teknol, vol. 73, no. 3, pp.
49–56, 2015, doi: 10.11113/jt.v73.4246.
E. Li, L. Bi, and W. Chi, ‘Brain-controlled leader-follower
robot formation based on model predictive control’, inIEEE/ASME International Conference on AdvancedIntelligent Mechatronics, AIM, Institute of Electrical andElectronics Engineers Inc., Jul. 2020, pp. 290–295. doi:
10.1109/AIM43001.2020.9158815.
S. Sanghavi, P. Rathod, P. Shah, and N. Shekokar, ‘Design
and implementation of a human following smart cart’, inProceedings of the 3rd International Conference on Smart
Systems and Inventive Technology, ICSSIT 2020, Institute©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
202

Powered by TCPDF (www.tcpdf.org)

Smart Assistive Trolley for

4.

5.

6.
7.
8.

of Electrical and Electronics Engineers Inc., Aug. 2020, pp.
1142–1149. doi: 10.1109/ICSSIT48917.2020.9214217.
S. Nagarajan, P. Banerjee, W. Chen, and B. A. Chin,‘Control of the Welding Process Using Infrared Sensors’,
1992.
H. K. Lee, S. Il Chang, and E. Yoon, ‘Dual-Mode
Capacitive Proximity Sensor for Robot Application:
Implementation of Tactile and Proximity Sensing
Capability on a Single Polymer Platform Using Shared
Electrodes’, IEEE Sens J, vol. 9, no. 12, pp. 1748–1755,
2009, doi: 10.1109/JSEN.2009.2030660.
Raspberry Pi Trading Ltd., ‘Raspberry Pi 4 Computer
Model B’, 2019.
‘Arduino Memory Guide | Arduino Documentation’.
Accessed: Jul. 28, 2023.

Dr. M. KA. Ahamed Khan
He completed his undergraduate degree
in Electronics and Communication
Engineering and postgraduate degree inElectronics and Control Engineering inIndia. Additionally, he obtained anadvanced diploma in Power Electronics
and Drives from Lucas-Nuelle inGermany in 2002, and a Diploma in Drives and Controls
from Woo Sun in Korea in 2014. He pursued a PhD in
Robotics, Power Electronics, and Controls in the UnitedStates and holds certifications. He is a Senior member of
the IEEE in the USA and a member of MIET in the UK.
Dr. Abdul Qayyum
He received his PhD in Electrical andElectronic Engineering from Universiti
Teknologi Petronas, Malaysia, in 2018.
He developed the totype. Previously, hecompleted his bachelor's degree incomputer engineering and master's
degree in electronic engineering from
Pakistan. He is a postdoctoral researcher at the Universityof Burgundy, France, where he studies cardiac MRI images
using deep learning approaches. He is currently affiliatedwith the National Heart and Lung Institute at Imperial
College London, UK.

DC Motor speed control and measurement | Arduino
Project Hub’.

‘Python - Process images of a video using OpenCV GeeksforGeeks’.
10. T. Pogge and M. Sengupta, ‘The Sustainable Development
Goals (SDGs) as Drafted: Nice Idea, Poor Execution’,Washington International Law Journal, vol. 24, 2015.
9.

Authors Introduction
Ms. Dina Ashraf Mohamed Talaat Koushek
She is currently pursuing a Bachelor's
degree in Electronics Engineering(Communication) with Honors at UCSI
University Malaysia.

Dr. Sivajothi A/L Paramasivam
He completed MEng in Manufacturing
Management at the University of South
Australia and then received his PhD in
Mechanical Engineering from Universityof Northumbria, Newcastle, UK. He is a
Senior Lecturer, currently Head of
Department for school of Engineering inthe same university. He is interested in
discipline-based and pedagogic research related toteaching and its associated framework, etc.

Dr. Mastaneh Mokayef
She has received her PhD from WirelessCommunication Centre Faculty ofElectrical Engineering in University
Technology Malaysia (UTM) in 2014.
She has also obtained her master’s degreefrom the faculty of engineering in 2009
from the University Technology
Malaysia. She has been working in UCSI
University, Malaysia, since 2015 in which she currently
serves as an Assistant Professor in the Faculty ofEngineering and Built Environment. Her research interests
include: Wireless communications, spectrum sharing
method, spectrum management, etc.
Mr. MHD Amen Summakieh
He received the B.Eng. degree (Hons.) in
communication
andelectronics
engineering from UCSI University,
Malaysia, in 2016, and the M.Eng.Sc.
degree from Multimedia University,
Malaysia, in 2020. His research interests
include heterogeneous LTE-advanced
cellular networks, user association,
metaheuristic algorithms, and antennas design.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
203

Parallel Cross Window Attention Transformer and CNN Model for Segmentation ofInstrument during Surgery
Abdul Qayyum

Imperial College, London, United Kingdom, Email: a.qayyum@imperial.ac.uk
Steven Niederer

The Alan Turning Institute, London, United Kingdom
M. K. A. Ahamed Khan

Department of Mechanical and Mechatronics Engineering, Faculty of Engineering, Technology and Built Environment,
UCSI University, Malaysia.
Email: mohamedkhan@ucsiuniversity.edu.my
Moona Mazher

Centre for Medical Image Computing, Department of Computer Science, University College London, UK
Email: m.mazher@ucl.ac.uk
Imran Razzak

University of New South Wales, Sydney, Australia
Mastaneh Mokayef

UCSI University, Faculty of Engineering, Malaysia
C. S. Hassan
UCSI University, Faculty of Engineering, Malaysia
Ridzuan, A
UCSI University, Faculty of Engineering, Malaysia

Abstract
Precise segmentation of surgical instruments is a fundamental component in the development of computer-aided
surgery systems by assisting the surgeons to navigate the patient’s body aiming to enhance the surgical precision andpatient safety. Though real-time tracking of surgical instruments is critically important in invasive computer-assisted
surgeries, it is challenging to achieve a highly sensitive and accurate system in complex surgical environment.
Recently, synthetic data for instrument segmentation in surgery (Syn-ISS) challenge using synthetic datasets isorganized to develop high performance methods for instrument segmentation. In this work, we present encoder anddecoder-based hybrid parallel cross window attention-based transformer during the feature extraction, which consistsof the multi-scale channel attention, convolutional layers, and Transformer layers, forming a unified block. Syn-ISS
challenge dataset comprised of two tasks. In first task1, they need to develop deep learning-based method for binary
instrument segmentation and in second task multiclass instrument segmentation is required. Experiments conducted
on Syn-ISS dataset achieved 0.993 F-score for task 1 and 0.993, 0.975, and 0.951 F-score for shaft, wrist, and jaw
segmentation respectively for Task 2.
Keywords: Deep Learning, Parallel Cross Window Attention, Transformer, 2D Instrument Surgery segmentation,
Dense Net.

to the confined working space and limited visual field-ofview. These visual obstructions elevate the risk of tissue
scars and tears. Therefore, the crucial task is totransparently remove or mask the surgical instrumentsfrom the background and subsequently fill the masked
region with appropriate background content. Automatedsegmentation of surgical instruments in MIS is currently
a focal point of research due to its significant practicalapplications [2].

1. Introduction
Minimally invasive segmentation using optical imaging
systems have gained popularity in modern healthcare dueto their advantages, including reduced patient recovery
time and lower mortality rates. Optical imaging has
enabled the use of robotic platforms such as the da Vinci
surgical system by Intuitive Surgery for complex
minimally invasive surgeries [1]. Nevertheless, duringendoscopic surgical suturing procedures, the presence of
surgical instruments can impede surgeons’ dexterity due©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
204

Abdul Qayyum, M. K. A. Ahamed Khan, Moona Mazher, Imran Razzak, Mastaneh Mokayef, C. S. Hassan, Ridzuan, A

The challenges associated with surgical instrument
segmentation are diverse and contingent upon factors
such as the source of dataset acquisition, the type of
surgical procedure, the specific instruments or tools inuse, image resolution, dataset scale, tool characteristics,
and challenging conditions like occlusions, rapid
appearance alterations, specular reflections, smoke, blur,and blood spatter. Segmentation of surgical instruments
has been formulated using both instance segmentation [2]and semantic and segmentation [2].

2.2. Parallel
Cross
Transformer Block:

ii.

iii.

Attention

An efficient hybrid segmentation framework consistingof integration of convolutional neural network andlearnable global attention heads using Efficient ParallelCross Attention module. We use the depth-wise separable
convolution as an efficient version of convolution
implemented by depth wise conv and pointwise conv,
where the depth wise convolution gathers the spatial
information while the pointwise convolution gathers
along the channel dimension. Furthermore, we haveconcatenated features from multi-window transformer
block with depth wise convolutional layer. In encoderside, we have used transformer-based block aided withcross attention window-based mechanism, however, wehave used normal 2DCNN based module at decoder-side.
The features are concatenated from window area partition
and window partition and further these features areconcatenated with depth wise convolutional layer
features and then pass these features to next layer of theencoder block.

Recently, Syn- ISS (synthetic data for instrument
segmentation in surgery) challenge using synthetic
datasets is organized to develop high performance
methods for instrument segmentation [3]. Recently, there
is a different method has been proposed using medical
imaging and signals [4], [5], [6], [7], [8] for classificationand segmentation. Based on our previous work on
segmentation [6], [7], [8], [9]. We presented encoderdecoder based hybrid transformer and CNN model for
instrument segmentation with parallel cross window
attention-based transformer block in encoder and 2D
Dense Net layer CNN blocks in decoder. The main
contribution in this work is:i.

Window

Unlike the vision transformer (ViT) that computes
relationship between tokens at each step of self-attention
module, swin transformer is based on computation of
attention within partition of non-overlapping local
windows of lower resolution feature map and original
image. In contrast to the original swin transformer thatuses patch merging layer to empower it for pixel leveltasks, we used rectangular-paralleled-piped windows toaccommodate non-square images using Parallel-Cross
Attention approach (window area partition and window
partition). To extract different feature maps from eachconvolutional block with parallel window-basedtransformer in encoder, each block consists of Parallel
Cross Window Attention Transformer block, patch
merging and depth wise [7] convolutional layer. The
proposed block is shown in Figure 1 (b).

Developed Parallel Cross Window Attention
Transformer encoder block for 2D
segmentation task.
Proposed 2D Dense Net block at decoder sideof proposed model using transformer-based
encoder features.
Compare performance on synthetic data for
instrument segmentation in surgery for binary
and multiclass surgery instrument
segmentation.

2. Methodology
2.1. Parallel Cross Window Attention Transformer
and CNN model for instrument segmentation

2.3 Convolutional Neural Network (CNN) block:

Due to the intrinsic locality of convolution, which isincapable of modelling long-range dependencies. In
addition, Transformer generates single scale featureswith only token wise attention, and it ignores therelationship among channels, thus subpar to tackle
situations such as segmenting multi scale lesion regions
in medical images. Considering these issues, we
introduce the hybrid Transformer block during thefeature extraction, which consists of the multi-scalechannel attention, convolutional layers, and Transformer
layers, forming a unified block. Our proposed model isbased on encoder and decoder layers, and we haveproposed a hybrid transformer and CNN model for
instrument segmentation. Our proposed model consistedof Parallel Cross window attention-based transformer
block on the encoder side and 2D Dense Net layer CNN
blocks on the decoder side. We have used Dense Net 201
based layers on the decoder side. The proposed model isshown in Figure 1 (a) and Figure 1 (b).

To better encode spatial location information and injectstrong inductive bias, we adopt convolutional block to
extract local spatial features. Specifically, given an input
feature F_(i-1), we adopt the convolutional block tomodel local spatial features, which are shown as follows:
𝐹𝑐𝑖 = 𝐶𝑜𝑛𝑣𝐵𝑙𝑜𝑐𝑘𝑠𝑖 (𝐹𝑖−1 ), 𝑖 ∈ {1,2,3,4}
𝐻

×

𝑊

×𝐶

(1)

𝑤ℎ𝑒𝑟𝑒 𝐹𝑐𝑖 ∈ 𝑅2𝑖+1 2𝑖+1 𝑖 is the local features, whichcontains 2D spatial location information, making it
possible to encode position information. The
convolutional blocks in each decoder stage consisted of
DensNet-201 based architecture is shown in Figure 1 (a).

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
205

Parallel Cross Window Attention

We adopted data augmentation of horizontal flips,
vertical flips, and random rescales. The network is trainedfor 200 epochs using the Adam optimizer and the weightdecay is 0.0001. We have used binary cross-entropy anddice loss used for training and optimization [10], [11].
[12]. The dataset has different spatial size, hence, we
resized each image and label sample to 512x512, whereas
we resized each sample to original input 2D image sizeby bilinear interpolation during inference time. The 25batch size is used during training. The model is training
on A6000 GPU machine with 4 GPUs and all model
codes are developed from scratch using Pytorch Library.

Fig. 1 (a) The proposed model based on hybridtransformer and Dense Net CNN for instrument andinstruments parts segmentation

4.0 Result
The training dataset has been divided into 80 percent
training and 20 percent validation. We have trained andvalidated our proposed model using 5-fold crossvalidation and based on the best validation score, theproposed model has been submitted for task 1 binary
segmentation and task 2 multiclass segmentation. We
have evaluated the performance using IOU, F-Score,
Recall, Precision, and HD. The best score produced byour proposed model based on validation dataset is shownin Table.1 for Task 1 and Task 2.

Fig. 1 (b) The proposed Transformer block based onparallel cross window depth wise attention module

Table 1. The performance analysis of proposed solution
for Task1 and Task2.

In the encoder block, the spatial input size has been
reduced with an increasing number of feature maps andon the decoder side, the input image spatial size has been
increased using a 2D Conv-Transpose layer. The input
features’ maps that are obtained from every encoder
block are concatenated with every decoder block feature
map to reconstruct the semantic information. The spatial
size doubled at every encoder block and feature maps arehalved at each decoder stage of the proposed model. The
feature concatenation has been done at every encoder anddecoder block except the last 1x1 convolutional layer.
The three-level deep-supervision techniques are applied
to get the aggregated loss between ground truth andprediction.

Algo
rith
ms
Propo
sed
mode
l

Ta
sk
s
Ta
sk
2

class
es

IO
U

FScoreRecallPrecis
ion

HD

shaft

0.98
68
0.94
97
0.90
86
0.98
68

0.99300.99230.99200.97530.9506

0.97530.95230.96950.9357

0.99330.99340.9909

14.2
3
10.6
7
12.8
8
0

wrist
jaw

Propo
sed
mode
l

Ta
sk
1

instrument

The visualization of validation input image for binary
and multiclass segmentation is shown in Figure 2. Our
proposed produced similar prediction masks as comparedto ground-truth segmentation masks.

3.0 Experiment
3.1 Dataset
Recently, Syn-ISS challenge [3] is introduce in
MICCAI-2023 to develop high performance methods for
instrument segmentation. Syn-ISS dataset is synthetic
instrument segmentation datasets consist of two maintasks. The task-1 is a binary segmentation to annotate all
pixels that contain an instrument and consist of 1200
instances of simulated scene along with computer
generated corresponding masks. Task-2 further focuseson segmentation of pixels belonging to different parts of
the instrument and consists of 1800 instances.

input

input

3.2 Network Setting©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
206

Abdul Qayyum, M. K. A. Ahamed Khan, Moona Mazher, Imran Razzak, Mastaneh Mokayef, C. S. Hassan, Ridzuan, A

Ground-truth

Ground-truth

Prediction

Prediction"AIROGS: Artificial Intelligence for robust glaucoma
screening challenge." IEEE Transactions on Medical Imaging(2023).
6. Chen, Zhihao, Alain Lalande, Michel Salomon, Thomas
Decourselle, Thibaut Pommier, Abdul Qayyum, Jixi Shi, Gilles
Perrot, and Raphaël Couturier. "Automatic deep learning-based
myocardial infarction segmentation from delayed enhancement
MRI." Computerized Medical Imaging and Graphics 95 (2022):
102014.
7. Qayyum, Abdul, Mona Mazhar, Imran Razzak, andMohamed Reda Bouadjenek. "Multilevel depth-wise context
attention network with atrous mechanism for segmentation of
COVID19 affected regions." Neural Computing andApplications (2021): 1-13.
8. Ahmad, Iftikhar, Abdul Qayyum, Brij B. Gupta, Madini O.
Alassafi, and Rayed A. AlGhamdi. "Ensemble of 2D residual
neural networks integrated with atrous spatial pyramid poolingmodule for myocardium segmentation of left ventricle cardiac
MRI." Mathematics 10, no. 4 (2022): 627.
9. Ahmad, R. F., Malik, A. S., Kamel, N., Amin, H., Zafar, R.,
Qayyum, A., & Reza, F. (2014, November). Discriminating thedifferent human brain states with EEG signals using Fractal
dimension: A nonlinear approach. In 2014 IEEE International
Conference on Smart Instrumentation, Measurement andApplications (ICSIMA) (pp. 1-5). IEEE.
10. Qayyum, Abdul, Moona Mazher, Tariq Khan, and ImranRazzak. "Semi-supervised 3D-InceptionNet for segmentationand survival prediction of head and neck primary cancers."
Engineering Applications of Artificial Intelligence 117 (2023):
105590.
11. Eisenmann, Matthias, Annika Reinke, Vivienn Weru, Minu
Dietlinde Tizabi, Fabian Isensee, Tim J. Adler, Patrick Godauet al. "Biomedical image analysis competitions: The state ofcurrent
participation
practice."
arXiv
preprint
arXiv:2212.08568 (2022).
12. Qayyum, Abdul, Aamir Malik, Naufal M Saad, and Moona
Mazher. "Designing deep CNN models based on sparse coding
for aerial imagery: a deep-features reduction approach."European Journal of Remote Sensing 52, no. 1 (2019): 221-239.

Fig. 2 The input, ground-truth, and predictedsegmentation masks. shaft, wrist, and jaws are shownusing a yellow, red, and green segmentation mask.
5.0 Conclusion
Ideally, surgical instrument segmentation is used inreal time, to identify tools being used as a surgery is being
performed. The integration of surgical instrument
segmentation into computer-aided surgery systems offers
numerous benefits, including real-time guidance,
instrument tracking, and improved surgical outcomes. In
this work, we presented an encoder and decoder-based
hybrid parallel cross window attention-based transformer
which consists of the multi-scale channel attention,
convolutional layers, and Transformer layers.
Experiments conducted on Syn-ISS challenge dataset
achieved 0.993 F-score for task-1 and 0.993, 0.975, and
0.951 F-score for shaft, wrist, and jaw segmentation
respectively for Task 2.

Authors Introduction
Dr. Abdul Qayyum
He is currently working at National
Heart and Lung Institute Imperial
College London, UK. Previously, he
was joined as lecturer at University ofBourgogne Franche-Comté France.
He received his Ph.D in electrical &electronics
engineeringwithspecialization in deep learning andimage processing in 2017 from Universiti Teknologi
Petronas Malaysia. His area of interest is machinelearning, deep learning and quantum machine learning forsignal processing and bioemdical imaging.

6. References
1. Cristina Gonz´alez, Laura Bravo-S´anchez, and Pablo
Arbelaez. Isinet: an instancebased approach for surgicalinstrument segmentation. In International Conference on
Medical Image Computing and Computer-Assisted Intervention,
pages 595–605. Springer, 2020.
2. Mobarakol Islam, VS Vibashan, and Hongliang Ren. Ap-mtl:
Attention pruned multi-task learning model for real-time
instrument detection and segmentation in robot-assisted surgery.
In 2020 IEEE international conference on robotics andautomation (ICRA), pages 8433–8439. IEEE, 2020.
3.https://www.synapse.org/#!Synapse:syn50908388/wiki/6205
16
4. Payette, Kelly, Hongwei Bran Li, Priscille de Dumast,
Roxane Licandro, Hui Ji, Md Mahfuzur Rahman Siddiquee,
Daguang Xu et al. "Fetal brain tissue annotation andsegmentation challenge results." Medical Image Analysis 88(2023): 102833.
5. De Vente, Coen, Koenraad A. Vermeer, Nicolas Jaccard, He
Wang, Hongyi Sun, Firas Khader, Daniel Truhn et al.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
207

Powered by TCPDF (www.tcpdf.org)

Parallel Cross Window Attention

Prof. Steven Niederer
He completed his undergraduate
degree in Engineering Science at the
University of Auckland in 2003 andhis DPhil at the University of Oxford
in 2008. In 2023, he moved toImperial College London as the Chair
in Biomedical Engineering at the
National Heart and Lung Institute. His
current work is focused on reducing barriers to adopting
digital twin technology, developing virtual patient cohorts
for in-silico trials, mapping organ scale function throughto cellular and molecular physiology, and using modellingand simulation to personalize and guide therapies.

Dr Cik Suhana Hassan

She currently works at UCSI
University. She received her PhD
from UTP Malaysia. She has been
doing research in MechanicalEngineeringandMaterials
Engineering. She is passionate about
turning environmental waste intovalue-added products as part of her quest to live a moreenvironmentally friendly life.
2.

Ts Amar Ridzuan Bin Abd Hamid
He is a lecturer of Mechanical andMechatronic
programmes
from
DepartmentofMechanicalEngineering,
UCSI
University,
Malaysia. He has completed his
Master Degree from Universiti Putra
Malaysia, Postgraduate Diploma from
UCSI, and a Bachelor Degree with
Hounours in Mechanical Engineering (Automotive) from
Universiti Teknikal Malaysia Melaka (UTeM), Malaysia.

Dr M. K. A. Ahamed Khan
He is currently working at UCSI
University, Malaysia. He received his
Ph.D in Robotics and controls from
USA. His area of research is robotics,
AI and controls. He has publishedmore than100 papers. He is also anIEEE Senior member. He is also thepast chair for IEEE RAS Malaysia chapter
Dr. Moona Mazher
She is a senior postdoc research
fellow at Department of Computer
Science, University College London.
She received her Ph.D. from the
University of Rovira i Virgili, Spain,
212 a specialization in Neuroscience
from
Universiti
Teknologi
PETRONAS, Malaysia in 2017.. Her
areas of interest are machine learning, deep learning,
medical imaging, signal processing, computer vision, andexplainable AI.
1.
Dr. Imran Razzak
He is a Senior Lecturer in HumanCentered Machine Learning in the
School of Computer Science andEngineering at University of New
South Wales, Sydney, Australia.
Previously, he was as a SeniorLecturer in Computer Science at
School of IT, Deakin University, Victoria. His area of
research focuses on connecting language and vision for
better interpretation of multidimensional data and spansover three broad areas: Machine Learning, Computer
Vision, and Natural Language Processing.
Dr. Mastaneh Mokayef
He is currently working at National
Heart and Lung Institute Imperial
College London, UK. Previously, he
was joined as lecturer at University ofBourgogne Franche-Comté France.
He received his Ph.D in electrical &electronics
engineeringwithspecialization in deep learning andimage processing in 2017 from Universiti Teknologi
Petronas Malaysia. His area of interest is machinelearning, deep learning and quantum machine learning forsignal processing and bioemdical imaging.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
208

Magnetic Resonance Spectroscopy (MRS) Reconstruction Using Style Transfer Deep Depth
wise Framework
Abdul Qayyum

National Heart & Lung Institute, Imperial College, London, United Kingdom,
Email: a.qayyum@imperial.ac.uk
Steven Niederer

National Heart & Lung Institute, Imperial College, London, United Kingdom
Alan Turning Institute, London, United Kingdom
M. K. A. Ahamed KhanUCSI University, Faculty of Engineering, Malaysia
Moona Mazher

Centre for Medical Image Computing, Department of Computer Science, University College London, UK
Imran Razzak

University of New South Wales, Sydney, Australia
Mastaneh Mokayef

UCSI University, Faculty of Engineering, Malaysia
C. S. Hassan

UCSI University, Faculty of Engineering, Malaysia
M. Ridzuan, A

UCSI University, Faculty of Engineering, Malaysia

Abstract
The human brain is a complex and heterogeneous organ composed of distinct compartments such as cerebral cortex,
the cerebellum, the brainstem, and the subcortical regions. To analyze the chemical composition of tissues in brain,
in vivo magnetic resonance spectroscopy allows non-invasive measurements of neurochemicals in either single voxel
or multiple voxels. The reconstruction spectra using 1/3rd of original data than current Edited-MRS scans will not
only result in four times faster edited-MRS scans but also extensively reduction in radiations. In this work, we present
a deep depth-wise channel attention module (DCAM) based fine-tuned network for magnetic resonance spectroscopy
image reconstruction. Besides, we have used channel-wise convolutions and average pooling without dimensionality
reduction. We have trained the initial network from scratch on track-1 simulated dataset, however due to the limited
dataset, we finetune the network on track-2 and track-3. Experiments are conducted on Edited-MRS-Rec-Challenge
dataset1 that showed significantly better performance.
Keywords: Deep Learning, Magnetic Resonance Spectroscopy, Style Transfer, depth-wise channel attention,
Reconstruction, Huber loss.

tomography (PET). The biochemical processes that occur
in the brain are highly complex and dynamic, and involve
the activity of numerous neurotransmitters, enzymes,
receptors, and other signaling molecules. Magnetic
Resonance Spectroscopy (MRS) is a non-invasive
imaging technique identical with nuclear magnetic
resonance (NMR) spectroscopy. It uses magnetic fields
and radio waves to analyze the chemical composition oftissues in the human body such as quantification ofGamma Aminobutyric Acid (GABA), which isoverlapped by creatine and glutamate.

1. Introduction
The brain is a highly complex and heterogeneous organthat consists of many different types of cells, including
neurons, glial cells, and endothelial cells. These cells
interact with each other in complex ways to perform awide range of functions, including perception, thought,
movement, and emotion. Due to its complexity andheterogeneity, studying the brain is a challenging taskthat requires a wide range of techniques and approaches.
These include anatomical imaging techniques, such as
magnetic resonance imaging (MRI), magnetic resonancespectroscopy (MRS) and computed tomography (CT),functional MRI (fMRI) and positron emission

MRS provides information about the concentrations ofvarious metabolites within tissues, which can be used todiagnose and monitor diseases. Besides, it can also be©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
209

Abdul Qayyum, Steven Niederer, M. K. A. Ahamed Khan, Moona Mazher, Imran Razzak, Mastaneh Mokayef, C. S. Hassan, M. Ridzuan, Aused to detect tissue changes in stroke and epilepsy. MRS
data is acquired as a series of spectra, each representing
the concentration of a different metabolite [2], [3]. The
spectra are typically acquired in a 3D volume, with eachvoxel (3D pixel) in the volume containing a spectrum andcan provide estimates of GABA levels in the brain,
however, it has limitations and may not always be precise
or accurate. Figure 1 shows the different chemical peaksof a suspected brain tumor. Besides, its high-quality datais needed to efficiently estimate GABA which requires
long scan times. Hence, Edited-MRS reconstruction
challenge is organized to investigate the machinelearning models for spectra reconstruct using 1/3rd ofdata than current Edited-MRS scans, which will not only
result in to four times faster edited-MRS scans but also
extensively reduction in radiations. In this work, we
present a deep depth-wise channel attention module(DCAM) for magnetic resonance spectroscopy image
reconstruction. Besides, we have used channel-wise
convolutions and average pooling without dimensionality
reduction.

has been used at the end of the proposed model. Spectral
input size is reduced with increasing the number of layers
in the encoder block and the spectral input size isrecovered via a 2D up sampling layer using a bilinear up
sampling method in the decoder side. The DCAM block
handles the input feature maps extracted from eachconvolutional block in the encoder side and further passes
these feature maps into the decoder block. The output ofeach encoder block is fed into the proposed DCAM. Later,
the outputs of DCAMs are concatenating them with thecorresponding decoder blocks. The output of the lastencoder block is passed to the DCAM, and the resulting
feature maps are directly concatenated with thecorresponding decoder block. Each max-pooling layer
receives the output from DCAM. In addition to DCAM,
we have implemented different other modules such as
channel attention module (CAM), spatial attention
module (SAM), channel spatial attention Module(CSAM). As we have larger dataset available for task 1,
we trained the proposed model from scratch on track 1.
However, there are only few samples available for track
2 and track 3, thus we have developed the style-transfer
based framework by finetuning the pre-trained model andfreezing the parameters of the encoder and decoder layers
except the bottom and last two layers where the weights
or parameters are finetuned using vivo dataset for track2 and track-3. Figure 3 shows the block diagram offinetune network for track-2 and track-3.

Fig. 1 MRS graph shows the different chemical peaksof a suspected brain tumor [1]Recently deep learning models have been used in
medical image analysis for segmentation, classification,and object recognition [4], [5], [6], [7]. The 2D and 3D
segmentation models have been proposed forsegmentation of medical imaging [8], [9], [10].
2. Methodology
2.1. Proposed
Reconstruction

Depth-Wise

Fig. 2 The proposed model used for track1 simulated
dataset

Edited-Mrs

In this work, we presented UNet based model for MSR
reconstruction tasks. Figure 2 and Figure 3 show the
proposed reconstruction framework that consists ofconvolutional blocks, Max-Pooling blocks, Up-Sampling
blocks, 1x1 Conv, depth wise channel attention module(DCAM) blocks (see figure 4. The DCAM block is
introduced in each encoder block after the convolutional
layer block. The convolutional block consists ofconvolutional layers with Batch-Normalization and
ReLU activation function to extract the different featuremaps from each block in the encoder side. The 2D maxpooling layer has been used to reduce the input spectral
size. The 1x1 convolutional layer with SoftMax function

Fig. 3 The proposed model used for track2 and track3
vivo dataset©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
210

Magnetic Resonance Spectroscopy (MRS))

The dataset consists of 5500 spectra (5000 ground-truth
spectra and 500 corrupted spectra). The evaluation of the
proposed model is performed on an evaluation set which
has similar features to that of train set. To evaluate theperformance, we have used mean squared error (MSE),
signal to noise ratio (SNR), linewidth (FWHM), peak
shapes GABA, GLX (PS). Evaluation of model is basedon task (simulated, in vivo homogeneous, in vivo
heterogeneous) and each metric is ranked individually
and well as combined (MSE, SNR, FWHM, PS with ratioof 40%,20%,2 0%, 20%, 20% respectively).
2.4 Parameter Setting

Fig. 4 Diagram of the proposed depth-wise channel
attention module (DCAM)

In this work, we β decreases from 15 to 1 during thetraining for each epoch. We have trained proposedmodels using Adam optimizer and dynamics Huber loss
function using 0.0001 learning rate and 200 epochs. The
dataset is normalized using clip [0,1] window. All models
use PyTorch and trained on Tesla V100 GPU machine[11], [12]. We have converted all dataset time series intofrequency spectrum. The challenge organizers provided
simulated and in vivo data training sets representing
GABA-edited MEGA-PRESS scans composed of two
sub spectra (ON and OFF).

Figure 4 is proposed depth-wise channel attention
module (DCAM) used in encoder side of the proposedmodel. In this experiment, we have used two loss
functions: Mean Squared Error (MSE) and Dynamic
Huber loss (LHuber). MSE is sensitive towards outliers
whereas L Huber is less sensitive to outliers than MSE
loss. For N predictions, MSE loss function is defined by;1
̌ 2
𝐿𝑀𝑆𝐸 = ∑𝑁𝑖 (𝑋𝑖 − 𝑋𝑖 )𝑁

(1)

3. Results

N and 𝑋 = (𝑥1 , 𝑥2 , … , 𝑥𝑁 ) are the ground truth MRI
̌spectrumand𝑋̌ = (𝑥
̌,
̌,
are
their
1 𝑥
2 … , 𝑥𝑁 )corresponding estimated MRI spectrum. The huber loss
function is defined;1

𝐿𝐻𝑢𝑏𝑒𝑟 = ∑𝑁𝑖 𝑧𝑖𝑁We have used scripts provided by organizers to add
varying noise, frequency, and phase shifts, that were used
for data augmentation purposes. We have developed
different variations of proposed framework (depth-wisechannel attention module (DCAM), channel attention
module (CAM), spatial attention module (SAM), channel
spatial attention Module (CSAM)) and compared theperformance. The frequency spectrum has been used as
input to train our proposed model for task 1, 2 and 3. We
have trained proposed model parameters from scratchusing 80 percent simulated data in frequency domain andvalidated using 20 percent of the frequency spectra ofsimulated dataset. In task two, vivo data (i.e., singlevendor data) in frequency domain has been used to finetune the trained model that was trained using simulated
dataset. Similarly, for task3, the heterogeneous in vivo
data with different frequency spectra has been used totrain the proposed model and later fine-tuned the
proposed model using different number of epochs. We
have used dynamic Huber and Dynamic Tukey loss
function for training and optimizing the parameters of our
proposed model. The combined loss (Huber and Tukey)
loss function provided better performance on validation
dataset. Figure 5 shows the comparative analysis. Table
1 describes the evaluation results of proposed edited
magnetic resonance spectroscopy reconstruction
framework. We can notice that our proposed framework
achieved significantly better mean squared error (8.89e03), signal to noise ratio (4.61e+01), linewidth (9.17e-02)
and peak shapes score (9.44e-01) respectively incomparison to baseline method and other methods.

(2)

Where 𝑁 is the batch size and 𝑧𝑖 is defined by;0.5(𝑥𝑖 − 𝑥̌𝑖 )2 ,𝑖𝑓 |𝑥𝑖 − 𝑥̌𝑖 | ≤ 𝛽𝑧𝑖 = {
2|
𝛽|𝑥𝑖 − 𝑥̌𝑖 − 0.5𝛽 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒(3)

Where 𝛽 is controlling hyperparameter. In our
experiment, β decreases from 15 to 1 during the training
for each epoch.
2.2. Simulation Results
In this section, we briefly first describe the dataset,followed by evaluation parameters, experimental setting,and evaluation. We have used preprocessing step
provided by challenge organizer to convert dataset into
Frequency domain spectra and added some noises(random Gaussian amplitude, random Gaussian
frequency and random Gaussian phase noise, combined).
2.3 Dataset and Evaluation Matrix
The test set may differ in traditional features, such as byvendor and echo time and it will be lower quality data i.e.,lower main SNR or larger frequency and phase shifts.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
211

Abdul Qayyum, Steven Niederer, M. K. A. Ahamed Khan, Moona Mazher, Imran Razzak, Mastaneh Mokayef, C. S. Hassan, M. Ridzuan, A

Table 1 Comparative evaluation of proposed framework(depth-wise channel attention module (dcam), channel
attention module(cam), spatial attention module(sam),channel spatial attention module (csam)) with baseline
results on task 1, task 2 and task 3
MSE

SNR

Linewidt
Shape
h
Score
Comparative evaluation for track 1
Baseline 4.37e 1.43e+0
2.70e-01
6.89e-02
1
01
Propose 5.79e 3.66e+0
5.50e-02
8.52ed DCAM
-03
1
01
Propose 7.33e 4.11e+0
6.44e-02
9.33ed CAM-03
1
01
Propose 7.88e 5.26e+0
7.37e-02
9.683e
d SAM
-03
1
-01
Propose 6.33e 4.01e+0
6.11e-02
8.10ed CSAM
-03
1
01
Comparative evaluation for track 2
Baseline 4.09e 1.46e+0
2.56e-01
7.09e-02
1
01
Propose 5.79e 3.66e+0
5.50e-02
8.52ed DCAM
-03
1
01
Propose 7.33e 4.11e+0
6.44e-02
9.33ed CAM-03
1
01
Propose 7.88e 5.26e+0
7.34e-02
9.68ed SAM-03
1
01
Propose 6.33e 4.26e+0
6.11e-02
8.12ed CSAM
-03
1
01
Comparative evaluation for Track3 with 2048
spectraBaseline 9.30e 3.77e+0
1.25e-01
9.21e-03
501
Propose 8.26e 5.11e+0
8.18e-02
9.92ed DCAM-04
2
01
Propose 7.12e 7.22e+0
5.27e-02
5.12ed CAM-04
201
Propose 5.32e 6.59e+0
4.22e-02
7.12ed SAM-04
2
01
Propose 7.88e 6.44e+0
6.22e-02
6.33ed CSAM
-04
2
01
Comparative evaluation for Track3 with 4896
spectraPropose 6.92e 5.95e+0
7.68e-02
9.94ed DCAM-04
501
Propose 7.09e 2.29e+0
7.63e-02
9.97ed CAM-04
201
Propose 6.10e 3.11e+0
6.41e-02
9.89ed SAM-04
2
01
Propose 6.88e 3.89e+0
6.11e-02
8.66ed CSAM
-04
2
01
Propose 6.33e 3.22e+0
6.83e-02
8.98ed DCAM
-04
2
01

Fig. 5 Comparative analysis: ground truth vs proposes
depth-wise channel attention module (DCAM) based
reconstruction
4. Conclusion
In this paper, we presented deep depth-wise channel
attention module (DCAM) based fine-tuned UNetarchitecture for magnetic resonance spectroscopy image
reconstruction. Due to the availability of large dataset fortask 1, we have trained network from scratch on track-1
simulated dataset followed by finetuning the network on
track-2 and track-3. Experiments are conducted on
Edited-MRS-Rec-Challenge dataset that showedsignificantly better performance for all parameters i.e.
mean squared error (8.89e-03), signal to noise ratio(4.61e+01), linewidth (9.17e-02) and peak shapes score(9.44e-01) respectively in comparison to baseline method
and other methods.
5. References
1. G. O¨ z, J. R. Alger, P. B. Barker, R. Bartha, A. Bizzi, C.
Boesch, P. J. Bolan, K. M. Brindle, C. Cudalbu, A. Dinc，er etal., “Clinical proton mr spectroscopy in central nervous systemdisorders,” Radiology, vol. 270, no. 3, pp. 658–679, 2014.
2. Z. Su, Z. Liu, M. Wang, S. Li, L. Lin, Z. Yuan, S. Pang, Q.
Feng, T. Chen, and H. Lu, “Three-dimensional reconstructionof kambin’s triangle based on automated magnetic resonanceimage segmentation,” Journal of Orthopaedic ResearchR, vol.
40, no. 12, pp. 2914–2923, 2022.
3. B. Fu, Y. Dong, S. Fu, Y. Mao, and D. N. Thanh, “Learning
domain transfer for unsupervised magnetic resonance imaging
restoration and edge enhancement,” International Journal of
Imaging Systems and Technology,
vol. 32, no. 1, pp. 144–154, 2022.
4. Payette, Kelly, Hongwei Bran Li, Priscille de Dumast,
Roxane Licandro, Hui Ji, Md Mahfuzur Rahman Siddiquee,
Daguang Xu et al. "Fetal brain tissue annotation andsegmentation challenge results." Medical Image Analysis 88(2023): 102833.
5. Qayyum, Abdul, Mona Mazhar, Imran Razzak, andMohamed Reda Bouadjenek. "Multilevel depth-wise context
attention network with atrous mechanism for segmentation of
COVID19 affected regions." Neural Computing andApplications (2021): 1-13.
6. De Vente, Coen, Koenraad A. Vermeer, Nicolas Jaccard, He
Wang, Hongyi Sun, Firas Khader, Daniel Truhn et al.
"AIROGS: Artificial Intelligence for robust glaucoma
screening challenge." IEEE Transactions on Medical Imaging(2023).

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
212

Magnetic Resonance Spectroscopy (MRS))

7. Qayyum, Abdul, Moona Mazher, Tariq Khan, and ImranProf. Steven Niederer
He completed his undergraduate
degree in Engineering Science at the
University of Auckland in 2003 andhis DPhil at the University of Oxford
in 2008. In 2023, he moved to Imperial
College London as the Chair in
Biomedical Engineering at the
National Heart and Lung Institute. His
current work is focused on reducing barriers to adopting
digital twin technology, developing virtual patient cohorts
for in-silico trials, mapping organ scale function throughto cellular and molecular physiology, and using modellingand simulation to personalize and guide therapies.

Razzak. "Semi-supervised 3D-InceptionNet for segmentationand survival prediction of head and neck primary cancers."
Engineering Applications of Artificial Intelligence 117 (2023):
105590.
8. Chen, Zhihao, Alain Lalande, Michel Salomon, Thomas
Decourselle, Thibaut Pommier, Abdul Qayyum, Jixi Shi, Gilles
Perrot, and Raphaël Couturier. "Automatic deep learning-based
myocardial infarction segmentation from delayed enhancement
MRI." Computerized Medical Imaging and Graphics 95 (2022):
102014.
9. Ahmad, Iftikhar, Abdul Qayyum, Brij B. Gupta, Madini O.
Alassafi, and Rayed A. AlGhamdi. "Ensemble of 2D residual
neural networks integrated with atrous spatial pyramid poolingmodule for myocardium segmentation of left ventricle cardiac
MRI." Mathematics 10, no. 4 (2022): 627.
10. Qayyum, Abdul, Chun Kit Ang, S. Sridevi, MKA Ahamed
Khan, Lim Wei Hong, Moona Mazher, and Tran Duc Chung.
"Hybrid 3D-ResNet deep learning model for automatic
segmentation of thoracic organs at risk in CT images." In 2020
International Conference on Industrial Engineering,
Applications and Manufacturing (ICIEAM), pp. 1-5. IEEE,2020.
11. Eisenmann, Matthias, Annika Reinke, Vivienn Weru, Minu
Dietlinde Tizabi, Fabian Isensee, Tim J. Adler, Patrick Godauet al. "Biomedical image analysis competitions: The state ofcurrent
participation
practice."
arXiv
preprint
arXiv:2212.08568 (2022).
12. Qayyum, Abdul, Iftikhar Ahmad, Mohsin Iftikhar, andMoona Mazher. "Object Detection and Fuzzy-BasedClassification Using UAV Data." Intelligent Automation & Soft
Computing 26, no. 4 (2020).

Dr M. K. A. Ahamed Khan
He is currently working at UCSI
University, Malaysia. He received his
Ph.D in Robotics and controls from
USA. His area of research is robotics,
AI and controls. He has publishedmore than100 papers. He is also anIEEE Senior member. He is also thepast chair for IEEE RAS Malaysia chapter
Dr. Moona Mazher
She is a senior postdoc research fellow
at Department of Computer Science,
University College London. She
received her Ph.D. from the Universityof Rovira i Virgili, Spain, 212 a
specialization in Neuroscience from
Universiti Teknologi PETRONAS,
Malaysia in 2017.. Her areas ofinterest are machine learning, deep learning, medical
imaging, signal processing, computer vision, andexplainable AI.

Authors Introduction
Dr. Abdul Qayyum
He is currently working at National
Heart and Lung Institute Imperial
College London, UK. Previously, he
was joined as lecturer at University ofBourgogne Franche-Comté France.
He received his Ph.D in electrical &electronics
engineeringwithspecialization in deep learning andimage processing in 2017 from Universiti Teknologi
Petronas Malaysia. His area of interest is machinelearning, deep learning and quantum machine learning forsignal processing and bioemdical imaging.

Dr. Imran Razzak
He is a Senior Lecturer in HumanCentered Machine Learning in the
School of Computer Science andEngineering at University of New
South Wales, Sydney, Australia.
Previously, he was as a SeniorLecturer in Computer Science at
School of IT, Deakin University, Victoria. His area of
research focuses on connecting language and vision for
better interpretation of multidimensional data and spansover three broad areas: Machine Learning, Computer
Vision, and Natural Language Processing.
1.
Dr . Mastaneh Mokayef
He is currently working at National
Heart and Lung Institute Imperial
College London, UK. Previously, he
was joined as lecturer at UniversityofBourgogne
Franche-Comté
France. He received his Ph.D inelectrical & electronics engineeringwith specialization in deep learningand image processing in 2017 from Universiti Teknologi
Petronas Malaysia. His area of interest is machinelearning, deep learning and quantum machine learning forsignal processing and bioemdical imaging.
2.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
213

Powered by TCPDF (www.tcpdf.org)

Abdul Qayyum, Steven Niederer, M. K. A. Ahamed Khan, Moona Mazher, Imran Razzak, Mastaneh Mokayef, C. S. Hassan, M. Ridzuan, A
Dr Cik Suhana Hassan

She currently works at the Faculty ofEngineering, Technology and Built
Environment, UCSI University. She
received her PhD from UTP
Malaysia. Suhana does research in
MechanicalEngineeringandMaterials Engineering. Dr. Suhana ispassionate
about
turningenvironmental waste into value-added products as part ofher quest to live a more environmentally friendly life.
Ts Amar Ridzuan Bin Abd Hamid
He is a lecturer of Mechanical andMechatronic programmes from
DepartmentofMechanicalEngineering, UCSI University,
Malaysia. He has completed his
Master Degree from Universiti Putra
Malaysia, Postgraduate Diploma
from UCSI, and a Bachelor Degreewith Hounours in Mechanical Engineering (Automotive)
from Universiti Teknikal Malaysia Melaka (UTeM),
Malaysia.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
214

Federated Learning on Brain Disease Research: Segmentation of Cerebral Small Vessel
Diseases (CSVD) using Multi-scale Hybrid Spatial Deep Learning Approach
Moona Mazher

Centre for Medical Image Computing, Department of Computer Science, University College London, UK
Email: m.mazher@ucl.ac.uk
Abdul Qayyum

National Heart & Lung Institute, Imperial College, London, United Kingdom
M. K. A. Ahamed KhanUCSI University, Faculty of Engineering, Cheras, Malaysia
Steven Niederer

National Heart & Lung Institute, Imperial College, London, United Kingdom
Alan Turning Institute, London, United Kingdom
Mastaneh Mokayef

UCSI University, Faculty of Engineering, Cheras, Malaysia
Ridzuan, A.

UCSI University, Faculty of Engineering, Cheras, Malaysia
C. S. Hassan

UCSI University, Faculty of Engineering, Cheras, Malaysia

Abstract
Federated learning is an emerging approach that enables large-scale decentralized learning without the need to sharedata among different data owners. This approach is particularly valuable in addressing data privacy concerns inmedical image analysis. However, existing methods often impose a strict requirement for label consistency acrossclients, which significantly limits its applicability. Various clinical sites may only provide annotations for specificorgans of interest, and there may be limited or no overlap in the labeled data among different sites. The human brain
receives nutrients and oxygen through blood vessels in the brain. The pathology of small vessels, i.e. mesoscopic
scale, is a vulnerable component of the cerebral blood supply and can result in major complications such as Cerebral
Small Vessel Diseases (CSVD). In this paper, we propose a hybrid architecture for medical image segmentation to
produce efficient representations from global and local features and adaptively aggregate them, aiming to fully exploit
their strengths to obtain better segmentation performance in federated learning. Furthermore, we propose a multiscale feature extraction module embedded at the bottom of the proposed model, which can efficiently extract hidden
multi-scale contextual information and aggregate multi-scale features. Experiments on segmentation over threedimensional rotational angiography of internal Carotid Artery with aneurysm (SHINY-ICARUS) challenge dataset
show the effectiveness of the proposed multiscale framework.
Keywords: Federated Learning, Deep Learning, Medical Image Analysis, 3D volume Segmentation, Cerebral Small
Vessel Diseases, Brain angiography.

data owners to collectively train deep learning (DL)models without sharing the raw data. Previous studies [1],[2] have showcased the viability of FL, particularly thefederated averaging (Fe-dAvg) algorithm in the context
of medical image segmentation. Federated Learning (FL),
as a collaborative machine learning technique, aims totackle these challenges by creating a joint predictivemodel across clients at multiple sites, especially within
distributed medical institutions or hospitals. FL allows

1. Introduction
Distributed big data and digital healthcare technologies
hold immense potential for advancing medical services.
However, challenges arise when attempting to develop
predictive models from diverse and intricate e-healthdatasets. Federated learning (FL) [1] has emerged as adecentralized learning paradigm that enables multiple©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
215

All Author’s Full Namefor the collective training of deep learning models using
distinct patient data from various hospitals for various
clinical applications, including medical image
segmentation. Given the non-convex nature of the
training objective in deep neural networks, averaging
locally trained models can result in suboptimal solutions
within the parameter space, potentially leading toperformance degradation [1]. However, a significant
issue with FL is the performance degradation it may
experience when dealing with data that are notindependently and identically distributed (non-iid), a
situation commonly encountered in medical image
datasets. Recent publications by various authors [2], [3],[4], [5] have focused on federated deep learning,
especially for segmentation and classification tasks.
Wicaksana et al. [3] introduced "FedMix: Mixed
Supervised Federated Learning for Medical Image
Segmentation," while Xu et al. [4] presented "Federated
Cross Learning for Medical Image Segmentation."
Additionally, Qiu et al. [5] proposed "Federated SemiSupervised Learning for Medical Image Segmentation
via Pseudo-Label Denoising."

1.

2.

3.

Present a 3D UNet-based segmentation framework
aided with hybrid attention and multi-scale features
for automatic brain vascular segmentation in thefederated learning environment.
The attention-guided feature fusion module exploits
the most useful features (both high-level features andlow-level features) between two adjacent layers andthe multiscale feature module assists the model inextracting precise features at different scales andtraining on each client and communicating with theserver model using the weighting aggregation
approach.
Extensive experiments were conducted on the
SHINY-ICARUS challenge dataset which showed
efficient segmentation of brain vascular.

2. Methodology
3D angiographies of brain vasculature segmentation
especially segmentation of visible vasculature connected
to one or more of the main feeding arteries of the brain isa challenging problem due to its complex nature. To
effectively analyze the vascular morphology andtopology, we present an efficient hybrid attention andmulti-scale feature aggregation 3D deep learning
framework for automatic brain vascular segmentation. In
the following section, we first present the overall
framework followed by details of each component. The
proposed vascular segmentation architecture is based ona classical U-shape encoder–decoder structure [13], [14],[15]. Besides, we have also integrated two core modules(hybrid spatial and channel attention module, multi-scalefeature extractor module) subtly and seamlessly whichhelps to select important multiscale contextual spatial
information and semantic information adaptively. The
hybrid spatial and channel attention module (HSCA)
suppresses the low-level background noise and retains
local semantic information of vessel structure, whereas
the multiscale feature extractor module (MSFE) assists inthe effective extraction of concealed multi-scale
contextual information as well as aggregating multi-scalefeatures. As a results, it helps to enhance the capability of
the network to deal with complex cases where vascularheavily varies in shape and size, and many areintertwined. Figure 1 illustrates the proposed framework.
To extract different feature maps from eachconvolutional block in the encoder, each block consistsof 3D convolutional layers with batch-normalization and
ReLU activation functions. We have used the 3D maxpooling layer to reduce the input spatial size of the image.
Notice that we have reduced the size of spatial input with
the increase in several layers and we have recovered thespatial resolution in the decoder. To recover the spatial
resolution in the decoder, we have applied 3D upsampling using bi-linear up-sampling. In this experiment,
we have used 3 × 3 × 3 kernel size in both the encoderand decoder and the number of feature map numbers to
16, 32, 64, 128, and 256 for each encoder. To downsample the spatial resolution. we have set the kernel size
to 2 × 2 × 2 for the 3D-MaxPool layer on encoder side.

As the expert knowledge usually required forannotating medical images is much more demanding anddifficult to obtain, various medical institutions have very
limited strong pixel-level annotated images and mostavailable images are unlabeled or weakly annotated.
Therefore, a realistic clinical mechanism that utilizes
every available supervision for cross-institutional
collaboration without data sharing is highly desirable. We
propose to develop a label-agnostic Mixed Supervised
Federated Learning approach that efficiently uses datalabeled in any form for medical image segmentation.
Specifically, in the absence of pixel-level labels, it will
effectively utilize unlabeled images as well as useful
information contained in the weakly labeled for
producing and selecting high-quality pseudo labels. We
will devise an effective adaptive weight assignment
across clients, where each client can learn an aggregation
weight. Adaptive weight assignment is essential to handle
inter-client variations in supervision availability.
Recently, the SHINY-ICARUS challenge was organized
to analyze vascular morphology and topology efficiently
to provide a platform and benchmark for brain vascularsegmentation [6]. Recent advancements in deep learningand imaging technology have enabled the development ofadvanced deep learning models and large-scale datasets[6], [7], [8]. Recently a lot of work proposed deeplearning-based segmentation methods for medical
imaging and signals [8], [9], [10], [11], [12]. In this work,
we present an efficient hybrid attention and multi-scalefeature aggregation 3D deep learning framework forautomatic brain vascular segmentation. Experiments are
conducted on the SHINY-ICARUS challenge whichshows the effectiveness of the proposed framework.
The key contributions of this work are:

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
216

Paper Title (First 4 Words)

Finally, we have used a transpose3D convolutional layer
with stride 2 and 2 × 2 × 2 kernel size for up-sampling thesize of each decoder. At the end, we have concatenatedthe output of each encoder block to the corresponding

decoder block. To generate the final output segmentation
map, we have used a 1 × 1 convolutional layer with
softmax function.

Fig. 1. Proposed Multi-scale Feature extraction Module (MSFE) and hybrid spatial and Channel attention module(HSCA) aided brain vascular segmentation framework. E1, E2, E3, E4 are encoders blocks and D1, D2, D3 and D4 aredecoders blocks respectively.
Our approach follows the structure of the FedAvg [1]algorithm, a widely recognized benchmark in the field ofFederated Learning (FL). This approach involves acentral server node and K client nodes. The central
server's main role is to manage the communication andcomputation processes among the various client nodes.
Meanwhile, the client nodes are primarily responsible fortraining the model using their local data and computing
devices. With a universal label set encompassing M
organs, each client node possesses a local dataset 𝐷𝑘 ,which is labeled with a subset of 𝑁𝑘 (≤ 𝑀))organs. Our
objective is to develop a segmentation network F(.,θ) for
all M organs by leveraging the labeled datasets {𝐷𝑘 }𝐾
𝑘=1 ,which are distributed across distinct client nodes andcannot be joined or centralized for training purposes. The
training process within the Federated Learning (FL)framework is comprised of Tcommunication rounds
between the server and client nodes. In each
communication round, denoted as the 𝑡𝑡ℎ round, eachclient node k initially downloads the parameters of thecurrent segmentation network, represented as 𝐹(. , 𝜃 𝑡 )from the server (referred to as the global model). This
process results in a local copy of the model, denoted as
the local model.

Subsequently, the client node proceeds to train its local
model using its local dataset, Dk, for a specified numberof E epochs. Following the local training, the servercollects the trained local models, 𝐹(. , 𝜃𝑘𝑡 ), from all
Kclient nodes and combines them into a new global
model through a parameter-wise averaging process.
𝐾𝜃 𝑡+1 = ∑|𝐷𝑘 |𝐾𝑘 ∑𝑗 |𝐷𝑗 |

. 𝜃𝑡

(1)

Since the local models are trained separately on theclient nodes, the server node is only required to transmit
the model parameters, as opposed to the raw data, fromthe clients. This approach allows the Federated Learning(FL) model to gain insights from distinct client datasets
without compromising data privacy. Figure 2 showed theprocess of federated learning environment for brain
tumor segmentation.
As labels for regions in brain vascular are sparse, hence,
we have used binary cross-entropy and weighted binary
cross-entropy loss [14] which can adjust the learning bias
between vessels and background. Besides, we also useddice similarity coefficient loss [15] to ensure the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
217

All Author’s Full Name

where α is the weight balance parameter between
weighted binary cross entropy (WBCE) and LDSC (Diceloss). We have empirically set α = 0.6.

segmentation of small vessels. Finally, we define the 3D
optimization loss function.
L = BCE + αWBCE + (1 − α)LDSC(2)

Fig. 2 Federated Learning environment for brain tumor segmentation
3. Results and Discussion

SHINY-ICARUS dataset consists of manuallysegmented cerebral arteries NIfTI volumes with 16-bit
representation. The images (resolution 0.227 × 0.227 ×
0.227mm)areacquiredbyinterventional
neuroradiologists from patients presenting intracranial
aneurysms, using a GE INNOVA 3D. The dataset
consists of contract images of most of the patient’s head.
In this experiment, we have divided the dataset into fivefold cross-validation based on the best validation score,
the proposed model uploaded in a docker container. The
challenge organized provided 35 volumes for training the
proposed model. A detailed description of the dataset canbe found [6]. A sample dataset with manual annotation isshown in Figure 3.

In this section, we first present details of the dataset
followed by the experiment and results. We haveevaluated and compared the performance using the Dice,
Jaccard index, volumetric similarity coefficient, andbalanced average Hausdorff distance (HD). SHINYICARUS challenge evaluation is based on two
overlapping matrices such as clDice5 and Dice. ClDices
measure the similarity while favoring both connectivity
and topology. Both clDice5 and Dice carry the same
weight for the evaluation.
3.1. Dataset(a) image(b) mask

Fig. 3 A sample dataset was used in our proposed model.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
218

Paper Title (First 4 Words)

3.2. Experimental Results

PyTorch library and trained from scratch. Figure 4
illustrates the visualization of the validation set. Table 1
shows the results of the proposed model using thevalidation dataset.

For evaluation purposes, the SHINY-ICARUS
challenge provides two test sets (primary and secret). We
have trained the proposed model using patches of input
volume due to the limitation of memory. We have found
160 × 160 × 160 the best patch for training. We havegenerated 10 random patches with a single epoch during
training and optimization. In this experiment, we used
Adam optimizer, batch size 2, and set the learning rate to
0.0001 and epoch to 1000. We have used a combined loss
functions function between the prediction mask and theground truth. We have used early stopping criteria andtraining was ended after 20 epochs on similar validation
dice. We have used NVIDIA GTX 3070 GPU having a
24GB memory machine and developed the model in

Figure 4 shows the predicted and ground truth
segmentation for subject1 and subject2 validation
datasets. The proposed model produced a similar
segmentation mask as compared to ground truth
segmentation. Our proposed model produced
overestimated some branches and we can remove these
extra segmentation branches using connectedcomponents.

Table 1. Results on the proposed model on the validation dataset.
Models

Dice

Re

Pr

clDiceclRe

clPr

3DUNet+MSFE

0.86740.99900.76640.8496

0.99880.7392

3DUNet+HSCA

0.89690.98820.82100.85730.98900.7566

3DUNet+HSCA+MSFE

0.9374

0.94320.93170.89680.95670.8440GT input

GT volume

GT input

GT volume

Predict

Predict Volume

Predict

Predict Volume

Fig. 4 The first shows the prediction and ground truth (GT) segmentation mask for validation subject 1 and the second
row shows results on subject 2.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
219

4. Conclusion
10.

In this paper, we presented an efficient brain vascularsegmentation deep framework aided with hybrid
attention and multi-scale feature aggregation modules.
Experiments are conducted on the SHINY-ICARUS
challenge which shows that our model is ranked 2nd on
the secret test and 3rd on the combo test that validates theeffectiveness of the proposed framework. We havedivided the dataset into five-fold cross-validation andbased on the best validation score, the proposed model
was uploaded in a docker container. Experiment results
showed efficient (dice and clDice5) segmentation ofbrain vascular.

11.

12.

References
13.

1. McMahan, Brendan, Eider Moore, Daniel Ramage, SethHampson, and Blaise Aguera y Arcas. "Communicationefficient learning of deep networks from decentralized
data." In Artificial intelligence and statistics, pp. 12731282. PMLR, 2017.
2. Liu, Quande, Cheng Chen, Jing Qin, Qi Dou, and PhengAnn Heng. "Feddg: Federated domain generalization on
medical image segmentation via episodic learning incontinuous frequency space." In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern
Recognition, pp. 1013-1023. 2021.
3. Wicaksana, Jeffry, Zengqiang Yan, Dong Zhang, Xijie
Huang, Huimin Wu, Xin Yang, and Kwang-Ting Cheng.
"FedMix: Mixed supervised federated learning for medical
image segmentation." IEEE Transactions on Medical
Imaging (2022).
4. Xu, Xuanang, Tianyi Chen, Han Deng, Tianshu Kuang,
Joshua C. Barber, Daeseung Kim, Jaime Gateno, Pingkun
Yan, and James J. Xia. "Federated cross learning for
medicalimage
segmentation."arXiv
preprint
arXiv:2204.02450 (2022).
5. Qiu, Liang, Jierong Cheng, Huxin Gao, Wei Xiong, andHongliang Ren. "Federated Semi-Supervised Learning for
Medical Image Segmentation via Pseudo-Label
Denoising." IEEE Journal of Biomedical and Health
Informatics (2023).
6. Moccia, S. et al., “Blood vessel segmentation algorithms—
review of methods, datasets and evaluation metrics,”
Computer methods and programs in biomedicine 158, 71–
91 (2018).
7. Payette, Kelly, Hongwei Bran Li, Priscille de Dumast,
Roxane Licandro, Hui Ji, Md Mahfuzur Rahman
Siddiquee, Daguang Xu et al. "Fetal brain tissue annotationand segmentation challenge results." Medical Image
Analysis 88 (2023): 102833.
8. De Vente, Coen, Koenraad A. Vermeer, Nicolas Jaccard,
He Wang, Hongyi Sun, Firas Khader, Daniel Truhn et al.
"AIROGS: Artificial Intelligence for robust glaucoma
screening challenge." IEEE Transactions on Medical
Imaging (2023).
9. Chen, Zhihao, Alain Lalande, Michel Salomon, Thomas
Decourselle, Thibaut Pommier, Abdul Qayyum, Jixi Shi,
Gilles Perrot, and Raphaël Couturier. "Automatic deeplearning-based myocardial infarction segmentation from

14.

15.

delayed enhancement MRI." Computerized Medical
Imaging and Graphics 95 (2022): 102014.
Qayyum, Abdul, Mona Mazhar, Imran Razzak, andMohamed Reda Bouadjenek. "Multilevel depth-wise
context attention network with atrous mechanism forsegmentation of COVID19 affected regions." NeuralComputing and Applications (2021): 1-13.
Ahmad, Iftikhar, Abdul Qayyum, Brij B. Gupta, Madini O.
Alassafi, and Rayed A. AlGhamdi. "Ensemble of 2D
residual neural networks integrated with atrous spatial
pyramid pooling module for myocardium segmentation ofleft ventricle cardiac MRI." Mathematics 10, no. 4 (2022):
627.
Qayyum, Abdul, Aamir Malik, Naufal M Saad, and Moona
Mazher. "Designing deep CNN models based on sparse
coding for aerial imagery: a deep-features reduction
approach." European Journal of Remote Sensing 52, no. 1(2019): 221-239.
Ahmad, R. F., Malik, A. S., Kamel, N., Amin, H., Zafar,
R., Qayyum, A., & Reza, F. (2014, November).
Discriminating the different human brain states with EEG
signals using Fractal dimension: A nonlinear approach. In
2014 IEEE International Conference on Smart
Instrumentation,
Measurement
andApplications(ICSIMA) (pp. 1-5). IEEE.
Qayyum, Abdul, Moona Mazher, Tariq Khan, and ImranRazzak.
"Semi-supervised
3D-InceptionNetforsegmentation and survival prediction of head and neckprimary cancers." Engineering Applications of Artificial
Intelligence 117 (2023): 105590.
Eisenmann, Matthias, Annika Reinke, Vivienn Weru,
Minu Dietlinde Tizabi, Fabian Isensee, Tim J. Adler,
Patrick Godau et al. "Biomedical image analysis
competitions: The state of current participation practice."
arXiv preprint arXiv:2212.08568 (2022)

Authors Introduction
Dr. Moona Mazher
She is a senior postdoc research
fellow at Department of Computer
Science, University College London.
She received her Ph.D. from the
University of Rovira i Virgili, Spain,
212 a specialization in Neuroscience
from
Universiti
Teknologi
PETRONAS, Malaysia in 2017.. Her
areas of interest are machine learning, deep learning,
medical imaging, signal processing, computer vision, andexplainable AI.
Dr. Abdul Qayyum
He is currently working at National
Heart and Lung Institute Imperial
College London, UK. Previously, he
was joined as lecturer at University ofBourgogne Franche-Comté France. He
received his Ph.D in electrical &electronics
engineeringwithspecialization in deep learning andimage processing in 2017 from Universiti Teknologi
Petronas Malaysia. His area of interest is machine learning,
deep learning and quantum machine learning for signal
processing and bioemdical imaging..

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
220

Powered by TCPDF (www.tcpdf.org)

All Author’s Full Name

Dr M. K. A. Ahamed Khan

He is currently working at UCSI
University, Malaysia. He received his
Ph.D in Robotics and controls from
USA. His area of research is robotics,
AI and controls. He has publishedmore than100 papers. He is also anIEEE Senior member. He is also thepast chair for IEEE RAS Malaysia chapter.
.
Prof. Steven Niederer
He completed his undergraduate degree
in Engineering Science at the
University of Auckland in 2003 and his
DPhil at the University of Oxford in2008. In 2023, he moved to Imperial
College London as the Chair in
Biomedical Engineering at the National
Heart and Lung Institute. His current work is focused on
reducing barriers to adopting digital twin technology,
developing virtual patient cohorts for in-silico trials,
mapping organ scale function through to cellular andmolecular physiology, and using modelling and simulation
to personalize and guide therapies.
Dr . Mastaneh Mokayef
He is currently working at National
Heart and Lung Institute Imperial
College London, UK. Previously, he
was joined as lecturer at University ofBourgogne Franche-Comté France. He
received his Ph.D in electrical &electronics
engineeringwithspecialization in deep learning andimage processing in 2017 from Universiti Teknologi
Petronas Malaysia. His area of interest is machine learning,
deep learning and quantum machine learning for signal
processing and bioemdical imaging.
Ts Amar Ridzuan Bin Abd Hamid
He is a lecturer of Mechanical andMechatronic
programmes
from
DepartmentofMechanicalEngineering,
UCSI
University,
Malaysia. He has completed his
Master Degree from Universiti Putra
Malaysia, Postgraduate Diploma from
UCSI, and a Bachelor Degree with
Hounours in Mechanical Engineering (Automotive) from
Universiti Teknikal Malaysia Melaka (UTeM), Malaysia.
Dr Cik Suhana Hassan

She currently works at UCSI
University. She received her PhD fromUTP Malaysia. She has been doing
research in Mechanical Engineeringand Materials Engineering. She ispassionate
about
turningenvironmental waste into value-added
products as part of her quest to live a more environmentallyfriendly life.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
221

Hybrid Classical and Quantum Deep Learning Models for Medical Image Classification
Moona Mazher

Centre for Medical Image Computing, Department of Computer Science, University College London, UK
Email: m.mazher@ucl.ac.uk
Abdul Qayyum

National Heart & Lung Institute, Imperial College, London, United Kingdom
M. K. A. Ahamed KhanUCSI University, Faculty of Engineering, Malaysia
Steven Niederer

National Heart & Lung Institute, Imperial College, London, United Kingdom
Alan Turning Institute, London, United Kingdom
Mastaneh Mokayef

UCSI University, Faculty of Engineering, Malaysia
C. S. Hassan

UCSI University, Faculty of Engineering, Malaysia
M. Ridzuan, A.

UCSI University, Faculty of Engineering, Malaysia

Abstract
Quantum machines enhance the capabilities of classical counterparts across various domains, notably in addressing
real-world challenges. The classification of brain MR images for tumor detection is a crucial diagnostic process inthe analysis of brain images. Traditional approaches, such as classical machine learning techniques and conventional
deep learning structures like convolutional neural networks, are frequently employed for image classification.
However, as the network size increases, training these models becomes increasingly arduous. Quantum algorithms
offer advantages by optimizing the performance of classical algorithms through the incorporation of the intrinsic
properties of quantum bits. In this paper, we proposed a hybrid classical and quantum convolutional neural network
for Alzheimer's disease (AD) classification. The proposed model was further validated on the brain tumor
classification task. The fundamental concept involves encoding data into quantum states, facilitating quicker
information extraction, and subsequently utilizing this information to discern the data class. The proposed model
results underscore the reliability and robustness and demonstrated by optimal performance accuracies across various
datasets, the proposed model substantiates its efficacy in detecting and classifying AD disease and brain tumors.
Keywords: Quantum machine learning, Deep learning, Classification, Alzheimer's disease, Brain Tumour, Hybrid
Quantum technique in medical imaging.

tau positron emission tomography (PET) [2]. Although
there have been significant advancements, numerous
emerging diagnostic and treatment approaches are still
primarily confined to research settings. As a result, theprimary means of diagnosis before death continues to rely
on traditional clinical evaluation, neuropsychologicaltesting [10] and magnetic resonance imaging (MRI) [11].
Mild cognitive impairment (MCI), which serves as an
early precursor to dementia, can also represent a subtle
initial manifestation of Alzheimer's disease (AD).
Diagnosing MCI as an early sign of AD demands a highlevel of clinical expertise from skilled specialists.
Alzheimer's disease (AD) is a neurological disorder thatcan be detected through brain imaging, and numerous

1. Introduction
Alzheimer’s disease (AD) is the most common reasonof dementia worldwide [1]. Dementia belongs to assess
abnormal changes in the brain and commonly interrupts
the communication between the brain cells [1]. Disrupted
communication between cells can lead to impairments in
an individual's cognitive functions, including memory
loss, emotional regulation, reasoning and decisionmaking, behavior, and language proficiency [1]. In recent
years, substantial progress has been achieved in theadvancement of cerebrospinal fluid (CSF) biomarkers [2]and cutting-edge imaging techniques like amyloid and©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
222

Moona Mazher, Abdul Qayyum, M. K. A. Ahamed Khan, Steven Niederer, Mastaneh Mokayef, C. S. Hassan, M. Ridzuan, A.

studies have concentrated on using machine learning or
deep learning methods to classify AD based on brain
images [3].

abnormalities using MRI analysis. The architecture
primarily emphasizes the utilization of classicalalgorithms for enhanced detection, thereby expediting thediagnosis and treatment processes. The substructure
integrated into constructing a quantum layer aims totransform classical data points into quantum states,
facilitating faster information extraction and improving
the efficiency of feature detection and pattern recognition.

Recently, various studies proposed deep learningmodels for AD classifications [3], [4]. They proposed
traditional deep learning classification models, in thisstudy we investigate a hybrid model based on classicaldeep learning and quantum machine learning. Recently
Quantum machine learning research has gained successin the medical community.

The Convolutional Neural Network (CNN) is a widely
employed architecture for a range of processing tasks,
such as image recognition, segmentation, andclassification [6], [7], [8], [9], [10], [11], [12]. It is
composed of three basic layers: Convolution, pooling,
and fully connected layers. The convolution and poolinglayers are involved in feature extraction from input
images, while the fully connected layer extracts features
to the output, facilitating classification based on theidentified features. A typical CNN structure comprises
multiple blocks that integrate convolution, pooling, andfully connected layers. We have proposed three blocks ofconv layers. Each block consisted of 2D Conv, 2D
MaxPool, and 2D Normalization layers [13], [14], [15].
We have used one fully connected (FC) layer and one
hybrid quantum layer for AD and brain tumor
classification. The structure of the proposed model isshown in Figure 1.

Few studies proposed quantum machine learning forbrain tumor classification [5]. We have proposed hybrid
deep learning and quantum layers for the classification of
AD and compared the performance with state-of-the-art
methods using a hybrid proposed approach. We proposed
first-time hybrid classical deep learning and quantum
machine learning models for AD classification. We havecompared performance using the brain tumor dataset
using our proposed technique.
2. Methodology
Our proposed approach combines a classical networkwith a quantum network to harness the strengths of bothworlds, creating a model that can effectively identify AD

Fig. 1 The structure of the proposed classical and Quantum deep learning network.

3. Quantum Layers

unique features of a quantum bit, primarily superpositionand quantum entanglement, to process data andinformation. Unlike a classical bit, a qubit's superposition
property enables it to simultaneously exist in multiple
states, including zero and one. In essence, qubits can be
described as state vectors within Hilbert space. The
hybrid layer has been proposed with classical deeplearning models for AD classification and further
compared on the brain tumor classification dataset.

Quantum supremacy manifests when solving intricate
problems more effectively than traditional classicalmethods [5]. Quantum mechanics provides thefundamental framework for the operation of quantum
machines. In a quantum computer, the fundamental unitof information processing is a quantum bit, or qubit, akinto the role played by classical bits in conventional
computers. A standard quantum computer utilizes the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
223

Hybrid Classical and Quantum

Figure 2 shows the complete block architecture of thequantum layers.

Fig. 2 Block of the quantum layer.

4. Dataset

Brain dataset comprises 7,023 human brain MRI scan
images in DICOM format, categorized into four distinct
classes, namely glioma, meningioma, no tumor, andpituitary [5].

We have used an Alzheimer's Disease Neuroimaging
Initiative (ADNI) dataset, consisting of 1,821 participants
based on magnetic resonance imaging (MRI) scans. After
rigorous inclusion criteria were applied, the dataset
comprised a total of 8,916 participants. Subjects werelabeled according to the clinical diagnoses provided by
the ADNI study cohort.
For any subjects with
documented dementia and a primary diagnosis of
Alzheimer’s disease dementia, an AD label was assigned
regardless of the presence of additional dementing
comorbidities. We compared the performance of our
proposed method using a brain tumor dataset. All
algorithms undergo training and performance validation
using the Kaggle Brain Tumor MRI dataset. The Kaggle

5. Results and Discussion
In this study, a series of tests have been performed toassess the benefits of employing quantum circuits in AD
disease and brain cancer classification. These tests
specifically compared the performance of a traditional
CNN with quantum circuits used for feature extraction,with a focus on highlighting the advantages of quantum
circuits. The proposed approach with Quantum layers
produced comparable performance using both datasets asshown in Table 1.

Table 1 Classification performance of brain tumor using classical and quantum deep learning models
Methods
A. Pashaei et al. [16]Classical CNN
Classical and Quantum CNN

Accuracy
93.68
94.33
95.33

precision-95.07
96.34

recall-94.3395.88

F1-score-94.98
94.23

Table 2 The Alzheimer classification using classical and quantum deep learning models.
Methods
Di Wang et al. [17]Classical CNN
Classical and Quantum CNN

Accuracy
87.25
88.33
89.56

precision-88.9090.87

recall-89.9091.63

F1-score-90.3991.22©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
224

Moona Mazher, Abdul Qayyum, M. K. A. Ahamed Khan, Steven Niederer, Mastaneh Mokayef, C. S. Hassan, M. Ridzuan, A.

Table 2 shows the performance of the proposed modelusing classical and hybrid deep learning models. The
quantum-based hybrid deep learning model produced
optimal performance on brain tumor and AD diseaseclassification tasks.

5.

6.

Furthermore, the results found have the capacity of thehybrid quantum model to enhance the efficiency of
medical image classification and diagnosis. However,additional research is necessary to evaluate its
performance in classifying different types of data. This
research offers valuable insights into the benefits ofintegrating quantum circuitry into CNN models for the
analysis of medical images, opening possibilities forquantum-enhanced machine learning within this field.

7.

8.

6. Conclusion
9.

The proposed study introduces an efficient Hybrid
Quantum deep learning model for the classification anddiagnosis of AD disease and Kaggle brain tumors
medical image datasets. The proposed model attains acomparable performance accuracy as compared to theconventional CNN's performance. The validation
accuracy results provide additional evidence of thissuperior performance. Future research efforts will bedirected towards assessing the resilience of hybrid
quantum-based models against adversarial attacks. The
research on quantum with classical deep learning models
will continue to enrich the body of knowledge in quantum
computing and its potential applications in medical
imaging research, ultimately driving progress in precise
diagnosis and enhanced patient care.

10.

11.

12.

References
1. Nichols, Emma, Jaimie D. Steinmetz, Stein Emil Vollset,
Kai Fukutaki, Julian Chalek, Foad Abd-Allah, Amir
Abdoli et al. "Estimation of the global prevalence of
dementia in 2019 and forecasted prevalence in 2050: ananalysis for the Global Burden of Disease Study 2019."
The Lancet Public Health 7, no. 2 (2022): e105-e125.
2. Palmqvist, Sebastian, Shorena Janelidze, Yakeel T. Quiroz,
Henrik Zetterberg, Francisco Lopera, Erik Stomrud, Y. I.
Su et al. "Discriminative accuracy of plasma phosphotau217 for Alzheimer disease vs other neurodegenerative
disorders." Jama 324, no. 8 (2020): 772-781.
3. Mahendran, Nivedhitha, PM Durai Raj Vincent,Kathiravan Srinivasan, and Chuan-Yu Chang. "Improving
the classification of alzheimer’s disease using hybrid gene
selection pipeline and deep learning." Frontiers ingenetics 12 (2021): 784814.
4. Qiu, Shangran, Matthew I. Miller, Prajakta S. Joshi, Joyce
C. Lee, Chonghua Xue, Yunruo Ni, Yuwei Wang et al.
"Multimodal deep learning for Alzheimer’s disease

13.

14.

15.

16.

dementia assessment." Nature communications 13, no. 1(2022): 3404.
Ajlouni, Naim, Adem Özyavaş, Mustafa Takaoğlu, Faruk
Takaoğlu, and Firas Ajlouni. "Medical Image Diagnosis
Based on Adaptive Hybrid Quantum CNN." (2023).
Payette, Kelly, Hongwei Bran Li, Priscille de Dumast,
Roxane Licandro, Hui Ji, Md Mahfuzur Rahman
Siddiquee, Daguang Xu et al. "Fetal brain tissue annotationand segmentation challenge results." Medical Image
Analysis 88 (2023): 102833.
Qayyum, Abdul, Iftikhar Ahmad, Mohsin Iftikhar, andMoona Mazher. "Object Detection and Fuzzy-BasedClassification Using UAV Data." Intelligent Automation& Soft Computing 26, no. 4 (2020).
De Vente, Coen, Koenraad A. Vermeer, Nicolas Jaccard,
He Wang, Hongyi Sun, Firas Khader, Daniel Truhn et al.
"AIROGS: Artificial Intelligence for robust glaucoma
screening challenge." IEEE Transactions on Medical
Imaging (2023).
Chen, Zhihao, Alain Lalande, Michel Salomon, Thomas
Decourselle, Thibaut Pommier, Abdul Qayyum, Jixi Shi,
Gilles Perrot, and Raphaël Couturier. "Automatic deeplearning-based myocardial infarction segmentation fromdelayed enhancement MRI." Computerized Medical
Imaging and Graphics 95 (2022): 102014.
Qayyum, Abdul, Mona Mazhar, Imran Razzak, andMohamed Reda Bouadjenek. "Multilevel depth-wise
context attention network with atrous mechanism forsegmentation of COVID19 affected regions." NeuralComputing and Applications (2021): 1-13.
Ahmad, Iftikhar, Abdul Qayyum, Brij B. Gupta, Madini O.
Alassafi, and Rayed A. AlGhamdi. "Ensemble of 2D
residual neural networks integrated with atrous spatial
pyramid pooling module for myocardium segmentation ofleft ventricle cardiac MRI." Mathematics 10, no. 4 (2022):
627.
Qayyum, Abdul, Aamir Malik, Naufal M Saad, and Moona
Mazher. "Designing deep CNN models based on sparse
coding for aerial imagery: a deep-features reduction
approach." European Journal of Remote Sensing 52, no. 1(2019): 221-239.
Ahmad, R. F., Malik, A. S., Kamel, N., Amin, H., Zafar,
R., Qayyum, A., & Reza, F. (2014, November).
Discriminating the different human brain states with EEG
signals using Fractal dimension: A nonlinear approach. In
2014 IEEE International Conference on Smart
Instrumentation,
Measurement
andApplications(ICSIMA) (pp. 1-5). IEEE.
Qayyum, Abdul, Moona Mazher, Tariq Khan, and ImranRazzak.
"Semi-supervised
3D-InceptionNetforsegmentation and survival prediction of head and neckprimary cancers." Engineering Applications of Artificial
Intelligence 117 (2023): 105590.
Eisenmann, Matthias, Annika Reinke, Vivienn Weru,
Minu Dietlinde Tizabi, Fabian Isensee, Tim J. Adler,
Patrick Godau et al. "Biomedical image analysis
competitions: The state of current participation practice."
arXiv preprint arXiv:2212.08568 (2022).
A. Pashaei, H. Sajedi and N. Jazayeri, "Brain tumor
classification via convolutional neural network and©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
225

Powered by TCPDF (www.tcpdf.org)

Hybrid Classical and Quantum

extreme learning machines", Proc. 8th Int. Conf. Comput.
Knowl. Eng. (ICCKE), pp. 314-319, Oct. 2018.
17. Wang, Di, Nicolas Honnorat, Peter T. Fox, Kerstin Ritter,
Simon B. Eickhoff, Sudha Seshadri, Mohamad Habes, andAlzheimer’s Disease Neuroimaging Initiative. "Deepneural network heatmaps capture Alzheimer’s disease
patterns reported in a large meta-analysis of neuroimaging
studies." Neuroimage 269 (2023): 119929.

Dr . Mastaneh Mokayef
He is currently working at National
Heart and Lung Institute Imperial
College London, UK. Previously, he
was joined as lecturer at University ofBourgogne Franche-Comté France.
He received his Ph.D in electrical &electronics
engineeringwithspecialization in deep learning andimage processing in 2017 from Universiti Teknologi
Petronas Malaysia. His area of interest is machine learning,
deep learning and quantum machine learning for signal
processing and bioemdical imaging.

Authors Introduction
Dr. Moona Mazher
She is a senior postdoc research
fellow at Department of Computer
Science, University College London.
She received her Ph.D. from the
University of Rovira i Virgili, Spain,
212 a specialization in Neuroscience
from
Universiti
Teknologi
PETRONAS, Malaysia in 2017.. Her
areas of interest are machine learning, deep learning,
medical imaging, signal processing, computer vision, andexplainable AI.

Dr Cik Suhana Hassan

She currently works at the Faculty ofEngineering, Technology and Built
Environment, UCSI University. She
received her PhD from UTP
Malaysia. Suhana does research in
MechanicalEngineeringandMaterials Engineering. Dr. Suhana ispassionate about turning environmental waste into valueadded products as part of her quest to live a moreenvironmentally friendly life.

Dr . Abdul Qayyum
He is currently working at National
Heart and Lung Institute Imperial
College London, UK. Previously, he
was joined as lecturer at University ofBourgogne Franche-Comté France. He
received his Ph.D in electrical &electronics
engineeringwithspecialization in deep learning andimage processing in 2017 from Universiti Teknologi
Petronas Malaysia. His area of interest is machine learning,
deep learning and quantum machine learning for signal
processing and bioemdical imaging.

Ts Amar Ridzuan Bin Abd Hamid.He is a lecturer of Mechanical andMechatronic
programmes
from
DepartmentofMechanicalEngineering,
UCSI
University,
Malaysia. He has completed his
Master Degree from Universiti Putra
Malaysia, Postgraduate Diploma from
UCSI, and a Bachelor Degree with
Hounours in Mechanical Engineering (Automotive) from
Universiti Teknikal Malaysia Melaka (UTeM), Malaysia.

Dr M. K. A. Ahamed Khan

He is currently working at UCSI
University, Malaysia. He received his
Ph.D in Robotics and controls from
USA. His area of research is robotics,
AI and controls. He has publishedmore than100 papers. He is also anIEEE Senior member. He is also thepast chair for IEEE RAS Malaysia chapter.

Prof. Steven Niederer
He completed his undergraduate
degree in Engineering Science at the
University of Auckland in 2003 andhis DPhil at the University of Oxford
in 2008. In 2023, he moved toImperial College London as the Chair
in Biomedical Engineering at the
National Heart and Lung Institute. His
current work is focused on reducing barriers to adopting
digital twin technology, developing virtual patient cohorts
for in-silico trials, mapping organ scale function through tocellular and molecular physiology, and using modellingand simulation to personalize and guide therapies.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
226

Global Stabilization of A Class of Nonholonomic Integrators via Discontinuous Control
Lixia Yan

The Seventh Research Division and the Center for Information and Control, School of Automation Science and ElectricalEngineering, Beihang University (BUAA), Beijing 100191, China
Yingmin Jia

The Seventh Research Division and the Center for Information and Control, School of Automation Science andElectrical Engineering, Beihang University (BUAA), Beijing 100191, China
Email: yanlixia@buaa.edu.cn, ymjia@buaa.edu.cn

Abstract
This paper investigates the discontinuous state feedback control for stabilizing a class of nonholonomic integratorswith drift terms. The control design relies on constraining state trajectory in an invariant set. To this end, we apply
constant controls to drive the states moving into the invariant set and then switch to a continuous control law withsuitable gain selections. It is proven in the Lyapunov sense that the proposed control scheme achieves global
exponential stabilization of the states, and the control switch would only occur at most once. Numerical simulations
are carried out to validate the proposed control law.
Keywords: Nonholonomic integrators, Discontinuous feedback, Nonlinear control

system are not considered in the literature mentioned
above.

1. Introduction
Nonholonomic integrators refer to one kind of control
systems that obstruct Brockett’s necessary condition for
the existence of static time-invariant stabilizers [1]. They
have attracted much attention in the control community
because various robotic systems can be converted into
nonholonomic integrators [2].

It is necessary to consider drift terms for utilizing theresults developed for pure nonholonomic integrators onpractical robotic systems. In [2], an adaptive leaderfollowing formation control scheme of multiple wheeledmobile robots was developed by applying techniques
associated with nonholonomic integrators, regulating
formation errors globally convergent to the neighborhoodof the origin. After converting the kinematic model of
unicycles into the form of nonholonomic integrators withdrift terms, a control law capable of rendezvous andtracking of networked unicycles was proposed in [7],which, however, only consider the case that the error
states are initialized inside an invariant set.

Due to non-integrable properties, only discontinuous,time-varying, or hybrid control laws are applicable tostabilize the nonholonomic integrators [3], [4], [5], [6].
Via forcing the state trajectories to move on a sliding
mode surface, a stabilization and a tracking control law
were developed in [3] for nonholonomic integrators,
achieving global asymptotical convergence of the states.
In [4], a novel logic-based hybrid control law with theswitching mechanism that achieved global exponential
stabilization was reported. Using virtual control andvariable structure design, the control scheme in [5]stabilized the states of nonholonomic integrators to zero
exponentially from any initial states. An alternative withthe control Lyapunov function approach for stabilizing
the nonholonomic integrators can be found in [6]. The
leading results in [3], [4], [5], [6] are helpful inunderstanding the structural properties and solution
trajectory of nonholonomic integrators. Yet, the drift
terms that play affect on the dynamic performance of the

Motivated by the discussions above, this paper makes
further endeavors on the control design of nonholonomic
integrators. The concerned nonholonomic integrators canbe viewed as an augmented version of that in [1] byadding a drift term. The control design involves a state
feedback control law and a constant control law, which
achieves asymptotical convergence of the states in an
invariant set and forces the state trajectory to move into
the invariant set, respectively. A simple switch would
occur if the initial states are outside the invariant set.
Lyapunov stability theory is utilized to prove the obtained
theoretical results.
The rest is organized as follows. Section 2 formulates
the control problem and presents the control design.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
227

Lixia Yan, Yingmin Jia

where g −k1 z2 tanh z1 + k1 z1 tanh z2 , k2  2k1  0 . The
effectiveness of the control law (3) is summarized in thelemma below.
LEMMA 1. The application of (3) on (1) achieves thatz1 → 0, z2 → 0, z3 → 0 as t → + and u1 , u2  L , if

Numerical simulations are carried out in Section 3.
Section 4 concludes the work briefly.

2. Main results
2.1. Problem formulation

 z1 = u1 z2 = u2z = z u − z u + f z , z( 1 2)
2 1
1 2
 3
where z1, z2 , z3, u1, u2  , f : 

z2 = −k1 tanh z2 +(1)

positive number 1 such that f ( z1 , z2 )  1  z1 , z2  .
The nonholonomic dynamic (1) obstructs the famous
Brockett’s necessary condition for the existence of fullstate time-invariant static stabilizer. To this end, we plan
to find a discontinuous control law for (1) so that
t →

t →



z12 + z22  z3 .

z2
( f + g + k2 z3 )z12 + z22

(2)

from any initial states.

f  1 z22 + z22
g  k1 z1 + k1 z2  2k1 z22 + z22
which, together with (3), demonstrates thatf +g
k z
u1  k1 ++ 2 3
22
z1 + z2
z12 + z22

Remark 1. Either kinematic or dynamic models of
various nonholonomic systems, including nonholonomic
unicycles and underactuated hovercrafts [7], can beconverted into the form of (1). Thus, the addressednonholonomic integrator (1) has general property though
its form is simple.

k2 z3

 3k1 + 1 +According to x  tanh x , x 

The discontinuous control design includes two steps.
First, we design a state feedback control law so that thestate trajectory is convergent to zero in an invariant set.
Second, we propose a constant control to force the states
to move into the invariant set. The design process above
is depicted in Fig. 1.

, we derive that
(9)

Using the comparison principle then yields [8],V (t )  e−2k1tV ( 0)(10)Therefore, one has z1 ( t ) , z2 ( t )   z1 ( 0 ) , z2 ( 0 ) e− k t

(11)

2
2

1

and

z3
z +z
2
1

u2 = −k1 tanh z2 +z1
( f + g + k2 z3 )z + z22(8)V  −k1 ( z + z ) = −2k1V
2
1

2
2



z3 ( 0 )z ( 0 ) + z22 ( 0 )2
1

Combining (8) and (12), we find out thatz3 ( 0 )u1  U 3k1 + 1 + k2
2
z1 ( 0 ) + z22 ( 0 )Fig. 1. The invariant set and switching mechanism.

(7)

z12 + z22

2.2. Control design

Design the following control law,z
u1 = −k1 tanh z1 − 2 2 2 ( f + g + k2 z3 )z1 + z2(4)

z3 = −k2 z3
Obviously, one has z3 ( t ) = z3 ( 0) e− k2t and obtains that z3
would converge to zero exponentially. Choose a positive
function,V = 0.5 ( z12 + z22 )(5)
The time derivative of V along with the solution
trajectory of (4) can be calculated as,
V = −k1 z1 tanh z1 − k1 z2 tanh z2  0(6)which implies that the states z1 and z2 converge to zero
asymptotically. Next, we prove the boundedness of thecontrol inputs and invariant properties of the set C . Dueto the forms of f and g , it is direct to obtain, and there is alim z1 = 0,lim z2 = 0,lim z3 = 0

3Proof. Substituting (3) into (1) results in,zz1 = −k1 tanh z1 − 2 2 2 ( f + g + k2 z3 )z1 + z2

The concerned nonholonomic integrator with a drift
term in this work holds the form below,

t →

Z 

k2  2k1  0 and Z ( 0 )  C

(12)

(13)

and u1 is bounded. Following the same routine above, we
can also prove the boundedness u2  U . Additionally,(3)

the inequality (11) and the fact z3 ( t ) = z3 ( 0) e− k2t imply
that z12 + z22  z3 , t  0 as k2  2k1 and Z ( 0 )  C .

2
1©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
228

Global Stabilization of A…

LEMMA 2. Given the constant control inputs u1c , u2c  in

Therefore, the set C is invariant. This completes theproof.
□The Lemma 1 shows that the state trajectory would
converge to zero asymptotically in the invariant set C . In
what follows, we would like to use constant controls todrive the state trajectory moving into the set C if thestates are not initialized therein.
Without losing generality, the initial states are assumed
to be outside the invariant set C , i.e., Z ( 0)  3 − C .

(14) satisfying u12c + u22c  c2 and Z ( 0) Proof. The proof is direct via following the derivations(14)-(21), and hence omitted.
□
According to Lemmas 1-2, the discontinuous control law
can be constructed asz2−k1 tanh z1 − z 2 + z 2 ( f + g + k2 z3 ) ,if Z  C
u1 = 
12
u ,if Z  3 − C
 1c
z1

−k1 tanh z2 + z 2 + z 2 ( f + g + k2 z3 ) ,if Z  C
u2 = 
12
u ,if Z  3 − C 2c

z2 ( t ) = z2 ( 0) + u2c t
(15)

t

0

 f ( z ( ) , z ( ))d satisfies, f ( z ( ) , z ( ))d    ( z ( 0) + z (0) )d
t

1

0

2

t

t

1

0

2

1 0

1

2

+1  ( u1c + u2c ) d
t

(16)

0

= c1t + c2t 2()

where c1 = 1 z1 ( 0 ) + z2 ( 0 ) , c2 = 0.51 ( u1c + u2c ) .
Let c0 = z3 ( 0) , c3 = 1 ( z1 ( 0) + z2 ( 0) ) +  z2 ( 0) u1c − z1 ( 0) u2c  , theRemark 2. Note that the time instant t1 is not necessarily
known for control switch in (22). We introduce theestimation on t1 to prove that the state trajectory would
move into the invariant set C in finite time.

estimation for z3 ( t ) can be given by,z3 (t )  c0 + c3t + c2t 2(17)

It hence follows that,z12 ( t ) + z22 ( t ) − z3 ( t )  ( z1 ( 0 ) + u1ct ) + ( z2 ( 0 ) + u2ct )2−c0 − c3t − c2t 2(22)

THEOREM 1. Given the discontinuous control law (22),the system (1) is globally asymptotically stable if
k2  2k1  0 and u12c + u22c  c2 .
Proof. According to Lemma 1, the states would converge
to zero asymptotically in the invariant set C by thecontrol law (22). Moreover, the states would be driven
moving into the set C in finite time if they are initialized
outside the invariant set. Thus, the discontinuous scheme(22) would always ensure the asymptotic convergence tozero of the states. The claims in the theorem areestablished immediately.
□+  f ( z1 ( ) , z2 ( ) )d
The term− C , there

is a finite time instant t1 given by (21) so that Z ( t1 )  C .

Consider the constant control inputs,
u1  u1c , u2  u2c(14)
Using (14) and integrating (1) with respect to time thenlead to,z1 ( t ) = z1 ( 0 ) + u1c t

z3 ( t ) = z3 ( 0) +  z2 ( 0 ) u1c − z1 ( 0 ) u2c  t

3

2

Remark 3. Due to the invariant property of C , the switchof the control law (22) would only occur once at most.

(18)= b2t 2 + b1t + b0

with

b2

u12c + u22c − c2 ,b1

2 z1 ( 0 ) u1c + 2 z2 ( 0 ) u2c − c3b0z ( 0 ) + z ( 0 ) − c0  02
12.3. Application to the tracking control of aunicycle robot(19)

This subsection illustrates the application of (22) on
the trajectory tracking of a unicycle robot. The kinematic
model of a unicycle robot can be given by
x = v cos , y = v sin  , = 
(23)

2
2

u1c , u2c are chosen satisfyingu + u  c2 , then we conclude from b2  0 and b0  0that there exists a finite time instant t1  0 so that,
Observing (19), if
2
1c

2
2c

z12 ( t1 ) + z22 ( t1 ) − z3 ( t1 ) = 0

z12 ( t ) + z22 ( t ) − z3 ( t )  0, t  t1

where  x, y  denotes the Cartesian position,  is theTorientation angle, u represents linear velocity and (20)stands for the angular velocity. Let  xr , yr ,r  denote theTreference trajectory generated by
xr = vr cosr , yr = vr sin r ,r = r
(24)where the reference velocities vr and r are bounded.
Define the following tracking errors,
z1 = ( x − xr ) cos + ( y − yr ) sin Such a t1 can be estimated as,−b1 + b − 4b2b0(21)2b2
The derivations above are gathered together in thefollowing lemma.
0  t1  t2

21

z2 =  −  rz3 = −2 ( x − xr ) sin  + 2 ( y − yr ) cos  + z1 z2©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
229

(25)

Lixia Yan, Yingmin Jia

reference signal by vr = 0.2,r = 0.1 with initial condition
xr ( 0) = 0, yr ( 0) = −2,r ( 0) = 0 . The initial pose of the

The time derivative of (25) can be calculated asz1 = v + 0.5 ( z3 − z1 z2 ) − vr cos z2

z2 =  − runicycle is chosen as x ( 0) = −2, y ( 0) = −2.5, ( 0) =  .
Meanwhile, the control coefficients k1, k2 , u1c and u2c are
set the same as previous case. We depict the position
trajectory and errors in Fig. 4 and Fig. 5, respectively.
As can be seen, the Fig. 4-5 illustrate the success ofapplying discontinuous control law (22) on the trajectory
tracking control of a typical unicycle robot. The posetracking errors are convergent to zero asymptotically.

(26)

z3 = z2 z1 − z1 z2 + 2 ( vr sin z2 − r z1 )Obviously, the error dynamics (26) features the samestructure as (1) if we define
u1 = v + 0.5 ( z3 − z1 z2 ) − vr cos z2
u2 =  − r

f = 2 ( vr sin z2 − r z1 )(27)

1 = 2 vr2 + r2
Therefore, applying the control law (22) on (26) would
steer the error (25) to zero globally asymptotically. As thestate transformation (25) is globally invertible, theoriginal tracking errors x − xr , y − yr and  −r would
converge to zero asymptotically from any initial states.
3. Numerical Simulations
This section validates the proposed control law (22) bynumerical simulations. To this end, we initialize theTnonholonomic integrator (1) by Z ( 0) = −1,1, −4 and set

Fig. 4. The position trajectory(*:starting point).

the control coefficients as k1 = 0.25, k2 = 0.55 . The drift
term is f = 0.2 ( z1 + z2 ) . The constant control inputs are
u1c = 0.2, u2c = 0.2 . We depict the results in Figs 2-3.

Fig. 5. The tracking errors.
4. Conclusion
Fig. 2. The state trajectories.
As can be seen from Fig. 2. the initial states are outside
the invariant set, and the control inputs are constant inthis phase. The control is automatically switched into
state feedback control law as soon as the switchingcondition z12 (t ) + z22 (t )  z3 (t ) is satisfied, which can be

This brief introduces a discontinuous global stabilizer
for a class of nonholonomic integrators with drift terms.
The proposed control scheme includes a state feedback
controller and a constant control law, fusing with a simple
criterion for control switching. It is proven in the
Lyapunov sense that the system states converge to zeroglobally asymptotically, with the control switchingoccurring at most once. In the future, the authors will
generalize the current control scheme to solve global
stabilization problems of other nonholonomic systems.

drawn from the turnings in Fig. 2. Then, the states areconstrained in the invariant set and converge to zero
asymptotically. In addition, the control inputs are always
bounded, which can be concluded from Fig. 3.

Acknowledgements
This work was supported in part by the NSFC
(62133001, 62227810) and the National Basic Research
Program of China (973 Program: 2012CB821200,
2012CB821201).

Fig. 3. The control inputs.
To validate the control law on solving the trajectory
tracking control problem of a unicycle robot, we set the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
230

Powered by TCPDF (www.tcpdf.org)

Global Stabilization of A…

References
1.

2.

3.

4.

5.

6.

7.

8.

R.W. Brockett, Asymptotic stability and feedback
stabilization, Differential Geometric Control Theory.
Birkhauser, 1983.
L. Yan and B. Ma, Adaptive practical leader‐following
formation control of multiple nonholonomic wheeledmobile robots, International Journal of Robust Nonlinear
Control, Vol.30(17), 2020, pp.7216-7237.
A. Bloch and S. Drakunov, Stabilization and tracking in
the nonholonomic integrator via sliding modes, Systems &Control Letters, Vol. 29(2), 1996, pp.91-99.
J.P. Hespanha and A.M. Stephen, Stabilization ofnonholonomic integrators via logic-based switching,
Automatica, Vol.35(3), 1999, pp.385-393.
W. Xu and W. Huo, Variable structure exponential
stabilization of chained systems based on the extended
nonholonomic integrator, Systems & Control Letters,
Vol.41(4), 2000, pp.225-235.
P. Braun, L. Grüne and C.M. Kellett, Feedback designusing nonsmooth control Lyapunov functions: A
numerical case study for the nonholonomic integrator.
2017 IEEE 56th Annual Conference on Decision andControl (CDC); 2017: IEEE.
L. Yan, B. Ma, and Y. Jia. Universal Control for Both
Rendezvous and Tracking of Multiple Nonholonomic
Unicycles. IEEE Transactions on Control of Network
Systems, 2023(Early Access).
H.K., Khalil, Nonlinear Systems (3rd Edition), Patience
Hall, 2002.

Authors Introduction
Dr. Lixia Yan
He received B.S. degree from Beijing
Jiaotong University, Beijing, China in
2013, and the M.S. and Ph.D degrees
from Beihang University, Beijing,
China in 2016 and 2021, respectively.
He is currently a postdoctor with theSeventh Research Division and theCenter for Information and Control at
Beihang University. His research interests include
nonlinear control theory, multiagent systems andembedded system applications. He is a member of IEEE.
Prof. Yingmin Jia
He received B.S. degree from Shandong
University, Jinan, China, in 1982, andthe M.S. and Ph.D. degrees from
Beihang University, Beijing, China, in1990 and 1993, respectively.
He is currently a professor with theSeventh Research Division and the
director of the Center for Information
and Control at Beihang University. His research interests
include robust control, robotic systems, spacecraft
coordination and on-orbit servicing. He is a senior memberof IEEE.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
231

Frequency Dependence Performance Limit of Vibration Absorbers
Jiqiang Wang

Zhejiang Provincial Engineering Research Centre for Special Aircrafts, Ningbo Institute of Materials Technology &Engineering, Chinese Academy of Sciences, Ningbo, China
Email: wangjiqiang@nimte.ac.cn
Jingjing Fang

Zhejiang Development & Planning Institute, Zhangzhou, China
Email: fjjzdpri@foxmail.com
Weicun Zhang

School of Automation and Electrical Engineering, University of Science and Technology Beijing, Beijing, China
Email: weicunzhang@ustb.edu.cn
Xinmin Chen

Zhejiang Provincial Engineering Research Centre for Special Aircrafts, Ningbo Institute of Materials Technology &Engineering, Chinese Academy of Sciences, Ningbo, China
Email: chenxinmin@nimte.ac.cn

Abstract

Optimal design of vibration absorbers has been extensively investigated. Most of the design methods areapproached by optimizing certain performance indices, resulting in a set of optimal parameters that are
independent of exogenous forcing frequencies. In practical designs, however, it is often desirable to know
the performance limits over a frequency band of interest. This problem is tackled in the present paper
where both lower and upper bounds are obtained. Numerical examples are given to validate thecorresponding designs.
Keywords: Vibration absorbers, optimal design, performance limit

1. Introduction

In review of the available results in the literature, it isseen that the optimization of performance indices leads to
the results that only optimize the pre-designated indices,
yet performance limits, particularly the frequencydependence performance bounding information cannot
be obtained. This frequency-dependence performance
limit of vibration absorbers is considered in the paper. It
is organized as follows: section 2 formulates the problem
to be considered. Section 3 proceeds to develop the lower
and upper bounds. Numerical examples are also provided
before a conclusion in section 4.

Tuned mass dampers (TMD) are widely used forvibration attenuation either at a frequency or over afrequency range. Many configurations have been
proposed [1], [2], [3] while one typical TMD consists ofa secondary mass-damper-spring system, attached to avibrating primary system. The secondary system is also
called an absorber whose parameters are to be designed.
Besides the classical “equal height” methods, many
approaches have been proposed for optimal design of
TMD parameters. Most of the developments are precededwith formulating optimization problems by optimizing aproperly chosen performance index. For example,
H2/H∞ forms of performance indices can be optimized
where even analytical solution can be found for optimal
parameter selections [4], [5]. A comparison between
different optimization criteria is given in [6], while moreelaborated examples can be found in [7]. As a result of
the optimization, optimal parameters can be obtained
which is usually represented as optimal damping andfrequency ratios as a function of the mass ratio.

2. Problem Formulation
The model can be represented as:

m1x1 + (c1 + c2 ) x1 − c2 x2 + (k1 + k2 ) x1 − k2 x2 = dm2 x2 − c2 x1 + c2 x2 − k2 x1 + k2 x2 = 0(1)
F = c1x1 + k1x1

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
232

Jiqiang Wang, Xiaoyu Yin, Jingjing Fang, Weicun Zhang, Xinmin Chen

X1 ( j )D( j )

where the dependence on time has been omitted for easy
reference. The frequency response property for thetransmission force can then be written down:()(k1 + jc1 ) − m2 2 + k2 + jc2F ( j )=D( j ) − m1 2 + k1 + k2 + j (c1 + c2 ) − m2 2 + k2 + jc2 − (k2 + jc2 )2(2)where F ( j) and D( j) are the Fourier transforms



(

)

of F (t ) and d (t ) , respectively.
The aim of the TMD design is to design the absorberparameters m2 , c2 , and k 2 in such a way, so thatcertain appropriate performance indices such asenergy/magnitude defined by H2/H∞ norms are
optimized. Although the optimization “routines” exist for
obtaining feasible solutions to the corresponding
optimization problems, it is still of great significance andinterest to seek the boundary or limit of performance. As
the limits will dictate the achievable performance, on the
one hand, they are not to be conquered by any form of
performance index to be optimized; on the other hand,
they will actually provide guidance to the designers if achoice of performance index is suitable by knowing its“distance” to the limits. Henceforth, performance limits
should be treated as a benchmarking that any design
through any optimization method with any performanceindex must compare with. These issues are treated in thefollowing sections.

2

2
2

2
2

2

, 
(5)
The relationship is strictly “greater than” implying thatthe lower bound is absolute and never to be attained.
Theorem 2: The performance of X 1 ( j ) D( j ) is

1 = k1 m1 , the minimum upper bound is 2 = k 2 m2 .
(2) As the lower bound, the upper bound is also inversely
proportional to the natural frequency 2 —toincrease the bound, 2 needs to be decreased!
(3) Yet one of the most important applications for upperbound is the assertion that the performance
X1 ( j ) D( j ) will always be attenuated overthe frequency bands where the upper bound is lessthan unity. This can be developed into a very usefuland powerful design methodology.
For example, assume c1 and m2 are unities, thenthe bound becomes:

X1 ( j )− m2 2 + k2 + jc2=2
D( j ) − m1 + k1 + k2 + j (c1 + c2 ) − m2 2 + k2 + jc2 − (k2 + jc2 )2(4)
Thus the objective of TMD design can be re-stated to

(

1

2

achieved at the natural frequency of the absorber

F ( j) D( j) is equivalent to optimizing
X1 ( j) D( j) . From equation (2), it is known:



1

2
1

frequency.
From the above result, a series of observations
follow:(1) While the minimum lower bound is attained atnatural frequency of the primary system

)

)

2 2

X 1 ( j )K 2 + M 2 4 + C 2 2

, 
(6)
D( j )h( )where: h() is a positive function dependent on

From equation (1), it is known that the following
frequency responses relationship holds:
F ( j) = k1 + jc1 X1 ( j)(3)
For any particular configuration, the primary systemparameters c1 and k1 are known, hence optimizing((k − m  ) + c  +  m (k 2+ c  )bounded from above by:

3. Performance Limit: Lower & Upper Bounds

(

1

)

reduce the transmission magnitude X 1 ( j ) D( j )+1c 2 5(7)(k −  ) + c 
2 2

2

2
2

2

Then a calculation for cubic unities of k 2 , c2 , and with a grid of 101010 shows that a set of solutions
exist for satisfying (7). This is shown in Figure1.

through the optimal selection of the absorber parameters
m2 , c2 , and k 2 . In the following, the performancebounds for attenuation of the magnitude ofX1 ( j) D( j) through the to-be-designed
parameters of m2 , c2 , and k 2 will be sought. This ispreceded by boldly stating the results.
Theorem 1 (Lower Bound): The performance ofX1 ( j ) D( j ) is bounded from below by:©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
233

Powered by TCPDF (www.tcpdf.org)

Frequency Dependence Performance Limit

Authors Introduction
Prof. Jiqiang Wang
He received his PhD degree inControl Systems in 2009 from theUnversity of Sheffield, UK. He iscurrently a Professor at Ningbo
Institute of Materials Technology&
Engineering,
ChineseAcademy of Sciences. He is also
the deputy director of Zhejiang
Provincial Engineering ResearchCentre for Special Aircrafts.
Ms. Jingjing Fang
She graduated with a bachelor’s
degree from Nanjing Universityof Aeronautics & Astronautics in2013 and a master’s degree from
Shanghai
AcademyofSpaceflight Technology in 2016.
She has been working on
instrumentation and testing ofrocket engines for many years
before moving to Zhejiang
Development&
Planning
Institute for her current career. She now focuses upon theeconomic and industrial analysis of cutting-edge
technologies in aerospace and high-end equipment sectors.

Figure 1: feasible choices of parameters k2 , c2 , and within a 101010 cubic unity grid
4. Conclusion
Frequency-dependent performance limits of tuned massdampers have been considered. It has been demonstrated
that the performance is not only bounded from below, butalso bounded from above. The existence of the lower
bound is very useful providing guidance upon best
performance to be expected. Combining with the concept
of attenuation by bounding from above, it has been shownthat this can lead to design methods with such features as
guaranteed performance.

Prof. Weicun Zhang
He received his M.S. degree in
Automatic Control from Beijing
Institute of Technology in 1989,
and the Ph.D. degree in Control
Theory and Applications from
Tsinghua University in 1993.
From March 1997 to May 1998,
he was a visiting research fellow
in Industrial and Operations
Engineering Department, University of Michigan at Ann
Arbor, From September 2006 to August 2007; he was avisiting professor in Department of Electrical andComputer Engineering, Seoul National University, South
Korea.

Acknowledgements
This work was supported by the Yongjiang Talent
Project of Ningbo (No. 2022A-012-G).
References
1.
2.

3.

4.

5.

6.

7.

S.G. Kelly, Fundamentals of Mechanical Vibrations, 2ndedition, McGraw-Hill, 2000.
Y. Ishida, Recent development of the passive vibration
control method, Mechanical Systems & Signal Processing
2012, 29: 2-18.
S. Bakre, R. Jangid, Optimal multiple tuned mass dampers
for base excited damped main system, International
Journal of Structural Stability & Dynamics 2004, 4(4):527-542.
O. Nishihara, T. Asami, Closed-form solutions to the exact
optimizations of dynamic vibration absorbers, Journal of
Vibration & Acoustics 2002, 124: 576-582.
T. Asami, O. Nishihara, A. Baz, Analytical solutions to
H∞ and H2 optimization of dynamic vibration absorbers
attached to damped linear systems, Journal of Vibration &Acoustics 2002, 124: 284-295.
G. Marano, R. Greco, B. Chiaia, A comparison between
different optimization criteria for tuned mass dampers
design, Journal of Sound & Vibration 2010, 329: 48804890.
G. Bekdaş, S. Nigdeli, Mass ratio factor for optimum tuned
mass damper strategies, International Journal of
Mechanical Sciences 2013, 71: 68-84.

Prof. Xinmin Chen
He graduated with a bachelor’s
degreefrom
Schoolof
Astronautics of Beijing Universityof Aeronautics & Astronautics(BUAA) in 1995, and graduatedfrom China Academy of Launch
Vehicle Technology (CALT) witha master degree in 1998, and a PhD
degree in 2009. In CALT hesuccessively served as assistant engineer, engineer, senior
engineer (2005) and chief engineer (2010). He has been
under the supervision of Academician YU Menglun ofChinese Academy of Sciences for many years and has
expertise in flight mechanics and aircraft designs.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
234

Adaptive Concurrent Learning Algorithm Based on Pontryagin’s Maximum Principle for
Nonlinear System Optimal Tracking Control with State Inequality Constraints
Yuqi Zhang

Beijing University of Posts and Telecommunications, School of Artificial Intelligence, Beijing 100876, China
Bin Zhang

Beijing University of Posts and Telecommunications, School of Artificial Intelligence, Beijing 100876, China
Email: zhangbinzdh@bupt.edu.cn

Abstract
In this paper, an adaptive iterative learning algorithm is proposed to solve the optimal tracking control problem(OTCP). Unlike the existing method, we select the finite-time horizon cost function to measure tracking performance.
Our method doesn’t require an accurate system dynamic and the concurrent learning (CL) technique is utilized tolearn the system identification model. Based on identification model, we present our concurrent iterative learningalgorithm under the Pontryagin’s framework to learn the solution for OTCP. The proposed algorithm overcomes thelimitation of Adaptive Dynamic Programming (ADP) methods when dealing with time-varying systems or situations
involving state inequality constraints. The algorithm's effectiveness is demonstrated through a numerical simulation.
Keywords: Optimal tracking control, Concurrent learning, State inequality constraints, Adaptive iterative algorithmand value function no longer holds in affine nonlinear
systems. Therefore, within the ADP framework, theproblem of state constraints for OTCP remains
unresolved.

1. Introduction
Optimal tracking control [1], [2] is a comprehensive
problem to find an optimal control input to minimize thetracking cost function to ensure that the system follows
the prescribed trajectory. Due to the complexity of thenonlinear systems, it is difficult to obtain analyticalsolutions for OTCP. This paper aims to design an
iterative algorithm to realize the tracking control without
the accurate system information.

This paper aims to develop a new adaptive iterativealgorithm to solve the OTCP with finite-horizon costfunction and state constraints. CL technique [9] is used tolearn the system identification model. Based on
Pontryagin’s framework and the identification model, we
design a new adaptive iterative method to learn the
optimal control input which minimize the tacking costfunction in prescribed time interval without the exact
system dynamic. Moreover, the system trajectory not
only tracks the predetermined trajectory, but also satisfies
the state constraints.

Many researchers apply Reinforcement learning (RL)
technique [3], [4], [5], [6] to solve the above problems.
The advantage of reinforcement learning is that it does
not require a precise prior knowledge of system dynamics.
Most of the methods use the ADP technique to solve thetracking Hamilton–Jacobi–Bellman (HJB) equation.
Approximation methods like neural networks (NNs) are
widely used in ADP literature for value function
approximation. Although ADP is effective, it still has
inherent technical obstacles. The HJB equation reduced
to the form of an ODE under an infinite-horizon costfunction and an affine nonlinear system. It is easier tolearn solutions compared to PDE.

2. Problem statement
We consider the following nonlinear dynamic system:𝑥̇ = 𝑓(𝑥, 𝑢, 𝑡), 𝑥(𝑡0 ) = 𝑥0(1)

where 𝑥 ∈ ℝ𝑛 is the system state with initial state 𝑥0 , and𝑢 ∈ ℝ𝑚 is the control input. The unknown map 𝑓: ℝ𝑛 ×
ℝ𝑚 × ℝ → ℝ𝑛 is Lipschitz continuous. It is assumed thatthe state must be limited to satisfy the following
inequality constraints:

Another important problem for the OTCP is the is theexistence of constraints. For control input constraints, anonquadratic performance function [7], [8] is used in the
optimal regulation problem. As for the state constraints
problem, the explicit expression between optimal control𝑠(𝑥, 𝑡) ≤ 0©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
235

(2)

Yuqi Zhang, Bin Zhang

where Φ(𝑒, 𝑢): ℝ𝑛 × ℝ𝑚 → ℝ𝑁 is the vector of basis
functions. Θ: ℝ → ℝ𝑁×𝑛 is the unknown time-varying
weight matrix. 𝜀 is the vector of approximation errors.
According to the Weierstrass approximation theorem, 𝜀can be reduced to zero with arbitrarily high precision as
𝑁 → ∞. We define the estimate of weight matrix and̂ and 𝑒̂ , respectively.
tracking error as Θ

where 𝑠 = (𝑠1 , 𝑠2 , … , 𝑠𝑧 )𝑇 ∈ ℝ𝑛 × ℝ → ℝ𝑧 is a zdimensional column vector function. Let 𝑥𝑑 (𝑡) ∈ ℝ𝑛 is aprescribed trajectory with initial state 𝑥𝑑0 , the dynamic of
tracking error and the constraints can be expressed as:{𝑒̇ = 𝐹(𝑒, 𝑢, 𝑡)𝑆(𝑒, 𝑡) ≤ 0

(3)
(4)

For error dynamic system (3), we define the historical𝑊
data stack as {𝔼𝑗 (𝑡), 𝕌𝑗 (𝑡), 𝔼̇𝑗 (𝑡), 𝑡 ∈ [𝑡0 , 𝑡𝑓 ]}𝑗=1 ,where 𝑒 = 𝑥 − 𝑥𝑑 is the tracking error, 𝐹(𝑒, 𝑢, 𝑡) =𝑓(𝑒 + 𝑥𝑑 , 𝑢, 𝑡) − 𝑥̇ 𝑑 and 𝑆(𝑒, 𝑡) = 𝑠(𝑒 + 𝑥𝑑 , 𝑡) . We
define the following finite-horizon cost function for
tracking performance:

which stores the tracking error, the control input, and thederivative of the tracking error. We have the following
assumption for history trajectory stack:𝑡𝑓

𝐽(𝑒, 𝑢) = ∫ 𝐿(𝑒, 𝑢)𝑑𝑡

𝑡
𝑡
𝑡
𝑡
𝑇
Assumption 1. Let Π 𝑡 = ∑𝑊
𝑗=1 Φ (𝔼𝑗 , 𝕌𝑗 ) Φ(𝔼𝑗 , 𝕌𝑗 ). It
is assumed that there exist positive constants 𝜇 > 𝜇 > 0,for ∀𝑡[𝑡0 , 𝑡𝑓 ], we have 𝜇𝐼 < Π 𝑡 < 𝜇𝐼.

(5)

𝑡0

where 𝐿(𝑒, 𝑢) = 𝑒 𝑇 𝑄𝑒 + 𝑢𝑇 𝑅𝑢 , 𝑄 ∈ ℝ𝑛×𝑛 and 𝑅 ∈ℝ𝑚×𝑚 are positive definite symmetric matrices. The
optimal tracking control problem in this paper aims tofind an optimal control input 𝑢∗ (𝑡), 𝑡 ∈ [𝑡0 , 𝑡𝑓 ] tominimize the cost function (5) subject to system (3) andinequality constraints (4).

We design the following adaptive updating law of the
̂𝑖 :𝑖-th estimates for tracking error 𝑒̂ 𝑖 and weight matrix Θ
̂ 𝑖 (𝑡))𝑇 Φ(𝑒, 𝑢) + 𝜔𝑒̃ 𝑖 , 𝑒̂ 𝑖 (𝑡0 ) = 𝑥0 − 𝑥𝑑0 (9)𝑒̂̇ 𝑖 = (Θ𝑖
̂ (𝑡) = Θ
̂ 𝑖−1 (𝑡) + (𝐼 − 𝜇Π 𝑡 )2 Φ(𝑒, 𝑢)(𝑒̃ 𝑖 )𝑇Θ𝑊𝑇

It is proved [10] that the above state constraints optimal
control problem can be reduced to solve a sequence ofunconstrained problem by minimizing the following
Kelley-Bryson penalty cost function as 𝑘 → ∞:𝐽(𝑒, 𝑢, 𝑟𝑘)𝑗=1

̂ 𝑖−1 (𝑡) + (𝐼 − 𝜇Π 𝑡 )2 Φ(𝑒, 𝑢)(𝑒̃ 𝑖 )𝑇 + 𝜇Π 𝑡 Θ
̃ 𝑖−1 (𝑡)
=Θ
(10)
{

𝑧

𝑡𝑓

1

where 0 < 𝜇 < is a positive constant. 𝑒̃ 𝑖 = 𝑒 − 𝑒̂ 𝑖 and𝜇̃𝑖 (𝑡) = Θ(t) − Θ̂ 𝑖 (𝑡) are 𝑖 -th estimation errors of theΘ
tracking error and the weight matrix, respectively. Under
Assumption 1, the adaptive updating law (10) guarantees
̂ 𝑖 converges to its true values.
that the weight matrix Θ= ∫ 𝐿(𝑒, 𝑢) + 𝑟 ∑ ℎ(𝑆𝑗 )𝑆𝑗2 (𝑒, 𝑡) 𝑑𝑡 (6)
𝑘

𝑡0

𝑗=1

1, 𝜎 > 0
where lim 𝑟 𝑘 = ∞ and ℎ(𝜎) = {0, 𝜎 ≤ 0.
𝑘→∞
The necessary condition for each optimal control
problem associated with 𝐽(𝑒, 𝑢, 𝑟 𝑘 ) is concluded by:

3.2. Iterative learning algorithm
In this subsection, we will provide our iterative
learning algorithm to solve the optimal tracking problem.
To start with, we define:𝑧𝜆̇ = −𝐻𝑒 − 2𝑟 𝑘 ∑ ℎ(𝑆𝑗 )𝑆𝑗 𝑆𝑗𝑒
𝑗=1

{

(7)

𝑧

𝑢 = argmin 𝐻(𝑒, 𝑢, 𝜆, 𝑡)𝑢{𝑇𝑃(𝑒, 𝑢, 𝑟, 𝑡) = 𝐿(𝑒, 𝑢) + 𝑟 𝑘 ∑ ℎ(𝑆𝑗 )𝑆𝑗2 (𝑒, 𝑡)
𝑗=1

where 𝐻(𝑒, 𝑢, 𝜆, 𝑡) = 𝐿(𝑒, 𝑢) + 𝜆 𝐹(𝑒, 𝑢, 𝑡).

𝑄(𝑒, 𝑢, 𝑟, 𝑡, 𝜆, Θ) = 𝑃(𝑒, 𝑢, 𝑟, 𝑡) + 𝜆𝑇 Θ𝑇 (t)Φ(𝑒, 𝑢)

3. Concurrent learning iterative algorithm

Algorithm 1. Iterative learning algorithm for
optimal tracking problem

3.1. System identification(11)

Step 1: Initialize parameters 𝑟 0 > 0, 𝑎0 > 0, 𝑏 0 > 0,Θ𝑡 ∈ [𝑡0 , 𝑡𝑓 ] , convergence error 𝜖1 > 0 , 𝜖2 > 0 ,𝜖3 > 0. Select the initial control input u0 (𝑡), 𝑡 ∈ [𝑡0 , 𝑡𝑓 ].
Update the initial estimation of tracking error by solving:First, we suppose that the unknown error dynamic (3)can be represented through a finite set of basis functions:𝐹(𝑒, 𝑢) = Θ𝑇 (t)Φ(𝑒, 𝑢) + 𝜀𝑇̂ 𝑖−1 (𝑡)) (Φ(𝔼𝑗𝑡 , 𝕌𝑗𝑡 )))+𝜇 ∑ Φ(𝔼𝑗𝑡 , 𝕌𝑗𝑡 ) (𝔼̇𝑗𝑡 − (Θ

̂ 0 (𝑡),(8)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
236

Adaptive Concurrent Learning Algorithm̂ 0 (𝑡))𝑇 Φ(𝑒, 𝑢) + 𝜔𝑒̃ 0 , 𝑒̂ 0 (𝑡0 ) = 𝑥0 − 𝑥𝑑0 (12)𝑒̂̇ 0 = (Θwhere 𝜃𝑖 and 𝜃𝑚 represent the angles of motors and theload, 𝐽𝑖 and 𝐽𝑚 are the moment of inertia for the motor
and the load, 𝑏𝑖 and 𝑏𝑚 are the resistance coefficients ofthe motor and load, respectively. 𝑢𝑖 is the input torque ofthe motor, 𝜏𝑖 is the torque transmitted when the driving
motor and load come into contact, which can be
expressed as the following dead zone function:

Calculate the initial tracking cost function:𝑡𝑓

𝐽(𝑒 0 , 𝑢0 , 𝑟 0 ) = ∫ 𝑃(𝑒 0 , 𝑢0 , 𝑟 0 , 𝑡)𝑑𝑡

(13)

𝑡0

Let 𝑖 = 0 and 𝑘 = 0.
Step 2: Calculate 𝑢

𝑖+1

and 𝜆 by following equation:̂ 𝑖 ), 𝜆𝑖 (𝑡𝑓 ) = 0
𝜆̇𝑖 = −𝑄𝑒 (𝑒 𝑖 , 𝑢𝑖+1 , 𝑟 𝑘 , 𝑡, 𝜆𝑖 , Θ̂𝑖 )𝑢𝑖+1 = 𝑎𝑟𝑔𝑚𝑖𝑛 𝑄(𝑒 𝑖 , 𝑢, 𝑟 𝑘 , 𝑡, 𝜆𝑖 , Θwhere 𝑘 is the torque torsion coefficient, 𝑧𝑖 = 𝜃𝑖 − γ𝜃𝑚is the angular error between the motor and the load, γ istransmission ratio, 2𝛼 is the backlash width. Due to thenon differentiability of the dead zone function, it isgenerally approximated by the following smooth
continuous differentiable functions:

(14)

𝑢

{

𝑘(𝑧𝑖 + 𝛼), 𝑧𝑖 ≤ 𝛼|𝑧𝑖 | < 𝛼𝜏𝑖 = {0,𝑘(𝑧𝑖 − 𝛼), 𝑧𝑖 ≥ 𝛼𝑖12+𝑎 ‖𝑢 − 𝑢𝑖 ‖1 + 𝑏 𝑖 ‖𝑢 − 𝑢𝑖 ‖22𝑖̂ 𝑖 by following equation:
Step 3: Calculate ê𝑖 and Θ
̂ 𝑖 (𝑡))𝑇 Φ(𝑒, 𝑢) + 𝜔𝑒̃ 𝑖 , 𝑒̂ 𝑖 (𝑡0 ) = 𝑥0 − 𝑥𝑑0𝑒̂̇ 𝑖 = (Θ
̂ 𝑖 (𝑡) = Θ̂ 𝑖−1 (𝑡) + (𝐼 − 𝜇Π 𝑡 )2 Φ(𝑒, 𝑢)(𝑒̃ 𝑖 )𝑇 (15){ Θ̃ 𝑖−1 (𝑡)+𝜇Π 𝑡 Θ2
𝜏𝑖 = 𝑘 (𝑧𝑖 − 𝛼 (− 1)) = 𝑘(𝑧𝑖 − 𝛼𝑔𝑖 )1 + 𝑒 −𝑟𝑧𝑖
Assuming two motors have the same parameters, thetransmission ratio γ is selected as 1, the parameter r in theapproximation function is selected as 10, and theparameters of the system are selected as Table 1.

Step 4: Calculate the tracking cost function:𝑡𝑓

𝐽(𝑒 𝑖+1 , 𝑢𝑖+1 , 𝑟 𝑘 ) = ∫ 𝑃(𝑒 𝑖+1 , 𝑢𝑖+1 , 𝑟 𝑘 , 𝑡)𝑑𝑡

(16)

𝑡0
𝑖+1

𝑖+1

𝑘)

𝑖

𝑖

𝑘)

𝑖

Parameter
𝐽𝑚
𝑏𝑚
𝐽𝑖
𝑏𝑖
𝑘
𝛼

𝑖

If 𝐽(𝑒 , 𝑢 , 𝑟 > 𝐽(𝑒 , 𝑢 , 𝑟 , let 𝑎 ⟵ 𝑎 + 𝑑𝑎and 𝑏 𝑖 ⟵ 𝑏 𝑖 + 𝑑𝑏 , where 𝑑𝑎 and 𝑑𝑏 are positive stepsize, then go to Step 2. Else go to Step 5.
Step 5: If ‖𝑢𝑖+1 − 𝑢𝑖 ‖ ≤ 𝜖1 , go to Step 6. Else let 𝑖 ⟵𝑖 + 1, 𝑎𝑖 ⟵ 𝑎0 , 𝑏 𝑖 ⟵ 𝑏 0 , go to Step 2.
̂ 𝑖+1 − Θ
̂ 𝑖 ‖ ≤ 𝜖2 , go to Step 7. Else let 𝑖 ⟵Step 6: If ‖Θ
𝑖 + 1, 𝑎𝑖 ⟵ 𝑎0 , 𝑏 𝑖 ⟵ 𝑏 0 , go to Step 2.

𝑘𝑔 ∙ 𝑚2
𝑁 ∙ 𝑚 ∙ 𝑠/𝑟𝑎𝑑
𝑘𝑔 ∙ 𝑚2
𝑁 ∙ 𝑚 ∙ 𝑠/𝑟𝑎𝑑𝑁 ∙ 𝑚/𝑟𝑎𝑑
𝑟𝑎𝑑

0.185
1.2
0.028
1.3
56
0.2

𝑇
Let 𝑥 = (𝜃𝑚 , 𝜃̇𝑚 , 𝜃1 , 𝜃̇1 , 𝜃2 , 𝜃̇2 ) . The target is selected
as 𝑥𝑑 (𝑡) = [1 0 1 0 1 0]𝑇 , we choose cost function:

Step 7: If 𝑆(𝑒 𝑖+1 , 𝑡) ≤ 𝜖3, go to Step 8. Else let 𝑖 ⟵𝑖 + 1 , 𝑎𝑖 ⟵ 𝑎0 , 𝑏 𝑖 ⟵ 𝑏 0 , 𝑟 𝑘 ⟵ 𝑟 𝑘 + 𝑑𝑟 where 𝑑𝑟 isthe positive step size, then go to Step 2.

20

6

2

𝑖=1

𝑗=1

1
𝐽(𝑒, 𝑢) = ∫(∑ 𝑒𝑖2 + ∑ 𝑢𝑗2 )𝑑𝑡
0 2

4. Simulation

the state constraints are assumed to be |𝑥2 | ≤ 0.4𝑟𝑎𝑑/𝑠,|𝑥4 | ≤ 0.4𝑟𝑎𝑑/𝑠 , |𝑥6 | ≤ 0.4𝑟𝑎𝑑/𝑠 , which can be
rewritten as: 𝑠1 = 𝑒2 − 0.4, 𝑠2 = −𝑒2 − 0.4, 𝑠3 = 𝑒4 −
0.4 , 𝑠4 = −𝑒4 − 0.4 , 𝑠5 = 𝑒6 − 0.4 , 𝑠6 = −𝑒6 − 0.4 .
The basis function is selected as Φ(𝑒, 𝑢) =(𝑒1 , 𝑒2 , 𝑒3 , 𝑒4 , 𝑒5 , 𝑒6 , 𝑔1 (𝑒1 , 𝑒3 ), 𝑔2 (𝑒1 , 𝑒5 ), 𝑢1 , 𝑢2 )𝑇 . The
initial state is 𝑥0 = (1.5 0 1.6 −0.1 1.5 0.1)𝑇 .
The initial parameters are set to be 𝑟0 = 𝑐0 = 𝑑0 = 1,
̂ 00 = 𝑟𝑎𝑛𝑑(6,10)𝑇 , 𝜖1 = 𝜖2 = 𝜖3 = 0.001.
ΘIn this section, we provide a nonlinear dual motor servo
system with backlash to validate the efficiency of ouralgorithm. The system can be modeled as:𝐽𝑖 𝜃̈𝑖 + 𝑏𝑖 𝜃̇𝑖 = 𝑢𝑖 − 𝜏𝑖{Table 1. The system parameters.
Value
Units

2

𝐽𝑚 𝜃̇𝑚 + 𝑏𝑚 𝜃̇𝑚 = ∑ 𝜏𝑖
𝑖=1

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
237

Powered by TCPDF (www.tcpdf.org)

Yuqi Zhang, Bin Zhang

3.

Hamidreza Modares and Frank L Lewis. Optimal tracking
control of nonlinear partially-unknown constrained-input
systems using integral reinforcement learning. Automatica,
50(7):1780–1792, 2014.
4. Ruizhuo Song, Frank L Lewis, Qinglai Wei, andHuaguang Zhang. Off policy actor-critic structure for
optimal control of unknown systems with disturbances.
IEEE transactions on cybernetics, 46(5):1041–1050, 2015.
5. Yuanheng Zhu, Dongbin Zhao, and Xiangjun Li. Using
reinforcement learning techniques to solve continuoustime non-linear optimal tracking problem without system
dynamics. IET Control Theory & Applications,10(12):1339–1347, 2016.
6. Fayez El-Sousy, Mahmoud M Amin, and Ahmed Al-Durra.
Adaptive optimal tracking control via actor-criticidentifier based adaptive dynamic programming forpermanent-magnet synchronous motor drive system. IEEE
Transactions on Industry Applications, 57(6):6577–6591,
2021.
7. Murad Abu-Khalaf and Frank L Lewis. Nearly optimal
control laws for nonlinear systems with saturating
actuators using a neural network hjb approach. Automatica,
41(5):779–791, 2005.
8. Bahare Kiumarsi, Frank L Lewis, Hamidreza Modares, Ali
Karimpour, and Mohammad-Bagher Naghibi-Sistani.
Reinforcement q-learning for optimal tracking control oflinear discrete-time systems with unknown dynamics.
Automatica, 50(4):1167–1175, 2014.
9. Girish V Chowdhary and Eric N Johnson. Theory andflight-test validation of a concurrent-learning adaptive
controller. Journal of Guidance, Control, and Dynamics,
34(2):592–607, 2011.
10. Milind M Lele and David H Jacobson. A proof of theconvergence of the kelley-bryson penalty function
technique for state-constrained control problems. Journalof Mathematical Analysis and Applications, 26(1):163–
169, 1969.

Fig. 1 and Fig. 2 are visual representations of theresults. We can see that compared with the initial
trajectory, the angles of load and motor have stabilizedaround the predetermined values.

Fig. 1. Comparisons of the tracking angles.

Fig. 2. Comparisons of the tracking angular velocities.

Authors Introduction

5. Conclusion
Mr. Yuqi Zhang

In this paper, we present a novel concurrent iterative
learning algorithm. The tracking cost function is selected
as a finite-horizon form. Our method does not requireprecise dynamic information and can handle with thetime-varying systems. Moreover, state constraints are
considered in the process of solving OTCP.

He received the B.S. degree in
Automation from Beijing Universityof Posts and Telecommunications,
Beijing, China, in 2020. He is currently
working toward the M.D. degree inControl Science and Engineering. His
research interests include differential
games and reinforcement learning.

Acknowledgements
Dr. Bin Zhang
He received his B.S. and Ph.D. degrees
both in control theory and applications
from Beihang University, Beijing,
China, in 2010 and 2016, respectively.
He is currently an Associate Professor
with the School of Artificial
Intelligence at Beijing University ofPosts and Telecommunications. His
research
interests
includereinforcement learning, multi-agent
systems, and intelligent control.

This work was supported by the National Natural
Science Foundation of China (Grant No. 61973044).
References
1.

2.

Firdaus E Udwadia. Optimal tracking control of nonlinear
dynamical systems. Proceedings of the Royal Society A:
Mathematical, Physical and Engineering Sciences,464(2097):2341–2363, 2008.
Mohamed Boukattaya, Mohamed Jallouli, and Tarak
Damak. On trajectory tracking control for nonholonomic
mobile manipulators with dynamic uncertainties andexternal torque disturbances. Robotics and autonomous
systems, 60(12):1640–1647, 2012.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
238

Privacy preserving Mean-square consensus for discrete-time heterogeneous multi-agent
systems with Communication Noises
Tongqing Yang

School of Mathematics and Statistics, Beijing Technology and Business University, 11 Fucheng Road, Haidian District,
Beijing, 100048, China
Lipo Mo

School of Computer and Artificial Intelligence, Beijing Technology and Business University, 11 Fucheng Road, Haidian
District, Beijing, 100048, China
Yingmin Jia

The seventh Division, Beihang university, 37 Xueyuan Road, Haidian District, Beijing, 100191, China
Email: yangtongqing08@126.com, molipo@th.btbu.edu.cn, ymjia@buaa.edu.cn.
www.btbu.edu.cn

Abstract
This paper investigates privacy-preserving mean-square consensus in distributed heterogeneous multi-agent systems
with communication noise on fixed undirected graphs. To mitigate the impact of communication noise, we introduce
a stochastic approximation step rule in the control protocol. Utilizing graph theory, stochastic analysis, and Lyapunov
theory, consensus conditions are derived. Subsequently, a cryptographic cryptosystem encrypts the designed protocol,
safeguarding against eavesdropping and information privacy loss between agents during consensus. Numericalsimulations confirm the efficacy of the proposed consensus protocol and privacy protection algorithm.

Keywords: Distributed, Heterogeneous multi-agent systems (HMAS), Consensus, Privacy preserving

been explored. A commonly used method is differential
privacy, where random noise is introduced to theinteraction state to obscure the true state [5]. However,this approach is susceptible to eavesdropping, as theadded noise compromises accuracy [6], [7]. Anotherapproach involves a state decomposition mechanism
introduced in literature. To address privacy concerns
without sacrificing accuracy, the study delves into thehomomorphic encryption scheme and extends theaverage consensus of first-order agent systems to secondorder agent systems [7], [8]. Nevertheless, practical
networks often operate in uncertain communication
environments with random perturbations. Consequently,
instead of assuming a deterministic form, literature
considers the mean square consensus of communicationnoise [9], [10]. However, it overlooks privacy protectionin the presence of communication noise, leaving thechallenge of achieving privacy-preserving consensus for
Heterogeneous Multi-Agent Systems (HMAS) with
communication noises unresolved. This paper presents a
distributed consensus algorithm for HMAS based on
cryptography. The main contributions of this paper are as
follows:

1. Introduction
In recent years, research on multi-agent systems has
profoundly impacted the development of engineering
technology [1]. Consensus control stands out as a key
issue in multi-agent system studies, aiming for theconvergence of state and output values of each agent
through a control protocol [2]. Extensive research results
exist for consensus control in multi-agent systems
comprising first or second-order agents [3]. However, thefocus is expanding to include Heterogeneous MultiAgent Systems (HMAS) consisting of both first-orderand second-order agents. HMAS demonstrates superiorload and task configuration capabilities, making itsuitable for more complex environments [4].
In the above research, the consensus control protocol
relies on the interaction of state information between
adjacent nodes to achieve consensus. For instance, agroup of agents aiming to converge at a specific location
may wish to keep their initial locations confidential. In
recent years, various privacy protection schemes have

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
239

Tongqing Yang, Lipo Mo, Yingmin Jia.

lim E x j (k ) − xi (k ) 2 = 0 i, j = 1, 2,

1) We propose a privacy consensus algorithm for
HMAS with communication noises based on partial
homomorphic cryptography;
2) The privacy protection for the initial state of theagent is analyzed in detail. In particular, compared withthe existing approach in [7], [8], our algorithm canachieve higher privacy when there exists an agent
connected with only one neighbor;3) The effectiveness of our method is verified bysimulation.

k →

lim E vi (k )

k →

2

= 0 i, j = 1, 2,,n,mLemma 1. Let { (k )},{ (k )} and {q(k )}, k = 0,1, , bereal
sequences,satisfying 0  q(k )  1, (k )  0，
 (k ),and (k )  0

 q(k ) = , q(k ) → 0, k → ,
k =0

 (k +1)  (1− q(k )) (k ) → 0, k → .

 (k ) +  (k ),then

2. Preliminaries and problem description
3. Main contents

The network of HAMS consisting of n agents isrepresented by an undirected connected graph G . For
HMAS with first-order and second-order agents, theLaplacian matrix of the graph G can be described as

3.1. Mean-square Consensus in Heterogeneous Multiagent Systems
We propose the following consensus protocol:

Ls + Dsf
L = −A fs−A sf ，L f + D fs nui (k ) =  (k ) aij ( x j (k ) + w ji (k ) − xi (k )) − k1vi (k ), i  Vm ,
j =1


n
u (k ) =  (k ) a ( x (k ) + w (k ) − x (k )), i = V .

i
ij
j
ji
i
n−m

j =1


where Ls and L f represent the Laplacian matrices of thegraph comprising the second-order agents and first-orderagents, respectively. A sf and A fs denote the adjacency

where  (k ) and k1 are the control gains, {wji (k ),matrix between the second-order agents (first-orderagents) and first-order agents (second-order agents)
respectively, Dsf and D fs denote their row sums.

k  0} are communication noise sequences.
Let yi (k ) = xi (k ) + vi (k )，i  Vm , then derived from (1)Paillier Cryptosystem: The Pailler crypto-system has
key generation, encryption and decryption functions. The
encryption operator E and the decryption operator D areusually used in the encryption and decryption process.
Now, we introduce the additive homomorphism propertyof the Paillier cryptosystem:
k

k

i =1

i =1

xi (k +1) = (1− T ) xi (k ) + Tyi (k ) .
Define  (k ) = [ x1 (k ),・, xm (k ), y1 (k ), , ym (k ), xm+1 (k ),, xn (k )] , W (k ) = [W (k ), ,WmT (k ),W1T (k ), , WnT (k )]T,  = diag{0, ,0, 1 , , n }, i = [ai1, ai 2 , , ain ], Wi (k)E( z ) k =  E( z ) = E( z ) = E(kz ).

T1= [w1i (k ), , wni (k )]T . Then the HMAS (1) can be
rewritten as

We assume that there are n agents in the HMAS, and thefirst group has m second-order agents, labeled Vm (k + 1) = [ I m+ n −  (k )T 1 − (1 −  (k ))T 2 ] (k )+ (k )T W (k )= {1,2, , m} , while the rest are first-order agents,
labeled Vn−m = {m +1, m + 2, , n} . The dynamic equation of the composed HMAS is as follows: xi (k + 1) = xi (k ) + vi (t )Tvi (k + 1) = vi (k ) + ui (t )T , i  Vm ,
 x (k + 1) = x (k ) + u (t )T , i  V .
i
i
n−m
 i

(2)

(3)

where

Im
−Im

1 = (1 − k1 ) I m + (Ls + Dsf ) (k1 − 1) I m
−A fsT0 Im
2 = (1 − k1 ) I m

0

(1)

−I m
(k1 − 1) I m00−A sf T 
(L f + D fs )T 

0
0 .
0

・
It is clear that  = [(k1 −1)1封m , 1m , 1n − m ] and 1n + m are the left
eigenvector and right eigenvector associated with zero
eigenvalue of  (k )T 1 + (1 −  (k ))T 2 , respectively.

where xi (k )  , vi (k )  and ui (k )  are the position, velocity and control input of agent i , respectively.
The sample time is 0  T  1 . All results can obtained
exist in n by using the Kronecker product.

Assumption 1. The additive independent noise {wji (k ),Definition 1. The HMAS consisted of (1) is said toachieve mean-square consensus if for any initialcondition, we havek  0} , E(wji (k )) = 0 , Var({wji (k )) = W .

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
240

Privacy Preserving Mean-square Consensus

Ei ( x j (k ))Ei (− xi (k ) + wi → j (k )) = Ei ( x j (k ) − xi (k )Assumption 2. Suppose that the control gains satisfy thefollowing conditions




k =0

k =0

+ wi → j (k )) → (Ei ( x j (k ) − xi (k ) + wi → j (k ))) ja (k )= Ei (a j (k )( x j (k ) − xi (k ) + wi → j (k )))1) a(k ) = ,0  a(k )  1, a(k + 1)  a(k )，2) a 2 (k )  .

then sends encrypted differences back to agent i .
5: Agent i decrypts the received ciphertexts withprivate key ski and computes weighted differences:

Lemma 2. If Assumptions 1-2 hold, 0  T  min{maxEi (a j (k )( x j (k ) − xi (k ) + wi → j (k ) + w j →i (k )))i  01Re2 (i ) + Im2 (i )

, max
i 0

1
Re2 (i ) + Im2 (i )}and k1  1+ max lii , lii ,→ a j (k )( x j (k ) − xi (k ) + w ji (k )) ai ( k )⎯⎯⎯→ aij ( x j (k ) − xi (k ) + w ji (k ))1,2, ,n , then 0   (k )T 1  1, 0  (1 −  (k ))T 2  1 ,6: The values calculated in step 5 are taken into
protocol (2) to update xi (k ) and vi (k ) .

where i , i = 1,2,

, n + m and i , i = 1,2, , n + m are theeigenvalues of 1 and 2 , 1 and 2 are the spectral

7: Set k  k +1.

radius of 2 and 2 , respectively. lii is the diagonal
element of the Laplacian matrix.

Definition 2 [7]: For a connected network of n nodes, theprivacy of the initial value xi (0) of node i is preserved if
an honest-but-curious adversary cannot estimate xi (0)with any accuracy.

Theorem 1. If Assumptions 1-2 hold, 0  T  min{maxi  01Re2 (i ) + Im2 (i )

, max
i 0

1
Re2 (i ) + Im2 (i )}and k1  1+ max lii , lii ,iSimilar to [7], the fixed coupling weight can be decomposed, aij = aij = ai (k )a j (k ) > 0, where ai (k ) is genera-= 1,2, ,n , then the HMAS (3) reach mean-squareconsensus.

ted when agent i transmits information and only known
by itself. Similarly, we split the noise delivered by acommunication link, i.e., wji (k ) = wi → j (k ) + wj →i (k ) in (2).

3.2. Privacy protection
In this paper, honest-but-curious and eavesdropper
were considered.

Theorem 2: In a connected network of nodes with system
dynamics given by (1), each agent adheres to the consensus protocol (2) and Algorithm 1 for state updates. An
honest-but-curious node, Eve, capable of receiving
messages from a neighboring node, Alice, cannot
ascertain Alice's initial state if Alice is also connected toanother legitimate node, Bob. Furthermore, Alice's
privacy remains undisclosed to external eavesdroppers.

Algorithm 1 Encrypted information exchange
Initialization: Each agent i initializes its state xi (0) ,the public key pki and private key ski are generated,and broadcasts pki to its neighbor(s) while keeps ski
private. Each time a message is sent, a independent
random variable wj →i (k ), i, j  Vm , k = 0,1,
isTheorem 3: In a connected network of nodes governedby system dynamics as described in (1), each agent
adheres to the consensus protocol (2) and Algorithm 1 forstate updates. If a node, Alice, is solely connected to therest of the network through an honest-but-curious node(or a colluding group of such nodes) Eve, and the noise
introduced by the communication link is symmetrical,
Eve can asymptotically deduce Alice's initial state.
Conversely, in the case of asymmetric noise, while Eve
can receive messages from its neighbor Alice, it lacks themeans to ascertain Alice's initial state.

added.
Input: xi (k ), k1 ; Output: ui (k ) .
1: Agent i encrypts xi (k ) with its public key pki ：−xi (k ) → Ei (−xi (k ))then sends Ei (−xi (k )) to each neighbor j  Ni , theequivalent of sending Ei (− xi (k ) + wi → j (k )) .
2: Each agent j encrypts x j (k ), and v j (k ) with its
public key pki :x j (k ) → Ei ( x j (k ))4. Numerical simulation

3: Each agent i generates the number ai (k ) and agent

j  Ni generates the number a j (k ) .

Consider the HMAS with eight agents, where nodes 14 represent first-order agents with input saturation andnodes 5-8 represent second-order agents. The network of
the eight agents between communication topology shown

4: Each agent j  Ni computes the encrypted
differences based on property of the Paillier
cryptosystem:©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
241

Powered by TCPDF (www.tcpdf.org)

Tongqing Yang, Lipo Mo, Yingmin Jia.

in Fig. 1(a). We assume k1 = 5 , given the initial value

4.

Y. Liu, H.B. Min, S.C. Wang, Z.G. Liu, S.Y. Liao,
Distributed consensus of a class of networked
heterogeneous multi-agent systems, J. Frankl. Inst.
351 (2014) 1700–1716.
5. X. Li, H. Yan, Z. Cheng, W. Sun and H. Li,"Protecting Regression Models With Personalized
Local Differential Privacy," in IEEE Transactions on
Dependable and Secure Computing, vol. 20, no. 2, pp.
960-974, 1 March-April 2023.
6. Zhang K, Li Z, and Wang Y, “Privacy-preserving
dynamic average consensus via state decomposition:
Case study on multi-robot formation control,”Automatica, vol. 139, May. 2022.
7. M. Ruan, H. Gao and Y. Wang, "Secure and PrivacyPreserving Consensus," in IEEE Transactions on
Automatic Control, vol. 64, no. 10, pp. 4035-4049,
Oct. 2019.
8. W. Fang, M. Zamani, and Z. Chen, “Secure andprivacy preserving consensus for second-ordersystems based on paillier encryption,” Syst.Control
Lett., vol. 148, p. 104869, 2021.
9. S. Guo, L. Mo, Y. Yu, Mean-square consensus ofheterogeneous
multi-agent
systems
withcommunication noises, Journal of the Franklin
Institute 355 (8) (2018) 3717-3736.
10. T. Li, J.-F. Zhang, Consensus conditions of multiagent systems with time-varying topologies andstochastic communication noises, IEEE Transactions
on Automatic Control 55 (9) (2010) 2043-2057.

x(0) = [1,2,3,4,5,6,7, 8] , v(0) = [1,2, 3,4] . Fig. 1 (b)封represents the case where the state of the agent converges
to 0. Fig. 1 (c) is the comparison between the real valueof the input of agent 1 and the encrypted ciphertext.

a. Network b. Convergencec. Control input
Fig.1. Numerical simulation picture

5. Conclusion
This paper explores privacy-preserving consensus in
distributed heterogeneous multi-agent systems with firstorder and second-order agents, addressing communication noise using graph theory, stochastic analysis,
Lyapunov theory, and cryptography. It proves that theclosed-loop system attains consensus. The designed
protocol is encrypted to prevent eavesdropping andinformation privacy loss. However, there are limitations
when a secure agent connects with an adversary alone.
Numerical simulations validate the consensus protocol's
effectiveness and the privacy protection algorithm's
ability to safeguard agent privacy.

Authors Introduction
Mr. Tongqing Yang
He received his Bachelor degree inEngineering in 2022 from the School of
Mechanical and Electrical Engineering,
Guizhou Minzu University in China. He iscurrently a master student in Beijing
Technology and Business University, China.

Acknowledgements
This work was supported by the National Natural
Science Foundation of China (6197020462).

Dr. Lipo Mo

References
1.

2.

3.

He received his B.S. degree in Department ofMathematics, Shihezi University, China, in
2003, and the Ph.D. degree in School ofMathematics and Systems Science from the
Beihang University, Beijing, in 2010.

W. Yue, Y. Yang and W. Sun, "Resilient Consensus
Control for Heterogeneous Multiagent Systems via
Multiround Attack Detection and Isolation
Algorithm," in IEEE Transactions on Industrial
Informatics, doi: 10.1109/TII.2023.3327175.
A. Nedic, A. Ozdaglar and P. A. Parrilo, "ConstrainedConsensus and Optimization in Multi-Agent
Networks," in IEEE Transactions on AutomaticControl, vol. 55, no. 4, pp. 922-938, April 2010.
H. Du, S. Li, and P. Shi, “Robust consensus algorithm
for second-order multi-agent systems with external
disturbances,” International Journal of Control, vol.
85, no. 12, pp. 1913–1928, 2012.

Yingmin Jia (Member, IEEE)He received the B.S. degree in control theory
from Shandong University, Jinan, China, in
1982, and the M.S. and Ph.D. degrees incontrol theory and applications from Beihang
University, Beijing, China, in 1990 and 1993,
respectively.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
242

Event-Triggered Consensus Control for Nonlinear Singular
Multi-Agent Systems under Directed Topology
Lin Li*, Tong Yuan

Automation Division, University of Shanghai for Science and Technology, Shanghai, China, 200093
Email: lilin0211@163.com, ytong77@163.com

Abstract
The event-triggered consensus problem for singular multi-agent systems with Lipschitz nonlinearity under directed
topology is investigated in this paper. A sampled-data-based event-triggered mechanism is constructed to decide when
the current data-packet should be broadcast. The objective is to design an event-triggered protocol such that theconsidered multi-agent system can achieve admissible consensus. Based on graph theory and singular system theory,
sufficient conditions of consensus of nonlinear singular multi-agent systems are derived. Finally, a numerical example
shows the effectiveness of our proposed approach.
Keywords: Singular multi-agent systems, Event-triggered scheme, Consensus control, Lipschitz nonlinearity

Euclidean norm of a vector or a matrix, respectively. In asymmetric matrix, * denotes the matrix entries implied bysymmetry. 𝐝𝐢𝐚𝐠{⋯ } stands for a block-diagonal matrix.

1. Introduction
In recent decades, the distributed coordinated control ofmulti-agent systems has attracted tremendous attention
due to its wide adoption in various fields, such as robot
formation [1], [2], [3], sensor network [4], [5], [6], neuralnetwork [7], [8], target tracking control [9] and so on.
Consensus is the basis problem of the coordinated control
of multi-agent systems. In practical, each agent may beequipped by the embedded microprocessor, which is withlimited computing and communication capabilities. How
to utilize these limited resources to operate systemefficiently becomes a concerned issue. Therefore, eventtriggered consensus has attracted much attention andachieved some results [10], [11], [12]. Singular systems,
can provide convenient and accurate description ineconomic systems, power systems and aircraft modelingfrom practical considerations. Some literatures havestudied the event-triggered for singular systems [13], [14],[15]. However, to the best of our knowledge, there are
few works regarding to the event-triggered consensus for
singular multi-agent systems (SMAS), which motivatesthe current study.

2. Problem formulation and preliminaries
2.1. Graph theory
A directed graph 𝒢(𝒱, ℰ, 𝒜) is considered in this paper,in which 𝒱 = {𝜈1 , 𝜈2 , ⋯ 𝜈𝑁 } is the node set, ℰ is the edge
set, and 𝒜 = [𝑎𝑖𝑗 ]𝑁×𝑁 is the adjacency matrix, 𝑎𝑖𝑗 > 0 if
𝜈𝑖 can receive the information from 𝜈𝑗 (𝑖, 𝑗 = 1,2, ⋯ 𝑁),
otherwise 𝑎𝑖𝑗 = 0 . The set of neighbors of the 𝜈𝑖 is
𝑗=𝑁
denotes as 𝒩𝑖 . Let 𝑑𝑖 = ∑𝑗=1 𝑎𝑖𝑗 be the in-degree of
node 𝜈𝑖 and 𝐷 = diag(𝑑1 , ⋯ , 𝑑𝑁 ). The Laplacian matrixof 𝒢 can be defined as ℒ = 𝐷 − 𝒜. Suppose the graph 𝒢
has a directed spanning tree.
2.2. Problem formulation
Consider a group of N agents, and the dynamic of 𝑖thagent is described by
𝐸𝑥̇ 𝑖 (𝑡) = 𝐴𝑥𝑖 (𝑡) + 𝐵𝑢𝑖 (𝑡) + 𝑓(𝑥𝑖 (𝑡)), 𝑖 = 1,2 ⋯ , 𝑁(1)where 𝑥𝑖 (𝑡) ∈ ℝ𝑛 , 𝑢𝑖 (𝑡) ∈ ℝ𝑚 denote the state and the
input, respectively. 𝐸, 𝐴 and 𝐵 are constant with𝑟𝑎𝑛𝑘(𝐸) = 𝑟 < 𝑛. The nonlinear function 𝑓(𝑥𝑖 (𝑡))satisfies ‖𝑓(𝜍1 ) − 𝑓(𝜍2 )‖ ≤ 𝜇‖𝜍1 − 𝜍2 ‖ , ∀𝜍1 , 𝜍1 ∈ ℝ𝑛with 𝜇 > 0 is the Lipschitz constant.

In the present study, we focus on the event-triggered
consensus control for a class of SMAS with Lipschitz
nonlinearity under directed topology. An even-triggeredmechanism excluding Zeno behavior is proposed, and an
artificial transition time-delay is introduced to design theevent-trigged consensus control protocol. Sufficientconditions that can guarantee the consensus of theconsidered SMAS are obtained. Throughout this paper,
𝑷 > 𝟎 (𝑷 ≥ 𝟎) means that 𝑷 is positive definite(positive semi-define). 𝑰𝑵 is the 𝑵 × 𝑵 identity matrix.
⨂ and ‖∙‖ are used to represent Kronecker product andDefinition 1: The SMAS (1) is said to achieve admissible
consensus if it is regular, impulse free while satisfying
𝐥𝐢𝐦 ∥ 𝒙𝒊 (𝒕) − 𝒙𝒋 (𝒕) ∥ = 𝟎, ∀𝒊, 𝒋 = 𝟏, 𝟐 ⋯ 𝑵.
𝒕→∞

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
243

Lin Li, Tong Yuan

where 𝜏𝑀 = max {𝜏𝑘𝑖 } , ∆ is a positive integer satisfying∆≥ 1. It is easy to see that𝑒𝑖 (𝑡𝑘𝑖 ℎ + ∆ℎ) = 0, 𝑡 ∈ [𝑡𝑘𝑖 ℎ + 𝜏𝑘𝑖 , 𝑡𝑘𝑖 ℎ + ℎ + 𝜏𝑀 ),𝑒𝑖 (𝑡𝑘𝑖 ℎ + ∆ℎ) = 𝑥𝑖 (𝑡𝑘𝑖 ℎ + ∆ℎ) − 𝑥𝑖 (𝑡𝑘𝑖 ℎ),

In this paper, we assume that the state of the multi-agent
system (1) is periodically sampled at a constant sampling
period 𝒉 > 𝟎. The sampling sequence is described by theset 𝕊𝟏 = {𝟎, 𝒉, 𝟐𝒉, ⋯ , 𝒌𝒉, ⋯ }. The transmission event isdetermined by an event-triggered scheme. The
transmission sequence of agent 𝒊 is described by the set
𝕊𝟐 = {𝟎, 𝒕𝒊𝟏 𝒉, 𝒕𝒊𝟐 𝒉, ⋯ , 𝒕𝒊𝒌 𝒉, ⋯ }.
The event detector is described as follows𝑡 ∈ [𝑡𝑘𝑖 ℎ + ∆ℎ + 𝜏𝑀 , 𝑡𝑘𝑖 ℎ + ∆ℎ + ℎ + 𝜏𝑀 )𝑒𝑖 (𝑡𝑘𝑖 ℎ + ∆ℎ) = 𝑥𝑖 (𝑡𝑘𝑖 ℎ + 𝑑ℎ) − 𝑥𝑖 (𝑡𝑘𝑖 ℎ),𝑖𝑖𝑡 ∈ [𝑡𝑘𝑖 ℎ + 𝑑ℎ + 𝜏𝑀 , 𝑡𝑘+1
ℎ + 𝜏𝑘+1)𝑖
𝑡𝑘+1ℎ = inf {𝑘ℎ|𝑘ℎ > 𝑡𝑘𝑖 ℎ, 𝜓𝑖 (𝑘ℎ) ≥ 0}
{𝜓𝑖 (𝑘ℎ) = 𝜒𝑖𝑇 (𝑡𝑘𝑖 ℎ + 𝑙𝑖 ℎ)Φ𝜒𝑖 (𝑡𝑘𝑖 ℎ + 𝑙𝑖 ℎ) −Then, we get
𝒙𝒊 (𝒕𝒊𝒌 𝒉) = 𝒙𝒊 (𝒕 − 𝝉𝒊 (𝒕) − 𝒆𝒊 (𝒕𝒊𝒌 𝒉 + ∆𝒉))Then, the event-triggered scheme (2) can be rewritten as(2)

𝛿𝑖 𝜆𝑇𝑖 (𝑡𝑘𝑖 ℎ + 𝑙𝑖 ℎ)Φ𝜆𝑖 (𝑡𝑘𝑖 ℎ + 𝑙𝑖 ℎ)}where 𝒌𝒉 = 𝒕𝒊𝒌 𝒉 + 𝒍𝒊 𝒉, 𝒍𝒊 ∈ ℕ, 𝜹𝒊 > 𝟎 is the thresholdparameter, 𝚽 is a positive definite matrix, and𝝌𝑻𝒊 (𝒕𝒊𝒌 𝒉 + 𝒍𝒊 𝒉)𝚽𝝌𝒊 (𝒕𝒊𝒌 𝒉 + 𝒍𝒊 𝒉) < 𝜹𝒊 [∑𝒋∈𝓝𝒊 𝒂𝒊𝒋 (𝒙𝒊 (𝒕𝒊𝒌 𝒉) −𝝌𝒊 (𝒕𝒊𝒌 𝒉 + 𝒍𝒊 𝒉) = 𝒙𝒊 (𝒕𝒊𝒌 𝒉 + 𝒍𝒊 𝒉) − 𝒙𝒊 (𝒕𝒊𝒌 𝒉){

𝝀𝒊 (𝒕𝒊𝒌 𝒉 + 𝒍𝒊 𝒉) = ∑

𝒋∈𝓝𝒊

𝑡0𝑖 ℎ

𝑡1𝑖 ℎ

2h

3h

4h

5h

6h

7h

𝑡2𝑖 ℎ

𝑡0𝑖 ℎ + 𝜏0 𝑡1𝑖 ℎ + 𝜏18h(𝑰𝑵 ⨂𝑬)𝒙̇ (𝒕) = (𝑰𝑵 ⨂𝑨)𝒙(𝒕) − (𝓛⨂𝑩𝑲)𝒙(𝒕 − 𝝉(𝒕))+ (𝓛⨂𝑩𝑲)𝒆(𝒌𝒉) + 𝐅(𝒙(𝒕))Let 𝑧𝑖 (𝑡) = 𝑥1 (𝑡) − 𝑥𝑖+1 (𝑡), 𝜀𝑖 (𝑘ℎ) = 𝑒1 (𝑘ℎ) − 𝑒𝑖+1 (𝑘ℎ),𝑇 (𝑡)]𝑇𝑧(𝑡) = [𝑧1𝑇 (𝑡), 𝑧2𝑇 (𝑡), ⋯ , 𝑧𝑁−1
= (𝑇1 ⨂𝐼𝑛 )𝑥(𝑡), 𝜀(𝑘ℎ) =𝑇𝑇𝑇[𝜀1 (𝑘ℎ), 𝜀1 (𝑘ℎ), ⋯ , 𝜀𝑁−1 (𝑘ℎ)] = (𝑇2 ⨂𝐼𝑛 )𝑒(𝑘ℎ), 𝑇1 =[1, −𝐼𝑁−1 ] ∈ ℝ(𝑁−1)×𝑁 , 𝑇2 = [0, −𝐼𝑁−1 ]𝑇 ∈ ℝ𝑁×(𝑁−1) . The

following system is immediate(𝑰𝑵−𝟏 ⊗ 𝑬)𝒛̇ (𝒕) = (𝑰𝑵−𝟏 ⨂𝑨)𝒛(𝒕) − (𝑻𝟏 𝓛𝑻𝟐 ⊗ 𝑩𝑲) 𝒛(𝒕 −𝝉(𝒕)) + (𝑻𝟏 𝓛𝑻𝟐 ⊗ 𝑩𝑲)𝜺(𝒌𝒉) + (𝑻𝟏 ⊗ 𝑰𝒏 )𝐅(𝒙(𝒕))(4)Now, the consensus problem of SMAS (1) is transformed
to the admissible of system (4).

9h

3. Main results
Theorem 1: Given 𝒉, 𝝉𝒎 , 𝝉̅𝑴 , and 𝝁 , the SMAS (1)achieves admissible consensus if there exist real matrices
𝑷, positive definite matrices 𝑸, 𝑹𝟏 , 𝑹𝟐 , 𝒁𝟏 , 𝒁𝟐 such thatthe following matrix inequalities hold𝑡3𝑖 ℎ

𝑡3𝑖 ℎ + 𝜏3

𝑡2𝑖 ℎ + 𝜏2

:sampling sequence𝒋

Substituting (3) into the system (1) yields

Under the event-triggered scheme (2), 𝒕𝒊𝒌+𝟏 𝒉 − 𝒕𝒊𝒌 𝒉denotes the transmission period of the Event generators.
Let 𝝉𝒊𝒌 denote the signal transmission delay of 𝒊th agent.
Suppose 𝝉𝒊𝒌 is bound, that is, 𝝉𝒊𝒌 ∈ (𝟎, 𝝉𝑴 ], where 𝝉𝑴 is apositive integer. In view of the effect of signaltransmission delay, the released states 𝒙𝒊 (𝒕𝒊𝒌 𝒉) will reachthe controller at the time instants 𝒕𝒊𝒌 𝒉+𝝉𝒊𝒌 .
Obviously, 𝒕𝒊𝒌 𝒉 + 𝝉𝒊𝒌 < 𝒕𝒊𝒌+𝟏 𝒉 + 𝝉𝒊𝒌+𝟏 (𝒌 = 𝟏, 𝟐 ⋯ ) .
Figure 1 shows the above-mentioned event-triggeredtransmission scheme.

h

𝒋

𝒋

𝒋
𝒂𝒊𝒋 (𝒙𝒊 (𝒕𝒊𝒌 𝒉) − 𝒙𝒋 (𝒕𝒌′ 𝒉))𝒋𝒋𝒋with 𝒌′𝒋 = 𝒂𝒓𝒈𝒎𝒊𝒏𝒑 {𝒕𝒊𝒌 + 𝒍𝒊 − 𝒕𝒑 |𝒕𝒊𝒌 + 𝒍𝒊 > 𝒕𝒑 , 𝒑 ∈ ℕ}.

0

𝑻

𝒋

𝒙𝒋 (𝒕𝒌′ 𝒉)))] 𝚽 [∑𝒋∈𝓝𝒊 𝒂𝒊𝒋 (𝒙𝒊 (𝒕𝒊𝒌 𝒉) − 𝒙𝒋 (𝒕𝒌′ 𝒉))]:transmission instant:arrived instant̅ 𝑻 𝑷𝑬̅≥𝟎
𝑬

Figure 1. Example of time evolution of the sampling
and transmission series of agent 𝒊𝑗𝑢𝑖 (𝑡) = −𝐾 ∑𝑗∈𝒩𝑖 𝑎𝑖𝑗 (𝑥𝑖 (𝑡𝑘𝑖 ℎ) − 𝑥𝑗 (𝑡𝑘 ′ ℎ))𝑗√𝝉𝒎 𝚿𝟏√𝝉𝑴𝒎 𝚿𝟐√𝝉𝒎 𝑳√𝝉𝑴𝒎 𝑴

∗

𝒁𝟏

𝟎

𝟎

𝟎

∗

∗

𝒁𝟐

𝟎

𝟎

∗

∗

∗

𝒁𝟏

𝟎

∗
𝚿𝟎 + 𝚼 + 𝚼 𝑻∗√𝝉𝒎 𝚿𝟏∗√𝝉𝑴𝒎 𝚿𝟐

∗
√𝝉𝒎 𝑳

𝒁𝟐 ]
√𝝉𝑴𝒎 𝑵

∗

𝒁𝟏

𝟎

𝟎

𝟎

∗

∗

𝒁𝟐

𝟎

𝟎

∗

∗

∗

𝒁𝟏

𝟎

∗

∗

∗

∗

𝒁𝟐

𝛀𝟏 =

In system (1), based on the event-triggered mechanism(2), consider the following control protocol

[

(3)
𝛀𝟐 =

where 𝒕 ∈ [𝒕𝒊𝒌 𝒉 + 𝝉𝒊𝒌 , 𝒕𝒊𝒌+𝟏 𝒉 + 𝝉𝒊𝒌+𝟏 ) , 𝒊 = 𝟏, 𝟐 ⋯ 𝑵, and𝑲 is the control gain matrix to be determined.

(5)𝚿𝟎 + 𝚼 + 𝚼 𝑻

[

where

Define the function 𝝉𝒊 (𝒕) as

φ0 = 𝐴̅𝑇 ℙ𝑇 + ℙ𝐴̅ + 𝑅1 + 𝑅2 + 𝜇2 𝐼,

𝝉𝒊 (𝒕)

𝛗𝟏 = −𝑸 + 𝝀(𝑻𝑻𝟐 𝑳𝑻 𝚲𝑳𝑻𝟐 ) ⊗ 𝚽,𝒕 − 𝒕𝒊𝒌 𝒉,𝒕 ∈ [𝒕𝒊𝒌 𝒉 + 𝝉𝒊𝒌 , 𝒕𝒊𝒌 𝒉 + 𝒉 + 𝝉𝑴 )𝒊𝒊
= {𝒕 − 𝒕𝒌 𝒉 − ∆𝒉, 𝒕 ∈ [𝒕𝒌 𝒉 + ∆𝒉 + 𝝉𝑴 , 𝒕𝒊𝒌 𝒉 + ∆𝒉 + 𝒉 + 𝝉𝑴 )𝒕 − 𝒕𝒊𝒌 𝒉 − 𝒅𝒉, 𝒕 ∈ [𝒕𝒊𝒌 𝒉 + 𝒅𝒉 + 𝝉𝑴 , 𝒕𝒊𝒌+𝟏 𝒉 + 𝝉𝒊𝒌+𝟏 )𝛗𝟐 = −𝝀(𝑻𝑻𝟐 𝑳𝑻 𝚲𝑳𝑻𝟐 ) ⊗ 𝚽,
𝛗𝟑 = −(𝑰𝑵−𝟏 ⊗ 𝚽) + 𝝀(𝑻𝑻𝟐 𝑳𝑻 𝚲𝑳𝑻𝟐 ) ⊗ 𝚽,

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
244

]

<𝟎

(𝟔)

<𝟎

(𝟕)

Event-triggered consensus Control

̅ 𝟎
𝚼 = [𝑳𝑬̅ (𝑴 − 𝑵)𝑬
̅ −𝑵𝑬
̅(𝑵 − 𝑴)𝑬̅ 𝟎
𝚿𝟏 = [𝒁𝟏 𝑨

𝒁𝟏 𝑳̅ 𝟎𝟎−𝒁𝟏 𝑳̅]𝑻 ,̅ 𝟎
𝚿𝟐 = [𝒁𝟐 𝑨

𝒁𝟐 𝑳̅ 𝟎𝟎−𝒁𝟐 𝑳̅]𝑻 ,

𝟎],

𝑻
𝑴𝒁−𝟏
𝟏 𝑴 𝜼(𝒕) + ∫̅ 𝑻 𝒁𝟐 𝑬̅ 𝒛(𝜶)𝒅𝜶𝒛̇ 𝑻 𝑬𝒕−𝝉(𝒕)𝒕−𝝉(𝒕)

̅ 𝒛̇ (𝜶)𝒅𝜶 ≤ (𝝉𝑴 − 𝝉(𝒕))𝜼𝑻 (𝒕)𝑬−𝟐𝜼𝑻 (𝒕)𝑵 ∫𝒕−𝝉̅𝑴ℙ = 𝑰𝑵−𝟏 ⨂(𝑬𝑻 𝑷 + 𝑻𝑺𝑻 ).

𝚿𝟎 =

𝒕−𝝉𝒎

𝒕−𝝉(𝒕)

𝑵𝒁−𝟏 𝑵𝑻 𝜼(𝒕) + ∫

𝟎

𝟎

ℙ𝑳̅

̅ 𝑻 𝒁𝟐 𝑬̅ 𝒛(𝜶)𝒅𝜶𝒛̇ 𝑻 (𝜶)𝑬

𝛗𝟎

ℙ

−ℙ𝑳̅

∗

−𝑰

𝟎

𝟎

𝟎

𝟎

∗

∗

𝛗𝟏

𝟎

𝟎

𝛗𝟐

∗

∗

∗

−𝑹𝟏

𝟎

𝟎

∗

∗

∗

∗

−𝑹𝟐

𝟎

It is obviously that

[∗

∗

∗

∗

∗

𝛗𝟑 ]

𝜺𝑻 (𝒌𝒉)(𝑰𝑵−𝟏 ⊗ 𝚽𝟏 )𝜺(𝒌𝒉) ≤ 𝝀𝒆𝑻 (𝒌𝒉)(𝑰𝑵 ⊗ 𝚽𝟏 )𝒆(𝒌𝒉)𝒕−𝝉̅𝑴By using the Kronecker product, the event-triggered
scheme can be written in the compact form as follows:
𝜺𝑻 (𝒌𝒉)(𝑰𝑵−𝟏 ⊗ 𝚽𝟏 )𝜺(𝒌𝒉)= 𝒆𝑻 (𝒌𝒉 )(𝑻𝑻𝟏 𝑰𝑵−𝟏 𝑻𝟏 ⊗ 𝚽𝟏 )𝒆(𝒌𝒉).

≤ 𝝀[𝒛(𝒌𝒉) − 𝜺(𝒌𝒉)]𝑻 [(𝑻𝑻𝟐 𝑳𝑻 𝚲𝑳𝑻𝟐 ) ⊗ 𝚽𝟐 ][𝒛(𝒌𝒉) − 𝜺(𝒌𝒉)]Proof: Firstly, similar to the analysis in [14], it can beproved that LMIs (6) and LMIs (7) can guarantee thatsystem (4) is regular and impulse-free.
Next, we will prove that the system (4) is
asymptotically stable. Construct the Lyapunov functional

where 𝝀 is the largest eigenvalue of 𝑻𝑻𝟏 𝑻𝟏 . 𝚲 =𝐝𝐢𝐚𝐠{𝜹𝟏 , 𝜹𝟐 , ⋯ 𝜹𝑵 }.
Since ‖𝒇(𝒙𝒊 (𝒕)) − 𝒇(𝒙𝒋 (𝒕))‖ ≤ 𝝁‖𝒙𝒊 (𝒕) − 𝒙𝒋 (𝒕)‖, Then,𝑭𝑻 (𝒛(𝒕))𝑭(𝒛(𝒕)) ≤ 𝝁𝟐 𝒛𝑻 (𝒕)(𝑰𝑵−𝟏 ⊗ 𝑰𝒏 )𝒛(𝒕). To sum up,
we can get

𝑽(𝐭) = 𝑽𝟏 (𝒕) + 𝑽𝟐 (𝒕) + 𝑽𝟑 (𝒕)̅ 𝑻 𝑷𝑬̅ 𝒛(𝒕) + (𝝉̅𝑴 − 𝝉(𝒕))𝒛𝑻 (𝒕 − 𝝉(𝒕))𝑸𝒛(𝒕𝑽𝟏 (𝒕) = 𝒛𝑻 (𝒕)𝑬− 𝝉(𝒕))𝒕𝑽̇(𝒕) ≤ 𝜼𝑻 (𝒕)𝛀𝜼(𝒕)
where

𝒕

𝒛𝑻 (𝜶)𝑬𝑹𝟏 𝒛(𝜶)𝒅𝜶 + ∫𝑽𝟐 (𝒕) = ∫𝒕−𝝉𝒎𝒛𝑻 (𝜶)𝑹𝟐 𝒛(𝜶)𝒅𝜶𝛀 = 𝚿 + 𝚼 + 𝚼 𝑻 + 𝝉𝒎 𝚿𝟏 𝒁𝟏−𝟏 𝚿𝟏𝑻 + (𝝉̅𝑴 − 𝝉𝒎 ) 𝚿𝟐 𝒁𝟏−𝟏 𝚿𝟐𝑻−𝟏 𝑻𝑻+ 𝝉𝒎 𝑳𝒁−𝟏
𝟏 𝑳 + (𝝉(𝒕) − 𝝉𝒎 )𝑴𝒁𝟏 𝑴
𝑻.
+ ((𝝉̅𝑴 − 𝝉(𝒕))𝑵𝒁−𝟏
𝑵
𝟏

𝒕−𝝉̅𝑴

𝒕

𝒕

𝑽𝟑 (𝒕) = ∫̅ 𝑻 𝒁𝟏 𝑬̅ 𝒛̇ (𝜶)𝒅𝜶𝒅𝜷
𝒛̇ 𝑻 (𝜶)𝑬∫𝒕−𝝉𝒎 𝒕+𝜷𝒕−𝝉𝒎+∫

The inequalities (6) and (7) are equivalent to 𝛀 < 𝟎, thatis 𝑽̇(𝒕) < 𝟎, which implies system (4) is asymptoticallystable. Hence, system (4) is admissible. From Definition
1, the multi-agent system (1) achieves admissible
consensus. This completes the proof.

𝒕̅ 𝑻 𝒁𝟐 𝑬̅ 𝒛̇ (𝜶)𝒅𝜶𝒅𝜷∫ 𝒛̇ 𝑻 (𝜶)𝑬

𝒕−𝝉̅𝑴

𝒕+𝜷

From 𝝉𝒎 ≤ 𝝉(𝒕) ≤ 𝝉̅𝑴 . And taking the derivative of𝑽(𝒕) with respect to 𝒕 along the solution of (4) yields
̅ 𝑻 (𝑰𝑵−𝟏 ⨂𝑷)𝑬̅ 𝒛(𝒕) +𝑽̇(𝒕) = 𝒛̇ 𝑻 (𝒕)𝑬𝑻𝑻̅̅ 𝒛̇ (𝒕) − 𝒛𝑻 (𝒕 − 𝝉(𝒕))𝑸𝒛(𝒕 − 𝝉(𝒕)) +𝒛 (𝒕)𝑬 (𝑰𝑵−𝟏 ⨂𝑷)𝑬𝒛𝑻 (𝑹𝟏 + 𝑹𝟐 )𝒛(𝒕) − 𝒛𝑻 (𝒕 − 𝝉𝒎 )𝑹𝟏 𝒛(𝒕 − 𝝉𝒎 ) − 𝒛𝑻 (𝒕 −
̅ 𝑻 𝒁𝟏 𝑬̅ 𝒛̇ (𝒕) + (𝝉̅𝑴 −𝝉̅𝑴 )𝑹𝟐 𝒛(𝒕 − 𝝉̅𝑴 ) + 𝝉𝒎 𝒛̇ 𝑻 (𝒕)𝑬
𝒕
𝑻
𝑻
𝑻
̅
̅
̅ 𝑻 𝒁𝟏 𝑬̅ 𝒛̇ (𝜶)𝒅𝜶 −𝝉𝒎 )𝒛 (𝒕)𝑬 𝒁𝟐 𝑬𝒛̇ (𝒕) − ∫𝒕−𝝉 𝒛̇ (𝜶)𝑬

4. Simulation example
Consider a SMAS with six agents shown in Figure 2.
There is a directed spanning tree in this graph.

𝒎𝒕−𝝉𝒎
̅ 𝑻 𝒁𝟐 𝑬̅ 𝒛̇ (𝜶)𝒅𝜶.
∫𝒕−𝝉̅ 𝒛̇ 𝑻 (𝜶)𝑬

6𝑴

For any free-weighting matrices L, M and N, we have𝒕

̅ 𝒛(𝒕) − 𝑬̅ 𝒛(𝒕 − 𝝉𝒎 ) − ∫𝟐𝜼𝑻 (𝒕)𝑳 [𝑬
𝒕−𝝉𝒎

𝟐𝜼

𝑻 (𝒕)𝑴̅ 𝒛̇ (𝜶)𝒅𝜶] = 𝟎𝑬𝒕−𝝉𝒎̅ 𝒛(𝒕 − 𝝉𝒎 ) − 𝑬̅ 𝒛(𝒕 − 𝝉(𝒕)) − ∫[𝑬̅ 𝒛̇ (𝜶)𝒅𝜶] = 𝟎𝑬𝒕−𝝉(𝒕)−𝟐𝜼𝑻 (𝒕)𝑴 ∫̅ 𝒛̇ (𝜶)𝒅𝜶 ≤ 𝝉𝒎 𝜼𝑻 (𝒕)𝑳𝒁−𝟏 𝑳𝑻 𝜼(𝒕)
𝑬

𝒕−𝝉𝒎

4

The dynamics of each agent described by (1)with 𝒇(𝒙𝒊 (𝒕)) = [𝟎 𝟎 −𝝁𝐬𝐢𝐧 (𝒙𝒊𝟏 )]𝑻 , and𝒕

𝟏
𝑬 = [𝟎𝟎𝒕+∫

3

Figure 2. The interaction topology of agents

where 𝜼(𝐭) = [𝒛𝑻 (𝒕), 𝑭𝑻 (𝐳(𝒕)), 𝒛𝑻 (𝐭 − 𝝉(𝒕)), 𝒛𝑻 (𝒕 −𝝉𝒎 ), 𝐳(𝒕 − 𝝉̅𝑴 ), 𝜺𝑻 (𝒌𝒉)]𝑻 . Then, we have
𝒕−𝝉𝒎

2

5

𝒕−𝝉(𝒕)
̅ 𝒛(𝒕 − 𝝉(𝒕)) − 𝑬̅ 𝒛(𝒕 − 𝝉̅𝑴 ) − ∫̅ 𝒛̇ (𝜶)] = 𝟎
𝟐𝜼𝑻 (𝒕)𝑵 [𝑬𝑬𝒕−𝝉̅𝑴−𝟐𝜼𝑻 (𝒕)𝑳 ∫1

̅ 𝑻 𝒁𝟏 𝑬̅ 𝒛̇ (𝜶)𝒅𝜶𝒛̇ 𝑻 𝑬

𝒕−𝝉𝒎

𝟎
𝟏
𝟎

𝟎
−𝟐
𝟎] , 𝑨 = [ 𝟎
𝟎
𝟐

𝟏
𝟎
𝟏

𝟏
𝟎. 𝟓
−𝟏 ] , 𝑩 = [𝟏]𝟎−𝟏Let 𝒉 = 𝟎. 𝟎𝟓, 𝝉𝒎 = 𝟎. 𝟎𝟐, 𝝉̅𝑴 = 𝟎. 𝟎𝟕, 𝝁 = 𝟎. 𝟓 and𝒙𝒊 = [𝒙𝒊𝟏 , 𝒙𝒊𝟐 , 𝒙𝒊𝟑 , 𝒙𝒊𝟒 , 𝒙𝒊𝟓 , 𝒙𝒊𝟔 ]𝑻 . The initial state is
selected as̅ 𝒛̇ (𝜶)𝒅𝜶 ≤ (𝝉(𝐭) − 𝝉𝒎 )𝜼𝑻 (𝒕)𝑬

𝒕−𝝉(𝒕)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
245

Lin Li, Tong Yuan

[𝒙𝟏 (𝟎) 𝒙𝟐 (𝟎) 𝒙𝟑 (𝟎) 𝒙𝟒 (𝟎) 𝒙𝟓 (𝟎) 𝒙𝟔 (𝟎)]𝟏. 𝟓
−𝟏𝟏 𝟏. 𝟓 −𝟏. 𝟓 𝟎. 𝟓=[ 𝟏
−𝟏. 𝟓 𝟏. 𝟓 𝟏−𝟏 −𝟏. 𝟓]−𝟏. 𝟓 𝟎. 𝟓 −𝟏 𝟏. 𝟓
𝟏
−𝟎. 𝟓
Solve the matrix inequalities (5), (6), (7), we can obtain
𝑲 = [𝟎. 𝟎𝟒𝟒𝟓 −𝟎. 𝟐𝟏𝟓𝟑 𝟎. 𝟏𝟒𝟖𝟗], and𝟐𝟗. 𝟒𝟏𝟕𝟒 𝟗. 𝟎𝟐𝟖𝟒
𝚽 = [ 𝟗. 𝟎𝟐𝟖𝟒 𝟒𝟑. 𝟒𝟖𝟎𝟔
𝟗. 𝟒𝟐𝟑𝟐 𝟑𝟐. 𝟖𝟗𝟖𝟕

𝟗. 𝟒𝟐𝟑𝟐
𝟑𝟐. 𝟖𝟗𝟖𝟕]
𝟒𝟖. 𝟗𝟕𝟓𝟖

The state trajectories of system (1) are shown in Figure
3, 4, 5. One can clearly see that all agent’s states canindeed reach consensus.
Figure 5. State trajectories 𝒙𝒊𝟑, (𝒊 = 𝟏, 𝟐 ⋯ 𝟔)
The transmission instants and release intervals areillustrated in Figure 6, which shows that the number ofthe sampled-data transmission is significantly decreased.

Figure 3. State trajectories 𝒙𝒊𝟏, ( 𝒊 = 𝟏, 𝟐 ⋯ 𝟔)

Figure 4. State trajectories 𝒙𝒊𝟐, (𝒊 = 𝟏, 𝟐 ⋯ 𝟔)

Figure 6. Transmission instants and release interval of
agent 𝒊(𝒊 = 𝟏, 𝟐)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
246

Powered by TCPDF (www.tcpdf.org)

Event-triggered consensus Control

13. X. Fan and Wang Z. Event-triggered sliding mode control
for singular systems with disturbance, Nonlinear Analysis
Hybrid Systems, 40(11), 2021, pp. 101011.
14. H. Wang, R. Lu and A. Xue, Network-based H∞ control
for singular systems with event-triggered sampling scheme,
Information Sciences, 329, 2016, pp. 540-551.
15. M. Li, J. Xia, J. Zhao. Event-triggered extended dissipative
control for networked singular systems. International
Journal of Control Automation and Systems, 19(1), 2021,
pp. 382–391.

5. Conclusion
Event-triggered consensus problem of nonlinear
SMAS with directed topologies has been investigated inthis paper. The conditions of achieving event-triggered
consensus were obtained, while consensus control gain
matrix is designed. The effectiveness of the proposed
method was verified by a numerical example. It would be
important and more practical to study consensus for
SMAS with leaders or under switching topologies, which
will be our further analysis.

Authors Introduction
Acknowledgements

Dr. Lin Li
She received her Ph.D. degree inControl
TheoryandControlEngineering from Beihang University
in 2010, Beijing, China. She iscurrently an Associate Professor at
University of Shanghai for Scienceand Technology. Her current researchinterests include robust control,
filtering, the multi-agent systems
control, and machine learning.

This work was supported by the National Natural
Science Foundation of China (No. 61673277).
References
1.

T. Liu and Z. Jiang, Distributed formation control ofnonholonomic mobile robots without global position
measurements, Automatica, 49(2), 2013, pp. 592-600.
2. X. Ge, Q. Han and X. Zhang, Achieving cluster formationof multi-agent systems under aperiodic sampling andcommunication delays, IEEE Transactions on Industrial
Electronics, 65(4), 2018, pp. 3417-3426.
3. X. Dong, B. Yu and Shi Z. Time-varying formation control
for unmanned aerial vehicles: theories and applications,
IEEE Transactions on Control Systems Technology, 23(1),2014, pp. 340-348.
4. J. He, H. Li and J. Chen, Study of consensus-based timesynchronization in wireless sensor networks, ISA
Transactions, 53(2), 2014, pp. 347-357.
5. H. Ji, L. Lewis and Z. Hou, Distributed informationweighted Kalman consensus filter for sensor networks,
Automatica, 77, 2017, pp. 18-30.
6. Y. Zhang and Y. Tian, A fully distributed weight design
approach to consensus Kalman filtering for sensor
networks, Automatica, 104, 2019, pp. 34-40.
7. C. Chen, G. Wen and Y. Liu, Adaptive consensus control
for a class of nonlinear multiagent time-delay systems
using neural networks, IEEE transactions on neuralnetworks and learning systems, 25(6), 2014, pp. 12171226.
8. G. Dong, H. Li and H. Ma, Finite-Time consensus tracking
neural network FTC of multi-agent systems, IEEE
Transactions on Neural Networks and Learning Systems,
32(2), 2020, pp. 653-662.
9. Y. Wang, H. Li and X. Qiu, Consensus tracking for
nonlinear multi-agent systems with unknown disturbanceby using model free adaptive iterative learning control,
Applied Mathematics and Computation, 365, 2020, pp.
124701.
10. D. V. Dimarogonas, E. Frazzoli and K. H. Johansson,
Distributed Event-triggered control for multi-agent
systems, IEEE Transactions on Automatic Control, 57(5),2012, pp. 1291-1297.
11. W. Zhu, Z. Jiang and G. Feng, Event-based consensus ofmulti-agent systems with general linear models,Automatica, 50(2), 2014, pp. 552-558.
12. E. Garcia, Y. Cao and D. W. Casbeer, Periodic eventtriggered synchronization of linear multi-agent systems
with communication delays, IEEE Transactions on
Automatic Control, 62(1), 2015, pp. 366-371.

Mr. Tong Yuan
He received his Master degree inApplied Mathematics from Shandong
University of Science and Technology
in 2022. He is currently a PhD student
at the University of Shanghai for
Science and Technology, China.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
247

An Improved Conversion Technique from EPNAT Models to VDM++ Specificationsfor Simulation of Abstract Software Behavior
Sho Matsumoto

Graduate School of Science for Creative Emergence, Kagawa University
2217-20 Hayashi-cho, Takamatsu-shi, Kagawa 761-0396, Japan
Ryoichi Ishigami

Graduate School of Science for Creative Emergence, Kagawa University
2217-20 Hayashi-cho, Takamatsu-shi, Kagawa 761-0396, Japan
Tetsuro Katayama

Department of Computer Science and Systems Engineering, Faculty of Engineering, University of Miyazaki
1-1 Gakuen-kibanadai nishi, Miyazaki-shi, Miyazaki 889-2192, Japan
Tomohiko Takagi

Department of Engineering and Design, Faculty of Engineering and Design, Kagawa University
2217-20 Hayashi-cho, Takamatsu-shi, Kagawa 761-0396, Japan
Email: s23g364@kagawa-u.ac.jp, takagi@eng.kagawa-u.ac.jp

Abstract
Formal software models based on EPNAT (Extended Place/transition Net with Attributed Tokens) can be converted
to VDM++ specifications that enable simulation of abstract software behavior before implementation processes.
However, the conversion technique has two problems, that is, (1) extracting all properties to be checked from the
VDM++ specifications requires time and effort, and (2) the structure of the VDM++ specifications has less readability
and maintainability. In this study, we improve the conversion technique by (1) adding features to extract an abstract
current state of software, and (2) dividing into classes that correspond to subnets of EPNAT models. This paper showsa new conversion rule, a new structure of VDM++ specifications, a simple example, and the discussion about theireffectiveness.
Keywords: Software model, Place/transition net, Vienna development method, Conversion techniquechecked in the simulation) from the VDM++
specifications requires time and effort. In the simulation,
a common VDM++ interpreter [3] is used to execute the
VDM++ specifications, and moreover, a particularfeature of our support tool to manage its original EPNAT
model needs to be used to process and obtain someproperties. Another is that the structure of the VDM++
specifications has less readability and maintainability.
The structure is not appropriately divided into modules
that correspond to the elements of its original EPNAT
model.

1. Introduction
Formal software models are useful to define
specifications and designs clearly, and to find failures
early in software development. EPNAT (Extended
Place/transition Net with Attributed Tokens), which isbased on place/transition nets and VDM++, has been
proposed as a formal software modeling language
especially for concurrent and distributed software that iscomposed of multiple subsystems (and/or systems) [1].
In an EPNAT model (that is, a formal software model thatis defined by using EPNAT), the behavior of eachsubsystem is defined as a subnet, and the interactions
between the subsystems are defined as glue transitions.
EPNAT models can be converted to VDM++
specifications, and enable simulation of abstract software
behavior before implementation processes [2].

To tackle the above problems, we propose an improved
conversion technique from EPNAT models to VDM++
specifications. In this technique, features to extract anabstract current state of software are added to the
VDM++ specifications in order to tackle the first problem.
Additionally, the structure of the VDM++ specifications
is divided into classes that correspond to subnets of
EPNAT models in order to tackle the second problems.
In general, the important things in the conversion of

However, the conversion technique has two problems.
One is that extracting all properties (that is, data to be©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
248

Sho Matsumoto, Ryoichi Ishigami, Tetsuro Katayama, Tomohiko Takagi

software models are correctness and automation [4], andthus we try to keep or improve them in this study. This
paper shows a new conversion rule, a new structure of
VDM++ specifications, and a simple example in section
2. The discussion about their effectiveness is given in
section 3, and then conclusion and future work are shownin section 4.

same behavior are expressed as symbols (the ranges aredefined by engineers). A symbol represents all thenumbers of attributed tokens in a corresponding range,
and therefore the number of markings to be covered canbe reasonably reduced. For example, in the purchasesubsystem of BSS (Fig. 1), an execution path by using
original markings [1, 0, 0]->t1->[2, 0, 0]->t1->[3, 0, 0]>t2->[2, 0, 0]->t2->[1, 0, 0] can be converted into an
execution path by using partitioned markings [1, 0, 0]>t1->[w, 0, 0]->t1->[w, 0, 0]->t2->[w, 0, 0]->t2->[1, 0, 0],
when the number of attributed tokens in p1 in the range of
two or more is represented as a symbol w. In this example,
the original markings [2, 0, 0] and [3, 0, 0] are replacedwith the partitioned marking [w, 0, 0].

2. Conversion Technique
The improvement of the previous technique to convert
from EPNAT models to VDM++ specifications consistsof introducing the features to obtain marking data
(discussed in section 2.1) and the basic class structure of
the VDM++ specifications (discussed in section 2.2). The
former is proposed for tackling the first problem, and thelatter is proposed for tackling the second problem.

In the simulation, if possible, a search algorithm should
avoid to revisit the same partitioned markings to improve
its efficiency. The features to obtain marking data will befrequently used in the simulation, since the marking data
is important properties to capture the abstract software
behavior. The way of adding the features to VDM++
specifications is discussed in the next section.

For the discussion in this study, we have constructed anexperimental EPNAT model of BSS (imaginary BusinessSupport Software) that is composed of a purchasesubsystem, a production subsystem, and a stock
management subsystem. Its overview is shown in Fig. 1.

2.2. Basic class structure of VDM++ specifications
t1

t2

p1

The structure of VDM++ specifications should be
divided into modules that correspond to the elements of
its original EPNAT model. Therefore, the basic classstructure of VDM++ specifications is constructed in thisstudy.

p2

The basic class structure is shown in Fig. 2. It is
composed of five kinds of classes, that is, “Software”,“Subsystem”, “AttributedToken”, “Places”, and“Marking”. “Subsystem” and “AttributedToken” are thesets of classes for specifying each subsystem. In each of
the sets, there are N classes (N expresses the number ofsubsystems), and one class peculiar to each subsystemneeds to be defined in the VDM++ specifications. The
others are classes for managing the data of all of thesubsystems. In the VDM++ specifications, they need to
be defined as one class, respectively.

p3

Fig. 1. Overview of an experimental EPNAT model.
2.1. Features to obtain marking data
The features to extract an abstract current state ofsoftware should be added to the VDM++ specifications.
In EPNAT models, the abstract states are expressed as the
distributions of attributed tokens, that is, markings.
Therefore, in this technique, the features to obtain
original markings and partitioned markings are added to
the VDM++ specifications. The idea of the partitioned
markings has been derived from equivalence partitioning
testing and coverability trees [5].

“Subsystem” is a part of “Software”, and corresponds
to the subnets of the original EPNAT model. There are
relations between “Subsystemi” and “AttributedTokeni”,
and also there are relations between “AttributedToken i”
and “Places” (1 ≤i≤N; ClassNamei expresses the i-thclass in the set ClassName). They mean that eachsubsystem keeps corresponding attributed tokens in its
own places. There are relations between “Marking” and“Subsystemi”, which means that original markings andpartitioned markings are derived from the places ofsubsystems. The operations that correspond to gluetransitions are defined in “Software”, since they have arole in the interaction between different subsystems. The
other transitions (hereinafter, referred to as normaltransitions) are peculiar to each subsystem, and thus theoperations that correspond to normal transitions are

An original marking is the list of the numbers of
attributed tokens in every place. For example, the original
marking in the purchase subsystem of BSS (Fig. 1) isexpressed as [p1, p2, p3] = [2, 0, 0]. In general, there is ahuge number of original markings in an EPNAT model,
and it is not so easy to cover all of them in the simulation.
In partitioned markings, the ranges of the numbers of
attributed tokens in which software seems to have the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
249

An Improved Conversion Techniquedefined in corresponding “Subsystemi”. Executing theoperations of the transitions changes the current marking,which corresponds to arcs. Therefore, the VDM++
specifications have no explicit elements that correspondto arcs.

specifications by using this technique. Also, we have
converted the experimental EPNAT model to VDM++
specifications by using the previous technique.
Hereinafter, the former is referred to as improved
specifications, and the latter is referred to as previous
specifications. We have compared the improved
specifications and the previous specifications, and havefound the followings.

Fig. 3 shows the overview of the VDM++
specifications of the experimental EPNAT model that are
based on the basic class structure. The classes“PurchaseSubsystem”, “StockManagementSubsystem”,
and “ProductionSubsystem” correspond to “Subsystem”
of the basic class structure. The classes “Customer”,“Goods”, and “ProductionMachine” correspond to“AttributedToken” of the basic class structure.
Marking
originalMarking
partitionedMarking
partitions
toOriginalMarking
toPartitionedMarking
createObject

Subsystem1
place1place2⁝
marking
fireNormalTransition1
fireNormalTransition2⁝

Placesnames
types (set or seq)
mapFromNamesToTypes
mapFromNamesToATTypes
createObject

AttributedToken1
currentPlaceattribute1
attribute2⁝
createObject•

Softwaresubsystem1
subsystem2⁝
fireGlueTransition1
fireGlueTransition2⁝•

Each set of the classes is referred
to as ClassName, and the i-thclass in each set is referred to as
ClassNamei.
There are N classes in each set.
(N is the number of subsystems.)
There are relations between the ith classes in the sets.

Fig. 2. Overview of the basic class structure.

•

BusinessSupportSoftware

PurchaseSubsystemstarting
previousPurchasepurchasingState
markinglogin
logoutmoveToPreviousPurchaseStatus
addBalanceerrorDueToInsufficientBalance
moveToPurchasingStatuspurchasepurchaseSubsystem
stockManagementSubsystemproductionSubsystem
putInCartcancelreportProblem
stopAndDeleteStockproduce

StockManagementSubsystem
dataUpdateCompleted
errorChecking
markingjudgementAsError
judgementAsNoError

Goods

CustomercurrentPlace
userName
cartGoods
balancesumPriceboughtGoods
createObject

currentPlace
goodsNamestockdefectiveFlag
defectiveContent
pricematerial
userNameWhoReportedTheError

ProductionSubsystemrunningState
stoppingState
marking

4. Conclusion and Future Work

autoMaintenancelargeScaleMaintenanceIn this paper, we have proposed an improved
conversion technique from EPNAT models to VDM++
specifications for the simulation of abstract software
behavior. The improvement consists of (1) introducing
the features to obtain marking data and (2) introducing
the basic class structure of the VDM++ specifications.
Regarding (1), the features will be frequently used in thesimulation, since the marking data is important properties
to capture the abstract software behavior. We haveconcentrated on obtaining marking data, but otherproperties also need to be collected to be efficiently
checked in the simulation. The features to collect otherproperties will be considered in future work. Regarding(2), the basic class structure establishes traceability
between EPNAT models and VDM++ specifications tosome extent, and avoids a certain kind of errors. However,VDM++ specifications by the improved conversion
technique are larger than VDM++ specifications by theprevious conversion technique. The basic class structure

ProductionMachinecurrentPlace
autoMaintenanceFlag
largeScaleMaintenanceFlag
createObject

createObject
Marking

Placesnames
types (set or seq)
mapFromNamesToTypes
mapFromNamesToATTypes
createObject

The improved specifications can additionallyprovide marking data (original markings andpartitioned markings). Also, the improved
specifications can provide the information about
individual attributed tokens by subsystems, andvariables by subsystems. These will be useful tosearch the VDM++ specifications for the failures ofsoftware in the simulation.
The elements of EPNAT models correspond to theelements of the improved specifications. There istraceability between EPNAT models and theimproved specifications to some extent. Additionally,
the improved technique can avoid the errors that theactions of normal transitions of a subsystem try toaccess the elements of other subsystems, since
subsystems are separately defined as classes in the
improved specifications.
The improved specifications and the previous
specifications consist of about 440 and 280 lines ofcode in VDM++, respectively. The previous
specifications do not have the features to obtainmarking data discussed in section 2.1, and theimproved specifications excluding the featuresconsist of about 340 lines of code. In any case, theimproved specifications are larger than the previous
specifications.

originalMarking
partitionedMarking
partitions

toOriginalMarking
toPartitionedMarking
createObject

Fig. 3. Overview of VDM++ specificationsof the experimental EPNAT model
based on the basic class structure.
3. Discussion
This section gives the discussion about theeffectiveness of this technique. We have manuallyconverted the experimental EPNAT model to VDM++©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
250

Powered by TCPDF (www.tcpdf.org)

Sho Matsumoto, Ryoichi Ishigami, Tetsuro Katayama, Tomohiko Takagi

may need to be refined further to improve the readability
and maintainability in future work.

Dr. Tetsuro Katayama
He received a Ph.D. degree in engineeringfrom Kyushu University, Fukuoka, Japan,
in 1996. From 1996 to 2000, he has been a
Research Associate at the Graduate Schoolof Information Science, Nara Institute of
Science and Technology, Japan. Since 2000
he has been an Associate Professor at theFaculty of Engineering, Miyazaki University, Japan. He iscurrently a Professor with the Faculty of Engineering,
University of Miyazaki, Japan. His research interests
include software testing and quality. He is a member of the
IPSJ, IEICE, and JSSST.

Acknowledgements
This work was supported by JSPS KAKENHI Grant
Number JP22K11976, and Young Scientists Fund of
Kagawa University Research Promotion Program 2021
(KURPP).
References
1.

2.

3.

4.

5.

Dr. Tomohiko Takagi
He received the B.S., M.S. and Ph.D.
degrees from Kagawa University in 2002,
2004 and 2007, respectively. He became an
assistant professor in 2008, and a lecturer in2013 in the Faculty of Engineering at
Kagawa University. Since 2018 he has been
an associate professor in the Faculty ofEngineering and Design at Kagawa University. His researchinterests are in software engineering.

T. Takagi and R. Kurozumi, Software ModelingTechnique and its Prototype Tool for Behavior of Multiple
Objects Using Extended Place/Transition Nets with
Attributed Tokens, Journal of Robotics, Networking andArtificial Life, Vol.7, No.3, pp.194-198, Dec. 2020.
S. Matsumoto, T. Katayama and T. Takagi, AutomatedRandom Simulation for Checking a Behavioral Model of
Systems Based on Extended Place/Transition Net with
Attributed Tokens, Proceedings of International
Conference on Artificial Life and Robotics, online, Japan,
pp.337-340, Feb. 2023.
J. Fitzgerald, P.G. Larsen, P. Mukherjee, N. Plat and M.
Verhoef, Validated Designs for Object-Oriented Systems,
Springer-Verlag London, 2005.
K. Lano and S. Kolahdouz-Rahimi, Model Transformation
Specification and Design, Advances in Computers, Vol.85,
pp.123-163, Elsevier, 2012.
T. Takagi and Z. Furukawa, Test Case Generation
Technique Based on Extended Coverability Trees,
Proceedings of 13th International Conference on Software
Engineering, Artificial Intelligence, Networking andParallel/Distributed Computing, IEEE, Kyoto, Japan, pp.
301-306, Aug. 2012.

Authors Introduction
Mr. Sho Matsumoto
He received the B.S. degree from Kagawa
University in 2023. He is a master's
student in the Graduate School of Science
for Creative Emergence at Kagawa
University. His research interests are insoftwareengineering,particularly
software quality control.
Mr. Ryoichi Ishigami
He received the B.S. degree from Kagawa
University in 2023. He is a master's student
in the Graduate School of Science forCreative
Emergenceat
KagawaUniversity. His research interests are insoftware engineering, particularly software
quality control.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
251

Prototype of RAGESS Which Is a Tool for Automatically Generating SwiftDiagram to Support
iOS App Development
Haruki Onaga*, Tetsuro Katayama*, Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, Naonobu Okazaki**

Department of Computer Science and Systems Engineering, Faculty of Engineering, University of Miyazaki,
1-1 Gakuen-kibanadai nishi, Miyazaki, 889-2192 Japan†
Department of Information Security, Faculty of Information Systems, Siebold Campus, University of Nagasaki
1-1-1 Manabino, Nagayo-cho, Nishi-Sonogi-gun, Nagasaki, 851-2195 Japan
E-mail: onaga@earth.cs.miyazaki-u.ac.jp, kat@cs.miyazaki-u.ac.jp, kita@sun.ac.jp,
yamaba@cs.miyazaki-u.ac.jp, aburada@cs.miyazaki-u.ac.jp, oka@cs.miyazaki-u.ac.jp

Abstract
It is difficult to understand the structure of large and complex mobile applications. To support iOS app development,
we proposed SwiftDiagram, a visualization of the static structure of Swift source code and confirmed its highusefulness. However, manually drawing SwiftDiagram is labor-intensive. This paper implemented a prototype of
RAGESS (Real-time Automatic Generation of SwiftDiagram System), a tool that automatically generates
SwiftDiagram by performing static analysis on Swift source code every time an iOS app build is successful.
Keywords: Software visualization, Mobile application, Swiftthe design and maintenance of iOS applications byvisualizing the following:

1. Introduction••

The smartphone and tablet application market isexpanding every year [1]. As a result, mobile applications
are becoming larger and more complex. The following
problems exist for developers of increasingly complex
mobile applications.
••

Static structure of type
Affected scope when making changes to the type

In addition, SwiftDiagram is composed of the following
four types of parts and one type of arrow. The appearanceof each part and arrow is shown in Fig. 1.

Difficult to understand the overall structure of theapplication
Difficult to keep track of where changes to thesource code may have an impact••

To solve the above problems through software
visualization, we have proposed SwiftDiagram.
SwiftDiagram is a diagram that visualizes the static
structure and impact scope of source code written in Swift,
a programming language used to develop iOS
applications.
However,manually
drawinga
SwiftDiagram takes time and effort. Therefore, this paper
implements RAGESS (Real-time Automatic Generationof SwiftDiagram System) to support the development of
iOS applications in Swift. RAGESS is a tool that
performs a static analysis of the Swift source code andautomatically draws the corresponding SwiftDiagram
when it detects that the target application builds
successfully.

•••

Header Part
Shows a type identifier, such as type categories andname
Details Part
Shows type components such as properties,
methods, protocol conformance, and so on
Extension Part
Shows a type extension
Nest Part
Shows a type that is declared nested inside anothertype
Affected scope (arrow)Shows that changes made to the type at the root of
the arrow may affect the component of the typepointed to by the arrow tip

2. SwiftDiagram
SwiftDiagram is a diagram that targets source code
written in the Swift programming language and supports©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
252

Haruki Onaga, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, Naonobu Okazaki

Fig. 1 Component parts of SwiftDiagram

Fig. 2 Overview of RAGESS••••

3. RAGESS

Slide bar to adjust arrows opacity
Button to display all arrows
Button to select a project directory
Button to select a build file

We implement RAGESS as a macOS application in Swift.
After launching RAGESS, the user can use RAGESS
functions by selecting the path to the project and its build
file for which the user wants to draw SwiftDiagram. This
chapter describes the overview and functions of
RAGESS.

3.2. Functions of RAGESS

3.1. Overview of RAGESS•RAGESS has the following two functions.

•Fig. 2 shows the overview of RAGESS implemented inthis paper. RAGESS consists of two areas, two slide bars,
and three buttons as shown below.
•••

Automatic real-time drawing of SwiftDiagram
corresponding the latest source code
Narrowing down the affected scope

RAGESS monitors the build file of the target project.
RAGESS statically analyzes the project's Swift sourcecode every time it detects a change in the build file andautomatically draws the corresponding SwiftDiagram.
This saves the user from the trouble of updating theSwiftDiagram and means that the source code
represented by the SwiftDiagram doesn’t contain any
syntax errors. Static analysis of Swift source code isArea displaying SwiftDiagram
Area displaying path and last modified date andtime
Slide bar to adjust the magnification rate in
SwiftDiagram©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
253

Prototype of RAGESS WhichFig. 3 Correspondence between SwiftDiagram and Swift source code in the sample project
performed by using SwiftSyntax [2], which is open
source by Apple. The user can scale the drawn
SwiftDiagram by operating the slide bar.

SwiftDiagram correspond to the structure name,properties, and dependencies by protocol conformance in
the Swift source code, respectively.

Representing the overall structure of a project in
SwiftDiagram could complicate the arrows representing
the affected scope and make it difficult for the user tounderstand the structure. When the user selects a header
part, RAGESS narrows down the display to only thosearrows that represent the affected scope of the changes
made to the type. The user can also operate the slide bar
to adjust the opacity of the arrows representing theaffected scope. The button to display all arrows, returns
the state of the narrowed down affected scope and theopacity of the arrows to their initial state, and displays all
the arrows that represent the affected scope for the entire
project.

Also, ProtocolA, SuperClass, SubClass, and StructB in
the SwiftDiagram extend arrows toward components ofthe type that may be affected by changes made to the type.
Hence, it is clear that RAGESS can statically analyze
Swift source code and draw the corresponding
SwiftDiagram.
5. Related work
Emerge [3] is a tool for visualizing Swift source code. It
is a browser-based tool that visualizes codebase anddependencies for multiple programming languages like
Swift, Kotlin, and TypeScript. Because it visualizes thesource code in a graph structure composed of nodes andedges, the user can explore and analyze it visually.

4. Application Example
By using an application example, we verify that the
RAGESS functionality works correctly for Swift sourcecode that defines multiple types. Fig. 3 shows thecorrespondence between the SwiftDiagram drawn by
RAGESS and the Swift source code in the example. Here,Fig. 2 shows a screen shot of RAGESS monitoring thebuild file of the project containing the Swift source codein the example, and when the build is successful.

Emerge analyzes source files under the source directorythat are written in a language set by the user and generates
files for displaying graphs in a Web browser. The user
can load the file with a Web browser to analyze thedependencies. However, Emerge cannot reflect changes
made to the target source code after analysis in the graphdisplayed in the Web browser. Also, because it takes
types and files as nodes, the user cannot analyze thedependencies of the type components. On the other hand,
RAGESS automatically draws SwiftDiagram every time
the target application builds successfully. In addition,
because arrows representing the dependencies areconnected to type components such as properties andmethods, the user can understand the dependencies inmore detail.

By looking at the StructA of the SwiftDiagram and theSwift source code in Fig. 3, the structure name andproperties of the structure in the SwiftDiagram
correspond to the structure name and properties of thestructure in the Swift source code, respectively. Similarly,
looking at StructC, the structure name, properties, anddependencies by protocol conformance in the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
254

Haruki Onaga, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, Naonobu Okazaki

And another tool for visualizing Swift source code is
Swiftcity [4], [5]. Swiftcity adapts the city metaphor,which has been studied for Java, C++, and otherlanguages, to unique type extensions, structures, andmore of Swift to map and visualize source code into cities
and buildings. The height of the building represents thenumber of lines of code (LOC) of the type, while thewidth and depth of the building represent the number ofmethods of the type. The user can use Swiftcity by
uploading a project file to the user’s Web browser.

RAGESS cannot coexist with the current Swift andSwiftSyntax versions and is not buildable.
Therefore, in this paper, we couldn’t evaluate theusefulness of RAGESS through experimentation inwhich subjects actually use RAGESS. We need to
implement RAGESS to adapt it to current Swift andSwiftSyntax versions and conduct subject
experimentation to confirm its usefulness.

Swiftcity is useful for getting an overview of theapplication. For example, yellow buildings represent typeextension, so if a city has many yellow buildings, itmeans that type extension are frequently used in its
source code. The variety of building colors also means
the use of all the standard features provided by Swift.
However, Swiftcity visualizes type categories and thenumber of methods, but doesn't visualize elements suchas properties and enumerated type cases, nor the
dependencies such as function calls and protocol
conformance. Therefore, use of Swiftcity is limited tounderstanding and analyzing an overview of theapplication in the maintenance process. On the other hand,
RAGESS can visualize the size of a type by the height of
the SwiftDiagram that combines each part and also
components and dependencies of the type.
Therefore, RAGESS provides a more detailed
visualization of the structure of applications than theother two tools that visualize Swift source code. In
addition, it updates the SwiftDiagram in real-time every
time the application builds successfully, thus reducing thetime the user needs to modify the deliverables.
6. Conclusion
In this paper, we have implemented RAGESS, a tool
statically analyzes the project's Swift source code every
time it detects a change in the build file and automaticallydraws the corresponding SwiftDiagram, in order tosupport the development of iOS applications in the Swiftprogramming language.

•

Implementation of SwiftDiagram editing
function
We enable SwiftDiagram to edit in the area
displaying SwiftDiagrams of RAGESS and reflect
the edits in the source code. This function could
assist the user in designing new developments oradding new function. We need to implement it
because we believe it feature will further increase
the usefulness of SwiftDiagram and RAGESS.

•

Implementation of search function
In the current RAGESS, the user must visually
search for the target type in the area displaying
SwiftDiagram to understand the structure of theapplication. We believe that implementing thefunction to search by type name or component willfurther reduce the user's time and effort.

•

Extension of supported syntax
RAGESS does not support type inference and can
only extract properties whose types are explicitly
indicated by type annotations. Since type inferenceis used in much Swift source code, we believe that
supporting type inference would further increase
the usefulness of RAGESS. And even if there are
more nested types within nested declared types,
RAGESS cannot extract that information. It is also
not yet compatible with Swift Macros released with
Swift 5.9. We plan to make RAGESS compatible
with them.

References1.

We have applied RAGESS to Swift source code andconfirmed that it can draw the corresponding
SwiftDiagram. We also have confirmed that RAGESS
can visualize the structure of the application in moredetail and in real-time compared to other tools thatvisualize Swift source code.

2.
3.
4.

Consequently, RAGESS is expected to support thedevelopment of iOS applications in the Swiftprogramming language.
5.

Future works are as follows:•

Evaluationofexperimentation

usefulness

by

Ministry of Internal Affairs and Communications,Japan, Information and Communications in Japan
WHITE PAPER 2022,
https://www.soumu.go.jp/johotsusintokei/whitepaper/eng/WP2022/2022-index.html
GitHub, swift-syntax, https://github.com/apple/swiftsyntax (Accessed 2023-12-14)
GitHub, emerge, https://github.com/glato/emerge(Accessed 2023-12-14)
Rafael Nunes, Marcel Rebouc a̧ s, Francisco SoaresNeto, Fernando Castor, Poster: Visualizing Swift
Projects as Cities, IEEE/ACM 39th International
Conference on Software Engineering Companion, pp.
368-370, 2017.
Swiftcity, https://swiftcity.github.io/swiftcity-app/(Accessed 2023-12-14)

subject©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
255

Powered by TCPDF (www.tcpdf.org)

Prototype of RAGESS Which

Authors Introduction
Naonobu Okazaki
He received his B.S, M.S., and Ph.D. degrees
in electrical and communication engineeringfrom Tohoku University, Japan, in 1986,
1988 and 1992, respectively. He joined theInformation Technology Research andDevelopment Center, Mitsubishi Electric
Corporation in 1991. He is currently aProfessor with the Faculty of Engineering, University ofMiyazaki since 2002. His research interests include mobile
network and network security. He is a member of IPSJ,
IEICE and IEEE.

Mr. Haruki Onaga
Haruki Onaga received the Bachelor's
degree in engineering (computer scienceand systems engineering) from the
University of Miyazaki, Japan in 2023. He
is currently a Master's student in Graduate
School of Engineering at the University ofMiyazaki, Japan. His research interests
include software development support through software
visualization.
Dr. Tetsuro Katayama
He received a Ph.D. degree in engineeringfrom Kyushu University, Fukuoka, Japan,
in 1996. From 1996 to 2000, he has been a
Research Associate at the Graduate Schoolof Information Science, Nara Institute of
Science and Technology, Japan. Since 2000
he has been an Associate Professor at theFaculty of Engineering, Miyazaki University, Japan. He iscurrently a Professor with the Faculty of Engineering,
University of Miyazaki, Japan. His research interests
include software testing and quality. He is a member of the
IPSJ, IEICE, and JSSST.
Dr. Yoshihiro Kita
Yoshihiro Kita received a Ph.D. degree insystems engineering from the University ofMiyazaki, Japan, in 2011. He is currently
an Associate Professor with the Faculty ofInformation Systems, University ofNagasaki, Japan. His research interests
include software testing and biometrics authentication.
Dr. Hisaaki Yamaba
He received the B.S. and M.S. degrees inchemical engineering from the Tokyo
Institute of Technology, Japan, in 1988 and1990, respectively, and the Ph D. degree insystems engineering from the University ofMiyazaki, Japan in 2011. He is currently an
Assistant Professor with the Faculty ofEngineering, University of Miyazaki, Japan. His researchinterests include network security and user authentication.
He is a member of SICE and SCEJ.
1.
Dr. Kentaro Aburada
He received the B.S., M.S, and Ph.D.
degrees in computer science and systemengineering from the University ofMiyazaki, Japan, in 2003, 2005, and 2009,
respectively. He is currently an Associate
Professor with the Faculty of Engineering,
University of Miyazaki, Japan. His research interests include
computer networks and security. He is a member of IPSJ and
IEICE.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
256

Extension to Support Types and Operation/Function Definitions in BWDM to Generate Test
Case Tool from the VDM++ Specification
Shota Takakura*, Tetsuro Katayama*, Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, and Naonobu Okazaki**Department of Computer Science and Systems Engineering, Faculty of Engineering, University of Miyazaki,
1-1 Gakuen-kibanadai nishi, Miyazaki, 889-2192 Japan†Department of Information Security, Faculty of Information Systems, Siebold Campus, University of Nagasaki
1-1-1 Manabino, Nagayo-cho, Nishi-Sonogi-gun, Nagasaki, 851-2195 Japan
E-mail: takakura@earth.cs.miyazaki-u.ac.jp, kat@cs.miyazaki-u.ac.jp, kita@sun.ac.jp,
yamaba@cs.miyazaki-u.ac.jp, aburada@cs.miyazaki-u.ac.jp, oka@cs.miyazaki-u.ac.jp

Abstract
Generating test cases from the formal specification description VDM++, which is a method for disambiguating
specifications, is time-consuming and labor intensive. Therefore, our laboratory has developed BWDM, a tool that
automatically generates test cases from VDM++ specifications. However, existing BWDM has problems that it only
supports integer types and cannot generate test cases for operation and function definitions including recursive
structure. Therefore, in order to improve the usefulness of BWDM, this paper extends BWDM to solve the above
problems. Consequently, it has confirmed that the use of extended BWDM can reduce the test case generation timecompared to manual test case generation.
Keywords: software testing, formal methods, VDM++, automatic generation, test cases

1. Introduction⚫

Today, our lives are supported by a lot of software. As aresult, software has become larger and more complex,
and software bugs have a greater impact on society. Oneof the causes of bugs in software is the use of natural
language in the upstream stage of software development.
This is because natural language contains ambiguity. One
method to solve this problem is to design software using
formal methods. VDM is one of the most common formal
methods used for this purpose, and VDM++ is a formal
specification description language used in VDM forobject-oriented modeling [1].
On the other hand, software design using VDM++
requires software testing. However, it is time-consuming
and labor-intensive to manually generate test cases from
specifications written in VDM++. In addition, manual
test case generation may cause leakage in testing.
Therefore, our laboratory has developed BWDM [2], [3],an automatic test case generation tool for VDM++
specifications. However, existing BWDM has the
problem that it only supports integer types and cannot
generate test cases for operation and function definitions
with recursive structure.

⚫

Addition of a function to generate test cases forenumerated type
Addition of a function to generate test cases for
operations/functions with recursive structure

In this paper, Chapter 2 describes the existing BWDM,
Chapter 3 describes the extended BWDM, Chapter 4shows application examples, and Chapter 5 discusses andconfirms the usefulness of BWDM. Chapter 6 is theconclusion.
2. BWDM
BWDM is a tool that automatically generates test cases
using the VDM++ specification as input. As examples,Fig. 1 shows the VDM++ specification and Fig.2 shows
the test cases generated from it by BWDM, respectively.
3. The Extended BWDMThis chapter describes the details of the extension of
BWDM. Fig. 3 shows the structure of the extended
BWDM.

Therefore, to improve the usefulness of BWDM, it isextended as follows.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
257

Shota Takakura, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, Naonobu Okazaki

Class calendar
functionsjudgeLeapYear: nat -> seq of charjudgeLeapYear(year) ==if(year mod 4 = 0 and ((year mod 100 = 0) =>(year mod 400 = 0))) then"Leap Year"else "Normal Year "end calendar

Function Name : judgeLeapYear
Argument Type : current_year : nat
Return Type : seq of (char)
Number of Test Cases : 17 cases(BVA:13 /SE:4)
Boundary Values for Each Argument
current_year : 4294967295 4294967294 0 -1 3 45 99 100 101 399 400 401
Test Cases for Boundary Value Analysis
No.1 : 4294967295 -> Undefined Action
No.2 : 4294967294 -> "Normal year"
No.3 : 0 -> "Leap year"
No.4 : -1 -> Undefined Action
No.5 : 3 -> "Normal year"
No.6 : 4 -> "Leap year"
No.7 : 5 -> "Normal year"
No.8 : 99 -> "Normal year"
No.9 : 100 -> "Normal year"
No.10 : 101 -> "Normal year"
No.11 : 399 -> "Normal year"
No.12 : 400 -> "Leap year"
No.13 : 401 -> "Normal year"

Fig. 1 Example of VDM++ specification
3.1. Addition of a Function to Generate Test
Cases for Enumerated TypeTo solve the problem of the existing BWDM only
supporting specific types, we extend Syntax Analyzer
and Symbolic Executor. Concretely, we extend BWDM
to generate test cases for enumerated type consisting of aset of unique identifiers.
In the extended Syntax Analyzer, if a type definition ofan enumerated type exists in the abstract syntax tree
analyzing process, the corresponding type declaration is
replaced with a list consisting of the declared name andvalues. If the substituted list exists in the syntax tree in
the Symbolic Executor, JavaAPI [4] which generates
enumerated type is used to generate constraints to beregistered with the SMT solver [5] used in the
propositional analyzer section. This additional process
can generate input data for enumerated type by symbolic
execution.

Fig. 2 Output when Fig.1 is applied to
the BWDM
recursively generating output data in the output data
generation section in Test Suite Generator.
4. Application Examples

3.2. Addition of a Function to Generate Test
Cases for Operations/Functions with Recursive
Structure

This chapter shows application examples to confirm thatthe extended BWDM works correctly.

To solve the problem of existing BWDM's inability togenerate test cases for operation and function definitions
with recursive structure, we extend Syntax Analyzer andTest Suite Generator.
In existing BWDM, when there is an operation/function
call in test suite generation, test cases cannot be generated
unless the parsing of the called operation/function iscompleted. This is the cause that the parsing of theoperation/function including recursive calls is notcompleted and the abstract syntax tree cannot be
generated.
Therefore, when there is an operation or function thathas self-recursive calls, an upper limit is set on thenumber of recursive calls, and the abstract parse tree
analysis process is modified so that it is executed after theparse tree analysis for all definitions is completed. This
allows the test data generation to be performed after theparsing of the VDM++ specification given as input iscompleted. Additionally, a function is added thatterminates the process if self-recursive calls are
performed more than a limited number of times when

4.1. Confirmation that Test Cases can be Generated
for Enumerated Type
Fig. 4, shows the VDM++ specification using
enumerated type used for verification, and Fig.5 shows
the results of applying it to the extended BWDM,
respectively.
From Fig. 4, it can see that in the "judgeLightColor"
function, "public TraficLight = <BLUE>|<YELLOW>|<RED>;"is defined as the type. As output, Fig.5 shows that test
cases can be generated for the definition using theenumerated type.
4.2. Confirmation that Test Cases can be Generated
for Definitions with Recursive Structure
Fig. 6 shows the VDM++ specification using a definition
with recursive structure used for verification, and Fig 7©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
258

Extension to Support Types

Fig. 3 The structure of the extended BWDMClass judgeLightColor

Function Name : judgeLightColor
Argument Type : color:
Return Type : seq of (char)Typespublic TrafficLight = <BLUE> | <YELLOW> | <RED>;functionsjudgeLightColor: TrafficLight ==> seq of charjudgeLightCoulor (color) ==if color = <BLUE >"The color is blue. "if color = <YELLOW>"The color is yellow"
if color = <RED>"The color is red."

Test Cases by Symbolic Execution
No.1 : <BLUE> -> "The color is blue"
No.2 : <YELLOW> -> "The color is yellow"
No.3 : <RED> -> "The clolor is red"Fig. 5 Output when Fig.4 is applied tothe extended BWDM
enumerated type. Thus, the usefulness of BWDM has
been improved.

end judgeLightColor

Fig. 4 VDM++ specification using enumerated type5.2. Evaluation of Test Case Generation for
Definitions with Recursive Structure

shows the results of applying it to the extended BWDM,
respectively.
From Fig. 6, it can see that in the "calcSum" function isdefined with recursive structure. As output, Fig.7 showsthat test cases can be generated for definition withrecursive structure.

As shown in Section 4.2, we have confirmed that theextended BWDM can generate test cases from definition
with recursive structure in the VDM++ specification ofFig. 6. Therefore, it can be said that this extension enables
BWDM to generate test cases from definitions withrecursive structure. Thus, the usefulness of BWDM has
been improved.

5. Discussion
5.1. Evaluation of Test Cases Generation for
Definitions using Enum Type5.3. Comparison with Manual Test Case GenerationTime

As shown in Section 4.1, we have confirmed that theextended BWDM can generate test cases from definitions
using enumerated type in the VDM++ specification ofFig. 4. Therefore, it can be said that this extension enables
BWDM to generate test cases from definitions using

Using the extended BWDM, we evaluate the test casegeneration time for the VDM++ specifications which
include definitions with enumerated type and recursive
structure, by comparing it to the human case. Two
VDM++ specifications, Fig. 4 and Fig. 6, are used in the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
259

Shota Takakura, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, Naonobu Okazaki

Table.1 Comparison of test case generation time

Class calcSum
factorial: nat ==> nat
factorial (value) ==if value = 0 then0else value + factorial (value - 1)

6. Conclusion
In order to improve the usefulness of BWDM, anautomatic test case generation tool for the VDM++
specification, two extensions have been made to theexisting BWDM to solve the problems that only supports
integer types and that it cannot generate test cases for
operation and function definitions with recursive
structure.
The application examples of the extended BWDM are
shown, and the test case generation function for
definitions using enumerated type and definitions withrecursive structure for expanding the range of supported
types is confirmed. We have confirmed that the extended
BWDM can reduce test case generation time by about 14
minutes compared to manually generating test cases from
VDM++ specifications. In addition, we have confirmedthat the extended BWDM can eliminate human errors.
Therefore, it can be said that the usefulness of BWDM
has been improved by the extensions in this paper.
The future issues are the following.

end calcSum
Fig. 6 VDM++ specification with recursive structure
Function Name : calcSum
Argument Type : value:nat
Return Type : nat
Boundary Values for Each Argument
value: 4294967295 4294967294 0 -1
Test Cases of Boundary Values
No.1 : 4294967295 -> Undefined Action
No.2 : 4294967294 -> Undefined Action
No.3 : 0 -> 0
No.4 : -1 -> Undefined Action
Fig. 7 Output when Fig.6 is applied tothe extended BWDM⚫
⚫

experiment. The manual verification is performed by atotal of five people, two graduate students and threefourth-year undergraduates in our laboratory. We
measure the time required to generate sufficient test cases
from the VDM++ specification, which are no leakage.
The time measurement is terminated when the subjects
have generated the correct test cases, and when the
manually test cases includes any incorrect test cases, themistakes are pointed out. The comparison results areshown in Table.1.
Table 1 shows that about 14 minutes are saved whenusing the extended BWDM to generate test cases
compared to manual test case creation. In addition,
human error has been observed in the manual test casecreation.
In this research, we have confirmed that an extended
BWDM that adds a test case generation function for
definitions using enumerated type and recursive structure
can reduce the time and human errors required for test
case generation. Therefore, it can be said that theusefulness of BWDM has been improved.

⚫

Support for types other than integer andenumerated type
Test case generation for input values with morerecursive calls
Test case generation for mutually recursivefunctions

References
1.
2.

3.

4.
5.

Overture
Project.
Manuals.
http://overturetool.org/documentation/manuals.html.
Accessed: 2023-12-13.
T. Katayama, F. Hirakoba, Y. Kita, H. Yamaba, K.
Aburada, and N. Okazaki. Application of Pirwise Tsting
into BWDM which is a Test Case Generation tool for the
VDM++ Specification. Journal of Robotics, Networking
and Artificial Life, Vol.6, No.3, pp.143-147, 2019.
T. Muto, T. Katayama, Y. Kita, H. Yamaba, K. Aburada,
and N. Okazaki. Expansion of Application Scope andAddition of a Function for Operations into BWDM which
is an Automatic Test Cases Generation Tool for VDM++
Specification. Journal of Robotics, Networking andArtificial, Vol.9. No.3, pp.255-262, 2022.
Z3:Packagecom.microsoft.z3.
https://z3prover.github.io/api/html/namespacecom_1_1
microsoft_1_1z3.html. Accessed: 2023-12-13.
Z3. https://github.com/Z3Prover/z3. Accessed: 2023-1214.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
260

Powered by TCPDF (www.tcpdf.org)

Extension to Support Types

Dr. Naonobu Okazaki
He received his B.S, M.S., and Ph.D. degrees
in electrical and communication engineeringfrom Tohoku University, Japan, in 1986,
1988 and 1992, respectively. He joined theInformation Technology Research andDevelopment Center, Mitsubishi Electric
Corporation in 1991. He is currently aProfessor with the Faculty of Engineering, University ofMiyazaki since 2002. His research interests include mobile
network and network security. He is a member of IPSJ,
IEICE and IEEE

Authors Introduction
Mr. Shota Takakura
Shota Takakura received the Bachelor's
degree in engineering (computer scienceand systems engineering) from the
University of Miyazaki, Japan in 2023. He
is currently a Master's student in Graduate
School of Engineering at the University ofMiyazaki, Japan. His research interests
include software testing, software quality, and formal
method.
1.
Dr. Tetsuro Katayama
He received a Ph.D. degree in engineeringfrom Kyushu University, Fukuoka, Japan, in
1996. From 1996 to 2000, he has been a
Research Associate at the Graduate Schoolof Information Science, Nara Institute of
Science and Technology, Japan. Since 2000
he has been an Associate Professor at theFaculty of Engineering, Miyazaki University, Japan. He iscurrently a Professor with the Faculty of Engineering,
University of Miyazaki, Japan. His research interests
include software testing and quality. He is a member of the
IPSJ, IEICE, and JSSST.
Dr. Yoshihiro Kita
Yoshihiro Kita received a Ph.D. degree insystems engineering from the University ofMiyazaki, Japan, in 2011. He is currently an
Associate Professor with the Faculty ofInformation Systems, University ofNagasaki, Japan. His research interests
include software testing and biometrics
authentication.
Dr. Hisaaki Yamaba
He received the B.S. and M.S. degrees inchemical engineering from the Tokyo
Institute of Technology, Japan, in 1988 and1990, respectively, and the Ph D. degree insystems engineering from the University ofMiyazaki, Japan in 2011. He is currently an
Assistant Professor with the Faculty of Engineering,
University of Miyazaki, Japan. His research interests include
network security and user authentication. He is a member of
SICE and SCEJ.
Dr. Kentaro Aburada
He received the B.S., M.S, and Ph.D.
degrees in computer science and systemengineering from the University ofMiyazaki, Japan, in 2003, 2005, and 2009,
respectively. He is currently an Associate
Professor with the Faculty of Engineering,
University of Miyazaki, Japan. His researchinterests include computer networks and security. He is amember of IPSJ and IEICE.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
261

Proposal of ASLA Which Is a Segmentation and Labeling Tool for Document Images Based on
Deep Learning
Kanta Kakinoki*, Tetsuro Katayama*, Yoshihiro Kita†,
Hisaaki Yamaba*, Kentaro Aburada*, and Naonobu Okazaki**

Department of Computer Science and Systems Engineering, Faculty of Engineering, University of Miyazaki,
1-1 Gakuen-kibanadai nishi, Miyazaki, 889-2192 Japan†
Department of Information Security, Faculty of Information Systems, Siebold Campus, University of Nagasaki
1-1-1 Manabino, Nagayo-cho, Nishi-Sonogi-gun, Nagasaki, 851-2195 Japan
E-mail: kakinoki@earth.cs.miyazaki-u.ac.jp, kat@cs.miyazaki-u.ac.jp, kita@sun.ac.jp,
yamaba@cs.miyazaki-u.ac.jp, aburada@cs.miyazaki-u.ac.jp, oka@cs.miyazaki-u.ac.jp

Abstract
This paper proposes a prototype of ASLA, segmentation and labeling tool for document images based on deeplearning, to reduce the time required for region segmentation and label generation. To evaluate the usefulness of
ASLA, we have compared the time required for region segmentation and label generation using ASLA and by hand,
and then confirmed the reduction in time. We also have confirmed that the rule-based region redividing method
achieves a high recall and precision.
Keywords: Region segmentation, Labeling, Document image, Rule-based region redividing

1. Introduction

However, these tasks are time-consuming and laborintensive in the manual. This paper proposes a prototypeof ASLA (Automatic Segmentation and Labeling tool
using AI), segmentation and labeling tool for document
images based on deep learning, to reduce the timerequired for region segmentation and label generation.

Because storage space is needed to store paper documents,
maintaining and managing their condition are costly.
Electronic documents are used to solve this problem [1].
The use of electronic documents enables the reduction of
paper and the cost of the maintenance and management.
Therefore, the use of electronic documents is becoming
more widespread.
The current situation of the electronic documents isonly a substitute for paper. As a new way to utilize
electronic documents, we focus on dividing electronic
documents into regions by their elements and generating
keywords and sentences as labels from the contents of theelements. In addition, the movement of the reader's lineof sight when they read the electronic document isrecorded as coordinate information. By realizing thesefeatures, it is possible to improve the efficiency of sales
activities by capturing the level of interest and concern ofcustomers, and to strengthen compliance by checking
whether important matters are properly explained tosubscribers of products or services.
To achieve the above, the following two tasks are
required.
• Region segmentation that divides the components ofan electronic document into some regions.
• Label generation that analyzes the contents of aregion and then generates labels according to thecontents of the region.

2. ASLA
2.1. Functions
ASLA takes a single column or two columns Japanese
document image as input. ASLA outputs segmentation
image and class identification on the input image. Table
1 shows the definitions of class and region in ASLA. The
output is a segmentation image and an analysis result file.
The segmentation image is a JPEG format image. Eachregion of the image is color-coded and visualized as red
for the text class region, blue for the figure class region,and green for the table class region. The analysis result
file is a CSV (Comma Separated Value) format file. Inthat file, the top-left x-coordinate, top-left y-coordinate,bottom-right x-coordinate, bottom-right y-coordinate,class name, and label pair in each region are recorded perline.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
262

Kanta Kakinoki, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, Naonobu Okazaki

Table 1 Definition of classes and regions in ASLA.
Class Name
TargetDefinition of regions
Text
Text and
Smallest rectangle
title, etc.
surrounding each element
FigureFigureSmallest rectangle
surrounding the figure
Table
Table
Smallest rectangle
surrounding the table

2.2. Dataset Preparation
Fig. 1 The structure of ASLA.
ASLA uses Cascade R-CNN [2] as the object detector forregion segmentation. In addition, ASLA uses
LayoutLMv3 [3] that is a pre-trained model specific to
document AI tasks as a training model for the object
detector. To improve the performance of region
segmentation for Japanese document images, we finetune the model by using a dataset of Japanese document
images prepared in advance.
The ASLA prototype uses 103 document images of thedoctoral dissertation of the Graduate School of
Educational and Developmental Sciences of Nagoya
University provided by the Nagoya University Academic
Repository [4] and 87 document images of the paper
presented at the 2021 Kyushu Branch Joint Conferenceof the Institutes of Electrical and Related Engineers [5] as
document images dataset for fine-tuning.

Fig. 2 The Process of rule-based region redividing.
•

2.4. Rule-Based Region Redividing

2.3. Structure

There are several studies that use object detectors to
divide regions on document images [3], [6], [7]. The
purpose is to roughly divide each region in a document
image, such as layout analysis of a document image.
Therefore, the existence of small misalignments between
the divided regions and the actual regions is not a major
problem. However, region segmentation in ASLA
extracts the characters within a region after the region isdivided. Therefore, if there is even a small misalignment,
some characters cannot be extracted. This causes the
problem that correct labels cannot be generated. Anotherproblem with segmentation by the object detectors existsthat it is difficult to detect small regions [6], [7].
Therefore, ASLA performs rule-based region
redividing. This eliminates the misalignment of regionswith region segmentation by the object detector, and also
divides smaller regions that were not detected before. Fig.
2 shows the flow of the rule-based region redividing. In
the region redividing process, each binding component
extracted by the binding component extraction process
determines which region is the same as the region thatwas divided using the object detector. Alternatively,
binding components that are far from any region are
divided as a new region. Fig. 3 shows an example of theregion redividing process in rule-based region redividing.
In Fig. 3, the region redividing process redivides theregion to eliminate misalignment of text region
segmented by Cascade R-CNN.

Fig. 1 shows the structure of ASLA. It consists of thefollowing three processing parts.
••

Analysis Result Output Part:
Analysis Result Output Part outputs an analysis
result file as a CSV format file based on the analysis
results obtained by Region Segmentation Part andLabel Generation Part.

Region Segmentation Part:
Region Segmentation Part at first generates a trainedmodel by fine-tuning LayoutLMv3 with a previously
prepared dataset. Next, it divides region by using
Cascade R-CNN from the input document image
with the generated trained model. This generates the
top-left and bottom-right xy-coordinates and classnames for each region. Thereafter, it redivides theregion to modify the misalignment of regions. The
rule-based region redividing is described later in
Section 2.4. Finally, based on the results of region
segmentation, a segmentation image is output. Since
this, text class regions are referred to as text region,
figure class regions as figure region, and table classregions as table region, respectively.
Label Generation Part:
Label Generation Part generates labels that are allnouns included in the text region for the text region.
For the figure and table regions, it generates a labelthat is the text of each caption. Here, it extracts
characters from Japanese text using Tesseract that is
an OCR tool, and it generates labels from theextracted characters using MeCab which is amorphological analysis tool.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
263

Proposal of ASLA WhichFig. 3 An example of the region redividing process inrule-based region redividing.
Fig. 6. Result output file image
Table 2. Comparison of the time required for
ASLA and manual region segmentation.
Single column
Two columns Average
Manual
2m6s
4m8s
3m7s
(Average)
ASLA
5s
6s
6s
Table 3. Comparison of the time required for
ASLA and manual labeling.
Time
Manual (Average)
8m40s
ASLA
5s

3. Evaluation

Fig. 4 Input document image sample

We evaluate the usefulness of ASLA developed in thispaper. It is evaluated in terms of the execution time andperformance of region segmentation.
3.1. Evaluation of the Execution Time
We compare the time required for region segmentationby using ASLA and manually. Two document images areused for region segmentation. They are single column
document image and two columns document image. In
the manual region segmentation, five subjects divide
regions on document images, and their working time ismeasured. Table 2 shows the comparison of the timerequired for region segmentation by using ASLA andmanually. From Table 2, we have confirmed that ASLA
reduces the time required for region segmentation by
about 3 minutes (97%) per document image.
We compare the time required for label generation byusing ASLA and manually. In the manual label
generation, five subjects generate labels on document
images that had been divided regions in advance, andtheir working time is measured. This labeling task
involves extracting all nouns within each region. Table 3
shows a comparison of the time required for ASLA andmanual label generation. From Table 3, we haveconfirmed that ASLA reduces the time required for label
generation by about 8 minutes (99%) per document
image. Based on the above, it can be said that ASLA
helps reduce the time required for label generation.

Fig. 5 Segmentation image

2.5. Input/Output Example
Fig. 4 shows an example of a document image to beanalyzed by ASLA. Fig. 5 shows the segmentation image
generated by applying the document image in Fig. 4 as
input for ASLA. Fig. 6 shows an image of an analysis
result file. From Fig. 5, we have confirmed that ASLA
can divide the document image into text, figure, and tableregions, as defined in Table 1.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
264

Kanta Kakinoki, Tetsuro Katayama, Yoshihiro Kita, Hisaaki Yamaba, Kentaro Aburada, Naonobu Okazaki

Table 4 Comparison of precision and recall for Cascade R-CNN and Mask R-CNN with and without rule-based region
redividing, respectively.
Precision
RecallIoU Text Figure Table Text Figure Table
Only Mask R-CNN
0.8 0.768 0.846
0.4
0.651 0.846 0.455
0.9 0.348 0.154
0.2
0.295 0.154 0.182
Mask R-CNN and rule-based region redividing
0.8 0.906 0.929
0.9
0.955
1.0
1.0
0.9 0.877 0.929
0.9
0.924
1.0
1.0
Only Cascade R-CNN
0.8 0.808 0.846
0.4
0.691 0.918 0.818
0.9 0.337 0.317
0.2
0.305 0.308 0.182
Cascade R-CNN and rule-based region redividing 0.8 0.945 0.942
0.9
0.955
1.0
1.0
0.9 0.936 0.942
0.9
0.955
1.0
1.0

3.2. Evaluation of Region Segmentation
Performance(97%). We also have compared the time required for label
generation by using ASLA and manually. As a result, wehave confirmed that the time required for label generation
per document image could be reduced by about 8 minutes(99%). Furthermore, we have evaluated the performanceof region segmentation. As a result, we have confirmedthat the rule-based region redividing achieves highprecision and recall even with a high IoU threshold value.
The future issues are as follows.

In addition to ASLA, there are other studies that divide
region on document images [6], [7]. These studies use
Mask R-CNN [8] as an object detector for region
segmentation. Here, we evaluate the performance ofregion segmentation for each of the Cascade R-CNN used
in ASLA and Mask R-CNN. The dataset used for
performance evaluation consists of 8 document images.
These document images consist of 4 single column
document images and 4 two columns document images.
The evaluation is based on the calculation of IoU for theregions and class names defined in Table 1 to calculate
the precision and recall. Furthermore, to evaluate theusefulness of rule-based region redividing, we calculate
the precision and recall with and without rule-based
region redividing.
Table 4 shows the calculated precision and recall.
From Table 4, we have confirmed that the combinationof Cascade R-CNN and rule-based region redividing used
in Region Segmentation Part of ASLA has the highest
values for both precision and recall. We have confirmedthat the usefulness of ASLA for region segmentation. In
addition, the rule-based region redividing also has
achieved the highest precision and recall when the IoU
threshold is set to a high value of 0.9. We have confirmedthat the usefulness of rule-based region redividing.

•••Support for document images with non-whitebackgrounds.
Automatic generation of sentences or nouns that
represent the contents of figures or tables.
Handling of text regions that do not make sense as
Japanese.

References
1.

2.

3.

4. Conclusion
4.

This paper has proposed a prototype of ASLA that is a
deep learning tool for region segmentation and label
generation, in order to reduce the time required to divide
document images and generate labels. The proposed
ASLA performs region segmentation and class
identification on the input document image, generates
labels each region, and outputs the segmentation image
and the analysis result file.
After applying the document images to ASLA, we haveconfirmed that ASLA works correctly. Furthermore, wehave compared the time required for region segmentation
with the use of ASLA and manually. As a result, we haveconfirmed that the time required for region segmentation
per document image could be reduced by about 3 minutes

5.

6.

7.

8.

Hirohito Shibata, “Paper vs. Electronic Media: Work
Efficiency and Environmental Impact” NIP & Digital
Fabrication Conference, Vol.27, pp.7-10, 2011.
Zhaowei Cai and Nuno Vasconcelos, “Cascade R-CNN:
Delving into high quality object detection,” Proc. of 2018
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), pp. 6154-6162, 2018.
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, andFuru Wei, ”LayoutLMv3: Pre training for Document AI
with Unified Text and Image Masking”, MM '22:
Proceedings of the 30th ACM International Conference on
Multimedia, pp. 4083-4091, 2022.
“NagoyaRepository”,
https://nagoya.repo.nii.ac.jp/(Accessed 2023-12-14)“The 74th Joint Conference of Electrical, Electronics andInformation
Engineersin
Kyushu”,
https://sites.google.com/jceee-kyushu.jp/2021/ (Accessed2023-12-14)
Canhui Xu, Cao Shi, Hengyue Bi, Chuanqi Liu, Yongfeng
Yuan, Haoyan Guo, and Yinong Chen: "A page object
detection method based on Mask R-CNN", IEEE Access,Vol.9, pp.143448-143456, 2017.
Sanket Biswas, Pau Riba, Josep Llados, Umapada Pal,
“Beyond document object detection: instance-levelsegmentation of complex layouts”, IJDAR 24, pp.269-281,
2021.
Kaiming He, Georgia Gkioxari, Piotr Dollar, and Ross B.
Girshick: "Mask R-CNN", International Conference on
Computer Vision(ICCV), pp.2961-2969, 2017.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
265

Powered by TCPDF (www.tcpdf.org)

Proposal of ASLA Which

Dr. Naonobu Okazaki
He received his B.S, M.S., and Ph.D. degrees
in electrical and communication engineeringfrom Tohoku University, Japan, in 1986,
1988 and 1992, respectively. He joined theInformation Technology Research andDevelopment Center, Mitsubishi Electric
Corporation in 1991. He is currently a Professor with theFaculty of Engineering, University of Miyazaki since 2002.
His research interests include mobile network and network
security. He is a member of IPSJ, IEICE and IEEE.

Authors Introduction
Mr. Kanta Kakinoki
He received the Bachelor's degree inengineering(computer science andsystems engineering) from the Universityof Miyazaki, Japan in 2023. He is currently
a Master's student in Graduate School ofEngineering at the University of Miyazaki,
Japan. His research interests include
software development support method
and imageprocessing.
Dr. Tetsuro Katayama
He received a Ph.D. degree in engineeringfrom Kyushu University, Fukuoka, Japan,
in 1996. From 1996 to 2000, he has been a
Research Associate at the Graduate Schoolof Information Science, Nara Institute of
Science and Technology, Japan. Since 2000
he has been an Associate Professor at theFaculty of Engineering, Miyazaki University, Japan. He iscurrently a Professor with the Faculty of Engineering,
University of Miyazaki, Japan. His research interests
include software testing and quality. He is a member of the
IPSJ, IEICE, and JSSST.
Yoshihiro Kita
He received a Ph.D. degree in systems
engineering from the University ofMiyazaki, Japan, in 2011. He is currently
an Associate Professor with the Faculty ofInformation Systems, University ofNagasaki, Japan. His research interests
include software testing and biometrics authentication.

1. Dr. Hisaaki Yamaba
He received the B.S. and M.S. degrees inchemical engineering from the Tokyo
Institute of Technology, Japan, in 1988 and1990, respectively, and the Ph D. degree insystems engineering from the University ofMiyazaki, Japan in 2011. He is currently an
Assistant Professor with the Faculty ofEngineering, University of Miyazaki, Japan. His researchinterests include network security and user authentication.
He is a member of SICE and SCEJ.
Kentaro Aburada
He received the B.S., M.S, and Ph.D. degrees
in computer science and system engineeringfrom the University of Miyazaki, Japan, in
2003, 2005, and 2009, respectively. He iscurrently an Associate Professor with theFaculty of Engineering, University ofMiyazaki, Japan. His research interests include computer
networks and security. He is a member of IPSJ and IEICE.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
266

A Design of a Modular Mobile Robot for Rescue Operations
Baris Celiker

Institute of Information Technology and Intelligent Systems, Kazan Federal University, Kazan, Russia
Shifa Sulaiman

Laboratory of Intelligent Robotics Systems (LIRS), Institute of Information Technology and Intelligent Systems, Kazan
Federal University, Kazan, Russia
Tatyana Tsoy

Laboratory of Intelligent Robotics Systems (LIRS), Institute of Information Technology and Intelligent Systems, Kazan
Federal University, Kazan, Russia
Email: Bariss.be@gmail.com, shifa_1910@it.kfu.ru, tt@it.kfu.ru

Abstract
Modular robotics is one of the subfields of mobile robotics, which is emerging as a new trend in various sectors.
Modular mobile robots can be reconfigured to perform a wide variety of tasks. In this paper, applications of modular
mobile robots in various sectors such as industry, space, surgery, rescue and entertainment tasks are discussed. Basedon the study, an improved design of a modular mobile robot for navigating through different terrains during a rescue
operation is presented. Simulation study of the robot is included to demonstrate a motion capability of the modular
mobile robot.
Keywords: Modular robots, Applications of modular robots, Design, Simulation study

mobile robots to invigilate the conditions of patients andlogistics tasks inside hospitals [14]. Modular robots arealso used in space explorations and rescue operations.
Ground and aerial modular robots are nowadays
employed in places of natural calamities such as earth
quakes [15], landslides and [16] and floods [17]. Modular
robots can reach any location due to its shape shifting,flexible design and motion capabilities.

1. Introduction
Robots were known to assist people in a wide variety oftasks such as industrial [1], [2] household [3], rescue [4],[5], medical [6], [7] space [8], defense [9], andentertainment [10] fields. The emergence of artificial
intelligence elevated a performance of robots [11]. Most
industrial operations are currently performed by robotic
systems due to its better performance capabilities
compared to human operators. Robots are proved to be
more precise and accurate in carrying out a task comparedto human operators. Employers are forced to build
unmanned workplaces due to a variety of factors such asincreasing wages, absence of skilled employees,
dangerous work conditions, etc. Hence, most industries
are replacing human operators with multi-tasking robotic
systems. However, some of the challenging tasks insideindustrial and service sectors are nowadays carried out ina collaboration mode. Such robots are known as
collaborative robots [12].

2. Applications of mobile and modular robot
Mobile manipulation combines the advantages ofmanipulation and navigation. Basiri et al. [18] developed
a mobile robot with a UR5e [19] arm for performing
activities in a construction field. The four-wheeledmobile platform was 78 cm long and 55 cm tall. The
platform can support a payload of 25 kg. Bricks forconstruction purposes were identified and moved using avision-based technique. Lio [20] was a service mobile
robot developed as a care assistant to patients admitted ina hospital. It consisted of a 6 degrees-of-freedom (dof)arm and a mobile base with 4 wheels.

Medical, service and space sectors have started to
explore the capabilities of mobile robots [13]. The major
reason is the multifunctional capabilities of mobile robots
especially modular mobile robots. These kinds of robots
can be controlled remotely to use in places orenvironments which are not accessible for a humanoperator to work. Service sectors are employing modular

Mobile manipulators can be used in cooperative andcollaborative tasks to compensate a growing demand fora workforce in industrial sectors. Robots in the industrial
sector must follow safety regulations and have been
proved to save a significant amount of time and money.
Collaborative robots were developed for carrying out©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
267

Baris Celiker, Shifa Sulaiman, Tatyana Tsoyactivities combinedly by human operators and robots in ashared configuration space [21]. A team of mobile
manipulators used for carrying payload was presented
in [22]. Each robot consisted of a 2 dof mobile base anda 2 dof planar arm for transferring an object from one
place to another. Another set of mobile manipulators used
for a collaborative 3D printing process was illustrated
in [23]. Experimental validation proved that using
collaborative robots increased a performance of acollaborative task.

unit, which was used for moving the robot in uneventerrains. Design of a homogenous modular robot
consisted of revolute and prismatic joints wasdemonstrated in [32]. It consisted of a rotary plate
docking mechanism that allows the robot to change
configurations during a task.
Robots are used for carrying out surgeries andoperations in the medical field. Versatile modular roboticarms are used in robotic surgeries, which can beassembled and dissembled based on task requirements.
Robotic capsules are used for carrying out an endoscopy
to detect internal diseases. Cameras and biochemicalsensors used in these devices can detect internal bodyissues. Numerous distinct terrestrial modular robotic
systems are employed for carrying out space related tasks.
However, due to extreme terrestrial conditions, only few
robots have been in active employment currently.
However, modular robots are preferred for space related
tasks since a reconfigurable robot can carry out multiple
tasks [33], [34]. Mobile modular robots have been also
used in military operations to reach complex terrains [35].

Modular robots can perform a variety of tasks and adapt
to the environment quickly. Connect-R [24] was amodular robot developed for implementing in anindustrial environment. Each module has the ability torotate around other modules, link to other modules, andextend based on the available space. A reconfigurable
modular robot named Deformable Modular Robot(Datom) was presented in [25]. Each module of the robot
was connected to other modules using a Face-centeredcubic lattice pattern. The robot has a capability toseparate from neighbouring modules during a task.
FreeSN [26] was a modular reconfigurable robot
consisted of strut and node modules. A node module wasconstructed using a low carbon steel spherical shell and amagnetic connector was used as a joining mechanism.
SMORES-EP [27] was a modular robot used inconstruction sector. It can self-assemble to generate 3
different topologies based on a task. A docking control
strategy was used for a successful docking of the robot.
A self-reconfigurable mobile robot with 2 dof with adocking mechanism was demonstrated in [28]. It
consisted of a hybrid mobile base with wheels and tracks.
An image processing technique was used for locating adocking station for the charging of the modular robot.

3. Advantages of modular robots
Generally, the cost of a robot manufactured is directly
proportional to the size of the robot due to material
requirements. Due to the high cost of industrial robots,
most of the industries are not able to employ such robots
for execution of tasks. However, modular robots arecompact and cost-effective in terms of their size andmaterials.
Capabilities of mobile robots available in current
market are not adequate to carry out complex tasks. In
comparison to mobile robots, modular robots are more
effective and successful when dealing with emergency
situations. Modular robots are employed for rescue
operations and in narrow pathways, where normal sizedmobile robots cannot navigate. Another major capabilityof a modular robot to transform the shapes and patternsof individual modules.

Modular robots are also used in household andentertainment sectors. Roombots [29] were a group of
modular robots designed to employ both as a payload
carrier in houses and also act as toys for children.
Roombots can act as adaptive and reconfigurable mobile
robots and used for supporting furniture during relocation.
The reconfigurable modular robot consisted of 12
modules with 36 dof. It can autonomously change shape,
grasp, and move furniture. KAIRO-II, a modular robot
developed for rescue operations was given in [30]. It
consisted of 6 drive modules interconnected using 5 joint
modules. KAIRO -II can ascend short stairs and steps
shorter than 55 cm. Modular robots are more efficient incarrying out multiple tasks compared to mobile robots.
Nowadays modular robots are used for navigating
through different terrains using different modes of basemotions. Unmanned ground and aerial modular mobile
robots are used in rescue applications. These mobile
robots can reach a specified location and invigilate theconditions of the environment using sensors attached tothem. A design of a modular robot capable of navigating
through uneven terrains was proposed in [31]. A hybrid
wheel base for the robot was developed with both wheels
and tracks. The robot consisted of a vertical translation

One of the most important requirements foraccomplishing a logistics-related operation is thecapability to provide required speed of navigation. The
ordered item is anticipated to be delivered as soon as
possible. Traditional mobile robots cannot traverse
through complex terrains due to mobile base mechanism
limitations. However, a modular robot can transform its
shape and pattern to control speeds during navigations.
One of the major issues faced by mobile robots is thelimited space for navigation. Modular robots are proved
to be traversable through narrow and curved pathways
without any difficulty. This is mostly due to their shape
shifting capabilities and ability to adapt to different©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
268

Design of a Modular Mobile Robot

environments. Hence, modular robots can be employedremotely to reach disaster zones for monitoring situations.

rectifier and a filter were equipped inside the robot tomanipulate incoming and outgoing currents. Magnetic
resonance technique can also be used for charging therobot.

4. A design of a mobile robot
A design process of a modular robot is very complex
since modular robots must be able to shift shapes basedon requirements. Specifications of a modular robot playa crucial role in completing a task successfully. A
traditional modular robot has 2 or more modules basedon the requirements. Multiple modules can be
manipulated for shifting shapes and configuration of therobot. In some circumstances, each module of a modular
robot can be operated as a single robot, with noconnectivity to other modules.

Wi Fi Coils
Fig. 2 WIFI equipped on modular robot
5. Simulation study

In this paper, a design of a modular mobile robot, which
can be used for search and rescue operations waspresented. The specific design (Fig. 1) allowed the robot
to enter areas inaccessible to human rescue operators. It
can carry necessary equipment, medicines, and has
capability to convey information to communicate with
online remote operators. A modular robot with two
modules each with 4 wheels was designed in Tinkercad
software [36], which is significantly more simple inmodelling and operating than popular Gazebo [37], [38]or Webots [39], [40] robotic simulators. The design ofeach module consisted of a mobile robot with 4 wheels.
One module with 4 wheels can be moved above the other.
An 8-wheeled single-decker vehicle can be created bylining up two distinct vehicles in a row with a rail system
in between the wheels. Self-rotating pallets were used forrotating one module and placing it on top of anothermodule. The two modules can also be used separately
depending on the task requirements. In such cases, GPStechnology is used to detect positions of each module andreach a location for docking.

A simulation study was conducted for analysing motion
capabilities of the modular robot. A task was assigned to
the robot to reach a goal location from an initial location
in a stacked configuration. The initial configuration of themodular robot is shown in Fig. 1. The rear module of themodular robot was sliding on to the top of front moduleusing the wheel rotations as shown in Fig. 3. The final
configuration of the robot when rear module reached the
top of front module is shown in Fig. 4. The rear module
can revert back to its initial pose using the same motion
shown in the Fig. 4. The designed modular robot can betraversed using 8 wheels or 4 wheels by keeping rear
module on the top based on the space constraints.

Selfrotating
pallets

Front
moduleFig. 3 Intermediate position of rear and front modules

Rear
moduleFig. 1 Modular mobile robot with 2 modules
Each module of the robot consisted of a hydrogen fuelbattery, a Raspberry PI 4 as processor and a WIFI module(Fig, 2). The batteries presented in the robot were charged
using a coil. Raspberry PI 4 was used to control an overall
process of the modular mobile robot. WIFI module wasused for receiving and sending information. The robot
was charged using a wireless charging method also. A

Fig. 4 Stacked configuration of the modular robot©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
269

Baris Celiker, Shifa Sulaiman, Tatyana Tsoy

8.

The modular robot was able to reach the goal location
in a stacked configuration. The motion of the robot wassmooth on a flat terrain.

9.

6. Conclusion
10.

In this paper, applications and advantages of available
designs of modular robots in various sectors werereviewed. Based on the study, a design of a modular robot
with two modules was introduced. The modules cantraverse individually or in combined pattern based on thespace constraints and assigned tasks. A simulation study
was carried out to analyze a motion capability of thedesigned modular robot. The nature of motion obtained
was smooth and able to complete the task within limited
time period. The motion capabilities of the designed
modular robot in complex terrains and 3D spaces will beanalyzed in future works. The energy capacity of themodular robot will be analyzed during a task fordetermining the operation runtime of the robot.

11.

12.

13.

Acknowledgements
14.

This work was supported by the Russian Science
Foundation (RSF), project ID 24-29-00564.
References
15.
1.

2.

3.

4.

5.

6.

7.

M. Mstafin, E. Chebotareva, E. Magid, H. Li and
E. Martinez-Garcia, Features of Interaction Between a
Human and a Gestures-Controlled Collaborative Robot inan Assembly Task: Pilot Experiments, International
Conference on Artificial Life and Robotics (ICAROB
2023), 2023, pp. 158-162.\
R. Sultanov, S. Sulaiman, H. Li, R. Meshcheryakov and E.
Magid, A Review on Collaborative Robots in Industrial andService Sectors, in 2022 International Siberian Conference
on Control and Communications (SIBCON), IEEE, 2022,
pp. 1-7.
J. Zhong, C. Ling, A. Cangelosi, A. Lotfi and X. Liu, On
the gap between domestic robotic applications andcomputational intelligence, Electronics, vol. 10(7), 2021,
p. 793.
E. Magid, Simulation Tools for Urban Search and Rescue
Robotics, International Conference on Artificial Life andRobotics (ICAROB 2023), 2023, pp. 4-11.
A. Dobrokvashina, S. Sulaiman, T. Gamberov, K.-H. Hsia
and E. Magid, New Features Implementation for Servosila
Engineer Model in Gazebo Simulator for ROS Noetic,in Proceedings of the International Conference on Artificial
Life Science and Technology, vol. 28, 2023, pp. 153-156.
B. Abbyasov, A. Dobrokvashina, R. Lavrenov,
E. Kharisova, T. Tsoy, L. Gavrilova, S. Bulatov, E. Maslak,
N. Schiefermeier-Mach and E. Magid, Ultrasound sensor
modeling in Gazebo simulator for diagnostics of abdomen
pathologies, The 15th Siberian Conference on Control andCommunications (SIBCON 2021), 2021, № 9438910.
E. Magid, A. Zakiev, T. Tsoy, R. Lavrenov and A.
Rizvanov, Automating pandemic mitigation, Advanced
Robotics, vol. 35(9), 2021, pp. 572-589.

16.

17.

18.

19.

20.

21.

22.

Z. Jiang, X. Cao, X. Huang, H. Li and M. Ceccarelli,
Progress and development trend of space intelligent robot
technology, Space: Science & Technology, 2022.
M. Payal, P. Dixit, T. V. M. Sairam and N. Goyal, Robotics,
AI, and the IoT in defense systems, AI and IoT‐BasedIntelligent Automation in Robotics, 2021, pp. 109-128.
K. J. Morris, V. Samonin, J. Baltes, J. Anderson and
M. C. Lau, A robust interactive entertainment robot for
robot magic performances, Applied Intelligence, vol. 49,
2019, pp. 3834-3844.
E. Magid, A. Pashkin, N. Simakov, B. Abbyasov,J. Suthakorn, M. Svinin, M and F. Matsuno, Artificial
intelligence based framework for robotic search and rescue
operations conducted jointly by international teams. Smart
Innovation, Systems and Technologies, vol. 154, 2019,
pp. 15-26.
A. Dobrokvashina, A. Zagirov, E. Chebotareva, K.-H. Hsia,
E. Magid and S. Sulaiman, Human robot interaction incollaborative manufacturing scenarios: prospective cases.
Siberian Conference on Control and Communications,(SIBCON 2022), 2022, pp. 1-6.
F. Rubio, F. Valero, and C. Llopis-Albert, A review ofmobile robots: Concepts, methods, theoretical framework,and applications, International Journal of Advanced
Robotic Systems, vol. 16(2), 2019.
R. Safin, R. Lavrenov, T. Tsoy, E. Magid, M. Svinin,
S. D. Roy and S. K. Saha, Prioritizing tasks within a robotic
transportation system for a smart hospital environment,
Lecture Notes in Computer Science (including subseries
Lecture Notes in Artificial Intelligence and Lecture Notes
in Bioinformatics), vol. 12998, 2021, pp. 182-193.
E. Magid, K. Ozawa, T. Tsubouchi, E. Koyanagi andT. Yoshida, Rescue Robot Navigation: Static Stability
Estimation in Random Step Environment, Lecture Notes In
Computer Science, vol. 5325, 2008, pp. 305-316.
E. Magid, F. Matsuno, J. Suthakorn, M. Svinin, Y. Bai,
T. Tsoy, R. Safin, R. Lavrenov, A. Zakiev, H. Nakanishi,
M. Hatayama and T. Endo, E-ASIA Joint Research
Program: development of an international collaborative
informational system for emergency situations managementof flood and land slide disaster areas, Artificial Life andRobotics, vol. 27(4), 2022, pp. 613-623.
Y. Bai, M. Svinin and E. Magid, Multi-Robot Control for
Adaptive Caging and Tracking of a Flood Area, 59th
Annual Conference of the Society of Instrument andControl Engineers of Japan (SICE), 2020, pp. 1452-1457.
M. Basiri, J. Gonçalves, J. Rosa, A. Vale and P. Lima, An
autonomous mobile manipulator to build outdoor structuresconsisting of heterogeneous brick patterns, SN Applied
Sciences, vol. 3, 2021, pp. 1-14.
R. Sultanov, S. Sulaiman, T. Tsoy and E. Chebotareva,
Virtual Collaborative Cells Modelling for UR3 and UR5
Robots in Gazebo Simulator, International Conference on
Artificial Life and Robotics (ICAROB 2023), 2023,
pp. 150-153.
J. Mišeikis, P. Caroni, P. Duchamp, A. Gasser, R. Marko,
N. Mišeikienė and H. Früh, Lio-a personal robot assistantfor human-robot interaction and care applications, IEEE
Robotics and Automation Letters, vol. 5(4), 2020, pp. 53395346.
R. Galin, A. Shiroky, E. Magid, R. Meshcheryakov and
M. Mamchenko, Effective Functioning of a Mixed
Heterogeneous Team in a Collaborative Robotic System.
Informatics and Automation (SPIIRAS Proceedings),
vol. 20(b), 2022, pp. 1224-1253.
C. P. Tang, R. Bhatt and V. Krovi, Decentralized kinematic
control of payload by a system of mobile manipulators,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
270

Powered by TCPDF (www.tcpdf.org)

Design of a Modular Mobile Robot

23.

24.

25.

26.

27.

28.

29.

30.

31.

32.

33.

34.

35.

in IEEE International Conference on Robotics andAutomation, vol. 3, April 2004, pp. 2462-2467.
X. Zhang, M. Li, J. H. Lim, Y. Weng, Y. W. D. Tay,
H. Pham and Q. C. Pham, Large-scale 3D printing by a teamof mobile robots, Automation in Construction, vol. 95,
2018, pp. 98-106.
M. E. Sayed, J. O. Roberts, K. Donaldson, S. T. Mahon, F.
Iqbal, B. Li, S. F. Aixela, G. Mastorakis, E. T. Jonasson,
M P. Nemitz, S. Bernardini and A. A. Stokes, Modular
robots for enabling operations in unstructured extreme
environments, Advanced Intelligent Systems, vol. 4(5),2022, pp. 2000227.
B. Piranda and J. Bourgeois, Datom: A Deformable
modularrobot
forbuilding
self-reconfigurable
programmable matter, in Distributed Autonomous Robotic
Systems: 15th International Symposium, 2022, pp. 70-81.
Y. Tu, G. Liang and T. L. Lam, FreeSN: A freeform strutnode structured modular self-reconfigurable robot-design
and implementation, International Conference on Robotics
and Automation (ICRA), 2022, pp. 4239-4245.
C. Liu, Q. Lin, H. Kim and M. Yim, SMORES-EP, a
modular robot with parallel self-assembly, Autonomous
Robots, vol. 47(2), 2023, pp. 211-228.
S. S. Sohal, B. Sebastian and P. Ben-Tzvi, Autonomous
Docking of Hybrid-Wheeled Modular Robots With an
Integrated Active Genderless Docking Mechanism, Journalof Mechanisms and Robotics, vol. 14(1), 2022, p. 011010.
S. Hauser, M. Mutlu, P. A. Léziart, H. Khodr,
A. Bernardino and A. J. Ijspeert, Roombots extended:
Challenges in the next generation of self-reconfigurable
modular robots and their application in adaptive andassistive furniture, Robotics and Autonomous Systems,
vol. 127, 2020, p. 103467.
L. Pfotzer, S. Ruehl, G. Heppner, A. Roennau and
R. Dillmann, KAIRO 3: A modular reconfigurable robot forsearch and rescue field missions, IEEE Int. Conf. on
Robotics and Biomimetics, 2014, pp. 205-210.
P. Kumar, W. Saab and P. Ben-Tzvi, Design of a multidirectional hybrid-locomotion modular robot withfeedforward stability control, in International Design
Engineering Technical Conferences and Computers andInformation in Engineering Conference, August 2017,vol. 58189, pp. V05BT08A010.
J. Baca, S. G. M. Hossain, P. Dasgupta, C. A. Nelson and
A. Dutta, Modred: Hardware design and reconfiguration
planning for a high dexterity modular self-reconfigurable
robot for extra-terrestrial exploration, Robotics andAutonomous Systems, vol. 62(7), 2014, pp. 1002-1015.
SpaceArchitect.org, 3D printed Mars habitat (3rd phase ofNASA’scentennial
challenge),
2018,
http://spacearchitect.org/portfolio-item/ 3d-printed-marshabitat-3rd-phase-of-nasas-centennial-challenge/.
M. A. Post and J. Austin, Knowledge-based selfreconfiguration and self-aware demonstration for modular
satellite assembly, in 10th International Workshop on
Satellite Constellations & Formation Flying 2019, April
2019.
J. Whitman, R. Bhirangi, M. Travers and H. Choset,
Modular robot design synthesis with deep reinforcement
learning, in Proc. of the AAAI Conference on Artificial
Intelligence, vol. 34(6), April 2020, pp. 10418-10425.

36.

37.

38.

39.

40.

E. A. Juanda and F. Khairullah, Tinkercad Application
Software to Optimize Teaching and Learning Process inElectronics and Microprocessors Subject, in 6th UPIInternational Conference on TVET 2020, 2021, pp.124-128.
A. Dobrokvashina, R. Lavrenov, E. Magid, Y. Bai and
M. Svinin, How to Create a New Model of a Mobile Robotin ROS/Gazebo Environment: An Extended Tutorial,
International Journal of Mechanical Engineering andRobotics Research, vol. 12(4), 2023, pp. 192-199.
D. Imameev, A. Zakiev, H. Li, E. Martinez-Garcia and
E. Magid, Modelling Autonomous Perpendicular Parking
Procedure for Car-like Robot Avrora Unior in Gazebo
Simulator, 6th International Scientific Conference on
Information, Control, and Communication Technologies(ICCT 2022), 2022, pp. 412-415.
A. Dobrokvashina, R. Lavrenov, E. Magid, Y. Bai,
M. Svinin and R. Meshcheryakov, Servosila Engineer
Crawler Robot Modelling in Webots Simulator,
International Journal of Mechanical Engineering andRobotics Research, vol. 11(6), 2022, pp. 417-421.
A. Dobrokvashina, R. Lavrenov, Y. Bai, M. Svinin and
E. Magid, Sensors modelling for Servosila Engineer
crawler robot in Webots simulator, Moscow Workshop on
Electronic and Networking Technologies (MWENT-2022),
2022, pp. 1-5.

Authors Biographies
Mr. Baris Celiker
In 2021, he graduated from the
Universty of Dokuz Eylul, Turkey.
Since 2022 he has been a master's
student at the Institute of Information
Technology and Intelligent Systems(ITIS), Kazan Federal University(KFU).
Dr. Shifa Sulaiman
In 2013, she received her Master's
degree in Machine Design from
Mahatma Gandhi University, India.
In 2013-2017 worked as an AssistantProfessorinvariousIndianEngineering Institutes. She earned
her PhD from National Institute ofTechnology,
Calicut,
India,
specialized in Humanoid Robotics in 2023. She is currently
working as a Research Associate at the Laboratory ofIntelligent Robotic Systems (LIRS) at ITIS, KFU, Russia.
Ms. Tatyana Tsoy
In 2012 she graduated fromInternational Area Studies Master's
Degree Program at the University ofTsukuba. In 2023 completed a PhD in
Computer science and information
processes at ITIS, KFU, Russia.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
271

Implementation of Bug1 and Bug2 Basic Path-Planning Algorithms fora TurtleBot 3 Robot in ROS Noetic
Ilya Spektor

HSE University, Moscow, Russian Federation
Aidar Zagirov

Intelligent Robotics Department, Kazan Federal University, 420008, Kazan, Russian Federation
Ramil Safin

Intelligent Robotics Department, Kazan Federal University, 420008, Kazan, Russian Federation
Evgeni MagidIntelligent Robotics Department, Kazan Federal University, 420008, Kazan, Russian Federation
HSE University, Moscow, Russian Federation
E-mail: ivspektor@edu.hse.ru, ai.zag@it.kfu.ru, safin.ramil@it.kfu.ru, magid@it.kfu.ru

Abstract
Mobile robots typically operate in a dynamically changing unknown environments. The Bug family algorithms weredeveloped to address path planning challenges for ground vehicles within 2D configuration spaces of unknown
environments. These algorithms utilize local sensory data about obstacles encountered during navigation. This paper
introduces the implementation of Bug1 and Bug2 local path planning algorithms in ROS Noetic. Traditional 2D Bug
algorithms are extended into 3D Gazebo simulation environment. Performance evaluation of Bug1 and Bug2 wereconducted using the TurtleBot3 Burger model in both simple convex and maze environments.
Keywords: BUG Algorithms, Path Planning, Navigation, Mobile Robots, ROS, Gazebo

1. Introduction

a target point until encountering an obstacle. The second
step involves circumnavigating the obstacle until an exit
condition is met.

There are two main path planning models distinguishedby the type of information they utilize. The first model,
known as path planning with complete information or
global navigation, assumes that a robot possesses
comprehensive information about its surroundings. In
contrast, the second model, known as path planning withincomplete information, introduces uncertainty about anenvironment [1]. It is essential to integrate both global
and local path planning strategies to ensure secure robotic
navigation. In this context, a primary objective of local
planners is to adhere to an initial plan provided by global
planners [2]. Thus, an implementation of a safe andreliable local navigation algorithm, tested in conditions
close to real-world scenarios, holds a significant
importance.

This paper outlines challenges and potential solutions
for implementing local path-planning algorithms within
3D environments, utilizing Bug1 and Bug2 as reference
models. It marks an initial step toward an implementationof more complex BUG algorithms [3], [4]. The
algorithms were implemented in ROS and evaluated in
Gazebo simulator using TurtleBot3 Burger [5]. Customdesigned environments were used for testing while as apart of our ongoing work the algorithms are being
validated employing a broad set of environments
generated by an automatic tool for Gazebo simulator [6]and a number of standard warehouses’ models [7].

The Boundary-following and Ultimate Goal (BUG)
family of algorithms is among the most renowned in thefield of local path-planning. The BUG algorithms follow
a simple cycle consisting of two states. In the first state,a robot pursues a directly or intricately calculated path to

The Bug1 and Bug2 algorithms are a foundation for theentire BUG family, encompassing algorithms such as
Class 1, 2 and 3 [8], Alg1 and Alg2 [9], [10], VisBug21
and VisBug22 [11], Wedgebug [12], Rev1 and Rev2 [13],2. Algorithms Overview©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
272

Ilya Spektor, Aidar Zagirov, Ramil Safin, Evgeni Magid

Distbug [14], TangentBug [15], CautiousBug [16] andothers. Both algorithms operate by transitioning between
two states: move-to-target and boundary-following (Fig.
1). These algorithms require a robot to be equipped witha touch sensor, rendering them useful for compact
machinery. Next, we offer a brief overview of Bug1 andBug2 algorithms. For a more in-depth analysis refer to theoriginal work of Lumelsky and Stepanov [1].

content of registers R2 and R3, determine a shorter way
along the boundary to Li, use it to get to Li. Increment
counter i and go back to step 1.
2.2. Bug2 Algorithm
Steps of Bug2 are executed at any point of a continuous
path as follows:
1) From point Lj-1, move along a straight line (Start,
Target) until one of the following events occurs:(a) If Target is reached: stop the algorithm.
(b) An obstacle is encountered: define hit point Hj
and proceed to step 2.
2) Follow an obstacle boundary:(a) If Target is reached: stop the algorithm.
(b) If line (Start, Target) is met at point Q suchthat distance d(Q) < d (Hj), and line (Q,Target) does not cross the current obstacle atpoint Q. Define leave point Lj = Q. Increment
counter j and go back to step 1.
(c) If the robot returns to Hj and, hence, completes
a closed curve (the obstacle boundary) without
defining next hit point Hj+1, it means that
Target cannot be reached, stop the algorithm.

Fig. 1. Trajectory of Bug2 algorithm. Red lines depict
the move-to-target state, while blue lines illustrate theboundary-following state.
We use the following notations for both algorithms:

3. Implementation• Target – a destination point that the robot should
eventually reach.
• Li (leave point) – a point defined when departing froman i-th obstacle to resume moving toward Target.
• Hi (hit point) – a point defined when an i-th obstacleis encountered.

Several challenges were encountered during Bug1 andBug2 algorithms’ implementation in ROS/Gazebo. The
first problem was a simulation of a touch sensor for the
TurtleBot3 Burger robot. It is equipped with a 180
degrees field of view range sensor for safety and integrity
reasons [5]. To simulate a touch sensor, we restricted thesensor’s vision range to ρv = 1.5Rb, where Rb is a radiusof a robot’s body circumcircle. Touch sensors usuallyprovide boolean values: true means that a robot is
touching an obstacle, false otherwise. In our
implementation we divided a field of view into fivequadrants (of 36 degrees each) and register collision
events within each quadrant independently. Anotherproblem is that Bug1 and Bug2 do not limit robot’s
forward and angular velocities [1]. The moststraightforward solution is to set a constant velocity.
Potential further enhancements may incorporate a PID
controller into the algorithm’s pipeline.

2.1. Bug1 Algorithm
For the Bug1 algorithm we introduce variable Qm that
stores an encountered point with a minimal distance
between an obstacle boundary and Target. Additionally,
we will introduce three registers: R1 to store coordinatesof the point Qm, R2 for a length of the obstacle boundaryfrom Hi to Qm, and R3 for a length of the obstacle
boundary from Qm to Hi. The following steps are
executed at every point of a continuous path:
1) From point Li-1 move toward Target along a straightline until one of the following events occurs:(a) Target is reached: stop the algorithm.
(b) An obstacle is encountered: define hit point Hi
and proceed to step 2.
2) Follow the obstacle boundary using a local direction(local information about the obstacle’s boundary):(a) If Target is reached: stop the algorithm.
(b) Otherwise, traverse the boundary and return to
Hi. Define new leave point Li = Qm andproceed to step 3.
3) Apply a test for target reachability. If Target is notreachable, stop the algorithm. Otherwise, using

Our implementation and validation were conductedusing Ubuntu OS, employing robust capabilities of Robot
Operating System (ROS) Noetic Ninjemys and theGazebo simulator [17], [18]. The system configuration
for the rigorous testing and validation is outlined in Table
1. For testing purposes, we selected the TurtleBot3
Burger (see Fig. 2) as the robot model [19]. The
algorithms were initially implemented in Python,leveraging its popularity and the ease of prototyping anddevelopment. However, recognizing the importance ofcomputational efficiency, we plan to reimplement thealgorithms in C++.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
273

Implementation of Bug1 and Bug2

Table 1. System configuration
Component
Specification
OS
Ubuntu 20.04 LTS
ROS
Noetic Ninjemys
Gazebo11.11.0
Processor
Intel Core i5-11300H @ 3.10GHz
RAM
24 GB

test scenario. When a point was deemed reachable, therobot successfully navigated its path to the target,
affirming the efficacy of the Bug1 and Bug2 algorithms.
The outcomes of the Bug1 and Bug2 algorithms werevisually demonstrated through simulations in Gazebo.
Fig. 3 illustrates a resulting path in an environment withconvex obstacles, achieved by the Bug1 algorithm. Fig. 4
presents a resulting path in the same environment by
Bug2. Notably, a difference was significant, with Bug2
path displaying a more straightforward trajectory, which
is observed in a substantial portion of the conducted tests.

Fig. 2. TurtleBot3 Burger in the Gazebo environment.
There are three ROS nodes created to simulate a cycleof switching the states:

Fig. 3. Path of Bug1 in the environment with convex
obstacles (white dashed line).

1) Main: operates continuously, responsible forcalculating an optimal time to switch between thestates based on predefined conditions.
2) Go-to-Target: orients and moves the robot toward
the target.
3) Follow-Wall: facilitates the circumnavigation of
obstacles. In the case of Bug1, there are two
instances of this node – one for clockwise rotations
and another for counterclockwise rotations.
Fig. 4. Path of Bug2 in the environment with convex
obstacles (white line).

4. Validation
To ensure the robustness and effectiveness of the Bug1
and Bug2 algorithm implementations, a comprehensive
validation process was undertaken in Gazebo simulator
using ROS navigation stack [19], [20]. The validationencompassed a diverse set of tests conducted in 3D
simulation world featuring individual complex obstacles
and a maze [6]. To ascertain the correctness of the target
point reachability determination, a testing scenario was
implemented for both Bug1 and Bug2 algorithms. The
validation process involved 30 tests for each algorithm onmaps with convex obstacles. Half of these tests featured
reachable targets, while the remaining half featured
unreachable targets.

5. Conclusion
In this work we presented the implementation of theBug1 and Bug2 algorithms, delving into the challenges
encountered while integrating local navigation
algorithms into realistic ROS/Gazebo simulations. This
article lays a ground for future implementations of more
complex BUG family algorithms.
Acknowledgements
This paper was supported by the Kazan Federal
University Strategic Academic Leadership Program(“PRIORITY-2030”).

Given the Bug1's requirement to fully circumnavigate
obstacles, testing in maze environments was timeconsuming. Thus, only 10 tests for each algorithm wereconducted on the maze map, with an equal distribution ofreachable and unreachable target scenarios. All target
points were accurately identified for reachability in every

References
1.

V. J. Lumelsky and A. A. Stepanov, Path-planning
strategies for a point mobile automaton moving amidst©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
274

Powered by TCPDF (www.tcpdf.org)

Ilya Spektor, Aidar Zagirov, Ramil Safin, Evgeni Magid

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

unknown obstacles of arbitrary shape, In Autonomous
robot vehicles, Vol. 2, 1990, pp. 403-430.
A. Apurin, B. Abbyasov, E. A. Martínez-García and
E. Magid, Comparison of ROS Local Planners for a
Holonomic Robot in Gazebo Simulator, In International
Conference on Interactive Collaborative Robotics, Cham:
Springer Nature Switzerland, 2023, pp. 116-126.
J. Ng and T. Bräunl, A Practical Comparison of Robot Path
Planning Algorithms given only Local Information, Centre
for Intelligent Information Processing Systems School ofElectrical and Electronic Engineering The University ofWestern Australia, 2005.
J. Ng and T. Bräunl, Performance comparison of bug
navigation algorithms, Journal of Intelligent and Robotic
Systems, Vol. 50(1), 2007, pp. 73-84.
R. Safin, T. Tsoy, R. Lavrenov, I. Afanasyev and E. Magid,
Modern Methods of Map Construction Using OpticalSensors Fusion, International Conference on Artificial Lifeand Robotics (ICAROB 2023), 2023, pp. 166-169.
A. Iskhakova, B. Abbyasov, T. Tsoy, T. Mironchuk,
M. Svinin and E. Magid, LIRS-Mazegen: An Easy-to-Use
Blender Extension for Modeling Maze-Like Environments
for Gazebo Simulator, Frontiers in Robotics andElectromechanics, Vol. 329, 2023, pp. 147-161.
A. Khazetdinov, A. Zakiev, T. Tsoy, R. Lavrenov and K.H. Hsia, Standard-complaint Gazebo warehouse modellingand validation, Proceedings of 13th International
Conference on Developments in eSystems Engineering(DeSE), 2020, p. 218-221.
H. Noborio, Several path-planning algorithms of a mobile
robot for an uncertain workspace and their evaluation,
Proceedings of the IEEE International Workshop on
Intelligent Motion Control, Istanbul, Turkey, 1990,
pp.289-294.
A. Sankaranarayanan and M. Vidyasagar, A New Path
Planning Algorithm For Moving A Point Object Amidst
Unknown Obstacles In A Plane, IEEE Conference on
Robotics and Automation, 1990, pp. 1930-1936.
A. Sankaranarayanan and M. Vidyasagar, Path Planning
For Moving A Point Object Amidst Unknown Obstacles In
A Plane: A New Algorithm And A General Theory For
Algorithm Development, Proceedings of the 29th
Conference on Decision and Control, 1991, pp. 1111-1119.
V. Lumelsky and T. Skewis, Incorporating range sensing
in the robot navigation function, IEEE Transactions on
Systems, Man, and Cybernetics, Vol. 20(5), 1990,
pp. 1058-1069.
S. L. Laubach and J. W. Burdick, An autonomous sensorbased path-planner for planetary microrovers, In
Proceedings 1999 IEEE International Conference on
Robotics and Automation (Cat. No. 99CH36288C), Vol. 1,
1999, pp. 347-354.
Y. Horiuchi and H. Noborio, 2001, May. Evaluation ofpath length made in sensor-based path-planning with thealternative following, In Proceedings 2001 ICRA. IEEE
International Conference on Robotics and Automation(Cat. No. 01CH37164), Vol. 2, 2001, pp. 1728-1735.
I. Kamon and E. Rivlin, Sensory-based motion planningwith global proofs, IEEE Transactions on Robotics andAutomation, Vol. 13(6), 1997, pp. 814-822.
I. Kamon, E. Rimon and E. Rivlin, TangentBug: A RangeSensor-Based Navigation Algorithm. The International
Journal of Robotics Research, Vol. 17(9), 1998, pp. 934953.
E. Magid and E. Rivlin, CAUTIOUSBUG: A Competitive
Algorithm for Sensory-Based Robot Navigation.

17.

18.

19.

20.

Proceedings of 2004 IEEE/RSJ International Conferenceon Intelligent Robots and Systems, 2004, pp. 2757-2762.
R. Sultanov, S. Sulaiman, T. Tsoy and E. Chebotareva,
Virtual Collaborative Cells Modelling for UR3 and UR5
Robots in Gazebo Simulator, International Conference on
Artificial Life and Robotics (ICAROB 2023), Vol. 28,
2023, pp. 149-152.
A. Apurin, A. Dobrokvashina, B. Abbyasov, T. Tsoy,
E. A. Martinez-Garcia and E. Magid, LIRS-ArtBul:
Design, modelling and construction of an omnidirectional
chassis for a modular multipurpose robotic platform,
Lecture Notes in Computer Science, Vol. 13719, 2022, p.
70-80.
F. U. Pereira, P. M. de A. Brasil, Marco Antonio De Souza
Leite Cuadros and Anselmo Rafael Cukla, Analysis ofLocal Trajectory Planners for Mobile Robot with Robot
Operating System, IEEE Latin America Transactions,Vol. 20(1), 2022, pp. 92-99.
A. Dobrokvashina, R. Lavrenov, T. Tsoy, E. A. MartinezGarcia, Y. Bai, Navigation stack for the crawler robot
Servosila Engineer, Proceedings of the IEEE 16th
Conference on Industrial Electronics and Applications(ICIEA), 2021, p. 1907-1912.

Authors Introduction
Mr. Spektor Ilya
Spektor Ilya is a fourth-year bachelor
student in Applied Mathematics’ atTikhonov
MoscowInstituteofElectronics and Mathematics, HSE
University, Russian Federation.

Mr. Aidar Zagirov
Aidar Zagirov is a master student in
intelligent robotics at the Institute ofInformation Technology and Intelligent
Systems, Kazan Federal University(KFU), Russia. He works as a research
assistant at the Laboratory of Intelligent
Robotic Systems at KFU.
Mr. Ramil Safin
Ramil Safin is a Ph.D. student and a
research assistant at the Laboratory ofIntelligent Robotic Systems, Institute ofInformation Technology and Intelligent
Systems, Kazan Federal University,
Russia. He teaches «Fundamentals oftechnical vision», «Robotic systems
sensors».
Professor Evgeni Magid
A Professor, a Head of Intelligent
Robotics Department and a Head ofLaboratory of Intelligent Robotic
Systems (LIRS) at Kazan Federal
University, Russia. Professor at HSE
University, Russia. Senior IEEE
member. He earned his Ph.D. degreefrom University of Tsukuba, Japan. He authors over 200
publications.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
275

Implementation of Alg1 and Alg2 Path Planning Algorithms for Mobile Robots Using ROS
NoeticAnastasia Yankova

HSE University, Moscow, Russian Federation
Timur Gamberov

Intelligent Robotics Department, Kazan Federal University, 420008, Kazan, Russian Federation
Tatyana TsoyIntelligent Robotics Department, Kazan Federal University, 420008, Kazan, Russian Federation
E-mail: aayankova@edu.hse.ru, TRGamberov@stud.kpfu.ru, tt@it.kfu.ru

Abstract
Two standard approaches for a robot path planning include a global and a local navigation. The later does not require
to store an environment model in a robot memory. This paper presents implementations of two local navigation
algorithms, Alg1 and Alg2, with a robot having no prior information about an environment and obstacles. It calculatesa path in a real time, continuously changing its states depending on correspondent conditions. The algorithms were
implemented for an existing differential drive robot Turtlebot3 Burger using Robot Operation System (ROS). Virtual
experiments were performed in the Gazebo simulator employing a simple 3D environment with only convex obstacles
and a small 3D maze.
Keywords: path planning, mobile robots, local navigation, Alg1, Alg2, Gazebo, ROS Noetic

path [21]. which is called the local navigation. In the local
planning approach a robot is being placed at a starting
position and must reach a target or report if it cannot bereached while no other information is known to the robot
in advance [22]. In this case, the robot uses local sensory
data [20] to detect obstacles within its radius of vision[23], which allows the robot to encounter an obstacle only
when it hits the obstacle in most algorithms [24].

1. Introduction
Mobile robots and autonomous navigation are
gradually integrated in various aspects of humanactivities, from occasional operations in dangerous
scenarios to daily social interactions. The former include
firefighting services [1], [2], urban search and rescue
operations [3], [4], and special military operations [5],while the later improve processes in education [6], [7],manufacturing [8], [9], medicine [10], [11], agriculture[12], [13], rehabilitation [14] and service tasks [15], [16].
These tasks require advanced sensory-based autonomous
navigation in various environments [17], [18].
Autonomous navigation allows a robot to decide on its
motion and actions, based on onboard sensory data about
its environmental and current location [19].

Boundary-following and Ultimate Goal (BUG) family
algorithms were designed to solve the local navigation
problem without generating a full map of an environment[22]. Following a BUG family algorithm, a robot could
operate in a broad variety of environments [25] and (byan algorithm design) attempts to construct a shortest path
toward its destination [26]. BUG algorithms have twomodes of motion: moving towards a target and following
an obstacle boundary. A robot goes towards the target
until it hits any obstacle. Then it starts to follow theboundary until a straight path towards the target becomes
clear again [27]. A condition that defines if the path isclear differs depending on a particular algorithm.

There are two types of path-planning: a global
navigation and a local navigation [20]. In the global
planning approach, a mobile robot has a well-defined
map of an environment in which the robot can build its
path. In more realistic settings, a robot deals withuncertain maps and relies on its sensors to plan a©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
276

Anastasia Yankova, Timur Gamberov, Tatyana TsoyIn BUG model a robot is considered as a point object[22] in a 2D-map. In this paper a path-planning sensorybased navigation was simulated in a 3D-environment
with a real mobile robot model, which employs Alg1 [28]and Alg2 [29] algorithms. The algorithms were
implemented using robot operating system (ROS) [30]and evaluated using Gazebo simulator [31].

the robot became closer to T than before. The algorithm
operates as follows [30]:0) Initialize iteration i to 0 and Q = d(S, T).
1) Increment i and proceed in the direction of T whilst
continuously updating Q to d(x, T) if Q < d(x, T),
where x is a current position. Q should now represent
the closest to T point where the robot has ever been.
Repeat this until one of the following occurs:• T is reached. Stop.
• An obstacle is hit. Define this point Hi. Go tostep 2.
2) Keeping the obstacle on the right, follow the obstacle
boundary continuously updating Q to d(x, T) if Q <d(x, T). Do this until one of the following occurs:• T is reached. Stop.
• A point y is found such that:- d(y, T) < Q and- The robot can move towards T at y.
Define this point as Li and go to step 1.
• A previously defined point Hj or Lj is
encountered such that j<i. Change the local
direction and return to Hi. When Hi is reached,follow the obstacle boundary keeping the wallon the left. This rule cannot be applied again
until Li is defined.
• The robot returns to Hi. T is unreachable. Stop.

2. Brief description of the algorithms
Alg1 and Alg2 algorithms belong to BUG family. Twostates of a robot under BUG strategy are: 1) go to thetarget; 2) follow the boundary. Most algorithms employ
a line from start point to target, which is called M-line.
The robot switches from state 1 to state 2 when it hits an
obstacle at a point that is defined as a Hit-point (H-point).
The switch from state 1 to state 2 occurs when the robot
decides to abandon the current obstacle at a point that isdefined as a Leave-point (L-point). Both H-point and Lpoint are defined differently by particular algorithms.
2.1. Alg1 algorithm
Alg1 algorithm improves basic Bug2 algorithm [24] ina sense of excluding multiple traces of long segments ofa path. It collects H-points (Hi) and L-points (Lj) of
previous iterations and uses this information to generate
shorter paths by changing a local direction to the opposite.
The algorithm works as follows [32]:3. Implementation details
Ubuntu operating system and ROS Noetic Ninjemys
were used. Gazebo simulator was employed fordebugging and verifying algorithms’ implementation
with Turtlebot3 Burger robot model (Fig. 1) from an open
source software kit [33].

0) Initialize iteration i to 0, define M-line as line
connecting start S and target T points.
1) Increment i and follow M-line toward T until either:• T is reached. Stop.
• An obstacle is hit. Define this point as Hi. Go tostep 2.
2) Keeping the obstacle on the right, follow the obstacle
boundary. Do this until one of the following occurs:• T is reached. Stop.
• A point y is found such that:- it is on M-line and- d(y, T) < d(x, T) for all x ever visited by the
robot along M-line and- The robot can move towards T at y.
Define this point as Li and go to step 1.
• A previously defined point Hj or Lj is
encountered such that j<i. Change the local
direction one and return to Hi. When Hi isreached, follow the obstacle boundary keeping
the wall on the left. This rule cannot be applied
again until Li is defined.
• The robot returns to Hi. T is unreachable. Stop.

Alg1 and Alg2 were implemented in Python3
programming language and arranged as a package with
two files, containing Alg1 and Alg2 respectively. A main
service uses services for going to a point and a wall
following in a clockwise and counter clock-wise order.
The following libraries and modules were used for all
the services:
•
•
•
•
•
•
•

2.2. Alg2 algorithm

rospy – a pure Python client library for ROS;
geometry_msgs module used for generating andsending messages for setting robot’s position;
sensor_msgs module to register laser range
finder’s measurements;
gazebo_msgs module to define and set a robot’s
current state;
tf module for angles’ operations;nav_msgs module for odometry;
std_srv module for ros services.

Alg2 algorithm upgrades Alg1 algorithm byabandoning M-line concept. The leaving condition is that

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
277

Implementation of Alg1 andpopular tools for environment construction for Gazebo
worlds, ranging from semiautonomous generators of
single environments from 2D images [18],[34] toautonomous generators of multiple environments,
including maze-like environments [35] and random step
environment generators [36] using ergonomic graphicaluser interfaces, it was decided to construct twoenvironments manually in order to ensure interestingcases for algorithms’ testing. The first constructedenvironment was bounded by an external non-traversable
black wall with five green towers and contained separate
convex obstacles in a form of nine identical columns. The
second environment was a maze. 25 and 20 test cases
within the convex environment and the maze wereconducted, correspondingly. For each test cases four runs
were conducted. In total, 90,5% of the tests weresuccessful while seven tests in the convex environment
and ten tests in the maze failed due to the problems thatwere stated in Section 3. Additional tests for the target
reachability were performed successfully in both
environment. Table 1 presents the system configuration.

Fig. 1. Turtlebot3 Burger in the Gazebo environment
A main difficulty that was encountered while
implementing the algorithms was a definition of suitable
constants that allow comparing distance measurements.
For example, one of the problems was to define athreshold ε>0 that could be employed to define two
distinct robot positions. This way we define that if a
Euclidean distance Dist between two points pk and pm isless than ε, it means that the two points coincide with eachother:
Dist(pk,pm)≤ ε <=> pk = pm

Table 1. System configuration
Parameter
Characteristics
Operation System details Ubuntu 20.0.4
Memory8.00 GBIntel(R) Core (TM) i7Processor
4510U CPU @ 2.00GHz
2.60 GHz(1)

A value of ε was chosen empirically to fit all types ofenvironments.
Another problem relates to the construction of the robotthat causes stuck because the robot sensor does notconsider the robot’s wheels, which poke out from themobile base. In some cases, the robot does not register an
obstacle when its wheels already have hit a convex corner.
A possible solution is an increase of a threshold δ thathelps defining a H-point as:Dist(pk,pobs)≤ δ => pobs = Hi

Fig. 2 presents a path constructed by Alg1 algorithm inthe convex environment. Due to the environment’s
simplicity, a path of Alg2 algorithm was almost identical.
Fig. 3 and Fig. 4 demonstrate paths with the same S, T
locations that were constructed in the maze environment
by Alg1 and Alg2 respectively. In this particular case
Alg2 outperformed Alg1; it is wrong to state that Alg2
always outperforms Alg1 in every case as a result
depends on the S, T and the environment selection.

(2)

where pobs is a point on a boundary of a currently hit
obstacle, pk is a current position of the robot, Hi is a newly
defined H-point. Yet, this may cause the algorithms’
failure since the robot may miss a H-point and thus skips
a switch of its state into the boundary following mode.
Finally, one more difficulty appeared only for Alg2
algorithm at step 1 when it checks whether the robot
became closer to T than before (Q < d(x, T)). While in amathematical sense this comparison is performed
constantly, in practice a particular small time step Δtbetween checking a new value of x should be selected.
The value of Δt was found empirically so that it
successfully works for both employed types of maps:
separate convex obstacles and mazes.
4. Validation

Fig. 2. A path of Alg1 in the convex environment isshown by white lines. The arrows depict the direction of
motion, the red star marks the target.

To validate the implementation of Alg1 and Alg2
algorithms 45 tests within two different types ofenvironments were conducted. While there exist several©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
278

Anastasia Yankova, Timur Gamberov, Tatyana Tsoy

2.

3.

4.

5.

6.

Fig. 3. A path of Alg1 in the maze

7.

8.

9.

10.

11.

Fig. 4. A path of Alg2 in the maze
5. Conclusions

12.

The paper presented the implementation of Alg1 andAlg2 algorithms in Python programming language for
Turtlebot3 Burger robot model using ROS Noetic. The
conducted tests successfully validated the implemented
algorithms in the simple convex environment and in themaze. A number of difficulties that were encountered
while implementing the algorithms are discussed.

13.

14.

Acknowledgment
15.

This paper has been supported by the Kazan Federal
University Strategic Academic Leadership Program(“PRIORITY-2030”).

16.

References
1.

J. Zhu, W. Li, D. Lin, H. Cheng and G. Zhao, Intelligent
fire monitor for fire robot based on infrared image
feedback control. Fire Technology, vol. 56, no. 5, 2020, pp.
2089–2109.

17.

R. Bogue, The role of robots in firefighting. Industrial
Robot: the international journal of robotics research andapplication, 48(2), 2021, pp.174-178.
E. Magid, K. Ozawa, T. Tsubouchi, E. Koyanagi andT. Yoshida, Rescue Robot Navigation: Static Stability
Estimation in Random Step Environment. Lecture Notes
in Computer Science, vol. 5325, 2008, p. 305-316.
E. Magid, Simulation Tools for Urban Search and Rescue
Robotics. International Conference on Artificial Life andRobotics (ICAROB 2023), 2023, pp. 4-11.
C. Chen, S. Wang, L. Li, S. Ke, C. Wang and X. Bu,Intelligent covert satellite communication for military
robot swarm. IEEE Access, vol. 8, 2019, pp. 5363–5382.
T. Tsoy, L. Sabirova, M. Abramsky and E. Magid,
Establishing effective teaching for robotics: a comparison
study of bachelor students participated in introduction to
robotics course. In Int. Conf. on Artificial ALife andRobotics, 2018, pp. 212–215.
E. Chebotareva and M. Mustafin, Android Based
Educational Mobile Robot Design and Pilot Evaluations.
International Conference on Artificial Life and Robotics(ICAROB 2023), 2023, pp. 146-149.
A. Dobrokvashina, S. Sulaiman, A. Zagirov,
E. Chebotareva, K.-H. Hsia and E. Magid, Human robot
interaction in collaborative manufacturing scenarios:
Prospective cases. Siberian Conference on Control andCommunications, (SIBCON 2022), November 2022, p. 16.
R. Galin and R. Meshcheryakov, Automation and robotics
in the context of Industry 4.0: the shift to collaborative
robots. In IOP Conference Series: Materials Science andEngineering, vol. 537, no. 3, May 2019, p. 032073.
E. Magid, A. Zakiev, T. Tsoy, R. Lavrenov and A.
Rizvanov, Automating pandemic mitigation. Advanced
Robotics, vol. 35, no. 9, 2021, pp. 572–589.
R. Safin, R. Lavrenov, K.-H. Hsia, E. Maslak,
N. Schiefermeier-Mach and E. Magid, Modelling a
TurtleBot3 Based Delivery System for a Smart Hospital in
Gazebo. The 15th Siberian Conference on Control andCommunications (SIBCON 2021), 2021, № 9438875.
J. Samaniego, E. López-González, E. Magid, andE. A. Martínez-García,
Path-Tracking
inDoubleSpiraliform Sowing Fields with Agricultural Robotics.
2023, Available at SSRN 4589037.
M.S.A. Mahmud, M.S.Z. Abidin, A.A. Emmanuel and H.S.
Hasan, Robotics and automation in agriculture: present
and future applications. Applications of Modelling and
Simulation, vol. 4, 2020, pp.130-140.
C. Eicher, M. Haesner, M. Spranger, O. Kuzmicheva,
A. Gräser and E. Steinhagen-Thiessen, Usability andacceptability by a younger and older user group regardinga mobile robot-supported gait rehabilitation system.
Assistive technology, 31(1), 2019, p. 25-33.
D. Ryumin, I. Kagirov, A. Axyonov, N. Pavlyuk, A.
Saveliev, I. Kipyatkova, M. Zelezny, I. Mporas and A.
Karpov, A multimodal user interface for an assistive
robotic shopping cart. Electronics, vol. 9, no. 12, 2020,
p. 2093.
E. Chebotareva, R. Safn, K.-H. Hsia, A. Carballo and E.
Magid, Person-following algorithm based on laser range
finder and monocular camera data fusion for a wheeledautonomous mobile robot, in International Conference on
Interactive Collaborative Robotics (ICR 2020). Lecture
Notes in Computer Science, vol 12336, 2020.
A. Eryomin, R. Safin, T. Tsoy, R. Lavrenov, and E. Magid,
Optical Sensors Fusion Approaches for Map Construction:
A Review of Recent Studies. In Journal of Robotics,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
279

Powered by TCPDF (www.tcpdf.org)

Implementation of Alg1 and

18.

19.

20.

21.

22.

23.

24.

25.

26.

27.

28.

29.

30.

31.

32.

Networking and Artificial Life, vol. 10(2), 2023, pp. 127130.
R. Lavrenov, A. Zakiev and E. Magid, Automatic mappingand filtering tool: From a sensor-based occupancy grid toa 3D Gazebo octomap. International Conference on
Mechanical, System and Control Engineering (ICMSC),
2017, pp. 190-195.
M. Mustafin, T. Tsoy, E.A. Martınez-Garcıa,
R. Meshcheryakov and E. Magid, Modelling mobile robot
navigation in 3D environments: camera-based stairs
recognition in Gazebo. Moscow Workshop on Electronic
and Networking Technologies (MWENT-2022), 2022, pp.
1-6.
E. Magid and E. Rivlin, CAUTIOUSBUG: A Competitive
Algorithm for Sensory-Based Robot Navigation.
Proceedings of 2004 IEEE/RSJ International Conferenceon Intelligent Robots and Systems, 2003, pp. 2757-2762.
I. Kamon and E. Rivlin, Sensory-based motion planningwith global proofs. IEEE Transactions on Robotics andAutomation, vol.13, 1997, pp. 814-822.
N. James and T. Bräunl, Performance comparison of bug
navigation algorithms. Journal of Intelligent and Robotic
Systems 50, no. 1, 2007, pp. 73- 84.
V. Lumelsky and T. Skewis, Incorporating range sensing
in the robot navigation function. IEEE Transactions on
Systems, Man, and Cybernetics, 1990, pp. 1058-1069.
V. Lumelsky and A. Stepanov, Path-planning strategies fora point mobile automaton moving amidst unknown
obstacles of arbitrary shape. In Autonomous robot vehicles,
1990, pp. 363-390.
H. Noborio, Evaluation of Path Length Made in
SensorBased Path-Planning with the Alternative
Following. Proceedings of the 2001 IEEE International
Conference on Robotics 8 Automation, 2001, pp. 17281735.
H. Noborio, Several Path-Planning Algorithms of a Mobile
Robot for an Uncertain Workspace and their Evaluation.
Proc. of the IEEE Intelligent Motion Contro, 1990, pp.
289-294.
A. Yufka and O. Parlaktuna Performance comparison of
bug algorithms for mobile robots. In Proceedings of the 5thinternational advanced technologies symposium, 2009, pp.
13-15.
A. Sankaranarayanan and M. Vidyasagar, A New Path
Planning Algorithm For Moving A Point Object Amidst
Unknown Obstacles In A Plane. IEEE Conference on
Robotics and Automation, 1990, pp. 1930-1936.
A. Sankaranarayanan and M. Vidyasagar, Path Planning
For Moving A Point Object Amidst Unknown Obstacles In
A Plane: A New Algorithm And A General Theory For
Algorithm Development. Proc. of the 29th Conference on
Decision and Control, 1991, pp. 1111-1119.
A. Apurin, B. Abbyasov, E.A.Martínez-García and E.
Magid Comparison of ROS Local Planners for a
Holonomic Robot in Gazebo Simulator. In International
Conference on Interactive Collaborative Robotics, Cham:
Springer Nature Switzerland, 2023, pp. 116-126.
A. Dobrokvashina, R. Lavrenov, E. Magid, Y. Bai and M.
Svinin. How to Create a New Model of a Mobile Robotin ROS/Gazebo Environment: An Extended Tutorial.
International Journal of Mechanical Engineering andRobotics Research, 12(4), 2023, pp. 192-199.
N. James and T. Bräunl, A Practical Comparison of Robot
Path Planning Algorithms given only Local Information.
Centre for Intelligent Information Processing Systems
School of Electrical and Electronic Engineering The
University of Western Australia, 2005.

33. F. U. Pereira, P. M. de A. Brasil, M. A. De S. L. Cuadrosand A. R. Cukla, Analysis of Local Trajectory Planners for
Mobile Robot with Robot Operating System. IEEE Latin
America Transactions, vol. 20(1), 2022, pp. 92-99.
34. B. Abbyasov, R. Lavrenov, A. Zakiev, K. Yakovlev,
M. Svinin and E. Magid, Automatic Tool for Gazebo
World Construction: From a Grayscale Image to a 3D
Solid Model. International Conference on Robotics andAutomation (ICRA), 2020, p. 7226-7232.
35. A. Iskhakova, B. Abbyasov, T. Tsoy, T. Mironchuk,
M. Svinin and E. Magid, LIRS-Mazegen: An Easy-to-Use
Blender Extension for Modeling Maze-Like Environments
for Gazebo Simulator. Frontiers in Robotics andElectromechanics, vol. 329, 2023, pp. 147-161.
36. R. Gabdrahmanov, T. Tsoy, Y. Bai, M. Svinin and
E. Magid, Automatic Generation of Random Step
Environment Models for Gazebo Simulator. Lecture Notes
in Networks and Systems, vol. 324, 2021, p. 408-420.

Authors Introduction
Ms. Anastasia Yankova
Anastasia Yankova is a masterstudent in System analysis andmathematical technologies at HSE
University, Russia. Previously she
obtained a bachelor degree in
information systems at ITMO
University, Russia.

Mr. Timur Gamberov
Timur Gamberov is a bachelor
student at Institute of Information
Technology and Intelligent Systems,
Kazan Federal University, Russia.

Ms. Tatyana Tsoy
In 2012 she graduated fromInternational Area Studies Master's
Degree Program at the University ofTsukuba. In 2023 completed a PhD in
Computer science and information
processes at ITIS, KFU, Russia.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
280

Implementation of VisBug-21 and VisBug-22 Path Planning Algorithms Using ROS Noetic
Viktoriia Mirzoian

HSE University, Moscow, Russian Federation
Maxim MustafinIntelligent Robotics Department, Kazan Federal University, 420008, Kazan, Russian Federation
Evgeni MagidIntelligent Robotics Department, Kazan Federal University, 420008, Kazan, Russian Federation
HSE University, Moscow, Russian Federation
E-mail: vimirzoyan@edu.hse.ru, mustafin@it.kfu.ru, magid@it.kfu.ru

Abstract
Local navigation algorithms are crucial for autonomous robots operating in unknown environments where a presenceof obstacles and dynamic changes pose significant challenges. A focus of these algorithms is to enable a real-time
path calculation, allowing a robot to adapt its states dynamically based on corresponding environmental conditions,despite an absence of prior knowledge about surroundings. This paper presents an implementation of the VisBug-21
and VisBug-22 algorithms, designed to address challenges of a local navigation. The algorithms were implemented
for a differential drive robot Turtlebot3 Burger using Robot Operation System (ROS). Virtual experiments were
performed in the Gazebo simulator employing a simple 3D environment that contained only convex obstacles and asmall 3D maze.
Keywords: Path-Planning, Mobile Robots, Local Navigation, VisBug-21, VisBug-22, Gazebo, ROS Noetic

model of a vision sensor [8] is proposed, which functions
similarly to a rangefinder, providing the robot withcoordinates of obstacles’ boundaries within a limited
radius of its visibility [9].

1. Introduction
Robotics researchers face a significant challenge whenit comes to efficient navigation [1]. A current focus in
robot motion planning concentrates around two different
models, each based on distinct assumptions about
available data for planning [2]. The first model, known asa path planning with complete information or the piano
movers problem, assumes perfect knowledge about arobot and obstacles [3]. This paper is concerned with thesecond model, which is a path planning with incomplete
information [4]. In this model, some degree of
uncertainty exists, and missing data are obtained fromlocal sources, e.g., laser range finders or cameras. One
advantage of this model is the ability to incorporate
sensory feedback, transforming a motion planning into a
continuous dynamic process. It also eliminates a need foran analytic representation of obstacle boundaries [5]. A
key issue in the motion planning with incomplete
information is how to integrate sensory data into aplanning function [6], [7].

2. Algorithms’ overview
BUG family of algorithms provides efficient andeffective ways for robots to navigate through populatedwith obstacles environments and reach their target
locations [10]. These algorithms are widely used inrobotics research and are valuable tools for path planningand navigation applications. VisBug-21 and VisBug-22
algorithms are an enhanced version of the basic BUG
algorithm [8], which allow the robot making more
informed decisions by introducing a vision control.
2.1. VisBUG-21
Incorporating vision into Bug2 algorithm is to mentallyreconstruct a segment of Bug2 path visible to the robot at
each step [11]. Using this reconstructed path, the robot
identifies a farthest point and moves directly towards it.
It is important to note that a segment continuity is a
crucial factor in this process, rather than remembering theentire segment itself [12], [13]. The algorithm iscomprised of two key components: a main body thatplans a motion along a path responsible for generating theThis paper describes how to implement twomathematically described algorithms that can guide amobile robot through complex environments with various
obstacles, using basic distance data provided by a range
finder or stereo vision. To achieve this goal, a simplified©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
281

Viktoria Mirzoian, Maksim Mustafin, Evgeni Magidnext intermediate target Ti+1, and a procedure “ComputeTi+1” that tests its reachability [8]. Typically, the main
body only operates in step S1, but step S2 is executed
when the robot is navigating a (locally) convex boundaryof an obstacle, preventing it from utilizing its vision todefine next intermediate target Ti+1.

Procedure Compute Ti-22 keeps identifying points
along the BUG2 path or a quasi- BUG2 path segment,
until a better point (in terms of its proximity to T), S’, isidentified on the line Start-Target. Then S’ becomes thestarting point of another quasi- BUG2 path segment, andthe process repeats. As a result, unlike algorithms Bug2
and VisBug-21, where each defined hit point has its
matching leave point, in VisBug-22 no such matchingnecessarily occurs.

Main body:
1. Move towards current target Ti, execute “ComputeTi-21” and check:a. If start point C = target point T, stop the procedure.
b. If T is unreachable, stop the procedure.
c. If C = Ti, go to step 2.
2. Move along the obstacle boundary, execute“Compute Ti-21” and check:a. If C = T, stop the procedure.
b. If T is unreachable, stop the procedure.
c. If C ≠ Ti, go to step 1.

3. Configuration
An environment configuration consists of Robot
Operation System (ROS), Gazebo, and the TurtleBot.
ROS was chosen due to its modular framework thatallows developers breaking down complex algorithms
into smaller, reusable software components known as
nodes. This modularity enables easier development,
testing, and maintenance of robot motion algorithms [14].
Additionally, ROS provides a wide range of sensor
drivers and libraries that make it easier to integratedifferent sensors into a robot’s system. This allows robots
gathering data from various sources, such as cameras,
LiDAR, or depth sensors, which is crucial for navigatingin unknown environments [15].

Procedure Compute Ti-21 builds a path that BUG2
algorithm would build, and then selects a farthest point
on a section of a path that it sees. This point becomes anext current target.
2.2. VisBug-22

The benefit of using Gazebo for programmingalgorithms for robot movement in unknown conditions isa realistic physics simulation environment for robots [16],[17] allowing developers to test and refine theiralgorithms in various unknown conditions without a needfor physical prototypes and without damage risks. This
enables faster development and iteration cycles. Gazebo
provides a highly customizable and extensible platform
where developers can create virtual environments withdifferent terrain types, lighting conditions, or weather
effects to simulate distinct unknown conditions [18]. This
flexibility allows for comprehensive testing androbustness evaluation of algorithms in various scenarios.

It is possible to create a different method, similar to
BUG2 mechanism but with a more opportunistic
approach compared to VisBug-21. Instead of strictly
following a BUG2 path, the robot can deviate from it if itspots more promising opportunities, while still ensuring
a convergence. This alternative process is referred to as
VisBug-22. A design of this algorithm bears a strong
resemblance to VisBug-21 algorithm, albeit with one
notable difference - the robot no longer prioritizes
making sure that all intermediate targets Tj are located on
the BUG2 path [11]. Instead, it strives to selectintermediate targets that are as close as possible to thetarget T while still being on the same line. This leads to a
distinct convergence mechanism and a behavior that sets
it apart from VisBug-21 algorithm. The algorithm ismade up of two parts: a main body, which is the same as
the main body of VisBug-21, and a procedure named«Compute Ti-22». This procedure calculates a next
intermediate target T based on a current position C of therobot and also checks if the target can be reached.

Fig. 2. Schematic representation of the search for"interesting" points
TurtleBot 3 Burger virtual model of a robot was
employed for experiments in Gazebo simulator (Fig. 1)
due to its simplicity that makes it accessible for users of

Fig.1. Turtlebot3 Burger in the Gazebo environment©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
282

Implementation of VisBug-21

Fig. 4. VisBug-21 path in the maze

Fig. 3. VisBug-21 path in the map with convex
pillars

Another difficulty was related to a practical identifycation of "interesting" points that were described in [1],[2]. At the current location C of the robot (Fig. 2), thealgorithm processes a grid of its sensor’s vision within its
radius r (an area inside the orange circle). The robot
rotates 360 degrees in place and searches for interesting
points (marked with blue dots) that form three groups:

all levels of expertise, including beginners [19], roundshape and differential drive chassis that ease local
navigation. The robot's small size allows navigating
through tight spaces, making it suitable for various
environments, including confined ones.
4. Implementation details- corners of obstacles (indicated as a, b, c, d, e, f);- visible points of obstacles’ boundaries at distance r(indicated as i);- visible intersections of M-line and obstacles (g, h).

Python 3 was utilized to implement VisBug-21 and
VisBug-22. The package includes two separate files
containing the algorithms for VisBug-21 and VisBug-22,
representing distinct processes. Additionally, the main
service offers user services for point navigation,clockwise and counterclockwise wall following. To
ensure the functionality of all services, the following
libraries and modules were employed [20], [21]:5. Validation
The algorithms’ implementation was validated in two
types of environments: a world of identical pillars of acylindric shape and a maze. Virtual tests in Gazebo werecarried out for reachable and unreachable targets. 20(Start, Target) pairs were selected in the first environmentand 25 pairs in the second. Each test was run multiple
times. While most tests were successful, 9.5% of the tests
failed due to situations at the boundaries of the obstacles
where the robot’s wheels were not considered by thesensors and thus the robot got stuck.

- rospy: is a Python client library specifically designed
for ROS.
- sensor_msgs module: records laser measurements.
- geometry_msgs module: generates and transmitts
messages pertaining to setting the robot's position andutilizing points.
- gazebo_msgs module: defines and configures therobot's current state.
- nav_msgs module: odometry use.
- tf module: allows angle transformations.
- std_srv module: involved in running ROS services.
A main difficulty in implementing the algorithm was a
vision system design for the robot. The original articles[8], [11] described only a visual sensor range concept. It
was decided to model the vision system using a
rangefinder package of ROS. The algorithm contains amethod that serves as a callback for processing laser scan
messages. The purpose of this function is to extract
specific ranges of data from the laser scan message andstore them as global variables. It defines global variables
for each range of 360 degrees around the robot, populates
a range dictionary with minimum range values from
specific ranges and assigns a maximal vision range value
if no valid readings are available.

Table 1 presents system configuration that was used
for the validation.
Table 1. System configuration

Parameter
Operation System details
MemoryProcessor

Characteristics
Ubuntu 20.0.4
16.00 GBAMD Ryzen 5 4600H
with Radeon Graphics
3.00 GHz©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
283

Viktoria Mirzoian, Maksim Mustafin, Evgeni Magid

Asian Control Conference (ASCC), May 2022, pp. 14131418.
8. V. Lumelsky and A. Stepanov, Path-planning strategies
for a point mobile automaton moving amidst unknown
obstacles of arbitrary shape. In Autonomous robot
vehicles, 1990, pp 363-390.
9. H. Noborio, Evaluation of Path Length Made in
SensorBased Path-Planning with the Alternative
Following. Proceedings of the 2001 IEEE International
Conference on Robotics 8 Automation, 2001, pp. 17281735.
10. H. Noborio, Several Path-Planning Algorithms of a
Mobile Robot for an Uncertain Workspace and their
Evaluation. Proc. of the IEEE Intelligent Motion Control,
1990, pp. 289-294.
11. V. Lumelsky and T. Skewis, Incorporating range sensing
in the robot navigation function. IEEE Transactions on
Systems, Man, and Cybernetics, 1990, pp. 1058-1069.
12. A. Yufka and O. Parlaktuna, Performance comparison of
bug algorithms for mobile robots. In Proceedings of the
5th international advanced technologies symposium,
2009, pp. 13-15.
13. A. Sankaranarayanan and M. Vidyasagar, Path Planning
For Moving A Point Object Amidst Unknown Obstacles
In A Plane: A New Algorithm and A General Theory for
Algorithm Development. Proc. of the 29th Conference on
Decision and Control, 1991, pp. 1111-1119.
14. N. James and T. Bräunl, Performance comparison of bug
navigation algorithms. Journal of Intelligent and Robotic
Systems 50, no. 1, 2007, pp. 73- 84.
15. A. Sankaranarayanan and M. Vidyasagar, A New Path
Planning Algorithm for Moving A Point Object Amidst
Unknown Obstacles in A Plane. IEEE Conference on
Robotics and Automation, 1990, pp. 1930-1936.
16. A. Iskhakova, B. Abbyasov, T. Tsoy, T. Mironchuk, M.
Svinin and E. Magid, LIRS-Mazegen: An Easy-to-Use
Blender
ExtensionforModeling
Maze-Like
Environments for Gazebo Simulator. Frontiers inRobotics and Electromechanics, vol.329, 2023, pp. 147161.
17. A. Khazetdinov, A. Zakiev, T. Tsoy, R. Lavrenov and K.H. Hsia, Standard-complaint Gazebo warehousemodelling and validation. Proceedings of 13th
International Conference on Developments in eSystems
Engineering (DeSE), 2020, p. 218-221.
18. R. Gabdrahmanov, T. Tsoy, Y. Bai, M. Svinin and E.
Magid, Automatic Generation of Random Step
Environment Models for Gazebo Simulator. Lecture
Notes in Networks and Systems, №324, 2021, pp. 408420.
19. E. Chebotareva, T. Tsoy, B. Abbyasov, J. Mustafina, E.
A. Martinez-Garcia, Y. Bai and M. Svinin, On the
Problems of SLAM Simulation for Mobile Robots in the
Arctic Conditions. Lecture Notes in Computer Science(including subseries Lecture Notes in Artificial
Intelligence and Lecture Notes in Bioinformatics), 12336,
2020, p. 34-44.
20. F. Duan, W. Li and Y. Tan, The Framework andFundamental Use of ROS. In Intelligent Robot:
Implementation and Applications, 2023, pp. 43-69.
21. A. Dobrokvashina, R. Lavrenov, T. Tsoy, E. A. MartinezGarcia and Y. Bai, Navigation stack for the crawler robot
Servosila Engineer. Proceedings of the IEEE 16th
Conference on Industrial Electronics and Applications(ICIEA), 2021, p. 1907-1912.

Fig. 5. VisBug-22 path in the maze
Fig. 3 present a path constructed by VisBug-21
algorithm in the convex environment. Fig. 4 and Fig. 5
demonstrate paths with the same S, T selection in themaze environment by VisBug-21 and VisBug-22
respectively.
6. Conclusions
The paper presented the implementation of VisBug-21
and VisBug-22 algorithms in Python programming
language for Turtlebot3 Burger robot model using ROS
Noetic. The implementation was validated in two
different environments of Gazebo simulator anddemonstrated a success in 90.5% of the test cases.
Acknowledgment
This paper has been supported by the Kazan Federal
University Strategic Academic Leadership Program(“PRIORITY-2030”).
References
1.

2.

3.

4.

5.

6.

7.

G. Huang, Visual-inertial navigation: A concise review.
In 2019 international conference on robotics andautomation (ICRA), May 2019, pp. 9572-9582.
J. Kerr and K. Nickels, Robot operating systems:
Bridging the gap between human and robot Proceedings.
44th IEEE Southeastern Symposium on System Theory(SSST), 2012.
G. E. Jan, K. Y. Chang and I. Parberry, Optimal path
planning for mobile robot navigation. IEEE/ASME
Transactions on mechatronics, 13(4), 2008, pp. 451-460.
E. Magid and E. Rivlin, CAUTIOUSBUG: A
Competitive Algorithm for Sensory-Based Robot
Navigation. Proceedings of 2004 IEEE/RSJ International
Conference on Intelligent Robots and Systems, 2003, pp.
2757-2762.
I. Kamon and E. Rivlin, Sensory-based motion planningwith global proofs. IEEE Transactions on Robotics andAutomation, vol. 13, 1997, pp. 814-822.
E. Magid, Simulation Tools for Urban Search and Rescue
Robotics. International Conference on Artificial Life andRobotics (ICAROB 2023), 2023, pp. 4-11.
I. Mavrin, T. Tsoy and E. Magid, Modified E3 exploration
algorithm for unknown environments with obstacles. 13th

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
284

Powered by TCPDF (www.tcpdf.org)

Implementation of VisBug-21

Authors Introduction
Ms.Viktoria Mirzoian
She is a master student of The
Internet of Things and cyber physicalsystems program at HSE University,
Russian Federation. She got abachelor’s degree in information
systems and technologies at SaintPetersburg
MiningUniversity,
Russia.
Mr. Maksim Mustafin
He is a master student of Intelligent
Robotics program at Institute ofInformation
Technology
andIntelligent Systems (ITIS), Kazan
Federal University (KFU), Russia. In
2023 he obtained a bachelor degree at
ITIS, KFU.
Professor Evgeni Magid
He is a Head of Intelligent Robotics
Department and a Head of Laboratoryof Intelligent Robotic Systems (LIRS)
at Kazan Federal University, Russia.
Professor at HSE University, Russia.
Senior IEEE member. Previously heworked at University of Bristol, UK;
Carnegie Mellon University, USA; University of Tsukuba,
Japan; National Institute of Advanced Industrial Scienceand Technology, Japan. He earned his Ph.D. degree from
University of Tsukuba, Japan. He authors over 200
publications.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
285

DistBug path planning algorithm package for ROS Noetic
Alexander Pak

Higher School of Economics University, Moscow, Russian Federation
Alexander EreminIntelligent Robotics Department, Kazan Federal University, Kazan, Russian Federation
Tatyana TsoyIntelligent Robotics Department, Kazan Federal University, Kazan, Russian Federation
E-mail: adpak_1@edu.hse.ru, aneremin@it.kfu.ru, tt@it.kfu.ru

Abstract
Algorithms of path-planning in an unknown environment play an important role in robotics. They do not require aprior information about obstacles’ locations around a robot and allow calculating a path in a real time. This article
presents an implementation of a sensory-based DistBug algorithm, which operates reactively using range data for
immediate decision-making without constructing a world model. The algorithm was programmed in Python using
robot operating system (ROS) and validated in the Gazebo simulator. For virtual experiments Turtlebot 3 Burger
mobile robot was employed. The experiments were conducted in two types of environment: an environment withconvex obstacles and a maze. The paper demonstrates analysis of experiments using several standard criteria of a path
quality estimation.
Keywords: BUG algorithm, path planning, sensor-based navigation.

(ROS) [13]. The algorithm was tested in the Gazebo
simulator [14]. The navigation was performed in a 3D
environment for the TurtleBot 3 Burger robot model [15].
The robot measures a distance to an obstacle with a 2D
LiDAR of 3.5 m range and 360 degrees field of view.

1. Introduction
A basic motion planning issue is to compute a path froma given starting point to a specified destination point [1].
In robotics, there are two approaches for path planning:
global and local [2]. Global planning typically relies on
preliminarily known data about an environment [3].
Local path planning uses measurements from sensors andknowledge of a target point [4] while the robot moves inan unknown environment avoiding obstacles [5].

2. A brief description of the DistBug algorithm
The sensory-based DistBug algorithm was designed toreach a goal in an unknown environment or indicate thatthe goal is unreachable [12]. The robot does not constructa model of the world and relies on sensory data to makedecisions [16]. The method consists of two behavior
models, represented as movement modes: a straight-line
motion towards the goal and an obstacle boundary
following. A condition for leaving an obstacle is based ona free range in a goal direction when the target is on a lineof sight or a distance to it gradually decreases. The
algorithm utilizes sensory data with a limited range ofview, and the robot moves in steps of a given size. The
core concept of the algorithm is the following:

The algorithms of the BUG family use only local
sensory information and odometry data to control
movement and address a challenge of local navigation
without a need to create a comprehensive map of anenvironment [6]. BUG algorithms have two motion
modes: moving toward a target and following a boundaryof an obstacle [7]. The robot moves in a straight line
towards the target until it reaches a minimum distance toan obstacle [8]. Further, the robot follows a boundary of
the obstacle [9]. The algorithm uses leaving conditions todetermine when to exit the obstacle boundary and headdirectly toward the target again [10]. The robot receives
measurements from its sensors to detect nearby obstacles
and plan a subsequent trajectory [11].

1) Moving from starting point S to target point T in astraight line:•

In this article, we present an implementation of the
DistBug [12] algorithm using robot operating system

If T is reached, the navigation is successfullycompleted.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
286

Alexander Pak, Alexander Eryomin, Tatyana Tsoy•

4. Implementation details

If an obstacle is encountered, a collision pointwith the obstacle is determined, and thealgorithm proceeds to step 2.

DistBug algorithm was implemented in Python
programming language, version 3.7. The implementation
relies on ROS libraries and modules, including rospy (aPython client library for ROS), geometry_msgs (contains
a type of messages for exchanging a position of the robot),sensor_msgs (contains types of laser measurement
messages and motion commands),
gazebo_msgs(contains types that determine a robot state), tf (manages
transformations between coordinate systems), nav_msgs(contains the odometry data type), std_srv (contains
packages for creating services in ROS) [22].

2) The robot follows a boundary of the obstacle, taking
steps of a specified size. If the robot reaches T, thenavigation is successfully completed. Otherwise, therobot leaves the obstacle and heads towards T from apoint of exit, referred as a leaving point. The selection of
the leaving point depends on achieving the following
conditions:••

The robot can move towards T without colliding
with the obstacle.
Freedist > 0, and (Currdist – Freedist ≤ 0 or Currdist– Freedist ≤ Bestdist), where Freedist is a distancewithin the line of sight from a current location toa nearest obstacle in a direction of T, Currdist is adistance from the current location to T, and Bestdist
is a difference between the distance from thecollision point with the obstacle to T and the stepsize.

3) If the robot cannot find a leaving point or it returns toa position it was at a previous step, then T is unreachable.
3. System setup
Fig. 1. Turtlebot3 Burger model in the Gazebo.

The algorithm was implemented in Noetic Ninjemys
version of ROS. The algorithm uses ROS topics to obtainmeasurements from sensors and wheel odometry [17],and to control the robot. Messages are transmitted using
standard types predefined in ROS: sensor_msgs::
PointCloud, geometry_msgs::Twist, nav_msgs::Odometry, and geometry_msgs::Point.

The algorithm’s implementation contains three nodes.
The first and second nodes regulate a robot motion
directly to the target and the clockwise wall following
motion, respectively. Each node implements a ROS
server to activate and deactivate the mode. The third node
launches a certain mode via a request to a corresponding
server. The server choice depends on measurements
received from the laser rangefinder. If the rangefinder
indicates an occupied space ahead of the robot while therobot is moving towards the target, the robot changes the
motion mode. Similarly, if a space in a direction of thetarget is free while the robot is moving along an obstacle
boundary, the robot switches the mode. When the stop
condition occurs, the algorithm turns off all nodes. To
evaluate a performance the algorithm logs its operation
time, all trajectory points (to calculate a path length), anda number of encountered obstacles during a path search.

The algorithm was tested in the ROS-compatible
Gazebo 11 simulator [18]. The Gazebo simulator creates
realistic environments that allow conducting experimentswith virtual robot models. The robot can be equippedwith any of the standard sensors, and Gazebo has asupport for third-party plug-ins. The Robot Visualization
tool (RViz, [19]) is used to visualize a robot's position,measurements from sensors, and a path trajectory. The
tool allows setting a target point as a 2D Nav Goal. A
message is sent to an appropriate topic and accepted by alocal planner.
The TurtleBot3 robot model (Fig. 1) with an opensource software was selected. The TurtleBot3 Burger[20] model was used, which has necessary sensors fornavigating an environment under DistBug protocol. The
robot's design allows making a transition from a 3D
configuration space to a 2D one, which simplifies a pathcalculation [21].

To configure the algorithm, the following parameters
were introduced: a timeout, a stopping threshold, a rangeof the rangefinder (i.e., radius of the robot’s vision), anda robot's step size. The stopping threshold was
empirically defined to determine a point where the robot
stops when a certain coordinate value is reached. The
timeout allows avoiding robot’s stuck cases: despite astatic environment, in some cases the TurtleBot 3 robot
can get stuck at an obstacle boundary if one of its wheel
touches an outer corner of the obstacle. This occurs due©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
287

DistBug path planning algorithm

to a rangefinder’s placement that may cause amisregistration of occupied cells of the environment as
free cells. If the robot gets stuck, a timer is activated andafter the timeout expires, the algorithm stops with anerror message. The rangefinder range and the step size areuser provided parameters. The range should be less than
the step size. A small step size causes a better precisionof a computed path with regard to a theoretical path of an
infinitely smell step and affects how often the obstacle
detour check is be performed. With a large step size arobot may miss the target. The rangefinder measurements
were divided into sectors by their direction. The value ofa sector was determined by a minimum distance to an
obstacle in the range.
5. Validation
To evaluate the implementation, tests were conductedin two types of environment: an environment with convex
3D obstacles of a cylindrical shape and a maze. The tests
included reachable and unreachable goals. In the lattercase the algorithm returned an error after reaching thestop state. The test results were recorded into a file that
contained tracked metrics for each (start S, target T) pair.

Fig. 3. A path of the DistBug in the maze
6. Conclusions
The paper presented an implementation of DistBug
algorithm in the ROS Noetic environment. The
evaluation was performed in two different types ofenvironment and with a set of different target points.
After reaching the stop condition, the control node logged
the metrics, which included a flag of arrival to the target,
a trajectory of the robot, a total time of the algorithm
execution, and a number of encountered on the way
obstacles. The tests demonstrated 97% of success. The
implemented method can be adapted for other robot
models and environments by empirically adjusting thetimeout and the stopping threshold, while setting
corresponding to the robot sensor range and the step size.
Acknowledgements
This paper has been supported by the Kazan Federal
University Strategic Academic Leadership Program(“PRIORITY-2030”).

Fig. 2. A path of the DistBug in the simple map. The
green star marks the start, the red star is the target.

References

We made 20 selection of (S,T) for the simple map (Fig.
2) and 15 pairs for the maze (Fig. 3). For each pair anumber of runs was performed and they demonstrated
approximately 97% success. The failures were cause by
the problem mentioned in Section 4. Fig. 2 and Fig. 3
present the resulted paths of the robot using DistBug inthe simple map and in the maze respectively. The virtual
experiments were conducted using a PC with Ubuntu
20.0.4(LTS) operation system, 6.00 GB memory and
AMD Ryzen 5 5500U processor with 2.10 GHz Radeon
Graphics. While the environments for testing wereconstructed manually, a part of our ongoing work is totest the algorithm in automatically constructedenvironments for Gazebo, e.g., [23], [24], [25].

1. A. Zakiev, R. Lavrenov, E. Magid and V. Indelman, Pathplanning for Indoor Partially Unknown Environment
Exploration and Mapping. ICAROB 2018: Proceedings ofthe 2018 International Conference on Artificial Life andRobotics, 2018, p. 399-402.
2. B. K. Patle et al, A review: On path planning strategies fornavigation of mobile robot. Defence Technology, vol. 15,
2019, pp. 582-606.
3. E. Magid, R. Lavrenov and A. Khasianov, Modified
spline-based path planning for autonomous ground vehicle.
Proceedings of the 14th International Conference on
Informatics in Control, Automation and Robotics(ICINCO 2017), vol. 2, 2017, p. 132-141.
4. H. Zhang, W. Lin and A. Chen, Path planning for themobile robot: A review. Symmetry, vol. 10, 2018, pp. 450.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
288

Powered by TCPDF (www.tcpdf.org)

Alexander Pak, Alexander Eryomin, Tatyana Tsoy

5. V. J. Lumelsky and T. Skewis, Incorporating range sensing
in the robot navigation function. IEEE transactions on
Systems, Man, and Cybernetics, vol. 20, 1990, pp. 10581069.
6. A. Sankaranarayanan and M. Vidyasagar, A new path
planning algorithm for moving a point object amidst
unknown obstacles in a plane. IEEE International
Conference on Robotics and Automation, IEEE, 1990, pp.
1930-1936.
7. V. J. Lumelsky and A. A. Stepanov, Path-planning
strategies for a point mobile automaton moving amidst
unknown obstacles of arbitrary shape. Algorithmica, vol. 2,
1987, pp. 403-430.
8. E. Magid and E. Rivlin, CAUTIOUSBUG: A competitivealgorithm for sensory-based robot navigation. 2004
IEEE/RSJ International Conference on Intelligent Robots
and Systems (IROS) (IEEE Cat. No. 04CH37566), IEEE,
vol. 3, 2004, pp. 2757-2762.
9. H. Noborio, Several path-planning algorithms of a mobile
robot for an uncertain workspace and their evaluation. Proc.
of the IEEE Int. Work. Intel. Motion Control, Istanbul,
Turkey, vol. 1, 1990, pp. 289-294.
10. A. Yufka and O. Parlaktuna, Performance comparison of
bug algorithms for mobile robots. Proceedings of the 5thinternational advanced technologies symposium, Karabuk,
Turkey, 2009, pp. 13-15.
11. Y. Horiuchi and H. Noborio, Evaluation of path length
made in sensor-based path-planning with the alternative
following. Proceedings 2001 ICRA. IEEE International
Conference on Robotics and Automation (Cat. No.
01CH37164), IEEE, vol. 2, 2001, pp. 1728-1735.
12. I. Kamon and E. Rivlin, Sensory-based motion planningwith global proofs. IEEE transactions on Robotics andAutomation, vol. 13, 1997, pp. 814-822.
13. A. Apurin, B. Abbyasov, E.A.Martínez-García and E.
Magid, Comparison of ROS Local Planners for a
Holonomic Robot in Gazebo Simulator. International
Conference on Interactive Collaborative Robotics, Cham:
Springer Nature Switzerland, 2023, pp. 116-126.
14. A. Dobrokvashina, R. Lavrenov, E. Magid, Y. Bai and M.
Svinin, How to Create a New Model of a Mobile Robot in
ROS/Gazebo Environment: An Extended Tutorial.
International Journal of Mechanical Engineering andRobotics Research, vol. 12, 2023, pp. 192-199.
15. R. Safin, R. Lavrenov, K.-H. Hsia, E. Maslak, N.
Schiefermeier-Mach and E. Magid, Modelling a
TurtleBot3 Based Delivery System for a Smart Hospital in
Gazebo. The 15th Siberian Conference on Control andCommunications (SIBCON 2021), 2021, № 9438875.
16. R. Safin, T. Tsoy, R. Lavrenov, I. Afanasyev and E. Magid,
Modern Methods of Map Construction Using OpticalSensors Fusion. International Conference on Artificial Lifeand Robotics (ICAROB 2023), 2023, pp. 167-170.
17. P. Marin-Plaza et al, Global and local path planning study
in a ROS-based research platform for autonomous vehicles,
Journal of Advanced Transportation, 2018, pp. 1-10.
18. B. Abbyasov, K. Kononov, T. Tsoy, E. A. Martinez-Garcia
and E. Magid, Experience in Efficient Real OfficeEnvironment Modelling in Gazebo: a Tutorial.
International Conference on Artificial Life and Robotics(ICAROB 2022), 2022, p. 673-677.
19. H. Maghfiroh and H. P. Santoso, Online Navigation of
Self-Balancing Robot using Gazebo and RVIZ. Journal of
Robotics and Control (JRC), 2(5), 2021.

20. Fabio Ugalde Pereira, Pedro Medeiros de Assis Brasil,
Marco Antonio De Souza Leite Cuadros and Anselmo
Rafael Cukla, Analysis of Local Trajectory Planners for
Mobile Robot with Robot Operating System, IEEE Latin
America Transactions, vol. 20, 2022, pp. 92-99.
21. N. James and T. Bräunl, Performance comparison of bug
navigation algorithms, Journal of Intelligent and Robotic
Systems, vol. 50, 2007, pp. 73-84.
22. A. Dobrokvashina, S. Sulaiman, T. Gamberov, K.-H. Hsia
and E. Magid, New Features Implementation for Servosilla
Engineer Model in Gazebo Simulator for ROS Noetic.
International Conference on Artificial Life and Robotics(ICAROB 2023), 2023, pp. 154-157.
23. R. Lavrenov, A. Zakiev and E. Magid, Automatic mappingand filtering tool: From a sensor-based occupancy grid toa 3D Gazebo octomap. International Conference on
Mechanical, System and Control Engineering (ICMSC),
2017, p. 190-195.
24. A. Iskhakova, B. Abbyasov, T. Tsoy, T. Mironchuk, M.
Svinin and E. Magid, LIRS-Mazegen: An Easy-to-Use
Blender Extension for Modeling Maze-Like Environments
for Gazebo Simulator. Frontiers in Robotics andElectromechanics, Vol. 329, 2023, pp. 147-161.
25. B. Abbyasov, R. Lavrenov, A. Zakiev, K. Yakovlev, M.
Svinin and E. Magid, Automatic Tool for Gazebo World
Construction: From a Grayscale Image to a 3D Solid
Model. International Conference on Robotics andAutomation (ICRA), 2020, p. 7226-7232.

Authors Introduction
Mr. Alexander Pak
He is a third-year bachelor student ofComputer Science and Engineering at
HSE University, Russia.

Mr. Alexander Eryomin
He received a BSc degree in Software
Engineering from the School of Spaceand Information Technology at theSiberian Federal University in 2022.
Currently, he is a second-year student ofMaster program in Intelligent Robotics
at the Institute of Information
Technology and Intelligent Systems at the Kazan Federal
University, Russia.
Ms. Tatyana Tsoy
In 2012 she graduated from the
University of Tsukuba. Since 2018 she
has been a PhD student in Robotics at
the Institute of Information Technology
and Intelligent Systems of Kazan
Federal University, Russia.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024289

On sensor modeling in Gazebo simulator
Niez Yuldashev

Intelligent Robotics Department, Kazan Federal University, 420008, Kazan, Russian Federation
Alexandra Dobrokvashina

Intelligent Robotics Department, Kazan Federal University, 420008, Kazan, Russian Federation
Roman LavrenovIntelligent Robotics Department, Kazan Federal University, 420008, Kazan, Russian Federation
Email: NBYuldashev@stud.kpfu.ru, dobrokvashina@it.kfu.ru, lavrenov@it.kfu.ru

Abstract
Sensor modeling in the Gazebo simulator is fundamental to robotics advancement. This review explores sensor
modeling intricacies, methodologies, and applications, while emphasizing a critical role of a precise sensor modeling.
Application scenarios demonstrate a sensor modeling's broad utility across fields including medical diagnostics,
autonomous navigation, and industrial automation. Differences in research focus, methodology, and implementation
underline a varied nature of sensor modeling studies. Key challenges include a need for more detailed world models.
The paper guides a research in sensor modeling and identifies crucial questions.
Keywords: Overview, Gazebo, Sensor modeling, Sensor simulation.

approaches to sensor modeling within the Gazebo
simulator. We will explore the challenges researchers
address in this domain, the methods employed for precise
sensor modeling, and the profound impact these models
have on the development of robotic systems. Moreover,
we will discuss the primary challenges researchers
encounter and outline potential directions for future
research in this rapidly evolving field.

1. Introduction
In contemporary robotics and autonomous systems,
simulation plays an important role [1]. It offers
researchers and engineers the means to create and test
robots in virtual environments, reducing costs andensuring safety during the development of robotic
systems [2]. However, for simulations to produce
realistic results, precise modeling of sensors, such as
cameras, lidars, infrared, and ultrasonic sensors, isparamount. These sensors are instrumental in perceiving
the robot's surroundings and interacting with them.
Accurate sensor models in simulators have become
fundamental for the successful development and testingof perception, navigation, and control algorithms for
robots.

2. Methodologies
The choice of a specific method and methodology formodeling sensors in the Gazebo simulator depends on
several key factors. Initially, this is the type of sensor thatis required to simulate, since different sensors havedifferent characteristics and requirements. For example,
the modeling of the camera requires accounting for
optical parameters [5], while the modeling of theultrasonic sensor can be based on acoustic properties [6].
The objectives of the study also play an important role inchoosing a methodology. If the main goal is to assess theaccuracy of the sensor, then the methods focused oncomparison with real dimensions can be the mostsuitable [7]. If the goal is to develop and test newalgorithms for processing data from the sensor, thenmodeling taking into account noise and distortion can be
more relevant [8]. The level of detail also plays a key role.
Highly detailed models of sensors may require large

In this review article, we delve into a significant issue
within the realm of robotic simulation, focusing on
sensor modeling in the Gazebo simulator [3]. Among
robotic simulators, Gazebo stands out for its powerful
functionality and flexibility. It allows for the creation ofcomplex 3D robot models and their environments, alongwith the simulation of a wide range of sensors [4].
The objective of our article is to provide readers withan overview of contemporary methodologies and©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
290

Niez Yuldashev, Alexandra Dobrokvashina, Roman Lavrenov

computing resources and time for simulation, while
simplified models can be less accurate, but more
effective in terms of computing resources [9].

simulation environments for testing and training in thefield of autonomous systems and robotics.
4. Distinctions

Some of the common methodologies include: Physicalmodeling, including taking into account the physicalproperties of the sensor and its interaction with theenvironment [10]. Methods of machine learning thatallow training sensors based on real data and apply it insimulations [11]. Geometric modeling, focused on theexact reconstruction of the geometry of the environmentand objects with which the sensor interacts [12].
Statistical modeling methods: include statistical
approaches to take into account various sources of noise
and uncertainty in the measurements of sensors [13]. This
may include methods for filtering, smoothing andassessing the state that allow you to take into account thestatistical characteristics of measurements.

Each scientific paper can focus on different aspects ofsensor modeling. For instance, one paper may investigate
camera modeling for robot navigation [17], while anothermay emphasize lidar sensors for obstacle detection [16].
Consequently, the research topic can substantially differ.
Researchers can use various methods and techniques
for sensor modeling. For example, some may develop
their own sensor models [18], [19], while others mightutilize standard tools and libraries available in Gazebo[20]. This can impact the accuracy of modeling and thescope of the results.
Studies may also differ in terms of research objectives.
Some papers may aim to optimize sensor
performance [21], while others seek to develop new dataprocessing algorithms or assess sensor reliability invarious scenarios [22].

Table 1 provides an overview of recent and noteworthy
publications related to sensor modeling within the
Gazebo simulator. This table showcases articles that haveleveraged Gazebo for simulating sensors, highlighting
the various contexts and applications where these models
have been employed.

Different studies may be oriented toward various typesof robotic systems. For example, research may target
mobile robots, autonomous vehicles [23], robot
manipulators [6], [24], or drones [25], and each of thesesystem types may have unique characteristics in sensor
modeling.

3. Common aspects
One of the key tasks of research involving sensor
modeling is to evaluate and analyze the accuracy ofvarious sensors, such as cameras, lidars, ultrasonic andinfrared sensors, in a virtual environment. This allows
researchers to understand how the sensors interact withthe environment and how accurate the data they
provide [5], [6].

Scientific papers can be directed toward diverse
applications of sensor modeling, such as medical
diagnostics [26], autonomous navigation [17], [27], andmore.

Research in this area often involves the creation andtesting of data processing algorithms for sensors. Thesealgorithms may include computer vision methods, image
processing, machine learning, and other techniques used
to analyze data collected from sensors [14].

These differences in research themes, methods, andobjectives make each scientific paper unique andspecialized. Collectively, they contribute to theadvancement of sensor modeling in the Gazebo simulator,
providing new knowledge and tools for developers andresearchers in the field of robotics.

Gazebo's sensor modeling research aims to create
virtual scenarios that are as close to real-world conditions
as possible. Researchers strive to accurately model theobjects with which sensors interact and whose condition
can affect the simulated sensor readings [6], [15].

5. Application scenarios
Modeling sensors in the Gazebo simulator finds
applications in various fields of robotics and autonomous
systems.

Researchers often use sensor simulations to validate thealgorithms of their robots. This allows them to test thevalidity of their designs under different conditions.
Research may also involve integrating sensor modelinginto real robotic platforms. This helps researchers adapt
the results of their work for use in actual robots [15], [16].

One of the key applications of sensor modeling in
Gazebo is the development and testing of autonomoustransportation systems, such as self-driving cars [30].
Virtual simulations enable engineers to assess navigationand control algorithms under diverse conditions,ensuring safety before these vehicles hit the roads.

In summary, research related to sensor modeling in the
Gazebo simulator aims to improve the understanding of
sensor behavior in robotics systems, develop efficient
data processing algorithms, and create realistic©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
291

Niez Yuldashev, Alexandra Dobrokvashina, Roman Lavrenov

Table 1. Relevant publications in Gazebo sensors simulation.
Sensor modeling context
A software package for ultrasoundsensor modeling in the Gazebo simulator
Incorporating sensors into the model
Incorporating sensors into the model
It is explained how to incorporate sensors into models in
GazeboShows physical integration of the main types of sensors in
UAV domain both for navigation and collision avoidance
Automatic tool allows creating realistic landscapes in Gazebo
simulation based on results of real world sensor-based
exploration
The simulated tactile sensor can produce high-resolution
images from depth-maps captured by a simulated optical sensor
Simulating the operation of LIDAR sensors in real-time,
considering LASER beam propagation and attenuation of
LASER energy in adverse weather conditions such as fog andrain
An algorithm for detection of obstacles and lines was
implemented for the IR sensors
The scalability investigation of LIDAR-type sensor simulation
in ROS-supported Gazebo
Evaluate a variety of visual and LiDAR SLAM algorithms in asimulated environment
Simulation of the RFID readers-antennas mounted on robots
reading RFID tags in the environment

Sensor type

Application

Reference

Ultrasound sensor

Medical Robotics[6]

USAR, AMR

[15]

Autonomous
Vehicles[23]

LRF, IMU sensor,
camerasLIDAR, IMU,
GPS/GLONASS,
Side sonars
LRF, IMU sensor,
camerasLIDAR, sonar,
radar,video

Robotics
Education, AMR
Autonomous
Vehicles, UAV

[28]
[16]

LIDAR, RGBDcamera

UAV, UGV[12]

GelSight tactile
sensorTactile Sensing
Visualization

[18]

LIDAR

Automotive

[7]

IR

Robotics
Education

[20]

LIDAR

AMR

[29]

LIDAR, RGB-Dcamera

Mobile Robots

[21]

RFID

UAV

[19]

A method to simulate the ARVA transceiver

ARVA

Search And
Rescue, UAV[10]

Test the control-level and sensor-level algorithms in simulation
environment

LIDAR

UAV

[25]

In the medical domain, Gazebo's sensor modelingcapabilities can be used to create virtual training
environments [31]. These environments allow medical
professionals to practice operating medical robots andother devices [6]. Additionally, it is beneficial for
research in the field of medical robotics, including
surgical robots and rehabilitation devices.

objectives in the realms of robotics and autonomous
systems.
6. Main challenges and future directions
While sensor modeling in the Gazebo simulator has
seen remarkable advancements, several challenges
persist, and promising future directions are emerging.
Here, we outline some of the key challenges and areas ofpotential growth.

Gazebo is frequently used in the educational field toteach students and researchers the fundamentals ofrobotics [28]. Sensor modeling permits students toexperiment with various sensor types and data processing
algorithms, enhancing their understanding of robotics
principles.

Many studies have identified the need for further
refinement of the world models used insimulations [6], [15]. Realism of sensor simulations
often depends on the complexity of the environment
being simulated. Future research may focus on enhancing
world models to better mimic real-world scenarios,
including factors like lighting, weather, and dynamic
objects.

Sensor modeling in Gazebo is invaluable for simulatingcomplex urban search and rescue (USAR) scenarios [15].
It allows researchers and first responders to test various
robotic systems that can navigate disaster-stricken areas,
locate survivors, and assess structural damage, all while
avoiding risks to human life.

Some articles focused on the complexity of sensory
synthesis [32]. As robotic systems become moresophisticated, sensor fusion, the integration of data frommultiple sensors, poses a growing challenge. Future
developments may revolve around creating more

These applications showcase the versatility of sensor
modeling in the Gazebo simulator. They represent only afraction of the possibilities offered by sensor modeling,with its utility extending to various challenges and©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
292

Niez Yuldashev, Alexandra Dobrokvashina, Roman Lavrenov

advanced sensor fusion models to handle the increasing
complexity of robotic applications.

References
1.

Balancing realism and computational efficiency
remains a fundamental challenge. Simulations with highlevels of detail can be computationally intensive, limiting
real-time performance [29], [33]. Researchers may
explore novel techniques to achieve a harmonious
balance between realism and efficiency.

2.

3.

Leveraging AI and machine learning for sensor
modeling is a burgeoning area [34]. Future research may
delve into integrating AI techniques for sensor datainterpretation, enabling robots to learn from simulated
sensor inputs.

4.

5.

As robots increasingly interact with humans, modelingsensors for human-awareness becomes vital. Future work
may focus on simulating sensors that enable robots toperceive and respond to human actions and intentions.

6.

With the rise of autonomy in robotics, future directions
include the development of sensor models that support
high-level decision-making in autonomous systems.
7.

In summary, sensor modeling in Gazebo continues to
evolve, aiming for realism, efficiency, and applicability.
Addressing these challenges and exploring these future
directions will contribute to the continued growth ofrobotics and autonomous systems.

8.

9.

7. Conclusion
In conclusion, this review article has explored the
intricate world of sensor modeling within the Gazebo
simulator. We've delved into the aspects of sensor
modeling, discussed current research directions, andidentified key research questions in this evolving field.
Through an examination of various methodologies,
application scenarios, and the distinctiveness of eachresearch endeavor, we've gained a comprehensive
understanding of how sensor modeling contributes to theadvancement of robotic systems. As the field of robotics
continues to expand, the importance of accurately
modeling sensors in virtual environments has become
increasingly clear. This review serves as a foundational
resource for further research and innovations in thedomain of sensor modeling within the Gazebo simulator.

10.

11.

12.

13.

Acknowledgements

14.

This work was supported by the Russian Science
Foundation (RSF), project ID 22-21-20033.
15.

K. Liu, D. Negrut, The Role of Physics-Based Simulatorsin Robotics, Annual Review of Control, Robotics, andAutonomous Systems, 4, 2021, pp 35-58.
K. Fernandez, The use of computer graphic simulation in
the development of robotic systems, Acta Astronautica,
17(1), 1988, pp. 115-122.
N. Koenig, A. Howard, Design and use paradigms for
Gazebo, an open-source multi-robot simulator,
International Conference on Intelligent Robots andSystems (IROS), 3, 2004, pp. 2149-2154.
R. Lavrenov, A. Zakiev, Tool for 3D Gazebo map
construction from arbitrary images and laser scans,International Conference on the Developments ineSystems Engineering, 2017, pp. 256-261.
T. Tsoy, R. Safin, E. Martinez-Garcia, S. Roy, S. Saha,
E. Magid, Exhaustive simulation approach for a virtual
camera calibration evaluation in Gazebo, International
Conference on Automation, Robotics and Applications(ICARA), 2022, pp. 233-238.
B. Abbyasov, A. Dobrokvashina, R. Lavrenov,
E. Kharisova, T. Tsoy, L. Gavrilova, S. Bulatov,
E. Maslak, N. Schiefermeier-Mach, E. Magid,
Ultrasound sensor modeling in Gazebo simulator for
diagnostics of abdomen pathologies, International
Siberian Conference on Control and Communications(SIBCON), 2021, pp. 1-6.
M. Hadj-Bachir, P. Souza, LIDAR sensor simulation inadverse weather condition for driving assistancedevelopment, 2019.
A. Stefanov, Distortion analysis of underwater acoustic
sensor networks, International Conference on New
Technologies, Mobility and Security (NTMS), 2015, pp.
1-4.
K. Demarco, M. West, A. Howard, A computationallyefficient 2D imaging sonar model for underwater robotics
simulations in Gazebo, OCEANS 2015 - MTS/IEEE
Washington, 10, 2015, pp. 1-7.
W. Zhao, J. Queralta, T. Westerlund, Sim-to-RealTransfer in Deep Reinforcement Learning for Robotics: aSurvey, IEEE Symposium Series on Computational
Intelligence (SSCI), 2020, pp. 737-744.
R. Lavrenov, A. Zakiev, E. Magid, Automatic mappingand filtering tool: From a sensor-based occupancy grid toa 3D Gazebo octomap, International Conference on
Mechanical, System and Control Engineering (ICMSC),
2017, pp. 190-195.
N. Hirsenkorn, T. Hanke, A. Rauch, B. Dehlink, R.
Rasshofer, E. Biebl, Virtual sensor models for real-time
applications, Advances in Radio Science, vol. 14, 2016,
pp. 31-37.
A. Buyval, I. Afanasyev, E. Magid, Comparative analysis
of ROS-based Monocular SLAM methods for indoor
navigation, International Conference on Machine Vision(ICMV), 2017.
A. Dobrokvashina, R. Lavrenov, E. Martinez-Garcia, Y.
Bai, Improving model of crawler robot Servosila"Engineer" for simulation in ROS/Gazebo, International
Conference on Developments in eSystems Engineering(DeSE), 2020, pp. 212-217.
J. García, J. Molina, Simulation in real conditions ofnavigation and obstacle avoidance with PX4/Gazebo
platform, IEEE International Conference on Pervasive
Computing and Communications Workshops (PerCom
Workshops), 2019, pp. 979-984.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
293

Powered by TCPDF (www.tcpdf.org)

On sensor modeling in

16. M. Mustafin, T. Tsoy, E. Martınez-Garcıa,
R. Meshcheryakov, E. Magid, Modelling mobile robot
navigation in 3D environments: camera-based stairs
recognition in Gazebo, Moscow Workshop on Electronic
and Networking Technologies (MWENT), 2022, pp. 1-6.
17. D. Gomes, P. Paoletti, S. Luo, Generation of GelSight
Tactile Images for Sim2Real Learning, IEEE Robotics
and Automation Letters, 6(2), 2021, pp. 4177-4184.
18. A. Alajami, R. Pous, G. Moreno, Simulation of RFID
Systems in ROS-Gazebo, International Conference on
RFID Technology and Applications (RFID-TA), 2022,
pp. 113-116.
19. A. Rafael, C. Santos, D. Duque, S. Fernandes, A. Sousa,
L. Reis, Development of an AlphaBot2 Simulator for RPi
Camera and Infrared Sensors, Robot 2019: Fourth Iberian
Robotics Conference, 2019, pp. 502-514.
20. R. Giubilato, A. Masili, S. Chiodini, M. Pertile, S. Debei,
Simulation Framework for Mobile Robots in PlanetaryLike Environments, International Workshop on
Metrology for AeroSpace (MetroAeroSpace), 2020, pp.
594-599.
21. R. Lavrenov, F. Matsuno, E. Magid, Modified splinebased navigation: Guaranteed safety for obstacle
avoidance, Interactive Collaborative Robotics, 2017,
pp. 123-133.
22. I. Shimchik, A. Sagitov, I. Afanasyev, F. Matsuno,
E. Magid, Golf cart prototype development andnavigation simulation using ROS and Gazebo, MATEC
Web of Conferences, vol. 75, 2016.
23. A. Klimchik, E. Magid, S. Caro, K. Waiyakan,
A. Pashkevich, Stiffness of serial and quasi-serial
manipulators: comparison analysis, MATEC Web ofConferences, vol. 75, 2016.
24. J. Zhu, C. Xu, A comprehensive simulation testbench foraerial robot in dynamic scenario using gazebo-ros,
Chinese Automation Congress (CAC), 2017, pp. 76647669.
25. A. Liu, F. Tendick, K. Cleary, C. Kaufmann, A Survey ofSurgical Simulation: Applications, Technology, and
Education, Presence: Teleoperators and Virtual
Environments, 12, 2003, pp. 599-614.
26. Y. Alborzi, B. Jalal, E. Najafi, ROS-based SLAM andNavigation for a Gazebo-Simulated Autonomous
Quadrotor, International Conference on Research and
Education in Mechatronics (REM), 2020, pp. 1-5.
27. A. Dobrokvashina, R. Lavrenov, E. Magid, Y. Bai,
M. Svinin, How to Create a New Model of a Mobile
Robot in ROS/Gazebo Environment: An Extended
Tutorial, International Journal of MechanicalEngineering and Robotics Research, 12(4), 2023,
pp. 192-199.
28. A. Saglam, Y. Papelis, Scalability of sensor simulation in
ROS-Gazebo platform with and without using GPU,
Spring Simulation Conference (SpringSim), 2020, pp. 1–
11.
29. P. Kaur, S. Taghavi, Z. Tian, W. Shi, A Survey on
Simulators for Testing Self-Driving Cars, International
Conference on Connected and Autonomous Driving(MetroCAD), 2021, pp. 62-70.
30. M. Tiryaki, Ö. Erin, M. Sitti, A Realistic Simulation
Environment for MRI-Based Robust Control ofUntethered Magnetic Robots With Intra-Operational
Imaging, IEEE Robotics and Automation Letters, 5(3),2020, pp. 4501-4508.
31. E. Cheborateva, R. Safin, K. Hsia, A. Carballo, E. Magid,
Person-Following Algorithm Based on Laser Range
Finder and Monocular Camera Data Fusion for a Wheeled

Autonomous Mobile Robot, Interactive Collaborative
Robotics, 2020, pp. 21-33.
32. J. Borrego, R. Figueiredo, A. Dehban, P. Moreno,
A. Bernardino, J. Santos-Victor, A generic visual
perception domain randomisation framework for Gazebo,
IEEE International Conference on Autonomous Robot
Systems and Competitions (ICARSC), 2018, pp. 237-242.
33. K. Chin, T. Hellebrekers, C. Majidi, Machine Learning
for Soft Robotic Sensing and Control, AdvancedIntelligent Systems, 2(6), 2020.
34. M. Mustafin, E. Chebotareva, E. Magid, H. Li,
E. Martinez-Garcia, Features of Interaction Between a
Human and a Gestures-Controlled Collaborative Robot inan Assembly Task: Pilot Experiments, International
Conference on Artificial Life and Robotics (ICAROB),
28, 2023, pp. 157-161.

Authors Introduction
Mr. Niez Yuldashev
Currently he is a master student at
Institute of Information Technology
and Intelligent Systems, Kazan
Federal University, Russia.

Ms. Alexandra Dobrokvashina
She received a master degree atInstitute of Information Technology
and Intelligent Systems, Kazan
Federal University in 2021.
Currently she works as a research
assistant at the Laboratory ofIntelligent Robotic Systems (LIRS)
at Kazan Federal University, Russia.

Assistant Professor Roman Lavrenov
Currently he works as an AssistantProfessor at Kazan Federal
University. He obtained his Ph.D. atKazan Federal University in 2020.
He teaches the following master's
courses:«Robotic
Operating
System», «Industrial robotics» andothers. Author of an online course and methodological
guide on the Robotic Operating System (ROS). His maininterests are path planning, robot operating system, andautonomous mobile robots.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
294

An Overview of Kinect Based Gesture Recognition Methods
Alexander Alexeev

Institute of Information Technology and Intelligent Systems, Kazan Federal University, Kazan, Russia
Tatyana Tsoy

Laboratory of Intelligent Robotics Systems (LIRS), Institute of Information Technology and Intelligent Systems, Kazan
Federal University, Kazan, Russia
Edgar A. Martínez-García

Robotics Laboratory, Department of Industrial Engineering and Manufacturing, The Autonomous University of Ciudad
Juarez, Chihuahua, Mexico
Evgeni Magid

Laboratory of Intelligent Robotics Systems (LIRS), Institute of Information Technology and Intelligent Systems, Kazan
Federal University, Kazan, Russia
HSE University, Moscow, Russia
E-mail: san73rus@gmail.com, tt@it.kfu.ru, edmartin@uacj.mx, magid@it.kfu.ru

Abstract
Visual sensors play an important role in a broad variety of robotic systems applications. Even though Kinect
technology appeared over 10 years ago, Kinect sensors are still actively employed by researchers around the world.
This paper presents an overview of Kinect and Kinect 2 sensors’ applications in a human gesture based control. We
analyzed existing research papers to estimate a popularity of particular feature extraction and gesture recognition
methods, recommendations on a distance between an object of interest and a sensor, reported accuracy and latency ofthe sensor. Our analysis is supposed to facilitate a selection of a suitable combination of methods for a particularapplication of Kinect sensor in gesture recognition while considering its performance.
Keywords: Microsoft Kinect, robotics, gesture control, gesture recognition.

2. Kinect in Robotics

1. Introduction

Kinect sensor (Fig. 1a) is a device introduced byMicrosoft in 2010 for XBOX 360 gaming console control.
A combination of an infrared sensor and RBG-D camera
makes Kinect suitable for human body tracking purposes.
Software Development Kit [10] allows developing acustom software for a variety of tasks including gesture
based robot control. The introduced by Microsoft in 2013
Kinect v2 was (Fig. 1b) a second generation of Kinect
device that also combines RGB-D and infrared sensors.
Similar to its predecessor, Kinect v2 is intended for a
gesture-based control. However, due to technicalimprovements such as adding a wide angle time of flight(ToF) camera [11] and upscaling a color camera
resolution to 1080p [12], it exhibits a better performancecompared to the first generation, which is demonstrated
in Section 4.

Robotics has achieved a significant progress, enabling
robots to autonomously plan routes [1] and interact withhumans in medicine [2], education [3], and rescue
operations [4]. Controlling robots with a voice andgestures is the most natural way of interaction [5] due todaily habits of using these methods by humans. Control
using gestures can be implemented with vision sensors[6], electromyography methods (that track human muscle
contractions [7]), employing a touch screen, an
accelerometer or other sensors [8]. This paper overviews
gesture control implementations in robotics using Kinect
sensor. Kinect is originally a Microsoft motion controller
for gaming that uses computer vision methods for a
gesture based control. Later Microsoft enabled a custom
application development with Software Development Kit
for Windows [9]. This study summarize information
about Microsoft Kinect use in robotics, comparesaccuracy of different gesture recognition methods andemphasizes a reported latency of Kinect in various tasks.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
295

Alexander Alexeev, Tatyana Tsoy, Edgar A. Martínez-García, Evgeni Magida recognition system demonstrates significant delays, adifferent method of control should be considered. In thissection, we summarize information about methods ofKinect based control and emphasize their accuracy.
Table 1 presents an analysis of studies that used various
combinations of feature extraction and gesturerecognition methods. The first column of the table
describes a particular task; the second column Dist
specifies a recommended optimal distance between an
object of interest (gesture producer) and a sensor inmeters; the third and the forth – a feature extraction FEM
and a gesture recognition methods GRM, respectively.
The fifth column Acc shows an accuracy of eachapproach in percents (as reported by its authors); the sixth
Skeleton emphasizes if an approach employes a skeleton
extraction; the last one Rf refers to a corresponding paper.
N/A in the table denotes that a paper does not containcorresponding data.

Fig. 1. Microsoft Kinect first generation and Kinect v2.
Kinect and Kinect v2 capture color and depth data of ascene and are broadly used in computer vision for object
recognition including gesture recognition. The distinctive
feature of Kinect family is its versatility compared tocounterpart sensors. Leap Motion Controller is also usedin robotics for gesture-based robot control [13], [14]. Yet,this device is designed to recognize only hand
movements, whereas Kinect does not have such
limitations and can be employed for a full-body gesturerecognition [15] and a face tracking [16]. In addition,
Kinect v2 contains an embedded microphone that enables
a voice control [17], which can augment a gesture-based
control. Leap Motion Controllers could be used in pairs,e.g., to recognize each hand separately [18], which may
increase the system's load. Similarly, Kinect sensors canform complicated systems for tracking and motion
capture system purposes [19], [20]. However, in somecases Kinect sensor may not be a right solution, e.g.,authors in [21] complained that Kinect is not compatible
with Linux systems as Kinect’s Software Development
Kit has only Windows OS support. To solve this issue,
the researchers employed Asus Xtion Pro Live with an
RGB-D camera similar to Kinect.

As Table 1 shows, a combination of the Histogram ofOriented Gradients (HOG) for feature extraction andSupport Vector Machine (SVM) for gesture recognition([24], [33]) is a popular approach among researchers. The
same combination was used at an earlier stage of [26]research, however, a deep learning AlexNet model
eventually was chosen due to a better performance. In[31] the authors emphasized an influence of a distance atwhich gestures were recognized by Kinect on arecognition accuracy. The highest accuracy of 87% wasachieved at a distance of 3 meters from the Kinect, while
the lowest accuracy of gesture recognition of 25% was
obtained at a distance of 5 meters from the sensor. The
importance of a classifier selection when controlling arobot with user-defined gestures was confirmed in [23],where the SURF and FLANN methods were employed,
to extract arbitrary points from an image and to recognize
gestures. During the experiments, it was discovered that
the FLANN library failed to achieve required results inrecognizing gestures as it could not match corresponding
features in a variety of gestures.
Majority of studies in Table 1 used a method of askeleton construction, where a skeleton consists of jointsof a human body or hands only. This approach allows amore accurate extraction of special points, which arefurther required for a gesture recognition. An alternative
approach built a 3D model of a hand [34]; yet the authors
did not provide accuracy indicators for the gesturerecognition, and thus a usability of the method for a real
gesture recognition system is questionable.

In robotics, Kinect could be used to control different
types of real robots. In [17] authors presented a Kinectbased industrial robot control and demonstrated that
Kinect could be successfully integrated into simulators.
In [22] Kinect was applied for a remote control of amechanical arm that replicates operator-defined gestures.
A gesture-based control using Kinect could also be
implemented in mobile robotics, where Kinect serves asa robot control panel [23].

4. Kinect Latency
3. Gesture Recognition Methods
Latency has a significant importance when operating
technical devices. Latency can be caused by hardware

Choosing a gesture recognition method for Kinect
based control is important, as it directly affects an overallsystem performance. If a gesture recognition method is
not accurate enough, not suitable for a particular task or©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
296

An Overview of Kinecte

Task

Dist, mTable 1. Gesture Recognition Methods
FEM
GRM

Static hand gesturerecognition
Hand gesture recognition

N/A

Touchless visualizationof 3D medical images

2.5-3.5

Histogram ofOriented Gradients

Hand gesture recognition

N/AMediaPipe Palm
Detector

Sign language
recognition for Arabic
speakers

N/A

Used (not specified
which one)

Dynamic hand gesturerecognition
Full body gesturerecognition

N/A

N/A

N/A

CNN

Smart home control
Sign language
recognition
Hand gesture recognition

3m
N/A

Self-developed
Histogram ofOriented Gradients
Histogram ofOriented Gradients

1.2–3.5

2.5-3 mHistogram ofOriented Gradients
Skeleton Extraction

and software factors. Moreover, a task-dependent
software delay sums up with a hardware delay, whichleads to a decrease in device performance. This section
discusses a latency between a user input and an expected
system output, which differ for a particular task.

Latency, ms
KinectSkeleton detection
106-500
Human fall incident detection 300
Contactless hand tracking for
89-576
surgical robot
Skeleton position estimation
100
Skeleton detection
200-400
Kinect v2
Human gait analysis
29-127
Human gait analysis
200
Skeleton detection
66-100
IRB4400 industrial robot
20
(only
a
controlhardware latency)
Rat behavior tracking
80-126
3D content capture
65-67

Skeleton

Rf

Support Vector
MachineArtificial Neural
Network
AlexNet (CNN)

98.3

No

[24]

97.8

Yes

[25]

96.5

No

[26]

MediaPipe
GestureRecognition
RandomForest
Classifier + AdaBoosting
Dynamic Time
Warping
Fast Dynamic
Time Warping +CNN
Self-developed
Dynamic Time
Warping
Dynamic Time
Warping

95.7

Yes

[27]

93.7

Yes

[28]

92

[29]

90.8

Kinectembedded
Yes

87
86

Yes
Yes

[31]
[32]

76.7

Yes

[33]

[30]

In [41] authors emphasized a 20 ms hardware latency
as a minimum latency level of the device itself, whichcannot be reduced. While official manufacture statedlatency data for the first generation of Kinect is notavailable, the study [36] reports the latency value of 100
ms. Based on the surveyed in Table 2 papers weconcluded that the average latency for the first generation
Kinect is 267 ms, while the average latency for Kinect v2
is 123 ms. However, even though the Kinect v2's averagelatency significantly improved over the first generation,
its effectiveness still depends on a specific task in which
it is used and could achieve up to 200 ms.

Table 2. Latency Measurements of Kinect
Task

Acc, %

Ref
[35]
[36]
[37]

5. Conclusions

[38]
[39]

This paper presents an overview of Microsoft Kinect
sensor applications for gesture based control in robotics.
Kinect first generation and Kinect v2 related research
papers were analyzed in terms of range and accuracy in a
number of typical tasks that require a sensor with RBGD capabilities. Based on the survey we concluded that the
average overall latency of the first generation Kinect is
267 ms, while the average latency for Kinect v2 is 123
ms. However, even though the Kinect v2's averagelatency has improved over the first generation, its
effectiveness naturally depends on a specific task andcould achieve up to 200 ms.

[40]
[41]
[42]
[43]
[44]
[45]

Table 2 presents ab information about latency
measurements of both generations of Kinect devices
employed in various tasks. Note that reference papers ofTable 1 and Table 2 are different since Table 1 papers did
not specify a latency as they considered a different aspectof Kinect usage.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
297

Alexander Alexeev, Tatyana Tsoy, Edgar A. Martínez-García, Evgeni Magid

Acknowledgement
15.

This paper has been supported by the Kazan Federal
University Strategic Academic Leadership Program("PRIORITY-2030").

16.

References
1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

Y. Bai, On Motion Planning and Control for Partially
Differentially Flat Systems. Robotica, vol. 39, no. 4, 2021,
pp. 718-734.
B. Abbyasov, Ultrasound sensor modeling in Gazebo
simulator for diagnostics of abdomen pathologies. 2021
International Siberian Conference on Control andCommunications (SIBCON), Kazan, 2021, pp. 1-6.
L. Gavrilova, Facilitating a preparatory stage of real-world
experiments in a humanoid robot assisted English
language teaching using Gazebo simulator. 2020 13th
International Conference on Developments in eSystems
Engineering (DeSE), Liverpool, December 2020, pp. 222227.
E. Magid, Artificial intelligence based framework for
robotic search and rescue operations conducted jointly byinternational teams. Proceedings of 14th International
Conference on Electromechanics and Robotics“Zavalishin's Readings” ER (ZR), 2019, pp. 15-26.
J. Berger and L. Shuang, Review of Interfaces for
Industrial Human-Robot Interaction. Current Robotics
Reports, vol. 1, 2020, pp. 27-34.
M. Mustafin, Features of Interaction Between a Humanand a Gestures-Controlled Collaborative Robot in an
Assembly Task: Pilot Experiments. The 2023 International
Conference on Artificial Life and Robotics(ICAROB2023), February 2023, pp. 157-161.
X. Zhang, A Framework for Hand Gesture Recognition
Based on Accelerometer and EMG Sensors. IEEE
Transactions on Systems, Man, and Cybernetics - Part A:
Systems and Humans, vol. 41, no. 6, 2011, pp. 1064-1076.
S. Berman and H. Stern, Sensors for Gesture Recognition
Systems. IEEE Transactions on Systems, Man, andCybernetics – Part C: Applications and Reviews, vol. 42,
no. 3, 2012, pp. 277-290.
Kinect – Starting to Develop with Kinect, Available at:https://learn.microsoft.com/en-us/archive/msdnmagazine/2012/june/kinect-starting-to-develop-withkinect (accessed 29 November 2023).
Kinectfor
Windows,
Availableat:https://web.archive.org/web/20120103103154/http://www.kinectforwindows.org/ (accessed 10 May 2023).
A. Corti, A metrological characterization of the Kinect V2time-of-flight camera. Robotics and Autonomous Systems,
vol. 75, 2016, pp. 584-594.
L. Jamhoury, Understanding Kinect V2 Joints andCoordinate
System,2018,
Availableat:https://lisajamhoury.medium.com/understanding-kinectv2-joints-and-coordinate-system-4f4b90b9df16 (accessed
29 November 2023).
A. Latif, Design and Control Of Autonomous Robot Using
Gesture Based Intuitive Interaction. International Journalof Advanced Computer Systems and Software
Engineering, vol. 1, no. 4, 2020, pp. 7-11.
D. A. Katsay, T. D. Isupova, and A. D. Biryukov, Noncontact control system for anthropomorphic robot. II

17.

18.

19.

20.

21.

22.

23.

24.

25.

26.

27.

28.

29.

International scientific and technical conference "PromEngineering", May 2016. pp. 441-444.
Y. Ma, Deep Learning-Based Upper Limb Functional
Assessment Using a Single Kinect v2 Sensor. Sensors, vol.
20, no. 7, 2020,1903 p.
L. Vladutu and C. Marian, Framework for Posture andFace Recognition using Kinect an ambient-intelligence
method. WSEAS Transactions on Computers, vol. 22,
2023, pp. 13-24.
W. Kaczmarek, Industrial Robot Control by Means ofGestures and Voice Commands in Off-Line and On-LineMode. Sensors, vol. 20, no. 21, 2020, 6358 p.
W. Qi, Multi-Sensor Guided Hand Gesture Recognition
for a Teleoperated Robot Using a Recurrent Neural
Network. IEEE Robotics and Automation Letters, vol. 6,
no. 3, 2021, pp. 6039-6045.
B. Gabbasov, I. Danilov, I. Afanasyev, and E. Magid,
Toward a human-like biped robot gait: Biomechanicalanalysis of human locomotion recorded by Kinect-based
motion capture system. Proceedings of the 10th
International Symposium on Mechatronics and its
Applications (ISMA15), 2015, pp. 1-6.
B. Gabbasov, I. Danilov, I. Afanasyev, and E. Magid, ZMP
trajectory from human body locomotion dynamics
evaluated by Kinect-based motion capture system.
Proceedings of the 11th Joint Conference on Computer
Vision, Imaging and Computer Graphics Theory andApplications (VISAPP), 2016, pp. 160-166.
N. Pavón-Pulido, IoT Architecture for Smart Control of an
Exoskeleton Robot in Rehabilitation by Using a Natural
User Interface Based on Gestures. Journal of Medical
Systems, vol. 44, 2020, pp. 1-10.
P. Tu and C. Huang, Mechanical arm teleoperation control
system by dynamic hand gesture recognition based onkinect device. The Journal of Engineering, 2019, vol. 2019,no. 23, 2019, pp. 9110-9113.
N. Nikiforov, Pilot studies on Avrora Unior car-like robot
control using gestures. Electromechanics and Robotics:
Proceedings of 16th International Conference on
Electromechanics and Robotics" Zavalishin's Readings"(ER (ZR) 2021), April 2021, pp. 271-283.
T. Ma and M. Guo, Research on Kinect-based gesturerecognition. 2019 IEEE International Conference on
Signal Processing, Communications and Computing(ICSPCC), IEEE, September 2019, pp. 1-5.
B. Ganguly, Kinect Sensor Based Single Person Hand
Gesture Recognition for Man–Machine Interaction.
Computational Advancement in Communication Circuits
and Systems: Proceedings of ICCACCS 2018, 2020, pp.
139-144.
J. Liu, An Improved Kinect-Based Real-Time Gesture
Recognition Using Deep Convolutional Neural Networks
for Touchless Visualization of Hepatic Anatomical Mode.
Journal of Image and Graphics, vol. 7, no. 2, 2019, pp. 4549.
M. Harris, Applying hand gesture recognition for user
guide application using MediaPipe. 2nd International
Seminar of Science and Applied Technology (ISSAT
2021), Atlantis Press, 2021, pp. 101-108.
B. Hisham and A. Hamouda, Supervised learningclassifiers for Arabic gestures recognition using Kinect V2.
SN Applied Sciences, vol. 1, 2019, pp. 1-21.
W. Lv, Gesture recognition in somatosensory game via
Kinect sensor. Internet Technology Letters, 2021, 6 p.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
298

Powered by TCPDF (www.tcpdf.org)

An Overview of Kinecte

30. M. Pfitscher, Activity gesture recognition on kinect sensor
using convolutional neural networks and FastDTW for the
MSRC-12 dataset. Intelligent Systems Design andApplications: 18th International Conference on Intelligent
Systems Design and Applications (ISDA 2018), vol. 1,
Springer International Publishing, December 2020, pp.
230-239.
31. H. Fakhrurroja, Hand State Combination as Gesture
Recognition using Kinect v2 Sensor for Smart Home
Control Systems. 2019 IEEE International Conference on
Internet of Things and Intelligence System (IoTaIS), IEEE,
November 2019, pp. 74-78.
32. C. Sun, T. Zhang, and C. Xu, Latent support vector
machine modeling for sign language recognition with
Kinect. ACM Transactions on Intelligent Systems andTechnology (TIST), vol. 6, no. 2, 2015, pp. 1-20.
33. K. N. Krisandria, B. S. B. Dewantara, and
D. Pramadihanto, HOG-based Hand Gesture Recognition
Using Kinect. 2019 International Electronics Symposium(IES), IEEE, September 2019, pp. 254-259.
34. E. E. Shelomtsev, Solution to the problem of gesturerecognition for controlling the behavior of a robot. Youthand modern information technologies: collection of papers
from the 11th International Scientific and Practical
Conference of Students, Postgraduates and Young
Scientists, Tomsk Polytechnic University, November 2013,
pp. 278-280.
35. M. A. Livingston, Performance measurements for theMicrosoft Kinect skeleton. 2012 IEEE Virtual Reality
Workshops (VRW), IEEE, March 2012, pp. 119-120.
36. A. A. M. Bigy, Recognition of postures and Freezing ofGait in Parkinson's disease patients using Microsoft Kinect
sensor. 2015 7th International IEEE/EMBS Conference on
Neural Engineering (NER), IEEE, April 2015, pp. 731-734.
37. Y. Kim, Kinect technology for hand tracking control ofsurgical robots: technical and surgical skill comparison to
current robotic masters. Surgical endoscopy, vol. 28, 2014,
pp. 1993-2000.
38. S. Feng, R. Murray-Smith, and A. Ramsay, Position
stabilisation and lag reduction with Gaussian processes insensor fusion system for user performance improvement.
International Journal of Machine Learning andCybernetics, vol. 8, 2017, pp. 1167-1184.
39. S. Park, An approach to the reduction of time delay in a
networked performance. Proceedings of the Asia-Pacific
Advanced Network, vol. 36, 2013, pp. 91-96.
40. D. Guffanti, The accuracy of the Microsoft Kinect V2
sensor for human gait analysis. A different approach forcomparison with the ground truth. Sensors, vol. 20, no. 16,
2020, 4405 p.
41. A. Amini, K. Banitsas, and S. Hosseinzadeh, A New
Technique for Foot-Off and Foot Contact Detection in a
Gait Cycle Based on the Knee Joint Angle Using Microsoft
Kinect v2. 2017 IEEE EMBS International Conference on
Biomedical & Health Informatics (BHI), IEEE, February
2017, pp. 153-156.
42. T. Kure and S. Kasahara, Lighterbody: RNN based
anticipated virtual body makes you feel lighter. 2021 IEEE
Conference on Virtual Reality and 3D User Interfaces
Abstracts and Workshops (VRW), IEEE, 2021, pp. 561562.
43. A. Aalerud and G. Hovland, Evaluation of perception
latencies in a human-robot collaborative environment.
2020 IEEE International Conference on Robotics andAutomation (ICRA), IEEE, 2020, pp. 5018-5023.

44. J. Taufatofua, P. Pounds, and J. Wiles, PiRat: An
autonomous framework for studying social behaviour inrats and robots. 2018 IEEE/RSJ International Conferenceon Intelligent Robots and Systems (IROS), IEEE, October
2018, pp. 7601-7608.
45. K.
Zhou,
RGBD-based
Real-time
VolumetricReconstruction System: Architecture Design and
Implementation. 2022 IEEE International Conference on
Visual Communications and Image Processing (VCIP),
IEEE, December 2022, pp. 1-5.

Authors Introduction
Mr. Alexander Alexeev
He received his Bachelor’s degree inRegional Studies in 2022 from theIntitute of International Relations,Kazan Federal University (KFU),
Russia. He is currently a master student
in Institute of Information Technologies
and Intelligent Systems (ITIS), KFU,
RussiaProfessor Edgar A. Martínez-García
He is a full Professor at the Universidad
Autónoma de Ciudad Juárez, Mexico;founder and Head of the Robotics
Laboratory; leader of the Mechatronics
academic body at the Institute ofEngineering and Technology, since 2007. He obtained his
Ph.D. degree in Robotics Engineering from the Universityof Tsukuba, Japan (2005).
Ms. Tatyana Tsoy
In 2012 she graduated fromInternational Area Studies Master's
Degree Program at the University ofTsukuba. In 2023 completed a PhD in
Computer science and information
processes at IT IS, KFU.

Professor Evgeni Magid
He is a Head of Intelligent Robotics
Department, a founder, and a Head ofLaboratory of Intelligent Robotic
Systems (LIRS) at Kazan Federal
University, Russia. Senior member ofIEEE. He obtained Ph.D. degree in
Robotics Engineering from the
University of Tsukuba, Japan.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
299

An investigation on the impact of human-robot interactions during an autonomous obstacle
avoidance task

Riham Salman

Institute of Information Technology and Intelligent Systems, Kazan Federal University, Kazan, Russia
Shifa Sulaiman

Laboratory of Intelligent Robotics Systems (LIRS), Institute of Information Technology and Intelligent Systems, Kazan
Federal University, Kazan, Russia
Renata Islamova

Laboratory of Intelligent Robotics Systems (LIRS), Institute of Information Technology and Intelligent Systems, Kazan
Federal University, Kazan, Russia
Tatyana Tsoy

Laboratory of Intelligent Robotics Systems (LIRS), Institute of Information Technology and Intelligent Systems, Kazan
Federal University, Kazan, Russia
Email: RikSalman@stud.kpfu.ru, shifa_1910@it.kfu.ru, rislamova@it.kfu.ru, tt@it.kfu.ru

Abstract
The purpose of this research is to investigate how an interaction between humans and robots influences a safety of anautonomous obstacle avoidance task. The research collected and analyzed data from surveys and interviews using a
combination of quantitative and qualitative methodologies. The findings contributed to our understanding of acomplex interplay between a human-robot interaction, a perception, and a robot navigation safety. Based on thesefindings, the study proposed a number of recommendations for improving both physical and psychological safety
aspects of an autonomous robot navigation.
Keywords: Human-robot interaction, HRI, autonomous obstacle avoidance, navigation safety

into the complex dynamics of human-robot interaction(HRI), which encompasses the objective behavior of therobot, subjective experiences and perceptions of thehuman [14], [15].

1. Introduction
Autonomous robots are becoming an integral part ofdiverse human environments due to the advancements ofrobotic technologies from personal healthcare [1] toautomated manufacturing fields [2], [3]. The progression
is expected to lead to a future in which both humans androbots coexist and interact on a regular basis [4], [5].
However, the integration of autonomous robots into
human environments is not without significant
challenges, particularly when it comes to ensure safety
during autonomous obstacle avoidance tasks [6], [7].
Autonomous obstacle avoidance [8] and humanfollowing [9] is a critical feature for any autonomous
robot operating in a human-populated environment [10].
It is a capability that allows a robot to navigate throughan environment avoiding static and dynamic obstacles,
including humans. A robot which can precisely avoid
obstacles is considered safe [11]. Understanding andaddressing various aspects of safety require acomprehensive exploration of the relationship between
humans and robots [12], [13]. This involves indulging

The objective of this research is to provide an in-depth
examination of the relationship between humans androbots, focusing on its impact on the safety of
autonomous obstacle avoidance. Specifically, theresearch will investigate how humans interact withrobots that are capable of autonomous navigation. The
investigation will contribute to a deeper understandingof this critical aspect of autonomous robotics, providing
insights that could help improve the design and operationof future autonomous robots. It will also address gaps in
the existing body of knowledge, particularly in theintersection of autonomous navigation, HRI, andperceived safety [16]. Surveys and interviews wereconducted in a group of people, who had experience withrobotic environments. The inferences obtained throughthese surveys and interviews were used to conclude about
the current status of HRI in various fields [17].

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
300

Riham Salman, Shifa Sulaiman, Renata Islamova, Tatyana Tsoy

2. Methodology
The methodology outlines the research approaches thatwere utilized to investigate the relationship between HRI
and the safety of autonomous obstacle avoidance. The
research began with a comprehensive review of existing
literature from databases, journals, and conference
proceedings related to HRI, robotic safety, andautonomous obstacle avoidance. Following thecomprehensive literature review, the research focused on
the development of an experimental setup, which
consisted of a robot equipped with a capacity forautonomous navigation and obstacle avoidance. A
restaurant delivering food to customers using a mobile
robot (shown in Fig. 1(a) was selected for the survey. The
robot can move to assigned tables and deliver food asshown in Fig. 1(b).

(a)

(b)

Fig. 2 Different phases of the robot work: (a) Robot
waiting for collecting food items (b) Moving to towards
a destination table (c) Robot reached the destinationtable (d) A customer taking the food item from the robot
The final step in the methodology involved thevalidation of the findings. This was done through peerreviews and expert consultations, ensuring the researchconclusions were robust, reliable, and could stand up to
scrutiny. This step served as a quality check and ensured
that the research had been conducted in amethodologically sound manner. Further, it also
provided an opportunity for receiving recommendations
for future studies, thereby adding to the overall value andimpact of the research. This methodological approach
aimed to provide a comprehensive, multi-dimensional
understanding of the research topic, combining thestrengths of both empirical and qualitative researchmethods.

Fig. 1 Robot navigation (a) A food delivery robot (b)The robot delivering food to customers
The robot avoided collision with static and dynamic
obstacles using IR sensors fixed on the base of the robot.
Fig. 2 depicts the stages of a food delivery using themobile robot. Fig. 2 represents the robot waiting for the
food delivery near the restaurant kitchen. Fig. 2(a) and (b)represent a motion of the robot towards the assigned tableand destination table respectively. A customer receiving
the food items are shown in Fig. 2(d).
To obtain a human-centered perspective on the safety
and comfort of autonomous robots, surveys and semistructured interviews were conducted with individuals
who had interacted with the robot. Questions were
designed to explore various phases of human-robot
interaction, including social acceptability, perceived
safety, and trust in the robot’s autonomous decisionmaking capabilities. The customers and employees
working in the restaurant were participated in the survey.
The gathered data were subjected to rigorous analyses
using suitable statistical techniques. This critical processof data analysis allowed the transformation of raw data
into meaningful information, thereby offering bothquantitative and qualitative insights into the correlation
between HRI and the safety of autonomous obstacle
avoidance.

3. Results/findings
In this section, we present and analyze the results
gathered from our research and the insights obtained fromthe HRI studies. we conducted surveys and interviews
with participants after they observed or interacted withthe robot. A total of 60 participants attended the survey
and expressed their opinions and suggestions regarding
the coexisting of humans and robots inside a restaurant
environment. The qualitative data suggested a generally
positive perception of safety, comfort, and trust in the
robot’s autonomous capabilities. Approximately 90% of
the participants reported that they felt safe with therobot’s movements and obstacle avoidance capabilities.
Interestingly, a correlation was observed between the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
301

An investigation on the

robot’s behavior and participant responses. Smoother andslower robot movements were associated with increased
comfort and trust ratings from the participants. About
80% of the participants conveyed that the robot fooddelivery was faster compared to human food delivery
services. However, some of the participants areconcerned about the precision of delivery and quoted
about interaction difficulties they faced during food order
updating.

obstacles present in a restaurant setting. In order toprevent collisions when moving from the food pickup
location to the destination table, the robot needs to be
employed with an effective collision avoidance approach.
By utilizing proper sensors and controllers, the robot cansteer clear of obstructions and prevent crashes during the
motions. Reserving separate paths for robots canminimize chances of collisions with obstacles. To help
users comprehend how the robot is moving, the robot canalso show path details on a screen or verbally during its
motion.

The perception of risk and social acceptability
associated with the autonomous robot were also includedin the survey. We interpreted that in a populated
environment, participants perceived the robot as lessrisky compared to a controlled environment, contrary to
our expectations. This might be attributed to the social
dynamics of a populated setting, where humans provide
an additional layer of unpredictability that makes therobot’s autonomous capabilities more evident andappreciated. The social acceptability of the robot was
rated highly by 82% of participants, signifying the
successful integration of social robotics principles in its
design and operation. However, 27% of the customers
preferred contactless delivery. 70% of the participantsconsidered robot usage inside a restaurant as an ecofriendly option. 20% of the participants chose human
workers in place of autonomous rover delivery due tosafety concerns. Most participants concerned about the
robot malfunctioning, and navigational errors.

The study’s findings also had several theoretical
implications. They reinforced the theories of autonomy,social robotics, proxemics, risk perception, and planned
behavior [15]. The robot’s autonomous performancealigned well with the theory of autonomy in robotics. The
feedback from participants suggested that social robotics
theories were applicable in real-world environments,
reinforcing the necessity of developing robots that
respect human social norms.
5. Conclusion
In conclusion, the research undertaken successfully
addressed the primary goals of investigating therelationship between HRI and the safety of autonomous
obstacle avoidance. The investigation revealed that theinteraction between humans and robots is multi-faceted.
It emphasized that the concept of safety in autonomous
obstacle avoidance is not limited to the physical ability of
the robot to navigate without causing collisions or
accidents. It also heavily depends on the subjective
feelings of safety, comfort, and trust experienced by thehumans who interact with or are in the vicinity of theserobots. Therefore, the importance of considering bothobjective and subjective factors when evaluating thesafety of autonomous robots in environments shared withhumans was strongly underscored. This research also
pointed towards potential areas for future study,
demonstrating that the quest for knowledge in thisdomain is far from exhausted. For instance, it suggested
the need for further exploration into how different design
elements and behavioral parameters of the robot might
influence human perception and interaction. This could
include aspects such as the robot’s size, color, speed,
motion patterns, and communication methods. Research
into these directions could yield interesting results thatmight further refine the approach to designing anddeploying autonomous robots in human environments.

4. Discussion
The study was designed to investigate the relationshipbetween HRI and the safety of autonomous obstacle
avoidance, contributing to the growing body ofknowledge in this field. The multi-dimensional
methodological approach, combining both empirical andqualitative research methods, was integral in producing acomprehensive understanding of the research topic. The
survey results regarding autonomous navigation andobstacle avoidance revealed that the robot was successful
in navigating the environments and avoiding obstacleswith a high success rate.
Results from the HRI revealed a generally positive
perception of the robot’s autonomous abilities, showing
trust in the robot’s decision-making and overallsatisfaction with their interaction. These findings
provided valuable insights into the subjective aspects ofrobot safety, complementing the quantitative dataobtained. Risk perception and social acceptability werealso significant factors in our study. Results suggested
that participants perceived the autonomous robot as safe
and socially acceptable, even in densely populated
environments.

Acknowledgements
This work was supported by the Kazan Federal
University Strategic Academic Leadership Program("PRIORITY-2030").

When performing HRI activities, robots must be
equipped with adaptive abilities to prevent accidents andtask incompletions. There are a lot of static and dynamic©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
302

Powered by TCPDF (www.tcpdf.org)

Riham Salman, Shifa Sulaiman, Renata Islamova, Tatyana Tsoy

14. K. Dautenhahn, Methodology & themes of human-robot
interaction: A growing research field, International Journalof Advanced Robotic Systems, vol. 4(1), 2007, pp. 15.
15. R. Stock-Homburg, Survey of emotions in human–robot
interactions: Perspectives from robotic psychology on 20
years of research, International Journal of Social Robotics,
vol. 14(2), 2022, pp. 389-411.
16. K.-H. Hsia, J.-H. Cai, S.-L. Pai and E. Magid, Control
System for Maintaining Safe Following Distance while
Driving, Journal of Robotics, Networking and Artificial
Life, vol. 5(4), 2019, pp. 261–264.
17. M. Mustafin, E. Chebotareva, H. Li and E. Magid,
Experimental Validation of an Interface for a HumanRobot Interaction Within a Collaborative Task, InInternational Conference on Interactive Collaborative
Robotics, Cham: Springer Nature Switzerland, 2023,
pp. 23-35.

References
1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

E. Magid, A. Zakiev, T. Tsoy, R. Lavrenov andA. Rizvanov, Automating pandemic mitigation, Advanced
Robotics, 35(9), 2021, pp. 572-589.
M. Shamout, R. Ben-Abdallah, M. Alshurideh, H. Alzoubi,
B. A. Kurdi and S. Hamadneh, A conceptual model for the
adoption of autonomous robots in supply chain andlogistics industry, Uncertain Supply Chain Management,
10(2), 2022, pp. 577-592.
M. Mustafin, E. Chebotareva, E. Magid, H. Li and
E. Martinez-Garcia, Features of Interaction Between a
Human and a Gestures-Controlled Collaborative Robot inan Assembly Task: Pilot Experiments, International
Conference on Artificial Life and Robotics (ICAROB
2023), 2023, pp. 158-162.
K. Merckaert, B. Convens, C. J. Wu, A. Roncone, M.
M. Nicotra and B. Vanderborght, Real-time motion
control of robotic manipulators for safe human–robot
coexistence,
Robotics
andComputer-Integrated
Manufacturing, 2022, pp. 73.
S. R. Schepp, J. Thumm, S. B. Liu and M. Althoff, Sara:
A tool for safe human-robot coexistence and collaboration
through reachability analysis, in 2022 International
Conference on Robotics and Automation (ICRA), 2022, pp.
4312-4317.
A. Dobrokvashina, S. Sulaiman, A. Zagirov,
E. Chebotareva, K.-H. Hsia and E. Magid, Human robot
interaction in collaborative manufacturing scenarios:
Prospective cases, in 2022 International Siberian
Conference on Control and Communications (SIBCON),
2022, pp. 1-6.
A. C. Simões, A. Pinto, J. Santos, S. Pinheiro and
D. Romero, Designing human-robot collaboration (HRC)
workspaces in industrial settings: A systematic literature
review, Journal of Manufacturing Systems, vol. 62, 2022,
pp. 28-43.
E. Magid, R. Lavrenov and A. Khasianov, Modified
Spline-based Path Planning for Autonomous Ground
Vehicle. In ICINCO (2), July 2017, pp. 132-141.
E. Chebotareva, E. Magid, A. Carballo and K.-H. Hsia,
Basic User Interaction Features for Human-Following
Cargo Robot TIAGo Base, Proceedings of 13th
International Conference on Developments in eSystems
Engineering (DeSE), 2020, pp. 206-211.
E. Chebotareva, R. Safin, K.-H. Hsia, A. Carballo and
E. Magid, Person-Following Algorithm Based on Laser
Range Finder and Monocular Camera Data Fusion for a
Wheeled Autonomous Mobile Robot, Lecture Notes in
Computer Science (including subseries Lecture Notes in
Artificial Intelligence and Lecture Notes in
Bioinformatics), vol. 12336, 2020, p. 21-33.
R. Lavrenov, F. Matsuno, E. Magid, Modified splinebased navigation: Guaranteed safety for obstacle
avoidance, Lecture Notes in Computer Science (inc.
subseries Lecture Notes in Artificial Intelligence andLecture Notes in Bioinformatics), vol. 10459, 2017,
pp. 123-133.
A. Zacharaki, I. Kostavelis, A. Gasteratos and I. Dokas,
Safety bounds in human robot interaction: A survey,
Safety science, 2020, pp. 127.
M. Rubagotti, I. Tusseyeva, S. Baltabayeva, D. Summers
and A. Sandygulova, Perceived safety in physical human–robot interaction—A survey, Robotics and Autonomous
Systems, 2022, pp. 151.

Authors Introduction
Ms. Riham Salman
In 2021, she graduated from the
University of Tishreen, Syria. Since
2022, she has been a master's student
at the Institute of Information
Technology and Intelligent Systems(ITIS), Kazan Federal University(KFU), Russia.
Dr. Shifa Sulaiman
In 2013, she received her Master's
degree in Machine Design from
Mahatma Gandhi University, India. In
2013-2017 worked as an AssistantProfessorinvariousIndianEngineering Institutes. She earned her
PhD from National Institute ofTechnology, Calicut, India, specializedin Humanoid Robotics in 2023. She is currently working
as a Research Associate at LIRS, ITIS, KFU, Russia.
Ms. Renata Islamova
She is currently pursuing bachelor's
degree program in linguistics as an
English and German interpreter at theInstitute of International Relations,
KFU, Russia. She is also working as alaboratory assistant at the Laboratory ofIntelligent Robotic Systems (LIRS) at
KFU
Ms. Tatyana Tsy
In 2012 she graduated fromInternational Area Studies Master's
Degree Program at the University ofTsukuba. In 2023 completed a PhD in
Computer science and information
processes at ITIS, KFU, Russia.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
303

A Comparative Analysis of Object Detection Methods for Robotic Grasping
Nikita Kolin

Intelligent Robotics Department, Kazan Federal University, 35 Kremlevskaya St., Kazan, 420008, Russian Federation
Elvira Chebotareva

Intelligent Robotics Department, Kazan Federal University, 35 Kremlevskaya St., Kazan, 420008, Russian Federation
Email: nikitakolin7@gmail.com, Elvira.Chebotareva@kpfu.ru

Abstract
The objects grasping is one of the fundamental robotic problems. Accurate and efficient real-time object detection is
crucial for successful grasping in robots equipped with monocular vision. Deep machine learning has made significant
progress in solving problems of object detection and image segmentation. At the same time, classical computer vision
methods do not lose their relevance and can also be used for these tasks. In this research, we conduct a comparative
analysis of the effectiveness the YOLOv8-seg neural network model versions for solving the image segmentation
problem with classical segmentation methods. The obtained results allowed us to formulate some recommendations
on the choice of a particular method for object detection depending on the surrounding environment conditions.
Keywords: Robotic grasping, Robotic arm, Object detection, Object segmentationin the review [9]. The overview of different approaches
to computer vision-based robotic grasping presented in[10] and [11]. The effectiveness of deep machine learningin object detection on RGB-D images for robotic
grasping is well demonstrated in the work [12]. At thesame time, detecting objects in RGB images seems to bea much more difficult task, since such images do notcontain depth information. Examples of works exploring
the challenges of robotic grasping using monocularvision and RGB images include works [13] and [14].

1. Introduction
Robotic grasping is one of the fundamental tasks ofrobotics, relevant to all its branches [1], including
industrial [2], service [3] and social robotics [4].
Grasping can be particularly important in collaborative
robotics, when a robot and a person use common tools or
perform joint assembly [5].
Obtaining information about the shape of the captured
objects is an important stage in the implementation ofgrasping [6]. The shape of objects can be determinedusing various sensors, including cameras and lidars. At
the same time, solutions that allow detect grasping
objects using monocular cameras are also of practicalinterest. Deep machine learning has made significant
progress in solving problems related to object detection[7] and image segmentation [8]. However, classicalcomputer vision methods do not lose their relevance andcan also be used for these tasks. In this paper, we present
the results of our experiments aimed at comparingclassical object detection methods with object detection
and segmentation using the YOLOv8-seg models family.
The aim of our research is to evaluate the accuracy andeffectiveness of various methods for detecting object
contours in robotic grasping.

Often, the effectiveness of grasping area detection
methods is evaluated using one of the popular datasets,such as Cornel dataset [15] and Jacquard dataset [16].
The use of publicly available datasets allows comparing
the effectiveness of different approaches. According to
the data from the “Paper with Code” platform [17] theaccuracy of various grasping methods based on machinelearning, estimated with the help of Cornel Dataset wasmade from 88% in 2016 for Multi-Modal Grasp
Predictor [18] to 98.2% in 2021 for Ainetter and
Fraundorfer CNN-based architecture [19] and 98.9% for
Efficient Grasping model by Cao et. al [20]. At the sametime, the accuracy estimates on the Jacquard dataset for
the Ainetter and Fraundorfer model [19] was 92.95%, andfor the Efficient Grasping model by Cao et. al [20] was
95.6%.

2. Related WorkAs it can be seen, estimates of the accuracy of different
methods may vary from the quality of the test dataset. In
addition, although public datasets provide an overview ofthe effectiveness of a given method, these datasets do notalways take into account the peculiarities of objects andIn the past few years, there has been a lot of research
reported on robotic grasping problems. Some approaches
to solving the problem of robotic grasping are presented©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
304

Nikita Kolin, Elvira Chebotareva

environments encountered in real cases. Therefore,
datasets that are close to the real conditions of a particular
task are always of special interest. In this research work
we will make a comparative analysis of various methodsof objects detection on own dataset, adapted to the real
object grasping task. As an example of a model thatprovides acceptable accuracy along with the highinference speed and usability we use the YOLOv8-seg
model for object segmentation. The YOLOv8-seg modelis based on variant of the U-Net architecture [21]. The
YOLO models family based on hybrid convolutional
neural network architecture [22], [23] and often used inrobotics.

training dataset through rotation and perspective
distortion.
3.3 Object detection methods
For our experiments, we used the YOLOv8n-seg and
YOLOv8s-seg models pre-trained on the COCO dataset
by Microsoft. YOLOv8n-seg and YOLOv8s-seg models
trained on a set of 368 images of size 640×480.
To extract the object’s contours we used two classicalmethods. The first method is based on the conversion to
HSV space and channel thresholding. And the second
method uses the Canny edge detector, as well as auxiliary
erosion and dilation transformations. The classificationof the extracted contours is based on a single feature − theaspect ratio of the bounding box.

3. Materials and Methods
3.1. Object detection problem statement
We model a situation in which the working cell of arobotic manipulator is equipped with a single monocularcamera fixed above the working area. During theexperiment, the camera does not change its position. Thiscamera transmits image of the table area, on which threeobjects can be placed at random: pliers, screwdrivers andPIR sensor. We assume that the objects are homogeneous,
so we can assume that the center of mass of each object
coincides, or is close, with its geometric center. Therefore,
the optimal gripping points must also be near thegeometric center of each object.

3.2. Evaluation
Each method of object detection and segmentation was
evaluated using two commonly used metrics: the Jaccard
index (1) − also known as the IoU (Intersection overUnion) metric, and the Dice coefficient (2) − also known
as the F1-score metric. These metrics are based on theformulas (1) и (2). In formulas (1) and (2) A is the set ofpixels belonging to the real object in the image, B is theset of pixels of the result of segmentation.

Thus, in the context of the task of detecting objects fortheir robotic grasping, we need to determine the object
class (pliers, screwdriver, or PIR sensor), the contours of
the object, the geometric center of the object (the centerof mass of the contour) and the orientation of the object.
Note that in order to calculate the center and orientationof an object in this case it is enough to define its contour.

𝐴∩𝐵
𝐴∪𝐵

(1)

|𝐴 ∩ 𝐵||𝐴| + |𝐵|

(2)

𝐼𝑜𝑈 =
𝐹1 = 2For each method, we also measured the inference time
per frame in the test video and calculated the inference
speed as the number of processed frames per second(FPS).

3.2 Datasets
For our experiments, we have prepared a set of 100
images with resolution 640480 pixels containing threedifferent target objects: pliers, screwdriver and PIR
sensor. The choice of objects is determined by the context
of collaborative tasks in which it is planned to use thisrobot in the future. In addition, these objects havedifferent shapes and colors, which can help identify
potential issues with specific object detection methods.
The images were taken under various lighting conditions
and against different backgrounds.

4. Experiment Results and Discussion
At the first stage of the experiment, we evaluated theaverage values of IoU and F1-score for 100 images fromour dataset. Then, at the second stage of our experiment,
we conducted the evaluation using 20 images withsignificantly different backgrounds compared to thebackgrounds of the images in the training dataset. The
results of these stages are presented in Table 1 and Table
2.

However, to test the hypotheses of our study, we also
prepared 20 images with backgrounds that differed fromthose in the training dataset images. A total of 165
annotated images were prepared for model training, with
132 included in the training dataset, 17 images in thevalidation dataset, and 16 in the test dataset. To improve
the quality of training, 236 images derived from theoriginal 132 training dataset images were added to theTable 1. The object detection methods accuracy obtained onimages with conditions similar to the training dataset.
MethodIoU
F1-scoreThresholding
0.4112
0.4658
Canny edge detector
0.17200.2257
YOLOv8n-seg
0.82480.8811YOLOv8s-seg
0.86120.9185©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
305

A Comparative Analysis of Object

Table 2. The object detection methods accuracy obtained onimages with conditions differing from the training dataset.
MethodIoU
F1-scoreThresholding
0.80470.8771
Canny edge detector
0.02520.0469YOLOv8n-seg
0.3186
0.3477
YOLOv8s-seg
0.22990.2535

background not included in the training dataset. The
method based on Canny edge detector also did not give
positive results.

Thus, in the case where the images background waspresent in the training dataset, the best result was
obtained by YOLOv8s-seg and YOLOv8n-seg models.
However, the YOLOv8n-seg model showed an
expectedly lower result compared to YOLOv8s-seg
model. The thresholding-based method showedsignificantly lower results than the YOLOv8-seg models,
but exceeded the Canny edge detection method.
Examples of object detection and segmentation using
models YOLOv8n-seg and YOLOv8s-seg are presented
in Fig. 1 and Fig. 2.

Fig. 3. The example of object contour detection using
a classic threshold-based method in cases where
YOLOv8n-seg and YOLOv8s-seg models failed.
Table 3 presents the results of FPS evaluation on thetest video for all the considered methods. The best
inference speed result belongs to the method based on the
Canny edge detector, followed by the threshold-based
method in second place, and the YOLOv8s-seg model
demonstrated the worst result.
Table 3. Results of FPS evaluation for the considered
methods.
MethodThresholding
Canny edge detector
YOLOv8n-seg
YOLOv8s-seg

Fig. 1. The example of successful recognition of
all objects using the YOLOv8n-seg model.
However, in cases where the background on which theobjects are located differed from the backgrounds in the
training images, the method based on thresholdprocessing of channels in the HSV color space showed
better results compared to both the YOLOv8s-seg and
YOLOv8n-seg models, as well as the method based on
Canny edge detector. Fig. 3 presents an example of object
detection and segmentation on an image with abackground different from the training dataset. In thisinstance, the YOLOv8s-seg and YOLOv8n-seg models
failed to detect object contours on the complex

FPS
21.8
32.6
4.2
2.1

5. Conclusion
The obtained results allowed us to formulate somerecommendations on the choice of a particular method for
object detection depending on the surrounding
environment conditions. In cases where the scene is notoverloaded with a large number of objects, theenvironmental conditions remain stable, and the objects
significantly differ from the background and haverelatively homogeneous texture, classical methods can bequite successfully applied to extract the contours of thetarget objects. Moreover, in contrast to machine learning
methods, these methods do not require the preliminary
collection of a large amount of training data andannotation, and they also characterized by higher
inference speed. However, if the environmental
conditions change dynamically, these methods become
practically inapplicable. In this case, the optimal solution
would be to train a previously pre-trained neural networkmodel. However, the training dataset must be close to thereal conditions in which recognition is planned, inparticular, it is necessary that the background on which
the target objects are located is the same as in realFig. 2. The example of successful recognition of all
objects using the YOLOv8s-seg model.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
306

Powered by TCPDF (www.tcpdf.org)

Nikita Kolin, Elvira Chebotareva

conditions. Otherwise, the quality of detection andsegmentation is significantly reduced.
Acknowledgements

14.

This paper has been supported by the Kazan Federal
University Strategic Academic Leadership Program(“PRIORITY-2030”).

15.

References

16.

1.
2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

R. Hodson, How Robots are Grasping the Art of Gripping.
Nature, vol 557, 2018, pp. 23–25.
M. Mustafin, E. Chebotareva, E. Magid, H. Li and E.
Martinez-Garcia, Features of Interaction Between a
Human and a Gestures-Controlled Collaborative Robot inan Assembly Task: Pilot Experiments, The 2023
International Conference on Artificial Life and Robotics(ICAROB), Oita, Japan, 2023, pp. 158–162.
K. Khusnutdinov, A. Sagitov, A. Yakupov, R. Lavrenov,
E. A. Martinez-Garcia, K.-H. Hsia and E. Magid,
Household Objects Pick and Place Task for AR-601M
Humanoid Robot. Lecture Notes in Computer Science,
11659, 2019, p. 139-149.
K. Khusnutdinov, A. Sagitov, A. Yakupov,
R. Meshcheryakov, K.-H. Hsia, E. A. Martinez-Garcia and
E. Magid, Development and Implementation of Grasp
Algorithm for Humanoid Robot AR-601M. Proceedings ofthe 16th International Conference on Informatics inControl, Automation and Robotics, 2019, pp. 379-386.
M. Mustafin, E. Chebotareva, H. Li and E. Magid,
Experimental Validation of an Interface for a HumanRobot Interaction Within a Collaborative Task. In
International Conference on Interactive Collaborative
Robotics (ICR 2023), Baku, Azerbaijan, 2023, pp. 23–25.
R. Abdulganeev, R. Lavrenov, R. Safin, Y. Bai and E.
Magid, Door handle detection modelling for Servosila
Engineer robot in Gazebo simulator. Siberian Conference
on Control and Communications, (SIBCON 2022), 2022,
p. 1-4.
V. Myrzin, T. Tsoy, Y. Bai, M. Svinin and E. Magid,
Visual data processing framework for a skin-based humandetection. In Interactive Collaborative Robotics: 6thInternational Conference, ICR 2021, Proceedings 6,
September 2021, pp. 138-149.
Y. Yu, C. Wang, Q. Fu, R. Kou, F. Huang, B. Yang, T.
Yang and M. Gao, Techniques and Challenges of Image
Segmentation: A Review. Electronics 12, no. 5, 2023, pp.
1199.
L. Wang, Z. Zhang, J. Su and Q. Gu, Robotic Autonomous
Grasping Technique: A Survey. 2021 5th Asian
Conference on Artificial Intelligence Technology(ACAIT), Haikou, China, 2021, pp. 287–295.
K. Kleeberger, R. Bormann, W. Kraus, and M. F. Huber
Kleeberger, A Survey on Learning-Based Robotic
Grasping. Current Robotics Reports 1, 2020, pp. 239–249.
Z. Xie, X. Liang and R. Canale, Learning-Based Robotic
Grasping: A Review. Frontiers in Robotics and AI 10,
2023, p. 1038658.
Q. Zhang, J. Zhu, X. Sun and M. Liu, HTC-Grasp: A
Hybrid Transformer-CNN Architecture for Robotic Grasp
Detection. Electronics 12, no. 6, 2023, p. 1505.
C. Veiga Almagro, R.A. Muñoz Orrego, Á. García
González, E. Matheson, R.M. Prades, M. Di Castro and

17.
18.

19.

20.

21.
22.

23.

M.F. Pérez, (MARGOT) Monocular Camera-Based Robot
Grasping Strategy for Metallic Objects. Sensors 23, no. 11,
2023, p. 5344.
W. Prew, T. Breckon, M. Bordewich and U. Beierholm,
Improving Robotic Grasping on Monocular Images Via
Multi-Task Learning and Positional Loss, 2020,
https://arxiv.org/abs/2011.02888
I. Lenz, H. Lee and A. Saxena, Deep learning for detecting
robotic grasps. The International Journal of Robotics
Research 34, 2013, pp. 705–724.
A. Depierre, E. Dellandréa and L. Chen, Jacquard: A Large
Scale Dataset for Robotic Grasp Detection. 2018
IEEE/RSJ International Conference on Intelligent Robots
and Systems (IROS), Madrid, Spain, 2018, pp. 3511–3516.
Paper with Code, https://paperswithcode.com/
S. Kumra and C. Kanan, Robotic grasp detection using
deep convolutional neural networks. 2017 IEEE/RSJ
International Conference on Intelligent Robots andSystems (IROS), Vancouver, BC, Canada, 2017, pp. 769–
776.
S. Ainetter and F. Fraundorfer, End-to-end Trainable Deep
Neural Network for Robotic Grasp Detection and
Semantic
Segmentation
from
RGB,
2022,
https://arxiv.org/abs/2107.05287
H. Cao, G. Chen, Z. Li, J. Lin and A. Knoll, Lightweight
Convolutional Neural Network with Gaussian-based
Grasping Representation for Robotic Grasping Detection,
2021, https://arxiv.org/abs/2101.10226
Ultralytics
YOLOv8
Tasks,
Ultralytics
Inc,
https://docs.ultralytics.com/tasks/
J. Redmon, S. Divvala, R. Girshick and A. Farhadi, You
Only Look Once: Unified, Real-Time Object Detection. In
Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition (CVPR), Las Vegas, NV, USA,2016, pp. 779–788.
J. Terven and D. Cordova-Esparza, A Comprehensive
Review of YOLO: From YOLOv1 and Beyond, 2023,
https://arxiv.org/abs/2304.00501.

Authors Introduction
Mr. Nikita Kolin
He received his Bachelor's degree inKazan Federal University, Russia, in
2022. Currently he is a Master's student at
the Institute of Information Technologies
and Intelligent Systems, Kazan Federal
University, Russia.
Assistant Professor Elvira Chebotareva
She received her PhD in physics andmathematics from Kazan Federal
University. She is currently an assistantprofessor in Laboratory of Intelligent
Robotic Systems (LIRS) at Kazan Federal
University, Russia.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
307

Vision-based autonomous navigation for medical examination using a UR3e manipulator
Bulat Abbyasov

Intelligent Robotics Department, Kazan Federal University, 420111 Kazan, Russian Federation
Aidar Zagirov

Intelligent Robotics Department, Kazan Federal University, 420111 Kazan, Russian Federation
Timur Gamberov

Intelligent Robotics Department, Kazan Federal University, 420111 Kazan, Russian Federation
Hongbing Li

Department of Instrument Science and Engineering, Shanghai Jiao Tong University, Shanghai, Minhang, 200240, China
Evgeni MagidIntelligent Robotics Department, Kazan Federal University, 420111 Kazan, Russian Federation
E-mail: abbyasov@it.kfu.ru, ai.zag@it.kfu.ru, TRGamberov@stud.kpfu.ru, lihongbing@sjtu.edu.cn, magid@it.kfu.ru

Abstract
Medical robotics is an emerging field of robotics within a healthcare sector. This interdisciplinary field focuses on
prototyping, building and developing advanced robots for numerous clinical application. Modern robotic technologies
have a tremendous potential for performing medical examination procedures such as ultrasonography using robotic
manipulators that act in an autonomous manner. A manipulator navigation plays a key role in a safe and efficient
exploration of a human body. This paper presents a development of a vision-based autonomous navigation system fora 6-DOF UR3e robotic arm. The developed system is based on a 3D point cloud and uses MoveIt for path planning.
Gazebo was used as a simulation framework to validate the navigation system.
Keywords: Medical robotics, Manipulator, Autonomous navigation, ROS, MoveIt, Gazebo

An autonomy enables an unskilled human worker toeasily configure a robotic system in a setup phase before
performing a medical procedure. Autonomous
manipulators execute tasks consistently without avariability that can be introduced by human operators[16]. This consistency is crucial in medical examination
scenarios where uniformity is essential. Manipulator
navigation plays a key role in a safe and efficientautonomous exploration of a human body [17]. The
exploration refers to autonomously move, position, andcontrol a trajectory of a manipulator within a designatedspace. Vision-based manipulator navigation is anapproach of guiding and controlling a robotic
manipulator's movement using visual information. A
robotic system relies on cameras or other vision sensors
to perceive and interpret an environment [18].

1. Introduction
The healthcare industry has been making a notable
progress in recent years with an advancement of robotics
technologies [1], [2]. Due to COVID-19 pandemic anautomation of medical procedures employing robotics
technologies became critical [3], [4]. Robots areprogressively being employed for a broad spectrum oftasks that include surgery [5], rehabilitation [6],diagnostics [7], and drug delivery [8], offering precision,
efficiency and remote operational capabilities [9].
Modern robotic technologies have a tremendous potential
for performing medical examination procedures using
robotic manipulators [10] that employ various endeffectors to provide additional functionalities within
specific tasks. Nowadays, robotic arms became a major
focus for medical research groups [11]. Robotic
manipulators feature a lightweight reconfigurable arm
design, high precision actuators and motion control
systems. These robots are used in medical scenarios that
require a high degree of precision and quality of a task
performance, e.g., tissue suturing [12], minimallyinvasive surgery [13], [14], and ultrasonography [15].

This paper presents a development of a vision-based
autonomous navigation system for medical examination
using a 6 degrees of freedom (DoF) UR3e robotic arm.
The developed system is based on 3D point cloud data.
3D point clouds are produced by a depth camera, whichmeasure physical dimensions of a human body. MoveIt
package [19] was used for motion planning, manipulation,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
308

Bulat Abbyasov, Aidar Zagirov, Timur Gamberov, Hongbing Li, Evgeni Magidand control of the robotic arm. Gazebo [20] was used asa simulation framework to validate the navigation system.

sensor_msgs/Image. Depth data captured by the virtual
Kinect camera is used by a 3D point cloud processing
module. Fig. 2 shows a 3D point cloud of an abdomen.

2. System setup
A simulation environment in Gazebo contains a UR3emanipulator model. UR3e is mounted on a supportingbase that allows to effectively cover a table used for
examinations. The Kinect camera model with a depth
camera plugin captures visual and depth data of a humanbody. A human body model is placed on the examinationtable within the manipulator workspace and the Kinect
camera range. UR3e is equipped with a transducer device
used as an end-effector. The virtual environment is
depicted in Fig. 3.

A UR3e manipulator (Fig. 1, left), manufactured by
Universal Robots, stands as a suitable robotic system for
automating low-weight processing tasks. The capability
to grab and move a medical tool is facilitated by its
payload capacity of 3 kg, providing a sufficient control tonavigate an end-effector. Flexible motion planning isachieved using the UR3e arm's capability for 360 degreesof rotation in all its joints. For a virtual robot setup a
ROS-based UR3/UR3e controller with simulation in
Gazebo was used (Fig. 1, right).

Fig. 1. UR3e robotic arm: a real robot (left) and avirtual manipulator model (right)

Fig. 3. Gazebo environment: a human, an
examination table and UR3e manipulator, that
explores a chest.

3. Navigation
A proposed navigation system consists of threemodules: a 3D point cloud processing, a path planningand a motion planning. The first module serves as afoundation for a spatial perception. The second module,
which is responsible for waypoint generation, takes theprocessed point cloud data and calculates series of
feasible waypoints for navigation. The third module
focuses on translating the planned paths into executablemotion commands for the manipulator.

Fig. 2. Depth data: a 3D point cloud (blue) of a regionof interest
A depth sensor is a type of imaging device that captures
depth data along with visual data. Unlike traditional
cameras that record color and intensity information,
depth cameras provide additional details about a distancefrom a camera to an object. In a Gazebo simulation
environment, integrating a Kinect camera model involves
configuring a depth camera plugin. ROS/Gazebo
provides a package gazebo_ros_pkgs that contains
wrappers and tools for using ROS with Gazebo. The
package includes gazebo_ros_openni_kinect plugin thatserves as a driver for interfacing a Kinect camera with
Gazebo simulation framework. The Gazebo camera
plugin publishes depth data on a ROS topic of type

A point cloud is a collection of data points defined bytheir coordinates in 3D space. Each point represents a
specific position in an environment and may include
additional information, such as color or intensity. The
module processes raw 3D point cloud data obtained fromthe depth camera. This step involves filtering input depth
data and removing invalid points. Point Cloud Library(PCL) library [21] was utilized to process raw 3D point
cloud data and create a digital representation of a humanbody’s specific region of interest.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
309

Vision-based autonomous navigation

Waypoint generation determines a set of points that
represents a desired path for the manipulator. The module
is responsible for generating waypoints and utilizes theprocessed 3D point cloud model as an input. Normal
vectors of points are employed to calculate a rotation
matrix for the end-effector. The output is a sequence offeasible waypoints for motion planning (Fig. 4, bottom).

4. Conclusions
This paper presented a vision-based autonomous
navigation system for medical examination using a 6DOF UR3e robotic arm. The developed system relies on
depth-sensing cameras and utilizes a 3D point cloud.
MoveIt motion planning framework was used for path
planning and control of the robotic arm. The validation of
the developed navigation system was conducted in
Gazebo simulation framework. As a part of our future
work, we plan to validate the proposed navigation systemon a real UR3e robot. To achieve this, we will use a real
Kinect depth camera and a human mannequin.

A motion planning process involves converting a
trajectory into a series of motion commands. The motion
planning module uses MoveIt framework to plan amotion for the robotic system, validate it and then executethe motion plan (Fig. 4, top). MoveIt uses inverse
kinematics solvers for determining joint configurations
and collision checking to ensure safe motion of UR3e.

Acknowledgements
This work was supported by the Russian Science
Foundation (RSF), project ID 24-29-00564.
References
1.

J. Holland, L. Kingston, C. McCarthy, E. Armstrong, P.
O’Dwyer, F. Merz and M. McConnell, Service robots in
the healthcare sector. Robotics, 10(1), 2021, p. 47.
2. M. Kyrarini, F. Lygerakis, A. Rajavenkatanarayanan, C.
Sevastopoulos, H. R. Nambiappan, K. K. Chaitanya and F.
Makedon, A survey of robots in healthcare. Technologies,
9(1), 2021, p. 8.
3. E. Magid, A. Zakiev, T. Tsoy, R. Lavrenov and A.
Rizvanov, Automating pandemic mitigation. Advanced
Robotics, vol. 35(9), 2021, p. 572-589.
4. A. Khamis, J. Meng, J. Wang, A. T. Azar, E. Prestes, A.
Takacs, I. J. Rudas and T. Haidegger, Robotics andintelligent systems against a pandemic. Acta Polytechnica
Hungarica, 18(5), 2021, pp.13-35.
5. H. Li, X. Nie, D. Duan, Y. Li, J. Zhang, M. Zhou and E.
Magid, An admittance‐controlled amplified force tracking
scheme for collaborative lumbar puncture surgical robot
system. The International Journal of Medical Robotics andComputer Assisted Surgery, 18(5), 2022, e2428.
6. Q. Liu, J. Zuo, C. Zhu and S. Q. Xie, Design and controlof soft rehabilitation robots actuated by pneumatic
muscles: State of the art. Future Generation Computer
Systems, vol. 113, 2020, pp. 620-634.
7. N. Simaan, R. M. Yasin and L. Wang, Medical
technologies and challenges of robot-assisted minimallyinvasive intervention and diagnostics. Annual Review ofControl, Robotics, and Autonomous Systems, vol. 1, 2018,
pp. 465-490.
8. E. B. Joyee and Y. Pan, Additive manufacturing of multimaterial soft robot for on-demand drug delivery
applications. Journal of Manufacturing Processes, vol. 56,
2020, pp. 1178-1184.
9. X. Kong, P. Gao, J. Wang, Y. Fang and K. C. Hwang,
Advances of medical nanorobots for future cancer
treatments. Journal of Hematology & Oncology, 16(1),2023, p. 74.
10. H. Afrisal, A. D. Setiyadi, M. A. Riyadi, R. Ismail, O.
Toirov and I. Setiawan, Performance Analysis of 4-DOF
RPRR Robot Manipulator Actuation Strategy for Pick andPlace Application in Healthcare Environment.
International Journal on Advanced Science, Engineering
and Information Technology, 12(6), 2022, pp. 2258-2265.

Fig. 4. Navigation in RViz: red markers denote
waypoints that MoveIt uses for motion planning
RViz software [22] is commonly used together withMoveIt for visualization of motion planning tasks. A
control ROS node is written to bring up the autonomous
navigation system and start path planning.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
310

Powered by TCPDF (www.tcpdf.org)

Bulat Abbyasov, Aidar Zagirov, Timur Gamberov, Hongbing Li, Evgeni Magid

11. L. Zongxing, L. Wanxin and Z. Liping, Researchdevelopment of soft manipulator: A review. Advances in
MechanicalEngineering,12(8),
2020,p.
1687814020950094.
12. A. Sagitov, T. Tsoy, H. Li and E. Magid, Automated open
wound suturing: detection and planning algorithm. J.
Robotics Netw. Artif. Life, 5(2), 2018, pp. 144-148.
13. J. Zhu, L. Lyu, Y. Xu, H. Liang, X. Zhang, H. Ding and Z.
Wu, Intelligent soft surgical robots for next‐generation
minimally invasive surgery. Advanced Intelligent Systems,
3(5), 2021, p. 2100011.
14. S. Nisar, A. Hameed, N. Kamal, O. Hasan and F. Matsuno,
Design and realization of a robotic manipulator forminimally invasive surgery with replaceable surgical tools.
IEEE/ASME Transactions on Mechatronics, 25(6), 2020,
pp. 2754-2764.
15. J. Cornejo, J. P. Perales-Villarroel, R. Sebastian and J. A.
Cornejo-Aguilar, Conceptual design of space biosurgeon
for robotic surgery and aerospace medicine. In 2020 IEEE
ANDESCON, October 2020, pp. 1-6.
16. Y. Liu, Z. Li, H. Su, L. Jiang and C. Y. Su, Whole bodycontrol of an autonomous mobile manipulator using series
elastic actuators. IEEE/ASME Transactions on
Mechatronics, 2021.
17. A. Kumar, M. Vohra, R. Prakash and L. Behera, Towards
deep learning assisted autonomous uavs for manipulation
tasks in gps-denied environments. In 2020 IEEE/RSJ
International Conference on Intelligent Robots andSystems (IROS), October 2020, pp. 1613-1620.
18. K. Asadi, V. R. Haritsa, K. Han and J. P. Ore, Automatedobject manipulation using vision-based mobile robotic
system for construction applications. Journal ofComputing in Civil Engineering, 35(1), 2021, p. 04020058.
19. M. Görner, R. Haschke, H. Ritter and J. Zhang, Moveit!
task constructor for task-level motion planning. In 2019
International Conference on Robotics and Automation(ICRA), May 2019, pp. 190-196.
20. N. Koenig and A. Howard, Design and use paradigms forgazebo, an open-source multi-robot simulator. In 2004
IEEE/RSJ international conference on intelligent robots
and systems (IROS) (IEEE Cat. No. 04CH37566), vol. 3,
September 2004, pp. 2149-2154.
21. C. C. Wong, H. M. Feng, Y. C. Lai and H. Y. Chen,
Manipulator system designs for drawing random objects
through point cloud posture estimation. Proceedings of theInstitution of Mechanical Engineers, Part B: Journal ofEngineering Manufacture, 236(14), 2022, pp. 1867-1884.
22. A. Dobrokvashina, R. Lavrenov, E. Magid, Y. Bai and M.
Svinin, How to Create a New Model of a Mobile Robot in
ROS/Gazebo Environment: An Extended Tutorial.
International Journal of Mechanical Engineering andRobotics Research, 12(4), 2023.

Mr. Aidar Zagirov
He obtained his B.Sc. degree from KFU in
2023. Currently, he is a first-year student
in the Master's program for Intelligent
Robotics at ITIS, KFU. Aidar works as a
research assistant at the Laboratory ofIntelligent Robotic Systems (LIRS) at
KFU, Russia.
Mr. Timur Gamberov
He is a 4rd year student at the ITIS, KFU,
Russia. He works as a research assistant at
LIRS, KFU.

Associate Professor Hongbing Li
He received his Ph.D. degree in MechanoMicro Engineering at the Tokyo Institute ofTechnology, Japan. Currently he works at
the Shanghai Jiaotong University as an
Associate Professor. His research interests
include surgical robots, surgical instrument
design, robot force control and haptic
perception for minimally invasive surgery.
Professor Evgeni Magid
He is a Head of Intelligent Robotics
Department, a founder, and a Head ofLaboratory of Intelligent Robotic Systems(LIRS) at Kazan Federal University,
Russia. Senior member of IEEE. He
obtained Ph.D. degree in Robotics
Engineering from the University ofTsukuba, Japan.

Authors Introduction
Mr. Bulat Abbyasov
Bulat Abbyasov received his master
degree at Institute of Information
Technology and Intelligent Systems(ITIS), Kazan Federal University in 2022.
Currently, he is a second year PhD student
in Robotics at ITIS, Kazan Federal
University (KFU).

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
311

Robot-Assisted Language Learning: Scientific Data Analysis
Karina Sadyikova

Institute of Information Technology and Intelligent Systems, Kazan Federal University, Kazan, Russia
Valeriya Zhukova

Laboratory of Intelligent Robotics Systems (LIRS), Institute of Information Technology and Intelligent Systems, Kazan
Federal University, Kazan, Russia
Roman Lavrenov

Laboratory of Intelligent Robotics Systems (LIRS), Institute of Information Technology and Intelligent Systems, Kazan
Federal University, Kazan, Russia
Email: sadyikovakarina@mail.ru, v.zhukova@it.kfu.ru, lavrenov@it.kfu.ru

Abstract
Robot-assisted language learning (RALL) is a direction in education that uses robots in the process of studying
foreign languages. In this paper, we present the results of our research work devoted to studying trends in thequantity of publications in the field of using social robots for learning foreign languages. Additionally, we haveanalyzed some published paper in the field of RALL, which has allowed us to identify several popular robot models
used in practical research in recent years. Among these models were the NAO, Pepper and DARWIN OP-2 robots.
We have found that these robots are actively used in the educational process for learning foreign languages. In
particular, these models are used to improve language and communication skills during implicit learning andinteractive games.
Keywords: Robot-assisted language learning, Foreign language learning, Robot teaching assistant, Social robotics

tasks: they can provide the necessary assistance to theelderly [11], they can be used for medical procedures
and examinations [12], [13] and find application as
personnel in hospitals [14], and they are also used inareas of high risk for humans, in rescue operations [15].

1. Introduction
The last decades have seen significant progress in thefield of Artificial Intelligence (AI), which has become
an integral part of human life. This development has
resulted in the emergence of many systems with theability to replace certain aspects of human activity.
Modern AI systems are innovative technologicalsolutions capable of functioning effectively in various
fields of activity, including manufacturing, medicine,finance, transportation, and others. Some studies
speculate about the prospects of symbiosis of humanand cybernetic systems [1]. Robotics is closely related toartificial intelligence and also applies the results ofprogress in this area [2]. Thanks to artificial intelligence,
it is possible to generate a three-dimensional space forrobot testing in simulation [3], generate a maze [4] andcreate obstacles to further calibrate algorithms forcomplex tasks [5]. Robots, thanks to machine learningalgorithms, are able to navigate in the Gazebo
simulation environment [6], [7], in real-world
environments [8], and can also follow humans [9] andindependently plan a route to move [10].

Currently, robotics is widely used in education [16].
There it finds application in the creation of programsand exercises for students [17], as well as can be an
assistant in learning foreign languages [18]. The use ofsocial robotics in education has great potential and is ofsignificant interest to researchers in both the humanities
and technical fields of science. Robot assisted language
learning (RALL) is a direction in education that uses
robots in the process of studying foreign languages.
Education using robots is aimed primarily at increasing
student motivation and the degree of interactivity of thelearning process itself, as well as adapting the
educational process to the individual needs of students.
The aim of this paper is to study the dynamics of thenumber of publications devoted to the use of social
robots for learning foreign languages. Additionally, we
will examine the robot models used by researchers in thefield of RALL.

The significant achievement of robotics is the direct
assistance to humans, which makes this field in demand
nowadays. Modern robots perform a huge range of©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
312

Karina Sadyikova, Valeriya Zhukova, Roman Lavrenov

2. Analysis of Publication Activity in the Field of
RALL

Scholar search engine. Among the most popular modelsof robots used in practical research were the NAO robot
and the Pepper robot from SoftBank Robotics, as well as
the DARWIN-OP2 robot from ROBOTIS.

In order to assess the interest of researchers in the fieldof RALL, we counted the number of published papers
for the query "robot-assisted language learning" in the
Google Scholar search engine. In total, based on the
results of this search query, we received more than 900
scientific papers. Among the collected papers, we haveselected those published no earlier than 2010 (totaling
889 papers) and counted the number of publications ineach year from 2010 to 2023. The resulting diagram of
the number of published papers in each year is presented
in Fig. 1. Based on the diagram presented in Fig. 1 wecan conclude that interest in social robotics and, inparticular, in robot-assisted language learning has
increased over the last decade.

The NAO robot model was referenced in 3 research
papers [19], [20], [21]. The study [19] examined theimpact of the NAO robot functioning as a teacher's
assistant on university students' English vocabulary
improvement. The research [20] implemented a systemof implicit learning utilizing two NAO robot models. In
study [21], the NAO robot was utilized in teaching asecond language to 5-year-old children. Two research
papers [22], [23] referenced the Pepper robot model. In[22], the Pepper robot was employed for interactive
game-based learning in single-user and team versions ofa quiz game. In study [23], a project was presented
which involved the Pepper robot being used as the basis
for developing for robotic teaching of the endangered
Ainu language. The DARWIN-OP2 robot was alsomentioned in two research papers [18], [24]. The study[18] presented the experimental verification of English
language lesson scenarios with the DARWIN-OP2 robot
in the Gazebo simulator. The study [24] introduced apilot experiment involving the interaction of preschool
children aged 5-6 years with the DARWIN-OP2 robot,
serving as a teacher's assistant.
4. Discussion

Fig.1 A number of published papers (X-axis) on robotassisted language learning in the Google Scholar searchengine per year (Y-axis)

In this section, we provide the general characteristicsof the NAO, Pepper, and DARWIN-OP2 robots, andalso review key characteristics of the studies presented
in the previous section.

According to Fig. 1, starting in 2019, interest in robots
assisting with foreign language learning began to grownoticeably. Thus, among the papers presented in the
search results there were 80 papers published in 2019,
and 148 papers published in 2022. This increase ininterest may be attributed, among other factors, to theimpact that the COVID-19 pandemic has had on themethods of communication in society in general, andwithin educational systems in particular. Due torestrictions on traveling and face-to-face meetings,
many people have turned to online education and digitalsolutions to learn foreign languages. Robots capable ofteaching languages provide a convenient and affordable
way to learn virtually. This has further influenced thegrowth of technology in this field. These results confirm
that social robotics and robot-assisted language learning
have great potential and will continue to attract theattention of researchers and society at large.

The robots NAO, Pepper, and DARWIN-OP2 sharecharacteristics that make them attractive for research in
the field of RALL. The robot NAO has small
dimensions - about 58 cm in height, and a humanoid
form with a head, arms, and legs. The DARWIN-OP2
robot has more compact dimensions - about 45 cm inheight, and a simpler design. Pepper is a taller robot about 120 cm in height. Like NAO and DARWIN-OP2,
Pepper has a head and two arms. However, unlike NAO
and DARWIN-OP2, Pepper does not have legs, andoverall, Pepper has a more monumental appearance.
Overall, the visual design of all three robots is
conducive to social interaction.
From the point of view of using these robot models in
an educational environment when learning foreign
languages, all three robot models also have a number ofcommon attractive characteristics. First of all, thesemodels are programmable models and can be adapted tovarious educational contexts. From a technical point ofview, robots NAO, Pepper and DARWIN-OP2 provide
ample opportunities for speech synthesis andrecognition, integration of computer vision and machinelearning. All this allows researchers to achieve a high

3. Robot Models Used in RALL
To identify the robots currently utilized in robotassisted language learning research, we conducted ananalysis of around 100 papers derived from the search
query "robot-assisted language learning" in the Google©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
313

Robot-Assisted Language Learninglevel of human-robot
interaction with children.

interaction,

includingthepublications since 2019, indicating a growing interest in
robots helping with foreign language learning. This
increase in interest can be partially attributed to theimpact of the COVID-19 pandemic, which has
prompted a shift to online education and digital
language learning solutions due to travel restrictions andsocial distancing measures. Among the most frequently
used robot models in RALL research were the NAO,
Pepper, and DARWIN-OP2 robots. These robots areused to improve language skills and communication in avariety of educational scenarios, including implicit
learning and interactive games.

In Table 1 we provide a comparative analysis of someof the characteristic features of the studies we indicatedin previous section, in which the NAO, Pepper and
DARWIN-OP2 robot models were used.
Table 1. Key characteristics of studies using theNAO, Pepper and DARWIN-OP2 robots.
The application area
Assistant English teacher for
university students improving their
NAO
vocabulary.
Implicit learning throughconversation.
PepperStudying English grammar inuniversities.
Learning an endangered language.
DARWIN-OP2
English lessons with preschool
children.
Objectives of the experiments
Studying the effect of a robot on
NAO
improving students' vocabulary.
Pepper
The effectiveness of game-basedlearning.
DARWIN-OP2
Interaction with preschool children
in English lessons.
Distinctive features of the studies
The use of implicit learning as a
NAO
primary learning style.
PepperUsing a robot as a play partner withan educational interactive quiz.
DARWIN-OP2
The optimization of the English
language learning process with thehelp of the Gazebo simulator andglobal human-robot interaction
scenarios

Acknowledgements
This work was supported by the Kazan Federal
University Strategic Academic Leadership Program("PRIORITY-2030").
References
1.

2.

3.

From the presented in Table 1 training scenarios, it
can be concluded that the NAO, Pepper and DARWINOP2 robots demonstrate a wide range of capabilities in
the context of educational environments. They havebeen successfully used in university classrooms, ranging
from helping students learn foreign language andimproving students' vocabulary to using them in gamebased learning. In addition, these robots successfullyinteract with preschool children, optimizing thelanguage learning process using advanced simulatorsand global human-robot interaction scenarios. Thus,
learning cases using these robots reflect theirapplicability in various educational contexts and theirability to effectively support and improve students'
language skills.

4.

5.

6.

5. Conclusions
Our analysis of the number of published papers related
to "robot-assisted language learning" demonstrates a
steadily growing interest among researchers in the
RALL field. Data obtained from a Google Scholar
search show a noticeable increase in the number of7.

М. Talanov, J. Vallverdú, A. Adamatzky et al.,Neuropunk Revolution. Hacking Cognitive Systems
towards Cyborgs 3.0, International Journal of
Unconventional Computing, Vol. 18 (2-3), 2023, pp.
145−201.
E. Magid, A. Pashkin, N. Simakov et al., Artificial
Intelligence Based Framework for Robotic Search andRescue Operations Conducted Jointly by International
Teams, Proceedings of 14th International Conference on
Electromechanics and Robotics “Zavalishin's Readings”
ER (ZR), Kursk, Russia, 2019. pp. 15−26.
E. Denisov, A. Sagitov, R. Lavrenov et al., DCEGen:
Dense Clutter Environment Generation Tool for
Autonomous 3D Exploration and Coverage Algorithms
Testing, in Interactive Collaborative Robotics. ICR 2019.
Lecture Notes in Computer Science, eds. A. Ronzhin, G.
Rigoll, R. Meshcheryakov, Springer, Cham, vol 11659,
2019, pp. 215−225.
A. Iskhakova, B. Abbyasov, T. Mironchuk et al., LIRSMazeGen: An Easy-to-Use Blender Extension for
Modeling Maze-Like Environments for Gazebo
Simulator, in Frontiers in Robotics and Electromechanics.
Smart Innovation, Systems and Technologies, eds. A.
Ronzhin, V. Pshikhopov, Springer, Singapore, vol 329,
2023, pp. 147–161.
R. Gabdrahmanov, T. Tsoy, Y. Bai, M. Svinin and E.
Magid, Automatic Generation of Random Step
Environment Models for Gazebo Simulator, in Robotics
for Sustainable Future. CLAWAR 2021. Lecture Notes in
Networks and Systems, eds. D. Chugo, M.O. Tokhi, M.F.
Silva, T. Nakamura, K. Goher, Springer, Cham, vol 324,
2022, pp. 408–420.
R. Safin, R. Lavrenov and E.A. Martínez-García,
Evaluation of Visual SLAM Methods in USAR
Applications
Using
ROS/Gazebo
Simulation,
Proceedings of 15th International Conference on
Electromechanics and Robotics “Zavalishin's Readings”,
Ufa, Russia, 2021, pp. 371−382.
E. Mingachev, R. Lavrenov, T. Tsoy, F. Matsuno, M.
Svinin, J. Suthakorn and E. Magid, Comparison of ROSbased monocular visual SLAM methods: DSO, LDSO,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
314

Powered by TCPDF (www.tcpdf.org)

Karina Sadyikova, Valeriya Zhukova, Roman Lavrenov

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

ORB-SLAM2 & DynaSLAM, in 5th International
Conference on Interactive Collaborative Robotics (ICR
2020), St. Petersburg, Russia, 2020, pp. 222–233.
E. Chebotareva, T. Tsoy, B. Abbyasov, J. Mustafina, E.A.
Martinez-Garcia, Y. Bai and M. Svinin, On the Problemsof SLAM Simulation for Mobile Robots in the Arctic
Conditions, in 5th International Conference on Interactive
Collaborative Robotics (ICR 2020), St. Petersburg,
Russia, 2020, pp. 33–44.
E. Chebotareva, R. Safin, K.-H. Hsia, A. Carballo and E.
Magid, Person-Following Algorithm Based on Laser
Range Finder and Monocular Camera Data Fusion for a
Wheeled Autonomous Mobile Robot, in 5th International
Conference on Interactive Collaborative Robotics (ICR
2020), St. Petersburg, Russia, 2020, pp. 21–33.
Y. Bai, M. Svinin, E. Magid and Y. Wang, On Motion
Planning and Control for Partially Differentially Flat
Systems, Robotica, Vol. 39, no. 4, 2021, pp. 718−734.
K. Khusnutdinov, A. Sagitov, A. Yakupov, R. Lavrenov,
E. A. Martinez-Garcia, K.-H. Hsia, and E. Magid,
Household Objects Pick and Place Task for AR-601M
Humanoid Robot, in 4th International Conference on
Interactive Collaborative Robotics (ICR 2019), Istanbul,
Turkey, 2019. pp. 139–149.
A. Sagitov, T. Tsoy, H. Li and E. Magid, Automatedopen wound suturing: detection and planning algorithm.
Journal of Robotics, Networking and Artificial Life, Vol.
5, no. 2, 2018, pp. 144−148.
A. Shafikov, T. Tsoy, R. Lavrenov et al, Medical
palpation autonomous robotic system modeling andsimulation in ROS/Gazebo, in 13th International
Conference on Developments in eSystems Engineering(DeSE), Liverpool, United Kingdom, 2020, pp. 200−205.
R. Safin, R. Lavrenov, T. Tsoy, E. Magid, M. Svinin, S.
D. Roy and S. K. Saha, Prioritizing tasks within a robotic
transportation system for a smart hospital environment, in
6th International Conference (ICR 2021), St. Petersburg,
Russia, 2021, pp. 182−193.
V. Myrzin, T. Tsoy, Y. Bai, M. Svinin and E. Magid,
Visual data processing framework for a skin-based
human detection, 6th International Conference (ICR
2021), St. Petersburg, Russia, 2021, pp. 138−149.
E. Chebotareva and M. Mustafin, Android Based
Educational Mobile Robot Design and Pilot Evaluation,in 2023 International Conference on Artificial Life andRobotics (ICAROB2023), Oita, Japan, 2023, pp.
145−148.
S. Bulatov, E. Kharisova, R. Lavrenov, B. Abbyasov andE. Magid, Structural Model of a Training Computer
Program for Improving Professional Skills of a Student ina Role of a District Polyclinic Physician, Journal of
Robotics, Networking and Artificial Life, Vol. 8, no. 2,
2021, pp. 122−126.
L. Gavrilova, A. Kotik, T. Tsoy et al., Facilitating apreparatory stage of real-world experiments in ahumanoid robot assisted English language teaching using
Gazebo simulator, in 2020 13th International Conferenceon Developments in eSystems Engineering, Liverpool,
United Kingdom, 2020, pp. 222−227.
H. Banaeian and I. Gilanlioglu, Influence of the NAO
robot as a teaching assistant on university students’vocabulary learning and attitudes, Australasian Journal of
Educational Technology, 37(3), 2021, pp. 71−87.
A. Khalifa, T. Kato and S. Yamamoto, Effect of Implicit
Learning in Joining-in-type Robot-assisted Language
Learning System, International Journal of Emerging

21.

22.

23.

24.

Technologies in Learning (iJET), 14(02), 2019, pp. 105–
123.
M. de Haas, P. Vogt and E. Krahmer, The Effects ofFeedback on Children's Engagement and Learning
Outcomes in Robot-Assisted Second Language Learning,
Front. Robot. AI, 7:101, August (2020).
I. Buchem, A. Mc Elroy and R. Tutul, Designing andProgramming Game-Based Learning with Humanoid
Robots: a Case Study of the Multimodal “Make or Do”
English Grammar Game with the Pepper Robot,
ICERI2022 Proceedings, Seville, Spain, 2022, pp. 15371545.
K. Nowakowski, M. Ptaszynski and F. Masui, Spicing up
the Game for Underresourced Language Learning:
Preliminary Experiments with Ainu Language-speaking
Pepper Robot, in 6st Workshop on Linguistic andCognitive Approaches to Dialog Agents (LaCATODA
2020), 2020.
L. Gavrilova, V. Petrov, A. Kotik et al., Pilot Study ofTeaching English Language for Preschool Children witha Small-Size Humanoid Robot Assistant, in 2019 12th
International Conference on Developments in eSystems
Engineering (DeSE), Kazan, Russia, 2019, pp. 253−260.

Authors Introduction
Ms. Karina Sadyikova
She received her Bachelor's degree in
Accounting Analysis and Audit from theFaculty of Economics, Kazan Federal
University, Russia, in 2022. She iscurrently a Master's student at the InstituteofInformation
Technologies
andIntelligent Systems, Kazan Federal
University, Russia.
Ms. Valeriya Zhukova
In 2022 she graduated from Kazan
Federal University with a degree inlinguistics as an English and Koreaninterpreter at the Institute of International
Relations. Currently is a Master’s student
in the foreign language teaching program
and works as a Research Assistant at the Institute ofInformation Technologies and Intelligent Systems, Kazan
Federal University, Russia.
Assistant Professor Roman Lavrenov
Currently he works as an AssistantProfessor at Kazan Federal University.
He obtained his Ph.D. at Kazan Federal
University in 2020. His main interests arepath planning, robot operating system,
and autonomous mobile robots.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
315

Monitoring Beehive Sound Levels with Arduino-based System
Ramis Kulmukhametov

Intelligent Robotics Department, Kazan Federal University, 420008, Kazan, Russian Federation
Ramil Safin

Intelligent Robotics Department, Kazan Federal University, 420008, Kazan, Russian Federation
Tatyana TsoyIntelligent Robotics Department, Kazan Federal University, 420008, Kazan, Russian Federation
Kuo-Hsien Hsia

National Yunlin University of Science and Technology, Taiwan
Evgeni MagidIntelligent Robotics Department, Kazan Federal University, 420008, Kazan, Russian Federation
Email: RNKulmukhametov@stud.kpfu.ru, tt@it.kfu.ru, khhsia@yuntech.edu.tw, magid@it.kfy.ru

Abstract
An automated beekeeping is a promising approach to addressing various issues associated with a beekeeping. Among
primary problems, a swarming procedure stands out as a major concern. An uncontrolled swarming can lead tosignificant financial losses. During a swarming period a potential for losing a bee swarm is high, therefore a noisemonitoring at this period gains a significant importance. Our long term research aims at a development of an intelligent
monitoring system for beehive conditions, based on a hive-generated noises analysis. This paper presents anexperiment that collected data about acoustic characteristics of bee states using an Arduino microcontroller and a
MAX9814 microphone module. The obtained data analysis is discussed.
Keywords: beehive condition monitoring, acoustic level analysis, Arduino microcontroller, microphone,anthropogenic disturbanceof some bees and, consequently, a decline in honeyproduction.

1. Introduction
Currently, beekeepers are increasingly facing an issue
of a rising mortality among honeybees, attributed tovarious factors [1]. This problem carries significant
ecological and economic consequences that may cause aheavy impact on a biodiversity of wild plants [2] andagricultural production [3], [4]. One of the mostdestructive phenomena in this field is Colony Collapse
Disorder, in which honeybees abandon their hives [5], [6],[7]. This colony collapse or demise is a result of multiple
negative factors that can act independently, combine, or
amplify one another [8]. A mortality of honeybees can be
attributed to various factors, which can be classified into
two main groups: a natural factor and a anthropogenic
factor. Natural factors include bee infestations by Varroa
mites and a swarming. The former are major parasites ofhoneybees, leading to a reduced hive viability and adecreased honey production [9]. A swarming, on theother hand, is a natural process of dividing a bee colony
into two parts [10]. During the swarming, some worker
bees and one or several queens leave a current nest to
establish a new colony elsewhere. However, this process
can pose a challenge for beekeepers as it involves a loss

To deal with Varroa mite a number of methods were
proposed including computer vision approaches [11],[12], [13] and real-time visualization systems [14] thatallow monitoring of honeybee activity by counting theirentry and exit from a beehive [15], [16]. Sensor fusion[17], [18] and fiducial markers [19] can also be
incorporated into the monitoring process to provide morerobust and accurate measurements. Researchers
identified a correlation between the swarming and a hive
weight, a sound, a temperature, and a humidity inside a
hive [20]. Particularly noteworthy is an analysis ofsounds produced by honeybees, which has proven to be
one of the most effective methods for detecting theswarming [21]. Scientific studies in this fielddemonstrated a correlation between specific soundfrequencies and the swarming process [22], [23], making
an acoustic analysis of bee sounds a promising avenue formonitoring and studying a condition of bee colonies.
This article presents results of a pilot experiment thatanalyzed a hive noise under normal conditions and after©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
316

Ramis Kulmukhametov, Ramil Safin, Tatyana Tsoy, Kuo-Hsien Hsia, Evgeni Magidbee disturbances. It is a part of our long term goal of
constructing an IoT-based monitoring system that could
improve a productivity of honeybees by a timely
detection of various events inside beehives and reporting
results of AI-based analysis to beekeepers.
2. Experimental Setup
The Arduino UNO microcontroller acts as a mainprocessor, performing key monitoring functions of theexperimental setup. The microcontroller receives data
from
MAX9814
microphone
module.
The
microcontroller and the microphone were placed together
into a cardboard box with a power and data wire
connected to a notebook computer. The box was placed
under a hive cover, above hive frames. The setup and thedevice placement within a beehive are depicted in Fig. 1.

Fig. 2. Acoustic monitoring flowchart
The device was placed into a beehive and themicrophone collected acoustic data from bees. The
experiment was divided into two phases, 30 seconds each.
The first phase corresponded to normal conditions of thebeehive. In the second phase an anthropogenic
disturbance was generated by knocking the hive for 3
seconds in order to provoke a response from the bees. The
phase aimed to investigate changes in the bees’ behavior.
a

b
-170

Sound level (dB)

Fig. 1. Experimental setup: (a) the device; (b) the
container placement inside a beehive
The device was programmed using the Arduino opensource software development platform. Arduino IDE wasused for programming the microcontroller and reading
data from the microphone. Upon powering up the
microcontroller, the program initializes variables andopens a serial port at a baud rate of 115200. Data aretransmitted through UART communication, and obtainedby the microphone values are converted to decibels. A
flow of the program is depicted in Fig. 2.

-190

-210
-230
-250
0

10

20

30

40

50

60

Time (s)3. Results
Fig. 3. An acoustic level at a human disturbance event
A pilot experiment targeted to examine a state and abehavior of bees in terms of acoustic changes underartificial human-made disturbances. The pilot experiment
was conducted on October 8, 2023, at 14:30:00 of thelocal time. By this time of the year, bees completed theirmain honey gathering season and started preparations forthe upcoming winter season.

Fig. 3 presents experimental results of acoustic levels
that were measured within the beehive for one minute. It
is worth noting that the measurements were taken during
a cool weather, which could have affected the acoustic
level, making it lower compared to the main honeyharvesting period. Initially, in a state of a rest, themeasured acoustic level was -240 dB. The highest
recorded acoustic level was -170 dB, which occurred at
the beginning of the second phase (14:30:31) when the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
317

Monitoring Bee Hive Soundbeehive was subjected to a 3-seconds knocking. At thismoment, the bees transitioned to an alarmed state, thatwas reflected in the acoustic level of -210 dB, which
gradually decreased over several seconds. Interestingly,
when the device was removed from the beehive, there
was a clustering of bees around its location. This behavior
could be interpreted as an attempt of the bees to protect
their colony. They formed a small cluster around thedevice, and as their stings proved ineffective against thepotential threat, they attempted surrounding the “intruder”
with their bodies, creating a defensive barrier. Thisbehavior impacted the acoustic level, which increased byseveral decibels in the period from 14:30:27 to 14:30:29,
prior to the anthropogenic disturbance.

5.

H. M. Al-Solami, N. A. Alkenani, A. G. Alghamdi, M. M.
M. Ahmed, K. Javeed and A. A. Dar, Influence andManagement of Colony Collapse Disorder (CCD)
Damaging European Honeybee Apis mellifera, Specialusis
Ugdymas, Vol. 2(43), 2022, pp. 3117-3130.
6. Nikita, A. Grover, P. Kalia, R. Sinha and P. Garg, Colony
collapse disorder: A peril to apiculture, Journal of Applied
and Natural Science, Vol. 14(3), 2022, pp. 729-739.
7. P. A. Woods, Illustrative Document of the Colony
Collapse Disorder, 2022.
8. F. Requier, L. Garnery, P. L. Kohl, H. K. Njovu, C. W. W.
Pirk, R. M. Crewe and I. Steffan-Dewenter, The
conservation of native honey bees is crucial, Trends inecology & evolution, Vol. 34(9), 2019, pp. 789-798.
9. K. S. Traynor, F. Mondet, J. R. de Miranda, M. Techer, V.
Kowallik, M. A. Y. Oddie. P. Chantawannakul and A.
McAfee, Varroa destructor: A complex parasite, cripplinghoney bees worldwide, Trends in parasitology, Vol. 36(7),2020, pp. 592-606.
10. R. A. Ellis, T. Weis, S. Suryanarayanan and K. Beilin,
From a free gift of nature to a precarious commodity: Bees,
pollination services, and industrial agriculture, Journal ofAgrarian Change, Vol. 20(3), 2020, p. 437-459.
11. S. Sevin, H. Tutun and S. Mutlu, Detection of varroa mites
from honey bee hives by smart technology var-gor: ahivemonitoring and image processing device, Turkish
Journal of Veterinary & Animal Sciences, Vol. 45(3), 2021,
pp. 487-491.
12. G. J. Tu, M. K. Hansen, P. Kryger and P. Ahrendt,
Automatic behaviour analysis system for honeybees using
computer vision, Computers and Electronics inAgriculR12re, Vol. 122, 2016, pp. 10-18.
13. M. Roy, R. A. Boby, S. Chaudhary, S. Chaudhury, S.
D. Roy, S. D. and S. K. Saha, Pose estimation of textureless cylindrical objects in bin picking using sensor fusion.
In 2016 IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS), October 2016, pp. 2279-2284.
14. E. Magid, R. Lavrenov, T. Tsoy, M. Svinin and R. Safin,
Real-time Video Server Implementation for a Mobile
Robot. Proceedings of 11th International Conference on
Developments in eSystems Engineering (DeSE), 2018, p.
180-185.
15. C. Chen, E.-C. Yang, J.-A. Jiang and T.-T. Lin, An
imaging system for monitoring the in-and-out activity ofhoney bees, Computers and electronics in agriculR12re,Vol. 89, 2012, pp. 100-109.
16. T. N. Ngo, K.-C. Wu, E.-C. Yang and T.-T. Lin, A realtime imaging system for multiple honey bee tracking andactivity monitoring, Computers and Electronics inAgriculture, Vol. 163, 2019, p. 104841
17. R. Safin, T. Tsoy, R. Lavrenov, I. Afanasyev and E. Magid,
Modern Methods of Map Construction Using OpticalSensors Fusion. International Conference on Artificial Lifeand Robotics (ICAROB 2023), 2023, pp. 167-170.
18. E. A. Martínez-García, Cyber-Physical Robotics: RealTime Sensing, Processing and Actuating. Cyber-Physical
Systems for Industrial Transformation. CRC Press, 2023,
pp. 57-74.
19. T. Tsoy, R. Safin, R. Sultanov, S. K. Saha and E. Magid,
Recommended criteria for qualitative comparison of
fiducial markers performance. Siberian Conference onControl and Communications, (SIBCON 2022), 2022, p.
1-5.
20. W. Hong, B. Xu, X. Chi, X. Cui, Y. Yan and T. Li, Longterm and extensive monitoring for bee colonies based on

Based on the experimental observations, it was
concluded that while the device could collect valuable
acoustic data, a safer placement of a microphone and theprocessing block should be selected (which could be doneusing a virtual environment [24]), e.g., on an outersurface of a back wall of a beehive that is opposite a bees’
entrance. This would protect the device from overheating
and other factors generated by the bees themselves andensure a more reliable data acquisition.
4. Conclusions
In this study, a part of the bee monitoring systemdesigned to collect data on an acoustic level within abeehive was successfully developed and tested. The
experiment aimed to study a bees’ behavior underconditions of an anthropogenic disturbance. This
experiment represents a step towards a better
understanding of bees behavior and their response tointerventions. Based on the experimental observations, it
was also concluded that a proper selection of themonitoring device and sensors’ location is crucial.
Acknowledgement
This paper has been supported by the Kazan Federal
University Strategic Academic Leadership Program("PRIORITY-2030").
References
1.

2.

3.
4.

A.-M. Klein and B. E. Vaissière, Importance of pollinatorsin changing landscapes for world crops, Proceedings of theroyal society B: biological sciences, Vol. 274, 2007, pp.
303-313.
D. A. Stanley, S. M. Msweli and S. D. Johnson, Nativehoneybees as flower visitors and pollinators in wild plantcommunities in a biodiversity hotspot, Ecosphere, Vol.
11(2), 2020, e02957.
D. L. Gazzoni and J. V. G. R. P Barateiro, Soybean yield
is increased through complementary pollination by honeybees, Journal of Apicultural Research, 2023, pp. 1-12.
G. MacInnis and J. R. K. Forrest, Pollination by wild bees
yields larger strawberries than pollination by honey bees,
Journal of Applied Ecology, Vol. 56(4), 2019, pp. 824-832.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
318

Powered by TCPDF (www.tcpdf.org)

Ramis Kulmukhametov, Ramil Safin, Tatyana Tsoy, Kuo-Hsien Hsia, Evgeni Magidinternet of things, IEEE Internet of Things Journal, Vol.
7(8), 2020, pp. 7148-7155.
21. A. Terenzi, S. Cecchi and S. Spinsante, On the importanceof the sound emitted by honey bee hives, Veterinary
Sciences, Vol. 7(4), 2020, p. 168.
22. K. I. Dimitrios, C. V. Bellos, K. A. Stefanou, G. S. Stergios,
I. Andrikos, T. Katsantas and S. Kontogiannis,
Performance Evaluation of Classification Algorithms to
Detect Bee Swarming Events Using Sound, Signals, Vol.
3(4), 2022, pp. 807-822.
23. G. Voudiotis, S. Kontogiannis and C. Pikridas, Proposed
smart monitoring system for the detection of bee swarming,
Inventions, Vol. 6(4), 2021. p. 87.
24. A. Zagirov, A. Apurin, E. Chebotareva, Modeling of
Human Actions in a Collaborative Robotic Space Using
AR601M Humanoid Robot: Pilot Experiments in the
Gazebo Simulator. International Conference on Artificial
Life and Robotics (ICAROB 2023), 2023, pp. 163-166.

Ms. Tatyana Tsoy
In 2012 she graduated from International
Area Studies Master's Degree Program atthe University of Tsukuba. In 2023
completed a PhD in Computer science andinformation processes at the Institute ofInformation Technology and Intelligent
Systems (ITIS) of Kazan Federal
University.
Associate Professor Kuo-Hsien Hsia
He received a Ph.D. degree in ElectricalEngineering at the National Sun Yat-Sen
University, Kaohsiung, Taiwan. He works
at the National Yunlin University of
Science and Technology.

Professor Evgeni Magid
He is a Head of Intelligent Robotics
Department, a founder, and a Head ofLaboratory of Intelligent Robotic Systems(LIRS) at Kazan Federal University,
Russia. Senior member of IEEE. He
obtained Ph.D. degree in Robotics
Engineering from the University ofTsukuba, Japan.

Authors Introduction
Mr. Kulmukhametov Ramis
He received a BSc degree from Ufa State
Aviation Technical University in 2022.
Currently, he is a first-year student ofMaster degree program in Intelligent
Robotics at the Institute of Information
Technology and Intelligent Systems at the
Kazan Federal University. He is also a
professional beekeeper.
Mr. Ramil Safin
Ramil Safin is a Ph.D. student and a
research assistant at the Laboratory ofIntelligent Robotic Systems, Institute ofInformation Technology and Intelligent
Systems, Kazan Federal University, Russia.
He teaches the following courses:«Fundamentals of technical vision»,«Robotic systems sensors», «Algorithms and data
structures» His main interest is computer vision for robotic
applications.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
319

Restoration of Guqin Music by Deep Learning Methods
Takashi Kuremoto1, Kazuma Fujino1, Hirokazu Takahashi1, Shun Kuremoto2, 3,
Mamiko Koshiba2, Hiroo Hieda3, Shingo Mabu2
1

Nippon Institute of Technology, Gakuendai 4-1, Miyashiro-Machi, Saitama, 345-8501, JAPAN
2
Yamaguchi University, Tokiwadai 2-16-1, Ube, Yamaguchi, 755-8611, JAPAN3
Institute of Future Engineering, Fukagawa 2-6-11, Koto-ku, Tokyo, 135-8473 JAPANE-mail: kuremoto.takashi@nit.ac.jp
Abstract

Guqin （古琴）music played an important role in the history of Asia cultures. The notation of Guqin ancient musicremained more than 600, however, only about 100 of them are played in nowadays. The reason is that the handwritten
Guqin notations named “Jianzi Pu” is hard to be understood, however, we challenge to restore the Guqin music bydeep learning methods and few Jianzi Pu images. VGG16 and YOLOv5 were adopted in the recognition experimentfor Guqin music restoration. For a well-known Guqin music “Sen-O-So”（仙翁操）, 55 kinds of single charactersof Jianzi Pu and 4,951 images of them were collected from 23 kinds of Sen-O-So version found by the Internet or
obtained by image processing such as rotation, enlarge (zoom-in), reduce (zoom-out), filtering, etc. The average
accuracies of VGG16 and YOLOv5 were 87.50% and 88.47% for the test data, respectively. Additionally, it was
realized an online output of Guqin music as its output of audio or video forms by YOLOv5 in this study.
Keywords: deep learning, VGG16, YOLOv5, SVM, Guqin, Jianzi Pu, AI music

1. Introduction
Many kinds of ancient cultures and civilizations havedisappeared in the past. A Chinese stringed instrument
named “Guqin” （古琴）(or “Qin”（琴）) (Figs. 1 and
2) has a history more than 3,000 years, however, few
players in the world exists in the last century. Recently,
more Chinese youngers are learning this ancient musicalinstrument, however, it is very difficult to use its notation,
named “Jianzi Pu”, which is a kind of handwritten
character using Chinese character and its original
characters (see Figs. 3 and 4). There are more than 600
Guqin musical pieces in the history, meanwhile, theperformed pieces in nowadays only about 100 [1], [2].
These Guqin music are translated from Jianzi Pu tomodern staff notation by experts of Guqin players, andthis translation process is called “Dapu” [3].
Jianzi Pu recognition by computer has been studied
since last century in China [4], [5], [6], however, thepractical results are not obtained.

Fig. 1 A musical instrument: Guqin (7 strings Qin).

Fig. 2 Playing Guqin.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
320

Takashi Kuremoto, Kazuma Fujino, Hirokazu Takahashi, Shun Kuremoto, Mamiko Koshiba, Hiroo Hieda, Shingo Mabu

In our previous work [7], [8], machine learning
methods such as support vector machine (SVM), deeplearning models VGG16, ResNet50, and the hybrid
models VGG16 with SVM, ResNet50 with SVM wereadopted to be classifiers of single character of Jianzi Pu.
In the comparison experiments using a dataset of a wellknown Guqin music “Sen-O-So”（仙翁操）, VGG16
with SVM had the highest average accuracies to 15 kindsof single characters data which were the first and thesecond line of Sen-O-So Jianzi Pu in 1,500 images, i.e.,
99.11% in training and 88.33% for unknown data.
In this study, we built a bigger dataset for total
single characters of Sen-O-So Guqin notation. 55 kindsof images of single characters were collected from 23
kinds of original Jianzi Pu from the Internet, and 4,951
single characters were obtained including data
augmentation by image processing such as rotation,
reversion, size enlargement (zoom in), size reduction(zoom out), and filtering. To realize online recognition,
deep learning model YOLOv5 [9], [10] was adopted into
the system as a classifier. The validation accuracy of
YOLOv5 after 300 training arrived at 88.47% which was
higher than the case of VGG16 87.50%. Additionally,
audio/video data of single characters of Sen-O-So werecreated by Japan Society for the promotion of Guqin [2],and they were matched to the recognition results by
YOLOv5.

Fig. 3 A sample of Guqin musical notation “Sen-O-So”.

Fig. 4 A sample of a single Jianzi Pu and its meaning inJapanese.

Fig. 5 A restoration system of Guqin music.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
321

Restoration of Guqin Music
ファイル名

画像

画像処理の詳細

2. A Guqin Music Restoration System
7 (14).jpg

A Guqin music restoration system, as shown in Fig. 5,
is proposed in this study. It is aimed to realize that whena Jianzi Pu image is presented to the system, Guqin music
in audio/video forms are output. The system isconstructed by a fine-tuned deep learning model
YOLOv5 [9], [10] and a matching process of the outputof YOLOv5 and a single character database in audio andvideo forms.

元の画像

7 (14).jpg_30.jpg

高さと幅を30pxに変更

7 (14).jpg_100.jpg

高さと幅を100pxに変更

7(14).jpg_200.jpg

高さと幅を200pxに変更

7 (14).jpg_blur.jpg

カーネルサイズ（7，7）標準偏差1.5で平滑化

7 (14).jpg_brighter.jpg

色の値を1.5倍

7 (14).jpg_darker.jpg

色の値を0.9倍

7 (14).jpg_noise.jpg

ランダムな場所を対象にRGBランダムに選んでその値をランダムな少数分±する

7 (14).jpg_r45.jpg

反時計回りに45度回転

7 (14).jpg_r90.jpg

反時計回りに90度回転

7 (14).jpg_r135.jpg

反時計回りに135度回転

7 (14).jpg_r180.jpg

反時計回りに180度回転

7 (14).jpg_r225.jpg

反時計回りに225度回転

7 (14).jpg_r270.jpg

反時計回りに270度回転

7 (14).jpg_r315.jpg

反時計回りに315度回転

Fig. 7 A sample of data augmentation.
Fig. 6 A sample of 55 single characters of Sen-O-So.

2.2. Deep Learning Models
A well-known deep learning model VGG16 [7], [8]and an online object recognition model YOLOv5 [9], [10]were investigated their recognition accuracies to single
characters of Jianzi Pu. The state of art of YOLO is
YOLOv8, and we are investigating its performance for
Jianzi Pu recognition recently.

2.1. A Database of Jianzi Pu
To train the deep learning models, a database of single
characters of a well-known Guqin music “Sen-O-So” in
Jianzi Pu form is built at first. 23 versions of Sen-O-So
Jianzi Pu images were collected by the Internet. 55 kindsof single characters were segmented from the Sen-O-So
notation, and totally 343 images of single characters wereobtained. Data augmentation is adopted by image
processing such as rotation, reflection, parallel translation,
enlarge (zoom-in), reduce (zoom-out), filtering to theoriginal single character images. As the result, 4,951
images of single characters were obtained as a databaseof Jianzi Pu. A sample of the original single character
images and a sample of data augmentation were shown in
Fig. 6 and Fig. 7, respectively.

3. Experiments and Results
Using the database of Sen-O-So described in Section
2, VGG16 and YOLOv5 were fine-tuned with 300 epochs.
The average recognition accuracies of the single
characters were 87.50% and 88.47% respectively.
The change of accuracies in training of two models isshown in Fig. 8. It can be confirmed that the accuracies(training accuracy and validation accuracy) of VGG16
were converged from 200 epoch (Fig. 8a), meanwhile, it
converged from 100 epoch in the case of YOLOv5 (Fig.
8b).

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
322

Takashi Kuremoto, Kazuma Fujino, Hirokazu Takahashi, Shun Kuremoto, Mamiko Koshiba, Hiroo Hieda, Shingo Mabu

given by their position detected by YOLOv5. Bymatching the recognized single characters to audio/video
data, the system output music and videos as shown in Fig.
10.
4. Conclusion
To restore Guqin （古琴）music written in Jianzi Pu
notation, a deep learning system constructed by YOLOv5
was developed in this study. The main contribution
includes the making of dataset of a well-known Guqin
music Sen-O-So (仙翁操) and the restoration systemdeveloped with artificial intelligence (AI) technology.
The experiment results showed the effectiveness of the
proposed system, and the future work of this study is tobuild more single character datasets of Jianzi Pu using
more Guqin musical notations.

a. VGG16

Acknowledgements
This work was supported by JSPS KAKENHI Grant (No.
22H03709, No.22K12152)
References
1. China Script, “The difference between Qin and So”,
http://chugokugo-script.net/chugoku-bunka/kin.html(2016) (in Japanese)
2. JapanSocietyfor
Promotion
ofGuqin:
https://www.guqin.jp/about
3. R. Takao, “A survey and analysis of the Dapu process in
Qin music”, Journal of the Musicological Society ofOchanomizu University, No. 10, 2008 pp.40-45 (inJapanese)
4. Z.X. Pan, C. L. Zhou: “Text Segmentation and Extraction
from Images of Guqin Jianzi Pu”, Mind an Computation，
Vol.1, No.2, pp. 286-295 (2007) (in Chinese)
5. Chen Shi: “Guqin Notation and Music Style Recognition”,
Computer Science (2016)
6. 王 利，孫 洋， 罗 兆麟， 張輝：「AI 自动翻译“减字谱”——以《流水》和《不染》为例」，Art Education,
Vol.3 (2019) (in Chinese)
7. B. Yang, M. Sato, S. Kuremoto, M. Koshiba, S. Mabu, H.
Hieda, T. Kuremoto, “Recognition of Guqin music by deeplearning methods”, Proceedings of the Electronics,
Information and Systems Conference Electronics,
Information and Systems Society, I.E.E. of Japan，pp.512515 (2022) (in Japanese)
8. Bowen Yang, Shun Kuremoto, Mamiko Koshiba, Shingo
Mabu, Hiroo Hieda, Takashi Kuremoto, “Recognition of
Guqin Notation using Deep Learning”, in Proceedings ofInnovative Application Research and Education(ICIARE2022), pp.61-64 (2022)
9. J. Redmon, S. Divvala, R. Girshick, A. Farhadi, You Only
Look Once: Unified, Real-Time Object Detection,
aiZiv:1506.02640v5 (2016)
10. G. Jocher, https://github.com/ultralytics/yolov5/ (2020)

epoch
b. YOLOv5
Fig. 8 Learning curves of VGG16 and YOLOv5.

Fig.9 A sample of audio/video data matched to asingle character of Jianzi Pu.

Fig.10 A sample of the processing result of the
proposed system.
Single characters of Jianzi Pu were played by aprofessional Guqin player, Mrs. Yuki Takei [2] andrecorded in audio/video forms as shown in Fig. 9. The
order of single characters in a Guqin musical notion was©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
323

Powered by TCPDF (www.tcpdf.org)

Restoration of Guqin MusicMr. Hiroo Hieda
He is a Fellow of the Institute for Future
Engineering.
Longtime,hedidcomprehensive research on science and art.
Currently, he researches on the history of
Qin ( 七 絃 琴 ) in early modern Japan,
especially in the Edo period. He published a book named“A study to the history of Qin in early modern Japan” (近世琴学史攷) in 2018.

Authors Introduction
Prof. Dr. Takashi Kuremoto
He received his B.E. degree from the
University of Shanghai for Science andTechnology, China, in 1986, M.E. andPh.D.
degreesfrom
Yamaguchi
University, Japan, in 1996 and 2014,
respectively. He worked as a systemengineer at the Research Institute ofAutomatic Machine of Beijing from 1986 to 1992 and until
2021, he was an Assistant Professor at Yamaguchi
University. He was an Academic Visitor of the School ofComputer Science, the University of Manchester, U.K., in2008. Currently, he is a Full Professor with the Department
of Information Technology and Media Design, Nippon
Institute of Technology, Japan. His research interests
include artificial neural networks, fuzzy, evolutionary
computation, time series forecasting, and intelligent
systems

Prof. Dr. Shingo Mabu
He received his B.E and M.E. degrees
from Kyushu University, Japan, in 2001
and 2003, respectively, and his Ph.D.
degree from Waseda University, Japan, in
2006. From 2006 to 2007, he was a visiting
lecturer at Waseda University, Japan, andfrom 2007 to 2012, an assistant professor at the same
university. From 2012 to 2017, he was an assistantprofessor at Yamaguchi University, Japan, and from 2017
to 2020, an associate professor at the same university. He
is currently a professor at the Graduate School of Sciences
and Technology for Innovation at the same university.

Mr. Kazuma Fujino
He received his B.E. degree from Nippon
Institute of Technology, Japan, in 2023. Heworks at SET Software Co., Ltd., Tokyo,
Japan, currently.

Mr. Hirokazu Takahashi

He received his B.E. degree from
Nippon Institute of Technology, Japan,
in 2023.

Mr. Shun Kuremoto
He received his B. F. degree from Shanghai
University of Finance and Economics,
China, in 1989, and M. A. in Commerce
degree from Waseda University, Japan, in
1997. Currently, he is a guest researcher of
the Institute for Future Engineering, Japan, as well as
Director of Japan Society of Promotions of Guqin, and adoctoral program student in Yamaguchi University, Japan.
Dr. Mamiko Koshiba1.
She received her B.E., M.E. and Ph.D.
degrees from Tokyo University ofAgriculture and Technology, Japan, in
2001, 2002 and 2005, respectively. She
worked as a postdoctoral neuroscientist at
National Center of Neurology and Psychiatry 2005 to 2007
and until 2013, she was a Visiting Associate Professor atTokyo University of Agriculture and Technology and from2013 to 2016 at Saitama Medical University, Currently,
she was an Associate Professor of Yamaguchi University,
Japan. Her research interests include application of
artificial neural networks for neuronal system evolution
and development.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
324

Constructive Nurse Scheduling Using Reinforcement Learning
Considering Variations in Nurse Work Patterns
Masato Nagayoshi

Niigata College of Nursing, 240 Shinnan-cho Joetsu, Niigata 943-0147, Japan
E-mail: nagayosi@niigata-cn.ac.jp
Hisashi Tamaki

Kobe University, 1-1 Rokkodai-cho, Nada-ku, Kobe, Hyogo 657-8501, Japan
E-mail: tamaki@al.cs.kobe-u.ac.jp

Abstract
It is very difficult to create a work schedule that satisfies all the different requirements in nurse scheduling. For thisreason, numerous studies have been conducted on the nurse scheduling problem. However, the created shift schedule
is often not practical as it is, because adjustments including various constraints and evaluation criteria are required.
Therefore, we have proposed a work revision method using reinforcement learning in a constructive nurse schedulingsystem. In this paper, we investigate the feasibility of creating a practical work schedule even when there are nurses
with different work pattern evaluations, i.e., the feasibility of creating a practical work schedule that takes each nurse's
life stage into consideration.
Keywords: nurse scheduling, reinforcement learning, constructive search, work patterns, life stages

1. Introduction

2. Constructive nurse scheduling system

Various studies have been conducted on the nurse
scheduling problem [1], which is the creation of a shiftschedule for nurses. However, for practical use,
adjustments including various constraints and evaluation
values are required, and the created shift schedule is often
not practical as it is, so many head nurses still feelburdened by creating shift schedules [2]. Therefore, wehave proposed a work revision method [3] using
reinforcement learning [4] on a constructive nurse
scheduling system [5].
In this paper, we investigate the feasibility of creating apractical work schedule even when there are nurses withdifferent work pattern evaluations, i.e., the feasibility ofcreating a practical work schedule that takes each nurse's
life stage into consideration. In addition, the work patternevaluations of the obtained modified work schedules arevisualized. Furthermore, none of the previous studies onnurse scheduling considered the evaluation of different
work patterns.

2.1. Features
The features of the constructive nurse scheduling system[5] are as follows.
1. The system creates a schedule for each day, startingfrom the first day.
2. The priority calculation can be extended to consider
detailed conditions.
3. It does not consider the evaluation value for theentire shift schedule for a month.
2.2. Work Revisions
The constructive scheduling system considers only thebasic constraints that would be required in a hospital withmany nurses, and the possibility exists that a feasible
solution that does not satisfy the head nurse is obtained.
For this reason, Kurashige et al. [5] describe thefollowing two procedures for the actual modification.
(1) A work shift of the nurse in the case that does notsatisfy the head nurse is manually exchanged with awork shift of another nurse.
(2) A work shift of the nurse in the case that do not satisfy
the head nurse is exchanged to other work shift as
designated work shift, and the rescheduling is done.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
325

Masato Nagayoshi, Hisashi Tamaki

Next, we introduce our proposed system [3] that learns
this exchange procedure using reinforcement learning.

modifications depends on which work shift is being
exchanged.

3. Work Revision Method Using Reinforcement
Learning

3.2. RL Agent
Q-learning [6] is applied to the proposed method to learn
an appropriate exchange procedure. The state space of
the RL agent consists of 4 dimensions: the previous
exchange days (1 to 30), the total number of violations
by all nurses for evening, late-night shift, and holiday:
Vnw (w=1,2,3), to be a Markov decision process. The
number of possible actions is 4, which is the exchange ofevening, late-night, holiday, and night shift.
1 step is defined as 1 exchange including unsuccessful
cases, 1 episode is defined as the time when the shiftschedule reaches the target state, or 100 steps passed.
Here, the target state is defined as the sum of violations
for all nurses and shifts, ∑n ∑v Vnw = 0 (excluding
over-holiday violations). The positive reinforcement
signal 𝑟𝑡 = 10 (reward) is given only when the target
state is reached and the reinforcement signal 𝑟𝑡 = 0 is
given at any other steps. At the start of each episode, theshift schedule will be in its initial state.
4. Computational Experiments

3.1. Problem Setting for Reinforcement Learning
The shift schedule created by the constructive nurse
scheduling system, which is created in order from thefirst day, satisfies the shift constraints (such as thenumber of nurses required for each day). On the other
hand, the shift schedule for the entire scheduling period(e.g., one month) is checked, there may be several cases
in which the nurse constraints (such as the limited
number of workdays) are not satisfied for each nurse.
Therefore, the number of violations Vnw of work shift
w is calculated as the number of days exceeding UTnw,
the upper limit of the number of assignments of work
shift w to each nurse n, from the work schedule, and arevision is repeated according to the following:𝑚𝑖𝑛 ∑n ∑v Vnw(1)

The following procedure is to be used for one revision.
(1) Select a work shift w0 that is the source of theexchange (usually the one with the most violations).
(2) Determine the nurse n0 with the highest number ofviolations in the shift w0.
(3) If the shift w0 is the night shift, the shift w0 with thehighest number of violations, whether it is evening or
late-night shift, is designated as w0 for the nurse n0.
(4) If there is a work shift that is below the lower limit of
the number of assignments for the nurse n0, that work
shift w1 is designated as a destination of the exchange
shift. If not, day shift without the upper and lower limitsof the number of assignments is used as the exchange.
(5) Determine the day d0 with the highest priority among
the days when the shift w0 is exchanged to w1 for nursen0.
(6) Deduce the group g(j0) in which the nurse n0 is incharge of a job j0, which is assigned as the shift w0.
(7) Determine a nurse n1 who belongs to group g(j0) andwhose shift on the day d0 is w1. If there is more than one
applicable nurse, determine the nurse n1 with the highest
priority among the nurses when the shift w1 is exchanged
to w0 on d0.
(8) The nurses n0 and n1 are exchanged their shifts on d0.
In case there is no corresponding nurses in any of theprocedures, the exchange is not valid. In addition, it isalso not valid to undo a previous exchange.
Here, minimizing the number of violations is a very
difficult problem, because the number of possible

Table 1. Evaluations of work pattern for 2 days.
shift
on previous day

day
eveninglate-nightholiday

day
15
0
0
23

shift on the day
eveninglate-night
1
13
5
0
8
5
3
0

holiday

11
12
4
17

Table 2. Evaluations of work pattern for 2 days
allowing transitions from the evening shift to thelate-night shift.
shift
on previous day

day
eveninglate-nightholiday

day
15
0
0
23

shift on the day
eveninglate-night
1
13
2
8
8
5
3
0

holiday11
7
4
17

Table 3. Parameters for experiments
Parameter
Value
0.1
𝛼Q
0.9
𝛾
0.1
𝜏
4.1. Nurse Scheduling Problem
The extended method is applied to a nurse schedulingproblem like that of Kurashige et al. [5]. First, a three-©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
326

Constructive Nurse Scheduling Using

shift system (day, evening, and late-night shift) is
adopted, and the number of nurses is 23, including thehead nurse. Furthermore, the number of positions isclassified as 3 (head nurse, assistant head nurse, andgeneral), the number of teams is 2 (A and B), and the skill
level is 3 (experienced, mid-career, and new). The otherconstraints are outlined below.
⚫

4.3. Results
The average of the numbers of steps required foraccomplishing the task and the average of the total
number of violations when accomplishing the task in
Situations 1 and 2 were observed during learning over 20
simulations, as described in Fig. 1 and Fig. 2,
respectively.
It can be seen from Fig. 1 and Fig. 2 that (1) in Situation
1, the target state was reached in about 5 steps, and thenumber of violations was reached at 3, (2) in Situation 2,
the target state was reached in about 2 steps, and thenumber of violations was reached at 2. Thus, we
confirmed that a feasible modified work schedule could
be created not only in Situation 1, but also in Situation 2,
where nurses have different work pattern evaluations.
The average of each nurse's work pattern evaluations foreach day (the average of the previous day's and the day's
work pattern evaluation, and the day's and the next day's
work pattern evaluation) and the total average of thework pattern evaluations on the modified work schedulefor Situations 1 and 2, respectively, are shown in Figs. 3
and 4. In addition, the 24, 25, and 26 days of Staff 6,
indicated by the red boxes in Fig. 4, represent the averageof the pattern evaluation of the late-night shift, evening
shift, and late-night shift, respectively, and the 7, 8, and9 days of Staff 17 represent the average of the work
pattern evaluation of the holiday, evening shift, and latenight shift, respectively.
It can be seen from Fig. 3 and Fig. 4 that (1) the total
average of the work pattern evaluations is the same for
Situation 1 and Situation 2.
Although additional experiments are needed, setting thework pattern evaluations to values that each nurse
him/herself desires will enable him/her to realize a work
style (work pattern) that suits him/her life stage and, in
addition, will help him/her to understand the work
schedule creation process.

Restrictions on the number of nurses for each shift:(1) Required number of day shift on weekdays isgreater than or equal to 10.
(2) Required number of day shift for weekends andholidays is 5.
(3) Required number of late-night shift is 5.
(4) Required number of evening shift is 5.

Next, Table 1 shows the evaluation of work patterns for
2 days with M = 2. The evaluation of work patterns canbe set using the current work schedule as a reference.
Here, we investigate the feasibility of creating a feasible
modified work schedule in the situation where all nurses
have the work pattern evaluations shown in Table 1(Situation 1) and only Staff 6 and Staff 17, theexperienced nurses of Team A and Team B, respectively,
have the work pattern evaluations allowing transitions
from the evening shift to the late-night shift shown in
Table 2 (Situation 2).
4.2. RL Agent
In the state space of the RL agent, the total number ofviolations is assumed to be [0, 2] and can take 3 states.
The computational experiments have been done withparameters as shown in Table 3. In addition, all initial Qvalues are set at 5.0 as the optimistic initial values.

Fig. 1. Required steps for accomplishing the task.

Fig. 2. Total number of violations when accomplishing
the task.

Fig. 3 Average evaluations of work pattern in therevised work schedule in Situation 1.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
327

Powered by TCPDF (www.tcpdf.org)

Masato Nagayoshi, Hisashi Tamaki

4. R. S. Sutton and A. G. Barto, Reinforcement Learning, A
Bradford Book, MIT Press, Cambridge, 2018.
5. K. Kurashige, T. Hashimoto and Y. Kameyama, Nurse
Scheduling System in Consideration of Versatility,
Journal of Japan Industrial Management Association,
56(2), 109-120, 2005.
6. C. J. C. H. Watkins and P. Dayan, Technical note: QLearning, Machine Learning 8, 279-292, 1992.

Authors Introduction
Dr. Masato Nagayoshi
He is an Associate Professor of Niigata
College of Nursing. He graduated from
Kobe University in 2002 and received
Master of Engineering from Kobe
University in 2004 and Doctor ofEngineering from Kobe University in2007. IEEJ, SICE, ISCIE member.

Fig. 4 Average evaluations of work pattern in therevised work schedule in Situation 2.
5. Conclusion
In this paper, we confirmed that the proposed method
can create a feasible modified work schedule in asituation where there are nurses with different work
pattern evaluations, specifically, where there are twonurses who can work from evening shift to late-night
shift. In addition, the work pattern evaluations of theobtained modified work schedules were visualized andthe total average of the work pattern evaluations in thesituation was compared with the total average of the work
pattern evaluations in the normal situation where evening
shift to late-night shift is not possible, and we confirmedthat the evaluation was equivalent to that in the normal
situation.
Our future projects include to confirm the effectivenessof the proposed method in situations where nurses
evaluate even more diverse work patterns, etc.

Dr. Hisashi Tamaki
He is a Professor, Graduate School ofEngineering, Kobe University. He
graduated from Kyoto university in 1985
and received Master of Engineering from
Kyoto University in 1987 and Doctor ofEngineering from Kyoto University in
1993. ISCIE, IEEJ, SICE, ISIJ member.

Acknowledgements
This work was supported by JSPS KAKENHI Grant
Number JP19K04906.
References
1. A. Ikegami, A Model for the Nurse Scheduling Problem,
IPSJ SIG Notes, 5, 1-6, 1996. (in Japanese)
2. H. Adachi, S. Nakamura, M. Nagayoshi and N. Okamura,
A Survey on The Present Status about Required Time and
Recognition of Supports for Managers to Prepare a Work
Timetable in The Medical Treatment and Supervision Act
Ward, Journal of Japan Academy of Psychiatric and
Mental Health Nursing, 30 (1), 59-65, 2021. (in Japanese)
3. M. Nagayoshi and H. Tamaki, An Approach of
Exchanging Work Shifts Using Reinforcement Learning
on a Constructive Nurse Scheduling System, Journal of
Robotics, Networking and Artificial Life, 9(2), 154-158,
2022.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
328

A Basic Study on Indicator of Transfer Learning for Reinforcement Learning
Satoshi Sugikawa

Osaka Institute of Technology, 1-79-1 Kitayama, Hirakata City, Osaka, 573-0196 Japan
Email: satoshi.sugikawa@oit.ac.jp
Kenta Takeoka

Osaka Institute of Technology, 1-79-1 Kitayama, Hirakata City, Osaka, 573-0196 Japan
Naoki Kotani

Osaka Institute of Technology, 1-79-1 Kitayama, Hirakata City, Osaka, 573-0196 Japan

Abstract
Reinforcement learning requires a lot of learning time for the agent to learn. Transfer learning is a method to reduce
this learning time, but it has the problem that the user does not know which knowledge is effective in which
environment until it is learned. Therefore, it is necessary for the user to consider the relationship between the sourceand destination when transferring knowledge. Therefore, this study proposes Indicator of adaptation criteria that candetermine this relationship in advance. In simulations, we demonstrate the usefulness of the proposed method byusing some example problems.
Keywords: Reinforcement learning, Transfer learning, Maze problems

2. Reinforcement learning

1. Introduction

Reinforcement learning is a branch of machine learningin which an agent learns through interaction with its
environment. Reinforcement learning is characterized by
the fact that the agent chooses an action to achieve acertain goal and receives a reward signal for that action
as it learns. Optimal behaviour rules are learnt by
repeating steps 1 to 4 below.

In recent years, machine learning has achieved rapid
growth with results in various fields such as natural
languageprocessing
and
image
processing.
Reinforcement learning [1], [2] which is capable of selflearning, is expected to further develop in all fields in the
future. Reinforcement learning has the problem that it
requires a lot of learning time because the agent learns
from scratch in a new environment by trial and error.

1.
2.

Many studies have been conducted to reduce learning
time, one of which is transfer learning. Transfer learning
is a method of adapting and reusing previously learned
material from similar tasks for a new task. Since there isno need to relearn, learning time can be reduced.
However, the effectiveness of the transferred material is
not known until it is transferred and learned. Therefore,
when transfer learning is performed, the user needs toconsider the relationship between the transferee and thetransferee source, but even then, there is a possibility thatthe learning will not be successful and negative transfers
will occur [3]. Therefore, this study proposes several
indicators of adaptive criteria that can discriminate inadvance the validity of knowledge.

3.
4.

Agent observes the state
Decides on a course of action based on cues frommeasures
Memorizes experience of which states, which
actions and which rewards
Seeks measures based on experience

Reinforcement learning is also formulated as a Markov
decision process, expressed as M=(S,A,P,R) as follows.
⚫
⚫
⚫
⚫

S: Set of states
A: Set of actions
P: State transition 𝑝(𝑠𝑡 = 𝑠 ′ |𝑠𝑡 = 𝑠, 𝑎𝑡 = 𝑎)
R: Behaviour rules π(𝑠, 𝑎) = 𝑅(𝑠𝑡 = 𝑠, 𝑎𝑡 = 𝑎)Reinforcement learning aims to acquire behaviour rulesthat maximize the expected reward.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
329

Satoshi Sugikawa, Kenta Takeoka, Naoki Kotani.

2.1. Q-learning
𝑄𝑐 (𝑠, 𝑎) = 𝑄𝑡 (𝑠, 𝑎) + τ𝑄 𝑠 (𝑠, 𝑎)
In this study, Q-learning [3] was used as a reinforcement
learning algorithm learning updates the Q-values that theagent associates with a combination of states and actions,and finds an action rule that selects the action with the
maximum Q-value in each state. The formula for
updating the Q-values is as follows.

4. Similarity Indicators
The following five indices are used to compute thesimilarity.
4.1. Similarity by pixel value
In this calculation example, the similarity by pixel value
is determined. As the mazes used in this study have thesame size, the percentage of pixel values matchedbetween the images is determined.

𝑄(𝑠𝑡 , 𝑎𝑡 ) = 𝑄(𝑠𝑡 , 𝑎𝑡 ) + η∗ (𝑅𝑡+1 + γ max 𝑄 (𝑠𝑡+1 , 𝑎)
− 𝑄(𝑠𝑡 , 𝑎𝑡 ))In this study, the ϵ-greedy method was also used as theagent's action selection method. In the ϵ -greedy method,
the agent acts randomly with a probability of ϵ andselects an action with a probability of 1- ϵ with amaximum Q value.

4.2. Cosine similarity between maze sequences
In this example calculation, the maze is transformed into
arrays and the cosine similarity between the source anddestination arrays is calculated. The transformed array is
the same as the array used for training, with the mazewalls set to 1 and the paths set to 0.
⚫ s is a vectorization of the base maze
⚫ t is a vectorization of the destination maze.
∑𝑖 𝑠𝑖 𝑡𝑖𝑐𝑜𝑠(𝑠, 𝑡) =√∑𝑖 𝑠𝑖2 √∑𝑖 𝑡𝑖2

3. Transfer learning
In transfer learning [4], [5], the goal is to reduce
learning time by transferring knowledge learned at thetransfer source as prior knowledge at the transfer
destination with similar tasks. However, if there is no
similarity between the source and destination, a negativetransfer may occur. Therefore, the user must consider thesimilarity between the source and destination in advance.
Therefore, it is necessary to determine the similarity
between the source and destination to determine whichknowledge is transferred to which destination.

4.3. Similarity using maxQ as weights
In this calculation example, maxQ is used to determine
the similarity. The total of the differences between themazes is calculated using maxQ of maze A, which is thesource of the transition, as the weight. The sum of thedifferences between the mazes is calculated using maxQof maze A as the weight, and divided by the total valueof maxQ.

In this study, we aim to formalize the adaptation
criterion by determining the similarity between mazesusing the maze problem.
3.1. learning model

p=1−

The algorithm used for learning in reinforcement
learning is Q-learning. There are five actions that theagent can take: up, down, left, right, left and right, and noaction. The reward design is +1 if the agent reaches thegoal, and -0.01 if it collides with a wall or a step elapse.

∑i(si − t i )maxQ si ,aa∑i maxQ si ,aa4.4. Cosine similarity using maxQ
Finally, in this calculation example, the Q values
obtained after learning are used to obtain the cosine
similarity. The Q values obtained after normal learning ineach maze (12 × 12 × 5) are changed to an array of
12 × 12by finding the maximum value (maxQ) in eachsquare. However, this is not known before application as
it is an outcome. It is calculated as part of the evaluation
for measuring similarity.

3.2. transition method
As the transfer method uses Q-learning, the transfer
learning adopts the value function transfer type.
The Q-table obtained by the learning of the transfer
source is transferred to the transfer destination agent.
The degree of re-use of the behavioural value function isadjusted using the transfer rate τ. The transfer rate τis adjusted in the range of 0 < τ < 1. In this study, τ
was set at 0.5.

cos(Maxa Q s , Maxa Q t ) =∑i Maxa Q si ,a Maxa Q ti,a√∑i Maxa Q2si ,a √∑i Maxa Q2ti,a©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
330

A Basic Study on

S The Similarity results show that maze B, maze D, and maze
C have the highest similarity in that order. Similarity by pixel
value, similarity between maze sequences, and similarity using
Qmax, in that order. The use of Q-values allows us to focus on
differences only in important locations. Fig. 7, Fig. 8, and Fig.
9 show results of the transfer learning. Transition from

5. Simulation
Five mazes were prepared to test the usefulness of theindex. Maze A is the original maze and was applied tomazes B, C, D, and E, respectively. Fig. 1, Fig. 2, Fig. 3,
Fig. 4, and Fig. 5 show the mazes.

Fig.1 Maze A

maze A to maze C is not shown in the figure because thegoal was not achieved. Reinforcement learning isrepresented by the red line and transfer learning by theblue line.

Fig.2 Maze B

Fig.7. Transfer learning from maze A to maze B, left panel
shows number of steps to goal and episodes, right panel shows
total reward and number of episodes.

Fig.3. Maze C

Fig.4 Maze D

Fig.8. Transfer learning from maze A to maze D, left panel
shows number of steps to goal and episodes, right panel shows
total reward and number of episodes

Fig.5 Maze E
The similarity of the mazes to each indicator is shownin Table 1.
Table 1 Similarity Results
Maze
A
to
Maze BMaze
A
toMaze C

Maze
A
toMaze DMaze
A
toMaze E

Similarity
by pixel value

82.95

77.08

80.02

79.44

Similarity
between mazesequences

80.29

75.96

78.71

77.66

Similarity
using Qmax

98.86

60.96

82.49

77.37

Cosinesimilarity
using Qmax

72.97

5.89

41.17

28.91

Fig.9. Transfer learning from maze A to maze E, left panel
shows number of steps to goal and episodes, right panel shows
total reward and number of episodes
From these figures, the goal is reached faster with transfer
learning than with reinforcement learning. This indicates thattransition learning is effective in these mazes. Based on the
results in the figures and the cosine similarity using Qmax, it
seems reasonable to focus on the Q value.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
331

Powered by TCPDF (www.tcpdf.org)

Satoshi Sugikawa, Kenta Takeoka, Naoki Kotani.

Fig. 10 shows the results of applying reinforcement learning
to the mazes. Fig. 11, Fig. 12, Fig. 13 and Fig. 14 show theresults of applying transition learning to each maze. Red
indicates locations with high rewards. The reward of maze A
was applied to each maze. Therefore, the maze was routed
from the top towards the goal. This demonstrates the validityof this simulation.

2.

3.
4.

5.

Leslie Pack Kaelbling. Michael L. Littman and Andrew W.
Moore: Reinforcement Learning-A Survey. Journal of
Artificial Intelligence Research.vol.4.237/285（1996）
C. J. C. H. Watkins: Learning from Delayed Rewards, PhD
thesis, King's College, Cambridge, UK（1989）
Haitham B. Ammar, Karl Tuyls, Matthew E. Taylor, Kurt
Driessens, Gerhard Weiss: Reinforcement Learning
Transfer via Sparse Coding, Proceedings of the 11th
International Conference on Autonomous Agents andMultiagent Systems（2012）
Matthew E. Taylor, Peter Stone: Transfer Learning for
Reinforcement Learning Domains, A Survey ,Journal of
Machine Learning Research, vol.10,1633/1685（2009）

Authors Introduction

Fig.10ResultofReinforcement Learning
on maze A

Fig.12 Result of Transfer
learning on maze C

Dr. Satoshi Sugikawa
He received his Dr. Eng. degrees from
Kobe University, Japan, in 2011. In
2013, he joined Osaka Institute ofTechnology, where he is currently an
Assistant Professor. His researchinterests
includeMathematical
Optimization. He is a member of IEEJ

Fig.11 Result of Transfer
learning on maze B

Mr. Kenta Takeoka
He received his Bachelor’s degree inEngineering in 2022 from the Faculty ofInformation Science and Technology,
Osaka Institute of technology in Japan.
He is currently a master student in
Osaka Institute of Technology, Japan

Fig.13 Result of Transfer
learning on maze D

Dr. Naoki Kotani

He received a Ph.D. degree from
Osaka University, Osaka, Japan, in
2011. He is an assistant professor ofInformation
ScienceandTechnology at Osaka Institute ofTechnology. His research interests
machine learning and robotics.
Kotani is a member of the ISCIE,
the SICE, the RSJ, the JSAI and hte IEEE.

Fig.14 Result of Transfer
learning on maze E
6. Conclusion
In this study, we conducted basic research on measures
focusing on environmental similarity in reinforcement
learning. Simulation results confirmed the validity offocusing on maxQ. The validity of the indicator was also
confirmed to some extent. In the future, the creation of
more accurate indicators is required.
References
1.

Sutton, R, S., Barto, A, G: Reinforcement Learning: An
Introduction. The MIT Pres（2007）©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
332

Machine Learning Approachto Predict Cooling Load for Existing Buildings
Makoto Ohara

Department of Information Technology, International Professional University of Technology in Osaka, 3-3-1
Umeda, Kita-ku, Osaka, 530-0001, Japan
Hideo Isozaki

Carbon Neutral Promotion Headquarters, Kobe University, 1-1, Rokkodai-cho, Nada-ku, Kobe 657-8501, Japan
E-mail: ohara.makoto@iput.ac.jp, isozaki@person.kobe-u.ac.jp
www.iput.ac.jp/osaka/

Abstract
The objective of this study is to predict air conditioning loads for existing buildings using operational data, weather
forecasts and visitor forecasts. The proposed prediction method is based on a neural network approach. However, it
is important to note that the proposed method does not learn the entire loads. Loads are divided into factors which
can be predicted by traditional thermodynamics and factors which are subject to machine learning. The proposed
method has been applied to an example instance using operational data from an underground mall in Kobe, and its
validity has been confirmed.
Keywords: Cooling Load Prediction, Air Conditioning System, Existing Building Data, Machine Learning

1.

Therefore
historicaloperatingdata
andcorresponding load data exist, they can be used forforecasting by machine learning. The large amountof training data can be used for the load prediction
without having to establish physical causal
relationships between load calculation conditions
and load values.
2. The actual situation regarding how the building isused is known, not the assumptions made at the timeof design.
3. The process of changing the parameters andimproving the accuracy of the predictions is
expected, as results are immediately available on
the correctness of the predictions.
The scope of this study is as follows.
1. The proposed approach predicts the hourly heatsource load for the next day in the existing building.

1. Introduction
If air-conditioning load are accurately predicted, it isexpected which the system is optimally operated, e.g.
setting of changes in chilled water supply temperature,
selection of heat source models, setting of operation
times, for saving energy [1].
The purpose of this study is to predict future load of an
air conditioning systems on the next day for which exists,
using operational data of existing buildings, weather
forecasts and visitor volume forecasts. The prediction
algorithm is combining mathematical calculation andneural network (hereafter NN) methods. The proposed
method has applied to an example based on operational
data of an underground shopping mall in Kobe andcompared with the actual measured cooling load.
Air conditioning load prediction for existing buildings
differ from load calculations for new buildings in thefollowings.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
333

Makoto Ohara, Hideo Isozaki

2.

3.

The target of forecasting is limited to cooling loads,
where more energy savings are expected, and does
not deal with heating loads.
If the predicted cooling load is spatially subdivided,
the approach is more valuable for air-conditioning
operations, although, load performance monitoring
points must also be taken at the same granularity.
The detailed measurement has become realistic in
recent years due to advances in IT technology,
however, this research aims the cooling load
forecast for the existing building, considering the
suppression of renovation work costs.

6.

for this load is the past room temperature of theroom concerned (going back about 20 hours).
System losses, .e.g. fan reheat, duct losses andmixing losses.

2.2. Load Factors Subject to Actuarial
Calculations
7.

Load due to human metabolism. The explanatory
variables for this load are current manpower, past
manpower (going back about 10 hours), andaverage work intensity.
8. Load due to heat dissipation of lighting fixtures.
The explanatory variables for this load are current
power consumption and past power consumption(going back about 10 hours).
9. Load due to heat dissipation from other indoor
equipment. The explanatory variables for this load
are current heat generation, and past heat generation(going back about 10 hours).
10. Outdoor air load. The explanatory variables for thisload are outdoor air intake, outdoor air temperature,and outdoor air enthalpy.

2. Air-Conditioning Load
With reference to the dynamic load calculation method
based on the response factor method [2], [3], the airconditioning loads were classified into two categories:
those estimated by mathematical calculations and thoseestimated by machine learning, assuming that eachelement has its own explanatory variables. The
explanatory variables for each load element wereselected on the basis of whether they could be measured
in a typical building.
In this section, the term "current" indicates the point intime at which the forecast is made.

3. Prediction Method for Cooling Load
A prediction method which combines mathematicalcomputation and machine learning approach is build.

2.1. Load Factors Subject to Machine Learning

3.1. Learning Phase1.

When training with measurement data, the data set iscomposed for NN [3] as follows.
• Teaching data are the differences between theoverall loads and the loads of Section 2.2.
• Input data are the explanatory variables for loads ofSection 2.1.

2.

3.

4.

5.

Load of exterior wall through-flow heat due totemperature difference between interior andexterior including through-flow load from glass.
The explanatory variables for this load are: current
outdoor air temperature, past outdoor air
temperature (going back about 50 hours), past daily
average outdoor air temperature (going back about
3 months), current horizontal solar radiation, andpast horizontal solar radiation (going back about 50
hours).
Load of interior wall penetration heat from adjacentroom. The explanatory variables for this load are thecurrent outdoor air temperature and the past outdoor
air temperature (going back about 50 hours).
Load due to solar radiation transmitted throughglass. The explanatory variables for this load are
current daylight hours, horizontal solar radiation,and historical horizontal solar radiation (going backabout 10 hours).
Load due to drafts. The explanatory variables forthis load are current outdoor air temperature, current
outdoor air humidity, current external winddirection and speed, and building opening condition.
Load due to heat storage of intermittent air
conditioning operation. The explanatory variable©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
334

Machine Learning Approach to

Total Load for Learning

Outside Temperature

Total Load forActuarial Calculation

Fig. 1 Monthly Cooling Load of the Example Instance in 2018 and 2019.
4.

4.2. Learning setting

Computational Experiment

Since the subject is an underground mall, the effect ofsolar radiation was ignored among the air conditioning
load elements. See [4] for the measurement method of thecurrent people flow. The settings of the NN method areshown in Table 2.

4.1. Example
An example of air conditioning load forecasting is
consists of two years of measured data for the overall air
conditioning load and the explanatory variables for each
air conditioning load for the underground shopping mallin Kobe, see Table 1. Fig. 1 shows the cooling loads of 2
years.

Input Layer

Table 2. Setting of NN method.
Input Size
28

Intermediate
Layer

Table 1. Outline of Underground Mall
Name
SanticaYear of construction 1963 (1st phase)
1966 (2nd phase)Management Entity Kobe Chikagai Co., Lt
Total floor area
19,109m2
Retail area
10,145m2
Business Hours
10:00-20:00 (merchandising)
11:00-21:00 (food andbeverage)
6:00 - 24:00 (passageway)

Numberof Layers

2

Type

Fully-Connected
Layer
ReLU

Activation
Function
Number of Units

20 (first layer)
10 (second layer)
Fully-Connected
Layer

Output Layer

Type

Others

Output Size
Batch Size

1
10Epoch Size
Trial

30
10

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
335

Powered by TCPDF (www.tcpdf.org)

Makoto Ohara, Hideo Isozaki

4.3. Prediction setting
In this manuscript, calculations are performed on
historical data sets to verify the performance of the NN
method. Where weather forecasts or other forecastingdata should be used in actual use, measurement values
are used. The people-flow prediction method is describedin [5].

3.
4.

4.4. Result
Fig. 2 shows the difference between normalized
predicted and actual measurements by time period. The
errors are particularly large between 8:00 and 10:00 andat 20:00. The prediction accuracy is expected to improvewhen the conditions at the start and end of cooling are
considered.

5.

Model and Examining the Identification Accuracy”,
Proceedings of Annual Conference of the Society ofHeating, Air-Conditioning and Sanitary Engineers ofJapan, E-6, 2020. (In Japanese)
T. Okatani, “Deep Learning”, Kodan-sha, 2015. (InJapanese)
M. Ohara, T. Matsumoto, Y. Suzuki, H. Nagahiro, H.
Tamaki, “An Estimation Method for People Flow with
Laser Rangefinders on an Underground Walkway”,
Proceedings of Annual Conference of the Institute of
Systems, Control and Information Engineers, 2018. (InJapanese)
M. Ohara, T. Matsumoto, K. Sakakibara, Y. Suzuki, H.
Nagahiro, H. Tamaki, “Pedestrian Density Prediction
Method for Air Conditioning System on UndergroundMall with Exterior Openings”, Proceedings of SICE SSI,
2020. (In Japanese)

Authors Introduction
Dr. Makoto Ohara
He received Doctor of Philosophy in
Engineering from Kobe University in 2012.
He works as an assistant professor inInternational Professional University ofTechnology in Osaka now. His researchinterests optimization for social systems. In
recent years he has also research intomachine learning approaches.
Mr. Hideo Isozaki
He received his Master in Engineeringfrom Waseda University in Japan. He is avisiting professor Carbon Neutral
Promotion Headquarters, Kobe University
in Japan.

Fig. 2 Result of Prediction.
5. Conclusion
In this study, a method for predicting air-conditioning
loads is investigated with the goal of efficient airconditioning operation in underground malls with open
areas. The proposed method has been applied theexample based on actual data from the underground mallin Kobe. Future issues include evaluation of predictions
based on actual use and examination of accuracy incomparison with existing studies.
References
1.

Y. Matsuo, K. Yokoyama, H. Ishino, S. Kawamoto,“Introduction to Dynamic Heat Load Calculations
for Air Conditioning Systems”, Japanese
Association of Building Mechanical and ElectricalEngineers, 1980. (In Japanese)2.

H. Oda, M. Nakao, S. Todo, H. Isozaki, “Study on an
Identification Model of Cooling Coil Characteristics of
Air Conditioners (Report 1) Setting Up a Gray-Box©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
336

Intelligent Logistics Handling Robot: Design, Control, and Recognition
Yanchao Bi

Tianjin University of Science and Technology, 1038 Dagu South Road, Hexi District, Tianjin, 300222, China
Jiale Cheng

Tianjin University of Science and Technology, 1038 Dagu South Road, Hexi District, Tianjin, 300222, China
Limei Wang

Tianjin University of Science and Technology, 1038 Dagu South Road, Hexi District, Tianjin, 300222, China
Yizhun Peng

Tianjin University of Science and Technology, 1038 Dagu South Road, Hexi District, Tianjin, 300222, China
Email: pengyizhun@126.com

Abstract
This study aims to investigate various aspects of intelligent logistics handling robots, including mechanical design,
automatic control, and image recognition. With the continuous development of the logistics industry andadvancements in automation technology, intelligent logistics handling robots play a crucial role in improving logistics
efficiency and reducing costs. Leveraging existing technologies, we have designed and developed an omnidirectional
mobile intelligent logistics handling robot equipped with a SCARA-type robotic arm. The robot integrates functions
such as task acquisition, global positioning, material detection, warehouse identification, material handling, andstacking, achieving a fully automated and streamlined logistics handling process.
Keywords: Logistics handling robots, Automatic control, Image recognition, Global positioning

structure chassis and planar articulated (SCARA) roboticarm [1], [2], [3], [5]. It integrates functions such as
positioning, environmental perception, path planning,material recognition and grasping, material placement,
and stacking. This integration significantly reduces thenumber of operators, enhances the speed and accuracy oflogistics operations, and greatly improves the efficiencyof the logistics system.

1. Introduction
In modern industrial production, logistics handling is atime-consuming and labor-intensive task. The modern
industry faces rapidly changing market demands and acomplex and dynamic production environment, making
traditional manual logistics handling methods relatively
outdated. In this situation, there is an urgent need for amore flexible, efficient, and adaptive logistics handlingapproach to meet the requirements of modern industry.
This new type of handling method should be able to adapt
to changes in different factory environments andproduction processes, completing a large number ofhandling tasks in a short time to improve production
efficiency. This is precisely the significant advantage that
intelligent logistics handling robots can bring to modern
industry.

2. Intelligent Logistics Handling Robot Overall
Program Design
2.1. Operational site
In this thesis, the robot's operation site is based on the
9th China University Students Engineering Practice andInnovation Competition "Intelligent +" track logistics
handling robot competition. The robot drives on the graylane, and the rest of the area is matte white or yellow. In
the competition venue, the start-stop zone, raw material
zone, roughing zone and temporary storage zone are set
up. The start-stop zone is blue in color and is used for the
robot to go back and forth. Fig. 1 shows the exact locationof the given raw material area, roughing area and staging
area within the competition venue.

The intelligent logistics handling robot studied in thisproject comprehensively designs and analyzes aspects
such as mechanical structure, robotic arm control andcoordinate calculation, chassis control and global
positioning, image recognition, and processing. A
multifunctional intelligent logistics handling robot(AGV) is designed based on the omnidirectional wheel©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
337

Yanchao Bi, Jiale Cheng, Limei Wang, Yizhun Pengchassis control board, includes external devices such as a
QR code recognition module, gyroscope, touchscreen,
and brushless motor driver. The robotic arm motion
control system integrates servos and utilizes closed-loop
stepper motors for precise control of the robotic arm. The
visual module, developed based on OpenCV, includes
functions for material color recognition and target ring
center recognition. The chassis control board, robotic arm
control board, and visual module are all powered
simultaneously by the power module.

Fig. 1 Schematic diagram of the robotics field
The raw material area adopts a round electric turntable
to arrange materials, and materials are identified by color.
The top surface of roughing area, staging area, finishingarea and finished product area are equipped with color
rings and or circles for measuring the accuracy ofmaterial placement.

Fig. 2 Overall design schematic

2.2. Mandate requirements

3. Design of Mechanical Structural Systems

The robot follows a "one-key" start-up procedure,
moving from the start-stop zone to the QR code panel
within a specified time to read the QR code. It obtains
two handling tasks (the sequence for handling materialsof three different colors). According to the task
requirements, the robot moves between the material area,rough processing area, and temporary storage area,
carrying out placement and stacking tasks. Aftercompleting the tasks, it returns to the start-stop zone.

3.1. Intelligent logistics handling robot overallmechanical structure design
To ensure the flexibility and efficiency of the robot, thechassis adopts a Mecanum wheel chassis, and themechanical arm structure adopts a standard three-axis
SCARA structure.

2.3. Design proposal
According to the task requirements, the overall designof the intelligent logistics handling robot is illustrated inFig. 2. In terms of mechanical structure, the chassis
adopts a Mecanum wheel [1], [5] double-layer shockabsorbing design, controlled by DJI brushless motors toachieve closed-loop control. The robotic arm employs aplanar articulated (SCARA) design, driven by stepper
motors and closed-loop stepper motors, allowing realtime reading of the angles of each joint for feedback
control.
The electronic control system is mainly divided into
three parts: chassis motion control system, robotic arm
motion control system, and visual recognition system.
The chassis motion control system, centered around thea.

Robot 3D drawing©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
338

Intelligent Logistics Handling Robot

b. Bottom viewFig. 4 Chassis structureb. Robot Physical Picture
Fig. 3 Image of Intelligent Logistics Handling Robot

3.3. Mechanical structure design of mechanical arm
This robotic arm is of the three-axis SCARA type [2],made of PLA material and manufactured using 3D
printing technology. It is lightweight in quality.

3.2. Chassis mechanical structure design
The chassis adopts a double-layer design with
Mecanum wheels, made of aluminum alloy withelectroplating process. The lower layer of the chassis
houses the electronic control system, while the upper
layer accommodates the mechanical arm, material
storage, and human-machine interaction module, among
others. The overall structure is well-defined.
The motion system of the chassis requires a high levelof grounding for the Mecanum wheels [5]. To ensurestable movement of the robot, the lower layer of thechassis adopts a double-plate coaxial structure, ensuringthat all four wheels touch the ground simultaneously.
This design increases shock absorption and solves the
problem of Mecanum wheels slipping when not in
contact with the ground [1].

Fig. 5 3D model of the robot arm
The upper arm of the robotic arm is connected to thearm axis through a linear bearing, while the lower arm isequipped with a thrust ball bearing to withstand axial
forces and reduce friction between the upper and lower
arms. The robotic arm can achieve three degrees of
freedom through the motion of stepper motors, including
vertical movement, rotation of the upper arm, androtation of the lower arm.
3.4. Mechanical Structure design of mechanical claw
The mechanical claw is one of the key components ofan intelligent logistics handling system. It is
manufactured using 3D printing technology and made of
PLA material.

a.

Top view©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
339

Yanchao Bi, Jiale Cheng, Limei Wang, Yizhun Peng

Z-axis. The process of solving the operational sciencemodel of the three-axis SCARA robotic arm is as follows:
Determination of joints, connecting rods and theirparameters. The robot arm uses a rotating chassis at thebottom, lifting and lowering using a stepper motor screw
Z-axis, and a rotating arm in the XY plane, for a total ofthree degrees of freedom, and the analysis process is
simplified by dropping the end-effector [3].
Simplify the model to set the coordinate system. Withthe vertical upward as the +Z-axis and the center of the
Z-axis as the coordinate origin from the top view,
establish a plane right-angle coordinate system with the
body as the reference system.
a.

3D drawing

Fig. 7 Simplified coordinate system model
As shown in Fig. 7, for the big arm of the robot arm, forthe small arm of the robot arm. For the rotation angle ofthe big arm, controlled by the bottom disk; for the rotation
angle of the small arm, controlled by the stepping motor
through the synchronous belt.

b. Concrete drawing
Fig. 6 Robotic Gripper Structure
The mechanical claw is used for grasping andidentifying materials. When grasping materials, themechanical claw needs to firmly grip the materials. The
mechanical claw is divided into active and passive parts,
with the active part being directly driven by a servo motor
to rotate, and the active part drives the passive part torotate through gears. The identification of materials
involves the use of a camera and a visual recognition
system. The camera is installed in the center of themechanical claw.

Inverse kinematic modeling calculations. We set the
point coordinates to (x, y, z), the length of the big arm to
be L1, and the length of the small arm to be L2, then thelength L3 = √𝐿12 + 𝐿22 . The angle of rotation of thelarge arm 𝜃1 and the angle of rotation of the small arm 𝜃2need to be solved for. This leads to Eq. (1). The rotation
angle of the big arm 𝜃1 and the rotation angle of the smallarm 𝜃2 can be found (Eq. (2)).
𝐿22 + 𝐿32 − 𝐿12
2 ∗ 𝐿2 ∗ 𝐿3
𝐿12 + 𝐿22 − 𝐿32{ cos(𝜃2 ) = − 2 ∗ 𝐿1 ∗ 𝐿2
𝐿22 + 𝐿32 − 𝐿12𝜃1 = 𝑎𝑟𝑐𝑐𝑜𝑠()
2 ∗ 𝐿2 ∗ 𝐿3
2
2
𝐿1 + 𝐿2 − 𝐿32𝜃=𝑎𝑟𝑐𝑐𝑜𝑠(−){ 2
2 ∗ 𝐿1 ∗ 𝐿2

4. Three-axis SCARA Robot Kinematic Modelcos(𝜃1 ) =4.1. Kinematic modeling of a three-axis SCARA
robotic arm
SCARA robotic arms are usually characterized by highaccuracy, high speed and repeatability [2], [3]. In thisproject, the SCARA robotic arm is designed as a threedegree-of-freedom, containing one rotational degree of
freedom in the XY plane, one lifting degree of freedom
in the Z-axis, and one moving degree of freedom in the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
340

(1)

(2)

Intelligent Logistics Handling Robot

4.2. Robotic arm electronic control system designcode logic is confusing and redundant. This system uses
structure variables to save the spatial coordinates of eachstep of the robotic arm action and information such as
gripping operation, and establishes structure arrays forstoring the information of the robotic arm action during
the whole operation process. After the robotic arm is
powered on, it initializes the peripherals and stepper
motors, servos and so on. After the initialization iscompleted, turn on the serial port to receive interrupts andwait for the chassis controller to send task execution
instructions. When the task execution instruction isreceived, the robotic arm starts to move, reads the action
information in the structure array, and executes it step bystep. The vision system will send the material color,target ring coordinates and other information to therobotic arm, and the robotic arm will calibrate and correct
the position through this information. When all theactions are completed, the robotic arm task is completed
and returns to the initial state. The workflow of therobotic arm motion control system is shown in Fig. 9.

(1) Robotic arm control board circuit design
The robotic arm control board uses STM32F405RGT6
as the main control chip [4], with an externally integrated
power module, stepper motor drive module andcommunication module. The power supply module part,the input voltage is 12V, using TPS5430 decompression
to generate 5V voltage, and then through the AMS1117
on the 5V voltage regulator to generate 3.3V, thus
generating all the voltages required for system operation.
The control board integrates the TMC2209 stepper motor
driver, which is powered by 12V and has the advantagesof silence, high speed and low failure, and is used foropen-loop control of the stepper motor. The closed-loop
stepper motor driver is controlled using a UART interface.
In order to realize a UART interface and multiple closedloop stepper motor driver communication problems, weuse a diode plus pull resistance way to enhance the serial
port receiving ability, shielding the interference signal.
(2) PCB Overall Layout
The control board of the robotic arm is a four-layer
board, and the shielding layer is set up to shield the power
supply and signals, which improves the signal stability ofthe system. As Fig. 8 are the front and back of the chassis
control board.

a.

Fig. 9 Flow chart of robotic arm motion control
systemTop view

5. Chassis Motion Control System Design
5.1. Mecanum wheel chassis kinematics model
The Mecanum wheel platform kinematic model is toestablish the relationship between the rotational speedof the four Mecanum wheels and the speed of thegeometric center [4]. The positive kinematics model is
to know the rotational speed of the four Mecanum
wheels and calculate the speed of the Mecanum wheel
platform center point CENTER; the inverse kinematics
model is to know the speed of the Mecanum wheel
platform center point CENTER and calculate the speedof the four Mecanum wheels.

b. Bottom viewFig. 8 Robotic arm control board(3) Robotic arm software design
The robotic arm performs numerous actions, and theuse of sequential execution of robotic arm actions will
occupy a large amount of program storage space, and the

As an example, the inverse kinematics model of the
Mecanum wheel is used to establish a coordinate©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
341

Yanchao Bi, Jiale Cheng, Limei Wang, Yizhun Pengsystem with the geometric center point CENTER of the
top view of the chassis as the origin [5]. The distance
between the CENTER point and the point of contact
between the Mecanum wheel and the ground is 𝑟. 𝑟𝑥
and 𝑟𝑦 denote the projected distances on the x-axis andy-axis, respectively, of the CENTER coordinate system(both positive). The velocity of the CENTER point is
known to be [𝑣𝑐 𝑤𝑐 ]𝑇 . 𝑣𝑐𝑥 and 𝑣𝑐𝑦 are thecomponents of a on the x and y axes, respectively.

Mecanum wheel as the encoding wheel. When the
robot starts to move, the Mecanum wheel starts to shift
rotation. According to the positive kinematics model ofthe Mecanum wheel, the velocity of the robot at thecurrent moment can be obtained from the rotational
speeds of the four wheels with respect to the current
coordinate system of the robot itself [5]. Assuming thatthe angle between the current robot's own coordinate
system and the world coordinate system is θ (Fig. 11).

Fig.10 Decomposition of Mecanum Wheel Platform
MotionFig.11 World Coordinate System and Own
Coordinate System

After calculation, the inverse kinematic equation ofMecanum's wheel can be obtained.

The expression for converting the robot's velocity in its
own coordinate system to the velocity in the world
coordinate system is Eq. (5).

𝑤1
1
1 1
𝑤2
[𝑤 ] =
3
𝑅 1
𝑤4
[1

−1
+1
−1
+1

−(𝑟𝑥 + 𝑟𝑦 )𝑣−(𝑟𝑥 + 𝑟𝑦 ) 𝑐𝑥
[𝑣𝑐𝑦 ]+(𝑟𝑥 + 𝑟𝑦 ) 𝑤𝑐+(𝑟𝑥 + 𝑟𝑦 )]𝑣𝑥
𝑐𝑜𝑠𝜃0 𝑣𝑐𝑥 + 𝑠𝑖𝑛𝜃0 𝑣𝑐𝑦𝑣[ 𝑦 ] = [−𝑠𝑖𝑛𝜃0 𝑣𝑐𝑥 + 𝑐𝑜𝑠𝜃0 𝑣𝑐𝑦 ]
𝑤𝑧
𝑤𝑐

(3)

With the same time units, the conversion to adisplacement expression is Eq. (6).

The positive kinematics equations are based on therotational speeds of the four Mecanum wheels to
calculate the velocity of the point CENTER, and thetheoretical analysis is similar to the inverse kinematics,
and the simplest way to do this is to invert Eq. (3), which
can be expressed as Eq. (4).
+1
𝑣𝑐𝑥
𝑅 −1
[𝑣𝑐𝑦 ] = [ −14𝑤𝑐
𝑟𝑥 + 𝑟𝑦+1+1−1
𝑟𝑥 + 𝑟𝑦+1−1
1
𝑟𝑥 + 𝑟𝑦

+1
𝑤1
+1
𝑤2
1 ] [𝑤3 ]𝑟𝑥 + 𝑟𝑦 𝑤4(5)𝑥𝑥
𝑐𝑜𝑠𝜃0 𝑥𝑐𝑥 + 𝑠𝑖𝑛𝜃0 𝑥𝑐𝑦𝑥[ 𝑦 ] = [−𝑠𝑖𝑛𝜃0 𝑥𝑐𝑥 + 𝑐𝑜𝑠𝜃0 𝑥𝑐𝑦 ]
𝜃𝑧
𝜃𝑐

(6)

In this case, the calculated displacement is the relative
displacement in the world coordinate system for an
instantaneous moment. Since the angle between the
robot's own coordinate system and the world coordinate
system is always changing, therefore, to obtain thedisplacement of the robot in the world coordinate systemover a period of time, it is necessary to integrate thedisplacement value calculated at any moment (discrete
data summation) [6], assuming that over a period of time,a total of N times the robot's displacement relative to the(4)

5.2. Full-field positioning method
The full-field localization method studied in thisproject does not require the addition of the rest of theextra sensors or encoders and uses the robot's own©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
342

Intelligent Logistics Handling Robotworld coordinate system at a given moment, thecumulative displacement expression is Eq. (7).

∑𝑁∆𝑥𝑥𝑖=0 𝑐𝑜𝑠𝜃𝑖 𝑥𝑐𝑥𝑖 + 𝑠𝑖𝑛𝜃𝑖 𝑥𝑐𝑦𝑖
𝑁[∆𝑥𝑦 ] = [∑𝑖=0 −𝑠𝑖𝑛𝜃𝑖 𝑥𝑐𝑥𝑖 + 𝑐𝑜𝑠𝜃𝑖 𝑥𝑐𝑦𝑖 ]
∆𝜃𝑧
∆𝜃𝑐

(7)

In Eq. (7), i represents the number of operations.
5.3. Chassis electronic control system design(1) Chassis control board circuit design
The main control and power scheme of the chassis
control board is the same as that of the robotic arm control
board, in addition to the integrated communication
module. The chassis motion control system adopts avariety of communication interfaces to communicate
withvariousperipherals,
mainlyincludingcommunication with gyroscope, QR code scanning
module through UART serial port, communication withtouch screen through SPI and IIC, and communication
with brushless motor driver through CAN [7].

b. Bottom viewFig.12 Chassis control board
5.4. Chassis software design
Prepare a chassis motion control system flowchartaccording to the task requirements. After the system is
powered on, the program starts running and the
peripherals are initialized. After the initialization iscompleted, the robot drives out of the start-stop area, runs
to the QR code board, and starts the QR code scanning
module to start scanning the QR code to receive the taskcode. The robot enters the next running task and proceeds
to the raw material zone. When the robot runs to the raw
material area, the vision system is activated to identify thecolor of the material according to the task code, and therobotic arm starts to grasp and place it into the bin on the
robot. After completing the grasping of all materials, therobot continues to move forward. When the robot travels
to the vicinity of the roughing area, the vision systemstarts to recognize the coordinates on the target ring in theroughing area. The chassis control board adjusts the robot
position according to the coordinate data so that the robot
is accurately close to the rough machining area. At thistime, the vision system starts to identify the center of thetarget ring, and the robotic arm will place thecorresponding color materials in the target ring according
to the position of the target ring. After all materials havebeen placed, the robot will pick up the materials again in
the order in which they were placed and place them in thebin on the robot. When the above operation is complete,
the robot will move on to the staging area and resume thematerial placement operation. This operation is the sameas the robot placing material in the roughing area. When
all tasks are completed, the robot returns to the start-stop
area.

(2) PCB Overall Layout
The PCB is fabricated using a two-layer board design
with careful planning and design of the overall layout. As
Fig. 12 show the front and back view of the chassis
control board.

a.

Top view©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
343

Yanchao Bi, Jiale Cheng, Limei Wang, Yizhun Peng

Fig.13 Chassis Motion Control System Flowchart
6. Visual identity system
6.1. Material Color Identification

b. Identification results
Fig.14 Visual identification of material colors

The task requires the robot to be able to recognize thecolor of the material and grab the material in the order of
the task code. This project uses OpenCV library for color
recognition and detects the center of gravity of thematerial in the image. The idea of specific color
recognition is to confirm the HSV color space thresholdof the color and binarize the image with this HSV
threshold to get a binary image [8]. At this point, theimage is expanded so that the outline of the material in
the image tends to be more like a circle. Based on thisoutline, it can be determined to circle the material anddetermine the coordinates of the center of gravity of thematerial.

6.2. Target ring center identification
The task requires the robot to be able to recognize thetarget ring coordinates and place the material accurately
into the target ring coordinates. This project uses the
OpenCV library to identify the center of the target ring,which is a circle with color. When dealing with the target
ring detection problem, this system does not directly
detect the target ring directly by means of Hough's circle
transformation, but first binarizes the image. After that,the contours in this image are searched and the contours
are filtered to find the contour of the target ring. The
contour of the target ring is drawn as the smallest
enclosing circle, which is very close to the position andsize of the target ring itself, and the circle is the center of
the target ring.

As in Fig. 14, the robot gripper is directly above theblue material, at this time, the visual recognition system
is turned on to identify the material color. The outline of
the material will be wrapped by the black outline line, andthe center of the material will be marked (center).

a.

As shown in Fig. 15, the robot gripper is directly above
the blue target ring, and at this time, the visual
recognition system is turned on to detect the center of thetarget ring. The center of the blue target ring will bemarked by a blue dot.

Identification processa.

Identification process©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
344

Powered by TCPDF (www.tcpdf.org)

Intelligent Logistics Handling Robot

4.

5.

6.

7.

b. Identification results
Fig.15 Visual recognition target ring center8.

7. Conclusion

Authors Introduction

This project designs an intelligent logistics handlingrobot (AGV) for industrial logistics scenarios, which isequipped with a chassis motion control system based onan omnidirectional moving chassis with Mecanum
wheels, a robotic arm motion control system based on aplanar articulated (SCARA) robotic arm, and a material
recognition system based on machine vision. The chassis
motion control system is used to control the robot motion,
pick up the task code, and perform full-field localization.
The robotic arm motion control system realizes material
gripping and placing operations through the movement of
the robotic arm. The visual recognition system thenidentifies the color of the material and the center of thetarget ring. The chassis motion control system sends
commands to the chassis motion control system to control
the robotic arm to perform tasks. When the robotic arm
performs the task, it needs to receive the data returned by
the visual recognition system to correct the coordinate
position. Each system cooperates with each other toaccomplish the task.

Mr. Yanchao Bi
He was admitted to Tianjin University of
Science and Technology, China in 2021,
majoring in Intelligent Science andAdvanced Manufacturing, School ofElectronic Information and Automation.
He is currently an undergraduate student of
Tianjin University of Science andTechnology, China.
Mr. Jiale Cheng
He was admitted to Tianjin University of
Science and Technology, China in 2021,
majoring in Intelligent Science andAdvanced Manufacturing, School ofElectronic Information and Automation.
He is currently an undergraduate student of
Tianjin University of Science andTechnology, China.
Ms. Limei Wang
She was admitted to Tianjin University of
Science and Technology, China in 2021,
majoring in Intelligent Science andAdvanced Manufacturing in the School ofElectronic Information and Automation. He
is currently an undergraduate student of
Tianjin University of Science andTechnology, China.

The design simulates the handling and palletizing tasks
under industrial operations, which will greatly improve
the rapidity and accuracy of logistics operations andgreatly improve the efficiency of the logistics system.
References
1.

2.

3.

Long Tao. (2023). Design of Sweeping Robot Based on
STM32 Single Chip Microcomputer.Journal of Physics:
Conference Series(1).
Dong Nguyen Minh, Hiep Do Quang, Nam Dao Phuong,
Tien Ngo Manh Duy Nam Bui. (2023). An Adaptive Fuzzy
Dynamic Surface Control Tracking Algorithm for
Mecanum Wheeled Mobile Robot.International Journal of
Mechanical Engineering and Robotics Research(6).
Pizá Ricardo, Carbonell Rafael, Casanova Vicente,
Cuenca Ángel Salt Llobregat Julián J..(2022).Nonuniform
Dual-Rate Extended Kalman-Filter-Based Sensor Fusion
for Path-Following Control of a Holonomic Mobile Robot
with Four Mecanum Wheels.Applied Sciences(7),Jing Yan, Xuyang Tian, Xiaoyuan Luo Xinping Guan.
(2019).Design of an Embedded Communication System
for Underwater Asynchronous Localization..Embedded
Systems Letters(3),97-100.
CorbalanFuertes Montserrat, Millan Maria S. Gilabert
Daniel. (2001). Computer vision system for color
discrimination.Univ. Politecnica de Catalunya (Spain)592595.

Huimin Xu, Gaohong Yu, Yimiao Wang, Xiong Zhao,
Yijin Chen Jiangang Liu. (2023). Path Planning ofMecanum Wheel Chassis Based on Improved A*
Algorithm.Electronics(8).
Zhao Nan Ibaraki Soichi. (2022). Novel kinematic modelof a SCARA-type robot with bi-directional angular
positioning deviation of rotary axes.The International
Journal of Advanced Manufacturing Technology(7-8),
4901-4915.
Rigatos G., Abbaszadeh M., Busawon K. Pomares
J..(2023).Nonlinear optimal control for a 4-DOF SCARA
robotic manipulator. Robotica(8), 2397-2450.

Dr. Yizhun Peng
He is an Associate Professor in Tianjin
University of Science &Technology. He
received a doctor ’ s degree in control
theory and control engineering from theInstitute of Automation, Chinese Academyof Science in 2006.His research field isintelligent robot and intelligent control.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
345

Greenhouse Design Using Visual Recognition and IoT Technology
Yuntian Xia,
College of Artificial Intelligence ,Tianjin University of Science and Technology, 300457，China；
Yizhun Peng*

College of Electronic Information and Automation, Tianjin University of Science and Technology,
300222，China
E-mail: *pengyizhun@tust.edu.cn
www.tust.edu.cn

Abstract
This device is to solve the traditional pesticide spraying method on the human body has a greater impact and otherissues, through the STM32-based visualisation of the intelligent greenhouse to achieve automatic spraying ofpesticides and remote monitoring and other functions, the establishment of a visualisation of the intelligent
greenhouse monitoring platform. This equipment through the MQTT protocol, not only through the Internet ofThings platform real-time monitoring of crop growth status in the greenhouse, but also through the platform todetermine whether to spray pesticides, data transmission, so as to use the cross slide to control the position of thenozzle, and then through the visual recognition algorithms to improve the accuracy of the visual recognition part of
the accuracy of the spraying of plants affected by insect pests, the realization of the digital intelligent greenhouse.
Keywords: STM32, Monitoring Platform, MQTT, Visual Recognition

1. Introduction

reducing air pollution and greenhouse gas emissions, andpositively impacting human health [4].
For the electrical control part of the project, we
carried out systematic modelling and simulation throughthe Proteus simulation platform to determine the optimal
operating conditions. Subsequently, we used Altium
Designer to create the design, programming anddebugging, and finally succeeded in building an efficient
intelligent greenhouse monitoring platform [5].
The rest of this paper is organised as follows. The
second part describes the hardware used and thetheoretical calculations. The third part describes themethod of controlling the nozzle and the visual
recognition algorithm. Part IV describes how the smart
shed works and shows some of the operational results.
Part V summarises the main points of the paper.

This project is dedicated to solving problems such as thepotential impact of traditional pesticide spraying methods
on the human body. Through the STM32-based [1]visualised intelligent greenhouse, automatic pesticide
spraying and remote monitoring functions are realised,and a complete visualised intelligent greenhousemonitoring platform is constructed. We adopted the
MQTT [2] protocol, which not only monitors the growth
status of crops in the greenhouse in real time, but also
replenishes different colours of light in a targeted manneraccording to the light conditions inside the greenhouse toimprove the efficiency of photosynthesis. This is not only
more energy efficient than traditional white light
supplementation, but also significantly increases crop
yields and helps to make the system carbon neutral.
In addition, our system also features indoor
temperature monitoring and control, which promotes theaccumulation of organic matter in crops by adjusting thetemperature difference between day and night. For a
more precise response to pests, we utilise a cross slide tocontrol the position of the nozzles and incorporate visual
recognition algorithms to improve the accuracy ofspraying on pest-affected plants [3]. For a sustainableenergy supply, we added solar panels to the exterior of
the greenhouse to provide a constant source of energy forthe system, while improving long-term human health,2.

Design of Hardware Structure

2.1. Selection of the main control board
This work needs to deal with a large amount of data, hasa more complex circuit, and needs a lot of memory, so itneeds a microcontroller that can deal with a lot of song
peripheral devices at the same time, so the STM32F4
became the team's first choice. the physical diagram of
the STM32F4 is shown in Fig. 1.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
346

Yuntian Xia, Yizhun Peng

The nozzle structure is to control different spraying
methods and the pump structure is used to spray different
concentrations of pesticides. The mechanical structure ismodelled as shown in Fig. 4.

Fig. 1 STM32F4 Physical2.2. Relay Modules
This work is controlled by mobile phone APP, throughthe Internet of Things platform, the corresponding signal
will be sent to the STM32 core board, and it will control
each module. In order to save power, they will use a relay
to control the switch of some modules, so that some
non-working modules are in the off state when they are
not working, this work uses the JQC-3FF-S-Z relay, thephysical picture is shown in Fig. 2.

Fig. 4 Mechanical structure modelling drawings
The main issues considered in the design: Since the crops
inside the greenhouse are scattered all over the place, amechanical structure that can bring the nozzles andpumps to every part of the greenhouse is needed, theyrefer to the mechanical structure that controls the
movement of the nozzles in the 3D printers, and they
designed and chose the above solution.
2.5. Sensor selection

Fig. 2 Relay Physical Diagram

In order to obtain more information about the greenhouse,
this work chooses a variety of sensors. The temperatureand humidity sensor can always sense the temperatureand humidity inside the greenhouse, which is convenient
for the adjustment of the internal temperature of thegreenhouse [6]; the light intensity sensor monitors thechange of the light intensity inside the greenhouse, so as
to judge what colour and intensity of light needs to be
reinforced; the visual sensor can intuitively observe thegrowth status of the plants inside the greenhouse andtransmit the information to the user's mobile phone APP,
so as to achieve real-time monitoring. The digitaltemperature and humidity sensor, light intensity sensor
and OpenMV are shown in Fig. 5.

2.3. Driver Modules
This work needs to be as precise as possible when it
comes to pesticide spraying so that the pesticide can besprayed where we need it, so we chose to use stepper
motors and stepper motor drives. The physical drawing of
the stepper motor and stepper motor drive is shown in Fig.
3.

Fig. 3 Physical drawings of stepper motors and stepper
motor drives
2.4. Mechanical components

Fig. 5 The digital temperature and humidity sensor, lightintensity sensor and OpenMV

The mechanical part is designed as shown in Fig. 7, its
motion system consists of two axes, X-axis and Y-axis, tocontrol the position of the nozzle and the pump.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
347

Greenhouse Design Using Visual

2.6. Motion control of nozzles and pumps

Dilution of 100 times or less: Diluent dosage = Weight ofthe original agent × (concentration of the original agent concentration of the formulated agent) / Formulation
Concentration of formulated agent
Dilution more than 100 times: Diluent dosage = weight ofthe original agent × concentration of the original agent /concentration of the formulated agent
Calculate the amount of diluent by the multiplier method:
Dilution less than 100 times: Diluent dosage = Original
agent weight × Dilution times - Original agent weight
Dilution more than 100 times: Diluent dosage = original
agent weight × dilution times
Mass percent concentration: Mass percent concentration(%) = solute mass/solution mass 100%
Molar concentration: molar concentration (mol) = solutemoles / solution volume (litres) [8].
Equivalent concentration: Equivalent concentration =gram equivalent of solute / volume of solution (litres)
Mass-volumeconcentration:
Massvolumeconcentration = mass number of solute (grams or
milligrams) / volume of solution (cubic metres or litres)

Motion control of spray nozzles and pumps means thatthe mobile phone APP or various sensors send signals to
the main board, so as to control the spray nozzles andpumps to move according to the predetermined paths andspeeds in the two-dimensional space through thecross-axis structure. They use stepper motors to drive thenozzles and pumps along the X-axis and Y-axis, and therotation angle and speed of the stepper motors arecontrolled by pulse signals sent from the main board, andthe frequency and number of pulse signals are determinedby the concentration of pesticides that need to be
configured [7].
The principle of this mechanism is to control the
movement of X-axis and Y-axis by two motors at thesame time, when the left and right motors are in the same
direction, they move towards X-axis, and when the two
motors are in the opposite direction, they move towards
Y-axis. The simultaneous action of the two motors
provides a more stable force than a single motor
controlling one axis, and reduces the weight of one motor
on the XY platform. The formula is shown in Fig. 6.

2.8. Solar panel steering design
A common method in solar panel steering design is
to use a microcontroller and a photoresistor module tocollect light intensity data in four directions, and thenconvert them into digital signals through ADC, and thendrive the servo or stepper motor to adjust the angle of thesolar panel through comparison and control algorithms.
The specific calculation process is as follows:
Collect light intensity data in four directions through
ADC, respectively x0 , x1 , y0 , y1, and can take theaverage value several times to improve the accuracy.
Calculate the difference between the two directions oflight intensity, respectively, dx = x0 - x1 , dy = y0 - y1,
these two values reflect the solar panel and the degree of
deviation from the sun. Based on the positive, negativeand magnitude of dx and dy, the direction and angle to be
adjusted are determined. For example, if dx > 0 and dy >0, the solar panel needs to be rotated to the upper left; if
dx < 0 and dy < 0, the solar panel needs to be rotated to
the lower right; and if dx = 0 and dy = 0, the solar panel
has been aligned with the sunlight.
According to the direction and angle to be adjusted,
calculate the corresponding PWM signals or pulse
numbers, and then generate PWM signals or pulse signals
to drive the servo or stepper motor to rotate thecorresponding angle by timer interrupt or other means.

Fig. 6 Formula2.7. Pesticide concentration configuration
They will configure pesticides in different concentrations
according to the needs of different crops
There are several formulas for pesticide formulation:
Conversion between per cent concentration and parts
per million concentration: parts per million (ppm) =10,000 × per cent concentration
Conversion between the multiplier method and thepercentage concentration: Percent concentration (%) =(original concentration/dilution times) × 100
Calculate the amount of diluent according to the active
ingredient:©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
348

Yuntian Xia, Yizhun Peng

Repeat the above steps until dx and dy are close to zero,i.e. the solar panel is perpendicular to the sunlight.
3.

Software design

3.1. Pesticide spraying design
When the vision system detects that a plant is infested, it
automatically feeds back a signal to the stepper motor
system. The stepper motor system responds quickly andaccurately drives the pesticide nozzle to position itself
near the infected plant. The stepper motors thenco-ordinate the spraying of the pesticide to ensure
efficient and precise spraying of the pesticide in theaffected area. This intelligent system not only responds topest problems in a timely manner, but also provides areliable solution for plant healthcare by maximising
spraying efficiency through the collaborative operation of
the vision system and stepper motors [3], [7]. The stepper
motor working process is shown in Fig. 7.

Fig. 9 The diseased plant dataset(2) Annotate
He carefully and systematically annotated the leaf dataset
and leaf disease dataset of this batch of plants. Throughthe annotation scheme designed by him, he captured andrecorded the characteristics of the plants at different
growth stages, from roots to leaves, from physiologicalparameters to environmental adaptations. And hecaptured the characteristics of plants suffering from pests,
diseases and so on. This kind of annotation not only
enriches the connotation of plant data, but also lays asolid foundation for subsequent data analysis and model
construction. The process diagram of leaf annotation isshown in Fig. 10, and the process diagram of disease
point annotation is shown Fig. 11.

Fig. 7 The stepper motor working process
3.2. Visual Inspection(1) Acquisition of plant leaf datasets
They acquired a detailed and rich batch of plant leaf
datasets and leaf disease datasets, a valuable resource thatwill fully support in-depth research. The data covers
multiple dimensions, including plant growth, dynamics ofphysiological parameters, and the potential impact ofenvironmental factors on tobacco quality. By carefully
analysing this dataset, they will be able to gain insight
into all aspects of plant growth. The plant leaf dataset isshown in Fig. 8, and the diseased plant dataset is shownin Fig. 9.

Fig. 10 The process diagram of leaf annotation

Fig. 11 The process diagram of disease point annotation(2) Identification
Fig. 8 The plant leaf dataset©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
349

Greenhouse Design Using Visual

Through CNN neural networks, he successfully
implemented the functions of leaf recognition and pest
identification [9], and cleverly combined them to form anefficient and comprehensive system. Through deeplearning and computer vision techniques, he accurately
recognised different types of tobacco leaves in the leaf
images, and based on this, the system was able to keenly
detect possible insect pests through careful algorithm
design. The leaf recognition and pest recognition diagram
is shown in Fig. 12. The final recognition result is shownin Fig. 13.

bring faster convergence speed at the same time, havebetter performance [10]. The activation function is shownin Fig. 14.

Fig. 14 activation function
4. Experiment4.1. Pesticide spraying programmes
In the smart greenhouse system, they have introduced anadvanced disease monitoring and prevention mechanism.
Once the system recognises the presence of a plantdisease, the automated control system is instantly
activated, transferring the information to the stepper
motor system. The stepper motors precisely manoeuvre
the pesticide nozzles to move to the exact location of theinfected plant. The intelligent system then implements
precise and efficient spraying according to a pre-set
pesticide spraying programme to maximise thesuppression of the spread of the disease.
Future research directions include further optimisingdisease detection algorithms, improving the system's
ability to accurately identify different disease types, andincorporating advanced pesticide selection and use
techniques to achieve more precise and sustainabledisease prevention and control strategies. Suchinnovations will lead to more sustainable and efficientplant health management solutions in the field of smart
agriculture.

Fig. 12 The leaf recognition and pest recognition diagram

Fig. 13 The final recognition result
3.3. Optimisation
Since the disease target is very small, the sensory field of
the original smallest detection head is still larger than thesize of the target, so it is necessary to improve theoriginal network, starting from the optimisation algorithm,
firstly, the activation function is improved, they use leaky
relu activation function in the middle layer, and theS-type activation function is selected for the last detection
layer, and at the same time, the mish activation function
is used; and the loss function is improved , using the
CIOU loss as the loss of the bounding box, which can

4.2. Mobile Application Programming
By running a well-designed mobile application, he
successfully implemented a real-time greenhousemonitoring system on his mobile phone. The application
provides an intuitive and detailed user interface thatenables users to easily monitor the status of thegreenhouse remotely and access key environmental
indicators in real time. Users can easily access data suchas temperature, humidity, light and soil moisture, as well

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
350

Yuntian Xia, Yizhun Peng

as plant growth conditions in each area of the greenhouse,anytime, anywhere, using their smartphones [11].
Through the mobile app, users can set customised
alarm thresholds, and the system will immediately send
notifications to alert users of any abnormal changes inenvironmental conditions. This real-time monitoring andresponse mechanism provides users with timely decision
support, enabling them to quickly take the necessary
measures to safeguard plant health and growth.
In addition, the mobile app supports remote controlof the greenhouse system. Users can adjust parameterssuch as temperature control, humidity regulation, light
management, and irrigation system through the mobile
phone to achieve intelligent remote control of thegreenhouse environment. Such flexibility enables users tomake adjustments according to real-time changing needs,
improving the adaptability and operability of thegreenhouse system. The running diagram of the
programme is shown in Fig.15.

improves the human-computer interaction experience,and improves efficiency while protecting people's safety,
and I believe that the demand for this type of product in
the agricultural market will grow.
References
1. Bufeng Zhang, Guoli Li, Puryu Liu. Design of remotemonitoring system for agricultural greenhouses based on
STM32.
ScienceandTechnology
Innovation,2023,(27):205-208.
2. Fengshuo Wang,Li Zeng. Design and implementation ofinternet of things platform based on MQTT protocol.
Computer Knowledge and Technology,2023,19(25):73-76.
3. Chen Peng, Jin-Lian Mo, Feng Wang. Research anddesign of machine vision in environmental monitoring
system of agricultural vegetable greenhouse. Journal of
Hunan College of Arts and Sciences (Natural Science
Edition),2021,33(02):68-72.
4. Meigui Ya. Safe production of agricultural products andthe use of biopesticides. Rural Practical Technology, 2023,(10):93-94.
5. Jun Jia, Jingyu Li, Fangjuan DONG. Discussion on theapplication of internet of things technology in the design ofmonitoring system for smart agricultural greenhouses.
Intelligent Agriculture Guide,2023,3(13):9-12.
6. Baoling Hu, Jun Ma, Lei ZHOU et al. Design ofintelligent control system for air temperature and humidity
in agricultural greenhouses. Journal of Lanzhou College of
Arts
and
Sciences(Natural
Science
Edition),2023,37(05):68-74.
7. Renming Tan, Renjie Zhang, Tao Jiang. Design ofclosed-loop stepper motor position control system based on
STM32. Journal of Heilongjiang Institute of Technology(General Edition),2023,23(01):58-62.
8. Feineng Zheng. Representation of pesticide concentrationand calculation of dilution. Plant Protection Technology
and Extension,1994, (06):19-20.
9. Tong Zhao,Chaofeng Sha. Revisiting test sample
selection for CNN-oriented models:considering model
calibration. Computer Science 1-13.
10. Guowei Dai, Zhimin Tian, Jingchao Fan et al. A neuralnetwork structure search oriented enhanced recognition
method for plant leaf diseases. Journal of Northwest
Forestry College,2023,38(05):153-161+193.
11. Xiaoyuan Bo,Jiaqi Wu,Zhiyi Hong et al. Design andresearch on the application programme of Huinong Huimin.
Modern Agriculture, 2020, (11):17-19.

Fig. 15 The running diagram of the programme
5. Conclusion
Nowadays, with the rapid development of artificial
intelligence and Internet of Things technology, thevisualised smart greenhouse designed in this paper isprecisely in line with the trend of technologicaldevelopment,
combiningtheknowledgeofmicrocontroller, machine vision, artificial intelligence,and many other things with each other to form a complete
system, and it is believed that in the future, this design
will be more widely used in the agricultural market. This
design combines the knowledge of multiple disciplines,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
351

Powered by TCPDF (www.tcpdf.org)

Greenhouse Design Using Visual

Authors Introduction
Mr. Yuntian Xia
He is currently pursuing his undergraduate
degree at the School of Artificial Intelligence,
Tianjin University of Science andTechnology. His research area is neuralnetworks.

Dr. Yizhun Peng
He is an Associate Professor in Tianjin
University of Science &Technology. He
received a doctor’s degree in control theory
and control engineering from the Institute ofAutomation, Chinese Academy of Science in2006.His research field is intelligent robot
and intelligent control.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
352

A Design of Modular Photovoltaic Environmentally Friendly Portable Stroller
Suqing Duan, Yizhun Peng*

College of Electronic Information and Automation, Tianjin University of Science and Technology, China
E-mail: *pengyizhun@tust.edu.cn
www.tust.edu.cn

Abstract
The work is centered on ROS and integrates speech recognition and natural language processing modules toenhance the environmental awareness of the stroller and monitor the baby's condition to ensure a timely andappropriate response. The Raspberry Pi is the main control unit of the stroller and connects to a cloud-based IoT
platform via the MQTT protocol. The platform facilitates seamless communication between the Raspberry Pi andcloud data for efficient data visualization on mobile devices. This innovative solution solves the challenge ofbalancing parental responsibilities with career advancement, promoting healthier and happier babies while allowing
parents to maintain a harmonious work-life balance.
Keywords: Natural Language Processing, MQTT Protocol, Data Visualization, Cloud Computing, Raspberry Pi

1. Introduction

MQTT protocol is built and the interaction scheme
between Raspberry Pi and cloud data is designed torealize the convenient interaction function of datavisualization in the mobile terminal [4].
The rest of the paper is organized as follows. The
second part introduces the system program design. The
third part introduces the working principle of the stroller,
which includes the design of automatic following andsafe obstacle avoidance function, automatic cradle
function design, microenvironment sensing system design,
baby crying sound sensing and recognition design andbaby expression recognition design. The fourth partsummarizes the main content of this paper.

In order to reduce greenhouse gas emissions and at thesame time enhance the safety and interactivity of thestroller, this work designs a modular photovoltaic
intelligent portable stroller system with active safety,
micro-environmental monitoring and remote monitoringof mobile terminals [1], and carries out a scientific andintelligent design of the software and hardware of thephotovoltaic environmentally friendly portable stroller
movement system and monitoring system [2].
The work uses polycrystalline solar panels to chargelithium-ion batteries, sets three power modes, and adopts
aluminum alloy and other carbon-saving materials, whichhelps to reduce air pollution and greenhouse gas
emissions, and responds to the national call for promoting
green and low-carbon development [3]. The work uses
STM32 development board as the bottom control center,
and DC motor and drive circuit to build the bottom drive
system hardware; adopts ROS as the core, and applies
IoT technology with voice recognition and natural
language processing module to realize the stroller's
perception of the surrounding environment and the baby's
state and make appropriate responses; adopts Raspberry
Pi as the upper host computer of the stroller, and builds a
cloud IoT platform and a cloud IoT platform based on
MQTT protocol, and uses Raspberry Pi as the upper hostcomputer. Raspberry Pi is used as the host controller ofthe stroller, and a cloud-based IoT platform based on

2. System Programming
The team carried out the overall design and technicalanalysis of the system for the modular photovoltaic smart
eco-friendly baby stroller. The function realization part ofthe stroller is a set of integrated motion device, whichcontains modules such as automatic cradle, autonomous
following, etc.; and the monitoring part is a complete setof intelligent IoT solutions, which contains modules suchas intelligent interaction, active safety, and visualsurveillance [5]. The multi-angle view of the smart
eco-friendly stroller is shown in Fig.1.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
353

Suqing Duan, Yizhun Peng

Fig.3 The schematic diagram of data processing
Fig. 1 The multi-angle view of the smart eco-friendly
stroller

3. How Strollers Work
3.1. Automatic Following and
Avoidance Function Design

The modular photovoltaic intelligent eco-friendly stroller
realizes the corresponding functions by controlling avariety of intelligent modules such as motor drive module,
autonomous motion module, automatic brake module,
automatic bed shaking module, deep learning module,
visual monitoring module, intelligent interaction moduleand so on through the embedded system, and theintelligent system based on the Internet of Things (IoT)
can make the stroller connected to the mobile terminaland provide real-time feedback for the parents on all
kinds of information of the baby in the stroller. The
system block diagram is shown in Fig. 2.

Safe

Obstacle

The automatic following function is calculated by theultrasonic distance measuring module through theacoustic wave reflection time length, to get the distance
between the stroller and the person being followed, andthrough the PID algorithm to control the motor drive
module to make the distance between the stroller and theperson being followed stay within the set range [6]; thesafety obstacle avoidance function also uses theultrasonic distance measuring module to measure thedistance, when the distance between the stroller and the
obstacle in front of the stroller is less than the set value,
the stroller brakes and stops. When the distance between
the stroller and the obstacle in front is measured to be lessthan the set value, the stroller brakes and stops. In signal
processing, Kalman filtering algorithm will be used toeliminate other interferences, to ensure that the smart
stroller follows accurately, so that parents can have amore secure experience.
3.1.1. Distance Measuring Sensor
The ultrasonic distance module was selected for the
distance sensor. Use low power consumption, low priceand easy to connect HC-SR04 ultrasonic distance sensor.
The module uses a transceiver split probe to range objects
within a distance of 1cm-6m, to achieve the function of
avoiding obstacles and following people. Ultrasonicdistance measurement structure is shown in Fig. 4.

Fig. 2 The system block diagram
The sensor data collected by the controller is susceptible
to external interference and is not highly accurate. In thiswork, different sensor data types are processed by mean
filtering, weighted recursive average filtering, medianfiltering, Kalman filtering, dithering filtering, first-order
hysteresis filtering, etc., and the data accuracy isimproved. The schematic diagram of data processing isshown in Fig. 3.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
354

A Design of Modular

3.2. Automatic Cradle Function Design
The design adopts the temperature and humidity
compound sensor DHT11. It has humidity measurement
elements to measure humidity and NTC temperature
measurement elements to measure temperature, and its
correction factor is stored in OTP memory by program.
3.2.1. Cradle Control Functional Design
Fig. 4 Ultrasonic distance measurement structure
The team added temperature and humidity sensors to thework, thus using different speeds of sound for distancemeasurements by monitoring different temperatures,increasing the accuracy of the distance measurements andfurther securing the safety of the stroller. The
temperature-speed of sound relationship graph is shownin Fig. 5.

3.1.2. Bluetooth Module

The function of this module is divided into
semi-automatic bionic rocking and fully automatic bionicrocking. When the semi-automatic cradle is turned on,parents can control it by connecting the cell phone APP
through the Bluetooth module, and the stroller will
simulate the parents' shaking once for every click on theturn on automatic shaking button; while when the fully
automatic cradle is turned on, the automatic cradle
module will work together with the cry detection moduleand the camera to automatically turn on the automatic
cradle function when the baby's cry is detected and thefunction will decide the direction and strength of shaking
according to the baby's posture fed back by the camera.
This function will decide the direction and strength ofrocking according to the baby's posture feedback from thecamera, so that the baby is in the most comfortable cradle
environment.

The system uses HC-05 Bluetooth module to realize

3.2.2. Attitude Sensing Module

Fig. 5 The temperature-speed of sound relationship graphtwo-way wireless function, through the micro-controller
control device and voice control system communication,
to realize the remote control cradle car front and back,left and right in all directions, or cell phone APP connects
to Bluetooth to realize the cell phone APP control
movement [7]. The cell phone Bluetooth app interface isshown in Fig. 6.

The stroller is equipped with posture sensing module
MPU6050, which sets the balance state data limit,
autonomously monitors the swinging condition of thecradle without voice control [8], and the Raspberry Pi
accepts the data and transmits it to the control board,which ensures that the cradle is always kept in ahorizontal state without human intervention, providing
safety for infants and toddlers to move around in thecradle. The test results of the attitude sensing module areshown in Fig. 7.

Fig.7 The test results of the attitude sensing module

Fig.6 The cell phone Bluetooth app interface©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
355

Suqing Duan, Yizhun Peng

3.2.3. Voice Interaction Module

its high resolution can detect a larger range of lightintensity changes.

The stroller is also equipped with voice interaction
module LD3320, which can recognize the user's
commands and make corresponding interactive
commands. Considering that some babies areaccustomed to listening to their parents' voices to sleep,
when the stroller is told to "play mommy's words" orother commands, it will play the audio recorded by theparents in advance to coax the child to sleep, which
reduces the pressure on parents' childcare. The test dataof the voice interaction module is shown in Fig. 8.

3.3.2. Temperature and Humidity Sensor
DHT11 is a temperature and humidity composite digital
signal sensor, the sensor to digital acquisition technology
and sensor technology integrated development, withyears of market reputation and excellent performance.
The sensor's clever single-wire serial interface design
scheme, so that power consumption, volume has a greater
reduction in the extremely demanding scenarios, thesensor can still maintain good performance and stability.
The temperature and humidity sensor operation interface
is shown in Fig. 9.

Fig.8 The test data of the voice interaction module
3.3. Design of Microenvironment Sensing Systems
Sensing system design, temperature and humidity sensors
can monitor the temperature and humidity in the stroller,
when the temperature is too high, it will remind the baby
moms and dads through the cell phone to ventilate in time,
and when the humidity is too low, it will prompt the baby
moms and dads to turn on the humidification device; thelight intensity sensor automatically senses the strength ofthe light around the sun, and when it meets with thestrong light environment, the sunshade will automaticallyextend and unfold, and play the effect of the sun, and if
the baby moms and dads want to let their children
sunbathe in the sun, they can turn it off the automatic
sun-shading function through the cell phone app and turnthe sunshade on.

Fig.9 The temperature and humidity sensor operationinterface
3.4. Infant Cry Sensing and Recognition Design
Infant cry perception and recognition design uses a cryrecognition system, when the baby cries, the detection
device will receive the cry signal, and then filtering
processing, feature extraction through the basic acoustic
features to get the feature data [9]. The results of thehuman infrared sensor test are shown in Fig.10.

3.3.1. Light Intensity Sensor
BH1750 internal by the photodiode, operational amplifier,
ADC acquisition, crystal, etc. PD diode through thephotovoltaic effect will be converted into an input lightsignal into an electrical signal, amplified by theoperational amplifier circuit, the voltage collected by the
ADC, and then through the logic circuit is converted into
a 16-bit binary number stored in the internal registers, inshort, is that the stronger the light, the greater thephotocurrent, the greater the voltage is. larger, and using

Fig.10 The results of the human infrared sensor test
3.4.1. Sound Classification
Inception-v4 network

Algorithm

Based

on

By studying Dunstan's theory of infant cry classification,
our team collected videos and audios of infant cries fromvideo websites and manually edited and labeled them toobtain the speech spectrograms of five categories of
infant cry sound samples, which are the speech©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
356

A Design of Modular

spectrograms of sound samples of cries due to wanting to
hiccup, being uncomfortable, feeling sleepy and wanting
to go to sleep, being hungry and wanting to eat, and being
bloated and in pain in the abdomen [10]. The five kindsof speech maps are shown in Fig. 11.

baby expression recognition, i.e., accurately calibrating
the position and size of the face in the image. Baby
expression images contain rich pattern features, such as
histogram features, color features, template features,
structural features and Haar features [11]. Baby
expression detection is to pick out this useful information
and use these features to achieve baby expression
detection. Baby expression detection is shown in Fig. 13.

Fig. 11 The five kinds of speech maps
3.4.2. Network Model Structure Details
The database used in the work is the personal
self-constructed infant crying database described in thispaper, with a total of 7,500 spectrograms converted from
infant crying data. In the experiment, the team used 6500
infant crying speech maps as a training set and 1000
infant crying speech maps as a test set. The categories
were categorized using Oberschländerstein's theory ofbaby crying for 5 categories, which were manuallylabeled.
The tool used to build the network is the Tensorflow deeplearning framework, and the task of categorizing different
categories of infant cries is accomplished by modifying
the Inception-v4 network model. The upper computer
gives different feedback for different cries as shown in
Fig. 12.

Fig. 13 Baby expression detection
Mainstream baby expression detection methods use
Adaboost learning algorithm based on the above features,
which combines some of the weaker classification
methods together to combine new very strongclassification methods, which can effectively improvethe detection speed of the classifier.
3.5.2. Baby Expression Image Preprocessing
Image preprocessing for baby expressions is a process ofprocessing images and ultimately serving feature
extraction based on face detection results. The original
image acquired by the system often cannot be useddirectly due to the limitations of various conditions andrandom interference, and it must be subjected to image
preprocessing such as grayscale correction and noise
filtering in the early stage of image processing. For thebaby expression image, the preprocessing process mainly
includes light compensation, gray scale transformation,
histogram equalization, normalization, geometriccorrection, filtering and sharpening of the baby
expression image.

Fig. 12 Different feedback for different cries
3.5. Baby Expression Recognition Design
The baby expression recognition system consists of fourmain components, namely: baby expression image
acquisition and detection, baby expression image
preprocessing, baby expression image feature extraction,and matching and recognition.

3.5.3. Baby Expression Image Feature Extraction
The methods of infant expression feature extraction aresummarized into two main categories, one is theknowledge-based characterization method; the other isthe characterization method based on algebraic features
or statistical learning.
Knowledge-based characterization methods are mainly
based on the shape description of infant expression
organs and the distance characteristics between them toobtain feature data that help to classify infant expressions.
Infant expression consists of eyes, nose, mouth, chin and3.5.1. Image Acquisition and Detection of Baby
Expressions
In terms of baby expression image capture, different baby
expression images can be captured by the camera lens.
When the baby is within the capture range of the capturedevice, the capture device automatically searches andcaptures the baby expression image; baby expression
detection is mainly used in practice for preprocessing of©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
357

Suqing Duan, Yizhun Pengother localities, the geometric description of these
localities and the structural relationship between themcan be used as an important feature for recognizing theface; the basic idea of the algebraic feature-based method
is to transform the high-dimensional description of theface in the spatial domain into a low-dimensionaldescription in the frequency domain or in other spaces,
and the characterization methods are the linear projection
characterization method and the nonlinear projection
characterization method.

responsibilities, and realizes the further penetration of
intelligence and advancement in the field of childcare.
References
1. Wan Tao, Feng Junsen, Chen Jin et al. Design andrealization of intelligent safety stroller control system[J].
Internet of Things. 2022,12(10):102-106.
2. Liu Xiaocheng. China's stroller: the current situation andcountermeasure analysis of expanding overseas market[J].
Foreign Economic and Trade Affairs.2011.(12):4-7.
3. Liu Yu, Fan Guangjie. Research on the problems andcountermeasures of promoting the concept of green low-carbon
development into high school civics class[J]. New curriculum
introduction. 2023(29):9-12.
4. Fu Bairui, Chen Jiaping. Embedded agriculture based on
Raspberry Pi and LabVIEW[J]. Electronic Design Engineering.
2023,31(20):119-123.
5. Liu Yuxi, Zheng Ming, Wang Xuebin et al. Design of a
new type of intelligent stroller[N]. Proceedings of theProfessional Committee on Educational Teaching Innovation ofChina Society for Educational Development Strategy Volume
IV.2019:311-314.
6. Wang Lufeng. Research on DC drive motor speed
regulation for electric vehicles based on PID control[J].
Automotive practical technology.2020(10):106-108.
7. Li Minghuan, Peng Jian, Tan Jianfeng et al. Intelligent
stroller based on wireless sensor network and human-computerinteraction control[J]. Internet of Things Technology.
2022,12(09):127-131.
8. Ding Zulei, Jiang Tianze, Wwn Xiuping. Research on
autonomous motion navigation system based on GPS
positioning[J].
Automation
andInstrumentation.
2022,37(08):1-4+29.
9. Qu Jin. Design of intelligent interaction system based on
speech recognition[J]. Automation and Instrumentation.2023,(01):221-225.
10. Yang Han, Wang Miaoqin, Li Yanxi, YANG Chongyao,
CHEN Maoqing, LI Tarojun. Knowledge graph analysis basedon CiteSpace domestic infant crying research field[J]. Chinese
Journal of Child Health Care.2022,30(10):1112-1117.
11. Liu Shupeng, Chen Zhiqiang, Chen Na et al. A set ofexperiments on Tensorflow-based CNN-RNN fusionarchitecture[J].
Industrial
ControlComputer.
2019,32(08):66-68.

3.5.4. Image Matching and Recognition of Baby
Expressions
The feature data of the extracted baby expression image
is searched and matched with the feature templates stored
in the database, and the result obtained from the matchingis outputted by setting a threshold value when thesimilarity exceeds this threshold value. Baby expression
recognition is to compare the face features to berecognized with the obtained baby expression featuretemplates, and to judge the baby state information basedon the degree of similarity.
4. Conclusion
The Modular Photovoltaic Environmentally Friendly
Portable Stroller's is dedicated to innovatively solvingsocial problems through a modular photovoltaic smart
portable stroller system, while reducing greenhouse gas
emissions and enhancing the safety and interactivity ofthe stroller. The work uses polycrystalline solar panels,
aluminum alloy and other environmentally friendly
materials, as well as three power modes, fully responding
to the initiative of green and low-carbon development. In
terms of hardware design, the STM32 development board
builds the underlying control center and implements theactive safety and motion system. In terms of software,
ROS serves as the core, combining with IoT technology,
speech recognition and natural language processing toequip the stroller with the ability to intelligently sense
and respond to the surrounding environment and the
baby's state. Through the Raspberry Pi as the hostcontroller of the upper computer, a cloud-based IoT
platform based on the MQTT protocol is built, which
realizes the convenient interaction of data and thevisualization and monitoring of the mobile terminal. By
improving the intelligent level of the stroller, it enables
parents to better balance their work and family

Authors Introduction
Ms. Suqing Duan
She is currently pursuing her undergraduate
degree at the School of Electronic
Information and Automation, Tianjin
University of Science and Technology. Her
research field is embedded system.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
358

Powered by TCPDF (www.tcpdf.org)

A Design of Modular

Dr. Yizhun Peng
He is an Associate Professor in Tianjin
University of Science &Technology. He
received a doctor’s degree in control theory
and control engineering from the Institute ofAutomation, Chinese Academy of Science in2006.His research field is intelligent robot
and intelligent control.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
359

"Teenage Mutant Ninja Turtles" - A Design of a Bionic Quadrupedal Rescue Robot
Hongpi Zhao, Yingfan Zhu, Zhihan Zhao, Xin Liang, Lei Lv, Yizhun Peng*

Tianjin University of Science and Technology, Tianjin, 300222,China
E-mail: *pengyizhun@126.com
www.tust.edu.cn

Abstract"Teenage Mutant Ninja Turtles - Bionic Quadrupedal Rescue Robot is a quadrupedal robot based on the principleof bionics, inspired by the quadrupedal animals in nature. The robot has rescue and life detection capabilities andcan perform rescue operations at disaster sites. The design of the robot enables it to operate efficiently and stably
in complex environments, and at the same time it has the qualities of bravery and toughness, which are in line with
the image of the Teenage Mutant Ninja Turtles. The robot can be used in a wide range of application scenarios,
such as earthquake, fire and other disasters, to provide more efficient and safer support for rescue work.
Keywords:Biomimetic Machinery,Artificial Intelligence, Life Detection, Wifi Module

as flat, rugged,vertical and muddy.It is equipped withacoustic obstacle avoidance system with superb
obstacle-crossing ability.It is also equipped with 4G
infrared transmission for working in dark environments
and entering collapsed pits for life detection andenvironmental exploration [2].
The rest of this article is organized as follows. Part II
describes the modules used in the hardware systemdesign. Part III describes the design of the bionic mobile
joint. Part IV builds the model and tests to determine thefoot structure. Part V summarises the main points of thepaper [3], [4].

1. Introduction
With the development of society, mankind is faced withmore and more natural disasters and man-made accidents,
such as earthquakes, fires and mining accidents, which
often result in a large number of casualties and property
losses, bringing great harm to society. At the scene ofthese disasters, rescuers often need to risk their lives toenter the danger zone to carry out life detection,
environmental exploration, material transportation, etc.
Constrained by the number of rescuers, capacity,equipment, time and other factors, it is difficult toachieve high efficiency, safety and comprehensiveness[1].
To solve this problem, a bionic quadruped rescue robot
called "Teenage Mutant Ninja Turtle" has beendeveloped. The animated image of "Teenage Mutant
Ninja Turtles" is a brave, invincible and courageous little
turtle, and the rescue robot designed by the team has thesame qualities of bravery and toughness, "Teenage
Mutant Ninja Turtles" animated image is a brave,
invincible and courageous little turtle, and the rescue
robot designed by the team has the same qualities ofbravery and toughness essence of the spirit, and therefore
the name of this rescue robot for the "Teenage Mutant
Ninja Turtles". The legs of "Ninja Turtle" use eight
servos as joint actuators, and the legs can be replacedwith wheel-type, bionic foot-type, negative pressure
suction cup-type and three kinds of modularized
structures to cope with different working scenarios, such

2.

The Hardware System Design

In terms of hardware design, the Teenage Mutant Ninja
Turtles Bionic Quadrupedal Rescue Robot has been
carefully PCB-drawn, including the design and layout ofseveral key modules, to ensure the stable and efficient
operation of the robot. The design of the PCB board isshown in Fig. 1.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
360

Hongpi Zhao, Yingfan Zhu, Zhihan Zhao, Xin Liang, Lei Lv, Yizhun Peng

3. Design of bionic mobile joints
The parts design drawing is shown in Fig. 3. Detailed
descriptions of functions and uses are shown in Table 1.

Fig. 3 Component design drawing
Table 1 Functions and uses
Fig. 1 The design of the PCB board

Functions and Uses
Transmission
elements
forconnecting different joints andactuators to ensure coordinatedmovement of the robot's limbs
bear
Used to provide a rotational
connection that allows the robot's
joints to rotate freely without
excessive friction
Servo 1
Used to control the swinging
motion of the thighs
Servo 2For controlling the elevation
movement of the lower leg
shaftsAllows the robot to control theattachment and release of suction
cups by varying air pressure, thus
enabling the ability to climb andattach to different surfaces
suction pad
Uses a vacuum or pneumatic
system to create a suction force toenure that the robot is firmly
attached to the target surface.
A sophisticated system has been designed for Teenage
Mutant Ninja Turtles Bionic Quadrupedal Rescue Robotin Pneumatic Adsorption System. The system has astrong adsorption and climbing function and ensuresstrong adsorption by negative vacuum -55KPa. The
suction cups can withstand wall loads from 1.25kg to
1.7kg, allowing the robot to safely support its own
weight and other additional loads. The design of thepneumatic adsorption system is complex and requiresconsideration of material selection, airflow control,
negative pressure generation and other factors to ensurethat the robot performs reliably in a variety ofenvironments. Such systems are not only used in bionicrobots, but are also widely used in industrial automation,
glass suction handling, and handling equipment [6]. The
pneumatic adsorption system is shown in Fig. 4.
Connectingrod

2.1 Circuit module section
Buck modules are an important part of power
management, used to reduce the input voltage to asuitable operating voltage for each electronic component,
helping to ensure system stability and power efficiency.
2.2 Motor drive module
Used to control the robot's motors, using PWM to control
the speed and direction of the motors.
2.3 Bluetooth module
Enables the robot to communicate with external devices
over Bluetooth, providing a convenient way for remotecontrol and data transfer.
2.4 Ultrasonic sensors
Measures distance by sending and receiving ultrasonicsignals for obstacle detection, navigation and localisation,
providing robots with environmental awareness.
2.5 Wi-Fi wireless mapping module
Using the Wi-Fi wireless mapping module, the robot cantransmit real-time images and video streams to remotedevices, providing powerful support for teleoperation andenvironmental exploration [5].
The hardware design is shown in Fig. 2.

Fig. 2 Hardware design Hardware design©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
361

Teenage Mutant Ninja Turtles

Fig. 4 Pneumatic adsorption system

4.Modelling and TestingFig. 6 Stress and deformation cloud diagrams

4.1 Multi-body dynamics modelling

Stress distributions also help to ensure that the robot
does not suffer from material fatigue or damage during
operation; deformation clouds show how the robot's
structure deforms under different loads. They arepresented through colour coding or arrows to visualise
the change in shape of the robot under stress.

In the mechanism dynamics simulation, a multi-bodydynamics model is adopted, focusing on the kinematicmechanism of the robot leg, and the swinging of thethigh joint, the lifting and lowering of the calf joint, andthe linkage between the thigh joint and the calf joint are
simulated and analyzed, respectively [7]. And the model
diagram is shown in Fig. 5.

5.System testing and results
Continuous optimizations and upgrades are performed toensure good product performance. Through continuous
product iterations, system testing and results are closely
monitored to validate and improve the effectiveness ofeach iteration round. Over the course of the program's
continuous development, five generations of products
have been witnessed with significant progress andimprovements.Firstly, the first generation of the productidentified a quadrupedal structural solution with afour-link structure, which significantly enhanced theairframe's manoeuvrability. However, this was only the
beginning [8]. The first generation is shown in Fig. 7.

Fig. 5 Multi-body dynamics model
4.2 Stress and deformation maps
The structure of the Teenage Mutant Ninja Turtles(TMNT) bionic quadrupedal rescue robot was studied in
depth in a finite element static analysis to evaluate its
performance
under
different
workloads
andenvironmental conditions. This analysis includes
the generation of stress and deformation maps, as shownin Fig. 6.

Fig. 7 First generation products
Subsequently, the introduction of the second
generation of the robot brought significant improvementswith the adoption of a high torque servo, MG996, to
make the robot more agile and the introduction ofmodularly designed legs to improve ease of maintenance.
The second generation is shown in Fig. 8.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
362

Hongpi Zhao, Yingfan Zhu, Zhihan Zhao, Xin Liang, Lei Lv, Yizhun Peng

At each iteration stage, we conducted system tests andanalysed the results to ensure stability and performance
optimisation of the new features. These tests included therobot's performance in various terrains and environments,
its manoeuvrability, the reliability of the automatic
obstacle avoidance function, the effectiveness ofomnidirectional movement, and the aesthetic design.
Fig. 8 Second-generation products

References

The third generation product further extends thefunction by adding tail and head structure, introducing
automatic obstacle avoidance function and wirelessmapping technology to make the robot more intelligent
and adaptable to different environments. The third
generation product is shown in Fig. 9.

1.

2.

3.

4.
5.

6.

Fig. 9 Third-generation products
7.

The highlight of the fourth generation is the additionof tracks and McNamee wheels, which give the robot theability to not only move in all directions, but also steer ina more flexible manner. The fourth generation is shownin Fig. 10.

8.

Gao Liang; Gao Jingsong; Xu Wei; Zhao Jie. Research
Status and Development Prospect of amphibian bionicrobot [J]. Robot Industry,2023
Research on the Development Status and trend of
biomimetic robot [J]. Menbao; Fan Xuekun; Chen
Yongxin. Robot Technology and Application,2019Gait planning of gecko Robot based on inverse
Kinematics [J]. Li Hu; Sun Shudong; Duan Jun.
Mechanical Science and Technology,2011
Infrared induction control system [J]. Gong Shulan; Guo
Wencheng. Private Science and Technology,2007(09)Application and Development of Rescue Robot
Communication System [J]. Chen Wei; Mali; Wang Zhu.
Fire Science and Technology,2016(08)Analysis of Gait Control and Switching Strategy for
Quadruped bionic Robot [D]. Zhao Shuangyu. University
of Electronic Science and Technology of China,2014
Coordinated Gait Control of Hexapod Robot [J]. Han
Jianhai, ZHAO Shushang, LI Jishun. Mechanical andElectrical Engineering,2004(04)Mechanism Analysis and Design of a New Hexapod
Robot [D]. Xie Zhong. Yanshan University,2019

Authors Introduction
Mr. Hongpi Zhao
He is currently pursuing his undergraduate
degree at the School of Electronic Information
and Automation, Tianjin University of
Science and Technology.

Fig. 10 Fourth generation products
Finally, the fifth generation product has undergone an
in-depth industrial design to resemble the appearance ofthe robot to a turtle, giving it a more vivid and adorable
image and enhancing its user-friendliness. The fifth
generation product is shown in Fig. 11.

Ms. Yingfan Zhu
She is currently pursuing her undergraduate
degree at the School of Electronic
Information and Automation, Tianjin
University of Science and Technology. Her
research area is robotics engineering.

Ms. Zhihan Zhao
She is currently pursuing her undergraduate
degree at the School of Electronic Information
and Automation, Tianjin University of
Science and Technology. Her research area isrobotics engineering.

Fig. 11 Fifth generation products©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
363

Powered by TCPDF (www.tcpdf.org)

Teenage Mutant Ninja Turtles

Mr. Xin Liang
He is currently pursuing his undergraduate
degree in the School of MechanicalEngineering at Tianjin University of Scienceand Technology. His research area ismechatronics.

Ms. Lei Lv
She is currently pursuing her undergraduate
degree in the School of MechanicalEngineering at Tianjin University of Scienceand Technology. Her research area isindustrial design.

Dr. Yizhun Pen
He is an Associate Professor in Tianjin
University of Science &Technology. He
received a doctor’s degree in control theory
and control engineering from the Institute ofAutomation, Chinese Academy of Science in2006. His research field is intelligent robot
and intelligent control.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
364

Design of Grass Lattice Planter for Complex Environment Based on Adaptive Suspension
Technology
Shaokai Tian, Wenqi Fu, Yizhun Peng*

Tianjin University of Science and Technology, China
E-mail: *pengyizhun@126.com
www.tust.edu.cn

Abstract
The Complex environmental grass grid growers based on adaptive suspension adopted the high-performance stm32
as the master chip, and use the Bluetooth communication module, so that users can easily control the vehicle
driving, feeding and the posture of the pressing wheel through the mobile phone application. In addition, we haveintroduced new mechanical structures such as adaptive suspension to ensure effective planting of grass squares incomplex terrain such as sloping land. Our vehicle uses a Mecanum wheel motion system to squeeze seeds byrotating the friction wheels at high speed.

Keywords:STM32F103ZET6, Adaptive suspension, Machine vision, Machine learning

planting. In the circuit of this project, the stm32
microchip as the main control chip and the Bluetooth as
the communication module can control the walking,feeding and grass wheel posture of the vehicle throughthe mobile phone APP [3], and then complete the grasssquare planting in the slope with the new adaptive
suspension and other mechanical structures. The method
is the McNham wheel motion system, which uses the
motor to drive the friction wheel and squeeze the tennis
ball at high speed to launch, and uses STM32 to control.
The remainder of this paper is organized as follows.
The second part introduces the mechanical structure
design for the implementation of the robot function. In
the third part, the hardware circuit design of the robot isgiven. In the fourth part, the electronic control program
design of the robot is given. The fifth part summarizes
the main content of this article.

1. Introduction
In recent years, severe weather has become more andmore frequent. In some inland China, the rainfall isdecreasing year by year, and the land desertification is
becoming increasingly serious. The sharp decrease ofarable land has led to the harsh environment such as
yellow sand, which seriously affects People's Daily life.
With the scientific concept of development to today's
clear waters and green mountains are gold and silver
mountains, China has formed a set of localized sandcontrol route: grass square sand fixing. Grass square sandis getting more and more attention because of its
extraordinary sand prevention effect [1]. However, mostof the existing grass square planting methods in China are
still artificial. Although there has been professional grass
planting equipment in recent years, the investigation andinterview found that most of the machines can only beapplied to flat land. Under rolling roads and dune
landforms, there is no machine to plant, and it is still
manual work. The harsh and changing environment in thedesert greatly increases the risk of manual work [2].
In order to conduct the actual test of the developed
grass square planting system, the team designed anddeveloped the corresponding mechanical structure thatcan control the movement and integrate the feeding and

2. A Mechanical Structure Design
2.1. An Adaptive structure design
In the four-wheel movement system, the front and rear
wheels on the same side are connected by the connecting
rod structure shown in the figure, and the left and rightside connecting rods are connected through two sets of©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024365

Shaokai Tian, Wenqi Fu, Yizhun Pengswing arm structure. The swing arm can move up anddown through the linear bearing, and finally the left front
wheel and the right rear wheel move up and down
synchronously, and the left rear wheel and the right front
wheel move up and down synchronously. The above
structure can realize the all-directional adaptive of
various slope landforms, and can operate in the desert
undulating terrain [4]. Adaptive suspension diagram isshow in Fig. 1.

Fig. 2 Details of the swing arm mechanism
2.3. A Material supply design
The double output motor 1 is flange and the power istransferred to the roller 4. The roller is fixed on thealuminum pipe base through the bearing and steel shaft.
The roller connected with the motor is the active wheel,
and the roller not connected with the motor serves as thedriven wheel.
During operation, the motor rotates, the active roller
rotates, the active roller and the driven roller pre-tighten,
the active rod rotates and drives, and produces friction
with the grass between the main and slave roller, so as torealize the feeding function.
The slot explains: adding the slot can realize theadjustment of forage with different thickness and width,
so as not to produce excessive stress between the rollers
and increase the practical service life of the structure.
Material supply mechanism diagram is show in Fig. 3.

Fig. 1 Adaptive suspension diagram
2.2. An Electric swing arm link design
In order to better adapt to the undulating terrain such as
desert and Gobi, the wheel group hanging swing arm isdesigned with three degrees of freedom, which can meetthe rotation of Pitch axis, Yang w axis rotation and Y axis
sliding at the same time.
The following is the details of the upper and lower
movement of the swing arm: the village sleeve and thescrew set the distance between the two steel swing arms,
the flange is fixed on the swing arm through the screw,
the straight bearing is fixed on the flange by welding, andthe steel column is fixed on the steel plate 1 and 2 withthe screw, thus fixed on the frame. The linear bearings
can move up and down around the steel column or rotate.
Structural details of the swing arm connecting rod contact
point: the sliding shaft and the crosshead are fixed,
through the sliding shaft, the sliding coat can be rotatedby a small margin, and the swing arm can be rotatedaround the shaft through the thrust bearing and steel
column. The distance of the distance is fixed between the
two links by rigid jacket. Through the above mechanism,
the movement realizes the slip and the rotation on thewhole node space.[5]Details of the swing arm mechanism
is show in Fig. 2.

Fig. 3 Material supply mechanism diagram
2.4. A Management mechanism design
This structure adopts multi-link structure, through therotating motion into linear motion, drive the upper rod 3
rotate around the hinge 8 , the swing of the upper rod 3
will make the shock absorber 4 compression, thentransfer the force to the lower rod 6 , the lower rod 6 will
rotate around the steel shaft centering by the horizontal
support, so as to realize the upper and lower movement of
the pressing wheel.
According to the calculation and analysis, thestructure has only one degree of freedom, so the controlof the structure can be fully realized by controlling thenumber of circles turned by the motor, and the structure is©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
366

A Design of Grass

simple and easy to use. Managing mechanism diagram isshow in Fig. 4.

3. A Hardware Circuit Design
3.1. STM32F103ZET6 Master control module
Up to 8 timers, containing a separate watchdog timer anda window-type timer, allowing for automatic microreset.
This chip has up to 72 MHz working frequency. The bus
system is a synchronous serial peripheral interface thatenables the MCU to communicate with various
peripherals in a serial manner to exchange information.
The MCU architecture features an easy-to-use STM 32
platform ideal for motor drive [7]. STM32F103ZET6
Master control module is show in Fig. 6.

Fig. 4 Managing mechanism diagram
2.5. A Design of shock absorption structure(1) The function of the shock absorber⚫ Realize the transmission of force;⚫ The expansion of the shock absorber 4 canensure the balance between the grass pressing
wheel and the ground of the force, to ensure
the quality and efficiency of planting;⚫ Bear the impact of the irregular ground, protect
the vehicle structure, improve the service life;(2) Advantages: This system does not use the rigid
connection, but uses the shock absorber as a flexible
connecting rod, thinking from suspension,
innovative application solved the rolling road may
produce the disadvantages of rigid stress stressstructure, and by ensuring the range between the
ground wheel always small fluctuations and always
fit the ground, so the car can not only in the plane
planting, but also can be in the plane planting (slope,tunnel, etc.), and has the very good practicalapplication [6]. Damper is show in Fig. 5.

Fig. 6 STM32F103ZET6
3.2. A4988 Stepper motor drive module
Can adapt to the effects of various natural environments
to better achieve work in harsh environments. The
selected A4988 DMOS micro-step drive module withconverter and overcurrent protection has more stableperformance. A4988 Stepper motor drive module is show
in Fig. 7.

Fig. 7 A4988 Stepper motor drive module
Fig. 5 Damper
3.3. The HC-12 Bluetooth module
The super-long transmission distance and acceptance
performance of the Bluetooth module can well realize the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
367

Shaokai Tian, Wenqi Fu, Yizhun Pengpurpose of long-distance control. The HC-12 Bluetoothmodule is show in Fig. 8.

Fig.10 L298N DC motor drive module
Fig. 8 The HC-12 Bluetooth module

3.6. Circuit logic control
The overall electrical route of the machine is arranged tomake the machine line structure stable and safe, and will
not be damaged due to external or internal interference,
especially for the line layout where the line isconcentrated. Circuit chassis design makes most of thecircuit placed in the back of the body, increase theconvenience of human-machine interaction, while thecircuit, convenient line arrangement.
Select the power line, signal line and other electronic
components needed for the robot, reasonably arrange theline of the robot, to ensure the stability and safety of therobot operation [8].

3.4. L298N DC motor drive module
L298N can be used to drive our 12V DC deceleration
motor, L298N can directly control the motor, set thecontrol level through the I / O input of the main control
chip, can drive the motor forward reversal, simple
operation, good stability, can meet the high current
driving conditions of the DC motor. L298N DC motor
drive module is show in Fig. 9.

3.7. A Chassis module design(1) Main components: Bluetooth module, STM32 main
control, L298N DC motor drive module, 12V DC
deceleration motor.
(2) Implementation mode: the main control board
receives the chassis control instruction through theBluetooth module, then calculates the corresponding
instruction as the corresponding control information,and controls the motor speed with a PWM control
pulse through the IO port.
(3) Electrical connection: the 12V power supply
directly supplies power to the L298N DC motor
drive module, and the motor drive module isconnected to the motor and directly supplies power
to the motor. The 12V power supply supplies power
to the STM32 main control board after the 12-5V
step-down module, and the development board
supplies power to the HC-12 Bluetooth module
through the 3.3V interface.

Fig. 9 L298N DC motor drive module
3.5. 12V DC deceleration motor
12V DC deceleration motor is widely used, because its
excellent performance is often used in smart home,electronic products, smart car driver, precision medical
equipment, etc. In this grass square unmanned planting
car, the price is appropriate, product depression purchaseand assembly and other excellent performance. L298N
DC motor drive module is show in Fig. 10.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
368

A Design of Grass

3.8. A Material supply module design

4. An Electric Control Program Design(1) Main components: Bluetooth module, STM32 main
control, L298N DC motor drive module, 12V DC
deceleration motor.
(2) Implementation mode: the main control board
receives the control instruction through theBluetooth module, then calculates the corresponding
instruction as the corresponding control information,and issues the PWM control pulse to control thefeeding motor through the IO port, so as to realize
the material supply and discharge.
(3) Electrical connection: the 12V power supply
directly supplies power to the L298N DC motor
drive module, and the motor drive module isconnected to the motor and directly supplies power
to the motor. The 12V power supply supplies power
to the STM32 main control board after the 12-5V
step-down module, and the development board
supplies power to the HC-12 Bluetooth module
through the 3.3V interface.

4.1. A System main body logic design
Program flow chart is show in Fig. 11.

Fig. 11 program flow chart

3.9. A Management module design

4.2. A Chassis system logic design(1) Main components: Bluetooth module, STM32
master control, A4988 step motor drive module, 42
step motor.
(2) Implementation mode: the main control board
receives the control instruction through theBluetooth module, then calculates the corresponding
instruction as the corresponding control information,and sends the PWM control pulse and IO level
control the rotation of the stepping motor throughthe IO port, so as to realize the control of the depthof the pressing structure.
(3) Electrical connection: the 12V power supply
directly supplies power to the A4988 stepping motor
drive module, and the motor drive module isconnected to the stepping motor to directly supply
power and control it. After the 12V power supply
supplies power to the STM 32 through the 12-5V
step-down module, and the development board
supplies power to the HC-12 Bluetooth module
through the 3.3V interface [9].

Chassis.c Three subroutines are set up, namely, chassis
motor initialization program, stepper motor initialization
program and chassis control program.
(1) In the chassis initialization program, complete is the
GPIO used for the chassis motor.
(2) In the initialization program of the stepping motor,
the initialization of the IO port of the control
stepping motor is completed.
(3) In the chassis control program, according to thecontrol information after the incoming processing,
set the PWM wave with different duty cycle ratio tocomplete the control of the motor speed and steering
under the corresponding situation.
4.3. A Logical design of the communication system
After receiving the message at the serial port, we need toprocess it. However, because there is more logic in thisproject, we set up a separate structure to integrate theinformation flow and facilitate us to process the data
uniformly.
In this structure, we integrate the information ofchassis speed, direction, stepping motor status andfeeding motor status. In the serial receiving interrupt, we
directly judge the received message in the interrupt, acharacter check function is called to judge the instruction©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
369

Powered by TCPDF (www.tcpdf.org)

Shaokai Tian, Wenqi Fu, Yizhun Pengaccording to the received characters and rewrite thevariables of the structure body.
In addition, in order to facilitate the user's control,
we also set a timer interrupt, for timing to clear
instructions, can realize the micro operation of the robot,
complete fine control.

8. Mei Yang, Muhua Liu, Feiyu Zhu ,et al, Design of Soil
Environmental Information Sensor Based on Bluetooth
Technology,
Acta
Agriculturae
Universitatis
Jiangxiensis,2023,4:pp.963-971.
9. Linfeng Li, Yong Wang, Yongchun Xie, Learning Space
Robotic Manipulation via Multi-View Visual Goal
Generation,
Aerospace
ControlandApplication,2022,2:pp.18-28.

5. Conclusion
Authors Introduction

Dunes or gobi desert governance is a long time, now
large machinery is mainly plane industry planting,planting can not complete the dunes, moreover, due to thedune itself has strong liquidity, not only lead to humanresources in the same repeated continuous consumption,
also has a potential danger: quicksand, a threat to people's
life and property safety. However, in order to create along-term sustainable green water peak, we shall not
ignore the one in the field of governance, so the teamproposed adaptive suspension has been practical, breakthrough the existing laying machinery technical problems,
can set foot on the ecological restoration in this field, andthe wireless control, no one to close interference doindustry, great guarantee the personal safety, the design
breakthrough existing laying machinery technicalproblems.

Mr. Shaokai Tian
He is currently pursuing his undergraduate
degree at the College of MechanicalEngineering, Tianjin University of Scienceand Technology

Ms. Wenqi Fu
She is currently pursuing her undergraduate
degree at the School of Electronic
Information and Automation, Tianjin
University of Science and Technology.

Dr. Yizhun Peng
He is an Associate Professor in Tianjin
University of Science &Technology. He
received a doctor’s degree in control
theory and control engineering from theInstitute of Automation, Chinese Academyof Science in 2006. His research field isintelligent robot and intelligent control.

References
1. Yaozong Wang, Xinbin Yue, Jiali Xie,etal,Desertification evolution in the sandy region to the east of
the Yellow River in Ningxia from 2000 to 2020,Journal of
Desert Research,2023,4:pp.31-40.
2. Chengzhi Hou, Danqing Huang, Dongwei Gui,etal,Spatiotemporal variations of climate extremes andinfluential factors in deserts and sandy fields of northern
China from 1961 to 2019,Scientia Geographica
Sinica,2023,8:pp.1495-1505.
3. Yan Zhang, Jiatong Li, Xiaoyi Song,et al,Survey of IoT
Device Security Detection,Journal of Computer Researchand Development,2023,10:pp.2271-2290.
4. Yang Liu, Unloading impact simulation and test of
luffing jib tower crane, Journal of Jilin University:
Engineering and Technology Edition,2022,6:pp.1292-1300.
5. Haimin Han, Design of Improved Damping Device Usedin Mechanical Equipment, Mechanical Engineering &Automation,2022,4:pp.113-114.
6. Chunyang Liu,Fan Chen,Zhangfei Wang,et al,Design oflane keeping assistance control system based on
STM32,Transducer
andMicrosystem
Technologies,2022,3:pp.68-71.
7. Jinhua Wang, Jia Liu, Yonghe Zhuang, et al, A new logic
control method of the high precision current/frequency
converter, Microelectronics & Computer,2023,6:pp.85-89.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
370

A Design of Intelligent Ecological Multifunctional Plant and Animal Breeding System
Suqing Duan, Yuntian Xia, Siyi Wang, Yizhun Peng*

College of Electronic Information and Automation, Tianjin University of Science and Technology, China
E-mail: *pengyizhun@tust.edu.cn
www.tust.edu.cn

Abstract
The intelligent ecological multi-functional system for animal and plant cultivation integrates PLC programmablecontrollers and microcontroller-based automatic control technology. This innovative approach amalgamatesaquaculture with horticulture, fostering a symbiotic environment for the breeding of animals and plants. The
system's operations are categorized into four key components: mechanical structure, environmental perception,
automatic control, and intelligent networking. Its comprehensive functionality encompasses plant monitoring,self-checking of temperature and humidity, group control for fertilizer replenishment, consistent temperature
regulation, full-spectrum illumination, advanced filtration, fogging supplementation, versatile water management,
fertilizer blending, and water purification through sterilization.
Keywords: Intelligent Ecology, PLC programmable controller, Environmental sensing, Eco-agriculture,
microcontroller, monitoring

1. Introduction

The rest of this paper is organized as follows. The
second part introduces the top ten functional designs. The
third part introduces the working principle of theintelligent ecological multifunctional plant and animal
breeding system. The fourth part is hardware design,
including mechanical structure part design, environment
sensing part design, automatic control part design andintelligent networking part design. The fifth part is thesoftware design. The sixth part summarizes the maincontent of this paper.

With the rapid development of science and technology,
intelligent technology has emerged in the field ofagriculture. Based on Siemens S7-1200 version PLC
programmable controller and AT89C51 microcontroller
automatic control technology [1]. this thesis aims tointegrate aquaculture and horticulture planting, andinnovatively realize plant and animal symbiotic
aquaculture. The most important feature of the system isthat it can realize ecological energy saving,environmental protection recycling and symbiotic
breeding of plants and animals, which pushes theintelligent ecological breeding to a new height.
The work is mainly composed of four modules:
mechanical structure, environmental sensing, automatic
control and intelligent networking. System functions
include plant monitoring, temperature and humidity
self-checking, group control and fertilizer renewal,
constant temperature and heating, full illumination,
multiple
filtration,
fogging
supplementation,
multi-purpose water, fertilizer mixing and sterilization ofwater purification and other special functions [2]. In
addition, the system introduces an animal feeding system,
which includes an adaptive feeding cabin and an
intelligent feeding system that monitors the physiologicalparameters of the animals and automatically feeds themat regular intervals and quantities in order to realize
intelligent feeding management.

2. Ten functional designs
Intelligent ecological multi-functional animal and plant
breeding system aims to realize fully automatic groupcontrol of plants and aquatic animals, which contains ten
key functions such as plant monitoring, temperature andhumidity self-checking, group control and fertilizer
renewal, constant temperature and heating, full
illumination and light supplementation, multiple filtration,and fogging supplementation. The application of thesetechnologies not only significantly reduces manualintervention and improves breeding efficiency andproduction, but also promotes the recycling of resources
and realizes environmentally sustainable production.
This intelligent ecological multi-functional plant andanimal breeding system realizes all-round monitoring andmanagement of plants and animals by integrating
advanced monitoring and control technologies,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
371

Suqing Duan, Yuntian Xia, Siyi Wang, Yizhun Penghumidification device to ensure that the photosynthesis,
respiration and transpiration of plants proceed normally.
Through time triggering and intelligent determination of
air humidity at 8 a.m., 11 p.m. and 5 p.m., the system
decides whether or not to humidify to meet the humidity
needs of the plants at different times of the day, thus
providing a suitable growing environment [3]. This
intelligent humidification system helps to maintain thephysiological activities of plants and improve thebreeding efficiency.

minimizing manual operations and thus improving
overall breeding efficiency and production. The top tenfunctional diagrams of the system are shown in Fig. 1.

2.4. Plant group control automatic fertilization design
The system performs real-time sensing and calculation
through sensors at the height of the plants, and judges thereasonable amount of fertilizer (CM/MG) according to
the growth status of the plants. In terms of time triggering,
the system intelligently sets 3:00 pm to 10:00 pm every
day as the fertilizer application time period to ensure thatthe plants get the right amount of nutrients during thistime period. In addition, soil sensors are used to monitor
the levels of nitrogen, phosphorus, potassium and otherfertilizers in the soil. The system intelligently determines
the soil condition according to the plant species and theseasons, and applies the right amount of fertilizer whenneeded. Such an intelligent fertilizer application
mechanism is designed to ensure that plants are suppliedwith appropriate nutrients, thereby improving breeding
efficiency and yield.

Fig. 1 The top ten functional diagrams of the system
2.1. Aquaponics Technology Design
Based on the concept of energy recycling, the system isguided by the concept of watering vegetables with fish
water and improving the growth environment of fish withvegetables, integrating aquaculture and hydroponic
cultivation technologies to build a harmonious
aquaponics system. In urban family application, thistechnology can reduce the purchase expenditure ofvegetables and fishery products and create a green andenergy-saving family living environment.

2.5. Plant group control automatic watering design

2.2. Temperature parameter change automatic
judgment sliding table contraction design

The system decides whether watering is needed bychecking whether the soil moisture reaches the set value
through real-time soil moisture monitoring every 10
minutes. In addition, the system intelligently takes into
account plant species and seasonal changes, and based onthese factors determines whether watering is required forthe day, ensuring that soil moisture meets the plant's
growth needs. This integrated intelligent watering
mechanism aims to provide plants with the right amountof water according to the actual situation, ensuring thatthey receive appropriate irrigation in different seasonsand growth stages, and improving breeding efficiency.

The system monitors the ambient temperature through an
RS485 industrial-grade temperature and humidity
transmitter to ensure that the slide table is intelligently
triggered to retract automatically in adverse climaticconditions (less than 15 degrees Celsius or more than 30
degrees Celsius) to prevent plants from being damagedby cold or overheating. Meanwhile, through intelligent
determination of the current time, the system realizes
artificial light supplementation at 18:00 pm to ensure thatplants can still obtain sufficient light when it is dark,
while it automatically extends at 8:00 am to absorb
sunlight to provide a good growing environment forplants. Such time determination and intelligent regulation
mechanism effectively guarantees the growth needs ofplants in different time periods.

2.6. Plant group control
replenishment design

automatic

nutrient

The system intelligently calculates the corresponding
levels of nutrients such as nitrogen, phosphorus andpotassium (CM/MG) through plant height sensing. In
terms of time triggering, the system is set toautomatically replenish nutrients once a month toimprove plant resilience and resistance and activate
microbial vitality in the soil [4]. This integrated highly
sensible and automatic replenishment mechanism helps to2.3. Plant group control automatic humidification
designThe system is equipped with a humidity sensor tomonitor the humidity of the plant growth environment inreal time. When the humidity is lower than 80%, theintelligent
systemautomaticallytriggers
the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
372

A Design of Intelligent

ensure that plants are supplied with appropriate nutrientsat different growth stages, improving their resilience andthus enhancing the ecological balance of the wholeaquaculture system.

effect, and the use of red-rich light sources for
supplemental lighting can advance plant flowering andfruiting and promote organ formation. Blue light helps
stomata open, promotes the entry of external carbon
dioxide, increases the rate of photosynthesis, and isconducive to protein synthesis. Blue light accelerates thegrowth of short-daylight plants, slows down the growthof long-daylight plants, and promotes leaf growth.
Artificial supplemental light using a blue light-rich lightsource can delay flowering and allow full plant growth[5].
Plant grow lights are electric light sources designed
to stimulate plant growth by emitting electromagnetic
waves suitable for photosynthesis, and are used inapplications where natural light is not available or where
additional supplemental light is required. In winter, when
daylight hours are insufficient to meet the needs of plants,
plant lights are used to extend light hours and prevent
plants from growing unduly. Meanwhile, the plant curtain
green light realizes cooling effect in summer by blocking
heat, shading strong sunlight and providing a cool indoor
environment; in winter, it prevents cold air from
intruding and provides thermal insulation. The ambient
light is used to provide extra light, highlight the color of
the fish, and meet the fish's need for light. In the absenceof sunlight, the ambient light creates a comfortableenvironment for the fish tank, creating a more perfectviewing effect. The finished rendering and exterior
design are shown in Fig. 2.

2.7. Automatic Oxygen Supply Design for Aquatic
Animals
According to the change of water temperature, automatic
oxygen supply is realized to ensure the oxygen demandof fish in different temperature ranges.
2.8. Automatic sterilization design
The system adopts the principle of ozone sterilization,
through which ozone acts on the cell membrane of
microorganisms, leading to cell death, so as to achieve
the purpose of sterilization, anti-inflammation anddetoxification. In addition, the system also applies UV
germicidal lamps, which utilize UV rays to destroy the
DNA or RNA molecular structure in the cells of various
microorganisms in the water, effectively killing various
pathogens in the water in order to keep the water quality
clean. This integrated sterilization system is designed toensure the hygienic condition of aquaculture water,
improve water quality, and guarantee the healthy growthof plants and animals.
2.9. Solar photovoltaic power generation system design
The system is centered on PLC and HMI, and is poweredby electricity from the solar photovoltaic power
generation system reserve. According to the light
condition, the system realizes automatic switching of
power supply to ensure the normal operation of theequipment under different light conditions. In addition,
the system intelligently utilizes LED plant growth lights
for automatic replenishment. When natural light isinsufficient, the solar PV system activates the LED plant
growth lights to provide an additional light source topromote plant growth. This integrated PLC and solar
photovoltaic power generation system, as well as theautomatic replenishment function of the LED plant
growth lamps, effectively guarantees the sustainable
operation of the system and the growth needs of plants
under different light conditions.

Fig. 2 The finished rendering and exterior design
3. Working Principle
Firstly, the ornamental and convenient management ofplants is realized through the design of plant sliding table.
The plant sliding table can display the plants in thebalcony or outdoor community, and when theenvironment changes, the plants can be cooled,humidified, and replenished with light through PLC
control in order to reduce the labor cost and improve thebreeding efficiency. The plant curtains are driven by 24V
motor for 90 degree opening and closing. Secondly, thesystem supports hydroponic plant cultivation, which cansave space utilization by putting the plant curtains away.
The root system of hydroponic plants can filter the fishtank water flowing through the plant curtains, realizing
the recycling of aquaponics resources. Finally, the system

2.10. Design of LED Plant Grow Lights for Yield
Improvement
Spectral regulation is crucial to plant growth, and red
light promotes chlorophyll formation and carbohydrate
synthesis, fuels the growth of long sunlight plants, delays
the growth of short sunlight plants, and promotes seedgermination. Theoretical and experimental evidenceshows that red light has the strongest photosynthetic©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
373

Suqing Duan, Yuntian Xia, Siyi Wang, Yizhun Peng

is equipped with an industrial control touch screen to
realize real-time feedback on the monitoring of plants andanimals and human-computer interaction.
For automatic sensor detection, ultrasonic ranging,soil sensors, environmental sensors, PLC time calculationand other multiple logic judgments are used to improve
the accuracy of plant control and help improve thesurvival rate of plants. Fertilizer pump adopts peristaltic
pump, and through the "one drop one number" control
method, it realizes the precision of fertilizer application
and avoids the situation of using too much or too little
fertilizer. In the filtration system, through a variety offiltration methods, such as hydroponic plant stalks, UV
lamps, ozone and active filters, etc., the depth of the fishtank water poisoning and filtration, to achieve the healthy
operation of the aquaponics system. The working
schematic is shown in Fig. 3.

three-dimensional Rubik's Cube structure is characterized
by a large available space, which is divided into fivedifferent unfolding surfaces, with different functions
corresponding to different surfaces, which are
specifically divided into the "display and observation
surface", "aquatic planting surface", "photovoltaic
surface" and "energy collection surface" [6]. The
different surfaces have different functions, specificallydivided into "display and observation surface", "water
planting surface", "photovoltaic surface", "energy
collection surface", "sliding platform structure surface"and "control and manipulation surface".The mechanicalstructure is shown in Fig. 5.

Fig. 5 The mechanical structure
4.2. Environment Sensing Part Design
Composed of various high-precision sensors, theenvironment sensing part is equivalent to the "eyes" and"ears" of the intelligent ecological multi-functional

Fig. 3 The working schematic
4. Hardware Design
The overall structure of the intelligent ecologicalmultifunctional plant and animal breeding system isdivided into the following four major components. The
overall structure is shown in Fig. 4.

animal and plant breeding system, which can
automatically detect the working state of the system andconvert various process parameters such as temperature,humidity, flow rate, liquid level and composition into
uniform standard signals. Physical quantities such astemperature, humidity, flow, liquid level, composition,etc. are transformed into uniform standard signals to meetthe requirements of information transmission, processing,storage, display, recording and control, etc. It is the firstlink to realize automatic detection and automatic control.
The high precision sensor map is shown in Fig. 6.
Fig. 6 The high precision sensor map

Fig. 4 The overall structure
4.1. Mechanical structure design

4.3. Automatic control part design

The three-dimensional Rubik's Cube structure is adopted,
which is inspired by the aerospace station. The

The system is mainly composed of four parts: controller,
sensor, actuator and transmitter. The controller is the core

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
374

A Design of Intelligentof the system, responsible for automated control tasks
without direct human intervention. Through a type ofwork, a state or a parameter in a machine, equipment or
production process, it operates automatically according topre-set rules. Electricity or other energy sources are usedand converted into drive action by means of motors orother devices [7]. Sensors are responsible for detecting
process parameters and transmitting the measured values
in the form of specific signals for display and regulation.
The role of the sensors is to convert various processparameters, such as temperature, flow rate, compositionand other physical quantities, into uniform standardized
signals, which are then transmitted to the regulator andtouch screen for regulation, indication and recording.
Such an automatic control system ensures the continuityof the production process and stabilizes the system
operation. The controller and controlled object diagram isshown in Fig.7.

Fig. 8 The smart Wi-Fi module diagram
5. Software Design
5.1. System logic circuit diagramEnvironment Sensing
Part Design
The software system of the Intelligent Ecological
Multifunctional Animal and Plant Breeding System is the"brain" of the system, which is responsible for monitoring,
controlling and coordinating various components torealize efficient animal and plant breeding. The system
logic circuit diagram is a graphical representation of theinternal logic structure of the software system, showing
the relationship and information flow between different
modules. The system logic circuit diagram is shown in
Fig. 9.

Fig. 7 The controller and controlled object diagram
4.4. Intelligent networking part design
As the communication hub of the intelligent ecologicalmulti-functional animal and plant breeding system, theintelligent Wi-Fi module plays an important role inconnecting with the outside world, including the close
connection with other intelligent devices. As the corecomponent of the smart home LAN, the module is able toindependently complete a variety of functions of thehome internal network without the intervention of otherdevices. It converts and shares information between
communication protocols, realizing efficient dataexchange between networks. Meanwhile, the dataexchange function between the smart Wi-Fi module andthe external communication network enables the smoothrealization of intelligent functions such as remote control,
scene control, linkage control and timing control. The
design ensures an efficient and seamless connection
between the smart farming system and external devices
and networks, further enhancing the intelligence of thewhole system [8]. The smart Wi-Fi module diagram isshown in Fig. 8.

Fig. 9 The system logic circuit diagram
5.2. Human-computer interaction diagrams
The human-computer interaction diagram of the softwaresystem of the intelligent ecological multifunctional
animal and plant breeding system is the part of the systemthat interacts with the user directly, providing the user
with a friendly operation experience through an intuitivegraphical interface. The human-computer interaction
diagram is shown in Fig. 10.

Fig. 10 The human-computer interaction diagram©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
375

Suqing Duan, Yuntian Xia, Siyi Wang, Yizhun Peng

5.3. Relay and Communication

improve the breeding efficiency and realize the effective
use of resources.

The relay and communication diagram of Intelligent
Ecological Multifunctional Animal and Plant Breeding
System software system includes PLC, sensors, relay
module, communication module, network connection,
data transmission protocol, remote monitoring equipment,
etc. PLC realizes equipment control through relay module,
sensors monitor environmental parameters, andcommunication module realizes equipment datatransmission and remote monitoring [9]. The relay andcommunication communication diagram is shown in Fig.
11.

5.5.1. Main Picture
The main drawing is shown in Fig. 13.

Fig. 13 The main drawing(1) Manual mode screen selection(2) Automatic mode screen selection(3) Data monitoring screen selection(4) Running status screen selection(5) Plant weekly record screen selection(6) Plant status screen selection
Fig. 11 The relay and communication communication
diagram

5.5.2. Manual Mode
The manual mode diagram is shown in Fig. 14.

5.4. Aquaculture System Logic Diagram
In the software system of Intelligent Ecological
Multifunctional Animal and Plant Farming System, thelogic diagram of aquaculture system is mainly related tothe monitoring, control and optimization of aquatic
ecology. The logic diagram of the aquaculture system isshown in Fig. 12.
Fig. 14 The manual mode diagram
The system status is divided into running and stopped,
and the water colonization system performs sterilization
and water filtration every 15 minutes. At the same time,
the plant aquatic curtains and outreach sliding table has
five states: not in motion (stopped), retracted (being
retracted), retracted to complete, stretching out (being
stretched out), stretched out to complete. The logic ofsuch states is clear and helps the user to understand thesystem operation status in real time.
5.5.3. Automatic Mode

Fig. 12 The logic diagram of the aquaculture system

The automatic mode diagram is shown in Fig. 15.
5.5. Human-computer exchange design
Our system integrates advanced technologies andprovides five control modes such as touch, WIFI, voice,Bluetooth, and physical buttons, which provide users
with more convenient means of controlling and managing
animal and plant breeding. At the same time, the system
also introduces intelligent self-learning function, which
can automatically update the best planting plan toFig. 15 The automatic mode diagram©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
376

A Design of Intelligent

When the system is running, the user can set the running
time of the automatic mode through the timer function toensure that the system automatically performs thebreeding tasks according to the preset plan within a
specific time period. At the same time, the systemmonitors the growth of the plants in real time and ensuresthat the necessary breeding operations, such as
fertilization and watering, are carried out at the right timethrough the real-time feedback of the plant height. In
addition, the system also takes into account the needs ofdifferent plants and provides a potted plant species
selection function, which allows users to choose theappropriate breeding method according to the specificplant species, ensuring a more intelligent and adaptablesystem.

Journalof
Agricultural
Mechanical
Chemistry.
2023,44(08):75-80.
4.Zhu Panpan, Ma Yanping, Zhou Zhongxiong et al. Traceelement zinc and plant nutrition and human health[J]. Fertilizer
and Health. 2021,48(05):16-18+23.
5.Long Jia-huan,Pu Min,Huang Zhi-wu et al. Research progressin spectral regulation of plant growth and development[J].
Journal of Lighting Engineering. 2018,29(04):8-16.
6.Zhang Xiaoyu,Liu Chang,Shi Liming et al. Optimized designof mobile deformable assembly with integrated support
structure of skinned dot matrix and space station application[J].
Journal of Solid Mechanics. 2022,43(05):551-563.
7.Fang Zewen. Application of configuration king and Siemens
PLC in wastewater treatment control system[J]. China
Equipment Engineering. 2022(09):269-271.
8.Song Ruibo,Zhang Yan,Lian Menghui. Design of smart homesystem based on internet of things[J]. Modern Industrial
Economy and Informatization. 2023,13(10):99-101+106.
9.Chen Ying. Intelligent monitoring and control ofelectromagnetic relay based on STM32 and LoRa
communication[J]. Information and Computer (Theoretical
Edition). 2023,35(13):55-57.

6. Conclusion
Our system is of great significance in promoting thedevelopment of modern agriculture. Facing the problemsof shortage of social agricultural workers, low production
efficiency and low value-added of agricultural products,
our design builds a unique and independent animal andplant breeding system based on technologies such as
artificial intelligence, Internet of Things, cloudcomputing and big data. In addition, our plant and animal
breeding system also has the potential for mixed plant
and animal breeding in large-scale agro-ecological farms.
Through smart WiFi and 3D management functions, thesystem is able to realize precise management of plants
and animals, which improves the efficiency ofagricultural production. More importantly, this system
helps to solve the problem of environmental pollution,
especially in large-scale agro-ecological farms, where ourtechnology can effectively minimize the impact on theenvironment and provide more high-quality ecologicalagricultural and sideline products. By enhancing people's
aesthetic requirements for a beautiful ecologicalenvironment, our work plays an active role in building amore sustainable and healthy agro-ecosystem.

Authors Introduction
Ms. Suqing Duan
She is currently pursuing her undergraduate
degree at the School of Electronic
Information and Automation, Tianjin
University of Science and Technology. Her
research field is embedded system.

Mr. Yuntian Xia
He is currently pursuing his undergraduate
degree at the School of Artificial Intelligence,
Tianjin
Universityof
ScienceandTechnology. His research area is neuralnetworks.

References

Ms. Siyi Wang
She is currently pursuing her undergraduate
degree at the School of Electronic
Information and Automation, Tianjin
University of Science and Technology. Her
research area is automation.

1.Wang Jianju. Siemens S7-1200PLC communication
research[J].
Southern
Agricultural
Machinery.
2021,52(18):146-149.
2.Yuan Quan, Liu Yongjun, Huang Weiwei et al.
Characteristics of water quality indicators and microbialcommunity diversity in fish-vegetable symbiotic aquaculture
system[J].
Jiangxi
Journalof
Agriculture.
2023,35(09):156-161+168.
3.Li Hongmei,Wu Jinji,Wang Liangming et al. Design and testof intelligent cultivation equipment for home fungus[J]. Chinese©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
377

Powered by TCPDF (www.tcpdf.org)

Suqing Duan, Yuntian Xia, Siyi Wang, Yizhun Peng

Dr. Yizhun Pen
He is an Associate Professor in Tianjin
University of Science &Technology. He
received a doctor’s degree in control theory
and control engineering from the Institute ofAutomation, Chinese Academy of Science in2006. His research field is intelligent robot
and intelligent control.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
378

Design of a Fully Automated Logistics Handling Platform
Hongpi Zhao, Jianfeng Qin, Yizhun Peng*

Tianjin University of Science and Technology, Tianjin, 300222, China
E-mail: *pengyizhun@126.com
www.tust.edu.cn

Abstract
With the progress of science and technology and the development of society, the world's countries on industrial
production efficiency, intelligent manufacturing transformation continues to grow. Therefore, we designed a fully
automatic logistics handling platform, which is a multi-modular device based on mechanical design,
microcontroller control, visual positioning, the device can realize autonomous identification, autonomoustransportation, improve production efficiency, the device has a wide range of applications, can be applied to foodpackaging, parts processing, intelligent manufacturing, and other automation scenarios.
Keywords: Mechanical design, Circuit design, stepper motor drive, logistics and transportation

1. Introduction
With the development of society and the progress ofscience and technology, artificial intelligence technology
is gradually changing the traditional mode of industrial
production [1]. Through the introduction of intelligent
manufacturing technology, some advanced products suchas unmanned technology, drone courier, intelligent
medical care and so on are born. This design is based on
the fully automatic logistics handling platform in thecontext of intelligent manufacturing technology, which
effectively solves the problem of time-consuming andlabor-consuming manual operation [1], [2], [3], [4].
The rest of this article is organized as follows. thesecond part introduces the mechanical model and related
structural analysis, the third part gives the finite element
analysis, and the fourth part shows the related hardware
designachieved by means of a screw slide and rack and pinion
mechanism. The main features of this system design arestability and reliability in the face of a variety of items tobe transported.
2.1Design of mechanical parts
The mechanical part of the transportation and handlingsystem of this design is mainly composed of thefollowing parts:(1) Screw slide: used to transport items from the starting
position to the designated position.
(2) Rack and pinion mechanism: used to ensure stability
and reliability during transportation.
(3) Pneumatic Cylinder: Used to carry out thetransportation and stamping assembly of the articles.
The design of the mechanical part is shown in Fig. 1.

2. Mechanical modeling and related structural
analysisAutomated item transportation and handling systems arean essential part of the production line in modern
factories. Its main purpose is to transport items from one
place to another and to perform handling and assembly
operations when needed. Such systems usually consist of
many different mechanical, electrical and control
components to accomplish various tasks. In this product,
the transportation and handling of items is mainly

Fig. 1 The Mechanical design drawings
2.2Component Introduction©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
379

Hongpi Zhao, Jianfeng Qin, Yizhun Peng

(1) Screw slide table: It is a kind of mechanism thatutilizes the principle of screw rotation to carry out
linear motion, which consists of threaded rod andslider, and pushes the slider to carry out linear
motion through the rotation of the threaded rod.
Screw slide has the advantages of high precision,
good stability and smooth movement, so it is widelyused in automated item transportation and handlingsystems. Screw slide is one of the most importantcomponents in the system and is responsible fortransporting items from the starting position to thespecified position. Screw slides usually consist of athreaded rod, a slider and a guideway. The threaded
rod rotates to drive the slide in a linear motion, while
the guide rails are used to ensure the orientation andstability of the slide.
(2) Rack and pinion mechanism: it consists of a set ofgears, racks and guiding shafts, which is mainly used
to realize the overturning and handling of the goods
and ensure the stability of the goods in thetransportation process. At the same time, the guiding
shaft in the rack and pinion mechanism can avoid thelateral displacement of the spiral slider in thetransportation process.
(3) Cylinder: It is used to realize the handling andstamping assembly of the articles. When the article
reaches the specified position, the cylinder starts towork, clamps the fixture to the article, and moves thearticle to the stamping assembly area for operation.
The system has a variety of functions, includingtransportation, handling and stamping assembly of items,
etc., and at the same time, it is characterized by stability
and reliability, easy operation, safety and high efficiency.
It is suitable for various application scenarios in logistics,
production and manufacturing industries.
3.

Mechanical design

the high requirements of precision machining.
3.3Manufacturing process
The mechanical parts in this system mainly take twokinds of machining methods: turning and milling, inwhich turning is mainly used for machining shaft systemparts, and milling is used for machining otherconfigurations of parts.
The following table provides a brief parts drawings,
drawings include a brief machining accuracy, assembly
requirements note. As shown in Table 1.
Table 1 Parts Drawings
Part Name

Summary Drawings
Machining precision

Turntable

Flip

The following focuses on the mechanical design of thehandling part of the automated article transportation andhandling system.

Table
Stand

3.1 Design Idea
The syringe pushes the rack and pinion to mesh with thegear to provide the circumferential force, the gear isconnected with the shaft in the handling mechanism
through the key to drive the shaft to rotate
circumferentially, and at the same time the shaft isconnected with the loading fixture in the handlingmechanism through the key to realize that the syringe
drives the loading fixture to rotate circumferentially torealize the handling of the goods.

Flip
Carrier

3.2Material selection
The material selection of the parts of this system is 6063
aluminum alloy, because 6063 aluminum alloy has
excellent weldability, extrudability and plating, good
corrosion resistance, toughness, plasticity, and can meet
©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
380

Design of a Fully

ensure its safety.
Next is the flip shaft, as the most core part, its safety isalso the most important part of the flip mechanism. As
shown in Fig. 3.

Flap axle

4.

Finite Element Analysis and Simulation

Finite element analysis is the use of mathematical
approximations to simulate real physical systems. Byutilizing simple but interacting elements, it is possible toapproximate a real system with a finite number of
unknowns to an infinite number of unknowns.
The tipping mechanism is based on mechanical design
to analyze the degree of deformation when it is subjected
to forces to ensure its safety. Because the material of thedevice is aluminum alloy 6063, by checking its
permissible stress is 160MPa, through solid works finite
element analysis, the following diagrams are obtained.
As shown in Fig. 2 is the cylinder table.

Fig. 2 Cylinder table
As can be seen from the figure, this design is not safe,
although the actual cylinder table is not subjected to sucha large force, but it can be added with reinforcement to

Fig. 3 Flap axle
From the above figure, it can be seen that the
maximum shear stress of this shaft is 2.553×10⁷Pa,
while the permissible stress of aluminum alloy 6063 is
1.6×10⁸PA.
Therefore this shaft is safe.
There is also the fit between the rack and pinion,
which is the key condition for the flip mechanism to beable to flip. This is shown in Fig. 4.

Fig. 4 Pinion
As can be seen from the figure, the maximum shear
stress is 3.102×10⁹Pa, which is too large compared withthe permissible stress, so it is not safe, but it is still
feasible in the flip mechanism because it is calculated not©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
381

Hongpi Zhao, Jianfeng Qin, Yizhun Peng

to use such a large force.
Overall, the designed tilting mechanism is completely
feasible.
5. Hardware design
To accomplish this, the team designed a PCB
motherboard that implements the above features. The
PCB schematic diagram is shown in Fig. 5.

Fig. 5 PCB schematic diagram
The design of this PCB board with STM32F103C8T6
as the main chip, equipped with a stepper motor driverand a three-way relay, realizes precise stepper motor
control and solenoid valve on/off. Through programming,the board can control the speed, direction and movement
distance of the stepper motor, while it can control theon/off of the circuit through the relays to realize
automation control.
Fig.7 Stepper Motor Driver and TB6612FNG Stepper
5.1 Solenoid Valve Drive:

Motor Driver Module
The actual principle of a stepper motor driver is toconvert a pulse signal from a control system into anangular displacement of the stepper motor. Whenever the
stepper driver receives a pulse signal, it drives the
stepper motor to rotate in a set direction at a fixed angle(called the "step angle"). The stepper motor rotates stepby step at a fixed angle. The angular displacement can becontrolled by controlling the number of pulses for precise
positioning. The speed and acceleration of the motor
rotation can also be controlled by controlling the pulse
frequency for speed regulation [5], [6], [7], [8], [9].

The Relay is shown in Fig. 6.

Fig. 6 Relay
Through the main control board to send high and low
level signals to control the relay on and off, and thencontrol the solenoid valve on and off.
5.2 Stepping motor drive:
This design mainly adopts two stepper motor driving
methods, the first one is stepper motor driver and thesecond one is TB6612FNG stepper motor driver module[4], [5], as shown in Fig. 7.

5.3 Voltage regulator moduleThis design voltage regulator module mainly uses thechip for AMS117-3.3V for 3.3v voltage regulator design.
6.

Product Summary

Through the test, the product can complete therecognition of objects and achieve the desired position toachieve flipping and stamping through the precise controlof the stepper motor, which can be adapted to theautomatic scenes such as product packaging and parts
processing.
References
1. Design and research on control system of handling robot
in automation perspective[J]. ZHANG Qiang;PAN©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
382

Powered by TCPDF (www.tcpdf.org)

Design of a Fully

Zhendong;TAO Chenghu; CAO Zhenbo. Science andTechnology Innovation Herald,2020
2. A practical design of variable interpolation drive
controller for stepper motor[J]. LI Lingjuan; LIU
Jinglin;WANG Can. Microtronic Motor,2008
3. Stepper Motor Speed Control Method Based on single
chip Microcomputer [J]. Yu Jianguo. Electric Times,2006
4. Overview of stepper motor [J]. Li Lihua; Zhang Yongli.
Private Science and Technology,2008
5. Cause Analysis and Solution of stepping motor loss [J].
Xie Min. New Technology and New Process,2008
6. New stepping motor [J]. Wang Cailong. Micro Electric
Machine,1984
7. Several Research on Mechanical Design Technology inthe Era of Intelligent Manufacturing [J]. Gao Songlin. Hubei
Agricultural Mechanization,2019
8. Preliminary Study on Design Problems of stepper Motor
Driver[J].
Tuo
Chaoyong.
Hunan
Agricultural
Machinery,2010
9. Development and Research of Small Capacity Stepper
Motor Intelligent Driver [J]. Jin Hailong. Automation andInstrumentation,2000

Authors Introduction
Mr. Hongpi Zhao
He is currently pursuing his undergraduate
degree at the School of Electronic Information
and Automation, Tianjin University of Scienceand Technology.

Mr. Jianfeng Qin
He is an Associate Professor in Tianjin
University of Science &Technology. He
received a doctor’s degree in control theory
and control engineering from the Institute ofAutomation, Chinese Academy of Science,in 2006.His research field is intelligent robot
and intelligent control.
Dr. Yizhun Pen
He is an Associate Professor in Tianjin
University of Science &Technology. He
received a doctor’s degree in control theory
and control engineering from the Institute ofAutomation, Chinese Academy of Science in2006. His research field is intelligent robot
and intelligent control.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
383

Design and Application of AI-based Brush Calligraphy and Painting Robot
Haibo Li, Yizhe Sun, Shuxin Wang

College of Artificial Intelligence, Tianjin University of Science and Technology, Tianjin, China
College of Electronic Information and Automation, Tianjin University of Science and Technology, Tianjin, China
Yizhun Peng*

College of Electronic Information and Automation, Tianjin University of Science and Technology, Tianjin, China
Email: pengyizhun@tust.edu.cn

Abstract
With the rapid development of digital technology and artificial intelligence, the innovation and exploration of
traditional painting forms in the field of digital art have become increasingly captivating. The focus of this researchis to create a brush calligraphy and painting robot system based on PLT files. It digitizes user creative instructions toautonomously generate brush calligraphy and landscape paintings. This technology not only advances the deep
integration of digital art and traditional culture but also opens new perspectives and vast possibilities for artistic
creation.
Keywords: Artificial intelligence, Calligraphy and painting robot, Brush word, Landscape painting, Digital art

Introduction
With the rapid progress of digital technology [1] andartificial intelligence, traditional art forms [2] areexperiencing a profound digital change, and brush painting
and calligraphy as a precious heritage of traditional Chinese
culture is no exception. This study aims to further explore
the influence of digital technology on traditional painting,and explore the widespread application of this technology
in the field of art creation. The core concept of the brushcalligraphy and painting robot system is planted on the basisof the PLT file. It receives the user's creative instructions ina digital manner and aims to independently draws brushcharacters and landscape painting. The rise of artificial
intelligence provides strong technical support for the designof brush calligraphy and painting robots. The algorithm ofmachine learning such as deep learning enables the machine
system to imitate the unique brush strokes of the brush, andeven understand the emotion and artistic conception
expressed by the artist in landscape painting. This
technology not only injects new vitality into traditional art,
but also expands the boundaries of art creation, andprovides users with more diversified and innovative
creative methods. In this study, we will introduce thetechnical principles, design frameworks and application
scenarios of the brush calligraphy and painting robot system.
Through in -depth excavation of these aspects, we expect toprovide in -depth insights for the integration of digital art
and traditional culture, and contribute to the innovation andinheritance of traditional culture of digital art.
1. The main text

The design scheme uses a robotic arm, a set of linear guidesystems, as well as painting tools such as pen, ink, paper,and inkstone. The robotic arm is mainly responsible forgrasping the brush and realizing dipping ink, writing andpainting [3]. The robotic arm is cleverly placed on the linear
guide to expand its movement range and improve accuracy
and efficiency [4]. For the overall design effect, see Fig. 1.

Fig.1 overall arrangement
2. Core structure of robotics
The robotic arm chooses the desktop four-axis robotic armof Yuejiang Technology. As shown in Fig. 2, the roboticarm works radius of 320mm, the load is 0.5kg, the repeatedpositioning accuracy is 0.2mm, and it has high accuracy inthe vertical direction. And in the horizontal direction, it canbe moved quickly, with a light weight and a small volume.
The length and width of the base are 158mm. You cancomplete the writing or painting work of this robot. The first©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
384

Haibo Li, Yizhe Sun, Shuxin Wang, Yizhun Peng

freedom of the robotic arm is the rotation freedom
composed of a shaft base. The second and third degree of
freedom is the rotation freedom composed of large arms andsmall arms. Four freedom is located at the end of the roboticarm, which is rotating freedom [5]. You can install a fixture
or suction cup at this location. The robot chooses a fixture
to pinch the brush.

First, convolutional neural networks can be used togenerate training of PLT files. The robot can draw a specificpattern or imitate a certain style. It can use convolutional
neural networks to train images, and then use thecharacteristics to generate the corresponding PLT files.
CNN can help identify the characteristics of strokes, lines,
and shapes in the image, thereby generating thecorresponding drawing instructions. Fig. 3-1 is the"chrysanthemum picture" entered by the user, Fig. 3-2 is the
picture form in the PLT file generated after program
processing.

Fig. 3-1 Original Picture

Fig. 3-2 Processed pictures

Secondly, convolutional neural networks can be used forimage recognition and PLT file generation. When the robot
receives the image input provided by the user, theconvolutional neural network can be used for image
recognition and identify the content and characteristics of
the image [7]. Then, the corresponding PLT file isgenerated according to the recognition result, so as to guidethe robot to complete the drawing task. Fig. 4 is thecalligraphy of the robot

Fig. 2 Manipulator
The core parameters are shown in Table 1.
Table 1. The core parameters of the roboticarm.
Weight3.4 kgBase Dimension
158 mm x 158 mm(Footprint)
Aluminum Alloy 6061,
Materials
ABS Engineering Plastic
Controller
Dobot Integrated Controller
Robot Mounting
DesktopNumber of Axes
4
Packing Size
421 mm x 334 mm x 352 mm(L X w X H)
Payload
500g
Max.Reach
320mm
Position
0.2mm
Repeatability
Communication
USB\WIFI*\Bluetooth*Power Supply
100-240 V,50/60 Hz
Power In
12V/6.5 A DC

Fig. 4 the calligraphy
3.1. The use of PLT files
The PLT file provides a digital approach for describing
painting instructions. By using PLT files, you can digitize
traditional paintings and landscape painting such as
landscape painting to make it meet the needs of modern
digital art.
The PLT format is a vector graphic format, also known as
HPGL (Hewlett Packard Graphics Language) format. It is adrawing language developed by HP, which was originallyused to control the output of the drawer. This format is now
widely used in fields such as drawing, engineering andmanufacturing.

3. The innovation point
The convolutional neural network (CNN) [6] is usuallyused in tasks such as image recognition and feature
extraction, and the PLT file is a format for describing thegraphic drawing instruction sequence. In the design of thebrush calligraphy and painting robot, the two can be usedwith each other.

The PLT file is a text file that can use any text editor toopen the viewing and editing. It contains drawing
commands and coordinates, which can be drawn throughthese commands.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
385

A Design and Applications

3.2. Convolutional neural network

5. Future outlook

Convolutional neural network (CNN) is a feeding neuralnetwork structure that is designed to process large image
data and performs well in the fields of image and voicerecognition. The network structure consists of one or more
convolutional layers, full connection layers, pooling layers,
and linear rectifier layers. It has the ability to capture thetwo -dimensional structure of the input data, especially inthe field of image processing.The components ofconvolutional neural networks include:

The design and application of brush calligraphy andpainting robots have opened a new era of deep integrationof digital art and traditional culture. In the future, thedevelopment of brush calligraphy and painting robots will
inject new vitality into the field of digital art, while
promoting traditional culture in the digital age. The
continuous evolution of this technology will bring more
possibilities to artists, educators, and cultural and creative
industries, and bring a more colorful future to our art world.

(1) Convoiced layer: The convolutional layer can
generate a set of parallel feature maps, and performs
operations by sliding the convolution nuclear on the input
image. The convolution nucleus and the input image
perform the computing of the component of the element andthe computing of the element. This process is called the rod.
The size of the convolution nucleus is small. Enter theimage through overlapping or parallel effects to share thesame weight and bias item to effectively extract image
features.

Acknowledgements
This work was strongly supported by the Innovation
Electronic Laboratory of Tianjin University of Science andTechnology. The author is particularly grateful for the helpof the laboratory.
References
1.

(2) Linear rectifier: Linear rectifier uses linear rectifier(RELU) as an incentive function to enhance the non -linear
characteristics of the entire neural network withoutchanging the output of the convolution layer.

2.

3.

(3) Pondalization layer: Pondization is a non -linear formof downgrade, of which the largest pooling is a common
form. This layer divides the input image into a rectangulararea, outputs the maximum value of each sub -area toachieve the purpose of reducing the data dimension.

4.

5.

(4) Complete connection layer: After the convolution andpooling layer, the neural network is high -level reasoning
through the complete connection layer. The neurons in thecomplete connection layer are connected to all theactivation in the previous layer, and activation is calculated
through imitation transformation, including the addition of
matrix multiplication and bias items.

6.

7.

4. Conclusion
This research focuses on the design and development of abrush calligraphy and painting robot system based on PLT
files. By digitizing user creative instructions, the systemautonomously generates brush calligraphy and landscapepaintings. The rise of artificial intelligence andconvolutional neural networks provides robust support for
the robot's design, enabling it to mimic unique brush strokes
and comprehend the emotions expressed by artists inlandscape paintings. This technology not only deepens theintegration of digital art and traditional culture but also
explores new avenues and vast possibilities for artistic
creation.

Li Pengyue. The study of the four -freedom robotic arm
drawing vector character and graphics [D]. Taiyuan
University of Technology, 2018.
Zhang Yu. The application of traditional Chinese visualelements in digital media art design [J]. Toys world, 2023,(06): 82-84.
Wang Miao. Innovation research in digital media art in theera of artificial intelligence [J]. Toys World, 2023, (06): 8-10.
Ding Hao. Study on image recognition and processing
algorithm based on neural networks [C] // Tianjin ElectronicSociety. Conference paper set. Tianjin City Investment
Group Asset Management Co., Ltd.;, 2023: 3.Doi:
10.26914/C.CNKIHY.2023.022845
Feng Kaili. Research on image dehuminating methods basedon convolutional neural networks [D]. Shandong
NormalUniversity,2023.DOI:10.27280/d.cnki.gsdsu.2023.0
00567
Lin Qianqi, Wang Yibo, Zhan Qiucheng, etc. The fourfreedom mechanical arm sports algorithm and steering gearslowdown design [J]. Mechanical engineer, 2023, (11): 6466.
Lu Ran. Six -axis calligraphy and painting robot trajectory
optimization control method Study [D]. Taiyuan Universityof
ScienceandTechnology,
2021.DOI:
10.27721/d.cnki.Gyzjc.2020.000297

Authors Introduction
Mr. Haibo Li
He is currently pursuing his undergraduate
degree at College of Artificial Intelligence,
Tianjin University of Science and Technology.
His research field is to Reinforcement learningand intelligent robots.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
386

Powered by TCPDF (www.tcpdf.org)

Haibo Li, Yizhe Sun, Shuxin Wang, Yizhun Peng

Mr. Yizhe Sun
He is currently pursuing his undergraduate
degree at College of Artificial Intelligence,
Tianjin University of Science and Technology.
His research field is to ROS and intelligent
robots.

Mr. Shuxin Wang
He is currently pursuing his undergraduate
degree at College of Artificial Intelligence,
Tianjin University of Science and Technology.
His research interests are image processing andneural networks.

Dr. Yizhun Pen
He is an Associate Professor in Tianjin Universityof Science &Technology. He received a doctor’s
degree in control theory and control engineeringfrom the Institute of Automation, Chinese
Academy of Science in 2006. His research field isintelligent robot and intelligent control.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
387

Intelligent Wheelchair System：Non-contact Heart Rate and Body Temperature Measurement
Dongpo Ma

Tianjin University of Science and Technology, 1038 Dagu South Road, Hexi District, Tianjin, 300222, China
Junsheng Zhang

Tianjin University of Science and Technology, 1038 Dagu South Road, Hexi District, Tianjin, 300222, China
Yizhun Peng*

Tianjin University of Science and Technology, 1038 Dagu South Road, Hexi District, Tianjin, 300222, China
Email: pengyizhun@126.com

Abstract
With the continuous development of society and human civilization, people, especially the disabled and the elderly, needmore and more to use modern high technology to improve their lives and improve the comfort of life. Therefore, this paper
introduces a single-chip microcomputer-based non-contact body temperature and facial recognition-based heart ratemeasurement method and applies it to the intelligent wheelchair system. Monitor the vital signs of the elderly, and give
early warnings in time when abnormal values occur, so that family members or caregivers can rescue them in time.
Keywords: Smart wheelchairs，Non-contact，heart rate，Body temperature measurement
Based on the stm32 microcontroller, the B-1 infrared
temperature measurement module is used to complete thenon-contact body temperature measurement. The physicaldiagram is shown in Fig. 1.

1. Introduction
Population aging has become a global phenomenon.
China's elderly population is huge, the aging rate is fast,
and the problem of elderly care and daily care for theelderly is becoming increasingly severe. Wheelchairs areindispensable means of daily life for the elderly and people
with lower limbs, and a cost-effective, comprehensive andreliable intelligent wheelchair can not only improve thequality of life of the disabled and the elderly, but also
reduce the burden on caregivers. Smart wheelchairs.
Research plays an important role in the development ofelderly care projects in China [1]. Since the developmentof the first intelligent wheelchair in the United Kingdom inthe 80s of the 20th century, the United States, Japan,
France and other countries have also carried out researchon intelligent wheelchairs [2], [3], [4], [5], [6]. The
research on intelligent wheelchairs in China started late,but in recent years, some research results have beenachieved. The research of intelligent wheelchairs based on
visual tracking [7], autonomous driving [8],electromyography [9], head [10], multi-sensor fusion [11],and voice [12] has also been progressing, but thesewheelchairs are still in the laboratory stage, which areexpensive, and their safety and real-time performancecannot meet the demand. Therefore, this study takes theabove problems as the starting point, and carries out thesystem improvement design on the basis of the existing
wheelchair machinery to meet the current needs.

Fig. 1 Photographs of the B-1
The module has a built-in 24-bit high-precision AD
conversion chip, and its working principle is that when theobject is higher than absolute zero (-273℃), it will radiate
infrared energy outward, and the infrared sensor receives
the infrared light reflected back by the object, and obtains
the induced temperature data through the high-precision
program algorithm in the CPU.
Table 1 Temperature measurement results
Temperature
Measurethetemperature/℃

1

2

3

4

36.7

36.2

37.3

35.9

36.8

36.0

37.1

36.2

True

2. Infrared body temperature measurement

temperature/℃©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
388

Dongpo Ma, Junsheng Zhang, Yizhun Pengdotted line and around the black dashed line. According to
the distribution of the Fig. 3 scatter plot, the distribution of
the sample points of the predicted value and the true value
is basically around the straight line y=x, which indicatesthat the consistency between the predicted value and thetrue value is good. Fig. 4 shows that the experimental
consistency is good

3. Non-contact heart rate measurement
3.1 Technical principle
There are two commonly used methods to measureheart rate in clinical practice: electrocardiogram (ECG)
and optical volume scanner (PPG) PPG is a method of
detecting blood volume pulses (BVP) in human blood
tissues using photoelectric means, such as fingertip pulse
oximeters, exercise bracelets, etc
Remote photoplethysmography (RPPG): Diffuse andspecular reflections of light after passing through the skin,both of which can be captured by the camera. According toLambert-Beer's law, the strength of a substance's
absorption of a monochromatic light is related to theconcentration of the absorbent substance and the thicknessof its liquid layer. Since the periodic beating of the heartcauses the same periodic change in the concentration ofblood in the blood vessels and capillaries, it leads to aperiodic change in the concentration of light absorbed by
the blood, so that the intensity of the diffuse reflected lightreceived also changes periodically, that is, the pulse wave
information.
When the heart contracts, the concentration of bloodincreases, the human skin absorbs more light and reflects
less light; When the heart stretches, the concentration ofblood decreases, the human skin absorbs less light, and thereflected light increases [13]. The schematic diagram isshown in Fig. 2.

Fig. 3 Scatter plot

Fig. 4 Bland-Altman
4 Experimental analysis

Fig. 2 Schematic diagram of the principle
3.2 Data set

Fig. 5 Comparison chart
Fig. 5 shows that the true value curve is in good
agreement with the predicted value curve. Table 1 shows

This experiment uses a UBFC dataset of a total of 10
RGB videos, recorded in an indoor environment, including
light changes and head movements, with a pixel resolutionof 640x480. At the same time, the fingertip oximeter wasused to synchronously record the real BVP signal and heart
rate value as label data.

that the error between the measured and true values ofbody temperature is within acceptable limits.
5 Conclusion
In addition to the non-contact heart rate and bodytemperature measurements presented in this article, a smart
wheelchair system is a highly intelligent device that
integrates a variety of sensors, controllers, and actuators to
help improve the quality of life for the elderly and people
with disabilities with reduced mobility. At the same time,

3.3 Test results
According to the distribution of the Bland-Altman
diagram, the blue scatters are basically within the red©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
389

Powered by TCPDF (www.tcpdf.org)

Intelligent Wheelchair System

the human-computer interaction module enables users toeasily communicate with the wheelchair. In the future, withthe continuous development of technology, smart
wheelchairs will be able to provide convenient and safe
travel solutions for more people in need.

Authors Introduction
Dongpo Ma
He received his bachelor's degree in physics
and optoelectronic engineering from Taiyuan
University of Technology in 2019 and is now agraduate student in Tianjin University of
Science and Technology

References
1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

Zhen Zhang, Xinmei Liu, Junling Yin ,etc. Design of
ARM-based intelligent wheelchair safety monitoring
system [J]. Foreign electronic measurement technology,
2021,40(6):138-143.
Lamia B, Noureddine E. Implementation of a biometricinterface in voice controlled wheelchairs[J]. Sound andVibration, 2020, 54(1):1-15.
Pawuś D, Paszkiel S. BCI wheelchair control using expert
system classifying EEG signals based on power spectrum
estimation and nervous tics detection[J]. Applied Sciences,2022,12(20):1-22.
Rosero P, Peluffo D, Batista V, et al. Intelligent system for
identification of wheelchair user’s posture using machinelearning techniques [J]. IEEE Sensors Journal, 2019, 19(5):1-8.
Abolfazl M. Human-robot interaction in rehabilitation andassistance: a review[J]. Current robotics reports,2020, 1(3) :131-144.
Timofei I V, Eugene V C, Gleb A U. The control system
based on extended BCI for a robotic wheelchair[J].
Procedia Computer Science,2018,123: 522-527.
Yan Li, Mingda Tang, Yingling Zhou, etc. Computer
vision-based wheelchair following system design [J].
Computer Engineering and Applications,2021,57(10):163172.
Chongyue Bai, Jianjun Wang, Xiaoxiao Cheng, etc. Fusion
laser SLAM realizes the optimization of spatial positioningof unmanned wheelchairs [J]. Advances in Lasers andOptoelectronics, 2022, 59(2):485-493.
Weijian Yang, Qingshan She, Qizhong Zhang, etc.
Intelligent wheelchair control method for shoulder andneck electromyography signal recognition [J]. Journal of
Shanghai Jiao Tong University,2016,50(6):949- 956, 962.
Cong Wu, Kun Zhang, Licai Yang. Design of attitudemonitoring device for the elderly based on inertial sensor[J]. Biomedical engineering research,2018,37(2):210-214.
Zhixin Han, Xiuwu Sui. Research on intelligent control
technology of myoelectric wheelchair based on multisource information fusion [J]. Modern Manufacturing
Engineering,2020(9):136-144.
Jie Hu, Hongliu Yu, Ping Shi, etc. A kind of multifunctional
intelligent wheelchair control system design [J]. Chinese
Journal of Rehabilitation Medicine,2016,31(11):12461249.
Weixiang Gao, Qing Li, Yuzhu Yang. Non-contact heart
rate detection based on PTZ camera [J]. Computer
applications and software,2023,40(09):43-50+98.
Weihua O, Longbao C, Jiahao X, et al.Non-contact heart
rate measurement from face video based on group sparse
representation[J].Digital Signal Processing,2024,144

Mr. Junsheng Zhang
In 2021, he will be admitted to the School ofElectronic Information and Automation, Tianjin
University of Science and Technology, China,
majoring in electronic information. He iscurrently a master's student at Tianjin
University of Science and Technology, China
Dr. Yizhun Pen
He is an Associate Professor in Tianjin
University of Science &Technology. He
received a doctor’s degree in control theory andcontrol engineering from the Institute ofAutomation, Chinese Academy of Science in2006. His research field is intelligent robot andintelligent control.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
390

Recognition and Localization Method for Automotive Axle Holes in Assembly Robots
Junsheng Zhang

Tianjin University of Science and Technology, 1038 Dagu South Road, Hexi District, Tianjin, 300222, China
Dongpo Ma

Tianjin University of Science and Technology, 1038 Dagu South Road, Hexi District, Tianjin, 300222, China
Yizhun Peng*

Tianjin University of Science and Technology, 1038 Dagu South Road, Hexi District, Tianjin, 300222, China
Email: pengyizhun@126.com
Abstract
During the production of automotive axle holes, the roundness error at the pipe opening leads to low detection efficiency
due to manual measurements, rendering real-time inspection unfeasible. This paper proposes a method for detecting andsorting the roundness of automotive axle holes based on visual inspection. Ensuring accuracy in grasping, it establishes the
kinematic model of the robot Aubo_I5. Targeting automotive axle holes for grasping, it employs adaptive thresholdsegmentation to highlight the section features of the axle hole. The Canny algorithm is then used to extract edge information,and finally, the least squares method is utilized to detect roundness errors for sorting the axle holes based on this error.
Keywords: Axle Hole, Assembly, Vision Deep, Learning
environment are critical issues affecting target recognition
1. Introduction

and localization.
2. Kinematic Modeling

The assembly of mechanical components constitutes thesubsequent phase in the entire manufacturing process.

Establish the joint coordinate system of the Aubo_I5

Manual assembly is characterized by low efficiency, androbot as shown in Fig. 1, and the structural model andthe assembly speed is affected by factors such as worker

dimensions of each joint as illustrated in Fig. 2. To morelabor intensity. Traditional assembly robots have mostly

accurately define the sorting model coordinate system, thebeen controlled through teaching or offline programming

gripper coordinate system Gxy is established at the end joint.

[1], [2]. Consequently, these robots strictly follow
predefined assembly procedures, lacking the ability topromptly respond to changes in the surrounding
environment or objectives, thus exhibiting poor robustness.
With

the

advancements

incomputerandvisualtechnologies, automation machinery on production lines
gradually possesses visual perception capabilities. This
enables real-time dynamic recognition and localization oftargets, facilitating visual-guided adjustments of robot
poses and grasping [3].
The emergence of deep learning has improved issues
regarding detection accuracy in visual recognition.
However, due to the complexity of the assembly process,Fig. 1 Structure of the Aubo_I5 Robot

uncertainties such as lighting conditions, backgrounds,
workpiece sizes, shapes, and other factors in the working©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
391

Junsheng Zhang, Dongpo Ma, Yizhun Peng

Gxpose relationship between the tool coordinate system of a

Z5

X6

robot relative to the polar coordinate system, given thed6

Z6

known motion angles of each joint. This is accomplished

Gz

d5

X5

d7

using

the

homogeneoustransformation

matrix𝑖−1
𝑖𝑇 between adjacent link pairs, as shown in Equation (1):

Z4

X4

𝐶𝜃𝑖
a4

𝑆𝜃𝑖 𝐶𝛼𝑖−1
𝑖−1
𝑖𝑇 = [
𝑆𝜃𝑖 𝑆𝛼𝑖−1

X3

0

Z3

−𝑆𝜃𝑖
𝐶𝜃𝑖 𝑐𝛼𝑖−1
𝐶𝜃𝑖 𝑆𝛼𝑖−100−𝑆𝛼𝑖−1
𝐶𝛼𝑖−1
0

𝑎𝑖−1
−𝑆𝛼𝑖−1 𝑑𝑖]𝐶𝛼𝑖−1 𝑑𝑖1

(1)

In the equation：𝐶𝜃𝑖 = 𝑐𝑜𝑠 𝜃𝑖 , 𝑆𝜃𝑖 = 𝑠𝑖𝑛 𝜃𝑖d4

According to the D-H parameter table of Aubo_I5,

a3calculate the homogeneous transformation matrices foreach of the seven joints 01𝑇、 12𝑇、 23𝑇、 34𝑇、 45𝑇、 56𝑇、 67𝑇
As shown below：

Z1

X2

d2

Z2

𝐶1 −𝑆1 0 0
𝑆1 𝐶1 0 00
] (2)1𝑇 = [
0
0
1 𝑑1
0
0
0 1
𝐶2 −𝑆2 0
0
0
0
−1
0
1
] (3)
2𝑇 = [
𝑆2 𝐶2
0 𝑑2
0
0
0
1
𝐶3 −𝑆3 0 𝑎3
−𝑆3 −𝐶3 0
02] (4)3𝑇 = [
0
0
−1 0
0
0
0
1
𝐶4 −𝑆4 0 𝑎4
−𝑆4 −𝐶4 0
03] (5)4𝑇 = [
0
0
−1 𝑑4
0
0
0
1
𝐶5 −𝑆5 0 0
0
0
1 0
4
] (6)
5𝑇 = [−𝑆5 −𝐶5 0 𝑑5
0
0
0 1

d1

X1 Z0

Obase
X0

Fig. 2 Robot Joint Coordinate System
Fig 2: Establishing the Joint Coordinate System of
Aubo_I5
Based on the joint coordinate system, the corresponding DH parameter table is shown in Table 1 [4], where each linkparameter is defined as follows:
αi-1 represents the distance moved along Xi-1 axis from
Zi-1 to Z_i;
ai-1 represents the angle rotated around Xi-1 axis from Zi1 to Z_i;
d_i represents the distance moved along Z_i axis from Xi1 to X_i;

𝐶6 −𝑆6 0
0
0
0
−1
0
5
]
6𝑇 = [
𝑆6 𝐶6
0 𝑑6
0
0
0
1

θ_i represents the angle rotated around Z_i axis from Xi1 to X_i.
Table 1 D-H Parameter Table for AUBO_I5 Robot(7)i𝛼𝑖−1 /𝑑𝑒𝑔

𝑎𝑖−1 /𝑚𝑚

𝑑𝑖 /𝑚𝑚𝜃𝑖 /𝑑𝑒𝑔

Combining the above formulas, the robot's forward

1

0

0

𝑑1 = 98.5

𝜃1

kinematics equation can be obtained by simultaneous.

0𝑑2 = 140.5

𝜃2

∘

2

90

3

180∘

𝑎3 = 408

0

𝜃3

4

180∘

𝑎4 = 376

𝑑4 = −19

𝜃4

5

−90∘

0

𝑑5 = 102.5

𝜃5

6

90∘

0

𝑑6 = 94

𝜃6

𝑛𝑥
𝑛𝑦
0
0 1 2 3 4 5
6𝑇 = 1𝑇 2𝑇 3𝑇 4𝑇 5𝑇 6𝑇 = [ 𝑛
𝑧
0

𝑜𝑥
𝑜𝑦
𝑜𝑧
0

𝑎𝑥
𝑎𝑦
𝑎𝑧
0

𝑝𝑥
𝑝𝑦
]
𝑝𝑧
1

(8)

2.2 Robot Inverse Kinematics Analysis
Inverse kinematics involves calculating the joint angles
θ_i of a robot given the pose of the tool coordinate system.

2. 1 Robot Forward Kinematics Analysis

Considering the robot's link coordinate systems, the axesof joints 4, 5, and 6 intersect at one point, satisfying the

Forward kinematics analysis involves determining the

Pieper criterion, indicating the existence of a closed-form©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
392

Recognition and Localization Method2 ） Threshold Segmentation: Highlighting the target
solution. Assuming the end posein Equation (8) is

contour features more effectively.

known, solving is performed using an analytical method,(1) Image Filteringand the inverse solutions for each joint are presented in

Image filtering refers to the suppression of noise in an

Table 2.

image while preserving detailed features, and its

Table 2: Inverse solution formula of joint

effectiveness𝑝𝑦𝜃1 = 𝑎𝑟𝑐𝑡𝑎𝑛− 𝑎𝑟𝑐𝑡𝑎𝑛𝑝𝑥±√1 − 𝑢1 2
𝑢1

directly

impacts

subsequent

image

processing. Common filtering algorithms include mean
filtering, median filtering, Gaussian filtering, and others.

±√1 − 𝑢2 21）Mean filtering is a typical linear filtering algorithm𝑢2
𝑜𝑥 𝑆1 + 𝑜𝑦 𝐶1
𝜃6 = 𝑎𝑟𝑐𝑡𝑎𝑛
𝑛𝑦 𝐶1 + 𝑛𝑥 𝑆1that replaces the grayscale value of a particular pixel with

𝜃5 = 𝑎𝑟𝑐𝑡𝑎𝑛

𝜃3 = 𝑎𝑟𝑐𝑡𝑎𝑛
𝜃2 = 𝑎𝑟𝑐𝑡𝑎𝑛

mathematical expression is as follows:1g ( x, y) = (i, j )Sxy f (i, j) (9)mnWhere represents the center point at, and is the size of±√1 − 𝑔2𝑔𝑘±√1 − 𝑘 2

𝜃4 = 𝑎𝑟𝑐𝑡𝑎𝑛

the average pixel value in its surrounding area. Its− 𝑎𝑟𝑐𝑡𝑎𝑛𝑢4
𝑣4

the filtering window. By controlling the window size, thefiltering effect can be adjusted. A larger value for results𝑣5− 𝜃3 + 𝜃2
𝑢5

in a more pronounced filtering effect.

Among which:𝑢1 =2）The median filtering algorithm is a common non-linear

𝑑4 − 𝑑5

filter that calculates the median value of points within aneighborhood range in an image for a particular pixel,
replacing the original pixel value. The mathematicalexpression is as follows:g ( x, y) = median(i , j )Sxy { f (i, j)}(10)√𝑝𝑥 2 + 𝑝𝑦 2
𝑢2 = −𝑎𝑥 𝑆1 − 𝑎𝑦 𝐶1
𝑢3 = 𝐶1 (𝑝𝑥 + 𝑑6 𝑜𝑥 − 𝑑5 𝐶6 𝑎𝑥 − 𝑆6 𝑛𝑥 ) +𝑆1 (𝑑5 𝐶6 𝑎𝑦 − 𝑝𝑦 − 𝑑6 𝑜𝑦 − 𝑆6 𝑛𝑦 )Median filtering controls the size of to adjust the𝑢3 2 + 𝑣3 2 − 𝑎3 2 − 𝑎4 2𝑔=
2𝑎3 𝑎4𝑢4 = 𝑎3 + 𝑎4 𝐶3 − 𝑎4 𝑆3
𝑣4 = 𝑎3 + 𝑎4 𝑆3 + 𝑎4 𝐶3
𝑢4 + 𝑣4𝑘=√𝑢4 2 + 𝑣4 2

filtering effect. Compared to mean filtering, medianfiltering not only removes salt-and-pepper noise but also
better preserves edge information.
3) Gaussian filtering is also a type of linear smoothingfilter. It involves the weighted averaging process of theentire image and is primarily used to remove unevenlighting or sensor noise.
To compare the effects of the three filtering methods,

𝑢5 = 𝐶1 (𝑜𝑥 𝑆5 + 𝑛𝑥 𝐶5 𝐶6 + 𝑎𝑥 𝑆6 ) −𝑆1 (𝑜𝑦 𝑆5 + 𝑛𝑦 𝐶5 𝐶6 + 𝑎𝑦 𝑆6 )experiments were conducted on the images, and the results

𝑣5 = 𝑜𝑧 𝑆5 + 𝐶5(𝑛𝑧 𝐶6 + 𝑎𝑧 𝑆6 )are shown in Fig. 3.
From the results, it's apparent that after applying median

2.3 Mage Preprocessing

filtering, not only has the original noise in the image andthe interference from oil stains on the workpiece surface

Before conducting roundness measurement, in order toreduced, but it has also effectively preserved the edge

eliminate the effects of metal debris, noise from thefeatures. Hence, median filtering is employed to filter thesurrounding environment, and surface wear on theimages of the pipe end face.

workpiece, the captured images need preprocessing. This

(2) Threshold Segmentation

involves removing noise and the influence of lighting

Image threshold segmentation is a crucial step inwhile retaining the edge features of the workpiece [5].

segmenting the target region. Determining the threshold is

Image preprocessing mainly includes:

a key aspect of image segmentation. If the threshold is set

1）Image Denoising: Using specific filtering algorithms

too high, target points may be considered as backgroundto reduce noise in the image while preserving edge features.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
393

Junsheng Zhang, Dongpo Ma, Yizhun Pengand ignored, while setting it too low might lead tobackground points being identified as target features.
Commonthresholdsegmentation

methods

includeadaptive thresholding algorithms, Otsu's method, iterative
threshold segmentation, and others. Due to varying surface
qualities of the hole, there's no fixed threshold selection.
Therefore, this paper conducts comparative experiments
among several commonly used segmentation algorithms tochoose the appropriate threshold segmentation algorithm
suitable for the cooling pipe imaging environment [6].

Fig. 4 Comparison of threshold segmentation
2.4 Edge Detection
After threshold segmentation, the next step is edge
detection for the borehole. To ensure better contour
detection, reducing noise interference is essential tomaintain the integrity and clarity of edge information.
Commonly used edge detection algorithms include the
Canny algorithm, Sobel algorithm, Laplacian algorithm,
etc. Compared to other edge detection algorithms, theCanny algorithm is less susceptible to noise interference.
Fig. 3 Comparison of filtering algs
After applying median filtering to the captured borehole[7].
The Canny algorithm first applies a Gaussian filter toregions, comparative experiments of three segmentationsmooth the image, reducing its sensitivity to noise. Then,algorithms were conducted, and the results are shown in

it calculates the gradients in the X and Y directions using

Fig. 4.

the Sobel operator, performs non-maximum suppression,

From the segmentation results above, it's evident that theand finally uses dual thresholds to determine edges. The

adaptive algorithm has better segmentation results,

Canny edge detection effect for the port block is shown inpreserving the edge features of the pipe end more

Fig. 5.

completely. Therefore, this paper adopts the adaptive
segmentation algorithm.
From the segmentation results above, it's evident that theadaptive algorithm has better segmentation results,
preserving the edge features of the pipe end more
completely. Therefore, this paper adopts the adaptive
segmentation algorithm.

Fig. 5 Canny detection algs

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
394

Recognition and Localization Method

The YoloV4 algorithm can be divided into four parts:

2.5 Borehole Circular Detection

input end, backbone feature extraction network, enhanced
After edge extraction, detecting circles is a crucial partfeature extraction network and output end. The input endof roundness measurement. The most commonly used

includes image preprocessing, converting the size of themethods for circular detection are the Least Squares Circle

input image to the specified input size, and normalizing the

Fitting and Hough Transform [8]. To measure circles

image information. Secondly, it is the backbone featureaccurately, it's essential to filter out the circular contours

extraction network. CSPDarknet53 formed after YoloV4 is

for measurement. As shown in the image, after pre-added to CSPNet serves as the backbone feature extraction

processing the image, the contour of the borehole isnetwork. YoloV4 's enhanced feature extraction network

extracted using Canny edge detection. The contour points

consists of a PANet structure, which is used to improve theof the borehole are then sorted for subsequent roundness

diversity of feature expressions and strengthen the fusion

measurement.

of feature information; the last is the output end, which is

Using the detected circle center as the reference [9], [10],used to perform classification and regression tasks andthe roundness error is represented by the difference

output the final prediction results.

between the maximum contour circle radius Rmax and the

YoloV4 has three outputs after passing through theminimum contour circle radius Rmin, expressed as Rerr=

backbone feature extraction network, namely L3, L4 andRmax－ Rmin.

L5. Among them, L3 and L4 will be output to the enhanced

erro = ( xmax − xc ) + ( ymax − yc ) −2.

feature extraction network for corresponding feature

2

(11)

fusion after one 1 × 1 convolution; L5 will be input to the( xmin − xc ) + ( ymin − yc )among which： ( xmax , ymax ),( xmin , ymin ) The coordinates ofthe farthest and nearest points from the fitted circle center,
respectively, correspond to the contour.
The principle of the least squares fitting circle involves
22

spatial pyramid pooling layer SPP after three convolutions,and then enter the enhanced feature Extract the network forfeature fusion. Finally, the final detection result can beobtained by sending it to the output terminal.

several steps: 1) detecting the actual contour, 2) applying

YoloV4’'s backbone extraction network CSPDarknet53 is

the least squares fitting to find a circle, 3) creating a circle

an improvement based on Darknet53. The original residualwith the least squares circle center and the maximum

module is changed to the CSP structure as shown in Fig. 7.

distance to the contour, and 4) creating a circle with the

The CSP structure changes the original residual module

least squares circle center and the minimum distance to theinto two parts, one part goes through the residual network

contour. The difference between the maximum andand the other part Directly merging with the output of theminimum radii of these circles is used as the basis forresidual network, this method can reduce the among ofevaluating roundness.

parameters and calculations while maintaining a high

The circle fitted using the least squares circle algorithm

accuracy.

for the axle hole contour is depicted in Fig. 6. The
coordinates of the fitted circle's center are (232.87733,

Conv

85.63195), and the fitted circle's radius is R = 63.36658.3
Input

Conv

Conv

Resblock

Conv

Fig. 7 CSP structure block
The PANet network is used to build a feature pyramid,
analyze semantic features from top to bottom, and use thefeature layer results obtained by down sampling and up

Fig. 6 Least squares circle fitting

sampling to reduce the loss of underlying feature
information during the propagation process of the FPN

3. Target recognition network based on YoloV4©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
395

Junsheng Zhang, Dongpo Ma, Yizhun Pengalgorithm by increasing the bottom-up feature fusion path.

are divided, 90% of which are used for training and 10%

problem [69], which improves the richness of the feature

for verification. Train for 100 epochs, and freeze the next

map and the detection accuracy of the algorithm. The

50 epochs. The initial value of the learning rate for the first

feature map after feature fusion is predicted in three

50 epochs is 0.01, and the batch size is 32. The initial value

different sizes by Yolo-head. The prediction results includeof the learning rate for the last 100 epochs is 0.001, and theoffset x, y, detection frame size w, h, and confidence levelbatch size is 16. The Adam optimizer is used during

c of target information. Finally, redundant ones aretraining, and the cosine annealing learning rate decay

eliminated through IOU and NMS. Detect the and NMS.

strategy is adopted.

Detect the frame and complete the target prediction.
4.2 Experimental results and analysis
3.1 Shaft hole type identification
From the global loss function curve in Fig. 9, we can see
As shown in Fig. 8, after the roundness detection of shaftthat the loss curve drops rapidly in the early stages ofholes of different sizes is completed, the YoloV4 algorithm

iteration, indicating that the model is fitting quickly andneeds to be processed to predict the category and position

the model's learning efficiency is high. When epoch is 40,

frame of the shaft hole. Before attitude detection, in orderthe network model gradually becomes stable. After 100

to remove the shaft hole target in the image, The effects ofiterations of training, the loss function of the final modelother complex backgrounds, noise, and light damage on

converged to 1.50. In addition, by observing thethe target surface existing in the sorting environment canconsistency of the loss curves of the training set and thebe used for subsequent shaft hole type identification. We

verification set, it can be seen that the generalization ability

have preprocessed the image above, and then use YoloV4of the model has reached the best state.

to predict the image.

Fig. 9 Loss value curve
Fig. 10 shows the detection effect of the Yolov4 model.
It can be seen that the network model accurately identified

Fig. 8 Shaft hole image after inspection by YoloV4

all buckles without missing detection. It can be seen from4 Experiments and Analysis

the above results that this method has a high accuracy indetecting targets and is practical.

4.1 Dataset and environment configuration
The data set used in the experiment of this article
contains a total of 6 categories, with 600 data for eachcategory. In order to prevent overfitting due to too little
data, this article uses Gaussian noise and random colortransformation for data enhancement. The final data thequantity is 3600 sheets.
The training and testing environment of this article isrunning on a Windows 10 system, NVIDIA GeForce RTX
Fig. 10 Yolov4 detection results

4060TI GPU, and 8G video memory. The 3600 data sets

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
396

Recognition and Localization Method

In this experiment, a plastic bottle was placed next to the

URBAN S，WURSTHORN S，LEITLOFF J，et al．
MultiCol bundle adjustment: a generic method forpose estimation，simultaneous self－calibration andreconstruction for arbitrary multi － camera systems［J］．International Journal of Computer Vision，
2017，121( 2) : 234－252．
4
CRAIG J J． Introduction to Robotics［M］． Yun
Chao, waiting for translation. Beijing: Machinery
Industry Press, 2006.
5
CHILLARÓN M ， VIDAL V ， VERDÚ G ．
Evaluation of image filters for their integration with
LSQR computerized tomography reconstruction
method［J］．PLOS One，2020，ne，2020，15( 3) :e0229113．
6
FU S J，ZHONG W B，LIANG L Z．Optical fiber
diameter non－contact measurement with computer
image processing［J］．Acta Scientiarum Naturalium
Universitatis Sunyatseni，2000，39( 2) : 24－27．
7
LI Y M ， TU J K ， XIANG H Z ， et al ．
Measurement of optical fiber geometric parameterswith Canny operator and binarization filtering［J］．
Optical Technique，2018，44( 5) : 513－518．
8
SHEN X J，DUAN X Y，YUAN W L，et al．
Research on circle detection algorithm based onconnected region labeling algorithm［J］． Computer
Engineering and Applications，2018，54( 21) : 95－
98．
9
LEI X Q，ZHANG Y D，MA W S，et al．Leastsquare fitting and error evaluation of the convex
contour of bearing roller［J］．Optics and Precision
Engineering，2018，26( 8) : 2039－2047．
10 CAI Z，WANG J L，LYU M F，et al．Roundness
error assessment based on improved cuckoo searchalgorithm ［ J ］ ． Modular Machine Tool ＆Automatic Manufacturing Technique，2020( 7) : 40
－44．
3

shaft hole as an interference object through the above steps,
and 50 sorting tests were conducted, in which the positionsof the plastic bottle and the shaft hole were randomly
placed. The robot grabbing effect is shown in Fig. 11. The
robot successfully avoided other interference and grasped
the target shaft hole.

Fig. 11 Robotic arm grabs shaft hole
5 Conclusion
The vision-based automobile shaft hole roundness
detection system proposed in this article improves theapplicability of detection and can be widely used inroundness error detection of various workpiece nozzles. In
the image pre-processing stage, by comparing various
threshold segmentation algorithms and filtering algorithms,
the adaptive threshold segmentation algorithm combinedwith the median filter algorithm was selected, which can
not only remove the noise in the image, but also retain theedge features of the nozzle. The Canny detection algorithm
is used to identify the contour of the nozzle, and the leastsquares method is used to detect the roundness error of thenozzle, and then combined with the YoloV4 algorithm tocomplete type recognition, and finally the robot completes

Authors Introduction

the shaft hole identification and grabbing work. The results
Junsheng Zhang

show that the designed detection system can not only

In 2021, he will be admitted to the SchoolofElectronic
Information
andAutomation, Tianjin University of
Science and Technology, China, majoringin electronic information. He is currently
a master's student at Tianjin University of
Science and Technology, China.

improve the real-time performance and efficiency ofdetection, but also complete the subsequent grabbing work.
References
1

2

CHURSIN Y A，REDKO L A，FEDOROV E M．
Enlargement of measuring zone in laser gauges
without sacrificing measurement accuracy ［ J ］ ．
Measurement，2019，131: 647－653．
LIAN M ． Methods and key technologies ofautomatic ultrasonic thickness measurement of large
thin－wall parts［D］．Dalian: Dalian University ofTechnology，2019．

Mr. Dongpo Ma
H He received his bachelor's degree inphysics and optoelectronic engineeringfrom Taiyuan University of Technology in2019 and is now a graduate student in
Tianjin University of Science andTechnology©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
397

Powered by TCPDF (www.tcpdf.org)

Junsheng Zhang, Dongpo Ma, Yizhun Peng

Dr. Yizhun Peng
He is an Associate Professor in Tianjin
University of Science &Technology. He
received a doctor ’ s degree in control
theory and control engineering from theInstitute of Automation, Chinese Academyof Science, in 2006. His research field isintelligent robot and intelligent control.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
398

Evaluation Standard of Error Recovery Planning Focused on Revival Processfrom Failures in Robotic Manufacturing Plants
Akira Nakamura

Department of Information Systems, Faculty of Engineering, Saitama Institute of Technology
1690 Fusaiji, Fukaya, Saitama 369-0293, Japan
Kensuke Harada

Robotic Manipulation Research Group, Systems Innovation Department
Graduate School of Engineering Science, Osaka University
1-3 Machikaneyama, Toyonaka 560-8531, Japan

Abstract

In recent times, intelligent robots have found applications across diverse fields. In scenarios demanding
both repetitive and non-routine tasks, the likelihood of work errors increases. To address this, we haveproposed methods encompassing both forward and backward recovery. Forward recovery suits minor
modifications, while our focus is on backward recovery for substantial failures. Our study introduces anovel evaluation method to discern the optimal recovery path among various options.
Keywords: error recovery, task stratification, error classification, evaluation standard

The concept of skills, which are motion primitives
comprising a task, is described in Section 2. The
fundamental technique for generating an error-recovery
path is described in Section 3. Multiple possibilities forvarious recovery paths are presented in Section 4.
Various evaluation standards and methods for selecting
the most suitable path using the new evaluation method
are considered in Section 5, and examples are presented
in Section 6.

1. Introduction
At the present day, information technology, particularly
generative AI, is making rapid strides, a trend mirrored in
the progress of robotic intelligence. As a consequence,robots face an increasing array of tasks during execution,
often grappling with challenging assignments prone toerrors and failures. This underscores the pressing need for
effective methods to address these issues [1], [2], [3], [4],[5].

2. Concept of Skill

Over several years, our research [6], [7], [8], [9], [10] has
focused on systematizing error recovery theory, resulting
in a method based on task stratification and error
classification concepts. The primary components of therobot system include sensing, modeling, planning, andexecution sequences (Fig. 1). When an error occurs, theprocess transitions to the recovery phase. This section
involves estimating the error's cause, classifying it, andcorrecting the original system. The refined process thenoperates on an enhanced, reliable system.

This section provides a brief overview of essential
aspects of these skills [11], [12], [13].
2.1. Skill primitives
Motion primitives constituting tasks, termed "skills,"
were derived by analyzing the human behavior. Threecrucial skills, "move-to-touch," "rotate-to-level," and"rotate-to-insert" (Fig. 2), are highlighted. A person's
behavior, including representative and similar skills,
serves as a model for the robot's motion primitives.

The proposed error recovery technology returns to theprocess before the step where the failure occurs and starts
over from there. In practice, not only this type of
backward recovery but also forward recovery, whichmoves forward after failure, is used. This study considers
various paths from failure to recovery execution.

2.2. Stratification of tasks
The use of task hierarchies, as shown in Fig. 3, iseffective for the execution of automated plants. The layer“task(i+1)” occurs one tier above the layer “task(i),” and thelayer “skill primitive” is represented by the lowest layer“task(0).”©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
399

Akira Nakamura, Kensuke Harada

3. Error Recovery

Main partRe-execute using
the system withimproved reliability

Sensing
Modeling
Planning
Execution

In an actual work environment, contrasting to an ideal
case, various factors can lead to errors in the execution ofa robot. This section outlines the error classification
concept and error recovery technique [6], [7], [8], [9], [10].

Failureoccurred!!

3.1. Error classification

Recovery part
Corrected system
Cause estimation anderror classification

Fig. 1 Robot task system with an error recovery function

The errors can be categorized into several groups basedon their possible causes. We considered four errorgroups: execution, planning, modeling, and sensing (Fig.
4).
3.2. Error recovery based on classification
First, if an error occurs, the cause is determined. Next,
appropriate corrections are made to the system based on
the tentative causes. The process returns to the previous
step, and the task is executed again in this step (Fig. 4).
Since a modified process was executed, the same error
was less likely to occur. If the error scale is small, theprocess returns to the previous step in the lowest
hierarchy (Figs. 4 and 5). Conversely, if the error scale islarge, the process returns to the previous step in thehighest-ranking layer of the hierarchy and is executed
again from that step (Fig. 5).

(a) Move-to-touch skill(b) Rotate-to-level skill(c) Rotate-to-insert skill

Fig. 2 Three fundamental skills…

task (i)The error recovery methods we proposed haveprimarily centered on rerunning the process by revertingto the step before an error occurs, constituting a backward
error recovery process. However, alternative recovery
procedures exist, including backward recovery forsignificant errors (failures) and forward recovery forminor errors. Additionally, errors may impact theenvironment, altering the arrangement or shape ofsurrounding objects, necessitating modified recovery
processes. This section introduces several recovery
procedures focusing on the degree of destruction of theenvironment surrounding an object according to [10].

task (1)This method resumes work in the same process from
subtaskp, the point in the middle of the process before thefailure in the original process occurs (Fig. 6(d)).

Consider an indicative task sequence composed of nsubtasks from the start to the goal (Fig. 6(a)). Suppose afailure occurs in the qth subtask during the process (Fig.
6(b)).

This method resumes work from a point in the middleof the original process before a failure occurs (Fig. 6(e)).
Unlike RS-II, a sequence from subtaskt to subtasku, notincluded in the original planning, restores theenvironment without causing problems for subsequent
work.

…

4. Various recovery paths

task (0) (Skill-primitive)

Fig. 3 Hierarchy of tasks

4.3. Recovery sequence III (RS-III): Restart from the
middle of a prior process (with another process)4.1. Recovery sequence I (RS-I): Complete restart

4.4. Recovery sequence IV (RS-IV): Restart from the
middle of a process that was supposed to come up
later (with another process)This method restarts work from the original starting
point using the same process (Fig. 6(c)). If needed, theenvironment is restored to its original state, and if
necessary, the original object or part is replaced with a
new one and executed.

This method restarts work from the middle of theoriginal process (Fig. 6(f)), differing from RS-III byreverting to its original process with subtasks scheduled
after the original process failure. It includes a sequence
from subtaskv through subtaskw, not in the original plan.

4.2. Recovery sequence II (RS-II): Restart from the
middle of a prior process (without another process)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
400

Evaluation Standard of Error Recovery

Change ofplanning parameters
SensingChange ofmodeling parameters

Modeling

Change ofsensing parameters

Class 4

Class 2no errorerrorClass 3error

task 1

Class 1

Planning

subStart task 1

task 2no errorsubtask n GoalClassification of error(a) Original task sequences from start to goal

Classification of error

error

task kClass 1： Execution error
Class 2： Planning error
Class 3： Modeling error
Class 4： Sensing error

no error

Failure Occurrence

subStart task 1

subsub- subtask q-1 task q task q+1

subtask n Goal(b) Failure Occurrence at subtaskq
Failure Occurrence

Fig. 4 Fundamental process flow
with error recovery

Start subtask 1

subsub- subtask q-1 task q task q+1

task (M)sub- Goal
task n(c) Original reproduction from start step
Failure Occurrence

subStart task 1

task (2)task (1)task (0)Modification Modification
Class N1

Task layer

Modificationsubtask p

subsub- subtask q-1 task q task q+1

sub- Goal
task n(d) Original repetition from step along the way (without another process)Class N2Class NM

The error recovery which returns
to the previous step

Fig. 5 The expression of task stratification andthe process flow of the error recovery

Failure Occurrence

subStart task 1

subtask p

subtask t

Table 1 Degree of correlation for each criteria△◎(ii) Time◯◯(iii) Reliability◎△(iv) Safety◎△(v) Finishing△◎(vi) Recovery data◎－(vii) Tool◎－(viii) Operator skill◎－(ix) Work efficiency◎－(x) Environmental impact◎－(xi) Damege◯◯

sub- Goal
task nsubtask u(e) Original repetition from step along the way (with another process)

Manufacturer Consumer／Operator／User(i) Cost

subsub- subtask q-1 task q task q+1

Failure Occurrence

subStart task 1

subtask p

subsub- subtask q-1 task q task q+1
subtask v

subtask s

sub- Goal
task nsubtask w(f) Restart from step in the middle of the original process scheduled to run later
Failure Occurrence

subStart task 1

subtask p

subsub- subtask q-1 task q task q+1

subtask r

subtask s

subtask x

sub- Goal
task nsubtask y(g) Continuation of work, including consideration of any recovery process that may be required later

Fig. 6 Various error recovery sequences

4.5. Recovery sequence V (RS-V): Continuation ofwork including consideration of recovery needed later

forward recovery process because it does notcontinue working after failure has occurred.
However, because recovery sequences aresubtasks scheduled to be executed after
failure of the original process, this study
included the recovery sequence RS- IV in theforward recovery process for convenience.

In this method, work continues as long as possible after
a failure. When further work becomes impossible, theenvironment is corrected, and work progresses (Fig. 6(g)).
This modification involves a sequence from subtaskx
through subtasky, not included in the original planning.

5. Selection of a recovery path

4.6. As mentioned above, this study briefly
categorizes the recovery sequences into five
types.Recovery sequences RS-I through RSIII are backward recovery processes, whereas
recovery sequence RS-V is a forward recovery
process. Recovery sequence RS-IV is not a

As explored in Section 4, a single failure often presents
multiple possible recovery processes. The study
advocates the use of evaluation standards to discern themost suitable process from several candidates. This
research incorporates 11 evaluation standards, expandingon the four to eight standards introduced in [9].

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
401

Akira Nakamura, Kensuke Harada

Work efficiency is considered as an evaluation standard.
Selecting an efficient work plan, that is, a route that is notunreasonable, wasteful, or uneven, is important. This
index is common to (i) cost and (ii) time, but even if it isan efficient route, it may not necessarily be cheap or fast;
therefore, it was set as a separate indicator.

Home

P10 (=P1 )P1
Home
Move-toapproachMove-betweenreference frames

P5

Pre-grasp

Approachdirection

P4

DepartureHand-open
Lift-up

P2

P9
P6

Move-todestination

P7
BringdownheightBringdown(x) Environmental impacts
Leave

Environmental indicators, including noise levels, areconsidered, accounting for overall environmental
problems such as air, ocean, and water pollution.

Lift-up height

Approach
Approachlength

Grasp

P3

P8
Display shelf(xi) Damage
Damage is a measure of the difference between an
object without and with errors. Although it may not bereadily visible externally, the quality of an object couldchange owing to errors in the production process.

Product stock

Fig. 7 Picking and placing task using a gripper

5.1. Various evaluation standards
5.2. Selection of a recovery path using evaluation
standards for each section(i) Cost
Cost is considered the most important evaluation
standard. A recovery process with the minimum practicalcost is selected.

Reference [9] details how to choose the best recovery
path for each evaluation standard. In contrast, this study
proposes specifying indicators for each section of therecovery process and selecting the path with the superior
total indicator. This prevents situations where a pathchosen by a single indicator may be partiallyinappropriate.

(ii) TimeTime is an important evaluation standard. Priority isgiven to paths that require shorter recovery times.
(iii) Reliability

Furthermore, evaluations may vary depending on therecipient of the standard. Table 1 delineates whether theevaluation is from the Manufacturer/Operator side or theConsumer/User side, indicating the degree of
involvement with symbols ◎, ◯, △, and - in descendingorder. For instance, a process with highmanufacturer/operator involvement may be chosen in thefirst half of the recovery stage, while a process with highconsumer/user involvement may be selected in thesecond half. This ensures optimal restoration on the
manufacturer/operator side and maximum satisfaction
with the finished object on the consumer/user side.

Reliability is considered a representative evaluation
standard. The path with a high success rate is prioritized
for accomplishing the recovery task.
(iv) Safety
Safety is considered a representative evaluation
standard. Priority is assigned to paths that are less likely
to harm people.
(v) Finishing
Finishing is considered an important evaluation
standard. It prioritizes paths displaying excellent
completion of the operation on the target object.

6. Example of error recovery in product display(vi) Recovery data

The task involves utilizing a manipulation robot toarrange products on shelves or in the display window ofa convenience store. The objective is to position andposture the products optimally, as a tidy arrangement
significantly boosts customers’ inclination to makepurchases, as opposed to disorganized placements. Fig. 7
displays an image illustrating a pick-and-place task thattransfers a single object from a product stock to a display
space. Furthermore, Fig. 8 showcases the commandsequence constituting a singular pick-and-place task. In
this instance, the task revolves around evenly arranging
12 identical rectangular products in a layout of three rows

Recovery data is considered as the evaluation standard.
Processes with a lot of useful data are prioritized.
(vii) Tool
The tool used for recovery is evaluated based on
workability, with priority given to tools displaying highefficiency.
(viii) Operator skill.
Operator skill during the recovery process is anevaluation standard, favoring processes managed by alarger number of skilled experts.
(ix) Work efficiency.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
402

Evaluation Standard of Error Recovery

Skill1： Move-to-approach
Skill2： Pre-grasp
Skill3： Approach
Skill4： Grasp

Transfer
Manipulation
Transfer
Manipulation

i=1

i=2

i=9

i=10

i=11

i=12

i=11

i=12

i=10

i=11

i=12

i=10

i=11

i=12

Minimum traceable units

Skill5： Lift-up
Skill6： Departure(a) Original task sequences {subtaski}(i=1,2,・・・,12)
Skill7： Move-betweenreference frames

i=1

Transfer

i=2

i=9

i=10

Skill8： Move-to-destination(b) Failure Occurrence at a moving task in subtask10

Skill9： Bring-down
Skill10： Hand-open

i=1

Manipulation

i=2

i=9

Skill11： Leave
Skill12： Home

Transfer

i=1～10: Returned tostock space

Fig. 8 Task sequence of picking
and placing a product(c) re-excution from start step

i=1

i=2

i=9

Putting boxes side by sidei =1

i=10: Returned tostock space
i=8: Correctlyaligned

i =5
i =2

i =4

(d) Original repetition from the first step of subtask(i=10) (with another process)
i=1

i=2

i=9

i=10

i=11

i=12

i =9
i =8

i =12

Fig. 9 Sequence of tasks related
to display of products

i=8: Correctly aligned(e) Original repetition from step along the way (with another process)

i=1

i=2

i=9

i=10

i=11

i=12

i=9
i=10
i=8

i=10
i=8

(i) Correct
placementtask(ii) Failure to
shift adjacentproductsi=10i=8(iii) Failure to
knock down
adjacent producti=8: Correctly aligned
if necessary(f) Continuing working to the end and aligning if necessary

Fig. 10 Successes and Failures in tasks ofdisplay of products

Fig. 11 Various recovery patterns

and four columns, as depicted in Fig. 9. Fig. 10 illustrates
an error scenario; Fig. 10(i) demonstrates a carry-andplace operation for a pre-planned 10th object.
Subsequently, Fig. 10(ii) portrays an instance where ahand interferes with the 10th object during the carry-andplace operation, causing the displacement of the 8th itemfrom its original position. Finally, Fig. 10(iii) presents a
scenario wherein the hand impacts the 8th item during thecarry-and-place operations for the 10th item, resulting in
the toppling of the 8th item.

recovery patterns. Fig. 11(c) demonstrates the error
recovery of the backward type back to the start, in which
the 1st object through the 8th object are returned to their
original stock space, and the task is rerun from the
beginning. Fig. 11(d) displays the error recovery of the
backward type, where the recovery task is to return the10th item to its original stock space, restore the 8th item
to its original position and orientation, and pick and place
the 10th item again. Fig. 11(e) shows the forward error
recovery, which executes a recovery task that places the10th item in its designated position even after an error
occurs and then returns the 8th item to its original
position and orientation, followed by the 11th pick-andplace. Fig. 11(f) shows a special type of forward error
recovery, which is a recovery task that continues the work
to the end even after an error occurs and returns the eighth

Fig. 11 shows various recovery patterns. Fig. 11(a)illustrates normal planning in which no errors occur. Fig.
11(b) depicts the point at which the aforementioned
failure occurred during the pick-and-place operation of
the 10th object. Figs. 11(c)–11(f) show the respective©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
403

Powered by TCPDF (www.tcpdf.org)

Akira Nakamura, Kensuke Harada

product to its original position and orientation, if
necessary. If a product is displayed in a store, it may notmatter much if it is misplaced or even fallen over; that is
how to deal with it. However, if it resonated with acustomer's willingness to buy, it is executed. Although
this is a particularly obvious example, it is sometimes
desirable to perform the first half of the recovery task
based on the operator’s criteria and the second half of therecovery task based on the user’s criteria.

8. A. Nakamura, N. Yamanobe, I. R. Alpizar, K. Harada and
Y. Domae, Cost-oriented Planning for Error Recovery inan Automation Plant, J. Robotics, Networking andArtificial Life, 6(4), 2020, pp. 225-230.
9. A. Nakamura, N. Yamanobe, I. R. Alpizar, K. Harada and
Y. Domae, Selection of Optimal Error Recovery Processusing Evaluation Standards in Automated Plants, J.
Robotics, Networking and Artificial Life, 8(3), 2021, pp.
211-217.
10. A. Nakamura and Harada, Error Recovery Patterns
Focusing on the Revival Process from Failures in
Manipulation Tasks, J. Advances in Artificial Life
Robotics, 3(4), 2023, pp. 230-237.
11. T. Hasegawa, T. Suehiro and K, Takase, A robot system
for unstructured environments based on an environment
model and manipulation skills, in Proc. IEEE Int. Conf.
Robot. Autom., (Sacramento, USA, 1991), 916-923.
12. A. Nakamura, T. Ogasawara, T. Suehiro and H. Tsukune,
Fine motion strategy for skill-based manipulation,
Artificial Life and Robotics, Springer, 1(3), 1997, pp. 147150.
13. A. Nakamura, K. Kitagaki and T. Suehiro, Using
simplified geometric models in skill-based manipulation,
Advanced Robotics, 18(8), 2004, pp. 835-858.

7. Conclusion
When an error arises during the primary task, theprocess transitions into the recovery phase. This section
explores diverse recovery paths utilizing our proposederror recovery method, grounded in both task
stratification and error classifications. We introduced amethod to systematically derive the optimal recovery
path, aligning with evaluation standards for each section.
As discussed, numerous recovery paths exist, andchoosing the right one poses challenges. While this paper
introduced a method for optimal path selection, it does
not delve into defining evaluation standards for eachsection. Addressing this aspect becomes imperative for
future research endeavors.

Authors Introduction
Prof. Akira Nakamura
He received the Ph.D. degree inElectrical Engineering from Keio
University in 1991. From 2021, he has
been working as a Professor at Facultyof Engineering of Saitama Institute ofTechnology. His research interests
include robot planning, vision andcontrol system.

References
1. B. R. Donald, Planning multi-step error detection andrecovery strategies, Int. J. Robot. Res., 9(1) (1990) 3-60.
2. T. Niemueller, G. Lakemeyer and S. S. Srinivasa, A
Generic Robot Database and its Application in Fault
Analysis and Performance Evaluation, in Proc. IEEE/RSJ
Int. Conf. on Intell. Robots Syst., (Vilamoura, Portugal,2012), 364-369.
3. E. D. Lello, M. Klotzbucher, T. D. Laet and H. Bruyninckx,
Bayesian Time-Series Models for Continuous Fault
Detection and Recognition in Industrial Robotics Tasks, in
Proc. IEEE/RSJ Int. Conf. on Intell. Robots Syst., (Tokyo,
Japan, 2013), 5827-5833.
4. E. Krabbe, E. Kristiansen, L. Hansen and D. Bourne,
Autonomous Optimization of Fine Motions for Robotic
Assembly, in Proc. IEEE Int. Conf. Robot. Autom., (HongKong, China, 2014), 4168-4175.
5. A. S. Wang and O. Kroemer, Learning Robust
Manipulation Strategies with Multimodal State Transition
Models and Recovery Heuristics, in Proc. IEEE Int. Conf.
Robot. Autom., (Montreal, Canada, 2019), 1309-1315.
6. A. Nakamura, K. Nagata, K. Harada, N. Yamanobe, T.
Tsuji, T. Foissotte and Y. Kawai, Error recovery using task
stratification and error classification for manipula-tion
robots in various fields, in Proc IEEE/RSJ Int. Conf. onIntell. Robots Syst., (Tokyo, Japan, 2013), 3535-3542.
7. A. Nakamura, K. Nagata, K. Harada and N. Yamanobe,
Technique of Recovery Process and Application of AI in
Error Recovery Using Task Stratification and Error
Classification, J. Robotics, Networking and Artificial Life,
5(1), 2018, pp. 56-62.

Prof. Kensuke Harada
He received his Doctoral degrees in
Mechanical Engineering from Kyoto
University in 1997. From 2016, he has
been working as a Professor at
Graduate School of Engineering
Science, Osaka University.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
404

Robotic Food Handling Utilizing Temperature
Dependent Variable-Stiffness Material
Rozilyn Marco

Division of Engineering Science, University of Toronto
Prashant Kumar, Xinyi Zhang, Weiwei Wan, Kensuke Harada

Robotic Manipulation Research Group, Osaka University
Osaka, Japan
E-mail: rozilyn.marco@mail.utoronto.ca, {kumar@hlab.,chou@hlab.,wan@,harada@}sys.es.osaka-u.ac.jp

Abstract
This paper presents a robotic end-effector that addresses challenges of automated robotic food handling. We utilize
a variable stiffness fabric on the finger surfaces of the gripper. Depending on its temperature, the finger stiffnesschanges to tackle grasping food items with different physical characteristics. Hard objects can be grasped with thehard mode of the finger while a fragile object can be safely grasped with the soft mode. This gripper design was
validated empirically through force and object-grasping experiments.
Keywords: robotic end-effector, food picking, variable stiffness, temperature dependence

utilizing this material to handle items requiring different
amounts of applied force by only altering the temperature
exposed to the gripper - eliminating the need forextensive or challenging control procedures required byother soft robotic grippers [5]. We attached the variable
stiffness material to the finger surface of a two-fingered
gripper and the compliancy increases as the temperature
becomes higher. We demonstrate the gripper can
successfully adapt to grasping a variety of food materials
by changing the finger stiffness according to the stiffness
properties of food. The effectiveness of the proposed
gripper is verified by experimental results.

1. Introduction
Within industry, the adoption of robotic automation isdriven by the promise of heightened efficiency and the
reduction of laborious, repetitive tasks that can exert bothmental and physical strain on humans [1]. The foodindustry is a continuous source of opportunities foradvancements in automation. Successful automated foodhandling can improve product quality and production
rates [2], however, is a challenge due to inherent
inconsistencies in food size, texture, and delicacy [3].
The automated processes may take significant time or failto adjust for these variances. This paper addresses thesechallenges by utilizing a compact, mechanically simple
design that takes advantage of the dynamic control of its
applied force and stiffness variability. The adjustablecompliance of the device's material is used in
combination with force feedback sensors to successfully
eliminate the chance of bruising that pinching or otherenclosing end-effectors can cause [2]. Force feedback ortactile sensors are commonly used in industrial robotics[4], allowing the design to remain industry compatible.
The goal of this paper is to demonstrate the potential of

2. Related Work
Food items that are heavy, slippery, fragile, or have aminimal-height profile commonly experience graspingdifficulties with currently proposed gripper designs [6],[7]. This gripper is compatible with objects having these
attributes that other developed variable-stiffness generalgrippers are not [8]. The design proposed by A.
Pettersson et al. utilizing magnetorheological fluid [9]also maintains a secure grasp through a gripper with highand adaptable compliance. However, an improvement©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
405

Rozilyn Marco, Prashant Kumar, Xinyi Zhang, Weiwei Wan, Kensuke Harada

this paper introduces is the compactness of the design —a crucial attribute for practical deployment in confined
spaces [10]. Other grippers that grasp food items by theside surfaces require the object's size to be predetermined[10]. The paper by G. Endo et al. [11] required the heightof the samples to be known and constant. Food items canvary in size and shape even between samples of the same
food, different apples for example, and this requirement
limits the success and robustness of the gripper. The
design proposed in this paper eliminates that requirement
through its dynamic force application and compatibility
with force feedback implementation. The gripper by R.
Maruyama et al. [12] was successful in handling fragileand soft food items but required predetermination of therequired force to apply to obtain a successful grasp. The
design of this gripper allowed for more innate tolerance
in this aspect, allowing it to work with food items notencountered previously.

without causing deformation or damage to samples inboth scenarios.
Between grasps, warm temperature can be applied toreturn the fabric to its neutral position. This methodologyof subjecting the gripper to warm temperature after eachgrasp will be applied during the experiments used in thispaper and can be seen in the flow chart of Fig. 1.

3. Design
The goal of the design was to achieve both simplicity andeffectiveness. Simple mechanical design minimizes therisk of contamination during usage due to component
separation [10] and the gripper’s achieved compactness,increases the ease of adoption in practical scenarios [10].
Relying on the variability of the gripper's compliancefrom temperature exposure as the means of adapting theforce application decreases the requirement for precise
mechanical human adjustments thus not requiring an
upskill in labour [13].

Fig. 1. Decision flow chart when picking an item.
3.2. Implementation
The base structure of the gripper was created throughCAD software and was 3D printed (Fig. 2) with the fabricattached via sewing. Aside from providing a strong
foundation, the base material is inconsequential to thefunctioning of the gripper. It can thus be substituted for ahygienic material such as stainless steel [14].

3.1. Concept
The robotic end-effector incorporates a variable stiffness
fabric as its primary functional component and is themain contributor to the gripper’s low mechanicalcomplexity. This fabric exhibits temperature-dependent
properties, becoming more flexible as the temperaturerises and stiffer as it decreases. When controlled, thisdynamic behaviour can be used to handle objects withdifferent structural integrities. Explicitly, the endeffector can be subjected to higher temperatures when thetarget for transportation is fragile or soft as the fabricwould apply less force while in this more compliant state.
Likewise, when the fabric stiffens when exposed to lower
temperatures, the gripper can exert greater force, making
it suitable for firmly grasping rigid food items. Thisultimately allows for a secure hold and movement

Fig. 2. Gripper physical implementation and CAD
model: (A) Front Profile, (B) Three Quarter Profile, and(C) CAD side profile of gripper (i) 73.7°, (ii) 20.5mm,
and (iii)2mm.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
406

Robotic Food Handling Utilizing

force gauge’s starting position (Fig. 3). To depict the
behaviour throughout the different segments of astandard pick-and-place operation, the carts were:• manually moved the same velocity until the final
displacement to mimic the initial grasping,• maintained its position for approximately 10 seconds
to simulate the effect the gripper would have during
the time required for transportation,• and then released, as if the object has arrived at its
final location.

An essential feature of the gripper base is the curved
nature at the bottom which guides the fabric beneath theitem and provides support to the bottom when the gripper
encloses it. This creates a secure position between the
fabric and the food item and prevents the item fromslipping through the bottom during transportation. To
prevent the loss of the curved formation (Fig.2.B), thefabric was attached to the base while exposed to warm
temperature. This was a method of pre-tensing the fabricto ensure the curved position remains taut as the fabric’s
innate stiffness is reduced at higher temperatures.
The 2mm between the bottom face and the fabric (Fig.
2.C.iii) enables effective grasping of items with narrow
height profiles. The angle of the stem and the radius of
the inner curvature of the base are 16.3° and 11mm
respectively. These values were chosen to minimize thegap between the hands of the gripper and maximize theamount of fabric held vertically. Minimizing the gap
between the hands increased the minimum width of fooditems that could demonstrate the enclosing property of
the gripper i.e. in Fig. 2.C.ii. Maximizing the amount ofvertical fabric increases the area that applies uniform
force along the side of an object. Future design iterations
will investigate methods of allowing these dimensions to
be variable, increasing the versatility of the gripper.
4.

Fig. 3. Force gauge experimental set-up: (A) gauge
cart and displacement blocker cart on track, (B)gripper and gauge prong, and (C) entire force gauge
setup.
4.2. Food picking experiment
The goal was to showcase and verify the final design's
practical ability to handle a variety of characteristics heavy, slippery, fragile, brittle, flat, and rigid.
Accordingly, corresponding food items were used during
testing as seen in Table 1 and Fig. 4.

Experiments

Table 1. Food Item Characteristics and Water
Temperature Assignment

The first experiment executed was a force gauge
experiment to observe the force applied to the fabricdepending on displacement and fabric temperature. The
second was a set of demonstrative experiments to display
the success of the gripper in a variety of arrangements
and sample types. Water was selected as the stimulus
medium as it possesses a high heat capacity, meaning it
requires a relatively high amount of energy to change its
temperature by one degree [15]. This ensured it
maintained at a considerably stable temperature
throughout the execution of an experiment sequence.

Item

Characteristics

A
B
C
D
E
F
G
H

Heavy – 66 gHard, non-uniform vertical profile
Malleable, slippery plastic surface
Soft, narrow
FragileHard, spherical
BrittleHard, narrow – 12.95 mm

Water
Temperature
Hot
Cold
Hot
Hot
Hot
Cold
Hot
Cold

4.1. Force gauge experiment
The force gauge experiment was conducted with thegripper to analyze the force applied to an object over time
under two temperatures and at three final displacement
positions. Specifically, 30°C and 67°C, and 10mm,
20mm, and 30mm respectively. The experiment setup
consisted of the force gauge sensor mounted onto a cartof a linear motion apparatus with a blocking cart
positioned at the desired displacement away from theFig. 4. Experimental food items.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
407

Rozilyn Marco, Prashant Kumar, Xinyi Zhang, Weiwei Wan, Kensuke Harada

The gripper was exposed to temperatures dependingon whether the physical properties of the target object are
better suited for greater or weaker applied force during
grasping. Each food item was tested individually on theplatform as well as in more complicated combinations
including multiple items on the platform grasped incontinual sequence and objects in a stacked formation(Fig. 5). A successful grasp is defined in this paper bymoving a sample from its initial to final location withoutcausing damage or the item slipping mid task.

Fig. 6. Results of the force gauge experiment plotted: (A)all displacements and (B) each displacement individually.
The constant velocity of the gripper movement meantthat time and displacement differed by a constant factor.
Therefore, while accounting for this factor, modelled
relationships with respect to time can proxy relationships
with respect to displacement. The constant velocity
portion of the experiment was isolated and time vs. force- or equivalently displacement vs. force - was analyzed
using both linear and second-order polynomial models.
A strong second-order polynomial relationship wasdetermined between the displacement of the gripper andthe force applied as seen with the R-squared values inTable 3. In future work, an analytical model will beformalized to depict the force a sample will experience
depending on the item’s width (i.e. the gripper
displacement) and considering the temperature.

Fig. 5. Experiment arrangements: (A) Sequential (B)sequential (C) stacked, and (D) combined stacked andsequential.
The gripper was attached to an industrial robot andintegrated with software that controlled grasp planningand execution while employing a depth camera for theidentification of item locations. The gripper's linear
actuator operates using force feedback to determine whenan adequate clamping width has been achieved.
5. Results
5.1. Force gauge
The results are seen in Table 2 and Fig. 6 for each final
displacement value, the fabric exposed to the cooler
temperature applied more force compared to when
exposed to a higher temperature. This confirms thepremise of the gripper i.e. force applied increases as thetemperature of operation decreases and vice versa.

Table 3. Force Gauge Experiment Model
Performance
Displacement

10 mm
20 mm
30 mm

Table 2. Force Gauge Experiment Key Values
Displacement

10 mm
20 mm
30 mm
10 mm
20 mm
30 mm

Max
Plateaued
Force
Force
Hot Temperature - 67°C
0.6 N
0.4N
1.8 N
1.3 N
3.8 N
2.8 N
Cold Temperature - 30°C
2.8 N
0.7 N
6.5 N
2.5 N
9.6 N
3.8 N

Velocity
10 mm
20 mm
30 mm

1.429 mm/s
1.818 mm/s
1.887 mm/s

Linear Model

Polynomial
Model

Hot Temperature - 67°C
0.909
0.971
0.969
Cold Temperature - 30°C
0.931
0.957
0.888

0.927
0.971
0.983
0.932
0.962
0.936

When the final displacement is achieved, the force
quickly decreases and then plateaus. This is ideal as there
is no prolonged period of unnecessary high force applied
after the fabric has molded to the shape of the target. As
seen in Table 2, there is a directly proportional

1.613 mm/s
1.538 mm/s
1.657 mm/s

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
408

Robotic Food Handling Utilizing

relationship between this plateaued value and thedisplacement of the gripper and an inversely proportional
relationship between the force and temperature. Given
that in practice, the displacement of fabric - i.e. a target's
width - is immutable, plateaued force can be controlledwith temperature. The relationship between thisplateaued force, the temperature of the fabric, and thefinal displacement value can be further investigated to
optimize and formalize the temperature used for various
targets with different ideal handling requirements.

Unsuccessful grasping attempts were seen withsamples with widths comparable to the maximum open
position width of the gripper and with the samples inexperiments with. These failures can be attributed toincorrect positioning of the gripper prior to closure. This
could be corrected by further developing the method of
identifying the desired initial location through thecalibration matrix of the software. Additionally, thegripper can be modified to have a bigger difference in the
open position width of the gripper compared to the sizeof the samples to have larger tolerance limits to accountfor the margin of error between generated and desired
locations.

5.2. Food picking
The gripper was successful in grasping and transporting
each food sample from its respective initial positions totheir final destinations. Fig. 7 shows the end-effector
gripping pose for each food item from Fig. 4 and Fig. 8
shows the gripper throughout experiment D from Fig. 5.

6. Conclusion
The robotic gripper proposed in this paper was designed
to utilize its variable stiffness property to successfully
grasp food samples with commonly difficult-to-handle
features without causing damage and while maintaining
mechanical simplicity and compactness. This concept
was formed on the basis that allowing the material of thegripper to do the bulk of adapting to the shape anddelicacy of a given sample permits the gripper to befunctional without human intervention or immediate
control. The gripper's variable force application was
validated in a force gauge experiment conducted under
two temperature conditions and at three displacement
values, confirming the viability of our design concept.
The practical capabilities of the gripper were
demonstrated experimentally with the gripper
performing food-picking sequences involving a diverse
range of items, each possessing distinct properties andarranged in both sequential and stacked configurations.
In future developments, enhancing the gripper's
versatility could involve the implementation of variablewidth and height dimensions to accommodate an evenbroader range of items. Moreover, expanding the rangeof temperatures utilized for more customized force
applications can be investigated for more tailoredhandling of samples.

Fig. 7. Gripping snapshots of each food item .

Acknowledgment
This work was supported by Osaka University throughthe LabFrontier Mini Program and the University ofToronto through the ESROP Global Program.

Fig. 8. Snapshots during the execution of experiment D .

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
409

Powered by TCPDF (www.tcpdf.org)

Rozilyn Marco, Prashant Kumar, Xinyi Zhang, Weiwei Wan, Kensuke Harada

References
1. Tomei G, Cinti ME, Cerratti D, Fioravanti M. Attenzione,
lavori ripetitivi, fatica e stress [Attention, repetitive works,
fatigue and stress]. Ann Ig. 2006 Sep-Oct;18(5):417-29.
Italian. PMID: 17089957.
2. F. Bader and S. Rahimifard, “A methodology for theselection of industrial robots in Food Handling,”
Innovative Food Science and Emerging Technologies, vol.
64, p. 102379, 2020.
3. C. Blanes, M. Mellado, C. Ortiz, and A. Valera, “Review.
Technologies for Robot Grippers in Pick and place
operations for fresh fruits and vegetables,” Spanish J. of
Agricultural Research, vol. 9, no. 4, p. 1130, 2011.
4. P. Li and X. Liu, “Common sensors in Industrial Robots:
A Review,” J. Physics: Conference Series, vol. 1267, no.
1, p. 012036, 2019.
5. D. Rus and M. T. Tolley, “Design, fabrication and controlof Soft Robots,” Nature, vol. 521, no. 7553, pp. 467–475,
2015.
6. S. Ma, L. Du, E. Tsuchiya and M. Fuchimi, "Paper-Made
Grippers for Soft Food Grasping," 2020 17th Int. Conf. on
Ubiquitous Robots (UR), 2020, pp. 362-367.
7. Z. Wang, H. Furuta, S. Hirai, and S. Kawamura, “A
scooping-binding robotic gripper for handling various
food products,” Frontiers in Robotics and AI, vol. 8, 2021.
8. D. Cardin-Catalan, S. Ceppetelli, A. P. del Pobil, and A.
Morales, “Design and analysis of a variable-stiffness
robotic gripper,” Alexandria Engineering J., vol. 61, no. 2,
pp. 1235–1248, 2022.
9. A. Pettersson, S. Davis, J. O. Gray, T. J. Dodd, and T.
Ohlsson, “Design of a magnetorheological robot gripper
for handling of delicate food products with varying shapes,”
J. of Food Engineering, vol. 98, no. 3, pp. 332–338, 2010.
10. Z. Wang, S. Hirai, and S. Kawamura, “Challenges andopportunities in Robotic Food Handling: A Review,”
Frontiers in Robotics and AI, vol. 8, 2022.
11. G. Endo and N. Otomo, "Development of a food handling
gripper considering an appetizing presentation," 2016
IEEE Int. Conf. on Robotics and Automation, 2016, pp.
4901-4906.
12. R. Maruyama, T. Watanabe and M. Uchida, "Delicate
grasping by robotic gripper with incompressible fluidbased deformable fingertips," 2013 IEEE/RSJ Int. Conf.
on Intelligent Robots and Systems, Tokyo, Japan, 2013, pp.
5469-5474.
13. L. N. K. Duong et al., “A review of Robotics andautonomous systems in the food industry: From the supply
chains perspective,” Trends in Food Science
andTechnology, vol. 106, pp. 355–364, 2020.
14. C. Faille, C. Cunault, T. Dubois, and T. Bénézech,“Hygienic design of food processing lines to mitigate therisk of bacterial food contamination with respect toenvironmental concerns,” Innovative Food Science andEmerging Technologies, vol. 46, pp. 65–73, 2018.
15. S. Jordan, Energy and Light. Milton Keynes: Open
University, 2009, p. 179.

Authors Introduction
Ms. Rozilyn Marco
She is currently an undergraduate
student at the University of Toronto
studying Engineering Science. She wasa visiting student at Harada Lab at Osaka
University.

Mr. Prashant Kumar
He received his B.Tach degree in
Mechanical Engineering in 2018 fromthe SRM University in India and his
M.E. in System Innovation from Osaka
University in 2023. He is currently aDoctoral student at Osaka University.
Dr. Xinyi Zhang
She received the Ph.D. degree from
Osaka University in 2023 while amember of the Harada Lab.

Dr. Weiwei Wan
He is currently an Associate Professor
with the School of Engineering Science at
Osaka University. He received the Ph.D.
in robotics from The University of Tokyo.
His research interests include smart
manufacturing and robotic manipulation.

Dr. Kensuke Harada
He is currently a Professor with the
Graduate School of Engineering
Science, Osaka University. He received
his Ph.D. degree from Kyoto University.
His research interests include mechanics
and control of humanoid robots androbotic hands.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
410

Fruits Maturity Classification for Harvesting Robot
Reno Muhammad Fadilla*1, Tokuo Tsuji*2, Tatsuhiro Hiramitsu*3, Hiroaki Seki*4, Kaihei Okada*5

Kanazawa University, Kakumamachi, Kanazawa, Ishikawa, 920-1192, Japan
Email: renofadilla01@stu.kanazawa-u.ac.jp*1, tokuo-tsuji@se.kanazawa-u.ac.jp*2, thiramitsu@se.kanazawa-u.ac.jp*3,
hseki@se.kanazawa-u.ac.jp*4, kaihei2112@stu.kanazawa-u.ac.jp*5
Abstract
This research proposes to create a machine vision using YOLOv5 model to detect maturity of the tomatoes on farm
field which categorized by raw, half-ripe, and ripe. The use of multiple color space RGB, HSV, and LUV dataset
aims to overcome external disturbances such as light and shadows. The results are the machine vision can differentiate
fruits by its category with minor bias and obtaining a good mAP50 rates with the greatest result is LUV color spaces
model. This machine vision with custom datasets in multiple color spaces can be implemented on farm field which isconsidered based on the purposes, environment and external disturbances. These models are developed forautonomous harvesting robot on farm field.
Keywords: Machine Vision, YOLOv5, Color Space

overcome this issues, execution process of CV can be
implemented using multiple color space identification.

1. Introduction
The gradual maturity of technology has developed inmany sectors, one of them is in the agricultural sector
called as precision agriculture. Many agricultural systems
still using traditional method, particularly in the maturity
detection level. The research on precision agriculture
aims to transform traditional method into digitalization
precisely [1]. Robotics and automation are expected tohave a significant impact on farms of the future byincreasing efficiency in production for fruits grading
which are gradually shifting to automation through a
Computer Vision based on fruits detection. In addition,
image processing can provide understanding ofindividual health, nutritional condition, and maturity
level. However, the complexity of the field environment
as well as the unstructured features of fruits resulting agreat challenge to the target detection, particularly formaturity level measurement [2]..

Color space basically can also describe as the way inwhich human eyes can perceive color vision. Color space
shaped from the development of color model which thecolor is associated with an accurate definition of the way
the components are to be inferred and the situations are
viewed with the set of resulting colors. There are manydifferent color spaces such as RGB, NTSC, LUV, HSV,CMYK, and HSI. Each color space has specified function
and color model [5].
In the recent years, many researches have conducted toaccomplish the best algorithm for ML, basically in image
processing, particularly in the fruit’s detection based on
its color space. Detection of seedling using machinetransplanted rice has been developed using machine
vision [6] which aimed to detect paddy seedlings using
many color spaces and plant the seeds using a paddy
machine. Research [7] conducted research by tomato
recognition and localize the object using YOLOv5
algorithm only in RGB images. Research [8] discussed
about a quality evaluation for two type of grapes using
random forest model to identify quality of each fruit type.
Another implementation of ML in image processing on
farming activity also developed for palm fruit [9] which
purposed to classified the oil palm fresh fruit based on the
textures and palm’s skin.

Computer Vision (CV) is a key point for future
implementation on farming technology, particularly inthe fruits grading and classification, yield estimation
counting, health condition, and maturity. Fundamentally,
CV generated using Machine Learning (ML) algorithm
which enable the analysis of massive quantities of data
more rapidly and precisely, instead traditional or manual
method. CV traditionally execute image by primary color
Red, Green, and Blue (RGB), so the images collected by
RGB based camera at RGB wavelength. In addition,
grading fruits can be classified by using color, texture,and share feature descriptors. RGB color detection has agood level accuracy when using CV, however, at somecondition, CV cannot detect object which has rough orirregular environment, such as on farm field. On the fam
field, the objects can be disturbed by external factor, forexample, sun rays, leaf, and shadows [3], [4]. To

This research proposed to determine maturity level for
tomato fruits based on three different color space using
ML and YOLOv5 model on the real farm implementation
to develop the previous similar researches. The maturity
level of the tomato was divided into three class those areraw, half ripe, and ripe. In addition, this research wasclassified the greatest result of dataset’s performancebased on its color space.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
411

Reno Muhammad Fadilla, Tokuo Tsuji, Tatsuhiro Hiramitsu, Hiroaki Seki, Kaihei Okada

containing many numbers of samples which consist of
single tomato for either red or green or orange tomato,group of tomato with the same color, multiple mixed
color type of tomato. In addition, the datasets also consistof multiple small sized images for either red or green ororange tomato. The small sized images used to recognize
fruits with far distance of camera or the fruits with a tiny
shape. To increase the diversity of the samples, thedatasets also contain multiple image characteristics suchas blur, dim, front-facing, upper-part, bright, andoverlapping objects.
Fig. 1. Flowchart of proposed system.

The datasets consist of three classes those areripe_tomato, raw_tomato, and half_ripe_tomato. The
ripe_tomato represent to mature fruit, the raw_tomato
represent to unripe fruit, and the half_ripe_tomato
represent the fruit is half mature [10]. The datasets also
distinguished based on the purpose those are train images
and validation images. The total sample images of trial
and validation for ripe_tomato is 192 and 136
respectively, the total samples of trial and validation for
raw_tomato class are 243 and 101 samples respectively,
and the total samples of trial and validation for
half_ripe_tomato class are 403 and 139 respectively. The
total samples from 334 images datasets areapproximately 838 samples for trial and 376 samples forvalidation. Each class basically is RGB color space andconverted to another color spaces HSV and LUV. Thesevariety of class and color spaces appointed to determine
the best color space to detect fruits maturity level.

2. Methodology
2.1. Proposed System
The overall flowchart of the proposed system ispresented on the Fig. 1. The specified fruits which usedas the main object is tomato. The system was proposedusing Machine Learning based on YOLOv5 Algorithmwith custom datasets. The used of YOLOv5 in thisresearch rather than another version of YOLO has
recently analyzed based on recognition and location of
on-farm fruits. The previous research [7] explained about
the YOLOv5 accuracy for on-farm moving robot.
YOLOv5 has a high-level accuracy rate for rapid image
detection with only 9.0 time/ms. Besides the accuracy
level can be reach for more than 95%. In addition, thenumber of objects detected by YOLOv5 is the highest
then other YOLO model. YOLOv7 is the latest model of
YOLO version however the average detection time is
13.59 time/ms and with a little bit lower accuracy ratecompared with YOLOv5. Based on this literature,
originally, this research purpose is to develop a machine
vision for on-farm robot. Thus, it is required a machine
vision to detect fruits rapidly and precisely. All of the test
has the same requirements such as 100 epochs, 32 batchwith custom datasets in 640 x 640 image size, and threeobject classes.

3. Results and Discussion
3.1. RGB Trial
This trial consists of several tests and each test has its
main purpose. The first test purposed the machine vision
to detect fruits and classified them into two classes
ripe_tomato and raw_tomato. However, the test result
has low quality mean Average Precision (mAP50) for
only 54% and 52% respectively with total 49%. Theseissues happened because the machine vision cannot
detect small-sized fruits for further scope of the images.
In the second trial, more images added to the dataset,
specifically for small-sized tomato for either red or green
tomato class with total additional datasets are 90 images
for both. The additional datasets including single small
sized red and green tomato, and multiple objects of smallsized tomato. The results of the second trial for
ripe_tomato and raw_tomato class have mAP50 92% and
83% with overall mAP50 is 87%. However, in the second
trial, the machine cannot detect orange tomato which isneither categorized as red nor green tomato. Thus, thefinal trial conducted to add more images and one datasetlabelled as half_ripe_tomato. The total images added forthe half_ripe_tomato class are 133 images containing
single tomato, multiple tomato, multiple mixed color
tomato, and the small-sized tomato. The total additional
images are 133, 328 images for overall class, and 823
samples in total for trial and 360 samples in total for

This research was conducted in many trials and three
sequences for identifying fruits based on three color
spaces those are RGB color space trial, HSV color spacetrial and LUV color space trial. This research conducted
to explain scientifically based on the data training. The
purpose of each trial is to achieve the best success rate.
Then, arrange each dataset depends on the color spaces
and implement it into real robot vision on-farm for future
concept.
2.2. Custom Datasets
Images of tomatoes used in training and testing werecollected from open-source image from internet andmanually captured using phone camera on the tomato
farm field located in West Java, Indonesia. The total
image of the data was obtained and store basically as
RGB format. The datasets for both trial and validation©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
412

Fruits Maturity Classification for3.4. Performance Evaluation from All Datasets
Based on this research, there are many advantages anddisadvantages to use multiple color space in a dataset.
Each color space has specific purpose to be used inmachine vision. HSV color space can be used to detectcolor-based image. HSV separates luma, or the image
intensity from chroma or color information. HSV can be
useful for histogram equalization, removing shadows,
lightning disturbance, etc. Besides, RGB color is thebasic color space in every image and captured by camera.
RGB color can be easily to use and formulated because
this color space not necessary to be converted from theoriginal image. However, RGB color space might be alsoeasily distracted with external factors such as shadow,
lightning, sunrays, etc. Meanwhile the LUV color spacecan be used to decouple color with the U and V
components which represent chromaticity values of color
image. Besides the L or Luminance can improve the RGB
picture which distracted from over light disturbance, etc.
The comparison from three color spaces which used inthis research shown in Table 1.

(a)(b)
Fig. 2. (a) Sample results RGB, and (b) P-C Curves.

(a)(b)
Fig. 3. (a) Sample results HSV, and (b) P-C Curves.

Based on the Table 1, the LUV and RGB color space
has similar mAP50 rates with the overall value 85% and
83% respectively. On the other hand, the LUV datasets
have a good performance in object detection proven by
the mAP50 rate shown on the Table 1 because LUV color
space are fundamentally suitable for restoration error
estimation, fast detection, and impulsive noise removal inreal-time color imaging. Based on the Table 1,
ripe_tomato class and raw_tomato class for LUV color
space’s mAP50 rate is higher than RGB color spacebecause some of the images in RGB overlapped byshadows and light as shown in the Fig. 6(a) and Fig. 5(a)respectively. The explanation of comparison between
RGB and LUV datasets also visualized in P-C Curve asshown in the Fig. 2(b) and Fig. 4(b) which shows the
RGB curve relatively fluctuating than LUV graph.
Nevertheless, the graph shows a great stability and raise
gradually with small fluctuations. In contrary, the HSV
has the lowest of all datasets training. The HSV datasets
conversion from RGB color space has the typical color
tone with its backgrounds such as leaf, trees, and grass.
The HSV datasets, either ripe_tomato or raw_tomato
class has a high mAP50 rates for more than 70%.
However, there is an issue regarding thehalf_ripe_tomato class, the machine vision cannot detect
the object significantly resulting only 57% of mAP50
value as shown on the Table 1. This value is nearly with
50% and the results should be evaluated to increase therates [11]. The P-C curves Fig. 3(b) showing therelatively unarranged graph for the half_ripe_tomato andthe prediction line increased gradually.

(a)(b)
Fig. 4. (a) Sample results LUV, and (b) P-C Curve.
validation. The sample images of final results shown on
the Fig. 2(a) and Precision-Confidence Curve (P-C
Curve) shown on the Fig. 2(b).
3.2. HSV Trial
HSV trial basically used the same number of total
datasets with RGB datasets. In the HSV trial, the images
converted into HSV images using Python code. The HSV
trial also use three classes called as ripe_tomato,
raw_tomato, and half_ripe_tomato. The trial test of thisHSV datasets conducted in 100 epochs. Brief results ofthe HSV trial are the machine vision cannot differentiate
raw_tomato and half_ripe_tomato class. This issue
happened because the HSV color space conversion
generates those tomato’s skin look exactly the same. The
sample images of final results shown on the Fig. 3(a) andPrecision-Confidence Curve (P-C Curve) shown on theFig. 3(b).
3.3. LUV Trial
LUV trial basically used the same datasets from theconverted RGB color space to LUV color space using
Python code. The LUV datasets also consist of threeclasses ripe_tomato, raw_tomato, and half_ripe_tomato.
The Trial test conducted in 100 epochs with the detail
information of sample final results shown on the Fig. 4(a)and Precision-Confidence Curve (P-C Curve) shown on
the Fig. 4(b).

The shadow and lightness disturbances occurred during
the dataset collection in this research as shown in Fig. 5
and Fig. 6. RGB image reflect lightness, such as sunrays,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
413

Reno Muhammad Fadilla, Tokuo Tsuji, Tatsuhiro Hiramitsu, Hiroaki Seki, Kaihei Okada

Table 1. Final results comparison.
Class
Trial
192
243
403
838

ripe_tomato
raw_tomato
half_ripe_tomato
Overall(a)(b)

Total Samples
Validation
136
101
139
376

RGB
88%
81%
80%
83%

(c)

(a)

Fig. 5. Lightness disturbance: (a) RGB, (b) LUV, (c)HSV.

mAP50 (Epochs = 100)
HSV
83%
77%
57%
72%

(b)

LUV
95%
84%
77%
85%

(c)

Fig. 6. Shadow disturbance: (a) RGB, (b) LUV, (c)HSV.

into white spot, whereas LUV image converts the whitespot into warm blue color as shown in the Fig. 5(b) which
was not distract the real color of the image and affecting
the machine vision detection more precisely. HSV
images turn the white spot into dark red color as shownin Fig. 5(c) and affect the real color of the images. HSV
images turn the shadow into lighter color compared withthe basic RGB image as shown in the Fig. 6(c) and Fig.
6(a) consecutively. This result influence the machine
vision to detect object more precisely. Moreover, LUV
also turn the shadows into lighter color than RGB shadow,
however, the shadows still configurable and not affectedmuch to fruit’s basic color as shown on the Fig. 6(b).
Basically, YOLOv5 model trained using a pre-training
model originally from RGB color which trained inanother color space LUV and HSV. Hence, the model
generated results in relatively similar to RGB color space.
Overall, LUV color space was the greatest datasets for
tomato classification. Therefore, the LUV can be suitablefor harvesting robot despite each color space has anadvantages and disadvantages in certain occasions andpurposes.

rate. In conclusion, all of the datasets in this research canbe used depending on the purposes, and object’s
circumstances which correlated with external
disturbances such as backgrounds, lightness, andshadows of the images. LUV color space has shown its
performance as the best datasets for rapid segmentationof tomato on farm and more suitable in many external
disturbances such lightness and shadows. On the other
hand, HSV datasets can be used to recognized tomatoes
on farm which has high light intensity and shadow
disturbance. Therefore, this color space can be moresuitable for harvesting robot with rapid segmentation.
References
[1]

[2]

4. Conclusion[3]This research has shown the model of ML which used
to determine the best datasets differentiated from color
space RGB, HSV, and LUV. According to the final
results, RGB and LUV datasets generate an outstanding
typical result for the machine vision, on the contrary, theHSV datasets obtain a relatively poor result. The bestclass in this research from overall datasets evaluation
were ripe_tomato and raw_tomato class. Meanwhile, thehalf_ripe_tomato class acquire intermittent results for all
datasets color space depends on the backgrounds andexternal distraction from the objects. Moreover, HSV
datasets need some additional images and adjustment of
the “Hue” value as required for increasing the success

[4]

[5]

[6]

E. Mavridou, E. Vrochidou, G. A. Papakostas, T.
Pachidis, and V. G. Kaburlasos, “Machine visionsystems in precision agriculture for crop farming,”
Journal of Imaging, vol. 5, no. 12. 2019. doi:
10.3390/jimaging5120089.
W. Jia, Y. Tian, R. Luo, Z. Zhang, J. Lian, and Y.
Zheng, “Detection and segmentation of overlapped
fruits based on optimized mask R-CNN application inapple harvesting robot,” Comput Electron Agric, vol.
172, 2020, doi: 10.1016/j.compag.2020.105380.
A. Bhargava and A. Bansal, “Fruits and vegetables
quality evaluation using computer vision: A review,”
Journal of King Saud University - Computer andInformation Sciences, vol. 33, no. 3. 2021. doi:
10.1016/j.jksuci.2018.06.002.
N. Ismail and O. A. Malik, “Real-time visualinspection system for grading fruits using computer
vision and deep learning techniques,” Information
Processing in Agriculture, vol. 9, no. 1, 2022, doi:
10.1016/j.inpa.2021.01.005.
H. Rubesh, D. Hema, and S. Kannan, “Interactive
Color Image Segmentation using HSV Color Space,”
Science and Technology Journal, vol. 7, p. 1, 2020,
doi: 10.22232/stj.2019.07.01.05.
H. Li et al., “An automatic approach for detecting
seedlings per hill of machine-transplanted hybrid rice

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
414

Powered by TCPDF (www.tcpdf.org)

Fruits Maturity Classification for

[7]

[8]

[9]

[10]

[11]

utilizing machine vision,” Comput Electron Agric, vol.
185, 2021, doi: 10.1016/j.compag.2021.106178.
T. Li et al., “Tomato recognition and location
algorithm based on improved YOLOv5,” Comput
Electron
Agric,
vol.
208,
2023,
doi:
10.1016/j.compag.2023.107759.
D. Pietro Cavallo, M. Cefola, B. Pace, A. F. Logrieco,
and G. Attolico, “Non-destructive and contactless
quality evaluation of table grapes by a computer visionsystem,” Comput Electron Agric, vol. 156, 2019, doi:
10.1016/j.compag.2018.12.019.
A. Septiarini, A. Sunyoto, H. Hamdani, A. A. Kasim,
F. Utaminingrum, and H. R. Hatta, “Machine vision
for the maturity classification of oil palm fresh fruitbunches based on color and texture features,” Sci
Hortic,
vol.
286,
2021,
doi:
10.1016/j.scienta.2021.110245.
Ifmalinda, Andasuryani, and I. Rasinta, “Classificationof tomato (Lycoersicon Esculentum Miil) ripenesslevels based on HSV color using digital image
processing,” in IOP Conference Series: Earth andEnvironmental Science, 2022. doi: 10.1088/17551315/1116/1/012062.
Q. Zheng, Z. Peng, P. Zhu, Y. Zhao, R. Zhai, and W.
Ma, “An Object Recognition Grasping Approach
Using Proximal Policy Optimization with YOLOv5,”
IEEE
Access,
vol.
11,
2023,
doi:
10.1109/ACCESS.2023.3305339.

Dr. Hiroaki Seki
He is currently a Professor in the
Faculty of Frontier Engineering,
Kanazawa University, specialty in thefields of robotics and mechatronics
automation. He received his Doctoral
Degree in Engineering from The
University of Tokyo in 1994.
Mr. Kaihei Okada
He is currently a master student in the
Division of Frontier Engineering,
Kanazawa University, Japan since2023 with research theme in the fields
of computer vision and robotics. He
received his Bachelor Degree inEngineeringfromKanazawa
University, Japan.

Authors Introduction
Mr. Reno Muhammad Fadilla
He is currently a Master Student in the
Division of Frontier Engineering,
Kanazawa University, Japan since2023 with research theme in the fields
of computer vision, and robotics. He
received his Bachelor of Engineeringfrom UIN Bandung, Indonesia in
2022.
Dr. Tokuo Tsuji
He is currently an Assoc. Professor in
the Faculty of Frontier Engineering,
Kanazawa University with researcharea in the fields of informatics, andintelligent robotics. He received his
Doctoral Degree in Engineering.

Dr. Tatsuhiro Hiramitsu
He is currently an Assistant Professor
in the Division of Frontier
Engineering, Kanazawa University,
Japan since 2019 with a research areain the fields of soft robotics andintelligent robotics. He received his
Doctoral Degree in Engineering.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
415

YOLO real-time object detection on EV3-Robot using FPGA hardware Accelerator
Dinda Pramanta

Kyushu Institute of Information Sciences, 6-3-1, Saifu, Dazaifu, 818-0117, Japan
E-mail: pramanta@kiis.ac.jp
Ninnart Fuengfusin, Arie Rachmad Syulistyo, Hakaru Tamukoh

Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan
E-mail: ninnart@brain.kyutech.ac.jp, syulistyo.arie-rachmad788@mail.kyutech.jp, tamukoh@brain.kyutech.ac.jp
Abstract
The growing demand for robots necessitates faster and more precise processing. However, running large Artificial
Intelligence (AI) models from cloud data centers to mobile robots via inference models uses considerable computation
resources, which leads to power limitations, particularly for mobile robots. The use of reconfigurable semiconductor
devices at the hardware level is a promising solution to this problem. We introduce the educational kit EV3-Robot
with a co-design methodology utilizing Field-programmable Gate Arrays (FPGA) Kria KV260 as a hardware
accelerator specifically for object detection. We apply the You Only Look Once (YOLO) model for object detection,
which provides real-time results for practical applications. Additionally, we analyze the processing times of the local
PC and EV3-Robot.
Keywords: EV3-Robot, FPGA, Object detection, YOLO.

In our previous study on hardware-based
communication [8], [9], the communication processes
played a pivotal role. The first step was to design theclient-server communication model, where the EV3Robot served as the server and the local PC as the client.
Next, we set the feedback for motor movements during
specific tasks, in this case, using Ev3dev Python Socket
Connection via Bluetooth to be able detect humans asobjects. After that, we execute the deep-learning
processor unit (DPU) of the YOLO model on the KV260
board. Finally, we evaluate the processing time.

1. Introduction
Japan has an ageing population and a decreasing
number of young people [1]. To address this issue, mobile
robots have gained attention [2]. For example, robots are
expected to perform daily indoor tasks, such as picking
up and delivering objects. To accomplish such tasks,
robots must be capable of recognizing and detecting
specific objects [3].
In the field of Artificial Intelligence (AI), Neural
Networks (NN) have been extensively researched [4], [5],[6]. Neural networks (NN) are equipped with state-ofthe-art perceptual abilities, adaptive learning, andsophisticated human interaction. However, processing
this information requires fast and accurate computation,
especially in mobile robots where real-time interaction isalmost unavoidable. This highlights the increasing needfor robots equipped with reliable NNs. In contrast, You
Only Look Once (YOLO) has gained attention for its fastand high accuracy [7].

2. Methodology
2.1. You Only Look Once version 3 (YOLOv3) as
detection system

Reconfigurable semiconductor devices at the hardware
level offer a promising solution to the challenge ofrunning large YOLO models from cloud to mobile robots
via inference models, which require significant
computation resources and result in low power
consumption. We present the educational kit EV3-Robot,
which utilizes a co-design methodology that employs
local-PC and Field-programmable Gate Arrays (FPGA)
Kria KV260 as a hardware accelerator for the YOLO
model.

Fig.1 Bounding box [10]Fig. 1 illustrates how YOLOv3 predicts the anchor box
by using the bounding box and size clustering. The
feature map cell is represented by 𝑐𝑥 , 𝑐𝑦 , and the preselected bounding box size is 𝑝𝑤 , 𝑝ℎ . The predicted
coordinates are 𝑏𝑥 , 𝑏𝑦 , 𝑏𝑤 , 𝑏ℎ which can be calculated©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
416

Dinda Pramanta, Ninnart Fuengfusin, Arie Rachmad Syulistyo, Hakaru Tamukoh.

through Eq. (1), (2), (3), and (4) where the 𝜎(𝑡𝑥 ) and𝜎(𝑡𝑦 ) are pixel values.
𝑏𝑥 = 𝜎(𝑡𝑥 ) + 𝑐𝑥
𝑏𝑦 = 𝜎(𝑡𝑦 ) + 𝑐𝑦
𝑏𝑤 = 𝑝𝑤 𝑒 𝑡𝑤𝑏ℎ = 𝑝ℎ 𝑒 𝑡ℎ

(1)
(2)
(3)
(4)

a.

Clockwise rotating for scanning “person” and“A” is an initial direction.

2.2. System architecture

Fig. 2. YOLO real-time object detection on EV3Robot using FPGA hardware accelerator.

b. Camera views on each direction
Fig. 4. EV3-Robot motor rotation

Fig. 2 displays a real-time implementation of YOLO
for object detection using an EV3-Robot. The EV3-Robotis equipped with various sensors, although we exclude
the input sensor functions in this system. The operating
system (OS) used is ev3dev, which is Linux-based andstored on a Secure Digital (SD) card. The connection to
the computer is established through Bluetooth andoperates at 2.4GHz. The robot is powered by an ARM-9
processor and includes a built-in mini-LCD display. It
has four input ports for sensor functions and four output
ports for actuators or motors. The EV3-robots motor isconnected using the output ports only for feedback.
2.3. Ev3dev Python Socket Connections for feedbacka.

Server diagram flowb. Client diagram flow
Fig. 3. Client-server communication systems.
Fig. 5. Feedback sequences.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
417

YOLO real-time object detection

Fig. 6 displays the application of DPU and its
integration into the KRIA KV260 board FPGA. Initially,
we prepare the YOLOv3 model [13]. Next, we proceed
to the pre-processing stage to install and verify thedependencies or libraries. Subsequently, we quantize themodel from 32-bit floating-point weights to 8-bit.

Fig. 3 illustrates a remote-control flow that utilizes
python socket connections [11] to set motor movements
as feedback. The communication involves the use ofInternet Protocol (IP) addresses to establish connections.
The local PC acts as a client and sends commands to an
EV3-Robot, which serves as the server, enabling control
over the robot.

Fig. 7 shows that we successfully generated thespecified model for the KV260 architecture model,
referred to as the “xmodel”. Finally, we load “xmodel”
into DPU deployment for real-time hardware acceleration.

Fig. 4 depicts a feedback design for detection thatutilizes the motor functions of the EV3-Robot. The
surrounding environment is scanned by setting up four
directional points, denoted as “A”, “B”, “C”, and “D”.
The direction is rotated 45° clockwise at regular intervals,
in this case, every 4 seconds.

3. Results and Discussion
3.1. Real-time remote-control communication ofclient-server systems

Fig. 5 displays the sequence of the motor rotation
process. The robot is equipped with a standby mode,
referred to as 'scanning for person', which enables it toscan while executing real-time object detection processes
until the desired class, 'person', is detected. Finally, we
have implemented an exit process triggered by the 'Esc'command from keyboard inputs to conclude the sequence.

Table 1. The latency results of the first fourcommand keys.
Keyboard
Message
LatencyKey(ms)Up“going UP! received”
522
Right
“going RIGHT! received”
652

2.4. Getting the weight data from cloud to localPCThe pre-trained COCO weight dataset [12] was utilized.
As we were testing real-time communication during
detection, we only focused on the specified class from
COCO, which is “person” (class_id = 0).

Down“going DOWN! received”

511

Left

“going LEFT! received”

661

Latency is a priority in client-server communications,especially for YOLO applications that require real-time
interaction and responsiveness from the EV3-Robot.
Achieving low-latency communication is crucial for
current system, so we define latency as the round-trip
time (RTT) divided by two in milliseconds (ms) as shownin (5).

This was added to the client part and a real-timeprocessing loop was performed for 'person' object
detection. The wireless camera is mounted to perform
object detection using the YOLOv3 model. It focuses on
specific classes and determines whether a class_id = 0(person) is detected. If so, it sends a message via thesocket connection.

𝐿𝑎𝑡𝑒𝑛𝑐𝑦 (𝑚𝑠) =2.5. Deep-learning Processor Unit (DPU) in
KV260

𝑅𝑇𝑇
2

(5)

The outcomes of the initial four command keys are
displayed in Table1. Each key is associated with aconfirmation message, allowing us to verify when the
instruction was issued. Our findings indicate that thelatency remains below 1,000 𝑚𝑠.

Fig. 6. DPU flow.

Fig. 7. YOLOv3 xmodel.
Fig.8. FPGA hardware accelerator task
©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
418

Dinda Pramanta, Ninnart Fuengfusin, Arie Rachmad Syulistyo, Hakaru Tamukoh.

Table 2. Result comparison during N-step = 1.
Local- FPS InferenceHardware“person”
PCtime (ms) Accelerator
ObjectDetection
13 th Gen
2.63
270.3
None
Success
Intel(R)Core(TM) i713700F
2.10
GHz.

3.78

264.4

FPGA
KV260

Acknowledgements
This research was supported by JSPS KAKENHI Grand
Numbers 23H03468.
References

Success1.

3.2. Real-time visualizations

2.

3.

4.

a.

5.

Local-PC without hardware accelerator.

6.

7.

8.

b.

Local-PC with hardware accelerator FPGA.
Fig. 9. EV3-Robot real-time visualizations
9.

Fig. 8 shows a task that FPGA hardware accelerator
task. Fig. 9(a) and Fig. 9(b) are displaying the real-timevisualization from the EV3-Robot's perspective. The
class detection for 'person' was achieved in the “B”
direction.

10.
11.

3.3. FPGA hardware accelerator

12.

Table 2 compares the local-PC and EV3-Robot's
performance during the detection task. The use of
hardware accelerators resulted in an increase in thenumber of frames per second (FPS) and a reduction in
inference time.

13.

Coulmas, F., 2007. Population decline and ageing inJapan-the social consequences (Vol. 16). Routledge.
Yoshimoto, Y. and Tamukoh, H., 2021. FPGA
implementation of a binarized dual stream convolutional
neural network for service robots. Journal of Robotics andMechatronics, 33(2), pp.386-399.
Hori, S., Ishida, Y., Kiyama, Y., Tanaka, Y., Kuroda, Y.,
Hisano, M., Imamura, Y., Himaki, T., Yoshimoto, Y.,
Aratani, Y. and Hashimoto, K., 2017. Hibikino-musashi@home 2017 team description paper. arXiv preprint
arXiv:1711.05457.
Hinton, G., 2018. Deep learning—a technology with thepotential to transform health care. Jama, 320(11), pp.11011102.
LeCun, Y., Bengio, Y. and Hinton, G., 2015. Deeplearning. nature, 521(7553), pp.436-444.
Paugam-Moisy, H. and Bohte, S.M., 2012. Computing
with spiking neuron networks. Handbook of natural
computing, 1, pp.1-47.
Redmon, J., Divvala, S., Girshick, R. and Farhadi, A., 2016.
You only look once: Unified, real-time object detection. In
Proceedings of the IEEE conference on computer visionand pattern recognition (pp. 779-788).
Pramanta, D., Morie, T. and Tamukoh, H., 2017.
Synchronization of Pulse-Coupled Phase Oscillators overMulti-FPGA Communication Links. J. Robotics Netw.
Artif. Life, 4(1), pp.91-96.
Pramanta, D. and Tamukoh, H., 2019, December. Highspeed synchronization of pulse-coupled phase oscillators
on multi-FPGA. In International Conference on Neural
Information Processing (pp. 318-329). Cham: Springer
International Publishing.
Redmon, J. and Farhadi, A., 2018. Yolov3: An incremental
improvement. arXiv preprint arXiv:1804.02767.
R. Hempel and D. Lechner, “ev3dev,” 2023. Available
online: https://www.ev3dev.org/, accessed on 30 January
2023. [Online]. Available: https://www.ev3dev.org/
Lin, T.Y., et al, 2014. Microsoft coco: Common objects incontext. In Computer Vision–ECCV 2014: 13th European
Conference, Zurich, Switzerland, September 6-12, 2014,
Proceedings, Part V 13 (pp. 740-755). Springer
International Publishing.
Darknet model, accessed on 30 January 2023. [Online]
Available: https://pjreddie.com/darknet/yolo

4. Conclusion
Real-time object detection on the EV3-Robot using
FPGA hardware acceleration has been successfully
implemented. Equipped with the FPGA Kria KV260
accelerator and the YOLO model, the EV3-Robotdemonstrates a practical and effective solution for realtime object detection. This paves the way for enhanced
performance and broader applications in the field of
mobile robotics.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
419

Powered by TCPDF (www.tcpdf.org)

YOLO real-time object detection

Authors Introduction
Dr. Dinda Pramanta
He earned his Bachelor's degree from
Telkom University in 2013 andcompleted both Master's and Doctoral
degrees at the Kyushu Institute ofTechnology in 2016 and 2020. After a
post-doctoral fellowship with NEDO,
Japan, from 2020 to 2021, he iscurrently an Assistant Professor at
Kyushu Institute of Information
Sciences since 2021. His research interests include neuralnetworks, hardware, and AI for educational applications.
Dr. Ninnart Fuengfusin
He earned his B.Eng. degree fromKing Mongkut’s University ofTechnology Thonburi, Thailand, in
2016, and his M.Eng. and D.Eng.
degrees from Kyushu Institute ofTechnology, Japan, in 2018 and 2021,
respectively. Currently a postdoctoral
researcher at the Graduate School ofLife Science and Systems Engineering,
Kyushu Institute of Technology, his research focuses on
deep learning, efficient neural network design, and digital
hardware design.
Mr. Arie Rachmad Syulistyo
A member of IEEE, he earned hisbachelor's degree in computer sciencefrom the University of Brawijaya,
Indonesia, in 2011, and his master's
degree from the University of Indonesia
in 2015. Currently pursuing a Ph.D. inlife science and system engineering at
Kyushu Institute of Technology, Japan, his researchinterests encompass computer vision and machinelearning.
Dr. Hakaru Tamukoh
A member of IEEE, he obtained his
B.Eng.
degreefromMiyazaki
University in 2001 and completed his
M.Eng. and Ph.D. degrees at the
Kyushu Institute of Technology in 2003
and 2006, respectively. After serving asa Postdoctoral Research Fellow, hebecame an Assistant Professor at theTokyo University of Agriculture and Technology from2007 to 2013. Currently a Professor at the Graduate Schoolof Life Science and Systems Engineering, Kyushu Instituteof Technology, his research interests encompass digital
hardware design, soft computing, and home-service robots.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
420

A Low Computational Cost Hand Waving Action Recognition Systemwith Echo State Network for Home Service Robots
Hiromasa Yamaguchi

School of Computer Science and Systems Engineering, Kyushu Institute of Technology,
680-4 Kawazu, Iizuka, Fukuoka 820-8502, Japan
Akinobu Mizutani

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu, Kitakyushu 808-0196, Japan
Arie Rachmad Syulistyo

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu, Kitakyushu 808-0196, Japan
Yuichiro Tanaka

Research Center for Neuromorphic AI Hardware, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu, Kitakyushu, 808-0196, Japan
Hakaru Tamukoh

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu, Kitakyushu 808-0196, Japan

E-mail: yamaguchi.hiromasa512@mail.kyutech.jp
https://www.brain.kyutech.ac.jp/~tamukoh/

Abstract
This study proposes a low computational cost hand-waving action recognition system for non-verbal communication
in home service robots. The system is based on an echo state network, which requires lower computational costs thanthat of deep neural networks (DNNs), and processes time-series data of skeletal coordinates of humans to recognize
hand-waving actions. Additionally, this study proposes and compares two types of Preprocessing ing methods of theskeletal coordinates to ensure the robustness of the human positions on the frame: one method extracts shoulder andarm angles, which are invariable regardless of the humans’ positions and the other normalizes the skeletal coordinates.
The experimental result shows that the proposed system has competitive accuracy and is robust to varying humanpositions.
Keywords: Action recognition, Home service robot, Low computational cost, Echo state network

1.

service robot. Therefore, the resources in the computers
are limited and the computational cost of the process must
be low.
In this study, we focus on hand-waving action
recognition, which indicates that a human is calling thehome service robot and aim to construct a system withlow computational cost. The proposed system recognizes
hand-waving action by providing skeletal information
extracted from videos using MediaPipe [5] to an echo
state network (ESN) [6], a lightweight machine learningmodel.

Introduction

The declining birthrate and aging population havebecome social problems in many countries, not only
Japan, and one of the solutions is the use of home service
robots [1]. For this reason, research on home service
robots has been active [2], [3], [4]. Home service robots
can expand the range of their use by acquiring not only
verbal but also non-verbal information such as humanfacial expressions and movements. In addition, sincehome service robots are required to judge situations andperform actions in real-time, computers for theirinformation processing should be mounted on the home

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
421

Hiromasa Yamaguchi, Akinobu Mizutani, Arie Rachmad Syulistyo, Yuichiro Tanaka, Hakaru Tamukoh

Table 1. Experimental results
Ours
Ours
Slow
UniFormer
(A)
(B)
Only
V2
Accuracy 0.667 0.771
0.625
0.646
Precision
0.875 0.928
0.667
0.622
Recall0.500 0.683
0.714
1.000
processing
3.445 3.446 20.519
29.607time[s]Table 2. Accuracy per subject for the proposed
method (Preprocessing B)
Subject
S1
S2
S3
S4
Accuracy
0.813
0.863
0.775
0.662

Fig. 1. Process flow of the proposed hand-waving
action recognition.
2.

Related Works

MMAction2 [7], provided by OpenMMLab, is a library
for action recognition. This library estimates humanbehaviors such as handshaking, hugging, hand waving,
etc. The tool offers powerful deep neural network(DNN)-based models for action recognition such as
SlowFast [8], UniFormerV2 [9], and VideoMAE V2 [10],however, the three models listed as examples have a huge
number of parameters (more than 20M) and arecomputationally expensive.
While the methods [8], [9], [10] directly feed input
images into DNNs, Doan proposed an action recognition
system with a long short-term memory model thatprocesses skeletal information [11]. Compared to theend-to-end processing models, the method is more
lightweight but still requires high computational costs
because it is based on the DNN.
3.

𝑋𝑙_𝑖𝑛 = tan−1(𝑦 𝑙_𝑤𝑟 − 𝑦 𝑙_𝑠ℎ )(𝑥 𝑙_𝑤𝑟 − 𝑥 𝑙_𝑠ℎ )(1)

𝑋 𝑟_𝑖𝑛 = tan−1(𝑦 𝑟_𝑤𝑟 − 𝑦 𝑟_𝑠ℎ )(𝑥 𝑟_𝑤𝑟 − 𝑥 𝑟_𝑠ℎ )

(2)

𝑙_𝑖𝑛

𝑿𝒊𝒏 = ( 𝑋𝑟_𝑖𝑛 )𝑋(3)

Preprocessing B shown in Eq. (4) and (5) use the xcoordinate of the left and right wrists (𝑥 𝑙_𝑤𝑟 , 𝑥 𝑟_𝑤𝑟 ), thenose ( 𝑥 𝑛𝑜𝑠𝑒 ), and the left and right shoulder(𝑥 𝑙_𝑠ℎ , 𝑥 𝑟_𝑠ℎ ), respectively. The coordinates of the wrists
are relative to the coordinates of the nose to ensure
robustness against parallel movement of the recognition
target. In addition, the wrist coordinates are normalized
by the length of the shoulder. The robustness against
human size in the frame is ensured by normalizing thelength of the shoulder. 𝑿𝒊𝒏 in Eq. (6) is the input to an
ESN.

Proposed Methods

We propose a lightweight hand-waving action
recognition system with MediaPipe [5] and an ESN [6].
The processing flow of the proposed method is shown in
Fig. 1. For each video frame, the system extracts skeletal
coordinates by using MediaPipe The skeletal coordinates
are then given to an ESN, which performs a binaryclassification of whether the hand is waving or not.
Before feeding the coordinates to an ESN, we apply
preprocessing to skeletal coordinates to classify theaction not depending on the human position on the frame.
We propose and compare two types of preprocessing inthis study.
Preprocessing A extracts the angles of the shoulders
and arms. Preprocessing A shown in Eq. (1) and (2) use
the x-coordinate of the left and right wrists (𝑥 𝑙_𝑤𝑟 , 𝑥 𝑟_𝑤𝑟 ),the left and right shoulder (𝑥 𝑙_𝑠ℎ , 𝑥 𝑟_𝑠ℎ ) , and the ycoordinate of the left and right wrists (𝑦 𝑙_𝑤𝑟 , 𝑦 𝑟_𝑤𝑟 ), andthe left and right shoulder (𝑦 𝑙_𝑠ℎ , 𝑦 𝑟_𝑠ℎ ) respectively.
𝑿𝒊𝒏 in Eq. (3) is the input to an ESN.

𝑋𝑙_𝑖𝑛 =

(𝑥 𝑛𝑜𝑠𝑒 − 𝑥 𝑙_𝑤𝑟 )|𝑥 𝑙_𝑠ℎ − 𝑥 𝑟_𝑠ℎ |(4)

𝑋 𝑟𝑖𝑛 =

(𝑥 𝑛𝑜𝑠𝑒 − 𝑥 𝑟_𝑤𝑟 )|𝑥 𝑙_𝑠ℎ − 𝑥 𝑟_𝑠ℎ |

(5)

𝑙_𝑖𝑛

𝑿𝒊𝒏 = ( 𝑋𝑟_𝑖𝑛 )
𝑋
4.

(6)

Experiments

4.1 Setup
We collected a total of 360 video files of actions from six
persons, of which 240 data points were used as training
data from one person, 24 data points were used asvalidation data from another person, and 96 data points
were used as test data from four persons. The CPU of thecomputer that performed the processing was a CPU (Intel(R) Core (TM) i7-6700K).
To verify whether the proposed method can be a
substitute for the prior cases, we conducted similar tests
using the behavior recognition models of the prior cases(however, these models can recognize multiple©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
422

A low computational costbehaviors). We used the SlowOnly [8] and the
UniFormerV2 [9] models provided by the MMAction2
model trained on the Kinetics-700 dataset [12] of 700
behaviors to predict the behaviors of the test data.

In the future, it will be necessary to construct a systemthat can recognize a waving person even if there are
multiple people in the frame because there is a possibility
that multiple people may be in the image in the real world.

4.2 Result

Acknowledgment

The averages of the accuracy, precision, recall rates, andprocessing time for the ten estimation results for the test
data are shown in Table 1. The proposed method has ahigher accuracy rate in hand-waving action recognition
than the other two methods. In addition, we can see that
Preprocessing B is superior to A in all indices. The
proposed method also has a shorter processing time thanthe other two methods.
Table 2 shows the accuracy for each subject using the
proposed method with Preprocessing B, which had thehighest accuracy. These results show that the accuracy
varies greatly depending on the subject.

This research is based on results obtained from a project,
JPNP16007, commissioned by the New Energy andIndustrial Technology Development Organization(NEDO).

5.

References
1.

T. Yamamoto, K. Terada, A. Ochiai, F. Saito, Y. Asahara
and K. Murase, Development of Human Support Robot as
the research platform of a domestic mobile manipulator,
Robomech Journal, Vol. 6, No. 4, 2019.
2. K. Isomoto, Y. Yano, Y. Tanaka, and H. Tamukoh, Robust
trash can lid opening system, Proceedings of the 2023
International Workshop on Smart Info-Media Systems in
Asia (SISA), 2023.
3. T. Shiba, T. Ono and H. Tamukoh, Object Search andEmpty Space Detection System for Home Service Robot,
Proceedings of the 2023 International Conference on
Artificial Life and Robotics (ICAROB2023), 2023.
4. A. Mizutani, Y. Tanaka, H. Tamukoh, K. Tateno, O.
Nomura and T. Morie, A knowledge acquisition systemwith a large language model and a hippocampus model forhome service robots, Proceeding of the Institute ofElectronics, Information and Communication Engineers(IEICE) Rep. vol. 123, no. 208, 2023, pp. 13–18.
5. C. Lugaresi, J. Tang, H. Nash, C. McClanahan, E.
Uboweja, M. Hays, F. Zhang, C. Chang, M. G. Yong, J.
Lee, Wan-Teh Chang, W. Hua, M. Georg and M.
Grundmann, MediaPipe: a framework for building
perception pipelines, 2019.
6. H. Jaeger: Short term memory in echo state networks,
GMD Report 152, 2002.
7. MMAction2
Contributors,
OpenMMLab's
Next
Generation Video Understanding Toolbox and Benchmark,https://github.com/open-mmlab/mmaction2, 2020.
8. C. Feichtenhofer, H. Fan, J. Malik and K. He, SlowFast
Networks for Video Recognition, Proceedings of the IEEE
international conference on computer vision, 2019, pp.
6202-6211.
9. K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, L. Wang and Y.
Qiao, UniFormerV2: Spatiotemporal Learning by Arming
Image ViTs with Video UniFormer, 2022.
10. L. Wang, B. Huang, Z. Zhao, Z. Tong, Y. He, Y. Wang, Y.
Wang, Y. Qiao, VideoMAE V2: Scaling Video Masked
Autoencoders with Dual Masking, Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern
Recognition (CVPR), 2023, pp.14549-14560.
11. T. Doan, An Efficient Patient Activity Recognition using
LSTM Network and High-Fidelity Body Pose Tracking,International Journal of Advanced Computer Science andApplications (IJACSA), VoL 13, 2022.
12. L. Smaira, J. Carreira, E. Noland, E. Clancy, A. Wu and A.
Zisserman, A Short Note on the Kinetics-700-2020 Human
Action Dataset, 2019.

Discussion

We consider the Preprocessing B shown in Eq. (4), (5)and (6) with normalization to the wrist coordinates to be
more accurate than the Preprocessing A using the angles
shown in Eq. (1), (2) and (3), because the rate offluctuation of the values is larger. In particular, the rateof fluctuation of the values for Preprocessing A is small
when the elbow is bent while hand waving. Therefore, we
consider that Preprocessing B is superior.
The proposed method has a low recall for
Preprocessing B. In Preprocessing B, we consider the
influence of the data from the wrist-fixed hand wave to
be significant. Preprocessing B makes predictions basedon fluctuation in wrist coordinates. Therefore, if the handis waving without wrist fluctuation, it cannot berecognized. Therefore, it is necessary to devise a systemthat allows recognition without wrist fluctuation.
As shown in Table 2, variations in accuracy wereobserved among the subjects. This is expected to becaused by the body size and waving habits of the subjects.
It is necessary to verify whether this problem can besolved by using data from multiple subjects in the training
data in the future.
6.

Conclusion

In this study, we proposed a hand-waving action
recognition system based on an ESN to construct a lowcomputational-cost hand-waving action recognition
system to be introduced into home service robots. The
results show that the accuracy of the lightweight model iscomparable to that of deep learning-based multipleaction recognition models.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
423

Powered by TCPDF (www.tcpdf.org)

Hiromasa Yamaguchi, Akinobu Mizutani, Arie Rachmad Syulistyo, Yuichiro Tanaka, Hakaru Tamukoh

Authors Introduction
Mr. Hiromasa Yamaguchi
He is currently an undergraduate student
at Kyushu Institute of Technology,
Japan. His research interests include
reservoir computing and home service
robots.

Mr. Akinobu Mizutani
He received the B.Eng. and the M.Eng.
degree from the Kyushu Institute ofTechnology, Japan, in 2021 and 2023,
respectively. He is currently a Ph.D.
student at the Graduate School of Life
Science and Systems Engineering,
Kyushu Institute of Technology, Japan. His researchinterests include brain-inspired artificial intelligence andits robot application.
Mr. Arie Rachmad Syulistyo
He received the bachelor degree incomputer science from the University ofBrawijaya, Indonesia, in 2011 and themaster degree in computer science fromthe University of Indonesia, Indonesia, in2015. He is currently pursuing a Ph.D.
degree in life science and systems engineering at Kyushu
Institute of Technology, Wakamatsu, Japan. His researchinterests include computer vision and machine learning.
Dr. Yuichiro Tanaka
He received the B.Eng., M.Eng., andPh.D. degrees from the Kyushu Instituteof Technology in 2016, 2018, and 2021,
respectively. He has also been a research
fellow at the Japan Society for the
Promotion of Science (JSPS) from 2019
to 2021. He has been an assistant professor at the ResearchCenter for Neuromorphic AI Hardware of the Kyushu
Institute of Technology since 2021. His research interests
include neural networks, home service robots, etc. He is amember of IEEE, IEICE, and JNNS.
Prof. Hakaru Tamukoh
He received the B.Eng. degree from
Miyazaki University, Japan, in 2001. Hereceived the M.Eng and the Ph.D. degreefrom Kyushu Institute of Technology,
Japan, in 2003 and 2006, respectively.
He was a postdoctoral research fellow of
21st century center of excellent program at Kyushu
Institute of Technology, from April 2006 to September
2007. He was an assistant professor of Tokyo Universityof Agriculture and Technology, from October 2007 to
January 2013. He had been an associate professor fromFebruary 2013 to March 2021 and is currrectly a professor
in the graduate school of Life Science and System
Engineering, Kyushu Institute of Technology, Japan. His
research interest includes hardware/software complex
system, digital hardware etc. He is a member of IEICE,
SOFT, JNNS, IEEE, JSAI and RSJ.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
424

A Rapidly Adjustable Object Recognition Systemthrough Language Based Prompt Engineering

Naoki Yamaguchi

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu, Kitakyushu, 808-0196, Japan
Tomoya Shiba

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu, Kitakyushu, 808-0196, Japan
Kosei Isomoto

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu, Kitakyushu, 808-0196, Japan
Hakaru Tamukoh

Research Center for Neuromorphic AI Hardware, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu, Kitakyushu, 808-0196, Japan
E-mail: yamaguchi.naoki892@mail.kyutech.jp
https://www.lsse.kyutech.ac.jp/

Abstract

We propose the use of language-based prompt engineering to achieve rapidly adjustable object
recognition in RoboCup@Home. The proposed prompt engineering involves humans adding features,such as the color and material of an object, into the text prompts inputted into Language Segment
Anything. In this research, we evaluated the effectiveness of our proposed method in three benchmark
tests for object recognition at RoboCup@Home 2023. The results show that the highest scores wereobtained in specific tasks, indicating that the proposed method applies to various recognition tasks.
Keywords: Home service robot, Object recognition, Prompt engineering, RoboCup@Home

hours to prepare 500,000 training images and 18 hours.
We aim to achieve a faster learning process and improve
operational efficiency.
This study proposes Language Segment Anything (Lang
SAM) [7] as a language-based object recognition system,
offering quick adjustability. To validate its effectiveness,
we conducted experiments at RoboCup@Home [8], aninternational competition focused on developing
practical home service robots.
Additionally, weevaluated operational efficiency by measuring inferencetime and power consumption, comparing it with thetraditional method.

1. Introduction
In recent years, service robots have attracted increasing
attention against the aging population background [1].
This trend has spurred active research in this domain [2],[3]. These robots operate in dynamic environments,
frequently encountering new and unfamiliar objects,
making the learning process of recognition technologies
and operational efficiency crucial. Ono et al.'s research[4] automates dataset generation and annotation,
reducing the time and cost involved in the learning
process while maintaining real-time performance using
You Only Look Once v4 (YOLOv4) [5]. However, theirexperiments [6] showed that YOLO learning took two©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
425

Naoki Yamaguchi, Tomoya Shiba, Kosei Isomoto, Hakaru Tamukoh

3. Proposed Method
3.1. Object recognition with language
For object recognition using language, our proposed
method employs Lang SAM. Lang SAM combines
Grounding DINO [12] and Segment Anything [13] into asingle model, enabling language-based object
recognition and segmentation. Fig. 1 shows a process
flow of object recognition with language.
3.2. Prompt engineering

Fig. 1. Process flow of object recognition with language

Adjust the text prompts that humans enter Lang SAM.
Lang SAM's recognition accuracy depends on the text
prompt. Below, we outline the procedure for adjusting
text prompts:

2. Related Work2.1. Object recognition
In object recognition, annotating datasets is an essential
process [9]. Annotation involves labeling each object inan image with its name, location, shape, and othercharacteristics. Typically, manual processes accomplish
this task. The performance of object recognition heavily
depends on the quality and quantity of the dataset, as wellas the precision of the annotations. Manually creatinglarge volumes of high-quality annotations can incur
substantial costs. Moreover, maintaining the accuracy of
manual annotations is challenging.

(i)

Take pictures of multiple scenes containing theobject of interest.
(ii) The initial text prompt shall be the name of therecognition target.
(iii) Input both the images and text prompts into therecognition system to check if the object iscorrectly recognized and assess the accuracy.
(iv) Improve the text prompts by human addition ofdetails like color and material, aiming torecognize objects not initially identified or toincrease accuracy.

2.2. Dataset generation
To address the issue of manual annotations in object
recognition datasets, research on Sim2Real, which
involves generating datasets using simulators, isprogressing [10], [11]. Utilizing simulators eliminates
human error and enables the rapid and high-quality
generation of datasets. Additionally, this approachensuresconsistency
inannotations.

4.

Experimental Condition

First, we compared the preparation costs for recognizing
new objects using the recognition models YOLOv8 [14]and Lang SAM. We generated the dataset for training
YOLOv8 using conventional methods. Next, we
implemented the proposed method on the Human
Support Robot developed by TOYOTA MOTORCORPORATION [15] to reveal its impact on enhancing
object recognition capabilities. The effectiveness of this
implementation was validated based on the scoresobtained at RoboCup@Home. Finally, we compared theoperational efficiency of YOLOv8 and Lang SAM,
focusing on their inference time and power consumption,
to determine which method was more efficient in realworld scenarios.

2.3. Problems
In addition to the challenges of dataset annotation, asignificant issue in object recognition is the need to train
recognition models with these datasets. Recognizing new
objects requires generating datasets for these objects andconducting training, which can be very costly. This
process involves substantial time and resources, mainly
when introducing new objects to the system, thereby
increasing the overall expense and complexity ofdeveloping effective object recognition models.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
426

A Rapidly Adjustable Object

Table 1. Specifications of theComputational System Used in Experiments

CPU
GPU
MemoryTable 3. Results of RoboCup@Home2023(proposed method: HMA)

11th Gen Intel® Core™ i7-11700@ 2.50GHz × 16
NVIDIA GeForce RTX 3080
32GB

1st
SG

Team

HMA

Score

420

2nd
Tech
UnitedEindhoven
410

Team

HMA

TRAIL

Score

1000

1000

Table 2. Time to prepare recognition model

Dataset[images]

Prepare time[hours]Dataset generator+ YOLOv8

500,000

18

Proposed method

102

MethodSR

3rdeR@sers+Pumas220
Tech
UnitedEindhoven
800

Table 4. Inference time and power consumption.

4.1. Time to prepare recognition model
We created a dataset to train YOLOv8 by generating
500,000 images using the method developed by the Ono
et al. system. Following this, we conducted manual
prompt engineering on these images. We then trained thisdataset using YOLOv8. Next, images from 10 scenes
were captured in real environments to serve as inputs forLang SAM's prompt engineering. We then manuallyconducted prompt engineering on these images. The time
taken to prepare these recognition models was measured.

Inference time
[msec]

Power consumption[W]

YOLOv871.8

95.7

Lang SAM

577.7

233.0

consumption for each of the prepared recognition models.
The inference time is the time recorded as the output ofeach recognition system, and this experiment uses a
maximum wattage recorded by the NVIDIA System
Management Interface [16] as a power consumption.
Table 1 shows the specifications of the PC used in theexperiments.
5. Experimental Result

4.2. Evaluate the effectiveness of the proposed
method

5.1. Time to prepare recognition model

The conditions of the second task performed at the
RoboCup@Home 2023 are summarized:

Table 2 shows the time taken for dataset generation,
training with YOLOv8, and the preparation of the
proposed method. YOLOv8 required 8 hours to create thetraining dataset and 10 hours for training, for a total of 18hours. In contrast, the proposed method required
approximately 2 hours for prompt tuning with 10 scene
images and text prompts.

(1) Storing Groceries (SG): The task involved
organizing and shelving items scattered on a table,
including categorizing objects and handling
unknown items.
(2) Stickler for the Rules (SR): In this task, the robot acts
as a party host, identifying guests not following
house rules and enforcing compliance by explaining
the
rules
to
them.
The rules are as follows:a. Remove shoes inside the house.
b. Do not enter prohibited areas.
c. Do not throw garbage on the floor.
d. Always hold a drink in hand.
This task primarily requires object recognition andhuman interaction technologies.

5.2. The score of RoboCup@Home 2023
Table 3 shows the top three teams and their scores in SG
and SR categories at RoboCup@Home 2023. The results
show that HMA achieved the highest SG and SR scores.
5.3. Inference time and power consumption
Table 4 shows the inference time and power consumption
between the traditional YOLOv8 method and the
proposed method. The results indicated that the proposed
method had approximately 8 times later inference timeand about 2.4 times higher power consumption than
YOLOv8.

4.3. Inference time and power consumption
In section 4.1, we measured the inference time and power©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
427

Naoki Yamaguchi, Tomoya Shiba, Kosei Isomoto, Hakaru Tamukoh

6. Conclusion and Discussion
In this study, we proposed using language-based prompt
engineering to achieve rapidly adjustable object
recognition in RoboCup@Home. We validated theeffectiveness of this approach through participation in
RoboCup@Home2023. We can adjust the proposed
method in just 2 hours, which is only one-ninth of thetime required by conventional methods. Furthermore,
The results from RoboCup@Home 2023 support thefeasibility and effectiveness of our approach in various
tasks in terms of scoring. However, compared to
traditional methods, it became clear that there are issues
with operational efficiency, such as inference time andpower consumption. We consider the need to selectrecognition models appropriately, considering the
preparation cost and operational efficiency.

10.

11.

12.

13.

14.

References
1. “FujiKeizaiGroup,”
https://www.fujikeizai.co.jp/report/detail.html?code=162208813,(Accessed 12/13/2023).
2. T. Shiba, T. Ono and H. Tamukoh, Object Search andEmpty Space Detection System for Home Service Robot,
Proceedings of the 2023 International Conference on
Artificial Life and Robotics (ICAROB2023), 2023.
3. K. Isomoto, Y. Yano, Y. Tanaka, and H. Tamukoh, Robust
trash can lid opening system, Proceedings of the 2023
International Workshop on Smart Info-Media Systems in
Asia (SISA), 2023.
4. T. Ono, D. Kanaoka, T. Shiba, S. Tokuno, Y. Yano, A.
Mizutani, I. Matsumoto, H. Amano and H. Tamukoh,
Solution of World Robot Challenge 2020 Partner Robot
Challenge (Real Space), Advanced Robotics, 2022, 36,
pp.870-889
5. A. Bochkovskiy, C. Wang, H. M. Liao, YOLOv4: optimal
speed and accuracy of object detection. 2020,
arXiv:2004.10934.
6. T. Shiba, A. Mizutani, Y. Yano, T. Ono, S. Tokuno, D.
Kanaoka, Y. Fukuda, H. Amano, M. Koresawa, Y. Sakai,
R. Takemoto, K. Tamai, K. Nakahara, H. Hayashi, S.
Fujimatsu, Y. Mizoguchi, M. Anraku, M. Suzuka, L. Shen,
K. Maeda, F. Matsuzaki, I. Matsumoto, K. Murai, K.
Isomoto, K. Minje, Y. Tanaka, T. Morie, and H. Tamukoh,
Hibikino-Musashi@Home 2023 Team Description Paper,
2023, arXiv:2310.12650.
7. L.
Medeiros,
Lang
Segment
Anything,
https://lightning.ai/pages/community/langsegmentanything-object-detection-and-segmentationwith-text-prompt (Accessed 12/13/2023).
8. RoboCup@Home,
https://athome.robocup.org/,(Accessed 12/13/2023).
9. Shu Yang, JingWang, Sheeraz Arif, Minli Jia, Shunan
Zhong, SAL-Net: Self-Supervised Attribute Learning for

15.

16.

Object Recognition and Segmentation, WirelessCommunications and Mobile Computing, 2021.
M. Denninger, M. Sundermeyer, D. Winkelbauer, D.
Olefir, T. Hodan, Y. Zidan, M. Elbadrawy, M. Knauer, H.
T. Katam and A. Lodhi, BlenderProc: Reducing theReality Gap with Photorealistic Rendering, Proceedings ofInternational Conference on Robotics: Science andSystems (RSS), 2020.
S. Max and B. Sven, Stillleben: Realistic Scene Synthesis
for Deep Learning in Robotics, Proceedings of IEEE
International Conference on Robotics and Automation(ICRA), 2020.
S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J.
Yang, H. Su, J. Zhu and L. Zhang, Grounding DINO:
Marrying DINO with Grounded PreTraining for Open-Set
Object Detection, 2023, arXiv:2303.05499
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L.
Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W. Lo, P,
Dollár and R. Girshick, Segment Anything, 2023,
arXiv:2304.02643
Ultralytics,
Ultralytics
YOLOv8Docs,
https://docs.ultralytics.com/ (Accessed on 12/13/2023).
T. Yamamoto, K. Terada, A. Ochiai, F. Saito, Y. Asahara
and K. Murase. Development of human support robot as
the research platform of a domestic mobile manipulator,
ROBOMECH journal, Vol. 6(1), 2019, pp. 1-15.
NVIDIA.DEVELOPER, System Management Interface
SMI,
https://developer.nvidia.com/nvidia-systemmanagement-interface (Accessed 12/13/2023).

Authors Introduction
Mr. Naoki Yamaguchi
He received the B.Eng. degree from the
National Institute of Technology, Ube
College, Japan, in 2023. He is a Master’s
degree student at the Graduate School ofLife Science and Systems Engineering,
Kyushu Institute of Technology. His
research interests include dataset creation and
Visualization.
Mr. Tomoya Shiba
He received the B.Eng. degree from
National
InstituteofTechnology,
Kagoshima College, Japan, in 2021. Hereceived the M.Eng. from Kyushu Instituteof Technology, Japan, in 2023. He iscurrently in a Ph.D. student in the graduate
school of Life Science and Systems
Engineering, Kyushu Institute of Technology. His researchinterest includes image processing, motion planning, anddomestic service robots.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
428

Powered by TCPDF (www.tcpdf.org)

A Rapidly Adjustable Object

Mr. Kosei Isomoto
He received the B.Eng. degree from
Kyushu Institute of Technology, Japan,
in 2023, respectively. He is a Master’s
degree student at the Graduate School ofLife Science and Systems Engineering,
Kyushu Institute of Technology. His
research interests include brain-inspired
artificial intelligence models and home service robots.
Prof. Hakaru Tamukoh
He received the B.Eng. degree from
Miyazaki University, Japan, in 2001. Hereceived the M.Eng and the Ph.D. degreefrom Kyushu Institute of Technology,
Japan, in 2003 and 2006, respectively. He
was a postdoctoral research fellow of 21st
century center of excellent program at
Kyushu Institute of Technology, from
April 2006 to September 2007. He was an assistantprofessor of Tokyo University of Agriculture andTechnology, from October 2007 to January 2013. He had
been an associate professor from February 2013 to March
2021 and is currrectly a professor in the graduate school ofLife Science and System Engineering, Kyushu Institute ofTechnology, Japan. His research interest includes
hardware/software complex system, digital hardware
design, neural networks, soft-computing and home service
robots. He is a member of IEICE, SOFT, JNNS, IEEE,
JSAI and RSJ.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
429

Development of A SayCan-based Task Planning System Capable of Handling Abstract Nouns
Kosei Yamao

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu, Kitakyushu, 808-0196, Japan
Daiju Kanaoka

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu, Kitakyushu, 808-0196, Japan
Kosei Isomoto

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu, Kitakyushu, 808-0196, Japan
Akinobu Mizutani

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu, Kitakyushu, 808-0196, Japan
Yuichiro Tanaka

Research Center for Neuromorphic AI Hardware, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu, Kitakyushu 808-0196, Japan
Hakaru Tamukoh

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology,
2-4 Hibikino, Wakamatsu, Kitakyushu, 808-0196, Japan
Email: yamao.kosei665@mail.kyutech.jp

Abstract

The task planning system is required to accomplish various requests from a human in real-world
environments. SayCan, one of the task planning systems, has high accuracy. However, its accuracy
decreases for requests that include abstract nouns of the ambiguous word/phrase. We propose a novel taskplanning system based on SayCan that introduces a function for checking concrete names of abstract
nouns and a rule-based skill extraction, enhancing accuracy. The proposed system facilitates theinterpretation of requests and enables appropriate task planning. The effectiveness of the proposed system
was demonstrated at RoboCup@Home, where it achieved high performance .
Keywords: Task planning, Home service robot, Large language model, RoboCup@Home1.

likelihood output from two modules. Although SayCan ishighly accurate, the accuracy decreases for commands
containing abstract nouns such as “fruit” and “drink.”
In this study, we propose a SayCan-based taskplanning system that introduces a function for checking
concrete names of abstract nouns and a rule-based skill
extraction. The contributions of this study are as follows.

Introduction

Recently, the demand for home service robots has been
increasing due to a low birth rate and an aging population,and research on such robots has been active [1], [2], [3],[4], [5], [6]. Highly accurate task planning is necessary torealize a general-purpose service robot that performs
appropriate actions in response to human requests.
SayCan, one of the task planning systems [7]. consistsof two modules: a Say module and a Can module. The
Say module outputs the likelihood of each skill based on
the language instruction, and the Can module outputs thelikelihood of each skill based on the robot's current state.
It decides which skill to execute next based on the⚫
⚫

More straightforward interpretation of humanrequests and more accurate task planning
Demonstrated high performance in a competition toevaluate the performance of home service robots©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
430

Kosei Yamao, Daiju Kanaoka, Kosei Isomoto, Akinobu Mizutani, Yuichiro Tanaka, Hakaru Tamukoh

Fig. 1. Schematic of command recognition
2.

Preliminaries and Related Works

2.1. RoboCup@Home
RoboCup@Home is an international competition for thetechnical development of home service robots [8].
RoboCup@Home includes two tests: General Purpose
Service Robot (GPSR) and the more challenging
Enhanced GPSR (EGPSR), in which robots listen tovarious commands and perform appropriate actions in thehome environment. In competitions, commands
generated from a command generator [9] are used. The
command generator may output commands that include
abstract nouns such as “fruit,” as in “Please find the fruitin the dishwasher.” The commands used in EGPSR are
more complex than those used in GPSR and require more
complex actions.

Fig. 2. Schematic of task planning

2.2. SayCan
3.

SayCan has a predefined skill set that the robot canperform, and it determines which skill to execute from agiven command. SayCan consists of the Say module andthe Can module. The Say module uses a large language
model (LLM) function to output the likelihood of eachword in a sentence. The Say module inputs the given
command and skill descriptions to the LLM and outputs
the likelihood of each skill. The Can module acquires
information about the external world from cameras andsensors and predicts which skills will most likely be
executed. The results of these two modules are combined
to determine the next skill to perform.
We think the Can module is difficult to implement onother robots because it is realized by reinforcement
learning. SayCan is shown in the paper to be highly
accurate task planning even with only a Say module
without a Can module. The experimental results in thepaper confirm that commands containing abstract nouns,
such as “Bring me a fruit,” reduce accuracy.

Proposed System

In this chapter, we describe the proposed system. The
proposed system consists of command recognition andtask planning modules.
3.1. Command recognition
Fig. 1 shows a schematic diagram of the function ofcommand recognition in the proposed system. If an
abstract noun such as an object, item, or food registeredin the database is included in a given command, theproposed system asks for the concrete name of theabstract noun. This process aims to obtain a concrete
name that the robot can recognize since it may notrecognize the object of the abstract noun. As an example,
as shown in Fig. 1(a), the robot asks for the concrete nameof the abstract noun “object” and replaces “object” with“apple” depending on the result of the response.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
431

Development of a SayCan-based

Table 1. Skill set of the proposed system

Skill
move
followfind_obj
find_person
observe_obj
observe_person
grasp_obj
put
pass_obj
answer_question
say
done

Task
go to the {PLACE}
follow the target
find the {OBJECT} on the{PLACE}find {PERSON}
look at the {PLACE} tocheck objects
look at the {PLACE} tocheck people
grasp the {OBJECT}
take the {OBJECT} to the{PLACE}
pass the {OBJECT}
answer a questionsay {LLM’s output}
done

Fig. 3. Base prompt for the proposed system
4.

Experiments

We conducted the following experiments to evaluate theproposed system. We use OpenAI's text-davinci-003 [10]as the LLM in the proposed system as a condition inexperiments.
Experiment 1. Evaluation experiment with command
output using the command generator [9]Experiment 2. Evaluation experiment using actual
competitions at RoboCup@Home 2023

3.2. Task Planning
Fig. 2 shows a schematic diagram of the task planning
function in the proposed system. The task planningconsists of three steps.

In Experiment 1, we randomly generated 100 GPSR
and 100 EGPSR commands, each using the command
generator. In this experiment, the object and location
names of the generated commands are those used in
RoboCup@Home 2023. We input the generated
commands into the proposed system and evaluate
whether the task plan obtained as the output of thegenerated commands is feasible to realize the given
commands. Specifically, we determined whether thesequence and targets of the skills included in the task
plans can accomplish the corresponding commands.
In
Experiment2,
weparticipatedin
RoboCup@Home 2023 held in Bordeaux, France, in July
2023 as Hibikino-Musashi@Home (HMA) team andevaluated the proposed system in the GPSR and EGPSR.

STEP1: Extraction of skill candidates
Extract skill candidates from the skill set using rule-based
constraints. The constraints are based on the order ofskills, such as “The first skill is the move,” “The sameskill is not executed consecutively,” and so on. This
process reduces the number of LLM inputs. Eliminatingskills that are very unlikely to be used is also expected to
improve the accuracy of task planning. Table 1 shows the
skill set of the proposed system. This skill set includes thebasic skills the home service robot needs to perform
various tasks. The say skill's speech content iscomplemented by LLM's output.

5.

STEP2: Generation of task candidatesExperimental Results

5.1. Experiment 1: Use the command generator

First, this process extracts object, place, and personnames registered in the database in advance as keywords
from the command recognition results. Then, generate
the task candidates by combining the keywords and skill
candidates. Table 1 shows the format of task candidates.

As a result, we confirmed that task plans for 61
commands in GPSR and 27 commands in EGPSR weresuccessful. In particular, we confirmed that the proposedsystem can successfully perform task planning by
checking the concrete names of the abstract nouns in thecommands to obtain the concrete names of the target
objects. For example, in the command “Please bring me
the fruit on the desk,” the robot was checking the concrete
name of the abstract noun “fruit,” and the response “apple”
was obtained. Table 2 shows examples of commands forwhich task planning was inaccurate and the possible
causes. We found in many cases that task planning
failures were due to incomplete skill sets and commands
that did not include the required actions.

STEP3: Determination of tasks to be executed
A combination of base prompts and candidate tasks are
input to the LLM, which outputs the likelihood of eachcandidate task. The robot then performs the task with thehighest likelihood. Fig. 3 shows the base prompt. Thebase prompt gives the LLM information, such as the date,time, and role the command should follow.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
432

Kosei Yamao, Daiju Kanaoka, Kosei Isomoto, Akinobu Mizutani, Yuichiro Tanaka, Hakaru Tamukoh

Table 2. Examples of commands that failed in task planning and possible causes

Command
Go to the desk, look for the mug, and place it on the entrance door
Could you please close the entrance door
Hand me some coke in a mug
Table 3. Results of RoboCup@Home 2023

1st
2nd
3rd

GPSR
Team
Score
Tidyboy
400
TRAIL
300
HMA
200

specific knowledge, such as the preferences and habits ofeach family member. For environment-specificknowledge acquisition and utilization, studies of episodic
memory acquisition models inspired by the hippocampus
are undergoing [12], [13], [14]. In the future, we will
integrate the episodic memory acquisition model into therobot to operate based on previously acquired
environment-specific knowledge instead of checking
concrete names of abstract nouns each time.

EGPSR
Team
Score
HMA
700
TRAIL
400
Tidyboy300

5.2. Experiment 2: RoboCup@Home 2023
In GPSR, the command given to the robot was, “Bringme the object behind the lemon from the cabinet.” In
response to this command, the robot asked for theconcrete name of the abstract noun “object.” The operator
responded, “Tropical juice,” however, the speech
recognition failed, and the system did not replace theabstract noun. It scored partial points because it brought
the object to the operator.
In EGPSR, in response to the command, “Getacquainted with Morgan at the exit, then find him in theliving room, please,” the robot found Morgan at the exit.
After that, the robot said to Morgan, “Hello, Morgan. I'm
HSR from Hibikino Musashi@Home”. Finally, the robot
moved to the living room and found Morgan. This
command was judged to be successful and received ascore.
Table 3 shows the top three teams and their scores in
GPSR and EGPSR at RoboCup@Home 2023. As a result,
HMA was ranked 3rd in GPSR and 1st in EGPSR,
showing that the robot works well under real-world
conditions.
6.

Causality
The command does not specify grasp.
No skill to open/close doors.
No skill to pour liquids.

Acknowledgments
This paper is based on results obtained from a project,
JPNP16007, commissioned by the New Energy andIndustrial Technology Development Organization(NEDO). This work was supported in part by JSPS
KAKENHI (23H03468, 23K18495).
References
1. “FujiKeizaiGroup,”
https://www.fujikeizai.co.jp/report/detail.html?code=162208813,(Accessed 12/12/2023).
2. T. Ono, D. Kanaoka, T. Shiba, S. Tokuno, Y. Yano, A.
Mizutani, I. Matsumoto, H. Amano and H. Tamukoh,
Solution of World Robot Challenge 2020 Partner Robot
Challenge (Real Space), Advanced Robotics, 2022, pp.
870-889.
3. Y. Yano, K. Isomoto, T. Ono, and H. Tamukoh,
Autonomous waiter robot system for recognizing
customers, taking orders, and serving food, Proceedings ofthe 26th RoboCup International Symposium, 2023.
4. K. Isomoto, Y. Yano, Y. Tanaka, and H. Tamukoh, Robust
trash can lid opening system, Proceedings of the 2023
International Workshop on Smart Info-Media Systems in
Asia (SISA), 2023.
5. T. Shiba, T. Ono and H. Tamukoh, Object Search andEmpty Space Detection System for Home Service Robot,
Proceedings of the 2023 International Conference on
Artificial Life and Robotics (ICAROB2023), 2023.
6. D. Kanaoka, Y. Tanaka and H. Tamukoh, Applying Center
Loss to Multidimensional Feature Space in Deep Neural
Networks for Open-set Recognition, Proceedings of 17th
International Conference on Computer Vision Theory andApplications (VISAPP2022), Vol. 5, 2022, pp. 359-365.
7. M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B.
David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, A.
Herzog, D. Ho, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, E. Jang,
R. J. Ruano, K. Jeffrey, S. Jesmonth, N. J. Joshi, R. Julian,
D. Kalashnikov, Y. Kuang, K.H. Lee, S. Levine, Y. Lu, L.
Luu, C. Parada, P. Pastor, J. Quiambao, K. Rao, J.
Rettinghouse, D. Reyes, P. Sermanet, N. Sievers, C. Tan,
A. Toshev, V. Vanhoucke, F. Xia, T” Xia’, P. Xu, S. Xu,

Discussion and Conclusion

In this study, we proposed a SayCan-based task planningsystem, which introduces a function to check the concrete
name of abstract nouns and a rule-based skill extraction.
We demonstrated the system's effectiveness throughexperiments using the proposed system and evaluations
in competitions. However, the proposed system has someissues, such as the incompleteness of the skill set and thefact that task planning sometimes fails for commands that
do not contain the required actions. To realize a moreaccurate task planning system, we think it is necessary todevelop skills such as those for opening/closing doors
and to devise prompts to optimize the effectiveness of
LLM [11].
In addition, for the system to operate in a real-world
environment, it is necessary to acquire environment-©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
433

Powered by TCPDF (www.tcpdf.org)

Development of a SayCan-based

8.
9.

10.

11.

12.

13.

14.

M. Yan, and A. Zeng, Do As I Can, Not As I Say:
Grounding Language in Robotic Affordances, arXiv
preprint arXiv:2204.01691, 2022.
“RoboCup@Home.” https://www.robocup.org/domains/3.
(Accessed 12/12/2023).
“RoboCup@Home
CommandGenerator.”
https://github.com/kyordhel/GPSRCmdGen. (Accessed
12/12/2023).
“OpenAI
GPT-3.5
API[text-davinci-003].”https://platform.openai.com/docs/models/gpt-3-5.
(Accessed 12/12/2023).
J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F.
Xia, E. Chi, Q. V. Le, and D. Zhou, Chain-of-thought
prompting elicits reasoning in large language models,
Proceedings of the Advances in Neural Information
Processing Systems (NeurIPS), vol. 35, 2022, pp. 24824–
24837.
Y. Tanaka, H. Tamukoh, K. Tateno, Y. Katori, and T.
Morie, A brain-inspired artificial intelligence model ofhippocampus, amygdala, and prefrontal cortex on home
service robots, Proceedings of the 2020 International
Symposium on Nonlinear Theory and Its Applications(NOLTA), 2020, pp. 138–141.
A. Mizutani, Y. Tanaka, H. Tamukoh, Y. Katori, K. Tateno
and T. Morie, Brain-inspired neural network navigation
system with hippocampus, prefrontal cortex, andamygdala functions, Proceedings of 2021 International
Symposium on Intelligent Signal Processing andCommunication Systems (ISPACS), 2021.
A. Mizutani, Y. Tanaka, H. Tamukoh, K. Tateno, O.
Nomura and T. Morie, A knowledge acquisition systemwith a large language model and a hippocampus model forhome service robots, Proceeding of the Institute ofElectronics, Information and Communication Engineers(IEICE) Rep. vol. 123, no. 208, 2023, pp. 13–18.

Mr. Kosei Isomoto
He received the B.Eng. degree from
Kyushu Institute of Technology, Japan, in
2023, respectively. He is currently a
Master’s degree student at the Graduate
School of Life Science and Systems
Engineering, Kyushu Institute ofTechnology, Japan. His research interests
include brain-inspired artificial intelligence and home
service robots.
Mr. Akinobu Mizutani
He received the B.Eng. and the M.Eng.
degree from the Kyushu Institute ofTechnology, Japan, in 2021 and 2023,
respectively. He is currently a Ph.D.
student at the Graduate School of Life
Science and Systems Engineering,
Kyushu Institute of Technology, Japan.
His research interests include brain-inspired artificial
intelligence and its robot application.
Dr. Yuichiro Tanaka
He received the B.Eng. M.Eng., andPh.D. degrees from the Kyushu Institute ofTechnology in 2016, 2018, and 2021,
respectively. He has also been a research
fellow at the Japan Society for the
Promotion of Science (JSPS) from 2019 to
2021. He has been an assistant professor at
the Research Center for Neuromorphic AI Hardware of the
Kyushu Institute of Technology since 2021. His researchinterests include neural networks, digital hardware
implementation, and home service robots. He is a memberof IEEE, IEICE, and JNNS.
Prof. Hakaru Tamukoh
He received the B.Eng. degree from
Miyazaki University, Japan, in 2001. Hereceived the M.Eng and the Ph.D. degreefrom Kyushu Institute of Technology,
Japan, in 2003 and 2006, respectively. He
was a postdoctoral research fellow of 21st
century center of excellent program at
Kyushu Institute of Technology, from April 2006 to
September 2007. He was an assistant professor of Tokyo
University of Agriculture and Technology, from October2007 to January 2013. He had been an associate professor
from February 2013 to March 2021 and is currrectly aprofessor in the graduate school of Life Science andSystem Engineering, Kyushu Institute of Technology,
Japan. His research interest includes hardware/software
complex system, digital hardware design, neural networks,
soft-computing and home service robots. He is a memberof IEICE, SOFT, JNNS, IEEE, JSAI and RSJ.

Authors Introduction
Mr. Kosei Yamao
He received the B.Eng. degree from the
National Institute of Technology, Ube
College, Japan, in 2023. He is a Master’s
degree student at the Graduate School ofLife Science and Systems Engineering,
Kyushu Institute of Technology. His
research interests include task planningand home service robots.
Mr. Daiju Kanaoka
He received the B.Eng. and the M.Eng.
degree from Kyushu Institute ofTechnology, Japan, in 2020 and 2022,
respectively. He is currently a Ph.D.
student at the Graduate School of Life
Science and Systems Engineering,
Kyushu Institute of Technology. His
research interests include object recognition, multimodal
learning, and home service robots. He is a student memberof IEEE.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
434

RoboCup@Home 2023: Stickler for the Rules Task Solutions
Tomoya Shiba

Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan
Hakaru Tamukoh

Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan

Email: shiba.tomoya627@mail.kyutech.jp, tamukoh@brain.kyutech.ac.jp
https://www.brain.kyutech.ac.jp/~tamukoh/

Abstract
This paper proposes a robot vision system that detects illegal persons who violate house rules such as wearing shoes
in a home environment. Such complex vision systems often require multiple AI systems including person detection,
object detection, and more. Our approach simplifies this by leveraging the combined capabilities of Grounding DINO
and SAM to detect rule violations effectively. The success of our method was proven at RoboCup@Home 2023,
where it secured the highest score among all participating teams.
Keywords: Service robot, Object recognition, Mobile manipulator, RoboCup@Home,

1. Introduction
Home service robots are gaining increased attention due
to the needs of aging societies with declining birth rates[1], [2], [3]. Their key functions include object
recognition, picking and placing, identifying people andenvironments, and interacting with humans. Our research
has involved using TOYOTA's home service robot HSR[4] and our Exi@ robot [5]. Various studies focusing on
the social implementation of service robots for home use
have been explored, often presented in competitions [6],[7], [8], [9]. These studies have mainly been proposedusing dedicated recognition models. However, homeenvironments vary greatly, and even within the same
household, the objects to be recognized change over time
and with the seasons. In RoboCup@Home competitions,the objects are usually announced in advance, leading
most teams to develop recognition systems tailored tothese specific objects [10]. However, some competitions
require the detection of items not disclosed beforehand,such as people, their clothing, and common householdobjects. Therefore, it has been challenging to prepare
recognition systems that can distinguish a wide variety ofshoes to enforce a no-shoes rule or ensure adherence to a
dress code at a party.

drink

shoes

Fig. 1. Detection of rule violations
2. Related works
2.1. Object recognition
We have been researching methods for home service
robots to detect the location of people and objects andthen perform some action in response. Specifically, we
have studied methods for tidying up and detecting people
within a home. In our previous research, we developed amethod for estimating the position of objects andavailable space, utilizing YolactEdge [14] and point
cloud information [15].
2.2. Problem

In this study, we propose a method for detecting ruleviolations in the competition task "Stickler for the Rules
Task" using the detection models Grounding DINO [11]and Segment Anything [12], which employ prompt
tuningforgeneralobjectrecognition.
We implement these methods on TOYOTA’s home
service robot HSR. As a validation, we used them in
RoboCup@Home [13], a competition for home service
robots, to verify their effectiveness. (Fig. 1)

The home environment contains many objects,
including furniture such as desks and shelves.
Additionally, the variety in human clothing is vast,requiring recognition of entirely different items
depending on the season. Therefore, preparing a modelthat accounts for all these variations in a short period ischallenging, and repurposing existing DNN models maylead to unstable recognition outcomes.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
435

Tomoya Shiba, Hakaru Tamukoh

3. Proposed Method

Table. 1. Results of Partial Recognition(Stickler for the Rules)
MethodDetection rate [%] Speed[s]Proposed Method
43.2
1.122
Yolo v8 (Detect)
23.1
0.103
MMDetection
24.2
0.239

3.1. RoboCup@Home: Stickler for the Rules Task
Here, I will explain the "Stickler for the Rules" task in
RoboCup@Home. This task is designed for home service
robots to identify party guests who have broken houserules and politely ask them to stop. The following fourhouse rules are set:
1. All guests must take off their shoes at the entrance.
2. No guests are allowed in the Black Room.
3. Guests are not allowed to leave garbage on thefloor.
4. All guests must always have a drink in hand.

4.2. RoboCup@Home task
At RoboCup@Home 2023, we tested the effectivenessof this task by featuring more than 5 individuals
committing rule violations, arranged in 3 different
patterns of person placement. The results are summarizedin (Table 2). As a result, we were able to perfectly address
two types of rule violations and achieve the highest score.
Table. 2. Results of RoboCup@Home2023(Stickler for the Rules)
Team
Score
HMA (Our team)
1000
Tidyboy
700
TRAIL
1000
Tech United800
eR@sers+ Pumas
300
RoboCanes-VISAGE
200

3.2. Proposed Method
We propose a method for detecting violations using two
Transformer-based models capable of prompt tuning.
Grounding DINO is used for detecting bounding boxes,
and Segment Anything is employed for segmentation
tasks within the bounding box area. This allows for theidentification of a wide variety of clothing and objects
held in hands (Fig. 2). Additionally, to obtain the threedimensional position, we use the pixel coordinates of thecentroid of the bounding box and depth images.

5. Discussion

PromptInput Image

All output

Prompt selecting

Fig. 2. SAM (Segmentation) & Grounding DINO (BBox)

4. Experiments and Results
4.1. Partial Recognition
To detect rule violations within a home, highrecognition accuracy, as well as ease of surveillance, arerequired. To evaluate these performances, we tested therecognition accuracy of partially visible individuals. In
this experiment, we prepared data with humans partiallyvisible in images and conducted a comparison between
YOLO v8 [16], MMDetection [17], and our proposed
method. The images used were from the open-source
human dataset DensePose-COCO [18], and we createdimages divided into quarters, reducing the image size tohalf, to test recognition accuracy. The results are
summarizedin(Table1)
As a result, our proposed method showed the highest
recognition performance in the same task. However, it
was also shown to be the most time-consuming method.

The proposed method has demonstrated higher
recognition accuracy for partial visibility compared toexisting methods. However, since it is based on a
Transformer model, it tends to have longer processed
times. When implementing this method as a function ofhome service robots, it may be effective to use theexisting models for fast recognition in some scenarios,and our method for its strong recognition accuracy incases of partial visibility. Additionally, the recognition
method using prompt tuning is often influenced by the
skill
of
the
promptsetter.
In the future, it will be necessary to establish objective
criteria for prompt tuning.
6. Conclusion
In this study, we proposed a method for utilizing
recognition models capable of prompt tuning to addressthe "Stickler for the Rules" task in RoboCup@Home, andconducted experiments in RoboCup tasks. The results
showed that our method was effective, achieving thehighest score in the same task. In the future, we aim toreclassify similar objects based on recognition scores andother information to reduce misrecognition.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
436

Powered by TCPDF (www.tcpdf.org)

Paper Title (First 4 Words)

References
Authors Introduction
1.
2.
3.
4.

5.

6.

7.

8.

9.

10.

11.

12.
13.
14.

15.

16.
17.

18.

T. Yamamoto, K. Terada, A. Ochiai, F. Saito, Y. Asahara,
and K. Murase, ROBOMECH Journal, 2019.
L. Iocchi, D. Holz, J. Ruiz-del-Solar, K. Sugiura, T. vander Zant, Artificial Intelligence, pp. 258-281, 2015.
H. Okada, T. Inamura, and K.Wada, Advanced Robotics,
2019.
Yamamoto, T., Terada, K., Ochiai, A., Saito, F., Asahara,Y., & Murase, K. (2019). Development of human support
robot as the research platform of a domestic mobile
manipulator. ROBOMECH journal, 6(1), 1-15.
Hori S, Yutaro I, Kiyama Y, et al. HibikinoMusashi@Home 2017 team description paper. Preprint.
2017. Avail-able from: arXiv:1711.05457
Savage J, Rosenblueth DA, Matamoros M, et al. Semantic
reasoning in service robots using expert systems. Robot
Auton Syst.2019;114:77–92..
Ishida Y, Morie T, Tamukoh H. A hardware intelligent
processing accelerator for domestic service robots. Adv
Robot. 2020 June;34(14):947–957.
Yoshimoto Y, Tamukoh H. FPGA implementation of abinarized dual stream convolutional neural network forservice robots. J Robot Mechatron. 2021;33(2):386–399
Taniguchi, A., Isobe, S., El Hafi, L., Hagiwara, Y., &Taniguchi, T. (2021). Autonomous planning based on
spatial concepts to tidy up home environments with service
robots. Advanced Robotics, 35(8), 471-489.
T. Shiba, A. Mizutani, Y. Yano, et al,. “HibikinoMusashi@Home 2023 Team Description Paper,”arXiv:2310.12650, 2023.
Liu, S., Zeng, Z., Ren, T., et al.: 'Grounding DINO:
Marrying DINO with Grounded PreTraining for Open-Set
Object Detection', 2023, arXiv:2303.05499
Kirillov, A., Mintun, E., Ravi, N., et al.: 'Segment
Anything', 2023, arXiv:2304.02643“RoboCup Federation website” (2023 12/15 accessed)
Liu, H., Soto, R. A. R., Xiao, F., & Lee, Y. J. (2021, May).
Yolactedge: Real-time instance segmentation on the edge.
In 2021 IEEE International Conference on Robotics andAutomation (ICRA) (pp. 9579-9585). IEEE.
T. Shiba, T. Ono, and H. Tamukoh, Object Search andEmpty Space Detection System for Home Service Robot,
The 2023 International Conference on Artificial Life andRobotics (ICAROB2023) OS17-2, Online, February 912(10), 2023.
Ultralytics.:'Ultralytics
YOLOv8Docs’,
https://docs.ultralytics.com/, (Accessed on 12/14/2023).
Chen, K, et al. "MMDetection: Open mmlab detection
toolboxandbenchmark."
arXiv
preprint
arXiv:1906.07155 (2019).
Güler, Rıza Alp, Natalia Neverova, and Iasonas Kokkinos.
"Densepose: Dense human pose estimation in the wild."
Proceedings of the IEEE conference on computer visionand pattern recognition. 2018.

Mr. Tomoya Shiba
He received the B.Eng. degree from
National Institute of Technology,
Kagoshima College, Japan, in 2021. Hereceived the M.Eng. from Kyushu
Institute of Technology, Japan, in 2023.
He is currently in a Ph.D. student in thegraduate school of Life Science andSystems Engineering, Kyushu Instituteof Technology. His research interest includes image
processing, motion planning, and domestic service robots.
Dr. Hakaru Tamukoh
He received the B.Eng. degree from
Miyazaki University, Japan, in 2001. Hereceived the M.Eng and the Ph.D. degreefrom Kyushu Institute of Technology,
Japan, in 2003 and 2006, respectively. He
was a postdoctoral research fellow of 21st
century center of excellent program at
Kyushu Institute of Technology, from
April 2006 to September 2007. He was an assistantprofessor of Tokyo University of Agriculture andTechnology, from October 2007 to January 2013. He iscurrently an associate professor in the graduate school ofLife Science and System Engineering, Kyushu Institute ofTechnology, Japan. His research interest includes
hardware/software complex system, digital hardware
design, neural networks, soft-computing and home service
robots. He is a member of IEICE, SOFT, JNNS, IEEE,
JSAI and RSJ.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
437

Offloading Intellectual Processing from Home Service Robots to Edge Devices
Yuma Yoshimoto

NIT, Kitakyushu College, 5-20-1 Shii, Kokuraminamiku, Kitakyushu, Fukuoka, 802-0985 Japan
Mizuki Kawashima

NIT, Kitakyushu College, 5-20-1 Shii, Kokuraminamiku, Kitakyushu, Fukuoka, 802-0985 Japan
Shun Yonehara

NIT, Kitakyushu College, 5-20-1 Shii, Kokuraminamiku, Kitakyushu, Fukuoka, 802-0985 Japan
Email: yoshimoto@kct.ac.jp, k19053mk@apps.kct.ac.jp, k19206sy@apps.kct.ac.jp

Abstract
In this study, we focus on extending the operational time of home service robots by offloading intellectual processing
to circuit devices such as Field Programmable Gate Arrays (FPGAs), which in turn reduces power consumption. The
core of our approach involves developing a method for implementing intellectual processing on FPGAs, coupled witha dynamic circuit reconfiguration technique. This enables the FPGA to adaptively respond to frequent task changes.
We present: (a) methods for transitioning circuits from a robot's control computer to FPGA in response to varying
tasks, and (b) an evaluation of the effectiveness of using FPGA to extend operational time under these rapidlychanging task conditions.
Keywords: Intelligent Processing, Edge Device, Service Robots, FPGA, Neural Networks

thereby reducing power consumption. To achieve this,
we propose the implementation of intelligent processing
on FPGAs and a method to rewrite the circuits on the
FPGA depending on the situation. This paper reports on
the initial investigation, including (a) implementing statemachines in robots and verifying the ability to constructappropriate state machines in response to user commands,(b) exploring methods to switch circuits from the robot
control computer to the FPGA, and (c) examining theeffectiveness of offloading the intelligent processing to
the FPGA.

1. Introduction
In recent years, service robots have gained attention
against the backdrop of a society experiencing declining
birthrates and an aging population. Service robots aredesigned to perform a variety of tasks in homes and stores,
such as tidying up and waitering, thereby reducing theworkload on humans. These robots are equipped withvarious intelligent processing capabilities such as object
recognition and voice recognition, allowing them tounderstand their surroundings and operate flexibly.
Recently, Neural Networks (NNs) have been increasingly
used for these intelligent processing tasks, and it is known
that NNs require a substantial amount of computational
power. The intelligent processing in service robots
demands real-time operation, but achieving this throughsoftware alone is challenging. Therefore, hardware
architecture acceleration is necessary.

2. System Configuration of Service Robots
This section describes the configuration of the service
robots targeted in this study.
2.1. Behavior Planning with State Machines
In service robots, processing is carried out through theconstruction and execution of state machines, as shownin Fig. 1. The process flow is as follows:

Today, GPUs are primarily used as the hardware
architecture for implementing NNs. However, running
numerous NNs simultaneously leads to issues withinsufficient hardware resources. Additionally, GPUs are
generally known for their high-power consumption,
which limits the operational time of robots.

(1) Listen to the Commands
The robot listens to the user's voice and understands thecommands as text.

This research aims to extend the operating time ofrobots by offloading the intelligent processing to circuit
devices like Field Programmable Gate Arrays (FPGAs),(2) Construction of the State Machine©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
438

Yuma Yoshimoto, Mizuki Kawashima, Shun Yonehara

The robot interprets the heard commands and decides inwhat order and which intelligent processes to utilize.

for AI processing in service robots ranges from 12 GB to
24 GB. Therefore, it is not feasible to implement an AI of
size 216 GB in the GPU of a robot control computer.
Thus, integrating this method into the edge side ischallenging. Additionally, incorporating this method into
robots would rely on external computing devices likecloud services, which introduces instability due to
network connections, making it difficult to achieve astable system.

(3) Execution of the State Machine
The robot executes the constructed state machine insequence.
At this stage of executing the state machine (3), the statemachine has already been constructed, and it is known inwhat order and which intelligent processes are needed.

3. Field Programmable Gate Array
An FPGA is a semi-custom LSI composed of a grid ofreconfigurable logic gates. FPGAs can achieve highcomputational performance by reconfiguring theirinternal logic circuits. Unlike GPUs, which are used for
AI processing and consist of block-based arithmeticarchitectures like multiplication and division, FPGAs
construct their architecture with logic operation units,
enabling lower power consumption and faster processing.
Therefore, it is conceivable that using FPGAs allows forreal-time processing.

① Listen to the Commands

Command:
Take the order from Mr./Ms. A② Construction the State Machine
To take the order, ...
Image Recognition: Find Mr./Ms. A
Navigation
Voice Recognition: Listen to the Order

3.1. Comparison of FPGA with GPU and CPU③ Execution the State Machine

Nakahara et al. implemented CNNs on FPGA andcompared it with CPU and GPU [4]. The comparison
table is shown in Table 2. They used the NVIDIA Jetson
TX1 board for comparison, which includes an embedded
CPU (ARM Cortex-A57) and an embedded GPU(Maxwell GPU). The CPU and GPU used Caffe (version0.14) and were compared when running VGG-11 with abatch size of 1 for latency measurement. As shown in
Table 2, FPGA is faster and consumes less power thanthe embedded CPU and GPU. Thus, a binary CNN on
FPGA is more suitable for embedded systems than CPU
and GPU.

Execute Image Recognition: NN (GPU)Execute Navigation
Execute Voice Recognition: NN (GPU)Fig.1 Constructing and Executing the State Machine
2.2. Required Intelligent Processing
Various intelligent processes are necessary for service
robots. For example, Tsuji et al. have implemented theNeural Networks (NN) shown in Table 1 for therealization of versatile robots. In conventional methods,
responding to human commands requires thesimultaneous activation of numerous NNs.

Table 2. Comparison with Embedded Platforms [5]Platform
Embedded CPU Embedded GPU
Device
ARM Cortex-A57 Maxwell GPU-1
FPS [s ]
0.23
36.7
Power
7
17
Consumption [W]

Table 1. AI Implemented in the Robot [1]AI
Model
Object Detection
Mask R-CNN, UOLS
Object Classification
CLIP (ViT-B/32)
Speech Recognition
Whisper
NLP and NLU
GPT-3

FPGA
Zynq 7020
421.9
2.3

2.3. Pathways Language Model (PaLM [2]),SayCan [3]3.2. FPGA Hardware Resource Utilization
There are developed by Google, this robot-specific AI
allows robots to autonomously plan actions when given
ambiguous verbal commands. For instance, if input like'I spilled my drink. Help me,' is given, PaLM-SayCan canmake decisions such as bringing a sponge or disposing ofan empty can. The number of parameters required for
PaLM is said to be 540B [2]. If these parameters aremanaged as float64, the memory size required for PaLM
amounts to 216 GB. However, the memory in GPUs used

Masatomo et al. implemented a model based on
YOLOv3-tiny with Depth wise Separable Convolution
on FPGA (Zynq-7020). They reported a BRAM
utilization rate of 85.71%, indicating that implementing
just one CNN on FPGA already exceeds 50% of thenecessary resources. Therefore, it is difficult toimplement other NNs simultaneously.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
439

Offloading Intellectual Processing from

4. Proposal-

We propose a method to control FPGA with a statemachine. Fig. 2 shows the diagram of the proposedsystem. The operating procedure of the system is asfollows:-① A control computer's state machine sends aninstruction for the AI mode to be executed to the
System on a Chip Manager (SoC Manager), whichthen instructs the FPGA Manager to rewrite.
② The FPGA Manager calls pre-defined circuit
modules and rewrites the intelligent processing
circuits on the FPGA.
③ The FPGA Manager sends input data for inference tothe intelligent processing circuit on the FPGA.
④ The intelligent processing circuit on the FPGA
completes the inference and sends the results back to
the FPGA Manager.
⑤ The FPGA Manager sends the inference results to the
SoC Manager, which then sends them to the statemachine.

The system's operation was also confirmed. The
operational verification was conducted at GPSR task of
RoboCup JapanOpen 2022 @Home League, abenchmark competition for service robots. This taskcompetes on how well robots can respond to input frompeople, evaluating the constructed state machine. The
robot was given instructions in English, and its ability to
go to a designated location and perform specific actions
was tested.
As a result of the experiment, the constructed statemachine was given the input 'Go to the Living room, findJennifer and answer her question.' First, the command
was correctly processed using the voice recognition
function, and then the robot moved to the living room
using the autonomous movement algorithm. Next, it usedperson recognition to find Jennifer and moved to her
location. After moving, it answered questions using thevoice recognition AI. It was confirmed that the statemachine was appropriately constructed for these states up
to the movement, and the states changed according to thesituational changes.

By repeating these steps, intelligent processing is
executed on the FPGA while time-sharing its resources.
Executing intelligent processing on the FPGA resolves
the issue of power consumption, and time-sharing the
FPGA resolves the resource issue.
Main Computer

State Machine

SoC Manager

System on a Chip(SoC)
Processing
System (CPU)Programmable
Logic (FPGA)FPGAManager

FPGA
Circuits

5.2. Consideration of FPGA Circuit Switching
MethodAs a preliminary step to constructing the system shownin Fig. 3, a method to switch the FPGA circuit from thecontrol computer was realized. The specificconfiguration is shown in Fig. 3. This was implemented
on the KV260 evaluation board equipped with FPGASoC, and its operation was verified. The operation flowis the same as described in Chapter 4. In the experiment,
the inference process was confirmed to be possible byswitching between two models of YOLO v3-Tiny [6],each trained on different datasets.

Fig.2 Overview of the Proposed System
5. Experimental
5.1. Construction of the State Machine
A system as shown in Fig. 1 was constructed, and it wasverified that it could (1) listen to commands, (2) constructa state machine, and (3) execute the state machine. In thisverification, the following technologies were
implemented as elemental technologies:-

Depth image processing to estimate the threedimensional position of objects
Speaking function using gTTS and HSR-API
Voice recognition to understand human speech using
WhisperContext understanding that interprets the meaningfrom recognized text strings

FPGA Board for Intelligent Processing (KV260)
System on a Chip (SoC)
Circuit Module

Recognition of opening and closing doors through
Depth image processing
SLAM for creating environmental maps andestimating self-position
Navigation allowing the robot to autonomously
move to a target location
Person and posture detection using skeleton
estimation AI
Object detection with Faster-RCNN

Main Computer

SoC Manager①SocketCommunication⑤SocketCommunication② Configuring the FPGA
Vitus AI APIFPGAManager

CPU
Processing System(PS)③ Input Data
Vitus AI API④ Estimate Results
Vitus AI API

FPGAProgrammable Logic (PL)

Fig.3 System Configuration©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
440

Intelligent
Processing
CircuitPowered by TCPDF (www.tcpdf.org)

Yuma Yoshimoto, Mizuki Kawashima, Shun Yonehara

5.3. Consideration of the Effectiveness When
Implemented on FPGA Circuit

References
1.

The object recognition AI was implemented on AMD's
KV260 FPGA evaluation board and an MSImanufactured notebook PC equipped with RTX3060.
The power consumption of each was measured both in a
steady state and during the operation of the object
recognition AI. The results are shown in Table 3. The
difference between the power consumption during theoperation of the object recognition AI and in the steadystate was 2 W for the FPGA and 96 W for the GPU, which
is a reduction to 1/48th. From this, it can be understoodthat the implementation of intelligent processing on
FPGA is advantageous in terms of power consumption
compared to GPU implementation.

2.
3.

4.

5.

Table 3. Difference in Power Consumption between
FPGA and GPU

6.

Power Consumption [W]During Object
Steady State Recognition AI Difference
Operation
KV260
MSI Stealth-15MB12UE-012JP

10
44

12
140

2
96

C. Tsuji, D. Komukai, et al, “TRAIL 2023 Team
Description Paper,” https://trail.t.u-tokyo.ac.jp/project/robocup2023/.
A. Chowdhery, S. Narang, et al., “PaLM: Scaling
Language Modeling with Pathways,” arXiv:2204.02311.
M. Ahn, A. Brohan, et al., “Do As I Can, Not As I Say:
Grounding Language in Robotic Affordances,”
arXiv:2204.01691.
H. Nakahara, H. Yonekawa, T. Fujii, M. Shimoda, S. Sato,“GUINNESS: A GUI Based Binarized Deep Neural
Network Framework for Software Programmers,” IEICE
Transactions on Information and Systems,” E102.D, 5,
pp.1003-1011, 2019.
M. Matsuda, Y. Araki, et al., “An FPGA Implementationof Object Recognition System with Low Power
Consumption using YOLOv3-tiny-based CNN,” 2022
“Hinokuni - Land of Fire” Information Processing
Symposium (HINOKUNI2022), 2022.
J. Redmon, A. Farhadi, “YOLOv3: An Incremental Improvement,” IEEE Computer Vision and Pattern
Recognition, pp. 1-8, 2018.

Authors Introduction
Dr. Yuma Yoshimoto
He received his B.Eng. degree from
National Institute of Technology(KOSEN), Maizuru College, Japan,
in 2016. He received his M.Eng. and
D. Eng. degrees from Kyushu
Institute of Technology, Japan, in2018 and 2021, respectively. And he
was JSPS researcher, in 2019 - 2021. He was a postdoctoral researcher at the Kyushu Institute of Technology,
Japan in 2021-2022. Currently, he is assistant professor at
the National Institute of Technology (KOSEN),
Kitakyushu College, Japan. His research interests include
deep learning, robot vision and digital hardware design. He
is a member of IEICE, IEEE.

5.4. Verification Using Images Captured by the
Robot

Images were captured using an RGB-D camera (Xtion
Pro) connected to the robot, and these images were
input into the system. As a result, an output was
obtained that shows what appears where in the images.
6. Conclusion
In this paper, we proposed a method for offloading
power-intensive intelligent processing to FPGA circuit
devices, aiming for extended operational times of service
robots. In our verification, as a first step towards realizing
the system, we (a) implemented a state machine in the
robot and confirmed the ability to construct anappropriate state machine in response to user commands,(b) explored a method for switching circuits implemented
on FPGA from the robot control computer, and (c)investigated the effectiveness of offloading intelligent
processing to FPGA. The results confirmed that (a) the
construction and execution of a state machine arepossible, as demonstrated in the RoboCup@Home
DSPL's GPSR, (b) it is feasible to switch actual circuits
from the robot control computer to FPGA, (c) significant
power reduction can be achieved when running the object
recognition AI on FPGA compared to GPU. In addition(d) the edge device (FPGA) estimated the images fromrobot camera (Xtion Pro). Future work will focus on
integrating these developments with the state machineand extending support to other intelligent processes.

Mr. Mizuki Kawashima
He enrolled at National Institute ofTechnology (KOSEN), Kitakyushu
College, Japan, in 2019. In 2021, hepursued the Information Systems
course, focusing on algorithms andcontrol. He commenced his researchin robotics in 2022.
Mr. Shun Yonehara
He enrolled at National Institute ofTechnology (KOSEN), Kitakyushu
College, Japan, in 2019. In 2021, hepursued the Information Systems
course, focusing on algorithms andcontrol. He commenced his researchin robotics in 2022.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
441

A Research on Performance Information Editing Support System for Automatic Piano
Yoshiki Horii

Kyushu Institute of Technology, 680-4 Kawazu, Iizuka-shi, Fukuoka, 820-8502, Japan
Eiji Hayashi

Kyushu Institute of Technology, 680-4 Kawazu, Iizuka-shi, Fukuoka, 820-8502, Japan
Email: hori.yoshiki311@mail.kyutech.jp, haya@mse.kyutech.ac.jp

Abstract
In 1996, Hayashi et al. from our laboratory developed an automatic piano playing device that requires data withnuances for each note to perform in a human-like piano playing. However, this device lacks the function to infer suchnuanced data. Therefore, prior research focused on developing a system to infer data with nuances for each note. Yet,this system required manual inference, consuming a significant amount of time. In this study, we have constructed asystem capable of automatically performing inference using deep learning. This system not only improves theaccuracy of piano playing inference but also contributes to the efficiency of the inference process.
Keywords: Automatic Piano, Computer Music, Deep Learning

1. Introduction

The device plays as instructed by inputting MIDI
(Musical Instrument Digital Interface) standard data, acommon standard for transferring and sharing
performance data among electronic instruments,
containing information like pitch and volume. By
recording a professional pianist's performance as MIDI
data and inputting it into the device, it's possible toreplicate their performance. However, due to the system's
design, inputting only the score data does not result in ahuman-like performance. To address this issue, efforts
are being made to develop a system that generates
human-like performances from score data. As a first step,efforts are being made to replicate the performances ofworld-renowned pianists.

The automatic piano playing device (Fig. 1) developed
by Hayashi and others, equips a grand piano's keyboard
with striking mechanisms and its pedals with driving
mechanisms, allowing for precise keystrokes and pedal
operations through computer control. During its
development, the behavior of the piano action was
analyzed using a unique mechanical model, examining its
dynamic properties. Additionally, to create optimal drive
waveforms according to each key, the results of the keywise behavior analysis were stored in a database forstable sound reproduction. The waveforms are generated
by referencing the database in accordance with the music
data. This technology enables stable performance, evenreplicating delicate and rapid repetitive strikes that arechallenging for professional pianists.

Previous research developed a system that infers datawith nuances for each note, but this system required
manual inference. Since even short musical scorescontain over a thousand notes, the inference process wastime-consuming and labor-intensive. Therefore, in thisstudy, we constructed a system that automatically
performs inference using deep learning.
2. Inference system using Deep Learning
2.1. Dataset
In this research, we used performance data of the pianist
Vladimir Davidovich Ashkenazy, which were recordedin accordance with the MIDI standard.

Fig. 1 Automatic piano playing device [1]©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
442

Yoshiki Hori, Eiji Hayashi

2.2. Performance data

2.5. Neural network configuration

In performance data, the raw state is often too complex
for direct machine learning application. Therefore, thisresearch involved separating the performance data into
performance information and score information andsetting parameters to make it manageable for machinelearning. This process essentially simplifies andorganizes the data, making it suitable for the algorithms
to process and learn from, thereby enhancing theeffectiveness of the machine learning application in thecontext of piano performance interpretation.

The inference systems developed to date have not been
built using deep learning. However, in developing an
inference system using deep learning, it is necessary tofind the optimal network structure for inference ofperformance information. For this reason, we decided toconstruct and verify a system with a simple network
structure. Fig. 2 shows a schematic of the constructed
network.

In Sections 2.3 and 2.4, the research details theparameters set for both performance information andscore information.
2.3. Performance information
Performance information includes data that
encapsulates the pianist's expression within theperformance data. In this study, four parameters were set
for the performance information. These parameters of theperformance information are shown in Table 1.

Fig. 2 Network overview

Table 1 The format of the performance information
Parameter
Velo
Gate
Step

Units
None
ms
ms

Time

ms

3. Inference Experiment and Results

AboutSound intensity
Length of note
Interval between the nextnoteTime of sound3.1. Data splitting
In this experiment, the performance data was divided
into training data, validation data, and test data. K-foldcross-validation is a statistical method for evaluating
generalization performance in which the data ispartitioned into K pieces, one of which is the validation
data, and the remaining K-1 pieces are used as training
data and evaluated with the test data. This is done by
dividing the data into K pieces, one of which is thevalidation data. Fig. 3 shows a schematic of K-fold crossvalidation. In this experiment, Prelude Op.28-7 byFryderyk Francszek Chopin was used as the test data.

In this research, Time is not a target of inference
because it can be calculated from the Gate and Step
values.
2.4. Score information
Score information is performance data that includes
musical notation such as notes and musical symbols. In
this research, five parameters were set from the score
information. Table 2 shows the parameters of the score
information.
Table 2 The format of the score information
Parameter
Key
Bar
Dyn
Tgate

Units
None
None
None
ms

Tstep

ms

About
Sound heightBar number
Dynamics markLength of note on thescore
Interval between the next
note in the score

Fig. 3 K-fold Cross-Validation overview©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
443

A Research on Performance

3.2. Validated network structure
In this experiment, the layers of neurons in intermediate
layer 1 and intermediate layer 2 were varied according to
the patterns shown in Table 3.
Table 3 Combination of neurons in intermediate layers 1
and 2
Patternnumber1
2
3
4

Layer 1

Layer 2

50
50
100
100

50
100
100
50

Fig. 6 Step of pattern 3

3.3. Results

Table 4 Correlation coefficient of performance
information

The performance information inferred by the inference
system and the pianist's performance information were
output as graphs for each Velo, Gate, and Step. The
graphs of the patterns with the highest correlation
coefficients are shown in Fig. 4, Fig. 5 and Fig. 6. The
correlation coefficients between the inferred performance
information and the pianist's performance information are
shown in Table 4.

Patternnumber1
2
3
4

Layer 1

Layer 2

Velo

Gate

Step

50
50
100
100

50
100
100
50

0.763
0.776
0.770
0.763

0.720
0.741
0.743
0.691

0.876
0.880
0.885
0.832

4. Consideration
Table 4 shows that Velo's correlation coefficient does
not change significantly for any of the patterns, andalthough Velo's correlation coefficient is high, thewaveforms of the graphs are not similar, as can be seenin Fig. 4. This is because the points with large value
fluctuations were learned as outliers, and the fluctuationsin the inferred values became smaller.
The correlation coefficient of Step is larger than that of
Velo and Gate. This is because the variation of the Step
waveform is more regular than the other values, making
it easier to learn, according to Fig. 6.
Furthermore, looking at the correlation coefficients forpatterns 2 and 3, we see that the correlation coefficients
for Velo, Step, and Gate are all large. This suggests thata network structure with a large number of neurons in thelayers close to the output layer is suitable for inference.

Fig. 4 Velo of pattern 2

5. Conclusion
In this study, we developed an inference system with asimple network structure using deep learning. Using thedeveloped system, we verified the optimal network
structure for inference of performance information. The
results showed that a network structure with a large
number of neurons in the layers close to the output layer
is suitable for inference.
Fig. 5 Gate of pattern 3

The future goal is to develop a more accurate inference
system. To this end, we will search for the optimal©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
444

Powered by TCPDF (www.tcpdf.org)

Yoshiki Hori, Eiji Hayashi

network structure by testing a system with an additional
number of neurons, layers, and a gating mechanism thattakes previous inputs into account.
References
1.

E. Hayashi, M. Yamane, H. Mori, Development of amoving coil actuator for an automatic piano, Int. J. Japan
Soc. Prec. Eng. 28 (1994), 164–169.

Authors Introduction
Mr. Yoshiki Hori
He received bachelor degree inEngineering in 2023 from mechanicalsystem engineering, Kyushu Instituteof Technology in Japan. He iscurrently a Master student at Kyushu
Institute of Technology and conducts
research at Hayashi Laboratory.

Prof. Eiji Hayashi

He is a professor in the Department
of Intelligent and Control Systems at
Kyushu Institute of Technology. He
received the Ph.D. (Dr. Eng.) degreefrom Waseda University in 1996. His
research interests include Intelligent
mechanics, Mechanical systems andPerceptual information processing.
He is a member of The Institute ofElectrical and Electronics Engineers (IEEE) and The Japan
Society of Mechanical Engineers (JSME).

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
445

Development of Autonomous Mobile Field Robot– Accuracy Verification of Self-Localization through Simulation Takamasa Hayashi

Graduate School of Computer Science and Systems Engineering, Kyushu Institute of Technology,
680-4 Kawazu, Iizuka-city, Fukuoka, 820-8502, Japan
Shintaro Ogawa1, Yuto Okawachi1, Tan Chi Jie1, Janthori Titan1,
Ayumu Tominaga2, Eiji Hayashi1, Satoko Seino3
1

Graduate School of Computer Science and Systems Engineering, Kyushu Institute of Technology,
680-4 Kawazu, Iizuka-city, Fukuoka, 820-8502, Japan2
Department of Creative Engineering Ribitucs and Mechatronics Course, National Institute of Technology,
Kitakyushu College, 5-20-1 Shii, Kokuraminamiku, Kitakyushu, Fukuoka, 802-0985, Japan
3
Graduate School of Engineering, Kyushu University,
744 Motooka, Nishi-ku, Fukuoka, 819-0395, Japan

Email: hayashi.takamasa784@mail.kyutech.jp, tominaga@kct.ac.jp,
haya@ics.kyutech.ac.jp, seino@civil.kyushu-u.ac.jp,
http://www.kyutech.ac.jp/

Abstract
In recent years, the increase in marine debris has become a significant challenge in terms of its collection. Costal
debris, a type of marine debris, can be collected by human hands, but the variety in shapes, and sizes presents
limitations to human-only collection efforts. To address this, I focused on developing an autonomous mobile robot,
establishing a simulation environment was considered crucial for facilitating smooth progress. This paper focuses on
self-localization, an essential aspect for autonomous movement. We replicated an actual coastal cleaning site within
the simulation environment and evaluated the accuracy of self-localization using an EKF (Extended Kalman Filter)with multiple sensors.
Keywords: Field Robot, Self-Localization, Gazebo, Simulation, Extended Kalman Filter (EKF)

powered by a gasoline engine. However, due to thevehicle’s structure, the turning radius was large and stable
traveling on steep slopes and rocky terrain was difficult,among other problems. To solve these problems, theplatform was changed. The new platform uses Agilex’s
BUNKER [2] as shown in Figure 1. The main changes tothe platform are that the traveling mechanism has two
opposing wheels and the drive wheels are crawlers. This
enables the platform to make super-clever turns,
improving maneuverability composed to conventional
platforms. In addition, the robot can climb hills with aslope angle of 36°, which is expected to enable it to run
at higher speeds. The robot is equipped with an RGB-D
sensor and 3D LiDAR as a visual system for autonomy.
An encoder is mounted inside the vehicle body tomeasure the rotation speed of the wheels.

1. Introduction
In recent years, the increase in marine debris has
become a significant challenge in terms of its collection.
Costal debris, a type of marine debris, can be collected byhuman hands, but the variety in shapes, and sizes presents
limitations to human-only collection efforts. To address
this, we focused on developing an autonomous mobile
robot, establishing a simulation environment wasconsidered crucial for facilitating smooth progress. This
paper focuses on self-localization, an essential aspect forautonomous movement. We replicated an actual costal
cleaning site within the simulation environment andevaluated the accuracy of self-localization.
2. Autonomous Mobile Field Robot “BUNKER”
In our previous study, we developed an autonomous
mobile field robot [1] based on Kawasaki Heavy
Industries’ KFX®90, an all-terrain vehicle (ATV)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
446

Takamasa Hayashi, Shintaro Ogawa, Yuto Okawachi, Tan Chi Jie, Janthori Titan, Ayumu Tominaga, Eiji Hayashi, Satoko Seino

3. Experiment
In the coastal environments shown in Figure 4 andFigure 5, the initial position of the robot was set as theorigin of spatial coordinates, and from there, threedestinations were set at (40, -5), (40, 0), (40, 5). The robot
was remotely operated to each destination five times toextract the necessary odometry information. For selflocalization, both individual odometries and multiple
odometries integrated via EKF were implemented,
resulting in seven estimation patterns. Mean Absolute
Error (MAE) was used for error calculation.

a. KFX®90
b.BUNKER
Fig. 1. Platform Overview
2.1. Simulation Construction
In this study, we not only represented the Agilex
BUNKER in the simulation environment but also
recreated the coast of Hokuto Mizukumi Park in
Munakata City, Fukuoka Prefecture, where actual beachcleaning is performed. We used 3D mapping technology
developed by prior research using drones. For thesimulation environment, we employed Gazebo, a 3D
dynamic simulator capable of efficiently and accurately
simulating groups of robots in complex indoor andoutdoor environments. Gazebo can also be integrated
with ROS (Robot Operating System) [3]. The projected
BUNKER and the coast on Gazebo are shown in Figure
2.

Fig. 4. Test environmenta. Coastal cross section

a. BUNKER
b. coast
Fig. 2. Built simulation environmentb. View from initial position
Fig. 5. Experiment in a simulation environment

2.2. EKF-Based Self-Localization System
The posture measurement of mobile robots is achieved
through various sensors and methods. However, themeasurements obtained from these sensors are not true
values but are considered to contain errors. In this study,
we implemented the integration of odometry using the
Extended Kalman Filter (EKF) shown in Figure 3, toachieve robust and low-error self-localization. We usedwheel odometry, RGB-D odometry, and LiDAR
odometry. The system was designed to accept any ofthese, individually or in combination, and produce anintegrated odometry output.

3.1. Result
Table 1 summarizes the errors in self-localization
results in the x-axis direction. In Pattern 4, the error was
minimal, with the odometry integrating wheel odometry
and RGB-D odometry via EKF achieving the highest
accuracy in self-localization. Additionally, focusing on
Pattern 2, the RGB-D odometry alone also showed
generally similar values. From Pattern 3, it was observedthat the LiDAR had an error of about 20 m, indicating it
was not very useful in the simulated environment
replicating an actual beach.

Fig. 3. Input/output data to/from EKF©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
447

Development of Autonomous Mobile

Table 1. Error in Self-Localization along the X-axis [m]IMU. we will also conduct verification on sandy beaches
different from those in this study.
References
1.

2.
3.

Ayumu Tominaga, Akihiro Koubara, Ryusuke Fujisawa,
Eiji Hayashi, Abbe Mowshowitz. Development of lidar
based navigation system for automation of tree harvestingprocess, Proceedings of International Conference on
Artificial Life and Robotics 2021, pp. 469-471
BUNKER-Agilex Robotics
ROS, Open Robotics

Authors Introduction

Table 2 summarizes the errors in self-localization
results in the y-axis direction. Compared to the results inthe x-axis direction, there were significantly larger
discrepancies overall. In every pattern, high-accuracy
self-localization was not achieved. While some input
destinations yielded accurate results, a stable trend was
not observed. Among them, the odometry that integrated
wheel odometry and RGB-D odometry via EKF had thesmallest error.

Mr. Takamasa Hayashi
He received his Bachelor’s degree inEngineering in 2023 from intelligent
and Control Systems, Kyushu Instituteof technology in Japan. He is currently
a Master student in Kyushu Institute ofTechnology and conducts research at
Hayashi Laboratory.

Table 2. Error in Self-Localization along the X-axis [m]Mr. Shintaro Ogawa
He received his Bachelor’s degree inEngineering in 2022 from intelligent
and Control Systems, Kyushu Instituteof technology in Japan. He is currently
a Master student in Kyushu Institute ofTechnology and conducts research at
Hayashi Laboratory.
Mr. Yuto Okawachi
He received his Bachelor’s degree inEngineering in 2023 from intelligent
and Control Systems, Kyushu Instituteof technology in Japan. He is currently
a Master student in Kyushu Institute ofTechnology and conducts research at
Hayashi Laboratory.

3.2. Consideration
Throughout this experiment, RGB-D odometry was
found to have an error and greatly contributed to theimprovement of self-localization accuracy. However, theaccuracy of LiDAR odometry was considerably low incoastal environments like those in this study, lacking
distinct landmarks for reference. The experimental
environment featured a slope descending to the right fromthe travel direction, as depicted in Figure 5, leading tovehicular lateral skidding, which mainly affected y-axis
self-localization. The introduction of an Inertial
Measurement Unit (IMU) is proposed as a solution for
more accurate vehicle posture determination, with priorresearch suggesting its substantial utility in enhancing
results.

Mr. Tan Chi Jie
He received his Master of Creative
Informatics from the Department ofComputer Science and Systems
Engineering, Kyushu Institute ofTechnology, Japan in 2023. He iscurrently a Doctoral student at Kyushu
Institute of Technology and conducts
research at Hayashi Laboratory.

4. Conclusion
In this study, we developed a simulation environment
for an autonomous mobile field robot and conductedaccuracy verification of self-localization. As a result,odometry that integrated wheel odometry and RGB-D
odometry using RGB-D and EKF was found to achieve
more accurate self-localization. In the future, we aim to
improve the accuracy of self-localization by adding an©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
448

Powered by TCPDF (www.tcpdf.org)

Takamasa Hayashi, Shintaro Ogawa, Yuto Okawachi, Tan Chi Jie, Janthori Titan, Ayumu Tominaga, Eiji Hayashi, Satoko Seino

Mr. Janthori Titan
He received his Bachelor ofEngineering in Robotic Engineeringand Automation System from the
Departmentof
Production
Engineering,
FacultyofEngineering,King
Mongkut’s
University of Technology North
Bangkok in 2022. He is currently a
Master student at Kyushu Institute ofTechnology and conducts research at Hayashi Laboratory.

Assist. Prof. Ayumu Tominaga
He is a professor in Department ofCreative Engineering Robotics andMechatronics Course at National
Institute of Technology Kitakyushu
College. He received the Ph.D. (Dr.
Eng.) degree from Kyushu Instituteof Technology in 2021. His researchinterests
includeIntelligent
mechanics, Mechanical systems andPerceptual information processing.
Prof. Eiji Hayashi
He is a professor in the Department
of Intelligent and Control Systems at
Kyushu Institute of Technology. He
received the Ph.D. (Dr. Eng.) degreefrom Waseda University in 1996. His
research interests include Intelligent
mechanics, Mechanical systems andPercentual information processing.
He is a member of The Institute of Electrical andElectronics Engineers (IEEE) and The Japan Society of
Mechanical Engineers (JSME).
Associate. Prof. Satoko Seino
She is an associate professor in
Graduate School of Engineering,
Kyushu university. She received thePh.D.
(Eng.)
from
KyushuUniversity. Her research interests
includecoastalandriverine
environmental conservation.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
449

Development of AR System for Grasping String Foods on Introduction of Industrial Robot
Yoshihiro Koyama

Graduate School of Computer Science and Systems Engineering, Kyushu Institute of Technology,
680-4 Kawazu, Iizuka-city, Fukuoka, 820-8502, Japan
Eiji Hayashi1, Akira Kawaguchi2
1

Kyushu Institute of Technology, 680-4 Kawazu, Iizuka-city, Fukuoka, 820-8502, Japan
Department of Computer Science, The City College of New York of The City University of New York
Email: koyama.yoshihiro272@mail.kyutech.jp, haya@ics.kyutech.ac.jp, akawaguchi@ccny.cuny.edu
2

Abstract
In recent years, the food service industry has been facing a labor shortage. However, the introduction of industrial robots is not easy due
to the high cost of equipment and system integration. Therefore, we are developing an Augmented Reality (AR) application for the
purpose of introducing robots to small and medium-sized companies. By using this application to perform tasks necessary for introducing
robots, such as teaching, cost reductions can be expected when introducing industrial robots. In a previous study, an AR-based graspingand serving simulation system was developed for solidified foods such as fried chicken and rice balls. In this study, we focused on stringshaped food items such as spaghetti and attempted to develop an AR system for grasping and serving a string-shaped object by controllinga gripper.
Keywords: AR, Factory Automation Robots, Unity, ROS, String Foods

Fig. 1 shows the appearance of the robot used in thisstudy. The robot arm UTRA 6-550 and the gripper FLXI
E-Type 2Finger manufactured by UmbraTek were usedin this study. Since this robot is required to perform thesame work as a human in a food factory, a 6-axis
vertically articulated robot with a high degree of freedomof movement was used.

1. Introduction
In recent years, the food service industry has been
facing a labor shortage. However, the introduction ofindustrial robots is not easy due to the high cost ofequipment and system integration. Therefore, we aredeveloping an Augmented Reality (AR) application forthe purpose of introducing robots to small and mediumsized companies. By using this application to performtasks necessary for introducing robots, such as teaching,cost reductions can be expected when introducing
industrial robots.
In a previous study, an AR-based grasping and serving
simulation system was developed for solidified foods
such as fried chicken and rice balls. However, since there
are a variety of foods that can be served in lunchboxes, itis necessary to develop a system that can simulate thegrasping and serving of not only solidified foods but also
other food items.

a.

In this study, we focused on string-shaped food items
such as spaghetti and attempted to develop an AR system
for grasping and serving a string-shaped object bycontrolling a gripper.

UTRA 6-550
b. FLXI E-Type 2 Finger
Fig. 1. Appearance of the robot

2.2. System Configurations
The system in this research is a simulation environmentthat reproduces the control of a gripper on a computer.
The system consists of two main platforms, ROS (Robot
Operating System) and Unity. ROS# [1], a Unity package,
was used for communication between Unity and ROS.

2. Methodology
2.1. Robot Overview©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
450

Yoshihiro Koyama, Eiji Hayashi, Akira Kawaguchi

In addition, AR Foundation was used for AR
application development. AR Foundation is a framework
developed by Unity for AR application development, andby using this platform and software, it is possible todevelop applications for different operating systems
without worrying about program differences. By using
the above platform and software, it is possible to develop
applications for different operating systems withoutworrying about program differences. In this study, weused Google ARCore XR Plug-in, an asset for AR
application development, assuming operation on Android
devices.

The calculation results are then sent to the Joint State
Subscriber, which draws the robot arm and gripper on the
Unity side. In the robot commander, the robot arm andgripper operate sequentially according to the motion
planning shown in Fig. 3.

Obi Rope is a particle-based physics engine based on
the XPBD method that can simulate the physics ofvarious objects.
The system configuration is shown below in Fig. 2. A
comparison of software versions with previous studies isshown in Table. 1.

Fig. 3. Robot Planning(a) Home Pose: Initial pose before motion is started(b) Pre Pick Pose: Preliminary pose before grasping
the object to be grasped(c) Pick Pose; Pose when grasping the object to be
grasped(d) Pre Place Pose: Preliminary posture before
placing the object to be grasped(e) Place Pose: Pose when placing the grasped
objectFig. 2. System configuration

Table 1. Comparison of software versions
Unity
ROS
ROS#
AR Foundation
ARCorePrevious Research2019.4.28f
Melodic
1.7.0
ARCore SDK for Unity
1.24.03. Results and Discussion

This research
2021.3.7f
Noetic1.7
4.2.7
ARCore XR Plugin
4.2.7

3.1. Results of implementation
The results of the implementation of this system aredescribed below. First, we confirmed that the robot arm
and gripper can be controlled by Unity, since the robot
operates in the same way in Gazebo and Unity. Next, thegrasping of the string object is shown in Fig. 4. Fig. 4
shows that the robot was able to lift the string object.
However, it did not reach the point where it was heaped
up, and the string object behaved in an unnatural manner.
Next, communication between Unity and ROS was
confirmed, as the robot control was confirmed, so
communication was possible during Unity execution.

2.3. Robot Accuracy of numerical integration
This section describes the operation of the system inthis study. Fig. 3 shows the motion planning of the robot.
Each posture of the robot is described below. First, thecoordinates of the grasping target and serving position aresent to the robot commander of ROS by Unity's Pose
Publisher. Next, the trajectory of the robot arm is
calculated using MoveIt based on the sent coordinates.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
451

Powered by TCPDF (www.tcpdf.org)

Development of AR System

2.

Position Based Dynamics ， Matthias Müller Bruno
Heidelberger Marcus Hennix John Ratcliff, 2006.

Authors Introduction
Mr. Yoshihiro Koyama
He received his Bachelor’s Degree inEngineering in 2023 from intelligent
and Control Systems, Kyushu
Institute of technology in Japan. He iscurrently a Master student in Kyushu
Institute of Technology and conducts
research at Hayashi Laboratory.

Fig. 4. Grasping
3.2. Considerations

Prof. Eiji Hayashi

First, we describe the grasping of string-like objects.
The Obi Rope asset used to reproduce the string-likeobject is a chain of spheres, and the XPBD [2] method isused to reproduce the string. We believe that the gripper's
pinching motion and the complicated structure of thegripper caused the string object to move in an unnaturalmanner. As a solution to this problem, we consider
designing and introducing a gripper with a simple
structure.

He is a professor in the Department ofIntelligent and Control Systems at
Kyushu Institute of Technology. He
received the Ph.D. (Dr. Eng.) degreefrom Waseda University in 1996. His
research interests include Intelligent
mechanics, Mechanical systems andPercentual information processing.
He is a member of The Institute ofElectrical and Electronics Engineers (IEEE) and The Japan
Society of Mechanical Engineers (JSME).
Prof. Akira Kawaguchi
He is a Professor and Head of theComputer Science Department of The
City College of New York. He has
been working together with Kyushu
Institute of Technology on a uniqueproject to make the future internet
more secure, resilient, and efficientunder the support of the U.S. National
Science Foundation. He received B.S. and M.S. degrees in
Administration Engineering from Keio University and
M.S. and Ph.D. degrees in Computer Science from
Columbia University.

Next, we discuss communication between ROS and
Unity in an AR environment. In this study, we use an
updated version of the software used in the system in theprevious study. We believe that this update has caused a
problem in the conversion to JSON format forcommunication using ROS#. To solve this problem, it isnecessary to check compatibility and find a newcommunication method.
4. Conclusion
In this study, we attempted to develop an AR system forgrasping and serving string-shaped food items, with themain objective of extending the simulation environment
for the application of AR applications for the introductionof industrial robots.
As a result of the implementation of the system in thisstudy, it was confirmed that the gripper can be controlled
and the Obi Rope can draw string-like objects in the AR
environment, in addition to the system in the previous
study. However, the communication between ROS and
Unity in the AR environment and the grasping andplacing of string-like objects could not be confirmed towork properly. In addition, we will continue to develop
an application that combines the system with previous
research.
References1.

ros-sharp, https://github.com/siemens/ros-sharp©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
452

An Image Analysis of Coastal Debris Detection-Detection of microplastics using deep learningYuto Okawachi

Department of Intelligent and Control Systems, Kyushu Institute of Technology, 680-4, Kawazu
Iizuka-city, Fukuoka, 820-8502, Japan
Shintaro Ogaw𝐚𝟏 , Takamasa Hayash𝐢𝟏 , Tan Chi Ji𝐞𝟏 , Janthori Tita𝐧𝟏 , Eiji Hayash𝐢𝟏 , Ayumu Tominag𝐚𝟐 , Satoko Sein𝐨𝟑

Department of Mechanical Information Science and Technology, Kyushu Institute of Technology
680-4, Kawazu, Iizuka-City, Fukuoka, 820-8502, Japan2
Department of Creative Engineering Robotics and Mechatronics Course, National Institute of Technology Kitakyushu
Colllege, 5-20-1 Shii, Kokuraminamiku, Kitakyushu, Fukuoka, 802-0985, Japan
3
Graduate School of Engineering, Kyushu University 744 Motooka, Nishiku, Fukuokacity, Fukuoka, 819-0395, Japan1

E-mail: ookawachi.yuuto693@mail.kyutech.jp,
tominaga@kct.ac.jp, haya@mse.kyutech.ac.jp,
http://www.kyutech.ac.jp/
Abstract
To address the issue of litter drifting ashore, this study developed a deep learning-based microplastic detection system.
The system employed yolov7 [1] as its deep learning network, complemented by SAHI (Slicing Aided Hyper
Inference) [2] as an additional vision library. yolov7 is renowned for its efficacy in real-time object detection. Our
experimental framework involved four tests, utilizing two variations of yolov7 - the standard model and yolov7-e6e- in conjunction with SAHI. The effectiveness of each test was quantified using metrics such as Intersection overUnion (IoU), Precision, Recall, F-measure, and Detection Time in seconds. For our dataset, we gathered images from
actual cleanup locations, such as Hokuto Mizukumi Park. The model's discriminator underwent 700 training iterations,with a learning rate set at 0.001. Experimental results showed that it detects fairly small microplastics.
Keywords: Deep learning, Object detection, YOLOv7

surveys there. To solve the problem of beached waste, awide range of data is needed. However, the regular andsystematic collection of data on the waste in various
places is challenging with the present means of survey.

1. Introduction
Drifted litter has become widespread throughout Japan.
In many regions, the composition of this litter, based onvolume, weight, and count, is predominantly artificial
rather than natural. Plastic litter does not decompose
naturally and, due to the influence of ultraviolet rays andwaves, breaks down into microplastics that drift in theocean. They are then washed up on the sandy beaches.
Microplastics, which may contain harmful substancesfrom the time of product manufacturing or accumulate
other toxicants, such as polychlorinated biphenyls, during
their drift, can be ingested by marine life, potentiallycausing adverse effects. These microplastics, with a sizeof less than 5mm, make them difficult to visually detecton the sand.

In this study, we aimed to realize labor-saving inbeached waste surveys by using deep learning for imagebased waste detection. Particularly, we focused on thechallenge of detecting microplastics, extremely small
items, previously undetected in other studies. To achievethis, we used deep learning to create a dataset with coastal
images and utilized it for both training the detection
model and validating its effectiveness. Fig.1 presents animage sample from the dataset. This research was
conducted with a specific focus on a single-detection
model, solely for microplastics.

A significant amount of waste washes up on the coasts ofJapan. Among this, small items like microplastics, often
buried in the sand, are difficult to detect by the current
survey methods. Furthermore, some coastal areas, which
are undeveloped or have wave-dissipating blocks, aredifficult to access, making it impossible to undertake©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
453

Yuto Okawachi , Shintaro Ogawa , Takamasa Hayashi , Tan Chi Jie , Janthori Titan , Eiji Hayashi , Ayumu Tominaga , Satoko Seino

Fig.3 ELAN, E-ELAN Structure
2.2. SAHI(Slicing Aided Hyper Inference)
SAHI is a lightweight vision library for large-scale
object detection and instance segmentation. As illustrated
in Fig.4, it divides the input image into small rectangles.
The model then performs an evaluation on eachsegmented image and finally merges and outputs thedetection results. While SAHI can be applied to any deeplearning model in principle, it has the limitation that theestimation time increases linearly, as it conducts the task
on each of the segmented images.

Fig.1 Examples Dataset
2. Methodology
2.1. Composition of YOLOv7
In this study, we employed the deep learning model
known as YOLOv7. YOLOv7 is an object detection
algorithm that operates at a higher speed than the existing
YOLO series. As shown in Fig.2 on the MS COCO
dataset, it achieves an AP value that significantly
surpasses the existing YOLO series. There are several
models within YOLOv7, and for our research, we used
the base model, YOLOv7 (plain), and YOLOv7-E6E.
Each model has a three-stage pyramid structure,generating multiple feature maps through repeatedupsampling and downsampling. Ultimately, three ofthese feature maps are used for estimation. The basemodel, YOLOv7 (plain), uses a network composed ofmultiple convolutional layers known as ELAN.
Additionally, YOLOv7-E6E employs an expanded
version of the ELAN network, called E-ELAN. The
structures of ELAN and E-ELAN are shown in Fig.3

Fig.4 Image segmentation image using SAHI
2.3. Dataset creation
The dataset used in this study was created using
photographs taken at real-world beach cleanup locations,
including Hokuto Miukumi Park. The main subject of thephotographs was microplastics found on the sand. To
enable the identification of these microplastics from otherbeach debris like shells and stones, we also included
images of only shells and stones. The only class created
was for microplastics. A total of 534 photographs weretaken, and the number of images for training and final test
datasets was tripled to 1,376 by using a brightness filter.
These were then divided into 977 for training, 291 forfinal test dataset, and 108 for the general test dataset.

Fig.2 Comparison of YOLOv7 with other real-time
object detectors

2.4. Identifier creation and evaluation methods
The identifier was trained 700 times using the createddataset. The study used models that were optimized to©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
454

An Image Analysis ofachieve the highest weighted sum of Average Precision(AP) and mean Average Precision (mAP) at a 1:9 ratio
during the training. Experiments were conducted using
two different types of YOLOv7 (plain) and YOLOv7E6E, as well as SAHI. The evaluation of the detector's
effectiveness was based on the range of the IoU value
being over 0.65, and the results were compared in terms
of precision, recall, F1-score, and time of estimation. The
F1-score, representing the harmonic mean of precisionand recall, and the time of estimation, which is the time
taken for the model to read the input image and produce
an output, were the main points of performance analysis.
The formulas for precision, recall, and F-score are shownin equations (1) to (4). Additionally, the confusion matrix
is presented in Table1
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =𝐼𝑜𝑈 =2 ∗ 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 ∗ 𝑅𝑒𝑐𝑎𝑙𝑙
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙

(2)

(4)

Table1. Confusion Matrix
Predictions
Positive
Actual Results

Negative

Positive

True Positive False Negative

Negative

False Positive True Negative

RecallF-scoreTime of estimation

0.771

0.679

0.722

0.157

Yolov7 + SAHI

0.801

0.671

0.730

1.63

YOLOv7-E6E

0.836

0.571

0.679

0.216

YOLOv7-E6E + SAHI

0.832

0.765

0.753

2.92

The detection results demonstrate that the detector canaccurately identify microplastics, indicating its practical
viability. However, as indicated in Table2, high precisionand recall rates were not achieved. A potential reason forthis could be the dataset's quality. The annotations for thedataset were manually done, and objects not visually
identifiable as trash were not labeled. The results showthat the detector can identify microplastics that were too
small to be labeled, leading to a lower precision rate.
Regarding recall, the failure to detect microplastics
resembling stones or shells in color resulted in lower
accuracy. The dataset images were captured with an
iPhone at resolutions like 3024 × 4032. To minimize thePC's load in the experiment, the input images for themodel were resized to 1280 × 1280, making smaller
objects even harder to recognize. Addressing these issues
could involve labeling even smaller objects andincreasing training data for items like stones and shells.
Adjusting hyperparameters such as batch size might alsoenhance accuracy. The increased estimation time due to
SAHI's use could be mitigated with software like
TensorRT [3], which accelerates deep learning inference.

(3)

𝐴𝑟𝑒𝑎 𝑜𝑓 𝐼𝑛𝑡𝑒𝑟𝑠𝑒𝑐𝑡𝑖𝑜𝑛
𝐴𝑟𝑒𝑎 𝑜𝑓 𝑈𝑛𝑖𝑜𝑛

Precision

Yolov7

As observed in Fig.5, the detector can identify even verysmall microplastics. It also effectively prevents false
detections of stones with similar shapes. Using SAHI
with both models resulted in improved accuracy acrossall metrics except for estimation time. Specificallyfocusing on the YOLOv7-E6E model, the use of SAHI
significantly increased the recall rate by nearly 0.2,
demonstrating that the model becomes more robust in
detecting microplastics with SAHI. However, the use of
SAHI resulted in an estimation time that wasapproximately ten times longer.

𝑇𝑟𝑢𝑒𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒(1)
𝑇𝑟𝑢𝑒𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 + 𝐹𝑎𝑙𝑠𝑒𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒

𝑇𝑟𝑢𝑒𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒
𝑅𝑒𝑐𝑎𝑙𝑙 =𝑇𝑟𝑢𝑒𝑃𝑜𝑠𝑖𝑡𝑖𝑣𝑒 + 𝐹𝑎𝑙𝑠𝑒𝑁𝑒𝑔𝑎𝑡𝑖𝑣𝑒𝐹 − 𝑠𝑐𝑜𝑟𝑒 =Table2. Score per model combination
Model

3. Results and Discussion

4. Conclusion
In a validation experiment, we developed a detector using
YOLOv7 to identify microplastics and evaluated its
accuracy. Accuracy was assessed using four metrics:
precision, recall, F1-score, and estimation time. The
results suggest that the detector achieves sufficiently
accurate detection for practical use. It is suggested thatmodifying the annotation method and adjusting
hyperparameters could lead to the development of a moreaccurate model.
Acknowledgements
This research was conducted at the Laboratory ofProfessor Hideharu Hayashi, Department of Intelligent
Systems Engineering, Faculty of Information
Engineering, Kyushu Institute of Technology. I am

Fig.5 Example of detection results©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
455

Powered by TCPDF (www.tcpdf.org)

Yuto Okawachi , Shintaro Ogawa , Takamasa Hayashi , Tan Chi Jie , Janthori Titan , Eiji Hayashi , Ayumu Tominaga , Satoko Seino

Mr. Tan Chi Jie

deeply grateful to Professor Hideharu Hayashi for his
valuable opinions and guidance throughout the research.
My heartfelt thanks also go to all members of the Hayashi
Laboratory for their guidance in my daily research
activities.

He received his Master of Creative
Informatics from the Department ofComputer Science and Systems
Engineering, Kyushu Institute ofTechnology, Japan in 2023. He iscurrently a Doctoral student at
Kyushu Institute of Technology andconducts research at Hayashi
Laboratory.

References
1.

2.

3.

Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan
Mark Liao, YOLOv7:Trainable bag-of-freebies sets newstate-of-the-art
forreal-time
objectdetectors
arXiv:2206.14651,2022.3,9,10
Akyon, Fatih Cagatay and Altinuc, Sinan Onur andTemizel, and Alptekin， Slicing Aided Hyper Inference
and Fine-tuning for Small Object Detection 2022 IEEE
International Conference on Image Processing(ICIP),2022.966- 970
NVIDIA Tensort.

Mr. Janthori Titan
He received his Bachelor ofEngineering in Robotic Engineeringand Automation System from the
Departmentof
Production
Engineering, Faculty of Engineering,
King Mongkut’s University ofTechnology North Bangkok in 2022.
He is currently a Master student at
Kyushu Institute of Technology andconducts research at Hayashi Laboratory.

Authors Introduction

Assist Prof. Ayumu Tominaga

Yuto Okawachi
He received his Bachelor’s degree inEngineering in 2023 from intelligent
and Control Systems, Kyushu Instituteof technology in Japan. He is currently
a Master student in Kyushu Institute ofTechnology and conducts research at
Hayashi Laboratory.

He is a professor in Department ofCreative Engineering Robotics andMechatronics Course at National
Institute of Technology Kitakyushu
College. He received the Ph.D. (Dr.
Eng.) degree from Kyushu Institute ofTechnology in 2021. His researchinterests
includeIntelligent
mechanics, Mechanical systems and Perceptual
information processing.

Mr. Shintaro Ogawa
He received his Bachelor’s degree inEngineering in 2022 from intelligent
and Control Systems, Kyushu
Institute of technology in Japan. He iscurrently a Master student in Kyushu
Institute of Technology and conducts
research at Hayashi Laboratory.

Prof. Eiji Hayashi
He is a professor in the Department ofIntelligent and Control Systems at
Kyushu Institute of Technology. He
received the Ph.D. (Dr. Eng.) degreefrom Waseda University in 1996. His
research interests include Intelligent
mechanics, Mechanical systems andPercentual information processing.
He is a member of The Institute of Electrical andElectronics Engineers (IEEE) and The Japan Society of
Mechanical Engineers (JSME).

Mr. Takamasa Hayashi
He received his Bachelor’s degree inEngineering in 2023 from intelligent
and Control Systems, Kyushu Instituteof technology in Japan. He is currently
a Master student in Kyushu Institute ofTechnology and conducts research at
Hayashi Laboratory.

Associate. Prof. Satoko Seino
She is an associate professor in
Graduate School of Engineering,
Kyushu university. She received thePh.D.
(Eng.)
from
KyushuUniversity. Her research interests
includecoastalandriverine
environmental conservation.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
456

A High-Speed Estimation Method of Parameters in Impulse Response
Toshiki Tanaka, Ivan Tanev

Doshisha University, 1-3 Tatara Miyakodani, Kyotanabe, Kyoto 610-0394, Japan
Tetsuo Hattori,

Kagawa University, 1-1 Saiwai-cho, Takamatsu, Kagawa 760-0016, Japan
Email: hattori@pe.kagawa-u.ac.jp

Abstract
This paper proposes a high-speed parameters estimation method for compartment model where output function is
described by the convolution between input function and impulse response, which is like a time-invariant linear
system. The proposed method uses linear regression analysis based on the equivalently transformed equation that canbe obtained using the processing of Differentiation of Convolution with Exponential function (DCE). In this paper,taking the parameters estimation problem of PET (Positron Emission Tomography) inspection system and RLC series
electrical circuit for examples, we show that the method can estimate parameters of those impulse responses in highspeed.
Keywords: Parameter estimation, Compartment model, Cumulative function, Linear regression analysis, PET
Inspection

observed by arterial blood sampling from the patient after
intravenous injection of FDG (18F-fluorodeoxyglucose)tracer. The Ce(t) means the tracer concentration before
metabolism in tissue, and Cm(t) is the tracer concentration
after metabolism. Ci(t) is measured by PET camera as thesum of Ce(t) and Cm(t). The flow parameters K1, k2, k3, k4([ml ・ g-1 ・ min-1] or [min-1]) show the transport andbinding rates of the tracer between two compartments.

1. Introduction
Compartment model has been used in the various fields
such as pharmacokinetics, chemical reaction system,
environmental diffusion, and electrical circuit, etc.
Especially, the estimation of parameters in PET (Positron
Emission Tomography) inspection is very important inpractice ([1], [2], [3], [4], [5], [6], [7], [8]).

Ci (t )k3

K1

In this paper, we propose a high-speed parameters
estimation method for compartment model where output
function is described by the convolution between input
function and impulse response, which is like a timeinvariant linear system ([9], [10], [11], [12]). The
proposed method uses linear regression analysis based onan equivalent equation that can be obtained from DCE(Differentiation of Convolution with Exponentialfunction).
Moreover, taking the parameters estimation problem of
PET inspection system and RLC series electrical circuit
for examples, we concretely describe that the method canestimate parameters of those impulse responses in highspeed.

C p (t )Cm (t )Ce (t )

k2

k4

Fig.1. 3-Compartment model of tissue in FDG PET
inspection [2], [4] with four tracer’s flow parameters K1,
k2, k3, k4.
 Ci ( t ) = Ce ( t ) + Cm ( t ) dCe ( t )= K1C p ( t ) − ( k2 + k3 ) Ce ( t ) + k4Cm ( t ) dt
 dCm ( t )= k3Ce ( t ) − k4Cm ( t ) dt

2. Proposed Estimation Method
2.1 Impulse Response of PET Compartment Model
The compartment model and parameters used in the
PET inspection is shown in Fig. 1. And the simultaneous
differential equations of the 3-comparement model isshown in Eq. (1). The Cp(t) means the tracer’sradioactivity concentration in plasma that can be directly(1)

In this model, the input function is Cp(t), and the output
is Ci(t). Let  denote the convolution, and let g(t) be theimpulse response of the compartment model described by
Fig. 1 and Eq. (1). Then the output function Ci(t)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
457

Toshiki Tanaka, Ivan Tanev, Tetsuo Hattori

including the flow parameters is described by thefollowing Eq. (2) and Eq. (3) ([2], [4]).

Ci ( t ) = g (t )  C p (t ) =  g (t −  )C p ( ) dIntegrating both sides of Eq. (7) twice with respect totime t, we have the following Eq. (8) using cumulative
functions such as

t

(2)

0

r (t ) = AC pII (t ) + BC pI (t ) + Cr I (t ) + Dr II (t ) (8)

where

where

K1

− t
−t
 g (t ) =  −  ( k3 + k4 −  ) e + (  − k3 − k4 ) e (3) ,  = 1 ( k + k + k ) ( k + k + k )2 − 4k k
2
3
4
2
3
4
2 4

2



C pI ( t ) C ( ) d , C (t )  C ( ) dr ( t )  r ( ) d , r ( t )  r ( ) d



I

t

0

t

t

II
p

p

II

0

0

t

I
p

0

 A = K1 ( k3 + k4 ) , B = K1
C = − ( k2 + k3 + k4 ) , D = −k2 k4

In this PET inspection, the problem is to estimate thevalues of flow parameters K1, k2, k3, k4 in the impulse
response g(t) from the two observed time series data,Cp(t) and Ci(t), where Ci(t) is normally measured as anoisy time series data r(t) (=Ci(t)+noise) by PET Camera.

(9)

I

(10)

Eq. (8) can be regarded as a linear regression equation
with objective variable r(t) and four explanatory
variables such as C pII (t ), C pI (t ), r I (t ), r II (t ) . Since the timeseries data r(t) and Cp(t) are given by PET Camera andblood sampling respectively, it is possible to obtain thevalues of parameters A, B, C, and D of Eq. (8), by linear
regression analysis ([10], [11]). Then we can estimate theparameters K1, k2, k3, and k4 from Eq. (11).
This is the main idea of the proposed method, that is, wetransform the simultaneous differential equations that
describes the system operating characteristics into an
equivalent equation that is convenient for linear
regression analysis.

2.2 Estimation Using Linear Regression Analysis
and DCE.
We equivalently transform the Eq. (1) to a linear
regression model (LRM) equation using cumulative
function, based on the following Theorem 1 ([9]) andTheorem 2 of DCE [12] that is derived from Theorem 1.
Theorem 1 [9]:Let f(x,t) and ∂f(x,t)/∂t be a continuous function andits continuous partial derivative function over the closed
section [a,b], respectively. And let the a and b are thefunctions a(t) and b(t) with respect to t, respectively.
Then, Eq. (4) holds.

2.3 Weighted LRM-DCE Method.
Generally, in many cases, it is expected that noise will
be mixed into the parameter estimation. For suchoccasion, we propose the Weighted LRM-DCE methodthat multiplies a known function w(t) to the both sides ofEq. (8) as weight function, as shown in Eq. (11).

b( t ) d b( t )f ( x, t )dx = f ( x, t )dx

a
t
a
(
)
( t ) tdt
db ( t )da ( t )+ f (b (t ) , t )− f ( a (t ) , t )
dt
dt

(4)
Theorem 2 (DCE: Differentiation of Convolution with
Exponential function) [12]:Let J(t) be the convolution between the functions
exp(-kt) and differentiable continuous function C(t) as
shown in Eq. (5), then from Theorem 1, Eq. (6) holds.

r (t )w(t ) = AC pII (t )w(t ) + BC pI (t )w(t ) + Cr I (t )w(t ) + Dr II (t )w(t )(11)
To determine the parameters A, B, C, and D in Eq. (9) is
equivalent to obtain the same parameters in Eq. (8). Thisweighting function w(t) has to be appropriately selectedaccording to the noise characteristics.
For example, as the weighting function w(t), we proposer(t)-1 that is the inverse of the observed r(t) in the region
where r(t)>0. Then we have the following Eq. (12).

dJ (t )
(5)
 J (t ) = (−kt ) J (t ) + C(t )J (t ) dte− kt  C (t ) = e− k (t − )C ( )d



0

(6)

1 = AC pII (t )r (t )−1 + BC pI (t )r (t )−1 + Cr I (t )r (t )−1 + Dr II (t )r (t )−1

Let r(t) be the measured value by PET camera at time t.
The r(t) normally includes the additive noise n(t), sor(t)=Ci(t) + n(t). However, for the simplicity, we simply
assume the case where the noise is not included in themeasured value, then r(t)=Ci(t).
Then, from Theorem 2, we obtain the Eq. (7).

(12)
3. Applied Experimentation
3.1 Programming Environment for Experiments
The programming environment for numericalcomputation used in the experimentation are as follows.
CPU: Intel Core i7-3770 (3.40GHz), RAM: 16GB,
OS: Windows 10 Professional (64bit),
Programming Language: Borland C++ 5.51 (32bit).

r ( t ) = ( k3 + k4 ) K1C p ( t ) + K1Cp ( t )( k, 2r''+(tk) 3= +d 2kr4(t)) r/ dt (2t,)C−p k(t2)k=4rdC(tp )(t ) / dt.(7)r' (t ) = dr (t )−/ dt

where©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
458

A High-Speed Estimation Method

As for the processing time for both noisy and noiselessr(t) by WLRM and LRM in the whole-time interval, thecomputing time of both method is less than 80 [ms].

3.2 Experiments for PET compartment Model
We have experimented for an input data Cp(t) (Fig. 2 (a))
that is generated based on the literature [6] and noiselessoutput data r(t) (Fig. 2 (b)) generated from the parameters(K1, k2, k3, k4) = (0.200, 0.130, 0.060, 0.007) using Eq. (2)and Eq. (3). Moreover, we also experimented for r(t) (Fig.
3 (a)) where the noise n(t)~N(0,1002) is added to theaforementioned noiseless r(t). In this noisy case, as amatter of fact, we used the smoothed r(t) (Fig.3 (b))
where moving average processing is done twice over therange of ±4 points before and after each time point of thenoisy r(t).
For the evaluation of estimation accuracy, we define therelative error as ( |(Estimated Value)-(True Value)| /(True Value))× 100[%].

(a) LRM for noiseless r(t).

A. Noiseless Case(a) Cp(t)(b) noiseless r(t)Fig.2. Generated input Cp(t) and noiseless r(t).

(b) WLRM for noiseless r(t).

B. Noisy Case(a) noisy r(t)(b) smoothed r(t)(c) WLRM and LRM for nosy r(t).

Fig.3. Generated noisy r(t) and smoothed r(t).

Fig. 4. Accuracy of the parameter estimation for K1, k2,
k3, and k4 by the methods of LRM and WLRM ([12]).

C. Results
We briefly call the LRM-DCE method and Weighted
LRM-DCE method as LRM and WLRM, respectively.
The estimated results of (K1, k2, k3, k4) by both LRM and
WLRM are shown in Fig. 4 for each parameter in eachtime interval.
Fig.4 (a) shows that the relative error rate (RER) of
parameters estimation by LRM becomes less than 0.4 [%]for noiseless r(t), if we take the time series data of 660
seconds (11 minutes) or longer for estimation computing.
Fig. 4(b) shows that the resultant (RER) by WLRM is lessthan 0.46[%] until 1800[sec]. Even in the whole-time
interval (=3600[sec]), the RER is less than 0.55[%]. Fig.
4(c) illustrates the resultant RER for noisy r(t) by bothWLRM and LRM, where WLRM shows that the rate isless than 3.3 [%] for all of four parameters in the wholetime interval, however the RER of LRM is worse than
WLRM.

3.3 Experiments for RLC Series Electrical Circuit
A. Equation and the Impulse Response
RLC series electrical circuit (RLC-EC) is one of most
simple compartment model，that consists of three basic
passive elements connected in series: a resistor R, an
inductor L, and a capacitor C. Eq. (13) shows the systemequation of RLC-EC with DC power supply (DC-PS) E(Fig. 5(a)), where we can regard it as the form of LRM
itself without DCE processing.

Ldi ( t )1 t+ Ri ( t ) +  i ( ) d = E
dt
C 0

(13)

Then, we have the following Eq. (14) for parameters
estimation, using cumulative functions.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
459

Toshiki Tanaka, Ivan Tanev, Tetsuo Hattori

Li (t ) + Ri I (t ) +where

i I ( t )i II ( t )(ii) Critical damping case: (R, L, C) = (10, 2.5, 0.1)
ER: (10.00010, 2.50001, 0.10000), RER < 0.01%.
(iii) Over-damping case: (R, L, C) = (10, 2.5, 0.2)
ER: (10.00000, 2.50003, 0.19999), RER <0.01%.

t
1 II
i (t ) =  Edt
0C

 i ( ) d i ( ) d =   i ( )d d
t

0

t

t

I

0

0 0

4. Conclusion(14)

In this paper, we have proposed a method forparameters estimation of impulse response incompartment model systems. The proposed idea is basedon the equivalent transformation from the simultaneous
differential equations of the system behavior into an
equation that is suitable for the estimation by linear
regression analysis.
For further study, we are going to experiment for more
complicated circuits and pursue the noise cancelation
method.

Considering the RLC series circuit as a system ofcompartment model, the parameters of impulse response
are those values of R[Ω], L[H], C[F], themselves. And,
there are three types of impulse responses, depending on
the magnitude relationship between (R/2L)2and 1/LC.
Let those impulse responses be g1(t), g2(t), g3(t),depending on the cases, (R/2L)2 < 1/LC, (R/2L)2 = 1/LC,(R/2L)2 > 1/LC, respectively.
(i) Oscillatory case: (R/2L)2 < 1/LC
g1 (t ) =

  1   R 2 
 R 
texp  − t   sin  −2  LC   2L  
 2L 
 L R − C  2 
1

References
Godfrey, K.: “Compartmental Models and Their
Application”, Academic Press (1983).
2. Hideo Onishi, Masanori Matsumoto, and Kazutaka
Masuda ed.: “Nuclear Medicine Testing Technology",
supervised by The Japanese Society of RadiologicalTechnology, Ohmsha (2002). (in Japanese)
3. Kinichi Hisada supervised, Norihisa Tonami, Atsushi
Kubo ed.: "Latest Clinical Nuclear Medicine", A New
Textbook of Clinical Nuclear Medicine (3rd edition),
Kanehara Shuppan Co., Ltd. (2005). (in Japanese)
4. R. P. Maguire and K. L. Leenders, ed.: “PET
Pharmacokinetic Course Manual”, International Society of
Cerebral Blood Flow and Metabolism, Summer School
2007. (2007)
5. Hiroshi WATABE, Yoko IKOMA, Yuichi KIMURA,
Mika NAGANAWA, and Miho SHIDAHARA: “PET
kinetic analysis—compartmental model”, Annals of
Nuclear Medicine, Vol. 20, No. 9, 583–588, (2006).
6. Kudomi N, Choi E, Yamamoto S, Watabe H, Kim K,
Shidahara M, et al.:” Development of a GSO detector
assembly for a continuous blood sampling system”, IEEE
Trans. Nuclear Science, Vol. 50, No.1, pp.70–73 (2003).
7. Naganawa M, Kimura Y, Ishii K, Oda K, Ishiwata K,
Matani A, “Extraction of a plasma time-activity curvefrom dynamic brain PET images based on independent
component analysis”, IEEE Trans. Biomedical
Engineering, Vol. 52, No.2, pp.201–210. 2005.
8. Hiroshi Hirayama, Tatsuo Otsuki: “Electrical Circuit
Theory (3rd Edition Revised)”, Institute of ElectricalEngineers of Japan (2008) (in Japanese)
9. Shigeichi Moriguchi, Kanehisa Udagawa, Shin
Hitotsumatsu: “Iwanami Mathematics Formula (I)”,
Iwanami Shoten (1987). (in Japanese)
10. Hirokazu Anai: “Practical Guide of Mathematical
Optimization”, Kodansha, ISBN978-4-06-156510-4(2016). (in Japanese)
11. William H. Press, Brian P. Flannery, Saul A. Teukolsky,
William T. Vetterling: "Numerical Recipes in C",
Cambridge University Press (1988).
12. Toshiki Tanaka, Tetsuo Hattori, Yusuke Kawakami, Yo
Horikawa, Yoshiro Imai: "Parameters Estimation of
Impulse Response in Compartment Model Using
Cumulative Function and Linear Regression Analysis",
IEEJ Transactions on Electronics, Information and
Systems, Vol.143 No.2, pp.185-191, 2023.
1.

(ii) Critical damping case: (R/2L)2 = 1/LC

1
 R 
g2 (t ) = t  exp  − t L 2L 
(iii)Over-damping case: (R/2L)2 > 1/LC
g3 (t ) =  R 2  1  
 R texp  − t   sinh    − 2
  2L   LC  
 2L 
R  L  −  2  C1

Fig. 5 shows the graphs of RLC circuit with DC-PS E andcumulative functions of i(t) in the case of oscillatory case.

t

(a)

(b)

t

t

(c)

(d)

Fig.5. (a) RLC series electrical circuit with DC-PS E.
(b) The i(t) in the oscillatory case. (c) iI (t). (d) iII (t).
B. Results of RLC parameters estimation
We have experimented for the parameters estimation,from generated time series data of 3 seconds current i(t)by taking 0.003[s] increments, that consists of 1001
pieces from time t=0 to t=3.
Estimated results (ER) and RER for each of above threecases with E=5[V] are as in the followings.
(i) Oscillatory case: (R, L, C) = (10, 2.5, 0.005)
ER: (10.00120, 2.49988, 0.00499), RER < 0.2%.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
460

Powered by TCPDF (www.tcpdf.org)

A High-Speed Estimation Method

Authors Introduction
Mr. Toshiki Tanaka
He received the Bachelor’s degree inElectronics & Information Engineeringfrom Kagawa University, Japan, in 2015.
In the same year, he joined Kinkei
System Co., Ltd. Currently, he is a
doctoral student at Doshisha University,
Japan, while working for the company.
His research interests include circuitdesign, numerical simulation, andparameter estimation, etc. He is a member of IEEJ.
Dr. Ivan Tanev
He received the Ph.D. degree in
Computer Engineering from SaintPetersburg
State
Electrotechnical
University, Russia, in 1993. He also
received the degree of Dr. Eng. from
Muroran Institute of Technology, Japan,
in 2001. Currently, he is a Professor at
Doshisha University, Japan. His
research interests include Evolutionary
Computations, Evolutionary Robotics, etc.
Dr. Tetsuo Hattori
He received the B.E. and M.E. degrees
in Electrical Engineering from
Doshisha University, and the Ph.D.
degree in Electronics & Information
Engineering from Osaka University,
Japan. After he worked for ToshibaEng. Co., Ltd., he had been with
Kagawa University from 1988 to 2015.
Currently, he is a Professor Emeritus at
Kagawa University. Member of IEEJ and IEEE.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
461

A Consideration on Amplification Function in BJT Evers-Moll Model and PTT (I)---- V-I Characteristics ---Shimon Hattori, Osamu Matoba

Kobe University, 1-1 Rokkodai-cho, Nada-ku, Kobe City 675-8061, Japan
Toshiki Tanaka

Kinkei System Corporation, 8-2-61 Nanko-Higashi, Suminoe-ku, Osaka 559-0031, Japan
Yusuke Kawakami

NIT Kagawa College, 355 Chokushi-cho, Takamatsu, Kagawa 761-8058, Japan
Tetsuo Hattori

Kagawa University, 1-1 Saiwai-cho, Takamatsu, Kagawa 760-0016, Japanhattori@pe.kagawa-u.ac.jp
Abstract
In 1989, the notion of Photon Transport Transistor (PTT) has been proposed by B.J. Van Zeghbroeck et al., at IBM
Research Center at that time. PTT is an optical coupling device of Light Emitting Diode (LED) and light receiving
diode (Photo Diode, PD) where the carrier of the base layer is light (Photon) only. Later in 1996, W. N. Cheung andPaul J. Edwards have shown that PTT can be a very low noise amplifier in a positive feedback circuit, based ontheoretical calculations of the noise figure by Quantum Mechanics. In this paper we consider the amplification
principle noticing the similarity of the VI characteristics between the Bipolar Junction Transistor (BJT) and PTT.
Keywords, PTT, LED, PD, BJT, positive feedback circuit, Ebers-Moll model, VI characteristics

For this discussion, firstly, we show the similarity of
the VI characteristics equation of BJT Ebers-Moll
Model (BJT EMM) and PTT in the emitter common (orgrounded) circuit as a positive feedback circuit. This isbecause, when using the BJT transistor for its
amplification function, a common emitter circuit isgenerally used.
And secondary, we show the circuit equations in orderto analyze how a fixed bias PTT circuit achieves DC
current amplification function.

1. Introduction
As is well known, the Bipolar Junction Transistor(BJT) has been invented in 1948. The transistor has astructure in which two diodes, each made of a P-type
and an N-type semiconductor, are further joined at thebase layer.
On the other hand, B. J. Van Zeghbroeck et al. at IBM
Research Laboratories presented the notion of a Photon
Transport Transistor (PTT) based on optical coupling
between a Light Emitting Diode (LED) and a Photo
Diode (PD) [1]. Later in 1996, W. N. Cheung and Paul J.
Edwards have shown that PTT can be a very low noise
amplifier in a positive feedback circuit, based ontheoretical calculations of the noise figure by Quantum
Mechanics [2].
After that, it has been reported that, using the PTT
that consists of high-brightness LED and PD ([3], [4],[5]). audio amplifier can be developed and that the PTT
expresses not only an amplification function but also aswitching function similar to a thyristor ([6], [7], [8]).
In this paper, we discuss the amplification function of
PTT in comparison with the conventional BJT.

2. BJT EMM and PTT
2.1. Circuit and Characteristic Equation
The BJT emitter common (or grounded) circuit and its
BJT EMM are illustrated in Fig. 1, Fig. 2, respectively.
Furthermore, a PTT emitter common circuit for small
signal amplification is shown in Fig. 3. In this section,
we show substantial correspondence between the two
devices of BJT EMM and PTT in their emitter common
circuits, by representing the VI characteristics of theirdevices.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
462

Shimon Hattori, Osamu Matoba, Toshiki Tanaka, Yusuke Kawakami, Tetsuo HattoriFig. 3 Fixed bias PTT emitter common circuit forsmall signal voltage amplification.

Fig. 1 BJT emitter common circuit for small signal
voltage amplification.

The approximate VI characteristic equations for PD
part and LED part of the PTT circuit in Fig. 3 are shownin Eq. (3) and Eq. (4), respectively. The meaning ofeach symbol is shown in Table 2.
We can point out that the pair of Eq. (3) and Eq. (4) iscorresponding to that of Eq. (1) and Eq. (2) as thesimilarity between the BJT EMM and PTT. The
difference of the two sets of pair equations is as follows.

IC =   I E − IC 0 ( exp ( cPDVBC ) − 1) , cPD =q
m2kTPD(3)

I E = I E 0 ( exp ( cLEDVBE ) − 1) , cLED =q
m1kTLED(4)

Fig. 2 BJT emitter common circuit that corresponds
to the where IE = IB+IC, based on Ebers-Moll model.
The VI characteristics equations of two diode parts in
Fig. 2 are shown as Eq. (1) and Eq. (2). The eachmeaning of symbols such as IC0, IE0, q, k, T is shown in
Table 1
 qV  
IC = 0 I E − IC 0  exp  BC  − 1 kT   qV  
I E = I E 0  exp  BE  − 1 kT  Table 2 The symbol list of the VI characteristic equation of
PTT circuit.

(1)

(2)
The Eq. (3) is an approximated VI characteristic
equation of PD, based on the Eq. (5) that can beobtained from the PD equivalent circuit as shown in Fig.
4 ([4], [5]).

Table 1 The symbol list of the VI characteristic equation of
the two diode parts in BJT Ebers-Moll Model.


qVD 

 IO = I L − I S  exp kT − 1 − I 'VD = VBC − IO RS
V −I R I ' = BC O SRsh©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
463

(5)

A Consideration on Amplification Function

be expressed as IL = γ*IE, where γ* is the proportionality
constant. Then, we obtain the Eq. (3) as the result of
introducing a correction term into the approximate
equation derived from Eq. (5).
2.2. Similarity and Difference in Diode Equation
The similarity points between BJT and PTT: Bothdevices are composed of two diode parts that arecollector-side diode and emitter-side one, and while thecollector-side diode is in a state where the current IC
flows in the opposite direction of the diode and includes
the generated current as proportional to the current IE ofthe emitter-side diode.
The differences in the equations for the two diodes in
BJT and PTT: Absolute temperature T of two diodes issame in BJT EMM, while it is different in PTT. And the
Ideality factors are 1 in BJT EMM, while they are
different in PTT.

IL：Generated current by incident light,
ID ： Current of diode, Cj ： Junction capacitance,
Rsh：Shunt resistance, RS：Series resistance,
I’：Current of Rsh, VD：Voltage of diode,
IO：Output current, VO：Output voltage.

Fig. 4. Equivalent circuit of PD ([4], [5]), where
the upper part corresponds to Base side and thelower Collector side.
The internal resistance Rs in Eq. (5) and in Fig. 4 isapproximated as 0 because it is a small resistance, and
VD = VBC = -VCB. The resistance Rsh is a large
resistance, so that I' is approximated as zero.
Furthermore, IL corresponds to Isc in Fig. 5 andrepresents the photocurrent proportional to the amountof light incident on the PD (amount of light received by
the PD).

2.3. Direct Current Amplification from Equation of
PTT Circuit
Under the assumption that the input signal voltage vi=0in Fig. 3, applying the Kirchhoff's law to the closed
circuit 1 (V→A→B→E→V) and closed circuit 2(A→B→C→A) in the PTT circuit of Fig. 3, we obtain
the equation on direct current circuit as shown in Eq. (6).

 RB I B + VBE = V
 RB I B + VBC − RC IC = 0(6)

On the other hand, if the ratio of current IC and IE is α,then IC = α IE. This is a variable commonly used in BJT,
and from IE = IB+IC, IB = (1-α) IE, and IC/IB = α/(1-α)(current amplification factor: hFE, β). By rewriting Eq.
(6) using this α, we have the following Eq. (7).

 RB (1 −  ) I E + VBE = V( RB −  ( RB + RC ) ) I E + VBC = 0

VOC：open circuit voltage, Isc：Photocurrent(7)

On the other hand, by transforming the Eq. (3) and Eq.
(4), we have another form of VI characteristic equation
as in the following ([13], [14]).

Fig.5. V-I characteristic of the equivalent circuit
as shown in Fig. 4 1   ( * I E − IC )   1   ( * − ) I E + 1 =   ln 
+ 1VBC =   ln 
IC 0
 cPD    cPD   IC 0
V =  1  ln  I E + 1
 BE  c   I
 LED   E 0 In the PTT of Fig. 3, the LED and the PD are
physically facing each other at a certain distance, so theamount of light received by the PD is considered to be
proportional to the amount of light emitted from theLED. Since the amount of light emitted from the LED is
proportional to the current IE flowing in the LED, IL can(8)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
464

Shimon Hattori, Osamu Matoba, Toshiki Tanaka, Yusuke Kawakami, Tetsuo Hattori

Acknowledgements

Assume that, when the power supply V increases to
V+ΔV, α and IE change to α+Δα, IE+ΔIE. Then. fromthe Eq. (7) and Eq. (8), we have the following Eq. (9)[13].

We would like to express our deepest gratitude toProfessor Emeritus Kensho Okamoto and Doctor
Junichi Fujita, who gave us the opportunity to do thisresearch and many advices.

 K2 
 =   I E
 K1 
VI E = 
 K 
1− ( RB I E )  2   RB (1 −  ) +cLED I E K1  

References
1.

2.

where




 1 
1
 K1 =  ( RB + RC ) +  I E
 cPD  ( * − ) I E + IC 0 



 1 
( * − ) 

K
=
R
−
R
+
R

+
(
)




2
B
B
C


 cPD  ( * − ) I E + IC 0 



3.

4.

(9)

5.

Eq. (9) shows that the amplification factor α and emitter
direct current IE will increase according to the increaseof the DC voltage V. However, we find out that when
the relation Eq. (10) holds, Δα=0, even if ΔV >0 and ΔIE>0.

 = *=

RB
RB + RC

6.

7.

(10)
8.

At that time, from Eq. (8), VBC = 0. When VBC = 0,
from our experiments for PTT with positive feedback
circuit as shown in Fig. 3, we have obtained the direct
current amplification factor β = 999 (α=0.999) [13],[14].
We consider that the same situation can happen in the
fixed bias emitter common circuit of BJT EMM,
because of the similarity of VI characteristic equation.

9.

10.
11.

3. Conclusion
12.

In this paper, from the similarity of the VI
characteristics between the BJT EMM and PTT, wehave discussed the expression of direct current
amplification function of PTT. As a further study, we
will analyze the voltage amplification function for input
AC signal of the PTT circuit based on the VI
characteristics.

13.

14.

B.J. Van Zeghbroeck, Ch. Herder, H. P. Meiner and W.
Walter, Photon transport transistor, Proc. of 1989 IEEE
International Electron Devices Meeting (IEDM), pp.543546 (1989-12)
W. N. Cheung and Paul J. Edwards, Characteristics of aClass of New Opto-Coupler Amplifiers with Positive
Feedback, IEEE Journal of QUANTUM ELECTRONICS,
Vol.32, No.3, pp.502-506 (1996-3)
OSRAM Opto-Semiconductors GmbH, OSLON® SSL
150, LH CPDP Datasheet (https://dammedia.osram.info/media/resource/hires/osram-dam-2493913/LH%20CPD
P.pdf)
Hamamatsu Photonics K.K., Si PIN photodiode S5107
Datasheet (https://www.hamamatsu.com/resources/pdf/
ssd/s5106_etc_kpin1033e.pdf)
Hamamatsu Photonics K.K., Characteristic and Usage(https://www.hamamatsu.com/resources/pdf/ssd/si_pd_te
chinfo.pdf)
Junichi Fujita, Kensho Okamoto, and Tetsuo Hattori,
Development of Novel Transistor and Thyristor
composed of LED and photodiode, Japanese Journal ofApplied Physics, ISSN 1347-4065, Vol.53, 05FEB18,
p.1-6 (2014-4)
Junichi Fujita, Daisuke Sato, Kensho Okamoto, andTetsuo Hattori, Novel Opt-coupling Transistor by LED
and PD and its Application, Journal of Robotics,
Networking and Artificial Life, ISSN 2352-6386, Vol.1,
No.3 pp.244-248 (2014-12)
Junichi FUJITA, Yusuke KAWAKAMI, Katsushi
TAKAHASHI, Kensho OKAMOTO, and Tetsuo
HATTORI, Novel Audio Amplifier and its Kansei
Evaluation,
International
JournalofAffective
Engineering, ISSN 2187-5413, Vol.13, No.4 pp.245-251(2014-12)
Katsuari Kamei, Fundamentals of Electronic Circuits forStudents, ISBN 978-4-320-08638-8, Kyoritsu Shuppan(2010-3). (in Japanese)
Nobuo Fujii ed., Handy-book Electronics, ISBN4-27403471-2, Ohmsha (2002-2). (in Japanese)
Akira Matsuzawa, Fundamental Electronics Circuits
Engineering---Focusing on Analog Circuits ---, IEEJ
University Course, ISBN978-4-88686-276-1, IEEJ,
Ohmsha (2014-2). (In japanese)
Isao Dendo, Basic Electronic Circuits, http://www.music.tohoku.ac.jp/attach/ kiso-denshikairo.pdf (2020-11).
Shimon Hattori, Osamu Matoba, Yusuke Kawakami,
Toshiki Tanaka, Tetsuo Hattori, Consideration on the
Expression of Amplification Function of PTT with
Positive Feedback Circuit Based on ApproximatedCharacteristic Equations of LED and PD, IEEJ Trans. on
Electronics, Information and Systems, Vol.143, No.2,
pp.165-171 (2023-2) (in Japanese)
Shimon Hattori, Osamu Matoba, Yusuke Kawakami,
Toshiki Tanaka, Tetsuo Hattori, Current Amplification
Function of PTT in Positive Feedback Circuit, Proc. of©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
465

Powered by TCPDF (www.tcpdf.org)

A Consideration on Amplification Function

2023 IEEE 12th Global Conference on Consumer
Electronics (GCCE), pp.84-88, Oct., 2023.

Dr. Tetsuo Hattori
He received the B.E. and M.E.
degrees in Electrical Engineeringfrom Doshisha University, Japan,
and the Ph.D. degree in Electronics
and Information Engineering from
Osaka University, Japan. After heworked for Toshiba Eng. Co., Ltd.,he had been with Kagawa University
from 1988 to 2015. Currently, he is a
Professor Emeritus at Kagawa University. His researchinterests include STEM education, signal & image
analysis, and hybrid computer, etc. Member of IEEJ andIEEE.

Authors Introduction
Mr. Shimon Hattori
He received the B.E. and M.E.
degrees in Electrical and Electronics
Engineering from Oita University,
Japan, in 2010 and 2012, respectively.
He is presently a doctoral student at
Kobe University, while he is working
for a company. He is interested in
information processing and computing
mechanism
using
optoelectronic
devices. Member of IEEJ.
Dr. Osamu Matoba
He received the Ph.D. (Eng.) degree,
from Osaka University, Japan, in
1996. From April 1996 to August
2002, he worked as an Assistant at
the Institute of Industrial Science, theUniversity of Tokyo, Japan. Since
September 2002, he has been with
Kobe University. He is currently aProfessor at the Graduate School of Systems andInformatics, also at Kobe University Center for Advanced
Interdisciplinary Research. His research interests include
measurement
optics,information
optics,andoptoelectronics. Member of IEEE, JSAP, OSJ, Senior
Member of LSJ, and also a Fellow of SPIE and OSA.
Mr. Toshiki Tanaka
He received the B.E. degree inElectronics
andInformation
Engineering from Kagawa University,
Japan, in 2015. In the same year, hejoined Kinkei System Co., Ltd.
Currently, while he is engaged inmeasurement and data management ofelectrical and electronic circuit
systems at the company, he is studying
as a doctoral student of Doshisha University, Japan. His
research interests include numerical simulation andparameter estimation. Member of IEEJ.
1.
Dr. Yusuke Kawakami
He received the B.E., M.E., andPh.D. degrees in Information System
Engineeringfrom
KagawaUniversity, Japan, in 2009, 2011,
and 2014, respectively. Since 2020,
he has been with National Instituteof Technology (NIT) Kagawa
College, Japan. Currently, he is an
Assistant Lecturer in NIT Kagawa
College. His research interests include Kansei
Engineering, electronics and machine control, image
processing and sound signal processing. Member of IEEJ.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
466

A Consideration on Amplification Function in BJT Evers-Moll Model and PTT (II)---- H Parameters in the Small Signal Amplifier Circuit---Shimon Hattori, Osamu Matoba

Kobe University, 1-1 Rokkodai-cho, Nada-ku, Kobe City 675-8061, Japan
Toshiki Tanaka

Kinkei System Corporation, 8-2-61 Nanko-Higashi, Suminoe-ku, Osaka 559-0031, Japan
Yusuke Kawakami
NIT Kagawa College, 355 Chokushi-cho, Takamatsu, Kagawa 761-8058, Japan
Tetsuo Hattori
Kagawa University, 1-1 Saiwai-cho, Takamatsu, Kagawa 760-0016, Japanhattori@pe.kagawa-u.ac.jp

Abstract
The notion of PTT (Photon Transport Transistor) has been proposed in 1989 as an optical coupling device of lightemitting diode (LED) and light receiving diode (Photo Diode, PD), where the carrier of the base layer is light(Photon) only. In this paper, in order to deal with the various applications of PTT circuit more theoretically, basedon the approximate VI characteristic equation of LED and PD, we newly derive the h parameters in PTT emittercommon circuit, while referring to the h parameter of Bipolar Junction Transistor (BJT).
Keywords: PTT, LED, PD, positive feedback circuit, amplification function, equivalent circuit, h parameters

the approximate VI characteristic equation of LED and
PD, we derive the h parameters in PTT emitter common
circuit, while referring to the h parameter of Bipolar
Junction Transistor (BJT) ([9], [10], [12]).
Moreover, from the similarities between PTT and BJT
Ebers-Moll Model (EMM), we discuss the essential
factors of amplification function in PTT and BJT.

1. Introduction
The notion of Photon Transport Transistor (PTT) has
been presented by B. J. Van Zeghbroeck et al. at IBM
Research Laboratories presented in 1989 {1}. The PTT
consists of the optical coupling between Light Emitting
Diode (LED) or Laser Diode and Photo Diode (PD).
Moreover in 1996, it has been theoretically shown that
the PTT can be a very low-noise transistor-like device
with an amplification function in a positive feedback
circuit [2].
Later, an audio amplifier prototype using PTT positive
feedback circuit with optical coupling between highbrightness LED and PD was developed, and it wasreported that the PTT even exhibited functions similar to
those of a thyristor ([3], [4], [5], [6], [7], [8]). Also,
while referencing the conventional studies on BJT ([9],[10], [11], [12]), authors have analyzed the current
amplification function of the PTT based on experimental
results ([13], [14]).
In this paper, in order to deal with the various
applications of PTT circuit more theoretically, based on

2. PTT and BJT EMM in Emitter Common Circuit
2.1. VI Characteristic of PTT and BJT EMM
The PTT with fixed bias in emitter common circuit isillustrated in Fig. 1. Generally, the emitter common
circuits are used for small signal voltage amplification.
The VI characteristic equations for each of the PD and
LED that make up the PTT are shown in Eq. (1) and Eq.
(2), respectively ([13], [14]). Each meaning of thesymbols in those equations is shown in Table 1. As for
the BJT and BJT EMM, those circuits without fixed bias
in emitter common are illustrated in Fig. 2(a) and Fig.
2(b). respectively. And, the VI characteristic equationsof two diode parts of the BJT EMM are shown in Eq. (3)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
467

Shimon Hattori, Osamu Matoba, Toshiki Tanaka, Yusuke Kawakami, Tetsuo Hattori

and Eq. (4), respectively, where α0 of Eq. (3) means thecurrent amplification constant.

 qV  
IC = 0 I E − IC 0  exp  BC  − 1 kT  (3) qV  
I E = I E 0  exp  BE  − 1 kT  (4)Comparing the pair of Eq. (3) and Eq. (4) with that ofEq. (1) and Eq. (2). we can expect that, the h parametersof PTT circuit using Eq. (3) and Eq. (4) will become theh parameters of BJT EMM by setting γ*=α, m1=1, m2=1.
Then, we only have to find out the h parameters of PTT.
2.2. H Parameter for Small Signal Equivalent
Equivalent Circuit

Fig.1. PTT emitter common circuit.

IC =   I E − IC 0 ( exp ( cPDVBC ) − 1) , cPD =q(1)
m2kTPDI E = I E 0 ( exp ( cLEDVBE ) − 1) , cLED =(2)q
m1kTLEDAs is well known, for small signal amplification circuitof BJT in emitter common circuit, the small signal
equivalent circuit with a set of h parameters (hie, hre, hfe,
and hoe) is used as shown in Fig. 3 ([9], [10], [12]).

Table 1. The symbol list of PTT characteristic formula.

(a) Precise equivalent circuit.

(b) Simplified equivalent circuit.
(a)Fig. 3. Equivalent circuit of BJT emitter common with hparameters.
The equivalent relation equations in Fig. 3 are as
follows.

vb  hieib + hrevc  hie , hre  ib 
i  = h i + h v  = h , h   v  c   fe b oe c   fe oe   c 

(b)

(5)

These h parameters are obtained from the total
differential equation for VBE and IC, assuming thefollowing functional relationship between f1 and f2.

Fig.2. (a) BJT emitter common circuit for small signal
voltage amplification. (b) BJT EMM without fixed bias
where α0 means the current amplification constant, and
IE = IB+IC.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
468

A Consideration on Amplification Function

Since we assume that VBE = f1 ( I B ,VCE ) , IC = f 2 ( I B ,VCE ) ,h fe = VBE  VBE VBE ( vb ) =  VCE = hieib + hrevc
 I B +  I B  VCE I ( i ) =  IC  I +  IC  V = h i + h v
 C c  I  B  V  CE fe b oe c B
 CE 
(6)−1

 I   I 
=  E    E   hoe VBE   IC 
(7)

I

− C 0  CPD  exp ( CPDVBC )  hie
1−  1− (13)

 − IC 0 
 1 −     ( exp ( CPDVBE )  exp ( −CPDVCE ) )

=
VCE

Since

  ( exp ( CPDVBE ) ) exp ( −CPDVCE ) VCE −I  =  C 0   

1
−


exp
−
C
V

 
( PD CE ) ) 
(
+
exp
C
V

( PD BE )


VCE


 C  I    V  
=  PD C 0   1 −  BE    exp ( CPDVBC ) 1 −     VCE  C I =  PD C 0   1 − hre   exp ( CPDVBC ) 1−  

I E I E 0 ( exp ( CLEDVBE ) −1)== I E 0  CLED  exp ( CLEDVBE )
VBE
VBE
= CLED  ( I E + I E 0 )  CLED  I E
(8)
and
(9)

then we have

hoe
hoe

CLED ( I E + I E 0 ) CLED  I E=

 − IC 0 
 1 −     exp ( CPD (VBE − VCE ) )

=
VCE

B

hre =

 exp ( CPDVBE ) VBE I 
− C 0   exp ( −CPDVCE ) 1−  1−  VBEI B  I
I − C 0  ( exp(CPDVBC ) − 1) 
 B
1
−

1
−

I C

hoe =
= 
VCE
VCE

 V  I   V   I  I 
VBE
=  BE  C  =  BE   E  C 
VCE I =const  IC  VCE   I E   IC  VCE  I E   ( I B + I C ) 

=
 =1
 I C   I C
=Similarly, we have hoe as shown in the following.

First of all, as for hre, we find the following relationshipbetween the other h parameter, hoe.

hre =

IC
I   exp ( CPDVBC ) =− C0  I B 1 −  1 −  
I B


(14)
(10)

As for hie, from Eq. (6), we have−1

 V  I   I   ( I + I ) V
hie = BE =  BE  E  =  E   C B I B  I E  I B   VBE   I B
Since IE = IB + IC in Eq. (1),IC =   (I B + IC ) − IC 0 ( exp ( cPDVBC ) − 1) , cPD =q
m2kTPD(11) I 
= E 
 VBE 

then

IC =
=

−1

( h + 1) = C
fe( h + 1)fe

LED ( I E + I E 0 )  I B I C 0 ( exp ( cPDVBC ) − 1)−
1−  
1−  

( h + 1)fe

CLED  I E(15)

  I B I C 0 ( exp ( −cPD (VCE − VBE ) ) − 1)−
1−  
1−  

So far, we have obtained the following equations for thefour parameters, hie, hfe, hre and hoe.

(12)

Therefore, we have the hfe of PTT as in the following.
©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
469

Shimon Hattori, Osamu Matoba, Toshiki Tanaka, Yusuke Kawakami, Tetsuo Hattori( hfe + 1)
V
hie = BE =
I B
CLED ( I E + I E 0 )h fe = IC =  − IC 0 CPD  exp ( CPDVBC )  hieI B 1 −   1 −  IC  CPD  I C 0 hoe = V =  1 −     1 − hre   exp ( CPDVBC )


CE


hoe
VBE
=
hre =
VCE CLED ( I E + I E 0 )

0, then the term exp(CPDVBC) nearly becomes 1.0. In thiscase we have the more simplified four parameters as
follows that are corresponding to Fig. 3(b).

 1 
 1 
 1−   
 1−    =hie = ,
CLED ( I E + I E 0 ) CLED  I E 0 exp(CLEDVBE )h fe 

, hre  0, hoe  0
1−  

(19)(16)

3. Conclusion
From Eq. (16), we can see the recursive relation
between hie and hfe, and another between hre and hoe. So,
we derive those parameters in the details, as follows.

h fe ==We have newly derived the four h-parameters in the
PTT small signal circuit, based on the VI characteristic
equation of LED and PD that composes the PTT. From
those computed h parameters, we find out that PPTfunctions the amplifier, because approximated andsimplified h parameters are very similar to the case of
BJT.

I

− C 0  CPD  exp ( CPDVBC )  hie
1−  1− ( hfe + 1)
IC 0

−
C

exp
C
V

(
)
PD
PD
BC
1−   1−  
CLED ( I E + I E 0 )Acknowledgements

So, we define the following K.

We would like to express our deepest gratitude toProfessor Emeritus Kensho Okamoto and Doctor Junichi
Fujita, who gave us the opportunity to do this research
and many advices.

IC 0
CPD  exp ( CPDVBC )  CLED ( I E + I E 0 )−1

1− 

K

References

Then we have

 

−1
h fe = − K   (1 + K ) 1− 

IC 0

−1
−
CPD  exp ( CPDVBC )  ( CLED ( I E + I E 0 ) )
1−  1− =I−1
1 + C 0  CPD  exp ( CPDVBC )  ( CLED ( I E + I E 0 ) )
1− 

1.

2.

3.

 CLED ( I E + I E 0 ) IC 0−
C  exp ( CPDVBC )1−  
1 −   PD
=

CLED ( I E + I E 0 ) +IC 0C  exp ( CPDVBC )1 −   PD

4.

(17)

Similarly, as for hre, we have the following equation.
 CPD  IC 0 
 1 −     exp ( CPDVBC )


hre =
C I 
CLED  I E 0  exp ( CLEDVBE ) +  PD C 0   exp ( CPDVBC ) 1−  

5.

6.

(18)

If we consider the case when VBC approximately equals

B.J. Van Zeghbroeck, Ch. Herder, H. P. Meiner and W.
Walter, Photon transport transistor, Proc. of 1989 IEEE
International Electron Devices Meeting (IEDM), pp.543546 (1989-12)
W. N. Cheung and Paul J. Edwards, Characteristics of aClass of New Opto-Coupler Amplifiers with Positive
Feedback, IEEE Journal of QUANTUM ELECTRONICS,
Vol.32, No.3, pp.502-506 (1996-3)
Junichi Fujita, Kensho Okamoto, and Tetsuo Hattori,
Development of Novel Transistor and Thyristor composedof LED and photodiode, Japanese Journal of Applied
Physics, ISSN 1347-4065, Vol.53, 05FEB18, p.1-6 (20144)
Junichi Fujita, Daisuke Sato, Kensho Okamoto, andTetsuo Hattori, Novel Opt-coupling Transistor by LED
and PD and its Application, Journal of Robotics,
Networking and Artificial Life, ISSN 2352-6386, Vol.1,
No.3 pp.244-248 (2014-12)
Junichi FUJITA, Yusuke KAWAKAMI, Katsushi
TAKAHASHI, Kensho OKAMOTO, and Tetsuo
HATTORI, Novel Audio Amplifier and its Kansei
Evaluation, International Journal of Affective Engineering，
ISSN 2187-5413, Vol.13, No.4 pp.245-251 (2014-12).
OSRAM Opto-Semiconductors GmbH, OSLON® SSL
150, LH CPDP Datasheet (https://dammedia.osram.info/
media/resource/hires/osram-dam-2493913/LH%20CPDP.pdf)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
470

Powered by TCPDF (www.tcpdf.org)

A Consideration on Amplification Function

7.

8.

9.
10.

11.

12.
13.

14.

Hamamatsu Photonics K.K., Si PIN photodiode S5107
Datasheet (https://www.hamamatsu.com/resources/
pdf/ssd/s5106_etc_kpin1033e.pdf)
Hamamatsu Photonics K.K., Characteristic and Usage(https://www.hamamatsu.com/resources/pdf/ssd/si_pd_tec
hinfo.pdf)
Nobuo Fujii ed., Handy-book Electronics, ISBN4-27403471-2, Ohmsha (2002-2). (in Japanese)
Katsuari Kamei, Fundamentals of Electronic Circuits forStudents, ISBN 978-4-320-08638-8, Kyoritsu Shuppan(2010-3). (in Japanese)
Akira Matsuzawa, Fundamental Electronics Circuits
Engineering---Focusing on Analog Circuits ---, IEEJ
University Course, ISBN978-4-88686-276-1, IEEJ,
Ohmsha (2014-2). (In japanese)
Isao Dendo, Basic Electronic Circuits, http://www.music.tohoku.ac.jp/attach/ kiso-denshikairo.pdf (2020-11).
Shimon Hattori, Osamu Matoba, Yusuke Kawakami,
Toshiki Tanaka, Tetsuo Hattori, Consideration on the
Expression of Amplification Function of PTT with
Positive Feedback Circuit Based on ApproximatedCharacteristic Equations of LED and PD, IEEJ Trans. on
Electronics, Information and Systems, Vol.143, No.2,
pp.165-171 (2023-2) (in Japanese)
Shimon Hattori, Osamu Matoba, Yusuke Kawakami,
Toshiki Tanaka, Tetsuo Hattori, Current Amplification
Function of PTT in Positive Feedback Circuit, Proc. of2023 IEEE 12th Global Conference on Consumer
Electronics (GCCE), pp.84-88, Oct., 2023.

Mr. Toshiki Tanaka
He received the B.E. degree inElectronics
andInformation
Engineering from Kagawa University,
Japan, in 2015. In the same year, hejoined Kinkei System Co., Ltd.
Currently, while he is engaged inmeasurement and data management ofelectrical and electronic circuit systems
at the company, he is studying as a doctoral student ofDoshisha University, Japan. His research interests include
numerical simulation and parameter estimation. He is amember of IEEJ.
Dr. Yusuke Kawakami
He received the B.E., M.E., and Ph.D.
degrees in Information System
Engineering from Kagawa University,
Japan, in 2009, 2011, and 2014,
respectively. Since 2020, he has beenwith National Institute of Technology(NIT) Kagawa College, Japan.
Currently, he is an Assistant Lecturer
in NIT Kagawa College. His researchinterests include Kansei Engineering, electronics andmachine control, image processing and sound signal
processing. He is a member of IEEJ.
Dr. Tetsuo Hattori
He received the B.E. and M.E. degrees
in Electrical Engineering from
Doshisha University, and the Ph.D.
degree in Electronics & Information
Engineering from Osaka University,
Japan. After he worked for ToshibaEng. Co., Ltd., he had been with
Kagawa University from 1988 to2015. Currently, he is a Professor
Emeritus at Kagawa University. His research interests
include STEM education based on ETT, signal and image
analysis, pattern recognition, and realization of hybrid
computer. Member of IEEJ and IEEE.

Authors Introduction
Mr. Shimon Hattori
He received the B.E. and M.E.
degrees in Electrical and Electronics
Engineering from Oita University,
Japan, in 2010 and 2012, respectively.
He is presently a doctoral student at
Kobe University, while he is working
for a company. He is interested in
information processing and computing
mechanism
using
optoelectronic
devices. Member of IEEJ.
Dr. Osamu Matoba
He received the Ph.D. (Eng.) degree,
from Osaka University, Japan, in
1996. From April 1996 to August
2002, he worked as an Assistant at
the Institute of Industrial Science, theUniversity of Tokyo, Japan. Since
September 2002, he has been with
Kobe University. He is currently aProfessor at the Graduate School of Systems andInformatics, also at Kobe University Center for Advanced
Interdisciplinary Research. His research interests include
measurement
optics,information
optics,andoptoelectronics. Member of IEEE, JSAP, OSJ, Senior
Member of LSJ, and also a Fellow of SPIE and OSA.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
471

Color Image Arrangement Using Histogram Matching
Yusuke Kawakami

National Institute of Technology (Kagawa College), 355 Chokushi-cho, Takamatsu, 761-8058, Japan
Email: riverjp2002@gmail.com, kawakami-y@t.kagawa-nct.ac.jp
Tesuo Hattori

Kagawa University, 1-1 Saiwai-cho, Takamatsu, 760-8521, Japan
Email: hattori@pe.kagawa-u.ac.jp
R. P. C. Janaka Rajapakse

Tainan National University of the Arts, 66 Daqi, Guantian Dist., Tainan 72045, Taiwan (R.O.C.)
Email: janakaraja@gmail.com

Abstract
For the arrangement processing of image using its histogram, we previously have presented a Histogram Matching
method based on Gaussian Distribution (HMGD). However, in the case where the brightness histogram of input
image has multiple peaks, the HMGD processing does not always bring good results. In this paper, we present an
improved histogram matching method using the reference histogram that is made by appropriate moving average(HMMA) processing over the histogram of input image. Also in this paper, we show the experimental results.
Keywords: Image processing, Histogram matching (HM), Moving average, Kansei impression

Fig. 1 shows the conceptual image of HMGD. Let f(x)and h(y) be two probabilistic density functions (PDF) on
real variables x and y, respectively. The PDF iscorresponding to histogram of image brightness level
which is discretely defined.

1. Introduction
Recently, automated image arrangement processing
such as enhancement of images that we have alreadyreported is usually used in the various imaging devices,for example, Digital Camera, Smart Phone and so on [1],[2], [3].
In our previous papers, we have presented a Histogram
Matching based on Gaussian Distribution (HMGD)
processing for the image arrangement [4], [5], [6], [7].
And we illustrated that HMGD processing can improve
feeling (or Kansei) impression of the original image [4],[5], [6], [7].
In this paper, we propose an improved Histogram
Matching method using the reference histogram that is
made by appropriate Moving Average (HMMA)
processing over the histogram of input image [8]. Also,in this paper, we provide the experimental results forsome color images and show its effectiveness of the
proposed method.

In addition, let y=(x) be a continuous and monotonic
increase function corresponding to cumulative
histogram of image brightness level between variables x
and y. And let y=(x) be defined by Eq. (1).
xy =  ( x ) + L  f ( x ) dx ................................. (1)0

At first, we have to expand brightness level of original
image histogram and convert into uniform distribution
histogram, because we aim to match Gaussian
distribution. From Eq. (1) and Fig. 1, we can derive Eq.
(2) and (3).

2. Principle
2.1. Histogram Matching as Example of GaussianDistribution [9]f ( x ) = h ( y ) ' ( x ) = h ( y ) Lf ( x ) .............. (2)h( y) =In the section, we describe the principle of histogram
matching method as an example of Histogram Matchingbased on Gaussian Distribution (HMGD).

1........................................................... (3)L©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
472

Yusuke Kawakami, Tesuo Hattori, R. P. C. Janaka Rajapakse

We understand the histogram of original image f(x)becomes uniform distribution h(y) by Eq. (3). Thismeans that brightness level of original image f(x) is
expanded to h(y).

Fig. 4 and Fig. 5 show another example of results andthe corresponding histogram, respectively. In this case,
we understand that the simple application of HMGD
processing gives poor perspective impression. And
HMMA image is enhancing its perspective and detail ofrocks in this image than original.
From the experimental results, we consider that
HMMA processing enhances the effect of the Kansei
impression effect more than HMGD processing.

Then, let g(z) and (z) be the function that is definedby Eq. (4) and Eq. (5), respectively.

 ( z −  )2 
 ............... (4)g ( z) =exp  −
2


2

2 2


1

y

y

y=(z)y=(x)

(L)

z

h(y)

y =  ( z ) = L  g ( z ) dz
0y

 ( z −  )2 =
exp  − 2 2  dz ........... (5)2 2 0


1

z

(x)
L
L

Here, Fig. 1 shows the relationship between y=(x)and y=(z). So we can be obtained following Eq. (6) andEq. (7).
x

L f ( x ) dx =0xz0

z z0+dz

2

x0+dx

x

f(x)

g(z)

x
z
 ( z −  )2 ddLL f ( x ) dx =exp− dz. .. (7)dx 0dz 2 2 0
2 2 


Fig.1 Conceptual image of histogram matching(example of HMGD).

If we perform Eq. (7),L ' ( x ) = L ' ( z ) ,

x0

z
1
L

 (z − ) 
exp  −
 dz .... (6)
2



2

2 2 0Lz(x0+dx)

4. Conclusionf ( x )  g ( z ) .......... (8)In this paper, for the color image arrangement
processing, we have described two histogram matchingbased methods: Histogram Matching based on GaussianDistribution (HMGD) and Histogram Matching basedon Moving Averaged histogram (HMMA).

That is, we understand that f(x) becomes Gaussian
distribution g(z) when we take the transform function as
Eq. (1) and Eq. (5).

And we have compared the two methods as to howeach processing brings about the Kansei impression
effect. Although the number of experimented images
that we have shown in this paper is limited, from theresults, we can find that the processing by HMMA moreenhances the contrast, brightness, and detail of original
image than by HMGD. That is, we consider that
HMMA processing can improve the Kansei impression.

Thus, histogram matching processing is the function
which defined by cumulative histogram transformation
the original histogram into arbitrary histogram.
3. Experimentation
Fig. 2 and Fig. 3 show the example of results and thecorresponding histogram respectively. In this case, we
understand that HMGD image is very enhancing
brightness and contrast than original image.
However, the touch of brush detail transforms
unnatural in this image. Also we understand that
HMMA image is enhancing contrast than original image
and the color tone keeps and slightly brighter thanoriginal image.

Since these evaluation of impression effects are
subjective at this point, so we will objectively
investigate the improvement of Kansei impression
through questionnaire surveys and characteristics
quantity in image, for future study.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
473

Color Image Arrangement Using

1000
900
800

Frequeny

700
600
500
400
300
200
100
0
1

25 49 73 97 121 145 169 193 217 241
Brightenssa.

Original imagea.

Original image histogram

1

25 49 73 97 121 145 169 193 217 241

1000
900
800

Frequency

700
600
500
400
300
200
100
0

Brightnessb.

b.

HMGD image

HMGD image histogram

1000
900
800

Frequency

700
600
500
400
300
200
100
0
1

25 49 73 97 121 145 169 193 217 241
Brightness

c. HMMA image histogram
Fig. 3. Corresponding histogram of Fig. 2

c. HMMA image
Fig. 2. Experimental results of original image, HMGD
image, and HMMA image©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
474

Yusuke Kawakami, Tesuo Hattori, R. P. C. Janaka Rajapakse

1600
1400
1200

Frequeny

1000
800
600
400
200
0
1

25 49 73 97 121 145 169 193 217 241
Brightenssa.

Original imagea.

Original image histogram

1

25 49 73 97 121 145 169 193 217 241

1600
1400
1200

Frequency

1000
800
600
400
200
0

Brightnessb.

b.

HMGD image

HMGD image histogram

1600
1400
1200

Frequency

1000
800
600
400
200
0
1

25 49 73 97 121 145 169 193 217 241
Brightness

c. HMMA image histogram
Fig. 5. Corresponding histogram of Fig. 4

c. HMMA image
Fig. 4. Experimental results of original image, HMGD
image, and HMMA image©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
475

Powered by TCPDF (www.tcpdf.org)

Color Image Arrangement Using

Dr. Tetsuo Hattori
He received the B.E. and M.E.
degrees in Electrical Engineeringfrom Doshisha University, and the
Ph.D. degree in Electronics &Information
Engineering
from
Osaka University, Japan. After heworked for Toshiba Eng. Co., Ltd,he had been with Kagawa
University from 1988 to 2015.
Currently, he is a Professor Emeritus at Kagawa
University. His research interests include STEM
education based on ETT, signal and image analysis,
pattern recognition, and realization of hybrid computer.
He is a member of IEEJ and IEEE.

References
1.
2.

3.

4.
5.

6.

7.

8.

9.

R. C. Gonzalez and R. E. Woods, Digital Image
Processing, Addison-Wesley Publishing Company, 1993.
B. Jahne, Digital Image Processing --Concepts,
Algorithms, and Scientific Applications-- 4th edition,
Springer, 1995.
E. S. Umbaugh, Computer Vision and Image Processing:
A Practical Approach Using CVIP tools, Prentice Hall
PTR, 1998.
W. Burger and J. M. Burge, Principles of Digital Image
Processing: Fundamental Techniques, Springer, 2009.
T. Izumi, T. Hattori, S. Sugimono, and T. Takashima,
Color Image Arrangement Using Elastic Transform on
Principal Component Axis (in Japanese), Journal of Japan
Society of Kansei Engineering Vol. 8(3) (2009), pp. 667674.
Y. Kawakami, T. Hattori, D. Kutsuna, H. Matsushita, Y.
Imai, H. Kawano, R.P.C. Janaka Rajapakse, AutomatedColor Image Arrangement Method Based on Histogram
Matching - Investigation of Kansei impression between
HE and HMGD -, International Journal of Affective
Engineering Vol. 14(2) (2015), ISSN 2187-5413, pp. 8593.
Y. Kawakami, T. Hattori, Y. Imai, H. Matsushita, H.
Kawano, R. P. C. Janaka Rajapakse, Kansei Impression
and Automated Color Image Arrangement Method,
Journal of Robotics, Networking and Artificial Life Vol.
1(1) (2014), ISSN 2352-6386, pp. 60-67.
Y. Kawakami, T. Hattori, Y. Imai, K. Ando, Y. Horikawa,
R. P. C. J. Rajapakse, Reference Histogram Matching
Based on Moving Averaged Brightness Histogram, IEEJ
Transactions on Electronics, Information and Systems
Vol. 138(9) (2018), ISSN 1348-8155, pp. 1131-1132.
Y. Kawakami, T. Hatttori, R. P. C. J. Rajapakse, Kansei
Impression of Color Image Arrangement Using
Histogram Equalization, Proceedings of ICESS2023 (The
8th International Conference on Electronics and Software
Science), 4pages.

Dr. Janaka Rajapakse
He received the M.S. and Ph.D.
degrees in Intelligence Science fromJapan Advanced Institute of Scienceand Technology (JAIST), in 2005
and 2008, respectively. Currently,
he is an Associate Professor at theGraduate Institute of Animation and
Film
Art,
TainanNational
University of the Arts, Taiwan. His
research interests include Computer
Graphics and Animations, Artificial Intelligence, Motion
Capture Techniques, Virtual Reality, Augmented Reality,
Haptic Interfaces, 3D Printing, Interactive Media andKansei Engineering. He is also a member of IEEE.

Authors Introduction
Dr. Yusuke Kawakami
He received the B.E., M.E., andPh.D. degrees in Information
System Engineering from Kagawa
University, Japan, in 2009, 2011,
and 2014, respectively. Since 2020,
he has been with National Instituteof Technology (NIT) Kagawa
College, Japan. Currently, he is an
Assistant Lecturer in NIT Kagawa
College. His research interests include Kansei
Engineering, image processing and sound signal
processing. He is a member of IEEJ.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
476

Methodology for Creativity Oriented STEM Education Based on ETT Theory
Tetsuo Hattori, Toshihiro Hayashi, Mai Hattori, Yoshiro Imai
Kagawa University, 1-1 Saiwai-cho, Takamatsu, Kagawa 760-0016, Japan
Email: hattori@pe.kagawa-u.ac.jp
Asako Ohno
Osaka Sangyo University, 3-1-1, Nakagaito, Daito-shi, Osaka 574-8530, Japan
Takeshi Tanaka
Hiroshima Institute of Technology, Saeki-ku, Hiroshima 731-5143, Japan

Abstract
Recently, the necessity of integrated and comprehensive methodology for STEM (Science, Technology,
Engineering and Mathematics) education is growing. In this paper, we propose an educational methodology
utilizing the viewpoint of Equivalent Transformation Thinking (ETT) theory which has been proposed by Dr.
Kikuya Ichikawa in 1955 as a principle of creativity. Especially, we show that the viewpoint is very useful not only
for new technology invention but also for STEM Education in the sense that it deepens insights of the contents to belearned, motivates students to study further, and inspires their creativity.
Keywords: STEM Education, Equivalent Transformation Thinking (ETT) Theory, Identification of Equivalence, 3Dimensional Computer Graphics (3DCG).

2.

1. Introduction

ETT Theory

As aforementioned, the ETT theory has been proposed
as a principle of creativity by late Dr. Kikuya Ichikawa
in 1955, after his investigation of the past discoveries
and inventions to analyze how creation was done. And
he revealed the investigation results that new technology
developments and creations were produced along with
the ETT process. Then he described the ETT process as
a form of symbolic logical expression which is called“Equivalence Equation” as shown in Fig. 1.

Recently, the necessity of integrated andcomprehensive methodology for Science, Technology,
Engineering and Mathematics (STEM) education seems
to be growing [1], [2], because the content of the STEM
field is becoming increasingly sophisticated.
For the STEM education, we propose the methodology
based on the viewpoint of Equivalent Transformation
Thinking (ETT) theory which was advocated as aprinciple of creativity by Dr. Kikuya Ichikawa in 1955,
after his investigation and analysis of the past
discoveries and technological inventions.
In this paper, we briefly explain the ETT theory ([3],[4], [5], [6], [7], [8]) and describe that the viewpoint of
ETT is very useful not only for new technology
invention but also for STEM education. This is because
the past inventions, discoveries and academic
developments in STEM field can be explained according
to the ETT theory.
Especially, we emphasize that the suitably combined
use of dynamic visualization technology based on the
viewpoint of ETT will be very effective for the students
to deeply understand the learning contents, to promote
their motivation, and to stimulate their creativity ([9],[10], [11]).

Ao: original system, Bτ: transformed (or arrival) system,
Σa: a set of special/peculiar properties and conditions
that holds the system Ao, cε: core essence or essential
meaning that reasons the equivalence under someconditions, Σ b: a set of necessary properties andconditions that hold the system, vi: viewpoint at a certain
meta level or objective level.

Fig. 1 Equivalence Equation by Dr. Kikuya Ichikawa in
his ETT Theory.
The description of “Equivalence Equation” means that,from a certain viewpoint vi, the original system Ao canbe transformed to the system Bτ according to theconditionalized essence cε, where some properties and©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
477

Tetsuo Hattori, Toshihiro Hayashi, Mai Hattori, Yoshiro Imai, Asako Ohno, Takeshi Tanaka

conditions Σa in Ao are discarded, and new elements ofproperty and condition Σb are added to the system Bτ.
When Dr. Ichikawa presented “Equivalence Equation”,
he also showed the schematic flowchart of ETT. And heinsisted that, if we make effort to develop something to
be desired according to the ETT flowchart, creative
achievement will be possible more efficiently.
As for the ETT theory, he mainly published papers at
the Creativity Research Workshop that was held by Dr.
Hideki Yukawa, the winner of 1949 Nobel Prize inPhysics. Dr. Yukawa called this Dr. Ichikawa’s theory
as "Identification Theory" in his own words and highly
praised it.

fluid, is equal to the weight of the fluid in the same
volume as the volume of body in the fluid (Fig. 3).
As shown in Fig. 3, if we imagine a situation that an
object is quietly floating on the water, and if we think ofthat it is equivalent to the situation where the water inthe same volume area of the object under the surface isquietly balanced (Fig. 3(c)), we understand thatArchimedes’ principle will be derived.

3. Application Examples of ETT Viewpoint3.1. Cartesian Coordinate System(a)

Looking back on history, it is possible to find out
many facts that the act of ETT and/or Identification ofEquivalence has made a progress of research in thefield of STEAM (Science, Technology, Engineering,
Art, and Mathematics), and that it has broadened ourhorizons and deepened our knowledge. Then, we
consider that utilization of the viewpoint of ETT isalso effective for STEM education.
For example, René Descartes provided “Cartesian
coordinate system” that gives a correspondence
between a point P on a plane and a pair of two
numbers (x, y), where x and y are quantities based on
the distances to two orthogonal axes (x-axis and yaxis), respectively, as shown in Fig. 2.

(b)

(c)

Fig. 3. Floating power (or buoyant force) that operates
on a body immersed in a fluid. (a) An object is quietly
floating on the water. (b) V means the same volume/areaof the object under surface. (c) The water in the same
volume area V is quietly balanced.
3.3. Mechanical Dynamic System (MDS) and Electric
Circuit System (ECS)For taking the relation between MDS and ECS as an
example, we can deal with the equations of two systems(see Fig. 4) as equivalent equation from both meaning ofmathematics and physics.
 d2x
dx x
dx
m 2 +r+ = F0 ( v =
)

 dt
dt k
dt

2
 L d q + R dq + q = E ( I = dq )
0

dt C
dt
 dt 2

Fig. 2 Cartesian coordinate system.
As a result, taking a parabolic curve C for an example,
it can be simply represented by a set of coordinates andalgebra, without showing the procedure of geometricdrawing, such as
C= {P(x, y) | y =ax2, a: constant number}.
This is an example that ETT not only opens the new
field of mathematics such as Analytic Geometry andAlgebraic Geometry, but also brings about the new wayof representation such as “Field” into physics, so we
consider that it is worth to enhance the idea of ETT.
Moreover, we should notice the viewpoint of ETT andappropriately point out the viewpoint of equivalence
when teaching the subject of STEM.

Mass m ⇔Inductance L, Responsiveness r ⇔Resistance R,
Compliance k (the inverse of spring constant) ⇔Capacitance C

(a)
F0

(b)

(c)

Fig. 4. (a) Differential equations and their corresponding
elements in MDS and ECS. (b) Conceptual image of
MDS. (c) Conceptual image of ECS.

3.2. Archimedes' Principle of Buoyancy
According to Archimedes' principle, floating power (orbuoyant force) that operates on a body immersed in a

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
478

Methodology for Creativity Oriented

3.4. Explanation of Faraday’s World First Motor
Faraday's world first electromagnetic rotating device isshown in Fig. 5, where the current flows from top tobottom into the left and right mercury containers.
Viewing from the above, when the current flows, bothof bar magnet and electrode (or conductor) rotate
clockwise.

the tower consisting of n disks, from the place at pillar
A (or start place) at the left end to the pillar C (ordestination) at the right end, using the pillar B in the
middle (as work place), under the constraints as below.
(i) Only one disk must be moved at a time, (ii) Do not
go outside the three sections, (iii) Do not put a disk overa smaller disk.

(a)Fig. 6. The “Tower of Hanoi” problem. (a) Initial state
when the number of disks n=5. From the left end, wecall three places as A, B, and C. (b) The smallest disk at
A is moved from A to C, and the next small disk is on
the way to move out.

(b)

(a)

(b)

For making the recursive program in the general caseof n disks, we note a pattern of disk moving, that willappear clearly when the number of disks n=2. That is,
the pattern is equivalent to what can be seen if we divide
n disks into two parts: upper parts (n-1 disks) and lowest
part (the bottom largest disk).
Imagine when the largest disk can be moved from A(start) to C (destination) under the constraints. Firstly,
we must move the upper parts (n-1 disks) from the place
A to B (workplace), after that, the largest disk (=lowestpart) can be moved from A to C. As the next step, theupper parts (n-1 disks) are to be moved from B to C.
This is the important turning point (Fig. 7).

(c)Fig. 5. Faraday’s World’s First Motor. (a) Documents
left at the Royal Institution. (b) Explanation of each partand direction of current． (c) Motion view of the Motor
made by 3-DCG “Blender”.
As for the force that causes rotation, there are different
explanations (or lectures) for the left and right sides. For
the left side, the current flowing through conductor
rotates the movable bar magnet by creating magnetic
according to Ampere's law. On the contrary, for theright side, the the current in conductor rotates because it
receives Lorentz force from the magnetic field made by
the fixed bar magnet.
However, there is a unified explanation from law ofaction and reaction. In the left side, like the left side, themagnetic field caused by the current in conductor exerts
a force on the fixed bar magnet, but since the bar
magnet is fixed and the conductor is floating, then thethe conductor rotates itself by the reaction force.
This unified explanation can be easily provided by 3DCG video, referring that an electric current generates amagnetic field around it and receives a reaction force
from the bar magnet via the magnetic field.

(a)

(b)

(c)

Fig.7. The turning point procedure of “Tower of Hanoi”
problem in moving n disks from the place A to C using
the place B. (a)Initial state. (b) and (c). The turning
point when the top n-1 disks are moved from A to B,
after that, largest disk left at A is moved to C.

3.5. Recursive Programming
Recursive programming is useful, but it is not so easy
to suitably write down. Taking the problem of “Towerof Hanoi” for example (Fig. 6), the problem is to move

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
479

Tetsuo Hattori, Toshihiro Hayashi, Mai Hattori, Yoshiro Imai, Asako Ohno, Takeshi Tanaka

Then, if we let TOH(n, S, W, D) represent a procedureof the “Tower of Hanoi” problem to move n disks fromthe start place S to destination D using workplace W, we
obtain the following symbolic formulation as a recursive
procedure.
TOH(n, A, B, C) =TOH(n-1, A, C, B) + Move thelargest of n disks from A to C+ TOH(n-1, B, A, C).
The procedure TOH(n, S, W, D) is concretely,
described by Excel VBA, as shown in the Fig. 8.

b( n) =

b(n) b(n -1) b(3) b(2)
 b(1) = 2nb(n -1) b(n - 2) b(2) b(1)Therefore, a(n)=b(n)-1=2n-1.

4.

Conclusion

In this paper, we have proposed a methodology for
integrated and comprehensive STEM education, basedon the viewpoint of ETT. Moreover, we have presented
some application examples of teaching way from theETT viewpoint, in order to show the usefulness. We
consider that the suitably combined use of visualization
tool such as 3-DCG will be more effective.
As a further study, we are considering that the solution
algorithm for the Tower of Hanoi problem can be
represented as the tree generation in tree language
notation. Then, we find out that the recursive algorithm
may be described as an iterative algorithm equivalent to
the recursive one, and that a fast solution algorithm canbe obtained. We would like to report this on anotheropportunity.

Sub main_Tower_of_Hanoi()
Dim n As Integer, Input_n As String
Input_n = InputBox("The number of disks ?")n = Val(Input_n)Call Hanoi_Tower(n, "A", "B", "C")End Sub'**********************************(a)

Sub Hanoi_Tower(n As Integer, S As String, W As
String, D As String)'/*Move n-1 disks from "S" to "W" via "D."*/If n >= 2 Then
Call Hanoi_Tower(n - 1, S, D, W)
End IfDebug.Print "Move the top disk No. " & n & " : " & S& "→" & D & "."'/*Move n-1 disks from "W" to "D" via "S."*/
If n >= 2 Then
Call Hanoi_Tower(n - 1, W, S, D)
End IfEnd Sub‘****************************************(b)

References
1. Yeping Li, Ke Wang, Yu Xiao & Jeffrey E. Froyd,Research and trends in STEM education: a systematic
review of journal publications, International Journal of
STEM Education, Springer Open, Vol.7, No.11,
https://doi.org/10.1186/s40594-020-00207-6G (2020)
2. Fitzgerald, M. S., & Palincsar, A. S., Teaching practices
that support student sensemaking across grades anddisciplines: A conceptual review, Review of Research in
Education, Vol.43, pp.227–248(2019)
3. Kikuya Ichikawa, Science of Creativity, NHK Publishing(1970) (in Japanese)
4. Kikuya Ichikawa, Creative Engineering, Lattice Co., Ltd.
(1977) (in Japanese)
5. Hideki Yukawa, Leap to Creation, Kodansha (1968) (inJapanese)
6. Equivalent Transformation Creativity Society ed.,Equivalent transformation theory that can be understoodgraphically, NIKKAN KOGYO SHINBUNSHA (2005)(in Japanese)
7. Hikaru Matsuki, Hideaki Ogawa, Yasuaki Kajisha, A
New Movement on the Theory of Equivalent Transformal
Thinking, The Japanese Society for Artificial Intelligence,
Vol.22, No.3, pp.392-398 (2007) (in Japanese).
8. Takeo HARUYAM, A Study on the Technology and Skill
Transfer from the Creative Engineering Point of View,
Journal of Japan Society for Production Management,
Vol.15, No.1, pp.1-6 (2008) (in Japanese)
9. Tetsuo Hattori, Rikiya Masuda, Yoshio Moritoh, Yoshiro
Imai, Yusuke Kawakami, Takeshi Tanaka, Utilization ofBoth Free 3DCG Software “Blender” and 3D Printing for
Early STEM Education, Proc. of IEEE TALE2020 (AnInternational Conference on Engineering, Technology and
Education), ISBN 978-1-7281-6942-2, pp.971-974 (2020)
10. Tetsuo Hattori, Rikiya Masuda, Toshihiro Hayashi,
Yoshiro Imai, Yoshio Moritoh, Yusuke Kawakami,
Takeshi Tanaka, A STEM Education Method Utilizing
3DCG Software Blender, Proc. of the Sixth International(c)Fig. 8. Excel VBA program for the Tower of Hanoi. (a)and (b). The program list. (c) (upper) Inputbox wheren=4 is input. (lower) The output at immediate window
in the case n=4.
The necessary number of steps a(n) to solve theproblem of n disks is decided from the above algorithm.
a(n)=a(n-1) +1+a(n-1) =2a(n-1) +1 (n≧2), (a (1) =1).
Let b(n)=a(n)+1, then b(1) =2 and b(n)=2b(n-1). Then,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
480

Powered by TCPDF (www.tcpdf.org)

Methodology for Creativity Oriented

Conference on Electronics and Software Science(ICESS2020), ISBN978-1-941968-62-8, The Society of
Digital Information and Wireless Communications(SDIWC), pp.38-43 (2020)
11. Tetsuo Hattori, Asako Ohno, Yoshiro Imai, Yusuke
Kawakami, Katsunori Shimohara, Takeshi Tanaka:“Creativity Oriented STEM Education Utilizing
Equivalent Transformation Thinking Theory andDynamic Visualization Technology”, Proc. on 2022
Annual Conference of Electronics, Information and
Systems Society, IEE of Japan, GS3-6, pp.1058-1062(2022) (in Japanese)

Dr. Asako Ohno
She received the B.S. from Osaka
University of Foreign Studies in2001. She also received the M.S. and
Ph.D. degrees in Human Sciencefrom Kobe University, in 2006 and2009, respectively. Currently, she isan Associate Professor at Osaka
Sangyo University. Her researchinterests include Intelligent Learning System, Learner
Behavior Analysis, Elementary Programming Education,etc.

Authors Introduction

Dr. Takeshi Tanaka
He received the B.E., M.E., and Ph.D.
degrees in Electrical and Material
Engineering
from
Hiroshima
University, Japan, in 1982, 1984, and1990, respectively. Currently, he is a
Professor at the Department ofElectronics and Computer Engineeringin Hiroshima Institute of Technology.
His research interests include Integrated Circuit
Manufacturing, Applications, and Education on
Semiconductor, etc.

Dr. Tetsuo Hattori
He received the B.E. and M.E.
degrees in Electrical Engineeringfrom Doshisha University, Japan,
and the Ph.D. degree in Electronics& Information Engineering from
Osaka University, Japan. After heworked for Toshiba Eng. Co. Ltd., hehad been with Kagawa University
from 1988 to 2015. Currently, he is a
Professor Emeritus there. Member of IEEJ and IEEE.
Dr. Toshihiro Hayashi
He received the B.E., M.E., and Ph.D.
degrees in Information and System
Engineeringfrom
Tokushima
University, Japan, in 1989, 1991, and1994, respectively. Since July 2004,
he has been with Kagawa University,
after he worked at Saga University
from April 1994 to June 2004.
Currently, he is a Professor at Kagawa University as the
director of Integrated Center for Informatics there.
Dr. Mai Hattori
She received the Ph.D. degree in
Medicine from Kagawa University,
Japan, in 2023. She is currently
engaged in the study on Education ofLife Science and Nutrition utilizing
Information Technology as a
researcher of Integrated Center for
Informatics at Kagawa University.
Dr. Yoshiro Imai
He received the B.E. degree inInformation Engineering from Kyoto
University, Japan, in 1980, and Ph.D.
from
Tokyo
UniversityofAgriculture and Technology, Japan,
in 2008. He had been a professor at
Kagawa University until the end of
March, 2021. Now he is a Professor
Emeritus there. His research interests include Computer
Architecture and e-Learning system Design, etc.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
481

Efficient Campus Shuttle Tracking and Management Mobile Application for College Campus
Andrea Tantay Gonzales

INTI International College Penang, Malaysia
Kavitha Thamadharan

INTI International College Penang, Malaysia
Neesha Jothi

UCSI University, Malaysia

Abstract
The purpose of this study is to address the current paper-based shuttle booking and tracking system at INTIInternational College, Penang. Shuttle Stalk, a Real-Time Campus Shuttle Booking and Tracking System was a
proposed solution that enables the students to register for the shuttle service through the mobile application,
reducing the administrative burden and tracking its current location in real-time, allowing them to plan theirjourneys more effectively. The application developed serves as a comprehensive solution for both students andadministrators, providing real-time shuttle tracking and streamlined management capabilities. The development of
this application is informed by extensive data collected through pre- and post-acceptance surveys, to ensure its
effectiveness and alignment with user needs. The mobile application not only enhances the efficiency of shuttle
services but also contributes to higher user satisfaction, improving the overall campus experience.
Keywords: Campus shuttle-tracking application, real-time shuttle location, Global Positioning System (GPS),
Google Maps, Firebase, Mobile Application

and universities (e.g., INTI International College andUniversities) appeals mostly to students, as it meets thecriteria of choosing a transportation mode. However, thelack of information regarding the transportation’s arrivaltime, current location, and seat availability makes
commuters or students reluctant to use public
transportation [8].

1. Introduction
Shuttle services are a form of public transportation that
respective schools and universities often provide toenable their students to travel from the pick-up point to
the institution’s location. However, there is no way to
track and see the shuttle’s arrival and departure time.
According to a study [3], the drawback of public
transportation (such as buses and shuttles) is that thecommuter has an unknown amount of time to wait for
their transport’s arrival. This results in commuters
opting for private vehicles, which adds consequences inenvironmental, social, and economic terms [7].

Therefore, a shuttle booking and tracking system was
proposed as a solution for the current paper-based
shuttle management at INTI International College,
Penang. It enables the students to book the shuttle
service through the mobile application, where they can
decide whether the registration may proceed or not
depending on the seating availability in the shuttle.
Moreover, the students will be able to track the shuttle’s
real-time location as well as efficiently mark theirattendance when the shuttle is nearby. Lastly, thestudents will be able to receive notifications from bothadmins and drivers if the shuttle is unavailable.

A study [7] shows that (from a survey sent to 3378
correspondence), 59% of the students opt for public
transportation such as buses and shuttles. Distance, time,and economic capabilities are the most important factors
when choosing a mode of transportation. With that inmind, the free transportation offered by some schools©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
482

Andrea Tantay Gonzales, Kavitha Thamadharan, Neesha Jothi

Hoque, et al. [1] developed a bus tracking system for
the International Islamic University Chittagong in
Bangladesh. Instead of using a physical GPS receiver,
the researchers made use of an Android smartphone’s
embedded location-aware hardware, making the systemcost and configuration time lesser. The system ismobile-based, therefore, improving student accessibility
and access. There were also two added features in thesystem, namely the driver accident reporting and seatavailability checking. These features are beneficial tostudents, however, the way that the seat availability
checking was implemented may be hazardous (as thedriver has to continuously accept/deny students’requests, even while driving). The UI for the application
may also need a lot of time getting used to.

2. Literature Review
The current shuttle management system in INTIInternational College, Penang enables the students to
use the shuttle through Google Form registration every
new semester. Attendance marking and tracking were
paper-based; the student's name and matriculation
number must be written manually by the student afterriding the shuttle. There was also no administrator
dashboard for the overall management of the shuttle
service on campus.
Fig. 1 shows the pre-acceptance survey which was sentout to 100 students with 63 respondents. Most of thestudents voted on having a real-time shuttle tracking
application to help plan transportation ahead of time.

Shibghatullah, et al. [11] implemented Tracker, a realtime location tracking system that makes use of GPSaware devices to track the current location of public
transportation. It caters to a wider audience and ismobile-based. It has a much-improved UI with detailed
information (e.g., real-time traffic conditions, error
messages) which makes it easier for users to navigate
through the application, as well as a notification feature,
allowing the users to receive updates from theadministrators. In terms of improvement, the application
does not have a way to check seat availability, and thenotification feature can be further improved by adding apush notification alert.
Fig. 1. Pre-acceptance survey result
3.
A real-time location tracking application is a type ofsoftware application where the location data of any two
or more devices are updated in each period to the
database so that both devices’ location data informationsynchronizes with each other. There has been muchresearch about the implementation of real-time location
tracking applications in a school and campus’ public
transportation
system,each
withdifferent
implementation methods and techniques.

Methodology

The research methodology used for this study was theWaterfall Methodology (as shown in Fig. 2). During
requirements gathering, feedback regarding the current
shuttle service on the campus was compiled fromadministrators and students alike. The design phase
implements a non-working prototype of the application
with the main features. Once the idea was confirmed,
the working prototype was implemented based on the
pre-acceptance survey result and the initial design of theapplication. Testing was done afterward, where theworking application was sent to the end users for the
purpose of gathering UI and feature-related issues andfeedback. Lastly, the application was updated based on
the result of the usability testing.

Saad et al. [8] conducted research on a real-time oncampus public transportation monitoring system for
Universiti Teknologi Malaysia. The researchers
implemented a GPS receiver hardware and attached it to
the campus bus, where it retrieves the latitude, longitude,
and speed of the bus. The data is then synchronized tothe student web application. This system is not as
portable and accessible, however, since students mostly
use mobile applications nowadays. The system mustalso adapt to the current design and UI trends. Lastly,
each of the campus transportation vehicles must have
GPS receiver hardware, which may add to the cost.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
483

Efficient Campus Shuttle Tracking

Fig. 2. Research Methodology
Fig. 3 shows the administrator website, where theadmins can manage the campus’ shuttle services. It
contains various managerial features like shuttle anddriver registration, where the admin can input the
shuttle details (e.g., plate number, seating capacity) as
well as create an account for the driver (to be used on
the driver’s side of the application). Route management
is also included on the website, where the admin can
assign a registered shuttle and driver to the route andenter its pickup/dropoff schedule and location. The
admins can also view the number of bookings made by
students, as well as the registered students.

Fig. 4. Student Application
Lastly, a driver application was implemented to foster
communication between the student and the driver’s
location (refer to Fig. 5). A listing of all available andassigned journeys to the driver can be seen on the home
page. When the driver starts a journey, the application
saves the device’s current location and updates the
database with it whenever the shuttle or driver moves
100 meters from the last location. The driver application
also has an emergency reporting feature that triggers
whenever the driver cancels a journey or ends a journey
early. The submitted report sends a push notification to
the student’s application side and cancels the journey
without penalizing the student.

Fig. 3. Administrator Website
A student application was also implemented as shown in
Fig. 4. It allows the students to book a shuttle based on
the pickup/dropoff location, as well as view the shuttle’s
current real-time location. The application has a built-in
attendance feature where the student can mark their own
attendance if the shuttle is nearby. Unmarked attendance
may result in the “no show” penalty embedded in theapplication that denies the student from booking the
shuttle. Moreover, the student may book the shuttle aslong as there is available seating in the vehicle. The
student can also receive push notifications fromadministrators and drivers alike whenever there are
updates regarding the shuttle service on the campus (e.g.,
shuttle maintenance). Lastly, an option to cancel the
shuttle booking was also in place if the journey time
was not within 2 hours.

Fig. 5. Driver Application©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
484

Andrea Tantay Gonzales, Kavitha Thamadharan, Neesha Jothi

4.

minimalistic design and similarity in layout with otherapplications (thus contributing to its ease of use).
Further improvements of the system focused more on
optimization, minor element obstruction, and otherrequested features by the users (e.g., personalization,polyline for Google Maps).

Results and Discussion

For this study, the application underwent usability
testing. Existing research [4], [5] shows that the amountof usability testers should be 5 testers (given that the
study does not focus on quantitative results, in whichcase, 20 testers are required). A total of 8 participants,
however, were compiled for this study as it ensures that
almost all usability problems were covered as much as
possible.

5.

In this paper, an efficient shuttle tracking andmanagement application has been developed for INTIInternational College, Penang campus. The proposedsystem proved beneficial to both the campus
administrators and students, especially when it comes tomanaging shuttle bookings and tracking the shuttle
location, compared to its initial paper-based and manual
shuttle management system. Furthermore, theapplication was a success in terms of user acceptance asproven by the result of the usability testing.

Table 1 below shows the result and overall feedback of
the participants from the usability testing.
Table 1. Usability testing results
Participant
no.
1

2

3

4

5

6

7

8

Conclusion

Feedback- Some parts of the text instructions whenlogging in and registering are obstructed.
- Overall, the application has nice featuresand is smooth and fast.
- Application is easy to use andstraightforward with minimalistic design.
- There should be an option for weekly
booking of the shuttle.
- There should be a polyline location to
track the shuttle route.
- Improve the arrangement of the input
boxes on the booking page.
- The application is easy to use as thelayout is pretty similar to otherapplications.
- There was a missing toast message forcancellation of shuttle booking.
- The application is easy to navigate
around- More optimization in the future updates- Overall application experience issmooth, and the interface is nice.
- There should be more customization
options for personalized application
experience.
- There should be a password
confirmation during registration.
Overall,theapplicationisstraightforward and easy to use.
- The application is not overwhelming touse.
- It takes some time to load
(optimization).

References
1.

2.

3.

4.

5.

6.

7.

8.

It can be seen from the table that the overall impressionof the system was smooth, easy to navigate, andstraightforward, with most of the participants liking its

Hoque, F., Chakma, R., Mahtab, S. S., Akter, R., &Ahmed, S. S. (2020). Design and Developing
Real Time Interactive IIUC Bus Tracking System.
JournalofInnovationinComputer
Science and Engineering, 9(2).
Nama, G. F., Rasyidy, F. H., Arum, R., & Mardiana.
(2018).
A
Real-time
Schoolchild
ShuttleVehicle Tracking System Base on Android Mobileapps.
International
JournalofEngineering & Technology.
Narkhede, P. V., Mahalle, R. V., Lokhande, P. A.,
Mundane, R. M., & Londe, D. M. (2018). Bus
Tracking Sytem Based on Location Aware Services.
International
JournalofEmerging
Technologies in Engineering Research (IJETER), 6(3).
Nielsen, J. (2012). How Many Test Users in a Usability
Study? Retrieved 27 October, 2023, fromNielsenNormanGroup:
https://www.nngroup.com/articles/how-many-testusers/
Nielsen, J., & Molich, R. (15 November, 2020). 10
Usability Heuristics for User Interface Design.
Retrieved 15 May, 2023, from Nielsen Norman Group:https://www.nngroup.com/articles/ten-usabilityheuristics/
Ramya, S., Ajitha, P., Brindha, A., Gayathri, A., &Kavitha, V. (2021). QR Scan Based System for
School Bus Tracking. International Journal of
Scientific Research & Engineering Trends.
Romanowska, A., Okraszewska, R., & Jamroz, K.
(2019). A Study of Transport Behaviour of
Academic
Communities.
Sustainability.
325
Saad, A. S., Hisham, A. A., Ishak, M. I., Fauzi, M. M.,
Baharudin, M. A., & Idris, N. H. (2018).
Real-time
on-Campus
PublicTransportation
Monitoring
System.
2018
IEEE
14th

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
485

Powered by TCPDF (www.tcpdf.org)

Efficient Campus Shuttle Tracking

International Colloquium on Signal Processing & its
Applications (CSPA 2018).
9. Saif, M. A., Zefreh, M. M., & Torok, A. (2019). Public
Transport Accessibility: A Literature Review.
Periodica Polytechnica Transportation Engineering.
10. Sham, R. B., Hussin, A., Abdamia, N., & Mohd, M.
(2019).
Developinga
Pollution
Free
Environment
Frameworkthrough
Technology
Integration(e-Hailing
App).
7th
AMER
International Conference on Quality of Life.
11. Shibghatullah, A. S., Jalil, A., Wahab, M. H., Soon, J.
N., Subaramaniam, K., & Eldabi, T. (2021).
Vehicle Tracking Application Based on Real Time
Traffic.
International
Journalof
Electricaland Electronic Engineering & Telecommunications.

Authors Introduction
Andrea Tantay Gonzales
She received her Bachelor’s inComputer Science (in collaboration with
Coventry University) from the School ofComputing, INTI International College
Penang, Malaysia in 2023. She recently
finished her internship in Sophic
Automation SDN BHD, Malaysia.
Kavitha Thamadharan
She received her Master's from the
School of Computer Sciences, Universiti
Teknologi Malaysia in 2015. She iscurrentlyaLecturerin
INTIInternational College Penang, Malaysia.
Her research interest areas are
Information Security and Information
Systems.
Dr. Neesha Jothi
She received her PhD from the School ofComputer Sciences, Universiti Sains
Malaysia in 2020. She is currently an
Assistant Professor in UCSI University,
Malaysia. Her research interest areas are
Data Mining in Healthcare and Health
Informatics.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
486

GCN Analysis of Task-Based fMRI Data for Diagnosis of Schizophrenia
Tejaswini Thota, Reuben Stephen John

School of Computer Science and Engineering, Vellore Institute of Technology, Chennai, India
R Dhanush

School of Electronics Engineering, Vellore Institute of Technology, Chennai, India
Amutha S

School of Computer Science and Engineering, Vellore Institute of Technology, Chennai, India
Heshalini Rajagopal

Department of Electrical and Electronics Engineering, MILA University, 71800 Nilai, Negeri Sembilan, Malaysia

Abstract
This study focuses on schizophrenia, impacting 24 million people globally, characterized by distorted
reality and delusions. We propose utilizing a GraphSAGE model and a Graph Convolutional Network(GCN) model on a task-based fMRI data to differentiate schizophrenia instances, with and without
auditory-visual hallucinations, utilizing healthy participants for comparison. The study focuses on
identifying specific auditory stimuli that can help significantly differentiate individuals experiencing
hallucinations. The GraphSAGE model performed better by achieving an accuracy of 97% and 96% on
the specified stimuli.
Keywords: Schizophrenia, Hallucination, Graph Convolution, GraphSAGE, fMRI

2. Literature Review

1. Introduction

This section provides concise information about priorliterature which has been consulted as reference during
the model preparation. The application of graph basedneural networks in the analysis of neural activity is an
emerging area of study. Saeidi et al [1] have proposed an
end-end GCN network on the benchmark HCP dataset,
which makes use of statistical features extracted from theconnectivity matrices as a node feature. Additionally,
standard node embedding algorithms such as NetMF,
RandNE, Node2Vec and Walklets were used to extract
node features and their performance was analyzed. The
paper acknowledges the importance of individual
differences in brain function by testing the model on agender-based and gF score-based sub-dataset. The effectof batch size was also accounted for. The proposed
framework achieved an accuracy of 97.7% and concluded
that NetMF and RandNE algorithms outperformed therest.
Li, X. et al [2] have employed a custom framework
BrainGNN, composed of Ra-GNN and R-pool blocks
which are responsible for generating the node
embeddings. The node embeddings are aggregated with
its neighbors and the updated representations areprojected onto a learnable vector which is fed to aclassifier. The model was utilized on two distinct
datasets: the Biopoint Autism Study Dataset (Biopoint)
and the Human Connectome Project (HCP) 900 Subject
Release [3]. The primary objective for the Biopoint
dataset [4] was the classification of autism affectedindividuals and healthy controls. Conversely, for the
HCP dataset, the focus was on classifying seven

Functional Magnetic Resonance Imaging(fMRI) has
emerged as a powerful tool in unraveling the intricaciesof mental disorders. Among the various types of fMRI,
the task-based fMRI serves as an ideal method forunderstanding the functioning of neurons as it captures
real-time brain activity as individuals perform specificcognitive tasks. In the context of schizophrenia, theanalysis of task-based fMRI data has proven to provide
valuable insights that in turn aid the diagnosis andtreatment of this disorder.
Graph Convoluted Network (GCN) is a type of neuralnetwork architecture designed to process data structured
as graphs. In these graphs, the nodes represent entities,
and the edges represent the relationship between theentities. 𝐆𝐂𝐍′𝐬 leverage information from theneighboring nodes to perform traditional convolution
operations. This methodology is suitable for analyzing
fMRI data as they model dynamic changes in brain
connectivity. In our investigation, we designate regionsof interest (ROIs) as nodes, and the edges represent thefunctional connectivity between these brain regions. We
leverage pertinent node attributes to encapsulate thevariability in brain activity corresponding to specificstimuli, serving as distinctive features for our classifier.
This methodology enables us to unveil patterns andrelationships in brain connectivity, offering enhanced
insights into the complex dynamics of schizophrenia andthe responses of individuals affected by it to auditory
stimuli.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
487

Tejaswini Thota, Reuben Stephen John, R Dhanush, Amutha S, Heshalini Rajagopal

cognitive and behavioral task states. BrainGNN
demonstrates an enhancement in average accuracy
ranging from 3% to 20% for autism classification.
Additionally, it achieves an average accuracy of 93.4%
for the classification on Human Connectome Project
dataset.

first 5 volumes were discarded to account for signal
stabilization. Brain extraction was done using the BET
tool. The functional data was then subjected to motion
correction using the MCFLIRT algorithm and spatialsmoothening (FWHM=5mm) was also applied (Fig 1.b) .
FLIRT was used to register the low-resolution functional
images to a high-resolution scan, and registration of thehigh-resolution scan to a standard MNI152) image. Using
the transformation matrix obtained from the previous
steps, we registered the functional sequence to thestandard MNI space (Fig 1.c).

3. Methodology
3.1. fMRI dataset and preprocessing
We obtained task fMRI data for 71 participants, 46
schizophrenic patients and 25 healthy controls subjectedto three experimental conditions, word lists, sentence lists
and reversed speech. White noise was the low-level
control condition to which the task results were compared
to. The schizophrenic patients are further classified into
two classes, 23 subjects who experience auditory visual
hallucinations (AVH+) and 23 subjects who do not(AVH-) (Table 1). The data was obtained using a 3T
Philips Ingenia scanner employing a T2*-weighted echoplanar imaging (EPI) sequence. The acquisition consistedof 341 volumes, and the following parameters wereapplied: a repetition time (TR) of 2000ms, echo time (TE)of 30ms and a flip angle of 70°. The in-plane resolution
was set to 3.5 × 3.5 mm, with a field of view (FOV)measuring 238 × 245 mm. The slice thickness was 3.5mm, and an inter-slice gap of 0.75 mm was maintained.
Slices, totaling 32 per volume, were acquired in aninterleaved order parallel to the AC-PC plane [5]. Refer(Fig. 1.a) for raw data.

Fig 1 (a) depicts raw fMRI data. Fig 1 (b)- depicts fMRI
data obtained after motion correction, brain extraction
and smoothenting. Fig 1 (c) depicts the fMRI data post
registration to MNI152 template3.2. Functional Graph Construction
The preprocessed functional sequence underwent
parcellation using the AAL atlas [7], dividing the brain
into 116 Regions of Interest (ROI's). Based on the studies
conducted by Soler-Vidal et al [5], the regions Left
Superior Temporal Cortex, Occipital Cortex, Precuneus
and Lingual Gyrus were found to exhibit a failure ofdeactivation in schizophrenic patients. The Left Superior
Temporal Cortex is associated with language processing
and speech comprehension. The Occipital Cortex and the
Lingual Gyrus play a key role in visual processing,
especially that of shapes and written words respectively.
Precuneus is associated with elements of self-awarenessand visuospatial processing. These regions areencompassed within the AAL atlas thereby was used forparcellation. The time points associated with specificactivities were then extracted, and voxel information wascollected. To derive the mean time series for each node,a bootstrapping technique was applied by randomly
selecting one-fourth of the voxels within each ROI. This
process is repeated ten times for augmentation.

Table 1: Participants description

The statistical features are obtained using the tsfreshalgorithm (shown in Fig. 2), which integrates elementsfrom hypothesis tests and feature significance testing [8].
Each feature vector produced undergoes a separate
evaluation for its relevance to the specified target byutilizing p-values. The features derived from tsfresh
encompass a comprehensive set of characteristics fromboth basic and advanced aspects of the time series.which
are represented as "absolute_sum_of_changes,",

The preprocessing of the data was carried out using the
FEAT module of FSL(FMRIB Software Library) [6]. The©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
488

GCN Analysis on task-based"benford_correlation", "variance," "standard deviation,""skewness,”, "quantile", “count above mean”, “countbelow mean”, “longest strike above mean”, “cubic
complexity” and “complexity index” [1]. We make use of
the Pearson correlation scores between the time series
with all nodes contained in the graph. This helps encode
information about the interconnectivity of the time seriesof each node that we have defined corresponding to thesetime points.

(𝑙+1)

ℎ𝑣

(𝑙)

(𝑙)

= 𝜎 (𝑊𝑖 ∙ 𝐶𝑂𝑁𝐶𝐴𝑇 (ℎ𝑣 , 𝐴𝐺𝐺𝑅𝐸𝐺𝐴𝑇𝐸𝑘 ({ℎ𝑢 ,(1)∀𝑢 ∈ 𝑁(𝑣)})))(𝑙)ℎ𝑣 denotes the node embedding for node 𝑣 at layer 𝑙.
𝑊𝑙 is the weight matrix for layer 𝑙. 𝜎 denotes the RELU
activation function and CONCAT is the concatenation
operation. (Fig 3.) denotes an overview of the modelarchitecture.

Fig. 3 Model architectureb) GCN:The proposed model architecture comprises threegraph convolutional layers. To enhance stability andaccelerate the training process, batch normalization isincorporated after each graph convolutional layer.
Rectified Linear Unit (ReLU) activation functions follow
each graph convolutional layer, introducing non-linearity
to the model. Dropout regularization is systematicallyapplied after each activation function during training to
mitigate overfitting risks. Global mean pooling isincorporated to aggregate node representations across theentire graph. The model's learnable parameters are furthertransformed by a linear layer to project the aggregatednode representations into the final output space. Lastly,
the model outputs are subject to a SoftMax activation
facilitating the derivation of class probabilities which is
given in Eq. (2).

Fig. 2 Flowchart representation of methodology
3.3. Model Architecturea) GraphSAGE:The 𝐺𝑟𝑎𝑝ℎ𝑆𝐴𝐺𝐸 model [9] utilizes a sequence of
three SAGE convolutional layers Eq. (1) to capture
hierarchical representations from graph-structured data.
Beginning with the extraction of node embeddings, eachlayer applies a SAGE convolution Eq. (1) followed by a
ReLU activation. The model concludes with a readoutlayer, employing global mean pooling to aggregate node
embeddings and produce a tensor summarizing graphlevel information. A dropout layer with a probability of
0.5 mitigates overfitting during training. The final linear
layer transforms the aggregated embeddings forclassification into the specified number of classes.

(𝑙+1)

ℎ𝑖

= 𝜎( ∑𝑗∈𝑁(𝑖)

1 (𝑙) (𝑙)𝑊 ℎ𝑗 )
𝑐𝑖𝑗

(2)

(𝑙+1)

ℎ𝑣
denotes the updated representation of node
layer (𝑙 + 1) . 𝜎 denotes the activation function. ∑denotes the weighted sum function. 𝑊 (𝑙) is the weight(𝑙)matrix at layer 𝑙 and ℎ𝑗 is the feature representation ofnode j. The normalization term is denoted by 𝑐𝑖𝑗 .
3.4. Training and TestingThis study utilized an 80-10-10 split for training,validation, and test sets, respectively. The validation split
was conducted on the training data. To augment the test
data, one-fourth of the voxels from the Regions ofInterest (ROIs) were randomly selected and bootstrapped
ten times. In the training set, four subjects from the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
489

Tejaswini Thota, Reuben Stephen John, R Dhanush, Amutha S, Heshalini Rajagopal

healthy class (class 0) and three each from AVH- (class1) and AVH+ (class 2) were included. This was done fortwo different experimental conditions namely “sentences”
and “reversed”. The model makes uses of Adam
optimizer with a learning rate of 0.001.
Cross-Entropy Loss was chosen as the loss function as itis primarily used for multi-class classification.

effective in situations where there is an imbalance
between classes.

3.5. Model Evaluation

The 𝐺𝑟𝑎𝑝ℎ𝑆𝐴𝐺𝐸 model was trained using functionalgraphs corresponding to the 'sentences' and 'reversed'
conditions. It achieved an accuracy of 97% after 20
epochs for the 'sentences' condition and a 95% accuracy
after 20 epochs for the 'reversed' condition. (Table 2 andTable 3) summarizes the performance of this model.

𝑀𝐶𝐶 =a) Precision: It denotes the extent to which a model
accurately recognizes positive instances among all the
instances it labeled as positive, highlighting its capability
to reduce the occurrence of false positives.

𝑇𝑃
𝑇𝑃 + 𝐹𝑃

Table 2: Evaluation metrics for GraphSAGE
Evaluation
Metrics
MCC
Accuracy (%)
F1 Micro (%)
F1 Macro (%)
F1 Weighted (%)
Bal. accuracy (%)
Precision
Recall score(3)b) Recall: Recall is an assessment of a model'seffectiveness in recognizing all positive instances among
the entire set of actual positive instances. It showcases themodel's capacity to reduce false negatives and encompass
all pertinent outcomes.

𝑅𝑒𝑐𝑎𝑙𝑙 =𝑇𝑃
𝑇𝑃 + 𝐹𝑁(4)

𝑇𝑃 + 𝑇𝑁𝑇𝑃 + 𝐹𝑁 + 𝑇𝑁 + 𝐹𝑁

Evaluation
Metrics
MCC
Accuracy (%)
F1 Micro (%)
F1 Macro (%)
F1 Weighted (%)
Bal. accuracy (%)
Precision
Recall score(5)d) Balanced Accuracy: Balanced Accuracy (BA) is ametric that considers class imbalances when assessing theoverall correctness of a classification model by
calculating the mean sensitivity across all classes.
𝑇𝑃
𝑇𝑁
𝐵𝐴 = 0.5 ∗ ()+()
𝑇𝑃 + 𝐹𝑁
𝑇𝑁 + 𝐹𝑃SENTENCES

REVERSED

0.957097.18%
97.18%
96.94%
97.14%
96.96%
0.97180.9696

0.935695.77%
95.77%
95.29%
95.92%
95.92%
0.94900.9592

Table 3: Evaluation metrics for GCNc) Accuracy: It signifies the gauge of the model'saccuracy in correctly categorizing instances across allclasses, providing insight into the overall correctness of
its predictions.

𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 =(8)

4. Result and Analysis

Evaluation metrics are described from Eqs. (3), (4), (5),(6), (7) and (8) such as precision, recall, accuracy,balanced accuracy, F1-scores and 𝑀𝐶𝐶 were used toassess the model’s performance.

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =𝑇𝑃 ∗ 𝑇𝑁 − 𝐹𝑃 ∗ 𝐹𝑁√(𝑇𝑃 + 𝐹𝑃)(𝑇𝑃 + 𝐹𝑁)(𝑇𝑁 + 𝐹𝑃)(𝑇𝑁 + 𝐹𝑁)

SENTENCES

REVERSED

0.895592.95%
92.95%
93.02%
92.97%
94.31%
0.9232
0.94310.743681.69%
81.69%
78.17%
81.30%
78.18%
0.87610.7818Fig. 4(a) and 4(b) depict the changes in training andvalidation loss, as well as training and validation
accuracy across different epochs for 𝐺𝑟𝑎𝑝ℎ𝑆𝐴𝐺𝐸 modelwith ‘sentence’ condition.

(6)e) F1 Score: It evaluates the model's capacity toaccurately recognize positive instances, taking into
account both precision and recall

𝐹1 𝑆𝑐𝑜𝑟𝑒 =2 ∗ 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 ∗ 𝑅𝑒𝑐𝑎𝑙𝑙
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙(7)f) MCC (Matthew’s Correlation Coefficient): Factors
in true positives, true negatives, false positives, and false
negatives, offering a well-balanced metric, particularlya)b)
Figure 4. GraphSAGE Training and validation a)
Loss b)Accuracy- for stimulus 1.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
490

GCN Analysis on task-based

for the 'reversed' condition. (Table 3) summarizes theperformance of this model.
Figs 4, 5, 6, 7 summarizes the trends of training loss
and accuracy vs validation loss and accuracy over thecourse of epochs. The decreasing trend over epochs withrespect to training loss is indicative of effective modeltraining. Concurrently, the validation loss which
represents the model's performance on unseen data
steadily decreases, signifying successful generalization.
The observed increasing trend in both training andvalidation accuracy plots suggests that the model islearning and generalizing effectively over epochs. The
rising training accuracy indicates successful adaptation to
the training data, while the parallel increase in validation
accuracy implies robust generalization to unseen data.
This
positive
trajectory
signifies
continuous
improvement in the model's performance, showcasing its
ability to capture underlying patterns in the data.

a)b)
Figure 5. GraphSAGE Training and validation a)
Loss b) Accuracy- for stimulus 2
Fig 5(a) and 5(b) denote the changes in training andvalidation loss, as well as training and validation
accuracy over epochs for 𝐺𝑟𝑎𝑝ℎ𝑆𝐴𝐺𝐸 model with‘reversed’ condition.

a)b)
Figure 6. GCN Training and validation a) Lossb)Accuracy- for stimulus 1a)b)
Figure 8. Confusion Matrix of GraphSAGE for a)
Stimulus 1 b) Stimulus 2.

a)b)Figure 7. GCN Training and validation a) Loss b)
Accuracy- for stimulus 2
Fig. 6(a) and 6(b) depict the changes in training andvalidation loss, as well as training and validation
accuracy across different epochs for 𝑮𝑪𝑵 model with‘sentence’ condition. Fig 7(a) and 7(b) denote thechanges in training and validation loss, as well as training
and validation accuracy over epochs for 𝑮𝑪𝑵 model with‘reversed’ condition.

a)b)

Figure 9. Confusion Matrix of GCN for a) Stimulus
1 b) Stimulus 2.

The GCN model was trained using functional graphs
corresponding to the 'sentences' and 'reversed' conditions.
It achieved an accuracy of 92% after 30 epochs for the'sentences' condition and 81% accuracy after 30 epochs©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
491

Tejaswini Thota, Reuben Stephen John, R Dhanush, Amutha S, Heshalini Rajagopal

Table 4: Comparative Results Analysis
Sentences
Metrics
Accuracy
Precision
Recall
F1-Score

GraphSAGE
AVH- AVH+
97%
0.97
0.94
1
1
1
0.91
98%
97%
95%
HC

HC
1
0.88
93%

Reversed
GCN
AVH- AVH+
93%
0.89
0.88
1
0.95
94%
91%

The confusion matrix outlines true positives, true
negatives, false positives, and false negatives, and is used
to assess prediction accuracy. Fig 8.a and 8.b depict theconfusion matrices of the 𝐺𝑟𝑎𝑝ℎ𝑆𝐴𝐺𝐸G model for boththe sentences and reversed conditions respectively. Fig
9.a and 9.b represent the matrices of the 𝑮𝑪𝑵 model forthe sentences and reversed conditions respectively.

GraphSAGE
AVH- AVH+
96%
1
0.89
0.95
0.97
1
0.91
98%
94%
93%
HC

HC
1
0.88
93%

GCN
AVH- AVH+
82%
1
0.63
0.47
1
64%
77%

5. Conclusion
Our study employs two different graph convolutionbased architectures namely 𝐺𝑟𝑎𝑝ℎ𝑆𝐴𝐺𝐸 and 𝑮𝑪𝑵. Boththese architectures examine the distinctions in brain
activity of 71 participants. The participant populationconsists of healthy controls, schizophrenic patients withhallucination (AVH+) and schizophrenic patients without
hallucination (AVH-). These groups were subjected tothree different auditory stimuli namely ‘words’,‘sentences’ and ‘reversed’. This analysis particularly
focuses on the ‘sentences’ and ‘reversed’ stimulus on theparticipant group. Our study primarily focusses to
identify the activation differences among the two stimuli.

On comparison of the model metrics of 𝐺𝑟𝑎𝑝ℎ𝑆𝐴𝐺𝐸, themodel exhibits proficiency in distinguishing between the'sentences' and 'reversed' conditions. Notably, for'sentences’ performs better than 'reversed' across all
metrics, refer (Table 2). These results underscore themodel’s discriminative capability, effectively capturing
nuanced distinctions in functional graphs associated withthe specified conditions. The overall performance isobserved to be enhanced for the 'sentences' condition andthis highlights the model's efficacy in this specificcognitive context.

The differences would be leveraged to classify among thethree classes. The fMRI data was preprocessed andregistered to the MNI152 template in order to bring it toa standard coordinate space. Parcellation was applied tothe preprocessed data through the AAL atlas. This atlascontains regions such as Lingual Gyrus and Left temporal
auditory cortex which play a primal role in visual
processing and speech perception [10].

The performance metrics for the 'sentences' and 'reversed'
conditions for the GCN are summarized in (Table 3). The
MCC, a measure considering true and false positives andnegatives, is notably high for both conditions, indicatingstrong classification performance. The 'sentences'condition exhibits higher values across these metrics,
suggesting enhanced classification ability for sentences
stimuli. Overall, these metrics collectively affirm theeffectiveness of the model in distinguishing between thespecified conditions, with 'sentences' achieving
significantly enhanced performance.

In the studies conducted by Soler-Vidal et al [5] on thedataset, it was concluded that out of the three conditions,
included in the experiment, the sentences condition andthe reversed condition together varied significantly fromthe words condition. Further the activation patterns
observed for ‘sentences’ and ‘reversed’ stimuli did notshow clear distinction. They also concluded that there
was no appropriate distinguishing factor between the
AVH+ and AVH- participants. Moreover, their studies
have highlighted a reduced activation of the left primary
auditory cortex when subjected to sentences and reversed
speech. The auditory cortex plays a primal role inreceiving and processing auditory information, including
speech comprehension. A reduced activation of thisregion indicates disruptions in the neural pathways
associated with auditory perception and interpretation[11][12].

The comparative analysis for the two models issummarized in (Table 4). From the metrics, it can beinferred that 𝐺𝑟𝑎𝑝ℎ𝑆𝐴𝐺𝐸 performs better by achieving
higher accuracy of 97% and 96% for both the conditions
while the GCN model achieves an accuracy of 93% and
82% for sentences and reversed condition respectively.
Between the given conditions, we can infer that the‘sentence’ condition results in a better accuracy for boththe models and can therefore be considered a moresuitable distinguishing stimuli among the subjects.

Through our analysis we were able to find that the graphconvolution-based 𝐺𝑟𝑎𝑝ℎ𝑆𝐴𝐺𝐸 model performs better
than the GCN model. Further, our results suggest subtle
distinctions within the AVH+ and AVH- classes, for boththe conditions as well as effectively distinguishing©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
492

GCN Analysis on task-based

6.

M.W. Woolrich, S. Jbabdi, B. Patenaude, M. Chappell, S.
Makni, T. Behrens, C. Beckmann, M. Jenkinson, S.M.
Smith. Bayesian analysis of neuroimaging data in FSL.
NeuroImage, 45:S173-86, 200
7. Automated anatomical labelling atlas 3. Rolls, E. T.,
Huang, C. C., Lin, C. P., Feng, J., & Joliot, M.,
Neuroimage,
2020,
206,
116189,doi:10.1016/j.neuroimage.2019.116189
8. Christ, M.; Kempa-Liehr, A.W.; Feindt, M. Distributedand parallel time series feature extraction for industrial big
data applications. arXiv 2016, arXiv:1610.07717.
9. Veličković, Petar, et al. "Graph attention networks." arXiv
preprint arXiv:1710.10903 (2017).
10. Shergill, S. S., et al. "SES04. 03 Auditory hallucinations:
Mapping the neural network using functional magnetic
resonance imaging." European Psychiatry 15.S2 (2000):
242s-242s.
11. Hamilton, Will, Zhitao Ying, and Jure Leskovec.
"Inductive representation learning on large graphs."
Advances in neural information processing systems 30(2017).
12. Ford, Judith M., et al. "Tuning in to the voices: a multisite
FMRI study of auditory hallucinations." Schizophrenia
bulletin
35.1(2009):
58-66.
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0276975

between healthy individuals and those with schizophrenia.
We speculate that the minor differences among thesentences and reversed conditions could be used as a
distinguishing factor among people with and without
auditory-visual hallucinations. Our findings are in linewith the results obtained by Soler-Vidal et al [5] andadditionally we conclude that though the activation
patterns for the ‘sentences’ and ‘reversed’ stimuli aresimilar, there are intricate differences which can beleveraged by our model to understand the effect ofauditory-visual hallucination on schizophrenic patients.
The current study can be further extended to specificregions that show significant changes in the activation
functions. These specific regions with the use of more
complex graph-based architecture and larger group of
subjects will provide notable insights in understanding
the changes in auditory and speech perceptions amongindividuals affected with schizophrenia with ranging
symptoms.

Acknowledgement
The first and second authors express their gratitude to
Vellore Institute of Technology (VIT), Chennai, forproviding valuable computational facilities and serving
as a source of motivation. Additionally, they extend
they’re thanks to their respective faculties for unwavering
support and guidance throughout the research endeavor.

Authors Introduction
Ms.Tejaswini Thota
She is currently pursuing his B.Tech degreein Computer Science with specialization in
Artificial Intelligence and Robotics from
School of Computer Science andEngineering,
VelloreInstituteofTechnology, Chennai, India.

References
1.

2.

3.

4.

5.

Saeidi, M.; Karwowski, W.; Farahani, F.V.; Fiok, K.;
Hancock, P.A.; Sawyer, B.D.; Christov-Moore, L.;
Douglas, P.K. Decoding Task-Based fMRI Data with
Graph Neural Networks, Considering Individual
Differences.
Brain
Sci.
2022,
12,
1094.
https://doi.org/10.3390/brainsci12081094
Li, X., Zhou, Y., Dvornek, N., Zhang, M., Gao, S., Zhuang,J., ... & Duncan, J. S. (2021). Braingnn: Interpretable brain
graph neural network for fmri analysis. Medical Image
Analysis, 74, 102233.
Van Essen, D.C., Smith, S.M., Barch, D.M., Behrens, T.E.,
Yacoub, E., Ugurbil, K., Consortium, W.M.H., et al.: The
wu-minn human connectome project: anoverview.
Neuroimage 80, 62–79 (2013)
Venkataraman, A., Yang, D.Y.J., Pelphrey, K.A., Duncan,
J.S.: Bayesian community detection in the space of grouplevel functional differences. IEEE transactionson medical
imaging 35(8), 1866–1882 (2016)
Joan Soler-Vidal and Paola Fuentes-Claramonte and Pilar
Salgado-Pineda and Nuria Ramiro and María Ángeles
García-León and María Llanos Torres and Antonio
Arévalo and Amalia Guerrero-Pedraza and Josep Munuera
and Salvador Sarró and Raymond Salvador and Wolfram
Hinzen and Peter McKenna and Edith Pomarol-Clotet(2022). Brain correlates of speech perception inschizophrenia patients with and without auditory
hallucinations.
OpenNeuro.
[Dataset]doi:
doi:10.18112/openneuro.ds004302.v1.0.1

Mr. Reuben Stephen John
He is currently pursuing his B.Tech degree in
Computer Science with specialization in
Artificial Intelligence and Robotics from
School of Computer Science andEngineering,
VelloreInstituteofTechnology, Chennai, India.
T

Dr. Dhanush R.
Dr. Dhanush R is an Associate Professor in
the School of Electronics Engineering, VIT
Chennai. His research interests include
Neuroscience,
Biomechanics.
Motor
learning, Motor Control.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
493

Powered by TCPDF (www.tcpdf.org)

Tejaswini Thota, Reuben Stephen John, R Dhanush, Amutha S, Heshalini Rajagopal

Dr. Amutha S
Dr. Amutha S is currently working as an
Assistant Professor, School of Computer
Science and Engineering, Vellore Institute ofTechnology, Chennai, India. Her area of
research includes Data mining, Machinelearning, Pattern detection, HCI, IoT/IoMT
and Deep learning.

Dr. Heshalini Rajagopal
She received her PhD and Master’s
degree from the Department of ElectricalEngineering, University of Malaya,
Malaysia in 2021 and 2016 respectively.
Her research interest includes image
processing, artificial intelligence andmachine learning©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
494

AR Based Application for Campus Navigation
Renuka Devi Rajagopal, Akshay Sripriya, Shakthi B, Manoj Rathinam S

School of Computer Science and Engineering, Vellore Institute of Technology, Chennai, India
Heshalini Rajagopal

Institute of Computer Science and Digital Innovation, UCSI University, 56000 Kuala Lumpur, Malaysia

Abstract
Modern campuses, encompassing diverse educational, medical, and commercial environments, often pose intricate
navigation challenges to both newcomers and regular visitors. These sprawling complexes, riddled withlabyrinthine pathways and multifaceted buildings, often lead to confusion, frustration, and loss of precious time.
Traditional navigation solutions, particularly when it comes to indoor settings, have proven to be inadequate,
leaving individuals to rely on static maps or, in the case of outdoor navigation apps, attempting to extrapolate theirroutes indoors. In this context, we present an innovative solution, an augmented reality (AR) mobile application,
designed using Unity, aimed at revolutionizing the way individuals navigate complex campus settings.
Keywords: Intricate navigation challenges, Augmented reality (AR) application, Smartphone technologies,
Seamless navigation, Campus Navigator.

groundbreaking augmented reality (AR) application
leverages the full potential of smartphone technologies
to provide users with precise and real-time geolocation
data. Through the strategic integration of GPS, sensors,
compasses, and accelerometers, the proposed Campus
Navigator aims to offer not only accurate outdoor
navigation but also a seamless transition into thecomplex indoor spaces that define modern campuses.
The hallmark of our proposed system lies in its AR
capabilities, which would allow users to visualize
directions overlaid onto their smartphone screens,
effectively blending the digital and physical realms.
Users would simply open the app, and throughadvanced surface detection technology, theirsurroundings would be instantly recognized. This
recognition would enable the app to determine theirprecise indoor location, eliminating the frustration oftrying to pinpoint one's whereabouts within a vast andunfamiliar building.
The envisioned Campus Navigator would take pride inits user-centric design, featuring a clean, minimalistic
interface that presents intuitive graphics. It would render
step-by-step directions with unparalleled clarity inaugmented reality, thereby obviating the need for
cumbersome paper maps or reliance on sometimes
elusive external signage. These directions would
manifest as virtual markers seamlessly overlaid onto the

1. Introduction
Outdoor navigation applications, leveraging the powerof GPS and geolocation technology, have indeed madesignificant strides in guiding users along the sunlit paths
and open spaces that define a campus's exterior. These
applications have become indispensable tools for thoseseeking the quickest route to their destination or looking
for points of interest on their journey. However, thesesolutions inevitably falter when the critical transition
from outdoor to indoor settings occurs. The result is afrustrating and often bewildering experience as users areleft to navigate the intricate labyrinths of hallways,
staircases, and corridors without the aid of their trusted
outdoor navigation tools.
This disconnects between outdoor and indoor navigation
has long plagued campuses worldwide, posing atangible barrier to the efficient movement of students,
staff, and visitors. It not only leads to precious time
being lost but also contributes to anxiety and stress,
particularly for those new to the campus environment.
Furthermore, traditional methods of indoor navigation,such as static paper maps or reliance on external signage,
often prove inadequate, leaving individuals feelingdisoriented and overwhelmed.
Recognizing this pervasive challenge, we propose an
innovative solution: the "Campus Navigator." This

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
495

Renuka Devi Rajagopal, Akshay Sripriya, Shakthi B, Manoj Rathinam S, Heshalini Rajagopal

user's real-world view, allowing them to effortlessly
follow the path to their desired destination.
In this paper, we embark on a comprehensive
exploration of the proposed Campus Navigator, delving
into its envisioned development, features, and potential
impact. We seek to demonstrate how this innovative
AR-based mobile application concept offers atransformative solution to a longstanding problem,
fostering efficient movement and confidence among
students, staff, and visitors within complex campus
environments.

interactive
campusenvironment
information.
Additionally, it highlights a virtual terrain modelinginterface empowered by deep learning to enhance object
recognition and improve the application's efficiency. Yu,
K. M., Chiu. [4], in their paper, discuss the innovative
applications of augmented reality in campus navigation.
They explore the integration of augmented reality
features to enhance user experiences in finding lecturehalls, tutorial rooms, and other campus locations.
Lautenschläger, B.. [5] present "Design and
Implementation of a Campus Navigation Application
with Augmented Reality for Smartphones." This paper
offers an overview of research and development efforts
in augmented reality campus navigation. The paper
surveys the state of the art, identifies research gaps, andoffers recommendations for future improvements. The
paper by Pawade, D., et al. [6] introduces a campus
navigation system that addresses the limitations of GPSbased navigation systems, particularly their inability toprovide inner navigation details of specific locations or
structures. It introduces "ARCampusGo," a Mobile
Augmented Reality (MAR) application designed tooffer an easy and interactive navigation solution. Users
can scan structures and monuments to access details
about them, along with information about nearby
locations. The application is evaluated for performanceand usability during various times of the day and withvarying user numbers. "ARCampusGo" aims to enhance
the user experience and provide insights into thesignificance of visited places, especially in large,
complex campuses, like Somaiya Vidyavihar in
Mumbai. In a doctoral dissertation by Hew, T. W. [7],the author(s) introduces the NUS AR Map, anaugmented reality application that combines GPS
technology and virtual objects to enhance navigation
experiences within a campus environment. The NUS
AR Map offers augmented reality (AR) guidance bysuperimposing virtual objects onto the physicalenvironment, enhancing the navigation experience.
However, this system has some limitations. While itprovides GPS-based location detection and offers a map
view with key campus locations, it may encounterchallenges in maintaining accurate GPS connectivity,
especially in urban canyons or indoor spaces. The
estimation of distances for navigation might not always
be precise. Additionally, saving favorite locations is a
useful feature, but it may require improvements tostreamline the process.

2. Literature Review
In the literature survey, existing research andapplications related to campus navigation, augmented
reality, and mobile-based navigation solutions were
explored. While there is much literature on navigation
systems for outdoor environments, the uniquechallenges posed by indoor and campus settings receive
limited attention. The paper by
Lu, F., Zhou, et. al.
[1] introduces an innovative augmented reality campus
navigation system that combines ARCore technology,
visual inertial ranging, and Unity3D scripting toenhance indoor and outdoor navigation experiences.
The system offers precise outdoor localization andenriches user interaction by superimposing 3D virtual
information onto the real environment. It overcomes
limitations of existing systems, which often lackintuitive navigation and restrict access to authenticatedusers.
Another paper by Kuwahara, et. al. [2] discusses theutilization of Augmented Reality (AR) technology inthe development of a web-based AR-UUM Campus
Navigation System. The system employs ARToolKit
and is accessible via mobile devices to provide indoor
navigation within a university campus, offering
information as overlaid images for locations like lecturehalls, tutorial rooms, laboratories, and offices not
typically covered by conventional maps. Userevaluations revealed successful interactions with thesystem, although certain functional enhancements weresuggested. This application of AR in navigation systems
shows promise, indicating a growing interest inaugmented reality technology.
In the paper by Lin. C. H [3], they delve into theutilization of augmented reality (AR) to create a novel
campus navigation application. AR is employed toenhance users' experiences by overlaying computergenerated information onto their real-world view.
Leveraging advanced AR technologies like computer
vision and object recognition, the paper introduces

The work by Chou, T. L, et al. [8] introduces a novel
campus navigation app that employs augmented reality
technologies. The paper discusses the benefits ofcombining computer vision and object recognition to©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
496

AR Based Application forprovide users with an interactive and immersive
navigation experience.

of interest, and outdoor pathways, werecollected and integrated into theapplication. This data formed thefoundation for accurate navigation andguidance.
(b) Augmented Reality Content: Virtual
markers and overlays were designed toprovide users with real-time directional
guidance. Augmented reality content was
created to enhance the user experience.

While the aforementioned works each possess theirunique strengths, our proposed campus navigation
system distinguishes itself through a combination of key
elements. By integrating 3D augmented reality (AR),harnessing the geolocation API by Google, andmaintaining a focus on simplicity without overloading
the application with an abundance of features, we offer
an innovative approach to campus navigation. Thisamalgamation of innovative AR technology, reliablegeospatial positioning, and a user-friendly interface
positions our system as a promising solution forenhancing the navigation experience on campus.

(iii)

Testing and Validation:(a) User Testing: The Campus Navigatorapplication underwent rigorous testingwith diverse user groups, including
students, faculty, and visitors. Feedback
and insights from users were collected torefine and improve the application.

3. Methodology(b) Accuracy Evaluation: The accuracy ofgeolocation data and the effectiveness ofaugmented reality overlays was assessed
through real-world testing scenarios.

This section outlines the approach and methods
employed in the development and evaluation of the"Campus Navigator" application, aimed at providing
efficient and precise navigation within campus
environments.
(i)

(ii)

(iv)

System Design and Development:(a) Requirement Analysis: The initial phase
involved an in-depth analysis of therequirements and objectives of theCampus Navigator application. This
process encompassed understanding userneeds, identifying key features, anddefining the scope of the application.
(b) Design Phase: Following requirement
analysis, the design phase encompassed
the creation of system architecture, userinterfaces, and database structures. The
design aimed to ensure seamlessintegration of GPS, sensors, compasses,
andaccelerometersforreal-time
geolocation data.
(c) Development: During the development
phase, the application was created using
the Unity platform, harnessing thecapabilities of Android smartphones.
Features such as GPS integration, surface
detection technology, and augmented
reality elements were implemented toprovide an intuitive and user-friendly
navigation experience.

Performance Evaluation:(a) Efficiency Assessment: The application's
efficiency in reducing navigation time andenhancing user satisfaction was evaluatedthroughcomparative
studieswithtraditional navigation methods.
(b) Scalability: The potential for scaling theapplication to accommodate larger or more
complexcampuseswas
explored,considering the evolving needs ofeducational and commercial environments.

(v)

Data Analysis and Reporting:(a) Data Analysis: The data collected during
testing and validation phases wereanalyzed to assess the application's
performance,
accuracy,
andusersatisfaction.
(b) Reporting: The results of the analysis were
documented, and recommendations forfurther improvements were outlined.

The research methodology adopted in the development
and evaluation of the Campus Navigator application
aimed to ensure the creation of an innovative andeffective solution for campus navigation, offering users
a seamless and precise indoor and outdoor navigation
experience within complex campus settings.

Data Collection and Integration:(a) Geospatial Data: Geospatial data of thecampus, including building layouts, points©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
497

Renuka Devi Rajagopal, Akshay Sripriya, Shakthi B, Manoj Rathinam S, Heshalini Rajagopal

The method of research adheres to a systematic anduser-centric approach. Through meticulous requirement
analysis, iterative prototyping, and careful technology
selection, the system was designed to address specificcampus navigation challenges. Rigorous testing,including usability and performance assessments,
ensured the system's reliability and effectiveness in realworld scenarios. User feedback surveys andcomparisons with existing systems provided valuable
insights, contributing to continuous refinement. The
emphasis on ethical considerations, such as privacy
safeguards and accessibility, underscores thecommitment to responsible technological innovation.
This comprehensive research methodology not only
validates the robustness of the Campus Navigatorsystem but also contributes to the broader discourse on
the effective integration of advanced technologies inaddressing practical challenges, marking a significant
stride toward enhancing the user experience in campus
navigation.

4.3. Geospatial Anchoring for Predefined
Locations
Predefined Geospatial Anchor:
The core of the Campus Navigator system relies onpredefined geospatial anchors. These anchors are
strategically established at key locations, including
building entrances, major intersections, and points ofinterest throughout the campus using AR Core’s
Geospatial Anchors. Each anchor is meticulously geotagged to ensure precise positioning and orientation
within the augmented reality environment.
Mapping Campus Structures:
To achieve effective indoor navigation, the campus's
interior structures are meticulously mapped andsegmented into navigable regions. Each region
corresponds to a predefined geospatial anchor. Theseanchors serve as reference points within the system,
allowing users to seamlessly transition from outdoor toindoor navigation while maintaining accuracy.

4. Proposed System

4.4. Augmented Reality (AR) Wayfinding

4.1. Geolocation API Integration
AR-Based Arrow/Pointer/Waypoint Overlay:
The hallmark of the Campus Navigator system is its useof augmented reality (AR) technology to provide users
with clear and intuitive wayfinding guidance. Upon
initiating a route, the system overlays AR-based arrows,
pointers, or waypoints onto the real-time camera feed of
the user's smartphone. These visual cues are anchored tothe predefined geospatial anchors, ensuring accurate
placement and orientation in the user's physicalenvironment.

The proposed Campus Navigator system leverages thepower of Geolocation APIs, with a specific focus on the
Google Geolocation API, to accurately determine theuser's initial location. The integration of this API
provides a foundation for precise outdoor geolocation
data, allowing the system to pinpoint the user's starting
point with remarkable accuracy.
4.2. QR Code Recognition for Initialization
QR Code Scanning:
To initiate the navigation experience, users can scan a
QR code strategically placed at various points within thecampus environment. This QR code serves as a unique
identifier for specific locations. Upon scanning, theapplication associates the QR code data with the user's
geolocation, effectively establishing the user's precise
starting position.
Advantages of QR Code Initialization:
The utilization of QR codes offers several advantages
for the Campus Navigator system:
Efficiency: QR codes allow for rapid and error-free
initialization, minimizing user effort and potential
location inaccuracies.
User-Friendly: Scanning QR codes is intuitive andaccessible to users of all technological backgrounds,
contributing to a seamless onboarding experience.

Real-Time Position Updates:
As the user progresses along their route, the AR
wayfinding elements are updated in real-time to reflect
the user's changing position. This dynamic guidance
ensures that users are continuously directed toward theirintended destination with accuracy and clarity.

4.5. User Interaction and Interface
Intuitive User Interface:
The user interface of the Campus Navigator is designedwith a minimalist and intuitive approach, prioritizing
ease of use and accessibility. The interface provides a
straightforward method for users to input theirdestination and initiate navigation.
Destination Selection:©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
498

AR Based Application for

Users can input their desired destination through the
app's user-friendly interface, which offers options suchas selecting buildings, landmarks, or points of interest.
Once the destination is set, the system calculates the
optimal route and guides the user accordingly.

Table 1: Application Runtime metrics

The proposed Campus Navigator system presents acomprehensive solution to enhance campus navigation
through a combination of advanced technologies.
Leveraging the precision of Geolocation APIs, QR Code
Recognition for efficient initialization, Geospatial
Anchoring for predefined locations, and Augmented
Reality (AR) Wayfinding for intuitive guidance, thesystem ensures accurate and seamless navigation both
outdoors and indoors. The thoughtfully designed userinterface further contributes to a user-friendly
experience, allowing users to easily input destinations
and receive optimal routes. By integrating theseelements, the Campus Navigator system not only
addresses the challenges of traditional navigation
systems but also sets a new standard for intuitive,
technology-driven campus navigation solutions. Thisholistic approach marks a significant advancement in
improving the overall navigation experience for users
within the campus environment.

MetricSample Value

Benchmark/Goal

Frame Rate (FPS)

30 FPS

30 FPS or higher

Tracking Accuracy

95% accuracy

High accuracy

Latency

25 ms

Low latency

Initialization Time

3 seconds

Fast initialization

10% per hour

Minimal battery drain

Data Usage

5 MB/hour

Efficient data usage

Memory Usage

150 MBCPU Usage

25%

Low CPU usage

Network Latency

50 ms

Low network latency

Battery
Consumption

User Engagement
Metrics

5. Result and discussion
5.1. Real-World Scenario TestingHigh user
engagement

1% errors

User Feedback

4.5 out of 5

Conversion Rates

5% conversion

High conversion rates

Retention Rate

70% retention

High user retention

2 seconds

Quick content loading

Scalability

100 objects

Scalable performance

App Size

50 MB

Small app size

App Launch Time

1.5 seconds

Quick app launch

Session Length

15 minutes

Crash Reports

0.5% crashes

Times

Fig. 1, Fig. 2, Fig. 3 and Fig. 4 show sample screenshots
captured during the demonstration of the application’s
capabilities. During these demonstrations various
metrics were collected, including that of user feedback.
These metrics are shown in Table 1.

usage

Error Rates

AR Content Load

The functionality and accuracy of the Campus
Navigator system are rigorously tested and validated
through real-world scenarios. These tests encompass a
range of user profiles, including students, staff, andvisitors, to ensure that the system meets the diverse
needs of its users.

10 mins/session

Efficient memory

Low error rate
Positive user
feedback

Longer session
lengths
Minimal crashes

5.2. User Feedback Integration
User feedback is actively sought and incorporated
into system improvements. Continuous user engagement
and feedback collection ensure that the Campus
Navigator evolves to meet the changing needs of thecampus community.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
499

Renuka Devi Rajagopal, Akshay Sripriya, Shakthi B, Manoj Rathinam S, Heshalini Rajagopal

Fig. 1 Screen showing AR path for user to follow.

Fig. 4 Screen showing AR path for user to follow.
6. Conclusion
In conclusion, the "Campus Navigator" represents apioneering solution to the intricate navigation
challenges faced within complex campus environments.
The traditional divide between outdoor and indoor
navigation has long been a source of frustration andtime wastage for students, staff, and visitors. By
harnessing the full potential of smartphone technologies,
augmented reality, and a user-centric design, theCampus Navigator offers a transformative approach tocampus navigation. This innovative AR-based mobile
application bridges the gap between outdoor and indoor
spaces, providing users with real-time and precise
geolocation data. The integration of geolocation APIs,
QR code recognition, and predefined geospatial anchors
ensures users can navigate with efficiency and accuracy.
Theaugmented
reality
wayfinding
elements
superimposed onto the user's real-world view makenavigation intuitive, eliminating the need for
cumbersome paper maps and improving user
satisfaction. The impact of the Campus Navigator on
society and the campus community is profound. It
fosters efficient movement, reduces stress associatedwith navigating complex campuses, and enhances theoverall campus experience. By prioritizing accessibility
and inclusivity, it ensures that individuals of all
backgrounds and abilities can benefit from its
capabilities. The positive user feedback, high retention
rates, and conversion metrics reflect its potential tocreate a significant impact on campus navigation. Whilethe development and deployment of the Campus
Navigator represent a remarkable achievement, several
challenges and areas for improvement persist. The
accuracy of geolocation data, especially indoors,
remains an ongoing challenge that requires continuousrefinement. Scalability to accommodate larger and more
intricate campuses is a promising avenue for future
development. User feedback and evolving technology

Fig. 2 Screen showing destinations for user tochoose from.

Fig. 3 Screen showing AR path for user to follow.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024500

Powered by TCPDF (www.tcpdf.org)

AR Based Application forwill drive further enhancements, ensuring theapplication remains at the forefront of campus
navigation solutions. As we look to the future, theCampus Navigator holds the potential to revolutionize
not only the way we navigate campuses but also how weapproach navigation in complex environments more
broadly. Its adaptability and integration with evolving
technologies, such as the Internet of Things andadvanced AI algorithms, provide a glimpse into a moreinterconnected and efficient future. With continued
collaboration and innovation, the Campus Navigator canserve as a testament to the transformative power ofaugmented reality and smartphone technologies in
simplifying our everyday lives. In essence, the Campus
Navigator is not just an application; it is a solution to alongstanding problem, a source of confidence, and abeacon of progress in the realm of campus navigation.
Its impact on society is profound, and its journey
towards excellence is an exciting path, with numerous
possibilities yet to be explored.

Authors Introduction
Dr. Renuka Devi Rajagopal
She is an Associate Professor in the
School of Computer Science andEngineering, VIT Chennai, India. Her
research interests include Cyber-Physical
Systems, Block Chain Technology, Data
Mining and Machine learning in the fieldof Health Care.
Mr. Akshay Sripriya
Akshay Sripriya is a student curently
enrolled in Integrated M.Tech Software
Engineering in VIT Chennai, motivatedby a passion for technology.
Mr. Shakthi B
He is a student curently enrolled in
Integrated M.Tech Software Engineeringin VIT Chennai, was driven by a passion
for cutting-edge technology.

References
1.

2.

3.

4.

5.

6.

7.
8.

Lu, F., Zhou, H., Guo, L., Chen, J., & Pei, L. (2021). An
ARCore-Based Augmented Reality Campus Navigation
System. Applied Sciences, 11(16), 7515.
Kuwahara, Y., Tsai, H. Y., Ieiri, Y., & Hishiyama, R.
(2019). Evaluation of a campus navigation application
using an AR character guide. In Collaboration
Technologies and Social Computing: 25th International
Conference, CRIWG+ CollabTech 2019, Kyoto, Japan,
September 4–6, 2019, Proceedings 25 (pp. 242-250).
Springer International Publishing.
Lin, C. H., Chung, Y., Chou, B. Y., Chen, H. Y., & Tsai,C. Y. (2018, April). A novel campus navigation APP
with augmented reality and deep learning. In 2018 IEEE
International Conference on Applied System Invention(ICASI) (pp. 1075-1077). IEEE.
Yu, K. M., Chiu, J. C., Lee, M. G., & Chi, S. S. (2015,
August). A mobile application for an ecological campus
navigation system using augmented reality. In 2015 8thInternational Conference on Ubi-Media Computing(UMEDIA) (pp. 17-22). IEEE.
Lautenschläger, B. (2012). Design and implementation ofa campus navigation application with augmented reality
for smartphones. Bachelor Thesis–University of Calgary.
Pawade, D., Sakhapara, A., Mundhe, M., Kamath, A., &Dave, D. (2018). Augmented reality based campus guideapplication using feature points object detection. Int. J.
Inf. Technol. Comput. Sci. (IJITCS), 10(5), 76-85.
Hew, T. W. (2018). AR-UTAR Kampar campus
navigation (Doctoral dissertation, UTAR).
Chou, T. L., & ChanLin, L. J. (2012). Augmented reality
smartphone environment orientation application: a casestudy of the Fu-Jen University mobile campus touringsystem. Procedia-Social and Behavioral Sciences, 46,
410-416.

Mr. Manoj Rathinam S
He is a student curently enrolled in
Integrated M.Tech Software Engineeringin VIT Chennai, skillfully merged asophisticated software engineering skill set
with a profound passion for technology.

Dr. Heshalini Rajagopal
She received her PhD and Master's degreefrom the Department of ElectricalEngineering, University of Malaya,
Malaysia in 2021 and 2016, respectively.
Her research interest includes image
processing, artificial intelligence andmachine learning.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
501

Development of Robotic Assistant for Health Care Sector with A Special Focus to Aid theGeriatric Patients
Narayanan Ganesh

School of Computer Science and Engineering, Vellore Institute of Technology, Chennai–600127, India
Heshalini Rajagopal

Institute of Computer Science and Digital Innovation, UCSI University, 56000 Kuala Lumpur, Malaysia
Email: ganesh.narayanan@vit.ac.in

Abstract
Human manual efforts in their day-to-day tasks are reduced by robotic helpers. This paper analyzes the robot createdto assist the aged people. The created robot can be controlled by voice commands with its own inbuilt microphone topick up human speech orders. This robot can do a variety of movements, turns, grab operations, move an object fromone location to another. Personal assistant robot is built using Microcontrollers. The results reveal that the developed
robot will be one of the best companions for the geriatric patients and will be an alternate solution will make a newmark in the health care sector.
Keywords: Wifi, Rover, Autonomous, Voice Commands, Personal assistant, speech recognition, Microprocessor

a different accent, voice identification is a difficult
undertaking. Robot voice accomplishes this by utilizing
the Bit Voicer Server, which supports 17 languages from
26 nations and regions.
These robotic helpers can be employed in a variety ofindustries, such as manufacturing and defense, forshaping, manufacturing, and tooling. The robot helper
reacts to verbal commands. The contribution of the paper
has introduction, study of literature, components used forresearch method, results and discussions, conclusion andreferences.

1. Introduction
Humans have progressed over time in their ability todesign new technologies that reduce human work andsave human life. People are frequently harmed whilehandling hazardous substances in the chemical andexplosives manufacturing industries due to a lack ofproper care or precaution. Physically challenged andelderly people have difficulty handling objects andrequire assistance in doing so. As a result, if a robotic
assistant that can be controlled with vocal instructions isdeveloped, it will be extremely useful. Assistant robots
can be employed for a variety of tasks, such as handling
dangerous chemicals and products in the chemicalindustry or in the household. These robotic helpers canbe utilized for shaping, manufacturing, and tooling in avariety of industries, including industrial, defense, andaerospace. Humans' primary mode of communication is
speech signals. Voice signals are used in almost every
discussion to interact. A microphone can be used totransfer sounds and various speech signals into electricalform. A computer-based system that transforms vocal
signals is known as voice recognition. Using a remote
server, this voice recognition technology can be utilized
to control and generate speech acknowledgement. A
robot voice can recognize hundreds of spoken commands
and carry out the required activity because everyone has

2. Study of Literature
Investment in industrial robots climbed up globally andorders for robots increased enormously reaching thehighest level ever recorded. Global growth is expected to
be over 7% each year on average. There are currently
over 600,000 home robots in use, with millions morepredicted in the coming years. In general, there are threemodes of communication such as isolated word or phrase
mode, continuous speech mode and small vocabularysystems. The Isolated word (or phrase) mode deals withthe user who speaks single words (or phrases) from a
vocabulary list in this mode Connected word mode: theuser speaks in this mode. The Continuous speech mode

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

502

Narayanan Ganesh, Heshalini Rajagopal

demonstrates the user to use a broad (sometimes
limitless) vocabulary to talk fluently that would specify
the way of recognizing Robot's voice. The Small
vocabulary systems recognizes up to 100 words. The
medium vocabulary systems can recognize up to 1000
words. The Broad vocabulary systems can recognize over1000 words. It has been proposed that a speech
recognition module is not required to govern a robot. In
this system, an android application is used to recognize
and process human voice, which is then transformed into
text using text recognition software. This study explores
the issues raised by the significant increase in life
expectancy, which has resulted in a demographic shift
toward an older population in modern developed
societies. The article highlights the limitations of present
home technologies in fulfilling the changing demands ofthe elderly, who are expected to spend more time at homeas they age. By 2050, the number of senior people
worldwide is expected to triple. The problem isexacerbated because wider adoption of these automated,
smarter, and more recent technologies is hindered bypeople's lack of familiarity with them. The study
proposes the integration of assisted living technologies
into a Mobile Robotic Platform (MRP) as a solution tothese difficulties. It also presents the concept of ASPiDA,
an all-encompassing system intended to assist the elderlyin their homes. [1]The study highlights the benefits of using Series Elastic
Actuators (SEAs) in rehabilitation robotics, including
their high compliance, shock tolerance, andbackdrivability. In uncertain human-robot interactions, ittackles the problem of creating an Assist-as-Needed(AAN)
techniquefor
multijoint
SEA-driven
rehabilitation robots. The suggested approach uses aniterative learning algorithm for robot-level dynamics anda fast time-scale controller for SEA-level dynamics,
modifying assistance in response to the assessment of
human-robot interaction. Experiments with healthy
participants verify the strategy, which is used to a twodegree-of-freedom SEA-driven robot, encouraging
proper motions with minimal help and adapting to thesubject's purpose [2].
The research investigates a robot-mediated therapy
approach for post-stroke rehabilitation that utilizes
performance-based help control. Three modes areavailable to accommodate different stages ofrehabilitation: Restriction Interaction Region (RIR),
Assist-as-Needed (AAN), and Zero Interaction Force(ZIF). AAN offers varying aid depending on motor
ability, RIR maintains safety with a significant assisting
force, and ZIF permits free motion referencing. The
strategy's success is validated by experimental resultswith able-bodied participants, which indicate correctfunctionality and the ability to modify modes and support
based on subjects' motor performance. With increasing

tracking error, the method adjusts by offering more
support and promoting active effort by adjusting theadaptive stiffness coefficient. Crucially, the dimensionsof the strategy can be tailored to suit various topics andneeds [3].
With an emphasis on agent embodiment, the study
investigates the creation of intelligent agents to facilitategroup discussions among senior citizens. User responses
to two types of agents—voice assistants and humanoid
robots—were compared. Two phases of the study wereconducted with older persons and a human facilitator: apreliminary study and an experimental study.
Notwithstanding their artificiality, both agent forms were
useful for the socially awkward duty of facilitating
discourse. Talkative personalities, on the other hand,found the "bodied" robot version to be less satisfying.
The results emphasize how crucial it is to take user
attributes into account when designing agents. Additional
observations and design implications are also presented,
with a focus on agent voice in particular [4].
Social robots are autonomous systems that interact withpeople in natural settings. They are used in healthcare toprovide support in private homes, hospitals, and nursing
homes. This assessment emphasizes technologicalfeatures and focuses on the condition of social robots
today. The three main categories of robots that arecovered are telepresence, companion, and humanoid
robots. Commercial applications, scientific literature(Scopus Elsevier), patent analysis (Espacenet), andsupplementary sources (Google search) are all includedin the analysis. The article offers a succinct summary ofsocial robots in healthcare by classifying different
devices and arranging their specifications [5].
Social robots are essential to the healthcare industry
because they can engage with people on their own in realworld settings, including nursing homes, hospitals, andprivatehomes.
This
assessment
emphasizes
technological features while highlighting the current
status of social robots. There is discussion of three
categories: telepresence robots, companion robots, andhumanoid robots [6].
The use of robotics and soft actuators in rehabilitation is
growing, with a focus on safe human-machine interaction.
They have benefits including robustness to a variety ofsettings, easy construction, complicated motions, andsafe contact. This review examines the state of soft
actuators in rehabilitation today, encompassing soft
materials and a range of powering techniques [7].
In order to improve healthcare decision-making, thisstudy presents a cognitive system for assistive robots thatuses artificial intelligence (AI). The technology
dynamically modifies robot actions to match thedemands of individual patients, proving its viability in an
actual setting [8].

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
503

Development of Robotic Assistant

Robots can help those who look after the elderly
population. Prior studies concentrated on developing
capacities rather than incorporating robots into theprovision of professional care. Design possibilities were
found in a field study conducted in a senior living facility.
These opportunities included improving caregiverworkflows, accommodating resident abilities, and giving
feedback to all stakeholders [9].
Recent developments in haptic guiding pose problems torobot-assisted training. As an alternative, Model
Predictive Controllers (MPCs) were investigated in thisstudy with 40 participants. The ball MPC increased
performance but limited variability, whereas the endeffector MPC encouraged movement variability andimproved learning. MPCs offer significant advantages
for neurological patient training since they demonstratepromise in enhancing motor learning in tasks withcomplicated dynamics [10].
By encouraging independence, CHARMIE, a flexible
healthcare and household assistance robot, tackles issues
faced by aging populations. It carries out activities likefall detection and room tidying with capabilities like map
development, safe navigation, and human-robot
interaction. In addition, CHARMIE helps with publichealth situations like as COVID-19 by offering safe
healthcare support [11].
Persons with low eyesight can navigate more easily
thanks to the Augmented Cane's sensors and clear
feedback. When used in place of a typical white cane, it
improved walking speeds by 18% for the visually
impaired and 35% for the sighted in tests. Around 250
million people worldwide will benefit from the opensource, affordable design's sophisticated navigation
features, which increase mobility and quality of life [12].
This paper offers a unique Force Exertion Ability
Enhancement (FEAE) and admittance control based
Robotic Assistive System (RAS) for senior mobility in awheeled mobile manipulator. The RAS offers usercontrolled, compliant behavior together with limited
horizontal guidance and vertical assistance. An
expensive force/torque sensor is not necessary whenusing a nonlinear disturbance observer. By increasing
Cartesian force exertion within joint torque limitations,
the FEAE boosts system performance. An experimental
validation using a 4-wheel omnidirectional mobile
manipulator shows that the suggested method works well[13]. In order to build gadgets that support aging invarious settings, such as residences, assisted living
facilities, nursing homes, and family housing, this article
examines the limitations of present technology. It lists
typical problems including financial hardship andloneliness and highlights how technology is necessary to
overcome obstacles unique to a given locality andpromote successful aging in place [14]. Although theconcept of the "Elderly Care Giver" influenced carerobotics, R&D obstacles have limited the practical roleof robots in care. The development of care robots
frequently depends on speculative scenarios and theirfocus on simple activities. Activities that address social,emotional, practical, political, economic, and ecologicalaspects of care ecosystems should be taken into account
if they are to be successful. To understand careorganization
withlimitedresources,
candidconversations about the motivations driving care robot
projects are necessary [15].

3. Components for Research Method
Raspberry Pi B+, Lidar, Body, Motors, Motor Driver,
Display, Mic, Camera and GPS.
3.1. Raspberry Pi B+
The Raspberry Pi Foundation, in collaboration with
Broadcom, developed a series of miniature single-board
computers (SBCs) in the United Kingdom. It has a level1 (L1) cache of 16 KB and a level 2 (L2) cache of 128
KB. The GPU is the primary user of the level 2 cache.
Raspberry pi model B+ is used for this project which
has Bluetooth. Fig. 1 shows the Raspberry Pi used forthis work.

Fig 1: Raspberry Pi©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
504

Narayanan Ganesh, Heshalini Rajagopal

3.2 Compatibility and Ethernet Compatibility Over
300mbps with 40 Gpio Pins.
Fig. 2 shows the flowgraph of RAM, IO, GPU, USB Hub.

Fig 4: The Body
Fig. 2 Flowgraph of RAM, IO, GPU, USB Hub

3.5 Electric Motors
A DC motor is an electronic device that runs
upon direct current which converts electrical energy
into mechanical rotational force. DC motors may be
powered directly from rechargeable batteries, which
is how the earliest electric cars got their start. DC
motors are still used today in a variety of applications,from toys to disc drives to steel rolling mills andpaper presses. The DC motor used for this work isshown in Fig. 5.

3.3 The Lidar
The TF-Luna Light Sensing and Ranging (LiDAR)
module (shown in Fig. 3) was presented as a time-offlight sensor for quick distance detection. GPIO pins
14/15 were used to connect the module to the RPi
through the UART serial interface. A plotter tool wasdeveloped as a means of testing the module's operationand viewing its behavior.

Fig. 5: Electric Motors
3.6 Motor Drivers
Using a Raspberry Pi Model B+ and this motor
driver kit (as shown in Fig. 6) and Python library, apair of bidirectional brushed DC motors can be
operated. The DRV8835 dual H-bridge motor driver
IC from Texas Instruments is used on the expansion
board, allowing it to function from a wide range ofvoltages. It consumes 1.5 V to 11 Vs of power whichis the best suitable range to work with microcontrollers. The board can deliver 1.2 A per channel
continuously. The board comes fully created with
SMD components, including a reverse battery
protection FET and the DRV8835 driver.

Fig. 3: LIDAR

3.4 The Body
The most important aspect of a robot is its flexible
body, which shields and surrounds the overall CPU andfragile boards. A body also aids in the structure of therobot's shape, and provides support to the whole system
while doing any task. The body of the robot is shown in
Fig. 4.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
505

Development of Robotic Assistant

3.9 Camera
This camera module as shown in Fig. 9 is smalland serving best performance and good for streamingin live.

Fig. 6: Motor Drivers
3.7 The Display
A liquid crystal display (LCD) (as shown in Fig.
7) is used as the user interface between human androbot. This operates on the principle of light
modulating properties of liquid crystals. Computer
monitors, televisions, instrument panels, andaeroplane cockpit displays are just a few of theapplications.

Fig. 9: Camera

3.9.1 Features of Raspberry Pi 5MP Camera Module•

Fully Compatible for Model B+ Raspberry Pi

•
•
•
•
•
•

5MP Omni vision 5647 Camera Module
Still Picture Resolution: 2592 x 1944
Video: Supports 1080p @ 25fps, 720x360p 60/90
15-pin Serial Interface with MIPI Camera
Size: 20 x 25 x 9mm
Weight 3g

3.10 GPS Module
The NEO-6M GPS Module (shown in Fig. 10) is acomplete GPS module that is based on the NEO-6M.
This is an upgraded GPS module that can be used with
Raspberry Pi.

Fig.7 The Display

3.8 Mic
A little microphone (as shown in Fig. 8) which canbe operated within the range of 1.5 to 2.5 volts is used
to detect the random sounds and human voice sounds
and convert them into AC or DC signals which will bepassed on the microcontroller.

Fig. 10: GPS Module
3.10.1 Features NEO-6M GPS Module:
Fig. 8 Mic● 5Hz position update rate
● EEPROM to save configuration settings● Rechargeable battery for Backup©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
506

Narayanan Ganesh, Heshalini Rajagopal● Supply voltage: 3.3 V● Configurable from 4800 Baud toanalyse the sounds to identify the most likely wordthat fits the audio. All of this is accomplished using
Natural Language Processing and Neural Networks.
By recognizing temporal patterns in speech, hidden
Markov models can improve accuracy. The working
model is shown in Fig. 11.

115200 Baud rates. (default 9600)● Support SBAS (WAAS, EGNOS, MSAS,GAGAN)● Separated 18X18mm GPS antenna
4 Results and Discussions
The robot's movement is controlled by speechcommands. The robot recognizes the voice commands
with mic and the internal op amp cuts off the noises
and passes on the clear voice sound. The vocal stream
is subsequently translated to text in real time via aninternet cloud server. The hardware platform is madeup of a moveable robotic arm incorporated within the
robot's body. Two robotic hands make up the roboticarm. The arm is utilized to place the hands in theirproper positions, and the hands are used to pick, holdand drop objects. It works similarly to human hands,
with the robotic arm acting similarly to our arm andthe robotic hands acting similarly to our hands. Two
DC motors control the robot's body movements.

Fig. 11 Working Model
4.3 Procedure of Working
The working procedures of the robot are as follows:

4.1 Modules used
4.1.1 Speech Recognition
The ability of a machine to listen to and recognise spoken
speech is referred to as voice recognition. The uttered
words can then be converted to text, a query can be made,and a response can be given using Python's speech
recognition. With the help of computer programs thattake in information from the microphone, process it, andconvert it into a proper form, you may accomplish voicerecognition in Python.
Speech recognition may appear futuristic, but it is alreadyin use. You can shout out your query on automated phone
calls, and virtual assistants like Siri and Alexa use speech
recognition as well.

•

Build the robot's body first•

Connect the motors, chipsets, and limbs.

•

Use WIFI to establish a reliableinternet connection and local
Network connection. Speak any ofthe pre-programmed commands todirect the robot's actions.

4.3.1 Voice Controls
Forward - The body moves forward.
Backward - The body moves backward.
Left - The robot moves to the left.
Right - The robot moves to the right.
Up – The robotic arm moves upward.
Down - The robotic arm glides downward.
Open - Robotic hands that are open
Close - Robotic hands are quite close together.
Bring Water - Brings a Water to you.
When a user speaks a command, the mic listens to
it first, and the voice input is processed by the speech
recognition module, The Google API is used toconvert the speech to text format. The matching drivercode of the code is executed if the text matches thecommands list. When a user commands the robot tobring a given object, the object's coordinates should
already be initialized, and the robot uses the GPS
module interface to reach the coordinate and bring theobject. The robot's arm picks the thing, and the Lidar

4.2 Process of speech recognition
Speech recognition starts by converting thesound energy into electrical signals using the
microphone and we also use some noise cancellation
techniques to acquire clear audio. After that, theelectrical energy is transferred from analogue todigital, and then to text. It takes the audio data andbreaks it down into sounds, then uses algorithms to©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
507

Development of Robotic Assistantsensor is used to detect and avoid obstacles along theway. The workflow is shown in Fig. 12.

industry to a relaxing environment within a homewherein the elderly people can live happily and at ease.
References1.

Keroglou, C., Kansizoglou, I., Michailidis, P., Oikonomou,
K. M., Papapetros, I. T., Dragkola, P., ... & Sirakoulis, G.
C. (2023). A Survey on Technical Challenges of Assistive
Robotics for Elder People in Domestic Environments: The
ASPiDA Concept. IEEE Transactions on Medical
Robotics and Bionics.
2. Han, S., Wang, H., & Yu, H. (2023). Human–Robot
Interaction Evaluation-Based AAN Control for Upper
Limb Rehabilitation Robots Driven by Series Elastic
Actuators. IEEE Transactions on Robotics.
3. Zhang, L., Guo, S., & Xi, F. (2023). Performance-based
assistance control for robot-mediated upper-limbs
rehabilitation. Mechatronics, 89, 102919.
4. Seaborn, K., Sekiguchi, T., Tokunaga, S., Miyake, N. P.,& Otake-Matsuura, M. (2023). Voice over body? Olderadults’ reactions to robot and voice assistant facilitators ofgroup conversation. International Journal of Social
Robotics, 15(2), 143-163.
5. Ragno, L., Borboni, A., Vannetti, F., Amici, C., & Cusano,N. (2023). Application of Social Robots in Healthcare:
Review on Characteristics, Requirements, TechnicalSensors, 23(15), 6820.
6. Suzuki, R., Karim, A., Xia, T., Hedayati, H., & Marquardt,
N. (2022, April). Augmented reality and robotics: A
survey and taxonomy for ar-enhanced human-robot
interaction and robotic interfaces. In Proceedings of the2022 CHI Conference on Human Factors in Computing
Systems (pp. 1-33).
7. Pan, M., Yuan, C., Liang, X., Dong, T., Liu, T., Zhang,J., ... & Bowen, C. (2022). Soft actuators and robotic
devices for rehabilitation and assistance. AdvancedIntelligent Systems, 4(4), 2100140.
8. Sorrentino, A., Fiorini, L., Mancioppi, G., Cavallo, F.,
Umbrico, A., Cesta, A., & Orlandini, A. (2022).
Personalizing care through robotic assistance and clinicalsupervision. Frontiers in Robotics and AI, 9, 883814.
9. Stegner, L., & Mutlu, B. (2022, June). Designing for
Caregiving: Integrating Robotic Assistance in Senior
Living Communities. In Designing Interactive Systems
Conference (pp. 1934-1947).
10. Özen, Ö., Buetler, K. A., & Marchal-Crespo, L. (2021).
Promoting motor variability during robotic assistanceenhances motor learning of dynamic tasks. Frontiers inneuroscience, 14, 600059.
11. Ribeiro, T., Gonçalves, F., Garcia, I. S., Lopes, G., &Ribeiro, A. F. (2021). CHARMIE: A collaborative
healthcare and home service and assistant robot for elderlycare. Applied Sciences, 11(16), 7248.

Fig. 12 Workflow
5 Conclusions
The personal assistant robot is built on a
microcontroller platform and has the ability to track
its present location. Improvements are also considered
in terms of prospective uses in the home, hospitals that
could aid the geriatric patients. Some of the areas thatcan be further investigated are the robot's mouth andmicrophone, its performance, and the effect of noiseon voice to text conversion. The robot's functioning isunaffected by the speaker's accent since voicecommands are handled by a cloud server that works
regardless of the speaker's accent. Using renewableenergy sources to power the robot would not only
reduce the cost of the robot, but it would also be
environmentally good. Another form of energy source
is solar energy. The robotic assistant developed has awide range of potential uses, from the chemical©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
508

Powered by TCPDF (www.tcpdf.org)

Narayanan Ganesh, Heshalini Rajagopal

12. Slade, P., Tambe, A., & Kochenderfer, M. J. (2021).
Multimodal sensing and intuitive steering assistance
improve navigation and mobility for people with impaired
vision. Science robotics, 6(59), eabg6594.
13. Xing, H., Torabi, A., Ding, L., Gao, H., Deng, Z.,
Mushahwar, V. K., & Tavakoli, M. (2021). An
admittance-controlled wheeled mobile manipulator formobility assistance: Human–robot interaction estimation
and redundancy resolution for enhanced force exertion
ability. Mechatronics, 74, 102497.
14. J. Miller, T. McDaniel and M. J. Bernstein, "Aging in
Smart Environments for Independence," 2020 IEEE
International Symposium on Technology and Society(ISTAS), Tempe, AZ, USA, 2020, pp. 115-123, doi:
10.1109/ISTAS50296.2020.9462211.
15. Van Aerschot, L., & Parviainen, J. (2020). Robots
responding to care needs? A multitasking care robot
pursued for 25 years, available products offer simple
entertainment and instrumental assistance. Ethics andInformation Technology, 22(3), 247-256.

Authors Introduction
Dr. Narayanan Ganesh
Dr. Narayanan Ganesh is a seniorassociate professor at the School ofComputer Science and Engineering,
Vellore Institute of Technology,
Chennai Campus. With a career
spanning nearly two decades inteaching, training and research, he
has established himself as an
authority in this field. His researchinterests are diverse and forward-thinking, encompassing
software engineering, agile software development,
prediction and optimization techniques, Internet of Things,
Robotics, deep learning, image processing and data
analytics.
Dr. Heshalini Rajagopal
She received her PhD and Master's
degree from the Department ofElectrical Engineering, University of
Malaya, Malaysia in 2021 and 2016,
respectively. She received the B.E(Electrical) in 2013 Currently, she isan Assistant Professor in UCSI
University, Kuala Lumpur, Malaysia.
Her research interest includes image
processing, artificial intelligence and machine learning.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
509

Development of a Desktop Application Restaurant Management System
Gabriela Maria AncillaInstitute of Computer Science and Digital Innovation, UCSI University, 56000 Kuala Lumpur, Malaysia
Heshalini Rajagopal

Department of Electrical and Electronics Engineering, MILA University, 71800 Nilai, Negeri Sembilan, Malaysia
Ismail Ahmed Al-Qasem Al-Hadi

Institute of Computer Science and Digital Innovation, UCSI University, 56000 Kuala Lumpur, Malaysia
Renuka Devi Rajagopal

School of Computer Science and Engineering, Vellore Institute of Technology, Chennai, India
Norrima Mokthar

Department of Electrical Engineering, Faculty of Engineering, University of Malaya, Malaysia
E-mail: heshalini@ucsiuniversity.edu.my, heshalini@gmail.com

Abstract
The number of restaurants has grown rapidly around the world, the awareness of managing it efficiently has
increased. This necessity yields the idea to invent a restaurant management system (RMS). This RMS will bringseveral benefits such as greater management of a restaurant, reduced resources cost, raise profit, and time-saving by
allowing the admin to manage the restaurant easily through the functionality provided in the system. The provided
features include adding, updating, and deleting information, stock calculation and auto re-order stock items,
ingredients management, and finance calculation.
Keywords: Restaurant Management System (RMS), Business Solutions, Administration Systemdaily workload and fulfill various task in one integrated
system using a software system and databases that keep
all the operational and personnel details. RMS offers its
benefits by providing functions to organize the details of
employees and their payroll, keep track of ingredients
based on their category, manage suppliers, keep sales
history, retrieve information easily, and calculate profits[1]. Using RMS, restaurants can get the opportunity to
speed up operations through the automated system,
increase personnel efficiency due to decreased timerequired for ingredients checking and manual ordertaking, and lessen labor costs by lowering the number of
employees needed. Thus, this project aims to develop a
restaurant management system that eases daily
operations, enhances performance, and solves the current
inefficiency of restaurants across Malaysia. A survey
will be distributed among Malaysian restaurant
managers and owners to study their opinion and

1. Introduction
Throughout the years, businesses in various sectors had
attempted to reach efficiency and gain the most profit
through their efforts. Methods and strategies have beendeveloped to overcome the problem of inefficiency.
However, most of the time, the advancement oftechnology is the solution to those problems in activity
and performance done by using the traditional system.
Technology has shifted the way various task is executedand decreased the need for people that used to performthe task manually. This situation is likewise applicable tomany restaurant owners who have tried to increase sales
and satisfy their customers with excellent services.
A restaurant management system was developed to help
owners manage personal operations such as customers,
employees, ingredients, suppliers, and sales. Restaurant
management system (RMS) simplifies the personnel’s©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
510

Gabriela Maria Ancilla, Heshalini Rajagopal, Ismail Ahmed Al-Qasem Al-Hadi, Renuka Devi Rajagopal, Norrima Mokthar

acceptance of RMS, the current and previous
implementation, and their perceived benefits of using
RMS. Besides, the survey will also help to understandwhether they have sufficient knowledge and resources torun and implement RMS.

Table 1: Comparison Table of Existing and Proposed
System

2. Existing System
In the existing system, the operational and daily tasks ofthe staff all need to be done manually. Activities such as
keeping details of the employees, suppliers, andinventories are taken by paper and pen which leads topaper wasting and overloaded document. Paper
documents can easily get lost or damaged and may be
hard to be found when needed. Besides, calculating andupdating the ingredients stock manually can be very
time-consuming and cause miscalculations. In certain
circumstances, a restaurant can be running out of stockand miss the time to re-order the ingredients which canaffect the restaurant's profits. Moreover, manuallycalculating the ingredients stock also require more
manpower and increase labor cost. Hence, this proposed
desktop-based restaurant management system isdeveloped to overcome those issues that restrain
restaurants to achieve efficiency. The system will
combine several functional features in an integrated
centralized system. The staff and supplier information as
well as the detail of all ingredients will be stored insidethe database which will ease information retrieval
whenever needed. The stock of ingredients will be
automatically ordered from the assigned supplier toavoid situations when the restaurant is running out ofstock. In addition, the finance calculation of therestaurant will also be performed inside the RMS which
involves calculating the employees' payroll, daily sales,
overall expenses, and profit. Eventually, the proposedsystem aims to replace all the tasks that are regularly
done manually with an automated system that canaccelerate and ease the restaurant's daily operation andadministration by executing several processes in one
system. Table 1 shows the comparison between theexisting and proposed system.

3. Prior Work
Various projects and studies related to restaurant
management systems have been conducted, aiming to
improve efficiency, customer satisfaction, and overall
performance. Rainer Alt (2021) emphasizes theimportance of supply chain management in restaurants tohandle raw materials efficiently and maintain smooth
operations [2]. Srikar Macha (2022) develops a web
application using modern technologies like MVC andbootstrap to optimize restaurant management [3]. W. B.
A. C. Piyatissa (2020) presents an online system fororders and reservations, offering convenience forcustomers and enhancing customer relationshipmanagement [4].
Emel Memis Kocaman (2021) highlights the widespread
adoption of RMS in restaurants, positively impacting
their operations and service standards, although it comeswith significant technical costs [5]. Prudveer Karne(2022) introduces a web-based application to facilitate
better communication between customers and staff,
improving order management and overall efficiency [6].
Ivan Wanyama (2019) implements a web-based
restaurant management system to enhance data
organization and decision-making [7].
These projects and studies collectively contribute to theadvancement of restaurant management systems,
enabling better resource management, improved
customer service, and enhanced overall performance inthe foodservice industry. As technology continues toplay a prominent role in daily life, the ongoingdevelopment of innovative restaurant management
systems promises to revolutionize the way restaurants
operate, making them more efficient, customer-oriented,
and adaptable to changing market demands.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
511

Development of a Desktop

4. Method
The methodology selected to develop this project is theSoftware Development Life Cycle (SDLC) which is amethod used to develop software by going throughseveral phases. The five phases of SDLC include
planning,
requirementofanalysis,design,implementation, and testing.
The system consists of 5 tabs which are the employees,
ingredients, suppliers, sales, and finance tabs that will befully controlled by admin. First, in the employees tab
admin can add/edit/delete employee details that consistof their personal information such as name, age, gender,
etc. Admin can filter and search the employee's table byID, name, and role. Then, the admin can manage theemployee payroll as well as do adjustments for salarydeductions and overtime. In the ingredients tab, theadmin can add/edit/delete the details of each ingredientsuch as its name, price, quantity, etc. The ingredients
will be grouped based on their category thus admin also
can add/edit/delete the ingredients category. There is an
ingredients table that the admin can search and filter
based on its barcode, name, and supplier name. There
will be a low-stock item button provided for the admin
to check the low-stock ingredients for them to re-orderand purchase the ingredient by filling in the re-stock
form. Thirdly, the admin can add/edit/delete supplierdetails in the suppliers tab which details then will be
used to identify which ingredients were produced bywhich supplier for re-stock purposes. In the sales tab, theadmin can add/remove/clear the food item from theorder table before adding it to the ongoing orders table.
There is a show chart button provided for the admin toview the bar chart of the overall monthly sales of therestaurant. Then, the admin can view the list of ongoingorders, do customer purchases, and add/edit/delete thefood menu within this sales tab. Lastly, on the financetab, the admin can view the list of overall invoices, GRN,
and employees' payroll expenses. Admin can likewise
add/edit/delete other expenses such as electricity, water,internet, etc. In the income statement tab, all the total
income, expenses, and final profit/loss as well as thetotal number of employees and suppliers can be viewed.
Eventually, the admin can log out from the system byclicking on the logout button provided. The flowchart ofthe proposed system is shown in Fig. 1.

Fig. 1. Restaurant Management System Flowchart
5. Result and discussion
The restaurant management system is implemented
using Java with Java Swing library inside Netbeans IDE
and MySQL for the database. The system consists of 5
main tabs which are the employees, suppliers,
ingredients, sales, and finance tabs. Fig. 2 shows the add
employee subtab allows the admin to save, update, anddelete employee personal information from database.
The admin must fill in all the text field there which
includes name, age, phone number, email, and address aswell as choose the radio button for gender. Fig. 3 is the
Search employee subtab allows admin to view the tableof information list of all the registered employees.
Admin can search and filter the employee’s information
based on the employee ID, name, role, or all.

Fig. 2. Add Employee Subtab©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
512

Gabriela Maria Ancilla, Heshalini Rajagopal, Ismail Ahmed Al-Qasem Al-Hadi, Renuka Devi Rajagopal, Norrima Mokthar

Fig. 5. Add Ingredient Subtab
Fig. 3. Search Employee Subtab
The add category subtab indicated in Fig. 6 is provided
for admin to save, update, and delete the ingredients
category. Admin can do that by simply fill in the
category name and the supplier ID will be automaticallyappear when admin select the supplier’s name provided
in the drop-down menu. Admin can likewise search andfilter the category table based on the category name. The
low stock item window in Fig. 7 is for admin to checkthe low stock item and re-order the stock by clicking on
the re-order stock button. Fig. 8 shows the re-orderstock form that will appear when admin click on the reorder stock button. Admin can simply key in theingredient ID on the search field. Then, the admin can
key in the desired re-stock quantity and the total pricewill appear.

Fig. 4 indicates the employee’s payroll subtab where theadmin can save, update, and delete the employee payroll
by searching on the employee ID and clicking on the
buttons provided. Besides, admin can alter theemployees’ payroll by clicking on the deduction or
overtime button depends on the employee number ofdays absence or total overtime work hours.

Fig. 4. Payroll Subtab
The add ingredient subtab in Fig. 5 allows the admin tosave, update, and delete the ingredients detail fromdatabase. The admin must fill in all the text field which
includes name, bar code, price, and quantity. The
supplier ID and name will automatically appear after theadmin select the ingredient’s category to show where thesupplier of the ingredient.

Fig.6. Add Category Subtab©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
513

Development of a Desktop

food name and its unit price will appear. The total pricewill be calculated once the admin key in the quantity of
the food. The add, remove, and clear button provided
there is used to manage the table order. After adding theorder, the total amount and quantity of the orders will becalculated and shown under the order table there. The
show chart button in Fig. 11 will show the graph of theweekly sales of the restaurant.

Fig. 7. Low Stock Ingredients Window

Fig. 10. Add Order Subtab

Fig. 8. Re-stock Ingredient Form
Fig. 9 shows the supplier tab which is generally used to
manage the supplier’s data such as their personal
information. Admin can save, update, and delete
supplier’s personal information from database. The
admin must fill in all the text field provided which
includes name, phone number, email, and address.

Fig. 11. Monthly Sales Chart
Fig. 12 exhibits the ongoing orders tab is used to view
the ongoing orders and its details. The table number will
be used to identify which orders belong to whichcustomer. When the customer wants to pay the bill,admin can click on the pay bill button. The pop-up
window as exhibited in Fig. 13 shows the example whenadmin want to complete the order of customer from table
number 3. Admin just need to key in the table number inthe search field and the food items from that table will belisted out. Then, the total item and amount of the order
will be calculated. After customer finish paying the bill,

Fig.9.Suppliers Tab
The add order subtab in Fig. 10 is primarily used to add
the customer order. When admin key in the menu ID, the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
514

Gabriela Maria Ancilla, Heshalini Rajagopal, Ismail Ahmed Al-Qasem Al-Hadi, Renuka Devi Rajagopal, Norrima Mokthar

admin can click on the add sales button to add the sales
data to database.

Fig.14. Menu List Subtab
The invoice subtab in Fig. 15 is used to list out all thesales of the restaurant. It includes the sale ID, total item,
total amount, and date. Admin can delete the sales when
there is a mistake by key in the sale ID or clicking on thelist and press on the delete button. The total of sales item
and amount will be automatically calculated. The
income statement subtab in Fig. 16 is where admin can
see the total income, expenses, employees, and suppliersof the restaurant. The profit or loss will be automaticallycalculated based on the sales and purchase of therestaurant. Lastly, admin can logout from the system byclicking on the logout button provided at the bottom leftof the system.

Fig. 12. Ongoing Orders Subtab

Fig. 15. Invoice Subtab
Fig.13.Re-stock Ingredient Form

Fig. 14 shows the menu list sub tab where admin cansave, update, and delete the restaurant menu item.
Admin can just simply type the food name and its pricein the text field provided.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
515

Development of a Desktop

increase management efficiency, minimize labor andresource costs, and enhance the performance ofrestaurant management. The system development isbased on the SDLC methodology in a determined timeand scope. User acceptance test is performed and hasreceived positive feedback from users. Overall, thesystem is targeted to bring benefits to most restaurants
and create efficiency by developing an integrated systemthat includes several functional features. This system canbe enhanced and improved for future development byadding more functional features.
References
Fig.16.Income Statement Subtab

1.

Based on the survey taken from 30 respondents ofrestaurant managers or owners, RMS has received
positive feedback where all of them agree with theefficiency of using RMS and would like to use thesystem for their restaurant. Thus, this project can bring
the solution to restaurant owners who wants to take thebenefits of using RMS. Fig. 17 shows the positive
feedback from users.

2.
3.
4.
5.

6.
7.

Alt, R. (2021). Digital transformation in the restaurant
industry: Current developments and implications. Journalof smart tourism, 1(1), 69-74..
K. Kuligowski, "Small Business Guide to a RestaurantManagement System," 1 July 2022.
Macha, S. (2022). Management System for a Restaurant..
Piyatissa, W. B. A. C. (2021). Web Based RestaurantManagement System (Doctoral dissertation).
Kocaman, E. M. (2021). Operational effects of using
restaurant management system: An assessment according
to business features. International Journal of Gastronomy
and Food Science, 25, 100408.
Karne, P. (2022). Management System for a Restaurant.
Wanyama, I., Arinaitwe, A., & Adraako, F. (2019). Webbased restaurant management system (Doctoral
dissertation, Makerere University).

Authors Introduction
Gabriela Maria AncillaShe has completed her B. Sc. (Hons)Business Information Systems, UCSI
University, Malaysia.

Fig. 17. User Acceptance Test Question on Willingness
to Use the RMS
Dr. Heshalini Rajagopal
She received her PhD and Master's degreefrom the Department of ElectricalEngineering, University of Malaya,
Malaysia in 2021 and 2016, respectively.
Her research interest includes image
processing, artificial intelligence andmachine learning.

6. Conclusion
The proposed restaurant management system isdeveloped to address the current issues in the existing
system and reduce the manual process for restaurant
daily operations. The traditional system that nowadays is
still implemented in many restaurants is time-wasting,demands a high cost, and often cause human error which
as a result cannot promote restaurant to perform in thebest efficient way. Thus, the proposed restaurant
management system is time-saving and error-freecompared to the manual system. It aims to fulfill all theneeds of the restaurant owners by developing a desktop
application restaurant management system which able toease the operational management of a restaurant,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
516

Powered by TCPDF (www.tcpdf.org)

Gabriela Maria Ancilla, Heshalini Rajagopal, Ismail Ahmed Al-Qasem Al-Hadi, Renuka Devi Rajagopal, Norrima Mokthar

Dr. Ismail Ahmed Al-Qasem Al-Hadi
He received the Ph.D. degree in
intelligent computing from Universiti
Putra Malaysia (UPM), in 2017. His
research
interests
includerecommendation
systems,machinelearning, and deep learning approaches.

Dr. Renuka Devi Rajagopal
Dr. R. Renuka Devi is an Associate
Professor in the School of Computer
Science and Engineering, VIT Chennai,
India. Her research interests include
Cyber-Physical Systems, Block Chain
Technology, Data Mining and Machinelearning in the field of Health Care.
Dr. Norrima Mokhtar
Norrima Mokhtar was appointed as alecturer to serve the Department ofElectrical Engineering, University of
Malaya immediately after graduatingwith her Master of Engineering. As
part of her career development, she
received SLAB/SLAI scholarship toattain her Ph.D. in 2012. She is now serving Department
of Electrical Engineering, University of Malaya.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
517

Face Recognition based on Attendance System
Koh Pei Cong

Institute of Computer Science and Digital Innovation, UCSI University, 56000 Kuala Lumpur, Malaysia
Heshalini Rajagopal

Department of Electrical and Electronics Engineering, MILA University, 71800 Nilai, Negeri Sembilan, Malaysia
Ghassan Saleh Hussein Al-Dharhani

Institute of Computer Science and Digital Innovation, UCSI University, 56000 Kuala Lumpur, Malaysia
Norrima Mokthar

Department of Electrical Engineering, Faculty of Engineering, University of Malaya, Malaysia
Dhanush Rachaveti

School of Electronics Engineering, Vellore Institute of Technology, Chennai, India
E-mail: heshalini@ucsiuniversity.edu.my, heshalini@gmail.com
www.ucsiuniversity.edu.my

Abstract
The use of face recognition technology for attendance tracking has grown popular in recent years. Hence, the main
goal of this project is to produce an accurate, fast, and robust face recognition based on an attendance system. The
system detects the user’s unique features, understand the identity of the user through face recognition technology,
and thus records the attendance from the face recognition dataset to the user that matches the user, in an attempt tohelp the user automatically check in with real-time attendance date. The system includes features such as facedetection, face recognition, distance estimation, and attendance recording.
Keywords: Face Recognition, Attendance, Convolutional Neural Network

mathematical objects and compared using linear
algebraic operations. The first linear algebra-based facerecognition algorithm was introduced by Turk andPentland in 1991 [4]. They proposed a method called"Eigenface" which used eigenvectors and eigenvalues torepresent the face space. This method involved
transforming the images into a lower-dimensional space
where faces could be represented as points in space.
This approach was later improved upon with theintroduction of "Fisherface" in 1997 by Belhumeur,
Hespanha, and Kriegman [5]. This method used linear
discriminant analysis to transform the face space into aspace where faces could be separated more easily. This
made it possible to use face recognition in a variety of
applications, including security and surveillance, bordercontrol, and airport security [5].

1. Introduction
A face recognition based on attendance system uses
machine learning techniques to identify individuals andtrack their attendance automatically [1]. It captures animage of the person's face, processes it to extract uniquefeatures, and compares it with the face recognition
datasets to determine the known faces [2]. If the known
face is detected, the system updates his or her
attendance record. This technology is commonly used inworkplace environments, educational institutions, andother organizations to streamline the attendance-taking
process and reduce manual errors [3].
In the 1980s and 1990s, researchers started to use linear
algebra for face recognition. This approach was basedon the idea that faces could be represented as©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
518

Koh Pei Cong, Heshalini Rajagopal, Ghassan Saleh Hussein Al-Dharhani, Norrima Mokthar, Dhanush Rachaveti

In the 2000s and 2010s, the development of mobile
devices and the widespread availability of the internet
paved the way for cloud-based face recognition systems,
which allowed for easy access and use from anywherein the world. This also paved the way for thedevelopment of mobile face recognition applications,which could be used on smartphones and other mobile
devices. In education, face recognition systems wereused for attendance tracking and to enhance student
safety [4], [6].
However, many conditions can affect the facerecognition attendance system, such as the camera
angles, the lighting conditions of the environment, thequality of the image, and various other factors. In
addition, the images will get low contrast and low
brightness if they are captured under low light
conditions [1]. There are several algorithms used byexperts to improve face recognition attendance systems,
such as the Eigenface algorithm, Principle Component
Analysis(PCA),
Fisherface
algorithm,
andConvolutional Neural Networks (CNNs) [5], [7], [8].
The choice of algorithm will depend on the specificrequirements of the application. However, CNN is
widely used in face recognition attendance systems dueto their high accuracy and ability to learn complex
features from large amounts of data.
Therefore, this project uses a built-in camera to capture
video from the camera, presents automatic attendance
tracking, develops a Graphical User Interface(GUI),
prevents system warnings and wrong execution fromunknown faces, and recognizes faces with high accuracy
and fast speed. The expected outcomes aim to ensurethat the system can recognize faces under the camera
angles, the lighting conditions of the environment, andthe quality of the image.

and PCA can lead to improved performance in facerecognition compared to using PCA alone [10].
Cahyono et. al (2020) designed a face recognition
system for automating the process of tracking employee
attendance. The linear SVM was used to classify thefaces of employees in real-time as they entered or left
the workplace. K-fold cross-validation was used toevaluate the performance and accuracy of a model. As aresult, the combination of FaceNet and SVM achieved
100% accuracy [2].
Qu et. al (2018) compared algorithms, such as PCA,
LBP, and CNN algorithms, for handling recognition
speed and accuracy. PCA was suitable for fast
recognition but with reduced accuracy, while CNN was
suitable for high-accuracy recognition but at the cost ofcomputational resources. LBP was a good compromise
between the two, offering a balance between recognition
speed and accuracy. It was possible to implement a
CNN on FPGA due to its high performance and low
power consumption [7].
Early work by Ilyas et. al (2019) was concerned withthe use of the Viola-Jones algorithm to detect faces inimages. Histogram Equalization was used in image
processing to enhance the contrast of an image. The
result showed that the recognition method using the
CMU PIE face database has a higher recognition rate
than the Extended Yale B face database [11].
KB and J (2020) designed a real-time face recognition
system using CNNs. Viola-Jones algorithm was used tocapture real-time input images from a camera. Then, the
AT&T database was used to detect faces in each image
and crop the image to include only the face. It waspossible to simplify the feature extraction process byhaving all images of equal size [12].
Rahouma and Mahfouz (2021) implemented a facerecognition system that leveraged the capabilities of the
API mobile vision for face detection. API Google’s
mobile vision was used to detect faces in the preprocessed images while Pearson correlation was used tocompare two face representations and determine theirsimilarity. As a result, the Pearson correlation spent lesstime but also produced a higher recognition rate thanother recognition methods [13].
Sarwar et. al (2021) developed an LBPH-based facerecognition system for visually impaired people. Haar
Cascade Classifier worked as a pre-processing step todetect faces in an image or video frame. The use of the
Euclidean distance helped to determine the similarity
between two faces and decide whether they belong to
the same person or not. The result showed that LBPH
gained higher average accuracy than the Fisherface
algorithm [14].
Wang et. al (2020) compared algorithms, such as SGD,
RMSprop, and Adam for evaluating the accuracy of theface recognition system. ReLU allowed the network to

2. Prior Work
This section reviews the methods which were used inthose studies to implement face recognition technology
and the corresponding to their results.
Shuhui and Xiaochen (2022) compared different SOTA
deep face recognition models to extract features from aface image even in the presence of masks. By using data
augmentation, the masked face recognition systembecame more robust to variations in the face images.
The result proved that an appropriate margin aided themodel’s establishment of flexible embedding [9].
Winarno et. al (2019) proposed a combination of CNNs
and PCA to identify and verify the attendance ofindividuals. CNN was used to extract features from aface image while PCA was used to reduce thedimensionality of the data and reduce computationalcosts. The result showed that the combination of CNNs

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
519

Face Recognition based onmodel complex features in the input images, such as theshapes of eyes, noses, and mouths. The result provedthat CNNs have significant practical application
potential when used for face recognition [15].
Early work by Wandzik et. al (2018) was concernedwith the use of an SVM classifier to distinguish between
real faces and morphed faces. SVMs were able tohandle high-dimensional data and can work well withnoisy data. The Euclidean distance was used to compare
the features of two faces and determined if they weresignificantly different. The result showed that theperformance of CNN-based approaches dramaticallyincreased [16].

3. Method
The face recognition based on attendance system
goes through five stages:(1) Set up the hardware: This involves installingthe cameras and other necessary hardware
components to capture the images and process
them in real-time.
(2) Collect and prepare the training data: A large
dataset of face images is collected andprocessed to extract the features needed fortraining the face recognition model.
(3) Train the face recognition model: The
extracted features are used to train a deep
Convolutional Neural Network (CNN) or othermachine learning model to recognize faces.
(4) Integrate face recognition system withattendance system: The face recognition modelis integrated with the attendance system so thatthe attendance of individuals can be recorded
and tracked in real-time.
(5) Test and evaluate the system: The system istested on a variety of real-world data toevaluate its accuracy and performance. Any
essential modifications are performed to raise
the system's efficiency.

Fig. 1. Use Case Diagram

4. Result and discussion
There is an interface design of face recognition based on
attendance system, as shown in Fig. 2. It is designed tobe simple to use and user-friendly for end users of all
ranges.

Fig. 1 represents two actors: the user and the admin.
A user can enter his or her personal information, such as
name, id, and course code. This information will be
stored in the system. Besides that, users can take face
images, train a model, view the result, or play an audio
file. If the username matches the result, the system
marks user attendance automatically. Users can exitfrom the program after using the system. On the other
hand, an admin can check user information, view theattendance records, and download an attendance list.

Fig. 2. Graphical User Interface (GUI) using QT Designer

The system includes six main functions, such as:
1. Play button: The user can click the play button
to play an audio file. This audio file introduces
the instructions should follow to complete all
the processes.
2.

Take image button: The user can click the takeimage button to register their faces. This button
will bring the user to another window, asshown in Fig.3.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
520

Koh Pei Cong, Heshalini Rajagopal, Ghassan Saleh Hussein Al-Dharhani, Norrima Mokthar, Dhanush Rachaveti

After the CNN model is trained, the model isevaluated on the test set to find the test score and test
accuracy, as shown in Fig. 6.

Fig. 6. Test Score(Loss) and Test Accuracy
Fig. 3: Take image window
3.

Then, the system uses focal length calculation toestimate the distance of the recognized face from thecamera, as shown in Fig.7.

Train model button: The user can click thetrain model button to train a CNN model. The
training process runs on the back-end
interpreter. Please give a few moments ofpatience until the window pops up a successmessage.

Fig. 7. Focal Length Calculation
After that, the system predicts images from eachdetected face in the frame, as shown in Fig. 8.

4.

Recognize face button: The user can click therecognize face button to recognize his or her
face. The result shows the name, id, and the
distance between an individual and the built-in
camera. The system will automatically record
the name, attendance date, and time.

5.

Download link: The user can click the
download file here to select the file todownload and select the destination folder tostore the file. The system will automaticallydownload files in the destination folder.

Fig. 8. Predict Image

The user can click the exit button to exit fromthe program immediately.

Lastly, the system updates the CSV file every time aface is recognized, as shown in Fig. 9.

6.

The system converts the images to grayscale, equalizes
the histogram, and normalizes pixel values between 0
and 1., as shown in Fig. 4.

Fig. 4. Preprocessing images
Next, the system preprocesses the data, creates the model,
applies image augmentation, and starts the training loop,
as shown in Fig. 5.
Fig. 9. Attendance records
The testing and evaluation phases are crucial for thedeveloped system. The system is tested on a variety ofreal-world data to evaluate its accuracy andperformance. Any essential modifications are performed
to raise the system's efficiency. This project covers unit
testing, functional testing, and user acceptance testing.
Fig. 5. Train CNN model©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
521

Face Recognition based on

Unit testing involves testing individual componentsof the system to ensure they function correctly. In thisproject, the unit testing helps to check the images arecaptured successfully, as shown in Fig. 10. Besides that,
this testing helps to check the model has been trained
successfully, as shown in Fig. 11. Moreover, it helps tocheck the face recognition done successfully, as shownin Fig. 12.

5. Conclusion
In conclusion, face recognition based on attendancesystem provides a functional solution for tracking
attendance automatically. The system includes features
such as face detection, face recognition, distanceestimation, and attendance recording. It exploits a pretrained Convolutional Neural Network (CNN) model forface recognition. Moreover, the system can recognize
faces with high accuracy and fast speed. Furthermore,
the system can accurately recognize known faces andunknown faces. In addition, the system stores
attendance records in a CSV file, making it easy to
manage and track their attendance. Besides that, thesystem works on a graphical user interface to interact
with end users easily.

Fig. 10. Capture Image Completed

References
1.

K. Alhanaee, M. Alhammadi, N. Almenhali, and M.
Shatnawi, “Face recognition smart attendance systemusing deep transfer learning,” in Procedia Computer
Science, 2021, vol. 192, pp. 4093–4102. doi:
10.1016/j.procs.2021.09.184.
2. F. Cahyono, W. Wirawan, and R. Fuad Rachmadi, “Facerecognition system using facenet algorithm for employee
presence,” in 4th International Conference on Vocational
Education and Training, ICOVET 2020, Sep. 2020, pp.
57–62. doi: 10.1109/ICOVET50258.2020.9229888.
3. H. Yang and X. Han, “Face recognition attendancesystem based on real-time video processing,” IEEE
Access, vol. 8, pp. 159143–159150, 2020, doi:
10.1109/ACCESS.2020.3007205.
4. I. Adjabi, A. Ouahabi, A. Benzaoui, and A. Taleb-Ahmed,“Past, present, and future of face recognition: A review,”
Electronics (Switzerland), vol. 9, no. 8. MDPI AG, pp. 1–
53, Aug. 01, 2020. doi: 10.3390/electronics9081188.
5. T. Schenkel, O. Ringhage, A. Márki, and J. Bae, “AComparative Study Of Facial Recognition Techniques
With focus on low computational power Bachelor Degree
Project in Information Technology Basic level 30 ECTS
Spring term 2019.”
6. V. Kumar, G. Jacek, M. Zurada, B. Raman, and G. R.
Gangadharan Editors, “Studies in Computational
Intelligence 885 Modern Approaches in Machine
Learning and Cognitive Science: A Walkthrough Latest
Trends in AI.”
7. X. Qu, T. Wei, C. Peng, and P. Du, “A Fast Face
Recognition System Based on Deep Learning,” inProceedings - 2018 11th International Symposium on
Computational Intelligence and Design, ISCID 2018, Jul.
2018,
vol.
1,
pp.
289–292.
doi:
10.1109/ISCID.2018.00072.
8. H. Zhi and S. Liu, “Face recognition based on genetic
algorithm,” J Vis Commun Image Represent, vol. 58, pp.
495–502, Jan. 2019, doi: 10.1016/j.jvcir.2018.12.012.
9. W. Shuhui and M. Xiaochen, “Dual-Proxy Modeling for
Masked Face Recognition,” in Procedia Computer
Science, 2022, vol. 208, pp. 145–151. doi:
10.1016/j.procs.2022.10.022.
10. Winarno, E., Al Amin, I. H., Februariyanti, H., Adi, P.
W., Hadikurniawati, W., & Anwar, M. T. (2019,
December). Attendance system based on face recognition

Fig. 11. Training Completed

Fig.12. Testing Completed
Functional testing involves testing the system as awhole to ensure that it meets the specified functional
requirements and performs correctly. In this project, thesystem recognizes known faces and unknown faces, as
shown in Fig. 13. Moreover, the system downloads anattendance file in the destination folder.

Fig. 13: Known face and Unknown face

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
522

Powered by TCPDF (www.tcpdf.org)

Koh Pei Cong, Heshalini Rajagopal, Ghassan Saleh Hussein Al-Dharhani, Norrima Mokthar, Dhanush Rachaveti

11.

12.

13.

14.

15.

16.

system using cnn-pca method and real-time camera. In
2019 International Seminar on Research of Information
Technology and Intelligent Systems (ISRITI) (pp. 301304). IEEE.
Ilyas, B. R., Mohammed, B., Khaled, M., & Miloud, K.
(2019, November). Enhanced face recognition system
based on deep CNN. In 2019 6th International
Conference on Image and Signal Processing and their
Applications (ISPA) (pp. 1-6). IEEE.
K. B. Pranav and J. Manikandan, “Design and Evaluationof a Real-Time Face Recognition System using
Convolutional Neural Networks,” in Procedia Computer
Science, 2020, vol. 171, pp. 1651–1659. doi:
10.1016/j.procs.2020.04.177.
K. H. Rahouma and A. Z. Mahfouz, “Design and
Implementation of a Face Recognition System Based on
API mobile vision and Normalized Features of Still
Images,” in Procedia Computer Science, 2021, vol. 194,
pp. 32–44. doi: 10.1016/j.procs.2021.10.057.
M. G. Sarwar, A. Dey, and A. Das, “Developing a
LBPH-based Face Recognition System for Visually
Impaired People,” in 2021 1st International Conferenceon Artificial Intelligence and Data Analytics, CAIDA
2021,
Apr.
2021,
pp.
286–289.
doi:
10.1109/CAIDA51941.2021.9425275.
D. Wang, H. Yu, D. Wang, and G. Li, “Face recognition
system based on CNN,” in Proceedings - 2020
International Conference on Computer Information and
Big Data Applications, CIBDA 2020, Apr. 2020, pp. 470–
473. doi: 10.1109/CIBDA50819.2020.00111.
Wandzik, L., Kaeding, G., & Garcia, R. V. (2018,
September). Morphing detection using a general-purpose
face recognition system. In 2018 26th European Signal
Processing Conference (EUSIPCO) (pp. 1012-1016).
IEEE.

Dr. Ghassan Saleh Hussein Al-Dharhani
He is currently an Assistant Professor
with the Institute of Computer Scienceand Digital Innovation, UCSI University,
Malaysia. His research interests include
Artificial Intelligence, Data Mining andKnowledge
Discovery,
MachineLearning, Data Science & Analysis
Dr. Norrima Mokhtar
Norrima Mokhtar was appointed as alecturer to serve the Department ofElectrical Engineering, University of
Malaya immediately after graduatingwith her Master of Engineering. She is
now serving Department of ElectricalEngineering, University of Malaya.
Dr. Dhanush R.
Dr. Dhanush R. is an Associate Professor
in the School of Electronics Engineering,
VIT Chennai, India. His research interests
include Neuroscience, Bio mechanics,
Motor learning, Motor Control.

Authors Introduction
Koh Pei Cong
He has completed his Bachelor of Science(Hons) in Computing, UCSI University,
Malaysia.

Dr. Heshalini Rajagopal
She received her PhD and Master's
degree from the Department of ElectricalEngineering, University of Malaya,
Malaysia in 2021 and 2016, respectively.
Her research interest includes image
processing, artificial intelligence andmachine learning.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
523

U-Reserve: Development of a Facility Reservation System for UCSI University
Esther Chong Jun Lynn, Neesha Jothi, Ismail Ahmed Al-Qasem Al-Hadi

Institute of Computer Science and Digital Innovation, UCSI University, Kuala Lumpur, Malaysia
Yeo Sook Fern

Faculty of Business, Multimedia University, Jalan Ayer Keroh Lama, Melaka 75450,
Malaysia Department of Business Administration, Daffodil International University, Dhaka 1207, Bangladesh
Email: neesha@ucsiuniversity.edu.my

Abstract
In today’s technology-driven era, there is an increasing emphasis on efficient and user-friendly methods to elevateuser satisfaction. UCSI University, Kuala Lumpur, currently relies on a manual facility reservation process,necessitating either in-person or email submissions. This outdated approach frequently leads to double bookings andunderutilization of resources. Thus, this project endeavours to create a web-based facility reservation system called
U-Reserve, employing the prototype methodology. U-Reserve was crafted using HTML, CSS, JavaScript, and coreJava (JSP and Servlet), while its database is administered through phpMyAdmin. Additionally, U-Reserve placesemphasis on incorporating preferred colour tones and font types of the target users, aiming to enhance user experienceand satisfaction. Through U-Reserve, the reservation request and cancellation processes are streamlined, and a datadriven dashboard facilitates informed decision-making regarding facility management.
Keywords: facility reservation system, web-based system, colour tone, design principle, data-driven decision-making,user permission management

measures, and enabling data-driven resource
optimization, which will benefit students, staff, lecturers,
and administrators at UCSI University.
The remainder of this paper includes sections on systemcomparison, literature review, development methodology,
conclusion, and future work, offering a comprehensive
overview of the development process, system features,and potential areas for further improvement.

1. Introduction
UCSI University’s ongoing expansion has posed
challenges in keeping facilities’ information up-to-date,leading to students and staff being unaware of available
resources. The Group Logistics Management Office(GLMO) depends on a manual reservation process,requiring students to email GLMO admin for facility
availability confirmation, leading to a three-day response
time. Subsequent booking procedures exacerbate
inefficiencies, risking overlapping reservations andincreasing the administrative workload.
Additionally, the manual confirmation process for Block
B’s discussion rooms at the library counter exacerbates
these challenges, leading to time wastage and increased
workload for library admins. This inefficiency
significantly impacts the overall experience andsatisfaction of students, staff, and lecturers at UCSI
University, potentially impeding academic performanceand participation in extracurricular activities [1], [2].
Therefore, this project introduces U-Reserve, a webbased facility reservation system aiming to address theissues associated with the current reservation process at
UCSI University. The system has been developed using
HTML, CSS, JavaScript, and core Java (JSP and Servlet)
to establish its functionality. For database management,
phpMyAdmin is used alongside the MySQL databasemanagement system. The primary objectives include
streamlining the reservation process, enhancing security

2. System Comparison
Comparison among four (4) existing facility reservation
systems—Bookeo [3], MRBS [4], Reservator [5], and
RoomBooker [6] highlights commonalities anddistinctive features, detailed in Table 1. These systems
share fundamental functionalities, enabling users to makereservations, check reservation history, and granting
administrators slot management capabilities. Theseessential features eliminate manual processes, prevent
double bookings, and ensure efficient slot control,
forming the foundation for U-Reserve. Table 1 illustrates
that Bookeo, MRBS, and Reservator offer reporting tools
for analyzing reservation trends. U-Reserve utilizes
graphs for visualizing data, aiding in decision-making.
Bookeo, MRBS, and RoomBooker send email
notifications to requesters—an absent feature in
Reservator. U-Reserve ensures prompt updates for users
upon request approval or rejection. Furthermore, MRBS
and Reservator display booked slots, whereas Bookeo
and RoomBooker exhibit available slots only. U-Reserve©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
524

Esther Chong Jun Lynn, Neesha Jothi, Ismail Ahmed Al-Qasem Al-Hadi

combines both, providing a comprehensive Home Page
table for clearer planning. Additionally, RoomBooker
features admin reservation management and userfriendly slot searches—attributes adopted by U-Reservefor enhanced control and convenience.

lives, effective user permission management remains apivotal component. It ensures system integrity,
operational efficacy, and fosters trust among both users
and administrators [7], [8], [9], [10].
3.1.1 Data-Driven Resource Optimization

Reservator

RoomBooker

U-Reserve

Make reservation.
Reservation
history
checking.
Slot management.
Report and analytics.
Email notifications.
Table showing booked slot.
Reservation management.
Search for available facilities
or slots.
Batch reservation.
Track reservation details.
View announcements andcontact information.

MRBS

Features

BookeoTable 1. Features Comparison Between Different Facility
Reservation Systems and The Proposed System.

√

√
√

√
√

√
√

√
√

√
√
√
×
×
×

√
√
√
√
×
×

√
√
×
√
×
×

√
×
√
×
√
√

√
√
√
√
√
√

×
×
×

√
×
×

×
×
×

×
×
×

√
√
√

√

Resource Management: Web-based systems have
revolutionized resource management, particularly indomains like facility management. These systems
provide a comprehensive platform for data analysis,
decision support, accessibility, and collaboration. In thecontext of facility management, they enable quantitativeand conceptual analysis, data sharing, remote
accessibility, and secure data management. Ultimately,
adopting web-based systems for resource management
empowers businesses to optimize their operations, reduce
costs, enhance safety, and improve overall efficiency. Byleveraging the capabilities of these systems,
organizations can ensure that their resource management
processes are both effective and efficient, ultimately
contributing to their success and competitiveness [11],[12].
Decision Making: Web-based systems have
revolutionized the decision-making landscape forbusinesses by offering tools and frameworks thatempower organizations to make informed, data-driven
decisions. Decision support systems, businessintelligence, analytics, interactive decision aids, andprovenance-based strategies all contribute to more
effective and efficient decision-making processes. In theera of big data and data science, web-based systems are
essential for organizations aiming to stay competitive andimprove business performance. The effective harnessingof data is a key driver of success, with web-based systems
playing a pivotal role in achieving this goal [13], [14],[15].

Moreover, MRBS introduces batch reservation options
for weekly, monthly, and yearly bookings. U-Reserve
customizes this feature for weekly reservations, offering
users flexible date selection to minimize unnecessary
bookings.
Furthermore,
U-Reserve
empowers
administrators to track reservation details, ensuring
seamless user contact. Users can also access department
announcements and contact information, streamlining
communication.
In summary, U-Reserve integrates functionalities from
Bookeo, MRBS, Reservator, and RoomBooker, while
incorporating unique elements to provide acomprehensive solution.

3.1.2 Design Principle

3. Literature Review
3.1.

System Design: It’s the core foundation for web-based
systems, addressing resource allocation, reliability,
security, usability, and learning system design. Its criticalrole ensures reliable performance, secures sensitive data,enhances usability, and promotes effective web-basedlearning. Good system design is essential for seamless
operation and user satisfaction in digital environments, beit large-scale enterprises or local host websites [16], [17].

Security

User Permissions: Managing user permissions is a
critical and multifaceted aspect within web-based
systems, serving as the cornerstone for security, privacy
protection, web personalization, and access control. As
emphasized in the referenced scholarly works, adopting
Role-Based Access Control (RBAC) models andpermission-based mechanisms equips web-based
systems with essential tools for handling user
permissions. The benefits are extensive, encompassing
the mitigation of security risks, safeguarding user
privacy, tailoring user experiences, and facilitating
secure, controlled resource access. In the evolving
landscape of web-based systems’ influence on our digital

Colour Theory: Color tone is a potent tool in web
design, influencing users’ perceptions, emotions, andbehaviours when used intentionally. With a well-matchedcolour scheme that complements content, designers craft
visually appealing designs and guide users effectively.
This approach fosters trust, improves user experience,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
525

U-Reserve: Development of aand drives engagement and satisfaction in web-based
systems [18], [19], [20].

Target Population: Students, lecturers, staff, andadmins of UCSI University, Kuala Lumpur.

Visual Appeal: Crucial in web system design, thelayout establishes flow and hierarchy, guiding users
effectively. Applying visual principles and aesthetics,
designers craft engaging, user-friendly systems,
contributing to satisfaction and success [21], [22].

Sampling Methods: Purposeful sampling entails
preselecting participants based on specific criteria toensure diverse insights into various system aspects.
Snowball sampling recruits additional participants
meeting specific criteria, allowing respondents to referothers.

3.2 Applications of Security, Data-Driven Resource
Optimization, and Design Principle on U-Reserve

Data Collection: Primary data collection involves
identifying functional and non-functional requirements,
as well as preferences for colour tone and font type for UReserve. Secondary data from sources such as journals
guides the acquisition of user requirements, supporting
project insights and results.

Table 2. Applications of Security, Data-Driven Resource
Optimization, and Design Principle on U-Reserve.
Area
Applications
Security.
● A role-based security system prevents
errors and safeguards system integrity.
Data-Driven● Decision
supportdesignforResource
administrators: comparing requests andOptimization
enhancing efficiency.
.
● Dashboard presenting visualised data fordata-driven facility management.
● Downloadable Excel report providing
valuable insights into facility utilisation.
Design● User preferences on colour tones andPrinciple.
font types are gathered for UI design.
● Logical page structure, consistentnavigation bar, and organised content
ensure coherence.
● Font size variations create a clear visualhierarchy on pages.

4.4 Analysis
Functional and Non-functional Requirements
Analysis: A total of seventy-five (75) users actively
participated in this survey via Google Forms, providing
valuable insights and feedback.
Functional requirements include checking availability,
making reservation requests, checking request status,
request history, and tracking reservation details. All
highlighted functional requirements identified byrespondents are integrated to enhance user satisfaction.
Priority will be given to enabling users to checkavailability, make reservation requests, and check request
status, as these received the highest percentages: 93.33%,
86.67%, and 74.67%, respectively.

4. Development Methodology
The project was completed using the prototype
methodology, as illustrated in Fig. 1. This approach wasselected due to its capacity to involve users in identifying
missing functionalities at an early stage [23].

4.3 Planning

Moving to non-functional requirements, feedback on
the ease of use of a facility reservation system shows that73.33% prefer high user-friendliness. U-Reserveprioritizes easy navigation and efficient user interactions
in line with this preference. The next query assesses
reliability and security, with 53.33% rating them as very
important, guiding U-Reserve’s robust security design.
The importance of 24/7 availability is evident, with65.33% considering it crucial for user convenience andefficiency. Lastly, 70.67% emphasize scalability’s
importance for a reservation system’s effectiveness.
Despite its local host development, U-Reserve needs tomeet availability and scalability as vital non-functional
requirements when adopted by UCSI University.

Location of Study: All research activities wereconducted at UCSI University, Kuala Lumpur, Malaysia,
and remotely via Microsoft Teams using an Internet
connection.

User Interface (UI) Preferences Analysis: The
feedback received from these one hundred (100) users via
Google Forms provided insights on their preferred colour
tones and font types for U-Reserve.

Fig. 1 Prototype Methodology [24]©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
526

Esther Chong Jun Lynn, Neesha Jothi, Ismail Ahmed Al-Qasem Al-Hadi

Four (4) commonly used colour tones - blue, red-violet,
green, and red - were selected for the user interface (UI)of U-Reserve. Respondents could choose their preferredcolour tone for each questionnaire question. Blue, linked
to familiarity and professionalism, emerges as the top
choice across multiple criteria: visual appeal (32%),
suitability (39%), calming effect (42%), professionalism(34%), modernity (30%), and accessibility (47%). Thispreference for blue aligns with research highlighting its
universal appeal, trustworthiness, and readability indigital environments. Notably, cool colours like blue andgreen consistently dominate preferences (60.83%),reflecting users’ preference for calming and trustworthy
associations. Emphasizing the importance of contrast in
U-Reserve’s UI, achieved through thoughtful design
considerations, is crucial for readability and enhancing
the overall user experience [25].
Additionally, respondents were asked to select a font
type—Josefin Sans + Dosis, Sans-Serif, Verdana, Arial,or Georgia—based on suitability, readability, andaesthetic appeal for the UI design. Verdana emerges as
the top choice, favoured by 38% for suitability, 48% forreadability, and 49% for aesthetic appeal. Sans-Serif andthe combination of Josefin Sans + Dosis also receive
varied support. The consistent preference for Verdanaaligns with research emphasizing its screen-centric
design, readability advantages, and aesthetic appeal,making it the optimal font type for a user-friendly
experience in a facility reservation system [26].

Fig. 3 Home Page of U-Reserve

4.5 Design

Fig. 4 Block B Facility Checking and Reservation Pageof U-Reserve

System Design:

Users can check availability by viewing the cell’scolour and make reservations by clicking on the cells.
Additionally, users can navigate to specific facility
reservation pages for Block B, as illustrated in Fig. 4, or
Block C, as illustrated in Fig. 5.

The following figures show the main pages of UReserve.
Fig. 2 displays a login page with ID number andpassword fields. Regardless of their role, users must
provide the correct ID number and password forauthentication.

Fig. 2 Login Page of U-Reserve
Fig. 5 Block C Facility Checking and Reservation Pageof U-Reserve

After a successful login, the home page, as
depicted in Fig. 3, is displayed, showcasing a navigationbar, announcement slides, contact information, and anavailability table for facility reservations.

After selecting a facility, users are redirected to
the reservation page, as depicted in Fig. 6 which is used
to fill in the details. Additionally, users can review theirreservation request list and check the status from thereservation history page, as shown in Fig. 7. Finally,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
527

U-Reserve: Development of aadmins can approve or reject reservation requests on thereservation approval and checking page, as illustrated inFig. 8.

Fig. 6 Reservation Page of U-Reserve

Fig. 9 Use Case Diagram for U-Reserve
Students and staff possess various capabilities,
including logging in, password management, accessing
announcements and contact details, checking facility
availability, making reservations, reviewing reservation
history, and canceling reservations. Additionally,lecturers enjoy extended privileges, allowing batch
reservations specifically for Block C classrooms. GLMO
and library admins have the highest permissions, sharing
fundamental functionalities with students and staff while
also conducting batch reservations, adding new slots, andoverseeing reservation requests. GLMO admins canaccess a dashboard and download usage reports for Block
C classrooms, whereas library admins focus on managing
discussion rooms in Block B, significantly contributing
to efficient oversight and management.

Fig. 7 Reservation History Page of U-Reserve

Fig. 8 Reservation Approval and Checking Page of
U-Reserve

Use Case Diagram:

Entity Relationship (ER) Diagram:

The U-Reserve use case diagram delineates three user
roles: students and staff, lecturers, and administrators(Group Logistics Management Office (GLMO) andlibrary admin), illustrated in Fig. 9.

Fig. 10 ER Diagram for U-Reserve©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
528

Esther Chong Jun Lynn, Neesha Jothi, Ismail Ahmed Al-Qasem Al-Hadi

In the ER Diagram displayed in Fig. 10, four entities are
represented: “facilitylist”, “availslot”, “reserve”, and“user”. The “facilitylist” entity stores data about all
facilities, utilizing “FacilityID” as its unique identifier,and includes mandatory attributes like “Block”,“Capacity”, “Equipments,” as well as optional attributes
such as “Images”, “Level”, and “Type”. The “availslot”
entity comprises mandatory attributes “Date” and“Time.” Identified by “ReserveID,” the “reserve” entity
contains mandatory attributes like “Date,” “Time,”“NumberOfUser,” and “status,” along with optional
attributes like “RequestReason” and “AdminReason”.
The “user” entity, marked by “ID,” encompasses
mandatory attributes like “Password”, “Name”, “Email”,
and “Role”. Relationships are established using UID
bars: a facility can have multiple available slots, eachuniquely linked to a facility through the “FacilityID”
attribute. Similarly, a facility can be prepared for multiple
reservations, each associated with a facility through thesame “FacilityID”. Finally, a user can make multiple
reservations, with each reservation linked to a user
through the “ReserveID” attribute.

automatically rejecting them, updating the status, andsending notifications.

Fig. 12 Activity Diagram for U-Reserve – Make
Reservation
As per Fig. 12, users reserve through two methods: via
the home page’s availability table or Block B/Block C
facility checking and reservation page. To reserve fromthe home page, users scroll, pick a date from the datepicker, check facility availability, select time and facility,
and click a cell. This redirects to the reservation page
where they input user count and reason, get confirmation,
and are sent to the reservation history. Alternatively, fromthe facility checking and reservation page, users choose
Block B/C, set date, time, capacity, select a facility, andclick “Reserve”. This proceeds to the reservation page
following the same steps.

Activity Diagram:

Fig. 11 Activity Diagram for U-Reserve – Approve or
Reject Reservation Requests
Fig. 13 Activity Diagram for U-Reserve – Batch
Reservation

In Fig. 11, users handle reservation requests byaccessing the reservation approval and checking page
through the “Request Approval” link. Here, they canbrowse and filter pending reservation requests by ID
number, facility, date, and time. Users review request
details and decide to approve or reject. Giving a reason
for rejection is necessary. The system updates the status
and notifies requesters via email. Approved requests
prompt the system to check for similar ones,

In Fig. 13, users perform batch reservations on the batch
reservation page. Here, users select the day of the week,
month, and year. The system retrieves corresponding
dates from the database and presents them. Users thenpick the facility, date, and time, enter user count andreasons, and submit the request. The system checks forexisting reservations for the selected slots. If available,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
529

U-Reserve: Development of athe system adds the new reservation request to the
database; otherwise, it skips it. This cycle repeats for eachselected date until completion. Users are notified of thesubmitted requests and redirected to the reservation
requests history page.

Fig. 16 Activity Diagram for U-Reserve – Check
Reservation History
Users can access their reservation request history
by clicking the “Reservation History” link on thenavigation bar, as depicted in Fig. 16. Upon reaching thereservation requests history page, users can observe all
their reservation requests. They have the option to input
the Reservation ID, select a status, or both to refine thehistory. Subsequently, the system connects to the
database, filters the results based on the provided criteria,
and displays them to the users.

Fig. 14 Activity Diagram for U-Reserve – Track
Reservation Details
In Fig. 14, users can monitor reservations for aparticular facility, date, and time via the reservation
details checking page. They choose the desired
parameters, and the system scans the database forreservations that match the criteria. If found, the system
displays the Reservation ID and requester’s ID number;otherwise, it indicates no reservations.

Fig. 17 Activity Diagram for U-Reserve – Export
Excel Report
In Fig. 17, users can generate an Excel format
utilization report from the dashboard page by clicking the“Dashboard” link on the navigation bar. They choose thepreferred month and year, and the system filters anddisplays the corresponding data. Clicking the “Export
Excel Report” button prompts the system to generate anddownload the report.

Fig. 15 Acytivit Diagram for U-Reserve – Add New
Slots

4.6 Development and Implementation

In Fig. 15, admins can include reservation slots byclicking the “Add New Slot(s)” button to access the Add
Slots page. On this page, they select the facility, date, andtime, and submit the details to the system. For each slot,
the system verifies its existence in the database; if alreadypresent, the system skips that slot. Upon completion of
the loop, the system informs admins about the number ofsubmitted slots.

Table 3 displays the tabulated data from the user
acceptance test (UAT) forms, utilising a Likert scaleranging from 1 for strongly disagree to 6 for strongly
agree, as suggested by research [27], as shown in Fig. 18.
The analysis used the arithmetic mean technique,following the methodology of [28]. A comprehensive
evaluation of different sections can be accomplished by
calculating the mean rating for all items within eachsection. A mean rating exceeding five (5) indicates user
agreement. In this three-scale representation of©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
530

Esther Chong Jun Lynn, Neesha Jothi, Ismail Ahmed Al-Qasem Al-Hadi

Assuming I had access tothe system, I intend to use
it.

agreement, encompassing 4 - Agree, 5 - Somewhat
Agree, and 6 - Strongly Agree, five (5) serves as the
midpoint.

Table 3 User Acceptance Test (UAT) Result.
1

2

3

4

5

6

Mean
Ratin
g

User Interface Design [27]Appropriate colour tone.
0

1

1

5

0

0

0

3

Appropriate font size.

0

0

0

3

7

Navigation is easy.

0

0

0

3

7

Navigation is clear and 0
concise.
Perceived Usefulness [29]Using the system in my 0job/study increases myproductivity.
Using
the
system 0
enhancesmyeffectivenessinmyjob/study.
I find the system to be 0
useful in my job/study.
Perceived Ease of Use [29]My interaction with the 0system is clear andunderstandable.
Interacting with the 0system does not require a
lot of my mental effort.
I find the system to be 0
easy to use.
I find it easy to get the 0system to do what I want
to do.
Output Quality [29]I have no problem with 0
the quality of thesystem’s output.
I rate the results from the 0system to be excellent.
Result Demonstrability [29]The results of using the 0system are apparent tome.
I would have no difficulty 0
explaining why using thesystem may be beneficial.
Behaviour Intention [29]

0

0

1

8

1
3
1
8
2
0
2
0
2
1

5.1000Appropriate font type.

1
0
9

0

0

4

9

1
7

5.4333

0

0

1

1
0

1
9

5.6000

0

0

2

1
1

1
7

5.5000

0

0

1

1
3

1
6

5.5000

0

0

1

1
0

1
9

5.6000001

85.6667

0

0

1

1
0

2
1
1
9

0

0

0

1
0

2
0

5.6667

In addition, Table 3 presents survey results indicating afavourable user perception of U-Reserve across various
aspects, such as UI design, effectiveness, ease of use,
output quality, result demonstrability, and behaviouralintention. The system meets user expectations andpreferences satisfactorily. However, potential areas for
improvement have been identified, and several
suggestions are currently under consideration. In essence,
U-Reserve stands as a well-developed system proficientin efficiently managing facility reservations for UCSI
University while assisting university staff in handling
lecturers’ timetables.

Fig. 18 Likert Scale [27]

Item

0

5. Conclusion and Future Work5.5000In conclusion, the development of U-Reserve, UCSI
University’s web-based facility reservation system,
followed a prototype methodology involving planning,
analysis, design, and implementation. This approachsuccessfully engaged users early in the process,incorporating feedback from user acceptance tests tomodify and enhance the system. Notable contributions
include streamlining the reservation process for students,
staff, lecturers, and administrators, reducing workloads,
and improving facility management through data-driven
analyses.

5.56675.56675.6667

U-Reserve meets user preferences for colour tone andfont type, fulfilling functional and non-functional
requirements. It stands out from existing systems by
integrating their features while introducing uniqueelements like tracking reservation details and offering
announcement and contact information. Moreover, thecomprehensive project report lays the groundwork for
implementing the facility reservation system at UCSI
University, signifying a substantial achievement for the
institution and its users.

5.6000

0

0

3

1
1

1
6

5.4333

0

0

2

9

1
9

5.5667

0

0

1

1
1

1
8

5.5667

0

0

4

1
1

1
5

5.3667

Future improvement suggestions for the facility
reservation system at UCSI University include
redesigning for formality and consistency, migrating to a
cloud-based or hosted server, implementing approval
periods for reservation requests, integrating with central
databases and the IIS, expanding reservation options, andenhancing the password reset mechanism. Theseenhancements aim to boost the system’s functionality,
security, and user experience, aligning it with the
evolving needs of the university community.
Acknowledgements
I extend my deepest appreciation to my first and second
supervisors, Asst. Prof. Ts. Dr. Neesha AP Jothi and Dr.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
531

U-Reserve: Development of a

Economics and Business, vol. 70, no. SI, pp. 43–54, Feb.
2023.
16. K.-S. Joo and J.-W. Woo, “Development of ObjectOriented Analysis and Design Methodology for Secure
Web Applications,” International Journal of Security and Its
Applications, vol. 8, no. 1, pp. 71–80, Jan. 2014.
17. Y.-M. Kim, “Factors Affecting University Library Website
Design,” Information Technology and Libraries, vol. 30, no.
3, Sep. 2011.
18. L. Khrouf and A. Frikha, “Web-surfers’ conative reactions
to the website’s dominant hue: mental imagery’s role,”
Internet Research, vol. 26, no. 5, pp. 1249–1268, Oct. 2016.
19. X. Zhang, W. Hu, and Q. Xiao, “Influences of Medical
Crowdfunding Website Design Features on Trust andIntention to Donate: Controlled Laboratory Experiment,”
Journal of Medical Internet Research, vol. 23, no. 5, p.
e25554, May 2021.
20. T.-C. Tung and H.-Y. Chen, “Integrating Conjoint Analysis
with TOPSIS Algorithm to the Visual Effect of Icon Design
Based on Multiple Users’ Image Perceptions,” Eurasia
Journal of Mathematics, Science and Technology
Education, vol. 13, no. 3, pp. 1025–1040, Dec. 2016.
21. D. Cyr, M. Head, H. Larios, and B. Pan, “Exploring Human
Images in Website Design: A Multi-Method Approach,”
MIS Quarterly, vol. 33, no. 3, p. 539, 2009.
22. S. Djamasbi, M. Siegel, and T. Tullis, “Generation Y, web
design, and eye tracking,” International Journal of HumanComputer Studies, vol. 68, no. 5, pp. 307–323, May 2010.
23. K. Kirpitsas and T. P. Pachidis, “Evolution towards Hybrid
Software Development Methods and Information Systems
Audit Challenges,” Software, vol. 1, no. 3, pp. 316–363,
Sep. 2022.
24. H. M. A. Wahab, M. H. Hassan, N. Mohd, Z. Hanafi, andHafizul, “WEB BASED INTELLIGENT APPOINTMENTSYSTEM,” Jul. 2023.
25. R. Kimmons, “Color Theory in Experience Design,”
Learner and User Experience Research, pp. 103–125, 2020,
Accessed: Nov. 14, 2023.
26. Banerjee and M. Bhattacharyya, “Selection of the optimum
font type and size interface for on screen continuous readingby young adults: an ergonomic approach,” Journal of
Human Ergology, vol. 40, no. 1–2, pp. 47–62, Dec. 2011.
27. C. K. N. C. K. Mohd and F. Shahbodin, “Personalized
Learning Environment: Alpha Testing, Beta Testing & User
Acceptance Test,” Procedia - Social and Behavioral
Sciences, vol. 195, pp. 837–843, Jul. 2015.
28. M. Osman, M. Maghribi, M. Nurzaid, P. Zulfikri, M.
Othman, and M. Shahrol, “Mobile cloud computing for mlearning application,” vol. 10, pp. 1055–1068, May 2018.
29. Ozdemir and A. K. Kabakus, “User acceptance of cloud
based hospital information system,” typeset.io, vol. 6, no. 1,
pp. 28–46, Mar. 2019.

Ismail Ahmed Al-Qasem Al-Hadi, for their invaluable
support, guidance, and mentorship throughout thisproject. Their patience and expertise have played acrucial role in shaping the direction and scope of thisstudy, and I am truly thankful for their contributions. I
also want to express my sincere gratitude to the lecturers,
administrators, staff, and students of UCSI University for
their cooperation in conducting the user acceptance tests(UATs) and providing valuable feedback. Special thanks
to Assistant Professor Ts. Dr. Ghassan Saleh for helping
me in finding more lecturers and staff to participate in my
UATs. Additionally, I acknowledge the support of myfriends, who assisted me in collecting details and imagesof the facilities.
References
1. Testech, “Effects of Poor Facilities Management Integrated Facilities Management,” Testech Group, Aug. 07,2022.
2. M. Mosley, “Why is Word of Mouth Marketing soImportant?,” Social Media Today, Jun. 25, 2017.
3. B. Hogan, “Bookeo Review, Pricing & Features: Popular
Online Appointment Booking Software for Small
Businesses,” SoftwarePundit, Feb. 26, 2018.
4. J. Beranek, “MRBS: Introduction,” mrbs.sourceforge.io,
2008.
5. Joelmertanen et al., “Reservator,” GitHub, Oct. 01, 2023.
6. Queen Margaret University, “Space and Timetabling
BOOKING A ROOM VER. 1.1 Edinburgh College | MIS
BOOKING A ROOM GUIDE 2 Contents,” 2019. Accessed:
Nov. 19, 2023.
7. J. Chen and T. Zhang, “Research and Implementation of
Role-Based Access Control Model Based on Partition
Number,” 2009 Second International Symposium on
Computational Intelligence and Design, vol. 2, Jan. 2009.
8. B. Mishra et al., “Privacy Protection Framework forAndroid,” IEEE Access, vol. 10, pp. 7973–7988, 2022.
9. T. Zhang and W. Tan, “Role-based dynamic access control
for Web services,” 2010 International Conference on
Computer Application and System Modeling (ICCASM
2010), vol. 4, Oct. 2010.
10. E. K. Zavadskas, A. Kaklauskas, M. Gikys, and N. Lepkova,“A multiple criteria decision support web-based system forfacilities management,” International Journal of Internet
and Enterprise Management, vol. 2, no. 1, p. 30, 2004.
11. S. Casadei, A. Pierleoni, and M. Bellezza, “Sustainabilityof Water Withdrawals in the Tiber River Basin (Central
Italy),” Sustainability, vol. 10, no. 2, p. 485, Feb. 2018.
12. F. Al-Hawari, M. Al-Zu’bi, H. Barham, and W. Sararhah,“The GJU Website Development Process and Best
Practices,” Journal of Cases on Information Technology,
vol. 23, no. 1, pp. 21–48, Jan. 2021.
13. Das, “Effectiveness of Web-Based Decision Making to
Deliver Decision-Support Information to Business Analyst
using a ‘Thin-Client,” Technoarete Transactions on
Advances in Data Science and Analytics, vol. 1, no. 1, pp.
1–7, Feb. 2022.
14. F. Provost and T. Fawcett, “Data Science and its
Relationship to Big Data and Data-Driven Decision
Making,” Big Data, vol. 1, no. 1, pp. 51–59, Mar. 2020.
15. L. Hurbean, F. Militaru, M. Muntean, and D. Danaiata,“The Impact of Business Intelligence and Analytics
Adoption on Decision Making Effectiveness andManagerial Work Performance,” Scientific Annals of

Authors Introduction
Ms. Esther Chong Jun Lynn
She is currently a degree student at UCSI
University, Kuala Lumpur, Malaysia,
studying for a B.Sc. (Hons) in BusinessInformation System.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
532

Powered by TCPDF (www.tcpdf.org)

Esther Chong Jun Lynn, Neesha Jothi, Ismail Ahmed Al-Qasem Al-Hadi

Dr. Neesha Jothi
She received her PhD from the School ofComputer Sciences, Universiti Sains
Malaysia in 2020. She is currently an
Assistant Professor in UCSI University,
Malaysia. Her research interest areas are
Data Mining in Healthcare and Health
Informatics.
Dr. Ismail Ahmed Al-Qasem Al-Hadi
He is a highly qualified and experienced
Assistant Professor who is currently
working at UCSI University in Kuala
Lumpur, Malaysia. He has a strong
background in computer science, with a
PhD in Intelligent Computing and a
Master’s degree in Computer Science. He also holds a
Bachelor’s degree in Comp Science.

Dr. Sook Fern Yeo
She is an Assistant Professor at the Facultyof Business, Multimedia University,
Melaka. She is currently holding theposition as of the Deputy Dean for
Research & Industrial Collaborations. Her
current research areas are service
innovations, consumer behaviour, social
media marketing, branding, supply chain
management, healthcare management and tourism
marketing..

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
533

The Smart Document Processing with Artificial Intelligence
Raenu Kolandaisamy, Heshalini Rajagopal

Institute of Computer Science and Digital Innovation, UCSI University, 56000 Kuala Lumpur, Malaysia
Indraah Kolandaisamy

School of Business Management, University Utara Malaysia 06010 Sintok, Kedah
Glaret Shirley Sinnappan

Tunku Abdul Rahman University of Management and Technology, Kuala Lumpur, Malaysia
E-mail: raenu@ucsiuniversity.edu.my

Abstract
This study focuses on the challenges and potential for Intelligent Document Processing (IDP) with Artificial
Intelligence (AI) to manage unstructured data. A large amount of data in many different forms, such as informationfrom the IoT, cybersecurity and more are produced during this modern Digital Age which has been widelydistributed through a wide range of unformatted formats. IDP utilizes AI technologies like Machine Learning (ML),
Natural Language Processing (NLP), and Computer Vision (CV), with the aim of converting unstructured data into
structured, usable information. The IDP, are explored in the findings and discussion section that emphasizes its
capability to automatically perform redundant tasks, reduce operating costs as well as improve employee
productivity.
Keywords: Artificial Intelligence (AI), Natural Language Processing (NLP), Document Processing, Computer
Vision (CV)Based on Forbes statistics, it reported that 95% ofbusiness organizations struggle to analyze unstructured
data due to they do not have the required expertise todeal with unstructured data [2]. In a business company,
80% to 90% of data is in the form of unstructured
format (Sukrutha, 2023). Business companies facesignificant challenges in effectively processing such
unstructured data. In addition, the process of
unstructured data analysis is always time-consuming
and costly for any organization [2].
However, all organizations are required to extract
valuable information from unstructured data to accessfunctional information. Thus, automated analysis is
crucial in helping the organization manage and accessuseful information in a structured manner. By
automating the unstructured data kept in digital formats,organizations are allowed to quickly gain insight into
their businesses, increase their competitive edge,
improve productivity, and drive innovations. Artificial
Intelligence-based (AI-based) technologies are criticalfor organizations that would like to adapt to automation
solutions [2].

1. Introduction
In today's world, everything is connected to a data
source, and digitization has made it possible to record
everything. The current electronic era has an abundanceof data available in various forms including the Internetof Things (IoT) data, cybersecurity data, smart city data,business data, smartphone data, social media data,
health data, COVID-19 data, etc. This data can be
structured, semi-structured, or unstructured [1].
With the emergence of new communication platforms
and diverse applications such as social media, mobile
applications, and digital marketing, the data delivered
lacks a typical format or predefined schemas like thestandard data and is unmanageable with the relational
database models. Data is generated in various forms like
text, audio, videos, emails, and images, which are
categorized as unstructured data. Unfortunately, suchdata lacks structure and standardization, which leads todifficulty for most organizations to edit, search, andanalyze [2].

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
534

Raenu Kolandaisamy, Heshalini Rajagopal, Indraah Kolandaisamy, Glaret Shirley SinnappanIntelligent Document Processing (IDP) is the conversion
process of unstructured data such as email, images, and
PDF documents to functional data. Intelligent
Document Processing is one of the processes carried out
by artificial intelligence (AI) and AI technologies are
one of the best automation processes of the next
generation. It involves Computer Vision (CV), Deep
Learning (DL), Machine Learning (ML), etc. for data
extraction [3].
The field of Artificial Intelligence (AI) is widely
renowned as one of the most exciting and promisingareas in the technology landscape. Analysts believe that
AI will play a vital part in any business sector andindustry. Besides that, the potential for value creation
across the economy with machine learning (ML), a
major subset of AI, is vast. It not only affects consumers,
but it also transforms the way businesses operateaTsyapewtheoxlet . [4].
The increasing demand for the effective use of
unstructured data necessitates the development of an
AI-based unstructured document processing framework.
This framework can aid organizations in extracting
valuable insights from unstructured data automatically.
Therefore, this research area is considered a significant
and an important research topic [2].
Extracting insights from unstructured data can bechallenging due to the lack of a predefined schema or
format. Therefore, it is more challenging to analyze andprocess. Besides that, it also required the users to have acertain level of understanding to utilize the AI
technologies and techniques. Thus, the main research
objectives of this study are listed as below:• To review the advantages and disadvantages of
IDP with AI-based techniques for unstructured
data.
• To understand the challenges with IDP to
manage unstructured data.

[6], it is shown that there are some potential issues andchallenges with using machine learning, such ascollecting sufficient and relevant training data. Besides
that, another challenge is the development methods toautomate document processing using machine learning,particularly for complex documents. Moreover, it
requires efficient and effective technology andtechniques to process and analyze fast streams of
unstructured data. The complexity of unstructured data
poses challenges for information extraction, especiallyfor streaming data. Structural variation and differencesin data formats are the critical obstacles associated with
unstructured big data [7]. To fully leverage the IDP andgain access to valuable insights, organizations must go
beyond simply knowing about it and actively
incorporate it into their management and utilization ofdata.
Extracting insights from unstructured data can bechallenging due to the lack of a predefined schema or
format. Therefore, it is more challenging to analyze andprocess. Besides that, it also required the users to have acertain level of understanding to utilize the AI
technologies and techniques. Thus, the main research
objectives of this study are listed as below:• To review the advantages and disadvantages of
IDP with AI-based techniques for unstructured
data.
• To understand the challenges with IDP to
manage unstructured data.
2. Literature Review
Intelligent Document Processing (IDP) is one of themost popular technologies in this modern era with ahigh demand focused on the automation of the way
humans work with different types of documents on adaily basis. Besides, Digital transformation is also oneof the trending topics nowadays as Robotic Process
Automation and other AI technologies such as ML work
toward automating the ordinary operations performedby users. These AI technologies work together toprocess documents intelligently with limited human
involvement [8].
The possibilities of electronically exchanging structured
information continue to rise significantly. Nevertheless,a large amount of business processes continue to centeron the sending, receiving, and processing of
unstructured documents. According to Prasad (2020),
only 20 % of data in enterprises is structured and therest of it is unstructured data with most of it locked in
documents. Companies with the adaptation of an
automated document processing solution reported a
reduction of 50% in their staffing and labor cost [9],while processing time is reduced to less than 20 %compared to manual processing time. It shows that

Intelligent Document Processing (IDP) is a powerful
technology that can transform unstructured and semistructured data into a structured format. By leveraging
advanced AI technologies like machine learning (ML)and natural language processing (NLP), IDP can capture,classify, and extract even the most complex andchallenging data. This extracted data can then bevalidated and automatically entered into existing
applications using Robotic Process Automation (RPA)
technology [5].
According to [4], although advanced analytics and ML
have been shown to provide a competitive advantage,
many enterprises still struggle with their adoption. The
lack of a clear AI strategy, talent, and data are major
barriers, along with soft factors like leadership,decision- making, and company culture. According to©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
535

The Smart Document Processing

automation of business document processing is the key
element for businesses to be more cost-effective andcost efficient.
Robotic Process Automation (RPA) refers to theautomation of service tasks to perform repetitive humantasks precisely with the assistance of AI. The developer
set the task instruction with the use of some form ofscreen recording and defining variables. Logging into
applications, copying and pasting data, opening emails,
filling forms are some of the examples of tasks.
Different from the traditional automation method, RPA
mainly emphasizes element identification instead ofscreen coordinates or XPath selections which results inbetter intelligent interaction with the user interface [9].
An alternative to the manual processing of documents isnecessary to limit human involvement, minimize error
rates and manage the time and cost efficiently andeffectively. In this case, the research of automatic
document processing has become one of the trending
topics in recent years. RPA, also known as digital
employee, has become a powerful tool to facilitate theautomatic processing of business documents. RPA
shows its effectiveness by enabling simple configurationof robots to handle high-volume and repetitive tasks
automatically based on predefined rules which not only
limits the human errors, but also improves the efficiency
and cost reduction [10].
AI-based approaches are capable of extracting useful
information from unstructured documents automatically.
AI-based approaches are used in numerous applications
such as invoice digitization, health record extraction,
metadata extraction, machine translation, text
summarization, insurance claims processing, andcontract analysis [2].
Natural Language Processing (NLP) is one of thesubfields of Artificial Intelligence that focuses on
empowering computers to understand and process
human languages, bringing them closer to human-likelanguage understanding. Named Entity Recognition(NER) is an important component of NLP systems,
which involves the process of automatic identificationof named entities in a given text or document. Named
entities refer to real-world objects such as people,
location, date, and time. Recognizing and extracting real
named entities is vital for various research areas,
including Question Answering and Summarization
Systems, Information Extraction, Machine Learning,
Semantic Web Search, Video Annotation, and etc [11]According to [12], significant improvements have been
made in extracting information from free text using
NLP techniques based on AI. NLP uses syntactic rules
to automatically scan and extract named entities likenames, locations, organizations, dates, and invoice
numbers from unstructured text using NER. The main
steps in NER involve text pre-processing, extracting

NER features, then training and classification. In text
pre-processing, the given text will be formatted to beunderstandable for machine learning algorithms. Then,
features are extracted to create a numericalrepresentation because the NLP model could not workon the text data. Lastly, the NER model will classify andcategorize words and phrases. [2].
Computer Vision (CV) is also a component of AI thatenables systems to identify and extract useful
information from scanned document images and othervisual inputs. The goal of computer vision is to extract
meaningful information from images with thedevelopment of technology that enables computers or
systems to understand and recognize digital images [13].
Machine Learning (ML) is a type of data-driven AI thatprovides the ability to learn about a system without
explicit programming [14]. AI Algorithms and Machine
Learning (ML) approaches have been effectively usedin real-world scenarios which include the digital
services, industry and commerce throughout the years.
ML acts as the teacher of the machines which “teach”
the efficient data handling skills by simulating thelearning concept of rational beings. It can be achieved
with Al algorithms that reflect diverse rational
paradigms including connectionist, genetics, statistics
and probabilities, based on cases, etc. The integration of
Al algorithms and the ML approach provide a
possibility for the exploration and extraction of
information to perform tasks such as classification,
association, optimization, grouping, prediction andpattern identification. [9]Since a large amount of enterprise data these days is
usually unstructured data which has no predefined
schema or format, it is more complicated andchallenging for users to collect, process and analyze thedata without proper structure [2]. In this case, Intelligent
Document Processing with AI is proven to be essential
and necessary for the processing of unstructured
documents.
However, there are several challenges and limitations
that might affect the performance of which need to beaddressed. There are major challenges regarding the useof Unstructured Data. Inaccurate text data is added by
OCR text extraction in the form of noisy datasometimes. This is a key problem related to data.
Unstructured data originating from multiple sources are
not standard and have different formats of data. The
lack of a sufficient supply of data and insufficientquality is also another challenge in data processing with
unstructured data. Moreover, it is difficult to obtain
information in a language that is complex such as
Arabic, without the creation of dictionaries. It isdifficult with information extraction methods todescribe the semantic and contextual relationships
among named entities in this ambiguous language.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
536

Raenu Kolandaisamy, Heshalini Rajagopal, Indraah Kolandaisamy, Glaret Shirley Sinnappan

Another challenge is posed by domain specific entities.
For example, the biomedical datasets are different fromany other dataset in terms of domain specific entities [2].
As the business process requires handling andprocessing different types of unstructured documentsfrom the clients or suppliers such as invoices, ID cards
and application forms, it poses a challenge as these
documents differ in type, form and layout which requireclassification based on their type and nature. Therefore,
the AI technologies used for document processing
should be able to extract the relevant fields from theparticular documents with the application of either
template-based or template-free approach. Moreover, AI
technologies should be able to process and improve thequality of scanned documents, which is often submittedwith low quality scanners or mobile devices. The
collection of relevant target data from documents iscomplicated when it comes to managing multi-pages of
unstructured documents which consist of tables withdata that span across different pages [2].

4. Result and discussion
A questionnaire is conducted to review and understandabout this study and research objectives. The
questionnaire results from the respondents are shown in
the Fig. 1.

3. Method
A case study method will be used for this study due
to various data and information gathering have been
undertaken to study this topic. A case study is a researchmethodology that assists in phenomenon exploration ina certain context via various data sources. Besides, itembarks on the exploration of phenomena fromdifferent perspectives to uncover various aspects. In thecase study, the study is conducted in real-time within its
naturally occurring context, with the consideration that
context will influence the outcome [15].
According to [16], case study research is a qualitative
approach in which the investigator explores real-life,
recent cases over time involving detailed data collection
from numerous sources of information, and reports acase description. To present the complexity of the issue
being examined, various methodological approaches
will be used. A case study will include a case study
database, a clear evidence chain, and multiple sources ofdata from research. The depth and richness of the casestudy description allow readers to have a better
understanding of the case and to determine theapplicability of findings beyond the setting. The
evidence sources for case studies include questionnaires,interviews,
documentation, direct
observations,participant observations, archival records etc. Therefore,case studies involve trustworthy information and act asa reliable research method since case study research isan in-depth and intensive research methodology [15].

Fig. 1 Results from the questionnaire
RPA is the automation technology that strives toautomate human tasks including business processes byutilizing software robots that interact with systems
through their user interface. It can help to increase
efficiency, decrease costs and efforts that humans spend
on repetitive tasks [17]. One of the advantages of RPA
is that it can automate numerous processes smoothly
and decrease operational costs significantly due to it canhandle repetitive tasks and time-saving as well as
resource-saving. Besides that, prior programming
knowledge is not a must to set up and implement RPA,
leading to accessibility to employees who are without
any programming knowledge and experience. Moreover,
RPA supports most of the regular business processes
with error-free automation and reduces the need forhuman intervention. Furthermore, software tools arescalable and do not get exhausted [2]. RPA can handle
tedious and redundant tasks, thus employees canconcentrate on more complicated and valuable tasks that
contribute more value to the company. The employees©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
537

The Smart Document Processing

with more meaningful tasks can invest more time todevelop new skills to be more qualified for their job[18].
On the other hand, RPA might be limited when it comesto reading unstructured documents with various layouts
and formats. The documents need to be in the sameformat for software bots to read accurately. In addition,even minor changes in the automation application also
require reconfiguration of the RPA software bot [2].
Moreover, it can be disruptive and less effective if
without the appropriate tools although it seems like a
good idea for a company to shift to a more advanced
business system [19].
Even though it has been proven that a competitive
advantage can be achieved with the use of advanced
analytics and ML, enterprises nowadays are still facing
significant challenges in the adoption of ML due to lackof clear strategy for AI, insufficient time and limited
data available. Soft aspects which include leadership,decision making, and company culture are also some of
the challenges which affect the adoption of ML [4].
It is possible to achieve adoption through the creation ofa tangible business value. However, as part of productor process innovation, the main challenge is to find theright purpose and to deliver the technology in acompliant manner. In these circumstances, for thedevelopment of business value it is imperative to havedeep experience in processes and a good understandingof commercial requirements. Efficiency can be achievedby automation and an increased pace of implementationof business processes, saving costs at the same time.
Effectiveness refers to higher business process quality
which consists of fewer errors and compliant process
execution [4].
ML is expected to address the clear business needs andintegrate them into the business process so that it candeliver efficiency, preferably in the form of an end-toend automation, simplifying the access of business users
to technology. In this respect, 'everyday AI' which iscapable of functioning without long data preprocessing
and training activities is expected. In order to achieve
this simplification, it is possible to integrate thetechnology into the process where the data is stored andimprove the user experience [4].
In addition, access to data and its quality are two of the
key elements that contribute to the success of ML.
Firstly, it is necessary to consider the aspects of data
gathering, access and transfer, anonymisation,
annotation, curation, secure storage and deletion.
Nevertheless, strict observance is required in the areasof legal frameworks and general regulations. When
more data is created or collected, AI must be trained andevolved over time. The algorithms are expected to learn
directly from user behavior, which is then used tointernalize their knowledge. ML is currently still

considered as an assistance system which coexists withhumans, and processes the data provided by humans.
This is expected to happen automatically, in a way thathuman correction is used to improve algorithms and that
this knowledge is not forgotten by the machine. Thiscan be described as continuous learning. As a result,users will be able to anticipate increasing accuracy overtime, build trust and depend on the results.
Furthermore, culture also plays an essential role in the
introduction of machine learning and therefore needs tobe adequately climate governed by innovation principles,resources, time, etc. The leadership's mindset andemployees' willingness to accept change are the main
reasons for this. It will have to be derived from andintegrated with the proper strategy. It seems thatenterprises with pragmatic and hands on attitudes tend
to be more agile. They are aware of business needs,
have well defined use cases, and provide the necessary
environment for implementation such as fewerorganizational obstacles, empowerment of staff, no fearof losing control or jobs. In fact, there seems to be areluctance and much slower adoption by customers who
are deeply entrenched in bureaucracy and hierarchies ofmanagers without an explicit innovation culture [4].
5. Conclusion
The study focused on the potential and challenges ofInformation Document Processing with Artificial
Intelligence techniques to manage unstructured data or
documents. The prior study conducted by the
researchers on IDP with AI techniques indicated that
there are numerous benefits and challenges for AI-based
techniques, but it is still a very popular and trending
technique that most companies are adapting andutilising. According to the findings of this study, AI
techniques such as RPA and ML have various benefits
to companies, such as increased efficiency, reduced
costs, and support for error-free automation of businessprocesses. In addition, they are also facing somechallenges, such as a lack of AI strategy, insufficientdata, and culture, in the adoption process for the IDP
with AI techniques. However, it can be overcome
through value creation, integration, data considerations,and fostering an innovation-driven culture. IDP with AI
techniques has been growing and increasingly importantin each industry. Therefore, it is vital for companies toemphasise, adopt and utilise the current technologies toimprove their business performance.
References
1.

Sarker, I. H. (2021). Machine learning: Algorithms, realworld applications and research directions. SN
computerscience, 2(3),160.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
538

Raenu Kolandaisamy, Heshalini Rajagopal, Indraah Kolandaisamy, Glaret Shirley Sinnappan

https://link.springer.com/article/10.1007/s42979-02100592-x
2. Baviskar, D., Ahirrao, S., Potdar, V., & Kotecha, K.
(2021). Efficient automated processing of the
unstructured documents using artificial intelligence: A
systematic literature review and future
directions.
IEEE
Access, 9,72894-72936.
https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumb
er=9402739
3. Sukrutha, K. S., Harini, S., & Kusuma, M. V. (2023). A
Study on Intelligent Document Processing Using AWS.
International Journal for Multidisciplinary Research(IJFMR),5(4),1-5.
https://www.ijfmr.com/papers/2023/4/4308.pdf
4. Janasz, T., Mortensen, P., Reisswig, C., Weller, T.,
Herrmann, M., Crnoja, I., & Höhne, J. (2021).
Advancements in ML-enabled intelligent document
processing and how to overcome adoption challenges
inenterprises.
Die
Unternehmung,75(3),
340-358.
https://www.nomos-elibrary.de/10.5771/0042-059X2021-3-340/advancements-in-mlenabled-intelligentdocument-processing-and-how-to-overcome-adoptionchallenges-inenterprises-jahrgang-75-2021-heft3?page=1
5. Cutting, G. A., & Cutting-Decelle, A. F. (2021).
Intelligent Document Processing--Methods and Tools in
the real world. arXiv preprint arXiv:2112.14070.
https://arxiv.org/ftp/arxiv/papers/2112/2112.14070.pdf
6. Sambetbayeva, M., Kuspanova, I., Yerimbetova, A.,
Serikbayeva, S., & Bauyrzhanova, S. (2022).
Development of Intelligent Electronic Document
Management System Model Based on Machine Learning
Methods. Eastern-European Journal of EnterpriseTechnologies,
1(2),115.https://www.researchgate.net/publication/359049293_Development_of_intelligent_electr
onic_document_management_system_model_based_on_
machine_learning_methods
7. Adnan, K., & Akbar, R. (2019). Limitations of
information extraction methods and techniques forheterogeneous unstructured big data. International
Journal of Engineering Business Management,
11,
1847979019890771.https://journals.sagepub.com/doi/epu
b/10.1177/1847979019890771
8. Fernando, L. (2023). Intelligent Document Processing: A
Guide for Building RPA Solutions (Edition 2023.1.).
Notion Press.
9. Ribeiro, J., Lima, R., Eckhardt, T., & Paiva, S. (2021).
Robotic process automation and artificial intelligence inindustry 4.0–a literature review. Procedia Computer
Science,
181,
51-58.
https://www.sciencedirect.com/science/article/pii/S18770
50921001393
10. Ling, X., Gao, M., & Wang, D. (2020). Intelligent
document processing based on RPA and machinelearning. In 2020 Chinese Automation Congress (CAC)(pp. 1349-1353).
11. Anandika, A., & Mishra, S. P. (2019). A study onmachine learning approaches for named entity
recognition. In 2019 International Conference on Applied

12.

13.

14.

15.

16.

17.

18.

19.

Machine Learning (ICAML) (pp. 153-159). IEEE.
https://ieeexplore.ieee.org/document/8989189
Macri, C. Z., Teoh, S. C., Bacchi, S., Tan, I., Casson, R.,
Sun, M. T., ... & Chan, W. (2023). A case study inapplying artificial intelligence-based named entity
recognition to develop an automated ophthalmic disease
registry. Graefe's Archive for Clinical and Experimental
Ophthalmology,
1-10.
https://link.springer.com/article/10.1007/s00417-02306190-2
Paneru, S., & Jeelani, I. (2021). Computer visionapplications in construction: Current state, opportunities& challenges. Automation in Construction, 132, 103940.
https://www.sciencedirect.com/science/article/pii/S09265
80521003915
Bose, P., Srinivasan, S., Sleeman IV, W. C., Palta, J.,
Kapoor, R., & Ghosh, P. (2021). A survey on recent
named entity recognition and relationship extraction
techniques on clinical texts. Applied Sciences, 11(18),8319. https://www.mdpi.com/2076-3417/11/18/8319
Rashid, Y., Rashid, A., Warraich, M. A., Sabir, S. S., &Waseem, A. (2019). Case study method: A step-by-stepguide for business researchers. International journal of
qualitative methods, 18, 1609406919862424.
https://journals.sagepub.com/doi/full/10.1177/160940691
9862424
Alpi, K. M., & Evans, J. J. (2019). Distinguishing casestudy as a research method from case reports as apublication type. Journal of the Medical Library
Association:
JMLA,
107(1),1.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6300237/
Santos, F., Pereira, R. and Vasconcelos, J.B. (2020),
Toward robotic process automation implementation: an
end-to-end perspective, Business Process Management
Journal, 26 (2), 405-420. https://doi.org/10.1108/BPMJ12-2018-0380
Costa, D., São Mamede, H., & Silva, M. M. D. (2022).
Robotic Process Automation (RPA) adoption: asystematic literature review. Engineering Management in
Production
andServices,
14(2),
1-12.
https://repositorioaberto.uab.pt/handle/10400.2/14058
Ansari, W. A., Diya, P., Patil, S., & Patil, S. (2019). A
review on robotic process automation-the future ofbusiness organizations. In 2nd International conferenceon advances in science & technology (ICAST).
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=337
2171

Authors Introduction
Dr. Raenu Kolandaisamy
He received his PhD from the Faculty ofComputer Science & Information
Technology, University Malaya in 2020.
He is currently an Assistant Professor in
UCSI University, Malaysia. His researchinterest areas are Wireless Networking,
Security, VANET and IoT.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
539

Powered by TCPDF (www.tcpdf.org)

The Smart Document Processing

Dr. Heshalini Rajagopal
Sh She received her PhD and Master's degreefrom the Department of ElectricalEngineering, University of Malaya,
Malaysia in 2021 and 2016, respectively.
She received the B.E (Electrical) in 2013
Currently, she is an Assistant Professor in
UCSI University, Kuala Lumpur,
Malaysia. Her research interest includes image
processing, artificial intelligence and machine learning.
Dr. Indraah Kolandaisamy
She is a senior lecturer in School ofBusiness Management, College ofBusiness, Universiti Utara Malaysia.
Indraah A/P Kolandaisamy holds a
doctorateinManagement
from
Universiti Kebangsaan Malaysia in
2015. Her D.B.A work is onorganizational citizenship behavior among public sector
in Malaysia. She obtained her MSc (Management) andBachelor in International Business Management (Hons)from Universiti Utara Malaysia respectively in 2007 and2005.
Dr. Glaret Shirley Sinnappan
She holds the position of AssistantProfessor in the Department ofInformation
andCommunication
Technology at Tunku Abdul Rahman
University College.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
540

Digital Security Challenges Faced by Business Organizations
Raenu Kolandaisamy, Heshalini Rajagopal

Institute of Computer Science and Digital Innovation, UCSI University, 56000 Kuala Lumpur, Malaysia
Indraah Kolandaisamy

School of Business Management, University Utara Malaysia 06010 Sintok, Kedah
Glaret Shirley Sinnappan

Tunku Abdul Rahman University of Management and Technology, Kuala Lumpur, Malaysia
E-mail: raenu@ucsiuniversity.edu.my

Abstract
Over the decades, the forms of cyber-attacks have evolved from disruption level, cybercriminal, followed by cyberespionage and lastly threatening level. Digital security has played a significant role in protecting enterprises fromany form of cyber-attacks, especially in today’s era of digitalization. Aligning with the global effort in emerging theconcept of Industrial Revolution 4.0 (IR4.0) in organizations, where sensitive data and confidential information canbe accessed at a fingertip of an employee. This paper discusses the difficulties of implementing digital security
solutions in an enterprise in terms of external potential cyber threats, internal cyber security roadblocks within theorganization and how Covid-19 pandemic imposed challenges towards cyber security in the organizations.
Keywords: digital security, cyber security, cyber-attacks, cybercriminal, cyber espionage, business organizations

analyse it in real time by using advanced big data
analytics for generating useful information of
integrating services, products and processes, andmaking decisive influence on existing business models.
Meanwhile, approximately 90% of business managers
in developed countries like the U.K. and U.S. forecast
information technology and digital technologies
contribute strategic values to the entire economy in the
future. On the other aspect, information technologies
will uncover a variety of new threats of cyber security
challenges and risks when more companies foster
innovation by increasingly popularizing new digitaltechnologies, while security information technologies
are changing and evolving to become a complicatedthreat (cyber security). Furthermore, information
security cases might have caused a minor technicalproblem 10 or 15 years ago, but today advanced andintentional cyber attacks are challenging us that mightcause large-scale events as a result of direct and indirectconsequences affecting business’s strategy andcompetitive advantage [2].
Browsing the internet, reading digital books, social
networking, searching content, and online shopping aretransforming our lives to better, efficient and productive

1. Introduction
The Information System has changed our lifestyle overthe past decades among banking systems, education,transportation, business, tourism, shopping. The rise ofinternet consumption has dramatically increased byorganizations and individuals with massive global
international trade and changes of human interaction in
the young generation. Digital economy can refer tointegration of new business models, goods, services andmarkets established on a basic business infrastructurewith digital technologies. Digital technologies can be
defined as consolidation of communication, computing,information, and connectivity technologies that are
driven by storage, software applications, sensors andbandwidth to the next generation of digital economy
through cloud computing [1].
The rapid development of science and technology is far
more than that such as the internet of things (IoT),
artificial Intelligence (AI), augmented reality (AR),drones, additive manufacturing (3D printing), robotics
and others. These latest technologies work fromextracting data from sensor conditions of physicaldevices, spreading and storing it rapidly on cloud to©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
541

Raenu Kolandaisamy, Heshalini Rajagopal, Indraah Kolandaisamy, Glaret Shirley Sinnappan

ones. However, most of the netizens and organizations
lack cyber security awareness and relevant knowledgeof internet hazards. Individuals and businessorganisations have limited knowledge to protect theirdevices and servers from cyber hazards such as financial
fraud, sensitive information theft, website spoofing,
Internet of Things (IoT) hacking, malware, distributed
denial of service attack (DDoS) as well as ransomware.
Cyber hazards come from the work of hackers also
known as “black hats” who committed cyber crime on
their own or with an organized criminal group. Largescale breach of cyber security has been rated by theWorld Economic Forum as one of the five mostvarious cyber hazards such as malware, phishing,
distributed denial-of-services attack (DDoS), man-inthe-middle attack (MMIT), ransomware and insider
threats [4]. Every cyber attackers’ intentions aredifferent, but mostly demanding for money, financial
information, personal information and even the business
infrastructure. Cyber attacks target individuals andbusiness organizations that have less cyber security
awareness and protection tools because they always
have key assets to be exploited by criminals. According
to “PricewaterhouseCoopers survey”, organizations that
undergo cyber attack relevant matter vanished 2.1% of
its value and lost more than 1.6 billion USD per case on
average [2]. Understanding the motive of cyberattackers and hazards is essential for internet users toprevent financial loss and irreversible consequences.
There are three aspects of motive to be proven as crime
which are motive, means, and opportunity that can betreated as a conventional crime. Means refer to an attack
tool or technical expertise to execute the attack, motiveis the aims, intention, purpose or reason of the attacker
selecting a target, opportunity can be seen as a timing,
vulnerability and weakness in the network of the target[5].
Ransomware is a kind of malicious software thatencrypts files of business organizations by demanding
ransom in exchange for a decryption key [6]. It is very
difficult to detect hijacked files and data before a cyberattacker initiates the attack. There are only 2 options forusers to solve the problem which are to pay ransom in
exchange for a decryption key to retrieve files and data
or disconnect the internet and format the computer [4].
However, it is nearly impossible for internet-based
business organizations to get rid of ransomware byformatting all data in digitized generation because it is
the foundation and infrastructure of the company. In
2012, Reveton is one of the recognised ransomware
trojans that ransom nearly $130,000 fines from victims
who believed that they have been charged forparticipating in illicit online activities. Businessorganizations weren't too worried about the attack
because it was affecting mostly individuals. By the endof 2016, an estimated $1 Billion was paid to the ransom
hackers from Cryptorlocker and CryptoWall. It gave ahuge profit trajectory to the hackers, something big
known windows exploits were on the way from the
National Security Agency (NSA). The worst and largest
ransomware attack that ever happened before in 2017,“WannaCry” (Fig. 1) ransomware attacked more than
230,000 computers of various industries which were
universities, government organization, hospital, bank,logistic, telecommunication companies, railways
companies and business organizations across 150
countries [7]. According to research from “SonicWall
Capture Labs”, ransomware cases have risen by 20%,

critical threats confronting the world today where thescope of threat globally is spreading rapidly and cost an
estimated US$6 trillion by 2021 [2]. Nevertheless,studies show that installing protective tools on thecomputers from factories does not maximize cybersecurity generally. Hence, this paper examines thechallenges faced while implementing cyber security
solutions into organizations’ systems in various
different perspectives.
2. Result and discussion
According to the Internet Crime Report publishedby the Cyber Division of the Federal Bureau of
Investigation (FBI) in the year of 2020, losses
exceeding USD 29 mil resulted from 2,474 ransomware
attacks registered worldwide has been recorded [3]. The
fact that cyber crimes against enterprises have increased
tremendously over the years has shown that the digital
security systems or practices within the organizations
still remained susceptible and vulnerable towards cyberattacks in any form or even worse, could not keep up
with the growing pace of these cyber threats. Thus, theroadblocks or difficulties to carry out theimplementation of the cyber protection have to be
discussed thoroughly in terms of external cyber hazards,
internal cyber security challenges in organization andchallenges of digital security during Covid-19 pandemic
in order to improve, correct and make changes towards
the current situation of cyber security support in theright direction.

2.1. External Potential Cyber Hazards
The Internet has improved the standard of living of
IoT users, but every pillar has two sides. Everything canbe done smoothly by one click as the same goes withcybercrime. That could so easily invade valuable
information of individual and business organizations by©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
542

Digital Security Challenges Faced

However, it’s not effective because these technicalcountermeasures are unable to analyze and differentiate
between phishing and general email. Furthermore,business organizations provide phishing attack
awareness training to their employees against phishingtechniques, tactics, and procedures (TTPs) throughemail security guidelines and standard operatingprocedures to increase cyber awareness [9].

with up to 121 million attacks recorded around theworld in the first half of 2020 [8].
Working of ransomware (Fig. 2) will be introduced into
4 steps. First, the ransomware starts with an unsolicitedemail usually designed to trick the victim into clicking
on an attachment or visiting a webpage. Second,ransomware leverages flaws in the computer’scommand-and-control server and forces it to downloada public key for running ransomware code. Third,
important data is encrypted on the system and demands
a ransom payment using cryptocurrency by providing
ransom fee instructions. Lastly, the decryption key will
be given by the attacker once the ransom is paid [4].

Fig. 3 Phishing attack mechanism
Distributed Denial of service attack (DDoS) attack is
one of the most common cyber crimes faced by businessorganizations when a group of botnets, server or
numerous connected internet devices flood the capacity
limit and resources of a targeted system as a result inservice degradation and unavailability by disrupted
normal traffic. Attacker will control his resources tooverwhelm the capacity limit of the server by sending
numerous requests to the targeted IP address of thewebsite in a short time that trigger a crash. DDoS
attacks could stay as long as 24 hours and up to days toretain access to the unavailability of the company's
server that lead to reputation damage and profit
minimization. There are vulnerabilities in the systemthat are exploited by the hackers although various
protections and prevention are applied.

Fig. 1 Activation of “WannaCry” on victim’s
computer

DDoS attacks (Fig. 4) are conducted with private
networks of internet-connected machines and any kinds
of IoT devices. Attackers can remotely control the
networks that are constituted by computers and devices
that are malware infected. These infected devices can becalled bots (or zombies), and a botnet where a group of
bots is connected in a network. Attackers are allowed tosend attack instructions to each bots when the botnet has
been established. When each bot sends requests to thevictim's IP address, possibly causing the server of thewebsite to be down where limit traffic is triggered
during a short period as a result of denial-of-service tonormal traffic. Because the server is difficult to identify
attack traffic and normal traffic from each bot is
considered a legitimate internet device [10].

Fig. 2 Ransomware mechanism

Phishing attacks (Fig. 3) is a social engineering
technique to bypass computer security and humandefence by using social networks, emails, mobile apps,
instant message, business email compromising or searchengine poisoning to steal user’s data, sensitive
information, credit card number and code, as well as
login credentials to make the victim perform an action.
There are more than 30% of cybercrime caused byphishing attacks and cost at least 8 billion USD
according to Verizon Data Breach Investigation (DBIR).
There are some technical countermeasures to mitigate
risk of phishing such as anti-phishing, email malware
detection, data loss prevention and spamming tools.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
543

Raenu Kolandaisamy, Heshalini Rajagopal, Indraah Kolandaisamy, Glaret Shirley Sinnappan

storage without permission and awareness. CCA could
be conducted by various methods such as structured
query language (SQL) injection attack, DDoS attack,malicious insider, side channel attack, man-in-themiddle (MITM) attack and abuse of cloud services.
Cloud computing attack is the most economicallysignificant threat in the world of virtualization
technology, confidential data breach is one of the key
diminish potential revenue loss and customer trust [11].
In fact, a massive cyber attack in Sweden in July 2017,a high confidential data stored at cloud system wasbreached, causing irreversible consequences that
exposed national secrets, national security, international
wrongdoing, and resignation of ministers. In fact, a
massive cyber attack in Sweden in July 2017, a highconfidential data stored at cloud system was breached,causing irreversible consequences that exposed nationalsecrets, national security, international wrongdoing, andresignation of ministers. In June 2017, British Airways
cloud system was being disrupted by hackers as a resultof financial loss at 114 million EUR due to over 100
flights being cancelled from London airports [2].
SQL injection (Fig. 5) attacks is a common attack wherean attacker inserts malicious code to unauthorizedaccess to a backend database to gain sensitive
information that was not intended to be disclosed or
private company data. Generally, SQL injection attacks
require an attacker to figure out the vulnerability of a
webpage or web application to perform unauthorizedaccess without an official account. For example, alogical user’s account coupled with usernames andpasswords that are registered at the database, attacker
could input malicious code like username = '1' or 2>1 -' and password = '111' to bypass source code of the web
application [14].

Fig. 4 DDos attack mechanism

Types of DDos Attack• Ping of Death (POD) attack
POD
attackexploits
theparticularcharacteristic based on the maximum size ofthe packet of transmission control protocol(TCP)/ internet protocol (IP) can be up to65,535 bytes in normal cases. Generally, in thecase of a large IP packet, it will be splitted into
numerous fragments and reassembled into acomplete packet to the recipient host. However,
POD attack exactly exploits the reassembling
packet size of 65,535 bytes greater than its
maximum to maliciously manipulate overflow
of memory allocated as a result of denial ofservice[11].
• Hypertext Transfer Protocol (HTTP) Flood
AttackHTTP flood attack is another method ofresource consuming by manipulating the HTTP
GET and HTTP POST requests whileconnecting with the victim’s machine [12].
HTTP works as a request-response protocol
and designs to allow communications between
servers and clients. HTTP GET and POST are
the two most common methods where GET
requests data from a particular resource and
POST sends data to a server to update or createa resource [13].
• Domain name system (DNS) Flood Attack
DNS is the “contact list” of the internet whichallows devices able to view the content of
websites. It sends a huge amount of DNS
requests to the target device in order tooverload it and reduce the speed of traffic.
Attack will be done by the botnet to generate alarge volume of traffic [12].
Cloud computing attack (CCA) is a malicious activity
that involves data breaches, manipulating, accounthijacking or eavesdropping from a user's virtual
machine on the physical server by injecting malware
from the hacker. Once the cloud system is deceived,
hacker can gain unauthorized access to the user's cloudFig. 5 SQL Injection mechanism
MITM (Fig. 6) occurs when an attacker establishes
independent connection with separate ongoing
conversations to different third parties by misleadingthem with the aim to inject, steal or exchange
information to manipulate the result. Third parties are
not aware that the attacker has control over the ongoing
conversation between both parties. All of the message
sent by one party to another will go through the attacker
and it allows the attacker to tamper the information tomislead both parties [14].

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
544

Digital Security Challenges Faced

ecosystem of the organization [17]. Thus, with such
irresponsible behaviour, the employees tend to put
lesser attention onto the data handling which leads tothe ignorance attitude towards the basic cyber security
practices and possibly violating the compliances ofcyber security. Not only that, employees’ careless work
behaviour such as inappropriate sharing confidential
information with third parties without permission,
inappropriate use of IT resources, losing their own
devices storing company data, etc. could lead to asignificant financial loss to the organization [18]. This
could be evident when the result from the IT Security
Risks Survey 2017 has indicated that almost 11% of thecyber attack incidents are due to careless anduninformed employees as shown in Fig. 7 below [18].
Hence, proven that the irresponsible act of theemployees could definitely lead to hard time for theorganization to implement the cybersecurity measures.
Fig.7 Most serious cyber attack vectors over the 12

Fig. 6 MITM mechanism
2.2. Internal Cyber Security Challenges within
the Organization
Other than external cyber security challenges in theform of various cyber threats, there are seven internal
challenges in implementing or improving the cybersecurity in an organization where these challenges could
be grouped into three different categories or foundations
for the organizations which are the three pillars ofPeople, Process and Technology [6]. Indeed, humanresources or employees are important assets to theorganization, however, also one of the internal factorsthat drive the organization to the risks of exposing themto the emerging and evolving cyber threats when theemployees
arelacking
cybersecurity
skills,demonstrating irresponsibility and conducting humanerror while dealing with sensitive data and processes [6].
Firstly, the lacking of cybersecurity skills is said to be
one of the challenges of cybersecurity in an organizationand getting worse four years in a row since 2017 where
this crisis has impacted almost 70% of the organizations
globally according to the fourth annual global study byboth the cybersecurity professionals, the Information
System Security Association (ISSA) and independent
industry analyst firm, the Enterprise Strategy Group(ESG) [15]. In fact, there are numerous factors that leadto such cybersecurity expert shortage crisis which could
impose significant cybersecurity risk to theorganizations, they are mainly due to global worker
shortages, limited hands on skills as well as experiencein handling cyber attacks and threats, the increasing
workload of employee resulting in lesser time to acquire
the relevant skill set, the incompetence of the employees
to learn or utilize the cybersecurity technologies to thefullest potential, etc [16]. Thus, it is clearly evident thatlack of skills in cybersecurity is one of the mostchallenging concerns in implementing cybersecurity
measures under the pillar of ‘People’ within theorganizations. Besides cybersecurity talent shortage, theirresponsible behaviour of the employees towards
cybersecurity is also one of the challenges faced by theorganizations in implementing the relevant measures.
Most of the employees in an organization would intend
to push the responsibility of the IT staff only, whereby
the cybersecurity implementation is not solely IT
departments responsibility, but all employees fromoperational level to management level within themonths in 2017.
Furthermore, human error has also contributed as an
obstacle towards the implementation of digital security
in an organization [6]. This could be proven from asurvey conducted by IBM in 2018, indicating that the
misconfiguration of cloud servers due to human error isthe biggest risk in cyber security in cloud computing
with 62% of survey participants made up of IT andsecurity professionals, while 55% of them noting misuseof employee credentials and improper access as one of
the human error contributing to challenges while
implementing cyber security measures, followed by
insecure interfaces at 50% [19]. In fact,
misconfiguration of systems usually allows cybercriminals to access large data platforms consisting ofusernames, passwords, credit card data, health data,
national identification numbers, email addresses, etc.
which could possibly be used to request for ransom
from the organizations. For example, one of thesignificant cyber attack incidents happened due tomisconfiguration of system and human error in 2018,
where a well-known marketing firm in the US, Exactis
was reported leaking approximately 340 million files tothe public including sensitive personal details ranged©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
545

Raenu Kolandaisamy, Heshalini Rajagopal, Indraah Kolandaisamy, Glaret Shirley Sinnappan

from national identification number to hobbsies bymaking them available through a server accessible toanyone [20]. This incident has definitely shown thathuman error within an organization especially involving
data handling and system configuration will definitely
lead to big loss to the organizations regardless offinancial or reputation.
The processes, which is the second pillar of anorganization, has played an important role incybersecurity implementation. There are three cybersecurity challenges identified under the pillar of‘Process’ in a recent research, which include the lack ofimplementation plans for cyber security measures
execution, the misplaced human resources and the lackof budget allocated for cyber security efforts within theorganization [6]. The readily compliance andregulations governed nationally such as cyber security
guidelines used in Malaysia including Information
Technology Instructions 2007, ICT Security Policy
2010, Electronic Government Act 1987, Public Sector
Cyber Security Framework 2016, etc., would need agood implementation plan within the organization inorder to apply the guidelines into processes and enforce
the cyber security strength of the organization [6].
However, most of the organizations do not emphasize
cyber security planning which ultimately leads to ahigher risk of monetary loss and reputational damage[21]. The mentioned phenomenon could be proven by
the results of the survey conducted by The Department
for Digital, Culture, Media & Sport (DCMS) in the UK
on 1,500 businesses from October to December 2017,
where only 38% of the businesses is aware of the newincoming data protection law, while only 27% fromthese businesses have made the changes on theorganization processes and less than half of thesebusinesses made changes to the cyber security practices
within the organization, leading to a total of 43% of
these businesses in the UK experienced cyber threats in2017 [22]. The lack of implementation plan could
involve cyber security governance, protection measures,
hardware maintenance plan, incident response plan,evaluation plan, documentation including policies,
Standard of Procedures (SoP), Work Instruction (WI),
etc. where the effectiveness of the cyber security
implementation is directly affected by the quality ofthese elements [21]. Therefore, detailed construction of
the implementation plan is indeed important to ensure
the cyber security measures are being implemented andexecuted accordingly in order to reduce the risks of
exposing the cyber physical asset of the organization
towards potential cyber hazards.
The next digital security challenge in an organization
under the pillar of ‘Process’ is poor human resource
management. For instance, new arrangements for cybersecurity talents in the workplace due to promotion to

managerial positions, assignments to new departments
or even the failure to retain cyber security talent would
lead to the reduction of trained and experienced talentsthat could counteract cyber threats in an organization [6].
Due to talent loss, more time would need to be
consumed to acquire capable talents, to train and nurture
the replacement, etc which would cost a knowledge gap
in terms of cyber security in the organization, leading to
the risks of cyber invasion [23]. In fact, a survey
conducted on cyber resilience by IBM Security and thePonemon Institute has shown that 75% of the survey
respondents found it difficult to hire and retain skilledcyber security professionals in respective organizations,while 48% of them has also pointed out the complexity
in operation due to various type of security tools being
implemented, causing the weakening in cyber resilience
due to the skill and abilities gap between existing talents
and tools deployed within the organization [24].
The sixth challenge in digital security within an
organization is the limited budget allocated for the
implementation and enforcement [6]. A detailed cybersecurity budget allocation should include risk analysis
cost, technology acquisition or software licensing costs,
training charges, insider threat reduction cost, etc. toreduce the potential risk of cyber internal breach andexternal threats [25]. However, most of the top
management in the organizations especially small andmedium enterprises do not prioritize cyber security
because they would be willing to maximize theprofitability instead of spending on cyber protection
measures for incidents that may not happen, but they did
not consider the possibility of cyber security failures
which would cost the organizations even more [26]. For
example, RiskIQ estimated financial loss of $17,700 per
minute will be imposed on the victims due to phishingattack while Accenture estimated Malware rates up to$2.6 million at most and Ransomware rates up to$646,000 on average [27]. Not only that, organizations
with improper cyber security could also be fined as it
would induce insecurity within users, for instance,
Google was fined at a cost of $57 million in 2019 due tothe failure in complying with General Data Protection
Regulation (GDPR) in France [28]. Thus, limited
budget allocation as a roadblock in cyber security of anorganization would definitely cause larger amounts offinancial and reputational losses in the future as therisks from organizations internally or cyber threats
externally are not being mitigated sufficiently.
Lastly, business organizations are facing cyber security
challenges when the technology advancement isevolving too fast, so does the advancement in cyberthreat vectors [6]. In fact, the cyber threats arecharacterized as asymmetrical and multi directional,
making the organizations being reactive instead ofproactive while facing the new form of cyber attacks[6].

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
546

Digital Security Challenges Faced

Also, as the world is currently accelerating towards
IR4.0, the organization especially in manufacturingindustry which adopting technologies such as Robotics
and Automation (R&A), Artificial Intelligence (AI) andMachine Learning (ML), Big Data Analysis (BDA) andCloud Computing (CC), etc, where these technologies
adapted are actually associated with the risks of being
attacked by cyber criminals [29]. For example, Booz
Allen Hamilton has pointed out the largest potential riskof AI and ML algorithms that worry the most is thepossibility of leaking the training data to cyber criminals
as this training data is sufficient enough to rebuild theoriginal AI model via reverse engineering [30]. Thus, it
is clearly shown that the benefits from ever evolving
technologies aligning with IR4.0 are always associatedwith greater risk of cyber hazards invasion.

vulnerability and risk analysis tests within theorganization to access the sustainment of the cybersecurity governance. Moreover, the policies, SOP and
WI related to cyber security measures and practices
should be comprehensive enough and made available toevery employee to ensure these guidelines or documents
are clear and easy to understand to avoid
miscommunication as well as guiding the employees intheir decision making process when facing issues
like cyber attacks attempts. Vendor management is also
one of the important processes to be focused onregarding cyber security because the vendors may
potentially serve as a medium for external cyber threats
invading the organizations, thus, it is important toensure the vendors engaged are equipped with the samelevel of security. [32]Recommendations are proposed in terms of the threemain pillars which are people, process and technology,
in order to help overcome the internal challenges facedby the organizations while implementing cyber security
measures.

Lastly, in the aspect of technological advancements,organizations are suggested to adopt AI-based cybersecurity management systems where the systems allow
to leverage up-to-date global cyber security knowledge
and industry specific threats to make significant
prioritization decisions as well as prompt incident
response when exposing to cyber threat, keeping trackof all organizations’ IT assets in a domain to perform
breach risk prediction and generating comprehensive
reports on recommendations and analysis of thedecisions proposed to all the involved stakeholders. [33]Firstly, human resources (HR) directors should play a
key role or even a leader to keep the organizations safe
from internal cyber breaches and employee-imposedcyber threat risks. As a HR director, a HR team should
be formed to work with the IT or cyber security
department to align the expectations ensuring the digital
security measures and policies to be implemented ispractical, ethical and appropriate. Besides, the HR
director and the team should also take ownership of the
security risk posed by employees by providing
sufficient education regarding the impact of theemployees behaviours towards the cyber security of theorganizations as well as identifying employees who may
be insiders allowing risks of external cyber threats to be
imposed on the company. Overall, the HR team plays avital role in shaping the workplace culture where theemployees are following the reviewed cyber security
measures and policies to help reduce the risks of cyberattacks. [31]3. Conclusion
In this paper, cyber security challenges faced by thebusiness organizations have been discussed andanalyzed in two different perspectives including theexternal potential cyber threats and internal factors
within the organizations. Notable external cyber hazards
such as the Ransomware, Phishing Attacks, DDoS
Attacks as well as Cloud Computing Attacks comprises
SQL Injection Attacks and Man-in-the-middle havebeen introduced and their mechanism also have been
discussed in this paper. Seven internal cyber security
challenges within the organizations have also beenhighlighted in terms of three different elements which
are people, process and technology, where by thechallenges discussed include the lack of cyber security
skills, irresponsible behaviour of employees, humanerror, lack of digital security implementation plan, poor
human resource management, constraint of budget
allocation for cyber security implementation and lastly
technology advancement. Indeed, both the internal andexternal cyber security obstacles faced by theorganization have allowed many digital security
breaches and invasions to happen, which leads tocritical negative impacts to the organizations, corporate

Secondly, in terms of processes, the management
systems within the organization are vital in combating
cyber security breaches. A proper management system
must be adopted to ensure all the employees are awareof their duties and responsibilities as well as toadequately manage the level of competence and interestof the employees in cyber security. Besides that, a seriesof enterprise cyber security governance activities should
be conducted from time to time to continuously improve
the strength of the cyber security, where the activities
include adopting enterprise risk management (ERM) to
prevent data loss, and also, conducting threat,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
547

Raenu Kolandaisamy, Heshalini Rajagopal, Indraah Kolandaisamy, Glaret Shirley Sinnappan

11. G. Levitin, L. Xing, and H. Huang, “Security of
Separated Data in Cloud Systems with Competing Attack
Detection and Data Theft Processes, Risk Analysis, vol.
39, no. 4, pp. 846–858, Oct. 2018.
12. L. Visalatchi, P.Yazhini, and M. Scholar, The survey
DDoS attack prevention and defense technique,International Journal of Innovative Science and ResearchTechnology, vol. 5, no. 2, pp. 65-68, Feb. 2020.
13. HTTP
Methods GET
vsPOST,
W3schools.com,
Accessed:
Aug.
28,
2021.
[Online]. Available:
https://www.w3schools.com/tags/ref_httpmethods.asp
14. N. Amara, H. Zhiqui, and A. Ali, Cloud computing
security threats and attacks with their mitigation
techniques, 2017 International Conference on CyberEnabled Distributed Computing and Knowledge
Discovery (CyberC), pp. 244-251, Oct. 2017.
15. Help Net Security, Cybersecurity skills shortage still theroot cause of rising security incidents, Accessed: Aug. 28,
2021.
[Online].
Available:
https://www.helpnetsecurity.com/2019/05/14/cybersecuri
ty-skills-shortage-causes-security-incidents/
16. J. Oltsik, ESG Research Report - The Life and Times ofCybersecurity Professionals 2020, ISSA, Accessed: Aug.
28, 2021. [Online].
Available:
https://2ll3s9303aos3ya6kr1rrsd7-wpengine.netdnassl.com/wp-content/uploads/2020/07/ESG-ISSAResearch-Report-Cybersecurity-Professionals-Jul2020.pdf
17. Cybersecurity regained: preparing to face cyber attacks,
Ernst and Young (E&Y). Accessed: Aug. 28, 2021.
[Online].
Available:
https://pdf4pro.com/view/cybersecurity-regainedpreparing-to-face-cyber-attacks-ey-559d48.html
18. “The Human Factor in IT Security: How Employees are
Making Businesses Vulnerable from Within,” Kaspersky
Daily. Accessed:Aug. 28, 2021. [Online]. Available:
https://www.kaspersky.com/blog/the-human-factor-in-itsecurity/
19. Kolandaisamy, R., Noor, R. M., Zaba, M. R., Ahmedy, I.,& Kolandaisamy, I. (2019, July). Markov chain based ant
colony approach for mitigating DDoS attacks using
integrated vehicle mode analysis in VANET. In 2019
IEEE 1st International Conference on Energy, Systems
and Information Processing (ICESIP) (pp. 1-5). IEEE.
20. T. Collins, Exactis leaks the private details of 340
MILLION people to cyber criminals, including theirphone numbers and home addressees, in a one of thebiggest security breaches of its kind, Mail Online,
Accessed: Aug. 28, 2021. [Online]. Available:
https://www.dailymail.co.uk/sciencetech/article5900071/Marketing-firm-Exactis-leaks-340-million-filescontaining-private-data.html
21. S. Ursillo and J. C. Arnold, Cybersecurity Is Critical for
all Organizations – Large and Small, IFAC, Accessed:
Aug. 28, 2021. [Online]. Available:
https://www.ifac.org/knowledge-gateway/preparingfuture-ready-professionals/discussion/cybersecuritycritical-all-organizations-larg e-and-small
22. The Department of Digital, Culture, Media & Sport(DCMS), “New figures show large numbers ofand national such as breach of trust, data, privacy,financial or economical losses as well as the damage of
reputation and public image. Therefore, further broadresearches and detailed studies on cyber security
especially the method of combating cyber threats andenhancing the strength of cyber security inorganizations are recommended so that the readers
could be provided with useful insights and ideas as wellas emphasizing the importance of everyone’s role inensuring the success of implementing effective andefficient digital security measures in the organizations
to instill the supportive behaviours and foster the cybersecurity culture within the companies.
References
M. Zwilling, G. Klien, D. Lesjak, Ł. Wiechetek, F. Cetin,and H. N. Basim, Cyber Security Awareness, Knowledge
and Behavior: A Comparative Study, Journal ofComputer Information Systems, pp. 1–16, Feb. 2020.
2. M. Spremić, and A.Šimunic. Cyber security challenges indigital economy, Proceedings of the World Congress on
Engineering, vol.1, pp. 4-6, Jul. 2018.
3. Internet Crime Complaint Center (iC3) Internet Crime
Report 2020, Federal Bureau of Investigation. Accessed:
Aug.
26,
2021.
[Online].
Available:
https://www.ic3.gov/Media/PDF/AnnualReport/2020_IC
3Report.pdf
4. S. Mohurle and M. Patil, A brief study of wannacry
threat: Ransomware attack 2017, International Journal ofAdvanced Research in Computer Science, vol. 8, no. 5,
pp. 1938-1940, 2017.
5. A. Abhishta, M. Junger, R. Joosten, and L. J. M.
Nieuwenhuis, A Note on Analysing the Attacker Aims
Behind DDoS Attacks, Intelligent Distributed Computing
XIII, Springer International Publishing, pp. 255–265, Jan.
2020.
6. C. S. Teoh, A. Kamil Mahmood, and S. Dzazali, Cyber
Security Challenges in Organisations: A Case Study in
Malaysia, 2018 4th International Conference on
Computer and Information Sciences (ICCOINS), Aug.
2018
7. C. Adams, Learning the lessons of WannaCry, Computer
Fraud & Security, vol. 2018, no. 9, pp. 6–9, Sep. 2018.
8. B. Hellard, 121 million ransomware attacks recorded in
the first half of 2020, Itpro.co.uk, Accessed: Aug. 26,
2021.
[Online].
Available:
https://www.itpro.co.uk/security/ransomware/356567/121
2-million-ransomware-attacks-in-the-first-half-of-2020
9. H. Shahbaznezhad, F. Kolini, and M. Rashidirad,
Employees’ Behavior in Phishing Attacks: What
Individual, Organizational, and Technological Factors
Matter?, Journal of Computer Information Systems, pp.
1–12, Oct. 2020
10. M. Aamir and S. M. A. Zaidi, DDoS attack detection
with feature engineering and machine learning: theframework and performance evaluation,” Int. J. Inf.
Secur., vol. 18, no. 6, pp. 761–785, Apr. 2019.
1.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
548

Powered by TCPDF (www.tcpdf.org)

Digital Security Challenges Faced

23.

24.

25.

26.

27.

28.

29.

30.

31.

32.

33.

businesses and charities suffer at least
one
cyber
attackinthepastyear,”
Gov.uk,Accessed:
Aug.
28,
2021.
[Online].
Available:
https://www.gov.uk/government/news/new-figures-showlarge-numbers-of-businesses-and-charities-suffer-atleast-one-cyber-attack-i n-the-past-year
Cipher,
What
Causes Cyber
Security
Projects
to
Fail?,
Accessed: Aug. 28,
2021.
[Online]. Available:
https://cipher.com/blog/what-causes-cyber-securityprojects-to-fail/
Help Net Security, 77% of orgs lack a cybersecurity
incident response plan, Accessed: Aug. 28, 2021.
[Online].
Available:
https://www.helpnetsecurity.com/2019/04/12/cybersecuri
ty-incident-response-plan/
Kolandaisamy, R., Subaramaniam, K., & Jalil, A. B.
(2021, March). A Study on Comprehensive Risk Level
Analysis of IoT Attacks. In 2021 International
Conference on Artificial Intelligence and Smart Systems(ICAIS) (pp. 1391-1396). IEEE.
S. Kabanda, M. Tanner, and C. Kent, “Exploring SME
cybersecurity practices in developing countries,” Journalof Organizational Computing and Electronic Commerce,
vol. 28, no. 3, pp. 269–282, Jul. 2018.
J. Fruhlinger, “Top cybersecurity facts, figures andstatistics,” CSO Online, Accessed: Aug. 28, 2021.
[Online]. Available:
https://www.csoonline.com/article/3153707/topcybersecurity-facts-figures-and-statistics.html
R. Dillet, French data protection watchdog fines Google$57 million under the GDPR, Tech Crunch, Accessed:
Aug. 28, 2021. [Online]. Available:
https://techcrunch.com/2019/01/21/french-dataprotection-watchdog-fines-google-57-million-under-thegdpr/
Baytamouny, M., Kolandaisamy, R., & ALDharhani, G.
S. (2022, April). AI-based Home Security System with
Face Recognition. In 2022 6th International Conference
on Trends in Electronics and Informatics (ICOEI) (pp.
1038-1042). IEEE. [30]. M. Korolov, How secure areyour AI and machine learning projects?,CSO Online.
Accessed: Aug. 28, 2021. [Online]. Available:
https://www.csoonline.com/article/3434610/how-secureare-your-ai-and-machine-learning-projects.html
Alt, R. (2021). Digital transformation in the restaurant
industry: Current developments and implications. Journalof smart tourism, 1(1), 69-74..
Putting human resources at the heart of cyber security,
PA Consulting. Accessed: Aug. 28, 2021. [Online].
Available:
https://www.paconsulting.com/insights/putting-humanresources-at-the-heart-of-cyber-security/
The three-pillar approach to cyber security: processes are
crucial, Det Norske Veritas. Accessed: Aug. 28, 2021.
[Online]. Available: https://www.dnv.com/article/thethree-pillar-approach-to-cyber-security-processes-arecrucial-162890
Using
Artificial Intelligencein
Cybersecurity, Balbix. Accessed: Aug. 28, 2021.

[Online]. Available:
https://www.balbix.com/insights/artificial-intelligence-incybersecurity/

Authors Introduction
Dr. Raenu Kolandaisamy
He received his PhD from the Facultyof Computer Science & Information
Technology, University Malaya in
2020. He is currently an AssistantProfessor in UCSI University,
Malaysia. His research interest areas
are Wireless Networking, Security,
VANET and IoT.

Dr. Heshalini Rajagopal
She received her PhD and Master's
degree from the Department ofElectrical Engineering, University of
Malaya, Malaysia in 2021 and 2016,
respectively. Her research interest
includes image processing, artificial
intelligence and machine learning.

Dr. Indraah Kolandaisamy
Dr. Indraah A/P Kolandaisamy is asenior lecturer in School of Business
Management, College of Business,
Universiti Utara Malaysia.

Dr. Glaret Shirley Sinnappan
Dr. Dr. Glaret Shirley A/P
Sinnappan holds the position ofAssistant
Professor
intheDepartment of Information andCommunication Technology at
Tunku Abdul Rahman University
College.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
549

The Study on Perception on E-Waste Among the People
Raenu Kolandaisamy, Heshalini Rajagopal

Institute of Computer Science and Digital Innovation, UCSI University, 56000 Kuala Lumpur, Malaysia
Indraah Kolandaisamy

School of Business Management, University Utara Malaysia 06010 Sintok, Kedah
E-mail: raenu@ucsiuniversity.edu.my
Abstract
E-waste is piling up at an alarming rate, which results in our environment's pollution, while only a few are being
disposed of correctly. This research aims to discover people's awareness of e-waste and how to handle it. An online
survey was done among 75 respondents, who were asked about their general knowledge of e-waste. Most
respondents are aware of the effects of e-waste but lack knowledge on how to tackle them, including questions
concerning e-waste management and its causes. In conclusion, the general public must be educated on e-waste to
reduce and dispose of e-waste efficiently and effectively.
Keywords: E-waste, Technology, Online Survey, Environment

swift growth of technology usage. The government havebeen emphasizing Vision 2020 lately, which includes anobjective to establish a scientific and progressive
society that is not only a consumer of technology but
also a contributor [2]. Another thing to note is the term
Industry 4.0, which aims to create a smart factory
environment. Alternatively, technology is known to easeday-to-day tasks. Due to this, technology has infused
into our lives and become an ongoing trend. To make it
clear, if the five big tech companies are combined, theywould have a higher revenue (up to 802 billion)
compared to Saudi Arabia, which has the 19th highest
GDP at 684 billion [3].

1. Introduction
The 20th century was rich with pivotal technologicaladvancement, from the birth of the television to thecreation of personal computers. Technologicaladvancement kickstarted the tech industries towards
more innovations that are rapidly improving, which led
to inventions such as the iPhone that paved the way formodern-day smartphones. Technology these days has
become mainstream. Statistics show an uphill trend
regarding the adoption of technologies in a household.
For example, colour TV use increased exponentially,
recording a 10 per cent adoption in 1966 to 96 per centin 1990 and a fairly stable adoption rate moving towards
the 21st century, staying around 90 per cent. The
adoption of computers has also shared the sameprosperity, skyrocketing from 20.70 per cent in 1992 to
89.30 per cent in 2016. The biggest winner of all this is
the smartphone. Since its introduction around the 21st
century, its adoption has rapidly increased, from 35 percent in 2011 to 73 per cent in 2016. Its predecessor, thecellular phone, is the only factor stopping the adoptionof smartphones from growing further since it also has anincrease of adoption from 10 per cent in 1994 to 93 percent in 2016 [1].

Unfortunately, there is always a catch. The everincreasing demand for technology causes techcompanies to manufacture more and people to upgrade
their technologies. All of this causes a domino effect,
which leads to e-waste. Electronic waste or e-waste
could be defined as any electrical and electronicequipment disposed of without considering it being
repurposed. The term covers any objects with screens,
temperature-related equipment like refrigerators, lamps,
large or small electronic appliances such as a vacuum or
dishwashers, and portable electronics ranging from
Global Positioning System (GPS) to calculators [4]. Aswith any production, the manufacturing of technologies
is not proportionate to its recycling work. As of 2016,44.7 million metric tons of e-waste were produced,

There are a multitude of factors feeding the growth of
the usage of technology. The role of the government andhuman nature to favour efficiency are the pivot of this©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
550

Raenu Kolandaisamy, Heshalini Rajagopal, Indraah Kolandaisamy

comparable to 6.1 kilograms per inhabitant (kg/inh),
which increased compared to the 5.8 kg/inh amount in2014 [5]. In other words, the perception of people
regarding e-waste remains uncertain. Therefore, thisstudy aims to investigate the general public's perceptionof e-waste in Klang Valley.

people constantly upgrade to more efficient electronics,
contributing to the ever-increasing e-waste problem.
The old electronic devices are thrown away without
considering the toxic materials from the electronic
devices released into the environment. The groundwater,
soil and air affect land and sea animals, causing harmful
substances to seep into groundwater. When e-waste isreleased, toxic chemicals are released into the air,damaging the atmosphere.

2. Background

Numerous electronic devices contain hazardous
materials like lead and barium. In particular, the releaseof lead into the environment can result in adverse healtheffects, including damage to the human blood, kidneys,
and peripheral nervous system.

2.1. Overview
Electronic waste and electronic waste identify
mechanical and digital products that have been
discarded. These wastes are also regarded as appliances
deemed refurbishable, recyclable, disposable andsaleable. The processes involved when managing ewaste are burdensome, as they cause pollution andnegatively impact human conditions, especially indeveloping countries.

2.4. Amount of Electronic Waste Worldwide
Increasing technological changes, media developments(tapes, computers, MP3), affordable market prices andthe deliberate design of short life span technologies
have culminated in a rapid buildup of accumulation ofe-waste internationally. An achievable countermeasurehas been developed, but in most situations, it isnecessary to implement it to work legally in the realworld, which is time-consuming even in the mostadvanced nations, especially due to the bureaucracy.
Different models and versions of TVs, processors,storage units, and audio modules have varying lifespans.
Among these, screen systems and processors arecommonly affected by electronic waste. Screen systems
are often discarded without attempts at repair, whereas
processors see a continuous influx of new and improved
versions to meet the demands of ever-advancing video
gaming graphics.

2.2. Definition
E-waste or digital waste is generated after completing
its useful life when an electronic item is discarded. The
rapid expansion of innovation and the economy
powered by usage contributes to producing a large
amount of e-waste every minute. The European WEEE
Directive classifies electronic and electrical equipment
for waste management purposes into ten categories
namely large household appliances (e.g., refrigerators,
washing machines), small household appliances (e.g.,
toasters, coffee makers), IT and telecommunications
equipment (e.g., computers, phones), consumer
equipment (e.g., TVs, radios), lighting equipment (e.g.,
lamps, fluorescent tubes), electrical and electronic tools(e.g., drills, saws), toys, leisure, and sports equipment(e.g., video game consoles), medical devices (e.g., Xray machines, pacemakers), monitoring and control
instruments (e.g., thermostats, smoke detectors), andautomatic dispensers (e.g. vending machines).

2.5. Environmental Impact
The methods used to manage e-waste in developing
countries are often questionable. It has caused many
issues that have an unfortunate effect on Mother Nature.
These range from airborne or liquid-borne material
leakage into the atmosphere, soils, water and foods,
affecting animals, humans and plants.

2.3. Current Situation of E-Waste
Electronic waste keeps growing. Today, people arebuying more electronic devices, which are being quieted
faster. For example, cell phones generally have a shortlifespan of one to two years. People use most of theirincomes to buy electronics yearly. According to a study,
more than 20 million tons of e-waste are produced
annually.

2.6. Electronic Waste Substances
Many electronic parts can be recycled in manufacturing
new computer devices, while others are reduced tomaterials that can be reused for industries as diverse asbuilding, flatware, and jewelry. Large-scale materials
include fiberglass, PVC, fabrics, gold, iron, copper,
quartz, beryllium and oil. Almost all electronics containlead-tin (as solder) and copper.

The environmental impact of e-waste is a growing
concern in today’s society. With the widespread use ofelectronic devices and rapidly advancing technology,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
551

The Study on Perception

2.7. Effects

Francisco Bay area and provides mail-inservices to those who cannot make it.

It is known that 13 percent of e-waste is processed indeveloping countries. Yearly, e-waste is generated at alarge sum (around 40 million metric tons) worldwide.
According to the United Nations Environment Program(UNEP), the European Union contributed up to 22.5
percent of the e-waste. Meanwhile, in China, thePhilippines,
Pakistan,India
andVietnam,unconventional ways of the procession of e-waste arelargely practiced, with 50-80 percent of the e-waste
procession. The processes involve dismantling, burningand shredding the items in backyards. These actions
cause bi-products that have been proven impactful (in abad manner) to mother nature and humans alike. Theiraction, however, is also due to the large amount of ewaste flowing in since they are handling e-waste
domestically and internationally.

• Recycle and dispose of e-waste properly.
Inappropriate throwing of e-waste is a more
hostile problem as we speak, as its volume isincreasing every year. For this reason, major
business brands provide their customers with aplatform to recycle their old electronics.
• Maintain the electronics.
Effectively taking care of electric gadgets andappliances is a way of killing two birds with
one stone since it could halt the production ofe-waste and, at the same time, save money
since it could function longer.
• Store data online
Cloud services are effective in minimizing
environmental effects. By keeping data online,
we can access the data from any place all overthe globe. There is no need to take otherstorage devices.

2.8. Methods Used to Reduce E-Waste
The production of devices and the use of the materials
that go into their manufacturing represent a major wayof embodied energy. Reducing the e-waste helps protect
the resources and minimizes the required total energy
taken from the earth.

3. Methodology• Recycling
Recycling the highly valuable materials of oldelectronics rather than making or mining newmaterials for new devices.
• Stem the Spread
Before buying a new electronic device, think
twice. We can use the old electronic as an MP3
player or GPS device. Please do not throw it
away.
• Sell and donate old electronics.
The best and easiest way to minimize theelectronic waste footprint is to sell or give itaway to the people who need it. Giving out anelectronic device to poor people is a good
example of reducing e-waste; not only that, but
it also makes us feel good.

This study is based on an online survey. The
questionnaire was prepared with multiple-choicequestions that comprised general information about the
respondents (age and gender) and knowledge in regards
to the topic, such as the proper e-waste disposal
technique, country produces the most e-waste, e-waste
hackable after being thrown away, country hosts thebiggest digital dumping ground, the main causes of ewaste, its effect on children within the proximity of the
dumping ground, environmental pollution and the main
victim of e-waste. The data obtained from the survey
was computed in Microsoft Excel for analysis. The
statistics of each question were presented in the form oftables, pie charts and bar charts, as shown in the next
section.

• Educate and organize
We may also educate our neighbors and local
businesses on recycling. Returning theseelectronics to the recycling flow will lower thepressureonthemining,refining,
manufacturing, and transportation industries
and reduce pollution. Make a much more
affordable trip to the local recycling centre toorganize a group project to clean up oldelectronics. For example, Green Citizen.com
organizes pickups for those in the San

In total, 75 respondents took this survey. 69.3 percent of
the respondents were male, and 30.7 percent werefemale. Therefore, most of the respondents are males.
The survey was taken by people ranging from the age of
19 and below or 20 to 49. None of the respondents over
50 years old participated in this survey. About 74.7
percent of people aged 19 and below took this survey,
while 25.3 percent of respondents aged 20 to 49 took
this survey. Seventy-six per cent of respondents choose
to give to a reliable e-waste company. Moreover, 18.7
percent of respondents choose to throw e-waste in the

4. Result and discussion©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
552

Raenu Kolandaisamy, Heshalini Rajagopal, Indraah Kolandaisamy

rubbish bin, while 4 percent throw e-waste by throwing
it in the streets, and 0.3 percent of respondents choose toburn e-waste. The right answer is to give it to an e-wastecompany, as shown in Fig. 1. Based on a study,
Switzerland and Germany tackled the e-waste problem
efficiently. All these countries have a specializedcompany that handles it, such as the Public Waste
Management Authorities (PuWaMa) of Germany or the
Producer Responsibility Organization (PROs) of
Switzerland [6].
Good Way to Dispose E-Waste

E-waste company

throw in street

burn itthrow it in rubbish bin

Fig. 3: Result of respondent regarding the prospect of EWaste being hackable
Fig. 3 represents respondents' view on whether they
think e-waste are possibly hackable once thrown away.
As we can see, 72 percent of respondents said yes while
28 percent of respondents said no. Fig. 4 illustrates thepercentage of people's predictions on which country
would have the largest digital dumping ground. Brazil
was the most chosen, at 41.3 percent, followed by
Ghana, which had 25.3 percent of the respondents’choices. Mexico came in third, sharing 18.7 percent,while Kenya received 14.7 percent. Most respondents

Fig. 1: Respondents way of disposing E-waste

chose Brazil as the world's biggest digital dumping
ground.

Fig. 2: The Country that produces the most E-Waste

Fig. 4: People’s view on who shares the biggest EWaste dumping ground

Based on Fig. 2, respondents think China produces themost E-waste, with 58.7 percent of respondents, USA at
25.3 percent, and Canada and Ghana, gaining 8 percent
of respondent choices. Unsurprisingly, China produces
the most e-waste, amounting to 7.2 tons per annum.
However, it is also worth noting that Norway tops it allin kg per capita, at 28.3 [7].

Fig. 4 illustrates the percentage of people’s predictions
on which country would have the largest digital
dumping ground. Brazil was the most chosen, at 41.3
percent, followed by Ghana, which had 25.3 percent of
the respondents’ choices. Mexico came in third, sharing
18.7 percent, while Kenya received 14.7 percent. Most
respondents chose Brazil as the world's biggest digital
dumping ground. However, statistics showed Ghana is
the worlds’ largest digital dumping ground [8].

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
553

The Study on Perception

percent of respondents chose soil pollution,17.3 per centchose water pollution, and 32 per cent chose air
pollution, concluding that soil pollution is the main
pollution mostly caused by waste. E-waste produces all
types of pollution. It affects the air from the open
burning and dust emitting into the air, while water ispolluted when chemicals used are improperly treated
and let out into the normal world. However, soil
pollution is the most harmful form of pollution causedby e-waste [11], [12]. Toxic chemicals would beabsorbed into the soil, which later would be used forfarming. This would then grow harmful foods thathumans would consume. In other words, it slowly andsilently kills people who ate the food produced.

Fig. 5: Main causes of E-Waste
Based on the above Fig. 5, our respondents regarded
improper disposal as the main cause of e-waste, gaining
44 percent of the responses. On the other hand, 11
percent of respondents chose population growth, 8
percent chose pollution, and 37 percent assumed due totechnological advancement. In truth, the main
contributor to e-waste is the failure to dispose of it
effectively. Countries usually lack appropriate
infrastructure, legislation, quantification measures andfunding, resulting in a mass accumulation of e-waste [9].

Fig. 7: Respondents’ view on the possible pollution due
to E-Waste

Fig. 6: Effect of E-Waste on children near the digital
dumping ground
The pie chart in Fig. 6 shows respondents’ views on theeffect of e-waste on children near digital dumping
grounds. 29.7 percent of the respondents chose theanswer of the possibility of the children getting sick.
51.4 percent chose the answer of children getting
chronic diseases, 17.6 percent of children would get achance to learn technology, and 1 per cent would getmoney. Children exposed to e-waste are highly likely to
get a chronic illness or disease. It could harm thenervous, immune, reproductive and digestive systems.
Due to this, they would have permanent, irreversible
damage to their body (World Health Organization [10].
The pie chart in Fig. 7 below shows the possible typesof pollution e-waste can do to the environment. 50.7

Fig. 8: People affected by E-Waste
The types of people affected by e-waste are shown in
Fig. 8. 62.2 percent of respondents chose that workers
working at digital dumping grounds are affected most©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
554

Powered by TCPDF (www.tcpdf.org)

Raenu Kolandaisamy, Heshalini Rajagopal, Indraah Kolandaisamy

27 percent of respondents chose people who dispose of
their e-waste, and 10.8 percent of respondents chose
people who recycle e-waste. In conclusion, workers
working at digital dumping grounds are affected themost by e-waste. The people that are most affected arethose who work near the dumping ground. As stated
before, e-waste causes air, water and soil pollution.
Therefore, anything they do has a chance of being
contaminated, thus harming them, especially in the long
term.

7.

8.
9.

10.
11.

5. Conclusion
E-waste is one of the rapidly growing environmental
problems around the world. This is because of the lackof awareness and appropriate information on disposingof these wastes. Another reason for the rapid growth of
E-waste is technological advancement. As technology
advances, the old technologies will be left out because
they are outdated. Furthermore, because of the evergrowing rate of technological advancement, we are
creating more waste than ever every day. Most of thesewastes are hazardous to the environment, and due to thegrowing amount of waste, we are damaging theecosystem more and more every day [13]. Moreover,very few authorities are making any necessary moves
yet. We hope the government can create more
campaigns and talk about e-waste, its destructive
properties and what we can do to stop thisenvironmental crisis. The school authorities must also
create awareness in kids and young adults so they would
not even destroy the future. We hope the governmentand school authorities take immediate action to reduce
this environmental crisis.

12.

13.

Authors Introduction
Dr. Raenu Kolandaisamy
He received his PhD from the Faculty ofComputer
Science&
Information
Technology, University Malaya in 2020.
He is currently an Assistant Professor in
UCSI University, Malaysia. His researchinterest areas are Wireless Networking,
Security, VANET and IoT.
Dr. Heshalini Rajagopal
She received her PhD and Master's degreefrom the Department of ElectricalEngineering, University of Malaya,
Malaysia in 2021 and 2016, respectively.
Her research interest includes image
processing, etc.

References
1.

2.

3.

4.

5.

6.

Benchmarking: An International Journal, 25(9), 35193540. https://doi.org/10.1108/BIJ-07-2017-0168
Kinhal, V. (2017). Highest e-waste generating nations in
the world. https://www.worldatlas.com/highest-e-wastegenerating-nations-in-the-world.html
Biello, D. (2014, Jan 1). E-waste dump among top 10
most polluted sites.
Chowrimootoo, D.J.M (2011). E-Waste causes, hazards,
barriers and approaches to effective management.
https://www.academis.edu/8958150/e-waste
https://www.who.int/ceh/risks/ewaste/en
Wilson, D. (2016). Impacts of WEEE (e-waste).
https://ewaste.ece.uw.edu./students/impacts-of-e-wasteon-the-environment/
Kolandaisamy, R., Subaramaniam, K., & Jalil, A. B.
(2021, March). A Study on Comprehensive Risk Level
Analysis of IoT Attacks. In 2021 International
Conference on Artificial Intelligence and Smart Systems(ICAIS) (pp. 1391-1396). IEEE.
Kolandaisamy, R., Noor, R. M., Zaba, M. R., Ahmedy, I.,& Kolandaisamy, I. (2019, July). Markov chain based ant
colony approach for mitigating DDoS attacks using
integrated vehicle mode analysis in VANET. In 2019
IEEE 1st International Conference on Energy, Systems
and Information Processing (ICESIP) (pp. 1-5). IEEE.

Ritchie, H., Roser, M. & Mathieu, E. (2019).
Technologicalchange.
https://ourworldindata.org/technology-adoption
Prime Minister’s office of Malaysia. (n.d.).
https://www.pmo.gov.my/vision-2020/malaysia-as-afully-developed-country/
Desjardins, J. (2019, April 3). How the tech giants maketheir billions. https://www.visualcapitalist.com/how-techgiants-make-billions/
California Department of Resources Recycling andRecovery(Cal
recycle).
(2019).
https://www.calrecycle.gov/electronics/whatisewaste
Balde, C.P., Forti, V., Gray, V., Kuehr, R. & Stegmann, P.
(2020). The global e-waste monitor 2020: Quantities,
flows
and
the
circulareconomypotential.
https://collections.umu.edu/eserv/unu:6341/Global-ewaste
Chaudhary, K. & Vrat, P. (2018). Case study analysis ofe-waste management systems in Germany, Switzerland,
Japan and India: A RADAR chart approach.

Dr. Indraah Kolandaisamy
She is a senior lecturer in School ofBusiness Management, College ofBusiness, Universiti Utara Malaysia.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
555

Emergence of Cybercrimes in Online Social Networks
Raenu Kolandaisamy, Heshalini Rajagopal

Institute of Computer Science and Digital Innovation, UCSI University, 56000 Kuala Lumpur, Malaysia
Indraah Kolandaisamy

School of Business Management, University Utara Malaysia 06010 Sintok, Kedah
E-mail: raenu@ucsiuniversity.edu.my

Abstract
The rise of social networking websites has been seen in recent years. Everyone will be spending most of their time
on social networking websites such as Facebook, Instagram and Whatsapp. The great advantage that this social
networking website offers benefited many of the users. It can help people to promote themselves or their businessto gain more popularity and also customers through these social networking websites. There are many cybercrimes
that can be identified which are identity theft, hacking, fraud and so on. The emergence of cybercrimes has createdan awareness so that the users will know what the common attacks are and how they can be prevented from being
lure and being a victim of this attack. This research will discuss about the attacks and how these attacks can beprevented by the users.
Keywords: Social Network, Cybercrimes, Digital World

1. Introduction
Social network has been growing in the 21st century
and has become one of the trend that people use tocommunicate everyday. Social network is connectingwith different individuals using the internet throughsocial media websites [1]. It allows many individuals to
be connected to each other and thus allowing many
people to have communication with each other aroundthe world. Social networking has been so popular
because of the great ability and the features that it offers.
The number of social networking sites has been growing
from time to time. Some of the popular social media
application and websites are Facebook, Whatsapp,
WeChat, Instagram, Twitter, Google + and Skype. An
example will be Facebook that have over 2 billions ofactive users each month based on Fig. 1 as of June 2017
[2].
Each of the social media platform have different
features and design in it. There are many reasons why
social networking has gain so much popularity over theyears. The first reason will be the opportunity to meet
new people. We live in a world that is full of different
culture and environment thus meeting new people fromdifferent countries all over the world is indeed a great
joy for many of us.

Fig. 1: Monthly users of social media applications
The second reason will social networking sites are user
friendly and thus is easy to use. Most sites aredeveloped with a very user friendly interface and is easy
to navigate which require a little knowledge to learn.
The third reason is that it is available to use on many
platforms such as desktop computers, tablets and also
smartphones with different operating system. The fourth
reason is that social networking sites are free of charge
to use and therefore many users will certainly want totry and use it [3]. Through gaining popularity to thissites, many also get the opportunity to advertise theirproduct and services thus making social networking agood place to start and grow your business. The fifth
reason is that social networking sites create theopportunity for graduates, businessmen and also

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
556

Raenu Kolandaisamy, Heshalini Rajagopal, Indraah Kolandaisamy

professionals to find their ideal jobs and also to meet thepeople that have the same interest as theirs.
The rise of social media certainly has a great impact on
the community and the great benefits it has to offer but
there also dangers in using social networking because of
the large community it possesses and thus making user
vulnerable to security threat and attacks. It can beshown that in recent years through social media thatmany have faced attacks such as scam and conned byusers on social networking sites trying to impersonate
another person seeking for donation and other personal
information such as bank details and so on. There arevery dangerous in the society and unaware users might
be lured so this studies to help to increase the awarenessof cybercrime in social network.
Cybercrime can be any activities that compromise on
the security of the users and try to exploit another user
using the internet. It is an illegal activity that involves
the use of computer and the internet through a platform
like social networking to gain or to cause destruction toanother user on the that network. Cybercrime is thegreatest attack based on the Fig. 2 that shows the
motivation of the attackers [4].

gift that can be obtained if they follow the steps tocomplete it on a website, but he or she sends it to hermany friends on Facebook thus making them to be a
victim as well because of the person doing things
unknowingly.
2. Literature Review
2.1. Detecting Compromised Accounts on Social
Networks
Social networking is one of the most used website intoday’s internet world. Social networking is a websitethat uses the internet to bring people together to share
information, ideas and also make new friends. This canalso be known as social media. Everyone loves to use
social media because it is convenient and free. Social
media websites such as Facebook, Instagram, Twitterand Whatsapp can all be accessed on various platform
such as smartphones, tablets and computers. This has
been a great advantage to many users around the world.
Social networking has been very demanding andpopular but the ever increasing of cyber criminals
activities are also increasing as well making it very
dangerous.
The study on detecting compromised accounts on social
networks by Manuel Egele, Gianluca Stringhini,
Christopher Kruegel and Giovanni Vigna will be
discussed. The act of compromising on social network
accounts has been a great and profitable job that has
benefit the cyber criminals. The reasons here is because
the speed and reliability of social medias account. By
targeting a social media account of a famous artist thathave many followers and by sending out fake messages
can be able to scam many thousands of users that havefollow that famous artist. It is very dangerous because
once an attacker managed to compromise a social
network account, the person can use it for many
purposes such as sending out spam messages to people
or even phishing web sites link. It is a higher benefit for
attackers to target large and popular company that haslarge amount of followers like news company.
Fake accounts and legitimated hacked accounts havedifferent characteristic so the system is harder to detectbecause it is not the fake accounts that is created by auser, it has been a good standing account but it is being
compromised. This research introduced COMPA which
is basically a detection system that is designed to
identify compromised social network accounts.
COMPA is use to monitor the habits of the user in asocial network account [5]. A person who uses social
media like Facebook usually have a fairly stablebehaviour from the messages that is being send or theamount of time that the person is active. How thisworks that COMPA will actually build a behaviouralFig. 2 Motivation behind attacks
Cyber criminal can be categorized into many aspect. It
can be broken down into a few different parts which are• A website lure a users to click on an image or alink to visit another website that can spreadmalware to the users computer to take control
and gain information from the users computer.
• A scammer try to claim to impersonate
someone to be a bank staff to gain personal
information such as password or even donation
by creating a fake account.
• The hackers hack your social networking
account to use as a way to get details fromclose people like family.
The danger to the victim here is that, they can lose theirinformation in a matter of minutes. It will be a big
problem to many society and what's more dangerous isthat one victim can lead another victim to be a victim.
For example, a victim receive an email regarding a free©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
557

Emergence of Cybercrimes in

profile for social network accounts and then it will start
to monitor and compare the new messages that is being
sent by the account, if it looks different from what it
usually is then Compa will flag it as being compromise.
There are two common use to detect compromise
account and the first is call suspicious group. Suspicious
group are those that send and do more than what theaccounts normally do. For example, account A usuallysend out 50 messages a day and averages about 45 permonth, but that particular month itself, it exceeded theregular threshold by sending over 500 messages a day.
It can be easily be detected especially if the amountraise too significantly.The second is call bulk
application. Bulk application are the one that send out
large amount of messages with the same amount of text
and words.
The studies have evaluated both social media websites
which are Facebook and Twitter. The data collection
method was to collect real time tweets from Twitter andran Facebook experiments on a large dataset crawled in2009. The advantages of COMPA is able to identify
groups of compromised accounts that are to distributemalicious messages on this accounts.
The limitation of this studies is that if the attacker isaware of COMPA, the attacker learn about the victim’s
behaviour before compromising so that COMPA cannot
detect the unusual behaviour of the account because of
the similarity of the action that is being possessed by theattacker that imitate the action of the victim. Besides
that, social networking sites are harder to gather data.

detecting cyberbullying on Twitter that has been doneby detecting the use of offensive language on Twitter.
The detection of of abusive language can include theuse of acronyms and words that can increase thecyberbullying classifier. Cyberbullying can be also
associated with the aggressive behaviour of the users on
Twitter. As certain words can be harder to detectbecause the words needs to be read as a whole sentence
to know whether it is offensive or not. In this case,plenty of information that needs to be utilized to achieve
and accurate detection to says that the particular tweet isdangerous and abusive. The detection of vulgar words
can also be used as this also a signal of offensive
behaviour. The cyberbullies can also use many ways totweet and of way is to use abbreviation or acronyms toshorten the sentence. The studies uses machine learningalgorithms to detect the words with various method.
They are four different ways that has been used which
are NB, LibSVM, random forest and KNN. This are all
the method that is used the various results andconclusions has showed that the study of abusive
language and cyberbullying will required the machine toadapt and new learn words as time goes on because newvulgar words with abbreviation may be form and thus
making it difficult to detect. The studies suggest thatusing other social medias can also improved thelearning process of the system so that cyberbullying canbe resolved and reduced in the community of social
networking users.
2.3. Organised crime goes online: realities andchallenges

2.2. Cyberbullying detection in the Twitter
networkThe Internet is like a vast galaxy of interconnected
networks that allows many transactions and events
taking place virtually from the whole world - including
cyber crimes and also Internet-facilitated organizedcrime. The study of the realities and challenges oforganised crime online by the author Anita Lavorgna
will be discussed.
Cyber crimes are known as a group of criminals thatuses ICT infrastructure to commit crimes, such as
phishing, malware injection, Denial of Service, and somore. However, in this world we are living in,organized crime groups (OCGs) have existed for longtime and have conducted crimes offline. This paper
mostly discussing on how these OCGs make use ofInternet or online presence to evolve themselves intoday’s situation where everyone is more Internetdependant.
The data for this paper are from interviews and also datacollected from law-enforcement officials and experts
researchers in the United Kingdom, United States, andthe Netherlands. 120 case studies regarding crime with

The rise of social networking websites has been such agreat impact to many users in the world. Many use thisopportunity to gain new friends, establish new
relationship and also to connected with family andfriends. Although it has been seen by many as a benefit
but the use of social networking websites has been so
popular that many have use it in the wrong manner toabuse it and gain advantages over it but causing harm toanother. The discussion in this literature review is by
Mohammed Ali Al-garadi , Kasturi Dewi Varathan and
Sri Devi Ravana which is about the experimental caseof cyberbullying detection in the Twitter social
network[6].
Firstly cyberbullying can be defined as the use of
information and communication technology on theinternet and it can on any social media platform forexample Facebook, Twitter, Google+ and also
Instagram to harass another user. This research andstudies is to focus on how cyberbullying happens on
Twitter. The study is to improve the detection rate on©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
558

Raenu Kolandaisamy, Heshalini Rajagopal, Indraah Kolandaisamy

the usage of Internet are also taken into the analysis of
this paper.
This paper focuses on two types of OCGs, one istraditional business-style groups, the other is mafia-stylegroups. The type crimes committed by these businessstyle OCGs are identified - mostly on trafficking
activities such as wildlife, human, psychoactive
substances (drugs), and also fake medicines. Thesecriminal activities are more to businesses motive andthus using the Internet to facilitate online. With the useof online social forums, instant messaging feature forexample, it is easier for these crime groups to conduct
and also expand their businesses ties with the client.
Mafia-style groups, however, were found to be still
grasping on the use of Internet technologies for theircriminal activities. Trafficking and gambling activities
were their main focus. However, mafia-style groups as
are mostly depend on a territorial and political trust,hence face-to-face interactions are needed. The highest
hierarchy of this group were found to be mostly leaded
by a leader who are not fancy of the use of the Internet.
However, as these groups keeps evolving in the future,
the usage of Internet in their criminal activities are to be
expected as new generations will lead in the future.
The previous points discusses the reality of these OGCs.
The challenges however, observed on legislation andpolicies for these Internet-mediated crimes. It is foundthat the regulatory policies regarding these crime
activities via the Internet are not enough to control them.
The instantaneous and universality of the Internet makesthese inhibits these regulatory policies to fully takeeffect. Other challenges found for legislation andpolicies are such as different approaches applied indifferent countries as some of these countries havedifference in experience in regards to tackling online
crimes and carrying out investigations. [7]take security lightly and this poses security and privacy
threat upon their account.
Among the security issues discussed in this paper are,firstly, identity theft. It is defined as an action of
acquiring someone’s identity without permission andmisusing it for malicious purposes, such as fraud. The
author(s) suggested that users of SNS should not simply
accept unknown friend requests and check therequesting account beforehand for safety.
Secondly, phishing is also a security issue found in SNS.
Preparators of phishing uses ‘mock-up’ websites to lureusers into logging in and thus exposing their sensitive
credentials such as login credentials. Author(s)suggested users to always use trusted devices to log into accounts and be wary of unknown websites. Other
security issues found in SNS are such as brute force
attack, Sybil attack, and Trojan attack.
Privacy issues in SNS were also discussed in this paper.
Privacy settings is said to be always overlooked by mostSNS users therefore this poses a huge privacy risk tothem as social networks get to hold users sensitive data.
It is found that some users even share their sensitive
data, whether aware or unaware.
Among privacy threats discussed are such as cyberstalking, whereby most SNS has made it easy for theselurkers to stalk users’ profiles without adding them as
friends. Author(s) suggested users to keep their profiles
privacy mode hidden and only to be shown to trusted
parties only. Secondly, cyberbullying also a privacy
threat found in SNS. Cyberbullying example are such ascommenting illicitly on people’s photos or posts andalso tarnishes one’s reputation online, mostly
anonymously. Author(s) suggested users to keep things
shared on SNS away from third party users and also bewary of them.
Other privacy threat includes profile cloning, whereby
preparators create another account with similar
information as the users’, and therefore performing
malicious actions online to the victim’s friends.[8]2.4. Privacy and Security Issues in Social Online
Networks
Social networking has becoming a necessity in today’ssociety as more people in the world are having easieraccess to the Internet. However, this new necessity canbring a potential danger especially to new and also“naive” users of online social networks, especially
teenagers. In this paper by Radhika Bhagat, Rajvi Modi,
Palaumi Patel, and Harshil Joshi discusses about thepotential privacy and security threats in social online
networks today.
Users of online social networks were discussed in thispaper. Most of the users using social network sites(SNS) to get informations and happenings all aroundthem instantaneously. Personal information were
propagated by them using SNS. Users of SNS tend to

2.5. Cybercriminal Networks, Social Ties andOnline Forums: Social Ties Versus Digital
Ties Within Phishing And Malware
Networks
As we are living in a digital age, everything is adapting
to it. Not just people, and organizations as a whole, buteven cyber criminals too. The usage of digital means
such as Internet, and its platforms such as forums, social
networking services, has made it easier for individuals
to get together and communicate anywhere and anytime.
This paper, authored by E. Rutger Leukfeldt, Edward R.
Kleemans and Wouter P. Stol, discusses about©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
559

Emergence of Cybercrimes incybercriminal networks applying social ties and/or
digital ties for their activities and recruiting.
A social network usually formed by actor to anotheractor with ties between them that defining theirconnection and relationship. A social tie are usuallyformed via social contact, for example, meeting in a
specific place such as cafes, to engage social activity.
Digital tie can be formed virtually online without social
contact such as using forums.
Social relationships are crucial in criminal networks.
However, social relationships are highly clustered withexisting relationships such as contacts from former
workplace, friends, family, and these relationships are
said to yield less opportunity. With Internet, theboundaries of traditional social network are blurred and“specific offender convergence setting” can take place
via online forums where these cybercriminals can get
together and share information online from different
places and also recruit new members or enablers.
The origins and growth of cybercriminal groups were
mostly investigated in this paper. The data for this paper
is taken from 18 Dutch police files on cybercrimes thatprovides them with knowledge cybercriminal networks
and its members.
Four models of growth were developed in this paper. It
is found that cybercriminal networks’ growth areassociated with recruitments of enablers - professional
and/or recruited. These enablers provides services tocybercriminals to give them access to useful andrelevant information such as credit card information andpostal information details. The four models of growthconsists of the first stage, that is “entirely through social
contacts”, second stage “based on social contacts;
forums for recruiting specialists”, third stage “based onforums, with social contacts used to recruit local
criminals” and fourth stage that is “entirely throughforums”.
It is found that most of the cybercriminal networks uses
social contacts entirely (first stage) to grow their
network and does not use online forums. However, in
future, forums may become a more common place forthe growth of cybercriminal networks in this digital era.
[9]•taken by the society in dealing with onlinesocial networks, especially in situations that
may cause cyber crimes to take place.
Social network visualization
A Facebook dataset from Stanford Network
Analysis Project (SNAP) are used todemonstrate and visualize how actors areconnected in social network such as Facebook.
[10] Social network analysis visualization tool
used is Gephi.

4. Result and discussion
We have found that the problem with cybercrimes willcontinue to increase as more users increase in the use ofsocial media. This is because the great number of users
can be a good target for the hackers and attackers
because it can be very lucrative if the hacker is being
able to compromise millions of users at once. That is
why online social media has also been a dangerous
platform to know friends because of the use of a device
and a face to face conversation has not been done andthat makes it very difficult to know the real person’s
identity. The person can be a boy on Facebook talking
to another person, but in reality she is a girl. That isreally a very dangerous act and it can cause many
unwanted things to happen on social media websites.
Nowadays, because people are also attracted by free
stuff and gifts only, it can be such an easy way to cheatpeople because of the lack of knowledge the users
possess and thus it is very dangerous for users to clickon links that link to malicious website that can harm theuser’s computers.
To find out how actors connected to each other in asocial network, we have used a social network analysis
tool software Gephi to visualize the network based on adataset collection from Stanford Network Analysis
Project (SNAP). This dataset is particularly fromFacebook. As shown in Fig. 3 this is the result of thevisualization projected from the dataset. The type of thedataset is undirected, it has 4039 nodes, and 88234
edges.

3. Methodology
The methodology used in this paper are are as follows:• Questionnaire
A set of questionnaire created in Google Forms
is distributed online for respondents to fill in.
The respondents sample size is taken among
students of UCSI University as a representativeof the public society. The questionnaire would
help to analyze the approaches and actions©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
560

Raenu Kolandaisamy, Heshalini Rajagopal, Indraah Kolandaisamy

Fig. 3 Result of the visualization projected from the
SNAP dataset
Fig. 4 shows our simulation on how a group of “rogue
actors” can affect the social network. These “rogue
actors” nodes are represented in color red. This showsthat a small group of them can have quite a spread
impact on the network. Note that this dataset is not a full
dataset from Facebook - it is collected from SNAP
survey participants on Facebook. However, a small
dataset can be observed quite complex.

Fig. 6 Number of friends on social networking site
Fig. 7 shows that most of the respondents accepts friend
requests even if they don’t know the person well.
Although the difference between those who answered
Yes and No is quite minimal, this shows that there arepeople still doing that practice.

Fig. 4 Simulation on how a group of “rogue actors” canaffect the social network
The questionnaire was distributed online and conducted
via Google Forms. Over 35 respondents responded tothe questionnaire. Questions asked were about their
online social network service they are using, how dothey use, how many connections they have, how do they
react with unknown person on social network, and so on.
The results are shown in the charts and graphs below.
Based on the chart above in Fig. 5 asking about social
network that is being used by the respondents andFacebook has the highest amount of users based on thesurvey done. It clearly shows that Facebook is still themost popular social network website that is use among
the people.

Fig. 7 Friend request acceptanceFig. 8 shows that most of the respondents do get a
private message from unknown person on online social
networks.

Fig. 8 Receiving private messages from unknown
person on online social networks
Fig. 9 shows that many messages that is being sent tothe user are mostly an advertisement with a link
attached to lure the users to go to another website. The
next is the attempt to get to know you more which isvery common in social networking sites because is aplatform to get to know more new friends. The last isjust a simple link without any description.

Fig. 5 Social Media used
The pie chart above in Fig. 6 shows that many people
have many friend on social networking website and it
can be clearly seen that many spend time on social
networking site making new friends.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
561

Emergence of Cybercrimes inFig. 13 shows that they are many people who use
Facebook do not set their post to public because andprefer to be private. This is a good measure of the users
to prevent unwanted use for comments by unwanted
users.

Fig. 9 The typical messages
Fig. 10 shows that many users will still click on the linkeven is a banking website. It shows that many users willclick if the link looks trustworthy. Which the yes has thehighest user.

Fig. 13 Privacy settings
Fig. 14 shows that many people do not check and see
who is the publisher of the content creator before
allowing others to access their Facebook page andcontent.

Fig. 10 Clicking links on website
Fig. 11 shows that many users are being added into
unknown groups by other people that they do not evenknow.
Fig. 14 Publisher details and privacy policy
Fig. 15 shows that many have not being cyber bullied
on social network sites and that is a good thing.

Fig. 11 Added to group chat by unknown person
Fig. 12 shows that there are people who are being
tagged by others into a picture which their faces are noteven in. This shows that many people are being byothers as a way of advertising their product.

Fig. 15 Cyberbullied
Fig. 16 shows that they are many users who will justaccept friend request without thinking twice whether the
person is real or not. This is pretty dangerous as this canallow hackers to compromise other users.

Fig. 12 Anonymous tagging©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
562

Raenu Kolandaisamy, Heshalini Rajagopal, Indraah Kolandaisamy

most users make is to create a password that is similar
to their birthday date or telephone number and thatmakes easy for many to easily compromise his or her
account. Besides that, talking to an unknown stranger
can also be very dangerous especially those who areseeking for new friends and are desperate, they tend to
do things without thinking of the danger.
From the findings gathered in the previous section, it isapparent that because of the way on how online social
networks users uses the platform, it contributes to theemergence of the cyber crime in online social networks.

Fig. 16 Accepting anonymous friend request
Fig. 17 shows that many users are still very unawarethat cyber crime is dangerous and this users will requireneed to have more knowledge if they want to keep
themselves safe.

References
1.

What is social networking? - Definition from
WhatIs.com", WhatIs.com, 2018. [Online]. Available:
http://whatis.techtarget.com/definition/social-networking.
[Accessed: 07- Mar- 2018].
2. J. Constine, Facebook now has 2 billion monthly users…
and responsibility, TechCrunch, 2018. [Online].
Available: https://techcrunch.com/2017/06/27/facebook2-billion-users/. [Accessed: 07- Mar- 2018].
3. M. Fita, M. Fita and M. Fita, 6 Reasons Why Social
Networking is So Popular These Days |, Brandignity.com,
2018.
[Online].
Available:
https://www.brandignity.com/2012/11/6-reasons-whysocial-networking-is-so-popular-these-days/. [Accessed:
07- Mar- 2018].
4. July 2017 Cyber Attacks Statistics, HACKMAGEDDON,
2018.
[Online].
Available:
https://www.hackmageddon.com/2017/08/24/july-2017cyber-attacks-statistics/. [Accessed: 07- Mar- 2018].
5. Manuel Egele, Gianluca Stringhini, Christopher Kruegel,and Giovanni Vigna, Towards Detecting Compromised
Accounts on Social Networks.
6. Kolandaisamy, R., Noor, R. M., Zaba, M. R., Ahmedy, I.,& Kolandaisamy, I. (2019, July). Markov chain based ant
colony approach for mitigating DDoS attacks using
integrated vehicle mode analysis in VANET. In 2019
IEEE 1st International Conference on Energy, Systems
and Information Processing (ICESIP) (pp. 1-5). IEEE.
7. A. Lavorgna, "Organised crime goes online: realities andchallenges", Journal of Money Laundering Control, vol.
18, no. 2, pp. 153-168, 2015.
8. R. B. R. M. P. Patel and M. H. Joshi, "Privacy andSecurity Issues in Social Online Networks,", National
Conference on Latest Trends in Networking and Cyber
Security, 2017.
9. E. Leukfeldt, E. Kleemans and W. Stol, "Cybercriminal
Networks, Social Ties and Online Forums: Social Ties
Versus Digital Ties within Phishing and Malware
Networks", British Journal of Criminology, p. azw009,
2016.
10. J. Leskovec and A. Krevl, SNAP Datasets: Stanford
Large Network Dataset Collection, 2014.

Fig. 17 Awareness on cyber crime
Fig. 18 shows that many are still in the middle and somestill think that social networking websites is still unsafe
and is dangerous to many people.

Fig. 18 Safety of social networking sites

5. Conclusion
Social networking has been really a good way tocommunicate with friends and family and thus make life
happier and more joyful. This is because of theconvenience that is offer and many users love using it
and have spend hours and hours a day just to go throughsocial media and looking at post posted by others. The
security of this social medias websites has improved
over the years but there are still many loopholes that becompromised by the hackers because of the lack ofknowledge that the users have and thus making users
more vulnerable to attacks. One common mistakes that

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
563

Powered by TCPDF (www.tcpdf.org)

Emergence of Cybercrimes in

Authors Introduction
Dr. Raenu Kolandaisamy
He received his PhD from the Faculty ofComputer
Science&
Information
Technology, University Malaya in 2020.
He is currently an Assistant Professor in
UCSI University, Malaysia. His researchinterest areas are Wireless Networking,
Security, VANET and IoT.

Dr. Heshalini Rajagopal
She received her PhD and Master's
degree from the Department of ElectricalEngineering, University of Malaya,
Malaysia in 2021 and 2016, respectively.
She received the B.E (Electrical) in 2013
Currently, she is an Assistant Professor in
UCSI University, Kuala Lumpur,
Malaysia. Her research interest includes image
processing, artificial intelligence and machine learning.
Dr. Indraah Kolandaisamy
She is a senior lecturer in School ofBusiness Management, College ofBusiness, Universiti Utara Malaysia.
Indraah A/P Kolandaisamy holds a
doctorate in Management from Universiti
Kebangsaan Malaysia in 2015. Her D.B.A
work is on organizational citizenshipbehavior among public sector in Malaysia. She obtained
her MSc (Management) and Bachelor in International
Business Management (Hons) from Universiti Utara
Malaysia respectively in 2007 and 2005.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
564

Development of a music recommendation application by using facial emotion recognition
Shengke Xie, Raenu Kolandaisamy, Ghassan Saleh

Institute of Computer Science and Digital Innovation, UCSI University, 56000 Kuala Lumpur, Malaysia
Heshalini Rajagopal

Department of Electrical and Electronics Engineering, MILA University, 71800 Nilai, Negeri Sembilan, Malaysia
E-mail: raenu@ucsiuniversity.edu.my

Abstract
Music is an important part of human life and culture, and it can affect people’ s emotions and moods. However,choosing music from a large library can be a challenging and time-consuming task. In this paper, we propose afacial expression recognition-based music recommendation system that can recommend suitable music whichmatches the user’ s current mood. The system uses a camera to capture the user’s face and a convolutional neuralnetwork model which trained facial emotion recognition database to recognize seven basic emotions: anger, disgust,fear, happiness, sadness, surprise and neutral. The paper contributes to the research of facial emotion recognition
and music recommendation and provides a convenient way for people to enjoy music.
Keywords: Facial Emotion Recognition, Music Recommend Systems, Convolutional Neural Network

1.

Introduction
With the development of the Internet of Things (IoT)
technology, various smart homes are entering the publiceye, and AI voice assistants used to unify the
management of various smart home appliances havealso emerged. People control the use of various
electrical appliances and furniture in the same networkby giving commands to the AI voice assistants with
their words, while many AI voice assistants have added
many other functions in addition to the basic home
control functions in order to increase the diversity offunctions, such as music recommendations. Whenpeople are looking for music to listen to, they are often
confused about the choice of music and are looking forsome suggestions to help them choose, since it is often
difficult to choose the most suitable music from a large
library of thousands of songs, and music
recommendations are one such feature that can reducethe difficulties users have in choosing music [1].
However, the music recommendation function 2.
sometimes does not meet the user's needs, and there areoften situations where the recommended music conflicts
with the user's current mood, such as recommending sadsongs when the user is happy, or recommending songs
with a strong rhythm when the user needs to calm down[2]. Therefore, how to make the computer quickly andaccurately recommend the right song to the user is aproblem that needs to be solved. It should be noted that
facial expressions account for two-thirds of humancommunication and are one of the most importantmeans of expressing human emotions. In the absence offacial expression recognition, AI is sometimes unable togive correct and rapid feedback through speech alone.
For example, the same phrase what's up may havedifferent meanings depending on the emotion, it may be
a greeting between acquaintances, or it may be aconcern to ask what difficulties you are experiencing, so
people hope that computers will be able to understandhuman emotions and give correct feedback, bringing
people a better experience with more intelligent humancomputer interaction [3].
For this problem, we will use a camera to capture theuser's image, use python deep learning based facial
expression recognition technology to identify the user's
current mood, and select the music that matches theuser's current mood from the music categories in the
database that apply to different moods to recommend
music to the user, thus trying to reduce the user's
anxiety in choosing music and improve the user's
experience
Literature Review
Communication is a bridge to build interpersonal
relationships, people send or obtain information throughcommunication, and facial expressions are an importantmeans and factor to help people understand theinformation conveyed by others in communication,according to the survey [4] the non-verbal component ofcommunication reaches about 55% of interpersonalcommunication, and as an important category of nonverbal component, the study of human facial
expressions is not only in medicine and psychology, its©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
565

Shengke Xie, Raenu Kolandaisamy, Ghassan Saleh, Heshalini Rajagopal

wide application prospects have also led to its
widespread interest in computer science [5].
As early as 1971, two American psychologists, Ekman
and Friesen, had systematically studied facial
expressions and in 1978 developed and defined theFacial Action Coding System (FACS), a system forrecognizing various human emotions by representing
different facial expressions in terms of different facialmuscle changes. The system encodes specific facialmuscle changes, called Action Units (AUs). Fig. 1
shows the main types of Aus.

emotions. Fig. 2 shows the main types of human facial
emotions represented by the various AUs combinations.

Fig.2. The AUs combinations that can be observed indifferent human facial emotions [5]3.

MethodThis system implements a music recommendation
system based on facial expression recognition on the PC.
Fig. 3 shows the steps for facial recognition.

Fig.3. Steps for facial emotion recognition
The aim of this system is to use the computer's powerfulcomputing resources to quickly and accurately
understand human emotions and recommend more
appropriate songs to the user, bringing a smarter
experience to people.
The system was developed on a PC with Windows 10,
64-bit operating system, Intel-i7-10510U CPU, 16G
Byte RAM, PyCharm 2022.2.2, OpenCV-3.4.4.19
compilation environment and python as the main
programming language. Tests of the system were also
carried out on this machine.
The process of using this system can be simply
summarized as capturing the user's image using the
local camera, using deep learning based facial
expression recognition technology to identify the user's
current mood, and recommending music that matches
the user's current mood from the downloaded music thathas been categorized into different mood folders, thus
trying to reduce the user's anxiety in choosing music
and improve the user's experience. Fig.4 shows theworkflow of this system.

Fig. 1. The main types of Aus
It is well-known that human beings have six basic
emotions (BEs), namely happiness, surprise, anger,sadness, fear and disgust, on the basis of which Du et al[6] have proposed 22 compound emotions (CEs) madeup of combinations of basic emotions with each other,and different combinations of AU classify these facial
emotions to facilitate identification, e.g. AU
combinations 1, 4, 20, 25 can be identified as fear

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
566

Development of a music5) Music classification module: count the emotion
value of each emotion word in the text according to theemotion dictionary, and classify the music files
corresponding to the lyrics according to the emotion
value.
6) Music recommendation module: according to theexpression recognition result, a song is randomly loaded
from the local music folder of the corresponding emotion7) Music playing module: play or pause the loadedsong.
Among them, the core function of the system is
Emotion recognition, the recognition result will bedirectly related to the quality of the system, theappropriateness of the recommended music.

Fig.5. Structure diagram of the system modules

3.1. User Interface Design
A clean and aesthetically pleasing user interface (UI)
provides a good user experience and helps users tointeract easily with the software system. Here the UI isdesigned by using PyQt5, a GUI framework for the
Python programming language, which makes it easy to
implement powerful user interfaces using a variety of UI
controls. The main controls used in this system are Label,
Button, Line Edit and Timer.

Fig. 4. System workflow diagram
According to the need, the design of this system asshown in Fig. 5 is mainly divided into seven modules
such as Image pre-processing, Face detection, Emotion
recognition, Music download, Music classification,
Music recommendation and Music playing:
1) Image pre-processing module: the image capturedby the local camera is pre-processed, the pre-processing
process is mainly to convert the colour image into a grey
scale image, which is convenient for face detection.

Fig. 6 show the design blueprint of the UI andscreenshots of the actual UI interface respectively, withthe functions of each UI control as follows:
1) close_btn: Button control, used to close the wholesystem.

2) Face detection module: face detection is
performed on the grayscale image, after the face is
detected, a rectangular box is drawn on the original
colour image to frame out the face, in preparation for
expression recognition.

2) title_label: Label control for scrolling the name of
the currently playing music file.
3) result_label: Label control, used to display the
result of face expression recognition.

3) Emotion recognition module: The detected face
image is fed into the expression recognition model for
expression recognition. The recognition model is a
convolutional neural network model with parameters
trained to recognize the face expression and return the
result.

4) camera_btn: Button control for turning the local
camera on and off.
5) play_btn: Button control to control the playing andpausing of the currently loaded music.

4) Music download module: The crawler crawls theMP3 file of the searched music from the music playing
website and downloads it to the local area, whilecrawling the corresponding lyrics and saving them in txt
format, so as to prepare for music classification.

6) next_btn: Button control for randomly loading apiece of music from the folder matching the expression
recognition result.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
567

Shengke Xie, Raenu Kolandaisamy, Ghassan Saleh, Heshalini Rajagopal

7) detect_btn: Button control that detects the user’s
current expression, returns the result and displays it in
the result_label.
8) searchMusic_LEdit: Line Edit control for enteringmusic names in preparation for music downloads.
9) download_btn: Button control for crawling anddownloading mp3 files and lyric content of the musicentered by the user.

Fig.7. Network structure parameters [7]10) camera_label: Label control to display the image
captured by the camera.

3.3. Design and implementation of the mainfunctions of the system
The music recommendation system based on facial
expression recognition is divided into seven modules,
and the main functions of each module are briefly
introduced above.
Image pre-processing module
In order to ensure consistency in face size andposition as well as face image quality in face images
and to improve the efficiency and accuracy of facerecognition, the images acquired by the camera need to
be pre-processed prior to face detection, which cansignificantly reduce the computational effort of thedevice. The main task of pre-processing is image
greyscale, which means that the image is unified into agrey-scale map of a specified size. In actual use willmeet the image acquisition effect is not good, such as
lighting and other factors cause image results are notgood, and ultimately affect the recognition results, andthe grayscale process can effectively remove these noise
influence, after the grayscale, the influence of the noiseis reduced to a minimum.
The method first normalises the input raw face
image, scaling the pixel value range from 0-255 tobetween 0-1, then scales the raw face image to a
specified image size of 48×48 and generates a set offace images of different sizes and orientations,providing more information for subsequent tasks suchas face recognition and expression recognition, thus
improving the accuracy and robustness of the algorithm.
The greyscale of the images is achieved using the
COLOR_BGR2GRAY method in OpenCV.

Fig.6. Design of the UI
3.2. Design of Convolutional Neural Networks
The design of the network structure affects themodel performance and the accuracy of theclassification. A good network structure can extract
features more efficiently and reduce the training time.
This system uses the convolutional neural network
structure designed by Zhou [7] as the basis forclassification. It achieved good recognition results on
both the FER-2013 test set and the dataset [8]. The
structure of this convolutional neural network consistsof a 48×48 input layer and a 1×1 convolutional layer
to form the first group of layer sets, and two consecutive
convolutional layers with filter sizes of 3×3 and 5×5
followed by a 2×2 maximum pooling layer to form thesecond and third group of layer sets. Each convolutional
layer uses ReLU as the activation function. The second
group of layers has one more convolutional layer than
the first group, which means that further refinement offeatures can be learned based on the first group of layers.
The detailed structure of the network parameters isshown in Fig. 7.

Face detection module
The prerequisite for expression recognition is thatthe recognized image contains a face component, so
face detection is a powerful guarantee for expression
recognition.
This
moduleuses
theface_detection_front.tflite
andface_detection_back.tflite models from [8] for facedetection.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
568

Development of a music

The method first initialises the face detector, uses
the detectFaces() method which from detector to detectfaces in the input image, and returns a detection resultcontaining the bounding box, key points and scores foreach detected face. Finally, the returned detection
results are used to obtain and return the absolutecoordinates of the face bounding boxes.

surprise, sadness, joy and disgust) and two emotions(negative and positive), and saves them in a csv file foreasy searching. We used this sentiment dictionary toanalyse the lyrics of the song to find the emotion withthe largest sentiment value among all the emotions in
the text of the lyrics as the final classified emotion, thus
determining the sentiment classification of the song.
The code first read the sentiment dictionary and map
the words to their corresponding sentiment values. Thenthe classify_sentiment() method below counts thenumber of each sentiment word in the input text and thetotal sentiment value based on the sentiment value ofthe word in the dictionary, thus returning the sentimentwith the highest sentiment value as the result.

Emotion recognition module
Emotion recognition is the core function of thissystem, image pre-processing, face detection modules
are all paving the way for the implementation of thismodule.
In this method we first load a pre-trained facial
emotion recognition model, read the video frames fromthe local camera using the capture.read() method in the
OpenCV computer vision library, resize the video
frames and greyscale them. Next, the faces returned by
the face detection module are recognized for theiremotions using the predict() method, and the predicted
results are annotated in the original image and displayedin the control camera_label.

Music recommendation module
The music recommendation module is mainly basedon the result of face emotion recognition, the music in
the emotion folder corresponding to the recognition
result will be played randomly using the method in the
random library in python. The next_song() method inthis module is also bound to the control next_btn, whichis also a method for randomly switching music.
The code first opens the path to the folder where theemotional music of the category is stored based on the
results of the face emotion recognition, searches thefolder for files ending in '.mp3', '.wav', '.ogg ' or '.m4a'
extension, saves the path to one of the files at random,and finally loads and plays the music using the methodin the pygame library.

Music download module
The music download module mainly uses python's
powerful crawler function to crawl music from music
websites. The module consists of three main parts:
music search, music crawling and music saving. Firstly,
the searchSong() method is used to send an HTTP GET
request to the url with the search content, and return alist of songs in JSON format to be prepared for themusic download.
Then the MusicDownload() method uses thereturned JSON list of songs to extract the song
information for the first song in the list, including thesong name, artist and song ID. Then the downloadSong()
method is called and passed the extracted song
information.
Finally, in the downloadSong() method, a request issent to the url consisting of the song ID and a responseis received. The music content is found in the JSON
format response, read and saved as an MP3 file in thelocal download folder, thus completing the download ofthe music and also preparing the music for classification.

Music playing module
The Music playing module is used to control theplaying and pausing of loaded music, and is
implemented by the methods provided in the pygame
library.
This method uses the Boolean variable ‘playing’
to determine if the music is playing and to control theplaying and pausing of the music. When the playing
variable is false, the music is not playing and can beplayed via the button, when the playing variable is true,
the music is already playing and can be paused via the
button.

Music classification module
In the music classification module, we will classify
the downloaded music according to the corresponding 4.
emotion and transfer the files to the corresponding
emotion folder. For this function we rely on a sentimentdictionary from [9], which assigns sentiment values toeach English word and the corresponding Chinese word
for eight basic emotions (anger, fear, expectation, trust,

Result and discussion
4.1. Under different light conditions
The tests were first carried out under different
lighting conditions and the results are shown in Figs. 8©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
569

Shengke Xie, Raenu Kolandaisamy, Ghassan Saleh, Heshalini Rajagopal

and 9. As can be seen from the above diagram, under
different lighting conditions, Fig. 8 shows the conditionof insufficient lighting and Fig. 9 shows the condition of
sufficient lighting, it can be seen from the figures thatthe function of the system expression recognition is notaffected by the lighting, but the accuracy of expression
recognition may be affected and reduced as a result.

Fig.11. Distance from camera 2m

Fig. 10 shows the results of the test at a distance of
0.3m from the camera and Fig. 11 shows the results ofthe test at a distance of 2m from the camera. After
testing, it was found that the recognition of faces within
two metres from the camera would not be affected;
however, once the face is more than two metres awayfrom the camera, the recognition will be affected andthe face may not be detected, so when using this system,
the user needs to be as close to the camera as possible sothat the face can be better captured for recognition.

Fig.8. Insufficient light

4.3. Recognition accuracy test
In this study, the 7 categories of "neutral", "surprise","happy", "angry", "sad", "fear" and "disgust" were madein turn as shown in Fig. 12. The first five expressions
matched the actual expressions, while the last two did
not match the actual expressions. The highest
recognition accuracy was achieved for the "happy" and"surprise" expressions, with over 90%, and for the"neutral", "sad" and "angry" expressions, with over 60%.
The intended "fear" expression was identified as "sad"and the intended "disgust" expression was identified as"fear". The reason for this may be, on the one hand, thelow recognition rate of these two expressions by themodel, and on the other hand, the ambiguity of these
two expressions, which could also be identified as "sad"and "fear" if judged by the human eye.

Fig.9. Sufficient light

4.2. Testing at different distances
The next test was conducted for different distances,with the face being closer to the camera and further
away. The test results are shown in Figs.10 and 11.

Fig.10. Distance from camera 0.3m

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024570

Development of a music

(a)

(b)

(c)

(d)

modules: image pre-processing, face detection, emotion
recognition, music download, music classification,
music recommendation and music playing. After thedesign was completed, the system was also tested in
actual use. The results showed that the system has ahigh accuracy rate of expression recognition and ishighly practical. However, the expression recognition
function in this system is still inadequate for theocclusion, side face and distance cases. At a later stage,
it is considered to add the expression images for theocclusion and side face cases to the existing dataset to
improve the face detection rate and thus the expression
recognition rate. In addition, the music classification
function of the system designed in this project is limited
to classifying music with lyrical content. In the future,
the authors will consider improving the musicclassification algorithm and adding the classification ofpure music without lyrics to the music classification
module of this system.

References
1.

(e)

(f)

2.

3.

(g)

4.

Fig. 12 (a) Neutral, (b) Surprised, (c) Happy, (d) Sad,(e) Anger, (f) Fear, (g) Disgusted expressions
5.

5.

Conclusion
Facial expression recognition has application needs inmany scenarios, and today more and more fields haveintroduced expression recognition technology, such as
the education field and the medical field. In order tomeet the requirements of a facial expression recognition
based music recommendation system, a convolutional
neural network model based on the FER-2013, JAFFE,
CK+ expression datasets was used in this project. A
music recommendation system based on facial emotion
recognition was successfully designed and implemented
in the PyCharm development environment using thepython programming language. It mainly includes seven

6.

7.

8.

S. Metilda Florence and M. Uma, ‘Emotional Detection
and Music Recommendation System based on User
Facial Expression’, IOP Conf. Ser.: Mater. Sci. Eng., vol.
912, no. 6, p. 062007, Aug. 2020, doi: 10.1088/1757899X/912/6/062007.
Song, Y., Dixon, S., & Pearce, M. (2012, June). A surveyof music recommendation systems and future
perspectives. In 9th international symposium on
computer music modeling and retrieval (Vol. 4, pp. 395410).
D. O. Melinte and L. Vladareanu, ‘Facial Expressions
Recognition for Human–Robot Interaction Using Deep
Convolutional Neural Networks with Rectified Adam
Optimizer’, Sensors, vol. 20, no. 8, Art. no. 8, Jan. 2020,
doi: 10.3390/s20082393.
K. Kaulard, D. W. Cunningham, H. H. Bülthoff, and C.
Wallraven, ‘The MPI Facial Expression Database — A
Validated Database of Emotional and Conversational
Facial Expressions’, PLOS ONE, vol. 7, no. 3, p. e32321,
Mar. 2012, doi: 10.1371/journal.pone.0032321.
B. C. Ko, ‘A Brief Review of Facial Emotion
Recognition Based on Visual Information’, Sensors, vol.
18, no. 2, Art. no. 2, Feb. 2018, doi: 10.3390/s18020401.
S. Du, Y. Tao, and A. M. Martinez, ‘Compound facial
expressions of emotion’, Proceedings of the National
Academy of Sciences, vol. 111, no. 15, pp. E1454–E1462,
Apr. 2014, doi: 10.1073/pnas.1322355111.
Kolandaisamy, R., Subaramaniam, K., & Jalil, A. B.
(2021, March). A Study on Comprehensive Risk Level
Analysis of IoT Attacks. In 2021 International
Conference on Artificial Intelligence and Smart Systems(ICAIS) (pp. 1391-1396). IEEE.
M. Lyons, S. Akamatsu, M. Kamachi, and J. Gyoba,‘Coding facial expressions with Gabor wavelets’, inProceedings Third IEEE International Conference on©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
571

Powered by TCPDF (www.tcpdf.org)

Shengke Xie, Raenu Kolandaisamy, Ghassan Saleh, Heshalini Rajagopal

9.

Automatic Face and Gesture Recognition, Apr. 1998, pp.
200–205. doi: 10.1109/AFGR.1998.670949.
https://github.com/luanshiyinyang/FacialExpressionReco
gnitionAuthors Introduction
Shengke Xie
He has completed his Bachelor ofComputer Science (Hons) Mobile
Computing and Networking, UCSI
University, Malaysia.

Dr. Raenu Kolandaisamy
He received his PhD from the Faculty ofComputer Science & Information
Technology, University Malaya in 2020.
He is currently an Assistant Professor in
UCSI University, Malaysia. His researchinterest areas are Wireless Networking,
Security, VANET and IoT.
Dr. Ghassan Saleh Hussein Al-Dharhani
He received his Ph.D. degree in
Computer Science from Universiti
Kebangsaan Malaysia (UKM). He iscurrently an Assistant Professor with theInstitute of Computer Science andDigital Innovation, UCSI University,
Malaysia. His research interests include Artificial
Intelligence, Data Mining and Knowledge Discovery, etc.
Dr. Heshalini Rajagopal
She received her PhD and Master's
degree from the Department of ElectricalEngineering, University of Malaya,
Malaysia in 2021 and 2016, respectively.
She received the B.E (Electrical) in 2013
Currently, she is an Assistant Professor
in UCSI University, Kuala Lumpur,
Malaysia. Her research interest includes image
processing, artificial intelligence and machine learning.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
572

A Model of Reaction-diffusion phenomena with Multiset Processing

Yasuhiro Suzuki

Nagoya University, Furocho, Chikusa-ku, Nagoya, 464-8601, Japan

Abstract
We propose a model of reaction-diffusion phenomena using Abstract Rewriting System on Multisets ARMS, which
is a model of Multiset Processing. Although proposed model is simple, computer simulations confirm that the Turing
pattern is generated.
Keywords: Reaction-Diffusion, Activator-Inhibitor, Multiset Processing , Abstract Rewriting System on Multisets,
ARMS

We model a system that performs activation andinhibition by diffusion coupling with two oscillators by amultiset rewriting system, Abstract Rewriting System on
Multisets, ARMS.

1. Introduction
A reaction-diffusion system is a mathematical model of
how the concentration of one or more substances
distributed in space changes under the influence of two
processes: local chemical reactions, in which substanceschange each other, and diffusion, in which substances
spread in space. One of the most famous reactiondiffusion systems is that proposed by Alan Turing.
Turing showed a system that is locally stable but
destabilized by diffusion. The spatio-temporal patternthat occurs in this system is called the Turing pattern [1].

2.1. Abstract Rewriting System on Multisets, ARMS
Abstract Rewriting System on Multisets, ARMS is amultiset rewriting system [2]. A multiset is defined as asimple set and a map, which returns the duplication ofelement. We denote the duplication (multiplicity) of anelement as M (a), for a ∈ A and in case c /∈ A, M (c) =0; for example M (a) and M (b) of {a, a, b, b} are 2, andM (c) = 0; in the mathematical description, a multiset isdescribed as; < sup, M () >, in which sup is a simple setof elements, in this paper we describe a multiset bydenoting the same alphabet in its number of multiplicity
such as {a, a, b, b} or a vector w =(M (a1) M (a2) . . . M(an)).

The reaction diffusion model, which Alan Turing
proposed is one in which two oscillators are coupled bydiffusion (Eq. 1), where the two oscillators mutuallyactivate and inhibit each other.
𝜕𝑢= 𝑓(𝑢, 𝑣) + 𝐷𝑢 ∇2 𝑢,
𝜕𝑡

The union of two multisets M1, M2 is the same as theunion of simple set and in vector description, the unionof multisets is addition of vectors w1 and w2. And
inclusion of sets is also the same as the simple set, when
M1(a) ≤ M2(a) for all a ∈ A, the multiset M1 is includedin M2 and we write M1 ⊆ M2.

(1)
𝜕𝑣= 𝑔(𝑢, 𝑣) + 𝐷𝑣 ∇2 𝑣,
𝜕𝑡
In the equation (1), 𝑢 ≡ 𝑢(𝒓, 𝑡), 𝑣 ≡ 𝑣(𝒓, 𝑡) , 𝑓, 𝑔 arereaction terms, 𝐷 is a diffusion coefficient and ∇2 is
Laplacian.

A reaction rule is a pair of multiset, we denote A# as aset of all combinations of multisets over A and in the
combinations, an empty multiset is included. A reaction
rule l → r, l, r ∈ A# is described as a pair of multiset
likewise chemical equations or a pair of its vector
expression; and in some case, we can describe a reaction

2. Methodology©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
573

Yasuhiro Suzuki

rule as a vector r, r = −l + r, it is simple and good for
examining the dynamics of an ARMS, but thisdescription can not illustrate when there are the same
species of element in the left-hand side and right-handside such as a, b → a, c; in this case l = (1, 1, 0) and r =(1, 0, 1) and r = −(1, 1, 0) + (1, 0, 1) = (0, −1, 1).

will eventually dissolve into the water. No matter how
long we keep looking at the surface of the water where
the ink has dissolved, there is no return of the drop of ink
when the dissolved ink is dropped on the surface of thewater again. In other words, diffusion is the process ofhomogenizing things.

A reaction is described as the rewriting of a multiset, if
the left-hand side of a reaction rule is included in a multiset, these elements in the multiset are excluded and theright-hand side of the rule is merged to the multiset; thecase when the multiset is a, a, b, b and the reaction rule isa, b → c, d, the left-hand side of the rule is included in
the multiset, {a, b} ⊆ {a, a, b, b} so the {a, b} is excluded
from the multiset and it is transformed to {a, b} and theleft-hand side of the rule {c, d} is merged to the set andwe obtain {a, b, c, d} by this reaction. By using vector
ex- pression, a reaction is the addition of vectors as w − l+ r,

If the state quantities of cells i and j are x_i and x_j, anddiffusion occurs between these cells, the amount ofdiffusion from cell j to cell i can be expressed as d=(x_j x_i)×D. D is called the diffusion coefficient. For example,
let us consider a sequence of cells [2,3,1,2]. (Thesequence of cells is indicated by enclosing the amount ofstate in each cell). For the cell with state 1, the state to theleft of this cell is 3, and the state to the right is 2. The stateof this cell is less voluminous than the left and right states.
Therefore, an inflow occurs from the left and right cells
to this cell, changing its state. The degree of inflow isdetermined by d=(x_j - x_i)×D. If the diffusion
coefficient D is 0.1, the inflow from the left neighbourd_left=(3-1)×0.1=0.2 and the inflow from the rightneighbour d_right=(2-1)×0.1=0.1. Thus, the sequence ofcells [2,3,1,2] becomes [2,2.8,1.3,1.9] due to this
diffusion. (Here, only the cell with state one is shown, but
in reality, other cells change in parallel as well.) In thisway, the state of each cell is averaged by diffusion.

in which l and r are vector expression of a reaction rule;the case when M = {a, a, b, b}, w = (2, 2, 0, 0) and thereaction rule is a, b → c, d, ({1, 1, 0, 0}, {0, 0, 1, 1}); andthe reaction is denoted as {2, 2, 0, 0} − {1, 1, 0, 0} ={1,1, 0, 0} and {1, 1, 0, 0} + {0, 0, 1, 1} = {1, 1, 1, 1}.
2.2. Activator-Inhibitor model
In the activator-inhibitor model, activator X increases
themselves by autocatalytic reaction (Eq. 2); X → X,
X. The reaction of activator X produces inhibitor Y is, X-> Y, X. X is put in both reaction equations because X
does not increase by this reaction, while only Y increases.
X, Y -> Y describes the inhibition of increasing X by Y,
where X decreases while Y does not change. Y → ∅denotes the decreases of Y, ∅ illustrates an empty set. We
give reaction coefficients for each rule. We give reaction
coefficients for each rule as k_1, k_2, k_3, k4. Eachcoefficient gives the probability of firing of the rule.

2.3 Diffusion in a two-variable system
The active/inhibited system of Eq (2) consists of two
variables, X and Y. Therefore, we will use a onedimensional sequence of X-only cells and a onedimensional sequence of Y-only cells. For example, if thesequence of cells representing the state of X is [2,3,1,2]
and the sequence of cells representing the state of Y is[0,0,1,0], the states of X and Y in the cell located second
from the leftmost are (1,1). When the diffusion
coefficient of X is 0.1, and that of Y is 0.3, if diffusion
occurs in cell (1,1), the state of the cell changes (Again,
only cell (1,1) change is shown).

𝑋 → 𝑋, 𝑋: r_1,
𝑘1

𝑋 → 𝑌, 𝑋: r_2,
𝑘2

3. Results and Discussion(2)
𝑋, 𝑌 → 𝑌: r_3,

We assume that the rightmost cell is connected to theleftmost cell; the ten cells are considered to be connected
in a ring. We show the result of simulation whose initialstates are;𝑘3

𝑌 → ∅: r_4.
𝑘1

Modelling diffusion：
X=[0,10,0,10,0,10,0,10,0,10,0,10],
The Eq. 2 above does not include diffusion. So, we needto consider diffusion separately. Why does diffusion
occur? If we put a drop of ink on clean water and watch
it for a long time, the ink spreading on the water's surface

Y=[0,10,0,10,0,10,0,10,0,10,0,10].

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
574

Powered by TCPDF (www.tcpdf.org)

A Model of Reaction-diffusion

4. Conclusion

We
set
the
reactioncoefficientsarek_1=k_2=k_3=k_4=0.01. The state quantity is first
updated by diffusion, as described in the previous section.
Next, reaction rules are applied in parallel to update thestate quantities. The update by diffusion and the update
by reaction rules are repeated, and this process is repeated.

The reaction-diffusion phenomenon has been modelled
and investigated as a partial differential equation bymodelling the activator-inhibitor system. We used
ARMS by modelling the activator-inhibitor system andconfirmed that the ARMS model shows Turing patternlike behavior.

When the diffusion coefficients of X and Y are
D_X=D_y=0.0, when there is no diffusion, there is nochange from the initial state. Such a state is called anequilibrium state (equilibrium means "balanced"). As
will be explained in detail in the next section, if X=Y (theamount of state of X is equal to the amount of state of Y),
neither the amount of state of X nor the amount of stateof Y will change in this reaction system. Therefore, since
X=Y holds in the corresponding cell in the initial state,
the system would remain in the initial state if there were
no diffusion.

Acknowledgements
This work was supported in part by KAKENHI
21K12108, 22H01444 and 23K06883.
References
1.

2.

What would happen if diffusion were to occur in a state
where the reaction is in equilibrium (X=Y)? First,
assuming that X and Y diffusion coefficients are the samewith D_X=D_y=0.1, X and Y change but always remain
X=Y. In other words, the reaction is always inequilibrium. Eventually, X and Y are homogenized bydiffusion, and the entire reaction reaches equilibrium.
This is true even if the diffusion coefficient is changed as
D_X=D_y=0.2,0.3⋯.

Turing, Alan Mathison. "The chemical basis ofmorphogenesis." Bulletin of mathematical biology 52(1990): 153-197.
Suzuki, Yasuhiro, and Hiroshi Tanaka. "Symbolic
chemical system based on abstract rewriting system and its
behavior pattern." Artificial Life and Robotics 1 (1997):
211-219.

Authors Introduction
Dr. Yasuhiro Suzuki
He received his Doctor degree inInformatics in 2001 from Graduate
SchoolofInformatics,
Kyoto
University, JAPAN. He is currently an
associate professor of Graduate Schoolof Informatics, Nagoya University,
JAPAN

Next, when X's diffusion coefficient is more significant
than Y's with D_x=0.3 and D_y=0.1, X and Y become
homogenized and almost X=Y, and the reaction
approaches equilibrium. On the other hand, when the
diffusion coefficient of Y is more significant than X's
with D_x=0.01 and D_y=0.3, Y becomes homogenized.
However, a larger or smaller pattern appears in X (Fig. 1)
shows an example of the results). In this case, X does notequal Y, and X and Y continue to change (increase).

Fig.1 Turing Pattern like behavior, where the diffusion
coefficient of X is 0.01, while Y, 0.1.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
575

Extract tactile qualities from time series data

Yasuhiro Suzuki

Graduate School of Informatics, Nagoya University, Furocho,
Chikusa, Nagoya 464-8601, Japan
E-mail: ysuzuki@i.nagoya-u.ac.jp
https://ysuzuki.info

Abstract
We proposed the Tactile Score to describe time-varying tactile sensation by the time variation of vertical force. Tactile
quality is essential in hardness/softness, roughness, and temperature. Hardness and softness can be extracted fromthe shape of the Tactile Score. Roughness can be extracted from the pattern of hardness and softness. The arbitrary
time series data can be interpreted as a tactile score by considering the time variation of the vertical force, and thehardness and softness are extracted from the time series data interpreted as the tactile score. This method can extract
different features from conventional data science methods.
Keywords: Data Science, Time series data, Tactile of Data, Tactile Score

method A method to quantify the impression received
from contents by assigning degrees to impression words,
learning correspondence between impression words andcontents, extracting correlation coefficients between
contents, and Extracting correlation coefficients, thedistance between contents Projecting the quantified
impression words and features of contents onto thefeature space and measuring the distance between them.

1. Introduction
Sensitivity differs from person to person. The same
greeting of "Good morning" may be perceived as
cheerful by some and noisy by others. There is no correct
answer to sensitivity, nor can it be generalized.
We can generalize if we take the average of many
people's sensitivities. Sensitivities that deviate from theaverage should not be ignored or directed towardsensitivities closer to the average.

2. Tactile Score
Why do we feel "cheer" from the children's "Goodmorning"? What is the difference between them? The
difference is "the way you say it. How we say it can becharacterized by the pitch and volume of our voice, but
let us look at the volume of our voice.

A general sensitivity search system is a system in which
evaluation criteria for content are modelled for eachindividual through instructional learning, and each user's
evaluation criteria model is used for searching.
The following algorithms have been used in a sensory
search; color histogram A method to extract features ofimages and videos; impression analysis using the SD©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
576

Yasuhiro Suzuki

When we think back to the children's "Good Morning,"
their voices gradually became louder and louder. So, let
us describe the change in "loudness" by using the music
notation. In musical notation, the higher the note is, thelower the note is, and the higher the note is, the lower thenote is. The middle line (the third line) is the "normal"
volume, and the lower the volume, the higher the volume,and the higher the volume, the lower the volume. The"rhythm of speech" is represented by the notes as they are.
We call this notation to describe tactile sense as theTactile Score [1] (Fig. 1).

3. Extract tactile qualities from time series data

Fig. 2 Roughness, read from Tactile Score
When we press a key, the piano makes a sound as it is. It
is impossible to adjust the sound later, as with a violin or
trumpet. So, are all piano performances the same? Of
course not. Even the same music can sound very different
depending on the pianist. One crucial element that makesa difference in piano playing is the "sense of touch. It is
often referred to as "the touch of the piano". The word
"touch" includes the meaning of "tactile sensation". The
sound of the piano, as this expression suggests, is thesound of the sense of touch.

Fig. 1 Tactile Score

2.1. Tactile Quality
There are several essential factors in the sense of touch,the most crucial being "hardness" and the second mostimportant being "roughness. These properties are called"tactile qualities. Is it possible to describe these tactile
qualities in tactile notation?

We examine the time series data of piano play andcompare Arthur Rubinstein [1887 - 1982] and Stanislav
Stanislavovich Bunin [1966-] playing Chopin's
Polonaisein A-flat major, Op.53 with WAV format file.
We split the data by 0.1 seconds each and used themean of the differences in the data as the hardness andthe standard deviation of the differences in the data as theroughness.

Think of two kinds of "springs": a stiff spring and a softspring. The softer spring will expand and contract morewhen these springs are pushed and pulled with the sameforce (difference in elastic force). Therefore, we call atactile spring with a slight change in size "hard" and atactile spring with a significant change in size "soft". If
we consider this as a way of saying "good morning,""hard" is a way of saying "no intonation," and "soft" is a
way of saying "full of intonation.

4. Discussion and Conclusion
Rubinstein is one of the foremost authorities on Chopin.
On the other hand, Bunin's style of playing is extremely
unique and different from Rubinstein's (Fig. 3).

Now we can describe the most critical tactile quality,"hardness," in tactile notation. What remains isroughness. Roughness is described as "smooth" when
there is little change in hardness and "rough" when there
is a significant change in hardness (Fig. 2). If we arewalking on the grass in a park and the hardness of thegrass is constant, we will feel "smooth".

Fig.3. Tactile qualities of piano play, above) R. Stein,bottom) S. S. Bunin, where the vertical axis illustrates
softness (the graph origin shows the hardest and far more
upper shows softer and the horizontal axis illustrates
roughness, the origin is the smoothest and far from originmeans rougher.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
577

Powered by TCPDF (www.tcpdf.org)

Extract tactile qualities from

Tactile quality analysis of sounds distinguished thedifferences in play style. In original music analysis, forexample, the power spectrum of sounds cannot clearlyshow the difference (Fig. 4).

Fig.4. Spectrum distribution of piano play, above) R.
Stein, bottom) S. S. Bunin, where the vertical axis
illustrates frequency, and the horizontal axis illustrates
Fig. 4 Tactile Query and result of retrieval; (Top)power.
The map of Tactile quality, “20181122-1152” is theReferences
tactile quality of inputted voice and others are
1. Y. Suzuki, R. Suzuki, Tactile score: a knowledge media
references.(Below) Japanese language in the middle,
for tactile sense. Springer Japan, 2014.
inputted language by voice (“it is a fine day and2. Kocyan, Wojciech. "The evolution of performance style in thevery good morning) and the system plays a music
history of the International Fryderyk Chopin Piano Competition
whose tactile quality is similar to the inputtedin Warsaw." Chopin Review 1 (2018): 20-37.
voiceF

Wojciech Kocyan is a Polish pianist. He is most wellknown for his performances of Chopin, and for his 2001
album, which was named as one of the 50 best classicalrecordings ever made by Gramophone in 2007. He
mentioned about the play style of Bunin.
Fig. 4areTactileQuery andretrieval;(Top)“therelaureatesof resultthe ofFirst
Prize
whose
The
map
of
Tactilequality,“20181122-1152”
theperformance style does not fit easily into whatis would
tactile quality of inputted voice and others areotherwise seem to be a quite clear picture of the stylistic
references.(Below) Japanese language in the middle,
preferences
Chopinby
Competition
are theinputted oflanguagevoice (“it isjuries.
a fine They
day and
Russianpianists
Stanislav
Bunin
(11th
very goodmorning)
and thesystemplaysCompetition,a music
1985)whose
and Yulianna
Avdeeva(16th Competition,2010).”
tactile quality
is similar
to the inputtedvoiceF(from [2] p32).

Authors Introduction
Dr. Yasuhiro Suzuki
He received his Doctor degree inInformatics in 2001 from Graduate
School of Informatics, Kyoto University,
JAPAN. He is currently an associate
professor of Graduate School ofInformatics, Nagoya University, JAPAN©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
578

Healthcare applications of vibrotactile stimulation developed by Tactile Score

Yasuhiro Suzuki

Nagoya University, Furocho, Chikusa-ku, Nagoya, 464-8601, Japan

Abstract
We constructed a system to convert the tactile score into vibratory tactile sensation. We conduct basic biologicalexperiments on electronic tactile stimuli and apply the results to medical and healthcare applications. We use two
types of electronic tactile stimuli. One is direct vibrating tactile stimulation, and the other is airborne vibrating tactile
stimulation. Verification experiments of this system have confirmed cosmetic effects such as skin collagenaggregation, improvement of hypertension and diabetes, and reduction of pain and nerve paralysis. In addition, aclinical study was conducted in which patients with dementia were exposed to vibrotactile sensation (low-frequency
sound) via air vibration and an improvement in cognitive function was confirmed.
Keywords: Healthcare, vibrotactile, Tactile scorethe sense of touch? There is a product that people haveused since time immemorial, across cultures and customs,
that is based solely on the sense of touch. It is the massage.
The mural paintings on the tombs of doctors from theEgypt period, depict a massage that is familiar to us today.
We may have yet to develop massage techniques for morethan 4000 years.

1. Introduction
Our hands have great power. There is a word in Japanesecalled, "Te-arte [1]". This word is the English term for"hand healing," or healing by laying one's hands on
someone. In Japan, hand healing has been valued sinceancient times.

2. Methodology
It seems which is not limited in Japan. Dr. Abraham
Varghese, a physician and professor at Stanford
University Medical School, gave a TED Talk in 2011
titled "A doctor's touch," in which he emphasized that,the importance of healing through the doctor's touch. He
addressed that:

We have studied massage and proposed a method fordescribing the sense of touch called the Tactile Score. In
our previous research, we found that tactile sensation can
only be expressed by the change in vertical force. So, we
decided to describe only the change in vertical force overtime. It is like describing music only in terms of pitchchanges.

"the most important innovation in medicine, to come in
the next 10 years, that is, the power of the human hand - to touch, to comfort, to diagnose and to bring about
treatment [2]”.

2.1. Tactile Score
Why do we feel "cheer" from the children's "Goodmorning"? What is the difference between them? The
difference is "the way you say it. How we say it can becharacterized by the pitch and volume of our voice, but
let us look at the volume of our voice.

The sense of touch is the earliest of the five senses todevelop, which is developmentally and biologically fully
formed at birth. A baby's sense of touch at birth, is said
to be superior to that of an adult.
However, the science and technology of the sense of
touch is, still in its developmental stage compared to theother senses. For example, the senses of audiovisual,smell and taste have their own "products". What about
touch? Can you think of some products based solely on©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
579

Yasuhiro Suzuki

When we think back to the children's "Good Morning,"
their voices gradually became louder and louder. So, let
us describe the change in "loudness" by using the music
notation. In musical notation, the higher the note is, thelower the note is, and the higher the note is, the lower thenote is. The middle line (the third line) is the "normal"
volume, and the lower the volume, the higher the volume,and the higher the volume, the lower the volume. The"rhythm of speech" is represented by the notes as they are.
We call this notation to describe tactile sense as theTactile Score [3] (Fig. 1)

transform a Tactile score into a vibrotactile. Since a
vibrotactile is a very low-frequency sound, we can record
it as a music file in WAV or MP3 format. Hence, we canexchange a vibrotactile translated from a Tactile score via
the Internet of electrical files.
By this Tactile score - vibrotactile translation, we cantransform tactile stimulation by human hands into
electrical data; this technique gives us the way to thedigital transformation of tactile sense, tactile DX.
We obtained a Tactile Score, which is converted tovibrations. In order to confirm that this transformation
works correctly, the biological responses of the vibrationtransformed from a Tactile score were compared to ahand massage using the same Tactile Score. Then, we
confirmed that the transformed vibration shows almost
the same biological response. So, we obtained a method
which allows a skilled technician's massage to beconverted to a vibratory massage using the Tactile Score(Fig. 2).

Fig. 1 Tactile Score
We had a method to convert tactile sensations into
vibrations but no general-purpose equipment to replay
the vibrations. Therefore, together with electrical andaudio equipment manufacturers, we developed a generalpurpose, face-shaped vibration presentation device (Fig.
3). This device allows users to receive massages by
distributing tactile sensations online. We are starting apilot service of internet delivery of vibration massage
around 2019, and more than 1,000 users are already using
the service.

2.2. Transform Tactile Score into Vibrotactile
Tactile score was for mainly hand massages. Hand
massage has long history, and its effectiveness has been
confirmed. However, its effectiveness would be affectedby skill of technician and even if well trained andexperienced technician, it is impossible to give massage
to more than two persons at the same time. And also, it isalso very difficult to confirm that the technician gives
exactly same tactile stimulations to patient.

Fig. 3 General-purpose vibrotactile display players;left) for face, right) for body.
With using this equipment, in order to verify theeffectiveness of the vibrating massage, we compared theskin condition before and after the treatment. We
examined that collagen fibers had agglomerated on theskin surface. It has known that the collagen fibers sinkfrom the surface of the skin due to ageing and otherfactors, but by applying transformed vibration, they roseto the surface of the skin.

Fig. 2 Transform Tactile Score into vibrotactile

To solve the difficulties, we transform the Tactile score
into vibrotactile. Since the Tactile score denotes thestrength of force, we transform the force into theamplitude of vibration. Hence, we found that applying
amplitude modification with the Tactile score can©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
580

Powered by TCPDF (www.tcpdf.org)

Healthcare applications of vibrotactile

3. Preliminary Results
We are now analyzing test data from users of our Internet
massage delivery system who visit hospitals for
hypertension and diabetes before and after receiving thevibration massage service. We have just started theanalysis and are surprised at the results. One user who
was taking medication for hypertension, had stopped
taking the user's hypertension medication. Because theuser's blood pressure became lower and felt terrible after
the user started the vibration massage. Now, the user's
blood pressure has stabilized in the normal range without
medication. We have confirmed several such cases. Since
blood pressure is subject to seasonal fluctuations, we willcontinue to monitor it over a long period.

References
1.
2.
3.

https://tearte.or.jp/archives/1076/(JapanTe-arteassociation web page, in Japanese)
https://www.ted.com/talks/abraham_verghese_a_doctor_s_touch?language=ja
Y. Suzuki and R. Suzuki, Tactile Score, Springer Verlag,
2014.

Authors Introduction
Dr. Yasuhiro Suzuki
He received his Doctor degree inInformatics in 2001 from Graduate
School of Informatics, Kyoto University,
JAPAN. He is currently an associate
professor of Graduate School ofInformatics, Nagoya University, JAPANIn addition, users who had high blood glucose levels
and were scheduled to apply insulin, have stabilized theirblood glucose levels at normal levels, after starting
vibration massage. We have monitored this user's blood
glucose levels regularly. We have observed that when theuser skips the use of vibratory massage, the blood glucose
levels rise again.
4. Conclusion
Our survey of past literature shows that when woundedsoldiers were transported in horse-drawn carriages forlong periods during the Greek period, their wounds
healed faster. Also, in the 18th century, it was confirmedthat patients with Parkinson's disease improved theirmotor functions when transported in a carriage for long
periods.
From the survey, our findings in preliminary results
would be a rediscovery of the wheel. Vibrotactiles had
been utilized in medicine before World War II and wereonce forgotten. However, in the 21st century, vibrotactile
has been rediscovered and applied in medical treatments;
at Tronto University, a vibration chair has been used totreat patients with dementia and Parkinson's, and also
vibroacoustic therapy has been used in northern Europe
or Germany.
The main difference between our method and others is
the Tactile score. Other methods have focused on soundor music, while we have been interested in tactile
stimulation. Hence, we have focused on how to design
the tactile stimulations.
Acknowledgements
This work was supported in part by KAKENHI
21K12108, 22H01444 and 23K06883.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
581

Image Collection Experiments of a Handy AUV for Offshore Structure Inspection
Keisuke Watanabe

Tokai University, 3-20-1 Orido, Shimizu-ku, Shizuoka, 424-8610, Japan
Koki Amano

Tokai University, 3-20-1 Orido, Shimizu-ku, Shizuoka, 424-8610, Japan
Shingen Urano

Tokai University, 3-20-1 Orido, Shimizu-ku, Shizuoka, 424-8610, Japan
Yasutaka Taniguchi

Tokai University, 3-20-1 Orido, Shimizu-ku, Shizuoka, 424-8610, Japan
Konosuke Watanabe

Tokai University, 3-20-1 Orido, Shimizu-ku, Shizuoka, 424-8610, Japan
Email: keisuke@tsc.u-tokai.ac.jp, 3ckgm001@mail.u-tokai.ac.jp

Abstract
In recent years, in Japan, with the decline in the working population, there has been a noticeable shortage of labor atconstruction sites and offshore operations. On the other hand, with the aim of realizing a GX society, it is expectedthat offshore wind power generation platforms will rapidly increase and be deployed offshore. Since it is not easy toaccess offshore platforms, it is necessary to automate equipment inspection work, and it is expected that the use of
AUVs will be particularly desirable for underwater inspection work. The most basic inspection work is assumed to
be to collect images of underwater structures. Therefore, in this study, we created an AUV and conducted an image
collection experiment using a pier owned by our university as an example to confirm its functionality.
Keywords: Offshore wind firm inspection, AUV, Underwater image

be easy. Generally, when maintaining offshore wind
power facilities, some specialized crews have to betransferred to the platform using a CTV (Crew Transfer
Vessel). If one platform can generate 10MW power, thenaround 4000 platforms must be installed to achieve the
40GW goal. In that case, it is not hard to imagine thatmaintenance costs will become huge because an operator
must hire CTVs which goes around those platforms. So
our concept is such that each platform should be equippedwith a unmanned inspection robotic system which
performs base inspection menus like taking images fromunderwater structures to up water windmill structures.

1. Introduction
In recent years, the worldwide development of windpower generation has been expanding and thedevelopment of offshore wind power is booming in someareas because the construction site of onshore wind
power is saturated. For example, in Europe, due tosaturation of onshore wind power sites, it is expected thatthe installation capacity of offshore wind firms willcontinue to increase, under the condition that theEuropean Union aims at reaching about 100 GW of
offshore wind capacity by 2030 [1].
In Japan, the goal of either reducing greenhouse gas
emissions to virtually zero or becoming carbon neutral by
2050 was set by the Cabinet recently. In 2020, thegovernment released its “Green Growth Strategy for
Carbon Neutrality by 2050”, which sets 14 action plantsthat Japanese industries should start making efforts torealize the net zero society. Under this strategy, wind
power is considered very important and set as one of the
14 action plans, with the goal of introducing 10 GW of
offshore wind power by 2030 and 30–45 GW by 2040.
Since Japan’s topography has few open plains, to achieve
this ambitious goal, accelerated development of offshore
wind firm is required [2] and many offshore platforms
will be installed off the coast of Japanese islands.

Image collection is one of the basic inspection menus
for offshore structures such as harbor structures [3].
Underwater structures are damaged by long-term use andmay be corroded by salty sea water. So, inspection
process should be performed to check the safety of thestructures. The regular safety inspection is very importantto prevent collapse accidents. The conventional
inspection of harbor structures used to be carried out by
professional divers. However, recently, the aging ofprofessional divers and the lack of successors havebecome a very critical issue in Japan. Also, while port
structure is shallower than approximately 20m, structureof an offshore wind platform like a semi-sub column islikely to be deeper than 30m, which makes divers limit
their activities in both time and space.

These offshore wind power platforms will be located
far from the shore and accessing many platforms will not

ROVs are widely used for offshore platform inspection[4], where it is assumed that an operator will operate the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
582

Keisuke Watanabe, Koki Amano, Shingen Urano, Yasutaka Taniguchi, Konosuke Watanabe

system by monitoring underwater camera images. In out
concept, the inspection procedure must be performed
unmanned, so even if the vehicle has a tether cable, thesystem should be developed as autonomous.
2. AUV System Development
2.1. System architecture
An external view of the developed AUV is shown in
Fig. 1. It has two pressure-tight vessels, one for thecontrol system and the other for the battery. A total of 6thrusters are installed, 2 each in the Surge, Sway, andHeave directions. The camera to capture images of thestructure is installed inside the acrylic window in front of
the upper hull and has a tilt mechanism. It is equippedwith sonar to detect obstacles in front and to the sides. A
DVL is attached to its bottom to measure altitude above
the ocean floor and speed over the ground. The battery is
stored in the lower hull, and buoyancy material is placed
above to improve self-stability in roll and pitch. ThisAUV is a prototype for developing various control
programs through experiments at sea, and was designed
to have an air weight of approximately 120N to make iteasier to handle on land. The dimensions of the AUV isshown in Fig. 2.
The architecture of the control system is shown in Fig.
3. This AUV is equipped with two microcomputers. One
computer was designed to mainly control input andoutput of sensors and actuators, and the other computer
was designed to control trajectory generation, control
mode switching, and image acquisition. It has both
wireless LAN and wired LAN, and when a tether cable isattached, it can be used so that it can be connected to a
USV or land-based computer and operated in ROV mode.
When surfacing and GNSS is available, it navigates using
GNSS coordinates, and when it starts diving, it uses deadreckoning from the GNSS coordinates just before diving.

Fig. 2. Dimension of the AUV

Fig. 3. Control system architecture

Fig. 4. The coordinate system for horizontal motion
2.2. Control algorithm
motion in the heave direction has little effect on the
motion in the horizontal direction. The coordinate system
is set as shown in Fig.4. The coordinate system is a spacefixed coordinate system (O-XY) based on latitude andlongitude, and a vehicle-fixed coordinate system (G-xy).
G is the vehicle center of gravity. The longitudinal
direction of the vehicle is the x-axis and the north is thex-axis. Let the angle formed by the X-axis and the x-axis
be the azimuth angle 𝜓. 𝜓 is positive when clockwisefrom the north. Space fixed Z axis and vehicle fixed z axis
is set as vertically downward to form the right-handed
system.

To implement the control algorithm, we assumed thatthe motion in the horizontal plane and the vertical motion
can be separated because when the velocity is small, the

The AUV's equation of motion includes a second-order
fluid force term and is nonlinear, but since the speed hereis slow, the fluid force is linearized by a representative
speed, and the linearly approximated equation of motion
is used as the dynamics model. Based on the linearly
approximated dynamics model, an optimal control

Fig. 1. External view of the AUV©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
583

Image Collection Experiments ofsystem of 1-type servo was designed and implemented.
Here, the system matrix depends on the coordinatetransformation matrix which consists of 𝜓 . Since thecoordinate transformation matrix changes according to
the motion of the vehicle, it is controlled by solving the
LQR problem for each step while updating the system
matrix at each control step[5], [6].
2.3. Sea experiment result
To verify all functions we designed and implemented,
we conducted a sea trial experiment at Tatsugo fishingport in Amami Oshima Island. The experiment site isshown in Fig.5. We set a trajectory along the quay of
Tatsugo fishing port as shown in the blue line in Fig.5,
and conducted an experiment in which we controlled thedepth and azimuth while following this trajectory. The
results of this experiment are shown from Figs.6 to Fig.9.
The following control modes as shown in Table 1 wereset for the trajectory tracking control, and the control
mode was changed when the error from the control target
value became less than the allowable value as shown inthe column of the transition condition.

Mode
0
1

2

3

4

5

Fig.6. DVL dead reckoning coordinate result

Table 1. Control modes set for the experiment.
TargetTransition condition
Azimuth 𝜓= 150 deg
Error of
(X,Y)<=1.0m
Depth = 0.5 m
Depth<=0.1m(X,Y)=(7.5cos 𝜓,
Then go to mode1
7.5sin 𝜓)
Error of
Azimuth 𝜓= 150 deg(X,Y)<=1.0m
Depth = 0.5 m
Depth<=0.1m(X,Y)=(15cos 𝜓,
Then go to mode2
15sin 𝜓)Error of
Azimuth 𝜓= -30 deg(X,Y)<=1.0m
Depth = 0.0 m
Azimuth<=10 deg(X,Y)=(15cos 𝜓,
Then go to mode3
15sin 𝜓)Error of
Azimuth 𝜓= -30 deg(X,Y)<=1.0m
Depth = 0.5 m
Depth<=0.1m(X,Y)=(7.5cos 𝜓,
Then go to mode4
7.5sin 𝜓)
Error of
Azimuth 𝜓= -30 deg(X,Y)<=1.0m
Depth = 0.5 m
Depth<=0.1m(X,Y)=(0,0)
Then go to mode5
Termination
Stop all thrusters

Fig.7. Azimuth angle control result

Fig.8. Depth control result (The depth sensor is
attached in the middle of the hull cylinder and theinitial depth when the AUV is floating is around
0.25m)

Tatsugo fishing port

Amami Oshima Island

Trajectory

Fig.5. AUV sea experiment site

Fig.9. Altitude from the sea bottom©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
584

Powered by TCPDF (www.tcpdf.org)

Keisuke Watanabe, Koki Amano, Shingen Urano, Yasutaka Taniguchi, Konosuke Watanabe

Authors Introduction
Dr. Keisuke Watanabe
He received his B.Engineering, M.E.,
ph.D in 1991,1993,1996,respectively
from the Department of Naval
Architecture and Ocean Engineering,
the University of Tokyo. He is a
Professor of the School of Marine
Science and Technology, Tokai
University in Japan. He is a member of
JASNAOE, MTS, IEEE etc..

Fig.10 Concrete image captured by the AUV

Mr. Koki Amano
He received his B.Oceanography in 2023
from the School of Marine Science andTechnology, Tokai University in Japan.
He is currently a master student in the
Graduate School of Marine Science andTechnology, Tokai University.

As shown in those Figures above, despite model
errors, all functions of the AUV operated smoothly,
and the control mission set as shown in Table 1 wassuccessful. Fig.10 shows an example of captured
images of the harbor concrete structure. Because thesea was highly transparent, we were able to obtain
relatively clear images. We plan to conduct further
research and improve AUV functionality.

Mr. Shingen UranoHe received his B.Oceanography in 2023
from the School of Marine Science andTechnology, Tokai University in Japan.
He is currently a master student in the
Graduate School of Marine Science andTechnology, Tokai University.

3. Conclusion
In this research, we developed an AUV platform andconducted trajectory tracking and image acquisition
experiments with the goal of unmanned inspection of
offshore wind power generation platforms. It was
confirmed that trajectory tracking, azimuth, and depth
control all functioned smoothly, and image acquisition
surveys could be carried out by setting a trajectory.

Mr. Yasutaka Taniguchi
He is currently an undergraduate student
in the School of Marine Science andTechnology, Tokai University.

References
1.

2.

3.

4.

5.

6.

E.P. Soares-Ramos, L. de Oliveira-Assis, R. Sarrias-Mena,
L.M. Fernández-Ramírez, “Current status and future
trends of offshore wind power in Europe”, Energy,
Vol.202, 2020, 117787.
K. Iwata, S. Kyoi, Y. Ushifusa, “Public attitudes ofoffshore wind energy in Japan: An empirical study using
choice experiments”, Cleaner Energy Systems, Vol.4,
2023, 100052.
J. Choi, Y. Lee, T. Kim, J. Jung and H. -T. Choi,"Development of a ROV for visual inspection of harbor
structures," 2017 IEEE Underwater Technology (UT),Busan, Korea (South), 2017, pp. 1-4
Capocci R, Dooly G, Omerdić E, Coleman J, Newe T, Toal
D. “Inspection-Class Remotely Operated Vehicles—A
Review”, Journal of Marine Science and Engineering.
2017; 5(1):13.
K. Watanabe, K. Amano, "Conceptual Design of
USV/UUV Combined Autonomous Platform for Offshore
Structure Inspection," 2023 IEEE International
Conference on Marine Artificial Intelligence and Law(ICMAIL), Taipei, Taiwan, 2023, pp. 1-6.
K. Watanabe, M. Shimpo, “Examination of Sea
Experimental System for Catamaran Type USV
Development”, The Journal of Japan Institute ofNavigation, 2022, Vol.146, pp.30-29(in Japanese)

Mr. Konosuke Watanabe
He is currently an undergraduate student
in the School of Marine Science andTechnology, Tokai University.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
585

Design of Disassembly-reassembly Type USV for Coral Reef Research
Keisuke Watanabe

Tokai University, 3-20-1 Orido, Shimizu-ku, Shizuoka, 424-8610, Japan
Koki Amano

Tokai University, 3-20-1 Orido, Shimizu-ku, Shizuoka, 424-8610, Japan
Gaku Minato

Tokai University, 3-20-1 Orido, Shimizu-ku, Shizuoka, 424-8610, Japan
Yasutaka Taniguchi

Tokai University, 3-20-1 Orido, Shimizu-ku, Shizuoka, 424-8610, Japan
Konosuke Watanabe

Tokai University, 3-20-1 Orido, Shimizu-ku, Shizuoka, 424-8610, Japan
Email: keisuke@tsc.u-tokai.ac.jp, 3ckgm001@mail.u-tokai.ac.jp

Abstract
Researchers are conducting physical surveys by diving and swimming to study the effects of microplastics and global
warming on coral reefs. The area, time, and water depth that can be investigated by diving are extremely limited. In
addition, to collect microplastics, it is necessary to tow the nets, but chartering a ship is expensive. Therefore, theauthors are developing a system that simultaneously operates a USV and a UUV to simultaneously observe the sea
surface and underwater. In this paper, we conducted a conceptual design, carried out fluid force measurements,
trajectory tracking experiments, and image recognition using AI, with the aim of realizing a lightweight USV that canbe divided during transportation to coral reef areas and reassembled on site for operation.
Keywords: USV/UUV combined, Disassembly-reassembly type USV, Coral reef monitoring, AI image processing

Additionally, by collecting small microplastics andidentifying them at the single particle level, it will bepossible to create some indicators regarding the potential
impact of microplastics on marine biodiversity and theirdistribution in the ocean. Currently, there are two
methods for collecting microplastics in coral reef waters:
chartering a ship and towing the nets, or having peopleswim and tow the nets. The former requires the ship to berouted from the port to the collection site, and costs suchas chartering fees are incurred. While the latter can beaccessed directly from the beach, the area in which it canbe retrieved is very limited.

1. Introduction
In recent years, marine plastic waste has been widely
recognized as a global environmental problem. Because
most plastics undergo very slow chemical or biologicaldegradation in the environment, debris can remain in theocean for years, decades, or even longer [1]. The term
microplastic refers to very small and ubiquitous plastic
particles less than 5 mm in diameter. They have been
separated into different fractions: large (1–5 mm) or
small (1 μm–1 mm) microplastics and the fraction below
20 μm (20 μm–1 μm) [2], [3]. Microplastics can adsorb
and carry hydrophobic chemicals that can have biologicaland toxicological effects on the environment [4].
Therefore, a clear understanding of the interactions ofsmall microplastics with the environment, especiallyliving organisms, is essential to assess potential healthhazards. To this end, it is important to have accurate
methods to quantify the mount of such particles in naturalenvironments. By monitoring the amount andcharacteristics of particles over time, along with thecoordinates from which they were collected, it will bepossible to define the extent of the effects they cause.

With the goal of improving these shortcomings andefficiently collecting more microplastics in coral reefwaters, we are developing a disassembly-reassembly type
USV, which we will introduce in this paper. The USV
under development will tow the net while automaticallyfollowing a set trajectory by a biologist to collectmicroplastics. The time and trajectory of microplastic
collection will be recorded by GNSS data. Since the work
is intended to be carried out even on remote islands, we
designed a structure that could be disassembled,transported by aircraft, etc., and reassembled on-site.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
586

Keisuke Watanabe, Koki Amano, Gaku Minato, Yasutaka Taniguchi, Konosuke Watanabe

2. USV System Development
2.1. System architecture
An external view of the developing USV is shown in
Fig.1. The dimensions of the USV is shown in Fig.2. As
shown in Fig.1, this USV consists of four hulls. The sizeof the hull was determined to fit within the typical
packaging size used in typical Japanese delivery services.
As shown in Fig.2, the thruster, computer, and GNSS
antenna are attached to the aluminum frame. The centerof the hull has a large opening so that the USV can collectmicroplastics while navigating, and a net for collecting
microplastics is attached to the rear of the USV. The
hardware components of control system is shown in Fig.3.
This USV is equipped with two microcomputers. One
computer is for motion control, and the other is for image
acquisition and processing. Image processing is supposed
to be used for collision avoidance when the tide is low
and the reef is exposed above the sea surface. As for thealgorithm of trajectory tracking, we hire optimal control
based 1-type servo [5].

Fig. 2. Dimensions of the USV

Fig. 3. Control system architecture

2.2. Drag force measurement of a hull
In order to estimate the maximum speed of the USV andconstruct a nominal model for designing the optimal
control system, it is necessary to estimate the drag force
coefficient. Therefore, as shown in Fig. 4, the hull was set
in a circulating water tank and the drag force wasmeasured by varying the flow velocity and angle of attack.
The flow velocity was increased every 0.2 m/s and wasmeasured from 0.4m/s up to 1.0 m/s. The measurement
results are shown in Fig.5. As the graph shows, it isalmost parabolic, confirming that the drag force is
proportional to the square of the flow velocity. The dragforce coefficient when moving forward is estimated using
the equation, where 𝐷 is drag force, 𝜌 is water density, 𝐴
is representative area of the hull, 𝑢 is velocity.
𝐶𝑑 = 1
2

𝐷
𝜌𝐴𝑢2

(1)
Fig. 4. Hull drag force measurement.

Fig.5 Drag force result
Fig. 1. External view of the USV

The drag force coefficient became 0.18 in average.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
587

Design of Disassembly-reassembly Type

3. Image Recognition Experiment Using AI
Since it is necessary to avoid obstacles during automaticnavigation, we are considering an image object
recognition function. We conducted an experiment using
YOLOv5 using our university's pier column as an
example. Fig.6 is an example of an image of the "pillar"
labeling that YOLO is trained to learn. About 100 photos
were taken to show the different shapes of the pillars on
the quay and the changes in the exposed parts of thepillars due to the ebb and flow of the tide, and were usedas training data. Fig.7 shows the results of image
recognition based on the above learning. It can be seenthat the pillars at the bottom of the quay could berecognized, but the probability of recognition was only
about 60%, and when sailing from an angle, it was often
not possible to recognize them. Fig.8 shows the results oflearning by adding additional training data, including
images from diagonal directions. The recognition rate
was over 80%, and the pillars could now be recognizedeven in images viewed from an angle. Image recognition
using AI is extremely important as a function of USV,
and in addition to detecting obstacles, it may also be
applied to recognizing coral reefs using underwater
cameras in coral reef surveys. YOLO is a very powerful
tool, but it is necessary to prepare various images as
training data, and how to obtain images for learning at thefirst site is a future issue. One possible solution would be
to fly a UAV (Unmanned Aerial Vehicle) in the researcharea and use the images captured by the UAV as learning
data. Floating debris such as large pieces of wood orplastic bottles can break the microplastic mesh if the USV
swims over them, so it must have a function to find themahead and avoid them.

Fig.8 Improved recognition result after more learning
4. Trajectory Tracking Experiment
In order to confirm the resolution of GNSS and confirm
the trajectory tracking performance, we conducted a
trajectory tracking experiment using a catamaran-type
USV that has been developed in our laboratory, rather
than the USV currently under development. The results
are shown from Fig. 9 to Fig.11. Fig.9 is a map of the
trajectory in the actual ocean area. Fig.10 shows theresults of thrust forces. Fig.11 shows converting thelatitude and longitude to the XY coordinate system withthe starting point as the origin. It was set to sail to the (10,10) point and then return to the origin. On the day ofthe experiment, the wind was strong and there was somedeviation from the straight trajectory, but we were able toconfirm that it had sufficient performance to collectmicroplastics.

Fig.6 An image example for pillar labelingFig.9 Trajectory result

Fig.7 An example of image recognition results

Fig.10 Thruster forces©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
588

Powered by TCPDF (www.tcpdf.org)

Keisuke Watanabe, Koki Amano, Gaku Minato, Yasutaka Taniguchi, Konosuke Watanabe

4.

5.

surface water samples of the subtropical island of Okinawa,
Japan”, Science of The Total Environment, 2021, Vol. 760,
143927
Richard E. Engler, “The complex interaction between
marine debris and toxic chemicals in the ocean”, Environ.
Sci. Technol., 46 (22) (2012), pp. 12302-12315
K. Watanabe, K. Amano, "Conceptual Design of
USV/UUV Combined Autonomous Platform for Offshore
Structure Inspection," 2023 IEEE International
Conference on Marine Artificial Intelligence and Law(ICMAIL), Taipei, Taiwan, 2023, pp. 1-6.

Authors Introduction
Dr. Keisuke Watanabe
He received his B.Engineering, M.E.,
ph.D in 1991,1993,1996,respectively
from the Department of Naval
Architecture and Ocean Engineering, theUniversity of Tokyo. He is a Professor of
the School of Marine Science andTechnology, Tokai University in Japan.
He is a member of JASNAOE, MTS,IEEE etc..

Fig.11 XY coordinate of the trajectory

Mr. Koki Amano
He received his B.Oceanography in 2023
from the School of Marine Science andTechnology, Tokai University in Japan.
He is currently a master student in the
Graduate School of Marine Science andTechnology, Tokai University.

Fig.12 The total image of the USV
5. Summary

Mr. Gaku Minato
He is currently an undergraduate student
in the School of Marine Science andTechnology, Tokai University.

In this study, we designed a USV with the goal of
automating and reducing the cost of microplastic
collection in coral reef waters, which is currently done
manually. A split-assembly type USV was designed tofacilitate transportation to remote islands. The conceptual
design drawing is shown in Fig.12. In order to realize the
obstacle avoidance function through image processing,
we implemented an object detection system using AI. In
order to construct a nominal model for control systemdesign, the hull's fluid force coefficient was identified
through experiments using a circulating water tank.
Through trajectory tracking experiments using existing
USV, we have confirmed that trajectory tracking ispossible within an error range of approximately 1m. We
plan to conduct microplastic collection experiments.

Mr. Yasutaka Taniguchi
He is currently an undergraduate student
in the School of Marine Science andTechnology, Tokai University.

Mr. Konosuke Watanabe

References
1.

2.

3.

He is currently an undergraduate student
in the School of Marine Science andTechnology, Tokai University.

Anthony L. Andrady, “Microplastics in the marine
environment”, Mar. Pollut. Bull., 62 (8) (2011), pp. 15961605
Julien Gigault, Alexandra ter Halle, Magalie Baudrimont,et.al., “Current opinion: what is a nanoplastic?”, Environ.
Pollut., 235 (2018), pp. 1030-1034
Christina Ripken, Domna G. Kotsifaki, Síle Nic
Chormaic, ”Analysis of small microplastics in coastal©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
589

Optimization method to improve visual SLAM in dynamic environment
Yufei Liu

Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan
Kazuo Ishii

Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan
Email: liu.yufei124@mail.kyutech.jp, ishii@brain.kyutech.ac.jp

Abstract
Aiming at the problem that the robustness of the classic visual SLAM system is greatly affected by dynamic target
feature points in the environment, a method is proposed to use a target detection algorithm to identify and eliminate
dynamic target feature points. First, use the target detection algorithm YOLOv5 to identify the collectedenvironmental images, and select the surrounding environment. Objects identified as dynamic targets in theenvironment, and then the target detection results are integrated into the feature extraction of the visual SLAM frontend, the feature points belonging to the dynamic target part of the extracted image feature points are removed, andthe remaining static feature points are used to map Construction and positioning, and finally testing on the TUM data
set. The results show that after using the target detection algorithm to eliminate dynamic feature points, the root meansquare error of the absolute trajectory error of the visual SLAM system in highly dynamic scenes is reduced by 97.89%,
effectively improving the positioning accuracy and robustness of the system.
Keywords: Visual SLAM, Deep learning, Feature detection, Position estimation, YOLOv5

2. Problem Description

1. Introduction

2.1. Overview of visual SLAM technology
With the development of autonomous mobile robot
platforms, robots are widely used in areas such as searchand rescue operations or delivery services required atindustrial sites and hotels. In these scenarios, the robot
needs to understand the entire area and the precise
location of the target object in the map based on the map
to complete autonomous navigation. In order to achieveautonomous navigation, mobile robots need to complete
two tasks: attitude estimation and map construction.
Therefore, Simultaneous Localization and Mapping(SLAM) is one of the research hotspots in the field of
mobile robots. It is a robot that only relies on its own
sensors to obtain external information of the unknown
environment to complete pose estimation and build anenvironment model. Technology. This technology iscurrently widely used in agriculture, disaster relief,
auxiliary medical and other robotic fields. According to
the different sensors used, SLAM is divided into visual
SLAM and laser SLAM. Visual SLAM uses cameras
mounted on the robot as sensors, including monocular,
binocular and depth cameras. Compared with lidar,
cameras are cheaper and more convenient to install anddebug. Moreover, cameras can collect more semantic
information in the environment. Therefore, visual SLAM
has gradually become a major research hotspot in thefield of autonomous navigation of mobile robots.

In recent years, various visual SLAM (Simultaneous
Localization and Mapping) techniques have emerged,
such as ORB-SLAM, ORB-SLAM2, VINS Mono,
RTABMap, PTAM, LSD-SLAM, DSO, among others,playing a crucial role in the field of autonomous robot
navigation [1]. The general algorithmic process of visual
SLAM begins with the acquisition of images using acamera to generate a sequence of images. Subsequently,
features are extracted from adjacent frames, and the
motion of the camera is estimated by minimizing pixel
intensity values through feature point matching. Pixelmatching methods between images include optical flow
and direct methods. Optical flow involves extracting
image feature points and estimating camera motion using
triangulation or Epipolar Geometry. Direct methods, on
the other hand, directly use pixel blocks or extract image
corner points to calculate motion estimation based ongrayscale values.
Following the motion estimation, noise is filtered fromthe data to obtain the optimal pose estimation. Maximum
a posteriori probability is then used to estimate the global
map. Campos et al. [2], building on the research by Artal
et al., released ORB-SLAM3 in 2020, which is a featurebased SLAM system and currently represents a notableapproach in the field.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
590

Yufei Liu, Kazuo Ishii

2.2. Impact of Dynamic Points on Visual SLAM

The underlying principle of these methods assumes thatthe feature points of dynamic objects deviate from thestandard constraints obtained in a static scene throughprocesses such as triangulation, fundamental matrix
estimation, epipolar lines, and reprojection error. During
the pose estimation process, these dynamic feature points
are treated as outliers. The correctness of feature
matching can be determined by examining whether the
extracted feature points violate these constraints, anddynamic points are subsequently excluded. However, theaccuracy of this method depends on the proportion of
static feature points in the scene. In scenarios with a highdensity of dynamic objects, it can significantly impact thereliability of pose estimation and the accuracy of map
construction.

ORB-SLAM3, as one of the classic algorithms in visual
SLAM, consists of two main components in its
algorithmic pipeline: the front end and the back end. The
front end, also known as visual odometry, processes
captured images through feature extraction and matching.
It solves the Epipolar Geometry relationship between
corresponding feature pixels, thereby estimating the poseparameters of camera translation and rotation. The backend includes nonlinear optimization, map estimation, andloop closure detection. Nonlinear optimization, differingfrom the previous use of Kalman filtering, employs
Bundle Adjustment (BA) to simultaneously optimize thesix degrees of freedom of camera pose parameters and theposes of landmarks in space.

3.2. Dynamic SLAM based on deep learning
The effectiveness of the Bundle Adjustment (BA)method in optimizing camera pose results depends onwhether the extracted matching feature points from thecaptured images exclusively static object features are.
However, real-world robot operating environments often
include dynamic objects. In scenarios with dynamic
targets, both feature-based SLAM algorithms and direct
methods SLAM algorithms struggle to differentiate
between feature types in regions with moving objects.
The matching points of dynamic target feature points canlead to misalignments, resulting in reduced accuracy inthe front-end visual odometry's estimation of camera
pose. This, in turn, causes a loss of camera pose tracking,leading to significant deviations between the robot's
motion trajectory and the constructed environmental map.

With the development of deep learning, many
researchers have introduced deep learning algorithms
into visual SLAM to mitigate the impact of dynamic
targets and improve its accuracy. The DynaSLAM(Dynamic SLAM) algorithm proposed by Berta Bescos
et al. [6] optimizes ORB-SLAM2 using Mask R-CNN(Region-based Convolutional Neural Network). By
combining geometry with deep learning, DynaSLAM
filters out dynamic feature points in the scene. The
algorithm has shown excellent results on the TUM
dataset, but its use of Mask R-CNN for pixel
segmentation leads to low real-time detection efficiency,
limiting its practical application in real-world
environments. DDL-SLAM (Dynamic Deep Learning
SLAM) [7] uses DUNet (Deformable Unity Networking)and semantic masks obtained through multi-viewgeometry to detect dynamic objects. It then employs animage restoration strategy to reconstruct the backgroundobscured by dynamic objects. Since the calculation of
dynamic object masks occurs at the pixel level, thismethod also falls short of achieving real-time
performance. In contrast to Mask R-CNN, another more
efficient object detection model is YOLOv5 (You Only
Look Once Version 5) [8]. YOLOv5 achieves detection
speeds of up to 45-155 frames per second (fps), which is9-30 times faster than the maximum 5 fps achieved by
Mask R-CNN. If the results of YOLOv5's object
detection can be integrated into dynamic visual SLAM
algorithms, it can partially compensate for the low
efficiency of Mask R-CNN.

3. Elimination of the dynamic points
3.1. Dynamic SLAM based on geometric methods
Currently, there are two main methods for mitigating
the impact of dynamic target feature points in detection
scenes. One approach is the traditional geometric-based
method. Sun et al. [3] detect moving objects bycomparing differences between adjacent frames, but thismethod suffers from poor real-time performance. Wanget al. [4] filter matching feature points in adjacent frames
using Epipolar Geometry. They also perform clustering
on depth images collected by RGB-D cameras to identify
independent dynamic objects in the scene. The accuracyof this algorithm is significantly influenced by the posetransformation matrix between adjacent frames. If there
are many highly dynamic objects in the scene, it can leadto substantial algorithmic errors. Lin et al. [5] use a fusionof image depth information and visual ranging to detectmoving objects. While this approach can determine theposition of moving targets in the scene, the algorithm's
accuracy is compromised due to the uncertainty of depth
information and cumulative errors in calculating thetransformation matrix between adjacent frames.

To address the issue of reduced localization accuracy
and robustness of visual SLAM in dynamic environments
due to the influence of dynamic targets, this paper
introduces a target detection thread based on the ORBSLAM3 algorithm to detect dynamic targets. Using the
YOLOv5 target detection network to identify objects in
input images, the semantic information of dynamic target
objects in the images is determined. Additionally, whilevisual SLAM extracts feature points in the front end, a
module is added to the tracking thread of visual SLAM toeliminate dynamic feature points based on the results of©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
591

Optimization method to improvetarget detection. Finally, only the remaining static featurepoints are used to estimate the pose change matrix
between adjacent frames, reducing the impact of dynamic
targets. This ultimately achieves high-precision
estimation and localization of the environmental
trajectory. The improved algorithmic flowchart is shownin Fig.1.
4. Experimental verification and result analysis
4.1. Experimental environment construction and data
set selection
Fig.1 Algorithm framework
4.3. Experimental data indicator analysis

his experiment involved testing the optimized visual
SLAM system proposed in this paper using video
sequences from the TUM dataset. The experimental
results were analyzed to evaluate the localization
accuracy of the SLAM system. The experiments wereconducted on the Ubuntu 20.04 operating system, with a
12th Gen Intel(R) Core(TM) i9-12900H 2.50GHz CPU,an NVIDIA GeForce RTX 3060 GPU with 12GB of
VRAM, and the PyTorch deep learning framework. The
algorithm's performance was tested on the
fr3_walking_xyz,
fr3_walking_half,and
fr3_walking_static
datasetsequences.
The
fr3_walking_xyz dataset depicts two individuals walking
and conversing in a fixed scene, with both the camera andpeople in motion, representing a high-dynamic scene.
The fr3_walking_half dataset builds upon this by having
the camera move along a semi-circular trajectory in the
air. The fr3_walking_static dataset, on the other hand,features relatively stationary objects, representing a lowdynamic scene.

a.

Fusion algorithm

4.2. Visual SLAM front-end feature extraction effectafter integrating YOLOv5
Capturing a frame for comparison with the original
ORB-SLAM3 algorithm's front-end feature extraction,
the results are illustrated in Fig. 2. The lower image
shows the feature extraction before integrating YOLOv5,
while the upper image demonstrates the effect after
fusion. It is evident that the visual SLAM front end, after
integrating YOLOv5, accurately identifies objects suchas people and computers. It successfully removes featurepoints on the dynamic target "person" while retaining
feature points on the static object "computer" within thedetected bounding box of the dynamic target. This
refinement enhances the precision of the results.
b. Original algorithm
Fig.2. Comparison of front-end feature extraction
results before and after integrating YOLOv5
Absolute Trajectory Error (ATE) describes thedifference between estimated poses and ground truth
poses, providing an intuitive expression of the algorithm's©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
592

Powered by TCPDF (www.tcpdf.org)

Yufei Liu, Kazuo Ishii

2.

global performance and accuracy. Relative Pose Error(RPE) calculates the differences in pose at the sametimestamps, typically used for estimating odometryerrors. The Root Mean Square Error (RMSE) is thenemployed to compute the overall value of this error. ATE
and RPE are two parameters that reflect the robustness
and stability of a visual SLAM system [9]. A lower
RMSE value calculated from ATE and RPE indicates a
better fitting performance.

3.

4.

5.

Capturing a frame for comparison with the original
ORB-SLAM3 algorithm's front-end feature extraction,
the results are illustrated in Fig. 2. The lower image
shows the feature extraction before integrating YOLOv5,
while the upper image demonstrates the effect after
fusion. It is evident that the visual SLAM front end, after
integrating YOLOv5, accurately identifies objects suchas people and computers. It successfully removes featurepoints on the dynamic target "person" while retaining
feature points on the static object "computer" within thedetected bounding box of the dynamic target. This
refinement enhances the precision of the results. The
improvement of the ATE and RPE performance of thisalgorithm compared with the original ORB-SLAM3
algorithm is shown in Table.1 and Table.2.

6.

7.

8.

9.

Campos C, Elvira R, Rodríguez J J G, et al. Orb-slam3:
An accurate open-source library for visual, visual–inertial,
and multimap slam [J]. IEEE Transactions on Robotics,
2021, 37(6): 1874-1890.
Sun Y, Liu M, Meng M Q H. Improving RGB-D SLAM in
dynamic environments: A motion removal approach[J].
Robotics and Autonomous Systems, 2017, 89: 110-122.
Wang R, Wan W, Wang Y, et al. A new RGB-D SLAM
method with moving object detection for dynamic indoor
scenes[J]. Remote Sensing, 2019, 11(10):1143.
Lin S F, Huang S H. Moving object detection from amoving stereo camera via depth information and visual
odometry[C]//2018 IEEE International Conference on
Applied System Invention (ICASI). IEEE, 2018:437-440.
Bescos B, Fácil J M, Civera J, et al. DynaSLAM: Tracking,mapping, and inpainting in dynamic scenes[J]. IEEE
Robotics and Automation Letters, 2018, 3(4) : 4076-4083.
Ai Y, Rui T, Lu M, et al. DDL-SLAM: A robust RGB-D
SLAM in dynamic environments combined with deeplearning[J]. IEEE Access, 2020, 8: 162335-162342.
Redmon J, Divvala S, Girshick R, et al. You only lookonce: Unified, real-time object detection[C]//Proceedingsof the IEEE conference on computer vision and patternrecognition. 2016:779-788.
Chang Z, Wu H, Sun Y, et al. RGB-D Visual SLAM Basedon Yolov4-Tiny in Indoor Dynamic Environment[J].
Micromachines, 2022, 13(2):230.

Authors Introduction
Ms. Yufei Liu
She graduated from Zhejiang Normal
University in China with a bachelor's
degree in 2017. Now she is studying
as a master's student at Kyushu
Institute of Technology in Japan.

Table.1 ATE performance comparison
Dr. Kazuo Ishii
He received his PhD from the
University of Tokyo, Japan, in 1996.
In 2011, he joined Kyushu Institute ofTechnology and is currently aprofessor in the Department of
Human Intelligent Systems. His
research interests include information
communications and marine robotics.
He is a member of IEEE.

Table.2 RPE performance comparison
This indicates that the proposed algorithm not only
enhances the detection capability in dynamic
environments but also preserves the accuracy of map
construction and localization in static scenes, as achievedby the original ORB-SLAM3 algorithm.
References1.

Zhaopeng G, Liu H, University P, et al. A survey of
monocular simultaneous localization and mapping[J].
CAAI Transactions on Intelligent Systems, 2015.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
593

Design of flexible mechanism for flexible manipulator
Huang Jiawei

Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan
Kazuo Ishii

Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan
Email: huang.jiawei311@mail.kyutech.jp, ishiii@brain.kyutech.ac.jp

Abstract
Based on the study of the kinematic limitations of the rigid manipulator structure and the characteristics of the existingflexible manipulator in different categories, this paper proposes a structure assumption of flexible manipulator basedon the advantages of high load of rigid joint and high flexibility of flexible manipulator. This paper designed a flexible
manipulator structure driven by wires. Using the forward kinematics analysis, the sport model of the structure. The
motion range of the end of the manipulator arm and the length changes of the wires were simulated with the movementof the model.
Keywords: Flexible manipulator, Wires driven, Kinematic model

SMA, full of Shape Memory Alloys, can be heatedto eliminate the deformation of alloys at a lower
temperature, making it revert to the initial state andachieving the effect of "memory". It can be driven bydirect heating of current, and its deformation can becontrolled by heating and cooling. It has been widelyused in aerospace and industrial intelligent
manufacturing. As shown in the Fig.1, the manipulator
can realize large spatial bending and complete thegrasping action of different positions.

1. Introduction
With the development of science and technology, rigid
robot devices have been widely used in social production
activities. It is well known that the industrial robot isapplied in the manufacturing industry. The rigid
manipulator consists of a rigid connecting rod connectedby a rigid motion pair, which can quickly and accurately
complete the corresponding control tasks [1]. However,
due to the rigid structure of the manipulator, its flexibility
is poor, and it cannot work properly in the complex andchangeable environment. Its structural characteristics
limit its application in dangerous and complex
environments, such as exploration and disaster relief,
nuclear power hazard management, space missions andother environments. If the manipulator is to be applied tothe above engineering fields, it needs to meet therequirements of safe interaction between the machine andthe environment, safe human-computer interaction, highflexibility and intelligence. The flexible manipulator ismade of soft materials. Compared with the rigid robot,
the manipulator has the characteristics of multiple
redundant degrees of freedom, flexibility and safe
human-computer interaction. Its characteristics greatly
make up for the shortcomings of rigid robots, so it has
been widely concerned and studied by scholars andinstitutions [2].

Fig.1 SMA flexible mechanical arm [3]2.2. Pneumatic-hydraulic driven
Pneumatic or hydraulic driven respectively use
compressed air and hydraulic oil as the power source ofthe mechanical arm bending. Pneumatic - hydraulicdriven manipulator is usually made of elastic material as
the main body, surrounded by a driving cavity. The
bending of the manipulator can be realized by changing
the volume deformation of the driving cavity by changing
the input of the power source as showed in Fig.2. Whenthe inner rubber air bag is filled with compressed air, theinternal pressure rises, the inner rubber air bag expands
along the radial direction, and then the force transmission

2. Classification of prior research
The development status of flexible manipulator isintroduced according to the driving mode classification.
The flexible manipulator is divided into the following
driving modes: SMA driven, pneumatic - hydraulicdriven, and wire driven, etc.

2.1. SMA driven©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
594

Huang Jiawei, Kazuo Ishii

effect of the outer fiber is transformed into the axial
contraction force to achieve the driving effect.

of the flexible manipulator. The flexible column is nonstretchable, incompressible and can be twisted arbitrarily.
Bend the flexible column by tensing and relaxing thethree driving wires to tilt the support plate. The outer partis covered with a spring to protect the flexible
manipulator. The single-section mechanical arm is shownin Fig. 4. Integrated model scheme is showed in Fig.5.

Fig.2 Pneumatic driven flexible manipulator [4]2.3. Wire driven
Wire driven is a popular driving method for flexible
manipulator. The bending motion of the flexible arm isgenerally realized by the stretching of the wire driven by
the motor. The wire driven flexible manipulator is driven
by the wires embedded in the interior of the flexible
manipulator to realize the deformation movement of theflexible manipulator as showed in Fig.3. Usually,mechanical equipment such as motor is used to generate
traction on the wire. The wire embedded in the flexible
manipulator with eccentric eccentricity generates axial
force on the manipulator while generating bending
moment on the neutral surface, making the flexible
bending motion of the manipulator.

a.

b.

A single segment robotic arm in an uprightposition

A single segment robotic arm in a bent position
Fig.4 Single segment robotic arm

Fig.3 Wire driven flexible manipulator [5]3. Model scheme
The flexible robot arm is composed of three layersof single-section robot arm, the height of the singlesection robot arm is 200mm, and the height of the entire
robot arm is 600mm. The central flexible column is a soft
plastic material with good flexibility and bearing capacity.
The hardness and elastic modulus of soft plastics can becontrolled by additives to adapt to different application
scenarios. The flexible column is used as the main body
©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
595

Design of flexible mechanism

α and curvature radius R to represent the correspondingarc length and chord length of the projection of the linefrom the anchor point of the lower support disk to thecurvature radius of the single-section mechanical arm, so
as to obtain the array of bending angle θ and rotation
angle α and curvature radius R represented by the lengthof the driving wires .The mapping relationship fromdriving space to configure space is Eq.(1).
√3(H1 − H2 )H1 + H2 − H32
√𝐻12 + 𝐻22 +𝐻32 − H1 H2 − H2 H3 − H3 H1

𝛼 = 𝑎𝑟𝑐𝑡𝑎𝑛
𝜃 = 𝑎𝑟𝑐𝑠𝑖𝑛
{

R=

6𝑟
(H1 + H2 − H3 )𝑟(1)

2√𝐻12 + 𝐻22 +𝐻32 − H1 H2 − H2 H3 − H3 H1

4.2. The mapping relationship from
configuration space to task space
Solving the mapping relationship between
configuration space and task space that is thehomogeneous transformation matrix from local reference
frame to global reference frame is solved. Matrix
elements are composed of configuration variables α、θ
and R. The solution is to use the D-H method, and thetransformation relationship between the coordinate
system from the lower support disk to the upper support
disk is regarded as a series of sequential rotation andtranslation results.
D-H method (Denavit-Hartenberg method) is amatrix general method for establishing the relative
position and attitude relationship of serial robots. It uses
four parameters of connecting link length, joint angle,
link offset and link twist to describe the structure andkinematic relationship of the series mechanism. Thesefour parameters are collectively referred to as D-H
parameters. The spatial geometric relation of each link
relative to the fixed reference coordinate system is
described by homogeneous transformation, and thespatial geometric relation between two adjacent links is
expressed by homogeneous transformation matrix. The
equivalent homogeneous coordinate transformation
matrix of the end actuator coordinate system relative tothe base coordinate system can be derived by thehomogeneous transformation, and the kinematics
equation of the robot can be established [7]. Generation
steps of homogeneous transformation matrix is showed
in Fig.7.

Fig. 5. Integrated model scheme

4. Kinematic analysis of model
The spatial mapping relationship of the robot arm isshown in Fig.6. This paper analyzes the kinematics modelof the flexible manipulator through forward kinematics.
The forward kinematics analysis of the manipulator arm
is to calculate the end pose of the manipulator arm
according to the information of the driving space.

Fig.6 Spatial mapping of kinematic [6]4.1. The mapping relationship from driving space
to configuration space
According to the length variation of three wires, thebending angle θ and the rotating angle α and radius ofcurvature R of the mechanical arm are solved bygeometric method. The solution method is to establish aset of equations by using bending angle θ, rotating angle©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
596

Huang Jiawei, Kazuo Ishii

the continuous flexible body is not subjected to external
forces, its initial state is reconnected with the Z-axis. The
angle between the projection of arc OO1 on the X-O-Y
plane and the positive direction of the Y-axis is set as α,and the angle between string d and the positive directionof the Z-axis is set as θ.
The free end point O1 of the center line of theflexible arm is obtained. The expression in the coordinate
system is Eq.5. The bending Angle is 0 to 90° and therotation Angle is 0 to 360°.
x=𝑦={Fig. 7 Generation steps of homogeneoustransformation matrix

𝐿

2θ
𝐿

2θ

(1 − 𝑐𝑜𝑠2𝜃)𝑠𝑖𝑛𝛼(1 − 𝑐𝑜𝑠2𝜃)𝑐𝑜𝑠𝛼

z=

𝐿

2θ

(5)

𝑠𝑖𝑛𝜃

The length L of the prototype single section flexible
arm in the natural state is set to 200mm, and the end point
motion relation is obtained by using MATLAB to get theend point motion range diagram (Fig.9).

The final homogeneous transformation matrix from
configuration space to task space is Eq. (2), Eq. (3) is therotation matrix. Eq. (4) is the translation matrix.
𝛩(𝑘) 𝛹(𝑘)
]
(2)
0
1
2
2
𝑐𝑜𝑠 𝛼𝑐𝑜𝑠𝜃 + 𝑠𝑖𝑛 𝛼 𝑐𝑜𝑠𝛼𝑠𝑖𝑛𝛼(𝑐𝑜𝑠𝜃 − 1) 𝑐𝑜𝑠𝛼𝑠𝑖𝑛𝜃𝛩(𝑘) = [𝑐𝑜𝑠𝛼𝑠𝑖𝑛𝛼(𝑐𝑜𝑠𝜃 − 1) 𝑠𝑖𝑛2 𝛼𝑐𝑜𝑠𝜃 + 𝑐𝑜𝑠 2 𝛼 𝑠𝑖𝑛𝛼𝑠𝑖𝑛𝜃 ] (3)−𝑐𝑜𝑠𝜃𝑠𝑖𝑛𝛼−𝑠𝑖𝑛𝛼𝑠𝑖𝑛𝜃𝑐𝑜𝑠Θ
Ψ(𝑘) = [𝑅𝑐𝑜𝑠𝛼(1 − 𝑐𝑜𝑠𝜃) 𝑅𝑠𝑖𝑛𝛼(1 − 𝑐𝑜𝑠𝜃) 𝑅𝑠𝑖𝑛𝜃](4)
𝐻(𝑘) = [5. Flexible manipulator motion space simulation
analysis:
The motion space diagram of the single-section
flexible manipulator is shown in Fig.8.
Fig. 9 Flexible manipulator motion space simulation
6. The simulation of length changes of the wires:
The vectors representation of the driving wires isobtained by superposition of the vectors, and then the
maximum wire length variation of the single flexible arm
is obtained by solving the modulus of the vector. The
activity limit of the initial single-section mechanical arm
is the rotation angle 0°-360° and the bending angle 0°90°. Through simulation analysis in Fig.10, when thebending angle is 90° and the rotation angle is from 0° to
360°, the length of the drawstring changes the most. The
simulation results are shown in the figure below. The
maximum variation value of wires length of single
section manipulator is about 95mm-265mm.

Fig. 8 Space analysis diagram of flexible manipulator
motionThe red part OO1 of the circular arc represents thecontinuous flexible body. The head end point O is the
fixed point, and the end point O1 is the free moving end.
Let the length of the continuous flexible body (spring) be
L. The arc OO1 corresponds to the chord length d. When

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
597

Powered by TCPDF (www.tcpdf.org)

Design of flexible mechanism

Dr. Kazuo Ishii
He received his Ph.D. degrees from
Tokyo University, Japan, in 1996. In
2011, he joined Kyushu Institute ofTechnology, where he is currently aProfessor of the Department of
Human Intelligence Systems. His
research interests include Underwater
Robot, Neural Network and Neural
NetworkFig. 10 The simulation of length changes of the wires

References
1.

2.

3.

4.

5.

6.

7.

XIAO Xiao. Research Status and Development Trend ofIndustrial
Robot
[J].
Wireless
Internet
Technology,2021,18(23):49-50.
Tsinghua University has made new progress in the field of
micro flexible robot [J]. Robot Technology andApplication, 2023, (02):5.
Wang Yurong, Xu Shengxin. SMA driven FangZhang fish
more joint research and development of flexible
manipulator analysis [J]. Journal of electronic testing,2020, (3): 18 to 19 + 25. DOI: 10.16520 / j.carol carroll
nki. 1000-8519.2020.03.006.
Yu Youhe, Li Jian, Zhou Yan et al. Pneumatic driving
bionic flexible manipulator design [J]. Mechanical andelectrical information, 2022, (17) : 23-26. DOI: 10.19514/ j.carol carroll nki cn32-1628 / tm. 2022.17.006.
Zhu Xiaojun, Wang Xueqian, Ma Yunxuan et al. Rope
hyper redundant space flexible manipulator displacement
teleoperation system design and experimental research [J].
Robot, 2022, 44 (01) : 9 to 18. DOI: 10.13973 / j.carol
carroll nki robot. 210228.
Engineering - Mechanical Engineering and Machinery;
University of Sousse Researchers Publish New Study
Findings on Mechanical Engineering and Machinery(Forward Kinematic Model Resolution of a Special
Spherical Parallel Manipulator: Comparison and RealTime Validation) [J]. Journal of Engineering,2020,2955-.
TANG Yue, Zheng Jinhui, Men Zhengxing et al. D-hmodeling and movement simulation is industrial robot [J].
China's heavy equipment, 2020, (01) : 31-32 + 40. DOI:
10.14145 / j.carol carroll nki. 51-1702 / th. 2020.01.011.

Authors Introduction
Mr. Huang Jiawei
He received his bachelor’s degree inengineering in 2020 from the
Mechanical design, manufacture andautomation;
Guangdong
ocean
university in China. He is currently amaster student in Kyushu Institute ofTechnology, Japan©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
598

Research on Driver Drowsiness Detection Method based on Deep Learning
Shi Puwei

Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku, Kitakyushu, Fukuoka 808-0196, Japan
Kazuo Ishii

Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku, Kitakyushu, Fukuoka 808-0196, Japan
Email: shi.puwei537@mail.kyutech.jp, ishii@brain.kyutech.ac.jp

Abstract
Drowsiness driving will pose a serious threat to the lives of drivers and others. Determining the state of the driverthrough face recognition has the advantages of low cost and convenience. Therefore, this study deploys the facerecognition model to the mobile phone, and finally realizes the recognition of the driver's Drowsiness by the front
camera of the mobile phone. The whole research is divided into three parts. The first part is to train the face 68keypoints recognition model based on the YOLO face algorithm. The second part is to deploy the trained model tothe mobile phone using ONNX and NCNN. The third part is to calculate EAR and MAR using several facial key
points, and finally complete the recognition of the driver's drowsiness state using EAR and MAR.
Keywords: YOLO face, ONNX, NCNN, EAR (Eye Aspect Ratio), MAR (Mouth Aspect Ratio)

Mott G E et al. concluded that the angle of the driver's
steering wheel and the lateral position of the vehicle had
the highest correlation with drowsiness driving bycomparing 87 vehicle parameters related to drowsinessdriving in waking and moderate fatigue states [4].

1. Introduction
Nowadays, the car has become an indispensable meansof transportation, but at the same time, driver drowsinessdriving this dangerous driving behavior is common, thedriver himself and others have brought indelible damage.
Drowsiness driving is due to the driver's high-intensity,
long time driving, resulting in excessive energy andphysical exertion, and then the reaction ability andcontrol level decreased, affecting the normal driving
operation of the situation. The probability of a drivercausing a traffic accident in drowsiness increases
dramatically. In Canada, more than 20% of traffic
accidents involve drowsiness driving. Surveys also showthat 20% of traffic accidents in the United States involve
drivers who are in drowsiness [1]. Therefore, it isimportant to study driver behavior and state recognition
methods to alert and intervene when dangerous situations
occur, to ensure the safety of drivers themselves and otherpeople and vehicles on the road.

As far as current research is concerned, recognition andearly warning through physiological parameters is easy
to interfere with normal driving and affect the drowsinesschange process; identification through vehicle driving
information is subject to factors such as driver skills androad conditions, and the recognition accuracy is relatively
low. The drowsiness recognition method based on thedriver's behavior and state has the characteristics of low
equipment cost and non-invasiveness. At present, therecognition of driver characteristics is mainly
concentrated in areas such as eyes, mouth and so on. At
the same time, the use of cameras for face recognition
also needs to consider the impact of changes in light on
the recognition ability of the model.
Methods of drowsiness recognition based on face can bedivided into two types. One is based on the entire image.
In this method, the model will infer and recognize theentire image, and then crop the different facial featureareas. The other is based on facial feature points. The
model will annotate the specified number of key points
on the entire image, and then infer facial features basedon the relationship between key points. Currently, themost common key points annotations include 98 key
points, 68 key points, and 29 key points. This study
chooses to use the most common 68 key points for model
modification and dataset building.

The current objective detection methods for drowsinessdriving are mainly divided into based on driver
physiological parameters; based on driver operating
behavior; and based on vehicle driving information. In
2007, Yamaguchi et al. extracted amylase from thedriver's saliva and determined whether the driver was ina drowsiness state by analyzing the changes in its
composition [2]. In 2013, Mbouna et al. extracted thedriver's eye state and pupil movement state as driver
fatigue features and used SVM to classify the driver's
drowsiness state, achieving a good recognition effect [3].

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
599

Shi Puwei, Kazuo Ishii

In summary, this study aims to recognize the facial
feature points of drivers through computer vision
methods, establish a driver fatigue recognition method,and complete the deployment of the model on thesmartphone side.
2. YOLO Face-based Face Recognition Model
2.1. Yolo Facea.

YOLO is a classic one-stage target detection algorithm
based on CNN (Convolutional Neural Network). The
original YOLO algorithm does not contain key points
detection. It can only annotate target objects on images
with bounding boxes. Later, some authors developed
YOLO pose for human pose detection based on YOLO.
YOLO pose adds 17 human body key points detection
based on YOLO. The YOLO face used in this study isbased on YOLO pose, changing the 17 key points to fivekey points regression. YOLO face treats face detection asa general target detection task. Therefore, its network
structure is not much different from the overall YOLO
pose, using only some special network modules to replace
the modules in the original structure [5].

Loss of training

b. Precision of validation
Fig. 1. Result of training
The detection performance of the model is evaluated interms of mAP (Mean Average Precision). mAP@0.5
represents the change of mAP when the IOU threshold is0.5, and mAP@0.5:0.95 represents the change of mAP
when the IOU threshold gradually increases from 0.5 to
0.95. These two mAPs are both greater than 0.9 after theend of training, and the training effect is good (Fig. 2).

The original YOLO face includes only 5 key points
detection. This study chooses to use 68 key points for
extracting face-related features. Therefore, the original
YOLO face needs to be modified. The modification ismainly reflected in the data structure. The data structureof each target in the original is a vector of length 16. The
length of data vector modified is 138.
2.2. Dataset for training
This study combines the 300w-3D dataset and the LaPa
dataset to build training and validation sets for the facerecognition model [6], [7]. Among them, the 300w-3D
dataset has the characteristics of large data volume,
accurate annotation, gender balance, age distribution
balance, and bright/dark balance. However, theopen/closed eye data in the 300w-3D dataset is notbalanced, especially the closed eye data is relatively small.
This will lead to poor model generalization ability, andeventually fail to detect closed eye data. Therefore, thisstudy screened out the closed eye data from the LaPa
dataset and combined it with the 300w-3D dataset to forma new dataset. The model is trained and validated using
this new dataset.

Fig. 2. mAP (Mean Average Precision)
To verify that the model still has good detection
capabilities under different lighting conditions, several
images are selected, and new images were obtained bydecreasing the average brightness of every image by 50
in turn, and they were sent to the model. The results showthat the model also has good detection ability in low
brightness conditions (Fig. 3).

2.3. Training Result
YOLO-tiny is selected as the network structure. After
training for 300 epochs, the model has basicallyconverged (Fig. 1).
Fig. 3. Face Detection under Different Brightness©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
600

Research on Driver Drowsiness

3. Android Smartphone Model Deployment

mouth visually. Taking eye as an example, among the 68
facial feature points, each eye occupies 6 feature points,
distributed as shown in Fig. 6. [10]This study uses ONNX and NCNN as "intermediate
representations" to build an inference engine based on the
NCNN framework to deploy the trained face recognition
model on Android phones.
In general, deep learning models rely on related
frameworks and environments, which are not suitable for
installation in mobile phones and other environments. In
addition, deep learning models require a lot of computing
power, which is difficult to run on phones without
optimization. ONNX and NCNN as "intermediate
representations" can solve this problem. ONNX (OpenNeural Network Exchange) was jointly released by
Facebook and Microsoft in 2017 for standard descriptionof model structure and parameter calculation graphs. It
can optimize network structures and reduce
environmental dependencies. NCNN is a lightweightneural network inference framework developed byTencent for mobile platforms [8], [9]. In short, the trainedmodel needs to be converted into ONNX form and theninto NCNN form before it can be deployed to a smart
phone. (Fig.4)

Fig. 6. Feature points of eye
When the eyes are open and closed, the relative positionof P1~P6 changes greatly, especially the verticalcoordinate changes between the P2/P6 and P3/P5 point
pairs are more obvious, and the horizontal coordinate
distance between the P1/P4 point pairs basically does notchange when the eye state changes. According to thechange law of the relative distance between the 6 featurepoints when the eye state changes, the current state of theeye can be clearly extracted, and the specific calculation
is as Eq. (1).

𝐸𝐴𝑅 =Fig. 4. Model transformation process‖𝑃2 − 𝑃6‖ + ‖𝑃3 − 𝑃5‖
2‖𝑃1 − 𝑃4‖(1)

4.2. Experiment for drowsiness detection basedon EAR and MAR

The effect is shown in Fig. 5, the experimental device
is an Android phone equipped with Snapdragon 888Soc(in strong light and low light environments, respectively).

This study uses the dataset from NTHU Computer
Vision Lab for related experiments and analysis. EAR
and MAR are computed for each frame of all the videos
in the training subset of the dataset. To minimize theeffect of noise on the data, exponential smoothing filter
and Gaussian filter are applied to the EAR and MAR ofeach sample respectively. The EAR and MAR graphs canbe obtained as shown in Fig.7 and Fig. 8. The downward
spike in the graph of EAR is the process of winking, andthe upward spike with a certain width in the graph of
MAR is the process of yawning.
By analyzing 142 video samples (71 drowsinesssamples and 71 non-drowsiness samples), it can be foundthat the change in the frequency of winking does notreflect the drowsiness state, because there are individual
differences in winking habits, and the frequency of
winking rises in the drowsiness state of some people anddecreases in other people. But there is always a longer
period of eye closing when people are in the drowsinessstate. Therefore, the presence of significantly longer thannormal winking time for eye closing time can be used for
drowsiness detection (Fig. 9). In addition, for MAR, then-sigma rule can be used to distinguish between yawningand normal speaking (Fig.7 and Fig. 10). The existenceof yawning can also be used for drowsiness detection. Insummary, drowsiness detection methods can beconstructed based on yawning and longer eye closing
time.

Fig. 5. The effect of the model running on a mobilephone
4. Drowsiness Detection Method by face key points
4.1. Eye and mouth state recognition based on EAR
and MAR
After completing the extraction of facial feature regions
or feature points, it is necessary to study the relationshipbetween the open and closed state of the eyes and mouthand drowsiness based on these characteristics. The
relationship was investigated in this study using EAR(Eye Aspect Ratio) and MAR (Mouth Aspect Ratio).
They are based on the same principle, the value of themcan reflect the opening and closing status of the eyes and©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
601

Powered by TCPDF (www.tcpdf.org)

Shi Puwei, Kazuo Ishii

References
1.

Li Xiaoxing, Deep Learning Based Fatigue Driving
Detection Methods, master’s thesis, University of Scienceand Technology of China, 2020.
2. Yamaguchi M, Deguchi M, Wakasugi J, Hand-heldmonitor of sympathetic nervous system using salivary
amylase activity and its validation by driver fatigue
assessment, Biosensors& Bioelectronics, 2006, Vol. 21(7),pp. 1007-1014.
3. Mbouna R O, Kong S G, Chun M G, Visual Analysis of
Eye State and Head Pose for Driver Alertness Monitoring,IEEE Transactions on Intelligent Transportation Systems,
2013, Vol. 14(3), pp. 1462-1469.
4. Mott G E, Forsman P, Short K R, Efficient driver
drowsiness detection at moderate levels of drowsiness,
Accid Anal Prev, 2013, Vol. 50(1), pp. 341-350.
5. Delong Qi, Weijun Tan, Qi Yao, YOLO5Face: Why
Reinventing a Face Detector, arXiv:2105.12931(CS/CV),
2021, https://arxiv.org/abs/2105.12931.
6. Xiangyu Zhu, Zhen Lei, Xiaoming Liu, Face Alignment
Across Large Poses: A 3D Solution,
http://www.cbsr.ia.ac.cn/users/xiangyuzhu/projects/3DD
FA/main.htm
7. Yinglu Liu, Hailin Shi, Hao She, A New Dataset andBoundary-Attention Semantic Segmentation for Face
Parsing, The Thirty-Fourth AAAI Conference on Artificial
Intelligence, 2020, pp. 11637-11644.
8. Open standard for machine learning interoperability,
https://github.com/onnx/onnx/tree/main
9. A high-performance neural network inference framework
optimized for the mobile platform,
https://github.com/Tencent/ncnn
10. Tereza Soukupová, Real-Time Eye Blink Detection using
Facial Landmarks, 21st Computer Vision Winterworkshop, Slovenia, 2016.

Fig. 7. EAR of a non-drowsiness sample

Fig. 8. MAR of a non-drowsiness sample

Fig. 9. EAR of a drowsiness sample
Authors Introduction
Mr. Shi Puwei
He received his Bachelor's degreein Automation from Guangxi
University, Institute of ElectricalEngineering in China.
He is currently a master student inKyushu Institute of Technology,
Japan

Fig. 10. MAR of a drowsiness sample
5. Conclusion
Dr. Kazuo Ishii
He received his Ph.D. degrees
from Tokyo University, Japan,
in 1996. In 2011, he joined
Kyushu Institute of Technology,
where he is currently a Professorof the Department of HumanIntelligence Systems.
His research interests include
Underwater Robot, Neural
Network and Neural Network.

In this study, a 68-keypoints face recognition model
was implemented based on YOLO face, after which themodel was deployed to Android phones through ONNX
and NCNN frameworks, and finally EAR and MAR were
calculated by key points and their relationship with driver
drowsiness driving was studied. According to theexperiment, it was concluded that based on yawning andlonger eye closing time can be used as a standard for
drowsiness determination. In the future, it is planned to
add this standard into the Android inference engine, andthe network structure of the face recognition model isadjusted to improve performance.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
602

An Analysis of Translational Motion for a Mobile Robot with Line-Symmetric Rollers
Arrangement
Kenji Kimura

Department of Control Engineering, National Institute of Technology, Matsue College,
14-4 Nishi-ikuma-cho, Matsue-shi,Shimane,690-8518, Japan
Kazuo Ishii

Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan
Email: k-kimura@matsue-ct.jp, ishii@brain.kyutech.ac.jp

Abstract
In fields such as logistics, robots are required to have efficient mobility. There are various types of rollers used in
mobile robots. Among them, omni-rollers have excellent omnidirectional mobility and are easy to control. In thisstudy, mechanism kinematics has been proposed that assumes arbitrary changes in the roller arrangement position ona circular mechanism and the roller arrangement has been evaluated from the viewpoint of speed efficiency.
Furthermore, we aim to evaluate the mobility of mobile robots by focusing on their translational components.
Moreover, we have examined the behavior of the area of the area generated by the end point of the robot velocity
vector and evaluated the velocity efficiency.
Keywords: Omni-roller, Translational motion, Motion analysis of mobile robotof the two drive rollers that drive the sphere should bedetermined. The optimal location has been identified to
be on the equator [8]. In a mechanism using rollers as
wheels, a transformation matrix is defined that associates
the input roller speed with the output robot speed(translation/rotation), and the “Image volume” and“Orthographic area” are used as an evaluation function
for movement efficiency. The roller arrangement wasevaluated in a previous study [9].
The present study focuses on sectional area by
considering the kinetic energy of a mobile robot in theline-symmetric roller’s arrangement which has onedimensional of freedom. Focusing only on the robot’s
translational motion, we derive a relational expression
between velocity efficiency and roller arrangement, andsubsequently analyze the behavior of the end point region
area of the robot velocity vector.
The rest of this study is as follows: Chapter 2 discusses
the kinematics of mobile robots. Chapter 3 derive
sectional area function. Chapter 4 conducted thesimulation. Finally, we present the summary and futuretasks.

1. Introduction
In recent years, industries such as logistics require
efficient mobile robots for object transport. Developmentof such mobile robot vehicles has attracted great attention
among the research community. The degrees of freedom
for omnidirectional movement (if holonomic properties
are provided) is three, expressed as the sum of atranslational component of 2 degrees of freedom and arotational component of 1 degree of freedom. The
holonomic movement mechanism is easy to control andhas excellent maneuverability in all directions due to the
independent drive characteristics of the wheels.
In particular, there are many moving mechanisms
equipped with three omni rollers, and the kinematics forthese mechanisms have been derived [1]. The three-roller
arrangement of this mechanism has a basic structure ofan equilateral triangle.
This basic arrangement is also used in the soccer robotsof RoboCup Meddle-sized-reague, such as RV-infinity[2], Musashi 150 [3], and NuBot [4]. Thus, a regularpolygon with the highest degree of symmetry among
triangles was used, and no theoretical research had been
conducted. However, recently, researchers have started
focusing on the kinetic energy of drive rollers for moving
mechanisms [5] and spherical conveyance [6].
In a movement mechanism using a sphere as wheels(or a mechanism using a sphere as a conveyance object),the optimal angle of the sphere rotation axis with respect
to the movement direction [7] and the placement location

2. Kinematics of Transfer Mechanism in case of
isosceles triangle three rollers arrangement
In a previous study, we have defined “Image volume”
and “Orthographic projection area” as evaluation
functions analysis roller contact location ([9]).
Section 2.1 introduces only on the robot translationalmotion and analysis robot motion.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
603

Kenji Kimura, Kazuo Ishii

Section 2.2 discusses kinematics for a mechanismthat adapts three omni-rollers in line symmetryroller’s arrangement.
2.1.
Liner
Transformation
mapping
correspondence of roller speed and robot speedforAs shown in Figure 1(a), the mobile robot that has acommon radius of all omni-wheels adapted the i-th rollers(𝑖 = 1,2,3) contact point 𝑷𝒊 on a circle. 𝑋– 𝑌 is the global
coordinate system (origin 𝑶). The robot translation speed
𝑻is 𝑽 = [𝑉𝑥 , 𝑉𝑦 ] and 𝜑 denotes robot direction. Robotrational speed 𝐿𝜙̇ ( 𝜙̇ : robot angular velocity) roller
peripheral speed 𝑣𝑖 are decomposed as translation androtational components. Contact point 𝑷𝒊 are adapted
angle 𝜃𝑖 on the circle that has a radius 𝐿 . Thus, thė ] iscorrespondence of [𝑣1 , 𝑣2 , 𝑣3 ] and [𝑉𝑥 , 𝑉𝑦 , 𝜙𝐿
represented in Figure 1(a).
As shown in Figure 1(b), linear transformation
̇ ]𝑻mapping 𝑓𝐴 : [𝑉𝑥 , 𝑉𝑦 , 𝜙𝐿
→ [𝑣1 , 𝑣2 , 𝑣3 ]𝑻 and𝑓𝐴 (𝑊) =Image𝑓𝐴 ∈ 𝐑3 is a parallelepiped domain fromcubic domain 𝑊.
In this study, we represented a sectional area of
𝑓𝐴−1 (𝑊) for a horizontal plane as follow:

Figure 1 Correspondence between three omni-rollers speed
̇ ]𝑻 (robot mobile speed (𝑉𝑥 , 𝑉𝑦 )[𝑣1 , 𝑣2 , 𝑣3 ]𝑻 and [𝑉𝑥 , 𝑉𝑦 , 𝜙𝐿
rotational speed 𝜙̇ ). (a) mobile robot and (b) translation
mapping.

𝑻

Forward kinematics ([𝑉𝑥 , 𝑉𝑦 , 𝐿𝜙̇] is determined from[𝑣1 , 𝑣2 , 𝑣3 ]𝑻 ) is represented as follows:
1
2sin
𝜃
𝑉𝑥
1
[ 𝑉𝑦 ] = −2 − 2cos 𝜃𝐿𝜙̇1[ 2 − 2cos 𝜃

𝑓𝐴−1 (𝑊) ∩ {𝑉𝑥 𝑉𝑦 − plane}−= {(𝑉𝑥 , 𝑉𝑦 , 0)| |𝑣1 |, |𝑣2 |, |𝑣3 | ≤ 1}

(1)

𝑓𝐴−1 (𝑊) = {(𝑉𝑥 , 𝑉𝑦 , 𝐿𝜙̇)| |𝑣1 |, |𝑣2 |, |𝑣3 | ≤ 1}

(2)

𝑊 = {(𝜈1 , 𝜈2 , 𝜈3 )||𝑣1 |, |𝑣2 |, |𝑣3 | ≤ 1}

(3)

Where

1
2sin 𝜃
1−
2 − 2cos 𝜃1
2 − 2cos 𝜃01
1 − cos 𝜃cos 𝜃−
1 − cos 𝜃 ]
𝑣1
𝑣
[ 2]
𝑣3
(5)

2.2 Kinematics for isosceles triangle three rollers
arrangement

3. Derive sectional area function
In this section, we calculate area using Eq. (1). Section
3.1 shows analysis of roller’s speed space, while Section
3.2 shows analysis of robot speed space.

In our previous work [9], we derived kinematics forarbitrary roller arrangement position as (𝜃1 , 𝜃2 , 𝜃3 ). In
this study, we restrain an isosceles triangular roller
arrangement (line symmetry roller’s arrangement) in
Eq. (6) of the previous study [9]Substituting 𝜃1 = 𝜃, 𝜃2 = 360° − 𝜃 and 𝜃3 = 0° with

3.1. Analysis of roller’s speed space(A) Setup for Cubic domain(𝜃1 , 𝜃2 , 𝜃3 ) in Eq. (6) of [9].
Inverse kinematics ([𝑣1 , 𝑣2 , 𝑣3 ]𝑻 is determined from𝑻
[𝑉𝑥 , 𝑉𝑦 , 𝐿𝜙̇] ) is represented as follows:𝑣1− sin 𝜃[𝑣2 ] = [ sin 𝜃
𝑣3
0

cos 𝜃
cos 𝜃
1

1 𝑉𝑥
1] [ 𝑉𝑦 ]1 𝐿𝜙̇

As shown in Figure 2(a), the cubic domain iscomposed of the following eight apexes: 𝑩𝟏 = (1, 1, 1),
𝑩𝟐 = (−1 , 1 , 1) , 𝑩𝟑 = (−1 , −1 , 1) , 𝑩𝟒 = (1 , −1 ,1) ,𝑩𝟓 = (1, 1, −1), 𝑩𝟔 = (−1, 1, −1),𝑩𝟕 = (−1, −1,−1), 𝑩𝟖 = (1, −1, −1). Additionally, sectional domain
denoted by 𝜋 is symmetrical with respect to the origin 𝑶.
For the sectional shape to be hexagonal like Figure 2(a),six dots 𝑷(𝑝, −,,,), 𝑸 (−,,𝑞 ,,), 𝑹(−,,,,𝑟), 𝑺(𝑠 ,,, −,),

(4)

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
604

An Analysis of Translational

𝑻 (,, 𝑡 , − ,), 𝑼 (,, − ,, 𝑢 ) (real parameter − , ≤𝑝, 𝑞, 𝑟, 𝑠, 𝑡, 𝑢 ≤ 1) should satisfy the following condition.
𝑝 = 𝑠, 𝑞 = 𝑡, 𝑟 = 𝑢(6)

As the two faces, 𝑩𝟏 𝑩𝟐 𝑩𝟑 𝑩𝟒 and 𝑩𝟓 𝑩𝟔 𝑩𝟕 𝑩𝟖 (face toface), are parallel, side 𝑷𝑸 and 𝑻𝑺 are parallel and havesame length.
(B) Plane representation as roller speed existent set
Focusing on 𝐿𝜙̇ -component of Eq. (5), onlytransrational motion is equivalent to 𝜙̇ = 0.
𝑣1 sin 𝜃 + 𝑣2 sin 𝜃 − 2 𝑣3 sin 2𝜃 = 0

Figure 2 Sectional shape of roller’s speed space 𝜋 ∩ 𝑊 in caseof an isosceles triangle roller arrangement. (a) Isometric view.
(b) Sectional view.

(7)

Eq. (8) represents a plane equation including origin by
three-dimensional combinate [𝜈1 , 𝜈2 , 𝜈3 ]𝑻 .

𝑓𝐴−1 (𝑊) ∩ {𝑉𝑥 𝑉𝑦 − plane} =𝜋={(𝑣1 , 𝑣2 , 𝑣3 )|𝑣1 sin 𝜃 + 𝑣2 sin 𝜃 − 2 𝑣3 sin 2𝜃 = 0}{(𝑉𝑥 , 𝑉𝑦 )| 𝑣1 𝑨́𝟏 + 𝑣2 𝑨́𝟐 + 𝑣3 𝑨́𝟑 , |𝑣1 |, |𝑣2 |, |𝑣3 | ≤ 1}

(8)

where

(C) Normal vector of plane

−

1

1

2sin 𝜃
1

2sin 𝜃
1

01

𝑨́𝟏 = − 2−2cos 𝜃 , 𝑨́𝟐 = − 2−2cos 𝜃 , 𝑨́𝟑 = [ 1−cos 𝜃 ]111−
1−cos 𝜃[ 2−2cos 𝜃 ][ 2−2cos 𝜃 ]Eq. (7) is equivalent to following expression.
𝑣1
1
sin 𝜃 〈[] , [𝑣2 ]〉 = 0
1−2 cos 𝜃 𝑣3

(10)

(9)

(11)

From Eq. (6) and property of transformation mapping
𝑓𝐴−1 , 𝑓𝐴−1 (𝜋 ∩ 𝑊) is decomposed as three parts.

Thus. [1,1, −2 cos 𝜃]𝑻 is normal vector of plane 𝜋. Also,sin 𝜃 > 0 and −2 cos 𝜃 > 0 for all 90° ≤ 𝜃 < 180°.
Figure 2(b) shows a section of the cubic domain withrespect to rectangle (𝑩𝟏 𝑩𝟑 𝑩𝟕 𝑩𝟓 ). It can be observed that
normal vectors shift counterclockwise direction from 𝜈3 axis (when 𝜃 = 90° ) to line 𝑩𝟑 𝑶 (when 𝜃 = 180° ).
Thus. 𝜋 ∩ 𝑊 satisfies the following property.

𝑓𝐴−1 (𝜋 ∩ 𝑊) = 2𝑓𝐴−1 (△ 𝑶𝑷𝑸)

(12)

+2𝑓𝐴−1 (△ 𝑶𝑸𝑹) + 2𝑓𝐴−1 (△ 𝑶𝑹𝑷)
Where

Property(i) Sectional shape is symmetrical with respect to origin(a) 𝜃1 = 90° ⇔ Rhomb(b) 90°＜ 𝜃1 ＜180° ⇔ Hexagon

𝜋 ∩ 𝑊 = 2(△ 𝑶𝑷𝑸 + △ 𝑶𝑸𝑹 + △ 𝑶𝑹𝑷)

(13)

Thus. Sectional area of 𝑓𝐴−1 (𝑊) ∩ {𝑉𝑥 𝑉𝑦 − plane} isrepresented as following.
𝐷𝑆𝑒𝑐 (𝜃) = ‖𝑓𝑨−1 (𝑷) × 𝑓𝑨−1 (𝑸)‖(ii) When 𝜃1 = 120° it has minimal area.

+‖𝑓𝑨−1 (𝑸) × 𝑓𝑨−1 (𝑹)‖ + ‖𝑓𝑨−1 (𝑹) × −𝑓𝑨−1 (𝑷)‖

3.2. Analysis of robot speed space(14)

Using Eq. (5), image of 𝜋 ∩ 𝑊 by transformation
mapping 𝑓𝑨−1 can be represented as follows

Where 𝑷(𝑝,−,,,), 𝑸 (−,,𝑞,,), 𝑹(−,,,,0)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
605

Powered by TCPDF (www.tcpdf.org)

Kenji Kimura, Kazuo Ishii

Reference𝑝=sin 𝜃 − sin 2𝜃
− sin 𝜃 − sin 2𝜃
,𝑞 =
sin 𝜃
sin 𝜃

(14)

[1]

[2]

4. Simulation[3]

This section presents the simulation findings,
including the evaluation values of “horizontal crosssectional area: 𝐷𝑆𝑒𝑐 ”
Simulations were performed at the various symmetricroller arrangement (Eq. (14) in 90° ≤ 𝜃 < 180°).
As shown in Figure 3, 𝐷𝑆𝑒𝑐 (90°) = 4.05[𝑚/𝑠]2 and𝐷𝑆𝑒𝑐 (180°) = ∞ [𝑚/𝑠]2 .
minimum value is
𝐷𝑆𝑒𝑐 (120°) =3.85[𝑚/𝑠]2 . Thus. The worst efficiency
roller arrangement in only translational motion is theequilateral triangular shape.

[4]

[5]

[6]

[7]

[8]

[9]

J.Tang, K.Watanabe, et al., “Autonomous control for anomnidirectional mobile robot with the orthogonal-wheel
assembly,” Journal of the Robotics Society of Japan. Vol. ,7, No.
1,pp. 51-60, 1999.
Y.Yasohara, K.Shimizu et al., Development of ball handlingmechanism for RoboCup MSL, 30th Fussy System Symposium, pp.
616-617, 2014.
S. Chikushi, M. Kuwada, et al., Development of Next-Generation
Soccer Robot”Musashi,50”for RoboCup MSL,30th Fussy System
Symposium, pp. 624-627,2014.
R. Junkai, X. Chenggang, X. Junhao et al., A control system foractive ball handling in the RoboCup middle size League, ChineseControl and Decision Conference(CCDC.)2016.
K. Kimura， K. Ishii, Efficiency Problem of Spherical Robot inTransfer Kinetic Energy, Journal of Robotics, Networking andArtificial Life, Vol.9, issue 1, pp.87-92, 2022.
K. Kimura, Koki Ogata, Hiroyasu Hirai, Takumi Ueda, K. Ishii,
Forward Kinematic of a Sphere Considering Slipping and Motion
Analysis in Three Rollers, Journal of Robotics, Networking andArtificial Life, Vol. 10, issue1, pp.25-32.2023.
K.Kimura, Y.Abematsu, H.Hirai, K.Ishii, Evaluation of Two
Roller Arrangement on a Hemisphere by Kinetic Energy, Journalof Robotics, Networking and Artificial Life, Vol.9, issue 3,
pp.233-239, 2022.
K. Kimura, K. Ishii, The Spherical Robot Transfer Problem With
Minimal Total Kinetic Energy, Proceedings of International
Conference on Artificial ALife and Robotics (ICAROB2021),
pp.266-270, 2021.
K. Kimura, Y. Shigyo, K. Ishii, An Analysis of Robot Speed
Efficiency for Mobile Robot Adapted Three Omni Rollers Using
Linear Transformation, Journal of Advances in Artificial Life
Robotics Vol. 3,issue4, pp. 242–249.2023.

Authors Introduction
Dr. Kenji Kimura
He is a Lecturer at the Department ofControlEngineering,
National
Institute of Technology, Matsue
College, and a Visiting Associate
Professor at Chuo University. He
received his ME (mathematics) from
Kyushu University in 2002 and his
PhD (engineering) from the Kyushu
Institute of Technology in 2020. His research interests
are spherical mobile robots.

Figure 3 Behavior of Area of domain of end point ofrobot speed 𝐷𝑆𝑒𝑐 (𝜃) (90° ≤ 𝜃 < 180°).

5. Conclusion
In this research, we focused on the translational motionof mobile robots. Furthermore, we derived the area of theregion where the end point of the robot’s velocity vector
exists as an evaluation function, investigated its behavior,and determined the minimum value. We get fact thatworst efficiency roller arrangement is the equilateraltriangular shape.
This is a one-parameter problem because the roller
arrangement is given as an isosceles triangle.

Dr. Kazuo Ishii
He is a Professor in the Kyushu
Institute of Technology, where he has
been since 1996. He received his
Ph.D. degree in engineering from
University of Tokyo, in 1996. His
research interests span both ship
marine engineering and Intelligent
Mechanics. He holds five patents
derived from his research.

Acknowledgements
This work was supported by JSPS Grant-in-Aid for
Scientific Research 23 K13279.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
606

Development of Teaching Materials for Robot Programming for Junior High School Students:
Student-Based Educational Activities
Kenji Kimura

Department of Control Engineering, National Institute of Technology, Matsue College,
14-4 Nishi-ikuma-cho, Matsue-shi,Shimane 690-8518, Japan
Youta Takano

Department of Control Engineering, National Institute of Technology, Matsue College,
14-4 Nishi-ikuma-cho, Matsue-shi,Shimane 690-8518, Japan
Email: k-kimura@matsue-ct.jp, y8otaman@gmail.com

Abstract
With the extensive use of robots in recent years, an age of co-existence between humans and robots is expected toarrive in the future. Therefore, providing robot education in the early stages of elementary and junior high school isnecessary to stimulate students’ interest in robots. Furthermore, educational institutions are becoming more active inrobot education as a part of their contribution to the local community. For elementary school students, a Beauto
Racer (Vstone) has been adopted as a teaching material for robot education, and teaching materials have beendeveloped for teaching line tracing. In this study, we will develop educational materials for junior high school
students on maze exploration using a Beauto Rover (Vstone). The students of Matsue National College ofTechnology took the initiative in this activity. The purpose is to improve their teaching skills.
Keywords: Robot education, Programing, Maze search, Student-based educational activities

1. Introduction

Table 1 Commercial microcontrollers for robotic education

AI robotic research is being promoted at anaccelerated pace, as obvious from RoboCup’s goal of"building an autonomous robot team that can beat theworld champion soccer team in 2050" [1]. Thus, an era
in which humans and robots will live together inharmony is expected in the near future; therefore,
providing a strong awareness regarding robots to futurechildren at an early age is critical. As shown in Table 1,many lesson designs use commercially available
microcomputers.
As an example of robotic education using Lego
Mindstorms NXT, the company offers numerous
courses in robotic education for students belonging tofirst to sixth grades. This includes student-centered
robotic contests centered on line tracing [2].
Furthermore, robotic education using Beauto Racer(Vstone) is being developed as a curriculum forelementary school students to learn programing on thetheme of line tracing [3].
In a junior robotic contest, participants used Lego
Mindstorms EV3 to perform line tracing for simulating
tomato harvesting [4].
Moreover, embedded programing using Lego
Mindstorms NXT was employed to teach maze
exploration programing [5].

Commercially available

Lesson content

Literature

microcontrollers
Lego Mindstorms NXT

Line trace[2][5]

Maze Search
Beauto Racer (Vstone)

Line trace[3]

Lego Mindstorms EV3

Line trace[4]

While most of these efforts are faculty-led, there arealso student-led collaboration activities in thecommunity that are aimed at improving the basic skillsof students and other members of society ([6]).
A student-led working group was organized in thisstudy to produce educational materials that will enable
students to quickly and easily learn maze search
programming.
The rest of this study is as follows: Chapter 2
introduce
information on lesson design andmicrocontrollers to be used for maze exploration
programming . Finally, we present the summary andfuture tasks.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
607

Kenji Kimura, Youta Takano

2. Planning and Preparation
2.1. View trovers and educational equipment

2.2. Lesson design for maze exploration programing

A Beauto Rover (Vstone) was used as the
microcontroller for robotic education. To organize
multiple wiring, a body is designed for it, as shown in
Figure 1. It was designed using Slid Works (Figure 1(a)),
prepared for processing by SPACEGEAR-U44(MAZAK) (Figure 1(b)), and laser was processed using
an acrylic plate (Figure 1(c)). This plane has arectangular shape with dimensions of 90 [mm] × 125[mm], and an image of the microcontroller after
attaching the protective cover is provided in Figure 1(d).
As shown in Figure 2, a single infrared sensor ( ① )was mounted on the head of the mechanism. First, thepower switch of the microcontroller ( ② ) was turned on,
and then a program created by an application on a PC
was sent to the CPU board ( ③ ) by connecting it to a
USB connector ( ④ ). Finally, the program proceed
button ( ⑤ ) to execute the program is turned on, andthe front tires ( ⑥ ) rotate to perform the motion on
execution.

(a)Assuming that we have lectured on how to transfer a
program to the microcontroller and execute it, we will
prepare problems 1–5, which are the key problems to besolved in a maze search. Table 2 shows the maze search
programing materials (powerpoint-slide) prepared forthe lecture.
The first problem is "1 second forward" and "turn right(90-degree rotation).” Participants were asked to think
about the number of rotations and how much they haverotated with respect to time (Figure 3(a)).
For problem 2, the exercise is to repeat problem 1
four times, where the useful "loop" function is taught
and guidance is provided (Figure 3(b)). For problem 3,
students are taught about conditional branching byhelping them understand the relationship between
senders and the IF function (Figure 3(c)).
A driving program is given as a mission in problem 4
to avoid obstacles when they are placed inpredetermined positions (this would use the knowledge
from problems 1-3) (Figure 3(d)).
Problem 5 is implementing a program that will workwith different placements of obstacles (note thedifference from problem 4) (Figure 3(e)).

(b)
Table 2 Class design(c)(d)
Figure 1 Process of creating a protective cover for the
microcontroller: (a). designing using SolidWorks, (b).
preparation for laser machining, and (c). laser processing. (d).
Microcontroller with a protective cover mounted(data) item

Notes on the Guidance

Question 1

Actionsforgoingstraight
andturningright.

Understandtherelationshipbetweenthenumberofrevolutions and timerequired to make a 90°
turn(relationshipbetween the number of
revolutions and angle ofthe turn)

Question 2

Iterative
operation

Ability to use the loop
function.

Question 3

IF Function

Ability to use the roleof sensors and IF syntax

Question 4

Traveling along
an approximate
known route

Summary of stationary,straight ahead, right/leftturn (basic driving)

Question 5

Unknown Maze
Running

Understandthedifference
betweendrivingonanapproximate
known
pathway

Figure 2 Function of each part of the robot©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
608

Development of Teaching Materials

(a)
(a)

(e)
Figure 3 Maze search programming materials (powerpointslide for lecture) and five Processes. (a). Basic driving, such as
going straight, turning right, and turning left. (b). Iterative
operations, (c). IF syntax, (d). multiple obstacle avoidance,and (e). maze search.

2.3. Maze selection
Obstacles were selected to create a maze, and threematerials used for creating a maze are listed in Table 3.
First one is recycling boxes, which are all around us andhave a flat surface. They are relatively responsive tosensors, but their nonuniform size makes it difficult tofreely change the maze path (Figure 4(a)). Further, inthe case of paper cups, the size is uniform and its unit
price is low. However, its shape resembles a part of acone, and the infrared sensor light reflects irregularly
and penetrates the gap between neighboring paper cups
(Figure 4(b)). Finally, the sensors responded well in the
cubic box case dimensions: 100 mm × 100 mm × 100mm). This is because they have flat surface, and theinfrared sensor responded relatively well because of its
uniform size. Furthermore, with cubic box, it is easy tofreely change the maze path (Figure 4(c)). Based on theabove, cubic boxes were adopted as the obstacle forpreparing maze.
The maze corridor’s width should be set at 200 mm,
which is equivalent to the width of two cubes. This
allows the robot to turn right and left in the passageway.

(b)

(c)

(a)
(d)

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
609

Powered by TCPDF (www.tcpdf.org)

Kenji Kimura, Youta Takano

Acknowledgements
The following four students of National Institute ofTechnology, Matsue College assisted us in this research.
We thank them from the bottom of our hearts.
Hiroto Bessho (5th year student), Kojyu Tanaka (2ndyear student), Konoka Mazda (2nd year student), andKisara Mitani (2nd year student).
References
(b)

[1]

[2]

[3]

[4]

(c)
[5]

Figure 4 Maze production choice: (a). recycled box, (b). paper
cup, and (c). cube box[6]

Table 3 Three candidates for maze production
Candidateobstacles
Recycling bin
Paper cup

Cube box [100mm per side](commerciallyavailable)

Advantage

Disadvantage

Gather easily
Low unit price
Uniform size

Uneven size
Sensorrecognition
unstableHigh cost

High degree of
freedomforplacement
Uniform size

I. Noda, H. Minakata, K. Kobayashi, RobotCup Aiming for theYear 2050 (Part 1), Journal of Japan Society for Fuzzy Theory
and Intelligent Informatics, Vol. 29, No. 1 (2017) pp. 2-13.
T. Hakata, H. Shibari, Y. Shimada, H. Otsuka, M. Nagata, T.
Matsumoto, H. Yamamoto, Education of Embedded System for
Senior Students via Line Tracer Robot, Research Reports ofKumamoto-NCT, Vol. 3 (2011) pp. 29-35.
N. Tominaga, Development of Programing Learning Curriculumthat Targets Elementary School Students, Ritsumeikan Teacher
Educational Studies, Vol. 4 (2014) pp. 81-90.
T. Matsuo, Y. Takemura, T.Sonoda, et al.Toward Smart Tomato
Greenhouse:The 6th Tomato-Harvesting-Robot Competition andRegulation Changes Aiming at Practical Application, Journal of
Robotics, Networking and Artificial Life, Vol. 9, No. 1 (2022)
pp. 13-19.
Y. Takemura, T. Kawasaki, K. Okazaki, Discussion about the
Effect of the Manufacturing Education University Student using
Robots, The 29th Fuzzy System Symposium, (2013) pp. 310-312.
H. Yonemori, N.Tsuguta, D. Sugiyama, Student-Based Regional
Collaboration Activities, Japanese Society for Engineering
Education, (2020) pp. 68-69.

Authors Introduction
Dr. Kenji Kimura
He is a Lecturer at the Department ofControlEngineering,
National
Institute of Technology, Matsue
College, and a Visiting Associate
Professor at Chuo University. He
received his ME (mathematics) from
Kyushu University in 2002 and his
PhD (engineering) from the Kyushu Institute ofTechnology in 2020. His research interests arespherical mobile robots.

is3. Conclusion
In this study, a Beauto Rover (Vstone) was used as
the microcontroller for robot programing. With only one
infrared sensor attached to the back of its head, thismicrocontroller can be easily operated by junior highschool students. To make robot programing learning
feasible, we produced programing materials thatprovided steps for realizing a successful maze
exploration.
We plan to conduct several workshops at ouruniversity using the developed robot programming
material for future research projects. In addition, weplan to conduct pre- and postlecture questionnaires toinvestigate changes in interest and awareness on maze
exploration and to evaluate the educational materials wehave developed.

Mr. Youta Takano
He is 5th year student at the
Department of Control Engineering,
National Institute of Technology,
Matsue College. His researchinterests include educational material
for robots and STEM education.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
610

Design and Software Production of Robotics Educational Designfor Elementary and Junior High School Student
Youta Takano

Department of Control Engineering, National Institute of Technology, Matsue College, 14-4 Nishi-ikuma-cho,
Matsue-shi, Shimane,690-8518, Japan
Kenji Kimura

Department of Control Engineering, National Institute of Technology, Matsue College
Email: d1923@matsue-ct.jp, k-kimura@matsue-ct.jp,

Abstract
The Ministry of Education, Culture, Sports, Science and Technology (MEXT) has been promoting cross-curricularlearning including STEAM education in its educational policy for 2021. The purpose of this study is to have students
experience not only control engineering but also basic programming techniques, and to have them become interested
in mathematical subjects in general, which are the basis for control engineering. The educational design andaccompanying software were designed using Beauto Balancer2 (Vstone) educational robot and Scratch, a visualprogramming editor developed by the Scratch Foundation. In addition, workshops were conducted, and the lesson
design was evaluated by a questionnaire.
Keywords: Robot education, Control engineering, Visual Programming, Beauto Balancer

setting of a robotics class for elementary and junior highschool students.
In recent years, it has become necessary to use someform of programming in controlling robots. However,there are few educational materials for elementary andmiddle school students that can help them understand andbecome interested in programming and control
engineering concepts. There have been no control
engineering-related studies targeting elementary andmiddle school students or aiming to educate them about
new concepts through robotics.
In this study, we will develop extensions to existingeducational robots and plan lessons that will enable
elementary and junior high school students to become
interested in a wide range of fields and challengethemselves to learn. The robot used in this study, "Beauto
Balancer2 (Vstone)," is a teaching material aimed atlearning control. Therefore, it is necessary to develop
new software to use it as a teaching material for
programming classes.

1. Introduction.
The Ministry of Education, Culture, Sports, Science andTechnology (MEXT) has made programming education
mandatory for elementary and junior high schools
starting in 2021. The purpose is to develop children's
programming abilities, discover their hidden potential,
and create opportunities for them to play an active role insociety in the future [1].
In addition, the Ministry of Education, Culture, Sports,
Science and Technology (MEXT) promotes STEAM
education and other cross-curricular learning to apply
learning in each subject and other areas to discoveringand solving problems in the real world. This is aimed at
fostering problem-solving skills that transcend subject
boundaries [2].
Hirata developed an experimental teaching material
that enables students to identify control objects in boththe time and frequency domains and to design control
systems while considering the relationship between them[3]. This teaching material is intended for beginning
students of control engineering.
Kato conducted an objective verification of educational
effectiveness through robot building and robot contests.
Here, as an objective method of measuring effectiveness,criteria for behavior evaluation were established andimplemented, and the behavior of the subjects wasevaluated by TAs [4]. This study cites education in the

2. Creation of home-grown software
2.1. Examination of specifications for home-grownsoftwareIn this study, Beauto Balancer 2 (Vstone) will be used.
Beauto Balancer 2 comes with the factory software(Balancer 2 Programmer).

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
611

Youta Takano, Kenji Kimura

The objectives sought for the home-made application
used in this study are as follows.

Table 1 shows the results of the study and review of thespecifications (A, B, and C). Initially, we adopted
proposal A, giving priority to the user interface (UI).
However, the communication problems could not be
solved, and the development process would have been toolarge, so we gave up on it. Next, they decided to adopt
proposal C and develop it in earnest, as it was judged tobe more effective in terms of man-hours among the three
proposals.

 To have the robot embody the behavior envisioned by
the subject.
 The difficulty level of the application should be suchthat elementary and junior high school students canoperate it without difficulty.
 Use of libraries (written in C) provided by Vstone

2.2. Software Development

To achieve these goals, we have defined the functions
to include or coexist with the functions of the original
software (Balancer 2 programmer) and to introduce theconcept of visual programming so that elementary andjunior high school students can easily operate thesoftware.
To achieve this, the following three compositions anddescription methods were considered.

As a result of the study in section 2.1, we adopted C
and produced two applications. The overall system
configuration and process flow are shown in Figure 1.
The first is a visual programming editor produced using
Scratch 3.6. The second is a communication software
written in C.
The flow of the entire system, its usage, and its
structure are described. First, a txt file was generated
from the visual programming editor, and using Scratch's
original block definition function, a block was created to
add a string to the list according to the contents of theaction (e.g., forward, stop). After the participants createdtheir own actions, we had them generate them using
Scratch's txt file output function for the list contents.
Next, we asked participants to start up thecommunication software. To reduce the burden on theparticipants, we implemented the software so that it could
be automatically loaded and executed by simply clicking
on the application to open it.
When the communication software is opened, it
automatically establishes USB HID communication andreads the txt file. At this time, we designed the softwareto read the txt file sequentially and issue communicationcommands according to the contents of each line.

A Desktop application only (written in Python)
B Console application (written in C)
C Scratch script + console application.
+ Genuine software (written in C)
Table 1 Study table for the software specification
advantage
Useofexternal
libraries facilitates UItuningdisadvantage
The stock library needs
to be re-written inadvance. Even if thelibrary is ported to
Python, there is no
guarantee that the serial
communication system
will work properly.

User operation unit
(so-called front end)
and the language of theinternal processing part(i.e., the bag-end)
Uniformity is possible,
allowing for flexible
design and description.

The nature of console
applications limits UI
tuning.

Troubleshootingiseasierbecausesoftware is created ordiverted by dividing
functionality.

Software tends to be
somewhat cumbersome
toconfigure,assoftware is divided byfunction, which is lessconsistent than otherproposals

A

B

C

High
degreeofdifficultyinintroducing
visualprogramming concepts.

Figure 1 Overall system design©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
612

Design and Software Production

3. Mock Class for Experiments

Step.2 "Let's feel physics and mathematics! "Next, mathematics, physics, and control engineering
were explained to elementary school students in an easyto-understand manner using specific examples. At first,
basic knowledge and concepts of mathematics andarithmetic (functions, derivatives, and integrals) wereexplained as necessary knowledge to explain physics
concepts. Figure 3 shows a slide explaining integration.

3.1. Design of mock class
A mock class was conducted for elementary and juniorhigh school students (hereinafter referred to as
participants). The evaluation was based on the results ofthe participants questionnaires. The dates of theexperiments were as follows Table 2.
Table 2 Planning mock classes
Date
December16, 2023

participant
4 persons(Breakdown:
Fourth grade. 2 students,(one each in sixth grade and eighth
grade))

Figure 3 Slide of Class Step 2

Title.
Student-centered lectures and hands-on practice ♪
〜Let's feel mathematics, physics, and control byusing butte balancer! 〜

Step.3 "Move the robot as you wish! "We defined the movements we wanted the subjects toperform with the robot and asked them to perform themin sequence. This was done using special software. The
explanation was given while displaying the Scratch
screen Figure 4.

The author was the instructor and one teachingassistant (TA), and the maximum number of participants
was set at five. Each mock class was held as a part of acraft class hosted by the school, and participants wererecruited from the public through posters distributed toelementary and junior high schools in Shimane
Prefecture. The aim of the classes was to encourage
students to become interested in mathematics, physics,
and all other fields by experiencing the mechanisms thatmake objects move as desired. The mock class consistedof three steps (Step.1-3).

Figure 4 Demonstration of Class Step 3

Step.1 "Let's get used to the robot! "After giving the students the course materials andexplaining how to use the robot and its software, we ask
them to touch the robot. Since this is the first time for theparticipants to touch the machine, we ask them to touch
it for the time being to familiarize themselves with it. At
this point, we ask them the question, "What is gain? andasked them to answer the question while touching thegenuine software. Figure 2 is a slide showing how to use
Beauto Balancer 2.

3.2. Evaluation of mock class
To test the effectiveness of the program, a pre-classquestionnaire and post-class questionnaire areadministered to the subjects. Subjects were asked 5questions before and after each class. The questionnaire
was given on a 5-point scale, with the higher the number,the more positive the evaluation. Table 3 shows thecontents of the pre-lesson questionnaire and Table 4shows the post-lesson questionnaire.
Table 3 Pre-Class Questionnaire
Q1: Do you like mathematics?
Q2: Do you like science?
Q3: Can you visualize the control?
Q4: Are you concerned about what math and arithmetic.
learned in school is used for?

Figure 2 Slide of Class Step 1

Q5: Wha science is used for in school. Are you curious?

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
613

Powered by TCPDF (www.tcpdf.org)

Youta Takano, Kenji Kimura

4. Conclusion.

Table 4 Post-Class Questionnaire

In this study, we developed software for robot
education for elementary and junior high school students
and planned and verified mock classes. The results of theverification showed that the level of interest and concern
increased.
In the future, we plan to conduct mock classes withsubjects with similar characteristics. In addition, we plan
to conduct a long-term post-lesson survey.

Q1: Have you developed an interest in mathematics?
Q2: Were you interested in science?
Q3: Did you have an interest in control engineering?
Q4: Did you want to know more about what math and.
arithmetic is used for in school?
Q5: What is the science you learn in school used for?
Do you want to know more about it?

Acknowledgements
Figure 5 shows the results of the questionnaire. The
horizontal axis represents each question 1-5 in thequestionnaire and the vertical axis represents the averageof the response results.
5.0

Pre-Class
Post-Class

4.0

ResultThis research was supported by the Mazda Foundation
Project Grant Related to the Promotion of Science andTechnology (23KJ-023). We would also like to thankKojyu Tanaka for his assistance with the control
experiment.
Reference

3.0
Ministry of Education, Culture, Sports, Science andTechnology, Guide to Elementary School Programming
Education (Third Edition) ,2020.
[2] Ministry of Education, Culture, Sports, Science andTechnology, Promotion of STEAM education and othercross-curricular learning, 2021.
[3] Mitsuo Hirata, An experimental apparatus for control
education to understand the relation between frequency
domain and time domain, Proceedings of the 57th
Automatic Control Union Conference, 2014, pp.606-609.
[4] Koichi Kato, et al.. Educational Effects of Creating a
Robot and Participating Robot Contest, Annual reports of
the Faculty of Education, Gunma University, series issue46, 2011, pp.125-147.
[1]

2.0

1.0
0.0
Q1

Q2

Q3

Q4

Q5

Question
Figure 5 Questionnaire results
3.3. Discussion of Verification Results
From Figure 5, the percentage increase in the survey
results was calculated and is shown in Table 5. As a result,
there is a large increase in some areas. However, Q4 canbe taken as showing a decrease.

Authors Introduction

Table 5 Percentage increase in survey results

Rate of increase [%]

Q1.

Q2.

Q3.

Q4.

Q5.

29

29

39

-6

7

Mr. Youta Takano
He is 5th year student at the Departmentof Control Engineering, National Instituteof Technology, Matsue College. His
research interests include educational
material for robots and STEM education

Due to the nature of the mock classes held this time, it
is difficult to conduct a long-term survey such as afollow-up survey, and the shortcoming is that it isinevitably a short-term survey. In addition, because therecruitment method was open-ended, the results werelikely to be influenced by the characteristics of the subject
population. For example, if the subjects were junior highschool students planning to take entrance exams todifficult-to-enter private high schools, the results of thesurvey would have changed significantly. Therefore, it isnecessary to gather subjects with similar characteristics
and ages, and to conduct long-term observation using
their parents' and academic records.

Dr. Kenji Kimura
He is a Lecturer at the Department ofControl Engineering, National Institute ofTechnology, Matsue College, and a
Visiting Associate Professor at Chuo
University. He received his ME(mathematics) from Kyushu University in2002 and his PhD (engineering) from the
Kyushu Institute of Technology in 2020.
His research interests are spherical mobile robots.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
614

Development of visual inspection system for low-reflective material utilizingof a string shadow
Keiji Kamei, Tomorou Kawahara

Department of Information Systems, School of Engineering, Nishinippon Institute of Technology
1-11 Aratsu, Kanda, Miyakogun, Fukuoka 800-0394, Japan
Yoshiyuki Daimaru

Nissan Motor Kyushu Co. Ltd., 1-3 Shinhama-cho, Kanda, Miyakogun, Fukuoka 800-0395, Japan
Email: kamei@nishitech.ac.jp

Abstract
Visual inspection of products in manufacturing plant is usually conducted by workers. However, that inspection
process by workers occasionally happen oversight due to careless check, so that automatic visual inspection system
is needed from manufacturer. Products made of low-reflective material are difficult to inspect by light reflection
because contours of defect areas are got blurred. To solve this problem, we propose to inspect products utilizing of astring shadow. Form our research, a string shadow was different between contours of defect area and its of non-defect.
On the other hand, former research did not success to detect defect area due to determination algorithm for defect. In
this paper, we apply linear approximation of a string shadow to defect detection. From the experimental results, we
succeed in detecting defects.
Keywords: Visual inspection system, low-reflective material, string shadow

This research aims to construct a visual inspection
system that can detect defects whose position is not fixed
and whose shape is unknown. In general, such visualinspections detect defects based on the reflection of linear
light such as fluorescent lights. On the other hand, thisstudy targets the products made of low-reflective
materials which do not reflect lights; so that, our visualinspection system is not able to use reflection of lights.
To overcome this difficulty, we propose to utilize a string
shadow to detect defective area. The shadow of string has
the property of changing shape based on distortion of thematerial when projected onto low-reflective materials.
Our method uses this property to detect distortion of lowreflective materials based on the shadow shape of string.
In this research, our method can flexibly detect bycharacterizing the defective area utilizing a string shadow
and determination the defective area not by patternmatching but by the distance to the approximate linear
line of a string shadow. From the experimental results, we
succeed in detecting defects on material.

1. Introduction
In the manufacturing industry, visual inspection in
manufacturing plants is used to ensure the quality of
products and parts. Most of the current visual inspections
are performed by workers. However, visual inspection
process by workers occasionally causes human errors
such as variation in pass/fail criteria for defective parts oroverlooking defective parts due to individual differences.
For those reasons, demand for development of visualinspection system by image processing has been
increased [1]. A template matching inspection method
based on image recognition is an automated method ofvisual inspection [2], [3]. In inspection by templatematching, the shape and location of the inspection target
must be fixed. That method is an effective method if
inspection items are simple and will not change
significantly in the future. However, it is not suitable forthe visual inspection system because it requires large costwhen there are complex decisions, multiple types ofinspection items or flexible responses to market trends.

2. Method for defective detection

In recent years, many studies has been conducted on
product inspection systems that apply image recognition
based on artificial intelligence(AI) [4]. In fact, we havebeen conducting research on image recognizers using
artificial intelligence in our research [5]. However, AIbased product inspection systems cannot explain thereasons for wrong decisions due to the problem of the AI
being a black box [6], [7]. So, this research does not apply
AI for visual inspection system.

We introduce the method of defective detection utilizing
a string shadow in this section.
2.1. Characterization from a string shadow
Out method for visual inspection is to light on a stringthat placed horizontally to the inspection target and toform a string shadow on it, then that judges defective or
non-defective from form of a string shadow. Fig. 1(a)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
615

Keiji Kamei, Tomorou Kawahara, Yoshiyuki Daimaru

shows the shadow of a non-defective material and (b) the
shadow of a defective material.
For preprocessing, the shadow image from the camera
is denoised by Gaussian filter, binarized, and then edgesmoothed again by Gaussian filter by OpenCV [8] asshown in Fig. 2.

(a)Defective area

Fig.3 Positional relationship.

(b)
Fig.1 String shadow. (a)non-defective. (b)defective.

defect detection is based on the sum of relative distances
between shadow and approximated linear line.

Non-Defective area
Fig.2 Example of after preprocessing.

Fig.4 Defective shadow after preprocessing.

2.2. Environment for inspection

When width of linear approximation is 𝛥𝑦 , theexamples of linear approximated line by start point 𝑦 andend point 𝑦 + 𝛥𝑦 are shown in Fig. 5. A wide black area
is created between approximated linear line and shadow
contour in defective case as Fig. 5(b).

The string shadow is affected by positions for light,
camera, string and material as follows.
1.

2.
3.

4.

Defective area

The closer the distance between light and a string,the more the light diffuses and a shadow becomes
more fading.
The closer the distance between a string and thematerial, the darker the shadow of a string.
When the distance between light and a string isabout 45 cm, light is not diffused and shadow isclearly projected.
The smaller the angle between the camera and thematerial, larger the change in the shape of the
shadow of the defective area.

(a)(b)
Fig.5 Differences of shadow. (a)non-defective.
(b)defective
When the approximated linear line is 𝑓(𝑦) and the
shadow curve is 𝑔(𝑦), gradient a of the approximatedlinear line is Eq. (1). Thus, linear line, 𝑓(𝑦) ,approximated equation is Eq. (2).

From these features for a string shadow, the positional
relationship is chosen to have the largest change in the
shape of shadow of defective area, as shown in Fig. 3.
2.3. Linear approximation of a string shadow andjudgement based on sum of distance between line andshadow𝑎=𝑔(𝑦0 +∆𝑦)−𝑔(𝑦0 )∆𝑦(1)

𝑓(𝑦) = 𝑦0 + 𝑎(𝑦 − 𝑦0 )(2)where, 𝑦0 is start point of linear line. The sum of relative
distances between shadow and approximated linear line
is defined as,

Fig. 4 shows that area of non-defective shadow has analmost constant gradient, whereas one part of gradientchanges for the defective one. When the contour on theupper side of shadow is approximated by linear line,𝑦+∆𝑦𝑆 = ∫𝑦|𝑓(𝑦) − 𝑔(𝑦)|𝑑𝑦(3)

𝑆 in Eq. (3) in defective case is larger than nondefective case from Fig. 5. 𝑆 is obtained by shifting thestarting point, then the sum of 𝑆 is used to determine nondefective and defective. Since this method makes relative©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
616

Development of visual inspection

judgments rather than absolute judgments, it is expected
to have a high degree of flexibility to suit a wide varietyof shapes. On the other hand, a problem with this method
is the possibility of successfully approximating the defectarea as a linear line depending on the value of 𝛥𝑦. To
solve this problem, starting point 𝑦 for approximatedlinear line is shifted several times to vary the sum of therelative distances, then determination of non-defects anddefects is based on comprehensive results. This will avoid
inspection errors by considering approximations which
are not coincidental successes of linear line
approximations.

The same variations are also observed in case ofdepression. From these results, it can be said that the
shadow contours are characterized by a large gradientvariation in the defective area.
The product to be inspected in this study does not havea horizontal surface when viewed from the side. In
addition, because the height varies depending on theinspection location, the shape of shadow and the location
where it is formed are continuously altered. A method ofdetermining by the rate of change of gradient could beconsidered, but it is difficult to define determination
sections because gradient varies depending on theinspection location and the shape of the product and sizeof defect.

3. Experiments
We first examine the variation of shadow contour withand without defects, and then examine the difference inthe sum of relative distances between approximatedlinear line and a shadow contour with and without defects.

3.3. Defect detection based on the sum of relative
distances between a string shadow and linear line
approximation
In this experiment, two non-defective shadow images
are utilized for each defect, shown as Fig. 8(a), to also
verify the sum of the relative distances of non-defective
shadows, shown as red and green area in Fig. 8(b).

3.1. Sample of visual inspection
Fig. 6 shows the non-defective and defective samples
used to be inspected. The defective sample has two kindsof defectives with different concavities, i.e., dent anddepression.

(a)(b)
Fig.8 Example of viewpoint. (a)dent defective. (b)nondefective(a)(b)
Fig.6 Samples. (a)non-defective. (b)defective

The results of linear line approximation of dent defectand non-defect are shown in Fig. 9 with the starting pointof approximation set to 0 and 𝛥𝑦 set to 200. Fig. 10shows the actual coordinates of a string shadow.

3.2. Comparison of shadow contour
Fig. 7 shows the variations in shadow contour
coordinates with and without a dent defect. The figure
shows that gradient varies linearly without dent, while
gradient with dent varies significantly around y at 500 to
700, which is the defective area, in the case of dent defect.

Fig.9 Liner line approx. for “Dent defect”
From figures, there is a large deviation between theapproximated linear line and the actual coordinates fordefect and non-defects with y-axis between 500 and 600.
The same deviation is also happened in case of depression.

Fig.7 Comparison results of shadow contour(dent)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
617

Keiji Kamei, Tomorou Kawahara, Yoshiyuki Daimaru

Table 1 shows the results of comparison for the sum of
relative distances between a string shadow and linear line
approximation in case of dent and depression defects.

4. Discussions
Real-time processing is required in the production
environment, and the camera frame rate is 30 fps, so
processing must be completed within 33[ms] before thenext image is received. The frame rate at 200[kpx] is
20[ms], on the other hand, at 900[kpx] of it is 55[ms];consequently, a frame rate drop occurred. So, it can be
expected that current PC can support up to about
500[kpx], otherwise stronger PC is needed.
5. Conclusions
In this study, defect images are characterized by the
shadow of a string to develop a defect detection method
for low-reflective materials, and defect features aredetected not by pattern matching but by the sum of
relative distances to an approximated linear line. Fromthe experiment, it is effective to shift the starting position
multiple times because the sum of relative distance varies
greatly depending on the starting position of theapproximated linear line. It can also be said that defectdetection is possible by using the maximum value as adefect determination criterion. We aim to further improve
the system by conducting experiments in actual
environments and trying out various kinds of samples.

Fig.10 Before liner line approx. for “Dent defect”
Table 1. Comparison of the sum of relative dist.

Sum ofrelative dist.

Kind ofdefectdent
depression

Nondefect1340
458

Nondefect2344
394

defect668
576

Table 1 shows that the sum of relative distance withdent is about 300 larger than without, and about 150
larger for depression. It can be said that the sum of
relative distances is smaller because of the gradualchange in shadow in case of depression defect. For this,
it is shown that linear line approximation successfully forslowly varying defects and might fail to detect defects.
To solve this, starting point 𝑦 for approximated linear
line is shifted, mentioned in Section 2.3. Let 𝑑 be thedistance of the start point 𝑦 from the image origin andshift 𝑦 19 times in total by +10 in the range of 𝑑 = 0 to
190, then obtain the sum of relative distances for 20 times.
Table 2 shows the results of shifting of starting point.

Acknowledgements
This work was supported by NISSAN Motor Kyushu
Co., Ltd, Japan. We also appreciate Nissan Motor
Kyushu Co., Ltd. for providing various test samples,computers, cameras, lights, and other equipment.
References
1.

Table 2. Results of shifting. ND1, ND2 and Df
correspond to “Non-defective 1”, “Non-defective
2” and “defective”, respectively.

Avg.
Max
Min

ND1

dent
ND2

Df

ND1

448.5

459.4

582.3

406.0

399.3

565.4

653
304

696
271

881
435

662
234

546
252

828
341

2.

depression
ND2
Df

3.

4.

Table 2 shows that the mean, maximum, and minimum
values differ by about 100 or more between theundistorted and the distorted samples. In particular, it canbe said that defects in products can be detected by using
the maximum value as a criterion for determination
because the maximum value differs by about 200
depending on occurrence of defects. From these results,
defect detection can be performed flexibly for defectdetection problems that cannot be solved by templatematching.

5.

6.

7.

8.

Newman, Timothy S., and Anil K. Jain. “A survey of
automated visual inspection,” Computer vision and image
understanding 61.2 (1995): 231-262.
Brunelli, Roberto. “Template matching techniques incomputer vision: theory and practice,” John Wiley & Sons,
2009.
Kamei, Keiji and Moriyama, Hiroyuki, “Recognition
Method of Target Objects for Autonomous Tomato
Harvesting Robot,” Proc. 2019 Int. Conf. on Artificial Lifeand Robotics, (ICAROB2019), pp.557-560.
Huang, Szu-Hao, and Ying-Cheng Pan. “Automated visualinspection in the semiconductor industry: A survey,”
Computers in industry 66 (2015): 1-10.
Kamei, Keiji, et al. “Development of the Image-based
Flight and Tree Measurement System in a Forest using a
Drone,” J. Robotics Netw. Artif. Life 7.2 (2020): 86-90.
Hachiya, Tatsuma and Kamei, Keiji. “Factor identificationand prediction of power demand using sparse modeling AI,”
Proc. the 12th Int. Conf. on Innovative Computing,Information and Control, (ICICIC, 2023), pp. 71.
DARPA. “Special Issue:DARPA's Explainable Artificial
Intelligence (XAI) Program,” APPLIED AI LETTERS,
Vol.2, Issure 4, 2021.
Adrian Kaehler and Gary Bradski. “Learning OpenCV 3,”
O'Reilly Media, Inc., 2016.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
618

Powered by TCPDF (www.tcpdf.org)

Development of visual inspection

Authors Introduction
Dr. Keiji Kamei
In 2007 he obtained his PhD at the
Department of Brain Science andEngineering, Kyushu Institute ofTechnology (Engineering). From
April, 2007 to March, 2014, he was alecturer, and from April, 2014 toMarch 2020, he was an associate
professor. April, 2020, he has been a professor,
Nishinippon Institute of Technology. His research interests
are Artificial Neural Networks, Artificial Intelligence,
Artificial Life and Computer Science. A member of JNNS,APNNS, IPSJ, IEICE, SOFT.
Mr. Tomorou Kawahara
He is a bachelor research member inKamei laboratory at Nishinippon
Institute of Technology. His researchinterest is image processing.

Mr. Yoshiyuki Daimaru
He is an employee of Nissan Motor
Kyushu Co. Ltd. and a co-researcher on
the corporate side of this research.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
619

Development of IoT-Based Remote Monitoring Module for Greenhouse Environment to
Facilitate Crop Growth Data Analysis
Moeko Tominga, Yasunori Takemura, Junya Era, Wataru Kaishita

Nishinippon Institute of Technology
1-11, Aratsu, Kanda-machi, Miyako-gun, Fukuoka, 800-0394, Japan
Eiji Mizoe, Tomoyasu Furukawa

Kumamoto Fruits and Vegetables Shipping Association Co. Ltd
3523, Okimachi, Yatsushiro-city, Kumamoto , 866-0013, Japan
Email: tominaga@nishitech.ac.jp

Abstract
As the shortage of agricultural population is becoming an issue, services that remotely monitor the agriculturalenvironment using IoT are being offered in the market. However, small- and medium-scale private farmers are unable
to purchase or rent IoT equipment and monitoring tools because they have to spend their budgets on maintenance andutilities in greenhouses to support environmental management, and the construction of data infrastructure has notbeen widely adopted. Therefore, in this experiment, we developed an agricultural IoT that can be provided to medium
and small farmers, and a module that enables collection, accumulation, and visualization of time-series data acquired
from sensors. The module provides an architecture that includes multiple sensing, networking, processing, andvisualization applications to keep deployment and management costs low. To demonstrate its feasibility, the proposed
IoT system was built, experimentally tested, and verified in a greenhouse in Yatsushiro, Kumamoto, Japan, byintermittently monitoring temperature, humidity, carbon dioxide concentration, solar radiation, and soil moisture
content in the greenhouse for two years. The results obtained confirm that the proposed IoT remote monitoring module
facilitates farmers to monitor the growing environment of their crops and enables them to improve their productivity
by progressively increasing their technology level.
Keywords: IoT，Agricultural, Monitoring System, visualization

productivity with fewer people. It is desired to acquire
environmental information on farms, which is a factor
that changes the quality and quantity of agricultural
products, and to visualize this information. In addition,
one of the challenges for future development is to analyze
the acquired data and weather data and utilize them as
time-series data for forecasting, thereby improving
productivity and ensuring the production of high-quality
and stable agricultural products [3]. Here, an example ofa farmer growing banana and coffee in Yatsushiro,
Kumamoto Prefecture, Japan, is presented. Banana andcoffee are mainly grown in tropical regions. Japanbelongs to the temperate zone, and farmers across thecountry are searching for new cultivation methods toenable the cultivation of tropical plants such as banana
and coffee in temperate Japan. Farmers in Yatsushiro
City, the site of this experiment, also grow bananas andcoffee and maintain a tropical environment in vast plastic
greenhouses. However, in Japan, which has four seasons,temperatures vary throughout the year and it is difficult
to maintain a constant environment inside thegreenhouses.

1. Introduction
Smart Agri and Agri Tech are attracting attention as
new forms of agriculture that utilize advanced
technologies such as robotics, ICT to enable labor-saving,high-quality production [1]. One of the technologies that
support this Smart Agri is the spread of the Internet ofThings (IoT), whereby various "things" are now
connected to the internet IoT refers to a system whereby
sensors are built into various "things", which are thenconnected to the internet, and a large amount of dataacquired via the internet It is a mechanism whereby large
amounts of data acquired via the internet are stored in thecloud. This accumulated large amount of data is known
as 'big data' (BD), and although BD cannot create value
simply by accumulating it, new value can be created byanalyzing the accumulated data [2].
In agriculture, where declining productivity due to anageing population and a lack of bearers is a problem,
there are high hopes for this IoT as a way to improve©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
620

Moeko Tominaga, Yasunori Takemura, Junya Era, Wataru Kaishita, Eiji, Mizoe, Tomoyasu Furukawa

However, in Japan, which has four seasons, thetemperature varies throughout the year and it is difficult
to maintain a constant environment in the vast
greenhouses, which affects the growth and production of
the plants. For this reason, many companies are
introducing agricultural IoT, collecting environmental
data in greenhouses and providing visualization services.
However, small- and medium-scale individual farmers
have to spend their budgets on the maintenance of theirgreenhouses, utility costs and other costs to support
environmental management, and are unable to purchaseor rent IoT equipment and monitoring tools, which meansthat the construction of data infrastructures has not
become widespread. Therefore, in this experiment, wedeveloped an agricultural IoT device that can be provided
to medium and small farmers, and built a system forcollecting, storing and visualizing time-series dataacquired from sensors.

The developed IoT device is shown in Fig. 2 and thespecifications are given in Table 1. As an MicroController Unit (MCU), the Arduino, which is also usedin other smart farming systems [4], [5], was used. The
developed IoT device was equipped with several types of
sensors to collect data in order to investigate theenvironmental information inside the greenhouse from allperspectives.
In order to develop an IoT device that can continue tooperate even in a harsh environment, a temperature andhumidity sensor was also attached to the inside of thegreenhouse, and a durability test was conducted toinvestigate at what temperature range the device would
break down. The results showed that failures occurred ataround 50 °C, so a fan was installed inside the chamber
to prevent failures due to overheating of the IoT device.
In addition, a Watch Dog Timer (WDT) was incorporated
into the software to monitor whether the MCU program
stopped or communication was interrupted, and if so,measures were taken to restart the program. By taking
countermeasures in terms of hardware and software, wewere able to develop an IoT device that could continue tooperate even in a harsh environment.

2. Development concept
The concept was set up as follows.
•

•
•
•

To collect on temperature, humidity, carbon,
dioxide concentration, moisture content and solar
radiation in order to investigate environmental
information in greenhouse from all perspectives.
Designed to continue to operate even under severe
environmental conditions and to ensure that dataacquisition is not interrupted.
Programmed to be able to be used universally atother sites, and designed so that sensors can beconnected using connectors.
Collect agricultural IoT devices that meet theabove conditions using inexpensive and highly
accurate sensors.

Sunlight
sensorCase with heatdissipation
MCU &Communication shield
Barometricpressure sensor

Temperature & Humidity
sensorCarbon dioxide
sensorSoil
sensors

A system of IoT modules configured according to theconcept is shown in Fig. 1. All sensor values are regularlycollected by microcomputer and stored in storage in thecloud. The user can view the values of these sensors
numerically and graphically at any time and fromanywhere.

Dig up & bury

Fig. 2 IoT devices developed (being implemented in agreen house in Yashiro).
Table 1 Specification of agriculture IoT module
Size
280.5 × 170 × 130(Length x width x depth)
MCU
Arduino Uno
Supply voltage
12[𝑉]Long Term Evolution
Line
(LTE)
Temperature sensor
Humidity Sensor
BME280
Barometric pressure sensor
Carbon dioxide sensor
S-300-3V
Soil moisture censer
WD-3-W-5Y
Sunlight sensor
PVSS-03

Fig. 1 System image of the IoT module©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
621

12-22202311:45:50
12-22

100.0%

35.2%

100.0%

35.2%

11:45:501033

1220237 50 22
p pm

7 50 p pm

12 22

55 00
19.8
00 pp pm
pm °C

99 7%

35 7%

99 7%

35 7%

pp m

227.0

22 50
50 pp pm
pm

202300 pp pm
pm

12-22
11:43:49
11:43:49
CC
2023CC
202312-22202312-22
12-22

11
32.3%
11 :0
:0 00
32.3%

32.3%
32.3%
100.0%

0 0098

[%LU
][ppm
X ]
[%LU
][p Xpm ]

206.0
1:10
16.911 1:10
°C
206.0
ppm

212.0
237.0
pp m

72.00

[%LUX
[pp
] m]
0.01950.0195
0

19.5

71 00

17 20

19.5

71 00

17 20

33.7 %[% ][% ]1:30 68%
1001033.5% 11 1:30
1033.5
hPa

[% ]
[% ]

[℃ ]
[℃ ]
[℃ ]

11
11 :5
:5 00 18.00
35.2 % 19.8
19.8
°C

11 1168.00
:4
:4 00

68%
m
m in
in

hPa

16.90

°C

0
73%

16.90

°C
19.4

LU X

[hPa ][hPa ]

11:2
0.0146
11:2 00
0.0146ppm[% 212.0
]

72.00

74%

73%

1033.40 009874%

72 % H

1033.432.3 %

11:45:50

68.00
m
m ax
ax

227
227 ppm
ppm

246
246 ppm
ppm

1033.6[hPa] 206
ppm
206
ppm
68%
1033.6hP
68%1033.3hP
74%
hPa

aa vv gg

18.00
cu
rre
cu
rre nn tt

237
237 ppm
ppm

237
237 ppm
ppm

217
217 ppm
ppm

206
ppm
206
ppm
17.80

°C

[% ]

[%19.8
]

231
ppm
231
ppm
69.00

[℃ ]

[℃ ]
17.80
16.90

19.8
°C
19.4
°C

69.00
72.00

Developed of IoT-Based Remote

°C

LU X

Wm/m
/m ^^ 221 0 0011 pkkpW

1034 hPa20231000
pp m
00 .7
50
W
7.7
50
pkkpm
5022
W /m
/m ^^ 22
12

hPa

74 %

227.0

33.7%
33.7%

33.7%
33.7%
35.2%

hPa
1033.3

0

18 °C
[% ]
[% ]

202312-22 °C
19.4

0

ppm
237.0

237 p p m

11 00202300
00 pp pp m
m hPa

206 pp m
99 7%

227.0

35 7%

68 %

68 % H

1033.40 0098

4. Field experiments and time series data obtained

0.0146
73%

19.5

71 00

17 20

750 ppm
00 5.500
W.500
00 pkkpm
W /m/m ^^ 22500 ppm

50 pkpm
00 2.2.2 50
50 k WW /m/m ^^ 22250 ppm
0 p pm202300 kk W
W /m/m ^^ 22
0 ppm

19.8
12-22 °C
11:43:49

[% ]

C

C
2023C

12-22
C

32.3%

212.0

33.7%

12 22

k W /m ^ 2
00 ppm0%0%0 kW /m ^2

100.0%

35.2%

99 7%

35 7%

ppm

11 :0 0
11:00
11 1:0
1:0 00 11:00

227.0

11 :1 0
11:10
11
11 :1
:1 00 11:10

[% ]

C

2023C
12-22

32.3%

33.7%

32.3%

33.7%

206.0
ppm

11:43:49
1 20230 0%1 kW /m ^2
11 00 0%
0%
12-22
100%

[hPa ][% ][% ]The developed agricultural IoT equipment was actually
installed in a plastic greenhouse in Yatsushiro to monitor
the growing environment of banana and coffee as shownin Fig. 5. The monitoring period started intermittently in
2021 and lasted for two years until 2023. The datainterval was 15 minutes. An example of the data collected
is shown in Fig. 6. This is temperature data, where
temperatures inside and outside the developed
agricultural IoT module were collected for one month. As
can be seen from Fig. 6, sensor values were successfully
retrieved and stored in cloud storage at 15-minuteintervals. The sensor values also show that the internal
temperature tends to be 1-5 ℃ higher than thetemperature outside the IoT module, but is kept almost
the same due to heat dissipation.

[℃ ]1 1:301 1 :4 0:5 019.811
11
:4
11
11 :3
:3 00 68%
11
:4 00
11 :5
:5 00 18.00
32.31033.5
% 11
33.7
%
68.00
11:40
hPa 11:30°C a v 11:50
m in
m ax
g
cu rre n t

m
m
av
cu
m in
in
m ax
ax
av gg
cu rrerre nn tt
in
ax
vg
cu
nt
227mppm
246mppm
237 appm
237rreppm
00 kW
0.01950.00684
00 kW/m
kW /m
/m ^2
^2
0.0195 kW/m
kW/m ^2^2
0.00684 kW/m
kW/m ^2^2
kW/m ^2^2
227 pp m
246 p p m
2 37 ppm
237 p p m
206 ppm
231 ppm
217 ppm
206 ppm
1033.619.8
0.0146kW/m^2
0.0488kW/m^20.0244kW/m^20.0146kW/m
0.0146 kW /m ^2
0.0488 kW/m ^2
0.0244 kW/m ^20.0146 kW/m ^2^20.0195
68%
69.00
17.80
206
pp m
231
ppm
2 17 ppm
206
ppm

212.0

[hPa]
1033.30hPa1033.40 0098
11 :2 0
11:20
11 11 :2
:2 0011:20

1 1:0 0 11:00
11
:0
11
:0 00
11:00

11 :1 0 11:10
11
:1
11
:1 00
11:10

11 :0 01 1:10

[% ]

74%

72.00

73%

71 00

[℃ ]
[℃ ]

19.4

16.90

°C

19.5

17 20

11 :4 0
11 :5 0
11:40
11:50
11 1:4
00
11
:5
1:4
11
:5 00
11:40
11:50
m in
m ax
cu rre n t
m inm ax av g a v g
m
in
m
aa xx
av g cu rre
nt
m[%
in ]
mav
rre
[%LU
][ppm
X ] m in[hPa ] m ax[%[℃
g ] av g cu
curre]nn tt
0 kW /m ^20.0195 kW/m
^20.00684
kW/m ^2
kW/m^2
227 ppm
246 ppm
237 ppm 0 237
ppm
99.7%
100%
99.8%
100%
99.7%
] ^2
0 kW /m ^2
0 .01 95 kW /m ^2
0.00
684 k W100%/m ^ 2 99.8%0 [℃
k W100%/m
0.0146 kW /m ^2
0.0488 kW/m
^20.0244
kW/m ^2
0.0146 206
kW/m^2
206 ppm
231 ppm
217 ppm
ppm
35.2%
35.7%
35.5%
35.2%
1033.519.80.0 146 k W35.2%
35.2%
35.7%
0 .0146 kW /m ^2
0 .04 88 kW68%
/m ^2
0.0
244 k W/m ^ 2 35.5%/m ^20.0146
68.00
18.00

hPa

°C

1033.60.0195hP

1 1 :2 011:20
11 11 :2
:2 00
11:20

2 50 p pm
0 p pm[% ]11 :3 011:30
11 1:30
1:30
11:30

7 5% kW /m ^2
0.750
77 5%
5%
75%
0.500
5 0% kW /m ^2
55 0%
0%
50%
1 0 00 p p m
0.250
2 5% kW /m ^2
22 5%
5%
25%
7 50 p pm
0%0 kW /m ^20%0%0%p pm
5 00

°C

[%LUX
[ppm
]
]
237.0

0 .5
00 k W /m ^ 2
11:45:50500
55 0%0% ppm0.500 kW /m ^2
0 .2
50 ppmk W /m ^ 2202325022 5%5%0.250 kW /m ^2

11:2 0

11
11 :2
:2 00
0.0146
11:20

hP

[% ]

1 k W /m ^ 2
11 00 0%0% pp m1000
1 kW /m ^2

0 2023.7 50 k W /m ^ 2
7750
5%712-22
5% ppm
0.750
kW /m ^2[%LU][p Xpm ]

[℃ ]

18

11 :0 01 1:10
206.0
11
11 :0
:0 00 33.7% °C 11
11 :1
:1 00
32.3%
11:00
11:10
ppm

11:30
1 1:30
11 1:30
1:30
11:30

m in

68%

0.0195 kW/m ^20.0146 kW /m ^2
11:2 00.0488 kW/m ^2
1 1:30

17.80

°C

11:40
1 1:4
0
11 1:4
1:4 00
11:40
m in
mav
a xg
m
in
m
ax
m
m
m in
in
m aa xx
0.00684^2
99.7% kW/m
100%
32.3%
32.3%
32.3%
32.3%
99.7 %
1 00%
0.0244 kW/m
^2
35.2%
35.7%
33.7%
33.7%
1 1 :4%0
33.7%
33.7%
35.2
3 5.7%

m ax

0 kW /m ^2

19.8

69.00

11:50
11
:5 0
11
:5
11
:5 00
11:50
av g cu rre n t
av
g cu rrent
av
av gg cu
cu rrerre nn tt^2
99.8%0 kW/m
100%
32.3%
32.3%
32.3%
32.3%
99 .8%
100%
0.0146 kW/m
^2
35.5%
35.2%
33.7%
33.7%
11.5%
:5 0
33.7%
33.7%
35
35 .2%

m in

m ax

avg

cu rre n t

C

227 ppm

246 ppm

237 ppm

237 ppm100%
1 0 0% C
11100%
00 00 %
%H
H

206 ppm

231 ppm

217 ppm

206 ppm

However, as a problem, the phenomenon of system
stoppages could appear. When the system is restarted, it
returns to a state where data can be continuously retrieved
again. The data in which this phenomenon appears isshown in Fig. 7. The period during which the sensor
values are not updated is the period during which thesystem is stopped. This has not yet been resolved. As acountermeasure, the system was monitored and improved
so that it can be restarted remotely in the event of a
stoppage. After the improvements, the system stoppages
have not been reproduced, but the program is being
logged so that the cause of the reproduction can beclarified.

75%
7 5%
7775%
55 %
%H
H
50%
5 0%
5550%
00 %
%
HW /m ^ 2
1 kH
25%
2 5%
5550%
% kH
HW /m ^ 2
0 22.725%0%% kH
HW /m ^ 2
0 .5000%
00%

11:00
11
:0 0
11
11 :0
:0 00
11:00

11:10
11 :1 011 11 :1
:1 00
11:10

11:20
1 1 :2 0
11 1:2
1:2 00
11:20

11:30
1 1:3011 11 :3
:3 00
11:30

11:40
11:50
1 1:4 0
11 :5 0
11 1:40
11
:5
1:40
11
:5 00
11:40
11:50
m in
max
av g cu rre n t
m
xx g cu rreavg
m inin
m aaav
avg
m in
max m
nt
99.7%
100%
99.8%
100%
32.3%
32.3%
32.3%
32.3%
H
74
%
H
72.2
%
71 3%
%2.3%
H
74 32
% .3%
H
72.2
%H
H
32.3 % 71
32 .3%
35.2%
35.7%
35.5%
35.2%
33.7%
33.7%
33.7%
33.7%
H
72
%
H
%
11 :4
11.7%
66 3%
%3.7%
H
72 33
%
H:5 0 68.6
68.6
%H
H
33.7
%0 66
33 .7%

0 .2 50 k W /m ^ 2
0 k W /m ^ 2
11 :0 0

11 :1 0

11 :2 0

11 :3 0m in

100%
100 % H
11 00H00 %
%H
H
100 %

m ax

av g

cu rre n t

0 kW /m ^20.0195 kW/m ^20.00684 kW/m ^2

0 kW/m ^2

0.0146 kW /m ^2

0.0488 kW/m ^2

0.0244 kW/m ^20.0146 kW/m ^2

75%
75 % H%H
H
75 %77H55 %50%
50 % H%H
H
1 50
0 0%%55H00 %25%
25 % H%H
H
25
7 5%%22H55 %
0%
0 %H
%H
H
0 % H00 %
5 0%

11:00
11 :0 0
11 :0 0
11:00

11:10
1 1 :1 0
1 1:1 0
11:10

11:20
1 1:2 01 1:2011:20

11:30
1 1 :3 0
1 1 :3 0
11:30

1 1:0 0

11 :1 01 1 :2 0

1 1:30

11:40
11:50
1 1:40
11 :5 0
11:4 0
11 :5 0
11:40
11:50
m in
nt
mm
in a x m aav
x g cu rreavg
avg
m in
mmainxcu rreavg
nt
avg
m ax
32.3% 71 32.3%
32.3%%H
74 32.3%%H
72.2%H
64 % H69
66.8
71 % H
74 % %
H H 7268.2 % H
33.7% 66 33.7%
33.7%%H
72 33.7%
%H
68.6
%H
65 %
H 0 68
66.8
1 1:4
11%
66 % H72 %H:5H0 6868
.6 % H

2 5%
0%

ho
hi

m in

max

av g

99.7%

100%

99.8%

100%

35.2%

35.7%

35.5%

35.2%

cu rre n t

Fig. 4 Visualization using SORACOM Lagoon( Only some data on coffee was extracted)
100 %1 0H0 % H
5 0 °C100 % H

4 75
0 °C%7H5 % H
75 % H

3 050°C%5H0 % H
1 0 0% 50 % H
22 00 °C°C
25 %2H5 % H
25 % H
1 70 5%°C
0 % H0 % H
05 0%°C 0 % H

3. System configuration diagram
11:00
11 :0 011
11 :0
:0 00 11:00

11:10
1 1:1 011 11 :1
:1 0011:10

11:20
1 1:20
11:20
11:20
11:20

11:30
1 1 :3 0
11:30
11:30
11:30

2 5%

ho
ho
ti
ti
0% hi
hi
to

11:40
11:4 0
11:40
11:4
00
11:4
m in
mmainx
m in
mma xin
71
64 % H69 %%H
H
64 % H
69
19.1
% H°C
19.1
°C
66
65 % H
68 %
%H
H
65 %
H 0 68
16.9
% H°C
1 1:4

11:50
11 :5 0
11:50
11
:5
11
:5 00
m ax
avg
cu rreavg
ntavg
m ax cu rreavgnt74%
H H 72.2
66.8 %
68 %
%H
H
66.8
19.5
6 8 %°C
H
19.5%°C°CH 19.3
19.3°C72
H H 68.6
66.8%%
68 %
%H
H
66.8
17.5
°CH0 17.2
6 8 %°CH
11%:5

The system configuration is shown in Fig. 3. The dataacquired from each sensor connected to the MCU was
A/D converted and the time-series data was stored in thedata server via the LET module. The data server used
SORACOM Harvest provided by SORACOM, andSORACOM Lagoon (Fig. 4) was used as thevisualization tool.
11 :0 0

11 :1 01 1 :2 0

1 1:305 0 °C100 % H
50
5 0 °C°C
4 0 °C 75 % H
40
4 0 °C°C

3 0 °C30
3 0 °C°C 50 % H
12 0 0°C% H
20
2 0 °C°C
25 % H
1 07 5°C% H
10
1 0 °C°C
0 °C 0 % H
005 0°C°C% H

11 :0 0 11:00
11:00
11 :0 025 % Hhotititi
0 % Hhitototo

1 1 :1 011:10
11:10
1 1 :1 0

11:30
11:30
11:30
11:30

max

av g

cu rre n t

32.3%

32.3%

32.3%

33.7%

33.7%

33.7%

33.7%

11:40
11:4
0
11:40
11:4 0m in
mma xin
m
m in
in
69
% H°C
19.1
19.1
19.2 °C°C
65 % H
68
% H°C
16.9
17.5 °C
1 1:40 16.9
°C

64 % H

Data collection device
11 :0 01 1 :1 0

Temperature
50
5sensor
0 °C°C50°C.0 h Pa
1 0 50

I2C

Humidity30°C
13 0 00°C.0 h Pa
30 °C
sensor20
100 % H
2200 °C°C°C

I2C

Barometricpressure
00 °C
°C
09 00
°C.0 h Pa
11:00
50 % Hsensor
11
:0 0
11:00
1 1:00I2C

40
4 0 °C°C
40 °C

9 50 .0 h Pa
101100 °C°C°C 7 5 % H

1 1:0025 % H
ti
ti
ti
a
0 %H
to
to
to
a

11:10
111:10
1 :1 0
11:1

1 1:2 00Soil moistureho
sensor50°C.0hih Pa
11050.00 50
h Pa

AnalogSunlight
111000.0.8°C.0 hh Pa
Pa3009 00
sensor
20 °C

Analog

1 1:1 0

11:20
11:20
11:20
1 1:2 0SORACOM Server

m in

m ax

avg

SORACOM
Lagoon

11:30
11:30
11:30
1 1:30m in

1033.40 hPa
1 1:20

11:50
11
:5 0
11:50
11 :5 0avg
nt
m ax cu rreavg
m
avg
m ax
ax
avg
66.8
68 %°C
H
19.5%°CH 19.3
19.5
9.3 °C19.8 °C°C 119.6°C
66.8
68 %°C
H
17.5%°CH 17.2
17.5
°C
7.2 °C18
11 :5°C0 117.8°C%H
72.2 % H
LTE 71 % H 74SORACOM
66 % H
72 % Harvest
H
68.6 % H

MCU

I2C

20
40 °C

1 1 :3 0

NB-IoT
ShieldCarbon dioxide
sensor
11 :0 051 09 .6°C
9950.0
50 .0 hh PaPa
10
4 0 °C°C
1 9 .4°C
3 009900.0°C.0 hh Pa
00
Pa

11:20
11:20
11:20
11:20

m in

32.3%

1 1hPa:3 01033.20

11:40
11:4
0
11:40
1 1 :4

11:50
11
:5 0
11:50
11 :5 0
m
in
m ax
avg
m
m in
in a v g m
m ax
ax cu rreavg
avg
nt
19.1
°C
19.5
°C 19.3
°C
19.2
19.8
19.6
°C
19.2 °C°C hPa19.8 °C°C
1 9.6hPa°C
1033.60 hPa
1033.501033.50
16.9
°C
17.5
°C
17.2
°C
17.5
°C
18
°C
17.8
°C
17.5 °C hPa 18°C°C
11:4 0 1033.30
11 :5
0 1 7.8hPa
1033.40 hPa
1033.30

0

m ax

m in

m ax

avg

64 % H

69 % H66.8 % H

68 % H

65 % H

68 % H66.8 % H

68 % Hcu rre n t

Fig. 5 Installation of the developed IoT modules(Right: coffee field, Left: banana field,
Top: before growth, Bottom: after growth)Fig. 3 System configuration diagram

12 90 .2°C
1 0 °C ti
1 9 aa11:001 1:00
11:00

11:1011:1 0
11:10

11:20
111:201:2 0

1 1:101 1:2 01 1 :1 0

11:20

11:30
111:30
1:30

m
m in
in

1 1:000 °C toaati Avg: 19.6 11 :0 01033.40
1 033 .4 0 hPah Pa
11 :3 01033.20
1 033 .2 0 hPa
h Pa
11:30

11:40
111:40
1 :4 0m
m ax
ax

1033.60
1 033.6 0 hPah Pa
11 :4 01033.40
1 033.4 0 hPah Pa
11:4 0

11:50
11
:5 0
11:50

m in a v g m ax cu rreavg
avg
cu rre nn tt
19.2 °C
19.8 °C
19.6 °C
1033.501033.50
1 03 3.50 hPah Pa
1 033.5 0 hPahPa
11:5°C0 17.8 °C
17.5 °C hPa 181033.30
1033.30
1 03 3.30 h Pa
1 033.3 0 hPahPa
11 :5 0©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
m in

m ax

avg

ti

19.1 °C

19.5 °C

19.3 °C

1050.02200 toh Pa
1 9 .616.9 °C

17.5 °C

17.2 °C

119.8
1000.0
1 99 .8.5 h Pa
1119.69 .4.6°C
5 90950.0
h Pa
14 90 .3°C

622

Moeko Tominaga, Yasunori Takemura, Junya Era, Wataru Kaishita, Eiji, Mizoe, Tomoyasu Furukawa

Tempreture [℃]35

Inside30

Outside

25
20
15
10

/4
11

5

6

7

8

9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 /1
12
Date

2

3

4

5

Fig. 6 Temperatures observed by IoT devices( 4 November 2023 - 4 December 2023)

Periods when data is not updated to the cloud.

Fig. 7 Temperatures observed by IoT devices(17 January 2022 – 28 December 2023)
3.

5. Conclusion
In this study, an IoT device for agriculture wasdeveloped and actual field observation experiments wereconducted in greenhouses. As a result, a device with thefollowing functions was developed.
•
•
•

•

4.

To continue to operate normally in greenhouses
where the humidity is above 90% and thetemperature exceeds 30°C.
Data is uploaded to the cloud every 15 minutes andcan be immediately visualized.
The device can observe multiple parameters(temperature, humidity, barometric pressure,
Sunlight, soil moisture and carbon dioxide
concentration in the air).
Robust against communication errors.

5.

M. R. Bendre, R. C. Thool and V. R. Thool, "Big data inprecision agriculture: Weather forecasting for future
farming," 2015 1st International Conference on Next
Generation Computing Technologies (NGCT), Dehradun,
India, 2015, pp. 744-750.
V. K. Shukla, R. S. Nair, F. Khan,”Chapter 13 - Smartirrigation-based behavioral study of Moringa plant for
growth monitoring in subtropical desert climaticcondition”, In Intelligent Data-Centric Systems, AI, Edgeand IoT-based Smart Agriculture, Academic Press, 2022,
pp. 227-240.
E. Ogunti, “IoT Based Crop Field Monitoring and
Irrigation Automation System”, International Journal of
Innovative Science, Engineering & Technology, Vol. 6
Issue 3, 2019, pp. 124-129.

Authors Introduction
Dr. Moeko Tominaga
She is an Assistant Professor at the
Department of Integrated System
Engineering, Nishinippon Institute ofTechnology, Japan. Her researchinterests are in multi-agent systems
and field robotics.

In the future, the durability of the system will be
increased by improving the modules so that the system
can run without stoppages for a long period of time.
Acknowledgements
This work was supported in part by Kumamoto Fruits
and Vegetables Shipping Association Co. Ltd.

Dr. Yasunori Takemura
He received his M.S., and Ph.D.
degrees from Kyushu Institute ofTechnology, Japan, in 2006 and 2009,
respectively. In 2010, he joined
Nippon Bunri University. In 2015 hejoined Nishinippon Institute ofTechnology, where he is currently an
Professor of the Department ofEngineering. His research interests
include robotics, IoT and AI
Technology.

References
1.

2.

J. Pretty, “Agricultural sustainability: concepts, principles
and evidence”, Philosophical Transactions of the Royal
Society B: Biological Sciences, Vol. 363(1491), 2007, pp.
447-465.
O. Elijah, T. A. Rahman, I. Orikumhi, C. Y. Leow and M.
N. Hindia, "An Overview of Internet of Things (IoT) andData Analytics in Agriculture: Benefits and Challenges,"in IEEE Internet of Things Journal, vol. 5, no. 5, 2018, pp.
3758-3773.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
623

Powered by TCPDF (www.tcpdf.org)

Developed of IoT-Based Remote

Mr. Junya Era
He worked as a steel plant controls
engineer from 2012-2019 and has
been at Nishinippon Institute ofTechnology since 2020, learning
about the IoT. He is an undergraduate
engineering student. His researchinterests are in IoT Technology.

Mr. Wataru Kaishita
He received his Bachelor's degreefrom the Nishinnippon Institute ofTechnology in 2022 and started his
postgraduate
studiesatthe
Nishinippon Institute of Technology
in 2022. He is currently in his firstyear of master's studies. His researchinterests are in the field robotics.
Mr. Eiji Mizoe
He is currenntry working inKumamoto Fruits and Vegitables
shipping Association Co. Ltd..
(Kumamotoseikabutsu
syukkakumiai).

Mr. Hiroaki Wagatsuma
He is currenntry CEO in Kumamoto
Fruits and Vegitables shipping
Association Co. Ltd.. (Kumamoto
seikabutsu syukka kumiai).

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
624

Development greenhouse environment prediction system using IoT data
Yasunori Takemura, Naoya Nishida, Moeko Tominga

Nishinippon Institute of Technology, 1-11 Aratsu, Kanda-town, Miyako-gun, Fukuoka, 800-0394, Japan
Eiji Mizoe, Tomoyasu Furukawa

Kumamoto Fruits and Vegetables Shipping Association Co. Ltd, 3523, Okimachi, Yatsushiro-city, Kumamoto ,866-0013, Japan
Email: takemura@nishitech.ac.jp

Abstract
Currently, the decrease in the working population is a significant problem in agriculture in Japan. In addition, theprices of imported vegetables and fruits have been rising due to international trade problems such as the weak yen.
This study has developed an environmental data storage system using IoT to assist the labor force and to understandthe cultivation data of crops that have not been cultivated in Japan. Using this big data, we have constructed andevaluated a forecasting system. In this study, we focus on the domestic cultivation of bananas grown in the tropics as
the subject to obtain data and make predictions. For learning, a system was constructed to infer the next day's
temperature by time-series analysis, using as input data obtained from the IoT. In this paper, we report a comparisonof two time series analysis algorithms.
Keywords: Time-series data analysis, ARIMA model, prophet model, IoT, agriculturecaused by the aging of the farming population and thelack of farmers to take on the workforce. Therefore, as asolution, it is expected to improve productivity with asmall number of people, and a necessary method toacquire environmental information on farms, which is afactor that changes the quality and quantity of agricultural
products [3]. As a solution to this problem, visualizationand prediction functions of environmental information
using IoT are expected. One of the challenges in future
development is to analyze the acquired IoT data andweather data and utilize them as time-series data inforecasting algorithms in order to improve productivity
and realize high-quality and stable crop production [4].

1. Introduction
For In recent years, the Internet of Things (IoT) has
become increasingly popular, and various " devices" arenow connected to the Internet. The IoT is a system inwhich sensors are embedded in various " products,"
connected to the Internet, and a large amount of dataacquired via the Internet is stored in the cloud. Thisaccumulated large amount of data is called "Big Data
(BD)," and today there are tens of billions of IoT devices
in the world, all of which continue to generate new data.
BD cannot create value simply by accumulating it, but it
can create new value by analyzing the accumulated data.
In addition, with the development of computer
technology, the computation time of AI (Artificial
Intelligence) has been shortened, and many companies
have developed home appliances and automobiles thatincorporate AI, making the IoT and AI a familiar
presence. As evidence of this, the number of IoT devices
is on an increasing trend with respect to transition data,and is expected to exceed 32.33 billion units by 2022 [1] .
In Japan, the government and companies are actively
working toward the realization of Society 5.0, and it isexpected that a society that uses information in a wide
variety of ways will expand in the future [2].

The purpose of this research is to build an AI using timeseries data with IoT technology, and to generate andvalidate prediction data using time-series data. The
experiment involves time-series prediction ofenvironmental data in a plastic greenhouse in Yatsushiro
City, Kumamoto Prefecture, where tropical crops are
harvested. The time-series forecasting algorithms werecompared using the ARIMA model [5] and Prophet [6],because of their ability to consider the effects ofperiodicity and external factors.
2. Methodology
2.1. ARIMA (Auto Regressive Integrated Moving
Average) model

For example, the utilization of IoT databases is expectedin various fields. One example is in the field ofagriculture, where the number of farmers is decreasing
year by year due to the problem of declining productivity©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
625

Yasunori Takemura, Naoya Nishida, Moeko Tominaga, Eiji Mizoe, Tomoyasu Furukawa

The ARIMA model [5], derived by Box & Jenkins, is amodel consisting of three components: AR (AutoRegressive) model, MA (Moving Average) model, andsummed components (integrated). Time series data may
have non-stationary time series processes when a trend
analysis is performed, which cannot be handled by the
ARMA model. Therefore, non-stationary time series
must be converted to stationary time series and then fit
into the ARMA model. The ARIMA model, which adds
a summation component to the ARMA model, is thenapplied to the non-stationary time series and aims to makethem stationary by taking the “d” th-order difference. The
general equation is shown in Equation (1) below.
𝒑̅)𝟐 √∑𝒏𝒊=𝟏(𝒚𝒊 − 𝒚̅)𝟐√∑𝒏𝒊=𝟏(𝒙𝒊 − 𝒙(3)

It is a statistic that evaluates the predictability of astatistical model using the residuals between observed
and theoretical values, with a smaller value indicating abetter fit. If the log likelihood of the model is L and thenumber of parameters in the model is p, the AIC is
calculated by the following Equation (4). In this paper,
the AIC is used as an evaluation criterion when selecting
parameters for the ARIMA model. L is the maximum
likelihood and p is the number of free parameters. In thepresent model, we adopted a low value of AIC to avoid acomplex model in determining the AR and MA
coefficients in the ARIMA model.

(1)

𝒋=𝟏

In the equation, t denotes time, d, i and j denote the timeat the regression point in time. e indicates the error height,
y is the output, and c,  and  are constants. In recent
years, it has been used as a forecasting system for
COVID-19 [7] , and has also been used in research suchas stock price forecasting [8].

𝐴𝐼𝐶 = −2ln (𝐿) + 2𝑝

(4)

3.3. Coefficient of determination R2
The coefficient of determination, also known as thecontribution ratio, is a measure of the explanatory powerof the predicted value of the target variable relative to theobserved value of the target variable in a regression
analysis. The coefficient of determination is generally
expressed as R^2. R^2 takes values between 0 and 1, andthe closer to 1, the more valid the regression equation is.
When the actual data are (xi,yi), the data estimated fromthe regression equation are (𝑦𝑖 , 𝑦̂),𝑖 and the mean value
obtained from the entire data is ( 𝑥̅ , 𝑦̅ ), the following
Equation (5) is shown.

2.2. Prophet model
Prophet is a library for time series analysis developed byMeta in 2017. The advantages of this model are that it canbe built without statistical knowledge, it can be trainedeven with missing values, and it is easy to interpret theprediction results. Prophet is based on the Generalized
Additive Model (GAM) [9], which is a model that uses a
It predicts future values by modeling and combining fourterms: the trend term (growth: g(t)), the seasonality term(seasonality: s(t)), the holidays effect term (holidays: h(t)),
and the error term ε(t)). The model equation is shown inEquation (2) below.

𝒚 = 𝒈(𝒕) + 𝒔(𝒕) + 𝒉(𝒕) + 𝜺(𝒕)∑𝒏𝒊=𝟏(𝒙𝒊 − 𝒙̅)(𝒚𝒊 − 𝒚̅)

3.2. Akaike’s Information Criterion (AIC) [10]𝒒𝒚𝒕 − 𝒚𝒕−𝒅 = 𝒄 + 𝒆𝒕 + ∑ ∅𝒊 𝒚𝒕−𝒊 + ∑ 𝜽𝒋 𝒆𝒕−𝒋
𝒊=𝟏

𝒓𝒙𝒚 =

𝑅2 = 1 −(2)

2∑𝑛𝑖=1(𝑦𝑖 − 𝑦̂)𝑖𝑛∑𝑖=1(𝑦𝑖 − 𝑦̅)2(5)

4. Data set and Experiment Environment
4.1. Experiment Environment

3. Evaluation Methodology
3.1. Correlation coefficient

The field for this experiment was a farmer in Yatsushiro
City, Kumamoto Prefecture, who grows bananas andcoffee. Since Japan belongs to the temperate zone,
farmers across the country are searching for new
cultivation methods to make the cultivation of tropicalplants such as bananas and coffee possible in temperate
Japan. The farmers in Yatsushiro City, which is the focusof this experiment, also grow bananas and coffee in a vast
plastic greenhouse, maintaining a tropical environment.
However, in Japan, which has four distinct seasons,temperatures vary throughout the year, making it difficult
to maintain a constant environment in the vast
greenhouses, which affects the growth and production of
the plants. For this reason, many companies are
introducing agricultural IoT, collecting environmental

The correlation coefficient expresses the strength of therelationship and correlation between two items as apositive or negative value, and can be objectively
analyzed in the range of 1 to -1. 1 is closer to a positive
correlation and -1 to a negative correlation. Equation (3)below shows the formula for obtaining the correlation
coefficient. In this research, we aim to use data obtained
from IoT data and weather data as explanatory variablesin the algorithm in order to increase the accuracy of data
prediction, and to select data by examining the correlation
coefficient with the output values.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
626

Development greenhouse environment prediction

Table 1. The planning and control components.
Item
Correlation
Coefficent
Temperature in plastic
0.886252
greenhouses 1 day ago
Average temperature 1 day
0.578104
before [JMA].
Lowest temperature 1 day
0.567764
ago [JMA].
Highest temperature 1 day
0.546740
ago [JMA].
Solar radiation 1 day ago
0.438165
Moisture content 1 day ago
0.365323(outside)
Maximum instantaneous
0.109610
wind speed 1 day ago[JMA].
Precipitation 1 day ago
0.086130[JMA].
Sunshine hours 1 day ago
0.077532[JMA].
Moisture content 1 day ago
0.015999(inside)Maximum wind speed 1 day-0.195937
ago [JMA].
Average wind speed 1 day-0.198114
ago [JMA].
Air pressure 1 day ago-0.284419
Humidity 1 day before-0.467905
Carbon dioxide-0.680186
concentration 1 day ago

data in greenhouses and providing visualization services.
However, small- and medium-scale private farmers haveto spend their budgets on environmental management,such as maintenance and utility costs in greenhouses, andare unable to purchase or rent IoT equipment andmonitoring tools, which has hindered the diffusion ofdata infrastructure construction. Therefore, in thisexperiment, we developed an agricultural IoT system thatcan be provided to small and medium farmers, andcollected, stored, visualized, and analyzed time-series
data acquired from sensors, with the goal of predicting
environmental information for the next day based ontime-series data.
4.2. IoT Device and Data set
Figure 1 shows the IoT device developed. The
developed IoT devices were equipped with multiple typesof sensors to collect data in order to investigate
environmental information in greenhouses from various
perspectives. In addition, we conducted a durability test
to determine the temperature range at which the device
malfunctions, aiming to develop an IoT device that cancontinue to operate even under harsh environmental
conditions. The results showed that the IoT devices failed
at temperatures around 50 degrees Celsius, so a fan was
installed inside the chamber to prevent failures due tooverheating of the IoT devices. In addition, a Watch Dog
Timer (WDT) was incorporated into the software tomonitor the microcontroller for program stoppages andcommunication interruptions, and to restart the programin the event of a stoppage. We developed IoT devices thatcan continue to operate even in harsh environments byimplementing countermeasures on both the hardware andsoftware sides.

The correlation coefficients between the output target,
temperature in the plastic greenhouse, and the other data
sets are shown in Table 1, and the explanatory variables
needed to predict the temperature in the greenhouse wereextracted from the threshold values. As a result, fiveitems were extracted as positively correlated: thetemperature in the greenhouse one day before, theaverage temperature one day before, the minimum
temperature one day before, the maximum temperature
one day before, and the amount of solar radiation one day
before. Two items, carbon dioxide concentration one day
before and humidity one day before, were selected as
negatively correlated items, and the temperature in thegreenhouse one day before had a high correlation
coefficient with the objective variable to be predicted.
Therefore, the high correlations were adopted as
explanatory variables in the model. As a pre-processing
step for time series forecasting, the data set is divided into
two types of data sets: training data and validation data.
In this experiment, in order to predict the temperature in
the greenhouse on the next day, the validation data wasset as the last three days, and the other data was used as
the training data. The other data were used as training
data. Data for training The training data are For the
training data, we used 14464 data sets from January 18 to
December 25, 2022. We used the temperature inside thegreenhouses as input and the temperature, carbon dioxidelevel, average temperature, humidity, and solar radiationof the previous day with high correlation coefficient as

From this device, temperature, humidity, air pressure,carbon dioxide, solar radiation, and soil moisture in thegreenhouse can be obtained. In addition, we also obtained
meteorological data supplied by the Japan
Meteorological Agency as necessary data for forecasting,such as altar instantaneous wind speed, precipitation,sunshine duration, maximum wind speed, and averagewind speed, and examined whether these data could be
utilized as explanatory variables for the model.

a.

Deviceb. Setting Environment
Fig. 1. IoT Device©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
627

Yasunori Takemura, Naoya Nishida, Moeko Tominaga, Eiji Mizoe, Tomoyasu Furukawa

explanatory variables. In addition, 145 pieces of datafrom December 26 to 28 were utilized as test data for
accuracy verification.

6. Conclusion
Then paper, we developed a model that acquires data inagriculture using IoT devices and utilizes the acquired
data to make predictions. The ARIMA model and theProphet model were used for development andcomparison. In the experiment, we examined thecorrelation between the output temperature and the dataacquired to use the explanatory variables, and compared
the models with and without the variables with highcorrelation values added as explanatory variables. The
results showed that the models with explanatory variables
were more accurate and the ARIMA model had the best
performance. Future issues include repeated experiments
to improve accuracy and evaluation through long-term
operation of the forecasting system.

5. Experiment Result
Analysis of the ARIMA model chorreogram shows thatthe data has a period of one year, and the candidate
parameters for AR(p) are in the range of 0 to 4, and thecandidate parameters for MA(q) should be designed in
the range of 0 to 2. Figure 2 and 3 show the prediction
results when good parameters are set based on theseresults. Figure 2 shows the forecasting results without
explanatory variables. The black dots indicate the values
that are the correct answers, the orange line shows thepredictions of the ARIMA model, and the blue line shows
the Prophet predictions. The value of R2 is 0.98 for the
ARIMA model, and 0.83 for Prophet's model. The valuesof R2 for the test data are 0.32 for the ARIMA model and
0.74 for the Prophet model. This indicates that theProphet model gives better predictions when there are no
explanatory variables. Next, Figure 3 shows the results ofthe model with additional explanatory variables. The
black dots indicate the values that are the correct answers,
the orange line shows the predictions of the ARIMA
model, and the blue line shows the Prophet predictions.
The ARIMA model, the value of R2 in the training data is
0.80, and for the Prophet model, the value is 0.85. The
predicted values for the test data were 0.88 for the
ARIMA model and 0.85 for the Prophet model. These
results indicate that the accuracy improves with theaddition of explanatory variables, and that the ARIMA
model is more accurate when explanatory variables areadded.

Acknowledgements
This work was supported in part by Kumamoto Fruits
and Vegetables Shipping Association Co. Ltd.
References
1. Ministry of Internal Affairs and Communications, 2022
White Paper on Information and Communications, Part 2,
Chapter., ICT Market Trends, Section 9, Trends in the Postal
and Letter Service Businesses.，p.111，2021, in Japanese
2. Ministry of Economy, Trade and Industry of Japan, “Future
Investment Strategy 2017 -Reforms towards the realization ofSociety 5.0-“, pp. 39-43, 2017, in Japanese
3. Prime Minister's Office: The Fifth Science and Technology
Basic Plan, Chapter 3: Measures to Address Economic andSocial Challenges, pp. 16-18, 2016, in Japanese
4. Shingo Otsuka, “Acquisition of Environment Data at Farmers”
Vol.30, No.1，p.35-41，2018, in Japanese
5. George E. P. Box ， Gwilym M. Jenkins ， Gregory C.
Reinsel，Greta M. Ljung：Time Series Analysis: Forecasting
and Control, 5th Edition, 2015.
6. Sean J. Taylor，Benjamin Letham：Forecasting at Scale，
2017.
7. Domenico, Benvenuto, at el. al, “Application of the ARIMA
model on the COVID-2019 epidemic dataset, Data in brief,
Vol29, pp.1-4, 2020
8. A. A. Ariyo, A. O. Adewumi and C. K. Ayo, "Stock Price
Prediction Using the ARIMA Model," 2014 UKSim-AMSS
16th International Conference on Computer Modelling and
Simulation, Cambridge, UK, 2014, pp. 106-112, doi:
10.1109/UKSim.2014.67.
9. Trevor Hastie and Robert Tibshirani：Generalized Additive
Models ， Statistical Science ， vol.1 ， No.3 ， pp.297-318 ，
1986
10. Akaike, H., "Information theory and an extension of the
maximum likelihood principle", Proceedings of the 2nd
International Symposium on Information Theory, Petrov, B. N.,
and Caski, F. (eds.), Akadimiai Kiado, Budapest: 267-281(1973)

Fig. 2 The experiment data using ARIMA andProphet model (No explanatory variables)

Data

D

Fig. 3 The experiment data using ARIMA andProphet model (No explanatory variables)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
628

Powered by TCPDF (www.tcpdf.org)

Development greenhouse environment prediction

Authors Introduction
Dr. Yasunori Takemura
He received his M.S., and Ph.D. degrees from Kyushu
Institute of Technology, Japan, in2006 and 2009, respectively. In 2010,
he joined Nippon Bunri University. In
2015 he joined Nishinippon Instituteof Technology, where he is currently
an Professor of the Department ofEngineering. His research interests
include robotics, IoT and AI Technology.
Mr. Naoya Nishida
He received his Bachelor’s degreeand M.S. in Engineering in 2021 and
2023
from
the
FacultyofEngineering, Nishinippon Institute oftechnology in Japan. He is currently
working in Information Technology
company.

Dr. Moeko Tominaga
She is an Assistant Professor at the
Department of Integrated System
Engineering, Nishinippon Instituteof Technology, Japan. Her researchinterests are in multi-agent systems
and field robotics.

Mr. Eiji Mizoe
He is currenntry working inKumamoto Fruits and Vegitables
shipping Association Co. Ltd..
(Kumamotoseikabutsu
syukkakumiai).

Mr. Hiroaki Wagatsuma
He is currenntry CEO inKumamoto Fruits and Vegitables
shipping Association Co. Ltd..
(Kumamoto seikabutsu syukka
kumiai).

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
629

The Development of SaaS for Quantifying the Amount of Drifted Debris on the Coast
Ayumu Tominaga

National Institute of Technology Kitakyushu College, 5-20-1 Shii, Kokuraminami-ku, Kitakyushu, Fukuoka, 802-0985,
Japan
Ryohei Komori

National Institute of Technology Kitakyushu College, 5-20-1 Shii, Kokuraminami-ku, Kitakyushu, Fukuoka, 802-0985,
Japan
Eiji Hayashi

Kyushu Institute of Technology, 680-4, Kawazu, Iizuka, 820-8502, Fukuoka, Japan
Email: tominaga@kct.ac.jp, k18082rk@apps.kct.ac.jp, haya@ics.kyutech.ac.jp

Abstract
In the process of collecting drifted debris along a coastal area, it has been required to quantify the collection results.
This study introduced a Software as a Service (SaaS) equipped with image processing capabilities for the detection
and identification of debris in photographs. The SaaS was developed to facilitate the automated weighing of thecollected debris. In the beach cleanup event in June 2023, 317 images capturing debris found during the coastal
cleanup were gathered on a cloud server from 90 participants via this SaaS. These images were subsequently analyzed
by object recognition AI executed on a cloud server, leading to the detection and identification of 954 instances ofdebris. This result represents the actual amount of debris collected during this cleanup event, and indicates that theachievements of the event were quantified via the SaaS.
Keywords: Coastal environment, Cloud edge, SaaS, AWS

Additionally, these latest technologies have not been
widely disseminated to the public.

1. Introduction
The impact of marine debris is a global issue [1]. The
familiar scenery of the beach can be destroyed by a large
amount of debris drifting to the coast after being carried
from urban areas through rivers. In Japan, an island
nation surrounded by the sea, this is a significant problem
that needs to be addressed as soon as possible. In recent
years, much research has been conducted on quantifying
the amount of beach-drifted debris. Especially, methods
based on high-resolution images acquired from UAVs orsatellites have been developed [2], [3], [4]. In bothproposed methods, deep learning has been actively
utilized, and certain results have been achieved for a local
region of the beach. These methods can quantify thephysical situation of the debris spread on the beachcompared to human-powered methods. However, theseimage processing-based methods have not been
implemented in an actual field. It is estimated that people
engaged in cleanup tasks may not consider utilizing suchtechnologies to improve the effectiveness of cleanup.

In this study, considering this situation, it is estimatedthat a system that can be used with already familiar
devices is needed for non-engineers or general consumers
participating in the quantifying work of debris. In otherwords, a method that can be completed with only asmartphone and its operation method is important topromote a quantitative understanding of waste. The
performance of smartphones equipped with highresolution cameras has been increasing year by year.
Furthermore, the advancement of AI through cloudcomputing, which is a core technology in Digital
Transformation (DX), has created a social situation
where high-performance PCs are not needed forindividuals. In this study, while utilizing image
processing-based litter detection and identification
methods similar to previous studies, an attempt has been
made to widely establish the work of quantifying debris
by developing a Software as a Service (SaaS) to provide
users with functions related to image analysis andinformation collection on the cloud. Users are required totake a photo containing the debris found on the beachduring the beach cleanup, and the identification processof the debris category and the quantification process for©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
630

Ayumu Tominaga, Ryohei Komori, Eiji Hayashi

each category are completed on the cloud. The SaaS
implemented with such functions has been developed. In
this paper, to socially implement the proposed method,the verification results are reported.

results are then gathered in cloud storage for easy
access.
• Users can review the data analyzed by AI through theapplication.
In this study, a web application with a graphical userinterface (GUI) has been selected and developed to meetthe previously specified requirements. Cloud storage andCPU resources on the cloud have been utilized toautomate image processing. By leveraging cloudcomputing, a large amount of information about thedebris can be collected through smartphones owned byindividuals.

2. Proposed method
2.1. Beach debris quantifying though cloudservice
In this study, an application integrating AI and cloudservices has been developed to facilitate the quantitative
assessment of beach-drifted debris. Fig. 1 represents a use
case of the application. Users of this application are
participants in a beach cleanup event. Typically, they
explore the beach, find debris, and pick it up. In the
proposed method, users are expected to take a photo
capturing the found debris. After finishing the cleanup
work (or during the cleanup), these photos will be
uploaded to cloud storage via the internet. Subsequently,
the photos will be analyzed by AI upon upload. Finally,
the information needed to quantify the debris, such as thetype of debris and the quantity of each type, will begathered as a database on the cloud. As for the
expectation of this proposed method, the event thatoriginally gathered labor to collect debris would
transform into an initiative constructing a large-scale
database focusing on the physical situation of the drifted
debris. This database will be provided to the participants
or organizers of the event, and it can be used forevaluating the results of the executed event or planning
the schedule for the next event.

Fig. 2 The system structure and network resources

3. Developed web application
3.1. Amazon Web Service (AWS)In developing the web application, Amazon Web
Services (AWS) was chosen as the platform forconstructing its network infrastructure. AWS, the cloudcomputing service provided by Amazon Web Services
Inc., has been offering services, including network
infrastructure. By integrating various services within
AWS, the web application developer can rapidly create
an application with extensibility and robustness.
3.2. The system structure
Fig. 2 shows the structure of the system (as the web
application). As mentioned above, resources such as
storage for data or image processing have been
implemented by integrating services provided by AWS(e.g., AWS Amplify, S3, AWS Lambda, etc.). The
system can be divided into two subsystems: a frontend
and a backend. The frontend is software that can be
executed on a web browser, essentially a web page
including a GUI. The resources required to implement thefunctions of the application are referred to as the backend,
and they exist on the cloud.

Fig. 1 Usecase of the application in the proposed method

2.2. Design of the application
To achieve the usecase as shown in Fig. 1, theapplication has been designed as follows:•••

Users can upload images without depending on a
specific location.
The images will be uploaded to cloud storage andautomatically analyzed by AI upon upload.
AI detects all debris in the image, classifies them,and quantifies the number of each debris type. TheseIn this study, the frontend was designed as a web page
with a file browser UI, allowing users to choose images
and buttons to initiate the upload of the selected image.
On the backend side, the implementation included (1)User identification and management functions, (2) The©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
631

The Development of SaaS••

storage region on the cloud, and (3) Virtual CPU
executing inferencing process. Particularly, the details of
the inference processing are explained in the next sectionin this paper.

Other plastics
OthersFig. 3 shows an example of input image including theactual beach-drifted debris, and Fig. 4 shows example ofinference result of Fig. 3 image.
4. Experiment in actual environment
The developed system has been implemented in an
actual beach cleanup event and verified. The event, heldin June 2023 at Hokuto-mizukumi Park (Munakata,
Fukuoka, Japan), aimed to have participants use thesystem. In this cleanup activity, approximately 70
individuals from 30 families participated, spending one
hour cleaning up the beach. It is estimated that 30 to 40
smartphones were used to utilize the system. The
participants were instructed to upload images capturing
the found debris after finished the cleanup activity. The
total number of gathered images was 317, and AI's
identification processing detected 954 beach-drifted
debris. Fig. 5 represents the numbers for each category ofdebris, with the majority falling under "Other plastics,"
including plastic fragments and buoys used in fisheries.
As shown in Fig. 5, plastics were detected significantly
more than other categories. Fig. 6 shows examples ofresults where debris identified as "Other plastics.". Fromthese results, it was confirmed that:

Fig. 3 Captured debris by user••

Fig. 4 Example of detection and identification debris by the
SaaS: Right side was identified “Styrofoam”. Left side was
identified “Other plastics”.

There were instances of misidentifying shells orcolored natural objects as plastics.
Single plastic objects were occasionally detected as
multiple plastic objects.

So it will suggest (1) participants could easily spot
plastics on the beach, leading to intensive collection, and3.3. Inference process based on DNN
In this study, a DNN-based inference is used to detect
captured debris in an uploaded image and classify its
category. In this process, mmdetection [5] was used as the
DNN framework, and Mask-RCNN [6] was employed as
the neural network architecture for detection of debris.
And, TACO dataset [7], it is large dataset for garbage wasutilized as the annotated data. In this study, the dataset
was classified by eleven categories as below:
•
•
•
•
•
•
•
•
•

Plastic bottle
Can
Glass
Metal
TabacoRubberRope & String
Styrofoam
Paper

Fig. 5 The result of number of each category of detected
debris©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
632

Powered by TCPDF (www.tcpdf.org)

Ayumu Tominaga, Ryohei Komori, Eiji Hayashi

7. P. F Proença and P. Simões, “TACO: Trash Annotations
in Context for Litter Detection,” arXiv:2003.06975, 2020.

(2) to enhance object detection accuracy, reconstructionof the learning dataset for debris is necessary.

Authors Introduction
Dr. Ayumu Tominaga
Dr. Ayumu Tominaga is assistantprofessor in Department of creativeengineering, National Institute ofTechnology, Kitakyushu College. He
received the Ph. D. from Kyushu Instituteof Technology. His research interests a
field robot such as for beach cleanup
robot, and social implementation of a cloud service. He isa member of The Japan Society of Mechanical Engineers(JSME).
Mr. Ryohei Komori
Mr. Ryohei Komori is student in
Department of creative engineering,
National Institute of Technology,
Kitakyushu College. His researchinterests include development of a SaaSor web service, application for
democratizing of AI.

Fig. 6 Example of the input and detected “Other plastics”

5. Conclusion
In this study, a cloud service-based web application wasdeveloped to promote the quantification of beach-drifted
debris. This web application, which can be used from asmartphone, aims to gather a large-scale beach debris
dataset through beach cleanup events. One advantage ofthis proposal is that many people can easily participate indebris quantification work. In the case of experimental
verification, the system successfully collected over 300
images capturing debris and detected over 900 debris
with the participation of about 70 cleanup event
participants. In the future, efforts will be made to enhance
the usability of the application, making it less stressful forusers. Additionally, long-term verification, spanningmonths or even years, will be conducted to address andimprove the physical situation of beach-drifted debris.

Prof. Eiji Hayashi
Prof. Eiji Hayashi is a professor in the
Department of Intelligent and Control
Systems at Kyushu Institute ofTechnology. He received the Ph.D. (Dr.
Eng.) degree from Waseda University in
1996. His research interests include
Intelligent mechanics, Mechanical systems and Perceptual
information processing. He is a member of The Institute ofElectrical and Electronics Engineers (IEEE) and The Japan
Society of Mechanical Engineers (JSME).

References
1. S. C. Gall and R. C. Thompson, “The impact of debris on
marine life,” Marine Pollution Bulletin, Vol. 92, Issues 1–
2, pp. 170-179, 2015.
2. K. Sasaki and et. al., “Coastal Marine Debris Detection andDensity Mapping With Very High Resolution Satellite
Imagery,” in IEEE Journal of Selected Topics in Applied
Earth Observations and Remote Sensing, vol. 15, pp.
6391-6401, 2022.
3. K. Moy and et. al., “Mapping coastal marine debris using
aerial imagery and spatial analysis,” Marine Pollution
Bulletin, Vol. 132, pp. 52-59, 2018.
4. Y. Taddia, and et. al., “UAV Approach for Detecting
Plastic Marine Debris on the Beach: A Case Study in the
Po River Delta (Italy),” Drones, Vo. 5, No. 140, 2021.
5. K. Chen and et. al., “MMDetection: Open MMLab
Detection Toolbox and Benchmark,” arXiv:1906.07155,
2019.
6. K. He and et. al.,“Mask R-CNN,” 2017 IEEE
International Conference on Computer Vision (ICCV),
2017.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
633

Development of antagonistic wire-driven joint mechanism capable of rapid motion andvariable stiffness
Katsuaki Suzuki

Kumamoto Industrial Research Institute3-11-38 Higashi machi,Higashi-ku, Kumamoto 862-0901, Kumamoto, Japan
E-mail: k-suzuki@kumamoto-iri.jp
Yuya Nishida

Life Science and Systems Engineering, Kyushu Institute of Technology
2-4 Hibikino,Wakamatsu-ku, Kitakyushu 808-0196, Fukuoka, Japan
E-mail: y-nishida@lsse.kyutech.ac.jp
Kazuo Ishii

Life Science and Systems Engineering, Kyushu Institute of Technology
2-4 Hibikino,Wakamatsu-ku, Kitakyushu 808-0196, Fukuoka, Japan
E-mail: ishii@brain.kyutech.ac.jp
Abstract
The advancement of digital transformation in manufacturing is expected to demand novel tasks for industrial
machinery unlike before. When adding mechanisms or actuators to enhance machine functionality, concerns arise
about potential system enlargement and increased complexity. The design concept of integrating multiple functions
into mechanisms using a few actuators and components serves as one means to address these issues. We aim toelucidate the structural arrangement capable of achieving three functions, including normal motion, rapid motion,and variable stiffness, using two electric actuators. In this paper, we introduce a newly devised antagonistic wiredriven joint mechanism aimed at improving the variable stiffness function within the articulated mechanism we havedeveloped, enabling three functionalities.
Keywords: Rapid motion, Variable stiffness, Link mechanism

estimation using ultrasonic waves before picking to
optimize the gripping rigidity in order to reduce damage
caused by food picking operations using robot hands [1].
To achieve automation, this system needs to besupplemented with a hand that can change the stiffnessof the fingertips. In addition, percussion testing is being
researched to predict the firmness of tomatoes todetermine their expiration date, to estimate the sugar
content of fruit, and to determine whether canned goods
are defective [2], [3]. To automate percussion testing, anappropriate hammering mechanism is required inaddition to a sensing system.
Adding additional mechanisms and actuators tointegrate multiple functions into a robot can increasesystem complexity and size. Therefore, a design conceptthat allows mechanisms to have multiple functions withfewer actuators and parts is one way to address theseissues.

1. Introduction
Digital transformation is being promoted through theuse of robots and digital technologies, such as the Internetof Things and artificial intelligence, with the aim ofincreasing productivity and manpower saving. In
response, robots will be required to perform tasks thathave not been done before, and it will be necessary to
promote research and development of robots that canperform movements and functions that have not yet been
generalized. One area is soft robotics, which can perform
rapid motion and variable stiffness. For example, in thefood industry, in order to realize the automation of theserving operation of prepared food, it is necessary for the
robot to grasp the hardness of the food and to pick andplace it according to the hardness of the food so that thesoft food does not fall apart. Mochizuki et al. areconducting research on non-contact acoustic impedance©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
634

Katsuaki Suzuki, Yuya Nishida , Kazuo Ishii

Aiming to realize three functions with a small numberof actuators, we developed a joint mechanism thatrealizes normal motion, rapid motion, and variable
stiffness with two motors [4], [5]. Our research results
have shown that these functions can be realized.
However, in the variable stiffness function, an endeffector does not rotate unless the load torque applied tothe end-effector exceeds the torque applied to the endeffector by the pre-compressed spring to increase
stiffness. In this paper, we propose a new antagonistic
wire-driven joint mechanism that can rotate the endeffector at the moment load torque is applied to the endeffector, even in a stiff state, in order to improve thefunctionality related to variable stiffness of joint
mechanisms that can realize the three functions.

through the wires, and the end-effector performs a rapid
motion. Next, consider the variable stiffness function. As
shown in Fig. 5, first assume a state in which the two
springs are compressed by 𝜀𝑠𝑝 and the two actuators arebraked. Then assume a state in which a load torque isMotor Cam Input pulley
Slider SpringEnd-effector
Fixed link
Main joint

Warm

Warm wheel

Wire
Passive pulleyFig. 1 Conceptual model

2. Antagonistic wire-driven joint mechanism

SliderFig. 1 shows the proposed mechanism. The proposedmechanism consists of the end-effector, a main joint, twowires, two springs, two linear guides, and two cam units.
Assume that the positions of the linear guide and main
joint are fixed in space. Additionally, the linear guide andslider are a prismatic pair. The internal structure of the
cam unit consists of a slider, a passive pulley, an input
pulley, a cam follower, a cam, a worm wheel, a worm,
and a motor.The function of the cam unit is to extend andretract the wire and release the energy stored in the spring.
Rotation of the motor causes the cam to rotate throughthe worm gear mechanism. Since the cam follower andthe input pulley are connected, when the input pulleyrotates, the cam follower performs an orbital motion.
Therefore, the input pulley rotates driven by the rotationof the cam. For ease of understanding, we will now
discuss the structure using the simplified proposedmechanism shown in Fig. 2.
The main difference between Fig. 1 and Fig. 2 is thatthe cam unit is represented by an electric actuator. As
shown in Fig. 3, during normal motion, the end-effector
rotates counterclockwise with actuator 1 in the active
state and actuator 2 in the passive state. For rapid motion,
as shown in Fig. 4, when the two actuators are active, thetwo sliders move toward the main joint and energy isstored in the two compression springs. Now, when
actuator 1 is in the braking state and actuator 2 is in thepassive state, the forces on the wires are out of balance
and the two sliders are moving away from the main joint.
Since actuator 1 is in the braking state, the compression
spring force acting on the sliders acts on the end-effector

SpringEnd-effector

Fixed link

Main joint
Wire
Electric linear actuator

Fig. 2 Simple modelFig. 3 Normal motion

Fig. 4 Rapid motion

Hand

Fig. 5 Variable-stiffness function
applied to the end effector and the end effector is rotatedcounterclockwise to 𝜃𝑜 . At this time, wire 2 is wrappedaround the end effector and slider 2 makes a translational
movement toward the main joint, increasing the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
635

Development of antagonistic wire-driven

angle 𝜃𝑜 was rotated from 0[deg] (initial posture) to+40[deg] and then from 0[deg] to -40[deg]. The analysis
results confirm that the load torque increases as the angle
𝜃𝑜 increases. Furthermore, as the value of 𝜀𝑠𝑝 increases,
the load torque required to rotate the joint increases. Even
when 𝜀𝑠𝑝 was set, the load torque applied to the endeffector did not have to exceed the torque applied to theend-effector by the pre-compressed spring. On the other
hand, it can be observed that the maximum displacement
angle decreases as the value of 𝜀𝑠𝑝 increases. This isbecause as 𝜀𝑠𝑝 increases, the distance between the slider
and the main joint becomes shorter and the range of
motion of the slider becomes smaller. Fig. 7 shows the
profile of the rotational stiffness 𝐾 and angle 𝜃𝑜 of theend-effector when the initial spring displacement 𝜀𝑠𝑝 is 0,5, 10, and 15 mm (Eq. (3)). It can be observed that as 𝜀𝑠𝑝
increases, the rotational stiffness of the main joint
increases. It can also be observed that the value of each
stiffness increases rapidly near the end point of the
motion. This is because as the angle 𝜃𝑜 increases, thedistance between the slider and the main joint becomesshorter, and the tension generated in the wire increases as𝛼∗ becomes smaller.

compressive force acting on spring 2. Therefore, thetorque due to the compression force of spring 2 acts on
the end-effector in the direction of resisting the load
torque. On the other hand, since the wire 1 originallywound around the end-effector is unwound, the slider 1
makes a translational movement in the direction awayfrom the main joint, and the compression force originallyacting on the spring 1 decreases. In other words, thetorque due to the compression force of spring 1 acts on
the end effector in the direction to assist against the load
torque. By using this load torque support mechanism, theend-effector can be rotated at the moment when a load
torque is applied to the end-effector. In addition, by
adjusting the value of 𝜀𝑠𝑝 , the compression force of thespring can be changed, and thus the rotational rigidity of
the joint can be varied. The relation between the load
torque 𝑇𝑞 and the internal force acting inside themechanism is expressed as follows based on theirgeometric relation:
𝛿𝑠𝑝1
𝛿𝑠𝑝2
(1)
𝑇𝑞 = 𝑘𝑠𝑝 𝑅𝑜 (−)
𝑠𝑖𝑛 𝛼1 𝑠𝑖𝑛 𝛼2
In Eq. (1), 𝛼∗ is expressed as follows:𝑙𝑠𝑝∗ 𝑙𝑤∗ + 𝑙𝑠 𝑅𝑜
𝛼∗ = 𝑡𝑎𝑛−1(2)
𝑙𝑠 𝑙𝑤∗ − 𝑙𝑠𝑝∗ 𝑅𝑜
The rotational stiffness 𝐾 of the end-effector is expressedusing the following equation by differentiating the load
torque 𝑇𝑞 exerted on the end-effector by the angle 𝜃𝑜 ofthe output link:
𝑑𝑇𝑞
(3)
𝐾=
= 𝐾𝑏2 − 𝐾𝑏1𝑑𝜃𝑜
In Eq. (3), 𝐾𝑏∗ is expressed as follows:
𝑅𝑜 2 𝑘𝑠𝑝 𝛿𝑠𝑝∗ 𝑙𝑤∗ 𝑙𝑠 𝑅𝑜3 𝑘𝑠𝑝 𝛿𝑠𝑝∗
− 2
2 )]
2
2 + 𝑙2 )(𝑅𝑜 + 𝑙𝑤∗
𝑙𝑠𝑝∗ (𝑙𝑠𝑝∗
𝑅𝑜 𝑘𝑠𝑝 𝑙𝑤∗
𝑠
𝐾𝑏∗ =
+
𝑙𝑠𝑝∗ 𝑠𝑖𝑛 𝛼∗
𝑠𝑖𝑛 𝛼∗ 𝑡𝑎𝑛 𝛼∗
[

0.2

0.1

0
=0mm

=5mm

-0.1

=10mm

=15mm

(4)

-0.2
-40

In Eq. (4), with 𝜀𝑠𝑝 = 0, 𝐾𝑏1 = 0 for counterclockwise
rotation of the output link and 𝐾𝑏2 = 0 for clockwise
rotation.

-30

-20

-10

0

10

20

30

40

Fig. 6 Relationship between the rotational angle ofend effector and load torque3. Analysis of output characteristics
Using the equations derived in Chapter 2, we analyzed
the output characteristics with respect to variable
stiffness. Fig. 6 shows the profile of the load torque 𝑇𝑞
and angle 𝜃𝑜 of the end-effector at initial spring
displacements 𝜀𝑠𝑝 of 0, 5, 10, and 15 mm (Eq. (1)).
Further, Table 1 shows the values of each parameter usedin the analysis. The analysis was performed assumingthat a load torque was applied to the end-effector and theTable 1. Design parameters©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
636

Powered by TCPDF (www.tcpdf.org)

Katsuaki Suzuki, Yuya Nishida , Kazuo Ishii

𝑘𝑠𝑝
𝑅
𝑙𝑠
𝑙𝑠𝑝𝑛
𝑚𝑎𝑥(𝜃𝑜 )𝑚𝑖𝑛(𝜃𝑜 )Spring constantRadius of end-effector
Length of slider
Natural length of spring
Maximum movement range
Minimum movement range

3.

500 N/m0.01m
0.02 m
0.02 m
+40°
−40°

4.

5.

YOKOI Masayuki. A Study on Diagnosis Method using
Hammering Test. JOURNAL OF OSAKA SANGYOUNIVERSITY Natural Sciences, 2021, 131: 59-69.
Suzuki, Katsuaki, et al. A new rotary actuator capable of
rapid motion using an antagonistic cam mechanism.
Journal of Advances in Artificial Life Robotics, 2020,
1.3: 143-151.
Suzuki, Katsuaki. Research on antagonistic wire-driven
joints with rapid motion and variable stiffness functions,Ph.D. thesis, Kyushu Institute of Technology, 2021.

8

Authors Introduction=0mm

Dr. Katsuaki Suzuki

=5mm

6

=10mm

He is a Researcher at the
Kumamoto Industrial ResearchInstitute, Japan. He received his
Ph.D. degree from the Kyushu
Institute of Technology in 2021.
His research interests include
joint mechanisms and theirapplications.

=15mm

4

2

0
-40

-30

-20

-10

0

10

20

30

40

Fig. 7 Relationship between the rotational angle ofend effector and rotational stiffness

Dr. Yuya Nishida
He is an Associate Professor at
the Graduate School of Life
ScienceandSystemEngineering, Kyushu Institute ofTechnology, Japan. His researchinterests include filed robotics,
its application, and dataprocessing.

4. Conclusion
In this study, we describes the structure of the newmechanism and demonstrated a method for realizing
normal motion, rapid motion, and variable stiffness.
Additionally, we derived expressions for variable
stiffness and analyzed. In the previously developed joint
mechanism, the end-effector does not rotate unless theload torque exceeds the torque applied to the end-effector
by the spring preloaded to increase the rigidity. As aresult of the analysis, it was theoretically clarified that the
newly proposed joint mechanism can solve this problem
and rotate the end-effector at the moment the load torqueis applied. Future issues include the search for a more
optimal link ratio and the need to verify the effectivenessof the system by fabricating an actual machine andevaluating its performance.

Prof. Kazuo Ishii
He is a Professor at the Graduate
School of Life Science andSystem Engineering, Kyushu
Institute of Technology, Japan.
His research interests include
field robots and intelligent robot
systems.

References
1.

2.

Mochizuki, Ryuugo, et al. A Method to Estimate
Reflection Coefficient of Ultrasonic Wave at a Boundaryof Two Media. Journal of Robotics, Networking andArtificial Life, 2023, 9.4: 343-348.
ALIUA, I., et al. THE NONDESTRUCTIVE METHODOFACOUSTICIMPULSERESPONSE
IN
CHARACTERIZINGTOMATOESFIRMNESS
COEFFICIENT DURING STORAGE.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
637

Gakken Hills Interdisciplinary Ekiden Competing with Humans, Animals, and Robots
Takuya Fujinaga

Fukuoka University
8-19-1 Nanakuma, Johnan, Fukuoka 814-0180, Japan
Moeko Tominaga

Nishinippon Institute of Technology
1-11, Aratsu, Kanda-machi, Miyako-gun, Fukuoka 800-0394, Japan
Daigo Katayama, Kazuo Ishii

Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu, Kitakyushu, Fukuoka 808-0196, Japan
E-mail: t.fujinaga.wz@fukuoka-u.ac.jp
www.lsse.kyutech.ac.jp/~sociorobo/gakkenekiden/

Abstract
The number of robots exiting in human living spaces has increased, and interaction between humans and robots has
become an important issue. The purpose of this study is to propose a future society where humans, animals, androbots coexist. We organize relay race, or ekiden in Japanese, where they compete together as runners. The
competition rules have been established to ensure that each runner can compete fairly. This paper explains thecompetition rules and discuss the fairness of the rules
Keywords: Robotics competition, Ekiden, Human-Robot Interaction, Robot coexistence societyof human society, there are many types of mobile
objects: humans, animals, and vehicles. There are few
examples of demonstration experiments or discussions
about the role of robots in such environments.
We have held “Gakken Hills Interdisciplinary Ekiden”
as the opportunity to think about the future society inwhich humans and robots coexist in harmony [6]. Thisekiden is the relay race which humans, animals, androbots compete together on the same field. We held the
8th race on May 13, 2023. This paper describes thecompetition rules, introduces the characteristics andtransition of participants from the 1st to the 8th races,and discusses the fairness of the rules.

1. Introduction
Various robotics competitions [1], [2] have been held
for social implementation with the development ofrobotics and artificial intelligence. In these competitions,robots mainly compete about functional performance.
On the other hand, the content of robotics competitions
changes to more challenging tasks aimed at coexistence
between humans and robots [3].
Traditionally, robots have been introduced into
industrial environments where safety has been ensured
by cages. However, in recent years, the environment inwhich robots work has been changed to match the daily
lives of humans. Research on human-agent interaction[4] and human-robot interaction [5] have aimed to the
future society in which humans and robots coexist inharmony. On the other hand, in the universal situation

2. Gakken Hills Interdisciplinary Ekiden©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
638

Takuya Fujinaga, Moeko Tominaga, Daigo Katayama, Kazuo Ishii

2.1. Overview- No sharp protruding parts or devices that may cause
harm to others.
- The robot has a function that enables it to be stopped
quickly in an emergency.
The advantages are applied to different categories ofrunners in this ekiden, and robots have different
advantages depending on their movement and operating
methods. Runners in the [R] are classified into thefollowing three classes.
- Direct operation class:requires human intervention (e.g. electric wheelchairs,
electric kickboards)- Remote operation class:no requires human intervention- Automatic operation class:applicable to three or more of the following five items- Do not require a remote controller or otheroperating device (e.g. human following).
- Do not require real-time operation by humans (e.g.
driving using satellite positioning systems).
- Make action selections based on environmental
recognition (e.g. obstacle avoidance).
- Have originality (e.g. does not use existingproducts as a base, or not wheel-based type).
- Submit a technical poster
Note that in this ekiden, the human intervention isdefined as pushing, carrying, or directly touching therobot to adjust its traveling direction by humans.

We organize this ekiden with the aim of making it astarting point for thinking about the future society inwhich humans and robots coexist and discussing thedirections and challenges of such a society. The 1st race
was held in 2014 and the 8th in 2023. Various runners
have participated so far: humans regardless of age or
gender, animals (mainly dogs) and robots with various
movement and operation methods. To ensure fairness
for the runners in this ekiden, where a wide variety ofrunners coexist, we have set our own competition rules.
The competition rules have been adjusted every year
based on law revision, developments in science andtechnology, and past competition results. In thefollowing, the current competition rules are described.
2.2. Rules for all runners
Runners are divided into categories [P] (Person), [A](Animal) and [R] (Robot). Category [P] is humans aged
18 years or older, [A] is dogs, cats, and other animals(excluding horses, cows, and other light vehicles) and[R] is robots and electric wheelchairs (excluding flying
robots). One of the rules for the [R] sets the use of anelectric motor as a prime mover, and electric
wheelchairs are classified as the [R].
One team consists of four runners, and, in principle,
the four runners are two males and two females.
However, the [A] and [R] can be classified as either
male or female. Note that within one team, the [A] and[R] can each have up to one runner participating. The
race distance differs depending on the category: about
1.6km for the [P] and [A], and about 0.8km for the [R].
For details of the course, see the ekiden website [7].
This ekiden sets advantages so that humans, animals,
and robots can compete fairly. The rank in thecompetition are determined based on the results ofapplying advantages to the race results. Details of theadvantages are stated in 2.4.

2.4. Advantages

2.3. Rules for robots

The advantages are determined by the age of the [P],the participation of animals, the robot specifications.
The results which are applied the advantages are used todetermine the final competition results and the rank.
Three types of advantages are defined: age advantage,
animal advantage and robot advantage. For the age
advantage, let 𝑁𝑝 and 𝐴𝑔𝑒𝑡𝑜𝑡𝑎𝑙 be the number ofparticipants in category [P] and their total age,respectively. If Eq. (1) is satisfied, 𝐴𝑑𝑣𝑝 seconds (Eq.
(2)) is subtracted from the race result.
(1)
𝐴𝑔𝑒𝑡𝑜𝑡𝑎𝑙 > 25 × 𝑁𝑝

The rules for robot participation are as follows. We
set these rules based on Japanese road traffic act.
- The max size is 120cm, 70cm, and 120cm in length,
width, and height, respectively.
- An electric motor is used as a prime mover.
- Traveling speed is 6 km/h or less.

(2)
𝐴𝑑𝑣𝑝 = 𝐴𝑔𝑒𝑡𝑜𝑡𝑎𝑙 × 2
For the animal advantage, let 𝑇𝐴 be the race time in
the [A]. The unit is seconds. The result 𝑅𝐴 seconds
which applying the animal advantage is calculated from
Eq. (3). Note that the method of calculating 𝑅𝐴 depends
on the value of 𝑇𝐴 ．

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
639

Gakken Hills Interdisciplinary Ekiden

1
𝑇𝐴 + 120 (𝑇𝐴 ≤ 600)
(3)
𝑅𝐴 = { 21
𝑇𝐴 + 360 (600 < 𝑇𝐴 ≤ 2400)
10
For the robot advantage, let 𝑅𝐴 be the race time in the[R]. The unit is seconds. The result 𝑅𝑅 seconds which
applying the robot advantage is calculated using one of
Eqs. (4) to (6). Eq. (4) is for the direct operation class( 𝑅𝑅(𝐷𝑖𝑟𝑒𝑐𝑡) ), Eq. (5) for the remote operation class(𝑅𝑅(𝑅𝑒𝑚𝑜𝑡𝑒) ) and Eq. (6) for the automatic operation
class 𝑅𝑅(𝐴𝑢𝑡𝑜) ). Note that 𝑇𝑅 is between 480 seconds (8minutes) and 2400 seconds (40 minutes).
1(4)
𝑅𝑅(𝐷𝑖𝑟𝑒𝑐𝑡) =𝑇 + 390
10 𝑅
1(5)𝑅𝑅(𝑅𝑒𝑚𝑜𝑡𝑒) =𝑇 + 270
10 𝑅
1(6)
𝑅𝑅(𝐴𝑢𝑡𝑜) =𝑇 + 150
10 𝑅
The constant number of these equations for theadvantages was determined based on the results ofprevious competitions results to ensure that all runners
can compete fairly.

Table 1 shows the transitions in the number ofparticipants from the 1st to the 8th races. The number ofrunners per team was six from the 1st to the 5th races,
and from the 6th race, it was four due to a change in therules. Regarding the proportion in each race, thecategory [P] accounts for 70-90%, the [A] was justunder 10%, and the [R] was just over 10%. The numberof participants in the [A] so far has been 15, all of themdogs. All runners in the [P] and [A] completed the race.
The number of participants in the [R] so far has been 45.
For the traveling method of the [R], two of the 45 robots
were multi-legged or bipedal walking types rather thanwheel-based type. For the operating method, six out 45
robots were semi- or full-automatic. Examples of its
Table 1 Transitions in the number of participants
Team
13
10
7
6
8
10

[P]
70
52
31
30
43
34

[A]
1
1
3
2
1
3

12
12

40
40

2
2

6
6

Table 2 Average and standard deviation (S.D.) of 8th race
results in each category (m: minutes, s: seconds)
Category[P]: male[P]: female[A][R]

Average

S.D.

8m21s
10m19s
7m04s
6m50s

1m33s
1m36s
16s
54s

Fig. 1: Average and standard deviation of each race
result in each category
operation methods were as follows: human tracking
using image processing, driving using waypoints, route
tracking using the global navigation satellite system,
simultaneous localization and mapping (SLAM) using alaser sensor and an inertial measurement unit. Runners
who completed the race within the time limit was 35
robots. The multi-legged and bipedal walking types
mentioned above were unable to complete the race. Of
the six autonomous robots, two were able to complete
the race within the time, one using waypoints and theother using SLAM.
Table 2 shows the average and standard deviation of
the results for each category in the 8th race. The resultsof the [A] and [R] were equal to or higher than those of
the [P]. Therefore, rules regarding advantages works,
and runners could compete regardless of theircharacteristics. The details are on the ekiden website [7].
This paper discusses the validity of rules regarding
advantages. Fig. 1 shows the average and standard
deviation of the competition results for each category
since the 6th race. The results for the [A] and [R] are theresults of applying the advantages. Note that the race
distance for the [R] has been changed since the 6th race,
so only the results from then on are shown. The resultsof the [P] have little variation each race. On the other
hand, the [A] was at a slow compared to the [P] in the
6th race, and furthermore, the standard deviation waslarge in the 7th race. The [R] is also at a slow comparedto the [P] in the 6th race. However, in the 7th race, the3. Results and discussion

Times
1
2
3
4
5
6

7
8

[R]
7
7
8
4
4
3

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
640

Powered by TCPDF (www.tcpdf.org)

Takuya Fujinaga, Moeko Tominaga, Daigo Katayama, Kazuo Ishii

robot advantage was revised based on the previous race
results of the [P], so the results of the 7th and 8th racesof [R] and [P] are fair. Based on this result, the animal
advantage was similarly revised for the 8th race. In the
8th race, the results of the [A] and [R] were equal to or
better than those in the [P]. Therefore, these results
show that for each runner to compete fairly, it isappropriate to set an advantage based on a certain
standard.

Authors Introduction
Dr. Takuya Fujinaga
He is an Assistant Professor at the
Department
ofElectronics
EngineeringandComputer
Science,
Fukuoka
University,
Japan. His research interests are in
agricultural robots, underwater
robots and field robotics.

4. Conclusions
We have held the ekiden for thinking about a futuresociety in which humans and robots coexist in harmony.
The 1st race was held in 2014, and the 8th in 2023. The
rules have been revised for each race based on previous
race results and law revision. With the spread of robots,laws regarding robots were established, and the rules forthe 8th race were determined based on the laws. This
paper showed validity of the rules from race results. We
have continued to discuss how robots coexist withhumans in future society through this ekiden.

Dr. Moeko Tominaga
She is an Assistant Professor at the
Department of Integrated System
Engineering, Nishinippon Instituteof Technology, Japan. Her researchinterests are in multi-agent systems
and field robotics.

Dr. Daigo Katayama
He is an Assistant Professor at
Kyushu Institute of Technology,
Japan. His research interests are in
welfare devices, mobile systems,
and field robotics.

References
1.

2.

3.

4.

5.

6.

7.

K. Ishii et al., “Report on the 8th Tomato-Harvesting
Competition toward Smart Agriculture”, Int. Conf. on
Artificial Life and Robotics, pp. 484-488, 2023.
H. A. Yanco et al., “Beyond Usability Evaluation:
Analysis of Human-Robot Interaction at a Major
Robotics Competition,” Human-Computer Interaction,19(1-2), pp.117-149, 2004.
H. Okada et al. “What is the Purpose of the World Robot
Summit Service Category,” J. of the Robotics Society ofJapan, 37(3), pp. 218-223, 2019. (In Japanese)
S. Yamada et al., “Mutual Adaptation to Mind Mappingin Human-Agent Interaction,” IEEE Int. Workshop on
Robot and Human Interactive Communication, pp. 105110, 2002.
R. Jahanmahin et al., “Human-robot interactions in
manufacturing: A survey of human behavior modeling,”
Robotics and Computer-Integrated Manufacturing, 78,102404, 2022.
T. Fujinaga et al., “Report on the 8th Gakken Hills
Interdisciplinary Ekiden,” The 31st Symposium on Fuzzy,
Artificial
Intelligence,
NeuralNetworks
andComputational Intelligence, 2023. (In Japanese)
Center for Socio-Robotics Synthesis, Kyushu Institute ofTechnology, “Gakken Hills Interdisciplinary Ekiden,”
https://www.lsse.kyutech.ac.jp/~sociorobo/gakkenekiden/(In Japanese, Access 2023-12-12)

Dr. Kazuo Ishii
He received his PhD from the
University of Tokyo, Japan, in 1996.
In 2011, he joined Kyushu Instituteof Technology and is currently aprofessor in the Department of
Human Intelligent Systems. His
research
interests
includeinformation communications andmarine robotics. He is a member ofIEEE.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
641

Development of a Low-Cost Underwater Robot for Research and Education
Takuya Fujinaga

Fukuoka University
8-19-1 Nanakuma, Johnan, Fukuoka 814-0180, Japan
E-mail: t.fujinaga.wz@fukuoka-u.ac.jp

Abstract
Underwater robots are utilized for underwater observation. Generally, the development of underwater robots
demands a significant amount of funds, posing a high barrier for research, development, and education. The aim of
this study is to reduce these barriers, facilitating broader implementation of research, development, and education in
the field of underwater robotics. This paper describes the development policy and cost of a low-cost underwater
robot and discusses considerations based on the results in the robotics competition.
Keywords: Underwater robot, Underwater drone, Remotely operated vehicle, Low-cost, Robotics competition

following are the minimum requirements for theirdevelopment: watertightness, pressure resistance, andadjusting buoyancy etc. There are also various types oftacit knowledge.
The aim of this study is to reduce barrier of thedevelopment cost, facilitating broader implementationof research, development, and education in the field of
underwater robotics. This paper describes thedevelopment policy and cost of a low-cost underwater
robot and discusses considerations based on the resultsof the robotics competition.

1. Introduction
Underwater robots are used as a means of underwaterobservation. Research and development of underwater
robots has been studied since the 1960s [1]. Recently,
underwater robots are called as underwater drones andbecome spread to society. When using underwater
robots as tools for underwater geological and biologicalsurveys etc., it is beneficial to use commerciallyavailable underwater robots. However, when
underwater robots are the subject of research anddevelopment or education, the specifications required
for the robot vary depending on the purpose. Therefore,
it is necessary to develop the robot independently.
Underwater robots come in a variety of sizes, from large
to small, and research and development has been carriedout depending on the purpose [2], [3]. There are also
underwater robots developed for education [4].
Generally, the development of underwater robots
demands a significant amount of funds, posing a highbarrier for research, development, and education.
Although some underwater robots have been developed
for low-cost, the development cost still ranges from$4,000 to $15,000 [5], [6], [7]. Underwater robots differ
from robots that move on land or in the air, and the

2. Method
2.1. Development policy
Underwater robots are mainly classified into five
types: glider, cruise, hovering, biomimetic, and vehicle.
The selection of each type is determined by the target
work and research policy. The glider-type generally
does not have propulsion devices and can adjust theirown buoyancy and center of gravity, enabling long-term
observation. The cruise-type has a long and slender
structure, so it has high propulsion efficiency and canconduct wide-ranging surveys. The hovering-type canperform more complex movements than the above-©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
642

Takuya Fujinaga

mentioned types by attaching multiple propulsors,
making detailed observations possible. The biomimetictype imitates the characteristics of fish and otheranimals, can be propelled by fish fins rather thanpropellers, and are superior to the above types in termsof energy efficiency and consideration for marine life.
The vehicle-type can move on the seabed or on the hullof a ship and perform work on those surfaces, such as
digging and cleaning. This study adopts the hoveringtype for the purpose of detailed underwater observation.
The underwater robot developed in this study isdesigned to operate underwater for about 30 minutes to
1 hour, targeting shallow part of the ocean with anobservation depth of 10m or less.
The main components of the hovering-type
underwater robot that performs underwater observation
are as follows: computers, sensors, thrusters, energy
sources. The computers control the underwater robot bycontrolling the propeller, acquiring sensor information,and selecting the actions. There are various types of
sensors depending on the purpose of observation, suchas cameras and acoustic devices. Underwater robots also
require positioning and navigation equipment and sonarsensor to measure their own posture, position, speed,
and surrounding environment. In some cases, sensors
account for a large portion of the development cost.
Electric thrusters are generally used as propulsion
devices, and a high degree of freedom in movement
performance is achieved by attaching multiple thrusters.
The energy sources need to be selected appropriately tooperate the above-mentioned components at the desired
operating time. In addition to the above components,
communication equipment, work equipment, ballast, etc.
are added as necessary.
The low-cost underwater robot in this study also
consists of computers, sensors, thrusters, and energy
sources. The sensors are equipped with a camera, depth
sensor, and inertial measurement unit (IMU). As an
option, add underwater lights to conduct underwaterobservations. In order to control this robot from theground and check the data acquired by the robot, theexternal computer on the ground and the internal
computer on the underwater robot are connected with an
optical fiber cable, and this robot is moored from theground using this cable. Therefore, this robot becomes aremotely operated vehicle (ROV). However, thedevelopment policy is to make it easy to expand thesensor so that it can be changed to an autonomous
underwater vehicle (AUV).
The functional requirements for the low-costunderwater robot are described above. As nonfunctional requirements, this robot is considered
portability and parts availability. For the portability, thedesign is based on the weight and ease of carrying by
one person. To ensure the ease of obtaining the parts,
use parts that can be easily purchased at online shoppingor home centers. Note that some parts are manufactured
using 3D printer and laser beam machining. Regarding
the development cost, previous studies has proposed acost of $4,000 to $15,000 for a low-cost underwater
robot. In this study, the development cost is less than
$2,000.
2.2. Low-cost underwater robot
The design drawing of the low-cos underwater robot
is shown in Fig. 1, and the components are shown in
Table 1. Although, there is a certain distance offset
between each component to show the components in Fig.
1, each component is connected as shown in the actual
appearance in Fig. 2. This robot mainly consists of thefollowing.
- Frame (No. 1 and 2 in Fig. 1)- Main box (No. 8) that stores the computers, depth
sensor, IMU, etc.
- Battery box (No. 9) that stores the batteries andcircuits- Thrusters (No. 21 and 22)- Camera unit (No. 4)- Underwater lights (No. 5)- Acrylic plates and joints to connect the above
components (No. 16-19)
The energy from the battery box is supplied to the mainbox via cable gland. A total of six thrusters are installed,
total of four ones in the front and rear for movement inthe xy-plane, and two in the center for movement in thez-axis direction. The camera unit is independent fromthe main box, and the data is acquired by connecting its
power and signal lines to the main box. Each box has
holes for cable glands, and the number of holes can beincreased or decreased as the number of sensors or
thrusters changes, ensuring expandability. Additionally,
when this robot adjusts its buoyancy, it fills the polyvinyl chloride (PVC) pipe of the frame with fresh water
to maintain neutral buoyancy. A cap (No. 3) is provided
to make it easy to put in and take out the water.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
643

Development of a Low-Cost

The specifications of this robot are shown in Table 2,
and the system architecture is shown in Fig. 3. This
robot is a size and weight that can be carried by one
person. In terms of watertightness and pressure
resistance, it can withstand water depths of 1.5m for 30
minutes. Testing at deeper water depths is a future task.
To access the internal computer from the external
computer, the ethernet connection is converted to an
optic fiber connection inside the main box, and thesignal is transmitted to the external computer throughthe optical fiber cable. The external computer is pairedwith the joystick device via Bluetooth, and the operator
can remotely control the underwater robot by operating
the joystick device. The internal computer is connectedto each sensor and a micro controller unit (MCU). The
MCU sends signals to the electronic speed controllers(ESCs), which controls the thruster speed, and to theunderwater lights and indicator LEDs. The LEDs are
installed to check the on/off status of the internal
computer and the operating status of this robot. Fig. 4
shows the circuit architecture inside the battery andmain boxes. The battery box has two power supplies:
11.1V for the thrusters and underwater lights, and 5V
for the computer and optical media converter. Note that
5V is converted from 7.4V batteries. For the software,
this robot distributes processing between the internal
computer and MCU and the external computer (Fig. 5).
The former acquires data from each sensor and controls
the thrusters, underwater lights, and indicator LEDs,
while the latter sends commands to the underwater robot
and analyzes the acquired data. This robot realizes
distributed processing using robot operating system(ROS).
The development cost for this robot is approximately$1,800 (250,000 yen). The proportions of parts by
categories are as follows: 6.0% for the frame, 9.6% forcable-related parts, 30.4% for the main box includinginternal computer, MCU, and ESCs, etc., and 6.8% for
the battery box including batteries, converters, and(a) Front view(c) Side view(d) Top viewFig. 1 Design of a low-cost underwater robot (see Table
1 for legend)
Table 1 Components of a low-cost underwater robot
No.

Component

No.

Component
Indicator LDEs

1

Lower frame

122Upper frame

13

3

Cap

14

4

Camera unit

15

5

Underwater Lights

16

6
7

Cable gland for thecamera unit
Cable gland for a
signal line

17
18

8

Main box

19

9

Battery box

20

10

Cable gland for a
power line

21

11

Switches

22

Cable
gland
for
thrusters
Cable gland for theunderwater lights
Depth sensor
Plate for fixing themain box
Plate for connecting
the two boxes
Plate for fixing thebattery box
Joint for fixing thetwo boxes
Thruster guards
Thrustersforclockwise propellers
Thrustersforcounterclockwise
propellers

relays, etc., 8.0% for thrusters, 13.6% for sensors,
17.6% for underwater lights, and 8.0% for otherincluding mechanical and electronic parts and 3D
printer materials. The frame is used PVC pipe. Products
from TAKACHI Electronics Enclosure Co., Ltd. areused for the cable gland, main and battery boxes. Theinternal computer is Jetson Nano and the MCU is
Arduino Micro. The type of battery is LiPo. The thruster
is a brushless motor for drones, and the motor has arotation speed of 3100rpm per unit voltage when no(b) Rear view

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
644

Takuya Fujinagaload is applied. The ESC, camera, depth sensor, etc. areproducts from BlueRobotics. The IMU is used BNO055
produced by BOSCH.

Dimensions

410 x 400 x 240mm (L x W x H)

Weight7.0kg (8.0kg when the pipe is filledwith water)

3. Results and discussion

Batteries

To evaluate the performance of this robot, it
participated in an underwater robotics competition heldon October 7, 2023 [8]. This robot entered in the AUV
category, but its operation method was ROV. The fieldof this competition consists of the following areas: forslaloming between buoys, for passing through gates, forobserving, for searching an acoustic source, for floating.
This robot challenged areas other than the searching an
acoustic source area. Fig. 6 shows the depth data andyaw angle data during this competition. This robot
started to dive into the field (Fig. 6), moved on thebottom ((ii)), traveled the slaloming area ((iii)), passed
the gate, and observed the target ((iv)). Then, this robot
turned its direction of travel toward the starting point,that is, rotated its heading 180 degrees, and returned tothe starting point. Finally, this robot attempted to float,
but the fuse for the 11.1V blown, making it impossible
to continue and the trial ended.
The results of this competition confirmed that thisrobot was able to move on the xy-plane at a speed ofapproximately 0.15m/s. Fig. 6 showed that the depth
and posture of this robot could be measured using eachsensor. In the observation area, the USB camera inside
this robot was able to read the target panel with QR
code and numbers printed on it. On the other hand,regarding the movement in the z-axis direction, the xyplane of this robot has a larger square measure thanother planes, so the underwater drag during floating or
diving is also large. For this reason, the load on thethrusters was greater in the z-axis movement than in thexy-plane one, and the fuse worked as a safety measure.

Sensors

11.1V, 2.2Ah x 2,
7.4V, 2.2Ah x 2
CPU: Quad-core ARM A57 @1.43GHz, Memory: 4GB
USB camera, Depth sensor, IMU

Communications

Ethernet, Optical LAN

ComputerFig. 3 System architecture for in/external of this robot

Fig. 4 Circuit architecture

Fig. 5 Software architecture of in/external computer

Fig. 2 View of developed low-cost underwater robot
Table 2 Specifications©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
645

Powered by TCPDF (www.tcpdf.org)

Development of a Low-Cost

3.

4.

5.

6.

Fig 6 Results of depth data and yaw angle
This robot used a glass tube type fuse, and the holder for
it is made of resin. During test before this competition,
the plastic holder melted due to the heat generated by
the load caused by the rotation of the thrusters.
Therefore, as a safety measure, a 5A fuse was used.
However, when designing this robot, cables wereselected with a permissible current of 15A. Based onthese findings, future tasks for this robot include
reconsidering the shape of this robot and redesigning thecircuit inside the battery box so that it can move in thez-axis direction. Additionally, the purpose of this study
is to develop a low-cost underwater robot platform andmake it open access. Towards this end, future tasks also
include conducting pressure tests, implementing depth
and heading control, dead reckoning, and underwater
object detection using each sensor.

7.

8.

Systems,” J. of Robotics and Mechatronics, 30(2), pp.
238-247, 2018.
T. Matsumura, et al. “Development of a Handy
Autonomous Underwater Vehicle “Kyubic,” Int. Conf. on
Artificial Life and Robotics, pp. 405-408, 2021.
H. Yamagata, et al., “Development of Underwater Robot
for Teaching Material of Basic Mechanics,” J. of the
Robotics Society of Japan, 33(3), pp. 181-188, 2015. (InJapanese)
B. Allotta et al., “A low cost autonomous underwater
vehicle for patrolling and monitoring,” J. Engineering for
the Maritime Environment, 231(3), pp. 740-749, 2017.
B. R. Page, et al. “Highly Maneuverable Low-CostUnderwater Glider: Design and Development,” IEEE
Robotics and Automation Letters, 2(1), pp. 344-349,
2017.
C. Edge at al., “Design and Experiments with LoCO
AUV: A Low Cost Open-Source Autonomous
Underwater Vehicle,” IEEE/RSJ Int. Conf. on Intelligent
Robots and Systems, pp. 1761-1768, 2020.
Techno-Ocean2023, http://ton23.underwaterrobonet.org/(In Japanese, Access 2023-12-12)

Author Introduction
Dr. Takuya Fujinaga
He is an Assistant Professor at the
Department
ofElectronics
EngineeringandComputer
Science,
Fukuoka
University,
Japan. His research interests are in
agricultural robots, underwater
robots and field robotics.

4. Conclusion
This paper described the development policy and costof a low-cost underwater robot. This robot was ahovering-type and ROV. The development cost was$1,800. This paper analyzed the sensor data anddiscussed the motion performance based on the resultsin the robotics competition. The next tasks are to
improve motion performance in the vertical direction
and make it an AUV using sensor data. The goal is toopen source the requirements and tacit knowledge forthe development of the low-cost underwater robot.
References
1.

2.

U. Tamaki, “Robots for Underwater World,” J. of the
Robotics Society of Japan, 22(6), pp. 692-696, 2004. (InJapanese)
Y. Nishida, et al., “Underwater Platform for Intelligent
Robotics and its Application in Two Visual Tracking©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
646

Enhancing Reconnaissance Missions Through Multiple Unmanned Systems in ROS
Anees ul Husnain

Department of Electrical Engineering, Faculty of Engineering, University of Malaya, Malaysia
The Islamia University of Bahawalpur, Pakistan
Norrima Mokthar

Department of Electrical Engineering, Faculty of Engineering, University of Malaya, Malaysia
Takao Ito

Hiroshima University, Japan
Siti Sendari

Universitas Negeri Malang, Indonesia
Muhammad Farris Kyasudeen

University Technology MARA (UiTM), Malaysia
Muhammad Badri M Noor

Universiti Malaya, Malaysia, Ifcon Technology Sdn Bhd, Malaysia
Heshalini Rajagopal

Department of Electrical and Electronics Engineering, MILA University, 71800 Nilai, Negeri Sembilan, Malaysia
Email: norrimamokhtar@um.edu.my

Abstract
The synergistic collaboration between UAVs and UGVs addresses the limitations of individual platforms, offering a
versatile solution for reconnaissance tasks in diverse environments. The proposed system employs ROS as theunderlying architecture to facilitate seamless communication and coordination among multiple UAVs and UGVs.
We delve into the intricacies of developing a robust communication framework that enables real-time data exchange
and decision-making, fostering a synchronized and adaptive operation. Furthermore, the article explores strategies
for path planning and navigation, considering the unique mobility constraints of UAVs and UGVs. Optimal coverage
is ensured through efficient exploration and coverage of the reconnaissance area, under by comparing raster-scan,
expanding spiral and zig-zag area exploration approaches. The article concludes by discussing potential extensions,such as the integration of machine learning techniques for enhanced autonomy and the scalability of the system forlarger-scale missions by presenting a ROS-based framework that maximizes the synergy between UAVs and UGVs.
Keywords: Heterogeneous unmanned systems, Cooperative Path Planning, Reconnaissance, UAV, UGV, ROS

cooperative reconnaissance strategy for multiple
unmanned vehicles in unknown areas has proven to bereliable and improves the heading errors of the vehicles[1]. Effective reconnaissance in modern military
applications possesses the key for a strategic advantage,
and also helps to neutralize surprise attacks, ambushes
and can prevent the damages through improvised
explosive devices (IEDs) with proactive sensory modules[2]. This is why combining Unmanned Ground Vehicles

1. Introduction
With the surge in utilization of unmanned systems
for research and their applications, the need forcooperation among heterogenous systems has made to
the spotlight. The cooperation among heterogeneous
systems exploits the diversity of features in unmannedsystem to achieve such mission objectives which are near
impossible for homogenous systems. Forming a©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024647

Anees ul Husnain, Norrima Mokthar, Takao Ito, Siti Sendari, Muhammad Farris Kyasudeen, Muhammad Badri M Noor, Heshalini Rajagopal(UGVs) alongside UAVs to coordinate area
reconnaissance combines the technical and tacticalcapabilities of both to achieve the same. However, there
are certain challenges linked with combing theseunmanned systems.

2.2. Measure of Objectiveness in coordination
Self-assembly is a technique to attain cohesion inheterogenous robots which is perceived as robotic
intelligence. The robots are required to assemble
themselves that may not be possible throughmaneuverability [4]. There are three primary objectives
under self-assembling the UAV and UGV to generate
maneuver instructions and capitalize on interaction
between them to work on complex tasks more efficiently.

This work addresses one of such challenges thatnaturally appears when the UAVs and UGVs are required
to perform a coordinated reconnaissance in an unknown
area. The nature of terrain and obstacles can affect theconditions required to maintain the reconnaissanceformation between the aerial and ground vehicles [3]. A
set of conditions is established which comprises two key
components: a). The unmanned vehicles, UAV and UGV,
have to explore the territory in close proximity. b).
Minimize the impact of terrain-based slowness of UGVon the UAV.
This work presents a novel technique andcoordination architecture that ensures the conditions
above and explores the area effectively.
2. Proposed Method(i)

Robustness: To reperform the task while
maintaining the cohesion in proximity incase of any missing area to be explored.

(ii)

Versatility: To adjust and readjust the
maneuvers because of uneven terrain andground obstacles for the UGV.

(iii)

Cost reduction: To minimize the cost for
the above. Here the cost is computed interms of the time lapsed and number ofturns a robot has to take. through design oflarge numbers of robots.

2.1. UAV / UGV Coordination Architecture
2.3. Attraction / Repulsion Mechanism in UAV /UGVThe requirements for reconnaissance formation are
primarily
derivedfromthedifferencesinmaneuverability between both of the vehicles and impact
of terrain on movability of UGV. The relative positionsof the heterogenous vehicles, reconnaissance objectives
and the environment data (grid map) are combined togenerate the planning data, as shown in Fig. 1. The
planning data further combines with the relative positionsof UAV and UGV, and the trajectory data from UAV is
fed to the path planning unit. The path planning unit,
considering the conditions mentioned in Section 1,
generates instructions for both UAV & UGV for
coordinated maneuvers alongside requirements likecollision avoidance.

There are two primary tasks which are being
performed at all the time: collision avoidance for the
ground vehicle and flocking of UAV with respect to theground vehicle. Collision avoidance in this work isbasically a necessary condition for the UGV to maintaina minimum safe distance from its surroundings objects.
This work is inspired from a recent published work
which engaged heterogenous robots in an ‘invisible
binding’ by implementing an “Attraction / Repulsion”
mechanism using Voronoi partitioning, based on
Voronoi Diagrams for collision avoidance [5]. However,
our approach exploits the idea of attraction-repulsion forheterogenous robots and generates path instructions in azone-based coordinate system. Aggregation anddispersion are the two phenomena which are derived
from the inter-robot’s distance and their distribution in
the area needed to be explored. This work mitigates thisproblem through establishing zones based on the
distances and focuses on reconnaissance of the territory.
The proposed model consists of four constraints i.e.
placement, repulsion, attraction, and orientation. The
placement may take place either randomly or with someprior information from the area of reconnaissance. The
model is based on a set of universal equations proposedby Ian Couzin cited by [6]. Referring to Eq. (1), Eq. (2)Fig. 1. UAV/UGV Coordination Architecture©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
648

Enhancing Reconnaissance Missions Throughand Eq. (3), where dr, do and da represent the radii ofrepulsion, operation and attraction zones with i and j
representing coordinates. Fig. 2 presents an illustration
for the three scenarios.
𝑑𝑟 (𝑡 + 𝐿̃) = − ∑𝑀𝑟𝑟𝑖𝑗(𝑡)

𝑗≠𝑖 |𝑟𝑖𝑗(𝑡)|𝑑𝑜 (𝑡 + 𝐿̃) = ∑

𝑀𝑜

𝑉𝑗(𝑡)

𝑗≠𝑖 |𝑉𝑗(𝑡)|𝑀𝑎
𝑟𝑖𝑗(𝑡)𝑑𝑎 (𝑡 + 𝐿̃) = − ∑𝑗≠𝑖 |𝑟𝑖𝑗(𝑡)|

(1)
(2)
(3)

Fig. 3. Parallel Maneuver Generation for UAV /UGV

3. Implementation
3.1. Simulation Setup
Robot Operating System (ROS) is an opensource
platform that provides libraries for and tools for a diverse
range of robotic applications. These libraries include thehardware abstraction, driver packages, data visualizers,
inter-module communication packages and protocols [7].
Gazebo on the other hand simulates the robotic
application in a simulated environment [8]. Both theseprojects are lead by a community called open robotics [9].

Fig. 2. Attraction Repulsion Mechanism with Placement
and operational zone

For this work, we consider HUSKY and HECTOR
which are emulated versions of opensource UGV and
UAV, respectively. The integration of Husky and Hector
within ROS provides the foundation of collaborativeautonomous systems, showcasing the potential
coordination among heterogeneous robots. There are four
key components for setting up this collaboration:(i) Sensor data Integration(ii) Communication of sensor data, control
instructions and status of UAV / UGV(iii) Motion Control and Mission Planning.
(iv) Mapping and Navigation.

2.4. Parallel Maneuver Generations for UAV / UGVHere it is important to mention that either of theheterogeneous robots has to be set as the leader orfollower. Moreover, the radii of the zones are adaptableaccording to the terrain and area under consideration. The
maneuver generation takes sensor and positioning datafrom both unmanned vehicles and create relative
positions of UAV and UGV, refer to the Fig. 1. This data
generates maneuvering instructions in a 2D workspace,assuming a constant altitude of the UAV. Fig. 3 presents
the maneuver generations of for both UAV and UGV. To
better understand the flow of an instruction generation,
consider the UAV and UGV in the placement zone.

3.2. Experimentation
The experimental scenarios are established for acomprehensive evaluation of the collaboration andexploration capabilities of Hector and Husky. There are
two primary considerations in mind before setting up
experimental scenarios:(i) To validate the ability of this collaboration
system for exploration and mapping(ii) To evaluate the adaptiveness of the systemagainst the challenges in the environment.
(Under this work, the challenges refer to static
obstacles and uneven terrain.)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
649

Anees ul Husnain, Norrima Mokthar, Takao Ito, Siti Sendari, Muhammad Farris Kyasudeen, Muhammad Badri M Noor, Heshalini Rajagopal

3.3. Flow of Activities

based on the combining the aerial and ground data. The
UGV is instructed to follow a programmed exploration
strategy which can be modified as needed, while the
Hector UAV performs the mapping of the aerial mappingof the environment.
Fig. 5 presents the initialization of exploration throughHusky in the pattern given to it. While Fig. 6 presents theaerial mapping of Hector of the environment shown in
Fig. 4.

(i) Initialize the ROS-Gazebo environment throughROS launch file. For uneven terrain, there is astock environment available in Gazebo which
has been deployed.
(ii) Deploy Hector UAV and Husky UGV equippedwith LiDAR sensors.
(iii) Command Husky for Autonomous Explorationand Hector for aerial mapping.
(iv) Initiate the program for exchanging Lidar andpositioning data between Husky and Hector andcompute relative positions. The relative dataand position histories are stored and explored
regions are marked with the coordinates of theenvironment.
(v) Constraints initialization: This is the main
program which generates the thresholds fromthe relative positions. This module also utilizes
LiDAR data from Husky for its collision
avoidance.
(vi) Creation of unified map that combines the
explored area from both of the platforms.
(vii) Collection of performance parameters which arepercentage of area explored, number ofmaneuvers required, and overall time taken.
(viii) Analysis of exploration data and suggestions for
optimizations.

Fig. 5. Expanding Circular Exploration lead by Husky
UGV, path trail in RVIZ visualizer

4. Results and Discussion
Since there isn’t any notion of geo-referenced
coordinates in Gazebo, therefore, the positioning data inthe simulation, for both the platforms, is collected in
Cartesian Coordinate space. Fig. 4 presents thescreenshot of deployment of both platforms while
coordinating repulsive motion.
Fig. 6. Aerial Mapping being conducted through Hector
UAV, LiDAR data being shown in RVIZ

5. Conclusion
The work demonstrated a successful integration of
two heterogeneous autonomous unmanned systems in
ROS-Gazebo environment, proving their potential to usethem for reconnaissance applications in unknown
workspaces. The key feature of this work was to propose
an attraction-repulsion mechanism among the aerial andground vehicles for a strategic and coordinatedexploration. This work is at the early stages ofexperimentation and possess the potential for further
enhancements which primarily include optimization of
exploration algorithms and strategies. Moreover, thisFig. 4. Hector and Husky Deployment in a stock
environment in Gazebo
The synergic collaboration between the two
autonomous platforms performed simultaneous mapping©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
650

Enhancing Reconnaissance Missions Throughwork just incorporates two unmanned systems which canbe expanded to exhibit a swarm intelligence behavior
among UAVs or UGVs. We believe that the future of
exploratory missions in unknown or hostileenvironments is very promising when it comes tounmanned systems.

Authors Introduction
Mr. Anees ul Husnain
Anees is currently pursuing his Doctoral
in Electrical Engineering at Universityof Malaya, Malaysia. He holds a
master’s
degreeincomputerengineering from UET Taxila, Pakistan.
He is working on autonomous pathgeneration of UAVs to monitor fugitive
emissions.

6. Acknowledgements
This work is conducted under research groupformed at University of Malaya, Kuala Lumpur in which
the individuals from diverse universities and industries
participate. The author gratefully acknowledges ACR
Lab, Faculty of Engineering, University of Malaya andtheir members for providing assistance duringassembling and testing.

Ir. Dr. Norrima Mokhtar
Norrima Mokhtar received the
Bachelor of Engineering (B. Eng)
degree in Electrical Engineering from
University of Malaya in 2000. She wasappointed as a lecturer to serve theDepartment of Electrical Engineering,
University of Malaya immediately
after graduating with her Master of Engineering. As part ofher career development, she received SLAB/SLAI
scholarship to attain her Ph.D. in 2012. She is now serving
Department of Electrical Engineering, University of
Malaya.

References
1.

2.

3.

4.

5.

6.

7.
8.
9.

H. Zhang, T. Yang, and Z. Su, "A formation cooperative
reconnaissance strategy for multi-UGVs in partiallyunknown environment," Journal of the Chinese Institute ofEngineers, vol. 46, no. 6, pp. 551-562, 2023/08/18 2023,
doi: 10.1080/02533839.2023.2227879.
J. Nohel, P. Stodola, J. Zezula, P. Zahradníček, and Z.
Flasar, "Area reconnaissance modeling of modular
reconnaissance robotic systems," The Journal of Defense
Modeling and Simulation, p. 15485129231210302, 2023,
doi: 10.1177/15485129231210302.
A. Wang, F. Jing, X. Huang, C. Gong, and S. Xu,"Structure Keeping Control for Heterogeneous Formations
Based on Consistency Theory and Graph Theory," in 2023
9th International Conference on Mechatronics andRobotics Engineering (ICMRE), 10-12 Feb. 2023 2023, pp.
139-145, doi: 10.1109/ICMRE56789.2023.10106518.
K. S. N. Ripon, E. Jakobsen, C. Tannum, and J. M.
Montanier, "Assessing the effect of self-assembly ports inevolutionary swarm robotics," in 2016 IEEE Symposium
Series on Computational Intelligence (SSCI), 6-9 Dec.
2016 2016, pp. 1-8, doi: 10.1109/SSCI.2016.7850181.
A. Eudes, S. Bertrand, J. Marzat, and I. Sarras,"Distributed Control for Multi-Robot Interactive
Swarming Using Voronoi Partioning," Drones, vol. 7, no.
10, doi: 10.3390/drones7100598.
M. Aguilera, J. A. R. Linan, and L. M. Torres-Treviño,"Change of Behavior on Swarm Robotics Using Repulsion
and Attraction," in 2015 Fourteenth Mexican International
Conference on Artificial Intelligence (MICAI), 25-31 Oct.
2015 2015, pp. 169-172, doi: 10.1109/MICAI.2015.32.
R. Wiki. "About ROS." Wiki. http://wiki.ros.org/(accessed January 21, 2024).
GazeboSim.
"AboutGazeboSimulator."
https://gazebosim.org/home (accessed January 21, 2024).
OpenRobotics. "About - Openrobotics." OpenRobotics.
https://www.openrobotics.org/ (accessed January 21,2024).

Dr. Takao Ito
He is Professor of Management ofTechnology (MoT) in Graduate SchoolofEngineering
at
Hiroshima
University. He is serving concurrently
as Professor of Harbin Institute ofTechnology (Weihai) China. He has
published numerous papers in refereed
journals and proceedings, particularly in the area ofmanagement science, and computer science. He has
published more than eight academic books including abook on Network Organizations and Information(Japanese Edition). His current research interests include
automata theory, artificial intelligence, systems control,
quantitative analysis of inter-firm relationships usinggraph theory, and engineering approach of organizational
structures using complex systems theory.
Dr. Siti Sendari
She is a lecturer at Universitas Negeri
Malang, Indonesia, specializing in
intelligent systems, robotics, andevolutionary computation. She holds
the position of Head of Laboratory intheDepartment
ofElectricalEngineering at the Faculty ofEngineering. Her work focuses on
intelligent systems and related areas,
contributing to the academic community.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
651

Powered by TCPDF (www.tcpdf.org)

Anees ul Husnain, Norrima Mokthar, Takao Ito, Siti Sendari, Muhammad Farris Kyasudeen, Muhammad Badri M Noor, Heshalini Rajagopal

Ir. Muhammad Farris Kyasudeen
He is associated with Universiti
Teknologi MARA (UiTM) in Malaysia.
He is part of the Faculty of ElectricalEngineering at UiTM's branch in Pulau
Pinang, where he holds the position of
Ir. (Ir.) indicating his professional
engineering qualification. He has also
received recognition for winning third place in acompetition related to Electrical Engineering at UiTM. His
contributions include research on an Autonomous Ground
Vehicle (COR-AGV) disinfectant system using far-UVC
light exposure.
Mr. Muhammad Badri M Noor
He is associated with Ifcon
Technology, as mentioned in the search
results. He is linked to the University of
Malaya, specifically in the Departmentof Computer System & Technology,
Faculty of Science Computer & IT.
Moreover, he has contributed to a
research paper on Remote Desktop Power Management
System using Single-Board Computer, showcasing his
expertise in this field.
Dr. Heshalini Rajagopal
She received her PhD and Master's
degree from the Department ofElectrical Engineering, University of
Malaya, Malaysia in 2021 and 2016,
respectively. She received the B.E(Electrical) in 2013. Currently, she is an
Assistant Professor in UCSI University,
Kuala Lumpur, Malaysia. Her researchinterest includes image processing, artificial intelligenceand machine learning.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
652

Illumination Effects on Facial Expression Recognition using Empirical Mode Decomposition
Hasimah Ali, Wan Khairunizam Wan Ahmad

Centre of Excellence Unmanned Aerial Systems (COE-UAS), Faculty of Electrical Engineering & Technology,
Universiti Malaysia Perlis, Arau, 02010, Malaysia
Hariharan Muthusamy

Department of Electronics Engineering, National Institute of Technology Uttarakhand, India
Mohamed Elshaikh

Department of Electronics Engineering & Technology (FTKEN), Universiti Malaysia Perlis, Arau, 02010,
Malaysia
E-mail: hasimahali@unimap.edu.my, hariharan@nituk.ac.id, khairunizam@unimap.edu.my, elshaikh@unimap.edu.my
Abstract
Facial expression recognition (FER) has been acknowledged as a significant modality that could bring facial
expression into human-machine interaction and make the interaction more efficient. However, the ability of FER to
operate in a fully automated and robust manner is still challenging. Illumination effects, for example, make the facial
expression images always contaminated with different levels of ambient noise (such as brightness variation) in acluttered background. Thus, this paper aims to investigate the illumination effects (brightness variations) on facial
expression recognition using empirical mode decomposition reconstruction techniques. In this framework, firstly, thenoisy facial expression images were simulated with the illumination effects using different brightness levels of 30%,
40%, 50%, 60%, and 70%. Then, the EMD will decompose the noisy facial expression images into a small set of
intrinsic mode functions (IMF), namely IMF1, IMF2, IMF3, and residue. Based on property held by EMD, the signals
are decomposed into several IMF components, each with a different time scale. Because the last several IMFs
represent the majority of illumination effects, various reconstruction techniques for IMFs have been investigated atvarious brightness levels. Feature reduction techniques Principal component analysis (PCA) and linear discriminant
analysis (LDA) have been employed to reduce the high-dimensional space of IMF features into low-dimensional IMF
features. The reduced IMF reconstructions were then used as input to the k-nearest neighbour classifier to recognise
the seven facial expressions. A series of experiments have been conducted on the JAFEE database using various
reconstruction IMFs together with PCA plus LDA. Based on the results obtained, the reconstruction of IMF1 + IMF2
+ IMF3 shows the highest accuracy in high illumination conditions, which is 99.06%.
Keywords: Illumination, Facial Expression Recognition, Empirical Mode Decomposition, PCA plus
LDA.

behaviour can be effectively extracted from their facial
expression. Facial expression recognition (FER) has been
acknowledge as a significant modality that could bringfacial expression into human machine interaction andmakes the interaction more efficient. However, theability of FER to operate in fully automated with robust
is still challenging. Illumination effects for example,
make the facial expression images always contaminatedwith different level of ambient noise (such as brightness
variation) in clutter background. In previous work,
limited study was conducted on illuminations effects for
facial expression recognition. For example, [3] havecombined the logarithm transform, discrete cosine

1 Introduction
Facial expression is the most natural and important tool
people have to show their emotional-state or deliver
information. For decades, facial expression has been
regarded as an important modality for conveying feelings
and attitudes, and it has had an impact on physicalcommunication. Specifically, facial expression is formedby contracting facial muscles [1]. Facial expression
recognition (FER) is a function that humans andcomputers can perform by locating the face from its
sources. Then, facial features will be analysed to classify
the emotion [2]. A lot of information about human©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
653

Hasimah Ali, M Hariharan,, Wan Khairunizam Wan Ahmad, M Elshaikhtransform, and illumination compensation as anormalised DCT to eliminate illumination variation
contained in the facial images of the JAFFE database. To
recognise the facial expression, the extracted featuresfrom the pre-processed image were combined with aneural network classifier. However, analysis of theeffects of illumination was not discussed. In [4], theauthors utilized the independence component analysis(ICA) and locality-preserving projection to compare theeffectiveness of those methods towards the illumination
variations on the face recognition. They found ICA
outperformed the results on YALEB database with 64
illuminations types. In [5], the authors utilized deeplearning approach for emotion recognition under
different pose and illumination variations. The
Convolution Neural Network were constructed using 15
layers (three layers per emotion) that learn from facial
expression database (CMU Multi-PIE) using input sizeof 32 x 32 in recognizing the five basic emotions.
Although they achieved 96.55%, selecting optimum the
input size considered to be computational cost. In [6] theauthors utilized a deep stacked convolutional
autoencoder in attempting reconstruct new input images
with better illumination for facial expression recognition.
The model used as pre-training in greedy layer-wise bylearning the image representation and then map it toapproximately reconstruction of input image. The
network model learns to encode the input images andcreate a feature vector form relatively similar
illumination, ignoring the level of luminance of the facial
images. Even though the used of deep learning isemerging, however handcraft features of the image
provides significant impact in understanding the
behaviour of spatial details of the images. Therefore, inthis study, we propose to investigate the illumination
effect on facial expression recognition (FER) by using
empirical mode decomposition (EMD) approach. In thisframework, the original facial expression database willgo through face detection in a manipulated illumination
condition, and then various illumination coefficients will
be analysed. The EMD will be used to decomposed theilluminated images (in our case brightness variations)
into a set of intrinsic mode functions (IMFs). These IMFs
will be used to further analyse the illumination effects
containing in facial expression images. The significant
IMFs will be reconstructed by removing the one contains
illuminations. In order to reduce dimensionality ofreconstructed IMF, PCA plus LDA will be adopted.

Finally, the reduced features of reconstructed IMFs will
be classified using k-NN classifier to recognize the sevenfacial expression under the illumination effects.

2 Materials and Method
Fig. 1 shows the framework of the proposed system. It
consists of five phases: facial expression image database,pre-processing, EMD-based feature extraction, IMF
reconstruction, and classification. In this framework,
firstly, the facial expression image was pre-processed todetect the local region of the face. Then, brightness
variations were simulated on the detected face. By
decomposing an image into a set of intrinsic mode
functions via a sifting process on the simulated brightnessof a facial expression database, the nonlinear technique,namely EMD, has been used as a feature extractionmethod. To assess performance, an IMF reconstruction
was performed on various IMFs. Later, feature reduction
was applied to the reconstructed image to reduce its
dimensionality before being subjected to the classifier.
The detailed description of the process is highlighted asfollows:

Fig. 1. Framework of the proposed method

2.1 Facial Expression DatabaseIn this study, the Japanese Female Facial Expression(JAFFE) database has been used. It contains 213
grayscale facial expression images of ten subjects, allJapanese women, with seven facial expressions: neutral(30 images), angry (30 images), sad (30 images), happy(31 images), disgust (29 images), surprise (31 images),and fear (32 images).

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
654

Illumination Effects on Facial

2.2 Image Pre-Processing: Face Detection

mode function, and numerical convergence criteria tostop the sifting phase.
In the EMD technique, there are two steps that must be
met to extract the intrinsic mode functions (IMFs). To
begin, the number of extremes and zero crossings must
be equal, or at most one. Second, the mean of its upperand lower envelopes must not be more or less than zero.
Those two steps satisfy the physically compulsory
process for defining a significant instantaneous
frequency. The EMD technique's entire procedure is asfollows: for input signal s(t):Face detection is a technique used to detect and localize
the face region from the unwanted background. The
original size of JAFFE image is 256 x 256 pixel was
cropped into 128 x 96 pixel that contains the face region.
Fig. 2(a) and Fig. 2(b) show the example of the original
and cropped image of seven facial expressions of a
JAFFE subject, respectively.

1.
2.

Fig. 2. Seven facial expression of JAFFE’s subject: (a)original and (b) cropped image

3.
4.
5.

2.3 Brightness Variation
In this study, illumination variation with different levelsof brightness was used to simulate illumination effects
that mimic real-world applications. The illumination
variations were simulated by varying the brightness
effect by controlling the intensity of each individual pixel
inside the image. The percentage illumination level usedin this project is 30%, 40%, 50%, 60%, and 70%. A
percentage greater or less than those specified will resultin either increased brightness or decreased darkness.

6.

Determine the local maxima and local minimaof s(t). Let d0(t) = s(t).
Incorporate the local maxima and the local
minima to develop the upper envelope eu(t)and lower envelope el(t), correspondingly,
using cubic B splines.
Calculate the mean of envelope m(t): m(t) =(eu(t) + el(t))/2.
Determine the details using d1(t)=d0(t)-m(t)(sifting phase).
Repeat steps 1-4 on the residual signal until thedetails signal dk(t) is included in the
IMF:c1(t)=dk(t)Procedures 1–5 should be repeated on theresidual rn(t)=s(t).Cn(t) to obtain all of thesignal's IMFs (c1(t), c2(t),..., cn(t)).

When the residual signal is either unaltered, monotone,
or a function with just one extrema, the procedure is
terminated. The advantage of the EMD approach is thatonce these IMFs are acquired, they may easily switch
back and forth from them to the original signal. Similarly,
the original signal may be reconstructed by combining allof the IMFs together, which can be adjusted to accountfor slight variations owing to the implant discovered in
the method. EMD can be expressed in mathematically as:N(1)
X (n) = C (t ) + R (t )2.4 Feature Extraction: Empirical Mode
Decomposition
Empirical mode decomposition (EMD) is a method of
decomposing a natural signal without leaving the time
domain. EMD decomposes any nonlinear signal into asmall set of finite intrinsic mode functions (IMFs).
Although EMD is focused on analysing one-dimensionalsignals, it can also be extended to bi-dimensional
empirical mode decomposition (BEMD) to analyse twodimensional signals while still having similar concepts
and procedures [7], [8]. The main advantage of EMD isthat it can extract data about local trends in the input
signal by calculating and quantizing oscillations. Suchoscillations can be quantized by local details andmatching a local trend. The fundamental ideas of EMD
are the implementation of the incorporation method suchas cubic splines, sifting phase to extract the intrinsic


n=1

n

N

According to the technique, lower-order IMFs record
rapid oscillation modes, whereas higher order IMFs
record low oscillation modes. In this research, twodimensional empirical mode decomposition (BEMD) isused to reduce illuminated face pictures into a limited
collection of intrinsic mode functions (BIMFs). The
BIMFs are features recovered at various scales or spatial
frequencies using phase sifting.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
655

Hasimah Ali, M Hariharan,, Wan Khairunizam Wan Ahmad, M Elshaikh

2.5 Dimensionality reduction: PCA plus LDA

Wlda = arg max

In this work, the PCA plus LDA has been employed as
dimensionality reduction on the extracted IMFs features
2.5.1

PCA

ii.

iii.

2.5.2

2.5.3

1 n
(ai −  )(ai −  )T
n i=1

Calculate both eigenvalues
eigenvectors ( vi ) of S as:

(

i )

where i = {1, 2, 3..N} is the number of input images.

(2)
3 Results and Discussion
andIn order to evaluate the effectiveness of the proposedsystem, the JAFFE database, which is publicly available,
has been used in this experiment. The facial images
obtained from successful face detection and applied withdifferent levels of illumination will undergo feature
extraction using the EMD technique in order to extract
the features called intrinsic mode functions (IMFs)
through sifting procedures. IMFs are extracted fromhigh-dimensional data. As a result, feature reduction
using PCA and LDA was used before being fed as input
to the k-NN classifier to classify all seven emotions:anger, disgust, fear, neutral, happiness, sadness, andsurprise.

(3)
Svi = i vi ,where i=1, 2, 3…n.
Lastly, sort the eigenvectors in descending orderthat associated to the largest eigenvalues. The
new projected space is given b:(4)
y =WTa ,which W is represent the transformation matrix
[9].
LDA

One of the main problem in pattern recognition is thecurse of dimensionality. LDA is one of the method used
to solve the issue of dimensionality by reducing thefeatures from higher-dimensional space to lowerdimensional space. The goal of LDA is to shape the classscatter by maximizing the between-class scatter andminimising the within-class scatter [9]. In this framework,
firstly we need to compute the between-class variance SB
as:n

S B =  Ni (i −  )(i −  )T

PCA plus LDA

To avoid singularity of the Sw, [10] proposed PCA plus
LDA. This framework involves two stages. First, original
feature vector having f-dimensional feature space ismapped onto t-dimensional intermediate feature space inwhich (t < f) via PCA. Then, project the intermediate (tdimensional) to k-dimensional feature space via LDA
resulting new projected space as:TT(7)z i = W pca
Wlda
xi

First, calculate the mean (µ) of the matrix, andcovariance matrix (S) as:

S=

(6)

in which wi (i = 1, 2,…m) is the set of SB and Sw
generalized eigenvectors based on the largest generalized
eigenvalues i of S B wi = i SW wi .

Although the intrinsic mode functions effectively
represent the different frequencies of oscillation fordifferent expressions (local trends) in the image, themode function responses can be large. A 128 x 96 image,for example, decomposes with the EMD resulting mode
function into a 128 x 96 = 12,288-dimensional vector. To
reduce the dimensions of the feature vector, we applied
PCA to our mode function. Let consider, if
A = {a1 , a2 , a3 ...aN } is a vector of N images.
i.

W T S BW= w1 , w2 , w3 ...wm 
W T SW W

3.1 Brightness Variation on Facial Expression
Images
The facial images from JAFFE database are simulatedwith different level of illumination which is 30%, 40%,
50%, 60% and 70%. This is done by adding andsubtracting the intensity of each pixel inside of the facial
images. Fig. 3 shows the simulated illumination of facial
expression images.

(5)

i =1

Second,c within-class scatter, Sw need to be computed by(ak −  i )(ak −  i )T .Finally, the lowerSW =i =1 ak Afeature
dimensional
projection, Wlda can be determined
i
via:



Fig. 3. Simulated illumination effects of facial
expression images at different brightness level©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
656

Illumination Effects on Facial

3.2 Performance of applied EMD on expression
images
The two-dimensional breakdown of a picture using thesifting process results in an interpretable display. EachIMF carries information on a certain scale that is
efficiently segregated. Fig. 4 depicts the EMD
decomposition of facial expression image into IMF1,
IMF2, IMF3, and residual. Note that, the IMF1 contains
local information about the smallest scales in the image,whereas the residue contains information about greater
scales [7]. Specifically, IMF1 corresponds to the lowest
time scale associated with the information's quickest time
fluctuation. As the decomposition process progresses, thetime scale increases, and the mean frequency of the mode
decreases. As observed in Fig. 4, the set of IMF reveals
the influent structure from the smallest to the largest
relative to the input images. The IMF1 shows the mostdistinct facial features, including the boundary lines
around the face region such as the mouth, nose, and eyes,
which are crucial for expression recognition.

Fig. 5. Reconstruction of different mode functions (30%brightness level): (a) IMF1+IMF2, (b) IMF1+IMF3 and(c) IMF1+residue, (d) IMF2+IMF3, (e) IMF2+residue,(f) IMF1 + IMF2 + IMF3, (g) IMF1 + IMF2 + residueand (h) IMF1 + IMF2 + IMF3+residue (reference)

3.4 Performance Analysis of Applied PCA plus
LDA on reconstructed IMFs
In this work, three reconstruction modes of IMFs havebeen investigated: IMF1+IMF2, IMF1+IMF2+IMF3,
and IMF1+IMF2+IMF3+residue. Due to the higher
dimensionality of feature space, the frameworks of PCA
plus LDA were applied. The rationale for using PCA plus
LDA is that, the feature vector space was reduced from ahigh-dimensional to a low-dimensional space by utilising
the PCA plus LDA framework as mentioned in Section
2.5. Fig. 6 shows the distribution of reduced features due
to IMF1+IMF2+IMF3+residue, whereas Fig. 7 shows the
distributionofreducedfeaturesdueto
IMF1+IMF2+IMF3 at a 30% brightness level. Note thatclasses 1, 2, 3, 4, 5, 6, and 7 are denoted as angry (red),
disgust (orange), fear (green), happy (cyan), neutral(blue), sad (purple), and surprised (pink), respectively. It
can be seen in Fig. 6 that there is a low degree of overlap
between the emotions of fear, anger, disgust, and neutral.
Fig. 7 shows that the class emotions are welldifferentiated from one another.

Fig. 4. EMD decomposition that associated with
30% brightness level

3.3

Performance Analysis on Facial Expression
Recognition using Reconstructed Intrinsic
Mode Function

According to [11], the majority of the illumination effects
is represented by the last several IMFs. To evaluate how
EMD can be employed to reduce the illumination effects
in the proposed system. Fig. 5 shows the different
combinations of the reconstructed mode functions. It
should be noted that Figure 5(h) is a reconstruction of all
IMF and a residue, and thus serves as a reference image.
As can be seen in Fig. 5, the illumination effect was
highly dominant in (b), (c), and (d). Whereas, (e), (f), and(g), the illumination effect appeared at a low degree.
Meanwhile, in (a), which is IMF1+IMF2, the effects
were almost gone. This observation agrees with [11] inwhich the several last mode functions are affected byillumination.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
657

Hasimah Ali, M Hariharan,, Wan Khairunizam Wan Ahmad, M Elshaikh

highest recognition rate, which is 99.06% at a 30%
brightness level. It seems that the combined IMF1+IMF2
is enough and would achieve similar recognition as in thereference. We agree that, the illumination effect may lie
inside the IMF3 and residue. When the brightness level
increased to 40% (Fig. 9), the combined IMF1+IMF2
again showed the highest recognition rate of 99.53%,
which is slightly higher by 0.53% than the reference.
In Fig. 10, at a brightness level of 50%, thecombined IMF1+IMF2+IMF3 shows the highest
recognition of 99.53%, which is the same as the reference.
Similar trends appear for brightness levels of 60% (Fig.
11) and 70% (Fig.12), where the combined
IMF1+IMF2+IMF3 outperformed the results, which are
99.53% and 98.59%, respectively. This can be inferredfrom the fact that as the brightness level increases, thecombined IMF without residue shows significant results.
Thus, we agree that the majority of illumination effects
may be represented by the last IMFs and residue.

Fig. 6. Distribution of reduced features of
IMF1+IMF2+IMF3+residue
for
seven
facialexpressions at 30% brightness level.

Fig. 7. Distribution of reduced features of
IMF1+IMF2+IMF3 for seven facial expressions at 30%
brightness level.

3.5 Experimental Results
Fig. 8. Accuracy of FER based on reconstructed IMF at
30% brightness level.

In this section, the reduced features using PCA plus LDA
were applied to the reconstruction IMFs to recognize theseven facial expressions. The objective is to investigate
the effectiveness of reconstruction IMFs in dealing withvariations in illumination. Three different modes of IMF
reconstruction
wereevaluated,namelyIMF1+IMF2+IMF3+residue (used as reference),
IMF1+IMF2+IMF3, and IMF1+IMF2 under different
levels of illumination, which are 30%, 40%, 50%, 60%,
and 70%. In this work, k-nearest neighbour was used asa classifier using a 10-fold cross validation strategy.
Fig. 8, Fig. 9, Fig. 10, Fig. 11, and Fig. 12
illustrate the recognition rate of IMF reconstruction of
facial expression recognition at brightness levels of 30%,
40%, 50%, 60%, and 70%, respectively. As observed inFig. 8, the reconstruction of IMF1+IMF2 shows theFig. 9. Accuracy of FER based on reconstructed IMF at
40% brightness level.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
658

Illumination Effects on Facial

4 Conclusion
This paper used an empirical mode decomposition
approach to present the effects of illumination on facial
expression recognition. Firstly, pre-processing of the
facial image has been done via face detection to extract
the local region of the face. Then, the face image wassimulated to mimic the illumination effects by varying
the brightness level between 30%, 40%, 50%, and 70%.
Then, the simulated facial images were decomposedusing empirical mode decomposition into a finite set of
intrinsic mode functions. This work (IMF1, IMF2, IMF3,
and residue) has been extracted via a sifting process.
According to the findings, illumination effects may havebeen observed in the last few IMFs. Three reconstruction
modes were considered for illumination analysis, which
are the combined IMF1+IMF2, IMF1+IMF2+IMF3, andIMF1+IMF2+IMF3+residue (as reference). Based on the
results obtained, at low brightness levels (30% and 40%),the combined IMF1+IMF2 gives the highest recognition
rate of FER. However, as the brightness level was further
increased to 50%, 60%, and 70%, the combined
IMF1+IMF2+IMF3 gave the highest recognition rate.
Therefore, we can conclude that as the brightness level
increases, the combined IMF without residue shows
significant results, and we agree that the majority ofillumination effects may be represented by the last IMF
and residue. However, more research on a larger dataset
using different machine learning techniques is required
to improve the robustness of FER to illumination effects.

Fig. 10. Accuracy of FER based on reconstructed IMF at
50% brightness level.

Fig. 11. Accuracy of FER based on reconstructed IMF at
60% brightness level.

Acknowledgment
The research is backed by the grant "IIRG003(b)-19IISS"
from the University of Malaya, Malaysia.
References
1.

2.
3.

Fig. 12. Accuracy of FER based on reconstructed IMF at
70% brightness level.

4.

Ali H, Hariharan M, Yaacob S, and Adom A H 2015 Facial
emotion recognition using empirical mode decomposition
Expert Syst. Appl. 42 p 1261.
Pantic M 2009 Facial Expression Recognition Encycl.
Biometrics p 400.
Zankhana H S and Kaushik V 2015 Performance analysis
of canny edge detection for illumination invariant Facial
Expression recognition International Conference on
Industrial Instrumentation and Control (ICIC).
Fernandes S L and Bala G J 2013 Comparative study on
ICA and LPP based face recognition under varying
illuminations and facial expressions International©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
659

Powered by TCPDF (www.tcpdf.org)

Hasimah Ali, M Hariharan,, Wan Khairunizam Wan Ahmad, M Elshaikh

Conference on Signal Processing , Image Processing &Pattern Recognition
5. Palaniswamy S and Suchitra 2019 A robust pose &
illumination invariant emotion recognition from facial
images using deep learning for human-machine interface
2019 4th International Conference on Computational
Systems and Information Technology for Sustainable
Solution (CSITSS) p 1.
6. Ruiz-Garcia A, Palade V, Elshaw M and Almakky I 2018
Deep Learning for Illumination Invariant Facial
Expression Recognition International Joint Conference on
Neural Networks (IJCNN), p 1.
7. Nunes J C, Bouaoune Y, Delechelle E, Niang O, and Bunel
P 2003 Image analysis by bidimensional empirical mode
decomposition Image Vis. Comput. 21 p 1019
8. Bhuiyan S M A, Khan J F, and Adhami R R 2010 A
bidimensional empirical mode decomposition method forcolor image processing Electrical Engineering Tuskegee
IEEE SiPS p 272
9. Belhumeur P N, Hespanha J P and Kriegman D J 1997
Eigenfaces vs. Fisherfaces: Recognition using classspecial linear projection. IEEE Trans. PAMI 19, p 711.
10. Pumlumchiak T. and Vittayakorn S. 2017 Facial
expression recognition using local Gabor filters and PCA
plus LDA 2017 9th International Conference on
Information Technology and Electrical Engineering(ICITEE), pp. 1-6.
11. Qing, Jiang J, & Yang Z 2010 Empirical mode
decomposition-based facial pose estimation inside video
sequences Optical Engineering, 49(3).

Prof Ir Dr. Wan Khairunizam Wan Ahmad
He received his B. Eng. degree inElectrical & Electronic Eng. from
Yamaguchi University and Ph.D. in
Mechatronic Eng. from Kagawa
University, in 1999 and 2009
respectively. He is currently is a Prof. atFaculty of Electrical Engineering &Technology, University Malaysia Perlis and also serve as
Director of Centre for Sustainable Academic Leadership
Development (LEAD), UniMAP. He is member of Boardof Engineer and Institute of Engineer, Malaysia. His
research interest is in Human-Computer Interaction (HCI),
Intelligent Transportation System, Artificial Intelligenceand Robotics.
Assoc. Prof. Dr. Hariharan M.
He is currently working as an Associate
Professor in the Department ofElectronics
Engineering,
National
Institute of Technology, Uttarakhand,India. He is a Senior Member of IEEE,
USA, Member of IE, India and Memberof IET, UK. He received Chartered Engineer status fromEngineering Council, UK on February, 2016. His major
research interests include Biomedical Signal and Image
Processing, Speech Signal Processing, Signal Processing
for Communications, AI/DL/ML Algorithms in VLSI:
Dr. Mohamed Elshaikh
He is a Senior Lecturer under Faculty ofElectronic Engineering Technology(FTKEN), University Malaysia Perlis,
Malaysia. He received his Ph.D. in
Computer Engineering from University
Malaysia Perlis, Malaysia, M.Sc
Electrical and Electronics Engineeringfrom University Technology Petronas (UTP), Malaysia
and B.Sc Engineering Technology (Computer
Engineering), University of Gezira, Sudan. His researchmainly is on computer networking related.

Authors Introduction
Ir. Dr. Hasimah Ali
She is senior lecturer at Faculty ofElectricalEngineering
Technology,
UniMAP. She received her PhD in
Mechatronic Engineering from UniMAP(2015), MSc in Mechatronic Engineeringfrom IIUM (2008) and BEng in
Mechatronic Engineering from IIUM
(2004). Her research interests are signal and image
processing, facial expression, ground penetrating radar,robotics gripper and IoT.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
660

Supercontinuum Generation Pump By a Molybdenum Disulfide Based Soliton Mode-Locked
Fiber Laser
Aeriyn D. Ahmad

Photonics Engineering Laboratory, Department of Electrical Engineering, University of Malaya, 50603 Kuala Lumpur,
Malaysia.
Norrima Mokhtar, Hamzah Arof, Sulaiman Wadi Harun

Department of Electrical Engineering, Faculty of Engineering, University of Malaya, Malaysia

Ahmad Haziq Aiman Rosol

Department of Electronic Systems Engineering, Malaysia–Japan International Instituteof Technology, Universiti Teknologi Malaysia, Jalan Sultan Yahya Petra, 54100 Kuala Lumpur,
Malaysia.
Email: aeriyndwierni@gmail.com, norrimamokhtar@um.edu.my

Abstract
In this letter, a highly stable soliton mode-locked Erbium-doped fiber laser (EDFL) is passively obtained using amolybdenum disulfide (MoS2) thin film as a saturable absorber (SA). The MoS2 thin film obtained via electrochemical
deposition technique is integrated into an EDFL cavity to generate mode-locked pulses operating at 1.88 MHz with apulse duration of 3.03 ps. Supercontinuum (SC) light is generated using the proposed soliton mode pulses operating
at 1560.4 nm as they are injected into a 100 m long highly nonlinear photonic crystal fiber (HN-PCF) after it isamplified to the output power of 17.8 dBm. The SC light operates in a wavelength range starting from 1360 nm to
more than 1750 nm with the intensity above -35 dBm. The proposed supercontinuum laser can be seen as a promisinglight source for metrology and sensing applications.
Keywords: Fiber laser, supercontinuum, molybdenum disulfide, soliton.

1. Introduction

Many works have been performed to understand thephenomenon as well as to implement the intended
practical devices. To achieve a wider continuum andhigher output power, mode-locked laser system thatprovides picosecond pulse train with high peak power
must be employed as a pump source. Such systems
utilizing passive saturable absorber (SA) as a modelocker have been widely reported over the last decade.

Super-continuum (SC) light sources combine thebroadband attributes of lamps with the high brightness
and spatial coherence of lasers. They have attracted
extensive attention in recent years owing to theirnumerous applications in optical coherent tomography[1], optical communication [2], metrology [3] andsensing [4]. Broadband SC lasers are normally generated
through a mode locked laser, which is coupled into ahighly nonlinearity fiber (HNLF). The nonlinear
mechanisms leading to the generation of SC include
stimulated Raman scattering (SRS), four-wave mixing(FWM), self-phase modulation (SPM), cross-phasemodulation (XPM), and dispersive-wave generation(DWG) [5]. Since the Raman Effect is self-phasematched and shifts light to longer wavelength byemission of optical phonons, the SC spreads to longer
wavelengths very efficiently. The short wavelength edge
arises from four-wave mixing, and often the shortwavelength edge is limited by increasing group velocity
dispersion in the fiber.

2. Methodology
2.1. Fabrication of MoS2 SA
Within the framework of this experimental investigation,
we propose a methodology that leverages the electrodeposition technique. More precisely, the experiment isdesigned to employ the electrodeposition process forcathodically depositing a molybdenum sulphoselenide
film. This deposition procedure will be implemented ontin oxide-coated conducting glass substrates, as well as
silicon and/or metal substrates [6]. The electro-deposition©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
661

Aeriyn D. Ahmad, Norrima Mokhtar, Hamzah Arof, Sulaiman Wadi Harun, Ahmad Haziq Aiman Rosol.

technique has undergone thorough investigation,
particularly in the fabrication of metallic alloy thin films.
As compared to alternative methods, electrodeposition
stands out for its scalability and cost-effectiveness. Thismethod is notably advantageous due to its non-vacuum
nature and its operation at room temperature [7]. A cyclic
voltammetry analysis and the electrodeposition of thethin film are conducted using a three-electrodeconfiguration. The electrolysis cell comprises an ITOcoated conductive film designated as the working
electrode (WE), where the deposition of the molybdenum
disulfide (MoS2) thin film takes place [8]. In contrast, agraphite rod serves as the counter electrode (CE), and asaturated calomel electrode (SCE) with an Ag/AgCl
reference system functions as the reference electrode [9].

subsequently amplified and introduced into the HN-PCF
for generating supercontinuum light.

980nm LD

Isolator

EDF

WDM

2.4m

SA
MOS2

980/1550

SMF
100m
Coupler80%
20%

PCF 100m

2.2. Cavity Characterization

EDFA
oscilloscopeFig 1. The SC laser system consisting of three main
components namely MoS2 based mode-locked laser,
optical amplifier, and a HN-PCF spool.

In this area, we carry out an experimental
demonstration of a simplified optical fiber-based
supercontinuum source operating a simple mode-locked
fiber laser saturating Molybdenum Disulfide (MoS2) as asaturable absorber (SA).

2.3. Mode-Locking of MoS2
With anomalous net cavity dispersion, fundamental
mode locking in the soliton regime is achieved from the
EDFL at the pump power of 76.8 mW. By adjusting thepump power up to the maximum pump power of 123.7
mW, the mode-locking state is still preserved. The output
spectrum of the output pulses measured by an optical
spectrum analyzer (OSA) is depicted in Fig. 2. It centeredat 1560.4 nm with a 3 dB spectral width of 2.2 nm. The
Kelly sidebands are clearly seen to appear at both sidesof spectrum symmetrically, indicating the EDFL operates
in conventional soliton mode-locking state. Fig. 3
illustrates the typical pulse train, which has uniform
intensity with the interval of two pulses of 536 ns. The
pulse period corresponds to the repetition rate of 1.88
MHz, which agrees with the cavity length. The radiofrequency spectrum is measured to study the stability ofsoliton pulse as shown in Fig. 4. The signal-to-noise ratio(SNR) at the fundamental repetition rate of 1.88 MHz is
measured to be higher than 60 dB, which indicates thepulsed laser operates at high stability. Fig. 5 shows the
autocorrelation curve, which was obtained by anautocorrelator to measure pulse duration. The full widthat half maximum (FWHM) is 4.70 ps, which indicatesthat the pulse duration is 3.03 ps considering the pulse
shape is Sech2.

The suggested supercontinuum (SC) laser system is
illustrated in Fig. 1, comprising three main keys: the
MoS2-based mode-locked Erbium-doped fiber laser(EDFL), an optical amplifier, and a 100-meter-longhighly nonlinear photonic crystal fiber (HN-PCF). The
mode-locked EDFL utilized a 2.4-meter-long Erbiumdoped fiber (EDF) pumped with a 980 nm laser diodeserving as the gain medium. The SA was tested byencapsulating the newly developed MoS2 thin film,
obtained through the electrochemical deposition
technique. We achieved the growth of the MoS2 thin film
on a transparent conductive indium tin oxide (ITO) film
by functionalizing MoS2 nano-flakes in the presence ofmonochloroacetic acid. This thin film was integrated into
the EDFL cavity to function as a mode-locker.
To enhance the cavity nonlinearity and align thedispersion with the nonlinearity, a 70-meter-longstandard single-mode fiber (SMF) with a group velocity
dispersion (GVD) of -21.7 ps2/km was incorporated into
the cavity. An isolator was employed to maintainunidirectional laser oscillation within the cavity. An
80:20 coupler was utilized to extract 20% of the output
laser while retaining 80% of the light within the cavity tosustain the oscillation. The GVD values for both EDF andwavelength division multiplexer (WDM) wereapproximately 27.6 ps2/km and -48.5 ps2/km,
respectively. The total cavity length amounted to 83
meters, with an anomalous net cavity dispersion of -1.59
ps2. The output from the mode-locked laser was©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
662

Supercontinuum Generation Pumped By

the output power and pulse energy are obtained at 0.79
mW and 0.42 nJ, respectively. To generate SC photons,
the soliton laser is amplified by the optical amplifier andthen launched into the HN-PCF. The amplifier boosts thesignal up to the output power of 17.8 dBm so that it caninitiate a spectral broadening in the so-called ‘long pulse’
regime. The HN-PCF used in the experiment has a zero
dispersion at 1550 nm with á length of 100 m andnonlinearity coefficient of around 11W-1 km-1.
Fig 2. Output Spectrum taken from OSA

Fig. 6. The average output power and pulse energy at
various launched pump power.

Fig 3. Output measured from oscilloscope

3. Results and Discussion
3.1. Generation of Supercontinuum LightAs the soliton picosecond pulses created throughsaturable absorption of MoS2 are coupled into a HN-PCF,
broadband SC photons are generated via various
nonlinear mechanisms as shown in Fig. 7. We observe an
SC light spanning from 1360 nm up to more than 1750
nm with output power of more than -35 dBm. As theamplified 1560.4 nm soliton pulse laser is injected into
the PCF, the SC spreads to longer wavelengths very
efficiently due to Raman effect, which is self-phasematched to allow a shifting of light to longer wavelengthby emission of optical phonons. The short wavelengthedge arises from parametric four-wave mixing, which
breaks up the higher-order solitons to produce
frequencies at wavelengths shorter than the zerodispersion wavelength.
The SC bandwidth can be further expanded byincreasing the peak power of the injected pulses. This
could be realized by improving the average power as wellas compressing the pulse duration of the mode-locked
laser. We expect a smaller pulse duration with the
improvement of modulation depth and non-saturable lossof the MoS2 SA. The use of an amplifier with higher
saturated output power is also expected to further expand
the bandwidth as well as improving the flatness andpower of SC light.

Fig 4. Radio Frequency Output at 30MHz spanFig 5. Autocorrelator
The average output power and pulse energy
characteristics are shown in Fig. 6. As the pump powerincreases, both output power and pulse energy increase
almost linearly. At the maximum pumping of 123.7 mW,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
663

Aeriyn D. Ahmad, Norrima Mokhtar, Hamzah Arof, Sulaiman Wadi Harun, Ahmad Haziq Aiman Rosol.

3.

Woodward, J.T., et al., Supercontinuum sources for
metrology. Metrologia, 2009. 46(4): p. S277.
4. Eslami Jahromi, K., et al., A broadband mid-infrared trace
gas sensor using supercontinuum light source: applications
for real-time quality control for fruit storage. Sensors, 2019.
19(10): p. 2334.
5. Dudley, J.M. and G. Genty, Supercontinuum light. Physics
today, 2013. 66(7): p. 29-34.
6. Li, X. and H. Zhu, Two-dimensional MoS2: Properties,
preparation, and applications. Journal of Materiomics,
2015. 1(1): p. 33-44.
7. Thomas, N., et al., 2D MoS2: structure, mechanisms, andphotocatalytic
applications.
Materials
Today
Sustainability, 2021. 13.
8. Hossain, M.A., et al., Electrochemical deposition of bulk
MoS2 thin films for photovoltaic applications. Solar
Energy Materials and Solar Cells, 2018. 186: p. 165-174.
9. Lee Lian Hong, J., et al., Ultrafast Erbium-doped Fiber
Laser Using Electrodeposition Coated MoS2 Thin Film As
Saturable Absorber. Materials Research Express, 2023.
10. Bres, C.S., et al., Supercontinuum in integrated photonics:
generation, applications, challenges, and perspectives.
Nanophotonics, 2023. 12(7): p. 1199-1244.

Fig. 7. SC generation due to the injection of theamplified soliton pulses into 100 m long HN-PCF.
From the broader bandwidth SC output, the potential
application can be implemented are pulse generation,pulse amplification, pulse compression, metrology,
spectroscopy, imaging, telecom and on-chip integration[10].
4. Conclusion

Authors Introduction
In conclusion, our research endeavors have yielded a
successful
demonstrationofbroad-spectrum
supercontinuum (SC) light generation through theimplementation of a newly developed mode-locked
Erbium-doped fiber laser (EDFL) utilizing Molybdenum
Disulfide (MoS2) as a saturable absorber. The MoS2 thinfilm, synthesized via the electrochemical deposition
technique, was seamlessly integrated into the EDFL
cavity. This integration facilitated the generation ofmode-locked soliton pulses operating at 1560.4 nm,
characterized by a repetition rate of 1.88 MHz and a pulse
duration of 3.03 picoseconds. The resultant pulses were
subsequently amplified to an output power of 17.8 dBm
and introduced into a 100-meter-long highly nonlinear
photonic crystal fiber (HN-PCF) to induce SC light. The
generated supercontinuum light spans a wavelength
range from 1360 nm to beyond 1750 nm. This
achievement holds significance in the context ofadvancing optical sources for applications in diverse
fields such as telecommunications, metrology, andsensing.

Mrs. Aeriyn D. Ahmad
She received her Bachelor’s degree inElectronic Engineering in 2017 from theFaculty of Engineering Technology,
Universiti Teknikal Malaysia Melaka(UTeM) in Malaysia. She was again
sponsored by Jabatan Perkhidmatan
Awam (JPA) under Program Pelajar
Cemerlang (PPC) by Malaysia Government for her
Masters of Engineering Telecommunication. She iscurrently a PhD student in University of Malaya under
Universiti Malaya Scholarship Scheme (UMSS).
Dr. Norrima Mokhtar
She received the Bachelor ofEngineering (B. Eng) degree inElectrical Engineering from Universityof Malaya in 2000. She was appointed
as a lecturer to serve the Department ofElectrical Engineering, University of
Malaya immediately after graduatingwith her Master of Engineering. As part of her careerdevelopment, she received SLAB/SLAI scholarship toattain her Ph.D. in 2012. She is now serving Department ofElectrical Engineering, University of Malaya.
Prof Dr Hamzah ArofHe received the B.Sc degree in Digital
Signal Processing & Image Processing
from Michigan State University, USA
and Ph.D in same area at University ofWales, Cardiff UK. His area of expertise
are time series economics, image
processing, robotics, biomedical signals
and photonics.

References1.

2.

Hartl, I., et al., Ultrahigh-resolution optical coherence
tomography using continuum generation in an air–silica
microstructure optical fiber. Optics letters, 2001. 26(9): p.
608-610.
Mori, K., et al., Supercontinuum lightwave source
generating 50 GHz spaced optical ITU grid seamlessly
over S-, C-and L-bands. Electronics Letters, 2003. 39(6):p. 544-546.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
664

Powered by TCPDF (www.tcpdf.org)

Supercontinuum Generation Pumped By

Prof Dr Sulaiman Wadi Harun
He received the B.E degree in Electricaland Electronics System Engineeringfrom
Nagaoka
UniversityofTechnology, Japan in 1996, and M.Sc.
and Ph.D. degrees in Photonics
Technology from University of Malaya
in 2001 and 2004, respectively. He has
nearly 20 years of research experienceson the development of optical fiber devices including fiber
amplifiers, fiber lasers, and fiber optic sensors. He also
involved in exploiting new nanomaterials such asgraphene, carbon nanotubes, black phosphorous,
topological insulators for various fiber lasers and sensors
applications. He has received about 10 research grants ofvalue over RM4M from the Ministry of Education and theMinistry of Science, Technology, and Innovation. Prof.
Harun has published more than 700 articles in ISI journals
and his papers have been cited more than 7000 times withan h-index of 37, showing the impact on the community.
Prof. Harun also worked as an adjunct Professor at
Airlangga University, Indonesia and Ton Duc Thang
University, Vietnam. He received a prestigious award of
Malaysian Rising Star 2016 from the Ministry of Higher
Education for his contribution in international
collaboration. He is the Fellow of Malaysian Academic of
Science, and also the founder and honorary advisor for the
Optical Society of Malaysia.
Dr. Ahmad Haziq Aiman Rosol
He holds a Bachelor of Engineering(Hons.) in Electronic Communicationfrom Universiti Teknologi MARA(UiTM) Shah Alam (2015), a Master of
Science in Electrical Engineering fromUiTM (2017), and a Ph.D. in Photonics
from Universiti Malaya (2021). He iscurrently a Postdoctoral Fellow at the Malaysia-Japan
International Institute of Technology (MJIIT), Universiti
Teknologi Malaysia Kuala Lumpur, where he conducts
research on innovative photonic technologies. His
expertise lies in the areas of pulse fiber laser andnonlinear optics.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
665

Adsorption Behavior of Arsenic and Selenium using NiZn Hydroxy Double Salts with Acetate,
Chloride, Nitrate and Sulfate Anions
Kaoru Ohe*, Ryosuke Tabuchi, Tatsuya Oshima

Faculty of Engineering, University of Miyazaki, 1-1 Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan*Corresponding author

Abstract
Arsenic(As) and selenium(Se) has become an increasingly serious water contamination worldwide, so thedevelopment of adsorbents to improve the adsorption performance of As and Se oxyanions is desired. In this study,
NiZn adsorbents intercalated acetate, chloride, nitrate, and sulfate anions to interlayers of NiZn hydroxy double salts
were prepared and their adsorption behavior of As and Se oxyanions was investigated. The Langmuir isotherm model,
characteristic of monolayer adsorption, fit the experimental data well with R2 >0.94. The adsorption capacity(qmax)
of Se(IV) was the other NiZn-AcO ≈ NiZn-Cl > NiZn-NO3 > NiZn-SO4 and that of Se(VI) was the order of NiZn-Cl> NiZn-AcO > NiZn-NO3 > NiZn-SO4. The qmax of as(V) on NiZn-Cl was about twice as large as that on NiZn-AcO,
NiZn-NO3 and NiZn-SO4. NiZn-Cl was found to adsorb As(V) and Se(VI) oxyanions more efficient than NiZn-AcO.
Keywords: NiZn hydroxy double salts, arsenic, selenium, adsorption, Langmuir isotherm modelxZn2x(OH)2(OCOCH3)2x.nH2O has a structure in which
1/4 of Ni2+ in brucite-like layer of Ni(OH)2 is removed
2+

1. Introduction

from the brucite-like layer and two Zn are locatedoutside the layers, just above and below the Ni2+ vacancy[5], [6]. The layers have positive charges in excess, which
are balanced by intercalation of acetate anion. The acetateanions bind directly to the tetrahedral sites along withthree OH- groups from the layer to satisfy thecoordination number of Zn2+. The acetate anions areexchanged with guest ions in aqueous solution. Most ofresearch on HDS is in the field of catalysis [7] and, to myknowledge, there are few applications in the aqueous
environment. We reported that NiZn-AcO is an effective
adsorbent for arsenic removal than MgAl Layered doublehydroxide [8]. Kozai et al. showed that NiZn hydroxyl
double salts is more effective than LDH for selenium
removal using selective coefficients [9], but adsorption
isotherms and adsorbent evaluation after adsorption
selenium were not studied.

Arsenic and selenium are known to be harmful tohuman body even at low concentration. The
environmental standard values for arsenic [1] andselenium [2] in drinking water are set by the WHO
guidelines as less than 0.01 mg/dm3. Arsenic in nature has
oxidation states of -3, 0, +3, and +5, and exists as
inorganic and organic arsenic. Inorganic arsenic is known
to more toxic than organic arsenic. Inorganic arsenic innatural water forms mainly oxyanions, arsenite (AsO33-,
As(III)) and arsenate (AsO43-, As(V)). The aciddissociation constants of As(III) are pKa1=9.23,
pKa2=12.13, and pKa3=13.4, and there are four chemical
forms, H3AsO3, H2AsO3-, HAsO32-, and AsO33-[3]. The
acid dissociation constants of As(V) are pKa1=2.24,
pKa2=6.96, and pKa3=11.5, and the chemical species are
H3AsO4, H2AsO4-, HAsO42-, and AsO43- [3]. Selenium
solubilized and released into waste water exists in +4 and+6 oxidation states. Chemical species of selenium withthese oxidation states are oxyanions selenite (SeO32-,
Se(IV)) and selenate (SeO42-, Se(VI)) with high solubility
and mobility in both aquatic and soil environment.. The
acid dissociation constants of Se(IV) are pKa1=2.62,
pKa2=8.23, and there are three chemical forms, H2SeO3,
HSeO3-, and SeO32- [4]. The acid dissociation constantsof Se(VI) are pKa1=1.7 and the chemical species are
HSeO4-, and SeO42- [4].
Hydroxyl double salts HDSs, one of inorganic anionexchanger with a general formula [MAII(1IIzx)MB 2x(OH)2(Y )]2x/z.nH2O. Specifically, the NiZn
double basic salt with acetate anion (NiZn-AcO) Ni1-

In this study, NiZn adsorbents with chloride, nitrateand sulfate anions in the interlayer of NiZn were preparedto improve for arsenic and selenium removal
performance and their adsorption behavior for arsenic
and selenium was evaluated.
2. Experimental©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
666

Kaoru Ohe, Ryosuke Tabuchi, Tatsuya Oshima

2.1. Preparation of NiZn adsorbents
NiZn-AcO, NiZn hydroxide double salt intercalating
acetate anion (AcO) was first prepared by modifying thesynthetic method reported by Rojas et al. [5]. NiZn-Cl
adsorbent was prepared by an ion-exchange method. One
gram of NiZn-AcO was added to 50 cm3 of 1 M NaClsolution and stirred at 333 K for 24h. The supernatant
was then wasted and fresh 1 M NaCl solution was added
and stirred. This procedure was repeated three times.
NiZn-NO3 and NiZn-SO4 were prepared in the sameprocedure as NiZn-Cl using 1 M Na NO3 and 1 M Na2SO4,
respectively. The resulting adsorbents were washedseveral times with deionized water, dried in a vacuum
drying oven and characterized by XRD, FT-IR, and CHN
elemental analysis.
Fig. 1 XRD patterns of NiZn-AcO, NiZn-Cl,
NIZn-NO3 and NiZn-SO4..

2.2. Adsorption test of arsenic and selenium

NiZn-AcO was determined to be 8.40 Å by subtracting
the thickness of the basic layer, Ni(OH)2, of 4.60 Å.
Similarly, peaks derived from layered compounds wereobserved in the diffraction patterns of NiZn-Cl, NiZnNO3, and NiZn-SO4, with d001 values of 8.09 Å, 9.14 Å,
and 9.14 Å, respectively, as shown in Fig. 1 [10].
Chemical formula of NiZn adsorbents was determinedusing atomic absorption analysis (Ni/Zn mole ratio),
CNH elemental analysis (amount of acetate) andTG/DTA (amount of adsorption water) [11]. Table 1
shows the chemical formulas of the NiZn adsorbent, inwhich acetate anion in NiZn-AcO is ion-exchanged with
Cl-, NO3- and SO42-. The anion exchange capacity of
NiZn-AcO is 2.45 meq/g. The amount of Cl-, NO3- andSO42-exchanged were in the order of Cl- (2.22 meq/g) >SO42- (2.09 meq/g) > NO3- (1.57 meq/g).

All adsorption test was carried out by batch method at
303 K. 1.0 × 10-4 M metal solution was adjusted to a given
pH using HCl solution or an aqueous NaOH solution in
the pH-dependent adsorption test. In the adsorption
isotherm experiments, metal concentrations wereprepared at 2.0 × 10-4 - 4.0 × 10-3 M and adjusted to theprescribed pH with the aqueous NaOH solution. Ten mg
of NiZn adsorbent and 15 cm3 of metal solution wasadded to a sample tube and shaken in at 303 K, 120 rpm
for 24 h. pH of the filtrate was measured. The metal
concentrations of the solutions before and after
adsorption equilibrium were measured using atomic
absorption spectrophotometer and ICP emission
spectrometer. The amount of adsorption(q) and theadsorption percentage (A%) were calculated according to
the following equations, respectively:q = (Cint – Ceq) v/w(1)
A = (Cint – Ceq) / Cint(2)
where Cint and Ceq are initial and equilibrium
concentration, respectively [mM], v is the volume ofsolution , w is the weight of adsorbent [g], q is the amountof arsenic or selenium adsorbed , A is the adsorption
percentage [%]Langmuir equation and its variant equation are given
in Eq.(3) and (4), respectively.
q = KLqmax Ceq / (1 + KLCeq)(3)
Ceq / q = Ceq / qmax + 1 / KL qmax
(4)
where qmax is the maximum adsorption capacity[mmol/g] and KL is the adsorption equilibrium constant[dm3/mmol]

Table 1. Chemical formula of NiZn adsorbents
NiZn adsorbents

Chemical formula

AcO

Ni0.67Zn0.34(OH)2(CH3COO)0.34 0.65H2O

Cl

Ni0.67Zn0.34(OH)2(CH3COO)0.03(Cl)0.31 nH2O

NO3

Ni0.67Zn0.34(OH)2(CH3COO)0.12 (NO3)0.22 nH2O

SO4

Ni0.67Zn0.34(OH)2(CH3COO)0.05 (SO4)0.15 nH2O

3.2. Adsorption isotherms of arsenic and selenium
To evaluate the adsorption performance of NiZn
adsorbents for As and Se oxyanions, adsorption
isotherms of e and As oxyanions at 303 K were conducted.
Langmuir isotherm model, characteristic of monolayer
adsorption, since the amount increases with increasing
concentration and approaches a constant value at highconcentrations. The Experimental data for the adsorptionof Se and As oxyanions using NiZn adsorbents fit
Langmuir plot well with R2 >0.94 (Fig. 2). The maximum
adsorption capacity (qmax) and adsorption equilibrium
constant (KL) for Se(IV), Se(VI) and As(V) were

3. Results and Discussion
3.1. Characterization of NiZn adsorbents
NiZn adsorbents prepared were identified by XRD and
FT-IR analysis. XRD diffraction pattern of NiZn-AcO isshown in Fig. 1. The diffraction peaks of 001, 002 and
003 are at 2θ = 6.8, 13.6, 20.6, respectively, in agreementwith the literature values [5]. The interlayer distance of©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
667

Adsorption behavior of arsenic

calculated from the Langmuir equation (Eq. (3) and (4))
and are shown in Table 2.

exchange percentage of As(V) was estimated to be 37.5%
and 85.9%. These results indicate that NiZn-Cl canadsorb Se(VI) and As(V) more effectively than NiZnAcO.
3.3. Effect of contact time of Se(VI) adsorption
The effect of contact time on Se(VI) adsorption by
NiZn-AcO and NiZn-Cl is shown in Fig. 3. The time toreach equilibrium for Se(VI) adsorptions was 5 min
which was extremely rapid. During the first 5 min 95.2%
of Se(VI) was adsorbed on NiZn-AcO, whereas NiZn-Cl
adsorbed 92.7% of Se(VI). These fast adsorption process
has been considered ion exchange reactions involving,which is similar to ion exchange in LDHs [12].

Fig. 2 Langmuir plots of Se(VI) on NiZn-AcO, Cl, -NO3 and -SO4....
Table 2. Parameters of the Langmuir adsorption
isotherms for Se(IV), Se(VI) and As(V)at 303 K.

Se(IV)

Se(VI)

As(V)

KL

pHeq

q max
[mmol/g][dm3/mmol]

NiZn-AcO

8.54±0.33

1.48

7.10

0.94

NiZn-Cl

7.88±0.39

1.44

9.98

0.994

NiZn-NO3

7.67±0.59

1.54

7.47

0.971

NiZn-SO4

7.35±0.39

0.909

4.09

0.965

NiZn-AcO

6.85±0.15

0.979

14.2

0.989

NiZn-Cl6.72±0.54

1.19

4.31

0.988

NiZn-NO3

6.60±0.42

0.983

12.8

0.997

NiZn-SO4

6.77±0.21

0.718

0.531

0.99

NiZn-AcO

10.3±0.32

0.446

287

0.998

NiZn-Cl9.67±0.43

0.947

14.3

0.997

NiZn-NO3

10.0±0.36

0.524

77.6

0.999

NiZn-SO4

9.59±0.49

0.564

11.1

0.988

Adsorbates Adsorbents

R2

Fig. 3 Time course of Se(VI) adsorption on NiZnAcO and NiZn-Cl...
3.4. XRD results of NiZn adsorbent after Se(IV)and Se(VI) adsorption
The basic brookite layer thickness does not changemuch during the process of anion exchange, but theinterlayer distance varies with the placement and size ofthe intercalated anions. When inorganic anions are
replaced by organic anions in the LDH interlayer, theLDH layer space increases significantly [12]. In thepresent study, the (003) diffraction peaks observed in

The maximum adsorption capacity (qmax) of Se(IV),
Se(VI), and As(V) by NiZn-Cl, -NiZnNO3, and SO4 wasalmost the same or increased compared to that of NiZnAcO, except for NiZn-SO4. Since the dissolved speciesof Se(VI) or As(V) have a significant effect on qmax, thesaturated adsorbed amount of Se(VI) or As(V) wasconverted to an equivalent unit by considering the mole
fractions of the main dissolved species at each pH, whichwere obtained from the acid dissociation constants. As
shown in Table 2, Se(VI) species are SeO42- from pH, theqmax of NiZn-AcO and NiZn-Cl are calculated to be 1.96
meq/g and 2.38 meq/g, respectively. Acetate in NiZnAcO and NiZn-Cl The Se(VI) exchange percentage wasestimated to be 80.0% and 107% by comparing theamount of acetate anion and Cl- (meq/g) in NiZn-AcO
and NiZn-Cl, respectively, as shown in section 3.1.
When Se(IV) was examined similarly, the exchange rateof Se(IV) by NiZn-Cl (82.7%) was slightly lower thanthat of NiZn-AcO (96.7%) From the pH given in Table 2,
the qmax of NiZn-AcO and NiZn-Cl were calculated tobe 0.918 meq/g and 1.91 meq/g, respectively, considering
the mole fraction of As(V) chemical species, and theFig. 4 XRD patterns of NiZn-AcO adsorbed (a)Se(IV) and (b) Se(VI)..and NiZn-Cl adsorbed (c)Se(IV) and (d) Se(VI)..

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
668

Powered by TCPDF (www.tcpdf.org)

Kaoru Ohe, Ryosuke Tabuchi, Tatsuya Oshima

NiZn-AcO and NiZn-Cl have the d003 values of 13.0 and
8.09 Å, respectively (Fig. 1), and these peaks shifted after
the adsorption of Se(IV) or Se(VI) (Fig. 4). and NiZn-Cl
host layer thickness is 4.6 Å, the NiZn-AcO interlayer
distance decreased significantly from 8.4 Å to 4.63 Å(Se(IV)), 4.52 Å (Se(VI)), and the NiZn-Cl interlayer
distance increased from 4.49 to 4.53 Å (Se(IV), Se(VI)).
Se(IV), Se(VI)). The interlayer distances after selenium
adsorption were almost equal, suggesting that theinterlayer acetate anions and Cl- were replaced by Se(IV)or Se(VI).

stability, Microporous and Mesoprous Mater., 112,
2008, pp. 262-272.
7.

8.

9.

4. Conclusions

10.

NiZn adsorbents NiZn-Cl, NiZn-NO3, and NiZn-SO4
were prepared by ion exchange method using NiZn-AcO
obtained by hydrothermal synthesis. Experimental
investigating the adsorption of arsenic and selenium
oxyanions from aqueous solutions using the NiZn
adsorbents indicated that the arsenic and selenium
oxyanions could be adsorbed on NiZn-AcO, NiZn-Cl,
NiZn-NO3, and NiZn-SO4. Se(IV), Se(VI) and As(V)were found to be adsorbed on NiZn adsorbent by
Langmuir isotherm model. The adsorption isotherm for
As(V) shows that NiZn-Cl has twice higher adsorption
capacity than NiZn-AcO. The adsorption of Se(IV) by
NiZn-NO3 was slightly increased compared to NiZn-AcO.
The exchange of acetate anion in the NiZn interlayer withchloride ions improved the adsorption performance ofAs(V) and Se(VI).

11.

12.

Authors Introduction
Dr. Kaoru Ohe
She received her Ph.D. degrees from
University of Miyazaki, Japan, in
2014. Currently she is an Associate
Professor of Center for Science andEngineering Education, Faculty ofEngineering. Her
researches are
separation engineering especiallyadsorption hazardous heavy metals
and oxyanions.

Acknowledgements
The authors are gratefully acknowledgement Prof. S.
Shimazu and Prof. T. Hara of Chiba University fortechnical advice with the preparation of NiZn-AcO
adsorbent.

Mr. Ryosuke Tabuchi
He received his Master’s degree inEngineering in 2019 from Faculty ofEngineering, University of Miyazaki
in Japan.

References
1.
2.

3.

4.

5.

6.

T. Hara, M. Ishikawa, J. Sawada, N. Ichikuni and S.
Shimazu, Creation of Highly Stable Monomeric Pd(II)Species in an Anion-Exchangeable Hydroxyl Double Salt
Interlayer: Application to Aerobic Alcohol Oxidation
Under an Air Atmosphere, Green Chem., 11, 2009,
pp.2034-2040.
K. Ohe, R. Tabuchi, T. Oshima, T. Hara and S. Shimazu,
Adsorptive removal of arsenic (III) and arsenic (V) from
aqueous solution using nickel-zinc hydroxyle double salts,
Kagaku gogaku Rombunshu, 45(2), 2019, pp.80-85.
N. Kozai, T. Ohnuki and S. Komarneni, Selenium
Oxyanions: Highly Selective Uptake by a Novel Anion
Exchanger, J. Mater. Res., 17, 2002, pp.2993-2996.
R. Rojas, C. Barriga, M. A. Ulibarri, P. Maletb and V.
Rive, Layered Ni(II)-Zn(II) hydroxyacetates. Anion
exchange and thermal decomposition of the hydroxysalts
obtained, J. Mater. Chem, 12, 2002, pp.1071-1078.
N. Kozai, H. Mitamura, H. Fukuyama, F. Esaka and S.
Komarneni, Synthesis and characterization of nickelcopper hydroxide acetate, NiCu(OH)3.1(OCOCH3)0.9･0.9
H2O, Microporous and Mesoprous Mater., 89, 2006,
pp.123-131.
T. You, G. F. Bance and H. Zhao, Selenium adsorption
on Mg-Al and Zn-Al layered double hydroxides, App.
Clay Sci.20, 2001, pp.13-25.

B. K. Mandal and K. T. Suzuki, Arsenic round the world :A review, Talanta, 58(1) , 2002, pp.201-235.
S. O. Okonji, G. Achari and D. Pernitsky, Environmetanl
inpacts of selenium contamination: A review on currentissues and remediation strategies in an aqueous system,
Water, 13(11), 2021, 1473
D. Mohan and C. U. Pittman Jr., Arsenic removal fromwater/wastewater using adsorbents-A critical review, J.
Hazardous Metals, 142(1-2), 2007, pp. 1-53.
S. Santos, G. Ungureanu, R. Boaventura and C. Botelho,
Selenium contaminated waters: Am overview ofanalytical methods, treatment options and recent advancesin sorption methods, Sci. Total Environ., 512-522, 2015,
pp.246-260.

Prof. Dr. Tatsuya Oshima
He received his Ph.D. degrees from
Kyushu University, Japan, in 2001.
Currently he is an Professor of the
Department of Applied Chemistry,
FacultyofEngineering.
His
researches are separation engineering(especially solvent extraction for
precious metal ions) and formulation
techniques for poorly water-soluble
drugs and nutrients.

R. Rojas, M. A. Ulibarri, C. Barriga and V. Rives,
Chromate intercalation in Ni-Zn layered hydroxidesalts, Appl. Clay Sci., 49(3), 2010, pp.176-181.
R. Rojas, M. A. Ulibarri, C. Barriga and V. Rives,
Intercalation of metal-edta complexes in Ni-Zn
layered hydroxysalts and study of their thermal©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
669

Parallel acoustic analysis based on the domain decomposition method with higher-orderelement
Amane Takei*, Makoto Sakamoto

Faculty of Engineering, University of Miyazaki, Japan
Akihiro Kudo

National Institute of Technology, Tomakomai college, Japan*Corresponding author E-mail: takei@cc.miyazaki-u.ac.jp

Abstract
Large-scale analyses, using numerical models with over 10 trillion elements, are required for the analysis of a large
space such as a concert hall with higher-frequency bands. Large spaces are often limited to low-frequency analysis.
In this study, the number of elements is reduced by wave acoustic analysis using higher-order elements. Based on the
results using higher-order elements, it is shown that it is possible to analyze a real environment model such as a live
music club and a concert hall.
Keywords: large-scale simulation, acoustical sound field, higher-order element

1. Introduction

2.1. Higher-order elements

Estimation of the sound field is important for improving
the quality of acoustic spaces such as concert halls andlive music clubs [1]. Scale model experiments and thecomputer simulations are used for estimation of the soundfield. Scale model experiments are used in many fields[2], however, creating models requires a lot of time andis expensive. On the other hand, computer simulation
creates a model and sound in a virtual space. Therefore,
in the computer simulation, it is easy to change theconditions of the analysis as well as to change materials
and shapes. However, a large-scale analysis, using a numerical model with over 10 trillion elements, is required
for the analysis of a large space with high frequency. n alarge-scale finite element steady-state acoustic analysis,
the iterative domain decom-position method [3] isproposed and applied as a parallelization technique. It isshown that the large-scale analysis becomes possible by
the iterative domain decomposition method [4]. In thisstudy, higher-order elements are introduced into aparallel finite ele-ment steady-state acoustic analysis
method and greatly reduce the number of elements.
Higher-order elements are not actively used because thematrix expands. In particular, there are some examples ofhigher-order elements, e.g., higher than the 3rd order element [5], however, there are few examples of acoustic
analysis. As far as we know, there is no example showing
the superiority of reducing the number of necessary
elements by applying higher-order elements, especiallyin large-scale acoustic calculations using the domain
decomposition method.

To reduce the number of elements, 2nd and 3rd orderelements are introduced. With higher-order elements, thenumber of nodes increases because the nodes are placedon the sides and faces of the element. Fig. 1 shows thenodal arrangement of the tetrahedras with 1st, 2nd, and3rd order elements. Table 1 shows the shape function ofeach element [6].

(a) 1st order elemenet(b) 2nd order(c) 3rd orderFig. 1. Nodal arrangement of each element
Table 1. Shape function
1st orderelm.
2nd orderelm.

3rd orderelm.

2. Finite element

𝑁𝑖 = 𝐿𝑖
𝑁𝑖 = 𝐿𝑖 (2𝐿𝑖 − 1)
𝑁𝑖 = 4𝐿𝑗 𝐿𝑘

i=0,1,…,3
i=0, 1, …,
3
i=4,…,9j, k=0,…,3

1
i=0,…,3
𝑁𝑖 = (3𝐿𝑖 − 1)(3𝐿𝑖 − 2)𝐿𝑖2
9i=4,…,15
𝑁𝑖 = (3𝐿𝑗 − 1)𝐿𝑗 𝐿𝑘
j, k=0,…,32
𝑁𝑖 = 27𝐿𝑗 𝐿𝑘 𝐿𝑙
i=16,…,19
𝐿 : Volume coordinate variable
N : Shape function
i : Number of nodes©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
670

Amane Takei, Akihiro Kudo, Makoto Sakamoto

2.2. Helmholtz equation𝑛∭ 𝐿𝑘1 𝐿𝑙2 𝐿𝑚
3 𝐿4 𝑑𝑥𝑑𝑦𝑑𝑧

In the 3-dimensional sound field, the wave equation for
velocity potential is expressed by the following equation:
𝜕2𝜙 𝜕2𝜙 𝜕2𝜙 1 𝜕2𝜙
+
+
−
=𝑞
(1)
𝜕𝑥 2 𝜕𝑦 2 𝜕𝑧 2 𝑐 2 𝜕𝑡 2
where 𝜙 is the velocity potential, 𝑐 is the speed of sound,and 𝑞 is the distribution function.
To consider the steady-state, the velocity potential isexpressed by Eq. (2). Using Eq. (1) and Eq. (2), theHelmholtz equation is obtained:
𝜙 = 𝛷𝑒 −𝑗𝜔𝑡(2)𝜕 2 𝛷 𝜕 2 𝛷 𝜕 2 𝛷 𝜔2
+
+
+
𝛷=𝑞
(3)
𝜕𝑥 2 𝜕𝑦 2 𝜕𝑧 2 𝑐 2
where 𝜔 is the angular frequency.
The velocity potential of Eq. (3) and calculate thesound pressure using the following equation are obtained:
𝑝 = 𝑗𝜔𝜌𝛷
(4)
where 𝑗 is the imaginary number, and 𝜌 is the medium
density.

𝛺𝑒

= 6𝑉𝑒

𝑘! 𝑙! 𝑚! 𝑛!
(𝑘 + 𝑙 + 𝑚 + 𝑛 + 3)!

(7)𝑘! 𝑙! 𝑚!
(𝑘 + 𝑙 + 𝑚 + 2)!

(8)∬ 𝐿𝑘1 𝐿𝑙2 𝐿𝑚
3 𝑑𝑥𝑑𝑦 = 2𝐴𝑒Γ𝑒

On the other hand, the calculation of ∇{𝑁}∇{𝑁}𝑇 in
Eq. (6.2) shown in the following equation.
𝜕𝑓
1
𝜕𝑓
𝜕𝑓
𝜕𝑓
𝜕𝑓
=
(𝑏1
+ 𝑏2
+ 𝑏3
+ 𝑏4
)
𝜕𝑥 6𝑉𝑒
𝜕𝐿1
𝜕𝐿2
𝜕𝐿3
𝜕𝐿4
𝜕𝑓
1
𝜕𝑓
𝜕𝑓
𝜕𝑓
𝜕𝑓
=
(𝑐
+ 𝑐2
+ 𝑐3
+ 𝑐4
)
𝜕𝑦 6𝑉𝑒 1 𝜕𝐿1
𝜕𝐿2
𝜕𝐿3
𝜕𝐿4
𝜕𝑓
1
𝜕𝑓
𝜕𝑓
𝜕𝑓
𝜕𝑓
=
(𝑑1
+ 𝑑2
+ 𝑑3
+ 𝑑4
)
𝜕𝑧 6𝑉𝑒
𝜕𝐿1
𝜕𝐿2
𝜕𝐿3
𝜕𝐿4

(9)

We convert the finite element equations of (5) to amatrix form as follows:
𝐾𝑢 = 𝑓.

(10)

2.3. Finite element formulation
2.5. Hierarchical domain decomposition method
To derive a weak form, the Galerkin method is applied
to Eq. (3). By applying the finite element approximationand discretization, the following equation is obtained:−𝑘 2 [𝑀]{𝛷} + 𝑗𝜔𝜌[𝐶]{𝛷} + [𝐾]{𝛷} = {𝑞}The original analysis domain is first divided into parts,
which are further decomposed into smaller domains
called subdomains. This is called the hierarchical domain
decomposition method (HDDM) [8], [9].

(5)3. Numerical experiment

where [・] is a matrix, {・} is a vector. In Eq. (5), thematrices [M], [C] and [K] can be calculated using Eq.
(6.1,2,3):[𝑀]𝑒 = ∭ {𝑁}{𝑁}𝑇 𝑑𝛺𝑒

3.1. Verification by benchmark problem
Fig.2 shows that test model for simulation. The model is
AHLV100 that is known as a reference model in
code_Aster. This is also described in the
ADVENTURE_sound manual as a sample. This
simulation was done to confirm the use of transient
analysis in ADVENTURE_Sound. To evaluate theaccuracy of the acoustic analysis code, an acoustic
benchmark problem is used. The analysis uses the test
model AHLV100 of Code_Aster [10], which is known asa representative benchmark problem among acoustic
problems (Fig. 4).
This model is an acoustic tube that has a length of 1[m], a height of 0.1 [m] and a width of 0.2 [m]. It has avibration boundary at the left end and a sound absorption
boundary at the right end. The other faces are given rigid
boundaries. The specific acoustic impedance Zn=445.9[kg/m3 ∙ s] is given as a sound absorption boundary
condition.
The accuracy is calculated from the average error offour points on the sound absorption boundary. The
formula for calculating the theoretical solution is asfollows:𝑝(𝑥, 𝑦, 𝑧) = 𝜌𝑐𝑉𝑛 𝑒𝑥𝑝(−𝑖𝑘𝑥)(11)where 𝜌 is the medium, c is the speed of sound, and 𝑉𝑛 isthe particle velocity.

(6.1)

𝛺𝑒

[𝐾]𝑒 = ∭ ∇{𝑁}∇{𝑁}𝑇 𝑑𝛺𝑒

(6.2)

Ω𝑒

1
∬ {𝑁}{𝑁}𝑇 𝑑𝛤𝑒
(6.3)
𝑍𝑛 𝛤𝑒
where N is the shape function, [𝑀]𝑒 and [𝐾]𝑒 are thevolume integrals, and [𝐶]𝑒 is a surface integral to thesound-absorbing boundary surface. 𝑍𝑛 is a specificacoustic impedance.
[𝐶]𝑒 = −2.4. Calculaion of element matrix
Let us consider the calculation {𝑁}{𝑁}𝑇 in Eqs. (6.1) and(6.3). This calculation uses an integration formula of the
3 or 2-dimensional finite element method. These
integration formulas are shown in the following
equations [7].

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
671

Parallel acoustic analysis based

3.3. Analyses using real environment models
To confirm the effectiveness of higher-order elements,
the sound field of the real environment model is analyzed.
The model used for analysis is concert hall model that isshown in Fig. 5.

Rigid boundary
Vibration boundary
Absorption boundary

Fig. 2. Acoustic benchmark problem AHLV100
3.2. Performance evaluation

The performance of higher-order elements isevaluated based on the number of elements,
accuracy rate, and memory usage. The performance
evaluation conditions are: 4.0 [kHz] for frequency,
air for medium, and 343 [m/s] for sound velocity.
The analysis uses a PC cluster composed of 5 PCs(40 cores) equipped with a multicore CPU (Intel
Core i7-9700K, 3.6GHz 8core, 32GB of memory).
Numerical results for the error rate, plottedagainst the number of elements, are shown in Fig. 3.
Results for the memory, plotted against the numberof elements, are shown in Fig. 4.

(a) Concert hall 3D CAD model(b) Concert hall model dimensions
Fig. 5. Concert hall model
The concert hall model is constructed based on real
spaces [11].
The concert hall model is 11.5 [m] wide, 7 [m] high,and 23.5 [m] deep. The sound source is set as a pair of
speakers at either end of the stage. Oakwood flooring, awooden stage, and a glass wool wall at the back of thehall are used as sound-absorbing boundary conditions.
The sound field is analyzed by applying a 400 [Hz] soundto these models.
The results of these analyses are shown in Table 2.
The visualization results are shown in Fig. 6, and theconvergence histories of the iterative method (COCG) are
shown in Fig. 7. The concert hall analysis using the 1 st
and the 2nd order elements are excluded from theevaluation because the number of elements exceeds 100
million. The data I/O library currently applied is 32 bit.
This library cannot use data sizes that exceed 100 million
elements. A 64-bit I/O library is currently underdevelopment.

Fig. 3. Error rate plotted against the number of elements

Table 2. Numerical results of the concert hall modelFig. 4. Memory usage plotted against the number of
elements

Element
type

Number ofelements

1st order

823,285,162

2nd order

102,910,645

3rd order

878,624

Number ofnodes

ElapsedTime[sec]

Memoryrequirement[MB/core]

4,051,235

53.97

391.97

The accuracy increases dramatically when a higherorder element is applied. In particular, the accuracychanges more rapidly as the order increases.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
672

Amane Takei, Akihiro Kudo, Makoto Sakamoto

Acknowledgements
This research was supported by Grant-in-Aid for
Scientific Research 22K19779.
References
1. T. Okuzono, M. Shadi and K. Sakagami, “Potential of Room
Acoustic Solver with Plane-Wave Enriched Finite Element
Method”, Appled. Science, Vol.10, No.6, 2020.
2. K. Suzuki, Y. Yamada, S. Koyanagi, T. Hidaka, “Basic study
on improvement of precision of measurement of room
acoustic characteristics using scale model and 3D soundfield auralization”, Acoustical Society of Japan (ASJ)
Academic journal, Vol. 74, No. 5,
3. K. Kowalczyk, M. Walstijn, “Room Acoustics Simulation
Using 3-D Compact Explicit FDTD Schemes”, IEEE
Transactions on Audio, Speech, and Language Processing,Vol.19, No.1, pp.34-46, 2011.
4. Y. Yasuda, T. Oshima, T. Sakuma, A. Gunawan, T.
Masumoto, “Fast multipole boundary element method for
low-frequency acoustic problems based on a variety of
formulations”, Journal of Computational Acoustics, Vol.18,
No.4, pp.363-395, 2010.
5. K. Okuzono, T. Otsuru, R. Tomiku, N. Okamoto,“Fundamental accuracy of time domain finite element
method for sound-field analysis of rooms”, Applied
Acoustics, Vol. 71, pp. 940-946, 2010.
6. K. Ueno Ed., Science of concert hall –Harmony of shapeand sound- (in Japanese, CORONA PUBLISHING Co., Ltd.
2012).
7. M. Koshiba, Fundamentals of the finite element method forlight and waves (in Japanese, MORIKITA PUBLISHING
Co., Ltd, 1990).
8. J. Mandel, “Balancing domain decomposition”,
Communications on Numerical Methods in Engineering,
Vol.9, pp.233-241, 1993.
9. M. Ogino, A. Takei, S. Sugimoto, S. Yoshimura, “Anumerical study of iterative substructuring method for finite
element analysis of high frequency electromag-netic fields”,
Computers and Mathematics with Applications, Vol. 72,
Issue 8, pp. 2020-2027, 2016
10. F. Stifkens, G. Rousseau, “Code_Aster Manuel deValidation”, No. V8. 22. 100, 1998.
11. STAR
PINE’SCAFÉ,Homepage:
http://mandala.gr.jp/SPC/, 2019.

(a) Visualization results (X-Y)(b) Visualization results (X-Z)
Fig. 6. Concert hall modelFig. 7. Converegence history of the COCG
4. Conclusion
This paper described a large-scale acoustic analysis
method using a domain decomposition method and theintroduction of higher-order elements. The performanceof proposed method was evaluated with higher-orderelements using of AHLV100. The error rate, number of
elements, and memory usage were the considered as
evaluation criterions. It was shown that the calculation
efficiency improved in higher-order elements. In
particular, in the 3rd order element, the calculation
efficiency was vastly improved. Furthermore, theaccuracy of higher-order elements was verified. The 2ndorder element and the 3rd order element were comparedin terms of the number of elements and calculation time.
Additionally, real environment model was analyzed,
namely, a small concert hall model. It was shown that thereal environment models could be successfully analyzed
by using higher-order elements.

Authors Introduction
Prof. Amane Takei
He is working as Associate Professor for
Department of Electrical and systems
Engineering, University of Miyazaki,
Japan. His research interest includes highperformance computing for computational
electromagnetism, iterative methods forthe solution of sparse linear systems,
domain decomposition methods for largescale problems. Prof. Takei is a member of IEEE, an expert
advisor of IEICE, a delegate of the Kyushu branch of IEEJ,
a member of JSST。.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
673

Powered by TCPDF (www.tcpdf.org)

Parallel acoustic analysis based

Prof. Makoto Sakamoto
He is presently a professor in the Facultyof Engineering, University of Miyazaki.
His first interests lay in hydrodynamics
and time series analysis, especially the
directional wave spectrum. He is atheoretical computer scientist, and his
current main research interests areautomata theory, languages and computation. He is alsointerested in digital geometry, digital image processing,computer vision, computer graphics, etc.
Prof. Akihiro Kudo
He received Ph.D. degree from Nagaoka
University of Technology He is a professor
in the Department of engineering forinnovation , National Institute ofTechnology, Tomakomai college. He is amember of Acoustical society of Japan,
Information and Communication Engineers(IEICE).

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
674

Sound field evaluation on the acoustical experiment using non-steady state analysis
Akihiro Kudo*

National Institute of Technology, Tomakomai college, Japan
Makoto Sakamoto, Amane Takei

Faculty of Engineering, University of Miyazaki, Japan*Corresponding author

Abstract
Identifying acoustic effects of sound waves emitted by a sound source on the listener is important for understanding
the localization of sound as perceived by the listener. In this presentation, we will analyze the effect of the presenceof a chair on the psychoacoustic effect using a practical model simulating a psychoacoustic experiment consisting ofa loudspeaker, a listener, and a chair. Non-steady state analysis is required to analyze changes in acoustic effects overtime. To analyze the interference of sound waves in the vicinity of the listener, a short Gaussian pulse-like solitary
wave is set up on the loudspeaker's vibrating surface. Since sound waves in the audible band are used for thesesimulations, a mesh size of less than 25 mm is used. The laboratory is 3m of depth and width, and 2.5m of height,
which means that the computational whole target consists of over 15 million elements. This simulation requires alarge number of calculations, so a parallel computing method with ADVENTURE_Sound that is an open-source
software for sound simulation, is used to run the simulation. From the simulation results of the non-stationary analysis
and consider the effect of the presence of the chair on the sound image localization.
Keywords: large-scale simulation, acoustical sound field, non-steady state analysis

1. Introduction

2.2. Simulation procedures with
ADVENTURE_Sound

In the experimental environments, there are objects todisturb sound wave propagation: wall, floor, ceil andchair, etc. Since the effects of these objects change thesound waves reaching the listener's ears, the accuracy ofsound image localization may also change. Our goal is toclarify the effect of these environments on the soundimage localization characteristics. To evaluate the timerelated effects of objects around the listener on the soundwaves reaching the listener's ears, this paper examines a
non-steady state sound field simulation is performed.

The procedure to use ADVENTURE_Sound is asfollows; 1) generate the mesh data, 2) add boundary
conditions (BCs), 3) subdivide the mesh data into
subdomains, 4) simulate the sound field distributions, 5)convert to the results to vtk data for visualization. On the
fourth procedure, it needs to calculate Helmholtz
equation in the time domain. For this, Newmark 𝛽method is adopted to the equation. As the boundary
conditions, input wave function and first and second
derivatives of that function.

2. Adventure Sound
2.1. Brief introduction

3. Preliminary test to simulate non-steady state
analysis

ADVENTURE_Sound is an open-source software forsound simulation. This allows us to simulate parallel
finite element analysis software for the sound fields inhuge space. Analyses are performed by solving the largescale linear system with parallel computing based on the
Mandel’s domain decomposition method (DDM) [1].

3.1. Test modelFig.1 shows that test model for simulation. The model is
AHLV100 that is known as a reference model in
code_Aster. This is also described in the
ADVENTURE_sound manual as a sample. This
simulation was done to confirm the use of transient
analysis in ADVENTURE_Sound.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
675

Akihiro Kudo, Makoto Sakamoto, Amane Takei

Fig. 1. Test model
3.2. Specs of the parallel computer

Fig. 2. An input wave for boundary conditions

Our parallel computer is called as TOMAHAWK.
Table 1 shows that the detailed specs of the parallel
computer.

3.4. Simulation results
Fig.3 shows that the simulation results at some time
points. In this figure, bright red color indicates that highintensity of the vector potential that is proportional to thesound pressure.

Table 1. Specs of parallel computer

CPU
Main memory/node[GB]Number of nodes
Total coresOS
CompilerIntel Xeon E52650L,1.8GHz
with 16 cores
L2 20480[KB]
32
8128
CentOS 6.2
gcc-4.4.7
mipch-3.2

t=999μst=1999μs

t=3899μs

3.3. Simulation conditions
Table 2 shows that analysis conditions.

Fig. 3. Simulation results at some time points

Table 2. Analysis conditions for the test model
Mesh size[m]Number of elements
Absorption boundary [kg/m2 s]
Medium(air) 𝑟0 [kg⁄m3 ]Speed of sound 𝑐0 [m/s]Time resolution ∆𝑡 [μs]Duration time to simulate[μs]Velocity potential {𝛷}𝑡
Full width at half maximum[ms]Cut-off frequency 𝑓𝑐 [Hz]3.5. Discussions

0.012
140, 604
445.9
1.3
343
1.0
4000
Gaussian pulse
0.264
646

From this figure, a single waveform with continuous andpositive intensity was observed and the waveform moves
from left side to right. Because the sound wave takes
2900μs to reach the right side and length of AHLV100 is
1m, estimated sound speed is 345m/s. This estimate isconsistent with the theoretical value. The simulation
results show that the use of ADVENTURE_Sound innon-steady state acoustical analysis is appropriate.
4. Simulation of acoustical experimental
environment

For boundary conditions as input wave, the single wave
is useful to analysis the transient sound field, because
sound reflections are observed clearly. In this study, agaussian pulse as velocity potential is adopted as follows,2{𝛷}𝑡 = 𝑒 −{𝛼(𝑡−𝑡0)} …
(1）
In the Eq. (1), decay parameter 𝛼 is set to 3141, delay
parameter 𝑡0 is set to 0.001[s].
An input wave for boundary conditions is shown in Fig.
2.

4.1. Simulation modelFig. 4 is the dimensions on the side view of simulation
model. Table 3 and 4 are dimensions of target andanalysis conditions for acoustical experimental
environment, respectively.
In this simulation model, there are one loudspeaker andsnowman pair that simulates an acoustical experimental
arrangement. The snowman simulates a listener’s head
and torso [2]. In addition, this model also includes a chair
to hold the listener in position. These are practical andtypical acoustical experimental environment. By©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
676

Paper Title (First 4 Words)

analyzing the sound waves at a position corresponding to
the ear of the Snowman model, it is possible to evaluate
the effect of sound reflections from the chair as an
obstacle on the sound image localization.

From these results, the sound wave reaches to the listenerat approx. 5ms and the time for gaussian pulse to show its
maximum value is 1ms, because it takes for approx. 4ms,
the simulation corresponds to the theoretical value.

Fig. 4. Dimensions on the side view of simulation
model
Table 3. Dimensions of the target
Room, width [m]×depth[𝑚]×height [m]Room volume [m3 ]Loudspeaker, width [m] ×depth [m]×height [m]Diameter of the listener head, torso [m]Distance from loudspeaker to listener [m]Backrest, width [m] ×depth [m] ×height[m]Seat, width [m] ×depth [m] ×height [m]Fig. 5. Simulation results on the xz-plane at y=1.5m
3×3×2.5
22.5
0.1×0.12×0.12
0.177, 0.3 [2]1.5
0.5×0.1×0.65
0.5×0.5×0.1

Table 4. Analysis conditions for the acousticalexperimental environment
Mesh size [m]Number of elements
Room wall, floor, ceiling [kg/m2 s]
Loudspeaker, snowman, chair [kg/m2 s]
Medium(air) 𝑟0 [kg⁄m3 ]Speed of sound 𝑐0 [m/s]Time resolution ∆𝑡 [μs]Duration time to simulate [μs]Velocity potential {𝛷}𝑡
Full width at half maximum [ms]Cut-off frequency 𝑓𝑐 [Hz]0.025
14,497,814445.9
4.56× 109
1.3
343
1.0
8000
Gaussian pulse
0.264
646

Fig. 6. Simulation results on the xy-plane at z=1.0m
4.3. Discussions

4.2. Simulation results

Fig. 7 is the snapshots on the xz-plane at y=1.5m.
From 6.7ms to 7.3ms, an increased sound pressure wasobserved bottom of the listener's torso due to the soundwaves reflecting off the seat of the chair.
The sound wave reaches the listener at approx. 5.2ms andthe sound wave reflected by the chair returns to thelistener at approx. 6.4ms, so this duration time is 1.2ms.
This might affect the spaciousness of sound image
although not directional accuracy of the sound image
localization, because this time interval exceeds theFig. 5 is the simulation results on the xy-plane at y=1.5m.
Since gaussian pulse is given to the surface of theloudspeaker, the sound waves are spread over. As soundwave travels from the loudspeaker, the sound directivity
can be observed on x-axis.
Fig. 6 is the simulation results on the xy-plane at z=1.0m.
The distance from the loudspeaker to the listener is 1.5m,
so it takes 4.4ms for the sound wave to reach the listener.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
677

Powered by TCPDF (www.tcpdf.org)

Akihiro Kudo, Makoto Sakamoto, Amane Takei

maximum time difference 0.6ms that humans use to
localize a sound image [3].

References
1.

2.

3.

J. Mandel, ”Balancing domain decomposition”,
Communications on Numerical Methods in Engineering,
Vol.9, pp.233-241, 1993.
S. Shimada, K. Sugiyama, H. Hokari, Wave equation andHead related transfer functions model (in Japanese,
Kameda book service, 2011).
B. C. J. Moore, An introduction to the Psychology ofHearing (ELSEVIER ACEDEMIC PRESS, 2007)

Authors Introduction
Prof. Akihiro Kudo
He received Ph.D. degree from Nagaoka
University of Technology He is a professor
in the Department of engineering forinnovation,
National
InstituteofTechnology, Tomakomai college. He is amember of Acoustical society of Japan,
Information
andCommunication
Engineers (IEICE).
Prof. Makoto Sakamoto
He is presently a professor in the Facultyof Engineering, University of Miyazaki.
His first interests lay in hydrodynamics
and time series analysis, especially the
directional wave spectrum. He is atheoretical computer scientist, and his
current main research interests areautomata theory, languages and computation. He is alsointerested in digital geometry, digital image processing,computer vision, etc.

Fig. 7. Snapshots on the xz-plane at y=1.5m
5. Conclusion
Purpose of this paper is to clarify the effect of theseenvironments on the human auditory characteristics,
especially sound image localization ability. The
simulation target is loudspeaker and snowman modelwith chair, was adopted as a typical acousticalexperimental environment. The simulation results
showed that 1) An increased sound pressure wasobserved bottom of the listener's torso due to the soundwaves reflecting off the seat of the chair, 2) The time
taken for sound wave reaching the listener to reflect off
the chair and return to the listener, was 1.2ms.

Prof. Amane Takei
He is working as Associate Professor for
Department of Electrical and systems
Engineering, University of Miyazaki,
Japan. His research interest includes highperformance computing for computational
electromagnetism, iterative methods forthe solution of sparse linear systems,
domain decomposition methods for largescale problems. Prof. Takei is a member of IEEE, an expert
advisor of IEICE, a delegate of the Kyushu branch of IEEJ,a director of JSST.

Since the time required for sound image localization is
up to 0.6ms and the sound wave delay caused by chairreflection is shorter than this time, the reflection on chair
may affect the spread of the sound image more than the
directional accuracy of sound image localization.
Acknowledgements
This research was supported by Grant-in-Aid for
Scientific Research 22K19779.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
678

A Deep Insight method with Morphological Analysis
Toyoaki Tomimoka

Graduate School of Engineering, Miyazaki University,
Japan, hm18026@student.miyazaki-u.ac.jp
Satoshi Ikeda*

Department of Computer Science and System Engineering, Miyazaki University,
Japan, bisu@cs.miyazaki-u.ac.jp
Makoto Sakamoto

Department of Computer Science and System Engineering, Miyazaki University,
Japan, saka2000@cc.miyazaki-u.ac.jp
Takao Ito

Graduate School of Engineering, Hiroshima University,
Japan, itotakao@hiroshima-u.ac.jp＊

Corresponding Author

Abstract
DeepInsight method, which interprets non-image data as image data, has garnered significant attention; however, auniversal approach for this method has yet to be standardized. Our objective was to formulate a comprehensive
method for DeepInsight, encompassing three key steps: strategic variable placement, effective feature extraction,and optimal model construction. Consequently, we successfully developed a model capable of predicting the Nikkei
Stock Average with an accuracy of approximately 60%, marking a significant stride towards establishing theversatility of the DeepInsight method.
Keywords: DeepInsight method, t-SNE, Convolutional Neural Network, Morphological analysis.

1.

Introduction

1.1

Research background2) Feature Combination Restrictions: Another approach
involves limiting combinations of features. Deep
Learning [3] achieves this by dividing the problem into
smaller regions, but it is most effective with data
exhibiting strong pixel correlations, such as images.
DeepInsight method [4], has emerged to extend deeplearning to non-image data, although a standardized
procedure for its application is yet to be established.

Machine learning faces challenges when dealing withhigh-dimensional data, known as the curse of dimensionality [1]. This issue arises because as the number ofdimensions increases, the potential combinations of datagrow exponentially, leading to computational challenges
and inadequate learning results. To address this, twoapproaches can be employed:
1) Dimensionality Reduction: One strategy is to reduce
dimensionality while preserving information. t-SNE [2]is a common technique that expresses data distances asconditional probabilities, keeping similar high dimensional data close in lower dimensions.

With the rise of social media, text mining utilizing
platforms like SNS has gained prominence. Predicting
stock prices has seen success by leveraging emotions
expressed on SNS. Researchers are exploring deeplearning analysis of news articles to build more accurate
models, especially as sentiment analysis on platforms like

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
679

Toyoaki Tomimoka, Satoshi Ikeda, Makoto Sakamoto, Takao Ito

SNS has proven effective. The broader adoption of thesetechniques holds promise for improving predictions andinsights across various domains.
1.2

𝑌 (𝑡) = 𝑌 (𝑡−1) + 𝜂

Here, t denotes the iteration, η is the learning rate, and α
is the momentum term.
2.2 Learning Image Data with t-SNE using CNN
A Convolutional Neural Network (CNN) comprises fully
connected, convolution, and pooling layers. In the fully
connected layer, units are interconnected, with weights
depicted in Fig. 2.1. Inputs and outputs to each unit are
one-dimensional vectors, and the output unit's value isdetermined by multiplying input values by connection
weights and adding bias.

Research purpose

The aim of this research is to transform the three stages
involving proper variable placement, feature extraction,and model construction into a versatile DeepInsightmethod. By utilizing the proposed method, we aim toaccurately predict tomorrow's stock price movements andvalidate the effectiveness of the approach.
２.

Research method

2.1

Summary of t-SNE

The t-SNE is a dimension reduction algorithm designed toreduce high-dimensional data to two or three dimensions.
The algorithm represents the proximity between points 𝑥𝑖
and 𝑥𝑗 on a high dimension as a ooint probability
distribution 𝑝𝑖𝑗 :Fig. 2.1 Fully Connected layer weights

𝑝𝑗|𝑖 + 𝑝𝑖|𝑗(𝑖 ≠ 𝑗), 𝑝𝑖𝑖 = 0.
𝑝𝑖𝑗 =
2𝑛

In the Convolutional layer, the outcomes of convolution
operations for each filter are generated and serve as one
unit in the subsequent layer. To derive the output, akin to
the Fully Connected layer, compute the inner product foreach filter, incorporate the bias, and apply the activation
function. The Pooling layer condenses information toconvert the input data into a more manageable format. Fig.
2.2 illustrates Max-pooling, wherein the maximum value
is selected from color-coded small areas.

2−||𝑥𝑖 −𝑥𝑗 ||
exp(
)
2𝜎𝑖 2

Here 𝑝𝑗|𝑖 =𝛿𝐶+ 𝛼(𝑌 (𝑡−1) − 𝑌 (𝑡−2) ).
𝛿𝑌for 𝑖 ≠ 𝑗.
2−||𝑥𝑖 −𝑥𝑘 ||∑𝑘≠𝑖 exp()
2𝜎𝑖 2And the proximity between points 𝑦𝑖 and 𝑦𝑗 in a low
dimension as a ooint probability distribution 𝑞𝑖𝑗 :2 −1(1 + ||𝑦𝑖 − 𝑦𝑗 || )

𝑞𝑖𝑗 =

2 −1

(𝑖 ≠ 𝑗).

∑𝑘≠𝑖 (1 + ||𝑦𝑖 − 𝑦𝑗 || )The oboective is to match the distance relationships
between data points in high dimensions with those in low
dimensions after dimension reduction. This is achieved by
setting set 𝑝𝑖|𝑗 = 𝑞𝑖|𝑗 , and the uullback-Leibler (uL)divergence is employed to measure the distance between
𝑝𝑖|𝑗 and 𝑞𝑖|𝑗 with the goal of minimizing the loss function:

Fig. 2.2 Max-pooling
2.3

news article
2020/10/1
****
****

𝑝𝑖𝑗
𝐶 = ∑ 𝐾𝐿(𝑃𝑖 ||𝑄𝑖 ) = ∑ ∑ 𝑝𝑖𝑗 log
𝑞𝑖𝑗
𝑖

𝑖

Morphological analysis𝑗Finally, stochastic gradient descent is used to minimize
the loss function. The gradient is given by

2020/10/2
****
****

2 −1
𝛿𝐶
= 4 ∑(𝑝𝑗𝑖 − 𝑞𝑗𝑖 )(𝑦𝑖 − 𝑦𝑗 ) (1 + ||𝑦𝑖 − 𝑦𝑗 || )𝛿𝑦𝑖𝑗number of word

Morphologicalanalysis

2020/10/1:
word A:5
word B:2
2020/10/2:
word A:3
word B:4

Fig. 2.3 Example of Morphological analysis

Using this gradient, is gradually updated using the renewal
formula:Morphological analysis dissects sentences into morphemes,
assigning parts of speech. Using a wordbook, the least-cost
method selects optimal words based on part-of-speech©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
680

A Deep Insight method,connection and word appearance. This process is applied
to news articles to count occurrences of words essential forstock price prediction, as illustrated in Fig. 2.3.
３.

Experiment

3.1

Create input data

Each generated image is labeled with 0, 1, or 2. Here, 0signifies that the next day's Nikkei Stock Average willremain unchanged (range from -100 yen to +100 yen), 1
indicates an increase (range from more than +100 yen), and2 indicates a decrease (range from less than -100 yen).
3.2

Firstly, split all the data into training and test sets. Allocate
80% of the total data to the training set and 20% to the test
set to ensure unbiased data representation. Subsequently,
normalize the input image data. Given that color values are
typically expressed as {0, 1, ..., 255}, normalizing the data
to the range [0, 1] can be achieved by dividing each value
by 255. Finally, apply a one-hot vector transformation to
the label data of the input images, converting them into abinary class matrix.

In the experiment, two datasets were used:
1) Daily stock prices of the 225 stocks constituting the
Nikkei Stock Average.
2) Word count in news articles.
For 2), news articles related to stocks from the uabutan
website [5] during the period from 2020 to 2021 were used.
Morphological analysis was applied to the news articles tocount the occurrences of words considered effective in
predicting stock prices (e.g., "weak yen"). A specialized
dictionary (with 563 registered words) tailored for stock
price prediction was employed for word counting. The data
for stock prices and word counts in news articles wereconsolidated into a single CSV datasheet (Table 3.1).
Table3.1
company1company2
・・
company225
word1
word2
・・

10/1
a1
a2
・・
a225
A1
A2
・・

3.3

・・
・・
・・
・・
・・
・・
・・
・・

Evaluation method

Display the accuracy rates for each training and test dataset
to assess the progress of learning and whether DeepInsightmethod is effective for prediction.
3.4

CSV datasheet
10/2
b1
b2
・・
b225
B1
B2
・・

Data processing

Experimental results

The number of training data is 175, and the number of test
data is 45. Among the 45 test data, the percentage of '0' was
16%, the percentage of '1' was 53%, and the percentage of'2' was 31%, indicating that the percentage of '0' wasrelatively small, with '1' accounting for more than half.

10/10
j1
j2
・・
j225
J1
J2
・・

The first row of Table 3.1 represents the dates for whichdata was obtained. In this study, one sheet was created fora continuous 10-day period. Lowercase letters (a1, a2, etc.)
represent daily stock prices for individual companies,
while uppercase letters (A1, A2, etc.) represent the word
count for each date. A total of 222 sheets of data werecreated for the entire data collection period. Using t-SNE,
the data from each of the 225 sheets were transformed into
image data composed of 783 points (225 for stock prices +563 for words), as illustrated in Fig. 3.1. These images will
be used as training and testing data for the Convolutional
Neural Network (CNN).
Fig. 3.2

Fig. 3.1 An example of t-SNE transformation

Experimental result

In Fig. 3.2, where the horizontal axis represents theepoch (number of executions) and the vertical axis
represents accuracy (correct answer rate), the accuracy rateof the training data reached nearly 100% at 25 epochs. The
test data achieved its highest accuracy rate of 59.6% at 27
epochs, but further increasing the number of epochs did notlead to a higher accuracy rate. Additionally, at epoch 10,on

CSV data©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
681

Powered by TCPDF (www.tcpdf.org)

Toyoaki Tomimoka, Satoshi Ikeda, Makoto Sakamoto, Takao Ito

the accuracy rates of the training data and test data weresimilar, but thereafter, a significant difference emerged.
Examining the conditional probabilities in Table 3.2, outof 45 test data items, 53% (24 items) were predicted as '1',
and among those, 58% (14 items) were correctly predicted
as '1'. In contrast, 31% (14 items) were predicted as '2', andamong those, 50% were correctly predicted as '2'.
Table3.20(16%)
1(53%)
2(31%)

Authors Introduction
Mr. Toyoaki Tomioka
He is a student of the Mechanical andInformation Systems Course, graduate
school of Engineering, University ofMiyazaki.

Conditional probability0
14%(1)
16%(4)
14%(2)

1
71%(5)
58%(14)36%(5)

2
14%(1)
25%(6)
50%(7)

Prof. Satoshi Ikeda
He received PhD degree from Hiroshima
University. He is an associate professor in
the Faculty of Engineering, University ofMiyazaki. His research interest includes
graph theory, probabilistic algorithm,
fractal geometry and measure theory.

4.Consideration
Our experiment aimed to turn the three steps of appropriate
variable placement, feature extraction, and appropriate
model construction into a general-purpose DeepInsightmethod. Appropriate variable placement was achieved byusing t-SNE to convert effective prediction words obtained
through morphological analysis of stock price statistical
data and news articles into image data. Feature extraction
utilized CNN, an effective method for image recognitionin machine learning. Despite struggling with suppressing
overfitting, we achieved a maximum accuracy ofapproximately 60%. Assuming the model was used for
actual trading, this resulted in an approximately 11%
increase in profits over one month, surpassing the averageannual interest rate for stock investments (3-10%).
These results indicate that the "appropriate variable
placement," "feature extraction," and "appropriate model
construction" approach is effective in DeepInsight. Futurechallenges include increasing the types of label data andimproving overall accuracy.

Prof. Makoto Sakamoto
He is presently a professor in the Faculty ofEngineering, University of Miyazaki. His
first interests lay in hydrodynamics andtime series analysis, especially the
directional wave spectrum. He is atheoretical computer scientist, and his
current main research interests areautomata
theory,languages
andcomputation. He is also interested in digital geometry, digital
image processing, computer vision, computer graphics, etc.
Prof. Takao Ito
He is Professor of Management ofTechnology (MOT) in Graduate School ofEngineering at Hiroshima University. He isserving concurrently as Professor of Harbin
Institute of Technology (Weihai) China. He
has published numerous papers in refereed
journals and proceedings, particularly in thearea of management science, and computer science. He has
published more than eight academic books including a bookon Network Organizations and Information (Japanese
Edition). His current research interests include automata
theory, artificial intelligence, systems control, quantitative
analysis of inter-firm relationships using graph theory, andengineering approach of organizational structures using
complex systems theory.

References
1.R.EBellman, “Dynamic Programming”, Princeton University
Press,1957
2.G.E.Hinton, Laurens van Maaten, ”Visualizing Data using tSNE”,Journal of Machine Learning Research 9 (2008) 25792605
3.G.E.Hinton, S.Osindero, Y.W.Teh, ”A Fast Learning Algorithm
for Deep Belief Nets”, NeuralComputation 18(7),2006,4pp.1527-1554
4.Tatsuhiko Tsunoda, ”DeepInsight: A methodology to transforma non-image data to an image for convolution neural networkarchitectu”, Scientific Repotrts. 10.1038/s41598-019-47765-6
5. “株探”©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
682

Support for Museum Exhibition of Small Fungi using AR Technology
Kakeru Takemura, Ota Hamasuna, Fumito Hamakawa

Graduate School of Engineering, University of Miyazaki, Japan
Satoshi Ikeda, Kaoru Ohe, Amane Takei, Makoto Sakamoto*

Faculty of Engineering, University of Miyazaki, Japan
Shuichi Kurogi

Miyazaki Prefectural Museum of Nature and History, Japan
Email: hm18022@student.miyazaki-u.ac.jp, saka2000@cc.miyazaki-u.ac.jp*Corresponding author

Abstract
There are 25 species of bioluminescent fungi that have been confirmed in Japan, and 12 species have been confirmedin Miyazaki Prefecture. Those fungi are very small. Therefore, it is difficult to observe the structure of fungi exhibitedin museums with the naked eye. The purpose of this research is to display 3DCG models of mushrooms using AR(Augmented Reality) technology in order to facilitate observation of these small mushrooms exhibited in museums.
Two hundred visitors to the museum were asked to use the application and complete a survey. In order to measure
the ease of observation of different app functions and mobile devices used in the survey, we divided the respondents
into four groups.
Keywords: AR, Exhibition Support, Museum, bioluminescent fungi

Table 1. Development Environment

1. Introduction

OS
SoftwareAugmented reality is a technology that superimposes
digital information on the real world. In recent years, AR
technology can be used not only on head mount display,
but also on mobile devices such as smartphones. This
study will examine the use of AR technology to support
observation in museums. There are 25 species of
bioluminescent fungi that have been confirmed in Japan,
and 12 species have been confirmed in Miyazaki
Prefecture. Those fungi are very small. Therefore, it isdifficult to observe the structure of fungi exhibited inmuseums with the naked eye. The purpose of thisresearch is to display 3DCG models of fungi using AR(Augmented Reality) technology in order to facilitate
observation of these small fungi exhibited in museums.

Smart phone
TabletWindows 10 Pro
Blender 2.93
Unity 2020.3.22f1
Visual Studio 2019
Galaxy S8
Yoga Tab 11

2.2. Target fungi
The bioluminescent fungi, Favolaschia peziziformis[Fig. 1], Cruentomycena orientalis [Fig. 2] and Favolus
manipularis [Fig. 3], were the subjects of this study.
2.3. 3DCG of fungi

2. AR Application Development
2.1. Development Environment

The CG model was created based on the images
provided.

An AR application was created for use withsmartphones and Tablets. The development environmentis shown (Table 1). Blender is a comprehensive 3DCG
software. It was used to create a 3DCG model of fungi[3], [4]. Unity was used to develop applications formobile terminals [1], [2].

Fig. 1 Favolaschia peziziformis©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
683

Kakeru Takemura, Ota Hamasuna, Fumito Hamakawa, Satoshi Ikeda, Kaoru Ohe, Amane Takei, Makoto Sakamoto, Shuichi Kurogii.

Explain that specimens of glowing mushrooms areon display.
ii.
Explain to users how to use the application.
iii.
Users use the app to make observations.
iv.
Users are free to finish their observations.
Show how the Questionnaire items [Table 2].
Table 2. Questionnaire items
1. Observation was fun.
2. I interested in fungi.
3. New discoveries were made.
4. I understand the structure.
5. It was easy to observe.
6. AR experience was good.
7. Operation was easy.
8. I want to use it in other exhibits.
9. I got the emotion of observing native fungi.
10. I wanted to touch the fungi.
11. Did you notice the hole?
12. Did you notice the rounded bottom?
13. Did you notice the reddish color?
14. Did you notice how the back of the fungus's
umbrella is shaped like folds?
15. Did you notice the reticulated underside of thefungus umbrella?

Fig. 2 Cruentomycena orientalis

Fig. 3 Favolus manipularis2.4. AR application
An image that serves as a marker is loaded, and an AR
3DCG model of a mushroom is displayed on top of it.
When the marker disappears from the screen, thedisplayed 3DCG model also disappears, and when thesame or another marker is scanned, the 3DCG model isdisplayed again. This application was used as the basic
application. In addition to the basic application, anotherapplication was prepared with a function that tilts themushrooms in the same direction when the user moves
his/her finger while touching the AR-displayed
mushrooms on the screen. A demonstration of theapplication in use is shown below.

3.3. Questionnaire Results
The questionnaire included questions to answer about theexperience of observing the mushrooms using theapplication and whether or not they noticed any
characteristics of the fungi. The questionnaire was basedon the MES [5]. Ten questions on a 5-point scale wereasked, five questions asking if the mushrooms wereobserved for their characteristics, and one open-ended
question. The results of the questionnaire are shown
below [Table 3], [Table 4].

3. Evaluation experiment

Table 3. Experiences observed using the app (average)

3.1. Experimental environment

item

The evaluation experiment was conducted at the
Miyazaki Prefectural Museum. A special exhibition was
being held and a place for observation using theapplication was set up next to the mushroom exhibit.
Visitors were asked to cooperate in the evaluation
experiment.

1
2
3
4
5
6
7
8
9
10

3.2. Experimental procedure
The experiment should take no more than 5 minutes.
After the observation using the app was over, theparticipants were asked to fill out a questionnaire. The
experiment was divided into four groups using
smartphones and tablets, basic apps and apps withadditional features. A total of 200 people, 50 in eachgroup, were surveyed. How the experimental procedureis shown.

Basic
apps
4.9
4.57
4.72
4.51
4.76
4.91
4.75
4.8
4.44
4.45

Additional
apps
4.92
4.58
4.71
4.47
4.78
4.87
4.63
4.68
4.49
4.45

Smartphone

Tablet

4.92
4.54
4.67
4.94
4.82
4.91
4.77
4.73
4.53
4.39

4.9
4.61
4.76
4.48
4.72
4.88
4.61
4.74
4.4
4.51

Table 4. About the characteristics of fungi
item
Basic
apps

NoticedDidn'tnotice.

11
88
%
12
%

12
68
%
32
%

13
89
%
11
%

14
58
%
42
%

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
684

15
66
%
34
%

Support for Museum Exhibition

Additional
apps

Noticed

Acknowledgements

87
%
13
%
84
%
16
%
91
%
9%

82
92
80
85
%
%
%
%
Didn't18
8% 20
15
notice.
%%%Smartp Noticed
69
85
63
71
hone
%
%
%
%
Didn't31
15
37
29
notice.
%%%%Tablet
Noticed
80
96
75
80
%
%
%
%
Didn't20
4% 25
20
notice.
%%%Since all items were 3 or higher in the experienceobserved using the app, it can be said that the app is an
aid to observation. In comparing the results between thebasic app and the app with additional functions, there wasa clear difference between Q7 and Q8. The respondents'
opinion that simple functions were easier to operate anduse was based on the belief that they wanted ease of use
rather than in-depth observation. For the question about
mushroom characteristics, differences were found at 12,
14, and 15. Apps with additional features for areas thatcannot be observed without looking directly at themushroom from the side, such as the back of themushroom, were a good result. Comparisons will bemade between the use of smartphones and tablets.
Differences were found in Q4, 5, 7, 9, and 10.
Smartphones can be handled with one hand, making iteasier to operate and observe from various angles, whichis thought to have led to better understanding ofmushrooms. Tablets were found to provide a sense of
immersion due to their large screens. In the questions
about mushroom characteristics, there was a 10%
difference in Q12, 13, and Q14, with tablet users being
able to observe mushrooms in more detail.

We would like to express our sincere gratitude to theprofessors who guided us in this research and to those
who cooperated in the questionnaire survey.
References
1.
2.
3.
4.
5.

多田憲孝,“Unity AR Foundation による AR アプリ開発入門”, インプレス R&D(in Japanese), 2021.
橋本直, “AR プログラミング -Processing レシピ-”,オーム社(in Japanese), 2012.
伊丹シゲユキ, “入門 Blender 2.9 ~ゼロから始める 3D制作”,秀和システム(in Japanese), 2020.
Benjamin, “Blender 2.9 3DCGモデリング・マスター”,
ソーテック社(in Japanese), 2021.
Mohd Kamal Othman, Engaging Visitors in Museums
with Technology: Scales for Measurement of Visitor and
Mulrimedia Guide Experience, Human-Computer
Interaction-INTERACT, 2011, pp. 92–99.

Authors Introduction
Mr. Kakeru Takemura
He is currently enrolled in the Master's
course in Mechanical Information
Systems,
Graduate
SchoolofEngineering, University of Miyazaki. His
current research theme is museum
support using AR technology.
Mr. Ota Hamasuna
He is a master student at Department ofComputer
ScienceandSystemEngineering, University of Miyazaki. His
current research interests are augmented
reality, computer graphics, and so on

4. Conclusion
We were able to confirm that AR technology supports
observation in museums. The apps with additional
functions were found to allow more detailed observationof the mushrooms. As for the difference between the
different devices, it was also found that the tablet device
allowed for more detailed observation of the mushrooms.
However, the basic function apps were better in terms ofease of use, while smartphones were rated higher in termsof ease of observation. Therefore, it is better to use asmartphone when carrying a mobile device forobservation when used in a museum. In this way, it isimportant to use different app functions and mobile
devices depending on the situation in which they are used.
In this study, we supported the observation of small
objects, but in the future we would like to consider
supporting the observation of very large objects that are
difficult to observe.

Mr. Fumito Hamakawa
He is a master student at Department ofComputer
ScienceandSystemEngineering, University of Miyazaki. His
current research interests are computer
graphics, cellular automaton simulation,and so on.
Prof. Satoshi Ikeda
He received PhD degree from Hiroshima
University. He is an associate professor in
the Faculty of Engineering, University ofMiyazaki. His research interest includes
graph theory, probabilistic algorithm,
fractal geometry and measure theory.
1.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
685

Powered by TCPDF (www.tcpdf.org)

Kakeru Takemura, Ota Hamasuna, Fumito Hamakawa, Satoshi Ikeda, Kaoru Ohe, Amane Takei, Makoto Sakamoto, Shuichi Kurogi

Dr. Kaoru Ohe
2.
She received her Ph.D. degrees from
University of Miyazaki, Japan, in 2014.
Currently she is an Associate Professor of
the Center for Science and Engineering
Education, Faculty of Engineering. Her
research is separation engineering
especially adsorption hazardous heavy
metals and oxyanions
Prof. Amane Takei
He is working as Associate Professor for
Department of of Electrical and systems
Engineering, University of Miyazaki,
Japan. His research interest includes highperformancecomputingforcomputational
electromagnetism,
iterative methods for the solution ofsparse linear systems, domain decomposition methods forlarge-scale problems. Prof. Takei is a member of IEEE,an expert advisor of IEICE, a delegate of IEEJ, a directorof JSST.
Prof. Makoto Sakamoto
He is presently a professor in the Facultyof Engineering, University of Miyazaki.
His first interests lay in hydrodynamics
and time series analysis, especially the
directional wave spectrum. He is atheoretical computer scientist, and his
current main research interests areautomata theory, languages, and computation. He is alsointerested in digital geometry, digital image processing,computer vision, computer graphics, virtual reality, etc.
Mr. Shuichi Kurogi
He is a curator of Miyazaki Prefectural
Museum of Nature and History. His
research interests are wild plants, fungiand lichens in Miyazaki Prefecture. His
current research interests are luminous
mushrooms, an endangered plant.
Member of Japanese Society for Plant
Systematics and Mycological Society of Japan.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
686

Automatic Selection of High-Grade Dried Shiitake Mushrooms
using Machine Learning
Ota Hamasuna, Kakeru Takemura, Kodai Hasebe, Fumito Hamakawa, Bidesh Biswas Biki

Graduate School of Engineering, University of Miyazaki, Japan
Satoshi Ikeda, Kaoru Ohe, Amane Takei, Makoto Sakamoto*

Faculty of Engineering, University of Miyazaki, Japan
Kazuhide Sugimoto

SUGIMOTO Co., Ltd., Japan*

Corresponding Author

abstract
Miyazaki Prefecture is blessed with the rich nature of the Kyushu Mountains, where many mushrooms have grownwild since ancient times, and the production of dried shiitake mushrooms ranks second in Japan after Oita Prefecture.
SUGIMOTO Co., Ltd., which will be cooperating with us this time, is a long-established company established in
1970, and at times it is necessary to sort over 1 ton. However, since selection is still done manually, it is very difficultfor employees to do the selection. The goal of this research is to determine the quality of shiitake mushrooms using
deep learning and video image processing.
Keywords: image classification, video processing, CNN, OpenCV

1. Introduction

3. Method

SUGIMOTO Co., Ltd. collects dried shiitake
mushrooms directly from approximately 600 producers
in Takachiho. Shiitake mushrooms are at their peak inspring and autumn, and at peak times, more than 1 ton of
shiitake mushrooms can be brought in a day [1].
However, shiitake mushrooms are still sorted manually,and sorting this much requires a lot of effort. The goal ofthis research is to use deep learning and video processing
to determine the quality of donko, a high-quality dried
shiitake mushroom.

3.1. Target of selection
The following two targets are selected. The one on theleft (Fig. 1) is a good shiitake mushroom, and the one on
the right (Fig. 2) is a bad one.

Fig. 1 Good item

Fig. 2 Bad item

2. Development environment
The following four main examples of bad items are listed
below. Fig. 3 shows a state where the pileus of a shiitake
mushroom part is broken. Fig. 4 shows a state in which
the pileus of a shiitake mushroom is greatly deformed.
Fig. 5 shows a state where there are holes caused by
insect damage. Fig. 6 shows the pileus of a shiitake
mushroom part being rubbed.

In creating this program, we developed it in the following
environment (Table 1). In the experiment, a stand was
fixed to a conveyor belt, and the smartphone camera was
attached to the end of the stand with the camera facing
down.

Table 1 Development environment
OS

Windows10

language

Pythoncameramoto g30

softwareiVCam

Fig. 3 Cracked

Fig. 4 Deformed

Fig. 5 HoleFig. 6 Rubbed

OpenCVsoftwarePIL
NumPy
Pandas©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
687

Ota Hamasuna, Kakeru Takemura, Kodai Hasebe, Fumito Hamakawa, Bidesh Biswas Biki, Satoshi Ikeda, Kaoru Ohe, Amane Takei, Makoto Sakamoto,
Kazuhide Sugimoto,

The example of a bad item mentioned this time is thefront side with a pileus of a shiitake mushroom on top.
Although there are characteristics of bad items on theback side as well, this experiment focused on the front
side. The reason why we focused on the front side is thatit takes a huge amount of time to collect samples from thefront and back sides, and the camera can only capture
either the front or back side due to the sorting process on
the conveyor belt.

items and bad items is displayed at the top right of thescreen. The count is performed when the final result isobtained.

3.2. CNN
Fig. 8 Program running

In the experiment, CNN was used as an algorithm todetermine whether an image was a good item or a bad
item. Explain about CNN. CNN (Convolution Neural
Network) is a network often used in image recognition
research. This CNN is characterized by being constructedby stacking layers with several unique functions, such as
convolutional layers and pooling layers [1]. Currently, it
is attracting increasing attention as it is being used in avariety of fields. The image in Fig. 7 shows the flow ofhandwritten digit classification using CNN. First,
handwritten digits are given as an input image, thefeatures of the image are extracted by convolution, andthe extracted features are aggregated in a pooling layer.
Finally, the fully connected layer transforms the output
into a one-dimensional form and outputs each selection
result as a probability.

4. Experiment content
An evaluation experiment was conducted to verify theusefulness of the developed sorting system. They sort out
17 good and 23 bad mushrooms, and tally up whether the
final result matches the shiitake mushrooms that were
flushed. The number of matches is defined as the numberof correct answers, and the accuracy is defined as follows.
Since the final result is updated sequentially until the flow
is finished, the last displayed final result is taken as thedetection result. The number of cards trained on the CNN
was 668 for good and 759 for bad, split 8:2 between
training and testing.
𝑎𝑐𝑐𝑢𝑟𝑎𝑐𝑦 =𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑎𝑛𝑠𝑤𝑒𝑟𝑠
× 100(%)
𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑠ℎ𝑖𝑖𝑡𝑎𝑘𝑒 𝑚𝑢𝑠ℎ𝑟𝑜𝑜𝑚𝑠 𝑠ℎ𝑒𝑑

5. Experimental result
As a result, the accuracy was low, with 41.1 % of good
items and 91.3 % of bad items. Most of the shiitake
mushrooms thrown away by the model were sorted out as
bad shown in Table 2.
Table 2 Program execution results
Good item
bad item
number of
17
23
flows
The number of7
21
correct answers
accuracy
41.1%
91.3%

Fig. 7 Classification of handwritten digits [2]3.3. Video image processing
In this research, we extract still images of shiitake
mushrooms from moving images and select them using amodel learned by CNN. The video image used in theexperiment was taken from directly above the conveyor
belt, and the shiitake mushrooms were flowing from left
to right. Fig. 8 shows an image of the program being
executed.
A still image extracted from a moving image is displayed
at the top left of the screen. Extract three still images ofthe shiitake mushroom using OpenCV. The extraction
location is when the shiitake mushrooms flow to the left,
center, or right side of the screen. The three images are
selected by the model, and the one with a majority of
selection results is the final result. At the bottom centerof the screen, a "〇" is displayed if the item is good, andan "×" is displayed if the item is bad. If the test is inprogress, "・・・" is displayed. The number of good

The figure below (Fig. 9) shows the selection results of
the CNN model in the still image state. Although theaccuracy of sorting out bad items as bad items was high,the accuracy in sorting out good items as good items waspoor.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
688

Automatic Selection of High-Grade2.

Saha, S.” A Comprehensive Guide to Convolutional
NeuralNetworks
—
the
ELI5
way
“.Medium.2018.[Online]https://towardsdatascience.com/
a-comprehensive-guide-to-convolutional-neuralnetworks-the-eli5-way-3bd2b1164a53,(accessedon2023/12/05)

Authors Introduction
Mr. Ota Hamasuna
He is a master student at Department ofComputer Science and System
Engineering, University of Miyazaki.
His current research interests areaugmented reality, computer graphics,
and so on.

Fig. 9 Results of still image selection using CNN
6. Conclusion
This research is still in its infancy, and theexperimental results were very poor. First, there is the
problem that it is difficult to distinguish between good
items at the CNN learning stage. It was difficult for me tosort by just looking at still images. When actually sorting,
not only the front side but also the back side is important.
Since this study was an experiment on only the front side,information on the back side was missing, and we believethat the accuracy is reduced. As a result, the accuracy ofmoving images deteriorated even more than the accuracyof still images. The reasons for the lower accuracy in thevideo images are that in the case of the video images, theshiitake mushrooms swayed on the conveyor belt,
making it impossible to obtain accurate images, and thatthe images to be trained by the CNN and the video images
on the conveyor belt had different lighting and size of theshiitake mushrooms within the frame. Future issues
include the following.
⚫ Increasing the number of images for learning⚫ Collect images of the back side of shiitake
mushrooms⚫ Organizing the environment in which still images
are collected and the environment in which video
images are taken⚫ Reconstruction of CNN model

Mr. Kakeru Takemura
Currently enrolled in the Master's
course in Mechanical Information
Systems,
Graduate
SchoolofEngineering, University of Miyazaki.
His current research theme is museum
support using AR technology.

Mr. Kodai Hasebe
He is a master student at Department ofComputer Science and System
Engineering, University of Miyazaki.
His current research interests are image
processing, machine learning, and soon.

Mr. Fumito Hamakawa
He is a master student at Department ofComputer Science and System
Engineering, University of Miyazaki.
His current research interests arecomputer graphics, cellular automaton
simulation, and so on.

Acknowledgment
I would like to thank my supervisor, Professor Masato
Sakamoto, for his guidance in preparing this thesis. We
would like to thank Kazuhide Sugimoto of Sugimoto
Shoten Co., Ltd. for providing information on shiitake
mushrooms for this study.

Mr. Bidesh Biswas Biki
He is master's student in Data Scienceat Department of Statistics from TU
Dortmund. He is an enthusiast of
Machine Learning and Deep Learning.
He has done some research work andindividual projects on supervised
learning especially on “Classification”
and “Regression” techniques. Also hasdone some work on Image Processing and NLP.

References1.

AIsmiley 編集部.” 画像認識でよく聞く「CNN」とは ？ 仕 組 み や 特 徴 を 1 か ら 解 説 (inJapanese)”.AISmiley.2023.[Online]
https://aismiley.co.jp/ai_news/cnn/,(accessedon2023/12/5).

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
689

Powered by TCPDF (www.tcpdf.org)

Ota Hamasuna, Kakeru Takemura, Kodai Hasebe, Fumito Hamakawa, Bidesh Biswas Biki, Satoshi Ikeda, Kaoru Ohe, Amane Takei, Makoto Sakamoto,
Kazuhide Sugimoto,Prof. Satoshi Ikeda
He received PhD degree from Hiroshima
University. He is an associate professor
in the Faculty of Engineering, Universityof Miyazaki. His research interest
includes graph theory, probabilistic
geometry, fractal geometry and measure
theory.

Prof. Kaoru Ohe
She received her Ph.D. degrees from
University of Miyazaki, Japan, in 2014.
Currently she is an Associate Professorof the Center for Science andEngineering Education, Faculty ofEngineering. Her
research isseparationengineering
especiallyadsorption hazardous heavy metals and oxyanions
Prof. Amane Takei
He is working as Associate Professor for
Department of Electrical and systems
Engineering, University of Miyazaki,
Japan. His research interest includes highperformancecomputingforcomputational
electromagnetism,
iterative methods for the solution ofsparselinearsystems,domaindecomposition methods for large-scale
problems. Prof. Takei is a member of IEEE, an expert
advisor of IEICE, a delegate of IEEJ, a director of JSST.
Prof. Makoto Sakamoto
He is presently a professor in the Facultyof Engineering, University of Miyazaki.
His first interests lay in hydrodynamics
and time series analysis, especially the
directional wave spectrum. He is atheoretical computer scientist, and his
current main research interests areautomata
theory,languages
andcomputation. He is also interested in digital geometry,
digital image processing, etc.

Mr. Kazuhide Sugimoto
He was Born in Takachiho town,
Miyazaki Prefecture. After working insales in the food service and apparelindustries, he joined SUGIMOTO Co.,
Ltd., a wholesaler of dried shiitake
mushrooms produced in Takachiho
town, in 2011 after the Great East Japan
Earthquake. In response to the current
harsh situation, such as aging contract farmers and sluggish
demand for dried shiitake mushrooms, he decided toprotect the producers by promoting new businessdevelopment, which he had experienced in sales. New
items developed using shiitake mushrooms from
Takachiho Township have become a standard item atsupermarkets and department stores outside the prefecture.
In March 2020, he was appointed as Representative
Director, and in 2021, he was selected as a Small andMedium Enterprise Supporter and GFP Ambassador.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
690

Predicting High Volatility Cryptocurrency Prices using Deep Learning
Tsutomu Ito

National Institute of Technology, Ube College, Japan
Kodai Hasebe, Fumito Hamakawa, Bidesh Biswas Biki

Graduate School of Engineering, University of Miyazaki, Japan
Satoshi Ikeda, Amane Takei, Makoto Sakamoto

Faculty of Engineering, University of Miyazaki, Japan
Md Riajuliislam

Department of Statistics, TU Dortmund, Germany
Sabrina Bari Shital

Computer Science and Engineering, Daffodil International University, Bangladesh
Takao Ito*
Hiroshima University, Japan＊

Corresponding Author E-mail: itotakao@hiroshima-u.ac.jp>

Abstract
Even if you want to make a profit from cryptocurrency, you are worried that you will lose money, and it is difficult
to afford it. There are a vast number of papers that study such unpredictable price fluctuations of cryptocurrency.
Currently, it is mainstream to use learning deep to predict the price of cryptocurrency. The goal of this research is topredict the price of cryptocurrency over the long-term using deep learning. The algorithms used are LSTM, GRU,
and Bi-LSTM. The targeted cryptocurrencies are Bitcoin, Ethereum, Litecoin, and Cardano. Finally, we will compare
it with previous research and verify the performance of our model.
Keywords: Price prediction, Cryptocurrency, LSTM, GRU, Bi-LSTM

network). The goal of this research is to use DNNs such
as LSTM, GRU, and Bi-LSTM to predict cryptocurrency
such as BTC, ETH, LTC, and ADA with higher accuracy
than previous research.

1. Introduction
Cryptocurrency is still not very familiar to Japan andmay feel like a distant entity, but it is actually somethingthat will affect our lives in the near future. The main usesof Bitcoin are investment and speculative management.
Cryptocurrency, which started out as a niche product usedby small online groups, has now become mainstream,
attracting the attention of both financial professionals andthe general public. With a market capitalization reachingbillions of dollars, this market has also become a new
arena for speculators [1]. Cryptocurrency assets lack
intrinsic value, regulatory oversight, lack of fixed-term
investment funds, thin order books, short-term
investment methods, and collective psychology, causing
cryptocurrencies to be highly volatile [2]. Therefore, it iseffective to perform technical trading on products withhigh volatility, and if you trade well, you can generatelarge profits [3]. Therefore, it is useful to predict the pricemovements of cryptocurrency using DNN (deep neural

2. Method
2.1. Cryptocurrency
Cryptocurrency is a currency that is exchanged only aselectronic data, does not have the power of compulsory
currency imposed by the state like legal currency, and isprimarily used for transactions over the Internet. Since
the advent of Bitcoin, which began operations in 2009,derivative cryptocurrency called altcoins have beencreated one after another, and with the appearance of
cryptocurrency exchanges that exchange legal currency
and cryptocurrency, holding cryptocurrency has become
easier. It spread rapidly [4]. There are various types ofcryptocurrencies such as BTC, ETH, LTC, NEM,
Ethereum Classic, LISK, etc., and the number is still
increasing. The targets to be predicted in this research are©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
691

Tsutomu Ito, Kodai Hasebe, Fumito Hamakawa, Bidesh Biswas Biki, Satoshi Ikeda, Amane Takei, Makoto Sakamoto, Md Riajuliislam, Sabrina
Bari Shital, Takao Ito

BTC (Bitcoin), ETH (Ethereum), LTC (Litecoin), and
ADA (Cardano).

The structure of Bi-LSTM (Bidirectional LSTM) has astructure that has with a forward LSTM and a reverse
LSTM is shown in Fig. 3. LSTM only flows in theforward direction, but when Bi-LSTM predicts data, forexample, given 𝑥𝑛 a signal to be predicted, prediction is
performed 𝑥𝐸𝑁𝐷 ~𝑥𝑛+1 using the Forward Layer in𝑥0 ~𝑥𝑛−1 the forward direction and the Backward Layerin the reverse direction. It has the characteristic of doing.
This allows it to handle more data than LSTM when thesame amount of data is used for learning. However, since
it handles a large amount of data, it takes longer toprocess than LSTM [7].

2.2. Algorithm
We will explain the three algorithms used in thisresearch to predict cryptocurrency: LSTM, GRU, and BiLSTM. First, I will explain LSTM (Long Short-Term
Memory). Although RNN (Recurrent Neural Network),
which is a neural network that remembers past states, issuitable for analyzing time-series data, it has not beenable to properly process long-term data [5]. LSTM
overcomes this weakness and can maintain long context
relationships. Data is controlled by three gates centeredon the memory cells. The three gates are called forgetgate, input gate, and output gate, and each uses a sigmoid
function to determine whether or not to pass data. Fig. 1
shows the contents of LSTM.

Fig.3. The structure of Bi-LSTM [8].

3. Experiment Content
The conditions for conducting the experiment aredescribed below.
⚫ Three algorithms were used in the experiment:
LSTM, GRU, and Bi-LSTM.
⚫ The four cryptocurrencies targeted for prediction
are BTC, ETH, LTC, and ADA.
⚫ There are two evaluation indicators: RMSE and M
APE.

Fig.1. The structure of LSTM [6].
GRU (Gated Recurrent Unit) is said to have similar
performance to LSTM, and requires less calculation than
LSTM, allowing for faster learning. LSTM is a cell thatmakes it possible to learn long-term features, which was
impossible with RNN, but it has the problem of highcomputational cost. High calculation costs are notdesirable, not only in machine learning. The solution is toeliminate memory cells and reduce the number of gatesto reduce computational costs. In LSTM, two states, thestorage cell, and the output value, are passed on to thenext cell, but GRU combines these states into one. In
addition,
LSTM requires one gate controller for input gate,
forget gate, and output gate, but in GRU, forget gate andinput gate are operated by one controller. Fig. 2 shows
the contents of the GRU.

3.1. Development environment
The model was trained and executed using Google
Colab, which allows experiments to be performed for free.
The number of CPU cores is 2, the number of threads is
4, the RAM is 12.7GB, and the disk is 225.8GB. The deeplearning models LSTM, GRU, and Bi-LSTM used in theexperiment were implemented in Python version 3.10.12.
We used python libraries such as Sklearn and Keras,
which are used for deep learning, and numpy and pandas,
which are used for numerical calculations and analysis.
3.2. Data
The dataset was collected from Yahoo Finance(https://finance.yahoo.com/). We collected the prices offour types of cryptocurrencies on a daily basis: Bitcoin(BTC), Ethereum (ETH), Litecoin (LTC), and Cardano(ADA). The training data consists of January 1, 2018 toJune 8, 2022 (80% of the data), and the test data consistsof June 9, 2022 to November 30, 2023 (20% of the data).
be done. Split the The data proportion of 80:20 for
training and testing is designed in this research. The

Fig.2. The structure of GRU [6].

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
692

Predicting High Volatility Cryptocurrency

detailed information of data specifications is listed inTable 1.
Attribute
Date
Open
High
Low
Close
Adj
Close
VolumeTable 1. Data specifications.
Explanation
Date of transaction
First traded priceHighest traded price
Lowest traded price
Last traded price
The closing price before thesplit is the adjusted price after
the split.
Quantity of trades completed
during the periodprevious studies [9], [10]. Here, we will verify whether
the model proposed in this study is effective. Whenmaking comparisons, we will verify that the period ofexperimentation is consistent with previous research.
Table 3 is a comparison between this research model andthe previous study [9] model, and Table 4 is a comparison
between this study and the previous study [10]. The
accuracy of Bi-LSTM in this study was good in allcomparisons, indicating that the model in this study iseffective.

Type
Date
Continuous
Continuous
Continuous
Continuous
Continuous

Table 3 Comparison of this study and previous study [9] .
Currency

Continuous

StudiesOur
model

3.3. Evaluation index
BTC

Root-Mean-Square Error (RMSE) and MeanAbsolute-Percentage error (MAPE) are used to evaluate
the algorithm in this study. The smaller the RMSE and
MAPE values, the better the prediction model
performance. In Eq. (1) and Eq. (2), n is the number of
elements, 𝑦𝑖 is the correct value, and 𝑦̂𝑖 is the predictedvalue.

[9]

(1)

𝑖=1
𝑛

1
𝑦𝑖 − 𝑦̂𝑖
𝑀𝐴𝑃𝐸 = ∑ |
|
𝑛
𝑦𝑖

RMSE

MAPE

LSTM
GRU
Bi-LSTM
LSTM
ARIMA
SARIMA

1184.7059
1094.3126
647.2073
1447.648
1288.51802.310.04260.04220.02210.030590.03479
0.04665

Table 4 Comparison of this study and previous study [10] .
Currency Studies
Methods
RMSE
MAPE
LSTM
836.2219
0.0318Our
GRU
901.6092
0.0312model
Bi-LSTM
752.4224
0.0290BTC
LSTM
1031.340
0.0397[10]GRU
1274.171
0.057
Bi-LSTM1029.362
0.036
LSTM
59.4498
0.0339Our
GRU
51.67240.0293model
Bi-LSTM
45.7691
0.0253ETH
LSTM
148.522
0.297
[10]
GRU
98.3140.148
Bi-LSTM
83.953
0.124
LSTM
3.620
0.049
Our
GRU
5.453
0.078
model
Bi-LSTM
1.988
0.020
LTC
LSTM
9.668
0.064
[10]
GRU
8.122
0.046
Bi-LSTM
8.025
0.041

𝑛

1
2
𝑅𝑀𝑆𝐸 = √ ∑(𝑦𝑖 − 𝑦̂)𝑖𝑛Methods(2)

𝑖=1

4. Experimental result
Table 2 shows the results predicted according to theconditions in Chapter 3. Evaluation values are rounded
down to the fifth decimal place. Bold letters in theevaluation value indicate the lowest value (the highest
value in terms of accuracy). The results showed that theaccuracy of Bi-LSTM was better than other algorithms
for all cryptocurrencies.
Table 2. Performance results of this research model.
Currencies
Methods
RMSE
MAPE
LSTM
807.7135
0.0332
BTC
GRU
873.7594
0.0362Bi-LSTM770.3047
0.0314
LSTM
21.0737
0.0123
ETH
GRU
33.09120.0192Bi-LSTM
15.18170.0092
LSTM
1.59910.0188LTC
GRU
1.76050.0207Bi-LSTM
1.19280.0130LSTM
0.02030.0578ADA
GRU
0.0236
0.0695Bi-LSTM0.01720.0526

6. Conclusion
In this study, we used three algorithms, LSTM, GRU,
and Bi-LSTM, to predict the prices of fourcryptocurrency: BTC, ETH, LTC, and ADA. The model
was evaluated using the evaluation indicators RMSE and
MAPE. The research results showed that Bi-LSTM madethe most accurate predictions for all cryptocurrency,
followed by LSTM and GRU. Finally, we conducted acomparative verification with previous studies andshowed that this study is more effective than previous
studies. Future works could include applying
reinforcement learning to previous studies [11] andpredicting price fluctuations based on the number oftweets [12].

5. Comparative verification
In the comparative verification, we compare theperformance of the model for predicting the price of
cryptocurrency in this study with other models in©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
693

Tsutomu Ito, Kodai Hasebe, Fumito Hamakawa, Bidesh Biswas Biki, Satoshi Ikeda, Amane Takei, Makoto Sakamoto, Md Riajuliislam, Sabrina
Bari Shital, Takao Ito

Review, 2018, vol.1, no.3. [Online]https://scholar.smu.
edu/datasciencereview/vol1/iss3/1 (accessed on 2023-1116)

References
1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

Bergsli, L. et al. “Forecasting volatility of Bitcoin”.
Research in International Business and Finance. 2022,
vol.59,
pp1-30.
(accessedon2023-11-16)[Online]https://doi.org/10.1016/j.ribaf.2021.101540
S, Ninomiya. “専門的見解：なぜ仮想通貨市場はボ
ラ テ ィ リ テ ィ が 高 い の か (in Japanese) ”
CoinPost.2018,[Online]https://qr.paps.jp/Xsj3v(accessed on 2023/12/07)
Hideki, N. “強化学習を利用した暗号資産の価格予
測 (in Japanese)”. Master’s thesis, Department of Data
Science, Graduate School of Data Science, Shiga
University.
2022.
https://shigau.repo.nii.ac.jp/record/14567/files/%E4%BF%AE%E5%A3%AB%E8%AB%96%E6%96%87%E5%85%A8%E6%96%87R4nagasawa.pdf(accessed on 2023-11-8)
CoinDesk. “暗号資産／仮想通貨とは何か？ クリ
プ ト と は ？ 初 心 者 に も わ か り や す く 解 説 (inJapanese)”.
CoinDesk.
2023.
[Online]https://www.coindeskjapan.com/learn/cryptocur
rency/ (accessed on 2023-11-9)
Yusuke, T. “RNN と LSTM(Long Short Term Memory)の 違 い と 特 徴 (in Japanese) ” . AUC+X.2019.
[Online]https://www.acceluniverse.com/blog/developers/2019/07/lstm.html (accessed on 2023- 11-9)
AFIRobotsBlog. “直感で理解する LSTM・GRU 入
門 - 機 械 学 習 の 基 礎 を マ ス タ ー し よ う ！ (inJapanese)”.
2023.
[Online]https://developers.agirobots.com/jp/lstmgruentrance-noformula/ (accessed on 2023-11-9)
Kai, O. "ディープラーニングを用いた低 SNR 下での生体信号抽出 In vivo Signal Extraction Under Low
SNR by Deep Learning (in Japanese) ". 338th Research
Meeting of the Tohoku Branch of the Society of
Instrument and Control Engineers, vol.4, pp1-8. https//www.topic.ad.jp/sice/htdocs/papers/338/3384.pdf(accessed on 2023-11-9)
Tracyrenee.” What is the difference between
Tensorflow’s
LSTM
and
BiLSTM?”.2023.
https://pub.aimind.so/what-is-the-difference-betweentensorflows-lstm-and-bilstm-d7b1f5d8c772(accessed on
2023- 11-9)
Dinshaw, C. et al. “Statistical Scrutiny of the Prediction
Capability of Different Time Series Machine Learning
Models in Forecasting Bitcoin Prices”. 2022 IEEE 4thInternational Conference on Cybernetics, Cognition andMachine Learning Applications (ICCCMLA), Goa, India,
2022,
pp.
329-336,doi:
10.1109/ICCCMLA56841.2022.9989057. (accessed on
2023-11-16)
Seabe, P. et al. “Forecasting Cryptocurrency Prices Using
LSTM, GRU, and Bi-Directional LSTM: A Deep
Learning Approach”. Fractal and Fractional 7, no. 2: 203.
[Online]https://doi.
org/10.3390/fractalfract7020203(accessed on 2023-11-16)
Hideki, N. “強化学習を利用した暗号資産の価格予測(in Japanese)”. Department of Data Science, Graduate
School of Data Science, Shiga University. 2023.
https://shiga-u.repo.nii.ac.jp/records/14567 (accessed on
2023-11-16)
Abraham, J. et al.”Cryptocurrency Price Pr ency Price
Prediction Using T ediction Using Tweet Volumes andolumes and Sentiment Analysis”. SMU Data Science

Authors Introduction
Prof. Tsutomu Ito
He is Associate Professor of the
Department of Business Administration
at National Institute of Technology, Ube
College, Japan. His current researchinterests include internet of things (IoT),
mechanicalengineering,artificial
intelligence (AI), automata theory,
quantitative analysis of Japanese
Keiretsu. Dr. Ito earned his doctor degree of Engineeringfrom Hiroshima University, Japan in 2018.
Mr. Kodai Hasebe
He is a master student at Department ofComputer
ScienceandSystemEngineering, University of Miyazaki.His
current research interests are image
processing, machine learning, and so on.

Mr. Fumito Hamakawa
He is a master student at Department ofComputer
ScienceandSystemEngineering, University of Miyazaki.
His current research interests arecomputer graphics, cellular automaton
simulation, and so on.

Mr. Bidesh Biswas Biki
He is master's student in Data Science at
Department of Statistics from TU
Dortmund. He is an enthusiast of Machine
Learning and Deep Learning. I have donesome research work and individual
projects on supervised learning especiallyon “Classification” and “Regression”
techniques. Also has done some work on Image Processing
and NLP.
Prof. Satoshi Ikeda
He received PhD degree from Hiroshima
University. He is an associate professor in
the Faculty of Engineering, University ofMiyazaki. His research interest includes
graph theory, probabilistic algorithm,
fractal geometry and measure theory.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
694

Powered by TCPDF (www.tcpdf.org)

Predicting High Volatility Cryptocurrency

Prof. Amane Takei
He is working as Associate Professor for
Department of Electrical and systems
Engineering, University of Miyazaki,
Japan. His research interest includes highperformance computing for computational
electromagnetism, iterative methods forthe solution of sparse linear systems, etc.
Prof. Takei is a member of IEEE, an expert advisor of
IEICE, a delegate of the Kyushu branch of IEEJ, a directorof JSST.
Prof. Makoto Sakamoto
He is presently a professor in the Facultyof Engineering, University of Miyazaki.
His first interests lay in hydrodynamics
and time series analysis, especially the
directional wave spectrum. He is atheoretical computer scientist, and his
current main research interests areautomata theory, languages, and computation. He is alsointerested in digital geometry, digital image processing,computer vision, etc.
Mr. Md Riajulii
He is master's student in Data Science at
Department of Statistics from TU
Dortmund. He is an enthusiast of Machine
Learning and Deep Learning. He has donesome research work and individual
projects on supervised learning especiallyon “Classification” and “Regression”
techniques. Also has done some work on Image Processing
and NLP.
Ms. Sabrina Bari Shital
She is a computer engineer with abackground in Computer Science andEngineering from Daffodil International
University. Her expertise encompasses
machine learning, deep learning, anddatabases. She is eager to contribute her
knowledge to the evolving landscape oftechnology.
Prof. Takao Ito
He is Professor of Management ofTechnology (MOT) in Graduate School ofEngineering at Hiroshima University. His
current research interests include
automata theory, artificial intelligence,systems control, quantitative analysis ofinterfirm relationships using graph theory,and engineering approach of organizational structuresusing complex systems theory.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
695

Simulation of Weathering Representation using Vertex and UV Information
Tsutomu Ito

National Institute of Technology, Ube College, Japan
Fumito Hamakawa, Kodai Hasebe

Graduate School of Engineering, University of Miyazaki, Japan
Satoshi Ikeda, Amane Takei, Makoto Sakamoto

Faculty of Engineering, University of Miyazaki, Japan
Takao Ito*

Hiroshima University, Japan＊

Corresponding Author E-mail: itotakao@hiroshima-u.ac.jp>

Abstract
In recent years, three-dimensional computer graphics (3DCG) technology has been developed. In addition, muchresearch has been done on weathering representations such as rust and moss for realistic representation. However,when outputting simulation results on an image, a large number of images are required to display different simulation
results in a three-dimensional space. In this paper, a simulation method using vertex information of a 3D model andsimple images is proposed. In this method, when the number of vertices is sufficient, the simulation result output
does not use images, thus reducing the data increase. This figure is one of the experimental results.
Keywords: Computer graphics, Vertices, Rust, Weathering, Visual simulation

To date, a plethora of research has been done on
weathering representations related to computer graphics.
However, most of them require the simulation results tobe output as images for textures. However, whensimulations are performed on multiple models, images
are required for the number of models, resulting in a large
amount of data. With the aim of being flexible enough todeal with existing models and real-time representations,a simulation method using vertex information and UVs
of 3DCG models was developed in this study.
In this study, the UV map used for the material
representation of the original 3DCG model is called thebase UV map, and the UV map for simulation is called
the simulation UV map.

1. Introduction
In recent years, three-dimensional computer graphics(3DCG) have been used in films, games, VR-based
education, and medicine. In this context, there is a needto make 3DCG look more realistic. However, it is not
subject to interferences such as environment or time andtherefore does not degrade as in reality. This is why
research into weathering expressions for CG is being
conducted to this day.
This study proposes a method to implement
weathering representations such as rust and moss using
vertex information and UVs, instead of outputting
simulation results as images for textures. The simulation
results were also applied to a 3DCG model in practice.
2. Research Background©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
696

Tsutomu Ito, Fumito Hamakawa, Kodai Hasebe, Satoshi Ikeda, Amane Takei, Makoto Sakamoto, Takao Ito

3. Development Environment

4.3. Dynamic environment acquisition

The development environment is shown in Table 1,with C# used as the programming language for Unity andPython for Blender.
Table 1. Development Environment.

Most of the previous studies assume a situation where
the object is in the same environment and in the same
shape at every step. However, some objects may change
dynamically, such as changing orientation over time or
increasing obstacles. Therefore, in this study, byincorporating vertex information into the local transition
rules, real-time changes such as changes in orientation
and up/down can be managed.

OS
Software

Windows10
Blender 3.6
Unity 2022.3.14f1
Visual Studio 2019
Adobe Illustrator 2023

4.

4.4. UV-based erosion representation
Determine UV coordinates from the degree of rust andmoss erosion at the vertices using the adjacency list. The
color of the eroded object is also determined by the user
entering a gradient image. In addition, as multiple UV
maps are used, they can be used in conjunction with thebase colors of existing 3DCG models as shown in Fig. 2
and Fig. 3.

Simulation Method

4.1. Adjacency list
In this study, simulations are performed using vertex
information in 3DCG. Vertex adjacency information was
obtained, and an adjacency list was created to represent
the phenomenon of rust and moss propagating locally inweathering simulations. The method for creating theneighborhood list used vertex information constituting
polygons and edges, although the method of searchingfor neighborhood information differs from software tosoftware. The adjacency list of vertexes v_i at index
number i is Fig. 1.

Fig. 2. Base color.

Fig. 3. Initial condition.

5. Experimental Results
This paper describes some of the experimental results,
mainly related to rust formation. The transition
conditions for rust generation are as follows, based on
previous studies [1], [2], [3].
Rust progression from 0 to the maximum rust value
𝑅𝑀𝐴𝑋 determined by the user.
Rust progression is assigned to each vertex.
Vertex 𝑣 as the initial rust occurrence has a low
probability of rust progression 𝑅𝑣 increases by 1.

Vertex adjacent to a vertex with progression greaterthan 1 has its 𝑅𝑣 increased by 1 by a function thattakes gravity and probability into account.

Vertexes above a certain level of progression arepeeled off.

UV placement of vertex 𝑣 at index number 𝑖 in step𝑡 is determined by 𝑅𝑣𝑡 /𝑅𝑀𝐴𝑋

Fig. 1. Adjacency list at vertex.
4.2. Initial condition setting
The susceptibility to rust and moss varies with theenvironment, such as humidity and sunlight, but also withthe material of the object, just as there is a difference inthe probability of rust between metal and plastic. Few
previous studies have considered the material of theobject and conducted simulations, and most of themassume that the entire object is made of the same material.
In this study, the user can control the erosion probabilityof the initial state like a mask image by painting directly
using the base UV map.

𝑖

In this experiment, simulated UVs are applied to themask textures in Fig. 4 and the color textures in Fig. 5.
The mask texture is a grayscale image and can be used toexpress textures such as the metallic feel and roughnessof the surface of the 3DCG model by inputting thesimulation results into software shaders. Besides, it can©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
697

Simulation of Weathering Representation

also be used to control the range of application of noise
textures to represent uneven and rusty surfaces or thebase color of an existing 3DCG model. Color textures
represent color scales. In addition, alpha values can beset, and the expression of peeling and holes above acertain value is implemented.

Fig. 4. Mask texture.

propagates in the left front direction because the object istilted in the left front direction. However, in thisexperiment, the rust propagated in the right-mostdirection as the object was tilted in the right-mostdirection as illustrated in Fig. 10 and Fig. 11.

Fig. 10. Mask result indifferent shapes.

Fig. 5. Color texture.

Fig. 11. Color result indifferent shapes.

5.1. Base experimental

5.4. Initial condition

In this experiment, simulations were performed on a
cube that was tilted. The results, shown in Fig. 6 and Fig.
7, show rust propagating according to gravity. In addition,
the strongly eroded areas have holes and peeling. The
rust-free area in Fig. 6 has no surface irregularity due tothe noise texture in Fig. 7, indicating that the mask
texture controls the texture of the 3DCG model.

In this experiment, the objects and initial conditions
were changed and simulated. The initial state is shown in
Fig. 3. Fig. 12 and Fig. 13 show that rust only occurs in
the most rust-prone white area in the initial state, and norust occurs in the most rust-prone black area.

Fig. 12. Mask result in
another object.
Fig. 6. Ex1 Mask result.

Fig. 7. Ex1 Color result.

6. Consideration

5.2. Verification of randomness

From 5.1 and 5.2, it was shown that rust propagates inthe direction of gravity depending on probability as inprevious studies, and that the location of rust and the
direction of rust propagation change depending on
probability. This indicates that the same mask texture andcolor texture can output different patterns of results forthe same object of the same shape existing at the sametime. This means that the simulation results do not need
to be output to an image, which may reduce the amountof data when multiple patterns of weathering
representation are desired for the same object. In
addition, since the simulation results are the erosion
degree for each vertex, they can be saved by outputting
them to a CSV file or other format.
From 5.3, it was shown that the simulation results canbe output in response to tilt even when the object shapeis changed. This is because the vertex information is

In this experiment, the same 3DCG model, the sametransition rules, and the same textures as in 5.1 were usedin the simulation. The results in Fig. 8 and Fig. 9 differ
from those in 5.1, indicating that the same image and thesame model can output different patterns of results
depending on the probability.

Fig. 8. Ex2 Mask result.

Fig. 13. Color result in
another object.

Fig. 9. Ex2 Color result.

5.3. Shape change
In this experiment, only the tilt of the object waschanged from 5.1 to the simulation. 5.1 shows that rust©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
698

Tsutomu Ito, Fumito Hamakawa, Kodai Hasebe, Satoshi Ikeda, Amane Takei, Makoto Sakamoto, Takao Itoreferenced in the transition rule, and it can respond tochanges in the tilt and size of the object during
simulation. In addition to vertex position information,vertex distance and the number of neighbors can also bereferenced. However, if the neighbor information
changes, such as when a vertex is deleted or added, it isnecessary to reacquire the neighbor information.
From 5.4, it is shown that it is possible to limit the
impact of the simulation by setting initial conditions.
This allows the base UV map and simple painting tocontrol the strength of the influence of different metals
and materials, without the In this study, only rust was
mentioned, but by changing the texture and transition
rules for each, the propagation of moss can be
represented as shown in Fig. 14.

[2]

[3]

次大会(in Japanese), 21C-2, 2 pages (2017).
田邉竜馬ほか,”本錆と副錆の挙動を考慮した錆の経
年 変 化 テ ク ス チ ャ 生 成”, ITE Technical Report(inJapanese), Vol.38, No.16, pp.95-98 (2014).
金澤功尚ほか,”オブジェクトの形状を考慮した錆によるエイジング画像シミュレーション”,画像電子学
会誌 Vol.46, No.4, pp.547-558 (in Japanese) (2017).

Authors Introduction
Prof. Tsutomu Ito
He is Associate Professor of the
Department of Business Administration
at National Institute of Technology, Ube
College, Japan. His current researchinterests include internet of things (IoT),
mechanicalengineering,artificial
intelligence (AI), automata theory,
quantitative analysis of Japanese
Keiretsu. Dr. Ito earned his doctor degree of Engineeringfrom Hiroshima University, Japan in 2018.
Mr. Fumito Hamakawa
He is a master student at Department ofComputer Science and System
Engineering, University of Miyazaki.
His current research interests arecomputer graphics, cellular automaton
simulation, and so on.

Fig. 14. Simulation results assuming moss in anotherobject.

7. Conclusion

Mr. Kodai Hasebe
He is a master student at Department ofComputer
ScienceandSystemEngineering, University of Miyazaki.
His current research interests are image
processing, machine learning, and so on.

In this study, we performed weathering simulations
using vertex adjacency and UV information. As in theprevious study, expressions such as rust and moss
propagation could be made without outputting them tothe texture. However, the number of vertices needs to be
increased for more realistic expressions and detailed
patterns. Therefore, if real-time changes are to beobserved, it is necessary to optimize the number ofvertices and to limit the number of objects to be simulated
at the same time.
For future works, it is necessary to pursue theimprovement of transition rules and the fields in which
this vertex technology can be utilized.

Prof. Satoshi Ikeda
He received PhD degree from Hiroshima
University. He is an associate professor in
the Faculty of Engineering, University ofMiyazaki. His research interest includes
graph theory, probabilistic algorithm,
fractal geometry and measure theory.

References[1]

鈴木雄太ほか,”3D オートマトンを用いた鉄の腐食による形状変化”,2017 年映像情報メディア学会年©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
699

Powered by TCPDF (www.tcpdf.org)

Simulation of Weathering Representation

Prof. Amane Takei
He is working as Associate Professor for
Department of Electrical and systems
Engineering, University of Miyazaki,
Japan. His research interest includes highperformancecomputingforcomputational
electromagnetism,
iterative methods for the solution ofsparselinearsystems,domaindecomposition methods for large-scale problems. Prof.
Takei is a member of IEEE, an expert advisor of IEICE, adelegate of the Kyushu branch of IEEJ a director of JSST.
Prof. Makoto Sakamoto
He is presently a professor in the Facultyof Engineering, University of Miyazaki.
His first interests lay in hydrodynamics
and time series analysis, especially the
directional wave spectrum. He is atheoretical computer scientist, and his
current main research interests areautomata theory, languages, and computation. He is alsointerested in digital geometry, digital image processing,computer vision, computer graphics, virtual reality, etc.
Prof. Takao Ito
He is Professor of Management ofTechnology (MOT) in Graduate Schoolof Engineering at Hiroshima University.
His current research interests include
automata theory, artificial intelligence,systems control, quantitative analysis ofinterfirm relationships using graphtheory, and engineering approach of organizational
structures using complex systems theory.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
700

Investigating the Engineering Interventions in the Conservation of Malaysia Heritage
Structures: A Review on Preserving Historical Edifices Through Advanced Civil Engineering
Techniques.
Muhammad Azizi Azizan

Faculty of Civil Engineering & Technology, Universiti Malaysia Perlis, Perlis, Malaysia
Centre of Excellence for Unmanned Aerial System, Universiti Malaysia Perlis, Perlis, Malaysia
Nurfadzillah Ishak

Faculty of Civil Engineering & Technology, Universiti Malaysia Perlis, Perlis, Malaysia
Centre of Excellence for Unmanned Aerial System, Universiti Malaysia Perlis, Perlis, Malaysia
Hazry Desa

Faculty of Electrical Engineering & Technology, Universiti Malaysia Perlis, Perlis, Malaysia
Centre of Excellence for Unmanned Aerial System, Universiti Malaysia Perlis, Perlis, Malaysia
E-mail: aziziazizan@unimap.edu.my

Abstract
This review delves into the realm of Malaysia's heritage conservation, spotlighting the transformative impact ofadvanced civil engineering techniques. Through the integration of Fiber-Reinforced Polymers (FRP) andnanotechnology, historical edifices are fortified, seamlessly blending modern engineering with architectural elegance.
Non-destructive testing (NDT) methods, including ground-penetrating radar and Finite Element Analysis (FEA),
empower conservationists with deep insights into structural intricacies, guiding targeted interventions. In the digital
sphere, 3D laser scanning captures intricate details, while Virtual Reality (VR) simulations facilitate immersive
exploration and informed decision-making. Beyond preservation, these technologies foster public engagement,
ensuring a collective understanding of Malaysia's cultural heritage. This harmonious fusion of tradition and cuttingedge engineering ensures the enduring legacy of Malaysia's architectural treasures.
Keywords: Heritage conservation, Non-destructive testing (NDT), Finite Element Analysis (FEA), 3D laser scanning,
Virtual Reality (VR)1.

Introduction

research endeavors to offer valuable insights into thesustainable preservation of our architectural heritage,ensuring that these treasures endure for generations tocome. In this pursuit, study will navigate the intersectionof history, culture, and engineering prowess, striving to
strike an intricate balance between preservation
imperatives and contemporary demands [5].

Heritage structures stand as invaluable remnants of ourcultural legacy, embodying the architecturalachievements and historical significance of bygone eras[1]. However, the preservation of these structures poses
intricate challenges that demand innovative engineeringsolutions [2], [14]. As urban landscapes evolve andmodernization accelerates, the need to safeguard these
historical edifices has never been more pressing [3]. This
review embarks on a comprehensive exploration of theengineering interventions essential in the conservation ofheritage structures, delving into the intricate
amalgamation of tradition and technology [4], [6].
Through a meticulous analysis of advanced civilengineering techniques, this study aims to shed light on
the nuanced methods employed to preserve the structural
integrity, cultural heritage, and aesthetic essence of thesebuildings. By examining the successes, challenges, andlessons learned from previous conservation projects, this2.

Preserving Historical Edifices in Malaysia

Conducting a thorough exploration of Malaysia's richhistorical and cultural heritage is vital. This involves
studying the evolution of architectural styles andunderstanding the cultural significance of historicaledifices within Malaysian society [6], [23]. The
architectural diversity in Malaysia, influenced by Malay,
Chinese, Indian, and colonial elements, forms a critical
backdrop for conservation efforts [7]. Malaysia'sarchitectural heritage is a tapestry woven from centuriesof cultural interactions and influences. The evolution of©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
701

Muhammad Azizi Azizan, Nurfadzillah Ishak, Hazry Desaarchitectural styles in Malaysia reflects the rich history of
the nation, characterized by the convergence of Malay,
Chinese, Indian, and colonial elements. Traditional
Malay architecture, exemplified by intricate wooden
carvings and steeply pitched roofs, showcases indigenous
craftsmanship [8], [19]. The Chinese influence is evident
in the intricate details of temples and clan houses,emphasizing symmetry and ornate decorations [9].
Indian architectural traditions, characterized by vibrantcolors and sculpted facades, have left an indelible markon religious structures across the country [10], [26].
Additionally, the colonial era introduced a blend ofEuropean styles, seen in buildings with ornate facades,
arched windows, and grand pillars, reflecting the British,
Dutch, and Portuguese colonial influences [11].

damaging the building fabric [14]. Ground-penetratingradar (GPR) operates on the principle of emitting highfrequency radio waves into the structure. These waves
penetrate the materials and reflect back when
encountering boundaries between different materials orvoids. By analyzing the time taken for the signals toreturn and their strength, engineers can create detailed
subsurface images without invasive measures.
Finite Element Analysis (FEA) is another civilengineering computational methods such as FEA
provide detailed insights into stress distributions andstructural vulnerabilities, aiding in targetedinterventions [2], [15]. FEA tools enable engineers tovisualize the impact of interventions on the structure's
aesthetics. Conservationists can assess how alterations
might affect the visual aspects of the edifice, ensuringthat preservation efforts align with historicalsignificance and architectural beauty. FEA can validate
restoration efforts. By comparing the digital model ofthe restored structure with historical data,conservationists can ensure that the restoration
accurately reflects the original design, maintaining thestructure's historical authenticity [16]. Incorporating
Finite Element Analysis into heritage conservation
efforts empowers engineers and conservationists withdetailed, data-driven insights. It enables targeted,efficient interventions while preserving the historicaland aesthetic essence of these invaluable structures.

Understanding the unique architectural amalgamation is
crucial in preserving historical edifices. Each cultural
influence represents a chapter in Malaysia's history,
encapsulating the nation's multicultural identity.
Preservation efforts must delicately balance these diverse
elements, ensuring that the architectural heritage reflects
the harmonious coexistence of cultures within Malaysiansociety [12]. The selection process involves collaboration
with local heritage organizations and experts. It's crucial
to choose diverse edifices, considering regional
representation and architectural styles. Identifying
overlooked structures that are significant to local
communities ensures a holistic approach to preservation[13].

3.2 Innovative Materials and Techniques
3.

Advanced Civil Engineering Techniques inConservations Work

Fiber-Reinforced Polymers (FRP) composites offer
lightweight and high-strength solutions for structural
reinforcement, preserving the original aesthetics [17].
Fiber-Reinforced Polymers (FRP) are composite
materials made of a polymer matrix reinforced withfibers such as glass, carbon, or aramid. These materials
offer exceptional strength-to-weight ratios anddurability, making them ideal for structural
reinforcement applications. FRP composites are used toreinforce historical structures without significantly
altering their original appearance. For instance, FRP
strips can be bonded to beams and columns, providing
additional strength without compromising the aesthetic
integrity [18]. In earthquake-prone regions, FRP
materials are employed to enhance the seismic
resilience of historical buildings. They can be
strategically placed to reinforce vulnerable areas,
ensuring the structure's stability during seismic events[19].

The preservation of historical edifices necessitates theintegration of advanced civil engineering techniques,
marking a paradigm shift in heritage conservation
practices. Traditional methods, while valuable, often
face limitations in addressing the complex challenges
posed by aging structures and evolving environmental
conditions. Advanced civil engineering techniques
encompass a spectrum of innovative approaches, fromcutting-edge materials science to sophisticatedcomputational modeling and non-destructive testing
methods. These techniques play a pivotal role in the
structural assessment, restoration, and long-termsustainability of historical buildings, ensuring thepreservation of cultural heritage for future generations.
3.1 Advanced Structural Assessment
Non-Destructive Testing (NDT) is a techniques likeground-penetrating radar (GPR) and ultrasonic testingenable engineers to assess internal structures without

Nanotechnology Applications Nanomaterials, such as
self-healing coatings, enhance the durability andresilience of historical structures against environmental©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
702

Investigating the Engineering Interventions

factors[20].
Nanotechnology
involves
themanipulation of materials at the nanoscale, where theirproperties exhibit unique characteristics. In heritage
conservation, nanomaterials are utilized for their ability
to enhance the durability, resilience, and protective
qualities of historical structures. Nanomaterials, such as
microcapsules containing healing agents, are embedded
in coatings applied to building surfaces. When the
structure experiences minor cracks due toenvironmental stresses, these capsules rupture,
releasing the healing agents and effectively repairing
the damage [21]. Hydrophobic nano coatings repel
water and prevent moisture infiltration, safeguarding
historical structures from water-related deterioration.
These coatings can be applied to various surfaces,
including stone, wood, and metal, preserving the
integrity of the building materials [22].

and assess their impact on the historical context [1].
Virtual Reality (VR) simulations immerse users incomputer-generated environments. Conservationists
use VR technologies to create virtual replicas ofhistorical structures, enabling interactive exploration
and analysis. Conservationists use VR simulations tovisualize proposed alterations or restoration efforts.
This immersive experience allows stakeholders toassess the visual impact of changes before
implementation, ensuring that modifications align withthe historical context below. VR simulations enhance
public engagement by allowing virtual tours ofhistorical sites. This interactive experience fosters
public interest, awareness, and appreciation for cultural
heritage, promoting advocacy for conservation efforts[28]. VR simulations serve as educational tools,
enabling students, architects, and conservationists tovirtually explore historical buildings.

The integration of FRP composites and nanotechnology
applications represents a forward-thinking approach in
heritageconservation,
combining
structural
reinforcement with advanced protective coatings.
These techniques not only strengthen historicalstructures but also contribute to their long-term
resilience against various environmental challenges,
ensuring the preservation of cultural heritage assets.

The integration of advanced civil engineering
techniques in heritage conservation signifies a
progressive approach toward preserving ourarchitectural legacy. By leveraging cutting-edge
methods and materials, conservationists can ensure thestructural integrity, authenticity, and longevity ofhistorical edifices, contributing significantly to thesafeguarding of cultural heritage for generations tocome. This evolving field continues to redefine theboundaries of conservation practices, marking a new
era in the intersection of engineering innovation andcultural preservation.

3.3 Preservation through Digital Technologies
3D Laser Scanning: High-resolution 3D scans aid in
documenting intricate architectural details, facilitating
accurate restoration [23]. 3D laser scanning involves
using lasers to create highly detailed, three-dimensional
representations of physical objects, including historicalbuildings. Laser scanners emit laser beams that bounce
off surfaces and return to the scanner. By measuring thetime, it takes for the laser to return, the scanner cancreate precise 3D models. 3D laser scanning captures
intricate architectural details, including ornate carvings,
moldings, and textures, with unparalleled accuracy.
This documentation serves as a digital record of thestructure, aiding in historical analysis and preservation
efforts [24]. Laser scanning provides precise
measurements, allowing conservationists to assess
dimensions, angles, and spatial relationships within thebuilding. These measurements are crucial for
restoration work, ensuring that new elements alignperfectly with the original design [25]. 3D models
generated through scanning enable virtual preservation.
Conservationists can digitally archive buildings,
allowing future generations to explore and study
historical structures in virtual environments [26].

4.

Conclusion

This comprehensive exploration into the conservationof Malaysia's rich heritage structures illuminates thepivotal role played by advanced civil engineering
techniques in preserving these historical gems. FiberReinforced Polymers (FRP) emerge as lightweight,
yet robust solutions, seamlessly merging modern
engineering with historical aesthetics. Collaboratingwith nanotechnology, self-healing coatings andhydrophobic nano coatings fortify structures against
environmental ravages, ensuring their endurance forposterity.
Non-destructive testing (NDT) methods like groundpenetrating radar (GPR) and ultrasonic testing,coupled with Finite Element Analysis (FEA), provide
profound insights into the structural intricacies ofhistorical edifices. These techniques empower
conservationists with precise stress distributions,
vulnerability
identifications,andpredictive
capabilities, guiding meticulous interventions that
preserve both the architectural integrity and historical

Virtual Reality (VR) Simulations: VR technologies
allow conservationists to visualize proposed changes©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
703

Muhammad Azizi Azizan, Nurfadzillah Ishak, Hazry Desa

[6]

Henderson, J. C. (2001). Conserving colonial heritage:
Raffles hotel in Singapore. International Journal of
Heritage
Studies, 7(1),
7-24.
DOI:
https://doi.org/10.1080/13527250119383[7] Lu, D. Architecture in the Age of Playfulness: Mappinga Framework for Global Historiography. In The
Routledge Companion to Contemporary Architectural
History (pp.
1-80).
Routledge.
DOI:
https://doi.org/10.4324/9781315674469[8] Rao, X., Zhou, J., Ding, K., Wang, J., Fu, J., & Zhu, Q.
(2022). Research on the cultural tracing of the patriarchal
clan system of traditional buildings in the Eastern
Zhejiang Province, China, based on space syntax: Thecase study of Huzhai in Shaoxing. Sustainability, 14(12),7247. DOI: https://doi.org/10.3390/su14127247[9] Khan, N. R. (2021). Art and architectural traditions ofIndia and Iran: Commonality and diversity. Routledge
India. DOI: https://doi.org/10.4324/9781003229421[10] Botti, G. (2023). Towards a Quantitative Turn: Frontiersof Architectural Historiography. In Designing Emerging
Markets: A Quantitative History of Architectural
Globalization (pp. 19-53). Singapore: Springer Nature
Singapore. DOI: https://doi.org/10.1007/978-981-991552-1_2[11] Rugkhapan, N. T. (2022). Between Toponymy andCartography: An Evolving Geography of Heritage in
George Town, Malaysia. In New Directions in Linguistic
Geography: Exploring Articulations of Space (pp. 327354). Singapore: Springer Nature Singapore. DOI:
https://doi.org/10.1007/978-981-19-3663-0_13[12] Berkes, F. (2007). Community-based conservation in a
globalized world. Proceedings of the National academyofsciences, 104(39),15188-15193.
DOI:
https://doi.org/10.1073/pnas.0702098104[13] El Masri, Y., & Rakha, T. (2020). A scoping review of
non-destructive testing (NDT) techniques in building
performance diagnostic inspections. Construction and
Building
Materials, 265,
120542.
DOI:
https://doi.org/10.1016/j.conbuildmat.2020.120542[14] Castellazzi, G., D’Altri, A. M., de Miranda, S., &Ubertini, F. (2017). An innovative numerical modelingstrategy for the structural analysis of historicalmonumental buildings. Engineering Structures, 132,229-248. Clark, S. (2021). FEA Validation in
Architectural Restoration. Journal of Architectural
Conservation,27(1),
45-58.
DOI:
https://doi.org/10.1016/j.engstruct.2016.11.032[15] Arias, P., Armesto, J., Di-Capua, D., González-Drigo, R.,
Lorenzo, H., & Perez-Gracia, V. (2007). Digital
photogrammetry, GPR and computational analysis of
structural damages in a mediaeval bridge. Engineering
FailureAnalysis, 14(8),1444-1457.
DOI:
https://doi.org/10.1016/j.engfailanal.2007.02.001[16] Sippach, T., Dahy, H., Uhlig, K., Grisin, B., Carosella,S., & Middendorf, P. (2020). Structural optimization
throughbiomimetic-inspired
material-specificapplication of plant-based natural fiber-reinforced
polymer composites (NFRP) for future sustainable
lightweight architecture. Polymers, 12(12), 3048. DOI:
https://doi.org/10.3390/polym12123048

resonance.
In the digital realm, 3D laser scanning meticulously
captures architectural nuances, laying the foundation
for precise restoration. Virtual Reality (VR)simulations then breathe life into historical spaces,
enabling immersive exploration and informed
decision-making. Beyond conservation, thesetechnologies foster public engagement, ensuring acollective understanding and appreciation of
Malaysia's cultural heritage.
As custodians of Malaysia's architectural legacy,
conservationists stand at the nexus of tradition andtechnology. Armed with these advanced tools, they
meticulously balance preservation imperatives andcontemporary demands. Through this synergy,
Malaysia's heritage structures not only withstand thetests of time but also continue to inspire, educate, andenrich future generations, ensuring the vibrantpreservation of the nation's cultural heritage.
5.

Acknowledgement

This research is fully supported by UniMAP
Commercialization grant, 9001-00751. The authors
fully acknowledged Universiti Malaysia Perlis for theapproved fund which makes this important research
viable and effective.
References
[1]

[2]

[3]

[4]

[5]

Volk, R., Stengel, J., & Schultmann, F. (2014). Building
Information Modeling (BIM) for existing buildings—
Literature review and future needs. Automation inconstruction,
38,
109-127.
DOI:
https://doi.org/10.1016/j.autcon.2013.10.023
Azpeitia Santander, A., Azkarate Garai-Olaun, A., & De
la Fuente Arana, A. (2018). Historic urban landscapes: A
review on trends and methodologies in the urban context
of the 21st century. Sustainability, 10(8), 2603.
DOI:https://doi.org/10.3390/su10082603
Ukabi, E. B., & Akçay, A. Ö. (2023). Conserving thehistorical identity of north Nicosia Walled City:
Exploring design approaches and implications from 1983
to2003. Buildings, 13(9),2199.
DOI:https://doi.org/10.3390/buildings13092199
Roger, J. A. (2022). Navigating the Currents andCountercurrents of Southern New Zealand’s
Human/River Relationship: An Environmental History ofRivers in Otago and Southland, 1890-1920 (Doctoral
dissertation,
UniversityofOtago).
DOI:
https://doi.org/10.9734/ajrcos/2023/v16i4386
Henderson, J. C., Liew, G., Ong, J., & Quek, B. (2013).
The use of urban built heritage for boutique hotels:
Examples from Malaysia and Singapore. Tourism
Review
International,
17(2),
63-74.
DOI:
https://doi.org/10.3727/154427213X13728688260758©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
704

Powered by TCPDF (www.tcpdf.org)

Investigating the Engineering Interventions[17] Hollaway, L. C. (2010). A review of the present andfuture utilisation of FRP composites in the civil
infrastructure with reference to their important in-service
properties. Construction and building materials, 24(12),2419-2445.
DOI: https://doi.org/10.1016/j.conbuildmat.2010.04.062[18] Freddi, F., Galasso, C., Cremen, G., Dall’Asta, A., Di
Sarno, L., Giaralis, A., ... & Woo, G. (2021). Innovations
in earthquake risk reduction for resilience: Recent
advances and challenges. International Journal ofDisaster
Risk
Reduction, 60,
102267.
DOI:
https://doi.org/10.1016/j.ijdrr.2021.102267[19] Idumah, C. I., Obele, C. M., Emmanuel, E. O., & Hassan,
A. (2020). Recently emerging nanotechnologicaladvancements in polymer nanocomposite coatings foranti-corrosion, anti-fouling and self-healing. Surfaces
andInterfaces, 21,
100734.
DOI:https://doi.org/10.1016/j.surfin.2020.100734[20] Zhu, D. Y., Rong, M. Z., & Zhang, M. Q. (2015). Selfhealing polymeric materials based on microencapsulated
healing agents: From design to preparation. Progress in
Polymer
Science, 49,
175-220.
DOI:https://doi.org/10.1016/j.progpolymsci.2015.07.00
2[21] Ruffolo, S. A., & La Russa, M. F. (2019). Nanostructured
coatings for stone protection: An overview. Frontiers inMaterials, 6,
147.
DOI:
https://doi.org/10.3389/fmats.2019.00147[22] Xu, Z., Wu, L., Shen, Y., Li, F., Wang, Q., & Wang, R.
(2014). Tridimensional reconstruction applied to cultural
heritage with the use of camera-equipped UAV andterrestrial laser scanner. Remote sensing, 6(11), 1041310434. DOI:https://doi.org/10.3390/rs61110413[23] Vileikis, O., Cesaro, G., Santana Quintero, M., Van
Balen, K., Paolini, A., & Vafadari, A. (2012).

Documentation in world heritage conservation: Towards
managing and mitigating change–The case studies ofPetra and the Silk Roads. Journal of Cultural Heritage
Management and Sustainable Development, 2(2), 130152. DOI:https://doi.org/10.1108/20441261211273635[24] Muchiri, P., Pintelon, L., Gelders, L., & Martin, H.
(2011). Development of maintenance function
performancemeasurement
frameworkandindicators. International
Journalof Production
Economics, 131(1),295-302.
DOI:https://doi.org/10.1016/j.ijpe.2010.04.039[25] Trillo, C., Aburamadan, R., Mubaideen, S., Salameen, D.,& Makore, B. C. N. (2020). Towards a systematic
approach to digital technologies for heritage
conservation. Insights from Jordan. Preservation,
Digital Technology & Culture, 49(4), 121-138. DOI:
https://doi.org/10.1016/j.ijpe.2010.04.039[26] Chandler, T., Richards, A. E., Jenny, B., Dickson, F.,
Huang, J., Klippel, A., ... & Prober, S. M. (2022).
Immersive landscapes: modelling ecosystem reference
conditions in virtual reality. Landscape Ecology, 1-17.
DOI: https://doi.org/10.1007/s10980-021-01313-8[27] Lefebvre, M., Espinosa, M., Gomez y Paloma, S.,
Paracchini, M. L., Piorr, A., & Zasada, I. (2015).
Agricultural landscapes as multi-scale public good andthe role of the Common Agricultural Policy. Journal ofEnvironmental Planning and Management, 58(12), 20882112. DOI: https://doi.org/10.1111/conl.12398[28] Najafi, P., Mohammadi, M., van Wesemael, P., & Le
Blanc, P. M. (2023). A user-centred virtual city
information model for inclusive community design:
State-of-art. Cities, 134,
104203.
DOI:
https://doi.org/10.1016/j.cities.2023.104203

Authors Introduction
Dr. Muhammad Azizi Bin Azizan
He received his PhD in Civil
Engineering from the Universiti
Malaysia Perlis. He is currently a
Senior Lecturer in the same institution.
He is Head of Project Integration &Management (PIM) at Centre of
Excellence for Unmanned Aerial
System (COEUAS), Universiti Malaysia Perlis.

Prof. Dr. Hazry Bin Desa
He obtained his PhD in Materials Scienceand Production Engineering (Robotics)
from Oita University and currently holds
the position as a Head at the Centre of
Excellence for Unmanned Aerial Systems(COE-UAS) at Universiti Malaysia Perlis(UniMAP).

Dr. Nurfadzillah Binti Ishak
She received her PhD in Building
Engineering from the Universiti
Malaysia Perlis. She is currently a SeniorLecturer in the same institution. She is
Head of Built Environment Intelligent(BELL) at Centre of Excellence for
Unmanned Aerial System (COEUAS),
Universiti Malaysia Perlis.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
705

Drones and Data: A Comprehensive Exploration of UAVs in Data Mining
Muhammad Azizi Azizan

Faculty of Civil Engineering & Technology, Universiti Malaysia Perlis, Perlis, Malaysia
Centre of Excellence for Unmanned Aerial System, Universiti Malaysia Perlis, Perlis, Malaysia
Nurfadzillah Ishak

Faculty of Civil Engineering & Technology, Universiti Malaysia Perlis, Perlis, Malaysia
Centre of Excellence for Unmanned Aerial System, Universiti Malaysia Perlis, Perlis, Malaysia
Hazry Desa

Faculty of Electrical Engineering & Technology, Universiti Malaysia Perlis, Perlis, Malaysia
Centre of Excellence for Unmanned Aerial System, Universiti Malaysia Perlis, Perlis, Malaysia
E-mail: aziziazizan@unimap.edu.my, nurfadzillah@unimap.edu.my, hazry@unimap.edu.my

Abstract
Encapsulates a comprehensive investigation into the symbiotic relationship between Unmanned Aerial Vehicles(UAVs) and data mining, as encapsulated in the discourse titled "Drones and Data." This exploration delves into themultifaceted applications and transformative impact of UAV technology within the data mining landscape. The
examination begins by elucidating the pivotal role of UAVs, highlighting their mobility, accessibility, and capability
to collect data from challenging or remote environments. As technology evolves, UAVs have emerged as versatile
platforms that redefine the data is collected and analysed across diverse sectors. The narrative unfolds through distinct
dimensions, encompassing precision agriculture, environmental monitoring, infrastructure inspection, mining andexploration, disaster response, and urban planning. The technological transitions facilitated by UAVs, emphasizing
the integration of machine learning algorithms, cloud-based data processing, and the collaborative synergy between
stakeholders. These advancements position UAVs as transformative tools that not only enhance the efficiency of
information acquisition but also open avenues for innovative solutions and insights. Therefore, this study a glimpse
into the intricate web of applications and technological advancements at the intersection of UAVs and data mining.
It serves as a scholarly guide, navigating the reader through the evolving landscape of "Drones and Data," UAVs playa central role in unlocking unprecedented insights and efficiencies, reshaping the future of data mining.
Keywords: Acceptance, Readiness; Unmanned Aerial Vehicle; Data Mining

1.

Introduction

2. Unmanned Advantages of UAVs in Data Mining
2.1. Mobility

In recent years, the integration of Unmanned Aerial
Vehicles (UAVs) in data mining applications has
witnessed a remarkable surge, reshaping the landscapeof information acquisition and analysis. UAVs,
commonly known as drones, have become pivotal tools
in various industries, offering unique advantages thatpropel the field of data mining into new frontiers [1].
As technology continues to advance, UAVs haveemerged as versatile platforms capable of transforming
how we collect and analyze data. Their applications
span a wide spectrum, from environmental monitoring
and agriculture to infrastructure inspection and disastermanagement. In the realm of data mining, UAVs play a
crucial role in augmenting our ability to gather
information with unprecedented efficiency andaccuracy [2].

UAVs provide unparalleled mobility, enabling accessto areas that are challenging or impossible for traditional
data collection methods. Whether navigating dense
forests, inspecting infrastructure, or monitoring vast
agricultural landscapes, UAVs offer a dynamic andflexible approach to data acquisition [3].
2.2. Accessibility
The ability of UAVs to access hard-to-reach or remotelocations is a game-changer in data mining. They
overcome geographical constraints and offer a costeffective alternative to manned missions, allowing for
more frequent and widespread data collection [4].
2.3. Data Collection in Challenging Environments
UAVs excel in collecting data from environments thatpose logistical or safety challenges. From rugged©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
706

Muhammad Azizi Azizan, Nurfadzillah Ishak, Hazry Desa

terrains to disaster-stricken areas, UAVs can swiftly andefficiently capture valuable information, contributing toa more comprehensive understanding of diverse
landscapes [5].

Table 1. Regulatory Environment
Regulatory Environment
AcceptanceReadinessThewillingnessofThe development ofregulatory bodies to adapt
regulatory
frameworks
andestablish
clear
that balance innovation
guidelines for UAV usage
with safety, fostering a
indataminingconducive environment
applications [9].
for UAV adoption [9].

2.4. High-Resolution Imaging
Equipped with advanced imaging technologies, UAVs
capture high-resolution aerial imagery, facilitating
detailed spatial analysis. This capability is particularly
beneficial in applications such as land-use mapping,environmental modelling, and infrastructure planning[6].

Table 2. Technological Infrastructure
Technological Infrastructure
AcceptanceReadiness
Embracing
UAV
Investment
inthetechnology requires anecessary
hardware,robusttechnological
software,andinfrastructure capable ofcommunication systems
supporting data-intensive
to accommodate theprocesses [10].
seamless integration of
UAVs into data mining
workflows [11].

2.5. Real-time Monitoring
UAVs enable real-time data acquisition andmonitoring, allowing for quick decision-making in
dynamic situations. This responsiveness is crucial inapplications like disaster response, where timely
information can save lives and mitigate the impact of
emergencies [7].

Table 3. Stakeholder Awareness and Education
Stakeholder Awareness and Education
AcceptanceReadiness
Recognition
andEducational
initiatives
understanding of theaimed
atraisingbenefits of UAVs in dataawareness about UAV
miningamongcapabilities, data security,
stakeholders, including
and the potential impact
industry
professionals,
on various sectors [13].
policymakers, and the
public [12].

The synergy between UAVs and data mining opens
up new avenues for exploration and discovery. This
introduction sets the stage for a comprehensive
examination of how UAVs are reshaping the field,
emphasizing their mobility, accessibility, andprowess in collecting data from challenging or
remote environments. As a delve deeper into thesubsequent sections, study will unravel themultifaceted applications and the transformative
impact of UAVs in the realm of data mining [8].
3.

Table 4. Cost Considerations
Cost Considerations
AcceptanceReadiness
Recognition of the costFinancial preparedness toeffectiveness of UAV
investin
UAV
technology compared toinfrastructure, training,
traditional data collection
andmaintenance,
methods [14].
considering the long-term
benefits and efficiency
gains [15].

UAV Readiness Index (URI) for Data
Mining: Navigating the TechnologicalLandscapeThe integration of Unmanned Aerial Vehicles(UAVs) into data mining practices is contingent on theacceptance and readiness of stakeholders withindiverse industries. This section introduces a UAV
Acceptance and Readiness Index, exploring thefactors influencing the adoption of UAV technology
for data mining and the preparedness of various
sectors to embrace these advancements.

Table 5. Data Security and Privacy
Data Security and Privacy
AcceptanceReadiness
Acknowledgment of the
Implementation of robust
importanceofdata
security
measures,security and privacy inencryption protocols, andUAV-enableddata
adherence to privacy
mining applications [16].
regulations to addressconcerns associated withdata collection from aerial
platforms [17].

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
707

Drones and Data:

Table 6. Industry Collaboration and Standards
Industry Collaboration and Standards
AcceptanceReadiness
Willingness of industries
Development
andto collaborate and share
adherence to industry
best practices for UAV
standards that streamline
integration into data
UAV adoption, ensuring
mining workflows [18].
interoperability
andconsistency
acrossapplications [19].

Meanwhile, the internal consistency of datareliability in this research was verified by using the
Cronbach’s Alpha coefficients through the pilot
survey, where 30 usable responses from the pilot
survey. It was found that the questionnairecoefficient of Cronbach’s alpha was more than 0.7
which was deemed as reliable for quantitative datacollection. In order to distribute the final survey, asimple random sampling was applied in this researchas the population sample is known. In this research,
the targeted population is 70 organizations fromgovernment agencies and development firms in
Kuala Lumpur. Kuala Lumpur has a high number ofdevelopments of township and is considered as apositively data mining active location in Malaysia. A
total of 56 required samples were successfully
obtained and usable for analysis. Data collected wereanalysed using IBM SPSS Statistical Software which
descriptive analysis has been employed in order toachieve the research objective.

Table 7. Training and Skill Development
Training and Skill Development
AcceptanceReadiness
Recognition of the need
Implementationoffor specialized training
training initiatives to build
programstoequip
a workforce capable ofprofessionals with theoperating, maintaining,skills required for UAVand extracting valuableassisted data mining [20].
insights from UAVcollected data [21].
Table 8. Public Perception and Acceptance
Public Perception and AcceptanceAcceptanceReadiness
Public perception andPublicawareness
acceptanceof
UAV
campaigns
tofostertechnology, considering
understanding
andfactors such as noise,
acceptance of UAVs,
visualimpact,andaddressing concerns andpotential disruptions [22].
showcasing the societal
benefits of UAV-enableddata mining [23].

5. Results and Discussion
5.1. The level of the acceptance and readiness ofUnmanned Aerial Vehicle implementation in data
mining procedures
Table 9 shows the means and standard deviation
based on each variable which is for capabilities factorof Unmanned Aerial Vehicle technology. The variablesconsist of Regulatory Environment (mean=4.60,
sd.=0.34), Technological Infrastructure (mean=4.47,
sd.=0.44), Stakeholder Awareness and Education(mean=4.43,
sd.=0.44),
Cost
Considerations(mean=4.41, sd.=0.47) and Data Security and Privacy(mean=4.38, sd.=0.52). Furthermore, for the readinesselements of implementation Unmanned Aerial Vehicle
technology are Industry Collaboration and Standards(mean=3.56,
sd.=0.44),
Training
andSkill
Development(mean=3.24,
sd.=1.38),
PublicPerception and Acceptance (mean=3.06, sd.=0.95). The
mean value score interpretation of the variables used inthis study were be interpreted.

The UAV Acceptance and Readiness Index
provides a comprehensive framework for assessing
the feasibility of integrating UAV technology into
data mining practices. As navigate through thesubsequent sections, each factor in this index will be
explored in detail, shedding light on the evolving
landscape of acceptance and readiness for UAVenabled data mining across various sectors.
4. Methods
A quantitative approach was employed in thisresearch. A self-administered questionnaire withclose-ended questions using a five-point Likert scaleranging. The final questionnaire has been through thepreliminary tests which consist of validity andreliability analysis. Content validity is applied byinviting a group of five experts to be selected to pretest and to review the set of questionnaires. They
consist of two academicians working in universities
as lecturers and researchers in the area of
construction and town planning and threepractitioners or project managers who haveexperience in managing development projects.

Table 9. Descriptive Analysis
Variables
Acceptance
Factors
Regulatory
Environment
Technological
Infrastructure

Mean

Std.
Deviation

4.60

0.34

4.47

0.44

Mean Value
Score
Interpretation©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
708

High
High

Muhammad Azizi Azizan, Nurfadzillah Ishak, Hazry Desa

The journey doesn't end here but propels us forward,
navigating the future where UAVs play a central rolein unlocking unprecedented insights and efficiencies
in the world of data mining.

Stakeholder
4.43
0.44
Awareness andHigh
Education
Cost
4.41
0.47
High
Considerations
Data Security and
4.38
0.52
High
Privacy
Readiness
Elements
Industry
3.24
1.38
Moderate
Collaboration andStandards
Training and Skill
3.06
0.95
Moderate
Development
Public Perception
3.12
1.37
Moderateand Acceptance
Note: The mean value categorized into three levels: low =1.00 to 2.66; moderate = 2.67 to 3.33; and high = 3.34 to
5.00.

7.

This work was funded by Universiti Malaysia
Perlis (UniMAP) under the Commercialization grant,
9001-00751.
References
1.

2.

Based on the result illustrated in the Table 9, it canbe concluded that in navigating the UAV Acceptanceand Readiness Index for Data Mining, our
exploration has unveiled critical dimensions shaping
the trajectory of UAV technology adoption within
the data mining landscape. The confluence ofregulatory adaptability, technological infrastructure,
stakeholder education, cost considerations, industry
collaboration, and public perception forms acomplex web influencing the acceptance andreadiness for UAVs. The journey through this index
demonstrates a forward momentum, wherein
regulatory bodies are increasingly adapting to UAV
applications, fostering an environment conducive toinnovation while ensuring safety and compliance.
Technological infrastructure investments have paved
the way for seamless UAV integration, with robust
hardware, software, and communication systems
supporting data-intensive processes. Stakeholder
awareness and education emerge as pillars ofknowledge empowerment, fostering a deeper
understanding of UAV benefits among professionals,
policymakers, and the public. Educational initiatives
play a crucial role in dispelling concerns, promoting
transparency,
andpositioning
UAVs
astransformative tools in data mining.
6.

Acknowledgement

3.

4.

5.

6.

7.

8.

9.

Conclusions
10.

As summarised, the UAV Acceptance andReadiness Index for Data Mining represents aholistic transition toward embracing UAVs as
integral components of data mining workflows. As
stakeholders align their strategies with thedimensions outlined in this index, the landscape ofdata mining is poised for transformative change [24].

11.

h. Yao, R. Qin, And X. Chen, Unmanned Aerial
Vehicle for Remote Sensing Applications—A
Review, Remote Sens. (Basel), Vol. 11(12), 2019,
pp. 1443.
n. Bayomi and J. E. Fernandez, Eyes in The Sky:
Drones Applications in The Built Environment
Under Climate Change Challenges, Drones, Vol.
7(10), 2023, pp. 637.
d. Giordan Et Al., The Use of Unmanned Aerial
Vehicles (UAVs) For Engineering Geology
Applications, Bull. Eng. Geol. Environ., Vol. 79(7),2020, pp. 3437–3481.
z. Zhang and L. Zhu, A Review on Unmanned
Aerial Vehicle Remote Sensing: Platforms, Sensors,
Data Processing Methods, And Applications,
Drones, Vol. 7(6), 2023, pp. 398.
w. Alawad, N. B. Halima, And L. Aziz, An
Unmanned Aerial Vehicle (UAV) System forDisaster and Crisis Management in Smart Cities,
Electronics (Basel), Vol. 12(4), 2023, pp. 1051.
h.-W. Choi, H.-J. Kim, S.-K. Kim, And W. S. Na, An
Overview of Drone Applications in The
Construction Industry, Drones, Vol. 7(8), 2023, pp.
515.
l. Gao and Z. Liu, Unraveling the Multifaceted
Nexus of Artificial Intelligence Sports and User
Willingness: A Focus on Technology Readiness,
Perceived Usefulness, And Green Consciousness,
Sustainability, Vol. 15(18), 2023, pp. 13961.
S. A. H. Mohsan, N. Q. H. Othman, Y. Li, M. H.
Alsharif, And M. A. Khan, 2023 Unmanned Aerial
Vehicles (Uavs): Practical Aspects, Applications,
Open Challenges, Security Issues, And Future
Trends, Intell. Serv. Robot.
a. Raghunatha, P. Thollander, And S. Barthel,
Addressing the Emergence of Drones – A Policy
Development Framework for Regional Drone
Transportation Systems, Transp. Res. Interdiscip.
Perspect., Vol. 18(100795), 2023, pp. 100795.
m. A. Azizan Et Al., The Effectiveness of Highway
Information Modeling in Kuala Perlis - Changlun
Roadway, 2020 In 2nd International Conference on
Materials Engineering & Science (IConMEAS 2019).
e. Fakhraian, I. Semanjski, S. Semanjski, And E.-H.
Aghezzaf, Towards Safe And Efficient Unmanned
Aircraft System Operations: Literature Review Of
Digital Twins’ Applications And European Union©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
709

Powered by TCPDF (www.tcpdf.org)

Drones and Data:

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

23.

24.

Regulatory Compliance, Drones, Vol. 7(7), 2023, pp.
478.
k. Al-Dosari And N. Fetais, A New Shift In
Implementing Unmanned Aerial Vehicles (Uavs) In
The Safety And Security Of Smart Cities: A
Systematic Literature Review, Safety (Basel), Vol.
9(3), 2023, pp. 64.
k. Al-Dosari, A. M. Deif, M. Kucukvar, N. Onat,
And N. Fetais, Security Supply Chain Using Uavs:
Validation and Development of a Uav-Based Modelfor Qatar’s Mega Sporting Events, Drones, Vol. 7(9),2023, pp. 555.
j. Matsimbe, W. Mdolo, C. Kapachika, I. Musonda,
And M. Dinka, Comparative Utilization of Drone
Technology Vs. Traditional Methods in Open Pit
Stockpile Volumetric Computation: A Case of Njuli
Quarry, Malawi, Front. Built Environ., Vol. 8, 2022.
k. Gunaratne, A. Thibbotuwawa, A. E. Vasegaard, P.
Nielsen, And H. N. Perera, Unmanned Aerial
Vehicle Adaptation to Facilitate Healthcare Supply
Chains in Low-Income Countries, Drones, Vol.
6(11), 2022, pp. 321.
h. J. Hadi, Y. Cao, K. U. Nisa, A. M. Jamil, And Q.
Ni, A Comprehensive Survey on Security, Privacy
Issues and Emerging Defence Technologies for
UAVs, J. Netw. Comput. Appl., Vol. 213(103607),
2023, pp. 103607.
u. Tariq, I. Ahmed, A. K. Bashir, And K. Shaukat, A
Critical Cybersecurity Analysis and Future Research
Directions for The Internet of Things: A
Comprehensive Review, Sensors (Basel), Vol. 23(8),2023, pp. 4117.
n.-A. Perifanis And F. Kitsios, Investigating the
Influence of Artificial Intelligence on Business
Value in The Digital Era of Strategy: A Literature
Review, Information (Basel), Vol. 14(2), 2023, pp.
85.
S. A. H. Mohsan, N. Q. H. Othman, Y. Li, M. H.
Alsharif, and M. A. Khan, Unmanned Aerial
Vehicles (UAVs): Practical Aspects, Applications,
Open Challenges, Security Issues, And Future
Trends, Intell. Serv. Robot, 2023.
L. Li, Reskilling and Upskilling the Future-Ready
Workforce for Industry 4.0 And Beyond, Inf. Syst.
Front., 2022.
a. M. Muhmad Kamarulzaman, W. S. Wan Mohd
Jaafar, M. N. Mohd Said, S. N. M. Saad, And M.
Mohan, Uav Implementations in Urban Planning andRelated Sectors of Rapidly Developing Nations: A
Review and Future Perspectives for Malaysia,
Remote Sens. (Basel), Vol. 15(11), 2023, pp. 2845.
b. Aydin, Public Acceptance of Drones: Knowledge,
Attitudes, And Practice, Technol. Soc., Vol.
59(101180), 2019, pp. 101180.
f. Nex Et Al., Uav in The Advent of The Twenties:
Where We Stand and What Is Next, Isprs J.
Photogramm. Remote Sens., Vol. 184, 2022, pp.
215–242.
h. Desa, M. Azizi Bin Azizan, M. S. A. Khadir, M.
S. Suhaimi, N. Z. Ramli, And Z. Hat, Feasibility
Study of UAV Implementation in Route Surveying,

J. Robot. Netw. Artif. Life, 2019.

Authors Introduction
Dr. Muhammad Azizi Bin Azizan
He received his PhD in Civil
Engineering from the Universiti
Malaysia Perlis. He is currently a
Senior Lecturer in the same
institution. He is Head of Project
Integration & Management (PIM) at
Centre of Excellence for Unmanned
AerialSystem(COE-UAS),
Universiti Malaysia Perlis.
Dr. Nurfadzillah Binti Ishak
She received her PhD in Building
Engineering from the Universiti
Malaysia Perlis. She is currently a
Senior Lecturer in the same institution.
She is Head of Built EnvironmentIntelligent (BELL) at Centre of
Excellence for Unmanned Aerial
System(COE-UAS),
Universiti
Malaysia Perlis.
Dr. Hazry Bin Desa
He obtained his PhD in Materials
Science and Production Engineering(Robotics) from Oita University andcurrently holds the position as a Headat the Centre of Excellence for
Unmanned Aerial Systems (COEUAS) at Universiti Malaysia Perlis(UniMAP).

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
710

Development of Variable Arm to Control the Manoeuvrability of Quadrotor
Lim Yi Hong, Hazry Desa, Muhammad Azizi Azizan

Centre of Excellence for Unmanned Aerial Systems (COE-UAS), Universiti Malaysia Perlis, Block E, Pusat Perniagaan
Pengkalan Jaya, Jalan Kangar – Alor Setar, 01000 Kangar, Perlis, Malaysia.
Muhammad Hassan Tanveer

Department of Robotics and Mechatronics Engineering, Kennesaw State University, Marietta, GA, 30067, USA.
Email: hazry@unimap.edu.my
www.unimap.edu.my

Abstract
This paper introduces the concept of a variable arm for a quadrotor which is able to perform the manoeuvrability ofthe quadrotor by changing the arm length’s. The variation in arm’s length affects the bending moment generated by
the thrust force, resulting in the tilting and movement of the quadrotor. The primary goal of this project is to develop
a quadrotor with an adjustable arm length to control its manoeuvrability effectively, thus minimizing the need for
additional thrust force during flight control. The study focuses on designing a quadrotor with the capability to extend
or retract its arms. The proposed concept relies on altering the bending moment through the variable arm to control
the quadrotor's manoeuvrability. A quadrotor equipped with a variable arm was successfully designed and tested, with
its performance evaluated in executing agile maneuvers. The experiment demonstrated that the variable arm induced
body rotation in the quadrotor, effectively regulating its manoeuvrability and the study validates the potential of thevariable arm approach for controlling quadrotor movement.
Keywords: N Retract, Control, Agile maneuvers, Body rotation, Performance evaluation, Validation.

1. Introduction

systems. Singanamalla et al. addressed reliability andenergy efficiency in emergency transmission within
wireless sensor networks [10], contributing to the broader
context of networked systems. Alzubi et al. conducted asurvey on specific IoT applications, providing insights
into the diverse applications of IoT across various
domains [11]. Furthermore, Kallam et al. explored lowenergy aware communication processes in IoT through agreen computing approach [12], aligning with the focus
on energy efficiency in the development of the variable
arm-controlled quadrotor. Suresh et al. proposed anenergy-efficient mechanism for leveraging IoT [13],setting the stage for considerations related to energy
efficiency in unmanned aerial systems. Mekala et al.
presented a computational intelligent sensor-rank
consolidation approach for the industrial Internet ofThings (IIoT) [14], providing valuable insights into
intelligent systems applied to industrial contexts.
Additionally, Reddy et al. focused on smart assistance for
elderly individuals in emergency situations using IoT[15], showcasing the potential applications of intelligent
systems in healthcare scenarios. Collectively, theseworks contribute to the foundation of knowledge
necessary for understanding and advancing thedevelopment of innovative systems, such as the variable
arm-controlled quadrotor presented in the current paper.

The quadrotor's working principle can be elucidated
through Newton's Third Law of Motion, which states thatevery action has an equal and opposite reaction [1]. Whenthe quadrotor's motor rotates the propeller, it generates adownward force on the air, a phenomenon explained by
Bernoulli's Principle [2]. In a standard quadrotor,
different motor speeds in the four motors allow for thrustgeneration, enabling precise control of the quadrotor's
movements [3]. As the size and weight increase, greater
thrust must be produced to lift the quadrotor effectively[4]. Typically, achieving this entails designing thequadrotor with higher-speed motors or larger propellers
to generate the necessary thrust to counter the quadrotor's
weight [5]. Consequently, this results in higher current
consumption and increased power usage, leading to
reduced flight endurance [6]. In this context, designing
the quadrotor using conventional methods can be bothcostly and inefficient [7]. This project proposes a variable
arm approach to control the quadrotor, utilizing different
arm lengths to regulate its manoeuvrability [8].
The research on variable arm-controlled quadrotors
draws on a foundation of related work in the fields ofsensor networks, IoT applications, and intelligent
systems. Kousik et al. leveraged a hybrid Convolution
Recurrent Neural Network for improved salient object
detection [9], showcasing advancements in intelligent©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
711

Lim Yi Hong, Hazry Desa, Muhammad Azizi Azizan, Muhammad Hassan Tanveer3(b) illustrates the vertical force produced by bothconstant thrust (𝐿𝑦 ) and the changed thrust (𝑅𝑦 ).

2. Materials and Methods

The quadrotor, depicted in Fig. 1, is designed in the formof a variable arm quadrotor, comprising 10 essential
components. These components include the Arduino
Mega, serving as the microcontroller, an ESC controller
responsible for regulating the RPM of the brushless DC
motor, the MDD3A driver for controlling the electric
actuator's movement, and an MPU 6050 sensor located at
the quadrotor's center to detect its acceleration androtation. Additionally, there are two sets of variable arms
and fixed arms, a LiPo Battery for power supply, an
A2212 2200KV brushless DC motor, and a base frame.
Specifically, the variable arm is engineered as an electric
actuator with a linear guide and designed to securely holdthe brushless DC motor.

(a)

(b)
Fig. 2. Free body diagram for (a) normal quadrotorand (b) variable arm quadrotor.

Fig. 1. Variable arm quadrotor.
3. Conceptual Modelling
The standard quadrotor achieves manoeuvrability byadjusting its speed to generate varying thrusts. The
quadrotor's design follows a plus configuration, which
offers simplicity in both calculation and analysis
compared to the cross configuration. Hence, thecalculations are conducted with two propellers whileassuming the other two propellers remain constant. In Fig.
2, a free-body diagram illustrates both the normal
quadrotor and the variable arm quadrotor, showcasing therelationship between angle, acceleration, and thedifference in thrust production with various arm lengths.
Fig. 2(a) displays the normal quadrotor, controlled bydifferent thrust forces, while Fig. 2(b) demonstrates thevariable arm quadrotor, which relies on adjusting the arm
length for control.
𝑊 = 𝐿 𝑐𝑜𝑠(𝐴𝑛𝑔𝑙𝑒) + 𝑅 𝑐𝑜𝑠(𝐴𝑛𝑔𝑙𝑒) = 3𝐿𝑦 + 𝑅𝑦

𝐹 = 𝑚𝑎

(a)

(1)
(2)

(b)

Based on Fig. 2, the quadrotor weighs 1.94 kg, andeach arm requires a thrust force of 4.76 N to maintainequilibrium. By assuming that only one thrust is changed
while the other three remain constant, Eq. (1) can beutilized to determine the force R required for thequadrotor to attain a specific angle. The acceleration
generated by Eq. (2) is depicted in Fig. 3(a) while Fig.

Fig. 3. (a) The acceleration and (b) changes of verticalforce.
When the angle increases, the constant thrustgenerates less vertical force. Therefore, to achieve
equilibrium for the quadrotor, the changed thrust should
be increased to enhance the vertical force produced.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
712

Development of Variable Arm

Additionally, this will result in a bending moment, asshown in Fig. 4(a), where the constant length of 200 mm
is used in Eq. (3).

Additionally, data from other axes are used to assess thestability and functionality of the variable arm.

(3)

𝑀 = 𝐹𝐿

Based on the bending moment in Fig. 4(a), theconcept of a variable arm is introduced, as demonstrated
in Fig. 2(b). In this configuration, the quadrotor can be
adjusted in angle by modifying the variable arm. When
keeping the thrust constant, altering the length of thethrust from the quadrotor's center affects the bending
moment, as indicated by Eq. (3).
By maintaining L1 and constant thrust S, the bending
moment on the quadrotor can be made equivalent to thatin Fig. 4(a) by adjusting the length of L2, as shown in Fig.
4(b). Consequently, increasing the length will lead to ahigher bending moment.

Fig. 5. Input signal and movement of the variable arm.
To control the extension or retraction of the variablearm, a signal is sent from the Arduino Mega
microcontroller to the MDD3A driver. Fig. 5 illustrates
the input signal for extending the variable arm. The
positive values indicate forward movement, while
negative values indicate backward movement. The
electric actuator's extension speed is set at 8 mm/s with a
30 mm stroke length. By incorporating program delaytime, the extension length of the variable arm can be
precisely controlled.
For data collection, the variable arm is extended todifferent lengths (8 mm, 16 mm, 24 mm, and 30 mm) andheld for a few seconds at each length to gather relevant
information. The entire extension cycle takes
approximately 88 seconds.

(a)

4.1.

With Constant Thrust Force

Fig. 6 and Fig. 7 demonstrate that as the arm length
increases, the rotation and acceleration of the quadrotor
also increase. However, it is evident that the relationshipbetween extension length and these variables is not linear.
To address this, a simplified graph for Fig. 6(b) and Fig.
7(b) was created, selecting the second highest positive
value or lowest negative value to minimize the influenceof noise from Fig. 6(a) and Fig. 7(a).

(b)
Fig. 4. Bending moment with (a) different thrust and (b)different length.
4. Result and Discussion
Based on the previous conceptual modeling, a prototype
was developed to observe and gather data regarding thequadrotor's movement with varying arm lengths. The
quadrotor is secured by a gripper and positioned in a waythat allows testing for rotation and acceleration. To
measure these parameters, an MPU6050 sensor is utilized,
providing data along the X, Y, and Z axes.
The quadrotor's design consists of 2 variable arms and2 fixed arms. Consequently, altering the length of thevariable arm causes the quadrotor to rotate along one axis
and move along another. Specifically, the quadrotor
rotates on the Y axis to move along the X axis.

(a)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
713

Lim Yi Hong, Hazry Desa, Muhammad Azizi Azizan, Muhammad Hassan Tanveer(b)Fig. 6. (a) Acceleration on X axis for dataset 1 (Ax1) &dataset 2 (Ax2) and (b) simplify data for acceleration on
X axis for dataset 1 (Ax1) & dataset 2 (Ax2).

(a)

(b)
Fig. 8. (a) Acceleration on Y axis for dataset 1 (Ay1) &dataset 2 (Ay2) and (b) rotation on X axis for dataset 1(Rx1) & dataset 2 (Rx2).

(a)

(a)
(b)
Fig. 7. (a) Rotation on Y axis for dataset 1 (Ry1) &dataset 2 (Ry2) and (b) simplify data for rotation on Y
axis for dataset 1 (Ry1) & dataset 2 (Ry2).
On the other hand, Fig. 8 and Fig. 9 display readings
for Ax, Az, Rx, and Rz, but with considerable noise.
Observing the graphs, most of the data for Ax, Az, Rx,
and Rz falls within a range of values from 3 to -3, 5 to 15,
0.2 to -0.2, and 0.15 to -0.15, respectively. However,there is a significant amount of data that exceeds theseranges, indicating an unstable condition for the quadrotor.
This instability could be attributed to various factors suchas environmental conditions like surrounding air flows,
unstable signals from the microcontroller, inconsistent
power supply, and the accuracy and sensitivity of thesensor being used.

(b)
Fig. 9. (a) Acceleration on Z axis for dataset 1 (Az1) &dataset 2 (Az2) and (b) rotation for Z axis for dataset1(Rz1) & dataset 2 (Rz2).
4.2.

With Variable Thrust Force

The variable thrust force incorporates astraightforward program for regulating the RPM ofbrushless DC motors, utilizing the range established fromearlier findings. During quadrotor operation, the craft istilted at an angle to facilitate movement, and thus, the Rx,
Rz, and Az ranges are utilized, with Ax being disregarded.
Az comes into play to ensure the quadrotor remains in an
equilibrium state and maintains its position.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
714

Development of Variable Arm

Consequently, this approach has the potential to reducenoise and enhance the quadrotor's overall stability.

(a)
(a)

(b)
Fig. 12. (a) Acceleration on Y axis for dataset 3 (Ay3) &dataset 4 (Ay4) and (b) rotation on X axis for dataset 3(Rx3) & dataset 4 (Rx4).

(b)Fig. 10. (a) Acceleration on X axis for dataset 3 (Ax3) &dataset 4 (Ax4) and (b) simplify data for acceleration on
X axis for dataset 3 (Ax3) & dataset 4 (Ax4).

(a)
(a)

(b)
Fig. 13. Acceleration on Z axis for dataset 3 (Az3) &dataset 4 (Az4) and (b) rotation on Z axis for dataset 3(Rz3) & dataset 4 (Rz4).

(b)Fig. 11. (a) Rotation of Y axis for dataset 3 (Ry3) &dataset 4 (Ry4) and (b) simplify data for rotation on Y
axis for dataset 3 (Ry3) & dataset 4 (Ry4).
In Fig. 10 and Fig. 11, as the variable arm length
increases, both rotation and acceleration also experiencea corresponding increase. To create a simplified
representation, we referred to the movement depicted inFig. 5 and extracted the second-highest positive value orlowest negative value for Fig 10(b) and Fig. 11(b). Thisapproach helps mitigate any potential noise influence
from Fig. 10(a) and Fig. 11(a) on the graphs.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
715

Lim Yi Hong, Hazry Desa, Muhammad Azizi Azizan, Muhammad Hassan Tanveer

concept utilizing an electric actuator and linear guide forthe variable arm was introduced, proving suitable forenhancing the manoeuvrability of the quadrotor by
adjusting the arm length.
Acknowledgment
This work was funded by the Universiti Malaysia Perlis(UniMAP) under the Commercialization Grant 900100748.
(a)References
1.

M. F. Ahmed and Y. S. Narayan, Fabrication and Testingof Quadcopter Prototype for Surveillance, International
Journal of Mechanical and Production Engineering
Research and Development (IJMPERD), June, 2018, pp.
99–105.
2. S. A. Khan, Z. Mehmood and Z. Afshan, Design, Analysis,
and Topology Optimization of a Landing Gear Strut for a
Quadcopter Upon Impact, International Conference on
Applied and Engineering Mathematics (ICAEM) (IEEE),
September 2021, pp. 37–42.
3. Y. R. Tang and Y. Li, Realization of the Flight Control for
an Indoor UAV Quadrotor, IEEE International Conferenceon Information and Automation (ICIA) (IEEE), August,2023, pp. 1278–1283.
4. N. Y. Kamil, D. Hazry, K. Wan and Z. M. Razlan,
Trajectory Tracking Based on Arm’s Length Variation,
Journal of Theoretical & Applied Information Technology,
Vol. 79(3), October, 2015, pp. 528-536.
5. P. E. I. Pounds, Design, Construction and Control of a
Large Quadrotor Micro Air Vehicle, Thesis for the Degreeof Doctor of Philosophy of the Australian National
University, September, 2007.
6. N. Y. Kamil, D. Hazry, K. Wan and Z. Razlan, A Novel
VAL: Quadrotor Control Technique for Trajectory
Tracking Based on Varying the Arm’s Length, ARPN J.
Eng. Appl. Sci, Vol. 11, 2015, pp. 9195–9204.
7. R. Sekaran, R. Patan, A. Raveendran, F. Al-Turjman, M.
Ramachandran and L. Mostarda, Survival Study on
Blockchain Based 6G-Enabled Mobile Edge Computation
for IoT Automation, IEEE access, Vol. 8, August, 2020,
pp. 143453-143463.
8. M. Rizon, C. Ang, M. I. Solihin, M. Z. Razlan, D. Hazry,
S. A. Bakar, W. Khairunizam and I. Zunaidi, Effects of
Variable Arm Length on UAV Control Systems, Journalof Robotics Networking and Artificial Life, Vol. 7(2),September, 2020, pp. 91–97.
9. N. Kousik, Y. Natarajan, R. A. Raja, S. Kallam, R. Patanand A. H. Gandomi, Improved Salient Object Detection
Using Hybrid Convolution Recurrent Neural Network,
Expert Systems with Applications, Vol. 166, March, 2021,
pp. 1-9.
10. V. Singanamalla, R. Patan, M. S. Khan, and S. Kallam,
Reliable and Energy‐Efficient Emergency Transmission
in Wireless Sensor Networks, Internet Technology Letters,Vol. 2(2), April, 2019.
11. J. A. Alzubi, R. Manikandan, O. A. Alzubi, O, N. Gayathri
and R. Patan, A Survey of Specific IoT Applications,International Journal on Emerging Technologies, Vol.
10(1), 2019, pp. 47-53.
12. S. Kallam, R. B. Madda, C. Y. Chen, R. Patan and D.
Cheelu, Low Energy Aware Communication Process in(b)Fig. 14. Frequency signal to brushless dc motor for (a)dataset 3 and (b) dataset 4.
In Fig. 12 and Fig. 13, the presented results still
exhibit some degree of noise. However, upon comparison
with the outcomes from the previous section, it becomes
evident that the quadrotor demonstrates enhanced
stability and balance. Fig. 14 illustrates that the frequency
signal sent to the motor undergoes alterations in a random
pattern, without showing any discernible relationship tochanges in the variable arm.
5. Conclusion

In this project, a quadrotor with a variable arm length wasconstructed. The variable arm is used to control the
manoeuvrability of the quadrotor. Data were collected infour sets, consisting of two with constant thrust force andtwo with variable thrust force. The analysis of theconstant thrust force data indicated that the quadrotor
exhibited instability, likely resulting from factors such as
surrounding airflow, an unstable signal from themicrocontroller, and fluctuations in the power supply. To
address these issues, a variable thrust force was
implemented to minimize noise and stabilize thequadrotor. The results displayed a clear linear
relationship between the variable arm length and therotation and acceleration of the quadrotor. As the arm
length increased, both the rotation and acceleration of thequadrotor also increased. Additionally, the findings
revealed that extending the variable arm eliminated theneed for extra thrust force, enabling the motor to fully
utilize its capacity in carrying the payload. The graphs
further indicated some noise variation, potentially causedby vibrations in the brushless DC motor. In conclusion, a©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
716

Powered by TCPDF (www.tcpdf.org)

Development of Variable Arm

IoT Using the Green Computing Approach, IET Networks,
Vol. 7(4), July, 2018, pp. 258-264.
13. K. Suresh, M. RajasekharaBabu and R. Patan, EEIoT:
Energy Efficient Mechanism to Leverage the Internet ofThings (IoT), International Conference on Emerging
Technological Trends (ICETT), October, 2016, pp. 1-4.
14. M. S. Mekala, P. Rizwan and M. S. Khan, Computational
Intelligent Sensor-Rank Consolidation Approach for
Industrial Internet of Things (IIOT). IEEE Internet ofThings Journal, Vol. 10 (3), February, 2023, pp. 21212130.
15. A. R. Reddy, G. P. Ghantasala, R. Patan, R. Manikandan
and S. Kallam, Smart Assistance of Elderly Individuals in
Emergency Situations at Home, Internet of Medical
Things: Remote Healthcare Systems and Applications,
April, 2021, pp. 95-115.

Authors Introduction
Mr. Lim Yi Hong
He completed his bachelor’s degree in
Mechatronic Engineering from Universiti
Malaysia Perlis and currently works as a
Mechanical
DesignEngineeratPentamaster Equipment Manufacturing
Sdn Bhd in Penang, Malaysia.
Prof. Dr. Hazry Desa
He earned his Bachelor of MechanicalEngineeringfrom
Tokushima
University, Japan, and proceeded topursue his Ph.D. at the Artificial Life andRobotics Laboratory, Oita University,
Japan, in 2003. Presently, he serves as the
director at the Center of Excellence for
Unmanned Aerial Systems (COE-UAS),
Universiti Malaysia Perlis, Malaysia.
Dr. Muhammad Azizi Azizan
He received his PhD in Civil
Engineering from the Universiti
Malaysia Perlis. He is currently a seniorlecturer in the same institution. He is
head of Project Integration &Management (PIM) at Centre of
Excellence for Unmanned Aerial System(COE-UAS), Universiti Malaysia Perlis.
Dr. Muhammad Hassan Tanveer
He is an Assistant Professor of Robotics
and Mechatronics Engineering at
Kennesaw State University. Prior to that,
he was a Post Doctoral Research
Associate at Virginia Tech, USA where
he was working on development of Biosonar sensor for UAVs navigations. He
received his Ph.D. in Robotics and Autonomous Sytems withspecilization in 'Collaboration of Heterogenous Team of
Robots' from University of Genova, Italy, in 2019.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
717

Development of IoT-Enabled Smart Water Metering System
Saw Di Wen, Hazry Desa and Muhammad Azizi Azizan

Centre of Excellence for Unmanned Aerial Systems (COE-UAS), Universiti Malaysia Perlis, Block E, Pusat Perniagaan
Pengkalan Jaya, Jalan Kangar – Alor Setar, 01000 Kangar, Perlis, Malaysia.
Abadal-Salam T. Hussain

Department of Medical Equipment Technology Engineering, College of Engineering Technology, Al-Kitab University,
Kirkuk, Iraq.
Muhammad Hassan Tanveer

Department of Robotics and Mechatronics Engineering, Kennesaw State University, Marietta, GA, 30067, USA.
Rizwan Patan

Department of Software Engineering and Game Development, Kennesaw State University, Marietta, GA, 30067, USA.
Email: hazry@unimap.edu.my
www.unimap.edu.my

Abstract
This paper introduces a smart water meter that utilizes the capabilities of the Internet of Things (IoT) to automate thecollection of meter readings. The primary goal of this project is to create an IoT-based device for reading water meters,
while simultaneously developing a compatible mobile application. Instead of relying on manual meter reading, which
requires human effort, this project proposes the use of an IoT-enabled water meter to collect the data automatically.
The device employs a camera and Convolutional Neural Network (CNN) for image processing, making it easy todetect the meter reading accurately. The IoT system architecture involves the use of an ESP32 CAM for data collection,a laptop as a gateway, and the Message Queuing Telemetry Transport (MQTT) protocol for data transfer. The
collected data is stored in Firebase's real-time database, and the mobile application is designed to monitor and analyze
the data. A functional prototype of the device is constructed and tested in a housing area. The collected data is thenmonitored through the developed mobile application. Lastly, the data is analyzed to assess the suitability of the
proposed method, and recommendations for future improvements are provided.
Keywords: IoT, Neural Network, Smart Water Management, Message Queuing system.

water and non-revenue water. Unbilled permitted use andwater losses are further categorized into apparent losses,
which include customer meter errors and unauthorizeduse, and true losses, such as leaks in mains, service
connections, and storage tanks [7].
Traditionally, water meters have been used forinvoicing water consumption, and manual meter readingis the prevailing method. However, this approach requires
a significant workforce and is prone to errors that canharm water companies. Moreover, manual meter reading
often necessitates entering indoor premises, adding to thechallenges faced by users and hindering daily operations[8]. Mechanical water meters, while reliable andaffordable, do not allow for automatic or real-timemonitoring of water usage. Consequently, water
management companies have relied solely on them,
despite the labor-intensive, time-consuming, and error-

1. Introduction
Water consumption is currently in high demand and has
been increasing due to population growth. Insufficientwater management capacity, unstructured management,
and the adverse effects of urbanization contribute to water
shortages [1], [2], [3]. Particularly in residential areas,households consume a significant amount of water daily[4], [5]. This excessive consumption, exceeding theavailable water supply, raises concerns about future water
shortages. Lack of public awareness, a significant
percentage of non-revenue water, inefficient water
demand management, and low water tariffs are generally
recognized as the main causes of excessive water usage[6]. Excessive water consumption is also attributed towater losses. To address this issue, the water balanceneeds to be calculated, distinguishing between revenue©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
718

Saw Di Wen, Hazry Desa, Muhammad Azizi Azizan, Abadal-Salam T. Hussain, Muhammad Hassan Tanveer,nd Rizwan Patan

prone nature of the process [8]. The market is graduallyshifting towards smart meter reading methods, which
eliminate the need for manual reading. This transition
began in industrialized Western countries, with the UnitedStates leading the adoption of smart meter reading
technology in the 1980s [9].
The automated collection and analysis of meter data
are facilitated by smart water metering (SWM) systems,
which incorporate various technologies. Customers canaccess their water usage data through online platforms
such as websites or mobile applications, allowing them tomanage their daily water usage more efficiently. Smart
metering improves operations by reducing costs
associated with operation and maintenance, per capita
usage, waste, and leaks. Key benefits include monitoringwater flow, distribution, and consumption, improving
access to clean and safe water, enabling frequent or realtime access to water consumption information and billing,
reducing the need for manual meter reading, enhancing
leak and fraud detection, and improving data collection
accuracy [10].
2.

2.2. Image Processing
The image processing algorithm used in this project has
been modified from an open-source codebase originallycreated by Jomjol [11]. When an image is captured, it issent to a web server for processing. On the server, theimage alignment is adjusted to a specific orientation tofacilitate smooth text recognition. Image segmentation
can also be fine-tuned within a certain range to accurately
detect numbers within the image. Additionally, artificial
intelligence has been integrated into the image processing
algorithm. By employing a Convolutional Neural
Network (CNN) and training it with various types ofwater meter images, the accuracy of number recognition
has significantly improved. The introduction of AI has
enabled the microcontroller to perform data computation
at the edge, rather than relying on cloud computing. Thisapproach minimizes delays in obtaining data. Instead of
waiting for data to be processed in the cloud, themicrocontroller can analyze the information locally,
resulting in real-time or near-real-time data processing.
2.3. Data Transfer

Methodology

The image processing software developed by Jomjol
utilizes MQTT to transfer data. The configuration for
MQTT can be pre-set within the web server [11]. In thisproject, the microcontroller remains powered on for 24hours, and data is collected at 30-minute intervals. To
facilitate data transfer using the MQTT protocol, an
MQTT broker must be installed on a device to host theserver. In this case, the Aedes broker was chosen due toits easy installation process. Additionally, the Aedes
broker is compatible with Node-Red, a programming tool
used for sending data to Firebase. With some coding, themicrocontroller can function as an MQTT publisher,
responsible for publishing data on a specific topic. The
Aedes broker then manages the published message andalerts subscribers who have subscribed to the same topic.
In this scenario, the published message refers to theprocessed data, while the laptop acts as a subscriber,
having Node-Red installed to receive the data. As a result,the microcontroller can establish communication with thelaptop for data transfer. Leveraging the capabilities ofNode-Red, the data can be further transmitted and stored
in the real-time database of Firebase.

2.1. Hardware Requirement
The primary microcontroller employed in this project isthe ESP32 Cam, which runs the main program. The
program code is uploaded to the microcontroller using a
programming software platform. Additionally, an SD
card is utilized to expand the memory capacity, enabling
the installation of the program, and recording of collected
data. The camera module, a crucial component, isconnected to the microcontroller and serves the purposeof monitoring the water meter by capturing images of themeter reading as shown in Fig. 1.

2.4. Cloud
For data transfer via MQTT, a connectivity platform isrequired, and in this case, a laptop is utilized as thegateway. The laptop acts as an intermediary, receiving
data from the microcontroller and sending it to the cloud.
Facilitating this communication is Node-Red, which has
been installed on the laptop. All the data transmitted by
the microcontroller is written into the Firebase real-time
database. This includes the initial water meter reading aswell as the most recent reading, along with timestamps
indicating when the data was captured. The presence ofdata stored in the Firebase real-time database isconsidered a successful transfer of data from themicrocontroller to the cloud at this stage. This mechanism

Water Meter with White and
Red BackgroundFig. 1. The construction of a smart water meter featuring
a display with white and red backgrounds.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
719

Development of IoT-Enabled Smartaligns with the cloud and network aspects in thearchitecture of IoT.

2.7. Circuit Connection

2.5. Application
The final component of the IoT architecture involves
application and data analysis. In terms of application,
users can access the water meter data through a dedicatedapp from anywhere, as long as they are connected to a WiFi network. Achieving this functionality involves
synchronizing the database with the developed mobile
application. Additionally, the app has the capability togenerate a bill based on the collected data.
2.6. Flowchart
Fig. 3. Schematic diagram.
Fig. 3 illustrates the schematic diagram of the Smart
Water Meter, showcasing the essential components of theelectronic circuit setup for this project. The ESP32 Cam
is a highly functional microcontroller board that
integrates a powerful ESP32 chip, along with built-in WiFi and Bluetooth capabilities. It serves as the primary
component of the setup. Additionally, the hardware
includes a camera module, enabling the device to stream
video and capture high-resolution images. Furthermore,
the microcontroller and an SD card are utilized to store
the programmed code, offering versatile and expandable
storage options. To enhance the lighting conditions forimage capture, an additional LED is incorporated toilluminate the surroundings.
Regarding the power supply, this project utilizes a DC
power source. To ensure compatibility with the hardware
components, a 5V converter is employed. Acting as a
crucial intermediary, the converter transforms theincoming AC power into a reliable 5V DC supply, whichpowers the main component of the project. Thisconfiguration guarantees a consistent and dependable
power source, facilitating the smooth operation of the
hardware and efficient execution of the programmed tasks.

Fig. 2. Smart water meter system flowchart.
This paper proposes a method for obtaining water meterreadings through a smart water meter system. To initiate
the system, the microcontroller is powered on using a 5V
DC power supply, enabling it to connect to a Wi-Fi
network. Once a stable Wi-Fi connection is established,
the program is designed to connect to the MQTT broker
as a client. Once all connections are established, themicrocontroller is ready to commence its operations.
The camera captures the first photo of the water meter
immediately upon activation, with the triggering interval
set to every 30 minutes. The captured image is thenuploaded to a web server for further processing, aimed at
extracting the data from the image. The text recognition
process concludes by sending the detected value to the
database via a tool called Node-Red.
The mobile application serves as a means for
consumers to retrieve and monitor data from the cloud. It
allows them to access the water meter readings and view
them on their mobile devices. Additionally,administrators can utilize the app to monitor the readings
and even send payment links to the cloud. Fig. 2 shows
the flowcharts of the developed smart water meter system.

3. Results and Discussions
3.1. Hardware Setup
A custom casing as shown in Fig. 4 has been designed
specifically for the ESP32 Cam to serve as storage. This
casing is positioned above the water meter and securely
fixed in place using supporting pillars. This ensures thatthe captured images remain clear and avoid any blurringor distortion.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
720

Saw Di Wen, Hazry Desa, Muhammad Azizi Azizan, Abadal-Salam T. Hussain, Muhammad Hassan Tanveer,nd Rizwan Patan

Fig. 4. Device casing - front view.

(a)3.2. Mobile Application
A mobile application has been developed with various
functionalities, including the ability to scan the Smart
Water Meter to establish a connection with the app. It
enables users to monitor their water consumption andgenerates bills based on their usage. The usage page
provides insights into the user's water consumption, while
the billing page generates the corresponding water bill.
The app retrieves data directly from the connected smart
water meter, ensuring accurate and up-to-date
information.
With this mobile application, users can easily track
their latest water usage and view the associated bill tounderstand the amount they need to pay. Thisfunctionality promotes increased awareness of current
water consumption, leading to potential cost savings. By
being more mindful of their water usage, users caneffectively manage their consumption and make informed
decisions to optimize their expenses. Fig. 5 illustrates thedisplay of information in the developed app, specifically
showcasing water usage details and the current usage bill.
This information can be accessed within the app and has
the capability to be printed for reference or
documentation purposes.

(b)Fig. 5. (a) Water usage page and (b) water bill page.
3.3. Data Analysis
Thorough analysis and discussion are conducted on thecollected data, considering the output results. This data is
obtained using the smart water meter device to retrieve
the meter reading. All the collected data is extracted into
an excel file to generate a graph for visualization asshown in Fig. 6. The blue line represents the value
obtained after image processing, while the orange line
represents the actual reading recorded from the captured
image of the water meter.
Fig. 6 reveals the presence of spikes in the graph,indicating instances where the image processing detected
false numbers from the water meter images. The primarycause of this issue is likely attributed to variations in thelighting conditions during image capture. The mostcommon error occurs when detecting the last 3-digit
number on the meter. Within the collected data, a clusterof errors is observed in the middle portion. This is
attributed to a specific number repeatedly reading
inaccurately, leading to the appearance of multiple spikes.
However, as the dial turns to a different number, the errors
diminish, resulting in fewer spikes toward the end of thedata.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
721

Development of IoT-Enabled Smart

Another factor contributing to the occurrence of
spikes is the presence of different background colors forthe digits, specifically the red color shown in Fig. 1. The
AI model used for the project was trained on images witha white background, which impacts the accuracy of themodel and prevents it from reaching the desired level of
performance.

4. Conclusion
Based on the aforementioned results, the system operates
smoothly without encountering any critical errors, despite
the detected data not being highly accurate. The accuracyof the data primarily depends on the AI model utilized forimage processing. This discrepancy arises because the AI
model was trained on a different type of water meter,
distinct from the one employed in this project. Without
proper training, the model's accuracy may becompromised. Nonetheless, the overall performance of
the AI model is commendable, as it accurately identifies
digits with a white background, exhibiting significantly
higher accuracy than those with a red background. Thus,
it can be concluded that a properly trained dataset greatly
enhances accuracy. Utilizing the collected data enables us
to ascertain the daily water usage for each household. Byleveraging this data, predictions can be made to determine
the water consumption in different areas, alerting thewater supply company accordingly. This valuable
information empowers the company to identify areas
requiring additional water supply, ensuring residents do
not face water shortages. Over time, as more data isgathered, the predictive capabilities of the system
improve, resulting in more accurate forecasts. In
summary, this IoT-based metering device resolves theissue of manpower required for water meter datacollection while also offering long-term cost savings.

Fig. 6. Comparison of manual water meter actual
reading between reading detected by the camera using
image processing technique.
3.4. Data Accuracy
The spike in Fig. 6 may happen due to the different
background color of the digits which is red color. The
accuracy is only 3.33% for the device detecting all 8-digit
numbers on the water meter. By reducing the last digit
number with the red background, the accuracy has
increased significantly to 53.33%. It is proved that the AI
model can detect all the digits with white background as
the accuracy for the first 5 digits is 100% accurate. The
AI model is not trained with red background digit
numbers, hence the accuracy for detecting the meterreading is not that high. Therefore, this explains theaccuracy result in Fig. 7 that this AI model used has a low
accuracy for detecting the meter digit number with red
background.

5. Future Recommendation
For the future implementation of this smart water metersystem, there is a need to enhance the design of the device
casing. The design should prioritize reliability and adhere
to sound design principles. Additionally, the issue ofsunlight penetration during the daytime must be
addressed by reimagining the casing product.
Furthermore, to address the common problem faced by allelectronic devices, a comprehensive plan for sustainableenergy usage in the power supply is necessary.
Additionally, to improve the accuracy of the system,
alternative algorithms can be explored and employed inconjunction with the AI model. Lastly, the mobile
application can be enriched with additional features toenhance user experience. For example, incorporating awater leak detection feature would enable users to receive
alerts regarding the condition of their pipes. Moreover,the billing system can be enhanced by expanding therange of payment methods, allowing for convenientonline payments and improving user-friendliness.
Acknowledgment
This work was funded by the Universiti Malaysia Perlis(UniMAP) under the Commercialization Grant 900100748.

Fig. 7. White and red background effects on water scale
meter accuracy.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
722

Powered by TCPDF (www.tcpdf.org)

Saw Di Wen, Hazry Desa, Muhammad Azizi Azizan, Abadal-Salam T. Hussain, Muhammad Hassan Tanveer,nd Rizwan Patan

References
Dr. Muhammad Azizi Azizan
He received his PhD in Civil Engineeringfrom the Universiti Malaysia Perlis. He iscurrently a senior lecturer in the same
institution. He is head of Project
Integration & Management (PIM) at
Centre of Excellence for Unmanned Aerial
System (COE-UAS), Universiti Malaysia
Perlis.

1.

M. Nadason, Joint Press Release by FOMCA, WECAM,
FAM: Celebrating World Water Day 2021 – Valuing
Water, March, 2022.
2. H. Abdul Rahman, A Review on Water Issues in Malaysia,
International Journal of Academic Research in Businessand Social Sciences, Vol. 11(8), August, 2021, pp. 860875.
3. S. Boatwright, S. Mounce, M. Romano and J. Boxall,
Integrated Sensor Placement and Leak Localization Using
Geospatial Genetic Algorithms, Journal of Water
Resources Planning and Management, Vol. 149 (9), June,
2023.
4. L. Canale, T. Cholewa, G. Ficco, A. Suita-Olcha, D. Di
Petra, P. Kolodziej and M. Dell’Isola, The Role ofIndividual Metering in Reducing Domestic Hot Water
Consumption in Residential Buildings: A Long-Term
Evaluation, Journal of Building engineering, Vol. 73,
August, 2023.
5. M. Kavya, A. Mathew, P. R. Shekar and P. Sarwesh, ShortTerm Water Demand Forecast Modelling Using Artificial
Intelligence for Smart Water Management, Sustainable
Cities and Society, Vol. 95, August, 2023.
6. N.S. Muhammad, J. Abdullah, N. A. Rahman and N. A.
Razali, Water Usage Behaviour: Case Study in a Southern
State in Peninsular Malaysia, IOP Conference Series:
Earth and Environmental Science, Vol. 646, 2021.
7. J. Cassidy, B. Barbosa, M. Damião, P. Ramalho, A.
Ganhão, A. Santos and J. Feliciano, Taking Water
Efficiency to the Next Level: Digital Tools to Reduce NonRevenue Water, Journal of Hydroinformatics, Vol. 23(3),May, 2021, pp. 453-465.
8. J. Marais, R. Malekian, N. Ye and R. Wang, A Review of
the Topologies Used in Smart Water Meter Networks: A
Wireless Sensor Network Application, Journal of Sensors,
Vol. 2016, October, 2016.
9. X. Cao, Design of Remote Water Meter Reading System
Based on GPRS Technology, Journal of Physics:
Conference Series, Vol. 1601, 2020.
10. T. Csaba, Challenges of Smart Water Metering, WM
Systems LLC - Innovation in Smart IoT Systems, Mar. 11,2021.
11. GitHub.
Retrieved
January
7,
2023,
from
https://github.com/jomjol/AI-on-the-edge-device

Dr. Abadal-Salam T. Hussain
He is currently a Ph.D. faculty assistantprofessor and head of the department for
Medical Instrumentation and TechniqueEngineering
at
AlkitabUniversity.
Previously, he was a staff member in the
FacultyofElectricalEngineering
Technology at Universiti Malaysia Perlis(UniMAP), Malaysia.
Dr. Muhammad Hassan Tanveer
He is an Assistant Professor of Robotics andMechatronics Engineering at Kennesaw
State University. Prior to that, he was a Post
Doctoral Research Associate at Virginia
Tech, USA where he was working on
development of Bio-sonar sensor for UAVs
navigations. He received his Ph.D. in Robotics andAutonomous Sytems with specilization in 'Collaboration of
Heterogenous Team of Robots' from University of Genova,
Italy, in 2019.
Dr. Rizwan Patan
He is currently working as a AssistantProfessor in the Department of Software
Engineering and Game Development at
Kennesaw State University, Marietta, USA.
He receives his Ph.D. in 2017 from the
School of Computing Science andEngineering at the Vellore Institute ofTechnology.

Authors Introduction
Mr. Saw Di Wen
He completed his bachelor’s degree in
Mechatronic Engineering from Universiti
Malaysia Perlis and currently works as a
Mechanical
DesignEngineeratPentamaster Equipment Manufacturing
Sdn Bhd in Penang, Malaysia.
Prof. Dr. Hazry Desa
He earned his Bachelor of MechanicalEngineering from Tokushima University,
Japan, and proceeded to pursue his Ph.D. at
the Artificial Life and Robotics
Laboratory, Oita University, Japan, in
2003. Presently, he serves as the director at
the Center of Excellence for Unmanned
Aerial Systems (COE-UAS), Universiti Malaysia Perlis,
Malaysia.
©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
723

Object Detection and Instance Segmentation with YOLOV8: Progress and Limitations
Law Jack Lee, Hazry Desa, Muhammad Azizi Azizan

Centre of Excellence for Unmanned Aerial Systems (COE-UAS), Universiti Malaysia Perlis, Block E, Pusat Perniagaan
Pengkalan Jaya, Jalan Kangar – Alor Setar, 01000 Kangar, Perlis.
Abadal-Salam T. Hussain

Department of Medical Equipment Technology Engineering, College of Engineering Technology, Al-Kitab University,
Kirkuk, Iraq.
Muhammad Hassan Tanveer

Department of Robotics and Mechatronics Engineering, Kennesaw State University, Marietta, GA, 30067, USA.
Email: hazry@unimap.edu.my
www.unimap.edu.my

Abstract
This research employs object detection and instance segmentation algorithms to distinguish between objects andbackgrounds and to interpret the detected objects. The YOLOV8 (You Only Look Once) framework and COCO
dataset are utilized for detecting and interpreting the objects. Additionally, the accuracy of detection, segmentation,and interpretation is tested by placing objects at various distances from the camera. The algorithm's performance was
evaluated, and the results were documented. In the experiments, a sample of 11 objects was tested, and 8 of themwere successfully detected at distances of 45cm, 75cm, 105cm, and 135cm. For instance, segmentation, segmentation
maps appeared clean when detecting a single object but faced challenges when multiple objects overlapped.
Keywords: Image Processing YoloV8, COCO, Segmentation, Object detection.

often heavily rely on human operators to analyze images
and spot potential issues [5]. This manual procedure canbe time-intensive and prone to errors, as human operators
might encounter challenges in consistently and preciselyinterpreting images or detecting subtle anomalies [4]. To
confront these challenges, this project introduces anobject detection and self-interpreting application
designed to augment monitoring systems.

1. Introduction
Traditional methods in object detection rely on handcrafted features and rules to identify and categorize
objects, whereas machine learning-based approaches
involve algorithms trained on extensive labeled image
datasets. Notably, machine learning-based methods offer
the advantage of handling complex and diverse
environments by learning and adapting to new data [1].
However, they can be computationally demanding andrequire substantial labeled data for training. In the past
few years, there have been notable strides in object
detection algorithms, especially with the rise of deeplearning-centered methods that have demonstrated
cutting-edge capabilities on a variety of benchmarks.
These methods utilize neural networks to learn featuresand classify objects, successfully finding applications invarious domains [2], [3]. Nevertheless, prevailingmonitoring systems might exhibit limited adaptability incomputer vision, leading to difficulties in accurately
detecting and classifying objects or features within
visually intricate settings, including those characterized
by low illumination, high contrast, or complex
backgrounds [4]. Furthermore, current monitoring setups

2. Related WorkThe research landscape in computer vision and deeplearning has witnessed significant advancements inrecent years, particularly in the domain of object
detection and segmentation. Notably, Kousik et al.
proposed a hybrid Convolution Recurrent Neural
Network for improved salient object detection [9].
Additionally, Mekala et al. introduced a Deep
Reinforcement Learning (DRL) based 4-r Computation
Model for Object Detection on Roadside Units (RSU)
using LiDAR on the Internet of Things (IoT) context [10].
Deep learning and image processing techniques forcancer diagnosis were effectively employed by Prassanna
et al. [11], while Ghantasala et al. focused on texture
recognition and image smoothing for microcalcification©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
724

Law Jack Lee, Hazry Desa, Muhammad Azizi Azizan, Abadal-Salam T. Hussain, Muhammad Hassan Tanveer

and mass detection in abnormal regions [12]. In thecontext of the Internet of Medical Things, Manimurugan
et al. utilized a deep belief neural network for effective
attack detection in smart environments [13]. Furthermore,
Natarajan et al. applied fully convolutional deep neuralarchitecture for the segmentation of nuclei inhistopathology images [14]. Lastly, Selvaraj et al.
proposed an optimal virtual machine selection approach
using swarm intelligence for anomaly detection [15].
These diverse works collectively contribute to thebroader understanding and application of advanced
techniques in the realm of computer vision and deeplearning.

Fig. 1. Object detection and instance segmentation.

3. Methodology

4. Results and Discussion

Overall Object Detection and Instance Segmentation
Segmentation is a specific type of image analysis that
focuses on identifying individual instances of objects andoutlining their boundaries [6]. Two essential computer
vision tasks, semantic segmentation, and object detection
are closely related to instance segmentation [7]. Semantic
segmentation seeks to correctly assign the appropriate
object category to every individual pixel present within
an image. However, it fails to differentiate among
separate instances of identical object classes. For instance,
if there are three dogs in an image, semantic segmentation
will only identify pixels belonging to one of these threedogs.
Conversely, object detection involves predicting boththe object's bounding box and its corresponding class forevery instance of an object present in the image. However,it does not provide per-pixel identification for each object
instance. Hence, object instance segmentation is notably
more challenging than semantic segmentation and object
detection because it seeks to both identify and label eachobject instance on a per-pixel basis [7].
To create a dataset, for example, one can collectimages of cats and dogs, organizing them into two
folders: "cats" and "dogs." Each folder should contain asimilar number of images. In each folder, a total 80% of
the images are allocated as training images, and theremaining 20% are used as test images. Annotations are
required for the training images, involving manuallydrawing bounding boxes and segmenting the dogs or catsfrom the background. These annotations produce
coordinates that are recorded in a blank text file, essential
for the machine's ability to locate objects during image
training and testing. Fig. 1 displays an example of object
detection, with a bounding box around the Region ofInterest (ROI), labelled with its name, and segmented
with a coloured mask layer within the ROI, utilizing the
COCO dataset [8].

4.1 Object Detection and Self-Interpreting Application
Within Different Distance.
To test the ability of detection and interpreting
application, the experiment was carried out in room. 11
objects were selected, and each object was placed in
45cm, 75cm, 105cm and 135cm apart from the webcam.
The results were recorded in Table 1 below. A tickindicates that the detection was successful, and a crossindicates that the detection was unsuccessful.

(a)

(b)
Fig. 2. (a) Scissor at 45 cm distance and (b) scissor at 75
cm distance from the camera.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
725

Object Detection and Instancefailed to do so at 75cm, 195cm, and 135cm distances.
Such limitations arise from the detection andsegmentation networks, as overlapping objects posechallenges in separation and precise localization. The
output image may look acceptable, but it merely
represents the network's best estimation of object
locations. When multiple objects overlap, the model may
struggle, resulting in inaccurate bounding boxes or masks,
which can sometimes lead to incorrect labels.

(a)

(a)

(b)
Fig. 3. (a) Scissor at 105 cm distance and (b) scissor
at 135 cm distance from the camera.
Fig. 2 and Fig. 3 present examples of various
distances captured by the webcam. The figures show apair of scissors, accurately detected, and identified by thesystem. Most large objects, such as a person, cell phone,keyboard, and bowl, were successfully detected andidentified. However, at distances of 105 cm and 135 cm,
the objects like the mouse, fork, and spoon were notrecognized. This lack of recognition was primarily due toobject occlusion. When the fork and spoon were held at
105 cm, the student's fingers obstructed part of the objects,
making it challenging for the Yolo model to detect them.
Furthermore, the webcam's resolution also impacted the
Yolo model's detection rate. Although the webcam
supported 1080p Full HD video quality, the output wascompressed to 640px x 640px for smoother video
operation. It is essential to note that this experiment was
conducted on a CPU-only machine, resulting insignificantly lower performance compared to using a
GPU-supported machine.
Instance Segmentation Within Different Distances.
The results obtained from the YOLOV8 model for
instance segmentation were observed and collected.
When a single object was detected at distances of 45cm,
75cm, 105cm, and 135cm, the segmentation maps of aperson appeared to be clean, as evident from Fig. 2 andFig. 3. However, in cases where multiple objects
overlapped, the model encountered difficulties inaccurate segmentation. Fig. 4 and Fig. 5 demonstrate theinconsistent segmentation of a keyboard at different
distances. While the keyboard could be segmented
successfully at 45cm when accompanied by a human, it(b)Fig. 4. (a) 45 cm distance (b) 75 cm distance away fromthe camera.

(a)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
726

Law Jack Lee, Hazry Desa, Muhammad Azizi Azizan, Abadal-Salam T. Hussain, Muhammad Hassan Tanveer

This work was funded by the Universiti Malaysia Perlis(UniMAP) under the Commercialization Grant 900100748.
References
1.

(b)Fig. 5. (a) 105 cm distance (b) 135 cm distance awayfrom the camera.

2.

Table 1. Instances of successful and unsuccessful attempts ofbeing detected.
Objects

3.

Distance away from camera (cm) on UAV
45

75

105

135

4.

Percentage (%)

Scissors

✓

✓

✓

✓

100

Person

✓

✓

✓

✓

100

Bottle

✓

✓

✓

✓

100

Cup

✓

✓

✓

✓

100

Cellphone

✓

✓

✓

✓

100

Mouse

✓

✓

×

×

50

Keyboard

✓

✓

✓

✓

100

Bowl

✓

✓

✓

✓

100

Spoon

✓

✓

×

×

50

Fork

✓

✓

×

×

50

Toothbrush

✓

✓

✓

×

75

5.

6.

7.

8.

5. Conclusion
The experiment involving object detection and a selfinterpreting system was carried out successfully,
employing the YOLOV8 framework and the COCO
dataset to collect the results. Out of the 11 objects that
underwent testing, 8 of them were effectively identified
at different distances: specifically, 45cm, 75cm, 105cm,
and 135cm. This led to the algorithm used in theexperiment achieving an accuracy rate of 72% in terms ofobject detection and interpretation. However, there were
instances where 4 objects failed to be detected accurately
at distances of 105cm and 135cm. This was attributed to
the lower image quality at these distances. Notably, thesegmentation maps displayed clean results when a single
object was detected, but inconsistency in segmentation
maps arose when multiple objects overlapped, mainly
due to the limitations of the detection and segmentation
network. In conclusion, while the experimentdemonstrated promising results with a 72% accuracy inobject detection and interpretation, challenges persisted
in cases of poor image quality and overlapping objects,
affecting the segmentation performance.

9.

10.

11.

12.

13.

Acknowledgements

H. Ishibuchi, T. Nakashima and T. Kuroda, A Fuzzy
Genetics-Based Machine Learning Method for Designing
Linguistic
Classification
Systems
withHigh
Comprehensibility, ICONIP'99. ANZIIS'99 & ANNES'99& ACNN'99, 6th International Conference on Neural
Information Processing, August, 2002.
J. Han, D. Zhang, G. Cheng, N. Liu and D. Xu, Advanced
Deep-Learning Techniques for Salient and CategorySpecific Object Detection: A Survey, IEEE Signal
Processing Magazine, Vol. 35(1), January, 2018, pp. 84100.
Z. -Q. Zhao, P. Zheng, S. -T. Xu and X. Wu, Object
Detection with Deep Learning: A Review, IEEE
Transactions on Neural Networks and Learning Systems,
Vol. 30(11), November, 2019, pp. 3212-3232.
S. Gundu, H. Syed and J. Harikiran, Human Detection in
Aerial Images using Deep Learning Techniques, 2ndInternational Conference on Artificial Intelligence andSignal Processing (AISP), April, 2022, pp. 1-10.
R. A. Suárez Fernández, J. L. Sanchez-Lopez, C.
Sampedro, H. Bavle, M. Molina and P. Campoy, Natural
User Interfaces for Human-Drone Multi-Modal Interaction,
International Conference on Unmanned Aircraft Systems(ICUAS), July, 2016, pp. 1013-1022.
A. M. Hafiz and G. M. Bhat, A Survey on InstanceSegmentation: State of the Art, International Journal ofMultimedia Information Retrieval, Vol. 9(3), July, 2020,
pp. 171-189.
L. Ye, Z. Liu and Y. Wang, Depth-Aware Object InstanceSegmentation, IEEE International Conference on Image
Processing (ICIP), February, 2018, pp. 325-329.
T. Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D.
Ramanan, P. Doll´ar and C. L. Zitnick, Microsoft COCO:
Common Objects in Context, Computer Vision–ECCV
2014: 13th European Conference, Zurich, Switzerland,
September, 2014.
N. Kousik, Y. Natarajan, R. A. Raja, S. Kallam, R. Patanand A. H. Gandomi, Improved Salient Object Detection
Using Hybrid Convolution Recurrent Neural Network,
Expert Systems with Applications, Vol. 166, March, 2021.
M. S. Mekala, R. Patan, A. H. Gandomi, J. H. Park and H.
Y. Jung, A DRL based 4-r Computation Model for Object
Detection on RSU using LiDAR in IloT. IEEE Symposium
Series on Computational Intelligence (SSCI), January,2022, pp. 01-08.
J. Prassanna, R. Rahim, K. Bagyalakshmi, R. Manikandan
and R. Patan, Effective Use of Deep Learning and Image
Processing for Cancer Diagnosis, Deep learning forCancer Diagnosis, September, 2020, pp. 147-168.
G. P. Ghantasala, S. Kallam, N. V. Kumari and R. Patan,
Texture Recognization and Image Smoothing forMicrocalcification and Mass Detection in Abnormal
Region, International Conference on Computer Science,
Engineering and Applications (ICCSEA), July, 2020.
S. Manimurugan, S. Al-Mutairi, M. M. Aborokbah, N.
Chilamkurti, S. Ganesan and R. Patan, Effective Attack
Detection in Internet of Medical Things Smart
Environment Using a Deep Belief Neural Network, IEEE
Access, Vol. 8, April, 2020, pp. 77396-77404.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
727

Powered by TCPDF (www.tcpdf.org)

Object Detection and Instance

14. N. A. Natarajan, M. S. Kumar, R. Patan, S. Kallam and M.
Y. N. Mohamed, Segmentation of Nuclei inHistopathology Images Using Fully Convolutional Deep
Neural Architecture, International Conference on
Computing and Information Technology (ICCIT-1441),
November, 2020, pp. 319-325.
15. A. Selvaraj, R. Patan, A. H. Gandomi, G. G. Deverajan and
M. Pushparaj, Optimal Virtual Machine Selection for
Anomaly Detection Using a Swarm Intelligence Approach,Applied Soft Computing Journal, Vol. 84, November,
2019.

Dr. Muhammad Hassan Tanveer
He is an Assistant Professor of Robotics
andMechatronics
EngineeringatKennesaw State University. Prior to that,
he was a Post Doctoral Research Associate
at Virginia Tech, USA where he wasworking on development of Bio-sonarsensor for UAVs navigations. He received
his Ph.D. in Robotics and Autonomous Sytems withspecilization in 'Collaboration of Heterogenous Team of
Robots' from University of Genova, Italy, in 2019.

Authors Introduction
Mr. Law Jack Lee
He completed his bachelor’s degree in
Mechatronic Engineering from Universiti
Malaysia Perlis and currently works as a
Mechanical
DesignEngineeratPentamaster Equipment Manufacturing
Sdn Bhd in Penang, Malaysia.

Prof. Dr. Hazry Desa
He earned his Bachelor of MechanicalEngineering from Tokushima University,
Japan, and proceeded to pursue his Ph.D.
at the Artificial Life and Robotics
Laboratory, Oita University, Japan, in
2003. Presently, he serves as the director
at the Center of Excellence for Unmanned
Aerial Systems (COE-UAS), Universiti
Malaysia Perlis, Malaysia.
Dr. Muhammad Azizi Azizan
He received his PhD in Civil Engineeringfrom the Universiti Malaysia Perlis. He iscurrently a senior lecturer in the same
institution. He is head of Project
Integration & Management (PIM) at
Centre of Excellence for Unmanned
Aerial System (COE-UAS), Universiti
Malaysia Perlis.
Dr. Abadal-Salam T. Hussain
He is currently a Ph.D. faculty assistantprofessor and head of the department for
Medical Instrumentation and TechniqueEngineering at Alkitab University.
Previously, he was a staff member in the
FacultyofElectricalEngineering
Technology at Universiti Malaysia Perlis(UniMAP), Malaysia.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
728

A high-performance motion planning method based on asymptotically optimal RRT
Tianbin Meng

School of Mechanical and Electronic Engineering, Beijing Jiaotong University
Beijing, Haidian District, China
Jiwu Wang

School of Mechanical and Electronic Engineering, Beijing Jiaotong University
Beijing, Haidian District, China
E-mail: mengtianbin888@163.com, jwwang@bjtu.edu.cn
www.bjtu.edu.cn
Abstract
The motion planning algorithm of the robot arm plays an important role in the working process of the robot arm,
especially in the complex environment, an efficient algorithm is more conducive to the robot arm to complete thecorresponding planning task. Aiming at the problems such as low exploration efficiency and poor planning path in
the current motion planning task of robotic arm, we propose a distance constraint mechanism. Based on 𝑅𝑅𝑇 ∗ and𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗, the algorithm uses halton sequence to generate random points and introduces the current lowest
cost path length. To avoid useless nodes extension. The simulation results show that the algorithm with distanceconstraint mechanism can improve the exploration efficiency and planning quality to some extent.
Key words: mechanical arm, Motion planning, Distance constraint, Improved 𝑅𝑅𝑇 ∗ algorithm, halton sequence

probability completeness, but cannot guarantee the optimal
solution and a large number of unnecessary waypointsearches. Frazzoli [6] proposed 𝑅𝑅𝑇 ∗ on the basis of
RRT, which is an algorithm with asymptotic optimality.
Compared with RRT, it adds two processes of re-selecting
parent nodes and rewiring, Therefore, an optimal or
suboptimal path can be found. Gammell [7] improved on
the basis of 𝑅𝑅𝑇 ∗ and proposed the 𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗algorithm. The principle is that after finding the firstfeasible solution, the subsequent sampling range is limited
to a high-dimensional ellipsoid determined by the cost of
the feasible solution. The convergence speed of 𝑅𝑅𝑇 ∗ is
improved.
To sum up, in order to solve the problem of low
efficiency and too many redundant nodes in the 𝑅𝑅𝑇 ∗computing planning. In this paper, a distance constraint
mechanism is introduced to improve 𝑅𝑅𝑇 ∗ and𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗ , which can greatly reduce the searchand expansion of unnecessary waypoints and plan a highquality path.

1. Introduction
The robot arm has changed from the original roughindustrial production equipment to today's multidisciplinary industrial automation equipment, entering all
corners of society, showing the powerful role of the robot
arm. At the same time, higher requirements are also put
forward for the motion planning of the robotic arm.
Therefore, the study of the motion planning of the roboticarm has important engineering value [1], [2].
At present, there are four main types of path planning
methods for robotic arms: graph search-based algorithm,
artificial potential field method, random sampling methodand deep reinforcement learning method [3]. The
algorithm based on graph search needs to be discretized in
the whole planning space, so it is not suitable for the spacewith higher latitude. The artificial potential field method iseasy to make the manipulator fall into the local minimum
under the influence of the influence field, and theefficiency is greatly reduced. Methods based on deep
reinforcement learning also have problems of poor processinterpretability and stability [4].
The method based on random sampling is one of the mostcommon methods for path planning of manipulators. Its
core idea is to replace the complex actual planning spaceby the simple planning space obtained by sampling, so as
to reduce the space complexity. the method based on
random sampling is efficient and suitable for highdimensional space. Lavalle [5] proposed a classical RRT
algorithm, which has strong exploration ability and2. Methodology
2.1. Random treemechanismgrowthdistance

constraint

𝑅𝑅𝑇 ∗ algorithm has asymptotic optimality, but the
random number growth in the sampling process has astrong blindness, thus reducing the efficiency of thealgorithm. 𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗ Although the sampling
points can be limited to an ellipse, the expansion of random
trees within the ellipse is still blind. Therefore, this paper
proposes a distance constraint mechanism, which
introduces the current lowest cost path length. When the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
729

Tianbin Meng, Jiwu Wang

random tree is extended, the minimum actual cost to reachthe new node is calculated plus the Euclidean distancefrom the new node to the end point. If the estimateddistance is greater than the current lowest cost path length,
the new node will not be extended, which increases thesearch efficiency of the algorithm to a certain extent. A
condition that determines whether a new node can be
expanded:𝑔𝑇 (𝑥𝑛𝑒𝑤 ) + ℎ̂(𝑥𝑛𝑒𝑤 ) < 𝑐𝑏𝑒𝑠𝑡(1)

Where 𝑔𝑇 (𝑥) represents the minimum actual cost fromthe starting point to the new node, ℎ̂(𝑥) represents the
Euclidean distance from the new node to the end point, and𝑐𝑏𝑒𝑠𝑡 is the current lowest cost path length.
Similarly, the current lowest cost path length is also
introduced in the process of rewiring in random trees.
When a new node is selected as the parent node at a certain
point in the neighborhood, if the minimum actual cost toreach the point plus the Euclidean distance from the point
to the end point is greater than the current lowest cost path
length, the parent node of the node will not be replaced. A
condition that determines whether a node in theneighborhood can replace the parent node:Fig.1 Random sampling method

Fig.2 Sampling method based on halton sequence𝑔𝑇 (𝑥𝑛𝑒𝑤 ) + ||𝑥𝑛𝑒𝑤 − 𝑥𝑛𝑒𝑎𝑟 || + ℎ̂(𝑥𝑛𝑒𝑎𝑟 ) < 𝑐𝑏𝑒𝑠𝑡(2)

Where ||𝑥𝑛𝑒𝑤 − 𝑥𝑛𝑒𝑎𝑟 ||indicates the Euclidean distance
between 𝑥𝑛𝑒𝑤 and 𝑥𝑛𝑒𝑎𝑟 .

It can be observed that the sampling points in the upper
left part of Fig. 1 are sparse, while the middle part is dense.
In Fig. 2, the distribution of sampling points is more
uniform, thus improving the ability to explore the space.

2.2. Random sampling point optimization strategy

3. Algorithm simulation analysis

𝑅𝑅𝑇 ∗ and 𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗ algorithms have theproblem of poor uniformity in the random sampling ofpoints in space, so the space cannot be explored more fully.
A random point sampling method based on haltonsequence is proposed for reference to halton sequence.
Random sampling method (Fig.1) and halton sequencebased sampling method (Fig.2) are shown in the figure
below.

In order to verify the validity of the distance constraint
mechanism, the 𝑅𝑅𝑇 ∗ ， 𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗ and theimproved 𝐷𝐶 − 𝑅𝑅𝑇 ∗ and 𝐷𝐶 − 𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗are simulated and analyzed experimentally in a threedimensional environment.
In MATLAB, simulation experiments in simple andcomplex obstacle avoidance environments are set
respectively. The simulation environment was set to thethree-dimensional space of [0.2 × 0.12 × 0.2], the searchstarting point was [−0.09 0.37 0.1], the target point was[0.09 0.43 0.1], the exploration step was 0.01, and thenumber of iterations was set to 100 to reduce therandomness of the algorithm. The experimental results areshown in the figure and table below.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
730

A high-performance motion

a. 𝑅𝑅𝑇 ∗ algorithm tracingb. 𝐷𝐶 − 𝑅𝑅𝑇 ∗ algorithm tracingc. 𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗ algorithm tracingd. 𝐷𝐶 − 𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗ algorithm tracing

Fig.3 Algorithm tracing in 3D simple scenes
Table 1. Performance comparison of algorithms in 3D simple scenes.

algorithm
𝑅𝑅𝑇 ∗𝐷𝐶 − 𝑅𝑅𝑇 ∗𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗𝐷𝐶 − 𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗average path length /m
0.20420.19340.19300.1906

As shown in Fig. 3, The blue line represents the extended
branch, and the green line represents the final path. 𝑅𝑅𝑇 ∗produces many useless branches. 𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗also has some useless branches, but fewer than 𝑅𝑅𝑇 ∗ . The
final path of the two is tortuous and the path length is long.
After the introduction of distance constraint mechanism,
the useless branches of 𝐷𝐶 − 𝑅𝑅𝑇 ∗ and 𝐷𝐶 −𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗ are significantly reduced, the final
path is smoother and the path length is shorter. Combined

average running time /s
6.67
5.41
8.42
5.72

Node usage /%6.62%
10.96%
6.38%
12.31%

with the data in Table 1, the average path of 𝐷𝐶 − 𝑅𝑅𝑇 ∗is reduced by 5.29% from 0.2042m to 0.1934m comparedwith 𝑅𝑅𝑇 ∗ . The average running time decreased by
18.90% from 6.67s to 5.41s; The node utilization rate isincreased from 6.62% to 10.96%, and the average path of
𝐷𝐶 − 𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗ and 𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗ isreduced from 0.1930m to 0.1906m by 1.24%. The averagerunning time decreased by 32.07% from 8.42s to 5.72s;
Node usage increased from 6.38% to 12.31%.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
731

Tianbin Meng, Jiwu Wang

a. 𝑅𝑅𝑇 ∗ algorithm tracingb. 𝐷𝐶 − 𝑅𝑅𝑇 ∗ algorithm tracingc. 𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗ algorithm tracingd. 𝐷𝐶 − 𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗ algorithm tracing

Fig.4 Algorithm tracing in 3D complex scenes
Table 2. Performance comparison of algorithms in 3D complex scenes.

algorithm
𝑅𝑅𝑇 ∗𝐷𝐶 − 𝑅𝑅𝑇 ∗𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗𝐷𝐶 − 𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗average path length /m
0.21910.20560.20030.1965

Fig. 4 shows the planning results of the algorithm in acomplex environment. Combined with the data in Table 2,
the average path of 𝐷𝐶 − 𝑅𝑅𝑇 ∗ decreases by 6.16%
from 0.2191m to 0.2056m compared with 𝑅𝑅𝑇 ∗ . The
average running time is 17.16s and 18.42s respectively, thenode utilization rate is increased from 7.63% to 14.75%,
and the average path of 𝐷𝐶 − 𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗ isreduced from 0.2003m to 0.1965m, a decrease of 1.89%.
The average running time was 20.41s and 21.95s
respectively, and the node utilization rate increased from
7.01% to 16.66%.
According to the analysis, since the introduction of
distance constraint mechanism reduces the expansion ofnodes that are useless for optimizing the path, the node
utilization rate is higher, and the node distribution is moreconcentrated near the final path, then the node is more
likely to optimize the cost of the current path. Secondly,
although the introduction of distance constraint
mechanism reduces the expansion of useless nodes, thenodes are more concentrated. In a simple obstacle
avoidance environment, the time to reduce the expansionof useless nodes is greater than the time to increase theprocess of parent node re-selection and rewiring, so theaverage running time /s
17.16
18.42
20.41
21.95

Node usage /%7.63%
14.75%
7.01%
16.66%

average running time is reduced. In complex obstacle
avoidance environments, parent node reselecting andrewiring processes increase the time more than the time to
reduce the expansion of useless nodes, so the averagerunning time increases. The distance constraint mechanism
proposed in this paper enables the algorithm to select andoptimize the path with higher efficiency during operation,
which makes the node utilization rate increase and the final
path length decrease. Especially in the simple obstacle
avoidance environment, the search efficiency of thealgorithm is improved to a large extent.
4. Conclusion
In this paper, a motion planning algorithm based on
𝑅𝑅𝑇 ∗ and 𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗ with distance constraint
mechanism is proposed to improve the problems of low
exploration efficiency and poor planning path in thecurrent motion planning tasks of robotic arms under
different complexity scenarios. It mainly reduces the
expansion of useless nodes when the random tree grows,
so that the algorithm can select and optimize the path with

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
732

Powered by TCPDF (www.tcpdf.org)

A high-performance motion

higher efficiency when running. Combined with haltonsequence sampling method, the space exploration abilityof the algorithm is improved. The simulation results showthat 𝐷𝐶 − 𝑅𝑅𝑇 ∗ and 𝐷𝐶 − 𝐼𝑛𝑓𝑜𝑟𝑚𝑒𝑑 − 𝑅𝑅𝑇 ∗ canreduce the path length, improve the node utilization rate,and improve the search efficiency of the algorithm to alarge extent, especially in simple environment.
References
1.

2.
3.

4.

5.
6.

7.

Fang H, Ong S, Nee A. Robot path planning optimiza
tion for welding complex joints [J]. INTERNATIONA
L JOURNAL OF ADVANC-ED MANUFACTURINGTECHNOLOGY, 2017.
La Valle S M. Motion planning[J]. IEEE
Robotics &Automation Magazine, 2011, 18 (2): 108-118.
Liang S. Research on off-line and on-line mo-tion pla
nning methods for multi-degree of freedom manipulator (in Chinese) [D]. Zhejiang University,2023.
Ying K C, Pourhejazy P, Cheng C Y, et al. Deep learningbased optimization for motion planning of dual-arm
assembly robots[J]. Computers & Industrial Engineering,
2021, 160: 107603.
Lavalle S M. Rapidly-Exploring Random Trees: A New
Tool for Path Planning[J].Research Report, 1999.
Karaman S, Frazzoli E. Incremental Sampling-based
Algorithms for Optimal Motion Planning[C]//Robotics:
Science and Systems 2010.2010.
Gammell J D, Srinivasa S S, Barfoot T D.In-formed RRT*:
Optimal Sampling-based Path Planning Focused via Direct
Sampling of an Admissible Ellipsoidal Heuristic[J].IEEE,
2014.

Authors Introduction
Mr. Tianbin Meng
He is a postgraduate in Beijing Jiaotong
University. His research interests are Robotarm motion planning.

Dr. Jiwu Wang
He is an associate professor, Beijing
Jiaotong University. His research interests
are Intelligent Robot, Machine Vision, andImage Processing.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
733

CSM-RRT*: an improved RRT* algorithm based on constrained sampling mechanism
Hang Yang

School of Mechanical and Electronic Engineering, Beijing Jiaotong University
Beijing, Haidian District, China
Jiwu Wang

School of Mechanical and Electronic Engineering, Beijing Jiaotong University
Beijing, Haidian District, China
Xueqiang Shang

Aero Engine Corporation of China
Beijing, Changping District, China
E-mail: 21121252@bjtu.edu.cn,jwwang@bjtu.edu.cn,shangxueqiang@163.com
www.bjtu.edu.cn
Abstract
The Rapidly Exploring Random Tree Star (RRT∗) is recognized as a better path planning algorithm, but its path
quality and path planning speed still have room for improvement. In this paper, an improved RRT* algorithm(CSMRRT*) based on constrained sampling mechanism is proposed .The entire planning process is divided into two steps:
fast exploration and optimization of the initial path. Subsequently, a dynamic sampling region consists of removed
redundant nodes and collision nodes is formed around initial path By prioritizing exploration within this dynamic
region, computational resources can be saved and the asymptotic optimal path can be quickly converged from theinitial path. Eventually, simulation results presented in various obstacle environments confirm the efficiency of CSMRRT*.
Keywords: Path planning, RRT*, Constrained sampling mechanism, Dynamic sampling region

provide transcendental information for the path.QuickRRT* [6] enlarges the set of possible parent vertices,which generates a better initial solution and converges tothe optimal faster than RRT*.PQ-RRT* [7] combines the
P-RRT* with the Quick-RRT* to generate better initialsolutions.
Further research on path quality and planning speed of
the RRT* has specific significance for solving the
problem in complex constraints. Based on constrained
sampling mechanism ,this paper proposes an improved
RRT*algorithm, CSM-RRT*.By comparing with otheralgorithms, effectiveness of the proposed algorithm isverified.

1. Introduction
Rapidly-exploring Random Tree(RRT) is a sampling- basedalgorithm. Many scholars have conducted research on it.
To further improve the speed of the RRT, a double-tree
algorithm RRT-Connect [1] makes the target tree and the starttree grow alternately. To improve the path quality, RRT*
[2] conducts the process of parent node selection andRewire for the samples. As the number of nodes increases,
the path gradually optimizes, but this process willconsume a lot of time.In order to plan the best possible
path within the same time, Informed RRT* [3] adopts anelliptical heuristic domain sampling method andcontinuously shrinks the elliptical region.In order toreach an optimum or near optimum solution at a much
faster rate, Smart-RRT* [4] accelerate the rate ofconvergence. In addition, P-RRT* [5]incorporates the artificial potential field in RRT* to

2. CSM-RRT*
This section introduces the strategies of the CSM-RRT*,
including the following aspects: ChooseParent,
Rewire and constrained sampling mechanism.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
734

Hang Yang, Jiwu Wang, Xueqiang Shang

2.1.2 Dynamic sampling region

2.1. Constrained sampling mechanism

As shown in Fig.3, after the initial path planning iscompleted, each node in the path has an exclusion region,
which contains a large number of redundant nodes andcollision points. For path optimization, the main goal ofpath optimization is to obtain an optimal or asymptotic
optimal path.In general, an optimal path can only beobtained when the path is close enough to the
obstacle.Therefore, redundant and collision nodes close
to the initial path should be employed to concentrate on
meaningful areas, while discarding nodes with higher
path costs.Starting from the node closest to the endpoint,
the nodes with the lowest path cost in the dynamic
sampling domain are selected to continuously improve
path quality. The initial path quickly converges to the
optimal path through cyclic iteration.

2.1.1. Node saved strategy
The regional exclusion mechanism(Fig.1) was proposed, inwhich random sampling points were eliminated in acircular area of the node to ensure the sparsity of theentire tree and improves the exploration efficiency.

Fig.1 regional exclusion mechanism.

X start represents the initial node, Xgoal represents thetarget,Xexcluded represents the nodes abandoned by theregional exclusion mechanism, and the circular dashedline represents the size of the exclusion region.
However, the xrand was eliminated rather thanxnew .Additionally, this method only focuses on thesparsity of the entire tree, without considering the sparsityof individual node. To address this issue,regional
exclusion mechanism and node saved strategy are
combined and applied to the path planning of themanipulator, ensuring the sparsity of both the entire tree
and individual sub-node. The results proved to have goodeffect.
As shown in Fig.2, after adopting the region exclusion
mechanism, the parent node O can extend 6 nodes if the
exclusion radius r is equal to step size, and the distance
between each node is equal to the step size. However, ingeneral, the distance of each node is greater than the stepsize, when a parent node extends 4 nodes. For a node, we
can record the number of expansions and collisions. And
the constraint condition is set to 5 times. When the sumof the number of extensions and collisions reaches 5times, it means that the node has been expanded, or there
are a lot of obstacles around it, and it is difficult tocontinue to expand. In this case, we can save itself toensure that the next sampling node will not appear in thevicinity again. At the same time, the redundant nodes of
the regional exclusion mechanism will also be saved
separately for path optimization.

Fig.3 Dynamic sampling region

With further optimization of the path, dynamic region
will change and contain many nodes. Nodes in dynamic
regions can be included multiple times, resulting inincreased runtime. To accelerate exploration, hash table
is used. In other words, even if a node is included in the
exclusion region many times, its distance to the starting
point only needs to be detected once, and the result is
saved and used directly later to avoid repeated distancedetection.
2.2. ChooseParent and Rewire
RRT* sets Xnear within a certain radius range of the
new node and as candidate parent nodes and candidate
child nodes during ChooseParent and Rewire.The
process is shown in Fig.4.

(a)Node extension range (b) collision detection failed node(a)chooseParent(b)Rewire

Fig.2 Node saved strategy
Fig.4 Process of ChooseParent and Rewire.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
735

CSM-RRT*: an improved RRT*

3. Simulation results and Analysis

Table1.Average of Tinit and T5% of the two algorithms in thesimple environment

In order to ensure the reasonability of the statisticalresults, 50 sets of valid data are gotten for each algorithm
when comparing each indicator. The experimental
environment is shown in Fig.5.

Algorithm

RRT*

Average(Tinit /s)
Average(T5% /s)

2.93
5.46

InformedRRT*
1.72
3.23

CSM-RRT*
0.84
1.56

3.1.2 Cluttered environment

Fig.7 shows the path planning results of the threealgorithms in the cluttered environment,and Table 2
shows the average value of Tinit and T5% .

(a)

(b)

Fig.5 Simulation environment (a)Simple environment(b)Cluttered environment. The entities are obstacles

The results are compared in four directions: initial path
planning time Tinit , the suboptimal path planning time
T5% , the initial path cost Cinit and the suboptimal pathcost Ccost .

(a)

(b)

(c)

Fig.7 Path planning results in the cluttered environment (a)RRT*; (b)Informed-RRT*; (c) CSM-RRT*As shown in Fig.7 and Table 2, it can be seen that as themap becomes more complex, the initial path searchefficiency of the CSM-RRT * algorithm is not assignificant as in a simple environment.However, its path
optimization performance is still much better than otheralgorithms, with an optimization speed increased by 2~5times.The CSM-RRT* algorithm only samples aroundthe initial path, rather than the entire environment,
avoiding path optimization in invalid areas and ensuring
fast convergence to the asymptotic optimal path.

3.1. Compare of algorithm convergence speeds
The convergence speed of the algorithm is compared byTinit and T5% .
3.1.1 Simple environment

Fig.6 shows the path planning results of the threealgorithms in the simple environment, and Table 1 shows
the average value of Tinit and T5% .

Table2.Average of Tinit and T5% of the two algorithms in the
cluttered environment

(a)

(b)

(c)

Fig.6 Path planning results in the simple environment (a) RRT*;(b)Informed-RRT*;(c) CMS-RRT*

Algorithm

RRT*

Average(Tinit /s)
Average(T5% /s)

4.64
12.93

InformedRRT*
2.76
5.02

CSM-RRT*
1.58
2.58

3.2. Compare of path quality
The path quality of the algorithm is compared by Cinit ,Ccost .At the same time, in order to avoid similar
simulation results, the number of algorithm iterations is
set to 2000 for the convenience of viewing algorithm
differences.

As shown in Table 1, in terms of T5% ,the CMS-RRT *algorithm respectively saved 2.3 seconds and 1 seconds
compared to RRT * and Informed-RRT*. The initial path
planning speed is 2.5 times faster than the RRT*
algorithm. Considering the impact of time saved by the
initial path, the CSM-RRT * algorithm did not show
significant performance in path optimization in simple
environments.

3.2.1 Simple environment

It can be seen from Table 3 that the path quality of
CSM-RRT * is not the beset among all initial paths.This
is mainly because the region exclusion mechanism andnode preservation strategy in the CSM-RRT * algorithm
focus more on fast exploration of the initial path.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
736

Powered by TCPDF (www.tcpdf.org)

Hang Yang, Jiwu Wang, Xueqiang Shang

However, the final path cost of CSM-RRT * algorithm
remains minimal, proving that CSM-RRT * is superior toother algorithms.

2.

Table3.Average of Cinit and Ccost of the two algorithms in thesimple environment

3.

Algorithm

RRT*

Average(Cinit )Average(Ccost )712.06703.85

InformedRRT*
707.58697.59CSM-RRT*
716.84692.214.

3.2.2 Cluttered environment

In order to verify the stability of the algorithm, 50
repeated experiments were conducted on algorithms, andthe final path cost simulation results(Fig.8) of the threealgorithms in cluttered environments were obtained.

5.

6.

7.

Perez AT, Karaman S, Shkolnik AC, et al. Asymptotically
optimal path planning for manipulation using incremental
sampling-based algorithms. IEEE/RSJ International
Conference on Intelligent Robots & Systems. 2011 Sep
25-30; San Francisco, USA; pp. 4307–4313.
GAMMELL JD, SRINIVASA SS, BARFOOT TD, et al.
Informed RRT*: Informed RRT*: Optimal Samplingbased Path Planning Focused via Direct Sampling of an
Admissible Ellipsoidal Heuristic[C]//2014 IEEE/RSJ
International Conference on Intelligent Robots andSystems. Chicago, USA: IEEE,2014, pp.2997-3004
Islam, F., Nasir, J., Malik, U., Ayaz, Y., & Hasan, O. .
Rrt*-smart: Rapid convergence implementation of rrt*
towards optimal solution. In 2012 IEEE International
Conference on Mechatronics and Automation ,2012, pp.
1651–1656.
Qureshi AH, Ayaz Y. Potential functions based samplingheuristic for optimal path planning. Auton Robot. 2015,
vol.40(6), pp.1079–1093.
Jeong IB, Lee SJ, Kim JH. Quick-RRT*: triangularinequality-based implementation of RRT*with improved
initial solution and convergence rate. Expert Syst Appl.
2019, vol.123, pp.82–90.
Li Y, Wei W, Gao Y, et al. PQ-RRT*: an improved path
planning algorithm for mobile robots. Expert Syst
Appl.2020,vol.152,pp.113425–113436.

Authors Introduction
Mr. Hang Yang
He is currently studying for a master's
degree in mechanical engineering at
Beijing Jiaotong University. He is mainly
engaged in research on robot path planning.

Fig.8 Path cost results in the cluttered environment

As shown in Fig. 8, the final path cost of the CSM-RRT* algorithm is much lower than those of the otheralgorithms, and the fluctuation between each path result
is very small, indicating stable optimization ability. The
performance indicates that with a limited iterations, theCSM-RRT* algorithm can quickly converge to the
optimal path from initial path every time.

Dr. Jiwu Wang
He is an associate professor, at Beijing
Jiaotong University. His research interests
are Intelligent Robots, Machine Vision,and Image Processing.

4. Conclusion
This paper proposes an improved RRT* algorithm, CSMRRT*, which has some advantages in planning speed ofinitial path, convergence speed and path quality. The
main idea of the CSM-RRT* algorithm is to remove andrecord the redundant nodes and collision points, control
the direction of path growth and optimize paths indynamic sampling region. Compare RRT* and InformedRRT*,it is verified that the CSM-RRT* algorithm has
good applicability to both simple and clutteredenvironments.

Mr. Xueqiang Shang
He is a senior engineer at Aero Engine
Corporation of China, primarily engaged inhydrauliccomponent
designandverification.

References
1.

Kuffner JJ, LaValle SM. RRT-connect: an efficient appro
ach to single-query path planning. Proceedings of IEEE
International Conference on Robotics and Automation.
2000 Apr 24–28; San Francisco, USA. pp. 1–7.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
737

Small Sample Object Detection Based on Improved YOLOv5
Yuxuan Gao

School of Mechanical and Electronic Engineering, Beijing Jiaotong University
Beijing, Haidian District, China
Jiwu Wang

School of Mechanical and Electronic Engineering, Beijing Jiaotong University
Beijing, Haidian District, China
Zixin Li

Aero Engine Corporation of China,Beijing, Changping District, China
Email: 18815511536@163.com, jwwang@bjtu.edu.cn,15311426793@163.com
www.bjtu.edu.cn

Abstract
Object detection is widely used in various production and life, such as mask detection and recognition during theepidemic, face recognition with masks. Object detection algorithm based on deep learning has always been an
important research content and implementation method in the field of object detection. Due to the large number oflead seals and fuses, their locations are not fixed, the lead seals and fuses have difficulties such as few sample datasets,complex target background and easy to be blocked, and strong reflective interference, and the conventional image
processing methods are difficult to solve the problem of effective object recognition. In this study, by expanding thedatasets, using different data enhancement methods, and training in the improved algorithm, the detection accuracy,
detection speed, and adaptability were effectively improved.
Keywords: YOLOv5; Deep learning; Object detection; Few-shot learning

In this study, we used different image processing
methods to further expand the original dataset andperform other data augmentations. At the same time, theautomatic annotation of the dataset is completed throughcode to ensure the accuracy of the annotation, and it istrained in the improved YOLOv5 network to improvedetection accuracy.

1. Introduction
Visual inspection, as a key application in related fields,
provides fundamental support for achieving automationand is a key link in promoting the development ofindustrial automation [1]. Feature based visual detection
methods have been widely used in industrial inspection,
but this traditional detection method relies too much on
manually designed feature extractors based on experience,
and when faced with changes in target morphology andinterference, the detection accuracy will significantly
decrease. With the development of machine vision andartificial intelligence technologies, deep learning has
become a new paradigm for object detection tasks in theindustrial field [2].
However, deep learning relies on a large amount oflabeled sample data during the training phase, which has
drawbacks in real application scenarios. In factories, it isoften difficult to obtain sufficient sample data due toissues such as a single model [3]. In such cases, thelimited number of labeled samples can lead to severe
overfitting of deep learning models and a decrease inrecognition accuracy. Moreover, with sufficient datasets,labeling data can be time-consuming and labor-intensive,
leading to issues of omission and mislabeling duringlabeling.

2. Methodology
2.1. Yolov5 algorithm
The core idea of YOLOv5 algorithm is to treat theobject detection task as a regression problem and segment
the feature map into L×L sized cells, M bounding boxes
are set for each cell to surround the target. Each cell is
responsible for detecting targets that fall within its area.
In one operation, the bounding boxes, localization
confidence, and probability of each category of targets
contained in all cells are predicted. Then, the detectedbounding boxes are filtered using NMS (Non Maximum
Suppression) method to obtain the position, category, andconfidence of the target to be tested.
The implementation of YOLOv5 algorithm can besummarized as the following steps:(1) Preprocessing. The input image is first preprocessed,
including scaling, cropping, and normalization
operations, to convert it into a format suitable for network
input.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
738

Yuxuan Gao, Jiwu Wang, Zixin Li(2) Feature extraction. The input image is fed into the
backbone network and the feature is extracted through the
backbone network.
(3) Feature pyramid network. YOLOv5 utilizes Feature
Pyramid Networks (FPN) [4] to fuse feature maps ofdifferent scales, capture targets of different sizes, andimprove detection accuracy.
(4) Predictive header. YOLOv5 uses a prediction headto predict each cell, obtaining bounding boxes (including
the position and size of the target), confidence(probability of target existence), and class probabilities.
(5) Decoding prediction. Decode and convert the outputof the prediction head into actual image coordinates,
including adjusting the predicted bounding box to theoriginal image size, and calculating the category andconfidence score of each target.
(6) Non maximum suppression. Use NMS algorithm toremove redundant detection boxes with high overlap, andfinally retain the most likely object detection result.
(7) Post processing. Perform post-processing on thedetection results processed by NMS, including
visualizing and saving the results.
The YOLOv5-7.0 network model structure is shown in
Fig. 1, which mainly includes Conv, CSPDarkNet, andSPPF modules.

ReLU activation function in the middle, and the second
full connected layer maps it back to the original numberof channels, This process can be regarded as a bottleneck
structure, which aims to extract the dependencies
between channels. The mathematical expression of thisprocess is shown in Formula (2):𝑠 = 𝐹𝑒𝑥 (𝑧, 𝑊) = 𝜎(𝑔(𝑧, 𝑊)) = 𝜎(𝑊2 𝜎(𝑊1 𝑧))(2)
Where, σ Represents the ReLU function, W1 represents
the parameters of the dimension reduction layer, W2
represents the parameters of the dimension increase layer,and z is the output result from the previous compression
step.
The last step is Re-scale, which multiplies thepreviously learned channel weight vector with theoriginal feature map to achieve the function of adaptively
adjusting the feature response of each channel andimprove the performance of the network model. The
network architecture of its algorithm module is shown in
Fig. 2.
Input
Residual

Re-ScaleC H W
Global pooling

C 1 1

Fully connected

C/r 1 1

ReLU

C/r 1 1

Fully connected

C 1 1

Sigmoid

C 1 1

C H W

OutputFig.2 The diagram of the squeeze-and-excitation
block
Fig.1 YOLOv5-7.0 model structure

Adding the SENet attention module to the last layer ofthe Backbone network can effectively reduce the false
detection rate and missing detection rate of lead seals andfuses, and improve the detection accuracy.

2.2. SENetAttention
In order to obtain attention in the channel dimension,
the input characteristic graph is convoluted to obtain thedimension H × W × C, and then compress the input
characteristic graph through global average pooling. This
step converts the characteristic graph of each channel into
the scalar of 1 × 1 × C, thus forming a channel description
vector, which captures the global spatial information ofeach channel and corresponds to the initial weight of eachchannel. This step is to compress (squeeze),as shown inequations (1) [5]:𝐻

3. Data processing
3.1. image processing
For the image acquisition work of the product object,considering the complex surface structure of the productand the obvious reflective effect of the metal material andpaint surface, the shooting effect is easily affected by thelighting conditions. According to this feature, the productis placed in a variety of lighting conditions, and different
types of camera equipment are used to collect images. At
the same time, local images and unmarked backgroundimages are taken in each lighting environment, in orderto maximize the diversity of target images.Sample isshown in Fig. 3.

𝑊1
𝑍𝑐 = 𝐹𝑠𝑞 (𝑥𝑐 ) =∑ ∑ 𝑥𝑐 (𝑖, 𝑗)
𝐻×𝑊

(1)

𝑖=1 𝑗=1

Where, H and W represent the height and width of thefeature map, and xc represents the input at each channel.
Then the second step is the excitation operation, whichuses the small network of the Fully Connected (FC) layer
to learn the relationship between channels. This network
usually contains two full connected layers. The first fullconnected layer is responsible for mapping the channel
description vector to a low dimensional space, with a©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
739

Small Sample Object Detectiona) Low brightness
environmentb) Normal lighting
environmenta) Original imageb) New samples 1c) New samples 2d) New samples 3c) High brightness
environment

Fig.3 Shooting in various lighting conditions
ImgAug is a powerful and flexible image enhancement
library based on Python language, which is mainly usedin the field of deep learning and computer vision. It
supports a variety of image processing libraries such as
OpenCV. By calling ImgAug's API, the image can betransformed in various aspects such as geometry, color,
contrast, etc., so as to expand the original image data set
and improve the generalization ability and robustness ofthe model. With the help of its official API document, thecorresponding Python code is written, and the image istransformed as follows. The process can be shown in Fig.
4.

Fig.5 New samples after image augmentation
Through the above image enhancement method, each
initial image is expanded to 6 new images, and finally
441 images are obtained as the final data set.
4. Experiment and Comparison
After completing the dataset production and modifying
the YOLOv5 model, conduct training and compare thetraining results of the unimproved model. At the sametime, verify the accuracy of the same image.
Baseline is the yolov5 model without adding SE
attention mechanism, and the training is carried out using
the non-expanded dataset. The dataset is as shown in the
figure. SENet is the yolov5 model with adding SE
attention mechanism, and the training is carried out using
the non-expanded dataset. SENet+dataset is the yolov5
model with adding SE attention mechanism, and thetraining is carried out using the expanded dataset. The
unexpanded dataset has 63 pictures, and the expanded
dataset has 441 pictures, as shown in the following Fig.
6.

Fig.4 Image enhancement process
Write python code to execute the above image
enhancement operation, and realize the automatic
annotation function of new images while realizing
various transformations of images. By reading the XML
annotation file corresponding to the original image, theposition and coordinate information of all the annotation
boxes are extracted, and according to the randomenhancement process of each new image, the code
automatically calculates and adjusts the information of
the annotation box, thus generating a new XML file thatis completely aligned with the target position in the newimage. This method avoids the tedious work of manual
marking and saves a lot of human resources. By
automatically updating the annotation information, theenhanced image is consistent with the original annotation,
thus improving the accuracy and reliability of the model.
The new sample image of the original image after image
enhancement is shown in Fig. 5.

a) Baselineb) Expand the dataset

Fig.6 Image expansion
The key data of the training results are shown in Table
1. Among them, mAP (mean Average Precision) refers to
the average precision, which comprehensively considers
precision and recall to measure the performance of themodel on detecting targets.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
740

Powered by TCPDF (www.tcpdf.org)

Yuxuan Gao, Jiwu Wang, Zixin Li

Table1. Comparison of data under different
training conditions

Baseline
SENet
SENet+ dataset

mAP
（%）
81.4
87.5
96.3

precision
（%）
81.4
89.5
96.9

the measurement accuracy requirements. Compared withthe unimproved method, the method proposed in thispaper improves detection accuracy and accuracy. Thismethod can provide good support for deep learning based
object detection research. However, due to the increased
computational complexity of the algorithm, the running
time is longer. In order to meet the real-time requirementsof object detection, the speed of the algorithm needs to befurther improved.

time
（ms）
9.9
9.3
12.3

From the table, it can be seen that compared to thebaseline reference benchmark, the use of SENet attention
mechanism in mAP has improved by 6.1%, and the
combination of SENet and expanded dataset has
improved by 8.2%; In terms of accuracy, compared toBaseline, the use of SENet attention mechanism
improved by 7.9%, and the combination of SENet andexpanded dataset improved by 7.4%.
The above data results demonstrate the advantages of
SENet+augmented dataset. The best weight files trainedusing Baseline, SENet, and SENet+augmented dataset
are tested on the images in the test set. The image
detection results are shown in Fig. 7.

References
1.

2.

3.

4.

5.

a) Baseline

Fu Binbin Application and development trend of industrial
machine vision [J] China Industry and Informatization,
2021 (11): 18-24
Guo Jing, Luo Hua, Zhang Tao Machine Vision andApplication [J] Electronic Science and Technology, 2014,
27 (07): 185-188
Xing Z, Chen X. Lightweight algorithm of insulator
identification applicable to electric power engineering[J].
Energy Reports, 2022, 8: 353-362.
Lin T Y, Dollár P, Girshick R, et al. Feature pyramid
networks for object detection[C]//Proceedings of theIEEE conference on computer vision and patternrecognition. 2017: 2117-2125.
Hu J, Shen L, Sun G. Squeeze-and-excitation
networks[C]//Proceedings of the IEEE conference on
computer vision and pattern recognition. 2018: 71327141.

b) SENet

Authors Introduction
Dr. Jiwu Wang

He is an associate professor, Beijing
Jiaotong University. His researchinterests are Intelligent Robot,
Machine Vision, and Image Processing.

Mr. Yuxuan Gao

He is a postgraduate in Beijing Jiaotong
University. His research interests are
deep learning and yolo.

c) NSENet+ Expand the dataset

Fig.7 Comparison of detection under different

training conditions
From the figure, it can be seen that the effect issignificantly improved after adding the SENet module
network. However, due to the limited amount of data,
there are still issues of false positives and missed alarms.
After expanding the dataset, it can be clearly observedthat not only is the accuracy further improved, but also
false positives are reduced. In summary, the addition of
SENet attention mechanism and the scalability of thedataset are helpful for model training and improving
detection accuracy.

Ms. Zixin Li

She is an engineer at Aero Engine
Corporation of China. Her research
direction is 3D reconstruction

5. Conclusion
In this study, by improving the YOLOv5 model andexpanding the dataset, the training accuracy is improved.
The experimental results show that this method can meet

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
741

Research on Gas Pore Prediction Method Based on Sand Core Characteristic Time
Xiaolong Wang

School of Mechanical and Electronic Engineering, Beijing Jiaotong University
Beijing, Haidian District, China
Qihua Wu

Weichai Power Co., Ltd.
Weifang, China
Jiwu Wang

School of Mechanical and Electronic Engineering, Beijing Jiaotong University
Beijing, Haidian District, China
Jinwu Kang

School of Materials Science and Engineering, Tsinghua University
Beijing, Haidian District, China
Na Li

Weichai Power Co., Ltd.
Weifang, China
Yucheng Sun

Weichai Power Co., Ltd.
Weifang, China
E-mail: 21121249@bjtu.edu.cn, wuqh@weichai.com, jwwang@bjtu.edu.cn, kangjw@tsinghua.edu.cn,
lin@weichai.com, sunyuc@weichai.com
Abstract
In the production of castings, gas pores are a prevalent defect, particularly in components where air-tightness is
crucial, such as cylinder blocks and heads. These defects primarily arise from the intrusion of gases into the casting
during the combustion of resin in the sand core. Due to the complexity of the sand core's gas evolution and venting
process, predicting gas pores using numerical simulation poses significant computational and time challenges. This
paper introduces a rapid prediction method for gas pores based on the characteristic time of the sand core. By setting
the heat transfer boundaries, initial conditions, and termination criteria for computation, the thermal conductivity ofthe sand core is adjusted. The termination computation time is used as the characteristic time of the sand core. This
time is then compared with a critical characteristic time to predict the distribution of gas pores. As the analysis of
more sand cores is incorporated, the precision of the critical characteristic time improves, leading to more accurate
predictions of gas pores in castings.
Keywords: Gas pore; Numerical simulation; Iron casting; Sand core; Resinfilling and solidification of molten metal, the sand cores
are subjected to heating, causing the organic binder
within them to decompose or burn at high temperatures,
generating gas [1], [2]. If the gas cannot be expelled fromthe core and instead infiltrates the molten metal, it resultsin entrapped gas pores [3]. The factors influencing theoccurrence of entrapped gas pores are numerous, making

1. Introduction
Gas pores are common and typical defects in cast ironand aluminum alloy castings, with entrapped gas pores
being a major type of pore defect. In complex castings
such as engine blocks and cylinder heads, numerous
intricately shaped sand cores are employed. During the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
742

Xiaolong Wang, Qihua Wu, Jiwu Wang, Jinwu Kang, Na Li, Yucheng Sun

it challenging to eliminate. This issue significantly
impacts product quality, sometimes leading to therejection of castings or the discovery of pore defects after
machining. The complex formation mechanism ofentrapped gas pores makes it difficult to predict using
numerical simulation methods accurately [4], [5], [6].
This paper adopts a strategy for identifying hazardous
sand cores and their critical points. Based on theprinciples of similarity in heat transfer within the sandcore and the transport of gas within it, a virtual heattransfer method for sand cores is employed. The paper
proposes a characteristic time indicator for predicting theoccurrence of casting porosity induced by sand cores.
This indicator is used to assess the critical points andseverity of risks associated with sand cores, enabling thedetermination of whether casting porosity will be
induced and predicting the location of gas pore formation.
The characteristic time values are stored in a database forvalidation against actual test results. Furthermore, thesevalues serve as a reference for predicting porosity in new
sand cores.

Heat transfer analysis is the foundation of the resin
burning, gas release and flow, the melt solidification, andthe formation of gas pores. The heat transfer inside thecasting, mold, and sand core during the casting process isexpressed by the Fourier equation,
𝜕𝑇
𝜕
𝜕𝑇
𝜕
𝜕𝑇
𝜕
𝜕𝑇
𝜌𝑐
=
(𝑘 ) +(𝑘 ) + (𝑘 ) + 𝑞 (1)𝜕𝑡 𝜕𝑥 𝜕𝑥
𝜕𝑦 𝜕𝑦
𝜕𝑧
𝜕𝑧
where 𝜌 is the density, 𝑐 is the specific heat capacity,
𝑇 is the temperature, t is the time, 𝑘 is the thermal
conductivity, and q is the latent heat term as the melt
solidifies.
The nominal thermal conductivity of the sand core isadjusted based on the permeability of the sand core, andthe adjustment formula is as follows:
𝐾
𝜆1 =
𝜆
(2)
100 0
where 𝜆0 is the real thermal conductivity of the sand core
(W/(m∙K)), 𝜆1 is the nominal thermal conductivity of thesand core (W/(m∙K)), 𝐾 is the permeability of the sandcore.
The nominal specific heat of the sand core is adjusted
based on the gas generation of the sand core, and theadjustment formula is as follows:
𝐺
𝑐1 =
𝑐
(3)
20 0
where 𝑐0 is the real specific heat of the sand core(J/(kg∙K)), 𝑐1 is the nominal specific heat of the sandcore (J/(kg∙K)), 𝐺 is the gas generation of the sand core.

2. Methods
2.1. Heat transfer analysis and modified thermal
property parameters
Gases
producedfromthehigh-temperaturedecomposition of binders are expelled through exhaustchannels. However, due to the intricate geometries ofsand cores and variations in the distances from different
points to the core head, some points, especially thosefarthest from the core head, may pose challenges in gas
evacuation. This difficulty increases the probability ofporosity formation. The term "distance" here is not
spatial but pertains to the movement of gas within thesand core, exhibiting a certain similarity principle withheat transfer within the sand core. To identify the point
farthest from the sand core, i.e., the most challenging gas
evacuation point, a virtual heat transfer method is
employed. The numerical simulation for virtual heattransfer focuses solely on the specific sand core inquestion, with other sand cores, moldings, and castings
excluded from the simulation. These excluded
components are, however, crucial for automaticallydetermining the core head. This determination isachieved by identifying areas where the sand core does
not contact the casting, signifying the presence of thecore head.

2.2. Determination of dangerous gas penetration
points in sand cores
During the simulation process, six distinct boundaryinterfaces are evident, as depicted in Fig. 1: metal-sandcore, metal-air, metal-sand mold, sand core-sand mold,sand mold-air, and sand core-air. The aforementioned
coating is applied to both the surface of the sand core andwithin the sand mold cavity. Consequently, the interfaces
involving metal-sand core, metal-sand mold, and sandmold-sand core are influenced by the presence of thiscoating. Its inclusion results in heightened thermal andpenetration resistance at these boundaries. Moreover,owing to the coating's exceptionally low gas permeability,
certain segments exhibit zero gas permeability at room
temperature, signifying an incapacity to permit gas
passage.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
743

Research on Gas Pore

3.1. Parameters
Predictions of pores were performed on a single cylinder
head. The casting consists of the intake manifold, exhaust
manifold, upper water jacket, lower water jacket, andchassis core.
Due to the relatively simple geometry of the intake andexhaust manifold cores, only the cores themselves wereconsidered in the determination of dangerous areas
during the assessment of the intake and exhaust cores.
For virtual heat transfer computations, the thermal
property parameters of the cores were set with a densityof 1.6 × 103 kg/m³, a nominal specific heat of 1 × 10 3
J/(kg∙K), and a nominal thermal conductivity of 0.5
W/(m∙K).

Fig. 1 Boundary setting conditions.

Establishing the boundary conditions for the sand core
involves setting the surface temperature of the core headto a constant 1500℃. The remaining exterior surfaces ofthe sand core are considered adiabatic. The initial
temperature of the sand core's elements is set at 25℃.
The virtual heat transfer computation is programmed toautomatically terminate when the volume of the unheatedportion of the sand core reaches 100 mm³, as illustrated
in Fig. 2. This moment is recorded as the characteristic
time of the sand core.

3.2. Gas pore
The virtual heat transfer simulation was conducted for the
casting and sand cores, and the computation was
terminated upon meeting the specified conditions. The
characteristic time for the sand cores was determined tobe 2.1 s. The critical units of each sand core, as illustrated
in Fig. 3, align with the experimental results, where thecritical characteristic time for gas pore generation in thesand cores was identified as 1 second. Consequently, thecasting exhibits gas pores, and the locations of thesepores are depicted in Fig. 4.

Fig. 2 Virtual heat transfer of sand core identifies the mostchallenging gas exhaust point.

When the characteristic time of the sand core exceeds thecritical time for gas pore initiation, it is determined thatthe sand core induces gas pores in the casting. Conversely,
when the characteristic time of the sand core is less thanthe critical time for gas pore initiation, it is determinedthat the sand core does not induce gas pores in the casting.
The topmost units in the unheated portion of the sandcore are selected, and these units are identified as thecritical points for gas pore initiation in the casting. Thesecritical points represent the locations in the sand core
most susceptible to inducing gas pores in the casting.
Save the characteristic time of the sand core to the sandcore characteristic time database, and adjust the criticalcharacteristic time based on the actual occurrence of gas
pores for each sand core.
3.

Fig. 3 Dangerous units in each sand Core

Pore prediction results©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
744

Powered by TCPDF (www.tcpdf.org)

Xiaolong Wang, Qihua Wu, Jiwu Wang, Jinwu Kang, Na Li, Yucheng Sun

Authors Introduction
Mr. Xiaolong Wang
He is currently studying for a master's degree
in mechanical engineering at Beijing Jiaotong
University. He is mainly engaged in researchon intelligent casting and porosity prediction.
Fig. 4 Casting pore results

Mr. Qihua Wu
He is a process engineer at Weichai Power
Co., Ltd., primarily engaged in research on
intelligent manufacturing.

Comparison with castings produced in the actual factory
workshop reveals a consistent distribution of gas pores,
validating the effectiveness of the proposed method.
4. Conclusion

Dr. Jiwu Wang
He is an associate professor, at Beijing
Jiaotong University. His research interests areIntelligent Robots, Machine Vision, andImage Processing.

This paper proposes a gas pore prediction method basedon the characteristic time of sand cores, enabling theforecasting of pores in castings. It addresses thechallenges associated with the complex gas generationand exhaust processes in sand cores, difficulties in
predicting them through numerical simulation techniques,
and the significant costs and time associated with such
simulations. This method provides guidance for practical
casting production. As the analysis of sand cores
increases, the critical characteristic time becomes more
precise, leading to more accurate gas pore predictions.

Dr. Jinwu Kang
He is an associate researcher and doctoral
supervisor
atTsinghua
University,
specializing in the fields of intelligent castingand simulation modeling.
Mrs. Na Li
She currently works at Weichai Power Co.,
Ltd., primarily responsible for the intelligent
manufacturing sector.

References
1.

2.

3.

4.

5.

6.

Daronde, S. et al. The Effect of Vacuum on the Mechanical
Properties of Sand Cast AA6061 Alloy. J. Mater. Eng.
Perform. 31, 262–271 (2022).
Wang, J., Jiang, H. & Jiang, N. Study on the pyrolysis ofphenol-formaldehyde (PF) resin and modified PF resin.
Thermochimica Acta 496, 136–142 (2009).
Wang, X. et al. Study on the Gas Release of 3D-Printed
Furan Resin Sand Core during the Casting Process.
Materials 16, 4152 (2023).
Mocek, J., Kaźnica, N. & Zych, J. Kinetics of Gases
Emission from Surface Layers of Sand Moulds. Archivesof Foundry Engineering; 2018; vol.18; No 1 (2018).
Carlson, K. D., Lin, Z. & Beckermann, C. Modeling theeffect of finite-rate hydrogen diffusion on porosity
formation in aluminum alloys. Metall. Mater. Trans. BProc. Metall. Mater. Proc. Sci. 38, 541–555 (2007).
Qian, X. et al. Gas evolution characteristics of three kindsof no-bake resin-bonded sands for foundry in production.
China Foundry 19, 140–148 (2022).

Mr. Yucheng Sun
He currently works at Weichai Power Co.,
Ltd., primarily responsible for the intelligent
manufacturing sector.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
745

Optimization Analysis of a Deep Learning-Based Model for Predicting Temperature Fields in
the Solidification Process of Castings
Yahui Yang

School of Mechanical and Electronic Engineering, Beijing Jiaotong University
Beijing, Haidian District, China
Jiwu Wang

School of Mechanical and Electronic Engineering, Beijing Jiaotong University
Beijing, Haidian District, China
Jinwu Kang
School of Materials Science and Engineering, Tsinghua University
Beijing, Haidian District, China
E-mail: 22126065@bjtu.edu.cn,jwwang@bjtu.edu.cn,kangjw@tsinghua.edu.cn

Abstract
In the foundry industry, accurate prediction of the temperature field of castings during solidification is crucial toensure high product quality and improve productivity. This study provides an in-depth investigation of the Unet deeplearning model used for temperature field prediction during solidification, focusing on the impact of key training
parameters such as optimiser selection, batch size, number of iterations, and loss function selection on the prediction
performance of the Unet model. The results of the study demonstrate the analysis of selecting and tuning theseparameters to improve the accuracy and reliability of model predictions. The results of this study not only provide
new ideas for the practice of deep learning applications in the foundry industry, but also help to improve the accuracy
and efficiency of the production proces.
Keywords: casting, deep learning, temperature field prediction, parameter optimization.

field during the solidification process of castings. The UNet [4], [5] network demonstrates exceptional
performance in the field of image segmentation,
particularly in achieving precise segmentation withsmaller datasets.

1. Introduction
In recent years, deep learning has expanded in field
prediction, simplifying traditional numerical simulations[1], [2] while maintaining accuracy. Specifically, incasting solidification, data-based methods predict
temperature fields [3] effectively. However, incorrect
network structures or training parameters can cause
underfitting or overfitting, leading to errors. Thus,
optimizing these parameters is vital for enhancing
accuracy, reducing costs, and speeding up design
iterations for precise temperature field prediction in
casting processes.
2. Model Introduction

Fig.1. Unet network model [6]In the prediction of temperature field during castingsolidification, due to the time-consuming computation of

Selecting an appropriate network model structure is
crucial for high-precision prediction of the temperature©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
746

Yahui Yang, Jiwu Wang, Jinwu Kang

the finite difference method, it is difficult to generate alarge amount of training data, and the encoder-decoder
network architecture is able to efficiently extract thesemantic features of the image and retain the positional
information, which is suitable for prediction tasks withsmall datasets. The Unet network model is shown in Fig.
1.
In this study, we employ a network model (Fig. 1) withthe classic U-Net structure to predict the temperaturefield during the solidification process of castings. The
model takes the mold images of sand castings as input,
and the output is the temperature field image after acertain time step. Both input and output are singlechannel images.

achieve optimal neural network prediction capabilities,
this study delves into several key factors affecting modeltraining, including the choice of optimizer, batch size,
number of iterations, and the selection of loss functions.
The following sections will present a detailed
comparative analysis of these crucial elements, aiming toelucidate how to optimize deep learning network
structures for higher accuracy in predictions. Thistranslation is intended for application in academic papers.
3.1. The Impact of Different Optimizers on thePrediction Accuracy of the Temperature Field
In this set of comparative experiments, the batch size
was set to 32, and the number of epochs was fixed at 200.
Mean Absolute Error(MAE) was chosen as the loss
function. To analyze the impact of different optimizers,
three distinct optimizers were selected for comparison:
Adam, Nadam and SGD. The learning rate for each was
adjusted using a cosine annealing learning rate strategy.
The experimental results are presented in Table 1.

3. Optimization Methods
In the realm of deep learning network architectures, theaddition or alteration of various modules cansignificantly impact the final predictive performance. To

Table 1. Comparison of results of different optimizers
Serial Number

Optimizer

Accuracy

Pearson Correlation Coefficient

Loss

Val-Loss

Training Time

1

Adam

0.9156

0.99890.00960.023066727.5657

2SGD

0.81980.76850.19770.176968961.8629

3

Nadam

0.91180.99890.00990.023665335.4521

randomly selected for visual analysis. Fig.2 demonstrates
the distribution of errors between the temperature fields
predicted by the model and the actual temperature fields,
under three different optimizers: Adam, Nadam, and SGD.
3.2. The Impact of Different Batch Sizes on
Prediction Accuracy
In this set of comparative experiments, building upon the
results of the previous set, Adam was selected as the
optimizer, with the number of epochs set to 200 and MAE
employed as the loss function. To investigate the impact ofbatch size on model performance, three different batch
sizes (16, 32, and 64) were established for comparative
analysis. The experimental results are shown in Table 2.

Fig.2. Error Plots of Predicted versus Actual Values Using
Three Optimizers

According to the data presented in Table 1, the model
demonstrates higher accuracy and correlation whenemploying the Adam optimizer, with the smallest loss
value observed. To further analyze and compare theseresults, samples at a specific moment (t=10 minutes) wereTable 2. Comparison of results for different batch sizes (batch size)
Serial Number

Optimizer

Accuracy

Pearson Correlation Coefficient

Loss

Val-Loss

Training Time1

16

0.90140.98890.0181

0.021768537.3182

2320.91180.99890.00960.023066727.5657

364

0.80890.99790.02140.028666357.1713©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
747

Optimization Analysis of a

Based on the data analysis from Table 2, it is observedthat setting the batch size to 32 results in higher model
accuracy and correlation, with the smallest loss value andshorter training time. Samples at the same moment (t=10minutes) were selected for visual analysis.

Error (MAE) used as the loss function. To explore theimpact of different numbers of epochs on model
performance, three distinct numbers of epochs (50, 200,
and 500) were established for comparative analysis. The
experimental results are shown in Table 3.
According to the experimental data in Table 3, the model
exhibits the highest accuracy and correlation, with thesmallest loss value and relatively short training time, when
the number of epochs is set to 200. With an increase in thenumber of epochs, the computational cost correspondingly
rises. Therefore, when selecting the optimal parameters forthe model, it is crucial to consider both prediction accuracy
and computational cost as key factors in determining themost suitable parameter settings. Fig. 4 displays error
distribution of the temperature field at different numbersof epochs (50, 200, 500), providing a more intuitive
assessment of model performance.

Fig.3. Error Plots of Predicted versus Actual Values for Three
Batch Sizes

Fig. 3 illustrates the distribution of errors between thepredictedand actual values of the model under threedifferent batch sizes (16, 32, and 64). The model exhibits
relatively smaller prediction errors when the batch size is
set to 32.
3.3. The Impact of Different Iteration Counts on
Prediction Accuracy
In this set of comparative experiments, building upon the
results of the first two sets, Adam was chosen as the
optimizer, with the batch size set to 32 and Mean Absolute

Fig.4. Error Plots of Predicted versus Actual Values for Three
Different Epoch Counts

Table 3. Comparison of results for different number of iterations (epoch)
Serial Number

Epoch

Accuracy

Pearson Correlation Coefficient

Loss

Val-Loss

Training Time

150

0.72640.89630.03020.0361

17488.4049

2200

0.91180.99890.00960.023066727.5657

3500

0.90830.99885.3330e-04

0.0040157621.8659

3.4. The Impact of Different Loss Functions on
Prediction Accuracy
In this set of comparative experiments, based on the
results of the previous three sets, we selected the Adam
optimizer with the number of epochs set to 200 (epoch =200). Three different loss functions were chosen forcomparative analysis: Mean Squared Error (MSE), Mean
Absolute Error (MAE), and their combination (MSE +MAE). The experimental results are presented in Table 4.

Fig.5. Error Plots of Predicted versus Actual Values for Three
Different Loss Functions

Based on the experimental data presented in Table 4, thehighest accuracy and correlation, along with the smallest

Table 4. Comparison of results of different loss functions
Serial Number

Loss Function

Accuracy

Pearson Correlation Coefficient

Loss

Val-Loss

Training Time

1MSE

0.92930.99904.9873e-04

0.003864514.8804

2

MAE

0.91180.99890.00960.023066727.5657

3

MAE+MSE

0.90830.99885.3330e-04

0.004067621.8659©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
748

Powered by TCPDF (www.tcpdf.org)

Yahui Yang, Jiwu Wang, Jinwu Kang

6. Han xiao. Research on Temperature Field Prediction

loss value and relativly short training time, are observed
when Mean Squared Error (MSE) is used as the loss
function. Thus, choosing MSE as the loss function is the
optimal parameter choice for the model. Fig. 5 illustrates
the visualization of the error distribution of the temperaturefield predictions when different loss functions (MSE,
MAE, and MSE+MAE) are employed.

Method of Sand Mold Casting Solidification Process Basedon U-NET [D]. Beijing Jiaotong University,2022

Authors Introduction
Ms. Yahui Yang
She is currently pursuing her master's
degree in Mechanical Engineering at
Beijing Jiaotong University. She is mainly
engaged in the research on deep learning
analogue simulation.

3.5. Experimental Results
In the four comparative experiments conducted, we
determined that the Adam optimizer is the optimal choice,surpassing Nadam and SGD in accuracy, correlation, andloss minimization. The model exhibits its best performancewith a batch size of 32, effectively balancing efficiency andaccuracy. Setting the number of epochs to 200 achieves abalance between computational cost and precision. Amongthe loss functions compared, Mean Squared Error (MSE)significantly excelled in predictive accuracy andminimizing loss.

Dr. Jiwu Wang
He is an associate professor, at Beijing
Jiaotong University. His research interests
are Intelligent Robots, Machine Vision,and Image Processing.

Dr. Jinwu Kang
He is an associate researcher and doctoral
supervisor at Tsinghua University,
specializing in the fields of intelligent
casting and simulation modeling.

4. Conclusion
In this study, we employed a data-driven Unet neuralnetwork model for temperature field prediction, optimizing
its structure and training parameters (optimizer, batch size,
epochs, loss function). The model, trained on 128x128
pixel image data, showed optimal performance with theAdam optimizer, a batch size of 32, 200 epochs, and Mean
Squared Error (MSE) as the loss function, achieving an
accuracy of 0.9293 and a correlation of 0.9990. Thesefindings offer new insights and valuable references for theapplication of deep learning in the casting industry.
Reference
1. Lee S M, Lee W J. Finite-element analysis on
thermornechanical behavior of a marine propeller casting inthe sand-casting process[J]. JOURNAL OF MATERIALSENGINEERING AND PERFORMANCE, 2005, 14(3):388-394.
2. Liu S, Bai L, Wang B, el. Numerical simulations ofsolidification and hot tearing for continuous casting of
duplex stainless steel[J]. JOURNAL OF IRON ANDSTEEL RESEARCH INTERNATIONAL, 2020, 27(6):643-655.
3. Cai S, Wang Z, Wang S, et al. Physics-Informed Neural
Networks for Heat Transfer Problems[J]. Journal of Heat
Transfer, 2021, 143(6): 060801.
4. Gao W, Zhao J, Peng S. UNet–Based Temperature
Simulation of Hot Dry Rock in the Gonghe Basin[J].
Energies, 2022, 15(17): 6162.
5. Raissi M, Perdikaris P, Karniadakis G E. Physics-informed
neural networks: A deep learning framework for solving
forward and inverse problems involving nonlinear partial
differential equations[J]. Journal of Computational Physics,
2019, 378: 686-707.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
749

Portable Green Energy Mobile Laptop Charging Station
Ammar A.M. Al-Talib

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Rodney Tan

Department of Electrical and Electronic, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Ang Aun Jie

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Noor Idayu Mohd Tahir

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Sarah ‘Atifah Saruchi

Faculty of Manufacturing and Mechatronic Engineering Technology,
Universiti Malaysia Pahang Al-Sultan Abdullah, 26600 Pekan, Pahang, Malaysia
Cik Suhana Hassan

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Amar Ridzuan Abd Hamid

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
E-mail: ammart@ucsiuniversity.edu.my, rodneytan@ucsiuniversity.edu.my, 1001747031@ucsiuniversity.edu.mynooridayu@ucsiuniversity.edu.my, sarahatifah@umpsa.edu.my, suhana@ucsiuniversity.edu.my,amar@ucsiuniversity.edu.my
Abstract
Mobile phones and laptop computers require electrical power to recharge when the battery is down. As a result, it
would be very useful if a portable charging station derived from renewable energy harvesting could be built, so that
individuals could recharge their phones and laptops whenever needed. The objective of this project is to design anddevelop a green energy mobile and laptop charging station that uses wind and solar energy and evaluate theperformance of the designed station under different working conditions. The efficiency of the power generated tocharge the station is achieving 95.6% for solar charging, which is considered a high efficiency for a renewable energy
charging station. Based on the analysis of the charging station results, it has been proven that it can provide sufficient
power and is safe for use as a portable mobile laptop charging station.
Keywords: Energy conversion, Portable power supply, Universal mobile laptop charging

1. Introduction
Electricity is a vital part of modern life and for acountry's economic growth and development. Over the
next two decades, average power consumption ispredicted to nearly double, with overall electricity
demand expected to rise by 2.3 percent per year on
average from 13,290 billion kWh in 2001 to 23,072
billion kWh in 2025 [1]. In Malaysia, home electric
power consumption has grown substantially over the past
years, resulting in high demand for energy to satisfy
increasing social and economic activities.

Fossil fuels are the most famous resources used.
However, these fuels are creating huge carbon emissions
to the atmosphere which has negatively impacted theenvironment. As the demand for electric power rises,
additional power generation is required to meet thedemand. Due to the depletion of conventional resources,
renewable energy resources are the most suitable
resources for generating electric power with little
environmental effect. Renewable or non-conventionalenergy are tidal energy, wind energy, hydropower,
biomass, geothermal energy, and solar energy [2]. It also

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
750

Ammar A.M. Al-Talib, Ang Aun Jie, Idayu M.Tahir, Sarah Atifah Saruchi, Cik Suhana Bt. Hasan, Amar Rizwan, Rodney Tancontributes to the achievement of Sustainable
Development Goal 7 (SDG7), which calls for universal
access to sustainable and renewable energy by 2030.
This project's goal is to create a portable dual-source
charging station that can charge mobile phones andlaptops using both solar and wind energy. A hybrid
power system is a system that combines two or more
renewable energy sources to provide continuous energy
generation to fulfil load needs. Researchers havediscovered that a combined solar and wind hybrid systemis the most reliable and efficient, as well as being
ecologically friendly and affordable in cost [3]. Several
recent research [4], [5], [6] have combined two nonconventional sources to create a dual-mode portablemobile charger. Wind and solar hybrid systems formobile chargers with minimal operating costs have been
designed in this research. Additional components arenecessary for laptop charging, which necessitates the useof an AC power source. When constructing a portablecharging station for mobile and laptop devices, certain
specifications and data are critical.

2.1. Fabricated prototype
The prototype has been successfully built using

Fig. 1 A block diagram of Hybrid Solar-wind Energy system[7]

2. System overview
Major components of the system include power inverter,
battery bank, system and battery controller, photovoltaic
modules, wind turbines, and, in some cases, theprescribed electrical demands. Fig. 1 from MEDA [7]illustrates the system's basic working principle. The
acquired energy from the wind turbine and PV array willbe transformed into electrical DC power and stored in thebattery bank. For the system under research, amonocrystalline solar panel and a vertical axis windturbine are recommended. As compared topolycrystalline
andthin-film
solarpanels,monocrystalline solar panels offer the highest efficiencyof 15–20 percent [8]. According to the research [9],vertical axis wind turbines are more efficient to build anduse in Malaysia's tropical and windy weather. A charge
controller for wind and solar power is required tomaintain your batteries properly supplied and avoid
overcharging. For DC loads, the electric power stored in
the battery bank may be utilized directly, however for AC
loads, an inverter is required to connect to the battery
bank. A power inverter is an electrical device thatconverts direct current (DC) from a battery bank toalternating current (AC). Finally, the system can be used
to charge mobile phones and laptops.

Fig. 2 Full components of prototype
No.
1
2
3
4
5
6
7
8

Table 1 List of prototype components
Component
Specifications
Power Inverter
200W
Dokio Solar charge
controller
PWM Solar Charge
controller
Battery Indicator
Voltmeter
Dokio Solar Panelconnector
SLA Battery
Plastic Ammo Box
4-in-1 output panel

30A
10A
N/A
N/A
12V, 7.2Ah
N/A
Voltmeter, switch,
USB ports, 12V
cigarette port

polystyrene boxes, a plastic ammo box, and cardboard.
The components of the prototype are labelled in Fig. 2and listed in Table 1. Fig. 3 shows the complete
fabrication of the prototype.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
751

Portable Green Energy Mobile

evaluate the performance of the designed station under
different working conditions. A multi-meter has beenused to measure the current and voltage output from thesolar panel and wind turbine, and then the energy output
over the state of the battery of the experiments have beenmeasured by the charge controller. The experiments have

The portable charging station consists of a Dokio solar
charge controller, a PWM solar charge controller, abattery indicator voltmeter, a 12V, 7.2Ah GP SLA
Battery, a 200W power inverter, and a 4-in-1 output
panel as shown in Fig. 4. All the components in the circuit
are linked in parallel with the 12V 7.2AH SLA Battery,

Table 2. Data collected for four different experiments conducted
Experiment

Fig. 3 The completed prototype

1

T
ri
al

1
2
3
4
5
6
1

Aver
age
Volta
ge
Gene
rated
(V)
12.75
12.67
13.50
13.24
12.85
12.72
12.41

Aver
age
Curre
nt
Gene
rated
(A)
0.935
0.873
1.450
1.140
0.982
0.955
0.340

ensuring that the station can function even if one of thecomponents fails. A switch, voltmeter, USB ports, and a
12V cigarette port are all included in the 4-in-1 output
panel. The charging station can be turned on and themobile phones can start charging through the USB port,
while a power inverter must be connected to the 12V
cigarette port to convert power into AC power for laptop
charging.

Spee
d of
Win
d
(m/s
)

734.95679.46
839.69837.85760.80745.23

-

-

-

238.1
7

-

451.9
9
493.1
9

12.62

0.450

3

12.65

0.480

19:09

1

11.30

0.250

23:00
(Not
complete
d)
23:12
(Not
complete
d)
22:30
(Not
Complete
d)
14:51

2

3

4

Light
Inten
sity
(W/
m2)

2

3

3. Result and Discussion

Solar
Irradia
nces
(W/m2
)

2:42
3:01
1:08
1:36
2:14
2:38
18:18
(Not
complete
d)
24:25

2

Fig. 4 The charging mobile phone and laptop with using thecharging station

Charging
Duration(Hour:
minute)

1

11.32

11.38

12.64

0.275

0.300

0.750

3.0

-

-

-

-

-

-

-

474.6
1

3.3

3.6

The system has been tested to evaluate the performanceof the designed station. The equipment used in theexperiments was an 80 W Mono-solar panel, a 100 W
wind turbine, and a 1000 W halogen lamp to generate
electrical power. Several experiments have been
conducted and the data were recorded and analyzed to©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
752

-

3.6

Ammar A.M. Al-Talib, Ang Aun Jie, Idayu M.Tahir, Sarah Atifah Saruchi, Cik Suhana Bt. Hasan, Amar Rizwan, Rodney Tanbeen conducted to fully charge the 12V 7.2Ah Sealed,
Lead Acid Rechargeable Battery for each trial ofexperiments.
Experiments 1 and 2 are solar power charging for thedesigned station, with experiment 1 using sunlight togenerate electrical power outdoor under the sun, whether
experiment 2 is by utilizing a halogen lamp as the sourceof light to simulate the solar energy. The wind power
charging for the station is evaluated with a power fan inexperiment 3 to simulate the wind energy because of the
low wind speed available outdoors. Experiment 4 is ahybrid solar and wind power charging system for thestation, with a halogen lamp and a power fan used indoors
to get more consistent and precise results. Table 2 shows
the data collected for the four different experiments
conducted.

Fig. 7 Overall average current generated for experiment 2.

Fig. 8 Overall average light intensity received forexperiment 2.

Fig. 5 Overall average current generated and solar irradiancefor experiment 1

Fig. 9 Overall average current generated with three different
speeds for experiment 3.
The average overall electrical current generated andoverall average solar irradiances received for experiment
1 are plotted and illustrated in Fig. 5 and Fig. 6 based on
the data obtained. In Fig. 7 and Fig. 8, the overall average
current generated and overall average light intensity
received for experiment 2 are plotted and shown. Fig. 9
illustrates the relationship between current generated andwind speed.
According to Fig. 5 and Fig. 6, it is shown that the higher
the solar irradiances come the higher the current
generation output result. The results reveal that even thelowest solar irradiation of 679.46 W/m2 can generate an
average of 12.67 volt and 0.873 amperes.
Fig. 7 and Fig. 8 are showing that as the light intensity
increase, the current generated and output power increase.

Fig. 6 Overall average solar irradiance received forexperiment 1.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
753

Portable Green Energy Mobile

3.1. Result comparison

As the solar panel receives more light intensity from thehalogen lamp, the current generated is increasing as wellas the power output. Although the light source to the solar
panel is stable, the current and power generation isrespectively low which is not utilizing the sunlight that
contains higher solar irradiances. As a result, it also
corresponds to the research done by [10], whereby thecurrent rises steadily with an increase in solar irradiancesor light intensity.
Fig. 9 shows that when the wind speed increase, thecurrent generated will also increase. However, it can still
be observed that the current is increasing with the speedof wind increase which gets a maximum of 0.3A with a
3.6 m/s wind speed. As a result, it can be concluded thatthe greater the wind speed, the greater the magnitude of
the current.
From Table 2, it is shown that an average of 0.75A is
generated with an average light intensity of 474.61 W/m2
and a constant speed wind of 3.6m/s. The results havebeen taken when both solar and wind harvesting systems
are working together to charge the station. The duration
to fully charge the station is around 14.85 hours which isfaster than experiments 2 and 3.
The duration to fully charge the battery is also dependent
on the renewable power supply. The higher the power
supply to charge the station, the shorter the duration of
the battery to be fully charged. The fastest battery of 7.2
Ah capacity could be fully charged in around 68 minuteswith a charging current of 1.45 A. To protect the battery,the battery is recharging at 12 V which is around 50 % to
avoid irreversible damage occurring. The fully charged
battery can fully charge an iPhone 11 with a battery
capacity of 11.9 Wh for 5.5 cycles and an Asus gaming
laptop with a battery capacity of 48 Wh for 1.8 cycles.

The average experimental results from experiment 1
through experiment 4 are plotted and illustrated in Figure
10. The average result is calculated by averaging themeasurements based on the state of charge of the battery,which acts as a comparison point. This is because thecharge controller will supply power following thebattery’s status, which means that the power delivered
will gradually grow following the status of the battery to
prevent the battery from being damaged.
According to Fig. 10, experiment 1 is the most efficientFig. 10 Average experimental result from experiment 1 toexperiment 4.
way, taking just 2.2 hours to fully charge the station withan average voltage of 12.95V and a current of 1.06A. It
took around 7.31 hours less time than the expected result
based on calculation. Even though the readings fluctuatedduring the experiment, it was still the most efficientmethod for solar charging when compared to experiment2 which utilized a halogen lamp as the light source.
From Fig. 10, experiment 2 took an average of 20.62
hours to completely charge the station battery, which isaround 18.4 hours longer than experiment 1. With anaverage of 12.56 V and 0.42A, the power generated was
likewise significantly lower than experiment 1. It tookaround 3.38 hours less time than the expected result
based on calculation. As a result, the most effective
approach for successfully charging the station batteries
for the solar harvesting system is charging with a solar
panel directly under the sun, which will maximize thepower generated among other methods.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
754

Ammar A.M. Al-Talib, Ang Aun Jie, Idayu M.Tahir, Sarah Atifah Saruchi, Cik Suhana Bt. Hasan, Amar Rizwan, Rodney Tan

Acknowledgment

Due to the low wind speed available in Malaysia,
experiment 3 aims to see how much power can begenerated at various wind speeds. It only can generate an
average of 11.33V and 0.275A, which only charged
station batteries to 60% in an average of 22.8 hours. Thisis because the charge controller utilized for the wind
harvesting system is a solar charge controller, resulting
in the charge controller power output being unstable andlow. Furthermore, the solar charge controller is only
designed for solar power systems with electricalcharacteristics that differ from those of wind power,
resulting in low power received to charge the station. As
a result, by adopting an appropriate wind charge
controller, the wind power charging system may be more
effective.
Experiment 4 feeds the station using both solar and wind
harvesting technologies. It aims to see if operating the
combined harvesting system at the same time as charging
the station is more efficient than working independently.
According to Figure. 10, the result showed an average of
12.64V and 0.75A, and it took an average of 14.8 hours
to fully charge the station. However, the required
duration to completely charge took 1.41 hours longer
than the theoretical calculation. This outcome might be
impacted by the fact that an inappropriate charge
controller is used for a combined solar and wind
harvesting system that prolongs the duration to be fully
charged. As a result, it can be proven that the combined
solar and wind harvesting system has a better
performance than a system that only harvested wind or
solar.

The authors would like to express their gratitude
towards the Faculty of Engineering, Technology and
Built Environment at UCSI University, for the labs andfacilities provided during the research.
References
1. N.A. Mahusin and R. Baharun, Utility Consumption
among Malaysian Electricity Users in Government
Buildings. International Symposium on Technology
Management and Emerging Technologies, 2014.
2. P. Pranjela and K.H Deepali, Non Conventional Sourcesof Energy-Applications of Solar Energy in Architectural
Buildings, International Journal of Computer Applications
2013
3. K.V. Anila, Design of a Hybrid Power Generation System
Using Solar-Wind Energy, M.S. Thesis, Deakin
University, Geelong, Australia, 2018.
4. R. Chakma, T. Chawaphan, K. A. A. Mamum, A. Chakma,
S. Harun, Portable Smart Phone Charger Using Human
Mechanical Energy by Gear Train with Hand Crank,
Journal of Electrical and Electronics Engineering, Vol 12,
Issue 3, pp. 20-25, June 2017.
5. Md. A. Rahaman, N. Hoque, N. K. Das, F. N. Maysha, MD.
M. Alam, Portable Dual Mode Mobile Charger with Hand
Crank Generator and Solar Panel, Indonesian Journal of
Electrical Engineering and Computer Science, Vol. 1, No.
2, February 2016, pp. 282 ~ 287.
6. P. Vijay, T. Manglani, P. Kumar, R. Meena, A. Khedia,
Wind and Solar Mobile Charger, International Journal of
Recent Research and Review, Vol. 7, Issue 4, Dec 2014.
7. Mahrashtra Energy Development Agency (MEDA), SmallWind Energy and Hybrid Systems Programme.
https://www.mahaurja.com/meda/en/off_grid_power/sma
ll_wind_solar_hybrid
8. A. Saleem, K. Mehmood, F. Rashid, The Efficiency ofSolar PV System, Proceedings of 2nd International MultiDisciplinary Conference 19-20 December 2016.
9. M. K. Johari, M. A. A. Jalil, M. F. M. Shariff, Comparisonof horizontal axis wind turbine (HAWT) and vertical axis
wind turbine (VAWT), International Journal ofEngineering & Technology, Vol 7, no 4, (2018), pp 74-80.
10. Z. Li, J. Yang, P. A. N. Dezfuli, Study on the Influence ofLight Intensity on the Performance of Solar Cell,
International Journal of Photoenergy, vol. 2021, 2021.

4. Conclusion
In summary, the design of a mobile dual source
charging station, by using wind energy and solar energy,
has been fabricated with several materials such aspolystyrene boxes, cardboard, and plastic ammo toolbox.
The performance of the designed station under different
working conditions was investigated and evaluated using
handphone and laptop as load. Based on the comparison,
the designed station has the best performance when bothsolar and wind harvesting systems are working at thesame time. Therefore, this study is promising to develop
a clean energy portable charging station that isenvironmentally friendly and able to act as a charging
station when there is no power supply available.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
755

Powered by TCPDF (www.tcpdf.org)

Portable Green Energy Mobile

Sarah ‘Atifah Saruchi
She graduated from Nagoya University,
Japan in Mechanical and Aerospace
Engineering. She received her Masterand Doctoral degrees from MalaysiaJapan
International
InstituteofTechnology(MJIIT),
Universiti
Teknologi Malaysia. Currently, she isworking as senior lecturer at Universiti Malaysia Pahang
Al-Sultan Abdullah, Pahang, Malaysia.
Her researchinterests include mechatronics and artificial intelligence

Authors Introduction
Ammar Abdulaziz Al T
He has finished his B.Sc and M.Sc
degrees in Mechanical Engineering fromthe University of Mosul/Iraq.
He has finished his Ph.D degree fromUPM University / Malaysia.
He is also a Chartered Engineer andMember of the Institute of MechanicalEngineers / UK. (CEng. MIMechE). He
has developed all the Postgraduate Programs at the Facultyof Engineering at UCSI University / Malaysia, and worked
as the Head of Postgraduate and Research department at
the same faculty for the years 2010-2018.
alib

Cik Suhana Hassan
She is an Assistant Professor at the
DepartmentofMechanicalandMechatronics Engineering of UCSI
University. She received her bachelor's
and master's degrees in 2009 and 2011,
respectively, from Universiti Teknologi
PETRONAS, and her PhD in 2019 from
Universiti Putra Malaysia. Her research interests include
the investigation of bio-composites for use in automotive
applications. She is also an active member of the materials
community, having been recognized as a Professional
Member of the Institutes of Materials Malaysia and a
Professional Technologist of the Malaysian Board ofTechnologists in the Material Science Technology field

Rodney Tan
He currently is an Associate Professor
at UCSI University, Malaysia. He
received his PhD degree in ElectricalEngineering in 2013 from UNITEN
Malaysia. His research interests include
Renewable Energy, Energy Storage
System and Power Electronic Converter.
Ang Aun Jie
He graduated from the MechanicalEngineering Department at UCSI
University in 2021, and currently is
working as an Engineer.

Amar Ridzuan Abd.Hamid
He is a Professional Technologist, Headof Programmes and lecturer of
MechanicalandMechatronic
programmes from Department of
MechanicalEngineering,
UCSI
University, Malaysia. He received his
Bachelor’s degree in automotiveengineering (Hons) from UTeM, Melaka in 2011 and his
Master’s degree in Mechanical Engineering from UPM,
Selangor in 2015. His research interests are in the design
and analysis of automotive compartments.

Noor Idayu Mohd Tahir
She graduated from University
Teknologi MARA, Malaysia in
Mechanical Engineering. She wasdesign engineer in oil and gas industries
for 7 years. Currently, she works as alecturer under the Department of
Mechanical and Mechatronics, Facultyof Technology and Built Environment, UCSI Kuala
Lumpur. Her research interests include design andautonomous robotics.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
756

Auto Indoor Hydroponics Plant Growth Chamber
Ammar A. M. Al-Talib

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Tew Hwa Hui

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Sarah ‘Atifah Saruchi

Faculty of Manufacturing and Mechatronic Engineering Technology,
Universiti Malaysia Pahang Al-Sultan Abdullah, 26600 Pekan, Pahang, Malaysia
Noor Idayu Mohd Tahir

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Nor Fazilah Binti Abdullah

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
E-mail: ammart@ucsiuniversity.edu.my, 1001955085@ucsiuniversity.edu.my, sarahatifah@umpsa.edu.my,nooridayu@ucsiuniversity.edu.my, norfa@ucsiuniversity.edu.my

Abstract
The objective of this project is to build an auto indoor hydroponics plant growing chamber that has an auto monitoring and controllingsystem. A ESP32 based hydroponics electrical system is built with the attachment of hardware components such as temperature andhumidity sensor, light intensity sensor, water level sensor, and water flow rate sensor. The software development of the system is throughArduino IoT Cloud platform, which has an overall suitability in terms of features, cost, and user intuitiveness for starters. Results haveshown that ESP32 can ensure stable power supply. After testing and validation, all of the electrical components are stored in a power
enclosure box to prevent contact with liquid. In short, the developed auto indoor hydroponics plant growth chamber has effectively
demonstrated the ability in easing the plant cultivation procedure for agricultural community.
Keywords: Auto Indoor Hydroponics, ESP32, Controlling and Monitoring Hydroponics

All the aforementioned issues have made it difficult toprovide adequate and nutritious food, particularly inurban areas. Therefore, new methods of producing
enough food must be developed in order to sustainably
feed the world's expanding population. New and modern
farming techniques marked a significant technologicaladvance for humanity. The substitution of liquid as the
new growing medium for traditional growing medium
can be the different approach for consistent crop
production, preservation of rapidly depleting land andfood availability. On that being in case, the rapid growthof hydroponic system is getting attention and increasing

1. Introduction
The global population is expected to reach 9.8 billion in
2050 [1]. Meanwhile, scientists anticipate that a growing
population will result in a decline in the amount ofavailable land. The scarcity of land in the future ispredicted to become a bottleneck in all industries, but
especially in agriculture [2]. Not to mention, a lack ofavailable land will make traditional farming difficult andincrease the likelihood of a food crisis due to rising fooddemand but low yield rates [3].

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
757

Ammar A. M. Al-Talib, Tew Hwa Hui, Sarah ‘Atifah Saruchi, Noor Idayu Mohd Tahir, Nor Fazilah Binti Abdullah

widespread all over the world. It is accounted to have aworld growth of 18.8% from 2017 to 2023 [4].
Hydroponics is derived from Greek word means Hydro“water” and Pons “labour”. It is a method to grow various
plants in water containing fertilizer without any soil byusing different inert mediums such as sand, gravel,
rockwool that act as a mechanical support [5]. The
growth of hydroponics plants must be in a controlledenvironment and often affected by different parameterssuch as electrical conductivity in nutrients, nutrients
circulation rate, PH, light intensity air and watertemperature. Hence, the plant can have growth in an
optimal condition with frequent monitoring for better
yield rate of the plants [2].
Therefore, this project aims to develop an auto indoorhydroponic system prototype with “auto” monitoring andcontrolling features with IoT capabilities.

Fig. 1. Water, Nutrient and pH Monitor and Control System

2.2. Temperature and Humidity Monitoring andControlling System
Temperature and Humidity around the plants is crucial as
it may affect the growth of plants in different temperatureenvironments.
Fig. 2. refers to the system used by S. Seni et al, 2020.
[7]. It is an indoor hydroponics project in Malaysia thatgrow Pak Choy. There were two environmental sensors
and two exhaust fans on each planting level on the system
to ensure proper air circulation and detection. It’s also
observed that the reading of sensors values on different
levels will be slightly different.
The temperature of the project is set within the allowable
operating range of Pak Choy by implementing
environment sensors and exhaust fans. The exhaust fanswill be operating when exceed the allowable operating
range. The project also identified that the temperatures
will periodically rise in the afternoon that will cause
continuous operation of exhaust fans in the afternoon.

2. Literature Review
In this section, understanding and evaluating themonitoring and controlling systems from past researchstudies can be well understood.
2.1. Water and Nutrient Flow Path Monitoring
and Controlling System
Water flow path and control takes a critical role inhydroponics system as the growth condition of plant was
highly determined by the nutrients absorbed.
Fig.1. refers to the system used by M. E. H. Chowdhury
et al., 2021 [6]. It contains two circulation paths: nutrient
solution circulation and clean water supply.
The pump on the clean water source will only operate
when the nutrient container does not meet the height of
the water level with the aid of the water level sensor. The
placement of the water level sensor was determined withthe water height formula used; the water level sensor of
the system will be placed 22.4cm vertically on the innercontainer to ensure enough 37L water in the tank.
An additional water flow sensor has been provided on
the nutrient container. The water flow sensor was capableof measuring volume circulation per day and water. The
system will notify when there is no water flow sensing.

Fig. 2. Temperature and Humidity Controlling System©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
758

Auto Indoor Hydroponics Plant

2.3. Light Monitoring and Controlling System

3. Methodology and Setup

Referring to another study about effects of Artificial
Night LED Lighting by L. S. Wei et al, 2019, [8]. The
proposed hydroponics system in Malaysia cultured theplant with a 18/6 (light/dark) photo period. Based on
Table 1 below, it shows that the plant was cultured with
8 hours of artificial lighting and 16 hours with naturallighting that continuously operate until plant maturityday.

In this section, the monitoring and controlling systems
from past research will be presented.
3.1. Mechanical Design and Setup
Fig. 4. shows the proposed hydroponics mechanicaldesign with multiple main components. The hydroponic
structure is arranged in a horizontal shape which has an
overall dimension of around 1.25m x 0.79m x 1.2m (W x
L x H) with 2 layers of planting platforms that can covera total of 72 plants with 38 plants each layer. The system
has additionally equipped with two ball valves that canhelp in controlling the water circulation and water levelto void overflow piping [9]. The system has been
installed in a chamber for easily controlled environments.
The hydroponics system has mainly been using UPVC
pipe as the structural material of the design for easily
accessible, fabrication and assembly friendly as well as
cost effective. Eventually, the chamber is also smaller insize as compared to a 160cm human being.

Table 1. Light Duration Used in the Study
Natural Light
without LEDs
Support8am to 6pm

Night-timewithout LEDs
Support
6pm to 12am

Night-timewith LEDs
support12am to 8am

2.4. IoT Monitoring System
Fig. 3. shows the overall system setup for the monitoringprocess. The proposed system is an Arduino Mega basedhydroponic system with MQTT protocol to communicate
between Arduino Mega and Node MCU for dataprocessing. This system proposed precise monitoring of
the nutrient solution's properties, such as electricalconductivity (EC), pH, and temperature, to ensure thatthe plants receive optimal levels of nutrients. The sensors
result will then convert to JSON format and sent to cloud
platform for monitoring. Overall, the proposed system is
an important step forward in the field of hydroponics
farming. It allows for precise and automated monitoringof nutrient solution levels reducing the need for manuallabor of the farming process.

Fig. 4. Hydroponics Design with Main Components

3.2. Electrical System Infrastructure
Fig. 5. shows the proposed electrical schematic designedwith Proteus. The proposed design has utilized ESP32 as
the mainboard and different sensors such as the DHT11
for temperature and humidity sensing, the YF-S201 forwater flow rate sensing, a water level sensor for nutrient
level sensing, as well as light sensor module for lightintensity sensing. Relays and actuating devices have beenused to control the system automatically. In general, thetotal cost of components required to set up the system isabout RM185.60

Fig. 3. IoT Monitoring Hydroponics System©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
759

Ammar A. M. Al-Talib, Tew Hwa Hui, Sarah ‘Atifah Saruchi, Noor Idayu Mohd Tahir, Nor Fazilah Binti Abdullah

Fig. 7. Auto Grow Light Flow Chart

Nutrient Level and Water Flow Rate Flow Chart
Fig. 8. shows the hydroponics system monitoring andcontrolling flow chart. The main controller, an ESP32,
will retrieve data from the water flow and water levelsensors. The monitoring system will notify the user when
the water level and water flow has reached the thresholdset. Meanwhile, the main microcontroller will thenpublish that data to the Arduino cloud to allow real-timemonitoring for the user.

Fig. 5. IoT Monitoring Hydroponics System

3.3. Software System Flow
Ventilation Flow Chart
Fig. 6. shows the hydroponics system monitoring andcontrol flow chart. The main controller, an ESP32, will
retrieve data from the sensor, DHT11. The ventilation fan
will turn on when the temperature have reached thethreshold set, vice versa. Meanwhile, the main
microcontroller will then publish that data to the Arduino
cloud to allow real-time monitoring for the user.

Fig. 8. Auto Grow Light Flow Chart

4. Results and Discussion

Fig. 6. Auto Ventilation Flow Chart

4.1. Testing, Analysis and Validation

Grow Light Flow Chart

Table 2 below shows the comparison of measured
voltage and actual voltage of power supply pin on the
microcontroller. The percentage error of most pins is within
an acceptable range. However, the percentage error forthe 3V pins is 10.33%, indicating that there is an issue
with the expansion board. This might be due to thevoltage drops across the pcb trace of the used expansion
board. Nonetheless, since the measured voltages are
3.31V, it is possible that there is wrong labelling on theFig. 7. shows the hydroponics system monitoring andcontrol flow chart. The main controller, an ESP32, will
retrieve data from the NTP servers and LDRs sensors.
The LED strips will turn on when the time has reached
the threshold set, vice versa. Meanwhile, the main
microcontroller will then publish that data to the Arduino
cloud to allow real-time monitoring for the user.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
760

Auto Indoor Hydroponics Plant

board. The 3V pins will be used as a 3.3V power supply
to the system.
Table 2. Measured Voltage and Actual Voltage
Input

5V
2A

23.30

23.00

1.30

Data at 30 Minutes

23.20

23.00

0.87

Average Error Percentage Difference (%)

1.00

Pin
Symbol

Measured
Voltage(V)

ActualVoltage(V)

Percentage
Error (%)

VIN

4.98

5.00

0.40

5V (1)

4.92

5.00

1.60

5V (2)

4.91

5.00

1.80

3.3V

3.31

3.33

0.60

Initial Data

60

58

3.45

3V (1)

3.31

3.00

10.33

Data at 5 Minutes

59

57

3.51

3V (2)

3.31

3.00

10.33

Data at 10 Minutes

59

57

3.51

Data at 15 Minutes

58

56

3.57

Data at 20 Minutes

58

56

3.57

Data at 30 Minutes

58

56

3.57

Average Error Percentage Difference (%)

3.53

Table 4. Measured and Actual Humidity

Input

Table 3 and Table 4, Are showing the comparison oftemperature and humidity of used sensor andmeasurement tools. The results are retrieved every 5
minutes until 30 minutes. Based on the results, it isobserved that the temperature measurements have an
average percentage difference of 1%. Nevertheless,humidity measurements have an average percentage of
3.53%. Both measurements are relatively acceptable.

No

Table 5 and Fig. 9 are for the Water Level Testing andAnalysis. It shows the ADC reading of the water levelsensor corresponding with the height of the measuring
tools. The ADC input channels of ESP32 have a 12-bit
resolution, this means that analog readings of 0 to 4095
shall be obtained. However, the measured value is only
ranging from 0 to 2019 only, this might be due to the
corrosion of the water level sensor. The testing results
helped in benchmarking the operating range of the waterlevel sensor. Therefore, the water level sensor can be usedin this system for water level measurements.

Table 3. Measured and Actual Temperature
Temperature (℃)

No

Relative Humidity(%)%
DHT 11 Measuring Difference
SensorTool

23℃

Temperature And Humidity Testing and Analysis

Input

Data at 20 Minutes%
DHT 11 Measuring Difference
SensorTool

Initial Data

23.50

23.20

1.29

Data at 5 Minutes

23.40

23.20

0.86

Data at 10 Minutes

23.40

23.20

0.86

No

Water Level Sensor(12-bit ADC Reading)

Measuring Tools(cm)

Data at 15 Minutes

23.30

23.10

0.87

1

0

0

2

976

1

3

1454

2

4

1703

3

5

2019

4

Table 5. Comparison of Level Sensor and Ruler

23℃©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
761

Ammar A. M. Al-Talib, Tew Hwa Hui, Sarah ‘Atifah Saruchi, Noor Idayu Mohd Tahir, Nor Fazilah Binti Abdullah

For the Light Intensity Testing, and based on themeasurement conducted by comparing the light intensity
sensor and lux meter in Table 5, it’s identifying that theroom lights itself is not enough to supply enough lightintensity on the hydroponics system. The percentage
difference of sensors and lux meter has less than 18% at
different room conditions. It indicates that the lightintensity sensor can be implemented in this prototype.
Nevertheless, it’s important to place the hydroponics
system near a room that has exposure to sunlight forenough brightness for plants to conduct photosynthesis.
Table 5. Light Intensity Sensor and Lux Meter
Room
Condition

Fig. 9. Actual Water Height Correspond to 12-bit ADC

Light Intensity
Lux Meter (lux)
Sensor (lux)% Difference

Top BottomTop

Sunlight 128.0

For the water flow rate sensor testing and analysis, amathematical equation is used to calculate the flowrate of the system is as follows, [6].
𝐹𝑙𝑜𝑤 𝑅𝑎𝑡𝑒 (𝐿/𝑚𝑖𝑛) =Room
LightsWithout
Lights𝑆𝑒𝑛𝑠𝑜𝑟 𝐹𝑟𝑒𝑞𝑢𝑒𝑛𝑐𝑦 (𝐻𝑧)Top

Bottom

Bottom

94.0

151.30 105.40 15.40% 10.82%

41.0

17.0

50.0

0

0

0

19.30 18.00% 11.92%
0

0%

0%

7.5

Based on Fig. 10, the obtained data suggests a linear
relationship between the valve angle of rotation and thewater flow rate, this proves that the water flow rateincreases with increasing the valve angle. The R2 valueof 0.9635 indicates that the data points are well- fitted to
the linear regression line, indicating that the valve angle
and water flow rate have a significant correlation.

For the Relay Testing, Table 6 shows the results ofcontrolling ventilation fan with relay, it is noticed that theventilation fan will be turned on every afternoon. The fan
will be turned off during mornings and nights. This is dueto the reason that the ventilation fan will be turned on whenreaching the state point.
Table 6. DHT11 and State of Fan
Threshold29.50℃

Temperature of State of VentilationTime
DHT 11 Sensor
Fan(a.m. / p.m.)(ON / OFF)(℃)
Morning
25.40
OFF
9 a.m.
Afternoon
30.30
ON
3 p.m.
Night
27.80
OFF
11 p.m.

Table 7 shows that the threshold was set, where the LED
strips will be only turned on during 12 a.m. to 8 a.m. The
LED strips will only turn on when it’s midnight.

Fig. 10. Relationship of Valve Angle and Water Flow Rate

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
762

Auto Indoor Hydroponics Plant

Final Implementation

Table 7. Results of Time and State of LED
Threshold12a.m. to
8a.m.

Time
(a.m. / p.m.)Morning
6 a.m.
Afternoon
12 p.m.
Mid-night
1 a.m.

State of LED Strips(ON / OFF)

Fig. 12. shows the final assembly of the hydroponics
system. The assembly of the hydroponics system started
with the structure, continued with circulation channels,
chamber and finally electrical and electronics system.

ON
OFF
ON

4.2. Hardware Assembly and Implementation
Fig. 11. shows the detail of the placement of the electricalcomponents. All the electronics components are placedin an enclosure to prevent direct contact with the waterand easy for maintenance. The temperature sensor is
located on the top layer of the hydroponics system.
Meanwhile, there is also a light intensity sensor that isplaced nearby. The second light intensity sensor is placedon the second layer. Next, the water flow rate and waterlevel sensor are placed along the flow of circulation andinside the nutrient container. The ventilation fan is placedon the front of the chamber due to the reason that it can
provide the lowest wire connection distance.

Fig. 12. Mechanical Final Assembly

4.3. Software and Interface Implementation
Fig. 13. and Fig. 14. shows the Arduino IoT cloud
dashboard and mobile app that measures the temperature,humidity, water flow rate, water level and light intensity
on top and bottom layer. There is only one page of thedashboard and app interface of the system for user
friendliness. The top part of the interface shows themeasurement values and status of sensors in real time.
Meanwhile, the bottom part of the interface shows themeasurement values retrieved over time.

Fig. 11. Placement of Electrical Components©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
763

Ammar A. M. Al-Talib, Tew Hwa Hui, Sarah ‘Atifah Saruchi, Noor Idayu Mohd Tahir, Nor Fazilah Binti Abdullah

Table 8. Comparison of Hydroponics System

SimilarWork
orCommercial
Solution
HypoZ

City Vertical
Farm L48

Plant
72
Capacity
Volume1.19 m3
1.48 m3
IoT Solution
Yes
None
Light
Yes
None
Monitoring
Temperature
M and C
None
Monitoring(M)
andControl
(C)
pH
None
None
Monitoring(M)
andControl(C)Power
58.665 W
18 W
Consumption
Price
RM 700.07
RM 2899.00
Information
Vendor Self Developed
CityFarm

Fig. 13. Web Interface of Arduino IoT Cloud System

Fig. 14. App Interface of Arduino IoT Cloud System

4.4. Hydroponic System Comparison
Table 8 shows two different similar commercial solutions
as compared to the proposed system which are all
solutions to indoor agriculture. It can be seen obviouslythat the proposed system offers more features with amonitoring and controlling system. Yet, remaining
significantly cheaper and affordable to anybody toventure into agriculture.
Besides that, the proposed system has a medium power
consumption as compared to the Grow IT premium kit.
This indicates that the system has consumed less energy
and power.
In general, the proposed system provides a cheaper andmore features alternative for indoor farming, which isalso suitable for growers that just started to venture into
agriculture.

GrowIT
Premium Kit
70
1.31 m3
None
None
None

None

120 W
RM 2999.00
GrowEatWell

5. Conclusion
The designed and tested prototype has shown successful
results in achieving the objectives of this project. There is
still room for improvements. It is recommended that thehydroponics structure can be attached with additionalfeatures such as air- conditioning and humidity
controlling devices to effectively control the growing
environment of the plants. Besides, features like pH andEC monitoring and controlling devices can be equipped,
this may help to adjust accurately nutrient solutions up takenby the plants.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
764

Powered by TCPDF (www.tcpdf.org)

Auto Indoor Hydroponics Plant

Authors Introduction

Acknowledgment

Ammar Abdulaziz Al Talib

The authors would like to express their gratitude towards
the Faculty of Engineering, Technology and Built
Environment at UCSI University/Malaysia for providing
all the facilities and labs to complete the research.

He received his B.Sc and M.Sc
degreesin
MechanicalEngineering from the Universityof Mosul Iraq.
He has finished his Ph.D degreefrom UPM University, Malaysia.
Member of the Institute of
Mechanical Engineers UK.
(CEng. MIMechE). He hasdeveloped all the Postgraduate Programs at the Faculty ofEngineering at UCSI University / Malaysia and worked
asthe Head of Postgraduate and Research department at
the same faculty for the years 2010-2018.

References
1.

2.

3.

4.

5.

6.

7.
8.

9.

Nations, U. World Population Projected to Reach 9.8
Billion in 2050, and 11.2 Billion in 2100 Available online:
https://www.un.org/en/desa/world-population-projectedreach-98-billion-2050-and-112-billion-2100 (accessed on
18 July 2022).
Shubhashree, B.S.; Hiremath, H.M. Design ofHydroponics System for Remote Automation. Int. J. Eng.
Res. 2020, 8, 3.
Okemwa, E. EFFECTIVENESS OF AQUAPONIC AND
HYDROPONIC GARDENING TO TRADITIONALGARDENING. Available online:
https://www.semanticscholar.org/paper/EFFECTIVENE
SS-OF-AQUAPONIC-AND-HYDROPONICGARDENINGOkemwa/4b3978feb2b103a04026bb4f7d4
6213b9a6a928 5 (accessed on 29 July 2022).
Aires, A. Hydroponic Production Systems: Impact on
Nutritional Status and Bioactive Compounds of Fresh
Vegetables. In Vegetables - Importance of Quality
Vegetables to Human Health; Asao, T., Ed.; InTech, 2018
ISBN 978-1-78923-506-7.
Sharma, N.; Acharya, S.; Kumar, K.; Singh, N.; Chaurasia,
O. Hydroponics as an Advanced Technique forVegetable
Production: An Overview. J. Soil Water Conserv. 2019,17, 364–371, doi:10.5958/2455- 7145.2018.00056.5.
Chowdhury, M.; Khandakar, A. Design, Construction andTesting of IoT Based Automated Indoor Vertical
Hydroponics Farming Test-Bed in Qatar. Sensors 2020,
20, doi:10.3390/s20195637.
Seni, S.; Audah, L. Automated Vertical Hydroponic
Farming. Evol. Electr. Electron. Eng. 2020, 1, 219–225.
Chua Mason, H.S.; Paramasivam, S.; Thing Thing, G.;Gee, C. Effects of Artificial Night LED Lighting on The
Loose Head Lettuce Growth in Hydroponic System; 2019;
Michael, G.; Tay, F.; Then, Y. Development ofAutomated Monitoring System for Hydroponics Vertical
Farming. J. Phys. Conf. Ser. 2021, 1844, 012024,
doi:10.1088/1742-6596/1844/1/012024.

Tew Hwa Hui
He is currently pursuing Bachelor of
Mechanical Engineering with Honours asfinal year student in Faculty ofEngineering Technology University,
Malaysia. His research interests areautomation & control,mechanical design,
and robotics.

Sarah ‘Atifah Saruchi
She received her B.Eng. in Mechanicaland Aerospace Engineering from
Nagoya University, Japan. She received
her Master and Doctoral degrees from
Malaysia-Japan International Institute ofTechnology(MJIIT),
Universiti
Teknologi Malaysia. Currently, she isworking as senior lecturer at Universiti
Malaysia Pahang Al-Sultan Abdullah, Pahang, Malaysia.
Her research interests are control, mechatronics and Ai.
Noor Idayu Mohd Tahir
She received her B.Eng in MechanicalEngineering from Universiti Teknologi
MARA, Malaysia. She was a design
engineer in oil and gas industries for 7
years. Currently, she works as a lecturer
under the Department of MechanicalandMechatronics,
FacultyofTechnology and Built Environment,
UCSI Kuala Lumpur. Her research interests include design
and robotics.
Nor Fazilah Abdullah
She received her Bachelor’s degree inaerospace engineering (Hons) from
IIUM, Gombak in 2010 and her Master’s
degree in Mechanical Engineering from
UKM, Bangi in 2015. Currently she ispursuing Doctoral of Philosophy
programme at UCSI University, Kuala
Lumpur. Her research interest is in biobased nanoparticle materials.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
765

A Design and Fabrication of a Solar Agriculture Water Pumping System
Ammar A. M. Al-Talib

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Noor Idayu Mohd Tahir

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Ain Atiqa Mustapha

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Amar Ridzuan Abd Hamid

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Sarah ‘Atifah Saruchi

Faculty of Manufacturing and Mechatronic Engineering Technology,
Universiti Malaysia Pahang Al-Sultan Abdullah, 26600 Pekan, Pahang, Malaysia
Yazan Shebli Ibrahim

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
E-mail: ammart@ucsiuniversity.edu.my, nooridayu@ucsiuniversity.edu.my, ainatiqa@ucsiuniversity.edu.my,amar@ucsiuniversity.edu.my, sarahatifah@umpsa.edu.my

Abstract
This study explores the use of a solar driven water pump. PV technology replaces conventional electricity and diesel
pumps by using solar energy to power DC or AC water pumps. The main objective of this study is to design andconstruct a solar-powered agriculture water pumping system and to evaluate its performance. The solar agriculture
water pumping system used in this project consists of a 40-watt monocrystalline solar cell with an efficiency
conversion of between 23% – 24%. This can supply power to a 16.8 W DC Flow Submersible Pump. It could lift thewater up to 5m and a flow range of 700 Liter/Hour. The system also includes a PWM 30A Solar Charge Controller
to regulate the input power to a 12V, 7.2A Sealed Rechargeable Battery. Finally, a 20m long watering kit with nozzles
irrigation system is connected to a 12V DC Submersible Pump to water the plants.
Keywords: Solar panel, water pumped, renewable energy, irrigation system, SPVWPS.

coal, and oil, make up 81%, while nuclear power
contributes only 5.7% [2]. Sustainable energy, such as
water, sunlight, and wind, is a solution that has noharmful impact on the environment and contributes toachieving sustainable energy goals [3]. The sun is themost significant power source on Earth, providing moreenergy in one hour than all nations produce in a year. It
delivers over 15,000 EJ of energy to the planet every day,which is more than 104 times the daily energy used byhuman activities [4]. However, only 0.1% of the sun's
energy is consumed for electricity [5]. AlexandreEdmond Becquerel observed the PV effect for the firsttime in 1839 [6]. The initial contemporary silicon solar

1. Introduction
Electricity is an essential element of nature and a widely
utilized form of energy that is vital for various humanactivities, such as lighting and work. With the everincreasing global population, energy consumption has
surged significantly in recent times. From 2005 to 2014,
global energy consumption increased by 18%, and it isexpected to grow by 35% by 2035 [1]. Renewable energy
sources, including biofuel, waste, hydro, solar, wind,geothermal, and thermal, Comprise merely 13% of global
energy consumption. Fossil fuels, such as natural gas,

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
766

Ammar A. M. Al-Talib, Noor Idayu Mohd Tahir, Ain Atiqa Mustapha, Amar Ridzuan Abd Hamid, , Sarah ‘Atifah Saruchi, Yazan Shebli Ibrahimcell was invented by Russel Ohl in 1946 [7]. Despite
technological advancements, solar power installation
costs five times more than coal, gas, or nuclear sources
for electricity generation [8]. Agriculture is one of theimportant sectors in emerging nations. According to a
World Bank estimate, rural agriculture areas employ
nearly 86 percent of the global population (World
Bank,2008). High water consumption coincides withperiods of high solar irradiation. As a result, PV water
pumping can help to alleviate the water shortage problem[9]. Climate change and population growth cause
droughts and agricultural losses, prompting farmers to
adopt solar-powered irrigation systems as an alternative
energy source [10]. The world is facing a crisis in waterand energy, both of which are critical for agricultural
productivity. To cater to the needs of an expanding
populace, a sustainable solution is necessary. The
employment of solar photovoltaic water pumping
systems is a feasible and economical substitute,especially in distant and undeveloped regions ofdeveloping nations. With continual innovation, thesesystems have gained widespread use in the industrial,
residential, and agricultural sectors [11]. Solar Powered
Photovoltaic Water Pumping Systems (SPVWPS) offers
an alternative to traditional water pumping systems that
rely on non-renewable sources like diesel, coal, and gas
for electricity. Diesel systems require a lot of fuel, cause
noise pollution, and result in environmental problems
such as acid rain and greenhouse emissions [12].
Replacing diesel pumps with solar PV pumps costs 2-4
times as much. SPVWPS have eco-friendly operations
and low maintenance costs [13]. Solar water pumping
systems are in the nascent phase of development andrequire addressing several obstacles, includingintermittent performance, steep upfront expenses, lowefficiency of 15%-16%, and weather conditions that
affect sun availability. Sun availability is not constant,and it varies from day to day, making it difficult to rely
solely on solar power [14]. The performance of SPVWPS
relies on factors such as the number of solar panels used,
the type of controllers employed, and the energy storage
system utilized. The pump, PV array adjustment, andambient conditions also impact system efficiency. The
maximum conversion efficiency of solar panels is 18%,
and controllers like MPP tracking and charge controllers
can increase efficiency and battery life. As SPVWPS only
generates electricity during daylight, energy storage
systems are used [15].

is connected to a 12V DC submersible pump for water
flow. This pump has a power range of 16.8W to 26.4W,
a Nylon PA66 shell is placed in a 20L water tank. It candeliver a flow range of 700 to 800L/H and is programmed
to operate at specific times via a timer. The pump isconnected to a 20m water pipe and is part of a watering
kit that includes spray nozzles, and tee connectors,
designed for plant irrigations. Fig. 1 depicts the overall
prototype.

Fig. 1 Solar agriculture water pumping system prototype
2.1. Design of solar agriculture water pumping
systemThe design of the system is presented in Fig. 2
SolidWorks has been employed to produce a detailed
representation of the design. The external frame of thesystem is constructed using wood. Additionally, two
sheets of plywood, each measuring 40cm by 45cm, areused as supporting pieces. One of these pieces
accommodates the solar panels, while the other houses
the battery, charge controller, and timer. A submersible
pump located inside the water tank and the tank itself areother essential components of the system.

2. Methodology and Experimental Setup
The experimental setup comprises a 40-wattmonocrystalline solar panel that generates a 12V DC
output and 1.5A current. This panel is linked to a 30A
PWM solar charge controller featuring an LCD controller
and dual USB, which regulates the charging of 12V
sealed lead-acid batteries with a 7.2AH capacity
connected to the system. In addition, the charge controller

Fig. 2 Different views of the prototype©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
767

Design and Fabrication of a Solar
Table 2 .The average outdoor readings through different
tilts angle of solar panel
Angle Charging Solar
Solar
Solar
Duration Panel
Panel
Irradiances(Hour)
Voltage Current (W/m2)
Input
Input
(V)
(A)
16°
5.76
12.68
1.25
961.67
25°
6.79
12.64
1.06
896.3
45°
7.58
12.44
0.95
783.5

The overall connection is illustrated as a block diagramin Fig. 3. To acquire an accurate experimental result andcompare it to the theoretical outcome.

Table 3. Overall average results for hoses of different
lengths
Tube
Voltage The
The
battery
length output
discharge discharge fully(V)rate
of rateof discharge
water
water
(hours)using a without anozzlesnozzle(L/min)(L/min)
5m
13
0.866
1.44
3.5
10m
12.98
0.636
1.10
3.28
15m
12.94
0.566
0.75
3.03
20m
12.9
0.458
0.625
2.47

Fig. 3 Overall connection of the prototype.

3. Results and Discussion
In this section, Table 1 presents the average results forthe three different solar panel orientation angles, with two
different direct solar radiation of 854.3 W/m2 and 542.7
W/m2. Table 2 presents the average results of charging
duration, solar panel voltage input, solar panel current
input, and solar irradiations when the solar panel of threedifferent tilt angles was placed directly under the sun.
Additionally, Fig. 3 is displaying the average results ofvoltage drops, the discharge rate of water under two
different conditions of discharge (nozzles connected tohoses of different lengths and free discharge), and thetime required for the battery to be fully discharged. The
readings shown in Table 3 are taken with a constant water
volume of one litre per minute.

3.1. Light Intensity
Fig. 4 illustrates the light intensity recorded by the solar
panel at different tilt angles. As the tilt angle increases,
the amount of light received by the panel decreases
significantly, particularly when the light source ispositioned further away from the solar panel. This
reduced light exposure results in a diminished input ofvoltage and current, ultimately leading to a lower power
output generated by the solar panel.

Table 1. The average readings throughout different tilt angles atIndoor two constant solar flux values.

585.7

27.7

508.1

36.05

Light Intensity (W/m2)

15.6

20

Solar Panel Current Input (A)

45

693.2

Solar Panel Voltage Input (V)

11.8

0.7
2
0.6
1
0.4
6

Charging Duration (Hour)

25

12.
64
12.
36
12.
31

Direct solar radiation of 542.7
W/m2

Light Intensity (W/m2)

10

Solar Panel Current Input (A)Charging Duration (Hour)

16

Solar Panel Voltage Input (V)Angle (°)

Direct solar radiation of 854.3
W/m2

12.2
8
12.1
7
12.0
6

0.36

367.7

0.26

340.9

0.2

303.4

Fig. 4 Light intensity vs different tilt angles at two
different indoor direct solar radiations

3.2. Solar Irradiations
Fig. 5 illustrates the solar panel's received solar irradianceat various tilt angles when placed directly under the sun.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
768

Ammar A. M. Al-Talib, Noor Idayu Mohd Tahir, Ain Atiqa Mustapha, Amar Ridzuan Abd Hamid, , Sarah ‘Atifah Saruchi, Yazan Shebli Ibrahim

Fig. 5 Solar irradiance received at various tilt angles
Furthermore, it is observed that the solar panel receives
the most radiation when placed at a smaller tilts angle,
which leads to the generation of more power this can beseen in Fig. 6. Hence, it can be concluded that the tilts
angle of the solar panel has a significant impact on theamount of radiation received and the subsequent power
generated.

Fig 1. Time required to charge battery against thestate of charge at direct solar radiation of 542.7 W/m 2

Fig 8. Time required to charge battery against thestate of charge under outdoor sun light
Fig. 6 Time required to charge battery against the stateof charge at direct solar radiation of 854.3 W/m2

Fig. 6 and Fig. 7 illustrate that the differences in the timerequired to charge the battery can be attributed to theamount of current generated by the solar panel, which isrelated to the amount of radiation received by the solar
panel.

3.3. Time required to charge the battery.
In Fig. 6, the charging duration of the battery for threedifferent tilts angles is shown as up to 10 hours when theheight difference between the solar panel and the 1000watt halogen bulb is 50 cm. Similarly, Fig. 7 illustrates
the charging duration of the battery for the same tilt
angles, but up to 20 hours and when the height differenceis 70 cm. Furthermore, in Fig. 8, the duration to charge
the battery for six hours is exhibited when it is situateddirectly under sunlight.

3.4. The discharge rate of water
Fig. 9 illustrates the collective findings of the water
discharge experiment conducted with hoses of varying
lengths and nozzle openings. The experiment involved
the flow of one litre of water per minute from a 20L watertank. The presented results are based on the different
percentages of nozzle openings tested.

Fig 9. Discharge rate of water against nozzles ofdifferent percentage opening©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
769

Design and Fabrication of a Solar

According to Fig. 9, the highest water discharge rateachieved is approximately 0.95 L/min, which occurred
when a 5m long hose is connected to the nozzle while
maintaining a 100% opening. Conversely, the slowest
water discharge rate of 0.4 L/min is observed when thehose is 20m long, and the nozzle opening percentage is20%, unlike the hoses of 10m and 15m lengths. It is worth
noting that friction between the water and the hose's innersurface, as well as changes in elevation or flow direction,
contribute to losses in water pressure, known as head loss,
during water pumping through a hose. Consequently, it
can be inferred that hoses with smaller nozzle opening
percentages offer more resistance to water flow, whichleads to reduced water pressure and discharge rates.

collection of more solar energy and facilitates fasterbattery charging. Moreover, the system is capable ofpumping water up to a height of 3 meters.

3.5. Maximum height water can reach

The authors would like to express their gratitude towards
the Faculty of Engineering, Technology and Built
Environment at UCSI University, for the labs andfacilities provided during the research.

To maintain long battery life, it has been found thatshorter irrigation pipes are more suitable. Furthermore,
the study revealed that the solar collector efficiency is80%. Overall, the designed solar irrigation system has
demonstrated its capability in achieving its intended
objectives, and the findings of the study have provided
valuable insights for future system improvements.
Acknowledgements

Fig.10 presents the findings obtained by using hoses ofdifferent lengths which are 0.5m, 1m, 1.5m, 2.0m, 2.5m,
and 3m to determine the maximum height that water canreach using a 12v DC submersible pump.

References1.

2.

Fig. 2 The discharge rate of water against hoses ofdifferent heights

3.

The graph shows that the pump can lift water to a
maximum height of 3m with a discharge rate of
approximately 0.321 l/min, while the minimum height of
0.5m has the fastest discharge rate of water at around 1.95
l/min. This indicates that the pump's lifting ability
decreases as the height of the hose increases, resulting ina decrease in the water discharge rate. This is due to thelonger hoses creating more resistance and reducing theflow rate.

4.

5.
6.

7.

4. Conclusion
The designed prototype for a solar irrigation system has
been fabricated and tested. The study indicates that more
solar radiation will result in higher charging power,
which is useful for running the solar irrigation system.
Additionally, the system is capable of pumping water up
to 20 meters by fully utilizing solar energy.

8.

9.

To maximize the efficiency of the system, it has beenfound that the solar panel should be tilted at an
appropriate angle for each location, which enables theE. Pérez-Denicia, F. Fernández-Luqueño, D. VilariñoAyala, L. Manuel Montaño-Zetina, and L. AlfonsoMaldonado-López, “Renewable energy sources forelectricity generation in Mexico: A review,” Renewable
and Sustainable Energy Reviews, vol. 78, pp. 597–613,
Oct. 2017, doi: 10.1016/J.RSER.2017.05.009.
M. Hosenuzzaman, N. A. Rahim, J. Selvaraj, M.
Hasanuzzaman, A. B. M. A. Malek, and A. Nahar, “Global
prospects, progress, policies, and environmental impact ofsolar photovoltaic power generation,” Renewable andSustainable Energy Reviews, vol. 41, pp. 284–297, Jan.
2015, doi: 10.1016/J.RSER.2014.08.046.
A. Q. Al-Shetwi, “Sustainable development of renewableenergy integrated power sector: Trends, environmental
impacts, and recent challenges,” Science of The Total
Environment, vol. 822, p. 153645, May 2022, doi:
10.1016/J.SCITOTENV.2022.153645
S. Bilgen, “Structure and environmental impact of global
energy consumption,” Renewable and Sustainable Energy
Reviews, vol. 38, pp. 890–902, Oct. 2014, doi:
10.1016/J.RSER.2014.07.004.
C. J. Chen, “Physics of Solar Energy,” Physics of Solar
Energy, Aug. 2011, doi: 10.1002/9781118172841.
L. M. Fraas, “History of Solar Cell Development,” LowCost Solar Electric Power, pp. 1–12, 2014, doi:
10.1007/978-3-319-07530-3_1.
S. Sharma, K. K. Jain, and A. Sharma, “Solar Cells: In
Research and Applications—A Review,” Materials
Sciences and Applications, vol. 06, no. 12, pp. 1145–1155,
2015, doi: 10.4236/msa.2015.612113.
N. Babakatcha, A. Yabagi, M. B. Ladan, and M. D.
Oladipupo,“HARNESSING
SOLAR
ENERGYPOTENTIAL AS AN ALTERNATIVE SOURCE OF
ELECTRICAL ENERGY IN NORTH CENTRAL,NIGERIA,” African Journal of Environment and Natural
Science Research, vol. 3, p. 86, 2020, Accessed: Mar. 28,
2023. [Online]. Available: www.abjournals.org
D. Zhou and Abdullah, “The acceptance of solar water
pump technology among rural farmers of northern
Pakistan: A structural equation model,” Cogent Food
Agric,
vol.
3,
no.
1,
2017,
doi:

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
770

Powered by TCPDF (www.tcpdf.org)

Ammar A. M. Al-Talib, Noor Idayu Mohd Tahir, Ain Atiqa Mustapha, Amar Ridzuan Abd Hamid, , Sarah ‘Atifah Saruchi, Yazan Shebli Ibrahim

10.

11.

12.

13.

14.

15.

10.1080/23311932.2017.1280882/SUPPL_FILE/OAFA_
A_1280882_SM1744.DOCX.
J. B. Rutibabara and W. M. Budzianowski,“Environmental and Economic Cost Analysis of a SolarPV, Diesel and hybrid PV-Diesel water Pumping Systems
for Agricultural Irrigation in Rwanda: Case study ofBugesera district INSTITUTE OF WATER AND
ENERGY SCIENCES (Including climate change),” 2018.
G. Li, Y. Jin, M. W. Akram, and X. Chen, “Research andcurrent status of the solar photovoltaic water pumping
system – A review,” Renewable and Sustainable Energy
Reviews, vol. 79. Elsevier Ltd, pp. 440–458, 2017. doi:
10.1016/j.rser.2017.05.055.
A. Mokeddem, A. Midoun, D. Kadri, S. Hiadsi, and I. A.
Raja, “Performance of a directly-coupled PV water
pumping system,” Energy Convers Manag, vol. 52, no. 10,
pp.
3089–3095,
Sep.
2011,
doi:
10.1016/J.ENCONMAN.2011.04.024.
S. S. Chandel, M. Nagaraju Naik, and R. Chandel,“Review of solar photovoltaic water pumping systemtechnology for irrigation and community drinking watersupplies,” Renewable and Sustainable Energy Reviews,
vol. 49, pp. 1084–1099, Oct. 2015, doi:
10.1016/J.RSER.2015.04.083.
B. Singh, U. Sharma, and S. Kumar, “Standalone
Photovoltaic Water Pumping System Using Induction
Motor Drive with Reduced Sensors,” IEEE Trans Ind Appl,vol. 54, no. 4, pp. 3645–3655, Jul. 2018, doi:
10.1109/TIA.2018.2825285.
R. Foster, M. Ghassemi, and A. Cota, “Solar energy:
Renewable energy and the environment,” Solar Energy:
Renewable Energy and the Environment, pp. 1–337, Jan.
2009, doi: 10.5860/choice.47-5672.

Ain Atiqa Mustapha
She received her Master's degreefrom the Faculty of Electronic &Computer Engineering, Universiti
Teknikal Malaysia Melaka, Malaysia
in 2017. She is currently a Tutor
under Department of Mechanical andMechatronics Engineering, UCSI
University, Kuala Lumpur. Her fieldof research is renewable energy andartificial intelligence.
Amar Ridzuan Abd.Hamid
He received his Bachelor’s degree inautomotive engineering (Hons) from
UTeM, Melaka in 2011 and his Master’s
degree in Mechanical Engineering from
UPM, Selangor in 2015.He is a
Professional Technologist, Head ofProgrammes and lecturer of Mechanicaland Mechatronic programmes from Department of
Mechanical Engineering, UCSI University, Malaysia. His
research interests are in the design and analysis ofautomotive compartments.
Sarah ‘Atifah Saruchi
She received her B.Eng. in Mechanicaland Aerospace Engineering from
Nagoya University, Japan . She
received her Master and Doctoral
degreesfrom
Malaysia-Japan
International Institute of Technology(MJIIT),
Universiti
Teknologi
Malaysia. Currently, she is working atUniversiti Malaysia Pahang Al-Sultan
Abdullah, Pahang, Malaysia. Her research interests include
mechatronics and artificial intelligence.

Authors Introduction
Ammar Abdulaziz Al Talib
He received his B.Sc and M.Sc
degreesin
MechanicalEngineering from the Universityof Mosul Iraq.
He has finished his Ph.D degreefrom UPM University, Malaysia.
Member of the Institute of
Mechanical Engineers UK.
(CEng. MIMechE). He hasdeveloped all the Postgraduate Programs at the Faculty ofEngineering at UCSI University / Malaysia and worked
asthe Head of Postgraduate and Research department at
the same faculty for the years 2010-2018.

Yazan Shebli Ibrahim
He has graduated form the
Mechanical Engineering Department
at UCSI University as a first class
graduand in 2023 . Currently, he isworking in the private sector.

Noor Idayu Mohd Tahir
She received her B.Eng in MechanicalEngineering from Universiti Teknologi
MARA, Malaysia. She was design
engineer in oil and gas industries for 7
years. Currently, she works as a lecturer
under the Department of MechanicalandMechatronics,
FacultyofTechnology and Built Environment,
UCSI Kuala Lumpur. Her research interests include design
and autonomous robotics.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
771

Design and Performance of a Power Generating Manual Treadmill
Ammar A.M. Al-Talib

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Sarah ‘Atifah Saruchi

Faculty of Manufacturing and Mechatronic Engineering Technology,
Universiti Malaysia Pahang Al-Sultan Abdullah, 26600 Pekan, Pahang, Malaysia
Cik Suhana Hassan

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Nor Fazilah Binti Abdullah

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Ain Atiqa Mustapha

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Ahmad Jelban

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
E-mail: ammart@ucsiuniversity.edu.my, sarahatifah@umpsa.edu.my, suhana@ucsiuniversity.edu.my, ,norfa@ucsiuniversity.edu.my, ainatiqa@ucsiuniversity.edu.my

Abstract
Treadmills are one of the most popular training equipment in the gym and at home. The working principle oftreadmills is by moving the belt with the human knee bending, which creates mechanical energy to turn the belt. A
gear or pulley and belt system connects to the generator along the axel line of the rolling bars. The power generated
by the DC generator is stored in a battery pack and could be used to charge phones or other equipment. It has beenfound that treadmills can provide an efficiency of 95% when the DC motor is used and 92% when the AC motor isused. The main objective of this study is to design and fabricate a powder-generating manual treadmill and to analyze
the performance of the system under different operation conditions.

Keywords: Manual Treadmills, Gym Renewable Energy

decreased [2]. Impact of the MCO on electricity
consumption has shown a sharp increase in residential
electricity consumption [3]. To address this issue, thetreadmill, which has become a popular fitness equipment
for both home and gym use, can be modified to generate
electricity. Originally the treadmill consumed energy tooperate [4]. The treadmill consists of a wide belt driven
by a variable-speed motor. However, by connecting it toa generator or an inverter, the kinetic energy generated bya person walking or running on the belt can be converted
into electrical energy. This energy can then be used to
power various devices or fed back into the power grid,

1. Introduction
In 2015, the United Nations General Assembly
established the Sustainable Development Goals (SDGs),
which consist of 17 objectives to be accomplished by
2030. SDG 7 aims to provide affordable, reliable,
sustainable, and modern energy for all [1] This goal isespecially relevant during the COVID-19 pandemic, as
measures such as movement control orders (MCO) havesignificantly increased residential energy consumption
while commercial and business energy consumption has©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
772

Ammar A. M. Al-Talib, Sarah ‘Atifah Saruchi, Cik Suhana Hassan, Nor Fazilah Abdullah, Ain Atiqa Mustapha, Ahmad Jelban

making the treadmill a potential solution to the increased
energy consumption during the pandemic [5].
Fig. 3 shows the modified treadmill at its charging status.
2. Methodology and Experimental Setup
As shown in Fig. 1, this experiment is set up with a 36 V
DC generator that is driven by a pulley linked by a belt,with the main motion generated by the user walking or
running on the treadmill belt [6]. The generator isconnected to a charging control that regulates the voltage
and current supplied to a 12 V, 7.2 Ah battery. The
battery then supplies a 200 W inverter, which converts
DC to AC power that can be used to charge devices suchas phones and laptops. A Watt meter is connected
between the generator and the charging controller tomeasure the current and voltage produced by thegenerator.
Fig. 3 Charging Laptop from The
Treadmill Battery

3. Results and Discussion
In this section, Table 1 shows the average results of 3
different people who tested the prototype. The results are
the readings from the watt meter for the voltage andcurrent while the average speed is from the speedometer
after 5 minutes of walking on the treadmill. As stated,
before the treadmill is not motorized, therefore the speed
is not constant, and the only way to maintain a constant
speed is dependent on the person himself. However, theadjustment of pace was constant to maintain a relatively
constant speed.

Fig. 1 Connection Illustration of The Setup

2.1. Design and Modifications
As the treadmill was already made, modifications are
required to convert the regular treadmill to manual one
that is able to generate electricity. The modifications canbe summarized as an assembly of a jack to have an incline,
support for the side, generator base, motor pulley andtreadmill roller. Fig. 2 shows the treadmill before
modifications.

Table 1. Average Speed for The Three Testers
Person 1
2.24

Speed
(km/hr)Current (A) 0.56
Voltage (v) 12.18
Power (W) 6.81

Person2
3.12

Person 3
5.05

0.86
12.53
10.63

1.19
13.24
15.43

3.1. Discussion
Fig. 4 shows the average result for the three-persons.
The results are combined in one graph, in terms of speed(km/hr), current (A), Voltage, and power (W).

Fig. 2 Treadmill Before the Modifications©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
773

Design and Performance of a Power(5)

From Eq. (1) for each rotation of the generator, a currentof 0.0029 A will be produced. The motor efficiency is
78% as stated by the manufacturer, which will beconsidered when calculating the current. The average
speed in Table 2 will be considered for these calculations.
Table 2. Average Speed Current Calculation

Fig. 4. Average Results for The Three Persons
From Fig. 4, it can be noticed that the higher the speed
the higher the voltage and current, and accordingly thepower generated. The data represents the average speed
that has been achieved by each user. The voltage andcurrent generated by the lowest speed of 2.24 km/h are
12.18 v and 0.56A respectively. This speed can beachieved by anyone as it is within the normal walking
speed. Speed 2 of 3.2 km/h induced a voltage of 12.53 v
and a current of 0.86 A. The results are the average of the
3 trials by each person. And lastly, speed 3 which is theaverage of the highest speed that can be achieved at 5.05
km/h induced a voltage of 13.24 v and current of 1.19 A.
Taking into consideration that the maximum current tocharge the battery as recommended by the manufacturer
is 0.1 to 0.25 of the battery capacity. The current achieved
falls within these values. Based on the results, it is evident
that there is a direct relationship between the speed,
voltage, current, and power output of the manual
Treadmill. When the speed of the person increases, thevoltage and current generated also increase, resulting inhigher power output.

Speed 2

Speed 3

2.24

3.12

5.05

37.28

52.0

84.11

Roller (RPM)

125.0

174.14282.0

Motor (RPM)

337.41

470.16

Current (A)

0.91

1.27

2.06

Current with78%
efficiency

0.71

0.99

1.61

Speed (km/hr)
Speed (m/s)

761.31

Eq. (6) is used to calculate the efficiency for actual
current and theoretical current.

(6)

Table 3. Actual Current and Theoretical Current
Efficiency

3.2. Theoretical Current
From the generator specifications the RPM rating is 3400,
while the current is 9.5 A for the 36 v. the generator
efficiency is stated to be 78%. Eq. (1) is used to calculate
the current produced by 1 RPM.
(1)

Actual Current (A)

0.56

0.87

1.19

Current with efficiency

0.71

0.99

1.61

Efficiency

78.88%

87.88%

73.91%

From Table 3, it can be noticed that the produced current
is less than the theoretical current and that is due to thelosses due to friction during the experiments.

The treadmill belt is connected to the roller pulley with aratio of 1:20 which is measured experimentally. The ratio
between the roller pulley and the generator pulley can becalculated using Eq. (2) to Eq. (5):

4. Conclusion(2)

In conclusion, the prototype of the manual treadmill isable to operate and generate electricity of different values
depending on the walking/ running speed of the user as it
is directly connected. The results have shown that thefaster the walking on the treadmill, the higher the current
generated which is required to charge a battery for later
use. The average 3 speeds are 2.24 km/h, 3.12 km/h, and(3)
For Motor RPM = Roller Pulley RPM × 2.7

Speed 1

(4)

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
774

Ammar A. M. Al-Talib, Sarah ‘Atifah Saruchi, Cik Suhana Hassan, Nor Fazilah Abdullah, Ain Atiqa Mustapha, Ahmad Jelban

7.

5.05 km/h which have generated a current of 0.56A,
0.86A, and 1.19 A respectively. While the voltage for the
3 speeds is 12.18 v 12.53 v and 13.24 v. For theapplication of the manual treadmill, 3 devices withdifferent battery capacities have been tested. The devices
are a smartphone, a smartwatch, and a laptop, withbatteries capacities of 4200 mAh, 455 mAh, and 56 Wh
respectively. After testing using USB and a normal
charger it has been found that the 7.2 Ah battery cancharge the smartform 4.8 times, the smartwatch 44.6
times, and the laptop 1.5 times. However, it is important
to note that the manual treadmill is designed to be anaffordable and accessible option for exercise beside being
successful in generating electrical power. The findings ofthis research align with target 13.2 of the SDG’s, whichemphasizes the integration of climate change measures
into policies and planning.

Authors Introduction
Ammar Abdulaziz Al Talib
He received his B.Sc and M.Sc
degreesin
MechanicalEngineering from the Universityof Mosul Iraq.
He has finished his Ph.D degree
from
UPM
University,
Malaysia. Memberofthe InstituteofMechanical Engineers UK.
(CEng. MIMechE). He hasdeveloped all the Postgraduate Programs at the Facultyof Engineering at UCSI University / Malaysia andworked as the Head of Postgraduate and Research
department at the same faculty for the years 2010-2018.

Acknowledgements
The authors would like to express their gratitude towards
the Faculty of Engineering, Technology and Built
Environment at UCSI University, for the labs andfacilities provided during the research.

Sarah ‘Atifah Saruchi
She received her B.Eng. in Mechanicaland Aerospace Engineering from
Nagoya University, JapanShe
received her Master and Doctoral
degreesfrom
Malaysia-Japan
International Institute of Technology(MJIIT),
Universiti
Teknologi
Malaysia. Currently, she is working atUniversiti Malaysia Pahang Al-Sultan
Abdullah, Pahang, Malaysia. Her research interests include
mechatronics and artificial intelligence.

References
1.

2.

3.

4.

5.

6.

Mustapa, S. I., Rasiah, R., Jaaffar, A. H., Abu Bakar, A.,& Kaman, Z. K. (2021). Implications of COVID-19
pandemic for energy-use and energy saving household
electrical appliances consumption behaviour in Malaysia.
EnergyStrategy
Reviews,
38,
100765.
https://doi.org/10.1016/j.esr.2021.100765

M. Yu, J. Kubiczek, K. Ding, A. Jahanzeb, and N. Iqbal,“Revisiting SDG-7 under energy efficiency vision 2050:
the role of new economic models and mass digitalizationin OECD,” Energy Effic, vol. 15, no. 1, p. 2, Jan. 2022,
doi: 10.1007/s12053-021-10010-z.
S. I. Mustapa, R. Rasiah, A. H. Jaaffar, A. Abu Bakar, andZ. K. Kaman, “Implications of COVID- 19 pandemic forenergy-use and energy saving household electricalappliances consumption behaviour in Malaysia,” Energy
Strategy Reviews, vol. 38, p. 100765, Nov. 2021, doi:
10.1016/j.esr.2021.100765
Md. T. Ullah, Md. A. Bin Karim, M. H. Uddin, and G. M.
Tauseef, “Harvesting green energy from wastage energy ofhuman activities using gymnasium bicycle at Chittagong
city,” in 2015 3rd International Conference on Green
Energy and Technology (ICGET), IEEE, Sep. 2015, pp. 1–
4. doi: 10.1109/ICGET.2015.7315085.
S. Kolgiri, “‘Design & Fabrication of Treadmill Bicycle’
Synopsis for B.E. (Mechanical Engineering) Project
Analysis Ergonomics aspects of Powerloom Industry
View project STATIC AND DYNAMIC ANALYSISFOR ROTOR SHAFT OF ELECTRIC MOTOR " View
project.”[Online].
Available:
https://www.researchgate.net/publication/331950519
Sahil, P. K. Sharma, N. Hari, N. Kumar, and D. Shahi, “Aninnovative technique of electricity generation and washingmachine application using treadmill,” in 2016 IEEE 1stInternational Conference on Power Electronics, Intelligent
Control and Energy Systems (ICPEICES), IEEE, Jul. 2016,
pp. 1–5. doi: 10.1109/ICPEICES.2016.7853524.
R.Harsha, “DESIGN AND FABRICATION OF
TREADMILL BICYCLE,” Chennai, T.Nagar, 2018.
[Online]. Available: www.jetir.org

Cik Suhana Hassan
She received her bachelor's andmaster's degrees in 2009 and 2011,
respectively,
fromUniversiti
Teknologi PETRONAS, and her PhD
in 2019 from Universiti Putra
Malaysia. She is an AssistantProfessor at the Department of
Mechanical and Mechatronics Engineering of UCSI
University. Her research interests include the investigationof bio-composites for use in automotive applications. She
is also an active member of the materials community,
having been recognized as a Professional Member of theInstitutes of Materials Malaysia and a Professional
Technologist of the Malaysian Board of Technologists in
the Material Science Technology field .

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
775

Powered by TCPDF (www.tcpdf.org)

Design and Performance of a PowerNor Fazilah Abdullah
She received her Bachelor’s degree inaerospace engineering (Hons) from IIUM,
Gombak in 2010 and her Master’s degree
in Mechanical Engineering from UKM,
Bangi in 2015. Currently she is pursuing
Doctoral of Philosophy programme at
UCSI University, Kuala Lumpur. Her
research interest in bio-based nanoparticles materials.

Ain Atiqa Mustapha
She received her Master's degreefrom the Faculty of Electronic &Computer Engineering, Universiti
Teknikal Malaysia Melaka,
Malaysia in 2017. She is currently a
Tutor under Department of
Mechanical and Mechatronics
Engineering, UCSI University,
Kuala Lumpur. Her field of researchis renewable energy and artificial intelligence.
Ahmad Abu Jelban
He graduated from the MechanicalEngineering Department at UCSI
University in 2022, and currently is
working as an Engineer.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
776

Smart Car Jack Using Internet of Things
Noor Idayu Mohd Tahir

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Ammar A.M. Al-Talib

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Zeiad Mahmoud Ebeid Abdelghaffar

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Cik Suhana Hassan

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
E-mail: NoorIdayu@ucsiuniversity.edu.my, ammart@ucsiuniversity.edu.my, 1001851834@ucsiuniversity.edu.my,suhana@ucsiuniversity.edu.my
Abstract
This research explores the integration of smart technologies in traditional car jacks, aiming to enhance
efficiency and user safety. By incorporating sensors and automation, the smart car jack introduced in thisstudy not only streamlines the lifting process but also provides real-time data on load distribution andstability. ethodologically, the study involves the design, prototyping, and testing of the smart car jack,assessing its performance in various scenarios. Results demonstrate a significant reduction in lifting timeand improved safety measures, positioning the smart car jack as a viable innovation in the automotive
industry. The implications of this technology extend to increased convenience for users and a potential
reduction in roadside accidents. This research contributes to the ongoing evolution of automotive tools andunderscores the benefits of merging smart technologies with traditional equipment.
Keywords: Internet of Things; Car Jack, Smart, ESP32

alleviating the burden on car owners during emergencies.
The focus is on developing a user-friendly and costeffective solution that can significantly reduce thephysical effort required for car maintenance tasks [2].

1. Introduction
This paper addresses the challenges of roadsideemergencies, such as flat tires, and the limitations ofconventional scissor car jacks that require physical effort
and can lead to backache problems. To tackle these issues,
the paper presents a modified car jack design that is safe,
easy to operate, and reduces physical effort for lifting andlowering vehicles during automobile repair, especiallyfor roadside situations. The study explores the integrationof Internet of Things (IoT) technology in automotive
tools, particularly in the development of smart car jacks.
The aim is to revolutionize the traditional car lifting
process, making it smarter, safer, and more efficient. By
incorporating IoT, the proposed smart car jack offers
convenience and enhanced control over vehicle
maintenance tasks [1].This paper introduces a novelsmart car jack design that leverages IoT technology to
improve the efficiency and safety of roadside repairs,

2. Methodology and Experimental Setup
A. Software and microcontroller
In this project, software holds a pivotal position indesigning and developing a smart car jack with Internetof Things (IoT) capabilities. To achieve the project's
objectives, the software utilized includes the Blynk
platform and the Arduino Integrated Development
Environment (IDE) [3]. Blynk serves as a mobile
platform, enabling remote control of various hardware
modules like Arduino and ESP32 over the internet.
With Blynk, developers gain access to an array offeatures, empowering them to control hardware
components, store, and display data, visualize©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
777

Noor Idayu Mohd Tahir, Ammar A.M. Al-Talib, Zeiad Mahmoud Ebeid Abdelghaffar, Cik Suhana Hassan

information, and more. It simplifies the process ofconnecting input/output components with hardware
devices and facilitates seamless data transfer between
them. The Blynk server, functioning as a cloud-based
backend service, establishes communication between
the smartphone application and the hardware, making
it highly convenient for developers to manage andcontrol their IoT systems efficiently [4]. In this project,
the ESP32 is utilized as the primary microcontroller.
The ESP32 is a powerful and versatile microcontrollerthat operates using the Arduino IDE, an open-source
software platform for writing and uploading code to
Arduino boards.

toughness and impact resistance compared to standard
PLA. This ensures the smart car jack's stability anddurability during the car lifting process. The final
design incorporates a movable scissor jack prototype
with mecanum wheels, operated by a stepper motor
controlled by the ESP32 microcontroller. The use of
PLA+ material for 3D printing ensures the structure's
strength and ability to withstand the pressure exerted
during the lifting operation.

The ESP32, integrated with the Blynk platform,
allows for enhanced remote control and monitoring of
the smart car jack's hardware components.

Fig. 2. System wiring diagram.
C. Coding and Programming
The code implemented in this project outlines thecore functionalities and control logic of the Arduino
sketch for the IoT-enabled car jack. It begins by
including essential libraries for Wi-Fi communication,
Blynk (the IoT platform), and Stepper motor control.
The Blynk template ID, obtained from the Blynk
website, along with the name and authentication token,
is defined to establish communication with the Blynkapp. Wi-Fi credentials are set to connect the ESP32
board to the network. In the setup function, the code
connects to Wi-Fi, initializes Blynk, and sets up the
motor control pins, ensuring the motors start in astationary state. The loop function continually checks
for incoming commands from the Blynk app throughvirtual pins. Corresponding functions are executed tocontrol the car jack's movements or perform otherdesired actions. Additional functions are defined for
precise motor control, including forward, backward,
left, and right movements. The Stepper1 function
governs the stepper motor's rotation based on specified
parameters for direction and the number of rotations
required [7]. This pseudocode provides a concise
overview of how the IoT-enabled car jack interacts
with the Blynk app, processes input commands, andcontrols the motors to achieve specific movements.

Fig. 1. Prototype Design
To establish the connection between the ESP32 andthe Blynk platform, the Blynk library was installed in
the Arduino IDE [5].This library provides an interface
for seamless communication between the ESP32 andthe Blynk cloud server and mobile application. Oncethe library is installed, developers can utilize the Blynk
API to program the ESP32, enabling control and realtime monitoring of the smart car jack's various
hardware functionalities [6].
B. Prototype Design
This section presents the final design for the smart car
jack project, as depicted in Fig. 1. The design featuresa scissor jack prototype mounted on a movable base,allowing for increased mobility. The prototype isequipped with mecanum wheels driven by a DC gear
motor, transforming it into a smart car jack. To control
the lifting mechanism, a steel rod is connected to a
stepper motor, which receives instructions from themicrocontroller, ESP32. The components of the design,
except for the steel rod, are 3D printed using robust
PLA+ (Polylactic Acid) material, renowned for its
improved mechanical properties like enhanced

D. Connections
Fig. 2 illustrates the interconnected setup of the Smart
Car Jack Prototype, comprising two power supplies,
three motor drivers, one stepper motor, four DC gear

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
778

Smart Car Jack Using

motors, and the ESP32 microcontroller. The motor
drivers are connected to the power supply and therespective motors, with specific pins interfacing themto the ESP32 for control signals.
Regarding the stepper motor, pin connections were
established using pins 25, 26, 21, and 19 as IN1, IN2,
IN3, and IN4 for its motor driver. These specific pins
allow precise control and coordination of the stepper
motor, significantly contributing to the overall
functionality and performance of the Smart Car Jack
Prototype.

3. Results and discussion

Fig. 3. The performance of the car jack with no load

Tests to analyse the performance of the smart car
jack: A. Prototype efficiency on different groundsurfaces. Table 1 shows the car jack performance whilemoving on different surfaces, The prototype
demonstrated varying speeds on different surfaces. It
achieved the highest average speed of 0.41 m/s onasphalt, performed moderately on grass with anaverage speed of 0.23 m/s, but encountered challenges
on sand, getting stuck in all trials and unable tocomplete the one-meter test. Surface conditions
significantly affected its mobility and effectiveness [8].

C. Analysing the performance of the car jack withdifferent loads.
The results reveal that the jacking up time increases
with the load on the jack, indicating a direct relationshipbetween them. Conversely, the jacking down timedecreases as the weight applied to the prototype increases,
showing an inverse relationship. These results provide
valuable insights for optimizing the prototype's
performance under varying load conditions, as illustrated
in Table 3.

Table 1. Test Results

Table 3. Test Results

D. Determining the maximum height of the prototype.

B. Analysing the performance of the car jack withno load.
The data of the car jack with no load has beentabulated in Table 2 and illustrated in Fig. 3 which showsthat the car jack's consistent and efficient performance inboth jacking up and jacking down processes without any
load. The mechanism is well-calibrated and operates
smoothly, ensuring reliable and convenient operations forusers during routine tasks [9].

This test findings determine the prototype's
maximum safe height as 30 cm, beyond which thestability and safety of the car lifting process could becompromised. Establishing this critical height limit
provides valuable guidance for users to safely operate thesmart car jack, ensuring both user and load safety duringlifting operations. Table 4 shows the stability of the car
jack with different heights.

Table 2. Test Results

Table 4. Test Results©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
779

Noor Idayu Mohd Tahir, Ammar A.M. Al-Talib, Zeiad Mahmoud Ebeid Abdelghaffar, Cik Suhana Hassan

E. Comparison between the jacking up speed ofmanual scissor car jack and the proposed prototype

5. References
1.

The smart car jack prototype reaches its maximum
height significantly faster than the manual scissor car jack,thanks to the use of a stepper motor that ensures precise
and consistent movements. The elimination of humanfactors during operation contributes to enhanced
efficiency and reliability, making the smart car jack an
appealing choice for users seeking faster and morereliable lifting operations, as shown in Table 5.

2.

3.

Table 5. Test Results
4.

5.

6.

4. Conclusion

Madanhire, I., Chatindo, T., & Mbohwa, C. (2019b).
Development of a Portable Motorized Car Jack.
http://www.edutwin.com.
S. R. J. Ramson, S. Vishnu, and M. Shanmugam,“Applications of Internet of Things (IoT) – An Overview,”
in 2020 5th International Conference on Devices, Circuits
and Systems (ICDCS), IEEE, Mar. 2020, pp. 92–95. doi:
10.1109/ICDCS48716.2020.243556.
ASEFA AND ADUGNA, “INSTITUTION OFTECHNOLOGY DEPARTMENT OF MECHANICALENGIENEERING MACHINE DESIGN PROJECT TWO
THE DESIGN OF ACME THREADED SCREW JACK,”
2022.
S. A. Pekhale and S. V Karanjkar, “International Journalof Modern Trends in Engineering and Research Design
Modification and Analysis of Electrically Operated Screw
Jack for Light Motor Vehicles,” 2016. [Online]. Available:
www.ijmter.com
B. Ezurike and M. Okwu, “Modified Screw Jack for
Lifting Operation in Industrial Setting,” International
Journal of Engineering and Technologies, vol. 13, pp. 39–
50,
Dec.
2017,
doi:
10.18052/www.scipress.com/ijet.13.39.
Pratheep, V. G., Tamilarasi, T., Ravichandran, K.,
Srinivasan, M., Someswaran, N., Vimal, S., & Praveen, S.
(2023). Design and fabrication of IOT based scissor jack.
020002. https://doi.org/10.1063/5.0148663
H. S. Purnama, T. Sutikno, S. Alavandar, and A. C.
Subrata, “Intelligent Control Strategies for Tuning PID of
Speed Control of DC Motor - A Review,” in 2019 IEEE
Conference on Energy Conversion (CENCON), IEEE, Oct.
2019,
pp.
24–30.
doi:
10.1109/CENCON47160.2019.8974782.
H. Taheri and C. X. Zhao, “Omnidirectional mobile robots,mechanisms and navigation approaches,” Mech Mach
Theory, vol. 153, p. 103958, Nov. 2020, doi:
10.1016/j.mechmachtheory.2020.103958.
M. Kelemen et al., “Distance Measurement via Using of
Ultrasonic Sensor,” Journal of Automation and Control,Vol. 3, 2015, Pages 71-74, vol. 3, no. 3, pp. 71–74, Dec.
2015, doi: 10.12691/AUTOMATION-3-3-6.

This study aimed to develop a smart scissor car jackwith IoT capabilities, incorporating mecanum wheels anda stepper motor, controlled through the Blynk application.
The prototype demonstrated remarkable performance,
lifting a 200 N toy car to 30 cm in just 13.13 seconds,
outperforming manual alternatives. The success of thisproject highlights the potential of IoT-integrated
automotive tools in enhancing efficiency, safety, andconvenienceinvehiclemaintenance.
Futurerecommendations include improving mobility with larger
wheels, implementing intelligent positioning systems,
and enhancing safety mechanisms through IoT sensors.
These enhancements will transform the smart car jackinto an advanced and user-friendly automotive solution
for various terrains and vehicles.

7.

Acknowledgment

Noor Idayu Mohd Tahir
She graduated from University Teknologi
MARA,
Malaysia
inMechanicalEngineering. She was design engineer in oil
and gas industries for 7 years. Currently,
she works as a lecturer under the
DepartmentofMechanicalandMechatronics, Faculty of Technology and Built
Environment, UCSI Kuala Lumpur. Her research interests
include design and autonomous robotics.

8.

9.

Authors Introduction

The author would like to thank UCSI University forproviding support for the research. The author also would
like to express his sincere appreciation to Ammar Al
Talib and Ir. Noor Idayu Binti Mohd Tahir for theirsupervision, guidance, and support throughout theprocess of preparation for the completion of this project.
The authors also like to thank Ms. Ain Atiqa Binti
Mustapha for providing valuable information andresources that were essential for this project.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
780

Powered by TCPDF (www.tcpdf.org)

Smart Car Jack Using

Ammar A.M. Al-Talib
He has finished his B.Sc and M.Sc degrees
in Mechanical Engineering from the
University of Mosul/Iraq. He has finishedhis Ph.D degree from UPM University /Malaysia. He is also a Chartered Engineer
and Member of the Institute of MechanicalEngineers / UK. (CEng. MIMechE). He hasdeveloped all the Postgraduate Programs at the Faculty ofEngineering at UCSI University / Malaysia and worked as
the Head of Postgraduate and Research department at thesame faculty for the years 2010-2018.

Mr. Zeiad Mahmoud Ebeid Abdelghaffar
He received his bachelor’s degree in
Mechatronic Engineering in 2023 at UCSI
University, Kuala Lumpur. His field of
research is Internet of Thing (IoT)
technology.

Cik Suhana Hassan
She received her bachelor's and master's
degrees in 2009 and 2011, respectively,from Universiti Teknologi PETRONAS,
and her PhD in 2019 from Universiti Putra
Malaysia. She is an Assistant Professor at
the Department of Mechanical andMechatronics Engineering of UCSI University. Her researchinterests include the investigation of bio-composites for use
in automotive applications.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
781

Gas Detection for Biogas System Using Internet-of-Things(IoT)
Ammar A.M. Al-Talib

Department of Mechanical & Mechatronics, Faculty of Engineering Technology and Built Environment, UCSI
University, 56000 Kuala Lumpur, Malaysia
Ian Hoh Wen Yang

Department of Mechanical & Mechatronics, Faculty of Engineering Technology and Built Environment, UCSI
University, 56000 Kuala Lumpur, Malaysia
Noor Idayu Mohd Tahir

Department of Mechanical & Mechatronics, Faculty of Engineering Technology and Built Environment, UCSI
University, 56000 Kuala Lumpur, Malaysia
Ayu Haslija Binti Abu Bakar

Department of Mechanical & Mechatronics, Faculty of Engineering Technology and Built Environment, UCSI
University, 56000 Kuala Lumpur, Malaysia
Nur Muhammad Afifi bin Zainal

Tenaga National Berhad
E-mail: ammart@ucsiuniversity.edu.my, 1001955881@ucsiuniversity.edu.my, NoorIdayu@ucsiuniversity.edu.my,
ayuhaslija@ucsiuniversity.edu.my, afifi.zainal@tnb.com.my

Abstract
Global warming remains one of the most detrimental by-products of industrialization. Fossil fuels contribute to themajority of greenhouse gases emitted but remain a popular option for the generation of energy. An easy fix for thisconundrum is to utilize other forms of fuel for energy generation which burns cleaner and renewable as opposed tofossil fuels. The aforementioned solution would be to use biomethane generated from waste products which burncleaner and comes from a renewable source. In this paper, an IoT based biogas monitoring system for biogas reactors
is proposed. An ESP-32 microcontroller system is deployed and tested to detect the presence of gas production. A
dashboard plotting the data obtained from sensors is designed to help user monitor parameters. Data obtained is
automatically uploaded to a Google Spreadsheet for data.
Keywords: Sensors; Biogas monitoring; Microcontroller; IoT

328,000,000 metric tons of carbon dioxide [1]. With theincrease in temperature comes the melting of the glaciers
which results in an increase in sea levels. The world
population continues to grow every year, and with thatthe demand for energy also continues to grow. In big
countries such as China, it was estimated that the
population in 2021 was 1.43 billion and 543.98 million
in 1950 [2]. The increase in population also contributes
to the increase in urbanization of a country. With theincrease in urbanization comes the increase in energy
demand as well. China’s energy consumption in 2003

1. Introduction
Fossil fuels have been lambasted for being one of themain contributors of global warming. The burning offossil fuels releases harmful chemicals into theatmosphere such as carbon dioxide and nitrogen oxides
into the atmosphere. Carbon dioxide contributes to thegreenhouse effect in which the heat from the sun istrapped in the atmosphere. Since the mid-19th century,
the United States alone were responsible for a whopping
29% of carbon dioxide emissions by humans or©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
782

Ian Hoh Wen Yang, Ammar A.M. Al-Talib, Noor Idayu Mohd Tahir, Ayu Haslija Binti Abu Bakar, Nur Muhammad Afifi bin Zainal

2.3. Prototype Setup

reached 1678 million tonnes coal equivalent (or MTCE)
where coal consumption took up 67% of the total
consumption, oil taking up 23% and hydroelectricity andnatural gas being 7% and 3%, respectively [3]. Thisshows that the burning of fossil fuels remains one of thepreferred fuels for energy generation.
With the continuous use of fossil fuels, comes the
detriment of the environment and eventual depletion ofthis fuel source as fossil fuels are not renewable. A
promising solution for this is to use biogas as analternative to fossil fuels. The decomposition of waste
products such as animal manure and food wastes throughanaerobic digestion of different microorganisms createsbiogas as one of the end products of the decomposition
process [4]. All that is required by the average householdto create their own biogas is an affordable way to monitor
the biogas production.
IoT helps connect previously unconnected and dumb
physical devices while simultaneously giving them theintelligence to act on a command or situation. It is likelythat the integration of IoT technology and other industries
can change the mode of economic development, achievegreen growth and low-carbon economy [5]. IoTtechnology has been widely used in environment
protection, industry monitoring, food tracing source,
logistic trading and other fields [6]. In this paper, asystem that utilizes IoT to monitor components in abiogas reactor will be presented.

The basic idea of the system is to monitor the biogas
generated in the biogas reactor and update the remote
database. The microcontroller collects data from thesensors and then sends it to the Blynk mobile application
and Google spreadsheets via IFTTT. Fig. 1 shows theoverall architecture of the system proposed while Fig. 2
shows the system flowchart. The reactor contains a
mixture of inoculum, food waste and rabbit dung. Biogas
produced in the reactor travels from the reactor to amason jar containing the sensors of the data acquisition
device.

Fig. 1 Overall System Architecture

2. Methodology and Experimental Setup
2.1. Materials
Biogas samples were obtained by mixing animal
manure and food waste in a blender and then adding
inoculum to the solution. Desiccant beads were includedin the biogas chamber to remove moisture.
2.2. Apparatus and Equipment
The microcontroller used for the data acquisition device
is an ESP-32. An MQ-4 semiconductor sensor was used
for the detection of methane. An MQ-135 sensor wasused to detect the presence of carbon dioxide in the air
while a DHT-11 sensor is used for detecting the presenceof temperature and humidity. A soldering set was used tocreate a circuit board for the data acquisition device.
IFTTT protocol was used to constantly update a Google
Sheets file with the data obtained from the sensors.

Fig. 2 System Flowchart
The data acquisition device is placed together with thebiogas reactor. The device measures the methane gas aswell as carbon dioxide produced by the biogas reactor
and then uploads the data to Blynk’s database and a
Google spreadsheet via IFTTT. The data can then be
viewed on Blynk’s dashboard. The data acquisition©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
783

Gas Detection for Biogas

device requires an active internet connection to send data.
A Telegram bot was created to receive alerts from thedevice if the data is out of the specified range.
2.3.1 Preheating Process
The MQ-4 and MQ-135 sensors require initialpreheating before the sensors provide stable readings.
The datasheet advises that the sensors be given a 24-hour
period to preheat. The sensors were found to provide
stable readings after 12-hours from the start of power on.
The subsequent use of the sensors requires roughly 10minutes of preheating.

Fig. 4 The dashboard on the computer

2.3.2 Sensor Calibration Process
The MQ-4 and MQ-135 sensors require calibration.
After the sensors were preheated and readings stabilized,
data from a reference sensor was used to assist in thecalibration process. Both readings were compared andthe difference in data was compensated for in the code
written for the ESP-32 microcontroller.
Fig. 5 The dashboard on the mobile application
2.3.3 Data Acquisition Device fabrication
2.3.5 Telegram Alert Function

The data acquisition device consists of an MQ-4 sensor
to detect methane, MQ-135 sensor to detect carbon
dioxide, and DHT-11 to detect temperature including
humidity. The microcontroller used for the device is an
ESP-32. Fig. 3 shows the circuit diagram of the device.

A Telegram message bot was created to receive alertsfrom the device in an event where a specific condition
has been met (Fig. 6). An example of this would be whenthe methane or carbon dioxide levels drop below a certain
level or when methane is detected. When the methane orcarbon dioxide levels drop from a high concentration toa lower concentration, it may signify that a leak is present
in the chamber and allows the user to perform data-driven
decision making.

Fig. 3 Data Acquisition Device Circuit Diagram
2.3.4 Blynk Dashboard and IFTTT
A dashboard was designed to show the data obtained by
the device. The dashboard displays the data in the formof a graph and the value at an instance on a gauge. The
data is provided in PPM (Parts Per Million) over time.
Fig. 4 shows the dashboard as viewed on a computer andFig. 5 shows the dashboard on the mobile application.

Fig. 6 Telegram Alert Function
2.3.6 Test Setup
The first prototype was built and tested as shown in Fig.
7. From the labels in Fig. 7, one represents the reference
sensor, two represents the biogas reactor containing foodwaste and animal manure, three represents the sensors ina mason jar, and four represents the microcontroller.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
784

Ian Hoh Wen Yang, Ammar A.M. Al-Talib, Noor Idayu Mohd Tahir, Ayu Haslija Binti Abu Bakar, Nur Muhammad Afifi bin Zainal

Fig. 9 Carbon dioxide comparison graphFig. 7 Prototype test setup
Fig. 10 Temperature comparison graph

3. Result and Discussion
3.1. Data Acquisition Device
In determining the sensor’s accuracy, the sensor
readings of the MQ-4 and MQ-135 were compared to asecondary set of data obtained in clean air and in biogas.
The DHT-11 sensor data was compared with anothermeasuring device that measures temperature andhumidity.
Fig. 8 shows results of the MQ-4 sensor in clean air.
The sensor readings fluctuate slightly when compared tothe secondary set of data due to noise. Fig. 9 shows theresults of the MQ-135 sensor in clean air. The data showsa similar trend when compared to the secondary set ofdata in clean air. Fig. 10 and Fig. 11 shows thecomparison between the data obtained by the DHT-11
sensor and the reference measuring device. It was foundthat the temperature data from the DHT-11 sensor
deviated about two degrees Celsius while the humidity
data was found to deviate about three percent. As the dataof the MQ-4 and MQ-135 fluctuates drastically, atabulation of data was created for comparison (Table 1).

Fig. 11 Humidity comparison graph

Table 1 Sensor Data in Clean Air
Sensor Data
MQ-4

MQ135

Reference(methane)

1

0

474

0

Reference(carbondioxide)
453

2

0

449

0

458

3

0

456

0

460

4

0

454

0

461

5

0

467

0

459

6

0

458

0

455

7

0

449

0

458

8

7

450

0

457

9

0

451

0

459

10

0

457

0

459

11

7

449

0

455

No.

Fig. 8 Methane comparison graph©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
785

Gas Detection for Biogas

Sensor Data
MQ-4

MQ135

Reference(methane)

12

3

457

0

Reference(carbondioxide)
456

13

8

454

0

453

14

1

461

0

456

15

0

460

0

458

16

9

446

0

459

17

0

457

0

456

18

0

453

0

458

19

0

457

0

462

20

0

454

0

463

21

0

448

0

462

22

0

456

0

460

23

5

457

0

461

24

2

455

0

465

25

1

464

0

466

26

4

465

0

461

27

3

453

0

462

28

7

446

0

461

29

0

467

0

458

30

3

440

0

461

31

4

465

0

463

No.

dioxide and is deemed acceptable. The data for Fig. 12
and Fig. 13 were tabulated in Table 2. The readings fromthe DHT-11 were affected by the heat generated by the
MQ-4 and MQ-135. Due to the reference sensor being
placed in a separate compartment, the data of the DHT11 and reference sensor was different and was thus
omitted.

Fig. 12 Comparison Between MQ-4 and Reference in
BiogasFig. 13 Comparison Between MQ-135 and Reference in
BiogasFig. 12 shows the results of the MQ-4 sensor in biogas.
It was observed that the MQ-4 sensor provided nearsimilar readings compared to the reference sensor until
near the end where a leak was likely the cause of the large
deviation. Fig. 13 shows the MQ-135 sensor in biogas.
The percentage of error was calculated using Eq. (1).
𝑃𝑒𝑟𝑐𝑒𝑛𝑡 𝑒𝑟𝑟𝑜𝑟 =𝐴𝑐𝑡−𝑀𝑒𝑎𝑠𝐴𝑐𝑡× 100%

Table 2 Sensor Data in Biogas
Sensor Data
No
.

MQ4

MQ135

Ref. (CH4)

Ref. (CO2)%error(CH4)%error
(CO2)

1

432

881

444

1451

2.703

39.283

2

438

882

442

1462

0.905

39.672

3

430

887

437

1459

1.602

39.205

4

440

897

435

1452

1.149

38.223

5

438

885

432

1471

1.389

39.837

6

436

884

430

1466

1.395

39.700

7

429

883

428

1468

0.234

39.850

8

434

885

425

1476

2.118

40.041

9

419

892

423

1473

0.946

39.443(1)

Where;
Act= Actual reading
Meas= Measured reading
It was discovered that the MQ-4 sensor had an error ofless than 6% and was deemed accurate. It was discoveredthat in higher carbon dioxide concentration, the MQ-135
becomes less sensitive and thus has a 41% error.
However, the MQ-135 did detect the presence of carbon©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
786

Ian Hoh Wen Yang, Ammar A.M. Al-Talib, Noor Idayu Mohd Tahir, Ayu Haslija Binti Abu Bakar, Nur Muhammad Afifi bin Zainal

Table 3 Power Consumption

Sensor Data
No
.

MQ4

MQ135

%error(CH4)%error(CO2)

Ref. (CH4)

1479

1.663

39.824

423

1482

1.655

40.351

MQ-4

Current
draw
(mA)
150

10

428

890

421

11

430

884

12

434

886

425

1479

2.118

40.09513

MQ-135

150

0.75

429

888

428

1485

0.234

40.202

DHT-110.3

0.0015

14

436

980

432

1482

0.926

33.873

ESP-32

240

1.2

15

419

890

435

1488

3.678

40.188

Total

540.3

2.7015

16

434

877

440

1485

1.364

40.943

17

436

922

447

1484

2.461

37.871

18

440

890

444

1487

0.901

40.148

19

445

885

442

1482

0.679

40.283

20

440

887

446

1475

1.345

39.864

21

437

889

442

1479

1.131

39.892

22

436

895

440

1484

0.909

39.690

23

435

892

447

1479

2.685

39.689

24

437

889

449

1485

2.673

40.135

25

436

889

451

1484

3.326

40.094

26

434

890

454

1490

4.405

40.268

27

438

892

456

1487

3.947

40.013

28

437

890

461

1493

5.206

40.388

29

439

892

463

1491

5.184

40.174

30

440

902

462

1486

4.762

39.300Ref. (CO2)Component

Power
(W)
0.75

3.3. Project Cost Analysis
The project cost analysis breakdown is shown in Table
4. Some of the components of the project such as thewires and soldering iron were not taken into account foras the cost is negligible due to only using a small amount.
Table 4 Cost Breakdown
Part
name
ESP-32

Individual
Price
(MYR)
15.00

MQ-4
MQ135
DHT11
PCB
Mason
Jar

*To save space methane and carbon dioxide has beenabbreviated to CH4 and CO2 respectively.

Quantity

Total
(MYR)

1

15.00

5.20

1

5.20

6.50

1

6.50

4.90

1

4.90

1.00

1

1.00

10.00

1

10.00

Total

42.60

3.2. Power Consumption
The current draw of the sensors including the ESP-32
were pulled from the data sheet. The power consumption
was calculated using Eq. (2).
𝑃𝑜𝑤𝑒𝑟 = 𝐶𝑢𝑟𝑟𝑒𝑛𝑡  𝑉𝑜𝑙𝑡𝑎𝑔𝑒(2)

4. Conclusion
The design proposed in this paper is a scalable
solution to monitor the production of methane and carbon
dioxide within the biogas which are important parameters
to determine the efficiency of the biogas reactor. The
proposed design was tested and was deemed capable indetecting the presence of biogas and carbon dioxide in
the biogas reactor.

Table 3 shows that the estimated power consumption of
the device totals up to 2.702W. Although the power was
calculated using the maximum rated current draw fromthe component’s respective datasheet, the actual power
consumption of the device could potentially be higher
due to energy loss in the form of heat.

Acknowledgment
The author would like to take this opportunity to thank
UCSI University, who provided the opportunity toundertake the challenge to create this biogas monitoring
device. Besides that, the author would like to thank Dr.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
787

Powered by TCPDF (www.tcpdf.org)

Gas Detection for Biogas

Ayu Haslija from the chemical side of the university for
preparing the biogas samples. A massive thank you to Ir.
Noor Idayu Mohd Tahir for supervising this project.

Noor Idayu Binti Mohd Tahir
She graduated from University Teknologi
MARA, Malaysia in MechanicalEngineering. She was a design engineer in
oil and gas industries for 7 years.
Currently, she works as a lecturer under
the Department of Mechanical andMechatronics, Faculty of Technology and Built
Environment, UCSI Kuala Lumpur. Her research interests
include design and autonomous robotics.

References
1.

2.

3.

4.

5.

6.

K. A. Hossain, Global warming and impact to third
world
countries,
http://www.ieomsociety.org/imeom/44.pdf (accessed
Jul. 27, 2023).
H. Ritchie et al., “Population growth,” Our world inData,
https://ourworldindata.org/populationgrowth?utm_campaign=Weekly%2Bnewsletter%2Bof%2BIgor%2BChykalov&utm_medium=email&utm_so
urce=Revue%2Bnewsletter#citation (accessed Jul.20,2023).
P. Crompton and Y. Wu, “Energy consumption in
China: Past trends and future directions,” Energy
Economics, vol. 27, no. 1, pp. 195–208, 2005.
doi:10.1016/j.eneco.2004.10.006
P. G. Kougias and I. Angelidaki, “Biogas and its
opportunities—a review,” Frontiers of Environmental
Science & Engineering, vol. 12, no. 3, 2018.
doi:10.1007/s11783-018-1037-8
R. Liu and J. Wang, “Internet of things: Application and
Prospect,” MATEC Web of Conferences, vol. 100, p.
02034, 2017. doi:10.1051/matecconf/201710002034
L. Wan, D. Sun, and J. Deng, “Application of IOT in
Building Energy Consumption Supervision,” 2010
International Conference on Anti-Counterfeiting,
Security
andIdentification,2010.
doi:10.1109/icasid.2010.5551509

Ayu Haslija binti Abu Bakar
She received her B. Eng in Chemical
Engineering in Universiti Teknologi
Malaysia. She then received her Master’s
degree and PhD in University Putra
Malaysia. Currently, she is a lecturer under
the Department of Chemical & Petroleum
Engineering in UCSI Kuala Lumpur. Her research interests
include Waste Management (solid & Liquid waste), Pup &Paper (Development of antibacterial packaging,nanocomposite), and Wastewater Treatment (Absorptionof pollutant in water & wastewater).

Nur Muhammad Afifi bin Zainal
He finished his B. Eng degree in Chemical
Engineering in University Malaysia
Pahang. He then continued his M. Sc
degree in University Kebangsaan
Malaysia. He is currently pursuing a PhD
in University Putra Malaysia whileworking for Tenaga Nasional Berhad
Research as a researcher under the Emission and Waste
Management Unit.

Authors Introduction
Ammar Abdulaziz Al Talib
He has finished his B.Sc and M.Sc degrees
in Mechanical Engineering from the
University of Mosul/Iraq. He has finishedhis Ph.D degree from UPM University /Malaysia. He is also a Chartered Engineer
and Member of the Institute of MechanicalEngineers / UK. (CEng. MIMechE). He
has developed all the Postgraduate Programs at the Facultyof Engineering at UCSI University / Malaysia and worked
as the Head of Postgraduate and Research department at
the same faculty for the years 2010-2018.
Ian Hoh Wen Yang
He completed his Bachelor’s degree in
Mechatronic Engineering from UCSI
University, Kuala Lumpur. He is slated tograduate in 2024.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
788

IoT Based Smart Mushroom Growing Kit
Ammar A.M. Al-Talib

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Cynthia Kuan Jing Ting

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Noor Idayu Mohd Tahir

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Ain Atiqa Binti Mustafa

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
Tan Yong Hui

Department of Mechanical and Mechatronics, Faculty of Engineering, Technology and Built Environment,
UCSI University, 56000 Kuala Lumpur, Malaysia
E-mail: ammart@ucsiuniversity.edu.my, 1001956246@ucsiuniversity.edu.my, NoorIdayu@ucsiuniversity.edu.my,
AinAtiqa@ucsiuniversity.edu.my, tanyh@ucsiuniversity.edu.my
Abstract
This project introduces an IoT-based smart mushroom growing kit to meet the rising global demand for high-quality
mushrooms. Various species of mushrooms can be efficiently grown in different environmental conditions with thehelp of IoT devices that enable farmers to regulate the climate condition according to the specific needs of each typeof mushroom. The kit employs sensors and actuators, including temperature, humidity, MQ-135, and ultrasonic sensors,along with an ESP32 camera, controlled by a microcontroller. The collected data is transmitted to an IoT platform via
Wi-Fi, facilitating real-time monitoring and control through a user-friendly dashboard on Blynk website and Blynk app.
This innovative system optimizes mushroom cultivation by adjusting environmental conditions, offering efficiency andprofitability. Users can remotely monitor and regulate the growth environment through their smartphones, enhancing
the overall mushroom cultivation experience.
Keywords: Internet of Things; Environmental Condition Control, Mushroom Growing Kit

the mushrooms fresh [4]. Despite the fact that,
Malaysia is a tropical country with suitable climatescondition to grow and produce various mushrooms
locally, Malaysia had imported more than five tons ofmushrooms every year since 2009 and has reached up
to 10 million tons in 2012 which cost more than USD
3.0 million from China [2]. This shows that, themushroom industry in Malaysia is inadequate as the
local mushroom production can barely supply enoughmushroom for local consumers [5]. There are several
techniques that can be used in mushroom cultivation.
Machine learning technique is one of the advanced
technologies used in large-scale farming for crops
monitoring and pest control. Fuzzy logic technique andcommand control programming were also used inurban farming and greenhouse to monitor theenvironmental parameters [6]. However, the cost of
implementing advanced technology can be relatively

1. Introduction
With the growth of cities and urbanization in
Malaysia, there is a growing demand for sustainable
food production within the urban areas. Mushrooms aretypes of fungi that are known as high protein food
which is beneficial for health. Mushrooms were mostly
used in health supplements and used as an ingredient inmeals to provide nutritional properties to the humanbody [1]. The demand for mushrooms in Malaysia was
expected to increase due to the health awareness among
the people increases [2]. Based on Hedley [3], theduration of the complete mushroom life cycle can varyfrom as short as one day to couple of years based on
the types of the mushroom species. Thus, it is best to
consume the mushroom fresh within a day. Other thanthat, drying, freezing, canning, pickling are thecommon processing and preservation methods to keep

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
789

Ammar A.M. Al-Talib, Cynthia Kuan Jing Ting, Noor Idayu Mohd Tahir, Ain Atiqa Binti Mustafa, Tan Yong Huihigh, making it difficult for small-scale farmers toafford [7].

consists of the microcontroller, sensors, actuators andpower supply. A three-pin plug was used to connect the
240V AC input power generated from the power
station to the house to the power supply transformer toconvert voltage 240V AC to 12V DC. Then, the 12V
DC output voltage positive and negative pole wereconnected to the terminal block to supply 12V DC to
the fan, water pump, LED light and buck converter. In
addition, the ESP32 microcontroller, MQ-135 air
quality sensor, HC-SR04 ultrasonic sensor, ESP32CAM and relay were connected to the output of thebuck converter where 12V DC is step down to 5V DC.
Lastly, the DHT22 temperature and humidity sensor
was powered with 3.3V DC from the ESP32
microcontroller.

The paper shows the development of an IoT-based
smart mushroom growing kits that allows beginners
and urban farmers to monitor and remotely control thegrowing condition of mushroom easily through smart
device without being worried or being at the presence
using Blynk IoT application and sensors. The
automated IoT smart mushroom growing system canhelp users to take care of the mushrooms and save user
time while enjoying growing mushrooms at home forcooking or to earn extra income.
2. Methodology and Experimental Setup
ESP32 microcontroller was used in this project. The
IoT platform chosen for this project is Blynk as it is a
mobile application-based platform, easy to use and it
offers a user-friendly interface to control and monitor
the IoT devices. The controlling parameters and output
devices of this project consist of temperature, relative
humidity, carbon dioxide level, water level, LED light,
fan, misting system and camera.
A. System block diagram

Fig. 2 System wiring diagram
The IoT-based smart mushroom growing kit was
designed to monitor the growth of mushrooms under
the desirable environmental condition by obtaining
data from the sensors and controlling the actuators
using IoT technology. Fig. 1 shows the block diagramof the IoT system where the input and output devices
were connected to the microcontroller and the ESP32
microcontroller communicates with Blynk IoT
platform through Wi-Fi internet connection.

C. System structure design
Fig. 3 shows the structure design of the mushroom
growing kit using SolidWorks software which consistsof the misting system, LED light, ESP32-CAM forvideo streaming, MQ-135 air quality sensor, DHT22
temperature and humidity sensor, a DC fan and an
ultrasonic sensor. The material used for the mushroom
growing kit structure is stainless steel with a dimensionof 450mm length, 360mm width and 690mm height.
The kit was designed to fit 10 mushroom bags inpyramid arrangement. The electrical component
section was designed to keep the electrical components
safe on top of the kit and the water tank section wasused to place the plastic container used as the watertank for the misting system, while the mushroom room
section was used to place the mushroom bags formushroom development. The components used in the
IoT-based smart mushroom growing kit were arranged
and labeled in Fig. 3 and Fig. 4.

Fig. 1 Block diagram of the system
B. Control System wiring diagram
The control system consists of three subsystems
which are the lighting, ventilation and misting system.
Fig. 2 shows the wiring connection of the system which©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
790

IoT Based Smart Mushroom

Fig. 5 Mushroom growing kit prototype
Fig. 3 IoT-based smart mushroom growing kit section

Fig. 6 LED Light setting on Blynk app interface
Desired value for temperature, relative humidity andCO2 concentration can be set by user in the Blynk app asshown in Fig. 7. Once the desired values are set, the data
will be read by ESP32 microcontroller, and the actuators
will operate according to the command given by the
ESP32 microcontroller to alter the kit environmental
temperature, relative humidity and CO2 concentration.
The user is able to monitor and control the kit from the
Blynk IoT application as measured readings from the
DHT22, MQ-135 and ultrasonic sensors are displayed on
the Blynk app IoT interface as shown in Fig. 8 and user
could view the live-stream video by clicking on the video
stream button on the Blynk app as shown in Fig. 9.

Fig. 4 IoT-based smart mushroom growing kit
components labels
3. Results and discussion
A. IoT-based smart mushroom growing kit prototype
Fig. 5 shows the kit prototype when the LED lightbutton is pressed as shown in Fig. 6. on Blynk IoTapplication. Blue light spectrum was used in the
prototype due to blue-colored LED light is the mosteffective color in enhancing the development andproduction of the oyster mushroom [8].

Fig. 7 Desired controlling variables setting on Blynk
app interface©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
791

Ammar A.M. Al-Talib, Cynthia Kuan Jing Ting, Noor Idayu Mohd Tahir, Ain Atiqa Binti Mustafa, Tan Yong Hui

Moreover, the desired value for relative
humidity was set at 70%. The relative humidity value
detected outside the prototype kit has a slight change
from 61% to 63% as shown in Fig. 11. However, it does
not reach the desired humidity value. The relative
humidity value measured from the DHT22 sensor inside
the prototype kit increases from 60.4℃ to 69.5℃
throughout the 10 minutes testing. This has proven thatthe misting system in the prototype kit was able to work
effectively by proving sufficient humidity required forthe mushroom development when the humidity detected
is low.

Fig. 8 Measured readings from sensors displayed on
Blynk app interface

Fig. 9 Video stream button
B. Sensor detection accuracy results
To analyze the accuracy of the sensors, thetemperature, relative humidity, CO2 concentration level
and water level readings obtained from a commercial air
detector, and a ruler were compared with the ultrasonicsensors used in the kit. After eight attempts of testing, theresult shown that temperature and humidity sensor(DHT22), air quality sensor (MQ-135) and ultrasonicsensors (HC-SR04) have high accuracy in detecting
temperature, relative humidity, CO2 concentration level
and water level as it has low percentage error below 6%.

Fig. 11 Relative humidity before and after IoT system
implementation graph
D. Mushroom harvest result

C. Results before and after IoT system implementation
A mushroom harvest cycle was conducted to observe
the performance of the IoT-based smart mushroom
growing prototype kit and the traditional mushroom
cultivation methods. Fig. 12 shows the mushroom
harvest using traditional method and Fig. 13 shows themushroom harvest using the IoT-based smart mushroom
growing prototype kit.

Desired temperature value was set at 27℃. Based on
Fig. 10, temperature value detected outside the prototype
kit remain constant at 28℃ and the temperature value
measured from the DHT22 sensor inside the prototype kit
reduced from 28.1℃ to 27.8℃ throughout the 10
minutes testing. This has proven that the prototype kit
was able to work according to the microcontroller
command by reading the desired value from the Blynk
IoT application to turn on the fan. However, theprototype kit was only able to reduce the temperature up
to 0.3℃ due to there is no cooler and heater implemented
in the mushroom growing kit.

Fig. 12 Mushroom harvest using traditional method
Fig. 10 Temperature before and after IoT system
implementation graph©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
792

IoT Based Smart Mushroom

Acknowledgment
The author would like to thank UCSI University forproviding support for the research. The author also would
like to express her sincere appreciation to Ammar Al
Talib and Ir. Noor Idayu Binti Mohd Tahir for theirsupervision, guidance, and support throughout theprocess of preparation for the completion of this project.
The authors also like to thank Ms. Ain Atiqa Binti
Mustapha and Dr. Tan Yong Hui for providing valuable
information and resources that were essential for thisproject.

Fig. 13 Mushroom harvest using prototype kit
The traditional harvest method took eight days for theprimordia formation while the mushroom harvest using
prototype kit only took five days which is three days
faster than the traditional harvest method. Hence, theduration for the mushroom fruitbody development using
traditional method is longer compared to the mushroom
fruitbody development using prototype kit. Other thanthat, the weight of the mushroom produced using the
traditional method was only 49 grams which is 11 grams
lesser than the 60 grams weight of the mushroom
produced using the prototype kit. The mushroom
produced from the prototype kit is bigger in size and thecolor is more even compared to the mushroom grownwith the traditional method. This shows that themushroom harvest using prototype kit is more efficientcompared to the traditional method. By using the
traditional method, user must continuously damp thecloth in water and place the cloth on top of the mushroom
bag once the cloth is dry. The prototype kit could
automatically control the environment condition toensure the surrounding has enough moisture which could
help in speeding up the mushroom development process
and to produce good quality mushrooms.

References
N. I. P. Samsudin and N. Abdullah, “Edible
mushrooms from Malaysia; a literature review on
their nutritional and medicinal properties,”
International Food Research Journal, vol. 26, no. 1,
pp.
11–31,
2019,
Available:
http://www.ifrj.upm.edu.my/26%20(01)%202019/(
2).pdf.
2. M. Zaffrie, M. Amin, and A. Harun,“Competitiveness of the Mushroom Industry in
Malaysia,” 2015.
3. E. Hedley, “The Life Cycle of a Mushroom,”
Teelixir,
Aug.
02,
2021.
https://teelixir.com/blogs/news/mushroom-lifecycle.
4. K. Sharma, “Mushroom: Cultivation and Processing,”
International Journal of Food Processing
Technology, vol. 5, pp. 9–12, 2015.
5. M. Zaffrie, M. Amin, A. Harun, M. Mukmin, and A.
Wahab, “Status and potential of mushroom industry
in Malaysia (Status dan potensi industri cendawan di
Malaysia),” Economic and Technology Management
Review, vol. 9, pp. 103–111, 2014.
6. M. A. Mohd Ariffin et al., “Enhanced IoT-BasedClimate Control for Oyster Mushroom Cultivation
Using Fuzzy Logic Approach and NodeMCU
Microcontroller,” Pertanika Journal of Science andTechnology, vol. 29, no. 4, Oct. 2021, doi:
https://doi.org/10.47836/pjst.29.4.34.
7. J. Ruan et al., “A Life Cycle Framework of Green
IoT-Based Agriculture and Its Finance, Operation,and Management Issues,” IEEE Communications
Magazine, vol. 57, no. 3, pp. 90–96, Mar. 2019, doi:
https://doi.org/10.1109/mcom.2019.1800332.
8. I. Roshita and S. Y. Goh, “Effect of exposure todifferent colors light emitting diode on the yield andphysical properties of grey and white oyster
1.

4. Conclusion
The proposed IoT-based smart mushroom growing
prototype kit has achieved all requirements to control
the environmental condition for the mushroom’s
development. The kit was capable of controlling theenvironmental variables based on the desired value set
by the user. An IoT system was created with Blynk
application to control and monitor the growth ofmushrooms through the smartphone. Other than that,users could obtain the data stored on the google sheet
for data analysis. The kit was considered a successful
innovation to support the sustainable development
goals (SDGs) to reduce the use of resources and waste
in the mushroom growing process and increases theefficiency and productivity of the mushroom growing
operation.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
793

Powered by TCPDF (www.tcpdf.org)

Ammar A.M. Al-Talib, Cynthia Kuan Jing Ting, Noor Idayu Mohd Tahir, Ain Atiqa Binti Mustafa, Tan Yong Huimushrooms,” AIP Conference Proceedings, 2018,
doi: https://doi.org/10.1063/1.5066751.

Dr. Tan Yong Hui
Dr. Tan Yong Hui finished her B.Sc and
M.Sc degrees in Biotechnology fromUTAR in 2011. She has finished her
Ph.D in mushroom biotechnology from
University of Malaya in 2022. She iscurrently a Lecturer under department of
Biotechnology.

Authors Introduction
Ammar Abdulaziz Al Talib
Dr. Ammar Al Talib has finished his
B.Sc and M.Sc degrees in MechanicalEngineering from the University ofMosul/Iraq. He has finished his Ph.D
degree from UPM University /Malaysia. He is also a Chartered
Engineer and Member of the Institute of
Mechanical Engineers / UK. (CEng.
MIMechE). He has developed all the Postgraduate
Programs at the Faculty of Engineering at UCSI University/ Malaysia and worked as the Head of Postgraduate andResearch department at the same faculty for the years
2010-2018.
Ms. Cynthia Kuan Jing Ting
She received her Bachelor’s degree in
Mechatronic Engineering in 2023 at
UCSI University, Kuala Lumpur. Her
field of research is Internet of Thing(IoT) technology.

Ir. Noor Idayu Binti Mohd Tahir
She
graduatedfromUniversity
Teknologi MARA, Malaysia in
Mechanical Engineering. She wasdesign engineer in oil and gas industries
for 7 years. Currently, she works as alecturer under the Department of
Mechanical and Mechatronics, Facultyof Technology and Built Environment, UCSI Kuala
Lumpur. Her research interests include design andautonomous robotics.
Ms. Ain Atiqa Binti Mustafa
She received her Master's degree fromthe Faculty of Electronic & Computer
Engineering,
Universiti
Teknikal
Malaysia Melaka, Malaysia in 2017. She
is currently a Tutor under department of
MechanicalandMechatronics
Engineering, UCSI University, Kuala
Lumpur.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
794

Design and Analysis of Artificial Magnetic Conductor for Metal Shielding Applications in
RFID Car Detection Applications
Eryana Hussin1,2
1

Faculty of Engineering, UCSI University, Malaysia
Electrical and Electronic Engineering, Universiti Teknologi PETRONAS, Malaysia
Email: 1EryanaEiyda@ucsiuniversity.edu.my, 2eryana_17007877@utp.edu.my

2

Azman Zakariya

Electrical and Electronic Engineering, Universiti Teknologi PETRONAS, Malaysia
Email: mazman_zakariya@utp.edu.my
Md. Ashraful Haque2,32Electrical and Electronic Engineering, Universiti Teknologi PETRONAS, Malaysia
3Electrical and Electronic Engineering, Daffodil International University, Dhaka, Bangladesh
Email: md_21000832@utp.edu.my
Nur Izzati Ali

Faculty of Electronic Engineering & Technology, University Malaysia Perlis, Malaysia
Email: nurizzatimohdali@unimap.edu.my
Abstract
In this paper, the process of developing the reformulation equation of the Square AMC is conducted. The
reformulation yielded results with less than a 5% difference for an AMC using FR-4 substrate with thickness of 3
mm. The equation for calculating capacitance was developed using the conformal mapping theory, with theassumption that inductance remains fixed at the thickness of the AMC. A comprehensive study analysis andintegration of an Artificial Magnetic Conductor (AMC) tailored explicitly for metal shielding in Radio Frequency
Identification (RFID) applications. The primary focus is to establish a stable and efficient communication link
between RFID readers and tags, ensuring reliable data transmission within metallic environments. The proposed AMC
structure exhibits a symmetrical design to facilitate seamless integration with RFID tags. This integration strategy
aims to optimize communication stability and signal fidelity despite challenging metallic surroundings.
Keywords: Artificial Magnetic Conductor (AMC), RFID, Dipole Antenna.

material. By applying the lumped circuit concept in [9],the modelling strategy based on characteristic modes, ageneric smartphone antenna’s impedance characteristic
can be achieved. The research was further extended by[10], who employed a similar circuit model to calculate
the resonant frequency and reflection phase of structures
based on Jerusalem crosses for artificial magnetic
conductors (JC-AMC) to intercept waves that would
typically strike the structure.

1. Introduction
The Artificial Magnetic Conductor (AMC) belongs to
the metamaterial structure group, specifically designed toenhance the performance of various antenna types,
including dipole, monopole, and patch antennas [1], [2],[3]. Employing AMC as the ground plane serves toenhance the antenna's gain by minimizing undesirable
back radiation and mitigating mutual coupling effects.
The AMC is characterized by its high surface impedance,
exceeding a thousand ohms, rendering it akin to an open
circuit or lossless structure under these conditions. The
reflection graph, which exhibits a phase angle of zero
degrees and a magnitude of +1 when the system is inresonance, serves as a defining feature of AMC behaviour[4], [5], [6].

In the paper presented by [11] includes the analysis ofa new equation model of resonant frequency for absorberbased on the AMC structure composed an array of square
patch. Using an equivalent LC circuit model to analyse
the square patch array's physical dimensions and material
properties, the resonant frequency is determined. The
comparable LC circuit model formula was derived using
the conformal mapping theory for neighbouring coplanar square patches. While the Gauss' law and the
Faraday's law are applied to the square patch that isparallel to the ground plane. This paper proposed amodification of the standard equation of square patch

In [7] and [8], AMC-based antenna designs were
explored to enhance gain and optimize radiation patterns
by manipulating the top layer of the Perfect Electric
Conductor (PEC) patch on a conductive substrate©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
795

Eryana Hussin, Azman Zakariya, Md. Ashraful Haque, Nur Izzati Ali

AMC for different parameters and produce smaller
discrepancy between the simulated and calculated
resonant frequency. The new equation was modified byintroducing the correction factor based on the parametricanalysis conducted. The difference of formulation forcapacitive element of the proposed structure based on theparametric analysis of patch width, substrate width andgap size.

The flowchart in Fig. 2 show the design steps ofdesigning the Square AMC using CST software. The
single structure is simulated using the frequency domain
solve with modified boundary setting shows in the
flowchart. The simulated result should meet thecharacteristics of the AMC by achieving the zeroreflection phase and high impedance at resonant [12],[13].

2. Methodology
The method of RFID tag for metal object detection
presented in the paper is divided into two parts; single
AMC design and array AMC with dipole antenna
integrations.
2.1. Single 2.45 GHz AMC design
The single AMC structure consists of three layers ofpatch, substrate, and ground plane. The size of the patch
must be smaller than the substrate to allow some gap
between the patches when integrating with the dipole
antenna in array arrangement. The resonant frequency
and bandwidth of the standard AMC structure presented
in square shape can be calculated using Eq. (1) and Eq.
(2) where the 𝐿 and 𝐶 represents the inductive andconductive elements of the structure which determine by
the Eq. (3) and Eq. (4). Fig. 1 shows the structure of thebasic square AMC with square patch on a FR-4 substratewith 3 𝑚𝑚 thickness and backed with a full ground plane.
1
𝑓=
(1)
2𝜋√𝐿𝐶
1 𝐿
(2)
√
𝜂0 𝐶
𝑤𝑝 𝜀° (1 + 𝜀𝑟 )
𝑤𝑠
(3)
𝐶=
𝑐𝑜𝑠ℎ −1 ( )𝜋𝑔
𝐿 = 𝜇° ℎ(4)where 𝑤𝑝 = patch width, 𝑤𝑠 = substrate width, 𝑔=gap
around patches, 𝜀𝑟 =substrate permittivity,ℎ =substrate thickness.
𝐵𝑤 =Fig. 2 The flowchart of single unit AMC design using
CST software.
2.2. Dipole antenna and Array AMC integration
The second part discussed about the proses of dipole
and array AMC integration. A simple dipole antenna will
be used to replicate the RFID Tag antenna. The
50𝛺 discrete port is used to replace the RFID chip at thecenter of the dipole antenna [14]. The single unit cellFig. 1 Structure of the basic square patch AMC, SPAMC©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
796

Design and Analysis of Artificial

AMC will be arranged in 2 × 2 and 3 × 3 arrangement
and will be presented Section 3.
Fig. 3 shows the flow chart of the dipole antenna andarray AMC integration. To maintain the symmetricalstructure, the array arrangement used in this chapter will
be in constant multiplication of 2 × 2 and 3 × 3
arrangement. It is important to analyze the effect ofadding the dipole antenna at different positions on the twodifferent array SP-AMC to see the effect of different
metallization around the dipole antenna.

reformulated again using the curve fit method. The
reformulation of Square Based AMC has been conductedby [11] and produced a maximum deviation of 9.59%. In
this paper, the new formulated proposed is expected to beless than 5% by using the MATLAB curve tools.
Table 1 shows the comparison of simulated andcalculated resonant frequency of the AMC with substrate
size of 50 mm and patch size varied from 18 mm to 21
mm. The calculated frequency using the standard formuladisplays a significant disparity with percentage different
more than 30% when compared to the simulated
frequency. The reformulation of the Eq. (5) was
conducted using the curve fit tools in MATLAB. The
reformulated resonant frequency (R) shown in Table 1
produce smaller discrepancy less than 1%. Fig. 4 shows
the comparison between the standard formula, simulated
and reformulation resonant frequency of the square AMC
with different patch size.
Table 1: Comparison of the S=simulated, C=Calculated,
R=Reformulated and percentage different CS= calculated
and simulated and RS = reformulated and simulated.
Patch
size
18
19
20
21

Frequency (GHz)
S
3.62
3.44
3.27
3.15

𝐶=(

C
4.81
4.59
4.38
4.19

R
3.58
3.43
3.28
3.15

Percentage
Different
CS
RS
33.0%
0.9%
33.5%
0.3%
34.0%
0.3%
33.1%
0.1%

1.63𝑤𝑝 𝜀0 (1 + 𝜀𝑟 )
)×
𝜋
(5)

𝑤𝑠
𝑐𝑜𝑠ℎ−1 ()
0.9(𝑤𝑝 − 𝑤𝑠)

2.8e+9Fig. 3 Flow chart of the dipole and array AMC

2.6e+92.4e+9Frequency (Hz)

3. Results and Discussion
3.1. Analysis of single unit cell AMC
In this section, the characteristics of the capacitive andinductive components will be investigated through thesimulated frequency. Then, the capacitance value iscalculated based on the resonant frequency. By applying
the conformal mapping theory, the inductance of the
Square AMC use in this paper will be based on the fixed
thickness of 3 mm and Fr-4 as substrate. By using the Eq.
(4), the inductance value is maintained at 3.77 nH. With
this, the capacitance value can be calculated using the Eq.
(1) and Eq. (3). The capacitance obtained from both
equations will be compared and the Eq. (3) will be

2.2e+92.0e+91.8e+91.6e+91.4e+91.2e+90.020

Simulated
Formulated Regression
Standard Reformulated
0.025

0.030

0.035

0.040

0.045

0.050

Patch size (mm)Fig. 4 Comparison of frequency based on standard
formula, simulated, and reformulated for the Square
AMC with different patch (𝒘𝒑 ) and gap (g) size.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
797

Eryana Hussin, Azman Zakariya, Md. Ashraful Haque, Nur Izzati Ali

The Eq. (5) is used to recalculate the frequency basedon the proposed structure in [15] and [16] respectively.
Table 2 shows the comparison of calculated frequency
based on the proposed equation and the actual resonant
frequency presented in the paper. Both use the same
substrate but with different thicknesses. The comparison
yields a percentage difference below 15%.

with the gap between the patches. The placement of thedipole antenna on the 2×2 array SP-AMC also affects theeffective capacitance and inductance of the SP-AMC,
hence modifying the mutual coupling and impedance ofthe structure. Based on the simulated graph, theoptimized result obtained when the dipole is place at thecenter top and bottom of the 2×2 array SP-AMC withreturn loss -28.03 dB.

Table 2: Comparison of frequency the Square Patch
AMC based on the previous papers.

Paper

Structure
dimension

𝐹𝑅 ,
GHz

𝐹𝐴 ,
GHz

Fig. 6 shows the surface current distribution of thedipole antenna on the 2×2 array SP-AMC. More inductedcurrent from the dipole antenna was distributed aroundthe top AMC structure compared to the bottom. The
AMC structure placed at the back of the dipole antenna
helps to realign the surface current to be in-phase with the
dipole antenna. Hence, overcome the problem of metal
object detection in RFID application.

Percentage
Different

10.8
× 12.4
3.40
3.90
12.8%
× 1.6
25.6
[16]
× 28.1
2.22
2.45
9.54%
× 1.25where 𝐹𝑅 = calculated frequency based proposed
reformulation in Eq. (5) and 𝐹𝐴 =actual resonant
frequency[15]

3.2. Dipole and AMC Integration and Analysis
This section will discuss the performance of the dipole
antenna on two different AMC array integrations. Dipole
antenna is placed at different positions on the array
antenna to obtain optimized results in term of gain,
directivity and efficiency. Fig. 5 shows the structure ofdipole antenna placed at two different positions on the
2 × 2 array AMC. Based on the simulated graph showsin Fig. 8, the dipole antenna on the 2×2 array SP-AMC
affects its interaction with the metallization componentsof the structure, leading to changes in impedance,
radiation pattern, near-field effects, and resonancebehavior. The overall structure for 2×2 array SP-AMC ismeasured at 92 𝑚𝑚 × 92 𝑚𝑚.

Fig. 6 Surface current distribution of the dipole
antenna backed with 2 × 2 array AMC.
The next analysis discussed on the performance ofdipole antenna on the 3 × 3 array AMC. The dipole isplaced at the top and bottom of the gap between patchedon left and right of the array the structure as shown in Fig.
7. The overall structure for 2×2 array SP-AMC ismeasured at 138 𝑚𝑚 × 138 𝑚𝑚 . Table 3 shows thecomparison of the performance of the dipole antenna on
two different square AMC array arrangements.

(a)(b)
Fig. 5 Placement of the dipole antenna with optimized
performance at the top center and bottom center on the
2 × 2 array AMC
As the dipole antenna position changed around the 2×2
array SP-AMC, the coupling and interaction with thenear-field area due to the patches of gap between thepatches has different effect to the efficiency of the energy
and radiation characteristics. For example, when the
dipole is placed at the top center and bottom center of the
2×2 array SP-AMC, the center of the dipole is in contact(a)(b)
Fig. 7 Placement of the dipole antenna with
optimized performance on the 3 × 3 array AMC©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
798

Design and Analysis of Artificial

Table 3: Comparison of the dipole antenna performancewith different AMC array arrangements.
Directivity
Structure
Gain (dB)(dB)
Dipole only
1.87
2.02
3.32
8.21
Dipole with 2 × 2
SP-AMC
3.75
7.58
Dipole with 3 × 3
SP-AMC
Fig. 9 shows the surface current distribution of thedipole antenna on the 3 × 3 array SP-AMC. More
inducted current from the dipole antenna was distributed
around the top AMC structure compared to the bottom.
The induced current are lesser as the distance between the
dipole and array cell increased due to the large gap usedin this structure. Therefore, for structure with big gap, it
is recommended to limit the array cell to achieve compact
structure.

Fig.9: Surface current distribution of the dipole antenna
backed with 3 × 3 array AMC
4. Conclusion and Recommendations
To conclude, the first part of this paper the newreformulation of square AMC has been proposed withsmaller discrepancy of less than 1% compared to thesimulated resonant frequency in CST software. The
second part discussed the performance of the simple
dipole antenna on two different arrays with optimized
results obtained at the top and bottom center of the SPAMC structure.

The return loss in Fig.8 shows the comparison between
the dipole antenna with and without the array AMC. It isobserved that the use of AMC degrades the bandwidth ofthe dipole antenna. However, the integration of dipole
and AMC shows in Table 3 improved 77% of gain and275% of directivity compared to dipole antenna without
AMC.

The dipole and array AMC structure proposed in thepaper can be used as metal shielding in RFID applications
such as car detection for parking or toll-system. The useof AMC to the dipole RFID tag can improve the detection
range as the AMC helps to reduce the back radiation of
the dipole and improve the directivity and gain of thestructure. The array arrangement in this paper is limited
to 3 × 3 to minimize the overall structure.
A bigger array is recommended for any application
which has no objection on the size limits. For low
frequency such as UHF RFID, the AMC structure withslots can be introduced to achieve compact size.
References

Fig.8: Return loss of the dipole antenna with andwithout the AMC

1.
2.

3.
4.

5.

Capolino, Filippo. Theory and phenomena ofmetamaterials. CRC Press, 2017.
Zharov, Alexander A., Ilya V. Shadrivov, and Yuri S.
Kivshar.
"Nonlinear
properties
ofleft-handed
metamaterials." Physical Review Letters 91, no. 3 (2003):
037401.
G. V. Trentini, “Partially reflecting sheet arrays,” IRE
Trans. Antennas Propag., vol. AP-4, pp. 666–671, 1956.
E. Carrubba, A, Monorchio, and G. Manara “Artificial
Magnetic Surface for Circular Polarization Movemenent”
Microwave and Optical Technology Letters, Vol 5,
pp.1782-1786, Aug 2010.
M Elena de Cos, Yuri Alvarez and Fernando Las-HEras,“Design and Characteristics of Planar Artificial Magnetic
Conductor in the RFID SHF Band” Proceeding of ForthEuropean Conference on Antenna and Propagation 2010,
pp. 1-5, Apr 2010

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
799

Powered by TCPDF (www.tcpdf.org)

Eryana Hussin, Azman Zakariya, Md. Ashraful Haque, Nur Izzati Ali

6.
7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

Sievenpiper D. F. “High Impedance Electromagnetic
Surface” PHD Thesis, University of California at Los
Angeles 1999.
D. SievenpiperLijun Zhang, Romulo F. Jimenez Broas,
Nicholas G. Alex´opolous, and Eli Yablonovitch, “HighImpedance Electromagnetic Surfaces with a Forbidden
Frequency Band” IEEE Transactions On Microwave
Theory And Techniques, Vol. 47, No. 11, November 1999,
pp: 2059 – 2074.
Constantin R. Simovski, Peter de Maagt, and Irina V.
Melchakova, “High-Impedance Surfaces Having Stable
Resonance With Respect to Polarization and Incidence
Angle” IEEE Transactions On Antennas And Propagation,
Vol. 53, No. 3, March 2005, pp: 908-914.
Jafargholi, A,Atlasbaf, Z. And Kamyab, M., "Miniaturized
dual-band dipole antenna loaded with metamaterial based
structure," 2011 19th Iranian Conference ElectricalEngineering (ICEE), pp.1-4, 17-19 May 2011.
Sujatha, M. N., and K. J. Vinoy. "A stacked ring-patch
artificial substrate for surface waves suppression and inphase reflection." In Industrial and Information Systems(ICIIS), 2010 International Conference on, pp. 75-79.
IEEE, 2010.
Kurniawan, Adit, Achmad Munir, and Levy Olivia Nur.
"Theoretical Analysis of Resonant Frequency for AMCbased Absorber Composed of Square Patch Array."International Journal on Electrical Engineering &Informatics 7.2 (2015).
Abu, M., E. E. Hussin, A. R. Othman, Fauzi Johar,
Norhidayah M. Yatim, and Rose F. Munawar. "Design Of
0.92 GHz Artificial Magnetic Conductor For Metal Object
Detection In RFID Tag Application With Little Sensitivity
To Incidence Of Angle." Journal of Theoretical & Applied
Information Technology (2014).
Abu, M., E. E. Hussin, A. R. Othman, N. M. Yatim, F. M.
Johar, and R. F. Munawar. "Design of stacked wafers
AMC at 920 MHz for metallic object detection in RFID
application." In 2013 IEEE Symposium on Wireless
Technology & Applications (ISWTA), pp. 236-239. IEEE,
2013.
Abu, M., E. E. Hussin, M. S. M. Isa, Z. Zakaria, and Z. A.
Baharudin. "Designing artificial magnetic conductor at
2.45 GHz for metallic detection in RFID tag Application."International Journal of Engineering and Technology(IJET) 6, no. 1 (2014): 427-435.
Askari, Hussain, Niamat Hussain, Domin Choi, Md Abu
Sufian, Anees Abbas, and Nam Kim. "An AMC-Based
Circularly Polarized Antenna for 5G sub-6 GHz
Communications." Computers, Materials & Continua 69,
no. 3 (2021).
Eardprab, Sanchai, Chuwong Phongcharoenpanich, andDanai Torrungrueng. "Improvement of a circular
microstrip antenna excited by four feeds and suspended
with artificial magnetic conductors." International Journalof Antennas and Propagation 2013 (2013).

Dr. Azman Zakariya
He joined Universiti Teknologi
PETRONAS in 1999. He is holding a
postion as Senior Lecturer in theElectrical and Electronic Engineering
Department. He obtained the B.Eng.
degree in Electrical & Electronic
Engineering in 1999 from Universiti
Teknologi Malaysia, Malaysia, Mastersof Science Degree in Communication And Signal
Processing in 2001 from University of Newcastle Upon
Tyne, and PhD in RF and Microwave in 2015 from
Universiti Sains Malaysia. Currently he is assigned as
Head of Research Group of Algriculture IOT (AIOT),
under Institute of Tranportation and Infrasture (ITI). His
research interest is in RF and Microwave Circuit Design,
Antenna and Wireless System and IOT. Systems.

Mr Md Ashraful Haque.
He is doing Ph.D. at the Department ofElectrical and Electronic Engineering,
Universiti Teknologi PETRONAS,
Malaysia, He got his B.Sc. inElectronics and Electronic Engineering(EEE) from Bangladesh's Rajshahi
UniversityofEngineeringandTechnology (RUET) and his M.Sc. inthe same field from Bangladesh's Islamic University ofTechnology (IUT). He is currently on leave from Daffodil
International University (DIU) in Bangladesh. His researchinterest includes microstrip patch antenna, sub 6 5G
application, regression model machine learning on antenna
design.
Dr. Nur Izzati Mohd Ali
Shei received her bachelor’s andmaster's degrees from Universiti
Teknikal Malaysia Melaka (UTeM). She
pursued her Ph.D. at Universiti
Kebangsaan
Malaysia(UKM).
Currently, she is a senior lecturer in the
Faculty of Electronic Engineering &Technology at Universiti Malaysia
Perlis. Her expertise lies in antenna design, particularly intransparent antennas, solar-integrated antennas, andwireless MIMO communication systems.

Authors Introduction
Ms. Eryana Hussin
She he received her Master's andBachelor’s degree in Electrical andElectronic Engineering from University
Teknikal Malaysia Melaka. She iscurrently pursuing her study under
Doctor of Philosophy in Electrical andElectronic Engineering in Universiti
Teknology Petronas, Malaysia. She is also a lecturer fromthe Department of Electrical and Electronics of UCSI
University, Malaysia. Her research interest covers in
Metamaterical structures, Antenna and Wireless System.
©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024800

Effect of Fiber Orientation on the Mechanical Performance of Natural Fiber Polymer
Composite Bicycle Frame using Finite Element Analysis
Kok Sem Too

Faculty of Engineering, Technology and Built Environment,
UCSI University, Kuala Lumpur, 56000, Malaysia
Cik Suhana Hassan

Faculty of Engineering, Technology and Built Environment,
UCSI University, Kuala Lumpur, 56000, Malaysia
Nor Fazilah Abdullah

Faculty of Engineering, Technology and Built Environment,
UCSI University, Kuala Lumpur, 56000, Malaysia
Ammar Abdulaziz Majeed Al-Talib

Faculty of Engineering, Technology and Built Environment,
UCSI University, Kuala Lumpur, 56000, Malaysia
Email: 1001748617@ucsiuniversity.edu.my, suhana@ucsiuniversity.edu.my,norfa@ucsiuniversity.edu.my, ammart@ucsiuniversity.edu.my

Abstract
In this research, the performance of oil palm empty fruit bunch (OPEFB) fiber-reinforced epoxy composite withvarying fiber orientation and stacking sequence as the material for mountain bike frame was studied utilizing ANSYS
software. The choice of OPEFB fiber was motivated by the fact that the waste by-product of oil palm extraction in
Malaysia alone might reach 70-80 million tons per year, with 90% of oil palm biomass lost as waste. The properties
of epoxy OPEFB composite in principal 1, 2, and 3 directions were calculated using Whitney and Riley estimates. 10
stacking sequences and five loading conditions were taken. The results show that the fibre orientation of epoxy
OPEFB composite on the bicycle frame had little effect on the performance contrary to the number of plies in thelaminate or number of laminates which had major effects.
Keywords: Biocomposites, Bicycle frame, Oil palm empty fruit bunch, Polymer composites

going to be manufactured and more carbon emissions aregoing to be produced. In Malaysia, there is also anotherproblem involving the oil palm industry. Oil palm empty
fruit bunches (OPEFB) are discarded by the industry as
waste in large quantities without utilization [2]. OPEFB
fibre has the potential to be used as reinforcing fibre incomposite materials. Therefore, utilizing OPEFB fibre in
a composite bicycle frame would ease both problems at
the same time.

1. Introduction
In the 21st century, companies and governments always
push towards green technologies, eco-friendly products,
and sustainability. In fact, in 2015, Sustainable
Development Goals (SDGs) were established by theUnited Nations (UN) to push for all countries to take partin sustainability for the conservation and preservation ofnature and resources for current and future generations[1]. SDGs were fully adopted by the Malaysian
government and taught in universities. With thisawareness, to reduce the use of resources like fuels andcut carbon emissions, bicycles become the ideal vehicle
for short-distance travel.

Many researchers have investigated the properties of
OPEFB fibre and OPEFB fibre in matrix. Gunawan et al.
[3] performed experiment to find out mechanicalproperties of 40 strands of OPEFB fibre and found thatthe average Young’s modulus is 11.88 GPa and theaverage tensile strength is 156.3 MPa. Zuhri [4]experimented on just a single strand of OPEFB fibre andfound that Young’s modulus is 1.7 GPa and tensile
strength is 71 MPa. Hassan [5] experimented on

While it is true that bicycles do not produce emissions,the manufacturing of steel, aluminium, or titanium
bicycle frames produces a lot of carbon emissions. With
the increasing popularity of bicycles, more bicycles are©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
801

Kok Sem Too, Nor Fazilah Abdullah, Ammar Abdulaziz Majeed Al-Talib, Cik Suhana Hassan

2.2. OPEFB fiber reinforced epoxy composite
properties

unidirectional OPEFB fibre-reinforced epoxy composite
in 0º, 45º, and 90º directions and found that the tensile
strengths are 30.5 MPa, 10 MPa, and 9.2 MPa
respectively.

Although many researchers have conducted
experiments on the properties of OPEFB fibre reinforced
epoxy composite, only the properties of Young’s
modulus, tensile strength, and flexural strength were
found. Properties such as shear modulus and Poisson’s
ratio in principal 1-2, 2-3, and 1-3 directions andcompressive strength and shear strength in principal 1, 2,
and 3 directions were not found. To find the moduli andPoisson’s ratios in all three directions of composite, many
theoretical estimates were developed. One suchtheoretical estimates is Whitney and Riley estimatesdeveloped by Air Force Materials Laboratory [7-8]. The
calculations of the moduli and Poisson’s ratios using
Whitney and Riley theory are less rigorous than othertheoretical estimates and only require Young’s modulus,
shear moduli, bulk moduli and Poisson’s ratios of thefibre and matrix. The theory also regards that the fibre
and matrix are isotropic. The compressive strength andshear strength of the composite were calculated using theequations in Chapter 5 of the book of Engineering
Mechanics of Composite Materials by Daniel and Ishai[9]. The properties of OPEFB fibre were obtained froman experiment done by Gunawan et al. [3] and theproperties of epoxy were obtained from Appendix A inthe book [9]. Table 2 shows the calculated properties ofthe epoxy OPEFB composite. Note that the fibre-tomatrix volume ratio is 60:40.

Utilizing unidirectional continuous OPEFB fibre in acomposite bicycle frame requires the study of the fibreorientation and stacking sequence to produce a bicycle
frame that can withstand all loading conditions. As
investigated by the researchers, the properties of OPEFB
fibre are adequate for the development of composite
bicycle frames.
2. Methodology
2.1. Mountain bicycle frame
The dimensions and geometry of the mountain bicycle
frame was adopted from a journal by Sajimsha B et al.
[6]. The modelling of the frame was completed on
SOLIDWORKS. Table 1 shows the parameters ofdimensions and geometry of the bicycle frame.
Table 1. Parameters of dimensions and geometryof the mountain bicycle frame
Parameter
Value
Head tube angle
73.5ºSeat tube angle
73.5º
Top tube length
580 mmSeat tube length
570 mmChain stay length
360 mmHead tube length
120 mmTable 2. OPEFB fibre properties
Young’s modulus 1 (GPa)8.848
Young’s modulus 2 (GPa)6.347
Young’s modulus 3 (GPa)
6.347
Poisson’s ratio 1-2
0.3184
Poisson’s ratio 2-3
0.32
Poisson’s ratio 1-3
0.3184
Shear modulus 1-2 (GPa)2.899
Shear modulus 2-3 (GPa)2.404
Shear modulus 1-3 (GPa)2.899
Bulk Modulus (GPa)5.937
Tensile strength 1 (MPa)142 MPa
Tensile strength 2 (MPa)49.46 MPa
Tensile strength 3 (MPa)49.46 MPa
Compressive Strength 1 (MPa)400 MPa
Compressive Strength 2 (MPa)143.4
Compressive Strength 3 (MPa)143.4
Shear Strength 1-2 (MPa)70.82
Shear Strength 2-3 (MPa)48
Shear Strength 1-3 (MPa)70.82

For composite material simulation, the shell model of thebicycle frame was exported to ANSYS. The meshing wasdone on ANSYS Mechanical and the completed meshedmodel has 12830 nodes and 12306 elements. Fig. 1 shows
the completed meshed model of the bicycle frame.

2.3. Stacking sequence
The stacking sequences of the plies were determinedfrom the journal of Thomas Jin-Chee LIU et al. [10] who
found 10 stacking sequences of 8-ply laminate thatFig.1 Shell model of mountain bicycle frame©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024802

produced better designs for composite bicycle frames.
The 10 stacking sequences are shown in Table 3. Eachstacking sequence is designated with a letter with 0, 45
and 90 indicating the fibre orientation angle in thecomposite. Fig. 2 shows examples of 0º fibre orientation
and 90º fibre orientation on the bottom bracket.

2.4. Loading conditions
The bicycle frame design and loading conditions
applied were based on Sajimsha B et al. [7] work. There
were five loading conditions simulated and analysed. All
the loading conditions are shown in Fig. 3. The firstloading condition is static start-up. It simulates thebicycle in a resting state with a rider sitting on the saddle
and the rider is about to start pedalling. A force of 700 N
is applied to the top of the seat tube implying the weightof the rider and a force of 200 N is applied to the bottom
bracket implying pedalling force. The head tube and rearbrackets are fixed supported. The second loading
condition is steady state pedalling. It simulates thebicycle is being pedalled out of saddle, meaning that therider is not sitting on the saddle instead standing up whilepedalling. Force of 1000 N is applied to the top of thehead tube implying the force of the rider’s hands pushingdown on the steering and force of 200 N is applied to thebottom bracket implying the pedalling force. The headtube and rear brackets are fixed supported. The third
loading condition is vertical impact. It is to simulate an
event of impact to the top of the seat tube. Downward
force of 2250 N is applied to the top of the seat tube. The
head tube and rear brackets are fixed supported. The
fourth loading condition is horizontal impact. Similarly
to third loading condition, it simulates an event of impact
to the top tube. Horizontal force of 2250 N is appliedalong the top tube. The head tube and rear brackets are
fixed supported. The fifth loading condition is rear wheel
braking. It simulates the rider is applying brake to the rear
wheel when coming to a stop. Horizontal force of 750 N
is applied to each of the rear bracket. The bottom bracket
is fixed supported.

Table 3. Composite stacking sequence
Designation
Stacking SequenceA[0/90/45/-45]sB[0/90/-45/45]sC[90/0/45/-45]sD[90/0/-45/45]sE[-45/45/0/90]sF[-45/45/90/0]sG[45/-45/0/90]sH[45/-45/90/0]sI[0/90/90/0]sJ[90/0/0/90]s

Thomas Jin-Chee LIU [10] used the same stacking
sequence of 8-ply laminate for the entire bicycle frame.
Other studies are more dynamic with having different
stacking sequence and number of plies for each section of
the bicycle frame. For instance, Hu et al. [11] separated
the bicycle frame into two sections: section A with 8-ply
laminate consisting of head tube, top tube, seat tube, andbottom bracket and section B with 5-ply laminate
consisting of seat stays and chain stays. Jung et al. [12]initially used the same stacking sequence of 8-ply
laminate for the entire bicycle frame then separated thebicycle frame based on the weak regions and failure
indexes into three sections with different stacking
sequences and 16-ply laminate only for section 3. Jung etal. [12] grouped the head tube into Group 1, top tube, seatstays, and chain stays into Group 2, and seat tube, top
tube, and bottom bracket into Group 3. Similarly, Chunet al. [13] initially employed the same stacking sequenceof 20-ply laminate throughout the bicycle frame and then
optimised different stacking sequences for each memberof the frame based on the initial results.

a.

To analyse whether the bicycle frame fails or not in aloading condition, multiple composite failure theories
were selected in composite failure tool on ANSYS.
ANSYS calculates the failure theories based on inverse
reserve factor (IRF) which tells a number that indicates
failure. IRF between 0 and 0.99 indicates that the bicycle
frame is still intact while IRF that exceeds 1 indicates thebicycle frame already fails. Composite failure theories
can be divided into three groups. Limit or noninteractive
theories like maximum stress and maximum strain
theories disregard interaction among different stresscomponents and only compare individual ply stresses or
strain with the corresponding ultimate strengths or strains[9]. Interactive theories like Tsai-Hill and Tsai-Wu
theories include all stress component in one expression
without reference to particular failure modes 9. Partiallyinteractive or failure-mode-based theories like Hashin
and Puck theories give separate criteria for fibre andinterfibre failures [9]. Five failure theories are chosen in
the composite failure tool: Maximum stress theory, TsaiWu theory, Tsai-Hill theory, Hashin theory, and Puck
theory.

0
b. 90
Fig. 2. Fibre orientation angle©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
803

Kok Sem Too, Nor Fazilah Abdullah, Ammar Abdulaziz Majeed Al-Talib, Cik Suhana Hassan

3. Results and Discussion
3.1. Initial analysis
The initial equivalent (von-mises) stress results show
good or even better performances than those of theconventional materials for some stacking sequences andloading conditions. All the stress values forcorresponding stacking sequences and loading conditions
are shown in Table 4. For comparison, the equivalent(von-mises) stress results produced by Deepak [14] werelisted in Table 5.

(a) Static start-up

Table 4. Initial equivalent (von-Mises) stress(b) Steady-state pedalling

StaticStartup

SteadyState
Pedalling

Vertical
ImpactHorizon-tal
ImpactRear
Wheel
Braking

A 11.242

4.588435.03411.12740.049B

11.2454.586435.051

11.12840.148C

9.77083.743530.2498.098132.523

D 9.7704

3.741730.2488.097633.298E

10.2484.149

31.7339.371934.823

F

10.2594.249731.7689.375835.377

G 10.235

4.144631.6959.472738.412

H 10.245

4.237931.7289.477338.717IJ

11.1964.629934.89610.99440.2399.77983.693930.2797.979432.685

Table 5. Equivalent (von-Mises) stress of
conventional materials
StaticStartup

SteadyState
Pedalli-ngVertical
Impact

Horizo
ntal
Impac
t

Rear
Wheel
Brakin-g

Steel9.8212

6.377331.21628.06917.746

Aluminium

10.9157.427033.72230.36417.780Titanium10.896

7.316834.60731.07118.022

CarbonFibre

10.94010.94034.83933.94119.498(c) Vertical impact

In static start-up, stacking sequence C, D, and J havelower stresses than all the conventional materials.
Stacking sequence E, F, G, and H have lower stresses
than aluminium 6061 T6, titanium grade 9 and carbon
fibre while stacking A, B, and I have higher stresses than
all the conventional materials. In steady state pedalling,
all the stacking sequences show lower stresses than all theconventional materials. In vertical impact, stacking
sequence C, D, and J have lower stresses than steel while
stacking sequence E, F, G, and H have lower stresses thanaluminium 6061 T6, titanium grade 9 and carbon fibre.
Stacking sequence A, B, and H have higher stresses than
all the conventional materials. In Horizontal impact, all
the stacking sequences have lower stresses than all theconventional materials contrary to rear wheel braking(d) Horizontal impact(e) Rear-wheel braking
Fig. 3. Loading conditions applied©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024804

condition which displays that all the stacking sequences
have higher stresses than all the conventional materials.
In each of the loading condition, the maximum stress isalways concentrated at the same region despite thedifferent stacking sequences. As can be seen in Fig. 4, thestress concentration is at the connection between seattube and seat stays in static start-up; connection between
down tube and bottom bracket in steady state pedalling;
connection between seat tube and seat stays in vertical
impact; connection between heat tube and top tube inhorizontal impact; intersection between chain stays inrear wheel braking.

(a) Static start-up

The bicycle frame will not necessarily fail at the stressconcentration region. The region may be able to hold the
maximum stress while other regions may not be able tohold their lower stresses. This is further seen with thelocation of maximum inverse reserve factor.
The initial total deformation results in Table 6 are notcomparable to the results of the conventional materials in
Table 7. As expected, the deformation in each loading
condition and stacking sequence is way higher than the
deformations for all the conventional materials. This isdue to the epoxy OPEFB composite having way lower
tensile and compressive moduli than all the conventional
materials. Thus, producing way higher deformations. The
deformations in rear wheel braking condition areunacceptable as 6 mm deformation is too high and will
possibly affect riding performance and stability. As
observed as well the different in deformations between
the stacking sequences in each loading condition is
insignificant.

(b) Steady-state pedalling(c) Vertical impact

Table 6. Initial total deformation induced.
StaticStartup

SteadyState
Pedalling

Vertical
ImpactHorizon-tal
ImpactRear
Wheel
Braking

A 0.29322

0.13132

0.89820.25591

6.8875

B 0.29323

0.13140.898230.25591

6.8863

C 0.29325

0.13792

0.89828

0.25626

6.8907

D 0.29325

0.13801

0.898290.25626

6.8895

E 0.29334

0.13446

0.89861

0.25608

6.8894F

0.29320.13663

0.898120.25618

6.8824

G 0.29333

0.13420.89859

0.25608

6.8927

H 0.29318

0.13636

0.898080.25617

6.886

I
J

0.29066

0.13167

0.89057

0.25316

6.83040.29083

0.136140.891130.253416.8401(d) Horizontal impact

Table 7. Total deformation of conventional
materials(e) Rear-wheel braking
Fig. 4. Stress concentration of each loading
condition©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
805

Kok Sem Too, Nor Fazilah Abdullah, Ammar Abdulaziz Majeed Al-Talib, Cik Suhana Hassan

Similarly to stress concentration, the bicycle frame
deforms at the same location in each loading condition
regardless of the stacking sequences. The bicycle frame
always deforms the most at the highest load location if
there is no fixed support at the location. The bicycle
frame deforms at the tip of the seat tube where force of
700 N is applied in static start-up; bottom bracket whereforce of 200 N is applied in steady state pedalling; tip of
the seat tube where force of 2250 N is applied in vertical
impact, connection between top tube and seat tube whereforce of 2250 N is applied to the top tube in horizontal
impact; both rear brackets where force of 750 N is applied
to each in rear wheel braking.

0.8. General trend can be seen here is that stacking
sequence A, B, G, H, and I produce lower IRF than all
the other stacking sequences in all loading conditions.
For initial results, stacking sequence A, B, G, H, and I arechosen to be general good stacking sequences. Table 9shows the top five stacking sequences in each loading
condition and they are composed of stacking sequences
A, B, G, H and I except for steady state pedalling thatswaps stacking sequence H for E. The (a), (b), (c) and (d)are indicators for the location of the maximum IRF which
occurs at only four locations; as shown in Fig. 5. Locationa is the connection between seat tube and chain stays;
location b is the connection between bottom bracket andseat tube; location c is the connection between bottom
bracket and down tube; location d is the chain stays
intersection. These locations are also the weak regions of
the bicycle frame. The designation inside the bracket
beside the IRF value is the location of the IRF. The
maximum IRF is greatly influenced by loading
conditions but not the stacking sequences except for static
start-up which shows variation in the maximum IRF
locations based on the stacking sequences.

Inverse reserve factor (IRF) is an indication if thebicycle frame will fail under the loading conditions. If theinverse reserve factor exceeds 1, the bicycle frame will
fail. As can be observed in Table 8, the bicycle frame has
relatively low IRF in static start-up and steady statepedalling but high IRF in vertical impact, horizontal
impact, and rear wheel braking, indicating that the frame
will most likely fail under these loading conditions.

Table 9. Top five stacking sequences of eachloading condition for IRF
Static Start-up

Table 8. Initial inverse reserve factor
StaticStartup
0.15252A

SteadyState
Pedalling
0.095640.52981

Horizon-tal
Impact0.2719Rear
Wheel
Braking
0.6079B 0.15182

0.095529

0.541290.27586

0.61455

C 0.20657

0.131490.801630.345440.84844

D 0.20922

0.13133

0.81165

0.349120.85809

E 0.20234

0.114120.801130.354340.80075

F 0.20584

0.11701

0.81488

0.35861

0.81305

G 0.15299

0.115210.52530.271020.71251

Vertical
ImpactH

0.15650.118210.53666

0.274160.72179

I
J

0.15032

0.096605

0.53767

0.273960.61409

0.20484

0.12935

0.79550.343250.84165[0/90/90/0]s[0/90/-45/45]s[0/90/45/-45]s[45/-45/0/90]s[45/-45/90/0]s0.15032 (a)0.15182 (a)0.15252 (a)0.15299 (c)0.1565 (c)Steady State Pedalling[0/90/-45/45]s[0/90/45/-45]s[0/90/90/0]s[-45/45/0/90]s[45/-45/0/90]s

0.09552 (c)0.09564 (c)0.09660 (c)0.11412 (c)0.11521 (c)Vertical Impact[45/-45/0/90]s[0/90/45/-45]s[45/-45/90/0]s[0/90/90/0]s[0/90/-45/45]s

The stacking sequence A, B, G, H, and I have slightly
higher IRF than 0.15 compared to stacking sequence C,
D, E, F, and J that have IRF higher than 0.2 in static startup. Stacking sequence A, B and I have IRF lower than0.1, stacking sequence E, F, G, H have IRF slight higher
than 0.11, and stacking sequence C, D, and J have IRF
higher than 0.12 in steady state pedalling.

0.5253 (b)0.52981 (b)0.53666 (b)0.53767 (b)0.54129 (b)Horizontal Impact[45/-45/0/90]s[0/90/45/-45]s[0/90/90/0]s[45/-45/90/0]s[0/90/-45/45]s

Stacking sequence A, B, G, H, I have slightly higher
IRF than 0.5 while stacking sequence C, D, E, F, and J
have IRF around 0.8 in vertical impact. Stacking
sequence A, B, G, H, and I have IRF slightly higher than
0.27 while Stacking sequence C, D, E, F, and J have IRF
higher than 0.34. Lastly, Stacking sequence A, B, and I
have slightly higher IRF than 0.6, stacking sequence G
and H have IRF slight higher than 0.7, and stacking
sequence C, D, E, F, and J have slight higher IRF than

0.27102 (b)0.2719 (b)0.27396 (b)0.27416 (b)0.27586 (b)Rear Wheel Braking[0/90/45/-45]s[0/90/90/0]s[0/90/-45/45]s[45/-45/0/90]s[45/-45/90/0]s

0.6079 (d)0.61409 (d)0.61455 (d)0.71251 (d)0.72179 (d)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
806

Table 12. Inverse reference factor for doublelaminate bicycle frame

While all the stresses, deformations, and IRFs werereduced half by just adding an additional laminate, thebicycle frame still had the same stacking sequence
throughout. Different stacking sequences for different
members of the bicycle frame is proven to be beneficial
for sustaining different loading conditions. Therefore,
based on the weak regions, the bicycle frame wassectioned into three sections: section 1 consists of top
tube, head tube, and down tube; section 2 consists of seattube alone; section 3 consists of seat stays, chain stays,
bottom bracket, and rear brackets. Section 1 would
employ stacking sequence B2 and I2, section 2 would
employ stacking sequence A2, B2, G2, and I2, and section
3 would employ stacking sequence A2, and H2. The
combinations of all the stacking sequences of eachsection are clearly listed in Table 13.

Fig. 5. Locations of maximum inverse reserve factor

3.2. Optimization
It was obvious that only one laminate of epoxy OPEFB
composite cannot be feasible for a good performancebicycle frame. The deformation was too high comparedto those of conventional materials and especially in rear
wheel braking with one laminate. The IRFs in vertical
impact, horizontal impact, and rear wheel braking werealso high with one laminate. To reduce the deformationand the IRF, another laminate of 8-ply was introduced,
creating two 8-ply laminates or 16 plies all together for
every member of the bicycle frame.

Table 13. Stacking sequences for section 1, 2,
and 3

The results, as shown in Table 10, 11 and 12, show
reduction in half for all the stresses, deformations, andIRFs in all the loading conditions. The equivalent (vonMises) stress was averagely reduced by 50.63% in static
start-up; 65.39% in steady state pedalling; 49.95% invertical impact; 41.55% in horizontal impact; 47.44% inrear wheel braking. The deformation was averagely
reduced by 57.13% in static start-up; 71.56% in steadystate pedalling; 57.68% in vertical impact; 51.79% inhorizontal impact; 60.56% in rear wheel braking. The
IRF was averagely reduced by 61% in static start-up;
70.98% in steady state pedalling; 59% in vertical impact;
63.69% in horizontal impact; 53.79% in rear wheel
braking.

Stac
king
K
L
M
N

Section 1

Section 2

Section 3[(0/90/90/0)2]s[(0/90/90/0)2]s[(0/90/90/0)2]s[(0/90/90/0)2]s[(0/90/90/0)2]s[(0/90/90/0)2]s[(0/90/-45/45)2]s[(0/90/-45/45)2]s[(45/-45/90/0)2]s[(0/90/45/-45)2]s[(45/-45/90/0)2]s[(0/90/45/-45)2]sOP[(0/90/90/0)2]s[(0/90/90/0)2]s[(0/90/45/-45)2]s[(0/90/45/-45)2]s[(45/-45/90/0)2]s[(0/90/45/-45)2]s

Q
R
S
T
U
V
W
X
Y
Z

[(0/90/90/0)2]s[(0/90/90/0)2]s[(0/90/-45/45)2]s[(0/90/-45/45)2]s[(0/90/-45/45)2]s[(0/90/-45/45)2]s[(0/90/-45/45)2]s[(0/90/-45/45)2]s[(0/90/-45/45)2]s[(0/90/-45/45)2]s[(45/-45/0/90)2]s[(45/-45/0/90)2]s[(0/90/90/0)2]s[(0/90/90/0)2]s[(0/90/-45/45)2]s[(0/90/-45/45)2]s[(0/90/45/-45)2]s[(0/90/45/-45)2]s[(45/-45/0/90)2]s[(45/-45/0/90)2]s[(45/-45/90/0)2]s[(0/90/45/-45)2]s[(45/-45/90/0)2]s[(0/90/45/-45)2]s[(45/-45/90/0)2]s[(0/90/45/-45)2]s[(45/-45/90/0)2]s[(0/90/45/-45)2]s[(45/-45/90/0)2]s[(0/90/45/-45)2]s

Equivalent (von-Mises) stress results, as shown in
Table 14, show insignificant difference between thestacking sequences except for a few stacking sequencesin some loading condition. Stacking sequence Q and Y
have lower stresses slightly above 4.5 MPa compared to
the rest of the stacking sequences that have stresses
slightly above 5 MPa in static start-up. All the stacking
sequences in steady state pedalling have stresses slightly
above 1.4 MPa and the difference between each other isinsignificant. Stacking sequence Q and Y have stresses
slightly above 14.3 MPa, stacking sequence R and Z havestresses slightly above 15.5 MPa, and all other stacking
sequences have stresses slightly above 16 MPa in vertical
impact. In horizontal impact, stacking sequence L, K, N,

Table 10. Equivalent (von-Mises) stress resultsfor double-laminate bicycle frame

Table 11. Total deformation results for doublelaminate bicycle frame©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
807

Kok Sem Too, Nor Fazilah Abdullah, Ammar Abdulaziz Majeed Al-Talib, Cik Suhana Hassan

P, R, M, O, and Q have stresses slight above 5.28 MPa
and stacking sequence T, S, V, X, Z, and U have stresses
slightly above 5.35. In rear wheel braking, stacking
sequence K, S, M, O, Q, U, W, and Y have stresses
slightly above 17.24 MPa and stacking sequence V, X, Z,
N, P, R, T, and L have stresses slightly above 18.09 MPa.
The stresses are not improved over the stresses of thesame case of double-laminate bicycle frame.

Table 16. Inverse reserve factor for doublelaminate sectioned bicycle frame
StaticStartup0.05958K(b)
0.06072L(b)
0.05919M(b)0.06034N(b)0.05918O(b)
0.06034P(b)
0.05900Q(b)
0.06016R(b)
0.05942S(b)0.06056T(b)
0.05902U(b)0.06017V(b)0.05902W(b)
0.0601X(b)0.0595Y(b)
0.0607Z(b)

Table 14. Equivalent (von-Mises) stress resultsfor double-laminate sectioned bicycle frame
StaticStartup
K 5.1354
L 5.1469
M 5.1607
N 5.1722
O 5.159
P 5.1709
Q 4.5435
R 5.0339
S 5.1407
T 5.1521
U 5.166
V 5.1775
W 5.1643
X 5.1762
Y 5.1354
Z 5.1469

SteadyState
Pedalling
1.42791.42181.429
1.42291.429
1.42291.429
1.42291.43061.42451.43161.42551.43161.42551.42791.4218

Vertical
Impact16.00916.04416.091
16.12616.08516.121
14.39715.54316.02316.05916.10516.141
16.1
16.13616.00916.044

Horizon-tal
Impact5.28875.28815.28925.28875.28925.28875.28935.28875.35025.34975.35085.35025.35085.35025.28875.2881

Rear
Wheel
Braking17.24118.09317.24218.09117.24218.09117.24218.09117.24118.09117.24218.09
17.24218.09
17.24118.093

Similar trend is observed for the total deformation for
double-laminate sectioned bicycle frame, as shown in
Table 15. The deformation difference between thestacking sequences in each loading condition is very
insignificant. The deformations do not see any
improvement over the deformations of the same case of
double-laminate bicycle frame. IRFs also do not see any
improvement over the IRFs of the same case of doublelaminate bicycle frame. The IRF difference between
stacking sequences in each loading condition is very
insignificant, as shown in Table 16.

K
L
M
N
O
P
Q
R
S
T
U
V
W
X
Y
Z

SteadyState
Pedalling
0.037681
0.0374850.037711
0.037515
0.037709
0.037514
0.037709
0.037514
0.037874
0.037678
0.037904
0.037708
0.037903
0.037707
0.037902
0.037707

Vertical
Impact0.378480.37830.37918
0.379
0.37918
0.379
0.37929
0.379110.37884
0.37867
0.37955
0.379370.379540.379370.37965
0.37948

Horizon-tal
Impact0.12185
0.121810.12188
0.12184
0.12188
0.12184
0.12188
0.12184
0.123150.123110.12318
0.123150.123190.123150.123190.12315

Vertical
Impact0.22218(b)0.22028(b)0.22057(b)
0.21869(b)0.22061(b)
0.21873(b)0.22007(b)
0.21818(b)0.22194(b)0.22004(b)0.22033(b)
0.21845(b)0.22037(b)
0.21849(b)0.22218(b)0.22028(b)

Horizontal
Impact0.10058(b)
0.09677(b)0.10025(b)0.09647(b)0.10024(b)0.09645(b)0.10006(b)
0.09626(b)0.10144(b)0.09759(b)0.1011(b)
0.09729(b)0.10109(b)
0.09727(b)0.10058(b)
0.09677
(b)

Rear
Wheel
Braking
0.29283(d)0.3086(d)0.29282(d)0.30858(d)0.29282(d)0.30857(d)
0.29282(d)0.30857(d)
0.29282(d)0.30857(d)
0.29281(d)0.30855(d)0.29281(d)0.30854(d)0.29283(d)0.3086(d)Overall, different stacking sequence for each section of
the bicycle frame does not produce any improvement at
all for epoxy OPEFB composite mountain bicycle framein this study. Major improvement of equivalent (vonMises) stresses, total deformations, and inverse reservefactor are seen with addition of laminate or ply. Unlike
other studies which found that major improvement of
performance by applying different stacking sequence todifferent member of the bicycle frame, all the otherstudies used synthetic fibre such as carbon fibre and glass
fibre which, in a composite, possess big difference
between the Young’s modulus 1 and 2, and tensile
strength 1 and 2. As calculated by using Whitney andRiley theory, epoxy OPEFB composite does not havevery big difference between the properties in principal 1
and 2 direction. Therefore, the fibre orientation of epoxy
OPEFB composite in each section of the bicycle frame
does not contribute much to the performance comparedto the fibre orientation of the entire bicycle frame which
saw some improvement in the initial results.

Table 15. Total deformation results for doublelaminate sectioned bicycle frame
StaticStartup
0.125150.12510.125360.125310.125360.125310.12540.12534
0.12530.125250.12552
0.125460.12552
0.125460.12555
0.1255

SteadyState
Pedalling
0.029985(c)0.029894
(c)
0.03
(c)
0.02991
(c)
0.03
(c)
0.029909
(c)
0.03
(c)
0.029909(c)
0.030027(c)0.029935(c)
0.030042(c)0.029951(c)
0.030042(c)0.02995(c)
0.029985(c)0.029894
(c)

Rear
Wheel
Braking2.70852.70572.71132.70852.71132.70852.71142.70862.71082.708
2.71372.71092.71372.71092.71382.711

4. Conclusion
The fibre orientation and stacking sequences of epoxy
OPEFB composite on mountain bicycle frame have beenstudied thoroughly. A total of 10 stacking sequences wereadopted from previous research. It was found that of the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024808

10 stacking sequences, five stacking sequences,[0/90/45/-45]s, [0/90/-45/45]s, [45/-45/0/90]s, [45/45/90/0]s, and [0/90/90/0]s, were found to be general
good stacking sequences. But the deformations and IRFs
of the five stacking sequences in some loading conditions
were still high. Therefore, the additional laminate wasintroduced and the results showed significant reductionby half in stresses, deformations, and IRF. The bicycle
frame was also sectioned for employment of different
stacking sequence in each section but the results showed
no improvement. Hence, for epoxy OPEFB composite
bicycle frame, additional laminate or ply will improve theperformance significantly but not employment ofdifferent stacking sequence in each member of the frame.

Experiments, Theory and Computations, Composite
Structures. 94(6), 2012, pp. 1973–1981.
9. M.D. Isaac and I. Ori, Engineering Mechanics ofComposite Materials, 2nd Edition, Oxford University
Press, 2006.
10. T.J-C. Liu and H-C. Wu, Fiber Direction andStacking Sequence Design for Bicycle Frame Madeof Carbon/Epoxy Composite Laminate, Materials &Design, 31(4), 2010, pp. 1971-1980.
11. Y. Hu, Y. Xiao, W. Shang, and J. Zhang, Effect ofFibre Direction and Stacking Sequence on Dynamic
Impact Performance of Composite Bicycle Frame,
International Journal of Crashworthiness, 22(5),2017, pp. 556-564.
12. H. Jung, J.A. Lee, K.Y. Kim, and H. Chun,
Determination of Number of Layers and Stacking
Sequence of Composite Bicycle Frame., Advanced
Materials Research, 123-125, 2010, pp. 531-534.
13. H.J. Chun, J.A. Lee, and K.T. Kang, Study of Design
Variable with Loading Condition for Composite
Laminate Bicycle Frame. In ECCM 2012 Composites at Venice, Proceedings of the 15th
European Conference on Composite Materials(ECCM 2012 - Composites at Venice, Proceedingsof the 15th European Conference on Composite
Materials), European Conference on Composite
Materials, ECCM, 2012.
14. P. Sarath, A. Deepak, H, Hrishikesh, N.S. Daniel andJinuchandran, Stress Analysis of Bicycle Frame
using Different Materials by FEA, GRD Journals forEngineering, 6(7), 2021, pp. 14-20.

Acknowledgements
The authors would like to express their gratitude
towards the engineering faculty of UCSI University.
References
1.

2.

3.

4.

5.

6.

7.

8.

P. Pradhan, L. Costa, D. Rybski, W. Lucht, and J.P.
Kropp, A Systematic Study of Sustainable
Development Goal (SDG) Interactions, Earth's
Future, 5, 2017, pp. 1169-1179.
M.K. Faizi, A.B. Shahriman, M.S. Abdul Majid,
B.M.T. Shamsul, Y.G. Ng, S.N. Basah, E.M.
Cheng, M. Afendi, M.R. Zuradzman, Khairunizam
Wan and D. Hazry, An Overview of the Oil Palm
Empty Fruit Bunch (OPEFB) Potential as
Reinforcing Fibre in Polymer Composite for Energy
Absorption Applications, MATEC Web ofConferences, 2017, 90. 01064.
F. E. Gunawan, H. Homma, S.S. Brodjonegoro, A.B.
Hudin and A. Zainuddin, Mechanical Properties of
Oil Palm Empty Fruit Bunch Fiber, Journal of Solid
Mechanics and Materials Engineering, 3(7), 2009,
pp. 943-951.
M.Z. Mohamed Yusoff, S.M Sapuan, and N. Ismail,Tensile Properties of Single Oil Palm Empty Fruit
Bunch (OPEFB) Fibre, Sains Malaysiana. 38(4),2009, pp. 525–529.
C.S. Hassan, C.W. Yeo, B.B. Sahari, S.M. Sapuan,
and N. Abdul Aziz, Mechanical Properties ofUnidirectional Oil Palm Empty Fruit Bunch(OPEFB) Fiber Reinforced Epoxy Composite, IOP
Conference Series: Materials Science andEngineering, 206, 2017, 012045.
B, Sajimsha, Analysis of Mountain Bike Frames by
ANSYS. International Journal for Research in
Applied Science and Engineering Technology, 7,
2019, pp. 1167-1175.
J.M. Whitney and M.B. Riley, (1996). Elastic
Properties of Fibre Reinforced Composite Materials,
AIAA Journal, 4(9), 1966, pp. 1537–1542.
A.P.S. Selvadurai and H. Nikopour, Transverse
Elasticity of a Unidirectionally Reinforced
Composite with an Irregular Fibre Arrangement:

Authors Introduction
Mr. Too Kok Sem
He received his Bachelor’s degree in
Mechanical Engineering in 2022 fromtheFacultyofEngineering,Technology & Built Environment,
UCSI University, Malaysia.

Dr. Cik Suhana Hassan
She received her bachelor's andmaster's degrees in 2009 and 2011,
respectively,
fromUniversiti
Teknologi PETRONAS, and her PhD
in 2019 from Universiti Putra
Malaysia. She is currently an
Assistant
Professor
attheDepartment of Mechanical andMechatronics Engineering of UCSI University. Her
research interests include the investigation of biocomposites characteristics for applications, particularly
automotive.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
809

Powered by TCPDF (www.tcpdf.org)

Kok Sem Too, Nor Fazilah Abdullah, Ammar Abdulaziz Majeed Al-Talib, Cik Suhana Hassan

Ms. Nor Fazilah Abdullah
She received her Bachelor’s degree inaerospace engineering (Hons) from
IIUM, Gombak in 2010 and her
Master’s degree in MechanicalEngineering from UKM, Bangi in2015. Currently she is pursuing
Doctoral of Philosophy programme at
UCSI University, Kuala Lumpur. Her
research interest in bio-based
nanoparticles materials.

Ammar Abdulaziz Al Talib
Dr. Ammar Al Talib has finished his
B.Sc and M.Sc degrees in
Mechanical Engineering from the
University of Mosul/Iraq.
He has finished his Ph.D degreefrom UPM University / Malaysia.
He is also is a Chartered Engineer
and Member of the Institute of
Mechanical Engineers / UK. (CEng.
MIMechE). He has developed all the Postgraduate
Programs at the Faculty of Engineering at UCSI University/ Malaysia and worked as the Head of Postgraduate andResearch department at the same faculty for the years
2010-2018.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
810

Modelling of Short-Circuit Protection for A Residential Grid-Connected BESS
Kong De Kang

Faculty of Engineering, Technology and Built Environment,
UCSI University, Kuala Lumpur, 56000, Malaysia
Farah Adilah Jamaludin

Faculty of Engineering, Technology and Built Environment,
UCSI University, Kuala Lumpur, 56000, Malaysia
Rodney H.G. Tan

Faculty of Engineering, Technology and Built Environment,
UCSI University, Kuala Lumpur, 56000, Malaysia
E-mail: farahadilah@ucsiuniversity.edu.my

Abstract
This research paper presents the power protection study on a grid-connected Battery Energy Storage System (BESS)
in a typical Malaysia low-voltage (LV) residential network. The BESS model and control algorithm is developed in
MATLAB/Simulink environment. The BESS model can charge and discharge its energy with an algorithm-controlledbidirectional AC/DC converter. The paper also presents two cases for BESS short circuit evaluation to investigate
stability of internal and external systems of BESS. The protection level on BESS is also optimized by introducing atime-delay characteristics model to coordinate circuit breakers in compliance with standards outlined in IEEE Std
1375-1998, IEC/EN 60898-2, IEC/EN 60947-2, UL 1077, and CSA 22.2. No.235. The paper presents BESS systemstability with and without overcurrent protection. As a conclusion, BESS was interrupted within stipulated time. Incase of internal short circuit protection failure, a backup protection will act on isolating BESS from the grid provided
Keywords: Battery Energy Storage System, Fault analysis, Power system stability, Protection scheme, Optimized
protection

emergency, and small-to-large scale power generation.
BESS may be the most versatile of the several storage
systems according to [5].
BESS has three primary components: an AC grid, abattery, and a battery management system [6]. The kindof battery used in the system determines the power output
and energy levels of BESS. Various research has beenpreviously conducted for modelling of BESS. In [7], theauthor has proposed a detailed modelling of BESS using
various power electronic components and control
modules have been discussed. On the other hand, asimplified BESS model using MATLAB/Simulink
logical-numerical modelling approach has been proposedby authors in [8] that can simulate the behaviors of atypical BESS.

1. Introduction
Lately, more than 200 energy storage demonstration
projects, at megawatt level and above, have been set up
across the world, indicating huge potential for further
expanded application of 154 Grid-scale energy storage
systems and applications energy storage technologies [1].
According to U.S. department of energy, ESS increases
grid resistant to disruptions, promotes clean electricity,and increases the economic value of wind and solar
power [2]. ESS is also essential to provide power at
optimal response speed to overcome the intermittence of
renewable sources and contribute to the mains voltage
regulation [3]. There are various types of ESS
technologies proposed by [4] used for transportation,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
811

Kong De Kang, Farah Adilah Jamaludin, Rodney H.G. Tan

Several issues have been faced by BESS due to its fault
behaviours. The primary issue is the high fault current
contributed from BESS [9]. The second issue is the rateof current rising [10], [11]. Regarding the two issues, thepower electronic switches used with BESS may also be
harmed by the current and its increasing rate [12]. The
sudden decline in battery state of charge (SOC), which
had an adverse effect on battery lifetime, is the final issue[13]. BESS can pose a significant safety risk to both users
and installers, with the potential to cause hazards such as
fire, flash burns, and explosion if it suffers from shortcircuit or fault if installed and used improperly [14].
Short circuits are a major contributor to the electrical andtemperature risks brought on by Li-ion batteries [15].
Accordingly, it is crucial to achieve power protection of
BESS to ensure its proper functions during critical time.
Therefore, two main areas of BESS protection areprotection of BESS itself for internal fault and isolating
BESS from a system for external fault [6]. Several
options to protection of stationary battery systems suchas circuit breakers and fuses have also been provided in[16] to DC system designer. The paper presents an LV
grid-connected BESS with and without overcurrent
protection of both internal and external circuit of BESS
alongside with fault analysis.

Fig. 1 Overview of BESS system protection modelin Simulink
2.2. BESS model
The three primary components of BESS are an AC
grid, a battery, and a battery management system aspresented in [6]. The BESS consists of a Li-Ion battery
and a bidirectional AC/DC converter. The nominal
voltage and ratings of the battery are 450V and 33.33Ah
respectively. The converter is designed using an averagemodel based VSC as the converter bridge. A subsystemthat consists of a series of mathematical blocks is built togenerate pulse-width modulation (PWM) signal for theconverter output as well as regulating the grid current. In
addition, the subsystem also provides charging anddischarging modes for BESS through current settings.
The converter circuit has a DC link capacitor forsmoothing the DC voltage as well as series parallel LCL
filter to reduce the Total Harmonic Distortion (THD) of
AC waveforms. Being the main charging source for
BESS, AC grid is designed at 230V which is a typical LV
residential network level in Malaysia. Simulation is run
to analyse the behaviours of BESS when there is no shortcircuit.

2. Methodology
2.1. System Design
As displayed in Fig. 1, the BESS is connected to atypical Malaysia LV network. The connection between
BESS and Grid is straight forward since no load isconnected to the network bus. BESS consists of a battery
module and a bidirectional AC/DC converter. The BESS
has utilized a battery module from manufacturer’s
datasheet. The battery module is referred to technicalspecifications outlined in manufacturer’s datasheet. With
the aid of a converter, the BESS can charge or discharge
the battery at constant voltage and current. The BESS is
protected by internal and external protection units againstinternal and external short circuit faults respectively.
Short circuit and power system stability can be analyzed
through DC and AC protection breaker monitoring
interface.

2.3. BESS short circuit evaluation
According to previous research on protection of BESS ina DC microgrid, fault on the bus at BESS side and fault
on DC terminals at battery side are the two fault locations
as identified in [12]. Based on the project model in thisresearch, similar fault locations have been determined ina block diagram as shown in Fig. 2. Three-phase fault
blocks from MATLAB/Simulink library are placed

across the identified fault locations as shown in Fig. 3.
The fault blocks are set to provide line-to-line fault as F1and line-to-line-to-ground fault as F2. F1 and F2 are©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
812

Modelling of Short-Circuit Protection

identified as internal fault and external fault, respectively.
The simulation is run to analyze the short circuit
behaviors of BESS in terms of voltage, current, and SoC
characteristics of its battery. The simulation is repeatedwith several short-circuit resistance adjusted in the fault
block parameter settings to observe the impacts ofdifferent network short-circuit levels on system stabilityof BESS and AC grid.

Fig. 2 Block diagram of short-circuited BESS network

Fig. 3 A short-circuited BESS model.

2.4 Proposed design of BESS protection scheme with optimization method
Circuit breaker as a main protection scheme has been provided in [6]. However, AC breaker for DC use must be
modified by adjusting the magnetic trip characteristics of an AC circuit breaker [16]. In this paper, a combination of AC
breaker (ACCB) and DC breaker (DCCB) are used for the external and internal power protection of BESS, respectively as
shown in Fig 4. During an external fault, F2, BESS will be isolated from AC bus whereas during an internal fault, F1,
battery will be isolated within the BESS system. An ideal switch is used as a basic circuit breaker to isolate battery or
BESS from the short circuit when the switch is opened. A circuit breaker trips according to its trip characteristics curve
based on the guidelines provided in [16]. A model that simulates B and C trip characteristics curves has been developed
using look-up table block from MATLAB/Simulink library. The trip characteristics used are referenced from amanufacturer datasheet for a miniature circuit breaker (MCB). As a common practice, B type curve is employed in DCCB
whereas C type curve is employed in ACCB.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
813

Kong De Kang, Farah Adilah Jamaludin, Rodney H.G. Tan

Fig. 4 Locations of circuit breaker.
Once the trip characteristics curves are verified, the curvemodel is employed in a subsystem. This subsystem isbuilt to generate signal as if it was a “relay” for opening
the circuit breaker. Some of the main parameters likecurrent ratings and breaking capacity are defined in thissystem. Trip time limit is used to define the maximum
and minimum trip time as indicated from trip curve frommanufacturer datasheet. The current ratings of a circuit
breaker are defined in [16]. Based on parameters ofgeneric
batterymodel
predefinedfrom
MATLAB/Simulink, BESS can discharge current at
approximate 14.5A. Whereas based on simulation, theAC bus is regulated by the converter at approximate 6A.
Therefore, the DCCB and ACCB current ratings from
calculation are 18.1A and 7.5A respectively. The current
ratings selection should be at least or above the calculated
values for common practice. From the available range ofcurrent ratings found in datasheet, DCCB and ACCB
current ratings are determined as 20A and 8A
respectively. In trip signal generator, MATLAB
functional codes are developed to compare the current
detected on the system the breaker is connected to withcurrent multiples of the trip curve. A time-delayed signalthat indicates trip will be sent to circuit breaker model toopen the switch. The tripping sequence of the circuit
breaker as coordinated by the trip signal generator
subsystem is summarized in a flowchart shown in Fig 5.
The trip curve model with trip signal generator subsystem
can optimize the protection scheme by introducing
delayed tripping in circuit breaker model approved withstandards outlined in IEEE Std 1375-1998, IEC/EN
60898-2, IEC/EN 60947-2, UL 1077, and CSA 22.2.
No.235. After installing the protection schemes in the
project model, system stability with protection isanalyzed.

Fig. 5 Tripping sequence of a circuit breaker model.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
814

Modelling of Short-Circuit Protection

3. Results and discussion

from the graph of SoC during BESS charging anddischarging states. Moreover, the voltage across theterminals of battery begins to decrease due to voltage
drop through the resistance. The gradual decrease involtage is caused by chemical changes occurring at thesurface of electrodes. Furthermore, the battery current
rises rapidly within 20ms to its peak at approximate
1.4kA. The gradual increase in the current is due tobattery inductance.

3.1 BESS behaviors under normal circumstances
Under normal circumstances, the voltage, current,
and SoC of BESS are displayed on graphs shown in Fig
6(a) and Fig 6(b). From the graphs, BESS is operating at
a constant voltage of 485V which is higher than its
nominal voltage, 450V due to other settings assumed bydefault in generic battery model. However, the operating
voltage is still within operating range based on datasheet.
The BESS is also charging and discharging at 3A which
is lower than current source from the grid due toimpedance losses. The current when BESS is discharging
opposes the polarity of current during charging as the
direction of current flow differs in both states. The initial
spike of current is due to initialization of the simulation.
However, it does not effect on system stability as it
happens only within 0.05s. SoC of BESS is increasing
steadily during charging state whereas decreasing
steadily during discharging state.

3.2. Analysis of BESS under external short circuit
From the graphs in Fig 7(a) and Fig 7(b), voltage ofbattery remains stable after external short circuit effects
at 0.1s. Therefore, the battery voltage does not drop due
to lower resistance on external circuit of BESS.
Moreover, the battery current decreases in both charging
and discharging states of BESS. Furthermore, the rate ofcharge and discharge of BESS also decreases. Therefore,
it is said that BESS does not display several issues as
discussed in [9]–[13] by neither having high discharged
current, voltage drop, nor sharp decrease in SoC except adecrease in power efficiency or performance of BESS.
The converter shows current blocking effect byfunctioning as a diode rectifier and pass the current
through only the reverse diodes. The grid filter between
ac grid and converter then limits the fault current throughthe diodes [10].

Fig. 6(a) BESS when charging.

Fig. 7(a) External short circuit of BESS during charging.

Fig. 6(b) BESS when discharging.
As seen from Fig 6(a) and Fig 6 (b), after a fault occurs
at 0.1s, the battery starts to discharge quickly as observed©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
815

Kong De Kang, Farah Adilah Jamaludin, Rodney H.G. Tan

short circuit does not cause BESS to discharge highcurrent but operating at lower power efficiency instead
due to current blocking effect of the converter. The
protection scheme using a combination of ACCB and
DCCB is designed for BESS power protection. The trip
characteristics curve models referenced frommanufacturer datasheet are developed to optimize thecircuit breakers in compliance with standards outlined inIEEE Std 1375-1998, IEC/EN 60898-2, IEC/EN 609472, UL 1077, and CSA 22.2. No.235 according to
manufacturer datasheet. AC breaker is also adjusted for
DC application in the modelling. Then, fault cases are
conducted son BESS to investigate the BESS systemstability with power protection. The simulation results
show that BESS operation is successfully interrupted in
0.02s by DCCB during an internal short circuit event. In
addition, ACCB trips with longer time delay to internalfault when DCCB fails to trip. Therefore, ACCB acts as
secondary protection in case of DCCB failure. During an
external short circuit event, ACCB will trip to protect
BESS from AC bus fault by means of BESS isolation.
DCCB will never trip because the converter diodes and
AC LCL filter have limited the fault current passing
through converter.

Fig. 7(b) External short circuit of BESS during
discharging.

Acknowledgement
Fig. 8 Grid stability during BESS external short circuits
under charging and discharging of BESS

The author would like to thank UCSI University forproviding the platform and resources for the students tocarry out the Final Year Project.

As seen from Fig 8, grid voltage level drops toapproximate 60V. Moreover, the grid current increases
rapidly to approximate 600A. The gradual decrease andincrease in voltage and current are due to grid impedance.
The short circuit occurs on the AC bus which is also
common point of coupling to the grid. Therefore, AC gridis directly affected by the short circuit.

References
1.

2.

4. Conclusion

3.

A grid-connected BESS model applicable to atypical LV network in Malaysia has been developed in
MATLAB/Simulink environment. The BESS can charge
or discharge energy by adjusting the current settings in
the converter block to achieve AC/DC bidirectional
conversion. Short circuit analysis has also been
conducted on the internal and external circuit of BESS.
In the internal circuit, faults are conducted across theterminals of battery. Whereas in the external circuit, fault
is conducted on the AC bus which is point of common
couplings for BESS terminals and AC grid. Based on thefault analysis, internal short circuit causes high current
discharged from BESS alongside with voltage drop andrapid decrease of SoC. These behaviors are damaging to
the battery lifespan of BESS. On the other hand, external

4.

5.

6.

7.

8.

Integrated ESS application and economic analysis, Gridscale Energy Storage Systems and Applications, pp. 153–
201, Jan. 2019.
Energy Storage: The Key to a Reliable, Clean Electricity
Supply | Department of Energy. 2022.
A. Stoppato and A. Benato, The Importance of Energy
Storage, pp. 1–26, Aug. 2017.
H. Abdi, B. Mohammadi-ivatloo, S. Javadi, A. R. Khodaei,
and E. Dehnavi, Energy Storage Systems, Distributed
Generation Systems: Design, Operation and Grid
Integration, pp. 333–368, Jan. 2017.
C. D. Parker, APPLICATIONS – STATIONARY | Energy
Storage
Systems:
Batteries,
EncyclopediaofElectrochemical Power Sources, pp. 53–64, Jan. 2009.
R. Hedding and P. Hayes, Protection of battery energy
storage systems, 2011 64th Annual Conference forProtective Relay Engineers, pp. 155–159, 2011.
X. Xu, M. Bishop, O. Donna G, and H. Chen, “Applicationand modeling of battery energy storage in power systems,”
CSEE Journal of Power and Energy Systems, vol. 2, no. 3,
pp. 82–90, Sep. 2016.
R. H. G. Tan and G. K. Tinakaran, Development of batteryenergy storage system model in MATLAB/Simulink,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
816

Powered by TCPDF (www.tcpdf.org)

Modelling of Short-Circuit Protection

9.

10.

11.

12.

13.

14.

15.

16.

International Journal of Smart Grid and Clean Energy, pp.
180–188, 2020.
M. Monadi, C. Gavriluta, A. Luna, J. I. Candela, and P.
Rodriguez, Centralized protection strategy for medium
voltage DC microgrids, IEEE Transactions on Power
Delivery, vol. 32, no. 1, pp. 430–440, Feb. 2017.
D. Salomonsson, L. Söder, and A. Sannino, Protection of
low-voltage DC microgrids, IEEE Transactions on Power
Delivery, vol. 24, no. 3, pp. 1045–1053, 2009.
D. M. Yehia and D. E. A. Mansour, Modeling andAnalysis of Superconducting Fault Current Limiter for
System Integration of Battery Banks, IEEE Transactions
on Applied Superconductivity, vol. 28, no. 4, Jun. 2018.
E. W. Nahas, D. E. A. Mansour, H. A. Abd El-Ghany, andM. M. Eissa, Accurate Fault Analysis and Proposed
Protection Scheme for Battery Energy Storage System
Integrated with DC Microgrids, 2018 20th International
Middle East Power Systems Conference, MEPCON 2018
- Proceedings, pp. 911–917, Feb. 2019.
M. Farhadi and O. A. Mohammed, A New Protection
Scheme for Multi-Bus DC Power Systems Using an EventClassification Approach, IEEE Trans Ind Appl, vol. 52, no.
4, pp. 2834–2842, Jul. 2016.
Government of Western Australia, Department of Mines,
Industry Regulation and Safety, Battery Energy Storage
Systems A guide for electrical contractors, 2019.
R. Zhao, J. Liu, and J. Gu, Simulation and experimental
study on lithium ion battery short circuit, Appl Energy, vol.
173, pp. 29–39, Jul. 2016.
1375-1998 - IEEE Guide for the Protection of Stationary
Battery Systems | IEEE Standard | IEEE Xplore. 1998.

Dr Rodney H.G. Tan
He currently is an Associate
Professor at UCSI University,
Malaysia. He received his PhD
degree in Electrical Engineeringfrom UNITEN in 2013. His researchinterests include Renewable Energy,
Energy Storage System and PowerElectronic Converter.

Authors Introduction
Mr. Kong De Kang
He received his Bachelor's
degree from the Department ofElectrical&Electronics
Engineering, UCSI University in
2023. He is currently working as
Electrical Engineer in one of
Mechanical&Electrical
Consulting Firm in Kuala
Lumpur, Malaysia.
Dr Farah Adilah Jamaludin
She is an Assistant Professor of
FacultyofEngineering
Technology&
Built
Environment, UCSI University.
She
graduatedfromtheDepartment of Electrical &Electronics
Engineering,
Universiti Putra Malaysia, in2008. She received her PhD
degree in Electrical Power Engineering from Universiti
Putra Malaysia in 2020. Her research interest is Power
System and Energy Management.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
817

An Intelligent Cargo/Warehouse Management System
Zhongheng Sun1, Yue Zhou1, Xun Sun1, Wenzhuo Fan1, Wenxuan Zhou1
1Maynooth International Engineering College, Fuzhou University, Fujian, China
E-mail: ZHONGHENG.SUN.2022@MUMAIL.IE, YUE.ZHOU.2022@MUMAIL.IE, XUN.SUN.2022@MUMAIL.IE,
WENZHUO.FAN.2022@MUMAIL.IE, WENXUAN.ZHOU.2022@MUMAIL.IE

Abstract
This paper presents a low-cost and user-friendly warehouse management system developed using Arduino and
ESP8266 hardware. The system accurately monitors temperature, humidity, and harmful gas concentration, and acts
as an intrusion detector in a warehouse environment. It also includes functions for managing and detecting goods
using RFID chips and for automatic delivery using unmanned intelligent vehicles. Data is uploaded to a cloud-based
relational database in real-time. The system provides a cost-effective and efficient solution for warehousemanagement.
Keywords: Arduino, ESP8266, Warehouse management system, RFID chips, Unmanned intelligent vehicles

monitored data to the cloud platform, which then transfers
the data to the control section for cargo management.

1. Introduction
In today's fast-paced business environment, efficient
warehouse management plays a crucial role in maintaining
a competitive edge [1], [2]. The ability to monitor andcontrol warehouse conditions accurately has a significant
impact on the quality of stored goods and overall operational
efficiency [3]. To address these needs, the development
team has developed a highly extensible, comprehensive,user-friendly,
quick-to-deploy,andcost-effective
warehouse management system [4].
The system offers real-time monitoring of temperature,humidity, harmful gas concentration, and intrusion
detection, ensuring optimal storage conditions andenhanced safety measures [5]. The collected data isseamlessly uploaded to the cloud, enabling easy access to
the warehouse conditions from anywhere.
Besides monitoring capabilities, the system also uses
ESP8266 and RFID chips to enable efficient goods
management and detection. By scanning the RFID chips [6]attached to goods boxes, the system can obtain detailed
information about the goods which includes quantity andtype. This information is then uploaded to a cloud-based
relational database [7], providing real-time visibility of
inventory levels and facilitating streamlined inventory
management.
Moreover, an experimental proportionally scaled small
car model is constructed. The small car can automatically
perform point-to-point transportation of goods. Through the
integration of routing algorithms, the system optimizes therouting of small cars, ensuring efficient and timelytransportation of goods from one location to another without
the need for human intervention. This automation reduces
manual labor and enhances operational efficiency.
The warehouse management system offers a costeffective and efficient solution that aligns with the demandsof modern business operations. By leveraging cloudcomputing, RFID, and unmanned intelligent vehicles, acomprehensive platform that enhances warehouseproductivity, accuracy, and adaptability is provided.
Figure 1 shows the overall structure of the system. The
monitoring system and management system return the

Figure 1 Overall system architecture diagram
2. Technical Background
The IoT-based warehouse monitoring and management
system, utilizing Arduino and ESP8266 hardware, offers acomprehensive solution for real-time monitoring of crucial
warehouse parameters. By continuously tracking
temperature, humidity, harmful gas concentration, andintrusion detection, the system ensures optimal storage
conditions and enhanced safety measures.
To enable seamless data integration and management,
the system leverages the Alibaba Cloud IoT platform. This
cloud-based infrastructure allows for the efficient uploading
and storage of data, providing a centralized hub formonitoring and analysis. The platform also facilitates easy
integration with third-party applications, thanks to Alibaba
Cloud's well-documented APIs, enabling rapid expansion
and deployment of new features.
In addition to real-time monitoring, the system
incorporates a cloud inventory management system basedon RFID recognition. By utilizing RFID technology, thesystem enables efficient and accurate goods tracking andmanagement. Each item is equipped with an RFID tag,which allows for quick and reliable identification, inventory
updates, and location tracking. This streamlines the818Zhongheng Sun, Yue Zhou, Xun Sun, Wenzhuo Fan, Wenxuan Zhou

inventory management process, reduces manual labor, andminimizes errors.
To facilitate the movement of goods within thewarehouse, the system included smart delivery cars. In our
experimental system, a proportional scaled small car modelis implemented. This automatic car operates based on
predefined algorithms and navigates through the warehouseto transport goods to their designated locations. The cars arealso able to operate manually using control from Android
devices. By automating this process, the system minimizes
human intervention, reduces handling costs, and improves
overall operational efficiency.
The integration of IoT in warehouse management
systems has enabled users to manage their data in real timeand proceed with accurate inventory tracking, demand
forecasting, and other crucial operations.
To develop the software for the system, Arduino IDE isutilized and Arduino UNO and ESP8266 development chip
are used and programmed using C language and Arduino
language, creating a robust and reliable software solutionthat connects seamlessly with the Alibaba Cloud IoT
platform using the WebSocket protocol. This protocol
ensures low latency, long-lasting connections, and powerefficient environment monitoring.

4. Monitoring System Design
For the monitoring system, Arduino Uno and ESP8266
in conjunction are chosen with the Arduino IDE to
implement its functionality. The Arduino Uno is a
microcontroller board based on the ATmega328P, while the
ESP8266 is a low-cost Wi-Fi microchip with a full TCP/IP
stack and microcontroller capability. These devices werechosen for their ease of use, low cost, and versatility
compared to other options. Figure 2 and Figure 3 show themonitoring system.

3. Literature Review
Zhang, B. et al. developed an intelligent warehousemanagement system utilizing the L298N chip, RC522 fornear-field communication, and W5500 module for network
communication [8]. The MSP430F5529 microcontroller
acts as the central control unit, overseeing tasks such as
motor control, data reading, and network communication.
The system facilitates smart cargo management, includingcard activation, encrypted data transmission, and remotemonitoring. This innovation streamlines warehouse
operations, saving resources, optimizing space, andensuring reliable security. Particularly beneficial for small
to medium enterprises and courier companies, it enhances
efficiency and reduces costs.

Figure 2 Picture of the monitoring system
In terms of specific functionality, a DHT11 temperatureand humidity sensor, an infrared human body sensor, and aharmful gas concentration sensor is connected to Arduino
Uno. The DHT11 sensor measures temperature andhumidity using a capacitive humidity sensor and athermistor. The harmful gas sensor detects the presence ofharmful gases such as carbon monoxide and methane. Bothsensors are connected using serial ports by the software
serial port function of Arduino. The infrared sensor detects
the presence of humans by measuring changes in infrared
radiation. It uses an analog input to transmit data to theArduino.

Christos Spandonidis et al. developed a smart container
monitoring system based on the Internet of Things [9]. Key
challenges such as weight reduction, fire smoke detection,
hardware safety, and logistics maintenance are addressed.
By integrating low-cost, low-power sensors, the system isable to track container status and detect critical events suchas fire/smoke, impacts, and accidental misuse. Provides
better and safer control over cargo loading/unloading
operations and flight processes. The experimental results
show that the system can make a breakthrough in the current
technical level of container technology and aircraft cargo
operation.
Xu Zhijie et al. took account of clear and real-time video
streams simultaneously [10]. They put forward hybrid
multichannel video stream transmission strategies for aremote surveillance system, in which high-quality
panoramic video and real-time interactive video are
combined together to satisfy surveillance system
requirements of high data stream quality and interactive
operation.

Figure 3 Hardware implementation map of the monitoring
systemAfter reading data from the sensors, the Arduino UNO
board hardware serial port transmits the data in real-time to
the ESP8266 for processing and transmission. The ESP8266
is responsible for processing the data and transmitting it toa cloud server for storage and analysis.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
819

An Intelligent Cargo/Warehouse

For the logic design of the ESP8266, MQTT and
WebSocket protocols are used for transmitting data. MQTT
is a lightweight messaging protocol for small sensors andmobile devices, while WebSocket is a protocol for fullduplex communication over a single TCP connection.
ESP8266 chip will receive data from the Arduino serial port
in bit storage format and perform simple frame header andtail check and bit processing. Then the ESP8266’s 2.4G WiFi module is used for establishing a connection to a provided
wireless network and to establish a connection with the
Alibaba Cloud server using the MQTT protocol over
WebSocket. The data is uploaded and displayed in real-timeon a front-end page. Figure 4 shows the front-end web pageand Figure 5 shows the Hardware implementation diagramof management module.

Figure 6 Photos of the RFID system and car
The implementation of RFID technology in thewarehouse management system has significant positive
impacts on management efficiency. By scanning RFID tags
attached to each item, users are able to accurately track thequantity, name and other relevant information of the cargo.
This data can then be transmitted to the cloud-based backend server, where it is processed and transformed into userfriendly visualizations. This allows warehouse managers
and personnel to have a clear overview of inventory levels,
stock movement, and overall warehouse performance at any
given time.
Furthermore, users can send instructions to the cars,
directing them to specific locations within the warehouse topick up or deliver goods. The system optimizes the routesof the car and ensures that goods of the same category are
stored together, maximizing efficiency and minimizing thetime required for manual handling.
With the integration of real-time data analytics andautomated car operations, the smart warehouse solution
provides a comprehensive and efficient approach toinventory management. The combination of RFID
technology, cloud-based computing, and automation
streamlines processes, reduces errors and enhances overall
productivity. As a result, warehouse operations become
more agile, cost-effective, and adaptable to changing
business demands.
In summary, the warehouse management module
leverages RFID scanning, cloud-based data processing, andintelligent delivery car systems to enable real-timeinventory tracking, visualization, and streamlined
operations. This intelligent solution empowers businesses toachieve higher levels of efficiency, accuracy, and scalability
in their warehousing processes.

Figure 4 Picture of the front-end web page

Figure 5 Hardware implementation diagram ofmanagement module
5. Management System Design

6. Effectiveness Validation

RFID technology is applied to scan and track goods
within the warehouse. The scanned information is thenuploaded to a cloud platform, where it is organized into
visualized tables, enabling real-time management.
Additionally, a feature on the platform that allows us to send
instructions to automated cars is implemented by the team,enabling them to transport goods efficiently. Figure 6 shows
the management module and small car are shown below.

To verify the effectiveness of the system, the teamconducted an on-site verification. The average number ofbacklogged items in the warehouse for incoming/outgoingshipments is recorded over two weeks, as well as the numberof instances where the warehouse's humidity andtemperature exceeded the standards (the temperature andhumidity test data is collected every hour in a day). The
system was not installed in the first week and then installedin the second week. Data changes in these two weeks weremonitored and analyzed, and the results are shown in Table
1. The graph of average data for using the warehousemanagement system as shown in Figure 7.
Table 1 The effect of using our warehouse management
system©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
820

Zhongheng Sun, Yue Zhou, Xun Sun, Wenzhuo Fan, Wenxuan Zhou
Items
Time

Number of
backlogged items

Temperature and Humidity
Exceedance Incidents Times

Time
Day1
Day2
Day3
Day4
Day5
Day6
Day7
Total
Average

Before
10
8
9
6
11
8
8
60
8.57

Before
7
7
8
11
9
9
7
58
8.29

After
5
3
4
2
4
1
1
20
2.86

In addition, the cloud service acts as a transition layer,allowing for seamless integration with third-party secondary
data analysis and prediction services. By leveraging theseexternal resources, users can gain deeper insights into theirdata, identify patterns, and make more informed decisions.
This integration enhances the analytical capabilities of thesystem, enabling users to extract valuable insights and drive
continuous improvement in their operations.

After
5
4
4
1
1
3
2
20
2.86

8. Conclusion
The study successfully developed a cost-effective, userfriendly and expandable warehouse management systemusing Arduino and ESP8266 hardware, which has uniqueadvances compared with similar systems [11]. It accurately
monitors temperature, humidity, gas concentration, andintrusions. The system utilizes RFID chips for goods
management and detection and unmanned intelligent
vehicles for automated delivery. Real-time data is uploaded
to a cloud-based database, providing an efficient and costeffective solution for warehouse management [12], [13],and also being effective for monitoring and making
decisions for controlling the carbon footprint of the venue
under monitoring.
Overall, this system enhances operational efficiency,
safety, and reliability. Further research can expand its
functionality and applicability in warehouse management.

Exceedance Incidents

Backlogged Items
9
8
7
6
5
4
3
2
1
0

9
8
7
6
5
4
3
2
1
0
BeforeAfter

BeforeAfter

Acknowledgement

Figure 7 Graph of average data for using the warehousemanagement system

This work was supported by Maynooth International
Engineering College, Fuzhou University.

7. Backend Cloud Service
The backend cloud service offers a comprehensive set offeatures designed to streamline and optimize users’
operations. With a wide range of functionalities, the service
serves as an effective tool to enhance the efficiency andeffectiveness of management and monitoring processes.
The main function of the cloud service is storing anddisplaying data. It provides a secure and reliable storage
infrastructure that ensures all the data are safely preserved.
The intuitive user interface allows for easy access to stored
data, enabling quick retrieval and analysis. By centralizing
the data of cargo to the cloud service, users can eliminate
the hassle of managing multiple data sources and improvedata consistency.
Moreover, the cloud service is designed to handle
connections between different terminals and data sources,
which facilitates seamless integration with various data
stream outputs, enabling real-time data synchronization andupdating. This functionality ensures that users have instantaccess to real-time information, enabling timely decisionmaking and accurate analysis.
One of the most prominent features of the cloud service
is its algorithmic path calculation capability. This featureenables users to exercise semi-automatic control over robots
or unmanned cars within their operations. The service cancalculate optimized paths for the robots, guiding them totransport goods to designated locations with minimal humanintervention. By automating this process, handling costs canbe significantly reduced, errors minimized, and overall
operational efficiency improved.

References
[1]

[2]

[3]

[4]

[5]

[6]
[7]

[8]

S. L. Ullo and G. R. Sinha, “Advances in Smart Environment Moni
toring Systems Using IoT and Sensors,” Sensors (Basel), vol. 20, no. 11, p. 3113, May 2020. doi: 10.3390/s20113113.
A. Čolaković, S. Čaušević, A. Kosovac and E. Muharemović, “A R
eview of Enabling Technologies and Solutions for IoT Based Smart
Warehouse Monitoring System,” in I. Karabegović (Ed.), New Tec
hnologies, Development and Application III. NT 2020. Lecture Not
es in Networks and Systems, vol 128. Cham: Springer, 2020. doi: 10.1007/978-3-030-46817-0_73.
Afzal, M. “IoT Based Real Time Warehouse Monitoring using
Sparkfun ESP8266 Thing Dev and Cayenne MyDevices,”
Conference Paper, April 2018.
D. Parida, A. Behera, J. K. Naik, S. Pattanaik and R. S. Nanda, "Real-time Environment Monitoring System using ESP8266 and Thing
Speak on Internet of Things Platform," 2019 International Conferen
ce on Intelligent Computing and Control Systems (ICCS), Madurai, India, 2019, pp. 225-229, doi: 10.1109/ICCS45141.2019.906545
1.
Zafar, S., Miraj, G., Baloch, R., Murtaza, D., & Arshad, K. (2018).
An IoT Based Real-Time Environmental Monitoring System Using
Arduino and Cloud Service. Engineering, Technology & Applied
Science Research, 8(4), 3238-3242.
M. Kamrul Hasan et al., “IoT-Based Warehouse Management
System,” ResearchGate, Oct. 2022.
M. K. Hasan, M. Junjie, A. K. M. Ahasan Habib, A. Al Mamun, T.
M. Ghazal and R. A. Saeed, "IoT-Based Warehouse Management
System," 2022 International Conference on Cyber Resilience(ICCR), Dubai, United Arab Emirates, 2022, pp. 1-6, doi:
10.1109/ICCR56254.2022.9995768.
Xu Zhijie, Wang Geng. A Hybrid Real-time Video Streaming
RemoteSurveillance
SystemBasedon
MobileRobot[C]//International Conference on Computer Networks andCommunication Technology (CNCT 2016). Advances in Computer
Science Research (ACRS). Paris: Atlantis Press, 2017, 54:458-466.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
821

Powered by TCPDF (www.tcpdf.org)

An Intelligent Cargo/Warehouse[9]

Spandonidis, C., Sedikos, E., Giannopoulos, F., Petsa, A.,
Theodoropoulos, P., Chatzis, K., & Galiatsatos, N. (2022). A novel
intelligent iot system for improving the safety and planning of air
cargo operations. Signals, 3(1), 95-112.
[10] Xu Zhijie, Wang Geng. A Hybrid Real-time Video Streaming
RemoteSurveillance
SystemBasedon
MobileRobot[C]//International Conference on Computer Networks andCommunication Technology (CNCT 2016). Advances in Computer
Science Research (ACRS). Paris: Atlantis Press, 2017, 54:458-466.
[11] J. Zhang et al., “Design and application of Internet of things-based
warehouse management system,” International Journal of
Production Research, vol. 56, no. 15, pp. 5217-5235, Aug. 2018.
[12] Jarašūnienė, A., Čižiūnienė, K., & Čereška, A. (2023). “Research on
Impact of IoT on Warehouse Management”. Sensors (Basel), 23(4),2213.
[13] W. C. Tan and M. S. Sidhu, “Review of RFID and IoT integrationin supply chain management,” Operations Research Perspectives,
vol. 9, p. 100229, 2022, doi: 10.1016/j.orp.2022.100229.

Mr. Wenzhou Fan
He is currently pursuing Bachelor of
Electronic Engineering as 3rd year
student in the Department ofMaynooth International Engineering
College, Fuzhou University, China.
His research interests are integrated
circuit development.

Authors Introduction
Mr. Xun Sun
He is currently pursuing Bachelor of
Electronic Engineering as 3rd year
student in the Department ofMaynooth International Engineering
College, Fuzhou University, China.
His research interests are embedded
system development.

Miss. Yue Zhou
She is currently pursuing a Bachelor of
Electronic Engineering as 3rd year
student in the Department ofMaynooth International Engineering
College, Fuzhou University, China.
His research interests are AI vision.

Mr. Wenxuan Zhou
He is currently pursuing a Bachelor of
Electronic Engineering as 3rd year
student in the Department ofMaynooth International Engineering
College, Fuzhou University, China.
His research interests are embedded
systems and hardware technology.

Mr. Zhongheng Sun
He is currently pursuing Bachelor of
Electronic Engineering as 3rd year
student in the Department ofMaynooth International Engineering
College, Fuzhou University, China.
His research interests are integrated
circuit development.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
822

A Comprehensive Approach to Design and Implement an IoT-Enabled Intelligent Shopping
Cart System with Obstacle-Aware Navigation and Enhanced Customer Engagement for
Elevated Consumer Experiences
Yao Chen1, Jiacheng Du1, Bo Peng1, Ningfei Wang1, Zehan Huang1, Wei Hong Lim2, Sew Sun Tiang2, Mastaneh Mokayef2,
Chin Hong Wong1*
1
Maynooth International Engineering College, Fuzhou University, Fujian, China
Faculty of Engineering, Technology and Built Environment, UCSI University, 1, Jalan Puncak Menara Gading, UCSIHeights, 56000 Cheras, Kuala Lumpur, Malaysia.
E-mail: yao.chen.2022@mumail.ie, jia.du.2022@mumail.ie, bo.peng.2022@mumail.ie, ning.wang.2022@mumail.ie,
zehan.huang.2022@mumail.ie, limwh@ucsiuniversity.edu.my, tiangss@ucsiuniversity.edu.my,mastaneh@ucsiuniversity.edu.my, chinhong.wong@mu.ie

2

Abstract
Supermarket shopping is an experience that everyone has in life. This project aims to design an intelligent and userfriendly shopping cart and interactive web page, to elevate the overall customer shopping experience. The shopping
cart designed in this project integrates multiple functionalities that allow users to access information about the items
within the cart via the interactive web page and exert control over the movement of the cart. While moving, theshopping cart can autonomously detect obstacles in its path and navigate around them. Simultaneously, users cancheck the total cost of their selected items by reviewing the cart's contents, enabling them to decide whether the cartshould continue following them. The project primarily utilizes an ultrasonic module to determine the cart's locationand trail a specific customer. The shopping cart employs automatic barcode scanning to identify various products,
and the WiFi module facilitates communication with the server via the MQTT protocol, enabling seamless interaction.
The fruition of this project serves as a tangible representation of the Internet of Things application, demonstrating
how connectivity through the internet enhances people's lives with greater convenience.
Keywords: Internet of Things, ultrasonic ranging, STM32singlechip, smart shopping cart

In the past, numerous researchers have conducted
diverse studies on smart shopping carts. Supermarket
owners benefit from these innovations in terms of timesavings, reduced manpower, and space efficiency,
leading to decreased investments. The smart shopping
trolley can be used in all retail shopping malls,
supermarkets, hypermarkets, and clothing showrooms [2].

1. Introduction
The widespread of information, and communication
technology, coupled with the advancements in Internet ofThings technology, has brought our life into the stage ofa smart city. The increase in electronic devices andapplications has significantly improved convenience inour daily lives. With the increasing prevalence of mobile
devices, the development of smart cities naturally falls
toward this technological landscape. Therefore, it isnecessary to develop intelligent equipment [1].

Several innovative concepts have emerged to
revolutionize shopping cart design, aiming to enhance theoverall shopping experience and reduce human efforts.
Sadia et al. [3] proposed the integration of RFID
technology to bolster the durability of productidentification tags. They suggest the implementation of
RFID in shopping carts, creating an intelligent shopping
system where all supermarket carts are equipped with
RFID tags to reduce wait times during the checkout
process and improve the customer purchasing experience.

In the shopping scene, an intelligent shopping cart cangreatly enhance the shopping experience. Essential andfundamental functions of an intelligent shopping cart
include automatic following the user as well as automatic
obstacle identification and avoidance. Anotherconvenient feature is the automated calculation of thetotal price of items in the shopping basket. This enables
customers to view the items and their respective prices on
the web page from their mobile phones, eliminating theneed for manual price calculations.

Gunawan et al. [4] installed an IOIO microcontroller
and an Android smartphone together with sensors and acontroller in a shopping cart. The basket is configured in
the shape of a two-wheeled mobile robot.

2. Literature Review©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
823

Yao Chen, Jiacheng Du, Bo Peng, Ningfei Wang, Zehan Huang, Wei Hong Lim, Sew Sun Tiang, Mastaneh Mokayef, Chin Hong Wong

Tai et al. [5] also utilized the ultrasonic sensor becauseof the advantages include a wide detection area, reduced
sensitivity to light, the ability to detect glass and shiny
surfaces, compact size, lightweight construction, minimal
memory usage, cost-effectiveness compared to Laser
Range Finder (LRF) or cameras, and lower power.
These investigations into shopping cart functionality
and application technology have significantly expanded
the research landscape of intelligent shopping carts. As aresult, tasks have become more focused and precise,yielding results that align smart shopping carts moreclosely with people's evolving needs.
3. Methodology
Figure 1Figure 1 shows a flow chart for the smart
shopping cart system from the user side whereas Figure
2 illustrates a flow chart for the operation of the shopping
cart.

Figure 2 Flow chart for the operation of the shopping
cart
When the shopping cart is started, it will detectobstacles by photoelectric switches and avoid them. The
ultrasonic sensor will provide real-time feedback on the
distance between the user and the shopping cart. If the
distance between the shopping cart and the user, as wellas the orientation of the shopping cart, exceeds the set
range, the shopping cart will move and adjust the distanceand direction. The mobile function of the shopping cart
will continue until the user stops the function on the
webpage. In addition, users can use the QR code scanning
module on the shopping cart to scan the QR code of the
product and add it to the shopping list on the webpage, tomanage the list and settle the payment on the webpage.
3.1. Microcontroller
STM32 F407 ZET6 is used as the main controller of the
shopping cart. By handling the information obtained fromsensors and servers, the controller is able to control the
movement of the shopping cart and communicate withthe user.

Figure 1 Flow chart for the smart shopping cart systemfrom the user side

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
824

A Comprehensive Approach to

3.2. PCB Module

printing technology is used to print the clamp to hold theultrasonic sensors as shown in Figure 5(b).

The PCB circuit board consists of STM32 F407ZET6
main control board, DC-DC buck module, A4950 dual
motor drive module, ultrasonic sensor, photoelectric
switch sensor, two-dimensional code module and motor
encoder as shown in Figure 3 and Figure 4. The circuitboard is highly integrated and the modular design makes
it easy to update and repair.

Figure 5 (a) The chassis and (b) the ultrasonic clamp
3.4. Mecanum Wheel
The outer ring of the Mecanum wheel as shown in
Figure 6 is equipped with rollers arranged a 45° to theaxle and contact with the ground. During rotation, thefriction generates a 45° reverse thrust force along theaxle. This oblique thrust force can be divided into two
vectors which are longitudinal and transverse. The entire
vehicle is propelled by two pairs of Mecanum wheels,
each featuring a mirrored arrangement of rollers. Each
wheel generates its own vector, and the combined forceof these vectors determines the final motion of the car[6].

Figure 3 Connection between the main board, sensors,
and wheels

Figure 6 Mecanum wheel

Figure 4 Block diagram for the circuit connection

Omnidirectional motion is achieved by installing four
Mecanum wheels at the corners of the chassis. The
overall velocity, V, can be obtained through Eq (1), where
𝑉𝑥 , and 𝑉𝑦 indicate velocity in the 𝑥 -direction and 𝑦 direction respectively and 𝜔 is the angular velocity, as
shown in Figure 7.

3.3. DC-DC Antihypertensive Module
The DC-DC antihypertensive module employs the
SY8205 chip, a high-efficiency synchronous stepdown
DC-DC converter capable of delivering a 5 A output
current. The SY8205 operates over a wide input voltage
range from 4.5 V to 30V and integrates the main switch
and synchronous switch with very low RDS(ON) to
minimize the conduction loss. The module consists oftwo voltage outputs, one 5V voltage and another output
to meet the diverse voltage requirements of various
modules.
The chassis for the shopping cart is crafted from a 6 mm
thick POM plate through laser cutting as shown in Figure5(a). The chassis is used to hold the shopping cart bracket,
circuit board and various sensors. Furthermore, 3DFigure 7: The direction of velocity.
→

→

→

→

𝑉 = 𝑉𝑋 + 𝑉𝑌 + 𝜔 × (𝑎 + 𝑏)
©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
825

(1)

Yao Chen, Jiacheng Du, Bo Peng, Ningfei Wang, Zehan Huang, Wei Hong Lim, Sew Sun Tiang, Mastaneh Mokayef, Chin Hong Wong
→

→

→

→

→

→

→

→

→

→

→

→

→

→

→

→

manipulated variable and the final control element
exhibits nonlinear behavior [5].

𝑉1 = |𝑉𝑋 | − |𝑉𝑌 | − |𝜔 × (𝑎 + 𝑏)|
𝑉2 = |𝑉𝑋 | + |𝑉𝑌 | − |𝜔 × (𝑎 + 𝑏)|

In the controlling of the Mecanum wheelbase, twodesired distances between the ultrasonic transmitter andreceivers are denoted as 𝑋3 and 𝑋4 serve as the input of
the cascade PID. To control the distance between the user
and the shopping cart, the distance (𝑋1 + 𝑋2 )/2 isconsidered as the feedback and 0.5 m is set as the desired
input for the Position PD control. To ensure the front of
the cart always faces the user, (𝑋1 − 𝑋2 ) is considered as
the feedback and zero difference as the desired input of
the Angle Loop PD control. With position control andangle control, 𝑉𝑥 , 𝑉𝑦 and 𝜔 can be calculated. By utilizing
the calculated 𝑉𝑥 , 𝑉𝑦 and 𝜔 as the input for the Velocity
Loop PID, the final PWM signal is conveyed to the motor
as shown in Figure 10.

𝑉3 = |𝑉𝑋 | + |𝑉𝑌 | + |𝜔 × (𝑎 + 𝑏)|
𝑉4 = |𝑉𝑋 | − |𝑉𝑌 | + |𝜔 × (𝑎 + 𝑏)|
3.5. Ultrasonic Ranging
To measure the distance between the user and the
shopping cart, two ultrasonic sensors were positioned at
the front end of the shopping cart as illustrated in Figure
8. These sensors were positioned roughly parallel to theuser's waist.

Ultrasonic sensor

Photoelectric

Figure 10: Block diagram of PID control.
Figure 8: Overall view of the shopping cart.
3.7. Obstacle Avoidance
By taking an ultrasonic transmitter in users’ hands, thereceivers can receive the ultrasonic waves and calculate
the distance between the transmitters and receivers.
These distances are denoted as 𝑋1 and 𝑋2 respectively [7].
The concept of the transmitters and the receivers is
illustrated in Figure 9.

Two E18-D80NK photoelectric switches are placed on
the front of the cart as illustrated in Figure 8. Upon any
obstacle is detected, the cart will stop following the user
and overtake the obstacle. As soon as the cart overtakes
the obstacle and no other obstacles exist, the cart willcontinue the job.
3.8. Server Communication
Advancements in the Internet of Things (IoT) haveenabled innovations in smart home and industrial
automation, offering possibilities for remote monitoring
and controlling devices. These solutions contribute toenergy efficiency and cost saving, as appliances aremonitored and controlled by small, resource-constrained
embedded devices [8].
To establish communication with the server, an esp8266 WiFi module is connected to the microcontroller
board. By using the Message Queuing Telemetry
Transport (MQTT) protocol, the controller is able topublish topics containing commodity information to theserver and subscribe to topics conveying user input fromthe server.

Figure 9: The concept of the transmitters and thereceivers
3.6. Cascade PID

In this system, the server initiates requests to the frontend controller through HTTP and utilizes the MQTT
protocol for communication with the embedded device.
The back-end is developed in Java, leveraging theSpringBoot framework to simplify the complexity of the

Cascade control is one of the most successful methods
for enhancing single-loop control performance,
particularly when the disturbances are associated with the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
826

A Comprehensive Approach todevelopment, and incorporates MySQL for databaseconnectivity and data persistence. In practical scenarios,
the server needs to establish a long connection with alarge number of embedded devices. The traditional
blocking I/O involves threading, and the frequentswitching between threads can lead to significant
overhead. Therefore, domestic smart sockets are
employed as the underlying I/O container to address thesechallenges.

resume following the user as needed, achieving the effectof remote communication.
There is also a shopping cart commodity settlement
interface where users can view the product information in
the shopping cart through the web page. Additionally,each time a product is added, users can monitor the total
cost and the quantity of items in the shopping cart throughthe settlement function. This process allows users toeasily track if they exceed their estimated budget,minimizing the need for users to independently calculate
commodity prices. Users can also modify the quantity ofgoods on the webpage or remove items from the shoppingcart, contributing to a smoother shopping experience as
illustrated in Figure 12. In short, this system enhances theuser's shopping experience by streamlining the checkout
process and providing greater control over the shopping
cart contents.

A prototype incorporating these features has beensuccessfully implemented and tested within a simple
home automation network to validate its functionalities.
The results demonstrate that the system responds rapidly,
preventing overconsumption and electrocution hazards.
This capability positions the technology to contribute tocreating safer and smarter homes in the next generation[9].
The multiplexing of threads is achieved using threadpool and asynchronous callback technologies. Thisapproach effectively reduces the overhead associatedwith thread creation and destruction, resulting in a
substantial improvement in the system's load capacity.
3.9. QR Code Scanner
QR codes are a cost-effective technique because they
are easily created and printed on a surface for distribution,
often being incorporated into existing print materials.
This is why the QR code was selected as a fundamental
component in the design of this shopping cart [10]. The
packaging of the goods is equipped with a corresponding
QR code. To identify commodity information, a QR code
scanner is employed, and the gathered data is transmitted
to the microcontroller. The QR scanner is attached to theback of the shopping basket as shown in Figure 11.

Figure 12: Example of web interface.
4. Results and Discussions
To assess the stability and accuracy of the shopping cartin actual operation, conducting relevant tests becomes
imperative.
4.1. Maximum Start DistanceTo evaluate the maximum distance capability of the
shopping cart's following function, a test involves
initiating the following function in front of the shopping
cart while it is in a static state. This method allows theobservation of the shopping cart's following stability
under varying distances between individuals and the cart.

Figure 11: QR code scanner
3.10. Interactive Webpage
Users can communicate with the shopping cart anddecide whether to enable the shopping cart to follow.
Alternatively, even when the shopping cart is not set tofollow during the shopping process, it can still be utilized
for remote communication, allowing the shopping cart toUnder unchanged conditions, the start distance between
the user and the shopping cart was set to 10 cm, 50 cm,
100 cm, and 200 cm with the desired follow distancemaintained at 40 cm. Figure 13, Figure 14, Figure 15 andFigure 16 show the results of the distance between the
cart and the user against time. In the legend, ‘Left’©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
827

Yao Chen, Jiacheng Du, Bo Peng, Ningfei Wang, Zehan Huang, Wei Hong Lim, Sew Sun Tiang, Mastaneh Mokayef, Chin Hong Wong

From the test results, when the distance between theuser and the shopping cart is between 10 cm to 200 cm,
the shopping cart can quickly adjust the distance to theuser after the start of the follow function. When the
distance between the user and the shopping cart exceeds
200 cm, the ultrasonic module experiences unstable data
reception. This instability prevents the shopping cartfrom promptly adjusting itself after activating the follow
function, resulting in a failure to follow accurately.
Therefore, the effective start distance for the shopping
cart is set between 20 cm and 200 cm.

represents the data from the left ultrasonic receiver, and‘Right’ indicates the data from the right ultrasonicreceiver.

4.2. Turning Test
To test the follow function of the shopping cart during
the user's turning process, two circular experiments wereconducted. The tester walked along circles with radii of
1m and 2m at normal speeds, while the shopping cartfollowed. Figure 17 and Figure 18 show the turning test
results.

Figure 13: 10 cm starting distance between the user andthe shopping cart against time.

Figure 14: 50 cm starting distance between the user andthe shopping cart against time.
Figure 17: Distance between the user and the shopping
cart against time for the tester walking along a circle
with a radius of 1 m.

Figure 15: 100 cm starting distance between the user
and the shopping cart against time.

Figure 18: Distance between the user and the shopping
cart against time for the tester walking along a circle
with a radius of 2 m.
According to the test results, it is observed that when
the turning radius is greater than 1 m, the shopping cart
promptly adjusts its distance to the user.
4.3. Walking Test

Figure 16: 200 cm starting distance between the user
and the shopping cart against time.

To evaluate the follow function in a shopping scenario,
two circular experiments were conducted. The tester©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
828

A Comprehensive Approach to

3.

walked at speeds of 1.5 m/s and 3 m/s, representing
normal walking and jogging speeds, while the shopping
cart followed. The test results are illustrated in Figure 19
and Figure 20.

4.

5.

6.

7.

Figure 19: Distance between the user and the shopping
cart against time with 3 m/s walking speed.

8.

9.

10.

Figure 20: Distance between the user and the shopping
cart against time with 1.5 m/s walking speed.
Based on the test results, when walking at a normal
speed and activating the follow function, the shopping
cart can successfully follow the user throughout the
shopping process.

H. Sadia, S. Jee, K. Pal, S. Singh, and M. Marbaniang,“Iot Application Based Advanced Shopping Trolley,” Int
J Eng Adv Technol, 2019.
A. A. S. Gunawan et al., “Development of smart trolleysystem based on android smartphone sensors,” inProcedia Computer Science, Elsevier B.V., 2019, pp.
629–637. doi: 10.1016/j.procs.2019.08.225.
W. W. Tai, B. Ilias, S. A. Abdul Shukor, N. Abdul Rahim,
and M. A. Markom, “A Study of Ultrasonic Sensor
Capability in Human Following Robot System,” in IOP
Conference Series: Materials Science and Engineering,
IOP Publishing Ltd, Dec. 2019. doi: 10.1088/1757899X/705/1/012045.
H. Taheri, B. Qiao, and N. Ghaeminezhad, “Kinematic
Model of a Four Mecanum Wheeled Mobile Robot,” Int J
Comput Appl, vol. 113, no. 3, pp. 6–9, Mar. 2015, doi:
10.5120/19804-1586.
A. Carullo and M. Parvis, “An Ultrasonic Sensor forDistance Measurement in Automotive Applications,”
2001.
P. MacHeso, T. D. Manda, S. Chisale, N. Dzupire, J.
Mlatho, and D. Mukanyiligira, “Design of ESP8266
Smart Home Using MQTT and Node-RED,” inProceedings - International Conference on Artificial
Intelligence and Smart Systems, ICAIS 2021, Institute ofElectrical and Electronics Engineers Inc., Mar. 2021, pp.
502–505. doi: 10.1109/ICAIS50930.2021.9396027.
A. Singaravelan and M. Kowsalya, “Design and
Implementation of Standby Power Saving Smart Socketwith Wireless Sensor Network,” in Procedia Computer
Science, Elsevier B.V., 2016, pp. 305–310. doi:
10.1016/j.procs.2016.07.360.
S. Statler, “Barcodes, QR Codes, NFC, and RFID,” inBeacon Technologies, Apress, 2016, pp. 317–331. doi:
10.1007/978-1-4842-1889-1_18.

Authors Introduction
Mr. Yao Chen
He is currently pursuing Bachelor of
Robotics and Intelligent Devices as
3rd year student in the Department ofMaynooth International Engineering
College, Fuzhou University, China.
His research interests are embedded
systems and machine learning.

5. Conclusion
This paper highlights the accomplishments indeveloping a smart shopping cart system with automatic
following, obstacle avoidance, and web interfacegenerated shopping lists. The acknowledgement ofpotential improvements, such as performance
optimization and enhanced user interfaces, adds a
realistic and forward-looking perspective. The mentionof future research directions, including adaptability todifferent shopping scenarios, demonstrates an awarenessof potential advancements.

Mr. Jiacheng Du
He is currently pursuing Bachelor of
Robotics and Intelligent Devices as 3rdyear student in the Department ofMaynooth International Engineering
College, Fuzhou University, China.
His research interests are embedded
systems and reinforcement learning.

References
1.

2.

M. Batty et al., “Smart cities of the future,” European
Physical Journal: Special Topics, vol. 214, no. 1, pp.
481–518, Dec. 2012, doi: 10.1140/epjst/e2012-01703-3.
Viswanadha V, Pavan Kumar P, and Chiranjeevi Reddy
S, “Smart Shopping Cart,” 2018.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
829

Powered by TCPDF (www.tcpdf.org)

Yao Chen, Jiacheng Du, Bo Peng, Ningfei Wang, Zehan Huang, Wei Hong Lim, Sew Sun Tiang, Mastaneh Mokayef, Chin Hong Wong

Mr. Bo Peng

Dr. Sew Sun Tiang
She is an Assistant Professor in Facultyof Engineering at UCSI University in
Malaysia. She received her PhD in
Electrical and Electronic Engineeringfrom Universiti Sains Malaysia in
2014. Her research interests are
optimization and antenna design.

He is currently pursuing Bachelor of
Robotics and Intelligent Devices as
3rd year student in the Department ofMaynooth International Engineering
College, Fuzhou University, China.
His research interests are data scienceand software development.

Ms. Ningfei Wang

Dr.Mastaneh Mokayef
She is an Assistant Professor in the
Faculty of Engineering at UCSI
University in Malaysia. She received
her
PhD
from
the
Wireless
Communication Centre Faculty ofElectrical Engineering at University
Technology Malaysia in 2014. Her
research interests include Wirelesscommunications, spectrum sharing
method, spectrum management, cellular communication
systems and Antenna design.

She is currently pursuing Bachelor of
Robotics and Intelligent Devices as
3rd year student in the Department ofMaynooth International Engineering
College, Fuzhou University, China.
Her research interests are data science,
data mining and data protection.
Mr. Zehan Huang
He is currently pursuing Bachelor of
Robotics and Intelligent Devices as
3rd year student in the Department ofMaynooth International Engineering
College, Fuzhou University, China.
His research interests are electronic
information and machine learning.

Dr. Chin Hong Wong
He is a Lecturer at MaynoothInternational Engineering College at
Fuzhou University, China. He
received his PhD in Electrical andElectronic
Engineering
fromUniversiti Sains Malaysia in 2017. His
research
interests
areEnergyharvesting, signals and systems, andmachine learning.

Dr. Wei Hong Lim
He is an Associate Professor in
Faculty of Engineering at UCSI
University in Malaysia. He received
his
PhD
in
Computational
Intelligence from Universiti Sains
Malaysia in 2014. His researchinterests are optimization andartificial intelligence.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
830

A Design of Four-Port Flexible UWB-MIMO Antenna for Wearable and IoT Applications
Jia Wei Tan1, Sew Sun Tiang1, *, Kim Soon Choong1, Mohammad Arif Bin Ilyas1, Mastaneh Mokayef1, Zhi Ying Yeoh1, Wei
Kang Lai1, Wei Hong Lim1

E-mail: 1001954282@ucsiuniversity.edu.my, tiangss@ucsiuniversity.edu.my, ChongKS@ucsiuniversity.edu.my,mohammadarif@ucsiuniversity.edu.my, mastaneh@ucsiuniversity.edu.my, 1001953809@ucsiuniversity.edu.my,100953677@ucsiuniversity.edu.my, limwh@ucsiuniversity.edu.my(1UCSI University, Malaysia)* Corresponding author
Abstract
This paper introduces a compact four-port MIMO antenna for ultrawideband (UWB) applications measuring 60 × 60
mm2 . Printed on a single layer flame resistant (FR-4) substrate (permittivity of 4.3, thickness 1.6mm), the antennafeatures four microstrip cells, each orthogonal to its neighbor for improved isolation. It includes a rectangular patchwith staircase slits and a stepped feed line. Square stubs at the top center enhance isolation. The antenna boasts a
significant return loss (-43.75dB), wide impedance bandwidth (1.967-12GHz), and an isolation below -15dB. Its
envelope correction coefficient (ECC) is under 0.02 with a moderate 4.4dBi gain. Although its 63% radiation
efficiency could be enhanced, the antenna’s ultrawide bandwidth and compactness make it suitable for UWB wearable
IOT applications.
Keywords: UWB antenna, MIMO antenna, MIMO performance.

1.

ensure that it does not harm human tissues since it is inclose contact with human skin [2].
However, the usage of more than one radiating
element, closely arranged to one another in MIMO maylead to high signal correlation and reduce the overallantenna efficiency. Therefore, appropriate decouplingneeds to be done between antenna elements for better
isolation in modern communication. The challenging partis to design UWB-MIMO with low isolation by keeping
minimum distance between antennas. In [1], a UWBMIMO antenna with square shaped decoupler andrectangular metallic stubs was designed to reduce mutual
coupling and improve isolation. The antenna offers stable
gain with peak gain at 6.8dBi, low ECC value of 0.001
and high diversity gain of 9.9. Besides, flexible substrates
such as polymer, paper, and Kapton are important in thedesigning of wearable UWB. In [3], a wristband MIMO
antenna is designed that covers frequency range of 2.7512GHz with silicone rubber as substrate material that isused to monitor children and patients, ensuring user safety.
In [4], the author proposed the use of liquid crystal
polymer (LCP) that operates between 2.9-10.86GHz
which results in a flexible MIMO antenna that performs
well in bending and on-body conditions, suitable inwearable fields. Another result obtained from [5], uses
bendable substrate FR-4 that operates up to 17GHz from
3.89GHz with stable gain.

Introduction

The frequency of ultra-wideband (UWB) technology
was set to 3.2-10.6GHz by the Federal Communication
Commission (FCC) which garnered great attention ofresearchers due to its promising solution to scarcity ofspectrum resources. Compared to traditional narrowband
technology, UWB offers more advantages and several
capabilities including short range, high transmission, highbandwidth, low energy usage, low complexity, and goodradiation. With this, UWB positions itself to being useful
in applications such as short-range communications,sensor networks, tracking, and positioning systems [1],[2].
In a multi-path environment, UWB systems are prone
to signal fading which degrades the performance of theentire system. To address this issue, Multi-Input-MultiOutput (MIMO) technology has been used to provide a
multi-element patch antenna for signal transmission andreception. This implementation significantly increases
spectral efficiency, reliability, and channel capacity
without utilizing additional power and spectrum. The
enhanced features of MIMO UWB antennas such as highdata rate and exclusion from interference make it a perfectcandidate for wearable and IoT applications especially inhealthcare industries. Radiation exposure from theantenna is heavily monitored to be as small as possible to©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
831

Jia Wei Tan, Sew Sun Tiang, Kim Soon Choong, Mohammad Arif bin Ilyas, Mastaneh Mokayef, Zhi Ying Yeoh, Wei Kang Lai, Wei Hong Lim

Moreover, the design of UWB is very challenging toachieve optimized frequency and time, stable polarization,
as well as low dispersion. In [6], the author proposed amethodology that introduces slotted ground structure
which is a widely used technique to improve theimpedance bandwidth and minimize the ground plane
effects in UWB. In [7], 4 ports rectangular monopole
antennas with step etching on ground plane and arrowshaped slot on radiating patch using FR-4 substrate
operates at frequency 3.1-10.6GHz, has high isolation of-17.4dB, ECC less than 0.001 which in turn decreases
mutual coupling. In the medical field, wearable devices
are achieving profound recognition when designed using
MIMO antenna due to their quality signal and highcapacity without increasing transmitted power.

This paper proposes the design of four port UWBMIMO antenna suitable for wearable and IoT applications
which is done by utilizing the CST software that involves
careful selection of flexible substrate properties, structureof microstrip patch, and feed network. The design uses
copper conductor for patch and FR-4 as substrate material
that covers wider bandwidth and higher efficiency
characteristic with low dielectric loss and high thermal
endurance. The width of the microstrip and gap between
the patches are adjusted to obtained impedance matchingof 50 Ohm. The finalized antenna is measured at different
parameters such as reflection coefficients, gain, radiation
pattern, current distribution, ECC, DG, radiation and total
efficiency for further analysis and improvements. Table1
shows summary of comparison between existing UWBMIMO antennas and proposed work.

Table 1. Summary Of Comparison Between Existing UWB-MIMO Antennas and Proposed Work
References

Operating
Frequency
(GHz)

Peak
Gain
(dBi)

Dimension(mm𝟑 )

Diversity
Gain
(DG)

Isolation(dB)

Substrate Type

24 × 30 × 0.13
65× 65 × 0.1

Envelope
Coefficient
Correlation(ECC)< 0.18< 0.01[3][4]

2.75-12.0
2.90-10.86

3.41
4

> 9.5
> 9.99

> 25
> 22

5.87
3.09
3.38
7
2.4

75 × 91 × 0.26
30 ×30 × 1.6
80 × 80 × 0.781
58× 58 × 0.8
26× 26 × 0.8< 0.02< 0.001< 0.07< 0.02

> 9.9

> 22
-17.4
> 18
~ 25

Silicon Rubber
Liquid Crystal
Polymer (LCP)
FR-4
FR-4
FR-4
FR4
FR4

[5]
[6]
[7]
[8]
[9]

3.89-17.09
2.9-12.4
2.6-11
3.6-16.00
1.5-3.6

[10]
[11]
[12]
[13]
Proposedwork

3.3-13.6
3.10-12
3.1-13.1
3.5-11
1.926-12

5.7
6.2
4
6
4.409

42 × 32.5 × 1.0
30× 30 × 1.6
45 × 45 × 1.6
24 × 30 × 0.13
60 × 30 × 1.187< 0.02< 0.001< 0.02< 0.03<0.02> 9.96> 9.9> 9.9985
> 9.9
>9.93

> 18
~ 17
< 17
< 20
>15

Textile
FR-4
FR-4
FR-4
FR-4

between. The design begins with a rectangular radiating
patch that uses a 50Ohm transmission line feeding
technique to feed signal to the resonating patch. The
ground plane size is decreased for low profile andoptimization for attaining enhanced impedance bandwidth.
Two symmetrical staircase-shaped slits are introduced on
both sides of the radiating patch to further enhance theimpedance bandwidth.

2. Material and Methods
2.1 Single Element Antenna Design
The configuration of the proposed UWB single element
microstrip antenna shown in Fig. 1. The fundamental
structure of the antenna consists of three layers which aremetal layer on top and bottom with substrate material, in©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
832

A Design of Four-Port

a. Front view of antenna design

b. Magnified area for staircase-like slits

Fig. 1 Geometry of proposed antenna design.
Table 2. Optimized Parameters for the Proposed FourPort MIMO Antenna.
Parameters
Values(mm)
Parameters
Values
(mm)

W1
31

W2
13

W3
3

W4
1.65

W5
2

L1
42

L2
15.6

L3
15.3

L4
0.1

L5
13
Fig. 2 Simulation results of return loss (S11) for singleslot antenna

Table 2 shows the optimized parameters for the proposedfour port MIMO antenna. The material chosen for theantenna patch is copper (annealed) because it is muchsofter and malleable which makes it easier to shapeduring the fabrication stage. On the other hand, theflexible dielectric substance used is flame-resistant (FR4)with dielectric constant of 4.3 and tangent loss of 0.019.
Some key features of this substrate material include low
cost, easily available and relative mechanical andelectrical stability. It is mostly found in microstrip andstripline circuits, as well as point-to-point digital radio
antennas. The thickness of the UWB microstrip antennais 1.8mm with copper (annealed) 0.1mm and substrate as
1.6mm respectively. Overall, the FR4 substrate based
UWB antenna, and its resonating patch has sizes of
31 × 42 × 1.7 mm3 and fed by a 3 ×17 ×0.1 mm3transmission line.
The operating range of antenna generated is 2.7410.59GHz shown in Fig. 2 which fulfills the requirementof UWB but is lesser compared to the values from thearticle [14] which obtain 2.9-11GHz.

2.2 Four-Element Flexible UWB Antenna
From the single element in Fig. 1, a four-port flexible
MIMO antenna array was designed as shown in Fig. 3 (b)with a square-shaped stub inserted between orthogonal
element antennas for better isolation performance.

a. Without isolator

b. With isolator branch

Fig 3. Four Port UWB MIMO antenna©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
833

Jia Wei Tan, Sew Sun Tiang, Kim Soon Choong, Mohammad Arif bin Ilyas, Mastaneh Mokayef, Zhi Ying Yeoh, Wei Kang Lai, Wei Hong Lim

3. Result and Discussion
The layouts were designed and simulated using CST
software to determine the antenna performance indifferent aspects.
The S11 parameter graph is a measure of how welldevices or lines are matched. A high return loss isdesirable for lower insertion loss. The simulated result
for return loss from the proposed UWB four-port antenna
is displayed in Fig. 4 (a). The first antenna result without
branch isolator (green line) shows much narrow
impedance bandwidth with frequency range of 3.92512GHz for S11 < -10dB. Besides, the return loss value is
acceptable, which is -22.49dB at resonant frequency of
4.42GHz but can be improved to reduce power reflectedto the system. This leads to the addition of a branchisolator (red line) to improve both return loss andbandwidth values. The S11 value almost increased
twofold with value highest return loss value of -43.94dB
at resonant frequency of 8.124GHz, larger bandwidth,
and wider frequency range of 1.926-12GHz. The rest of
the resonant frequencies for S11 below -10dB arerecorded at 2.292GHz, 4.14GHz, 6.6GHz and11.076GHz.
The remaining parameters are also compared such as
S12, S13 and S14 to determine the isolation as shown in
Fig 4 (b), (c), (d). The measured operating frequency
covers 2.0-12GHz. The implementation of branchisolators affects S12, S13, and S14 whereby most of them
are below -15dB (red line) and this shows good isolation
properties. With an almost similar isolation design, theresult from A. A. Ibrahim, et al [13] shows better
isolation which is greater than -20dB when the isolation
ports are added.

(c) S13(d) S14
Fig 4. Results of return loss and isolation loss
The gain and directivity for the final proposed antenna on
different resonant frequencies is tabulated in Table 3
while the far field pattern for resonant frequency,
8.124GHz is shown in Fig. 5 (a) and (b).
Table 3. Gain and Directivity Values for Different
Resonant Frequencies
Resonant
Gain
Directivity
Frequency (GHz)
(dBi)
(dBi)
2.292
4.14
4.407
6.36
6.6
8.124
11.076(a) S11(a) Gain at resonant frequency of 8.12GHz.
(b) S12©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
834

A Design of Four-Port(b) Directivity at resonant frequency of 8.124GHz.

(c) Radiation pattern at 6.6GHz.

Fig 5. Gain and directivity at resonant frequency of
8.124GHz.
Fig. 6 (a), Fig. 6(b), Fig. 6(c), Fig. 6(d), and Fig. 6(e)shows the radiation pattern of the proposed four-port
flexible antenna on H-plane and E-plane at different
resonant frequencies. The simulation and test results
were obtained when port 1 was used as excitation. The
H-field and E-field exhibit a directional pattern whereby
the energy is concentrated and focused on specificdirections at degrees of 44° and 178° for all resonant
frequencies. It allows far-distance communication due tohigher gain and directivity.

(d) Radiation pattern at 8.124GHz.

(e) Radiation pattern at 11.076GHz.
(a) Radiation pattern at 2.292GHz.

Fig 6. Radiation pattern on H-plane and E-plane of four
port MIMO UWB antenna at different resonant
frequencies
The surface current distribution is measured at port 1, 2,
3 and 4 with resonant frequencies at 2.292GHz, 4.14GHz,
6.6GHz, 8.124GHz and 11.076GHz and tabulated inTable 4. Fig. 7 shows the pattern of the selected resonant
frequency, 8.124GHz, where the rest have similar current
diagrams. The current density is most evident at theantenna feedline denoted by warm colors of red andorange color map which contributes to stronger radiation
and better performance but can result in higher potential
for losses and prone to heating easily due to highdissipation of electromagnetic fields and energy. The restof the areas that are mostly displayed in blue or green
colors experience low current density whereby it has(b) Radiation pattern at 4.414GHz.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
835

Jia Wei Tan, Sew Sun Tiang, Kim Soon Choong, Mohammad Arif bin Ilyas, Mastaneh Mokayef, Zhi Ying Yeoh, Wei Kang Lai, Wei Hong Lim

weaker radiation and signal strength but less susceptible
to losses.

The diversity performance of the UWB-MIMO antenna
was evaluated by Envelope Coefficient Correlation(ECC) and diversity gain (DG). ECC parameters reflect
the degree of correlation between adjacent elements of
MIMO antennas and are expected to be less than 0.5. The
ECC parameters between the antenna’s ports are shownin Fig. 9 (a) which are less than 0.02 in the operating
frequency band and indicate good isolation between ports.

Table 4. Surface Current Value at Port 1 to 4 at
Different Resonant Frequencies.
Resonant
Port 1
Port 2
Port 3
Port 4Frequency (A/m)
(A/m)
(A/m)
(A/m)
(GHz)
2.292
37.0092 37.9358 38.3408 37.0869
4.14
6.6
8.124
29.5007 33.4613 30.3317 34.7738
11.07637.0092 37.9358 38.3408 37.0869

DG evaluates the quantified improvement in signalto-noise ratio when antenna in the MIMO system receive
the RF signal. The DG parameters between different
ports are given in Fig. 9 (b) It shows that satisfying DGvalues are achieved with values above 9.938 for thewhole operating frequency band (1.926-12GHz) which
gives better diversity characteristics.

Fig. 7 Surface current distribution of four-port UWBMIMO antenna at resonant frequency, 8.124GHz.

(a) ECC

The average radiation and total efficiency of the antennais only recorded between 6GHz-12GHz due to mesh
limitation in frequency domain. Fig. 8 shows the value indB (magnitude) which is approximately 55% and 54%
after converted to percentage form. These values are
considered marginally acceptable because good antenna
design can reach up to 80%. Hence, the design can still
be further improved by careful calculation of geometry
and dimensions, matching networks to achieve maximum
performance whereby it provides better signal strengthand reduces interference which are important factors for
improving the reliability and operability of wearable
devices in IoT applications.

(b) DGFig 9. ECC and DG values for four-port UWB MIMO
antenna.
Conclusion
This paper proposed a small, compact, and flexible fourport UWB MIMO antenna designed for wearable and IoTapplications with overall dimensions of 60 × 60 × 1.8 mm3.
The proposed antenna was arranged orthogonally to eachother to form a four-port MIMO antenna structure and thefinal design shows that the antenna can operate in thewhole UWB band covering from 1.926 to 12GHz withmoderate isolation of -15dB. The isolation was achievedby adding a square-shaped stub at the top center withelongated strips extending to the edge of the substrate. It

Fig. 8 Radiation efficiency and total efficiency of fourport UWB MIMO antenna.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
836

A Design of Four-Portfor UWB communication,” Heliyon, vol. 9, no. 1, Jan.
2023.
12. A. Wu, M. Zhao, P. Zhang, and Z. Zhang, “A Compact
Four-Port MIMO Antenna for UWB Applications,”
Sensors, vol. 22, no. 15, Aug. 2022.
13. A. A. Ibrahim, M. I. Ahmed, and M. F. Ahmed, “Asystematic investigation of four ports MIMO antenna
depending on flexible material for UWB networks,” Sci
Rep, vol. 12, no. 1, Dec. 2022.
14. M. Mustaqim, B. A. Khawaja, H. T. Chattha, K. Shafique,M. J. Zafar, and M. Jamil, “Ultra-wideband antenna forwearable Internet of Things devices and wireless bodyarea network applications,” International Journal of
Numerical Modelling: Electronic Networks, Devices andFields, vol 32, no.6 Nov. 2019.

has a high return loss value of -43.65dB, total efficiency
of approximately 60%, gain of 4.4dBi, and good diversity
performance, with ECC less than 0.02 and DG larger than
9.93dB. In summary, the proposed antenna can beconsidered as a viable candidate for UWB-MIMO
wireless applications.
Acknowledgements
This work was supported by the UCSI University
Research Excellence & Innovation Grant (REIG) withproject code of REIG-FETBE-2022/038.
References
1.

Abbas, N. Hussain, M. A. Sufian, J. Jung, S. M. Park, andN. Kim, “Isolation and gain improvement of a rectangularnotch uwb-mimo antenna,” Sensors, vol. 22, no. 4, Feb.
2022.
2. P. Kumar, P. Kumar, T. Ali, P. Kumar, and S. Vincent,“Ultrawideband Antennas: Growth and Evolution,”
Micromachines, vol. 13, no. 1. MDPI, Jan. 01, 2022.
3. T. Govindan et al., “On the design and performance
analysis of wristband MIMO/diversity antenna for smart
wearable communication applications,” Sci Rep, vol. 11,
no. 1, Dec. 2021.
4. J. Zhang, C. Du, and R. Wang, “Design of a Four-Port
Flexible UWB-MIMO Antenna with High Isolation for
Wearable and IoT Applications,” Micromachines (Basel),vol. 13, no. 12, Dec. 2022.
5. Desai, J. Kulkarni, M. M. Kamruzzaman, S. Hubalovsky,
H. T. Hsu, and A. A. Ibrahim, “Interconnected CPW Fed
Flexible 4-Port MIMO Antenna for UWB, X, and Ku Band
Applications,” IEEE Access, vol. 10, pp. 57641–57654,
2022.
6. T. Ali and R. C. Biradar, “A Miniaturized Volkswagen
Logo UWB Antenna with Slotted Ground Structure andMetamaterial for GPS, WiMAX and WLAN Applications,”
2017.
7. W. Naktong and A. Ruengwaree, “Four-Port Rectangular
Monopole Antenna for UWB-MIMO Applications,” 2020.
8. P. Kumar, S. Urooj, and F. Alrowais, “Design andimplementation of quad-port MIMO antenna with dualband elimination characteristics for ultra-wideband
applications,” Applied Sciences (Switzerland), vol. 10, no.
5, Mar. 2020.
9. I. Elfergani et al., “Low-profile and closely spaced fourelement mimo antenna for wireless body area networks,”
Electronics (Switzerland), vol. 9, no. 2, Feb. 2020.
10. S. Kumar et al., “Wideband Circularly Polarized Textile
MIMO Antenna for Wearable Applications,” IEEE Access,vol. 9, pp. 108601–108613, 2021.
11. A. H. Jabire, S. Sani, S. Saminu, M. J. Adamu, and M. I.
Hussein, “A crossed-polarized four port MIMO antenna

Authors Introduction
Ms. Jia Wei Tan
She is currently pursuing
BachelorofEngineering
Communication and Electronics
with Honors as final year student
in Faculty of Engineering,Technology
and
Built
Environment, UCSI University,
Malaysia. Her research interests
are communication and antenna design.
Dr. Sew Sun Tiang
She is an Assistant Professor in
Faculty of Engineering at UCSI
University in Malaysia. She
received her PhD in Electrical andElectronic Engineering from
Universiti Sains Malaysia in 2014.
Her research interests are
optimization and antenna design.
Dr. Kim Soon Choong
He obtained his B.Eng., M.S., andPh.D. degrees in electrical andelectronicengineering
fromUniversiti Kebangsaan Malaysia
in 2012, 2015, and 2019,respectively. He currently serves
as an Assistant Professor in the
Department of Computer Engineering (Artificial
Intelligence) within the Faculty of Engineering
Technology and Built Environment at UCSI Kuala
Lumpur. His research focuses on areas such as
medical signal processing and instrumentation.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
837

Powered by TCPDF (www.tcpdf.org)

Jia Wei Tan, Sew Sun Tiang, Kim Soon Choong, Mohammad Arif bin Ilyas, Mastaneh Mokayef, Zhi Ying Yeoh, Wei Kang Lai, Wei Hong Lim

Dr. Mohammad Arif Bin Ilyas
He (M’32) was born in Malaysia in 1991.
He received his PhD in ElectricalEngineering from Universiti Tun
Hussein Onn Malaysia in 2020. He also
earned his bachelor’s degree inelectronic engineering at the same
university. Currently, he is a lecturer in
the Faculty of Engineering, Technology,
and Built Environment at UCSI University. His main areasof research interest are machine learning, artificial
intelligence, the Internet of Things, embedded systems, andoptical wireless communication, especially visible light
communication.

Dr. Wei Hong Lim
He is an Assistant Professor in Facultyof Engineering at UCSI University in
Malaysia. He received his PhD in
Computational
Intelligence
fromUniversiti Sains Malaysia in 2014. His
research interests are optimization andartificial intelligence.

Dr. Mastaneh Mokayef
She has received her PhD from WirelessCommunication Centre Faculty ofElectrical Engineering in University
Technology Malaysia (UTM) in 2014.
She has also obtained her master’s
degree from the faculty of engineeringin 2009 from the University Technology
Malaysia. Her research interests include
Wireless communications, spectrum sharing method,
spectrum management, cellular communication systems andAntenna design.
Ms. Zhi Ying Yeoh
She is currently pursuing Bachelor ofEngineering Electrical and Electronics
with Honours as final year student inFaculty of Engineering, Technology and
Built Environment, UCSI University,
Malaysia. Her research interests are
power electronics and simulation.

Mr. Wei Kang Lai
He is currently pursuing Bachelor of
Electrical and Electronics Engineering as
final year student in Faculty ofEngineering, Technology and Build
Environment,
UCSI
University,
Malaysia. His research interests areelectrical and solar system design.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
838

Investigate Power Efficiency in PLECS and MATLAB Software by Designing USB 5Watt Charger
Zhi Ying Yeoh1, Kim Soon Chong1, Sew Sun Tiang1, Mohammad Arif Ilyas1*, Jia Wei Tan1, Wei Kang Lai1, Wei Hong
Lim1

Email: 1001953809@ucsiuniversity.edu.my, ChongKS@ucsiuniversity.edu.my, tiangss@ucsiuniversity.edu.my,
MohammadArif@ucsiuniversity.edu.my, 1001954348@ucsiuniversity.edu.my,1001953677@ucsiuniversity.edu.my, limwh@ucsiuniversity.edu.my(1UCSI University, Malaysia)
Abstract

In power electronics, accurately assessing simulation tools is key for precise, reliable electronicsystem designs. This study compares MATLAB Simulink and PLECS in modeling a 5W USB
charger’s power characteristic. The charger, using an AC-DC full bridge rectifier and DC-DC
flyback topology, delivers a stable 5VDC at 1A. The analysis focuses on power efficiency andthermal characteristics, incorporating real-life components for detailed insights. Results show
PLECS, specialized in power electronics, surpasses MATLAB in accuracy and consistency. This
research aids in understanding simulation tools' effectiveness, guiding engineers and researchersin power efficiency evaluations for electronic systems.
Keywords: Power electronics conversion, MATLAB Simulink, PLECS, USB charger, simulation, power
efficiency, thermal analysis.

1. Introduction
Power electronics conversion is an essential
technology utilized in the landscape of modern
technology, permeating various aspects of daily life
applications. Power electronics converters serve as atransformative agent in the regulation and control ofelectric energy flow.

Fig. 1 Power converter topologies.

A static power converter is a converter made up
with the architecture of electrical components suchas transformers, diodes, capacitors, and inductors.
Electrical components work together transforming
electrical energy from one form to another. From Fig.
1, notable ideal power converter can be observed,
controlling the flow of power between two sources
aiming to enhance the power efficiency [1]. Power
conversion can delve into four categories which are
DC to DC converter, AC to AC converter, DC to AC
converter and AC to DC converter.

In recent years, significant improvements in powerelectronics technology. These advances include
breakthroughs in converter topologies, control
strategies, and power semiconductor devices, which
together have contributed to the continuous
improvement and widespread application of modern
power conversion systems [2].
USB chargers are one of the illustrations of daily
life, where the fundamental principles of power
conversion. USB chargers are ubiquitous forcharging electronics devices such as laptops andsmartphones. In contemporary times, USB chargers
are designed with emphasis on fast charging ofelectronics devices [3]. However, this design offaster charging rates can lead to increased energy©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
839

Zhi Ying Yeoh, Kim Soon Chong, Sew Sun Tiang, Mohammad Arif Ilyas, Jia Wei Tan, Wei Kang Lai, Wei Hong Lim

loss and heat generated, potentially diminishing theperformance of power efficiency.

The DAB converter operates by modulating thephase-shift of the switches to control the power flow
and achieve AC-DC or DC-AC conversion. It
implements both frequency and time-domain
analysis to capture the transient behavior of theconverter, offering the advantages of steady-state
frequency-domain analysis [5]. The converter can be
configured to achieve full soft-switching operation,high efficiency, minimize distortion in input current,
and provide output short-circuit protection [6]. One
notably advantage of DAB is seamless four-quadrant
operation [6], [7], [8], [9].

This paper focuses on using a USB 5-Watt charger
as an evaluation method to evaluate the performanceof MATLAB Simulink and PLECS software in
accurately modeling and predicting the power
characteristics of the charger. By conducting adetailed comparative analysis, this study aims to
elucidate the disparities in results obtained from eachsoftware, shedding light on the strengths andlimitations of MATLAB Simulink and PLECS in thecontext of power efficiency simulations.

2.2. DC- DC Flyback Converter
The outcomes of this research will not only
contribute to a better understanding of thecapabilities of these simulation tools but will also
assist engineers and researchers in making informed
decisions when assessing power efficiency invarious electronic systems.

A Flyback converter, specifically DC-DC converter
is a type of power converter that uses a transformer
to transfer energy from the input to the output.
The working principles of flyback converter isstoring energy in the transformer during on timeswitching transistor, and then releasing it to the
output during off time of switching transistor, as
outlined in Fig. 3. This allows for voltage conversion
and isolation between the input and output. The
flyback converter can be used in various applications,such as AC-DC power supplies, power factor
correction circuits, and active-clamp flyback
converters [10].

2. Related WorkIn the pursuit of advancing understanding of powerelectronics and simulation tools, a comprehensive
review of related work lays the groundwork for our
exploration. This section begins by exploring the
AC-DC bridge converters, and DC-DC Flyback
converter, essential components in designing the 5Watt USB charger. By continuously adjusting the
Proportional-Integral value to fine-tune and achieve
the desired output voltage in the 5-Watt USB charger.
Understanding converters is crucial for operatingand evaluating their role in enhancing power
efficiency, and thermal analysis.
2.1. AC-DC Bridge Converter

Fig. 3 The Operation of Two States of Flyback
Converter.

The AC-DC bridge converter, also known as thedual active bridge (DAB) converter, is a powerelectronic device used for converting alternating
current (AC) to direct current (DC) and vice versa
and the process known as rectification. The DAB
converter, as outlined in Fig. 2 utilizes a bidirectional
power flow and high-frequency isolation with abridge topology featuring two sets of switches and atransformer [4].

2.3. Proportional-Integral
A Proportional-Integral (PI) controller is a type of
feedback control system commonly used in powerelectronics to regulate the output of the power
converters. It combines proportional (P) and integral(I) control actions to achieve the desired output.
The P value generates an output proportional to thecurrent error, while the I value aids eliminating any
residual steady-state error by integrating the error
over time. In this project, the PI controller isimplemented in the DC-DC Flyback converter inregulation to achieve desired output of USB charger.

Fig. 2 DAB Converter Topology.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
840

Investigate Power Efficiency in

2.4. Thermal Analysis
Thermal analysis is an important aspect of power
USB charger configuration, due to it bringing effecton the reliability and safety of the system. The proper
working of the charger's thermal considerations is abasic part of power electronics design. Several
studies have been conducted on the thermal designof wireless chargers for electric vehicles [11],miniaturization and thermal design of battery
chargers [12], and thermal analysis of transformers
in on-board chargers for electric vehicles [13]. Thesestudies use simulation tools to predict losses andtemperature and establish thermal networks ofheating components to calculate hotspot
temperatures.
2.5. Power Efficiency
Power efficiency in USB chargers can be describedas the charger's ability to convert electrical powerfrom the source to the device, allowing it to be
charged with minimal loss. It is a critical factor in
determining the charging time and energy
conservation.

Fig. 4 Overview of 5-Watt USB Charger.
3.1. The Design Procedure of 5-Watt USB
Charger
The design and implementation of a 5-Watt USB
power charger involves the integration of an AC-DC
full bridge rectifier employing a DC-DC flyback
topology.

Achieving high conversion efficiency is essential
for optimal operation and robustness under varying
conditions, as seen in the context of mobile battery
charger ICs, where techniques like zero current
switching (ZCS) are employed [14]. Therefore,
power efficiency in USB chargers is essential for
improving device charging, energy conservation,and the development of innovative charging
technologies.

The primary objective of this charger is to deliver a
maximum output current of 1A with a consistent
output voltage of 5VDC. Achieving this goal
requires the inclusion of a 5-ohm load resistor in thecircuit.

3. Materials and Methods

To ensure the charger's adaptability for global use,
it accepts two AC input voltages (𝑽𝒊𝒏 ), as outlined inTable 1, providing corresponding output DC voltage,( 𝑽𝒐𝒖𝒕 ). The switching frequency, ( 𝒇𝒔𝒘 ) is set at
40kHz to optimize the performance of the charger.

Overview of 5-Watt USB charger described overall
input AC voltage to output DC voltage, as outlinedin Fig. 4. The process begins with an input of
141/325 VAC with a frequency of 50/60Hz. This
input voltage is directed through the AC-DC bridge
converter. In this stage, the AC input voltage
undergoes rectification, transforming it into apulsating DC voltage. Subsequently, the pulsating
DC voltage undergoes filtering and smoothing toensure a stable waveform. The smoothed DC voltage
then progresses through the DC-DC flyback
converter, where it is stepped down to the desired
level, providing isolation and regulation in thetransformation process. To fine-tune and optimize
the output results, a PI controller is applied, ensuringthat the output voltage remains regulated at 5V DC
with a current capacity of 1A.

Table 1. The Calculation of AC Voltage Input with
Frequency.
AC Voltage

Input Voltage

Frequency

100V

100 x √2 = 141.42V

50/60Hz230V

230 x √2 = 325.26V

50/60HzThe Flyback equation was employed to calculate
the winding ratio (𝑛), critical inductance (Lc), andcritical capacitance (Cc) for component selection in
the high-frequency transformer and Flyback circuit
capacitor. This calculation involves utilising known
variables such as resistance (R), duty cycle (D) andoutput ripple voltage (𝛥𝑉𝑜).

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
841

Zhi Ying Yeoh, Kim Soon Chong, Sew Sun Tiang, Mohammad Arif Ilyas, Jia Wei Tan, Wei Kang Lai, Wei Hong Lim

The proportional-integral (PI) value was tuned fora settling time within 0.1 seconds without
overshooting.

Number of turns ratio of transformer in flyback
converter to calculate the value of Lc and Cc.
𝑛=

𝑁2
𝑉𝑜𝑢𝑡 (1 − 𝐷)=𝑁1𝑉𝑖𝑛 ∗ 𝐷(1)

For diode selection, the circuit's maximum current
was observed on an oscilloscope, guiding the choiceof a diode capable of withstanding the surge forward
current. Finally, measurements were conducted toassessmetal-oxide-semiconductor
field-effecttransistor (MOSFET) temperature and power
efficiency.

Critical Inductance and critical capacitor for
Flyback circuit.
𝐿𝑐 =(1 − 𝐷)2 ∗ 𝑅 𝑁2 2( )2 ∗ 𝑓𝑠𝑤
𝑁1(2)

𝐷 ∗ 𝑉𝑜𝑢𝑡𝑅 ∗ 𝛥𝑉𝑜 ∗ 𝑓𝑠𝑤

(3)

𝐶𝑐 =

3.2. The Simulation on MATLAB Simulink and
PLECS
MATLAB Simulink and PLECS were utilized forrunning simulations, which play a significant role inthe selection of components for the power converter.
The design overview of the AC-DC 5-Watt USB
charger on MATLAB Simulink and PLECS softwareare shown on Fig. 6 and Fig. 7. Additionally, thePWM PI feedback control was implemented toaddress steady-state error, requiring lower forward
gain.

Following the calculation of Lc and Cc using the
Flyback equation, the appropriate high-frequency
transformer, and inductor was selected.
To mitigate ripple voltage and ensure converter
stability, a higher capacitor value was subsequently
chosen in accordance with the formula. The selectionof a higher capacitor value is essential for reducing
ripple voltage, even though it may result in a higher
peak current at the input. These components
specifications are outlined in Table 2.

The PI controller ensures a stable 5V output undervarying input voltages by generating a corrective
signal to address system faults. Manual adjustments
to the proportional and integral values of the PI
controller were made until the desired output wasstabilized without overshooting.

Table 2. The Specification of the Converter.
Converter Type

AC-DC
BridgeRectifier,
DC-DC Flyback Topology

Switching
DeviceSwitchingFrequency
HF
Transformer

MOSFET (C3M0120065J
CREE was used)
20kHz -100kHzby

AC to DC DiodeEE Series (EE16 74091 Flyback
Transformer by MYRRA wasused)
IN5819

Capacitor

250uF

Inductor

6000uH

Controller Type

Output Voltage

PWM PI Feedback Control (Noovershoot and 0,1 Second
Settling)
Region Independent (230/100
VAC at 50/60Hz)
5 VDC

Output Current

1A

Load Type

Resistor (5-ohm)

Input Voltage

The flyback capacitor is adjusted to 250
microfarads during PI value tweaking to prevent
excessive ripple. The PI value should be at P=0.06
and I=16, it was subsequently found.
The MATLAB Simulink environment is configured
with the solver ode23t (modified Stiff/Trapezoidal),as outlined in Fig. 5 to facilitate analysis before
conducting performance tests.

Fig. 5 The Configuration Parameters of the
Simulink Solver.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
842

Investigate Power Efficiency inFig. 6 The MATLAB Model of the AC – DC 5-Watt USB Charger.

Fig. 7 The PLECS Model of the AC – DC 5-Watt USB Charger.
were conducted using a 𝑓𝑠𝑤 value of 40kHz and a PI
value of P=0.06 and I=16. The results from both
simulations are tabulated and compared to eachother.

4. Results and Discussion
The 5-Watt USB Charger simulation results, asdetailed in Table 3, are presented utilizing both
MATLAB Simulink and PLECS. The simulations©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
843

Zhi Ying Yeoh, Kim Soon Chong, Sew Sun Tiang, Mohammad Arif Ilyas, Jia Wei Tan, Wei Kang Lai, Wei Hong Lim

Table 3. Comparison of the Simulation Results of the AC – DC 5-Watt USB Charger.

Voltage Region

Input Voltage
Input Current
Input Power

Frequency

50Hz
141V
60Hz
50Hz
325V
60Hz

MATLAB
138.9V
45mA
6.31W
138.9V45mA
6.307W
323.1V
22.6mA
7.307W
323.2V22.5mA
7.302W

Output Voltage
Output Current
Output Power

PLECS
138.83V
43.5mA
6.03W
138.9V43.4mA
6.034W
323.23V
18.5mA
5.989W
323.25V
18.5mA
5.989W

MATLAB
5.002V
1.00A
5.003W
5.002V
1.00A
5.004W
4.996V
999mA
4.992W
4.996V
999mA
4.992W

PLECS
4.985V997mA
4.969W
4.985V997mA
4.970W
4.987V997mA
4.973W
4.987V997mA
4.974W

4.1. The Output Voltage, Current and Power4.2. Thermal Analysis
The result indicates that the USB charger produces
an output voltage of 5VDC and an output current of
1A and shown in Fig. 8.

Fig. 9 performs the scope thermal analysis of the
input voltage at 230/100 VAC in MATLAB
Simulink (a) and PLECS (b). The result of MOSFET
temperature reaches a steady-state value of 25 °C in
MATLAB Simulink and 25.06 °C in PLECS, after asimulation time of 0.1 seconds.

Fig. 8 The Scope of Voltage, Current, and Power
Output at 325/100V AC with 50/60Hz.
Table 4 performs detail comparison on the
MOSFET temperature after simulation at different
voltage regions and frequencies.

(a)

The MOSFET temperature is slightly higher in
PLECS than in MATLAB Simulink. This isconsistent with the results of the simulation setup,
which showed that PLECS software utilized a moredetailed thermal model of the MOSFET. As can beseen from Fig. 9 (a) and (b), the MOSFET
temperature is slightly higher in PLECS than in
MATLAB Simulink.
Table 4. The MOSFET Temperature After
Simulation
Voltage
RegionFrequency

141V

50Hz

25 °C

25.06 °C

60Hz

25 °C

25.06 °C

50Hz

25 °C

25.01 °C

60Hz

25 °C

25.01 °C(b)
Fig. 9 The scope thermal analysis at 230/100VAC
at MATLAB Simulink (a) and PLECS (b).

325V

MOSFET
Temperature
MATLABPLECS©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
844

Investigate Power Efficiency inWhile the same datasheet was implemented, thisdoes not demonstrate that the MATLAB Simulink
software is superior for simulating the circuit
because the MOSFET's specific parameter differsfrom that of the PLECS Software. This is because
the MOSFET in the PLECS Software is based on thecharacteristics of the actual MOSFET that have been
implemented by the manufacturer, CREE. On theother hand, the parameters for the MOSFET in the
MATLAB/Simulink Software are manually enteredbased on the datasheet.

4.3. Power Efficiency
Power efficiency results can be observed on
differences input voltages and frequencies in Table
5. The table showed power efficiency was also
compared with using MATLAB and PLECS
software. The results in the MATLAB are between
68% and 79%. On the other hand, the power
efficiency in PLECS is more consistent at 82.35% to
83.04%. It is notably underscoring the difference inthe power efficiency in both software.

When simulated in 0.1 seconds, the MOSFET
temperature of each software is slightly comparable.
Nevertheless, the temperature of the simulation in
PLECS Software is slightly higher than that of thesimulation in MATLAB Simulink Software. This isbecause MOSFET’s specific parameter varies
depending on the software.

Table 5. Power Efficiency After Simulation.
Voltage
Region141V

325V

Frequency

Power Efficiency
MATLAB

PLECS

50Hz

79.29%

82.35%

60Hz

79.35%

82.37%

50Hz

68.31%

83.04%

60Hz

68.37%

83.04%In MATLAB Simulink software, the MOSFET ismanually inserted based on the datasheet, which
may result in less precision than the model used in
the PLECS software. Whereas in PLECS software,
MOSFET is based on the manufacturer, CREE. The
parameters and thermal model of MOSFET areneeded to add into PLECS thermal library.
The ways to solve the numerical methods involved
to solve the system of equations also can impact theoutcome of simulations in both MATLAB Simulink
and PLECS Software, which might result in somedegree of error throughout the simulation results.

Comparing to MATLAB Simulink, theperformance of PLECS on simulation power
efficiency USB charger exhibits a 3.06%
improvement at 141 VAC with 50/60Hz. Moreover,
PLECS further outperforms, demonstrating asignificant 14.73% increase in power efficiency witha 325V AC input voltage at 50/60Hz frequency.

Step size, solver parameters, and the accuracy of
the used models are only a few of the variables that
affect how accurate the simulation is. It is advised todecrease the step size or raise the solver accuracy inboth MATLAB and PLECS software and evaluate if
the simulation results converge to improve theaccuracy of the simulation results.

MATLAB is a general-purpose software used inengineering fields for numerical analysis, datavisualization, and simulation of complex systems,
using a discrete-time simulation method.
MATLAB's general approach to simulation may not
be as accurate in modelling power electronics
components, resulting in less consistent power
efficiency results.

5. Conclusion
This paper successfully constructs a 5-Watt USB
power charger, exploring AC to DC full bridge
rectifier and DC-to-DC Flyback converters. Eachcircuit component's parameter adjustments produce
the 5VDC, 1A output, necessitating a PI controller
for stability. This controller manages P = 0.06 and
I=16, minimizing the difference between reference
and actual voltage. Two simulation software werecompared for their performance and outcomes.
PLECS excels, designed specifically for powerelectronics, offering greater accuracy and faster,
memory-efficient, showing a 3.06% advantage at
141V and 50/60Hz. Moreover, at 325V AC input
voltage with 50/60Hz frequency, PLECS excels
further, exhibiting 14.73% increase in power

PLECS is designed specifically for simulating
power electronics systems, using a continuous-time
simulation method with specialized models for
simulating power electronic components. The
different simulation methods and models used bythese two software tools could result in variations insimulation outcomes, including power efficiency.
PLECS specialized models accurately represent the
behaviour of power electronic components leading
to more precise simulations and consistent power
efficiency results.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
845

Zhi Ying Yeoh, Kim Soon Chong, Sew Sun Tiang, Mohammad Arif Ilyas, Jia Wei Tan, Wei Kang Lai, Wei Hong Lim

Applied Power Electronics Conference and Exposition(APEC), IEEE, Jun. 2021, pp. 426–431.
12. J. Weimer, D. Koch, M. Nitzsche, J. Haarer, J. RothStielow, and I. Kallfass, “Miniaturization and Thermal
Design of a 170 W AC/DC Battery Charger Utilizing GaN
Power Devices,” IEEE Open Journal of Power Electronics,
vol. 3, pp. 13–25, 2022.
13. B. K. Kushwaha, R. R., S. Jadhav, and S. T. S.,“Thermal Analysis of 2.2kW Transformer of On-Board
Charger for Electric Vehicles,” in 2023 Second
International Conference on Electronics and Renewable
Systems (ICEARS), IEEE, Mar. 2023, pp. 311–314.
14. Y. D. Kuan, S. R. Lee, S. M. Lee, M. F. Sung, and H.
S. Chiou, “The Performance Analysis of a Portable
Charger System Using Direct Methanol Fuel Cells,” inProceedings of the 35th International MATADOR
Conference, London: Springer London, 2007, pp. 129–132.

efficiency compared to MATLAB Simulink. The
simulation study provides a comprehensive
understanding of converter interactions andhighlights PLECS Software advantages in powerelectronics simulations over MATLAB Simulink
Software.
Acknowledgement
This research received no specific grant from any
funding agency in the public, commercial, or notfor-profit sectors.
References
1. F. Bordry and D. Aguglia, “Definition of Power
Converters,” CAS - CERN Accelerator School: Power
Converters, Baden, Switzerland, pp. 15–43, Jun. 2015.
2. M. G. Simões and F. A. Farret, Modeling PowerElectronics and Interfacing Energy Conversion Systems.
Wiley, 2016.
3. C.-J. Chen, C.-H. Cheng, P.-S. Wu, and S.-S. Wang,“Unified Small-Signal Model and Compensator Design of
Flyback Converter With Peak-Current Control at Variable
Frequency for USB Power Delivery,” IEEE Trans PowerElectron, vol. 34, no. 1, pp. 783–793, Jan. 2019.
4. J. Zhai, F. Wang, L. Ye, and R. Li, “Optimal Strategy of
Triple Phase Shift Transient DC Bias for Dual Active
Bridge DC-DC Converter,” E3S Web of Conferences, vol.
185, p. 01088, Sep. 2020.
5. A. Singh, A. K. Yadav, and A. Khaligh, “Steady-State
Modeling of a Dual-Active Bridge AC–DC Converter
Considering Circuit Nonidealities and Intracycle
Transient Effects,” IEEE Trans Power Electron, vol. 36,
no. 10, pp. 11276–11287, Oct. 2021.
6. C. S. Wong, K. H. Loo, and L. Cao, “A Single-stageThree-phase Bidirectional AC-DC IPT Converter basedon SWISS-Rectifier for EV Charging Applications,” in
IECON 2022 – 48th Annual Conference of the IEEE
Industrial Electronics Society, IEEE, Oct. 2022, pp. 1–6.
7. Filho, “Predictive Control Applied to a Single-Stage,Single-Phase Bidirectional AC-DC Converter,” IEEE
Access, vol. 10, pp. 34984–34995, 2022.
8. S. Bagawade, I. Askarian, M. Pahlevani, and P. Jain, “A
New Discrete Four-Quadrant Control Technique for GridConnected Full-Bridge AC–DC Converters,” IEEE J
Emerg Sel Top Power Electron, vol. 10, no. 2, pp. 2362–
2379, Apr. 2022.
9. J. Liu, Y. Cheng, Y. Jia, and H. Wu, “Optimal Design ofa Constant Frequency Series-Resonant DC/DC Converterwith Wide Voltage Gain Range for Single-Stage Isolated
AC/DC Power Conversion,” in 2022 International PowerElectronics Conference (IPEC-Himeji 2022- ECCE Asia),IEEE, May 2022, pp. 1036–1043
10. C. Wang, D. Sun, X. Zhang, W. Gu, and S. Gui, “A
Constant Current Digital Control Method for PrimarySide Regulation Active-Clamp Flyback Converter in
CCM Mode,” in 2021 IEEE Energy Conversion Congress
and Exposition (ECCE), IEEE, Oct. 2021, pp. 3501–3506.
11. B. Zhang, J. Deng, L. Li, Z. Wang, S. Wang, and G.
Guidi, “Thermal Analysis and Design of a 30kW EV
Wireless Charger with Liquid-Cooled Shell for Magnetic
Coupler and Integrated Power Converter,” in 2021 IEEE

Authors Introduction
Ms. Zhi Ying Yeoh
She is currently pursuing
Bachelor of Engineering
Electrical and Electronics
with Honors as final year
student in Faculty ofEngineering,Technology
and Built Environment,
UCSI University, Malaysia.
Her research interests areelectrical
and
power
electronics.
Ms. Jia Wei Tan
She is currently pursuing
BachelorofEngineering
Communication
andElectronics with Honors as final
year student in Faculty ofEngineering, Technology and
Built Environment, UCSI
University, Malaysia. Her
research
interests
arecommunication and antenna design.
Mr. Wei Kang Lai
He is currently pursuing
Bachelor of Electrical andElectronics Engineering as final
year student in Faculty ofEngineering, Technology and
Build
Environment,
UCSI
University,
Malaysia.
His
research interests are electricaland solar system design.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
846

Powered by TCPDF (www.tcpdf.org)

Investigate Power Efficiency in

Dr. Sew Sun Tiang
She is an Assistant Professor in
Faculty of Engineering at UCSI
University in Malaysia. She
received her PhD in Electricaland Electronic Engineering from
Universiti Sains Malaysia in
2014. Her research interests are
optimization
andantennadesign.
Dr. Kim Soon Choong
He obtained his B.Eng., M.S.,
and Ph.D. degrees in electricaland electronic engineering from
Universiti Kebangsaan Malaysia
in 2012, 2015, and 2019,respectively. He currently serves
as an Assistant Professor in the
DepartmentofComputerEngineering (Artificial Intelligence) within the
Faculty of Engineering Technology and Built
Environment at UCSI Kuala Lumpur. His research
focuses on areas such as medical signal processing,etc..
Dr. Mohammad Arif Bin Ilyas
He received his PhD in ElectricalEngineering from Universiti Tun
Hussein Onn Malaysia in 2020.
Currently, he is a lecturer in the
FacultyofEngineering,Technology,
and
Built
Environment at UCSI University.
His main areas of researchinterest are machine learning, artificial intelligence,
the Internet of Things, embedded systems, etc..

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
847

Design and Simulation and Performance of Grid Connected Photovoltaic System for
Small, Tall Building in Malaysia
Wei Kang Lai1, Kim Soon Chong1,*, Mohammad Arif Bin Ilyas1, Sew Sun Tiang1, Zhi Ying Yeoh 1, Jia Wei Tan1, Wei
Hong Lim1
1

Department of Electrical and Electronics Engineering, Faculty of Engineering, Technology and Built
Environment, UCSI University, 56000 Cheras, Kuala Lumpur, Malaysia
E-mail: 1001953677@ucsiuniversity.edu.my, chongks@ucsiuniversity.edu.my,
MohammadArif@ucsiuniversity.edu.my, tiangss@ucsiuniversity.edu.my, 1001954348@ucsiuniversity.edu.my,1001953809@ucsiuniversity.edu.my, limwh@ucsiuniversity.edu.my(1UCSI University, Malaysia)
Abstract
In Malaysia's rapidly urbanizing landscape, sustainable energy for small, tall buildings is increasingly vital.
This study addresses this need through the design, simulation, and performance analysis of grid-connected
photovoltaic systems tailored for these unique structures. Utilizing AutoCAD and PVSyst for design andsimulation, the research details rooftop array dimensions, PV panel wiring, and system components alignedwith MS 1837-2018 standards. Results indicate the designed system can save 526.70MWh annually, costing
RM 339,306.98, with a 9.8% return on investment (ROI) and a 13-year breakeven point, emphasizing its
sustainability and economic viability.
Keywords: PVSyst, AutoCAD, Cost saving, ROI, Breakeven point

1.

panels, system sizing, inverter selection, and system
integration with existing electrical infrastructure.
In Malaysia, the rapid growth of urbanization
has led to an increased demand for sustainableenergy solutions. Despite the potential benefits ofgrid-connected photovoltaic (PV) systems inmeeting this demand, there is a lack ofcomprehensive research addressing the design,
simulation, and performance analysis tailoredspecifically for the unique challenges posed by small,
tall buildings in the Malaysian context. The main
objective of this paper is to bridge this gap byinvestigating the optimal design parameters,
conducting detailed simulations, and performing athorough performance analysis of grid-connected
PV systems. The outcomes of this study are
expected to provide valuable insights into thefeasibility and efficiency of implementing gridconnected PV systems in this specific urban setting,contributing to the advancement of sustainableenergy solutions for the region.

INTRODUCTION

In recent years, as non-renewable energy sources
such as coal, oil, natural gas, chemical energy, andnuclear fuel have begun to run out and sustainable
development has begun to be threatened, the use ofrenewable energy has become increasingly
important. All these energy sources are also harmful
to the environment and can lead to environmental
problems such as greenhouse gas (GHGs) emissions
and hazardous chemical waste [1], [2], [3].
Therefore, research and development of renewableenergy sources is crucial.
Among various renewable energy options,rooftop solar photovoltaic (PV) systems are a
popular and effective method for collect solar energy[1]. A grid-connected rooftop solar photovoltaic
system is a solar power generation system that isconnected to the grid and can transmit excess energy
back to the grid. In addition, Malaysia is one of thecountries with the greatest potential for solar energy
utilization due to its strategic location near theequator. Through the research, the monthly solar
irradiance for Malaysia is estimated at 400-600
MJ/m2 as well as its hot and sunny weather
throughout the year, thus the potential for solar
power is very large [4]. However, the design andimplementation of a grid-connected rooftop solar
PV system requires careful consideration of several
factors, including the selection of appropriate solar

2. METHODOLOGY
2.1. Calculation and Selection for Solar PanelIn this study, one of the buildings in the UCSI
university was selected for installing the PV panels.
The selected building, named Block C, located at No.
1, UCSI Heights, Jalan Puncak Menara Gading,
Taman Connaught, 56000 Cheras, Federal Territoryof Kuala Lumpur. As shown in Fig. 1, the satellite
view shows that there are 12 faces on the rooftop©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

848

Wei Kang Lai, Kim Soon Chong, Mohammad Arif Bin Ilyas, Sew Sun Tiang, Zhi Ying Yeoh, Jia Wei Tan, Wei Hong Lim

with the tilts of 30 degrees in each azimuth. The areaof the rooftop was calculated to be 3194m2.
However, only 8 faces are decided to install the PV
panels which is 2129m2 and the other 4 faces are forcleaning and technical maintenance. These data
were then used to create a 3D model of the rooftop
in 3D AutoCAD, and the appropriate PV array size
was decided using 1120 modules of SunPower X22360 solar panels based on calculated by using
expected power for each orientation which is 50kW
and spaces on the rooftop. Below is the calculation:

at exterior placement not only minimizing energy
waste from cable resistance but also enhance
efficiency by reducing the length of the DC wire
between the solar panel and the inverter.

3,194𝑚2
×8
122= 2,129𝑚

𝐴𝑟𝑒𝑎 𝑓𝑜𝑟 𝑝𝑙𝑎𝑐𝑖𝑛𝑔 𝑃𝑉 𝑝𝑎𝑛𝑒𝑙𝑠 =50,000𝑊
360𝑊= 140 𝑚𝑜𝑑𝑢𝑙𝑒𝑠

Figure 1: Satellite view of a building’s rooftop

𝑁𝑜. 𝑜𝑓 𝑃𝑉 𝑝𝑎𝑛𝑒𝑙𝑠 =2.2. Solar panel layout design
The floor plan of the roof PV panels was constructbased on the Google Maps satellite images as shownin Fig. 1. The dimensions of the SunPower X22-360
were obtained from the data sheet in order to arrange
the panels in the most effective and aesthetically
acceptable way possible. The route of the DC wires
from the PV panels to the inverter and the AC cablesfrom the inverter to the electrical panel was also
determined at this point. Thorough the analysis, a 3Dmodel of PV system design on the rooftop wasshowed in 3D AutoCAD shown in Fig. 2.

𝑇𝑜𝑡𝑎𝑙 𝑛𝑜. 𝑜𝑓 𝑃𝑉 𝑝𝑎𝑛𝑒𝑙𝑠 = 140 𝑚𝑜𝑑𝑢𝑙𝑒 × 8= 2,129𝑚2
𝑃𝑜𝑤𝑒𝑟 𝑔𝑒𝑛𝑒𝑟𝑎𝑡𝑒𝑑 = 1,120 𝑚𝑜𝑑𝑢𝑙𝑒𝑠 × 8= 403,200𝑊
Apart from that, to convert the DC power
generated by the solar panels into AC power, themodel of Goodwe GW50KLV-MT inverter had been
selected.
The SunPower X22-360 Solar panel is a highperformance solar panel designed for residential andcommercial applications. In a six-story building,
SunPower's X22-360 solar panels can be used togenerate renewable energy to offset the building's
energy consumption. The electricity generated by
the solar panels can be used to power lighting,heating, ventilation, and air conditioning (HVAC)
systems and other electrical equipment in thebuilding, while excess energy can be fed back to thegrid to generate revenue for the university through a
net metering program. Additionally, the X Series
panels have a 25-year warranty and are made to belong-lasting and robust. This long-term durability
improves X-Series panels a cost-effective, sustained
producing electricity for many years [5].
The Goodwe GW50KLV-MT is a three-phasegrid-tied inverter intended to transform the direct
current (DC) power produced by the solar panel
arrays into alternating current (AC) power that canbe supplied into the utility system. Goodwe
GW50KLV-MT has also built-in safety features likeground fault protection (GFP) and surge protection
devices (SPDs) for both DC and AC inputs, which
provide both the safety of the solar PV system andthe building's occupants [6].
The inverters of the system were installed atexterior placement to create a more spacious indoor
environment and address safety concerns related topreventing electrical hazards. Placing the inverters

Figure 2: 3D Modelling rooftop design
2.3. Wire Connection
Leapfrog chain wiring, as outlined in Fig. 3 is thebest and suitable wiring design for string panels thatare not organized in a straight line due to its
simplicity, ease of use, cost saving andelectromagnetic loop reduction which represent lessvoltage drop [7], [8].

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

849

Design and Simulation andrequirements for the installation of grid-connected
PV systems. This diagram provides a clear andconcise representation of the system components,
including the arrangement of PV array, inverters, DC
and AC disconnects, and electrical panels. It also
includes details on the system grounding andprotection devices, such as surge protectors andovercurrent devices [9].

Figure 3: Leapfrog Wiring
According to the MS 1837-2018 standard,
single-line diagram of the system shown in Fig. 4
was designed and outlined the minimum

Figure 4: Single-line diagram
In Fig. 4, the direct current (DC) electricity
produced by the 5 strings of 7 solar panels wasrouted through DC surge protective devices (SPDs)positioned before and after the PV DC switch. The
DC SPDs safeguard the system against surges andovervoltage. The DC electricity was then routed
through the PV DC switch to the eight generators.
Each inverter has four MPPT (maximum power
point tracking) sources to optimize electricity
production from the panels.
The wires before and after the PV DC switch
and the wires before and after the PV AC switch
were selected as MC4 compatible for the head andtail of the wire and through calculation. Due to thedesign of 7 module of solar panels in seriesconnection and each module can generate 60.6V at
maximum power point, the output of the voltage willbe 424.2V.
Through the calculation, the voltage for thewires required to withstand is at least 424.2V. Thus,
due to the critical safety aspect, THHN/THWN-2
wire which can withstand 600V was selected toprevent the wiring from overheating and overloadthat may cause fire hazard and short circuit. While
the other reason for choosing THHN/THWN-2 wire
was the temperature rating for this wire is 90 degrees
Celsius and could use in wet conditions [10].

2.4. Selection in PVsyst software
In Fig. 5, it depicts the desired tilts and azimuths.
Azimuths of 0°, 30°, 90°, 120°, 180°, -120°, -90°, 30° have been selected for placing the solar panelson the rooftop.

Figure 5: Orientations (Tilts and Azimuths)
In the PVSyst software, the selection andapplication of dimensions, photovoltaic modules
and inverters are shown in Fig. 6. The SunPower
X22-360 model for the solar panel and the Goodwe
GW50KLV-MT model for the inverter were chosen©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

850

Wei Kang Lai, Kim Soon Chong, Mohammad Arif Bin Ilyas, Sew Sun Tiang, Zhi Ying Yeoh, Jia Wei Tan, Wei Hong Lim

for each orientation in PVSyst software. The
inverters will conduct 4 MPPTs each, as theoperating DC voltage needs to fall within the MPPT
voltage range of the inverter. Therefore, 5 strings
that are connected in parallel will be connected to
one of the MPPT for the inverter. A total of 20
strings, each consisting of 7 solar panels per string,
will be subject to application based on the expected
power determined in PVSyst software, which is 50
kW.
After all the values and the selection wasapplied, the voltage maximum power point obtained
is 367V and 418V at 60 degrees and 20 degrees,
while AC voltage is 531V. Moreover, the value ofarray nominal power at Standard Test Conditions(STC) also determine as 50.4kWp.

Besides, to account for the various variables
that may influence the system's real performance as
opposed to the idealized circumstances anticipatedin laboratory testing, the Thermal Losses, Ohmic
Losses, Power Loss at Maximum Power Point(MPP),
Soling Loss, and Aging Losses has been applied in
the PVsyst software. These variables may include
shading, weather, soiling, inefficient modules, andinverters, among other practical and environmental
problems that could eventually result in a drop in
power production. It is possible to get a moreaccurate estimate of the system's real energy yieldand financial success by including these losses in thesimulation. It is also possible to find prospective
areas for optimisation and development.

Figure 6: Sizing and system applied in PVSys
The accumulation of dust or debris can alsodegrade the panel's performance [11]. Therefore,leaving space between panels is essential, providing
convenient for maintenance and cleaning. Overall,allocating sufficient space for solar panels is vital,
ensuring optimal performance, power efficiency,
and longevity of solar photovoltaic systems.

2.5. Arrangement of Solar Panels
The spacing between the solar panels is an essential
factor in the design of solar photovoltaic systems, as
shown in Fig. 7. The rationale behind providing
adequate space was to ensure that there is proper
ventilation and air flow around the panels. This aids
to keep the panel cool, subsequently enhances
performance.
Minimizing shading is imperative, it cansignificantly reduce the output of photovoltaic
systems. Therefore, by leaving space between panels,
can mitigate the risk of shading, which occurs when
one panel casts a shadow on another panel.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

851

Design and Simulation andtypically exhibits a default voltage drop of 0.7 V.
Concurrently, ohmic losses emerge from theresistance in wires and connectors, contributing to apotential decline in the solar system's output power.
To quantify these losses within the subfield,
the loss fraction was specified as STC which is 2%.
In this scenario, the solar system achieves a total
power output of 400 kW at STC and ohmic losses
are stipulated at 2%, the resultant power loss due toohmic factors amounts to 2 kW.
Apart from that, the yearly soiling loss factor
indicates the decrease in solar panel energy
production caused by dust and dirt accumulation on
the surface. It varies depending on the location andenvironmental conditions of the solar panel
installation such as rainfall frequency, wind patterns,
and proximity to dust sources can affect the rate ofsoiling. The annual soiling loss factor was found to
be 3% analysed by PVSyst software, reducing solar
panel’s performance. This outcome can be used todetermine the cleaning frequency needed to ensure
the solar panel performing optimally.
The PVSyst module analysis revealed anaverage degradation of 0.4% per year with a 10-year
loss factor. This indicates that the module's
efficiency is expected to decrease by 0.4% each year
over the course of 10 years.
The analysis also revealed a mismatch due todegradation and resulting in Root Mean Square of
Current (Imp RMS) and Root Mean Square of
Voltage (Vmp RMS) dispersions of 0.4% per year.
Owing to the degradation of the panels, the current
and voltage output from the solar panels will
experience slight deviations over time. Thesefindings highlighted the importance of regular
maintenance and monitoring of the solar panels toensure optimal performance and efficiency over the
long-term.

Figure 7: Spacing between Solar panels
2.6. Budget of the projectAn expenditure of 911,192 dollars, equivalent to
RM 4,309,026.97, as shown in Table 1, wasanticipated for the acquisition of PV panels and solar
inverters utilized in this project.
Table 1: Total cost of material
Product

Model

Total
unit

Singleunit
price(dollar)Total
price ofall units(dollar)PV
panel

SunPower
X22-360

1,120

800

896,000

Solar
inverter

Goodwe
GW50KLVMT

8

1,899

15,192

Total

911,192

3. RESULT AND DISCUSSIONS
3.1. PVsyst Analysis
The array loss analysis simulated by PVSyst
software as outlined in Fig. 8. The series diodeFigure 8: Array Loss analysis©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

852

Wei Kang Lai, Kim Soon Chong, Mohammad Arif Bin Ilyas, Sew Sun Tiang, Zhi Ying Yeoh, Jia Wei Tan, Wei Hong Lim

calculated to be 1,641.8 kWh/m2, indicating a
reduction in the total energy incident on the moduledue to the tilt angle and orientation.
According to the PVSyst analysis the obtained
results affirm that the PV system was performing
well in terms of energy production. The highspecific production or high output of 1,306
kWh/kWp/year indicates that the system was welldesigned and well-maintained, which was a positivesign. In addition, the total energy injected into the526,695kWh grid indicated a critical commitment to
the overall electricity supply on site.
The analysis of the results indicates that, in thegiven location, the PV module tilt point and thedegree of inclination are not optimally configured.
This was reflected in the comparatively lower valueof Globinc in comparison to GlobHor.
By focusing on the DiffHor value of 963.42
kWh/m2, the location of the PV system is not highly
shaded, which was a positive factor for energy
production. The average ambient temperature during
the study period of 27.72℃ was considered within
the cycle and is also within the ideal temperature
range for photovoltaic system performance.
Apart from that, the average performance ratio(PR) that indicates the efficiency of the system inconverting the solar energy into electrical energy
was calculated to be 0.796, equivalent to 79%,
which is close to 80%. Nevertheless, for the year
2010 is found to be between ~70 and ~90% andshows a median PR of 84%. However, the analysis
shows that it does not getting 84% the PV system
due to the imperfection of tilts. To solve this problem,
it is recommended to add an adjustable mounting
system for optimal tilt and performance [12].

Within the total PV power characteristic, thetotal number of utilized PV modules is 1,120,
generating a nominal (STC) output of 403 kWp as
shown in Fig. 9, which is consistent with thecalculated value. The covered module area and unit
area are 1,826m2 and 1,648 m2 respectively.
The total power of the inverters that have
converted was 400kWac and the number of theconverter used is 8 units. The ratio of PV to inverter
power is measured as the DC/AC ratio whereby
ideal value is 1.25. Nevertheless, the simulation
analysis indicates the power nominal (Pnom) ratio is1.01, falling short of the recommended healthy ratio
for the PV system.

Figure 9: Total PV power and inverter power
In Fig. 10, it shows the total global horizontal
irradiation (GlobHor) for the given area was foundto be 1,783.4 kWh/m2, whereas the total horizontal
diffuse irradiation (DiffHor) was calculated to be
963.42 kWh/m2. During this period, the averageambient temperature (T_Amb) was 27.72℃ . The
total global incidence in coll. plane (Globinc) was©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

853

Design and Simulation and

Figure 10: Balance and main results
In this study, the efficiency of the PV system
has comprehended by the system output power
distribution graph as shown in Fig. 11. The graphshows that the system operates at optimum
efficiency, with a peak energy injection of 9
MWh/class of 2.5 kW, corresponding to a power
injection of 240 kW. Notably, the majority of theenergy injected into the grid falls within a stable
range of 5 to 7 MWh/class of 2.5 kW, suggesting
efficient and consistent operation. The distribution
graph provides useful insights into the system's
performance that can be used to find potential areas
for development and optimize the functioning of the
PV system. By knowing the power levels at which
the system produces the most energy, the system canbe operated to maximize energy output at thesepower levels.

Figure 11: Power distribution graph3.2. Estimate Cost Saving, Breakeven point
and Return of Investment (ROI)
According to the analysis, 526.70 MWH can begenerated throughout the year, with an average of
43.891 MWH per month. The electricity bill is
calculated using the formula provided by Tenaga
National Berhad (TNB), as shown below:
Electricity Bill = (Consumed electricity x First 200
kWh rate) + (Consumed electricity x Next 100 kWhrate) + (Consumed electricity x Next 300 kWh rate)+ (Consumed electricity x Next 300 kWh rate) +(Consumed electricity x Next 901 kWh onwards permonth) *(6% tax)

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

854

Wei Kang Lai, Kim Soon Chong, Mohammad Arif Bin Ilyas, Sew Sun Tiang, Zhi Ying Yeoh, Jia Wei Tan, Wei Hong Lim

At the beginning of the investment, it resulted
in a net loss and no returns due to the high upfront
cost of the system which is RM 4,309,026.97 [13].
In Year 13, the ROI has become positive after 12
years as shown in Table 3. Beyond the 13th year, it
is expected to be in a profit-making state, where thesaving from the electrical bill exceeds the total costof panels ordered. Regarding the breakeven point
analysis, it shows this project needs to take 12.69
year which is almost 13 years to reach the breakevenpoint.

Table 2: Year of the ROIElectricity BillRM 28,146.24
RM 339,306.98

Per Month
Per YearThrough the calculation, user can saveminimum up to RM 28,146.24 per month and RM
339,306.98 per year as shown in Table 2. Besides,
the use of electricity from the solar panels areprioritised instead of using electricity from the TNB
grid whereby surplus electricity will not be wasted
and exported back to TNB.

4.

CONCLUSIONYear

ROI = (Net profit / cost of investment) x
100

1

-92%

2

-84.4%

3-76.6%

4-68.8%

5

- 61%

6

-53.2%

This study examines the potential of installing asolar photovoltaic (PV) system at a small, tallbuilding in Malaysia. The analysis considers factors
such as the building's energy demand, available
space, and the financial feasibility of the proposal.
The findings of the analysis reveal that a 403-kW
solar PV system has the capacity to generate
approximately 526695 kWh of electricity and helps
to save up to RM 339,306.98 per year which cansatisfy the building's power requirements.
Apart from cost saving and power supply, theperformance of PV systems is not optimal. In orderto take full advantage of the potential capabilities ofthe system and reduce performance differences, key
recommendations for the future include the adoptionof adjustable mounting systems. This technology
facilitates dynamic tilt regulation, ensuring optimalalignment of photovoltaic panels with incoming
sunlight, thereby improving overall energy
conversion efficiency. However, the calculation of
the ROI and the breakeven point shows it takes 13
years to reach profit-making state with no expenses.
This is because of the upfront cost of the system ishigh to begin with.

7-45.4%

5.

8-37.6%

9-29.8%

This research received no specific grant from any
funding agency in the public, commercial, or notfor-profit sectors.

10

-22%

REFERENCES

11-14.2%

12

-6.4%

13

1.4%

14

9.2%

1. M. Alsulaiman and N. Mohammadi, “Optimal Pitch
Distance and Tilt Angle of PV Power Plant for
Different Climate,” 2020.
2. S. I. Mustapa, L. Y. Peng, and A. H. Hashim, “Issues
and challenges of renewable energy development: A
Malaysian experience,” in Proceedings of theInternational Conference on Energy and Sustainable
Development: Issues and Strategies, ESD 2010, IEEE
Computer Society, 2010.
3. F. Chien, L. Huang, and W. Zhao, “The influence ofsustainable energy demands on energy efficiency:
Evidence from China,” Journal of Innovation andKnowledge, vol. 8, no. 1, Jan. 2023.
4. W. S. W. Abdullah, M. Osman, M. Z. A. A. Kadir, and
R. Verayiah, “The potential and status of renewable

𝑅𝑂𝐼 =𝑁𝑒𝑡 𝑝𝑟𝑜𝑓𝑖𝑡
× 100
𝐶𝑜𝑠𝑡 𝑜𝑓 𝐼𝑛𝑣𝑒𝑠𝑡𝑚𝑒𝑛𝑡

𝑅𝑀339,307 − 𝑅𝑀4,350𝑘
× 100
𝑅𝑀4,350𝑘= −92%

𝑌𝑒𝑎𝑟 1 =Table 3: Year of the ROI

Breakeven point (in year)= Fixed Costs / Gross Profit Margin= RM 4,309,026.97/ RM 339,306.98= 12.69 year

ACKNOWLEDGE©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

855

Powered by TCPDF (www.tcpdf.org)

Design and Simulation andenergy development in Malaysia,” Energies (Basel), vol.
12, no. 12, 2019.
5. F. Reil, I. Baumann, J. Althaus, and S. Gebhard,“Evaluation of current standards and practices for thesimulation of wind-blown sands and their applicability as
accelerated ageing tests for PV modules,” in Conference
Record of the IEEE Photovoltaic Specialists Conference,
Institute of Electrical and Electronics Engineers Inc., 2013,
pp. 1537–1541.
6. “MT G2 Series User Manual Grid-Tied PV Inverter.”[Online]. Available: www.goodwe.com
7. T. Bravo Mahachi and A. Johan Rix, “Energy yield
analysis and evaluation of solar irradiance models for autility scale solar PV plant in South Africa,” 2018.
8. A. Eric El Obeid, “BEST-PRACTICES GUIDELINES
AND LESSONS LEARNT FOR ON-GRID AND PVDIESEL HYBRID SYSTEMS.” [Online]. Available:
www.lb.undp.org/DREG
9. “MALAYSIAN STANDARD Installation of gridconnected photovoltaic (PV) system (Second revision)
DEPARTMENT OF STANDARDS MALAYSIA
DEVELOPMENT OF MALAYSIAN STANDARDS,”
2018, [Online]. Available: http://www.sirim.my
10. C. Marketing, “THHN/THWN-2 600V CT RoHS
CABLE.” [Online]. Available: www.nexans.co
11. F. Saeed and A. Zohaib, “Quantification of Losses ina Photovoltaic System: A Review †,” Engineering
Proceedings, vol. 11, no. 1, 2021.
12. N. H. Reich, B. Mueller, A. Armbruster, W. G. J. H.
M. Van Sark, K. Kiefer, and C. Reise, “Performance ratio
revisited: Is PR > 90% realistic?,” in Progress in
Photovoltaics: Research and Applications, Sep. 2012, pp.
717–726.
13. C. Houston, S. Gooberman-Hill, R. Mathie, A.
Kennedy, Y. Li, and P. Baiz, “Case study for the return on
investment of internet of things using agent-basedmodelling and data science,” Systems, vol. 5, no. 1, Mar.
2017.

Ms. Jia Wei Tan
She is currently pursuing Bachelor
of Engineering Communicationand Electronics with Honors asfinal year student in Faculty ofEngineering, Technology and
Built
Environment,
UCSI
University, Malaysia. Her researchinterests are communication and antenna design.
Dr. Sew Sun Tiang
She is an Assistant Professor in
Faculty of Engineering at UCSI
University in Malaysia. She
received her PhD in Electrical andElectronic
Engineering
fromUniversiti Sains Malaysia in 2014.
Her
research
interests
areoptimization and antenna design.
Dr. Kim Soon Choong
He obtained his B.Eng., M.S., andPh.D. degrees in electrical andelectronicengineering
fromUniversiti Kebangsaan Malaysia in
2012, 2015, and 2019, respectively.
He currently serves as an AssistantProfessor in the Department of Computer Engineering(Artificial Intelligence) within the Faculty ofEngineering Technology and Built Environment at
UCSI Kuala Lumpur. His research focuses on areas
such as medical signal processing andinstrumentation.
Dr. Mohammad Arif Bin Ilyas
He received his PhD in ElectricalEngineering from Universiti Tun
Hussein Onn Malaysia in 2020. He
also earned his bachelor’s degree inelectronic engineering at the same
university. Currently, he is a lecturer
in the Faculty of Engineering,Technology, and Built Environment at UCSI
University. His main areas of research interest aremachine learning, artificial intelligence, etc.

Authors Introduction
Mr. Wei Kang Lai
He is currently pursuing Bachelorof Electrical and Electronics
Engineering as final year student
in Faculty of Engineering,Technology
and
Build
Environment, UCSI University,
Malaysia. His research interests
are electrical and solar systemdesign.
Ms. Zhi Ying Yeoh
She is currently pursuing Bachelor
of Engineering Electrical andElectronics with Honors as final
year student in Faculty ofEngineering, Technology and
Built
Environment,
UCSI
University, Malaysia. Her researchinterests are electrical and powerelectronics.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

856

A Design of Dual-Band Coplanar Waveguide (CPW) Printed Antenna for 1.9-3.6GHz
Applications
Jia Wei Tan1, Sew Sun Tiang1, *, Kim Soon Choong1, Mohammad Arif Bin Ilyas1, Mastaneh Mokayef1, Zhi Ying Yeoh1, Wei
Kang Lai1, Wei Hong Lim1

E-mail: 1001954282@ucsiuniversity.edu.my, tiangss@ucsiuniversity.edu.my, ChongKS@ucsiuniversity.edu.my,mohammadarif@ucsiuniversity.edu.my, mastaneh@ucsiuniversity.edu.my, 1001953809@ucsiuniversity.edu.my,100953677@ucsiuniversity.edu.my, limwh@ucsiuniversity.edu.my(1UCSI University, Malaysia)* Corresponding author
Abstract
A dual-band coplanar waveguide (CPW) printed antenna for IoT application is proposed, mounted on a low-profile
RT/Duroid 5880 substrate with dielectric constant of 2.2, loss tangent of 0.0009 and a standard height of 0.787mm.
This design aims to cover the major frequency bands from LTE to Bluetooth/Wi-Fi band/WiMax/Zigbee, Extended
IMT and 5G whereby the bandwidth range between 1.7 GHz to 3.6 GHz. The antenna is miniaturized through the
CPW technique and has a rectangular size of 60 × 30 × 1.187 mm3 . The design and simulation of the result records
a return loss of -25.12dB, peak gain of 4.23dBi, voltage standing wave ratio (VSWR) close to 1, omnidirectionalradiation, and current distribution. Radiation efficiency reaches approximately 94%, with a total efficiency of 89.4%
between 1.9GHz-3.6GHz.
Keywords: Dual-band CPW, Internet of Things, impedance bandwidth, reflection coefficient, radiation pattern,current distribution

1. Introduction

presence of the additional strip can support even or odd
quasi-TEM mode at low frequencies as well as TE mode
at high frequencies [3]. The modification of thedimensions of the signal strip or ground plane such as
thickness, length and width play a vital role indetermining the characteristics of a CPW-fed antenna [2].
There are a few main challenges in designing a dualband CPW antenna which include miniaturization, large
gain, high bandwidth coverage, low return loss,
impedance matching, power, and radiation efficiency.
Some of the techniques introduced to satisfy theabovementioned criteria includes the usage ofmetamaterials and complementary split ring resonator(CSRR) to increase gain or implementation of a magneto
electric (ME) to achieve wider impedance bandwidth,
higher gain, and compatible radiation patterns on bothelectric and magnetic plane [3].
Hence, in this paper, the design of this waveguide isdone by utilizing the CST software which involves
careful selection of substrate properties, structure of
CPW, and feed network. The simulation result allows
optimization and evaluation of antenna performance overa desired frequency range. The proposed antennapresents a dual-band coplanar waveguide using copperWith increasing world population and the incorporationof Internet of Things (IoT) in multiple computer
networks, microelectronics and modern communication,
there are higher demands for better, flexible, and advanceantenna module to support the embedded systems [1].
Antennas play a primary role in supporting such systems
because they enable long-distance communication andefficient use of frequency spectrum that allows
maximization of data transmitted over a given frequency
band. An example is the dual-band antenna which isdesigned to support two frequencies that can cover
multiband applications and provide sufficient bandwidth
for the operating frequencies as well as reduced
interference by neighbor devices.
In recent years, coplanar waveguide (CPW) has
garnered immense attention across the globe especiallyin modern wireless communication systems due to theirsimple integration with microwave integrated circuits,
low radiation loss and reduced dispersion as compared tomicrostrip antenna [2]. This type of waveguide has anadditional third conductor centered in the slot region andspecifically useful in developing active circuitry. The©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
857

Jia Wei Tan, Sew Sun Tiang, Kim Soon Choong, Mohammad Arif bin Ilyas, Mastaneh Mokayef, Zhi Ying Yeoh, Wei Kang Lai, Wei Hong Lim

conductor and RT/Duroid 5880 as substrate material. The
width of the microstrip and gap between the sides of theground is adjusted to obtained impedance matching of 50
Ohm. The main consideration of the paper is to develop
a design that operates on all major frequency bands fromLTE (1900MHz) to Extended LTE (2100MHz),
Bluetooth/Wi-Fi band/WiMAX/Zigbee (2.4GHz) and 5G(3.4-3.6GHz). Table 1 shows the summary of comparison
between existing CPW-fed antennas and proposed work.

Table 1. Summary Of Comparison Between Existing CPW-Fed Antennas and Proposed Work.
References

Feeding
Technique

[2]

CPW

[3]
[4]

CPW
CPW

[5]

CPW

[6]
[7]
[8]

CPW
CPW
CPW

[9]
[10]

CPW
CPW

Proposedwork

CPW

Operating
Frequency(GHz)2.54-2.69/
5.68-6.05
2.9-21.0
1.10-2.70/
3.15-3.65
3.24-8.29/
9.12-11.25
1.28-4.50
5.15-7.29
2.45/
4.0-6.0
2.19-2.51
3.04-10.70/
15.18-18.0
1.90-3.60

Gain (dBi)

Dimension(mm𝟑 )

Substrate Type

5.30
3.44
8.90

18.2 × 20 × 1.6

FR-4

23.0 ×10 ×0.8
35.0 × 25 × 1.6

FR-4
FR4

4.33
4.90
3.5
2.25
0.38
0.765
> 4.2
3.94

24.90 × 20 × 1.6

FR-4

73 × 65 × 1.52
20 × 8.7 × 0.4
30 × 30 × 2

Rogers RO4232
FR-4
Glass

46.5 × 29.0 × 0.76
47 × 25 × 0.135

Rogers RT6002
PET

4.23

60 × 30 × 1.187

RT/Duroid 5880

2. Related Works

The material chosen for the antenna patch is copper(annealed) because this form of copper is much softer andmalleable which makes it easier to shape during the
fabrication stage. On the other hand, the dielectric
substance used is the RT/duroid 5880 with dielectric
constant of 2.2 and tangent loss of 0.0009. Some key
features according to the datasheet of this substratematerial include low moisture absorption, resistancetowards chemicals, isotropic and uniform chemical
properties over frequency. It is mostly found in
microstrip and stripline circuits, as well as point-to-point
digital radio antennas. The thickness of the dual-band
CPW is 1.187mm with copper (annealed) 0.4mm andsubstrate as 0.787mm respectively.
The detailed geometry of the proposed antenna isshown in Fig. 2. Some of the parameters such as length
and width of the main rectangular patch were taken fromthe values by S. Das, H. Islam, T. and Bose, N. Gupta [7]and is progressively modified to fit the requirements of
our antenna. The characteristic impedance of the feedline can be calculated with a certain formula, however, in
this paper, we considered utilizing the coplanar
waveguide calculator found online which requires value
including substrate thickness, width of antenna strip,
ground plane spacing, and dieletric substrate [11]. We ran
a couple of trials and errors to achieve matching
impedance of 50 Ohm and the result was to adjust

The configuration of the proposed antenna shown in Fig.
1 is inspired by the basic design from S. Das, H. Islam, T.
and Bose, N. Gupta [7]. The fundamental structure of theantenna consists of two layers which are antenna layer on
top and dielectric substrate at the bottom. Our design
assumes no ground plate is placed underneath thedielectric which means it is of free space or air.

a. Front View

b. Bottom View

Fig 1. Structure of proposed antenna design.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
858

A Design of Dual-Bandantenna width to 2mm while spacing between the edge of
feed element and ground plane is 0.1933mm. The
calculator equations are based on the ‘Coplanar
Waveguide Circuits, Components, and Systems’
textbook written by R. N. Simons [12].
The equations applied in the online calculator are justan approximation of the CPW impedance as it does notconsider the thickness of antenna strips, hence may notdisplay the most accurate results. When creating criticaldesign, the usage of 3D electro-magnetic analysis of

CPW needs to be considered. The gap between the
ground plates and antenna feed line is calculated to be
0.193mm which is less than half of the substrate
thickness to control leakage of electromagnetic energy infree space. The antenna design is gradually modified tofulfil the required frequency range and the evolution of
the proposed antenna design is shown in Fig. 3. The
detailed antenna parameters of the final design are
depicted in Table 2.

Fig 2. Geometry of proposed antenna.
Table 2. Proposed CPW Antenna Design Parameters Value.
Parameters
Value(mm)Parameters
Value(mm)Parameters
Value(mm)

L
60
L11
39.0
W5
3.0

L1
10
L12
27.0
W6
8.52

L2
13.5
L13
0.5
W7
5.5

L3
2.5
L14
6.5
W8
3.5

L4
1.0
R1
2.0
W9
7.0

L5
9.0
R2
1.5
W10
10.7

L6
11.75
W
40
W11
1.0

L7
13.5
W1
8.5
W12
4.11

L8
5.0
W2
6.5
W13
13.41

L9
12
W3
8.5
W14
2.0

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
859

L10
33.0
W4
3.0
W15
0.5

Jia Wei Tan, Sew Sun Tiang, Kim Soon Choong, Mohammad Arif bin Ilyas, Mastaneh Mokayef, Zhi Ying Yeoh, Wei Kang Lai, Wei Hong Lim

a. Antenna 1

b. Antenna 2c. Antenna 3d. Antenna 4

Fig 3. Evolution process of antenna.
3. Result and Discussion

More detailed parametric studies need to be done such as
modifying the major length and width of antenna anddieletric substrate, positions of rectangular strips,
separation distance between feed line and ground plates
to achieve wider frequency spectrum. Additional tuning
and optimization are also needed to attain the appropriate
frequency response.

The different layout of CPW antennas were designed andsimulated to determine the antenna performance indifferent aspects including S11-parameter (return loss),reflection coefficients, gain, radiation pattern, current
distribution, , and voltage standing wave ratio (VSWR).
The S11 parameter graph shows the return loss of theantenna design which is a measure of the reflection
coefficient. The simulated result for return loss from thefirst to the final proposed antenna is displayed in Fig. 4.
The first antenna result (red line) shows a narrow
impedance bandwidth less than 30% (2.0-2.6GHz) for
S11 < -10dB threshold. Besides, the return loss value isfairly small, which is -13.88dB and this may cause more
power reflected back to the system which is not desirable.
This leads to our second antenna (green line) which
is designed to increase the return loss as can be seen in
Fig. 4. By adding two rectangular patches with 1mm
width on each side of the ground plate as shown in Fig.
3 (b), the return loss improved to -21.47dB.
Moving on, the goal of antenna 3 (blue line) is toincrease the impedance bandwidth to fulfill the criteria
for IoT applications which covers all major frequency
bands between 1.7GHz to 3.6GHz. By changing thelength between the newly added rectangular strips withthe ground plate, the bandwidth coverage improved fromthe initial antenna design which is 2.0GHz-2.6GHz to
1.97GHz-3.58GHz with more than 80% of the desired
spectrum. This design also produces two resonant
frequencies at 2.215GHz and 3.28GHz.
Lastly, the addition of circular slots on the corner ofthe rectangular patches further decreases the return loss
to -25.12dB (orange line) and improved the impedance
bandwidth to 1.88GHz-3.62GHz. However, one of the
major frequency spectrums failed to be covered by thisdesign, which is the LTE with frequency of 1.7GHz.

Fig 4. Simulation result of S11 parameter (return loss)
for all four antennas.
The gain for final proposed antenna is shown in Fig 5
with the peak gain captured at 4.23dBi when theoperating frequency is around 3.6GHz. At resonant
frequencies of 2.215GHz and 3.28GHz, the gain
observed is 2.59dBi and 3.92dBi respectively.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
860

A Design of Dual-Band

Fig 5. Simulation result of gain for final proposed
antenna.
c. Far field radiation pattern at 2.215GHz.

The radiation pattern of the final proposed antenna at
2.215GHz and 3.28GHz frequencies viewed in 3D isshown in Fig. 6 (a) and (b). Based on the results, theantenna design is omnidirectional, or commonly referred
to as ‘omni’ whereby its shape is similar to a bagel. This
type of antenna radiates and receives signal in all
directions equally. Along its axis, this antenna evenly
emits electromagnetic radiation in all directions. An
omnidirectional antenna has the benefit of being able tosend and receive signals from any direction without therequirement for exact targeting. The directivity recorded
at 2.2GHz and 3.28GHz is 2.665dBi and 3.28dBi
respectively. The radiation pattern in E-field and H-field
can also be viewed in 2D for operating frequency at
2.215GHz and 3.28GHz as shown in Fig. 6 (c) and (d).

d. Far field radiation pattern at 3.28GHz.
Fig 6. Radiation pattern of final proposed antenna.
The surface current distribution is measured at bothresonant frequencies of 2.215GHz and 3.28GHz as
shown in Fig 7 (a) and (b). In both figures, the current
density is most evident at the antenna strip denoted by
red and orange color map which has higher potential for
losses and prone to heating easily due to high dissipationof electromagnetic fields and energy. Some methods toreduce these losses include having proper grounding andoptimizing the feed network as well as geometry design.
The rest of the areas that are mostly drawn in blue orgreen area experience low current density whereby itindicates that the performance of the antenna still has
room for improvement.

a. 3D radiation pattern at 2.215GHz.

b. 3D radiation pattern at 3.28GHz.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
861

Jia Wei Tan, Sew Sun Tiang, Kim Soon Choong, Mohammad Arif bin Ilyas, Mastaneh Mokayef, Zhi Ying Yeoh, Wei Kang Lai, Wei Hong Lim

Based on Fig. 10, efficiencies are recorded in terms ofmagnitude (dB). The highest radiation efficiency andtotal efficiency converted into percentage are 98% and92.4% respectively at 2.35GHz. The radiation and total
efficiency for resonant frequencies of 2.2GHz and3.28GHz are mostly above 90% and values are tabulated
in Table 3.
Table 3. Radiation efficiency and total efficiency ofdifferent resonant frequencies.
Resonant
Radiation
Total
Frequency
efficiency (%)
efficiency (%)
(GHz)
2.2
92.2%
86.6%
3.28
94.6%
90%

a. Current distribution at 2.215GHz.

b. Current distribution at 3.28GHz.
Fig 7. Current distribution on final proposed antenna.
Fig 10. Radiation efficiency and total efficiency ofdual-band CPW.
4. Conclusion
In this proposed paper, the rectangular dual-band CPW
antenna structure with RT/duroid 5880 as substrate thathas standard height of 0.787mm and dieletric constant of
2.2 is designed and simulated using CST software. The
design can be classified for the usage on the applicationsof Internet of Things as it can satisfy most of the major
frequency bands from LTE to Bluetooth/Wi-Fi
band/WiMax/Zigbee, Extended IMT and 5G. However,
it is also worth mentioning that the proposed design fails
to cover a lower bandwidth spectrum of LTE which is
1.7GHz and this can be due to various factors such as
dimensions, size, and layout of the antenna design aswell as substrate material. The final antenna design has
a wide impedance bandwidth of 1.9GHz-3.6GHz,
improve in overall antenna gain with highest peak value
at 4.23dBi in the two working bands and VSWR
approximately close to 1. The final antenna design
demonstrates
monopole
radiation
patterns(omnidirectional) characteristics and high radiation
efficiency up to 98% between the stated impedance
bandwidths. The proposed antenna also exhibits
moderate return loss of -29.54dB with an acceptable sizeof 1800m2 and total efficiency of approximately 92.4%
which makes it a comparable candidate for portablehandheld communication devices.

Fig 8. Smith chart of the final proposed antenna.
Smith chart is shown in Fig. 8. The VSWR value plottedat 2.215GHz is 1.11 and at 3.28GHz is 1.38 as shown in
Fig. 9 whereby both values can be considered close to 1.
This value of VSWR indicates a well-matched antenna
with the transmission line and also reduces power loss
due to impedance mismatch.

Fig 9. VSWR graph of the final proposed antenna.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
862

A Design of Dual-Band

Authors Introduction

Acknowledgements
Ms. Jia Wei Tan

This work was supported by the UCSI University
Research Excellence & Innovation Grant (REIG) withproject code of REIG-FETBE-2022/038.

She is currently pursuing Bachelor ofEngineering
Communication
andElectronics with Honors as final year
student in Faculty of Engineering,Technology and Built Environment,
UCSI University, Malaysia. Her researchinterests are communication and antenna

References
D.M Pozar, Microwave Engineering, 4th Edition, UnitedStates of America: John Wiley & Sons Inc, 2012.
[2] R. Vivek, S. Sreenath, P.V. Vinesh, and K. Vasudevan,“Coplanar waveguide (CPW)-fed compact dual band
antenna for 2.5/5.7GHz applications,” Progress In
Electromagnetics Research M, vol. 74, pp. 51-59, 2018.
[3] P. Ranjan, A. Maurya, H. Gupta, S. Yadav, and A. Sharma,“Ultra-wideband CPW fed band-notched monopole
antenna optimization using machine learning,” Progressin Electromagnetics Research M, vol. 108, pp, 27-38,
2022.
[4] Q. Awais, H.T. Chattha, M. Jamil, Y. Jin, F.A. Tahir, and
M.U. Rehman, “Novel dual ultrawideband CPW-fed
printed antenna for interent of things (IoT) applications”,
Wireless Communications and Mobile Computing, vol.
2018.
[5] M. M. Alam, R. Azim, N. M. Sobahi, A. I. and Khan, and
M. T. Islam, “A dual-band CPW-fed miniature planarantenna for S-, C-, WiMAX, WLAN, UWBm and X-band
applications,” Scientific Reports, vol. 12, 2022.
[6] J. Kulkarni, and C.Y.D. Sim, “Wideband CPW-fed ovalshaped monopole antenna for Wi-Fi 5 and Wi-Fi 6
applications”, Progress in Electromagnetic Research C,
vol. 107, pp. 173-182, 2021.
[7] S. Das, H. Islam, T. and Bose, N. Gupta, “Ultra wide band
CPW-fed circularly polarized microstrip antenna forwearable
applications”,
Wireless
Personal
Communications, vol. 108, pp. 87-106, 2019.
[8] N. Morales-Centla, R.Torrealba-Melendez, E. I. TamarizFlores, M. Lopez-Lopez, C.A Arriaga-Arriaga, J.M.
Munoz-Pacheco, and V. R. Gonzalez-Diaz, “Dual-band
CPW graphene antenna for smart cities and IoTapplications,” MDPI, 2022.
[9] S. Ahmad, N. Cherif, S. Naseer, U. Ijaz, Y.S. Faouri, A.
Ghaffar, and M. Hussein, “A wideband circularly
polarized CPW-fed substrate integrated waveguide based
antenna array for ISM band applications,” Elsevier Ltd,2022.
[10] S.G.Kirtania,B.A. Younes, A. R. Hossain, T. Karacolak,
and P.K. Sekhar, “CPW-fed flexible ultra-wideband
antenna for IoT applications,” Micromachines, vol. 12, pp.
453, 2021.
[11] Coplanar
Waveguide
Calculator.
everythingRF.
Microwave101.com.
https://www.microwaves101.com/calculators/864coplanar-waveguide-calculator[12] R. N. Simons, Coplanar Waveguide Circuits, Components,
and Systems. John Wiley & Sons Inc, pp. 15-21, 2001.
[1]

design.
Dr. Sew Sun Tiang
She is an Assistant Professor in Faculty ofEngineering at UCSI University in
Malaysia. She received her PhD in
Electrical and Electronic Engineeringfrom Universiti Sains Malaysia in 2014.
Her research interests are optimization
and antenna design.
Dr. Kim Soon Choong
He obtained his B.Eng., M.S., and Ph.D.
degrees in electrical and electronicengineering from Universiti Kebangsaan
Malaysia in 2012, 2015, and 2019,respectively. He currently serves as an
Assistant Professor in the Department ofComputerEngineering(Artificial
Intelligence) within the Faculty of Engineering Technology
and Built Environment at UCSI Kuala Lumpur. His research
focuses on areas such as medical signal processing andinstrumentation.
Dr. Mohammad Arif Bin Ilyas
He received his PhD in ElectricalEngineering from Universiti Tun Hussein
Onn Malaysia in 2020. He also earned hisbachelor’s
degreeinelectronicengineering at the same university.
Currently, he is a lecturer in the Faculty ofEngineering, Technology, and Built
Environment at UCSI University. His main areas of researchinterest are machine learning, artificial intelligence, etc.
Dr. Mastaneh Mokayef
She has received her PhD from WirelessCommunication Centre Faculty ofElectrical Engineering in University
Technology Malaysia (UTM) in 2014. She
has also obtained her master’s degree fromthe faculty of engineering in 2009 from the
University Technology Malaysia. Her
research interests include Wireless communications,spectrum sharing method, spectrum management, cellularcommunication systems and Antenna design.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
863

Powered by TCPDF (www.tcpdf.org)

Jia Wei Tan, Sew Sun Tiang, Kim Soon Choong, Mohammad Arif bin Ilyas, Mastaneh Mokayef, Zhi Ying Yeoh, Wei Kang Lai, Wei Hong Lim

Ms. Zhi Ying Yeoh
She is currently pursuing Bachelor ofEngineering Electrical and Electronics
with Honours as final year student inFaculty of Engineering, Technology and
Built Environment, UCSI University,
Malaysia. Her research interests are
power electronics and simulation

Mr. Wei Kang Lai
He is currently pursuing Bachelor of
Electrical and Electronics Engineering as
final year student in Faculty ofEngineering, Technology and Build
Environment, UCSI University, Malaysia.
His research interests are electrical andsolar system design.
Dr. Wei Hong Lim
He is an Assistant Professor in Faculty ofEngineering at UCSI University in
Malaysia. He received his PhD in
Computational
Intelligence
fromUniversiti Sains Malaysia in 2014. His
research interests are optimization andartificial intelligence.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
864

Optimized Convolutional Neural Network Towards Effective Wafer Defects Classification
Koon Hian Ang1, Koon Meng Ang1, Chin Hong Wong2,3, Abhishek Sharma4, Chun Kit Ang1, Kim Soon Chong1, Sew Sun
Tiang1*, Wei Hong Lim1*
1

Faculty of Engineering, Technology and Built Environment, UCSI University, Kuala Lumpur 56000, Malaysia
2
Maynooth International Engineering College, Maynooth University, Maynooth, Co Kildare, Ireland
3
Maynooth International Engineering College, Fuzhou University, Fujian, 350116, China
4
Department of Computer Science and Engineering, Graphic Era Deemed to be University, Dehradun 248002, India
Email: 1001850063@ucsiuniversity.edu.my, 1001436889@ucsiuniversity.edu.my, chinhong.wong@mu.ie,
abhishek15491@gmail.com, angck@ucsiuniversity.edu.my, ChongKS@ucsiuniversity.edu.my,tiangss@ucsiuniversity.edu.my, limwh@ucsiuniversity.edu.my

Abstract
Semiconductor defect inspection is crucial for yield improvement but is hindered by manual inspection's subjectivity
and error. This paper employs Convolutional Neural Networks (CNNs) for automated wafer defect classification,
addressing the challenges of time-intensive training and complex hyperparameter tuning. We propose the Arithmetic
Optimization Algorithm (AOA) to efficiently optimize CNN hyperparameters like momentum, initial learning rate,
maximum epochs, and L2 regularization. Our method reduces the trial-and-error in hyperparameter tuning. Using the
AOA-optimized ResNet-18 model, our simulations show superior performance in defect classification compared tothe unoptimized model, demonstrating its effectiveness and practical potential.
Keywords: Arithmetic optimization algorithm, Convolutional neural networks, Hyperparameter optimization, Wafer
defect classification

classification [8], [9], owing to their capability to learn
the nonlinearity between inputs and outputs byautomatically extracting relevant information from raw
data. Pre-trained CNN architectures like GoogleNet [10].
AlexNet [11], VGG [12], and ResNet [13] have beensuccessfully adapted using transfer learning to tackle newtasks, benefiting from reduced data and training timerequirements. The efficacy of these pre-trained networks
in addressing new tasks through transfer learning process
is heavily influenced by the hyperparameter settings
employed during their training phase.
In conventional practice, the fine-tuning of CNN
hyperparameters relies on a labor-intensive and timeconsuming trial-and-error approach. To streamline thisprocess, metaheuristic search algorithms (MSAs),
drawing inspiration from various natural phenomena[14]—including swarm intelligence, natural evolution,
physics, mathematical principles, and human activities—
have been developed. These algorithms are effective in
addressing complex problems [15], [16], [17], [18], [19],[20], [21], [22] and are particularly adept at
hyperparameter tuning in CNNs [23], thanks to theirrobust global search capabilities. One such algorithm is
the Arithmetic Optimization Algorithm (AOA) [24],which derives its strategy from the distribution behaviors
of basic arithmetic operations such as addition,
subtraction, multiplication, and division.
In this study, we employ a pre-trained network, ResNet18, retrained on new datasets using transfer learning for

1. Introduction
The advent of the fourth industrial revolution (IR4.0)
has escalated the need for semiconductor industries to
produce increasingly complex integrated circuit chips.
This complexity arises from the need to pattern and etchmore components on semiconductor wafers, catering todiverse chip specifications like lifespan, size, memorystorage, and access speed. Consequently, this escalationin production demands heightens the likelihood ofprocess-induced defects on wafer surfaces, adversely
affecting the manufacturing yield. A key step in
mitigating this yield reduction involves the identificationand classification of wafer defect patterns, which often
correlate with various manufacturing stages, including
contamination, robot handoff, and flow leakages.
Accurate identification of these defects enables engineers
to pinpoint and rectify the underlying issues, thereby
enhancing chip production yield [1].
Most semiconductor industries currently depend on
manual visual inspection for detecting wafer defects, amethod plagued by subjectivity and a high risk oferroneous classifications due to long-term fatigue. To
address these limitations, there is a growing need for an
automated machine vision system, integrated with an
optimized deep learning model, for reliable wafer defectclassification. Convolutional Neural Networks (CNNs)
are a widely used deep learning technique [2], [3], [4], [5],[6], [7]. particularly effective in wafer defect©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
865

Koon Hian Ang, Koon Meng Ang, Chin Hong Wong, Abhishek Sharma, Chun Kit Ang, Kim Soon Chong, Sew Sun Tiang, Wei Hong Lim

the purpose of wafer defect classification. The AOA is
utilized to optimize four critical hyperparameters of the
CNN: momentum, initial learning rate, maximum epochs,
and L2 regularization. The efficacy of this optimized
CNN model in classifying wafer defects is then assessedand benchmarked against the performance of acorresponding unoptimized CNN model.

In the initialization phase of the AOA, a set of potential
solutions is generated, each falling within predefined
dimensional boundaries, to address specific optimization
problems. Subsequently, the algorithm calculates the
Math Optimizer Accelerated (MOA) value, which
dictates the search strategy between exploration andexploitation phases. The MOA function ismathematically expressed as:
𝑀𝑎𝑥 − 𝑀𝑖𝑛
𝑀𝑂𝐴(𝐶𝐼𝑡𝑒𝑟 ) = 𝑀𝑖𝑛 + 𝐶𝐼𝑡𝑒𝑟  (
)
(1)
𝑀𝐼𝑡𝑒𝑟

2. Related Works
2.1. Conventional CNN and ResNet-18

where 𝑪𝑰𝒕𝒆𝒓 denotes the current iteration number, 𝑴𝑰𝒕𝒆𝒓 isthe maximum number of iterations, and Min and Max are
the lower and upper limits of the accelerated function,
respectively. The AOA enters the exploration phase if the
MOA value is lower than a randomly generated number
between 0 and 1; otherwise, it proceeds to the
exploitation phase.
During the exploration phase of the AOA, both
Multiplication and Division operators are employed toenhance the search space coverage and introduce adiverse array of solutions. This approach leverages theirpotential for high dispersion and distributed values. The
position of each i-th AOA solution in the d-th dimension
is updated as follows:

A typical CNN architecture comprises threefundamental components: convolutional layers (Conv)
for feature extraction, pooling (Pool) layers for reducing
the size of feature maps, and fully connected (FC) layers
for classification. Table 1 illustrates an example of astandard CNN model.
Table 1. Architecture of a typical CNN.
Layer
1
2
3
4
5
6
7

Layer
Type
Input
Conv 1Pool 1
Conv 2Pool 2
FC 1
FC 2#Feature
Maps
1
6
6
16
16
1
1

Feature Map
Size
14×14
7×7
4×4
4×4
2×2
120
84

Filter
Size
5×5
2 ×2
5×5
2×2

𝑥𝑖,𝑗 (𝐶𝐼𝑡𝑒𝑟 + 1)
𝑥𝑗𝑏𝑒𝑠𝑡 ÷ (𝑀𝑂𝑃 + 𝜀) × ((𝑈𝐵𝑗 − 𝐿𝐵𝑗 ) × 𝜇 + 𝐿𝐵𝑗 ) ,𝑟1 < 0.5
𝑥𝑗𝑏𝑒𝑠𝑡 × 𝑀𝑂𝑃 × ((𝑈𝐵𝑗 − 𝐿𝐵𝑗 ) × 𝜇 + 𝐿𝐵𝑗 ) ,=He et al. [13] introduced ResNet models, known for
their deep architectures, which have exhibited
remarkable convergence and high accuracy. In 2015,
ResNet models achieved first place in the ImageNet
Large Scale Visual Recognition Challenge (ILSVRC)
and the Common Objects in Context (COCO)
classification challenge. ResNet is characterized bymultiple stacked residual units and comes in various
configurations, including 18, 34, 50, 101, 152, and 1202
layers, each with specific operations tailored to its
architecture. ResNet-18, a model offering an optimalbalance between depth and performance, comprises fiveconvolutional layers, one average pooling layer, and afully connected layer with SoftMax activation. ResNet50 includes 49 convolutional layers, culminating in a
fully connected layer. Considering the balance between
computational efficiency and performance, ResNet-18 ischosen for this study.

{

otherwise(2)

where 𝑥𝑖,𝑗 (𝐶𝐼𝑡𝑒𝑟 + 1) denotes the j-th dimension of the ith solution in the next iteration (𝐶𝐼𝑡𝑒𝑟 + 1), with j ranging
from 1 to D, i ranging from 1 to I, and I is the population
size; 𝐶𝐼𝑡𝑒𝑟 ranges from 1 to 𝑀𝐼𝑡𝑒𝑟 , the latter is thealgorithm's maximum iteration number; 𝑥𝑗𝑏𝑒𝑠𝑡 is the j-thdimension of the best solution; 𝑈𝐵𝑗 and 𝐿𝐵𝑗 are the upperand lower boundaries in the j-th dimension; 𝑟1 is arandom number between 0 and 1; 𝜇 adjusts the searchrange. Define the Math Optimizer Probability (MOP) as:1

𝑀𝑂𝑃(𝐶𝐼𝑡𝑒𝑟 ) = 1 −𝐶𝐼𝑡𝑒𝑟 𝛼(3)1𝑀𝐼𝑡𝑒𝑟 𝛼where 𝑀𝑂𝑃(𝐶𝐼𝑡𝑒𝑟 ) represents the function value at
iteration 𝐶𝐼𝑡𝑒𝑟 , and α controls the exploitation accuracy.
In the exploitation phase of the AOA, Subtraction and
Addition operators are utilized to precisely target the
search regions identified by the most promising AOA
solution. This step leverages the operator's characteristicof being densely concentrated yet having low dispersion.
The position of each i-th AOA solution in the d-th
dimension is updated as follows during this phase:

2.2. Arithmetic optimization algorithm (AOA)
In 2021, Abualigah et al. [24] introduced the AOA, a
MSA inspired by the characteristics of the four basicarithmetic
operations:addition,subtraction,
multiplication, and division. The AOA operates throughthree primary phases: initialization, exploration, andexploitation, strategically navigating the solution space toaddress optimization challenges effectively.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
866

Optimized Convolutional Neural Network

𝑥𝑖,𝑗 (𝐶𝐼𝑡𝑒𝑟 + 1)

Additionally, to enhance ResNet-18's performance inclassifying wafer defects, the AOA is implemented to
optimize the network's training hyperparameters during
the transfer learning process. The AOA generates
solution vectors representing four key decision variables:
momentum, initial learning rate, maximum epochs, andL2 regularization. The ranges for these variables are
specified in Table 3. The effectiveness of each AOA
solution is assessed based on the classification accuracy
achieved. Fig. 1 illustrates the AOA-optimized ResNet18 framework developed for wafer defect classification.

𝑥𝑗𝑏𝑒𝑠𝑡 − 𝑀𝑂𝑃 × ((𝑈𝐵𝑗 − 𝐿𝐵𝑗 ) × 𝜇 + 𝐿𝐵𝑗 ) ,𝑟2 < 0.5
𝑥𝑗𝑏𝑒𝑠𝑡 + 𝑀𝑂𝑃 × ((𝑈𝐵𝑗 − 𝐿𝐵𝑗 ) × 𝜇 + 𝐿𝐵𝑗 ) ,={otherwise(4)Here, 𝒓𝟐 represents a randomly generated number
between 0 and 1, following a uniform distribution.
The AOA algorithm iteratively executes these
exploration and exploitation processes until predefined
stopping criteria are met. The optimal solution generated
by AOA at the end of the optimization process is adopted
as the best set of hyperparameters for training the ResNet18 network to classify wafer defects.

Table 3. Lower and upper bounds of training
hyperparameters to be optimized.
Hyperparameter
Lower Limits Upper Limits
Momentum
0.5
0.9
Initial Learning Rate
0.01
0.1
Maximum Epochs
5
10
L2 Regularization
1 × 10−4
5 × 10−4

3. Proposed Optimized Deep Learning Model
3.1. Preprocessing of dataset with wafer defects

AOA-Optimized ResNet-18 for Wafer Defect Classification
Input: N, D, UBj, LBj
01:
Initialize 𝐶𝐼𝑡𝑒𝑟 = 0;02:
for i =1 to I do
03:
Randomly generate solution 𝑥𝑖 ;04:
𝐶𝐼𝑡𝑒𝑟 = 𝐶𝐼𝑡𝑒𝑟 + 1;
05:
end for06:
while 𝐶𝐼𝑡𝑒𝑟 ≤ 𝑀𝐼𝑡𝑒𝑟 do
07:
Decode the hyperparameters from 𝑥𝑖 ;08:
Train the ResNet-18 using transfer learning based on
hyperparameters decoded from 𝑥𝑖 ;09:
Evaluate the accuracy 𝑓(𝑥𝑖 ) of ResNet-18;
10:Calculate MOP using Eq. (3);11:Calculate MOA using Eq. (1);12:
for i =1 to I do
13:
for j =1 to D do
14:
if rand > MOA then15:
Update 𝑥𝑖,𝑗 (𝐶𝐼𝑡𝑒𝑟 + 1) with Eq. (2);
16:
Else
17:
Update 𝑥𝑖,𝑗 (𝐶𝐼𝑡𝑒𝑟 + 1) with Eq. (4)18:
end if
19:
end for20:
Decode the hyperparameters from 𝑥𝑖 ;21
Train the ResNet-18 using transfer learning
based on hyperparameters decoded from 𝑥𝑖 ;22:
Evaluate the accuracy 𝑓(𝑥𝑖 ) of ResNet-18;
21:
Update 𝑥𝑖 , 𝑓(𝑥𝑖 ), 𝑥 𝑏𝑒𝑠𝑡 , 𝑓(𝑥 𝑏𝑒𝑠𝑡 )22:
𝐶𝐼𝑡𝑒𝑟 ← 𝐶𝐼𝑡𝑒𝑟 + 1;
23:
end for24:
end while
Output: Optimal values of momentum, initial learning rate,
maximum epochs and L2 regularization decoded from the best
AOA solution represented as 𝑥 𝑏𝑒𝑠𝑡 .

The optimized ResNet-18 architecture undergoes
training and evaluation for wafer defect classification
utilizing the WM-811K dataset [25]. This particular
dataset is noted for its significantly imbalanced
distribution across various classes. To mitigate potential
biases and overfitting stemming from this imbalance, weformed a new dataset using an under-sampling method.
This approach involved compiling 30,000 images
classified as 'None defect' from the WM-811K dataset,along with all other images categorized as defects. The
resultant dataset, balanced in terms of defect and nondefect images, is detailed in Table 2, providing acomprehensive breakdown of the image categories.
Table 2. Number of defect and non-defect wafer
images after preprocessing stage
Layer
Defect Type#Labeled Images
1
Center
4,294
2
Donut
555
3
Edge-Loc
5,189
4
Edge-Ring
9,680
5
Loc
3,593
6
Near-Full
149
7
Random866
8
Scratch
1,193
9
None
30,0003.2. Hyperparameter tuning of ResNet-18 with AOA
The revised WM-811K dataset, detailed in Table 2, isused to train the chosen deep learning model, ResNet-18,
for wafer defect classification via transfer learning. To
tailor the pre-trained network for this specificclassification task, several modifications are made to boththe input datasets and the configuration of ResNet-18.
The size of each input image is increased from 48×48×1
to 224×224×3, the padding size of the initial convolution
layer is adjusted to align with the new input size, and theoutput size of the fully connected layer is altered to 9 toreflect the nine defect types identified in this study.

Fig.1 Pseudocode of optimizing the hyperparameters of
ResNet-18 using AOA for wafer defect classification.
4. Performance Studies
4.1. Simulation settings
The pre-processed WM-811K dataset is randomly
partitioned into three subsets for the purposes of training,validation, and testing. Specifically, the dataset allocation©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
867

Koon Hian Ang, Koon Meng Ang, Chin Hong Wong, Abhishek Sharma, Chun Kit Ang, Kim Soon Chong, Sew Sun Tiang, Wei Hong Lim

comprises 70% for training, 10% for validation, and 20%
for testing. Additionally, all input images are uniformly
resized to a dimension of 224×224×3, and the minimum
batch size for processing is set as 32.

of precision, the AOA-optimized model achieves a scoreof 0.9059, exceeding the unoptimized model's 0.8751.
The F1 score and AUC for the optimized model are
0.8972 and 0.9397, respectively, both higher than theunoptimized model's scores of 0.8523 and 0.9002. The
enhanced validation accuracy of the AOA-optimized
ResNet-18 indicates its superior performance inpredicting classes of unseen samples during training,
while its greater testing accuracy suggests a moreaccurate prediction of new, unseen data classes. The
improved recall and precision scores signify the model'sheightened ability to correctly identify more true positive
samples and effectively reduce false positives.

4.2. Performance comparisons
The classification effectiveness of both unoptimized
ResNet-18 and AOA-Optimized ResNet-18 in
identifying wafer defects is evaluated using several key
metrics: recall, accuracy, precision, F1 score, and AUC.
Consider TP, TN, FP, and FN as the true positive, true
negative, false positive, and false negative counts,
respectively, generated by the deep learning model
during the classification process. Recall, reflecting themodel's capability to detect positive samples, iscalculated as:
𝑇𝑃(5)
𝑇𝑃 + 𝐹𝑁
Accuracy, indicating the proportion of correctlypredicted data, is defined as:
𝑅𝑒𝑐𝑎𝑙𝑙 =𝑇𝑃 + 𝑇𝑁(6)
𝑇𝑃 + 𝑇𝑁 + 𝐹𝑃 + 𝐹𝑁Precision, measuring the model’s accuracy in predicting
a sample as positive, is expressed as:
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 =𝑇𝑃(7)𝑇𝑃 + 𝐹𝑃The F1 score, the harmonic mean of precision and recall,is calculated by:
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =(a)𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 × 𝑅𝑒𝑐𝑎𝑙𝑙(8)
𝐹1 𝑆𝑐𝑜𝑟𝑒 = 2 × ()
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑅𝑒𝑐𝑎𝑙𝑙
Lastly, AUC, representing the area under the receiver
operating characteristic curve, provides an overall
effectiveness measure. The quantitative performance ofboth the unoptimized and AOA-Optimized ResNet-18
models is detailed in Table 4. Additionally, their
qualitative performance is analyzed based on the
confusion matrices depicted in Fig. 2.
Table 4. Quantitative performance comparison
between optimized and unoptimized ResNet-18
Metrics
Unoptimized
AOA-Optimized
ResNet-18
ResNet-18
Validation Accuracy
0.85600.8983
Testing Accuracy
0.83880.8923
Recall
0.84820.8956
Precision
0.87510.9059F1 Score
0.85230.8972AUC
0.90020.9397(b)
Fig.2 Confusion matrices produced by: (a) Unoptimized
ResNet-18 and (b) AOA-Optimized ResNet-18 whenclassfying wafer defects.

Table 4 illustrates that the AOA-optimized ResNet-18
surpasses the unoptimized model in all assessed metrics.
It shows a higher validation accuracy of 0.8983 comparedto the unoptimized model's 0.8560, and a testing accuracyof 0.8923 versus 0.8388. Additionally, the recall score of
the AOA-optimized model stands at 0.8956,
outperforming the unoptimized model's 0.8420. In terms

Qualitative results, derived from confusion matrices ofboth the AOA-optimized and unoptimized ResNet-18
models as depicted in Fig. 2, align with the quantitative
findings in Table 4. The AOA-optimized model
demonstrates superior accuracy in classifying various©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
868

Optimized Convolutional Neural Network

3. M. Alrifaey et al., Hybrid deep learning model for fault
detection and classification of grid-connected photovoltaic
system, IEEE Access 10, 2022, pp. 13852-13869.
4. T. Berghout, L. H. Mouss, T. Bentrcia and M. Benbouzid, A
semi-supervised deep transfer learning approach for rollingelement bearing remaining useful life prediction, IEEE
Transactions on Energy Conversion 37(2), 2022, pp. 1200-1210.
5. O. Friha, M. A. Ferrag, M. Benbouzid, T. Berghout, B.
Kantarci and K.-K. R. Choo, 2DF-IDS: Decentralized anddifferentially private federated learning-based intrusion
detection system for industrial IoT, Computers & Security 127,
2023, pp. 103097.
6. T. Berghout, M. Benbouzid, Y. Amirat and G. Yao, Lithiumion battery state of health prediction with a robust collaborative
augmented hidden layer feedforward meural network approach,IEEE Transactions on Transportation Electrification 9(3), 2023,
pp. 4492-4502.
7. L. S. Chow, G. S. Tang, M. I. Solihin, N. M. Gowdh, N.
Ramli and K. Rahmat, Quantitative and qualitative analysis of
18 deep convolutional neural network (CNN) models withtransfer learning to diagnose COVID-19 on chest X-ray (CXR)
images, SN Computer Science 4(2), 2023, pp. 141.
8. U. Batool, M. I. Shapiai, M. Tahir, Z. H. Ismail, N. J. Zakaria
and A. Elfakharany, A systematic review of deep learning forsilicon wafer defect recognition, IEEE Access 9, 2021, pp.
116572-116593, 2021.
9. S. Chen, Y. Zhang, M. Yi, Y. Shang and P. Yang, AI
classification of wafer map defect patterns by using dualchannel convolutional neural network, Engineering Failure
Analysis 130, 2021, pp. 105756.
10. C. Szegedy et al., Going Deeper with Convolutions, 2015
IEEE Conference on Computer Vision and Pattern Recognition(CVPR), Boston, MA, USA, 2015, pp. 1-9.
11. A. Krizhevsky, I. Sutskever and G. Hinton, ImageNetclassification with deep convolutional neural networks, Neural
Information Processing Systems 25, 2012.
12. K. Simonyan and A. Zisserman, Very deep convolutional
networks for large-scale image recognition, arXiv 1409.1556,
2014.
13. K. He, X. Zhang, S. Ren and J. Sun, Deep Residual Learning
for Image Recognition, 2016 IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), Las Vegas, NY, USA,2016, pp. 770-778.
14. M. F. Ahmad, N. A. M. Isa, W. H. Lim and K. M. Ang,
Differential evolution: A recent review based on state-of-the-art
works, Alexandria Engineering Journal 61(5), 2022, pp. 38313872.
15. M. Shaari et al., Supervised evolutionary programming
based technique for multi-DG installation in distribution system,
IAES International Journal of Artificial Intelligence 9, 2020, pp.
11.
16. K. M. Ang et al., Modified teaching-learning-based
optimization and applications in multi-response machining
processes, Computers & Industrial Engineering 174, 2022, pp.
108719.
17. A. Sharma, A. Sharma, V. Jately, M. Averbukh, S. Rajput,
and B. Azzopardi, A novel TSA-PSO based hybrid slgorithm
for GMPP tracking under partial shading conditions, Energies,
15, 2022, pp. 3164.
18. A. Singh, A. Sharma, S. Rajput, A. K. Mondal, A. Bose and
M. Ram, Parameter extraction of solar module using the sootytern optimization algorithm, Electronics 11, 2022, pp. 564.
19. C. Hassan, V. Durai, S. Sapuan, N. A.A and M. Z. Mohamed
Yusoff, Mechanical and crash performance of unidirectional oilpalm empty fruit bunch fibre-reinforced polypropylene
composite, Bioresources 13, 2018, pp. 8310-8328.

wafer defects, including Center, Edge-Loc, Near-Full,
Random, and Scratch, compared to the unoptimized
version. Notably, the unoptimized model frequently
misclassifies Edge-Loc defects as Edge-Ring, leading tosignificant errors, and incorrectly labels 14.5% of 'No
defect' cases as Edge-Ring defects. Both the quantitativeand qualitative analyses suggest that optimizing ResNet18's hyperparameters through AOA effectively corrects
the misclassification issues observed in the unoptimized
model, resulting in a marked enhancement in the model'sability to classify wafer defects accurately and efficiently.
5. Conclusion
This study is designed to demonstrate the effectivenessof optimizing hyperparameters in deep learning models
for wafer defect classification. In our approach, theArithmetic Optimization Algorithm (AOA) is employed
to fine-tune four key hyperparameters—momentum,
initial learning rate, maximum epoch, and L2
regularization rate—of the ResNet-18 model. This model,
initially pre-trained on ImageNet, was further refined ona wafer defect dataset via transfer learning. Our extensive
simulation studies indicate that the AOA-optimized
ResNet-18 model surpasses the unoptimized version inrecall, accuracy, precision, F1 score, and AUC,
showcasing enhanced capabilities in accurately
identifying and classifying wafer defects.
The findings reveal the significance of hyperparameter
optimization in developing deep learning models for
wafer defect classification. Future research could
investigate the applicability of AOA in fine-tuning otherdeep learning models within this domain. Additionally,
exploring AOA for identifying optimal deep learning
network architectures offers promising avenues forrobust wafer defect classification. Incorporating morefeatures and diverse data sources could further refine theproposed method's efficacy. Finally, subsequent studies
should assess the practicality of implementing these
optimized deep learning models in industrial settings forreal-world wafer defect classification applications.
Acknowledgements
This work was supported by UCSI University’s
Research Excellence & Innovation Grant (REIG) withproject code of REIG-FETBE-2022/038 and Billion
Prima Sdn. Bhd.’s Industry Research Grant with projectcode of IND-FETBE-2023/006.
References
1. J. C. Chien, M. T. Wu and J. D. Lee, Inspection andclassification of semiconductor wafer surface defects using
CNN deep learning networks, Applied Sciences 10(15), 2020,
pp. 5340.
2. B. Jdid, W. H. Lim, I. Dayoub, K. Hassan and M. R. B. M.
Juhari, Robust automatic modulation recognition through joint
contribution of hand-crafted and contextual features, IEEE
Access 9, 2021, pp. 104530-104546.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
869

Powered by TCPDF (www.tcpdf.org)

Koon Hian Ang, Koon Meng Ang, Chin Hong Wong, Abhishek Sharma, Chun Kit Ang, Kim Soon Chong, Sew Sun Tiang, Wei Hong Lim

20. E. Natarajan, C. Hassan, C. K. Ang, M. S. Santhosh, S.
Ramesh and R. Sasikumar, Modeling of multiwall carbon
nanotubes reinforced natrual rubber for soft robotic applications- A comprehessive presentation 46(9), 2021, pp. 3251-3258.
21. A. Machmudah et al. Cyclic path planning of hyperredundant manipulator using whale optimization algorithm,
International Journal of Advanced Computer Science andApplications 12(8), 2021, pp. 677-686.
22. K. Palanikumar, J. Nithyanandam, E. Natarajan, W. H. Lim
and S. S. Tiang, Mitigated cutting force and surface roughness
in titanium alloy-multiple effective guided chaotic multi
objective teaching learning based optimization, Alexandria
Engineering Journal 64, 2023, pp. 877-905.
23. K. M. Ang et al., Optimal design of convolutional neuralnetwork architectures using teaching and learning-based
optimization for image classification, Symmetry 14(11), 2022,
pp. 2323.
24. L. Abualigah, A. Diabat, S. Mirjalili, M. Abd Elaziz and A.
H. Gandomi, The Arithmetic Optimization Algorithm,
Computer Methods in Applied Mechanics and Engineering 376,
2021, pp. 113609.
25. J.-S. R. Jang. MIR-WM811K wafer map,
http://mirlab.org/dataSet/public/

Dr. Abhishek Sharma
He is a Research Assistant Professor atGraphic Era Deemed to be University inIndia. He received his PhD from
University of Petroluem & Energy
Studies in 2022. His research interests
are artificial intellgence and powerelectronics.

Dr. Chun Kit Ang
He is the Dean and Associate Professor
in Faculty of Engineering at UCSI
University in Malaysia. He received his
PhD in Mechanical and Manufacturing
Engineering from Universiti Putra
Malaysia in 2014. His research interestsareartificial
intelligence,
soft
computing, robotics and mechatronics.
Dr. Kim Soon Chong
He is an Assistant Professor in Facultyof Engineering at UCSI University in
Malaysia. He received his PhD in
Electrical, Eleronics & System
Engineering
fromUniversiti
Kebangsaan Malaysia in 2022. His
research interests are biomedical andhealthcare technoologies.

Authors Introduction
Mr. Koon Hian Ang
He received the Bachelor of
Mechatronics
Engineeringwith
Honours in Faculty of Engineering,Technology and Built Environment,
UCSI University, Malaysia, in 2023.
His research interests are machinelearning,
deep
learning,
andoptimization algorithm.

Dr. Sew Sun Tiang
She is an Assistant Professor in Facultyof Engineering at UCSI University in
Malaysia. She received her PhD in
Electrical and Electronic Engineeringfrom Universiti Sains Malaysia in
2014. Her research interests are
optimization and antenna design.

Dr. Koon Meng Ang
He received the B.Eng. degree in
Mechatronic
Engineeringwith
Honours and PhD in Engineering fromUCSI University, Malaysia, in 2019and 2023, respectively. His researchinterests are swarm intelligence,machine learning and deep learning.

Dr. Wei Hong Lim
He is an Associate Professor in Facultyof Engineering at UCSI University in
Malaysia. He received his PhD in
Computational
Intelligence
fromUniversiti Sains Malaysia in 2014. His
research interests are optimization andartificial intelligence.

Dr. Chin Hong Wong
He is a Lecturer in MaynoothInternational Engineering College at
Fuzhou University in China. He
received his PhD in Electrical andElectronic Engineering from Universiti
Sains Malaysia in 2017. His researchinterests are Energy harvesting andcontrol system.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
870

Tackling Photovoltaic (PV) Estimation Challenges: An Innovative AOA Variant for Improved
Accuracy and Robustness
Rayan Mohammed Noor Mohammed Bakhit1, Abhishek Sharma2, Tiong Hoo Lim3, Chin Hong Wong4,5, Kim Soon Chong1, Li
Pan1, Sew Sun Tiang1*, Wei Hong Lim1*
1
Faculty of Engineering, Technology and Built Environment, UCSI University, Kuala Lumpur 56000, Malaysia
Department of Computer Science and Engineering, Graphic Era Deemed to be University, Dehradun 248002, India
3
Faculty of Engineering, Universiti Teknologi Brunei, Bandar Seri Begawan 1410, Brunei Darussalam
4
Maynooth International Engineering College, Maynooth University, Maynooth, Co Kildare, Ireland
5
Maynooth International Engineering College, Fuzhou University, Fujian, 350116, China
Email: 1001955674@ucsiuniversity.edu.my, abhishek15491@gmail.com, lim.tiong.hoo@utb.edu.bn,
chinhong.wong@mu.ie, 1002060534@ucsiuniversity.edu.my, tiangss@ucsiuniversity.edu.my,limwh@ucsiuniversity.edu.my

2

Abstract
Optimizing photovoltaic (PV) cell/module modeling is key to advancing solar power and achieving net zero carbon
goals. Challenges in accurate PV parameter estimation arise from environmental variability, aging, and incomplete
manufacturer data. Traditional Arithmetic Optimization Algorithm (AOA) often struggles with premature
convergence due to imbalanced exploration and exploitation. This paper presents an enhanced AOA variant,incorporating chaotic maps and oppositional-based learning to better balance the optimization process. Our extensive
simulations show that this improved AOA variant significantly enhances accuracy and robustness in PV cell/moduleparameter estimation compared to the conventional method.
Keywords: Photovoltaic module/cell, Parameter estimation, Arithmetic optimization algorithm

1. Introduction

Existing photovoltaic (PV) cell/module parameter
estimation approaches are broadly categorized into
deterministic and metaheuristic methods. Deterministic
methods, simpler and less computationally intensive,
often fall short in accuracy under varied environmental
conditions [3]. In contrast, metaheuristic methods,
drawing inspiration from natural phenomena, exhibit
superior global search capabilities, ease of
implementation, and scalability. Consequently, they havebecome highly effective for diverse global optimization
challenges [4], [5], [6], [7], [8], [9], [10], [11]. These
metaheuristic methods also excel in addressing complex,
multimodal PV cell/module parameter estimation
problems [12], [13], [14], [15], [16]. Despite substantial
progress using metaheuristic approaches, challenges
persist due to the intricate, nonlinear interplay of PV
module parameters and variable operating conditions.
Therefore, developing robust metaheuristic methods formore precise PV parameter estimation is vital. Suchadvancements will enhance PV system optimization,
maximizing solar energy utilization and contributing to asustainable future.
Arithmetic Optimization Algorithm (AOA) [17] is amathematics-inspired metaheuristic method that employs
four basic arithmetic operations (division, multiplication,
addition, and subtraction) with varying exploration andexploitation strengths for solving optimization problems.

The Intergovernmental Panel on Climate Change(IPCC) has underscored the urgent necessity to address
global warming, advocating for significant efforts toattain net zero carbon emissions by 2050. This imperative
demands a transition to renewable energy sources, withsolar power emerging as a key player due to its
sustainability and minimal environmental footprint [1].
Solar energy, captured through photovoltaic (PV) cells
and modules, offers versatility, catering to a range ofapplications from residential heating to extensive solar
farms [2]. Yet, the efficacy of solar systems is criticallydependent on the performance of PV arrays, which areprone to degradation under harsh outdoor conditions.
Precision in PV system modeling is crucial forenhancing performance. This process encompasses
various models, including single-diode, double-diode,and triple-diode, each demanding distinct parameter suchas photocurrent and resistances [3]. Often, theseparameters are not readily available from manufacturers,
necessitating their estimation from experimental data.
This task is further complicated by the aging of PV
systems and fluctuating environmental factors [3]. The
intricate and multimodal characteristics of PV parameter
estimation, especially under diverse irradiance andtemperature conditions, pose considerable challenges.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024871

Rayan Mohammed Noor Mohammed Bakhit, Abhishek Sharma, Tiong Hoo Lim, Chin Hong Wong, Kim Soon Chong, Li Pan, Sew Sun Tiang, Wei HongLim

In PV cell/module parameter estimation, the AOA
population
signifies diverse
diode parameter
combinations. Since its introduction, AOA and its
variants have been effectively applied to various realworld optimization problems [18], [19], [20], [21], [22].
However, AOA, like many metaheuristic methods, relies
on a conventional approach for generating its initial
population, which often lacks intelligent, systematic
initialization. This approach tends to produce initialsolutions that are either trapped in local optimum or
distant from the global optimum, thus impacting thesolution's accuracy and the algorithm's convergence
speed [23]. This limitation affects AOA's efficiency incomplex, multimodal challenges such as PV cell/moduleparameter estimation.
In this paper, we introduce an enhanced variant of AOA,
termed AOA with Modified Initialization Scheme (AOAMIS), specifically designed to tackle complex andmultimodal PV cell/module parameter estimation
problems. The MIS module combines chaotic map
strengths and dynamic oppositional learning (DOL) to
generate a higher-quality initial population withimproved fitness and diversity. Utilizing the nonrepetitive and ergodic nature of chaotic maps, the MIS
module enhances initial population diversity, thereby
increasing the algorithm's robustness for complex issues.
Concurrently, the DOL mechanism within MIS module
effectively accelerates AOA's convergence by generating
opposite solutions for those significantly distant from theglobal optimum. The efficacy of AOA-MIS, using
various diode modeling techniques, is then benchmarked
against the original AOA.

diode parameters, the error function for PV cells is
expressed as:

Fig.1 Equivalent circuit of PV using TDM with J = 3.

{

(2)

𝑗=1→𝐽

for 𝑗 = 1, … , 𝐽

Root mean square error (RMSE) is commonly
employed as the objective function, defined as:
𝐾

1
𝑅𝑀𝑆𝐸 = √ ∑ 𝑓 2 (𝑋)
𝐾

(3)

𝑘=1

where k and K represent the indices and total count of
measured current data points, respectively. The aim is tofind a solution that minimizes the RMSE in PV
cell/module parameter estimation.
3. AOA-MIS
3.1. Proposed MIS module
The process of generating an initial population in the
proposed MIS module is described as follows. Initially, achaotic population of size N is generated using a modified
sine chaotic map, a departure from the traditional
initialization schemes. This map, based on deterministic
equations, can display stochastic behavior at external
levels. The ergodic and non-repetitive nature of chaotic
map promotes a more comprehensive search in thesolution space, preventing initial solutions from being
trapped in local optima and thus enhancing the diversity
and robustness of the initial population against premature
convergence. Define 𝝑𝒕 as the output of a chaotic
variable at the t-th iteration, for 𝒕 = 𝟏, … , 𝑻, updated as
per the modified sine chaotic map with a bifurcation
coefficient 𝝁 = 𝝅:The electrical properties of PV systems can be modeledusing different approaches: single diode model (SDM),
double diode model (DDM) and triple diode model(TDM) as shown in Fig. 1. Let 𝒋 = 𝟏, . . , 𝑱 be the diodeindex in the model, where 𝑱 = 𝟏 is for SDM, 𝑱 = 𝟐 for
DDM and 𝑱 = 𝟑 for TDM. For a given output voltage V,
the output current I each diode model is determined as:𝑞(𝑉+𝐼𝑅𝑠 )𝑉 + 𝐼𝑅𝑠()− ∑ 𝐼𝑠𝑠𝑑𝑗 [𝑒 𝑛𝑗 𝑘𝑇 − 1]𝑅𝑠ℎ𝑞(𝑉+𝐼𝑅𝑠 )𝑉 + 𝐼𝑅𝑠()− ∑ 𝐼𝑠𝑠𝑑𝑗 [𝑒 𝑛𝑗 𝑘𝑇 − 1] − 𝐼
𝑅𝑠ℎ

𝑋 = [𝐼𝑝ℎ , 𝑅𝑠 , 𝑅𝑠ℎ , 𝐼𝑠𝑠𝑑𝑗 , 𝑛𝑗 ],2. PV Cell/Module Parameter Estimation Problem

𝐼 = 𝐼𝑝ℎ −𝑓(𝑋) = 𝐼𝑝ℎ −

(1)

𝑗=1→𝐽

where 𝐼𝑝ℎ and 𝐼𝑠ℎ are the photogenerated line current andshunt resistor line current, respectively; 𝐼𝑠𝑠𝑑𝑗 is thesaturation current of the j-th diode, 𝑅𝑠 and 𝑅𝑠ℎ are theseries and shunt resistances, respectively; 𝑛𝑗 is theideality factor of the j-th diode, 𝑇 is the absolutetemperature, 𝑘 is Boltzmann's constant, and q is the unit
charge. Accurate modeling using SDM, DDM, and TDM
necessitates the accurate estimation of parameters such as
𝐼𝑝ℎ , 𝑅𝑠 , 𝑅𝑠ℎ , 𝐼𝑠𝑠𝑑𝑗 and 𝑛𝑗 for 𝑗 = 1, … , 𝐽.
In PV cell/module parameter estimation using
metaheuristic methods, appropriate objective functions
are formulated based on the discrepancy between theexperimental current and the model's predicted current.
Considering X as a candidate solution with undetermined𝜗𝑡+1 = sin (𝜇𝜗𝑡 )(4)

The final iteration output 𝝑𝑻 initializes the d-th
dimension of each n-th chaotic solution, as in Eq. (5),where 𝑿𝑳𝒅 and 𝑿𝑼
𝒅 are the lower and upper bounds of thed-th decision variable, respectively. This results in achaotic population 𝐏 𝐂 = [𝑿𝑪𝟏 , … , 𝑿𝑪𝒏 , … 𝑿𝑪𝑵 ].
𝐶𝑋𝑛,𝑑= 𝑋𝑑𝐿 + 𝜗𝑇 (𝑋𝑑𝑈 − 𝑋𝑑𝐿 )(5)

Despite its advantages, chaotic map may still generate
initial solutions far from the global optimum, potentiallyslowing algorithm convergence. To mitigate this, a DOL
operator is applied to the chaotic population 𝐏 𝐂 to

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
872

Tackling Photovoltaic (PV) Estimation Challenges

𝑋𝑛,𝑑 (𝐶𝐼𝑡𝑒𝑟 + 1) =𝑏𝑒𝑠𝑡𝑑 − 𝑀𝑃 × [(𝑋𝑑𝑈 − 𝑋𝑑𝐿 )𝜇 + 𝑋𝑑𝐿 ], 𝑟𝑎𝑛𝑑2 < 0.5{
𝑏𝑒𝑠𝑡𝑑 + 𝑀𝑃 × [(𝑋𝑑𝑈 − 𝑋𝑑𝐿 )𝜇 + 𝑋𝑑𝐿 ], Otherwise

produce opposite solutions for each n-th chaotic solution.
This DOL operator broadens the initial population's
coverage of the solution space, enhancing the chance offinding fitter solutions. The opposite solution for eachdimension, i.e., 𝑿𝑶𝒏,𝒅 , is calculated using Eq. (6),corresponding to 𝑿𝑪𝒏,𝒅 , where 𝒓𝟏 , 𝒓𝟐 ∈ [𝟎, 𝟏] arerandomly generated numbers. This results in an𝑶𝑶
opposition population, 𝐏 𝐎 = [𝑿𝑶
𝟏 , … , 𝑿𝒏 , … 𝑿𝑵 ].
𝑂
𝐶
𝐶
𝐶
𝑋𝑛,𝑑
= 𝑋𝑛,𝑑+ 𝑟1 [𝑟2 (𝑋𝑑𝑈 + 𝑋𝑑𝐿 − 𝑋𝑛,𝑑) − 𝑋𝑛,𝑑]The proposed AOA-MIS iterates through this search
process, following Eqs. (7) to (10), until predefinedtermination criteria are met, as illustrated in Fig. 2. Uponcompletion, the optimal diode parameters in the bestsolution are decoded to address the PV cell/moduleparameter estimation problems.

(6)
AOA-MIS for PV Cell/Module Parameter Estimation
Inputs: D, N, 𝑀𝐼𝑡𝑒𝑟 , T, 𝑀𝑎𝑥, 𝑀𝑖𝑛, θ
01: Initialize 𝐏𝐂 ← ∅, 𝐏𝐎 ← ∅ and 𝐶𝐼𝑡𝑒𝑟 ← 0;
02: for each n-th solution do
03:
for each d-th dimension do
04:
Randomly generate 𝜗𝑡 ∈ [0,1], where t = 0;05:while 𝑡 ≤ 𝑇 do
06:
Update 𝜗𝑡 using Eq. (4);07:
Update 𝑡 ← 𝑡 + 1;
08:
end while𝐶09:
Calculate 𝑋𝑛,𝑑with Eq. (5);𝑂
10:Calculate 𝑋𝑛,𝑑with Eq. (6);11:
end for𝐶
𝑂
12:
Update 𝐏𝐂 ← 𝐏𝐂 ∪ 𝑋𝑛,𝑑and 𝐏𝐎 ← 𝐏𝐎 ∪ 𝑋𝑛,𝑑;13: end for
14: Merge two populations as 𝐏𝐂 ∪ 𝐏𝐎 ;15: Fitness evaluation of all solutions stored within themerged population set of 𝐏𝐂 ∪ 𝐏𝐎 using Eq. (3);16: Sort the solutions within 𝐏𝐂 ∪ 𝐏𝐎 from best to worst
based on their fitness values;
17: Select the top N solutions from the sorted 𝐏𝐂 ∪ 𝐏𝐎 as
the initial population, i.e., 𝐏 = [𝑋1 , … , 𝑋𝑛 , … . , 𝑋𝑁 ].
18: Assign the first solution of P and its fitness as best
and 𝑓(𝑏𝑒𝑠𝑡), respectively;
19: while 𝐶𝐼𝑡𝑒𝑟 ≤ 𝑀𝐼𝑡𝑒𝑟 do
20:Update MOA and MP with Eqs. (7) and (8);21:
for each n-th solution do
22:
if 𝑟𝑎𝑛𝑑1 > 𝑀𝑂𝐴 then /*Exploration*/
23:
Update 𝑋𝑛,𝑑 (𝐶𝐼𝑡𝑒𝑟 + 1) with Eq. (9);24:else /*Exploitation*/25:
Update 𝑋𝑛,𝑑 (𝐶𝐼𝑡𝑒𝑟 + 1) with Eq. (10);26:
end if27:
Fitness evaluation of 𝑋𝑛 (𝐶𝐼𝑡𝑒𝑟 + 1) with
Eq. (3);28:
Update the 𝑋𝑛 , 𝑓(𝑋𝑛 ), 𝑏𝑒𝑠𝑡 and 𝑓(𝑏𝑒𝑠𝑡),
with greedy selection method;29:
end for30:
𝐶𝐼𝑡𝑒𝑟 ← 𝐶𝐼𝑡𝑒𝑟 + 1;
31: end while
Output: 𝑏𝑒𝑠𝑡 and the corresponding PV model;

The two populations, 𝐏 𝐂 and 𝐏 𝐎 , are combined into asingle set 𝐏 𝐂 ∪ 𝐏 𝐎 , resulting in a total population size of
2N. Each solution within this merged population set isevaluated for fitness using Eq. (3), focusing on RMSE.
The solutions are then rearranged based on their fitness
values, from the best to the worst. The top N solutions
from this sorted set 𝐏 𝐂 ∪ 𝐏 𝐎 are selected as the initial
population for the AOA-MIS algorithm, denoted as 𝐏 =[𝑿𝟏 , … , 𝑿𝒏 , … . , 𝑿𝑵 ].
3.2. Iterative Search Mechanisms of AOA-MIS
After generating the initial population P with theproposed MIS module, each n-th solution of AOA-MIS
is iteratively updated using search mechanisms akin to
those in the original AOA.
At each iteration, the Math Optimizer Accelerated(MOA) function value is updated for the proposed AOAMIS to toggle between exploration and exploitation:
𝑀𝑂𝐴(𝐶𝐼𝑡𝑒𝑟 ) = 𝑀𝑖𝑛 + 𝐶𝐼𝑡𝑒𝑟 (𝑀𝑎𝑥−𝑀𝑖𝑛
𝑀𝐼𝑡𝑒𝑟

)

(10)

(7)

where 𝐶𝐼𝑡𝑒𝑟 and 𝑀𝐼𝑡𝑒𝑟 represent the current andmaximum iteration numbers, respectively; 𝑀𝑖𝑛 and 𝑀𝑎𝑥
are the minimum and maximum values of MOA.
Concurrently, the Math Optimizer Probability (MP)function, guiding the search range for each solution andinfluenced by the critical parameter θ for exploitation
efficiency, is updated:
1

𝐶𝐼𝑡𝑒𝑟 ⁄𝜃
(8)
𝑀𝑃(𝐶𝐼𝑡𝑒𝑟 ) = 1 − ()
𝑀𝐼𝑡𝑒𝑟Based on MOA's value, a random number 𝒓𝒂𝒏𝒅𝟏
decides the search strategy (exploration or exploitation)
at each iteration for updating every d-th dimension of then-th solution, 𝑿𝒏,𝒅 . In the exploration phase (𝒓𝒂𝒏𝒅𝟏 >𝑴𝑶𝑨), Multiplication or Division is randomly chosen:
𝑋𝑛,𝑑 (𝐶𝐼𝑡𝑒𝑟 + 1)
𝑏𝑒𝑠𝑡𝑑 ÷ (𝑀𝑃 + 𝜀) × [(𝑋𝑑𝑈 − 𝑋𝑑𝐿 )𝜇 + 𝑋𝑑𝐿 ], 𝑟𝑎𝑛𝑑2 < 0.5={
𝑏𝑒𝑠𝑡𝑑 × 𝑀𝑃 × [(𝑋𝑑𝑈 − 𝑋𝑑𝐿 )𝜇 + 𝑋𝑑𝐿 ], Otherwise(9)
where 𝑟𝑎𝑛𝑑2 is a random number between 0 and 1;
𝑏𝑒𝑠𝑡𝑑 is the d-th dimension of the current best solution;
𝜀 is a small positive number preventing division by zero;𝜇 is a control parameter.
In the exploitation phase (𝑟𝑎𝑛𝑑1 ≤ 𝑀𝑂𝐴), Addition or
Subtraction operator updates 𝑋𝑛,𝑑 :Fig.2 Workflow of proposed AOA-MIS in solving thePV cell/module parameter estimation problems.
4. Results and Discussions
4.1. Simulation settings
In this section, the proposed AOA-MIS is applied tosolve the PV cell parameter estimation problem involves
the test case of R.T.C. France solar cell using SDM,
DDM and TDM approaches. The proposed AOA-MIS
and original AOA are implemented in MATLAB 2021a
on a personal computer consisting of an Intel® Core TM©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024873

Rayan Mohammed Noor Mohammed Bakhit, Abhishek Sharma, Tiong Hoo Lim, Chin Hong Wong, Kim Soon Chong, Li Pan, Sew Sun Tiang, Wei HongLim

i7-HQ CPU, 2.50 GHz, and 16 GB RAM laptop. For allcompared techniques, the population size and maximum
iteration numbers are set as 30 and 1000, respectively.

for both algorithms. The data in Table 3 reveal that AOAMIS consistently outperforms the original AOA in termsof Max, Min, and Mean RMSE values. Additionally,
AOA-MIS shows superior consistency in achieving
lower RMSE values, as evidenced by lower SD values inboth SDM and DDM test cases.

4.2. Performance analysis
The current and voltage experimental values for the
R.T.C. France solar cell were recorded under standard
conditions: 1000 W/m² at 33°C. This subsection applies
the proposed AOA-MIS to estimate five and seven
unknown parameters of the SDM and DDM, respectively,
representing the R.T.C. solar cell at these conditions. The
AOA-MIS results are compared with those from theoriginal AOA. RMSE serves as the performance index toevaluate AOA-MIS's effectiveness.
Table 1 and Table 2 showcase the optimized parameters
derived from AOA-MIS and the original AOA for the
R.T.C. France solar cell, using the SDM and DDM,
respectively, along with their corresponding RMSE
values. Parameters yielding better results, as indicated bylower RMSE values, are highlighted in bold. Table 1
reveals that AOA-MIS’s estimation of the fiveparameters (i.e., 𝐼𝑝ℎ , 𝑅𝑠 , 𝑅𝑠ℎ , 𝐼𝑠𝑠𝑑 and 𝑛 ) for the SDM
leads to higher modeling accuracy for the R.T.C. France
solar cell, as evidenced by lower RMSE values,
compared to the original AOA. Similarly, Table 2 showsthat AOA-MIS’s estimation of the seven parameters (i.e.,
𝐼𝑝ℎ , 𝑅𝑠 , 𝑅𝑠ℎ , 𝐼𝑠𝑠𝑑,1 , 𝐼𝑠𝑠𝑑,2 , 𝑛1 and 𝑛2 ) for the DDM yields
more competitive RMSE values.

Table 3. Statistical results of the RMSE valuesof R.T.C France solar cell represented bydifferent diode modelling methods
Model
RSME
AOA
AOA-MISMin
5.3572×10 -3
3.2507×10-3
Mean
3.0009×10 -2
2.4830×10-2
SDM
-1
Max
1.4549×10
6.7476×10-2
SD4.2432×10 -2
2.0059×10-2Min1.7449×10 -1
𝟏. 𝟕𝟎𝟓𝟎 ×10 -1
Mean
1.9299 ×10 -1
𝟏. 𝟕𝟖𝟕𝟓 ×10 -1
DDM
Max
2.1688×10 -1
𝟏. 𝟖𝟗𝟑𝟖 ×10 -1SD1.2098 ×10 -2
𝟓. 𝟖𝟗𝟐𝟐 ×10 -3

The superior performance of the proposed AIA-MIS
over the original AOA can be attributed to several factors.
PV cell/module parameter estimation is a complex,
multimodal real-world optimization problem, largely due
to the nonlinear relationship between model parametersand varying operating conditions such as temperature andirradiance levels. The complexity of this problem
escalates with the number of diodes used in modelling(e.g., five parameters in SDM, seven parameters in DDM
and so on), further complicating the optimization task.
Table 1, Table 2, and Table 3 suggest that the quality of
the initial population is crucial in enabling AOA toaccurately model PV cell/module parameters. The
original AOA, using a conventional initialization scheme,
tends to generate the initial solutions in local or nonoptimal regions, leading to premature convergence due tothe absence of intelligent methods and knowledge of thesurrounding search environment. Moreover, there is asignificant risk of initializing solutions far from theglobal optimum, slowing down the algorithm's
convergence speed. In contrast, the MIS module in AOAMIS generates an initial population of higher quality interms of fitness and diversity. The chaotic map's nonrepetitive and ergodic nature in the MIS module promotesa more comprehensive search of the solution space,minimizing the risk of local optima entrapment andpremature convergence. DOL, another key mechanism in
MIS, accelerates convergence by enhancing the
exploration of search range through generating opposite
solutions from those initialized by the chaotic map. Thissynergistic effect of the chaotic map and DOL in the MIS
module bolsters AOA-MIS's robustness in navigatingcomplex, multimodal search spaces, thereby enhancing
its accuracy in solving PV cell/module parameter
estimation problems.

Table 1. Optimized parameters obtained for
R.T.C France solar cell using SDM.
Parameters
AOA
AOA-MIS𝐼𝑝ℎ
0.754830.758220.052023
0.039672
𝑅𝑠
97.80569.689𝑅𝑠ℎ
𝐼𝑠𝑠𝑑
1.5746×10 -8
1.5411×10 -7
1.24851.4316𝑛RSME
5.3572×10 -3
3.2507×10-3
Table 2. Optimized parameters obtained for
R.T.C France solar cell using DDM.
Parameters
AOA
AOA-MIS0.73552
0.80916𝐼𝑝ℎ
0.0044392
0
𝑅𝑠
2.668
1.7305𝑅𝑠ℎ00𝐼𝑠𝑠𝑑,100𝐼𝑠𝑠𝑑,2
1.14281.0141𝑛1
1.9816
1.235
𝑛2
RSME
1.7449×10 -1
1.705×10 -1

To further assess the effectiveness of the proposed
AOA-MIS in addressing the PV cell/module parameter
estimation problems, Table 3 provides a statisticalanalysis of the RMSE values for the R.T.C. France solar
cell using both SDM and DDM models, comparing
AOA-MIS and the original AOA. This analysis includes
performance metrics such as the minimum (Min),
maximum (Max), mean (Mean), and standard deviation(SD) of the RMSE values from multiple simulation runs

5. Conclusion
In this paper, we introduce an enhanced version of theArithmetic Optimization Algorithm, termed AOA-MIS,
to address the complex and multimodal challenges of PV©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
874

Tackling Photovoltaic (PV) Estimation Challenges

cell/module parameter estimation. The innovation of
AOA-MIS lies in integrating chaotic maps and DOL
mechanisms into the Modified Initialization Scheme(MIS) module, thereby generating an initial population
with improved fitness and diversity. Simulation results
reveal that AOA-MIS, with its superior initial population
quality, outperforms the original AOA in estimating
parameters for the SDM and DDM in representing the
R.T.C. France solar cell, tested under standard conditionsof 1000 W/m² at 33°C. As one of the future works, AOAMIS could be further enhanced by incorporating an
adaptive search mechanism, potentially increasing its
robustness and efficacy in addressing various
complexities in PV cell/module parameter estimation.

10.

11.

12.

Acknowledgements

13.

This work was supported by UCSI University’s
Research Excellence & Innovation Grant (REIG) withproject code of REIG-FETBE-2022/038 and Billion
Prima Sdn. Bhd.’s Industry Research Grant with projectcode of IND-FETBE-2023/006.

14.

References

15.

1.

2.

3.

4.

5.

6.

7.

8.

9.

M. Premkumar and T. R. Sumithira, Design andimplementation of new topology for solar PV basedtransformerless forward microinverter, Journal of
Electrical Engineering & Technology 14(1), 2019, pp.
145-155.
D. Jung, A. Salmon and P. Gese, Agrivoltaics for farmers
with shadow and electricity demand: Results of a prefeasibility study under net billing in central Chile, AIP
Conference Proceedings 2361(1), 2021, pp. 030001.
Z. Gu, G. Xiong and X. Fu, Parameter extraction of solar
photovoltaic cell and module models with metaheuristic
algorithms: A review, Sustainability 15(4), 2023, pp. 3312.
E. S. Ghith and F. A. A. Tolba, Real-time implementationof an enhanced proportional-integral-derivative controller
based on sparrow search algorithm for micro-robotics
system, IAES International Journal of Artificial
Intelligence 11(4), 2022, pp. 1395-1404.
E. S. Ghith and F. A. A. Tolba, LabVIEW implementationof tuning PID controller using advanced control
optimization techniques for micro-robotics system.
International Journal of Mechanical Engineering andRobotics Research 11(9), 2022, pp. 653-661.
E. S. Ghith and F. A. Tolba, Real-time implementation ofan enhanced PID controller based on marine predator
algorithm (MPA) for micro-robotics system, 2022 3rdInternational Conference on Artificial Intelligence,
Robotics and Control (AIRC), Cairo, Egypt, 2022.
K. M. Ang et al., Modified teaching-learning-based
optimization and applications in multi-response machining
processes, Computers & Industrial Engineering 174, 2022,
pp. 108719.
C. Hassan, S. Sapuan, N. Abd. Aziz and M. Z. Mohamed
Yusof, Effect of chemical treatment on the tensile
properties of single oil palm empty fruit bunch (OPEFB)
fibre. Trends in Textile Engineering & Fashion
Technology 3(2), 2018, pp. 1-7.
E. Natarajan, C. Hassan, C. K. Ang, M. S. Santhosh, S.
Ramesh and R. Sasikumar, Modeling of multiwall carbon

16.

17.

18.

19.

20.

21.

22.

23.

nanotubes reinforced natural rubber for soft robotic
applications - A comprehensive presentation 46(9), 2021,
pp. 3251-3258.
C. Hassan C. W. Yeo, B. Shari, M. S. Salit and N. Abdul
Aziz, Mechanical properties of unidirectional oil palm
empty fruit bunch (OPEFB) fiber reinforced epoxy
composite. IOP Conference Series: Materials Science andEngineering 206(1), 2017, pp 012045.
L. Yao et al., Demand bidding optimization for an
aggregator with genetic algorithm. Energies 11(1), 2018,
pp. 1-22.
M. Premkumar et al., A reliable optimization frameworkfor parameter identification of single-diode solar
photovoltaic model using weighted velocity-guided grey
wolf optimization algorithm and Lambert-W function, IET
Renewable Power Generation 17(11), 2023, pp. 27112732.
A. Sharma et al., Identification of photovoltaic moduleparameters by implementing a novel teaching learning
based optimization with unique exemplar generation
scheme (TLBO-UEGS), Energy Report 10, 2023, pp.
1485-1506.
A. Sharma et al., Performance investigation of state-ofthe-art metaheuristic techniques for parameter extractionof solar cells/module, Scientific Reports 13, 2023, pp.
11134.
A. Sharma et al., An effective method for parameter
estimation of solar PV cell using grey-wolf optimization
technique, International Journal of Mathematical,
Engineering and Management Science 6(3), 2021, pp. 911931.
H. Bakır, Comparative performance analysis ofmetaheuristic search algorithms in parameter extraction
for various solar cell models, Environmental Challenges
11, 2023, pp. 100720.
L. Abualigah, A. Diabat, S. Mirjalili, M. Abd Elaziz andA. H. Gandomi, The arithmetic optimization algorithm,
Computer Methods in Applied Mechanics andEngineering 376, 2021, pp. 113609.
E. S. Ghith and F. A. A. Tolba, Tuning PID controllers
based on hybrid arithmetic optimization algorithm andartificial gorilla troop optimization for micro-robotic
systems. IEEE Access 11, 2023, pp. 27138-27154.
S. Rajput et al., A novel opposition-based arithmetic
optimization algorithm for parameter extraction of PEM
fuel cell. Electronics 10(2), 2021, pp. 2834.
K. M. Ang et al., Training feedforward neural networks
using arithmetic optimization algorithm for medical
classification, in Advances in Intelligent Manufacturingand Mechatronics. Lecture Notes in Electrical Engineering,eds. M. A. Abdullah et al. Springer, Singapore, vol. 988,
2023.
A. A. Mohamed et al., A novel hybrid arithmetic
optimization algorithm and salp swarm algorithm for data
placement in cloud computing, Soft Computing 27(9),2023, pp. 5769-5780.
X. Shi, X. Yu, and M. Esmaeili-Falak, Improved
arithmetic optimization algorithm and its application tocarbon fiber reinforced polymer-steel bond strength
estimation, Composite Structures 306, 2023, pp. 116599.
M. F. Ahamad, N. A. Mat Isa, W. H. Lim and K. M. Ang,
Differential evolution with modified initialization scheme
using chaotic oppositional based learning strategy,
Alexandria Engineering Journal 61(12), 2022, pp. 1183511858.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
875

Powered by TCPDF (www.tcpdf.org)

Rayan Mohammed Noor Mohammed Bakhit, Abhishek Sharma, Tiong Hoo Lim, Chin Hong Wong, Kim Soon Chong, Li Pan, Sew Sun Tiang, Wei HongLim

Dr. Sew Sun Tiang
She is an Assistant Professor in Facultyof Engineering at UCSI University in
Malaysia. She received her PhD in
Electrical and Electronic Engineeringfrom Universiti Sains Malaysia in 2014.
Her research interests are optimization
and antenna design.

Authors Introduction
Mr. Rayan Mohammed Noor Mohammed Bakhit
He received the Bachelor of
Mechatronics
Engineeringwith
Honours in Faculty of Engineering,Technology and Built Environment,
UCSI University, Malaysia, in 2023.
His research interests are renewableenergy, and optimization algorithm.

Dr. Wei Hong Lim
He is an Associate Professor in Faculty ofEngineering at UCSI University in
Malaysia. He received his PhD in
Computational
Intelligence
fromUniversiti Sains Malaysia in 2014. His
research interests are optimization andartificial intelligence.

Dr. Abhishek Sharma
He is a Research Assistant Professor atGraphic Era Deemed to be University
in India. He received his PhD from
University of Petroluem & Energy
Studies in 2022. His research interests
are artificial intellgence and powerelectronics.
Dr. Tiong Hoo Lim
He is a Senior Assistant Professor andDirector of Planning Development
Office at Universiti Teknologi Brunei in
Brunie Darussalam. He received his
PhD in Computer Science from
University of York, United Kingdom.
His research interests are artificial
intelligence, advanced technology,
competitive and forecasting analysis.
Dr. Chin Hong Wong
He is a Lecturer in MaynoothInternational Engineering College at
Fuzhou University in China. He
received his PhD in Electrical andElectronic Engineering from Universiti
Sains Malaysia in 2017. His researchinterests are Energy harvesting andcontrol system.
Dr. Kim Soon Chong
He is an Assistant Professor in Facultyof Engineering at UCSI University in
Malaysia. He received his PhD in
Electrical, Eleronics & System
Engineering
fromUniversiti
Kebangsaan Malaysia in 2022. His
research interests are biomedical andhealthcare technoologies.
Ms. Li Pan
She received her Master of Engineeringin Computer Technology degree from
Huazhong University of Science andTechnology, China in 2008. She iscurrently a Doctoral research student in
UCSI University, Malaysia.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
876

Deep Learning in Manufacturing: A Focus on Welding Defect Classification with CNNs
Tin Chang Ting1, Hameedur Rahman2, Tiong Hoo Lim3, Chin Hong Wong4,5, Chun Kit Ang1, Mohamed Khan Afthab
Ahamed Khan1, Sew Sun Tiang1*, Wei Hong Lim1*
1

2

Faculty of Engineering, Technology and Built Environment, UCSI University, Kuala Lumpur 56000, Malaysia
Faculty of Computing and Artificial Intelligence, Air University, Islamabad Capital Territory 44000, Pakistan
3
Faculty of Engineering, Universiti Teknologi Brunei, Bandar Seri Begawan 1410, Brunei Darussalam
4
Maynooth International Engineering College, Maynooth University, Maynooth, Co Kildare, Ireland
5
Maynooth International Engineering College, Fuzhou University, Fujian, 350116, China
Email: 1002058096@ucsiuniversity.edu.my, rhameedur@gmail.com, lim.tiong.hoo@utb.edu.bn,
chinhong.wong@mu.ie, angck@ucsiuniversity.edu.my, mohamedkhan@ucsiuniversity.edu.my,tiangss@ucsiuniversity.edu.my, limwh@ucsiuniversity.edu.my

Abstract
Welding is integral to modern manufacturing, yet the complex process often leads to defects, impacting the qualityof the final product. Recent advances in deep learning, particularly Convolutional Neural Networks (CNNs), haveshown remarkable results in applications like defect recognition. This study evaluated AlexNet, ResNet-18, ResNet50, ResNet-101, MobileNet-v2, ShuffleNet, and SqueezeNet for their effectiveness in identifying welding defects,
using accuracy, precision, sensitivity, specificity, and F-score as metrics. The dataset covered defects like cracks, lackof penetration, porosity, and a no-defect class. Our analysis shows that most of these architectures deliver promising
results in accuracy, sensitivity, specificity, precision, and F1-score, highlighting their potential in defect recognition.
Keywords: Convolutional neural network, Classification, Deep learning, Welding defects

categories, including visual or manual inspection,
radiographic testing with ionizing radiation sources (suchas gamma rays or X-rays), eddy current testing, ultrasonic
testing, and dye penetrant testing [2], [3]. However, eachof these techniques has inherent limitations. For example,
eddy current testing is only applicable to metallic
specimens, while X-ray radiographic testing poses
potential health risks due to prolonged exposure toradiation. Additionally, the majority of current weld
surface defect recognition relies on manual inspection by
NDT experts analyzing these radiographic images. Thismanual process of interpreting and evaluating images canbe complex, subjective, inconsistent, time-consuming,labor-intensive, and prone to errors, particularly when
distinguishing between defects with similar features [4].
Consequently, developing an automated inspection
solution that offers more efficient and accurate
recognition of weld surface defects is essential to
overcome the drawbacks of manual inspections.
The adoption of automated machine vision systems,
integrating deep learning techniques, offers a promisingsolution for weld surface image classification challenges.
Convolutional Neural Networks (CNNs), renowned fortheir ability to learn nonlinear relationships between
inputs and outputs, mimic the human brain's learning
process and have been effectively applied in various realworld scenarios. These applications include signal/image
classification [5], [6], [7], cybersecurity [8], [9], medical
diagnosis [10], [11], [12], fault detection [13], [14], [15]1. Introduction
Welding is a crucial industrial process integral tovarious production sectors, including high-performanceindustries like aerospace, automotive, marine, and power
generation. This complex process involves multiple
parameters that directly impact the quality of the weld
joint. Factors such as welding procedures, methods,
environmental conditions, and the operator’s skill level
can lead to various defects during the welding ofpipelines and pressure vessels [1]. Common defects
include blowholes, cracks, incomplete fusion, incomplete
penetration, slag inclusions, and undercutting, whichsignificantly compromise the sealing and strength of the
products. In addition to welding parameters, unforeseen
events in the manufacturing process can also cause weld
defects. Therefore, rigorous welding quality inspection
and testing are essential during manufacturing, especiallyfor products like pipelines and pressure vessels. This is
critical to identify the root causes of welding defects andimplement targeted corrective measures to ensure
product quality and safety [1].
Considerable research has been devoted to addressing
the challenges of weld defect detection and identification.
These defects are typically detected using nondestructive testing (NDT) techniques, favored for their
non-invasive interaction with specimens. Common NDT
methods for weld defect detection fall into various©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
877

Tin Chang Ting, Hameedur Rahman, Tiong Hoo Lim, Chin Hong Wong, Chun Kit Ang, Mohamed Khan Afthab Ahamed Khan, Sew Sun Tiang,
Wei Hong Lim

and prediction [16], [17]. Encouraged by deep learning's
success, numerous network architectures have beendeveloped, such as AlexNet [18], ResNet-18 [19],ResNet-50 [19], ResNet-101 [19], MobileNet-v2 [20],ShuffleNet [21], and SqueezeNet [22]. Transfer learning,a method for adapting these existing CNN architectures
to new tasks, has gained popularity. It utilizes smaller
datasets and reduces training time, making it an efficientapproach for training CNNs in new application domains.
While previous studies [4], [23], [24], [25], [26], [27]have explored welding defect classification using deeplearning, the optimal selection of CNN architectures
remains relatively unexplored, leaving manyarchitectures yet to be assessed. Addressing this gap, ourpaper presents an extensive study evaluating theperformance of seven popular CNN architectures:
AlexNet,ResNet-18,
ResNet-50,
ResNet-101,
MobileNet-v2, ShuffleNet, and SqueezeNet, inclassifying weld defects from digital radiographic images.
Our aim is to objectively identify the most effective CNN
architecture using various performance metrics. This
study could lead to replacing manual inspection methods
with a more accurate automated weld defectclassification system, potentially reducing production
costs and increasing throughput.

To facilitate deep learning model training, the
RIAWELC dataset is divided into three subsets: 70% fortraining, 10% for validation, and 20% for testing.
Additionally, all radiographic images from the dataset are
resized to match the input requirements of the selected
pretrained network, as outlined in Table 2, ensuring
compatibility with the network architectures.
Table 2. Summary of seven pretrained networks
Pretrained
Depth Size Parameter
Input
Networks(MB) (Millions)
Size
AlexNet8227
61.0
224×224
ResNet-18
18
44
11.7
224×224
ResNet-50
50
96
25.6
224×224
ResNet-101
101
167
44.6
224×224
MobileNet-v2
53
13
3.5
224×224
ShuffleNet
50
5.4
1.4
224×224
SqueezeNet
18
5.2
1.24
227×2272.2. Transfer learning of pretrained networks
Training CNNs from scratch for specific tasks presents
challenges due to significant resource requirements, suchas training time, infrastructure, and input datasets.
Transfer learning offers a viable solution by transferringknowledge from one or more source domains to adifferent target domain. In this study, we utilize transfer
learning to extract the learnable parameters from selected
pretrained network architectures (namely, AlexNet,
ResNet-18, ResNet-50, ResNet-101, MobileNet-v2,
ShuffleNet, and SqueezeNet) for the welding defectradiographic images classification tasks.
Specifically, the last three layers of these pretrained
networks are replaced with a new fully-connected layer,
a SoftMax layer, and a classification output layer.
Additionally, the original output layers are substituted
with new output layers tailored to four classes: lack ofpenetration (LP), cracks (CR), porosity (PO), and no
defect (ND). These modified networks are then trainedusing the RIAWELC dataset, as detailed in Table 1.

2. Methodology
2.1. Data acquisition and preprocessing
This study focuses on training and evaluating a deeplearning model using the RIAWELC welding defectdataset [28]. Unlike other commonly used datasets such
as GDXRay [29] and WDXI [30], RIAWELC offers alarger, open-source dataset, which is beneficial fortraining CNNs without the risk of overfitting.
The RIAWELC dataset comprises 24,407 radiographic
images, each of size 224×224 pixels, categorized into
four classes of welding defects: lack of penetration (LP),cracks (CR), porosity (PO), and no defect (ND). A
representative image for each defect type is displayed inFig. 1, and Table 1 details the distribution of images
across these classes. The substantial size of RIAWELC
dataset supports the development of automated methods
for identifying and classifying welding defects, crucial
for reliable inspection and quality control.

2.3. Hyperparameter tuning
Stochastic Gradient Descent (SGD) is a popular
optimizer in deep learning training, valued for its ability
to effectively balance accuracy and efficiency. In thisstudy, SGD is employed to train selected pretrained
network architectures with the RIAWELC dataset,aiming to minimize the cross-entropy loss function.
However, the performance and convergence of SGD are
influenced by various training hyperparameters,
including momentum, initial learning rate, L2
regularization, maximum epoch, and batch size.
Specifically, the initial learning rate dictates the step
sizes in the parameter update process. L2 regularization
helps to prevent network overfitting by adding a penalty
term to the loss function. The maximum epoch limits thenumber of iterations to avoid overfitting, and the batch
size determines the number of samples used per iteration,
impacting the stability, speed, and memory usage of SGD.

(a)
(b)
(c)
(d)
Fig.1 Sample of radiographic images for each welding
defect class from RIAWELC dataset: (a) LP, (b) CR, (c)PO and (d) ND.
Table 1. Data distribution of RIAWELC dataset
for each welding defect class.
Defect Types
CR
PO
LP
ND
No. of Image
7,635
6,320
4,452
6,000

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
878

Deep Learning in Manufacturing

To facilitate a fair comparison in performance
evaluations, the hyperparameters for all selected
pretrained networks are standardized. This includes
setting the momentum to 0.9, the initial learning rate to
𝟐 × 𝟏𝟎−𝟑 , L2 regularization to 0.5, the maximum epoch
to 5, and the batch size to 32.

3. Results and Discussions
The performance of all selected pretrained networks,
including AlexNet, ResNet-18, ResNet-50, ResNet-101,
MobileNet-v2, ShuffleNet, and SqueezeNet, inclassifying radiographic images of welding defects iscomprehensively evaluated. To facilitate a quantitative
comparison, each CNN architecture’s effectiveness in thewelding defect classification task is assessed using thefive performance metrics previously mentioned, namely
accuracy (Acc.), sensitivity (Sens.), specificity (Spec.),
precision (Prec.) and F1 score. The simulation results aredetailed in Table 3, where the highest values attained byeach CNN architecture for these metrics are highlighted
in boldface.

2.4. Performance metrics
The overall performance of all pretrained CNN
networks in classifying welding defect radiographic
images is evaluated using five key metrics: accuracy,
sensitivity, specificity, precision and F1 score. These
metrics are derived from the true positive (TP), true
negative (TN), false positive (FP), and false negative(FN) outcomes produced during the testing phase of the
CNN architectures. The mathematical formulations foreach performance metric employed in this study are as
follows.
Accuracy, indicating the overall correctness of the
CNN architecture's predictions, is calculated as:
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 =𝑇𝑃 + 𝑇𝑁𝑇𝑃 + 𝑇𝑁 + 𝐹𝑃 + 𝐹𝑁

Table 3. Quantitative performance evaluation of
all selected CNN architectures for welding defectclassification task
CNN
Acc. Sens.
Spec.
Prec.
F1
architecture
(%)
(%)
(%)
(%)
(%)
AlexNet71.43 71.43 90.48 80.21 70.92
ResNet-18
78.57 78.57 92.86 80.56 78.92
ResNet-50
82.14 82.14 94.05 83.93 82.49
ResNet-101
85.71 85.71 95.24 87.50 86.06
MobileNet-v2
71.43 71.43 90.48 69.00 69.06
ShuffleNet
75.00 75.00 91.67 80.63 75.98
SqueezeNet
71.43 71.43 90.48 75.00 70.92(1)

Sensitivity, measuring the CNN architecture's ability tocorrectly identify positive results, is computed as:
𝑇𝑃
𝑇𝑃 + 𝐹𝑁

𝑆𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦 =Table 3 reveals a significant observation across all
CNN architectures: the identical values of accuracy andsensitivity when classifying the RIAWELC dataset. This
uniformity implies that the models' ability to correctlypredict positive instances (sensitivity) significantly
impacts their overall accuracy. Among these, ResNet-101(85.71%) and ResNet-50 (82.14%) stand out with the bestand second-best performance in both accuracy andsensitivity, highlighting their superior detection
capabilities for welding defects in radiographic images.
Conversely, architectures with lower complexity, such as
AlexNet, MobileNet-v2, and SqueezeNet, demonstrateless effectiveness, each recording 71.43% in bothaccuracy and sensitivity.
Moreover, Table 3 indicates that all selected CNN
architectures consistently achieve higher specificity andprecision compared to their accuracy and sensitivity. Thissuggests they are more effective in correctly identifying
true negatives (specificity) and ensuring accurate positive
predictions (precision). Specifically, ResNet-101 exhibits
the highest specificity (95.24%), while AlexNet shows
the lowest (90.48%), indicating a broad capability acrossthese models to identify non-defective instances
accurately. Precision, however, varies more across themodels, with ResNet-101 leading at 87.50%, suggesting
its higher accuracy in classifying weld defects.
Conversely, MobileNet-v2, despite similar accuracy andsensitivity to other models, records the lowest precision(69.00%), implying a higher rate of false positives.
The F1 score, reflecting the harmonic mean of precisionand sensitivity, offers a more balanced evaluation ofmodel performance. ResNet-101 and ResNet-50 achieve(2)

Specificity, gauging the CNN architecture’s capacity toaccurately identify negative results, is determined as:𝑆𝑝𝑒𝑐𝑖𝑓𝑖𝑐𝑖𝑡𝑦 =𝑇𝑁
𝑇𝑁 + 𝐹𝑃(3)

Precision, assessing the accuracy of positive predictions
among all positive cases identified, is calculated as:

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =𝑇𝑃
𝑇𝑃 + 𝐹𝑃(4)

F1 Score, the harmonic mean of precision and sensitivity,
is defined as:

𝐹1 𝑆𝑐𝑜𝑟𝑒 = 2 × (𝑆𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦 × 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛)
𝑆𝑒𝑛𝑠𝑖𝑡𝑖𝑣𝑖𝑡𝑦 + 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛(5)

The abovementioned five performance metrics provide acomprehensive assessment of the CNN architectures’capabilities in distinguishing different classes of welding
defects.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
879

Tin Chang Ting, Hameedur Rahman, Tiong Hoo Lim, Chin Hong Wong, Chun Kit Ang, Mohamed Khan Afthab Ahamed Khan, Sew Sun Tiang,
Wei Hong Lim

the highest F1 scores (86.06% and 82.49%, respectively),
confirming their robust overall performance inclassifying the radiographic images of welding defects.
In contrast, AlexNet, MobileNet-v2, and SqueezeNet
exhibit lower F1 scores (70.92%, 69.06%, and 70.92%,
respectively), indicating a compromise between
sensitivity and precision in these models.
Analysis of the simulation results in Table 3 reveals that
more complex CNN architectures, such as ResNet-101
and ResNet-50, consistently outperform simpler ones like
AlexNet and SqueezeNet across all performance metrics.
This superior performance of advanced models like
ResNet-101 and ResNet-50 highlights the significance ofchoosing the right model for complex image
classification tasks, such as welding defect classification.
It suggests that deeper networks are more adept at
extracting essential features necessary for accurately
identifying welding defects. The enhanced capability ofthese complex CNN models to discern intricate patterns
in data meets the expectations associated with deeper
network architectures.
However, it is important to note that while these
sophisticated CNN architectures offer improved
classification accuracy, they also demand greater
computational resources. In contrast, simpler CNN
models, despite being less precise, are morecomputationally efficient and may be preferable inscenarios with limited resources. In practical applications,factors like available computational power, latency
considerations, and the criticality of accurately detecting
defects should guide the selection of CNN architectures.
Balancing these considerations is key to effectively
deploying CNNs in real-world tasks.

manufacturing companies seeking to enhance their
quality control processes. In this study, thehyperparameter settings for all pretrained networks aremanually configured. It is anticipated that theirclassification performance could be further improved by
employing advanced metaheuristic search algorithms tooptimize these hyperparameter settings, thereby refining
the training process of the pretrained networks.
Acknowledgements
This work was supported by UCSI University’s
Research Excellence & Innovation Grant (REIG) withproject code of REIG-FETBE-2022/038 and Billion
Prima Sdn. Bhd.’s Industry Research Grant with projectcode of IND-FETBE-2023/006.
References
1.

Q. Feng, R. Li, B. Nie, S. Liu, L. Zhao, and H. Zhang,
Literature review: Theory and application of in-line
inspection technologies for oil and gas pipeline girth weld
defection, Sensors 17(1), 2017, pp. 50.
2. L. Yin et al., A novel feature extraction method of eddy
current testing for defect detection based on machinelearning, NDT & E International 107, 2019, pp. 102108.
3. C. Pei, D. Yi, T. Liu, X. Kou and Z. Chen, Fully
noncontact measurement of inner cracks in thick specimen
with fiber-phased-array laser ultrasonic technique, NDT &E International 113, 2020, pp. 102273.
4. K. Ding, Z. Niu, J. Hui, X. Zhou, and F. T. S. Chan, A weld
surface defect recognition method based on improved
MobileNetV2 algorithm, Mathematics 10(19), 2022, pp.
3678.
5. B. Jdid, W. H. Lim, I. Dayoub, K. Hassan and M. R. B. M.
Juhari, Robust automatic modulation recognition throughjoint contribution of hand-crafted and contextual features,
IEEE Access 9, 2021, pp. 104530-104546.
6. K. M. Ang et al, Optimal design of convolutional neuralnetwork architectures using teaching-learning-based
optimization in image classification, Symmetry 14(11),2022, pp. 2323.
7. K. M. Ang et al., An innovative approach for automatedconvolutional neural network design for image
classification, Mathematics 11(19), 2023, pp. 4115.
8. T. Berghout, M. Benbouzid and Y. Amirat, Towards
resilient and secure smart grids against PMU adversarial
attacks: A deep learning-based robust data engineering
approach, Electronics 12(12), 2023, pp. 2554.
9. T. Berghout and M. Benbouzid, EL-NAHL: Exploringlabels autoencoding in augmented hidden layers offeedforward neural networks for cybersecurity in smart
grids, Reliability Engineering & System Safety 226, 2022,
pp. 108680.
10. L. S. Chow, G. S. Tang, M. I. Solihin, N. M. Gowdh, N.
Ramli and K. Rahmat, Quantitative and qualitative
analysis of 18 deep convolutional neural network (CNN)
models with transfer learning to diagnose COVID-19 onchest X-ray (CXR) images, SN Computer Science 4(2),2023, pp. 141.
11. A. Qayyum et al., Hybrid 3D-ResNet Deep Learning
Model for Automatic Segmentation of Thoracic Organs at
Risk in CT Images, 2020 International Conference on

4. Conclusions
This paper introduces a deep learning-based machine
vision inspection algorithm that combines pretrained
CNN architectures and transfer learning for classifying
four types of welding defects—lack of penetration,
cracks, porosity, and no defect—using radiographic
images from the RIAWELC dataset. The objective is tothoroughly analyze the efficacy of seven different
pretrained CNN architectures: AlexNet, ResNet-18,
ResNet-50, ResNet-101, MobileNet-v2, ShuffleNet, andSqueezeNet, in addressing welding defect classification
tasks. This analysis employs various performance metrics,
including accuracy, sensitivity, specificity, precision, andF1 score. Simulation studies reveal that these pretrained
networks exhibit diverse performance levels in defectclassification, with ResNet-101 emerging as the mosteffective, achieving 85.71% accuracy, 85.71% sensitivity,
95.24% specificity, 87.50% precision, and an F1 score of
86.06%. In contrast, MobileNet-v2 shows the least
effectiveness, with the lowest scores across all metrics:
71.43% accuracy and sensitivity, 90.48% specificity,
69.00% precision, and an F1 score of 69.06%. These
results offer valuable insights into the applicability of
pretrained deep learning networks for welding defectclassification, providing a significant benefit to©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024880

Deep Learning in Manufacturing

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

23.

24.

25.

26.

27.

Industrial Engineering, Applications and Manufacturing(ICIEAM), Sochi, Russia, 2020, pp. 1-5.
H. Rahman et al. A systematic literature review of 3D deeplearning
techniquesincomputed
tomographyreconstruction. Tomography 9(6), 2023, pp. 2158-2189.
M. Alrifaey, W. H. Lim and C. K. Ang, A novel deeplearning framework based RNN-SAE for fault detection ofelectrical gas generator, IEEE Access, 9, 2021, pp. 2143321442.
M. Alrifaey et al., Hybrid deep learning model for fault
detection and classification of grid-connected photovoltaic
system, IEEE Access 10, 2022, pp. 13852-13869.
T. Berghout et al. Federated learning for condition
monitoring of industrial processes: A review on fault
diagnosis method, challenges and prospect. Electronics
12(1), pp. 158.
T. Berghout, M. Benbouzid, Y. Amirat and G. Yao,
Lithium-ion battery state of health prediction with a robust
collaborative augmented hidden layer feedforward neuralnetwork approach. IEEE Transactions on Transportation
Electrification 9(3), 2023, pp. 4492-4502.
A. A. Abdelhamid et al. Deep learning with dipper throatedoptimization algorithm for energy consumption
forecasting in smart households. Energies 15(23), 2022, pp.
9125.
A. Krizhevsky, I. Sutskever and G. E. Hinton, ImageNetclassification with deep convolutional neural networks,
Communication of the ACM 60(6), 2017, pp. 84-90.
K. He, X. Zhang, S. Ren and J. Sun, Deep Residual
Learning for Image Recognition, 2016 IEEE Conference
on Computer Vision and Pattern Recognition (CVPR), Las
Vegas, NV, USA, 2016, pp. 770-778.
M. Sandler et al, MobileNetV2: Inverted Residuals andLinear Bottlenecks. 2018 IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR), Salt
Lake City, UT, USA, 2018, pp. 4510-4520.
X. Zhang, X. Zhou, M. Lin and J. Sun, ShuffleNet: An
Extremely Efficient Convolutional Neural Network for
Mobile Devices. 2018 IEEE/CVF Conference on
Computer Vision and Pattern Recognition (CVPR), Salt
Lake City, UT, USA, 2018, pp. 6848-6856.
F.Iandola et al. SqueezeNet: AlexNet-level Accuracy with
50x Fewer Parameters and <IMB Model Size, Preprint,
submitted
November
4,
2016.
https://arxiv.org/pdf/1602.07360
S. Kumaresan, K. S. Jai Aultrin, S. S. Kumar and M. Dev
Anand, Deep learning-based weld defect classification
using VGG16 transfer learning adaptive fine tuning.
International Journal on Interactive Design and
Manufacturing 17, 2023, pp. 2999-3010.
W. Hou, Y. Wei, J. Guo, Y. Jin and C. Zhu, Automatic
detection of welding defects using deep neural network,
Journal of Physics: Conference Series 933, 2017, pp.
012006.
W. Du, H. Shen, J. Fu, G. Zhang and Q. He, Approaches
for improvement of the X-ray image defect detection of
automobile casting aluminium parts based on deeplearning, NDT & E International 107, 2019, pp. 102144.
S. Kumaresan, K. S. Jai Aultrin, S. S. Kumar and M. Dev
Anand, Transfer learning with CNN for classification of
weld defect, IEEE Access 9, 2021, pp. 95097-951078.
D. Say, S. Zidi, S. M. Qaisar and M. Krichen, Automatedcategorization of multiclass welding defects using the Xray image augmentation and convolutional neural network,
Sensors 23(14), 2023, pp. 6422.

28. S. Perri, F. Spagnolo, F. Frustaci and P. Corsonello,
Welding defects classification through a convolutional
neural network, Manufacturing Letters 35(42), 2022, pp.
29-32.
29. D. Mery et al, GDXray: The database of X-ray images for
nondestructive testing, Journal of Nondestructive
Evaluation 34(42), 2015, pp. 42.
30. W. Guo, H. Qu and L. Liang, WDXI: The Dataset of Xray Image for Weld Defects, 2018 14th International
Conference on Natural Computation, Fuzzy Systems andKnowledge Discovery (ICNC-FSKD), Huangshan, China,
2018, pp. 1051-1055.

Authors Introduction
Mr. Tin Chang Ting
He is currently pursuing the Bachelor of
Mechatronics
Engineeringwith
Honours in Faculty of Engineering,Technology and Built Environment,
UCSI University, Malaysia. His
research interests are machine learning,
deep learning, and optimization
algorithm.
Dr. Hameedur Rahman
He is an Associate Professor in Facultyof Computing and AI at AIR
University in Pakistan. He received his
PhD in Computer Science from
Universiti Kebangsaan Malaysia in
2018. His research interests are
Virual/Augmented Rality, Image
Processing, Data Mining, Artificial
Intelligence, Natural Language Processing andCyberSecurity.
Dr. Tiong Hoo Lim
He is a Senior Assistant Professor andDirector of Planning Development
Office at Universiti Teknologi Brunei
in Brunie Darussalam. He received his
PhD in Computer Science from
University of York, United Kingdom.
His research interests are artificial
intelligence, advanced technology,
competitive and forecasting analysis.
Dr. Chin Hong Wong
He is a Lecturer in MaynoothInternational Engineering College at
Fuzhou University in China. He
received his PhD in Electrical andElectronic Engineering from Universiti
Sains Malaysia in 2017. His researchinterests are Energy harvesting andcontrol system.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
881

Powered by TCPDF (www.tcpdf.org)

Tin Chang Ting, Hameedur Rahman, Tiong Hoo Lim, Chin Hong Wong, Chun Kit Ang, Mohamed Khan Afthab Ahamed Khan, Sew Sun Tiang,
Wei Hong Lim

Dr. Chun Kit Ang
He is the Dean and Associate Professor
in Faculty of Engineering at UCSI
University in Malaysia. He received
his PhD in Mechanical and
Manufacturing Engineering from
Universiti Putra Malaysia in 2014. His
research interests are artificial
intelligence, soft computing, robotics
and mechatronics.
Dr. Mohamed Khan Afthab Ahmed Khan
He is an Assistant Professor in Facultyof Engineering at UCSI University in
Malaysia. His research interests are
artificial intelligence, robotics, control,
medical rehabilitation and powerelectroncs.

Dr. Sew Sun Tiang
She is an Assistant Professor in Facultyof Engineering at UCSI University in
Malaysia. She received her PhD in
Electrical and Electronic Engineeringfrom Universiti Sains Malaysia in
2014. Her research interests are
optimization and antenna design.

Dr. Wei Hong Lim
He is an Associate Professor in Facultyof Engineering at UCSI University in
Malaysia. He received his PhD in
Computational
Intelligence
fromUniversiti Sains Malaysia in 2014. His
research interests are optimization andartificial intelligence.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
882

Enhancing Global Optimization Performance of Arithmetic Optimization Algorithm with a
Modified Population Initialization Scheme
Tin Chang Ting1, Hameedur Rahman2, Meng Choung Chiong1, Mohamed Khan Afthab Ahamed Khan1, Cik Suhana Hassan1,
Farah Adilah Jamaludin1, Sew Sun Tiang1*, Wei Hong Lim1*
2

1
Faculty of Engineering, Technology and Built Environment, UCSI University, Kuala Lumpur 56000, Malaysia
Faculty of Computing and Artificial Intelligence, Air University, Islamabad Capital Territory 44000, Pakistan
Email: 1002058096@ucsiuniversity.edu.my, rhameedur@gmail.com, ChiongMC@ucsiuniversity.edu.my,mohamedkhan@ucsiuniversity.edu.my, suhana@ucsiuniversity.edu.my, FarahAdilah@ucsiuniversity.edu.my,tiangss@ucsiuniversity.edu.my, limwh@ucsiuniversity.edu.my

Abstract
Arithmetic Optimization Algorithm (AOA) is widely used to solve global optimization problems. However, it often
faces premature convergence challenges in complex optimization scenarios. A key factor affecting AOA's
performance is the solution quality of the initial population. The conventional initialization scheme, despite its
prevalence, lacks reliability in ensuring high-quality solutions due to inherent stochastic processes. To address thisissue, we propose a modified initialization scheme that improves initial population quality by integrating chaotic maps
and oppositional-based learning. Through extensive simulation studies, we demonstrate that the enhanced AOA,
equipped with this new initialization scheme, exhibits superior performance in solving a range of benchmark functions
with improved accuracy.
Keywords: Arithmetic optimization algorithm, Population initialization, Chaotic map, Oppositional-based learning

1. Introduction

Recognizing the limitations of traditional optimization
methods, there is an urgent need in the Industrial
Revolution 4.0 era to develop more intelligent and robust
optimization algorithms. These algorithms must be
capable of providing efficient solutions to a broad
spectrum of increasingly complex optimization problems.
Metaheuristic Search Algorithms (MSAs) have emerged
as effective solutions, drawing on search mechanisms
inspired by natural phenomena. Existing MSAs fall into
four categories based on their natural inspirations [3]:evolutionary algorithms, swarm intelligence, humanbased algorithms, and physics-based algorithms.
Compared to traditional methods, MSAs offer several
advantages, including potent global search capabilities,
straightforward
implementation,
andenhancedscalability. They exploit the unique strengths of theirrespective inspirations, enabling them to effectively
address a variety of complex optimization challenges as
outlined in [4], [5], [6], [7], [8], [9], [10], [11], [12], [13].
The Arithmetic Optimization Algorithm (AOA) [14],introduced in 2021, is a physics-based algorithm withsearch mechanisms inspired by the four basic arithmetic
operations: division, multiplication, addition, andsubtraction. These operations vary in their explorationand exploitation strengths, essential for solving
optimization problems. Like other MSAs, achieving aproper balance between exploration and exploitation is
crucial for enhancing AOA's performance. Consequently,
several enhancement schemes, including hybridization

Optimization is pivotal in real-life engineering design,where it seeks the most effective solutions whileaccommodating various stakeholders' criteria. Accurate
problem definition and modeling are essential for
resolving these design challenges. This process includes
establishing clear objectives, identifying both technicaland non-technical constraints, and optimizing decisionvariables. Nonetheless, real-world engineering design
problems frequently present complexity and challenges
due to factors like high dimensionality, numerous
constraints, conflicting objectives, and data uncertainty.
In the era of Industrial Revolution 4.0 (IR 4.0),
engineering systems have become more intricate, often
involving non-differentiable, nonlinear, multimodal, andnon-continuous functions. While traditional optimization
methods such as Newton's method are prevalent, they
face limitations in addressing these complex, real-world
engineering design challenges. These methods often
depend heavily on initial solutions and are typicallysuited to specific problem types, limiting their scalability
for diverse, complicated optimization tasks [1]. Most
traditional methods also rely on gradient information to
identify optimal solutions, which is not always feasible inreal-world scenarios that frequently present as black box
functions [2]. Furthermore, the limited global searchcapability of traditional optimization methods increases
the risk of premature convergence.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
883

Tin Chang Ting, Hameedur Rahman, Meng Choung Chiong, Mohamed Khan Afthab Ahamed Khan, Cik Suhana Hassan, Farah Adilah
Jamaludin, Sew Sun Tiang, Wei Hong Lim[15], [16], [17], the introduction of new learning
mechanisms [18], [19], [20] and others, have beendeveloped to improve the balance in AOA variants. Since
its introduction, both the original AOA and its enhanced
variants have been successfully applied to a wide array ofengineering optimization problems [15], [16], [19], [21],[22], [23], [24], [25].
The quality of the initial population is a key factor in
determining the performance of MSAs. Despite
numerous variants of the AOA being introduced in recent
years, many still rely on conventional methods torandomly generate their initial populations. This
conventional approach, while simple to implement, does
not intelligently leverage environmental information
around the solution regions during initialization [3]. As aresult, it often mistakenly places some solutions in local
or suboptimal regions, leading to premature convergence.
Additionally, if initial solutions are far from the global
optimum, the algorithm's convergence speed may becompromised. These drawbacks can significantly impact
the robustness and effectiveness of AOA in addressing
complex real-world optimization challenges.
This paper presents the Multi-Chaotic Dynamic
Oppositional Learning (MCDOL) module as anenhancement to the AOA’s population initialization
scheme, culminating in a new variant named MCDOLAOA. The MDCOL module is designed to produce an
initial population with enhanced solution quality, both interms of accuracy and convergence rate, by harnessing
the strengths of multiple chaotic maps and the dynamic
oppositional-based learning (DOL) mechanism. Notably,
the diverse ergodic properties of various chaotic maps are
employed for a comprehensive solution space search,enhancing population diversity and mitigating the risk ofinitializing solutions in local optima. Additionally, theinclusion of DOL in MCDOL module promotes more
rapid global optimum identification by expanding the
exploration of solution regions. The effectiveness of
MCDOL-AOA is evaluated through benchmarking
against the original AOA across a range of benchmark
functions with varying characteristics.

for specific optimization problems. Therefore, thesynergistic integration of multiple chaotic maps promises
to significantly improve the algorithm’s robustness inaddressing various complex problems.
The MCDOL module incorporates five distinct chaotic
maps — Circle, Logistic, Piecewise, Sine, and Tent —for generating a chaotic population. Each chaotic map israndomly chosen to generate different dimensions of theinitial solutions within this population. To facilitate this,define 𝝑𝒕 as the output of a chaotic variable at the t-th
iteration, where 𝒕 = 𝟏, … , 𝑻 . The population
initialization in the MCDOL module, using these
multiple chaotic maps, follows five criteria: (a) the Circle
map is used if 𝟎 ≤ 𝝑𝟎 < 𝟎. 𝟐, (b) the Logistic map for𝟎. 𝟐 ≤ 𝝑𝟎 < 𝟎. 𝟒, (c) the Piecewise map for 𝟎. 𝟒 ≤ 𝝑𝟎 <𝟎. 𝟔, (d) the Sine map for 𝟎. 𝟔 ≤ 𝝑𝟎 < 𝟎. 𝟖, and (e) the
Tent map for 𝟎. 𝟖 ≤ 𝝑𝟎 ≤ 𝟏. 𝟎. The specific equations
for these five chaotic maps, Circle, Logistic, Piecewise,
Sine, and Tent, are detailed in Eqs. (1) to (5).
0.5
𝜗𝑡+1 = mod (𝜗𝑡 + 0.2 − ( ) sin(2𝜋𝜗𝑡 ) , 1)
2𝜋(1)𝜗𝑡+1 = 4 𝜗𝑡 (1 − 𝜗𝑡 )

(2)

𝜗𝑡
,
𝑃
𝜗𝑡 − 𝑃,
0.5−𝑃𝜗𝑡+1 =
1 − 𝑃 − 𝜗𝑡,
0.5 − 𝑃
1 − 𝜗𝑡,{𝑃

0 ≤ 𝜗𝑡 < 𝑃𝑃 ≤ 𝜗𝑡 < 0.5(3)
0.5 ≤ 𝜗𝑡 < 1 − 𝑃
1 − 𝑃 ≤ 𝜗𝑡 < 1

𝜗𝑡+1 = sin (𝜋𝜗𝑡 )(4)𝜗𝑡
𝜗𝑡 < 0.7
0.7
𝜗𝑡+1 = {(5)
10(1 − 𝜗𝑡 ) 𝜗𝑡 ≥ 0.7
3
𝑳𝑼
Let 𝑿𝒅 and 𝑿𝒅 denote the lower and upper bounds ofthe d-th dimensional decision variable, respectively, for𝒅 = 𝟏, … , 𝑫. At the final iteration 𝒕 = 𝑻, a chaotic value
𝝑𝑻 is generated from a randomly selected chaotic map.
This value initializes the d-th dimension of each n-th
chaotic solution, as outlined in Eq. (6). The resulting
chaotic population, denoted as 𝐏 𝐂 = [𝑿𝑪𝟏 , … , 𝑿𝑪𝒏 , … 𝑿𝑪𝑵 ],encompasses all solution members formed using multiple
chaotic maps.

2. Methodology
2.1. Proposed MCDOL module
In this subsection, we detail how the proposed MCDOL
module generates an initial population of superior quality.
The MCDOL module leverages multiple chaotic maps tocreate a diverse set of initial solutions, forming a chaotic
population. This approach contrasts with conventional
initialization schemes, which often result in poorly
distributed initial solutions. The deterministic nature ofchaotic maps, coupled with their ability to exhibit
stochastic behavior through ergodic properties, allows fora more comprehensive exploration of the solution space.
Unlike some prior studies [26], [27], [28] that rely on asingle chaotic map, this paper explores the benefits ofusing multiple chaotic maps for population initialization.
We hypothesize that each map's unique ergodic
characteristics can enhance the algorithm's performance

𝐶𝑋𝑛,𝑑= 𝑋𝑑𝐿 + 𝜗𝑇 (𝑋𝑑𝑈 − 𝑋𝑑𝐿 )(6)

While various chaotic maps exhibit differing levels ofrobustness to local optima, they may still generate initialsolutions distant from the global optimum, potentiallyslowing the algorithm's convergence. To address thisissue, a DOL operator is applied to the chaotic population
𝐏 𝐂 , generating opposite solutions for each n-th chaotic
solution. This DOL operator is expected to broaden theinitial population's coverage of the solution space,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
884

Enhancing Global Optimization Performance𝑋𝑛,𝑑 (𝐶𝐼𝑡𝑒𝑟 + 1) =𝑏𝑒𝑠𝑡𝑑 − 𝑀𝑃 × [(𝑋𝑑𝑈 − 𝑋𝑑𝐿 )𝜇 + 𝑋𝑑𝐿 ], 𝑟𝑎𝑛𝑑2 < 0.5{
𝑏𝑒𝑠𝑡𝑑 + 𝑀𝑃 × [(𝑋𝑑𝑈 − 𝑋𝑑𝐿 )𝜇 + 𝑋𝑑𝐿 ], Otherwise

increasing the likelihood of identifying fitter solutions.
The opposite solution for each dimension, denoted as
𝑪𝑿𝑶𝒏,𝒅 , is determined using Eq. (7), corresponding to 𝑿𝒏,𝒅 ,with 𝒓𝟏 , 𝒓𝟐 ∈ [𝟎, 𝟏] . Consequently, this generates an𝑶𝑶
opposition population, 𝐏 𝐎 = [𝑿𝑶
𝟏 , … , 𝑿𝒏 , … 𝑿𝑵 ].
𝑂
𝐶
𝐶
𝐶
𝑋𝑛,𝑑
= 𝑋𝑛,𝑑+ 𝑟1 [𝑟2 (𝑋𝑑𝑈 + 𝑋𝑑𝐿 − 𝑋𝑛,𝑑) − 𝑋𝑛,𝑑]

As illustrated in Fig. 1, MCDOL-AOA continues thisiterative search, following Eqs. (8) to (11), until pre-set
termination criteria are fulfilled. Upon completion,
optimal decision variables in the best solution are
decoded to solve the specific optimization problems.

(7)

The two populations, one generated using multiple
chaotic maps (𝐏 𝐂 ) and the other using the DOL operator(𝐏 𝐎 ), are merged to form a combined population set 𝐏 𝐂 ∪𝐏 𝐎 with a total size of 2N. The fitness of each solution
member in this merged population set is evaluatedaccording to the predefined objective function. These
solution members are then ordered based on their fitness
values, from the best to worst performing ones. The top
N solution members from this ordered population set
𝐏 𝐂 ∪ 𝐏 𝐎 are chosen as the initial population for the
proposed MCDOL-AOA algorithm, represented as 𝐏 =[𝑿𝟏 , … , 𝑿𝒏 , … . , 𝑿𝑵 ].

MCDOL-AOA for Global Optimization
Inputs: D, N, 𝑀𝐼𝑡𝑒𝑟 , T, 𝑀𝑎𝑥, 𝑀𝑖𝑛, θ
01: Initialize 𝐏𝐂 ← ∅, 𝐏𝐎 ← ∅ and 𝐶𝐼𝑡𝑒𝑟 ← 0;
02: for each n-th solution do
03:
for each d-th dimension do
04:
Randomly initialize 𝜗𝑡 ∈ [0,1], where t = 0;05:
if 0 ≤ 𝜗0 < 0.2 then06:
Select Circle map in Eq. (1);07:
elseif 𝟎. 𝟐 ≤ 𝝑𝟎 < 𝟎. 𝟒 then08:
Select Logistic map in Eq. (2);09:elseif 𝟎. 4 ≤ 𝜗0 < 0.6 then10:
Select Piecewise map in Eq. (3);11:elseif 𝟎. 6 ≤ 𝜗0 < 0.8 then12:
Select Sine map in Eq. (4);13:elseif 0.8 ≤ 𝜗0 ≤ 1.0 then14:
Select Tent map in Eq. (5);15:
end if16:
while 𝑡 ≤ 𝑇 do
17:
Update 𝜗𝑡 with the selected chaotic map;18:𝑡 ← 𝑡 + 1;
19:
end while
𝐶
20:
Generate 𝑋𝑛,𝑑using Eq. (6);𝑂
21:
Generate 𝑋𝑛,𝑑using Eq. (7);22:
end for𝐶
𝑂
23:
Update 𝐏𝐂 ← 𝐏𝐂 ∪ 𝑋𝑛,𝑑and 𝐏𝐎 ← 𝐏𝐎 ∪ 𝑋𝑛,𝑑;
24: end for
25: Merge the two population sets as 𝐏𝐂 ∪ 𝐏𝐎 ;26: Evaluate the fitness values of all solutions stored
within 𝐏𝐂 ∪ 𝐏𝐎 ;27: Rearrange the solutions stored within 𝐏𝐂 ∪ 𝐏𝐎 frombest to worst based on their fitness values;
28: Select the top N solutions from the sorted 𝐏𝐂 ∪ 𝐏𝐎 as
the initial population, i.e., 𝐏 = [𝑋1 , … , 𝑋𝑛 , … . , 𝑋𝑁 ].
29: Assign the first solution of P and its fitness as best and𝑓(𝑏𝑒𝑠𝑡), respectively;
30: while 𝐶𝐼𝑡𝑒𝑟 ≤ 𝑀𝐼𝑡𝑒𝑟 do
31:
Update MOA and MP with Eqs. (8) and (9);32:
for each n-th solution do
33:
if 𝑟𝑎𝑛𝑑1 > 𝑀𝑂𝐴 then /*Exploration*/
34:
Update 𝑋𝑛,𝑑 (𝐶𝐼𝑡𝑒𝑟 + 1) with Eq. (10);35:else /*Exploitation*/
36:
Update 𝑋𝑛,𝑑 (𝐶𝐼𝑡𝑒𝑟 + 1) with Eq. (11);37:
end if38:
Fitness evaluation of 𝑋𝑛 (𝐶𝐼𝑡𝑒𝑟 + 1);
39:
Update the 𝑋𝑛 , 𝑓(𝑋𝑛 ), 𝑏𝑒𝑠𝑡 and 𝑓(𝑏𝑒𝑠𝑡),
40:
with greedy selection method;41:
end for42:
𝐶𝐼𝑡𝑒𝑟 ← 𝐶𝐼𝑡𝑒𝑟 + 1;
43: end while
Output: 𝑏𝑒𝑠𝑡 and 𝑓(𝑏𝑒𝑠𝑡)

2.2. Iterative Search Processes of MCDOL-AOA
After generating an initial population P of superior
quality using the MCDOL module, each n-th solution in
MCDOL-AOA is iteratively updated through searchmechanisms similar to those in the original AOA.
During each iteration, the Math Optimizer Accelerated(MOA) function value is adjusted in MCDOL-AOA toalternate between exploration and exploitation phases:
𝑀𝑂𝐴(𝐶𝐼𝑡𝑒𝑟 ) = 𝑀𝑖𝑛 + 𝐶𝐼𝑡𝑒𝑟 (𝑀𝑎𝑥−𝑀𝑖𝑛
𝑀𝐼𝑡𝑒𝑟

)

(11)

(8)

where 𝐶𝐼𝑡𝑒𝑟 and 𝑀𝐼𝑡𝑒𝑟 represent the current andmaximum iteration counts; 𝑀𝑖𝑛 and 𝑀𝑎𝑥 are theminimum and maximum values of MOA. Simultaneously,
the Math Optimizer Probability (MP) function, which
dictates the search range for each solution and is
influenced by the critical parameter θ for exploitation
efficiency, is updated as:
1

𝐶𝐼𝑡𝑒𝑟 ⁄𝜃
(9)𝑀𝑃(𝐶𝐼𝑡𝑒𝑟 ) = 1 − ()
𝑀𝐼𝑡𝑒𝑟
A random number 𝒓𝒂𝒏𝒅𝟏 determines the searchstrategy (exploration or exploitation) at each iteration for
updating the d-th dimension of the n-th solution, 𝑿𝒏,𝒅 .
During the exploration phase (𝒓𝒂𝒏𝒅𝟏 > 𝑴𝑶𝑨), either
the Multiplication or Division operator is selected:
𝑋𝑛,𝑑 (𝐶𝐼𝑡𝑒𝑟 + 1)
𝑏𝑒𝑠𝑡𝑑 ÷ (𝑀𝑃 + 𝜀) × [(𝑋𝑑𝑈 − 𝑋𝑑𝐿 )𝜇 + 𝑋𝑑𝐿 ], 𝑟𝑎𝑛𝑑2 < 0.5={
𝑏𝑒𝑠𝑡𝑑 × 𝑀𝑃 × [(𝑋𝑑𝑈 − 𝑋𝑑𝐿 )𝜇 + 𝑋𝑑𝐿 ], Otherwise(10)
where 𝑟𝑎𝑛𝑑2 is a random number between 0 and 1;
𝑏𝑒𝑠𝑡𝑑 denotes the d-th dimension of the current bestsolution; 𝜀 is a small positive number to prevent divisionby zero, and 𝜇 is a control parameter.
In the exploitation phase (𝑟𝑎𝑛𝑑1 ≤ 𝑀𝑂𝐴), either the
Addition or Subtraction operator is used to update 𝑋𝑛,𝑑 :Fig.1 Workflow of proposed MCDOL-AOA in solving
the global optimization problems.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
885

Tin Chang Ting, Hameedur Rahman, Meng Choung Chiong, Mohamed Khan Afthab Ahamed Khan, Cik Suhana Hassan, Farah Adilah
Jamaludin, Sew Sun Tiang, Wei Hong Lim

3. Results and Discussions
3.1. Simulations settings

evidenced by lower Emean values. In the scalable unimodal
function’s category (F1 to F7), MCDOL-AOA surpasses
the original AOA in 5 of the 7 functions based on Emean.
For the 6 scalable multimodal functions (F8 to F13),
MCDOL-AOA demonstrates superior Emean values in 4
functions (F8, F9, F12, and F13) and equals the original
AOA in one function (F10). When dealing with the fixeddimension multimodal functions (F14 to F23), MCDOLAOA outperforms the original AOA in 4 functions (F15,
F18, F22, and F23) and equals its performance in 3 otherfunctions (F16, F17, and F19).
Analysis of the simulation results in Table 1 indicatesthat MCDOL-AOA significantly outperforms theoriginal AOA in solving larger-scale benchmark
functions (F1 to F13) with D = 100, evidenced by lower
Emean values in 9 out of 13 functions. However, theperformance of MCDOL-AOA is relatively on par withthe original AOA in fixed-dimension problems (F14 toF23) with smaller dimensions (D = 2, 3, 4, and 6). Thesefindings suggest that the MCDOL module in MCDOLAOA effectively produces initial populations withsuperior fitness and diversity compared to the original
AOA, yielding enhanced optimization results. The
benefits of the MCDOL module are particularly notablein larger dimensional sizes (e.g., D = 100), as utilized inthis study. The multiple chaotic maps’ non-repetitive andergodic properties within the MCDOL module facilitatea more exhaustive exploration of the solution spaceacross various problem types, thereby minimizing therisk of local optima entrapment and premature
convergence. Additionally, the DOL mechanism within
the MCDOL module contributes to faster algorithmic
convergence by broadening solution space exploration
through the generation of opposite solutions from thoseinitiated by the chaotic maps.

In this section, we compare the performance of
MCDOL-AOA with the original AOA using 23
benchmark functions, each with distinct characteristics as
outlined in [17]. Functions F1 to F7 are scalable unimodal
functions, while F8 to F13 are scalable multimodal
functions, all set at a dimension size of D = 100.
Functions F14 to F23, on the other hand, are fixeddimension multimodal functions with D ranging from 2
to 6. Both MCDOL-AOA and the original AOA were
implemented on MATLAB 2021a, running on a personal
computer equipped with an Intel® Core™ i7-HQ CPU at
2.50 GHz and 16 GB RAM. For both algorithms, thepopulation size and maximum number of iterations are
set at 30 and 1000, respectively.
3.2. Performance analysis
The evaluation of MCDOL-AOA's performance, andits comparison with the original AOA across all 23
benchmark functions, is detailed in Table 1. We employ
two performance metrics: mean error (Emean) and standard
deviation (SD), to assess the algorithms' accuracy andconsistency in solving these functions. Lower values of
Emean and SD are preferable, indicating the algorithm's
consistent and accurate resolution of the givenbenchmark functions.

Fun
F1
F2
F3
F4
F5
F6
F7
F8
F9
F10
F11
F12
F13
F14
F15
F16
F17
F18
F19
F20
F21
F22
F23

Table 1. Performance comparison of MCDOLAOA and original AOA using 23 benchmark
functions with different characteristics.
Original AOAMCDOL-AOA
Emean
SD
Emean
SD
2.048e-04 5.662e-05
1.938e-04
5.371e-05
0.01120.01150.01070.0011
0.09150.02290.09340.02020.05230.0076
0.05160.008998.0900.089998.0600.115815.1310.774714.9300.81602.609e-05 3.060e-05
3.574e-05
3.667e-05-1.53e+04646.631-1.53e+04631.5096.244e-05 1.478e-05
6.195e-05
1.080e-05
0.00161.452e-04
0.00161.767e-04
0.00850.02490.0176
0.05190.891
0.06550.08880.07409.906
0.00309.904
0.00309.864
4.317
11.1922.979
0.00550.01250.00280.0075-1.0325.375e-12-1.0328.016e-12
0.398
5.620e-07
0.398
3.516e-07
18.3
28.08016.5
25.318-3.8635.441e-05-3.8634.105e-06-3.2940.0511-3.2740.0592-8.5542.522-8.141
2.979-8.0773.155-8.3503.267-8.5903.094-8.751
3.091

4. Conclusion
In this paper, we introduce an enhanced version of the
AOA, termed MCDOL-AOA, designed to solve complex
global optimization problems with improved accuracy.
The novelty of MCDOL-AOA resides in its integrationof multiple chaotic maps and DOL mechanisms into a
modified initialization scheme, the MCDOL module.
This module aims to generate an initial population ofsuperior quality, focusing on fitness and diversity.
Simulation results demonstrate that MCDOL-AOA,
benefiting from the enhanced initial population quality
provided by the MCDOL module, surpasses the original
AOA in solving 23 distinct benchmark functions.
Notably, the performance gains of MCDOL-AOA are
more pronounced in solving functions with larger
dimensions, specifically D = 100. This underscores theefficacy of the multiple chaotic maps and DOL
mechanisms within the MCDOL module in boosting thealgorithm’s robustness against premature convergenceand enhancing its convergence speed. Future work will
explore the application of MCDOL-AOA to real-world
engineering optimization problems, including machine

The results in Table 1 showcase MCDOL-AOA'simpressive accuracy, outperforming or matching theoriginal AOA in 16 of the 23 benchmark functions, as©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
886

Enhancing Global Optimization Performance

13. A. Sharma et al., An effective method for parameter
estimation of solar PV cell using grey-wolf optimization
technique, International Journal of Mathematical,
Engineering and Management Science 6(3), 2021, pp. 911931.
14. L. Abualigah, A. Diabat, S. Mirjalili, M. Abd Elaziz andA. H. Gandomi, The arithmetic optimization algorithm,
Computer Methods in Applied Mechanics andEngineering 376, 2021, pp. 113609.
15. A. A. Mohamed et al., A novel hybrid arithmetic
optimization algorithm and salp swarm algorithm for data
placement in cloud computing, Soft Computing, 27(9),2023, pp. 5769-5780.
16. M. Issa, Enhanced arithmetic optimization algorithm forparameter estimation of PID controller, Arabian Journalfor Science and Engineering, 48(2), 2023, pp. 2191-2205.
17. L. Abualigah and A. Diabat, Improved multi-corearithmetic optimization algorithm-based ensemble
mutation for multidisciplinary applications, Journal ofIntelligent Manufacturing 34(4), 2023, pp. 1833-1874.
18. S. B. Aydemir, A novel arithmetic optimization algorithm
based on chaotic maps for global optimization,
Evolutionary Intelligence 16(3), 2023, pp. 981-996.
19. W.-K. Hao, J.-S. Wang, X.-D. Li, H.-M. Song and Y.-Y.
Bao, Probability distribution arithmetic optimization
algorithm based on variable order penalty functions tosolve combined economic emission dispatch problem,
Applied Energy, 316, 2022, pp. 119061.
20. Y.-J. Zhang, Y.-F. Wang, Y.-X. Yan, J. Zhao and Z.-M.
Gao, LMRAOA: An improved arithmetic optimization
algorithm with multi-leader and high-speed jumping basedon opposition-based learning solving engineering andnumerical problems, Alexandria Engineering Journal61(12), 2022, pp. 12367-12403.
21. E. S. Ghith and F. A. A. Tolba, Tuning PID controllers
based on hybrid arithmetic optimization algorithm andartificial gorilla troop optimization for micro-robotic
systems. IEEE Access 11, 2023, pp. 27138-27154.
22. S. Rajput et al., A novel opposition-based arithmetic
optimization algorithm for parameter extraction of PEM
fuel cell. Electronics 10(2), 2021, pp. 2834.
23. K. M. Ang et al., Training feedforward neural networks
using arithmetic optimization algorithm for medical
classification, in Advances in Intelligent Manufacturingand Mechatronics. Lecture Notes in Electrical Engineering,eds. M. A. Abdullah et al. Springer, Singapore, vol. 988,
2023.
24. X. Shi, X. Yu, and M. Esmaeili-Falak, Improved
arithmetic optimization algorithm and its application tocarbon fiber reinforced polymer-steel bond strength
estimation, Composite Structures 306, 2023, pp. 116599.
25. A. Sharma et al., Performance investigation of state-ofthe-art metaheuristic techniques for parameter extractionof solar cells/module, Scientific Reports 13, 2023, pp.
11134.
26. Y. Wang, S. Gao, Y. Yu, Z. Wang, J. Cheng and T. Yuki,
A gravitational search algorithm with chaotic neuraloscillators, IEEE Access 8, 2020, pp. 25938-25948.
27. Z. Xu, S. Gao, H. Yang and Z. Lei, SCJADE: Yet anotherstate-of-the-art differential evolution algorithm, IEEJ
Transactions on Electrical and Electronic Engineering
16(4), 2021, pp. 644-646.
28. A.B. Ozer, CIDE: chaotically initialized differential
evolution, Expert Systems with Applications 37(6), 2010,
pp. 4632-4641.

learning model training and scheduling optimization, toassess its practicality and effectiveness.
Acknowledgements
This work was supported by UCSI University’s
Research Excellence & Innovation Grant (REIG) withproject code of REIG-FETBE-2022/038 and Billion
Prima Sdn. Bhd.’s Industry Research Grant with projectcode of IND-FETBE-2023/006.
References
1.

J. C. Nash, On best practice optimization methods in R,
Journal of Statistical Software 60(2), 2014, pp. 1-14.
2. B. Petrongolo and A. P. Chirstopher, Looking into theblack box: A survey of the matching function, Journal of
Economic Literature 39(2), 2001, pp. 390-431.
3. M. F. Ahamad, N. A. Mat Isa, W. H. Lim and K. M. Ang,
Differential evolution with modified initialization scheme
using chaotic oppositional based learning strategy,
Alexandria Engineering Journal 61(12), 2022, pp. 1183511858.
4. E. S. Ghith and F. A. A. Tolba, Real-time implementationof an enhanced proportional-integral-derivative controller
based on sparrow search algorithm for micro-robotics
system, IAES International Journal of Artificial
Intelligence 11(4), 2022, pp. 1395-1404.
5. E. S. Ghith and F. A. A. Tolba, LabVIEW implementationof tuning PID controller using advanced control
optimization techniques for micro-robotics system.
International Journal of Mechanical Engineering andRobotics Research 11(9), 2022, pp. 653-661.
6. E. S. Ghith and F. A. Tolba, Real-time implementation ofan enhanced PID controller based on marine predator
algorithm (MPA) for micro-robotics system, 2022 3rdInternational Conference on Artificial Intelligence,
Robotics and Control (AIRC), Cairo, Egypt, 2022.
7. C. Hassan, S. Sapuan, N. Abd. Aziz and M. Z. Mohamed
Yusof, Effect of chemical treatment on the tensile
properties of single oil palm empty fruit bunch (OPEFB)
fibre. Trends in Textile Engineering & Fashion
Technology 3(2), 2018, pp. 1-7.
8. E. Natarajan, C. Hassan, C. K. Ang, M. S. Santhosh, S.
Ramesh and R. Sasikumar, Modeling of multiwall carbon
nanotubes reinforced natural rubber for soft robotic
applications - A comprehensive presentation 46(9), 2021,
pp. 3251-3258.
9. C. Hassan C. W. Yeo, B. Shari, M. S. Salit and N. Abdul
Aziz, Mechanical properties of unidirectional oil palm
empty fruit bunch (OPEFB) fiber reinforced epoxy
composite. IOP Conference Series: Materials Science andEngineering 206(1), 2017, pp 012045.
10. L. Yao et al., Demand bidding optimization for an
aggregator with genetic algorithm. Energies 11(1), 2018,
pp. 1-22.
11. K. M. Ang et al., Modified teaching-learning-based
optimization and applications in multi-response machining
processes, Computers & Industrial Engineering 174, 2022,
pp. 108719.
12. A. Sharma et al., Identification of photovoltaic moduleparameters by implementing a novel teaching learning
based optimization with unique exemplar generation
scheme (TLBO-UEGS), Energy Report 10, 2023, pp.
1485-1506.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
887

Powered by TCPDF (www.tcpdf.org)

Tin Chang Ting, Hameedur Rahman, Meng Choung Chiong, Mohamed Khan Afthab Ahamed Khan, Cik Suhana Hassan, Farah Adilah
Jamaludin, Sew Sun Tiang, Wei Hong Lim

Authors Introduction

Dr. Sew Sun Tiang
She is an Assistant Professor in Facultyof Engineering at UCSI University in
Malaysia. She received her PhD in
Electrical and Electronic Engineeringfrom Universiti Sains Malaysia in 2014.
Her research interests are optimization
and antenna design.

Mr. Tin Chang Ting
He is currently pursuing the Bachelor of
Mechatronics
Engineeringwith
Honours in Faculty of Engineering,Technology and Built Environment,
UCSI University, Malaysia. His
research interests are machine learning,
deep learning, and optimization
algorithm.

Dr. Wei Hong Lim
He is an Associate Professor in Facultyof Engineering at UCSI University in
Malaysia. He received his PhD in
Computational
Intelligence
fromUniversiti Sains Malaysia in 2014. His
research interests are optimization andartificial intelligence.

Dr. Hameedur Rahman
He is an Associate Professor in Facultyof Computing and AI at AIR University
in Pakistan. He received his PhD in
Computer Science from Universiti
Kebangsaan Malaysia in 2018. His
research interests are Virual/Augmented
Rality, Image Processing, Data Mining,
Artificial Intelligence, Natural Language Processing andCyberSecurity.
Dr. Meng Choung Chiong
He is an Assistant Professor in Faculty ofEngineering at UCSI University in
Malaysia. He received his PhD in
Chemical Enigneering from Universiti
Teknologi Malaysia. His researchinterests are artificial intelligence,mechatronics and alternative fuels
combustion.
Dr. Mohamed Khan Afthab Ahmed Khan
He is an Assistant Professor in Facultyof Engineering at UCSI University in
Malaysia. His research interests are
artificial intelligence, robotics, control,
medical rehabilitation and powerelectroncs.

Dr. Cik Suhana Hassan
She is an Assistant Professor in Facultyof Engineering at UCSI University in
Malaysia. She obtained her PhD in PhD
in Mechanical Engineering from
Universiti Putra Malaysia. Her researchinterests are artifical intelligence, bio
composite and finite element analysis.
Dr. Farah Adilah Binti Jamaluddin
She is an Assistant Professor in Facultyof Engineering at UCSI University in
Malaysia. She obtained her PhD in PhD
in Eletrical Power Engineering from
Universiti Putra Malaysia. Her researchinterests are artifical intelligence,electrical insulation and high voltage
engineering.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
888

Enhancing Precision Object Detection and Identification for Autonomous Vehicles throughYOLOv5 Refinement with YOLO-ALPHA
Guandong Li1, Yanzhe Xie1, Yuhao Lu1, Zongyan Wen1, Jingzhen Fan1, Yuankui Huang1, Qinghong Ma1, Wei Hong Lim2,
Chin Hong Wong1*
1

Maynooth International Engineering College, Fuzhou University, Fujian, China
Faculty of Engineering, Technology and Built Environment, UCSI University, 1, Jalan Puncak Menara Gading, UCSIHeights, 56000 Cheras, Kuala Lumpur, Malaysia.
E-mail: guandong.li.2022@mumail.ie, yanzhe.xie.2022@mumail.ie, yuhao.lu.2022@mumail.ie,
zongyan.wen.2022@mumail.ie, jingzhen.fan.2022@mumail.ie, yuankui.huang.2022@mumail.ie,
qinghong.2021@mumail.ie,, limwh@ucsiuniversity.edu.my, chinhong.wong@mu.ie

2

Abstract
Advancing swiftly in contemporary society, the rapid growth of autonomous driving technology suggests its potential
adoption across continents. The realization of fully autonomous driving relies on proficiently detecting, classifying,
and tracking road objects such as pedestrians and vehicles. This research employs the YOLOv5 neural network,
enhancing it with YOLO-ALPHA. Modifications, encompassing freeze and attention mechanisms, serve to refine
accuracy and expedite training. Furthermore, adjustments to the activation function aim to stabilize precision andrecall. The integration of an FCN based on semantic segmentation theory contributes to improved accuracy indetecting road conditions during autonomous driving. Consequently, this enables the successful and highly accurate
functionality of automatic identification.
Keywords: Object detection, Autonomous vehicle, YOLOv5, FCN, Attention mechanism, Freeze mechanism,
Activation function

Utilizing the PASCAL VOC (The PASCAL VisualObject Classes) dataset, this study systematically alters
various modules and introduces distinct mechanisms, thispaper changes a different module and adds a disparate
mechanism each time. Then, by evaluating precision,
recall, mAP, and mAP0.5 indices of the adapted models,
it identifies the optimal placement for incorporating anattention mechanism, freezing layers to prolong training
duration, and selecting the activation function that
demonstrated superior performance on the PASCAL
VOC dataset.

1. Introduction
In recent years, the rapid advancement of technology
has led to a surge in the popularity of autonomous driving
systems. Real-time image processing is a pivotal
technology in autonomous driving. Despite reaching an
advanced stage, autonomous driving technology faces
persistent challenges in low recognition accuracy andsluggish real-time image processing, giving rise to safetyconcerns. These issues give rise to safety concerns.
Although automated driving technology has reached acertain level of maturity, it encounters limitations inadverse weather conditions, complex environments, andnighttime operations. automatic driving sensor
recognition accuracy tends to decline, compromisingautonomous vehicle safety. Consequently, there is apressing need to enhance automatic driving technology tobolster autonomous vehicle reliability and safety.

This paper presents a novel detection technique named
YOLO-ALPHA. The approach involves incorporating anattention mechanism at various layers within the YOLO
network. Specifically, inserting the Squeeze-andExcitation (SE) attention mechanism (Self-Attention
Mechanism) after the Concat module on the 16th layer ofthe network could enhance both the precision and recallof the model. Additionally, adding a freeze mechanism in
the YOLO backbone could accelerate curve fitting.
Moreover, replacing the combined structure of SiLU
(Sigmoid-weighted Linear Units) and LeakyReLU with
the SiLU activation function contributes to a smootherprecision and recall curve, thereby, improving thestability of the model by 0.169%.

This paper introduces an attention mechanism into theobject detection algorithm, fine-tunes activation
functions, adds a freezing mechanism and conducts acomparative analysis with the FCN (Fully Convolutional
Networks) algorithm. This algorithm offers an enhanced
detection speed and accuracy. The implications of thisresearch are highly relevant to the enhancement ofautonomous driving safety.

2. Literature Review©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
889

Guandong Li, Yanzhe Xie, Yuhao Lu, Zongyan Wen, Jingzhen Fan, Yuankui Huang, Qinghong Ma, Wei Hong Lim, Chin Hong Wong

Convolutional neural networks (CNNs) often sufferfrom information loss, resulting in gradientdisappearance and detection inaccuracies during image
processing. Researchers continuously strive to mitigate
semantic information loss and enhance overall network
performance. Lan et al. [1] conducted research to
improve the YOLO network, focusing on addressing
losses in pedestrian information during image processing.
They introduced additional passthrough layers, including
a Route layer and Reorg layer, or reconstructed theprimitive YOLOv2 network. The resulting YOLO-R
network specializes in pedestrian detection, effectively
improving accuracy and reducing false detection rates.
Wu et al. [2] presented a vehicle detection system in
CARLA, introducing YOLOv5-Ghost with adjusted
layer structures for reduced computational complexity.
The study achieved improved detection accuracy andspeed. Lu et al. [3] enhanced a YOLOv5-based model forcrack and vehicle detection datasets, addressing sample
imbalance and small object presence. Dodia and Kumar[4] compared modern object detectors with YOLO forvehicle detection, suggesting enhancements to improvevehicle prediction capacity. Kaloev et al. [5] investigatedactivation functions in deep learning, emphasizing theirrole in introducing nonlinearity. Xiao et al. [6] proposed
intelligent layer freezing during training to accelerate
training on various networks. Zhao et al. [7] improved an
infrared detection model based on YOLOv5s, adding an
SE-Net module. Kaymak et al. [8] experimented with
FCN architectures in the context of autonomous driving.
The study concludes that applying YOLO to vehicle
detection faces challenges, leading researchers to
improve the algorithm through various modifications.

all classes present in the dataset. A lower curvature in the
curve indicates enhanced stability and a moreconsistently changing trend, signifying ideal modeltraining stability and convergence.

3. Design Methodology
Fig. 1 shows the flow chart to tackle imminent
challenges related to object identification, classification,and tracking in the domain of autonomous driving. Given
the effectiveness of YOLOv5 in object identification
tasks, as evidenced by Xiao [9] and Kaymak [8], andrecognizing the utility of FCN in autonomous driving
scenarios, this research chooses to employ both methods
for real-time image processing in autonomous driving
contexts. To ensure a fair comparison of results acrossdifferent networks using the control variate method, aconsistent dataset, PASCAL VOC 2012, was used to train
and validate both YOLO and FCN networks. PASCAL
VOC 2012 encompasses 20 classes commonly
encountered in autonomous driving scenarios, such as
bicycle, bus, car, motorbike, and person.

Fig. 1 Flow chart
Then, the project aims to enhance the YOLO algorithm
by adding an attention mechanism, freezing backbone,
and activation function.
Incorporating an attention mechanism enables the
network to focus on crucial input features while
disregarding inconsequential ones. Through the
integration of SE attention mechanisms in different layersof the YOLO structure and subsequent training, theresearch evaluates the performance enhancement trend
by scrutinizing index curves and determining the optimal
location for implementing the SE attention mechanism.
Freezing the backbone is a training strategy aimed atexpediting training speed by halting parameters in the
backbone upgrade. By comparing the result indexes ofmodules employing freezing backbone, freezing alllayers, and not freezing layers separately, the projectseeks to identify the optimal freezing method.

Following multiple rounds of training and prediction in
YOLO, this work collects and scrutinizes detection andclassification outcomes using TensorBoard, relying on
index curves such as precision, recall, mAP, and mAP0.5
curvature. Higher precision values denote the network's
proficiency in accurately classifying items, whileelevated recall signifies the network's efficacy inrectifying previous classification errors. The mAP and
mAP0.5 metrics evaluate the detection accuracy across

Activation functions amplify the nonlinearity of
convolutional neural networks. By replacing the originalactivation function with SiLU, Mish, LeakyReLU, and
ReLU in the modified YOLOv5s, the project utilizes the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
890

Enhancing Precision Object Detection

Fig. 3 shows minimal differences observed between thefrozen backbone curve and the original model curve interms of mAP, precision, and recall at the highest training
rounds. Despite a reduction in training time by 2 hours,
image accuracy slightly decreases from 0.7166 to 0.6716.
The key conclusion drawn is that freezing trained
parameters
enhancestraining
speed
without
compromising accuracy. Freezing all layers completely
leads to virtually no accuracy, emphasizing theimportance of trained parameters. Freezing the backbone
during training enhances training efficiency, as theunchanging feature extraction network consumes lessmemory, leading to faster training.

average curvature of the precision curve to compare andassess the smoothness and stability of the network.
Ultimately, the study amalgamates the proven optimal
strategies from the preceding analysis to formulate
YOLO-ALPHA, an enhanced version of YOLOv5. The
performance of YOLO-ALPHA is then juxtaposed withthat of YOLOv5 to affirm its improvement.
4. Results and Discussions
The object detection output images from YOLOv5s
depict vehicles on the road using bounding boxes.
However, these bounding boxes lack information
regarding the distance between vehicles, a crucial
consideration in congested road scenarios, particularly
during peak commuting hours, a vital aspect ofautonomous driving. To precisely discern the outlines ofnearby vehicles and pedestrians, pixel-wise object
detection and classification with FCN are employed. Fig.
2 illustrates the contrast in detection results between
YOLOv5s and FCN. The depiction from YOLOv5s
shows vehicles with pink bounding boxes, outlining theedges of cars as straight lines that do not accurately matchthe actual boundaries. On the other hand, FCN, withpixel-wise detection, unveils the precise boundaries ofcars, enabling autonomous vehicles to effectively avoid
collisions.

Fig. 2 Freezing Mechanism graphs for (a) mAP_0.5, (b)mAP_0.5:0.95, (c) Precision and (d) Recall
Experimental results suggested that adding the SE
attention mechanism to the bottom or middle layers of the
network yields optimal outcomes. Fig. 4 illustrates that
adding the attention mechanism between layers 16 and 17
achieves higher accuracy than the original version,validating the hypothesis. Fig. 5 further confirms that
placing the SE attention mechanism between the middle
and bottom layers produces the best results.

Fig. 3 SE adding locations in the original YOLOv5
networkFig. 2 Detection result between YOLOv5s and FCN©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
891

Guandong Li, Yanzhe Xie, Yuhao Lu, Zongyan Wen, Jingzhen Fan, Yuankui Huang, Qinghong Ma, Wei Hong Lim, Chin Hong Wongboundaries. Enhancements, including the incorporationof attention mechanisms, freezing the network's
backbone, and modifying the activation function, areevaluated based on precision, recall, mAP, and mAP0.5
curves. These identified mechanisms and modules,
known to augment the performance of YOLOv5s, areamalgamated into an upgraded version referred to as
YOLO-ALPHA. However, the collective optimal
modules identified in previous research resulted in adegradation of performance. Following meticulous
analysis, the freezing backbone mechanism is pinpointed
as the cause. Subsequent refinement by removing thefreezing backbone mechanism reveals that YOLOALPHA showcases performance improvements incomparison to the original YOLOv5s.

Fig. 4 Attention Mechanism graphs for (a) mAP_0.5, (b)mAP_0.5:0.95, (c) Precision and (d) Recall
Five activation functions (ReLU, Leaky ReLU, Mish,
SiLU, and the original activation function) undergo
comparative analysis. Fig. 6 displays the precision, recall,mAP, and mAP0.5 indexes at epoch 50 for networks
employing various activation functions. The SiLU
activation function is selected for the PASCAL VOC
dataset due to its smoother curve and increased stability,
as tabulated in Table 1.

Fig. 6 Unsuccessful Result for (a) mAP_0.5, (b)mAP_0.5:0.95, (c) Precision and (d) Recall

Fig. 5 Activation Function
Table 1 Activation Function Data

Fig. 7 depicts a less satisfactory outcome whencombining the optimal freezing mechanism, attention
mechanism, and activation function. The precision
initially starts high but continuously decreases, indicating
performance degradation. Removing the attention
mechanism results in Fig. 8 that surpass those of theoriginal YOLOv5 network, affirming the performanceenhancement of the upgraded YOLO-ALPHA network.

Fig. 7 Result of Applying SE and SiLU excluding
Freezing Mechanism graph for (a) mAP_0.5, (b)mAP_0.5:0.95, (c) Precision and (d) Recall
References
1.

5. Conclusion
In this investigation, YOLOv5s is employed to carryout object detection, classification, and tracking, with the
FCN (Semantic Segmentation) technique serving as asupplementary method to precisely delineate object

2.

W. Lan, J. Dang, Y. Wang, and S. Wang, “Pedestrian
detection based on yolo network model,” in 2018
IEEE International Conference on Mechatronics andAutomation (ICMA), 2018, pp. 1547–1551.
T.-H. Wu, T.-W. Wang, and Y.-Q. Liu, “Real-time
vehicle and distance detection based on improved©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
892

Enhancing Precision Object Detection

3.

4.

5.

6.

7.

8.

9.

yolo v5 network,” in 2021 3rd World Symposium on
Artificial Intelligence (WSAI), 2021, pp. 24–28.
Z. Lu, L. Ding, Z. Wang, L. Dong, and Z. Guo,
“Road condition detection based on deep learningyolov5 network,” in 2023 IEEE 3rd International
ConferenceonElectronic
Technology,
Communication and Information (ICETCI), 2023,
pp. 497–501.
A. Dodia and S. Kumar, “A comparison of yolo
based vehicle detection algorithms,” in 2023
International Conference on Artificial Intelligenceand Applications (ICAIA) Alliance Technology
Conference (ATCON-1), 2023, pp. 1–6.
M. Kaloev and G. Krastev, “Comparative analysis ofactivation functions used in the hidden layers of deepneural networks,” in 2021 3rd Inter- national
Congress on Human-Computer Interaction,
Optimization and Robotic Applications (HORA),
2021, pp. 1–5.
X. Xiao, T. Bamunu Mudiyanselage, C. Ji, J. Hu, and
Y. Pan, “Fast deep learning training throughintelligently freezing layers,” in 2019 International
Conference on Internet of Things (iThings) andIEEE Green Computing and Communications(GreenCom) and IEEE Cyber, Physical and Social
Computing (CPSCom) and IEEE Smart Data
(SmartData), 2019, pp. 1225–1232.
H. Zhao, Z. Liang, D. Cai, and Y. Wang, “An
improved method for infrared vehicle and pedestrian
detection based on yolov5s,” in 2022 International
Conference on Machine Learning, Cloud Computing
and Intelligent Mining (MLCCIM), 2022, pp. 377–
383.
Kaymak and A. Uc¸ar, “Semantic image
segmentation for autonomous driving using fully
convolutional networks,” in 2019 International Artificial Intelligence and Data Processing Symposium(IDAP), 2019, pp. 1–8.
B. Xiao, J. Guo, and Z. He, “Real-time object
detection algorithm of autonomous vehicles basedon improved yolov5s,” in 2021 5th CAA
International Conference on Vehicular Control andIntelligence (CVCI), 2021, pp. 1–6.

Mr. Yanzhe Xie
He is currently pursuing Bachelor of
Robotics and Intelligent Devices as 3rdyear student in the Department ofMaynooth International Engineering
College, Fuzhou University, China. His
research interest is robotics and machinelearning
Mr. Yuhao Lu
He is currently pursuing Bachelor of
Robotics and Intelligent Devices as 3rdyear student in the Department ofMaynooth International Engineering
College, Fuzhou University, China. His
research interests are data science andembedded system.
Mr. Zongyan Wen
He is currently pursuing Bachelor of
Robotics and Intelligent Devices as 3rdyear student in the Department ofMaynooth International Engineering
College, Fuzhou University, China. His
research interests are embedded systems
and machine learning.
Mr. Jingzhen Fan
He is currently pursuing Bachelor of
Robotics and Intelligent Devices as 3rdyear student in the Department ofMaynooth International Engineering
College, Fuzhou University China. His
research
interests
aresoftwaredevelopment and system learning.
Mr. Yuankui Huang
He is currently pursuing Bachelor of
Robotics and Intelligent Devices as 3rdyear student in the Department ofMaynooth International Engineering
College, Fuzhou University, China. His
research
interests
areelectronic
information and machine learning.
Mr. Qinghong Ma
He is currently pursuing Bachelor of
Robotics and Intelligent Devices as 3rdyear student in the Department ofMaynooth International Engineering
College, Fuzhou University, China. His
research
interests
areelectronic
information and machine learning.

Authors Introduction
Mr. Guandong Li
He is currently pursuing Bachelor of
Robotics and Intelligent Devices as 3rdyear student in the Department ofMaynooth International Engineering
College, Fuzhou University, China.
His research interests are embedded
systems and machine learning.

Dr. Wei Hong Lim
He is an Associate Professor in Faculty ofEngineering at UCSI University in
Malaysia. He received his PhD in
Computational
Intelligence
fromUniversiti Sains Malaysia in 2014. His
research interests are optimization andartificial intelligence.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
893

Powered by TCPDF (www.tcpdf.org)

Guandong Li, Yanzhe Xie, Yuhao Lu, Zongyan Wen, Jingzhen Fan, Yuankui Huang, Qinghong Ma, Wei Hong Lim, Chin Hong Wong

Dr. Chin Hong Wong
He is a Lecturer at MaynoothInternational Engineering College at
Fuzhou University, China. He received
his PhD in Electrical and Electronic
Engineering from Universiti Sains
Malaysia in 2017. His research interests
are Energy harvesting, signals andsystems, and machine learning.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
894

A Game Framework Based on the Disinformation Warfare in Russo-Ukrainian War
Jumpei Ono

Aomori University, Kobata, Aomori, Aomori, 030-0943, Japan
Takashi Ogata

Iwate Prefectural University, Sugo, Takizawa, Iwate, 020-0611, Japan
Email: j.ono@aomori.ac.jp, t-ogata@iwate-pu.ac.jp

Abstract
From 2022, when Russia began its invasion of Ukraine, to the present, the number of speeches and information
directly/indirectly related to Russo-Ukrainian war has continued to increase in mass communication venues such as
SNS and TV. This study proposes the idea of a game system that simulates the spread of disinformation by focusing
on the information published by Ukraine as a list of disinformation speakers, especially on the topics they talk about.
The game system is intended to provide players with psychological immunity against disinformation by simulating
the spread of disinformation.
Keywords: Disinformation, Russo-Ukrainian war, Game System

For example, fake news, which spreads sensational orconspiratorial content, including falsehoods, is a typicalexample of disinformation. Today, battles that make fulluse of disinformation, in addition to the conventional
domain of warfare, are positioned as a new domain of
warfare and are called cognitive warfare [9]. On the other
hand, Ogata, who believes that falsehood is not the onlymeans of attacking opponents in battles that affect theperception of the target, has proposed the term “narrative
warfare” as a domain in which attacks are maderegardless of whether the content is true or false [6]. For
example, one method might be to present a distorted
conclusion as the conclusion of a logic that only mentions
the truth about individual elements. Maan have focusedon narrative warfare, and they have been exposed to it
from the time of the emergence of the Islamic State to thepresent day [10]. Narrative warfare in Russo-Ukrainian
war has also received attention [11].

1. Introduction
Russo-Ukrainian war has two parts. On February, 2014,
Russia annexed Crimea. Also, in February 2022, Russia
has invaded the Donbass region of Ukraine. In this war,
the discourse and information directly and indirectly
related to Russo-Ukrainian war continues to grow in masscommunication spaces such as social networking sites
and television. In this context, Russia sees theinformation space as another battlefield, with NATO [1]and Ukraine (https://cpd.gov.ua/) reporting on thesituation. Ogata [2], [3], [4], [5] has collected discourses
on Russo-Ukrainian war and summarized them as“Russo-Ukrainian War as Narrative Warfare”[6], while
Ono and Ogata have published articles on this war,
proposing the concept of a narrative-generating gamebased on Russo-Ukrainian war[7], [8].
This study proposes the idea of a game system (we callthe game system as simply “game” in the following
description) that simulates the spread of disinformation
by focusing on the discourses compiled by Ogata [2], [3][4] [5] and published by Ukraine as a disinformation
speaker list, especially the topics that the speakers talkabout. The game is designed to help players develop
psychological immunity to disinformation by simulating
the spread of disinformation.

Attempts to position players on the disseminators of
disinformation have been experimented with by
Roozenbeek and Linden [12] with Bad News [13]. Bad
News is a web game that provides a virtual experience ofa fake news speaker. In this game, each player follows thegame’s instructions and gradually increase their influenceon public opinion by selecting one of fake new candidates.
The player can efficiently increase their influence byselecting more appropriate fake news, and their influence
decreases if they select an inappropriate fake for theoccurred event. Roozenbeek and Linden found that thisgame allows each player to “cognitive resistance against
fake news strategies.” There are similar games on themarket, such as Plague Inc. [14]. Plague Inc. is asimulation game. Each player in the game becomes an

2. Background
Disinformation is information that influences aperson’s perception and distorts their decision making.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
895

Jumpei Ono, Takashi Ogata

epidemic and aims to wipe out humans. The player canselect the game mode of fake news.

Non-Player Character (NPC)3.



Proposed Game System

The structure of the game is presented after an overview
of the proposed game. At last, an example of the gameplay is shown.



3.1. Overview of the proposed game

Trust: if this value is 0, the NPC has completely lost
trust in the faction. This value also indicates
resistance to false information.
Characteristics: What personality traits the NPC has.
If there are no characteristics worth mentioning, it
is set to null. Currently, it indicates if the NPC hasconspiracy theorist traits.

Event
The game proposed in this paper is a game in which theplayer is a Russian spreading disinformation about the
Ukrainian model. The goal of each player is to convert
more Ukrainian faction to the Russian faction (“faction”
in this paper does not mean a particular political party,
people who believe in the state.). The game presents theplayer with events that occurred during Russo-UkrainianWar, and the player chooses the disinformation to spreadaccordingly. The Ukrainian faction is a set of non-player
characters (NPCs) that are not controlled by the player,
and each NPC has a trust parameter. This trust represents
the level of trust in Ukraine, and the disinformation
chosen by the player will fluctuate this value. The
Ukrainian faction will switch to the Russian faction if thetrust parameter falls below a certain value.

Characteristics: Classification of the event.
Disinformation is more influential if it matches the“Characteristics” in the disinformation.

DisinformationThis work does not include factors such as the spreadof misinformation (“misinformation” is false informationthat is spread without malicious intent), the phenomenonof disinformation having a greater impact due to biased
information on social networking sites, or the range ofinfluence of disinformation or the number of people
affected. These elements are considered important for
simulating disinformation and will be discussed as future
topics.

Strength: The degree to which the disinformation
affects the NPC’s trust.
Reality: The degree to which the disinformation isfactual, indicating the likelihood that the NPC will
accept the disinformation; if this value is low, thelikelihood of acceptance is low.
Characteristics: Classification of the disinformation.
If the disinformation is appropriate to the nature ofan event, it will have a greater influence on the
NPC’s trust.

Disinformation in the proposed game based onСПІКЕРИ, ЯКІ ПРОСУВАЮТЬ СПІВЗВУЧНІ
РОСІЙСЬКІЙ ПРОПАГАНДІ НАРАТИВИ (Speakers
who promote narratives that are in tune with Russian
propaganda,
https://cpd.gov.ua/reports/spikery-yakiprosuvayut-spivzvuchni-rosijskij-propagandi-naratyvy2/). This list was revised once, and before and after therevision, excluding duplicates, the list contained 89
speakers. In this paper, if the same speaker sent the samedisinformation before and after the revision, it wasclassified as a single source, and if different speakers sent
the same disinformation, they were classified as different
sources. The classification was then done in a bottom-up
fashion and organized as disinformation used in the game.
Each classification and the example of its breakdown areprovided as the Appendix.
The trust value of the NPC is reduced based on thefollowing Eq. (1) and. Eq. (2) First determine whether the
NPC is affected by the disinformation (Eq. (1)), and thendetermine the degree of influence (Eq. (2)). If the NPC iscommitted to a conspiracy theory, the trust value used in
the Eq. (1) and Eq. (2) Regarded as half the value.

3.2. The flow in a game play
A player becomes the sender of disinformation andchanges the parameters of non-player characters (NPCs)
made up of several people, to change the NPC’s faction.
The game repeats the turn as many times as the numberof events that make up the scenario, with (i) thegeneration of events, (ii) the selection of disinformation,and (iii) the calculation of effects as a single turn. At theend of the turn, the amount of trust in the Ukrainianfaction of NPCs is scored.
When choosing disinformation, the player chooses oneof the disinformation candidates. The mechanism
changes the parameters of all NPCs based on the selected
disinformation.

𝑃 = 100 + (𝑅 − 𝑇)
𝐴 = 𝑆 − 𝑇/4

3.3. Components of the proposed game

(1)
(2)

P: Probability of decreasing trust, A: Amount of trust
decreases, R: Reality in selected disinformation, T: Trustthat the NPC has, S: Strength in selected disinformation

The proposed game has non-player characters (NPCs),
events, and disinformation. These attributes are used to
calculate the score that evaluates the player, or are values
related to the game play.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
896

A Game Framework Based3.4. A game session example and a short
discussion

exposed to some type of disinformation and the influenceof disinformation for all NPCs are gradually increased.

The following shows the example of a game session.
There are five NPCs. One NPC in the five NPCs believesin the conspiracy theory. The session has six events.
“Disinformation” in the following description means thefake information selected by a player. “NPCs” in thedescription includes the changed “Trust” in the respective
NPC (“Number”).

4.

Conclusion

In this paper, we proposed the idea of a game system
related to disinformation in the context of RussoUkrainian war and developed a simple prototype dealing
with disinformation. We used one pattern of
disinformation used actually in Russo-Ukrainian war.
This game framework, including several NPCs and ahuman player, simulates the spreading process of
disinformation. The goal of a player in one session is tochange NPCs from Ukrainian faction to the Russian
faction. The player may acquire the knowledge to prevent
influences of disinformation through the game process.
In the future, we would like to extend the scale of thegame.

Event 1: “Russia have launched an invasion.”
Disinformation: “The U. S. is provoking/inspiring
Russia.”
NPCs: Number 0: Trust 82 (conspiracy thinking) /Number 1: Trust 95 (nil) / Number 2: Trust 95 (nil) /Number 3: Trust 95 (nil) / Number 4: Trust 95 (nil)
Average: 92.4, Total trust: 462/500
Event 2: “Russia occupied a city.”
Disinformation: “Russia has not attacked civilians.”
NPCs: Number 0: Trust 62 (conspiracy thinking) /Number 1: Trust 88 (nil) / Number 2: Trust 88 (nil) /Number 3: Trust 88 (nil) / Number 4: Trust 88 (nil)
Average: 82.8, Total trust: 414/500

Acknowledgements
This research was supported by JSPS KAKENHI(JP21K17870).

Event 3: “A massacre occurred.”
Disinformation: “The massacre in Bucha is a fake.”
NPCs: Number 0: trust 9 (conspiracy thinking) / Number
1: trust 50 (nil) / Number 2: trust 88 (nil) / Number 3:
trust 50 (nil) / Number 4: trust 50 (nil)
Average: 49.4, Total trust: 247/500

References
1. E. Lange-Ionatamišvili, Analysis of Russia’s information
campaign against Ukraine. Executive Summary. Executive
Summary, Riga: NATO Strategic Communications Centreof Excellence, 2014.
2. T. Ogata, The Narrative warfare of Russo-Ukrainian war
in Japan: Introduction, The Japanese Society for Artificial
Intelligence SIG-LSE0C31, 70, 9-82, 2022.
3. T. Ogata, Survey, analysis, and critique of the social
discourse in contemporary Japan based on an example:
Post-narratological study using the phenomenon of“Hashimoto Toru”, Proc. of the 39th Annual Meeting ofJapanese Cognitive Science Society, 632-640, 2022.
4. T. Ogata, War and narrative generation amateur talk:Beyond the landscape, in sad Japan where there is also a
World’s Commander Hashimoto-kun, The JapaneseSociety for Artificial Intelligence SIG-LSE0C31, 17-142,
2022.
5. R. Abe, J. Ono and T. Ogata, Concept acquisition andsentence generation of Russian-Ukrainian war based on
linguistic data from TV news and commentary programs,
The Japanese Society for Artificial Intelligence SIGLSE0C31, 71, 161-175, 2023.
6. T. Ogata, Monogatarisen Toshite no Russo Ukraine Senso,
Shin’yosya, 2023.
7. J. Ono and T. Ogata, Narrative generation game based on
the Ukraine War, The Japanese Society for Artificial
Intelligence SIG-LSE0C31, 69, 143-148, 2022.
8. J. Ono and T. Ogata, Designing a narrative generation
game based on the Russian Invasion of Ukraine,
Proceedings of the 2023 International Conference on
Artificial Life and Robotics (ICAROB2023), 732-736,
2023.
9. Y. Koizumi, K. Kuwabara, and K. Komiyama, Nise Joho
Senso: Anata no Atama no Naka de Okoru Tatakai, 2023.

Event 4: “The war continues.”
Disinformation: “Ukraine/Zelensky wants war.”
NPCs: Number 0: trust 0 (conspiracy thinking) / Number
1: trust 32 (nil) / Number 2: trust 80 (nil) / Number 3:
trust 32 (nil) / Number 4: trust 32 (nil)
Average: 35.2, Total trust: 176/500
Event 5: “The war continues.”
Disinformation: “Russia will win the war.”
NPCs: Number 0: trust 0 (conspiracy thinking) / Number
1: trust 10 (nil) / Number 2: trust 70 (nil) / Number 3:
trust 10 (nil) / Number 4: trust 10 (nil)
Average: 20.0, Total trust: 100/500
Event 6: “Russia has retreated.”
Disinformation: “They should not hunt down Russia.”
NPCs: Number 0: trust 0 (conspiracy thinking) / Number
1: trust 0 (nil) / Number 2: trust 57 (nil) / Number 3: trust
0 (nil) / Number 4: trust 0 (nil)
Average: 11.4, Total trust: 57/500
In the above session, the value of trust to Ukraine of
the conspiracy theorist (Number 0) is rapidly reduced. In
the other NPCs, the NPC of Number 2 continues to havethe value of trust to Ukraine. However, the values of trustof the other NPCs, Number 1, 3, and 4, finely become 0.
When we see the entire process, we know the accelerated
growth of suspicion to Ukraine.
In the game, the value of trust is also the power to resist
the influence of disinformation. NPCs are repeatedly©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
897

Powered by TCPDF (www.tcpdf.org)

Jumpei Ono, Takashi Ogata

Russia into war with Ukraine. / Zelensky provoked Russia
into launching special operations against it. / The U.S. hasprovoked Putin for years. / The events in Ukraine are a
provocation by the U.S. and NATO. / The U.S., throughNATO, provoked Russia to war in Ukraine. NATO and the
U.S. provoked Putin. NATO provoked Putin. NATOprovoked Russia The West provoked Putin. / The Westprovoked Russia. / The U.S. provoked Russia. /The Westprovoked Putin. / Arming Ukraine provokes Russia. /Zelensky is provoking Putin. / NATO has crossed the “redline.” / The U.S. and UK have crossed Putin’s “red line.” /For eight years, Ukrainians have been provoking Russia. /Putin’s decision to invade Ukraine was NATO’s fault. /NATO intentionally deployed on the Russian border. / The
U.S. and Europe provoked the Russian president. / The U.S.
is responsible for Russia’s attack on Ukraine. / The U.S.
needs to stop the madness in Ukraine provoked by the U.S.

10. Mann, A., Narrative Warfare. Narrative Strategies Ink.,
2018.
11. N. Aleksejeva (Reserch coodinator) and A. Carvin (Ed.),
Narrative Warfare: How the Kremlin and Russian News
Outlets Justified a War of Aggression against Ukraine,2023.
12. J. Roozenbeek and S. V. D. Linden, Fake news game
confers psychological resistance against online
misinformation, Palgrave Communications, 5 (65), 2019.
DOI: https://doi.org/10.1057/s41599-019-0279-9
13. Cambridge Social Decision-Making Lab., Bad News
Game. https://www.getbadnews.com/en/play, 2018.
14. Ndemic Creations, Plague Inc., Ndemic Creations, 2021.

Appendix: The types of disinformation
The following categories are based on the СПІКЕРИ, ЯКІПРОСУВАЮТЬ
СПІВЗВУЧНІ
РОСІЙСЬКІЙ
ПРОПАГАНДІ НАРАТИВИ (Speakers who promotenarratives that are in tune with Russian propaganda,
https://cpd.gov.ua/reports/spikery-yaki-prosuvayutspivzvuchni-rosijskij-propagandi- naratyvy-2/). “~~~” is aspace to insert a specific word, one of the U.S., NATO,
Ukraine, or Zelensky.
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.

Authors Introduction
Dr. Jumpei Ono
He received his bachelor’s degree fromthe Faculty School of Software andInformation Science, Iwate Prefectural
University in 2010. He received his MS
and PhD from the Graduate School ofSoftware and Information Science,
Iwate Prefectural University in 2014
and 2018, respectively. He worked as
an information and communication technology instructor
at the Vocational School of Digital Arts Sendai from 2018
to 2020. He is currently an assistant professor at theFaculty of Software and Information Technology, Aomori
University.

~~~ is provoking/inspiring Russia
It is a proxy war between ~~~ and Russia
The West is running a propaganda campaign
Russia-hatred (Russophobia)
NATO expansion is bad
Ukraine is suppressing Russian speakers in the country
Ukraine/Zelensky wants war
Ukraine is committing acts against humanity
Bucha massacre is a fakeUkraine has a biological weapons laboratory
We should not give arms to Ukraine
Ukraine started war before February 24/Maidan
Revolution
Ukraine is a Nazi
Ukraine should make concessions
Ukraine did not comply with the Minsk agreements
Russia will win the war
Sanctions are not getting through to Russia
Russia has not attacked civilians~~~ should negotiate with Russia/Putin~~~ should not blame Russia/Putin~~~ should not hunt down Russia
Crimea is Russian territory
Russia has freedom of speech

Dr. Takashi Ogata
He received his BSS from Waseda
University in 1983, his MS from
Tsukuba University in 1992, and his
PhD from the University of Tokyo in
1995. He has garnered industrial
experience from working at softwaredevelopment companies since 1983. He
has been an associate professor in the
Faculty of Engineering at Yamanashi University since1997 and is currently a professor in the Faculty of Softwareand Information Science at Iwate Prefectural University,where he has worked since 2005.

For example, 1. “~~~ is provoking/inspiring Russia”
includes the following contents. Other types also include
many contents. Small amount of contents are classified as“Other” and was not discussed in this paper.
The U.S. and Ukraine have actively provoked Russia into
aggression./ The U.S. and Ukraine have aggressively
provoked Russia into aggression. / Putin has been humiliatedand Russia persecuted for decades. / The West provoked©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
898

A Comparative Analysis of Eye Tracking between Veteran and Novice during RadiologicalInterpretation
Yuka Naito

Graduate School of Economics, Chuo University, Higashinakano Hachioji-shi, 192-0393, Japan
Jun Nakamura

Faculty of Global Management, Chuo University, Higashinakano Hachioji-shi, 192-0393, Japan
Yoshinobu Ishiwata

Graduate School of Medicine, Yokohama City University, 3-9, Fukura, Yokohama Shi Kanazawa Ku, 236-0004, Japan
Email: a19.3gem@g.chuo-u.ac.jp, jyulis.77f@g.chuo-u.ac.jp, ishi_y@yokohama-cu.ac.jp

Abstract
With the development of AI, AI may substitute for physicians in radiological interpretations in the future. The
purpose of this paper is to identify factors that could form the foundation for algorithms enabling AI to explore
pathological findings. This study focuses on the "angles of gaze trajectory" and conducts a comparative analysis
between veteran and novice. The results indicate commonalities and variances in the angles of gaze trajectory
between veteran and novice, particularly 10, 350, and 180 degrees. Both Veteran and Novice have a relatively
higher frequency of gaze movement around 10 and 350 degrees compared to other angles. Veteran has a relatively
higher frequency of gaze movement around 180 degrees compared to other angles, while novice has a relatively
lower frequency around 180 degrees.
Keywords: Eye-tracking, Angles of eye trajectory, Artificial Intelligence (AI)analysis is conducted on how participants can detect thiscue in terms of gaze trajectory and movement velocity[3]. Studies comparing the gaze patterns of veteran andnovices are also prevalent. For instance, an analysis ofgaze fixation points during chest X-ray image
interpretation shows that experts promptly identify
lesions and repeatedly confirm their presence while alsoscanning other areas, whereas novices tend to fixate on
structurally complex regions [4]. In research on
overlooking and missing findings during medical image
interpretation, it has been demonstrated that experts
thoroughly observe photographs, strongly focusing on
the accurate location of anomalies, while novices exhibit
more inconsistent observation patterns, with their gaze
often diverted to complex positions where organs
overlap [5]. This line of research on tacit knowledge andexpertise in skill-demanding scenarios has unveiled thatveterans and novices differ in terms of "where to look,"
influencing their subsequent performance [6]. As apreliminary step, it is necessary to extract andquantitatively analyze the gaze patterns of experts
before linguistically formulating such characteristics.
Therefore, the purpose of this paper is to conduct a
comparative analysis between novices and veteransregarding the angles of gaze trajectories, which could
serve as the foundation for algorithms exploring
findings in artificial intelligence (AI). Furthermore, theobjective is to propose recommendations for AI to learn

1. Introduction
Machine learning and deep learning advancements,
coupled with high-performance GPUs, have led to asignificant breakthrough in the field of artificial
intelligence [1]. Particularly in image recognition, AI isconsidered to surpass human capabilities [2]. In therealm of healthcare, a critical application of AI's image
recognition capabilities is evident, with one prominent
example being the interpretation of CT images. Given
the current shortage of radiologists, AI is expected toachieve a level of expertise in image interpretation thatcan serve as a viable alternative to human physicians.
To attain this capability, various factors contributing to
the training of AI in image interpretation must be
identified. One such factor is the ocular movement ofphysicians during image interpretation. This paper aims
to discuss the significance of physicians' gaze patterns
as one of these factors, emphasizing the need to uncoverand understand these aspects to enhance the learning
process of AI in image interpretation.
2. Previous Research
Numerous studies have explored gaze measurementsof physicians during the interpretation of radiographic
images, a certain cue is presented on the screen, and an©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
899

Yuka Naito, Jun Nakamura, Yoshinobu Ishiwata

from the disparities in gaze trajectories between novices
and veterans, ultimately enabling more accurate andeffective image diagnostics in the future.

(971, 810) to (988, 867), and subsequently to (974, 816),
the angle of gaze movement is denoted by Angle 1 inFig. 1. The directional movement is measured clockwise.
Similarly, when moving from (988, 867) to (974, 816)
and further to (1013, 837), the angle of gaze movement
is represented as Angle 2 in Fig. 1.

3. Experiment
The gaze measurement experiment was carried out asfollows.
⚫ Participants: One veteran radiologist (with over 10
years of experience) and one novice radiologist(with 1~3 years of experience).
⚫ Data:15 cases with abnormal findings.
⚫ Experimental method: Tobii Pro Lab eye-tracking
software was employed. The camera was fixed.
Participants were presented with CT images on acomputer screen and performed interpretation of
CT images by scrolling through the images using amouse, following their usual reading methods.

Fig.1 Angles of gaze movement

4. Analysis Methods
5. Results
Duchowski provides guidelines regarding datacollection and interpretation in the context of eye
tracking. The book introduces applications and instancesof success in fields such as medicine, psychology, andhuman factors engineering, underscoring theeffectiveness of employing eye tracking in the medical
domain [7]. Furthermore, Holmqvist et al. offer an indepth exposition of data analysis methods in eye
tracking research. Of particular note, fixation points
serve as a frequently employed metric [8]. However, theanalyzed metric in this paper is the angle of the gazemovement as it sequentially moves from one fixation
point to the next in succession. (Henceforth, this shall bereferred to the angle of eye movement.) The reason forthis is to confirm whether the direction of eye movement,
namely the trajectory of the gaze, when radiologists
interpret CT images is consistent with the characteristics
of everyday human gaze as elucidated in previous
research [9]. The characteristics of human eye
movement elucidated in this previous research aredetailed in the subsequent discussion section. The
employed eye-tracking software in this study
automatically aggregates parameters such as fixation
points, fixation counts, fixation duration, andcoordinates of fixation points, subsequently generating
this information as output data. The trajectories
connecting successive fixation points (hereinafterreferred to as gaze trajectories) are visually indicated by
red lines, enabling direct observation.
The angles of gaze movement were defined based on
the eye-tracking data obtained using Tobii Pro Lab andanalyzed using a custom-developed program. In thiscontext, angles of gaze movement refer to the directionchanges of gaze trajectories as it progresses acrossdifferent fixation points. The concept is illustrated in theFig. 1, where black circles represent fixation points(pixel), and connecting lines depict the gaze trajectory
between these points. When the coordinate shifts from

The results of the measured angles of gaze movement,
as captured by the software, are presented in Fig. 2 andFig. 3.

Fig.2 Angles of gaze movement in veteran (withabnormal findings)

Fig.3 Angles of gaze movement in novice (withabnormal findings)
A comparison between veteran and novice revealed
notable differences in angles of gaze movement. In thegraphs, the numbers displayed on the outercircumference represent the angles, while the numbers
within the circumference, namely the radii, indicate the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
900

A Comparative Analysis of Eye

frequency of the corresponding angles. Skilled
practitioners exhibited a higher frequency of gazemovement angles around 180 degrees within the 360degree spectrum. This is demonstrated in Fig. 2. In Fig.
2, angles such as 10 degrees, 350 degrees, 190 degrees,and 180 degrees were observed with higher frequencies.
When examining the results for novice, as shown in Fig.
3, angles such as 350 degrees and 10 degrees were more
pronounced in their gaze movement. Comparing theresults presented in Fig. 2 and Fig. 3, it is evident thatveteran frequently exhibit gaze movement angles near
180 degrees, a pattern not as pronounced in novice
practitioners. This suggests that veteran tend to pause
and carefully observe between fixation points whenshifting their gaze, while novice appear to transition
their gaze more promptly from one point to another.

in previous studies [9]. The research has been elucidated
that repetitive movements along horizontal lines are
more frequent than those along vertical lines. In otherprevious research [10], it has revealed that the capability
to accurately perceive information is higher in thehorizontal direction of eye movements compared to thevertical direction. This paper also found that novice
exhibited more frequent back-and-forth movements
along horizontal lines than vertical ones. Thisphenomenon can be attributed to reasons such as therelatively easier execution of horizontal movements
compared to vertical ones, as well as the cautious
tendency of novices due to the difficulty in anticipating
the location of abnormal findings. Consequently,
novices may opt to observe a broader scope of the CT
image.

6. Discussion

7. Conclusions

The paper highlighted differences in gaze movement
between veteran and novice during interpretation. The
analysis of angles of gaze movement, a hypothesis wasformulated. The hypothesis suggests that skilled
practitioners pause for careful observation between
fixation points when shifting their gaze, whereas
novices shift their gaze more promptly from one point toanother, lacking this intermediate pause.
Considering that novices possess less experience thanskilled practitioners, they might find it challenging topredict the location of anomalies. This could potentiallyexplain why novices tend to survey the entire CT images
thoroughly and quickly. In contrast, veteran can infer
the approximate location of anomalies, allowing them toconcentrate on specific areas. Interviews withexperienced practitioners aligned with this explanation.
Inexperienced individuals may have attempted toobserve a broad area as much as possible. As a result,the gaze of novice may have exhibited movements along
the Z-axis (the head-foot direction in CT images), while
simultaneously
surveying
the
XY
plane
comprehensively. Conversely, veteran tend to direct
their gaze along the Z-axis (head-foot direction).
Therefore, in a series of similar CT images, should ananomaly be detected, it could be recognized as a flicker.
According to the previous research [9], thecharacteristics of human gaze encompass distinct roles
for central vision and peripheral vision, where graspingstimuli occurs through peripheral vision andsubsequently confirming details via central vision
constitutes a fundamental process. Veterans shift theirgaze along the Z-axis (head-foot direction) with
peripheral vision, and subsequently enhance theresolution with their central vision when anomalies aredetected. This phenomenon aligns with the description
provided by adept physicians as "recognizing anomalies
as flickers."
Additionally, the gaze patterns of veteran revealed
repeated back-and-forth movements along lines near
180 and 360 degrees. Similar repetitive movements
along horizontal and vertical lines have been observed

The purpose of this paper is to conduct a comparative
analysis between novices and veterans regarding theangles of gaze trajectories, which could serve as thefoundation for algorithms exploring findings in artificial
intelligence (AI). Furthermore, the objective is to
propose recommendations for AI to learn from thedisparities in gaze trajectories between novices andveterans, ultimately enabling more accurate andeffective image diagnostics in the future. This paper
specifically analyzed the angles of gaze movement andspeed of veteran and novice radiologists. However, it isimportant to note that this research had limitations interms of a limited number of participants and sample
size. Future studies should involve a larger sample size
to further investigate the elements of algorithms during
radiological interpretation.

8. Acknowledgements
The development of a program for analyzing gazemovement angles in this study was made possible
through the support of Focus Systems Corporation. The
authors express their deep gratitude for their assistance.
References
1.

2.

3.

T. Hirasawa, Y. Ikenoyama, M. Ishioka, K. Namikawa, Y.
Horiuchi, H. Nakashima, T. Tada, and J Fujisaki,
Endoscopic AI Diagnosis of Gastric Cance, The journalof the Japan Laser Medical Society, 42(4), 2021-2022,
pp.250-260
T. Hirasawa, T. Tada, and J. Fujisaki, The Future of
Gastric Cancer Diagnosis with the Utilization of Artificial
Intelligence (AI), The journal of the Japan Endoscope forFire Extinguishers Society, 62(12), 2020, pp.3031-3140
N. Tsutsumi, M. Abe, T. Someya, and T. Matsumoto,Fundamental Consideration for Eye Movement Analysis
during X-ray Image Interpretation. Journal of the Japan
Society of Medical Image and Information Science, 17(3),2000, pp.152-160©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
901

Powered by TCPDF (www.tcpdf.org)

Yuka Naito, Jun Nakamura, Yoshinobu Ishiwata

4.

T. Matsumoto, T. Iinuma, Y. Tateno, K. Fukuhisa, K.
Ohata, M. Kaneko and S. Ikeda, Gaze Point Analysis of
Physicians during Interpretation of Chest X-ray Images,
Medical Imaging Technology, 35(4), 1990, pp.212-216
5. T. Terashita, T. Sato, T. Ogura, and K. Doi, Development
and Evaluation of an Eye-tracking-based Application for
Measuring Medical Image Interpretation Abilities.
Transactions of the Japanese Society for Artificial
Intelligence, 23(1), 2021, pp.43-46
6. R. Fukuda, K. Yoshida, Y. Ono, J. Matsubara, and M.
Kudo, Feature Extraction of Skilled Caregivers using Eye
Tracking, 2011
7. Andrew T. Duchowski, Eye Tracking Methodology,
Theory and Practice, Springe, 2017
8. K. Holmqvist, M Nystrom, R. Andersson, R. Dewhurst,
H/ Jarodzka, and J. Weijer, Eye Tracking: A
Comprehensive Guide to Methods and Measures, Oxford
University Press, 2011
9. K. Hashimoto, K. Ushiki, M. Nakamura, G. Watanabe,
and N. Ogawara, Consideration on Stimulus
Attractiveness and Arrangement in Video Playback.
Journal of the Information Processing Society of Japan:
Human and Computer Interaction, 2006(3), 2006, pp.7581
10. K. Murata, T. Miyake, and M. Moriwaka, Effectivenessof Gaze Input Systems - Identification of Conditions to
Ensure Point Accuracy and Speed and the Influence ofMovement Direction. Ergonomics, 45(4), 2009, pp.226235

Authors Introduction
Ms. Yuka Naito
She is a 1st year graduate student at thefaculty of Economics, Chuo University,
Japan. The areas of his research interest
are Experimental Economics, Behavioral
Economics, Cognitive Psychology, andArtificial Intelligence.

Dr. Jun Nakamura
He is a professor at the Faculty of Global
Management, Chuo University, Japan.
Previously he was working as a professor
at the Graduate School of Engineering
Management, Shibaura Institute ofTechnology. His main research areas are
Cognitive Science and Humanities.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
902

Development of Notification System to Prevent Working Productivity from
Declining Caused by Increased Carbon Dioxide Concentration
Yohei Kamoda

Chuo University, Faculty of Global Management, 742-1 Higashinakano, Hachioji-City
Tokyo, 192-0393, Japan
Jun Nakamura

Chuo University, Faculty of Global Management, 742-1 Higashinakano, Hachioji-City
Tokyo, 192-0393, Japan
Email: kamoda.yohei@gmail.com, jyulis.77f@g.chuo-u.ac.jp
Abstract
In association with the change of work style to remote, the home environment should be considered interms of carbon dioxide concentration (CDC) which has negative effects on the human body, such asless cognitive abilities. To challenge this problem, the authors developed an alert system that supports
remote workers to be notified of an increase in CDC, by combining Raspberry Pi, CDC sensor, andSlack. As a result, the developed alert system was able to support the user in keeping the CDC in theroom below the set threshold. In addition, the system shows that a significant increase in CDC can beobserved in the room when it comes to insufficient ventilation.
Keywords: Carbon Dioxide Concentration (CDC), Raspberry Pi, Working Productivity

1.

Introduction

books, and beds [4]. However, different from theperspective above, this paper focuses on thecomposition of indoor air. According to Azuma(2018), increased indoor CDC due to humanbreathing and other factors can cause negative
impacts on the human body, such as dizziness,headaches, and reduced cognitive and problemsolving abilities [5]. Therefore, remote workers
need to maintain indoor air quality by ventilating
and other ways. However, the increase in CDC
cannot be perceived easily, and it is difficult toknow the appropriate timing to ventilate. Therefore,
there is a possibility that the increase in CDC iscausing a decrease in working productivity even if
remote workers do not perceive it.
The purpose of this paper is to prevent anincrease in CDC and a decrease in working
productivity by using the system developed to
notify the appropriate timing of ventilation.

In recent years, the spread of the Internet andthe coronavirus have prompted many companies toadopt remote work. According to the Pew ResearchCenter (2022), remote work rates have not declined
much even after the peak of the pandemic [1]. In
addition, many workers have a favorable attitude
toward remote work after the pandemic (Da Silva
et al., 2023) [2]. Therefore, remote work will likely
continue to be promoted after the end of pandemic.
In consideration of the situation mentioned
above, working productivity is noteworthy.
Generally, the promotion of remote work in thewake of the coronavirus is interpreted as anincrease in productivity of work. In fact, there is anargument that the shift to remote work leads toincreased productivity, by thinking from theperspective of the amount of work, work-lifebalance, and job satisfaction (Kurdy, 2023) [3].
However, most of the papers referring to
productivity in remote work simply compare
productivity before the introduction of remote
work with after doing that. In other words, there islittle discussion from the perspective of changes in
productivity caused by the home environment.
When pursuing productivity in remote work, ahome environment that has become the working
environment should be focused. Krivonossova(2022) points out that distractions in remote work
are caused by the home environment, such as TVs,

2.

Previous Research to Set Threshold

The unit used to express CDC is the "ppm
(parts per million)”. To determine the CDC
threshold for the notification, it should be referredto the indoor CDC standards published by various
public agencies.
The Wisconsin Department of Health Services(2023) states that 400 to 1,000 ppm is a reasonable
standard for CDC indoors. Moreover, it points outthat from 1,000 to 2,000 ppm will cause complaints©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
903

Yohei Kamoda, Jun Nakamuraof drowsiness and poor air quality, and from 2,000
ppm and above will cause health problems such as
headaches, drowsiness, decrease in concentration
to work, and mild nausea [6]. In addition, Japan's
Ministry of Health, Labour and Welfare (2016) has
set the environmental health management standard
for CDC at 1,000 ppm in its Building Sanitation
Law because CDC levels above 1,000 ppm cause
fatigue, headaches, tinnitus, and breathlessness inthe human body [7].
From the above, it is reasonable to set thethreshold used in this paper at 1,000 ppm. Thus, thestandard for notification used in the developed
system in this paper can be set at 1,000 ppm.
3.

available on GitHub created by Ueda (2018) isutilized to simplify source code [8]. There are two
measurement functions using this. The first is torecord measurement results as csv file for analysis.
The second is to immediately check the results at
any given time through a small display.
3.3.

In the system, Slack, the communication toolin business, was used for the CDC notifications.
The purpose of this is to send a notification to theuser to urge them to ventilate when the CDC value
exceeds the threshold level of 1,000 ppm.
In addition, three conditions were set for
notification. First, the Raspberry Pi must beconnected to the Internet. Second, the notificationis sent when the measured CDC value exceeds
1,000 ppm. Third, the judgment on whether to senda notification is made every five minutes.

Specifications of Developed System

3.1. System Overview
Fig. 1 shows a photograph of the overall CDC
measurement and notification system created inthis paper, and Fig. 2 shows a wiring diagram ofthe system. The system consists of a small
computer called "Raspberry Pi model 4B" and two
modules called "MH-Z19C" and "LCD 1602". The
computer and display are connected by an i2c
communication interface, "PCF8574".

4. Methods
Measurementof

Evaluation

and

Data

4.1. Environment during Measurement of
Carbon Dioxide Concentration Data
The data measurements in this paper wereconducted by the authors in their room. A
simplified figure of the room where the datameasurements were taken is shown in Fig. 3. To
measure CDC as close as possible to the air that aperson breathes, the sensor was placed close to theperson. At this time, the position of the sensor was
adjusted to avoid direct contact with the exhaled
breath to obtain accurate data. Also, the "naturalventilation system" in this paper refers to the 24hour ventilation system in the bathroom outside the"door", and the room is ventilated by taking in
fresh air through the gaps around the "door” in theexperimental environment.
When measuring data, test measurements wereconducted under the same environment for at least
60 minutes prior to the start of the measurement toconfirm that the CDC remained stable at less than
1,000 ppm. In addition, the working human was
not away from the seat during the measurement
and was assumed to be engaged in constantbreathing activity.

Fig. 1. Overall Picture of CDC Measurement andNotification System

Fig. 2. Wiring Diagram of the System
The functions implemented in this system canbe divided into measurement and notification.
These two functions are intended to solve theproblem stated in Chapter 1.
3.2.

Notification Function

Measurement Function

The CO2 sensor module, MH-Z19C, was used
to measure CDC. In terms of the program code tomeasure CDC, the function from the library©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
904

Development of Notification System

Fig. 4. Transition of Carbon Dioxide Concentration
when Ventilation is Conducted at the Time of
Notification

Fig. 3. Environment of Room during Measurement
4.2. System Evaluation

As shown in Table 2 and Fig. 4, it can be said
that ventilation at the time of the notification
contributes to keeping the CDC in the room below
1,000 ppm. However, when the "Window" wasclosed, the CDC rapidly rose and exceeded 1,000
ppm. In the end, 135 minutes of ventilation time
was required out of 300 minutes. However, it
would be fair to say that positive results wereobtained for the question, "Can the notification
system help to keep the CDC in the room below
1,000 ppm?"

In the evaluation of the system, the effect of thenotification system on the increase in CDC isevaluated. Assuming a situation of inadequate
ventilation, the "natural ventilation system" isstopped. Then, the "Window" in Fig. 3 is opened at
the time of the notification to urge the user toventilate the room. As a result, the system will be
evaluated whether it has been able to contribute to
reducing the CDC in the room below 1,000 ppm.
5. Results

6. Discussion

Under the environment and conditions
described in Chapter 4, an experiment was
conducted by opening the "Window" andventilating the room when the measured CDC
value in the room exceeded 1,000 ppm, and a Slacknotification was received. Table 1 shows the date
and time of the measurement and the total
measurement time, and Table 2 shows the elapsed
time until when the notification was sent and the
conducted ventilation time. Also, the CDC
transition obtained from this measurement isshown in Fig. 4. In Fig. 4, the period during which
the ventilation was conducted is colored.

It is fair to say that the system developed in thispaper can help to prevent the increase of CDC in aroom, as shown in the results of Chapter 5. In
addition, it also showed that a room withinadequate ventilation has a high possibility ofincreasing CDC causing not only reduced
productivity but also health problems. In otherwords, the system developed in this paper can
prevent the decline in working productivity byknowing the adequate timing to ventilate. Also, thesocial value of this system in realizing the goal of
improving productivity in remote operations canbe well recognized.
However, this paper does not actually compare
working productivity. Therefore, it is necessary toconsider the possibility that Slack notifications
may interfere with concentration on work. From
this viewpoint, this system will not necessarily leadto increased working productivity based on the
only verification in this paper.
In addition, the system developed in this paper
only notifies the user of appropriate ventilation
timing. Thus, the control of CDC in the room
depends on the opening and closing of windows by
the user. This is sufficient for remote work but
needs further expansion to be utilized in otherlocations. As Fig. 4 shows, in rooms withinadequate ventilation, ventilation by opening andclosing windows only becomes a temporary
measure. Considering that the CDC rose

Table 1. Date and Total Time of Measurement for
Fig. 1
Start date and time
2023/9/27 19:00

End date and time
2023/9/28 0:00

Total Time
300 minutes

Table 2. Elapsed Time from Start of Measurement
to Notification and Ventilation Time
Elapsed Time (minutes)
65
95
120
190

Ventilation Time (minutes)
15
15
30
35

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
905

Powered by TCPDF (www.tcpdf.org)

Yohei Kamoda, Jun Nakamura

immediately after the window was closed, aventilation system that automatically adjusts the
CDC independent of user behavior would beneeded to maintain the appropriate working
environment.
Also, according to OPEN ACCESSGOVERNMENT (2021), since the COVID-19
pandemic, workers' awareness of flexible work
styles and better working environments has
increased [9]. Therefore, the system developed inthis paper can be used to improve the working
environment at a more advanced stage than the
promotion of remote work. The importance of thissystem will be recognized, and its demand will risein the current situation that COVID-19 has
promoted not only remote work but also awarenessof the work environment.

2.

3.

4.
5.

6.

7. Conclusion
7.

The purpose of this paper is to prevent anincrease in CDC and a decrease in working
productivity by using the system developed to
notify the appropriate timing of ventilation.
The CDC notification system developed in thispaper could be utilized to improve the working
environment for remote workers. As shown in Fig.
4, the system was able to help keep the CDC in theroom below 1,000 ppm by encouraging ventilation
at the appropriate time. However, it does notindicate whether it was possible to prevent
working productivity decline. Therefore, further
verification of working productivity will benecessary in the future.
The data measurements in this paper wereconducted in the limited environment of theauthors’ room, and the number of experimental
subjects is insufficient. Therefore, it will benecessary in the future to acquire a large amount ofdata and to consider detailed environmental
conditions such as the location of ventilation fans,actual flow of air, the number of people, room size,
and temperature. This would provide detailed
findings and suggestions for ventilation.

8.

9.

in America.
Da Silva, A. D., Georgarakos, D., & Weißler, M.
(2023). How people want to work – preferences
for remote work after the pandemic. ECB
Economic Bulletin, (1).
Kurdy, D. M., Al-Malkawi, H.-A. N., & Rizwan,
S. (2023). The impact of remote working on
employee productivity during COVID-19 in the
UAE: the moderating role of job level. Journal ofBusiness and Socio-economic Development. 3(4),339-352.
Krivonossova, M. (2022, September 9). 6 Pros
and Cons of Remote Work. University Wire.
Azuma, K. (2018). Effects of inhalation exposure
to carbon dioxide on human health in indoor
environment. Indoor Environment, 21(2), 113-120.
Wisconsin Department of Health Services. (2023,
March 29). Carbon Dioxide. Retrieved December
3,
2023,
from
https://www.dhs.wisconsin.gov/chemical/carbond
ioxide.htm
Ministry of Health, Labour and Welfare. (2016,
October 31). [ventilation requirements for indoor
air quality] shitsunai kuuki sitsu no tame no
hitsuyou kankiryou (in Japanese).
Ueda, T. (2018, November 29). Mhz-19. GitHub.
Retrieved
December
3,
2023,
from
https://github.com/UedaTakeyuki/mhz19/wiki/mh_z19
OPEN ACCESS GOVERNMENT. (2021, March
2). Why is it important to have a great working
environment?

Authors Introduction
Yohei Kamoda
He is a student of Faculty of
Global Management at Chuo
University. He is the president of
the "V.Code" programming clubat the university. He also holds
various certifications, including
the Fundamental Information
Technology
Engineer
Examination.
Dr. Jun Nakamura
He is a professor of Faculty of
Global Management at Chuo
University inJapan. He
graduated from the Departmentof Technology Management for
Innovation,
SchoolofEngineering, University ofTokyo, with highest honors. His area of research is
cognitive sciences.

Acknowledgement
The cost of purchasing the computer andmodules necessary to develop the system was paidby a scholarship from the Toru Yoshikawa
Foundation. The authors would like to express thedeepest gratitude to the Foundation for theirfinancial support for this research.
References
1.

Pew Research Center. (2022, February 19).
COVID-19 Pandemic Continues To Reshape Work©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
906

Visualization of the Skilled Physician’s Gaze characteristic during Diagnosis
Taiki Sugimoto

Graduate School of Economics, Chuo University, Higashinakano Hachioji-shi, 192-0393, Japan
Jun Nakamura

Faculty of Global Management, Chuo University, Higashinakano Hachioji-shi, 192-0393, Japan
Yoshinobu Ishiwata

Graduate School of Medicine, Yokohama City University, 3-9, Fukura, Yokohama Shi Kanazawa Ku, 236-0004, Japan
Email: a19.3gem@g.chuo-u.ac.jp, jyulis.77f@g.chuo-u.ac.jp, ishi_y@yokohama-cu.ac.jp

Abstract
This study examines gaze movement differences in diagnosis between skilled and beginner physicians, aiming to
identify factors influencing diagnostic speed and developing human resource and enhance artificial intelligence's
diagnostic capabilities. Results reveal that experienced physicians, on average, spent 61% less diagnostic time than
beginners, covering 49% of the gaze distance on the X and Y-axes. (comparison of the cumulative distance of gazeon the screen). Despite increased movement on the Z-axis (comparison of the scrolling speed of the CT scan results),
skilled physicians moved 2-3 times faster, effectively narrowing attention and identifying specific areas.
Keywords: Experimental Economics, Eye-tracking, Cognitive Science, Artificial Intelligence (AI)the understanding, aims to contribute to the realization ofdiagnosis like that of skilled physicians by beginner
physicians and AI.
Diagnosis using imaging test results requires time andeffort to master, resulting in a difference in the speed atwhich skilled physicians and beginner physicians canmake a diagnosis. This is thought to be due to thedifference in knowledge and experience between skilled
and beginner physicians based on previous research [2].
This paper visualizes and analyzes the eye movement ofboth physicians resulting from the difference inknowledge and experience.
Humans are thought to possess the property of being
able to make automatic inferences about stimuli withwhich they have frequent contact through practice, andthis process of developing automaticity in judgment
through practice is called procedural processing [4].
Procedural processing has been found to enable quicker
decisions, improve reaction speed, and emphasize
rational and practiced judgment [5]. It is thought thatsuch procedural judgment is also made by experienced
physicians when they use imaging test results to makediagnoses.
It is difficult to verbalize and provide effective
guidance for diagnoses that are made unconsciously
using such procedural judgments. Therefore, in this paper,by using a method [6] to visualize the eye movements ofexperienced physicians and beginner physicians during
diagnosis by measuring eye gaze and to analyze thedifferences from a cognitive perspective, this paper aims
to clarify the factors that lead to the speed with which

1. Introduction
Examination work in the medical industry is generally
considered time-consuming and highly labor-intensive totrain one CT service technician to acquire the knowledge
necessary to prepare interpretation reports [1]. In addition,a nationwide survey of the actual status of the reading
system for chest interpretation of radiograms for medical
examinations, based on a questionnaire targeting 239
radiologists in 2018 [2], also indicated that it is difficult
to train radiologists in a short period of time and requires
a certain level of clinical and reading experience. In
addition, the results of the survey indicated that theburden on radiologists is increasing due to the aging of
the workforce and that the spread of CT scans will further
reduce the manpower needed for the interpretation ofradiograms, which may lead to the breakdown of thecurrent radiogram system.
This has led to increased awareness of human resource
development and medical AI development. Medical AI isbeing developed using deep learning. However, because
deep learning has a black box problem in which thedecision-making process cannot be identified by humans,
the development of medical AI that can explain theprocess is being explored in the medical field, where
safety is a high priority [3].
Considering the above issues, this study does not aim
to contribute to the development of medical AI based on
deep learning. This study aims to understand humans
through the diagnosis of skilled physicians, and based on©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
907

Taiki Sugimoto, Jun Nakamura, Yoshinobu Ishiwata⚫ Beginner (1~3 years of career) 1 person
With the cooperation of 2 physicians⚫ Abnormal findings 15 people (cases)⚫ No findings 15 people (cases)
A total of 30 people (cases) were tested forinterpretation of head CT scan results, and eye gaze wasmeasured using Tobii Pro Lab Screen Edition.
When interpreting the 30 cases, both physicians were
not informed of the findings, the examination images
were random, and both physicians diagnosed the cases inthe same order, beginning and ending the diagnosis ofeach case at the physician's own discretion.
In comparing the experienced physician and beginner
physician in this paper, the following two types ofanalyses were conducted.
⚫ Comparison of the cumulative distance of gazeon the screen (X-axis and Y-axis movement):
Abnormal findings - 15 cases.
⚫ Comparison of the scrolling speed (Z-axis
movement) of CT scan results: Abnormal
findings - 3 cases.
The cumulative distance of the gaze (movement of the
X-axis and Y-axis) was performed with reference to thevelocity-detection method [6], which measures the
distance of gaze movement on the screen. In addition, in
this paper only analyzed abnormal findings (15 cases)from 30 cases.
The scrolling speed of the imaging test results(movement of the Z-axis) is an analysis unique to thispaper; the movement of the Z-axis is an expression also
used in the interpretation of radiogram and corresponds
to the scrolling speed (diagnostic time and movement)
between images of CT test results that capture crosssections. In the analysis, the diagnostic time for each
image was recorded at the time the examination images
were switched. The recording was done manually, andthe error in the timing of when the physician switched
images was kept within a range of 0 microseconds to 4
microseconds or less. If the physician did not change thescroll direction within 5 images, the time was recorded atintervals of 5 images, and if the physician did change thescroll direction within 5 images, the time was recorded
each time an image was switched.
Both analyses were conducted with the purpose of
focusing on gaze movements in the X-axis, Y-axis, andZ-axis to investigate whether the range of gazemovements of the hypothesized experienced physician's
eye would be narrower than the range of gaze movementsof the beginner physician's eye.

experienced physicians make diagnoses.
2. Purpose
The purpose of this paper is to contribute to humanresource development and Medical AI development byclarifying factors that lead to the speed of making adiagnosis through a cognitive approach, such asmeasuring the eye movement of experienced physicians
and beginner physicians during the interpretation ofradiograms.
This paper is a study of novelty and academic
significance. Conventional economics has focused on the
results of decision-making and attempted to understandhuman behavior through mathematical decision-making
models. However, this paper attempts to understandhuman behavior through a cognitive approach such as eye
measurement by visualizing and analyzing the diagnostic
process during the interpretation of a radiogram by using
imaging tests. In addition, unlike previous studies thatanalyzed decision-making by comparison, this paper is
unique in that it analyzes decision-making in the processof developing automaticity [4] as mentioned above.
This paper has social significance in that it aims tocontribute to the development of beginner physician anddevelopment of medical AI by making the gaze at thetime of diagnosis visible and clarifying factors that leadto the speed of giving a diagnosis.
3. Hypothesis
The hypothesis was developed as differences in theway experienced physicians and beginner physicians
move their gaze during the interpretation of radiograms.
Experienced physicians have a narrower range of eye
movement when searching for the presence or absence ofsymptoms/points of symptoms compared to beginner
physicians. Since both interpretations of radiograms by
experienced physicians and batting by experienced hittershave in common that they are performed using procedural
judgment [4], a process in which the aforementioned
automaticity of judgment has developed, previous studies[7] in which the gaze of experienced hitters while batting
was studied were consulted. The results showed thatexperienced batters fix their gaze on the entire pitcher intheir peripheral vision, fixing their gaze by on the pointof attention prediction, rather than on a wide area asbeginner batters gaze at a wide range. Based on thisfinding, it was hypothesized that the gaze movements ofexperienced physicians would be narrower than those of
beginner physicians.

5. Results
Comparison of cumulative distance of gaze (X-axis
and Y-axis movements)
Fig.1, Fig.2 below shows a comparison of the
cumulative distance of gaze when the same case wasdiagnosed by an experienced physician and a beginner
physician. Due to space limitations, the figures for eachcase are not shown here, but rather the total cumulative
distance and the total normalized diagnosis time for all

4. Methods
In this paper, experiments were conducted under thefollowing conditions.
Yokohama City University Hospitals⚫ Experienced (more than 10 years career) 1
person©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
908

Visualization of the Skilled⚫

cases, table summarizing their sum, mean, and variance,
and figure showing the mean cumulative distance for thecases with findings (15 cases). The horizontal axis
represents the time of diagnosis.
Table 1. X-axis and Y-axis movement in Abnormal
Finding cases.
Experienced_normalized time

Experienced_cumulative distance of gaze

Beginner_normalized time

Beginner_cumulative distance of gaze

Case1

66.99

31042.43

100100626.18

Case4

84.16

99326.15

10087496.00

Case5

57.23

115339.97

100128587.59

Case6
Case10
Case13
Case14
Case15
Case17
Case18
Case19Case20
Case21
Case24
Case27SUM
AVERAGEVARIANCE

43.91
70.13
64.24
55.58
58.08
52.90
39.92
77.37
61.02
46.54
83.93
86.07
948.0763.20
205.2566014.34
63618.04
42570.61
43520.29
44683.93
40321.40
27541.30
52913.18
34938.71
34720.16
66331.85
76429.26
839311.6355954.11
608630602.91

100
100
100
100
100
100
100
100
100
100
100
100
1500
100
0

180581.15
112969.58
73038.85
90969.10
96657.33
84090.70
173406.17
85824.34
134411.07
168388.14
154028.48
166290.33
1837364.99
122491.00
1325636396.42

Experienced physician never exceeded thecumulative distance of the beginner physician'sgaze for more than half of the diagnostic time inall 15 cases except Case 4 (Fig.2).

Comparison of scrolling speed (Z-axis movement) ofimaging test results
Fig.3 below is representative Figure that shows acomparison of the representative Z-axis gaze shift of thegaze when the same case was diagnosed by anexperienced physician and a beginner physician.

Fig. 3. Z-axis movement
The following two points were clarified.
⚫ Experienced Physician move back and forth
1~1.5 times between imaging test results until
beginner physician confirm all imaging test
results (Fig.3).
⚫ Compared to the Beginner Physician, who
tended to move back and forth along the Z-axis
at many points and was not able to narrow down
the range of attention, the Experienced
Physician tended to identify the points ofattention and gradually narrow down the rangeof attention along the Z-axis (Fig.3).

Fig.1. Average of X-axis and Y-axis movement

6. Discussion
Fig.2. Case 4 X-axis and Y-axis movement
The purpose of this paper is to contribute to humanresource development and medical AI development byclarifying factors that lead to the speed of making adiagnosis through a cognitive approach, such asmeasuring the eye movement of experienced physicians
and beginner physicians during the interpretation ofradiograms.
In analyzing the gaze, this paper hypothesized that
Experienced physicians would have a narrower range ofeye movement when searching for the presence orabsence of symptoms and the location of symptoms than
beginner physicians.
As a result of the analysis, it can be said that the
hypothesis is valid according to the following two
analysis results for the movement of the X-axis and Yaxis.
⚫ Experienced physician will require an averageof 49% of the cumulative gaze distance of the
beginner physician for all 15 cases except Case
4 (Fig.2).

The horizontal axis represents diagnostic time. Amongthe diagnoses made by experienced physician andbeginner physician diagnosticians, the diagnosis
completion time of the one with the longest diagnosis
time was set to 100 and normalized. The vertical axis
represents the cumulative distance of eye movements
until the end of the diagnosis, calculated by dividing alleye movements in the screen where the examination
image is displayed in Tobii Pro Lab by the coordinates of
the X-axis and Y-axis.
A comparison of the results for 15 cases revealed thefollowing three points.
⚫ Experienced physician require an average of
61% of the diagnostic time of beginner
physician for all 15 cases. (Table 1, Fig.1).
⚫ Experienced physician will require an averageof 49% of the cumulative gaze distance of the
beginner physician for all 15 cases except Case
4 (Fig.2).

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
909

Powered by TCPDF (www.tcpdf.org)

Taiki Sugimoto, Jun Nakamura, Yoshinobu Ishiwata⚫

Experienced physician never exceeded thecumulative distance of the beginner physician'sgaze for more than half of the diagnostic time inall 15 cases except Case 4 (Fig.2).
In addition, about Z-axis,⚫ Compared to the Beginner Physician, who
tended to move back and forth along the Z-axis
at many points and was not able to narrow down
the range of attention, the Experienced
Physician tended to identify the points ofattention and gradually narrow down the rangeof attention along the Z-axis (Fig.3).
This is related to the hypothesis that the range of eye
movement of Experienced physicians is narrower than
the range of eye movement of beginner physicians.
However, in all the cases investigated in this paper, thenumber of eye movements back and forth on the Z-axis
for Experienced physicians was greater than that for
beginner physicians Thus, in addition to substantiating
the hypothesis, the experienced physicians made fewer
eye movements between the X-axis and Y-axis than the
beginner physicians, but made more eye movements in
the Z-axis, suggesting that the experienced physician may
perceive and diagnose imaging test results in a threedimensional rather than a planar manner.
These findings and discussions in this paper will leadto human understanding through the skilled physician's
diagnosis and contribute to the future development of
human resource training and medical AI.

Finally, in terms of future prospects, this paper will
share the results of this analysis with physicians, and by
deepening the discussion, explore effective humanresource development methods and way of developing
medical AI.
8. Acknowledgements
I would like to thank Mr. Konishi of Focus System, Inc.
for his support and advice on this study. I would also liketo thank Professor Takizawa, H., for his advice on thispaper from the perspective of Experimental Economics.
References
1.

2.

3.

4.
5.

7. Conclusions

6.

The purpose of this paper was to contribute to humanresource development and medical AI development byfocusing on the differences in eye movement between
experienced physicians and beginner physicians during
diagnosis and to identify factors that lead to the speed atwhich they make a diagnosis.
As a result of analyzing eye movement during
diagnosis in two ways: cumulative distance of eye
movement (movement on the X-axis and Y-axis) andscrolling speed of imaging test results (movement on the
Z-axis), it was found that experienced physician was
faster than beginner physician in making a diagnosis,
moving less distance on the X and Y axes, and usually
moving less than the distance of beginner physician. On
the Z-axis, however, they tended to identify the area of
focus and gradually narrow the range of attention but
traveled more distance and in a shorter time than the
beginner physician.
Next, the limitations of this paper are discussed. Dueto technical problems, this paper has not been able toanalyze the Z-axis shift of all cases. Besides improving
the reliability of the analysis results, it would be desirable
to continue analyzing the Z-axis shift, which may lead to
new discoveries. In addition, this paper only compares
one experienced physician and one beginner physician
each. In the future, it is desired to increase the number ofsubjects and to include intermediate physicians who are
between the levels of the two physicians in this paper toincrease the reliability of the results.

7.

H. Ichikawa, H. Yamaguchi, T. Noda, and K. Tanaka, The
effect of CT interpretation assistance by radiological
technologists in off hours emergency medical care. Journalof Japanese Society for Emergency Medicine 17(4), 2014,
pp. 535-542
H. Mitomo, T. Nakayama, K. Ashizawa, C. Endo, T.
Kobayashi, M. Sato, K. Shibuya, T. Sobue, D. Takenaka,
K. Nishii, M. Harada, S. Maeda, Y. Maruyama, H. Miura,
K. Murata, and M. Sagawa, Nationwide Questionnaire on
the Actual Status of an Interpretation System for Lung
Cancer Screening by Chest Radiography. The Japan Lung
Cancer Society 58 (4), 2018, pp. 243-251
M. Terasaki, Y. Terasaki, and A. Shimizu, Artificial
Intelligence (AI) and Pathological Diagnosis. Journal of
Nippon Medical School 19 (2), 2023, pp. 72-77
E. R. Smith, Model of social inference processes.
Psychological Review 91(3), 1984, pp. 392–413
M. A. Zarate and E.R. Smith, Person Categorization and
Stereotyping. Social Cognition 8(2), 1990, pp. 161–185
R. A. Monty, and J. W. Senders, Eye Movements: On-line
measurement, analysis, and control. Hillsdale: Psychology
Library Editions: Perception, 1976
T. Katou, and T. Fukuda, Visual Search Strategies ofbaseball batters during the preparatory phase of batting.
The Japanese journal of ergonomics 38(6), 2002, pp. 333340

Authors Introduction
Mr. Taiki Sugimoto
He is a 1st year graduate student at thefaculty of Economics, Chuo University,
Japan. The areas of his research interest
are Experimental Economics, Behavioral
Economics, Cognitive Science, andArtificial Intelligence.

Dr. Jun Nakamura
He is a professor at the Faculty of Global
Management, Chuo University, Japan.
Previously he was working as a professor
at the Graduate School of Engineering
Management, Shibaura Institute ofTechnology. His main research areas are
Cognitive Science and Humanities.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
910

Toward Temporal Taste Sensing: Contrasting Human Experience with Machine Perception inFood Appreciation
Hiroki Fxyma

Kobe University, 1-2 Tsurukabuto, Nada-ku, Kobe, Japan

Abstract
This paper explores the temporal aspects of food and beverage appreciation, particularly focusing on the interplay
between taste sensors and human sensory experiences. It begins by discussing the fundamental principle of taste
sensors, which analyze chemical substances but often fail to capture the complexities of human taste perception. To
address this limitation, the concept of "artificial tongues" is introduced, aiming to replicate human taste perception
more accurately. The paper then delves into the temporalities inherent in our eating experiences, categorized into
three dimensions: the time of the object, the time of embodied cognition, and the time of the phenomenon. Four
temporal aspects of the phenomenon are proposed: non-temporal, linear temporality, circular temporality, and point
temporality, which are shaped by the eater's attention. The discussion further examines the motivations behind thetemporality of food, highlighting cognitive and physical constraints such as memory and oral capacity. Additionally,
the paper investigates temporalities in dessert appreciation, using the example of a strawberry mille-feuille to illustrate
how the arrangement of ingredients and the sequence of consumption create unique temporal experiences. Finally, it
concludes by emphasizing the need for philosophical and theoretical advancements to bridge the gap between humansensory experiences and machine sensing, thereby enhancing our understanding of taste perception in both realms.
Keywords: Temporality, Artificial tongues, Taste sensors, Aesthetics

and the taste that remains after swallowing (aftertaste).
These are computed based on the ease with which taste
components detach from the lipid membrane, which is ataste sensor, thereby detecting, for example, a richaftertaste. However, none of the current taste sensors
measure changes in perception over time, such as fatigue
with the same taste. Also, effects such as contrast
between ingredients (e.g., strawberry after cream) must
be measured separately and evaluated by humans.

1. Introduction
1.1. Taste sensor and the temporality
The basic principle of taste sensors is to analyze chemical
substances that are taste components. Sugar meters andsalt meters are the most basic of such sensors. However,a precise analysis of the chemical composition of fooddoes not represent the taste that humans perceive. The
characteristics of human taste perception include, forexample, the fact that humans perceive different chemical
substances (sucrose and aspartame) as the same taste
stimulus(“sweetness”)
as
well
as
the
inhibitory/synergistic effect of one taste on the perceived
intensity of another taste. To reflect such cognitive
characteristics of human perception and output "the taste
perceived by humans" instead of simply detecting taste
substances, sensors are referred to as artificial tongues.
As Toko [1] reviews, there are two types ofcommercialized electronic tongues in the world. One is
the taste sensing systems SA402B and TS-5000Z,
commonly known as the taste sensor, and the other is the
Astree II e-tongue.

1.2. Temporalities in our eating experience
Taste and smell are gaining status in aesthetics, buttheories concerning their basic appreciation are
insufficient. Fxyma [2] proposed the basic theory for
temporality of food and beverage appreciation.
According to the study, the temporality underlying foodappreciation is classified into three categories: the time ofthe object, the time of the embodied cognition, and thetime of the phenomenon. Next, Fxyma proposed four
aspects of the time of the phenomenon: non-temporal,
linear temporality, circular temporality, and point
temporality. These four temporalities are not determineda priori but generated by the eater’s attention.

These electronic tongues are intelligent sensors thatreflect the human taste perception system in theircomponent analysis as information processing, but theirdetection of temporality is limited to a small portion. The
SA402B and TS-5000Z developed by Toko et al. cananalyze taste as it is perceived on the tongue (initial taste)

2. Motivations of the Temporality of food
Temporality as a phenomenon in the human eating
experience arises from cognitive and physical constraints.
In this section, I outline the primary constraints: memory©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
911

Hiroki Fxyma(cognitive constraints) and oral capacity (physicalconstraints).

one meal to the stomach in several to several tens of times.
In the case of eating the entire amount of food at once,
there is no temporality. The division of food itself
becomes the basic unit of temporality, such as order andrepetition. For example, beef steak is divided into several
pieces with a knife, finely crushed with several times thenumber of chewing, and sent to the stomach.

The first constraint is memory. Memory is generally
divided into sensory memory, short-term memory, andlong-term memory. The retention period of sensory
memory varies by sensory modality, but is short, ranging
from less than one second to several seconds. In contrast,
its retention capacity is larger than that of short-term
memory. Sensory information is then stored as short-term
memory, medium-term memory, and then long-term
memory. Human senses cannot reproduce stimuli. We
can recall pain and taste, but this is not a reproduction of
the pain or taste "itself. If it were possible to reproduce
the pain itself, we would experience intense pain again
every time we explain the situation of a serious injury.

Human meals are embedded in multiple cyclicaltemporalities. The time span ranges from seconds, suchas repeated chewing and swallowing, to levels such as
three meals a day (rise and fall of blood sugar in somehours), and complex cyclical cycles such as foodaversion/preference learning (in some hours to days), or
seasonal foods (in a year). When analyzing food withtaste sensors, it is liquefied with a mixer and thecomposition of flavor components is analyzed. Here,temporalities such as division with a knife or repetitive
chewing are disregarded, and analysis is conducted astimeless, ignoring point time or time passage.

On the other hand, a machine's sensors can reproduce
the stimuli as numerical values; the values measured 10
years ago can once again be input into the sensors; thewine opened 10 years ago and the wine opened now canbe tasted "at the same time". The human being seems to
be comparing the two, but one is the sensor data of theactual wine in front of us, while the other is the edited andpartly erased/enhanced long-term memory of the wine of
10 years ago.

The analysis times increase significantly when thenumber of steps in the implicit integral is expanded. In
the implicit method, the solutions of simultaneous
equations are obtained by inserted Newton-Raphson
method, and numerical substitutions into symbolic matrix
within this process require the computation processing
time. The symbolic matrix is enlarged by increasing thenumber of implicit stages, which takes more computation
time. Therefore, it is important to implement an efficient
substitution process into a symbolic matrix for the
implementation of higher order implicit methods in MBD
analysis.

This means that the time possessed by the machine isbasically point temporality in Fxyma's temporality
classification. The present and the past appear
simultaneously in the here and now. It is not Linear
temporality because there is no passage of time or order.
It is also different from Circular temporality because it
recreates the same stimulus in the past. Circular
temporality is about finding repetition in different
experiences. For example, spring is repetitive, but thecontent of the spring experience is not identical every
year. The reproduction of exactly the same stimuli isdifferent from the concept of Circular temporality. Thus,
the simultaneous appearance of the here and now as acondensed time of the present and the past is Point
Temporality.

3. Temporalities in Dessert Appreciation
In this section, I will examine the process by whichhumans experience food, using one dish of dessert as an
example. This dessert shown in Fig. 1 is provided at the
pastry shop Assiette Dessert Maruyama, located in
Nishinomiya City, Japan.

This constraining characteristic of human memory to beedited, forgotten, and reinforced over time gives rise to
Linear Temporality or Circular temporality. When eatinga parfait, the first strawberry, the next taste of cream, andthe subsequent strawberries are not experienced
separately, but have a linear continuity or a circular
temporality structure as repetitions of strawberries. Whena person eats a parfait, temporal memory is passed onfrom one bite to the next; if the same taste continues in
the mouth, the person becomes bored, and if the sweet
flavor of the cream is followed by the sour jam, there is acontrast. In the repetition of strawberries and cream, thestrawberry is presented with a different role each time it
appears. Here, more than the summation of the elements,
an aesthetic experience emerges.

Fig.1 The Strawberry Millefeuille of Assiette Dessert
Maruyama.

The second constraint is the physical structure of thenasal and oral cavities. Generally, the volume of the oralcavity is smaller than that of the stomach, and humans
divide food during meals, sending the entire amount of

The strawberries scattered on the plate play different
roles in the beginning and the end of the dish. Of course,
they also play a role in giving an impact at the moment©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
912

Powered by TCPDF (www.tcpdf.org)

Toward Temporal Taste Sensing

strawberries scattered on the plate serve as “privileged
moments [3], bringing the consciousness of the eater backto the cyclical time.

they are presented. However, this function is a visual one.
The strawberries eaten at the beginning of the mealfunction to tell the taste of the strawberries or to introduce
the flavor of the strawberries used as an ingredient. In
order to understand the function of the strawberries in thelatter stages, it is necessary to interpret the flavor of themillefeuille in the main body of the dish. The millefeuilleis characterized by the horizontal structure of the cut
layers, rather than the more common style in which layersof pie and cream are piled up vertically.

4. Conclusion
This paper examines the differences between humansensory experiences and machine sensing through thedescription of the experience of eating a dessert on a plate.
I argue that the human experience of eating is temporal innature, primarily motivated by two constraints: thelimitations of memory and oral capacity. In contrast,
machine taste sensing, without these constraints, is anexperience characterized by "point time" or"timelessness," where past, present, and future
information are analyzed homogeneously.

In a typical layered millefeuille, one must cut the pastrywith a fork to achieve the intended size, which is not only
cumbersome but also risks disrupting the balance
between the pie pastry and the strawberries by causing
cream to spill out. However, in Maruyama’s millefeuille,
a bite-sized portion is pre-cut, allowing one to enjoy thepie pastry, cream, and strawberries atop withoutcompromising their balance.

The current limitations of sensing in reflecting the
temporal aspects of the human eating experience,
especially linear or cyclical time, do not stem frominadequacies in sensor capabilities or the scientificknowledge supporting them.

In Maruyama’s mille-feuille, the meticulous
calculation behind this bite-sized composition becomesfully evident as one progresses through consuming it. As
the layers of cream and pastry are meticulously arrangedside by side, defying the conventional definition of amille-feuille, scooping up a set with a fork automaticallyresults in an accompanying strawberry, completing amouthful. The pastry is not of the rigid variety but rather
consists of delicate, thin layers stacked upon one another.
When pressed against the custard cream with the tongue
and teeth, the pastry dissolves effortlessly without
requiring a bite, instantly enveloping the tongue in thesmoothness of the cream. There is no sensation of
strawberry juice separating from the custard cream andflowing out. While the strawberries provide a fibroustexture and a refreshing acidity to the tip of the tongue,
the carefully controlled juice harmoniously melds withthe pastry and cream, creating an impression of“strawberry custard cream” rather than separating out.

Rather, the deficiency lies in the philosophical andtheoretical understanding of what should be sensed. Welack theories to explain the experience and phenomena ofconsuming a dessert or savoring a glass of wine, which
limits taste sensing to mere component analysis. Taste
sensors themselves are capable of detecting taste
components more “accurately” than humans,“objectively” across various conditions, and with an
unparalleled sensitivity to subtle taste components. If
taste sensors are to transcend mere component analysis
and aim to reproduce "eating" as an intellectual behaviorof machines, what is required is philosophical andtheoretical leadership in science.
References[1]

K. Toko, Y. Tahara, M. Habara, Y. Kobayashi, and H.
Ikezaki, “Taste Sensor: Electronic Tongue with Global
Selectivity,” in Essentials of Machine Olfaction and Taste,
1st ed., T. Nakamoto, Ed., Wiley, 2016, pp. 87–174. doi:
10.1002/9781118768495.ch4.
[2] H. Fxyma, “The Temporality of the Aesthetic Appreciationof Food and Beverages,” Contemp. Aesthet., 2024.
[3] E. Souriau, “Time in the plastic arts,” J. Aesthet. Art Crit.,
vol. 7, no. 4, pp. 294–307, 1949.

The perfect harmony of pastry, cream, and strawberries
creates an impression of "strawberry custard cream"
where everything melts together in the mouth, to theextent that one might almost forget the awareness of"eating strawberries" as they progress through the middle
and later stages. To prevent this forgetting, the role of the
strawberries scattered on the plate from the middle to theend comes into play. The strawberries scattered on theplate ensure a return to the main theme that mightotherwise be lost due to the remarkable harmony.

Authors Introduction

This mille-feuille is likely intended to be consumed
from left to right. Evidence of this can be found in theabsence of cream at the right end, indicating a
progression. In the overall linear progression of time,each bite-sized millefeuille forms a condensed, Point
Temporality through the perfect harmony of its
ingredients. While the repetition of millefeuille layers
from left to right introduces a recursive CyclicalTemporality, there comes a moment when continuity
threatens to turn into monotony and a sense of timelessmonotony (i.e., Atemporality). At this juncture, theDr. Hiroki Fxyma
He is currently an Assistant Professor
at Kobe University. He completed his
doctoral and master's degrees at Keio
University. His current research topic
is the aesthetic appreciation of taste
and its cognitive process.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
913

Deep Learning Based Prediction of Heat Transfer Coefficient
Using Spectrogram Images from Boiling Sound
Fuga Mitsuyama

Department of Mechanical and Control Engineering, Kyushu Institute of Technology,
1-1 Sensui-cho, Tobata-ku, Kitakyushu, Fukuoka, 804-0015, Japan
Ren Umeno

Department of Mechanical and Control Engineering, Kyushu Institute of Technology,
1-1 Sensui-cho, Tobata-ku, Kitakyushu, Fukuoka, 804-0015, Japan
Tomohide Yabuki

Department of Mechanical and Control Engineering, Kyushu Institute of Technology,
1-1 Sensui-cho, Tobata-ku, Kitakyushu, Fukuoka, 804-0015, Japan
Tohru Kamiya

Department of Mechanical and Control Engineering, Kyushu Institute of Technology,
1-1 Sensui-cho, Tobata-ku, Kitakyushu, Fukuoka, 804-0015, Japan
E-mail: mitsuyama.fuga621@mail.kyutech.jp, umeno.ren696@mail.kyutech.jp,
yabuki.tomohide556@mail.kyutech.jp, kamiya@cntl.kyutech.ac.jp
Abstract
Cooling methods based on boiling have attracted attention as a thermal solution for electronic equipment. In thissituation, it is necessary to measure the heat transfer coefficient (HTC) to design more efficient cooling systems. In
this paper, we propose a method to predict of the HTC from boiling sound data using deep learning techniques. The
accuracy improved by 1.12% compared to the conventional method through the development of Convolutional
Neural Network (CNN) incorporate Convolutional Block Attention Module (CBAM).
Keywords: Boiling Sound, Heat Transfer Coefficient (HTC), Convolutional Neural Network (CNN), CBAM.

1. Introduction
The heat generation density of CPUs and GPUs isincreasing year by year as they become more highly
integrated [1]. Under these circumstances, the coolingmethod using water boiling is currently attracting
attention. Boiling cooling has a high heat transfer
coefficient (HTC) compared to gas forced convection
cooling using heat sinks and fans [2], which is still in use
today. Furthermore, since water serves as the boiling
medium, it is environmentally friendly and economical.
Since the boiling absorbs heat through the evaporation of
the boiling liquid and the convection induced by the
bubble motion [3], the HTC significantly depends on thenumber and size of the bubbles. In fact, the measured
HTC shows a significant change when the number andshape of the bubbles are varied, as shown in Fig. 1.
Therefore, it is important to accurately measure the HTC
to develop more efficient and reliable cooling systems.

This paper proposes a method for predicting the HTC
from spectrogram images of the boiling sound using
Convolutional Neural Network (CNN). This method isbased on the idea that there is a relationship between
boiling sound and HTC. We conduct a classification task
by categorizing the HTCs obtained through experiments
into five classes. Then, we apply Convolutional Block©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
914

Fuga Mitsuyama, Ren Umeno, Tomohide Yabuki, Tohru Kamiya

Attention Module (CBAM) [4] to a CNN model toimprove the classification accuracy.

a robust model against noise. We specify the labels foreach class ranging from 10–20, 20–30, 30–40, and 40–50[kW/m2/K]. These label numbers represent the HTC.

2. Method
2.3. Convolutional Block Attention Module(CBAM)

2.1. Creation of dataset

In order to improve the accuracy of the conventional
CNN model, we incorporate CBAM and compare its
accuracy with that of the conventional model. The
structure of the CNN model with CBAM is shown in Fig.
3. The CBAM architecture is based on the paper by S.
Woo et al [4], where it is found that higher accuracy
results can be obtained by connecting the Channel
Attention Module and the Spatial Attention Module inseries. The Channel Attention Module is a mechanismthat learns and weights the importance of information ineach channel based on the correlation of features between
channels. The Spatial Attention Module is a mechanismthat learns and weights the importance of information in
the spatial axis, allowing the extraction of important
information in the spatial direction in the feature map.

We experimented with sound data acquisition to create
a dataset of spectrograms. Then, this acquired sound data
was transformed into a spectrogram using the Fast

Fig. 1. Boiling state transitions and associated changes in
the quantity and shape of bubbles

Fourier Transform (FFT) [5] to generate a 200 x 200[pixels] image. Fig. 2 gave an overview of the

3. Experimental Result and Discussion
3.1. Evaluation
The evaluation method is the Five-Fold Cross
Validation and average accuracy, which is the average ofthe accuracy of each class in the five-class classificationand the average of the accuracy for all data in each dataset.

Fig. 2. Overview of experimental setup for sound dataacquisition

Table 1 The number of data in each classClass [kW/m2/K]
Number
10–20
880
20–30
440
30–40
440
40–50
900
50–60
420
Total
3080

experimental setup. A 1-mm-thick, 10-mm-squarecopper plate soldered to the top of a copper heater block
into which cartridge heaters are inserted was used as aheat-transfer wall. Thermocouples inserted in the copper
block were used to the surface measure temperature andheat flux. Before the boiling experiment, the pure waterused as the boiling medium was degassed for more than
1 hour. Boiling sound was measured using Miniature
Hydrophone Type 8103® [6] at a sampling frequency of
50 [kHz]. Table 1 shows the number of data for each the
HTC class.
2.2. Classification by CNN
The model is based on the CNN model developed by
Tabata et al [7] and Sinha, K.N.R et al [8]. A basic blockof CNN in Fig. 3 consists of a convolution layer (with akernel size of 3 x 3 and 96 channels), batch normalization,
ReLU functions (as activation functions) and average
pooling (with a kernel size of 2 x 2). Then, four stacks ofthese blocks downsample the input images and construct©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
915

Deep Learning Based Prediction

Fig. 3. Configuration of the proposed model with CBAM

(Accuracy is defined as the proportion of correct andpredicted labels that match).

provide typical data for these classes, and we canrecognize horizontal lines of constant frequency near thecenter of the images. On the other hand, Fig. 5 is an
example of images that could not be classified correctly,
showing that the features appeared in Fig. 4 are notclearly represented or are faded due to noises in Fig. 5.
The CBAM mechanism may have made it more difficult
to classify images like those in Fig. 5 by over-extracting
features from data with typical features like those in Fig.
4.

3.2. Result
The results of the proposed method using the datasets
obtained from the experiments in Section 2.1 are shownin Table 2 and compared with the those of theconventional method. The accuracy in each classdecreased by 1.09% in the 10–20 [kW/m2/K] class andby 0.95% in the 30–40 [kW/m2/K] class but increased
significantly in the other classes. In particular, theaccuracy in the 40–50 [kW/m2/K] class improved by
3.38%. The average accuracy of the proposed method is
91.61% compared to 90.49% for the conventional
method. That's an improvement of 1.12%.

4. Conclusion
In this paper, the incorporation of CBAM into theconventional CNN model decreased accuracy by 1.09%
in the 10–20 [kW/m2/K] class and by 0.95% in the 30–40[kW/m2/K] class, but significantly increased accuracy inthe other classes, with the average accuracy improvingby 1.12%. Future work will include increasing thenumber of experiments to build a CNN model that ismore robust to noise, as one reason for the decrease inaccuracy is the limited number of datasets. In addition,
we will identify features common to each class and builda model that takes these features into account, with theaim of predicting the HTC in more detail and further
improving accuracy.

3.3. Discussion
The main reason why the average accuracy was
improved except for the 10–20 [kW/m2/K] and 30–40[kW/m2/K] classes is that CBAM was able to extract the
response of the constant-frequency component of thehorizontal line, which is unique to each HTC. Thatallowed CBAM to maintain high accuracy even for noisy
data sets. On the other hand, the inaccuracy could be
attributed to the presence of various noises added during
the creation of dataset, which aimed to construct a modelthat can be robust to noise. We have concluded that the
spectrograms obtained from the experiments in Section
2.1 may not reflect the unique frequency characteristics
found at each HTC. The spectrograms shown in Fig. 4©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
916

Powered by TCPDF (www.tcpdf.org)

Fuga Mitsuyama, Ren Umeno, Tomohide Yabuki, Tohru Kamiya

Table 2 Results of five classes classification from spectrograms using two CNN model
10 ~ 2020 ~ 3030 ~ 4040 ~ 50
50 ~ 60
Average Accuracy [%]CNN
CNN + CBAM

95.01
93.92

84.90
86.90

88.45
87.50

90.08
93.46
7.

1000
[Hz]

-60
-80
-100

-60
-80

8.

-100

-120
-120
-140

89.80
91.15

90.49
91.61

M. Tabata, H. Lu, T. Kamiya, S. Mabu, S. Kido,“Automatic Classification of Respiratory Sound
Considering Hierarchical Structure”, 2022 The 22ndInternational Conference on Control, Automation and
Systems (ICCAS), pp. 2-3, 2022.
Sinha, K.N.R et al., “Deep Learning the Sound of Boilingfor Advance Prediction of Boiling Crisis”, Cell Reports
Physical Science, pp. 3-12, 2021.

-140
-160
[dB]

0

-160

Authors Introduction

1 [s]

0

Fig. 4. Typical spectrogram of 10–20 and 30–40
2[kW/m /K] class
1000
[Hz]

-60
-80
-100

-60
-80
-100

-120

Mr. Fuga Mitsuyama
He received his Bachelor’s degree inEngineering in 2023 from the Faculty ofEngineering,
KyushuInstituteoftechnology in Japan. He is currently amaster student in Kyushu Institute ofTechnology, Japan.

-120
-140
-140
-160
[dB]

[dB]
0
0

-160

1 [s]

Fig. 5. Spectrogram of failed classification in the2
10–20 and 30–40 [kW/m /K] class

Reference
1.

2.

3.

4.

5.

6.

N. Unno, “[Towards t he Realization of Boiling Cooling
Devices
Using
BubbleRefinement
Boiling]
Kihoubisaikahuttou wo tukatta huttoureikyakudebaisu
jitugen ni mukete”, JSME TED Newsletter, No.89, pp. 24, 2019. (In Japanese)
M. Suzuki, K. Kawaguchi, T. Terao, E. Tanaka, H. Tanaka,“Boiling Refrigerant Type Compact Cooling Unit for
Computer Chip (Comparison with Other Type Cooling
Unit for Computer Chip)”, The Japan Society of
Mechanical Engineers, pp. 97-98, 2000.
T. Tanaka, K. Miyazaki, T. Yabuki, “Observation of heattransfer mechanisms in saturated pool boiling of water byhigh-speed infrared thermometry”, International Journalof Heat and Mass Transfer, 170, 121006, 2021.
S. Woo, J. Park, JY. Lee, IS. Kweon, “Cbam:
Convolutional block attention module”, Proceedings ofthe European conference on computer vision (ECCV), pp.
3-8, 2018.
H.J. Nussbaumer, “Fast Fourier Transform andConvolution Algorithms”, Springer-Verlag, pp. 80-111,
1982.
MrHBK, “MINIATURE HYDROPHONE TYPE 8103”,
https://www.bksv.com/-/media/literature/ProductData/bp0317.ashx, Nov. 22, 2023.

Mr. Ren Umeno
He received his Master’s degree inEngineering in 2023 from the Faculty ofEngineering,
KyushuInstituteoftechnology in Japan. He is currently adoctoral student in Kyushu Institute ofTechnology, Japan.
Dr. Tomohide Yabuki
He is an Associate Professor in the
Department of Mechanical and Control
Engineering at Kyushu Institute ofTechnology and is a PRESTO researcher ofJapan Science and Technology Agency(JST). His research interests are in the areasof boiling heat transfer, MEMS thermal measurement
techniques, and single cell thermal analysis.
Dr. Tohru Kamiya
He received his B.A. degree in ElectricalEngineering from Kyushu Institute ofTechnology in 1994, the Masters and Ph.D.
degree from Kyushu Institute ofTechnology in 1996 and 2001,
respectively. He is a professor in the
Department of Mechanical and Control Engineering at
Kyushu Institute of Technology. His research interests arefocused on image processing, medical application of image
analysis, etc. .

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
917

A Study on Classification of Faulty Motor Sound Using Convolutional Neural Networks
Md Shafayet Jamil

Graduate School of Engineering, University of Miyazaki, 1-1, Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
Praveen Nuwantha Gunaratne

Interdisciplinary Graduate School of Agriculture and Engineering, University of Miyazaki, 1-1, Gakuen KibanadaiNishi, Miyazaki, 889-2192, Japan
Hiroki Tamura

Faculty of Engineering, University of Miyazaki, 1-1, Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
Email: z322t01@student.miyazaki-u.ac.jp, ti20060@student.miyazaki-u.ac.jp, htamura@cc.miyazaki-u.ac.jp

Abstract
Classification of sound has its usage in various fields in today’s world. In this paper we will go through the soundclassification techniques for the detection of faulty machines with the help of the sound data produced by the machine.
The focus is towards determining the pertinency of audio classification methods to detect faulty motors by theirsounds; both in noisy and noise-free scenarios; so that the requirement of human inspection can be reduced in factories
and industries. Noise reduction plays such an important role in improving accuracy of detection some researchers
simulated data by adding noise for benchmarking their models. Hence noise reduction is widely used in audio
classification tasks. Among various available methods, we have implemented an autoencoder for noise reduction. We
have conducted the classification tasks on both noisy and denoised data using Convolutional Neural Network.
Accuracy of classification on the data denoised using autoencoder is compared with the noisy ones. For classification,
we used spectrogram, Mel-frequency cepstral co-efficient (MFCC) and Mel-spectrogram images. These processes
yield promising results in distinguishing faulty motors by their sound.
Keywords: Faulty motor detection, autoencoder, convolutional neural network (CNN), audio classification

1. Introduction
Audio data is one of the most common multimedia
types used in almost every sector of modern life
throughout the past decades. The availability of audio
recording devices has also been increased and the usageof audio data can be found in security surveillancesystems, health monitoring systems, and various
autonomous systems along with the inseparable usage indaily human life [1], [2], [3], [4]. Successful utilization ofaudio resources depends on the efficiency ofclassification (process of identification of pre-definedlabel for audio signals [5]), transcription, and perceptionof underlying contents through a bunch of transforming
algorithms and machine learning approaches carried out
on various sources of audio data, as in, voices, music,
environmental sound, traffic sound etc. [1]. Our research
aimed at distinguishing the faulty motors from the normal
ones by means of the soundwave processing followed bytraining with convolutional neural network so that it canbe used in factories to reduce the requirement for humaninspection, and in preventing hazards. As the adequacy ofclassification often gets drastically affected by noise [6],we also have incorporated an autoencoder based noise
reduction model in parallel to the classification process.
CNN has many proven outcomes in the field of audio

classification [7], [8], [9]. Convolutional neural networks
have been used widely for classification since it has beenfound competent for both image classification e.g.
vehicle classification [9] and sound classification [10]on various fields namely environmental soundclassification (ESC) [11], animal sound classification[10], [12], vehicle classification [7], [13], emotion
detection [14], violence detection [15] etc.
In this paper we will be showing results of applying
Lenet-5 on image transforms of the audio signal. Lenet-5
was developed on the context of classification of images
especially text [16]. However, while we can transform
sound data into various image representations it can be an
auspicious tool for classification of audios as well. We
found approaches with applying CNN for the task ofEnvironmental sound classification [11] and deploying
Lenet for acoustic scene detection [17]. Despite the time
consumption issues with large scale datasets CNNs arewell suited for classification tasks within a few
parameters [18]. For our research we have used Lenet-5
for classifying audios by means of transforming audios
into feature images. We have used 3 types of feature-©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
918

Md Shafayet Jamil, Praveen Nuwantha Gunaratne, Hiroki Tamura

Fig. 1 Overview of faulty motor classification
images (Mel-Frequency Cepstral Coefficient, Melspectrogram, and Spectrogram) from both the noise-freeoriginal and smoothed (with an Autoencoder) audio
chunks. Finally, we fed the classifier with both (noisy andsmoothed) type of feature images separately andcompared the resultant accuracy for noise association andthe above mentioned 3 types of feature images as well.
The process yields promising results. The whole process
is shown in Fig. 1.

3.1. Noise reduction
Autoencoders are used for noise reduction in various
audio recognition experiments [19]. An autoencoder is acomposition of an encoder and a decoder block having
convolutional and transpositional layers respectively. It
can reconstruct the input without noise through a seriesof compressions and then decompressions [20]. We
optimized the autoencoder model described in [20] byaccompanying 4 convolutional layers and 4 transposition
or deconvolutional layers as shown in Fig. 2. Also, in thetransposition layers hyperbolic tangent activation (tanh)
is used instead of PRelU for keeping the negative
elements [21] in the resultant audios and to preserve anear-zero mean [22]. The results showing the smoothingwith both types of activations are shown in Fig. 3 and Fig.
4 by superimposing the smoothed wave (yellow) onto theoriginal one (blue).

2. Dataset and Preprocessing
Our dataset is comprised of 890 samples of motor sounds,
divided into two classes, 645 normal sound and 245
samples of abnormal sound; both are of variable length
audio file in the form of ‘.wav’ file at 48000Hz. The audio
data was acquired using electronic stethoscopes. In
preprocessing we split 3 seconds long chunks from theaudio samples, having a length of 3 * 48000; and thesamples which’s lengths are less than 144000 (3 * 48000)
were zero padded. Finally, each datapoint is normalized
in such a way that the underlying values of the data
stream contain real numbers ranging from -1.0 to 1.0 only.

Input
Encoder
Conv1D (128, k =7)
Conv1D (32, k =5)
Conv1D (16, k =3)
Conv1D (8, k=3)
Decoder
Transpose (8, k =3)
Transpose (16, k=3)
Transpose (32, k=5)
Transpose (128, k=7)
Output

3. Methodology
For classifying the uniform length audios of normal andabnormal motor sound; derived from the preprocessing;
we divided the task into 2 standalone pipelines. The firstone involves smoothing with autoencoder and the second
one doesn’t incorporate any noise reduction measure.
In the later sections we have discussed these steps.

Fig. 2 Autoencoder Model©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
919

A Study on Classification

3.2. Feature extraction
The selection of features has significant influence onclassification accuracies [4]. That is why we fed both thenoisy and smoothened audios through a transformation
process which yield 3 types of feature images- a.
Spectrogram, b. Mel-spectrogram, c. Mel-frequency
cepstral coefficient (MFCC). We used a Python-based
library called Librosa [23] for these processing.
a. Spectrogram
Transforming one dimensional audio signal into matrix
like representation makes them suitable for training withneural networks [4]. Spectrograms are such visual
depictions of frequency-wise strength of an audio which
can improve classification results [24]. A sample
spectrogram of a 3-second-long normal motor sound isshown in Fig 5 (a).

Fig. 3 Smoothing with autoencoder (tanh activation)b. Mel-spectrogram
Mel scaled spectrograms can represent the signal
analogous to the human auditory system. The formula for𝑓converting f Hz into m mel is 𝑚 = 2595 log10 (1 +)
700
[4]. The mel-spectrogram of a 3 second sound ofabnormal motor looks like Fig. 5 (b).

Fig. 4 Smoothing with autoencoder (PRelU activation)c. Mel-frequency cepstral coefficient (MFCC)
MFCC incorporates Discrete Cosine Transform and has
a compressed representation signal [4] so it is quite usefulfor training audio features. We have put an MFCC imageof a smoothened normal motor sound (3 second) in Fig.
5 (c).
3.3. Classification
Image classification networks also have good results onsound data [24]. So, we trained these feature images witha CNN based model Lenet-5 (shown in Fig. 6) [16] which
was proposed for classifying handwritten characters withrequiring low preprocessing. Our implementation of
Lenet-5 has an identical structure. It has 2 convolutional
layers working as feature extractors followed by 3 fully
connected layers working as classifier, as it appears in Fig.
7.

Fig. 5. Feature image samples a) Spectrogram b) Melspectrogram c) MFCC

Fig. 6 Generic architecture of Lenet-5 [16]©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
920

Md Shafayet Jamil, Praveen Nuwantha Gunaratne, Hiroki Tamura

5. Conclusion

Feature extractor
C1: Conv2d (3, 32, k=5)
S2: AvgPool2d (k=2)
C3: Conv2d (32, 64, k=5)
S4: AvgPool2d (k=2)
Classifier
C5: Convolution (128)
Gaussian Con.: Linear (84)
Output: Linear (2)Sound classification with the help of CNN could be afeasible solution to various realistic problems especiallyin automation of maintenance and monitoring scenarios
in large scale factories. It can reduce the need for
manpower and improve Mean time to response (MTTR)
by providing automated detection of faulty machines inindustries. The scope of this paper could also be easily
extended to other fields where a small number of classes
are needed to be classified with high precision.

Fig. 7 Structure of the implemented Lenet-5 model.

Table 1. Performance with and without autoencoder
With
Without
FeatureAutoencoder Autoencoder
Spectrogram
98.39871898 98.71897518
Mel-spectrogram 99.93327996 99.90659194
MFCC
99.82652789 99.91638471

Table 2. Performance by dividing motors into twogroups (motor 1-50 and motor 51-93)
FeatureFig. 8 Loss function of the autoencoder
Spectrogram
Melspectrogram
MFCC

4. Results
The results derived from the various experiments can bedivided into 4 parts. Firstly, the performance of theautoencoder in terms of loss function (refers to Fig. 8).
We have trained the autoencoder model up to 1.6% ofloss using the metrics of mean squared errors (MSE),
but the autoencoder model with 2.3% loss yield the best
result during classification. Secondly, the straightforward performance comparison between MFCC, Melspectrogram, spectrogram image-based classification of
the model. Thirdly, comparison between the performancewith and without using autoencoder smoothing, results ofsecond and third steps are summarized in Table. 1.
Fourthly, reviewing the impact of different dataset
combinations, amounts data, and biases as shown in
Table. 2. The best accuracies (%) found in these threesteps are shown in Table 1 and Table 2. To put it in anutshell, we can deduce that at least 97% accuracy could
be expected from this approach in classifying 2 classes ofmotors (normal and abnormal) where a totally unknown
type of motor sound is not present in the testing data. And
as for noise reduction, it is not evident to have positive
influence on classification accuracy with the current
configuration of the model of the autoencoder that has
been used for smoothing.

Motor1–50
normal:Motor 5193
abnormalMotor51–
93
normal:Motor 150
abnormalMotor1–50
normal:Motor150
abnormal

Motor
51–93
normal:Motor
51-93
abnormal

97.001

97.26

98.86

98.29

99.7

99.72

99.95

99.88

99.9

99.9

99.95

99.99

References
1.

2.

3.

4.

5.

6.

L. Gao, K. Xu, H. Wang, and Y. Peng, “Multirepresentation knowledge distillation for audio
classification,” Multimedia Tools and Applications, vol.
81, no. 4, pp. 5089–5112, Jan. 2022, doi:
https://doi.org/10.1007/s11042-021-11610-8.
C. Clavel, T. Ehrette, and G. Richard, “Events Detection
for an Audio-Based Surveillance System,” IEEE Xplore,
Jul.
01,
2005.
https://ieeexplore.ieee.org/document/1521669
Y.-T. Peng, C. Lin, M. Sun, and K.-L. Tsai, “Healthcare
audio event classification using Hidden Markov Models
and Hierarchical Hidden Markov Models,” Jun. 2009, doi:
https://doi.org/10.1109/icme.2009.5202720.
M. Turab, T. Kumar, M. Bendechache, and T. Saber,“Investigating Multi-feature Selection and Ensembling for
Audio Classification,” International Journal of Artificial
Intelligence & Applications, vol. 13, no. 3, pp. 69–84, May
2022, doi: https://doi.org/10.5121/ijaia.2022.13306.
Tuomas Virtanen, M. D. Plumbley, and D. Ellis,“Computational Analysis of Sound Scenes and Events.”
Springer,2017,
https://link.springer.com/book/10.1007/978-3-319-634500
J. Meyer, L. Dentel, and F. Meunier, “Speech Recognitionin Natural Background Noise,” PLoS ONE, vol. 8, no. 11,
Nov.
2013,
doi:
https://doi.org/10.1371/journal.pone.0079279©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
921

Powered by TCPDF (www.tcpdf.org)

A Study on Classification

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

K. W. Cheng et al., “Spectrogram-based classification on
vehicles with modified loud exhausts via convolutional
neural networks,” Applied Acoustics, vol. 205, p. 109254,
Mar.
2023,
doi:
https://doi.org/10.1016/j.apacoust.2023.109254.
C. Yang, X. Gan, A. Peng, and X. Yuan, “ResNet Basedon Multi-Feature Attention Mechanism for Sound
Classification in Noisy Environments,” Sustainability, vol.
15,
no.
14,
p.
10762,
Jan.
2023,
doi:
https://doi.org/10.3390/su151410762.
M. A. Butt et al., “Convolutional Neural Network Based
Vehicle Classification in Adverse Illuminous Conditions
for Intelligent Transportation Systems,” Complexity, vol.
2021,
pp.
1–11,
Feb.
2021,
doi:
https://doi.org/10.1155/2021/6644861.
F. Merchan, A. Guerra, H. Poveda, H. M. Guzmán, and J.
E. Sanchez-Galan, “Bioacoustic Classification of
Antillean Manatee Vocalization Spectrograms Using Deep
Convolutional Neural Networks,” Applied Sciences, vol.
10,
no.
9,
p.
3286,
May
2020,
doi:
https://doi.org/10.3390/app10093286.
Z. Mushtaq and S.-F. Su, “Environmental soundclassification using a regularized deep convolutional
neural network with data augmentation,” Applied
Acoustics, vol. 167, p. 107389, Oct. 2020, doi:
https://doi.org/10.1016/j.apacoust.2020.107389.
Yin, Y., Tu, D., Shen, W., & Bao, J. ,“Recognition of sickpig cough sounds based on convolutional neural networkin field situations,” Information Processing in Agriculture,
Nov.
2020,
doi:
https://doi.org/10.1016/j.inpa.2020.11.001.
M. A. Hedeya, A. H. Eid, and R. F. Abdel-Kader, “ASuper-Learner Ensemble of Deep Networks for VehicleType Classification,” IEEE Access, vol. 8, pp. 98266–
98280,
2020,
doi:
https://doi.org/10.1109/access.2020.2997286.
R. Cai, L. Lu, H.-J. Zhang, and L. Cai, “Highlight soundeffects detection in audio stream,” Jan. 2003, doi:
https://doi.org/10.1109/icme.2003.1221242.
S. Pfeiffer, S. Fischer, and W. Effelsberg, “Automatic
audio content analysis,” Proceedings of the fourth ACM
international
conferenceonMultimedia
MULTIMEDIA
’96,
1996,
doi:
https://doi.org/10.1145/244130.244139.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradientbased learning applied to document recognition,”
Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324,
1998, doi: https://doi.org/10.1109/5.726791
Venkatesh Duppada and Sushant Hiray, “Ensemble Of
Deep Neural Networks For Acoustic Scene Classification,”
arXiv (Cornell University), Aug. 2017, doi:
https://doi.org/10.48550/arxiv.1708.05826.
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet
Classification with Deep Convolutional Neural Networks,”
Communications of the ACM, vol. 60, no. 6, pp. 84–90,
May 2012, doi: https://doi.org/10.1145/3065386.
S. Alharbi et al., “Automatic Speech Recognition:
Systematic Literature Review,” IEEE Access, vol. 9, pp.
131858–131876,
2021,
doi:
https://doi.org/10.1109/access.2021.3112535.
K. Bajaj, D. K. Singh, and Mohd. A. Ansari,“Autoencoders Based Deep Learner for Image Denoising,”
Procedia Computer Science, vol. 171, pp. 1535–1541,
2020, doi: https://doi.org/10.1016/j.procs.2020.04.164.
Swalpa Kumar Roy, S. Manna, Shiv Ram Dubey, and B.
B. Chaudhuri, “LiSHT: Non-Parametric Linearly Scaled
Hyperbolic Tangent Activation Function for Neural

Networks,” arXiv (Cornell University), Dec. 2018, doi:
https://doi.org/10.48550/arxiv.1901.05894.
22. S. R. Dubey, S. K. Singh, and B. B. Chaudhuri,“Activation functions in deep learning: A comprehensive
survey and benchmark,” Neurocomputing, vol. 503, pp.
92–108,
Sep.
2022,
doi:
https://doi.org/10.48550/arXiv.2109.14545.
23. B. McFee et al., “librosa: Audio and Music Signal Analysis
in Python,” Proceedings of the 14th Python in Science
Conference, 2015, doi: https://doi.org/10.25080/majora7b98e3ed-003.
24. Boddapati, V., Petef, A., Rasmusson, J., & Lundberg, L.,“Classifying environmental sounds using image
recognition networks,” Procedia Computer Science, vol.
112,
pp.
2048–2056,
Jan.
2017,
doi:
https://doi.org/10.1016/j.procs.2017.08.250.

Authors Introduction
Mr. Md Shafayet Jamil
He received his bachelor’s degree inscience in 2016 from the Institute ofInformation Technology, Jahangirnagar
University, Bangladesh. He is currently amaster’s student in University ofMiyazaki, Japan.
Mr. Praveen Nuwantha Gunaratne
He received his Bachelor’s degree inEngineering in 2018 from the Faculty ofEngineering, University of Moratuwa, Sri
Lanka. He is currently a Doctoral student in
University of Miyazaki, Japan
Prof. Hiroki Tamura
He received the B.E. and M.E. degree from
Miyazaki University in 1998 and 2000,
respectively. From 2000 to 2001, he wasan Engineer in Asahi Kasei Corporation,
Japan. In 2001, he joined Toyama
University, Toyama, Japan, where he wasa Technical Official in the Department of Intellectual
Information Systems. In 2006, he joined Miyazaki
University, Miyazaki, Japan, where he was an AssistantProfessor in the Department of Electrical and Electronic
Engineering. Since 2015, he is currently a Professor in the
Department of Environmental Robotics. His main researchinterests are Neural Networks and Optimization Problems.
In recent years, he has had interest in Biomedical Signal
Processing using Soft Computing.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
922

Design and Development of a Flexible Active Ankle Joint Orthosis for Locomotion Assistance
Praveen Nuwantha Gunaratne

Interdisciplinary Graduate School of Agriculture and Engineering, University of Miyazaki, 1-1, Gakuen KibanadaiNishi, Miyazaki, 889-2192, Japan
Hiroki Tamura

Faculty of Engineering, University of Miyazaki, 1-1, Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
Email: ti20060@student.miyazaki-u.ac.jp, htamura@cc.miyazaki-u.ac.jp

Abstract
Active ankle joint orthosis is an artificial wearable device that is closely fitted to the human ankle and combines
human intelligence and intentions with the powered robot joints. However, most of the existing designs have only
focused on one degree of freedom (DOF) of the ankle joint: namely plantarflexion-dorsiflexion. Alternatively, a
modular power-assist flexible active ankle orthosis is proposed, addressing anthropomorphic architecture of design,and supporting multiple DOFs of ankle for locomotion assistance of the physically weak individuals. The proposed
ankle joint orthosis was designed with the aid of CAD tools followed by fabrication of a working model. Electricallypowered systems with gear transmissions were adapted to support the principal motions during normal Gait. In
parallel, the design of the control algorithm was carried out based on the EMG signals. The future advancements havebeen focused on developing a novel control method that provides sufficient flexibility to assist a wide variety oflower-limb motions.
Keywords: Active ankle orthoses, power-assist robots, anthropomorphic design, joint axes mapping, rehabilitation

musculoskeletal strength, motor control, and gait),human locomotion assistance (i.e. targeted at paralyzed
patients who have lost motor and sensor function in theirlower limbs) and human power augmentation (i.e.
enhance human strength and endurance duringlocomotion and enable individuals to perform tasks thatthey cannot easily perform by themselves).

1. Introduction
Every human being would prefer to spend their entire
lifetime as an independent individual without becoming
dependent on others for locomotion. According tostatistics around the globe, the elderly population exceeds
10% of the total population and conversely, the numberof people suffering from neurological or musculardisorders is on the rise. As such, the medical industry has
shown a growing interest in using exoskeleton robots forrestoring patients’ mobility and to alleviate health careissues arising from locomotion difficulties.

The ankle joint complex is concerned, encompassing
the lower leg and foot, serves as the crucial linkfacilitating interaction between the lower limb and theground. Especially the essential aspect for activities likewalking and daily tasks. Despite experiencing significant
compressive and shear forces during walking, the ankle's
robust structure provides a high level of stability. The key
ankle joint complex movements include sagittal plane,
plantarflexion-dorsiflexion, transverse plane, abductionadduction, and frontal plane, inversion-eversion [1].

With the continuous development in biomedical
engineering field over the past few decades, the robotic
orthosis devices have been used for many different
applications. The active orthosis/ robotic exoskeleton isan artificial wearable device that is powered by the meansof electric motors, pneumatics, hydraulics, or linear
actuation methods. In the current market, active orthoses
are classified as upper extremity systems, lowerextremity systems, and full-body systems. The lower
extremity exoskeletons (LEE) are designed to support
anatomical functions of different joints of the lower
extremity. In general, hip, knee and ankle joints haveseven degrees of freedom (DOF) per limb. The LEEs areprimarily developed for three main types of applications
such as gait rehabilitation (i.e. helping patients withmobility
disordersintherehabilitationofWhile commonly referred to as the 'ankle joint,' thefoot's movement is facilitated by several articulations.
The ankle joint complex comprises the subtalar joint(talocalcaneal), tibiotalar joint (talocrural), andtransverse-tarsal joint (talocalcaneonavicular) [1]. The
subtalar joint's geometry permits ankle inversion andeversion, with a primary role in these movements.
Although other motions are possible, the bulk of footinversion and eversion occurs here. The talocrural joint
acts as a hinge, mainly contributing to plantar- and©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
923

Praveen Nuwantha Gunaratne, Hiroki Tamura

dorsiflexion. The transverse tarsal joint is considered partof the functional unit with the subtalar joint, sharing acommon axis, and contributing to foot inversion-eversion.
The combining motions at the subtalar and talocrural
joints results in three-dimensional supination andpronation [2], [3].

inversion and eversion to complement this flexion
movement [7].
As illustrated in Fig. 2, the anatomical axes of talocruraland subtalar joints are placed in an oblique sense thatcause foot to move across all three planes allowing
pronation and supination to occur during walking [6], [8].
Hence, constraining motions to a single plane can lead toabnormal joint movements, poor muscle recruitment andincreasing overall energetic cost. Therefore, it is essential
to facilitate multiple DOF to improve exoskeleton
performance by active or passive means.

While some suggest the talocrural joint is multi-axial
due to internal rotation during dorsiflexion and externalrotation during plantarflexion, evidence indicates it may
be uniaxial. The observed simultaneous motions are
attributed to its oblique axis. Similar to the talocrural joint,
the subtalar joint has an oblique axis, contributing tomultiple motions during plantar- and dorsiflexion,
leading to pronation and supination [4].
In the past, there was a traditional belief that
dorsiflexion and plantarflexion were exclusively
associated with the motion of the talocrural joint, while
inversion and eversion were thought to occur solely at thesubtalar joint. However, recent perspectives have rejected
the idea of completely segregating these motions to eachjoint. Although the majority of plantar and dorsiflexion isstill attributed to the talocrural joint, there is now
acknowledgment that a few degrees of these motions also
involve the subtalar joint [5].
Despite some of the research efforts aimed at creating
active ankle orthoses to aid human movements,
challenges persist in effectively supporting natural
human motions. These challenges revolve around thegoal of enabling limbs to move freely and comfortably,
particularly in relation to the corresponding oblique axes.
2. Design Considerations

Fig. 1. The outputs from gait analysis representing ankle
complex rotation in sagittal and frontal planes [6].

Designing a device that will effectively interact with thehuman dynamic conditions is a difficult task toaccomplish, especially at the distal end of limbs. Gait
analysis can be used as an objective tool for quantifying
motion of lower limb joints and forces that act upon thesejoints. However, gait analysis cannot separate it joint
wise due to the major limitation of accurately measuring
talus motion using skin-mounted markers. Fig. 1 depicts
example gait analysis data of the ankle joint complex
kinematics [6]. During a normal gait cycle, the stancephase begins at heel strike, where the dorsiflexors are
eccentrically contracting to lower the foot to the groundbased on the sagittal motion of the ankle. Then the ankle
moves from plantarflexion to dorsiflexion during which
the shank rotates forward around the ankle and after thatthe foot rotates around the forefoot phase continuing until
maximum plantarflexion (approximately 14°) being
achieved at toe-off. In the swing phase of walking, theankle undergoes dorsiflexion, allowing the foot to lift off
the ground and then transitioning to a slightplantarflexion upon heel strike. Simultaneously, there ismotion at the subtalar joint, involving around 15° of

Fig. 2. The axis of rotation at the talocrural joint andsubtalar joint - oblique axes of rotation (red) are shown in
different views; a) talocrural joint; b) subtalar joint.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
924

Design and Development of

The two significant DOF, namely, plantarflexiondorsiflexion and inversion-eversion are externally
powered using electrical actuators and remaining DOF,
namely, internal- external rotation is passively supported.
In addition, with the intention of maximizing user
compatibility and performance, the mechanisms
proposed in the mechanical design have compatible joint
axes for each DOF. The CAD model of the flexible active
ankle joint orthosis is shown in Fig. 3.

3.1. Plantarflexion/ dorsiflexion assembly
The movement of the ankle predominantly takes place inthe sagittal plane, where plantarflexion and dorsiflexion
are mainly observed at the talocrural joint. Various
research studies have suggested a range of motion (ROM)in the sagittal plane, typically falling between 65° and 75°.
This encompasses a span from 10° to 20° of dorsiflexion
to 40°–55° of plantarflexion [9]. Here the plantarflexiondorsiflexion axis of the suggested ankle joint brace isaccurately aligned with the talocrural axis of the natural
joint. Fig. 4 illustrates the plantar-/dorsiflexion assemblywith the component axis.

Fig. 3. CAD model of the proposed ankle joint orthosis;a) conceptual design; b) CAD model.

3.2. Inversion/ eversion assembly
In the frontal plane, the complete range of motion isaround 35°, consisting of 23° of inversion and 12° ofeversion [10]. The axis of inversion-eversion in theflexible active ankle joint brace is accurately aligned withthe subtalar axis of the natural joint. The assembly
indicated in Fig. 5 has been mechanically formed andpositioned normal to the subtalar axis of ankle.

Fig. 4. The axis of rotation and osteo-kinematics at theplantar-/dorsiflexion assembly a) component axis (red)[8]; b) developed plantar-/dorsiflexion assembly.

3.3. Internal/ external rotation assembly
The internal/external rotation mechanism is crafted andadjusted to closely align with the ankle's axis of rotation.
A stabilizer system is employed to maintain the assemblyin its neutral position.

4. Prototype Development
The prototype incorporates tri-planar motion withoblique axes. The spacing between the two motion axes
(sagittal and frontal plane motion) is adaptable toaccommodate the wearer, making it one of the mosthuman-like structures among the initial design concepts.
Following multiple manufacturing processes at the
production facility, the final prototype was successfully
assembled. Fig. 6 illustrates the conclusive form of theactive ankle orthosis device, while Table 1 provides
detailed specifications outlining the design features of thedeveloped prototype.

Fig. 5. The axis of rotation and osteo-kinematics at the
inv-/ eversion assembly a) component axis(red) [8];b) developed inv-/ eversion assembly.
3. Mechanical Design
This flexible active ankle joint orthosis is a multipurpose
robotic ankle exoskeleton that can perform rehabilitation
exercises as well as locomotion assistance. It is worn as abilateral system and supports all three DOF of the ankle.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
925

Powered by TCPDF (www.tcpdf.org)

Praveen Nuwantha Gunaratne, Hiroki Tamura

294,
Oct.
2008,
doi:
https://doi.org/10.1016/j.jmbbm.2008.01.005.
4. A. Lundberg, O. Svensson, G. Nemeth, and G. Selvik,“The axis of rotation of the ankle joint,” The Journal of
Bone and Joint Surgery. British volume, vol. 71-B, no. 1,
pp. 94–99, Jan. 1989, doi: https://doi.org/10.1302/0301620x.71b1.2915016.
5. J. H. Calhoun, F. Li, B. R. Ledbetter, and S. F. Viegas, “A
Comprehensive Study of Pressure Distribution in the
Ankle Joint with Inversion and Eversion,” Foot & Ankle
International, vol. 15, no. 3, pp. 125–133, Mar. 1994, doi:
https://doi.org/10.1177/107110079401500307.
6. C. L. Brockett and G. J. Chapman, “Biomechanics of theankle,” Orthopaedics and Trauma, vol. 30, no. 3, pp. 232–
238,
Jun.
2016,
doi:
https://doi.org/10.1016/j.mporth.2016.04.015.
7. M. Nordin, Basic biomechanics of the musculoskeletal
system by Margareta Nordin Victor H. Frankel.
Philadelphia: Lippincott Williams And Wililins, 2001.
8. “ Ankle and Foot,” Clinical Gate, Mar. 18, 2015.
https://clinicalgate.com/ankle-and-foot/ (accessed Nov. 16,2023).
9. S. K. Grimston, B. M. Nigg, D. A. Hanley, and J. R.
Engsberg, “Differences in Ankle Joint Complex Range of
Motion as a Function of Age,” Foot & Ankle, vol. 14, no.
4,
pp.
215–222,
May
1993,
doi:
https://doi.org/10.1177/107110079301400407.
10. R. N. Stauffer, “Total Ankle Joint Replacement,” Archivesof Surgery, vol. 112, no. 9, p. 1105, Sep. 1977, doi:
https://doi.org/10.1001/archsurg.1977.01370090087018.

Fig. 6. Prototype of the proposed mechanical structure

Table 1. Design specifications of the flexible active
ankle joint orthosis
Characteristic
ROM (Degree)
Actuation method
Transmission

Max. torque output (Nm)PlantarDorsiflexion
55/20
Motor
drivenBevel andspur gearsystem
22.3

InversionEversion
35/25
Motor
drivenSpur gearsystem2.3

5. Conclusion
Authors Introduction

The distinctive feature of the design for the flexible active
ankle joint brace lies in its capability to meet thefunctional and ergonomic demands of the natural ankle
joint. This is accomplished through precise alignment of
the orthosis axes of rotation with the oblique axes ofrotation found in the talocrural and subtalar joints of theankle. The innovative mechanism, inclusive of the drive
units, is situated anteriorly to the shank and foot segments.
The proposed active ankle brace is easily adaptable for
performing ankle rehabilitation exercises, serving as arobotic therapy module to aid paralyzed individuals inregaining or enhancing their functional mobility, or
assisting those with physical weakness in their motion.
Future work will involve creating an EMG-based
controller for the flexible active ankle joint orthosis. Thisaims to enable real-time control by accurately
anticipating the patient's intended motion through precise
predictive capabilities.

Mr. Praveen Nuwantha Gunaratne
He received his Bachelor’s degree inEngineering in 2018 from the Faculty ofEngineering, University of Moratuwa, Sri
Lanka. He is currently a Doctoral student
in University of Miyazaki, Japan

Prof. Hiroki Tamura
He received the B.E. and M.E. degreefrom Miyazaki University in 1998 and
2000, respectively. From 2000 to 2001,
he was an Engineer in Asahi Kasei
Corporation, Japan. In 2001, he joined
Toyama University, Toyama, Japan,
where he was a Technical Official in the
Department of Intellectual Information Systems. In 2006,
he joined Miyazaki University, Miyazaki, Japan, where he
was an Assistant Professor in the Department of Electricaland Electronic Engineering. Since 2015, he is currently aProfessor in the Department of Environmental Robotics.
His main research interests are Neural Networks andOptimization Problems. In recent years, he has had aninterest in Biomedical Signal Processing using Soft
Computing.

References
1.
2.
3.

H. Gray. Gray's anatomy: with original illustrations byHenry Carter. Arcturus Publishing, 2009.
Cailliet R: Foot and Ankle Pain. Philadelphia: FA Davis
Co, 1968“Biomechanics of the ankle joint and clinical outcomes of
total ankle replacement,” Journal of the MechanicalBehavior of Biomedical Materials, vol. 1, no. 4, pp. 276–

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024926

A study on the Real-Time Biomechanical Analysis of Lumbar Burden Utilizing Stereoscope
CamerasTaufik Hidayat Soesilo

Graduate School of Engineering, University of Miyazaki, 1-1 Gakuen Kibanadai-Nishi, Miyazaki-shi,
Miyazaki, 889-2155, Japan
Praveen Nuwantha Gunaratne

Interdisciplinary Graduate School of Agriculture and Engineering, University of Miyazaki, 1-1, Gakuen KibanadaiNishi, Miyazaki, 889-2192, Japan
Hiroki Tamura

Faculty of Engineering, University of Miyazaki,1-1 Gakuen Kibanadai-Nishi, Miyazaki-shi, Miyazaki, 889-2155, Japan
E-mail: ti22068@student.miyazaki-u.ac.jp, ti20060@student.miyazaki-u.ac.jp, htamura@cc.miyazaki-u.ac.jp
Abstract
Lumbar region is susceptible to strain and stress due to various physical activities and occupational task. To study
about the lumbar burden, researchers have developed many tools but most of existing design can only use static
imaging or doesn’t provide real-time update. The proposed system records and examines the change of body posturein real time by utilizing the capabilities of stereoscope cameras. By using MediaPipe algorithms, 2D key-points
representing body joint can be extracted from images. Afterward, using the Direct Linear Transform (DLT) the
corresponding 3D key-points can be calculated using obtained 2D key-points. With the 3D key-points the body angles
can be computed and used to calculate the wight of the lumbar burden using JACK calculation. Finally, the proposedsystem reached to its aim to study about lumbar burden and adjust the person needs in real-time.
Keywords: Lumbar burden, real-time biomechanical analysis, computer visionembedded on the smartphone to measure changes in body
inclination [4]. This change in body inclination will thenbe used to estimate the lumbar burden that occurs on the
body and classify the movements the body makes [5]. On
the other hand, the Kinect sensor [6-8] uses a camera tocapture human images and a depth sensor to measure thedepth of key-points captured by the camera. This data
will then be used to obtain 3-dimensional body key points.
Afterward, the 3D key-points are being used to estimate
the body posture, angles, and the biomechanical of thehuman body. Each approach has its own advantages anddisadvantages. for example, smartphones can be used in
all conditions. However, based on the measurement
criteria, several devices need to be attached to the bodyand cannot be monitored in real-time. As for Kinect
sensor, the sensor can find out various information onparts of the body at once, but Kinect sensor cannot be
used under sunlight.

1. Introduction
The lumbar region of the spine, commonly referred toas the lower back, is susceptible to strain and stress due
to various physical activities and occupational tasks.
Around 577.0 million people worldwide, or approx. 7.5%of the total population, were projected to have low backpain at any given time in 2017, and that number is
continuing to rise [1]. This incident occurs due to doing
the same activities that affect the lumbar region over along period of time. Examples of activities that cause thisinclude agricultural work, nursing, construction, etc.
Therefore, this needs to be monitored and corrected
before it gets exaggerated. One of the consequences thatcan occur due to ignoring pain in the lumbar region isdisability [2], [3].
Therefore, many researchers are trying to monitor andimprove body movements with various approaches.
Among these methods, the approach that is often to be
seen is using smartphones and Kinect sensors [4], [5], [6],[7], [8]. The approaches have already taken theirrespective advantages. However, when it is applied onreal scenarios, such approaches demand several
additional features. For example, smartphone-based
approach, it employs gyro and accelerator sensors

The proposed system is designed for real-time
biomechanical analysis of the body, considering the use
in different environmental conditions. This research
employs a camera stereoscope to analyze the body andcarry out comprehensive calculations in order to identify
the lumbar burden in real time.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
927

Taufik Hidayat Soesilo, Praveen Nuwantha Gunaratne, Hiroki Tamura

allows the system to effectively detect movements within
a 10-meter range while minimizing noise interference.

2.2. Data Acquisition
Data collection is carried out according to the flowchart
depicts Fig.2. In order to carry out biomechanicalanalysis, the system needs to obtain 3D key-points so thatmonitoring of the desired body parts can be obtained [9].
Initially, the camera captures an image of the subject
positioned in front of it. Then the obtained image
undergoes processing via the MediaPipe algorithm toextract 2D key-points. Given that the images are derived
from two distinct perspectives through two cameras, apair of key-point perspectives right and left cameras,
illustrated in Fig.3. To achieve 3D key-points, some
calculations are conducted using DLT method. However,calibration is needed to derive the translation and rotation
parameters of the camera in order to carry out DLT
calculation. The obtained 3D key-points serve as
foundational data for extracting biomechanical insightsfrom the body. Afterwards, in order to compute lumbar
burden, it’s necessary to deepen the understanding ofbody posture, one of which is through the body angle.
This angle is obtained through vector calculations
utilizing the known 3D key-points coordinates. This
process involves presenting the 3D key-points within athree-dimensional environment, as visualized in Fig.4.

Fig. 1. Camera setup

Fig. 2. System Flowchart
2. Methodology

Fig. 3. 2D key-points from left and right cameras

2.1. System setup
The study employed a stereoscope system comprising
two cameras to investigate the activities. The cameras
were strategically placed on a 1-meter-high camera
support with a 1-meter separation, as depicted in Fig. 1.
Distance between the two cameras was carefully
addressed, as overly close placement could constrain theview, leading to more noise and less precision. Use of
stereoscopic approach creates a three-dimensional image,and it leads to a comprehensive understanding ofcomplex movement patterns involved in walking and thefactors that influence them. Further, this configuration©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024928

A study on the

3. Results and Discussion
In order to know the effect of load on lumbar burden, acomparison is needed by providing different loads. In
recent research, the load given was divided into 3 loads,
namely 0 kg or no load, 2.5 kg load, and 5 kg load. The
movement carried out by the subject is the same
movement as shown in Fig 7. but the system simulates
the target of lifting a different load. The results of thelumbar burden analysis can be seen in the Fig. 8.
Based on the results, it can be observed that the system
can analyze the lumbar burden according to the
movements made by the subject. Moreover, the load that
occurs in the lumbar region is directly proportional withthe load that being lifted. The maximum load that occurs
in the lumbar region when the subject was at the lowest
point and tries to lift the weight; 2098 N for 0 kg, 2843 N
for 2.5 kg, and 3612 N for 5 kg (Table 2). Further,minimum, and average values can also be found in Table
2.

Fig. 4. 3D key-points

2.3. Lumbar burden calculation
The estimation of lumbar burden involves thecomputation of body posture angles, as illustrated in
Fig.5. Calculations are performed according to formulas
Eq. 1, 2, 3, 4, 5, 6.
𝐴 = (𝑥1 , 𝑦1 , 𝑧1 )(1)

𝐵 = (𝑥2 , 𝑦2 , 𝑧2 )(2)

𝐶 = (𝑥3 , 𝑦3 , 𝑧3 )(3)𝑎 = (𝑥𝑎 , 𝑦𝑎 , 𝑧𝑎 ) = ((𝑥1 − 𝑥2 ), (𝑦1 − 𝑦2 ),(𝑧1 − 𝑧2 ))𝑏 = (𝑥𝑏 , 𝑦𝑏 , 𝑧𝑏 ) = ((𝑥3 − 𝑥2 ), (𝑦3 − 𝑦2 ),(𝑧3 − 𝑧2 ))(𝑥𝑎 𝑥𝑏 + 𝑦𝑎 𝑦𝑏 + 𝑧𝑎 𝑧𝑏 )−1
𝜃 = cos ()(√𝑥𝑎2 + 𝑦𝑎2 + 𝑧𝑎2 ∙ √𝑥𝑏2 + 𝑦𝑏2 + 𝑧𝑏2

(4)
(5)

(6)

Fig. 5 Vector coordinates

This angle calculation can then be used to carry out
biomechanical analysis by applying JACK calculations.
The JACK computation demands data on body posture(such as body angle), subject-specific information suchas height and weight, and the weight of the object
intended for lifting. Fig.6 represents the coordinates
involved, and Table 1 provides a detailed description of
the corresponding angles denoted by its abbreviations.
In this study, the height of the participated subject was
160 cm with a body weight of 50 kg, with different load
settings of objects, ranging from 0 kg (absence of a load)
to 2.5 kg and 5 kg. The proposed system records thecomputed results in an Excel file, facilitating
comprehensive post-analysis.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
929

Taufik Hidayat Soesilo, Praveen Nuwantha Gunaratne, Hiroki Tamura

Fig. 6 Respective vector coordinates

Load (kg)

Table 1. The coordinate details.
𝑇𝐴
𝑅𝑈𝐴
𝐿𝑈𝐴
𝑅𝐹
LF
RT
LT
RLL
LLL

Trunk anteversion angle
Right upper arm angle
Left upper arm angle
Right forearm angle
Left forearm angle
Right thigh angle
Left thigh angle
Right lower leg angle
Left lower leg angle

Lumbar Burden (N)
Max

Min

Average

0

2098

790

1655

2.5

2843

1299

2327

5

3612

1770

2974

4. Conclusion
Based on the results it can be concluded that the
proposed system can carry out biomechanical analysis todetermine the lumbar burden effectively. The procedure
was executed in real-time, also enabling monitoring of
the system within any occupational settings. Further, thesystem demonstrated a robust performance undersunlight, showing its adaptability to diverse
environmental conditions.
For future work, the observed results will undergo avalidation process using an established system to assess
the efficacy of the developed system. Furthermore, as asystem constraint, the current study is limited to thedetection of a single person. Therefore, in order to
conduct real-time analysis of lumbar burden for multiple
persons, the system requires further development.
References

Fig. 7 JACK angle for Lumbar Burden calculation
1. Wu A, March L, Zheng X, Huang J, Wang X, Zhao J, Blyth
FM, Smith E, Buchbinder R, Hoy D. Global low back pain
prevalence and years lived with disability from 1990 to
2017: estimates from the Global Burden of Disease Study
2017. Ann Trans Med 2020; 8(6): 299-313.
2. Global
HealthGroup
Data
Exchange
http://ghdx.healthdata.org/gbd-results-tool accessed Nov
20, 2023.
3. Hartvingsen J, Hancock MJ, Kongsted A, Louw Q, Ferreira
ML, Genevay S, Hoy D, Karppinen J, Glenn Pransky,
Sieper J, Smeets RJ, Underwood M. What low back pain is
and why we need to pay attention. Lancet 2015; 386: 21452191.
4. N.D. Nath, R. Akhavian, A.H. Behzadan, Ergonomic
analysis of construction worker’s body postures using
wearable mobile sensors, Appl. Ergon. 62 (2017), 107–117.
5. H. Tamura, K. Sakurai, K. Tanno, Y. Fuse. A Study on the
Lumbar Burden Evaluation of Work using One Smartphone.
Journal of Robotics, Networking and Artificial Life, Vol.
5(3); December (2018), pp. 173–179.
6. Liu, P.-L.; Chang, C.-C.; Li, L.; Xu, X. A Simple Method
to Optimally Select Upper-Limb Joint Angle Trajectories
from Two Kinect Sensors during the Twisting Task for
PostureAnalysis.
Sensors
2022,
22,
7662.
https://doi.org/10.3390/s22197662.
7. Singh, L.P., Kumar, P. & Lohan, S.K. (2023) Developmentof a real ‐ time work ‐ related postural risk assessment
system of farm workers using a sensor‐based artificial

Fig.8. Lumbar burden analysis

Table 2. Lumbar burden Max, Min, and Average©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
930

Powered by TCPDF (www.tcpdf.org)

A study on theintelligence approach. Journal of Field Robotics, 1 – 14.
https://doi.org/10.1002/rob.22215.
8. Christopher B., Oliver B., Pietro M., Tobias H., Laura J.,
Alexender M., Marco T., Verena N., Using real-time
feedback of L5/S1 compression force based on markerless
optical motion capture to improve the lifting technique inmanualmaterials
handling,
https://doi.org/10.1016/j.ergon.2022.103350.
9. Temuge B. Real time 3D body pose estimation using
MediaPipe.
Github.
https://github.com/TemugeB/bodypose3d accessed Nov 14,2022.

Authors Introduction
Mr. Taufik Hidayat Soesilo
He received his Bachelor’s degree inEngineering in 2021 from the Faculty ofEngineering, University of Brawijaya inIndonesia. He is currently a master student in
University of Miyazaki, Japan

Mr. Praveen Nuwantha Gunaratne
He received his bachelor’s degree inengineering in 2018 from the Faculty ofEngineering, University of Moratuwa, Sri
Lanka. He is currently a doctoral student in
University of Miyazaki, Japan.
Prof. Hiroki Tamura
He received the B.E. and M.E. degree from
Miyazaki University in 1998 and 2000,
respectively. From 2000 to 2001, he was an
Engineer in Asahi Kasei Corporation, Japan.
In 2001, he joined Toyama University,
Toyama, Japan, where he was a Technical
Official in the Department of Intellectual Information
Systems. In 2006, he joined Miyazaki University,
Miyazaki, Japan, where he was an Assistant Professor in
the Department of Electrical and Electronic Engineering.
Since 2015, he is currently a Professor in the Departmentof Environmental Robotics. His main research interests are
Neural Networks and Optimization Problems. In recent
years, he has had interest in Biomedical Signal Processing
using Soft Computing.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
931

Verification of Determination Possibility using Convolutional Autoencoder for Machine Tool
Abnormality Detection
Yuta Sumoto

Faculty of Engineering, University of Miyazaki, 1-1, Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
Praveen Nuwantha Gunaratne

Interdisciplinary Graduate School of Agriculture and Engineering, University of Miyazaki, 1-1, Gakuen KibanadaiNishi, Miyazaki, 889-2192, Japan
Hiroki Tamura

Faculty of Engineering, University of Miyazaki, 1-1, Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
Email: hi20024@student.miyazaki-u.ac.jp, ti20060@student.miyazaki-u.ac.jp, htamura@cc.miyazaki-u.ac.jp

Abstract
The purpose of this research is to clarify the cause of failure and to improve the accuracy of abnormality detection by
predicting the noise added to the machine tool from the results of the convolutional autoencoder. Data obtained froman acceleration sensor mounted on the machine tool are reconstructed using a convolutional autoencoder, and theaverage absolute error is calculated. The maximum value of the average absolute error of the training data is used as
the threshold value for abnormality detection. Multiple simulated data with different amplitude values based on acomposite sine wave with a specific frequency and random numbers within a specified amplitude value wereverified. In this paper, the characteristics of each type of noise and the parameters of the optimal model were examinedfrom error rate and error distribution.
Keywords: machine tools, deep learning, anomaly detection, convolutional autoencoder, noises

Initially, the machine with the attached sensor
experienced a breakdown, and there were no prospects
for its reprocessing. Subsequently, the sensor wastransferred to a similar type of machining equipment.
After the transfer, only normal data was obtained fromthe factory's machines. As abnormal data obtained before
the transfer exhibited specific frequency components,
simulated data resembling those characteristics was
generated. Parameters for multiple models were createdbased on different numbers of training data, and abnormal
detection was verified by introducing specific noise to thetraining data [3] to assess how much noise is required todetect anomalies. Moreover, considering the possibilityof identifying malfunctioning machine components bypinpointing the noise, we investigated the distribution ofreconstruction errors.

1. Introduction
In factories, machine failures can be categorized assudden breakdowns and deteriorating faults [1].
Preventing failures proactively during maintenance is
exceptionally challenging, and excessive maintenance or
machine breakdowns can result in significant losses forcompanies. Examples include production downtime
leading to manufacturing time losses, increased laborcosts due to stoppages, and the generation of defective
products. The purpose of this study is to reduce losses andenhance productivity by providing maintenance timing
suggestions and early detection of abnormalities inmachining equipment.
In collaboration with businesses, this research involved
installing acceleration sensors on machining equipmentused in semiconductor manufacturing. The data acquired
from the sensors is in the form of continuous time-series
data. Since explicit abnormal judgments are significantly
fewer than normal data, it is necessary to perform
anomaly detection without explicit labels for normal andabnormal data. Consequently, anomaly detection was
conducted using a convolutional autoencoder [2].

2. Proposed method
The proposed method consists of two steps: Learningand parameter saving, and Testing validation.
2.1. Learning and parameter saving
Initially, a convolutional autoencoder was created byreferencing the Keras tutorial [4] for parameter©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
932

Yuta Sumoto, Praveen Nuwantha Gunaratne, Hiroki Tamura(m/s²), y (m/s²), and z (m/s²). For this study, only the time(s) and x (m/s²) columns were utilized. To assess theaccuracy of each parameter, twelve CSV files were
utilized for training. The process depicted in Figure 1 wasrepeated twelve times, and the resulting parameters were
saved. The model parameters were denoted as model_1,
model_2, ..., model_n corresponding to the number oftraining data (n). With an epoch count of 5 per training, a
total of 60 training iterations were conducted as theprocess was repeated 12 times. (Fig. 3).
Analyzing the frequency components of normal andabnormal data obtained from the machine before thetransfer revealed that normal data covers a wide
frequency range, whereas abnormal data is concentrated
in the high-frequency band. The results of the FFT (Fast
Fourier Transform) applied to normal and abnormal data
are shown in Fig. 4.
Based on these results, a composite wave of sine waves
with specific frequencies was added to the initial training
data to create simulated abnormal data. The added
composite waves consisted of low frequencies at 40Hz
and 90Hz, and high frequencies at 410Hz and 460Hz.

Fig. 1. Flowchart of learning and parameter saving
inheritance (Fig. 1). The training data was standardizedusing the mean and standard deviation of the data. A
sequence with a timestep of 100 was generated using asequential model, and training was conducted. A callback
was implemented to halt training if the validation loss did
not improve for five consecutive epochs. The
hyperparameters were set as follows: epochs = 5, batch
size = 4096, and validation split = 0.1.
After training, the parameters were saved, and the model
was used to reconstruct the training data. The absolutemean error between the actual data and the reconstructeddata for each sequence was calculated, and the maximum
value was set as the threshold for testing.
2.2. Testing validation
The process involves extracting one CSV file at a timefrom the dataset. The training data is then standardizedusing the mean and standard deviation of the respective
file. Test data is reconstructed using the specified model
parameters. For each sequence, the absolute mean error
between the test data and the reconstructed data is
calculated. Instances where this error exceeds thepredefined threshold are considered anomalies. (Fig. 2)
3. ExperimentFig. 2. Flowchart of testing validation

The sampling frequency of the experimental data was
500Hz, and each file contained 600,000 data points(equivalent to 20 minutes) with columns for time (s), x©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
933

Verification of Determination Possibility

error is approximately 10% and investigate whether there
are any distinctive features in the distribution due to theadded noise.
4. Results
4.1. Parameter Comparison
The order of accuracy in detecting anomalies forabnormal data is as follows:• Low-frequency components: model_3, model_5,
model_11, model_2 (Fig. 5)• Random noise: model_3, model_11, model_12,
model_5 (Fig. 6)• High-frequency components: model_11, model_12,
model_10, model_3 (Fig. 7)

Fig. 3. Loss function

For data with added low-frequency components andrandom noise, the parameters of Model_3, trained using
three datasets, exhibited the most effective anomaly
detection. Regarding data with added high-frequency
component noise, Model_11, trained using eleven
datasets, demonstrated the most proficient anomaly
detection.

Fig. 4. Frequency analysis results of normal andabnormal data
Additionally, simulated data was generated by adding
random numbers within specified maximum andminimum values. The specified absolute values for
amplitude and random numbers ranged from 0.1 to a
maximum of 0.8.

Fig. 5. Parametercomponent)

comparison(Low-Frequency

3.1. Testing validation
Test the three types of simulated data using twelve
parameters (model_n) with varying amounts of training
data. Based on the results, examine the optimal amountof training for anomaly detection.
3.2. Testing validation
Detect the three types of simulated data using the bestparameters. Based on the results, evaluate how wellanomalies are detected when various levels of noise areadded and investigate the threshold for anomaly detection.
Fig. 6. Parameter Comparison (Random Data)

3.3. Testing validation
When the three types exhibit the same error rate,illustrate the distribution of the mean absolute errors
between the actual data and the reconstructed data foreach type. Compare the mean absolute errors when the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
934

Yuta Sumoto, Praveen Nuwantha Gunaratne, Hiroki Tamura

5. Conclusion
In the experimental data, step count, andhyperparameters employed in this study, the parametersof model_3 and model_11 demonstrated high accuracy inanomaly detection. From this, it was found that under thecurrent conditions, abnormalities could be detected withhigh accuracy if three or 11 or more pieces of normal data,
each with a total of 600,000 pieces of data, were trained.
From experiments using parameters trained three times,
it was observed that, irrespective of whether the noise isperiodic, high-frequency, or low-frequency, anomalies
were detected when noise with an amplitude value or
specified absolute value exceeding 0.3 was introduced.
While it was not possible to identify the characteristics ofthe noise from the distribution of reconstruction errors, it
is possible to estimate the amplitude of the added noisefrom the error percentages.

Fig.
7. Parameter Comparison (High-Frequency
Component)

Currently, data is being measured in the factory andsaved in specific files. Processing is set to analyze thesecond-to-last CSV file every 20 minutes. If anomalies
are detected above 10%, the number 1 is added to thearray; if below 10%, the number 0 is added. An alert istriggered when anomalies are detected continuously threetimes, and the array is output daily.

Fig. 8. Comparison of Model_3

4.2. Simulated Data Comparison
Based on the results from 4.1, a comparison of the three
types of simulated data was performed using model_3(Fig. 8). For all three types, errors reached 10% when the
additional amplitude and specified absolute values
ranged from 0.3 to 0.4. Additionally, it was observed thaterrors sharply increased for values equal to or greater than
0.4.
4.3. Noise Identification

Fig. 9. Low Frequency- Additional Amplitude 0.325

When the error is approximately 10%, the distributionof actual data and reconstructed data for the three types isshown. Additionally, when the error is approximately
10%, the amplitude of the synthesized wave for lowfrequency is 0.325, for high-frequency is 0.35, and thespecified absolute value for random data is 0.3 (Fig. 9,
Fig. 10 and Fig. 11). Differences in items such as mean,
median, standard deviation, minimum, 25th percentile,
75th percentile, and maximum, based on the obtained
basic statistical information, are on the order of onethousandth. It can be observed from the distribution thatwhen the anomaly rate is the same, there is little
difference.
Fig. 10. High Frequency- Additional Amplitude 0.35

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
935

Powered by TCPDF (www.tcpdf.org)

Verification of Determination Possibility

Authors Introduction
Mr. Yuta Sumoto
Yuta Sumoto born in 2001. He is currently
studying in department of environmental
robotics, and will receive the B.Eng from
University of Miyazaki in 2022. His
current research is Abnormality detection
using auto encoder.

Mr. Praveen Nuwantha Gunaratne
He received his Bachelor’s degree inEngineering in 2018 from the Faculty ofEngineering, University of Moratuwa, Sri
Lanka. He is currently a Doctoral student
in University of Miyazaki, Japan

Fig. 11. Random- Specified Absolute Value 0.3
References
1.

2.

3.

4.

J. Guo, Y. Miki, Y. Fujita, K. Kiritoshi, and K. Ito,
“Anomaly Detection of Seismogram withAutoencoder.” Accessed: Dec. 10, 2023. [Online].
Available:
https://www.jstage.jst.go.jp/article/pjsai/JSAI2020/0/JSAI2020_4L2GS1303/_pdf/-char/ja
S. Yan, H. Shao, Y. Xiao, B. Liu, and J. Wan, “Hybrid
robust convolutional autoencoder for unsupervised
anomaly detection of machine tools under noises,”
Robotics and Computer-Integrated Manufacturing,
vol. 79, p. 102441, Feb. 2023, doi:
https://doi.org/10.1016/j.rcim.2022.102441.
T. Mine et al., “Anomaly detection of mechanicalequipment with autoencoder,” 2016. Accessed: Dec.
10,
2023.
[Online].
Available:
https://www.jstage.jst.go.jp/article/fss/35/0/35_506/_pdf/-char/ja
K. Team, “Keras documentation: Timeseries anomaly
detection using an Autoencoder,” keras.io.
https://keras.io/examples/timeseries/timeseries_ano
maly_detection/

Prof. Hiroki Tamura
He received the B.E. and M.E. degree from
Miyazaki University in 1998 and 2000,
respectively. From 2000 to 2001, he wasan Engineer in Asahi Kasei Corporation,
Japan. In 2001, he joined Toyama
University, Toyama, Japan, where he wasa Technical Official in the Department ofIntellectual Information Systems. In 2006, he joined
Miyazaki University, Miyazaki, Japan, where he was an
Assistant Professor in the Department of Electrical andElectronic Engineering. Since 2015, he is currently aProfessor in the Department of Environmental Robotics.
His main research interests are Neural Networks andOptimization Problems. In recent years, he has had interestin Biomedical Signal Processing using Soft Computing.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
936

Basic Research for High-speed Heart Sound Determination using AI
Riku Nakashima

Faculty of Engineering, University of Miyazaki,
1-1, Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
Praveen Nuwantha Gunaratne

Interdisciplinary Graduate School of Agriculture and Engineering, University of Miyazaki, 1-1, Gakuen KibanadaiNishi, Miyazaki, 889-2192, Japan
Hiroki Tamura

Faculty of Engineering, University of Miyazaki,
1-1, Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
Email: hi20031@student.miyazaki-u.ac.jp, ti20060@student.miyazaki-u.ac.jp, htamura@cc.miyazaki-u.ac.jp

Abstract
In recent years, many researchers have focused on enhancing the current situation of home healthcare systems. In thispaper, we developed a system that uses AI to quickly determine whether auscultation sounds are normal or not. In
our analysis, heart sounds were imaged, and their abnormalities were identified using machine learning tools. The
proposed approach uses a Yolo_v7 model-based anomaly detection and the Wavelet Transform was employed toanalyze the acquired heart sound data. Proposed system resulted with an 90% accuracy rate or over on the basis of 5
seconds of heartbeat data, in various recording environments.
Keywords: Auscultation sounds, abnormality identification, YOLO_v7, Wavelet Transform

accuracy rate based on 5 seconds of heartbeat data. This
paper compares Yolo model with different durations ofphonocardiography data and elucidates strategies forachieving higher confidence scores.

1. Introduction
In today's society, the demand for home medical care is
rapidly increasing due to the aging population. However,home medical care requires a physician to visit the patientin person to make a diagnosis, which poses problems
such as securing time for physicians. A possible solution
to this problem would be to make medical equipment
readily accessible to ordinary households, enabling
individuals to personally use and evaluate theeffectiveness of the equipment. However, as proficiency
in medical knowledge is crucial for operating medical
equipment, the current situation hinders its widespreadutilization. To identify an approach suitable for thegeneral public, eliminating the necessity for a visit to aphysician or medical expertise, our team has attempted tocreate a system capable of swiftly assessing the normalityof heart sound data obtained through auscultation.

2. Methodology
Initially the Speech data, illustrating frequency featuresof heart sounds, went through wavelet transformation at
various intervals to generate images. Afterwards, thetransformed images were utilized to train multiple Yolomodels, facilitating a comparative analysis to identify thesuperior model. Here, the assessment was conductedbased on confidence scores assigned to the normal andabnormal classes. For the Yolo training data set, theimages were divided into three parts: training data, validdata, and test data. Table 1 illustrates the learning
environment used in this analysis, and the Fig. 1 depicts
the experimental procedure in brief.

In this paper, machine learning tools were employed toanalyze the heart sound data. Heart sound data fromdiverse recording environments was utilized in this paper.
The acquired heart sound data was transformed into
images through wavelet transforms, and the yolov7
model was employed for abnormality detection.
Additionally, a k-fold approach was applied to a limited
dataset to enhance accuracy. The findings revealed a 90%

3. Experiment
In the section 3.1, the ratio of training data to validation
data and test data was set at 7:2:1 in the Yolo model. The
subsequent phases, starting from section 3.2, the ratio
was adjusted to 6:2:2. The offline datasets used wereheart sound data auscultated by Sono-Support clinical©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
937

Riku Nakashima, Praveen Nuwantha Gunaratne, Hiroki Tamura

technologists (dataA – 10 healthy subjects and 40 sicksubjects), heart sound data from former laboratory
members (dataC - 23 healthy subjects), and heart sounddata of few laboratory members (jibun - 2 healthy
subjects). The data from physioNet (dataB~dataB_F),
comprising 2558 healthy subjects, 665 sick subjects wasused as online data [1].

3.1. Experiment 1
In experiment 1, multiple Yolo models (Yolo_v2,
Yolo_v5, and Yolo_v7) were employed to jointly train
normal and abnormal heart sound images and assess thesuperiority of each model.[2], [3].
3.2. Experiment 2

Table 1. Learning environment of the YOLO modelMultiple Yolo models with favorable confidence scores
were employed, and machine learning was conducted oneach of them for varying durations (3, 5, and 10 seconds),
comparing their performance in terms of the percentage
of correct responses. However, a 10-second wavelet
transform could not be generated due to insufficientseconds in imaging dataB_B. Fig. 3, 4 was generated by
performing a wavelet transform on a single speechsample, cut out for a fixed time width of 1 second
intervals and converting it to an image.

Fig. 1. Experimental Procedure
Fig. 2, 3, and 4 illustrate the images generated by the
wavelet transform used in this paper. The vertical axis
represents frequency, the horizontal axis signifies a
specific time width of the experimental data, and thecolor bar indicates amplitude. The wavelet transform is
output as an image after being processed using the Morlet
function with MATLAB. A one second shift was used toincrease the number of data by shifting the image by few
data. Due to the variation in the number of seconds inheartbeat, the number of images produced varies as well.
The k-fold method involves creating k splits of the
training data images and training them k times (k=5 in thiscase). The splits were randomly divided into 8 for the
training data and 2 for the validation data. Those with aconfidence score of less than 0.1 between normal andabnormal were excluded due to their classification fallingwithin gray zones as lacking in reliability. The formulas
for calculating the percentage of correct answers Eq. (1)and the percentage of target hits Eq. (2) are presented
below.

Fig. 2. Wavelet transform 3 sec.

Fig. 3. Wavelet transform 5 sec.

3.3. Experiment 3

𝑁𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑖𝑚𝑎𝑔𝑒𝑠
𝑃𝑒𝑟𝑐𝑒𝑛𝑡𝑎𝑔𝑒 𝑜𝑓 𝑐𝑜𝑟𝑟𝑒𝑐𝑡 𝑎𝑛𝑠𝑤𝑒𝑟𝑠 =× 100 (1)𝑇𝑜𝑡𝑎𝑙 𝑛𝑢𝑚𝑏𝑒𝑟 𝑜𝑓 𝑖𝑚𝑎𝑔𝑒𝑠

𝑃𝑒𝑟𝑐𝑒𝑛𝑡𝑎𝑔𝑒 𝑜𝑓 𝑡𝑎𝑟𝑔𝑒𝑡 ℎ𝑖𝑡𝑠 =𝑡𝑟𝑢𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒
× 100
𝑡𝑟𝑢𝑒 𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒＋𝑓𝑎𝑙𝑠𝑒 𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒

In Experiment 3, the accuracy of the 5-second wavelet
transforms, identified as the best on real data in Section
3.2. was enhanced by employing the k-fold method. Thismethod involved covering a small data set and increasing
the number of training patterns.

(2)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
938

Basic Research for High-speed

limited dataset and enhancing learning patterns throughk-fold techniques. Future research will focus on strategies
to enhance accuracy by expanding the dataset. Further,
considering the satisfactory average accuracy of the 3second wavelet transform, experiment with k-fold on the
3-second version will be continued.
Table. 3. Wavelet transform 3-second correct response
rate

Fig. 4. Wavelet transform 10 sec.
4. Results
4.1. Results of Experiment 1

Table. 4. Wavelet transform 5-second correct responserate

The correctness of machine learning responses for each
Yolo was determined based on the agreement between
the label and the test judgment. A comparison of the meanconfidence scores of the correct answers revealed thesuperiority of Yolo_v7, as illustrated in Table 2.
Table. 5. Wavelet transform 10-second correct response
rate

Table 2. Confidence score results of each YOLO
Yolov2 Yolov5 Yolov7
Confidence Score
0.957
0.973
0.997

4.2. Results of Experiment 2
Machine learning with Yolo_v7 was carried forward, as
it came out as the best among the other Yolo models
based on the results of Section 4.1. Consequently, thecorrect response rates were as follows: 3 seconds, 90.6%
for normal and 89.2% for abnormal (as shown in Table3); 5 seconds, 86.8% for normal and 90.6% for abnormal(Table 4); and in 10 seconds, 68.7% for normal and
94.4% for abnormal (Table 5). These three results
demonstrate that the 3-second cutout yields the highest
average correct response rate. However, upon comparing
with actual data (dataA), the 5-second wavelet transform
exhibits the best results. While deciding which result toadopt presents a doubt, move forwarded with k-fold using
the 5-second wavelet transform, as it yields the best
results on real data.

References
1.

2.

3.

“ Classification of Heart Sound Recordings: The
PhysioNet/Computing in Cardiology Challenge 2016v1.0.0,”physionet.org.
https://physionet.org/content/challenge-2016/1.0.0/“Yolov7”(Japanese)
Qiita,
Oct.
10,
2022.
https://qiita.com/johnrocky/items/a4c24e55acaf7dcccdd9(accessed Dec. 01, 2023).
“YOLOv5”(Japanese)
Qiita,
Oct.
27,
2021.
https://qiita.com/gotty21/items/37b52cec9542fd87bd5f(accessed Dec. 01, 2023)

Authors Introduction
4.3. Results of Experiment 3

Mr. Riku Nakashima
He was born in 2001. He is currently
studying indepartment of environmental
robotics, and will receive the B.Eng from
University of Miyazaki in 2024. His
current research is Basic Research for
High-speed Heart Sound Determination
using AI.

The outcomes obtained through k-fold training on a 5second wavelet transform revealed accuracy rates of
90.3% on normal and 93.6% on abnormal, which are over90%.
5. Conclusion
Yolo_v7 enhanced the accuracy to 90% in diverse
recording environments, achieved by optimizing a©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
939

Powered by TCPDF (www.tcpdf.org)

Riku Nakashima, Praveen Nuwantha Gunaratne, Hiroki Tamura

Mr. Praveen Nuwantha Gunaratne
He received his Bachelor’s degree inEngineering in 2018 from the Faculty ofEngineering, University of Moratuwa, Sri
Lanka. He is currently a Doctoral student
in University of Miyazaki, Japan
Prof. Hiroki Tamura
He received the B.E. and M.E. degree from
Miyazaki University in 1998 and 2000,
respectively. From 2000 to 2001, he was an
Engineer in Asahi Kasei Corporation,
Japan. In 2001, he joined Toyama
University, Toyama, Japan, where he was aTechnical Official in the Department of Intellectual
Information Systems. In 2006, he joined Miyazaki
University, Miyazaki, Japan, where he was an AssistantProfessor in the Department of Electrical and Electronic
Engineering. Since 2015, he is currently a Professor in the
Department of Environmental Robotics. His main researchinterests are Neural Networks and Optimization Problems.
In recent years, he has had interest in Biomedical Signal
Processing using Soft Computing.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
940

Prediction of High-Energy Electron Flux at Geosynchronous Orbit using a Neural Network
TechniqueAmi Iwabu

Kyushu Institute of Technology, 1-1 Sensui-cho, Tobata-ku, Kitakyushu-shi,Fukuoka, 804-8550, Japan
Kentaro Kitamura

Kyushu Institute of Technology, 1-1 Sensui-cho, Tobata-ku, Kitakyushu-shi,Fukuoka, 804-8550, Japan
E-mail: iwabu.ami593@mail.kyutech.jp, kitamura.kentaro375@mail.kyutech.jp
https://www.kyutech.ac.jp

Abstract
The radiation belt, where the high-energy particles are predominant in near earth space from the low earth orbit (LEO)
to the geostationary orbit (GEO), sometimes causes satellite malfunction. Therefore, the objective of this study is topredict the high-energy electron flux at GEO with the energy above 2 MeV after 24 hours with higher accuracy forthe safety satellite operation in terms of the space weather science. In this study the various kinds of solar wind datafrom satellite observations and ground geomagnetic observation data in 1999 were used for the Recurrent Neural
Network (RNN). Prediction results were evaluated by the prediction efficiency, which is derived from both predicted
and actual variation data. As a result, the prediction using combined data of solar wind and geomagnetic data showshighest prediction efficiency of 0.72.
Keywords: Neural network, Spacecraft, High-energy electron flux

component of solar wind magnetic field data, Bz: northsouth component of solar wind magnetic field data, E:
electron flux with energy above 2 [MeV], and AE index,
Dst index, and UT: universal standard time as groundobservation data, The same neural network was used tomake predictions. In their study, data from 1999 to 2006
were used to predict the energetic electron flux 24 hours
later by combining space-based and ground-based data,
and the prediction efficiency was 0.61. There is room tofurther improve this prediction efficiency by the numberof input data and the way they are combined.
In this study, with the aim of further improving theprediction accuracy, we use various combinations ofground-based observation data and solar wind data as
input data to predict the energetic electron flux over the
next 24 hours. The results are also evaluated using
prediction efficiency (PE).

1. Introduction
The region in space where high-energy particles areconcentrated is called the radiation belt, and the electrons
that exist there are called radiation belt electrons. The
radiation belts consist of the inner and outer belts. In
particular, the outer belt sometimes extends into
geostationary orbit, where many satellites are operated[1], [2], [3]. High-energy electrons with energies above
MeV are known to cause satellite malfunctions andfailures [4], [5].
So far, several studies have been conducted to predict
fluctuations in high-energy electron flux by using only
ground-based observation data as input parameters or
combining ground-based observation data with solar
wind data observed in space [6]. Fukada et al. [7] used
only ground-based AE index and Dst index data during
magnetic storms as input data and made predictions using
a neural network. In their study, predictions were madefrom 2 to 12 hours later, and the prediction efficiency wasas high as 0.71. However, since only the case of magnetic
storms was used as training data, it was not practicalbecause it could not predict the starting point where thehigh-energy electron flux fluctuates significantly.
Nakamura et al. [8] used seven solar wind data as input
data: Vsw: solar wind velocity, Bx: solar direction

2. Dataset
In this study, solar wind velocity (Vsw), north-south
component of the solar wind magnetic field (Bz), andhigh-energy (>2 MeV) electron flux (E), which are solar
wind data, and AE index, AU index, AL index, Dst index,
and Universal Time (UT), which are ground-based
observation data, as input parameters. AE, AU and AL
indices are used as a proxy of the Auroral activity, and©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
941

Ami Iwabu, Kentaro Kitamura

Dst index apparently represents the intensity of theMagnetic storms. We also used Akasofu ε parameter
calculated by Eq. (1) [9], which efficiently represent a
total energy transported from the solar wind into
magnetosphere. Vsw and Bz were obtained from the solar
wind probe ACE, which was acquired from Omni Web,and E was obtained from the geostationary satellite
GOES10, which was acquired from the Space Weather
Prediction Center (NOAA) of the U.S. National Oceanicand Atmospheric Administration. The AE, AU and AL
indices, and Dst index, which are ground-based
observation data, were obtained from the Kyoto
University Geomagnetic World Data Analysis Center.
In this study, one year of data in 1999 was used, of
which 80% was for training data and 20% was for test
data. The respective parameters are shown in Fig. 2-1.
Akasofu 𝜀 = 𝑉𝑠𝑤 × |𝐵|2 × 𝑓(𝑡) × (7𝑅𝑒)2
Re：Earth radius
B：Solar wind magnetic field average𝑡correlated and uncorrelated portions, indicating acomplex relationship among various parameters.

3. Recurrent Neural Network (RNN)
Neural networks are systems that mimic themechanisms of the human brain in order to perform
processes such as recognition, memory, and judgment on
a computer. There are two main types of neural networks:
supervised learning and unsupervised learning, andsupervised learning is used in this study. Supervised
learning is a learning method in which an input signal isgiven and the output signal is repeatedly compared witha teacher signal, and the coupling loadings of each neuron
are modified to reduce the error and adapt to a given
problem [10].
In this study, a recurrent neural network (RNN) was
employed to learn and predict the data. Here, RNN is aneural network that recursively repeats learning in each
unit and is suitable for use on continuous data, as shownin Fig.3-1.

(1)

4

where 𝑓(𝑡) = {sin ( )} , and the value of t is determined2
by Bz using the following equation.
𝐵
Bz > 0 𝑡 = arctan | 𝑦 |𝐵𝑧

Bz < 0𝐵𝑡 = 𝜋 − arctan | 𝑦 |𝐵𝑧

Fig.3-1 Structure of RNN

4. Results
In this study, predictions were made for various
combinations of parameters as shown in Table 4-1, andthe results were evaluated using the prediction efficiency(PE) shown in Eq. (2),𝑃𝐸 = 1 −𝑀𝑆𝐸

𝑉𝐴𝑅
𝑀𝑆𝐸 = ∑𝑁(𝑓 − 𝑥𝑖 )2
𝑁 𝑖=1 𝑖
1 𝑁
𝑉𝐴𝑅 = ∑𝑖=1(𝑥𝑖 − 𝑥̅ )2𝑁1𝑥̅ = ∑𝑁𝑥
𝑁 𝑖=1 𝑖(2)

1

Fig.2-1 Data for each parameter in 1999

where 𝑥𝑖 in each equation is the actual observed value
and 𝑓𝑖 is the predicted value.
Predictions were made five times for each epoch ineach case. The results for each case are shown in Fig.4-1
to Fig.4-5. The horizontal axis is the number of epochs,
the vertical axis is PE, and the average value of the five
predictions in each epoch is shown as a dot.

Fig.2-1 shows that the temporal variations of solar
wind parameters are basically correlated with that of
the high-energy electron flux. However, in the short
time-scales, the relationship is not necessarily a oneto-one correspondence, but rather a mixture of©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
942

Prediction of High-energy Electron

Table 4-1 Input data for each forecast
Case 1
Vsw, E, Bz, Dst index, AE index
Case 2
Vsw, E, Bz, Dst index, AE index, UT
Case 3
Vsw, E, Dst index, AEindex, Akasofu 𝜀Case 4
Vsw, E, Bz, Dst index, AU index, AL
index
Case 5
Vsw, E, Bz, Dst index, AU index,

PEPE at each epochPE

PE at each epoch

50

0.75
0.7
0.65
0.6
0.55
0.5
0.45
0.4
0.35
0.3

70

80

90

100

Fig.4-5 PE for case 5

60

70

80

90

From Fig.4-1 to Fig.4-5, it seems that there is nocorrelation between the number of epochs and prediction
efficiency between epochs 50 and 100.
Next, Table 4-2 shows the average prediction
efficiencies obtained from the results for all epochs ineach case.

100

Epoch

Fig.4-1 PE for case 1

Table 4-2 Comparison of average PE in each case

PE at each epoch

Case
Average
PE

0.75
0.7
0.65
0.6

PE

60

Epoch

50

1
0.61

2
0.51

3
0.61

4
0.57

5
0.57

0.55

Table 4-2 indicates that Case 2, in which UT (theproxy of the satellite position) was added as input data to
Case 1, shows the worst PE, while the case 1 shows thehighest PE. On the other hand, Case 3, in which Akasofu
ε (the total energy inflow from the solar wind) was added,
had the same PE as Case 1.
Case 1 with an epoch number of 70, PE shows thehighest value of 0.72. The prediction results for this caseare shown in Fig.4-6 and Fig.4-7.

0.5
0.45
0.4
0.35
0.3
50

60

70

80

90

100

90

100

Epoch

Fig.4-2 PE for case 2
PE at each epoch

PE

0.75
0.7
0.65
0.6
0.55
0.5
0.45
0.4
0.35
0.3

0.75
0.7
0.65
0.6
0.55
0.5
0.45
0.4
0.35
0.3
50

60

70

80

Epoch

Fig.4-3 PE for case 3
PE at each epoch

PE

0.75
0.7
0.65
0.6
0.55
0.5
0.45
0.4
0.35
0.3

Fig.4-6 Comparison of predicted and observed data at
the highest PE

50

60

70

80

90

100

Epoch

Fig.4-4 PE for case 4©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
943

Ami Iwabu, Kentaro Kitamura

From the prediction results, the trend of the change inhigh-energy electron flux is captured. In addition, in the
short time scale, we find that the prediction shows a little
underestimation at the time of decreasing of E, but during
E has an energy of 104 or more (alert level for the satellite
malfunction), the prediction is highly accurate. In
conclusion, the current result suggests that the RNN
method proposed in this study is able to predict theperiods considered important for the safe operation of thesatellite.
Acknowledgements
Fig.4-7 Comparison of predicted and observed data at
the highest PE

The solar wind data obtained by ACE spacecraft andthe high-energy electron flux data were provided by the
National Oceanic and Atmospheric Administration(NOAA). The Dst index, AE index AU index and AL index
were provided from the World Data Center (WDC) for
Geomagnetism, Kyoto.

In Fig.4-6 and Fig.4-7 show the comparison of thetemporal variations for both predicted and observed highenergy electron flux. The horizontal axis shows the date
and the vertical axis shows the energetic electron flux.
The blue line shows the actual observed data, and theorange line shows the predicted data. The more theorange line, the predicted data, overlaps the blue line, theobserved data, the higher the prediction efficiency andaccuracy.
Fig.4-6 shows that the forecast results are largely
consistent for large changes. Fig.4-7 shows that thepredicted values are lower than the actual observed
values in many areas where the value of E is small.

References
1.

Van Allen, J. A. G. H. Ludwing, E.C. Ray, and C. E.
McIlwain (1958), Observations of high intensity radiation
by satellites 1958 Alpha and Gamma, Jet Propulsion, 28,
588-592.
2. Van Allen, J. A., and L. A. Frank (1959), Radiation aroundthe Earth to a radial distance of 107,400 km, Nature, 183,
430.
3. Baker DN, McPherron RL, Cayton TE, et al. (1990) Linear
prediction filter analysis of relativistic electron properties
at 6.6 Rea. J Geophys Res 95:15 133-15 140.
4. Lohmeyer, W. Q., and K. Cahoy (2013), Space weather
radiation effects on geostationary satellite solid-state
power amplifiers, Space Weather, 11, 476–488,
doi:10.1002/swe.20071.
5. Baker, D. N. (2000), The occurrence of operational
anomalies in space-craft and their relationship to space
weather, IEEE Trans. Plasma Sci.,28(6), 2007–2016,
doi:10.1109/27.902228.
6. Koga S, Koshiishi H, Matsumoto H, et al. (2006)
Prediction of electron flux variations at radiation belt (inJapanese). JAXA-SP-05, p 24.
7. Fukata M, Taguchi S, Okuzawa T, and Obara T (2002)
Neural network prediction of relativistic electrons ingeosynchronous orbit during the storm recovery phase:
effects of recurring. Ann Geophys 20:947-951.
8. Nakamura Y, Kitamura K, Tokunaga M, et al. (2009)
Prediction of electron flux at Geosynchronous Obit byusing the Neural Network (in Japanese). JAXA-SP-09, p 6.
9. Akasofu, S.-I. (1981), Energy coupling between the solar
wind and the magnetosphere, Space Sci. Rev., 28, 121.
10. Yoshitomi Y (2002), Neural network (in Japanese),
Asakura shoten.

5. Discussion
In Case 2, in which UT (the positional information of
the satellite) was added as input data to Case 1, resulted
PE is lower than that in Case 1. On the other hand, PE in
Case 3 is almost same as that in Case 1, while Case 3 isadded one parameter (the Akasofu ε which is the total
number of energy inflows from the solar wind) to Case 1
same as Case 2. The difference of the added parameterscauses significant difference in consequent result. Thissuggests that the spatial information might reduce the PE,
rather than the increasing of number of input parameters.
Especially, using one parameter that include theintegrated several information can efficiently improve PE,
rather than superposing several parameters that has
individual information. On the other hand, the input
parameters of solar wind data might have potential to
improve the PE compared to that of ground data.
6. Conclusion
In each case, there is no correlation between thenumber of epochs and the PE, which means that the PE is
not dependent on the number of epochs between 50 and
100, and there is no need to set the number of epochs
above 100.
From the comparison of the average prediction
efficiency for each case, the accuracy of prediction ishigher when the parameters to be used as input data haveintegral information rather than when the number of input
data is increased.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
944

Powered by TCPDF (www.tcpdf.org)

Prediction of High-energy Electron

Authors Introduction
Ms. Ami Iwabu
She received her Bachelor’s degree inEngineering in 2023 from the Faculty ofEngineering,
KyushuInstituteoftechnology in Japan. She is currently amaster student in Kyushu Institute ofTechnology, Japan

Dr. Kentaro Kitamura
He is a Professor of Kyushu Institute ofTechnology, and director of Laboratory of
Lean Satellite Enterprises and In-orbit
Experiments (LaSEINE). He received his
BS and MS degrees in science from
Kyushu University, in 1996 and 1998,
respectively. Then he received his Ph.D.
degrees in science (Earth and Planetary Science) from
Kyushu University in 2001.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
945

Image Gradient-based Monocular Visual-Inertial Odometry
Tae Ihn Kim

Robotics Lab, Hyundai Motor Company,
Uiwang-si, Gyeonggi-do, 16082, Republic of Korea
Jae Hyung Jung

Dept. of Aerospace Engineering, Seoul National University / ASRI
Seoul, 08826, Republic of Korea
Chan Gook Park

Dept. of Aerospace Engineering, Seoul National University / ASRI
Seoul, 08826, Republic of Korea
E-mail: natigerya@snu.ac.kr, lastflowers@snu.ac.kr, chanpark@snu.ac.kr

Abstract
This paper presents image gradient-based monocular visual-inertial odometry (VIO) algorithm, using image gradientmeasurements, robust to illumination change. We expand the measurements from the reprojected feature locations on
the image coordinates to the corresponding image gradients. The iterated EKF and low-pass pyramid are adapted toreduce the linearization error in the multi-state constraint Kalman filter (MSCKF) measurement update process. We
verify that our proposed algorithm outperforms both conventional indirect and direct MSCKF-based VIO algorithms
by evaluating the pose estimation performance.
Keywords: Illumination Change, Image Gradient, Iterated Extended Kalman Filter, Low-pass Pyramid, VisualInertial Odometry

1. Introduction

we apply iterated extended Kalman filter (EKF) and lowpass pyramid to minimize linearization error. Lastly, weevaluate our proposed algorithm at both simulation andreal-world datasets and demonstrate that our proposed
algorithm outperforms both conventional indirect anddirect methods in the environment where illumination
changes.

An accurate ego-motion estimation of a vehicle in a
global navigation satellite system (GNSS) denied
environment has been a challenging task in robotics andautonomous driving for several decades [1]. To tackle
this issue, the ego-motion estimation algorithm using
camera information, so-called Visual Odometry (VO),has been studied [2], [3], [4]. VO incrementally estimatesa relative 6-DOF pose between consecutive images.
The most representative fusion of camera and alternative
sensors is visual-inertial odometry (VIO), which
additionally uses IMU; accelerometers, and gyroscopes.
Despite various VIO algorithm studies, several
challenging environments exist for ego-motion
estimation, still being tackled. One representativeenvironment is where the photometric consistency
assumption is violated due to the illumination change.
The main contribution of this paper is threefold. First,
inspired by the studies on image gradient-based VO
algorithms [5]. we adapted the novel measurement,
image gradient, to the VIO framework. Image gradient is
known to be robust to illumination change. We utilize
image gradients in the horizontal and vertical image
directions as measurements of the multi-state constraint
Kalman filter (MSCKF)-based VIO algorithm. Second,
unlike the conventional MSCKF-based VIO algorithm,

2. Preliminaries
In this paper, the 3D pose is described at the current IMUaffixed frame, {𝐵𝑘 }, at time 𝑘 with respect to a global
frame, {𝐺}. The global frame is the initial body framealigned with gravity. Unlike EKF, MSCKF considers
multiple images and utilizes the geometric constraint toestimate the ego-motion of the vehicle. Therefore,
MSCKF includes the previous camera poses in the state.
{𝐶𝑖 } is a camera frame of the 𝑖th element in the sliding
window of MSCKF.
The mathematical expression is described as follows. For
𝐶
example, 𝑝𝐶𝑙𝑙𝑓𝑖 is the position of camera frame describedas the translation from the 𝑙th camera frame, 𝐶𝑙 to the 𝑖th
3D feature position, 𝑓𝑖 with respect to 𝐶𝑙 .

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
946

Tae Ihn Kim, Jae Hyung Jung, Chan Gook Park

3.

State Representation

image are utilized, the residual is expanded to thedifference between the anchor and the 𝑙th image gradientalong two directions as,

The MSCKF state, 𝑋𝑀𝑆𝐶𝐾𝐹 is formulated as a
combination vector of the IMU state, 𝑋𝐼𝑀𝑈 , and theprevious camera state, 𝑋𝑐𝑎𝑚 , in the sliding window.
(1)

(9)

(2)
𝐶
𝐺
𝐺
where 𝑝̂𝐶𝑙𝑙𝑓𝑖 = ℎ(𝑝̂𝐺𝑓, 𝜃̂𝐶𝐺𝑙 , 𝑝̂𝐺𝐶
) . 𝛻𝑥 𝐼𝑖,𝐴 and 𝛻𝑦 𝐼𝑖,𝐴 are𝑙𝑖
the image gradients along the horizontal and verticaldirection of the anchor image. The anchor frame isdefined as the camera image where the 𝑖 th feature iscaptured in the sliding window. 𝑟𝑖,𝑙 , 𝑧𝑖,𝑙 and 𝑧̂𝑖,𝑙 are theresiduals, measured gradients and estimated gradients of
the 𝑖th feature on the 𝑙th image, respectively. HX𝑖,𝑙 and
Hf𝑖 are state and feature position Jacobian matrix,𝐺respectively. 𝑋̃𝑘 is the error state at step 𝑘 and 𝑝̃𝐺𝑓
is the𝑖error feature position.
The residuals of the 𝑖th feature on all image frames areaccumulated to form a block vector of residuals of the 𝑖thfeature. Since the measurement is extended from theimage feature projection location on the image
coordinates to image gradients, Jacobian matrices of state,
𝐻𝑋𝐶 , and feature position, 𝐻𝑝𝐺 , are newly derived as(3)
𝑞𝐺𝐵 is the unit quaternion describing the rotation frombody frame {𝐵} to the global frame {𝐺} (or the body𝐺𝐺
attitude with respect to {𝐺}, 𝑝𝐺𝐵
and 𝑣𝐺𝐵are the body
position and velocity with respect to {𝐺}, respectively,
𝑏𝑎 and 𝑏𝑔 are accelerometer and gyroscope bias,
𝐺
respectively, and 𝑞𝐺𝐶𝑙 and 𝑝𝐺𝐶
are the 𝑙 th camera𝑙attitude and position, respectively.
True position, velocity, accelerometer and gyroscope bias
are expressed as 𝑋 = 𝑋̂ + 𝑋̃ , where 𝑋̂ and 𝑋̃ areestimated and error state, respectively, and true attitude
as 𝑞 = 𝛿𝑞 ⊗ 𝑞̂ , where ⊗ denotes quaternion
multiplication. Then, the quaternion error is derived asfollows.
(4)

𝑙

𝐺𝑓𝑖

below.

Therefore, the error state is defined as below.
(5)
(6)

(10)

(7)
4. Measurement Model𝛻𝑥 and 𝛻𝑦 are gradients along horizontal and verticaldirection of the image, respectively. u𝑑 and v𝑑 are
undistorted image pixels along horizontal and verticaldirection of image, respectively. 𝑋𝐶𝑙 is the 𝑙 th camera

In our proposed algorithm, the measurement of theconventional MSCKF, which is the reprojected feature
location on the image coordinate, is extended to theimage gradients along the horizontal and verticaldirection of the image, 𝑧𝑔𝑟𝑎𝑑𝑖𝑒𝑛𝑡 ,pose.

𝛻 2 𝐼𝑙𝜕(𝑢𝑑 ,𝑣𝑑 )implies the gradients of the gradient image.

Then, the residual is projected to a left null-space toeliminate the terms related to the 3D feature position. The
block vectors of the residuals of the 𝑖 th feature are
accumulated to form a block vector of all residuals andproceed QR decomposition to reduce computational
complexity.
Unlike EKF, which utilizes prior estimate as alinearization point, iterated EKF iteratively computes
posterior estimate, which is more accurate since themeasurement is reflected. The MSCKF measurement
update equation is summarized as follows.

(8)
𝐶

𝐺
𝐺
where 𝑝𝐶𝑙𝑙𝑓𝑖 = ℎ(𝑝𝐺𝑓, 𝜃𝐶𝐺𝑙 , 𝑝𝐺𝐶) . 𝑧𝑔𝑟𝑎𝑑𝑖𝑒𝑛𝑡,𝑖,𝑙 is the𝑙𝑖measurement of the 𝑖th feature on the 𝑙th image. 𝜋(∙) is a
reprojected feature location, 𝐼𝑖,𝑙 (𝜋(∙)) is an intensity at
the reprojected feature location, 𝛻𝑥 𝐼𝑖,𝑙 (𝜋(∙)) and𝛻𝑦 𝐼𝑖,𝑙 (𝜋(∙)) are gradients along horizontal and verticaldirections of image. 𝑛𝑔𝑟𝑎𝑑𝑖𝑒𝑛𝑡,𝑖,𝑙 is the noise of the 𝑖thfeature on the 𝑙th image.

5. Measurement Update(11)

The measurement update of our proposed algorithm
proceeds when either a feature is lost from tracking or thelength of the feature track exceeds the size of the sliding
window, which are the same measurement update
conditions of the conventional MSCKF-based VIO
algorithm [6], [7]. Since the novel measurement, image
gradients along the horizontal and vertical direction of the𝛿𝑋𝑘+ is a posterior estimated state, which is the correction
term with respect to the error state from 𝑋̂𝑘+ = 𝑋̂𝑘− ⊞𝛿𝑋𝑘+ . ⊞ is the operator defining the error state as theestimated error state and the correction term. ℎ(𝑋̂𝑘− ) is
the measurement equation of the system, 𝐾𝑘 is the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
947

Image Gradient-based Monocular Visual-Inertial

Kalman gain, and 𝑃𝑘− is the prior covariance of theestimated error. 𝑇𝐻𝑘 is the upper triangular matrix
computed from the QR decomposition. 𝑄1𝑘 is the unitarymatrix computed from the QR decomposition, whose
columns form bases for the range of 𝐻𝑋 . 𝑦𝑘 is themeasurement, and 𝑅0𝑘 is the covariance matrix of themeasurement.
The EKF-based MSCKF uses the prior estimate state as
the linearization point. Therefore, the measurement is notreflected on the linearization point. However, the iterated
EKF-based MSCKF first computes the posterior estimate
state using the prior estimate state to reflect themeasurement on the linearization point and iteratively
computes the posterior estimate state to reduce thelinearization point. Therefore, the adaptation of theiterated EKF to MSCKF is made through modification ofequation 11 tosequences collected in the same indoor environment, andMAV travels 94.86 m in 134.8 s on average.
The experiment result is summarized in Fig. 1. It reportsthat the average 3D position RMSE of the feature-based
method is less than that of the intensity-based method.
This result implies that the illumination change strongly
influences the image intensity, while the reprojectedfeature position is less influenced. The feature-based
method is less influenced because the extracted feature
position's accuracy depends on motion-blur, not theillumination change. Average 3D position RMSE of theintensity-based method is reduced by utilizing the image
gradient
7. Conclusion
In this paper, we have proposed the utilization of image
gradients as measurements in the monocular MSCKFbased VIO algorithm with an adaptation of iterated EKF
with the low-pass pyramid. The novel measurement isused to tackle the illumination change environment,
where the photometric-consistency assumption isviolated. Unlike the conventional method, which utilizes
the image feature coordinates, our proposed algorithm
utilizes the image gradients along the horizontal andvertical direction of the image with the adaptation of
iterated EKF and low-pass pyramid. The performance ofthe proposed algorithm is analyzed by comparing it tothat of feature-based and intensity-based methods at thereal-world dataset. Our experimental result reports thatour proposed algorithm's accuracy of both 3D attitude
and position estimation outperforms both feature-based
and intensity-based algorithms.

(12)+
𝛿𝑋𝑘,𝑖+1
is the correction term at the 𝑖 + 1th iteration and+ℎ(𝑋̂𝑘,𝑖) is the measurement equation of system, where
posterior is reflected. 𝐾𝑘,𝑖 , 𝑇𝐻𝑘,𝑖 , and 𝑄1𝑘,𝑖 are the Kalman
gain, the upper triangular matrix, and the unitary matrix,
respectively, at the 𝑖th iteration. The combination of theiterated EKF and the low-pass pyramid results in a
reduction of the residual, which implies the improvementof linearization error due to the high non-linearity of theimage gradients.

Acknowledgements
This research was supported by the National Research
Foundation of Korea (NRF) funded by the Ministry of
Science and ICT, the Republic of Korea.
(NRF-2022R1A2C2012166)
References
1. M. Tang, Z. Chen, “Robot Tracking in SLAM with
Masreliez-Martin Unscented Kalman Filter”, International
Journal of Control, Automation and Systems, vol. 18, no.
9, pp. 2315-2325, 2020.
2. F. Zhang, D. Clarke and A. Knoll, “Visual Odometry basedon a Bernoulli Filter”, International Journal of Control,
Automation and Systems, vol. 13, no. 3, pp. 530-538, 2015.
3. W. Jeong, J. Moon, and B. Lee, "Error improvement invisual odometry using super-resolution", International
Journal of Control, Automation and Systems, vol. 18, no.
2, pp. 322-329, 2020.
4. L. Wang, S. Xia, H. Xi, S. Li, and L. Wang, "Robust Visual
Inertial Odometry Estimation Based on Adaptive
Interactive Multiple Model Algorithm", International
Journal of Control, Automation and Systems, vol. 20, no.
10, pp. 3335-3346, 2022.
5. J. Quenzel, R. A. Rosu, T. Labe, C. Stachniss and S.
Behnke, “Beyond Photometric Consistency: Gradientbased Dissimilarity for Improving Visual Odometry andStereo Matching”, 2020 IEEE International Conference on
Robotics and Automation (ICRA), pp. 272-278, 2020.

Fig. 1. Attitude and position RMSE

6. Experimental Results
The proposed algorithm is evaluated at a real-world
dataset to verify that it is more robust to the illumination
change than the feature-based and intensity-based
methods. EuRoC real-world dataset is collected from an
on-board hex-rotor helicopter. This dataset provides
stereo images, IMU data, and 6-DOF ground-truth.
Illumination change due to the camera's automatic
exposure time is included in the image. The performanceof our proposed algorithm is analyzed on five different©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
948

Powered by TCPDF (www.tcpdf.org)

Tae Ihn Kim, Jae Hyung Jung, Chan Gook Park

6. M. Li and A. I. Mourikis, “High-precision, Consistent
EKF-based Visual-Inertial Odometry”, The International
Journal of Robotics Research, vol. 32, no. 6, pp. 690-711,
May 2013.
7. A. I. Mourikis and S. I. Roumeliotis, “A Multi-State
Constraint Kalman Filter for Vision-aided Inertial
Navigation”, 2007 IEEE International Conference on
Robotics and Automation (ICRA), pp. 3565-3572, 2007.

Authors Introduction
Mr. Tae Ihn Kim
He received the M.S. degree in the
Department of Aerospace Engineeringfrom Seoul National University,
Republic of Korea, in 2021. Hereceived the B.S. degree in Mechanicaland Aerospace Engineering from Seoul
National University, Republic ofKorea, in 2019. His research interests
include Visual Odometry and Visual-Inertial Odometry.
Mr. Jae Hyung Jung
He received the M.S., and Ph.D.
degrees in the Department of
Aerospace Engineering from Seoul
National University, Republic ofKorea. He received the B.S. degree intheDepartment
ofAerospace
Engineering from Pusan National
University, Republic of Korea, in 2017. His researchinterests include visual odometry and vision-aided inertial
navigation for mobile robots.
Dr. Chan Gook ParkHe received the B.S., M.S., and Ph.D.
degrees in control and instrumentationengineering from the Seoul National
University, Seoul, South Korea, in
1985, 1987, and 1993, respectively. He
was as a Post-Doctoral Fellow with
Prof. J. L. Speyer about peak seeking
control for formation flight at the
University of California at Los
Angeles, Los Angeles, CA, USA, in 1998. From 1994 to
2003, he was an Associate Professor with Kwangwoon
University, Seoul. In 2003, he joined the Faculty with the
School of Mechanical and Aerospace Engineering,
National University, Seoul, where he is currently aprofessor. In 2009, He was a Visiting Scholar with the
Department of Aerospace Engineering, Georgia Instituteof Technology, Atlanta, GA, USA. He served as a chair ofIEEE AES Korea Chapter until 2009. His current research
topics include advanced filtering techniques, highprecision inertial navigation system (INS), visual-inertial
odometry (VIO), INS/GNSS/IBN integration, andsmartphone-based/foot-mounted
pedestriandead
reckoning (PDR) systems.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
949

A Method for Embedding Multiple Photographic Images in a Photographic Image
Naoki Kouno

Department of Information Systems, University of Nagasaki, Japan
Kanya Goto

Graduate School of Regional Design and Creation, University of Nagasaki, Japan
Toru Hiraoka

Department of Information Systems, University of Nagasaki, Japan
E-mail: hiraoka@sun.ac.jp

Abstract
A method has been proposed for embedding another photographic image (image B) in a photographic image (image
A). An image (image A') is generated by embedding information of image B in image A, and then an image (imageB') is restored by extracting information from image A'. The conventional methods can only embed one photographic
image in one photographic image. Therefore, we extend the conventional method and propose a method forembedding multiple photographic images in one photographic image. In our method, as more images B are embedding
in image A, the image quality of image A' deteriorates, but the image quality of images B' dose not deteriorate. To
verify the effectiveness of our method, experiments using various photographic images were performed. As a resultof the experiments, the relationship between the number of images B and the image quality of image A' was clarified.
Keywords: Multiple photographic images, Embedding, Restoring, Digital watermark

larger difference between images B and B' than theconventional method [2]. In the conventional methods [2],[3], it is important not to be aware that image B isembedded in image A', so we focus on the conventional
method [3].

1. Introduction
Digital watermarking [1] methods have been proposedthat uses two photographic images of the same size with
256 gradations and embed another photographic image in
one of the photographic images [2], [3]. The conventional
methods [2], [3] embed one photographic image (imageB) in one photographic image (image A). An image(image A') is generated by embedding information ofimage B in image A, and then an image (image B') is
restored by extracting information from image A'. Images
A and A' are similar, and images B and B' are similar. To
make people unaware that image B is embedded in image
A', the difference between images A and A' must be
visually unrecognizable. In the conventional method [2],each pixel value of pixels in image B whose horizontal
and vertical positions are odd numbers is embedded in 4
pixels in image A. At that time, the pixel values in image
A are changed within ±2, and then image A' is generated.
In the conventional method [3], each pixel value of pixels
in image B whose horizontal and vertical positions are thevalues of the tolerance sequence with first term 2 andtolerance 3 is embedded in 8 pixels in image A. At thattime, the pixel values in image A are changed within ±1,
and then image A' is generated. It is known throughexperiments that the conventional method [3] reduces thedifference between images A and A' about 60%
compared to the conventional method [2]. On the other
hand, the conventional method [3] is known to have a

Since the conventional method [3] can only embed one
photographic image in one photographic image, we
extend the conventional method [3] and propose amethod for embedding multiple photographic images in
one photographic image. In our method, as more images
B are embedding in image A, the image quality of image
A' deteriorates, but the image quality of images B' dose
not deteriorate. To verify the effectiveness of our method,experiments using various photographic images were
performed. As a result of the experiments, therelationship between the number of images B and theimage quality of image A' was clarified.
2. Our Method
Our method generates image A' by embedding images
B𝑜 (𝑜 = 1,2, ⋯ , 𝑂) in image A, where 𝑂 is the number ofimages B𝑜 to be embedded in image A, and then images
B′𝑜 are restored from image A'. A conceptual diagram ofour method is shown in Fig. 1. The pixel values for spatial
coordinates (𝑖, 𝑗) (𝑖 = 1,2, ⋯ , 𝐼; 𝑗 = 1,2, ⋯ , 𝐽) of images
A, B, A' and B' are defined as 𝑓A,𝑖,𝑗 , 𝑓B𝑜,𝑖,𝑗 , 𝑓A′,𝑖,𝑗 and𝑓B′𝑜,𝑖,𝑗 , respectively. Images A, B𝑜 , A' and B′𝑜 are the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
950

Naoki Kouno, Kanya Goto, Toru Hiraoka

When embedding images B𝑜 in image A, our methodchanges the pixel values of image A within plus or minus
2𝑂 − 1, and then generates image A'. As more images B𝑜
are embedding in image A, the image quality of image A'
deteriorates.

same size and have 256 gradations from 0 to 255. The
methods for embedding images B𝑜 in image A andrestoring images B′𝑜 from image A' are described below.

2.2. Restoring method
Let the binary representations of 𝑓A′,𝑖,𝑗 %2𝑂 be 𝑏A′,𝑜,𝑖,𝑗 ,and the following relationship (Eq. (5)) is established.
Fig. 1. Conceptual diagram of our method
𝑓A′,𝑖,𝑗 %2𝑂 = ∑𝑂𝑜=1 2𝑂−𝑜 𝑏A′,𝑜,𝑖,𝑗

2.1. Embedding method

The pixel values𝑓B′𝑜,𝑘,𝑙 for spatial coordinates (𝑘, 𝑙) arerestored by the following Eq. (6).

Let the binary representations of 𝑓B𝑜,𝑘,𝑙 be 𝑏B𝑜 ,𝑝,𝑘,𝑙 (𝑝 =1,2, ⋯ ,8 ), and the following relationship (Eq. (1)) isestablished, where 𝑘 (= 2,5,8, ⋯) and 𝑙 (= 2,5,8, ⋯) are
the values of the tolerance sequence with first term 2 andtolerance 3.
𝑓B𝑜 ,𝑘,𝑙 = ∑8𝑝=1 28−𝑝 𝑏B𝑜 ,𝑝,𝑘,𝑙(6)𝑓B′𝑜,𝑘,𝑙 = 128𝑏A′,o,𝑘−1,𝑙−1 + 64𝑏A′,o,𝑘,𝑙−1 +32𝑏A′,o,𝑘+1,𝑙−1 + 16𝑏A′,o,𝑘−1,𝑙 + 8𝑏A′,o,𝑘+1,𝑙 +4𝑏A′,o,𝑘−1,𝑙+1 + 2𝑏A′,o,𝑘,𝑙+1 + 𝑏A′,o,𝑘+1,𝑙+1(1)

The pixel values 𝑓B′𝑜,𝑘′,𝑙′ for spatial coordinates (𝑘′, 𝑙′)other than (𝑘, 𝑙) are restored by the following Eq. (7) andEq. (8).

Information is respectively embedded in 𝑓A,𝑘−1,𝑙−1 ,𝑓A,𝑘,𝑙−1 , 𝑓A,𝑘+1,𝑙−1 , 𝑓A,𝑘−1,𝑙 , 𝑓A,𝑘+1,𝑙 , 𝑓A,𝑘−1,𝑙+1 , 𝑓A,𝑘,𝑙+1
and 𝑓A,𝑘+1,𝑙+1 using the values 𝑏B𝑜 ,1,𝑘,𝑙 , 𝑏B𝑜 ,2,𝑘,𝑙 , 𝑏B𝑜 ,3,𝑘,𝑙 ,𝑏B𝑜 ,4,𝑘,𝑙 , 𝑏B𝑜,5,𝑘,𝑙 , 𝑏B𝑜 ,6,𝑘,𝑙 , 𝑏B𝑜 ,7,𝑘,𝑙 and 𝑏B𝑜,8,𝑘,𝑙 , and then𝑓A′,𝑘−1,𝑙−1 , 𝑓A′,𝑘,𝑙−1 , 𝑓A′,𝑘+1,𝑙−1 , 𝑓A′,𝑘−1,𝑙 , 𝑓A′,𝑘+1,𝑙 ,𝑓A′,𝑘−1,𝑙+1 , 𝑓A′,𝑘,𝑙+1 and 𝑓A′,𝑘+1,𝑙+1 are generated. Notethat 𝑓A′,𝑘,𝑙 is the same value as 𝑓A,𝑘,𝑙 . Since 𝑓A′,𝑘−1,𝑙−1 ,𝑓A′,𝑘,𝑙−1 , 𝑓A′,𝑘+1,𝑙−1 , 𝑓A′,𝑘−1,𝑙 , 𝑓A′,𝑘+1,𝑙 , 𝑓A′,𝑘−1,𝑙+1 ,𝑓A′,𝑘,𝑙+1 and 𝑓A′,𝑘+1,𝑙+1 are calculated in the sameprocedure, only the case of 𝑓A′,𝑘−1,𝑙−1 is described below.
The values 𝑐A,𝑘−1,𝑙−1 , which are integers greater than orequal to 0 and less than 2𝑂 , are calculated from the pixel
values 𝑓A,𝑘−1,𝑙−1 by the following Eq. (2).
𝑐A,𝑘−1,𝑙−1 = 𝑓A,𝑘−1,𝑙−1 % 2𝑂

𝑓B′𝑜,𝑘′,𝑙′ =𝑑𝑚,𝑛 =2∑2
𝑚=−2 ∑𝑛=−2 𝑓B′𝑜 ,𝑘′+𝑚,𝑙′+𝑛 𝑑𝑚,𝑛𝑑𝑚,𝑛1

√𝑚2 +𝑛2(7)(8)

where 𝑚 and 𝑛 are the relative positions from the target
pixel. Note that Eq. (7) is calculated using only the pixel
values 𝑓B′𝑜,𝑘,𝑙 obtained in Eq. (6).
3. Experiments(2)

In this experiment, Lena image shown in Fig. 2 wasused as image A, and 4 photographic images shown in
Fig. 3 were used as images B𝑜 . The size of all image A
and images B𝑜 was 256 * 256 pixels. Visual andquantitative evaluations were performed to verify theeffectiveness of our method.

where % represents a remainder operation. The values
𝑐B,𝑘−1,𝑙−1 , which are integers greater than or equal to 0
and less than 2𝑂 , are calculated from the values 𝑏B𝑜 ,1,𝑘,𝑙by the following Eq. (3).
𝑐B,𝑘−1,𝑙−1 = ∑𝑂𝑜=1 2𝑂−𝑜 𝑏B𝑜 ,1,𝑘,𝑙

(5)

(3)

The values 𝑓A′,𝑘−1,𝑙−1 are calculated by the following Eq.
(4).
Fig. 2. Lena image (image A)
𝑓A′,𝑘−1,𝑙−1 = 𝑓A,𝑘−1,𝑙−1 − 𝑐A,𝑘−1,𝑙−1 + 𝑐B,𝑘−1,𝑙−1(4)

If 𝑓A′,𝑘−1,𝑙−1 is smaller than 0, we must add 2𝑂 to𝑓A′,𝑘−1,𝑙−1 . If 𝑓A′,𝑘−1,𝑙−1 is greater than 255, we must
subtract 2𝑂 from 𝑓A′,𝑘−1,𝑙−1 .
(a) Airplane (b) Barbara(c) Boat (d)Cameraman
Fig. 3. Various photographic images (images B𝑜 )©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
951

A Method for Embeddingin which one or 3 images B𝑜 were embedded, wereslightly recognizable as different from image A, andimage A', in which one or 4 images B𝑜 were embedded,
were clearly recognizable as different from image A.

3.1. Evaluation of embedded images
Averages of absolute values of the differences between
the pixel values of image A and images A' were
calculated. Hereinafter, the averages are referred to as
difference averages. The difference averages for 4 casesof embedding one image B𝑜 are shown in Table 1. The
difference averages for 6 cases of embedding 2 images
B𝑜 are shown in Table 2. For example, "(a)+(b)" in Table
1 means that 2 images B𝑜 (a) and (b) of Fig. 3 areembedded. The difference averages for 4 cases of
embedding 3 images B𝑜 are shown in Table 3. The
difference average for one case of embedding 4 images
B𝑜 is shown in Table 4. From Table 1 to 4, it was foundthat the larger the number of images B𝑜 to be embedded
in image A, the larger the difference averFiage.

(a)
(b)
(c)
(d)
Fig. 4. Images A' for 4 cases of embedding one image B𝑜Table 1. Difference averages between image A andimages A' for 4 cases of embedding one image B𝑜Difference averages
Symbols of images B𝑜
(a)
0.443
(b)
0.442
(c)
0.443
(d)
0.442

(a) + (b)(a) + (c)(a) + (d)(b) + (c)(b) + (d)(c) + (d)Fig. 5. Images A' for 6 cases of embedding 2 images B𝑜Table 2. Difference averages between image A andimages A' for 6 cases of embedding 2 images B𝑜Difference averages
Symbols of images B𝑜(a) + (b)1.093(a) + (c)1.100(a) + (d)1.092(b) + (c)1.099(b) + (d)1.096(c) + (d)1.127(a) + (b) + (c) (a) + (b) + (d) (a) + (c) + (d) (b) + (c) + (d)Fig. 6. Images A' for 4 cases of embedding 3 images B𝑜Table 3. Difference averages between image A andimages A' for 4 cases of embedding 3 images B𝑜Difference averages
Symbols of images B𝑜(a) + (b) + (c)2.302(a) + (b) + (d)2.291(a) + (c) + (d)2.320(b) + (c) + (d)2.316(a) + (b) + (c) + (d)Fig. 7. Images A' for one case of embedding 4 images B𝑜
3.2. Evaluation of restored images

Table 4. Difference averages between image A andimages A' for one case of embedding 4 images B𝑜Difference averages
Symbols of images B𝑜(a) + (b) + (c) + (d)4.711

Difference averages between images B𝑜 and images
B′𝑜 restored from images A' were calculated. The
difference averages in this case are the same value nomatter how many images B𝑜 are embedded. The
difference averages are shown in Table 5. Additionally,
images B′𝑜 are shown in Fig. 8. Images B′𝑜 do notchange no matter how many images B𝑜 are embedded.
From Table 5, it was found that the difference averages
were somewhat larger, ranging from 6 to 13 values.
However, from Fig. 3 and 8, it was found that images B′𝑜
are fully recognizable as same as images B𝑜 , althoughimages B′𝑜 were disturbed at the edges.

Next, the differences between image A and images A'
were visually evaluated. Images A' for 4 cases of
embedding one image B𝑜 are shown in Fig. 4, Images A'
for 6 cases of embedding 2 images B𝑜 are shown in Fig.
5. Images A' for 4 cases of embedding 3 images B𝑜 areshown in Fig. 6. Image A' for one case of embedding 4
images B𝑜 is shown in Fig. 7. From Fig. 2 and 4 to 7, it
was found that images A', in which one and 2 images B𝑜were embedded, were visually unrecognizable asdifferent from image A. It was also found that images A',©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
952

Powered by TCPDF (www.tcpdf.org)

Naoki Kouno, Kanya Goto, Toru Hiraoka

Table 5. Difference averages between images B𝑜 andimages B′𝑜Difference averages
Symbols of images B𝑜
(a)
8.265
(b)
12.717
(c)
6.642
(d)
8.191

3.

T. Hiraoka and K. Goto, A method to reduce changes inphotographic images with embedded images, ICIC
Express Letter.

Authors Introduction
Mr. Naoki Kouno
He is a student in the Department ofInformation Systems, University ofNagasaki. His research interests inculede
image processing and data analysis.

Fig. 8. Images B′𝑜
Mr. Kanya Goto

4. Conclusion

He received B.Eco. degree from Keio
University in 2006. He is currently a
CEO in Movenext Corporation. His
research
interests
includenonphotorealistic rendering and electronic
watermark.

We proposed a method for embedding multiple
photographic images (images B𝑜 ) in one photographic
image (image A). An image (image A') was generated byembedding information of images B𝑜 in image A, andthen an image (images B′𝑜 ) was restored by extracting
information from image A'. In our method, as more
images B were embedding in image A, the image qualityof image A' deteriorated, but the image quality of images
images B′𝑜 did not deteriorate. To verify theeffectiveness of our method, experiments using various
photographic images were performed. As a result of theexperiments, it was found that images A', in which one
and 2 images B𝑜 were embedded, were visuallyunrecognizable as different from image A. Additionally,
it was found that images A', in which one or 3 images B𝑜were embedded, were slightly recognizable as different
from image A, and image A', in which one or 4 images
B𝑜 were embedded, were clearly recognizable asdifferent from image A.

Dr. Toru Hiraoka
He received B.Des., M.Des. and D.Eng.
degrees from Kyushu Institute of Designin 1995, 1997 and 2005, respectively. He
is currently a Professor in University ofNagasaki. His research interests include
non-photorealistic rendering and disasterprevention.

A future task is to be able to apply our method tophotographic images of different sizes.
Acknowledgements
This work was supported by JSPS KAKENHI Grant
Number JP23K11727 and The Telecommunications
Advancement Foundation Grant.
References
1.

2.

A. C. Pavan and M. T. Somashekara, An overview on
research trends, challenges, applications and future
direction in digital image watermarking, International
Research Journal on Advanced Science Hub 5(1), 2023, pp.
2582-4376.
T. Hiraoka, A method for embedding another photographic
image in a photographic image, ICIC Express Letter 14(4),2020, pp. 311-317.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
953

Generation of Flowing-Line Images Using Vertical and Horizontal Smoothing Filters
Karin Kuroki

Department of Information Systems, University of Nagasaki, Japan
Toru Hiraoka

Department of Information Systems, University of Nagasaki, Japan
E-mail: hiraoka@sun.ac.jp

Abstract
We propose a non-photorealistic rendering method for automatically generating flowing-line images fromphotographic images. Flowing-line images consist of unidirectional lines with smooth curves. Our method is executedby an iterative calculation using vertical and horizontal smoothing filters. To verify the effectiveness of our method,
we conducted an experiment using various photographic images to confirm that flowing-line patterns can be generated
on the entire image. Additionally, we conducted an experiment to visually examine how flowing-line patterns
generated by changing the values of the parameters in our method change.
Keywords: Non-photorealistic rendering, Flowing line, Vertical smoothing filter, Horizontal smoothing filter,
Automatic generation

similar to flowing-line images, non-photorealistic
rendering methods for generating ripple images has been
proposed [6],[7]. Ripple images are composed ofcontinuous lines with fluctuations and are generatedusing inverse Sobel filter [6] or intensity gradients [7].
Flowing-line images are represented by the lines closer tothe binary value of black and white than ripple images,
and flowing-line images and ripple images have different
impressions. The conventional method [8] is executed byan iterative calculation using circular-sector-typesmoothing filter and inverse filter [9]. Ripple images [8]are more similar to flowing-line images than ripple
images [6],[7], but flowing-line patterns are slightly more
linear than ripple patterns. In the conventional methods[6],[7],[8], ripple patterns cannot be generated in whiteand black areas. On the other hand, our method cangenerate flowing-line patterns in the white and black
areas. Through an experiment using various photographic
images, our method examines that flowing-line patterns
can be generated on the entire image. Additionally,
through an experiment that changes the values of theparameters in our method, how the generated flowingline patterns change is examined.

1. Introduction
Many studies [1],[2] have been conducted on nonphotorealistic rendering using computer technology.
Since non-photorealistic rendering targets all expressions
other than realistic expressions by photorealistic
rendering, there are various methods for dealing withvarious styles and uses. We focus on non-photorealistic
rendering methods that use image processing to generate
non-photorealistic images that are different fromconventional art expressions such as oil paintings andwatercolors. As non-photorealistic rendering methods ofthe unprecedented art expressions, many methods forgenerating labyrinth images [3], cell-like images [4], andmoire-like images [5] from photographic images havebeen proposed. Labyrinth images are composed ofequally spaced lines like a maze and are generated using
minimum spanning trees, cell-like images are composedof cell-like patterns with cell membranes and cell nuclei
and are generated using inverse iris filter, and moire-like
images are a kind of op art and are generated using
bilateral filter and unsharp musk.
We propose a non-photorealistic rendering method forautomatically generating flowing-line images fromphotographic images. Flowing-line images are composedof unidirectional (here vertical) lines with smooth curves.
The black lines are thickly represented at the edges andblack areas of photographic images, and are represented
along the edges in the nearly vertical direction. Our
method is executed by an iterative calculation using
vertical and horizontal smoothing filters. As images

2. Our method
Our method is largely executed in three steps. First,
since flowing-line patterns are unlikely to occur in thewhite and black areas, gray scale transformation is
performed on a photographic image. The gray scaletransformation eliminates pixel values close to white andblack, and eliminates areas where flowing-line patterns
cannot be generated. Second, processing is performed©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
954

Karin Kuroki, Toru Hiraoka

where 𝑊2 is the window size and 𝑙 is the
positions in the window.
A flowing-line image is obtained after 𝑇 times
iteration of Steps 2 and 3.

using horizontal smoothing filter and inverse filter. Third,
processing is performed using vertical smoothing filter.
A flowing-line image is generated by iterating throughthe second and third steps. A flow chart of our method isshown in Fig. 1.

3. Experiments
We conducted two experiments. First, we visuallyconfirmed flowing-line patterns by changing the valuesof the parameters in our method using Lighthouse image
shown in Fig. 2. Second, we applied our method to eight
photographic images shown in Fig. 3. All photographic
images used in the experiments were 512 * 512 pixels and
256 gradations.

Fig. 1. Flow chart of our method
Fig. 2. Lighthouse image
Details of the steps in Fig. 1. are explained below.
Step 0: Let the input pixel values on coordinates (𝑖, 𝑗) ofa gray-scale photographic image be 𝑜𝑖,𝑗 . The pixel
values 𝑜𝑖,𝑗 have value of 𝑈 gradation from 0 to𝑈 − 1.
Step 1: Let the pixel values that have undergone grayscale transformation as shown in the following
equation be 𝑓𝑖,𝑗 .
𝑈−2𝐷−1𝑓𝑖,𝑗 = 𝑜𝑖,𝑗
+𝐷
(1)
𝑈−1
where 𝐷 is a natural number. The larger the valueof 𝐷, the fewer pixels that are close to white or
black. When the value of 𝐷 is 0, the pixel values𝑓𝑖,𝑗 have the same pixel values as the original
photographic image.
Step 2: Let the pixel values of the image at the 𝑡 -th(𝑡)(0)
iteration number be 𝑓𝑖,𝑗 , where 𝑓𝑖,𝑗 = 𝑓𝑖,𝑗 . Let
the pixel values that horizontal smoothing filter isapplied as shown in the following equation be(𝑡)𝑠1,𝑖,𝑗 .
(𝑡)𝑠1,𝑖,𝑗 =Fig. 3. Photographic images
3.1. Experiment with changing parameter values
Flowing-line images generated by changing theiteration number 𝑇 were visually confirmed using
Lighthouse image. The value of 𝑇 was set to 5, 10, 20,
and 40. The values of the other parameters 𝐷, 𝑊1 , and 𝑊2
were set to 10, 6, and 3, respectively. The results of theexperiment are shown in Fig. 4. As the value of 𝑇 waslarger, flowing-line patterns became clearer and wereexpressed finely.

(𝑡−1)

𝑊

1
∑𝑘=−𝑊
𝑓𝑖+𝑘,𝑗1

2𝑊1 +1(2)where 𝑊1 is the window size and 𝑘 is the
positions in the window. Let the pixel values that
inverse filter is applied as shown in the following(𝑡)equation be 𝑔𝑖,𝑗 .
(𝑡)

(𝑡−1)

𝑔𝑖,𝑗 = 𝑓𝑖,𝑗(𝑡)

− 𝑠1,𝑖,𝑗 + 𝑓𝑖,𝑗Flowing-line images generated by changing the valueof the parameter 𝐷 were visually confirmed using
Lighthouse image. The value of 𝐷 was set to 0, 10, 20,
and 30. The values of the other parameters 𝑇, 𝑊1 , and 𝑊2
were set to 40, 6, and 3, respectively. The results of theexperiment are shown in Fig. 5. As the value of 𝐷 waslarger, flowing-line patterns became clearer in the whitearea at the bottom of Lighthouse image. On the other
hand, as the value of 𝐷 was larger, it became moredifficult to recall Lighthouse image.

(3)

(𝑡)
(𝑡)
If 𝑔𝑖,𝑗 is smaller than 0, then 𝑔𝑖,𝑗 must be set to 0,(𝑡)(𝑡)
and if 𝑔𝑖,𝑗 is greater than 𝑈 − 1, then 𝑔𝑖,𝑗 must

be set to 𝑈 − 1.
Step 3: Let the pixel values that vertical smoothing filter
is applied as shown in the following equation be(𝑡)𝑓𝑖,𝑗 .
(𝑡)𝑓𝑖,𝑗 =

𝑊

(𝑡)

2
∑𝑙=−𝑊
𝑔𝑖,𝑗+𝑙2

2𝑊2 +1

Flowing-line images generated by changing thewindow size 𝑊1 were visually confirmed using(4)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
955

Generation of Flowing-Line Images

Lighthouse image. The window size 𝑊1 was set to 2, 4,
6, and 8. The values of the other parameters 𝑇, 𝐷, and 𝑊2
were set to 40, 10, and 3, respectively. The results of theexperiment are shown in Fig. 6. As the value of 𝑊1 waslarger, the interval of flowing-line patterns became wider.

were set to 40, 10, and 6, respectively. The results of theexperiment are shown in Fig. 7. As the value of 𝑊2 waslarger, flowing-line patterns became smoother.
Furthermore, in Figs. 4 to 7, it was also found thatflowing-line patterns can be generate in the white area atthe bottom of Lighthouse image.

Flowing-line images generated by changing thewindow size 𝑊2 were visually confirmed using
Lighthouse image. The window size 𝑊2 was set to 1, 2,
3, and 4. The values of the other parameters 𝑇, 𝐷, and 𝑊1(a) 𝑇 = 5(a) 𝑊1 = 2(b) 𝑇 = 10(c) 𝑊1 = 6(d) 𝑊1 = 8
Fig. 6. Flowing-line images in the case of the window
size 𝑊1 = 2, 4, 6, and 8(c) 𝑇 = 20(d) 𝑇 = 40
Fig. 4. Flowing-line images in the case of the iteration
number 𝑇 = 5, 10, 20, and 40(a) 𝐷 = 0(b) 𝑊1 = 4(a) 𝑊2 = 1(b) 𝐷 = 10(b) 𝑊2 = 2(c) 𝑊2 = 3(d) 𝑊2 = 4
Fig. 7. Flowing-line images in the case of the window
size 𝑊2 = 1, 2, 3, and 4(c) 𝐷 = 20(d) 𝐷 = 30
Fig. 5. Flowing-line images in the case of the parametervalue 𝐷 = 0, 10, 20, and 30©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
956

Karin Kuroki, Toru Hiraoka

3.2. Experiment using eight photographic images

4. Conclusion

Our method was applied to eight photographic images
shown in Fig. 3. The values of the parameters 𝑇, 𝐷, 𝑊1 ,and 𝑊2 were set to 40, 10, 6, and 3, respectively. The
results of the experiment are shown in Fig. 8. For all
flowing-line images, flowing-line patterns werecomposed of unidirectional (here vertical) lines withsmooth curves, and could be automatically generated on
the entire image. Looking at more details, the black lines
were thickly represented at the edges and black areas(clothes, hair, etc. in the lower left image) of
photographic images, and were represented along theedges in the nearly vertical direction.

We proposed a non-photorealistic rendering method forgenerating flowing-line images from gray-scale
photographic images using vertical and horizontal
smoothing filters. Through an experiment using nine
photographic images, it was found that our method can
automatically generate flowing-line patterns on the entire
images. Additionally, through an experiment that thevalues of the parameters in our method were changed, it
was found that the interval and smoothness of flowingline patterns can be changed.
A subject for future study is to expand our method for
application to color photographic images, videos, andthree-dimensional data.
Acknowledgements
This work was supported by JSPS KAKENHI Grant
Number JP23K11727 and The Telecommunications
Advancement Foundation Grant.
References
1.

2.

3.

4.

5.

6.

7.

8.

9.

P. Haeberli. Paint by numbers, abstract image
representations, ACM SIGGRAPH Computer Graphics
24(4), 1990, pp. 207-214.
J. Lansdown and S. Schofield, Expressive rendering, areview of nonphotorealistic techniques, IEEE Computer
Graphics and Applications 15(3), 1995 pp. 29-37.
K. Inoue and K. Uraham, Halftoning with minimum
spanning trees and its application to maze-like images,
LSEVIER Computers & Graphics 33(5), 2009, pp. 638647.
T. Hiraoka, M. Hirota, K. Inoue and K. Urahama,
Generating cell-like color images by inverse iris filter,
ICIC Express Letters 11(2), 2017, pp.399-404.
T. Hiraoka and K. Urahama, Generation of moirepicture-like color images by bilateral filter, IEICE
Transactions on Information and Systems E96-D(8),2013, pp. 1862-1866.
T. Hiraoka and K. Urahama, Generating ripple-pattern-like
images by inverse Sobel filter, Journal of the Institute of
Image Electronics Engineers of Japan 44(2), 2015, pp.
349-352.
T. Hiraoka, Generation of ripple images using intensity
gradient, ICIC Express Letters 13(12), 2019, pp. 10971101.
T. Hiraoka, Generation of arbitrarily-oriented ripple
images using circular-sector-type smoothing filter andinverse filter, Journal of Robotics, Networking andArtificial Life 6(4), 2020, pp. 213-216.
Z. Yu and K. Urahama, Iterative method for inverse
nonlinear image processing, IEICE Transactions on
Fundamentals E97-A(2), 2014, pp. 719-721.

Fig. 8. Flowing-line images©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
957

Powered by TCPDF (www.tcpdf.org)

Generation of Flowing-Line Images

Authors Introduction
Ms. Karin Kuroki
She is a student in the Department ofInformation Systems, University ofNagasaki. His research interests
inculede image processing and nonphotorealistic rendering.

Dr. Toru Hiraoka
He received B.Des., M.Des. and
D.Eng. degrees from Kyushu
Institute of Design in 1995, 1997 and2005, respectively. He is currently aProfessor in University of Nagasaki.
His research interests include nonphotorealistic rendering and disasterprevention.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
958

Human Motion Recognition from Multiple Directions and Its Gait Cycles Analysis
Miki Ooba

Graduate School of Engineering, Kyushu Institute of Technology, 1-1 Sensui, Tobata-ku, Kitakyushu, 804-8550, Japan
Yui Tanjo

Faculty of Engineering, Kyushu Institute of Technology, 1-1 Sensui, Tobata-ku, Kitakyushu, 804-8550, Japan
Email: ooba.miki558@mail.kyutech.jp, tanjo@cntl.kyutech.ac.jp

Abstract
It is crucial for individuals to keep walking and stay healthy to prevent receiving nursing care. This paper proposes amethod of recognizing walk motions and analyzing the gait cycle of a human focusing on his/her posture. We use 43
structural features defined from human joint coordinates obtained using OpenPose and 18 figural features from humandomain images and their difference images. The feature vector containing these 61 features is used for the recognitionof walk motion by Random Forest. In the experiment, we applied the method to recognizing six types of motions andanalyzed the walk gait cycles of five persons, and obtained satisfactory results.
Keywords: OpenPose, images of human area, human walk motion, structural feature, figural feature, gait cycle

1. Introduction
According to the Ministry of Health, Labour andWelfare's ‘Physical Activity Standards for Health
Promotion’ [1], increasing the amount of daily physicalactivity can reduce the risk of deterioration of life
functions and other problems. Therefore, it isrecommended to maintain a certain intensity of physicalactivity (walking or equivalent movements). The purposeof this research is to recognize human daily motions andanalyze the information obtained from them.
Previous studies include posture and motion
recognition methods that use a chest-mounted camera by
the first person viewpoint called MY VISION [2][3],gait recognition by integrating the gait silhouette input
into a CNN [4], and a method that evaluates movement
based on the skeletal trajectory of the whole body using
depth data [5]. There is a motion recognition method [6]using an extension of Motion History Image (MHI)
called Triplet Motion Representation Images, with
Histograms of Oriented Optical Flow. However,attaching a measurement device to a person may cause
unnatural movements. It is necessary to consider an
actual living environment.

(a)(b)
Fig. 1. Figural and structural features. (a) Human
domain image, (b) joint coordinates
2. Foreground Extraction
To recognize a motion from the shape features of ahuman domain image, the human domain is extracted
from the image as the foreground. In this paper, thesequential background estimation method based on theGaussian mixture model [8][9] is used to extract a humandomain corresponding to background change. Then, thenoise on the extracted human domain is removed byexpansion and contraction processing and it is trimmed toa rectangle of a specified scale. Figural features are
extracted from the human domain using the binary andframe difference images shown in Fig. 1(a). After having
the foreground, figural features are extracted by the stepin 3.1In this paper, we propose a method of recognizing
human posture and gait motion based on human joint
positions and image features, as well as a method ofanalyzing gait motion based on the gait cycle, using arandom forest classifier [7] based on a 61-dimensionalfeature vector.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
959

Miki Ooba, Yui Tanjo

3. Feature Extraction1) Knee, ankle, and elbow angles: The knee angle
𝜃𝑘𝑛𝑒𝑒 [deg] is calculated using the inner product of the hipand knee vectors as follows;

We use figural features and structural features torecognize human motion and its gait cycles.
3.1 Figural Features
Our aim is to extract effective features from humanposture images. Eight kinds of figural features used in the
proposed method can be categorized as follows.

𝒂 = (𝑥ℎ𝑖𝑝 − 𝑥𝑘𝑛𝑒𝑒 , 𝑦ℎ𝑖𝑝 − 𝑦𝑘𝑛𝑒𝑒 )(1)𝒃 = (𝑥𝑎𝑛𝑘𝑙𝑒 − 𝑥𝑘𝑛𝑒𝑒 , 𝑦𝑎𝑛𝑘𝑙𝑒 − 𝑦𝑘𝑛𝑒𝑒 )(2)𝜃𝑘𝑛𝑒𝑒 =

1) Image aspect ratio: The ratio of the height to the widthof the human domain image.

180𝒂∙𝒃cos −1‖𝒂‖‖𝒃‖𝜋(3)

where (𝑥ℎ𝑖𝑝 , 𝑦ℎ𝑖𝑝 ), (𝑥𝑘𝑛𝑒𝑒 , 𝑦𝑘𝑛𝑒𝑒 ), (𝑥𝑎𝑛𝑘𝑙𝑒 , 𝑦𝑎𝑛𝑘𝑙𝑒 ) are
the coordinates of the hip, knee and ankle, respectively.
Using the same method, the ankle angles 𝜃𝑎𝑛𝑘𝑙𝑒 (theinner product of toe and knee)and elbow angle 𝜃𝑒𝑙𝑏𝑜𝑤(the inner product of wrist and shoulder) are also obtained.

2) Percentage of a white pixel area in a rectangle: The
percentage of a white pixel area in the human domain
image.

2) Ankle-toe, Hip-ankle angle, and Body tilt: The angle
between the ankle and toe relative to the horizontal line
is defined as the ankle-toe angle 𝜃𝑖𝑛𝑠𝑡𝑒𝑝 [deg] and isgiven by;3) Hand swing (upper body width): The difference
between the rightmost and the leftmost points at 60% of
the height of the upper body of the human domain image.
4) Shoulder height: The y coordinate at which the numberof white pixels per line exceeds a threshold when theimage is scanned from the upper left to the lower right.
The threshold was experimentally set at 40 pixels to
distinguish between a stretch out the back posture andothers.

𝜃𝑖𝑛𝑠𝑡𝑒𝑝 =180
𝑦𝑎𝑛𝑘𝑙𝑒 − 𝑦𝑡𝑜𝑒
tan−1𝜋𝑥𝑎𝑛𝑘𝑙𝑒 − 𝑥𝑡𝑜𝑒

(4)

where (𝑥𝑡𝑜𝑒 , 𝑦𝑡𝑜𝑒 ) is the coordinates of the toe as shownin Fig. 1(b). Similarly, we determine the hip-ankle angle𝜃ℎ𝑖𝑝−𝑎𝑛𝑘𝑙𝑒 (The angle between the vertical line of the hipankle and the horizontal line of the left and the right hip)and the body inclination 𝜃𝑏𝑜𝑑𝑦𝑡𝑖𝑙𝑡 (The angle between theneck and the vertical line of the waist center).

5) Length of the contour line at the bottom of the humanregion image: The length of the contour line at the bottom
40% of the height of the person area image.

3) Steps, Hip-ankle distance, and Wrist-shoulder
distance

6) Body asymmetry: The center of gravity of the lower
40% portion of the person area, and the percentage of thearea to the left of the center of gravity in the white pixel
area.

(a) Scale transformation: Using the length of a person's𝑡ℎ𝑖𝑔ℎ𝑠 [cm], the parameter 𝑠𝑐𝑎𝑙𝑒 [cm/pixel], whichindicates how many centimeters one pixel corresponds toin the current frame, is calculated by the following;7) Position of the center of gravity: The distance of thecenter of gravity of the head from the centerline of theimage.

𝑠𝑐𝑎𝑙𝑒 =

8) Number of white pixels of the difference image: The
number of white pixels in the 25% height area from thebottom of the difference image.

𝑡ℎ𝑖𝑔ℎ𝑠√(𝑥ℎ𝑖𝑝 − 𝑥𝑘𝑛𝑒𝑒 )2 + (𝑦ℎ𝑖𝑝 − 𝑦𝑘𝑛𝑒𝑒 )2(5)(b) Step width, Hip-ankle distance, Wrist-shoulder
distance: The step width is calculated using the Euclidean
distance between the left heel and the right heel asfollows;

Since features 2), 3) and 5) take periodic values, theirminimum and maximum values are also extracted andused as features. In this way, 18-dimensional figuralfeatures are obtained from the image of the humandomain.

𝑠𝑡𝑒𝑝_width= 𝑠𝑐𝑎𝑙𝑒× √(𝑥𝑅ℎ𝑒𝑒𝑙 − 𝑥𝐿ℎ𝑒𝑒𝑙 )2 + (𝑦𝑅ℎ𝑒𝑒𝑙 − 𝑦𝐿ℎ𝑒𝑒𝑙 )23.2 Structural Features(6a)where (𝑥𝑅ℎ𝑒𝑒𝑙 , 𝑦𝑅ℎ𝑒𝑒𝑙 ) and (𝑥𝐿ℎ𝑒𝑒𝑙 , 𝑦𝐿ℎ𝑒𝑒𝑙 ) are thecoordinates of the right and the left heel, respectively.
Concurrently, the Euclidean distance between hip-ankle
and wrist-shoulder is calculated as follows;

We extract the following six structural features (43dimensions) using human joint coordinates provided
from OpenPose [10].

∗ _# = 𝑠𝑐𝑎𝑙𝑒 × √(𝑥∗ − 𝑥# )2 + (𝑦∗ − 𝑦# )2©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
960

(6b)

Human Motion Recognition from

Table 1. The number of frames in the video.

where * indicates hip (or wrist), and # indicates ankle (orshoulder). (𝑥∗ , 𝑦∗ ) and (𝑥# , 𝑦# )are the coordinates of hip
and ankle or wrist and shoulder, respectively.

MotionNormalForward leaning
Fall
Help
Sit
Sleep
Total

4) Walk speed: The feature 𝑤𝑎𝑙𝑘_𝑠𝑝𝑒𝑒𝑑 [cm/s] indicates
how long the coordinates of the waist center (𝑥𝑐0 , 𝑦𝑐0 ) in
the current frame have moved from (𝑥𝑐𝑛 , 𝑦𝑐𝑛 ) in theprevious 𝑛 frames, and is given by the following
equation;
𝑤𝑎𝑙𝑘_𝑠𝑝𝑒𝑒𝑑= 𝑠𝑐𝑎𝑙𝑒√(𝑥𝑐0 − 𝑥𝑐𝑛 )2 + (𝑦𝑐0 − 𝑦𝑐𝑛 )2 × 𝑓𝑟𝑎𝑚𝑒_𝑟𝑎𝑡𝑒
×
𝑛

E
2192
2512
278
565
849
860
7256

(a) walk(b) fall and sleep
Fig. 2. Direction of the motion
5. Experiment5) Direction of motion: The direction of motion in thehorizontal (x-axis) and in the vertical (y-axis) is obtainedby vectorizing the displacement of the coordinates of thewaist center during 𝑛 frames and multiplying it by 𝑠𝑐𝑎𝑙𝑒.

5.1. Experimental Method
One fixed camera is set up at a height of 90 cm from anindoor floor to simulate a real-life situation. A person
appears in the video, and the whole body is taken a video.
The experiment was conducted on five subjects (22-23
years old). They are referred to as A, B, C, D, and E.
Table 1 shows the number of frames contained in each
video.

(8)6) Difference of the foot joint heights and wrist height:
The height of the knee, and heel ℎ𝑒𝑖𝑔ℎ𝑡_∗ [cm] is
calculated by the difference between the heights of theright and the left ℎ𝑒𝑖𝑔ℎ𝑡_∗ , as follows;ℎ𝑒𝑖𝑔ℎ𝑡_∗ = |(𝑦𝑅∗ − 𝑦𝐿∗ )| × 𝑠𝑐𝑎𝑙𝑒

The number of frames
B
C
D
2101 1878 2124
2830 2214 2290
407
405
309
327
511
375
1039 1336
877
712
991
649
7416 7335 6624

(7)

where (𝑥𝑐0 , 𝑦𝑐0 ), (𝑥𝑐𝑛 , 𝑦𝑐𝑛 ) are the coordinates of thecenter of the waist between the current frame andprevious 𝑛 frames, whereas the frame_rate is a unit thatindicates the number of the images makeup in one secondin a video.

𝑥 = (𝑥𝑐0 − 𝑥𝑐𝑛 ) × 𝑠𝑐𝑎𝑙𝑒𝑦 = (𝑦𝑐0 − 𝑦𝑐𝑛 ) × 𝑠𝑐𝑎𝑙𝑒

A
2351
3577
400
394
1322
1141
9185

Six types of motions: Normal walking, forward leaning
walk, falling, asking for help (beckoning to the camera),
sitting, and sleeping are captured on video. Fig. 2 shows
the direction.

(9)

Gait cycle [11]: Gait cycle (seven periods) recognitionis performed for normal walking, i.e. when walking fromthe right to the left, the target of the gait cycle is left foot,whereas when walking from the left to the right, the targetof the gait cycle is right foot. So we simply call it leftright foot. We manually select seven periods of thenormal walk and put the selected periods into sevenclasses for training. The seven classes (periods) are
Loading Response (LR), Mid Stance (MSt), Terminal
Stance (TSt), Pre swing (PSw), Initial Swing (ISw), Mid
Swing (MSw), Terminal Swing (TSw). In the gait cycle,
58 of the 61-dimensional features mentioned in section 3
are used for training. However, posture Initial Contact at
the start of the cycle is omitted.

where * indicates knee, or heel, (𝑥𝑅∗ , 𝑦𝑅∗ ) and (𝑥𝐿∗ , 𝑦𝐿∗ )are the coordinates of the right and the left knee, or heel,
respectively. Moreover, the height from floor to toe andthe height from floor to wrist are calculated in the sameway.
Features such as knee and ankle angles are calculatedwith the left and the right foot, respectively. The feature
values 1), 3), and 6) are then repeated for similar values
in each gait cycle (e.g., in the walking motion, a person
repeatedly bends and extends his/her knees). Therefore,
in addition to the features 1) to 6) above in the current
frame, the minimum and maximum values of thesefeatures in the past 𝑛 frames are included as features.
This results in 43 dimensional features.

5.2. Method of Evaluation

Finally, from the figural and structural features, we
represent a human posture using a 18+43=61dimensional feature vector.

Leave-one-out cross-validation was applied to theexperimental data. The percentage of correct frames to allframes is used for evaluation. In the gait cycle analysis,left-right balance is evaluated by determining the leftright difference in the percentage of each period to thefull video.

4. Learning and Identification
The proposed method uses Random Forest [7] as adiscriminator.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
961

Miki Ooba, Yui Tanjo

Table 2. Result of motion recognition
MotionNormal walk
Forward leaning
Fall
Help
Sit
Sleep

A
97.0
91.5
81.3
79.7
100
92.3

B
99.4
98.4
75.2
82.0
100
96.1

Precision [%]CD
93.8 99.6
89.6 98.9
87.7 87.7
95.1 99.2
92.5 10099.0 93.1

E
98.8
95.6
78.4
99.3
100
90.6

Ave
97.8
94.7
82.0
92.1
98.2
94.2

Fig. 3. Normal walk and the result of recognition

Table 3. Result of gait cycle recognition
PhasesStance
phase

Swing
phase

Periods(class)
LR
MSt
TSt
PSw
ISw
MSw
TSw

A
45.0
88.7
97.4
55.8
87.9
84.7
91.8

B
61.4
90.7
93.6
77.6
90.4
83.0
81.0

Precision [%]CD
63.9 76.8
90.8 94.5
79.0 98.0
77.3 68.5
86.8 87.4
81.6 80.4
66.1 90.9

E
67.4
85.3
81.2
67.8
75.2
82.5
78.5

Table 4. The difference of left-right foot.
Ave
62.4
89.9
89.6
67.8
85.4
82.6
81.9

PhasesStance phase

Swing phase

6. Results and Discussion

Periods(class)
LR
MSt
TSt
PSw
ISw
MSw
TSw

left-right foot differences [%]
A
B
C
D
E
0.3
-5.4 -2.4
2.0
2.5
4.4
-0.3
0.3
0.8
-1.3
-1.9
1.4
0.7
0.9
0.8
-2.1
2.3
1.0
-1.1
1.1
0.6
-4.3
1.0
-3.7
0.4
-0.4
1.4
0.1
2.7
0.7
-0.8
4.8
-0.7 -1.7 -4.2

for human health, and to identify the cases where multiple
movements and directions are mixed. If we can provide
accurate and appropriate advices to individuals, we will
be able to offer better health services.

6.1. Results
Tables 2 and 3 show the motion and gait cycle
recognition results, respectively. Note that, help and sit
motion are recognized without considering the direction.
Fig. 3 shows some of the images of the recognition results.
The upper left of the image shows the motion and gait
cycle recognition results. Table 4 shows the different
ratios of each period of the gait cycle of walking from theright to the left (the gait cycle of left foot) and the left tothe right (the gait cycle of right foot) of each subject.

References
Cabinet Office, “Physical Activity Standards for Health
Promotion 2013”, 2013.
2. J. K. Tan, T. Kurosaki, “Estimation of self-posture of apedestrian using MY VISION based on depth and motion
network”, Journal of Robotics, Networking and Artificial
Life, Vol.7, No.3, pp.152-155, 2020.
3. Z. Liu, J.K. Tan, “Analysis of human walking postureusing a wearable camera”, International Journal of
Innovative Computing, Information and Control, Vol.19,
No.3, pp.805-819, 2023.
4. H. Chao, K. Wang, Y. He, J. Zhang, “GaitSet: Cross-view
gait recognition through utilizing gait as a deep set”, IEEE
Transactions on Pattern Analysis and Machine Intelligence,
Vol.44, No.7, pp.3467-3478, 2022.
5. B. B. Amor, A. Srivastava, P. Turaga, G. Coleman, “A
framework for interpretable full-body kinematic
description using geometric and functional analysis”,
IEEE Transactions on Biomedical Engineering, Vol.67,
No.6, pp.1761-1774, 2022.
6. J. Cao, Y. Yamashita, J. K. Tan, “Human motion
recognition using TMRIs with extended HOOF”, Journalof Robotics, Networking and Artificial Life, Vol. 7, No. 4,
pp. 231-235, 2021.
7. L. Breman, “Random forests”, Machine Learning, Vol. 45,
No. 1, pp. 5-32, 2001.
8. C. Stauffer, W. E. L. Grimson: “Adaptive backgroundmixture models for real-time tracking”, Proceedings ofConference on Computer Vision and Pattern Recognition,
Vol. 2, pp. 246-252, 1999.
9. A. Shimada, D. Arita, R. Taniguchi, “Increment anddecrement of Gaussians in adaptive mixture-of-Gaussian
background models”, MIRU2006, pp. 741-751, 2006.
10. Z. Cao, T. Simon, S. Wei, Y. Sheikh, “Realtime multiperson 2D pose estimation using part affinity fields.”, Proc.
1.

6.2. Discussion
Motion Recognition: The overall average accuracy was
95.2%. As shown in Table 2, Fall was less accurate thanother motions, and the variation among subjects was
greater than others. In fact, there are individual
differences in the way the subject falls and shifts
diagonally. Collecting more training data is needed.
Gait cycle: The overall average accuracy was 82.4%. In
many cases, the gait cycle class was recognized as theclass before or after the class. This is because walking isa continuous motion and the before and after postures are
similar. From Table 4, it is expected that subject A's Mst
is shorter than that of the left, indicating a slight decreasein the muscle strength of the right leg. Since analysis of a
subject gait cycle is dependent on the recognition rate, therecognition accuracy in each class needs to be improved.
7. Conclusion
In this paper, we proposed a motion recognition andgait cycle analysis method using 61 features representing
human posture and Random Forest as a discriminator. It
was applied to the recognition of 6 motions and theanalysis of 7 periods of gait cycles. Further study isneeded to obtain more accurate values, such as step width,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
962

Powered by TCPDF (www.tcpdf.org)

Human Motion Recognition from2017 IEEE Conference on Computer Vision and Pattern
Recognition, No. 121, pp. 1302-1310, 2017.
11. S. Demura, “Fall prevention for community seniors frombasic theory of falls to intervention practice”, kyorin-shoin,
pp.114-116, 2012. (in Japanese)

Authors Introduction
Ms. Miki Ooba
She received B.E. degree inIntelligent Control Engineering
Course, Faculty of Engineering,
Kyushu Institute of Technology. Her
research interests include motion
analysis of elderly people in order toimprove their QOL.

Dr. Yui Tanjo
Dr. Tanjo is currently a professor with
the Department of Mechanical andControlEngineering,
KyushuInstitute of Technology. Her current
research interests include ego-motion
analysis by MY VISION, threedimensional shape/motion recovery,
human detection, and its motion
analysis from video. She was awarded
SICE Kyushu Branch Young Author’s Award in 1999, the
AROB Young Author’s Award in 2004, the Young
Author’s Award from IPSJ of Kyushu Branch in 2004, andthe BMFSA Best Paper Award in 2008, 2010, 2013 and2015. She is a member of IEEE, The Information
Processing Society, The Institute of Electronics,
Information and Communication Engineers of Japan.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
963

A Method of Improving the QOL of the People with Visual Impairment by MY VISION
Shun Kitazumi

Graduate School of Engineering, Kyushu Institute of Technology, 1-1 Sensuicho, Tobata-ku, Kitakyushu, 804-8550,
Japan
Yui Tanjo

Faculty of Engineering of Engineering, Kyushu Institute of Technology, 1-1 Sensuicho, Tobata-ku, Kitakyushu, 8048550, Japan
Email: kitazumi.shun215@mail.kyutech.jp, tanjo@cntl.kyutech.ac.jp

Abstract
Visually impaired people face several difficulties in indoor activities, such as spending excessive time in locating
objects. This paper proposes a method for assisting object acquisition by detecting desired objects and guiding users
to them. The method requests a user to express the object he/she wants to acquire verbally and utilizes speech
recognition to detect the specified object. Subsequently, the system guides in voice the user's hand to the location of
the desired object. The performance of the method is experimentally shown. The method contributes to enhancing the
comfort of indoor activities of visually impaired and, in this way, improves their quality of life.
Keywords: MY VISION, Visually impaired, QOL, Object acquisition, Network diagram

part in supporting indoor activities for people with visualacuity difficulties.
Currently, an application has been released that uses asmartphone to capture a registered object with a camera
and read it aloud to the user[8]. This application is very
useful for identifying objects that are similar in shape, butsince it assumes that the object's location can be
accurately identified and captured by the camera, it does
not work unless the visually impaired person knows theobject's location. In addition, as a study that addresses theguidance of object acquisition paths, a system has been
proposed by J.P. Docto et al. [9], in which a smart glove
is worn by a visually impaired person and guides him or
her to the object utilizing the built-in camera and sensors.
This system does not require text input or touch operationand is easy for visually impaired people to handle.
However, in practical use, there are issues related topracticality, such as the time and effort required to put on
and take off the gloves, the weight of the gloves when
they are worn, and the resistance caused by the wiring.
In this study, we propose a system that provides voice
guidance on a route to the object requested based on MY
VISION (a Magic eYe of a Visually Impaired for Safety
and Independent actiON), thereby reducing the burden ofvisually impaired people and making the system moreintuitive and practical. Specifically, the system canprovide real-time guidance using a notebook PC and a
webcam, regardless of location, and enables it by voice
without text input or button operation. This is expected tosolve the practical limitations of smartphone applications
and smart gloves that have been pointed out in previous
research. Therefore, the system proposed in this study can

1. Introduction
Many people around the world suffer from visually
impaired vision, a condition that interferes with their
daily lives due to persistent vision loss. According to areport [1] published by the World Health Organization(WHO) in 2019, the number of people with visualdifficulties worldwide is estimated to be at least 2.2
billion, and there are concerns that the number of people
affected will increase further due to population growth
and aging.
In response to this worldwide social problem ofvisibility difficulties, a great deal of research anddevelopment has been conducted to support the daily
lives of people with visibility difficulties. For example,
there has been the development of a pedestrian crossing
navigation system using smart glasses to enable people
with vision difficulties to safely walk across pedestriancrossings alone [2][3][4][5], and the development of asystem to assist people in getting on and off public
transportation [6][7].
As in the above studies, much of the research on
assisting people with vision difficulties tends to focus on
supporting outdoor activities. However, there are many
problems that occur during indoor activities, such as
spending excessive time for acquiring an object whenlooking for it at home. Therefore, the purpose of thisstudy is to improve the quality of life (QOL) of people
with visual acuity difficulties by focusing on a proposal
for an object acquisition support method, which plays a©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
964

Shun Kitazumi, Yui Tanjo

be an important step toward improving the quality of lifeof people with vision difficulties during indoor activities
by supporting smooth acquisition of the target object
when they are looking for something at home.
2. Methodology
2.1. Voice recognition
Fig. 1 Object detection results. (The target object
is drawn in a blue bounding box and the user’s
hand is in a green box.)

In this study, when a user has an object that he/shewants to acquire, the user is asked to tell the object loud(speech transmission) to the proposed system, and thetarget object’s name is recognized by analyzing the
speech. By asking the user to say the trigger word before
the object name, the noun that follows the trigger word is
known as the target object’s name. In the system, "MYVISION" is used as the trigger word. For example, if theuser wants to acquire a cell phone, he/she just say, "MYVISION, find a cell phone". The system then guides theuser's hand to the location of the desired object using
voice guidance.

Moreover, if an object to be detected is not detected in
the current frame, then the system estimates theundetected position by referring to the previous network
diagram.
The first condition for selecting the network diagram tobe referred to is when the number of detected objects isthree or more and the number of detected objects is thelargest (Eq. (1)), and the second condition is when the
network diagram consists of the same type of objects fora certain period (Eq. (2)).

2.2. Object detection
In the proposed system, we use YOLOv7 (You Only
Look Once version 7) [10] to detect a target object in realtime. Note that in the system, objects that are present on
the desk and easily portable (e.g., remote controls, cell
phones, cups) are considered as a class of objects to be
detected. In addition, by combining YOLOv7 and
DeepSORT (Deep Learning Simple Online and Realtime
Tracking) [11], object features are extracted on eachframe, and each object is assigned a unique ID for
tracking. Fig. 1 shows the object detection results.

{

𝑜𝑏𝑗_𝑐𝑛𝑡𝐹(𝑡) ≥ 3
𝑜𝑏𝑗_𝑐𝑛𝑡𝐹(𝑡) ≥ 𝑚𝑎𝑥_𝑜𝑏𝑗_𝑐𝑛𝑡(1)Here, F(t), obj_cntF(t) and max_object_cnt are current frame,
the number of objects detected in the current frame and the
maximum number of objects detected in each frame,
respectively.

𝑆={

𝑆 + 1 if ∀𝐸(𝐹(𝑡)) = ∀𝐸(𝐹(𝑡 − 1))0otherwise(2)

𝑆 ≥ 𝑇ℎ ⟹ SetReferenceModel(∀𝐸(𝐹(𝑡)))2.3. Route guidance to a target object

Here, S, E(F(t)), and Th are the variable for state continuity,node in F(t) and threshold, respectively.

If a target object is detected, the direction of the object
location (referred to as a route hereafter) is guided byvoice guidance according to the clock position. We use
MediaPipe Hands [12] to estimate the user's hand area
and align the user's hand with the target object. When thetarget object is in the 9 o'clock direction (left direction),
the system announces "move to 9 o'clock direction" withvoice.
The proposed method not only focuses on the hand andthe target object, but also employs proposed a network
diagram to represent the positional relationship among
the surrounding objects based on the object detection
results. In the network diagram, nodes are defined as thecenter coordinates of the object, and the edges are thedistances in pixel between objects. The network diagram
is introduced because the system can find the target object
position smoothly, and also can find the position of thetarget object more precisely, when the target is very close
to the other objects.

The object location estimation method first calculates
the amount of object movement common to the current
frame and the reference network diagram, respectively,
and derives the average amount of movement (Eq. (3)).
The position of the undetected object is then estimatedbased on the coordinates of the model and the average
displacement (Eq. (4)). The meanings of the letters in theequations are shown in Table 2.
𝑛

𝑥̅ =

1
∑(𝑥𝑡,𝑖 − 𝑥𝑚𝑜𝑑𝑒𝑙,𝑖 )
𝑛
𝑖=1
𝑛

1
𝑦̅ = ∑(𝑦𝑡,𝑖 − 𝑦𝑚𝑜𝑑𝑒𝑙,𝑖 )
𝑛
{

(3)

𝑖=1

(𝑖 = 1, ⋯ , 𝑛){𝑥𝑒𝑠𝑡,𝑘 = 𝑥𝑚𝑜𝑑𝑒𝑙,𝑘 + 𝑥̅
𝑦𝑒𝑠𝑡,𝑘 = 𝑦𝑚𝑜𝑑𝑒𝑙,𝑘 + 𝑦̅©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
965

(4)

A Method of Improving

Table 2. The meanings of Eq. (3) and Eq. (4)Average displacement in the 𝑥𝑥̅ , 𝑦̅or y-axis direction
Number of common nodes𝑛between current frame andreference modelx or y coordinates of the𝑥𝑡,𝑖 , 𝑦𝑡,𝑖common node in the current
frame
x or y coordinate of the𝑥𝑚𝑜𝑑𝑒𝑙,𝑖 , 𝑦𝑚𝑜𝑑𝑒𝑙,𝑖common node in the modelx or y coordinate estimates for
𝑥𝑒𝑠𝑡,𝑘 , 𝑦𝑒𝑠𝑡,𝑘missing nodes
x or y coordinates of the nodein the model corresponding to
𝑥𝑚𝑜𝑑𝑒𝑙,𝑘 , 𝑦𝑚𝑜𝑑𝑒𝑙,𝑘𝑥𝑒𝑠𝑡,𝑘 or 𝑦𝑒𝑠𝑡,𝑘(b) Network diagram of the current frame

The first condition for starting object location
estimation is when a reference network diagram exists,
and the second condition is when the number of detected
objects is two or more and there are two or more objects
in common with the reference network diagram. Fig. 2
shows the results of object location estimation using theabove condition.
In the proposed system, a monocular RGB camera isused to align the hand with the target object without using
a depth camera. The condition for determining object
acquisition is achieved by employing the bounding box
positional relationship between the target object and thehand region, and by judging the change in handorientation that occurs when the hand gestures comecloser to the camera after grasping the object. By adding
the latter condition, it is expected to prevent the error
judgment of acquisition as successful simply because thebounding boxes of the two objects overlap, even when thedepths are different, and the objects are not touched.
Furthermore, the system also incorporates a process thatchecks whether or not the object closest to the handmatches the target object by referring to the network
diagram and announces when the subject has acquired thewrong object. When the above three conditions are
satisfied, the target object is judged to have been acquired,
and the acquisition is communicated by voice ascompleted. Fig. 3 shows an example that the object
acquisition decision was fulfilled.

(c) Network diagram of the reference model(d) Network diagram reflecting estimation
results(e)Before reflecting location estimation(f) After reflecting location estimation
Fig. 2 Result of object location estimation(When the target object is estimated, it is surrounded bya red bounding box.)(a) Network diagram of the current frame©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
966

Shun Kitazumi, Yui Tanjo

References
WHO:“"World
reportonvision",https://www.who.int/publications/i/item/9789241516570,
2019, (Accessed 2023-12-04).
2. H. Son, D. Krishnagiri, V. Jeganathan, J. Weiland:“Crosswalk guidance system for the blind”, 2020 42nd
Annual International Conference of the IEEE Engineeringin Medicine & Biology Society (EMBC), pp.3327-3330,
2020.
3. J. K. Tan, T. Ishimine, S. Arimasu: “Walk Environment
Analysis Using MY VISION: Toward a Navigation
System Providing Visual Assistance”, International
Journal of Innovative Computing, Information and Control,Vol. 15, No.3, pp.861-871, 2019.
4. J. K. Tan, K. Kitagawa: “A Method of Navigating a
Visually Impaired Person Using MY VISION”, Journal of
Robotics, Networking and Artificial Life, Vol.9, No. 1, pp.
25-30, 2022.
5. T. Kumano, J. K. Tan: “Traffic signs and signals detection
employing the my vision system for a visually impaired
person”, ICIC Express Letters, Part B: Applications pp.
385-391, 2015.
6. S. M. Cruz, L. A. M. Hernández, et al.: “An Outdoor
Navigation Assistance System for Visually Impaired
People in Public Transportation", IEEE Access, vol.9,
pp.130767-130777, 2021.
7. J. K. Tan, Y. Hamasaki, Y. Zhou, I. Kazuma: “A Methodof Identifying a Public Bus Route Number Employing MY
VISION”, Journal of Robotics, Networking and Artificial
Life Vol.8, No.3, pp.224-228, 2021.
8. INTEC
Inc.
：
https://www.intec.co.jp/news/2020/0318_1.html,(Accessed 2023-12-05).
9. J. P. Docto, A. I. Labininay, J. F. Villaverde: “Third eye
hand glove object detection for visually impaired using
You Only Look Once (YOLO)v4-tiny algorithm”, 2022
IEEE International Conference on Artificial Intelligence inEngineering and Technology (IICAIET), pp.1-6, 2022.
10. Y. Wang, A. Bochkovskiy, H. Liao: “"YOLOv7: Trainablebag-of-freebies sets new state-of-the-art for realtime object
detectors”, arXiv preprint arXiv:2207.02696, 2022.
11. N. Wojke, A. Bewley, D. Paulus, “Simple online andrealtime tracking with a deep association metric”, 2017
IEEE International Conference on Image Processing(ICIP), pp. 3645–3649, 2017.
12. MediaPipe,
https://developers.google.com/mediapipe,(Accessed 2023-12-05).
1.

Fig. 3 Image on completion of the object acquisition
that satisfies the three conditions

Fig. 4 Arrangement where other objects are present
in the guidance path to the target object (target
object: remote controller)

3. Experimental Result
Five objects were selected as candidate objects foracquisition: a book, a cell phone, a cup, a plastic bottle,
and a remote controller. An experiment was conducted to
select an acquisition target from these objects and to
make a robot guide a blindfolded user along a path to thetarget object. As shown in Fig. 4, we placed other objects
on the guided path to the target object to verify if it ispossible to approach the target object again after
announcing that the wrong object has been grabbed. The
target object position was changed in the directions of 9,
10, 12, 2, and 3 o'clock with respect to the hand position
to ensure that there was no dependence on the object
placement. The evaluation index was Accuracy which
represents the percentage of correct answers acquired. A
total of 25 acquisition experiments were conducted with
5 choices among 5 objects times 5 object positions,
resulting in an Accuracy of 92%.
4. Conclusion
In this paper, we proposed a system that provides voice
guidance on the path to object acquisition based on MY
VISION and contributes to improving the quality of life
during indoor activities of the people with visiondifficulties. In addition, by utilizing the information
obtained from Mediapipe Hands and network diagrams,
we developed a practical and intuitive system thatprovides all information and guidance by voice, as wellas countermeasures in case that object tracking isinterrupted.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
967

Powered by TCPDF (www.tcpdf.org)

A Method of Improving

Authors Introduction
Mr. Shun Kitazumi
He received his Bachelor’s degree inEngineering in 2022 from the Facultyof Engineering, Kyushu Institute oftechnology in Japan. He is currently
a master student in Kyushu Instituteof Technology, Japan

Dr. Yui Tanjo
Dr. Tanjo is currently a professor with
the Department of Mechanical andControlEngineering,
KyushuInstitute of Technology. Her current
research interests include ego-motion
analysis by MY VISION, threedimensional shape/motion recovery,
human detection, and its motion
analysis from video. She was awarded
SICE Kyushu Branch Young Author’s Award in 1999, the
AROB Young Author’s Award in 2004, the Young
Author’s Award from IPSJ of Kyushu Branch in 2004, andthe BMFSA Best Paper Award in 2008, 2010, 2013 and2015. She is a member of IEEE, The Information
Processing Society, The Institute of Electronics,
Information and Communication Engineers of Japan.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
968

Human Behavior Segmentation and Recognition Using a Single-camera
Jing Cao

Graduate School of Engineering, Kyushu Institute of Technology, 1-1 Sensui, Tobata-ku, Kitakyushu, 804-8550, Japan
Yui Tanjo

Faculty of Engineering, Kyushu Institute of Technology, 1-1 Sensui, Tobata-ku, Kitakyushu, 804-8550, Japan
Email: cao.jing644@mail.kyutech.jp, tanjo@cntl.kyutech.ac.jp

Abstract
In recent years, elderly people living alone account for a large proportion of the elderly population, and the issue ofsafety has also been a matter of great concern for the public. Considering the importance of monitoring the behavior
and activities of the elderly and detecting abnormal movements, this paper proposes a method that can segment humanbehavior into each action and identify the action from the videos taken by a single camera. It uses features that canrepresent the shape of the human area in the depth direction, as well as the features such as motion direction and speed.
The performance and effectiveness of the method are verified by experiments.
Keywords: Behavior segmentation, Motion recognition, Optical flows, TMRIs, Ex-HOOF, MHI

single camera are only suitable for the motion on theplane perpendicular to the optical axis of a camera, andthe motion in the direction of the camera optical axis
(toward or away from the camera) is not dealt with.
Therefore, an extended 3D-MHI [8] and a reverse MHI
method [9] have been proposed. However, [8] has the
problem of high computational cost of creating 3D
images, and, with [9], the recognition rate needs to be
improved.

1. Introduction
Nowadays, the world's population is aging. The number
and proportion of older people in the population is
growing in every country in the world. Population aging
will become one of the most significant social changes in
the 21st century. By 2030, 1 in 6 people in the world willbe 60 years or older (16%), and the share of people aged
60 and above will increase from 1 billion in 2020 to 1.4
billion. By 2050, the number of people aged 80 and overis expected to triple between 2020 and 2050, to 426
million [1]. Especially in Japan, 30% of the population isover 60 years old. Some social problems, such as solitary
death which more than 50% of people worry about, arealso intensifying [2]. Moreover, 67.1% of the respondents
felt that the domestic security situation in Japan has
deteriorated in the past 10 years [3]. Considering these
problems, it is necessary to develop a system that can
detect the behavior of the elderly and detect abnormal
behaviors such as crime and theft, so as to realize a safe
and secure society.

In this study, we propose a new method to describe themotion in the depth direction, making it possible torealize the segmentation and recognition of behaviors
containing these motions. We segment human behavior
by extracting features from the human area and thenselect key frames of the segmented actions. The key
frames can be described through TMRIs (Triplet Motion
Representation Images) [10]. The shape features of
TMRIs, the features of optical flow and the changes of
motion are also extracted to analyze and identify the
motion of the frame. Finally, the recognition results ofthese frames are counted, and the final result of behavior
recognition is obtained.

In related research on behavior recognition, X. Yang etal. [4] proposed a behavior segmentation and recognition
method based on CSI, but it is not aimed at thesegmentation of continuous human behavior. W. Xing etal. [5] proposed a behavior segmentation method basedon posture histograms and adjusted sliding windows.
This method requires learning the features of various
postures. In related research on action recognition using
computer vision, the conventional methods include theFlow Vectors method [6] and the MHI (Motion History
Image) method [7]. However, these methods using a

2. Methodology
2.1. Behavior segmentation
When human behavior consists of several types of
motions, it is necessary to separate the behavior into
respective motions to analyze the behavior. In this paper,
we propose an automatic behavior segmentation methodthat does not require prior learning or training. The
feature points are set at equal distances on the contour©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
969

Jing Cao, Yui Tanjo

majority voting, the analysis of behavior composition iscompleted.

line of the human area. Then feature points are tracked
through the LK method [11] to obtain the optical flow,
and RANSAC [12] is applied to find the true value by
removing the influence of outliers. Changes in the
direction of human movement can be distinguished fromchanges in the trajectory of the center of gravity
coordinates and the average value of optical flow.
Therefore, we calculate the point where a sudden change
in trajectory occurs in the x-axis direction of the center ofgravity coordinates and the boundary point between plus
and minus of the average value of optical flow separately.

Fig.1 The definition of the ground truth.
In Equation (1) and (2), if the change of the current
frame exceeds a certain threshold, division is performed
from the current frame and the number of the previous
frame is used as the division point．
𝑋_𝑑𝑖𝑓𝑠𝑢𝑚𝑛𝑢𝑚𝑡𝑚𝑝 + 1 𝑖𝑓 𝑡ℎ1 << 𝑡ℎ2
𝑛𝑢𝑚𝑡𝑚𝑝 = {𝑛𝑢𝑚𝑡𝑚𝑝1𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

2.2. Motion recognition
To capture the motion in the depth dimension, we
employ TMRIs to represent actions. We identify andclassify actions by leveraging their shape features,
extracting directional velocities from the movements, andobserving changes in the human region.

(1)

𝑛𝑒𝑤−𝑛𝑢𝑚𝑡𝑚𝑝

𝑋_𝑑𝑖𝑓𝑠𝑢𝑚 =∑𝑛=𝑛𝑒𝑤(𝑥𝑛 − 𝑥𝑛−1 )𝑛𝑢𝑚𝑡𝑚𝑝(2)

TMRIs is an extension of the conventional MHI. It is
designed to express human motion, including motion
along the camera optical axis, using three types of motion
history images: newness, density, and depth. Amongthem, newness represents the MHI, density illustrates theappearance frequency of the foreground in the past τframes, and depth indicates the depth information
obtained from the FoE detection results. The shapefeatures of TMRIs, is defined as 𝑽𝑇𝑀𝑅𝐼𝑠 . Ex-HOOF(Extended Histogram of Oriented Optical Flow) [10] isused to extract the speed and directional information of
motions that are not readily visible in TMRIs features.
The feature of Ex-HOOF is denoted as 𝑽𝐸𝑥−𝐻𝑂𝑂𝐹 .

where, th1 and th2 are thresholds. 𝑛𝑢𝑚𝑡𝑚𝑝 is a parameterthat determines whether or not to divide from the current
frame. 𝑋_𝑑𝑖𝑓𝑠𝑢𝑚 is the average rate of change of thecenter of gravity’s x-axis coordinate, and new is thenumber of the current frames.
When the direction of human motion changes, it canalso respond to the average of optical flow. Theintersection of the average optical flow 𝑚𝑎𝑣𝑒 and the xaxis is determined to be the division point. 𝑚𝑎𝑣𝑒 isdefined by the following formula.

𝑚𝑎𝑣𝑒 =∑𝑁
𝑘=1 𝑚𝑘𝑁

Additionally, to represent more complex and detailed
motion features, we incorporate changes in the centroid
and area (AC) of the human region. The feature can be
defined as 𝑽𝐴𝐶 = (𝑭𝐴 , 𝑭𝐶 ) . Among them, the area
feature 𝑭𝐴 = (𝐴𝑟𝑒𝑎𝜏𝑎𝑣𝑒 , 𝐴𝑟𝑒𝑎𝜏𝑠𝑑 ), where 𝐴𝑟𝑒𝑎𝜏𝑎𝑣𝑒 is theaverage value, and 𝐴𝑟𝑒𝑎𝜏𝑠𝑑 denotes the standard deviationof the change of the foreground area. The formulas are as
follows;(3)

where, 𝑚𝑘 represents the value of the k-th optical flow,
and N denotes the total number of optical flows.
Then the division points are determined by matching
the points of the two features after performing datasmoothing within the threshold. The video can be divided
using the obtained division points. After that, key frames
are selected in each divided video, as shown in Fig. 1. In
the figure, the orange points are the division points
detected by features, which separate the behavior into
several motions. The green points are the points that
divide the motion into four equal parts, are also extracted
as key frames.

𝐴𝑟𝑒𝑎𝜏𝑎𝑣𝑒 =1 𝜏𝑡−𝑖∑ 𝐴𝑟𝑒𝑎𝑐𝑜𝑚𝑝
𝜏
𝑖=0

1 𝜏
𝑡−𝑖 − 𝐴𝑟𝑒𝑎 𝑎𝑣𝑒 )2𝐴𝑟𝑒𝑎𝜏𝑠𝑑 = √ ∑ (𝐴𝑟𝑒𝑎𝑐𝑜𝑚𝑝
𝜏
𝜏
𝑖=0

(4)
(5)

𝑡
Here, 𝐴𝑟𝑒𝑎𝑐𝑜𝑚𝑝
is the changes of areas between theframes, given by𝑡
𝐴𝑟𝑒𝑎𝑐𝑜𝑚𝑝=

In the end, by identifying key frames and determining
the classification of the actions they represent through(𝐴𝑟𝑒𝑎𝑡 − 𝐴𝑟𝑒𝑎𝑡−𝑝 )𝐴𝑟𝑒𝑎𝑡©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
970

(6)

Human Behavior Segmentation and350

𝑽 = (𝑽𝑇𝑀𝑅𝐼𝑠 , 𝑽𝐸𝑥−𝐻𝑂𝑂𝐹 , 𝑽𝐴𝐶 )

300

(11)

250
200
150

Action recognition is conducted using the k-nearest
neighbors (k-NN) method in the proposed method.

100
50
0
1

a.

51

101

151

201

251

301

351

401

3. Results and Discussion

Change in the trajectory of the center of gravity
coordinate in the x-axis

3.1. Accuracy of behavior segmentation

6
4
2

In the behavior segmentation experiment, three people(labelled as 1,2,3) perform behavior A, B, C and D. The
specific motion sequences are as follows: A: Walking left
rear → Walking right → Walking front, B: Walking rightrear → Walking left → Walking front, C: Walking rear
→ Walking left → Walking right front, D: Walking rear
→ Walking right → Walking left front. In thesegmentation experiments, we verify the accuracy andrecall of segmentation point selection based on the
trajectory of the centroid in the x-axis and the average
direction of optical flow. The example of the features of
behavior A is shown in Fig. 2. The recall rate (Recall) is
defined by

0
-2

1

51

101

151

201

251

301

351

401

-4
-6
-8

b.

Average change in optical flow in the x-axis
direction
0.15
0.1
0.05
0
1

51

101

151

201

251

301

351

401

-0.05
-0.1
-0.15

c.

Average change in optical flow in the y-axis
direction

Recall=𝑃𝑇×100[%]
𝑃𝐺𝑇(12)

Fig.2. Features used for behavior segmentation.
Here, 𝑃ALL is the total number of calculated segmentation
points, 𝑃𝑇 is the number of segmentation points that were
accurately segmented, and 𝑃𝐺𝑇 is the number of groundtruth segmentation points.
In the experiment, a motion between two motions, suchas turning around or standing, is called a transition action.
If the division point is within a transition action, this point
is considered a correct division point. If one behavior
consists of four consecutive motions, the definition of theground truth division points is shown in Fig. 3.

Fig.3 The definition of the ground truth.
The centroid feature 𝑭𝐶 = (𝐶𝑥𝜏𝑎𝑣𝑒 , 𝐶𝑦𝜏𝑎𝑣𝑒 , 𝐶𝑥𝜏𝑠𝑑 , 𝐶𝑦𝜏𝑠𝑑 ).
means the average value and 𝐶𝜏𝑠𝑑 indicates thestandard deviation of the change in the coordinate of thecenter of gravity. They are defined by
𝐶𝜏𝑎𝑣𝑒

1 𝜏
𝐶𝑥𝜏𝑎𝑣𝑒 = ∑ (𝐶𝑥𝑡 − 𝐶𝑥(𝑡−𝑖−𝑝) )
𝜏
𝑖=0
1 𝜏
𝑎𝑣𝑒
𝐶𝑦𝜏 = ∑ (𝐶𝑦𝑡 − 𝐶𝑦(𝑡−𝑖−𝑝) )𝜏
𝑖=0Table 1 shows the results of behavior segmentation,yielding an average recall of 91.7%.
3.2. Recognition rate of the motions(7)

In the motion recognition experiment, four people
performed 12 kinds of motions, and features were created
for these motions. In the experiment, we use leave-oneout cross-validation to evaluate the accuracy of motion
recognition. The recognition rate R is defined by

(8)

1 𝜏
𝐶𝑥𝜏𝑠𝑑 = √ ∑ ((𝐶𝑥𝑡 − 𝐶𝑥(𝑡−𝑖−𝑝) ) − 𝐶𝑥𝜏𝑎𝑣𝑒 )2
𝜏
𝑖=0

(9)

1 𝜏
𝐶𝑦𝜏𝑠𝑑 = √ ∑ ((𝐶𝑦𝑡 − 𝐶𝑦(𝑡−𝑖−𝑝) ) − 𝐶𝑦𝜏𝑎𝑣𝑒 )2
𝜏
𝑖=0

(10)
𝑅=

𝑁𝑇
× 100[%]
𝑁𝐴𝐿𝐿

Finally, by integrating the above features, we get afeature vector V, as shown below.
©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
971

(13)

Jing Cao, Yui Tanjo

Here, 𝑁𝑇 is the total number of correctly recognizedfeatures and 𝑁𝐴𝐿𝐿 represents the total number of features.

4. Conclusion
In this paper, we proposed a human behavior
segmentation and recognition method, which can handle
the behavior including movements in the depth direction.
For each behavior, the segmentation method proposed in
the paper effectively identifies division points, enabling
the recognition of segmented actions. The average recall
rate for behavior segmentation has reached 91.69%.
Furthermore, in action recognition, the proposed method
achieved an average recognition rate of 97.25%. Thisstrongly validates the effectiveness of the proposed
approach.

The recognition rates obtained using TMRIs, EXHOOF, TMRIs + EX-HOOF, and the proposed method(TMRIs + EX-HOOF + AC) are shown in Table 2.
As shown in the table, the proposed method achieved
an average recognition rate of 97.25%. This is because
TMRIs represent depth information determined by FoE
detection results. In addition, Ex-HOOF includes
movement speed and direction information, and thedetailed feature AC expresses changes in the area of the
motion region and the center of gravity. Therefore, in the
final proposed method, the average recognition rate fordaily activities is 99.1%, and the average recognition rate
for falling activities is 89.84%. Furthermore, the average
recognition accuracy for motions that include movementin the depth direction is 96.60%, and the average
recognition accuracy for other motions is 94.85%. The
average recognition accuracy of the proposed method
using TMRIs + EX-HOOF + AC is about 17.41% higher
than the method using only TMRIs.
Table 1. The result of behavior segmentation
Number of division points
Per- BehaOptical
Positivesonvior
Trajectory
Integration
flow
number
A
23
9
6
3
B
33
20
9
4
1
C
46
17
10
4
D
27
17
7
4
A
21
11
5
3
B
17
10
4
3
2
C
28
9
5
4
D
30
19
6
4
A
28
18
6
3
B
20
16
6
4
3
C
27
10
7
4
D
28
14
6
3
Average

In the future, our focus will be on achieving a highspeed and fully automated process of behavior
segmentation through recognition. Additionally, despite
the current good recall rate, there is still a need to enhance
the accuracy of the automatic segmentation method to
reduce the subsequent workload of recognition andsegmentation adjustments.
References

Recall[%]1.

United Nations, Department of Economic and Social
Affairs, Population Division, World Population Prospects
2022. (Online Edition).
2. Cabinet Office, 2022 White Paper on Aging Society(Overall version), 2022, pp.2-6. (in Japanese)
3. Cabinet Office, Outline of the 2022 Public Opinion Poll on
Security, 2022, pp.24. (in Japanese)
4. X. Yang, J. Cheng, X. Tang et al., CSI-based humanbehavior segmentation and recognition using commodity
Wi-Fi. J Wireless Com Network 2023(46), 2023.
5. W. Xing, W. Wang et al., A Novel Method for Automated
Human Behavior Segmentation. Apr. 2016.
6. E.L. Andrade, R.B. Fisher, and S. Blunsden, Detection of
emergency events in crowded scenes, Proceedings of IEEE
International Symposium on Imaging for Crime Detection
and Prevention, Hong Kong, China, 2006, pp.528-533.
7. A. Bobick, J. Davis, The recognition of human movement
using temporal templates, IEEE Transactions on PatternAnalysis and Machine Intelligence, vol.23(3), 2001,
pp.257-267.
8. Y. Yamashita, J.K. Tan, and S. Ishikawa, Human motion
description and recognition under arbitrary motion
direction, Proceedings of SICE Annual Conference 2017,
Kanazawa, Japan, 2017, pp.110-115.
9. J.K. Tan, S. Okae, Y. Yamashita, and Y. Ono, A methodof describing a self-occlusive motion – A reverse motion
history image, International Journal of Biomedical Soft
Computing and Human Sciences, vol.24(1), 2019, pp.1-7.
10. J. Cao, Y. Yamashita, J.K. Tan, Human motion recognition
using TMRIs with extended HOOF, Journal of Robotics,
Networking and Artificial Life, vol.7(4), 2021, pp.231-235.
11. B.D. Lucas, T. Kanade, An Iterative Image Registration
Technique with an Application to Stereo Vision, Proc. ofInt. Joint Conf. on Artificial Intelligence, 1981, pp.674679.
12. M. A. Fischer, R. C. Bolles, Random sample consensus: A
paradigm for model fitting with applications to image
analysis and automated cartography, 1981, pp.381-395.

100.0
100.0
100.0
100.0
75.0
75.0
100.0
100.0
75.0
100.0
100.0
75.0
91.7

Table 2. The recognition rate of the motions
Recognition rate [%]MotionWalk left
Walk right
Walk front
Walk back
Walk left front
Walk right front
Walk left rear
Walk right rear
Fall left
Fall right
Fall front
Fall rear
Average

TMRIs

ExHOOF

TMRIs+ ExHOOF

87.5
70.3
78.1
73.1
83.8
87.8
83.4
83.1
91.9
79.4
61.9
69.4
79.8

44.4
93.1
75.3
97.8
96.9
72.8
92.2
94.4
63.8
71.3
44.4
63.1
78.8

93.1
99.4
99.4
100.0
96.3
94.7
96.6
95.3
91.9
75.6
80.6
86.3
94.2

TMRIs
+ ExHOOF
+ AC
99.4
100.0
99.7
100.0
99.1
100.0
98.8
95.9
91.9
88.1
89.4
90.0
97.3

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
972

Powered by TCPDF (www.tcpdf.org)

Human Behavior Segmentation andAuthors Introduction
Ms. Jing Cao

She received B.E. from Republic of China
and M.E. from the Graduate School ofEngineering,
KyushuInstitute ofTechnology, Japan in 2020. She isacquiring the D.E. in the same University.
Her research interest includes computer
vision, machine learning and motion recognition.
Dr. Yui Tanjo
She is currently a professor with the
Department of Mechanical and Control
Engineering,
KyushuInstituteofTechnology. Her current research interests
include ego-motion analysis by MY
VISION, three-dimensional shape/motion
recovery, human detection, and its motion analysis fromvideo. She was awarded SICE Kyushu Branch Young
Author’s Award in 1999, the AROB Young Author’sAward in 2004, the Young Author’s Award from IPSJ of
Kyushu Branch in 2004, and the BMFSA Best Paper
Award in 2008, 2010, 2013 and 2015. She is a member ofIEEE, The Information Processing Society, The Instituteof Electronics, Information and Communication Engineersof Japan.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
973

Online Classroom Student Engagement Analysis using Enhanced YOLOv5
Shuai Wang

Institute of Computer Science and Digital Innovation, UCSI University, No.1, Jalan Menara Gading,
Cheras, Kuala Lumpur, Malaysia
Abdul Samad Shibghatullah

Institute of Computer Science and Digital Innovation, UCSI University, No.1, Jalan Menara Gading,
Cheras, Kuala Lumpur, Malaysia
Javid Iqbal

Institute of Computer Science and Digital Innovation, UCSI University, No.1, Jalan Menara Gading,
Cheras, Kuala Lumpur, Malaysia
Kay Hooi KeoyUCSI Graduate Business School, UCSI University, No.1, Jalan Menara Gading,
Cheras, Kuala Lumpur, Malaysia
E-mail: 1002268166@ucsiuniversity.edu.my,abdulsamad@ucsiuniversity.edu.my,javid@ucsiuniversity.edu.my,
keoykh@ucsiuniversity.edu.my
www.ucsiuniversity.edu.my

Abstract
The surge in online education has underscored the pressing issue of cyberbullying in virtual classrooms. This paper
introduces an inventive method for early cyberbullying detection by analyzing students' engagement and emotional
responses in online classrooms. The proposed SFER-YOLOv5 model, integrating Student Facial Expression
Recognition with an enhanced YOLOv5 object detection model, incorporates transformative optimizations. These
include Soft NMS for Non-Maximum Suppression, the integration of a Channel Attention (CA) module within the
YOLOv5 architecture, and the use of Enhanced Intersection over Union (EIOU) as the bounding box regression loss
function. This approach identifies diminished engagement and emotional irregularities, offering a proactive
framework for mitigating cyberbullying in online classrooms.
Keywords: Cyberbullying Mitigation, Emotional Analysis, Facial Expression Recognition, Online Classroom.

harassment or cyber aggression, refers to the malicious
use of digital technology to target others with acts ofaggression, insults, threats, and more. Particularly in
online classroom environments, both students andeducators may find themselves vulnerable tocyberbullying, which can severely impact their emotional
well-being, psychological state, and overall learning
experience.
In the online classroom environment, students andteachers communicate through screens, and teachers canonly judge students’ listening status based on their facial
expressions. Analyzing changes in students’ facial
expressions can help teachers better understand theirlistening status and make timely adjustments to the

1. Introduction
With the rapid advancement of information technology
and the global rise of online education, online teaching
platforms have become an integral part of modern
education. The sudden outbreak of the COVID-19
pandemic in 2020 has had a significant impact on ourlearning, work, and daily lives. Traditional face-to-faceteaching and communication between students andteachers have become a major challenge. Many
universities have chosen online teaching as an alternative.
However, accompanying this advancement are a host ofchallenges, one of which is the issue of cyberbullying in
the digital realm. Cyberbullying, also known as online©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
974

Shuai Wang, Abdul Samad Shibghatullah, Javid Iqbal Thirupattur, Kay Hooi Keoyteaching mode [1]. In 1971, American scholar Ekman etal. [2] conducted extensive experiments on facial
expressions, categorizing them into six primaryemotions: happiness, surprise, fear, sadness, disgust, andanger. Students’ emotions in class can be categorized as
positive, negative, or neutral. Positive emotions include
happiness and surprise. When students show positive
emotions, it indicates their willingness to accept theknowledge taught in class, and they are actively engaged
in listening and thinking. Negative emotions include
sadness, anger, fear, and disgust, which indicate thatstudents have a dislike or lack of attentiveness towards
the knowledge taught by the teacher. This can be anindication of their lack of concentration in class. Neutral
emotions suggest an average level of student engagement
in listening during class. Therefore, analyzing students’facial expressions in online classrooms holds greatsignificance.
Faced with this challenge, many educational institutions
and online teaching platforms are turning their attention
to how to effectively detect and address cyberbullying invirtual classrooms. Facial expression recognition
technology has garnered significant attention due to its
potential applications in emotion analysis and affective
state monitoring. Currently, there is limited research on
facial expression recognition based on YOLOv5. This
paper proposes a facial expression recognition method
based on an improved YOLOv5 model, aiming toachieve timely detection and intervention of
cyberbullying incidents in online classrooms. The
original YOLOv5 algorithm achieved an overall
recognition accuracy of 73.1% and 83.4% on the Fer2013
dataset and a self-constructed dataset for the three
expressions involved in the study: happiness, sadness,
and neutral. Through improvements in NMS, and the
addition of an attention mechanism module, theimproved YOLOv5 algorithm further enhances therecognition accuracy on the Fer2013 dataset and the selfconstructed dataset.

online harassment, such as verbal abuse, threats,
spreading false information, and exclusion from online
discussions, that occur within virtual classroom
environments. Cyberbullying can have detrimental
effects on students’ learning outcomes, psychologicalwell-being, and social relationships [5]. Therefore, it isimperative to develop effective strategies andmechanisms to detect and prevent cyberbullying
incidents in online classrooms. Previous research on this
topic has adopted various techniques and perspectives,
such as behavior analysis, natural language processing,social network analysis, machine learning, and emotion
recognition, to identify and intervene in potential
bullying situations that students may face in virtual
classrooms. Moreover, some studies have highlighted therole of educational interventions and awareness-raisingamong educators and students about cyberbullying,aiming to foster a safer and more positive online learning
environment [6].
2.2. YOLOv5 and object detection
In May 2020, UltralyticsLLC released YOLOv5 [7], alightweight model with an image inference speed of up
to 0.007 second, developed using python. It can process
140 FPS and can meet the real-time requirement forvideo sequences. The four network structures
Yolov5s\5m\5l\5x are basically same in principle andcontent, controlled respectively by the parameters ofwidth_multiple and depth_multiple in width and depth.
The size and accuracy of the four versions of the modelincrease sequentially. In practical applications, theappropriate size of the model can be selected based on
different scenarios. The YOLOv5s model has small depth
and fewer parameters, making it more applicable to realtime tasks in facial expression recognition due to its
faster inference speed compared to the other three
versions. Since the release of YOLOv5, its versions havebeen updated and iterated, and this paper is based on theversion 6.0 for improvement.
In summary, prior research has highlighted the urgencyof addressing online classroom cyberbullying and thepotential of facial expression recognition technology foremotion analysis. Additionally, the advancements in
object detection methods, particularly YOLOv5, provide
a solid foundation for integrating object detection into
cyberbullying detection frameworks. This study builds
upon these foundations to propose a novel model SFERYOLOv5 for detecting cyberbullying incidents in online

2. Related work
2.1. Online classroom cyberbullying
Online education has become increasingly prevalent inrecent years, especially due to the COVID-19 pandemic,
which has posed new challenges and opportunities foreducators and students. One of the major challenges iscyberbullying [3], [4], which refers to various forms of©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
975

Online Classroom Student Engagementclassrooms using an enhanced YOLOv5 model for facial
expression recognition.

study, the improved Soft NMS is used instead of theoriginal NMS, effectively improving the performance ofthe detection.

3. Methodology
3.2.2. Coordinate attention module

Currently, there is limited research on facial expression
recognition based on YOLOv5, and this method presents
a unique opportunity for enhancing online classroom
engagement and security. The original YOLOv5
algorithm achieved an overall recognition accuracy of
72% and 83.1% for the three facial expressions ofhappiness, sadness, and neutrality on the Fer2013 dataset
and the self-built dataset. Through improvements in the
non-maximum suppression (NMS) algorithm, loss
function, and the addition of an attention mechanism
module, the improved YOLOv5 algorithm achieved
further improvement in recognition accuracy on the
Fer2013 dataset and the self-built dataset.

In this study, we introduce the Coordinate Attention(CA) mechanism, which incorporates positional
information into channel attention. Experimental results
reveal that incorporating the CA attention mechanism
module into the network backbone improves detection
accuracy to some extent, particularly in classroom
settings with numerous students or when students in theback rows are at a considerable distance from the camera.
This enhancement is attributed to the CA module (Fig. 1),
which reinforces channel features in the feature map,
enabling the network to acquire more detailed andeffective information.

3.1. Data collection and processing
As students’ understanding of knowledge can bebroadly categorized into three levels: mastery, confusion,and non-mastery, in order to gain a more accurate
understanding of the facial expression feedback
corresponding to these three levels, we conducted asurvey in the form of a questionnaire to investigate therelationship between mastery levels and facial expression
feedback. The survey involved higher vocational schools,
and undergraduate institutions, with a total of 520
questionnaires collected (male: 287, female: 233). Seven
basic expressions were selected, including anger, surprise,
disgust, happiness, fear, sadness, and neutral.
This study used the Fer2013 dataset and a selfconstructed dataset of student facial expressions in theclassroom. The Fer2013 dataset contains some non-facial
images and mislabeled images. To improve theexperimental accuracy, the dataset was carefully selected,
cleaned, and filtered, resulting in a total of 11,000 images.
The self-constructed dataset of student facial expressions
in the classroom was collected through video monitoring,and it consists of 668 images. The images were annotatedby extracting frames from the video footage.

Fig. 1 The improved network structure.
3.2.3. Improved Loss function
We have improved the initial loss function byincorporating the Efficient-IOU (EIOU) to computewidth and height losses, replacing the aspect ratio
calculation based on the CIOU foundation. Furthermore,
Focal Loss has been introduced concurrently to mitigate
the challenge posed by sample imbalance.
4. Experiments and results
The experimental platform in this study was a 64-bit
Windows 10 operating system with a Gen Intel Core i511400H CPU and NVIDIA RTX 3050 graphics card. The
deep learning framework used was PyTorch, with adevelopment environment consisting of PyTorch 1.8 andPython 3.7. The number of training epochs was set to 300,
and the dataset was divided into training, validation, and3.2. Enhanced YOLOv5
3.2.1. Improved Non-Maximum Suppression (NMS)
Both NMS and Soft NMS utilize predictedclassification confidence as a measure, where higher
confidence indicates more accurate localization. In this©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
976

Shuai Wang, Abdul Samad Shibghatullah, Javid Iqbal Thirupattur, Kay Hooi Keoytesting sets following a 6:2:2 ratio. To effectively
evaluate the experimental results, a comparative
experiment was conducted on the self-constructed dataset
and the Fer2013 dataset before and after algorithm
improvement. The evaluation criteria were detection
accuracy (P) and mean average precision (mAP@0.5).
Table 1 presents the experimental outcomes on our
self-constructed dataset. Fig.2 illustrates the detection
accuracy (P) curve of the self-constructed dataset,comparing YOLOv5 with SFER-YOLOv5.
Table 1. Experimental results on self-constructed dataset.
Expression

YOLOv5

SFER-YOLOv5

Fig.3 P curve on the FER-2013 dataset.

P

mAP@0.5P

mAP@0.5

happy

0.816

0.853

0.851

0.887

sad

0.845

0.867

0.889

0.909

5. Conclusion

neutral

0.842

0.857

0.879

0.871

all

0.834

0.859

0.87

0.889

This study addresses the challenges of low real-time
detection rates and poor timeliness in recognizing
students' facial expressions in complex classroom
environments while simultaneously focusing on
mitigating and preventing cyberbullying incidents. Our
SFER-YOLOv5 model improves the non-maximum
suppression (NMS) by replacing it with Soft NMS,
enhances the feature extraction capability by adding theCoordinate Attention (CA) mechanism module, andimproves the representation ability of target boxes byusing EIoU instead of CIoU. Our experimental results arehighly encouraging, demonstrating a 3.9% increase indetection accuracy (P) and a 3.0% increase in mAP@0.5
on the self-constructed dataset. Beyond facial
expressions, future research will integrate poses andgestures for a comprehensive analysis of online
classroom dynamics. This enhances our cyberbullying
prevention by understanding students' engagement andemotions more comprehensively.

Fig.2 P curve on the self-constructed dataset.
Table 2 presents the experimental results on the Fer2013 dataset. Fig. 3 illustrates the P curve of the Fer-2013
dataset.

References

Table 2. Experimental results on FER-2013 dataset.
YOLOv5

1.

SFER-YOLOv5

ExpressionP

mAP@0.5P

mAP@0.5

happy

0.863

0.918

0.876

0.911

sad

0.636

0.594

0.669

0.626

neutral

0.694

0.799

0.722

0.515

all

0.731

0.770

0.756

0.784

2.

3.

J. Bao, X. Tao, Y. Zhou, "An Emotion Recognition
Method Based on Eye Movement and Audiovisual
Features in MOOC Learning Environment", IEEE
Transactions on Computational Social Systems, 2022.
P. Ekman, W.V. Friesen, "Constants across cultures in theface and emotion", Journal of personality and social
psychology, 17.2 (1971), 124.
N. Shakee, R. K. Dwivedi, "A survey on detection of
cyberbullying in social media using machine learningtechniques", Intelligent Communication Technologies andVirtual Mobile Networks: Proceedings of ICICV 2022,
Singapore: Springer Nature, pp. 323-340, 2022.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
977

Powered by TCPDF (www.tcpdf.org)

Online Classroom Student Engagement

4.

5.

6.

7.

A. Bozyiğit, S. Utku, E. Nasibov, "Cyberbullying
detection: Utilizing social media features", Expert
Systems with Applications, 179: 115001, 2021.
R. Kumar, A. Bhat, "A study of machine learning-basedmodels for detection, control, and mitigation of
cyberbullying in online social media", International
Journal of Information Security, 21(6), 1409-1431, 2022.
G.W. Giumetti, R.M. Kowalski, "Cyberbullying via social
media
and
well-being", Current
Opinionin
Psychology, 45, 101314, 2022.
X. Zhu, et al, "TPH-YOLOv5: Improved YOLOv5 basedon transformer prediction head for object detection ondrone-captured scenarios", Proceedings of the IEEE/CVF
international conference on computer vision, pp. 27782788, 2021.

Authors Introduction
Mr. Shuai Wang
He is currently pursuing the Ph.D. degree in
Computer Science programme with theInstitute of Computer Science and Digital
Innovation, UCSI University, Kuala
Lumpur, Malaysia. He is also working in
the Department of Physics and Electronic
Engineering, Yuncheng University, Yuncheng, Shanxi,
China. His current research interests include computer
vision, machine learning and multi-modal learning.
Dr. Abdul Samad Bin Shibghatullah
He received the Ph.D. degree in computerscience from the Brunel University of
Uxbridge, United Kingdom. He iscurrently an Associate Professor at
Institute of Computer Science & Digital
Innovation, UCSI University, Kuala
Lumpur, Malaysia. His current research interests include
optimization, modelling and scheduling, Robotics.
Dr. Javid Iqbal Thirupattur
He works as an Assistant professor at
UCSI University, Malaysia. He holds a
Ph.D. degree in Information andCommunication Technology from the
National Energy University Malaysia. His
research interests include Multimedia,
Augmented Reality, and Virtual Reality.
Dr. Keoy Kay Hooi
He received the Ph.D. degree at Sheffield
Hallam University, United Kingdom in
2006. He is currently an Associate
Professor, UCSI University, Kuala Lumpur,
Malaysia. His current research interests
include machine Learning, block chain
technology & management information
system.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
978

A Method of Recognizing Body Movements Based on a Self-viewpoint Video
Iichirou Moribe

Graduate School of Engineering, Kyushu Institute of Technology, 1-1 Sensui, Tobata-ku, Kitakyushu, 804-8550, Japan
Yui Tanjo

Faculty of Engineering, Kyushu Institute of Technology, 1-1 Sensui, Tobata-ku, Kitakyushu, 804-8550, Japan
Email: tanjo@.cntl.kyutech.ac.jp

Abstract
The most critical human sensory function resides in vision. This paper focuses on utilizing visual information,
specifically self-perspective footage, to identify individual movements. Existing researches require third-party filming
to recognize human body movements and states. The proposed method, on the other hand, simply attaches a camera
to the human head and enables the recognition of the subject's actions. Consequently, it becomes easier to monitor
daily movements of a human and gather his/her data on body kinetics. This approach would be beneficial in scenarios
involving individuals engaging in risky behavior or, during a certain emergency, providing valuable assistance.
Keywords: My VISION, Posture estimation, Optical flow, HSV conversion

1. Introduction

2. Methodology

Vision is the most crucial function among humansensory organs. Human intakes a vast amount of
information through vision. Therefore, self-perspective
footage can potentially serve as the primary source of
information from an individual. In fact, it has been
evident that visual information is more potent inperceiving human posture and bodily movements
compared to non-visual information [1]. Therefore, it wasconsidered possible to recognize the filmmaker's physicalstate by analyzing self-perspective footage.
Maintaining physical health has become morecommonplace, exemplified by the widespread use ofsmartwatches. Smartwatches offer various functions suchas recording heart rate, blood oxygen levels, and sleep
patterns. By utilizing these features to log daily physicalconditions, it becomes possible to promptly recognize
deviations from normal bodily states.
Hence, this study aims to develop a method foranalyzing and recording an individual's activities from aself-perspective video. Similar research includes studies
on self-posture estimation [2], methods utilizing singleeye images based on HOG features [3], and methods
employing deep learning [4]. However, these methods
focus on action estimation or recognition rather thanunderstanding a person's activity. Additionally, thetechniques using HOG-based single-eye images or
employing deep learning require capturing individuals
externally, which is entirely different from the selfperspective video approach in this paper.

This section describes feature extraction methods. The
features to be extracted are the values of the hue and thenorm of the optical flow on an image.
2.1. Detection of area similarity
This subsection describes the method for detecting
area similarity. By dividing the image into multiple
blocks and comparing the H (Hue) histogram features of
specific blocks, area similarity is determined.
2.1.1 HSV Conversion
First, the RGB color image, which serves as the input
image, is converted from the RGB color space to the HSVcolor space. The RGB color space represents a color
space with red, green, and blue as coordinate axes, while
the HSV color space represents a color space with hue,saturation, and value as coordinate axes.
2.1.2 Creation of H(Hue) histogram
In the proposed method, a histogram based on the Hue(H) is constructed. The values of S (Saturation) and V(Value) are not taken into account. The H histogram, as
depicted in Fig. 1, represents the hue values on thehorizontal axis and the frequency count on the vertical©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
979

Iichirou Moribe, Yui Tanjo

axis. Although the original range of hue values is 1 to 360,
for expedited processing, it is scaled to 1 to 180.

2.2.2 Feature point matching
The Lucas-Kanade method is used for matching
feature points [6]. The Lucas-Kanade method is one of
the leading methods for deriving optical flow and iscomputationally less expensive than the method thatfinds all pixels in the image by searching for them.

50000

25000

0
1

180

2.2.3 Removal of outliersFig. 1. H (Hue) Histogram
The optical flow is obtained by the feature point
matching, but the optical flow obtained from the actual
video contains many outliers. Outliers are false flows
obtained by matching different feature points and need tobe removed for accurate analysis. Outliers are removed
by applying RANSAC to modelling with homographymatrices.

2.1.3 Calculation of similarity
The H histogram is used to determine the similarity
between two images. Here, the similarity is calculatedusing the following formula. The meanings of thecharacters in Eq. (1) are given in Table 1.
𝑆 (𝐴, 𝐵) =𝐴
𝐵
∑𝑁
𝑖=0 min(ℎ𝑖 , ℎ𝑖 )𝑁𝐴∑𝑖=0 ℎ𝑖

An example of the optical flow obtained by the above
process is shown in Fig. 3. Note that the green points inthe figure represent the end points of the optical flow.

(1)

Table 1. Meanings of the characters in Eq. (1)N
Number of pixels in the imageℎ𝑖𝑥
The 𝑖th element of histogram 𝑥
min(𝑥𝐴 , 𝑥𝐵 )Minimum value of 𝑥𝐴 𝑎𝑛𝑑 𝑥𝐵
2.2 Derivation of optical flow
This subsection describes the derivation of optical flow.
The optical flow is the movement vector of the camera,or the subject obtained by mapping feature points in two
consecutive frame images extracted from the video.

Fig. 3. Derived optical flow.
2.3 Normal state recognition

2.2.1

Feature point extraction and description
In this subsection, the method of normal state
recognition is first described. Next, the methods forclassifying states are described.

With respect to two consecutive frame images, thefeature points in the first frame image are focused on.
After extracting the feature points, the feature values ofeach feature point are described, and the feature points
are matched in the two frame images. The Shi-Tomasi
corner detection method is used for feature point
extraction and feature point description [5]. The image
from which the feature points were extracted is shown in
Fig. 2.

2.3.1

Feature extraction methods

The input image is separated into nine blocks and eachblock image is assigned a number as shown in Fig. 4.
Focusing on blocks 2, 5 and 8, the three block images are
HSV-transformed and H histograms are created. The H
histograms are compared to obtain the similarity between
block 8 and block 2, and between block 8 and block 5,
respectively. The similarity between block 8 and block 2
and between block 8 and block 5 is calculated bycomparing the H histograms.
Block 1

Block 2Block 3Block 4Block 5Block 6Block 7Block 8Block 9Fig. 4. Separated input image

Fig. 2. Input image with feature point extraction©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
980

A Method of Recognizing

From these results, it can be read that 𝑆̅ is greater in the'looking down' state among the three states. This meansthat in the 'looking down' state, the similarity of block 2,
block 5 and block 8 is greater. It can also be seen that
𝑉𝑚𝑎𝑥 in the 'stumble' state is the largest among the threestates. This indicates that the motion in the 'stumble' stateis the most intense.

Furthermore, in the method, the number and norm of
optical flows are the key features. First, the average valueof the norm of the optical flow is calculated with eachframe, and then the maximum value of the norm in thewhole video is calculated. This value is used to evaluate
the intensity of the motion of the entire video. The
average value of the norm of the optical flow is calculatedby the following equation. The meanings of thecharacters in Eq. (2) are shown in Table 2.

2.3.3

State analysis

𝑁𝑓

𝑉𝑓 =

1
∑ 𝑉𝑖
𝑁𝑓

In the proposed method, the motion states are classified
into three states. Classification 1: If the value of 𝑆
satisfies Eq. (3), the image is classified as a 'looking
down' image. If the value of 𝑆 does not satisfy Eq. (3), theimage is classified as 'other' image. 'other' can be either'normal' or 'stumble'. The state of Classification 1 isdetermined using the following formula.

(2)

𝑖=1

Table 2. Meanings of the characters in Eq. (2)Total number of optical flows obtained𝑁𝑓between the 𝑓th frame and the f+1th frame
Norm of each optical flow
𝑉𝑖
2.3.2

𝑆 − 𝑆̅ ≥ 𝑇𝐻1

Prior statistics(3)

Here 𝑆 is the similarity in the current state and 𝑆̅ is theaverage of the similarity in the normal state. 𝑇𝐻1 is thethreshold value in Classification 1.

Since the estimation of each state is done bycomparison with the normal state, it is necessary to know
in advance what properties each state has in relation to
the normal state. For the three states, feature extraction iscarried out using the above procedure with the threevideos.

𝑆𝑟 is the proportion of the images in the image groupthat are classified as 'looking down' in Classification 1,
and if the value of 𝑆𝑟 satisfies Eq. (4), the image isclassified as "looking down". If the value does not satisfy
Eq. (4), it is assumed to be in the 'other' state and proceeds
to the next state estimation. If the value of 𝑉𝑚𝑎𝑥 issatisfies Eq. (5), the state is judged as the 'stumble' state.
If none of the above conditions is satisfied, the state is
judged as normal. Eq. (4) and Eq. (5) are shown below.

The results of the validation of the properties of thenormal state are presented in Table 3. The results of thevalidation of the properties of the 'looking down' state arepresented in Table 4. The results of the validation of theproperties of the 'stumble' state are presented in Table 5.
Table 3. Results of the validation of the properties of
the normal state
Scene
𝑉𝑚𝑎𝑥
𝑆̅
Video 1
0.53
118Video 2
0.40
109Video 30.51
124

𝑆𝑟 − 𝑆̅𝑟 ≥ 𝑇𝐻2(4)

𝑉𝑚𝑎𝑥 − ̅̅̅̅̅̅
𝑉𝑚𝑎𝑥 ≥ 𝑇𝐻3(5)

Here 𝑆𝑟 and 𝑉𝑚𝑎𝑥 are the evaluated values of similarity
and intensity of motion in the current condition,
respectively. 𝑆𝑟̅ and ̅̅̅̅̅̅
𝑉𝑚𝑎𝑥 are the average values ofsimilarity and intensity of motion, respectively, in thenormal condition. 𝑇𝐻2 and 𝑇𝐻3 are the threshold values
for each condition.

Table 4. Results of the validation of the properties of
the 'looking down' state
Scene
𝑉𝑚𝑎𝑥
𝑆̅
Video 1
1.29
113Video 2
1.19
110Video 3
1.06
75.6

3. Experimental Result
3.1. Experimental methods

Table 5. Results of the validation of the properties of the'stumble' state
Scene
𝑉𝑚𝑎𝑥
𝑆̅
Video 1
0.61
271Video 2
0.54
181
Video 30.45
194

An experiment is conducted with a camera worn on thehead. Fig. 5 shows an image of the camera worn by asubject during the experiment.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
981

Powered by TCPDF (www.tcpdf.org)

Iichirou Moribe, Yui Tanjo

References
1.

2.
(a)
(b)
Fig. 5. Image of a worn camera. (a) side view,(b) rear
view.

1.

2.

The details of the experimental method are as follows:3.

Feature extraction was carried out from five videos
in the 'normal' state, and a database was created byaveraging the feature values in each video.
Eight videos containing each of the three states 'normal', 'looking down' and 'stumbling' - taken at
different locations and at different dates and times
were used as input videos to estimate each of thestates.

4.

5.

The parameters during the experiment are shown in
Table 6.
Table 6. Parameters’ values
Value
Threshold 1
0.35
𝑇𝐻1 in Eq. (2-2)
Threshold 2
0.65
𝑇𝐻2 in Eq. (2-3)
Threshold 355
𝑇𝐻3 in Eq. (2-4)

6.

D. Lee, E. Aronson: “Visual proprioceptive control ofstanding in human infants” ，Attention Perception &Psychophysics，vol. 15, No.3, pp. 529-532，1974．
J. K. Tan, T. Kurosaki: “Estimation of self-postureof a pedestrian using MY VISION based on depth
and motion network” ， Journal of Robotics ，Networking and Artificial Life，vol. 7，No. 3，
pp. 152-155，2020.
K. Onishi, T. Takiguchi, Y. Ariki: “3D human posture
estimation using the HOG features from monocular
image” ， 19th International Conference on pattern
Recognition，pp. 1-4，2008．
Z. Cao, T. Simon, S. E. Wei, Y. Sheikh: “Realtime
multi-Person 2D pose estimation using partaffinity fields” ， Proceedings of the IEEE
Conference on Computer Vision and Pattern
Recognition，pp. 7291-7299，2017．
J. Shi, C. Tomasi: “Good features to track” ，IEEE Conference on Computer Vision and Pattern
Recognition，1994．
B. D. Lucas, T. Kanade: “An iterative image
registration technique with an application to stereo
vision” ，Proceeding of Imaging Understanding
Workshop，pp.121-130，1981．
Authors Introduction

3.2. Result
Mr. Iichirou Moribe
Mr. Moribe received his Bachelor’s
degree in Engineering in 2022 from theControl Engineering, Kyushu Institute oftechnology in Japan. He is currently amaster student in Kyushu Institute ofTechnology, Japan.

The results of the experiment are shown in Table 7. For
each of the eight videos, the number of correct responses
for the 'normal', 'looking down' and 'stumble' states were6, 7 and 7 respectively.
Table 7. Results of the experiment.
statenormal looking down stumble
normal6
02
looking down
1
70
stumble1
0
7

Dr. Yui Tanjo
Dr. Tanjo is currently a professor with
the Department of Mechanical andControl Engineering, Kyushu Institute ofTechnology. Her current researchinterests include ego-motion analysis by
MY
VISION,
three-dimensional
shape/motion
recovery,humandetection, and its motion analysis from video. She wasawarded SICE Kyushu Branch Young Author’s Award in
1999, the AROB Young Author’s Award in 2004, theYoung Author’s Award from IPSJ of Kyushu Branch in
2004, and the BMFSA Best Paper Award in 2008, 2010,2013 and 2015. She is a member of IEEE, The Information
Processing Society, The Institute of Electronics,
Information and Communication Engineers of Japan.

4. Conclusion
In this paper, we proposed a method for estimating theactivity state of the camera wearer from the selfviewpoint video obtained using a self-viewpoint camera.
Three human walk states were detected by comparing H(Hue) histograms and using features calculated from theoptical flow. In the experiment, three states weredetected: 'normal', 'looking down' and 'stumble' states.
The results showed that estimation was successful inmore than 75% of all states. However, in the estimationof all states, misrecognition occurred in which a different
state was estimated. Future issues include the estimationof states in case of sudden changes in vision andestimating more precise states corresponding to the angleof looking down.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
982

Supporting Safe Walk of a Visually Impaired Person at a Station Platform Based on MY
VISION
Shintaro Yamada

Graduate School of Engineering, Kyushu Institute of Technology, 1-1 Sensui, Tobata-ku, Kitakyushu, 804-8550, Japan
Yui Tanjo, Seiji Isihkawa*

Faculty of Engineering, Kyushu Institute of Technology, 1-1 Sensui, Tobata-ku, Kitakyushu, 804-8550, Japan*Emeritus Professor of Kyushu Institute of Technology
Email: tanjo@cntl.kyutech.ac.jp

Abstract
When individuals with visual impairment go out, public transportation such as trains and buses is commonly used.
However, many of them experience accidents, such as falling from train platforms or tripping due to unexpectedcontact with other passengers. To solve this problem, we propose a method using the MY VISION system which
detects obstacles that may pose risks to individuals with visual impairment. The proposed method detects obstacles
such as passengers’ pillars and platform edges at train stations. We employ an RGB-D camera for capturing frontal
views of a user, use depth images to detect the edge of obstacles and level differences, and give warning to the visually
impaired user based on the distance between him/her and the detected obstacle. Experimental results show satisfactory
performance of the method.
Keywords: Visually impaired, Safety, Train station platform, Obstacles detection, MY VISION

2. Method

1. Introduction

2.1 Obstacles detection

Public transportation is often used as a means ofmobility for individuals with visual impairments, andmore than half of them frequently utilize railways [1].
However, approximately 30% of visually impaired
individuals have experienced falls from station platforms,
resulting in an average of 76 fall incidents annually [2].

Initially, the coordinates conversion is performed fromthe image coordinate system to the camera image
coordinate system. The transformation formula is given
as follows.

There are methods for detecting steps and obstacles toassist visually impaired individuals in walking [3], [4].
However, the methods that use depth images to acquire
the walking plane and to detect steps and obstacles areaffected by noise, as the distance from the camera
increases, it results in a narrow and less accurate
detection range. Additionally, some methods use color
information, but they are often designed for indoor use
and are influenced by the factors such as time of day andweather when used outdoors.

(𝑥 − 𝑐𝑥 ) × Z(x, y)𝑓𝑥(1)(𝑦 − 𝑐𝑥 ) × Z(x, y)𝑓𝑦(2)

𝑍(𝑥,𝑦)=𝑑(3)

𝑋(𝑥, 𝑦) =Y(x, y)=Here 𝑐𝑥 is the optical center in the x-axis direction, 𝑐𝑦 is
the optical center in the 𝑦-axis direction, 𝑓𝑥 is the focal
length in the x-axis direction, and 𝑓𝑦 is the focal length in
the 𝑦-axis direction.

In this paper, we propose a method that detects stepedges directly using depth images and determines the
distance from the pedestrian to the step. The proposed
method, designed to assist visually impaired individuals
on station platforms, involves region segmentation using
Graph Based Segmentation on depth images and thedetection of step lines using Line Segment Detector.

Changes in Y(x,y) and 𝑍(x,y) with respect to the change
in the y-coordinate, as well as the changes in 𝑋(x,y) and𝑍(x,y) with respect to the change in the x-coordinate, areexpressed by the following equations.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
983

Shintoro Yamada, Yui Tanjo, Seiji IshikawadY(x,y)=Y(x,y)-Y(x,y+k)dy(4)dZ(x,y)=Z(x,y)-Z(x,y+k)dy(5)d𝑋(x,y)=𝑋(x,y)-𝑋(x+k,y)d𝑥dZ(x,y)=Z(x,y)-Z(x+k,y)d𝑥segments at the edges of planes with different heights.
Linear approximation is performed using RANSAC on
the endpoints of the line segments detected by LSD todetect the edges of the train platform. The detected lines
are represented by the equation 𝑦 = 𝑎𝑥 + 𝑏 using thecoordinates (x, y), and the gradient a helps determine thedirection of the platform edge from the perspective of theobserver. The conditional statements are as follows.

(6)

𝑅𝑖𝑔ℎ𝑡 𝑖𝑓 𝑎 > 𝑡ℎ1
𝐷𝑖𝑟 = { 𝐿𝑒𝑓𝑡 𝑖𝑓 𝑎 < 𝑡ℎ2
𝐹𝑟𝑜𝑛𝑡
𝑒𝑙𝑠𝑒

(7)

The method computes the change in angle of thegradient of the Z coordinate with respect to the gradientof the Y coordinate at the point (x,y), as well as the change
in angle of the gradient of the Z coordinate with respect
to the gradient of the X coordinate. If these angles exceed
a certain threshold, the method judges it as an obstacle
region. The formula and conditional statement are shownas follows.

dY(x,y)dy)d_YZ(x,y)=arctan (dZ(x,y)dy(8)d𝑋(x,y)d_𝑋Z(x,y)=arctan ( d𝑥 )dZ(x,y)d𝑥(9)

2.4 Distance from a step edge
Let 𝑔𝑐 be the point where the user of the proposed
method is standing. It is defined as 𝑔𝑐 = (width/2, height).
Let 𝑔𝑝 be a point on the detected line segment. Referringto the depth information on the left and the right of thepoint 𝑔𝑝 which is the nearest point from 𝑔𝑐 , the𝐻
distance𝐷𝑚𝑖𝑛
is defined as the distance from the platform
edge to the user. It is formulated as follows.

𝐷=min (|𝑔𝑐 − 𝑔𝑝 |)𝑝=1,2,…,𝑝(13)𝑑(𝑥, 𝑦, 𝑑𝑒 ) = 𝐷

(14)

𝐻
𝐷𝑚𝑖𝑛
= 𝑚𝑖𝑛 (𝑑(𝑥 + 1, 𝑦, 𝑑𝑒 ∗ ),𝑑𝑒∗d𝑋(x,y)d_Y𝑋(x,y)=arctan ( d𝑥 )dY(x,y)d𝑦(10)

𝑜𝑏𝑗𝑒𝑐𝑡1 𝑖𝑓𝑑𝑌𝑍 > 𝑡 𝑎𝑛𝑑 𝑑𝑋𝑍 > 𝑡ℎ 𝑎𝑛𝑑 𝑑𝑌𝑋 = 0={0𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒

(11)

(12)

𝑑(𝑥 − 1, 𝑦, 𝑑𝑒 ∗ ))(15)

3. Experimental result
In the performed experiment, an RGB-D camera wasmounted on the chest of a subject, and video taking was
conducted at a train platform. The proposed method wasapplied to the captured depth images, and comparison
with ground truth was performed to assess the accuracyof the method. Experimental videos were captured in foursituations involving obstacles such as people and luggage.
Each video consists of 95 frames. Accuracy evaluation isdone using the following formula.

2.2 Step detection at a station platform
The method designates the regions at both ends of thearea, excluding the obstacle region as candidate regions
where steps are present. Graph Based Segmentation(GBS) is applied to these candidate regions. By applying
GBS, it is possible to segment and merge regions withsimilar features in the image. This allows for thedetection of steps by dividing regions according todifferent planes of varying height.

𝒐𝒗𝒆𝒓𝒍𝒂𝒑 =𝐺𝑇 ∩ 𝑂𝐴>𝑇
𝑂𝐴

𝒂𝒏𝒈𝒖𝒍𝒂𝒓 𝒆𝒓𝒓𝒐𝒓 = |𝑎𝐺𝑇 − 𝑎𝑑 | < 𝑇𝑞𝑎(17)𝒅𝒊𝒔𝒕𝒂𝒏𝒄𝒆 𝒆𝒓𝒓𝒐𝒓 = |𝑏𝐺𝑇 − 𝑏𝑑 | < 𝑇𝑞𝑏(18)

2.3 Line detection at a step edge
Line Segment Detector (LSD) is then applied to thecandidate regions which contain the steps detected
through region segmentation. LSD detects candidate line

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
984

(16)

Supporting Safe Walk of

(b)

(a)

(d)

(c)

(e)

Fig.1. Experimental result : (a) Original color image, (b) depth image, (c) ground truth, (d) obstacle
detection, (e) result of platform edge detection（in blue color）

Table 1. Experimental result

Location

Obstacle

Percentage of correct line
segment location [%]Percentage of correct
angles [%]Percentage of correct
distance [%]The left platform edge

Person

100

94.5

98.6

The right platform edge

Person

100

91.0

98.0

The left platform edge

Person & luggage

100

88.0

97.0

The right platform edge

Person & luggage

98.1

92.7

89.0

Here 𝐺𝑇 represents the area of the ground truth line
segment, 𝑂𝐴 represents the area of the line segment
detected by the proposed method, and T is the threshold.
In this experiment, 𝑇 is set to 0.5. The success of thedetection is determined when the ratio of overlapping line
segment areas exceeds the threshold. 𝑎𝐺𝑇 represents the
distance on the image between the midpoint of the groundtruth line segment and 𝑔𝑐 , whereas 𝑎𝑑 represents the
distance on the image between the midpoint of thedetected line segment and 𝑔𝑐 . 𝑇𝑞𝑎 is the threshold. In thisexperiment, 𝑇𝑞𝑎 is set to 15 pixels. The success of thedetection is declared when the error between 𝑎𝐺𝑇 and 𝑎𝑑
is below the threshold. 𝑏𝐺𝑇 represents the angle between
the ground truth line segment and the x-axis, and𝑏𝑑 represents the angle between the detected line segment
and the x-axis. 𝑇𝑞𝑏 is the threshold. In this experiment,
𝑇𝑞𝑏 is set to 5 degrees. The success of the detection isdeclared when the error between 𝑏𝐺𝑇 and 𝑏𝑑 is below thethreshold. An example of experimental results is shownin Fig. 1.

mounted RGB-D camera. This camera captures frontal
scenes of a user with visual impairment, when he/she iswalking. By detecting platform edges, the proposed
method is able to contribute to the safety of a user withvisual impairment who takes a train. Future work
includes improving the proposed method for real-timeprocessing and conducting validation experiments.

References
1.

2.

3.

The overall experimental results are presented in Table
1. The average accuracy for line segment position, line
segment angle, and line segment distance were 99.5%,
91.6%, and 95.6%, respectively.
4.

4. Conclusion
This paper proposed a method of detecting the edgesof steps on a train platform automatically using a chest-

5.

Japan Federation of the Blind and Visually Impaired,
Mainichi Shimbun/Japan Federation of the Blind
Questionnaire Survey on Railway Stations for the Visually
Impaired Survey Results, http://nichimou.org/wpcontent/uploads/2017/03/mainichiannke-to.pdf, (2022-98)
Ministry of Land, Infrastructure, Transport and Tourism,
Study Group for Safety Improvement at Station
Platforms
www.milt.go.jp/tetudo/tetudo_fr7_000015.html,(2022-98)
K. Imai, I. Kitahara, and Y. Kameda, Detecting walkable
plane areas by using RGB-D camera and accelerometer forvisually impaired people, 2017 3DTV Conference, The
True Vision - Capture, Transmission and Display of 3D
Video (3DTV-CON), Copenhagen, Denmark, pp. 1-4,
2017.
Yuna Seibu, Haiyuan Wu, Tadayoshi Shioyama,
Fundamental Study of Image-Based Solo Walking System
for the Visually Impaired - Detection of Platform Edge,
Human Interface News and Report, 13, pp.467-474, 1998.
Yuki Kawaguchi, Joo Kooi Tan, Seiji Ishikawa, Takashi©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
985

Powered by TCPDF (www.tcpdf.org)

Shintoro Yamada, Yui Tanjo, Seiji Ishikawa

Shinomiya, Supporting Safe Walk of a Visually Impaired
Person at a Railway Station Platform Based on MY
VISION, The 2022 International Conference on Artificial
Life and Robotics(ICAROB2022), pp. 447-450, 2022.

Authors Introduction
Mr. Shintaro Yamada
Mr. Yamada received his B.E. degree in
Department of Control Engineering in
2022 from the Faculty of Engineering,
Kyushu Institute of Technology in Japan.
He is acquiring the M.E. in Kyushu
Institute of Technology.
Dr. Yui Tanjo
Dr. Tanjo is currently a professor with the
Department of Mechanical and Control
Engineering,
KyushuInstituteofTechnology. Her current research interests
include ego-motion analysis by MY
VISION, three-dimensional shape/motion
recovery, human detection, and its motion
analysis from video. She was awarded SICE Kyushu
Branch Young Author’s Award in 1999, the AROB Young
Author’s Award in 2004, the Young Author’s Award fromIPSJ of Kyushu Branch in 2004, and the BMFSA Best
Paper Award in 2008, 2010, 2013 and 2015. She is amember of IEEE, The Information Processing Society, The
Institute of Electronics, Information and Communication
Engineers of Japan.
Prof. Seiji Ishikawa
Professor Ishikawa graduated from Tokyo
University and was awarded BE, ME and
PhD there. He is now Emeritus Professorof Kyushu Institute of Technology. He
was Visiting Researcher of the Universityof Sheffield, UK, and Visiting Professor of
Utrecht University, NL. His researchinterests include visual sensing & 3-D shape/motion
recovery. He was awarded The Best Paper Award in 2008,2010, 2013 and 2015

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
986

Development and evaluation of a learning support robot for vector learning
Kosei Machida

Tokyo Gakugei University, 4-1-1 Nukuikita Koganei, Tokyo, 184-8501, Japan
Shinichi Imai

Tokyo Gakugei University, 4-1-1 Nukuikita Koganei, Tokyo, 184-8501, Japan
Email: m224203x@st.u-gakugei.ac.jp, shimai@u-gakugei.ac.jp

Abstract
Vectors studied in high school are a new concept that differs from those studied in previous scalars. Therefore, it has
been pointed out that difficulties arise in the conceptual formation of vectors. This study aims to develop a learning
support robot that can visualize vector information by robot movements, and to acquire vector concepts for students
through classroom practice using the robot. The robot operates on a piece of imitation paper, marking the start andend points and connecting them with arrows to visualize the vectors. In a class using the robot, the student predicts
the sum of the vectors, and the robot confirms it. A questionnaire was administered before and after the class. The
results revealed significantly higher mean values when comparing the pre- and post-assessments.
Keywords: Educational Support, Robot, Omni-wheel, Omnidirectional movement, High school mathematics

It has been noted that the use of robots in schools canhave a positive impact on students. The use of robots, aspart of STEM education, has been noted to have apositive impact on students' understanding of science,technology, engineering, and mathematics; students
acquire a broader range of knowledge because robots areassociated with technological applications that span
multiple disciplines; and the use of robots can increase
students' motivation and engagement in the classroom.
The use of robots can have a positive impact on students
by increasing their motivation and engagement in theclassroom. [3], [4], [5], [6]1. Introduction
Recent large-scale surveys such as OECD-PISA and
TIMMS have shown that "while there has been some
improvement in the percentage of students who
responded positively to the enjoyment of learning
mathematics and its relationship to the real world, there
are still issues in terms of motivation to learn, such as thefact that the percentage is still low compared to othercountries" and in upper secondary schools, "themotivation to learn mathematics is not high. [1] Therefore,
in the Guidelines for the Course of Study for Senior HighSchools announced in 2018, it is stated that in senior highschool mathematics "problems that reflect the real world
are handled, students who are not highly motivated to
study mathematics are made aware of the significance ofstudying mathematics, and this will lead to increased
motivation and the development of mathematical ability.
This will lead to the development of mathematical skills.

Therefore, the purpose of this study is to develop alearning support robot for learning vectors and to proposea class in which students can acquire the concept ofvectors in addition to numerical calculations. By using
the robot, students can learn vectors not only on theblackboard but also through concrete objects, which is
expected to enhance their conceptual understanding.
Through the proposed class, we expect to achieve thelearning effects of understanding the concept of vectors
and improving the motivation to learn.

"Vectors" are "quantities that have magnitude anddirection," a new concept that is different from the"quantities that express only magnitude" that students
have learned in the past. Therefore, various difficulties
have been pointed out in the concept formation of vectors.
One reason for this is that vector learning is conducted
only at the level of abstraction on the blackboard, making
it difficult to formulate concepts. In addition, lack ofunderstanding of the phases of vectors has also been
pointed out [2], which may be caused by a lack ofunderstanding of the concept of vectors themselves.

2. Robots to be developed.
2.1. Definition of robot requirements
The robot is used to consider the problem of two vectors
with respect to the sum of the vectors, and to develop arobot that represents the motion of the vectors in terms of
the robot's motion. The angle between the two vectors
shall be limited to 0-180°. The robot should be easy tooperate for high school students to acquire the concept of©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
987

Kosei Machida, Shinichi Imai

vectors. To visualize the vectors (trajectory of the robot),the robot is placed on a piece of imitation paper.
2.2. Robot Functions
The speed of the robot's motor is defined as the vector
magnitude, the direction of the tires is defined as thevector direction, and the trajectory of the robot's progressis defined as the vector motion. When studying vector
sums, the robot's trajectory can be defined as the sum ofvectors by simultaneously performing the motions of thetwo vectors before combining the vectors.

Fig.1 Robots to be developed.

2.3. Robot Design
This Robot will be developed using Artec Robo from
Artec, Inc. The robot to be developed uses Artec blocks,
DC motors, servo motors, and a battery box Studuino: bit.
The omni wheel is an omni-directional moving
mechanism. [7] The omni wheel is an omni-directional
moving mechanism, and the wheels using the omni wheel
can move in any direction. [8] Studuino: bit is used as the
microcontroller. Artec blocks are used to assemble therobot. The servo motors are used to change the directionof the tires. DC motors are used to rotate the wheels,
which allow the robot to move on a flat surface. Servo
motors are used to change the direction of the tires.
Electric power is supplied from a battery box. The size ofthe robot is about the same as the size of the classroom.
The size of the robot should be large enough for students
to hold in their hands and manipulate on a desk.

Fig.2 Omni wheel movement
However, 𝑉1 is assumed to move in the direction 𝑉𝑥 . If
𝑉1 = 𝑉2 , the body moves straight ahead. The use of an
omniwheel allows the robot to move without turning, so
the sum of vectors can be moved in a direction different
from the direction in which the wheels rotate. The wheels
can be changed to any position by using servo motors in
the upper positions where the wheels are mounted. This
allows the direction of the wheel to be changed.

2.4. Correspondence between robot motion andvectors
Fig. 1 shows the appearance of this teaching material.
Hereafter, the two wheels at the bottom of the robot will
be referred to as Wheel A and the two wheels at the topof the robot in Fig. 1 as Wheel B. In Fig. 1, the directionof rotation and speed of the tires are specified for Wheel
A and Wheel B, respectively, and the trajectory of therobot's motion is referred to as vector motion. The motionof the omni wheels is shown in Fig. 2. As shown in Fig.
2, the velocity in the 𝑥-direction is 𝑉𝑥 and the velocity inthe 𝑦-direction is 𝑉𝑦 . Let the rotational velocities of theomniwheel be 𝑉1 and 𝑉2 , respectively. considering the 𝑥axis as the axis horizontal to 𝑉1 , the 𝑥 -axis as 0 rad,counterclockwise as positive, and the angle between thetwo wheels as 𝜃 , the calculation of the velocity
components can be expressed by the Eq. (1). With 𝑉𝑥 and𝑉𝑦 known, the direction of movement is determined, andthe direction of rotation and rotation speed of 𝑉1 and 𝑉2
are controlled.
𝑉1
𝑉 = [ 1] = [cos⁡(𝜃)
𝑉2

𝑉𝑥
0
][ ]
𝑠𝑖𝑛(𝜃) 𝑉𝑦

2.5. Image Analysis
Image analysis is performed to check whether the sumof vectors can be represented by the robot's motion. For
the image analysis, we use a camera stand and a video
shot from the top of the robot with an iPhone13. The
frame rate of the video is 30 fps. PV Studio 2D PRO is
used as the image analysis software [9]. Place a marker at
the center of the robot and record the axis of the robot's
movement. The 𝑥 -axis and 𝑦 -axis are set so that thecenter becomes the origin. The frame when the robot
starts moving is used as the reference frame. Because of
the blurring of the axis, the frame when the axis increases
or decrease in a certain direction without any increase or
decrease in the axis is the frame of the start of the
movement. Similarly, the last frame in which the axis
increases or decrease in a constant direction without
increasing or decreasing back and forth is the frame at theend of the motion. Image analysis is performed by
combining the two motions when the sum of vectors isdecomposed. The analyzed data is quantified, theoreticalvalues are derived, and a comparison of the motion and(1)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
988

Development and evaluation of

the theoretical values is made to confirm that the sum ofvectors can be expressed.
2.5.1. Image Analysis Theoretical Value
Define the first values of x-axis and y-axis (axis of theinitial position) 𝑥1 and 𝑦1 . Thus, the 𝑥-axis and 𝑦-axis
values in frame 𝑛 can be defined as 𝑥𝑛 and 𝑦𝑛 . The
general forms of the x - and y-axis are 𝑥𝑛 and 𝑦𝑛 ,respectively, where 𝑥1 and 𝑦1 are first term values.
The robot is assumed to move in the minus direction of
the x -axis. Therefore, if the robot stops moving at frame𝑚, the total number of frames can be represented by 𝑚.
If the range of frames in the x -axis and y-axis directions
is defined as 𝑎 and 𝑏, respectively, then 𝑎 and 𝑏 can be
expressed by Eq. (2) and Eq. (3), respectively.
𝑎 = |𝑥𝑚 − 𝑥1 |(2)

𝑏 = |𝑦𝑚 − 𝑦1 |(3)

Fig.3 Results of Image Analysis.
When the number of frames is half, that is, when the
direction of the vector sum changes, there is a differenceof several frames between the measured and theoreticalvalues. This may be due to the motor. We believe that theindividual differences in the motors caused the slight
discrepancies.
In addition, there was a discrepancy in the y-axis
between the initial value and the final value of themeasured value. This can be attributed to friction.
Friction is considered to have shifted the y-axis byapproximately 0.05 meter compared to the initial position.

If the values advancing in the x-axis and y-axis
directions per frame are defined as 𝑔 and ℎ, respectively,
then 𝑔 and ℎ can be represented by Eq. (4) and Eq. (5),respectively.
𝑔=

𝑎
𝑚

𝑏
ℎ=𝑚
2

The results show that the approximate shape of the sumof the vectors can be represented, although there is anerror of a few centimeters from the theoretical value. In
order to improve these deviations, we believe that the
problem can be solved by making the arrows connecting
the trajectories thicker. We also believe that the learning
about the robot mechanism can be expanded by having
the students consider the reasons for the deviations fromthe theoretical values.

(4)
(5)

In the case of ℎ, the vector shows two directions, so thenumber of frames is 𝑚/2, since the direction changes
after half the time. Therefore, the values 𝑥𝑛 and 𝑦𝑛 of thex and y-axis in the 𝑛𝑡ℎ frame can be expressed by Eq. (6),Eq. (7) and Eq. (8). However, when 𝑛 = 1, 𝑥1 = 𝑔 and𝑦1 = ℎ.
𝑥𝑛 = 𝑥𝑛−1 − 𝑔⁡(𝑛 ≧ 2)

3. Classroom Practice
3.1. Class Structure(6)𝑦𝑛 = 𝑦𝑛−1 − ℎ(1 < 𝑛 ≤ 𝑚/2)

(7)𝑦𝑛 = 𝑦𝑛−1 + ℎ(𝑚/2 < 𝑛 ≤ 𝑚)

(8)

In the class, the sum of vectors is represented bymoving a robot on paper, attaching points to the startingand ending points of the robot, and connecting thesepoints with arrows. The goal of the class is "to be able toconsider the sum of vectors by transforming thedecomposed vectors. The structure of the class is shownin Table 1. A questionnaire will be administered before
and after the class.

When the number of frames is half, 𝑦𝑛 can be expressedin two equations for 𝑛/2, since the sum of the vector’schanges direction after half the number of frames has
elapsed, since the vectors indicate two directions.
2.5.2. Results and Discussion of Image Analysis

Intro.

In PV Studio 2D PRO, each displacement of one axis
results in a movement of one meter. Based on this,comparing the results of image analysis with thetheoretical values, the motion when the left and rightwheels are moved simultaneously is shown in Fig. 3.

Dev.
Con.

Table 1. Class Flow.
Explanation of how to use the robot.
About vectors and sums in one direction.
Sum of vectors in two directions
Same magnitudes on the left and rightDifferent magnitudes of left and rightConfirmation of the sum of vectors©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
989

Powered by TCPDF (www.tcpdf.org)

Kosei Machida, Shinichi Imai

Table 2. Results of pre-post comparison of questionnaires (t-test (with correspondence))
Pre

Post

M

SD

M

SD

t -valuep -valueeffect d

1. Do you think you enjoy learning?

3.25

1.24

3.50

1.16

1.17

.261

0.21

2. Do you think you are able to participate willingly in class?

3.50

1.21

3.75

1.00

0.70

.497

3. Would you like to find out more about what you have learned?

2.31

1.08

3.00

1.10

2.71

.016

4. Can relate what they have learned to what they have studied and to what is happening in their daily lives and society.
2.38

1.20

2.80

1.13

1.00

.333

5. Can base one's opinions and thoughts on reasons and in a well-sequenced manner.

2.31

1.01

2.81

1.28

1.52

.150

6. Can imagine and look ahead and think about what could happen in relation to what they are learning.

2.31

0.95

3.63

1.09

3.88

.001

7. Organize materials and information gathered for a purpose into diagrams and tables.

2.56

1.21

3.06

1.34

1.94

.072

8. Can express facts and results based on the regularity and lawfulness of things.

2.50

1.03

3.25

1.29

2.54

.023

questionnaire

0.23
*

0.64
0.32
0.44

**

1.30
0.39

*

0.65

*p < .05 ;**p < .01※In the post-survey, add "Through the use of the robot in the class" at the beginning of the survey item.

concepts using test of understanding of vector", Journal of
Physics: Conference Series, vol2098, pp.1-5(2021)
3. Lai Poh Emily Toh, Albert Causo, Pei-Wen Tzuo, I-Ming
Chen, and Song Huat Yeo："A Review on the Use of
Robots in Education and Young Children", Educational
Technology & Society, 19 (2), pp.148-163(2016) ．
4. D. Lancheros-Cuesta and R. Fabregat: "Educational
Robotics Intervention in the Motivation of Students," inIEEE Revista Iberoamericana de Tecnologias delAprendizaje, vol. 17, no. 2, pp. 131-139, (2022).
5. S. Bruder and K. Wedeward: "Robotics in the classroom,"
in IEEE Robotics & Automation Magazine, vol. 10, no. 3,
pp. 25-29, (2003).
6. Kai Wang, Guo-Yuan Sang, Lan-Zi Huang, Shi-Hua Li
and Jian-Wen Guo："The Effectiveness of Educational
Robots in Improving Learning Outcomes”, A MetaAnalysis, Sustainability 2023, 15, 4637, pp.1-16(2023)
7. Artec Robo official website, (Accessed on November 25,
2023)
8. S. Long, T. Terakawa, M. Komori and T. Ougino:"Analysis of Traveling Strategies for Driving OmniWheeled Vehicle Around a Corner," in IEEE Access, vol.
8, pp. 104841-104856, (2020).
9. PV STUDIO website, (Accessed on November 13, 2023)
10. Ministry of Education, Culture, Sports, Science andTechnology: Learning Innovation Project Demonstration
Study Report (2014)

3.2. Survey Summary
On October 11, 2023, 17 high school sophomores (16pre-survey, 17 post-survey) will be surveyed. The
subjects are students who have not studied vector sums.
The questionnaire will be administered using a five-item
method with reference to the literature [10]. In order tomeasure knowledge, survey questions were administered
before and after the survey. The pre-survey was
conducted before the class, and the post-survey was
conducted after the class. A t-test (with correspondence)
was used to analyze the differences in attitudes toward
independent.
3.3. Result
The results of the survey are shown in Table 2;
significant differences were found in three items. For theknowledge question, the response rate for Question 1 was
62.4% before and 93.8% after the response. For question
2, the pre-response rate was 18.8% and the post-response
rate was 50.0%.
4. Conclusion

Authors Introduction
In this study, we developed a learning support robotthat represents vector sums. The robot was also used in aclass. A questionnaire survey was conducted, andsignificant differences were found. In addition, thepercentage of correct answers in the test was higher than
before and after the class. In the future, we plan to further
analyze the questionnaires and modify the robot and theclasses.

Mr. Kosei Machida
He e received his Bachelor's Degree(Education) in 2022 from the Faculty ofEngineering,
Toyko
Gakugei
University in Japan. He is currently amaster student in Tokyo Gakugei
University, Japan.
Dr. Shinichi Imai

Acknowledgements

He graduated doctor course at
department of engineering in
Hiroshima University. He works at
department of education in Tokyo
Gakugei University. His researcharea is about control systemdesign, educational engineering.

We would like to thank Artec Corporation for lending
us a robot to be used in our classroom practice. We would
like to express our deepest gratitude to Artec Corporation.
References
1.

2.

Ministry of Education, Culture, Sports, Science andTechnology: "The Revisions of the Courses of Study for
Elementary and Secondary School (2018) "BRA Latifa, E Purwaningsih, and S Sutopo;"Identificationof students’ difficulties in understanding of vector©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
990

A Feasibility Study on Methods to Measure the Strain on Young Children's Bodies.
Sachiko Kido

Interdisciplinary graduate school of Agriculture and Engineering, University of Miyazaki,,
1-1, Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
Praveen Nuwantha Gunaratne

Interdisciplinary graduate school of Agriculture and Engineering, University of Miyazaki,,
1-1, Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
Hiroki Tamura

Faculty of Engineering, University of Miyazaki,
1-1, Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
E-mail: t321z01@student.miyazaki-u.ac.jp, ti20060@student.miyazaki-u.ac.jp, htamura@cc.miyazaki-u.ac.jp

Abstract

This paper investigates the effective methods of measuring the burden on the infant's body. Initially, the“the burden on the body and the lumbar spine burden are the same” was defined. The calculations andcomparisons were carried out based on the two methods: the AnyBody Modeling System and thesmartphone application Yo-bukun (a lumbar spine burden measurement application). The lumbar burden
was calculated using the AnyBody Modelling System. In addition, an iPhone running the Yo-bukun
lumbar burden measurement application was placed close to the heart position in order to simulate thelumbar burden. The results were compared with the respective average values and validated.
Keywords: Infants, Physical activity, Lunbar burden,of the human body, including muscle activity, muscle andantagonist muscle forces, elastic energy of tendons, joint
forces, and joint moments, during motion. While
extensively used for simulating joint movements andanalyzing motion capture data from video, research using
this system has predominantly focused on adults, withlimited exploration in the context of infants.

1. Introduction
In contemporary Japan, where the decline in children's
physical fitness has emerged as a prominent concern [1],a comprehensive analysis of children's physical activity,
particularly among young children, holds significant
relevance. Various methodologies are employed tomeasure physical activity, encompassing continuousheart rate recording, electromyography, pedometers,
calorie counters, and accelerometers [2], [3], [4]. Notably,
smartwatches like the Apple Watch have gained
popularity in recent years. However, the majority of thesemethods are tailored for adult use. Even amongsmartwatches, those designed for children are geared
towards elementary school students and older, lacking
compatibility with younger children. Accelerometers,
pedometers, and behavior analysis are frequently
employed for measuring physical activity in young
children, yet a prevailing challenge in research utilizingthese methods is their inadequacy for accommodating theneeds of this demographic. Hence, there is an immediate
necessity to investigate measurement methods that alignwith the body size of infants when assessing their
physical activity.

The Yo-bukun application [6], [7], a collaborative
effort between the Faculty of Engineering at Miyazaki
University and Densan Corporation in 2022, is an iPhone
application quantifying and displaying lower back burden.
This application is calibrated when the user stands
upright, and calculates the amount of lumbar burden
based on the tilt of the iPhone. This iPhone application
facilitates the easy measurement of lower back burden
and has not been previously applied to infants.
This paper investigates the effective methods ofmeasuring the burden on the infant's body. The
calculations and comparisons were carried out based on
the two methods: the AnyBody Modeling System and thesmartphone application Yo-bukun (a lumbar spine
burden measurement application).
2. Methodology

The AnyBody Modeling System, developed at
Aalborg University in Denmark, is a widely utilizedmusculoskeletal mechanics analysis software globally.
The AnyBody Modeling System [5] employs inverse
dynamics analysis to calculate forces acting on each partIn this paper, we will compare the results of the
anybody modeling system and the “Yo-bukun” analysis
application, which are set to the height and weight ofinfants. As a measurement system, the AnyBody©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
991

Sachiko Kido, Praveen Nuwantha Gunaratne, Hiroki Tamura

Modeling System is accurate if the movement ismeasured correctly. However, the AnyBody Modeling
System is not realistic to accurately measure the burdenof a moving infant. The “Yo-bukun” analysis application,
who wears a smartphone, has fewer restrictions and cantake measurements over a long period of time.
The value of the “Yo-bukun” analysis application is
unclear how accurate for infants. Therefore, in order toconfirm the accuracy of the “Yo-bukun” analysis
application for infants, we conduct the comparative
verification using the parameter settings as those for
infants.
2.1. The AnyBody
procedure

Modeling

System's

analysis

In this paper, we use the AnyBody Modeling System toanalyze the sample basic motion. Set the analysis settings
in the AnyBody Modeling System to the height andweight of a typical infant, and calculate the amount oflumbar strain during movement in a pseudo manner.

Fig. 1. Sequence of balance movements from behindwith the AnyBody Modeling System analysis

2.2. The Yo-bukun’s analysis procedure
Set the settings in the “Yo-bukun” analysis application
to the same height and weight as in Chapter 2.1. We moveby hand the smartphone according to the sample basic
motion. In this way, we can simulate the basic motion on
our smartphone.
3. Results and Discussion
3.1. Analysis results with the AnyBody ModelingsystemFig. 1 shows a series of balancing movements viewed
from behind. The color of the muscles used changes,
indicating areas of high strain. The fact that the lower
back is colored in addition to the leg on which the bodyweight is placed indicates that the burden is placed
mainly on the lower back. The posture with the maximum
burden on the lower back is considered to be posture 5, in
which the subject is off-balance. In the "balance" posture,
the average value was 231[N] for approximately 8
seconds, and the maximum value was 283[N] (Fig. 1).

Fig. 2. Sequence of squat movements from behindwith the AnyBody Modeling System analysis
According to the AnyBody Modeling System analysis,
two of the movements put the burden primarily on the
lumbar region, thus meeting the definition of the “theburden on the body and the lumbar spine burden are thesame”.

Fig. 2 shows a series of squatting movements, viewed
from behind. In this movement, too, the color of thelumbar region changes, and it can be said that the lumbar
region is burdened. The posture with the maximum
lumbar load is posture 5 in the standing up motion. In the
crouching movement, the average value forapproximately 10 seconds was 322[N], and the maximum
value was 484[N].

3.2. Analysis results with the Yo-bukun
In the context of balance movements, the approximate
average load over a duration of 8 seconds was 164[N],with a maximum load reaching 204[N].

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
992

Powered by TCPDF (www.tcpdf.org)

A Feasibility Study on

In the squatting movement, the approximate averageload over a duration of 10 seconds was 228[N], with amaximum load of 320[N].

5.

3.3. Comparison of Analytical Results between
Two Software Applications

6.

In the Squatting movement, the average load was
322[N] in the AnyBody Modeling System and 228[N] in
the Yo-bukun, resulting in AnyBody Modeling System
exhibiting a slightly higher value of approximately 90[N].
The average value for balance movements was also 231[N] for the AnyBody Modeling System and 164[N] for
Yo-bukun, with the AnyBody Modeling System being
about 70[N] higher. Although there was a slight
discrepancy in the average values for both "Squatting"and "Balance," the fact that similar numerical values wereobtained with both software applications. The burden on
the body of infants during these two movements isapproximately 200[N] to 350[N]. Although we predictthat the difference between the two results is due todifferences in the standards of the skeletal models used.

7.

Ms. Sachiko Kido
She received her M.S. from Nakamura
Gakuen University, Graduate School of
Human Development in 2015. Currently
enrolled in the doctoral program at theGraduate School of Agriculture andEngineering,
University of Miyazaki.
Assistant Professor, Department of Early Childhood
Education, Higashikyushu Junior College, 2010. Lecturer,
Faculty of Education, Miyazaki International University,
since 2019. Her main research interest is the developmentof motor skills in young children.

In this paper, the physical burden on infants wasmeasured using two different software applications. A
comparison of the average values obtained from the two
software applications revealed no significant differences,suggesting that the burden on infants during the two
movements can be considered to range fromapproximately 200[N] to 350[N]. The analysis time
capability of the AnyBody Modeling System isapproximately ten seconds. In contrast, the Yo-bukun
enables prolonged analyses. The absence of significant
differences between the AnyBody Modeling System andYo-bukun in this study suggests the potential formeasuring the physical load during infant body activities
using Yo-bukun. In our future work, the validations arenecessary by increasing the number of infant subjects.

Mr. Praveen Nuwantha Gunaratne
He received his Bachelor’s degree inEngineering in 2018 from the Faculty ofEngineering, University of Moratuwa, Sri
Lanka. He is currently a Doctoral student in
University of Miyazaki, Japan
Prof. Hiroki Tamura
He received the B.E. and M.E. degree from
Miyazaki University in 1998 and 2000,
respectively. From 2000 to 2001, he was an
Engineer in Asahi Kasei Corporation,
Japan. In 2001, he joined Toyama
University, Toyama, Japan, where he wasa Technical Official in the Department ofIntellectual Information Systems. In 2006, he joined
Miyazaki University, Miyazaki, Japan, where he was an
Assistant Professor in the Department of Electrical andElectronic Engineering. Since 2015, he is currently aProfessor in the Department of Environmental Robotics.
His main research interests are Neural Networks andOptimization Problems. In recent years, he has had interestin Biomedical Signal Processing using Soft Computing.

References

2.

3.

4.

H. Tamura, K. Sakurai, K. Tanno, and Y. Fuse, “AStudy on the Lumbar Burden Evaluation of Workusing One Smartphone,” Proceedings ofInternational Conference on Artificial Life andRobotics, vol. 23, pp. 550–553, Feb. 2018, doi:
https://doi.org/10.5954/icarob.2018.os14-1

Authors Introduction

4. Conclusion

1.

Prefecture”, the Japan Journal of Human Growth andDevelopment Research, vol. 90, 2021, pp. 57-64.
M. Damsgaard, J. Rasmussen, S. T. Christensen, E. Surmaand M. D. Zee, “Analysis of musculoskeletal systems in
the AnyBody Modeling System”, Simulation Modelling
Practice and Theory, vol. 14, 2006, pp. 1100-1111.
H. Tamura, K. Sakurai, K. Tanno, and Y. Fuse, “A Studyon the Lumbar Burden Evaluation of Work using One
Smartphone,” Journal of Robotics, Networking andArtificial Life, vol. 5, no. 3, pp. 173–173, Jan. 2018, doi:
https://doi.org/10.2991/jrnal.2018.5.3.7.

Japan Sports Agency, “Results of the 2022 National
Physical Fitness and Exercise Capacity and Exercise
HabitsSurvey”,
https://www.mext.go.jp/sports/b_menu/toukei/kodomo/ze
ncyo/1411922_00004.html, (Last viewed on December 15,2023).
A. Sugimoto, “The Measurement of Physical Activity : A
Review of Recent Progress”, The Japanese Journal of
Rehabilitation Medicine, vol. 37, 2000, pp. 53-61.
S. Tanaka, “A Study of the Relationship between YoungChildren’s Motor Abilities and Physical Activities :Focusing Five Year Old Children’s Physical Activity
Countsina Day, The Hoikugaku Kenkyu vol. 47, no.2,
2009, pp. 8–16.
R. Tsuda, M. Watanabe and K. Suzuki, “Daily physicalactivities of public nursery school children during
winter：a case study of eight public schools in Ishikawa©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
993

An Integration of Contact Force Models with Multibody Dynamics Analyses for Human Joint
Mechanisms and Effects of Viscoelastic Ground Contact
Shintaro Kasai

Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan
Dondogjamts Batbaatar

Mongolian University of Science and Technology, 8th khoroo, Baga toiruu 34, Sukhbaatar district Ulaanbaatar,
Mongolia 14191
Hiroaki Wagatsuma

Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan
Email: kasai.shintaro660@mail.kyutech.jp, dondogjamts.b@must.edu.mn, waga@brain.kyutech.ac.jp

Abstract
In human movement and rehabilitation analyses, human joint dynamics is a key to consider the incorporation ofspring-damper components, flexible bodies and contact forces analytically. In the present study, an analytical method
for human gaits were introduced to integrate those essential elements, and viscoelastic properties of musculoskeletal
system were modeled with the absolute nodal coordination formula (ANCF) method representing flexible body
motions. A contact force model simulates interactions between different body segments and the environment. The
proposed system is applied to a slider crank mechanism, demonstrating its capabilities in human joint motion analysis
using integrated dynamic model within the framework of multibody dynamics (MBD), which realizes
dynamic/inverse dynamics for human biomechanics.
Keywords: Multibody dynamics (MBD), Contact force model, Absolute nodal coordinate formula (ANCF), humanbiomechanics[4], [5], [6], [7]. In order to reproduce the movement of a
joint and analyze in detail the increase/decrease in theload that occurs on its components, it is necessary toconstruct and analyze models in which the muscles andligaments are replaced with spring damper components
or flexible bodies. The absolute nodal coordinate formula(ANCF) [8], [9], [10] has been proposed as an analysis
method that incorporates finite element methods for
systems composed of flexible bodies in MBD. The
ANCF allows the analysis of systems that include large
deformation of flexible bodies by dividing the body into
multiple elements. Furthermore, by considering theeffects of viscoelastic ground contact [11], [12], it
becomes possible to analyze human motion such as
human gait.

1. Introduction
Joint disorders progress irreversibly due to exercise andaging, and the number of patients who have joint
disorders is increasing. These joint disorders affect gaitand daily life [1] and they are one of the major issues in
the current aging society. Physical therapy treatments forjoint disorders include step-by-step treatment that
restricts extension by restraining joints using joint
orthosis, and rehabilitation that applies appropriate joint
loads. From the viewpoint of prevention of the disease by
joint load reduction, patient-specific order-made assistive
devices using 3D printers and flexible materials [2] arealso developing. However, the formulation ofrehabilitation programs and the design of orthotics andassist devices are performed through subjective
evaluations based on the experience of physical therapists
and prosthetists. Therefore, the standard numericalevaluation index is needed, and modeling human joint
mechanisms and analyzing their movements plays animportant role in the formulation of the programs anddesign for the devices.

In this study, for the implementation of an analyticalsystem for the joint load reduction analysis based on
MBD, we demonstrate an integrated dynamic model byapplying a flexible body, spring-damper components andground contact force to the crank model and analyzing it
dynamically.
2. Methodology
2.1. Integrated dynamic analysis based on MBD

The dynamic and inverse dynamic analysis of humanmotion is possible with multibody dynamics (MBD) [3],©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
994

Shintaro Kasai, Dondogjamts Batbaatar, Hiroaki Wagatsuma

The motion of a multibody system (MBS) composed ofrigid bodies is described by a differential algebraic
equation (DAE) such as Eq. (1) based on MBD theory
using Lagrange multiplier λ and acceleration equation γ.
Here, 𝑀𝑅 , Φ𝑅 and 𝑄𝑅𝐴 mean the mass matrix of rigid
bodies, constraint matrix and external force vector
respectively. Subscript 𝑞 means the Jacobian matrix
obtained by partially differentiating the constraint matrix
with respect to the generalized coordinate matrix 𝑞.

2.2. Contact force in external force vector
In the continuous contact model, the normal force 𝐹𝑁 isexpressed by Eq. (5) using the generalized stiffness
parameter K and the relative penetration or indentation 𝛿.
Based on the theory of elasticity by Herts [11], theexponent n is equal to 2⁄3.
𝐹𝑁 = 𝐾𝛿 𝑛

[

𝑀𝑅
Φ𝑅𝑞

𝑇
Φ𝑅𝑞

0

𝑞̈
𝑄𝐴
] [ 𝑅] = [ 𝑅 ]
𝜆

(1)

𝛾

Considering the damping term, 𝐹𝑁 can be written as
follows with hysteresis damping factor 𝜒. Here, 𝑐𝑟 , 𝛿̇ and𝛿̇ (−) mean coefficient of restitution, relative
approach/departing velocity and initial approach velocity
respectively.

Considering incorporating a flexible body into MBS,
the generalized coordinate matrix of the flexible body isdefined and its motion is described based on ANCF.
According to Fig. 1, the generalized coordinate matrix ofthe flexible body 𝑖 is described as Eq. (2) with x-y
coordinates at each node and their slope [13].
𝑖𝑖𝑖𝑖𝑞𝐹𝑖 = [𝑒1𝑖 , 𝑒2𝑖 , 𝑒3𝑖 , 𝑒4𝑖 , ⋯ 𝑒4𝑁
, 𝑒4𝑁
, 𝑒4𝑁
, 𝑒4𝑁
]
𝑛 −3
𝑛 −2
𝑛 −1
𝑛

𝑇

𝐹𝑁 = 𝐾𝛿 𝑛 + 𝜒𝛿 𝑛 𝛿̇

(2)

0
𝑀𝐹
Φ𝐹𝑞

𝑇
Φ𝑅𝑞
𝑄𝑅𝐴
𝑞̈ 𝑅
𝑇 ][
Φ𝐹𝑞 𝑞̈ 𝐹 ] = [𝑄𝐹𝐴 ]𝜆𝛾
03𝐾(1 − 𝑐𝑟 2 )4𝛿̇ (−)
3(1 − 𝐶𝑟 2 ) 𝛿̇
𝐹𝑁 = 𝐾𝛿 𝑛 [1 +∙ (−) ]
4𝛿̇
𝜒=𝑄𝐴
𝑄 = [ 𝑅𝐴 ] + 𝐹𝑆𝐷 + 𝐹𝐶
𝑄𝐹

(7)
(8)

(3)
When sliding occurs in contact between the body and theground, the dynamic friction force is described as Eq. (9)using kinetic friction coefficient μ, tangential velocity
vector 𝑐𝑓 and dynamic correction coefficient 𝑐𝑑 based on
friction velocity.

𝑄𝑅𝐴 and 𝑄𝐹𝐴 are external force vectors consist of gravity
force and elastic force due to deformation of flexible
body. When the analytical model consists of springdamper components and the ground contact model, thetotal external force vector 𝑄 𝐴 includes force vectors 𝐹𝑆𝐷
and 𝐹𝑐 which are generated by spring-damper andviscoelastic ground contact as Eq. (4).
𝐴

(6)When 𝜒 is expressed as Eq. (7), substituting it into Eq.
(6), 𝐹𝑁 becomes Eq. (8).

The DAE of flexible multibody dynamics (fMBD)
which incorporates flexible bodies based on ANCF canbe expressed as Eq. (3) by expanding Eq. (1).
𝑀𝑅
[ 0
Φ𝐹𝑞

(5)

𝐹𝑓 = −𝜇𝐹𝑛 𝑐𝑓 𝑐𝑑(9)Consider the case where body 𝑖 contacts the straight lineax + by + c = 0 at contact point CP according to Fig. 2,
the coordinate and velocity at point cp are as Eq. (10).

(4)

𝑥𝑐𝑝
𝑥𝑖
𝑟 cos 𝜃𝑖 − sin 𝜃𝑖[ 𝑦 ] = [𝑦 ] + [ 𝑖 ] []0 sin 𝜃𝑖cos 𝜃𝑖
𝑐𝑝
𝑖
𝑥̇ 𝑐𝑝
𝑥̇
𝑟
[ ] = [ 𝑖 ] − [ 𝑖 ] 𝜃̇𝑖
𝑦̇𝑐𝑝
𝑦̇ 𝑖
0
{

(10)

The relative penetration δ can be expressed as theshortest distance between the contact point and thestraight line as Eq. (11).
𝛿=Fig. 1 The definition of the generalized coordinates ofthe flexible body I based on ANCF.

|𝑎𝑥𝑐𝑝 + 𝑏𝑦𝑐𝑝 + 𝑐|
√𝑎2 + 𝑏 2

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
995

(11)

An Integration of Contact

Fig. 3 Integrated dynamic model of slider crank

The driving constraint Φ𝐷 is given to 𝐿1 as Eq. (14) andangular velocity ω is equal to 2𝜋⁄5 [𝑟𝑎𝑑 ⁄𝑠 ].
(0)

Φ𝐷 = 𝜃1 − 𝜔𝑡 − 𝜃1Fig. 2 Viscoelastic contact with the tilted grounddescribed as 𝑎𝑥 + 𝑏𝑦 + 𝑐 = 0

𝑐𝑟 = (𝑥̇ 𝐺 − 𝑥̇ 𝑐𝑝 ) cos 𝜑 + (𝑦̇ 𝐺 − 𝑦̇𝑐𝑝 ) sin 𝜑(14)

In this slider-crank model, the endpoint at 𝐿2 collides
with 𝐿3 and the effect of viscoelastic contact occurs
without sliding. The contact force between these links is
described as Eq. (15).

Since the relative approach velocity 𝛿̇ in Eq. (8) andtangential velocity 𝑐𝑟 in Eq. (9) are perpendicular andparallel to the straight line, they can be written as Eq. (12)when the ground line moves as [𝑥̇ 𝐺 , 𝑦̇ 𝐺 ]𝑇 . Here the angle
between x-axis and the straight line is as 𝜑 =𝑡𝑎𝑛 −1 (− 𝑎⁄𝑏).
𝛿̇ = (𝑥̇ 𝐺 − 𝑥̇ 𝑐𝑝 ) sin 𝜑 − (𝑦̇ 𝐺 − 𝑦̇𝑐𝑝 ) cos 𝜑=023(1 − 𝑐𝑟 2 ) 𝑥̇ 3 − 𝑥̇ 2∙]4𝛿̇ (−)
𝐹𝑓 = 0𝐹𝑁 = 𝐾(𝑥3 − 𝑥2 )3 [1 +(15)

3. Results and Discussion(12)

3.1. Dynamic analysis of the integrated slider-crankmodel based on MBD
Numerical analysis of the dynamic analysis based on
MBD was performed by solving the DAE of theanalytical system with MATLAB. In numericalcomputation, the fourth-order Runge-Kutta Gill’s method[14] was used as the numerical integration method. The
time step width is ℎ = 1.0 × 10−5 [𝑠]. The parameters ofspring-damper components are 𝑘 = 50, 𝑐 = 0.1 and theparameters of viscoelastic contact are K = 4.0 ×
105 , 𝑐𝑟 = 0.1 and μ = 0.45. The results of the dynamic
analysis for each element of the integrated slider-crankmodel are shown in Fig. 4. The x-coordinates of link L2
with and without flexible material are compared as shownin Fig. 5. The right graph in Fig. 5 shows the difference
between each movement at first contact. When the model
consists of a flexible body, the viscoelastic contact causes
the deformation of it.

2.3. Slider-crank model for validation of
integrated analysis
In order to validate the MBD integrated analysis, aflexible body, a spring-damper component and theviscoelastic contact model are applied to the slider-crankmodel as shown in Fig. 3 to compare the changes in the
behavior of the mechanism with and without thesecomponents. When 𝑁𝑒 which is the number of division
elements of the elastic beam 𝐹1 based on ANCF is equal
to 8, the kinematic constraint equation between 𝐹1 andrigid links 𝐿1 and 𝐿2 of the slider-crank in Fig. 4 isdescribed by Eq. (13).
𝑙1cos 𝜃12𝑙1
𝑒2 − 𝑦1 − sin 𝜃1
Φ𝐿𝐾1𝐹1
2[ 𝐾 ]==0𝑙2
Φ𝐿2 𝐹1𝑒33 − 𝑥2 + cos 𝜃22𝑙2[ 𝑒34 − 𝑦2 + 2 sin 𝜃2 ]𝑒1 − 𝑥1 −(13)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
996

Shintaro Kasai, Dondogjamts Batbaatar, Hiroaki Wagatsuma

Fig. 4 The result of the dynamic analysis of theintegrated slider-crank model.

Fig. 6 Integrated dynamic models of slider crank. The
model a) consists of rigid linkages, b) is replaced andhas a flexible body and c) is applied with a flexible
body and a spring-damper component.

Fig. 5 The transition of x-coordinate at CP. The rightgraph is looking at the point in time when the effect ofviscoelastic contact occurres.

Fig. 7 The difference of contact force at contact point
between the slider-crank models
analysis system to incorporate flexible bodies and a
viscoelastic contact model. The use of flexible materials
and spring-damper components in the joint mechanism
enables gait analysis of walking robot legs that utilize
nonlinear elastic properties and dynamic analysis of theeffect of joint assistive devices on human walking motion,and is effective for the design and development of suchrobot legs and support devices. In further analysis, it isnecessary to apply this integrated dynamic analysis to
joint drive control for the robot leg models and the
combination models of human gaits and assistive devices.

3.2. Constraint force by adopting viscoelastic
contact model
MBD-based dynamic analysis was performed for threemodels with different components in Fig. 6 to analyze thechanges in contact force on the body of the analyticalsystem. Fig. 7 shows the result of the dynamic analysis,
and the deflection of the flexible body and the contractionof the spring-damper component reduces the relative
penetration of the contact point, resulting in smaller
contact forces compared to the rigid slider-crank model.
It is possible to analyze the force by the viscoelastic
contact which changes according to the components
applied to the MBS.

Acknowledgments
This work was supported in part by JSPS KAKENHI(16H01616, 17H06383), Grant-in-Aid for JSPS Fellows(23KJ1754), Project on Regional Revitalization ThroughAdvanced
Robotics(Kyushu
InstituteofTechnology/Kitakyushu city, Japan) and Kitakyushu
Foundation for the Advancement of Industry, Scienceand Technology (FAIS). The authors gratefully

4. Conclusion
Through this research, the integrated dynamic analysis
of models which has contacts between objects or objects
and ground was shown to be possible by expanding the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
997

Powered by TCPDF (www.tcpdf.org)

An Integration of Contact

acknowledge ARIZONO Orthopedic Supplies Co., Ltd.
for assistance with the formulation of exoskeleton-typeassistive devices in their products.

Authors Introduction
Mr. Shintaro Kasai
He received his Master’s degree inEngineering in 2023 from the Graduate
School of Life Science and System
Engineering, Kyushu Institute oftechnology (Kyutech) in Japan,. He iscurrently a doctoral course student inKyutech, Japan and JSPS Research
Fellow (DC1).

References
1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

L. Zhang, G. Liu, B. Han, Z. Wang, Y. Yan, J. Ma and P.
Wei, “Knee Joint Biomechanics in Physiological
Conditions and How Pathologies Can Affect It: A
Systematic Review,” Applied Bionics and Biomechanics,
2020, Article Number 7451683.
Y. J. Choo, M. Boudier-Revéret and M. C. Chang, “3D
printing technology applied to orthosis manufacturing:narrative review,” An Palliat Med 2020; 9(6): 4262-4270.
doi: 10.21037/apm-20-1185
J. A. C. Ambrósio, Impact of Rigid and Flexible Multibody
Systems: Deformation Description and Contact Models, in
Virtual Nonlinear Multibody Systems. NATO ASI Series(Series II: Mathematics, Physics and Chemistry), eds. W.
Schiehlen and M. Valášek, Springer, Dordrecht, vol 103,
2003, pp. 57-81.
P. E. Nikravesh, Planar Multibody Dynamics: Formulation,
Programming with MATLAB, and Applications, 2nd edn.,
CRC Press, Boca Raton, 2018.
K. Komoda and H. Wagatsuma, Energy-efficacy
comparisons and multibody dynamics analyses of legged
robots with different closed-loop mechanisms, Multibody
System Dynamics 40, 2017, pp. 123–153.
D. Batbaatar and H. Wagatsuma, A Proposal of theKinematic Model of the Horse Leg Musculoskeletal
System by Using Closed Linkages, Proceedings of the2019 IEEE International Conference on Robotics andBiomimetics (ROBIO), Dali, China, 2019, pp. 869–874.
J. Baumgarte, Stabilization of constraints and integrals of
motion in dynamical systems, Computer Methods in
Applied Mechanics and Engineering 1(1), 1972, pp. 1–16.
A. A. Shabana, "Computer Implemantation of theAbsolute Nodal Coordinate Formulation for Flexible
Multibody Dynamics," Nonlinear Dynamics, vol. 16, pp.
293-306, 1998.
L. G. Maqueda, A. A. Mohamed and A. A. Shabana, "Useof General Nonlinear Material Models in Beam Problems:Application to Belts and Rubber Chains," Journal ofComputational and Nonlinear Dynamics, vol. 5, pp.
21003-1-21003-10, 2010.
A. A. Shabana, An overview of the ANCF approach,justifications for its use, implementation issues, and future
research directions, Multibody System Dynamics, 2023.
11. Flores, P., Machado, M., Silva, M.T. et al. On thecontinuous contact force models for soft materials in
multibody dynamics. Multibody Syst Dyn 25, 357–375(2011). https://doi.org/10.1007/s11044-010-9237-4
D. Batbaatar and H. Wagatsuma, A Viscoelastic Contact
Analysis of the Ground Reaction Force Differentiation in
Walking and Running Gaits Realized in the Simplified
Horse Leg Model Focusing on the Hoof-Ground
Interaction, Journal of Robotics, Networking and Artificial
Life, Vol. 8(2); September (2021), pp. 78-84.
M. A. Omar and A. A. Shabana, "A Two-Dimensional
Shear Deformable Beam for Large Rotation andDeformation Problems," Journal of Sound and Vibration,
vol.
243,
no.
3,
pp.
565-576,
2001.
doi:10.1006/jsvi.2000.3416
Wolfram Research, Inc., Runge-Kutta Gill’s method,
https://mathworld.wolfram.com/GillsMethod.html

Dr. Dondogjamts Batbaatar
He received his M.S. in the field of
mechatronics
fromMongolian
University of Science and Technology(MUST), Mongolia and Ph.D. degreefrom Kyushu Institute of Technology,
Japan in 2015 and 2021. He is currently
a vice dean of research affair in Schoolof Mechanical Engineering Transportation in MUST. His
research interests include computational non-linear
dynamics and bio-inspired robotics.
Dr. Hiroaki Wagatsuma
He received his M.S., and Ph.D. degrees
from Tokyo Denki University, Japan, in
1997 and 2005, respectively. In 2009, hejoined Kyushu Institute of Technology,
where he is currently a Professor of the
Department of Human Intelli- gence
Systems. His research interests include
non-linear dynamics and robotics. He is amember of IEEE.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
998

Haptic Sensation Enhancement via the Stochastic Resonance Effect and Its Application to
Haptic Feedback for Myoelectric Prosthetic Hands
Yoshitaka Mizumoto

Okayama University, 1-1-1 Tsushima-naka, Kita-ku, Okayama, 700-8530, Japan
Taro Shibanoki

Okayama University, 1-1-1 Tsushima-naka, Kita-ku, Okayama, 700-8530, Japan
Email:mizumoto-y@s.okayama-u.ac.jp, shibanoki@okayama-u.ac.jp

Abstract

In this paper, we propose a highly realistic haptic feedback method for myoelectric prosthetic hands. Although several haptic feedback methods have been studied, this study attempts to create realistic sensory
feedback through prosthetic hands by not only using feedback methods but also by improving the haptic
sensation based on the stochastic resonance effect. In the experiment, contact information obtained froma microphone attached to the fingertip of a prosthetic hand was transferred through a vibrotactile stimulator near the elbow fossa, and white noise vibration was also applied near the elbow fossa to verify the
improvement of tactile sensitivity of the fingertips. The results demonstrated the possibility of transmitting
tactile information for myoelectric prostheses through sensory enhancement of the user.
Keywords: Myoelectric Prosthetic Hand, Haptic Sensation, Stochastic Resonance, Haptic Feedback

can now be transmitted. Alessia et al. measured the acceleration of the fingertip and controlled the oscillator of asocket to enable discrimination of roughness [7], Christian et al. detected pressure from a silicon bulb and confirmed that the magnitude of pressure could be discriminated by inflating the silicon pad at the cut using a monofilament [8].
These various sensory feedback methods provide only
pseudo feedback, and it is difficult to reproduce actual
sensation. This is due to practical issues such as the danger of direct sensory feedback, including pain, and thefact that the cutaneous sensation near the amputation site
is different from that at the fingertip, where feedback isprovided in the noninvasive approach. Therefore, we
consider improving the user's sensation.
Minamizawa et al. have made it possible to reproduce
tactile sensations through vibration stimulation by recording and playing back tactile data [9]. Jianyao et al.
measure the acceleration of the fingertip and transmit thevibration as an audio signal to enable roughness discrimination [10]. Research has also been conducted to enhance the sensory perception on the human side of receiving feedback, enhancing the ability to detect signals byapplying noisy vibrations to the human hand or foot [11],[12]. This phenomenon is called stochastic resonance,and looking at the case of vibration to the upper limb, it
has been reported that applying noise to the wrist and fingers improves tactile sensitivity at the fingertips [13],1. Introduction
To realize a prosthetic hand that resembles a human
hand, it is necessary not only to consider the mechanismof the prosthetic hand but also to establish a sensory feedback method, such as tactile sensation. Sensory information has a vital function in daily activities because thesense of touch conveys various properties or informationof objects [1]. However, it is said that when prosthetic
users touch objects with their prosthetic hands, they feelvibrations through the socket, making it difficult to accurately acquire tactile information through the prosthetic
hand. Although myoelectric prosthetic hands have
evolved to imitate motor functions of human hands [2],many current devices are deemed insufficient given theirinability to deliver relevant sensory information.
The senses are broadly classified as cutaneous anddeep senses, and restoration of these senses shows its improvement for the daily life of forearm amputees [3]. Various methods of cutaneous sensation feedback such as
touch, pressure, pain, warmth and cold have been investigated. In the past, a simple transducer attached to asmall speaker [4] was proposed, and in recent years, Ueda
et al. have transmitted the warmth or coldness of an object
by transmitting the temperature detected by a temperature
sensor at the fingertip to a device on the upper arm [5],Osborn et al. developed electronic skin, which can transmit pain to the user by detecting differences in the shapeand texture of objects at the fingertips and sending signals
to peripheral nerves [6], and various other skin sensations©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
999

Yoshitaka Mizumoto, Taro Shibanoki[14]. However, the application of these sensory enhancements using audio feedback and stochastic resonance toprosthetic hands has not been fully investigated.
This study applies these methods to a prosthetic handand provides tactile feedback near the elbow fossa using
voice. In addition, by focusing on the phenomenon ofprobability resonance, we attempt to enhance the sensation obtained by the sensory feedback method of the prosthetic hand, thereby increasing the realism of the sensation through the prosthetic hand.

(b) Myoelectric prosthetic hand(a) Human handFig. 1. Scenes from the verification experiment.

indicate that fingertip sensation cannot be obtained
through the socket of the prosthetic hand. From a questionnaire survey after the experiment, the participant answered that when they touched the sandpaper with theirprosthetic hands, they felt almost no vibration through thesockets, suggesting that discrimination was impossible.

2. Sensory feedback and enhancement
The proposed myoelectric prosthetic hand sensory
feedback method is based on the TECHTILE toolkit [9]and consists of a microphone (AT9904, audio-technica),
an audio amplifier (LP-2024A+, Lepy), and vibration actuators (AFT14A903A, Alps Alpine). In this paper, a microphone is attached to the fingertip of a prosthetic handto record friction and vibration sounds when touching an
object as an input signal for vibration stimulus generation.
The signal is output as vibration through an audio amplifier using an actuator attached near the elbow fossa.
In this study, we further focused on the stochastic resonance effect to improve sensory function. The vibration
used is low-pass filtered white noise with a cutoff frequency of 300 Hz and is output from a PC via an audio
amplifier with another actuator. These processes generatetactile sensations and simultaneously enhance the sensation near the elbow fossa, resulting reproduction of human fingertip sensation.

4. Sensory enhancement experiment
4.1 Determination of vibration perception threshold
To determine the vibration perception threshold (themaximum vibration that the participant cannot feel) before the experiment, the participant was asked to wear aneye mask, relax with hands on a desk, and present the vibration. The intensity of noise was increased and decreased until the participant was unable to distinguish between it being on or off. The average value was obtainedby repeating each trial three times. The threshold is referred to as 1T.
4.2 Verification of the effect of stochastic resonance

3. Verification of differences in fingertip sensation
We tested whether the tactile sensitivity was affectedby applying white noise near the elbow fossa to one
healthy university student. The participant was asked totouch a piece of sandpaper with his hand and a prosthetic
hand (Fig. 2), and the percentage of correct responses foreach was recorded. Five vibration intensity conditions(0.5 T, 0.75 T, 1 T, 1.25 T, and 1.5 T) were applied randomly. One trial was conducted for each vibration condition with a 30-second rest between trials. The hand andprosthetic hand were tested separately.
Fig. 3 shows the experimental results. The result for
the no-vibration condition (no-vib) was obtained from theexperiment in Section 3. In both cases, the correct response rate tended to be higher than that of the no-vibration. A questionnaire survey of the participant after theexperiment showed that there was no significant difference in the perception of the vibration caused by thewhite noise.
Fig. 4 shows signals recorded from the microphone
while touching sandpapers with the prosthetic hand andtheir FFT results. The threshold value was set to -30 dB,
and the range from the first detection of a signal above
the threshold value to the last detection of a signal above
the threshold value was clipped. The extreme values of
the absolute values of the signals were extracted and averaged to obtain the amplitude. The results show that theroughness of #400 tends to be easily discernible, with an

In this section, we first discuss the difference between
the cutaneous sensation at the fingertips of a person andthe sensation when using a prosthetic hand. In this section,
we first describe the difference between the cutaneous
sensation at the fingertips of a person and the sensation
when using a prosthetic hand. We compared and verified
the sensitivity of the skin sensation at the fingertips whentouching an object and the sensation of the prosthetic
hand near the elbow through the socket.
The participant was a healthy university student seatedon a chair wearing an eye mask and earplugs (Fig. 1) In
the experiment, the participant was given a piece of paperwith five pieces of sandpaper of different roughness(#400, #600, #800, #1000, and #1200.) on the front side
and one randomly selected piece of sandpaper from thefive pieces on the front side on the back side, and asked
to touch all the sandpapers on the front and back sides.
The sandpaper with the same coarseness as the one on thereverse side was selected from the sandpaper on the front
side, and the percentage of correct responses was recorded. The test consisted of one trial in which each roughness was presented twice at random (10 in total). One trial
was performed with the human hand and one trial withthe prosthetic hand.
The discrimination rate of the human hand was 70%,
and that of the prosthetic hand was 10%. These results©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1000

Correct rate [%]Haptic Sensation Enhancement via

Correct rate [%](b) Myoelectric prosthetic hand(a) Human handFig. 2. Scenes from the sensory enhancement experiment.
100
80
60
40
20
0

(a) Experimental scene(b) Experimental result
Fig. 5. Tactile feedback experiment.

5. Simulation of tactile generation near the
elbow fossa
Although we were able to improve the tactile sensitivity of the fingertips by adding white noise near the elbow
fossa in Section 4, the result of the prosthetic hand waslow (30%). Therefore, we conducted an experiment inwhich vibration stimuli were applied near the elbow fossa
based on the measured audio signals.
The participants were two healthy university students.
White noise vibration was added simultaneously withfeedback from the vibratory stimulation of the audio signal using two vibrotactile stimulators attached near the
elbow fossa. The same procedure as in Section 3 was used
to record the percentage of correct responses by touching
the sandpaper with the prosthetic hand in five vibration
intensity conditions.
The results of the experiment are shown in Fig. 5(b),which compares the cases without voice feedback (NoFB, the result shown as No-vib in Section 4), with tactile
feedback only (No-vib), and with tactile feedback andwhite noise vibrations. The results show the average oftwo participants. Except for 0.75T, the result was higher
than that without vibration, and the highest response rate
was obtained when the vibration intensity was 1T. A
questionnaire survey of the participants after the experiment suggested that the correct rate increased because theparticipants were able to feel strong vibrations due to thetactile feedback. These results indicate that some fingertip sensation can be obtained by vibration stimulation
based on audio signals near the elbow fossa and tactile
sensitivity enhancement by white noise vibration.

(a) Human hand(b) Myoelectric prosthetic handFig. 3. Results of the sensory enhancement experiment.
#400

0.2
0.1
0 0
-0.1
-0.2

1
2
Time [s]

0.2
0.1
0
0
-0.1
-0.2

#600
3

0

1

2
Time [s]

#1000
1

2

#800
3

0

1
2
Time [s]

3

#1200
3 0

Time [s]123Time [s](a) Sound waveformSound pressure level Sound pressure level
[dB]
[dB]

100
80
60
40
20
0

#400

0

#600

#800

-50

-100
0

0

10000 20000 0
Frequency [Hz]10000 20000 0
Frequency [Hz]

#1000

#1200

10000 20000 0
Frequency [Hz]10000 20000
Frequency [Hz]10000 20000
Frequency [Hz]

-50

-100

0

(b) FFT
Fig. 4. Recorded signals and their FFT results.

6. Verification of tactile sensation generation with aprosthetic hand

amplitude of approximately 0.021, which is larger thanthe others. In addition, #1000 tended to be misidentified
as #800. This may be because the amplitude of #800 was
about 0.012, close to the amplitude of #1000 of about
0.009, and the frequency band peaks were similar ataround 900 Hz. The amplitude of #600 and #1200 wereapproximately 0.017 and 0.014, respectively. However,#600 had a peak near 2500 Hz and #1200 had a peak near
400 Hz, which may have affected the discriminability ofthe two samples. The identification rate through thesocket of the myoelectric prosthetic hand was 30%.

Based on the results obtained in Section 5, an experiment was conducted with a healthy university student as
a participant to verify whether tactile feedback and white
noise vibration could provide a realistic sensation whentouching objects with a prosthetic hand. In the experiment,
three different textures of cloth (nylon, polyester, and cotton) were traced with the prosthetic hand to test whether
it could reproduce the sense of touch.
From the experimental results, when the participant
touched the cloth with the prosthetic hand, the participant
could not feel much difference in texture and could not
reproduce the tactile sensation. Fig. 6 shows the soundwaveforms and FFT results of touching the cloth with the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1001

Yoshitaka Mizumoto, Taro Shibanoki
CottonSound pressure level
[dB]

0.2
0.1
0 0
-0.1
-0.2

1
2
Time [s]

Nylon
3

0

1
2
Time [s]Polyester
30

12Time [s]hul R.Kaliki, Nitish V.Thakor, “Prosthesis with neuromorphic multilayered edermis perceives touch andpain”, Sci. Robot., Vol. 3, No. 19 (2018), pp. 57-61.
7. Alessia Silvia Ivani, Federica Barontini, Manuel G.
Catalano, Giorgio Grioli, Matteo Bianchi, Antonio
Bicchi, “VIBES: Vibro-Inertial Bionic Enhancement
System in a Prosthetic Socket”, 2023 International
Conference on Rehabilitation Robotics, 2023.
8. Christian Antfolk, Anders Bjorkman, Sven-Olof Frank,
Fredrik Sebelius, Goran Lundborg, Birgitta Rosen,“Sensory feedback from a prosthetic hand based on airmediated pressure from the hand to the forearm skin”,
Journal of rehabilitation medicine, Vol. 44, No.
8(2012), pp. 702-707.
9. Kouta Minamizawa, Yasuaki Kakehi, Masashi Nakatani, Soichiro Mihara, Susumu Tachi, “TECHTILE
toolkit: a prototyping tool for designing haptic media”,
ACM SIGGRAPH 2012 Emerging Technologies, pp.
387-392(2012).
10. Jianyao Zhang, Hiroyuki Kajimoto, “A Robust Approach for Reproducing the Haptic Sensation of Sandpaper With Different Roughness During Bare Fingertip
Interaction”, ERONTIERS IN VIRTUAL REALITY, Vol.
3(2022).
11. Marius Dettmer, Amir Pourmonghaddam, Beom-Chan
Lee, Charles S Layne, “Effects of aging and tactile stochastic resonance on postural performance and posturalcontrol in a sensory conflict task”, Somatosens Mot Res,
Vol. 32, No. 2(2015), pp. 128-135.
12. Carly C. Sacco, Erin M. Gaffney, Jesse C. Dean, “Effects of White Noise Achilles Tendon Vibration on
Quiet Standing and Active Postural Positioning”, J
Appl Biomech, Vol. 34, No. 2(2018), pp. 151-158.
13. Leah R Enders, Pilwon Hur, Michelle J Johnson, Na Jin
Seo, “Remote vibrotactile noise improves light touchsensation in stroke survivors’ fingertips via stochastic
resonance”, Journal of NeuroEngineering and RehabilitationI, 10:105(2013).
14. Komi Chamnongthai, Takahiro Endo, Shohei Ikemura,
Fumitoshi Matsuno, “Stiffness Discrimination by Two
Fingers with Stochastic Resonance”, International
Conference on Human Haptic Sensing and Touch Enabled, Springer, Cham, 2020.

3(a) Sound waveform
Cotton0

Nylon

Polyester

-50

-100

0

10000 20000 0
10000 20000 0
10000 20000
Frequency [Hz]Frequency [Hz]Frequency [Hz](b) FFT
Fig. 6. Recorded signals and their FFT results.
prosthetic hand. The amplitudes of both results were almost the same (0.006) and the peak frequency bandwidths were similar at around 100 Hz, suggesting that the
feedback vibration stimuli were similar and therefore
could not be determined.

7. Conclusion
This paper showed that vibrotactile feedback and stochastic resonance effects near the elbow fossa can provide some fingertip sensation when touching an object
through a prosthetic hand. However, it is difficult to distinguish objects with small amplitude or similar frequency bands. In the future, we plan to improve the feedback method to detect slight differences in vibration.
Acknowledgements
A Robust Approach for Reproducing the Haptic Sensation of Sandpaper With Different Roughness During
Bare Fingertip Interaction.
References
1. Kairu Li, Yinfeng Fang, Yu Zhou, Honghai Liu, “NonInvasive Stimulation-Based Tactile Sensation for Upper-Extremity Prosthesis: A Reviw”, IEEE Sens. J, Vol.
17, No. 9(2017), pp. 2625-2635.
2. Larisa Dunai, Martin Novak, Carmen Garcia Espert,“Human Hand Anatomy-Based Prosthetic Hand”, Sensors 2021, 21(1), 137.
3. Luis Vargas, He Huang, Yong Zhu, Xiaogang Hu, “Object Recognition via Evoked Sensory Feedback during
Control of a Prosthetic Hand”, IEEE Robotics and Automation Letters, Vol. 7, No. 1(2022), pp. 207-214.
4. John W. Hill, “A Describing Function Analysis ofTracking Performance Using Two Tactile Displays”,
IEEE Transactions on Man-Machine Systems, Vol. 11,
No. 11(1970), pp. 92-101.
5. Yuki Ueda, Chiharu Ishii, “Development of a feedback
device of temperature sensation for a myoelectric prosthetic hand by using Peltier element”, International
Conference on Advanced Mechatronic Systems, pp.
488-493, 2016.
6. Luke E.Osborn, Andrei Dragomir , Joseph L.Betthauser, Christopher L.Hunt, Harrison H.Nguyen, Ra-

Authors Introduction
Mr. Yoshitaka Mizumoto
He received the B.E. degree from Okayama University in 2022. He is currently
a master course student in Okayama University His research interests focus on
myoelectric prosthetic hand©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1002

Powered by TCPDF (www.tcpdf.org)

Haptic Sensation Enhancement via

Dr. Taro Shibanoki
He received the D.Eng. degree from Hiroshima University in 2012. He was a Research Fellow of the JSPS since 2011, an
Assistant Professor in Hiroshima University since 2013 and an Assistant Professor,
Lecturer, and an Associate Professor in
Ibaraki University since 2014, respectively. He is currently an Associate Professor in Okayama University. His current research interests focus on human-machine interfaces.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1003

PID Parameter Tuning of a Low-Cost DC Motor Speed Control for Mobile Robot Application
Munkh-Erdene Ayurzana

Mongolian University of Science and Technology, 8 th khoroo, Baga toiruu 34, Sukhbaatar district Ulaanbaatar,
Mongolia 14191
Erkhembayar Gankhuyag

Mongolian University of Science and Technology, 8 th khoroo, Baga toiruu 34, Sukhbaatar district Ulaanbaatar,
Mongolia 14191
Dondogjamts Batbaatar

Mongolian University of Science and Technology, 8 th khoroo, Baga toiruu 34, Sukhbaatar district Ulaanbaatar,
Mongolia 14191
Naranbaatar Erdenesuren

Mongolian University of Science and Technology, 8 th khoroo, Baga toiruu 34, Sukhbaatar district Ulaanbaatar,
Mongolia 14191
Email: B232340036@must.edu.mn
Abstract
Precise control of DC motors is vital for robotics, industrial automation, and mechatronics. Traditional PID(Proportional-Integral-Derivative) control methods, while widely used, often require offline parameter tuning, which
can be time-consuming and suboptimal for real-time applications. This paper proposes a GUI (Graphical User
Interface) based approach for PID tuning of a DC motor model. The proposed method utilizes a MATLAB-based
parameter estimation model with GUI to continuously monitor and update PID controller parameters based on realtime data from the Arduino-controlled DC motor setup.
Keywords: PID parameter tuning, Graphical user interface (GUI), DC motor control

combining PID controllers with other methods like fuzzy
logic control, genetic algorithms [1],1. Introduction
Despite its simplicity, Proportional-IntegralDerivative (PID) control remains remarkably efficient,offering a three-term solution to address both transientand steady-state responses in numerous real-world
problems. Since its invention in 1910, largely attributed
to Elmer Sperry's ship autopilot, and the development ofstraightforward tuning methods by Ziegler and Nichols in1942 [1], [2]. PID control has witnessed widespread
adoption [3]. DC motors are widely employed as
actuators for wheeled mobile robots, robots controlled byprogrammed microcontrollers, or computers thatnavigate autonomously based on sensor information.
These robots typically have two, three, or four wheels,
each driven by a DC motor for smooth maneuverability[4], [5], [6]. Two primary control applications for DC
motors are speed and position control. Speed controllers
regulate robot movement speed, while position
controllers ensure accurate movement to predetermined
positions. [7], [8]. Recent research on DC motor speed
control focuses on stabilization, with many works

and particle swarm optimization. Similarly, PIDbased position controllers have been combined withartificial intelligence techniques like fuzzy logic andgenetic algorithms to enhance accuracy [9], [10], [11].
While PID control exhibits remarkable effectiveness,
optimal tuning remains a challenge, particularly indynamic environments encountered by wheeledmobile robots. However, these approaches often lackuser-friendliness and adaptivity.
This study utilizes one of the DC motors from a twowheeled mobile robot as the plant. The DC motor'smathematical model is developed by acquiring input
and output signals from an open-loop experiment [10].
The applied working voltage serves as the input signal,and the electronically measured speed is the output
signal. The resulting model's step response iscompared to the original motors to validate its
accuracy. A PID-based speed controller is proposed,with initial parameters obtained from the relay
feedback experiment and MATLAB based PID©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1004

Munkh-Erdene Ayurzana, Erkhembayar Gankhuyag, Dondogjamts Batbaatar, Naranbaatar Erdenersuren

tuning integrated with GUI. These parameters arethen manually fine-tuned based on practical PID
tuning knowledge to achieve optimal performance
without overshoot or steady-state error. The paper is
structured as follows: Section 1 presents the
background and related work, Section 2 outlines the
proposed approach and methodology, Section 3
details the implementation and results, and Section 4
concludes the study and highlights its potential
impact.

2.3.

By torque balance (energy balance) in the system themechanical equations can be stated.
𝑇𝑒 − 𝑇𝜃̈𝑎 − 𝑇𝑏 − 𝑇𝐿 = 0

Mathematical model of DC motor

Before we create a GUI interface, we need to start off
with a Simulink model of the DC motor system with PI
controller. The DC motor has to be modeled first, in
Simulink and the mathematical equations behind thefunctioning of the motor has to be understood before
advancingfurther

𝒅
𝒅𝒕

𝒊𝒂 =

𝟏
𝑳𝒂

𝑑𝑡

∙ (𝒊𝒂 ∙ 𝑹𝒂 + 𝑲𝒆 ∙ 𝜽̇𝒂 − 𝑽𝒊𝒏 )

(10)
3.

Results and Discussion
The experiment's foundation lay in meticulous datacollection, enabling the analysis of the intricate interplay
between voltage and RPM. A 140 second data acquisition
process captured the voltage profile, characterized by a
20-second ramp-up to 24V followed by a 20-second
minimum, and this pattern mirrored the observed RPM
variations with relay feedback experiment was shown in
Fig. 2.

Fig.1 DC Motor model
In order to model a DC Motor in Simulink, we have todefine the mathematical equations on which it functions,
The DC Motor functioning diagram is given in Fig.1.
2.2.

Electrical Characteristics

By using Kirchoff’s voltage law in the DC-motor modelaccording to Fig. 1 DC motor model following equation
is derived
𝑉𝑖𝑛 − 𝑉𝑅𝑎 − 𝑉𝐿𝑎 − 𝑉𝑒𝑚𝑓 = 0(1)

Where, 𝑉𝑖𝑛 , 𝑉𝑅𝑎 , 𝑉𝐿𝑎 and 𝑉𝑒𝑚𝑓 is the input-voltage,
voltage over the armature resistance, voltage drop over
the armature inductance and voltage induced by the coilrespectively. The equations for the voltage parts in the
DC-motor are,𝑉𝑅𝑎 = 𝑖𝑎 ∙ 𝑅𝑎

(2)

𝑑
𝑖
𝑑𝑡 𝑎

(3)

𝑉𝐿𝑎 = 𝐿𝑎

𝑉𝑒𝑚𝑓 = 𝐾𝑒 ∙ 𝜃̇𝑎(5)

Where 𝑇𝑒 is the electromagnetic torque, 𝑇𝜃̈𝑎 , is torquegenerated from the rotational acceleration of rotor. 𝑇𝑏 , isthe torque due to the friction and angular velocity in the
motor. 𝑇𝐿 is the torque of mechanical load (external load),
equations for the first three parts are
𝑻𝒆 = 𝑲 𝒕 ∙ 𝒊𝒂(6)𝒅𝑻𝜽̈𝒂 = 𝑱𝒎 ∙ 𝜽̇𝒂(7)𝒅𝒕
𝑻𝒃 = 𝒃𝒎 ∙ 𝜽̇𝒂(8)
Substituting equation (5) with (6), (7) and (8) yields
following differential equation,𝑑𝑉𝑖𝑛 − 𝑖𝑎 ∙ 𝑅𝑎 − 𝐿𝑎 ∙ 𝑖𝑎 − 𝐾𝑒 ∙ 𝜃̇𝑎 = 0(9)

2. Methodology
2.1.

Mechanical Characteristics [2]Fig.2 Relay output
From the relay feedback experiment, the following
result was obtained.

(4)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1005

PID Parameter Tuning ofcharacterized [12], [13]. This could lead to difficulties intuning the PID controller effectively.

Fig.3 System without PID tuning

Fig. 5 Iteration for finding motor parameters

The open-loop experiment Fig. 3 revealed significant
overshoot and sustained oscillations in the DC motor
speed due to suboptimal PID settings. Manuallyconfiguring the parameters or using the time-consuming
Ziegler-Nichols method might not be ideal for real-time
applications.

As shown in Fig. 4 the measured data overlaid with thesimulated data. The simulated data comes from the modelwith the estimated parameters listed in Table 1.
Comparing the response of the system before and after
the estimation process clearly shows that the estimation
successfully identified the model parameters and thesimulated response accurately matches the experimental
data.

3.1 Parameter estimation and tuning

Table 1. DC Motor parameters
From the mathematical model of DC motor following
initial parameters are to be known such as moment ofinertia of the rotor Jm, damping ratio of the mechanicalsystem bm, electric resistance Rm, electric inductance Lm,
armature constant Kt, before being tuned with a PID
controller. While manufacturers may provide values forsome of these DC motor parameters, they are only
approximate. Its necessary to estimate these parameters
as precisely as possible for our model to ascertain
whether it is an accurate representation of the actual DC
motor system.

Constants
𝐾𝑡
𝑅𝑚
𝐿𝑚
𝑏𝑚
𝐽𝑚

Value
0.0035401
0.0065388
8.758e-05
0.0001263
4.3114e-05

Unit[𝑁 ∙ 𝑚/𝐴]
[𝑂ℎ𝑚]
[𝐻]
[−]
[𝑘𝑔 ∙ 𝑚2 ]While from PID tuning result following initial PID
parameters are determined in Table 2.
Table 2. Parameters used in the numerical simulation.

Controller
PID

PID Parameters
𝐾𝑝
𝐾𝑖
0.01
0.061

𝐾𝑑
0

After the fine-tuning process in MATLAB, it was foundthat the final Kp = 0.01, Ki = 0.061 and Kd = 0. The initialvalue of Kp = 0.9 gives big oscillation output. The finetuning is performed to adjust the Kp, such that the P
controller output results in only one overshoot and no
oscillation, although steady state error is still not zero.
This final value of Kp is 0.01.
Fig.4 Parameter estimation plot
Again, these values are of important for designing andtuning a PID controller for the DC motor. The PID
controller will use these parameters to calculate themotor's input voltage in order to achieve the desired
speed. Furthermore, without knowing the specific valuesof the parameters, it is difficult to say much about the
specific performance of the DC motor. However, the factthat these are estimated parameters (Fig. 5) suggests thatthe motor has not been precisely modeled or©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1006

Munkh-Erdene Ayurzana, Erkhembayar Gankhuyag, Dondogjamts Batbaatar, Naranbaatar Erdenersuren

First of all, getting a proper data is one of the mostimportant steps. Based on the collected data, picking thecorrect model is directly affects the PID tuning response
and therefore directly affects the possibilities of tuning.
At last, figuring the optimal response through MATLABPID tuner directly affects the quality of controlling.
b) GUI for standalone application
Once identification and tests are complete, user can shut
down MATLAB and use a standalone application asshown in Fig. 8 to control the DC motor. Standalone
application requires less resources to run compared tomultistep manual identification.
Fig.6 System with PID tuning
The Fig. 6 illustrates the response of a DC motor speed
control performance after it has been tuned with PID
controller. The blue line shows the reference input, thered line shows the actual output speed of the motor, which
initially lags behind the reference input and then abruptly
rise to meet it. There are some oscillations in the motor
speed, which indicates that the system is damped due
external disturbances or externally given mechanical load.
The system is returned to stable state approximately after
1 second under the varying operating conditions.

Fig.8 Speed controller performance at final stage
with standalone application

3.2. GUI for motor control

In Fig. 8 graph depicts the reference tracking responseof the DC motor after it has been tuned with the PID
controller. The blue line shows the reference input, which
is direction of motor changes with time. The red line
shows the actual output of a motor, which closely tracks
the reference input with minimal overshoot, and settling
time, and there is no steady-state error. Finally, ananalysis was performed to ensure that the controller was
well-tuned with GUI of standalone application.

In order to simplify the dc motor control process, a GUI
is developed that allows a user to quickly identify model,
tune controller coefficients and send reference data wascreated. Two types of GUIs were created.
a)

GUI for identification

4. Conclusion
This project developed an interactive GUI to optimize
PID controller parameters for a DC motor SIMULINK
model in real-time. This approach enables continuous
adaptation to varying operating conditions, achieving
significant improvements compared to traditional offlineor manual tuning methods. Experimental results
demonstrate enhanced control accuracy and systemstability, confirming the effectiveness of the proposed
method. The PID controller, adjusted through the GUI,
optimizes the motor's input voltage to minimize the error
between the desired and actual output. This is achievedby utilizing a parameter estimation tool based on the DC
motor's mathematical model. Careful tuning of the PID
gains ensures a system that is both responsive and stable.
The developed GUI serves as a valuable tool for robot
designers and researchers, enabling real-time control
optimization and hands-on learning of control systems
concepts. It offers undergraduate students a practicallearning experience in the classroom, fostering theirunderstanding of PID control and its applications.

Fig.7 Identification GUI MATLABThis GUI (presented on Fig. 7) allows user to control
motor speed manually, send identification data, identify
model, tune PID, set object constants and deploy modelwith new coefficients and constants to Arduino. After
uploading a model to Arduino, user can check the qualityof tuned controller by reconnecting to Arduino, settingreference velocity manually and checking the response.
The quality of controlling depends on several factors.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1007

Powered by TCPDF (www.tcpdf.org)

PID Parameter Tuning of

Authors Introduction
Mr. Munkh-Erdene Ayurzana
He is currently studying for his
Bachelor’s degree in MechanicalEngineering at the Mongolian Universityof Science and Technology (MUST). His
research interests include control system
and robotics.

Acknowledgements
This work was supported in part by Mongolia-Japanhigher Engineering Education Development (MJEEDJICA) through the joint research project “Development ofindustrial, service and intelligent systems based on
advanced technologies” (J23A16).

Mr. Erkhembayar Gankhuyag
He is currently studying for his Bachelor’s
degree in Mechatronics Engineering at theMongolian University of Science andTechnology (MUST).
His researchinterests include control system andmechatronics.

References
1.

2.

3.
4.

5.

6.

7.
8.

9.

10.

11.
12.

13.

Astrom, K. J. and T., Hagglund, PID Controllers: Theory,
Design and Tuning, ISA, Research Triangle, Par, NC,(1995)S. K. Suman and V. K. Giri, "Speed control of DC motorusing optimization techniques based PID Controller,"
2016 IEEE International Conference on Engineering andTechnology (ICETECH), Coimbatore, India, 2016, pp.
581-587
Visioli, Antonio. (2012). Research Trends for PID
Controllers. Acta Polytechnica. 52. 144-150.
Supriyo, B., Suharjono, A., & Atmaja, K. W. (2021).
Modelling and Control of DC Motor Speed and Position
for Wheel Mobile Robot Application. Proceedings of theInternational Conference on Innovation in Science andTechnology (ICIST 2020), 80–84.
Fu, Yuheng & Zhou, Qinyou. (2023). Analysis andapplication research of mobile robot navigation related
technologies. Applied and Computational Engineering. 9.
92-96.
Khan, H.; Khatoon, S.; Gaur, P.; Abbas, M.; Saleel, C.A.;
Khan, S.A. Speed Control of Wheeled Mobile Robot by
Nature-Inspired Social Spider Algorithm-Based PID
Controller. Processes 2023, 11, 1202.
K. Ogata (2009) “Modern Control Motorering, 5th
Edition”, Dorling Kindersley,”.
Kiam Heong Ang, G. Chong and Yun Li, "PID control
system analysis, design, and technology," in IEEE
Transactions on Control Systems Technology, vol. 13, no.
4, pp. 559-576, July 2005
Vikhe, Pratap & Punjabi, Neelam & Kadu, C B. (2015).
DC Motor Speed Control Using PID Controller In Lab
View. IJISME. 3. 38-41.
Vikhe, Pratap & Punjabi, Neelam & Kadu, C B. (2014).
Real Time DC Motor Speed Control using PID Controllerin LabVIEW. International Journal of Advanced Researchin Electrical, Electronics and Instrumentation Engineering.
03. 12162-12167.
Messaadi, Nassim & Amroun, Abdelkader. (2021). Speed
Control of DC Motor Using Fuzzy PID Controller.
Nilawar, Vaishnavi & Sonare, Aniket & Samrutwar,
Shrirup. (2022). Implement PID using MATLAB for
motion control application.
Abdul-Kareem Mohammed, J. (2011). Modeling, Analysis
and Speed Control Design Methods of a DC Motor.
Engineering and Technology Journal.

Dr. Dondogjamts Batbaatar
He received his M.S. in the field of
mechatronics from Mongolian Universityof Science and Technology (MUST),
Mongolia and Ph.D. degree from Kyushu
Institute of Technology, Japan in 2015
and 2021. He is currently a vice dean of
research affair in School of MechanicalEngineering Transportation in MUST. His researchinterests include computational non-linear dynamics andbio-inspired robotics.
Dr. Erdenesuren Naranbaatar
He received his M.S. in the field of
mechatronics from Mongolian Universityof Science and Technology (MUST),
Mongolia and Ph.D. degree from
University of Ulsan, South Korea in 2002
and 2013. he is currently an Associate
Professor of the Graduate department in
School of Mechanical Engineering Transportation in
MUST. His research interests include computer vision androbotics.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1008

Reinforcement Learning DDPG Algorithm Based Wheeled Mobility Aid Robot Control
Methods
Junkai Li, Mohd Rizon Mohamad Juhari, Tiang Sew Sun

Faculty of Engineering, Technology and Built Environment, UCSI University, Kuala Lumpur 56000, Malaysia
E-mail: 1002162416@ucsiuniversity.edu.my, mohdrizon@ucsiuniversity.edu.my, tiangss@ucsiuniversity.edu.mywww. ucsiuniversity.edu.my

Abstract
When using wheeled walker robots to help individuals with limited walking ability improve their mobility, thestability of the robot's motion control and trajectory tracking accuracy are critical. In this paper, a new trajectory
tracking method for wheeled walking robots is proposed by combining the Deep Deterministic Policy Gradient(DDPG) algorithm in reinforcement learning with a Proportional Integral Differential (PID) controller. The article
first analyzes the kinematic model of the chassis of the wheeled walking robot, and then introduces the designprinciple and structure of the adaptive PID controller that combines the DDPG algorithm and the PID controller.
Finally, the effectiveness of the research scheme and control strategy is verified by joint simulation experiments, andthe results show that this DDPG-based PID controller can automatically adjust the parameters when tracking the
trajectory to ensure the accuracy of the trajectory, and it has a strong anti-interference capability.
Keywords: Wheeled helper robot, trajectory tracking, DDPG algorithm, PID controller.

gravity is not stable and caused by the fall and othersecondary injuries, but also in a narrow space to achieve
any direction of movement, to give the patient a good
sense of spatial movement and real walking experience,wheeled walking robot with its stable chassis and flexible
spatial movement ability, in terms of safety andpracticality with the Advantages.
When using wheeled walking robots for rehabilitation
training, due to the complexity of the external
environment, the wheeled walking robots are required tofollow the desired trajectory as much as possible, which
requires the robots to be able to track an ideal trajectorywith a time function according to the actual position andmotion state, and complete the tracking of each point in
the trajectory according to the time requirements
Therefore, this paper applies the reinforcement learningalgorithm to the traditional proportional-integralderivative (PID) controller to realize adaptive PID

1. Introduction
In recent years, with increasing aging, the elderly show
significant physiological decline, limb flexibility andother basic abilities, increasing the risk of falls.
Meanwhile, as people's lifestyles change, there are more
and more lower limb motor dysfunctions caused by
sports injuries, traffic accidents, or diseases, making
there a huge demand for intelligent assistive devices [1].
Walking robots for the elderly and patients with lower
limb motor dysfunction can be categorized into two
types: exoskeleton robots [2] and wheeled assisted
walking robots [3]. However, the center of gravity
control of most exoskeleton rehabilitation robots is notsatisfactory, and the safety of the rehabilitation process
cannot meet the requirements [4]. Wheeled walking
robot itself has a stable chassis, not only can well avoid
the patient in the training process due to the center of©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1009

Junkai Li, Mohd Rizon Mohamad Juhari, Tiang Sew Sun

parameter tuning in order to adjust the robot position
more accurately so that the robot can complete trajectory
tracking.
Although the traditional PID control has the advantagesof simplicity of use, easy to implement, no static error,
etc., its disadvantage is that it cannot realize the online
adjustment of parameters, so when it encounters a strong
interference, it is bound to have the phenomenon of
prolonged recovery time and increased overshooting,which affects the stability of the motion of the chassis ofwheeled robots. A large number of scholars haveconducted research on adaptive PID, and introduced theidea of online parameter adjustment in the traditional PID,
which improves the response speed of the system. At
present, adaptive PID control methods mainly include:
fuzzy PID controller [5], the method requires a large
amount of a priori knowledge, there are parameter
optimization problems; neural network-based adaptive
PID control [6], the method can be achieved without
identifying the complex nonlinear system to achieveeffective control, but in the use of supervised learning to
optimize the parameters, the acquisition of the teacher'ssignals is relatively difficult; evolutionary algorithm
adaptive PID controller [7], the method, although theacquisition of a priori knowledge of the lower
requirements, but the computation time is longer, it isdifficult to achieve real-time control in practicalapplications; reinforcement learning adaptive PID
controller [8], proposed the Actor-Critic algorithm toachieve adaptive tuning of PID parameters, which makes
use of AC algorithms of model-free online learning The
algorithm utilizes the model-free online learning
capability of AC algorithm, but the convergence speed of
AC algorithm is slow and the training time is long.
In this paper, we adopts the deep deterministic policy
gradient (DDPG) algorithm, which is based on the ActorCritic (AC) framework, to enhance the performance of
the PID controller and the deep Q-network (DQN)
algorithm is added on the basis of the deterministic policy
gradient (DPG) algorithm. This algorithm can not only
update in a single step like DQN, but also has theadvantages of high data utilization and fast convergenceof DPG. In order to realize the adaptive adjustment of
PID parameters, a PID controller based on DDPG
algorithm is proposed. In the simulation experiments, thekinematic model of the omnidirectional chassis of thewheeled walking robot in our laboratory is used to verify
the superiority and generality of the proposed method.

controller that combines the DDPG algorithm and the
PID controller.
2.1. Kinematic modeling of assistive robots
The schematic and kinematic model of the robot's
omnidirectional wheel chassis is shown in Fig. 1. Taking
the world coordinate system 𝑥𝑤 𝑜𝑤 𝑦𝑤 as a reference, therobot moves in the motion coordinate system 𝑥𝑜𝑦 with avelocity of magnitude 𝑣 toward an angle 𝛼 from the 𝑌𝑤
axis. 𝑉𝑥 , 𝑉𝑦 denote the horizontal and vertical travel
speeds of the robot relative to the motion coordinate
system, respectively [9].

Fig. 1 Kinematic modeling of assistive robots
When the robot is moving in any direction, e.g., at anangle 𝛼 from the 𝑦 -direction with velocity 𝑣 , themagnitudes of the horizontal and vertical velocities of therobot respectively:
𝑉𝑥 = 𝑣𝑠𝑖𝑛𝛼
𝑉𝑦 = 𝑣𝑐𝑜𝑠𝛼( 1)

Let the angle between the omnidirectional wheels and the𝑥-axis, measured from the center of the robot chassis, be𝜃 ,and the distance from the center of the chassis to thefour wheels be 𝑎 , then the wheel speed of eachomnidirectional wheel is degree is:𝑉1 = −𝑠𝑖𝑛𝜃𝑉𝑥 + 𝑐𝑜𝑠𝜃𝑉𝑦 + 𝜃𝑎̇𝑉2 = 𝑠𝑖𝑛𝜃𝑉𝑥 + 𝑐𝑜𝑠𝜃𝑉𝑦 − 𝜃𝑎̇𝑉3 = 𝑠𝑖𝑛𝜃𝑉𝑥 + 𝑐𝑜𝑠𝜃𝑉𝑦 − 𝜃𝑎̇𝑉4 = −𝑠𝑖𝑛𝜃𝑉𝑥 + 𝑐𝑜𝑠𝜃𝑉𝑦 + 𝜃𝑎̇

(2)

Where 𝑉1 , 𝑉2 , 𝑉3 , 𝑉4 denote the linear velocities of thefour omni-directional wheels respectively and 𝜃̇ denotes
the angular velocity of the robot writing Eq. (1) in matrix
form gives:

2. Proposed Method
This section will introduce the kinematic model of thewheeled walking robot chassis. Then, it will describe thedesign principles and structure of the adaptive PID©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1010

Reinforcement Learning DDPG Algorithm

𝑉1
−𝑠𝑖𝑛𝜃
𝑉2
𝑐𝑜𝑠𝜃
[ ]=[
𝑉3
−𝑠𝑖𝑛𝜃
𝑉4
𝑐𝑜𝑠𝜃

𝑐𝑜𝑠𝜃
𝑠𝑖𝑛𝜃
𝑐𝑜𝑠𝜃
𝑠𝑖𝑛𝜃

𝑎
𝑎
]
𝑎
𝑎

(3)

𝑎
𝑎
]
𝑎
𝑎

(4)

environment more stable, which in turn accelerates theconvergence of the algorithm.
In summary, the incremental PID algorithm is used torealize the trajectory tracking control of the wheeled
robot, which requires the design of two PID controllers,i.e., the transverse position 𝑋 controller and thelongitudinal position 𝑌 controller, and the block diagramof the control system is shown in Fig. 2. The input of thetransverse PID controller is 𝑥𝑒 , i.e., the transverse
deviation of the wheeled robot system; the output is 𝑣𝑥 ,the transverse speed of the wheeled robot system. theoutput of the PID controller goes to the speed distribution
controller, which calculates the speeds of the four
omnidirectional wheels according to Eq. (2) to correct thetransverse deviation of the wheeled robot. The principleof longitudinal position 𝑌 control is the same as theprinciple of transverse position X control, i.e., 𝑦𝑒 is thelongitudinal error, 𝑣𝑦 is the longitudinal velocity.

Let the transformation matrix 𝑅 be:
−𝑠𝑖𝑛𝜃
𝑐𝑜𝑠𝜃
𝑅=[
−𝑠𝑖𝑛𝜃
𝑐𝑜𝑠𝜃

𝑐𝑜𝑠𝜃
𝑠𝑖𝑛𝜃
𝑐𝑜𝑠𝜃
𝑠𝑖𝑛𝜃

Let the positional attitude of the robot at 𝑌𝑤 𝑂𝑤 𝑋𝑤 be 𝑃 =[𝑥, 𝑦, 𝜃]𝑇 , and the differential of this positional attitude
be 𝑃̇ , and the velocities of each of the robot's wheels𝑉𝑏 =[𝑉1 , 𝑉2 , 𝑉3 , 𝑉4 , ], then the equations of kinematics of therobot's differential omnidirectional wheels are:𝑃̇ = 𝑅 −1 ∙ 𝑉𝑏

(5)

2.2. Incremental PID Control Principle
Digital PID control can be categorized into two types:
positional PID and incremental PID. incremental PID
does not require the use of the accumulated value of pastdeviations, which effectively reduces the system's
computational error. Incremental PID is an algorithm thatrealizes PID control by controlling the increment. The
formula is as follows:

Fig.2 Block diagram of two PID controller systems

2.3. DDPG algorithm
Reinforcement learning is a machine learning methodthat learns behavioral strategies through trial and error,
while deep learning is a machine learning method that
performs high-level abstraction and feature extraction
throughmulti-layer
neuralnetworks.
Deep
Reinforcement Learning combines the feature extraction
capability of Deep Learning with the decision-making
capability of Reinforcement Learning to autonomously
learn and improve the performance of decision-making
strategies. DDPG (Deep Deterministic Policy Gradient)algorithm is an optimization algorithm based on the
Actor-Critic algorithm [5] framework, which is able tobetter select the optimal policy in continuous actions suchas robot movement. The DDPG algorithm is based ondeterministic policy gradient and refers to the experience
pool replay mechanism and Double-Depth Q-Network(DDQN) objective network method to update the
network parameters and realize the self-tuning of PID
parameters, and its algorithm structure is shown in Fig. 3.

𝑢(𝑡) = 𝑢(𝑡 − 1) + 𝐾𝑖 (𝑡)𝑒(𝑡) + 𝐾𝑝 ∆𝑒(𝑡) + 𝐾𝑑 (𝑡)∆2 𝑒(𝑡) (6)𝑒(𝑡) = 𝑦𝑑 (𝑡) − 𝑦(𝑡)∆𝑒(𝑡) = 𝑒(𝑡) − 𝑒(𝑡 − 1)∆2 𝑒(𝑡) = 𝑒(𝑡) − 2𝑒(𝑡 − 1) + 𝑒(𝑡 − 2)(7)

In these equations, 𝑦𝑑 (𝑡) denotes the current actual
signal value, 𝑦(𝑡) denotes the current system output
value of the current system, 𝑒(𝑡) denotes the output errorof the system, ∆𝑒(𝑡) denotes the first difference of theerror, and ∆2 𝑒(𝑡) denotes the second difference of theerror.
The use of incremental PID controllers in control
system design can optimize the use of computational
resources. The method relies only on the last threesampling points to determine the control increments, thus
reducing the computational burden and storage
requirements. This facilitates fast training of the DDPG
algorithm and storage of sample data. In addition, theincremental PID controller ensures stable operation in theevent of a system failure, and since its output is thechanging value of the control quantity, it makes the
rewards obtained in a reinforcement learning©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1011

Junkai Li, Mohd Rizon Mohamad Juhari, Tiang Sew Sun

Then update the Critic evaluation network according to∇𝑎 𝑄(𝑠, 𝑎|𝜃 𝑄 ), and at the same time pass ∇𝑎 𝑄(𝑠, 𝑎|𝜃 𝑄 ) to
the actor, update the actor evaluation network according
to Eq. (9), and finally update the parameters of the target
network as follows:′𝜃 𝑄 ← 𝜏𝜃 𝑄 + (1 − 𝜏)𝜃 𝑄′{ 𝜇′𝜃 ← 𝜏𝜃𝜇 + (1 − 𝜏)𝜃𝜇′Fig.3. Block diagram of DDPG algorithm structure
Actor gives a new parameter 𝜇 according to the strategy
network, denoted as 𝑎𝑡 = 𝜇(𝑆𝑡 |𝜃𝜇 ) + 𝑁𝑡 , where 𝜃𝜇denotes the parameter of the neural network, 𝐺𝑘 denotes
the noise.After one update iteration, the state changes
from 𝑆𝑡 to 𝑆𝑡+1 , and obtains the reward 𝑅𝑡 .
(𝑆𝑡 , 𝐴𝑡 , 𝑆𝑡+1 , 𝑅𝑡 ) is stored in the experience pool. Finally,
samples are drawn from the experience pool to train thestrategy network and evaluation network.
The strategy network and evaluation network are
updated as shown in Fig. 4. The actor and the Critic
consist of two identical networks, denoted as: actor
evaluation network 𝜇(𝑠|𝜃𝜇 ), actor estimation network
𝜇 ′ = (𝑠𝑡 |𝜃𝜇′ ) , Critic evaluation network 𝑄(𝑠, 𝑎|𝜃 𝑄 ) ,Critic estimation network 𝑄′(𝑠, 𝑎|𝜃 𝑄′ ).

(11)2.4. General Structure of Adaptive PID Controller
Based on DDPG Algorithm
The design idea is to combine the DDPG algorithm on
the basis of incremental PID controller, so the structure
design is shown in Fig. 5.

Fig.5. adaptive PID controller based on DDPG algorithm
The system block diagram is based on the traditional
PID with the addition of closed-loop control by the
DDPG algorithm. The speed allocation module allocatesthe speeds of the four wheels according to Eq. (2), andthe scores are obtained through a self-designed reward
function, and output to the DDPG algorithm. According

Fig.4. updating network parameters
The actor evaluation network updates the parameters
according to the following objective function:𝜇=1∑(∇𝑎 𝑄(𝑠, 𝑎|𝜃 𝑄 )|𝑆=𝑠𝑡,𝐴=𝜇(𝑆𝑡) ∙ ∇𝜃𝜇 𝜇(𝑠|𝜃𝜇 )|𝑆=𝑠𝑡 ) (8)𝑁𝑡where ∇𝑎 𝑄(𝑠, 𝑎|𝜃 𝑄 ) is the gradient of the Critic
evaluation network, and the gradient is updated bycomputing the loss function. The loss function iscomputed as follows:𝐿=

12∑(y𝑡 − 𝑄(𝑠𝑡 , 𝑎𝑡 |𝜃 𝑄 ))
𝑁

(9)

𝑡

where 𝑦𝑡 is referred to as the TD target, which consists ofan immediate reward and an estimated discounted reward.
The formula is as follows:′′y𝑡 = r𝑡 + 𝛾𝑄′(𝑠𝑡+1 , 𝜇 ′ (𝑠𝑡+1 |𝜃𝜇 )|𝜃 𝑄 )(10)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1012

Reinforcement Learning DDPG Algorithm

to the state obtained by the system output, the DDPG
algorithm continuously carries out trial and error training
until it selects the action group with the highest score,
which is the optimal PID parameter, and outputs it to thetwo PID controllers, adaptively adjusts the parameter,
and the error under the parameter is minimized, and theparameter outputs it to the speed allocation module, andthen outputs the speeds of the four wheels to the wheeledwalking robots' chassis system, which realizes the
trajectory tracking control. The controller uses theparameter vector of the PID controllers, i.e. the 6parameters to be tuned by the two PID controllers, as theaction space.
Critics rate each reinforcement learning cycle using thesystem's state variables and the defined reward function
𝑅𝑡 to generate the TD error 𝛿𝑇𝐷 (𝑡) and evaluate the valuefunction 𝑄𝑡 , where 𝛿𝑇𝐷 (𝑡) is provided directly to theactor and Critic, and the reward 𝑅𝑡 is used to evaluate thequality of the current behavior.
𝑅1 = −0.00001
𝑅2 = −10, 𝑒(𝑡) ≥ 1 × 10−5
𝑅3 = 0, 𝑒(𝑡) < 1 × 10−5
𝑅4 = −200, 𝑒(𝑡) > 0.5{𝑅𝑡 = 𝑅1 + 𝑅2 + 𝑅3 + 𝑅4

Table 1. DDPG Algorithm Parameters
Parameter
Value
Actor learning rate
0.005
Critic learning rate
0.001
Discount rate
0.99
Soft update parameter
0.001
Total training steps
50000
Experience pool capacity 1000000

Table 2. PID controller Parameters
Parameter

Value

𝐾𝑝1

5.0

𝐾𝑝2

5.0

𝐾𝑖1

2.0

𝐾𝑖2

2.0

𝐾𝑑1

0.2

𝐾𝑑2

0.2

(12)
4. ResultExperiment shows the simulation results of trajectory
tracking realized by traditional PID controller in the Fig.
6. The solid line shows the desired trajectory and thedashed line shows the tracked trajectory. It can be seenthat the method can also realize the trajectory tracking,the overshoot is 4.5%, and the tracking trajectory is far
away from the target trajectory, and the error is large. Fig.
7 shows the simulation results of trajectory tracking
control using the PID controller based on DDPG
algorithm proposed in this paper. The overshoot of thismethod is 2.7%, and the trajectory is closer to the target
trajectory with less error than the traditional PID control.

𝑅1 is the reward for the speed of the four wheels, which
is set to a score that has less impact on the system since
the main reward comes from the error rather than theamount of control. 𝑅2 and 𝑅3 denote the scores when theerror is lower or higher than the error tolerance interval,
respectively, and 𝑅4 denotes the score of -200 when thesystem error is more than 0.5. Finally, the four scores are
summed up to form the final reward function. Based on
the continuous trial-and-error training, the maximum
reward value is obtained, and thus the optimal PID
parameter values are obtained.

3. Simulation experiment
In this paper, trajectory tracking control simulation
experiments are carried out in python environment.
Based on the chassis modeling of the wheeled walking
robot, the system simulation model was established in
Simulink. In the simulation process, the initial position of
the controlled object is set as [0,1], the desired trajectory
is set as a cosine curve along the x-axis, and the samplingtime is set as 1ms. The parameters of the adaptive PID
control based on the DDPG algorithm are listed in Table
1 and Table 2.

Fig.6 Traditional PID for trajectory tracking©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1013

Junkai Li, Mohd Rizon Mohamad Juhari, Tiang Sew Sun

trajectory tracking control. The method has good
generality and generalization.
References
1. Hou Zeng-Guang, Zhao Xin-Gang, Cheng Long, Wang
Qi-Ning, Wang Wei-Qun. Recent Advances in
Rehabilitation Robots and Intelligent Assistance
Systems. ACTA AUTOMATICA SINICA, 2016
2. Zhou, Jinman, Shuo Yang, and Qiang Xue. "Lower limb
rehabilitation exoskeleton robot: A review." Advances in
Mechanical Engineering 13.4 ,2021
3. Neuhaus, Peter, and H. Kazerooni. "Design and control ofhuman assisted walking robot." Proceedings 2000 ICRA.
Millennium Conference. IEEE International Conferenceon Robotics and Automation. Symposia Proceedings (Cat.
No. 00CH37065). Vol. 1. IEEE, 2000
4. Nokata M, Ikuta K, Ishii H. 11 Safety Evaluation Methodof Rehabilitation Robots. In: BIEN Z Z, STEFANOV D.
Advances in Rehabilitation Robotics[M]. Lecture Notes inControl and Information Science, 2006, 306: 187-198.
5. Savran A. A Multivariable Predictive Fuzzy PID Control
System [J]. Applied Soft Computing,2013, 13(5): 26582667.
6. Chen J H, Huang T C. Applying Neural Networks to online Updated PID Controllers for Nonlinear Process
Control [J]. Journal of Process Control,2004, 14(2): 211230.
7. Saad M S, Jamaluddin H, Darus I Z M. Implementation of
PID controller tuning using differential evolution andgenetic algorithms [J]. International Journal of Innovative
Computing, Information and Control, 2012, 8(11): 77617779.
8. Pomerleau A, Desbiens A, Hodouin D. Development andevaluation of an auto-tuning and adaptive PID controller[J]. Automatica, 1996, 32(1): 71-82.
9. Mollaret C, Mekonnen A A, Lerasle F, et al. A multimodal perception based assistive robotic system for theelderly [J]. Computer Vision and Image Understanding,
2016, 149: 78-97.

Fig.7 PID control for trajectory tracking based on
DDPG algorithm
Compared with the traditional PID controller, themaximum error of this controller is reduced by 48.12%,
as listed in Table 3. In addition, the controller's overshootis reduced by 40% compared to a traditional PID
controller, which improves user safety during assisted
walking and training.
Table 3. Comparison of results
Controller

Maximum error

OvershootTraditional PID

0.2097

4.5%

DDPG+PID

0.10882.7%Comparative results show that the controller is able to
track the desired trajectory more accurately when the user
uses the wheeled mobility aid robot for rehabilitation
training. It also has a strong anti-interference ability due
to the trial-and-error mechanism of reinforcement
learning, which increases the safety and comfort oftraining.
5. Conclusion
In order to ensure that the elderly and patients withlower limb dysfunction can move accurately according to
the pre-set desired trajectory when they use wheeledwalking robots for rehabilitation training in complex
environments, this paper proposes a reinforcement
learning method that combines the DDPG algorithm witha PID controller. This method not only solves the
problem that the traditional PID cannot adjust theparameters online, but also reduces the storage spacerequirement of the controlled system, thus reducing thecomputation time of the system. Simulation results showthat the PID control based on the DDPG algorithm has
the advantages of high tracking accuracy, small
overshooting amount and strong adaptability, whichenables the wheeled walking robot to realize accurate

Authors Introduction
Mr. Junkai Li
He is currently pursuing Doctor ofPhilosophy (Engineering) in Faculty ofEngineering,Technology
and
Built
Environment, UCSI University, Malaysia.
His research interests are machine learning,
optimization
algorithm
andopticalmeasurement.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1014

Powered by TCPDF (www.tcpdf.org)

Reinforcement Learning DDPG AlgorithmProf. Dr. Mohd Rizon Mohamad Juhari
He is a Professor in Faculty ofEngineering at UCSI University in
Malaysia. He received his PhD in
Engineering from Oita University,
Japen in 2002. His research interests areface analysis, pattern recognition andvision for mobile robot.
Assistant Professor Ir Ts Dr Tiang Sew Sun
She is an Assistant Professor in Facultyof Engineering at UCSI University in
Malaysia. She received her PhD in
Electrical and Electronic Engineeringfrom Universiti Sains Malaysia in 2014.
Her research interests are optimization
and antenna design.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1015

Unsupervised image registration based on Residual-connected DRMINE
for diagnostic metastatic bone tumors
Shogo Baba1, Tohru Kamiya1, Takashi Terasawa2, Takatoshi Aoki2
1

2

Kyushu Institute of Technology, 1-1 Sensui, Tobata, Kitakyushu, Fukuoka 804-8550, Japan
University of Occupational and Environmental Health, 1-1, Iseigaoka, Yahatanishi, Kitakyushu, Fukuoka, 807-8555,
Japan
Email: baba.shogo274@mail.kyutech.jp, kamiya@cntl.kyutech.ac.jp
Abstract
Metastatic tumors are frequently identified through follow-up surveillance using computed tomography (CT) scans.
However, CT scans produce more than 100 images in an examination, which imposes a significant burden on
radiologists and entails a potential risk of misdiagnosis. Temporal subtraction is utilized in Computer-Aided
Diagnosis (CAD) and proves to be an effective technique in aiding image interpretation process for the radiologists.
In this study, we focus on the preliminary stage of CAD development specialized in bone metastasis extraction, witha particular emphasis on rigid registration. We propose a novel rigid registration technique by augmenting DRMINE,
which estimates mutual information using neural networks, with skip connections and normalization. From the threedatasets, ten images were selected randomly from the cervical, thoracic, and lumbar regions. These images were thenaugmented with rotation as well as horizontal and vertical translations to create modified versions. The registration
accuracy was assessed based on the Full Width at Half Maximum (FWHM) of the difference images. In the proposed
method, FWHM values for the thoracic and lumbar regions of the spine exhibited a maximum reduction rate of 2.8%
and a minimum reduction rate of 0.533%. However, the cervical spine region exhibited superior FWHM results withDRMINE compared to the proposed methodology. The proposed method was influenced by the capture area, but it
indicated the potential to provide stable registration as the standard deviation decreased for all FWHM values.
Keywords: Computed Tomography, Rigid registration, Unsupervised learning, MINE, DRMINE

applied in computer-aided diagnosis (CAD) systems forsupporting radiographic interpretation. However, themere subtraction of a simple current image and a past
image results in the occurrence of differential artifacts
due to positional discrepancies between the images.
Therefore, it becomes crucial to perform registration
processing as a preprocessing step. In typical image
registration frameworks, prior to performing non-rigid
registration, the target image pairs are often pre-aligned
based on rigid registration or affine transformations.
However, unless there is a significant improvement in theaccuracy of affine deformations, the performance ofalignment will be greatly diminished [5], [6].
In recent years, numerous studies have been conducted
on registration techniques, particularly in the domain ofdeep learning. However, deep learning requires a
substantial amount of annotated data for training, whichhampers the practical application of deep learning
methods
inmedicalresearch.
Consequently,
unsupervised learning, which does not rely on annotateddata, has garnered significant attention as a potential
solution to this limitation. Differentiable registration withmutual information and matrix exponential (DRMINE)[7] is an unsupervised learning-based rigid registration
model that utilizes Mutual Information Neural Estimation(MINE) [8] to estimate mutual information using a neuralnetwork, instead of relying on conventional
approximations of mutual information.

1. Introduction
As per the estimates of the World Health Organization(WHO) in 2019, cancer is the first or second leading
cause of death in 112 of 183 countries [1]. Furthermore,
there exists the possibility of cancer metastasizing to
distant locations through the vascular and lymphatic
systems, with the spine, in particular, being the mostsusceptible site for skeletal metastasis [2], [3]. Metastaticbone tumors necessitate prompt identification and timely
intervention. However, as metastatic bone tumors
commonly lack discernible symptoms, they are often
detectedthrough
Computed
Tomography(CT)examinations during follow-up observations. Therefore,
the assessment of osseous structures harboringmetastases and the early detection thereof predominantly
rely upon CT scanning as the most efficacious modality.
However, in CT examinations, where 100 to several
hundred cross-sectional images can be obtained in an
examination, the burden on the diagnosing radiologist
increases significantly, and the results vary due todifferences in interpretation experience, which poses a
significant challenge.
The Temporal Subtraction technique (TS) [4] is amethod that involves performing calculations between
two images taken at different times to extract lesions thatappeared during a specific period. This technique is©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1016

Shogo Baba, Tohru Kamiya, Takashi Terasawa, Takatoshi Aoki

𝐵𝑖 is an affine transformation on a two-dimensional
plane. This group comprises six generators, where 𝑀𝑒𝑥𝑝
denotes the matrix exponential operation.

In this study, we focused on rigid registration andproposed a model incorporated with skip connections andnormalization in the MINEnet framework within the
DRMINE model.

0 0 1
0𝐵1 = [0 0 0] 𝐵2 = [00 0 002. Materials and methods
2.1. Dataset

0 0
0 −1 00 1] 𝐵3 = [1 0 0]
0 0
0 0 0

1 0 0
1 0 0
0 1 0
𝐵4 = [0 1 0] 𝐵5 = [0 −1 0] 𝐵6 = [1 0 0]
0 0 0
0 0 0
0 0 0

We conducted experiments employing TOSHIBA
Aquilion PRIME and TOSHIBA Aquilion ONE, utilizing
data from three datasets encompassing imaging ranges
spanning from the cervical to lumbar spine levels. The
image size was set at 512×512, with a slice thickness of
1.0 mm and pixel size ranging from 0.625 to 0.820 mm.
We conducted experiments by randomly selecting ten
images each from the cervical, thoracic, and lumbar
regions based on the data of these three datasets cases.
Furthermore, in order to replicate the displacement inpositioning, an image was created with horizontal andvertical movements of 15 pixels each along the x-axis andy-axis, along with a 10° rotation (Bicubic interpolation).
Fig. 1 illustrates the image transformation, depicting theimage before and after the conversion.
Our computing system consisted of an Intel Core i77500U CPU and an NVIDIA Tesla V100 GPU with
16GB of memory.

Using the multi-resolution pyramid technique [9], [10],[11], the fix image pyramid :𝑇𝑙 and the moving image
pyramid :𝑀𝑙 （𝑙 = 1, .... , 𝐿）. 𝐿 is the maximum level inthe image pyramid.
MINE is a method for estimating mutual information
utilizing neural network architectures. MINE learns using
the error backpropagation method, similar toconventional neural networks, in order to maximize themutual information between the fixed image and the
distorted moving image. θ denotes the parameters of theneural network utilized to accomplish MINE. MINEnetis constructed with two hidden layers composed of 100
neurons each, and utilizes the non-linear activation
function ReLu between the hidden layers.
However, note also that image structures are slightly
shifted through multi-resolution image pyramids. So, atransformation matrix suitable for a coarse resolution
may need a slight correction when used for a finer
resolution. To alleviate this issue, in the second item, we
employ the parameterization of matrix exponentialfunctions and introduce a dedicated additional parameter
vector 𝑣 1 = [𝑣11 , ⋯ , 𝑣61 ] for the finest level of resolution,
optimizing the multiresolution approach. Fig. 2 illustrates
DRMINE registration algorithm.

Fig. 1 The image transformation
Left: Original image, Right: transformed image

2.2. DRMINE registration algorithm
MINE utilizes the principle of Donsker-Varadhan (DV)duality to compute mutual information (MI),𝑠𝑢𝑝
𝑀𝐼 = 𝑓 𝐽(𝑓)(1)where 𝐽(𝑓) is the DV lower bound.

Fig. 2 DRMINE algorithm

𝐽(𝑓) = ∫ 𝑓(𝑥, 𝑧)𝑃𝑋𝑍 (𝑥, 𝑧)𝑑𝑥𝑑𝑧− log(∫ 𝑒𝑥𝑝(𝑓(𝑥, 𝑧))𝑃𝑋 (𝑥)𝑃𝑍 (𝑧)𝑑𝑥𝑑𝑧) (2)2.3. Our method

MINE uses a neural network to compute f(x,z) and uses
Monte Carlo technique to approximate the right hand sideof (2). MINE claims that computation of (1) scales muchbetter than histogram-based computation of MI.
The optimization of DRMINE is as follows.
𝑚𝑎𝑥𝐿6
𝑣1 , ⋯ , 𝑣6
1
𝑣1 , ⋯ 𝑣66 {∑ 𝑀𝐼𝑁𝐸(𝑇𝑙 , 𝑊𝑎𝑟𝑝(𝑀𝑙 , 𝑀𝑒𝑥𝑝 (∑ 𝑣𝑖 𝐵𝑖 )))
𝑙=2
𝑖=1
𝜃

In order to enhance the registration technique utilizing
DRMINE, it is of paramount importance to precisely
estimate mutual information. MINEnet is a method thatutilizes neural networks for the estimate mutual
information, and thus, we postulated that the effective
propagation of differentials plays a pivotal role in theestimate mutual information. The conventional approachcomprises a simplistic two-layer structure, whereas the
proposed methodology constructs a three-layerarchitecture by incorporating additional layers into
MINEnet. Furthermore, we introduced skip connections

6+ 𝑀𝐼𝑁𝐸(𝑇1, 𝑊𝑎𝑟𝑝(𝑀1 , 𝑀𝑒𝑥𝑝 (∑(𝑣𝑖 + 𝑣𝑖1)𝐵𝑖 )))} (3)𝑖=1©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1017

Unsupervised image registration based[12] between each layer. Additionally, we believed thatthe normalization of the hidden layers also influences theprecision of the neural network. Hence, we incorporated
layer normalization [13] into the model, as depicted inFig. 3. Moreover, we set the number of neurons in eachlayer to 100, similar to DRMINE.
To demonstrate the enhancement in accuracy attributed
to the incorporation of normalization, this study
conducted a comparative analysis of precision among
three methodologies: the conventional approach, a model
enriched solely with skip connections in the DRMINE
framework (Method [a]), and the proposed technique(Method [b]).

3. Experiments and results
3.1. Results
From Table 1, the method[b] yielded the lowest average
FWHM for both thoracic and lumbar regions of all dataof datasets. The mean reduction rate was found to be
1.389%, with a maximum reduction rate of 2.8% and aminimum reduction rate of 0.294%. However, in thecervical spine region, DRMINE yielded superiorprecision results across all data of datasets. The model
constructed solely with skip connections withoutnormalization did not show a clear improvement inaccuracy compared to DRMINE. Therefore, theenhancement of accuracy cannot be achieved solely
through the addition of skip connections. By combining
Layer Normalization, we have achieved an enhancementin the accuracy of estimations.
Fig. 5, Fig. 6, Fig. 7 depict the results obtained fromalgorithms applied to the cervical, thoracic, and lumbar
regions. The proposed methodology led to the observablereduction of residuals in the contour region.

2.4. Evaluation methods
The evaluation of image registration accuracy involved
the computation of the Full Width at Half Maximum(FWHM) from the histogram of the difference image.
This difference image was aligned using a reference
image and registered with the moving image (Fig. 4).
FWHM represents the width at which the distribution
encompasses half of its maximum value. A smaller
FWHM indicates a reduction in the presence of artifacts

attributed to residuals.

Fig. 3 Proposed network

Fig. 4 FWHM (Subtract image histogram)

Table1. Average FWHM and Rate of increase or decrease
Rate[a]: The comparison between DRMINE and Method[a], Rate[b]: The comparison between DRMINE and Method[b]▲: Rate of increase , ▼: Rate of decrease
Datasets

Data[A]

Data[B]

Data[C]Level

DRMINE

Method[a]Method[b]Rate[a](%)

Rate[b](%)

C-spineT-spineL-spineC-spineT-spineL-spineC-spineT-spineL-spine1.2686±0.0182
1.2251±0.0249
1.1812±0.0237
1.3628±0.0224
1.3955±0.0197
1.3187±0.0252
1.2645±0.0180
1.3194±0.0147
1.2680±0.0106

1.2805±0.0177
1.2065±0.0196
1.1794±0.0194
1.3724±0.0193
1.3984±0.0111
1.3153±0.0279
1.2688±0.0200
1.3140±0.0121
1.2684±0.0154

1.2712±0.0151
1.1918±0.0140
1.1633±0.0136
1.3667±0.0116
1.3819±0.0127
1.2958±0.0115
1.2693±0.0166
1.3101±0.0044
1.2613±0.0067

0.928▲
1.543▼
0.158▼
0.699▲
0.254▼
0.207▲
0.344▲
0.026▲
0.417▼

0.200▲
2.800▼
1.546▼
0.286▲
0.981▼
1.764▼
0.381▲
0.712▼
0.533▼©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1018

Shogo Baba, Tohru Kamiya, Takashi Terasawa, Takatoshi Aoki

Fix imageMoving image

DRMINE
Fig.5 Cervical spine region

Method[a]Method[b]Fix imageMoving image

DRMINE
Fig. 6 Thoracic spine region

Method[a]Method[b]Fix imageMoving image

DRMINE
Fig. 7 Lumbar spine region

Method[a]Method[b]Thrombosis Research, Vol.133, No.52, pp.149-157, 2014.
3. R. Guillevin et al., “Spine Metastasis Imaging: Review of
the Literature”, Journal of Neuroradiology, Vol. 34, No.5,
pp.311-321, 2007.
4. Kano A. et al., “Digital image subtraction of temporally
sequential chest images for detection of interval change”,
Medical Physics, Vol.21, No.3, pp.453-461, 1994.
5. Maintz et al., “A survey of medical image registration.
Medical image analysis”, Vol.2, No.1, pp.1-36, 1998.
6. Pluim et al., “Mutual-information-based registration of
medical images: a survey”, IEEE Transactions on Medical
Imaging, Vol.22, No.8, pp.986-1004. 2003.
7. Nan A. et al., “Drmime: Differentiable mutual informationand matrix exponential for multi-resolution image
registration”, In: Medical Imaging with Deep Learning.
PMLR, pp.527-543, 2020.
8. Belghazi et al., “Mutual information neural estimation., In:International conference on machine learning”, PMLR,
pp.531-540. 2018.
9. P. Thevenaz, et al., “A pyramid approach to subpixel
registration based on intensity”, IEEE Transactions on
Image Processing, Vol.7, No.1, pp.27-41, 1998.
10. S. Kruger et al., “Image registration using multiresolution
frequency domain correlation”, in BMVC, pp.1-10, 1998.
11. H. S. Alhichri et al., “Multi-resolution image registration
using multi-class Hausdorff fraction”, Pattern Recognition
Letters, Vol.23, No.1-3, pp.279-286, 2002.
12. He K. et al., “Deep residual learning for image recognition”,
In Proceedings of the IEEE Conference on Computer Vision
and Pattern Recognition, pp. 770-778, 2016.
13. Ba Jimmy Lei et al., “Layer normalization”, arXiv:1607.
06450, 2016.

4. Discussion and conclusion
From Table 1, it can be inferred that the contour's shapeexerts an influence on the FWHM. However, it is positedthat the stratification of the proposed methodology was
rather shallow, resulting in an insufficient learning
efficacy. Therefore, we contemplate that further
deepening the layers and establishing interconnections
between them through skip connections may lead to an
improvement in precision. Furthermore, the mere
addition of skip connections did not exert any influenceon the enhancement of accuracy and stabilization of
estimation precision. This demonstrates the contributionof adding layer normalization not only to theimprovement of accuracy but also to the stabilization of
estimate mutual information.
The proposed methodology has demonstrated further
enhancement and stabilization of precision in rigid
registration. Additionally, it has suggested its utility inrigid registration of medical images from anunsupervised learning perspective.
References
1. OECD, Health at a Glance 2021: OECD indicators, pp.143,
OECD publishing, 2022.
2. David Stegner et al., “Mechanistic Explanation for Platelet
Contribution to Cancer Metastasis”,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1019

Powered by TCPDF (www.tcpdf.org)

Unsupervised image registration based

Authors Introduction
Mr. Shogo Baba
He received his Master degree in Medical
Science from Kagoshima University Japan
in 2021. He is a Ph.D. student at Kyushu
Institute of Technology Japan.

Tohru Kamiya, Prof. Ph.D
He received his B.A. degree in ElectricalEngineering from Kyushu Institute ofTechnology in 1994, the Masters andPh.D. degree from Kyushu Institute ofTechnology in 1996 and 2001,
respectively. He is a professor in the
Department of Mechanical and Control Engineering at
Kyushu Institute of Technology. His research interests arefocused on image processing, medical application of image
analysis.
MD. Takashi Terasawa
He received his MD degree from university ofoccupational and environmental health japan
in 2011 and PhD in medicine in 2020.
He is a radiologist at the university.
MD. Takatoshi Aoki
He is a professor of Radiology at Universityof Occupational & Environmental Health inJapan, and the vice president of Japanese
Musculoskeletal Society. His clinical andresearch interests include the imaging modalities ofmusculoskeletal pathology, especially of tumor, arthritis,and bone metabolism.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 20241020

Developing a Smart Belt for Monitoring Elderly Activities Based on Multi-Modal Sensors
Integration and Internet of Things
Abdul Jalil*, Pujianti Wahyuningsih, Najirah Umar, Muhammad Risal, Suwatri Jura, A. Edeth Fuari Anatasya

Universitas Handayani Makassar,
Adhyaksa Baru Street, No. 1, Makassar, South Sulawesi, 90231, Indonesia.
Email: abdul.jalil@handayani.ac.id, ujiwahyuningsih@handayani.ac.id, najirah@handayani.ac.id,
risal@handayani.ac.id, suwatrijura@handayani.ac.id, edetanatasya@handayani.ac.id.
Abstract
This study aims to develop a smart belt that can be used to monitor elderly activities at home by integrating MultiModal Sensors (MMS) and the Internet of Things (IoT). The MMS used in this study to collect elderly information is
the IMU sensor, vibration sensor, push button, and ESP32 to process IoT data. Furthermore, the IoT platform used totransmit elderly information data from the smart belt to the family smartphone is Blynk. In this study, the smart belt
can monitor the activities of the elderly when walking, sitting, lying down, and sleeping. After that, the smart belt can
provide warning alerts when the elderly carry out abnormal activities at home. The results of this study show that thesmart belt can monitor the activities of the elderly when walking, sitting, lying down, sleeping, and warning alerts on
the family smartphone based on the integration of MMS and IoT.
Keywords: Smart Belt, Elderly, Multi-Modal Sensor, Internet of Things.

sensor [5]. Firebase IoT platform was used to transmit
elderly information from the device to the smartphone of
the elderly family. Furthermore, Naeim et al. havedeveloped a device for monitoring elderly activities forsafety based on mobile IoT [6]. In that study, theresearchers used a low-cost prototype device to measureheart pulse, detect falls, and determine the location of theelderly, then used Blynk, Firebase, and Google Assistantfor the IoT platform.

1. Introduction
The elderly is a person who has reached the age of 60
years or older [1]. During this phase, the elderly will havea reduced ability to perform activities, which means that
their family will need to provide them with special
assistance. In addition to monitoring the condition of theelderly at home, the busy activities of family members
outside the home can pose an additional problem whencaring for an elderly relative who is alone at home.
Therefore, the urgency of this research is to build a smart
belt device that can help family members monitor theirelderly activities based on the integration of Multi-Modal
Sensors (MMS) and the Internet of Things (IoT).

According to the results of the study shown previously,
it can be seen that some researchers have developed
devices to monitor elderly activities using IoT. Therefore,
the contribution of this study is to develop a smart beltthat can be used to monitor elderly activities based on the
integration of MMS and IoT. In this study, we used the
IMU sensor, vibration sensor, and push-button for MMS,
and Blynk as an IoT platform to transmit elderly
information from the smart belt to elderly activities whenthe elderly is walking, sitting, lying down, and sleeping.

Some researchers have conducted studies on thedevelopment of devices to monitor elderly activities
based on IoT technology. Hua et al. have developed adevice that can be used to monitor elderly activities using
ICE (IoT Cares for the Elderly) [2]. In that study, theresearchers used a heartbeat sensor, a body temperature
sensor, and an Intel Edison platform to monitor elderly
activities based on the Internet of Things. Narasinghe etal. have conducted a study to monitor and detect activities
of elderly care by integrating various technologies ofwearable and non-wearable devices connected to thewireless network [3]. The researchers in that study used
the heart rate sensor and the PIR sensor to detect elderly
activities and the cloud system to store elderly
information using an internet connection. Cheng et al.
have presented an application to monitor elderly carewith IoT [4]. In that study, the researchers used theThingSpeak IoT platform and the ADXL345
accelerometer to monitor the elderly while doing
activities at home. Rupasinghe et al. conducted the designand development of an IoT-based device to track thephysical activities of the elderly using an accelerometer

This paper consists of a fourth chapter to elaborate on
our study results. In the second chapter, we explain themethod used in our study to develop a smart belt tomonitor elderly activities. The third chapter presents theresults of our study. Then in the last chapter, we show theconclusion and future work of our study.
2. Methodology
Figure 1 shows the architecture design developed in ourstudy to build the smart belt based on the integration of
MMS and IoT to monitor elderly activities.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1021

Abdul Jalil, Pujianti Wahyuningsih, Najirah Umar, Muhammad Risal, Suwatri Jura, A. Edeh Fuari Anatasya

Fig.1. Architecture design
According to the information in Fig. 1, it can be seenthat the sensor devices used to develop MMS in our study
consist of the IMU sensor, vibration sensor, and pushbutton. The function of the IMU sensor in this study is todetect the activities of the elderly when they walk, sit, lie
down, and sleep. We detected it when the degree of
motion of the IMU sensor was observed following theactivities of the elderly. IMU sensor is a sensor device
that can be used to detect the degree of motion of an
object according to acceleration and gyroscope [7]. The
vibration sensor function in the smart belt is to confirm
that the elderly is moving when the IMU sensor detectsthat the elderly walk, sit, and lie down. Then after that,when the IMU sensor detected the elderly in a sleep
position and the vibration sensor did not detect the elderly
moving, the system confirmed that the elderly was in asleep state. The vibration sensor is a sensor device thatcan detect the vibration of an object based on themechanical quantity received by the sensor and thenconvert it into electrical current [8]. Furthermore, thepush button function is a security button that sends
elderly information when the elderly need helps or are in
an abnormal condition.

Fig.2. Flowchart systems

In this study, the microcontroller used to read andprocess MMS and IoT data is ESP32. ESP32 is a
microcontroller that can be used to process sensor data
for the controller, is low-cost and low-power, and is
integrated with the Wi-Fi and Bluetooth module on aboard [9]. The IoT platform used to transmit IoT data
between the smart belt and the smartphone in this study
is Blynk. Blynk is an IoT platform that can be used tocommunicate IoT data between devices to an iOS orAndroid smartphone over an Internet connection [10].
Furthermore, Fig. 2 shows the flowchart systemdeveloped in our study to operate the smart belt based on
IoT.

Based on the information in Fig. 2, it can be seen that,in the first step, the smart belt reads the data input
received from the IMU sensor, vibration sensor, and pushbutton, then processes the IMU and vibration sensor data
to detect elderly activities. When the system detects thatthe elderly is walking, the system then sends theinformation from the smart belt to the family's
smartphone to inform the elderly that they are walking
using Blynk IoT, connected over the Internet connection.
Furthermore, when the elderly is sitting, lying down,
and sleeping, the system sends the information data to thefamily's smartphone based on the information from the
IMU and vibration sensor, respectively. When the system
detects that the push button is pressed, the Blynk IoT thensends the warning information to the family that theelderly need helps or are in an abnormal condition. The
systems then run repeatedly until the power on the smart
belt is turned off.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1022

Developing a Smart Belt

3. Results and Discussion

activities of the elderly when walking, standing up, sitting,and lying down, respectively. Fig. 5 shows the elderly
information on the family smartphone using the Blynk
application based on the Android OS when the elderlywalk.

We have developed a smart belt that can be used tomonitor the activities of the elderly based on the
integration of MMS and IoT. Figure 3 shows the smart
belt developed in our study.

Waist Belt

Belt Head:• IMU Sensor• Vibration• Push Button• Battery

(a)

(b)

(c)

Fig.3 The smart belt developed in this study.
Based on the information in Fig. 3, to implement theexperiment, we used an IMU sensor (KKHMF, MPU6050) to detect elderly activities when walking, sitting,lying down, and sleeping according to the degree ofmovement of the IMU sensor (acceleration andgyroscope), a vibration sensor (DKARDU, Piezoelectric)
to confirm elderly activities when walking, sitting, lying
down, and sleeping according to the vibration of elderly
moving, then push button as trigger for the smart belt tosend a warning to the family when the elderly is anabnormal condition. The microcontroller used in ourstudy to process MMS data and IoT is ESP32 (ESP32
DEVKIT V1) and then used the Blynk platform totransmit IoT data between the smart belt and the family
smartphone. Furthermore, Table 1 shows the
DATASTREAM ID and data type used to transmit theelderly information from the smart belt to the family
smartphone using Blynk.

(d)Fig.4. Activities of the elderly when (a) walking, (b)standing, (c) sitting, and (d) lying down.

Table 1. The DATASTREAM ID and data typeare used to transmit elderly information.
Elderly Information DATASTREAM
Data TypeWalking
V1
INT, 0/1
Sitting
V2
INT, 0/1
Lying Down
V3
INT, 0/1
Sleeping
V4
INT, 0/1
Warning
V5INT, 0/1

Fig.5. Information of elderly when walking.
4. Conclusion
In this study, the development of a smart belt based on
the integration of MMS and IoT that can be used tomonitor elderly activities when walking, sitting, lying
down, sleeping, and under abnormal conditions was
carried out. We used an IMU sensor, vibration sensor,
and push button for MMS and ESP32 as a microcontroller
to process MMS and IoT data, then Blynk for the IoT
platform. In our study result, the smart belt can detect theelderly walking, sitting, lying down, and sleeping on thefamily smartphone.

According to the information in Table 1, we use INT
data type 1 or 0 to confirm the activities of the elderly
when they walk, sit, lie down, sleep, and have abnormal
conditions. When the system detected that the elderly was
doing activities (walking, sitting, lying down, sleeping,
and abnormal conditions), the system then sent the INT
data of 1 to confirm that the elderly was doing theactivities. If not, the system sends 0 data information to
the family smartphone. Furthermore, Fig. 4 shows the©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1023

Powered by TCPDF (www.tcpdf.org)

Abdul Jalil, Pujianti Wahyuningsih, Najirah Umar, Muhammad Risal, Suwatri Jura, A. Edeh Fuari Anatasya

References

Ms. Pujianti Wahyuningsih
She received her Bachelor’s degree inInformation Systems in 2015 fromSTMIK AKBA Makassar, Indonesia.
Then received her Master’s degree in
Computer Systems in 2017, from STMIK
Handayani Makassar, Indonesia. Her
research interests include information
systems engineering and management information
systems.

1.

J. P. Putri and J. Natalia, The Dynamics of Aging Process
Adaption From the Late Adulthood to the Elderly in Panti
Werdha X. Psikostudia: Jurnal Psikologi, Vol. 11, No. 3,
2022, pp. 467-478.
2. B.D.C. Hua, H. Fahmi, L. Yuhao, C.C. Kiong and A.Harun,
Internet of Things (IoT) Monitoring System for Elderly,International Conference on Intelligent and Advance
System (ICIAS), 2018.
3. N. Narasinghe, RPS. Kathiarachchi and M.W.P.
Maduranga, IoT based Health Monitoring and Activity
Detection for Elderly Care, 13th International Research
Conference General Sir John Kotelawala Defense
University, 2020, pp. 396–401.
4. B.J. Cheng, M.M.A. Jamil and R. Ambar, Elderly Care
Monitoring System with IoT Application, ICITAM SCI
863, 2020, pp. 525–537.
5. I.D.M.S. Rupasinghe and M.W.P. Maduranga, Towards
Ambient Assisted Living (AAL): Design of an IoT-based
Elderly Activity Monitoring System, I.J. Engineering andManufacturing, 12(2), 2022, pp. 1–10.
6. M.K.M. Naeim, G.C. Chung, I.E. Lee, J.J. Tiang and S.F.
Tan, A Mobile IoT-based Elderly Monitoring System for
Senior Safety, International Journal of Technology, Vol.
14(6), 2023, pp. 1185–1195.
7. W. Suryanto, Supriyanto, E. Hartantyo, T. Juhana,
Budiarjo, A.S. Handaru and J. Satriani, IMU (Inertial
Measurement Unit) Device for Internet of Things Based
Disaster Early Warning System: Application andInnovation, IOP Conf. Series: Earth and Enviromental
Science, 2020.
8. D.N. Ilham, R.A. Candra, A. Budiansyah, E. Sipahutar,
M.K. Harahap and F. Anugreni, Implementation ofVibration Sensor and Pin Lock using Keypad for Charity
Box Security, International Journal of Multidisciplinary
Sciences and Arts, 1(2), 2022, pp. 125-133.
9. D. Hercog, T. Lerher, M. Truntič and O. Težak, Design
and Implementation of ESP32-Based IoT Devices,
Sensors, 23(15), 2023, pp. 1-20.
10. D D. Babić, I. Jovović, T. Popović, N. Kovač and S. Čakić,
An Internet of Things System for Environmental
Monitoring Based on ESP32 and Blynk, 26th International
Conference on Information Technology (IT), Zabljak,
Montenegro, 2022, pp. 1-5.

her

Ms. Najirah Umar
She received her Bachelor’s degree inInformatics Engineering in 2000 fromSTMIK Handayani Makassar, Indonesia.
She then received her Master’s degree inElectrical Engineering in 2007, from
Hasanuddin University, Indonesia. His
research interests include software
engineering and database management
systems.
Mr. Muhammad Risal
He received his Bachelor's degree in
Computer Systems in 2006 from STMIK
Handayani Makassar, Indonesia. Thenreceived her Master’s degree in ElectricalEngineering in 2011, from Hasanuddin
University, Indonesia. His researchinterests include embedded systems,
microcontroller, networking, and Internet of Things.
Ms. Suwatri Jura
She received her Bachelor's and Master's
degrees in Computer Systems, 2010 and2015, respectively, from STMIK
Handayani Makassar, Indonesia. His
research interests include information
systems and software engineering.

Ms. A. Edeth Fuari Anatasya
She received her Bachelor’s degree inInformation Systems in 2012 fromSTMIK AKBA Makassar, Indonesia. She
then received her Master’s degree in
Computer Systems in 2015, from STMIK
Handayani Makassar, Indonesia. Her
research interests include informatics
engineering and management information systems.

Authors Introduction
Abdul Jalil
He received his Bachelor's and Master's
degrees in Computer Systems, 2012 and2015, respectively, from STMIK
Handayani Makassar, Indonesia. He thenreceived his Doctor of Engineering in
2023 in the Department of Computer
Science and Systems Engineering,
Kyushu Institute of Technology, Japan.
His research interest includes robotics, embedded systems,
network, and Internet of Things.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1024

A penalized motion detection model for extracting ionospheric echoes from low signal-to-noise
ratio Ionogram video images
Yuu Hiroshige

Department of Creative Informatics, Kyushu Institute of Technology, 680-4 Kawazu,
Iizuka, 820-8502, Japan
Akiko Fujimoto

Department of Artificial Intelligence, Kyushu Institute of Technology, 680-4 Kawazu,
Iizuka, 820-8502, Japan
Akihiro Ikeda

National Institute of Technology, Kagoshima College, 1460-1 Shinko, Hayato,
Kirishima, Kagoshima, 899-5193, Japan
Shuji Abe

International Research Center for Space and Planetary Environmental Science, Kyushu University,744, Motooka, Nishi-ku, Fukuoka, 819-0395, Japan
Akimasa Yoshikawa

International Research Center for Space and Planetary Environmental Science, Kyushu University,744, Motooka, Nishi-ku, Fukuoka, 819-0395, Japan
Email: hiroshige.yuu147@mail.kyutech.jp, fujimoto@ai.kyutech.ac.jp, a-ikeda@kagoshima-ct.ac.jp,
abeshu@i-spes.kyushu-u.ac.jp, yoshikawa.akimasa.254@m.kyushu-u.ac.jp
www.kyutech.ac.jp

Abstract
Measuring the altitude distribution of electron density in the upper atmosphere, known as the ionosphere, using HighFrequency radio wave reflections often causes the low signal-to-noise ratio of ionospheric echoes due to radio
frequency interference. We propose a model for converting low-signal-to-noise-ratio ionospheric echo video images(Ionogram) into noise-reduced images using image processing techniques, for tracing the ionospheric echoes from
Ionogram. The proposed method consists of three processing parts: noise removal optimized for individual Ionogram
images, extraction of ionospheric echoes by penalized background subtraction technique, and fine-tuning of
ionospheric echo signals using a minimum spanning tree algorithm. The proposed model successfully reproduces fine
Ionograms with 98% recall and 99% precision.
Keywords: Ionospheric Observation, Computer Vision, Motion Detection, Time-series data

1. Introduction

measure and analyze the fluctuating ionosphericenvironment at high speed and with high accuracy.

The ionosphere is a part of Earth’s upper atmospherefrom about 100km to high altitudes and is very closely
related to our modern life. The ionosphere has asignificant impact on the radio propagation forcommunications, satellite positioning system, etc. The
structure of ionosphere is not stable and variable in thetime of day, season, latitude, and solar activity, becauseof the differences of the energetic photo-ionization
process. The irregularities and disturbances in theionosphere cause various severe problems, for example,
lower accuracy of the radio navigation due to the global
positioning system/Global Navigation Satellite System
scintillation, disruptions in High Frequency (HF) radio
wave bands or interferences due to anomalous
propagation, etc. In order to utilize radio propagation
safely and securely in modern society, it is essential to

The fluctuating ionospheric environment ischaracterized by electron density variations in this region.
The fluctuating ionospheric environment is characterized
by electron density variations in this region. Ionosonde, avertical ionospheric sounder, has been widely used todetect the ionospheric electron density as a function ofheight, using HF radio wave reflections. Ionosonde emits
radio impulses with increasing frequency, measuring thetime delay of radio signals received back from thedifferent ionospheric layers. Fig. 1a shows an ionogram,which is a visualization of the output of an ionosondemeasurement. The ionogram represents the received echo
intensities in a virtual height (time-of-flight) against
transmitted frequency. To analyze the fluctuating
ionospheric environment, it is important of theinterpretation of the ionogram, the so-called “scaling” of©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1025

Yuu Hiroshige, Akiko Fujimoto, Akihiro Ikeda, Shuji Abe, Akimasa Yoshikawa

ionograms. In other words, the scaling is the detection of
the ionospheric echo on the ionograms and the extractionof characteristics of E and F layers (virtual heights: h'E,
h'F, h'F2, etc. or critical frequencies: foE, foF1, foF2,etc.) from ionograms [1].

approach for scaling which includes the elimination
process of background noises.

Classical manual scaling by professional researchers
has been used, but several methods of automatic scaling
have been proposed in recent decades. There are twomain approaches to automatic scaling: image processing(computer vision) approaches [2], [3] and deep-learning
image generation approaches [4], [5]. The both methods
perform the scaling of ionograms with higher accuracy,
but the expected ionogram for scaling is high signal-tonoise ratio (SNR) image. Here, the signal is theionospheric echo, and the noise is the background noise,such as radio frequency interference (RFI) from otheremissions. In Fig. 1a, the noise appears as the vertical
strip. The ionogram visualization process also generates
noise [6]. Ionogram images are visualized using spectral
analysis techniques such as the Fast Fourier Transform(FFT) algorithms on original data obtained by theionosonde. Artificial noises are generated by this process.
The noise due to RFI and FFT complicates automatic
information extraction from ionogram through scaling [7].

a. Original Ionogramb. Full denoised Ionogramc. Histogram of signal
d. Histogram for signal
intensity for Fig .1a
intensity of Fig. 1bFig. 1. Comparison of the signal intensities of original
and full denoised ionograms recorded at 00:06:00 UT,
January 1, 2019.
In this paper, we provide a new method to scaling of
FM-CW ionogram with low SNR. The new scaling
method can be applied to ionograms containing any
ionospheric echo class (E and F layers) for quasi-realtime detection. The shape and position of the ionogram
echoes vary with time, due to daily and seasonal changes
in the ionospheric electron density structure and its solar
activity dependence. Therefore, our study focuses ontime-variant scaling based on time-series analysis. The
proposed method consists of three parts. First, we remove
the noise from the ionogram with low SNR, optimizing
the noise elimination threshold value for individual
ionogram image. Second, we extract the ionosphericechoes based-on penalized background subtraction
technique. Finally, the ionogram with high SNR areregenerated by fine-tuning of ionospheric echo signals
using a minimum spanning tree algorithm.

Ionosonde transmitting power by a typical HF radar ishigh power, several tens of kW. In ionogram images
acquired by such high-power ionosondes, the signal
strength of the ionospheric echo and the backgroundnoise are distinctly different, and the ionospheric echo issufficiently strong. This means that scaling can be
performed from ionogram images with high SNR. On theother hand, our ionosonde observations with Frequency
Modulated Continuous Wave (FM-CW) type radar,
operated by the joint research with Kyushu Institute ofTechnology and Kyushu University, have a maximum
peak power of 20 watts. Although the advantage of FMCW radar is its high gain even at low power, the reflectedecho strength from the ionosphere is relatively weakcompared to a typical HF radar. Therefore, the signal
strength of the ionospheric echoes is often comparable tothe background noise, and it is difficult to separate fromeach other. Since the scaling process is performed from
ionogram images with low SNR, a simple noise
elimination processing is not sufficient.

2. Dataset
The dataset used in this study consists of 960 ionogram
data observed every 90 seconds on January 1, 2019, at the
FM-CW radar ionosonde located at Sasaguri (130.34° E,
33.37° N). Fig. 1a shows a sample of the ionogram image
data obtained on January 1, 2019, at 00:06:00 UT(Universal Time). Ionogram image shows the signals
with frequency values from 2 MHz to 9 MHz and altitude
values from 0 km to 700 km. In order to treat theionograms as a two-dimensional array, we consider theclass width as an array with 278 x 238 bins with afrequency of 0.25 MHz and an altitude of 2.9296875 km.

There are two challenges in automatic ionogram scaling
for FM-CW ionogram. First, as described above, scaling
accuracy is poor when the input image is an ionogramwith low SNR. Second, in terms of the quasi-real-timemonitoring of the ionospheric condition, the processing
time on scaling is required within few seconds. We
sequentially operate the FM-CW ionosonde with every 3
minutes. A low-cost computer, which is usually expected
to perform with no high processing speed, is required on
the observation site. The deep learning approaches havea processing time of 2-3 minutes, and real-time scaling
requires a computer with high processing speed. We try
to propose a new algorithm without deep-learning

As mentioned above, the ionogram is usuallycontaminated with RFI noise appearing as vertical strips,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 20241026

A penalized motion detection

shown in Fig. 1a. As shown in Fig. 1, the signal intensity
distribution in histograms is significantly different
between the original ionogram (Fig. 1c) and the fulldenoised ionogram (Fig. 1d). Fig. 1b was produced bymanual removal of the background noise from Fig. 1a.
The signal distribution in Fig. 1d only corresponds to theionospheric echoes. Comparing between Fig. 1c and Fig.
1d, it is obvious that ionospheric echoes are highly
intense relative to noises. In other words, the almost
background noises are in lower value of signal intensity.

Ridge regression is a method which further improves
generalization performance by adding an L2
regularization term to multiple regression analysis. For
each raw ionogram, Ridge regression estimates thecoefficients, 𝑤 , of multiple regression models for theboundary threshold value as the objective variable, 𝑦, andwhen you give the frequency, mean and variance of themode of the signal intensity as the explanatory variable,
𝑋. The loss function of the ridge regression, 𝐿, is shownin Eq. (2), where the weight of the L2 regularization termis α = 0.7.

3. Methodology
The proposed method consists of three processing parts:(1) Weak Signal Reduction, (2) Penalized Background
Subtraction, and (3) Fine-tuning Ionogram. In theprocessing part on Weak Signal Reduction, the divided
value between the ionospheric echoes and thebackground noises is automatically estimated based-on
Ridge Regression [8]. After this process, almost
background noises are eliminated, and we get thedenoised ionogram remaining enough ionospheric echoes.
The processing part of Penalized Background Subtraction
enables the extraction of ionospheric echoes with motion
detection algorithms. Finally, the processing part of Finetuning Ionogram provides the regenerated ionogram,
removing the remained noises and interpolating theionospheric echoes from the detection region using theminimum spanning tree in the graph structure. The
following subsections describe the detail algorithm ofeach part.

𝐿 = (𝑦 − 𝑋𝑤)𝑇 (𝑦 − 𝑋𝑤) + 𝛼‖𝑤‖22(2)

𝑦̂ = 𝑋 𝑇 𝑤(3)The equation for the objective variable is given by Eq. (3).
The above equation yields a model for estimating thethreshold for removing an appropriate low-intensity
signal. The ionogram after applying this algorithm isshown in Fig. 2.

3.1. Weak Signal Reduction
Fig. 2. Ionogram with Weak Signal Reduction
applied to Fig. 1a

As pre-processing, the weaker RFI noises are
eliminated by aligning the average signal strength in thealtitude direction to a constant value, which is the modeof histogram of signal intensity. We define theelimination function of RFI noises as below:
𝑹𝑓,ℎ = 𝑰𝑓,ℎ + 𝑀𝑂 − 𝜇𝑓3.2. Penalized Background Subtraction
As shown in Fig. 2, it is still impossible to completely
remove the noise from the ionogram, even after removing
the low intensity signal. This is because the noise signal
is more intense than the ionospheric echoes. We use
Background Subtraction method [9] to extract of theionospheric echoes from the ionogram and to ignore theremained intense background noise. The Background
Subtraction method is one of the motion detection
techniques to detect moving objects in a video frame. A
moving object is detected by comparing a frame of video
with its corresponding background image. It is importantto generate a background image without moving objects.
In this study, we consider an ionospheric echo as themoving object and an ionogram image with only noisepresent as the background image. The processing part of
Penalized Background Subtraction is divided into four
steps and the following subsections describe the detailalgorithms.

(1)

where 𝑹𝑓,ℎ and 𝑰𝑓,ℎ represent the power of the original
ionogram as input and the eliminated ionogram as output,
respectively. The subscript 𝑓and ℎ indicate the positionsof the frequency and altitude values on the ionogram. 𝑀𝑂
represents the mode value of signal intensity histogram.
𝜇𝑓 is the average of signal intensity at each frequency bin.
As shown in Fig. 1, it is possible to separate theionospheric echoes and the background noise by
estimating the boundary value between them. However,the distribution of signal intensity is different eachionogram. This suggests that it is necessary to estimate
the threshold value according to the signal distribution ofeach ionogram. We estimate this threshold of boundary
between the ionospheric echoes and the backgroundnoise by ridge regression [8].

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1027

Yuu Hiroshige, Akiko Fujimoto, Akihiro Ikeda, Shuji Abe, Akimasa Yoshikawa

3.2.1. Penalty based on ionospheric echo shape[10]. In the present work, a normal distribution isassumed for each signal with the same frequency value
after removing weak intensity signals, and signals thatdeviate from the 95% confidence interval are penalized
by detecting them as anomalies. By penalizing with theseprocedures, we ideally aim to get the background image
contains only noise.

As shown in Fig. 1b, ionospheric echoes in theionogram are observed mainly as line shapes extending
in the frequency direction. This suggests that the signal
intensity of ionospheric echoes is higher at specificaltitudes than at other altitudes in each frequency band
bin. In fact, as shown in Fig. 3b, the distribution of signal
intensity against the altitude in interest frequency bin, 4.0
MHz, shows two significant peaks corresponding to theionospheric echoes.

1𝑝𝑓𝑓,ℎ (𝑰[𝑡]𝑓,ℎ ) = {𝑰[𝑡]𝑓,ℎ𝑰[𝑡]𝑓,ℎ(𝑝𝑒𝑛𝑎𝑙𝑖𝑧𝑒𝑑)(4)(else)

3.2.2. Penalty based on one previous detection
frame image

a. Original Ionogram data b. Signal intensity as aand the interest
function of the altitude
frequency, 4.0 MHz, infor the interest
red vertical line
frequency, 4.0 MHz
Fig. 3. Signal distribution in altitude direction,
recorded at 06:01:30 UT, January 1, 2019

The background subtraction method can efficiently
detect moving objects under the constraint that thebackground scene is stationary. However, the accuracy
decreases under dynamically changing backgroundimage. Ionograms are no exception to this problem, as the
position and shape of background noise as well as
ionospheric echoes change from moment to moment.
Therefore, the ionospheric echoes detected from theprevious ionogram are used to estimate the backgroundimage.

For generating the background image from the input
ionogram image including the ionospheric echoes, weonly penalize the corresponded frequency into thesignificant ionospheric echoes in each ionogram, toreduce the amplitude of signal power. The detail
operating flow is as shown in Fig. 4. First, a vector of thesignal intensity is extracted for each frequency bin fromthe two-dimensional matrix of the ionogram. Next, thecomponents that penalize the extracted vectors are
selected according to the flowchart shown in the Fig. 4.

This method assumes a two-dimensional Gaussian
distribution centered on the detected component. Here,the frequency value is 𝑓 and the height value is ℎ, fromwhich the mean 𝜇𝑓 , 𝜇ℎ , and standard deviations 𝜎𝑓 and𝜎ℎ are calculated, respectively. The covariance 𝜌 is given
by Eq. (5), considering the characteristic that theionospheric echoes vary in the frequency direction. Eq.
(6) and (7) show the weight Gaussian. The generated
background image is penalized by considering theprevious ionospheric echo detection region.
8
𝜌=(
0
𝑔

𝑓𝑓,ℎ =

0)4

12𝜋𝜎𝑓 𝜎ℎ √1 − 𝜌2

(5)
𝑒𝛼

(6)

2

𝛼=−

(𝑓 − 𝜇𝑓 )2𝜌(𝑓 − 𝜇𝑓 )(ℎ − 𝜇ℎ )1{−2(1 − 𝜌2 )
𝜎𝑓 2
𝜎𝑓 𝜎ℎ
(ℎ − 𝜇ℎ )2
+
}
𝜎ℎ 2

(7)

3.2.3. Penalized background image generation
algorithm

Fig. 4. Determination flow of penalization for thegeneration of background image

In addition to the background image update algorithm
for the Background Subtraction method [9], thebackground image, 𝑩[𝑡] ,is updated by adding the weightterms of Eq. (4) and Eq. (6) as follows.

𝑝
The penalty, 𝑓𝑓,ℎ , is given as Eq. (4), where 𝑰[𝑡]𝑓,ℎ is thesignal intensity of the unprocessed ionogram array at time
𝑡 . We use anomaly detection in deciding whether topenalize. This is designed based on the Hotelling’s theory©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 20241028

A penalized motion detection𝑝𝑩[𝑡] = 𝛼 ∙ 𝑩[𝑡 − 1] + (1 − 𝛼) ∙ 𝑓𝑓,ℎ (𝑰[𝑡])𝑔−𝛽 ∙ 𝑓𝑓,ℎ (𝑭[𝑡 − 1])𝑝composed of odd-numbered widths several times isapplied to the detected ionogram with two-dimensional
array. If there are many signals around the element ofinterest that have been labeled as ionospheric echoes by
the background subtraction method, then they should beionospheric echoes, otherwise they should be considered
as noise.

(8)𝑔where α and β are weights for 𝑓𝑓,ℎ and 𝑓𝑓,ℎ . In this study,
we set 𝛼 = 0.5 and 𝛽 = 8.0 ,which are assumed thesame level of influence of each term.
3.2.4. Detection region determination

The filter 𝐻𝑖,𝑗 consists of an 𝑛 × 𝑛 square matrix, withall elements 1. It is also scanned with stride 1 to maintainthe size of the two-dimensional array of the ionogram.
The convolution equation using this filter is shown in Eq.
(9). Let 𝐷𝑏𝑠 be the array output by the algorithm in theprocessing part of Penalized Background Subtraction and
𝐷 be the array after impulse noise removal.

The positions where the result of subtracting thebackground image obtained by Eq. (8) from the input
image is a positive value is detected as ionosphericechoes, 𝑫[𝑡]𝑓,ℎ , as shown in Eq. (9). It is assumed thatbackground regions are distributed where the difference
values are small. Thus, 𝑡ℎ𝑟𝑒𝑠ℎ in Eq. (9) is set to the classvalue of the mode in the histogram of difference values
between the input and background images.

𝑛𝐷𝑓,ℎ =
1 (𝑰[𝑡]𝑓,ℎ − 𝑩[𝑡]𝑓,ℎ ≥ 𝑡ℎ𝑟𝑒𝑠ℎ)
𝑫[𝑡]𝑓,ℎ = {0 (𝑒𝑙𝑠𝑒)

Original Ionogram𝑏𝑠( ∑ ∑ 𝐷𝑓+𝑖,ℎ+𝑗 ∙ 𝐻𝑖,𝑗 ≥ 2 + 𝑛)𝑖=−𝑛 𝑗=−𝑛

{0

(9)

(10)

(𝑒𝑙𝑠𝑒)In this study, impulse noises are removed by
performing the convolution operation, changing the filter
width n in the order of 5, 7, 9, and 11. Ionograms before
and after the removal of impulse noises are shown in Fig.
6. It is obvious that this process reduces extracted excess
noises and does not disappear the detected signal ofionospheric echoes (Fig. 6b).

Fig. 5 shows that our method can detect ionosphericechoes. This indicates that our method is also effective
for ionograms with high signal-to-noise ratios.

a.

1

𝑛

b.

Ionogram detection
data after processing
for Fig. 2
Fig. 5. Ionospheric echo detection by Penalized
Background Subtraction, original ionogram data
recorded at 00:06:00UT, January 1, 2019

a. Ionospheric echo
b. Ionospheric echo
detection results
detection results after
before processing
processing
Fig. 6. Comparison of ionograms with and without
Impulse noise reduction processing

3.3. Fine-tuning Ionogram

3.3.2. Optimization by a minimum spanning tree

As shown in Fig. 6a, the background subtraction
method can be used to detect ionospheric echoes, but it
can also leave isolated noises. In addition, the signal
should be detected might have missing or incorrectly
detected as the ionospheric echo. For solving these
problems, we propose two post-processing methods: theconvolution operation to remove noise that appears as
isolated echoes, and the optimization for interpolating
and denoising of the detection region using an algorithm
based on a minimum spanning tree.

First, for all points detected as ionospheric echoes,
these detection points are connected for each E and F
layer. The points are connected so that they form aminimum spanning tree, considering the linear shape of
the ionospheric echoes. Next, edges with large distances
between vertices are removed. The graph to which theremoved vertices belong is judged to be noise andremoved. For the remained edges after the above
processes, the detection area is interpolated between thetwo vertices that compose the edge. The result is shownin Fig. 7.

3.3.1. Denoise impulse noise
The convolution is, an algorithm of image processing,used to reduce of impulse noises in the present work. The
multiplication a two-dimensional square matrix©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1029

Yuu Hiroshige, Akiko Fujimoto, Akihiro Ikeda, Shuji Abe, Akimasa Yoshikawa

signals such as those seen in equatorial spread F are notexpected to give meaningful results.
5. Conclusion
Automatic scaling of ionograms presents various
challenges for the ionogram image contaminated withnoises. In this study, we proposed an algorithm toautomatically extract ionospheric echoes from ionogramswith low signal-to-noise ratio.

a.

Ionospheric echo
b. Ionospheric echo
detection results before
detection results after
processing
processing
Fig. 7. Comparison of ionograms with and without
optimization process by minimum spanning tree

Our algorithm is based on statistical thresholding
according to the signal intensity distribution of eachionogram, image processing techniques focusing on time
variability and ionospheric echo characteristics, noise
reduction filtering, and graph theory. Although it isnecessary to define some parameters, it is possible toextract ionospheric echoes from ionograms in quasi-realtime without the huge amount of teacher data as in deeplearning.

4. Result and discussion
To evaluate the accuracy of our automatic scaling
method, we compared the accuracy of ionograms
obtained by our algorithm with that of manually scaled
ionograms. The evaluation is based on the absolute errorof the critical frequencies’ foE and foF of each layer ofthe ionogram and the percentage of absolute error within
0.5 MHz indicating that sufficient accuracy was achieved.
The accuracy of the proposed method is compared withthat of the Weighted Background Subtraction without
each weight term (Table 1). The average processing timeof the proposed algorithm was 7.1 seconds per ionogram.
The proposed model successfully reproduces fineionograms with 98% recall and 99% precision.

On the other hand, for frequent noise and special
ionospheric echo classes, the algorithm may not
guarantee accuracy. The algorithm should be improved to
be robust in these cases as well for the future work.
Acknowledgements
This work was supported in part by JSPS KAKENHI
Grant Numbers JP21K03646, JP21H04518, JP20H01961,
JP22K21345.

Table 1 shows that the detection results weresatisfactory despite the low SNR of the ionograms. From
Table 1 and the detection result shown in Fig. 7b, it isclearly found that the addition of a weight term to thebackground subtraction algorithm was effective. The
processing time with few of seconds indicates that the
proposed method is useful to generate quasi-real-time
ionograms with high SNR.

References
1.
2.

Table 1. Performance Comparison with different weightterms in this study
Model
Absolute
error
(MHz)

foE

Main
result
0.47

Base+
Base+
𝒇𝒑𝒆𝒏𝒂𝒍𝒕𝒚 𝒇𝒈𝒂𝒖𝒔𝒔
0.93
0.92

Base

foF

0.26

0.44

0.42

2.6

Rate ofaccuracy
(%)

foE

67%

50%

50%

47%

foF

83%

49%

61%

15%

3.

4.

0.99

5.

6.

7.

On the other hand, the continuity of ionospheric echoes
remains an issue. Some ionospheric echoes are missing
from the detection results of our method. This is due tothe altitudinally banded intermodulation noise in theionograms. This is a factor that reduces the signal
strength, and thus causes the background subtraction
method to fail to detect the ionospheric echoes. Also,
although not present in this study's data set, scattered

8.

B. Zolesi, L. R. Cander, "Ionospheric Prediction and
Forecasting", Springer, 2014
Z. Chen, S. Wang, S. Zhang, G. Fang, J. Wang,“Automatic scaling of F layer from ionograms”, Radio
Science, Vol. 48, No. 3, pp. 334–343, 2013.
Z. Chen, Z. Gong, F. Zhang, G. Fang, “A new ionogramautomatic scaling method”, Radio Science, Vol. 53, No. 9,
pp. 1149–1164, 2018.
C. De La Jara, C. Olivares, “Ionospheric echo detection indigital ionograms using convolutional neural networks”,
Radio Science, Vol. 56, No. 8, 2021.
Z. Xiao, J. Wang, J. Li, B. Zhao, L. Hu, L. Liu, “Deeplearning for ionogram automatic scaling”, Advances inSpace Research, Vol. 66, No. 4, pp. 942–950, 2020.
F. Arıkan, O. Arıkan, S. Salous, “A new algorithm forhigh-quality ionogram generation and analysis”, Radio
Science, Vol. 37, No. 1, pp. 1-11, 2002.
M. D. E. Turley, A. J. Heitmann, R. S. Gardiner-Garden,“Ionogram RFI Rejection Using an Autoregressive
Interpolation Process”, Radio Science, Vol. 54, No. 1, pp.
135–150, 2019.
D. E. Hilt, D. W. Seegrist, “Ridge: a computer program forcalculating ridge regression estimates”, Research Note
NE-236. Upper Darby, PA: U.S. Department ofAgriculture, Forest Service, Northeastern Forest
Experiment Station, pp. 1-7, 1977.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1030

Powered by TCPDF (www.tcpdf.org)

A penalized motion detection

9.

R. Cutler and L. Davis, "View-based detection andanalysis of periodic motion", Proceedings. FourteenthInternational Conference on Pattern Recognition (Cat.
No.98EX170), Brisbane, QLD, Australia, Vol. 1, pp. 495500, 1998.
10. H. Hotelling, “Multivariate Quality Control Illustrated by
Air Testing of Sample Bombsights”, In: Eisenhart, C.,
Hastay, M.W. and Wallis, W.A., Eds., Techniques of
Statistical Analysis, McGraw Hill, New York, pp. 111-184,
1947.

Dr. Akimasa Yoshikawa
He is a Professor at the Faculty of
Sciences, Kyushu University in Japan. He
graduated from the Department of
Physics, Yamaguchi University, in 1989.
He received his Ph.D. degree in Sciencefrom Kyushu University in 1998. His
research interest is space plasma physics.

Authors Introduction
Mr. Yuu Hiroshige
He received his Bachelor’s degree in
Computer
ScienceandSystems
Engineering in 2022 from the Faculty ofComputer
ScienceandSystems
Engineering, Kyushu Institute oftechnology in Japan. He is currently amaster student in Kyushu Institute ofTechnology, Japan.
Dr. Akiko Fujimoto
She is an Associate Professor at theFaculty of Computer Science andSystems Engineering at Kyushu Instituteof Technology in Japan. She graduatedfrom the Department of Earth andPlanetary Sciences, Kyushu University,
in 2005. She received her Ph.D. degree in
Science from Kyushu University in 2010.
Her research interest is Space Weather and Upper
Atmospheric Sciences.
Dr. Akihiro Ikeda
He is an Associate Professor of the
Department of Liberal Arts and Sciences
at National Institute of Technology,
Kagoshima College in Japan. He
graduated from the Department of Earth
System Science, Fukuoka University, in
2005. He received his Ph.D. degree in
Science from Kyushu University in 2010.
His research interest is Upper Atmospheric Sciences.
Dr. Shuji Abe
He is a Research Fellow at theInternational Research Center for Spaceand Planetary Environmental Science,
Kyushu University in Japan. He
graduated from the Department of Earthand Planetary Sciences, Kyushu
University, in 1999. He received his
Ph.D. degree in Science from Kyushu
University in 2005. His research interest is space weather.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1031

Verification Experiments on the Lower Back Burden caused by Posture and Environment
during Lifting Operations
Tomoka Kimura

Graduate School of Engineering, University of Miyazaki, 1-1, Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
Yutaro Fujino

Faculty of Engineering, University of Miyazaki, 1-1, Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
Sachiko Kido

Interdisciplinary Graduate School of Agriculture and Engineering, University of Miyazaki, 1-1, Gakuen KibanadaiNishi, Miyazaki, 889-2192, Japan
Praveen Nuwantha Gunaratne

Interdisciplinary Graduate School of Agriculture and Engineering, University of Miyazaki, 1-1, Gakuen KibanadaiNishi, Miyazaki, 889-2192, Japan
Hiroki Tamura

Faculty of Engineering, University of Miyazaki, 1-1, Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
E-mail: hi19014@student.miyazaki-u.ac.jp, hi20035@student.miyazaki-u.ac.jp, z321t01@student.miyazaki-u.ac.jp,
ti20060@student.miyazaki-u.ac.jp, htamura@cc.miyazaki-u.ac.jp

Abstract
One of the measures to prevent back pain is the use of appropriate posture corrections. In generally, the Squat method(method of keeping the knees bent and the waist as straight as possible) is recommended over the Stoop method(method of keeping the knees straight and the waist bent). The results of previous studies using acceleration andmuscle potentials have shown that the lifting with knees in a kneeling position reduces the amount of load born by
the lower back. However, few studies discuss the case where there is an obstacle between the subject and the object
to be lifted and further, the scenarios where the object is away from the body's center of gravity. Therefore, thisresearch focus on analyzing three types of movements using the AnyBody musculoskeletal mechanics analysis
software, the Yo-bukun back pain prevention application, and the Delsys surface EMG and verifying the amount oflower back burden when there is an obstacle between the lifting object and the subject. This paper presents theverification results.
Keywords: Lower back burden, Posture correction, EMG, The body's center of gravity

possible). The Japanese Ministry of Health, Labour andWelfare's "Guidelines for Prevention of Back Pain in the
Workplace" recommends the knee-bend lifting method.
The results of the previous study and the analysis using
the "Yo-bukun" application for measuring back strain
showed that the lifting method with the knees bent
reduced the amount of back strain compared to the lifting
method without knees bent [1], [2], [3]. In addition, theprevious study showed that the lifting method with theknees bent was associated with a lower EMG burden

1. Introduction
Frequent lifting activities in the construction andnursing care industries place a heavy burden on the lower
back and can easily lead to lower back pain. One of themeasures that can be taken by individuals to prevent backpain is to improve posture. There are two main types oflifting movements: the Stoop method (keeping the knees
straight and the hips flexed) and the Squat method(keeping the knees flexed and the hips as straight as©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1032

Tomoka Kimura, Yutaro Fujino, Sachiko Kido, Hiroki Tamura

when the lifting object was located close to the body's
center of gravity [4], [5].
However, few previous studies have discussed the casewhere there is an obstacle between the lifting object andthe subject and also when the object is far from the body's
center of gravity. Therefore, the purpose of this paper is
to analyze three types of movements using the AnyBody
musculoskeletal mechanics analysis software, Yo-bukun
back pain prevention application, and the Delsys surface
electromyograph, in order to verify the amount of backburden when there is an obstacle between the lifting
object and the subject.

the presence of an obstacle and the distance between thecenters of gravity. The size of the obstacle was
51(W)×10(D)×51(H)cm. The subject's center of gravity
was set at the pelvis. The experimental apparatus isshown in the Fig. 2.
The initial measurement was a subject lifting the object
as close as possible to its body's center of gravity without
placing any obstacles. The step wise process involves a
subject lifting the object and remaining for 5 seconds,
before lowering it. The distance between the center ofgravity of the object and the subject's center of gravity
was 10 cm. In the second measurement, the obstacle is inplace, and the subject lowers the object to behind the
obstacle starting from an apparatus configuration as inFig. 2, subsequently returns it to the top of the obstacle.
Upon lowering the object behind the obstacle, the subject
stood upright and maintained the position for a durationof 5 seconds.

2. Methodology
In this paper, lumbar burden was measured using
AnyBody, Yo-bukun applications, and Delsys EMG.
2.1. Measuring equipments
AnyBody is a musculoskeletal mechanics analysis
software from Terrabyte Co. Basically, it calculates theforces acting on each part of the human body, such asmuscle activity, muscle force, and joint moments, whena movement is applied to a musculoskeletal model using
inverse dynamics analysis.
Yo-bukun is an application developed by the
University of Miyazaki in collaboration with DENSAN
Corporation for the prevention of back pain for theiPhone. It is calibrated when the user stands upright, andcalculates the amount of lumbar burden based on the tiltof the iPhone.
Delsys is the generic name for Trigno EMG system
incorporated with some additional sensors developed by
Delsys. In addition to EMG, IMU sensors (accelerometer,gyroscope, and magnetometer) are built in to collectkinematic data (angles) at the same time.

Fig.1 . EMG sensor placements

2.2. Measurement method
A video camera footage was used as the method of
measurement for the analysis by AnyBody application.
Under Yo-bukun approach, subjects were asked to keep
their iPhones installed with Yo-bukun application, in
their chest pockets for the measurement. EMG based
approach employs two EMG sensors placed on thesubject's erector spinae. The positioning of the lumbar
EMG sensors is shown in the Fig. 1. The left side sensor
is considered as the first channel and the right side sensor
is the second channel.
The subjects were all healthy adult males, 174±3 cm
tall and 22±1 years old. Subject identifiers are A, B, andC respectively. The movement to be measured was thelifting and lowering of an object while kneeling on one
knee. The dimensions of the object (a box), are 35(W)×23.5(D)×31(H)cm and the weight is 1 kg. Three typesof movements were measured to examine the effects of

Fig.2 . Experimental apparatus (includes the lifting
object and the obstacle)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1033

Experiments on the Lower Back Burden

the potentials recorded when the subject is at rest for a 5second interval between lifting and lowering movements.

3. Results
This chapter outlines the measurement outcomes for thethree equipment types and provides a comparison
between them. The movements are individually analyzed
for lifting and lowering at center-of-gravity distances of
10 cm, 30 cm, and 45 cm, respectively.
3.1. Similarity between AnyBody and Yo-bukun
To validate the reliability of Yo-bukun, the similarity
between AnyBody and Yo-bukun was assessed.
Verification utilized the measurement data of subject B,
analyzable with A. The coefficients of determination foreach center-of-gravity distance are presented in Table 1.
A somewhat stronger correlation was evident in the data
at a center-of-gravity distance of 10 cm, while robust
correlations were observed at center-of-gravity distancesof 30 cm and 45 cm.

Fig.3 Posture in which the center of gravity of both thesubject and the object 10 cm

Table 1. Correlation between AnyBody and Yo-bukun(Subject B)
Center ofgravity distance
(cm)
10
30
45

Fig.4 . Posture in which the center of gravity of both thesubject and the object 30 cm
Then, the subject knelt down once more and raised theobject back over the obstacle. The distance between theobject's center of gravity and the subject's center ofgravity was 30 cm. The third measurement closely
resembled the second, with the distance between thecenter of gravity of the object and that of the subject
being 45 cm. The postures observed during suchmeasurements at the distance between the center ofgravity of the object and that of the subject being 10 cm
and 30 cm are depicted in Fig 3 and Fig. 4.

R2
0.55820.92890.9812

Table 2. The rates of increase in Yo-bukun measurements
Subject
Identity

A

B

After that the transition of the amount of burden appliedon the lower back during such movements were
calculated using AnyBody and Yo-bukun. Later, theanalysis incorporates the maximum and average valuesof the amount of burden observed here. The Delsys
measured the EMG potentials of the lower back during
movements. To reduce the influence of noise, the rate ofincrease was calculated by dividing the difference
between the mean value of the EMG during movement
and the mean value of the EMG at rest by the mean valueof the EMG at rest. The resting EMG potentials refer toCCenter ofgravity
distance
(cm)
10
30
45
10
30
45
10
30
45

Raising
movement
(Up)
%
97.39
110.71120.7090.68
140.64172.14137.98
197.84
196.63Lowering
movement(Down)%
107.02104.21105.34108.78132.38163.56157.31
193.90162.033.2. Analysis results using Yo-bukun
The rate of increase was also determined for the Yobukun in comparison with myoelectric potentials. The
rate of increase in the Yo-bukun measurements for thethree movements is shown in Table 2. The comparisons©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1034

Tomoka Kimura, Yutaro Fujino, Sachiko Kido, Hiroki Tamura

for the raising and lowering movements are shown in Fig.
5 and 6, respectively. The average rate of increase wasobtained by excluding the rate of increase at 10 cm. The
average rate of increase was 156.44% for the lifting
motion and 143.57% for the lowering motion.

Table 2. The rates of increase in lumbar EMG potentials
Subject
IdentityA

B

C

Center ofgravity
distance
(cm)
10
30
45
10
30
45
10
30
45

Raising
movement
(Up)
%
30.43
230.24339.7859.08
352.31
247.7616.20
518.76737.72Lowering
movement(Down)%
5.73
239.78296.6968.08
357.99284.3016.48
534.77654.72

Fig.5. Yo-bukun Increase rate (up)Fig.7. EMG Increase rate (up)Fig.6. Yo-bukun Increase rate (down)

3.3. Analysis results using Delsys device
The rates of increase in lumbar EMG potentials during
the three movements are shown in Table 3, andcomparisons for raising (up) and lowering (down)
movements are shown in Fig. 7 and Fig. 8, respectively.
In all cases, the rate of increase was lowest when thecenter of gravity distance was 10 cm for each subject. For
Subjects A and C, the rate of increase was higher when
the center of gravity was 45 cm than when the center ofgravity was 30 cm, for both lifting and lowering. For
subject B, the rate of increase was higher when the centerof gravity was 30 cm than when the center of gravity was
45 cm. The average rate of increase was obtained by
excluding the rate of increase at 10 cm. The average rateof increase was 404.43% for the lifting motion and394.71% for the lowering motion. The average increase
was 2.59 and 2.75 times that of Yo-bukun for each
operation.

Fig.8. EMG Increase rate (down)©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1035

Experiments on the Lower Back Burden

3.

4.

Fig.9 . Posture (Left: Subject B, Right: Subject C)

5.

4. Conclusion
The rate of increase of EMG potentials during the
movement was different in Subject B than in the othersubjects. The possible reason for this is the difference inposture. The most notable difference is the curvature of
the back. A comparison of the posture of Subject B and
Subject C at a center of gravity distance of 45 cm isshown in the Fig. 9. The yellow circle at the pelvis
position is the center of gravity. The white line is a
simplified representation of the back. It can be seen that
subject B has a straighter back than subject C. This
indicates that subject B has a straighter back than subject
C. This suggests that Subject B had less muscle stretch
than Subject C, and that the rate of increase inmyopotential was less pronounced in Subject B than in
the other subjects.

Authors Introduction
Ms. Tomoka Kimura
She received her Bachelor’s degree inEngineering in 2023 from the Faculty ofEngineering, Miyazaki University inJapan. She is currently a master student
in Miyazaki University, Japan.

Mr. Yutaro Fujino
Yutaro Fujino Born in 2001. He iscurrently studying in the department ofenvironment robotics, and will receive
the B,Eng from University of Miyazaki
in 2024. His current reseasrch isAnalysis of Low Back Burden Utilizing
a Smartphone Application Called Yobukun.

In this paper, we analyzed movements using three
types of measurement devices and verified the amount oflumbar burden in a situation where there is an obstacle
between the subject and the lifting object. The analysis
results shows that the change in the amount of lumbar
burden due to the difference in the center of gravity
distances was more pronounced in the electromyograms
than in the measurements from the posture-based devices,such as the Anybody and the Yo-bukun. Therefore, it ispossible that even in the one-kneeling lifting method, asignificant distance between the object and the operator'scenter of gravity might not result in a reduction ofelectromyographic back burden.

Ms. Sachiko Kido
She received her M.S. from Nakamura
Gakuen University, Graduate School of
Human Development in 2015. Currently
enrolled in the doctoral program at theGraduate School of Agriculture andEngineering, University of Miyazaki.
Assistant Professor, Department of Early
Childhood Education, Higashikyushu Junior College,
2010. Lecturer, Faculty of Education, Miyazaki
International University, since 2019. Her main researchinterest is the development of motor skills in young
children.

References
1.

2.

M. F. Antwi-Afari, H. Li, D. J. Edwards, E. A. Pärn, J. Seo,and A. Y. L. Wong, “Biomechanical analysis of risk
factors for work-related musculoskeletal disorders during
repetitive lifting task in construction workers,”
Automation in Construction, vol. 83, pp. 41–47, Nov. 2017,
doi: https://doi.org/10.1016/j.autcon.2017.07.007.
I. Kingma et al., “Lumbar loading during lifting: acomparative study of three measurement techniques,”
Journal of Electromyography and Kinesiology, vol. 11, no.
5,
pp.
337–345,
Oct.
2001,
doi:
https://doi.org/10.1016/s1050-6411(01)00011-6.
C. Larivière, A. Bertrand. Arsenault, D. Gravel, D.
Gagnon, and P. Loisel, “Surface electromyography
assessment of back muscle intrinsic properties,” Journalof Electromyography and Kinesiology, vol. 13, no. 4, pp.
305–318, Aug. 2003, doi: https://doi.org/10.1016/s10506411(03)00039-7.

H. Tamura, K. Sakurai, K. Tanno, and Y. Fuse, “A Studyon the Lumbar Burden Evaluation of Work using One
Smartphone,” Journal of Robotics, Networking andArtificial Life, vol. 5, no. 3, pp. 173–173, Jan. 2018, doi:
https://doi.org/10.2991/jrnal.2018.5.3.7.
H. Tamura, K. Sakurai, K. Tanno, and Y. Fuse, “A Studyon the Lumbar Burden Evaluation of Work using One
Smartphone,” Proceedings of International Conference on
Artificial Life and Robotics, vol. 23, pp. 550–553, Feb.
2018, doi: https://doi.org/10.5954/icarob.2018.os14-1.

Mr. Praveen Nuwantha Gunaratne
He received his Bachelor’s degree inEngineering in 2018 from the Faculty ofEngineering, University of Moratuwa, Sri
Lanka. He is currently a Doctoral student
in University of Miyazaki, Japan.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1036

Powered by TCPDF (www.tcpdf.org)

Tomoka Kimura, Yutaro Fujino, Sachiko Kido, Hiroki Tamura

Prof. Hiroki Tamura
He received the B.E. and M.E. degreefrom Miyazaki University in 1998 and
2000, respectively. From 2000 to 2001,
he was an Engineer in Asahi Kasei
Corporation, Japan. In 2001, he joined
Toyama University, Toyama, Japan,
where he was a Technical Official in the
Department of Intellectual Information Systems. In 2006,
he joined Miyazaki University, Miyazaki, Japan, where he
was an Assistant Professor in the Department of Electricaland Electronic Engineering. Since 2015, he is currently aProfessor in the Department of Environmental Robotics.
His main research interests are Neural Networks andOptimization Problems. In recent years, he has had interestin Biomedical Signal Processing using Soft Computing.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1037

Development of Smartphone Application for Calculating the Low Back Pain Risk
Seigo Imura

Faculty of Engineering, University of Miyazaki, 1-1, Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
Praveen Nuwantha Gunaratne

Interdisciplinary graduate school of Agriculture and Engineering, University of Miyazaki, 1-1, Gakuen KibanadaiNishi, Miyazaki, 889-2192, Japan
Hiroki Tamura

Faculty of Engineering, University of Miyazaki, 1-1, Gakuen Kibanadai-Nishi, Miyazaki, 889-2192, Japan
Email: hi20006@student.miyazaki-u.ac.jp, ti20060@student.miyazaki-u.ac.jp, htamura@cc.miyazaki-u.ac.jp

Abstract
This study primarily relies on a smartphone application, developed within our research institute known as Yo-bukun,
for the real-time estimation of lumbar burden. The Yo-bukun is capable of estimating the lumbar burden of a subject(the user) by placing an app. installed smartphone in the subject’s chest pocket, while the subject is lifting/ relocating
an object. The subject's movements are assessed through sensors embedded in the smartphone and certain aspects of
their physical information initially fed into the app. are used in the estimation formula for determining lumbar burden.
In the current scenario, Yo-bukun lacks the capability to ascertain whether the user is holding an object or not;consequently, it can only estimate the lumbar burden for limited cases of a subject holding an object. To address such
limitations, the proposed system integrates voice recognition to facilitate lumbar burden estimation, considering thepresence or absence of an object. Further, it was incorporated with the capability to recalculate lumbar burden after
measurements, enabling prospective studies.
Keywords: Lumbar burden estimation, smartphone application, voice recognition.

The second issue arises upon the completion of
measurement, in which the user has not been given thecapability to modify the set information and reevaluate
the lumbar burden. In that case, if a measurement has
already been conducted with incorrect information, it
demands repeating the same measurement. Therefore,
this paper outlines the functionalities introduced toaddress and resolve these identified challenges.

1. Introduction
The smartphone application called Yo-bukun allows
you to estimate the lumbar burden when holding an object
in real time by placing a smartphone in the chest pocket[1], [2]. The estimation method of the Yo-bukun involves
utilizing smartphone sensors such as the acceleration
sensor and gyro sensor to assess the user's state (standing,sitting, turning, crouching, walking). Subsequently, theuser's physical information is incorporated, and theestimation formula used in MIYADAI Pro developed byProfessor Mitarai [3] is applied to perform the estimation.
Here, the user needs to set the user’s physical informationand the weight of the object being held by the user before
taking measurements. During measurement, the lumbar
burden is estimated at every second in real-time.

2. Methodology
This paper describes the proposed method that can be
resolved only using a smartphone, emphasizing userfriendly ease of operation.
2.1. First problem-solving method
To address the problem of measurement being possible
only in the limited situation where the user holds anobject, a voice recognition function has been introduced
to the Yo-bukun. The user articulates speech, while
initially holding an object and when releasing it, and thisvoice information is utilized to determine the object
possession status. The judgment method can be describedas follows: upon detecting the initiation of object holding,the system assumes continuous possession until the

However, few constraints have been identified in its
functionality. The initial challenge is the inability todetermine whether the user is actively holding an object
during measurement or not. Therefore, throughout themeasurement the lumbar burden is approximated under
the assumption that the user is consistently holding an
object. As a result, the Yo-bukun can’t estimate thelumbar burden regarding movements occurring before
the user holds an object or after putting an object down.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1038

Seigo Imura, Praveen Nuwantha Gunaratne, Hiroki Tamura

moment of object release is identified. Conversely, if thesystem detects the timing when the user puts down an
object, it assumes that the user does not hold an object
until the point of object holding initiation is recognized.
In this way, the status of object possession can be
determined using voice recognition.

4.1. First experiment
In the first experiment, Yo-bukun was employed forlumbar burden assessment, using specific verbal cues
during both the Start and End of the object handlingprocess. The weight of the object was set at 2 kg. To
ensure the optimal accuracy in estimation result, thesmartphone was securely affixed to a vest, positioned inclose proximity to chest area. Initially, the configuration
was set for the subject to commence the measurement
without holding an object as the Yo-bukun also offers
configurability to specify the status of holding an object
or not, at the initialization. The Fig. 1 depicts thefollowing operational sequence proceeded during theexperiment:a) Start the measurement without any object.
b) Uttered a specific word and lifted an object after 11
seconds.
c) Raised the upper half of the body.
d) Safely place the object on the floor by uttering aword after 21 seconds.
e) End the measurement by raising the upper half of thebody.

2.2. Second problem-solving method
The limitation was explained wherein, the postmeasurement, user is unable to modify pre-measurement
information and recalculate lumbar burden again.
Utilizing the estimation formula utilized in Yo-bukun, thekey parameters for lumbar burden estimation, extending
beyond the pre-measurement information provided by theuser were identified. These additional parameters arerecorded for each lumbar burden estimation.
Consequently, even if the user modifies the premeasurement information after the completion of
measurement, the proposed system facilitates the use ofrecorded parameter data for any recalculations of lumbar
burden.
3. Implementation Procedure
3.1. First problem-solving Procedure
With the use of Apple's speech recognition framework[4], a voice recognition feature was incorporated with the
Yo-bukun and developed a system that assesses thelumbar burden based on the user’s object possession
status. This advancement enables the estimation oflumbar burden solely through a smartphone, considering
the user's object possession status. The proposed systemestimates lumbar burden based on a user-selected preset
weight if an object is held. In instances where the user is
not holding an object, the system assumes an object
weight of 0 kg. Furthermore, this approach replaces
traditional button operations as the commencement(Start) and cessation (End) of measurement have been
automated through voice recognition.
3.2. Second problem-solving Procedure
A review was conducted on the fundamental parameters
crucial for lumbar burden estimation and the systemdevelopment was carried forward for their systematicrecording during measurements. Moreover, the system
was facilitated with the use of recorded parameters toreassess lumbar burden subsequent to the measurement.
As a result, users possess the capability to modify premeasurement information, enabling recalculation oflumbar burden based on these alterations.
4. Experiment
Two experiments were conducted to validate thefunctionality of the introduced system.

Fig. 1. Operation sequence of the first experiment©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1039

Development of Smartphone Application

measurement. Therefore, it was deduced that lumbar
burden could be estimated through voice recognition,
considering the presence or absence of an object.
5.2. Second experiment result
The reevaluated results, depicting changes on object
weight from 2 kg to 5 kg, are illustrated in Fig. 3. During
different measurements when no object is held, therecorded weight is maintained at 0 kg, yielding consistentmeasurements. Subsequently, the object's weight isadjusted to 5 kg, and recalculation is executed during
object holding. As a result, no variations were observed
during the intervals from 0 to 11 seconds and from 21
Seconds until the conclusion of the measurement when
the person was not holding an object. The fluctuations inresults were only evident from the 12 to 20 seconds timeinterval when the person was holding an object.
Therefore, the results of the second experiment illustrate
the possibility of parameter adjustments and postmeasurement recalculations.

Fig. 2. Graph illustrating lumbar burden estimation
during the first experiment.

6. Conclusion
Through the integration of voice recognition into Yobukun, the system is now capable of estimating lumbar
burden, considering whether the user is holding an object
or not. The start and end of measurements which werepreviously reliant on manual button press within theapplication, can now be automated via voice recognition.
By recording the parameters essential for lumbar burden
estimation during the measurement, users have theflexibility to modify pre-set information even after
completing the measurement, allowing the re-calculationof lumbar burden. The implementation of this system has
also simplified the prospective studies of how changes inobject weight or physical information during the same
movement influence lumbar burden.

Fig. 3. Graph showing the change in lumbar burden
estimation during the second experiment; graph for theobject weight 2kg (blue), 5kg object (orange)
4.2. Second experiment
The second experiment was conducted to reassess theestimation results obtained from the initial trial bymodifying user-input information. In this subsequent trial,
an examination was made to determine the impact on
estimation results by altering the weight of an object to 5kg.
5. Experiment result
5.1. First experiment result

References
1.

The experimental results are illustrated in Fig. 2. The
first experiment result outlines the procedure fordetermining whether the presence or absence of an object
can be considered. During the measurement, the Yobukun is modified to enable the recording of userpredefined information and the other parameters essential
for estimating lumbar burden. Consequently, therecorded parameters included the weight of an object,with the weight being logged every second. From this, it
is evident that the Yo-bukun assesses lumbar burden
based on the criteria that a weight of 0 kg indicates the
person is not holding an object, whereas a weight of 2 kg
implies the person is holding an object. The firstexperiment result revealed that the weight was 0 kg forthe initial 10 seconds, 2 kg from seconds 11 to 20 Sec.,and returned to 0 kg thereafter until the end of the

2.

3.

4.

Hiroki Tamura, Keike Sakurai, Koichi Tanno, Yasufumi
Fuse, “A Study on the Lumbar Burden Evaluation of Workusing One Smartphone,” Journal of Robotics, Networking
and Artificial Life 5(3) (2018) 173-179
Mizuki Maiguma, Hiroki Tamura, Koichi Tanno, “A
Study on the Lumbar Burden Evaluation of Work using
One Smartphone,” The 2018 International Conference on
Artificial A Life and Robotics 1-4
K. Mitarai, "Software Resources at Miyazaki University,"
Miyazaki
University.
[Online].
Available:
https://www.cc.miyazakiu.ac.jp/mitarai/room707/soft%20MIYADAI.html.
Apple Inc., "Speech Framework," Apple Developer.
[Online].
Available:
https://developer.apple.com/documentation/speech?langu
age=objc.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1040

Powered by TCPDF (www.tcpdf.org)

Seigo Imura, Praveen Nuwantha Gunaratne, Hiroki Tamura

Authors Introduction
Mr. Seigo Imura
Seigo Imura born in 2001. He iscurrently studying in department ofenvironmental robotics, and will
receive the B.Eng from University ofMiyazaki in 2024.
His current research is Developmentof Smartphone Application for
Calculating the Low Back Pain Risk.

Mr. Praveen Nuwantha Gunaratne

He received his bachelor’s degree
in engineering in 2018 from theFaculty of Engineering, Universityof Moratuwa, Sri Lanka. He iscurrently a doctoral student in
University of Miyazaki, Japan.

Prof. Hiroki Tamura
He received the B.E. and M.E. degreefrom Miyazaki University in 1998 and
2000, respectively. From 2000 to
2001, he was an Engineer in Asahi
Kasei Corporation, Japan. In 2001, hejoined Toyama University, Toyama,
Japan, where he was a Technical
Official in the Department ofIntellectual Information Systems. In
2006, he joined Miyazaki University, Miyazaki, Japan,
where he was an Assistant Professor in the Department ofElectrical and Electronic Engineering. Since 2015, he iscurrently a Professor in the Department of Environmental
Robotics. His main research interests are Neural Networks
and Optimization Problems. In recent years, he has hadinterest in Biomedical Signal Processing using Soft
Computing.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1041

A Computational Approach for Global Trade Analysis Sensitive to Free Trade Agreement
Circumstances: A Case Study Focusing on the Great Mekong Subregion
Ahmad Altaweel

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan
Email altaweel.ahmad770@mail.kyutech.jp
Bo-Young Lee

Logistics Revolution Korea Co., Ltd.
5th FI. 47, Gangnam-daero 101-gil, Seocho-gu, Seoul 06034, Republic of Korea
Email: bylee@logisroad.com
Masayuki Fujiwara

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan
Email: m-fujiw@brain.kyutech.ac.jp
Jang-Sok Yoon

Logistics Revolution Korea Co., Ltd.
5th FI. 47, Gangnam-daero 101-gil, Seocho-gu, Seoul 06034, Republic of Korea
Email: kjsyoon@logisroad.com
Hiroaki Wagatsuma

Graduate School of Life Science and Systems Engineering, Kyushu Institute of Technology
2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan
Email: waga@brain.kyutech.ac.jp

Abstract
In the Global Trade Analysis Project (GTAP), GDP and economic statistical indices can be analyzed for forecasting
future trends in multiple countries by using the GTAP database and GEMPACK utilities, which realize the numericalcalculation based on the concept of Computable General Equilibrium (CGE) models. Even with such well-prepared
tools with the official database, appropriate forecasting is still difficult due to the sensitivity of the Free Trade
Agreement (FTA) circumstances. FTA scenarios with a uniform tariff reduction can be assumed in general, while anabrupt and unreasonable change may occur in the simulation depending on the network structure of trading countries
and the upper and lower bounds of tariffs in the time course. In the present study, we focused on the Great Mekong
subregion (GMS) and explored possible methods to calculate substantially.
Keywords: computable general equilibrium (CGE), General Equilibrium Modelling PACKage (GEMPACK),
Global Trade Analysis Project (GTAP), Free trade agreements (FTA).

1. Introduction

To lower this technical barrier involved, this research
investigates the roadmap to handle the implementation of
the GTAP model. It discusses preparing data andmeasures the effect of altering the growth rates andreduction in the tariff value. Moreover, it forecasts thechanges in the real GDP among the countries in the Great
Mekong subregion (China, Lao-PDR, Myanmar,
Vietnam, Thailand, and Cambodia).

Computable General Equilibrium (CGE) analysis has
emerged as a competitive tool for policy evaluation. Its
widespread application includes research on trade policy,
regional infrastructure development, and environmental
protection. However, challenges in the technicalimplementations explained in [1], [2], [3], datapreparation, and interpreting the simulation results posesignificant burdens in the current state of CGE modeling.

2. Methodology©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1042

Ahmad Altaweel, Bo-Young Lee, Masayuki Fujiwara, Jang-Sok Yoon, Hiroaki Wagatsuma

2.1. Simulation environment

method (e.g. Gragg, Johansen, Euler, 1,3,5 steps) in the
CMF file. Subsequently, the model is executed, and thecoefficients of the model are filled with the actual data.
Thesimulation
generates
a
set
of
files
(.tab, .cmf, .exe, .sl4, .slc, .har) associated with the model.
To retrieve the solution, users can utilize ViewSol,
AnalyzeGE, or ViewHAR to access data in '.slc' and '.sl4'
files.

2.1.1. General Equilibrium Modelling PACKage(GEMPACK)One of the fundamental characteristics of the CGE
models is their computational capability, specifically,
they can generate numerical solutions to shocks(changes) using the provided database and parameters.
GEMPACK short for General Equilibrium Modelling
Package is a set of software created for solving applied
general equilibrium models. This package comprises
various tools for managing economic models, databases,
and simulations. In GEMPACK, data is typicallypresented in the format of HAR files, which are headerarray files capable of storing multiple numeric and stringarrays with dimension labels, accommodating up to sevendimensions. The outcome of the simulation is also
multiple HAR files or solution files (e.g., .slc, .sl4) can besaved in header files.
2.1.2.

2.2. Data and Model Preparation
2.2.1.

The GTAP-RD model is calibrated to the most recent
database produced by the Global Trade Analysis Project
GTAP database version 11 [2]. The database GTAP 11
reconciles different data sources at a global scale andoffers a time series of 5 reference years (2004, 2007, 2011,2014, and 2017), distinguishing 65 sectors in each of 141
countries and 19 aggregate regions [4]. The GTAP
database is typically supplied in the default format of the
HAR file (flows file, parameters, sets,..etc). This
database has been precisely adjusted to achieve
equilibrium in the world’s initial state [5].

Modifying header files

It is often required to adjust the input parameters, shock
files, and process the model's outputs, and that involves
modifying the data within header files. The conventional
approach for handling HAR files has been to employ
executable programs provided by GEMPACK (e.g.,seehar.exe or modhar.exe). However, utilizing these
programs may pose a challenge. Manual transfer of data
between HAR files and data editing software (e.g., Excel)
using a basic clipboard method is also a common practice.
However, this process is highly manual, time-consuming,and prone to errors. Another approach for modifying
HAR files is to create a data manipulation TABLO Input
file using ViewHAR. That can be done by exporting theheader file as a TABLO Code from the Export menu
within the ViewHAR program. This process
automatically generates the foundational code, which canbe further edited using the Tablo language to include
specific filenames, formulas, updates, assertions, andother adjustments tailored to the model's requirements. In
addition, this approach requires creating a Command file(CMF) to define the input, output, and auxiliary file. Thiscan also be generated automatically through the menu
(view/Create CMF) within the Tablo file (.tab file).
2.1.3.

GTAP Database

2.2.2.

The GTAP-RD modelThis study uses GTAP-RD, a dynamic multi-region
model that is described in detail in [6]. GTAP-RD model
was built on the latest GTAP 7 framework [7] and offers
features including, capital accumulation, andinternational capital and investment mobility. In addition,
GTAP-RD offers flexibility in terms of closures as well
as a more convenient way to perform a modification to
the model due to consistent variables, coefficients, andequation-naming conventions.
2.3. Forecasting Process
Quantitative policy analysis is typically framed within
the confines of a specific scenario (numerical projection).
This scenario assumes that the economy will be affectedby future adjustments in the macroeconomic forecasts ofeach country and the expected policy changes. The
baseline scenario provides a reference point forevaluating the impact of policy changes [3] and shouldreflect as closely as possible the changes expected tooccur in the world economy. For example, FTAs that are
already in effect [8]. In most baselines, external
information about certain future trends of themacroeconomic drivers, for example, population, labor
force growth, and other drivers are taken from external
projections, such as the United Nations Population
Division, World Bank, IMF, OECD, CEPII…, etc.). The
forecasts gathered from the mentioned sources may lack

Executing a model in GEMPACKGEMPACK compiles a model by interpreting a model
written in the TABLO language into a system of
equations and generating an executable file (e.g.,‘gtapv7.exe‘). Users must specify the endogenous andexogenous variables, along with the chosen solution©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1043

A Computational Approach for

information and may not align. This necessitates someprocessing to guarantee the availability of data for all
countries and relevant years. Shaping a baseline isimportant for long-term economic issues, such as climatechange, the Intergovernmental Panel on Climate Change(IPCC) has created standard scenarios for future
greenhouse gas (GHG) emissions so-called Shared
Socioeconomic Pathways (SSPs). The SSPs are five
scenarios that form a consistent set of socioeconomic
drivers widely used in the climate modeling community,
and also in the broader global modeling community [3].
2.3.1.

5vnm

Vietnam18

sas

Oceania6

mmr

Myanmar,East Timor

19

eas

East Asia

7

phlPhilippines

20

nwa

North America

8

idn

Indonesia

21

lamLatin America

9

mys

Malaysia22

wer

Western Europ

10

brn

Brunei

23

men

11

sgp

Singapore

24

ssa

12

jpn

Japan

25

row

Middle East
Sub-Saharan andNorth Africa
Rest of World

kor

South
Korea

13

Simulation design and policy scenarios
The analysis begins with the 2017 GTAP databaseserving as the base year (Fig. 1). It shifts forward by three
years to establish 2020 as the benchmark year.
Subsequently, for each period, there is a five-yearincrement until reaching the year 2050 in all scenarios.

The baseline projection runs from 2017 to 2050 andincludes past data and projections for growth in real GDP,
population, and the skilled and unskilled labor supply for
all regions. The baseline projection data comes from the
Shared Social-Economic Pathways (SSP) database macro
forecasts for 230 countries for the period of 2007-2100.
This paper uses the SSP2 projections for the baseline(business-as-usual scenario). In the baseline scenario, we
did not incorporate policy changes caused by existing andongoing FTAs. Absence of the policy changes in thebaseline may affect the simulation results of the policy
scenarios [9]. The baseline projection is run under thedefault GTAP-RD closure for all regions and to make the
GTAP model follow a chosen growth path, real GDP(qgdp) is swapped with the region-wide technologicalchange (afereg) to be calculated endogenously. In
subsequent model runs, technological change are thenmaintained at those estimated levels, while GDP is
endogenously computed in the base rerun.
The study focuses on global logistics in Southeast Asiaand the Great Mekong subregion, and therefore, countries
in this area are the targets of the analysis and are treatedas individual countries. Countries in other regions areconsolidated into regional units, with each treated as ahypothetical country in the GTAP Model.
We aggregated the GTAP database to 25
countries/regions and 10 sectors (Table 1) The region of
the rest of Southeast Asia contains Myanmar and East
Timor after making all the other countries in this region
as individual countries.

Fig. 1. Forecasting Process and sequence of years
We utilized the GTAP-RD aggregation utility database[10] by making the necessary adjustments to align it with
GTAP database version 11. This involved modifying theregion mapping from the original 230 countries to fit the
160 countries of the GTAP database and aggregate years
to match the relevant years in the study.
Scenario 1 (High Case): The SSP5 scenario percent
change for each of real GDP (qgdp), Population (pop),
skilled and unskilled labor qe(ENDWL,REG) is applied
as target shocks to the years from 2025~2050. In addition,
import tariff within the six regions in the GMS area was
reduced gradually in the S1 Scenario but maintained
between the rest of the world and GMS regions. The
reduction periods are as follows (Table 2):

Table 1. Regional aggregation of the GTAP Data Base.
NOID

COUNTRY/R
EGION

NO.

ID

COUNTRY/REGION

1

lao

Lao-PDR

14

ind

INDIA

2

chn

China

15

aus

India

3

tha

Thailand16nzl

Australia

4khm

Cambodia17

xoc

New Zealand

Table 2. Tariff Reduction Schedule between the GMS
countries.
PeriodPercent change(2020~2025)→-20%(2025~2030)→-20%©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1044

Ahmad Altaweel, Bo-Young Lee, Masayuki Fujiwara, Jang-Sok Yoon, Hiroaki Wagatsuma(2030~2035)→(2035~2040)→

factor of production growth rate in the remarkablecountries.

-20%
-40%

It is improbable that an agreement would lead to thecomplete removal of all import barriers, and therefore,
this experiment provides an upper bound for the benefitsof a free trade agreement that can be captured by themodel.
To control the change in the tariff, we used the “rate%”function in the CMF file of the shocks as a policy fileafter associating the new levels variable TMS_L (powerof import tariff) Eq. (1) with the previously declared
linear variable tms in the GTAP model [2].
Fig.2. GDP value change in GMS countries
𝑇𝑀𝑆_𝐿(𝑐, 𝑠, 𝑑) = 𝑉𝑀𝑆𝐵(𝑐, 𝑠, 𝑑)/𝑉𝐶𝐼𝐹(𝑐, 𝑠, 𝑑)(1)

VMSB: the value of imports of commodity c from s to d
at domestic (basic) prices.
VCIF: the value of imports of c from s to d at CIF prices(tradeable only).

4. Conclusion
In this study, the authors employed the GTAP model toestimate the future GDP value in the Great Mekong
subregion area. The study delves into the technicalintricacy of implementing modifications in the growthrate and reduction of the tariff rate. It simulates a free
trade agreement among the involved countries.

After adding the level variable TMS_L, it is possible tomodel the removal of all import taxes by utilizing the“final_level” function Eq. (2) from GEMPACK as
follows:
𝐹𝑖𝑛𝑎𝑙_𝑙𝑒𝑣𝑒𝑙 𝑇𝑀𝑆_𝐿(𝐶𝑂𝑀𝑀, 𝐺𝑅𝐸𝐺, 𝐺𝑅𝐸𝐺)= 𝑢𝑛𝑖𝑓𝑜𝑟𝑚 1 ;Subsequently, the authors intend to expand the range ofFree trade agreements, encompassing the agreements
currently in force in the baseline scenario. Additionally,
they plan to refine the calibration to the growth rate in the
GMS area.

(2)This function will remove the import taxes; thereby advalorem (tariff rate) is equal to zero and the power levelof the tariff is equal to one from the year 2040 in the S1
scenario. The uniform reduction in the tariff rate isapplied to all commodities within the six countries andregions in the Great Mekong subregion by making a new
region set GREG in the model to contain the GMS
countries.

Acknowledgment
The authors would like to thank Professor Ryuichi
Shibasaki for their invaluable suggestions in forecasting
international trade amounts with respect to the GTAP
framework and Dr Trang Tran for technical support andassistance related to GTAP experimental methods. This
work was supported in part by the cooperative research
project on digital logistics between the Kyushu Instituteof Technology and Logistics Revolution Korea Co., Ltd.

Scenario 2 (Low Case): The SSP4 scenario for thepercent changes in the real GDP, population, and labor
force are implemented for the years 2025~2050. This
scenario takes a pessimistic stance on the advancement ofa trade agreement, envisioning no reduction, and predicts
a modest growth rate.

References
1. K. Pearson, M. Horridge, and E. Corong, “Hands-on
Computing with RunGTAP and WinGEM To Introduce
GTAP and GEMPACK.
2. M. Horridge, “GEMPACK manual.” Dec. 17, 2019.
3. R. Dellink, D. Van Der Mensbrugghe, and B. Saveyn,“Shaping baseline scenarios of economic activity with CGE
models: introduction to the special issue,” J. Glob. Econ.
Anal., vol. 5, no. 1, pp. 1–27, Jun. 2020, doi:
10.21642/JGEA.050101AF.
4. A. Aguiar, M. Chepeliev, E. Corong, and D. Van Der
Mensbrugghe, “The Global Trade Analysis Project (GTAP)
Data Base: Version 11,” J. Glob. Econ. Anal., vol. 7, no. 2,
pp. 1–37, Dec. 2022, doi: 10.21642/JGEA.070201AF.

3. Results and Discussion
3.1. Estimated Gross domestic product growth
The estimated values of real GDP are displayed in Fig 2.
As described in Section 2.3.1, It is exogenously given in
the baseline scenario. Looking at the high case theincrements in the value of GDP come from the benefit ofchanges in the tariff reduction and the increase in the total©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1045

Powered by TCPDF (www.tcpdf.org)

A Computational Approach for5. M. Ivanic, “GEMPACK simulations in R: Ademonstrationof running the GTAP modeland processing its results
entirely in R using packages HARr and tabloToR,” J. Glob.
Econ. Anal., vol. 8, no. 1, pp. 1–20, Jun. 2023, doi:
10.21642/JGEA.080101AF.
6. A. Aguiar, Corong, Erwin, and van der Mensbrugghe,
Dominique, “The GTAP Recursive Dynamic (GTAP-RD)
Model: Version 1.0,” Present. GTAP-RD Model GTAP
Conf. Organ. Sess. OnImproving Baseline Proj. GTAPRecursive Dyn. RD Model Framew., Feb. 2019.
7. E. Corong, H. Thomas, M. Robert, M. Tsigas, and D. Van
Der Mensbrugghe, “The Standard GTAP Model, version 7,”
J. Glob. Econ. Anal., vol. 2, no. 1, pp. 1–119, Jun. 2017, doi:
10.21642/JGEA.020101AF.
8. K. Itakura, “Impact of the Regional Comprehensive
Economic Partnership (RCEP): A Global Computable
General Equilibrium (CGE) Simulation,” Econ. Res. Inst.
ASEAN East Asia, Oct. 2022.
9. K. Itakura, “Impact of liberalization and improved
connectivity and facilitation in ASEAN,” J. Asian Econ.,
vol.
35,
pp.
2–11,
Dec.
2014,
doi:
10.1016/j.asieco.2014.09.002.
10. GTAP-RD Data Aggregation Utility.

Dr. JangSok Yoon
He received his M.S. and Ph.D. degrees
from Kyunghee University, in South
Korea, in 2004 and 2008. He is thefounder, CEO, and Consultant over 23
years of Logistics Revolution Korea
Co., Ltd He has served as a logistics andsupply chain consultant to more than
300 companies and governments.
Dr. Hiroaki Wagatsuma
He received his M.S., and Ph.D.
degrees from Tokyo Denki University,
Japan, in 1997 and 2005, respectively.
In 2009, he joined the Kyushu Instituteof Technology, where he is currently
an Associate Professor of the
Department of Human Intelligence
Systems. His research interests include
non-linear dynamics and robotics. He
is a member of IEEE.

Authors Introduction
Mr. Ahmad Altaweel
He received his Bachelor's degree inElectronics
andCommunication
Engineering, from the Faculty ofElectricalandMechanicalEngineering, ALBAATH University,
Syria in 2013. He is currently a Ph.D.
student at Kyushu Institute ofTechnology, Japan.
Ms. BoYoung Lee
She received her Bachelor's degree in
Human Relations from Keio University
in Japan. She then received her M.S. inlogistics from Inha University in Korea
and is currently a Ph.D. student. She
works as a logistics consultant in the
Logistics Revolution Korea Co., Ltd.
Dr. Masayuki Fujiwara
He graduated from the Fukuoka
Institute of Technology(FIT) in 2015.
He then went on to earn his M.Sc. and
Ph.D. (Knowledge Science) from Japan
Advanced Institute of Science andTechnology(JAIST) in 2017 and 2023.
Until 2020, he was a Research Fellow of
the Japan Society for the Promotion of
Science (JSPS) (DC1) and now is a researcher at Kyushu
Institute of Technology(Kyutech). His research focuses on
the EEG studies and neural modeling of (non-)symbolic
communication from the perspective of cognitive
neurodynamics.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1046

Trigger circuit design and system integration
for simultaneous measurement of human EEG, motion, and gaze
Masayuki Fujiwara, Phan Hoang Huu Duc

Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan
Email: m-fujiw@brain.kyutech.ac.jp, phan.duc-hoang516@mail.kyutech.jp
Laurent Bougrain, Patrick Hénaff

Université de Lorraine, CNRS, LORIA, F-54000 Nancy, France
Email: {laurent.bougrain, patrick.henaff}@loria.fr
Hiroaki Wagatsuma

Kyushu Institute of Technology, 2-4 Hibikino, Wakamatsu-ku, Kitakyushu, 808-0196, Japan
Email: waga@brain.kyutech.ac.jp

Abstract
Simultaneous measurement of EEG, motion, and gaze in humans has the potential to lead to the discovery of newscientific insights. In order to achieve these simultaneous measurements, it is necessary to manage triggers and time
information between measurement devices, as well as to correct time offsets. However, the management of accurate
triggers and time information requires the design of a dedicated circuit board and the integration of TransistorTransistor Logic signal voltage information. In this study, we report on the building of a trigger circuit and an
experimental system using it to solve these problems. We created a home-made trigger circuit board for voltage
integration and combined it with a commercially available microcomputer to realize an integrated trigger circuit andmeasurement system.
Keywords: Trigger circuit design, Electroencephalography (EEG), Motion, GazeThe simultaneous measurement of various types of
information, such as movement and gaze, in conjunction
with human electroencephalography (EEG), holds
particular significance due to their intimate involvementwith neural activities. As introduced by Iturrate et al. [7],in the case of humans, the information derived from thegaze is directed to the V1 area within a span of 60-90 ms,
reaches the motor cortex in approximately 220 ms, andthe expression of fingertip movements is processed witha latency roughly between 280-400 ms (it is noteworthy
that the process is expedited in monkeys [8]). Measuring
and analyzing brain activities during this process isbeneficial to understanding human cognitive functions.

1. Introduction
Simultaneous recording of physiological andbehavioral data in standard data formats and analysis ofthese data in combination may lead to new scientificdiscoveries. So, in recent years, multimodal measurement
methods and data formats have been attracting attention.
In response to this situation, data acquisition through the
Lab Streaming Layer (LSL) [1], new standard data
formats, such as Brain Imaging Data Structure (BIDS) [2],[3], Neuroscience Information eXchange [4] and OpenmetaData Markup Language [5] (NIX-odML) [6], havebeen proposed to integrate and handle multimodal data
and meta information.

In the previous work, we have proposed amethodological design [9] to reveal human-robot
interactions in the real world by managing TransistorTransistor Logic (TTL) signals. This proposal provided a
specific event analysis based on data from multiple
instruments and contributed to a reproducible
experimental scheme in which each element could beinterchangeable in the future.

For these simultaneous recordings, an experimenter isfaced with the necessity to manage the trigger and time
information between measuring instruments and tocompensate for time offsets. In fact, while there are paid
software and hardware types of equipment that manage
the trigger and time information, it is so challenging tohandle between measuring instruments that do notsupport such information.

However, to achieve accurate trigger and time
information to be managed, a dedicated and versatilecircuit board and its system must be designed to integrate©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1047

Masayuki Fujiwara, Phan Hoang Huu Duc, Laurent Bougrain, Patrick Hénaff, Hiroaki Wagatsuma

Fig. 1 Recording System Integration for human EEG, motion, and gaze data synchronization.
the voltage information of the TTL signal.

depends on detecting the pressing of the Axis Neuron'srecording button using software for the perception neuron
with a global hook function on Windows.

Therefore, this study reports on the fabrication of a
trigger circuit and an experimental system using it tosolve the problems mentioned above. We built a trigger
circuit board for voltage integration and combined it witha commercially available microcontroller to integrate thetrigger circuit and the measurement system.

2.2. Home-made Trigger Circuit andMicrocomputer
Fig. 2 shows the fabrication and design of the trigger
circuit board. We used Autodesk Fusion 360 [10] to unify
circuit design (Fig. 2a) and board design (Fig. 2b). The
designed circuit was fabricated to be integrated with themicrocomputer (Fig. 3). Moreover, this circuit has two
input ports (blue and yellow cables), but multiple input
ports can be managed by increasing the number of levelshift circuits. For measurement instruments that do nothave a trigger output, a timer function in the
microcomputer is used to output a square wave (e.g., 5.00
[V]) with accuracy in the order of microseconds. The
trigger and synchronization are managed byadding/subtracting the above-mentioned multiple trigger
inputs to/from this voltage. If other measurement
equipment has a trigger output, it can be integrated after
checking the output voltage.

2. Proposed Trigger circuit design and system
integration s
2.1. Recording System Integration
The integrated measurement system of our recording
instruments is shown in Fig. 1. The system uses a homemade trigger circuit and a microcomputer for integrated
management of the voltage and time information output
from each measurement instrument with and without
trigger output function. The following is a description of
the actual measurement instruments that were used:⚫
⚫
⚫

g.tec USBamp and g.tec GAMMAbox for EEG
recording;
Perception Neuron 2.0 for Motion Capturing;
Tobii Pro Glasses 2 for Eye Tracking.

2.3. Testing Equipment information
We observed the trigger signals using a digital
oscilloscope to test the integrated trigger circuit board
and its recording system. Actually, we connected
multiple measurement instruments to the input ports ofthe trigger circuit to capture the changes in the trigger

Perception Neuron 2.0 does not have a trigger output
function. Nevertheless, the trigger and time information
can be obtained by combining a trigger circuit and a
microcomputer, as described in the next section. Notethat the timing of the first output of the trigger signal©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1048

Trigger circuit design anda.

Schematic circuit diagram
b. Circuit board diagram
Fig. 2 Fabrication and Design of Trigger Circuit Board
signal. The settings used for observation were an
amplitude of 2.00 [V]/scale and a period of 100 ms/scale.
3. Results and Discussion
We measured the trigger signal from the integrated
trigger circuit board (Fig. 4). Fig. 4a shows the case inwhich only the trigger circuit was used, and a squarewave (5.00 [V]) was measured. Fig. 4b shows asimultaneous measurement, including trigger signals
from other measurement devices and the timing ofmultiple trigger inputs. These recorded trigger signals
showed the management we expected.
As a limitation, if the measurement start time of themeasurement device depends on the recording software,
the accuracy of the trigger may be affected by the system
clock of the Operating System (OS), leaving room forfurther study. Currently, in this case, a dedicatedapplication is developed and used to output the trigger
from the Universal Serial Bus port to detect themeasurement start time, which may cause an accuracy
problem.

Fig.3 Integrated Trigger Circuit Board

4. Conclusiona.

b.

We designed a custom trigger circuit board to measure
EEG, motion, and eye movement simultaneously. This
trigger circuit board and its system integrate the voltage
information of multiple TTL signals, allowing for
accurate triggering and time synchronization between
different measurement instruments. It also may suggest apotential way to provide data from measurement
instruments that do not have trigger output functions withrespect to existing measurement methods for integrating
multimodal data.

Only Integrated Trigger Circuit Board

Future work is to investigate the accuracy of triggeringat system time, etc., when relying on measurement
software., i.e., the management of time information
between the OS and the trigger circuit.

Combined with Other Measuring Instruments
Fig.4 Recorded Trigger Signals©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1049

Powered by TCPDF (www.tcpdf.org)

Masayuki Fujiwara, Phan Hoang Huu Duc, Laurent Bougrain, Patrick Hénaff, Hiroaki Wagatsuma

Acknowledgements
Mr. Phan Hoang Huu Duc
He received Bachelor degree inBiomedical
Engineering,from
Department of Biomedical Engineering,
Faculty of Applied Science, Ho Chi Minh
City University of Technology (HCMUT)
in 2021. He is currently studying
Neuroscience in Kyushu Institute ofTechnology from 2022.

We are grateful to Guangyi Ai for collaboration on thisproject. This work was supported in part by JSPS
KAKENHI Grant Numbers JP16H01616, JP17H06383,
JP23K20008 and Université de Lorraine and Kyutech
Collaborative Project Fund 2023-24 "Human/Robot
Social Interactions: engagement and affect analysis
during gaming tasks".

Dr. Laurent Bougrain
He earned his MSc in artificial
intelligence from université Paris 6, his
BS in psychology from université Nancy
2 and his Ph.D in Computer Sciencefrom Université de Nancy 1 in 2000. He
is leading the Neurosys/Neurorhythms
team since 2015 at LORIA. His research focuses on braincomputer interfaces, motor systems, machine learning,feature extraction and selection, post-stroke rehabilitation,
artificial intelligence, and monitoring of generalanesthesia. He is currently an Associate Professor ofComputer Science at the University of Lorraine.

References
1.

The
lab
streaming
layer,
https://github.com/sccn/labstreaminglayer
2. K. Gorgolewski, T. Auer, V. Calhoun, et al. The brain
imaging data structure, a format for organizing anddescribing outputs of neuroimaging experiments.
Scientific Data, 3, 2016, 160044.
3. C. R. Pernet, S. Appelhoff, K. J. Gorgolewski, et al. EEGBIDS, an extension to the brain imaging data structure forelectroencephalography. Scientific Data, 6, 2019, 103.
4. A. Stoewer, C. J. Kellner, J. Benda, T. Wachtler and J.
Grewe, File format and library for neuroscience data andmetadata, Frontiers in Neuroinformatics. Conference
Abstract: Neuroinformatics 2014.
5. J. Grewe, T. Wachtler, and J. Benda, A bottom-up
approach to data annotation in neurophysiology, Front.
Neuroinformatics, 5(16), 2011.
6. nix-odML-converter,
https:/github.com/G-Node/nixodML-converter
7. A. Grimaldi, A. Gruel, C. Besnainou, J. N. Jérémie, J.
Martinet, L. U. Perrinet, Precise spiking motifs inneurobiological and neuromorphic data, Brain sciences,vol.13 no.1, 68, 2022.
8. S. J. Thorpe and M. Fabre-Thorpe, Neuroscience. Seeking
categories in the brain. Science, vol.291 no.5502, pp.260–
263, 2001.
9. M. R. V. Sanchez, S. Mishima, M. Fujiwara, G. Ai, M.
Jouaiti, Y. Kobryn, S. Rimbert, L. Bougrain, P. Hénaff, H.
Wagatsuma, Methodological Design for Integration of
Human EEG Data with Behavioral Analyses into HumanHuman/Robot Interactions in a Real-World Context, ICIC
Express Letters, 14(7), pp.693-701, 2020.
10. Autodesk
Fusion360,
https://www.autodesk.co.jp/products/fusion360/overview

Dr. Patrick Hénaff
He earned his M.Sc and Ph.D in
Robotics from Université Pierre et Marie
Curie in 1989 and 1994. He worked as alecturer at the University of CergyPontoise and a researcher at LISV
(Systems Engineering Laboratory at the
University of Versailles) and the
University of Cergy-Pontoise. Since
2013, he has been a Professor at the Ecole des Mines deNancy, University of Lorraine, and worked in the LORIA
laboratory. He led the Complex Systems, AI, and Robotics
department at LORIA from 2015 to 2022, and theComputer Science department at Mines Nancy from 2018
to 2023.
Dr. Hiroaki Wagatsuma
He received his M.S., and Ph.D. degrees
from Tokyo Denki University, Japan, in
1997 and 2005, respectively. In 2009, hejoined Kyushu Institute of Technology,
where he is currently an Associate
Professor of the Department of HumanIntelligence Systems. His researchinterests include non-linear dynamics
and robotics. He is a member of the IEEE society.

Authors Introduction
Dr. Masayuki Fujiwara
He graduated from Fukuoka Institute ofTechnology (FIT) in 2015. He then went
on to earned his Master and Ph.D
(Knowledge Science) degrees fromJapan Advanced Institute of Science andTechnology (JAIST) in 2017 and 2023.
Until 2020, he was a Research Fellow of
the Japan Society for the Promotion of Science (JSPS)(DC1) and is now a researcher at Kyushu Institute ofTechnology (Kyutech). His research focuses on the EEG
studies and neural modeling of (non-)symbolic
communication from the perspective of cognitive
neurodynamics.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1050

Terminal Synergetic Controller for Car’s Active Suspension System Using Dragonfly
Algorithm
Tinnakorn Kumsaen

Department of Chemical Engineering, Faculty of Engineering, Khon Kaen University, Khon Kaen, 40002, Thailand
Sorn Simatrang

Nacres Co., Ltd, Bangkok 10150, Thailand
Arsit Boonyaprapasorn

Department of Mechanical Engineering, Chulachomklao Royal Military Academy, Nakhon-Nayok, 26000, Thailand
Thunyaseth Sethaput

Sirindhorn International Institute of Technology, Thammasat University, Pathum Thani, 12120, Thailand
Email: tinnakor@kku.ac.th, urarl.urarl9@gmail.com, robosorn@gmail.com, thunyaseth@siit.tu.ac.th
Abstract
This research introduces a terminal synergetic controller (TSC) designed for the active suspension system of
automobiles through the implementation of the dragonfly algorithm (DA). The proposed controller aims to enhancethe dynamic performance of a car's suspension using the DA in tuning the system parameters. The stability of thedesigned controller is proved through the application of Lyapunov stability theory. Through iterative optimization
processes, the TSC approach seeks to achieve an optimal balance between ride comfort and vehicle handling. The
simulation results demonstrate that the proposed controller enhances convergence properties and alleviates thepresence of chattering. The results indicate that the proposed approach with the optimal parameters provided insights
into its potential application in improving the overall suspension system.
Keywords: Active suspension, Synergetic control, Feedback control, Metaheuristic; Swarm-based optimization

the proposed controller is to improve the stability of thecar by utilizing the DA for the precision tuning of systemparameters.

1. Introduction
Since the primary objective of controlling active
suspension systems is disturbance rejection, in thiscontrol application, it is imperative to select a potential
feedback control method that can effectively counteract
disturbances. Even though the robustness andenhancement of the convergence of the control system
can be achieved by using the concept of sliding mode
control (SMC), the method has a key drawback from thechattering phenomenon. This drawback can be avoided
by using the synergetic control (SC) which was proposedby Kolesikov et al. [1], [2]. The SC method allows thedesigner to achieve the desired characteristics of thecontrol system including parameter insensitivity androbustness if the macro variables are selected properly.
Sliding surfaces in sliding mode control can be utilized as
macro variables in the SC method. Consequently, theconvergence time of the control system under the SC
method can be improved by choosing macro variables
and/or dynamic evolutions with terminal attractors [3].
Swarm-based optimization algorithms, including the
dragonfly algorithm (DA) [4], enable the optimal
selection of controller parameters. The benefit ofachieving an improved convergence rate is obtained
when a feedback controller is combined with the DA [5].

2. Methodology
Lemma 1: Consider a nonlinear system in Eq. (1):x(t ) = f (x(t )) ,where x(t ) denotes a state eettor and x() :

(1)

→

n

,.

f () :→ is suffitiently smooth nonlinear mapping.
If there exists a positiee-definite and tontinuous
Lyapunoe funttion V (, ) : n  → [0, ) satisfied with
Eq. (2)nn

V (x(t ), t )  −V  (x(t ), t ), t  t0 , V (t0 )  0 , (2)where   0 ,  is a tonstant exponent suth that0    1 , and t0  0 is the initial time. Then,V (x(t ), t ) = 0, t  Ts ,(3)

where Ts is so tall the settling time and determined as

Ts = t0 +In this study, we present the TSC method specificallydesigned for the active suspension system of automobiles.
Based on the quarter-car model, the primary objective ofV 1− (x(t0 ), t0 )
 (1 − )2.1. Mathematical model©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1051

(4)

Tinnakorn Kumsaen, Arsit Boonyaprapasorn, Sorn Simatrang, Thunyaseth Sethaput

It is worth noting that the reasons for introducing  andadditional dynamics of Eq. (8) were elaborated in detailin [6]. To summarize, introducing  can provide us thedegree of freedom to guarantee the stability of the zerodynamics corresponding to the chosen output in Eq. (9)as we will further discuss at the end of section 2.3. In
addition, it will play an important role in the tradeoff
between ride quality and rattle space usage.

In this study, the model of the suspension systemderieed from Newton’s law presented in [6], [7] was used.
Fig. 1 shows a quarter-car model of an active suspension
which includes the hydraulic actuator in the suspension
system. The model can be extended by considering thehydraulic dynamics can be found in [6]. However, in thisstudy, the model views the hydraulic actuator in thesuspension system as the control input ua and ignores thehydraulic dynamics as shown in Eq. (5):

Mb xs + Ka ( xs − xw ) + Ca ( xs − xw ) − ua = 0
Mus xw + Ka ( xw − xs ) + Ca ( xw − xs ) + Kt ( xw − r) + ua = 0(5)where Mb represents the car body mass and M us thewheel. The displacement of car body and wheel aredenoted by xs and xw .The stiffness of the active
suspension system are defined by Ka and Kt . The
damping coefficient is represented by Ca . The force ua
is the actuator force from the hydraulic system. The road
disturbance is defined by r and it is assumed to bebounded.

Fig. 1. A quarter-car model with active suspension
system

According to [6], [7] by letting x1 = xs x2 = xs x3 = xw
and x4 = xw , the corresponding state space representationis shown in Eq. (6):2.2. Controller design
According to [1], [2], [3], the terminal synergetic
controller design procedure can be presented as follows:Firstly, define the macro variable of the controller.

x1 = x2
x2 = −1
 Ka ( x1 − x3 ) + Ca ( x2 − x4 ) − ua Mb

x3 = x41
x4 = Ka ( x1 − x3 ) + Ca ( x2 − x4 ) − Kt ( x3 − r ) + ua 
M us

p

(6)

 = e+( e) q.

(10)

where p and q are positive odd numbers such that p  q
Second, define the dynamic evolution of the macro
variable as Eq. (11):The control objective is to control the error in Eq. (7) toconverge to zero:me = x1 − x3 ,( ) n + = 0(7)

where 1  m n  2 , m and n are positive odd numbers,

where x3 is the filtered signal of the wheel displacement,  0 is parameters to be determined by the designer andx3 . The filter is defined as:x3 =  ( x3 − x3 ) .

affect the convergence time of the macro variable toconverge to zero. Finally, determine the control input asfollows. By Eq. (10), the derivative of the macro variable
is,

(8)

where   0 is the parameter to be determined later. The
output is defined as:ye = x1 − x3 .

(11)

p

−1

p(e) q e
 = e+q(9)

Recall that e = x1 − x3 , then.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1052

(12)

Terminal Synergetic Controller fore = x1 − x3 = x2 +  ( x3 − x3 )( ) = − 1  2 2  = −V 
V ( ) = −






And,

3

3

4


2 

(18)

Based on Lemma 1, the settling time T1 is determined as K ( x − x ) + Ca ( x2 − x4 ) −  2 ( x − x ) −  x + ua
e=− a 1 3
Mb



n
m

(13)

( (t ) )T1 = t0 + 1− 0.
2  (1 − )2 − 2

Mb

(19)

(14)
Then, this implies that the manifold  = 0 , t  T1 .
From Eq. (10),

By using the Eq. (11) – Eq. (12) and Eq. (14), we candetermine the controller ua as

e = − e




q
  (− )2ua = M b  − e+  ( x3 − x3 ) +  x4 p−1
 p(e) q 
 (15)+  Ka ( x1 − x3 ) + Ca ( x2 − x4 )

(20)

n
m

p

whenever the manifold  = 0 , where 





q
p

and

q

. Furthermore, notice that   0 and both of p ,p
q are positive odd numbers such that p  q , then   0
and 0    1 by construction. Again, by Lemma 1, theerror e(t ) converges to zero within the time:−1

Note that to guarantee that the term (e) q will not cause
the singularity, the additional condition must be imposedon the parameters p , q , m , n ,which the following

2 − 2
e(T1 ) )(T =T +m( p − q)
pn + m ( q − p )i)1.
 0 and ii)
pn
qn

2

1

21−  (1 − )(21))

Hence, x1 (t ) converges to x3 (t ) within T = t0 + T1 + T2

2.3. Proof of stability

which is a finite real number. Note that, since we choose
the output y x1 − x3 , we can obtain the zero dynamics

The Lyapunov function is defined in terms of the macro
variable as1
V ( ) =  2
2

in the matrix form as Eq. (22) [6] by setting the output
identically equal to zero.

(16)

z = Az z + Bz r
where zBy Eq. (12), the derivative of Eq. (16) is determined as−1p(e) q e V ( ) =  =   e +

q 



x4 

T

Az

Bz

0 0 Kt
M us  .

By substituting the controller in Eq. (15) into Eq. (17), we
yield,( − )V ( ) = 

x3



 −
 0

 2 Mb
 −
M us


p

(17)

 x3

(22)




0 
0
1  and  2 M b − Kt 
Mb 


 −
M us 
 M usTnm

and r is assumed to be a bounded road disturbance i.e.,r  M r where Mr is a positive constant.

Recall that m and n are positive odd number such that
2
n+m

=
and
.
0.5  n  1 and let define  =m2m
Note that 0.75    1 and   0 by tonstruttion, then,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1053

Tinnakorn Kumsaen, Arsit Boonyaprapasorn, Sorn Simatrang, Thunyaseth Sethaput

One tan easily realize that the matrix Az is Hurwitz if andFig. 2 and Fig. 3 illustrate that the proposed TSC
method can effectively guide all state responses to thedesired levels of displacement and velocity. Nevertheless,upon comparing the controller with and without optimal
parameters obtained from DA, it is evident that the TSC
with optimal parameters exhibits a superior convergence
rate performance. Fig. 4 demonstrates that the TSC with
optimal parameters can alleviate the presence ofchattering.

only if   0 by using the Ruth-Hurwitz criterion. It isalso worth noting that there is no finite escape time
phenomenon that occurs no matter how large themagnitude of road disturbance r is since the zerodynamics Eq. (22) is a linear system with z is the statevector of the system and r is the disturbance. From thedesign perspective, one can choose a parameter  such thatthe matrix Az is sufficiently robust enough to dominate thebounded road disturbance r to guarantee the convergenceof the state z to zero. Then, the state x1 and x2 converge tozero for the following reasons. Since x1 converges to x3 infinite time by Eq. (18) and the argument from Eq. (20) i.e.,e = − e when the manifold  = 0 . Then, whenever
z

 x3

x3

x4  is identically equal to zero, it implies
Tthat x1 is identically equal to zero and then it also implies
that x2 = 0 i.e., x2

x1

Hence, the proof of locallyterminal time stability is completed
2.4. Controller parameter tuning
By using the algorithm of DA, the controller
parameters of the design control input as shown in Eq.
(15) are selected optimally to minimize the cost function
or performance index which is defined as
Fig. 2. Comparison of state response x1 , x3 , and x3 ofthe control system before and after DA tuning

J = ua T ua

3. Simulation Results and Discussions
To show the capability of the designed controller, themodel in Eq. (6) with the system parameters from [8], [9]are used as the simulation example. The systemparameters are defined as follows: Mb = 290 kg,

Mus = 59 kg, Ka = 16,812 N/m, Kt = 190,000 N/m,
Ca = 1000 Ns/m The controller parameters are given by
DA with 40 search agents and 20 iterations. The
controller parameters before tuning are arbitrarily definedas 1 = 0.228 , 1 = 0.9431 , 1 = 1.3356 , and thecontroller parameters after tuning are 2 = 3.9648
2 = 0.10942 ,  2 = 3.7505 . Both cases are using p = 5 ,q = 3 , m = 7 , n = 5 . We formulate the road disturbance
based on the principles outlined in [6]. This disturbanceis mathematically described as a single bump with aheight of 5 cm ( a = 0.025 ), defined as:r ={a (1 − cos8 t ) , 3.5  t  3.75
0, otherwise

Fig. 3. Comparison of state response x2 and x4 of thecontrol system before and after DA tuning©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1054

Powered by TCPDF (www.tcpdf.org)

Terminal Synergetic Controller for

9.

A.G. Alleyne, P. Neuhaus, and J.K. Hedritk, “Applitationof nonlinear control theory to electronically controlledsuspensions,” Vehitle System Dynamits, eol. 22, no. 5–6,
pp. 309–320, Jan. 1993.

Authors Introduction
Dr. Tinnakorn Kumsaen
He received his B.S degree in chemicalengineering from Khon Kaen University,
Thailand, M.S. degree in systemengineering from Case Western Reserve
University (CWRU), and Ph.D. inchemical engineering from CWRU. He iscurrently a faculty member at chemical engineering, Khon
Kaen University. His research interests are control system,
process design simulations & integrations, and renewableenergy.

Fig. 4. Comparison of state response of the control
input before and after DA tuning
4. Conclusion
In this study the TSC method was applied forcontrolling of the active suspension system. The
controller parameters are selected optimally based on thealgorithm of DA. The proposed controller effectively
stabilizes the active suspension system by Lyapunovstability analysis. Simulation of the control system wascarried out to validate the efficacy of the proposed
controller. The simulation results demonstrate that the
proposed controller enhances convergence properties.
Additionally, the observed decrease in chattering in thecontrol input signals, a significant issue in sliding mode
control, further supports the potential application of the
proposed TSC method in enhancing the overall
performance of the suspension system.

Mr. Sorn Simatrang
He received a B.S. in electricalengineeringfromChulalongkorn
University, Thailand, and an M.S. in
systems and control engineering from
Case Western Reserve University, Ohio,
USA in 2006 and 2015, respectively.
Currently, he works as a control systems
engineer at Nacres Co., Ltd, Bangkok, Thailand. His main
research interests include output feedback control
problems, adaptive control problems for nonlinear
systems, and applications of nonlinear control theory tosystems biology and mechanical systems.
Dr. Arsit Boonyaprapasorn
He received a B.S. in MechanicalEngineering from King Mongkut ’s
University of Technology (KMUTT),
Thailand, in 1998. He continued his
postgraduate studies at Case Western
Reserve University, Cleveland, Ohio,
U.S.A., obtaining an M.S.in Systems andControl Engineering in 2003 and a Ph.D. in MechanicalEngineering in 2009.Since then, he has served as a lecturer
at Chulachomklao Royal Military Academy, Nakhon
Nayok, Thailand. His research interests revolve aroundnonlinear feedback control and robotics.

References
1.

2.

3.

4.

5.

6.

7.

8.

A.A. Kolesnikoe, “Introduttion of synergetit tontrol,”
Proceedings of the 2014 American Control Conference,
Portland, OR, USA, Jun. 4-6, 2014, pp 3013-3016
A.A. Kolesnikov, Modern Applied Control Theory:
Synergetic Approach in Control Theory. Moscow, Russia:
Integracia-TSURE publ, 2000.
Y. Feng, X. Yu, and Z. Man, “Non-singular terminal
sliding mode tontrol of rigid manipulators,” Automatita,
vol. 38, no. 12, pp. 2159–2167, Dec. 2002
S. Mirjalili, “Dragonfly algorithm: a new meta-heuristic
optimization technique for solving single-objective,
discrete, and multi-objettiee problems,” Neural Computing
and Applications, vol. 27, no. 4, pp. 1053–1073, May 2015
A.
Boonyaprapasorn,
S.
Kuntanapreeda,
P.S.
Ngiamsunthorn, T. Kumsaen, and T. Sethaput, “Timevarying sliding mode controller for heat exchanger withdragonfly algorithm,” International Journal of PowerElectronics and Drive Systems, vol. 13, no. 4, p. 3958, Aug.
2023.
J.S. Lin and I. Kanellakopoulos, “Nonlinear design ofattiee suspensions,” IEEE Control Systems Magazine, eol.
17, no. 3, pp. 45–59, Jun. 1997.
J.S. Lin and I. Kanellakopoulos, "Nonlinear design ofactive suspensions," Proceedings of the 34th IEEE
Conference on Decision and Control, New Orleans, LA,
USA, 1995, pp. 3567-3569.
A. Alkyne and J.K. Hedritk, “Nonlinear Control of a
Quarter Car Attiee Suspension,” Proteedings of the 1992
American Control conference, Chicago IL, pp. 21-25.

Dr. Thunyaseth Sethaput
He received his B.S. degree inmechanical engineering from Sirindhorn
International Institute of Technology,
Thammasat University, Thailand andPh.D. in systems and control
engineering, from Case Western Reserve
University, Ohio, USA. He is anassistant professor and chairperson of mechanicalengineering program at Sirindhorn International Instituteof Technology, Thailand. His research interests are control
systems, systems biology, biomedical mechanics,
simulation modeling.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1055

Rehabilitating Flood-Damaged Cars for
Sustainable Car Rental Services: A Web-Based System
Pon Xiao Qi

Institute of Computer Science and Digital Innovation, UCSI University, Kuala Lumpur, Malaysia
Abdul Samad bin Shibghatullah

Institute of Computer Science and Digital Innovation, UCSI University, Kuala Lumpur, Malaysia
Kasthuri Subaramaniam

Institute of Computer Science and Digital Innovation, UCSI University, Kuala Lumpur, Malaysia
Email: 1001749088@ucsiuniversity.edu.my, abdulsamad@ucsiuniversity.edu.my,kasthurisuba@ucsiuniversity.edu.my

Abstract
Reusing flood-damaged cars gives a unique opportunity in the car rental market in today's fast-paced world, where
sustainable solutions are crucial. This study represents the result of efforts to design and construct a web-based car
rental system that utilizes flood-damaged cars. The system will be developed using Rapid Application Development(RAD) methodology. Surveys and interviews are conducted to collect potential users’ feedback. The results showthat due to global economic recession and the increased awareness of global warming, citizens will prefertransportation that is eco-friendly and cost effective if the vehicle is in good condition and properly fixed. This
project intends to transform the car rental environment by leveraging a strong technological stack and a user-centric
design. With a strong emphasis on user experience and system efficiency, the web-based platform promises not only
to ease seamless vehicle rentals but also to promote sustainability in the automotive sector. The favorable results ofuser acceptability testing confirm the system's success in reaching its objectives. GPS features and transparent
vehicle history disclosure will be included for future enhancements.
Keywords: Car Rental System, Eco-Friendly, Flood

In 2023, 71% of companies will have websites. This is arise over prior years, owing primarily to the influence of
the Covid-19 epidemic. Companies have realized thevalue of having a web site to reach a bigger audience andstay competitive in today's digital market because of themove to online commerce and remote employment. The
trend of companies creating a website has been hastenedby the Covid-19 pandemic, and it is probable that thistrend will continue.

1. Introduction
The car industry has long struggled with the issues ofsustainability
andenvironmental
friendliness.
Simultaneously, natural calamities such as floods haveresulted in many flood-damaged cars that, despite being
salvageable, are frequently underutilized. Recognizing thepotential of combining these two factors, this project will
create a web-based automobile rental system that willreuse flood-damaged vehicles. This system solves theenvironmental challenges connected with underutilized
cars while also catering to the increasing demands ofmodern consumers seeking economical and sustainabletransportation alternatives by providing a digital platform
for vehicle renters.

The latest research shows that 28% of all company
activity is now done online. This transition to online
shopping reflects the internet's growing significance intoday's business environment and the expanding trend of
e-commerce.
Recent studies show that 43% of small firms intend to
spend money in 2023 to improve the functionality of theirwebsites. This phenomenon highlights the growing
significance of having a strong online presence forcompanies of all sizes. A business' success in today's
digital environment depends on having a well-designed
and optimized website, as consumers are increasingly
turning to the internet to make purchasing decisions [1].

2. Methodology
2.1 Latest Statistic on Website Usage
In 2023, there are approximately 1.13 billion websites
on the internet. Only a small portion of the world's 1.13
billion websites are regularly updated and used. Only
200,121,724 of the 1.13 billion websites are regularly
maintained and viewed, a startling 82% of which areinactive. Each three seconds, a brand-new website is
created.

Moreover, 5.18 billion people used the internet at thestart of the second quarter of 2023. The number of©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1056

Pon Xiao Qi, Abdul Samad bin Shibghatullah, Kasthuri Subaramaniam

individuals using the internet is increasing; according to
the latest recent data, there were 147 million more users
globally connected in the year before April 2023. Due to
growing internet availability globally, the number ofpeople who are "unconnected" to the internet has fallen to
2.85 billion, with the majority of these people living inSouthern and Eastern Asia as well as Africa.

season, the east coast and eastern Malaysia are typicallyaffected by floods during December to January when thenortheast monsoon sweeps. Most floods in Malaysia
typically originate from persistent, heavy rains that cause
runoff because of surplus water supplies that exceed thecapacity of streams and rivers. Many states in Peninsular
Malaysia, including Kelantan, Johor, Pahang, Perak,
Kuala Lumpur, and Selangor, as well as east Malaysia's
Sabah and Sarawak, are home to the majority of floodprone areas [6].

Mobile phones now account for approximately 60% of
all web traffic and are used by 95% of internet users
globally to access the internet at least periodically as
shown in Fig 1. People spend more than 57 percent of
their online time on mobile devices. However, in theworld's major economies, more than 6 in 10 internet users
still utilize laptops and desktop computers for at least
some of their online activity [2].

2.3 Overview of Car Rental System
The development of technology caused fierce
competition among businesses to win over more clients[7]. Technology aids in decision-making by allowing theowner of the car rental to concentrate on the usage of IT
to gain a competitive edge [8]. Car rental companies
frequently struggle with transactions that are inefficient.
The tenant must make individual calls to the rental
provider to inquire on the car's availability [9]. Distance isa problem for communication and information
technologies, yet a website allows us to access a wide
range of information from around the globe. Onlinetransactions are those when the buyer and seller conduct
their business over the Internet. Web-based Information
Systems were created because of the quick advancementof information technology. Research on general models
for the study and design of web-based information
systems has been done in numerous papers [10]. Due tothe current rapid growth of information technology, it isessential to access information quickly and accurately tofacilitate decision-making. Due to the accessibility of
information and the effective and efficient processing oforder data, using web design as a tool to promote therental of vehicles is highly beneficial in attracting moreconsumers and making transactions easier.

Fig 1. Statistics of internet usage

2.2 History of Flood in Malaysia
According to studies conducted specifically in Malaysia,
the fast-rising number of flood events in recent decades
are the result of unregulated human activity in rivers, thecutting down of forests, and extreme weather eventsbrought on by climate change [3]. Since 2001, there havebeen 143 floods on average annually, more than 90% of
which have been flash floods. Such often recurrent floods
pose a serious risk to the lives and property of Malaysian
inhabitants, severely damaging the nation's infrastructure[4].

2.4 Problem Statement
Natural calamities, such as floods, have caused severe
damage to automobiles in Malaysia in recent years,
bringing huge financial damages to individuals.
Depending on the type and level of damage, it might cost
the automobile owner between RM20,000 and RM40,000.
Repair costs will be higher for luxury vehicles such as
BMW, Porsche, Range Rover, Peugeot, Mini Cooper, andso on. Due to the amount of damage in the car, some carowners may be unable to afford the repair costs or may
opt to sell it and acquire a new car. These cars will almost
certainly wind up in landfills, contributing to pollution.
This is because not all vehicles have flood insurance.
Driving a flood-damaged car that has not been adequately
repaired might be hazardous to the driver's health andthreaten the safety of other vehicles on the road. Mold canform in flood-damaged cars if wet conditions are nothandled, causing respiratory difficulties for the driver andpassengers. A web-based car rental system for flooddamaged vehicles can provide a convenient option for

According to information from the Country Disaster andRisk Profile of the United Nations Office for Disaster
Risk Reduction (UNDRR), floods in Malaysia account for
98% of average yearly damage between 1990 and 2014.
According to a report from the Malaysian Department of
Irrigation and Drainage published in 2003, flooding
affects an average of 29 000 km2, or 9 percent of thenation's total land area, and more than 4.82 million people(22 percent of the population) per year [5].
Two different forms of rainfall generate floods. The firstkind of rainfall is of a moderate intensity and lasts for along time over a large region. The second kind is localized
heavy rain that lasts for a short time. Floods in Malaysia
are predictable to happen. While the west coast ofPeninsular is primarily plagued by thunderstorms from
September to November because of the inter-monsoon©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1057

Rehabilitating Flood-Damaged Cars forthese car owners to sell their flood-damaged vehicles
through a reputable platform. The owner can utilize themoney from selling the flood-damaged car to our platform
to buy a new car. Our platform will then repair theseflood-damaged vehicles and rent them out at a low cost.

ensure that the system's performance, usability, andfunctionality are satisfactory.
3.2 User Functionality
Users should be able to quickly register on the suggested
website. Important information such as their name andcontact information will need to be obtained throughout
the registration process. Users must be able to securely
log in using their selected credentials (username andpassword) once registered, allowing them to tailor theirinteractions with the system and log in or out as needed.

Without a web-based car rental system, flood-affected
car owners must either continue to use the vehicle,
dispose it in landfills, or spend time contacting
secondhand car dealers who are prepared to acquire theirflood-damaged vehicle, which is a time-consuming
process. The goal of this project is to reduce the numberof vehicles that wind up in landfills, hence reducing
environmental effect. Furthermore, reusing flooddamaged vehicles through rental businesses is a moresustainable option than scrapping them. Furthermore, anaffordable automobile rental system including flooddamaged vehicles would provide transportation options
for people who might not be able to purchase a carotherwise. This accessibility would allow people tocommute to job, education, or other important places,
making their lives easier. Furthermore, these flooddamaged vehicles might be used in the film business. Our
platform ensures that vehicles are thoroughly inspected
and repaired prior to rental, ensuring renters' safety.

Access to a comprehensive list of available cars is an
important component of the user experience. This list
should include specific information about each car, suchas its model, year, rental fee, and a brief description. A
search option should be included to improve user ease,allowing consumers to filter available cars based onparticular parameters such as location and rental term.
This enables users to quickly select the appropriate carthat suits their individual requirements.
Furthermore, reserving a car for a specific time should
be a simple process. Users should be able to enter theirpreferred pick-up and return dates, allowing for smoothbooking procedures. Users should also be able to alter orcancel previous bookings to accommodate changes inplans if the request is made at least three days before thescheduled pick-up date.

3. Implementation
3.1 Method

The Touch & Go payment system, which wasrecommended by 96% of poll respondents, will be
included in the system. This strategy offers quick and easy
payment processing, which improves the entire customer
experience.

The proposed car rental website is built using the Rapid
Application Development (RAD) methodology. The RAD
methodology is divided into four phases: user
requirements, user design, prototype, and implementation.
During the user requirements phase, system requirements
will be gathered through surveys and questionnaires. Oncethe requirements are defined, we will move into the user
design phase. The emphasis here was on creating thehigh-level structure of the web-based car rental system.
We envisioned the system's components, theirinterrelationships, and the general architecture that would
support its functionality.

Additionally, the system will highlight how importantuser feedback is. Users should be able to provide ratingsand comments about their rental experience after
completing a booking. Future users can benefit greatly
from these user reviews, which offer insights and support
well-informed decision-making. A thorough booking
history that provides a summary of prior reservations for
simple access and tracking of rental activity should also
be available to users. To safeguard our clients, ourbusiness will offer a warranty on every vehicle. However,GPS features and transparent vehicle history disclosure
will be included for future enhancements (Fig 2).

The prototyping step marked the move from concept toactual representation. At this point, we'll start working on
a workable model of the system. This prototype served as
proof of concept as well as a visual depiction of thesystem's primary functions. It enabled my supervisors andpotential customers the opportunity to interact with asimplified version of the system and obtain a hands-on
grasp of how it would work. This phase included
continuous testing and validation to ensure that the systemremained reliable and of high quality throughout
development.
The final phase, implementation, marked the web-based
automobile rental system's preparedness for deployment.
To ensure a successful launch, several critical procedures
will be followed. Extensive testing will be carried out to©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1058

Pon Xiao Qi, Abdul Samad bin Shibghatullah, Kasthuri Subaramaniam

References
1.

Haan, K. (2023) Top website statistics for 2023, Forbes.
Availableat:https://www.forbes.com/advisor/business/software/websitestatistics/#:~:text=There%20are%20about%201.13%20billi
on,are%20actively%20used%20and%20updated.
(Accessed: 30 May 2023).
2. Digital around the world - datareportal – global digital
insights (no date) DataReportal. Available at:https://datareportal.com/global-digital-overview (Accessed:
30 May 2023).
3. Aliagha, G.U., Mar Iman, A.H., Ali, H.M., Kamaruddin, N.
and Ali, K.N., 2015. Discriminant factors of flood
insurance demand for flood‐hit residential properties: acase for M alaysia. Journal of Flood Risk Management,
8(1), pp.39-51.
4. Nasiri, H. and Shahmohammadi-Kalalagh, S., 2013. Floodvulnerability index as a knowledge base for flood risk
assessment in urban area. Journal of Novel Applied Science,2(8), pp.269-272.
5. D'Ayala, D., Wang, K., Yan, Y., Smith, H., Massam, A.,
Filipova, V. and Pereira, J.J., 2020. Flood vulnerability andrisk assessment of urban traditional buildings in a heritage
district of Kuala Lumpur, Malaysia. Natural Hazards andEarth System Sciences, 20(8), pp.2221-2241.
6. Safiah Yusmah, M.Y., Bracken, L.J., Sahdan, Z.,
Norhaslina, H., Melasutra, M.D., Ghaffarianhoseini, A.,
Sumiliana, S. and Shereen Farisha, A.S., 2020.
Understanding urban flood vulnerability and resilience: acase study of Kuantan, Pahang, Malaysia. Natural Hazards,
101, pp.551-571.
7. Adiyanto, N., 2021. Customer Relationship Management(CRM) Based On Web To Improve The Performance Of
The Company. IAIC Transactions on Sustainable Digital
Innovation (ITSDI) The 1st Edition Vol. 1 No. 1 October
2019, 32.
8. Soegoto, E.S., 2018, November. Information Technology
Based Entrepreneurship Education in University. In
International Conference on Business, Economic, Social
Science and Humanities (ICOBEST 2018) (pp. 316-320).
Atlantis Press.
9. Rozi, N.F., Ruswiansari, M., Rachman, A., Wardhana, S.R.
and Istiyanto, L., 2019. The Development of LIDI: A WebBased Car Rent Marketplace Application in Sidoarjo,
Indonesia. In IOP Conference Series: Materials Science andEngineering (Vol. 462, No. 1, p. 012052). IOP Publishing.
10. Welzer, T., Eder, J., Podgorelec, V., Robert, W., Ivanovic,
M., Gamper, J., Morzy, M., Tzouramanis, T., Darmont, J.
and Kamišalić, A., 2019. New Trends in Databases andInformation Systems. In 23rd European Conference on
Advances in Databases and Information Systems (ADBIS2019) (Vol. 1064). Springer.

Fig 2. Use Case Diagram

4. Conclusion
Administrators should be equipped with powerful tools
to efficiently manage automobile listings. This involves
having the power to update current listings, remove outof-date ones, and add new cars to the inventory. Thisguarantees the accuracy and currentness of the vehicle
inventory.
Besides this, administrators need to be able to supervise
and manage every user account. This includes having thepower to activate and deactivate admin and user accounts
as necessary. Admins also can make new admin accounts,
which will improve platform integrity and security.
All bookings must be managed efficiently.
Administrators should be able to approve or cancel
reservations based on availability and particular criteria.
Admin should be able to manage user reviews as well.
The administration of user reviews is equally important inmaintaining the platform's positive and trustworthy
reputation.
Moreover, admins should be able to export booking
history records to aid in record-keeping and analysis. This
information is crucial in accounting and auditing. Admins
should also be able to track the platform's financial
performance over time by viewing revenue data, which
will help them assess the platform's viability and growth.
Lastly, both users and administrators should have accessto a visual booking calendar that displays all bookings
made on certain days. This calendar tool aids in efficientplanning and management, improving the system's overall
functionality.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1059

Powered by TCPDF (www.tcpdf.org)

Rehabilitating Flood-Damaged Cars for

Authors Introduction
Ms. Pon Xiao Qi
She is currently pursuing B. Sc. (Hons)Business Information Systems at Institute ofComputer Science and Digital Innovation(ICSDI), UCSI University, Kuala Lumpur,
Malaysia.

Dr. Abdul Samad bin Shibghatullah
He received the bachelor accounting
degree from Universiti Kebangsaan
Malaysia, Bangi, Malaysia in 1999, theM.Sc. degree in computer science from the
Universiti Teknologi Malaysia, Skudai,
Malaysia in 2002, and the Ph.D. degree incomputer science from the Brunel University of Uxbridge,
United Kingdom. He is currently an Associate Professor at
Institute of Computer Science etc.
Dr. Kasthuri Subaramaniam
She is currently an assistant professor at
Institute of Computer Science and Digital
Innovation (ICSDI), UCSI University,
Kuala Lumpur, Malaysia. She earned bothher bachelor’s degree in computer scienceand a master’s degree in computer sciencefrom the University of Malaya. Her research interests
include human-computer interaction, etc.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1060

Optimizing E-Invoicing Rollout: Adaptive E-Invoicing Rollout (AER) Framework for
Navigating Malaysia's Digital Transformation
Koh Chee HongUCSI University, Kuala Lumpur, Malaysia
Email: 1002265962@ucsiuniversity.edu.my
Abdul Samad Bin Shibghatullah

UCSI University, Kuala Lumpur, Malaysia
Email: AbdulSamad@ucsiuniversity.edu.my
Abstract
In the dynamic landscape of Southeast Asia's digital economy, Malaysia is gearing up for a transformative shift withthe mandatory rollout of electronic invoicing (e-Invoicing) beginning June 2024. Recognizing the critical need for astructured approach to this transition, this study presents the design and evaluation of an Adaptive E-Invoicing Rollout(AER) Framework, which is aimed at facilitating a successful e-Invoicing implementation across Malaysia's diverse
business sectors. The framework is developed through a qualitative methodology that synthesizes insights from anextensive literature review, expert interviews, case studies, and the Delphi technique. The AER Framework stands
out for its comprehensive approach, incorporating a theoretical predictive model that assesses various variables
influencing e-Invoicing success. By employing the correlation coefficient, the model quantifies the relationships
between key variables, providing a structured mechanism to predict and enhance the successful adoption of eInvoicing. This model is particularly designed to be adaptable and dynamic, evolving with ongoing insights andchanges in the e-Invoicing landscape. The study's findings offer a significant contribution to the digital economic
literature and practice, providing policymakers, businesses, and stakeholders with a robust tool for strategic planningand effective decision-making. As Malaysia navigates its digital transformation, the AER Framework serves as a
critical guide, ensuring that the e-Invoicing rollout is not only compliant with regulations but also conducive toenhancing fiscal transparency, economic efficiency, and business operations. Furthermore, the framework's
adaptability and predictive nature make it a relevant model for other countries considering similar digital transitions,marking it as an asset in the broader context of digital finance evolution.
Keywords: Electronic Invoicing, Digital Transformation, Adaptive Framework, Predictive Model, Qualitative
Methodology, Malaysia, Correlation Coefficient, Stakeholder Strategy, Fiscal Transparency

this research. This study aims to contribute to the body ofknowledge by providing an empirical evaluation of thefactors influencing e-Invoicing success in Malaysia,
thereby guiding policymakers and businesses in theirdecision-making processes. Moreover, it will offer anovel predictive model to anticipate the trajectory of eInvoicing adoption and its ripple effects on the digital
economy, drawing on the insights from the ASEAN
Digital Economy framework [6].

1. Introduction
In the burgeoning digital economy of Southeast Asia,
Malaysia stands on the precipice of a significant
transformation with the impending mandatory rollout of
e-Invoicing beginning June 2024. The introduction of eInvoicing is not merely a technological shift, but astrategic maneuver poised to streamline tax compliance,
bolster fiscal transparency, and catalyze economic
efficiency. However, despite the clear benefits and thecomprehensive guidelines issued by the Inland Revenue
Board (IRB), the successful adoption and implementationof e-Invoicing across diverse business ecosystems
presents a complex challenge.

The novelty element of this study is encapsulated in its
comparative analysis approach, juxtaposing the
Malaysian e-Invoicing rollout with global best practices[7], [8] and its predictive nature, which seeks to forecast
the long-term implications of e-Invoicing on the digitaltransformation of the Malaysian economy [9], [10]. By
examining the intersection of digital payments, data
regulations, and e-Invoicing [5], [11], along with thecritical factors for cloud-based e-invoice service adoption[12] and the effects of e-government measures on 2.
taxpayer compliance [13], this research will carve out anovel paradigm in the digital economic literature,
specifically tailored to the Malaysian context. The
proposed research is poised to offer a robust, scalable,
and adaptive framework for e-Invoicing implementationin Malaysia, contributing significantly to the literature

The significance of this study lies in its potential to bridge
the gap in understanding the multifaceted dimensions of
e-Invoicing adoption, from the perspective of compliance[1],[2] to the readiness of technological infrastructure[3],[4]. As noted by Chan [5], the digital economy
collaboration and the integration of digital payment
systems are crucial areas of development in ASEAN,
with Malaysia being no exception. Yet, a comprehensive
framework that encompasses all aspects of the transition
to e-Invoicing is absent, underscoring the necessity for©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1061

Koh Chee Hong, Abdul Samad Bin Shibghatullah

and providing practical implications for stakeholders
across the board.

invoicing success rate. It indicated both the strength anddirection of their relationship.
Pearson's Correlation Formula Simplified:

3. Methodology𝑟=
The methodology for the research study, " Optimizing EInvoicing Rollout: Adaptive E-Invoicing Rollout (AER)
Framework for Navigating Malaysia's Digital
Transformation" will be composed of a qualitative
approach, given the pre-implementation state of einvoicing in Malaysia.

𝑛∑𝑥𝑦 − (∑𝑥)(∑𝑦)√[𝑛∑𝑥 2 − (∑𝑥)2 ][𝑛∑𝑦 2 − (∑𝑦)2 ]Where:• r (correlation coefficient): This was the desired
outcome of the analysis. It is a value between -1
and 1 that indicated how strongly two variables
were related.
• n (number of pairs): This represented the total
number of observations or data points available.
Each pair represented a set of two variables,such as a particular e-invoicing feature's
implementation and its observed impact onoverall success.
• Σxy (sum of the product of paired scores): Foreach pair of variables, they were multiplied
together and then all these products weresummed up. This sum helped understand thejoint variability of the two variables.
• Σx and Σy (sum of x scores and y scores):
These were simply the totals of all the first andsecond variables in the pairs respectively.
• Σx² and Σy² (sum of the squares of x scoresand y scores): Similar to Σx and Σy, but eachvariable was squared before they were summed
up. This component helped in understanding the
spread or distribution of each variable.

3.1. Literature Review
A comprehensive literature review was conducted tosynthesize global and local e-invoicing practices,
identifying critical themes and practices that inform a
predictive model for e-invoicing success. The key
findings are summarized in Table 1.0: Key Findings from
Literature Review.
3.2. Expert Interviews
Insights were gathered from industry experts tounderstand the operational, technical, and complianceaspects of e-invoicing implementation. The synthesized
perspectives are documented in Table 2: Table 2.0:
Theoretical Predictive Model Variables for E-Invoicing
Success.
3.3. Case Study Analysis
An analysis of various regional e-invoicing
implementations provided criteria for success,highlighting parallels and distinctions across ASEAN
countries. Findings are presented in a comparative format.

The results of this predictive analysis, including
correlation coefficients and predictive importance ratings,
were summarized in Table 2.0: Theoretical Predictive
Model Variables for E-Invoicing Success. This table
served as an essential tool for stakeholders, offering
insights into the influential factors of e-invoicing success.
It enabled stakeholders to anticipate potential outcomes
more effectively and align strategies accordingly.

3.4. Delphi Technique
A Delphi study established consensus on predictive
parameters for e-invoicing success, integrating expert
opinions into the AER Framework's development criteria.
Parameters such as system integration readiness,compliance rates, and user adoption were discussed,
echoing the approach suggested by Wagiman et al. [14]in applying the Delone & McLean Information Systems
Success Model to assess net benefits.

3.6. Documentation and Reporting
The meticulous documentation and reporting of findings
throughout the research process ensured that theframework would be dynamic and subject to continuous
improvement. This living document approach allows theframework to evolve as new insights emerge, maintaining
its relevance and effectiveness.

3.5. Theoretical Predictive Analysis
The theoretical predictive model was developed from an
integration of literature reviews, expert interviews, andcase study findings. It evaluated various variables
impacting the success of e-invoicing, each assigned arating from 1 to 5 for its predictive importance.

4. Result/Discussion

4.1. Development of the AER Framework

Application of the Correlation Coefficient:

The "Adaptive E-Invoicing Rollout (AER) Framework"is the outcome of a methodical approach that
amalgamates a comprehensive literature review, expert
interviews, comparative case studies, Delphi consensus,

The correlation coefficient, a crucial metric in this model,
was calculated to measure the relationship between pairsof variables, such as regulatory compliance and e-©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1062

Optimizing E-Invoicing Rollout: Adaptive

Table 2 Theoretical Predictive Model Variables for EInvoicing Successand theoretical predictive analysis. This blend has
formulated a detailed and adaptable guide suitable for thenuanced needs of Malaysia's e-invoicing initiative.

Variable(Factor)4.2. Literature Review Outcome
The outcome of the literature review of e-invoicing
related journal papers were consolidated in the Table 1
below:

Cintegration(System & Data
Integration
Capability)
Aadoption(Rate ofAdoption
Among Users)
Rreadiness(Organizational&
Technological
Readiness)
Ccompliance(Compliancewith Legal &Tax
Requirements)
Uusage (Actual
Usage &
Utilization
Rates PostImplementation)

Table 1 Key Findings from Literature Review
Theme

Key
Findings

DigitalEconomy &E-Invoicing

Implementation & Adoption
Challenges

Benefits
Outlook&

Regulatory
Aspects&
Compliance

Studies note
the
rapid
growth of einvoicing
andregulations
in ASEAN
Identifies
technology,
operational,
andcompliancebarriers to einvoicing
adoption
Discusses
the potential
benefits of einvoicing
and forecasts
its adoption
trajectory
Focuses on
the role ofregulations
and
legal
frameworks
infacilitatingor hindering
e-invoicing

Predictive
Model
Relevance
Highlights thesignificance ofCintegration
and Aadoption
inthepredictivemodel.
Reinforces theimportance of
RreadinessandCcompliance

Reference

Suggests
Uusage as a
criticaloutcomemeasure in themodel.

[7], [8],[14], [17]Underscores
Ccomplianceas a vital
predictor
inthe model.

[3], [11],[13], [19][5], [6][1], [9], [5],[16], [18],[20]

Predictive
Importance(Scale: 1-5)
4

Correlation
Coefficient

Description

0.85

Technical
integration
capability [10]

3

0.65

User acceptance
rate [9]40.75Business andinfrastructure
preparedness
[16]

5

0.90

Regulatory
adherence [13]

4

0.80

Utilization postimplementation[1]

In this table:• Predictive Importance is quantified on a scaleof 1-5, where 1 is the lowest and 5 is the highest
level of importance. This allows a moreobjective comparison of factors.
• Correlation Coefficient remains a numericvalue indicating the strength and direction of therelationship between each variable and thesuccess of e-invoicing implementation.
• Description is a concise statement of what eachvariable represents.
• References are provided for each variable,
offering direct access to the underlying studies
or sources.
The predictive model is a robust, theoretical constructbased on identified variables derived from qualitative
insights and empirical evidence from similar initiatives.
Each variable is assigned a predictive importance and acorrelation coefficient, indicating how strongly it's
expected to influence the success of the e-invoicing
rollout. This model not only aids in forecasting outcomesbut also assists in identifying areas requiring strategic
focus and resource allocation, making it a grounded andpractical tool for stakeholders. A conceptual model was
defined to guide future quantitative studies. The variables
identified for the predictive model are enumerated inTable 5: Predictive Model Variables for E-Invoicing
Success. Informed by Hernandez-Ortega’s work [10] andother seminal studies, this model was conceptualized to
guide stakeholders in anticipating and preparing for the

4.3. Predictive Analysis for E-Invoicing Success
The critical component of the AER Framework is thetheoretical predictive model, which was designed toforecast and evaluate the success of the e-invoicing
rollout. Drawing insights from the comprehensive
literature review, this model incorporates various
variables identified as significant predictors of einvoicing success. Each variable, substantiated by theliterature, is accompanied by a correlation coefficientindicating its impact and relevance in forecasting the
successful implementation of e-invoicing, asconsolidated in Table 2 below.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1063

Koh Chee Hong, Abdul Samad Bin Shibghatullahpotential outcomes of the e-invoicing rollout. The
formula for the theoretical predictive model is as follows:other countries embarking on a similar journey. As thelandscape of digital finance continues to evolve, the AER
Framework is poised to remain a relevant, effective toolfor navigating the future of e-invoicing.

Psuccess=f(Cintegration,Aadoption,Rreadiness,Ccompliance,Uusage)

Where:•
Psuccess is the probability of successful einvoicing rollout.
•
Cintegration represents system and data
integration capability.
•
Aadoption indicates the rate of adoption•
Rreadinessreflects
organizationalandtechnological readiness.
•
Ccompliance is the degree of compliance withlegal and tax requirements.
•
Uusage measures actual usage and utilization
rates post-implementation.

5. Conclusion

This model, as illustrated in Fig. 1, is significantly
grounded in empirical evidence and industry insights,
making it a novel and practical contribution to the fieldof e-invoicing and digital economy strategies.

References

Malaysia's transition to mandatory e-Invoicing reflects a
crucial shift in its digital economy. This study developed
an adaptive framework, utilizing a qualitative
methodology to integrate insights from various sources,
culminating in a theoretical predictive model. This model
offers stakeholders a robust tool for strategic e-Invoicing
implementation. The AER emerges as a practical guide,significant for Malaysia's digital transformation andrelevant for similar global contexts, ensuring continued
efficacy in evolving digital finance landscapes.

1.

Suendra, A. A. P. (2023). Analysis of the Implementationof E-Invoice 3.0 in Increasing Taxpayer Compliance at
Kpp Pratama Bandar Lampung Dua. Data: Journal of
Information Systems and Management, 1(1), 50-62.
2. Irawati, I., Darmawan, H., Sofyan, M., & Serebryakova, T.
(2022). The effect of the implementation of E-Invoice 3.0
and taxable person compliance to value added tax revenue
at KPP Pratama Pasar Rebo. Ilomata International Journalof Tax and Accounting, 3(1), 1-12.
3. Balakrishnan, V., & Shuib, N. L. M. (2021). Drivers andinhibitors for digital payment adoption using the CashlessSociety Readiness-Adoption model in Malaysia.
Technology in Society, 65, 101554.
4. Ooi, K. B., Lee, V. H., Tan, G. W. H., Hew, T. S., & Hew,
J. J. (2018). Cloud computing in manufacturing: The next
industrial revolution in Malaysia?. Expert Systems with
Applications, 93, 376-394.
5. Chan, M. (2024). Malaysia: Digital Payments, Data
Regulations, and AI as Most Promising Areas for Digital
Economy Collaboration. In The ASEAN Digital Economy(pp. 76-96). Routledge.
6. Cheung, P., & Taojun, X. (Eds.). (2023). The ASEAN
Digital Economy: Towards an Integrated Regional
Framework. Taylor & Francis.
7. Olaleye, S. A., Sanusi, I. T., Dada, O. A., & Agbo, F. J.
(2023). A bibliometric review of global visibility, impact
and adoption of electronic invoicing: The past and thefuture. Heliyon.
8. Harris, H., Guru, B. K., & Avvari, M. V. (2011). Evidenceof firms' perceptions toward Electronic Payment Systems(EPS) in Malaysia. International journal of business andinformation, 6(2), 226.
9. Nguyen, A. H., Nguyen, T. P., & Dang, G. T. T. (2020).
Determinants Of E-Invoice Adoption: Empirical evidence
from Vietnam. The Journal of Asian Finance, Economics
and Business (JAFEB), 7(7), 311-321.
10. Hernandez-Ortega, B. (2012). Key factors for the adoption
and subsequent use of e-invoicing. Academia. Revista
Latinoamericana de Administración, (50), 15-30.
11. Kamaruzaman, K. N., Handrich, Y. M., & Sullivan, F.
(2010). E-commerce adoption in Malaysia: Trends, issues
and opportunities. ICT strategic review, 11.

Fig.1 AER Framework - Theoretical Predictive Model

4.4. Conclusion and Strategic Implications
The AER Framework emerges as a strategic,
comprehensive guide for e-invoicing adoption in
Malaysia. It is not just a theoretical construct but apractical, dynamic tool reflecting the complexities andrequirements of real-world implementation. The
framework's adaptability, combined with its predictive
capability, positions it as a model for digitaltransformation, not only for Malaysia but potentially for©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1064

Powered by TCPDF (www.tcpdf.org)

Optimizing E-Invoicing Rollout: Adaptive

12. Lian, J. W. (2015). Critical factors for cloud based einvoice service adoption in Taiwan: An empirical study.
International Journal of Information Management, 35(1),98-109.
13. Rokhman, A., Handoko, W., Tobirin, T., Antono, A.,
Kurniasih, D., & Sulaiman, A. (2023). The effects of egovernment, e-billing and e-filing on taxpayer compliance:
A case of taxpayers in Indonesia. International Journal of
Data and Network Science, 7(1), 49-56.
14. Wagiman, A. N., Aspasya, G. S., & Prawati, L. D. (2023).
Net Benefit on E-Invoice Implementation: Applying theDelone & McLean Information Systems Success Model.
In E3S Web of Conferences (Vol. 388). EDP Sciences.
15. Hannah, B., & Nani, G. (2021). Improving procurement
performance in the public sector with the implementationof E-procurement: a study of selected Metropolitan andMunicipal Assemblies in the Ashanti region of Ghana.
16. Liew, S. Y. (2022). The determinants of Malaysia’s
revenue: the role of e-filing, tax administration and tax
compliance (Doctoral dissertation, UTAR).
17. Groznik, A., & Manfreda, A. (2015, October). E-Invoicing
and e-government–impact on business processes. In
DIEM: Dubrovnik International Economic Meeting (Vol.
2, No. 1, pp. 204-217). Sveučilište u Dubrovniku.
18. Alaweti, M. F., Azizan, N., & Faryadi, Q. (2013). The
Impact of Experience and Perception on the Efficiency of
E-procurement in Malaysia. Journal of Communication
and Computer, 10, 1475-1483.
19. Panayiotou, N. A., & Stavrou, V. P. (2021). Government
to business e-services–A systematic literature review.
Government Information Quarterly, 38(2), 101576.
20. Waithaka, R. K., & Kimani, J. G. (2021). Effect of eprocurement practices on supply chain performance.
Global Journal of Purchasing and Procurement
Management, 1(1), 32-42.

Authors Introduction
Mr. Koh Chee Hong
A seasoned financial technologist
specializing in SAP FICO, SAP
BPC/BW, and Oracle NetSuite, he iscurrently pursuing a PhD in Computer
Science at UCSI University. He holds
a First Class Honors B.Tech in
Business Information System from
Universiti Teknologi PETRONAS.
Dr Abdul Samad Bin Shibghatullah
He is a Senior Lecturer in UCSI withacademic qualifications including a
PhD and MSc in Computer Science,along with a Bachelor's degree in
Accounting. His areas of focus include
Modelling, Simulation, Optimization,
and Artificial Intelligence.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1065

App Alert System for Smart Phones
Chee Kin Hoe

Institute of Computer Science & Digital Innovation, UCSI University,
UCSI Heights, 1 Jalan Puncak Menara Gading, Kuala Lumpur, WP Kuala Lumpur 56000, Malaysia
Kasthuri Subaramaniam

Institute of Computer Science & Digital Innovation, UCSI University,
UCSI Heights, 1 Jalan Puncak Menara Gading, Kuala Lumpur, WP Kuala Lumpur 56000, Malaysia
Abdul Samad bin Shibghatullah

Institute of Computer Science & Digital Innovation, UCSI University,
UCSI Heights, 1 Jalan Puncak Menara Gading, Kuala Lumpur, WP Kuala Lumpur 56000, Malaysia
E-mail: 1002163440@ucsiuniersity.edu.my, kasthurisuba@ucsiuniversity.edu.my,
abdulsamad@ucsiuniversity.edu.my

Abstract
In this project the researcher chosen area is the development of an App for forgetful users to take their phones
from their cars it functions as a medium to take their phones from their cars. the target users will be senior adults
who may be careless or may have memory deteriorating diseases such as dementia or Alzheimer. The researcher
used surveys to gather information and used various online resources. The research methodology used by the
researcher is modified waterfall model for its simplicity and ability to reverse to different phases. The researchmethodology used by the researcher is modified waterfall model for its simplicity and ability to reverse todifferent phases. Overall, the project can be deemed as success as the researcher has successfully developed an
App for users to remind them to take their phone before exiting their personal vehicle.
Keywords: Development of an App, Senior Adults, Surveys, Modified Waterfall Model

1. Introduction
Since the introduction of the first smartphone, madeby IBM, it was called the IBM Simon it had a calendar,
address book, clock, notepad, keyboard, touchscreenand email capability. Since then, humans have beenusing this device to benefit themselves in daily tasks
such as navigating, working and using it as a form ofentertainment [3]. But with good the bad always come.
Below are the problem that users are facing:• Elderly users who have dementia may forget totake their smartphones.
• The underground basement parking is too dark
and did not notice their smartphones in the holder.
• Buying a new smartphone is affordable but
repairing or buying a new car is a liability.
With the problems identified, we propose to build a
notification App for forgetful users when the car
engine has been shut off. The app will notify the userwhen the ignition has been turned off. The App will

be user friendly and simple. he objectives of theresearch include:• To identify why users always forget to take theirphones from their cars.
• To design a notifying system• To develop a prototype Notifying App• To evaluate the system

2. Methodology
2.1 Impact of Smartphones in Malaysia
According to Malaysian Communications andMultimedia Commission (MCMC) the use ofsmartphones and their apps has dramatically increased
since the introduction of affordable 5G smartphones in
2021. According to the survey conducted by MCMC

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

1066

Chee Kin Hoe, Kasthuri Subaramaniam, Abdul Samad Bin Shibghatullah

the number of smartphone users in 2021 is 94.8%
compared to a mere 78.0% in 2018 [1], [2]. The higher
usage of smartphones indicates people are much moredigitally connected. As shown in (Fig.1) below.

smartphones for voice calls and 80.2% benefit from it
once a day.

2.3 Methodology that had been used in app
development
2.2 What kind of Apps are used in Malaysia
For the researcher’s case it is developing an App forforgetful users to take their smartphones from theircars. As for this the researcher have discussed withhis/her supervisor the choices the researcher has fordeveloping the App. The researcher may have somefactors to consider such as suitability of themethodology, the confidence of the researcher usingthat methodology, and factoring the time constraint [5][6]. After considering all the factors, the researcher has
decided to go with the modified waterfall model as cansee in (Fig.2).
According to the survey created by the researcher
titled “FYP Survey: Forgetfulness, Phones & Cars”
has received a total of 40 respondents. This is all
possible without the use of various distribution
channels such as Facebook, Word of Mouth,
Instagram and family members.

According to the HPUS 2021, MCMC has found outthat communication remained the top activity amongsmartphone users [1]. The survey has found out that
82.9% of the respondents use text messaging. On
which it found that 97.3% of users performed at least
once a day. Not only that, 78.9% of users
communicated via social media and statistics showed
94.4% users performing it at least once in a day [4].
While in the realm of voice communication, thesurvey has found that 78.6% of respondents use

3.1 Survey Results
In the first question the researcher has asked what
age is the respondent. According to the survey, 18
respondents are in between 31-60 years old while 15
are young adults in between 18-30 years of age andanother 7 which are seniors in the golden years.
According to the survey conducted by the researcher.
The number of respondents that currently possesseddiseases that deteriorates memory are as follow.
Respondent that has dementia are 10 respondents
while 6 have Alzheimer disease.

Fig.1. Percentage distribution of Smartphone andfeature phone users, 2017 to 2021.

Fig.2. Modified Waterfall model

3.2 Use Case Diagram
Shown below in (Fig.3) is the Use Case Diagram
which represents the methodology used during thedevelopment of the App. The Use Case Diagram isused to organize and identify the requirements
required for the Forgetful Alarm App.

3. Results and Discussion©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

1067

App Alert System For

Fig.4. Usefulness of the App.

Fig.3.Use Case Diagram

4. Conclusion
In conclusion all the objectives of the App
development had been met and users who used the app
had shown interest and can see the potential of this app.
Based on the results of this researcher it can beconcluded that this app can benefit users in helping
them to remember to take their phones from their car.
Some recommendations had been elevated from theusers during their time using the app. One of therecommendations received during the evaluation is tohave reminders to take medication or take an insulinjab. To improve the overall usability of the app theresearcher has found some ways to improve the app.

3.3 Implementation
In this section, the implementation of the App is
discussed. To develop the App the researcher usedJava as the backend programming language and use
Extensible Markup Language (XML) to develop thefrontend of the App. Android Studio was also used indeveloping the App [7].

3.4 Evaluation
After successfully developed the App. The researcher
will conduct testing to determine whether the Appsatisfy the requirements that had been identified [8],[9]. To ensure the apps functionality, the researcher has
conducted various ways of testing to test the App. The
various ways are Unit Testing and User Acceptance
Testing.

References
1.

2.

3.5 User Acceptance Testing3.

Before the user acceptance test, the researcher would
ask the participants which would be selected from theprevious survey to download the App and use the app
in their daily routine. Participants are asked tocommute to work as usual but with a twist of putting
their phones in their phone holders and charging it
using the cars cigar lighter that can be function as acharging port. In (Fig.4) majority of the participants
has agreed that the app is useful.

4.

5.

6.

"Hand Phone Users Survey 2021 (HPUS 2021),"
MalaysianCommunications
And
Multimedia
Commission, Cyberjaya, 2021.
Z. Lazzari, "Technology & How We Communicate," 27August2018.
[Online].
Available:
https://smallbusiness.chron.com/role-cell-phonecommunication-today-31479.html. [Accessed 25 May
2023].
B. Jung, "What Are the Benefits of Smartphone
Technology," 12 February 2019. [Online]. Available:
https://smallbusiness.chron.com/role-cell-phonecommunication-today-31479.html. [Accessed 25 May
2023].
Indeed Editorial Team, "What Is an App? Types ofApps and Examples," 11 March 2023. [Online].
Available:
https://www.indeed.com/careeradvice/career-development/what-is-an-app. [Accessed
25 May 2023].
"Mayo Clinic," 12 October 2022. [Online]. Available:
https://www.mayoclinic.org/diseasesconditions/dementia/symptoms-causes/syc-20352013.
[Accessed 22 May 2023].
"Dementia cases set to ruse 312 per cent by 2050: Is
Malaysia prepared?," 8 June 2022. [Online]. Available:©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

1068

Powered by TCPDF (www.tcpdf.org)

Chee Kin Hoe, Kasthuri Subaramaniam, Abdul Samad Bin Shibghatullah

7.

8.
9.

https://adfm.org.my/dementia-cases-set-to-rise-312per-cent-by-2050-is-malaysia-prepared/. [Accessed 25
May 2023].
"Mayo Clinic," 02 February 2023. [Online]. Available:
https://www.mayoclinic.org/diseasesconditions/alzheimers-disease/symptoms-causes/syc20350447. [Accessed 25 May 2023].
R. A. Armstrong, "Folia Neuropathological," Whatcauses Alzheimer's disease, p. 20, 2013.
E. Ng, "A brief history of cars in Malaysia," 9 August2022. [Online]. Available: https://mytukar.com/blog/abrief-history-of-cars-in-malaysia/. [Accessed 27 May
2023].

Authors Introduction
Chee Kin Hoe
He is currently an undergraduate
student, pursuing his study in Diploma
in Information Technology from
Institute of Computer Science andDigital Innovation (ICSDI) at UCSI
University, Malaysia.

Dr. Kasthuri Subaramaniam
She is currently an assistant professor at
Institute of Computer Science andDigital Innovation (ICSDI), UCSI
University, Kuala Lumpur, Malaysia.
She earned both her bachelor’s degree
in computer science and a master’s
degree in computer science from the
University of Malaya. Her research interests include
human-computer interaction, etc
Dr. Abdul Samad bin Shibghatullah
He received the bachelor accounting
degree from Universiti Kebangsaan
Malaysia, Bangi, Malaysia in 1999, theM.Sc. degree in computer science fromthe Universiti Teknologi Malaysia,
Skudai, Malaysia in 2002, and the
Ph.D. degree from the Brunel
University of Uxbridge, United Kingdom. He iscurrently Associate Professor at UCSI University,
Kuala Lumpur, Malaysia. His current research interests
include optimization, etc.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024

1069

Developing a Prototype Hand Gesture Recognition System in Interpreting American Sign
Language
Kong Seh Chong

Institute of Computer Science and Digital Innovation, UCSI University, Malaysia
Kasthuri Subaramaniam

Institute of Computer Science and Digital Innovation, UCSI University, Malaysia
Ismail Ahmed Al-Qasem Al-Hadi

Institute of Computer Science and Digital Innovation, UCSI University, Malaysia
Email: 1002058020@ucsiuniversity.edu.my, kasthurisuba@ucsiuniversity.edu.my,ismailAhmed@ucsiuniversity.edu.my

Abstract

Hand gestures of sign language is a form of non-verbal communication which have been used by most people in their
daily life. Sign language is not only used by people with speaking issues but it is also unconsciously used by normal
people during their daily interaction with others. This is because it is a way to express their current feelings or themeaning they wanted to convey to others. On other hand, sign language is an important alternative used by people withhearing impairment or speaking obstacles so they can communicate with others. However, not everyone from all walksof life has learned sign language so there will be problems of interaction between them and people with speaking issues.
Thus, this research focuses on developing a hand gesture recognition system for accurately interpreting American Sign
Language (ASL) so that it can deliver a message that can be understood by others and enable efficient communication. In
our system, it will utilize computer vision techniques to analyze hand postures and movements which will include handsign recognition, finger tracking, and motion estimation. With the pre-developed libraries like OpenCV and MediaPipe
are employed in the system so it can recognize and classify ASL gestures based on extracted features. Extensive datasetsof ASL hand gestures are collected and annotated to enhance the system's accuracy and robustness. The developed
system aims to improve human-computer interaction, enabling seamless communication between deaf individuals andtechnology. The potential applications include real-time interpretation of ASL gestures for enhanced accessibility andinclusivity.
Keywords: Human-Computer Interaction, American Sign Language, Hand Gesture Recognition System.

language, where each hand sign will represent different
alphabets from A to Z which then can form a specificword. As a result, this project purpose is aboutdeveloping a Hand Gesture Recognition System that
can help in reading sign language. With this system, it
can help people with hearing impairment or speaking
disabilities to communicate with others more efficiently
as well as allow non-signers to get know more about
what actually a sign language is [7], [8].

1. Introduction
In this age of technology, there are many IT related
tools or systems that provide various benefits to all
walks of life which include Hand Gesture Recognition
System (HGRs). The Hand Gesture Recognition
System is a type of technology that uses computer
vision and machine learning to interpret the motion andsign of individuals [1]. Therefore, this technology cancontribute a significant transformation to society
nowadays by providing a particular group of peoplehuge benefits who are born with hearing impairment or
muted. This is because people with hearing impairment
or mute usually rely on sign language, and it is an
important tool of communication for them. There is a
specific meaning from each gesture in sign language
which can construct a complex meaning with the
combination of various gestures. In American sign

Hand gestures are something that are typically used indaily communication including communicating withpeople having hearing problems. According to theresearch, hand gestures can be defined as a type of bodylanguage that can be expressed through the center of thepalm, finger position and the shape constructed by thehand [9]. There are two classifications of hand gestures
which will be the static and dynamic. The static©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1070

Kong Seh Chong, Kasthuri Subaramaniam, Ismail Ahmed Al-Qasem Al-Hadi

gestures mostly have to do with the shape of the handwhereas the dynamic gestures deal with a variety ofhand movements [9]. For instance, we can find static
gestures from the American Sign Language (ASL) at
the figure (Fig. 1) in which most of its hand gestures
are in a particular shape or pattern that represent analphabet excluding j and z. Dynamic gestures aremostly related to body language or hand gestures that
require movement such as waving, shaking other hands,
or even showing the J and Z letter in ASL [11]. It is
hard to track the dynamic gestures so in our project wewill be focusing more on the ASL static gestures andthink of future improvement for the dynamic hand
gestures recognition system.

2

3

Indian Sign
Language
Recognition
Using Random
ForestClassifier [2]2021

A Vision-based
System for
Recognition ofWords used inIndian Sign
Language
Using
MediaPipe [6]2022

1.Random
ForestClassifier••

1.Random
ForestClassifier
2.MediaPipe

•

•

4

Communicatingwith the Deafand Hard of
Hearing
through
Automatic
Arabic Sign
LanguageTranslator [5]2021

1.Feature
Extraction
2.Random
ForestClassifier

•

5

Smart
Communication
System Using
Sign Language
Interpretation
[3]

2022

1.MediaPipe
2.Light
WeightRandomForestClassifier•Fig. 1 American Sign Language [10].

•High accuracy
even oncomplexdatasets.
Robust to
missing data asusers may notalways perform
gesturesperfectly.
Able to learncomplexrelationships
between thefeatures of thesign language
gestures.
Achieving highaccuracy in
97.4%.
Achieving best
recognition
accuracy out of
all otherclassification
algorithms.

Achieving good
accuracy onrecognition
about 94.69%
Ability inadapting todifferent sign
language
datasets byusing different
featureextraction
techniques andhyperparametertuning

2. Methodology
2.1 Comparison Of Different Methodology in Hand
Gestures Recognition System
A comparison of methodology is shown in the tablebelow (Table 1).
Table 1 Comparison Table of methodology
in Hand Gesture Recognition System
Sr.
No

Paper Name

Year
of
PublicTechniques

Advantages

1

Hand Gesture
Recognition via
Lightweight
VGG16 andEnsembleClassifier [4]

2022

1.CNN
2.Random
ForestClassifier••Fig. 2 Proposed concept of Hand Gesture Recognition
System
2.2 Analysis
Random forest
is great inhandling large
input variables.

We will be revealing the outcomes and analyzing theinformation about the Hand Gesture Recognition
System in interpreting ASL from the survey organizedthat was distributed on the network through emailingand messaging (Fig. 2). Based on the outcomes of thesurvey, we will know about whatever this project can
provide a system in interpreting ASL for

Having thehighestclassification
rate.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1071

Developing a Prototype Hand

communication between normal people and deaf orpeople with hearing problems.

Table 2. Results of Unit Testing
Procedure
Description
Click Start
Camera Button
Click Stop
Camera Button

2.3 Methodology
In this report, we have employed a quantitative
method to gather information regarding the Hand
Gesture Recognition System in interpreting ASL. The
chosen approach for data collection was a survey form
created using Google Forms. The survey form consistedof 2 parts, the first part is about the demographics of the
respondents and the second part is about the system
inquiries. There are a total of 16 questions that focus onwhich the 12 questions from the total are mostly asking
about the Hand Gesture Recognition System and ASL.
The questions included in the survey form were
carefully selected to extract essential information
required for our research. We aimed to understand more
about the Hand Gesture Recognition system so that we
can successfully develop the system in interpreting
ASL.

Recognizing the
ASL handgestureClick Exit
ButtonClick Save
Character
ButtonClick Clear
Texts Button

Enable to savemessage ashistory
Click History
Message Button

The survey form consisted of multiple-choicequestions that allowed respondents to select multiple
answers from a given list of options. This format
enabled us to capture a range of basic understanding of
the respondents in the Hand Gesture Recognition
System. In addition to multiple-choice questions, we
also included rating questions in the survey form. These
questions employed a scaling approach to measure
respondents' agreement or disagreement with specificstatements related to the Hand Gesture Recognition
System. This format allowed us to assess the level ofagreement towards certain measures or opinions aboutthe Hand Gesture Recognition System in interpreting
ASL.

Click ASL
Display Button

Click Testing
Page Button
On History
Message page,
double click onmessage
On History
Message page,
click clear
buttonOn History
Message page,
click clear
history button
Click Text to
SpeechInside messagebox of History
Message page

By utilizing the survey form with these question
formats and employing various distribution methods,
we intended to gather comprehensive insights into theHand Gesture Recognition System. For reference, theappendix of this report contains 15 survey questions
pertaining to the Hand Gesture Recognition System ininterpreting ASL.

On testing page,
click startcamera button

3. Results and Discussion

On testing page,
click stop
camera button
On testing page,
information box
displayIn this section, it will be discovering all the validationand testing that are done after the development of theHand Gesture Recognition system. This chapter is avery significant part that needs to be taken in the entire
process of the project development as it will be thesources to know what are the weaknesses andlimitations of the current system and the improvementthat it needs to enhance in the future. The results areshown in Table 2 and Table 3.

Expected Result

ActualResultWorked as
expected
Worked as
expected

Test
ResultPass

Worked as
expected

Pass

Worked as
expected
Worked as
expected

Pass

Worked as
expected

Pass

Worked as
expected

Pass

Worked as
expected

Pass

Worked as
expected

Pass

Worked as
expected

Pass

Worked as
expected

Pass

Able to clear the fullshown messages on the
below the label of “Full
Message:”
All the history on thepage will be able toclear

Worked as
expected

Pass

Worked as
expected

Pass

Able to hear the voiceof messages to speech
Information of saved
message will berecorded such as Date,
Time and Message
contents
It will activate thecamera frame for hand
gesture recognition
process
It will stop the operationof camera frame

Worked as
expected
Worked as
expected

Pass

Worked as
expected

Pass

Worked as
expected

Pass

After camera operated,
it will show different
information such asrecognized alphabets,
hands, interpretation
and FPS

Worked as
expected

Pass

The camera is started
after button was clicked
The camera frame isstopped after the button
was clicked
All the static ASL hand
gesture is recognizedsuccessfully on bothright and left hand
Close the entire
programAble to save therecognized ASL hand
gesture to text file
Able to clear the
textboxwiththerecognized ASL hand
gestureClick the Save
Messages button to
store it as history at
history message page
Open up a new framethat redirect to theHistory Message page
Allow users to view on
the ASL gesturesinformation on new
frameOpen up a new framethat redirect to thetesting page
Allow long message to
be view in full on thelabel of “Full Message:”©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1072

Pass

Pass

Pass

Kong Seh Chong, Kasthuri Subaramaniam, Ismail Ahmed Al-Qasem Al-Hadi

Table 3. Results of System Testing
Procedure
Description
Recognizing
ASL
hand
gestures throughwebcamSaving the ASL
hand gestures

ASL information
display

Saving the text in
Message History
Testing featurefor users

Converting Text
to Speech

Stop the system
operation

Expected Result
To ensure the users wereable to recognize theASL
handgesturethrough the start camera.
To ensure that the users
are able to save therecognizedASL
alphabets to the textbox.
To ensure that the users
are able to view types of
ASL hand gestures at thepage.
To ensure that the users
are able to save and view
back the history of text.
To ensure that the users
are able to familiarize
themselves with the ASL
hand gesture systembefore directly using it.
To ensure the messages
from the deaf or people
with hearing problem
can be heard by theothersTo ensure that the users
are able to quit thesystem by clicking theexit button.

ActualResultWorkedas
expected

Test
ResultPass

Workedas
expected

Pass

Workedas
expected

Pass

Workedas
expected
Workedas
expected

Pass

Workedas
expected

Pass

Workedas
expected

Passconquer these limitations stated in this project.
Nevertheless, our hand gesture recognition system has
already attained the objectives and solve some of the
problems stated in the project.
References
1.

2.

Pass

3.

4.

5.

4. Conclusion

6.

In conclusion, the project is researched and analyzed
comprehensively in order to ensure that the objectives
and aims that have been decided at the proposal phase
are accomplished. After the development and analysisof the hand gesture recognition system, it can be statedand concluded that all objectives that were planned in
the proposal phase have all been accomplished. Since
the hand gesture recognition system is specially createdfor people with hearing problems to have an interaction
with others through the gesturing and recording
characters of ASL which form a text to bring themeaning they are expressing. At the same time, it may
also serve as an educational tool to help people who
might be interested in American Sign Language andwant to learn it by using the system. This is how the
research project's key goals will be accomplished.
However, there are no perfect system exist in the world
which include our hand gesture recognition system. For
instance, one of the limitations of our hand gesturerecognition system are unable to recognize in dark
environment or classify dynamic gestures. Therefore,
there will be future works to be accomplish in order to

7.

8.

9.

10.

11.

A. Kumar C, C. Venkatesh, L. Vaishnavi D A, and H. S,
Computer vision based Hand gesture recognition
system, vol. 20, no. 7, pp. 2859–2866, Jul. 2022.
doi:doi: 10.14704/nq.2022.20.7.NQ33365
A. S, A. Potluri, S. M. George, G. R, and A. S, “Indiansign language recognition using random forestclassifier,” 2021 IEEE International Conference on
Electronics,
ComputingandCommunication
Technologies(CONECCT),
2021.
doi:10.1109/conecct52877.2021.9622672
D. Bisht, M. Kojage, M. Shukla, Y. P. Patil, and P.
Bagade, “Smart Communication System Using Sign
Language Interpretation,” 2022 31st Conference ofOpen Innovations Association (FRUCT), May 2022.
doi:10.23919/fruct54823.2022.9770914
E. L. Ewe, C. P. Lee, L. C. Kwek, and K. M. Lim,
“Hand gesture recognition via lightweight VGG16 andensemble classifier,” Applied Sciences, vol. 12, no. 15,
p. 7643, 2022. doi:10.3390/app12157643
G. Latif, J. Alghazo, N. Mohammad, and R. Alghazo,“Communicating with the deaf and hard of hearing
through Automatic Arabic Sign Language translator,”
Journal of Physics: Conference Series, vol. 1962, no. 1,
p.
012055,Mar.
2021.
doi:10.1088/17426596/1962/1/012055
S. Adhikary, A. K. Talukdar, and K. Kumar Sarma, “A
Vision-based system for recognition of words used inindian sign language using MediaPipe,” 2021 Sixth
International Conference on Image Information
Processing(ICIIP),Feb.
2021.
doi:10.1109/iciip53038.2021.9702551
Thwe, P.M., (2019) ‘Hand Gesture Detection and
Recognition System: A Critical Review’. International
Journal of Computer (IJC), vol. 32, no. 1, pp. 64-72.
Li, G. et al. (2017) ‘Hand gesture recognition based on
convolution neural network’, Cluster Computing,22(S2), pp. 2719–2729. doi:10.1007/s10586-017-1435x.
M. Oudah, A. Al-Naji, and J. Chahl, “Hand gesturerecognition based on Computer Vision: A review oftechniques,” Journal of Imaging, vol. 6, no. 8, p. 73,
2020. doi:10.3390/jimaging6080073
M. Yasen and S. Jusoh, “A systematic review on hand
gesture recognition techniques, challenges andapplications,” PeerJ Computer Science, vol. 5, 2019.
doi:10.7717/peerj-cs.218
N. Bargellesi, M. Carletti, A. Cenedese, G. A. Susto,
and M. Terzi, “A random forest-based approach forhand gesture recognition with wireless wearable motion
capture sensors,” IFAC-PapersOnLine, vol. 52, no. 11,
pp. 128–133, 2019. doi:10.1016/j.ifacol.2019.09.129©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1073

Powered by TCPDF (www.tcpdf.org)

Developing a Prototype Hand

Authors Introduction
Mr. Kong Seh Chong
He is a student majoring in Bachelor ofComputer Science (Honours) at Instituteof Computer Science and Digital
Innovation (ICSDI), UCSI University,
Kuala Lumpur, Malaysia.

Dr. Kasthuri Subaramaniam
She is currently an assistant professor at
Institute of Computer Science and Digital
Innovation (ICSDI), UCSI University,
Kuala Lumpur, Malaysia. She earned bothher bachelor’s degree in computer scienceand a master’s degree in computer sciencefrom the University of Malaya. Her
research interests include human-computer interaction, etc.
Dr. Ismail Ahmed Al-Qasem Al-Hadi
He is currently an assistant professor at
Institute of Computer Science and Digital
Innovation (ICSDI), UCSI University,
Kuala Lumpur, Malaysia. He has a PhD inIntelligent System. His research interests
include AI and Data Mining.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1074

Minature Resource Planner
Adim Khalid AldireejahInstitute of Computer Science and Digital Innovation, UCSI University, Kuala Lumpur, Malaysia
Kasthuri Subaramaniam

Institute of Computer Science and Digital Innovation, UCSI University, Kuala Lumpur, Malaysia
Ghassan Saleh Hussein Al-Dharhani

Institute of Computer Science and Digital Innovation, UCSI University, Kuala Lumpur, Malaysia
Email: 1002061045@ucsiuniversity.edu.my, kasthurisuba@ucsiuniversity.edu.my,ghassan@ucsiuniversity.edu.my

Abstract
Due to the complexity and high cost of typical Enterprise Resource Planning (ERP) systems, medium-sizedenterprises (SMEs) often face challenges in their implementation. To address this, the article develops a tailored,
cost-effective ERP system specifically designed for SMEs, considering their unique needs and limitations. The
research analyzes existing ERP systems in large organizations and conducts surveys and interviews to identify
common issues faced by SMEs when adopting standard ERP systems. The paper's main goal is to create anaffordable and user-friendly ERP system that empowers SMEs to overcome financial obstacles in an increasingly
digital world. The system addresses key business processes such as financial accounting, employer management,
and procurement. Additionally, the report highlights the interest homeowners have in using ERP systems forhousehold tasks, indicating a substantial market and demand. Emphasizing the importance of a flexible Home
Resource Planning (HRP) system, the study aims to design a cost-effective ERP solution suitable for both homes
and SMEs. The proposed ERP system's impact on SME operating performance, productivity, and decision-making
is thoroughly examined. The conclusion underscores the significance of customized ERP systems for homes and
SMEs in the current digitized environment. The developed small-scale ERP system proves successful in helping
SMEs and homes overcome financial challenges and reach their full potential.
Keywords: SMEs, Resource Planner, ERP

ERP has been expanding quickly, with more customers
and users on a daily basis.

1. Introduction
Organizations nowadays are trying to attain optimal
efficiency, get an edge over their competitors, andaccomplish their objectives as quickly as possible. ERP,or enterprise resource planning, aids in achieving
previous corporate goals. As a definition, ERP is apiece of software that contains all the components
necessary for a corporation to be managed and operated
with maximum effectiveness, avoiding the mostpossible issues. Enterprise resource planning works byintegrating technology from many industries—
including human resources, finance, manufacturing,
supply chain, and more—in one environment. In otherwords, it automates numerous business processes withthe least amount of effort while remaining in the samesetting. Most large businesses globally have alreadyembraced ERP, and more and more small and mediumsized businesses are trying to do the same to compete;
they find it both cost-effective and necessary to keep
pace [1]. Although the scope and close integration of
ERP has only just become available, the market for

Any firm may gain a lot from using an ERP system.
For instance, it can improve and automate the majorityof daily tasks an organization must do by removing theneed for human data entry, saving time and money [2].
Second, ERP keeps everything organized and in asingle system, which removes the need for numerous
technologies. Third, ERP can quickly analyse andidentify the saved data to provide managers with
operational insights. Overall, enterprise resource
planning provides seamless data flow between any
organization’s
sectorswhilemaintaining
customizability so that it can follow the business needs
and strategies.
It has become a must for the future of any
establishment that is looking to improve and remain on
the popular side of the market [3]. Throughout this
proposal, a deeper analysis of the significant
characteristics, challenges of implementation, anddifficulties related to ERP systems will be done toultimately gain a thorough understanding of their©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1075

Adim Khalid Aldireejah, Kasthuri Subaramaniam, Ghassan Saleh Hussein Al-Dharhani

function in promoting organizational success. alsomentioning some issues and their solutions, using myapproach to resolve them, and expanding theaccessibility of ERPs to everyone, without exception.

uniqueness lies in its modular approach, allowing
organizations to integrate only the required modules.
With an approachable UI, community-driven ecosystem,
and cloud-based options, Odoo is an affordable andscalable solution for businesses of all sizes and sectors.
Known for fitting SMEs, Odoo is simple and free to test
for new users.

2. Methodology
2.1 Brief History

2.3 ERP Benefits

ERP history traces back to the 1960s with material
requirements planning (MRP) systems, pioneered by J.I.
Case in collaboration with IBM. The increasing
complexity of workplaces led to the need for crossfunctional data flow in areas like decision-making,product ordering, inventory management, accounting,human resources, and distribution. This complexity
gave rise to the first ERP system, aimed atconsolidating diverse functions into a unified platform.
ERP systems emerged commercially in the late 1980s
and early 1990s, initially designed for the demands oflarge, complex organizations. Recognized for theircomplexity and high costs, ERP systems have evolved
over time [4].

Streamlined Operations: Streamlined operations areachieved through the reduction of delivery and cycle
times, strategically minimizing delays in data retrieval
and reporting. By optimizing these processes, theorganization enhances overall efficiency andresponsiveness, contributing to a more agile andcompetitive operational framework.
Cost Reduction: The realization of cost reduction
stems from a comprehensive, enterprise-wide review oforganizational decisions. This holistic approach not
only saves valuable time but also enhances control overfinancial aspects. Through the identification andelimination of inefficiencies, the organization not only
achieves monetary savings but also establishes a morerobust and cost-effective foundation for futureendeavours.

2.2 Existing System In The Market
SAP ERP, by SAP, a leading corporate software
solutions provider, automates key business operations
like accounting, sales, and human resources. SAP ECC,
the main component, handles transactions, reporting,and integration. SAP S/4HANA, the latest version,offers real-time data processing and improved analytics.
Known for a well-explained UI, SAP ERP is popularamong giants like PayPal and NBA. Recently integrated
with cloud capabilities, SAP ERP provides acustomizable cloud solution for global business models
and revenue growth [5].

Adaptability and Flexibility: The system's inherent
adaptability and flexibility empower the organization toundergo simple reorganization swiftly. This adaptability
extends to accommodating modifications in businessprocesses, ensuring that the enterprise remains agile inresponse to evolving market dynamics. This capability
becomes a cornerstone for sustained innovation andresilience in an ever-changing business landscape.
E-Commerce and E-Business Integration: The systemseamlessly supports online shopping experiences andfosters a collaborative environment conducive toeffective teamwork. This collaboration, in turn, enables
effortless integration with e-commerce systems. By
embracing and facilitating digital interactions, theorganization not only capitalizes on the growing trendof online commerce but also positions itself forenhanced connectivity and synergy across various
aspects of its e-business operations.

Oracle ERP, from Oracle Corporation, optimizes
finance, supply chain management, and humanresources. Renowned for dependability and scalability,
it suits both small and large businesses. Oracle ERP iscustomizable to meet industry needs and offers
extensive capabilities in data analytics, reporting, andsystem integration. With a solid reputation and broad
support services, Oracle ERP has been a market leader
for over a decade.
Microsoft Dynamics 365, part of Microsoft's businessapplication package, is a popular ERP system. It covers
accounting, supply chain management, sales, andcustomer support. Unique for seamless integration with
Microsoft products like Office 365, it provides aconsistent user experience. Microsoft Dynamics 365
ERP offers advanced analytics, AI-driven insights, andcloud-based
deployment
options,
facilitating
operational growth, insights, and automation.

2.4 ERP Drawbacks
Time-consuming deployment, the time and resources
required to accomplish an ERP deployment are
frequently considerable [6]. The collecting of
requirements, system configuration, data migration,
testing, and user training all need resources fromorganizations.
Organizational disruption can result from theimplementation phase, which forces staff members to
become used to new procedures and systems. Employee

Odoo ERP, an adaptable open-source system, isamong the fastest-growing ERP systems. It offers tools
for accounting, manufacturing, and sales. Odoo's©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1076

Minature Resource Planner

reluctance to change and temporary productivity
declines may result from this shift.

needs. Project planning outlines the implementation
schedule, resource allocation, and key tasks after
vendor selection. Configuration and customization of
the ERP system to match unique requirements andprocedures follow, including data transfer withaccuracy. User training introduces the new systemthrough sessions, workshops, and documentation.
Testing and quality control ensure functionality, withvarious tests like user acceptability and functionality
testing. Upon readiness, the system is deployed, withongoing support to address user queries. Continuous
improvement is emphasized, with regular monitoring
and feedback collection for streamlined procedures.

Challenges with complexity and customization,
because of their extensive nature, ERP systems may bevery complicated. Adapting the system to a given
organization's requirements can be difficult and may
call for advanced technical knowledge.
2.5 ERP In Different Industries
Healthcare Industries: ERPs are pivotal in healthcare,streamlining processes like patient scheduling,
inventory control, and invoicing. Key benefits include
increased operational effectiveness and compliancewith healthcare data protection laws like HIPAA.

2.8 Quantitative Methodology
Data collection involved in-depth interviews andorganized surveys to gather user and stakeholder
insights. Surveys targeted user groups, focusing on
quantitative aspects like performance indicators, systemusability, and user satisfaction. A comprehensive
analysis of comparable ERP systems enriched
contextual knowledge, dissecting and comparing
various systems to understand ERP dynamics. The
investigation explored the advantages, disadvantages,
and unique features of existing ERPs for designing anddeveloping the Home Resource Planning (HRP) system.

Financial Sector: In finance, ERPs are indispensable
for managing financial data, reducing data loss risks,and ensuring compliance with industry laws,emphasizing the importance of rapid and secure accessto accounting information [7].
Education Sector: ERPs play a vital role in education,
simplifying tasks like resource organization, grading,and student enrollment. They provide colleges andschools with seamless support for effective operations.
Manufacturing Industries: In manufacturing, ERP
systems prove invaluable throughout the workday,
efficiently managing logistics, invoicing, and ensuring
data organization [8]. Beyond these functionalities, theyalso assist employees in maintaining accessible personaldata, highlighting the depth of their capabilities.

2.9 Survey Structure
The survey comprised two parts. The first assessed
respondents' ERP knowledge, experience, andunderstanding of fundamental features. The second partdelved into comprehensive testing methods, evaluating
the system's stability, performance, and user satisfaction
in challenging circumstances. Survey sections included
Knowledge and Familiarity with ERP, HRP Interest,
ERP Experience, Future ERP Use, and an Additional
Comments section for open-ended feedback.

2.6 Comparison Table
Table 1 shows the comparison of the ERP providers.
Table 1. ERP Comparison Table
ERP Provider
Odoo
Microsoft
Oracle365
Deployment
OnOnOnpremises
premises,premises,, Cloud
Cloud
CloudOpen source
Yes
No
No
Customization
HighlyModerate
Moderate
Scalability
Yes
Yes
Yes
User-Friendly
Yes
Yes
Yes
Cost-effective
Yes
VariesVariesPopularity
Growing
PopularPopularCommunity
open
Strong
ActiveSAP
Onpremises, CloudNo
Moderate
Yes
Yes
VariesPopular
Active2.10 Qualitative Methodology
Qualitative methods, including interviews andpersonal
comments,
supplemented
statistical
approaches for a holistic understanding. One-on-one
interviews covered participants' experiences with ERP
systems, potential benefits of an HRP system for dailytasks, and desired features. Questions focused onfamiliarity with ERP systems, willingness to use an
HRP system, valued features, anticipated impacts, andchallenges faced with ERP systems in the past.

2.7 ERP Implementation
3. Implementation

When implementing an ERP system, a comprehensive
plan is crucial. It begins with a detailed requirements
analysis to identify organizational issues and objectives.
Vendor selection follows, involving research to choose
an ERP provider that aligns with the organization's

3.1 Functional Requirements
User registration and authentication: Users are able toset up accounts with distinctive passwords and©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1077

Adim Khalid Aldireejah, Kasthuri Subaramaniam, Ghassan Saleh Hussein Al-Dharhani

usernames. During the login process, the systemauthenticates and verifies the user's credentials.

3.3 Web-based Resource Planner

User-Specific Views: Based on their role andpreferences, every user is shown a customized
dashboard or view. The interface is customized by thesystem to provide appropriate information, including
appointments, finances, people, and duties, for various
users.
Service Selection: A variety of services are available
for users to select from, such as appointment scheduling,money, personnel management, and chore organization.

Fig 1. HRP Home-Page
The Homepage in Home Resource Planning (HRP) isan intuitive user interface that offers a smooth sign-up
or log-in process for current users (Fig. 1). The design
places a high value on simplicity, guaranteeing users aseamless onboarding experience. Furthermore, theplatform provides a brief yet useful feature summary,enabling customers to rapidly understand thepossibilities it provides. A dedicated "Learn More" page
offers a thorough examination of the web application
for individuals looking for more specific details. This
page functions as an in-depth manual, providing
information on the features, and advantages.

Data Entry and Management: Within the service
category they have chosen, users can enter and manage
data. To ensure data accuracy, the system offers
capabilities for data entry, modification, and deletion.
Notification and Reminder System: Users are able tochoose how they want to be notified and reminded.
Reminders and automatic notifications are sent by thesystem in accordance with user-specified schedules andactions.
Data Integration: Users may access and use
information from several modules by means of the ERP
system's smooth integration of data from the otherservices. Maintaining data consistency helps toguarantee accuracy and dependability.
3.2 Non-Functional Requirements
Security: To safeguard user data and preserve privacy,
the ERP system complies with industry-standard
security standards. Sensitive data is protected by robust
encryption and authentication processes.
Scalability: Exhibits the capacity to increase with thenumber of users and volume of data. meets
predetermined performance goals and maintains peak
performance as the system grows.

Fig 2. HRP Home-Page Responsiveness
It's also important to note that the majority of its pages
adjust to the screen size, even on mobile devices.
Whether on a laptop, tablet, or desktop, HRP will try toadjust the size to fit the screen (Fig. 2).

Performance: Reduces downtime and guarantees
quick user request responses. Supports several users
concurrently and effectively handles data transfers
without appreciable performance reduction.
Usability: The design of the user interface places ahigh value on simplicity and clarity, making it easy forusers to navigate and finish activities.
Dependability: An ERP system with highdependability has fewer mistakes and interruptions.

Fig 3. HRP Main Page
Similar to the home page, the system home page issimple to navigate; users may select the service theywant to use here as shown in Fig. 3.. Additionally, theheader and footer are always there and allow users to©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1078

Minature Resource Planner

select from the services offered on each page. At thispoint, it is worth noting that the purpose of HRP is toprovide similar services any ERP can but with easiernavigation.

of vital corporate data while accommodating thevarious needs of ERP modules (Fig. 6). The smoothfunctioning of the database is essential to the overall
effectiveness and success of any ERP system.

Fig 4. HRP Services
Fig 7. User Authentication
Since HRP uses user authentication (Fig. 7), thatallows each user to have their own view which willallow data to be secure and no mix-ups can happen
between users.

One of the features offered by the web-based app is
the ability for users to add individuals to the database,modify or remove their information, or even look for a
specific record as shown in Fig. 4.. Each record has aunique ID to help with identification, and each user will
see their data in a unique way, guaranteeing that eachuser's data is private and personal.

4. Conclusion
As a result of their ability to offer a centralized systemthat unifies several company processes such as supply
chain management, manufacturing, finance, and humanresources—resource planners are incredibly helpful tobusinesses. They have been becoming more popular by
the day owing to their utilization in the business world.
Small firms can increase their productivity andefficiency considerably by using an ERP system, but it
does require careful design, sufficient funding, andgood user support and training. A number of other
implementation-related aspects are covered, such as
small business owners' as well as homeowners’ interest
in utilizing such a system, their familiarity with ERP
software, and their long-term goals for system adoption
or upgrades. The analysis also showed that different
small business owners have different long-term plans
for adopting or upgrading their systems; some want tomove to more simple solutions, while others would
rather learn more about them in general.
Additionally, Homeowners' interest in utilizing an
ERP system for household tasks is also emphasized.
The research revealed a significant market and need forsuch systems, with over 80% of participants expressing
a strong interest in an HRP system designed for duties
connected to the household. In order to satisfy the needsof a diverse variety of customers with a wide range ofuser requirements and expectations, the researchhighlights the significance of having an adaptive andversatile HRP system.
Introducing HRP to the market and matching it withthe current trends might have a lot of benefits for homes
and company owners alike, as well as for anyone
looking to improve efficiency and organization in theireveryday lives. An HRP system is a promising addition
to homes and small enterprises, with the potential torevolutionize resource management.

Fig 5. Pop-ups and Forms
The user will always be informed if an activity was
successful or not with pop-up pages and messages thatappear after any action (Fig. 5). In any online
application, error management is Considered one of themost important features for a web-app/website to have.

Fig 6. Database Schema
The ERP system's database is distinguished byadvanced development and maximum functionality.
With a strong performance, it effectively organizes andmaintains large amounts of data, which adds to thesystem's overall dependability and efficiency. Because
the database is designed to be scalable, data can be
stored and retrieved quickly, guaranteeing a flexible andresponsive system. Its challenging architecture creates asafe and organized space for the archiving and retrieval©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1079

Powered by TCPDF (www.tcpdf.org)

Adim Khalid Aldireejah, Kasthuri Subaramaniam, Ghassan Saleh Hussein Al-Dharhani

References
1.
2.
3.

4.

5.

6.

7.

8.

K. Hayes, “ERP industries: What industries use ERP
software?,” SelectHub raquo, 2023.
E. Watson, “Using ERP systems in education researchgate,” Using Erp Systems In Education, 1999.
S. Singh, S. Singh, and S. C. Misra, “Postimplementation challenges of ERP system inpharmaceutical companies,” International Journal of
Quality & Reliability Management, 2023.
B. Sampath and K. A. Mathappa, "Analyzing the Impact
of ERP on Improving Business Operations Using
Oracle," Latest Trends in Multidisciplinary Research &Development, vol. 35, 2023.
U. Malhotra, "Secure and Compatible Integration of
Cloud-Based ERP Solution: A Review," International
Journal of Intelligent Systems and Applications inEngineering, vol. 11, no. 9s, pp. 695-707, 2023.
S. L. Z. Zahra and T. Siswanto, "Implementation of
Odoo-Based ERP in The Case Study of Micro, Small,and Medium Enterprises (MSME) 'Woody Moody
Jakarta'," Intelmatics, vol. 3, no. 2, pp. 68-77, 2023.
B. Arifianto, S. S. Azhar, and D. F. Murad, "Evaluation
and Recommendation of Odoo Enterprise Resource
Planning System Operation & Maintenance Module," in2023 8th International Conference on Business andIndustrial Research (ICBIR), IEEE, May 2023, pp. 102107.
S. Tongsuksai, S. Mathrani, and K. Weerasinghe,"Influential characteristics and benefits of cloud ERP
adoption in New Zealand SMEs: a vendors’ perspective,"
IEEE Access, vol. 11, pp. 23956-23979, 2023.

Authors Introduction
Mr. Adim Khalid Aldireejah
He is currently pursuing Bachelor ofComputer Science (Hons) at Institute ofComputer Science and Digital Innovation(ICSDI), UCSI University, Kuala Lumpur,
Malaysia.

Dr. Kasthuri Subaramaniam
She is currently an assistant professor at
Institute of Computer Science and Digital
Innovation (ICSDI), UCSI University,
Kuala Lumpur, Malaysia. She earned bothher bachelor’s degree in computer scienceand a master’s degree in computer sciencefrom the University of Malaya. Her
research interests include human-computer interaction,
human personality types, etc.
Dr. Ghassan Saleh Hussein Al-Dharhani
He is currently an assistant professor at
Institute of Computer Science and Digital
Innovation (ICSDI), UCSI University,
Kuala Lumpur, Malaysia. His researchinterests include cybersecurity, ethicalhacking and machine learning.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 20241080

Developing Cloud-based Sportswear Website
Wei Yee Lim

Institute of Computer Science and Digital Innovation, UCSI University, Malaysia
Kasthuri Subaramaniam

Institute of Computer Science and Digital Innovation, UCSI University, Malaysia
Raenu Kolandaisamy

Institute of Computer Science and Digital Innovation, UCSI University, Malaysia
Email: weiyee55@gmail.com, kasthurisuba@ucsiuniversity.edu.my, raenu@ucsiuniversity.edu.my

Abstract
This research is going to develop a cloud-based sportswear website that can handle high volumes of traffic during eventsor seasonal sales, assist customers in selecting the right size for their sportswear, the interface should have options, size
charts, fitting instructions, and recommendations. A study on the existing system will be conducted, design, develop andevaluate on the sportswear website also will be carried out. The system development approach used on this website is
Rapid Application Development (RAD) model. The reason to choose RAD is because this method is time efficient,flexibility and adaptability. With RAD, functional software is delivered quickly. We can quickly build prototypes, iterateon them, and gather feedback from stakeholders with a website project. There are four of the steps in Rapid Application
Development (RAD) methodology life cycle, which is define the requirements, prototypes, rapid construction andfeedback gathering, and cutover.
Keywords: Cloud computing, Sportswear trend, E-commerce

Cloud computing provides computer resources as aservice over the internet, offering dynamic scalability
and virtualization. Users' information is stored remotely,
allowing accessibility anytime, anywhere. The threebasic cloud computing service models are Infrastructure
as a Service (IaaS), Platform as a Service (PaaS), andSoftware as a Service (SaaS), each serving distinct
purposes in the cloud ecosystem [6].

1. Introduction
The Internet, especially with the advent of cloudcomputing, has brought about significant changes in ourworld. Cloud computing, a highly advanced
technological product, has emerged as a key player inthe 21st century's era of globalization, promoting social
normalization and expanding social equality [1]. This
innovation relies on the World Wide Web and the rapid
growth of Internet technology, offering a solid andscalable security structure crucial for its real-world
applications.

The e-commerce landscape is evolving, and cloudcomputing is playing a pivotal role in shaping its
technical architecture. Different types of e-commerce,such as B2B, B2C, C2C, C2B, B2A, and C2A, cater todiverse business models and transactions. This project
specifically adopts the B2C model, focusing on selling
athletic apparel and related items directly to individual
users through a sportswear website [2].

Distinguishing
e-commercefromtraditionalcommerce, e-commerce conducts transactions online,
allowing users to shop, trade, auction, and accessvarious services. The advantages of e-commerce
include lower costs, increased convenience, and accessto a broader range of products. Cloud computing,viewed as the next revolution in the science andtechnology industry, is expected to significantly impact
enterprise operations when combined with e-commerce.
It eliminates the complexities and costs associated withtraditional business applications, providing sharedinfrastructure and scalability.

Amazon EC2 (Elastic Compute Cloud) exemplifies
how cloud computing can benefit e-commerce
enterprises. Instead of purchasing hardware andsoftware, businesses can rent them, reducing systembuilding costs. EC2, a popular cloud system, allows
users to rent cloud-based applications, providing
flexibility and cost-effectiveness. In this project, EC2 is©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1081

Wei Yee Lim, Kasthuri Subaramaniam Raenu Kolandaisamy

chosen as the cloud server to store website data,
leveraging the benefits of Amazon Web Services(AWS).

3. Results and Discussion
The questionnaire results reveal key insights into
consumer behavior and preferences for online
sportswear shopping. Regardless of age, respondents
engaged in sports are more inclined to shop for
sportswear online, with those in their 20s and 30s
showing the highest likelihood. Older age groups,
especially those between 51 to 60 years old, show areluctance to adopt e-commerce, possibly due to limited
familiarity with technology.

The impact of social media on customer buying
behavior is noteworthy, with 3.6 billion users acrossvarious platforms. Social media platforms enhancecustomer connection, broaden reach, and provide
businesses with opportunities to interact with andinfluence customer behavior. Building brand equity
through social media is recognized as a valuable asset in
the contemporary market [3].

Device usage for online shopping is evenly split
between desktop and mobile, with respondents lesslikely to use tablets. JD Sports emerges as more popularthan Nike and Adidas, attributed to its diverse range ofbrands and products. User-friendly navigation menus
and simple, clean website designs are highly valued byrespondents.

Sportswear has become a fashion trend, influenced bychanges in lifestyle, the demand for functional
performance clothing, and the growing interest in
fitness outerwear [7]. Activewear manufacturers thrive
as fitness-conscious consumers seek appropriate gear.
The activewear market, particularly in the United States,
is robust, with Nike and Adidas leading in revenue.
Women's sportswear sales surpass those of men's andchildren's, and this trend is expected to continue into the2020s [4], [5], [8].

Product preferences highlight running shoes andactivewear as top choices, driven not only by sports
engagement but also as fashion items. Eco-friendly andsustainable sportswear gains significant preference,reflecting a growing awareness of environmental impact
among consumers.

2. Methodology
The Rapid Application Development (RAD) approach
revolutionizes software application development with
its iterative and user-centric methodology. Primarily
designed for projects requiring ongoing user
participation, RAD's strength lies in its adaptability andresponsiveness to changing requirements. Contrary to
the traditional waterfall development paradigm, RAD,
introduced in 1991, thrives on flexibility and efficiency,
making it particularly effective for managing modest
and medium-sized projects.

In terms of information, sizing guides, material details,
and product reviews are considered crucial. The
website's content, including blog articles, styling tips,
and athlete endorsements, is appreciated by half of the
respondents.
Checkout preferences are divided, with half opting for
guest checkout and the other half preferring accountcreation before purchase. Touch ′n Go is the favored
payment method, emphasizing the importance of
seamless transactions and a secure checkout process.

The RAD process unfolds through distinct phases.
During the Planning and Define Requirements phase,
meticulous research and requirement gatherings takeplace to grasp the project's objectives, audience, scope,and establish measurable goals. A questionnaire sent to
website clients ensures a comprehensive understandingof their needs. Prototyping follows, where the project
team collaboratively creates an early working model of
the website, incorporating user feedback throughiterative cycles and casual interviews to refine featuresand eliminate design errors.

Email is the preferred mode of communication forcustomer support, chosen by half of the respondents
over phone and live chat. Privacy and data security areparamount, with customers expecting companies tosafeguard their information.
Overall, respondents express satisfaction with their
online shopping experiences, emphasizing theimportance of trust, reasonable pricing, and quality
products for a successful e-commerce venture.

Rapid Construction and Feedback Gathering phasetranslates user feedback into practical developmenttasks, including coding and testing. Adjustments aremadetoapplication
prototypes
basedonrecommendations, and thorough testing ensureseffective functionality. Once the website is launched,
the Maintenance phase kicks in, focusing on regularupkeep to keep the site current, secure, and operational.
This involves content updates, software and plugin
maintenance, issue resolution, and adapting to evolving
user input or business requirements.

3.1 Implementation
This section focuses on a dynamic use case as shownin Fig. 1, emphasizing the synthesis of various diagrams
to provide a comprehensive understanding of a specificsystem. Starting with a clever use case scenario analysis,
An Entity-Relationship (ER) diagram (Fig. 2) visualizes
the database structure. Concrete observations, such as
screenshots of the admin and customer website sections,
are integrated to bridge conceptual and practical aspects,©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 20241082

Developing Cloud-based Sportswear

offering a holistic overview of the system's design andoperation (Fig. 3).

Fig. 3 Class Diagram

3.2 Evaluation

Fig. 1 Use Case Diagram

Evaluation begins with three crucial testing stages:
unit testing, integration testing, and system testing.
Together, these stages allow us to thoroughly assess thefunctionality and dependability of the proposed website.
Table 1, Table 2, Table 3 and Table 4 show thesummary of the testing for each of the stages.
Table 1. Login Unit Test

Fig. 2 Entity Relationship Diagram

Procedure
Description

Expected Result

ActualResultTest
ResultUsername text
field

Allow user to
key in theirusername.

Working as
expected

Pass

Password
field

text

Allow user to
key in theirpassword.

Working as
expected

Pass

Login button(Admin)Allow admin toauthenticate toaccessthewebsite.

Working as
expected

Pass

Login button(Customer)

Allow customer
to authenticate
to access thewebsite.

Working as
expected

Pass

ForgetPassword
buttonAllow user to
reset
their
password.

Working as
expected

Pass©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1083

Wei Yee Lim, Kasthuri Subaramaniam Raenu Kolandaisamy

Table 2. Integration Test
Procedure
Description

Expected Result

ActualResultTest
ResultLogin function
should connect
to database tovalidatetheuser.

The system able
to connect todatabaseandauthenticate theuser based on
their username
and password.

The system
connects todatabase andverifies the
user.

Pass

Customer placeorder function
should connect
to database torecord customer
order.

The system able
tostorecustomer orderin the database.

The system
connects todatabase andstore
customerorder.

Pass

Admin
side
function should
connect
todatabasetoupdate productdetail.

The system able
to store productdetail in the
database.

The system
connects todatabase andstore
product
detail.

Pass

All website functions are operational and working
seamlessly

Yes

Error message when username or password
incorrect

Yes

Alert successful message when action is performed

Yes

Able to logout

Yes

4. Conclusion
In summary, the construction and hosting of a cloudbased sportswear website on Amazon EC2 involved
careful consideration of critical factors, particularly inresponse to the evolving landscape of e-commerce
accelerated by the COVID-19 pandemic. The adoptionof Amazon EC2, a reliable and scalable cloudcomputing solution, underscored the importance ofleveraging
technology
forresilienceandcompetitiveness in the e-commerce industry. The Rapid
Application Development (RAD) methodology
facilitated a collaborative and iterative development
approach, aligning with the dynamic nature of ecommerce. Rigorous testing, including unit tests,
integration tests, system tests, and user perception tests,
ensured the website's reliability, functionality, and user
satisfaction. This effective combination of digitaltechnologies, cloud computing, RAD methodology, andcomprehensive testing positions the sportswear website
for success in the competitive e-commerce landscape,addressing challenges posed by the pandemic andestablishing a robust foundation for sustained success inthe dynamic online business environment.

Table 3. Admin side System Testing
Expected Result

Actual Result

Admin able to login to thesystem.

Admin can login successfully
by insert username andpassword.

Admin able to view producton website.

All product is showed on thewebsite.

Admin can add new product.

Product can be added by insert
all the product detail.

Admin can amend productdetail.

Product can be amended byinsert all the amend detail.

Admin can delete product.

Product can be deleted byclicking the confirm delete
button.

Admin can search user list.

User list is showed whensearch button is clicked.

Admin can add new admin.

New admin can be added byinsert all the admin detail.

Admin can inactivate or
activate user.

User can inactivate or activate
when button is clicked.

Admin can logout from thensystem.

Admin can logout successfully
when logout button is clicked.

References
1.

2.

3.

4.
Table 4. User Perception Test
Function

Yes/NoAll the button are working perfectly

Yes

5.

Able to login

Yes

6.

All text field allow user to enter text

Yes

Dropdown menu allow user selection

Yes

Navigation menu allow user selection

Yes

7.

Kysh L.M., “Adaptation Of B2c E-Commerce To The
Conditions Of The Covid-19 Pandemic,” East European
Scientific Journal, vol. 12, no. 64, 2020.
S. Ramizulhasan, Dr. N. Wajid, and K. Azam,“Changing Dynamics of Online Trust and Retailers
Ethics: A B2C study of high valued products in Pakistanfrom the COVID-19 perspective,” International Journalof Business and Economic Affairs, vol. 6, no. 6, 2022.
doi:10.24088/ijbea-2021-66003
Ahmed, I. (2019). A brief review: security issues incloud computing and their solutions. Telkomnika,17(6),2812-2817.
Hussainn, W., Hussainn, F.K., Hussainn, O.K., Damiani,E., & Chang, E. (2017). Formulating and managing
viable SLAs in cloud computing from a small to
medium service provider's viewpoint: A state-of-the-art
review. Journal of Information System, 71(10), 240-259
Namburu, “Analysis Of Cloud Computing Services
Using Amazon EC2,” Course Hero, 2023.
S. Shokeen and A. Singh, Sci-Hub | Deploying an Ecommerce website using Amazon Web Services ...,
https://sci-hub.se/10.1109/IC3I46837.2019.9055586(accessed Nov. 26, 2023).
What is Amazon EC2? - amazon elastic compute cloud,https://docs.aws.amazon.com/AWSEC2/latest/UserGuid
e/concepts.html (accessed Nov. 27, 2023).

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1084

Powered by TCPDF (www.tcpdf.org)

Developing Cloud-based Sportswear

8.

X Li, “Nike website analysis,” Academia.edu,
https://www.academia.edu/5711105/Nike_Website_Analysis (accessed Jun. 6, 2023).

Authors Introduction
Ms. Wei Yee Lim
She is a student majoring in B. Sc. (Hons)Business Information Systems at Instituteof Computer Science and Digital
Innovation (ICSDI), UCSI University,
Kuala Lumpur, Malaysia.

Dr. Kasthuri Subaramaniam
She is currently an assistant professor at
Institute of Computer Science and Digital
Innovation (ICSDI), UCSI University,
Kuala Lumpur, Malaysia. She earned bothher bachelor’s degree in computer scienceand a master’s degree in computer sciencefrom the University of Malaya. Her
research interests include human-computer interaction, ecommerce and etc.
Dr. Raenu Kolandaisamy
He is currently an assistant professor at
Institute of Computer Science and Digital
Innovation (ICSDI), UCSI University,
Kuala Lumpur, Malaysia. He has a PhD in
Computer Science. His research interests
include Wireless Networking, VANET and
IoTs.

©The 2024 International Conference on Artificial Life and Robotics (ICAROB2024), J:COM HorutoHall, Oita, Japan, 2024
1085