SUGISAKAMASANORI

Journal of Advances in Artificial Life Robotics
Vol. 4(1); June (2023), pp. 1–5
ON LINE ISSN 2435-8061; ISSN-L 2435-8061
https://alife-robotics.org/jallr.html

Research Article

Industrial Robot Introduction through Augmented Reality
System
Takuya Matsumoto1, Eiji Hayashi2, Sakmongkon Chumkamon2, Ayumu Tominaga3, Abbe Mowshowitz4

Department of Intelligent and Control Systems, Kyushu Institute of Technology, 680-4, Kawazu, Iizuka-city, Fukuoka, 820-8502, Japan
Department of Mechanical Information Science and Technology, Kyushu Institute of Technology, 680-4, Kawazu, Iizuka-city, Fukuoka, 820-8502, Japan3
Department of Creative Engineering Robotics and Mechatronics Course, National Institute of Technology Kitakyushu College, 5-20-1 Shi,Kokuraminamiku, Kitakyushu-city, Fukuoka, 802-0985, Japan
4
Department of Computer Science, The City College of New York, 160 Convent Avenue, New York, NY 10031, USA
1
2ARTICLE I N FO
Article History
Received 02 December 2022
Accepted 02 August 2023

Keywords
ARFactory automation robots
ARCore
Unity
ROS eABSTRACT

In this research, we developed an application that allows AR simulations to check safety ranges
and operation details. Furthermore, we created a user-friendly GUI, suitable for individuals withlimited knowledge in robot development. This application was developed using ARCore in Unity.
By establishing communication between ROS (Robot Operating System) and Unity to control
virtual robots, we achieved visualization of ROS-based robots in the AR environment. With thisAR application, the need for physical safety design and verification of safety ranges during theintroduction of robots in small and medium-sized enterprises has been significantly reduced. Asa result, cost reduction in the implementation process can be expected, and it offers a potential
solution to the labor shortage issues in such enterprises. In conclusion, the effectiveness of theAR application developed in this research has been confirmed.

© 2022 The Author. Published by Sugisaka Masanori at ALife Robotics Corporation Ltd.
This is an open access article distributed under the CC BY-NC 4.0 license(http://creativecommons.org/licenses/by-nc/4.0/).

environment through devices like smartphones or AR
glasses. By operating a virtual robot model in an AR
environment, SMEs can assess food preparation

1. Introduction
In recent years, the issue of labor shortages has become
increasingly critical for small and medium-sizedenterprises (SMEs). In particular, the food service
industry faces challenges in finding adequate labor tomeet its demands. To address this, automation using
robots is seen as a promising solution to improve
productivity and cope with the scarcity of available
workforce. However, the high costs associated withrobot introduction, including design, parts, andintegration expenses, present significant barriers,
especially for SMEs with limited financial resources.
In this context, this study explores the potential ofAugmented Reality (AR) technology as a means tosupport SMEs in robot introduction. AR technology
superimposes virtual information onto the realprocesses and safety ranges within their factory without
the need for physical robot installations and operational
tests. This approach is expected to considerably reduceoverall introduction costs, encompassing design andsystem integration expenditures.
The primary focus of this research is the development ofan AR application tailored for SMEs to evaluate thesafety range of robots during their introduction phase.
The application is designed with the assumption that its
users may not have prior experience in robotdevelopment. Consequently, we have prioritized userfriendliness and ease of operation during theapplication's development. This paper presents theCorresponding author’s E-mail: matsumoto.takuya130@mail.kyutech.jp URL: http://www.kyutech.ac.jp

1

Once the real robot's actions are completed, a message
indicating the end of the process will be displayed.

details of the application's design, evaluation, andverification process.
2. Application Details
2.1. AR System Configuration
In this study, we utilized three primary technologies forthe development of the AR application. Specifically, weintegrated the ARCore SDK into Unity, enabling theutilization of ARCore, a software dedicated to AR
development, for creating the application. Fig. 1
presents a schematic diagram illustrating the AR system.

Fig. 1. AR System Configuration

Fig. 2. The Display of AR Simulation

3. Robot System

2.2. AR Application

This chapter describes the Real Robot itself, the image
recognition system used for it, and the operation flow ofthe Real Robot.

The application has two main parts. One part is for AR
simulation, and the other part is for AR simulation. Theother part is for grasping with a real robot. Thisapplication is to operate a real robot after AR simulation.
The real robot part is explained in Chapter 3. RobotSystem.

3.1. Robot Configuration
In the foodservice industry, robots are required toperform tasks equivalent to those of humans,necessitating more human-like and flexible movements.
Therefore, to achieve a certain level of high freedom ofmovement, a 6-axis vertically articulated robot isemployed. The model used for this purpose is theUmbraTek UTRA6-850. Fig.3.1 shows the appearance.
Furthermore, at the robot's end-effector, as shown inFig.3.2, an RGB-D camera was equipped for object
recognition purposes. Additionally, a stepper motor wasused for the gripper to enable grasping of solid fooditems. The stepper motor allows the gripper to open andclose, facilitating the handling of solid food products.

2.2.1. Simulation Part in AR
The application flow is depicted in Fig. 2, proceeding in
the following order: (a), (b), (c), and (d).
(a) Select Object Scene
Users select the type, quantity, and placement position ofsolidified food by dragging and dropping within theapplication. After making selections, they transition to
the simulation screen where AR robots conduct thesimulation by pressing the "Plan" and "Execute" buttons.
Additionally, users have the option to start over from thebeginning by pressing the "Reset" button.
(b) Simulate Scene in AR
By pressing the "SetObject" button, the selected virtual
object in (a) is placed at the same real-world coordinates.
Users can then use the "Pick" and "Place" buttons toinstruct the AR robot to perform pick-and-place actions.
(c) Start Real UTRA Robot Scene
After the AR robot simulation is completed, buttons tocontrol the real robot's movements appear. By pressingthese buttons, the real robot performs pick-and-placeactions with the same objects as used in the AR
simulation.
(d) End Scene

Fig. 3.1. Appearance of UTRA6-850

2(a) Home:
Placement for visualization processing at the home
position before and after grasping.
(b) Pre-pick (Previous pick):
Motion before grasping the object(c) Pick:
Motion to grasp the object(d) Post-pick (Postpone pick):
Motion to lift the grasped object up once(e) Pre-place (Previous place):
Motion before placing the object on the plate(f) Place:
Placing an object on the table

Fig. 3.2. Appearance of the End Effector

3.2. System Configuration
During the gripping and serving process, it is essential toidentify the position of the solidified food. For thispurpose, an object recognition system was employed.
Initially, the information of the object RGB Image iscaptured using an RGB-D camera situated on the robot'stip, and instance segmentation is conducted.
Subsequently, the two-dimensional coordinates of thecenter of gravity are computed based on the generated
segmentation image. Additionally, depth information isacquired to determine the 3D center-of-gravity
coordinates for the object. Moreover, a microcon isutilized to control the grippers' opening and closing
motions via a PC. Fig. 4 showcases both the image
acquired from the RGB-D camera and the corresponding
processed image(a)Home(b)Pre-Pick(c)Pick(d)Post-Pick(e)Pre-Place(f)Place

Fig. 5. Each motion phase of a real robot

4. Experiment(a) Original image(RGB)

4.1. GUI Evaluation Experiment[1](b) Segmented Image

The AR application is intended for small and mediumsized industrial robots. They are not familiar with robot
development and are therefore not familiar with how tooperate the application. The application cannot be used
if the operation method is complicated. Therefore, theGUI application was designed to be simple enough to beused by first-time users. Therefore, the following
experiments were conducted to confirm that the GUI is
simple enough to be used by first-time users of theapplication.
First, gather 10 subjects and divide them into two groups,
A and B, with 5 subjects in each group. 2.
Next, the five subjects in Group A were given a full
explanation of how to use the application, and asked to(c) Image with marked centroid coordinates
Fig. 4. The information from the RGB-D camera and theprocessed images after each step

3.3. Real Robot Grasping part
Next, we describe the motion flow for grasping solidified
food using the robot, hardware, and image recognition
system. The operation phase is divided into six phasesfrom (a) to (f). The motion is performed in the order of(a) to (f). Fig.5 shows the robot in each phase.
3

ARCore in Unity. By establishing communication
between ROS and Unity to control virtual robots, weachieved visualization of ROS-based robots in the ARenvironment. With this AR application, the need forphysical safety design and verification of safety ranges
during the introduction of robots in small and mediumsized enterprises has been significantly reduced. As aresult, cost reduction in the implementation process canbe expected, and it offers a potential solution to the labor
shortage issues in such enterprises. In conclusion, theeffectiveness of the AR application developed in thisresearch has been confirmed.

use the application. The other five participants in GroupB will be asked to use the application without any
explanation of how to use it. If the subjects were able toperform the sequence of operations from "(a) the object
selection screen" to "(d) the display of the serving
results", they were able to successfully use theapplication.
However, if the subject is unable to complete theoperation due to the user's operation or comprehensionof the operation method, the subject is considered to havefailed. In this case, the subject is considered to havefailed. For example, the user could not press the start
button even though the start button was displayed on theactual device. This means a case in which the functions
installed in the application were not fully usable. In otherwords, when it stops in an emergency due to danger, it is
not regarded as a failure, and the process is restarted
from the beginning.
The percentage of successes in each of the above groups
is calculated and compared. Table 1 shows the results.
Table 1. The results of user success or failure using theapplication

References
1. Kazuya Ikegami, Hidehiko Okada. [ Toward Usability
Quantification] Usability teiryouka ni mukete(inJapanese). NEC Technical Journal, Vol.61 No.2/2008, pp.
53-56.

Authors Introduction
Mr. Takuya Matsumoto
He received bachelor degree inEngineering in 2022 from intelligent andControl Systems, Kyushu Institute ofTechnology in Japan. He is currently aMaster student at Kyushu Institute ofTechnology and conducts research at
Hayashi Laboratory.

4.2. Discussion

Prof. Eiji Hayashi

He is a professor in the Department ofIntelligent and Control Systems atKyushu Institute of Technology. He
received the Ph.D. (Dr. Eng.) degreefrom Waseda University in 1996. His
research interests include Intelligent
mechanics, Mechanical systems andPerceptual information processing. He
is a member of The Institute of Electrical and Electronics
Engineers (IEEE) and The Japan Society of MechanicalEngineers (JSME).

Experiments 4.1 were conducted to assess the usabilityof the developed application. The results from 4.1
revealed that while the application can be utilized
effectively with proper guidance, it proved challenging
for first-time users. Particularly, a significant number ofusers encountered difficulties when they want the AR
Robot to grasp the virtual objects. To address this issue,future improvements could involve developing anapplication that prevents the acceptance of subsequent
operation commands while the virtual robot is in motion.
By doing so, it is anticipated that the number of failuresduring this stage will be reduced.

Dr. Sakmongkon

He received Doctor of Engineering
degree from Kyushu Institute ofTechnology in 2017. He was apostdoctoral Guangdong Technology
Presently he is a postdoctoral researcher
in Kyushu Institute of Technology since2019. His research interests include
factory automation robots and social robots.

5. Conclusion
In this research, we developed an application that allowsAR simulations to check safety ranges and operationdetails. Furthermore, we created a user-friendly GUI,
suitable for individuals with limited knowledge in robot
development. This application was developed using4Prof. Abbe Mowshowitz
He received the Ph.D. degree from
University of Michigan in 1967.
He has been professor of computerscience at the City College of New
York and member of the doctoral
faculty at the Graduate Center of
the City University of New Yorksince 1984. His current researchinterests lie in two areas are organizational and managerial
issues in computing, and network science. In addition toteaching and research, He has acted as consultant on theuses and impacts of information technology (especiallycomputer networks) to a wide range of public and privateorganizations in North America and Europe.
Projected Assist. Prof. Ayumu Tominaga
He is a professor in Department ofCreative Engineering Robotics andMechatronics Course at National
Institute of Technology Kitakyushu
College. He received the Ph.D. (Dr.
Eng.) degree from Kyushu Institute ofTechnology in 2021. His researchinterests
includeIntelligent
mechanics, Mechanical systems and Perceptual
information processing

5