SUGISAKAMASANORI

Journal of Advances in Artificial Life Robotics
Vol. 1(1); June (2020), pp. 2–6
ON LINE ISSN 2435-8061; ISSN-L 2435-8061
https://alife-robotics.org/jallr.html

Research Article

Robot Motion and Grasping for Blindfold Handover
Sakmongkon Chumkamon, Kota Kawamoto, Umaporn Yokkampon, Eiji Hayashi
Kyushu Institute of Technology, Japan

ARTICLE INFO
Article History
Received 11 November 2019
Accepted 06 June 2020
Keywords
Factory automation
Motion planning
Human-Robot interaction

ABSTRACT

Autonomous robots in human-robot interaction (HRI) recently are becoming part of human lifeas the number of services or personal robots increasingly used in our home. In order to fulfill thegap of HRI merit, we would like to propose a system of the autonomous robot motion and graspingcreation for assisting the disabled person such as the blind people for handover tasks to help blindpeople in pick and place tasks. In this paper, we develop the robot motion to receive the object byhanding over from the blindfold human to represent the blindness. To determine the target of theobject and human hand, we implement the 6DOF pose detection using a point cloud and handdetection using the Single Shot Detection model in Deep learning for planning motion using
9DOF arm robot with hand. We finally experiment and evaluate the tasks from blindfold-robot
handover tasks..

© 2022 The Author. Published by Sugisaka Masanori at ALife Robotics Corporation Ltd
This is an open access article distributed under the CC BY-NC 4.0 license(http://creativecommons.org/licenses/by-nc/4.0/).

various movement such as robot arm motion, vehicle,
drone [2].

1. Introduction
In order to push country economic, the smartautomation in industrial sector is keys to improve
nowadays. Since the merit of artificial intelligence
research could facilitate various tasks in industrial.
Therefore, we are able to develop the factory that could
produce products faster with higher precision. In smart
factory, the robot is important to assembly because it ismore reliable and accuracy than human.
The manipulator robot actually is controlled by joint
position however we usually control the robot in
Cartesian space therefore we have to calculate theinverse kinematic (IK). One of popular method of IKrecently is Trac-IK and KDL which can calculate thesolution more than related method [1]. Since the robot
also need to plan the trajectory according to theenvironments such as avoiding the collision or moving
to the target. The recent motion planning method isOpen Motion Planning Library which is popular,
reliable and fast for solving the trajectory motion inFig. 1. The overview system of challenge experimentof handover tasks from blindfold

Corresponding author’s E-mail: chumkamon.sakmongkon637@mail.kyutech.jp, kawamoto@mmcs.mse.kyuetech.ac.jp,
jiraphan@mmcs.mse.kyutech.ac.jp, may@mmcs.mse.kyuetech.ac.jp, haya@mse.kyutech.ac.jp

2Since the robot could not move by itself, the user haveto teach the robot to move according to the tasks.
However, sometimes it is hard to teach the robot
because of the difficult environment. Therefore, weproposed the teaching less system that the robot cangenerate motion according to the situation. We
implement the perception system to let the robot canautonomously planning its motion depending on thetasks. This teaching less system is shown in Fig. 1.
Collaborative robot (Cobot) becomes import recently
since it helps the worker achieve the tasks easier andmore reliable. There is the research on assembly tasks
or Pick and Place recently [3]. However, the recent
research is lack for implement the robot for disabled
people therefore our application also can apply forservice tasks for blind people. Since the blindness
recently is lack support from the society, we therefore
attempt to initiate the experiment that can apply forblindness using subject with blindfold.
In this paper, we propose the robot system that cangenerate motion to grasping the object from human in
any poses by let the human blind their eye that thehuman cannot help the robot to grab the object. Inaddition, this system might be useful for blind people inservice tasks such helping the blind people keep thething to its place.
In the rested paper, we organize the section in 5 section
includingFig. 2. The robot system and hand for grasping.

Fig. 3. Example color image and depth image from handdetection.

2. Concept of Motion Generation for Handoverfrom Blindfold
In this section, we explain the overview system
including robot hardware configuration, robot
perception, motion planning and grasping pose forhandover.
2.1. Hardware Configuration
From the beginning, we have proposed the concept idea
and the objective of this system. Consequently, we
design the hardware configuration. Firstly, we utilize
the robot arm which is industrial robot from Motoman
Yaskawa as an operator for getting the object fromblindfold. The robot was construct by 7 degree of
freedom including one joints in base, two joints inshoulder, two joints in elbow, and two joints in wrist.
For grippe, we use D-Hand robot which 2 DoF that cancontrol finger angle and gripper state. The robot isshown in the Fig. 2.

Fig. 4. 3D point cloud of hand and object position estimation
for robot grasping.

3

Fig. 5. 3D point cloud of environment and subject with robot model during handover experiment in different views

T1

T2

T3

T4

T5

T6

T7

T8

Fig. 6. The snapshots during object handover experiment from blindfold to robot

point represents object and lower point represents handposition.

2.2. Perception
We develop perception system in order to estimate thetarget position for motion which is based on the handdetection and object position estimation. Firstly, weemploy RGBD camera to getting image. In perception,
hand detection is implemented using Deep Learning
based Single Shot Detection [4] then we extract thelocation of hand from color image and depth image. We
estimate the hand position by project the depth image in
Cartesian space. Afterward we extract area of the points
cloud of hand and object in the palm. We then remove
the hand point by skin filter and get only object pointand estimation the position. Fig 3 shows the result whenthe robot can detect the hand extracting in color anddepth image. Fig. 4 shows the results in point cloud of
position estimation of hand and object which the upper2.3. Motion Planning
Since the robot knew the position of the object, we thencan plan the motion for the robot. Firstly, we set thehome pose to initialize the robot joints to prepare forgrasping object from blindfold. When the robot candetect hand and object then it would plan motion using
OMPL and KDL inverse kinematic. In thisimplementation, we utilize MoveIt API for organize themotion plan and collision protection among theenvironment. After complete detection, then robot
would approach the object prepare for grasping thenmove to grasp from blindfold who does know thecurrent object pose and robot hand position that human4

Planning Library and KDL inverse kinematic which arecapable to plan the motion to grasp the object fromhandover tasks. By the experiment and result, weconclude that the robot could generate the motion andgrasping for handover task
For the future work, we would like to the improve robust
pose estimation for grasping using 6DoF pose detection.
Since robot can know the object pose then the robot cangrasp with the reliable position. Moreover, we would
like to improve the reliable of system for real service
task to assist blind people to improve human life.

cannot help the robot grip object easier. Therefore, oursystem is pretty unique among the current research thatusually let human know robot hand and can help robot
to grip object.
3. Experiment
From the above section, we explained our system andthe configuration then in this section we explain thedemonstration and evaluation of our system. Since we
proposed this system for handover from blindfold infactory automation or human service sector. We theninvite subject to do experiment and blind his eye topreventing subject helping robot to grasp which is
unique experiment comparing to recent research andslightly challenge.
The experiment starts from the robot initializing
position and wait for subject hand detection with object.
In Fig. 6 shows experiment snapshot during thehandover from blindfold starting from T0 to T8. In T0
the robot prepares for hand detection in home pose.
After the robot can detect hand then the robot moves topre-grep position in T2. T3, the robot moves move tograsp from subject handover the object robot without
seeing the robot because subject is blindfold. T4 after
handover completely, the robot moves to post-grip
position. Then in T5 and T6 robot move to place thebottle into the box at the pre-place position. T7 and T8,the robot put the bottle into the box softly.
In Fig. 5 shows the 3D image captured during thehandover experiment. The first, second and third show
the side view, top view and front side view respectively.

References
1. Beeson, Patrick, and Barrett Ames. TRAC-IK: An opensource library for improved solving of generic inverse
kinematics. In 2015 IEEE-RAS 15th International
Conference on Humanoid Robots (Humanoids), pp. 928935. IEEE, 2015.
2. Sucan, I.A., Moll, M. and Kavraki, L.E., 2012. The open
motion planning library. IEEE Robotics & Automation
Magazine, 19(4), pp.72-82.
3. Zhu, Z. and Hu, H., 2018. Robot learning fromdemonstration in robotic assembly: A survey. Robotics,7(2), p.17.
4. Liu, Wei, Dragomir Anguelov, Dumitru Erhan, ChristianSzegedy, Scott Reed, Cheng-Yang Fu, and Alexander C.
Berg. "SSD: Single shot multibox detector." In European
conference on computer vision, pp. 21-37. Springer,Cham, 2016.
=========================================

Authors Introduction

Dr. Sakmongkon Chumkamon
Dr.
Sakmongkon
Chumkamon
receivedDoctor of Engineering
degree from Kyushu
Institute of Technology in2017.
Hewasapostdoctoral researcher atGuangdong University ofTechnology in 2017-2019.
Presently he is a postdoctoral researcher in Kyushu
Institute of Technology since 2019. His researchinterests include factory automation robots andsocial robots.

4. Conclusion and Future Works
This paper proposed the autonomous motion planningof manipulator robot to utilize in factory automation for
handover tasks. The handover tasks are important forassembly production since there are many worker androbot operating together. However, our paper also isdesigned to utilize for blind people as a service robot.
Because this system can assist blind people for keeping
things to their place. This might help blind people liveeasier.
In this paper, we already present the system for motion
plan where the system can be employed for factory andhuman life such as blind people. For the perception inthis system, we implement hand position estimation
using deep learning to deploy for motion planning. For
the motion system, we utilize The Open Motion

5

Kota Kawamoto
Kota Kawamoto received
Bachelor Engineering andMaster Engineering degreein Computer Science andSystems Engineering fromKyushu
InstituteofTechnology in 2018 and2020
respectively.
Currently, he works as aengineering in Bosch Corporation Company.
Ms. Umaporn Yokkampon
Ms. Umaporn Yokkampon
received Master of Science
degree in Applied Statistics
from King Mongkut’s
University of Technology
North Bangkok in 2019.
Presently she is a doctoral
student in the Departmentof Computer Science andSystems Engineering at Kyushu Institute ofTechnology. Her research interests include data
mining, anomaly detection and time series analysis.
Prof. Eiji Hayashi
Prof. Eiji Hayashi is a
professor in the Department
of Intelligent and Control
Systems at Kyushu Instituteof Technology. He received
the Ph.D. (Dr. Eng.) degreefrom Waseda University in1996. His research interests
includeIntelligent
mechanics, Mechanical systems and Perceptual
information processing. He is a member of The
Institute of Electrical and Electronics Engineers(IEEE) and The Japan Society of MechanicalEngineers (JSME).
===========================================

6