Internationalof Computational
Intelligence
Systems
Journal ofJournal
Robotics,Networking andArtificial Life
Vol. XX(Z);Month(Year),xx–yy5(4); March(2019),
pp.pp.
232–235
DOI: 10.1080/XXXXXXXXXXXXXX;ISSN
ISSN2405-9021
XXXX–XXXXDOI: https://doi.org/10.2991/jrnal.k.190220.005;
print; online
2352-6386 online
https://www.atlantis-press.com/journals/ijndc
https://www.atlantis-press.com/journals/jrnal

Research Article

Motion-Data Driven Grasp/Assembly Planner
Kensuke Harada1,2,*, Natsuki Yamanobe2, Weiwei Wan1,2, Kazuyuki Nagata2, Ixchel G. Ramirez-Alpizar1, Tokuo Tsuji3
Graduate School of Engineering Science, Osaka University, Toyonaka 560-8531, JapanIntelligent Systems Research Institute, National Institute of Advanced Industrial Science and Technology (AIST), Tsukuba 305-8568, Japan
3
Institute of Science and Technology, Kanazawa University, Kadoma-cho, Kanazawa 920-1192, Japan
1
2

ARTICLE INFOABSTRACT

Article History

This paper proposes a robotic grasp/assembly planner using a database of complex and skillful motion. In our proposed planner,
if a skillful motion is included in the solution path, motion data included in the database is used. In this paper, we first explain thethree-layered structure of the motion data. Then, we explain the proposed grasp/assembly planner. Finally, we show a numericalexample to confirm the effectiveness of our proposed method.

Received 18 October 2018
Accepted 21 November 2018
Keywords
Robotic assembly
motion planningskillful motion
database© 2019 The Authors. Published by Atlantis Press SARL.
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION

2. THREE-LAYERED DATA STRUCTURE

In factory environment, industrial robots are expected to assemble
a product. However, robotic assembly is difficult since a robot has
to assemble a variety of objects. Sometimes, a robot has to realize avery skillful and complex motion.

In this section, we explain the structure of the motion data andmethod for storing skillful motion to the database [1]. An assemblytask is composed of multiple task elements where each task e­ lement
is composed of a set of motion sequence. According to this classification, the first, the second and the third layers are respectively
named as “work”, “task” and “action” layers. Figure 1 shows anexample of assembly of a toy airplane. Since a single action usually includes multiple objects to be assembled, the object held bya robot is named as the “main object” and the object assembled tothe main object is named as the “sub object”. The following is a setof information included in the database:On the other hand, this research aims to fully automate theproduct assembly process and proposes a novel robotic grasp/
assembly planner. In conventional grasp/assembly planners[2,3,6,7], we simultaneously planned the grasping posture of apart, the assembly sequence and the motion of a robot. However,conventional grasp/assembly planners can just realize a series ofpick-and-place motion of a robot. For the purpose of realizing
more complex and skillful motion using a grasp/assembly planner, we consider incorporating the database of skillful motion.
The motion data can be generated using several methods suchas human motion capture and robotic motion generation methods. Especially, our motion generation tool utilizes the previously proposed three-layered structure of motion data. In our
proposed grasp/assembly planner, if the solution path includes
complex and skillful motion, such motion is realized using thedatabase of skillful motion.
This paper briefly outlines our novel grasp/assembly planner using
the motion data and shows a simple numerical example. The rest of
this paper is organized as follows; we first explain the three-layered
data structure in Section 2. The robotic grasp/assembly planner isexplained in Section 3. Application of motion data to the grasp/
assembly planner is explained in Section 4. A numerical example
is shown in Section 5.
Corresponding author. Email: harada@sys.es.osaka-u.ac.jp*•• Motion strategy (A sequence of task/motion)•• Human (Identified name etc.)•• Robot (Name, type, hand)•• Object Name•• Object category (e.g., JICFS code)•• Object shape•• Object parameters (Mass, CoG, material, stiffness, etc.)•• Grasping configuration (Main object)•• Fixture configuration (Sub object)•• Time series of the main object’s pose w.r.t the sub object•• Control method and its parameters•• Force information applied by the hand
The data can be obtained by several methods such as a humanmotion capture and a motion generation tool. The data is stored
in the YAML format where some examples are shown in Figure 1.

K. Harada et al. / Journal of Robotics, Networking and Artificial Life 5(4) 232–235

233

Figure 1 | Three layered data structure and data expression using YAML format.

On the other hand, we have constructed a software tool manually constructing a skillful motion and store it to the database [1].
Among the three-layered structure of database, our proposed
grasp/assembly planner especially uses the action data corresponding to the edge of the AND/OR graph of assembly which will be
explained in the next section.

3. GRASP/ASSEMBLY PLANNER
DEFINITION
This section shows the overview of the grasp/assembly planner
which has proposed in Harada et al. [2] and Wan and Harada [3].
Let us consider a product composed of m parts P = (P1, · · · , Pm). Let
us also consider assembly of parts by C = (C1, · · · , Cr). For example,
the assembly Cl composed of the parts Pu, Pv, and Pw can be definedas Equation (1)Cl = {( Pu , Pv , Pw ), (Tuv , Tuw ), (nuv , nuw )}(1)

where Tuv and nuv denote the homogenous matrix and theapproach vector, respectively, of P v relative to Pu used for an
assembly task.
To assemble a product, a gripper has to grasp an assembly of parts.
We consider a set of n grippers H = (H1, · · · , Hn) where Hj denotes
the j-th gripper. We consider using a grasp planner to calculate agrasping posture where Gljk denotes the k-th grasping posture ofthe assembly Cl by the gripper Hj. By iteratively calculating grasping posture with changing the approach position, we consider
constructing a set of grasping postures Glj = (Glj1, · · · , Gljk) of theassembly Cl grasped by the gripper Hj.
Every possible assembly sequence can be expressed using theAND/OR graph G(V, E) where the node V is composed of theFigure 2 | AND/OR graph of our assembly task where the red lines denotethe edges with which the action data is associated.

assembly of parts Cl, l = 1, · · · , r (Figure 2). In the grasp/
assembly planner introduced in this section, we assume that theAND/OR graph is defined in advance. To obtain robot motion toassemble a given product, we first solve the AND/OR graph [4,5]and obtain a possible assembly sequence. Then, we consider constructing the manipulation graph from the assembly sequence
including a set of grasping posture. By solving the manipulationgraph, we simultaneously plan grasping posture and robot motion.
The method is detailed in our previous papers [2,3].

234

K. Harada et al. / Journal of Robotics, Networking and Artificial Life 5(4) 232–235

Figure 3 | Simulation result of toy airplane assembly. (a) Picking the body, (b) Placing the body to the assembly position, (c) Picking the wing, (d) Fitting
the wing to the body, (e) Picking the cockpit, (f) Fitting the cockpit to the body, (g) Picking the bolt, (h) Screwing the bolt 1, (i) Screwing the bolt 2,(j) Screwing the bolt 3.

4. APPLICATION OF ASSEMBLY
MOTION DATA
The grasp/assembly planner explained in the previous subsection
just can realize a series of pick-and-place motion. We consider
extending this grasp/assembly planner for more complex tasks
using the database of skillful assembly motion data. In the AND/ORgraph shown in Figure 2, the wing is picked by the gripper and put
on the body, and then the cockpit can be picked and put on thewing. However, the bolt cannot be assembled to the wing just using
a pick-and-place motion. The edges marked in red shows thatthe assembly motion data is required for realizing this assembly
sequence. We consider associating the action Yaml file corresponding to this assembly sequence. To obtain an assembly sequence, weconsider solving the AND/OR graph. If an action Yaml file is associated to the solution edge, we consider using the action Yaml file
to generate the robot motion. Otherwise, we consider constructing
the manipulation graph and solving it to obtain the robot motion.

5. RESULT
We consider performing the robotic assembly of a toy airplane. We
prepared the skillful motion of the robot rotating the bolt stored
in the database by the YAML format. We first obtained the solution path of the AND/OR graph. Since the solution path includes
the screwing motion of the bolt, we used the action YAML data of
the screwing motion. The result is shown in Figure 3. As shown inFigure 3, when the robot assembles the bolt to the wing, the robot
tries to rotate the bolt using the task motion data.

6. CONCLUSION
In this paper, we proposed a motion-data driven robotic grasp/
assembly planner. A motion data includes an information on skillful motion which cannot simply be realized using conventional
grasp/assembly planners. This paper first explained the structureof motion data. Then, we explained the proposed grasp/assemblyplanner. Finally, we showed a numerical example of a toy airplaneassembly.

ACKNOWLEDGMENT
This research was conducted under the support of NEDO
AI-Robotics Project.

REFERENCES[1] N. Yamanobe, T. Tsuji, K. Harada, K. Nagata, R. Hanai, W. Wan,et al. Construction of cloud database for task motion generation, in: Annual Conference on the Robotics Society of Japan,
1B2-07, 2016.
[2] K. Harada, K. Nakayama, W. Wan, K. Nagata, N. Yamanobe, I.G.
Ramirez-Alpizar, Tool exchangeable Grasp/Assembly Planner,
in: M. Strand, R. Dillmann, E. Menegatti, S. Ghidoni, (Eds.),
Intelligent Autonomous Systems 15. IAS 2018. Advances inIntelligent Systems and Computing, Vol. 867, Springer, 2018.
[3] W. Wan, K. Harada, Developing and comparing single-arm anddual-arm regrasp, IEEE Robotics and Automation Letters, IEEE,
2016, pp. 243–250.
[4] L.S.H. de Mello, A.C. Sanderson, AND/OR graph representationof assembly plans, CMU Research Showcase, CMU-RI-TR-86-8,
1986, pp. 1–17.
[5] L.S.H. de Mello, A.C. Sanderson, Automatic generation ofmechanical assembly sequences, CMU Research Showcase,CMU-RI-TR-88-19, 1988.
[6] T. Siméon, J.-P. Laumond, J, Cortés, A. Sahbani, Manipulation
planning with probabilistic roadmaps, Int. J. Robot. Res. 23(2004), 729–746.
[7] K. Harada, T. Tsuji, J.-P. Laumond, A manipulation motion
planner for dual-arm industrial manipulators, 2014 IEEE
International Conference on Robotics and Automation (ICRA),IEEE, Hong Kong, China, 2014, pp. 928–934.

K. Harada et al. / Journal of Robotics, Networking and Artificial Life 5(4) 232–235

235

Authors Introduction
Dr. Kazuyuki Nagata

Prof. Kensuke HaradaHe received his PhD in Engineeringfrom Tohoku University in 1999.
He is currently a Senior Research
Scientist with Intelligent Systems
Research Institute of Advanced
Industrial Science and Technology(AIST).

He received his Doctoral degrees in
Mechanical Engineering from Kyoto
University in 1997. From 2016, hehas been working as a Professor at
Graduate School of Engineering
Science, Osaka University.

Dr. Natsuki Yamanobe

Prof. Ixchel G. Ramirez-Alpiar
She received the PhD Degree fromthe University of Tokyo in 2007.
She is currently a Senior Research
Scientist with Intelligent Systems
Research Institute of Advanced
Industrial Science and Technology(AIST).

Prof. Weiwei Wan

She received the PhD degree inMechanical Engineering from Osaka
University, Osaka, Japan, in 2013.
Since 2016, she has been working as an Assistant Professor at theGraduate School of Engineering
Science, Osaka University.

Prof. Tokuo Tsuji
He received his PhD degree from theUniversity of Tokyo in 2013. From
2017, he has been working as anAssociate Professor at the Graduate
School of Engineering Science,Osaka University.

He received his Doctoral degrees
from Kyushu University in 2005.
From 2016, he has been working as
an Associate Professor at Institute of
Science and Engineering, KanazawaUniversity.