Internationalof Computational
Intelligence
Systems
Journal ofJournal
Robotics,
Networking andArtificial Life
Vol. XX(Z);Month(Year),xx–yy6(4); March(2020),
pp.pp.
258–261
DOI: 10.1080/XXXXXXXXXXXXXX; ISSN
online
DOI: https://doi.org/10.2991/jrnal.k.200222.008;
ISSNXXXX–XXXX
2405-9021; eISSN
2352-6386
https://www.atlantis-press.com/journals/ijndc
https://www.atlantis-press.com/journals/jrnal

Research Article

User Interface and Motion Planner for Task Database
Tokuo Tsuji1,*, Natsuki Yamanobe2, Kei Ikeda3, Kensuke Harada2,3Institute of Science and Technology, Kanazawa University, Kadoma-cho, Kanazawa 920-1192, JapanIntelligent Systems Research Institute, National Institute of Advanced Industrial Science and Technology (AIST), 1-1-1 Umezono, Tsukuba 305-8568, Japan
3Graduate School of Engineering Science, Osaka University, 1-3 Machikaneyama, Toyonaka 560-8531, Japan
12ARTICLE INFOABSTRACT

Article History

In this paper, we present user interface modules for handing task motion data on robotic manipulation. The database covers
robot motion, human motion and object information. The authors especially present its data structure, data registration, datasearch, Graphical User Interface (GUI), and Application Programming Interface (API). The task motion data is registered using
vision sensor and GUI. The robot work motion is generated automatically with planning module and modification module.

Received 11 November 2019
Accepted 24 December 2019
Keywords
Robotic assembly
motion planningtask planning
database© 2020 The Authors. Published by Atlantis Press SARL.
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION
Most robots repeat a determined motion in mass production.
However, in the case of variable production, it is necessary to teacha new motion to robot according to changes in the product. It issaid that robot teaching and device setting costs are much larger
than the cost of the robot body. Therefore, frequently teachingrobot motions and changing devices according to product changes
will lead to a significant increase in cost.
Human flexibly realize a wide variety of tasks. It is considered thatthe experiences of various tasks are abstracted, conceptualized, andaccumulated as skills and knowledge. Then the skills and knowledge are applied to new tasks.
The authors have constructed a cloud database [1–4] that accumulates various information related to motions and shares amongrobots for motion generation. The database accumulates human
work motions and teaching data for robots, work procedure, andobject information such as the shape of the object, motion trajectories during work.
This research aims to construct the database for a robot adaptively
performing a task. Figure 1 shows the overview of its framework.
The database is composed of the object data and teaching motion
data. When a robot performs a novel task, a robot download andmodify the data so as to fit the novel task.

coordinates of the object and is independent of the configuration of robot. A motion can be shared in general robots. The
hierarchical description makes it possible for robots to reuseappropriate level of motion.
An assembly of a product is composed of multiple task elements
where each task element is composed of a set of motion sequence.
Therefore, the first, the second and the third layers are respectively
named as “work”, “task” and “action” layers. Since multiple objects
are related in a single action, two objects are named as follows. Theobject held by a robot is named as the “main object” and the object
assembled to the main object is named as the “sub object”. The
action data include the trajectory which is the time series of relative posture of the main object and the sub object. If the positionof the sub object changes, the motion is adapted using the relative
postures in the action data. The three layers are effective for reusing
motion. For example, the action “tightening screw” can be used inanother task data. The task data can also be used in another work.
Figure 2 shows an example of assembly of a toy airplane. The task
is “fixing the wing to the body” and composed of two actions: “putthe wing on the body”, and “put and rotate the screw”. At the firstaction, the main object is the upper wing and the sub object is theupper body. At the second action, the main object is the screw andthe sub object is the upper wing.
The following is a set of information included in the database:•• Motion strategy (A sequence of task/motion)

2. THE STRUCTURE OF THE DATABASE•• Human (Identified name etc.)

The structure of database is designed by considering versatility and reusability. The motion is described by the sequence of•• Robot (Name, type, hand)

Corresponding author. Email: tokuo-tsuji@se.kanazawa-u.ac.jp•• Object category*•• Object name

T. Tsuji et al. / Journal of Robotics, Networking and Artificial Life 6(4) 258–261

259•• Object shape•• Object parameters (Mass, CoG, material, stiffness, etc.)•• Grasping configuration (Main object)•• Fixture configuration (Sub object)•• Time series of the main object’s pose w.r.t the sub object•• Control method and its parameters•• Force information applied by the hand
The data can be obtained by several methods such as a humanmotion capture and a motion generation tool. The data is stored
in the YAML format where some examples are shown in Figure 2.
YAML is a human-readable data-serialization language.

Figure 3 | Search window for the database.

3. INTERFACES OF DATABASES
3.1. Motion Searching Function
Figure 3 shows the searching function of the motion databaseplugin. As keywords used for the search, we set the name of parts,
the category of parts, the feature of object data, and object’s weight.
Inputting keywords in the search window, we obtain the candidatesof objects and motion data. The Figures 3 and 4 show an example
where we search the keyword “shaft” and obtained the object model“shaft1” and “shaft2” and corresponding motion data.
Figure 4 | Search result of task database.

Figure 5 | Vision system for measuring object motion grasped by humanand robot motion generation using the motion data.
Figure 1 | Framework on motion generation of industrial robot fromdatabase where the database includes information on object dataand teaching motion data.

3.2. Object Trajectory of Human Motion
Vision sensor is used generating action data. The trajectory of an
object is measured by vision and stored to the database. In Figure 5,
an example is shown. A depth camera is used for measuring theposition of a plastic bottle. The trajectory of a plastic bottle grasped
by human is stored to the database as an action data. After a robot
plans the grasping position on the plastic bottle, the robot moves
the same trajectory of human.

3.3. Graphic User Interface

Figure 2 | An example of task for toy assembly.

Graphic user interface of Choreonoid is implemented for registering the action data in the database. The main object and thesub object can be selected by clicking a button on the window.
The object position of the main object can be operated by mouse
motion and be set with the value. The trajectory of the main object
is saved as YAML file.

260

T. Tsuji et al. / Journal of Robotics, Networking and Artificial Life 6(4) 258–261

The grasp position can be registered manually using the graphicinterface. The robot motion can be generated according to themotion of the object.

4. MOTION GENERATION USING DATABASE
4.1. Motion Planning for Generating
Task Motion

Figure 8 | Screwing units used in the experiment. (a) Screwing unit 1.
(b) Screwing unit 2.

The authors have implemented motion planner. Robot motion ofthe task data is generated by using grasp planning and path planning [5]. The hand position on the object position is determined bygrasp planning. The path planning is used for connecting the ends
of multiple actions. After once grasp position of the main object isdetermined, the robot motion is generated automatically according
to the motion of the main object. If the hand collides the sub object
during an action, grasp position is planned again.

4.2. Motion Modification
Let us explain the feature of objects defined in this research.
Figure 6 shows three examples of features. In case of a bolt, diameter, pitch and length are the features. In case of a gear, diameter andthe number of teeth are the features. In case of a L shaped object,
the dimension of the object is the feature.
Figure 7 shows the modification module of motion data.
The inputs to this module are the features of the new main and subobjects and the existing motion data of similar but different object.
An example of the main and sub objects, we can consider the bolt
and nut. As an example of new objects, we can consider the newbolt with different pitch and length. By considering the differenceof feature between objects, we modify the motion data and make it
fit to the newly appeared objects.

Figure 9 | Experimental result of motion modification. (a) The robot
grasped the screwing unit. (b) The robot put the screwing unit on the baseplate. (c–f) The robot screwed the screwing unit.

In our experiment, we used two screwing units with different
features as shown in Figure 8. The first unit has the screw length
12 mm and base thickness 15 mm. The second unit has the screw
length 18 mm and base thickness 25 mm. We prepared the motion
data to assemble the first unit and modify this motion data to fit to
the second unit.
Figure 9 shows the experimental results. As shown in this figure,the robot adaptively generates the motion data and generates thescrewing motion of different screwing unit.

5. CONCLUSION
Figure 6 | Definition of features. (a) Bolt. (b) Gear. (c) L shaped object.

Task database and the interfaces are implemented for robotic
motion generation. The task data is independent with the configuration of robots. We designed three layers for task data. The
data of each layer can be reused. The GUI and vision sensor is usedfor storing task data. The robot motion is generated with search
module, planner module, and modification module.

CONFLICTS OF INTEREST
The authors declare they have no conflicts of interest.

REFERENCES
Figure 7 | Modification module of motion data.

[1]

N. Yamanobe, T. Tsuji, K. Harada, K. Nagata, R. Hanai,
W. Wan, et al., Construction of cloud database for task motion

T. Tsuji et al. / Journal of Robotics, Networking and Artificial Life 6(4) 258–261[2]

generation, Annual Conference on the Robotics Society ofJapan, The Robotics Society of Japan, Yamagata, 1B2-07, 2016.
K. Harada, K. Nakayama, W. Wan, K. Nagata, N. Yamanobe,
I.G. Ramirez-Alpizar, Tool exchangeable grasp/assembly planner,
Proceedings of the International Conference on Intelligent
Autono­mous Systems and Computing, Springer, Cham, 2019,
pp. 1–13.

[3]
[4]
[5]

261

W. Wan, K. Harada, Developing and comparing single-arm anddual-arm regrasp, IEEE Robot. Autom. Lett. 1 (2016), 243–250.
K. Harada, N. Yamanobe, W. Wan, K. Nagata, I.G. RamirezAlpizar, T. Tsuji, Motion-data driven grasp/assembly planner,
J. Robot. Netw. Artif. Life 5 (2019), 232–235.
T. Siméon, J.P. Laumond, J. Cortés, A. Sahbani, Manipulation
planning with probabilistic roadmaps, Int. J. Robot. Res. 23(2004), 729–746.

AUTHORS INTRODUCTION
Prof. Tokuo Tsuji

Mr. Kei Ikeda
He received his Doctoral degrees fromKyushu University in 2005. From 2016, hehas been working as an Associate Professor
at Institute of Science and Engineering,
Kanazawa University.

Dr. Natsuki Yamanobe
She received the PhD Degree from theUniversity of Tokyo in 2007. She is currently a Senior Research Scientist withIntelligent Systems Research Instituteof Advanced Industrial Science andTechnology (AIST).

He received his Master degrees fromGraduate School of Engineering Science,Osaka University in 2019.

Prof. Kensuke HaradaHe received his Doctoral degrees inMechanical Engineering from Kyoto
University in 1997. From 2016, he has
been working as a Professor at Graduate
School of Engineering Science, OsakaUniversity.