Journal of Robotics, Networking and Artificial Life
Vol. 5(4); March (2019), pp. 283–288
DOI: https://doi.org/10.2991/jrnal.k.190402.002; ISSN 2405-9021 print; 2352-6386 online
https://www.atlantis-press.com/journals/jrnal

Using Importance Ranks to Derive Suitable Timing of VisualSensing in Manipulation Task Containing Error Recovery
Akira Nakamura1,*, Kazuyuki Nagata1, Kensuke Harada2, Yukiyasu Domae11
Automation Research Team, Artificial Intelligence Research Center, National Institute of Advanced Industrial Science and Technology (AIST),
Koto-ku, Tokyo 135-0064 Japan
2
Robotic Manipulation Research Group, Systems Innovation Department, Graduate School of Engineering Science, Osaka University, Toyonaka 560-8531, Japan

ARTICLE INFOABSTRACT

Article History

In general, a manipulation task can be composed of many skill primitives. Therefore, it is desirable to carry out plural visual
sensing in most skill primitives; however, performing sensing all the time is difficult. In this paper, we propose the addition ofimportance ranks to the attribute of skill primitives in order to derive a suitable timing for performing sensing. Furthermore, weshow that the skill primitives distinguished by their high importance ranks considerably correlates with error recovery.

Received 12 October 2018
Accepted 14 November 2018
Keywords
Manipulation skill
importance rankerror recovery
error classification
task stratification© 2019 The Authors. Published by Atlantis Press SARL.
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION
A robot came to play an active role in many fields from this century. In the robot manipulator, studies came to be performed forthe manipulation robot which played an active part in various categories, similarly. Therefore, the use of robot manipulator worked
in several domains such as power plants for maintenance and manufacturing factories for production (Figure 1) has been researchedupon considerably [1–3].
These manipulation tasks tend to be complex. We have considered the technique that the complicated tasks could be carriedout as easy as possible by using the concept called the skills[1–3]. The “skills” are technique to resolve one task into thesequence of motion primitives by making reference to person’s
operational execution. If each motion primitive is feasible using
a manipulator, it is possible to carry out the task using the robot
in many cases.
The robot causes failure as well as a person, too. In late years
practice of the difficult task performed by a robot is often
demanded, and studies of error recovery have been also proposed [4–7]. However, most are studies of recovery whichcorrects difference in trajectory of manipulator. The study oferror recovery in which restoration from failure is performed
is hardly done.
We studied the technique of recovery for various error classes
from a small error to big failure [8,9]. About small-scale trajectory
Corresponding author. Email: a-nakamura@aist.go.jp*

adjustment, error recovery is carried out by a forward correction
process. However, it is difficult to correct big error with the methodof the trajectory adjustment. In the big failure, error recovery is
performed by a backward correction process which returns to theprevious step.
A task performed by manipulator consists of several skill primitives,and it is desirable that visual sensing is performed in most of skill
primitives. However, performing sensing all the time is difficult
because of some restrictions. In this paper, we propose to distinguish skill primitives according to their importance ranks to selectthe primitives with suitable timing of visual sensing. Furthermore,
we show that the skill primitive distinguished according to a highrank is deeply related to the skill primitive in which error recovery is considered in advance. In addition, visual sensing is assumed
to be performed precisely to stop movement before and after skill
primitives are executed.
In the next section, skills performed by manipulator and the stratification of manipulation tasks are explained. The procedure ofmarking manipulation skill by the importance ranks is proposed
in Section 3, and a concept of error recovery and a method of error
classification are described in Section 4. Section 5 presents theeffectiveness of applying importance ranks to skill primitives using
examples of repacking tasks.

2. STRATIFICATION OF TASKSIn this section, fundamental matter about manipulation skill is
explained, the details can be found in Hasegawa et al. [1].

A. Nakamura et al. / Journal of Robotics, Networking and Artificial Life 5(4) 283–288

284

Figure 1 | Maintenance robot.

Figure 2 | Manipulation hierarchy.

2.1. Concept of Skills
We observed and analyzed human behavior in motion, and it wasderived that movement of one task consists of sequence of plural
meaningful primitives of motion. We named such primitives of
movement “skills” [1]. For example, there are move-to-touch skill,rotate-to-level skill, and rotate-to-insert skill. In assembly task, aparticular task can consist of these skill primitives. Therefore, theseskills mean fundamental skills. There are various methods of theresolution to the sequence with respect to the task. We proposeda method using fluctuations of number of contacting points in a
motion primitive of skill [2]. Furthermore, a lot of skills can bederived by observing and analyzing human behavior in motion.

We researched the timing appropriate to visual sensing in skill
sequences with error recovery in Nakamura et al. [10]. It is efficientthat a visual sensing is performed by considering the difficulty inaccomplishing a skill primitive, as shown in Case 4 of Section 4 ofNakamura et al. [10]. The degree of necessity of a visual sensing
is derived based on the consideration of the following idea in thispaper. An importance rank derived from the degree of the necessity of sensing was added to the attribute of each skill primitive.
Moreover, the priority of performing visual sensing in each skill
primitive can be derived by deciding the whole value of the importance rank in the total task. To decide the value of an importancerank, an operator’s teaching is allowed, although it is desirable todecide the value automatically.
Let us consider importance ranks with a transfer task (Figure 3)
and grasping and gripper-open tasks (Figure 4) as an example. In
the transfer task, it is designated that a skill primitive in which strictvisual sensing is needed has first priority and that a skill primitive
with grasping one or more objects has second priority (Table 1). In
the grasp and gripper-open skills, the skill primitive in which thecontact state, with the grasped object, changes has the top priority(Table 2). The timing of efficient visual sensing in a total task can bederived by considering an importance rank in each skill primitive
according to Tables 1 and 2.

4. ERROR RECOVERY USING
CLASSIFICATIONThe real performance may differ from the ideal performance. Errors
occasionally happen by various reasons in actual manipulation.
Our ideas of classifying errors and a stream of error recovery processwith hierarchy of task are described in this section (for more details,
please refer to Nakamura and Kotoku [8]).

2.2. Stratification of Tasks
Figure 3 | Transfer task.

Figure 2 shows a hierarchy of manipulation tasks [8]. If the servo
layer is disregarded, the skill layer means the lowest layer marked
the task(0) layer. This skill layer is composed of elementary motion
primitives. Further, the task(1) layer is a layer on one of task(0) layer.
About the layer above it, the task(i + 1) layer is located one tier above
the task(i) layer, similarly. And, the top layer is task(max).

3. ADDITION OF IMPORTANCE RANKS IN
SKILL PRIMITIVES TO DERIVE SUITABLE
TIMING FOR VISUAL SENSING
Ideally, visual sensing should be performed twice in each skill
primitive to derive the exact posture of the objects before e­ xecution
and confirm the last state of objects after execution. However, in
actuality, it is difficult to carry out sensing at all sensing points
because of the restrictions of time and hardware.

Figure 4 | Grasping and gripper-open task.
Table 1 | Importance rank of transfer skill
Important rank Conditions
HT (1)HT (2)
LT (= HT (3))

The target position and orientation are demanded strictly.
The hand is grasping one or more objects.
Other than the above.

HT (1) > HT (2) > LT (= HT (3)).

A. Nakamura et al. / Journal of Robotics, Networking and Artificial Life 5(4) 283–288

Table 2 | Importance rank of grasp and gripper-open skills
Important rank

Conditions

H (1)
LG (= HG (2))

The contact state with the grasped object changes.
Other than the above.

G

HG (1) > LG (= HG (2)).

285

Nakamura and Kotoku [8]. Error recovery is executed througha backward correction process in this process. At the step of confirmation in each skill primitive task(0)(i0), the outcome is decided as correct or error by using an automated-means or a human judgment.
Error recovery is practiced based on the following classifications.
Class 1: It is judged that the error is an execution error, and task(1)(i1)
is practiced repeatedly with no correction of the parameters.
Class 2: It is decided that the error is a planning error, and task(1)(i1)
is carried out repeatedly with correcting planning parameters.
Class 3: It is decided that the error is a modeling error, and task(1)(i1)
is carried out repeatedly with correction of modeling parameters.
Class T(1): It is decided that the error is a sensing error, and task(1)(i1)
is carried out repeatedly using different sensing parameters.
Class T(2): task(2)(i2) is practiced repeatedly after the required changes
are performed and goes back to the initial position one tier above
layer task(1)(i1).
::Class T(max): task(max)(imax) is carried out repeatedly after necessarychanges are done and goes back to the initial position at (max − 1)tier above layer task(1)(i1).
Class T(max + 1): When too many changes are judged to be necessary,
the process being carried out is aborted.

5. IMPORTANCE RANKS IN SKILLPRIMITIVES WITH ILLUSTRATIONOF REPACKING TASKS

Figure 5 | Process flow with error recovery.

4.1. Classifying Errors
The reasons of failures in practice of manipulation are attributable to several types of errors. We grouped the states of errors into
plural classes, namely execution, planning, modeling, and sensing,according to causes [8].

Let us consider manipulation robots playing an active role in physical distribution. Then, take into account tasks of repacking objects
from a large box into a small box at a distribution center and theprocesses of error recovery for the robot tasks. Here, a representative task of lifting up an object pointed out by using the parallel
jaw gripper on the robot [11] is selected, and an importance rank is
considered for each skill primitive according to Section 3.
The principal errors in the repacking tasks are described in thissection. And the process of recovery is explained with respect to eacherror. Moreover, we take into account the relation between typicalerrors of the task and skill primitives with high importance degrees.

5.1. Sequence of Repacking Tasks

Merely remedying the origins of these errors does not always reachperfection of recovery. When the environment of robotic work changes
by the error very much, it may be needed to go back to a former step.

Let us consider the task of picking up and placing objects such as
Plastic bottles by using a manipulation robot with a gripper. The
processes are shown in Figure 6. Figure 7 shows the flow of the task
sequence if there is no error. These motion primitives are as follows.

4.2. Recovery Based on Classifying Errors(Skill1) Move-to-approach: Moving to the starting point of theapproach motion.

A generalized flow showing process of stratified tasks taking
error recovery into consideration was proposed in Nakamura andKotoku [8]. Figure 5 illustrates the central portion of Figure 10 in(Skill2) Pre-grasp: Opening to grasp the object.
(Skill3) Approach: Moving at low speed until arriving at the grasping point.

286

A. Nakamura et al. / Journal of Robotics, Networking and Artificial Life 5(4) 283–288

5.2. Importance Ranks of Skills
in Repacking Tasks
Here, we take into consideration the importance ranks of skills in
the repacking task. An importance rank of each skill primitive isderived (Section 3) as follows:
Skill1: LT,		Skill2: LG,		 Skill3: HT (1),Skill4: HG (1),Skill5: HT (2),Skill6: HT (2),Skill7: HT (2),Skill8: HT (2),Skill9: HT (1),Skill10: HG (1), Skill11: LT,		Skill12: LT.

Let us suppose that the following relation (1) exists among theimportance ranks.
	   H T (1) > H G (1) > H T (2) > L (= LT = LG ) (1)Figure 6 | Picking and placing task using a gripper.

These skill primitives, in turn, can be arranged as follows based on
the importance ranks with inequality (1).
First priority: Skill3, Skill9.
Second priority: Skill4, Skill10.
Third priority: Skill5, Skill6, Skill7, Skill8.
Fourth priority: Skill1, Skill2, Skill11, Skill12.

5.3. Candidate Errors in Repacking Tasks
Next, we will consider the representative types of errors occurredin the tasks.
(1) Errors when grasping and lifting.
(1a)	Error about height; the robotic hand cannot attain to abottle in Skill3 (Figure 8a).
(1b)	Error of movement in parallel with the bottom of the box;
the open gripper of hand is not put around the cap of abottle in Skill3 (Figure 8b).
Figure 7 | Task sequence of picking and placing a bottle.

(Skill4) Grasp: Grasping the target object.
(Skill5) Lift-up: Lifting the grasped object.

(1c)	Error in which a bottle cannot be pulled up owing to tightpacking in a box in Skill5 (Figure 8c).
(2) Errors when carrying an object.
(2a)	Error in which a bottle is dropped in Skilli (i = 6 – 8)
(Figure 9).

(Skill6) Departure: Moving to a particular point in a reference frameof a box from which the object is taken.
(Skill7) Move-between-reference-frames: Moving to a particular point
in a reference frame of a box, in which the object must be placed.
(Skill8) Move-to-destination: Moving to the destination point.
(Skill9) Lower: Bringing down the grasped object.
(Skill10) Hand-open: Opening to place the object.
(Skill11) Leave: Moving to the safe area.
(Skill12) Home: Going back to the initial point for the continuous task.

Figure 8 | Errors when grasping and lifting.

A. Nakamura et al. / Journal of Robotics, Networking and Artificial Life 5(4) 283–288

287(2) Cases (1c) and (3d)		 Class 1: Execution is simply repeated.
		 Class 2: The process is restarted from the planning step after
the revision of the planning methods such as the motion withslightly shaking in lifting or lowering.
		 Class 3: The process is executed repeatedly from the modelingstep after modification of the geometric models representing
the bottles or box.
		Class T(1): [The same as (1)].
Figure 9 | Errors when carrying.

(3) Case (2a)		 In most cases, the task of picking up a bottle in the box ischanged to the task of lifting the bottle that has dropped tothe bottom of the transition path; Class T(2) is chosen. The skill
sequence of this task is executed from the beginning.
(4) Cases (3b) and (3c)Figure 10 | Errors when packing.

(3) Errors when packing.
(3a) 	Error of movement in parallel with the bottom of the box;
a gap exists between the target position in Skill9. No movement of another bottle is performed (Figure 10a).
(3b)	Error in which the bottle grasped by the gripper pushes
the neighboring bottles in the box in Skill9 (Figure 10b).
(3c)	Error in which both (3a) and (3b) occur simultaneously(Figure 10c). Gaps occur both between the positions of
the grasped bottle and surrounding bottles.
(3d)	Error in which Skill9 is stopped before completion(Figure 10d) because the available space is too small.

5.4. Error-recovery Processes
The recovery process from each error is explained in this section.
The following numbers and letters in brackets coincide with thosedescribing the candidate errors in the previous section. The corrections were carried out based on definite or indefinite causes derived
from the error classification, and the process restarts from the corresponding step.
(1) Cases (1a), (1b), and (3a)		 Class 1: Execution is simply repeated.
		 Class 2: The process is executed repeatedly from the planning step after the planning parameters are modified fromthe gap.
		 Class 3: The process is executed repeatedly from the modelingstep after modification of the geometric models of the bottles.
		Class T(1): The process is executed repeatedly from the sensing
step after adjustment of the coordinate system.

		 The task according to scenario is finished, and the process
moves to the initial position of the task sequence at one point.
Next, the task in which the bottle should be moved to the correct position is executed. If necessary to move many bottles, it
is desirable to investigate whether or not the paths of correction converge efficiently.

5.5. Sensing Time in Repacking Tasks
We considered typical errors in 5.3 and recovery processes of errors
in 5.4. Furthermore, we introduced the importance ranks of skill
primitives in 5.2 so that Skill3 (Approach skill) and Skill9 (Lower
skill) became high priority tasks. Then, these are included in skill
primitives treated in 5.3 and 5.4. That is, Skill3 is relevant to (1a) and(1b), and Skill9 is relevant to (3a)–(3d). Thus, the skill primitive inwhich error recovery is almost considered has a deep relation withthe skill primitive in which visual sensing is needed. Moreover, skill
primitives that are important alongside the above-mentioned skills
are grasp and gripper-open skills, i.e., Skill4 and Skill10, respectively,
and transfer skills: Skill5–Skill8. Especially, Skill5 and Skill7 are skill
primitives with high importance ranks, as in Cases (1) and (2) in
5.3, respectively.
When we take into account error recovery processes in the task
composed of several skill primitives, the total number of skill
primitives in which our error-recovery procedure [8,9] can beapplied becomes very larger. It is difficult to consider those all
the recovery processes, since great effort is necessary for planning of these recovery processes. When using the importanceranks proposed in Section 3, it becomes relatively easy to choiceskill primitives, for which it is suitable to consider an errorrecovery procedure.

6. CONCLUSION
In general, a manipulation task can be composed of many skill
primitives. It is desirable to perform plural visual sensing in mostskill primitives as geometric modeling and task planning are
carried out before execution, and task achievement is confirmed
after execution. However, the performance of sensing at all sensing

A. Nakamura et al. / Journal of Robotics, Networking and Artificial Life 5(4) 283–288

288

points is difficult because of the restrictions of time and hardware.
In this paper, we proposed the addition of importance ranks to theattribute of skill primitives to derive suitable timing to perform
visual sensing. The use of the proposed method eases the selectionof skill primitives in which visual sensing should be performed.
Furthermore, we showed that skill primitives distinguished according to their high importance ranks considerably correlate withthose in which error recovery is considered in advance. This characteristic can be used for selecting skill primitives in which error
recovery planning is carried out in advance.
In the future, we will apply our proposed procedure to sensing,modeling, planning, and execution in many kinds of tasks using
robotic hands. We will attempt to apply our procedure of error
recovery to tasks executed by actual robots.

CONFLICTS OF INTEREST
There is no conflict of interest.

REFERENCES[1] T. Hasegawa, T. Suehiro, K. Takase, A model-based manipulation system with skill-based execution, IEEE Transactions on
Robotics and Automation, IEEE, 1992, pp. 535–544.
[2] A. Nakamura, T. Ogasawara, T. Suehiro, H. Tsukune, Skill-based
backprojection for fine motion planning, Proceedings of IEEE/
RSJ International Conference on Intelligent Robots and Systems(IROS’96), IEEE, Osaka, Japan, Japan, 1996, pp. 526–533.
[3] A. Nakamura, T. Ogasawara, K. Kitagaki, T. Suehiro, Using
robust and simplified geometric models in skill-based manipulation, Proceedings 2001 IEEE/RSJ International Conference onIntelligent Robots and Systems. Expanding the Societal Role ofRobotics in the Next Millennium (Cat. No. 01CH37180), IEEE,
Maui, HI, USA, USA, 2001, pp. 139–145.
[4] B.R. Donald, Planning multi-step error detection and recovery
strategies, Int. J. Robot. Res. 9 (1990), 3–60.
[5] C.M. Baydar, K. Saitou, Off-line error prediction, diagnosis andrecovery using virtual assembly systems, Proceedings 2001 ICRA.
IEEE International Conference on Robotics and Automation(Cat. No. 01CH37164), IEEE, Seoul, South Korea, South Korea,
2001, pp. 818–823.
[6] M. Scheutz, J. Kramer, Reflection and reasoning mechanisms for
failure detection and recovery in a distributed robotic architecture for complex robots, Proceedings 2007 IEEE International
Conference on Robotics and Automation, IEEE, Roma, Italy,
2007, pp. 3699–3704.
[7] P. Pastor, M. Kalakrishnan, S. Chitta, E. Theodorou, S. Schaal,
Skill learning and task outcome prediction for manipulation, 2011
IEEE International Conference on Robotics and Automation,
IEEE, Shanghai, China, 2011, pp. 3828–3834.
[8] A. Nakamura, T. Kotoku, Systematization of error recovery inskill-based manipulation, Artif. Life Robot. 14 (2009), 203–208.
[9] A. Nakamura, K. Nagata, K. Harada, N. Yamanobe, T. Tsuji,
T. Foissotte, et al., Error recovery using task stratification anderror classification for manipulation robots in various fields,2013 IEEE/RSJ International Conference on Intelligent Robots
and Systems, IEEE, Tokyo, Japan, 2013, pp. 3535–3542.
[10] A. Nakamura, K. Nagata, K. Harada, N. Yamanobe, The suitabletiming of visual sensing in error recovery using task stratificationand error classification, J. Robot. Netw. Artif. Life 4 (2017), 134–137.
[11] A. Nakamura, K. Nagata, K. Harada, N. Yamanobe, Error recovery using task stratification and error classification for manipulation robots in physical distribution, Trans. Control Mech. Syst. 3(2014), 122–131.

Authors Introduction
Dr. Akira Nakamura

Prof. Kensuke Harada
He received the PhD degree in ElectricalEngineering from Keio University in
1991. He is currently a research staff at
National Institute of Advanced Industrial
Science and Technology (AIST). His
research interests include robot planning,
vision and control system.

He received his Doctoral degrees in
Mechanical Engineering from Kyoto
University in 1997. From 2016, he has
been working as a Professor at Graduate
School of Engineering Science, Osaka
University.

Dr. Yukiyasu Domae

Dr. Kazuyuki Nagata
He received his PhD in Engineeringfrom Tohoku University in 1999. He iscurrently a Senior Research Scientist
with Intelligent Systems ResearchInstitute of Advanced Industrial Scienceand Technology (AIST).

He received his PhD degree from
Hokkaido University in 2014. He iscurrently a team leader with Artificial
Intelligent Research Center (AIRC),
National Institute of Advanced Industrial
Science and Technology (AIST).