The 2009 IEEE/RSJ International Conference onIntelligent Robots and Systems
October 11-15, 2009 St. Louis, USA

Segmentation method of human manipulation task based on
measurement of force imposed by a human hand on a grasped object
Kazuya Matsuo, Kouji Murakami, Tsutomu Hasegawa, Kenji Tahara, and Ryo Kurazume
Abstract— This paper proposes a segmentation method of
human manipulation task based on measurement of contact
force imposed by a human hand on a grasped object. We
define an index measure for segmenting a human manipulation
task into primitives. The indices are calculated from the set ofthe contact forces measured at all the contact points during amanipulation task. Then, we apply the EM algorithm to the setof the indices in order to segment the manipulation task into
primitives. These primitives are mapped onto the robotic handto impose appropriate contact forces on a grasped object. In
the experiments, manipulation tasks performed in daily humanlife have been successfully segmented.

I. Introduction
A multi-jointed multi-fingered robotic hand is a potentiallydexterous hand like a human hand [1]. However, it is very
diﬃcult for us to manually and directly write a motion
program of multiple fingers moving synchronously and cooperatively to execute a task. Programming-by-demonstration[2][3] is often used in order to overcome this diﬃculty. In
the programming-by-demonstration, a motion program forthe robotic hand is generated based on the motion data ofactual tasks executed by a human.
Direct mapping of joint angle trajectories of the human
hand onto the robotic hand has been reported [4] forprogramming-by-demonstration. However, the direct mapping can not be applied to the robotic hand whose configuration is diﬀerent from that of the human hand. Anothermethod is mapping of positions of human fingertips onto
the robotic hand [5]. This method generates joint angle
trajectories of the robotic hand from the positions of thehuman fingertips by solving inverse kinematics of the robotic
hand. This method may easily fail in keeping stable graspof an object because it does not assure stable grasping
force. Contact information between the palm and the grasped
object is not mapped onto the robotic hand by this position
mapping.
In order to overcome these problems, higher level mappingof symbolized manipulation primitives onto the robotic hand
has been proposed. This method recognizes a task fromthe motion data of the human hand: continuous human
hand motion is segmented, and symbolized according to themeaning of a particular motion in the context of the task.
Each symbol represents a particular manipulation primitive.
The task is represented as a sequence of symbols. The robotic
hand will therefore be able to autonomously perform various
K. Matsuo, K. Murakami, T. Hasegawa, K. Tahara, and R. Kurazume
are with Faculty of Information Science and Electrical Engineering,
Kyushu University, 744 Motooka, Nishi-Ku, Fukuoka, 819-0395, JAPAN.

matsuo@irvs.is.kyushu-u.ac.jp

978-1-4244-3804-4/09/$25.00 ©2009 IEEEtasks by executing a sequence of corresponding manipulation
primitives if the symbols representing manipulation primitives are recognized. Bernardin et al. [6] presented a method
to recognize continuously executed sequences of graspinggestures.
Particular shapes of grasps by the human hand are usuallyused as keys to recognize manipulation primitives in related
work on human motion recognition [6][7][8][9][10]. Classified particular shapes of the human hand are called grasp
types. Cutkosky proposed a grasp taxonomy [11] consistingof 16 hand shapes used by humans working with tools andmetal parts. Kamakura, an occupational therapist, proposeda grasp taxonomy [12] consisting of 14 hand shapes usedin daily life. These grasp taxonomies are classified based on
observations of particular shapes of the human hand.
However, the grasp types do not contain force information
imposed on the object grasped by the human hand. So,
the robotic hand may not impose the appropriate force
on the grasped object when a mapping of shape-based
grasp type is applied. Required force information consists ofmagnitude and direction of the forces at the contact points.
The force information imposed by the human hand is an
important property to be mapped onto the robotic hand in the
programming-by-demonstration. The robotic hand imposes
the appropriate force on the grasped object when a mappingof force-based primitive is applied.
This paper proposes a segmentation method of humanmanipulation task based on contact force imposed on an
object grasped by a human hand. The remainder of the paper
is organized as follows. Section II explains the segmentation
method of manipulation task based on force measurement at
the contact points. The designed device to measure forces
imposed by a human hand is described in Section III.
Section IV describes the experiments for segmenting humanmotion of a screw manipulation task and a writing task. The
conclusions of the present study and the future work arepresented in Section V.
II. Segmentation Method of Manipulation Task
We propose a segmentation method of human manipulation task. First, we measure contact force imposed on an
object grasped by a human hand during a manipulation
task. Then, we define an index measure for segmenting
the manipulation task. The index is calculated from theset of the contact forces at all the contact points. We callthe index contact force index. We assume that a humanmanipulation task is composed of primitives: each of them ischaracterized by particular distribution pattern of force and1767torque at contacts on the object with the hand so that theappropriate interaction and constraint are generated to satisfy
the task requirements. The time series data of contact force
are segmented to be primitives by applying a cluster analysis
to the set of the contact force indices.
A. Analyzing grasping force of a human hand on an object
A human worker adjusts the magnitude and direction ofcontact force when he grasps an object so that suitable
constraints and external force are generated to perform theparticular task. Diﬀerent force is required for the same object
if the task context is diﬀerent. For example, the grasping
force of a hammer when the worker carries it is diﬀerent fromthe force required when he hits a nail with the hammer eventhough the grasping shape of the hand is similar. Strategy toadjust the grasping force may be acquired through experienceand learning of the worker. It would be possible to recognize
the task primitive by analyzing the time series of grasping
force even though the grasping shape of the hand does notchange.
It is very diﬃcult to map all the forces of a human hand
imposed at all the contact points onto a robotic hand. Thus,
we calculate the contact force index to evaluate and recognize
the requirements of grasp which in turn leads to segment the
manipulation task into primitives. Then, the primitives are
mapped to the actual robotic hand.

3) The grasping force applied to the object by the human
hand is composed of the contact forces at all thecontact points.
We explain the formalization to calculate the contact force
index. Firstly, we construct two matrices: W f is the matrix
consisting of the contact force vectors at all the contact points
and W t is the matrix consisting of the torque vectors at all
the contact points. Two matrices are expressed as:(1)
W f = f1 f2 · · · f N ∈ 3×N ,t i = pi × fi (1 ≤ i ≤ N),
W t = λt 1 λt 2 · · · λt N ∈ 3×N ,(3)where fi is the contact force vector at the i th contact point, t iis the torque vector at the i th contact point, pi is the position
vector of the i th contact point, λ is a torque multiplier that
relates units of torque to units of force, and N is the numberof contact points (Fig. 1). fi and pi are expressed with respect
to a same coordinate system. We have enforced |λt i | ≤ | fi |by choosing λ = 1r , where r is the maximum distance of thecontact points from the torque origin. This will ensure thatthe contact force index is independent of object scale.

C1

B. Contact force index
Many works have been reported on the contact force
imposed on an object grasped by a robotic hand. Forceclosure [13] is often used to characterize the grasp stability.
The force closure means any wrench can be expressed bya positive combination of contact point wrenches. A robotic
hand performing the force closure can influence the object
such that any external disturbance can be nullified. Graspquality [14] is an index for evaluating the stability of a grasp.
Miller and Allen computed the grasp quality from a shape ofa wrench space. These are indices for evaluating the stabilityof a grasp performed by a robotic hand. Thus, these are
not suitable for an index for extracting primitives from amanipulation task performed by a human hand.
We propose the contact force index: an index for extracting
primitives from a manipulation task performed by a humanhand. The contact force index is calculated from the shapesof two convex hulls formed by the contact forces and thetorques at all the contact points. The contact force index
is suitable for mapping onto a robotic hand because themapping using the contact force index does not constrain thenumbers and the positions of contact points on the object.
So, a robotic hand is able to apply the precision grasp to thismapping. We formulate a method for calculating the contact
force index on the assumptions as follows:
1) A contact between a human hand and a grasped object
is the point contact with friction.
2) The normal direction of a contact force at each contact
point depends upon the contacting surface of theobject.

(2)

p11

f1

O

f2

p22

C
C2
Fig. 1. Defined vectors to calculate the contact index: fi is the contact
force vector at the i th contact point, pi is the position vector of the i thcontact point, and Ci is the i th contact point (i = 1, 2).

Secondly, we calculate the singular values of W f and W t inorder to investigate the shapes of two convex hulls formedby the total fi and the total t i respectively. The singularvalues of W f and W t represent the axes of ellipsoids whichapproximate the two convex hulls respectively. The singularvalues of W f and W t are calculated as follows:
W f = U f Σ f V Tf ,
Σ f = diag(σ f 1 , σ f 2 , σ f 3 )(σ f 1 ≥ σ f 2 ≥ σ f 3 ≥ 0),

W t = U t Σ t V Tt ,
Σ t = diag(σt1 , σt2 , σt3 )(σt1 ≥ σt2 ≥ σt3 ≥ 0),

(4)
(5)
(6)
(7)

where σ j ( j = f 1, f 2, f 3, t1, t2, t3) are the singular values,
U k (k = f, t) are 3-by-3 orthogonal matrices, and V k are
N-by-N orthogonal matrices.

1768

Lastly, the contact force index (CFI) is composed of thecalculated 6 singular values.
(8)
CFI = σ f 1 σ f 2 σ f 3 σt1 σt2 σt3 .
If manipulation primitives segmented by using the contact
force index are mapped onto a robotic hand, we can decide
the appropriate contact force imposed by the robotic hand.
The magnitude of the appropriate contact force is decided
based on the singular values. The direction of the appropriate
contact force is decided based on U f , V f , U t , and V t .

We attach the pressure sensors onto the surface of theobject to be manipulated. We have developed sensing device
as shown in Fig. 3. Total of 60 pressure sensors (Flexi Force,
Nitta Corp.) are attached to the surface of the device. The
device is composed of a brass 6-sided regular prism and awooden cone on its top. Ten pressure sensors are installed oneach side face of the 6-sided prism. The mass of the device
is 253[g].
The size of the device and the placement of the installed
60 pressure sensors are shown in Fig. 4. The orange circles
in Fig. 4 indicate pressure sensors distributed on the device.

C. Segmentation of manipulation task into primitives byapplying a cluster analysis
We apply a cluster analysis to the set of the contact
force indices in order to segment a manipulation task into
primitives (Fig. 2). We employ the expectation-maximization(EM) algorithm [15] for this cluster analysis. The EM
algorithm segments data into the specified number of clusters
under the assumption that each cluster shows a normal
distribution. Each contact force index calculated from contact
forces includes the force measurement error. We suppose thatthe distribution of the force measurement error is the normal
distribution. Thus, we use the EM algorithm.

EM algorithm
time

The contact
force index

ð1

ð2

1st cluster

2nd cluster

Fig. 4. Size of the contact force sensing
device and placement of the 60 pressure sensors.
Fig. 3.
Photograph of thecontact force sensing device.

A photograph of the sensing part of the pressure sensor isshown in Fig. 5. The sensing part has a circular shape whose
diameter is 9.5[mm], and the thickness of the sensing part is
0.2[mm]. The specifications of the pressure sensor are shownin TABLE I. The pressure sensor measures normal forces toitself.

3rd cluster

ð3

9.5

Fig. 2.
Cluster analysis to the set of the contact force indices forsegmentation of manipulation task into primitives.

10.5

Fig. 5.

Each of clusters segmented by the EM algorithm is treatedas a manipulation primitive. We arbitrarily select the numberof the extracted primitives by choosing the specified numberof the clusters.

Photograph of the sensing part of the pressure sensor.

TABLE I
Specifications of the Pressure Sensor
Maximum capacity
Resistance under no-load
Resistance under maximum-loadHysteresis
Linearity

III. Device for Measuring Contact Force
To implement our method, we need to measure contact
force imposed by a human hand on a grasped object. One
direct way would be to attach pressure sensors on the contacting surface of the human hand. Discreet miniature pressure
sensors are still too large and thick to attach to the human
hand surface. It would obstruct natural motion of human
hand during object manipulation. Thin film-type pressure
sensor easily deforms itself according to the finger motion if
it is attached to the finger surface, thus it outputs erroneouscontact pressure. In addition, joint angles of fingers must be
measured to obtain contact force direction with respect tothe 3D task coordinate system. Joint angle measurement of
human hand is diﬃcult. Error of joint angle measurement
badly aﬀects the performance of our method.

[mm]

110[N]
10[MΩ]
20[kΩ] ± 15%< 4.5%< ± 5%

The contact force is decomposed into the normal force
and the tangential force. Commercially available thin film
sensor measures the normal force only. So, we investigate
the results of our method in the case of measuring only thenormal force in this report. To analyze the influence of thetangential components is future work.
IV. Experiments
We applied our method to two diﬀerent manipulation tasks
performed in daily human life: a screw manipulation task anda writing task. A human subject performed the manipulation

1769

tasks with the contact force sensing device. The subject wasmale of age 25 years.
A. Screw manipulation task
The subject rotated the contact force sensing device like ascrewdriver. The screw manipulation task is executed in theorder explained as follows:
1) Noncontact between the device and a hand (Fig. 6-a).
2) Rotating the device clockwise (Fig. 6-b).
3) Noncontact between the device and the hand.
4) Repeat 2) and 3) twice.
5) Rotating the device counterclockwise (Fig. 6-c).
6) Noncontact between the device and the hand.
7) Repeat 5) and 6) twice.
He touches diﬀerent parts of the device when he regrasps it in
the step 2) and 5). The influence of the wiring of the device
on sensor reading is negligible during this manipulation task.

Fig. 8. Segmented the screw manipulation task into 2 primitives.
1), 3), and 6) are segmentations labeled by ‘noncontact’. 2) is a segmentation
labeled by ‘clockwise’. 5) is a segmentation labeled by ‘counterclockwise’.
Sampling rate of data count is 33[Hz].

Fig. 9.

a: noncontact
Fig. 6.

b: clockwise

c: counterclockwise

Snapshots of the screw manipulation task.

We applied our method to the screw manipulation task.
Sampling rate of the contact force sensing device is 33[Hz].
In the implementation of the method, the parameters are
determined as follows: the coordinate system of the contact
force sensing device is defined as shown in Fig. 7. The
magnitude and the direction of the contact force vector fi
is obtained from the output and the position of the i thsensor. A position vector pi is decided from the center
position of the sensing part of the i th sensor and the originof the coordinate system. The value of r is approximately
95.9[mm]: the maximum distance of the sensors from theorigin.

Fig. 7.

Coordinate system of the sensing device.

1) Comparison between primitives extracted by themethod and segmentations of human motion by human intuition: We applied our method to the screw manipulation task
in order to segment the task into 2 primitives. The result isshown in Fig. 8.
Horizontal axis shows the elapsed time. Vertical axis
shows the extracted primitives as the clusters. The colors

Segmented the screw manipulation task into 3 primitives.

of the data and the division by the vertical lines indicate
3 groups classified through visual observation by humanintuition: ‘noncontact’, ‘clockwise’, and ‘counterclockwise’.
These groups are classified independently of the primitives
extracted by our method. The 2 primitives (Fig. 8) are
interpreted as ‘noncontact’ and ‘rotation (clockwise & counterclockwise)’. The 2 primitives are similar to segmentationsof human motion by human intuition.
2) Selection of the number of the primitives by choosing
the number of the clusters: We also applied our method tothe screw manipulation task in order to segment the task into
3 primitives. The result is shown in Fig. 9. The 3 primitives(Fig. 9) are interpreted as ‘noncontact’, ‘rotation (clockwise& counterclockwise)’, and ‘approach or release’. ‘approachor release’ means the transitions between 2 segmentations
labeled by ‘noncontact’ and ‘rotation’.
3) Advantage of contact force index against raw sensor
output: We segmented the screw manipulation task into 2
primitives by applying the EM algorithm to the set of theraw sensor outputs: the 60 dimensional data provided fromthe 60 pressure sensors of the device. The segmentation resultis shown in Fig. 10.
The extracted 2 primitives by using the 6 dimensional
contact force indices (Fig. 8) are more similar to the groups
classified by human intuition than those by using the raw
sensor outputs (Fig. 10). Rotation of the device is a causeof this result. For the convenience of comprehension, we
discuss a case of only the 6 pressure sensors. We explain two
grasping situations: they are diﬀerent in the contact points(Fig. 11-(a) and Fig. 11-(b)). The numerals in Fig. 11 indicate
the serial numbers of the 6 pressure sensors. Every force

1770

Fig. 10. Segmented the screw manipulation task into 2 primitives by using
the 60 dimensional data provided from the contact force sensing device.

imposed at the contact points is the same magnitude ‘F’. A
sensor output is expressed as a vector consisting of outputsof the 6 pressure sensors. Thus, the sensor output in Fig.
11-(a) is F a = (F 0 F 0 F 0), and the sensor output in Fig.
11-(b) is F b = (0 F 0 F 0 F). In addition, the sensor output
in the case of ‘noncontact’ is F o = (0 0 0 0 0 0).
The Euclidean distances between F a , F b , and F o arecalculated as follows:|F a − F b | = 6F,(9)|F a − F o| = |F b − F o| = 3F,(10)... |F a − F b | > |F a − F o| = |F b − F o|.

(11)

The Euclidean distance between the two grasping situations(F a and F b ) is greater than that between ‘noncontact’ (F o)and each grasping situation (see Eq. (11)). Therefore, thetwo grasping situations are extracted as diﬀerent primitives
in the case of using the raw sensor outputs. In contrast, thecontact force indices of F a and F b are of equal magnitude.
Therefore, the two grasping situations are extracted as thesame primitive in the case of using the contact force indices.

6

6

1

5

2
4

(a)

3

1

5

2
4

3

(b)

Fig. 11. Two grasping situations which are diﬀerent in the contact points.

7: Moving the tip of the device.
8: Drawing a vertical line.
9: Moving the tip of the device.
10: Drawing a horizontal line.
11: Moving the tip of the device.
12: Drawing a horizontal line.
13: Grasping the device for writing.
14: Carrying the device back where it was (Fig. 12-e).
15: Changing a grasping form and putting the device
on the desk (Fig. 12-f).
16: The device is on the desk without a touch.
We classified the 16 segmentations mentioned above into 8groups by human intuition.
I:
The device is on the desk without a touch (1, 16).
II:
Taking up the device (2).
III:
Carrying the device (3, 14).
IV:
Changing a grasping form (4, 15).
V:
Grasping the device for writing (5, 13).
VI:
Drawing a vertical line (6, 8).
VII: Drawing a horizontal line (10, 12).
VIII: Moving the tip of the device (7, 9, 11).
We segmented the writing task into 8 primitives in thesame way described in Section IV-A. The segmentation resultis shown in Fig. 13. We successfully segmented the writingtask into the 8 primitives which are similar to the 8 groups
classified by human intuition.
V. Conclusions and Future Work
A. Conclusions
We defined the contact force index for segmenting humanmanipulation task into primitives. The contact force index
was calculated from the set of the contact forces at all thecontact points. Then, we proposed a segmentation methodof human manipulation task into primitives by applying the
EM algorithm to the set of the contact force indices. In theexperiments, we applied our method to manipulation tasks
performed in daily human life: a screw manipulation task anda writing task. The results of the experiments are summarized
as follows:
1) We successfully segment the tasks into primitives
which are similar to segmentations of human motion
by human intuition.
2) The primitives are extracted regardless of the relative
position/orientation between a human hand and an
object during the manipulation tasks.
B. Future work

B. Writing task
The subject drew lines with the contact force sensing
device. The sequence of writing task is as follows:
1: The device is on a desk (Fig. 12-a).
2: Taking up the device.
3: Carrying the device above paper (Fig. 12-b).
4: Changing a grasping form for writing (Fig. 12-c).
5: Grasping the device for writing (Fig. 12-d).
6: Drawing a vertical line.

We will segment various manipulation tasks performed indaily human life into primitives. In addition, we will map
the primitives onto robotic hands in the programming-bydemonstration.

1771

References[1] A. Bicchi, Hands for Dexterous Manipulation and Robust Grasping:
A Diﬃcult Road Toward Simplicity, IEEE Trans. on Robotics andAutomation, Vol. 16, No. 6, 2000, pp. 652–662.

a: The device is on the desk.

b: Carrying the device above paper.

c: Changing a grasping form for writing.

d: Grasping the device for writing.

e: Carrying the device back where it was.

f: Changing a grasping form andputting the device on the desk.

Fig. 12.

Fig. 13.

Snapshots of the writing task.

Segmented the writing task into 8 primitives.

[2] K. Ikeuchi and T. Suehiro, Toward an Assembly Plan from Observation
Part I: Task Recognition With Polyhedral Objects, IEEE Trans. on
Robotics and Automation, Vol. 10, No. 3, 1994, pp. 368–385.
[3] Y. Kuniyoshi, M. Inaba, and H. Inoue, Learning by Watching: Extracting Reusable Task Knowledge from Visual Observation of Human
Performance, IEEE Trans. on Robotics and Automation, Vol. 10, No.
6, 1994, pp. 799–822.
[4] K. J. Kyriakopoulos, J. Van Riper, A. Zink, and H. E. Stephanou,
Kinematic analysis and position/force control of the Anthrobot dexterous hand, IEEE Trans. on Systems, Man, and Cybernetics, Part B,
Vol. 27, No. 1, 1997, pp. 95–104.
[5] J. Hong and X. Tan, “Calibrating a VPL DataGlove for teleoperating
the Utah/MIT hand”, Proc. of the 1989 IEEE Int’l Conf. on Robotics
and Automation, 1989, pp. 1752–1757.
[6] K. Bernardin, K. Ogawara, K. Ikeuchi, and R. Dillmann, A Sensor
Fusion Approach for Recognizing Continuous Human Grasping Sequences Using Hidden Markov Models, IEEE Trans. on Robotics,
2005, Vol. 21, No. 1, pp. 47–57.
[7] J. Aleotti and S. Caselli, “Grasp Recognition in Virtual Reality for
Robot Pregrasp Planning by Demonstration”, Proc. of the 2006 IEEE
Int’l Conf. on Robotics and Automation, 2006, pp. 2801–2806.
[8] L. Y. Chang, N. S. Pollard, T. M. Mitchell, and E. P. Xing, “Feature
selection for grasp recognition from optical markers”, Proc. of the2007 IEEE/RSJ Int’l Conf. on Intelligent Robots and Systems, 2007,

pp. 2944–2950.
[9] S. Ekvall and D. Kragic, “Grasp Recognition for Programming by
Demonstration”, Proc. of the 2005 IEEE Int’l Conf. on Robotics andAutomation, 2005, pp. 760–765.
[10] M. Kondo, J. Ueda, Y. Matsumoto, and T. Ogasawara, “Perception of
Human Manipulation Based on Contact State Transition”, Proc. of the2004 IEEE/RSJ Int’l Conf. on Intelligent Robots and Systems, 2004,
pp. 100–105.
[11] M. R. Cutkosky, On Grasp Choice, Grasp Models, and the Designof Hands for Manufacturing Tasks, IEEE Trans. on Robotics andAutomation, 1989, Vol. 5, No. 3, pp. 269–279.
[12] N. Kamakura, M. Matsuo, H. Ishii, F. Mitsuboshi, and Y. Miura,
Patterns of static prehension in normal hands, The American J. of
Occupational Therapy, 1980, Vol. 34, No. 7, pp. 437–445.
[13] J. K. Salisbury and B. Roth, Kinematic and Force Analysis of Articulated Hands, ASME J. of Mechanisms, Transmissions, and Automationin Design, 1982, Vol. 105, pp. 33–41.
[14] A. T. Miller and P. K. Allen, “Examples of 3D Grasp Quality
Computations”, Proc. of the 1999 IEEE Int’l Conf. on Robotics andAutomation, 1999, pp. 1240–1246.
[15] A. P. Dempster, N. M. Laird, and D. B. Rubin, Maximum Likelihoodfrom Incomplete Data via the EM Algorithm, J. of the Royal StatisticalSociety, Series B (Methodological), 1977, Vol. 39, No. 1, pp. 1–38.

1772