Currently under review at IEEE Transactions on Robotics, c IEEE Transactions on Robotics.
This copy is for personal use only.
Preprint Submitted to IEEE TRANSACTIONS ON ROBOTICS

1Hierarchical Fingertip Space: A Unified Framework
for Grasp Planning and In-Hand Grasp Adaptation
Kaiyu Hang, Miao Li, Johannes A. Stork, Yasemin Bekiroglu, Florian T. Pokorny,
Aude Billard and Danica Kragic

Abstract—We present a unified framework for grasp planningand in-hand grasp adaptation using visual, tactile and proprioceptive feedback. The main objective of the proposed framework
is to enable fingertip grasping and address problems of changed
weight of the object, slippage and external disturbances. For thispurpose, we introduce Hierarchical Fingertip Space (HFTS) concept as a representation enabling optimization for efficient grasp
synthesis. Grasp synthesis is followed by a grasp adaptation stepthat consists of both grasp force adaptation through impedancecontrol and regrasping/finger gaiting when the former is not
sufficient. Experimental evaluation is conducted on an Allegro
hand mounted on a Kuka LWR arm.
Index Terms—Fingertip grasping, Hierarchical Fingertip
Space, grasp synthesis, grasp adaptation

I. I NTRODUCTIONRASP planning and in-hand grasp adaptation are twocomplex problems that have commonly been studied
separately. Lots of contributions to these problems have been
made during the past two decades considering stability modeling and estimation, task based grasping, object representation,
grasping synergies and grasp adaptation [1]–[10].
In this paper, we present a framework for fingertip grasping
considering an integrated approach to grasp planning and inhand grasp adaptation. The main objective of the framework
is to address the problem of grasp instability due to problems
such as changed weight of the object, slippage or external
disturbances such as collision. The framework implements
both grasp force adaptation through impedance control andregrasping/finger gaiting when the former is not sufficient. We
integrate our previous work on Hierarchical Fingertip Space(HFTS) based grasp synthesis [11] and grasp adaptation [9].
The approach consists of i) a pre-grasping phase executing
grasp synthesis on an efficient representation including bothobject and hand properties, ii) grasp execution, and iii) a postgrasping phase where tactile feedback and experience are used
for in-hand grasp adaptation, see Fig. 1 and Fig. 2.
In the pre-grasping phase, grasp synthesis is formulated
as a combinatorial optimization problem considering grasp
stability, contact locations and finger gaiting in an integrated
manner. In the post-grasping phase, tactile feedback provides
information of the stability of the executed grasp. An offline

G

K. Hang, J. A. Stork, Y. Bekiroglu, F. T. Pokorny and D. Kragic are with
KTH Royal Institute of Technology, Stockholm, Sweden, as members of theComputer Vision & Active Perception Lab., Centre for Autonomous Systems{kaiyuh, jastork, yaseminb, fpokorny, dani}@kth.se.
M. Li and A. Billard are with Learning Algorithms and Systems Laboratory(LASA) at École Polytechnique Fédérale de Lausanne (EPFL), Switzerland{miao.li, aude.billard}@epfl.ch.

Fig. 1. A visualization of the proposed Hierarchical Fingertip Space concept:
Initial fingertip locations are determined by optimizing grasp stability andadaptability using a hierarchical discretization of the object surface and an
impedance controller is used to balance grasping forces. If a large disturbanceoccurs, the grasp is adapted by fingertip gaiting to maintain grasp stability.
The new fingertip location is computed using an optimization in the HFTS.

learned probabilistic model is used to assess the grasp stability
and initiate an adaptation of grasp forces, followed by finger
gaiting if needed. To the best of our knowledge, this is so far
the only system that accomplishes grasp synthesis, stability
estimation, online replanning and in-hand adaptation in a
unified framework and evaluated on a real physical system.
Compared to the state of the art and our previous work in[11] and [9], our integrated system:
closes the loop between grasp planning and control
through stability estimation and finger gaiting;• introduces a Gaussian Process-based cluster analysis for
the Hierarchical Fingertip Space representation that is
considering surface properties of objects and viable contact locations;• optimizes grasp adaptability and demonstrates informed
finger gaiting by considering viable hand configurations
and object shape knowledge.
•

We review the related work in Sec. II and present themethodology in Sec. III, Sec. IV and Sec. V. We evaluate
in Sec. VI and then conclude in Sec. VII.

2

Preprint Submitted to IEEE TRANSACTIONS ON ROBOTICS

Pre-Grasping
Augmented Fingertip
Space

GP Based Clustering for
Hierarchical Fingertip Space

Grasping

Post-Grasping+

Fingertip Gaiting
Keep
Monitoring

Hierarchy

KeepMonitoring

MonitorStability

Yes

StableGP Clustering?

Probabilistic Model for Stability Estimation
and Grasp Adaptation

No

Adapt
Stiﬀness?

Adaptability Prioritized
Reachability

Grasp Optimization

Impedance Control
in Virtual Frame

Adapt
Stiﬀness
Keep
Monitoring

No

Fingertip Gaiting
in HFTS

Yes

Fig. 2. Schematic overview of the system: Pre-grasping: After the Hierarchical Fingertip Space is generated, grasps are synthesized by a multi-level refinement
strategy. Grasping: The synthesized hand configuration is used to execute the grasp. Post-grasping: Once tactile feedback is available, grasp stability is monitored
by a learned probabilistic model. If a grasp is estimated as unstable, the stability is maintained through force adaptation or finger gaiting.

II. R ELATED W ORK
The area of object grasping covers several problems: grasp
stability analysis, data-driven grasp synthesis and planning,object and task representation, grasp adaptation [6], [10], [12]etc. Although each of these have been studied extensively during the past couple of decades there are rather few systems thathave addressed grasp synthesis and in-hand grasp adaptationin an integrated manner.
In terms of object representation, there are many examples
of works that rely on encoding shape properties of objects:
Reeb Graph [13], Medial Axis [14], [15], hierarchical box
decomposition [16], super-quadrics [17]–[20]. More recent
work demonstrates topological analysis of shape for grasping
and caging [21], [22]. Our HFTS proposes a hierarchicalmethod for shape representation that encodes both the global
and local shape of the object.
Classical work formulates contact-level grasp synthesis
as an optimization problem [8], [12], [23]–[27] for which
the objective—grasp stability—is commonly performed using
force analysis in the contact wrench space [28]. The problemof estimating feasible hand configurations has also been addressed in this context [2], [29]. To account for uncertainties
in physical properties of objects, grasp friction sensitivity [30]and independent contact regions [31] have been investigated.
Our approach formulates fingertip grasping as an optimization
problem considering grasp stability, adaptability and handreachability to prepare a grasp for future adaptive execution
against physical uncertainties.
Approaches to force based grasp control range from geometry based analytic methods [32]–[34] to learning-based frameworks for force optimization [35], [36]. In-hand manipulation
has been addressed as finger gaiting with a rolling contact
model and quasi-static assumption [37], [38]. Hybrid position
and force control has also been addressed [39]–[42] as well as
impedance control [43]–[46]. Our approach allows for grasp
stabilization through both contact force adaptation and finger
gaiting planned in real-time using tactile feedback and the
proposed Hierarchical Fingertip Space.
In realistic tasks, the ability to retain a stable grasp on an
object is an integral property of robust systems. A grasp originally estimated as stable may be perturbed while performinga manipulation with the held object. This is also valid forcases where some properties of the object change - weight canchange if a glass held by the robot gets filled, environmental
changes can affect friction coefficients, collision may cause
slippage, etc. In addition, many of these properties may not
be exactly known to start with. Thus, in-hand grasp adaptation
may be needed after a grasp has been applied on an object. For
this purpose, relying on visual feedback is not sufficient andmany of the recent approaches facilitate haptic and proprioceptive information [7], [47]–[53]. Finger gaiting may be further
required when applying higher grasping force does not suffice[9], [54]. Our work here builds upon [9], [54] and additionallyallows for replanning during grasp execution.
III. H IERARCHICAL F INGERTIP S PACE AND G RASPO PTIMIZATIONIn the pre-grasping phase, we formulate fingertip grasp
synthesis as an optimization problem considering each object
represented as a point cloud P = {pi ∈ R3 | i ∈ {1, ..., np }}.
We seek m contact locations, Cg = {c1 , ..., cm |ci ∈ P}, on
the object surface and a hand configuration, Jg ∈ Rd whered are controlled joint angles.
There are two concepts we define: Fingertip Space and Hierarchical Fingertip Space (HFTS). Fingertip Space represents
a finite set of contacts on an object surface that are locally flatand large enough for a fingertip [11]. We denote the Fingertip
Space as Φ(P) = {φ1 , ..., φnf } ⊂ P and an element of this set
φi is called a Fingertip Unit. Fingertip Space Φ is parametrized
by locations and normals of Fingertip Units. We extract theΦ(P) from P based on the estimated surface curvature from a
set of points within one fingertip size1 r, around the potential
contact, N r (pi ) ⊂ P. The fingertip space of P is given by
Φ(P) = {φi | K(N r (φi )) ≤ κ, φi ∈ P}

(1)

where K(N r (pi )) is the local surface curvature estimated fromN r (pi ) and κ ∈ R is the empirically determined curvaturethreshold. In the rest of this paper, we write Φ instead of
Φ(P). Fig. 3(left) shows an example of Fingertip Space. To
enable finger gaiting, we want our Fingertip Space to encode
the space around each Fingertip Unit in an efficient manner.
To achieve this, we put a penalty term on admissible regions
1 For the SynTouch sensor used in this work, the fingertip size r is 14mm,
http://www.syntouchllc.com/

HANG et al.: HIERARCHICAL FINGERTIP SPACE: A UNIFIED FRAMEWORK FOR GRASP PLANNING AND IN-HAND GRASP ADAPTATION

3

Fingertip hierarchy
Parent nodes

Fig. 3. Left: Fingertip Space with attached penalties rendered by jet colormap,
in red are points that have been filtered out by Eq. (1), note that there areperception noise in planar areas. Middle: GP for the spray model represented
by 20 cluster centers. Right: partitions of fingertip units rendered with different
colors.

using a logistic function. Let c(φi ) ∈ P \ Φ be the closest
point to Fingertip Unit φi that has been rejected by Eq. (1),the penalty wi is computed as:
1
(2)
wi =
−γkφi −c(φi )k1+ewhere γ ∈ R+ is an elasticity factor, see Fig. 3(left).
A. Multilevel refinement of Fingertip Space
Given the large number of Fingertip Units per object,formalizing grasp optimization on all combinations of theseis computationally impractical. A more feasible strategy is
to apply Surrogate Models or multilevel refinement [55],[56], that recursively approximate the original optimization
problem in a hierarchy of simpler, more tractable problems i.e.
surrogate models. We first explain grasp synthesis for a single
fingertip contact optimization (gaiting) and then continue with
optimization for multiple fingertip contacts.
Surrogate approximation of Φ is constructed by recursively
grouping Fingertip Units by cluster analysis using geometricproperties. For the optimization of a single contact in Φ
we construct a hierarchy of surrogate approximations of Φ(see Fig. 4) as a similarity-based graph GΦ = (EΦ , VΦ ),with the hierarchy levels i ∈ {0, . . . , l − 1} representing
different scales of surrogate approximations. Φ is recursively
partitioned into smaller sets of fingertip units, denoted asφ̂i,j ⊂ Φ, and is represented as a node φi,j ∈ VΦ in graphGΦ , where i is the level of φi,j in the hierarchy and j is
the index of the partition in level i. We partition the set
Φ in a top-down manner, with parent φi,j nodes split into
children nodes if |φ̂i,j | > 1. Ultimately, the bottom levelof GΦ consists of nodes representing single fingertip units,|φ̂0,j | = 1. Experimentally and as shown in Fig. 4, the numberof partitioning centers for the second top level is set to 20 andin the remaining levels to 4 children similar to [8].
Given real sensor data, there is noise associated with thecomputation of surface normals. To address this, we employ
a Gaussian process based filter with Thin Plate Spline kernel[57]. Higher sampling frequency for GP centers is used inareas of higher curvature, see Fig. 3(middle). The distribution of centers captures the geometric similarities (locations
and normals) and therefore relate to the similarities in thegrasp wrench space [58]. GP partitioning is regulated using

Fig. 4. Surrogate models represented as a graph. Fingertip unit partitions are
represented as nodes in different levels of the hierarchy and the connectivities
in this graph are represented by edges. Extra connectivities defined by Eq. (3)are exemplified in level l − 3 for φl−3,1 , with red edges for 2 hops and blueedges for 4 hops.

a threshold Tp , so that if |φ̂i,j | ≤ Tp , a node is not further
divided by GP partitioning but it is split up into all its
fingertip units. Nodes consisting of single fingertip units arecopied to the next level as long as some other nodes can bepartitioned. This guarantees a balanced partitioning tree, andhence a valid surrogate approximation for every level in thehierarchy. As discrete optimization relies on relevant neighbors
in the solution space, we introduce connectivity by introducing
additional edges between nodes and their siblings into EΦ .
More precisely, the extended edge set consists of parent-childP
Sedges and sibling-edges EΦ = EΦ
∪ EΦwhich are given as:PEΦ= {{φi,j , φi−1,k } ∈ VΦ × VΦ | φ̂i−1,k ⊆ φ̂i,j }SEΦ= {{φi,j , φi,k } ∈ VΦ × VΦ | hop(φi,j , φi,k ) ≤ h}

(3)

The function hop(φi,j , φi,k ) denotes the hop distance betweenP
φi,j and φi,k along edges in EΦ
. The hop limit h ∈ N defines
the size of the neighborhood and is set to 4 in our experiments,
resulting in neighborhoods of size e.g., ca 4cm in the second
top level. Using the definitions above, we can now define thei-th surrogate approximation of the Fingertip Space Φ as:(GΦ )i = ((VΦ )i , (EΦ )i )[(VΦ )i = {φi,j }(4)j(EΦ )i = {{φi,j ,φi,k } | {φi,j , φi,k } ∈ EΦ }which is a subgraph of GΦ and an approximation at i-th
resolution level.
We define the mean location and orientation of the set offingertip units contained in the partition φ̂i,j as p(φ̂i,j ) ∈ R3
and n(φ̂i,j ) ∈ R3 , this will be used later for stability analysis.
In terms of Eq. (2), the penalty assigned to a node φi,j isdefined as:
X
1
wi,j =wk(5)|φ̂i,j |φk ∈φ̂i,j

Given the hierarchy GΦ of surrogate approximation models,
we can optimize a fingertip location in a top-down manner.
By optimizing the contact in a coarse to fine fashion, a finalcontact will be found in the bottom of the hierarchy. Next, we
will investigate the grasp synthesis with multiple contacts.

4

Preprint Submitted to IEEE TRANSACTIONS ON ROBOTICS

B. Hierarchical Fingertip SpaceIn the previous section, we introduced the similarity-basedgraph GΦ for a single fingertip. For m fingertips, we define theproduct graph ΛΦ = (VΛ , EΛ ) named Hierarchical Fingertip
Space (HFTS) as in Eq.(6). Thus, nodes in VΛ represent
combinations of m contacts, λi,j = (φi,j1 , ..., φi,jm ), and thegraph-distance between nodes in the same level reflects thesimilarity of the individual contacts. Formally, the HFTS is
defined as:ΛΦ = G1Φ × · · · × Gm
ΦVΛ = {λi,j =(φi,j1 , ..., φi,jm ) | λi,j ∈ VΦ1 × . . . × VΦm }(6)kwhere GkΦ = (VΦk , EΦ) is the surrogate hierarchy for the k-th
fingertip. The penalty value for a set of contacts is defined asminimum of all individual contact penalties:∗wi,j= min{wi,j1 , . . . , wi,jm }(7)

Optimization on ΛΦ requires definition of neighborhoods
and we define two types of edges for EΛ : 1) Edges between
nodes and their parent, EΛP , such that ΛΦ inherits the hierarchy
levels from the individual GkΦ , and 2) edges between nodesof the same level, EΛS , for which the individual contacts are
identical or siblings in their graph GkΦ , respectively. Formally,
we obtain EΛ = EΛP ∪ EΛS :
(k)

(k)

P
}
EΛP = {{λi,j1 , λi−1,j2 } | ∀k : {λi,j1 , λi−1,j2 } ∈ EΦ
(k)

(k)

S
}
EΛS = {{λi,j1 , λi,j2 } | ∀k : {λi,j1 , λi,j2 } ∈ EΦ

(8)

(k)

where λi,j ∈ VΦk is the k-th item of tuple λi,j . Similarly tothe surrogate models for a single fingertip contact, we define
the i-th surrogate approximation of multiple fingertip graspingin HFTS as:(ΛΦ )i = ((VΛ )i , (EΛ )i )[(VΛ )i = {λi,j }(9)j(EΛ )i = {{λi,j1 ,λi,j2 } | {λi,j1 , λi,j2 } ∈ EΛ }C. Grasp Optimization in HFTS
So far, we described the solution space for grasp synthesis
using nodes λg ∈ ΛΦ from different levels, which are
combinations of contacts on the object surface. However, torealize the contacts with a robot hand, we additionally need thejoint angles Jg ∈ Rd . A valid grasp solution, g = (λg , Jg ),is a combination of contact positions and joint angles.
1) Grasp Stability: During the pre-grasping phase, when
we synthesize a grasp, only visual information of object isavailable and we need to evaluate or predict grasp stability
without feedback. This can be done using contact based force
closure analysis [12], [28]: Given a grasp solution g, the grasp
quality measure Q(λg ) ∈ R is the minimum offset between
the origin of the wrench space and the convex hull spannedby friction cones of contacts parametrized by positions andnormals [23]. The value is positive when the grasp is force
closed and larger for more stable grasps.

2) Grasp Reachability: Not all combinations of contacts
λg can be realized by a given robotic hand and we canclassify contacts into reachable or unreachable using a function
R∗ : VΛ → {0, 1} so that the optimization can be constrained
to reachable grasps with R∗ (λg ) = 0. Since a robotic hand
can have many degrees of freedom and complicated coupledkinematics, and we seek a computation time on the scale ofmilliseconds during grasp optimization, it can be too costly
to analytically compute R∗ (λg ) in each optimization step.
For this, various forms of constraints have been formulated[59], [60]. To achieve required speed and precision, we linearly relax it as a measure of dissimilarity between λg andthe closest known reachable contacts λ∗g of grasp solutiong ∗ = (λg∗ , Jg∗ ). The reachability measure of m contacts λg
is then reformulated as a residual R(λg ) ∈ R+ :R(λg ) = C(λg ) − C(λg∗ )(10)where C(·) ∈ R6×(m−2) is an affine invariant encoding of mcontacts in terms of its contact locations and normals [11].
To generate a collection of viable grasps, we randomly
sample hand configurations and save the encoded contacts andcorresponding hand configuration Jg into a k-d tree like data
structure T offline with the query time O(n log n). Using T ,we can compute the residual by lookup and find the handconfiguration for realizing the contacts if the residual was
small.
(11)
T : λg 7→ (Jg∗ , R(λg ))3) Grasp Adaptability: We use grasp adaptability to enable
finger gaiting already in the grasp synthesis stage. By decomposing the hand Jacobian and calculating the manipulability[61] of a hand configuration in the tangential plane of contacts,
we measure the adaptability of a grasp, denoted as A(Jg ) ∈R+ . Concretely, given the Jacobian Jf (Jg ) ∈ R3×n and thenormal nf ∈ R3 of fingertip f , the Jacobian can be rotatedby Rf ∈ R3×3 such that the last row of Jˆf (Jg ) = Rf Jf (Jg )corresponds to the movement of fingertip in the direction offf (Jg ) ∈ R2×n ,nf . The first two rows of Jˆf , denoted by J
are then the projection of Jf in the tangential plane of thefingertip normal.
XqT
ff (Jg )J
ff (Jg )(12)
detJ
A(Jg ) =f

Note that we can assume that the fingertip normal (on the
robot hand) and the fingertip unit normal will be similar when
the grasp is realized if R(λg ) is small. An example of grasp
adaptability measure is shown in Fig. 5. Since this measureis hand configuration based, it is affine invariant, and hence
grasp pose independent.
In order to capture grasp stability, reachability and adaptability in the grasp optimization, the optimization objective isdefined as follows:
Priority 1: Maximize

θ(g)

(13)

Priority 2: MaximizeA(Jg )(14)α ∈ R+(15)with
θ(g) = Q(λg ) − αR(λg ),HANG et al.: HIERARCHICAL FINGERTIP SPACE: A UNIFIED FRAMEWORK FOR GRASP PLANNING AND IN-HAND GRASP ADAPTATION

5

Algorithm 1 Surrogate-Based Optimization in HFTS

Fig. 5. Grasp Adaptability for fingertip 1. Adaptability is computed forfingertip positions sampled in joint space. The colored volume shows fingertip
positions and their adaptability values.

where α is a weighing factor to account for the hand size. To
optimize the second objective, we use a sorted lookup tablefor R(λg ) which returns the most adaptable joint configurationin the area of the best grasp according to A(λg ) [62], [63]when querying reachability residuals (line 7 and 10 in Alg. 1).
As we can see in Fig. 6, for the same contact locations,
there can be multiple hand configurations for realizing it,
however, our prioritized lookup table will always return thehand configuration with the best adaptability.

Input: stopCondition, ΛΦ , maxIter
Output: grasp g = (λg , Jg )1: for i = l − 1 to 0 do
2:
if i = l − 1 then
⊲ Initialization
3:
λg ← random from (ΛΦ )i4:else⊲ Extend to Lower Surrogate
5:
λg ← random child of λg
6:end if7:(Jg , R(λg )) ← T (λg )8:for 1 to maxIter do⊲ Optimize Surrogate
9:λg′ ← random neighbor of λg ∈ (ΛΦ )i
10:(Jg′ , R(λg′ )) ← T (λg′ )11:
if Pr (g, g ′ ) ≥ rand(0, 1) then12:g ← g′
13:
end if14:
if stopCondition(g) then
⊲ Good Solution
15:
break
16:
end if17:
end for
18: end for

In cases when the final reachability residual R(λg ) 6= 0, a
local optimization of joint configuration by linear interpolation[64] is required to realize desired contacts. To avoid too small
and time consuming incremental improvements at each level,
we utilize a stopCondition. It can be set to false if we want
to explore the space until convergence or we control thenumber of iterations by setting a threshold for the optimization
function in Eq. (15).
IV. G RASP ADAPTATION
A synthesized grasp is executed using a simple position
control [9]. When contacts are made, tactile readings areavailable and an impedance controller [3] is used to regulate
grasp stiffness. The controller is formulated in a virtual frame(VF) defined in terms of fingertip locations as

Fig. 6. Grasps with same contacts and different adaptabilities: the left grasp
has the highest adaptability.

Having defined the objective function, we can now proceed
to grasp optimization. Using a surrogate-based optimization
metaheuristic, we need to find solutions on each of the surrogate approximations and extend them to the next model. For
optimization in each model, we adopt stochastic hill climbing
which can escape from local optima by means of randomness.
Switching between two solutions g and g ′ is determined by
the probabilistic function in Eq. (16):−1wg θ(g) − wg′ θ(g ′ )′Pr (g, g ) = 1 + exp(16)ζ

where wg is the penalty assigned to a tuple of contacts definedby Eq. (7). The randomness in the optimization is determinedby ζ, it makes the optimization more random when a large
value is chosen, while it behaves more like pure hill climbing
if a small value is applied. The grasp optimization algorithm
is shown in Alg. 1.
For realizing the grasp, we can transform the hand baseto the pose where the fingertips meet the contact locations.

Ro = [vx , vy , vz ] ∈ SO(3)p3 − p1vx =kp3 − p1 k
(p2 − p1 ) × vx
vz =k(p2 − p1 ) × vx k
vy = vz × vx

(17)

where p1 , p2 and p3 ∈ R3 are locations of the fingertips, seeFig. 7.

Fig. 7. Left: Virtual frame Ro , vx , vy and vz defined by fingertip locations.
Right: Virtual springs used by the impedance controller. A virtual spring (red)is superimposed on the impedance controller (between fingertip and the newlocation p̂) when a fingertip gaiting is requested.

A grasp in the VF is denoted ĝ = (K, L, S) where K =(Kx , Ky , Kz ) ∈ R3 is grasp stiffness and L = (L1 , L2 , L3 ) ∈6Preprint Submitted to IEEE TRANSACTIONS ON ROBOTICS

R3 is grasp rest length, i.e. the distance between each fingertip
and the center of VF. S = (S1 , S2 , S3 ) ∈ R57 represents thetactile readings, in our case from SynTouch sensors.
Grasp stability is monitored using a probabilistic representation relying on a Gaussian Mixture Model Θ that is trained
offline, see Fig. 8. As described in detail in our previous work[9], Θ is trained over K, L, S parameters for a variety ofobjects. Given Θ, grasp stability is estimated byp(ĝ|Θ) =ng
Xπi N (ĝ|µi , Σi )(18)

Fig. 9. Breadth-first search in HFTS for fingertip gaiting optimization. The
green path shows how the search fringe evolves, and the red edges show thepruned path due to the 2 criteria defined in Alg. 2.

i=1

where ng is the number of Gaussian components, each ofwhich has a prior πi . N (ĝ|µi , Σi ) is the Gaussian distribution
with mean µi and covariance Σi .
Learned Model
Centers of Gaussians
Grasps before Stiffness Adaptation
Grasps After Stiffness Adaptation
Grasps Require Finger Gaiting
Nearest Gaussian Center

Fig. 8. GMM model Θ for grasp stability estimation and decision making
for grasp adaptation. The gray ellipsoids depict the Gaussian components of
Θ, dots and circles show grasps, represented by K, L, S in different stages,
and the red lines illustrate how grasps are switched between different stages
by grasp stiffness adaptation and fingertip gaiting.

A grasp ĝ is considered unstable if the log likelihood of
Eq. (18) is smaller than a predefined threshold, which is set
to −100 as in our previous work [9]. If a grasp ĝ is unstable,
we compute its Mahalanobis distance to each component in Θand denote the minimum distance as md . If md is within two
standard deviations, we apply force adaptation by changing
stiffness K to the value obtained by computing the maximum
expectation of K conditioned on grasp rest length L and tactile
readings S. The details of this process have been described indetail in our previous work [9]. Otherwise, a finger gaiting
strategy is initiated as explained in detail in next section.
V. R EGRASPING BY FINGER GAITING
Stiffness adaptation is not enough in cases when there isan upper bound on the force exerted by the hand. Thus, tostabilize a grasp, the system initiates finger gaiting. Finger
gaiting is defined as an optimization problem based on thecurrent rest length L represented in VF:θ∗ (λg ) = kL − L∗ k + βR(λg )(19)+where R(λg ) is the reachability defined in Eq. (10), β ∈ R
is a weighing factor to account for the hand size and L∗ is thedesired rest length obtained from the closest Gaussian centerĝ ∗ = (K ∗ , L∗ , S ∗ ) in terms of md . The reasoning above is tofind the closest stable and reachable grasp to the current one,taking into account the current tactile readings.

For the robot hand we use in this work, we can only relocate fingertip F 1 or F 2, as shown in Fig. 7, since relocating
the thumb F 3 leaves the grasp without contacts on opposite
sides of the object. Our strategy of choosing between F 1 andF 2 is straightforward: we compute the optimization for F 1
and F 2 in parallel for choosing objective value from Eq. (19).
Our optimization procedure employs breadth-first search in ΛΦstarting from the initial contact. The search is terminated ineach branch when the reachability measure grows beyond apredefined threshold ǫR . Since we move only one finger, weneed an additional rule:
(
(i)
(i)
T rue, ∃i : i 6= fo ∧ λg 6= λg′
Prune(λg , λg′ , fo ) =F alse, otherwise.
(20)
where fo is the fingertip to be relocated, λg is the nodethat represents the current grasp contacts, and λg′ is the
new solution. Since the search fringe can go upwards in thehierarchy graph ΛΦ , this rule asserts that only a single fingertip
is moved while the remaining two are kept fixed. The mainidea is sketched in Fig. 9 and the procedure summarized in
Alg. 2. Note that it includes the penalty factor from Eq. (7).
Algorithm 2 Fingertip Gaiting by Optimization in ΛΦInput: ΛΦ , λg , ǫR , fo
Output: p̂⊲ New Location
1: Ro = R(λg )∗2: λ ← λg
3: Queue.push(neighbors of λg )4: while Queue is not empty do
5:
λg′ ← Queue.pop()
6:
if R(λg′ ) > Ro + ǫR or P rune(λg , λg′ , fo ) then7:
continue⊲ Pruning8:
end if9:
if w1 ′ θ ∗ (λg′ ) < w1 ∗ θ ∗ (λ∗ ) thenλg10:
λ∗ ← λg ′11:
end if⊲ Breadth-First
12:
Queue.push(neighbors of λg′ )13: end while
14: p̂ ← λ∗(fo )⊲ New Location for fo

A. Fingertip Gaiting in Practice
When grasp stability changes rapidly, the system would
need to switch between impedance and position control frequently. To avoid this, we stay in impedance control mode during finger gaiting by sliding the finger to the desired position.
To allow this, we formulate fingertip gaiting using impedancecontroller defined in VF. A virtual spring with stiffness k is

HANG et al.: HIERARCHICAL FINGERTIP SPACE: A UNIFIED FRAMEWORK FOR GRASP PLANNING AND IN-HAND GRASP ADAPTATION

Grasp Adaptability Distribution
·104
3
2
1
·10−2

0
2.4

2.6

2.8

3

3.2

3.4

3.6

3.8

4

3.71

3.52

3.56

3.67

3.5

3.54

Ju
g

4

M
ilk

Grasp Adaptability (×10−2 )

3

ra
y
Sp

e2
ttl
Bo

Bo

ttl

e1

2
Ri
ve
lla

defined to connect the current location of the moved fingertip
and p̂, which is equivalent to a fingertip impedance controller
superimposed on the original grasp controller. An example offingertip F 1 gaiting is depicted as in Fig. 7.
The stiffness k of the virtual spring is determined by the
distance dp̂ ∈ R between the fingertip’s current location p̂ andan empirical parameter Γ ∈ R as: k = dp̂ Γ. In this way, thefingertip will be slided towards p̂ while keeping the contact
on the object. Since p̂ is computed in the HFTS, we ensurethat the desired position is on the object surface. If a newgoal position is requested during finger gaiting, the system
will either continue to the new position if the same fingertip
is concerned or stop the current gaiting and initiate gaiting
with another fingertip. Another example situation is depicted inFig. 10 where fingertip F 2 stopped moving before the desired
position is reached, since the grasp was estimated as stable on
the way.

7

Fig. 11. Upper: Grasp adaptability distribution of 106 hand configurationsin the reachability lookup table. Lower: Average grasp adaptabilities for the
100 grasps generated in the evaluations for all objects.

Fig. 10. The rivella bottle is grasped by the Allegro hand and a human isapplying random perturbations on top of it. The red and green points areshowing the new locations for fingertip F 1 and F 2 computed by Alg. 2 with
virtual springs in the virtual frame.

VI. E XPERIMENTAL EVALUATION
We perform experimental evaluation with an Allegro hand
mounted on a Kuka LWR arm. The hand is equipped with
SynTouch2 tactile sensors on three fingertips. The systems
performance is evaluated using six objects shown in Fig. 12,
which are tracked using the OptiTrack3 real-time motion
tracking system. The evaluation presented below demonstrates
the performance of the grasp synthesis system alone as wellas the integrated system for grasp adaptation.

Fig. 12. Six example objects used in the evaluation: there is both variationin global geometry as well as local surface properties.

A. Grasp Synthesis
Grasp synthesis is performed on a point cloud representationof objects generated offline. We also generated a reachability
table with 106 hand configurations using rejection sampling:grasps are first uniformly sampled in the hand joint space
and those with adaptabilities larger than 0.02 are kept. The
motivation for choosing the adaptabilities larger than 0.02
is shown in Fig. 11 (upper), showing the distribution of theadaptability.
Alg. 1 generates both contact locations and hand configurations. Simple position based control is used to execute a grasp[9]. A few examples are shown in Fig. 12 and Fig. 13.
For evaluating the performance of the grasp planner, we
repeat the grasp optimization according to Alg. 1 for eachtest object. In order to keep an equal number of iterations
2 http://www.syntouchllc.com/
3 http://www.naturalpoint.com/optitrack/

Fig. 13. Example grasps generated by Alg. 1 with stopCondition(g) that,as soon as the grasp is stable and the reachability residual is smaller than
0.006, we stop optimizing on the current level of ΛΦ and continue on thenext level.

for each repetition of the algorithm, we set maxIter = 100
and stopCondition(g) = f alse. For each object, we run
the algorithm with random initialization until we achieve
100 stable and collision free grasps. Evaluation results are
summarized into a table shown in Fig. 14.
First, Fig. 14 shows that the number of levels of the graphGΦ are between 4 and 5 when Tp = 10, or between 3 and
4 when Tp = 40. This indicates that our system produces
similar depth of the HFTS independent of the shape of theobject. However, the shape of the object affects the number of8Preprint Submitted to IEEE TRANSACTIONS ON ROBOTICS

Tp

10

40

#Levels
6
6
5
6
5
6
3
3
4
3
4
3

#Nodes
5672
6321
3227
5124
3204
6926
2839
3203
3002
2399
2912
3310

Time(s)
1.81
1.77
1.64
1.54
0.96
2.04
0.45
0.64
0.81
0.44
0.73
0.62

SR(%)
96.15
93.46
89.29
98.03
97.09
94.34
94.34
94.34
87.72
99.01
96.15
92.59

θ(×10−2 )
5.23
4.74
4.96
3.29
4.05
3.71
5.01
4.56
4.99
3.17
4.12
3.62

Fig. 14. Evaluation of Alg. 1: #Units: number of fingertip units in Φ(P),#Levels: number of levels in graph GΦ (including the top level with only
one node), #Nodes: number of nodes in graph GΦ , Time(s): average time inseconds for one run of the algorithm, SR: success rate of synthesizing a stablegrasp. θ: optimized objective value defined in Eq. (15). The evaluations were
implemented in Python on a machine with Ubuntu 12.04 running on Intel
Core i7-2820QM 2.30GHz processors.

nodes at each level, given that some branches are terminatedfor objects of simpler geometry, such as the milk package.
Regarding the success rate (SR), it shows the number of runs
needed to generate 100 stable and collision free grasps on eachobject. We can see that SR lies at approximately 90%. Fig. 11
shows average adaptabilities for the 100 stable grasps for eachobject. Average adaptability values, computed by Eq. (12), arelarge showing that our methodology considers the adaptability
effectively.

B. Grasp Adaptation
Once a grasp is executed and contacts established, thesystem will enter the post-grasping phase and start monitoring
the stability based on tactile feedback. Now, the impedancecontroller is used to control the grasp using GMM based model
Θ. Our tactile based grasp stability estimator is trained in thesame way as in [9]. The log likelihood threshold for Eq. (18) is
set to −100 for stable grasps as we found empirically that theobject drops at a value below −130. For the force control of
the hand, we set the initial grasp stiffness K = (12, 2, 2) and
use it for the execution of all grasps, as described in Sec. IV.
For the evaluation, we run two sets of experiments: 1) We
continuously increase the objects’ weight by filling themto evaluate the maximum weight each grasp can withstand,and 2) we shake the grasped objects by linearly increasing
acceleration in different directions to evaluate the maximum
acceleration each grasp can withstand. For comparison, we
conduct the same experiments without any grasp adaptationand on the system proposed in [9] which does not consider
object shape information when relocating fingertips.
1) Testing maximum weight: For each object, we executethe best out of 100 grasps generated in Sec. VI-A and alignthe object with vertical axis as shown in Fig. 18. We thengradually fill object with black pepper beans and record the
maximum weight the grasp can withstand. The maximum
weight is reached when the stability estimator predicts unstablegrasp for more than 2 seconds or if the object drops. We repeat
this test for each grasp 5 times and summarize the results in
Fig. 15. For comparison, we perform the same tests for thesystem proposed in [9] and a system without adaptation.

Objectbottle1bottle2jug
rivellamilk
spray

Weight
34
39
112
24
34
63

Without
55.1 ± 7.11
62.8 ± 6.63
125.3 ± 14.90
36.0 ± 6.96
63.5 ± 8.20
75.7 ± 7.21

With [9]153.1 ± 12.31
102.3 ± 13.38
147.4 ± 9.62
76.5 ± 9.4
151.8 ± 7.24
102.2 ± 6.02

Improved
165.3 ± 13.27
121.3 ± 9.91
162.1 ± 13.12
92.7 ± 7.45
157.4 ± 8.35
121.6 ± 7.15

Fig. 15. The comparison of the supported object weights(Unit:gram). without:
without grasp adaptation; with [9]: with grasp adaptation in [9]; improved:
the new adaptation approach in this paper.

Naturally, the system without any adaptation performs theworst and the integrated system outperforms the system from[9]. This is since our system: i) takes into account grasp
reachability during the exploration, and ii) the new locationis computed in the HFTS, thus ensuring it is valid, avoiding
problems shown in Fig. 16, and iii) considers two fingers for
gaiting, resulting in increased flexibility.

Fig. 16. The risk of moving a fingertip to an non-existing position present
in [9] is addressed by using our HFTS representation. The red point shows
the fingertip position before gaiting.

A quantitative evaluation of the proposed system and thesystem in [9] has been conducted with respect to optimization
residual. We first execute the grasp in simulation and thentrigger the fingertip gaiting by sending desired rest lengths
randomly sampled around the current values within a ball ofradius 20mm. The result is shown in Fig. 17. Due to the object
shape constraint, the systems cannot provide zero residuals.
Our system performs much better for non-planar objects giventhat HFTS representation considers shape in an effective way.
Optimization Residual (mm)Object(#Units)bottle1(2736)
bottle2(3102)
jug(2671)
rivella(2273)
milk(2696)
spray(3207)
bottle1(2736)
bottle2(3102)
jug(2671)
rivella(2273)
milk(2696)
spray(3207)

8

Alg. 2

With [9]

Milk

Jug

6
4
2
0
Rivella Bottle1 Bottle2

Spray

Fig. 17. The results of fingertip gaiting optimization residual. 105 desired
rest lengths are randomly sampled around the current rest length within a ballof radius 20mm.

An example of the supported weight test for the rivellabottle is shown in Fig. 18. In the beginning when the object
is not too heavy, the likelihood p(ĝ|Θ) is larger than −100and the grasp stiffness K is constant. As the weight increases,
grasp becomes unstable and stiffness adaptation is initiated.
Stiffness changes rapidly when the weight increases, and when
the force adaptation is not able to handle the current weight, afinger gaiting is triggered and fingertip F 2 is relocated. After
finger gaiting, grasp stiffness is decreased since the new grasp

HANG et al.: HIERARCHICAL FINGERTIP SPACE: A UNIFIED FRAMEWORK FOR GRASP PLANNING AND IN-HAND GRASP ADAPTATION

requires less force to be stable. As the weight increases again,
the whole process is repeated, resulting in F 1 finger gaiting.

F 2 Gaiting

F 1 Gaiting

Force Adaptation

kKk

20

15

0

1

0

1

2
Time (seconds)

3

4

3

4

p(ĝ|Θ)

−50
−100
−150
2
Time (seconds)

Fig. 18. A record of supported weight test of a grasp on rivella bottle. Upper:
The norm of grasp stiffness and fingertip gaiting. Lower: Likelihood for grasp
stability estimation defined in Eq. (18).

J0
−30◦

J1
30◦

J2
2◦

Init. K = (Kx , Ky , Kz )(12, 2, 2)

J3
−60◦

J4
−20◦

Horizontal Acc.
2m/s2 – 8m/s2

J5
0◦

J6
−60◦

Vertical Acc.
2m/s2 – 8m/s2

9Similarly to the supported weight test, we evaluate eachgrasp by measuring the maximum acceleration it can withstand. The criterion is similar: the maximum acceleration isrecorded when the grasp is predicted as unstable for more than
2 seconds or if the object drops. The shaking test is conductedin both directions separately and on each object by filling itwith 10g, 20g, 30g, 40g and 50g black pepper beans. Each test
is repeated 5 times. For comparison, we also conduct the sametest for a system without adaptation and the system proposedin [9].
Experimental results are summarized in Fig. 20. If the
maximum acceleration rate is 8m/s2 , it means that the grasp
has been kept stable during the test. On the other hand, if the
maximum acceleration rate is 2m/s2 , it means that the grasp
could not withstand any shaking. We can see that our systemoutperforms both the system without the adaptation and thesystem proposed in [9]. The strength of our approach is that we
ensure that the finger gaiting has resulted in an actual contact
with the object which is not the case in [9]. In addition, theflexibility of gaiting two fingers provides additional strength.
Additional quantitative results are shown in Fig. 21. We cansee that the average computing time of Alg. 2 is between 20ms
and 40ms. The average number of explored nodes shows thatthe pruning is efficient since less than 5% of all nodes in GΦ
are considered. Note that the computation time and numberof explored nodes are heavily dependent on the connectivityof graph GΦ : less nodes in the graph does not mean lesscomputing time. Therefore, the connectivity in GΦ indirectly
measures how complex an object is in the context of thissystem.
VII. C ONCLUSION

Fig. 19. The setup of grasp shaking test, in which the arm shakes each grasp
in horizontal and vertical directions. J0 to J6 are the joint values for the
initial pose of shaking test. When shaking horizontally, the shaking direction
is fixed to be perpendicular to the palm.

2) Shaking Test: External disturbances, such as collision,
may occur once a grasp has been executed. To evaluate
the proposed system, we designed a shaking test. We first
execute the best out of the 100 generated grasps for eachobject according to Eq. (15), and then rotate the arm tothe configuration shown in Fig. 19. Thereafter, we start toshake the arm in either vertical or horizontal directions while
linearly increasing the acceleration from 2m/s2 to 8m/s2 .
The shaking magnitude is limited to 10cm in either directions,which means that the hand is accelerating in the first 5cm anddecelerating in the second 5cm. After every period of shaking,we increase the acceleration by 1m/s2 and therefore have 14
shakes for every test.

We have presented a unified framework for grasp planning and in-hand grasp adaptation using visual, tactile andproprioceptive feedback. The proposed Hierarchical Fingertip
Space defines a hierarchy of surrogate solution spaces offingertip grasping enabling both planning and adaptation. By
augmenting the fingertip space in terms of local geometry andspatial relations, as well as optimizing hand configurations
with respect to grasp adaptability, we demonstrated efficientplanning and adaptation. Moreover, the probabilistic modelfor grasp stability estimation and adaptation has shown its
feasibility in closing the loop between grasp replanning andcontrol. We have evaluated the performance of the proposedsystem quantitatively and shown that the proposed system significantly improves the robustness of grasp execution. It also
outperforms our previous work reported in [9]. To the best of
our knowledge, this is so far the only system that accomplishes
grasp synthesis, stability estimation, online replanning and inhand adaptation in a unified framework, as well as evaluating
this on a real physical system.
However, as a basic drawback of most learning based
approaches, our probabilistic model is experience based, andhence relying on the training data, i.e. limited number ofobjects and examples to generalize from. As a potential future
work, we plan to design an active learning strategy to update
this model iteratively using new experiences over time, so as10Preprint Submitted to IEEE TRANSACTIONS ON ROBOTICS

Jug

Rivella
8

6

4

2

6

4

20

30

40

50

10

4

20

30

40

10

50

20

30

40

Additional Weight (g)Additional Weight (g)Additional Weight (g)Bottle2

Milk

Spray

6

4

2

6

4

2
20

30

40

50

50

8
Maximum Acceleration (m/s2 )8
Maximum Acceleration (m/s2 )

8

10

6

2

2
10

Maximum Acceleration (m/s2 )8
Maximum Acceleration (m/s2 )

HA
VA
HB
VB
HC
VC

Maximum Acceleration (m/s2 )Maximum Acceleration (m/s2 )8Bottle1

6

4

2
10

Additional Weight (g)

20

30

40

50

Additional Weight (g)

10

20

30

40

50

Additional Weight (g)Fig. 20. Results of shaking tests on grasps shown in Fig. 12. In the legend, H and V refer to horizontal shaking test and vertical shaking test respectively.
A, B and C refer to 3 grasp strategies: grasp without adaptation, grasp adaptation in [9] and the grasp adaptation proposed in this paper.
Objectbottle1bottle2jug
rivellamilk
spray

Avg. Duration(ms)
261.2
320.1
414.4
447.9
392.6
502.7

Avg. Improvement
66.21
75.17
70.72
52.39
47.11
57.26

Avg. Comp. Time(ms)
30.7
32.1
19.4
38.6
24.9
29.2

Avg. Err.(m)0.00740.00620.00420.00450.00570.0068

Avg. # Nodes
279.2
221.5
140.4
194.1
137.6
197.6

# Gaiting
6
4
5
12
9
7

Fig. 21. Results for the horizontal shaking tests when the objects are filled with 20g of pepper beans (from left to right): average duration for one time offingertip gaiting; Average stability likelihood improvement after fingertip gaiting; Average computation time of Alg. 2 for each computation; Average errors
between achieved rest lengths and the rest lengths computed by Alg. 2; Average number of nodes explored in Alg. 2; Number of fingertip gaiting required
during a shaking test with 14 shakes. The evaluations were implemented in C++ and run on a machine with Ubuntu 12.04 running on Intel Core i7-2820QM
2.30GHz processors.

to evolve the model in a long term to generalize it to a broader
set of objects, without retraining the models from scratch.
ACKNOWLEDGEMENT
This work was supported by the European Union Seventh Framework Program FLEXBOT - FP7-ERC-279933 andROBOHOW - FP7-288533.
R EFERENCES[1] K. Shimoga, “Robot grasp synthesis algorithms: A survey,” The International Journal of Robotics Research, vol. 15, no. 3, pp. 230–266, 1996.
[2] C. Borst, M. Fischer, and G. Hirzinger, “Calculating hand configurations
for precision and pinch grasps,” in Proc. IEEE/RSJ Int. Conf. Intell.
Robot. Syst., 2002.

[3] K. Tahara, S. Arimoto, and M. Yoshida, “Dynamic object manipulation
using a virtual frame by a triple soft-fingered robotic hand,” in Proc.
IEEE Int. Conf. Robot. Autom., 2010.
[4] T. Yoshikawa, “Multifingered robot hands: Control for grasping andmanipulation,” Annual Reviews in Control, vol. 34, no. 2, pp. 199 –
208, 2010.
[5] J. Romano, K. Hsiao, G. Niemeyer, S. Chitta, and K. Kuchenbecker,“Human-inspired robotic grasp control with tactile sensing,” IEEE
Transactions on Robotics, vol. 27, no. 6, pp. 1067–1079, Dec 2011.
[6] A. Sahbani, S. El-Khoury, and P. Bidaud, “An Overview of 3D Object
Grasp Synthesis Algorithms,” Robot. Auton. Syst., vol. 60, no. 3, pp.
326–336, 2011.
[7] Y. Bekiroglu, D. Song, L. Wang, and D. Kragic, “A probabilistic
framework for task-oriented grasp stability assessment,” in Proc. IEEE
Int. Conf. Robot. Autom., 2013.
[8] K. Hang, J. A. Stork, F. T. Pokorny, and D. Kragic, “Combinatorial
optimization for hierarchical contact-level grasping,” in Proc. IEEE Int.
Conf. Robot. Autom., 2014.

HANG et al.: HIERARCHICAL FINGERTIP SPACE: A UNIFIED FRAMEWORK FOR GRASP PLANNING AND IN-HAND GRASP ADAPTATION[9] M. Li, Y. Bekiroglu, D. Kragic, and A. Billard, “Learning of grasp
adaptation through experience and tactile sensing,” in Proc. IEEE/RSJ
Int. Conf. Intell. Robot. Syst., 2014.
[10] J. Bohg, A. Morales, T. Asfour, and D. Kragic, “Data-driven grasp
synthesis – a survey,” IEEE Transactions on Robotics, vol. 30, no. 2,
pp. 289–309, 2014.
[11] K. Hang, J. A. Stork, and D. Kragic, “Hierarchical fingertip space formulti-fingered precision grasping,” in Proc. IEEE/RSJ Int. Conf. Intell.
Robot. Syst., 2014.
[12] A. Bicchi and V. Kumar, “Robotic grasping and contact: A review,” inProc. IEEE Int. Conf. Robot. Autom., 2000.
[13] J. Aleotti and S. Caselli, “A 3d shape segmentation approach for robot
grasping by parts,” IEEE RAS, vol. 60(3), pp. 358–366, 2012.
[14] M. Przybylski, T. Asfour, and R. Dillmann, “Planning grasps for robotic
hands using a novel object representation based on the medial axis
transform,” in Proc. IEEE/RSJ Int. Conf. Intell. Robot. Syst., 2011.
[15] N. Vahrenkamp, M. Przybylski, T. Asfour, and R. Dillmann, “Bimanual
grasp planning,” in Proc. IEEE-RAS Int. Conf. Humanoid Robots, 2011.
[16] K. Huebner, “BADGrA toolbox for box-based approximation, decomposition and GRasping,” Robot. Auton. Syst., vol. 60, no. 3, pp. 367 –
376, 2012.
[17] R. Pelossof, A. Miller, P. Allen, and T. Jebara, “An svm learning
approach to robotic grasping,” in Proc. IEEE Int. Conf. Robot. Autom.,2004.
[18] G. Biegelbauer and M. Vincze, “Efficient 3d object detection by fitting
superquadrics to range image data for robot’s object manipulation,” inProc. IEEE Int. Conf. Robot. Autom., 2007.
[19] T. T. Cocias, S. M. Grigorescu, and F. Moldoveanu, “Multiplesuperquadrics based object surface estimation for grasping in service
robotics,” in Optimization of Electrical and Electronic Equipment (OPTIM), 2012, pp. 1471–1477.
[20] D. Berenson, R. Diankov, K. Nishiwaki, S. Kagami, and J. Kuffner,“Grasp planning in complex scenes,” in Proc. IEEE-RAS Int. Conf.
Humanoid Robots, 2007.
[21] J. A. Stork, F. T. Pokorny, and D. Kragic, “Integrated motion and clasp
planning with virtual linking,” in Proc. IEEE/RSJ Int. Conf. Intell. Robot.
Syst., 2013.
[22] F. T. Pokorny, J. A. Stork, and D. Kragic, “Grasping objects with holes:
A topological approach,” in Proc. IEEE Int. Conf. Robot. Autom., 2013.
[23] C. Borst, M. Fischer, and G. Hirzinger, “Grasping the dice by dicing
the grasp,” in Proc. IEEE/RSJ Int. Conf. Intell. Robot. Syst., 2003.
[24] B. Mirtich and J. Canny, “Easily computable optimum grasps in 2-d and3-d,” in Proc. IEEE Int. Conf. Robot. Autom., 1994.
[25] J.-P. Saut and D. Sidobre, “Efficient models for grasp planning with amulti-fingered hand,” Robot. Auton. Syst., vol. 60, no. 3, pp. 347 – 357,
2012.
[26] C. Rosales, J. Porta, and L. Ros, “Grasp optimization under specificcontact constraints,” IEEE Transactions on Robotics, vol. 29, no. 3, pp.
746–757, June 2013.
[27] I. Gori, U. Pattacini, V. Tikhanoff, and G. Metta, “Three-finger precision
grasp on incomplete 3d point clouds,” in Proc. IEEE Int. Conf. Robot.
Autom., 2014.
[28] C. Ferrari and J. Canny, “Planning optimal grasps,” in Proc. IEEE Int.
Conf. Robot. Autom., 1992.
[29] C. Rosales, J. Porta, R. Suarez, and L. Ros, “Finding all valid handconfigurations for a given precision grasp,” in Proc. IEEE Int. Conf.
Robot. Autom., 2008.
[30] K. Hang, F. T. Pokorny, and D. Kragic, “Friction coefficients and grasp
synthesis,” in Proc. IEEE/RSJ Int. Conf. Intell. Robot. Syst., 2013.
[31] M. Roa and R. Suarez, “Computation of independent contact regions forgrasping 3-d objects,” IEEE Transactions on Robotics, vol. 25, no. 4,
pp. 839–850, 2009.
[32] K. Hertkorn, M. Roa, and B. Ch, “Planning in-hand object manipulation
with multifingered hands considering task constraints,” in Proc. IEEE
Int. Conf. Robot. Autom., 2013.
[33] C. Borst, M. Fischer, and G. Hirzinger, “Grasp planning: how to choosea suitable task wrench space,” in Proc. IEEE Int. Conf. Robot. Autom.,2004.
[34] R. Platt, A. H. Fagg, and R. A. Grupen, “Null-space grasp control:
Theory and experiments,” IEEE Transactions on Robotics, vol. 26, no. 2,
Apr. 2010.
[35] C. Xiong and Y. Xiong, “Neural-network based force planning for
multifingered grasp,” Robot. Auton. Syst., vol. 21, no. 4, pp. 365 – 375,
1997.
[36] Y. Xia, J. Wang, and L.-M. Fok, “Grasping-force optimization for
multifingered robotic hands using a recurrent neural network,” IEEE

11

Transactions on Robotics and Automation, vol. 20, no. 3, pp. 549–554,
June 2004.
[37] L. Han and J. Trinkle, “Dexterous manipulation by rolling and finger
gaiting,” in Proc. IEEE Int. Conf. Robot. Autom., 1998.
[38] J.-P. Saut, A. Sahbani, S. El-Khoury, and V. Perdereau, “Dexterous
manipulation planning using probabilistic roadmaps in continuous grasp
subspaces,” in Proc. IEEE/RSJ Int. Conf. Intell. Robot. Syst., 2007.
[39] Z. Li, P. Hsu, and S. Sastry, “Grasping and coordinated manipulation
by a multifingered robot hand.” The International Journal of Robotics
Research, vol. 8, no. 4, pp. 33–50, 1989.
[40] T. Yoshikawa and X.-Z. Zheng, “Coordinated dynamic hybrid position/force control for multiple robot manipulators handling one constrained object,” The International Journal of Robotics Research, vol. 12,
no. 3, pp. 219–230, 1993.
[41] M. Buss, H. Hashimoto, and J. Moore, “Dextrous hand grasping force
optimization,” IEEE Transactions on Robotics and Automation, vol. 12,
no. 3, pp. 406–418, Jun 1996.
[42] T. Takahashi, T. Tsuboi, T. Kishida, Y. Kawanami, S. Shimizu, M. Iribe,
T. Fukushima, and M. Fujita, “Adaptive grasping by multi fingered handwith tactile sensor based on robust force and position control,” in Proc.
IEEE Int. Conf. Robot. Autom., 2008.
[43] S. A. Schneider and R. H. Cannon, “Object impedance control forcooperative manipulation: theory and experimental results,” IEEE Transactions on Robotics and Automation, vol. 8, no. 3, pp. 383–394, 1992.
[44] H. Liu and G. Hirzinger, “Cartesian impedance control for the DLR
hand,” in Proc. IEEE/RSJ Int. Conf. Intell. Robot. Syst., 1999.
[45] K. Tahara, K. Maruta, A. Kawamura, and M. Yamamoto, “Externallysensorless dynamic regrasping and manipulation by a triple-fingered
robotic hand with torsional fingertip joints,” in Proc. IEEE Int. Conf.
Robot. Autom., 2012.
[46] M. Li, H. Yin, K. Tahara, and A. Billard, “Learning object-level
impedance control for robust grasping and dexterous manipulation,” inProc. IEEE Int. Conf. Robot. Autom., 2014.
[47] K. Hsiao, S. Chitta, M. Ciocarlie, and E. Jones, “Contact-reactive
grasping of objects with partial shape information,” in Proc. IEEE/RSJ
Int. Conf. Intell. Robot. Syst., 2010.
[48] K. Hsiao, L. Kaelbling, and T. Lozano-Perez, “Task-driven tactile
exploration,” in Proceedings of Robotics: Science and Systems, 2010.
[49] A. Morales, M. Prats, P. Sanz, and A. P. Pobil, “An Experiment in
the Use of Manipulation Primitives and Tactile Perception for Reactive
Grasping,” in Robotics: Science and Systems - Robot Manipulation:
Sensing and Adapting to the Real World, 2007.
[50] C. Zito, M. Kopicki, R. Stolkin, C. Borst, F. Schmidt, M. Roa, andJ. Wyatt, “Sequential trajectory re-planning with tactile information gain
for dexterous grasping under object-pose uncertainty,” in Proc. IEEE/RSJ
Int. Conf. Intell. Robot. Syst., Nov 2013.
[51] H. Dang and P. K. Allen, “Grasp adjustment on novel objects usingtactile experience from similar local geometry.” in Proc. IEEE/RSJ Int.
Conf. Intell. Robot. Syst., 2013.
[52] H. Dang, J. Weisz, and P. Allen, “Blind grasping: Stable robotic grasping
using tactile feedback and hand kinematics,” in Proc. IEEE Int. Conf.
Robot. Autom., 2011.
[53] Y. Bekiroglu, J. Laaksonen, J. A. Jorgensen, V. Kyrki, and D. Kragic,“Assessing grasp stability based on learning and haptic data,” IEEE
Transactions on Robotics, vol. 27, no. 3, pp. 616–629, June 2011.
[54] E. L. Sauser, B. D. Argall, G. Metta, and A. G. Billard, “Iterative
learning of grasp adaptation through human corrections,” Robot. Auton.
Syst., 2012.
[55] S. Koziel, D. Ciaurri, and L. Leifsson, “Surrogate-based methods,” inComputational Optimization, Methods and Algorithms, ser. Studies in
Computational Intelligence. Springer Berlin Heidelberg, 2011, vol.
356, pp. 33–59.
[56] C. Walshaw, “Multilevel refinement for combinatorial optimisation problems,” Annals of Operations Research, vol. 131, no. 1-4, pp. 325–372,
2004.
[57] N. Sommer, L. Miao, and A. Billard, “Bimanual compliant tactile
exploration for grasping unknown objects,” in Proc. IEEE Int. Conf.
Robot. Autom., 2014.
[58] F. T. Pokorny and D. Kragic, “Classical grasp quality evaluation: New
theory and algorithms,” in Proc. IEEE/RSJ Int. Conf. Intell. Robot. Syst.,
2013.
[59] S. El Khoury, M. Li, and A. Billard, “Bridging the gap: One shot grasp
synthesis approach,” in Proc. IEEE/RSJ Int. Conf. Intell. Robot. Syst.,
Oct 2012, pp. 2027–2034.
[60] B. Huang, S. El-Khoury, M. Li, J. Bryson, and A. Billard, “Learninga real time grasping strategy,” in Proc. IEEE Int. Conf. Robot. Autom.,May 2013, pp. 593–600.

12Preprint Submitted to IEEE TRANSACTIONS ON ROBOTICS[61] T. Yoshikawa, “Manipulability of robotic mechanisms,” The International Journal of Robotics Research, 1985.
[62] L. Guilamo, J. Kuffner, K. Nishiwaki, and S. Kagami, “Efficient prioritized inverse kinematic solutions for redundant manipulators,” in Proc.
IEEE/RSJ Int. Conf. Intell. Robot. Syst., 2005.
[63] Y. Aydin and M. Nakajima, “Database guided computer animation ofhuman grasping using forward and inverse kinematics.” Computers &Graphics, vol. 23, pp. 145–154, 1999.
[64] M. Powell, “A direct search optimization method that models theobjective and constraint functions by linear interpolation,” in Advancesin Optimization and Numerical Analysis, ser. Mathematics and Its
Applications. Springer Netherlands, 1994, vol. 275, pp. 51–67.

Kaiyu Hang received the B.S. degree in information and communication engineering from Xi’An
Jiaotong University, Xi’an, China in 2010 and the
M.Sc. degree in communication systems from KTH
Royal Institute of Technology, Stockholm, Sweden
in 2012.
He is currently a Ph.D student at the Centre
for Autonomous Systems and the Computer Vision
and Active Perception Lab at KTH Royal Instituteof Technology, Stockholm, Sweden. His research
include robotic precision/fingertip grasping, objecthand integrated representation, adaptive grasping and grasp manifold learning.

Miao Li received the Bachelor and Masters degreeboth from the School of Mechanical Science andEngineering, Huazhong University of Science andTechnology(HUST), China, in 2008 and 2011 respectively. From Sep. 2011, he joined thecurrent Department of Mechanical Engineer-ing at cole Polytechnique Fdrale de Lausanne (EPFL) in Switzerland, as a doctoral assistant. His current researchinterests mainly focus on robust robotic grasping,
impedance control of dexterous manipulation usingtactile sensing.

Johannes A. Stork received the B.S. degree incomputer science from Albert Ludwigs University ofFreiburg, Freiburg, Germany, in 2009 and the M.S.
degree in computer science from Albert Ludwigs
University of Freiburg, Freiburg, Germany, in 2012.
He is currently pursuing the Ph.D. degree in computer science and robotics at the School of Computer
Science and Communication, KTH Royal Institute ofTechnology, Stockholm, Sweden.
From 2009 to 2012, he was a Research Assistant with the Social Robotics Lab, Albert Ludwigs
University of Freiburg, Freiburg, Germany. His research interest include
representation learning, robotic in-hand manipulation, robotic grab synthesis.

Yasemin Bekiroglu received the M.Sc. degree incomputer engineering from Karadeniz Technical
University, Trabzon, Turkey, and the M.Sc. degree
in applied artificial intelligence from Dalarna University, Dalarna, Sweden. She completed her Ph.D.
studies in the Computer Vision and Active Perception Lab (CVAP), in the School of Computer
Science and Communication, at the Royal Instituteof Technology (KTH), in Stockholm, Sweden, in
2012. Currently, she is a postdoctoral researcher
at CVAP. Her research interests include robotics,
computer vision and machine learning.

Florian T. Pokorny is a Postdoctoral Researcher
at KTH Royal Institute of Technology. He is interested in robotic manipulation and machine learning
methods. He received his PhD in mathematics fromthe University of Edinburgh and, before that, hecompleted Part III of the mathematical tripos at
the University of Cambridge and obtained a BSc(first class) in Mathematics from the University of
Edinburgh.

Aude Billard is Professor of Micro and MechanicalEngineering, and the head of the LASA Laboratory
at the School of Engineering at the Swiss Federal
Institute of Technology in Lausanne. She received
a M.Sc. in Physics from EPFL (1995), a MSc. in
Knowledge-based Systems (1996) and a Ph.D. in
Artificial Intelligence (1998) from the University of
Edinburgh. She was the recipient of the Intel Corporation Teaching award, the Swiss National Science
Foundation career award in 2002, the Outstanding
Young Person in Science and Innovation from the
Swiss Chamber of Commerce and the IEEE-RAS Best Reviewer award
in 2012. Aude Billard served as an elected member of the Administrative
Committee of the IEEE Robotics and Automation society (RAS) for two
terms (2006-2008 and 2009-2011) and is the chair of the IEEE-RAS TechnicalCommittee on Humanoid Robotics. Her research interests focus on machinelearning tools to support robot learning through human guidance. This extends
also to research on complementary topics, including machine vision and its
use in human-robot interaction and computational neuroscience to develop
models of motor learning in humans.

Danica Kragic (SM12) received the M.Sc. degree inmechanical engineering from the Technical University of Rijeka, Rijeka, Croatia, in 1995 and the Ph.D.
degree in computer science from the Royal Instituteof Technology (KTH), Stockholm, Sweden, in 2001.
She is currently a Professor with the School ofComputer Science and Communication, KTH. Her
research include computer vision, object grasping
and manipulation, and humanrobot interaction. Her
recent work explores different learning methods forformalizing the models for integrated representationof objects and actions that can be applied on them. This work has demonstrated how robots can achieve scene understanding through active explorationand how full body tracking of humans can be made more efficient.
Dr. Kragic received the 2007 IEEE Robotics and Automation Society Early
Academic Career Award. She is a member of the Swedish Royal Academyof Sciences and the Swedish Young Academy. She has chaired the IEEE
Robotics and Automation Society (RAS) Technical Committee on Computerand Robot Vision and, since 2009, has served as an IEEE RAS Administrative
Committee member.