Grasp Moduli Spaces, Gaussian Processes, andMultimodal Sensor Data
Florian T. Pokorny, Yasemin Bekiroglu, Johannes Exner, Mårten Björkman, and Danica Kragic
Centre for Autonomous Systems/CVAP, CSC
KTH Royal Institute of Technology
Stockholm, Sweden

Abstract—We discuss our recent and ongoing work on
deformation-based grasp synthesis and transfer which formulatesa joint space of shapes and grasps – a Grasp Moduli Space– within which both grasp configurations and object shapes
can be continuously deformed. In this context, we propose theuse of Gaussian Process-based implicit surface representations.
These shape representations complement our previous workon using spherical harmonics as well as explicit cylindricalcoordinates to model the object shape component of a Grasp
Moduli Space. We provide a preliminary experiment with these
Gaussian Processes showing how shape representations can beobtained using multimodal visual and haptic information anddiscuss how these representations can be continuously deformed
for the purpose of transferring and generalizing known grasps.

I. I NTRODUCTION AND M OTIVATION
Humans are able to grasp and manipulate objects seeminglywith little effort. Furthermore, we are able to easily generalize
grasps and manipulation sequences from a specific demonstration, and we can transfer grasps between similar shapes
such as bottles or tools. In robotics, on the other hand, thepredominant approach to grasp synthesis has been to solve
the grasping problem for each specific object instance by completely reinitializing the developed algorithms at hand. Popular
approaches to grasp synthesis such as the sampling-based
approach of GraspIT [7], various heuristics-based approaches
such as a reduction of the object model onto its medial axis[11], and approaches decomposing the object into parts suchas boxes [6] have been developed and typically require a new
execution of the underlying grasp algorithm when a new object
is encountered.
When we think about the graspable objects surroundingus, we observe that one can consider almost any type ofobject as being part of family of similar objects which vary
continuously with respect to various parameters such as, mosttrivially, height, width and depth, but also in certain latent styleparameters. Handles of tools such as screw drivers can varyin their precise shape, for example, while it is reasonable toassume that such shapes should be concentrated around a mean
shape resembling a cylinder. Humans are able to quickly adapt
to such variations - a capability which we aim to endow robots
with in this ongoing work. Our approach to grasp transfer isbased on a continuous deformation of a known grasp and shape
to novel objects with similar shape.
The key representation which we aim to develop further
for this purpose is the notion of a Grasp Moduli Space[9] modeling shapes and grasps jointly. The term moduli

Fig. 1: A joint grasp/shape deformation and optimization in G sph (see [10]).
space here is inspired by its use in deformation theory inmathematics [1]. To represent shapes, we are investigating
Gaussian Process implicit surfaces [13, 4, 12] providing aprobabilistic shape estimate from point-cloud data.
II. G RASP M ODULI S PACES
In order to model both a shape and a particular grasp ona shape, we need to firstly define an appropriate shape spacemodeling the admissible objects. Secondly, we need to makeprecise what we mean by a grasp. Here, we consider a grasp g
as a tuple of m contact points ci ∈ R3 on an object’s surface,unit normal vectors ni ∈ S2 at those contact points, and acenter of mass z ∈ R3 : g = (c1 , . . . , cm , n1 , . . . , nm , z) ∈R3m × (S2 )m × R3 . In order to determine whether a grasp
is force-closed or not [2], we consider the L1 grasp quality
function Q defined by [5]. Q, for m contact points, is thena function on the R3m × (S2 )m × R3 , and this function
is in fact tame in the sense that it is Lipschitz continuouswith a particular, easily computable, Lipschitz constant undervariations in grasps, as shown in our recent work [8]. Grasps
with positive grasp quality under Q are then considered morestable the larger Q is. We have recently begun to consider
an appropriate simple space of smooth surfaces which allows
for a deformation-based grasp synthesis [9]. There, we considered the moduli space of smooth surfaces with cylindricalcoordinates and with grasps with m contact positions – called
G cyl – as well as the space of surfaces of revolution withgrasp configurations with m contact positions G rev ⊂ G cyl .
Both G rev and G cyl are infinite dimensional and the underlying
shape spaces of surfaces of revolution Mrev and of surfaces
with cylindrical coordinates Mcyl are convex allowing us toconstruct shape space subsets from finitely many example
surfaces by means of convex combinations of those surfaces.

Similarly, we can jointly and continuously deform a grasp
configuration in cylindrical coordinates while the underlying
object shape is deformed. In [9], we combined such continuous
deformations in grasps and shapes, where an initial stablegrasp on a surface was continuously optimized using a gradientascent while the underlying surface was deformed towards
a novel shape. Recently [10], we have further extended thisframework to be applicable directly with point cloud-data bymeans of a shape space defined using spherical harmonics.
There, we defined a shape space of surfaces with sphericalcoordinates Msph and a resulting Grasp Moduli Space G sphof grasp configurations on such surfaces. Fig. 1 illustrates
an example grasp/surface deformation in G sph with 3 contact
points. An interesting aspect of this approach is that it allows
us to quantify uncertainty in grasping both in terms of shape
uncertainty, for example due to partial or unreliable sensor
data, and contact point uncertainty, e.g. when we cannot
precisely position the robot hand.
III. G AUSSIAN P ROCESSES AND S HAPE D EFORMATIONS
A natural framework within which noisy data can be
described is that of Gaussian Process Regression. We are
currently investigating two specific kernel choices to allow us
to represent shapes and their deformations in a continuous
manner. In the recent work [3], we have shown that thethin plate kernel can provide useful shape representations
from a single view kinect point-cloud P if this point-cloudis supplemented by a set P 0 of points obtained from haptic
data of a Schunk Dexterous Hand touching the object ofinterest, and where an exploration strategy is guided towards
points of maximal variance under the GP regression model.
There, an object’s surface is represented as an implicit surface
S = f −1 (0), and f : R3 → R is described using the thin-platekernel k(xi , xj ) =√2|r|3 − 3Rr2 + R3 , where r = |xi − xj |and we set R = 3L, and L is the kinect data’s boundingbox side-length (see [3]). Yet another interesting GP kernel isthe Matérn kernel which reduces to the exponential kernel
for its parameter ν = 0.5 and to the Gaussian kernel asν → ∞. We are √in particular√considering the Matérn kernelkν= 32 (r) = (1 + l3r ) exp (− l3r ) with length-scale l.
An advantage of representing a shape by a GP is that we canobtain an estimate for the variance as well as the mean shapedetermined by the sensor data. Observe that, since a shapeis represented by a mean function m. We can, for any two
shapes S1 , S2 corresponding to GP means m1 , m2 consider acurve γ(t) = (1 − t)m1 + tm2 , t ∈ [0, 1] deforming S1 to S2 .
Since Gaussian Processes can model a rich family of shapes,
we propose to study these in the context of the Grasp Moduli
Space framework. In Fig. 2, we provide an initial illustrationof an experiment where two GP-based shape representations
are being deformed into each other using the above convex
combination of their means. The initial and final shapes are
determined from multimodal sensor data from [3], with red
points corresponding to single view visual data from a Kinect
sensor, while blue points were obtained from haptic sensors
on a Schunk Dexterous Hand. Since these GPs represent aFig. 2: Box and spray bottle models with haptic (blue points) and subsampled single view visual (kinect, red points) sensor data are displayed in
the first row. Shape deformations γ(t) between these shapes based on a Matérnkernel GP (middle row) and a thin plate GP (bottom row) are displayed fort equal to 0, 0.06, 0.12, 0.5, 1.0 from left to right. Data points are assigned
function value 0 and are supplemented by an interior point (value −1) and 15
exterior points (value 1). We transformed the point-cloud data to√lie in a cube
centered at the origin and of side-length L = 3 and set R = 3L for thethin-plate kernel and l = 1 for the Matérn kernel. In our Gaussian Process
regression, we also assume that our sensor data is corrupted by a Gaussian
noise term with σ 2 = 0.1.

shape not as a parametrized surface as in [9, 10], but only
as a level set, the representation of grasp contact points inthis setting will have to be adapted. A natural Grasp Moduli
Space G consists of mean functions corresponding to smooth
embedded surfaces and grasp contact points on the surfaces
Sh parametrized by h. More generally, this could be extendedto include the full robot hand configuration during a grasp.
Many open problems, which we are looking forward to discuss
during this workshop, exist in this direction:
Which optimization methods are most effective in optimizing a grasp’s quality, and more generally a grasp’s
task specific utility, in a general Grasp Moduli Space
where shapes are parametrized using Gaussian Processes?
• How can the GP’s variance information be incorporated
in a deformation-based grasp synthesis framework?
• How can grasp configurations best be modeled probabilistically in conjunction with the GP’s shape estimate?
• How can prototypical grasp/shape configurations in G bedetermined automatically?
•As a first step, we are currently experimenting with various
optimization methods which optimize the robot hand’s joint
configuration while we deform a known shape/grasp configuration towards a novel shape. Methods beyond a simple
gradient ascent will likely be necessary here to ensure a rapid
convergence and the numerical stability of the optimization of
the hand’s joint configuration while maintaining contact during
a shape and grasp deformation.

R EFERENCES[1] D. D. Ben-Zvi. Moduli spaces. In Timothy Gowers, June
Barrow-Green, and Imre Leader, editors, The Princeton
companion to mathematics. Princeton Univ. Press, 2009.
[2] A. Bicchi and V. Kumar. Robotic grasping and contact:
A review. In IEEE ICRA, pages 348–353 vol.1, 2000.
[3] M. Björkman, Y. Bekiroglu, V. Högman, and D. Kragic.
Enhancing visual perception of shape through tactile
glances. In IEEE/RSJ IROS, pages 3180–3186, 2013.
[4] S. Dragiev, M. Toussaint, and M. Gienger. Gaussian process implicit surfaces for shape estimation and grasping.
In IEEE ICRA, pages 2845–2850. IEEE, 2011.
[5] C. Ferrari and J. Canny. Planning optimal grasps. IEEEICRA, pages 2290–2295, 1992.
[6] K. Huebner, S. Ruthotto, and D. Kragic. Minimum
volume bounding box decomposition for shape approximation in robot grasping. In IEEE ICRA, pages 1628–1633, 2008.
[7] A. T. Miller and P. K. Allen. Graspit!: A versatile
simulator for robotic grasping. IEEE Robotics andAutomation Magazine, 11:110–122, 2004.
[8] F. T. Pokorny and D. Kragic. Classical grasp quality
evaluation: New theory and algorithms. In IEEE/RSJIROS, 2013.
[9] F. T. Pokorny, K. Hang, and D. Kragic. Grasp moduli
spaces. In Proc. of Robotics: Science and Systems, 2013.
[10] F. T. Pokorny, Y. Bekiroglu, and D. Kragic. Grasp moduli
spaces and spherical harmonics. In IEEE ICRA, 2014.
[11] M. Przybylski, T. Asfour, and R. Dillmann. Planninggrasps for robotic hands using a novel object representation based on the medial axis transform. In IEEE/RSJIROS, pages 1781–1788, 2011.
[12] C. Walder, B. Schölkopf, and O. Chapelle. Implicit
surface modelling with a globally regularised basis ofcompact support. In Computer Graphics Forum, volume 25, pages 635–644. Wiley Online Library, 2006.
[13] O. Williams and A. Fitzgibbon. Gaussian process implicit
surfaces. Gaussian Proc. in Practice, 2007.