Regrasping and Unfolding of Garments
Using Predictive Thin Shell Modeling
Yinxiao Li† , Danfei Xu† , Yonghao Yue† , Yan Wang§ , Shih-Fu Chang†§ , Eitan Grinspun† , Peter K. Allen†
Abstract— Deformable objects such as garments are highly
unstructured, making them difficult to recognize and manipulate. In this paper, we propose a novel method to teach a twoarm robot to efficiently track the states of a garment from an
unknown state to a known state by iterative regrasping. The
problem is formulated as a constrained weighted evaluation
metric for evaluating the two desired grasping points during
regrasping, which can also be used for a convergence criterion
The result is then adopted as an estimation to initializea regrasping, which is then considered as a new state for
evaluation. The process stops when the predicted thin shellconclusively agrees with reconstruction. We show experimental
results for regrasping a number of different garments including
sweater, knitwear, pants, and leggings, etc.

I. I NTRODUCTION
Robotic manipulation of deformable objects is a difficult task especially for sequential manipulations such as
unfolding a garment. One particular manipulation task is
how to unfold a garment from a random pose to a desiredconfiguration [14][3][4]. This is a difficult task because thestates of a deformable object are not easy to track andpredict owing to large-dimensional state spaces. In this paper,
we propose a system to recognize, manipulate, and unfold
a deformable garment into a desired configuration. Key
contributions of our paper are:
- A constrained weighted metric for evaluating grasping
points during regrasping, which can also be used for aconvergence criterion- A fast, two-stage deformable registration algorithm that
integrates simulated results with online localization anduses a novel non-rigid registration method to minimize
energy differences between source and target models- A semantically labeled garment database built with offline simulation that contains 37 different garments suchas sweaters, pants, shorts, dresses, scarves, etc.
- A method for analysis of local curvature using IR scandata for stable grasp- Experimental results showing that our method applied
on a Baxter robot is able to recognize a garment froman arbitrary pose, successfully regrasp it to put it into aknown configuration, and place it flat on a table
We are interested in unfolding the garment efficiently
as a part of a pipeline for manipulating deformable garments. After several steps of regrasping, the robot holds† Department Computer Science, Columbia University, New York, NY,USA{yli@cs., dx2143, yonghao@cs., eitan@cs.

allen@cs.}columbia.edu§ Department of Electrical Engineering, Columbia University, New York,NY, USA {yanwang, sfchang}@ee.columbia.edu

Fig. 1.
Our application scenario: a Baxter robot picks up a garment (asweater here), and a Kinect captures depth images to recognize the pose of
the sweater (left). The final unfolding result is shown on the right.

the garment at two desired positions. Using a sweater as
an example, we defined the optimal grasping positions on
the two sleeves, respectively. The regrasping is built on therecognition pipeline described in our previous work [10],which applies a volumetric approach to identify the garment
pose using a 3D reconstructed mesh model. We extend thisprevious work by developing a registration-search frameworkthat looks for an optimal position over the entire mesh model.
This position is then adopted as a regrasping point in 3dimensional space and guides the other hand to approachand regrasp. The complete pipeline of a robot folding a
garment from a random state is shown in Figure 2, where
our work described in this paper spans initial grasping, pose
estimation, regrasping for unfolding, and placing the garment
flat on a table.
II. R ELATED W ORK
Kita et al. [6][7] have done a series of work on garment pose recognition by grasping at one or more points
using a shape matching algorithm. The observation usedin this work is just three depth images from top, side,and front views, which may limit its accuracy on more
complicated garments. The examples shown in the paper
are all relatively rigid garments, lacking further exploration
with different materials where deformation is reasonably
complex. Willimon et al. [18][19] worked on classifying theclothing through interactive perception using a single roboticarm. The heavy dependence on the color-based segmentation,
however, makes the method sensitive to the texture variance.

Regrasp

Improperpose

Proper pose

Initial state

Initial grasp

Pose estimation

UnfoldPlace flat

Fold

UnfoldRegrasp

Fig. 2. T OP ROW: The entire pipeline of dexterous manipulation of deformable objects. B OTTOM ROW: If the recognition is not successful, the robot
will regrasp the object and repeat the step of pose estimation (the red rectangle).

Wang et al. [17], and Miller et al. [13] have done work inclothing recognition by manipulation with a PR2 robot. Theirmethod achieves good results but may lack accuracy because
of self-occlusion. Our recent work [9] on recognition andpose estimation of deformable clothes uses a Kinect depth
sensor. The method recognizes garment category and poseon individual depth images and uses majority voting to get acomprehensive result. A further work [10] first reconstructs
a 3D model of the object via a Kinect, and then searches adatabase of simulated models to predict the pose.
Osawa et al. [14] proposed a method using dual-arm
to unfold a garment from pick up. However, from theexamples in the paper, this method requires a garment withunique color and a clean background for color-based image
segmentation. The PR2 robot is probably the first robotthat has successfully unfolded deformable objects such asa towel or a T-shirt [12]. The visual recognition in thiswork targets corner-based features, which may not exist in
many soft garments. The subsequent work has improved theprediction of the state of a garment using a HMM framework
by regrasping at the lowest corner point [3]. However, thismethod also requires a clean background, and thus limits theapplicability.
The work that is closest to ours is by Doumanoglou etal. [4]. This work has impressive results for unfolding of anumber of different garments. They use a dual-arm industrial
robot to unfold a garment guided by a set of depth images
which provide a regrasping point. This method achieves
promising accuracy. A major difference between this workand ours is our use of simulated predictive thin shell models
for garments to automatically create a large database ofgarments and their poses. Their training set is a numberof physical garments that have been grasped at different
grasping points to create feature vectors for learning. Given
the physical nature of this training set, it can be very timeconsuming to create, and may have problems encompassing
a wide range of garments and different fabrics which we canmore easily accommodate in the simulation environment. We
also use an online registration of a reconstructed volumetricmesh with the simulated model to find regrasping points.
By this method, we can choose arbitrary regrasping points
without having to train the physical model for the occurrenceof the grasping points. This allows us to choose any pointon the garment at any time as the regrasping point. We havealso developed a closed loop tactile feedback algorithm thatallow us to perform a local search at the regrasping pointthat alleviates the instability of the grasp during regraspingas mentioned in [4].
In our method, the regrasping point is located by mapping
the pre-determined point from simulation mesh to the reconstructed mesh. Therefore, a fast and accurate registration
algorithm plays a key role in our method. Rigid or nonrigid surface registration is a fundamental tool to find shapecorrespondence. A thorough review can be found in [15].
Our registration algorithm builds on previous techniques forrigid and non-rigid registrations. First, we use an iterative
closest point method [2] to rigidly align the garment. We
use a distance field to accelerate the computation. Next, we
perform a non-rigid registration to improve the matchingby locally deforming the garment. Similar to [8], we findthe correspondence by minimizing an energy function that
describes the deformation and the fitting.
III. P ROBLEM F ORMULATION
A. Framework
Our objective is to put the garment into a certain configuration[10], such that the garment can be easily placed
flat on a table for the folding process. we formulate it as amathematical optimization problem:
max f (xL , xR ).

xL ,xR(1)

Here we use two vectors xL , xR ∈ R2 to describe theconfiguration, i.e. the grasping points on the UV mappingof the garment.1 and the function f is an evaluation function
for such a configuration. We seek a principled way to build
1 Each garment mesh is defined in a U V 2-dimensional parameter space.
The grasping points are chosen from a particular set of U V parameters.

B. Optimization Objective
We need a quantitative function defined on the configuration to evaluate its quality. While this can be computedon the continuous surface of the garment, we can also
discretize the garment into a set of anchor points Sg , which
typically contains about 15 points for a garment. After such
quantization, the garment pose recognition can be treated asa discrete classification problem, which the current robotics
system is able to handle reliably. This also simplifies thedefinition of the objective function, which then becomes a
2D score table or a matrix, given our robot has two arms.
Although the configuration variables xL , xR is quantizedfor the sake of reliable pose estimation, we use a smoothprobabilistic Gaussian model to describe the objective function to allow efficient optimization. Specifically, given known
locations of grasping points xL , xR , we treat the left andright hand of the robot independently and use the product ofthe probability of two Gaussians as the objective function:f (xL , xR ) = N xL | x∗L , ΣL · N xR | x∗R , ΣR , (2)where xL , xR ∈ Sg . Here N (x | x∗ , Σ) is a 2D Gaussian
distribution with x∗ as the mean, and Σ as the covariance on
the U V mapping. Typically, x∗L , x∗R are the target grasping
points, and Σs control the tolerance and convergence speed(or number of folding actions to reach the target configuration) of the regrasping process.
In practice, due to the sensor errors of occlusion anddeformation, it is not feasible to get a deterministic noise-freeobservation of xL and xR . Instead, our previous work [10]provides a probabilistic distribution p(xl , xr | y), in which
y is the observed mesh model. p is the probabilistic measurement of the confidence given certain grasping points
xl , xr . Therefore according to the conditional probabilistic
decomposition, the objective can be rewritten as,
f (xL , xR |y) =XN xl |x∗L , ΣL N xr |x∗R , ΣR p(xl , xr |y).

(3)

xl ,xr ∈Sg

Intuitively, this objective function measures the closeness ofeach grasping point to the optimal grasping points, scaled by
the predicted probability of the current grasping points being
accurate. To avoid numerical problems, we do the optimization on ln f (·), and obtain the objective after substituting thereal probabilistic density function of Gaussian:X ∗
ln f (xL , xR ) =− (xl − x∗L )T Σ−1
L (xl − xL )xl ,xr ∈Sg

Experimentally, we use the geodesic distance on the garment
surface to measure the quality of the regrasping. In our
settings, there is no reason to have more weights in one
direction against another. Therefore the covariance matrices
are set as σl I and σr I, resulting in the objective function
which needs to be maximized as,X σl kxl − x∗L k2
ln f (xL , xR ) = −xl ,xr ∈Sg
 (5)
∗ 2
+ σr kxr − xR k − ln p(xl , xr |y) .
The related parameters in the objective such as σL and σR are
set depending on the desired configuration. For example, forsweaters, we set x∗L and x∗R on the elbow of the two sleeves.
The Gaussian formulation ensures a smooth decrease fromthe expected grasping points, as visualized in Figure 3 as an
example.
1.0

distancea feedback loop for garment regrasping, which allows us tograsp at pre-determined points on the garment.
For example, the candidate garment is a sweater to be
unfold and place flat. A desired configuration is having(x∗L , x∗R ) on the elbows of the sleeves. And our goal is tofind a series of regrasping procedures that will converge toa configuration of (x∗L , x∗R ).

0.0

(a)

(b)

Fig. 3. Visualization of the defined objective used in this paper. (a) showsa sweater and (b) show a pair of pants. (c) is the visualization of distancefunction given a mesh. The color bar on the right shows the normalization
distance.

IV. I MPLEMENTED M ETHOD
A. Deformable Registration
After obtaining the location of the current grasp point, weseek to register the reconstructed 3D model to the groundtruth garment mesh to establish point correspondences. The
input to the registration is a canonical reference (“source”)
triangle mesh S that has been computed in advance andstored in the garment database, and a target triangle mesh
T representing the geometry of the garment grasped by the
robot, as acquired by 3D scans of the grasped garment.
The registration proceeds in three steps. First, we scale thesource mesh S to match its size to the target mesh T . Next,
we apply an iterative closest point (ICP) technique to rigidly
transform the source mesh S (i.e., via only translation androtation). Finally, we apply a non-rigid registration techniqueto locally deform the source mesh S toward the target T .
Scaling First, we compute a representative size for eachof the source and target meshes. For a given mesh, let ai
and gi be the area and barycenter of the ith triangle. Thenthe area-weighted center c of the mesh is
c=

NS
X
i


∗
− (xr − x∗R )T Σ−1
(x
−
x
)
+
ln
p(x
,
x
|y)
.
r
l
r
R
R
(4)

(c)

ai gi

NS
.X

ai ,

(6)

i

where NS is the number of vertices of the source mesh S.
Given the area-weighted center, the representative size l ofthe mesh is given by
l=

NS
X

NS
.X
ai kgi − ck
ai .

i

(7)

i

Let the representative sizes of the source and target meshes
be lS and lT , respectively. Then, we scale the source meshby a factor of lT /lS .
Computing the rigid transformation We use a variant of
ICP [2] to compute the rigid transformation. ICP iteratively
updates a rigid transformation by (a) finding the closest point
wj on the target mesh T for each of the vertices vj ofthe source mesh S, (b) computing the optimal rigid motion(rotation and translation) that minimizes the distance between
wj and vj , and then (c) updating the vertices vj via this rigid
motion.
To accelerate the closest point query, we prepare a grid
data structure during preprocessing. For each grid point,
we compute the closest point on the target mesh using
fast sweeping [16], and store for runtime using both thefound point and its distance to the grid point as shown in
Figure 3, (c). At runtime, we approximate the closest point
query for vertex vj by searching only among those eightprecomputed closest points corresponding to the eight grid
points surrounding vj , thereby reducing the complexity of
the closest point query to O(1) per vertex.
After establishing point correspondences, we computethe optimal rotation and translation for registering vj withwj [2]. We iteratively compute point correspondences andrigid motions until successive iterations converge to a fixed
rigid motion, yielding a rigidly registrated source mesh S̄.
Non-rigid registration Given a candidate source meshS̄ obtained via rigid registration, our non-rigid registration
seeks the vertex positions vj of the source mesh S thatminimizeES̄,T (S) = Efit (S, T ) + Edef (S, S̄),

(8)

where Efit (S, T ) penalizes discrepancies between the source
and target meshes, and Edef (S, S̄) seeks to limit and regularize the deformation of the source mesh away from its rigidly
registrated counterpart S̄. The term
Efit =

geometry of the reference mesh S̄:The deformation term Edef (S, S̄), derived from a physically based energy (e.g., see [5]), is a sum of three terms

NS
X

2

(dist(gi )) Āi ,

(9)

i=1

penalizes deviation of the source and target meshes. Here gi
is the barycenter of the triangle i, and dist(gi ) is the distancefrom gi to the closest point on the target mesh. As in therigid case, we use the precomputed distance field to query
for the distance.
It might appear that the fitting energy Efit could be triviallyminimized by moving each vertex of mesh S to lie on mesh
T . In practice, however, this does not work because all of thegeometry of the precomputed reference mesh S̄ is discarded;
instead, the geometry of this mesh, which was precomputed
using fabric simulation, should serve as a prior. Thus, we
introduce a second term to retain as much as possible theEdef (S, S̄) = κEarea + βEangle + αEhinge ,(10)where α, β and κ are user-specified coefficients.The term

2
NS
X
1 Ai
Earea =
− 1 Āi ,
(11)
2 Āi
i=1
penalizes changes to the area of each mesh triangle. Here Ai
is the area of the triangle i, and ¯· refers to a corresponding
quantity form the undeformed mesh S̄. The term

2
NS X
3
X
1 θik
− 1 Āi ,(12)
Eangle =6 θ̄iki=1 k=1
penalizes shearing of each mesh triangle, where θik is thekth angle of the triangle i. The term Ehinge [5]X
Ehinge =(θe − θ̄e )2 kēk/h̄e ,(13)epenalizes bending, measured by the angle formed by adjacenttriangles. Here θe is the hinge angle of edge e, i.e., the angle
formed by the normals of the two triangles incident to e; kēkis the length of the edge e, and h̄e is a third of the sum of
the heights of the two triangles incident to the edge e.
We used the secant-version of the L-M method[11] to seek
the source mesh S that minimizes the energy Eq.(8). Sample
registration results are shown in Figure 4.
B. Grasping Point Localization
We use a pre-determined anchor point (e.g., elbow on thesleeve of a sweater) to indicate a possible regrasping point.
The detection of the regrasping point can be summarized in
two steps: global localization and local refinement.
Global localization is achieved by deformable registration.
The registered simulation mesh will provide a 3D regrasping
point from the recognized state which will be then mapped
onto the reconstructed mesh.
In order to improve the regrasping success rate, we propose a step of local refinement. The point on the actual
garment may be hard to grasp for several reasons. One isthat during the garment manipulation steps, such as rotation,
the curvature over the garment may change. Another reason
is that when considering the width of robot hand gripper,a ridge curve with proper orientation and width should beselected for regrasping. We consider the proper orientation
as a direction perpendicular to the opening of the gripper.
Therefore, we propose an efficient 1D blob curvature detection algorithm that can find a refined position in the local
area over the garment surface via an IR range sensor.
In our experiment, the Baxter robot is equipped with a IR
range sensor close to the gripper as shown in Figure 6 top.
Once the gripper moves to the same height of the predicted
3D regrasping point from registration, it will perform ahorizontal scan search, moving from one side to the other,so that the IR sensor will scan over the full local curvature.

We then apply a curvature detection algorithm that convolves the IR depth signal with a fixed width kernel, where
the width is determined by the opening of the gripper. Here
we use a Laplacian-Gaussian Kernel g 00 (x):x2
1x2
− 2 )e− 2σ2
(14)
4
σ
σ
where x is the depth signal, and σ is the width parameter.

g 00 (x) = (C. Convergence
After the regrasping is finished, we evaluate the current
grasping configuration by the objective function f (·). An
f (·) greater than a given ξ means the grasping points are on
the desired positions, and the robot will then stop regraspingand enter the placing flat mode. The two arms will open toslightly stretch the garment and place it on a table.
V. E XPERIMENTAL R ESULTS
To evaluate our results, we tested our method on several
different garments such as sweater and pants for multiple
trials, as shown in Figure 7 left. A high resolution videoof our experimental results is online at http://www.cs.
columbia.edu/˜yli/ICRA2015.
A. Robot Setup
In our experiments, we use a Baxter research robot, whichis equipped with two arms with seven degrees of freedom.
To improve grasp stability and form a closed loop controller,
we add tactile sensors to each of the grippers as shown in
Figure 6. Each robot hand is equipped with a IR range sensor
which is used for local curvature detection, as described in
section IV-B. A Kinect sensor is mounted on a fixed frame
at a height of 1.2 meters to capture the depth images.
B. Garment Database and Real-time Recognition
Our recognition method consists of two stages, offlinemodel simulation and online recognition. In the offline model
simulation stage, we use a physics engine [1] to simulate
the stationary state of the mesh models of different types ofgarments in different poses. More specifically, we manuallylabel each garment in the database with the key grasping
points such as sleeve end, elbow, shoulder, chest, and waist,
etc. For each grasping point, we compute the garment layoutby hanging under gravity in the simulator. The simulated
meshes are used as the training data for 3D shape-basedmatching and pose recognition [10]. In our experiments, we
assume the category of the garment is known. Therefore, westart with the garment pose estimation.
Below we summarize the pose recognition method, details
can be found in [10]. We first pick up the garment at a
random point. In the online recognition stage, we use a
Kinect sensor to capture depth images of different views of
the garment while it is being rotated by a robotic arm. The
garment is rotated 360◦ clockwise and then 360◦ counterclockwise to obtain about 550 depth images for an accurate
reconstruction. We reconstruct a 3D mesh model from thedepth image segmentation and volumetric fusion. Then with
an efficient 3D feature extraction algorithm, we build up a

Fig. 4. Registration examples. F IRST ROW: A sweater grasped at elbow.
S ECOND ROW:A pair of pants grasped near knee. Each row depicts fromleft to right: a reconstructed mesh, the predicted mesh from the database,rigid registration only, and rigid plus non-rigid registration.

binary feature vector and finally match against the offline
database for pose recognition. One of the outputs is a highquality reconstructed mesh, which is used for 3D registration
and accurate regrasping point prediction, as described below.
C. Registration
As described in section III, we apply both rigid andnon-rigid registrations. The rigid registration step mainly
focuses on mesh rescaling and alignment, whereas the nonrigid registration step refines the results and improves the
mapping accuracy. In Figure 4, we compare the difference
between using rigid registration only and using rigid plus
non-rigid registration side by side. We can clearly see thatwith non-rigid registration, the two meshes are registered
more accurately. In addition, the location of the designatedgrasping points on the sleeves are also closer to the groundtruth points. Note that for the fourth row, after the alignmentby the rigid registration algorithm, the state is evaluated as alocal minimum. Therefore, there is no improvement by thefollowing non-rigid registration. But as we can see from thevisualization, such a case is still good enough for finding
point correspondence.
D. Search for Best Grasping Point by Local Curvature
Once we choose a potential grasping point, we can perform a search to find the best local grasping point for thegripper. We are trying to a find a fold in the vicinity of thepotential grasping point with a high local curvature tuned to
the gripper width that allows for a stable grasp. The opening
size of the gripper is approximately 8cm and empirically weset σ = 10 in the equation 14. A plot of its signal, as well as
the convoluted signal, are shown in Figure 6 left and right.
We can clearly see that the response from the filter is at aminimum where the grasping should take place. The tactile

Fig. 5. Examples of each step in our unfolding procedure. For each row from left to right is: a snapshot of initial pick up, a 3D reconstructed mesh, apredicted mesh from database, the predicted mesh with weighted Gaussian distribution distance, predicted regrasping point on the 3D reconstructed mesh,a snapshot of regrasping, and finally a snapshot of unfolding. T OP ROW: The Baxter robot unfolds a sweater following pick up. B OTTOM ROW: The
Bater robot unfolds a pair of pants following pick up.

sensors then assure that the gripper has properly closed on
the fabric.
−5

0.4

10

x 10

8

0.3

6
Convolution results

Distance to garment surface

IR range scan
0.35

0.25

0.2

4

2

0.15

0

0.1

−2

0.05

0.35

0.4
0.45
0.5
Gripper’s position along robot X−axis

0.55

−4

0.35

0.4
0.45
Gripper’s position along robot X−axis

0.5

0.55

Fig. 6. IR range sensor scan example. L EFT: Single reading plot fromthe IR range sensor. R IGHT: Convoluted result of the sensor reading and a
Laplacian-Gaussian kernel with different kernel size. The lowest point (inred) is the place the gripper should grasp.

E. Iterative regrasping
Figure 5 shows two examples (sweater and pants) of
iterative regrasping using the Baxter robot. The robot first
picks up a garment at a random grasping point. Once thearm reaches a pre-defined position, the last joint of thearm starts to rotate and the Kinect will capture the depth
images as it rotates, and reconstruct the 3D mesh in realtime. After the rotation, a predicted pose is recognized [10]as shown in the third image of each row. For each pose,
we have a constrained weighted evaluation metric over thesurface to identify the regrasping point as indicated in the
fourth image. By registration of the reconstructed mesh andpredicted mesh from the database, we can map the desired
regrasping point onto the reconstructed mesh. The robot thenregrasps by moving the other gripper towards it. With our 1D

blob curvature detection method, the gripper can move to thebest curvature on the garment and regrasp, which increases
the success rate. The iterative regrasping stops when thetwo grasped points are the designated anchor points on thegarment (e.g., elbows on the sleeves of a sweater).
Figure 7 left shows 7 sample garments in our test, andthe table on the right shows the results. For each garment,
we perform 10 unfolding tests. We have on average an 83%
successful recognition rate for the pose of the objects over
all the garments. We have on average an 87% successful regrasping rate for each garments, where regrasping is defined
as a successful grasp of the other arm on the garment. 80%of the time we are able to successfully unfold the garment,
placing the grippers at the designated grasping points.
Unsuccessful unfolding occurred when either the gripper
lost contact with the garment, or the gripper was unable tofind a regrasping point. Although we did not perform thisexperiment, it is possible to restart the method after one of
the grippers loses contact as an error recovery procedure.
For the successful unfolding cases, we also report the average number of regrasping attempts. The minimum numberof regrasping attempts = 1. This happens when the initial
grasping is at one of the desired positions, and the regrasping
succeeds at the other desired position (i.e., two elbows on
the sleeves for a sweater). In most cases, we are able tosuccessfully unfold the garments using 1 − 2 regraspings.
Among all these garments, jeans, pants, and leggingsachieve high success rate because of their unique layout
when grasping at the leg position. The shorts are difficult
for both recognition and unfolding steps possibly because
its ambiguous appearances in different grasping points. One

Garment

# of
Trial

Successful
Recognition

Successful
Regrasping

Successful
UnfoldingSweatshirt
Sweater
Knitwear
Jeans
Pants
Leggings
ShortsAverage

10
10
10
10
10
10
10
10

9/10
8/10
9/10
9/10
8/10
8/10
7/10
8.3/10

8/10
8/10
9/10
9/10
10/10
9/10
8/10
8.7/10

8/10
7/10
8/10
9/10
9/10
8/10
7/10
8.0/10Avg. # of
Regrasps
Success Only
1.6
1.6
1.7
1.3
1.4
1.4
1.9
1.6

Fig. 7. L EFT: A picture of our test garments. R IGHT: Results for each unfolding test on the garments. We evaluate the results by recognition, regrasping,
unfolding, and regrasping attempts for each test. The last row shows the average of each evaluation component.

observation is that in a few cases, when the recognition is
not accurate, our registration algorithm was sometimes able
to find a desired regrasping point for unfolding. This is an
artifact of the geometry of pant-like garments where thedesignated regrasping points are at the extreme locations on
the garments.
We also show that after grasping at two desired points, therobot will proceed to place the garment on a table. In our
experiments, we use cardboard to simulate a table area. As
shown in Figure 8, the robot is able to place the garment flatwith a simple move when grasping at a pair of two desired
grasping points. With such a flat configuration, the robot canbegin to fold it, which is part of our future work.

Fig. 8. The Baxter robot places a garment flat on a table. L EFT: The
garment is a sweater and the two desired grasping points are on the sleeves.
R IGHT: The garment is a pair of pants and the two desired grasping points
are on the lower leg parts.

VI. C ONCLUSION
In this paper, we propose a novel solution for the problem of unfolding a garment to a desired configuration via
regrasping. The unfolding procedure contains garment pickup, pose estimation, regrasping, and placing flat on a table.
We use simulated predictive thin shell models for garments
to automatically create a large database of garments andtheir poses, which can be used in a learning algorithm tofind object pose. We also create a real-time 3D volumetricmodel from 3D scanning which can be used for registration
with the simulated models. The regrasping point is foundby a fast, two-stage deformable object registration algorithm
that uses a novel non-rigid registration method to minimize
energy differences between source and target mesh models
The unfolding state is determined by a constrained weighted
objective function for the current two grasping points. Experimental results show that our method is able to unfold anumber of common garments with a high success rate.

Acknowledgments We’d like to thank J. Weisz, A. Garg,and J. Augustin for many discussions. We’d also like tothank NVidia Corporation, Intel Corporation, and Takktile
LLC for the hardware support. This material is based uponwork supported by the National Science Foundation underGrant No. 1217904 and in part by the JSPS Postdoctoral
Fellowships for Research Abroad.
R EFERENCES[1] Maya, http://www.autodesk.com/products/autodesk-maya/.
[2] P. J. Besl and N. D. McKay. A method for registration of 3-d shapes.
IEEE Trans. Pattern Anal. Mach. Intell., 14(2):239–256, feb 1992.
[3] M. Cusumano-Towner, A. Singh, S. Miller, J. F. OBrien, and P. Abbeel.
Bringing clothing into desired configurations with limited perception.
In Proc. ICRA, 2011.
[4] A. Doumanoglou, A. Kargakos, T-K Kim, and S. Malassiotis. Autonomous active recognition and unfolding of clothes using random
decision forests and probabilistic planning. In Proc. ICRA, May 2014.
[5] E. Grinspun, A. N. Hirani, M. Desbrun, and P. Schröder. Discrete
shells. In Proceedings of the 2003 ACM SIGGRAPH/Eurographics
Symposium on Computer Animation, pages 62–67, Aire-la-Ville,
Switzerland, Switzerland, 2003.
[6] Y. Kita and N. Kita. A model-driven method of estimating the stateof clothes for manipulating it. In Proc. WACV, 2002.
[7] Y. Kita, T. Ueshiba, E-S Neo, and N. Kita. Clothes state recognition
using 3d observed data. In Proc. ICRA, 2011.
[8] H. Li, R. W. Sumner, and M. Pauly. Global correspondence optimization for non-rigid registration of depth scans. In Proceedings of the
Symposium on Geometry Processing, pages 1421–1430, 2008.
[9] Y. Li, C-F Chen, and P. K. Allen. Recognition of deformable object
category and pose. In Proc. ICRA, June 2014.
[10] Y. Li, Y. Wang, M. Case, S-F Chang, and P. K. Allen. Real-timepose estimation of deformable objects using a volumetric approach.
In Proc. IROS, September 2014.
[11] K. Madsen, H. B. Nielsen, and O. Tingleff. Methods for non linear
least squares problems. Tech Report, 2004.
[12] J. Maitin-Shepard, M. Cusumano-Towner, J. Lei, and P. Abbeel. Cloth
grasp point detection based on multiple-view geometric cues withapplication to robotic towel folding. In Proc. ICRA, 2010.
[13] S. Miller, J. Berg, M. Fritz, T. Darrell, K. Goldberg, and P. Abbeel.
A geometric approach to robotic laundry folding. IJRR, 2012.
[14] F. Osawa, H. Seki, and Y. Kamiya. Unfolding of massive laundry andclassification types by dual manipulator. Journal of Advanced andIntelligent Informatics, 11(5), 2007.
[15] G. K. L. Tam, Z-Q Cheng, Y-K Lai, F. C. Langbein, Y. Liu, D. Marshall, R. R. Martin, X-F Sun, and P. L. Rosin. Registration of 3d point
clouds and meshes: A survey from rigid to nonrigid. Visualization andComputer Graphics, IEEE Transactions on, 19(7), July 2013.
[16] Y-H R. Tsai. Rapid and accurate computation of the distance function
using grids. Journal of Computational Physics, 178(1), 2002.
[17] P-C Wang, S. Miller, M. Fritz, T. Darrell, and P. Abbeel. Perception
for the manipulation of socks. Proc. IROS, 2011.
[18] B. Willimon, S. Birchfield, and I. Walker. Classification of clothing
using interactive perception. In Proc. ICRA, 2011.
[19] B. Willimon, I. Walker, and S. Birchfield. A new approach to clothingclassification using mid-level layers. In Proc. ICRA, 2013.