Goal-Conditioned Model Simplification for
Deformable Object Manipulation
Shengyin Wang∗ , Matteo Leonetti† , Mehmet Dogar∗∗ School of Computing, University of Leeds, Leeds, United Kingdom

Email: scswan@leeds.ac.uk, m.r.dogar@leeds.ac.uk† Department of Informatics, King’s College London, London, United Kingdom

Email: matteo.leonetti@kcl.ac.uk

Abstract—Deformable object manipulation has been a challenge for a long time in robotics due to its high-dimensional
configuration space and complex dynamics. In this work, we
explore the idea of goal-conditioned model simplification which
has a great potential to improve motion planning, perception, andpolicy learning. Two workflows are proposed for objects that canbe approximated by lines (1D linear model) and surfaces (2D flatmodel) respectively to find the minimal number of key points
which are used to reduce the action search space for motion
planners. Representative tests on ropes and cloth are conductedwhich demonstrate the effectiveness of the proposed method.

I. I NTRODUCTION & R ELATED W ORK
Due to the ubiquitous existence of deformable objects andtheir extensive applications [1], [2], there has been a great
deal of work dedicated to the development of deformableobject manipulation methods. Detailed survey of deformablemodelling, planning, control, and learning can be found by
Zhu [3], Yin [4], Bhagat [5], and Arriola-Rios [6].
The intrinsic high-dimensional configuration space of deformable objects make both perception and planning complex
problems, especially for traditional search or optimizationbased methods [7]–[9]. To circumvent this problem, earlier
work focused on predefined features such as key points [10],contours [11], or wrinkles [12]. However, defining such features requires expert knowledge about the deformable object.
Learning based methods are also employed to extract a compact representation of the deformable object directly from raw
sensor inputs [13], and achieve faster planning, however thelatent space is not explainable or intelligible to humans. Although researchers proposed various schemes to alleviate thechallenges posed by high-dimensional configuration spaces,
they are mostly designed for a particular task [11] or requirelarge amount of training data [14].
Another challenge for deformable-object manipulation isthat the dynamics of the object is much more complex thanrigid objects. Unlike rigid objects, it is difficult to derive
an explicit model for deformable objects to describe theirdynamical behaviors due to high-dimensionality of the configuration space, material characteristics, and contacts withenvironments [13]. Several simulators have been developed
based on mass-spring systems, particle based dynamics, or
finite element models [6], to support deformable object manipulation, including softgym [15], mujoco [16], and pybullet[17]. Although these simulators achieve similar deformablebehaviors to the real deformable object, planning motions orlearning policies [18]–[20] based on high fidelity simulators
is still time consuming due to a vast number of rollouts. To
reduce the time cost, learning based methods [21] are employed to learn the dynamic model of the deformable objects
either in pixel space [22] or latent space [13] before rolling out
for planning. However, the interpretability of these models ispoor, and the number of samples required for training a neuralnetwork model is still forbiddingly high. Power et al. [23] use
simple models to improve learning efficiency for controllingcomplex systems. Nonetheless, only a few tasks are performed
and choosing the simple model manually is non-trivial.
As pointed out by Lin et al. [15], a compact representationthat captures key features relevant to the goal could makeplaning faster and learning more accurate. For example, when
flattening cloth, four corners are enough for both representing
the state and achieving the task, whereas for side folding tasks,
additional two middle points are indispensable for specifying
the goal state and can give intuitive information for action
selection. Thus, our key idea to circumvent these problems
is designing a goal-conditioned model simplification scheme
to automatically extract key features that are most relevant toperform the task while keeping the representation as concise
as possible. Centering on this idea, we developed line-fitting
and mesh simplification methods for linear and flat deformableobjects separately. Once the key points are acquired, we use it
to simplify the action space for a motion planer, which shows
better performance and faster convergence speed for all the
given deformable manipulation tasks on both ropes and cloth.
II. P ROBLEM F ORMULATION
Considering a deformable object modelled by a mass-spring
system, in which a grid of mass points are subjected to gravity,
Hookean spring force, and contact forces, the full state of
the object when stabilized can be described by the positionof all mass points. Suppose there are No mass points in theobject model, each of which is deemed as a particle indexed
by i, (i = 1, ..., No ) with pi = [xi , yi , zi ] representing theposition of the ith particle. Then, the configuration at time t
can be defined as ξt := [pi | ∀i ∈ {1, ..., No }].

To simplify the planning problem, we assume the robot
pickers can pick any given particle in the object. The available
actions at each time t move a particle with the picker by acertain distance along a target direction. To let the planner
allow a picker not to pick any particle, N one is added forgrasping nothing, thus the action at time t can be given as:(⟨pi , δxt , δyt , δzt ⟩
(k)
at =
N one
(1)
h
i
(1) (2)
at = at , atwhere the superscript k denotes the k th picker, in this work weassume two pickers k ∈ {1, 2}. We constrain the movement of
the picker complying to the quasi-static assumption: |δxt | <=∆x, |δyt | <= ∆y, |δzt | <= ∆z; ∆x, ∆y, and ∆z are motion
limits along each axis in Cartesian space. Thus, the originalaction space has size (2 × No × R3 + 2).
The planning problem is to find a sequence of Na actions,τ = a1 , a2 , ..., aNa , that minimizes a goal-conditioned costfunction cg (τ ). For instance, the goal can be a triangle shapefor rope folding and a diagonally folded shape for cloth folding(last column in Fig. 5). Admissible action sequences are thosethat do not cause the object to stretch excessively. Formally,
the optimization problem solved by the planner is:min
cg (τ )
(2)
C(at ) < 0, ∀t ∈ {1, ..., Na },s.t.
where C(at ) is the stretching constraint for the action at stept, as defined in the softgym simulator in practice.
III. M ETHODOLOGY
To solve the planning problem defined in Equation 2 weused Cross Entropy Method (CEM). However, due to the highdimensional action space, CEM requires a large number ofrollouts. To reduce the complexity of the planning problem,
we simplify the model by reducing the number of graspable
points based on the goal configuration.
A. Model Simplification
Given the goal configuration of the deformable object, we
aim to find the smallest accurate model that contains themost important and relevant key particles to achieve the task.
In most cases, the number of key particles, Ns , is muchsmaller than the original number, i.e. Ns ≪ No , making
the action space of the planner much smaller. To extractsuch particles from the given deformable object, we used two
different methods, depending on whether the object can beeffectively approximated by a one or two-dimensional entity.
In the former case we used piece-wise line-fitting, while in
the latter case we used a mesh-simplification method. It is
worth noting that manipulation still happens in 3D space while
objects can be approximated by a set of lines or surfaces.
1) Line Fitting for Ropes: We start with two key points,
Ns = 2, the positions of which can be anywhere in the space,and iteratively increase the number of key points (see Fig. 1)
until we have a good fit to the goal configuration. At each
iteration, (i.e. for each value of Ns ), we solve a Quadratic

Figure 1. Linear fitting (in red) for the goal rope configuration (in blue).

Programming (QP) problem, to find the optimal piece-wiseline fit, which minimizes the error:
XNoiNs( )])2 ,E=(pi − fj=0
(3)
n=1
No
Ns
(·) denotes the piece-wise line equation interpowhere fj=0
lated from the given key points qj , j ∈ {1, ..., Ns }, with
Ns
Ns
(1) as the last(0) representing the first key point, fj=0
fj=0
Ns
th
key point, and fj=0 (i/No ) representing the i point on thefitted line corresponding to the ith particle on the original rope.
For example, in Fig. 1, the blue dots represent the original
particles, the red stars represent the simplified key particles,
and the interpolated particle qj can be extracted from the red
dotted line. The iterative process stops at the smallest numberof key points, Ns , for which the error E is below a thresholdτl . For example in Fig. 1, our process stops at the rightmost
iteration with Ns = 5.
2) Mesh Simplification for Cloth: For objects that can beapproximated by surfaces, we rely on a widely used meshprocessing toolkit called Meshlab [24], and use its built-in
module named Quadric Edge Collapse Decimation [25] tosimplify the meshes to a given number of triangles (startingfrom two triangles with four key points). To measure the errorof the simplified mesh with respect to the original one, (i.e.
in place of the error E above), we use the Hausdorff distance[26]. Similarly to the rope simplification, an iterative process
stops at the smallest number of triangles whose error is below
a threshold τs .
B. CEM based Planning
CEM [27] is a well established optimization algorithm,
which has been applied to address plenty of manipulation
problems including deformable objects. To demonstrate theeffectiveness of the proposed model simplification methods,
CEM-Original (all points on the model are graspable) and
CEM-Simple (only extracted key points are graspable) are
implemented on a set of tasks for rope and cloth manipulation.
IV. E XPERIMENTS
To evaluate the effectiveness of the proposed method, a setof experiments are conducted to illustrate how the simplified
model can be used to improve the performance of motion
planning for deformable objects covering both ropes and cloth.
CEM-Original is implemented as a benchmark method tocompare the performance of CEM-Simple. The simulator usedin this paper is based on softgym [15] which is built on topof PyFlex [28] provided by Nvidia. All tests are conducted ona workstation with Intel Xeon(R) CPU E5-2665 @2.40GHz,
32 GB of RAM, and Nvidia Quadro K4000 GPU.
A. Tasks
1) Rope Straightening [15]: This task refers manipulatinga rope from a randomized initial configuration to a slack
and straightened goal configuration. The cost function is theaccumulation of the absolute difference between the current
distance of the endpoints and the original length of the rope.
2) Rope Folding [15]: For the task of Rope Folding, thetwo pickers need to deform the rope into a specific shape,a triangle in this test, from an initially straightened state.
The cost is also an accumulative quantity at each step, whichis computed by the bipartite matching distance between thecurrent positions and the goal positions of all particles.
3) Cloth Side Folding [15]: Cloth Side Folding task aims
to fold a flattened cloth into half sideways. The cost functionis the accumulative step costs composed by two parts: one isthe distance of the corresponding particles between the two
halves; another is the displacement of the still half from its
initial state which penalizes the cloth being dragged away.
4) Cloth Diagonal Folding [29]: This task is very similar to
the Cloth Side Folding task, with the only difference of using
the diagonal halves instead of sideways halves. The structureof the cost function is also same by replacing sideways
correspondence with diagonal matching relations.

Figure 2. Model simplification costs (first row: rope straightening & folding;second row: cloth side & diagonal folding)Figure 3. Simplified models (rope straightening & folding; cloth side &diagonal folding)

B. Simulation Results
1) Model Simplification: Model simplification costs and thesimplified models are shown in Fig. 2 and 3 respectively. As
the results show, only the two end points are necessary for therope straightening task, while another middle point is required
for the folding task. Mesh simplification for cloth is much
more complex than rope simplification: a simplified mesh with
only six points and six triangles can be used to approximate
a complex mesh when folded in half sideways; meanwhile, apiece of diagonally folded cloth can be approximated using
four points and four triangles. In this way, the points that are
most relevant to complete the task are extracted to facilitatemotion planning.
2) CEM based Motion Planning: Based on the extracted
key points, CEM-Simple is implemented to demonstrate its
effectiveness compared to CEM-Original. For rope manipulation, each task is tested for ten times, while only one roundof optimization is conducted for cloth manipulation using thebuilt-in CEM method in softgym due to the time cost. Planningcosts per iteration during optimization are shown in Fig. 4,
where CEM-Simple achieves lower cost and faster convergence than CEM-Original for all tasks. As the results show, forrope manipulation tasks, CEM-Original achieves similar costto CEM-Simple because the dimensionality difference between
them is much smaller than those for cloth manipulation tasks
where CEM-Original cannot find a feasible plan while CEMSimple succeeds with the same number of simulation steps.
Snapshots for rope folding and cloth diagonal folding tasks
are shown in Fig. 5, which validate that the proposed method
can find more effective motion plans for rope manipulation,
and achieve complex cloth folding tasks whereas using theoriginal action space failed.

Figure 5. Motion plans: rope folding based on CEM-Simple & CEM-Original;
cloth diagonal folding based on CEM-Simple & CEM-Original (first column
as initial state, the last two columns as final state and goal state respectively;rope straightening and cloth side folding are omitted due to limited space)

V. C ONCLUSION
In this paper, we proposed model simplification methods forropes and cloth, and established a deformable object manipulation scheme based on goal conditioned model simplification

which shows better performance in several tasks. Currently,
we are doing research on hierarchical planning to address thecase where only the final goal is not enough to achieve the

Figure 4. Motion planning cost (first row: rope straightening & folding;second row: cloth side & diagonal folding)task and intermediate sub-goals are required. Future work will
focus on model simplification for learning and planning.
R EFERENCES[1] D. Seita, N. Jamali, M. Laskey, A. K. Tanwani, R. Berenstein,
P. Baskaran, S. Iba, J. Canny, and K. Goldberg, “Deep transfer learningof pick points on fabric for robot bed-making,” in The International
Symposium of Robotics Research. Springer, 2019, pp. 275–290.
[2] X. Li, X. Su, and Y.-H. Liu, “Vision-based robotic manipulation offlexible pcbs,” IEEE/ASME Transactions on Mechatronics, vol. 23, no. 6,
pp. 2739–2749, 2018.
[3] J. Zhu, A. Cherubini, C. Dune, D. Navarro-Alarcon, F. Alambeigi,
D. Berenson, F. Ficuciello, K. Harada, X. Li, J. Pan et al., “Challenges
and outlook in robotic manipulation of deformable objects,” arXiv
preprint arXiv:2105.01767, 2021.
[4] H. Yin, A. Varava, and D. Kragic, “Modeling, learning, perception, andcontrol methods for deformable object manipulation,” Science Robotics,
vol. 6, no. 54, p. eabd8803, 2021.
[5] S. Bhagat, H. Banerjee, Z. T. Ho Tse, and H. Ren, “Deep reinforcement learning for soft, flexible robots: Brief review with impending
challenges,” Robotics, vol. 8, no. 1, p. 4, 2019.
[6] V. E. Arriola-Rios, P. Guler, F. Ficuciello, D. Kragic, B. Siciliano, andJ. L. Wyatt, “Modeling of deformable objects for robotic manipulation:
A tutorial and review,” Frontiers in Robotics and AI, vol. 7, p. 82, 2020.
[7] P. Jiménez, “Survey on model-based manipulation planning of deformable objects,” Robotics and computer-integrated manufacturing,
vol. 28, no. 2, pp. 154–163, 2012.
[8] A. Doumanoglou, A. Kargakos, T.-K. Kim, and S. Malassiotis, “Autonomous active recognition and unfolding of clothes using random
decision forests and probabilistic planning,” in 2014 IEEE international
conference on robotics and automation (ICRA). IEEE, 2014, pp. 987–
993.
[9] Y. Li, Y. Yue, D. Xu, E. Grinspun, and P. K. Allen, “Folding deformableobjects using predictive simulation and trajectory optimization,” in 2015
IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS). IEEE, 2015, pp. 6000–6006.
[10] J. Maitin-Shepard, M. Cusumano-Towner, J. Lei, and P. Abbeel, “Cloth
grasp point detection based on multiple-view geometric cues with application to robotic towel folding,” in 2010 IEEE International Conferenceon Robotics and Automation. IEEE, 2010, pp. 2308–2315.
[11] S. Miller, J. Van Den Berg, M. Fritz, T. Darrell, K. Goldberg, and
P. Abbeel, “A geometric approach to robotic laundry folding,” TheInternational Journal of Robotics Research, vol. 31, no. 2, pp. 249–
267, 2012.
[12] L. Sun, G. Aragon-Camarasa, P. Cockshott, S. Rogers, and J. P.
Siebert, “A heuristic-based approach for flattening wrinkled clothes,”in Conference Towards Autonomous Robotic Systems. Springer, 2013,
pp. 148–160.
[13] W. Yan, A. Vangipuram, P. Abbeel, and L. Pinto, “Learning predictive
representations for deformable objects using contrastive estimation,”
arXiv preprint arXiv:2003.05436, 2020.
[14] P. Zhou, J. Zhu, S. Huo, and D. Navarro-Alarcon, “Lasesom: A latentand semantic representation framework for soft object manipulation,”
IEEE Robotics and Automation Letters, vol. 6, no. 3, pp. 5381–5388,
2021.
[15] X. Lin, Y. Wang, J. Olkin, and D. Held, “Softgym: Benchmarking
deep reinforcement learning for deformable object manipulation,” arXiv
preprint arXiv:2011.07215, 2020.
[16] E. Todorov, T. Erez, and Y. Tassa, “Mujoco: A physics engine for modelbased control,” in 2012 IEEE/RSJ international conference on intelligent
robots and systems. IEEE, 2012, pp. 5026–5033.
[17] C. Erwin and B. Yunfei, “Pybullet a python module for physics simulation for games,” PyBullet, 2016.
[18] D. Seita, A. Ganapathi, R. Hoque, M. Hwang, E. Cen, A. K. Tanwani,
A. Balakrishna, B. Thananjeyan, J. Ichnowski, N. Jamali et al., “Deepimitation learning of sequential fabric smoothing from an algorithmic
supervisor,” in 2020 IEEE/RSJ International Conference on Intelligent
Robots and Systems (IROS). IEEE, 2020, pp. 9651–9658.
[19] J. Matas, S. James, and A. J. Davison, “Sim-to-real reinforcement
learning for deformable object manipulation,” in Conference on Robot
Learning. PMLR, 2018, pp. 734–743.

[20] R. Jangir, G. Alenya, and C. Torras, “Dynamic cloth manipulation withdeep reinforcement learning,” in 2020 IEEE International Conferenceon Robotics and Automation (ICRA). IEEE, 2020, pp. 4630–4636.
[21] X. Ma, D. Hsu, and W. S. Lee, “Learning latent graph dynamics
for deformable object manipulation,” arXiv preprint arXiv:2104.12149,
2021.
[22] R. Hoque, D. Seita, A. Balakrishna, A. Ganapathi, A. K. Tanwani,
N. Jamali, K. Yamane, S. Iba, and K. Goldberg, “Visuospatial foresight for multi-step, multi-task fabric manipulation,” arXiv preprint
arXiv:2003.09044, 2020.
[23] T. Power and D. Berenson, “Keep it simple: Data-efficient learning forcontrolling complex systems with simple models,” IEEE Robotics andAutomation Letters, vol. 6, no. 2, pp. 1184–1191, 2021.
[24] P. Cignoni, M. Callieri, M. Corsini, M. Dellepiane, F. Ganovelli,
G. Ranzuglia et al., “Meshlab: an open-source mesh processing tool.”in Eurographics Italian chapter conference, vol. 2008. Salerno, Italy,
2008, pp. 129–136.
[25] M. Garland and P. S. Heckbert, “Surface simplification using quadric error metrics,” in Proceedings of the 24th annual conference on Computer
graphics and interactive techniques, 1997, pp. 209–216.
[26] N. Aspert, D. Santa-Cruz, and T. Ebrahimi, “Mesh: Measuring errors
between surfaces using the hausdorff distance,” in Proceedings. IEEE
international conference on multimedia and expo, vol. 1. IEEE, 2002,
pp. 705–708.
[27] R. Rubinstein, “The cross-entropy method for combinatorial and continuous optimization,” Methodology and computing in applied probability,
vol. 1, no. 2, pp. 127–190, 1999.
[28] Y. Li, J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba, “Learning
particle dynamics for manipulating rigid bodies, deformable objects, andfluids,” arXiv preprint arXiv:1810.01566, 2018.
[29] R. Hoque, D. Seita, A. Balakrishna, A. Ganapathi, A. K. Tanwani, N. Jamali, K. Yamane, S. Iba, and K. Goldberg, “Visuospatial foresight forphysical sequential fabric manipulation,” Autonomous Robots, vol. 46,
no. 1, pp. 175–199, 2022.