Cloth Manipulation and Perception Competition
Irene Garcia-Camacho1 , Júlia Borràs1 , Berk Calli2 , Adam Norton3 and Guillem Alenyà1

Abstract— In the last decade, several competitions in robotic
manipulation have been organised as a way to drive scientificprogress in the field. They enable comparison of different
approaches through a well-defined benchmark with equal
test conditions. However, current competitions usually focus
on rigid-object manipulation, leaving behind the challenges
that suppose grasping deformable objects, especially highlydeformable ones as cloth-like objects. In this paper, we wantto present the first competition in perception and manipulationof textile objects as an efficient method to accelerate scientificprogress in the domain of domestic service robots. To do so,
we selected a small set of tasks to benchmark in a common
framework using the same set of objects and assessment methods. This competition has been conceived to freely distributethe Household Cloth Object Set to research groups working
on cloth manipulation and perception and participate on thechallenge. In this work, we present an overview of the tasks
that are proposed in the competition, detailed descriptions of thetasks and more information on the scoring and rules are provided in the website http://www.iri.upc.edu/groups/
perception/ClothManipulationChallenge/.

I. I NTRODUCTION
Establishing well-defined benchmarks that serves to evaluate different robotic manipulations strategies across researchgroups in a standard manner is still a pending challenge inrobotics. Several initiatives of benchmarks for grasping andmanipulation have been proposed in recent years, including
the definition of standard object sets such as the wellknown YCB Object Set [1], which goal was to facilitate
the fair comparison of approaches among the community ofmanipulation research. However, a benchmark is successfulwhen it is widely accepted and used by the community
at which it is targeted, what is difficult since applying
benchmarking protocols is a thorough and time-consuming
process and current benchmarks are usually too specific forthe tackled task. The best practice to reach this acceptancewithin a scientific community is to arise the benchmark fromthe community itself. For this reason, the approach chosen
by the authors of the YCB object set to let others know
about their set was to distribute several of them at a seriesof tutorials at robotic conferences. Another good approachto reach acceptance is to get active participation through theorganisation of robotic competitions.
This work receives funding by MCIN/ AEI /10.13039/501100011033 andby the ”European Union” under project BURG (PCI2019-103447).
1 Institut de Robòtica i Informàtica Industrial, CSIC-UPC {igarcia,
jborras, galenya}@iri.upc.edu
2Worcester
Polytechnic
Institute,
Massachusetts,
USA

bcalli@wpi.edu
3Universityofadam norton@uml.edu

Massachusetts

Lowell,USA

Similarly to the approach followed by the YCB authors,
the motivation of the presented competition is to distributethe Household Cloth Object Set [2] among research groups
working on the field. In addition, we want to bring together researchers in a common setup using these objects
to benchmark the performance of autonomous manipulationand perception solutions involving textile objects acrosshousehold applications. This object set was created as anextension of the YCB object set targeting textile objects tofoster benchmarking in textile manipulation and therefore
support progress in this field, what can further push towards
developing service robots that can operate in unstructured
human environments.
Our objective is to attract the participation of both perception and manipulation groups working with cloth-likeobjects. For this reason, the tasks that participants will tackle
are divided into two tracks; (1) perception and (2) manipulation. Teams will be able to participate on one or severalof the tasks, since together form a complete pipeline. This
competition is aimed to be held as part of the Service Trackof the Robotic Grasping and Manipulation Competition [3]that will take place at the next IROS 2022, in order toincrease reachability and ease the participation of teams. We
believe that putting efforts on defining test conditions that canbe replicated at any robotic lab seems a reasonable approachto enhance reproducibility and comparability of results inrobotics in the near future.
Based on the current challenges in manipulation of deformable objects, we selected a small set of tasks that include
problems that have already been tackled in literature but thatstill require efforts to be considered as solved problems, forexample grasp point detection [4–6] and folding of clothes[6–8]. With this competition we provide the first attemptof comparing current research approaches in perception andmanipulation under the same test conditions in regards ofobjects used and assessment measures, providing relevant
results in benchmarking cloth manipulation.
Our objectives are twofold:To organize a robotic competition focusing on theperception and manipulation challenges of highly deformable objects such as textiles to bring together
research groups in a common framework.
• To distribute of the Household Cloth Object Set toresearch groups of the field to set the grounds on using
common objects to foster benchmarking in manipulationof textiles.
•

II. R ELATED WORK
A. Robotic manipulation competitions
Due to their potential to accelerate research activities
effectively, several challenge programs have been organisedin the past years focusing on manipulation of objects inrealistic applications as industrial or service scenarios. Someof the more popular are the Amazon Picking Challenge [9],the RoboCub@Home [10] or the ARM project of DARPA.
Nevertheless, these competitions are usually focused on
manipulation of rigid objects, lacking a wide range of
deformable objects, specially cloth-like objects, that suppose
an increase on the challenges to solve. To the best of
our knowledge, there has been no textile perception andmanipulation competitions. This can be due to the highcomplexity that suppose highly deformable objects since it
breaks fundamental assumptions in robotics such as rigidity,
low-dimensional state space and known dynamic models.
Regarding benchmarking, in similar way as with competitions, we find very few works that have attempted to definea benchmark focused on cloth manipulation. A recent work
is presented in [11], where benchmarks for three bimanualcloth manipulation tasks are presented.
B. Challenges in deformable object manipulation
During the past years, several surveys [12–14] have collected the main challenges that appear according to the different components of which manipulation systems is composed,such as sensing, control, modelling or gripper design. Mostof these surveys considers state estimation of deformableobjects as one of the most important challenges, speciallywhen tackling cloth-like objects due to the problems indealing with self-occlusions and generating meaningful representation of their configurations. Many of the works that
described approaches to deal with this issue [6, 7, 15]rely on holding the garment in the air to recognise thegarment’s configuration. However, before grasping the fabric
it is first necessary to locate suitable grasping points. This isconsidered another challenge in perception, finding several
works that used different perception techniques to solve it
according to the cloth state and manipulation goal [4, 6, 16,17]. Regarding grasping and manipulation, the most common
identified challenges rely on the capabilities of precision in
grasping, so that the robot is capable of firmly grasp single
layers of a cloth before it bends under the contact of thegripper itself. This issue has been currently dealt in someworks with the design of custom grippers [18] that reduce
the need for precision. Another issue comes from the fact
that, in contrast to rigid object manipulation where planners
generate motions that avoid contacts, in textile manipulation
is usually quite indispensable to consider contacts with theenvironment in order to properly grasp the fabric in many
configurations, for example to grasp a flat cloth that lays ona table [8, 19]. In addition, in order to properly manipulate
an object it is necessary to plan a sequence of manipulations
that brings the cloth into the desired configurations, what
is also a challenging problem mainly due to the difficulty

in reasoning about a deformable object at a semantic level.
A recent research [20] have worked on a way to represent
states of cloth in a compact manner in order to enable
clear representations of cloth manipulation tasks for planningpurposes.
III. C OMPETITION DESCRIPTIONIn this section we present an overview of the proposed
tasks for the competition, which have been selected basedon the current trends and challenges in textile manipulationand perception. More information regarding the organisation,
instructions on how to register to participate and competition
dates is available in the related website 1 , as well as detailed
descriptions on the scoring and rules of each task.
The present competition is comprised of two tracks; (1)perception and (2) manipulation. On one hand, the first trackis composed of a task which solution provides the main
perception skills necessary to manipulate clothes. Dependingon the methodology used to solve the task, several perception
challenges can be solved, including cloth segmentation, state
recognition or feature detection. On the other hand, themanipulation track includes two of the most common cloth
manipulation tasks, which require of basic manipulation
skills to perform them. Some of the manipulation challenges
to be solved to perform these tasks includes precision in thegrasping, plan the approach to the cloth so it does not deform
before it is grasped correctly, possibility of environmental
contact (in cases where the corner is touching the table), etc.
We consider a common setup for all the tasks. The objects
that will be used are part of the household cloth object
set, having objects of different sizes, colors and textures. In
addition, a table of minimum dimensions will be necessary
to perform the tasks. We do not restrict the characteristicsof the table, but any necessary considerations (e.g. specificcolor of the table) will have to be reported. Any perception
system can be used, but a bonus scoring will be added for
those perception systems which are part of a robotic system(e.g. camera placed on the head of a mobile manipulator
or included in a robotic arm). In similar way, any robotic
system with grasping capabilities can be used in the tasks ofthe manipulation track. This includes no restrictions on thegripper hardware, being able to use custom design grippers
that ease the manipulation of textiles.
A. Perception Track
Before being able to grasp an object it is first necessary toperceive it and locate a suitable grasping point. This is doneby sensing the environment, usually with vision systems to
obtain global information. We propose this task, where wewant to assess not only the ability of detecting the corners
and edges of the cloth but also the correct selection ofgrasping points according to the cloth configuration.
1 http://www.iri.upc.edu/groups/perception/ClothManipulationChallenge/

1) Task 1.1. Grasp point detection: This task consists
on selecting a suitable grasping point on a cloth that canbe either crumpled, folded or flat on a table. The selected
point must correspond to a graspable corner of a single layerof the cloth, providing the necessary information, such as
coordinates and approach direction vector. To do so, it isnecessary to first detect their relevant features as the corners
and edges.
Steps:
1) Place the cloth on top of the table. Follow the protocol
guidelines to place them in the indicated configuration.
2) Start the perception system.
3) Record the input images and point cloud clearly marking the selected point and the detected edges andcorners.
4) Repeat steps 1 to 3 three times changing the object andits configuration.
Rules and Evaluation: The assessment of this task willconsist on the correct selection of a suitable grasping point.
To do so, it will be necessary to provide the results of thedetected features of the cloth (edges and corners) and thecoordinates of the grasping point, as well as the approach
vector. The total execution time will be also taken into
account in order to limit it to a reasonable time. Notice that as
this task is aimed to assess the perception capabilities, during
the competition the objects must be used in its original form,
not being allowed to modify their properties in any way or
add any type of markers that ease the perception.
B. Manipulation Track
This track focuses on the manipulation skills for textiles.
The tasks of this track are based on two cloth manipulation
tasks that are most studied in literature [21]; folding and unfolding. These two tasks include several manipulation skillspresent in many other cloth manipulation tasks. Both tasks
can be performed and evaluated independently but teams areencourage to execute them continuously, completing the endto-end pipeline, by adding a bonus scoring.
Since some perception skills are also necessary for thistrack, although they are not the object of evaluation, we offer
the possibility of simplifying it by using some markers on
the cloth (e.g. dying relevant parts as corners and edges).
Nevertheless, a bonus score will be added to those strategies
that do not need this help and perform the tasks using thecloth objects in its original form.
1) Task 2.1. Unfolding: This task consists on grasping
the cloth and manipulating it until it is placed flat on thetable. The cloth initial configuration can be either crumpled
or folded and the final configuration of the task is flat. Thistask can be performed in many different ways, having nolimitations on the strategies used. That is to say, it can be
performed for example with a bimanual robotic system bygrasping directly the corners, performing edge tracing to thesecond corner or by pushing and sliding the cloth on thetable until it is flat. However, the evaluation will take into
account several parameters as the total execution time or thenumber of state transitions performed since the task starts.

Steps:
1) Place the cloth in the indicated configuration, either
folded or crumpled. Follow the configuration protocol
to set them correclty.
2) Start the execution of the task and start the timer.
3) Save the result by taking a zenithal picture of the flat
cloth.
4) Repeat steps 1 to 3 five times, changing at each trial
the object and its configuration.
Rules and Evaluation: This task will be consideredsuccessful when the cloth has been placed laying flat and iscompletely inside of the table. The quality of the placement
will be evaluated by measuring the resulting area of thecloth and computing the error with respect to a predefined
template. In addition, the total execution time and the numberof state transitions [20] (i.e. changes in the grasp location,
grasp type or cloth configuration) will also be taken into
account. The task ends when the cloth is placed flat on thetable or when the maximum execution time is exceeded.
2) Task 2.2. Folding: The last task of the track consists
on folding the cloth two times once is flat on the table. In
order to correctly perform this task, the object must be folded
following the folding protocol.
This task can be a continuation of Task 2.1, being able
to execute it in sequence right after the previous one, performing an end-to-end pipeline of the whole task of folding
a cloth starting from a crumpled configuration.
Steps:
1) Place the object flat in the table and place the robotic
system in a form that the cloth is inside its reachability.
2) Start the task execution and the timer.
3) Save the result by taking a zenithal picture of the folded
cloth.
4) Perform five trials repeating steps 1 to 3, changing theobject.
Rules and evaluation: A single fold is consideredsuccessful when the opposing corners are together. The task
is considered successfully completed when the two folds are
performed according to the protocol. In similar way to Task
2.1, the quality of the result will be evaluated by comparing
the final area of the folded cloth with the expected area,
which for the case of two folds will be 1/4 of the total areaof the flat cloth. Take into account that in order to avoid overengineering of the solution, the day of the competition theorganisers may indicate to change a bit the initial positionof the flat cloth by rotating or moving it over the table.
IV. C ONCLUSIONSWe propose the first competition in manipulation and perception of highly deformable objects, such as textiles, with acompromise between tasks that can be implemented by mostof current research groups but that offer challenges to beconsidered as fully solved. We believe that the organisationof this kind of competition is a great way to improve andcompare state-of-the-art approaches and that will provide
usefull knowledge for the deformable object manipulation
community.

R EFERENCES[1] B. Calli, A. Walsman, A. Singh, S. Srinivasa, P. Abbeel, and A. M.
Dollar, “Benchmarking in manipulation research: Using the yaleCMU-berkeley object and model set,” IEEE Robot. Autom. Mag.,
vol. 22, no. 3, pp. 36–52, 2015.
[2] I. Garcia-Camacho, J. Borras, B. Calli, A. Norton, and G. Alenya,“Household cloth object set: Fostering benchmarking in deformableobject manipulation,” IEEE Robotics and Automation Letters, pp. 1–
1, 2022.
[3] “Robotic grasping and manipulation competition 2020.” https://rpal.cse.usf.edu/competitioniros2020/.
[4] A. Ramisa, G. Alenya, F. Moreno-Noguer, and C. Torras, “Using
depth and appearance features for informed robot grasping of highly
wrinkled clothes,” in IEEE Int. Conf. Robot. Autom., pp. 1703–1708,
2012.
[5] C. Bersch, B. Pitzer, and S. Kammel, “Bimanual robotic cloth manipulation for laundry folding,” in IEEE/RSJ Int. Conf. Intell. Robot.
Syst., pp. 1413–1419, 2011.
[6] J. Maitin-Shepard, M. Cusumano-Towner, J. Lei, and P. Abbeel, “Cloth
grasp point detection based on multiple-view geometric cues withapplication to robotic towel folding,” in IEEE Int. Conf. Robot. Autom.,
pp. 2308–2315, 2010.
[7] A. X. Lee, H. Lu, A. Gupta, S. Levine, and P. Abbeel, “Learning
force-based manipulation of deformable objects from multiple demonstrations,” in IEEE Int. Conf. Robot. Autom., 2015.
[8] A. Doumanoglou, J. Stria, G. Peleka, I. Mariolis, V. Petrik, A. Kargakos, L. Wagner, V. Hlavac, T.-K. Kim, and S. Malassiotis, “Folding
clothes autonomously: A complete pipeline,” IEEE Trans. Robot.,vol. 32, no. 6, pp. 1461–1478, 2016.
[9] C. Eppner, S. Höfer, R. Jonschkowski, R. Martin-Martin, A. Sieverling,
V. Wall, and O. Brock, “Lessons from the amazon picking challenge:
Four aspects of building robotic systems,” in Proceedings of the 26th
IJCAI, pp. 4831–4835, 2017.
[10] T. Wisspeintner, T. Zant, L. Iocchi, and S. Schiffer, “Robocup@home:
Scientific competition and benchmarking for domestic service robots,”
Interaction Studies, vol. 10, pp. 392–426, 12 2009.
[11] I. Garcia-Camacho, M. Lippi, M. C. Welle, H. Yin, R. Antonova,
A. Varava, J. Borras, C. Torras, A. Marino, G. Alenyà, and D. Kragic,“Benchmarking bimanual cloth manipulation,” IEEE Robot. and Autom. Let., vol. 5, no. 2, pp. 1111–1118, 2020.
[12] P. Jiménez, “Visual grasp point localization, classification and state
recognition in robotic manipulation of cloth: An overview,” Robotics
and Autonomous Systems, vol. 92, pp. 107–125, 2017.
[13] J. Sanchez, J.-A. Corrales, B.-C. Bouzgarrou, and Y. Mezouar,“Robotic manipulation and sensing of deformable objects in domestic
and industrial applications: a survey,” Int. J. Robot. Res., vol. 37, no. 7,
pp. 688–716, 2018.
[14] J. Zhu, A. Cherubini, C. Dune, D. Navarro-Alarcon, F. Alambeigi,
D. Berenson, F. Ficuciello, K. Harada, J. Kober, X. LI, J. Pan, W. Yuan,
and M. Gienger, “Challenges and outlook in robotic manipulation of
deformable objects,” IEEE Robot, Autom. Mag., pp. 2–12, 2022.
[15] S. H. Huang, J. Pan, G. Mulcaire, and P. Abbeel, “Leveraging appearance priors in non-rigid registration, with application to manipulationof deformable objects,” in IEEE/RSJ Int. Conf. Intell. Robot. Syst.,
pp. 878–885, 2015.
[16] K. Yamazaki and M. Inaba, “Clothing classification using image
features derived from clothing fabrics, wrinkles and cloth overlaps,” in2013 IEEE/RSJ Inte. Conf. on Intel. Rob. Syst., pp. 2710–2717, 2013.
[17] P. Monsó, G. Alenyà, and C. Torras, “Pomdp approach to robotized
clothes separation,” in IEEE/RSJ Int. Conf. Intell. Robot. Syst., 2012.
[18] M. J. Thuy-Hong-Loan Le, A. Landini, M. Zoppi, D. Zlatanov, and
R. Molfino, “On the development of a specialized flexible gripper for
garment handling,” Journal of Automation and Control Engineering,
vol. 1, no. 3, 2013.
[19] P. N. Koustoumpardis, K. X. Nastos, and N. A. Aspragathos, “Underactuated 3-finger robotic gripper for grasping fabrics,” in Robotics in
Alpe-Adria-Danube Region (RAAD), 2014 23rd Int. Conf. on, pp. 1–8,
2014.
[20] J. Borràs, G. Alenyà, and C. Torras, “Encoding cloth manipulations
using a graph of states and transitions,” arXiv:2009.14681, 2020.
[21] J. Borràs, G. Alenyà, and C. Torras, “A grasping-centered analysis for
cloth manipulation,” IEEE Transactions on Robotics, vol. 36, no. 3,
pp. 924–936, 2020.