Journal of Robotics, Networking and Artificial Life
Vol. 5(4); March (2019), pp. 217–222
DOI: https://doi.org/10.2991/jrnal.k.190220.001; ISSN 2405-9021 print; 2352-6386 online
https://www.atlantis-press.com/journals/jrnal

Research Article

Self-Generated Dataset for Category and Pose
Estimation of Deformable Object
Yew Cheong Hou, Khairul Salleh Mohamed Sahari*
Department of Mechanical Engineering, Universiti Tenaga Nasional, Selangor, Malaysia

ARTICLE INFO
Article History
Received 24 July 2018
Accepted 16 November 2018
Keywords
Deformable object
robotic manipulation
computer visionparticle-based model

ABSTRACTThis work considers the problem of garment handling by a general household robot that focuses on the task of classification andpose estimation of a hanging garment in unfolding procedure. Classification and pose estimation of deformable objects such as
garment are considered a challenging problem in autonomous robotic manipulation because these objects are in different sizes
and can be deformed into different poses when manipulating them. Hence, we propose a self-generated synthetic dataset forclassifying the category and estimating the pose of garment using a single manipulator. We present an approach to this problem
by first constructing a garment mesh model into a piece of garment that crudely spread-out on the flat platform using particlebased modeling and then the parameters such as landmarks and robotic grasping points can be estimated from the garment
mesh model. Later, the spread-out garment is picked up by a single robotic manipulator and the 2D garment mesh model issimulated in 3D virtual environment. A dataset of hanging garment can be generated by capturing the depth images of real
garment at the robotic platform and also the images of garment mesh model from offline simulation respectively. The synthetic
dataset collected from simulation shown the approach performed well and applicable on a different of similar garment. Thus, thecategory and pose recognition of the garment can be further developed.
© 2019 The Authors. Published by Atlantis Press SARL.
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION
Handling deformable object by robot is a challenging problem inrobotic research. To teach a general household robot to handle
these deformable objects is difficult compared with rigid object due
to the variety of sizes, textures and poses for these objects. Figure 1
shows a pipeline for the autonomous garment handling by a robot.
To manipulate it from initial state to final desired state successfully,
the category and pose of the garment should be recognized by the
robotic system. The computer vision technique integrated with therobust machine learning algorithm through garment perception
should be acquired by a robot to manipulate these objects dexterously by recognized their deformed configuration. In this research,
we are focusing the task of classify the category and recognizing
the pose of hanging garment grasped by single robotic manipulator. The main idea of this work is to generate a dataset of hanging garment that combine the real and synthetic images from thereal garment and deformable model respectively. The deformation
poses of the garment can be reduced by hanging the garment undergravity. A dataset of hanging real garment by robotic manipulator
can be obtained by capturing the depth images. For deformableobject modeling, the particle-based model is constructed to thegarment in an image and then used into the simulation to acquire
the synthetic dataset. This work is extended to include the 2D garment mesh model that able to represent the real garment that not
Corresponding author. Email: khairuls@uniten.edu.my*found in the database. The dataset will take advantages of this algorithm to generate large number of data for learning purposes without using professional graphic software. The main contribution inthese works are:•• The simple 2D garment mesh model is extracted from the real
garment that crudely spread-out on the flat platform. The multiple poses of hanging garment model under gravity are simulated,and different viewpoints of the model are captured as an image.
•• Different types and sizes of garment are simulated and the dataset is generated from the synthetic data of hanging garment meshmodel integrated with the real data.

2. RELATED WORK
Modeling and simulation are commonly used to compute the deformation of the deformable objects and becomes essential in future
autonomous robotic manipulation since the result produced fromsimulation is seeming real and applicable. In robotic manipulation,
the research on autonomous laundry service for general householdrobot has gained attention and increase significantly. The end-toend task of garment folding had been decomposed into several subtasks, the existing approaches had been proposed from different
aspects to solve the particular problem to achieve its accuracy androbustness. Generally, the procedure of autonomous garment handling can be categorized into several research areas. The majority

218

Y.C. Hou and K.S.M. Sahari / Journal of Robotics, Networking and Artificial Life 5(4) 217–222

into the real garment. Mariolis et al. proposed an approach of recognize the pose for garment using convolutional neural network.
Their model is trained by the combination of real and synthetic
dataset in 2D garment model [6]. Li et al. presented a series of laundry works in recognition and manipulation of garment. Initially, themodel-driven approach based on precomputed deformable modelof hung garments has been proposed. A set of deformable models
had been trained using the method of support vector machine torecognize the category and pose of garment [7]. They improve theirrecognition approach using KinectFusion technique to reconstructa deformable 3D model [8].

3. PARTICLE-BASED POLYGONAL MODEL
We suppose a piece of garment that is crudely spread-out and uniformly placed on a flat platform. We perform a computer vision
analysis to a single color image that captured from the top view fromthe garment. By applying the dynamic programming algorithm, thereal garment is matched by the particle-based polygonal modeland its parameters are extracted. The flowchart for the model
generation is described as shown in Figure 2.

3.1. Segmentation and Contour Finding

Figure 1 | The pipeline for garment handling in autonomous robotic
manipulation.

From the input image, the color of garment had been assumed
sufficiently different compared with background color. Hence, theobject in an image that represents garment can be acquired using
GrabCut algorithm [9]. Contour finding is used to mainly extract
the contour and centroid of the garment object in the binary image asshown in Figure 3 [10]. Additional parameters such as contour area,straight bounding rectangle and rotated rectangle with minimum

contributions in the literature of garment handling mainly focus on
the grasping skills, recognition, and motion planning strategy of
unfolding and folding.
There has been previous work on the recognition and manipulation of deformable objects. A series of works had been proposed
Kita et al. [1] in pose estimation for a hanging garment grasped
by a single end-effector using single view image and stereo image.
The hanging garment models are obtained from the physic simulation by a set of predefined grasping points to generate a deformablemodel database. The approach is observed the garment from different views and checking the consistency during pose recognition.
Maitin-Shepard et al. [2] propose an approach for folding a pile ofcrumpled towels using PR2 robot. Later, Miller et al. [3] proposeda parametrized polygonal model to describe the garment perception in the folding process. The contour of a piece of garment had
been fitted by the predefined polygonal model iteratively in order
to identify its category and parameters from the polygonal model.
Doumanoglou et al. [4] proposed a dual-arm industrial robot
to unfold different garments such as sweater and trouser. They
focused on classification and pose recognition of hanging garment
and then proceeded to unfold it into desired configuration. Striaet al. [5] proposed an approach using polygonal model incorporating relative angle and length of contour segments to fit the model

Figure 2 | Flowchart for the particle-based polygonal model generation
for synthetic dataset.

Y.C. Hou and K.S.M. Sahari / Journal of Robotics, Networking and Artificial Life 5(4) 217–222

area can be obtained. The category of garment can be estimatedbased on these related information.

3.2. Category Classification
For simplicity, the garment is supposed uniformly placed on theplatform by robot. The category of garment such as towel, shirt andtrouser can be estimated using multiple lines detection and comparison to the contour as shown in Figure 4 [10]. The length ofwidth and height at different layers can be measured, the category
can then be estimated based on the relationship between each layer.

3.3. Garment Discretization and Particle
Mesh Fitting
The garment can be discretized into separated parts such as
main part and sleeves as shown in Figure 5. From the information obtained in the category classification, the width and height219

of rectangle can be measured and the main part of the garment
can be extracted and later the other parts such as sleeved can also
be extracted using the contour finding discussed in Section 3.1.
For each type of garment, they can be separated into several parts
depending on their categories. Each separated part has parameterssuch as rectangle and centroid. Thus, rectangle can be transformed
to a mesh lattice for each contour in the garment.

3.4. Model Matching and Combination
The particles at the boundary in particle mesh lattice are matchedto the contour for each discretized part as shown in Figure 6 [11].
The particles in the internal structure of mesh lattice will realignto more organized and keep the distance between each other toprevent particles combine together. Lastly, the mesh lattice of mainpart and sleeve are combined to generate a completed particle-­
based polygonal model as shown in Figure 7.

3.5. Simulate the Hanging Garment
The particle-based modeling is applied into the dynamic programming and used to simulate the movement and deformationof the garment mesh model. The approach in the previous work
is adopted and the mass-spring model is utilized in this work [12].
The hanging process of garment mesh model generated froma real garment is simulated as shown in Figure 1, a piece of
crudely spread-out garment mesh model is placed flat on the

Figure 3 | Contour details of the garment in an image.

Figure 4 | Polygonal model for each category of garment. Measure thewidth and height at different layers by subtracting the outer points ofcontour either horizontally and vertically.

Figure 5 | Discretize the towel, shirt, trouser into main body and sleevesrespectively. Particle mesh fitting to the rectangle for each category ofgarment.

Figure 6 | The contour are matched by particle mesh lattice.

Figure 7 | Model combination between discretized parts of garment toform a completed particle-based polygonal model.

220

Y.C. Hou and K.S.M. Sahari / Journal of Robotics, Networking and Artificial Life 5(4) 217–222

simulated environment. For each step, a specified particle from thegarment mesh model will pick up by robot as a grasping point thenmoves it to the hanging position following the simulated trajectory.

3.6. Generate Synthetic Dataset
One of the hardest problems need to be solved in deep learning is
to collect the large scale data into the desired format. Collecting
the data for different categories of real garment are very time consuming particularly encompassing the different types, sizes andmaterial properties with its deformation. Hence, a training set
that gather relevant information of garment which can be used invisual recognition purposes is required. The features such as colors
and textures of the garments are less significant in this work. The
dynamic programming modeling is applied to the garment meshmodel, the deformation poses of the model hanging by robotic
manipulator is simulated. By applying different physical propertiesof the garment mesh model, the model look visually similar withdepth images can be produced from the virtual simulation. An
amount of images that simulate the deformation of real garment
can be acquired. A series of images are captured for each pose ofgarment, the model is rotated vertically with 360° and the virtual
camera captures the image of the rotated garment by increasing one
rotating degree until full rotation is achieved.

garments have been acquired by a Kinect depth sensor placedin front of the Baxter robot at a fixed height. The second Kinect
sensor is placed at the head of Baxter robot which used to capture
the images for a piece of crudely spread out garment on the flat
platform. As shown in Figure 8, the garment mesh model extracted
from the real garment, the selected particles are predefined andsimulated in virtual simulation and the deformation poses of thegarment model can be acquired.

4.2. Real and Synthetic Dataset
To collect the real garment dataset, a piece of the garment placedon the flat platform is hung by a Baxter robot and a Kinect depth
sensor located in front of the Baxter robot. The hanging garment
is then rotated 360° vertically and the color and depth images arecaptured. As shown in Figure 9, the depth images are thresholded
to obtain the garment from the background. The 200 × 340 pixels
depth images that only consists of a piece of hanging garment are

4. PRELIMINARY RESULTS
4.1. Robotic Manipulation Setup
Robotic manipulations have been conducted using a Baxter robot
with two Kinect depth sensor. The depth images of the hanging

Figure 8 | Sample of particle-based polygonal model is extracted for eachcategory of garment. Selected particles defining the poses of the hanging
garments. Orange markers define the poses on real garments from theparticle-based model.

Figure 9 | Example images of color, depth, threshold 1 (white), threshold 2 (black), synthetic and particle-based model of hanging garment grasped by
specified point.

Y.C. Hou and K.S.M. Sahari / Journal of Robotics, Networking and Artificial Life 5(4) 217–222

extracted from 640 × 480 original images. For category classification, three categories which are towels, shirts and trousers are
selected in this current work. Total nine different real garments
have been used as the samples, three real garments for each category. Each garment have been hung two or three times and about
360 depth images have been acquired for each grasping point.
For pose recognition, 21 poses for shirts, 15 pose for trousers andnine poses for towels are selected by defined by the markers. Totalaround 48,600 depth images including towels, shirts and trousers
are collected, the procedure of image acquisition is considered slow
and time consuming.
For synthetic dataset, a large dataset of synthetic depth images areconstructed using computer graphic software and dynamic programming algorithm respectively. Using Blender software, the totalnine garment models which three models of towels, three modelsof trousers and three models shirts are constructed with different
in size and shape. For simplify task in designing and model simulation, the garment models are constructed in 2D by assuming
the front and back sides of shirt and trouser are not separated
during grasped by robotic arm. In addition, the 2D garment model
also can be generated using the proposed dynamic programmingalgorithm. The particle-based polygonal model can be generated
from a crudely spread out garment placed on the flat platform. The
towel models consists of 81 particles, the trouser models consistsof 121 particles and the shirt model consists of 209 particles areconstructed. Due to the symmetry axes of the garment and reducethe dense of the particles in the model, only selected particles in thegarment model that matching with the markers in the real g­ arment
are simulated. The hanging garment models can be simulated byeach of their particles. Different views of the hanging garment
model can be acquired by rotating the model 360° in front of virtual
camera that placed at fixed height as similar as possible with the
setup of Kinect sensor for image acquisition in robotic platform.
The approximately 97,200 virtual depth images have been generated.
For synthetic dataset, the condition in the virtual environment is
set as close as possible to the depth images in real dataset. Hence,
the images acquired from the virtual simulation are comparablewith real depth images, then allow to use as training dataset in thelearning model.

5. CONCLUSION
In this work, an approach to generate the dataset of hanging
garment using both real and synthetic images is proposed. The­garment mesh model is extracted from the crudely spread-out
garment using particle-based polygonal model, then the different
poses of hanging garment can be simulated in the virtual environment. Experiment results demonstrate the proposed approach isapplicable in dataset collection that need a large volume of datato train the learning model. Using this proposed approach, different types and sizes of garment mesh model can be extracted
and simulated rather than design them in the graphics software.
For future improvement, this proposed approach will be including more complex and different category of garment such as
long-sleeved shirts and socks, and evaluate its performance oncethe datasets are extended. Thereafter, the further works such asrecognition and unfolding a piece of garment can be considered in the garment handling process. In addition, the proposed

221

approach may also benefit to the pipeline of autonomous garment
folding in robotic system which real garment can be represented
by the particle-based polygonal model and the robotic grasping
point can be defined.

ACKNOWLEDGMENT
This work is supported by the Ministry of Higher Education,
Malaysia through research grant: 2017/FRGS/V3500.

REFERENCES[1] Y. Kita, T. Ueshiba, E.S. Neo, N. Kita, Clothes state recognition
using 3D observed data, 2009 IEEE International Conference on
Robotics and Automation, IEEE, Kobe, Japan, 2009, pp. 1220–
1225.
[2] J. Maitin-Shepard, M. Cusumano-Towner, J. Lei, P. Abbeel,
Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding, 2010 IEEE
International Conference on Robotics and Automation, IEEE,
Anchorage, AK, USA, 2010, pp. 2308–2315.
[3] S. Miller, J. van Den Berg, M. Fritz, T. Darrell, K. Goldberg,
P. Abbeel, A geometric approach to robotic laundry folding, Int. J.
Robot. Res. 31 (2012), 249–267.
[4] A. Doumanoglou, A. Kargakos, T.K. Kim, S. Malassiotis,
Autonomous active recognition and unfolding of clothes using
random decision forests and probabilistic planning, 2014 IEEE
International Conference on Robotics and Automation, IEEE,
Hong Kong, China, 2014, pp. 987–993.
[5] J. Stria, D. Průša, V. Hlaváč, L. Wagner, V. Petrík, P. Krsek, et al.
Garment perception and its folding using dual-arm robot, 2014
IEEE/RSJ International Conference on Intelligent Robots andSystems, IEEE, Chicago, IL, USA, 2014, pp. 61–67.
[6] I. Mariolis, G. Peleka, A. Kargakos, S. Malassiotis, Pose andcategory recognition of highly deformable objects using deeplearning, 2015 International Conference on Advanced Robotics(ICAR), IEEE, Istanbul, Turkey, 2015, pp. 655–662.
[7] Y. Li, C.F. Chen, P.K. Allen, Recognition of deformable object
category and pose, 2014 IEEE International Conference on
Robotics and Automation (ICRA), IEEE, Hong Kong, China,
2014, pp. 5558–5564.
[8] Y. Li, Y. Wang, M. Case, S.F. Chang, P.K. Allen, Real-time pose
estimation of deformable objects using a volumetric approach,2014 IEEE/RSJ International Conference on Intelligent Robots
and Systems, IEEE, Chicago, IL, USA, 2014, pp. 1046–1052.
[9] C. Rother, V. Kolmogorov, A. Blake, “GrabCut”: interactive
foreground extraction using iterated graph cuts, ACM Trans.
Graphics 23 (2004), 309–314.
[10] C.H. Yew, K.S.M. Sahari, Real-time modeling and parameter
approximation of dexterous garment folding by robot, Artif. Life
Robot. 2018, 1–8.
[11] Y.C. Hou, K.S.M. Sahari, “Real-time modeling and parameter
approximation of dexterous garment folding by robot,” Artif. Life
Robot. 2018.
[12] Y.C. Hou, K.S.M. Sahari, Y.W. Leong, D.N.T. How, H. Seki,
Particle-based perception of garment folding for robotic manipulation purposes, Int. J. Adv. Robot. Syst. 14 (2017), 1–14.

222

Y.C. Hou and K.S.M. Sahari / Journal of Robotics, Networking and Artificial Life 5(4) 217–222

Authors Introduction
Khairul Salleh Mohamed Sahari

Yew Cheong Hou
He obtained his bachelor and master
degrees from Universiti Tenaga
Nasional in 2009 and 2012 respectively. He is currently pursuing his
PhD at Universiti Tenaga Nasional.

He is currently a Professor at the
Department of Mechanical Enginee­
ring, Universiti Tenaga Nasional.
He received his PhD from Kanazawa
University, Japan in 2006. His
research interest includes robotics,
mechatronics, Artificial Intelligenceand Machine Learning.