Visuotactile Affordances for Cloth Manipulation with Local Control
Authors: Neha Sunil
Shaoxiong Wang
Yu She
Edward Adelson
Alberto Rodriguez Garcia
Venue: CoRL 2022 Poster (2022)
Keywords: Multi-modal learning
Cloth manipulation
Tactile control

Abstract: Cloth in the real world is often crumpled, self-occluded, or folded in on itself such that key regions, such as corners, are not directly graspable, making manipulation difficult. We propose a system that leverages visual and tactile perception to unfold the cloth via grasping and sliding on edges. Doing so, the robot is able to grasp two adjacent corners, enabling subsequent manipulation tasks like folding or hanging. We develop tactile perception networks that classify whether an edge is grasped and estimate the pose of the edge. We use the edge classification network to supervise a visuotactile edge grasp affordance network that can grasp edges with a 90% success rate. Once an edge is grasped, we demonstrate that the robot can slide along the cloth to the adjacent corner using tactile pose estimation/control in real time.

Visuotactile Affordances for Cloth Manipulation with Local Control
Neha Sunil" Shaoxiong Wang"), Yu She?, Edward Adelson', Alberto Rodriguez' Massachusetts Institute of Technology PPurdue University <nsunil, wang sx, albertor>emit. edu shey@purdue edu adelson@csail. mit. edu
Abstract: Cloth in the real world often crumpled, self-occluded, folded itself such that key regions, such corners, are not directly graspable, mak- ing manipulation difficult. We propose system that leverages visual and tactile perception to unfold the cloth via grasping and sliding on edges. By doing so, the robot is able to grasp two adjacent corners, enabling subsequent manipulation tasks like folding or hanging. As components of this system, we develop tac- tile perception networks that classify whether an edge is grasped and estimate the pose of the edge. We use the edge classification network supervise visuotac- tile edge grasp affordance network that can grasp edges with 90% success rate. Once edge is grasped, demonstrate that the robot can slide along the cloth to the adjacent corner using tactile pose estimation/control in real time.
Keywords: Multi-modal learning, Cloth manipulation, Tactile control
(a)
No Fabric
Edge
All Fabric
(b)
Corner
Fold
Vision
Affordance
Tactile
Supervision
Figure 1: Visuotactile affordance for sliding. Using the visual depth image input, affordance network scores each pixel in terms potential edge grasp success. The affordance image shown is trained first in simulation and then fine-tuned with tactile self-supervision from real grasps. In this particular cloth configuration, the adjacent corner is not directly visible or graspable, motivating first grasping an edge and then sliding. We demonstrate tactile sliding in configurations: (a) sliding down with the cloth in a stable configuration and (b) horizontal sliding suitable for longer towels.
1 Introduction
The robotic manipulation of highly deformable objects a growing research area due to its applica- bility in various fields such as assembling cable harnesses in factories, industrial garment manufac- turing, and assisted dressing and laundry folding in the home and healthcare industry. Our goal is
Authors with equal contribution.
This work was done at MIT.
6th Conference Robot Learning (CoRL. 2022), Auckland, New Zealand.

---

framework that combines visual and tactile sensing of local features (specifically cloth corners and edges in our system) for perception and control of deformable objects. Within this paper, we focus on the manipulation of cloth.
Manipulating deformable objects is difficult due to their continuous and highly-underactuateed ture. Applicable high-dimensional models are challenging to identify or simulate. Perception is another key challenge, given that the object is easily occluded by the robot's grippers and even self- occluded by the object itself. These issues pose an inherent difficulty in object representation since full state estimation and modeling is very challenging for real-time manipulation.
Most cloth manipulation studies address these challenges with visual perception [1]. A common first step is unfolding the cloth to make specific features visible by holding it up flattening it on table. Then, the robot perceive and directly grasp features (like corners collars) for subsequent manipulation tasks. However, these features may still not be directly graspable, as in the corner example in Fig. 1.
In contrast, we want to approach real-time, closed-loop manipulation that is closer to how humans manipulate deformable objects. Instead of directly looking for adjacent corners on crumpled cloth to pick it up along an edge, unfold the cloth while sliding until we feel corner. As shown in Fig. 1, once an edge is successfully grasped, we can slide to the adjacent corner, which was not otherwise graspable. This bottom-up approach of using local features (specifically corners and edges) for perception and control, enables novel manipulation skills.
We leverage tactile perception, along with vision, to localize and manipulate local features. Vision provides global guidance help with state identification and grasp point localization (in the style of visual affordance-baset grasping [2, 3]). We introduce visuotactile affordances where we learn grasp affordances self-supervised with tactile feedback. Tactile feedback from the gripper aids in the classification and estimation of the state of local features. This is especially useful when vision occluded by the gripper. Finally, we use tactile perception for grasp adjustments, local control, and grasp success confirmation. In this paper, the key technical contributions are:
Tactile Perception of Local Features. We demonstrate a strategy for local feature pose estimation and classification where we show that it is possible to distinguish between edge, corner, fold, all fabric, and no fabric grasps with 92% accuracy (Sec. 4.1).
Visuotactile Grasp Affordances. A grasp localization affordance network pre- pre-traine in simulation and fine-tuned with tactile supervision with 90% grasp success rate (Sec. 4.2).
Tactile Sliding. Tactile based controllers to slide gripper along the edge of cloth in two different configurations (Sec. 4.3).
Ultimately, our key experimental contribution is a demonstration of our framework picking a crum- pled towel from the ground, grasping a corner using vision, grasping an edge using our visuotactile affordance network, and sliding to an adjacent corner, reaching configuration which can be used fold or unfold the towel.
2 Related Work
Perceiving the full state of cloth [4, 5, 6] modeling its dynamics [7, 9] challenging and time-consuming approaches. In this section, we instead highlight works that grasp and manipulate local features for cloth folding and unfolding.
Manipulation of Local Features of Deformable Objects. Much of the work on cloth-centric perception for manipulation focuses on corner detection and ridge detection for determining grasp points for future manipulation tasks [10]. Both can be done using computer vision techniques, often using filters (e.g. Harris corner detection [11] and Gabor filters for wrinkles [12]). However, finding
other more specific local features often requires first flattening the cloth [13, 14, 15, 16] or hanging it from specific grasp points [17, 18, 19] to prevent self-occlusion of the cloth.

---

Learning-baat -baiid perception approaches enable more flexibility with different local features and start- ing configurations. However, determining ground truth labels is challenging for cloth. Qian et al. (2020) trains a network to segment edges and corners cloth from a depth image by using painted cloth as a ground truth label [20]. Simulators can also provide these labels [21, 22], but Sim2Real transfer is an issue, given the uncertain dynamics of cloth. In this work, we first train our edge grasp affordance network in simulation and fine-tune the robot using tactile perception for supervision.
Most work in fabric manipulation either uses incremental pick-and -a---addllccee movements against table [14, 23, 21] or holds up the cloth and regrasps while hanging (17, 18, 19]. However, the sliding skill simplifies the task of finding two adjacent corners order to fold a piece fabric, especially when the second corner is not immediately visible or graspable. Sahari al. (2010) holds up a corner of the fabric and uses gravity trace downward to find the second corner with low-resolution sensory feedback [24]. Similarly, Yuba al. (2017) executes open-loop "pinch and slide" motion along the top edge of piece of fabric [25]. Zhou et al. (2020) use low resolution tactile sensing to learn sliding policy in latent space [26]. Our high-resolution tactile sliding allows us to consistently follow longer towels while being robust to disturbances in both stable and horizontal configurations.
In previous work, Yu et (2019) slides along cables using GelSight [27] for tactile perception to estimate the pose of the cable in grasp while sliding [28]. Gelsight, and other vision-based tactile sensors, convert touch vision by using camera visualize the deformation of the contact surface. We also approximate shear force by tracking the black markers on the sensor surface as proxy for friction force while sliding.
Perception Local Features. Visual grasp affordances are proposed for learning grasping con- figurations with data-driven methods [3]. The affordance is a metric to predict the grasp success rate under different configurations. Each configuration is represented an individual pixel and discrete rotation angle. Affordances
be labeled manually [3], but alternatively self-supervised manner [29, 2, 30, 16]. In these cases, the supervision for affordances comes from either simulation the visual outcome move out of the camera frame for visual evaluation. In contrast, our visuotactile approach fine-tunes the affordance trained in simulation under visual occlusion. The learned visuotactile affordance allows for a smooth transition between grasping and sliding.
be learned in
real system, which usually requires the end-effector to
3 Cloth Manipulation
Vision Tactile Open-loop
Rotate
fabric
tansion
below Grasp Affordence
not adgo
fabric (R]
Grasp Edge
Slide Corner
Pick
Corner
Fold/ Unfold
(R)
edee (R)
tension
Figure 2: State machine for cloth manipulation. Colors in the state machine reflect sensing modal- ities required for each action and transition conditions between actions. The two key components, visuotactile grasp affordances (highlighted blue) and tactile sliding to corner (highlighted red), are discussed in Sec. 4.2 and Sec. 4.3 respectively. Below, we illustrate the cloth state with respect to each gripper.
In order fold or unfold the cloth, first pick the towel up from a surface, grasp corner, grasp an edge adjacent to that corner, and slide to the adjacent corner (See Fig. 2). Once the robot is grasping two corners, it can complete the folding unfolding task open-loop.

---

Tactile Sliding, In the cloth sliding task, two grippers start near each other, grasping the cloth edge. The goal is to traverse as much of the fabric edge as possible or to specified goal condition (a corner, this work) without losing the grasp, maintaining an estimate of the edge pose within the sliding gripper.
Sliding along the edge of short towels can be done with regrasps if the cloth is kept in a stable configuration under gravity. We use simple control strategy for sliding along a cloth edge while hanging. However, a longer towel is more likely to be entangled, susceptible to disturbances, and partially resting on the table. In addition, the adjacent corner is likely self-occluded, making the sliding skill more useful. Thus, we extend the work Yu et al. (2019) for sliding along cables to longer towels [28]. Here, a moving gripper with fixed grip on the cloth pulls the cloth through stationary gripper while attempting to keep the edge visible in the tactile image of the stationary gripper, shown in Fig. 1. Both sliding strategies discussed in Sec. 4.3. Perception of the cloth edge is more difficult than for cables. Therefore, we a supervised data-driven method to estimate the pose of the fabric edge, explained in Sec. 4.1.
Grasp Affordance for Sliding. In order to grasp an edge way that allows for sliding, a single layer fabric should be aligned within the gripper to avoid collisions. We train visuotactile grasp affordance network that takes the depth image as input and outputs an affordance heatmap, indicating the grasp viability of each point the image. This network is originally trained in simulation. Then, we fine-tune this network robot using the tactile information from the grasp attempt supervision (Sec. 4.2).
4 Method
Pose Estimation
Tactile Sliding
Grasp Supervision
Single-frame Input
Classification
Fabrk
Edge
Foblicc
Corner
Multi-frame Input
Figure 3: Tactile perception. The perception network used for sliding along the fabric edge takes a single-frame frame input and outputs the pose of the edge, including states where no edge is visible (the no fabric and all fabric cases). For our grasp supervision network, we sample multiple frames from the grasp attempt and classify between additional categories.
4.1 Tactile Perception of Local Cloth Features
Tactile perception is used for pose estimation and classification of local features within the grip. All of the following tactile perception networks share a similar architecture and strategies for data augmentation.
Pose Estimation. Applying the same cable pose estimation method [28] to slide along the edge of soft cloths would be challenging because the fabric is relatively thin and covers a large area (see Fig. 3), unlike the cable which creates a distinct imprint. We found that traditional computer vision edge detection techniques are unsuccessful at consistently isolating the fabric edge due to the fabric texture, wrinkles in the sensor's gel surface, and noise in the tactile image.
Our pose network takes down-sample depth image input and outputs the and coordinates of the center of the edge and the orientation of the edge, which can be directly used for tactile sliding.

---

The network has convolutional layers followed by max pooling, and 3 fully connected layers. We choose a simple network architecture to optimize for speed to use in our sliding controller.
The dataset is generated from a small number of hand-labeled tactile images (frames from a video), and then augmenting the dataset by randomly varying the maximum depth threshold of each tactile image, and further augmenting the images by introducing random translation and rotation (see Appendix for details). The human labeling process involves clicking two points to define the cloth edge if visible, or otherwise classifying the image as all fabric or no fabric. This labeling can then be transformed to all the synthetically augmented images.
Classification. The tactile sliding perception network that outputs pose also classifies whether (1) there is no fabric on the sensor, (2) the sensor is covered fabric, (3) the sensor sees an edge (Fig. 3). We collect data from videos of sliding with different gripper widths. The final network train has 150 raw training images which augment 30,000 depth images.
The tactile sliding network can classify between no fabric, edges, and all fabric given a single frame when the gripper is pressing against the towel. To supervise our edge grasp affordance network, we also need to distinguish between edges and folds. Fold classification especially challenging because the final tactile image can look identical other categories. Therefore, use a sequence of images from the grasp attempt input which introduces temporal information, instead of just the final grasp. The tactile image starts showing imprint earlier with multiple layers of fabric.
We sample every tactile depth images of the grasp attempt and choose the last 5 as input to the network. During the augmentation process, we shift the initial frame for sampling. We skip depth augmentation because timing crucial this network, but continue translate and rotate the images. We record 330 grasp attempts per category and augment this over 6000 per category.
4.2 Visuotactile Grasp Affordances
In this work, there are three kinds of grasp points we are interested in: generic (for picking), corner, and edge. All three use vision to choose grasp points and can use either visual or tactile perception to verify grasp success (Fig. 2). We discuss the generic and corner grasps in Sec. 4.4.
Edge grasp point localization is more challenging than corners because direction of the edge. Furthermore, visual edge detection be difficult since edges look similar to folds along the contour of the cloth. The cloth edge can also easily twist in on itself, but successful grasp for sliding requires grasping single layer. Moreover, aligning the edge within the gripper to avoid collision is crucial for grasp success. the cloth collides with the gripper, it easily deforms, preventing a successful grasp. Given these additional considerations, we train a network that learns visuotactile edge grasp affordances.
also consider the
We first train this affordance network in simulation. Then, we refine our fine-tuning technique while transferring between slightly different simulation environments. Finally, we fine-tune this network on a robot using the tactile information from the grasp attempt's success/failure as supervision.
Training in Simulation. We use the same Blender simulation setup [21] for training the affordance net- work using U-Net [31], as shown in Fig. 4. The in- put a depth image of the hanging cloth, and the output is the affordance heatmap. The ground-truth labels of the affordances in simulation determined for each pixel by geometric computing, specif- ically (1) the percentage of neighboring cloth nodes that are classified as an edge, (2) whether the
Depth
Affordance
max threshold - grasp
max < threshold rotate
U-Net
Figure 4: Affordance architecture and rotation policy.
5

---

gripper will collide with cloth, (3) whether the gripper would grasp a single or multiple layers of fabric, and (4) whether the point is reachable. All of these criteria can be determined in simulation because we have access to the full state of the cloth. The tactile sensor implicitly checks all of these criteria on the real system when classifies grasp as an edge or non-edge grasp. The simulated dataset contains 200 cloth configurations rotated in increments of 15", yielding 4800 depth images with corresponding affordance images.
Fine-tuning in Simulation. As precursor to sim-to-real transfer, we first try sim-to-sim transfer to validate our network architecture and tune parameters. While the network in simulation has access to an entire labeled image during training, during fine-tuning, the robot can only test a single pixel at a time. We create a second simulated dataset with slightly different camera location, cloth size, and cloth stiffness. We change several network parameters (see Appendix for details) and ultimately find that fine-tuning with single pixel updates is feasible with an experience replay buffer.
Fine-tuning on Robot with Tactile Supervision. The real depth images the robot have regions of zero-depth due to the offset in location between the projector and sensor of the depth camera. Also, items in the background affect the affordance. Therefore, we mask out all non-cloth regions in both simulation and deployment and add random black rectangles to the simulated depth images during training for robustness against regions with zero-depth (see Appendix for details).
During data collection, we grasp corner, mask the reachable regions, and attempt to grasp the cloth at different points with fixed orientation before resetting and grabbing reachability mask avoids collisions between both grippers, chooses the side of the cloth closest to the grasping gripper, and filters out the bottom of the cloth to only grasp adjacent edges. The cloth is rotated increments of 15 for a whole rotation with grasp attempts per rotation. We implement a minimum affordance threshold to increase the number of positive examples (Fig. 4). Each grasp attempt is labeled with the tactile multi-frane classification network's 's confidence that the grasp is an edge. We collected around 3,000 grasps, which takes about 15 hours. We validate the network with 110 hand-labeled data collected offline.
corner. The
4.3 Tactile Sliding
Sliding in Stable Configuration. Once one gripper holds corner and the other holds an adjacent edge, we use the tactile signal to slide downward. The sliding gripper moves forward and back to keep the edge close to the target position within the grip until the robot reaches a corner. We use proportional controller with respect to the fabric edge's 's offset from the target position to slide along the edge. We consider an increase in shear (as determined by GelSight marker displacement) above a certain threshold from the fixed gripper to indicate that the sliding gripper has reached the thicker corner.
Sliding along Long Towels. We follow the methods of [28] slide along the edge of longer towels in horizontal configuration. We learn a linear dynamics model that finds the relationship between state (defined by cloth edge position and orientation and robot position) and pulling angle. We use this model in an LQR controller that keeps the edge close to the target state as the cloth is pulled through (see Appendix for details).
4.4 Cloth Manipulation System
In order to complete the system, we also need to pick the towel up from the ground and then grasp a corner. Although the specific grasp location with respect to the rest of the cloth is unknown, this initial grasp is helpful for partially unfolding the towel using gravity. This allows for more of the towel be visible for downstream tasks. For our specific grippers, the towel grasp point cannot be completely flat the surface (wrinkles are much easier to grasp). Therefore, sample from the highest points choose a random grasp point.

---

Comer grasp points are common for flattening and folding tasks. However, corners are not always visible when the cloth on a surface due to self-occlusions. Furthermore, visible corners on a table often flat on the surface, which would require modifications to our gripper design to grasp them. So, instead, we look for corners on the hanging cloth by choosing the lowest point.
For the hardware setup, use dual UR5 robot arms, which are equipped with parallel-jaw grippers servoed with dynamixel motors (Dynamixel XM430- RM430-W210-1 Robotis). We mount two GelSight sensors [32], one on each gripper. We use Kinect Azure camera to capture RGB-D images.
5 Results
Tactile Perception. Our pose estimation network for tactile sliding has mean average of 3.2 mm for the center position of the edge and 4.1" for the orientation of the edge. The overall sensing area is 30 mm wide. These error ranges are within the expected human labeling enror and this performance is experimentally sufficient for tactile sliding. We compare this network to one trained on a dataset without augmentation, which has a similar position error, but significantly larger orientation error of 10.2". This network runs at 33 Hz within the sliding controller.
The multi-frame classification network used for grasp affordance supervision has an overall classi- fication accuracy of 92%. Since affordance network only needs distinguish edges from other grasps, we also report a non-edge classification accuracy of 98%.
Visuotactile Grasp Affordance. Our network trained in simulation successfully grasps edges in 21.3% of the given configurations, out of which only 39.7% have valid, accessible edge grasps. By introducing our threshold rotation policy (Fig. 4), our network successfully grasps an edge for 56.6% of initial configurations in which 95.0% of configurations had valid grasps. Rotations crucial to finding and accessing valid edge grasps.
We compare our fine-tuned affordance network to three other baselines: (1) Segmentation-based
(a)
(c)
(d)
visual edge seg- mentation network [20] combined with surface normal and right side-reachability cost metrics (see Appendix for details), (2) Sim2Real: the network trained in simula- tion that is directly transferred to the robot,
Vision
and (3) Real2Real: a network trained di- rectly on the collected robot data. We
evaluate the networks offline using pre-
Affordance
cision@k [33], using this metric due
the unbalanced nature of the evaluation set and because it does not need set thresh- old. We found that Sim2Real performs the worst because of the differences be- tween the simulated and real robot envi- ronments. However, the fine-tuned net- work outperforms the Real2Real network, showing that the network from simulation provides overall structure.
Figure 5: Fine-tuned edge grasp affordance results. The learned affordances effectively reflect the edge graspability in different configurations, where is graspable for (a) the entire right edge and (b) no edge. The last towels have edges that twist such that they are only accessible from (c) the top portion and (d) the bottom portion. The proximity to the rest of the fabric (c) or the misalignment of the surface normal (d) would cause collisions during the grasp attempt.
We further evaluate the networks in de- ployment by reporting the average num- ber of grasp attempts before identifying an edge (as confirmed by tactile perception) and the overall success rate of true edge grasping. The threshold was chosen independently for each network and tuned to maximize performance the offline dataset. The cloth is grasped by the corners and evaluated with 10 different configurations.
7

---

Offline Evaluation
Real Robot Deployment
Precision @k
Average attempts
Success Rate
Segmentation-baan [20]
65.0%
4.2
65.0%
Sim2Real
45.0%
1.3
40.0%
Real2Real
62.5%
3.3
50.0%
Sim2Real + Finetune (Ours)
80.0%
1.6
90.0% %
Table 1: Quantative results in real world for visual affordance learning with tactile supervision. We
use = 40 for precision @k. since there are around 40 positive samples out of 110 in the real-world data for offline evaluation. For evaluation during deployment, compare methods with 10 different trials. Each model has similar initial cloth configurations for each trial.
Table shows the experiment results. The failure modes include false- aasseppositive edge tactile classifi- cations and rotating 360 In general, more grasp attempts increase the likelihood of a false positive edge classification. Our fine-tuned network outperforms both baselines with an overall success rate of 90%. The Sim2Real network often misses grasp opportunities and completes full rotation with- out successful attempts. The Real2Real network is more likely to grab inside the cloth without the underlying structure from simulation. The robot often grabs an edge along with fold, resulting in false positives from tactile classification. The segmentation-bant that would fail due to collisions with other regions of fabric. The fine-tuned network effectively identifies graspable edges, considering both global and local geometry and semantics for calculating affordance (Fig. 5).
network fails to identify grasps
When sliding while the towel is in a stable configuration, we find that sliding controller performs differently with the thin and thicker edges of the towel. The gripper traverses 100% of the thin edge (sliding all the way to the corner) no matter the initial edge position and 100% of both the thin and thicker edges if the initial edge pose is near the inner edge of the gripper (n = 5 trials). The sliding distance decreases (cloth slips from grip before reaching the corner) as the initial edge pose starts closer to the fingertip. At centered initial pose, the sliding gripper travels an average of 95% of the towel length (14.0 em for the thin edge and 12.7 em for the thick edge).
Tactile Sliding.
For sliding in the horizontal configuration, we slide until the end of the robot workspace (smaller than the towel's total length). Similar to sliding in a stable configuration, we find that if the cloth edge starts near the gripper's 's inner edge, the robot can consistently slide for the entire length of the workspace in all trials. If the initial edge pose is in the center of the gripper, the robot travels an average of 76% of the workspace (43 em out of 56 em).
Discussion
We develop robot system for cloth manipulation that uses both visual and tactile perception of local features for local control. Our tactile perception systems are used for both pose estimation for controlled tactile sliding and supervision for learning visuotactile grasp affordances. Both networks are trained with small amounts of real data by augmenting the tactile depth images. Our grasp affordance network learns global structure from simulation and is fine-tuned using tactile supervision during real robot grasp attempts. This network outperforms all of our baselines.
Limitations and Future Works. We use multiple neural networks for our local approach to percep- tion and control, which lends itself generalization because we do not rely on the global structure of the deformable object. However, while pose network works most towels, but the tactile classification network is specific to the training towel's thickness and texture and the fine-tuned af- fordance network is specific to the towel's dynamics and shape. These networks could be more generalizable trained with a dataset with more variety. Our fine-tuning framework allows for in- troducing new cloth types without needing to train from scratch (given an applicable classification network). Our tactile approach allows certain modules be independent of specific cloth parame- ters, for example, towel length texture while sliding. As another limitation, tactile sliding success is dependent on the initial cloth edge position. Adding a tactile regrasp step to adjust the cloth edge position with tactile feedback before sliding could help address this issue.
8

---

Acknowledgments
We thank the anonymous reviewers for their helpful comments in revising the paper. Toyota Re- search Institute (TRI), Amazon Research Awards (ARA), the Google Faculty Research Awards, and GentleMAN and BiFrost projects of SINTEF provided funds to support this work. Neha Sunil is supported by the National Science Foundation Graduate Research Fellowship [NSF-1122374).
the Office of Naval Research (ONR) (N00014-18-1-255
References
[1] Jiménez and C. Torras. Perception of cloth in assistive robotic manipulation tasks. Natural Computing, 19(2):409-433 4311,-43111000 2020.
[2] J. Mahler, J. Liang. S. Niyaz, M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg, Dex- net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metrics. arXiv preprint arXiv: Ttt.::/70309122. 2017.
[3] A. Zeng, Song, K.-T. Yu, E. Donlon, F. R. Hogan, M. Bauza, D. Ma, O. Taylor, M. Liu, E. Romo, al. Robotic pick-and-place of novel objects clutter with oulti-affordance grasp- ing and cross-domaie image matching. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3750-3757. IEEE, 2018.
[4] C. Chi and D. Berenson. Occlusion-robust deformable object tracking without physics simula- tion. Intelligent Robots and Systems (IROS), 2019 IEEE International Conference on. IEEE, 2019.
[5] Y. Li, Y. Wang, M. Case, S.-F. Chang, and Allen. Real-time pose estimation of de- formable objects using volumetric approach. In 2014 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 1046-1052. IEEE, 2014.
[6] C. Chi and S. Song. Garmentnets: Category-level pose estimation for garments via canon- ical space shape completion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3333, 2021.
[7] W. Yan, A. Vangipuram, P. Abbeel, and L. Pinto. Learning predictive representations for deformable objects using contrastive estimation. arXiv preprint arXiv:2003.05336 2020.
[8] R. Hoque, D. Seita, A. Balakrishna, A. Ganapathi, A. K. Tanwani, N. Jamali, K. Yamane, Iba, and K. Goldberg. Visuospatial foresight for multi-step, multi-task fabric manipulation. arXiv preprint arXiv:2003.09044 2020.
[9] J. Wang, H. Gang, S. Ancha, Y.-T. Chen, and D. Held. Semi-supervised 3d object detection via temporal graph neural networks. In 2021 International Conference on 3D Vision (3DV), pages 413-422, IEEF, 2021.
[10] B. Willimon, S. Birchfield, and I. Walker. Model for unfolding laundry using interactive per- ception. In IEEE International Conference on Intelligent Robots and Systems, 2011. ISBN 9781612844541.
0t::0.1109/RROS.20011
[11] C. Harris and M. Stephens. A Combined Corner and Edge Detector. 2013. doi: 10.5244/c...23.
[12] K.
Yamazaki, K. Nagahama, and M. Inaba. Daily clothes observation from visible surfaces based on wrinkle and cloth-overlap detection. In MVA, pages 275-278, 2011.
[13] Sundaresan, J. Grannen, Thananjeyan, A. Balakrishna, M. Laskey, Stone, J. Gon- zalez, and K. Goldberg. Learning rope manipulation policies using dense object descriptors trained on synthetic depth data. In 2020 IEEE International Conference Robotics and Au- tomation (ICRA), pages 9411-9418, IEEE, 2020.

---

[14] Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel. Learning to manipulate deformable objects without demonstrations, 2019.
[15] R. Hoque, K. Shivakumar, S. Aeron, G. Deza, A. Ganapathi, A. Wong, J. Lee, A. Zeng, V. Van- houcke, and K. Goldberg. Learning to fold real garments with one arm: A case study in cloud-based robotics research. arXiv preprint arXiv:2204.10297 2022.
[16] H. Ha and S. Song. Flingbot:
The unreasonable effectiveness of dynamic manipulation for cloth unfolding. In Conference on Robot Learning, pages 24-33, PMLR, 2022.
[17] A. Doumanoglou, A. Kargakos, T.-K. Kim, and S. Malassiotis. Autonomous active recognition and unfolding of clothes using random decision forests and probabilistic planning. In 2014 IEEE international conference on robotics and automation (ICRA), pages 987-993, IEEE, 2014.
[18) M. Cusumano-Towwen A. Singh, S. Miller, J. F. O'Brien, and P. Abbeel. Bringing clothing into desired configurations with limited perception. In 20/1 IEEE international conference on robotics and automation, pages 3893-3900. IEEE, 2011.
[19] J. Maitin-Shepard,, M. Cusumano-Towwn J. Lei, and P. Abbeel. Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding. In 2010 IEEE International Conference on Robotics and Automation, pages 2308-2315. IEFE, 2010.
[20] J. Qian, T. Weng, L. Zhang, B. Okorn, and D. Held. Cloth region segmentation for robust grasp selection. In IEEE International Conference on Intelligent Robots and Systems, 2020. ISBN 9781 777222162222 doi: 10/1
121.
[21] A. Ganapathi, P. Sundaresan, B. Thananjeyan, A. Balakrishna. D. Seita, J. Grannen, M. Hwang, R. Hoque, J. E. Gonzalez, N. Jamali, K. Yamane, S. Iba, and K. Goldberg. Learning to smooth and fold real fabric using dense object descriptors trained on synthetic color images. arXiv, 2020.
[22] E. Corona, G. Alenya, A. Gabas, and C. Torras. Active garment recognition and target grasping point detection using deep learning. Pattern Recognition, 74:629-641, 2018.
[23] R. Hoque, D. Seita, A. Balakrishna, A. Ganapathi, A. K. Tanwani, N. Jamali, K. Yamane,
S. Iba, and K. Goldberg. VisuoSpatial foresight for multi-step, multi-task fabric manipulation, 2020.
[24] K. S. M. Sahari, H. Scki, Y. Kamiya, and M. Hikizu. Edge tracing manipulation of clothes based on different gripper types. Journal of Computer Science, 2010. ISSN 15493636. doi: 10.3844/jcssp.2000.77..
[25] H. Yuba, S. Arnold, and K. Yamazaki. Unfolding of rectangular cloth from unarranged start- ing shapes by certainty. Advanced Robotics, 2017. ISSN 15685535. doi: 10. 1000/0009 (00000069166.201111
robot with mechanism for managing recognition eгror and un-
[26] W. Zhou, S. Bajracharya, and D. Held. Plas: Latent action space for offline reinforcement learning. arXiv preprint arXiv:2011.07211 2020.
[27] W. Yuan, S. Dong, and F. Adelson. Gelsight: High-resolution robot tactile sensors for estimat- ing geometry and force. Sensors, 17(12)22762, 2017.
[28] Y. She. Wang, S. Dong, N. Sunil, Rodriguez, and Adelson. Cable manipulation with tactile-reactive gripper. arXiv preprint arXiv:1910.02866 2019.
[29] A. Zeng, Song, Welker, J. Lee, Rodriguez, and T. Funkhouser. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. In 2018 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 4238-
4245. IEEE, 2018
10

---

[30] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. IEEE Transactions on Robotics, 36(4):1307-1319 2020.
[31] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical im- age segmentation. In International Conference on Medical image computing and computer- assisted intervention, pages 234-241. Springer, 2015.
[32] S. Wang, Y. She, B. Romero, and E. Adelson. Gelsight wedge: Measuring high-resolution
3d contact geometry with compact robot finger. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages
IEEE, 2021.
[33] M. Sanderson. Test collection based evaluation of information retrieval systems. Now Pub- lishers Inc, 2010.
[34] R. Hogan, M. Bauza, O. Canal, E. Donlon, and A. Rodriguez. Tactile regrasp: Grasp adjustments simulated tactile transformations. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2963-2970. IEEE, 2018
[35] Y. Hou, Z. Jia, A. M. Johnson, and M. T. Mason. Robust planar dynamic pivoting by regulating inertial and grip forces. In Algorithmic Foundations of Robotics XII, pages 464-479. Springer, 2020.
[36] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint a41i:: 2014.
[37] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, Antonoglou, D. Wierstra, and M. Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602. 2013.


Reviews:
[{"review_id": "ox4ZBaLGf_", "reviewer": "Reviewer_MA9Y", "summary": "This paper develops a manipulation framework for picking up and unfurling cloth in tabletop scenarios (e.g. cloth randomly draped on table). The system works by (1) picking up cloth from the table; (2) re-positioning the cloth so it can be grasped at a (detected) corner, and (3) unfurling the cloth by “following” an edge from the detected corner. The authors found that tactile data (from GelSight) was crucial both as an **input** *and* as a **supervision source** for training the necessary perception models (e.g. corner/edge detectors). Data for training these models was collected both in simulation and in the real world. The authors validate these design choices with an ablation study, and demonstrate qualitative behavior in their video.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n* This paper offers a compelling examples for how to use tactile information to solve real world robotics problems. It is particularly interesting that the touch data is used to improve the vision model through supervision, as well as being used as an input.\n* The authors clearly did some great task specific engineering and came up with clever strategies for unfolding.\n* The video does a good job explaining the method and contextualizing results.\n\n**Weaknesses**\n* The experiments only discuss success rates in terms of affordance learning, but *not* in terms of task success. This makes it hard to contextualize their improvement over baselines for the final task. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper showcases an impressive framework for solving a challenging real world task. I particularly enjoyed reading how the authors exploited unique sensing capabilities (e.g. tactile data), and what was required to successfully transfer their perception stack from sim to real. While I would like more context on task success (see Issues section), I don’t believe this issue is major enough to warrant rejection.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "The authors should address the weakness pointed out above, by giving a clearer idea of how their method improves end-task (i.e. cloth unfolding) success rate in realistic scenarios. \n", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "__tzNlNHSzv", "reviewer": "Reviewer_6ikq", "summary": "This paper presents a system that uses vision and tactile perception to unfold clothes by grasping and sliding. The system uses two UR arms, depth camera and vision-based tactile sensors attached to the robots' grippers. Perception is handled with neural networks that are responsible for the classification of grasping edges, supervision of visuo-tactile affordances detection, and pose-estimation/control for sliding. The system is deployed on real robots effectively.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents advances on a relevant and challenging topic in robot manipulation. The problem of cloth manipulation involves several challenges, that are addressed in a systematic and clear way in the proposed work.\nThe paper is well structured, it is clear and easy to read. Challenges and contributions are presented in a clear way, and the literature review seems comprehensive to the best of my knowledge. Figures, diagrams and videos are good and helpful to get a grasp of the contributions.\nThe experiments include an informative set of comparisons, and the relative discussion includes useful insights and analysis.\nThe limitations reported point out relevant shortcoming of the proposed method.\n\nWhile the paper explains the experimental setup and method thoroughly, implementation details are missing: for example details related to the networks trained and used in the experiments, how many parameters were used, how long the training time is, etc. Some information on how the robot is controlled, and how the state machine (fig.2) is deployed would be very useful as well to provide a complete picture of the experimental setup.\nThe analysis would benefit from a discussion on what the failure modes are and why they happen. For example, what are the limitations on the perception side? do you have a quantitative measure of the uncertainty of the tactile classifier? how does that affect the final performance? on the visual perception, what would make your setup more robust? \nThe sliding motion seems really hard to obtain, and the results presented are remarkable. Do you have quantitative measures of the robustness or reliability of this controlled movement? How many repetitions are consistently executed on the same piece of cloth? What part of your setup/architecture would further improve the overall performance? \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presents a relevant piece of work, that tackles a challenging and important problem in robot manipulation. The execution of the experiments and the research advances are interesting and, although not general, very promising.\nWhile some implementation details are missing, the work is well executed and the experimental results are remarkable.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "Missing or limited details: please add implementation details for reproducibility, as well as quantitative measures indicating uncertainty/effectiveness in your experiments, for example prediction/classification uncertainty, robustness to noise/disturbances, reliability/repeatability of the perception/control loop."}}, {"review_id": "_7XDD4-POM", "reviewer": "Reviewer_a2rY", "summary": "The paper addresses the problem of cloth grasping by presenting a system that leverages both visual and tactile perception. The proposed system uses tactile perception (edge estimation network, tactile sliding network) to enble closed-loop local control under the occlusion induced by gripper. Another vision-based grasp affordance netowrk is pretrained in simulation and fine-tuned with tactile supervision, and provides global guidance to state identification and grasp point localization. The proposed system is evaluated in real-world and demonstrates impressive performance in real-world cloth grasping.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n*  The paper tackles a very important problem in cloth manipulation: cloth grasping. When designing the system, the authors take the common failure mode of cloth grasping into account,  such as gripper collission and grasping wrong number of layers, missed grasp.\n\n* Compared to previous cloth grasping approach, it leverages tactile perception, and in a really clever way: 1. it enables closed-loop manipulation with occlusion; 2. it provides supervision for the affordance learning.\n*The proposed system outperforms all baselines in real world and achieves a success rate of 90%.\n\n\nLimitations:\n* The evaluation is conducted on a simplified setup.  The towel is lifted by the robot so that is partially unfolded by the gravity. I don't think edge detection on a hanging cloth is a very difficult task, as there aren't many folds or wrinkles. Many of the current cloth folding pipelines can only be applied when the cloth is placed on the table.  It would be nice to see the effectiveness of the proposed system on on-table crumpled cloth.\n* The system is only evaluated for towel, which is kind of limited. I doubt it if will work for clothes with more complicated geometries, such as Tshirt (multiple corners), as double-layer clothes. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper tackles a very important but challenging problem in deformable object manipulation: cloth grasping, and an interesting approach. Accurate cloth grasping can be really helpful for the downstream tasks, such as cloth flattening and folding. I believe this paper will have a significant impact on the field and I strongly recommend for acceptance.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "It will be nice if the authors can provide results of the proposed method under more challenging setting (without lifting-up, clothes other than towel), or at least the discussion about it. It will help the community to better understand the limitation of the method."}}, {"review_id": "BLTS2YuNcwu", "reviewer": "Reviewer_Bk2e", "summary": "This paper describes a method for manipulating a towel from its initial configuration to a configuration where adjacent corners are grasped, after which subsequent tasks could be performed like folding. Using a bimanual robot setup, they train a tactile classifier to distinguish between edges, folds, and non-edges to confirm successful edge grasps. A \"tactile affordance network\", trained from synthetic depth images and fine-tuned in real, predicts viable edge grasps given a towel dangling from one gripper, and uses the above classification network to self-supervised collect 1000s of additional real datapoints. An additional output of the classification network is the position and orientation of the edge, used for closed-loop control during sliding.\n\nIn experiments, they test on (1) a small towel, and (2) a large towel, with different manipulation strategies for each. The system exhibits great performance on small towels, and performs reasonably on larger, more challenging towels.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The approach proposed is a sensible, well-motivated strategy for interacting with large deformable 2D objects.\n- The related work section is thorough and well contextualized.\n- The methods presented are technically sound, well-executed, and feasible to scale to different types of cloth with additional work on data collection.\n\nOverall I enjoyed the paper, and feel with some writing clarity changes and additional experiments it is a valuable contribution to the deformable object manipulation community.\n\nSuggested changes:\nIn my view there are two key weaknesses that should be addressed, and some smaller comments which I left in a list below.\n1. $\\textbf{Writing clarity}$: The writing is unclear in a some places, specifically around experiment details and the construction/usage of the classification network(s). \n- How was the threshold for grasp execution chosen? \n- How is the orientation of an edge grasp determined from the pixel in the depth image selected via the affordance network?\n- How is the 92% tactile classification accuracy evaluated? using a test set from the original data? What is the distribution of classes in the test set and how many of each?\n- It seems to me that the corner classification from the tactile network is never used in the pipeline, and instead shear force serves as a proxy for detecting a corner while sliding. Why not use the available corner classification network for this part of the system? The experiments section states that corners are grasped by \"choosing the lowest point\" while dangling a towel, but it's unclear if the corner classification is used here to confirm a grasp, and what happens if a corner is not grasped. If it's never used, I would suggest removing it from fig 4 and the video/text for clarity. \n- Fig 4 seems to conflict with the classification section: in the text it's stated that the tactile classification network distinguishes between {no fabric, all fabric, edge}, and the grasp supervision network can additionally distinguish between edges and folds, however corners are never mentioned. Fig 4 suggests that the grasp supervision network can detect corners\n- The success rate of corner grasping (sec 4.4) is not discussed: is it 100%?\n- The initialization of experiments is unclear: is the procedure described in sec 4.4 used to initiate every experiment? What happens if the corner grasping during initialization fails? is this reflected in the success rates presented? Is the same procedure used to initialize horizontal sliding as for vertical sliding?\n\n2. $\\textbf{Experiment scope}$: I feel the main benefit of this method is its ability to deal with large towels because of its active perception and dynamic sliding primitives. Using small towels does validate the system, but I feel is less convincing than testing on larger towels where visual methods would struggle more.\n- Though large towels are experimented with (horizontal sliding), the full pipeline of grasp affordance+tactile sliding seems to never be tested on larger towels, and I wonder why this is.\n- There should ideally be experiments highlighting the entire pipeline's success rate on manipulating a (small) towel from a crumpled state to folded, since this is the primary goal the paper describes, rather than reporting success on grabbing an edge and on sliding separately.\n1. $\\textbf{Miscellaneous notes}$:\n- This paper uses towel edge sliding with tactile sensing as an evaluation, which is worth mentioning in the related work: https://arxiv.org/pdf/2011.07213.pdf\n- It seems like there should be different thresholds for grasp affordance for the sim2real and finetuned datasets, since the sim2real network's confidence could be lower having never seen real towels before. However, lower confidence doesn't necessarily imply lower success (if all outputs are scaled to be lower but ordered the same way), so it would be a fairer comparison to choose a different threshold for each network using the same standardized technique (eg, based on statistics of grasp confidence on the test set)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think the paper presents a compelling system for manipulating towels using a hybrid of vision+tactile perception, and should be accepted as is with changes to writing which improve clarity and some additional experiments showcasing performance on a full folding task. If full folding experiments on long towels were included, I would increase my vote to a strong accept. Right now, I would consider myself somewhere between a weak and strong accept because of the limited experiments with larger towels.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "I think the writing changes suggested in strengths/weaknesses should be addressed, and experiments should be run with full pipeline rollouts folding small towels from an initial crumpled state to report overall success rate on the goal task.  However I understand the difficulties of adapting the system to conduct full pipeline rollouts with large towels, so this aspect is less important during revisions, although would significantly strengthen the paper if included!"}}]

Author rebuttal:

Decision: Accept (poster)