Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper addresses the problem of cloth grasping by presenting a system that leverages both visual and tactile perception. The proposed system uses tactile perception (edge estimation network, tactile sliding network) to enble closed-loop local control under the occlusion induced by gripper. Another vision-based grasp affordance netowrk is pretrained in simulation and fine-tuned with tactile supervision, and provides global guidance to state identification and grasp point localization. The proposed system is evaluated in real-world and demonstrates impressive performance in real-world cloth grasping.
strengths_and_weaknesses: Strengths:
*  The paper tackles a very important problem in cloth manipulation: cloth grasping. When designing the system, the authors take the common failure mode of cloth grasping into account,  such as gripper collission and grasping wrong number of layers, missed grasp.

* Compared to previous cloth grasping approach, it leverages tactile perception, and in a really clever way: 1. it enables closed-loop manipulation with occlusion; 2. it provides supervision for the affordance learning.
*The proposed system outperforms all baselines in real world and achieves a success rate of 90%.


Limitations:
* The evaluation is conducted on a simplified setup.  The towel is lifted by the robot so that is partially unfolded by the gravity. I don't think edge detection on a hanging cloth is a very difficult task, as there aren't many folds or wrinkles. Many of the current cloth folding pipelines can only be applied when the cloth is placed on the table.  It would be nice to see the effectiveness of the proposed system on on-table crumpled cloth.
* The system is only evaluated for towel, which is kind of limited. I doubt it if will work for clothes with more complicated geometries, such as Tshirt (multiple corners), as double-layer clothes. 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper tackles a very important but challenging problem in deformable object manipulation: cloth grasping, and an interesting approach. Accurate cloth grasping can be really helpful for the downstream tasks, such as cloth flattening and folding. I believe this paper will have a significant impact on the field and I strongly recommend for acceptance.
issues: It will be nice if the authors can provide results of the proposed method under more challenging setting (without lifting-up, clothes other than towel), or at least the discussion about it. It will help the community to better understand the limitation of the method.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper describes a method for manipulating a towel from its initial configuration to a configuration where adjacent corners are grasped, after which subsequent tasks could be performed like folding. Using a bimanual robot setup, they train a tactile classifier to distinguish between edges, folds, and non-edges to confirm successful edge grasps. A "tactile affordance network", trained from synthetic depth images and fine-tuned in real, predicts viable edge grasps given a towel dangling from one gripper, and uses the above classification network to self-supervised collect 1000s of additional real datapoints. An additional output of the classification network is the position and orientation of the edge, used for closed-loop control during sliding.

In experiments, they test on (1) a small towel, and (2) a large towel, with different manipulation strategies for each. The system exhibits great performance on small towels, and performs reasonably on larger, more challenging towels.

strengths_and_weaknesses: Strengths:
- The approach proposed is a sensible, well-motivated strategy for interacting with large deformable 2D objects.
- The related work section is thorough and well contextualized.
- The methods presented are technically sound, well-executed, and feasible to scale to different types of cloth with additional work on data collection.

Overall I enjoyed the paper, and feel with some writing clarity changes and additional experiments it is a valuable contribution to the deformable object manipulation community.

Suggested changes:
In my view there are two key weaknesses that should be addressed, and some smaller comments which I left in a list below.
1. $\textbf{Writing clarity}$: The writing is unclear in a some places, specifically around experiment details and the construction/usage of the classification network(s). 
- How was the threshold for grasp execution chosen? 
- How is the orientation of an edge grasp determined from the pixel in the depth image selected via the affordance network?
- How is the 92% tactile classification accuracy evaluated? using a test set from the original data? What is the distribution of classes in the test set and how many of each?
- It seems to me that the corner classification from the tactile network is never used in the pipeline, and instead shear force serves as a proxy for detecting a corner while sliding. Why not use the available corner classification network for this part of the system? The experiments section states that corners are grasped by "choosing the lowest point" while dangling a towel, but it's unclear if the corner classification is used here to confirm a grasp, and what happens if a corner is not grasped. If it's never used, I would suggest removing it from fig 4 and the video/text for clarity. 
- Fig 4 seems to conflict with the classification section: in the text it's stated that the tactile classification network distinguishes between {no fabric, all fabric, edge}, and the grasp supervision network can additionally distinguish between edges and folds, however corners are never mentioned. Fig 4 suggests that the grasp supervision network can detect corners
- The success rate of corner grasping (sec 4.4) is not discussed: is it 100%?
- The initialization of experiments is unclear: is the procedure described in sec 4.4 used to initiate every experiment? What happens if the corner grasping during initialization fails? is this reflected in the success rates presented? Is the same procedure used to initialize horizontal sliding as for vertical sliding?

2. $\textbf{Experiment scope}$: I feel the main benefit of this method is its ability to deal with large towels because of its active perception and dynamic sliding primitives. Using small towels does validate the system, but I feel is less convincing than testing on larger towels where visual methods would struggle more.
- Though large towels are experimented with (horizontal sliding), the full pipeline of grasp affordance+tactile sliding seems to never be tested on larger towels, and I wonder why this is.
- There should ideally be experiments highlighting the entire pipeline's success rate on manipulating a (small) towel from a crumpled state to folded, since this is the primary goal the paper describes, rather than reporting success on grabbing an edge and on sliding separately.
1. $\textbf{Miscellaneous notes}$:
- This paper uses towel edge sliding with tactile sensing as an evaluation, which is worth mentioning in the related work: https://arxiv.org/pdf/2011.07213.pdf
- It seems like there should be different thresholds for grasp affordance for the sim2real and finetuned datasets, since the sim2real network's confidence could be lower having never seen real towels before. However, lower confidence doesn't necessarily imply lower success (if all outputs are scaled to be lower but ordered the same way), so it would be a fairer comparison to choose a different threshold for each network using the same standardized technique (eg, based on statistics of grasp confidence on the test set)
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think the paper presents a compelling system for manipulating towels using a hybrid of vision+tactile perception, and should be accepted as is with changes to writing which improve clarity and some additional experiments showcasing performance on a full folding task. If full folding experiments on long towels were included, I would increase my vote to a strong accept. Right now, I would consider myself somewhere between a weak and strong accept because of the limited experiments with larger towels.
issues: I think the writing changes suggested in strengths/weaknesses should be addressed, and experiments should be run with full pipeline rollouts folding small towels from an initial crumpled state to report overall success rate on the goal task.  However I understand the difficulties of adapting the system to conduct full pipeline rollouts with large towels, so this aspect is less important during revisions, although would significantly strengthen the paper if included!
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper develops a manipulation framework for picking up and unfurling cloth in tabletop scenarios (e.g. cloth randomly draped on table). The system works by (1) picking up cloth from the table; (2) re-positioning the cloth so it can be grasped at a (detected) corner, and (3) unfurling the cloth by “following” an edge from the detected corner. The authors found that tactile data (from GelSight) was crucial both as an **input** *and* as a **supervision source** for training the necessary perception models (e.g. corner/edge detectors). Data for training these models was collected both in simulation and in the real world. The authors validate these design choices with an ablation study, and demonstrate qualitative behavior in their video.

strengths_and_weaknesses: **Strengths**
* This paper offers a compelling examples for how to use tactile information to solve real world robotics problems. It is particularly interesting that the touch data is used to improve the vision model through supervision, as well as being used as an input.
* The authors clearly did some great task specific engineering and came up with clever strategies for unfolding.
* The video does a good job explaining the method and contextualizing results.

**Weaknesses**
* The experiments only discuss success rates in terms of affordance learning, but *not* in terms of task success. This makes it hard to contextualize their improvement over baselines for the final task. 
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper showcases an impressive framework for solving a challenging real world task. I particularly enjoyed reading how the authors exploited unique sensing capabilities (e.g. tactile data), and what was required to successfully transfer their perception stack from sim to real. While I would like more context on task success (see Issues section), I don’t believe this issue is major enough to warrant rejection.

issues: The authors should address the weakness pointed out above, by giving a clearer idea of how their method improves end-task (i.e. cloth unfolding) success rate in realistic scenarios.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a system that uses vision and tactile perception to unfold clothes by grasping and sliding. The system uses two UR arms, depth camera and vision-based tactile sensors attached to the robots' grippers. Perception is handled with neural networks that are responsible for the classification of grasping edges, supervision of visuo-tactile affordances detection, and pose-estimation/control for sliding. The system is deployed on real robots effectively.
strengths_and_weaknesses: The paper presents advances on a relevant and challenging topic in robot manipulation. The problem of cloth manipulation involves several challenges, that are addressed in a systematic and clear way in the proposed work.
The paper is well structured, it is clear and easy to read. Challenges and contributions are presented in a clear way, and the literature review seems comprehensive to the best of my knowledge. Figures, diagrams and videos are good and helpful to get a grasp of the contributions.
The experiments include an informative set of comparisons, and the relative discussion includes useful insights and analysis.
The limitations reported point out relevant shortcoming of the proposed method.

While the paper explains the experimental setup and method thoroughly, implementation details are missing: for example details related to the networks trained and used in the experiments, how many parameters were used, how long the training time is, etc. Some information on how the robot is controlled, and how the state machine (fig.2) is deployed would be very useful as well to provide a complete picture of the experimental setup.
The analysis would benefit from a discussion on what the failure modes are and why they happen. For example, what are the limitations on the perception side? do you have a quantitative measure of the uncertainty of the tactile classifier? how does that affect the final performance? on the visual perception, what would make your setup more robust? 
The sliding motion seems really hard to obtain, and the results presented are remarkable. Do you have quantitative measures of the robustness or reliability of this controlled movement? How many repetitions are consistently executed on the same piece of cloth? What part of your setup/architecture would further improve the overall performance? 



robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper presents a relevant piece of work, that tackles a challenging and important problem in robot manipulation. The execution of the experiments and the research advances are interesting and, although not general, very promising.
While some implementation details are missing, the work is well executed and the experimental results are remarkable.
issues: Missing or limited details: please add implementation details for reproducibility, as well as quantitative measures indicating uncertainty/effectiveness in your experiments, for example prediction/classification uncertainty, robustness to noise/disturbances, reliability/repeatability of the perception/control loop.
review_title: null

Meta-review: metareview: The paper is of interest and addresses an important problem relating to deformable object manipulation. It utilizes a hybrid of vision and tactile perception for manipulating towels. The idea is original with very good technical quality. The work was presented well and guides the reader appropriately. The reviewer concerns were adequately addressed. Application to more complex tasks remains a concern, if further addressed, it can significantly improve the paper. However, with the current experiments, the paper makes sufficient contributions to the community. 

confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)