JA-TN: Pick-and-Place Towel Shaping from Crumpled States based on TransporterNet with Joint-Probability Action Inference

Authors: halid abdulrahim kadi
kasim  terzić
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-05 21:58:46

Abstract: Towel manipulation is a crucial step towards more general cloth manipulation. However, folding a towel from an arbitrarily crumpled state and recovering from a failed folding step remain critical challenges in robotics. We propose joint-probability action inference JA-TN, as a way to improve TransporterNet's operational efficiency; to our knowledge, this is the first single data-driven policy to achieve various types of folding from most crumpled states. We present three benchmark domains with a set of shaping tasks and the corresponding oracle policies to facilitate the further development of the field. We also present a simulation-to-reality transfer procedure for vision-based deep learning controllers by processing and augmenting RGB and/or depth images. We also demonstrate JA-TN's ability to integrate with a real camera and a UR3e robot arm, showcasing the method's applicability to real-world tasks.

Main paper: 
 introduction: 
 Despite many recent efforts in the robotic community, cloth manipulation remains a key challenge due to its complex deformation and severe self-occlusion [1, 2] . Towel shaping is a set of critical subtasks of towel manipulation, including flattening and folding a rectangular fabric. Regarding initial states, folding can be classified into folding-from-flattened and folding-from-crumpled tasks, where the latter is a more complex problem [3] -a controller must flatten the towel first before folding it into a goal state; in addition, a robust controller should recover from failure states.Most data-driven approaches train on a square fabric for flattening [4, 5, 6, 7, 8] , folding-fromflattened [9, 10] , or both [11, 12, 13, 14] . While some demonstrate their generalisability to other rectangular fabrics [4, 9, 12, 13] , only a few can achieve an intended shaping on various types of fabrics from a large set of different initial states; they are based either on intermediate goal instructions [12, 13, 14 ] or a hierarchical system using hybrid action primitives [1, 15] . Our method JA-TN is the first to demonstrate various folding-from-crumpled tasks (see Figure 1b ) using singlegripper pick-and-place (P&P) controllers on various rectangular fabrics varying in sizes, length-towidth ratios and colours. In a nutshell, we make the following contributions:1. Our JA-TN towel-shaping controller extends the TransporterNet architecture [16] with our joint-probability action inference and achieves state-of-the-art performance for flattening, folding-from-flattened and folding-from-crumpled tasks in simulation.2. We develop three standard benchmark domains -Mono Square Fabric, Rainbow Square Fabrics, and Rainbow Rect Fabrics -based on SoftGym [3] with towel-shaping tasks and corresponding oracle expert policies that are vital for the success of JA-TN.8th Conference on Robot Learning (CoRL 2024), Munich, Germany.  
related work: 
 Robotic Cloth Manipulation Classical cloth manipulation systems are restricted to a narrow range of configurations, whereas data-driven approaches allow robust and general controls in open environments with a broader range of contextual configurations [2] . A large amount of data-driven research has tried to address cloth flattening and folding (including fabrics and garments) since 2018 [17, 18] . These methods mainly vary regarding their action space. (1) Gravity-based methods stretch the article in the air using dual grippers by picking the two key points [19] then adopt fling action [20] to flatten the article; finally, a folding heuristic method, such as g-fold [21] , to achieve target shape.(2) Pick-and-place (-drag) methods employ a single or dual gripper to pick the key points on the article to transport them towards target locations for accomplishing the flattening and folding [22, 23, 9, 7] . (3) Adopting hybrid and variations of the above two action primitives demonstrated significant results in achieving many types of garment manipulation [1, 24, 18, 15] .Data-driven P&P Towel Shaping Many data-driven controllers in P&P towel shaping utilise keypoint detection [24, 25] , template registrations [1] , optical flow [26, 9] , visual descriptors [27] , point clouds [18] and mesh reconstructions [4, 28] as intermediate representations. Some others adopt end-to-end training [17] , which produces a latent space that can be used for policy learning [22, 13, 6] and planning [23, 12, 7] . Single-purpose deep reinforcement learning (RL) can flatten a crumpled towel [17, 7, 22, 4, 29] and perform specific folding-from-flattened tasks [29] with specifically engineered states and rewards biased towards the domain. Mesh-based planning methods [28] , such as VCD and VCD-GI [4] , and latent planning methods, such as PlaNet-ClothPick [7] , achieve state-of-the-art (SoTA) performance on flattening a fabric. Goal-conditioned RL has struggled with general towel-shaping tasks due to the difficulty of engineering proper dense reward functions and learning from sparse ones [23, 12, 6] . Similar to our approach, Wu et al. [8] also utilise a TransporterNet to train a pick-and-place visual affordance map to flatten a square fabric without demonstration. Their method depends on pixel-wise reward function using domain-specific heuristics for self-supervised learning, and it only reaches 76% coverage in simulation and 68% coverage in real trials; it is not clear how it can be applied to various shapes of fabrics and folding tasks.In contrast, successful approaches for folding adopt certain variations of imitation learning. Ganapathi et al. [11] use learned visual correspondences [30] across different fabric configurations to account for the spatial relationship between the current and the goal image; their system can perform a diverse range of fabric manipulation tasks, including various types of folding using a single demonstration trajectory as guidance. FabricFlowNet [9] achieves multi-step fabric folding from canonical states (flattened at the camera centre) with one or two grippers using goal-directed images; its underlying goal-condition policy learns to generate a pick heatmap conditioned on optical flows produced by a pre-trained flow network, then it selects the place action based on the optical flow and the chosen pick position, but it does not support folding from crumpled or arbitrary flattened states. Foldsformer [10] achieves multi-step folding-from-flattened by utilising a space-time attention mechanism [31] to capture the instruction information of demonstrations, but it cannot flatten. TransporterNet (TN) [16] is a vision-based end-to-end method that preserves the spatial structure of the RGB-D input to infer pick and place positions on images; Seita et al. [6] extend the approach to goal-conditioned control domains and apply it to cloth-shaping and bag manipulation tasks in simulation. 
method: 
 TransporterNet (TN) is a vision-based end-to-end behaviour-cloning approach for P&P object manipulation [16] . It comprises of pick q pick (•|x) and place q place (•|x, a pick ) networks where the latter is composed of convolutional key f key and query f query networks. TN learns to infer the pick-action heatmap and selects the most probable action a pick = arg max x,y q pick ((x, y)|x), then uses the query-map of the region-of-interest of the pick action to cross-correlate to the key-map of the observation x for estimating its place action, i.e., a place = arg max x,y f query (x[a pick ]) * f key (x).Place network samples rotational instances of the region of interest (before feeding into the query network) to produce heatmap channels that signal the rotational degrees of the end-effector when placing. It adopts random-pivot rotation as an augmentation technique on RGB-D input images during training. The action maps use one-hot encoding with a value of 1 at the target pick position and 0 elsewhere. In inference, it selects the best combination of pick and place positions based on a selected pick position to act on the environment. Vanilla TN fails on fabric flattening tasks in Pybullet Raven Benchmarks [6, 16] and the simulation is far from realistic. However, TN shows promising results in other P&P domains so we assess and extend it in our towel-shaping tasks.To ensure an efficient selection of P&P positions during test time, we first mask the pick-heatmap with an object mask to filter out the pick positions that are not on the object, and we select the best joint probability of P&P, i.e., a * = arg max apick,aplace p mask (a pick ) × q(a pick |x) × q(a place |a pick , x).We sample 10 of the best picks to perform inference in practice and select the most probable one that differs from the last pick action. We utilise centre-pivot random rotation instead of random-pivot for data augmentation and update the network with a batch size of 10 (instead of the original 1). In the rest of the paper, we refer to this method as JA-TN. The success of JA-TN highly depends on high-quality expert trajectories, which we generate by manually designed oracle policies.Oracle Towel Smoothing (OTS) One-gripper P&P flattening, or smoothing, is a complex problem even if the true state of the article is known, and a controller should address different crumpled states of the towel. Table 1 presents the heuristics of our oracle algorithms in various scenarios.Flattening case-policy picks and places the visible corner furthest from its target flattening position, which is calculated by scanning the whole pixel space to find the minimum total distance towards the targets from all cloth corners -considering symmetry including both sides and the vertical as well as horizontal displacements of the towel. Revealing-Corner policy picks the particle on the clothborder and closest to a hidden corner and its target flattening position, then the policy drags the towel to the opposite of the hidden corner. Revealing-Edge policy targets the hidden edge-point furthest from its closest cloth border, then picks the cloth-border point and places it on the opposite side of the hidden point. Untwisting grabs the corner whose one cloth edge is above the other edge that excludes the corner and drags the grabbed point towards the perpendicular direction of the distance between existing twisting corners. Dragging-to-Opposite picks the cloth-border point furthest from the camera centre and drags it to the opposite side of the centre. Note that the OTS needs to use a more significant dragging distance for adjustment while addressing larger fabrics due to larger friction. The proposed algorithm is relatively robust regarding the fabric's initial crumpled states, size, and length-to-width ratio. Still, it may fail to address higher-order twisting and cases where a corner is wrapped inside the towel. Table 2 shows the policy can generate expert trajectories for behaviour cloning algorithms. Oracle Towel Folding (OTF) Oracle policy for folding assumes a flattened article and maps its surface to a 2-dimensional plane ranging from 0 to 1 -(0, 0) represents the top-left particle of the towel whilst (1,1) denotes the bottom-right. OTF regards a specific folding as a sequence of P&P positions on the article map.Training Details of JA-TN and TN We train JA-TN and TN using 2000 OTS demonstrations for flattening, 1000 OTF demonstrations for corresponding folding-from-flattened tasks and the combination of both for folding-from-crumpled tasks. We train JA-TN with 80,000 update steps and vanilla TN with 40,000 update steps as it starts to overfit beyond this (see Figure 9b and 9c ). We also make the following modifications to adapt the vanilla TN to our shaping tasks. We remove rotation-variant sampling because it is not helpful in our domain. We use image resolution 128×128 instead of the original 320×160. Rather than synthesising the top view from 3 separate cameras as in the original paper, we use a single camera to capture the top view of the towel. Moreover, we clip the depth image between 1.47 and 1.51 (as the camera is 1.5m above the surface in simulation) and normalise both RGB and depth images to range from -0.5 to 0.5. 
experiments: 
 We implement our methods in SoftGym [3] , which is a particle-based cloth simulator, and we develop three domains: (1) Mono Square Fabric (MSF) contains a 40cm × 40cm square fabric that has the same back and front colours across the trials; (2) Rainbow Square Fabrics (RSF) contains a set of square fabrics whose side-lengths are uniformly sampled from 20cm to 70cm, and the colours of this fabric are also uniformly sampled from all possible colours; (3) Rainbow Rectangular Fabrics (RRF) is a similar setting to RSF but the length and the width of a fabric can be different. All three domains include a top-down camera that looks down at the centre of the world surface from a height of 1.5m. Cloth stretch, bend, and shear stiffness are set to (0.8, 1, 0.9) with a particle radius of 6.25mm and a particle mass of 0.5.Towel Flattening We employ bounded (clipped between 0 and 1) normalised coverage (NC) and normalised improvement (NI) to evaluate the step-wise performance of flattening controllers [4] . We additionally use success rate (SR) and steps-to-finish (#2F), on success or end of the trial, to assess the robustness and efficiency of the methods; a flattening trial succeeds when the controller can achieve above 0.99 of NC within 30 steps. We evaluate the controllers on 65 trials of MSF and 56 trials of RRF. Figure 3a shows the final step performance of JA-TN against SoTA controllers in the evaluation setting of Lin et al. [4] . Table 2 shows improvement of our JA-TN compared to vanilla TN in terms of performance and efficiency in both MSF and RRF domains.Towel Folding from a Flattened State For folding-from-flattened tasks, we use standard mean particle distance (MPD) between the particle pairs in possible corresponding folding groups [9] as well as additional largest particle distance (LPD) to evaluate the step-wise performance. A folding succeeds when LPD goes below a predefined threshold value (7cm for diagonal-cross folding and 4.5cm for others) within 25 steps. All folding-from-flattened tasks in the 3 domains share 30 test trials where the article is randomly rotated and placed within the camera view. We train JA-TN and TN on 1,000 demonstrations for each folding and examine (1) all types of folding illustrated in Figure 1b on MSF, (2) diagonal-cross, all-corner inward, corners-edge inward and rectangular folding in RSF, and (3) side, double-side and rectangular foldings in RRF. We did not examine double-straight folding [10] as it is challenging even for the oracle to achieve with a single gripper. Table 3 and Figure 3b show that JA-TN performs robustly on these folding tasks in all 3 domains, and it matches the SoTA performance of Foldsformer [10] , a method that cannot fold from crumpled.Towel Folding from Crumpled Folding-from-crumpled tasks share the same initial states as the ones for flattening and the same success criteria as folding-from-flattened tasks, but they must finish within 55 steps. We train JA-TN on 2,000 flattening + 1,000 folding-from-flattened demonstrations.We examine (1) all-corner inward, diagonal cross, corners-edge inward, double-side and rectangular foldings in MSF and (2) the latter in RRF. Table 4 shows JA-TN's high performance in the MSF setting and its potential in the more complex RRF setting.Ablation Study We conduct ablation tests of JA-TN on flattening and double-side folding tasks in RRF. Table 5 shows that mask filtering is essential for the success of flattening agent as No Mask ablation tends to choose the pick position of No-Op policy, whilst the joint-action inference is crucial for the success of folding agent as it reduces ineffective folding steps. Training with a larger batch size and using centre-pivot rotation in data augmentation are critical for the system's success in both tasks, as the former reduces overfitting and improves the learning (see Figure 9b and 9c ), and the latter avoids feeding the networks with undesirable actions.Sim2Real Transfer Many data-driven P&P towel-shaping methods developed in SoftGym demonstrate the ability to transfer to real trials mostly using depth-only images [4, 10, 9, 8] . We retrain JA-TN with masked RGB (and/or depth) images in simulation and adopt a segment-anything model [32] with heuristic mask selection in physical inference -the largest average colour difference between the masked and background regions. We employ a top-down RealSense D435i camera positioned 0.72 metres above the table surface and rely on human judgment to assess the success of a trial. We choose one square (sky-blue, 14cm) and one rectangular (white, 15cm×7cm) fabric for physical evaluations. During the physical experiment, we feed RGB(-D) camera input to JA-TN and ask it to generate pick-and-place action primitives (see Figure 4b ). To isolate the effect of grasping errors common in this domain such as misgrasping and grasping multiple layers, we then manually perform each action on the real cloth (human in the loop) before repeating the process. This eliminates grasping errors and allows us to focus on the quality of the learned policy operating on live RGB-D images. This demonstrates that our simulation-trained system can create effective policies on real towels based on real camera input (see Figure 2 ). Whilst flattening and foldingfrom-flattened systems transfer nicely, flattening-from-crumpled agents cannot directly transfer; we must stitch the flattening and folding controllers with a predefined mask area. Table 6 shows our method's plausibility in physical trials on most tasks using RGB images, but note that it struggles to perform double-side and rectangular folding. We have also conducted experiments using a UR3e robot arm and an Active Parallel Gripper with a printed stick-like extension, which have managed to successfully flatten real fabrics in a completely autonomous fashion (see Figure 4c ). 
discussion: 
 JA-TN can effectively flatten various fabrics from highly crumpled states in both simulation and physical applications, regardless of their colour, size, length-to-width ratio, or background. It can also perform different folding types from flattened and crumpled states. To our knowledge, this is the first learning algorithm capable of excelling in all three tasks. It has potential applications in other P&P domains and object shapes, as we did not introduce inductive biases into the learning process apart from the Oracle policies -Figure 10 in the Appendix demonstrates that JA-TN can be applied to other complex shapes of cloth-like deformable objects, such as T-shirts.Behaviour cloning methods like JA-TN, trained with around 20,000 transitional steps, are much more data-efficient compared to reinforcement learning (RL) methods, such as PlaNet-ClothPick (around 1 million steps) [2] and VCD (around 100,000 steps) [4] for fabric flattening. For folding, it is hard to engineer reward functions for each type; Lee et al. [13] (using Hindsight Replay Buffer [33] ) and VSF [12] (using visual MPC planning [34] ) adopt a goal-condition strategy to bypass the problem, but these systems do not outperform other supervised learning and imitation learning methods such as FabricFlowNet [15] , Foldsformer [10] and our JA-TN. To our knowledge, no RL algorithm has demonstrated the ability to fold directly from a crumpled state.Limitations JA-TN struggles with highly wrapped and severely twisted states in simulations and folded states (both outside and inside) in reality due to its inability to distinguish folds when both sides are the same colour. While it reliably performs most folding-from-flattened tasks, we see room Real-depth images often suffer from noise, blurriness, and holes, even after calibration and filtering. To address this, we apply a cloth mask, heuristically selected from a segmentation network (green line), and set the depth image background to the camera height (purple line). Since fabrics vary in thickness, we standardise the cloth-top and cloth-bottom to 0 and 1 for robustness, then invert these values so 0 represents the background and deepest view, while 1 represents the cloth top (red line). We also normalise RGB images and apply mask segmentation to RGB images, ensuring the policy remains background-agnostic (purple line).for improvement by addressing rectangular folding in simulations and reality as key picking points are often wrapped inside during the process; the system cannot distinguish when one side of the fabric overlaps at the bottom but still needs completion, so it may require a goal configuration to improve its recognition capabilities. Lastly, JA-TN only transitions from flattening to folding when it perceives the fabric as perfectly flattened, making physical trials difficult.We aim to address the identified limitations of the shaping controller and enhance the transferability of JA-TN in folding from a crumpled state. Additionally, we plan to extend JA-TN's capabilities in two-gripper cloth manipulation to achieve more effective rectangular and doublestraight foldings. We also intend to evaluate our method's effectiveness in other garment-shaping tasks. Furthermore, we seek to use goal-conditioned methods to unify various folding techniques, eliminating the need for training specialised systems for each folding type. 
conclusion: 
 We propose a Joint Probability TransporterNet (JA-TN) that achieves state-of-the-art performance for flattening and folding-from-flattened tasks. We believe JA-TN can be applied to other pick-andplace domains as we do not introduce inductive biases towards our domain apart from the Oracle expert scripts. Our experiments on real towels and real camera images show our method's promise in real-world towel-shaping trials, and we plan to improve the performance of folding-from-crumpled controllers and their transferability. 
initialisation script for crumpled states in simulation: 
 We run 1000 trials for Mono Square Fabric (MSF), 2000 for Rainbow Square Fabrics (RSF), and 5000 for Rainbow Rectangular Fabrics (RRF), where the trial's configuration ID seeds the random state. Crumpled states are initialised as follows: (1) the script selects a particle on the fabric and lifts it to a random height between 0 and 40cm; (2) after it stabilises in the air, the cloth is dropped; then (3) the script relocates the fabric (without changing the crumpled state) to a random position within a 30cm circle centred around the camera; (4) then, 70% of the time, it again selects a random particle and drags the cloth towards a random position in a circle with a radius of 30cm and a height of 10cm; it drops the cloth and re-centres it randomly again; finally, ( 5) the cloth's re-centring and random rotation continue until all particles are within the camera view. This results in a wide range of complex initial states. Heuristics for OTF Rectangular, side, and double-side foldings are more challenging to achieve with one-picker operations, as they tend to conceal key pick-points inside the article, leading to failure; hence, OTF carefully produces the folding sequence based on fabric properties, which avoids such scenarios and the overlapping of the two sides in double-side folding.Limitations of Oracles As shown in Figure 6 , the flattening Oracle struggles with highly twisted and wrapped states, but this only occurs once in all 56 tested trials in simulation. Figure 7 demonstrates that the folding Oracle cannot detect failing states until reaching designated folding steps, and rectangular and double-side foldings are particularly challenging due to the concealed key points. Preprocessing of Real Images As shown in Figure 4b , the real depth image can be noisy and blurry, and it can also contain holes even after calibration and being applied depth pre-processing Figure 7 : A Rectangular-Folding Failure of Oracle Towel Folding filters provided by Realsense SDK. To address this, we apply a cloth mask and make the background of the depth images the same as the camera height. However, different fabrics have different thicknesses, so we map the cloth-top and cloth-bottom consistently to 0 and 1 to make our system robust to cloth thickness. Then, we flip the values so that 0 represents the background and the deepest part of the view, and 1 represents the cloth top. We also map the values to 0 and 1 for RGB images by dividing 255 from the original RGB image. Then, we applied mask segmentation on depth and RGB images so that the policy can be agnostic to the view background, a common practice in this domain [9, 10] . Training details of Sim2Real Policies To transfer the learned policy into the real world, apart from remapping, value flipping and masking as in preprocessing real-depth images, we also add Gaussian noise of 0.01 and Gaussian blurring with a kernel of 11, during training, to the simulated depth to emulate the real depth images (see Figure 8b ). We also add a Gaussian noise of 0.02 to the simulated RGB images. Figure 8a shows how these augmentations influence JA-TN performance on flattening.Robot Setup As shown in Figure 4a , we use a 6-degree-of-freedom UR3e robot arm, an Active Parallel gripper and a Realsense D435i camera to conduct our robotic experiments. The camera is 0.72 meters above the table surface; we manually adjust the camera pose so that the depth images give an even depth reading across the view, and the camera is parallel to the robot base axes. We also 3D-printed a stick-like thin gripper extension to grasp the target cloth point with higher precision; whilst open, the tip-to-tip distance is 2.4cm. The robot working space is a 28cm × 28cm square region on table syrface. We adopt MoveIt 2's Python API under ROS 2 Humble [35] with OMPL's RRTConnect [36] motion planner to achieve quasi-static pick-and-place robot control as well as establishing the communication between the robot and GPU program.The transformation between the normalised pixel space (see the raw RGB image in Figure 4b ) and the world space (same as robot base space) is calculated using the cropping offset and scale of work space, the intrinsic of the camera and the extrinsic transformation matrix between the camera frame and the robot's base frame. The calibration of the extrinsic parameters can be carried by eye-to-hand calibration, but it does not necessary gives a precise solution; hence a further fine-tuning is required.For each pick-and-place action, the end-effector goes as deep as it can on the table to grasp the cloth (i.e., the tip of the end-effector is at the same height as the robot's baseline). Then, it picks the cloth up 5cm above the surface to go to the place point that is 3cm above the surface to release the cloth. While taking each picture, the robot gets to a home position that makes it completely absent from the camera view. 
 
 Appendix: The gripper can misgrasp the cloth, grasp multiple layers of the cloth, and the cloth can attached to the gripper after release; the solutions for tackling these challenges are out of the scope of this paper. Also, we encounter edge (and/or corner) alignment issues during folding tasks. Currently, planning each motion trajectory takes 2 seconds, and we plan to explore other motion planners to improve efficiency. Additionally, the robot's workspace is limited because the planned trajectory fails to produce smooth joint movements when it approaches its base. Besides, the current robotic system cannot automatically detect the success states, and it only stops when it reaches defined maximum steps. Lastly, we must manually run the associated configuration for an intended task.  


Decision: This paper studies pick-and-place actions for towel manipulation by building on the Transporter Network architecture. The rebuttal convinced two reviewers who rated "weak reject" to increase scores to "weak accept." There is somewhat limited technical novelty here; for example the "mask-filtered" part should be de-emphasised; in fact this area chair strongly recommends removing that term from the paper title all together. 

Nonetheless, the empirical results appear good enough for the robot learning community. The authors are encouraged to place more emphasis on describing their real world manipulation results. There has been some increased emphasis on the revised version from the rebuttal but there should be even more emphasis (for maximum impact).


Strengths:
- It tackles the important problem of shaping and adjusting deformable towels.
- Extending and improving Transporter Networks is impressive.

Weaknesses:
- The paper may be more experimental (instead of proposing a new algorithm) and thus it would help to have substantial real world robot experiments.
- The contributions may be limited compared to current work. For example, some of these might reduce to hyperparameter tuning (e.g., changing batch size). Also, filtering pick points so that they only go on the cloth might not be enough of a technical contribution.
- Changing colours of the fabric is interesting but also less interesting from a novelty and generalisation perspective, as a reasonable method should be robust to object color.
- It also requires a substantial amount of demonstrations.

Reviews: review: title: title: Sim2Real method for cloth manipulation with novel learning method and planner distillation
paper_summary: paper_summary: See Review
main_review: main_review: ## Summary

This paper introduces MJ-TN (Mask-Filtered Joint-Probability TransporterNet), a method for robotic towel manipulation that can flatten, fold from flattened states, and fold from crumpled states. The approach extends the TransporterNet architecture with mask-filtered joint-probability action inference to improve operational efficiency. The authors develop three benchmark domains based on SoftGym with towel-shaping tasks and corresponding oracle expert policies. They evaluate MJ-TN's performance against state-of-the-art controllers in simulation and demonstrate its ability to transfer to real-world scenarios. The method is tested on various types of fabrics and folding tasks, showing robust performance across different scenarios. The authors claim the following contributions:

* Development of MJ-TN, which achieves state-of-the-art performance for flattening, folding-from-flattened, and folding-from-crumpled tasks in simulation.
* Creation of three standard benchmark domains with towel-shaping tasks and corresponding Oracle expert policies.
* Demonstration of sim-to-real transfer capabilities for MJ-TN in physical trials.

### Strengths

Overall, the work is a nice contribution to the cloth manipulation literature. It makes good use of learning and provides demonstrations of a real system. In particular, the key strengths of the paper are:

* Comprehensive evaluation: The authors test their MJ-TN method across a wide range of scenarios, including different fabric types, sizes, and folding tasks. They also compare their results against state-of-the-art methods in both flattening and folding tasks, providing a thorough assessment of their approach's capabilities.

* Real-world evaluation: The paper demonstrates the method's ability to transfer from simulation to real-world scenarios, showing its potential for actual robotic applications. This sim-to-real transfer is a crucial step in bridging the gap between simulated environments and practical implementations.

### Weaknesses

* It is interesting to me that the authors are distilling a policy from a fully observed simulation to a partially observed robot deployment. It would be nice to see some discussion of this approach (in comparison to, e.g., RL)

* While the testing on the robot is quite good, further experiments in more diverse conditions would help to support the claims.

* It is a small point, but it would be good to see some evaluation of the Oracle policies that generate the data. Are there any configurations they struggle in?
questions: questions: I'm curious if the authors have any responses to my critiques above. I think there is room for improvement, however, I don't think there are any specific issues that need to be addressed prior to acceptance.
limitations: limitations: 2
review_summary: review_summary: Overall, this is a good demonstration of learning on a robotic platform that demonstrates strong performance on an interesting problem
title: title: Although the results show improvement, creating more benchmarks, oracle policies and providing more experiments maybe is recommended.
paper_summary: paper_summary: The paper addresses the problem of fabric folding from crumpled state on square and rectangular fabrics. The proposed method extends Transporter Networks, TN trained using behavior cloning (BC) predicts pick and place points directly from a 3D reconstruction of multi-view camera images. The extended version as proposed adds a mask to filter out the predicted pick points not on the object and uses the joint probability of filtered pick and place to determine both the pick - place actions during inference leading to improvement in performance. In addition, the paper introduces 3 benchmarks along with oracle expert policies. These policies behave as primitives (e.g., Revealing-corner, Revealing Edge) and are used to collect data for behavior cloning. Quantitative and Qualitative results are shown for simulation and real experiments.
main_review: main_review: ## Strengths

The paper addresses an important problem and is of interest to the sub-community of non-rigid body manipulation. The expert primitives/policies designed can be widely used by the community for further data collection on different kinds of materials. In addition, improved inference performance shows that adding the mask helps the success rate. Extensive experiments are done on the 3 benchmarks proposed that show the effectiveness of the mask filter and joint probability for action prediction during inference. An ablation study justifies the use of the proposed two contributions.

## Weaknesses

The technical contribution is minor as it adds on top of existing works [1], [2].

1. A mask of the object is applied to the existing output of the TN network.
2. A single RGB-D image is used instead of 3D-reconstruction from multiple camera view images.
3. Training tricks like data augmentation.
The variety of objects experimented on is minimal.

[1] Deep imitation learning of sequential fabric smoothing from an algorithmic supervisor.
[2] Transporter networks: Rearranging the visual world for robotic manipulation
questions: questions: The reviewer suggests experimenting with more realistic non-rigid bodies or even a larger variety of non-rigid objects (ropes) and providing

1. oracle policies
2. benchmarks

for the same would make this a good contribution to the community.  However, this may require using a different physics simulator, which the reviewer understands may not be feasible.

The reviewer also recommends highlighting the paper's contributions to distinguish it from the related work.
limitations: limitations: 3
review_summary: review_summary: Although the results show improvements, the contribution is minor. More oracle policies and benchmark design on garments and other non-rigid bodies would be of more interest to the community as a benchmarking and dataset paper.
title: title: Want to see real robot videos and photos
paper_summary: paper_summary: This paper presents a method to fold a piece of fabric from difficult initial states such as crumpled state. The method improves upon the Transporter Network by adding a mask around the piece of fabric and a tuning a few hyperparameters such as batch size. The authors show positive results in simulation and mention real robot transfer but real robot videos are missing in the supplementary material. It is difficult to see that there is a real robot from the photos in the paper (such as in Fig. 9).
main_review: main_review: Strength:
1. Tackles a difficult robotics problem
2. Shows positive results in the sim and claims successful sim to real transfer

Weakness:
1. It’s unclear what the contribution of the paper is. From line 93: The method of filtering only points on the cloth is an obvious choice one would make when applying an existing pick-and-place architecture for cloth folding. It doesn’t make sense to sample pick points not on the cloth. Similarly, changing batch size from 1 to 10 is hyper-parameter tuning and doesn’t count as a contribution.
2. I am not opposed to purely experimental papers and in fact, I encourage such papers. If indeed the paper is purely experimental, then it should be positioned as so with a very strong emphasis on real robot results. There should be learnings presented explaining details of which choices mattered the most in good policy performance on the real robot. In that way, the paper would add value to the robotics community allowing others to re-produce their approach and extend the learnings to similar domains.
2. Difficult to appreciate that there is a real robot involved from the photos and videos presented in the Appendix and supplementary material. The real robot results mentioned would be the strongest part of the paper and the authors should emphasize more on them with photos and a video and details on sim to real transfer choices
questions: questions: Please refer to the weakness above
limitations: limitations: 3
review_summary: review_summary: The paper tackles a difficult robotics problem but I feel it needs restructuring and more experimental evidence. Post rebuttal edit: I think the paper looks much better now with the increased focus on real robot results. Hence, I now recommend accepting it.