FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfolding
Authors: Huy Ha
Shuran Song
Venue: CoRL2021 Oral (2021)
Keywords: Dynamic manipulation
Cloth manipulation
Self-supervised learning

Abstract: High-velocity dynamic actions (e.g., fling or throw) play a crucial role in our everyday interaction with deformable objects by improving our efficiency and effectively expanding our physical reach range. Yet, most prior works have tackled cloth manipulation using exclusively single-arm quasi-static actions, which requires a large number of interactions for challenging initial cloth configurations and strictly limits the maximum cloth size by the robot's reach range. In this work, we demonstrate the effectiveness of dynamic flinging actions for cloth unfolding with our proposed self-supervised learning framework, FlingBot.
Our approach learns how to unfold a piece of fabric from arbitrary initial configurations using a pick, stretch, and fling primitive for a dual-arm setup from visual observations. The final system achieves over 80% coverage within 3 actions on novel cloths,  can unfold cloths larger than the system's reach range, and generalizes to T-shirts despite being trained on only rectangular cloths.
We also finetuned FlingBot on a real-world dual-arm robot platform, where it increased the cloth coverage over 4 times more than the quasi-static baseline did. The simplicity of FlingBot combined with its superior performance over quasi-static baselines demonstrates the effectiveness of dynamic actions for deformable object manipulation.

FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfolding
Huy Ha Shuran Song
Columbia University
httpss//lingboo.sscoolmmiia
Abstract: High-velocity dynamic actions (e.g., fling or throw) play a crucial role in our everyday interaction with deformable objects by improving our effi- ciency and effectively expanding our physical reach range. Yet, most prior works have tackled cloth manipulation using exclusively single-arm quasi-static actions, which requires a large number of interactions for challenging initial cloth config- urations and strictly limits the maximum cloth size by the robot's reach range. this work, demonstrate the effectiveness of dynamic flinging actions for cloth unfolding with our proposed self superviid learning framework, FlingBot. Our approach learns how to unfold piece of fabric from arbitrary initial configura- tions using pick, stretch, and fling primitive for a dual-arm setup from visual ob- servations. The final system achieves over 80% coverage within actions on novel cloths, can unfold cloths larger than the system's 's reach range, and generalizes to T-shirts despite being trained on only rectangular cloths. We also finetuned Fling- Bot on a real-world dual-arm robot platform, where it increased the cloth coverage over 4 times more than the quasi-static baseline did. The simplicity of FlingBot combined with its superior performance over quasi-static baselines demonstrates the effectiveness of dynamic actions for deformable object manipulation.
Keywords: Dynamic manipulation, Cloth manipulation, self-supervised learning
1 Introduction
High-velocity dynamic actions play cru- cial role in our everyday interaction with de- formable objects. Making our beds in the mor- ing is not effectively accomplished by picking up each corner of the blanket and placing them in the corresponding comners of the bed, one by one. Instead, we are more likely to grasp the blanket with two hands, stretch it, and then unfurl it with a fling over the bed. This fluid high-velocity flinging action is an example of dynamic manipulation [1], which is used to im- prove our physical reachability and action effi- ciency allowing us to unfold large crumpled cloths with little as interaction.
Figure 1: Cloth unfolding with dynamic inter- actions. Given a severely crumpled cloth, Fling- Bot uses high-speed fling to unfurl the cloth with as little as one interaction. In this paper, demonstrate that such dynamic actions can ef- ficiently unfold cloths, generalizable to different cloth types, and improve the effective reach range of the system.
From goal-conditioned folding [2] to fabric smoothing [3, 4), prior works have achieved success using exclusively single-arm quasi- static interactions (e.g.. pick & place) for cloth manipulation. However, these approaches require large number of interactions for challenging ini- tial configurations (e.g.. highly crumpled cloths) rely on strong assumptions about the cloth (e.g., predefined keypoints). Additionally, since the robot arm cannot manipulate the cloth at locations it can't 't reach, the maximum cloth size is greatly limited by the robot arm's 's reach range.
Sth Conference Robot Learning (CoRL. 2021), London, UK.

---

In this work, we focus on the task of cloth unfolding, where the goal is to maximize the cloth's 's coverage. Because unfolding reveals key features of the cloth for downstream perception and ma- nipulation, is is a typical first step for many cloth manipulation pipelines. An ideal cloth unfolding approach should be:
Efficient: the approach should reach a high coverage with a small number of actions from arbi- trarily crumpled initial configurations.
Generalizable: the algorithm should not rely on heuristics (e.g.. grasp predefined key points). This is especially important when unfolding is only the initial stage of cloth perception and manipulation pipeline, where key points are not visible or severely occluded, and when the system must handle cloth types unseen during training, which may not contain the predefined key points.
Flexible Beyond the Workspace: the approach should work with cloths of different sizes, includ- ing large ones which lie outside the robot's physical reach range.
To achieve this goal, we present FlingBot, a self-supervised algorithm that learns how to unfold cloths from arbitrary initial configurations using a pick, stretch, and fling primitive for a dual-arm setup. At each time step, the policy predicts value maps from its visual observation and picks actions greedily with respect to its value maps. To provide the supervision signal, the system computes the difference in coverage of the cloth before and after each action - the delta-coverage from the visual input captured by a top-down camera. FlingBot achieves over 80% coverage within 3 actions on novel cloths and increases the cloth's coverage by more than twice that of pick & place and pick & drag quasi-static baselines on rectangular cloths. Our approach is flexible to large cloths whose dimensions exceed the robot arm's reach ranges and generalizes to T-shirts despite being trained on rectangular cloths. We fine-tune our approach in the real world, where, average, it increased the cloth coverage over 4 times more than the quasi-static pick & place baseline did. In summary:
Our main contribution is in demonstrating the effectiveness of dynamic manipulation for cloth unfolding through our self-supervised learning framework, FlingBot.
We propose parameterization for the dual-arm grasp of our fling primitive, which enables the application of a simple yet effective single-arm grasping technique [5, 6, 7] to dual-arm grasp- ing while satisfying dual-arm safety constraints. The simplicity of FlingBot combined with its superior performance over quasi-static baselines further emphasize the effectiveness of dynamic actions for deformable object manipulation.
We present custom simulator! built on top of PyFlex [8, 9], CUDA accelerated simulator, which supports the loading of arbitrarily shaped cloth meshes. We hope this open-source simulator expands cloth manipulation research to more complex cloth types.
2 Related Work
A convincing argument for the addition of dynamic actions to exclusively quasi-static cloth manipu- lation pipelines would need to demonstrate their superior performance on core cloth manipulation skill. As a typical first step for many cloth manipulation tasks, cloth unfolding is a popular and important problem setting for studying deformable object manipulation. The goal of cloth unfolding is to maximize the coverage of the cloth on the workspace, which exposes key visual features of the cloth for downstream applications. However, achieving fully unfolded cloth configuration from a crumpled initial configuration remains challenging problem. Additionally, doing efficiently for many different types of cloths, some larger than the system's 's reach range, is extremely challenging.
Quasi-static Cloth Manipulation with Expert Demonstrations. Prior works have explored us- ing heuristics and identifying key points such as wrinkles [10], corners [11, 12, 13], edges [14], or combinations of them [15], but fail generalize to severely self-occluded cloth configurations, to when required key points aren't 't visible, or non-square cloths. Recent reinforcement learning ap- proaches [3, 16] relied cloth unfolding expert demonstrations in quasi-static pick-and-drag action
Pleasse visit https://tingboo..s.ouubb. for experiment videos, code, simulation environment, and data.

---

Primitive
Parameterization
Action Sequence
Pick
Stretch
Fling
Place
a) Fling
Pick
Lift
Move
Place
Pick Place
Figure 2: Action Primitives. The dynamic Fling primitive starts with two-arm grasp the left L and right R grasp locations with center point C, followed by fixed stretch, fling, and place routine. A good fling could unfold highly crumpled cloth in little single step. In contrast, the quasi- static Pick & Place primitive, which grasps at the start location S, lifts, moves, then places at the end position specified by the arrow tip, requires many steps for such challenging cloth configurations.
spaces, While they sidestep the exploration problem, expert demonstrations can be sub-optimal and brittle if from hard-coded heuristics [3] or expensive if from humans [16].
Self-supervised Quasi-static Cloth Manipulation. Bypassing the dependency on an expert, self- supervised cloth manipulation has been demonstrated in unfolding with a factorized pick & place action space by Wu al. [4] and goal conditioned folding using spatial action maps [7, 6, by Lee et al. [2]. However, all these approaches operate entirely in quasi-static action spaces.
Dynamic Cloth Manipulation. In contrast to quasi-static manipulation, which are "operations that can be analyzed using kinematics, static, and quasi-static forces (such as frictional forces slid- ing contacts)" [1], dynamic manipulation involves operations whose analysis additionally requires "forces of acceleration". Intuitively, dynamic manipulation (e.g., tossing [6]) involve high-velocity actions which build up objects' momentum, such that the manipulated objects continue to move after the robot's end-effector stops. Such dynamic manipulation results in an effective increase in reach range and a reduction the number of actions to complete the task compared to exclusively quasi-static manipulation. However, prior works on dynamic cloth manipulation have either relied on cloth's vertex positions [17] (so only practical in simulation), motion capturing system with markers combined with human demonstrations [18], or custom hardware [19, 20].
In contrast, our algorithm achieves high performance from severely crumpled initial cloth configu- rations through self-supervised trial and error and learns directly from visual input without requiring expert demonstrations or ground truth state information.
3 Method
The goal of cloth unfolding to manipulate cloth from an arbitrarily crumpled initial state to flattened state. Coneretely, this amounts to maximizing the cloth's coverage on the workspace surface. Intuitively, dynamic actions have the potential to achieve high performance on cloth un- folding by appropriately making use of the cloth's mass in high- action to unfold the cloth (Sec. 3.1). From top-down RGB image of the workspace with the cloth, our policy decides the next fling action (Sec. 3.2) by picking the highest value action which satisfies the system's 's con- straints (Sec. 3.3). predicts the value each action with a value network (Sec. 3.4) which is trained in a self-supervised manner to take actions that maximally increases the cloth's 's coverage. The supervision signal is computed directly from the visual observation captured by the top-down camera. After training in simulation, we finetune and evaluate the model in the real world (Sec. 3.5).
3.1 Advantages of the Fling Action Primitive
Quasi-static actions, such as pick & place, rely on friction between the cloth and ground stretch the cloth out. The complexity of friction forces between the workspace and the (nonvisible, ground- facing) surfaces of the cloth means the cloth's final configuration may be difficult to predict from

---

Workspace
Scaled Images
Predicted Maps
Highest Value
Fling Action
spe
300
Valor Nerwork
2.0c
1.3c
L.CG
12 estutions
Reachability
Figure 3: Method Overview. From top-down RGB image a), our policy evaluates batch of different action rotations and scales by transforming the observation b) then predicting the corre- sponding batch of value maps c). The highest value action d), corresponding to the maximum value pixel which also satisfies the arms' reachability constraints e) is chosen. Finally, the chosen pixel's location and its observation's 's transformation is decoded into the fling action parameters (i.e., center point, distance, relative orientation between gripper).
only visual observations, especially for novel cloth types or significantly different friction forces (i.e.: simulation v.s. real). In addition, such systems can't manipulate points on the cloth to regions outside of their physical reach range, which limits their maximum cloth dimensions. Meanwhile, dynamic actions largely rely on cloths' mass combined with a high-velocity throw to do most of its work. Since a single dynamic motion primitive can effectively unfold many cloths, dynamic unfolding systems can Iearn a simpler policy which generalizes better to different cloth types. In addition, they can reach higher coverages in smaller numbers of interactions and throw corners of cloths larger than the system's reach range, effectively expanding their physical reach range.
We propose to use a dual-arm pick, stretch, and fling primitive. Here, a dual-arm system stretches the cloth between the arms, which unfolds it in one direction, and then flings the cloth, which makes use of the cloth's 's mass to unfold it in the other direction. The combination of stretching and flinging ensures that, given two appropriate grasp points, our motion primitive should be sufficient for single step unfolding when possible.
3.2 Fling Action Primitive Definition
To achieve an efficient, generalizable, and flexible cloth unfolding system, we argue that the system requires two arms operating in dynamic action space. To this end, we've designed pick, stretch, and fling motion primitive for dual-am system, where each arm is placed on either side of the cloth workspace, follows. First, the arms perform a top-down pinch-grasp on the cloth at locations L, E R$. Second, the arms lift the cloth to 0.30m and stretch the cloth taut in between them. Third, the arms fling the cloth forward 0.70m at 1.4m then pull backwards 0.50m at 1.4m which swings the cloth forwards. Finally, the arms place the cloth down to the workspace and release their grips. The entire motion primitive is visualized in (Fig. 2a).
Instead of learning all steps of our primitive, we can fix the stretching step to always stretch the cloth as much as possible without tearing the cloth. We also fix the fling speed and trajectory from the observation that the real world system could robustly unfold the cloth using a wide range of fling parameters (i.c.: fling height, fling speed) if given a good grasp (see supplementary materials). Thus, the problem of learning to pick, stretch, and fling reduces down to the problem of learning where to pick, which is parameterized with only two grasp locations, one for each arm.
3.3 Constraint Satisfying Fling Action Parameterization
dual-arm grasp is parameterized by two points L, R E R which denotes where the left and right arm should approach from a top-down grasp respectively, and requires 6 scalars. Without loss of generality for the purposes of grasping from top-down RGB-D (GB-Dine input, the third dimension could be specified by depth information. This reduces L and R to two points in R?, each representing pixels to grasp from the visual input and uses only 4 scalars in total.
However, to minimize collisions between two arms, we wish to impose a constraint that T. is always left of R, and vice versa. Additionally, the grasp width (i.e., the length of the line L R2) be greater than the physical limit of the system and smaller than the minimum safe distance limit between the two arms Directly using I. and R will entangle these two contraints (see supplementary materials),

---

making them difficult to satisfy. To make these constraints linear and independent, we propose a 4- scalar parameterization, which consists of the pixel C' E RR2 at the center the line -- R, an angle 0 E R denoting the planar rotation of the line L - R. and a grasp width w E IR denoting the length of the line in pixel space. To constrain to on the left of R. we can constrain [-90", 90%]. while w can be directly constrained to appropriate system limits.
3.4 Learning Delta-Coverage Maximizing Fling Actions
The naive approach for learning the action parameters (Cas Cg., 0, w) by directly predicting these 4 scalars does not accommodate the best grasp's equivariances to the cloth's translation, rotation, and scale. However, the learner should be equipped with the inductive bias that cloths in identical configurations have identical optimal grasp points for flinging relative to the cloth.
To this end, we propose to use spatial action maps [5, 6, 7]. By predicting grasp values with constant scale and rotation in pixel space from the transformed images, spatial action maps can recover grasp values with varying scales and rotations world space by varying the transformation applied to the image. Concretely, given visual observation from the top-down view (Fig. 3a), we generate batch of rotated and scaled observations (Fig. 3b) then predict the corresponding batch of dense value maps (Fig. 3c). Each pixel in each value map contains the value of the action parameterized by that pixel's 's location, giving C, and its observation's 's rotation and scaling, giving @ and w respectively (Fig. 3f). The value network is supervised to regress each pixel in the value map to the ground truth delta-coverage - the difference coverage before and after the action. Thus, by picking the grasping action with the highest value (Fig. 3d), the system picks grasp points which expects would lead to the greatest increase in cloth coverage. By its architecture, the value network is equivariant to translation. By rotating and scaling its inputs, it is also equivariant to rotation and scale.
We use 12 rotations in [-909, (--00,90%, 90"]. 8 scale factors in [1.00, 2.75], both at even intervals, and 64 x 64 RGB images. To make the use the scale range, we crop the image such that the cloth takes up two-thirds of the image width and height before applying one of the 8 scale factors. We also filter out and reject all grasp point pairs which are out of reach for either arm.
Self-Supervised Learning. The value network is trained end-to-end with self-supervised trials in simulation then finetuned in the real world. Each fling step labeled with its normalized delta coverage, which is computed by counting cloth mask pixels from a top-down view, then dividing by the cloth mask pixel count of flattened cloth state. While our experiments have plain black workspace surface colors and non-textured colored cloths for casy masking, our approach can be combined with image segmentation or background subtraction approach to obtain the corresponding cloth mask to work with arbitrary cloth and workspace textures. Note that this supervision signal (i.e., cloth mask) is only needed during training.
The policy interacts with the cloth until it reaches 10 timesteps or predicts grasps on the workspace. The latter stopping condition indicates that the policy does not expect any possible action further improve the cloth coverage. All policies are trained in simulation until convergence, which takes around 150,000 simulation steps, or 6 days on a machine with GTX 1080 Ti. The network is trained using the Adam optimizer with a learning rate of le-3 and weight decay of 1e-6.
3.5 Experiment Setup
Simulation Setup. Our custom simulator² is built on top of the PyFleX [8] bindings to Nvidia FleX provided by SoftGym [9], and can load arbitrary cloth meshes, such as T-shirts, through Python API. The observations are rendered using Blender 3D where the cloth HSV color is sampled uni- formly between [0.0, 1.0), [0.0, 1.0). and [0.5, 1.0) respectively, and the background is dark grey with procedurally generated normal map to mimic the wrinkles on the real-world workspace surface. We further apply brightness, contrast, and hue jittering on observations to help with the transfer
"Our custom simulatorss is publicty accessible at http://iiib...ooooommbbaeooooies
5

---

Init
Step 1
Step 2
Step
Init
Step
Step
Step
Normal Reet (Real)
0.964
Fling
Done
Done
0.348
0.563
0.674
0.674
0.291
0.322
0.473
0.527
P P
0.305
0.412
0.447
0,848
0.276
0.931
Large Rect (Real)
Fling
Done
0.342
0.432
0.6.88
0.676
0.335
0.642
0.701
7.789
P &
0.283
0.893
0.192
0:4.9
0.959
Fling
Shirt (Real)
0.433
P& P
Figure 4: Qualitative Results in Real World. Cloth coverages are labeled the top right corner. Red and green circles represent grasps by left and right arms, placed above and below field view, respectively. FlingBot discovered through trial-and-error two-arm corner and edge grasp when corners and edges are visible. While the pick place baseline discovered a similar strategy, requires significantly more steps achieve coverage lower than FlingBot's.
to real. We manually tuned our simulation fling speed to qualitatively match the real-worke cloth dynamics (see supplementary materials on real world robustness fling parameters).
Real World Setup. Our real-world experiment setup consists of two UR5s, where one is equipped with Schunk WSG50 and the other with an OnRobot RG2, facing each other and positioned 1.35m apart. The top-down RGB-D image is captured with an Intel RealSense D415. To help with pinch grasp success in real, we used 2-inch rubber mat as the workspace and rubber gripper finger covers. To mitigate the reliance on specialized force sensors, we chose to implement stretched cloth detection using only second Intel RealSense D415 capturing frontal view of the workspace. By segmenting the cloth from the front camera RGBD view of the workspace, then checking whether the top edge of the cloth mask is flat as proxy for the cloth being stretched, we can pull the arms' end effectors apart until the top of the cloth is no longer bent.
4 Evaluation
We design a series experiments to evaluate the advantage of dynamic actions over quasi-static actions in the task of cloth unfolding. The system is evaluated on its efficiency (i.e., reaching high coverages with a small number of actions), generalization unseen cloth types (i.e., flattening different types of T-shirts when only trained rectangle eloths), and reach range (i.e., on cloths with dimensions larger than its reach range). Finally, we evaluate the algorithm's 's performance the real-world setup. Please visit http://finngoo..s.oooumbaaaaaaaa
for experiment videos.
The performance is measured by the final coverage, delta-coverage from initial coverage, and the number of interactions. All our coverage statistics are normalized by the coverage of the cloth in flattened configuration and can be computed from top-down camera. To evaluate our policy, load task from the unseen test datasets then run the policy for 10 steps until the policy predicts grasps on the floor.
Metrics

---

Sim Rectangular Cloths
Large Rectangular Cloths
Shirts
1.0
1.0
1.0
Covenge tha
Concagg (Na
Covcage
0.6
9.66
844
0.4
0.4
10
4
10
0
2
10
Epieade Siep
Fpieode Stap
Epiende Step
Real Normal Rectangular Cloths
Real Large Rectangular Cloths
Real Shirts
1.0-
1.0
10-
Covetugs (50
088
Coverage (S)
0.8
Oovetage (S0
9.4
0.2
0.2
Upisode Step
Lppisodd Stop
Eppisode Step
FlingBot
Bot-
Stretch & Drag
Pick Drag
Pick & Place
Figure 5: Coverage v.s. Steps. With 95% confidence interval shaded. FlingBot can achieve high coverage within few interaction steps, while the quasi-static baselines never reach high coverages even with significantly more interaction steps. This demonstrates the difficulty of unfolding from highly crumpled initial configurations and dynamic action's 's superior efficiency.
4.1 Task Dataset Generation
Each task is specified by a cloth mesh mass, stiffness, and initial configuration. The cloth mesh is sampled from one of three types: (1) Normal Rect, which contains rectangular cloths with size within the reach range. Edge lengths are sampled from [0.40m, 0.65m], (2) Large Rect, which contains rectangular cloths with at least one edge larger than the reach range (0.70m) and (3) Shirt, which contains subset of shirts sampled from CLOTH3D's s [21] test split, all of which are resized be within the reach range. The cloth mass is sampled from [0.2kg, 2.0kg] and an internal stiffness Finally, the cloth's initial configuration is varied by holding ran- domly grasped the cloth at random height between [0.5m, 1.5m] then dropping and allowing the cloth to settle (similar to I Lee et al. and tier-3 in Seita et al.). resulting in severely crumpled config- uration. Please refer to supplementary materials for more details on our task generation process.
from [0.85kg/s², 0.95kg/s*
In simulation, the policy is trained on 2000 rectangular cloths sampled evenly between Normal Rect and Large Rect, and evaluated on 600 novel tasks split evenly between Normal Rect, Large Rect, and Shirt cloths. In real, the simulation policy is deployed to collect real world experience 150 Normal Rect episodes (257 steps), optimized on both simulation and real world data, then evaluated on 10 novel tasks in each cloth type.
4.2 Approach Comparisons
We compare against 3 quasi-static baselines, [Pick & Place] (similar to Lee et al. [2], visualized in Fig. 2b), [Pick & Drag] (similar to Seita et al. [22]), and [Stretch & Drag] (identical to [Pick & Drag] with an extra stretch step). Our policy, [FlingBot], predicts the optimal two-arm grasp locations for the flinging primitive (Sec. 3.2). In addition, we compare against [Fling-Reg], which does not exploit the task's equivariances (Sec 3.4) and thus completely fails to perform the task (Tab. 1). Further, we trained [FlingBot-S], which is identical [FlingBot] but also learns the fling speed. However, Tab. 1 and Fig. 5 show there aren't significant performance gains for [FlingBot-S] over [FlingBot], so we prefer the simpler [FlingBot] for comparisons with baselines. We provide more information about baselines in the supplementary material.
4.3 Results
Better Efficiency, In this experiment, we compare the unfolding efficiency between quasi-static and dynamic actions. From the Normal Rect column in Tab. 1, [FlingBot] increases the coverage of the cloth (+63.1%) by more than two times that of [Pick&Place] and [Pick&Drag] baselines (+29.2%, and 24..%%). Additionally, from Fig. 5, [FlingBot] achieves 80% within 3 interactions (sim- ulation normal cloth), while the quasi-static baselines never reach such high coverage even with
7

---

significantly more interaction steps or with a stretching subroutine. This demonstrates the difficulty of unfolding from highly crumpled initial configurations and dynamic action's superior efficiency.
Increased Reach Range. In this experiment, we investigate these approaches' performance on Large Rect cloths. These cloths are not only challenging for quasi-static baselines, because the arms can't manipulate the cloth at locations beyond its reach range, but also for our flinging policies, because the arms can't fully stretch or lift the cloth off the ground. Despite these challenges, [FlingBot] achieves 79. 9.2% (Tab. I, column Large Rect). Com- pared to the quasi-static baselines, [FlingBot] in- creases the coverage by +52.0%, which is roughly twice that of the quasi-static baselines ( (+27.1% +24.8%, +2).1%) These results show how high- velocity actions could effectively expand the phys- ical reach range, allowing the system to be more flexible with extreme cloth sizes.
Normal Rect Large Rect
Shirt
Pick&Place Pick&Drag Stretch&Drag Fling-Reg
53.0 24,2
52.0/ 24.8 79.8 33.4
58.0/
54.2/ 27.1 72.4 26.0
77.2/
50.2/23.1 90.3 43,9
29.1
/0.5
7.6
FlingBot-S FlingBot
92.6/ 63.8 78.9 51.7 93.5 47.1
91.9/
79.2 52.0 93.3 46.8
Table I: Simulation Experiments (Final / Delta Coverage).
Normal Rect Large Rect
Shirt
Pick&Place FlingBot
43.2 / 13.0
38.4/ 11.0 42.7 19.9 88.5 54.9 89.2 65.2
81.9 55.8
Table 2: Real World Experiment (Final / Delta Coverage).
Generalize to Unseen Cloth Types. This experiment investigate how well these approaches, trained on only rectangular cloths, can generalize to unseen cloth types (i.e.: T-shirts). Qualitative real world (Fig. 4) and simulation (in supp.) results suggest that our flinging policy has learned to grasp key- points on cloth (i.e., corners, edges) when it sees them, or otherwise fling to reveal these features. FlingBot's 's generalization performance (93.3% in Tab. 1, column Shirt) to novel cloth geometries can be attributed to this strategy, since a cloth of any type can be unfolded by grabbing one of its edges, stretching, then flinging it in a direction perpendicular to the edge. Through self-supervised exploration, our approach discovered grasping strategies which were manually designed in heuris- tic based prior works, while being more generalizable to different cloth configurations and types. Meanwhile, all quasi-static baselines exhibited worse cloth unfolding efficiency. The Sim Shirts plot in Fig, 5 shows that our flinging policies take only 3 actions to reach their maximum coverages, while the quasi-static baselines take upwards of 8 steps to reach lower maximum coverages.
Evaluating Real-Worle Unfolding. Finally, we finetune and evaluate our simulation models from Tab. I with wolld experience on pair of URS arms. Task generation is automated using the robot arms by randomly grasping the cloth at height 0.50m then dropping it back on the workspace. We use a 0.35m x 0.45m cloth for Normal Rect, 0.40m x 0.70m bath towel for Large Rect, and 0.45m x 0.55m T-shirt for Shirt. The system collected 257 experience steps over 150 cloth tasks for finetuning in total. The performance is reported averaged over 10 test episodes, where real-world grasp errors are filtered out (see supplementary materials). From Tab. 2, we report that our policy achieves over 80% all cloth types, which outperforms the quasi-static pick & place baseline by over 40%. Additionally, we report that the pre-finetune performance of FlingBot on Normal Cloths is 69.8%, justifying our decision finetune get 12.1% improvement. Overall, our flinging primitive takes a median time of 16.7s. While the pick & place primitive only takes a median time 8.88, it is unable to reach the high coverages even with many more interaction steps (Fig. 5, bottom row). Both primitives incur additional overheads of 2.5s for preparing the transformed observation batch and 2.9s for a routine which checks whether the cloth stuck to the gripper after the gripper opened (details in supplementary materials).
Conclusion and Future Work
We proposed fling primitive and a self-supervised learning algorithm for learning the grasp param- eters for the cloth unfolding task, resuling in a policy which is efficient, generalizable, and works with cloth sizes beyond the reach range of the system in both simulation and the real world. Fu- ture work could explore dynamic actions for more complex (i.c., goal-condition) cloth manipulation tasks by combining with quasi-static action spaces and leveraging cloth pose estimation [23].
8

---

Acknowledgments
The authors would like to thank Eric Cousineau, Benjamin Burchfiel Naveen Kuppuswamy, other researchers in Toyota Research Institute, Zhenjia Xu, and Cheng Chi for their helpful feedback and fruitful discussions. We would also like to thank Google for their donation of URS robot hard- ware. This work was supported in part by the Amazon Research Award and the National Science Foundation under CMMI-2037101.
References
[1] M. Т. Mason and K. M. Lynch. Dynamic manipulation. In Proceedings of 1993 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS '93), volume 1, pages 152- 159 vol.1, 1993. doi:10. Noi:10.1109/IRO (009/ROS..999.58.....
[2] R. Lee, D. Ward, A. Cosgun, V. Dasagi, P. Corke, and J. Leitner. Learning arbitrary-goal fabric folding with one hour real robot experience, 2020.
[3] D. Seita, A. Ganapathi, R. Hoque, M. Hwang, E. Cen, A. K. Tanwani, A. Balakrishna, B. Thananjeyan, J. Ichnowski, N. Jamali, K. Yamane, S. Iba, J. F. Canny, and K. Goldberg. Deep imitation learning of sequential fabric smoothing policies. CoRR, abs/ Abss/191.00454, 2019. URL http:/ttt/aaaa //axii/ wwtp/iiii.oroaeoooooeo. http/bb//199000 04854.
[4] Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel. Learning to manipulate deformable objects without demonstrations. CoRR, abs/1910. 13439, 2019. URL http://arxiv.orz abs/1910. 13439.
[5] A. Zeng, S. Song, S. Welker, J. Lee, A. Rodriguez, and T. Funkhouser. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning. 2018.
[6] A. Zeng, Song, J. Lee, Rodriguez, and Funkhouser. Tossinghot: Learning to throw arbitrary objects with residual physics. 2019.
[7] J. Wu, X. Sun, A. Zeng, S. Song. J. Lee, S. Rusinkiewicz, and T. Funkhouser. Spatial action maps for mobile manipulation. Robotics: Science and Systems XVI, Jul 2020. doi: 10.1560777 rss. 20s.2020.vvi..m
URL http: //xx doi. org.o...........ooooo..o....0... 15607 /RSS. 2020 XVI.
[8] Y. Li, J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. In ICLR, 2019.
[9] X. Lin, Y. Wang, J. Olkin, and D. Held. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation, 2020.
[10] L. Sun, G. approach for flattening wrinkled clothes. In TAROS, 2013.
W. Cockshott, S. Rogers, and Siebert. heuristic-based
[11] B. Willimon, S. Birchfield, and I. Walker. Model for unfolding laundry using interactive per- ception. In 20// JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 4871-4876, 2011. doi: 10. 109//OO, 2011.609506.
[12] D. Seita, N. Jamali, M. Laskey, R. Berenstein, A. K. Tanwani, P. Baskaran, S. Iba, J. F. Canny, and K. Goldberg. Robot bed-making: Deep transfer learning using depth sensing of deformable fabric. CoRR, abs/1 809.09810, 2018. URL http: / arxxiv org/ 11099 .9810..
[13] J. Maitin-Shepard, M. Cusumano- Towner, J. Lei, and P. Abbeel. Cloth grasp point detection based on multiple-view geometric cues with application robotic towel folding. In 2010 IEEE International Conference on Robotics and Automation, pages 2308-20 2315. IEEE, 2010.
[14] D. Triantafyllou, I. Mariolis, A. Kargakos, S. Malassiotis, and N. Aspragathos. geometric approach to robotic unfolding of garments. Robotics and Autonomous Systems, 75:233-24 2233-,24,, 2016.

---

[15] H. Yuba, S. Arool, and K. Yamazaki. Unfolding of a rectangular cloth from unarranged starting shapes by dual-armed robot with a mechanism for managing recognition error and uncertainty. Advanced Robotics, 31(10):544 556, 2017.
[16] A. Ganapathi, P. Sundaresan, B. Thananjeyan, A. Balakrishna, D. Seita, J. Grannen, M. Hwang, R. Hoque, J. E. Gonzalez, N. Jamali, K. Yamane, S. Iba, and K. Goldberg. Learning dense visual correspondences in simulation to smooth and fold real fabrics, 2020.
[17] R. Jangir, G. Alenya, and C. Torras. Dynamic cloth manipulation with deep reinforcement learning. arXiv preprint arXiv:1910.14475, 2019.
[18] B. Balaguer and S. Carpin. Combining imitation and reinforcement learning to fold deformable planar objects. In 201 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1405-1412. IEEE, 2011.
[19] Y. Yamakawa, A. Namiki, and M. Ishikawa. Dynamic manipulation of a cloth by high-speed robot system using high-speed visual feedback. IFAC Proceedings Volumes, 44(1):8076-808), 2011.
[20] M. Shibata, T. Ohta, and S. Hirai. Robotic unfolding of hemmed fabric using pinching slip motion. In The Abstracts of the international conference on advanced mechatronics: toward evolutionary fusion of IT and mechatronics: ICAM 2010.5, pages 392-397. The Japan Society of Mechanical Engineers, 2010.
[21] H. Bertiche, M. Madadi, and S. Escalera. Cloth3d: Clothed 3d humans. In European Confer- ence on Computer Vision, pages 344-359. Springer, 2020.
[22] D. Seita, P. Florence, J. Tompson, E. Coumans, V. Sindhwani, K. Goldberg, and A. Zeng. Learning Rearrange Deformable Cables, Fabrics, and Bags with porter Networks. In ICAR, 2021.
Trans-
[23] C. Chi and Song. Garmentnets: Category-level pose estimation for garments via canonical space shape completion. In The IEEE International Conference Computer Vision (ICCV), 2021.
10


Reviews:
[{"review_id": "u1VzH75Oine", "reviewer": "Reviewer_X3S4", "summary": "The paper proposes FlingBot, an approach to cloth unfolding using dynamic flinging motions. The key idea of the approach is to grasp a crumpled object with two hands, lift the object and fling it such that it can be placed on the support surface in an unfolded state. This process is repeated several times until the object cannot be unfolded further.\n\nThe paper decomposes the problem as follows: (i) a hand-crafted dynamic motion primitive for flinging motions. (ii) a task-specific representation of a bimanual grasp point for flinging. (iii) A learned value network for predicting effective bimanual grasp points for flinging. (iv) A self-supervised learning curriculum for learning the grasp value network in simulation and fine-tune it in real.\n\nThe paper shows that the learned dynamic flinging skill results in a more successful, efficient and generalizable unfolding capability then previous pick & place policies for the same task.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe paper is very enjoyable to read, and makes an important contribution to the problem of deformable object manipulation in general, and cloth unfolding in particular. It strikes an excellent balance between leveraging general techniques such as reinforcement learning and sim2real and combining them with task-specific motor primitives and representations to solve the task at hand efficiently.\n\nThe proposed approach is rigorously evaluated, both in sim and in real, and sensible ablations and baselines are considered.\n\nThe figures are beautifully crafted and enable the reader to immediately grasp the key ideas of the approach (Fig 1-3) and how it performs (Fig 5).\n\n\n\n## Weaknesses\n\nThe paper writing, in particular Sections 1-3 could be more to the point. Some key concepts such as \"delta coverage\" are used throughout the text but only defined in Section 4.1. Furthermore, the paper states that the real world pipeline suffers from failure cases, but it is not quantified how often these occur, e.g. how many episodes had to be discarded because of a grasping failure. That raises concerns about the practical applicability of the method, and the paper should add a section on how these issues can be mitigated practically.\n\nFurthermore, it would be interesting to add visual ablation studies to understand when the value network breaks down.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper makes an important contribution to CoRL, both for the concrete task of cloth unfolding and for deformable object manipulation in general. The paper offers a good mix of careful task-specific engineering that enables solving the cloth unfolding task effectively, and a set of general techniques such as self-supervised learning in simulation. To me, this paper constitutes the most impressive account of both simulated and real-world cloth manipulation to date.\n", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "### Writing\n\n- \"Delta coverage\" is used throughout the text but only defined in Section 4.1. Define it explicitly when introduced for the first time (see next bullet point)\n- L52-L56: This explanation is a bit hard to understand when read for the first time. What is a \"value map of the same dimension\"? What does it mean for a flinging action to be parametrized by a pixel? What is delta coverage? I would recommend to rewrite this paragraph by explaining the approach's idea in simpler terms, and explicitly define all technical terms.\n- L75-77 I find that sentence hard to parse. What is the minimally convincing argument? Do the authors mean that demonstrating superior performance on the cloth unfolding task would be a convincing argument? Please reformulate\n- L80 Nitpick: The meaning of this sentence is devoid, as the simplicity of an objective generally bears no relationship to the simplicity of the problem.\n- L104 State explicitly in which way the task considered here is more challenging (-> more crumpled initial state)\n- L123 I don't follow why the network must learn friction effects, please elaborate\n- L169-172 I found this section a bit difficult to follow. I would suggest to make explicit references to Fig 3c-e and guide the reader\n- L184 It would be helpful to repeat here that training occurs in sim (I was surprised by \"task is sampled\")\n\n\n### Real world failure cases\n\n- State explicitly how many episodes had to be discarded because of grasping failures\n- Elaborate on how the real world grasp failures could be mitigated in future work\n\n### Technical questions\n\n- Why is the observation so aggressively scaled down to 64x64? I'm wondering how much \"crumpling\" remains visible in such a downscaled image\n- L208 How do you determine \"qualitative match\"? What is the reason that the simulator behaves so differently/requires half the fling speed. What would happen if the fling speed would not match? Experimental results on the robustness to fling speed in sim would be interesting (if you have some)\n\n### Value network ablation studies\n\nIt would be interesting to understand what the value network learns, and in particular how it generalizes. Is it actually paying attention to the item shape or the crumples? Or a mix? If it would only look at shape, it would probably never stop on the T-Shirt task as it has never seen irregularly shapes items during training. Are there irregularly shaped items that would break the network?\n\nI would propose (maybe for an extended journal version of this paper) to systematically feed different item shapes and crumpling states to the network and study the responses of the value network, e.g. whether there are cases where it would infinitely manipulate unfolded cloth, or where it does not notice crumples. \n"}}, {"review_id": "p6hE7-ZyWez", "reviewer": "Reviewer_BAzn", "summary": "This paper proposes to use dynamic flinging actions with a dual-arm robot for the task of cloth unfolding instead of quasi-static actions. It uses a self-supervised framework based on spatial action maps with the novel parameterization of a fling action primitive. The system takes an image of the scene as input and outputs a value map that provides the parameters for the primitive. The experiments demonstrate that this primitive has better performance than pick and place actions and can be transferred to a real robot; they also demonstrate generalization to different types of cloth.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The proposed parameterization for the dual-arm picking actions is novel and very reasonable.\n- The use of the dynamic action is well-motivated in the introduction.\n- The method is evaluated on real robot experiments which demonstrate clear improvement over the baselines.\n- The self-supervision pipeline is well-constructed to be able to generalize to unseen articles of clothing and does not rely on specific keypoints on the clothes.\n- The supplementary video is very helpful.\n\nLimitations:\n- The scope/impact of the main novelty (fling motion) is a bit limited to the specific task of unfolding.\n- The paper shows the benefit of the fling action primitive which consists of the stretch motion and a fling motion. How much benefit comes from flinging vs stretching? For example, what if the primitive were pick-stretch-drag instead of pick-stretch-fling? Additional ablations on this primitive would strengthen the argument of “the effectiveness of dynamic actions”.\n- To better demonstrate the generalizability as well as the failure cases of the method, it would be great to test on more types of shirts or other types of clothes in the real world experiment.\n- There are a number of details that are unclear - see below.\n\nOther comments and questions:\n- The paper suggests that fling speed prediction does not provide significant performance gain. There might be several reasons for that: (1) The training method could be improved. What if a self-supervised method is used instead of DDPG? For example, the fling speed can be discretized into N bins and the predicted value map can have N channels. (2) The benefit of the fling speed might be more obvious when tested over a wider variety of clothes. A piece of silk might require a very different fling speed compared to a heavy towel. (3) Maybe the fling speed should be correlated with the mass of the cloth which might be difficult to learn from the image.\n- How many different cloths are used for the real-world training? How many real-world actions are performed for this data collection? \n- Some other figures from the supplementary video can be added to the supplementary text to allow the reader to better understand the method, e.g. cross-over constraint, grasping parameterization; many key parts of the paper were only understood after I saw the supplementary video.\n \nMinor points:\n- Fig. 3 is not mentioned in the main text. It could probably be mentioned in Section 3.4 or at the beginning of Section 3.\n- Line 213-215 “ The stretching subroutine in our primitive is implemented by pulling the arms’ end effectors apart until the top of the grasped cloth is no longer bent” - how is this computed and implemented?  Details should be added to the appendix.\n- Line 264: It would be great to have a brief description of the difference between “pick and place” and “pick and drag”.\n- Lines 122-123: “Quasi-static actions, such as pick & place, rely on strategically chosen pick and place points and friction between the cloth and ground to stretch the cloth out. This means these systems must learn complex grasp point selection policies capable of reasoning about the effects of friction on cloth unfolding.” This is not obvious to me - why is friction reasoning important for policies that use pick and place primitives?  Is there a reference to support this claim?  It doesn’t seem like past work that uses pick and place primitives explicitly reasons about friction.\n- Line 151 says that L,R are in R^3 and then line 153-154 says that they are in R^2; this is somewhat confusing. What happened to the extra dimension? Was some assumption made here that isn’t explicitly stated?\n- Line 162: “ To constrain L to be on the left of R, we can constrain θ ∈[−90◦,90◦]” - a visualization (such as the one in the supplementary video) would be helpful here; this can be added to the supplementary text.\n- Line 180-181: “we crop the image such that the cloth takes up at most two-thirds of the image width and height before further scaling it by the 8 scale factors above.” The scale factors are [1.00, 2.75]; this is confusing: if the image is first scaled to take up 2/3 of an image and is later scaled by 2.75, it will then take up 1.83 of the image, e.g. larger than the image size.\n- Figure 5 visualization - what do the green and red points each indicate? \n- For the top left image in Figure 5, it looks like the red point is not on the cloth.  Shouldn’t this episode get discarded because only one pick point is on the cloth? (Line 337: “When there is a grasp failure in an episode, we discard the episode.”)\n- Line 191: “The policy interacts with the cloth until it reaches 10 timesteps or predicts grasps on the floor.” - don’t you mean on the table?\n- It is unclear how Flex and Blender are linked; do you import the cloth configuration from Flex into blender for each step in training?\n- “Both primitives incur additional overheads of 2.5s for preparing the transformed observation batch” - this seems very slow for just resizing and rotating images.  Is something else going on here?\n- “2.9s for a routine which checks whether the cloth is stuck to the gripper after the gripper is opened.” - how does this work?  Details can be added to the appendix.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper investigated the use of dynamic flinging motion for the cloth unfolding task based on a self-supervised pipeline with spatial action map. The experiments demonstrated the effectiveness of the proposed method and its applicability to the real robot. However, the main contribution of dynamic action is limited to the specific task. Also, it would be great if more experiments can be provided to support the generalizability and the effectiveness of the dynamic action.\n\n# Post-rebuttal Update\nI appreciate the clarifications and the additional experiments, such as stretch & drag, which further demonstrates the effectiveness of the fling motion. I would like to keep my current rating. One minor suggestion is to add a section to the Appendix to describe Figure 3 (instead of just adding the figure without any references to it).", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "- Ablation of pick-stretch-drag instead of pick-stretch-fling\n- Test on more types of shirts or other types of clothes in the real world experiment\n- Clarify missing details of the method and experiments\n"}}, {"review_id": "g3vwCk2QJVy", "reviewer": "Reviewer_LQc9", "summary": "This work demonstrates the impressive effectiveness of dynamic fling motion for cloth unfolding.  The parametrization combined with spatial action maps makes it easy and safe to learn dual-arm grasps. The system is trained in a self-supervised fashion, using delta coverage of the cloth as supervision signal. Experiments in simulation and real environments show its efficiency over the pick&place strategy, increased reach range, and generalization to unseen cloth type.\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- The effectiveness of the dynamic fling motion for cloth unfolding is impressive, novel, and beneficial for downstream cloth manipulation.\n- The parametrization combined with spatial action maps shows an easy but effective way for learning dual-arm grasps.\n- It is appreciated to evaluate the system in both simulated and real experiments thoroughly.\n- The paper is well written and enjoyable to read.\n\nWeaknesses\n- For sim2real, the authors mentioned mixing sim & real data. It will be very helpful to illustrate how important it is? Since 150 real data points are not much compared to the data in simulation, does it improve a lot or marginally?\n- For introducing the spatial action maps, on Page 4, line 169-172. The long sentence makes it difficult to interpret quickly, especially for scaling. Figure 3c&d makes it clearer. Rephrasing the paragraph or adding references & descriptions to the figure can improve the readability of the method.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Unfolding a piece of cloth can remove its self-occlusion, which can benefit downstream cloth manipulation tasks. The paper demonstrates the impressive effectiveness of dynamic fling motion for cloth unfolding. The learning diagram is easy but effective for dual-arm manipulation. The experiments are conducted in both simulated and real environments thoroughly.\n", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Issues as stated in the weakness section:\n\n- For sim2real, the authors mentioned mixing sim & real data. It will be very helpful to illustrate how important it is? Since 150 real data points are not much compared to the data in simulation, does it improve a lot or marginally?\n- For introducing the spatial action maps, on Page 4, line 169-172. The long sentence makes it difficult to interpret quickly, especially for scaling. Figure 3c&d makes it clearer. Rephrasing the paragraph or adding references & descriptions to the figure can improve the readability of the method.\n"}}, {"review_id": "fg3XqTU3NB-", "reviewer": "Reviewer_fiDj", "summary": "The paper proposes a human like, alternative strategy to unfolding cloth: using 2 robot arms to grab the cloth and then fling it (apply a high velocity oscillation). Flinging policies are trained in simulation. Three different categories of cloth are considered. A flinging action primitive is hand designed and a simple but effective action parameterization is used that can overcome some difficulties of dual arm setups. Top down images are captured and fed to a value network, which predicts spatial action maps. The best action parameters are chosen, and the primitive is executed. The methods are compared in simulation against pick & place and pick & drag baselines and show improved efficiency (fewer steps), performance on large cloths, and generalization to unseen cloths. The methods are also compared in real against a pick & place baseline and show improved performance. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper tackles an increasingly popular problem in deformable object manipulation. The method is unique, intuitive, and simple (no state estimation) and seems to perform significantly better than baselines in terms of efficiency, performance, and generalization on complex cloth initializations. Training is done primarily in simulation with fine tuning in the real world. Results are evaluated in both simulation and the real world. The paper is very well written and the video is extremely clear and well illustrated.\n\nAs the authors imply in the limitations section, the dynamic flinging strategy and associated findings may be less transferrable to folding cloth than existing pick & place baselines. There are also a few questions. What were the failure cases were in terms of generalization? Which cloths and t-shirts did it fail on and how/why? Please discuss this more. How does performance compare against baselines when the initial state is only lightly or moderately crumpled? Maybe an optimal strategy for unfolding is an initial flinging followed by pick & place. More questions are listed in Issues below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This is a really nice paper on an interesting problem. It provides an unfolding strategy that is clearly the right choice for highly crumpled initial states and demonstrates its effectiveness over benchmarks in simulation and real world. The visuals, video, and clarity also serve as an example for other research work in the field.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Fair: Some knowledge of the area"}, "extra_text": {"issues": "How were the parameters for the dynamic primitive selected? What was that process like? More insight there would be useful.\nIs lifting to 0.30 m sufficiently generalizable to long cloth?\nHow do you know when the cloth is taut during the stretching stage?\nWhy is learning a velocity unhelpful? When would you expect it to be helpful?\nI was confused by how the width and angle of the picking actions are represented or selected from the action value maps. Please clarify this.\nAll the videos are at 5x speed. Please provide videos at 1x as well.\nWere there any notable challenges in sim2real? Would be great to discuss this."}}]

Author rebuttal:

Meta-review: This paper addressed the problem of cloth unfolding with dynamic flinging actions of a bimanual system. At the initial reviews, all four expert reviewers have had a positive rating with three Strong Accepts and one Weak Accept. The reviewers appreciated the simple yet effective approach in this work to enabling new robot behaviors of deformable object manipulation. Some questions have been brought up by the reviewers, including clarifications of model details and discussions on failure cases. The authors provided satisfactory responses to these questions. At the end of the discussion period, the reviewers retained their original ratings. The AC agreed with the reviewers' positive assessments of this work, in particular, the empirical robot behaviors demonstrated in this work. For this reason, we recommended the acceptance of this paper for an oral presentation.
Decision: Accept (Oral)