FabricFlowNet: Bimanual Cloth Manipulation with a Flow-based Policy
Authors: Thomas Weng
Sujay Man Bajracharya
Yufei Wang
Khush Agrawal
David Held
Venue: CoRL2021 Poster (2021)
Keywords: deformable object manipulation
optical flow
bimanual manipulation
cloth manipulation

Abstract: We address the problem of goal-directed cloth manipulation, a challenging task due to the deformability of cloth. Our insight is that optical flow, a technique normally used for motion estimation in video, can also provide an effective representation for corresponding cloth poses across observation and goal images. We introduce FabricFlowNet (FFN), a cloth manipulation policy that leverages flow as both an input and as an action representation to improve performance. FabricFlowNet also elegantly switches between bimanual and single-arm actions based on the desired goal. We show that FabricFlowNet significantly outperforms state-of-the-art model-free and model-based cloth manipulation policies that take image input. We also present real-world experiments on a bimanual system, demonstrating effective sim-to-real transfer.  Finally, we show that our method generalizes when trained on a single square cloth to other cloth shapes, such as T-shirts and rectangular cloths. Video and other supplementary materials are available at: https://sites.google.com/view/fabricflownet.

FabricFlowNet: Bimanual Cloth Manipulation with Flow-based Policy
Thomas Weng, Sujay Bajracharya, Yufei Wang, Khush Agrawal, and David Held Robotics Institute, Carnegie Mellon University, USA (tweng, sbajrach, yufeiw2, khusha, dheld cmu. edu
Abstract: We address the problem of goal-directed cloth manipulation, chal- lenging task due to the deformability of cloth. Our insight is that optical flow, a technique normally used for motion estimation video, can also provide an ef- fective representation for corresponding cloth poses across observation and goal images. We introduce FabricFlowNet (FFN), a cloth manipulation policy that leverages flow as both an input and formance. FabricFlowNet also elegantly switches between dual-arm and single- actions based on the desired goal. We show that FabricFlowNet significantly outperforms state-of-the-an model-free and model- cloth manipulation poli- cies. We also present real-world experiments on bimanual system, demonstrat- ing effective sim-to-real transfer. Finally, we show that our method generalizes when trained single square cloth to other cloth shapes, such as T-shirts and rectangular cloths. Video and other supplementary materials are available at: http://itee..oolll.oooom/o/// wtwt//eeee.fa.rricfoo.eeet...
action representation to improve per-
Keywords: deformable object manipulation, optical flow, bimanual manipulation
Introduction
Cloth manipulation has wide range of applications domestic and industrial settings. However, it has posed challenge for robot manipulation: compared to rigid objects, fabrics have higher- dimensional configuration space, can be partially observable due to self-ocelusions in crumpled configurations, and do not transform rigidly when manipulated. Early approaches for cloth manipu- lation relied on scripted actions; these policies are typically slow and do not generalize to arbitrary cloth goal configurations [27, 13, 3].
Recently, learning-basee approaches have been explored for cloth manipulation [18, 33, 42, 29, 32], including model-free reinforcement learning to obtain policy [21, 39]. For cloth manipulation policy to be general to many different objectives, must receive a representation of the current folding objective. A standard approach for representing goal-conditioned policy is to input an image the current cloth configuration together with an image of the goal [21, 32].
We will show a number of downsides to such an approach when applied to cloth manipulation. First, the policy must learn to reason about the relationship between the current observation and the goal, while also reasoning about the action needed to obtain that goal. These are both difficult learning problems; requiring the network to reason about them jointly exacerbates the difficulty. Additionally, previous work has used reinforcement learning (RL) to try to learn such a policy [21, 39]; however, reward function a fairly weak supervisory signal, which makes it difficult to learn a complex cloth manipulation policy. Finally, while many desirable folding actions more easily and accurately manipulated with bimanual actions, previous learning-bass methods for goal-conditioned cloth manipulation have been restricted to single-arm policies.
In this paper, we introduce FabricFlowNet (FFN), goal-conditioned policy for bimanual cloth manipulation that uses optical flow to improve policy performance (see Fig. 1). Optical flow has typically been used for video-related tasks such as object tracking and estimating camera motion. We demonstrate that flow can also be used in the context of policy learning for cloth manipulation; we use an optical flow-type network estimate the relationship between the current observation and sub-goal. We use flow in two ways: first, as an input representation to our policy; second, after es- timating the pick points for a pick-and-place policy, we query the flow image to determine the place
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Observations
Flow
Observation
FabricFlowNel
Predicted Action
Goal
Collect data simulation
Train FabricFlowNet
Real world Folding
Achieved Fold
FabricFlowNet (FFN) overview. We collect dataset of random actions and ground truth flow to train FFN. FFN learns predict flow and uses both an input and action representation in manipulation policy. FFN successfully performs single and dual-arm folding in the real world.
Figure I:
actions. Our method is learned entirely with supervised learning, leveraging ground truth particles from simulation. Our method learns purely from random actions without any expert demonstrations during training and without reinforcement learning.
Our learned policy can perform bimanual manipulation and switches easily between dual and single- arm actions, depending what is most suitable for the desired goal. Our approach significantly outperforms our best efforts to extend recent single-arm cloth manipulation approaches bimanual manipulation tasks [18, 21]. We present experiments dual-arm robot system and in simulation evaluating our method's ' cloth manipulation performance. FabricFlowNet outperforms state-of-the- art model-based and model-free baselines, and provide extensive ablation experiments to demon- strate the importance of each component of our method to the achieved performance. Our method also generalizes with additional training to other cloth shapes and colors. This paper contributes:
A novel flow-based approach for learning goal-conditioned cloth manipulation policies that can perform dual-arm and single-arm actions
A test suite for benchmarking goal-conditions cloth folding algorithms encompassing and ex- panding on goals used in previous literature [18, 21, 15]; we perform extensive experiments using this test suite to evaluate FabricFlowNet (FFN), baselines [18, 21], and ablations, demonstrating that FFN outperforms previous approaches.
Experiments to demonstrate that FFN generalizes to other cloth colors and shapes, without training on such variations.
Related Work
Bimanual Manipulation. A large body of research exists on dual-arm, or bimanual, manipula- tion [35]. Dual-arm systems allow for more complex behaviors than single-arm systems at the cost of greater planning complexity [14, 34), leading to research on closed kinematic chain plan- ning [36, 4], composable skill learning [40, 7]. and rewarding synergistic behavior [8]. Prior work has also explored bimanual cloth manipulation [31], including establishing diverse set of bench- mark tasks [16]. Cloth manipulation is highly underactuated task, and bimanual manipulation enables controlling multiple cloth points [5]. common approach for cloth flattening to lift cloth with one arm and regrasp it with the other arm until it reaches the flattened configura-
tion [23, 10. 27, 13, 3]. Previous work in this direction uses hard-coded policies [27, 13, 3], whereas learn to achieve arbitrary folded configurations. Tanaka et al. [37] learn bimanual actions for goal-conditioned folding, using a voxel-based dynamics model to predict how actions will change the cloth state. However, optimizing this dynamics model can slow down inference time compared to our model-free approach. Dynamic bimanual manipulation has also been explored in simula- tion from ground-truth keypoints [20] and for unfolding cloth in the real world [17]; we perform real-world bimanual folding using depth image observations.
Learning for Cloth Manipulation. Prior works have proposed various hand-defined representa- tions for cloth manipulation, such as parameterized shape models [28] or binary occupancy fea- tures [22]. Recent approaches use contrastive learning to learn pixel-wise latent embeddings for

---

cloth [15, 6]. Both contrastive learning [15] and conl-condit transporter networks [32] have been applied imitate expert demonstrations. Our approach doesn't require expert actions, just sub-goal states provided at test-time to define the task. In contrast to these previous representations, our method uses a flow-based representation, which we found to significantly outperform previous methods for goal-based cloth manipulation.
Other approaches have applied policy learning techniques to single-arm cloth smoothing [39, 33J. In contrast, we learn a policy that performs either single and dual-arm cloth manipulation; further, our focus on goal-conditioned cloth folding, rather than smoothing. For cloth manipulation, Lee er al. [21] learns model-free value function, but is limited by its discrete action space, and further, they do not use a flow-based representation, which we show leads to large benefits. Prior methods for learning condiiondt policies have used self-supervised learning to learn an inverse dynamics model for rope [29, 30] but such approaches have not been demonstrated for cloth manipulation. Lippi et al. [25] plan cloth folding actions in latent space, but do not demonstrate generalization to unseen cloth shapes. Other papers use an online simulator [23], or learn a cloth dynamics model in latent space [42], pixel-space [18], or over graph of keypoints [26]. Unlike these model-based methods, our method is model-free and does not require online simulation or time-expensive CEM planning, leading to much faster inference. Further, we compare our approach to state-of-the-art approaches for cloth manipulation [21, 18] and show significantly improved performance.
Optical Flow for Policy Learning. Optical flow is the task of estimating per per-pixel correspondences between two images. typically for video-related tasks such as object tracking and motion estimation. State-of-the-arr approaches use convolutional neural networks (CNN) to estimate flow [12, 19, 38). Optical flow between successive observations has previously been used as an input representation to capture object motion for peg insertion [11] or dynamic tasks [1]. Within the domain of cloth manipulation, Yamazaki et al. [41] similarly use optical flow on successive observations to identify failed actions. We use flow not to represent motion between successive images, but to correspond the cloth pose between observation and goal images, and to determine the placing action for folding. Argus el al. [2] use flow in visual servoing task to compute residual transformations between images from a demonstration trajectory and observed images. In contrast, we learn a policy with flow to determine what cloth folding actions to take, not how to servo to desired pose.
3 Learning a Goal-Conditioned Policy for Bimanual Cloth Manipulation
3.1 Problem Definition
Our objective is to enable robot to perform cloth folding manipulation tasks. Let each task be defined by sequence of sub-goal observations G {aq. ......... , ..,% x ). each of which can be achieved by single (possibly bimanual) pick-and-place action from the previous sub-goal. We require sub- goals, rather than a single goal, because a folded cloth can be highly self-occluded such that a single goal observation fails to describe the full goal state. Defining a task using a sequence of sub-goals is found in other recent work [30]. Similar to prior work [30, 29], even if the sub-goals are obtained from an expert demonstration, nonetheless do not assume access to the expert actions; this is realistic assumption if the sub-goals are obtained from visual observations of a human demonstrator.
We assume that the agent does not have access to the sub-goal sequence G during training that must execute during inference. Thus, the agent must learn a general goal-conditioned policy a, T(x, 9), where is the current observation of the cloth and a, E A is the action selected by the policy. In our approach, we input each sub-goal 2 sequentially to our policy: a; = т (xe, 2²)). A goal recognizer [30] can also be used to decide which sub-goal observation to input at each timestep. For convenience, we will interchangeably refer to x! as goal or sub-goal.
3.2 Overview
A common approach for a goal-conditioned policy is to input the current observation o, and the goal observation x. directly into to a neural network representation of a policy [30, 21] or a Q- function [39, 21). However, the network must reason simultaneously about the relationship between the observation and the goal, as well as the correct action to achieve that goal. Our first insight is that we can improve performance by separating these two components: we will learn to reason about the relationship between the observation and the goal, and separately use this relationship to reason
3

---

Policy
Observation
Flow
PickNet
Points
Flow Image
Goal
(a) Naive system
(b) FabricFlowNet (FFN)
Figure 2: (a) A naive approach goal-conditioned policy learning is to input observation and goal images directly to the policy and predict the action. (b) FabricFlowNet separates representation learning from policy learning; it first estimates the correspondence between the observation and goal as a flow image. The flow is then used as the input to PickNet for pick point prediction, and as way to compute place points without requiring additional learning.
actions. Specifically, represent this relationship using "flow image" which indicates the correspondence between the current observation 2; and sub-goal xi. Thus we propose using the flow image f improved input representation of the policy, rather than directly inputting the observation and goal observation x!
Our second insight is that pick and place action space; prior methods that learn pick and place policies for deformable object manipulation predict place points using the policy network, either explicitly [33, 32, 42, 29, 30, 39] or implicitly by transforming the inputs to a Q-function [21]. Instead, we simplify the problem by leveraging flow: our policy network only learns predict the pick points. For the place point, we query the flow image f for the flow vector starting at the predicted pick location, and use the endpoint of that vector the place point.
also flow in the output representation of the policy. We use
We demonstrate that using flow the two ways described above for our policy achieves significantly improved performance compared to prior work. Furthermore, our approach extends naturally to dual-arm manipulation, allowing us to casily transition between single and dual-arm actions.
A schematic overview of our system can be found in Fig. 2b, We first compute the flow f between the current observation Fa and goal xς- Next, we imput the flow to a policy network (PickNet), which outputs pick points p;. We then query the flow image f(p;) to determine the place points for each robot arm. Further details our approach are described below.
3.3 Estimating Flow between Observation and Goal Images
We learn flow to use it as an input representation to our pick prediction network, and as an action representation for computing place points. Given observed depth image 2; and desired goal depth image uy, we estimate the flow f = (fz f2), mapping each pixel (u, v) in Ie to its corresponding coordinates (u', (') = (u + fl (u),
(v)) in a² This task formulation differs from standard optical flow tasks as the input image pairs (Iz, I are not consecutive images from video frames.
To capture the complex correspondences between Iy and x. we train convolutional neural network to estimate the flow image f (see Appendix for details). The training loss we use to supervise the network is endpoint error (EPE), the standard error for optical flow estimation. EPE is the Euclidean distance between the predicted flow vectors f and the ground truth f", averaged over all pixels:
LEPE N EN |lfe - fl We use cloth simulation to collect training examples with ground truth flow. The simulator provides the ground- --utt correspondence between the particles of the cloth in different poses. The simulation cloth particles are not as dense as the depth image pixels; as a result, we only have ground-truth flow supervision for a sparse subset of the pixels that align with the cloth particles. Thus, we mask the loss to only supervise the flow for the pixels that align with the location of the cloth particles. We train the flow network using data collected from random actions. See Sec. 3.6 for more details the simulator, data collection, and network training.

---

PickNet2
Flow Image
Heatmap H2
PickNet1
Heatmap H,
Gaussian on Pick Point
Ground Truth Pick Points
Goal
BCE Loss
Figure 3: PickNet architecture. We utilize two-network architecture for bimanual manipulation, where the second pick point conditioned on the prediction of the first pick point.
3.4 Learning to Predict Pick Points
Our bimanual action space A consists of actions P2, 92) where p and q are the pick and place points respectively, paired according to the subscripts. We train neural network called PickNet to estimate the pick points P1. P2- Crucially, the input to PickNet is flow image f. timated between the current depth image zz and the desired goal depth image x4, as described in the previous section. The flow image indicates, for each pixel (u, v) in the current observation, the location (u, v) that the pixel has moved to in the goal observation. Our flow network (Sec. 3.3 above) reasons about the observation-goal relationship, so that the policy network (PickNet) only needs to reason about the action, specifically the two pick points (p1. P2); computing the place points described in Sec. 3.5.
For dual-arm actions, the pick points must be estimated conditionally, as the location of pick point Pi the cloth influences the optimal location of pick point P2, and vice versa. To decouple this conditional estimation problem, we propose a two-network architecture, PickNetl and PickNet2, to estimate the pick points (see Fig. 3). This architecture used two networks for pick-conditioned placing; we instead use two networks to condition dual-arm picking. PickNet1 is fully convolutional network that receives flow image a single heatmap H₁ estimating the optimal pick points for arm 1. We compute the first pick point P1 arg max, H1 (p). The second network, PickNet2, predicts the second arm's pick point p2 conditioned on Pl: PickNet2 takes as input both the flow image f and an additional image with 2D Gaussian centered on Pl, and otherwise identical PickNetl. PickNet2 outputs heatmap H2. from which we compute the second pick point: = arg max, H2(p). The two-network architecture decouples the conditionally dependent pick point predictions and does not require us to resort to heuristics to extract two pick points from single heatmap. We refer to PickNetl and PickNet2 together as "PickNet."
inspired by Wu al. [39], which
input and outputs
To train PickNet, we collect a dataset of random actions (see Sec. 3.6 for details) and record the current observation Tt, the bimanual action a (P1, P2, Q1, (2). and the next observation 2++1- We also estimate the flow f from It to 24+1, as explained Sec. 3.3. We create ground truth pick heatmaps H1; for arm using the recorded random action a, by placing 2D Gaussian N(p,, o) on each ground truth pick location p;. We then supervise PickNet using the binary cross-entropy (BCE) loss between predicted heatmaps H1, H2 and ground truth heatmaps Hi Hr. However, might be unclear to the network which pick point should be output by PickNet1 and which should be output by PickNet2. We compute the loss for both possible correspondences and use the minimum:
Incr H, H,, H,, HJ) = BCE(H, HT) + BCB HJ, HJ)
Lric = min(/nen H1, H2, Hi, H3), (ncr(H2, H1, Hi Hb)
At inference time, PickNet outputs the pick points pr, P2, computed from the argmax of H1, H2 respectively, as described above.
3.5 Estimating the Place Points from Flow
After estimating the pick points P1.P2 from flow, the remaining step to predict a bimanual pick and place action a = (P1, P2, q .22 estimate the place points @ı, 82- straightforward approach would be to train the network to predict place points qı, 42, similar to the pick points Pt, P2 as
5

---

described above (see Fig. 2a). Instead, our approach uses the flow image to find the place points, so that the place points do not have to be learned separately.
Our approach makes the assumption that, to achieve a desired subgoal configuration, the point picked on the cloth should be moved to its corresponding position in the goal image (which is estimated by the flow). This is simplifying assumption, since it is possible that the picked point will shift slightly after it released by the gripper; our method does not take into account such small move- ments. Using this assumption, to compute the place points Q1, 92, we query the flow f at each pick point P1, P2 to estimate the delta between the pick point location in the observation image and the corresponding location of the pick points in the goal image. We use these predicted correspondences as the place points: qi fp,) + Pt. for each arm i.
Q1,92,
Action predictions estimated by our approach can produce nearly overlapping pick and place points, indicating that arm 1 and arm should perform identical actions. We observe this hehavior from PickNet when the goal is best achieved with a single-arm action, rather than a bimanual one. On a real robot, grippers are likely to collide if grasping points that are too close. Therefore, to switch between executing a single-arm or bimanual action, we compute the L2 pixel distance between pick points dpick = |ppi p2ll2 la and place points dpnnc Il 92ll2. We use single-arm action when either distance is smaller than a threshold a, which we set to 30 for all experiments.
3.6 Implementation Details
We use SoftGym [24], an environment for cloth manipulation built on the particle-based simulator Nvidia Flex, collect training datasets. The simulator models cloth as particles connected by springs. We use pickers that simulate a grasping action by binding to the nearest cloth particle within threshold to execute pick and place actions in SoftGym. We collect data by taking random actions, biased towards grasping corners of the cloth. We demonstrate that we are able to train our method in SoftGym and then transfer the policy the real world. Details on the data collection, as well as the network architecture and training details, can be found in Appendix Sec. A.l.
4 Experiments
4.1 Simulation Experiments
Experiment Setup. We evaluate FabricFlowNet (FFN) and compare to state-of-the-art baselines in the SoftGym [24] simulator; real-world evaluations are below in Sec. 4.2. Our experiments focus on folding tasks, and we assume that cloth smoothing method (e.g., [17, 33)) is used to flatten the cloth before folding is executed. Our eтror metric the average particle position error between the achieved and goal cloth configuration. We evaluate on two sets of goals: 40 one-step goals that can be achieved with a single fold action, and 6 multi-step goals that require multiple folding actions. The multi-step goals each consist of sequence of sub-goal images, with the next sub-goal presented after each action. This protocol follows from our problem formulation in Sec. 3.1, and is similar to the protocol in Nair et al. [29]. Our goals include test goals from Ganapathi et al. [15] and Lee et al. [21] that are achievable with one arm, as well as additional goals more suitable for two-arm actions (see Appendix Fig. S2 for the full set of goals).
We compare our method Fabric-VSF [18], which learns visual dynamics model and uses CEM to plan using the model. We only use Fabric- with RGB-D input, as depth-only input performs poorly for folding tasks [18]. FabricFlowNet only uses depth and does not rely on RGB, which enables our method to transfer easily to the real world without extensive domain randomization. We also compare to Lee et al. [21], model-free approach. We extend the the original single-arm method to a dual-arm variant and compare against both. For both our method and the baselines, we only allow each method to perform one pick-end-place action for each subgoal (e.g. one pick and place action for each single-step goals). Additional baseline details can be found in the Appendix.
4.1.1 Simulation Results
Table I contains our simulation results for all methods. We report average particle distance error (in mm) for one-step goals only, multi-step goals only, and over both one-step and multi-step goals. Our results show that FFN achieves the lowest error over all goals and has the fastest inference time.

---

Table 1: Mean Particle Distance Error (mm) and Inference Time (sec) on Cloth Folding Goals
Method
One Step (n=40)
Multi Step (n=6)
All (n=46)
Inference Time
Lee al., 1-Arm [21]
16.18 08.38
26.20 : 16.31
17.49 + 10.10
0.04
Lee el al., 2-Arm
36.62 + 14.51
47.71 + 21.95
38.07 + 15.82
~ 0.04
Fabric-VSF [18]
6.31 + 06.55
21.33 11.20
8.27 + 08.90
420
FabricFlowNet (Ours)
4.46 + 02.62
25.04 +: 22.88
7.14 + 11.06
~ 0.007
We also investigate whether using flow as a goal recognizer improves performance. When an ob- servation closely matches the goal, the flow for all points is close to zero. We leverage this fact by evaluating FFN with "iterative refinement": we allow the policy to take multiple actions per subgoal to try to further minimize the flow between the observation and subgoal. When the average flow between observation and current subgoal reaches a minimum threshold, the policy moves forward to the next subgoal. FFN with iterative refinement achieves mean error of 6.62 over all goals 7.14 without refinement. Additional details on iterative refinement can be found in the Appendix, along with additional results from baseline variants, crumpled initial configurations, and end-to-end training.
4.1.2 Ablations
We run series of ablations to evaluate the importance of the components of system; results averaged over all 46 goals are in Table 2. Additional details and results are in Appendix Sec. D. Our ablations are designed to answer the following questions:
What is the benefit of using flow as input? We modify PickNet to receive depth images of the observation and goal imput to the network ("NoFlowIn"), as is commonly done in previous work on goal-conditioned RL [21, 32]. In this ablation, the PickNet needs to reason about both the rela- tionship between the observation and the goal, as well the action. In contrast, our method uses the flow network to compare the observation and goal: the picknet separately reasons about the action. What is the benefit of using flow to choose the place point? In this ablation, we train network to predict the place points directly ("NoFlowPlacee") This is in contrast to our approach where we use the flow field, evaluated at the pick point f(:)), compute the place point qi for arm i. Our ap- proach leads to a 32.4% improvement, showing the benefit to using flow as an action representation. What is the performance with no flow? We combine the above two ablations and remove flow en- tirely, ("NoFlow""; ours has 60.4% improvement). The above ablations all indicate the strong benefit of using flow as both an input and action representation for cloth manipulation.
What is the benefit of biasing the data collection to grasp corners? Our method uses prior knowledge about cloth folding tasks to bias the training data and pick corners of the cloth. In this ablation, we choose pick points randomly ("NoCornerBias", ours has 35.5% better performance). What is the performance with a simpler architecture? We also compare our architecture for PickNet (Sec. 3.4) to simpler architecture that takes as input the flow image 1 and outputs a two heatmaps, one for each pick point ("NoSplinPickNet", ours has 2.1% better performance).
Does the loss formulation in Eq. 1 improve performance? We compare our method to an ablation where the first ground-truth heatmap is used to supervise PickNetl and similarly for the second, i.e. Lrick = InCE H1, H2, Hi; Hz). ("NoMinLoss""; ours has similar performance).
Table 2: Mean Particle Distance Error (mm) for Ablations over All Goals (n=46)
NoFlowIn
NoFlowPlace
NoFlow
NoComerrBias
NoSplitPickNet
NoMinLoss
FFN (Ours)
9.37
10.56
18.02
11.07
7.29
7.15
7.14
4.2 Real World Experiments
We evaluate FabricFlowNet in the real world and demonstrate that our approach successfully ma- nipulates cloth real robot system.
Experiment Setup. Our robot system consists of two 7-DOF Franka Emika Panda arms and a single wrist-mounted Intel RealSense D435 sensor (See Fig. 1). We plan pick and place trajectories using Movelt! [9]. We evaluate on a 30x30 em towel, using 6 single-step and multi-step goals (see Fig. 4) that form representative subset of our simulation test goals.
7

---

To transfer from simulation to the real world, we align the depth between real and simulated images by subtracting the difference between the average depth of the real support surface (i.e. the table) and the simulated surface. We mask the cloth by chler-thrs the background; see Appendix for details. We found that these simple techniques were sufficient to transfer the method trained entirely in simulation to the real world, because we use only depth images as input. Simulated depth images match reasonably well to real depth images, unlike RGB images.
4.2.1 Real World Results
Gool
Gool
Achieved
Xchivved
(a) One-step Square Cloth
(b) Multi-step Square Cloth
Figure 4: Qualitative results for FFN on real world experiments. FFN only takes depth images as input, allowing easily transfer cloth different colors.
Fig. provides qualitative real world results, showing that successfully achieve many of the goals. Our website (link in abstract) contains videos of these trials.
We compare FabricFlowNett to the NoFlow ablation from Sec. 4.1.2. Both methods used the same sim-to-real techniques described in the previous section. While we do not have access to the true cloth position error in the real world, Intersection-over-Unnboo (IoU) the achieved cloth masks serves as a reasonable proxy metric [21]. FFN achieves 0.80 mean loU over trials for the square cloth, compared to 0.53 for NoFlow. See the Appendix for additional details.
Gool
Goal
Achieved
Achioved
(a) Rectangular Cloth
(b) Printed T-shirt
Figure 5: Generalization new cloth shapes for FFN trained only on square cloth in simulation. FFN achieves single and multi-step goals for rectangular fabric and printed T-shirt.
Generalization. In addition to evaluating the folding policy square cloth for various goal con- figurations, we also test the generalization our method to other shapes of cloth. We evaluate the performance of FFN trained only on square cloth folding goals for rectangular cloth well as a T-shirt. These fabrics are also thinner than the square blue towel used the real world experiments above. Fig. 5 shows that FFN trained square yellow cloth in simulation is able to generalize to other cloth shapes, textures, and colors (FFN only receives depth images as input). See Appendix for additional details.
5 Conclusion
In this work we present FabricFlowNet. a method which utilizes flow learn goal-conditioned fabric folding. We leverage flow to represent the correspondence between observations and goals, and as an action representation. The method is trained entirely using random data in simulation. Our results show that separating the correspondence learning and the policy learning can improve performance on an extensive suite single- and dual-arm folding goals simulated and real environments. Our experiments also demonstrate generalization to different fabric shapes, textures, and colors. Future work on flow-based fabric manipulation could incorporate actions beyond pick and place, such as parameterized trajectories or dynamic actions.

---

Acknowledgments
This work was supported by the National Science Foundation (NSF) Smart and Autonomous Sys- tems Program (1IS-1849154, LG Electronics, and NSF Graduate Research Fellowship (DGE- 1745016).
References
[1] Artemij Amiranashvili, Alexey Dosovitskiy, Vladlen Koltun, and Thomas Brox. Motion per- ception in reinforcement learning with dynamic objects. In Conference Robot Learning, pages 156-168. PMLR, 2018.
[2] Max Argus, Lukas Hermann, Jon Long, and Thomas Brox. Flowcontrol: Optical flow based visual servoing. arXiv preprint arXiv:2007.0022 2020.
[3] Christian Bersch, Benjamin Pitzer, and Sren Kammel. Bimanual robotic cloth manipulation for laundry folding. In 2011 IEEE/RS International Conference on Intelligent Robots and Systems, pages 1419, 2011.
[4] Ricard Bordalba, Lluis Ros, and Josep M Porta. A randomized kinodynamic planner for closed-chain robotic systems. IEEE Transactions on Robotics, 2020.
[5] Júlia Borràs, Guillem Alenyå, and Carme Torras. A grasping-centered analysis for cloth ma- nipulation. IEEE Transactions Robotics, 36(3):924 926,936, 2020.
[6] Cheng Chi and Shuran Song. Garmentnets: Category-level pose estimation for garments via canonical space shape completion. arXiv preprint arXiv:2104.05117 2021.
[7] Rohan Chitnis, Shubham Tulsiani, Saurabh Gupta, and Abhinav Gupta. Efficient bimanual manipulation using learned task schemas. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 1 1149-1155. IEEE, 2020.
[8] Rohan Chitnis, Shubham Tulsiani, Saurabh Gupta, and Abhinav Gupta. Intrinsic motivation for encouraging synergistic behavior. International Conference Learning Representations, 2020.
[9] Sachin Chitta, loan Suean, and Steve Cousins. Moveit![ros topics]. IEEE Robotics Automa- tion Magazine, 19(1):18-19, 2012.
[10] Marco Cusumano-Towre Towner, Arjun Singh, Stephen Miller, James F. O'Brien, and Pieter Abbeel. Bringing clothing into desired configurations with limited perception. 20/ IEEE Interna- tional Conference Robotics and Automation, pages 3893-3900, 2011.
[11] Siyuan Dong, Devesh K Jha, Diego Romeres, Sangwoon Kim, Daniel Nikovski, and Alberto Rodriguez. Tactile-rl for insertion: Generalization to objects of unknown geometry. arXiv preprint arXiv:2104.01167, 2021.
[12] Alexey Dosovitskiy, Philipp Fischer, Eddy Ilg, Philip Hausser, Caner Hazirbas, Vladimir Golkov, Patrick Van Der Smagt, Daniel Cremers, and Thomas Brox. Flownet: Learning opti- cal flow with convolutional networks. In Proceedings of the IEEE international conference computer vision, pages 2758-2766. 2015.
[13] Andreas Doumanoglou, Jan Stria, Georgia Peleka, Ioannis Mariolis, Vladimr Petrk, Andreas Kargakos, Libor Wagner, Velav Hlav, Tae-Kyun Kim, and Sotiris Malassiotis. Folding clothes autonomously: A complete pipeline. IEEE Transactions on Robotics, 121))1441-14778 2016.
[14] Aaron Edsinger and Charles Kemp. Two arms are better than one: A behavior based control system for assistive bimanual manipulation. In Recent progress in robotics: Viable robotic service to human, pages 345-355, Springer, 2007.
[15] Aditya Ganapathi, Priya Sundaresan, Brijen Thananjeyan, Ashwin Balakrishna, Daniel Seita, Jennifer Grannen, Minho Hwang, Ryan Hoque, Joseph E Gonzalez, Nawid Jamali, et al. Learn- ing dense visual correspondences simulation to smooth and fold real fabrics. arXiv preprint 20X3 2020.

---

[16] Irene Garcia-Camacha Camacho, Martina Lippi, Michael C. Welle, Hang Yin. Rika Antonova, Anas- tasiia Varava, Julia Borras, Carme Torras, Alessandro Marino, Guillem Aleny, and Danica Kragic. Benchmarking bimanual cloth manipulation. IEEE Robolics and Automation Letters, 5(2):1111-1118, 2020.
[17] Huy Ha and Shuran Song. Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding. arXiv preprint arXiv: 105. 10..03655, 2021.
[18] Ryan Hoque, Daniel Seita, Ashwin Balakrishna, Aditya Ganapathi, Ajay Tanwani, Nawid Jamali, Katsu Yamane, Soshi Iba, and Ken Goldberg. VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation. In Proceedings of Robotics: Science and Systems, Corvalis, Oregon, USA, July 2020.
[19] Eddy 1lg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2462-2470, 2017.
[20] Rishabh Jangir, Guillem Alenyå, and Carme Torras. Dynamic cloth manipulation with deep reinforcement learning. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4630-4636. IEEE, 2020.
[21] Robert Lee, Daniel Ward, Akansel Cosgun, Vibhavari Dasagi, Peter Corke, and Jurgen Leitner Learning arbitrary-goal fabric folding with one hour of real robot experience. Conference on Robot Learning, 2020.
[22] Yinxiao Li, Yan Wang, Michael Case, Shih-Fu Chang, and Peter K. Allen. Real-time pose estimation of deformable objects using a volumetric approach. In 2014 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1046-1052, 2014.
[23] Yinxiao Li, Danfei Xu, Yonghao Yue, Yan Wang, Shih-Fu Chang, Eitan Grinspun, and Pe- ter K. Allen. Regrasping and unfolding of garments using predictive thin shell modeling. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2015.
[24] Xingyu Lin, Yufei Wang, Jake Olkin, and David Held. Softgym: Benchmarking deep rein- forcement learning for deformable object manipulation. In Proceedings of (CoRL) Conference on Robot Learning, November 2020.
[25] Martina Lippi, Petra Poklukar, Michael C Welle, Anastasiia Varava, Hang Yin, Alessandro Marino, and Danica Kragic. Latent space roadmap for visual action planning of deformable and rigid object manipulation. In 2020 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5619-56266 IEEE, 2020.
[26] Xiao Ma, David Hsu, and Wee Sun Lee, Learning latent graph dynamics for deformable object manipulation. arXiv preprint arXiv:2104. 12149, 2021.
[27] Jeremy Shepar,, Marco Cusumano-Towner Jinna Lei, and Pieter Abbeel. Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel fold- ing. In 2010 IEEE International Conference on Robotics and Automation, pages 2308 2315, 2010.
[28] S. Miller, J. V. D. Berg, Mario Fritz, Trevor Darrell, Ken Goldberg, and P. Abbeel. A geometric approach to robotic laundry folding. The International Journal of Robotics Research, 31:249 267,, 2012.
[29] Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Combining self-supervised learning and imitation for vision-based rope ma- nipulation. In 2017 IEEE international conference on robotics and automation (ICRA), pages 2146-2153, IEEE, 2017.
[30] Deepak Pathak, Parsa Mahmoudieh, Guanghao Luo, Pulkit Agrawal, Dian Chen, Yide Shentu, Evan Shelhamer, Jitendra Malik, Alexei Efros, and Trevor Darrell. Zero-shot visual imita- tion. In ICLR, 2018
10

---

[31] J. Sanchez, J. Corrales, B. Bouzgarrou, and Y. Mezouar. Robotic manipulation and sensing of deformable objects in domestic and industrial applications: a survey, The International Journal of Robotics Research, 37:688 716, 2018.
[32] Daniel Seita, Pete Florence, Jonathan Tompson, Erwin Coumans, Vikas Sindhwani, Ken Gold- berg, and Andy Zeng. Learning to rearrange deformable cables, fabrics, and bags with goal- conditioned transporter networks. arXiv preprint arXiv:2012.083 033.00038..... 2020.
[33] Daniel Seita, Aditya Ganapathi, Ryan Hoque, Minho Hwang, Edward Cen, Ajay Kumar Tan- wani, Ashwin Balakrishna, Brijen Thananjeyan. Jeffrey Ichnowski, Nawid Jamali, Katsu Ya- mane, Soshi Iba, John Canny, and Ken Goldberg. Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisor. In IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.
[34] Ruhizan Liza Ahmad Shauri and Kenzo Nonami. Assembly manipulation of small objects by dual-arm manipulator. Assembly Automation, 2011.
[35] Christian Smith, Yiannis Karayiannidis, Lazaros Nalpantidis, Xavi Gratal, Peng Qi. Dimos V. Dimarogonas, and Danica Kragic, Dual arm manipulationa survey. Robotics and Autonomous Systems, 60(10):1340-133 2012.
[36] Ral Surez, Jan Rosell, and Nstor Garea. Using synergies in dual-arm manipulation tasks. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 5655-5661, 2015.
[37] Daisuke Tanaka, Solvi Arnold, and Kimitoshi Yamazaki. Emd net: An encodemanipulatede- code network for cloth manipulation. IEEE Robotics and Automation Letters, 33)):1771-1778,, 2018.
[38] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European Conference on Computer Vision, pages 402-419. Springer, 2020.
[39] Yilin Wu, Wilson Yan, Thanard Kurutach, Lerrel Pinto, and Pieter Abbeel. Learning to Manip- ulate Deformable Objects without Demonstrations. In Proceedings of Robotics: Science and Systems, Corvalis, Oregon, USA, July 2020.
[40] Fan Xie, Alexander Chowdhury, M. Clara De Paolis Kaluza, Linfeng Zhao, Lawson Wong, and Rose Yu. Deep imitation learning for bimanual robotic manipulation. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balean, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 2327-2337. Curran Associates, Inc., 2020.
[41] Kimitoshi Yamazaki, Ryosuke Oya, Kotaro Nagahama, Kei Okada, and Masayuki Inaba. Bot- tom dressing by dual-arm robot using clothing state estimation based on dynamic shape changes. International Journal of Advanced Robotic Systems, 13(1):5, 2016.
[42] Wilson Yan, Ashwin Vangipuram, Pieter Abbeel, and Lerrel Pinto. Learning predictive repre- sentations for deformable objects using contrastive estimation. Robotics: Science and Systems, 2020.


Reviews:
[{"review_id": "zT7knCXjWQ8", "reviewer": "Reviewer_a9Nv", "summary": "In this work, a goal-conditioned cloth manipulation policy is developed that uses optical flow to improve its performance. Two networks are used: the first network estimates flow between observation and goal images; the second network use flow from the first network to predict pick points and place points. The proposed method was tested both in simulation and on a real robot setup, compared against other model-free and model-based cloth manipulation policies, with promising results shown in the experiments. \n\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main contribution of the paper is the presentation of a flow-based approach for learning goal-conditioned cloth manipulation policies. As mentioned by the authors a common approach for goal-conditioned policy learning requires the network to learn the relationship between observation and the goal and simultaneously to use this relationship to reason over the actions. With their method, these two tasks are separated and the relationship task is performed using the flow network (but this could be better justified, see more comments below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presents an interesting idea for cloth manipulation (cloth folding in this work) by taking optical flow as the input to predict the pick and place points in the manipulation. It is claimed that the proposed method is better predicting the pick and place points compared to the ones having the observation learning and action prediction together. However, this is not well justified as essentially the proposed method can be extended to connecting the Flow network and PickNet together, i.e., learning the relationship between observation and the goal first and then predicting the actions in one network, trained in an end-to-end manner.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "1. In this work, the relationship between observation and the goal and the actions predictions are done separately, how will it affect the performance if two tasks are performed in a network?\n2. The ablations: I was not able to find (later found that in appendix) the information on what type of data these experiments were conducted: “one-step”, “multi-step” or “all”; and also how many repetitions were done to achieve the presented results (in table 2). This can be found in Appendix, however, this seems relevant enough to add in the main text shortly.\n3. For the real world experiments, how many trials were conducted?\n4. Table 1 presents the average value over several experiments conducted, It would be additionally informative to show a standard deviation for each average value. \n5. How was alpha taken as 30 for all experiments? For fabrics of different sizes, alpha is to be different, isn't it?\n6.\tThe article is well organized and easy to read, however, in my opinion, the conclusion section could have been elaborated a bit more.\n7.\tLine 194, I suggest rephrasing the statement, instead “learned in Sec. 3.3” could be “as explained in Sec. 3.3” or “learned as explained in Sec.3.3”.\n8.\tPickNet – policy network, is composed of two sub-components PickNet1 and PickNet2, this naming convention can be a bit confusing at first glance. This is well explained in the text but without a proper read the PickNet1 and 2 can be considered as some variants of PickNet.\n9.\tSome information in the paper seems unnecessarily repeated: \na.\tPrevious methods for goal-conditioning policy have to learn current and goal relationship and reasoning, while the presented method separates these two tasks – this information appears in lines: 126-129, 174-175, and shortly in conclusions (323).\nb.\tThe improvement compared to the previous works – lines: 73, 87, 95, 142. (the context in these usages is sometimes a bit different but it still seems like repeating).\n"}}, {"review_id": "8Qo4vWMuD8R", "reviewer": "Reviewer_sL8a", "summary": "The authors proposing using dense point-to-point cloth correspondence (flow) as a representation for cloth manipulation. The authors propose training a flow model in simulation which produces point correspondences between observations (initial configurations) and goal states of the cloth. From this representation they then predict pick points and use the predicted flow to implicitly determine placement locations, resulting in a final pick-and-place style policy. They show convincing results in a bimanual tabletop environment with two Panda arms.\n\nUpdate (Post Author Response):\nI appreciate the author's response to my questions; fundamentally my opinion of this paper still stands: It proposes an interesting and practical representation for cloth manipulation and they present fairly compelling evidence for its efficacy in the regime that it was designed for (fairly nicely behaved scenes and initial conditions). I think this paper is a useful addition to the field and advocate for its acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strengths of this work lie in its simplicity and effectiveness. The authors show that simple point-to-point correspondence, trained in sim, generalizes well to the real world and results in a usable system (even ~2cm of placement error in the multi-step version is pretty impressive compared to prior work). The authors also nicely ablate the differences between the flow-based input and the flow-based output parameterization. Interestingly (and surprisingly for me), most of the benefit came from the output parameterization using flow. I also found the paper to be well written and clearly presented, without unnecessary pomp and circumstance to try to dress up the work (at the cost of clarity and understandability). This is refreshing and great to see!\n\nThe biggest weakness of this paper was the lack of dynamics incorporation. Generally, these sorts of pick and place policies are inefficient compared to policies that incorporate dynamic effects (such as grasping and shaking a cloth to flatten it before the next fold). This isn't a fatal flaw, but it would strengthen the paper to briefly discuss this limitation and how a flow-based representation might fit into a dynamically aware paradigm in the future.\n\nAnother question is how well the long-term temporal correspondence works when dealing with highly crumpled initial conditions. By relying so heavily on good long-horizon flow/correspondence predictions, it seems that this method might struggle with such cases (e.g. a crumpled ball of a shirt). It would also strengthen the paper to evaluate such examples (even if the answer is \"this representation works less well in those cases\").", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper is relatively novel (correspondence-based representations and flow have been around for quite a while, as the authors themselves note, but their application to cloth folding in this way is novel). In the relatively controlled domain the authors explore, this approach appears effective and the work is well-written and includes useful, and in some cases surprising (see my reaction to the ablation section) information for other practitioners.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "I have no major issues that I need addressed. In a vacuum (i.e. not having discussed with other reviewers) would advocate for acceptance in the paper's current state.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "3d1-loHFe_g", "reviewer": "Reviewer_RKFg", "summary": "The paper contributes a novel and efficient framework for bimanual cloth manipulation. Specifically, the proposed method leverages the concept of optical flow to represent both the cloth pose and actions. Without relying of expert demonstrations or explicit reward signals, the proposed approach is shown to be capable of learning cloth manipulation policies that generalize to new cloth shapes without additional training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n+ The paper addresses a timely and well-motivated problem.\n+ The use of optical flow for cloth manipulation seems to be a novel and ingenious application of well understood concept from a different filed (computer vision / estimation).\n+ The experimental design includes multiple tasks (including those that require multiple steps), ablation experiments, a real-robot platform, and comparisons against SOTA approaches to cloth manipulation. Further, the results clearly demonstrate the utility and relative benefits of the proposed approach over existing approaches.\n+ The method neither relies on expert demonstrations nor explicit reward signal. It simply leverages random explorations to collect data for supervised learning of flow prediction\n+ The method autonomously switches between single and dual arm behavior depending on the distance between pick (or place) points.\n\n**Weaknesses**\n\n- The related work section on \"Policy Learning for Cloth Manipulation\" merely lists model-based and model-free methods for cloth manipulation, and mentioned that experimental comparisons are carried out. A qualitative discussion on the similarities and differences (much like how it is done in other subsections of related work) would be a lot more useful here.\n- The approach assumes that the appropriate number of sub goals required for any given goal is known and each of those sub-goals provided to the robot.\n- The approach relies on a simulator for ground-truth. While this reliance is not a weakness in itself, the paper needs to make this more important. For instance, this reliance must be disclosed in the introduction.\n- The paper does not cite or utilize an important related work (Garcia-Camacho et al., RAL, 2020) that contributes a comprehensive collection of evaluation metrics and benchmark scenarios for bimanual cloth folding that would have made the evaluation much stronger. Further, this prior somewhat weakness the second contribution of the paper relating to the test suite for benchmarking\n\n**References**\n\n- Garcia-Camacho, Irene, et al. \"Benchmarking bimanual cloth manipulation.\" IEEE Robotics and Automation Letters 5.2 (2020): 1111-1118.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper proposes a novel and effective approach to learning bimanual cloth manipulation policies. Barring some relatively minor weaknesses in the experimental evaluation, the proposed method appears to be much more efficient and somewhat more effective than SOTA approaches.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "1. Please considering removing the last bullet from the list of contributions as evaluation of a proposed framework can not considered as a contribution. Having said that, it might be appropriate to list the ability of the proposed framework to general to other cloth colors and shapes as a contribution.\n2. How robust is the proposed approach expected to be variations in the background?\n3. It seems difficult to know how many sub-goals are necessary for a given task/goal. Can exiting methods (e.g., Ref. [33]) address this challenge? \n4. For the ablation experiments, please include the error associated with the full method in Table 2 (currently, only relative improvements are reported in text and the actual error is only included in the supplementary material).\n5. I could not quite understand the motivation for not including key information in Table S2 from the appendix in the manuscript (e.g., many variants of the Fabric-VSF baseline from this table do better than what is reported in Table I for Fabric-VSF). Why are the best performing variants of the baseline not included in the manuscript? \n6. Unlike the proposed approach and Fabric-VSF, why was Lee et al.'s ability to generalize to new cloth shapes not evaluated?"}}]

Author rebuttal:

Meta-review: All three reviewers were generally positive about the paper, and the main weaknesses were based on small queries and clarifications, which should be addressed in the rebuttal. Along with these, I would also like the authors to particularly address two issues raised by the reviewers. Firstly, I would like to hear motivation for the need to separate the flow network and the pick network, rather than training a single end-to-end network, as raised by Reviewer a9Nv. Secondly, I would like to hear the authors' option on whether the need to manually specify the sub-goals for a particular task, as mentioned by Reviewer RKFg, is a significant weakness of the work.

-----

After the reviews, authors have provided an updated paper which addresses the reviewers' main issues, including some new experiments. This is a new state-of-the-art method for deformable object manipulation, with good real-world results, and all three reviewers recommend acceptance of the paper.
Decision: Accept (Poster)