Iterative Interactive Modeling for Knotting Plastic Bags
Authors: Chongkai Gao
Zekun Li
Haichuan Gao
Feng Chen
Venue: CoRL 2022 Poster (2022)
Keywords: Plastic Bag Manipulation
Learning from Demonstrations

Abstract: Deformable object manipulation has great research significance for the robotic community and numerous applications in daily life. In this work, we study how to knot plastic bags that are randomly dropped from the air with a dual-arm robot based on image input. The complex initial configuration and terrible material properties of plastic bags pose challenges to reliable perception and planning. Directly knotting it from random initial states is difficult. To tackle this problem, we propose Iterative Interactive Modeling (IIM) to first adjust the plastic bag to a standing pose with imitation learning to establish a high-confidence keypoint skeleton model, then perform a set of learned motion primitives to knot it. We leverage spatial action maps to accomplish the iterative pick-and-place action and a graph convolutional network to evaluate the adjusted pose during the IIM process. In experiments, we achieve an 85.0% success rate in knotting 4 different plastic bags, including one with no demonstration.

Iterative Interactive Modeling for Knotting Plastic Bags
Chongkai Gao
Zekun Li
Department of Automation Tsinghua University
Department of Automation Tsinghua University
gck200mails tsinghua, edu. cn
1izk21 @maiis.mmil.....mmmmm tsinghua. edu. cn
Haichuan Gao
Feng Chen
Department of Automation
Department of Automation, Tsinghua University
Tsinghua University
LSBDPA Beijing Key Laboratory
ghc18@mails. tbc18emails.tsinnhuaa edu.
chenfeng@mail wwt.gmmmmil.tsinnaaaaaaaa edu.
Abstract:
Deformable object manipulation has great research significance for the robotic community and numerous applications in daily life. In this work, we study how to knot plastic bags that are randomly dropped from the air with dual robot and image input. The complex initial configuration and intricate physical properties of plastic bags pose challenges for reliable perception and planning. Directly knotting it from random initial states difficult. this work, we propose Iterative Interactive Modeling (IIM) first adjust the plastic bag standing pose with imitation lcarning to establish a high-confidence keypoint skeleton model, then perform a of learned motion primitives knot it. We leverage spatial action maps to accomplish the iterative pick-and-pe aed--acd action and graph convolutional network evaluate the adjusted pose during the IIM process. experiments, we achieve an 85.0% success rate in knotting 4 different plastic bags including one that has no demonstration.
Keywords: Plastic Bag Manipulation, Learning from Demonstrations
Introduction
Deformable object manipulation (DOM) has been long standing problem in robotics. Researchers have been studying manipulating various kinds of deformable objects. from linear objects [], fab-
(a)
rics [3, 4], papers [5] to elastic and elasto-plastic objects [6. 7, 8]. Apart from them, plastic bag, perhaps the most widely used application plas- tics in our daily life, has remained unexplored in robotics literature due to the high complexity in- volved in modeling and controlling its deforma- tion. Endowing robots with the ability to ma- nipulate plastic bags spawn diverse industrial and domestic applications in warehouses, garbage dumps, and supermarkets. Knotting plastic bags is one of the most representative manipulation tasks on plastic bags for a robot show its dexterity in the real world. In this paper, we study how to knot plastic bags that are randomly dropped from the air with a dual arm robot and raw image inputs.
(d)
Figure 1: Overview of our work. (a) A randomly dropped plastic bag. (b) Inaceu- rate depth measurement from Intel Realsense D435i. (c) Unstable keypoint detection the initial state. (d)(e) We propose IIM to first iter- atively interact with the plastic bag to shape to a standing pose, then build high-confidence 3D keypoint skeleton with triangulation. (f) Knot tying with learned motion primitives.
A randomly dropped plastic bag is shown in Fig- ure I(a). Knotting it from such an irregular initial configuration requires locating and taking the han- dles out from the mess and delicate coordination
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

of dual arms to tie the knot. Compared to previous knot tying tasks that have been well studied on ropes [1, 9, 10. 11]. this task poses new challenges for both perception and planning. because of the complex initial configuration and the intricate material and dynamics property of plastic bags. For perception, as shown in Figure (bb), the translucent and non-Lambertian surface of plastic bags makes depth sensing inaccurate, thus 3D vision based models such as point cloud [12, 13, 14] or voxels [15] are not suitable for our task. Meanwhile, 2D vision models such as 2D keypoints [16] are not reliable due to the severe self-occlusion and partial observation problem caused by com- plex initial configurations (Figure I(c)), For planning, mathematical physical models are difficult to build for plastic bags since their physical parameters knowledge, no simulator can be employed. Learning a dynamics model for planning [7, 13, 15] is also infeasible since motions of plastic bags not quasi-static. Although some image-based imi- tation or reinforcement learning approaches [3, 4. 17, 18] can directly learn a visual policy without visual representation physical model, their learned policies either employ just small action space such as pick-and-place that is not enough to accomplish the knot tying task, or are unable to cope with complex initial configuration that appear in a randomly dropped plastic bag. Therefore, directly knotting the plastic bag from initial configurations is hard for both perception and planning.
very hard to estimate, and to the best of our
Fortunately, the elasto-plastic property of plastic bags gives us another way to accomplish the knot tying task. Elasto Elasto-plastic objects can be shaped to a predefined target state in equilibrium and un- changing. This is called deformation control [19], and previous works have studied how to learn dynamics model [13] or estimate physical parameters [20] of clasto-plastic objects such as sponges, plasticine, or clay by interacting with them during this shaping process. However, for our task, this interactive shaping process is more valuable for reducing difficulties of perception and planning: shaping a plastic bag from a randomly dropped state to standing pose (Figure (d)) is a process of straightening it, and the partial observation and self-occlusion problem can be gradually alleviated during this process. This allows to build a more reliable visual model (such as keypoint skeleton, Figure (())), and the knot tying task can be realized from the standing pose based on this model. This adjusting process can be accomplished with only iterative pick-and-place actions [17, 4, 21] using recent advances, thus we can tackle the original problem by developing a visual learning policy to straighten the plastic bag and then acquiring a reliable perception model to perform the down-streaming knot tying task.
In this work, we propose Iterative Interactive Modeling (IIM) for knotting plastic bags randomly dropped from the air with only image input. We train the robot to first shape the plastic bag to standing pose with the help of demonstrations and then establish keypoint skeleton model with multi-view stereo images to knot it with a set of learned adaptive motion primitives. Specifically, the robot iteratively performs different kinds of top-down pick-and-place actions on part of the plastic bag to outspread it, meanwhile, a task progress module evaluates if the pose is good enough for knot tying. We leverage spatial action maps [17, 18] to accomplish the pick-and-place -lace action, and train a graph convolutional network as the task progress module with the same demonstrations to evaluate the keypoint skeleton during the adjusting process. To enable keypoint detecting on plastic bags, we provide the first 2D plastic bag keypoint dataset PBPose with 43,200 images to train an off-the- shelf 2D keypoint detection model RLE [22]. After the IIM process, we lift the plastic bag into air with geometrically constrained planning to knot the plastic bag with set of motion primitives with trained action parameters from CNN. In our experiments, we achieve an 85.0% success rate in tying four different plastic bags (one of them has no demonstration) that randomly dropped from the air with our dual Kinova Gen3 arms equipped with standard Robotiq 2F-85 grippers, with only 100 demonstrations (1.5 hours) provided for each plastic bag. In summary:
We propose Iterative Interactive Modeling (IIM) for complex elasto-plastic object manipu- lation that iteratively shapes the object to facilitate more reliable perception and planning.
We leverage spatial action maps, graph convolutional networks and the RLE model to per- form IIM on plastic bags, and train motion primitives to accomplish the knot tying task.
We build the first dual-arm robotic system to knot plastic bags randomly dropped from the air with the provided PBPose dataset and small number of demonstrations.
2 Related Works
Deformable object manipulation (DOM) has long been a challenging area of robotics research. The challenges of DOM come from two properties of deformable objects: the infinite degrees of freedom and the complex non-linear dynamics, which lead to difficulties for perception, modeling, and plan-

---

ning. In this section, we compare different techniques for modeling and manipulating deformable objects and discuss whether they are suitable for knotting plastic bags.
2.1 Visual Representations of Deformable Objects
Keypoints [23, 24, 25] are commonly used representations. They are sparse representations for the structure of target objects, and grasping points usually generated from them. Keypoint detec- tors can be trained either with supervised or unsupervised ways [26] to find keypoints with image or point cloud inputs to allow robots to manipulate the object based on some geometry loss on the keypoints. Dense visual descriptors [8, 27, 28, 29] are recently used as a real-time pixel-wise dense representation for many kinds of objects, and humans can specify grasping points on it. Besides, the 3D vision community has made great progress in modeling deformable objects, with voxels [30], meshes [31], convexes [32], or implicit functions such as flow-based model [33] or neural radiance fields [34]. and most of these model can represent the deformable object in high fidelity. Other works seek to use particles [35, 14, 36] that are down-sampled from point cloud data to represent deformable objects. These methods usually combine graph neural networks and differentiable sim- ulators [37, 38] to learn the dynamies of objects. Lastly, visual policy learning methods simply use high-dimensional feature embedding learned from deep neural networks object representations for manipulation [3, 4, 1,, 39] and have achieved great success in various tasks.
For plastic bags, current commercial depth sensors fail to give accurate depth estimation because of the translucent and -amberarnt surface, thus 3D vision based methods are not feasible. Since we aim to manipulate the plastic bag rather than studying how to build refined representation, those high-fidelity representations are not necessary: they all degenerate to grasping points [28] in practice for manipulation. Thus, we choose keypoint skeleton as the representation of plastic bags.
2.2 Manipulating Deformable Objects
A lot of work aims to build dynamics model for the target object first and then perform motion planning on it to manipulate the object. Conventional works build the dynamics model mathe- matically [40, 41, 42], and some of them have achieved promising results recently by planning on simpler approximated dynamics and using local controllers to tackle actual complex dynamics [43, 44, 45). Other works aim to learn the dynamics model from visual data [13, 15, 7] or tactile data [46, 47]. Recently, reinforcement learning and imitation learning methods are developed to di- rectly manipulate the object without dynamics model, and have achieved success on various tasks [3, 48, 1, 10, 17, 49]. Some of them train the robot in simulation with fast virtual experiences and then transfer the learned policy directly to real robots or with sim-to-real methods [27, 50]. Others try to directly train the robot in the real world with its own experience or expert demonstrations [4].
For manipulating plastic bags, dynamics models are hard to build since the complex non-lineat dynamics, and the motion of plastic bag is not always quasi-static: some parts of the plastic bag may collapse after it has been moved. In this work, follow the idea of most visual policy learning approaches that directly learn a pick-and-place policy with a limited action space to accomplish the adjusting task, and use geometrically constrained planning to lift the plastic bag the air to knot with a set of learned motion primitives.
3 Method
The goal of this paper is to enable a dual arm robotic system to knot plastic bags from initial ran- domly dropped states. We use keypoint skeleton as the visual representation for plastic bags. The robot is trained to iteratively adjust the plastic bag standing pose to build complete and high- confidence keypoint model with the help of a task progress module, then tie the knot with a set of learned motion primitives. We introduce the problem setting in 3. the keypoint detection model in 3.2, the iterative interactive modeling process in 3.3, and the knotting process in 3.4.
3.1 Problem Statement
We use the most common type of plastic bags that are colored, medium-sized (33.7~42.5 em wide and 40.2~42.3 em long when laid out), translucent, and have two looped handles, as shown in Figure 2(a). We choose four kinds of plastic bags that vary in size and color. In order to make the tying problem possible, we put some items in the plastic bag to resist indoor airflow. The plastic bag is randomly dropped from the air to form an initial state. The knot we tie in this work is kind of lan Knot [9] which can be tied with two motion primitives (Action I and Action 2) from the stage of hanging on grippers, as illustrated in Figure 2(e): 1) The right arm grasps the rear side of the left
3

---

(a
items
(c) Initial Stage
Aciin
A
Finai Stage
Figure 2: (a) Plastic bags and fillers used in this work. We give demonstrations on white, red and black bags. The grey plastic bag is used to show the generalization of IIM. We fill plastic bags with different items varying in size and shape, including cloths, paperboards, bottles, tape volumes, and dry batteries. (b) The PBPose dataset. Images are captured various backgrounds, knotting states, occupied states, illumination conditions, placement status (standing or lying down), and angles of view. (c) We design two motion primitives for tying an Ian Knot in the air.
handle to allow the left arm go back; 2) The right arm rotates 90" let the left arm grasp the front side of the right handle. The knot is tightened by pulling two handles away from each other.
The whole experimental setup is shown in Figure 3. We use two identical Kinova Gen3 (6DoF) arms and Robotiq 2F-85 grippers for our experiments. We mount two Intel Realsense D435i cameras Cieft and Crighe the end of each arm respectively the end-effector perspective. The distance between the bases of two arms is 51.2cm. Cright looks straight down at the XY plane from top view to get right within the range of 75.4em x 56.6cm with the plastic bag at the center. On the other side, we get Teftt by moving Cleft aiming at the plastic from set of sipeaa-separated poses to get stereo images.
get image inputs Ief/ and Fright from
3.2 3D Keypoint Skeleton for Plastic Bags
Directly 3D keypoint detecting methods and reconstruction methods are not suitable for plastic bags discussed in section 2. Thus in this work detect 2D keypoints of plastic bags with image input and reconstruct 3D keypoint skeleton with multi-view geometry. This requires plastic bag keypoint dataset and well-trained 2D keypoint detection model.
To this end, we provide PBPose, the first 2D plastic bag keypoint dataset with 43,200 images of four different kinds of plastic bags. Images in PBPose vary from the background, knotting state, occupied state, illumination condition, placement status, and angle view, as shown in Figure 2(b). For each image, we manually label 19 keypoints of the plastic bag from handle to bottom to establish full skeleton of it, along with point visible properties and the bounding box. Based on this dataset, we train RLE network [22] f for 2D keypoint detection. takes as input of single frame from Tieft and outputs predicted 2D joints of the target plastic bag along with confidences each point.
From the illustrated performances and (e)), we can see that it is not robust and reliable to detect keypoints from random initial config- urations. Although this problem can be mitigated by providing more training data at different initial configurations, is hard to cover enough cases the infinite continuous initial states. Thus this work sidestep this problem by shaping the plastic bag to standing pose.
plastic bags at initial states and standing poses in Figure I(c)
3.3 Iterative Interactive Modeling
We adjust the plastic bag from its initial state by iteratively picking and placing it with expert demon- strations, and automatically evaluate the pose with task progress module. We introduce these modules one by one in the following part. Note we do not require the whole adjusting process to be quasi-static, which means the plastic bag can partially collapse before it can steadily stand.
Observation and Action Space Definition: At every step, we get h multi-view images Imm {Teeee- -left ft from Cteft (h- 15 in this work), and one top-view image Fright from Cright- The robot can choose two kinds of pick-and-place action at each step: 1) single-arm picking action

---

right
Pick- tokk-and-pace
lieft
Generating grasping points
G
Knot tying
kreecdinkkyppoinsskkeeeo
Figure 3: The IIM iteration for knotting plastic bags. At each step, the right camera gets a top-down view image right, and the left camera gets a sequence of side view images Tej: We use spatial action maps module ø to get the grasping points, and use sparse reconstructed point cloud to get the grasping depth. At the same time, we detect 2D keypoints from Lief with facilitate the task progress module G determine which picking action to use, and when the adjusting pose is good enough to support building 3D keypoint skeleton to tie the knot.
Pi that goes top-down to location E 3R3 and picks up to 0.55m height along the axis; 2) dual-arm picking action p2 that goes top-down to locations L, R E R and picks up 0.55m height along the Z axis respectively. After each picking, the arm can choose to go to a left/staying/right placing position to release the plastic bag, shown in Figure 4. pi is used for most pick-and-place actions and p2 is only used for final steps to get a good standing pose.
The robot needs to choose which action to use, and grasping and placing locations at every step. Specifically, the zy locations of grasping points are determined by our learned visual policy. The parameter (grasping depth) is determined by first running sparse reconstruction of stereo images by structure from motion algorithm (SfM) to get sparse point cloud (using COLMAP [51] this work), then calculating the average height of the top- 10 points on a cylinder with radius of lem with the center of the bottom surface of the grasping point to get the grasping depth. For determining the placing location, we segment right and calculate the relative sizes of the left-side and right-side areas. If the ratio of two areas less than 0.5 or greater than the robot will place the plastic bag to the smaller-side placing point. Otherwise it will release the gripper in situ, shown in Figure 4.
Demonstration Collection: human expert demonstrates how to accomplish the adjusting process for the robot. For each demonstration trajectory D = {Pen Pel. PPa-1 }. it includes & picking actions, where co: c C denote picking categories. For each Pe,, we record T// and Tright before performing the action, and the grasping locations L, R E IR2 from the perspective of Cright, along with the grasping angles Oz, On. Thus, Pe, (Teett Tright, I., R, Or, ).. The expert al- ways grasps handles of the plastic bag for all picking actions. This is important establish a unified grasping principle to avoid ambiguity for imitation learning. P2 may be used several times at the end of trajectory to form good standing pose. In this paper, find 100 demonstration trajectories are enough for each kind of plastic bag, which only take 1.5 hour for human to demonstrate.
Visual Grasping Module: The robot determines the grasping points { (LL, 0y.), (R, 0R)) from Lright with spatial action maps [17, 21, 18), which have shown promising results for learning heatmaps of visual-affordances over pixels with fully convolutional networks. They can recover pixel-wise grasping affordances of different graspings with input images being rotated to achieve a discrete set of possible actions. Concretely, shown in Figure 4, given image from Lright, we first segment out the plastic bag from the background and generate 16 rotated images (multiples 22.5"), then pass them through the fully-convolutional network 6 predict the corresponding set of heatmaps within the same size of the input image. In this work we use a pixel resolution of 160 x 120. The pixel with higher predicted probability among all 16 maps is
suitable for grasping.
The problem our case is that we have to determine one grasping point for p1 but two grasping points for p2. To this end, we propose a Selection Rule based on safe distance constraints to avoid collisions of grippers: for pı and P2, the pixel with the highest value corresponds to (R, 0z). When choosing (L, 0r) for p2, we first delete nearby areas (a circle of 5 pixels radius) of (R, 0R) on all 16
5

---

Sogment roate
Networks
ere
Selection rules & Distance constraints
One/two grasp points
Pick up
Plaze
down view segimenttioon
Depth eddimation from
To the smaller side
Tright
Caposdates
pick-and- and-place process. Fright is segmented and rotated to get copies. We send these images through network & and choose one or two grasping point(s) from the output candidates according to our selection rules and distance constraints. We get the grasping depth of these points by calculating the depths of top- points above the grasping points and perform the picking action. After picking it upward, we determine where to place the plastic bag by segmenting the top-down view and compare relative sizes of left-side and right-side area of the plastic bag.
Figure 4:
output images, and select the pixel with the highest value in the remaining area. If this pixel has value higher than 0.8, then we choose it as (L, 0t), otherwise we abort p2 and only perform p1.
Task Progress Module: To evaluate the quality of poses during the adjusting process to determine when the robot can use p2 and when the pose good enough to facilitate knot tying, we propose to use graph convolutional network to classify the keypoint skeleton during the IIM process to predict the task progress stages. We represent the 2D keypoint skeleton as graph formed by state S = (O,E), where vertices O are the keypoints. Concretely, o; = (x,, Cz), where and c; are 2D pixel locations and confidences of each keypoint respectively. The edges & are the connectivity of each pair of keypoints that are predefined. At each step, graph convolutional network G' gets graphs abstracted from stereo image inputs of Lefft and detected keypoint skeletons by f and calculates the mean state S. Then it classifies current step to three categories: 1) ordinary picking step (using Pi): 2) final picking step (using P2): 3) ending step (no picking action will be chosen). We train G with the standard cross entropy loss and the same demonstrations collected above.
After IIM, get well-standing plastic bag. We recover the 3D keypoint skeleton of by perform- ing triangulation from detected 2D keypoints of stereo images of Lej with the help of calibrated camera intrinsic and extrinsic parameters. The next step is to knot the plastic bag with this model.
3.4 In-air Knotting Plastic Bags by Learning Motion Primitives
As shown in Figure 2(c), tying an Ian Knot needs to first lift the plastic bag to the air to eliminate the influence from ground. This re- quires the direction that allows each arm to into each ring on handles. as shown Figure 5(a). There are six keypoints on each ring, and they are not on the same plane in R33 Thus we here calculate ring plane that minimizes the total least squares (TLS) distance from each point to it. Then we calculate the perpendicular direction of each ring plane, and let this perpen- dicular go through the center point of the ring, which is the average of 3D coordinates all the ring points. By TLS, we know this center point is on the ring plane too. We set each robot arm on its perpendicular respectively and place each gripper 0.1m away from the center point of each ring. Then we close both grippers and move the arm along the perpendicular to insert into the ring, and keep both end effectors horizontal with a 0.25m distance to lift the bag to height of 0.5m. Finally we open both grippers to reach the initial stage for tying an Ian Knot.
robot amm
xby
(a)
Figure 5: (a) Calculating the inserting direction. (b) Different initial stages for tying Ian Knot. (c) End-effector view before action I and action 2. Red stars represent goals.
The second step is to perform the two motion primitives. However, as shown in Figure 5(b), the dangling handles of plastic bags on the gripper may hang in different specific places. This requires both actions to adaptively choose their goals (the grasping points, see Figure 5(c)) according to

---

Table 1: Success rates of different methods. FSR of baseline methods are calculated by using our method to complete the missing parts of them.
Metires
Plastic Bags
DI
RP
VBE-S
VHEET
HKT
Ours
Red, Black. White
4.0%6(/6)) 5.09(1120)
(8/6%
(42066))
(3966)))9/0)
(5336))))
ASR
Grey (Unleamed)
10.0%;(2/20)
(0.0%(8/20)
(0.0%(
80.0%
Red, Black, White Grey (Unleamed)
4 (1566))5/60)
(5966)))))
KSR
(8/0%)
90.0% (18/20)
Red, Black. White Grey (Unlearndd
4.0%((/60) (3.3%)
70.0% (42/60)
63. (38366))
38.3%
86.7%
FSR
0.0%((0/20) 10.05%(2/20) 30.0%(6/20) (5.0%)
25 25.0%)(5/20)
80.0%%(16/20
actual situations. To this end, we train a 4-layer convolutional neural network using Huber loss with hidden size=256 to regress grasping points for both actions directly in IR with images input, as shown in Figure 5(c). Following up demonstrations in 3.3, we extend each demonstration with a knot tying part. For each demonstration, we record two images from right and left end-effector views respectively for action I and action 2, and record corresponding grasping locations in RSS
4
Experimental Results
4.1 Metrics and Baselines
We evaluate our method on success rates of knot tying on three plastic bags that have demonstra- tions them and new type of plastic bag (grey) that has no demonstration on it to show the generalization ability of our method. Concretely, we evaluate: a) Adjusting Success Rate (ASR): if the plastic bag is successfully adjusted to a standing pose from randomly dropped initial state to facilitate a valid inserting and lifting action (evaluated by human); b) Knot Tying Success Rate (KSR): if the plastic bag is successfully knotted from randomly in-air initial stage. c) Full Task Success Rate (FSR); if the plastic bag is successfully knotted from randomly dropped initial state. For ASR and FSR, we say one attempt fails if it does not succeed in 10 steps of grasping. We show the effectiveness of different modules in our method by a set of ablated versions:
Directly Inserting without Adjusting (DI): This method aims to directly find the handles of the plastic bag to lift it with the same method in 3.4 when it is just dropped from the air by the keypoint skeleton detected at the initial state. This is used to show the necessity of IIM.
Random Picking (RP): The robot randomly picks up a point of the plastic bag based on the seg- mented plastic bag area from top-down view image Liigtt and randomly generated a single grasping point in this area. This is used to show the effectiveness of the spatial action maps ø.
Vision Based Keypoint Skeleton Evaluation (VBE): This method evaluates the goodness of the adjusted pose of the plastic bag directly from images rather than using a GCN to process the keypoint skeleton. This is used to show the effectiveness of the task progress module G. We use channel-win concatenated side view images (VBE-S) and top- down view images (VBE-T) for classifying stages in this baseline respectively.
Hard-coded Knot Tying (HKT); This method ties the Ian Knot with the same primitives of our method but use hard-coded goals for each action. This is used to show the effectiveness of learned motion primitives by CNN.
4.2 Quantitative Results
Table 1 shows the ASR, KSR, and FSR of different baselines. Our method achieves the best success rates in all metrics. For ASR, DI can barely knot the plastic bag, since most of the initial states do not support high-confidence keypoint detection (as shown in Figure 6). but occasionally randomly dropped plastic bag can form a good standing pose. RP achieves similar results. Random picking can stretch out the plastic bag to some extent, but it can never end up with good standing pose, which needs a dual-arm picking action P2. VBE methods achieve much better results than the above two methods on the training plastic bags. especially for VBE-S. However, their performance drops dramatically on the grey plastic bag. This is because although images can also provide information about poses of plastic bags, they do not extract and use the essential information for evaluating poses such as the keypoint skeleton, and are susceptible to different illumination conditions. Meanwhile we only provide hundreds of images for each plastic bag. which may not be sufficient for image- based classification. That is why VBE methods are not as good as IIM on training plastic bags and lack generalization abilities to new plastic bags. For KSR, the hard-coded knot tying method can achieve an average 41.25% success rate. Most of failures of HKT happen in action 2. The goal position in action 2 varies because affected by the grasping results of action 1. A hard coded action 2 will miss the front side of the right handle or just grasp a wrong part of the plastic bag.
7

---

Figure 6: IIM processes in white and red plastic bags. Top row: side view and detected keypoint skeleton of plastic bags at each step. Middle row: top-down view and the predicted grasping points and directions. Bottom row: reconstructed 3D keypoint skeletons and the predicted actions each step. We can see at both initial states, handles interleave with each other, but the robot can finally adjust them to standing poses with iterative wock-and-place actions.
Figure 8: (a) Different initial configurations. (b) Spe- cial initial configurations that support directly lifting. (c) Failure cases. The filler is thrown out.
Figure 7: All task progresses along with grasping steps.
4.3 Qualitative Results
We show the qualitative results of white and red plastic bags in Figure and all task progress curves evaluated by G (mean and std) in Figure 7. IIM shows great generalization ability adjust plastic bags from different initial states to the standing poses. Most IIM processes finish in 4 steps. Some of them have more steps finish because of harder initial configurations: one handle too closer to the surface below it, or two handles interweave. In this situation the robot may grasp up more parts than one handle, but after several pick-and-place, the interlaced part can disentangle from each other and subsequent graspings become normal, as shown in the red plastic bag in Figure 6.
The inaccuracy of depth estimation is the main reason that causes failed grasping. Some graspings fail to catch the bag (above the right grasping point). Others exceed the right grasping depth. The first
Failure Cases:
will make the robot stuck in loop. The second case will make the robot grasp the body of the plastic bag directly. This may lead to a violent jolt when placing the plastic bag. which almost equals starting all over again. Sometimes the filler items fall out, which leads to a failed grasping, as shown in Figure 8(c). Most of the attempts can finish in 5 grasps, but some attempts (around 15%) need more steps due the above reason, and 13.3% of attempts fail.
5 Discussion and Future Works
In this work we propose Iterative Interactive Modeling for knotting plastic bags that are randomly dropped from the air with a dual arm robot. We show the effectiveness various visual learning methods such spatial-action maps and keypoint detection models on plastic bags. IIM is a gen- eral type of interactive perception [52] for modeling complex elasto-plastic objects: interacting with explicit representation model. Modules in our method can be replaced by
objects to complete other specific methods for different objects. For example, the representation model (keypoint skele- ton in this work) can be dense descriptors [8, 27, 28] partical-basee graphs [13, 14, 36), and the completion algorithm (imitation learning in this work) can be graph-based completion algorithms.
The limitations of our work are: 1) The keypoint detector depends on our custom dataset PBPose, and IIM based on human demonstrations. These make our method lack quick extensibility to other objects and situations. 2) The assumption of this work is that the plastic bag can stand still after being adjusted. For some very soft/hard plastic bags, or some fillers that do not support stand- ing, our method would not be effective. 3) Our method cannot handle extremely difficult initial configurations such as the handles being pressed underneath the plastic bag. Future works may seek to employ more actions such as pushing to tackle these situations.

---

Acknowledgments
We would like to thank Tianren Zhang and Yizhou Jiang for their insightful comments of the whole work, and Qualcomm China WRD UR Program. This work was supported by the National Natural Science Foundation of China 62176133.
References
[1] A. Nair, D. Chen, P. Agrawal, P. Isola, Abbeel, J. Malik, and S. Levine. Combining self- supervised learning and imitation for vision-based rope manipulation. In 2017 IEEE interna- lional conference on robotics and automation (ICRA), pages 2146-2153. IEEE, 2017.
[2] S. Kudoh, T. Gomi, R. Katano, T. Tomizawa, and T. Suehiro. In-air knotting of rope by a dual- arm multi-finger robot. In 2015 IEFE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 6202-6207. IEFE, 2015.
[3] H. Ha and S. Song. Flingbot: cloth unfolding. Conference on Robot Learning, pages 24-33. PMLR, 2022.
The unreasonable effectiveness of dynamic manipulation for
[4] R. Lee, D. Ward, A. Cosgun, V. Dasagi, P. Corke, and Leitner. Learning arbitrary-goal fabric folding with one hour of real robot experience. arXiv preprint arXiv:2010.03200 2020.
[5] A. Namiki and S. Yokosawa. Robotic origami folding with dynamic motion primitives. In 2015 IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3623-5628 IEEE, 2015.
[6] S. Duenser, J. M. Bern, R. Poranne, and S. Coros. Interactive robotic manipulation of elastic objects. In 2018 IEEE/RS/ International Conference Intelligent Robots and Systems (IROS), pages 3476-3481. IEEE, 2018.
[7] B. Shen, 7. Jiang, C. Choy, I., J. Guibas, S. Savarese, A. Anandkumar, and Y. Zhu. Acid: Action-conditional implicit visual dynamics for deformable object manipulation. arXiv preprint arXiv:2203.06886, 2022.
[8] P. R. Florence, I.. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. arXiv preprint arXiv: 1806. 08756, 2018.
[9] K. Suzuki, M. Kanamura, Y. Suga, H. Mori, and T. Ogata. In-air knotting of rope using dual- arm robot based deep learning. In 202/ JEEE/SS International Conference Intelligent Robots and Systems (IROS), pages 6731- IEEE, 2021.
[10] A. Wang, T. Kurutach, K. Liu, P. Abbeel, and A. Tamar. Learning robotic manipulation through visual planning and acting. arXiv preprint arXiv: 1905.04411 2019.
[11] Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel. Learning to manipulate deformable objects without demonstrations. arXiv preprint arXiv:1910.13433 2019.
[12] X. Lin, Y. Wang, Z. Huang, and D. Held. Learning visible connectivity dynamics for cloth smoothing. In Conference on Robot Learning, pages 256-266. PMLR, 2022.
[13] H. Shi, H. Xu, Z. Huang, Y. Li, and J. Wu. Robocraft: Learning see, simulate, and shape clasto-plastic objects with graph networks. arXiv preprint arXiv:2205.0200 02990, 2022.
[14] Y. Li, J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba. Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids. arXiv preprint arXiv:1810.0156. 2018.
[15] 7. Xu, Z. He, Wu, and S. Song. Learning 3d dynamic scene representations for robot manipulation. arXiv preprint 2xXiv:2011.01968 2020.
[16] I.. Manuelli, W. Gao, P. Florence, and R. Tedrake. kpam: Keypoint affordances for category- level robotic manipulation. arXiv preprint arXiv: 1903. 06684, 2019.

---

[17] A. Zeng, S. Song, J. Lee, A. Rodriguez, and T. Funkhouser. Tossingbot: Learning to throw arbitrary objects with residual physics. IEEE Transactions on Robotics, 3674)11307-1319, 2020.
[18] A. Zeng, S. Song. S. Welker, J. Lee, A. Rodriguez, and T. Funkhouser. Learning synergies between pushing and grasping with pervised deep reinforcement learning. In 2018 IEEF/RSI International Conference on Intelligent Robots and Systems (IROS), pages 4238 4245. IEEE, 2018.
[19] Smolen and A. Patriciu. Deformation planning for robotic soft tissue manipulation. In 2009 Second International Conferences on Advances in Computer-Humal Interactions, pages 199-20 199-204. IEEE, 2009.
[20] S. Luo, J. Bimbo, R. Dahiya, and H. Liu. Robotic tactile perception of object properties: A review, Mechatronics, 48:54-67, 2017.
[21] J. Wu, X. Sun, A. Zeng, S. Song, J. Lee, S. Rusinkiewicz, and T. Funkhouser. Spatial action maps for mobile manipulation. arXiv preprint arXiv:2004.099 09/40, 2020.
[22] J. Li, S. Bian, A. Zeng. C. Wang, B. Pang, W. Liu, and C. Lu. Human pose regression with residual log-likelihood estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1025-11034, 2021.
[23] Maitin- Shepard, M. Cusumano- Towner, J. Lei, and P. Abbeel. Cloth grasp point detection based on multiple-view geometric cues with application to robotic towel folding. In 2010 IEEE International Conference on Robotics and Automation, pages 2308-2315, IEEE, 2010.
[24] D. Seita, N. Jamali, M. Laskey, R. Berenstein, A. K. Tanwani, P. Baskaran, S. lba, J. Canny, and K. Goldberg. Robot bed-making making: Deep transfer learning using depth sensing deformable fabric. arXiv preprint arXiv: 809.09810, 26, 2018
[25] S. Miller, J. Van Den Berg, M. Fritz, T. Darrell, K. Goldberg, and Abbeel. A geometric approach to robotic laundry folding. The International Journal of Roboties Research, 31(2): 249-267, 2012.
[26] T. D. Kulkarni, A. Gupta, C. Jonescu, S. Borgeaud, M. Reynolds, A. Zisserman, and V. Mnih. Unsupervised learning of object keypoints for perception and control. Advances neural information processing systems, 32, 2019.
[27] A. Ganapathi, P. Sundaresan, Thananjeyan, A. Balakrishna, D. Seita, J. Grannen, M. Hwang, R. Hoque, J. E. Gonzalez, N. Jamali, et al. Learning dense visual correspondences in simulation to smooth and fold real fabrics. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 11515-11522. IEEE, 2021.
[28] Florence, L. Manuelli, and R. Tedrake. Self-supervised correspondence in visuomotor policy learning. IEEE Robotics and Automation Letters, 5(2):492-449, 2019.
[29] L. Yen-Chen, Florence, J. T. Barron, T.-Y. Lin, A. Rodriguez, and P. Isola. Nerf- supervision: Learning dense object descriptors from neural radiance fields. arXiv preprint arXiv:2203.0191 2022.
[30] C. B. Choy, D. Xu, J. Gwak, K. Chen, and S. Savarese. 3d-r2n2: A unified approach for single and milti-view 3d object reconstruction. European conference on computer vision, pages Springer, 2016.
[31] T. Groueix, M. Fisher, V. G. Kim, B. C. Russell, and M. Aubry. A papier-máché approach to learning 3d surface generation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 216-224, 2018.
[32] B. Deng, K. Genova, S. Yazdani, S. Bouaziz, G. Hinton, and A. Tagliasacchii. Cvxnet: Learn- able convex decomposition. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 31-44, 2020.
10

---

[33] C. Jiang, J. Huang, A. Tagliasacchi, L. Guibas, et al. Shapeflow: Learnable deformations among 3d shapes. arXiv preprint arXiv:2006.07966 07982, 2020.
[34] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Martin-Brualla, Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865 (065-587,200 2021.
[35] D. Mrowca, C. Zhuang, E. Wang, N. Haber, L. F. Fei-Fei, J. Tenenbaum, and D. L. Yamins. Flexible neural representation for physics prediction. Advances in neural information process- ing systems, 31, 2018.
[36] Y. Li, T. Lin, K. Yi, D. Bear, D. Yamins, J. Wu, J. Tenenbaum, and A. Torralba. Visual grounding of learned physical models. In International conference on machine learning, pages 5927-5936. PMLR, 2020.
[37] Y. Hu, L. Anderson, T.-M. Li, Q. Sun, N. Carr, J. Ragan-Kelley, and F. Durand. Difftaichi: Differentiable programming for physical simulation. arXiv preprint arXiv:1910.00935 2019.
[38] Holl, V. Koltun, and N. Thuerey. Learning to control pdes with differentiable physics. arXiv preprint arXiv:2001.07457, 2020.
[39] Y. Li, S. Li, V. Sitzmann, Agrawal, and A. Torralba. 3d neural scene representations for visuomotor control. In Conference on Robot Learning, pages 1 112-123. PMLR, 2022.
[40] B. Frank, C. Stachniss, N. Abdo, and W. Burgard. Efficient motion planning for manipulation robots in environments with deformable objects. In 2011 JEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2180-2185. IEEE, 2011.
[41] M. Saha and P. Isto. Manipulation planning for deformable linear objects. IEEE Transactions on Robotics, 23(6):114-11150, 2007.
[42] M. Moll and I.. E. Kavraki. Path planning for deformable linear objects. IEEE Transactions on Robotics, 22(4):625-636, 2006.
[43] D. McConachie, M. Ruan, and D. Berenson. Interleaving planning and control for deformable object manipulation. In Robotics Research, pages 1019-1036. Springer, 2020.
[44] D. Berenson. Manipulation of deformable objects without modeling and simulating deforma- tion. 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 4525 4532. IEEE, 2013.
[45] Z. Hu, P. Sun, and J. Pan. Three-dimensional deformable object manipulation using fast online gaussian process regression. IEEE Robotics and Automation Letters, (12):979-986.1 2018
[46] C. Wang, S. Wang, B. Romero, F. Veiga, and E. Adelson. Swingbot: Learning physical fea- tures from in-hand tactile exploration for dynamic swing-up manipulation. In 2020 JEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 5633-5640 IEEE, 2020.
[47] V. Chu, I. McMahon, L. Riano, C. G. McDonald, Q. He, M. Perez-Tejada, M. Arrigo, T. Darrell, and K. J. Kuchenbecker. Robotic learning of haptic adjectives through physical interaction. Robotics and Autonomous Systems, 63:279-292, 2015.
[48] A. X. Lee, A. Gupta, H. Lu, S. Levine, and P. Abbeel. Learning from multiple demonstrations using trajectory-aware non-rigid registration with applications to deformable object manipula- tion. In 2015 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5265 5272. IEEE, 2015.
[49] C. Matl and R. Bajesy. Deformable elasto-plastic object shaping using an elastic hand and basel-based reinforcement learning. In 2021 IEEE/RSJ International Conference on Intelli- gent Robots and Systems (IROS), pages 3955-3962. IEEE, 2021.
[50] Matas, S. James, and A. Davison. Sim-to-real reinforcement learning for deformable object manipulation. In Conference on Robot Learning, pages 734-743. PMLR, 2018.
11

---

[51] J. L. Schonberger and J.-M. Frahm. sructure-fromt from-rroiinnn motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1104-4113, 2016.
[52] J. Bohg, K. Hausman, B. Sankaran, O Brock, D. Kragic, S. Schaal, and G. Sukhatme. Inter- active perception: Leveraging action in perception and perception in action. IEEE Transactions on Robotics, 33(6):1273-122) 2017.
12


Reviews:
[{"review_id": "HP4ohZtuGBOf", "reviewer": "Reviewer_5bQ2", "summary": "This paper tackles a novel and interesting task: knotting plastic bags. The proposed method, Iterative Interactive Modeling (IIM), contains keypoint detection, single/dual arm pick and place, and knotting primitive. Finally, the method is trained and evaluated in the real-world setup.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Knotting plastic bags is a novel task and it is very common in our daily life.\n- Although there are many modules in the proposed method (IIM), each of them is carefully chosen and the whole pipeline is well designed. I really appreciate this system design.\n- Keypoint is a good choice for the intermediate representation. It is low-dimension, explicit, and able to contain almost all the necessary information to describe the bag state.\n- A lot of real-world experiments are executed to demonstrate the performance of this method.\n\nWeakness:\n- It will be better to describe each module in more detail. For example, what's the training supervision for spatial action map. The demonstration can only support positive data. Then how about negative data, all remaining pixels are negative?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Knotting plastic bags is challenging enough and very novel in robot learning community.\nAlthough no novel submodule are proposed in this paper, It takes advantage of each submodule and uses the appropriate intermediate representation (key points) to build a system very carefully.\nFInally, A lot of real-world experiments are executed.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- It will be better to include more details of each submodule. For example, how to train the spatial action map using demonstration.\n- The dual arm pick-and -place policy may be potentially improved. For example, FlingBot uses a scaled and rotated spatial action map to predict the dual-arm picking policy.\n- I just realized the picking policy is pretty straightforward: pick one handle. Key point prediction can already tell where are the handles, Is it possible to integrate this information into the pick-and-place policy."}}, {"review_id": "FAj7bscR1sU", "reviewer": "Reviewer_eMeZ", "summary": "The paper is about how to knot plastic bags. The high level idea is that estimating the state of plastic may not be possible at all times as a result the task is decomposed in to straightening up the bag first until the keypoint estimation becomes good and then solve the task based on the predicted keypoints on the plastic bag. The method is relying on a task progress module to decide which model to use and which phase of the task the robot is. The task progress module uses graph convolution network on the predicted keypoints along with their confidences to determine whether the robot needs to straighten up the plastic bags, approach the handles for knotting, and do the actual knotting. The model for straightening the bags is basically learned from demonstration data using action maps. For the final knotting, 2 planes are fitted from keypoints detected on each bag handle and that normal of that plane will be the approach direction of arms. \n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe task is important and challenging to tackle.\nExperiments are done on a dual-arm manipulator.\n\nWeaknesses:\nIMO, the method needs to be re-ordered. I think it flows better if it starts from task progress module and then go over all the other modules.\nI am not super clear on the actual knotting process. Is it open loop? It seems like the 3D keypoint detections are trained with only non-interactive data. Also, wrist mounted cameras basically don’t see much at close distance\nIt seems like learning from demonstrations work decently well for straightening up the plastic bug. Why not do it for the approach and knotting process? Why is the keypoint detection crucial for this task?\nThe method seems extremely engineered for this specific task.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think overall the problem is challenging and this paper will at least present a compelling baseline for future works to build on top of.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "See weaknesses above.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "58iif5Iwu__", "reviewer": "Reviewer_NmgN", "summary": "The paper describes a learning method for knotting plastic bags with a dual arm robot. The proposed iterative interactive modeling (IIM) method will first adjust the bad to a predefined-pose for better keypoints detection then perform few primitives to knot it. The paper combines GNN, 2D keypoints detection RLE model and motion primitives together to accomplish the knot task with 100 demonstrations for each plastic bag. Also, the work provides a 2D plastic bad keypoints dataset as well.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The paper addresses a very interesting and novel task -- plastic bag knotting. And the paper tries to shape the plastic bag to equilibrium pose for better perception and planning input instead of directly knotting the plastic bag from random initial configurations. \n2. The presentation for method is clear and straight forward. Figures clearly express the algorithm step by step.\n3. There are sufficient quantitative results, baseline comparison and real world execution videos look promising.\n\nWeaknesses:\n1. It seems to me the visual grasping module assumes the robot conducts 4DOF top-down picking and placing actions to shape the plastic bag. And the heatmap is predicted from top down images. However, what if the handle needs to be grasped from side? Similar to the Transporter Networks, is that possible to extend this module to 6DOF grasping?\n2. The performance on new grey bag drops a lot compared to the training result. Could you explain more about it? To me, the train and test objects are nearly the same except a little bit size, color different. However adding data augmentation to the CNN layers inside your modules should be able to handle those tiny changes and generalize well. It would be better if you could show the train/test loss curve to show the model is not overfitting. \n3. As you mentioned in the limitation part, the method assumes the bag needs to be hard enough to be able to stand still in the first phase. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper shows a novel solution of knotting bag task by first shaping the bag's pose then executing motion primitives. \nIt would be better if you could show how you collect the expert data in a video and some real execution heatmaps generated from the visual grasping module. As to the crappy depth issue from colmap, maybe structure light sensor is a way to get the better depth for the z location in picking. \nThough there are some limitations, concerns for the paper, I'd like to accept this paper. And I am looking forward the answers I described above during the rebuttal period and happy to discuss.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "N/A", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "2ZX0AWe24dW", "reviewer": "Reviewer_6XFW", "summary": "The authors proposed the Iterative Interactive Modeling (IIM) to first adjust the plastic bag to a standing pose with imitation learning to establish a high-confidence key point skeleton model. Directly knotting plastic bags from random initial states is difficult because of the complex initial configuration and intricate material and dynamics properties of plastic bags. To solve this problem, the authors used keypoint skeletons to obtain a visual representation from the complex shape of the plastic bag. The robot is trained to iteratively adjust the plastic bag to a standing pose to build a complete and high confidence keypoint model, then tie the knot with a set of learned motion primitives. The authors demonstrated that a plastic bag dropped randomly from the air can be tied together by a dual-armed robot using the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors robustly estimate the grasping point of a plastic bag, which is difficult to recognize, and two additional motion primitives enable Ian Knot in the air with a dual-armed robot. Despite being a very challenging task, it is wonderful to show that the task can be performed with high accuracy with only 100 demonstrations. The supplementary video and source code attached were helpful to the reader's understanding. After clearly stating the difficulty of recognizing plastic bags directly from 3D keypoints in a related work, the authors propose a method for solving the problem. The authors propose a method for generating 3D keypoints using 2D keypoints and multi-view geometry. The specific model structure is referenced in the source code, and the main body of the text describes the data collection conditions and training methods, allowing the reader to conduct additional experiments. Chapters 3.3 and 3.4 were similarly clearly written with issues and methods that the reviewer were able to understand well. The evaluation experiment describes the success rate of the task based on three evaluation metrics using four different comparison methods. In addition, the authors mention generalization performance using untrained plastic bags and failure cases, and the reviewers consider the experimental results are sufficient.\n\nMinor comments are as follows.\n\"A human expert demonstrates how to accomplish the adjusting process,\" but what exactly is being demonstrated?\nHow long does the task take to execute? It would be good to add the execution time to the text and display the playback speed (e.g. 4x) of the video.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The authors' research is high quality because they have performed a challenging task with high accuracy, and furthermore, the proposed method is novel and well described.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "The paper was clearly written and the reviewer pointed out only two minor corrections wrote in \"Strengths And Weaknesses\".\t", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)