Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper describes a learning method for knotting plastic bags with a dual arm robot. The proposed iterative interactive modeling (IIM) method will first adjust the bad to a predefined-pose for better keypoints detection then perform few primitives to knot it. The paper combines GNN, 2D keypoints detection RLE model and motion primitives together to accomplish the knot task with 100 demonstrations for each plastic bag. Also, the work provides a 2D plastic bad keypoints dataset as well.
strengths_and_weaknesses: Strengths:
1. The paper addresses a very interesting and novel task -- plastic bag knotting. And the paper tries to shape the plastic bag to equilibrium pose for better perception and planning input instead of directly knotting the plastic bag from random initial configurations. 
2. The presentation for method is clear and straight forward. Figures clearly express the algorithm step by step.
3. There are sufficient quantitative results, baseline comparison and real world execution videos look promising.

Weaknesses:
1. It seems to me the visual grasping module assumes the robot conducts 4DOF top-down picking and placing actions to shape the plastic bag. And the heatmap is predicted from top down images. However, what if the handle needs to be grasped from side? Similar to the Transporter Networks, is that possible to extend this module to 6DOF grasping?
2. The performance on new grey bag drops a lot compared to the training result. Could you explain more about it? To me, the train and test objects are nearly the same except a little bit size, color different. However adding data augmentation to the CNN layers inside your modules should be able to handle those tiny changes and generalize well. It would be better if you could show the train/test loss curve to show the model is not overfitting. 
3. As you mentioned in the limitation part, the method assumes the bag needs to be hard enough to be able to stand still in the first phase. 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper shows a novel solution of knotting bag task by first shaping the bag's pose then executing motion primitives. 
It would be better if you could show how you collect the expert data in a video and some real execution heatmaps generated from the visual grasping module. As to the crappy depth issue from colmap, maybe structure light sensor is a way to get the better depth for the z location in picking. 
Though there are some limitations, concerns for the paper, I'd like to accept this paper. And I am looking forward the answers I described above during the rebuttal period and happy to discuss.

issues: N/A
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper tackles a novel and interesting task: knotting plastic bags. The proposed method, Iterative Interactive Modeling (IIM), contains keypoint detection, single/dual arm pick and place, and knotting primitive. Finally, the method is trained and evaluated in the real-world setup.
strengths_and_weaknesses: Strengths:
- Knotting plastic bags is a novel task and it is very common in our daily life.
- Although there are many modules in the proposed method (IIM), each of them is carefully chosen and the whole pipeline is well designed. I really appreciate this system design.
- Keypoint is a good choice for the intermediate representation. It is low-dimension, explicit, and able to contain almost all the necessary information to describe the bag state.
- A lot of real-world experiments are executed to demonstrate the performance of this method.

Weakness:
- It will be better to describe each module in more detail. For example, what's the training supervision for spatial action map. The demonstration can only support positive data. Then how about negative data, all remaining pixels are negative?

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Knotting plastic bags is challenging enough and very novel in robot learning community.
Although no novel submodule are proposed in this paper, It takes advantage of each submodule and uses the appropriate intermediate representation (key points) to build a system very carefully.
FInally, A lot of real-world experiments are executed.
issues: - It will be better to include more details of each submodule. For example, how to train the spatial action map using demonstration.
- The dual arm pick-and -place policy may be potentially improved. For example, FlingBot uses a scaled and rotated spatial action map to predict the dual-arm picking policy.
- I just realized the picking policy is pretty straightforward: pick one handle. Key point prediction can already tell where are the handles, Is it possible to integrate this information into the pick-and-place policy.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper is about how to knot plastic bags. The high level idea is that estimating the state of plastic may not be possible at all times as a result the task is decomposed in to straightening up the bag first until the keypoint estimation becomes good and then solve the task based on the predicted keypoints on the plastic bag. The method is relying on a task progress module to decide which model to use and which phase of the task the robot is. The task progress module uses graph convolution network on the predicted keypoints along with their confidences to determine whether the robot needs to straighten up the plastic bags, approach the handles for knotting, and do the actual knotting. The model for straightening the bags is basically learned from demonstration data using action maps. For the final knotting, 2 planes are fitted from keypoints detected on each bag handle and that normal of that plane will be the approach direction of arms. 

strengths_and_weaknesses: Strengths:
The task is important and challenging to tackle.
Experiments are done on a dual-arm manipulator.

Weaknesses:
IMO, the method needs to be re-ordered. I think it flows better if it starts from task progress module and then go over all the other modules.
I am not super clear on the actual knotting process. Is it open loop? It seems like the 3D keypoint detections are trained with only non-interactive data. Also, wrist mounted cameras basically don’t see much at close distance
It seems like learning from demonstrations work decently well for straightening up the plastic bug. Why not do it for the approach and knotting process? Why is the keypoint detection crucial for this task?
The method seems extremely engineered for this specific task.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think overall the problem is challenging and this paper will at least present a compelling baseline for future works to build on top of.
issues: See weaknesses above.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The authors proposed the Iterative Interactive Modeling (IIM) to first adjust the plastic bag to a standing pose with imitation learning to establish a high-confidence key point skeleton model. Directly knotting plastic bags from random initial states is difficult because of the complex initial configuration and intricate material and dynamics properties of plastic bags. To solve this problem, the authors used keypoint skeletons to obtain a visual representation from the complex shape of the plastic bag. The robot is trained to iteratively adjust the plastic bag to a standing pose to build a complete and high confidence keypoint model, then tie the knot with a set of learned motion primitives. The authors demonstrated that a plastic bag dropped randomly from the air can be tied together by a dual-armed robot using the proposed method.
strengths_and_weaknesses: The authors robustly estimate the grasping point of a plastic bag, which is difficult to recognize, and two additional motion primitives enable Ian Knot in the air with a dual-armed robot. Despite being a very challenging task, it is wonderful to show that the task can be performed with high accuracy with only 100 demonstrations. The supplementary video and source code attached were helpful to the reader's understanding. After clearly stating the difficulty of recognizing plastic bags directly from 3D keypoints in a related work, the authors propose a method for solving the problem. The authors propose a method for generating 3D keypoints using 2D keypoints and multi-view geometry. The specific model structure is referenced in the source code, and the main body of the text describes the data collection conditions and training methods, allowing the reader to conduct additional experiments. Chapters 3.3 and 3.4 were similarly clearly written with issues and methods that the reviewer were able to understand well. The evaluation experiment describes the success rate of the task based on three evaluation metrics using four different comparison methods. In addition, the authors mention generalization performance using untrained plastic bags and failure cases, and the reviewers consider the experimental results are sufficient.

Minor comments are as follows.
"A human expert demonstrates how to accomplish the adjusting process," but what exactly is being demonstrated?
How long does the task take to execute? It would be good to add the execution time to the text and display the playback speed (e.g. 4x) of the video.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The authors' research is high quality because they have performed a challenging task with high accuracy, and furthermore, the proposed method is novel and well described.
issues: The paper was clearly written and the reviewer pointed out only two minor corrections wrote in "Strengths And Weaknesses".
review_title: null

Meta-review: metareview: Reviewers felt positively that the task is interesting and novel.  The proposed method is reasonable, well-designed, and presented clearly.  The results appear to be very promising, and reviewers were happy to see so many real robot experiments.

Reviewers felt that their major concerns have been resolved during the rebuttal period. 

Reviewers also noted a few limitations of the paper: the proposed method has strong assumptions and restrictions on the types of bags that can be used. Reviewers also requested to see a video showing the expert data collection as well as some real execution heatmaps generated from the visual grasping module. 

Although the reviewers felt that the paper should likely be presented as a poster and not as an oral presentation, there was also some feeling that perhaps the paper could be nominated as “Best Systems Paper” due to the impressive results of the overall proposed system on a very challenging task.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)