Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes an image-based learning framework for object rearrangement. In particular, three major challenges are addressed: (1) Objects are closely located with each other. (2) The planner can determine which objects are subject to rearrangement. (3) The planner can resolve a situation when other objects occupy the goal locations of target objects. The proposed method is evaluated on simulations and real-world experiments and shows its promising performance in terms of success rate, accuracy, and efficiency.
strengths_and_weaknesses: Strengths:
- The proposed framework is novel, and it is convincing why this learning approach should work.
- The results from both simulation and experiments look promising.

Weaknesses (details are included in the Summary Of Recommendation and Issues sections):
- Contributions seem less significant as the justification of the fundamental limitations of the previous methods is not well addressed.
- Object sequencing has some critical limitations, which may fail in practice in environments containing many objects.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is well-written overall, and the presentation is clear. Although the evaluation results are promising, I am concerned that the contributions may not be significant enough compared to existing methods. The authors state that the previous methods cannot handle the challenges considered in this work as those papers do not treat these challenges. However, that does not mean those methods fail completely with clutter and swap. Since the authors did not evaluate those methods themselves, a more substantial justification is needed by providing insights into the fundamental limitations of those methods. A novel extension to the previous methods is selectivity, which I think is the weak component in the framework, but the same principle also seems to work for them. Moreover, the current setting of the problem having a bin that is not a part of the workspace makes selectivity not a very challenging task.
issues: Major comments:
- Contributions compared to existing methods: As discussed above, please explain in the introduction or related work section why previous methods cannot handle clutter and swap. Unless there are clear reasons, the capabilities of existing methods made in Tables 1 and 2 do not look fair. I believe this would enhance the contributions of this work considerably. 
- Object sequencing 1: An exponential number of subgraphs can be generated, prohibiting working with a large number of objects in practice. How long does it take to solve 20 objects in the evaluation? 
- Object sequencing 2: As pointed out in the limitations section, a straight line path is very naive. When many objects are located closely, motion planning must be used to avoid collisions. In this regard, the high task completion rates in Table 2 are surprising to me. Incorporating motion planning in the learning framework is not straightforward as it requires the notion of configuration space. 
- Object sequencing 3: It is unclear how object sequencing handles swaps in Section 3.3. Please explain this. Also, in Figure 1, it seems to clear non-target objects first. Is it always expected to occur?
- POMDP in Section 3.1: It is confusing how solving a POMDP is related to the framework. Is it referring to deep Q-learning, or are they two separate things?
- Section 3.2: The proposed approach does not seem to find a placement right behind an object as that area will be considered as occlusions. If it is incorrect, please explain this.
- In the evaluation, it is not intuitive what makes the proposed method handle clutter. Is it because of feature-based action selection that makes pixel-wise prediction?

Minor comments:
- Do NeRP and IFOR consider the same assumptions and inputs (e.g. both initial and goal images)? 
- I was not able to understand what the authors mean by rotating an image in Section 3.1. How do you maintain the resolution of a rotated image before feeding it into FCN?
- In Section 2, "changes in environment observability increase the difficulty of back-tracking and replanning": I don't understand the relationship between changes in environment observability and backtracking and replanning. Please clarify this. 
- In Section 3.3, what is "relabeling of a vertex" for?
- In Section 3.3, it is hard to understand how A* can be used to calculate the GED between graphs.
- In Figure 3, images are too dark to see. 
- In Figure 1, there are typos. (1-10) -> (1-11) and (11-15) -> (12-15).
- Typo in Line 223: "needs identify" -> "needs to identify".
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This work presents a novel method for tabletop object rearrangement amid clutter for cases where a subset of the objects may need to be removed from the scene. It works through sequencing the objects to manipulate through a measure of the graph edit distance from the current state to the goal state, where each object is manipulated by either a learned grasp or a learned push primitive (both implemented as fully convolutional Q-value maps), and where each grasped object is either moved away from the scene or placed in a location with the greatest feature correspondence to the goal scene. Extensive evaluation is done in simulation, and it is demonstrated that the learned policies can successfully transfer to a real robot.
strengths_and_weaknesses: Strengths:
* Clearly written and largely easy to follow
* The problem addressed is important to the field of robotics
* The proposed solution is novel in several respects (in particular the scene-graph edit based object sequencing and the depth-image cross correlation aspect of the feature-based repositioning). 
* The experiments are detailed and convincingly demonstrate the approach works well, and the video clearly demonstrates the approach can be adapted to work on a real robot
* The discussion of limitations is thorough 

Weaknesses:
* Several of the figures (1, 3, 4) have images and/or text that are so small that even zooming in does not make it easy to make out the details in the PDF, and that would be hard to understand in a printed version.
* I felt some of the text not covering the main novel ideas or results could have been cut or moved to allow for more results or analysis to be included in the main paper instead of the appendix. In particular, the stanard RL notation preliminaries are covered in section 3.1 and the summary of the Q-value map method (which as stated has a lot of overlap with [20]) to have been moved to the appendix or cut down. I feel including the planning step and ablation results in the main text would have been more informative.
* While the citation of the relevant prior work is well done, there is very little summary of it. I feel that for readers less familiar with the cited works would hard a hard time understanding which are the key novel contributions of this work and which parts are largely similar to prior work (such as the Q-value map). 
* While the ablation results for the action sequencing is informative, I was left wondering how well a simple hand-coded heuristic (such as moving each non-target object to the bin first, and then repositioning each object by Q-value) might perform. In general, it is not intuitive why the first set of actions would not be just removing all non-target objects from the scene, followed by planning to reposition the remaining objects to the target.
* The goal specification including an image of the final scene (needed for template matching) seems a bit limiting -- the method would be more powerful and widely applicable were it able to work with just a scene-graph based of the goal scene. 
* In some parts the paper can be a bit dense with extremely long paragraphs, and it seemed that it could have been edited a bit more to make reading easier.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: While I feel this paper could still be improved in various ways, I still think the clear presentation of ideas, novel approach to an important problem, and impressive results strongly merit it being accepted.
issues: See my summary of weaknesses - I feel the paper could still be improved with fairly substantial re-writing and iteration of the figures.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Selective Object Rearrangement in Clutter presented an image-based learned method for table-top object rearrangement in clutter using parallel jaw gripper. The method consists of three stages, including a graph-based object sequencing to select the next object to manipulate (using the current scene graph and goal scene graph's graph edit distance), feature based action selection to map image to robot actions (select between pushing or grasping), and a correspondence-based placement policy. Overall, authors claim that the method is the first to address concurrently object selection from initially cluttered scene, discarding the unrelated objects, and dealing with cases where goal location is occupied. Experiments show that the method is able to outperform SOTA methods in less restrictive settings, while also providing reasonable performance in more restrictive settings. 
strengths_and_weaknesses: Strengths
* Real robot experiments: The authors demonstrated zero-shot transfer from simulation to real robot. Cool!
* Interesting choice of actions: Push & Grasp, where grasp is preferred, push designed to singulate objects in clutter.
* Experiments are well designed, including definition of a clutter coefficient. Quantitative results of experiments analyzed, and convincing justifications given for observed behavior.
* Outperforms existing SOTA methods.

Weaknesses
* Pre-defined assumptions: Pre-defined window crop size for visually defining grasped object, would be better to see some ablations on generalization to out-of-distribution objects, would the assumption still hold?
* Dependence on pre-trained methods: UOIS-NET-3D is used to provide set of object segmentation masks. How does the system react to errors in the predicted segmentation mask?
* Missing experiment: It appears the NeRP baseline has adequate performance when only Swap is handled. It would be interesting to see how the presented method works when only swap is handled as well.
* Can the authors comment on whether this is an online method, and what kind of computational resources is required to run the pipeline?
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper presented a novel comprehensive method tackling selective object rearrangement in clutter. Overall, the technical contributions are sound and clearly explained, with a full suite of experiments (simulation and real-robot). Through the reported results, it appears that the method outperforms existing baselines by a large margin, and is able to be scaled to real-world experiments zero-shot. There are a few points that I am hoping the authors can comment on / provide experiments for as outlined in the strengths & weaknesses section, including an evaluation of the method's dependence on pre-trained methods robustly performing, impact of pre-defined assumptions (i.e. fixed window crop size for template matching), and more careful comparison with NeRP.
issues: See strengths and weaknesses section.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a system for image-based object rearrangement in cluttered tabletop environments where given a goal image, the robot plans a sequence of grasp and pushing actions to achieve the configuration corresponding to the goal image. The presented approach has 3 components: (1) an object sequencing method, which constructs a scene graph and encourages the robot to manipulate objects which  result in a graph closer in topology to the scene graph corresponding to the goal image, (2) an action selection method, which selects a grasping or pushing action on sequenced objects based on their likelihood of success, and (3) a placement module which places objects such that they match their pose in the goal image. The action selection method and placement module are very similar to ideas from prior work, but the object sequencing method appears to be relatively novel. The constructed system is also impressive: the videos demonstrate the efficacy of the approach on robot hardware.
strengths_and_weaknesses: Strengths:

(1) The presented method is clearly described and for the most part all components of the system are easy to understand.
(2) The resulting system is deployed on physical robotic hardware and videos suggest that the proposed method is able to successfully rearrange configurations with up to 15 objects. This is a very challenging task and the effort put into deploying this system in hardware experiments is much appreciated.
(3) The graph based object sequencing method is very interesting and novel to the best of my knowledge. I appreciate the ablation in Section 4.2, which appears to demonstrate the utility of the approach as opposed to heuristics based only on grasp Q-values (such as in Mech Search by Danielczuk et al.)
(4) I appreciate the extensive evaluation of the proposed method on a number of different numbers of initial objects and goal objects in simulation.

Weaknesses:

(1) The novelty of the work with respect to prior work needs to be made a bit more clear. The paper claims that this is the first paper that allows the robot to selectively rearrange objects from image observations in clutter. While the authors claim that prior TAMP approaches that incorporate learning-based vision models [14, 15, 16, 17] struggle to scale to adversarial environments, the reason for this should be expanded in more depth, and preferably, the authors should experimentally show that prior methods which seem in principle capable of addressing the same problem setting are intractable for the problem settings considered in this work.

(2) The comparison to prior work in simulation experiments does not appear calibrated. The authors should implement prior methods in the same setup as their algorithm is evaluated in. Otherwise, the numbers in Table 2 do not seem to be a fair comparison, and it is hard to draw any meaningful conclusions about the relative performance of the presented method. It seems important to implement at least 2 of [14, 15, 16, 17, 24, 25] based on which the authors believe will be the strongest baselines in their experimental setup. 

(3) The discussion of the physical experiment results is insufficient in the main body of the paper. It is important the paper discusses quantitive results such as the number of trials, success rate, etc... of these experiments for the reader to be able to evaluate the significance of the results. It is also preferable to compare to prior methods in physical experiments, but it is OK to omit this if sufficient comparison is at least provided in simulation.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper presents an impressive robotic system for object rearrangement with convincing evaluation on physical hardware. However, the experimental evaluation with respective to prior work and reporting of results in physical experiments need substantial improvement to fully evaluate the contributions of the paper. I am happy to update my recommendation if these issues are sufficiently addressed, but I am currently learning towards rejection for these reasons.
issues: Major issues:

1) Please clearly describe in more detail why prior TAMP methods with learning-based vision models [14, 15, 16, 17] would not scale to the scenarios evaluated in this paper and ideally provide experimental evidence for this argument. Without this, it is difficult to understand why the presented algorithm is needed with respect to prior existing work.

2) It would be good to implement some prior algorithms as discussed in the weaknesses above in simulation to provide a calibrated comparison to prior work. Currently, there does not seem to be concrete experimental evidence that the proposed approach provides significant benefit over prior approaches.

3) Please report full quantitive details of physical experiments as indicated in the weaknesses section above.

More minor issues, but which would benefit from clarification:

4) Why are pushes considered successful based on pixel change in the image? If the purpose of a push is to increase grasp access, shouldn't pushes be considered successful based on the degree to which the increase grasp Q-values?

5) The grasp success criteria is a bit strange. Just because the antipodal distance between parallel jaws is larger than a predefined threshold doesn't seem to necessarily suggest that the executed grasp will be stable when the object is lifted. More clarification on this point would be much appreciated.

6) The learning objective in Section 3.2 is somewhat confusing. Specifically, I am confused by how $\varphi_t^{\text{PLACE}}$ and $\phi_t^{\text{PLACE}}$ are related to each other. Given $\varphi_t^{\text{PLACE}}$ (which requires no learning to compute since a freezed Resnet backbone is used to featurize images), it seems like all you would need to do is place the object at the location where $\varphi_t^{\text{PLACE}}$ is the highest.

7) I am also confused by the learning objective in Section 3.3. The authors initially state that a pre-trained SimGNN is used to determine graph similarity. Given this, I am confused why any learning is necessary at all (and am thus confused by the loss definition $\mathcal{L}_s$: given this graph similarity measure, you have all the information you need to determine which object to sequence next.
review_title: null

Meta-review: metareview: The paper proposes a framework for object rearrangement in clutter given a goal image, using graph-based object sequencing, action selection (pushing or grasping), and a placement module. The reviews are mixed, and the current rating of the paper is: weak reject, strong accept, weak accept, weak reject. Reviewers agree that the object sequencing method is interesting, and that the real robot videos are impressive and compelling. However, reviewers also raise several valid questions and concerns including:

* Experiments do not appear to be calibrated: comparisons to prior work should be performed in the same experimental setup for an informative comparison.
* Clarification on novelty claims: why the proposed algorithm is better positioned to handle clutter and swap.

Authors are encouraged to respond to reviewers' comments and questions.

Updates:

Reviewers appreciate the authors' detailed responses, and agree that the presented ideas are interesting, with compelling real robot results on cluttered tabletop rearrangements that involve both translation and rotation. The idea of distinguishing pick and push for rearrangement also adds value to the community. The experiments and ablations in simulation provide helpful quantitative evidence to the efficacy of the approach. There are still a few outstanding concerns on whether the comparisons to prior methods are perfectly calibrated – the authors have clarified that to address this, they have tried to match the experimental setup from prior work as closely as possible: including same data input format, data generation process, and evaluation metrics. Given that the prior methods did not release code for the authors to run on their system, these experiments should suffice (effectively demonstrating the same task but with clear notable improvements).
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)