Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a mechanical search method in cluttered shelf environments where the object of interest is initially not visible. The system moves other objects in the scene through prehensile and pushing gripper actions to find and clear the path to the target object. The authors formulate the task into a model-based optimal control problem and come up with the existence function/graspability function, which works jointly as the cost function of the solver. MPC is used to find the best local robot action that fulfills the objective. The authors conduct experiments in both the simulation and the real-world robotics system. Promising results are presented to demonstrate the effectiveness of the proposed method.
review: Strengths:
1. The paper overall has a smooth logic flow and it is easy to follow. 
2. The problem setting that the object of interest is not visible initially is hard yet interesting and practical. In addition, the shelf environments bring more challenges to the problem like limited workspace, limited relocation space and severe occlusion, etc. This problem happens a lot in real-world scenarios if the agent can only observe the environment through a fixed camera. Solving this task can truly benefit real robotics applications.
3. The setting where all objects except the object of interest are initially unknown is a good practice as that's what happened in the real world, the agent is more likely to have only the observation and its target.
4. Since the object is not visible at first, the design of the existence function is smart. Also, the formulation of the model-based optimal control problem and the modification based on it makes good sense. The usage of MPC also fits well in this problem.
5. The authors provide most of the design details of the proposed method.
6. The real-world demos are comprehensive enough to demonstrate both successes and failures.

Weaknesses:
1. The authors claim that one of the challenges in the shelf environment is that the agent can only make observations from the front direction which makes the object of interest not observable in many cases. This is actually because of the setup in this paper (fixed camera) but not the nature of the shelf environment. If the robot has an in-hand camera that can observe the environment from different poses(although they are all at the front of the environment), I am sure that the red cylinder in the environment shown in Figures 1 and 4 can be seen even without relocating any objects. If this is the case, the system can directly figure out the objects in collision with the grasping path and relocate them according. Thus, the authors should include this camera setting in the introduction section to better motivate the proposed solution.
2. Claiming the object pose $x \in SE(3)$ is not accurate. First, since the target object is always placed on the surface, it is $(x, y)$ out of $(x, y, z)$. Then in both the figures and the video, the target object is always a red cylinder placed in a straight-up pose. I don't see there are any rotations included (at least no row and pitch, I cannot tell about the yaw because it does not matter for this object), which means it is at best $(yaw)$ out of $(row, pitch, yaw)$.  In all, the only thing that is certainly included in the calculation is a 2D point $(x,y)$. Although the authors did mention it as $\mathbb{R}^2$ in the experiment section, it creates some inconsistency.  
3. In the search-for-grasp method, there are two cases where $f_t(x) = 1$, 1. the target object is observed 2. the target object is still hidden but occluded by others. Since the overall objective of the problem is to retrieve the target object, whenever the target object is observed, the agent should directly aim to clear the path and get it out of the shelf. In the current method, since it just uses $f_t(x)$, the agent may still make efforts to perform other actions instead of directly clearing the path to the target object. Having a design that distinguishes between these two cases can benefit the method (minimizing $J_t$ until the objects show up, there is no need to keep doing it at this point, like the search-and-grasp method).
4. There are some robot-object collisions shown in the real demo video. Once some objects fell due to collision, it can cause problems that can potentially prevent the robot from achieving the objective.
5. The paper does not have a novel core learning-based module. The pushing dynamics model and the object recognition model are adapted from other works (not sure if there are any modifications introduced). The depth segmentation network is based on the DGCNN, not sure if there are any modifications introduced. Learning is of great usage in the pipeline but not the one of the main contributions.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The work is convincing and provides new ideas to solve the object grasping tasks in cluttered shelf environments by formulating it as a model-based optimal control problem. The proposed method is technically sound. Results from the simulation and the real-world robot demonstrate the effectiveness of the method. This work has enough contributions to the TAMP area in the robotics field. However, I think the novelty in the robot learning part is limited or not addressed enough in the current version of the paper. Thus, to the best of my knowledge, my recommendation is weak accept considering its contribution to the robotics field.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper tackles finding and grasping occluded objects on cluttered shelves using standard grippers. The key idea is representing target visibility and graspability as indicator functions and optimizing these via pushing and pick-place actions. 3D superquadric recognition enables estimating the functions from partial views. Model predictive control leverages the estimates to plan actions over a short horizon that reveal and enable grasping the target object. Experiments in simulation and real-world settings validate the proposed search-and-grasp and search-for-grasp methods can effectively retrieve objects in cluttered spaces comparable to oracle methods with full state knowledge. Key contributions are the visibility and graspability formulations, recognition-based estimation, and optimal control framework to enable mechanical search with standard grippers.
review: Originality: This is the first work I'm aware of that addresses mechanical search on cluttered shelves with standard grippers. The idea of using indicator functions to guide rearrangement actions is novel. Estimating these based on 3D recognition is also a new approach. The optimal control formulation using dynamics of learned indicator functions is unique.

Quality: The technical approach is sound, leveraging superquadric recognition, PyBullet physics, and model predictive control. The experiments evaluate performance in simulation and real-world settings. Limitations are properly acknowledged regarding gripper constraints and recognition errors.

Clarity: The paper is well-written and clearly conveys the key ideas, approach, and results. The problem is motivated early on, and technical sections provide sufficient details to understand the methodology.

Significance: This research tackles a difficult problem in robotic manipulation that arises in real-world conditions. The ability to mechanically search for and retrieve occluded objects in tight, cluttered spaces can enable many useful applications. The proposed methods advance the state of the art in using standard grippers rather than specialized tools.

Relevance: This work is relevant to the CoRL audience since it addresses a challenging robot learning problem involving mechanical search, interactive perception, and manipulation planning under uncertainty. The methods leverage learning for 3D recognition, dynamics modeling, and optimal control. Evaluation is extensive on both simulation and physical robots.

Limitations: As mentioned by the authors, the fixed grasping approach limits general applicability. Collision checking for the graspability function is also computationally expensive. Recognition errors can propagate. Larger-scale experiments would be useful to better characterize robustness. The superquadric representation also limits the object shapes. Only a quite limited variety of objects with limited complexity and poses is considered.

In summary, I believe this paper makes a good contribution in advancing mechanical search capabilities for retrieving occluded objects from cluttered shelves using standard grippers. The technical approach is novel, of good quality, and clearly conveyed.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper makes technically strong contributions in advancing mechanical search capabilities using standard grippers rather than specialized tools. The novel formulation using visibility and graspability indicator functions provides a unique optimal control approach to sequential object rearrangement for retrieving occluded targets. Extensive experiments demonstrate effective performance on cluttered shelves in both simulation and real robot settings. While the object shapes are limited and impact potential incremental, the key ideas show promise in addressing a challenging problem. Overall quality and technical merit justify acceptance, though real-world applicability would benefit from more extensive validation and generalization beyond basic geometries.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a system for finding and grasping occluded objects in shelves without specialized hardware beyond a parallel jaw gripper. The authors propose an objective function that takes into account two functions, $f(x)$ and $g(x)$, representing existence and graspability. They then use shooting methods to minimize this objective over the course of a rollout. There are two versions of the method considered, one where graspability is considered as part of the objective, and one where it is considered as a separate step after visibility is achieved. The system employs 3d object recognition methods based on recognizing implicitly defined superquadrics, to recognize the geometry of objects within a scene at each step in the real world.
review: This paper proposes a system for grasping occluded objects in shelves via rearrangement. I believe the paper has the following strengths:

- The paper’s methods and findings are presented in a clear and digestible manner.
- Physical experiments show the practicality of this system.
- Well-defined problem statement.
- Presents a principled approach to the problem of mechanical search for extracting a target object.

However, I have the following concerns about the proposed method and its fit for this conference.

- Given that 3D object recognition is the only learning-based component in this work, my feeling is that the paper’s emphasis should be more on the application of this method to mechanical search and what challenges & solutions are present in application to mechanical search. This is a good robotics paper, but I don’t know if it is suitable for CoRL given that the only learning is the application of an object detection model from prior work.
- The success rate demonstrated in the real world experiments is quite low for 5 and 6 objects. The failure cases include incorrect 3D recognition but do not elaborate further. Again, given that this is the only learning in the paper, it would be good to explain this further. In simulation, the success rate is much higher; why is this? Is object recognition in the real world harder for some reason?
- It is unclear to me what impact the superquadric-based object recognition method actually has. No ablations are provided to indicate whether simply segmenting objects and using that segmentation mask with some simple symmetry prior in 3d would give a sufficient shape to use for collision checking. Why does implicitly defining these objects’ shape via superquadrics help? Is there any indication that recognizing subtle variations in these objects’ shapes improves collision checking, etc. If not, isn’t the object recognition method overkill for the problem?

More minor points.

- Using the word “Verify” for experiments may signal some confirmation bias; better to use words like “evaluate” instead.
- Would be nice to have standard error or standard deviation in the table, or at least an explanation of what the distribution & variance for # of steps looks like.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think that the work has potential, but it would be good to see more novelty on the learning side for this problem beyond the object recognition, where it is unclear how useful the superquadric object detection method is. i.e. Which listed limitations could learning potentially help tackle?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a practical method for finding and grasping target objects on cluttered shelves using a standard robot gripper. The authors develop an updated mechanical search algorithm based on 3D reconstruction of the scene, which is used to estimate the existence function and graspability function of the target object. The authors formulate the problem as a model-based optimal control problem and provide algorithms for both search-and-grasp and search-for-grasp methods. Experimental results in both simulation and real-world environments demonstrate the effectiveness of the proposed method.
review: The paper leverages 3D reconstruction of the scene to estimate the existence function and graspability function, which enables the robot to find and grasp target objects in cluttered shelves without specialized tools. This is a practical and innovative approach that has not been explored extensively in previous research. The paper provides a well-structured framework for mechanical search and grasping, with specific algorithms for search-and-grasp and search-for-grasp methods. The formulation of the problem as a model-based optimal control problem is clear and logical. The experimental results in both simulation and real-world environments validate the effectiveness and real-world applicability of the proposed method. The comparison with cases where all object information is assumed to be known demonstrates that the proposed method performs similarly. 

Since the abstract mentions that existing works primarily focus on specialized tools with suction grippers, it would be interesting to see a comparison between these works and the proposed algorithm based on 3D reconstruction. Providing this contextual background will help the reader understand the novelty and originality of the proposed method. The paper lacks sufficient implementation details for reproducing the study. It would be helpful to provide more information on the specific techniques and algorithms used in the 3D reconstruction, estimation of the existence function and graspability function, and model-based optimal control. The paper could benefit from more thorough evaluation and ablation studies. It would be valuable to compare the proposed method with widely-known baselines in the field and conduct additional experiments to further analyze the performance and limitations of the method.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: More details on baselines and implementations are needed.
review_title: null

Meta-review: Reviewers felt that the paper tackles an interesting and challenging problem, with a reasonable and well-explained solution.  The biggest concern was that it is unclear whether Superquadrics are important for this framework and whether there are any quantifiable advantages over standard methods for object recognition.
Decision: Accept (Poster)