Leveraging 3D Reconstruction for Mechanical Search on Cluttered Shelves
Authors: Seungyeon Kim
Young Hun Kim
Yonghyeon Lee
Frank C. Park
Venue: CoRL 2023 Poster (2023)
Keywords: Mechanical search
Object rearrangement
Prehensile and Non-prehensile manipulation

Abstract: Finding and grasping a target object on a cluttered shelf, especially when the target is occluded by other unknown objects and initially invisible, remains a significant challenge in robotic manipulation. While there have been advances in finding the target object by rearranging surrounding objects using specialized tools, developing algorithms that work with standard robot grippers remains an unresolved issue. In this paper, we introduce a novel framework for finding and grasping the target object using a standard gripper, employing pushing and pick and-place actions. To achieve this, we introduce two indicator functions: (i) an existence function, determining the potential presence of the target, and (ii) a graspability function, assessing the feasibility of grasping the identified target. We then formulate a model-based optimal control problem. The core component of our approach involves leveraging a 3D recognition model, enabling efficient estimation of the proposed indicator functions and their associated dynamics models. Our method succeeds in finding and grasping the target object using a standard robot gripper in both simulations and real-world settings. In particular, we demonstrate the adaptability and robustness of our method in the presence of noise in real-world vision sensor data. The code for our framework is available at https://github.com/seungyeon-k/Search-for-Grasp-public.

Leveraging 3D Reconstruction for Mechanical Search on Cluttered Shelves
Seungyeon Kim m
Young Hun Kim Yonghyeon Lee² Frank Chongwoo Park' 'Seoul National University, Koora Institute For Advanced Study (ksy, yhun) Groootics.... snu. ac.kr, ylee@kias. .r, fcpesnu. ac.kr
Abstract: Finding and grasping a target object on a cluttered shelf, especially when the target occluded by other unknown objects and initially invisible, re- mains significant challenge in robotic manipulation. While there have been advances in finding the target object by rearranging surrounding objects using specialized tools, developing algorithms that work with standard robot grippers remains an unresolved issue. In this paper, we introduce a novel framework for finding and grasping the target object using standard gripper, employing pushing and bick-and-place actions. To achieve this, we introduce two indicator functions: an existence function, determining the potential presence of the target, and (ii) graspability function, assessing the feasibility of grasping the identified target. We then formulate model-based optimal control problem. The core compo- nent of our approach involves leveraging 3D recognition model, enabling effi- cient estimation of the proposed indicator functions and their associated dynamics models. Our method succeeds finding and grasping the target object using a standard robot gripper in both simulations and real-works settings. particular, we demonstrate the adaptability and robustness of our method in the presence of noise in real-worke vision sensor data. The code for our framework is available https: / /gi thub. www.............eeeeeeee.eeeeee. wwon///enggeoon/////
for-Grasp--rrrrrr -ra---rrappppep public.
Keywords: Mechanical search, Object rearrangement, Prehensile and Non-
prehensile manipulation
Introduction
Finding and grasping a desired target object on a cluttered shelf where the target occluded by unknown objects and initially not visible vision sensor is a significant challenge in robotic manipulation. This task is further complicated when the pose of the vision sensor fixed. In such scenarios, the robot must rearrange surrounding objects to identify the target's pose and grasp it, all while avoiding collisions with the shelf and nearby objects. The geometric characteristics of the shelf, which allow visual observations solely from the front and limit the manipulator's 's workspace, add another layer of complexity.
Previous research on finding and grasping the target object, termed mechanical search, has focused on (i) simple environments, such flat tables bins [1, 3, 5, 6, 8), or (ii) complex shelf environments but with the assumption that all object shapes and poses are known [9, 10, 11. 12, 13, 14, 15]. Recent studies have ventured into more realistic shelf scenarios where the surrounding objects are unknown. However, these often depend on specialized, elongated tools to manipulate the objects within the tight shelf spaces [16, 17, 18]. The challenge of finding and grasping the target object using a standard robot gripper, devoid of specialized tools, remains unsolved. our knowledge, we are the first offer a practical method for mechanical search on a cluttered shelf using a standard robot gripper as shown in Figure 1.
Equal contribution
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Using a standard gripper introduces
t (not found)
1 (not found)
== (found)
new practical challenges that were not addressed by existing methods.
Recognition Observation
In shelf space densely populated with unknown objects, identifying collision-free spaces and generat- ing collision-free actions are crucial. However, when using standard grip- per, as opposed to specialized elon-
gated tools, accounting for collisions between the gripper, robot links, ob- jects, and shelves becomes much more challenging (e.g.. imagine sce- narios in which the robot is required to reach its arm deeper into the shelf). A distinguishing feature of our approach, which serves as the solution to the aforementioned challenge and sets it apart from existing methods, is the utilization of 3D reconstruction model.
Pick-and-place
Pushing
Target grasping
Figure A 3D recognition based mechanical search and grasping of the target object (red cylinder).
In this paper, we propose a novel optimal control framework for mechanical search with practical algorithms that leverage 3D reconstruction models. First, we introduce two indicator functions (denoting the target's candidate pose by SE(3)): (i) existence function f(z) that indicates if the target can be present at æ and (ii) a graspability function g(x)) that indicates if the target is graspable. The objective then becomes to rearrange surrounding objects until only one existable and graspable pose x' remains, i.c., there exists a unique x¹ E SE((3 such that f(x*) 1 and g(x") = 1, which leads to straightforward definition of a cost function using f and g.
Second, leverage 3D object recognition model to effectively estimate the functions S.g g and their corresponding dynamies models; therefore formulate tractable model- optimal con- trol problem. Specifically, we employ recent 3D recognition model rooted in superquadric prim- itives [19]. Notably, the superquadric representation allows for rapid collision checks, depth image rendering, and the utilization of pushing dynamics models [20], which leads to effective estimations of f and and their dynamics models. To mitigate accumulated estimation errors during optimal control, we adopt the model predictive control with a short time horizon.
Through experiments in both simulations and real-world scenarios, have validated the effective- ness of our 3D reconstruction- based approach. Our method is able to find and grasp the target object using a standard two-finger robot gripper, even in the presence of noise from vision sensor data in real-world settings.
2 Related Works
Mechanical search. Earlier studies have focused algorithmic approaches to finding the target object by manipulating surrounding objects through pushing or pick-and-plas actions [1, 2, 3]. De- spite their contributions, these methods make many environmental assumptions and they are com- putationally intensive and slow. Subsequent efforts have leveraged the POMDP model and solvers like DESPOT [4] or POMCP [5]. More recent studies have employed deep learning- -based modules, such as object segmentation and recognition networks, to enhance search effectiveness [6. 7]. An- other significant work uses a 3D shape recognition to predict occluded geometries from the vision sensor, leveraging this information enhance the search process
[8]. Our work aligns with [8] in leveraging the 3D recognition model to effectively solve the mechanical search problem. How- ever, we employ implicit object representation, offering various computational advantages for the mechanical search problem.
Mechanical search on shelves. Given that shelves are commonly used to store objects in home en- vironments or logistic warchouses, mechanical search shelves has emerged critical research area [16, 17, 18]. The inherent constraints of shelves introduce challenges, including potential col- lisions with the shelf, constraints object removal, and limited camera views. An extension of

---

the X-ray method [7] has been proposed to address these challenges [16]. Subsequent studies have introduced novel tools to augment the capabilities of robotic actions, such as stacking [17, 18). Our approach is distinct because it incorporates the graspability of the target object during mechanical searches. While previous studies often utilize specialized grippers, thereby eliminating the need to consider the graspability of the target. our use of standard grippers necessitates careful considera- tion of potential collisions during the grasping of the target object. This requires 3D scene reasoning, leading us to adopt 3D recognition model, which is departure from the 2D reasoning commonly employed in prior works. More detailed reviews on related works can be found in Appendix A.
A General Framework for Mechanical Search and Grasping
In this section, we propose general framework for mechanical search and provide two specific algorithms: (i) search-and-gue grasp and (ii) search-for- grasp. using the model-based optimal control formulation. The search-and-grann method executes actions first to find the target and then executes additional actions to rearrange the objects to make the target graspable, while the search-for-gee grasp method integrates the search and grasp processes into a cohesive framework so that the target object's graspability taken into account in the search phase.
Our focus lies specifically scenarios where the target object is known where its information is given as the color and geometry and is fully occluded by other unknown objects. We assume that the robot can interact with objects either by pushing or pick-and-place actions to rearrange the objects. When the robot performs the pick-and-place action, it is only allowed to place an object in another empty space on the shelf (i.e., object stacking is not allowed). For the target object, we assume that it is placed in a straight-up pose (i.e., not tilted) and not stacked on top of the other surrounding objects. Also, when the robot rearranges the surrounding non-target objects, the target object is assumed to remain stationary.
We denote the target object's pose by E SE(3): since the target object is not tilted and stacked, sufficient to represent the object pose as æ E SE(2). Instead of dealing with the continuous pose space SEE)), to simplify numerical computations, we re- strict our attention to finite subset X' SE(2). The two core components in our framework are (i) an ex- istence function f X > {0, }} and (ii) graspa- bility function X + {0, 1}. The existence func- tion f(x) indicates whether the target object can be present at the pose x or not. For example, given an observation shown in Figure 2 (Upper), consider two candidate target poses
Given Observation
Candidate poses
Rendered image
Target
f(x)
Camera
X (i.e., the red cylinders in Figure 2 Lower). For fxx to be 1, the rendered image must match the observation (there are other conditions as well, details are in the next section), otherwise, f(x) = 0. In practice, considering the discretization resolution of X, f (ar) considered to be if the target object pose can exist near æ. Natu- rally, it captures the uncertainty of the target object's pose because Enex fx) represents the number of possible object poses; the greater ExEx f(x), the more uncertain. We assume X' is sufficiently densely discretized so that Exex f(z) is lower bounded by (i.e., the target object must exist in at least one & E X).. The graspability function g(x) indicates whether the target object at the pose x is graspable or not.
f(x)
Figure 2: Illustration on the existence func- tion f()) Upper: Observation. Lower: Can- didate poses and hypothetical rendering re- sults.
Denote the existence and graspability functions at a discrete timestep by fe(x)) and (x( and pushing or pick-and-place action by ai We assume that the dynamics models for f((x)) and (x))) given and denoted by F and G such that (x+1(x)= == Flfe, (x) and (z) == a.))).
3

---

Additionally, we assume that the uncertainty of the target object pose is not increasing in t, i.e., Exex f((x)) 2 Exex (z)( regardless of a,- This non-increasing uncertainty assumption comes from the prior assumption that the target object does not move when an action is applied because is impossible for an object suddenly can exist where it could not originally exist.
Given initial fo and go and dynamics models F and G, we formulate search-for- grasp method as model-based optimal control. Specifically, the goal is to find a sequence of actions do..... 4T for fixed time T, so that fr(z") 1 for only one xx E X - if there are more than one, then we cannot specify at which x the target object exists - and that is graspable, i.e., (r((")())))))))) (x*)== = 1. For this purpose, we formulate the following model-based optimal control problem:
min
Σ + TEX
(1)
such that (+) = F(f,, (x) and It+1 (z) = G(g,, (z), and a is a hyperparameter, Mini- mizing the first term in (1) makes ft have only one such that fr()") ) 1 since the existence function is lower bounded by -, and minimizing the second term enforces gr(x") = 1. Without the second term, even though the target object becomes visible, there is guarantee that it will be graspable.
In practice, we prefer to perform a minimum number of actions needed for the task, rather than for a pre-determined fixed length T. We introduce a discount factor y E (0, 1] and propose modified version of the optimal control problem:
T Σ 7 t=1
min
Σ 1=0
1 1- -J)) -" JT +
J -Jo
where J. := Eaxx fe(x) + f(())(1- 9x()). ) As + 1, the new objective function converges to the original one in (1) since Tu constant. For y < 1, Tr for T are now taken into account, and thus minimizing (2) leads to accomplishing the task quickly. A 7 that is too small can make the problem difficult, so appropriate level of y needs to be found.
Additionally, we propose computationally lighter version, a search-and-grast method. In the cost term T, in (2), the computation cost gi (z) is very high since it requires the planning of multiple collision-free paths (as we explain in the next section more details). The function ge (1)) must be calculated for all æ such that (z)
thus as Σ. (x) bigger, the computation cost increases. In the search-and-ges -arcas method, we first find the target by using the first term only J. = En, (1r).. Then, after the target is found at z", use the second term J. 1- gx(") to rearrange the objects to make the target graspable. Since the target pose is already specified at the search phase, in the second stage, we need to compute the graspability function Jr only at one x", which significantly reduces the total computation time.
4 3D Object Recognition-basel Mechanical Search
In this section, as one practical way to implement our framework, we propose an object recognition- based approach. We assume that a scene contains multiple unknown objects and known target object, and the RGB-D camera's pose is fixed so only capture side of the scene (e.g., shelf strategy
containing objects from the front view). From the partial view information of the scene, is to first recognize the shapes and poses of the objects and then to use the recognition results to estimate the existence and graspability functions ff and g and the dynamics models F and G.
4.1 Object Recognition via Superquadries
In this paper, we employ the shape class called the superquadries, family of geometric shapes which represent diverse shapes ranging from boxes, cylinders, and ellipsoids to other symmetric complex shapes. The implicit function of a superquadric surface at the canonical pose (i.e., T

---

1x is the identity matrix) has the following form:
x
y
2
+
+
=
0,
(3)
Q2
a3
where 9 (a1, 02 03, e e2) E is the shape parameter.
Especially, 41, 02, d3 controls the sizes and e1. 62 controls the geometric shapes (some ex- amples are shown in Figure 3). Given pose of the object E SE(3), the implicit surface equation transformed accordingly. Recently, given a partial observation of object, model that predicts the 3d full shape of the object by using the superquadric representation has been proposed [19]; we use this model for 3D ob- ject recognition. We note that to efficiently rep- resent more complex 3D shapes, we can use latent variable models, e.g., using the autoen-
2.0
1.0
Cylinder
Ellipsoid
Bicone
0.2
Box
Octahedron
2.0 e
0.2
Figure Examples of superquadrics.
coder framework [21. 22, 23, 24, 25, 26]. We
include details about the model architectures and the training details in Appendix B.1.
4.2 Existence and Graspability Function Estimates f and
Given partial visual observation o, let set of estimated superquadric shape parameters and poses be denoted by {(q', T' Further, assume that an indicator variable (0, 1) indicates whether the target object is visible (i.e., c 1) not (i.e., 0) in the observation is available. If target is visible (i.e., 1), we assume that the target object's pose can be estimated accurately. Using these estimates and we obtain the estimates for the existence and graspability functions denoted by f(x; s, c) and s):;
The key idea is to locate the recognized superquadric objects in the simulation, as well as the target object hypothetical pose E X. If 1, the existence function f. since we know which x² the target object exist, is 1 only at x", i.e.. f'x=; 1) and at other E X. If O, (ar; s, = 0) is defined to be if (i) depth rendering results with and without the target object are identical and (ii) there is collision between the recognized objects, the environment, and the target object. Otherwise f(;; 8) = O. The graspability function 0(æ; s) is defined to be if we can find a collision-free grasping trajectory of the robot gripper, where all possible collisions between the robot arm, gripper, environment, and multiple objects should be taken into account. The depth rendering, collision checking, and grasp planning can be performed efficiently using 3D recognition results, and the details on the computation of the existence and graspability functions are given in Appendix B.2 and B.3, respectively.
4.3 Approximate Dynamics Models f and 9
Then, we construct the approximate dynamics models for fo and gr by using (i) the dynamics of se denoted by 8t+1 S(s,, 0+) where just transform the selected object for the pick-and-place action and use pre-trained SE({)-equivariant pushing dynamics model for pushing action [20] - and (ii) the function estimates and ğ. Since f takes the classification label c as an input, we need a dynamics model for cz. but it is hardly possible to know if the target will be visible or not in the future given an action a, We take a conservative strategy and assume that the visibility of the target object ž does not change at + i.e., ē = a. Then, for given Sa and ge at time approximate dynamics models are defined as follows:
(()) = ((())))))((())) f((z)f(;; &41)) a))x))))(x)) = 0(1)
(4)
5

---

where 8141 = S(st, ar) and ē++1 = The existence function F(fe, (x) is 1 if both ft
and 8411, are I, which guarantees the non-increasing uncertainty condition for F, i... Exex f((x) 2 E (x)..
4.4 Sampling-based Model Predictive Control
Given an initial observation Op and function estimates fo(=)) f(1; so, co), 90 = [(æ; so), by using the approximate dynamics models F, I. we can solve the optimal control problem in (2). Since both the function estimates and dynamics models have numerical errors, we perform the model predictive control (MPC) where we iterate the following procedure for T times: for o..... --1 (i) update initial estimates of ft: It from a new observation oz, (ii) solve (2) with short time horizon M T, and (iii) take only the first action de. Details for sampling-based model predictive control can be found in Appendix B.4.
5 Experiments
In this section, we conduct a comparative analysis of the two proposed methods, the search-and- grasp and search-for-grasp methods, and evaluate both methods show robust performance find and grasp the occluded target object in both simulation and real-world environments.
Environment. The simulation and real-world experiments share the same robot, the same camera, and the same shelf environment; the simulation experiments are conducted by the Pybullet simulator. We use the 7-dof Franka Emika Panda robot with parallel-jaw grip- per and Azure Kinect DK camera sensor looking into the shelf at fixed location (See Figure 4). The raw input visual observation from the camera an RGB-D, which is then converted point cloud data. In the simula- tion environment, we assume that the ground- truth segmentation masks of objects are avail- adopt
Robot environment
Objects including target
Simulation
Real-world
Figure 4: The left column shows the simulation and real environments, and in the right column, objects used in each environment are visualized. In particular, the target object is marked in red in the simulation; the red-taped can is the target ob- ject in the real experiment.
able; for real-world experiments, color-based heuristic segmentation method. We use syllnder-shaped and cube-shaped objects of various sizes which are visualized in Figure 4. More details on the environment are in Appendix C.I. Throughout the experiments, we restrict our attention to cylindrical target object. Considering the rotational symmetry of the target object and the assumption that the target object stands upright, the target pose space X' can be reduced to IR. We note that if the target object is box, the whole SE(2) space should be considered as the target pose space; the experimental results when the target object is box can be found in Appendix D.3.
Action space. The action space AA composed either pushing or pick-and sock-and-place actions. A pushing action is defined by (T push: d) E SE()) TR3 where Tush E SE(3) an initial gripper pose and d 3 displacement of the gripper with a fixed orientation. A pick-and placeplcce action is defined by (T (Tggssp Tplace) SE(3) SE()). which gripper poses when it grasps and places the object, respectively. If the gripper collides with other objects, it is judged to be invalid and is not sampled. For details on the actions and the action sampling method, we refer to Appendix B.5.
Evaluation metrics. We report the find- and grasp-success rates separately. If a sequence of our actions makes the target visible within 10-time steps, it is considered as a find-success. If makes the target not only visible but also graspable and we can find a collision-free path for taking the object within 10-time steps, it is considered grasp success. Additionally, measure the average number of actions required to find or grasp the target.

---

t=0
t=1
t=
t=
t= 4
t=
RGB
Depth
Recog.
Pushing
Pushing
Pushing
Pushing
Pick-and placee
Target grasping
Figure 5: An example trajectory simulation manipulation. Each column shows the camera input and action selection at each time step. In the simulation, surrounding objects are blue and the target object is red.
The number of objects
8
METHOD
Find
Grasp
Find
Grasp
Find
Grasp
Find
Grasp
Suce.
0.978
0.939
0.939
0.761
0.878
0.622
0.844
0.528
Search-anddGrray
Steps
1.392
1.562
2.178
2.635
2.589
3,473
2.664
3.947
Suce.
0.983
0.928
0.933
0.794
0.9
0.678
0.889
0,578
O-Scarch-for-Greet
Steps
1.407
1.647
2.077
2.769
2.377
3.574
2.831
4.125
Suce.
0.983
0,928
0,928
0.789
0.889
0,656
0.878
0.606
E-Scarch-and-- Grarch-and-Grasp
Steps
1.362
1.611
2.222
2.739
2.581
3,432
2.981
3.78
Suce.
0.972
0.922
0.933
0.756
0,922
0.672
0.894
0.656
R R-Search-for Search-forGrrasp Grasp
Steps
1.331
1.651
2.196
2.765
2.301
3.281
2.901
3,975
Table 1: Simulation manipulation results
5.1 Simulation Experiments Results
In this section, we evaluate our methods in the simulation experiments and empirically show that they can find and grasp the fully-occluded target object. To evaluate our method, we have created 180 scenarios for each number of surrounding objects in {2, 4, 6, 8), so total of 720 scenarios; for each scenario, a random selection (allowing duplicates) is made among the given objects in Figure 4. More details simulation experiment settings provided in Appendix C.2. We note that there some scenarios where the target object is not graspable even with the maximum number of actions, so the maximum average success rate is slightly lower than 1.
To evaluate our 3D reconstruction-uee based mechanical search method, and particular see how much recognition error affects the task performance, also test the cases where the ground-truth poses and shape parameters of the surrounding objects (not the target object) are available, and denote them oracle. In these cases, the the recognition module is not used and the ground-truth information of the surrounding objects are directly used for solving optimal control. We put the letter 'O' in front of the method name in the oracle experiments, and 'R' in the experiments using the 3D recognition (c.g., O-Search seirrchh-nn--Cae aatt---d-rrrppp Grasp and Search-for-Graas
Figure shows an example how our recognition based search-for-gres gorggssp method acts on the simu- lation experiment. The search-for-gue for-grasp method succeeds in finding the target object in four pushing actions and then makes the target object graspable by performing an additional pick-and-place ac- tion. The success rates and the average number of actions for finding and grasping the target are shown in Table 1. First, the performance differences between oracle and recognition are not signifi- cant, which means that the 3D recognition error does not significantly affect performance. Second, march-for-grasy has no difference in performance from search-and-gras ange--p when the number of sur- rounding objects is small, but shows better performance when the number is large. This suggests that considering graspability is of great help when finding the target. Further experimental results with more diverse scenarios are provided in Appendix D. and D.2.
7

---

5.2 Real-world Experiments Results
We adopt the R-Search-for-Grrap method for finding and grasping the desired target object in real-world en- vironment. Figure 6 shows real- world manipulation result. The target object is occluded by the two objects, and the target object is found through three pushing actions. The found tar- get object not graspable at = 3, an additional pushing action applied make the target object graspable'.
t= 0
t=1
t=2
t=3
t=4
RGB
Recog. Depth
Pushing
Pushing
Pushing
Pushing
Target grasping
Figure 6: Search-for-ee saarch-fo--grapp real-world manipulation results
Failure cases. Table 2 shows the manipulation success rates in real-world experiments. We design 5 test scenarios for each num- ber of objects, and used object configurations for each experi- ment are in Appendix C.3. few failure failure cases occur, es- pecially when the number of the surrounding objects increases. Most of the reasons for failure cases are that there's no solu- tion of rearranging the objects in our designed action space (see Appendix B.5) and (ii) that incorrect 3D recognition causes erro- neous updates of existence and graspability maps; for example. if the recognition model recognizes an object as inaccurately large, the existence map may be under- estimated, i.e., it decided that f(z) 0, but the target object can exist at x E SE(2).
Num.
Find
Grasp
3
5/5
5/5
4
5/5
4/5
5
4/5
2/5
6
3/5
2/5
Table 2: Search-for-grasp real-
world manipulation results.
5.3 Limitations and Future Directions
First, since our method considers single superquadric--haaped objects, it is trivial to apply directly more complex objects. One future direction to extend method is exploit the searches that attempt to represent the complex objects as multiple superquadrics [27, 19, 28, 29]. Even if each object is represented with several superquadrics, we can compute the existence function and the graspability function in the same manner. Second, our method highly depends on the 3D recognition model, inaccurate recognition can cause several problems. There can be robot-object collisions because we check the collision between the robot action and recognized objects (see Ap- pendix B). Moreover, some inappropriate action decisions can be made due erroneous existence map estimation. To overcome the limitation, additional information such RGB images should be utilized to enhance the recognition performance [30, 31]. Lastly, since the graspability of the object is checked using only a limited number of grasping trajectories (see Appendix B.3), there can be cases where it is determined that the object cannot be grasped even if there is grasping trajec- tory. As a research direction to overcome this, the graspability can be checked with more diverse trajectories quickly using network-based planner [32].
Conclusion
We have introduced novel mechanical search and grasping framework for cluttered shelves and developed practical algorithms named search-and-grash and search-for-grasp. utilizing pre-trained 3D object recognition model. Our approach uses pushing and pick-and-place actions to rearrange occluding objects, making the desired target object both visible and graspable. Unlike prior research which often relies on specialized elongated tools, we employ a standard two-finger gripper with robot actions including pushing and pick-and-place, and 6-DoF grasping. Our method achieves a 90-65% success rate in simulation experiments depending on the number of objects varying from to 8, and maintains comparable efficacy in real-world settings, even in the observation noise.
'The real-world manipulation videos can be found this Youtube link.

---

Acknowledgments
S. Kim, H. Kim, and C. Park were supported in part by SRRC NRF grant RS- RS-2023-002280552 IITP-MSIT grant 2021-0-02068 (SNU AT Innovation Hub), HITP-MSIT grant 2022-0-00480 (Train- ing and Inference Methods for Goal-Oriented AI Agents), KIAT grant P0020536 (HRD Program for Industrial Innovation), ATC+ MOTIE Technology Innovation Program grant 20008547, SNU-AIIS, SNU-IAMD, SNU BK21+ + Program in Mechanical Engineering, and SNU Institute for Engineering Research, Y. Lee was the beneficiary of an individual grant from CAINS supported by KIAS Indi- vidual Grant (AP092701) via the Center for and Natural Sciences Korea Institute for Advanced Study.
References
[1] I.. L.. Wong, I., P. Kaelbling. and T. Lozano-Pérez. Manipulation-baset active search for oc- cluded objects. In 2013 IEEE International Conference on Robotics and Automation, pages 2819. IEFE, 2013.
[2] M. Gupta, T. Rühr, M. Beetz, and Sukhatme. Interactive environment exploration in clutter. In 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages IEEE, 2013.
[3] M. R. Dogar, M. C. Koval, A. Tallavajhula, and S. S. Srinivasa. Object search by manipulation. Autonomous Robots, 36: 153-167, 2014.
[4] J. K. Li, D. Hsu, and W. S. Lee. Act to see and see to act: Pomdp planning for objects search in clutter. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5701-5707. IEEE, 2016.
[5] Y. Xiao, S. Katt, A. Pas, S. Chen, and C. Amato. Online planning for target object search in clutter under partial observability. In 2019 International Conference on Robotics and Au- tomation (ICRA), pages 8241-8244 IEEE, 2019.
[6] M. Danielczuk, A. Kurenkov, A. Balakrishna, M. Matl, D. Wang, R. Martin-Martin, A. Garg, S. Savarese, and K. Goldberg. Mechanical search: Multi-step retrieval target object oc- cluded by clutter. 2019 International Conference on Robotics and Automation (ICRA), pages 1614-1621. IEEE, 2019.
[7] M. Danielezuk, A. Angelova, V. Vanhoucke, and K. Goldberg. X-ray: Mechanical search for an occluded object by minimizing support of learned occupancy distributions. In 2020 JEEE/RS/ International Conference Intelligent Robots and Systems (IROS), pages 9577-9584. IEEE, 2020.
[8] A. Price, L. Jin, and D. Berenson. Inferring occluded geometry improves performance when retrieving an object from dense clutter. In Robotics Research: The 19th International Sympo- sium ISRR, pages 376-392. Springer, 2022.
[9] Lee, Y. Cho, C. Nam, Park, and C. Kim. Efficient obstacle rearrangement for object manipulation tasks in cluttered environments. In 2019 International Conference Robotics and Automation (ICRA), pages 183-189. IEEE, 2019.
[10] J. Lee, C. Nam, J. Park, and C. Kim. Tree search-based task and motion planning with pre- hensile and non non-preiin prehensee manipulation for obstacle rearrangement in clutter. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 2516-85222 IEEE, 2021.
[11] E. Vieira, D. Nakhimovich, K. Gao, Wang, J. Yu, and K. Bekris. Persistent homology for effective non-prehensile manipulation. In 2022 International Conference on Robotics and Automation (ICRA), pages 1918-1924. IEEE, 2022.

---

[12] S. Park, Y. Chai, S. Park, J. Park, K. Lee, and S. Choi. coni-muuonomous teleoperation via learning non-prehensile manipulation skills. In 2022 International Conference on Robotics and Automation (ICRA), pages
IEEE, 2022.
[13] C. Nam, J. Lee, Y. Cho, Lee, D. H. Kim, and C. Kim. Planning for target retrieval using a robotic manipulator cluttered and occluded environments. arXiv preprint arXiv: (ikiv:9900 Seei199077.999 2019.
[14] C. Nam, Lee, S. H. Cheong, Cho, and C. Kim. Fast and resilient manipulation plan- ning for target retrieval in clutter. In 2020 IEEE International Conference Robotics and Automation (ICRA), pages 3777-3783. IEEE, 2020.
[15] C. Nam, S. H. Cheong, J. Lee, D. H. Kim, and C. Kim. Fast and resilient manipulation planning for object retrieval in cluttered and confined environments. IEEE Transactions on Robotics, 37 (5): 15:3999-1555 2021.
[16] H. Huang, M. Dominguez- Kuhnne, V. Satish, M. Danielczuk, K. Sanders, J. Ichnowski, A. Lee, A. Angelova, V. Vanhoucke, and K. Goldberg. Mechanical search on shelves using lateral access x-ray. In 2021 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2045-2052. IEEE, 2021.
[17] H. Huang, M. Danielezuk, C. M. Kim, L. Fu, Z. Tam, J. Ichnowski, A. Angelova, B. Ichter, and K. Goldberg. Mechanical search on shelves using novel "bluction" tool. 2022 International Conference Robotics and Automation (ICRA), pages
IEEE, 2022.
[18] H. Huang, L. Fu, M. Danielezuk, C. Kim, Z. Tam, J. Ichnowski. A. Angelova, Ichter, and K. Goldberg. Mechanical search shelves with efficient stacking and destacking of objects. In Robotics Research, pages 205-221. Springer, 2023.
[19] S. Kim, T. Ahn, Y. Lee, J. Kim, M. Y. Wang, and F. C. Park. Dsqnet: deformable model- based supervised learning algorithm for grasping unknown occluded objects. IEEE Transac- tions on Automation Science and Engineering. 2022.
[20] S. Kim, B. Lim, Y. Lee, and F. C. Park. Se (2)-equivariant pushing dynamics models for tabletop object manipulations. In Conference on Robot Learning, pages 427-436. PMLR, 2023.
[21] Y. Lee, S. Kim, J. Choi, and Park. A statistical manifold framework for point cloud data. In International Conference on Machine Learning, pages
PMLR, 2022.
[22] Y. Lee and C. Park. On explicit curvature regularization in deep generative models. arXiv preprint rrXiv:2309.102333 2023.
[23] Y. Lee, Kwon, and F. Park. Neighborhood reconstructing autoencoders. Advances Neural Information Processing Systems, 34:536-546. 2021.
[24] Y. Lee, Yoon, Son, and F. Park. Regularized autoencoders for isometric representation learning, In International Conference on Learning Representations, 2021.
[25] C. Jang, Lee, Y.-K. Noh, and F. Park. Geometrically regularized autoencoders for non- euclidean data. In The Eleventh International Conference on Learning Representations, 2022.
[26] Y. Lee. A geometric perspective autoencoders. arXiv preprint arXiv:2309.082477 2023.
[27] D. Paschalidou, A. O. Ulusoy, and A. Geiger. Superquadries revisited: Learning 3d shape parsing beyond cuboids. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10344-10353, 2019.
10

---

[28] W. Liu, Y. Wu, S. Ruan, and G. S. Chirikjian. Robust and accurate superquadrie recovery: A probabilistic approach. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2685, 2022.
[29] W. Liu, Y. Wu, S. Ruan, and G. S. Chirikjian. serching-rrimitives: Shape abstraction from signed distance function. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8771-8780, 2023.
[30] C. Xie, Y. Xiang, A. Mousavian, and D. Fox. Unseen object instance segmentation for robotic environments. IEEE Transactions Robotics,
2021.
[31] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead,
A. C. Berg, W.-Y Lo, al. Segment anything. arXiv preprint arXiv:2304.0264. 2023.
[32] A. Fishman, A. Murali, C. Eppner; B. Peele, B. Boots, and D. Fox. Motion policy networks. In Conference on Robot Learning, pages 967-977. PMLR, 2023.
[33] Y. Ye and J. K. Tsotsos. Sensor planning for 3d object search. Computer Vision and Image Understanding, 73(2):145-166, 1999.
[34] K. Sjö, D. G. López, C. Paul, P. Jensfelt, and D. Kragic. Object search and localization for an indoor mobile robot. Journal of Computing and Information Technology, 17(1):67-80, 2009.
[35] T. Kollar and N. Roy. Utilizing object-object and object-scene context when planning find things. In 2009 IEEE International Conference Robotics and Automation, pages 2168- 2173. IEEE, 2009.
[36] Ma, T. H. Chung, and Burdick. A probabilistic framework for object search with 6-dof pose estimation. The International Journal of Robotics Research, 30(10):1209-1223: 2011.
[37] A. Aydemir, K. Sjöö, J. Folkesson, A. Pronobis, and P. Jensfelt. Search the real world: Active visual object search based on spatial relations. In 2011 IEEE International Conference on Robotics and Automation, pages 2818-2824. IEEE, 2011.
[38] M. Hanheide, C. Gretton, R. Dearden, N. Hawes, J. Wyatt, A. Pronobis, A. Aydemir, M. Göbelbecker, and H. Zender. Exploiting probabilistic knowledge under uncertain sensing for efficient robot behaviour. In IJCAI, pages 2442 2442-2449, 2011.
[39] A. Aydemir, M. Göbelbecker, A. Pronobis, K. Sjöö, and P. Jensfelt. Plan-based object search and exploration using semantic spatial knowledge in the real world. ECMR, pages 13-18. Citeseer, 2011.
[40] S. Gupta. J. Davidson, S. Levine, R. Sukthankar, and J. Malik. Cognitive mapping and planning for visual navigation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2616-2625, 2017.
[41] Y. Zhu, R. Mottaghi, E. Kolve, J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE interna- tional conference on robotics and automation (ICRA), pages 3357-3364. IEEE, 2017.
[42] A. Mousavian, A. Toshev, M. Fišer, J. Košecká, A. Wahid, and J. Davidson. Visual representa- tions for semantic target driven navigation. In 2019 International Conference on Robotics and Automation (ICRA), pages 8846-888 8852. IEEE, 2019.
[43] D. S. Chaplot, D. P. Gandhi, A. Gupta, and R. R. Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems, 33: 4247-4258, 2020.
[44] Y. Yang, H. Liang, and C. Choi. A deep learning approach to grasping the invisible. 1EEE Robotics and Automation Letters, 5(2):2232-24) 2239,, 2020.

---

[45] J. Varley, C. DeChant, A. Richardson, J. Ruales, and P. Allen. Shape completion enabled robotic grasping. 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 2442-2442 2447. IEEE, 2017.
[46] W. Yuan, T. Khot, D. Held, C. Mertz, and M. Hebert. Pen: Point completion network. In 2018 International Conference 3D Vision (3DV), pages 728-737, IEEE, 2018.
[47] H. Kato, Y. Ushiku, and T. Harada. Neural 3d mesh renderer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3907-3916, 2018.
[48] S. Liu, S. Saito, W. Chen, and H. Li. Learning to infer implicit surfaces without 3d supervision. Advances in Neural Information Processing Systems, 32, 2019.
[49] L. Mescheder, M. Occhsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning reconstruction in function space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4460-446 046004470. 2019.
[50] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. Deepsdf: Learning con- tinuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2019.
[51] M. Van der Merwe, Q. Lu, B. Sundaralingam, M. Matak, and T. Hermans. Learning continuous 3d reconstructions for geometrically aware grasping. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 11516-11522 IEBE, 2020.
[52] T. E. Boult and A. D. Gross. Recovery of superquadrics from depth information. In Proc. Workshop on Spatial Reasoning and Sentor Fusion, pages 128-137, 1987.
[53] A. Makhal, F. Thomas, and A. P. Gracia. Grasping unknown objects in clutter by superquadric representation. In 2018 Second IEEE International Conference on Robotic Computing (IRC), pages 292-299, IEEE, 2018.
[54] G. Vezzani, Pattacini, and L. Natale. grasping approach based on superquadric models. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 1579-1586. IEEE, 2017.
[55] G. Vezzani, U. Pattacini, G. Pasquale, and L. Natale. Improving superquadric modeling and grasping with prior on object shapes. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 6875-6882 IEEE, 2018.
[56] F. Solina and R. Bajesy. Recovery of parametric models from range images: The case for superquadrics with global deformations. IEEE transactions pattern analysis and machine intelligence, 131-):13 1990.
[57] Y. Lee, J. Baek, Y. M. Kim, and F. C. Park. Imat: The iterative medial axis transform. In Computer Graphics Forum, volume 40, pages 162-181. Wiley Online Library, 2021.
[58] Y. Wang. Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon. Dynamic graph cnn for learning on point clouds. Acm Transactions On Graphics (tog), 38(5):1-12, 2019.
[59] H. W. Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97, 1955.
[60] A. D. Gross and T. E. Boult. Error of fit measures for recovering parametric solids. In ICCV, 1988.
[61] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications the ACM, 65(1):99-106, 2021.
12

---

[62] D. Paschalidou, L. V. Gool, and A. Geiger. Learning unsupervised hierarchical part decompo- sition of 3d objects from a single rgh image. Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pages 1060-1070, 2020.
[63] I. Oblak, J. Sircelj, V. Strue, P. Peer, F. Solina, and A. Jaklič. Learning to predict superquadric parameters from depth images with explicit and implicit supervision. IEEE Access, 9: 9:1087- 1102, 2020.
[64] M. Gadelha, R. Wang, and S. Maji. Shape reconstruction using differentiable projections and deep priors. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 22-30, 2019.
13

---

Appendix
A Related Works
A.1 Mechanical Search on Shelves
Object search and mechanical search. The goal of object search (sometimes called as target object retrieval) is to find target object from unknown environments. Some works have focused on the active perception problem of making decisions of the sequence of camera poses to find a target object using a camera-mount methods have been proposed in terms of target-driven visual navigation [40, 41, 42, 43]. However, cluttered environment on tabletop or environment
mobile robot [33, 34, 35, 36, 37, 38, 39]; recently, deep learning-
in more complex environment, such where objects are placed on a shelf, it may be impossible find target object by controlling only the position of the camera. To solve these issues, interactive perception-based methods in which the robot can change the environment to find the target object- - have been proposed. Object search using interactive perception is recently called mechanical search.
Mechanical search methods. The earlier works have attempted to solve the problem of searching the target object via performing pushing or grasping actions to the surrounding objects in algorith- mic manners [1, 2, 3]. Although these methods have made significant contribution to the research topic of mechanical search, many assumptions are made in the environment to make the problem tractable, and they are generally computationally complex and therefore slow. To improve these methods (e.g.. relaxing the assumptions), several works have proposed a POMDP model and its solver (e-g., DESPOT [4] or POMCP [5]) for mechanical search. recent work provides a general- ized formulation of mechanical search and solve this problem effectively using deep learning-bass -based perception module (e.g., object segmentation and recognition network) and grasping module (e.g., pre-trained Dex-Net) [6]. The follow-up paper proposes novel perception module and policy that minimizes the support of learned occupancy distributions obtained from the perception, and claims that the proposed method outperforms the previous methods [7]. Another work propose a approach that predicts the occluded geometries from the vision sensor image and then utilize this information to efficiently find the target object [8]. Our work is also in the spirit of [8] utilizing the 3D shape recognition module to solve the mechanical search prob- lem efficiently (e.g., reduce the number of total actions), but we use implicit representation for the recognized objects utilize them for efficient and effective action decision (sce Appendix A.3).
3D shape recognition-
Mechanical search on shelves. As the shelves are often used to store the objects in home envi- ronments or logistic warehouses, mechanical search on shelves are being studied as an important research topic [16, 17, 18). Object manipulation on the shelves more challenging because of the several task constraints: the manipulator must not collide with the shelf, the objects cannot be re- moved from the shelf, and only a nearly-lateral camera view is available. These constraints limit the action space of the manipulator and the amount of visual information that can be obtained from the vision sensor. An earlier work proposes an extension of the previous method named lateral access X-ray [7] to solve laterally-accessiblee mechanical search [16]. The follow-up studies use novel tools to extend the robot action space from just pushing to pushing-and and--aaspiingggg [17] and stacking [18].
The main difference from these works is that the graspability of the target object is taken into con- sideration when performing mechanical search. The existing studies use custom long suction gripper specialized for mechanical search, so the graspability of the target object does not need to be considered separately. On the other hand, when using the standard robot gripper (e.g., parallel- jaw gripper), we need to find a trajectory that does not collide with the other surrounding objects and the shelf to grasp the target object. To find a non-collide trajectory, the 3D reasoning of the current scene inevitable for 6-DOF grasping, and accordingly, a 3D recognition model has been adopted in our method. The existing works only perform 2D reasoning for the scene since they do not have to consider the graspability, and the point that we perform mechanical search task adopting 3D recognition is also different from other works.
14

---

A.2 Object Rearrangement for Target Object Grasping
The object rearrangement generally refers to the problem of finding the feasible paths the objects that move the objects from their initial configuration to desired final configuration, and in fact, lot of various object rearrangement studies has been conducted; in this subsection, we only focus on the object rearrangement researches for grasping the target object. An earlier work propose algorithm to remove the surrounding objects using prehensile manipulation to grasp the target object without robot-object collisions [9]. Since the action space is limited only by prehensile manipulation, object rearrangement algorithms using non non- non-prehensile manipulation have also been conducted; for exam- ple, these algorithms are based on tree-search [10), persistent homology [11], and semi-autonommous tele-operation [12]. We note that unlike mechanical search, these papers assume that the information about the target object (and sometimes information about the environment) known. Other works focus on more general cases where the target object is possibly occluded [13, 14, 15]. If the target object is occluded, the proposed performs algorithm find the target similar to the mechanical search. It is worthy to note that our problem is more challenging since the surrounding objects can be removed in previous studies, but cannot in our case. Also, these studies first find the target object and then grasp it when the target occluded; we argue in this paper that finding a target object while simultaneously considering whether can be graspable is more efficient.
Grasping the invisible. It is valuable considered in [44]. Their work also considers the problem of grasping the target object while consid- ering the mechanical search problem. They named this problem grasping the invisible and introduce a deep learning- based end-to-end method, more specifically, a critic function that maps the visual observations to the expect rewards of robot pushing or grasping actions. This paper is the same in that it addresses the same problem as ours, but the proposed methods so far are limited to a specific environment and may require lot of data for the model to generalize other environments. We develop a method that can be applied in various environments by using object recognition, which known to be well generalizable to unseen scenes [19, 20), rather than end-to-end method.
that our problem setting is the closest to the problem
A.3 Shape Recognition-based Robot Manipulation
Numerous approaches have been proposed for the recognition of complete 3D shapes based on partial observations like depth images. Some of these methods employ explicit representations such as occupancy grid [45], point cloud [46], or mesh [47]. However, due to the limited reso- lution of these representations, they often result imprecise shape predictions. To address this issue, recent studies have explored the use of neural implicit functions to learn implicit 3D repre- sentations for objects [48, 49, 50, 51]. In our research, utilize superquadric functions, which strike balance between shape expressiveness, computational efficiency, and the number of pa- rameters required [52]. Superquadric functions have found applications in robotic manipulation tasks such as grasping [53, 54, 55]. Although we represent each object as a single superquadric function in our paper, our approach can be casily extended to encompass general implicit repre- sentations, particularly deformable superquadries [56, 19] or collection of superquadries [27]. To efficiently express more complex 3D shapes, we can use latent variable models, e.g., autoencoder framework [21, 22, 23, 24, 25, 26), with sufficiently low-dimensional latent vectors.
15

---

B Implementation Details for Our Methods
B.1 Details for Object Shape Recognition
The object shape recognition is an algorithm that takes a partial observation from (synthetic or real- world) RGB-D camera as input and outputs the 31) shapes of the objects in the scene. Especially, the input a partial point cloud of the scene obtained from a depth camera C IR (the RGB image only used to check whether the target object detected or not) and output the superquadric representations {q,, Te J=1 where qi the shape parameter, T, E SE(3) the pose, and N the number of the objects. To design model that performs this task, use the same method as in the previous work [20]. We first segment partially observed point cloud P into a set of object point clouds {P:),=1 and then convert each segmented point cloud P. to superquadric representation (T)).). The segmentation and superquadric recognition processes are based on neural network models. Each model trained from synthetic dataset obtained through simulation environment, and the trained models are directly applied to both the simulation environment and the real-world environment. The overall object shape recognition process described in Figure 7.
Partially observed point cloud
Segmented
Superquadric representation
Depth image
point cloud
Figure 7: Overall process for object shape recognition.
Point cloud processing. For partially observed point cloud P obtained from simulation environ- ment, to bridge the sim-to-real gap, noise is added to each point of the point cloud x E P using the map x + x mv where v is uniformly sampled on S and m is sampled from a Gaussian with zero-mean and standard deviation 0.001, as done in [57]. The all points corresponding to the shelf in the point cloud are removed and up/downsampled so that the number of points is 2048, i.e., P {x, E J== where n 2048. For partially observed point cloud P obtained from real- world environment, the shelf points
removed through the known shelf shape information and pose; since there's 's noise on shelf pose and observed point cloud, additionally remove the points corresponding to the floor of the shelf where the objects are placed through RANSAC plane fitting. Then as in the case of the simulation environment, the point cloud is up/downsampled so that the number of points is 2048.
Point cloud segmentation. After the point cloud is processed, it is separated into several object point clouds (P: via segmentation network. For the network architecture, use the same chitecture used in [58]. To train the segmentation network, first find bipartite matching between ground-truth and predicted segmentation labels using the Hungarian matching algorithm [59], and then define the loss function as the segmentation loss between the matched labels. This loss function invariant to the point permutation of prediction, so the network can learn segmentation labels and can be trained faster and more accurately accordingly.
Superquadric recognition. Each segmented object point cloud P, is then converted to the 3D full shape represented by superquadric qi E R and T, E SE(() via recognition network proposed in [19, 20J. For the network architecture, we use the same architecture used in [20]; the input representation a point cloud with 4-dimensional points P' = [x., E R for each point X,j, the first three components of x.y is equal to X, and the last element of each point is if Xij E P; and 0 otherwise for all 1...., n and output representation is (g:: T:). The input and output of the superquadric recognition model are described in Figure To train this recognition network, we
adopt the training loss function the difference between ground-truth and predicted object shapes. For the ground-trun truth shape, use the point cloud uniformly sampled from the surface of the object
16

---

- {x} E 1 where ny 512. Then we use the distances from the ground-truth point cloud to the predicted superquadric as the loss function. Especially, the distance form proposed in [60] used. The distance ő is defined as follows: for the superquadric surface equation expressed as
52
I
2
(tr, y1 v)
(5)
d3
the distance å between a point x E RR and superquadric surface S(x) 1 =0 defined by
6 6(xo S) ||x| S
where || denotes the Euclidean norm. Accordingly, the training loss function for the recognition network is defined as:
L 1 ng
Σ OT x),
where S is defined by the predicted superquadric parameters {a1: 02 03, E1, e2, and T is the predicted object pose.
We generate training dataset to train the above two networks, the segmen- tation network and the superquadric recognition network. To generate data, we randomly generate N ob- jects consisting of cubes and cylin- ders with various shape parameters (i.e., width, height, depth for cube, and radius, and height for the cylinder); the number of objects N varies from 2 to 8. The generated objects are then placed in random position and orientation on the shelf. After that, we construct a data tuple consisting of 3 components: (i) par- tially observed point cloud from depth camera, (ii) segmentation label for each point on P. and (ii) ground-truth point cloud (i.e., point cloud sampled from ground-truth shape) for each object po for 1, .... N. For each number of objects N, collect data until the numbers of the data tuples become 5000/100 100 for training/validatinn set, respectively. As result, the total numbers of the data tuples are 35000/700 for training/validation validation set, respectively. The validation set is used to select the best model for the segmentation network and the superquadric recognition network.
Segmented point cloud
Pi = (x1, E ER")"
P1 (x2) E RR)
Pk = (xx) E [R))
Superquadric recognition (21)
Superquadric representation
(g..T))
(92, T2)
(xx)TT)
Figure 8: Input and output representation of the su-
perquadric recognition model.
B.2 Details for Existence Function Estimate
In this section, we describe how to calculate the existence function estimate f. The function fa;; s, c) is defined by the recognized superquadric parameters & {(g, Tı) and the visibility of the target object c E {0, 13. The function's 's input hypothetical pose æ E X C SE(3) and output an indicator whether the target object can be present at the pose x (i.e., f(z;; 8,c) c) = 1) or not (i.e., fI;; know which =² the target object exist
0). The calculation of is trivial if c 1 since the existence function f
only at x², i.e., f X. So we consider only the case where 0, i.e., the target object is not visible. If 0, (x;88,) is defined to be if (i) depth rendering results with and without the target object at E X are identical and (ii) there is no collision between the recognized objects, the environment, and the target object. Otherwise f(x;; s) = O. We describe how to compute the above two conditions by taking advantage the superquadric as an implicit function.
and 0 other
Depth rendering condition. We check the depth rendering condition, i.e., calculate the function fa : X > {0, where fa(x)) if the depth rendering results with and without the target object at
17

---

Recognized objects
SQ function
Occupancy V
Visibility
Camera
Ray points
Figure 9: Overview of depth image rendering process from recognized superquadric functions.
A' are identical and fa( 0 otherwise. To calculate this, depth rendering function that takes the 3D object shapes input and outputs the corresponding depth image D E R is required; the intrinsic and extrinsic parameters of the camera used for depth rendering are known. In this paper, depth image can be rendered quickly using the superquadric implicit function. The overview of the depth rendering process from recognized superquadric functions described in Figure 9.
We first obtain the camera rays from the camera's intrinsic parameters, extrinsic parameters, and the resolution of the camera described in [61]. and we denote each ray's equation corresponding to [b., 4 + R$ where k 1...., H, i = 1,... W, and
the (k, 1)-th pixel of the depth image tu and ty are the near and far bounds to measure distance; in this paper, we set this values as 0 and 1.5, respectively. Each ray is a straight line rtt(t) = o + tdgt where o E RR3 the position of the camera pose and d&t E S is a direction vector of the ray. The camera rays are shown Figure 9.
obtain N superquadric parameters and their poses {(g, T.) We recall that the superquadric equation with the superquadric parameters qi is ex- pressed by
After recognizing the objects,
y/
2
S(x, ys 2;))= =
+
(8)
Q2
03
where = (41, 02 C1, 22). Then, we convert the parameters from recognition to implicit func- tions S,(x) = S (T; x: 4:) = for i I.....N. N.. Using the obtained implicit functions, we
calculate the signed distance-like superquadric function F as:
F F(x) = min (x)
Using this superquadric function, we can approximately calculate the occupancy function V(x) where V(x) = the point x is occupied and V(x) = O otherwise. The occupancy can be calculated using the equation
V(x) = (p(1 - (xx)))
(10)
where a the sigmoid function and p is scaling factor that adjusts the sharpness of V we set p = 1000 as proposed in [62, 63]. Then the visibility function Au((!) which indicates whether
point is visible the rendered camera image - on ray point ria(t) can be obtained
(t = exp(-r f.' ((Ta(()))
(11)
with large enough r - we set = 100 as proposed [64]. Finally, the depth value Dar on the (k, 1)-th pixel of the rendered depth image D is calculated
DEI = in + [" (ta(t)dt,
(12)
where = 1,.., H and = 1,.., W. The integrals above can be calculated numerically after uniformly dividing E [r].
The integration is required for all (k: 1) pairs obtain perfect depth image D, but is not necessary to calculate for all (k, pairs to check the depth rendering condition. Instead, we additionally
18

---

propose an algorithm that (i) finds pixels around the recognized object (i.e., S,'s) and the target object and (ii) calculates the depth value only for those pixels.
We additionally propose a modified method to calculate the depth rendering condition efficiently. The key idea is that since our region of interest is the objects the shelf, the above calculation will also be done only near the objects. The modified algorithm is based on the fact that arbitrary
superquadric (g. T can be contained in an ellipsoid E; in detail, for q (a1, 22 43, c e2), the ellipsoid equation E is given by
x²
y
2²
E( V.
+
(13)
3a
3a2
303
and the superquadric (g, 7) is contained in the ellipsoid E(T- S
Using this fact, the modified method (i) reduces the number of camera rays Fkt required for cal- culation, and (ii) efficiently samples the ray points ri((() required for calculation in each ray Tit- For the first one, for each ray's equation (t)(t)
+ tdla, we calculate whether Tke Tia(t) (t) meets the ellipsoids EXT S 1. If (t) does not meet any ellipsoid of the object, the depth value Du is set to Ey, and otherwise, D&z is calculated following (12). Therefore, we only need to consider the rays that meet at least one of the object ellipsoids. For the second one, instead of uniformly dividing [fn, ty] for numerical integration (11) and (12), we use more efficient method for dividing; we obtain the intersection points between the ray and the ellipsoid, and then uniformly divide the chord corresponding to the intersections. Since all of the above calculations exist in closed-form, they have little effect the amount of calculation.
Using the above depth rendering module, we finally obtain the function fa X - {0, 1} to check the depth rendering condition; especially, (z) = 1 if the MSE (Mean Squared Error) between the rendered depth images with and without the target object at E X' is lower than the threshold - we set this threshold as 0.001 and (=) = 0 otherwise.
Collision condition. We check the collision condition, i.e., calculate the function fc X + {0, 1) where (z) = 1 if there is no collision between the recognized objects, the environment, and the X' and fc(x) 0 otherwise. This function also can be quickly evaluated using the superquadric implicit function.
target object
We first obtain the target object point cloud sampled from the surface of the target object P. {xf E 1==1. where n. 512. Sim- ilar the case of depth rendering condition, also get the superquadric implicit functions S; for == 1,..., N from the recognized ob- jects. We note that a point x R3 is inside the i'th object S, when the value Sx(x) is less than I and otherwise outside. We utilize this fact check whether the target object collides with the recognized objects or not. The func- tion fo X + {0, }} is defined as:
Candidate poses
Collision checking
f,(x)
Target
f
Figure 10: Illustration on the collision condition fe(z). Candidate poses and collision checking re- sults.
fe(x) 1(min Sx(x)) > 1) f.j
(14)
where i = 1, ...,. N is the object index, j = 1, no is the point index of the target object point cloud, and 1()) is the indicator function. The calculation of the collision condition described in Figure 10.
Existence function estimate. Using the above functions to check the conditions, fa and far define the existence function estimate f X 10, as follows,
f(x) f(()))
In the other words, the existence function is at pose X if both the depth rendering condition and the collision condition holds and 0 otherwise.
19

---

Candidate grasp poses
Grasp trajectory
Recognized objects
Object
Object 2
Object
30cm
Object
Object
Object
Figure 11: The examples of the candidate grasp poses for various object shapes (Left) and the robot trajectory for selected grasp pose (right).
Computational cost. The calculation time takes 0.013 seconds in average where W H = 18, 564, " = 7, N = 4, and no = 512 with GeForce RTX 3090 and Gen Intel(R) Core(TM) i9- 1900K @ 3.50GHz. We design the existence function calculated batch-wise with respect to the target pose æ to accelerate the computation time for MPC.
B.3 Details for Graspability Function Estimate 9
In this section, we describe how to calculate the graspability function estimate 9. The function g(x; 8) is defined by the recognized superquadric parameters = {(y,, 7:)}. The function's 's input a hypothetical pose x E X SE(3) and output an indicator whether a collision free grasping trajectory of the robot gripper- all possible collisions between the robot arm, the robot gripper, the shelf, and the surrounding objects should be taken into account g(z; or not (i.c., g(z; 0). We then describe how compute the graspability function by also taking the advantage of the superquadrics as an implicit function similar to the calculation of the existence function.
be find the pose (i.e.,
Candidate grasping trajectories. To check the graspability, i.e., check whether collision- free grasping trajectory exists, generate can- didate grasping trajectories to grasp the target ob- ject. To achieve this, we first generate candidate grasp poses for the target object using a simple rule-based method as introduced in [20]. In this paper, we generate side grasp poses according to the shape and size of the superquadrie. The pairs of the antipodal points of the target objects are sampled since use two-finger gripper, and the grasp poses with distance of bigger than 7.5cm between antipodal points are rejected since the maximum gripper width of the Franka gripper is 8cm. To stably grasp the target object on the shelf, we tilted the grasp poses about 20 relative to the ground. Accordingly, we generate Ngr grasp poses (Tg")
Grasping trajectory
With link &
Collision checking
Collide
Figure 12: Illustration on the grasp trajectory collision detection. Candidate grasp trajectories and collision checking results.
Ngr === where Ngr between 10 and 30; the examples of the grasp poses for various ob- ject shapes shown in the left of Figure 11. Then, additionally have to plan the trajectory of the robot arm for each grasp pose. For grasp pose T E SE(3), we design robot trajectory where the gripper approaches about 30cm along the z-direction of the gripper frame as shown in the right of Figure 11. Finally, we get Ngr candidate grasping trajectories for the target object.
Grasp trajectory collision detection. After generating grasping trajectories, we should check whether the robot amm following the trajectory collides with the surrounding objects or the shelf. We first obtain the afterimage mesh of the gripper and Franka's links 7 and 6 when the robot fol-
20

---

lows the trajectory of the grasp pose Th and obtain the point cloud Pe = {xx E 1 sam- pled from the afterimage mesh, where or = 2048. We get the superquadric implicit functions (i = S (T; x; q:) = for = 1..., N from the recognized surrounding objects. We addi- tionally represent the shelf as the superquadric implicit functions; shelf can be represented by five boxes so five implicit functions (x)1 .... SN+5(x) are additionally considered. We recall from the collision condition of existence function that point R$ is inside the i'th object S; when the value (x) is less than and otherwise outside. The collision function d. X {0, 1) is defined
as:
(o(x)) (max min S((x)) 1) i.
(15)
= - 1 (max min (x <= 1)
(16)
where: 1, ,..., N 5 is the object index, j , ,... Aer is the point index of the target object point cloud, i = 1, ..., Ngr is the grasp pose index, and (-) is the indicator function. The calculation of the collision of the grasp trajectories is described in Figure 12. In practice, the modified collision detection function 9c.m X' + {0, we used is:
(x) min Σ (x <=
(17)
Graspability function. We define the graspability function estimate 9 X {0, 1) the same with gc, i.e.
g(z) Gc(x).
(18)
The modified graspability function estimate is the same with Ic,ms i.c.,
g ((x) 9c,m (x))
(19)
Computational cost. The calculation time takes 0.01 28 seconds in average, where N + 5 6. Pggr 2048, and Ngr 32 with GeForce RTX 3090 and Gen Intel(R) Core(TM) 19-11900K @
3.50GHz.
B.4 Details for Sampling-based Model Predictive Control
To solve the iterative optimization in MPC (where we set 10), we use M 3 and take sampling-based approach. The actions space A, whose elements either pushing pick-and- place actions, is defined based the object recognition results and a discrete set that can be efficiently searched through sampling (details included in Appendix B.5). We stop the MPC iteration if the target is visible and graspable, so the total number of actions can be less than T. The objective function is slightly modified so that, in J. = Ersx fr(x)) + fr(x) - (x(x)), the graspability function g, (x) which originally could take 0 or 1 now can take 1,0, -1, -2, -...... where g(x) = -n + 1 indicates that generated grasping trajectories for the target at æ collides with least n objects. This modified objective function provides more dense signals, making the optimization problem easier solve, while not changing the optimal solution.
B.5 Details for Action Sampling Manipulation
In this section, we describe how actions are sampled when performing mechanical search and pro- vide more rigorous information about actions, including pick-and-plad action and pushing action. The details of the pushing action and the pick-and-place place action including sampling processes and scene prediction are described Figure 13.
21

---

Recognized objects
Selected object
Sample action
Predicted scene
Pick-and-place Pushing
SQPD-Net
[21]
Transform
Figure 13: Sampling process and predicted scene after applying the action for pushing and pick- and-place actions.
Pushing action. A pushing action is defined by (Tush d) E SE(3) x Rs, where Twss E SE(3) an initial gripper pose and E R is a displacement of the gripper. To sample the initial gripper pose push. index i, from the recognized object indices i I..., N is selected and then direction (left or right) for push- ing the selected object i, is selected. Then Tush is defined from the pose and the shape parameters of the selected object (q..,Ti,) T,,) and the direction to push; (i) the gripper is tilted about 30 degrees along the y-axis of the gripper frame and (ii) the distance between the selected object and the gripper (i.e., Chamfer distance between point clouds sampled from the meshes) is set to lem. The push- ing action is described in Figure 14. If the gripper at the pose Tpush collide with the surrounding objects, the action is rejected; for colli-
Pushing action
Object
d
30
Gripper
Figure 14: Visual description
of the pushing action.
sion checking, we use the method used in Section B.2. The displacement of the gripper d is sampled from a discrete set {5, 10, cm. For each sampled action, we predict the next state 8:41 using pre-trained pushing dynamics model named SQPD-Net [20].
Pick-and-place action. pick-and-place action is defined by (T (Terasp: place) SE(3) SE(3), which are gripper poses when it grasps and places the object, respectively. To sample the grasp pose T 'gaasp an index ig from the recognized object indices i = 1, ...., N is selected and grasp pose is selected from the candidate grasp poses of the selected object (4.,, ); we generate the grasp poses using the same strategy we used in Section B.3. Then a gripper pose to place Tplace is selected from the poses which the grasped object does not overlap (i.e., not collide with) other surrounding objects and the shelf; we use the collision checking method used in Section B.2. We check the collision of the trajectory when grasping and placing and the action rejected if there a collision; use the same method used in Section B.3 when checking collision. For each sampled action, we predict the next state 81+1 by just applying the transformation grasp placee to the selected object.
Action sampler. We first check which objects can be pushed or grasped (for pick-and-plase action); whether an object can be pushed or grasped noted in the action description section above. Let Ip fipl...., (pN,) and 4, fig...... ... igN,, be the set of indices of the graspable and pushable objects, respectively. Then, randomly choose 30 indices (allowing duplicates) from the multi-set G + P = (ig1.-.. ,9N,, ip...... ... (pN,). For each chosen index, we select one action (pushing action if index is in Tp and pick-and-place action if index is in Ty) through the method described above. The action sequences for MPC are sampled in the similar manner.
22

---

C Experimental Details
C.1 Additional Details for Environment
As we note in Section 4, the simulation and real-world experiments share the same environment including the robot arm, the RGB-D camera, and the shelf. For this reason, we first build en- vironment in the real-world and import this environment into simulation. The intrinsic parameters of the camera and the shapes the robot and shelf are accessible advance; in order to match the two environments equally, the relative poses of the shelf and the camera to the robot base (i.e., relative SE(3)) are required. The shelf pose relative to the robot obtained by measuring with a ruler, and the camera pose relative the robot obtained from the calibration using the shape of the robot arm. More specifically, the camera pose is obtained through ICP (Iterative Closest Point) algorithm between the point cloud of the robot observed by the camera and the point cloud of the robot obtained from the known shape of the robot.
C.2 Additional Details for Simulation Experiments
Object configuration. We have created 180 scenarios for each number of surrounding objects in (2, 4, 6, 8), so total of 720 scenarios; for each scenario, random selection (allowing duplicates) made among the given objects.
Initial scene setting. We first randomly drop the selected objects on the shelf. Then, we place the (red) target object on the shelf where it is not visible from the current camera image. If there is no place to place the target object, or the target object is not graspable because of the collision of the gripper with the shelf, the scenario is discarded.
Target detection. In the simulation environment, ground-truth segmentation mask can be used from the synthetic camera. If part of the target object is observed for more than 100 pixels on the camera image (c.f. the resolution of the camera is 1280 x 720), it is considered to have succeeded in finding the target object in that scenario.
C.3 Additional Details for Real-world Experiments
Object configuration. We have created scenarios for each number of surrounding objects in {3, 4, 5, 63. total of 20 scenarios; for each scenario, a pre-defined object set is used according to the number of surrounding objects shown in Figure 15.
3 Objects
4 Objects
Objects
6 Objects
Figure 15: Pre-defined object set used for real-world experiments.
Initial scene setting. We put the given surrounding objects and the target object so that the target object (red cylinder) not visible in the initial camera view.
Target detection. In the 3D recognition process, we segment the observed point cloud. We first calculate the average RGB value of the points in each segmented point cloud. Then, we calcu- late the MSE between these average RGB values and the RGB of the target object (in this case, [0.7282 0.1558, 0.2099)), and if there is exactly one segmented point cloud with MSE smaller than 0.1. the target object said to be found.
23

---

D Additional Experimental Results
D.1 3-Object Toy Experiment
In this experiment, we compare the performance of the pro-
Objects on Shelves
Search-ane and grasp
and-grasp
Search for grasp
posed methods, search-and-grasp and search-for-grasp. and espe- cially highlight the advantages of the scarch-for-grasp method. Let consider a situation where a yellow cylindrical target object fully occluded by two larger boxes as described in the left of Figure 16. The search-and-grasy method may successfully find the target object by rearranging the two boxes, but does not guarantee the target object's graspability. As illustrated in the middle of Figure 16, the identified object may not have a collision-free robot trajectory for grasping in some cases. Consequently, additional actions would be necessary manipulate the environment and make the target object graspable. On the other hand, in the case of the search-for-grasp method, the target object's graspability is taken into account in the searching phase, so the two boxes are rearranged in way that the target object becomes not only visible but also graspable as shown in the right of Figure 16. In summary, when the target object is occluded by multiple objects, the bearch-for-gran method can be efficient in terms of the number of actions. To verify this, we design an additional simulation experiment named 3-object toy experiment as described below.
Fully-occluded Target object
Comparison of search-and-gret grasp and search-for- grasp methods to find the target object (yellow cylinder). This figure is a conceptual figure, not the result of imple- menting the methods.
Figure 16:
Object configuration. We have created 200 scenarios in the pybullet simulation environment with two large cylindrical objects and one small cylindrical target object. The radius and height of the target cylinder is fixed, and those of two large cylinders are randomly selected big enough to occlude the target.
Initial scene setting. The position of the camera center and the positions of three cylindrical objects straight line in the x-y plane that the target object occluded by the other two objects the shelf. The position of the target object is fixed, while the (x, y) coordinates of the other objects' positions are randomly selected on the straight line.
Target detection. Target detection method same with Appendix C.2.
Search-and-grass
Search-for-grast t=1
t=
t= 1
t=
RGB
0. Recog. Depth
Pick-and- place
Pick-and- place
Pick-and- place
Target grasping
Pick-and- place
Pick-and- place
Target grasping
Pushing
Example trajectories of simulation manipulation for R- searchhaand-eeee grasp (Left) and R- search-for. foth--oo-ggraspp grasp (Righr). Each column shows the camera input and action selection at each time step. In the simulation, surrounding objects are blue and the target object is red.
Figure 17:
Figure 17 shows the mechanical search results by search-and-grasp and search-for-grasp. In this subsection, only recognition-basee based (i.e., R-search-and grasp and -search-for-graae The search-and- sarch-and-grasp grasp method succeeds in finding the target object in two sock-and-lace actions as de-

---

sired, but when target object is found, it cannot be grasped due to collision with surrounding objects. So, this method makes the target object graspable by performing two additional actions as shown in the left of Figure 17. In the case of the search-for-rasye method, it also succeeds finding the target object in two pick-and-place actions, and at this time, the target object becomes graspable at the same time as shown in the right of Figure 17. This difference of the number of actions is due the difference in whether the graspability of the found object is taken into account when searching for the object. Although the search-for-ee sarch-p the search-and- grasp method, the search-for-grant grasp method can be efficient in terms of number of actions in these situations.
grasg method is slightly more computationally expensive than
In order to quantitatively verify this fact, we measure (i) the average grasp-success rate and the average number of actions to (ii) find and (iii) grasp the target object for the two methods, and the results are shown in Table 3. We note that the minimum number of actions required to find and grasp is 2. The average number of actions to find lower for search-and- grasp than for bearch-for-grayy since the search-and grasp method only focuses on finding the target object. On the other hand, the average number of actions to grasp is lower for search-for-gee -orggs than for search-and-grasy Unlike the search-and-grasy meth- ods, the search-for-grasp simultaneously search for target object while increasing the probability of grasping the found target object. As result, this fact has the effect of reducing the average num- ber of actions. In conclusion, we verify that the search-for-grapp method can be more efficient than the search-and-gee sarrchhandgraasp method in terms of the number of actions when the target object is occluded by multiple objects.
METHOD
Succ.
Find
Grasp 2.919
R- Search-andd -rassppp
0.985
2.065
R- Search-for-Grasy
0.995
2.255
2.497
Table Manipulation results for 3-object toy ex- periment.
D.2 Mechanical Search via Only Pushing Only Pick-and-place
This experiment evaluates the performance of two cases where only one action type (e.g., pick-and- place or pushing) is allowed, demonstrating that both motions are essential for mechanical search tasks on shelves. We only recognition-based method in this experiment.
Object configuration. We have created 75 scenarios for each number of surrounding objects in (4,8), 8), so a total of 150 scenarios; for each scenario, a random selection (allowing duplicates) is made among the given objects.
Action sampler. Instead of action sampler described in B.5, if only push is allowed, the action is sampled only in Ip. If only pick-and-place is allowed, the action is sampled only in Tg.
Initial scene setting Initial scene setting is the same with Appendix C.2.
Target detection. Target detection method is the same with Appendix C.2.
Figure 18 shows the mechanical search results by our methods using only hick-and-place and only pushing, In the first example, using only pushing succeeds in finding and grasping the target object in one action as desired, but using only pick-and-placs fails since there is no valid pick-and place action. In the second example, using only pick-and-place succeeds in finding and grasping one action, and using only pushing succeeds in finding the target object but there no pushing action to move the blue cylinder to be removed since it becomes close the wall of shelf. As such, there cases where using only pick-and-place and only pushing show different trajectories.
To quantitatively investigate the roles of the pushing and pick-and-place and the results are shown in Table 4. First, allowing only one type of action degrades the performance significantly as shown in the table. Specifically, using only pick-and-place highly degrades the performance of finding the target object compared the case where both actions are used, and on the other hand, using only pushing highly degrades the performance of grasping the target object, even though the success rate in finding is higher than using only pick-and-place.
25

---

Example 1
Example 2
Pick-and-Place t=
Pushing
Pick-and-Place
Pushing
t=0
t= t=
t=
t=
3
Recog Depth RGB
Not
Pushing
Target
Pick-and- place
Target
Pushing
Pushing
Pushing
Pushing
available
grasping
grasping
Figure 18: Example trajectories of simulation manipulation using only pick-and-place and only pushing. Each column shows the camera input and action selection at each time step. the sim- ulation, surrounding objects blue and the target object is red. (Left) A scenario where only pick-and-place fails but only pushing succeeds. (Right) A scenario where only pick-and-pla suc- ceeds but only pushing fails.
The number of objects
METHOD
ALLOWED ACTION
Find
Grasp
Find
Grasp
Succ.
0.973
0.787
0.893
0.613
Both
Steps
2.164
2.559
3.269
3.957
Suce.
0.76
0.627
0.627
0.48
R-Search-and-Gre
Pick-and- Place
Steps
1.404
1.596
2.255
3.361
Succ.
0,907
0.573
0,813
0.507
Push
Steps
2.559
3.628
3,098
4.763
Suce.
0.96
0.813
0.92
Both
Steps
2.139
2.77
3.101
3.889
Succ.
0.76
0.627
0.693
0,467
Seaarch-for-Grap
Pick-and- Place
Steps 1.386
1.617
2.327
3.543
Suce. 0.92
0.613
0.773
0.56
Push
Steps 2.609
3.826
3.034
Table 4: Simulation manipulation results for the one-type action experiments.
This result is because pick-and-place and pushing have different strengths and weaknesses. The pick-and-place is efficient in terms of the average number of actions because it can move the objects farther away, but the actions which can be performed are limited due to the fewer objects the robot can grasp in a cluttered scene. The pushing is better than pick-and-place for finding target objects because the robot can manipulate more objects than pick-and-place, but because the robot can't t move the objects that far, it requires higher number of actions because can't move objects that far. In summary, both pick-and-plan place and pushing are required to find the target and grasp the target object.
D.3 Mechanical Search with Box Target Object
This experiment evaluates the performance of our methods for mechanical search tasks with a box target object. In this case, the target pose space should be X' = SE((), i.e. position (x, y) of the target box on the shelf and its z-axis angle as orientation. We only use recognition- based method in this experiment.
Object configuration. We have created 40 scenarios for each number of surrounding objects in (4,8), 8). a total of 100 scenarios; for each scenario, random selection (allowing duplicates) is made among the given objects. The target object is replaced with a box shape instead of a cylinder.
Other settings are following the section C.2.
Figure 19 shows the mechanical search results by search-for-gee grasp the task of the box target object. The search-and aoacchhnn-graap grasp method succeeds in finding the target object in three pick-and-place
26

---

t=0
t=1 t= t=3
t= 4
RGB
Depth
Recog-
Pick-and-plas
Pick-and-place
Pick-and-plas
Pick-and-place
Target grasping
Figure 19: An example trajectory of simulation manipulation for R-scarch-for-grant for the box- shaped target object. Each column shows the camera input and action selection at each time step. In simulation, surrounding objects are blue and the target object is red.
number objects
8
METHOD
Find
Grasp
Find
Grasp
Suce.
0.925
0.775
0.825
0.525
R- R-Search-and-Grrap
Steps
1.568
3.323
2.394
4.81
Suec.
0.95
0.7
0.85
0.6
R-Search-for-Grayy
Steps
1.652
3.286
2.412
5.167
Table Simulation manipulation results for the box target object
actions and succeeds in grasping the target object in additional one pick-and- placce action. Table shows the performance of our methods with the box target object. Compared to the cylinder target object case, the grasp success rate lower when the number of objects is 4. This is because the approach direction of the grasping trajectory is limited to one in the case of the box as shown in the Figure 11; we note that the cylinder can be grasped in any direction due rotational symmetry. However, when the number objects are large (i.e., the number of objects 8), the box target object case shows similar performance to the cylinder target object case. This is because the advantage of having various grasping approach directions of cylinders is lost in highly cluttered environment. In summary, we conclude that our algorithm also works on the target pose space of SE(() on the box target object experiment.
27


Reviews:
[{"review_id": "yX8GmDFzRWJ", "reviewer": "Reviewer_bz1q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a practical method for finding and grasping target objects on cluttered shelves using a standard robot gripper. The authors develop an updated mechanical search algorithm based on 3D reconstruction of the scene, which is used to estimate the existence function and graspability function of the target object. The authors formulate the problem as a model-based optimal control problem and provide algorithms for both search-and-grasp and search-for-grasp methods. Experimental results in both simulation and real-world environments demonstrate the effectiveness of the proposed method.", "summary_of_recommendation": "More details on baselines and implementations are needed.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Can the authors more details on the implementation of the 3D reconstruction and estimation of the existence function and graspability function? How was the pushing dynamics model and depth rendering utilized?\n2. Have the authors compared the proposed method with existing baselines? It would be valuable to see how the proposed method performs compared to other approaches.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper leverages 3D reconstruction of the scene to estimate the existence function and graspability function, which enables the robot to find and grasp target objects in cluttered shelves without specialized tools. This is a practical and innovative approach that has not been explored extensively in previous research. The paper provides a well-structured framework for mechanical search and grasping, with specific algorithms for search-and-grasp and search-for-grasp methods. The formulation of the problem as a model-based optimal control problem is clear and logical. The experimental results in both simulation and real-world environments validate the effectiveness and real-world applicability of the proposed method. The comparison with cases where all object information is assumed to be known demonstrates that the proposed method performs similarly. \n\nSince the abstract mentions that existing works primarily focus on specialized tools with suction grippers, it would be interesting to see a comparison between these works and the proposed algorithm based on 3D reconstruction. Providing this contextual background will help the reader understand the novelty and originality of the proposed method. The paper lacks sufficient implementation details for reproducing the study. It would be helpful to provide more information on the specific techniques and algorithms used in the 3D reconstruction, estimation of the existence function and graspability function, and model-based optimal control. The paper could benefit from more thorough evaluation and ablation studies. It would be valuable to compare the proposed method with widely-known baselines in the field and conduct additional experiments to further analyze the performance and limitations of the method.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "kTlmEerMXh", "reviewer": "Reviewer_QSGc", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper tackles finding and grasping occluded objects on cluttered shelves using standard grippers. The key idea is representing target visibility and graspability as indicator functions and optimizing these via pushing and pick-place actions. 3D superquadric recognition enables estimating the functions from partial views. Model predictive control leverages the estimates to plan actions over a short horizon that reveal and enable grasping the target object. Experiments in simulation and real-world settings validate the proposed search-and-grasp and search-for-grasp methods can effectively retrieve objects in cluttered spaces comparable to oracle methods with full state knowledge. Key contributions are the visibility and graspability formulations, recognition-based estimation, and optimal control framework to enable mechanical search with standard grippers.", "summary_of_recommendation": "This paper makes technically strong contributions in advancing mechanical search capabilities using standard grippers rather than specialized tools. The novel formulation using visibility and graspability indicator functions provides a unique optimal control approach to sequential object rearrangement for retrieving occluded targets. Extensive experiments demonstrate effective performance on cluttered shelves in both simulation and real robot settings. While the object shapes are limited and impact potential incremental, the key ideas show promise in addressing a challenging problem. Overall quality and technical merit justify acceptance, though real-world applicability would benefit from more extensive validation and generalization beyond basic geometries.  ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Questions:\n\n* Why use a learned model for pushing prediction rather than also simulating it like the pick-and-place actions?\n* Have you experimented with more complex object shapes beyond basic geometries? How robust is the method to shape diversity?\n* How is occlusion handled in the recognition model? Does it limit how cluttered the scene can be?\n* What specific limitations did you observe in the real robot experiments? How can they be addressed?\n* Does rearrangement strategy matter? Is your method optimized for minimally disturbing the scene?\n\nPotential issues:\n\n* Expand the object shapes used in experiments to better demonstrate generalizability.\n* Provide ablation studies to isolate the impact of key algorithmic components.\n* Benchmark on larger datasets to characterize robustness. How does performance degrade with scene complexity?\n* Demonstrate benefits over ablations without the optimal control formulation.\n* Compare to more baselines like end-effector picking without rearrangement.\n* Discuss computational efficiency - how does it scale? Could optimizations be made?\n* Validate more real-world factors like pose estimation noise, partial occlusions, novel objects.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Originality: This is the first work I'm aware of that addresses mechanical search on cluttered shelves with standard grippers. The idea of using indicator functions to guide rearrangement actions is novel. Estimating these based on 3D recognition is also a new approach. The optimal control formulation using dynamics of learned indicator functions is unique.\n\nQuality: The technical approach is sound, leveraging superquadric recognition, PyBullet physics, and model predictive control. The experiments evaluate performance in simulation and real-world settings. Limitations are properly acknowledged regarding gripper constraints and recognition errors.\n\nClarity: The paper is well-written and clearly conveys the key ideas, approach, and results. The problem is motivated early on, and technical sections provide sufficient details to understand the methodology.\n\nSignificance: This research tackles a difficult problem in robotic manipulation that arises in real-world conditions. The ability to mechanically search for and retrieve occluded objects in tight, cluttered spaces can enable many useful applications. The proposed methods advance the state of the art in using standard grippers rather than specialized tools.\n\nRelevance: This work is relevant to the CoRL audience since it addresses a challenging robot learning problem involving mechanical search, interactive perception, and manipulation planning under uncertainty. The methods leverage learning for 3D recognition, dynamics modeling, and optimal control. Evaluation is extensive on both simulation and physical robots.\n\nLimitations: As mentioned by the authors, the fixed grasping approach limits general applicability. Collision checking for the graspability function is also computationally expensive. Recognition errors can propagate. Larger-scale experiments would be useful to better characterize robustness. The superquadric representation also limits the object shapes. Only a quite limited variety of objects with limited complexity and poses is considered.\n\nIn summary, I believe this paper makes a good contribution in advancing mechanical search capabilities for retrieving occluded objects from cluttered shelves using standard grippers. The technical approach is novel, of good quality, and clearly conveyed.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "XT0VeV4fXr", "reviewer": "Reviewer_kape", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a mechanical search method in cluttered shelf environments where the object of interest is initially not visible. The system moves other objects in the scene through prehensile and pushing gripper actions to find and clear the path to the target object. The authors formulate the task into a model-based optimal control problem and come up with the existence function/graspability function, which works jointly as the cost function of the solver. MPC is used to find the best local robot action that fulfills the objective. The authors conduct experiments in both the simulation and the real-world robotics system. Promising results are presented to demonstrate the effectiveness of the proposed method.", "summary_of_recommendation": "The work is convincing and provides new ideas to solve the object grasping tasks in cluttered shelf environments by formulating it as a model-based optimal control problem. The proposed method is technically sound. Results from the simulation and the real-world robot demonstrate the effectiveness of the method. This work has enough contributions to the TAMP area in the robotics field. However, I think the novelty in the robot learning part is limited or not addressed enough in the current version of the paper. Thus, to the best of my knowledge, my recommendation is weak accept considering its contribution to the robotics field.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1. It's better for the authors to revise the structure of the paper by having a proper related work section.\n2. The authors are encouraged to add the fixed camera setting in the introduction when talking about the challenges (the target object is not visible) that shelf environments introduced.\n3. Please revise the candidate pose issue mentioned in the weaknesses section or provide additional details/experiments to support the $SE(3)$ claim.\n4. It will be beneficial to give some explanation about why the object-to-robot collisions happen in the real-world demo. Do they come from the sensor noise or the inaccurate object models or some other facts in the motion planning method you used? The reasonable can be added to the limitation section.\n5. How does the agent choose between pushing action and picking-and-place action during the MPC process? Do they have the same chance of being selected? Additional info is needed for the readers to fully understand the pipeline.\n6. In line 216, I think the authors want to say Appendix Part C instead of B. Also, Appendix B.1 is empty. Consider removing or adding related content.\n7. The real-world instance segmentation in cluttered shelf environments is challenging because of the merge of edges between objects as well as the poor lighting conditions. The author can provide additional info other than just saying it is a DGCNN if there are modifications applied. Although this is not the contribution of this work, but it is the key to making the real-world experiment work.\n8. In line 312, what does it mean when the authors mention the gripper holds the object in a straight line? Does it mean the gripper will only try to approach the object from the front but not from the side?\n9. The work itself is convincing. But the authors are encouraged to provide justifications for why this work is suitable for CoRL as the novelty in the robot learning part is limited. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n1. The paper overall has a smooth logic flow and it is easy to follow. \n2. The problem setting that the object of interest is not visible initially is hard yet interesting and practical. In addition, the shelf environments bring more challenges to the problem like limited workspace, limited relocation space and severe occlusion, etc. This problem happens a lot in real-world scenarios if the agent can only observe the environment through a fixed camera. Solving this task can truly benefit real robotics applications.\n3. The setting where all objects except the object of interest are initially unknown is a good practice as that's what happened in the real world, the agent is more likely to have only the observation and its target.\n4. Since the object is not visible at first, the design of the existence function is smart. Also, the formulation of the model-based optimal control problem and the modification based on it makes good sense. The usage of MPC also fits well in this problem.\n5. The authors provide most of the design details of the proposed method.\n6. The real-world demos are comprehensive enough to demonstrate both successes and failures.\n\nWeaknesses:\n1. The authors claim that one of the challenges in the shelf environment is that the agent can only make observations from the front direction which makes the object of interest not observable in many cases. This is actually because of the setup in this paper (fixed camera) but not the nature of the shelf environment. If the robot has an in-hand camera that can observe the environment from different poses(although they are all at the front of the environment), I am sure that the red cylinder in the environment shown in Figures 1 and 4 can be seen even without relocating any objects. If this is the case, the system can directly figure out the objects in collision with the grasping path and relocate them according. Thus, the authors should include this camera setting in the introduction section to better motivate the proposed solution.\n2. Claiming the object pose $x \\in SE(3)$ is not accurate. First, since the target object is always placed on the surface, it is $(x, y)$ out of $(x, y, z)$. Then in both the figures and the video, the target object is always a red cylinder placed in a straight-up pose. I don't see there are any rotations included (at least no row and pitch, I cannot tell about the yaw because it does not matter for this object), which means it is at best $(yaw)$ out of $(row, pitch, yaw)$.  In all, the only thing that is certainly included in the calculation is a 2D point $(x,y)$. Although the authors did mention it as $\\mathbb{R}^2$ in the experiment section, it creates some inconsistency.  \n3. In the search-for-grasp method, there are two cases where $f_t(x) = 1$, 1. the target object is observed 2. the target object is still hidden but occluded by others. Since the overall objective of the problem is to retrieve the target object, whenever the target object is observed, the agent should directly aim to clear the path and get it out of the shelf. In the current method, since it just uses $f_t(x)$, the agent may still make efforts to perform other actions instead of directly clearing the path to the target object. Having a design that distinguishes between these two cases can benefit the method (minimizing $J_t$ until the objects show up, there is no need to keep doing it at this point, like the search-and-grasp method).\n4. There are some robot-object collisions shown in the real demo video. Once some objects fell due to collision, it can cause problems that can potentially prevent the robot from achieving the objective.\n5. The paper does not have a novel core learning-based module. The pushing dynamics model and the object recognition model are adapted from other works (not sure if there are any modifications introduced). The depth segmentation network is based on the DGCNN, not sure if there are any modifications introduced. Learning is of great usage in the pipeline but not the one of the main contributions.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "A9jAJy2qSIq", "reviewer": "Reviewer_e7zE", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a system for finding and grasping occluded objects in shelves without specialized hardware beyond a parallel jaw gripper. The authors propose an objective function that takes into account two functions, $f(x)$ and $g(x)$, representing existence and graspability. They then use shooting methods to minimize this objective over the course of a rollout. There are two versions of the method considered, one where graspability is considered as part of the objective, and one where it is considered as a separate step after visibility is achieved. The system employs 3d object recognition methods based on recognizing implicitly defined superquadrics, to recognize the geometry of objects within a scene at each step in the real world.", "summary_of_recommendation": "I think that the work has potential, but it would be good to see more novelty on the learning side for this problem beyond the object recognition, where it is unclear how useful the superquadric object detection method is. i.e. Which listed limitations could learning potentially help tackle?", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "- What are all the substantial differences between prior mechanical search work such as [19] and your proposed method? They feel quite similar. While hardware used is different, they also use a lookahead algorithm and similar pushing and pick & place actions. The main difference appears to be the actual grasping of the target object.\n- In the video, certain rollouts are listed as failures after a few suboptimal actions. I am unsure of why the algorithm seems to be unable to recover from these situations.\n- Please let me know if there are any factual inaccuracies in my understanding of your work :)", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a system for grasping occluded objects in shelves via rearrangement. I believe the paper has the following strengths:\n\n- The paper’s methods and findings are presented in a clear and digestible manner.\n- Physical experiments show the practicality of this system.\n- Well-defined problem statement.\n- Presents a principled approach to the problem of mechanical search for extracting a target object.\n\nHowever, I have the following concerns about the proposed method and its fit for this conference.\n\n- Given that 3D object recognition is the only learning-based component in this work, my feeling is that the paper’s emphasis should be more on the application of this method to mechanical search and what challenges & solutions are present in application to mechanical search. This is a good robotics paper, but I don’t know if it is suitable for CoRL given that the only learning is the application of an object detection model from prior work.\n- The success rate demonstrated in the real world experiments is quite low for 5 and 6 objects. The failure cases include incorrect 3D recognition but do not elaborate further. Again, given that this is the only learning in the paper, it would be good to explain this further. In simulation, the success rate is much higher; why is this? Is object recognition in the real world harder for some reason?\n- It is unclear to me what impact the superquadric-based object recognition method actually has. No ablations are provided to indicate whether simply segmenting objects and using that segmentation mask with some simple symmetry prior in 3d would give a sufficient shape to use for collision checking. Why does implicitly defining these objects’ shape via superquadrics help? Is there any indication that recognizing subtle variations in these objects’ shapes improves collision checking, etc. If not, isn’t the object recognition method overkill for the problem?\n\nMore minor points.\n\n- Using the word “Verify” for experiments may signal some confirmation bias; better to use words like “evaluate” instead.\n- Would be nice to have standard error or standard deviation in the table, or at least an explanation of what the distribution & variance for # of steps looks like.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers felt that the paper tackles an interesting and challenging problem, with a reasonable and well-explained solution.  The biggest concern was that it is unclear whether Superquadrics are important for this framework and whether there are any quantifiable advantages over standard methods for object recognition.
Decision: Accept (Poster)