SlowGuess/NeurIPS_2023_OCR huggingface dataset:
So as to complete the given object detection task using several given images and some semantic words (optional), based on the previous point cloud segmentation, we perform the point cloud re-projection, sub-image fusion, YOLO detection, and SIFT matching [17] to compute the probability of object detection. We also would like to manipulate the objects in the workspace to free some FOV, so we estimate the move-ability  $r_{move}$  of each object by the learning-based Grasp Pose Detection (GPD) toolbox [23], the k-means clustering algorithm, and the ROS moveit toolbox. The details of the implementation are presented in supplementary materials. In the planning stage, for each particle, we will randomly sample a value for this object and compare it with the move-ability  $r_{move}$  to identify its move-ability.
5 Belief tree reuse for online POMDP solver with growing state space
Here, we present a POMDP solver, GPOMCP, reusing the previous belief tree when the state space of the problem is growing. In this kind of POMDP problem, the observation obtained by real measurement causes the unpredicted new state, which means that the whole belief tree should be cut and a totally new belief tree should be computed by sampling. It is time-consuming. As shown in Fig. 2, we solve the problem by following steps:
- Step 1: When state space grows from  $s \in S$  to  $(s, s_{add}) \in S \times S_{add}$ , in the old belief tree  $\mathcal{T}$ , we can pick out the branches corresponding to old sub-state  $s$  and set it as the root of new tree  $\mathcal{T}'(s)$ .
- Step 2: This new tree  $\mathcal{T}'(s)$  has many particles with history  $h$ . For each particle, we can update the future state  $s'$  by the added state vector  $s_{add}'$  corresponding to new objects based on the transition function  $T(s_{add}, a, s_{add}')$  and observation function  $Z(s_{add}', a, o_{add})^3$ . It is noted that, in the state updating, we should consider the effect  $s' \rightarrow s_{add}'$  from the old state vector  $s$  corresponding to the old object. So we can get the updated histories about some or all particles within the given time limitation.
- Step 3: Then, starting from the root, we can grow the belief tree  $\mathcal{T}'(s)$  using these updated particles. It is noted that, in building the new belief tree  $\mathcal{T}'(s)$ , the new observation nodes with newly detected objects may be introduced, which means that the generated belief tree  $\mathcal{T}'(s)$  may be not a sub-tree of the old belief tree  $\mathcal{T}$ .
- Step 4: Then, we perform the Monte Carlo tree search (MCTS) for all branches including the new actions and new observations. The action selection strategy follows the upper confidence bound (UCB) and the belief distance  $C_b \sum_{o \in O} Z(s'_new, a, o') \| b((s', s'_add) : s' \to s'_add) - b((s', s'_add) : s' \rightleftarrows s'_add) \|_1 + V^*(b, a) + C_N \sqrt{\frac{\log N(b)}{N(b, a)}}$ , where  $C_b$  and  $C_N$  are coefficients set by users.
We know that the current optimal action should be selected following equation Eq. (2).
$$
\max  _ {\boldsymbol {a} \in A} \left(\sum_ {\boldsymbol {s} _ {\text {n e w}} \in S} R (\boldsymbol {s} _ {\text {n e w}}, \boldsymbol {a}, \boldsymbol {s} _ {\text {n e w}} ^ {\prime}) \boldsymbol {b} (\boldsymbol {s} _ {\text {n e w}}) + \gamma \sum_ {o \in O} \sum_ {\boldsymbol {s} _ {\text {n e w}} ^ {\prime} \in S ^ {\prime}} Z (\boldsymbol {s} _ {\text {n e w}} ^ {\prime}, \boldsymbol {a}, \boldsymbol {o} ^ {\prime}) T (\boldsymbol {s} _ {\text {n e w}}, \boldsymbol {a}, \boldsymbol {s} _ {\text {n e w}} ^ {\prime}) V ^ {*} (\boldsymbol {b} (\boldsymbol {s} _ {\text {n e w}} ^ {\prime}: s ^ {\prime} \rightleftharpoons s _ {a d d} ^ {\prime}))\right) \tag {2}
$$
<sup>3</sup>It is noted that because our considered actions  $\pmb{a}_{1}$  and  $\pmb{a}_{2}$  just perform on the old objects, the reward function does not need to be updated. Our reward function follows  $R((s, s_{add}), a_{1}, (s', s_{add}') = R(s, a_{1}, s'))$ , because the old action  $\pmb{a}_{1}$  and  $\pmb{a}_{2}$  just operate one object each step.
4Here, the belief of the old state does not consider the effect from the new object  $s_{add}^{\prime}\nrightarrow s^{\prime}$
where  $s_{new} = (s, \emptyset)$  and  $s_{new}' = (s', s_{add}')$ . Considering two ways (GPOMCP and the direct resampling way), in the sub-tree of the new belief tree, the only difference is from the last terms  $V^{*}(b(s_{new}' : s' \rightleftarrows s_{add}')$  (correct by building tree) and  $V^{*}(b(s_{new}' : s' \rightarrow s_{add}')$ ) (GPOMCP, without considering partial effect). Then, we have the following Theorems:
Theorem 1. We are considering the unweighted particles for approximating the extended full-correct belief  $\pmb{b}(s_{new}': s' \rightleftarrows s_{add}') = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}(s_{new}' = s_{new}'': s' \rightleftarrows s_{add}')$  and the reused approximated belief  $\pmb{b}(s_{new}': s' \rightarrow s_{add}') = \frac{1}{N} \sum_{i=1}^{N} \mathbb{I}(s_{new}' = s_{new}'': s' \rightarrow s_{add}')$ . Assuming the reward function is Borel measurable and bounded, denoted  $R_{\text{max}} = \|R\|_{\infty}$ , the belief L1 distance is limited in  $\| \pmb{b}(s_{new}' : s' \rightleftarrows s_{add}') - \pmb{b}(s_{new}' : s' \rightarrow s_{add}') \|_1 \leq \delta$ , and the optimal action selected by building the whole tree is same as the one selected by approximated value  $\max_a V(\pmb{b}(s_{new}), \pmb{a}) = \max_a \hat{V}(\pmb{b}(s_{new}), \pmb{a}) = a^*$ , the optimal value function of POMDP problem using our method  $V^*(\pmb{b}(s_{new}))$  and the optimal value function using the direct resampling way  $\hat{V}^*(\pmb{b}(s_{new}))$  will satisfy the following bounding equation:
$$
\left\| V ^ {*} \left(\boldsymbol {b} \left(\boldsymbol {s} _ {\text {n e w}}\right)\right) - \hat {V} ^ {*} \left(\boldsymbol {b} \left(\boldsymbol {s} _ {\text {n e w}}\right)\right) \right\| _ {1} \leq \frac {\gamma R _ {\max }}{1 - \gamma} \delta \tag {3}
$$
Proof. Please refer to the supplementary material.
This theorem indicates that the reuse technology works well with a performance bound when the reused belief tree shows good approximations to the exact accurate belief tree. To enhance the estimation of the upper bound, we introduce a term  $C_b \sum_{o \in O} Z(s_{new}', a, o') \| b(s_{new}' : s' \to s_{add}') - b(s_{new}' : s' \rightleftharpoons s_{add}') \|_1$  to the UCB strategy when reusing the previous belief tree.
It is noted that Theorem 1 is correct only under the assumption  $\max_{\pmb{a}} V(\pmb{b}(\pmb{s}_{\text{new}}), \pmb{a}) = \max_{\pmb{a}} \hat{V}(\pmb{b}(\pmb{s}_{\text{new}}), \pmb{a}) = \pmb{a}^*$ , which means that there are some bounds for the performance guarantee shown in Theorem 1 using the reuse approximation.
Discussion 1. In fact, our belief tree reuse can be considered as the wrong estimation  $\triangle$  of the action and measurement node values in the correct belief tree. When the distance between the value function for the optimal action  $V(\pmb{b}(\pmb{s}_{\text{new}}), \pmb{a}^*)$  and the value function for any sub-optimal action  $V(\pmb{b}(\pmb{s}_{\text{new}}), \pmb{a}_{\text{sub}}^*)$  is relatively large satisfying  $V(\pmb{b}(\pmb{s}_{\text{new}}), \pmb{a}^*) - V(\pmb{b}(\pmb{s}_{\text{new}}), \pmb{a}_{\text{sub}}^*) >> \triangle$ , which means that the estimation error  $\triangle$  will not affect the final solution and the convergence of our solver follows the method directly rebuilding the whole tree.
Following Discussion 1, we would like to get the exact performance bound to ensure that the obtained solution remains the same as the optimal solution using the fully-rebuilt belief tree. Firstly, we have the following corollary:
Corollary 1. With the same assumption as Theorem 1, given the action  $\mathbf{a}$ , the optimal action-value function of POMDP problem using our method  $V^{*}(\mathbf{b}(\mathbf{s}_{\mathrm{new}})$ ,  $\mathbf{a}$ ) and the optimal value function using the direct resampling way  $\hat{V}^{*}(\mathbf{b}(\mathbf{s}_{\mathrm{new}})$ ,  $\mathbf{a}$ ) will satisfy the following bounding equation:
$$
\left\| V ^ {*} \left(\boldsymbol {b} \left(\boldsymbol {s} _ {\text {n e w}}\right), \boldsymbol {a}\right) - \hat {V} ^ {*} \left(\boldsymbol {b} \left(\boldsymbol {s} _ {\text {n e w}}\right), \boldsymbol {a}\right) \right\| _ {1} \leq \frac {\gamma R _ {\max }}{1 - \gamma} \delta \tag {4}
$$
Proof. Just follows the similar proof process shown in Theorem 1, so we ignore it here.
Corollary 2. With the same assumption as Theorem 1, if the distance between the value function for the optimal action  $V(\pmb{b}(\pmb{s}_{\text{new}}), \pmb{a}^*)$  and the value function for any sub-optimal action  $V(\pmb{b}(\pmb{s}_{\text{new}}), \pmb{a}_{\text{sub}}^*)$  is larger than  $\frac{2\gamma R_{\text{max}}}{1 - \gamma} \delta$ , the optimal action obtained by the tree reuse approximation way will be same as the optimal action using the direct resampling way  $\max_{\pmb{a}} V(\pmb{b}(\pmb{s}_{\text{new}}), \pmb{a}) = \max_{\pmb{a}} \hat{V}(\pmb{b}(\pmb{s}_{\text{new}}), \pmb{a}) = \pmb{a}^*$ .
The pseudocode Algorithms of the GPOMCP solver are shown in supplementary materials. Our proposed method follows the common procedure with four alternating stages, planning, execution, obtaining observation, and filtering. The optimal action for the current belief
is selected based on GPOMCP. Inside the POMDP solver, the Monte Carlo update and a revised UCB strategy considering the belief error proposed in Section 5 are applied.
In this section, we showcase our method's effectiveness via simulations conducted using C++ and Python codes within the Fetch robot simulator. The project operates on multiple Ubuntu 18.04 desktops equipped with an Intel Core i7-13700k processor, running exclusively on CPU. We evaluate our method using various object settings in different scenarios.
Problem settings We implement our approach in a realistic gazebo living environment (Fig. 3a). The 3D point cloud map and 2D occupancy grid map, generated using [16], of this environment with furniture are available before the object searching task (Fig. 3b). There are multiple workspaces on the surfaces of the large furniture, like tables, beds, and desks, in this planning environment. Some unknown objects including a target object are set inside the different workspaces sometimes with an occlusion relationship. Four photos of the target object from different viewing directions are available for the object detector.
(a) Gazebo environment
(b) Corresponding point cloud and occupancy grid maps
Figure 3: The planning environment
We manually set several candidate poses that can observe partial workspace for the move base action  $\text{Move}(\pmb{p}^t, \omega_m)$ . The candidate robot head motion  $(l_p, l_t)$  are set within  $l_p \in \{-\frac{1}{12}\pi, 0, \frac{1}{12}\pi\}$ ,  $l_t \in \{-0.5, 0, 0.5\}$ , and the lift motion  $l_h$  is limited in  $l_h \in \{0.0, 0.2, 0.4\}$ . The FOV of the Fetch camera has a 60-degree horizontal view angle with a 480/600 height-width ratio. In the transition and observation function, using these actions, we add the nearest (0.5 meters to the camera center) and farthest planes (1.7 meters to the camera center) to the FOV and form a hexahedron. This hexahedron-shaped FOV will move based on the robot's motion and configuration change. As mentioned in Section 3, each object is represented as 8 grids based on its optimal oriented box, and the occlusion is computed using perspective projection of the cube vertices. The grid size of the grid world is set as 2 cm. A discussion about different grid sizes is shown in the supplementary materials. The log-odds value of these grids is initially set as unknown satisfying  $\log \frac{0.5}{1 - 0.5} = 0$ . These odds values will be updated after each real-world observation and then used to sample the potential position of the guessed target object. The planning time is limited to 60 s/step, the discounted factor  $\gamma = 0.9$ , the threshold to the object declaration  $C_d^t = \log(8/2)$  and  $C_d^o = \log(2/8)$ , the thresholds for grid updating  $\nu_p = 0.1$  and  $\nu_n = -0.1$ , the re-initialized grid values of the guessed target object are set as 0.2, the reward values  $R_{max} = 10^5$ ,  $R_{c_t} = 5 \times 10^4$ ,  $R_{c_o} = 10^4$ ,  $R_{min} = -1$ , and  $R_{ill} = -10^3$ .
Comparison results Because a lot of existing object search approaches are based on the pure POMCP method with different problem settings, eg. [27, 24], we compare our approach using Bellman (Ours_b) or Monte Carlo update (Ours_m) with the POMCP method without using/using the guessed target object (denoted as POMCP, POMCP_f). The comparison is implemented based on 4 different scenarios with different object numbers, including  $LOOSE_{1}$  (4 objects in 1 workspace),  $LOOSE_{2}$  (6 objects in 2 workspaces),  $Hidden_{1}$  (7 objects
Here, we use the Monte Carlo update because the visual measurement model in object search is not well predictable. The robot will frequently receive unexpected measurements, which are not in the belief tree. It breaks the advantage of the Bellman update (explore deeper) of Adaptive Belief Tree (ABT) solver [15]. We think that, in other applications, the Bellman update should be better.
in 1 workspace), and Covered<sub>1</sub> (7 objects in 1 workspace), as shown in Fig. 4. The target object is the blue snack box shown in the blue circle. Only the black headset box and the dish rack are not movable, and the others are all movable. Green arrows show the candidate robot poses. "Tick" means the target objects can be observed easily without changing the robot configuration. "Half tick" means the target can be observed if we change the robot configuration. "Cross" means the target object is fully hidden by the obstacle objects and cannot be seen from that pose even changing the robot configuration, including the lift of the robot, if the obstacle objects are not removed before. For each scenario, 20 trials are conducted and statistical results are reported. More results with different problem settings are shown in the supplementary materials.
(a)  $LOOSE_{1}$
(b)  $LOOSE_{2}$
(c) Hidden
Figure 4: The planning environment
(d) Covered
The obtained  $95\%$  confidence interval of the discounted cumulative reward, the steps, and the success rate are reported in Table 1. For clarity, we present one sequence of POMDP-generated actions (Fig. 5), of which the discounted cumulative reward is 57691.7, to complete the Covered<sub>1</sub> scenario by moving its pose, changing its configuration, declaring, and removing the target object within 14 steps. The robot removes the obstacles (objects 3 and 4) to free the visual occlusion and finally declares and removes the target objects. The sub-images in the red boxes show the probability of the grid world for the guessed target objects, which is updated using Eq. 1, and the ones in yellow boxes show the detected camera images. We can find that, with the growth of the robot's actions, the probability of most grids far away from the target object decreases quickly and the ones near the target object increase a lot, which guides the robot to make more effort to observe the area with the target object.
Figure 5: Actions to complete the Covered<sub>1</sub> scenario with 7 objects in 14 steps.
Our method, utilizing Monte Carlo update, demonstrates superior performance in complex object search scenarios such as cases like Hidden<sub>1</sub> and Covered<sub>1</sub>, where predicting camera observations is challenging. In simpler cases like Loose<sub>1</sub>, our method performs comparably to the POMCP method when using our introduced guessed target object. The guessed target object proves to be beneficial in activating the robot in situations with limited information. By incorporating our approximated belief tree reuse, we retain more useful branches after receiving observations, leading to further improvement in overall performance. The concept of the guessed target object aligns with our natural tendency to assume the presence of certain objects in specific locations and adjust our estimations based on visual observations.
**Limitation** Our main limitations mainly stem from the errors and failures from perception and navigation components, not our focused planning part. The first limitation is the
Table 1: Discounted cumulative reward, steps, and successful rate (within 50 steps)
Scenarios	\( {Loose}_{1} \)	\( {Loose}_{2} \)
POMCP	66003.7 ± 16269.5 | 13.5 ± 6.8 | 90%	26175.8 ± 8220.3 | 29.2 ± 7.7 | 65%
\( {\mathrm{{POMCP}}}_{f} \)	78629.6 ± 8472.1 ± | 7.2 ± 0.7 | 100%	38462.6 ± 11221.2 | 23.2 ± 6.6 | 95%
\( Ours_{b} \)	84701.5 ± 5546.8 | 6.9 ± 0.6 | 100%	33265.5 ± 10085.3 | 22.2 ± 4.2 | 100%
\( Ours_{m} \)	79788.0 ± 4787.4 | 7.4 ± 0.6 | 100%	51574.1 ± 17930.9 | 17.4 ± 2.9 | 100%
Scenarios	\( Hidden_{1} \)	\( Covered_{1} \)
POMCP	40958.7 ± 8070.9 | 20.1 ± 2.7 | 100%	17700.2 ± 3616.6 | 38.3 ± 5.1 | 65%
\( {\mathrm{{POMCP}}}_{f} \)	45815.2 ± 7260.9 | 19.4 ± 2.8 | 100%	31506.9 ± 6249.7 | 32.3 ± 6.0 | 80%
\( Ours_{b} \)	45038.6 ± 7631.0 | 18.2 ± 2.7 | 100%	26130.6 ± 3368.7 | 30.2 ± 4.6 | 90%
\( Ours_{m} \)	55574.8 ± 6225.3 | 15.3 ± 2.0 | 100%	34397.8 ± 7381.6 | 26.7 ± 4.8 | 95%
pre-existing map  $\mathcal{M}$  which may not be available for a real robot but can be solved by introducing advanced sementic SLAM method. Achieving  $100\%$  primitive action success in reality is impractical, affecting our method's success rate. Another limitation is that our framework is based on point cloud segmentation to identify different objects. It may generate incorrect bounding boxes for objects with extensive contact areas. This misinterpretation can result in erroneous data association, impacting belief updates and removing actions. Furthermore, YOLO and SIFT object detection techniques may struggle in scenarios with limited SIFT features and YOLOv5 toolbox coverage, especially in low-light environments. However, we believe that addressing these limitations is possible by incorporating more advanced perception and navigation methods.
We propose a novel object-level POMDP formulation with growing state space, a guessed target object, and its grid world for object search in 3D with real-world sensors and solve it using a novel GPOMCP method based on Monte-Carlo tree search and belief tree reuse. We demonstrate the effectiveness of our approximation in cases with similar beliefs. Our method achieves object search with only maps, photos, and onboard sensors without relying on additional object information, like geometric models, making it highly practical for real-world robot applications. By utilizing the guessed target object, our method can successfully locate fully hidden objects even without prior information, by removing obstacles—a challenge for existing approaches. Our future directions involve dropping prior information, such as pre-built point cloud map  $\mathcal{M}$ , and exploring continuous action domains. Through our experiments, we showcase the efficiency of our method, and we anticipate that our results will inspire further advancements in this field.
This research is supported by the ANU Futures Scheme.
[1] Alper Aydemir, Andrzej Pronobis, Moritz Göbelbecker, and Patric Jensfelt. Active visual object search in unknown environments using uncertain semantics. IEEE Transactions on Robotics, 29(4):986-1002, 2013.
[2] Wissam Bejjani, Wisdom C Agboh, Mehmet R Dogar, and Matteo Leonetti. Occlusion-aware search for object retrieval in clutter. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4678-4685. IEEE, 2021.
[3] Lawrence Yunliang Chen, Huang Huang, Michael Danielczuk, Jeffrey Ichnowski, and Ken Goldberg. Optimal shelf arrangement to minimize robot retrieval time. In 2022 IEEE 18th International Conference on Automation Science and Engineering (CASE), pages 993-1000. IEEE, 2022.
[4] Yongbo Chen, Shoudong Huang, and Robert Fitch. Active slam for mobile robots with area coverage and obstacle avoidance. IEEE/ASME Transactions on Mechatronics, 25(3):1182-1192, 2020.
[22] Zachary Sunberg and Mykel Kochenderfer. Online algorithms for pomdps with continuous state, action, and observation spaces. In Proceedings of the International Conference on Automated Planning and Scheduling, volume 28, pages 259-263, 2018.
[23] Andreas Ten Pas, Marcus Gualtieri, Kate Saenko, and Robert Platt. Grasp pose detection in point clouds. The International Journal of Robotics Research, 36(13-14):1455-1473, 2017.
[24] Arthur Wandzel, Yoonseon Oh, Michael Fishman, Nishanth Kumar, Lawson LS Wong, and Stefanie Tellex. Multi-object search using object-oriented pomdpds. In 2019 International Conference on Robotics and Automation (ICRA), pages 7194-7200. IEEE, 2019.
[25] Yang Yang, Hengyue Liang, and Changyun Choi. A deep learning approach to grasping the invisible. IEEE Robotics and Automation Letters, 5(2):2232-2239, 2020.
[26] Juyong Zhang, Yuxin Yao, and Bailin Deng. Fast and robust iterative closest point. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3450-3466, 2021.
[27] Kaiyu Zheng, Anirudha Paul, and Stefanie TELlex. A system for generalized 3d multi-object search. arXiv preprint arXiv:2303.03178, 2023.
[28] Kaiyu Zheng, Yoonchang Sung, George Konidaris, and Stefanie Tellex. Multi-resolution pomdp planning for multi-object search in 3d. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2022-2029. IEEE, 2021.