SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yongbo Chen*  
School of Computing, Australian National University  
Canberra, ACT, 2601, Australia  
Yongbo.Chen@anu.edu.au
Hanna Kurniawati  
School of Computing, Australian National University  
Canberra, ACT, 2601, Australia  
Hanna.Kurniawati@anu.edu.au
Efficiently searching for target objects in complex environments that contain various types of furniture, such as shelves, tables, and beds, is crucial for mobile robots, but it poses significant challenges due to various factors such as localization errors, limited field of view, and visual occlusion. To address this problem, we propose a Partially Observable Markov Decision Process (POMDP) formulation with a growing state space for object search in a 3D region. We solve this POMDP by carefully designing a perception module and developing a planning algorithm, called Growing Partially Observable Monte-Carlo Planning (GPOMCP), based on online Monte-Carlo tree search and belief tree reuse with a novel upper confidence bound. We have demonstrated that belief tree reuse is reasonable and achieves good performance when the belief differences are limited. Additionally, we introduce a guessed target object with an updating grid world to guide the search in the information-less and reward-less cases, like the absence of any detected objects. We tested our approach using Gazebo simulations on four scenarios of target finding in a realistic indoor living environment with the Fetch robot simulator. Compared to the baseline approaches, which are based on POMCP, our results indicate that our approach enables the robot to find the target object with a higher success rate faster while using the same computational requirements.
Finding an object in a partially known and cluttered environment is critical for many robotics tasks. For instance, a robot operating in human environments, such as our homes may need to find mugs, glasses, or books that may be placed on top of various furniture like beds, tables, or chairs and partially occluded by other objects. While the positions of the furniture are commonly known, the positions of the smaller objects, such as the mugs and glasses, are not generally known and move very often. Humans can typically find those smaller objects by relying on prior knowledge, assuming the object they are searching for has been seen before[1] and removing occlusions as necessary. This kind of target objects-oriented problems is also called "Mechanical Search" problem [7].
This paper proposes a planning method based on a novel solver, called Growing Partially Observable Monte-Carlo Planning (GPOMCP), that enables mobile manipulators to do just that: Finding a target object that might be occluded by other objects in a partially unknown environment by moving to positions with better visibility or by removing occlusion when necessary.
We assume the robot is operating in a home environment that consists of multiple rooms and is provided with 3D point cloud and 2D occupancy grid maps of the environment and a set of the target object's photos. These maps contain the furniture but do not contain information about the objects on top of the furniture. GPOMCP computes an efficient search policy that utilizes onboard sensors and its ability to remove objects within a designated search region to find the target object even if it is partially or completely occluded while accounting for localization and perception errors. To handle the partial observability nature of the problem, GPOMCP is based on the Partially Observable Markov Decision Process (POMDP). A unique feature of GPOMCP is it can efficiently modify policies when the number of state variables increases. This feature allows GPOMCP to model the environment incrementally, starting from an environment map with no objects on top of the furniture to a more faithful map where all the objects on top of the furniture have been included in the map and everything in-between, as needed until the target object is found.
In addition to the above solver, we propose a POMDP model for the object-finding problem that exploits GPOMCP's ability to modify policies as the number of state variables increases. To this end, the model maintains a set of state variables that represents information about the robot's guess of the target object, which we refer to as a guessed target object. A belief over this set of state variables represents the robot's uncertainty over the characteristics of the target object, such as the pose, whether the object can be moved or not, etc. As new observations are perceived, the robot refines its belief about the guessed target object, and eventually, the belief has a high chance to converge to the correct target object positions based on suitable parameter settings. Experimental results on four different scenarios using a Fetch robot simulator in Gazebo indicate that the above model and GPOMCP significantly outperforms POMCP[20].
A POMDP is a general mathematical formulation for a sequential decision-making problem where the state is not fully observable by the agent. Formally, it is defined as an 8-tuple  $< S, \mathcal{A}, O, T, Z, R, b_0, \gamma >$ , where the state space  $S$  denotes the set of all possible states of the robot and the environment; the action space  $\mathcal{A}$  denotes the set of all actions the robot can perform; the observation space  $O$  means the set of all observations the robot can perceive. At each time step, a POMDP agent is at a state  $s \in S$ , but this state is generally not known exactly by the agent and therefore the POMDP agent uses a belief  $b$  —a distribution over the state space—to represent this state uncertainty. The set of all beliefs is referred to as the belief space and denoted as  $\mathcal{B}$ . At each time step, the agent performs an action  $a \in \mathcal{A}$  which may result in the agent arriving at a new state. However, the effect of actions is uncertain and is represented by a conditional probability function, namely the transition function  $T(s, a, s') = Pr(s'|s, a)$ . Once the agent arrives at a new state (which may not be fully known), the agent perceives perceives an observation  $o \in O$ , governed by the observation function  $Z(s', a, o) = Pr(o|s', a)$ , which is a conditional probability function that represents the observation the robot may perceive after performing action  $a \in \mathcal{A}$  in state  $s' \in S$ . Together with perceiving an observation, the agent also receives an immediate reward, governed by the reward function  $R(s, a, s') \in \mathbb{R}$ , which can be parameterized by a state, an action, a state-action pair, or a tuple of state, action, and subsequent state. The estimate on the initial state is called the initial belief and denoted as  $b_0$ . Last but not least,  $\gamma$  is a discount factor following  $0 < \gamma < 1$  [13]. The goal of solving a POMDP problem is to find an optimal policy  $\Pi^*(b) = \operatorname*{argmax}_{a \in \mathcal{A}} Q(b, a)$  for belief  $b \in \mathcal{B}$ , where the  $Q(b, a)$ -value is the value of executing action  $a$  when the agent is at belief  $b$  and continuing optimally afterwards.
POMDP solvers can be broadly classified into two main classes based on their real-time performance: offline and online solvers. While offline solvers can achieve optimal solutions for small-scale problems, such as the point-based POMDP algorithm [19, 14], they require too much memory for real-world applications. Most online POMDP solvers commonly explore the sparse belief sampling with a collection of particles and perform a Monte-Carlo tree search (MCTS) to grow the belief tree within the limited planning time. State-of-the-art online methods include Partially Observable Monte Carlo Planning (POMCP) [20], Adaptive Belief
Tree (ABT) [15], Determinized Sparse Partially Observable Tree (DESPOT) [21], Partially Observable Monte Carlo Planning with Observation Widening (POMCPOW) [22], and Adaptive Discretization using Voronoi Trees (ADVT) [9] methods. Among them, POMCP and POMCPOW methods rely on the Monte Carlo backup, while others use the Bellman backup. Although most of these methods are designed to explore belief nodes more deeply in continuous or large action and observation spaces, continuous state, action, and observation spaces continue to pose a significant challenge.
A substantial category of mechanical search methodologies is closely intertwined with advancements in learning technologies, including deep reinforcement learning (RL) [12], deep Q-learning [25], receding horizon planning guided by neural network-based distribution [2], deep-geometric inference systems [7, 10]. For example, in [12], the authors use a deep RL procedure that combines teacher-aided exploration, a critic with privileged information, and mid-level representations to uncover a target object occluded by a heap of unknown objects. Except for the learning-based methods, some other methods exploit the classical technologies, like the adaptive horizon exploration algorithm [8] and mixed-integer program (MIP) [3]. A shared characteristic among these methods is their specialized application in specific environments, primarily centered around scenarios involving shelves and boxes.
Some other works on object search are solved using POMDP planning approaches, such as a recent MCTS-based mechanical search method that relies on a 1D occupancy distribution for the current camera observation in cluttered shelves. [11]. The Object-Oriented POMDP (OO-POMDP) [24], which is very popular, considers the state and observation spaces to be factored by a set of  $n$  objects, where each belongs to a class with a set of attributes, and the beliefs about different objects are independent for the small computational reason. Many previous works also limit the workspace to a 2D plane with simple transition and observation models. For example, Aydemir et al. proposed a 2D POMDP formulation for the object search in a room by calculating candidate viewpoints [1]; The object search task has been extended to 3D with an object size-friendly property using a multi-resolution planning algorithm [28]. We follow the same Object-Oriented idea because of the good applicability of the real-world application but go forward one step to consider the occlusion relationship for the belief update between different objects in a complicated 3D environment, including some challenging cases with partial and fully occluded objects.
3 POMDP formulation for object search
Our focused object search task is to find a movable target object  $\{obj_{tar}\}$  in several clustered workspaces that contain an unknown number  $n > 1$  of obstacle objects  $\{obj_{obs}\}$  with unknown locations. To facilitate robot navigation, we have access to point cloud and occupancy grid maps representing the planning environment, including furniture. However, we do not possess the point clouds of the obstacle objects  $\{obj_{obs}\}$  or the target object  $obj_{tar}$ . Therefore, during the execution process, we rely on on-board sensors to detect and identify these objects  $(obj_{obs} \cup obj_{tar})$ . Since we gather information about the objects using real-world sensors, they are not mutually independent in our scenario and may influence each other. This differs from the assumption made in OO-POMDP.
State space  $S$ . A state  $s = \{s_r, s_{o^0}, s_{o^1}, \dots, s_{o^n}\} \in S$  is represented as two parts including robot configuration and object-oriented factors, where  $s_r$  is the robot state and  $s_{o^i}$  denote the object state of the  $i$ -th object. A robot state is written as  $s_r = (p, l_h, l_p, l_t)$ , where  $p$  is the 6D robot base pose,  $l_h$  is the lift height,  $l_p$  and  $l_t$  are the pan and tilt angles of the robot head. At the beginning stage, the robot may not get into the workspace, so no detected object is available. In order to guide the robot toward meaningful actions, we introduce a guessed target object  $s_{o^0} = (p_{o^0}, s_{o^0}^x, s_{o^0}^y, s_{o^0}^z, g_{o^0}, m_{o^0}, u_{o^0}) \in \mathbb{R}^{1 \times 20}$ , where  $p_{o^0} = (p_{o^0}^p, p_{o^0}^o)$  is the 6D object pose (3D position  $p_{o^0}^p$  and 3D rotations represented by quaternion  $p_{o^0}^x$ ;  $s_{o^0}^x$ ,  $s_{o^0}^y$ , and  $s_{o^0}^z$  mean the sizes along the principal 3D axis of the object;  $g_{o^0} \in \mathbb{R}^{1 \times 8}$  denotes 8 grid odds values to identify whether this object is a target or obstacle object, which will be updated using the scores obtained from image matching.  $m_{o^0} \in \mathbb{R}$  is the odds value to evaluate the move-ability of the guessed target object;  $u_{o^0} \in \{-2, -1, 0, 1\}$  is the int front value to claim whether this object is updating or not. Specifically, -2 means this object is
Figure 1: Grid update in one frame
Figure 2: Main steps for GPOMCP
removed after performing the removing action; -1 denotes that the object is still updating without performing declaring and removing actions, 0 and 1 means this object has been declared as an obstacle object and a target object. The belief of the position  $\pmb{p}_{o^0}^p$  of the guessed target object is saved in a grid world  $\mathcal{G}_f$  generated by all workspaces  $\mathcal{W}$  using many odds values  $\text{Odd}(\mathcal{G}_f)$  with a given resolution. Inside each grid, the position is sampled uniformly and its probability is obtained by its odds value. The odds values are updated based on the field of view (FOV) of the camera using real-world measurement in the excursion process, which is similar to the update of the occupancy grid map in Simultaneous Localization and Mapping (SLAM) [4], following:
$$
\begin{array}{l} \log O d d \left(\mathcal {G} _ {f} | z _ {1: t}\right) = \log O d d \left(\mathcal {G} _ {f} | z _ {t}\right) + \log O d d \left(\mathcal {G} _ {f} | z _ {1: t - 1}\right) \\ O d d \left(\mathcal {G} _ {f} | z _ {1: t}\right) = P \left(\mathcal {G} _ {f} | z _ {1: t}\right) / P (\neg \mathcal {G} _ {f} | z _ {1: t}) \\ \end{array}
$$
where  $P(\mathcal{G}_f|z_{1:t})$  and  $P(\neg \mathcal{G}_f|z_{1:t})$  means the probability of the object belonging to and not belonging to this grid based on multiple observations  $z_{1:t}$ ;  $\text{Odd}(\mathcal{G}_f|z_{1:t})$  is the corresponding odd value. In the belief tree search, we do not update the odds values  $\text{Odd}(\mathcal{G}_f)$  and it will be updated after the real-world excursion. In the planning stage, this grid world is just used to sample the potential position of the guessed target object in the root node. The guessed target object is special with a constant orientation (set as  $(0, 0, 0, 1)$ ), size  $(0.1, 0.1, 0.1)$ , and move-ability value (set as 100, movable). The grid values  $g_{o^0}$  and the declared value  $u_{o^0}$  are update-able in the belief tree search but need to be reinitialized as the given value after each excursion. Fig.1 shows the scenario about the grid world for the guessed target object. In fact, the guessed target object variable within the state vector saves the belief about the position of the target object. It not only encodes the probability information derived from the grid world with odds values but also facilitates efficient planning within Monte Carlo tree search with low computational complexity, acceptable memory cost, and coding friendliness. More explanations to the guessed target object are shown in the supplementary materials. The other object  $s_{o^i}$ ,  $i \neq 0 \in \mathbb{R}^{1 \times 20}$  follows the same format and all parameters should be updated both in belief tree search and real-world excursion. It is noted that the pose of the other objects  $s_{o^i}$  is estimated based on the point cloud which is independent of the grid world  $\mathcal{G}_f$ . The newly detected objects after each real-world excursion will be appended to the state vector and form a growing state space  $(s, s_{add}) \in S \times S_{add}$ .
Action space  $\mathcal{A}$ . Our method is task-level planning with many primitive actions instead of control-level planning. The key purpose of setting these actions is to change the FOV of the robot camera to cover the workspace and further operate objects. The action space consists of 5 types of primitives: Move  $(\pmb{p}^t, \omega_m)$  means to move the robot base to a given target pose  $\pmb{p}^t$ ; Head  $(l_p, l_t)$  denotes the actions to change the angles of the robot head pan (left/right) and tilt (up/down) to  $l_p$  for the pan and  $l_t$  for the tilt; Lift  $(l_h)$  represents to move the robot height to  $l_h$ ; Declaring  $(u_{o^i})$  means to declare the updating object  $(u_{o^i} = -1)$  to be the target object  $(u_{o^i} = 1)$  or the obstacle object  $(u_{o^i} = 0)$ ; Removing  $(i, \omega_r)$  is the action to remove  $i$ -th object using robot manipulator. All actions are assumed to have a  $100\%$  success rate for simplicity<sup>2</sup>.
<sup>2</sup>Even in the gazebo Fetch simulator, it is hard to ensure that each action is successfully performed, especially for removing actions. In order to guarantee the  $100\%$  success rate, after the robot's common control level operation, we call the gazebo server "set_model_state" to physically set the model state with some Gaussian noises, which will be improved in future work.
Observation space  $\mathcal{O}$ . The main focused observation is a FOV viewing frustum  $V$  projected from the robot head camera. Because we view each object by an oriented box with 8 grids, for each observation, we assume to update at most the 4-nearest grids with a noise negative log-odds value  $-c_{o} + \eta$ , a noise positive log-odds value  $c_{o} + \eta$ , a noise near-zero log-odds value  $\eta$ , where  $\eta$  follows the Gaussian distribution. Following the blind spot caused by the other objects, sometimes, the robot camera can not detect all 4-nearest grids. We set the observation space to be discrete. By fusing all  $n_{o}$  detected objects, we can get the observation to be  $\{\{i,\dots ,j\} ,\{o(i),\dots ,o(j)\} \} ,| \{i,\dots ,j\} | = n_{o}$ , where  $\{i,\dots ,j\}$  records the observed object with at least one updated grid and  $\{o(i),\dots ,o(j)\}$  are their updated log-odds value.
Transition function  $T$ . If the actions  $Move(\pmb{p}^t, \omega_m)$ ,  $Head(l_p, l_t)$ , and  $Lift(l_h)$  are applied to the robot, the camera location and orientation will change with some noises. The camera motion follows the rigid transformation and the static structure information of the applied robot is read from  $tf$  message. The declaring action  $Declaring(u_{o^i})$  will change the robot status  $u_{o^i}$  based on the comparison between the mean value of the minimal 2 log-odds of 8 grid values and the thresholds  $C_d^o$  for the obstacle object and  $C_t^o$  for the target object. If no removing action is performed to the  $i$ -th object, the  $i$ -th object is static and rigid. The removing action  $Removing(i, \omega_r)$  on  $i$ -th object will move the  $i$ -th object outside the workspace and set the status  $u_{o^i}$  to -2, which means they will not block the observation of the other objects.
Observation function  $Z$ . The coordinates of 8 grids of the  $i$ -th object can be computed using center point  $p_{o^i}^p$  and the orientation  $p_{o^i}^o$  of the object pose and its size  $s_{o^0}^x, s_{o^0}^y, s_{o^0}^z$  estimated by point cloud. After performing one action, the visible grids of different objects belonging to the 4 nearest grids will be updated. The updated value will be decided based on the mean log-odds value of all grids. If the original mean log-odds value is larger than a positive threshold  $\nu_p > 0$ , the updated value will be set to be positive  $c_0 + \eta$ . If it is smaller than a negative threshold  $\nu_n < 0$ , the updated value will be set to be negative  $-c_0 + \eta$ . Otherwise, the value between the positive and negative thresholds will generate a near-zero  $\eta$ . So as to encourage the exploration of the robot, the initial grid values  $g_{o^0}$  of the guessed target object are reinitialized to be larger than the positive threshold  $\nu_p > 0$  after each real-world excursion.
Reward function  $R$ . Our expected sequential actions involve the robot moving to the workspace for observation, continuously changing its viewpoints to gather more measurements, updating odds values in 8 grids, then declaring the updating object to be the obstacle or target object, and finally removing the obstacle objects for the more visible area or directly the target object to complete the whole task. Hence, if the whole task is completed successfully with the removal of the target object, the robot will receive a large positive reward  $R_{max} \gg 0$ . As the prerequisite action, correctly identifying the obstacle and target objects yields relatively large rewards ( $R_{c_0}$  and  $R_{c_t}$ , respectively), where  $R_{max} \gg R_{c_t} > R_{c_0} \gg 0$ . All actions incur a negative step cost ( $R_{min} < 0$ ). The cost of the removal action  $Removing(i, \omega_r)$  is set as  $2R_{min}$  due to its complexity. Illegal actions incur a large negative penalty ( $R_{ill} \ll 0$ ), including collisions with occupied grids in the occupancy grid map, attempting to remove immovable objects, declaring an object that doesn't meet the condition, removing updating objects without declaration, and trying to remove objects that have already been removed.
4 Action execution and perception with on-board sensors
Our perception part is carefully explored with many useful outputs, like estimated object pose, estimated object size, object move-ability, and object detector, fusing both point cloud data and the image data without using manual quick response (QR) codes. Our framework is suitable for all mobile robots with 2D Lidar and RGBD cameras, but specifically, we mainly consider the Fetch robot here. Move base actions are implemented using a ROS interface move_base and the interaction with the AMCL-based navigation stack [18]. The robot lift height and head joints are controlled by following the joint-space trajectories on a group of joints based on a ROS interface ros_control [5]. Removing action is to pick up the object and place it in some given areas outside the workspace using the ROS tool moveit [6].
To simplify the scenario and reduce the navigation effort, we assume the availability of a pre-built point cloud map  $\mathcal{M}$  of the robot's environment, including furniture and known
objects, which can be created beforehand through environment mapping and reused during planning. Then, the 3D point cloud  $\mathcal{F}_j$  in  $j$ -th frame from the depth camera is matched with the map  $\mathcal{M}$  by iterative closest point (ICP) [26]. The 3D ICP is initialized by matching 2D Lidar points with the 2D occupancy grid map. As the frame number increases, the point clouds of objects and environment  $\mathcal{P}_j = \mathcal{M} \bigcup \mathcal{F}_0 \bigcup \dots \bigcup \mathcal{F}_j$  are becoming more and more complete. After removing the original map  $\mathcal{M}_j' = \mathcal{P}_j / \mathcal{M}$ , the point clouds for  $n$  newly detected objects  $\{o_0, \dots, o_{n-1}\} \in \mathcal{M}_j'$  are extracted by point cloud segmentation using the Euclidean cluster extraction method. For estimating object pose and size, we obtain the minimum oriented bounding box for each object using principal component analysis. The point cloud segmentation is applied followed by data association based on the Mahalanobis distance of centroid points and point-wise mean distance. Additional details can be found in the supplementary materials.