ROS-HPL: Robotic Object Search with Hierarchical Policy Learning and Intrinsic-Extrinsic Modeling

1 INTRODUCTION . Robotic Object Search ( ROS ) is a task where an intelligent agent ( a.k.a . robot ) is expected to take reasonable steps to approach the user-specified object in an unknown indoor environment . ROS is an essential capability for assistant robots and could serve as the enabling step for other tasks , such as the Embodied Question Answering ( Das et al. , 2018a ) . Recently , ( deep ) reinforcement learning ( RL ) has demonstrated its power at enabling robots with autonomous behaviors ( Arulkumaran et al. , 2017 ) , such as navigating over an unknown environment ( Mirowski et al. , 2016 ; Zhu et al. , 2017 ) , manipulating objects with robot ’ s end effectors ( Gu et al. , 2017 ; Popov et al. , 2017 ; Rajeswaran et al. , 2017 ) , and motion planning ( Chen et al. , 2017 ; Everett et al. , 2018 ) . Under the RL setting , a robot learns behavioral policy by maximizing the expected rewards that are estimated by interacting with the environment physically and/or virtually . The estimated rewards serve as the reinforcement signals for robot to update its policy . A well-known challenge to train agent to perform ROS with RL is the sparse reward issue , due to the fact that the environment and/or the location of the target object are typically unknown . With welldefined reward functions , such as the ones in Atari games ( Mnih et al. , 2015 ) , the agents are shown to achieve extremely promising performance . However , it is a well-known challenge to define the reward function under the real-world scenarios ( Abbeel & Ng , 2004 ) . Typically , for the real-world applications such as object search or target-driven visual navigation , prior research prefers to construct the reward function in terms of the distance between the robot ’ s current location and the target location under the assumption that the full information of the environment is known ( Mousavian et al. , 2018 ; Wang et al. , 2018b ; a ) . Given an unknown environment , a straightforward way is to set a high reward when robot reaches the final goal state while at all other intermediate states , the reward is either zero or with a small negative value ( Zhu et al. , 2017 ) . More recently , Ye et al . ( 2018b ) 1 Our code and models can be found in https : //XXX.XXX/XXX . presented a relatively denser reward function which is based on the bounding box of the target object from robot ’ s detection system , the reward is still not defined among the situations that the target object is not detected . Thus , learning in a sparse reward setting ( reward is only defined for a small subset of states ) is a fundamental challenge for RL agents to perform ROS well , especially dealing with real-world scenarios and in complex environments . To overcome the sparsity issue in reward settings , hierarchical RL has shown its superiority under the sparse reward setting ( Kulkarni et al. , 2016 ; Le et al. , 2018 ; Levy et al. , 2018 ) . It aims to learn multiple layers of policies , in which the higher layer breaks down the task into several easier sub-tasks and proposes corresponding sub-goals for the lower layer to achieve . This kind of hierarchical policy is biologically plausible and intuitive . For human beings , when asked to accomplish a complicated task , we rarely plan over the atomic steps directly towards the target goal . Instead , we plan through a sequence of semantic meaningful sub-goals . Motivated by this observation , we put forward a novel two-layer hierarchical policy learning paradigm to deal with the sparse reward challenge . One way to contruct the hierarchical policy learning is by adopting the option-critic framework ( Bacon et al. , 2017 ) to generate the sub-goals automatically . By specifying the number of desired sub-tasks or sub-goals , the option-critic framework can automatically abstract the whole action sequences into the specified number of the sub-goals . However , the automatically generated sub-goals typically lack semantically meaningful senses , thus are difficult for humans to understand and hurts the overall system ’ s interpretability . In the other way of constructing the hierarchical policy learning , as shown in work ( Kulkarni et al. , 2016 ; Das et al. , 2018b ) , the sub-goal space could be designated from human knowledge with semantic meaningful terms . Constructing the hierarchy following human knowledge is intuitive and interpretable . Whenever the high-level layer proposes a sub-goal from the sub-goal space , the lowlevel layer then aims to achieve it without considering the overall goal again . However , although their results are promising for certain tasks , it is worth to mention that there is no guarantee the prior human knowledge is always perfect . The low-level layer policy which focuses only on achieving the sub-goal proposed by the high-level layer policy without keeping the final goal in mind , may drift away from being ultimately successful . This is equivalent to the scenario that without the long-term goal to provide human beings a clear direction and focus down the road , our short-term achievements are not likely to add up to something substantial . Taking the advantages from these two ways , we put forward a system with a higher level of generalization capability upon previous work ( especially , Bacon et al . ( 2017 ) and Kulkarni et al . ( 2016 ) ) and present a novel framework dubbed as ROS-HPL : Robotic Object Search with Hierarchical Policy Learning and Intrinsic-Extrinsic Modeling , to mitigate the aforementioned challenges ( see Fig . 1 left ) . Our contributions could be summarized in three-fold . First , we fully utilize the prior knowledge provided by human to make the hierarchy interpretable . Our high-level layer plans over a sub-goal space from human knowledge in order to achieve the final goal . Second , when a sub-goal is proposed by the high-level layer , the low-level layer learns the action policy by deliberating between achieving the immediate sub-goal and the action that is helpful for achieving the final goal . Third , we build our framework with a novel intrinsic and extrinsic rewards setting . Though the rewards in this paper are designed for object search , the method itself is general and can be applied to other tasks . To validate ROS-HPL , we conduct extensive sets of experiments on the House3D ( Wu et al. , 2018 ) simulation environment , and report the observed results that validate the efficiency and efficacy of the proposed system over other state-of-the-art systems for object search . 2 OUR APPROACH . First , we define the robotic object search ( ROS ) task . Formally speaking , when a target object is specified and provided with an image , the robot is asked to search and approach the object from its random starting position . The RGB image from the robot ’ s on-board camera is the only source of information for decision making . None of the environment information , such as the map of the environment or the location of the target object could be accessed . Once the area of the target object in the robot ’ s viewpoint ( the image captured by its camera ) is larger than a predefined threshold , the agent stops and we consider it as a success . In this work , we present a novel two-layer hierarchical policy for the robot to perform the object search task , motivated by how human beings typically conduct object search . In the following sections , we first describe the hierarchy of policies . Then we introduce two kinds of reward functions , i.e . extrinsic rewards and intrinsic rewards , and we make use of these two reward functions to formulate the solution . Finally , we describe the network architecture adopted for learning the two-layer hierarchical policy . 2.1 HIERARCHY OF POLICIES . Our hierarchical policy has two levels , a high-level policy πh and a low-level policy πl . At time step t , the robot takes the image captured by its camera as the current state st . Given a target object or goal g , the high-level layer proposes a sub-goal sgt ∼ πh ( sg|st , g ) and the low-level layer takes over the control . The low-level layer then draws an atomic action at ∼ πl ( a|st , g , sgt ) to perform . The robot will receive a new image/state st+1 . The low-level layer repeats Nt times till 1 ) the low-level layer terminates itself achieving the termination signal term ( st+Nt , g , sgt ) ; 2 ) the low-level layer achieves the sub-goal sgt . Either way , the low-level layer terminates at state st+Nt , and then returns the control back to the high-level layer , and the high-level layer proposes another sub-goal . This process repeats until 1 ) the goal g is achieved , i.e . the robot finds the target object successfully ; 2 ) a predefined maximum number of atomic actions has been performed . For ROS , we define the sub-goal space as { approach obj|obj is visible in the robot ’ s current view } . We argue three reasons for the sub-goal space definition , a ) approaching an object that shows in the robot ’ s view is a more general and relatively trainable task shown by Ye et al . ( 2018a ) . It also aligns well with the goal of the hierarchical reinforcement learning by breaking down the task into several easier sub-tasks ; b ) approaching a related object may increase the probability of seeing the target object . As soon as the target object is captured in the robot ’ s current view , the task becomes an object approaching task ; c ) as also suggested by Kulkarni et al . ( 2016 ) , specifying sub-goals over entities and relations can provide an efficient space for exploration in a complex environment . Moreover , in case there is no object visible in the robot ’ s current view , we supplement one additional candidate sub-goal : non-goal-driven exploration . The atomic action space for the low-level layer is adopted with navigation in mind , namely { move forward / backward / left / right , turn left / right } . 2.2 EXTRINSIC REWARDS AND INTRINSIC REWARDS . We define two kinds of reward functions . The extrinsic rewards re are defined for our object search task , thus are goal dependent . Further , we also introduce the intrinsic rewards ri for the low-level sub-tasks . The intrinsic rewards are hereby sub-goal dependent . We specify the two reward functions respectively as follows . Extrinsic rewards re . Without loss of generality , to encourage the robot to finish the object search task , we provide a high extrinsic reward ( in practice , 100 ) when the robot reaches the final goal state . At all other intermediate states , the extrinsic rewards are set to 0 . Formally , ret ( st−1 , at−1 , st , g ) = 100 if and only if st is a goal state , otherwise ret ( st−1 , at−1 , st , g ) = 0 . Intrinsic rewards ri . To facilitate the robot perform the sub-task , i.e . approaching the object specified in the sub-goal sg which shows in the robot ’ s current view , we adopt the following reward function ( Ye et al. , 2018a ) as the intrinsic rewards . To be specific , let art be the area of the object in robot ’ s view at time step t ( known as st ) , the intrinsic reward rit ( st−1 , at−1 , st , sg ) = art if and only if art > art−1 , art−2 , ... ar0 , otherwise rit ( st−1 , at−1 , st , sg ) = 0 . Note that art can be easily calculated based on the robot ’ s detection outputs , e.g . the size of the detected bounding box . In practice , we normalize art to the range of [ 0 , 100 ] .

Summary: The paper proposes an intuitive 2-layer hierarchy for robotic object search. The high-level policy does subgoal selection, whereas the low-level layer handles explicit control. Notably, the low-level policy is trained to be aware of both the subgoal and the final goal. The authors conducted a series of ablations, demonstrating the value of training the low-level policy to be final goal-aware, and empirically demonstrated the strength of their method compared to other baselines.