ROS-HPL: Robotic Object Search with Hierarchical Policy Learning and Intrinsic-Extrinsic Modeling
Authors: Xin Ye
Shibin Zheng
Yezhou Yang
Venue:  (2020)
Keywords: Robotic Object Search
Hierarchical Reinforcement Learning

Abstract: Despite significant progress in Robotic Object Search (ROS) over the recent years with deep reinforcement learning based approaches, the sparsity issue in reward setting as well as the lack of interpretability of the previous ROS approaches leave much to be desired. We present a novel policy learning approach for ROS, based on a hierarchical and interpretable modeling with intrinsic/extrinsic reward setting, to tackle these two challenges. More specifically, we train the low-level policy by deliberating between an action that achieves an immediate sub-goal and the one that is better suited for achieving the final goal. We also introduce a new evaluation metric, namely the extrinsic reward, as a harmonic measure of the object search success rate and the average steps taken. Experiments conducted with multiple settings on the House3D environment validate and show that the intelligent agent, trained  with our model, can achieve a better object search performance (higher success rate with lower average steps, measured by SPL: Success weighted by inverse Path Length). In addition, we conduct studies w.r.t. the parameter that controls the weighted overall reward from intrinsic and extrinsic components. The results suggest it is critical to devise a proper trade-off strategy to perform the object search well.

Under review as a conference paper at ICLR 2020
ROS-HPL: ROBOTIC OBJECT SEARCH WITH HIERARCHICAL POLICY LEARNING AND INTRINSIC-EXTRING MODELING
Anonymous authors Paper under double-bling blind review
ABSTRACT
Despite significant progress in Robotic Object Search (ROS) over the recent years with deep reinforcement learning based approaches, the sparsity issue in reward setting as well as the lack of interpretability of the previous ROS approaches leave much to be desired. We present novel policy learning approach for ROS, based on a hierarchical and interpretable modeling with intrinsic/extrinsic reward set- ting, to tackle these two challenges. More specifically, we train the low-level policy by deliberating between an action that achieves an immediate sub-goal and the one that is better suited for achieving the final goal. We also introduce a new evaluation metric, namely the extrinsic reward, as a harmonic measure the ob- ject search success rate and the average steps taken. Experiments conducted with multiple settings on the House3D environment validate and show that the intelli- gent agent, trained with our model, can achieve better object search performance (higher success rate with lower average steps, measured by SPL:: Success weighted by inverse Path Length). In addition, we conduct studies w.r.t. the parameter that controls the weighted overall reward from intrinsic and extrinsic components. The results suggest is critical to devise a proper trade-off strategy to perform the ob- ject search well.
INTRODUCTION
Robotic Object Search (ROS) is a task where an intelligent agent (a.k.. robot) is expected to take reasonable steps approach the user-specified object in an unknown indoor environment. ROS is an essential capability for assistant robots and could serve as the enabling step for other tasks, such the Embodied Question Answering (Das et al. 2018a).. Recently, (deep) reinforcement learning (RL) has demonstrated its power at enabling robots with autonomous behaviors (Arulkumaran et al. 2017), , such as navigating over an unknown environment (Mirowski al., [016 Zhu et 2017)) manipulating objects with robot's 's end effectors (Gu et al., 2017, Popov et al. 2017 Rajeswaran al. 2017). and motion planning (Chen et al.) [201)] 7 Everett et al. (018).. Under the RL setting, robot learns behavioral policy by maximizing the expected rewards that are estimated by interacting with the environment physically and/or virtually. The estimated rewards serve as the reinforcement
signals for robot to update its policy.
well-known challenge to train agent to perform ROS with RL is the sparse reward issue, due to the fact that the environment and/or the location of the target object are typically unknown. With well- defined reward functions, such as the ones in Atari games (Mnih et al. 2015). the agents are shown to achieve extremely promising performance. However, it is well-known challenge define the reward function under the real-world scenarios(Abbedd & Ng 2004). Typically, for the real-world applications such as object search or target-driven visual navigation, prior research prefers con- struet the reward function in terms of the distance between the robot's current location and the target location under the assumption that the full information of the environment is known (Mousavian et al., 2018 Waag et al.) 2018b a). Given an unknown environment, straightforward way is to set high reward when robot reaches the final goal state while at all other intermediate states, the reward is either zero or with a small negative value (Zhu et al. 2017). More recently, Ye et al. (2018b)
Our code and models he found in https: /XXx XXX/XXX

---

Under review as a conference paper at ICLR 2020
Top-down 2D Map
Estrinsic rewards Encouragin ng the robal to schleve the goal
Intrinsic rewards Encouraging the robot to achieve the sub-gosl
Approach
clock
Goal: music
Approach
Subrgaali
Agproach
music player
Figure I: Left: An illustration of the ROS-HPL framework. The robot puts a of its effort in maxi-
mizing the intrinsic rewards and achieving the nub-goal, and 1 of its effort in maximizing the extrinsic rewards and achieving the goal. Right: Trajectory generated from our method (a = 0.8) for searching the target object music player under our SSTTING-STSE.
presented relatively denser reward function which based on the bounding box of the target ob- ject from robot's detection system, the reward is still not defined among the situations that the target object is not detected. Thus, learning in a sparse reward setting (reward only defined for a small subset of states) is fundamental challenge for RL agents to perform ROS well, especially dealing with real-world scenarios and in complex environments.
To overcome the sparsity issue in reward settings, hierarchical RL has shown its superiority un- der the sparse reward setting (Kulkarni et al. 2016 Le et al.] 2018: Levy et al., 2018). It aims to learn multiple layers of policies, in which the higher layer breaks down the task into several easier sub-tasks and proposes corresponding sub-goals for the lower layer to achieve. This kind of hierar- chical policy is biologically plausible and intuitive. For human beings, when asked accomplish a complicated task, we rarely plan over the atomic steps directly towards the target goal. Instead, we plan through sequence of semantic meaningful sub-goals. Motivated by this observation, we put forward a novel two-layer hierarchical policy learning paradigm to deal with the sparse reward challenge.
One way to contruct the hierarchical policy learning is by adopting the option-critic framework (Bacon et al. 2017) to generate the sub-goals automatically. By specifying the number of desired sub-tasks or sub-goals, the option-critic framework can automatically abstract the whole action se- quences into the specified number of the sub-goals. However, the automatically generated sub-goals typically lack semantically meaningful senses, thus are difficullt for humans understand and hurts the overall system's 's interpretability.
In the other way of constructing the hierarchical policy learning, shown in work (Kulkarni al. 2016 Das et al. 2018b), the sub-goal space could be designated from human knowledge with se- mantic meaningful terms. Constructing the hierarchy following human knowledge is intuitive and interpretable. Whenever the high-level layer proposes a sub-goal from the sub-goal space, the low- level layer then aims to achieve it without considering the overall goal again. However, although their results are promising for certain tasks, it is worth to mention that there is no guarantee the prior human knowledge is always perfect. The low-level layer policy which focuses only on achieving the sub-goal proposed by the high-level layer policy without keeping the final goal in mind, may drift away from being ultimately successful. This is equivalent the scenario that without the long- term goal to provide human beings clear direction and focus down the road, our short-term achievements are not likely to add up to something substantial.
Taking the advantages from these two ways, we put forward a system with a higher level of gener- alization capability upon previous work (especially, [acon et al. (0117) and Kulkarni et al. (2016) and present a novel framework dubbed as ROS-HIPL: Robotic Object Search with Hierarchical Pol- icy Learning and Intrinsic-Extrinsic Modeling, to mitigate the aforementioned challenges (see Fig. left). Our contributions could be summarized in three-fold. First, fully utilize the prior knowl- edge provided by human to make the hierarchy interpretable. Our high-level layer plans over a sub-goal space from human knowledge in order achieve the final goal. Second, when sub-goal
2

---

Under review as a conference paper at ICLR 2020
is proposed by the high-level layer, the low-level layer learns the action policy by deliberating be- tween achieving the immediate sub-goal and the action that is helpful for achieving the final goal. Third, we build our framework with a novel intrinsic and extrinsic rewards setting. Though the wards in this paper are designed for object search, the method itself is general and can be applied to other tasks. To validate ROS-HPI, we conduct extensive sets of experiments iet al. 2018) simulation environment, and report the observed results that validate the efficiency and efficacy of the proposed system over other state-of-the-ar systems for object search.
the House3D (Wul
OUR APPROACH
First, we define the robotic object search (ROS) task. Formally speaking, when a target object is specified and provided with an image, the robot is asked to search and approach the object from its random starting position. The RGB image from the robot's on-board camera the only source of information for decision making. None of the environment information, such as the map of the environment or the location of the target object could be accessed. Once the area of the target object in the robot's viewpoint (the image captured by its camera) is larger than a predefined threshold, the agent stops and we consider it as a success. In this work, we present a novel two-layer hicrarchical policy for the robot to perform the object search task, motivated by how human beings typically conduct object search. In the following sections, we first describe the hierarchy of policies. Then introduce two kinds of reward functions, i.e. extrinsic rewards and intrinsic rewards, and we make use of these two reward functions to formulate the solution. Finally, we describe the network architecture adopted for learning the two-layer hierarchical policy.
2.1 HIERARCHY OF POLICIES
Our hierarchical policy has two levels, a high-level policy Ta and low-level policy m. At time step I, the robot takes the image captured by its camera as the current state 8t. Given a target object or goal g, the high-level layer proposes a sub-goal sgt TA (sglss 9) and the low-level layer takes over the control. The low-level layer then draws an atomic action ag ~ Ty (alast, Sr. g, sg.) to perform. The robot will receive new image/state 8t+1- The low-level layer repeats N₂ times till 1) the low-level layer terminates itself achieving the termination signal term(s:-N, 9, age); 2) the low-level layer achieves the sub-goal sg.. Either way, the low-level layer terminates at state Nt+N,, and then returns the control back to the high-level layer, and the high-level layer proposes another sub-goal. This process repeats until 1) the goal g is achieved, i.e. the robot finds the target object successfully; 2) a predefined maximum number of atomic actions has been performed.
For ROS, we define the sub-goal space as {approach objlobj is visible in the robot's current view). We argue three reasons for the sub-goal space definition, a) approaching an object that shows in the robot's view is a more general and relatively trainable task shown by Ye al. (2018a). It also aligns well with the goal of the hierarchical reinforcement learning by breaking down the task into several easier sub-tasks; b) approaching a related object may increase the probability of seeing the target object. As soon as the target object is captured in the robot's 's current view, the task becomes an object approaching task; c) as also suggested by Kulkarni et al. (2016). specifying sub-goals over entities and relations can provide an efficient space for exploration in complex environment. Morcover, in case there is no object visible in the robot's current view, we supplement one additional candidate sub-goal: non-goal-dre -oal--diven exploration. The atomic action space for the low-level layer is adopted with navigation in mind, namely {move forward / backward / left / right, turn left / right).
2.2 EXTRINSIC REWARDS AND INTRINSIC REWARDS
We define two kinds of reward functions. The extrinsic rewards r are defined for our object search task, thus are goal dependent. Further, we also introduce the intrinsic rewards r² for the lowelevel sub-tasks. The intrinsic rewards are hereby sub-goal dependent. We specify the two reward functions respectively as follows.
Extrinsic rewards r Without loss of generality, to encourage the robot to finish the object search task, we provide high extrinsic reward (in practice, 100) when the robot reaches the final goal state. At all other intermediate states, the extrinsic rewards are set to 0. Formally, of 9,-1, St, g) =
100 if and only if 8; is goal state, otherwise rf (st-1, at 8t, 0.
3

---

Under review as a conference paper at ICLR 2020
Intrinsic rewards r'. To facilitate the robot perform the sub-task, i.e. approaching the object specified in the sub-goal sg which shows in the robot's s current view, we adopt the following reward function (Ye et al.] (2018) as the intrinsic rewards. To be specific, let ar, be the area of the object in robot's s view at time step /. (known as 8t), the intrinsic reward (8:- 0t-1, (t.88)))) sg) ar, if and
only if ar, > ar,-1, 20r -rr otherwise ri (, at-1; 8t; sg) = O. Note that ary can be easily calculated based on the robot's detection outputs, e.g. the size of the detected bounding box. In practice, we normalize ar; to the range of [0, 100).
2.3 MODEL FORMULATION
We formulate the ROS task in terms of the two rewards introduced in Sec. 2.2 starts from an initial state so, it proposes a sub-goal sgo aiming to achieve the final goal g (locat- ing and approaching the target object). To achieve the final goal, we can optimize the discounted cumulative extrinsic rewards, expected over all trajectories starting at state su and sub-goal sgu, which EI>;= >'rE+1 |so I, sgo]- If and only if the robot takes minimal steps to the goal state, the discounted cumulative extrinsic rewards are thus maximized.
When the robot
The discounted cumulative extrinsic rewards also known as the state action value Qf Suttonn & Barto 2018) for our high-level layer, i.e. E. =t= > y'r 74 |80 = 8. 9, $90 = sg) Q% (8. g. sg).
Following the option-critie framework (Bacon et al. 2017), we unroll the Q% (s, g, sg) as,
8 Ti (als, g, +g)
(8,
Σ
ri 9, =g. ao a
Σ (i g, ag) Qf g. ag,a).
where the state action value (s, sg. a) for our low-level layer is the discounted cumulative extrinsic rewards after taking action a under the state s, goal g and sub-goal sg. Given the transition probability P(s'|s. is, a) which denotes the probability of being state s' after taking action at state s, Qi (s.g, 9.s.gg sg. a) can be further formulated as,
Qi 9, 80, a) Σ (( (8, >U( sg, sg.8 8'))
Ulg. =(1 (1-term(s', ,9, (g) (s..)))))))
term(s' .0 ('',
(i(s'' = Σ (sg' 9 sg').
(2)
We parameterize (a(as, 9, sg) and term(s, g, sg) as 0z and 8, respectively. According to Bacon et al. (2017), On, and Og can be updated by Equation 3 and to optimize Q (s, g. sg). We refer interested readers to Bacon al. (2017) for more details.
Oz + Oz, Von, log (als, (g) (ateeaaa-@(.s...e...... sg)).
(3)
0. - 0, - te,terms, (8', (Q) .g (f(.''g))
(4)
In addition, we adopt Q-learning method MMnih et al. 2015) to learn Q (s, sg) and generate sub- goal sg arg max,g Qi (s, g, 8g). We parameterize Q (s, 8g) with the and update its value towards the 1- 1-step extrinsic return Ri (s, .',.s)''))))))) + UUn, on (g. sg, s'), and consequently the can be updated as follows.
One + One Vene [A - Qa.. (s. 9. sg))²
(5)
For the low-level policy layer, when we build the hierarchical policy from the sub-goal space, we inherently assume that achieving the sub-goal is helpful for achieving the final goal. We define the sub-goal as approaching a related object by doing so the robot is more likely to observe the target one. Without loss of generality, we introduce low-level layer puts in achieving the sub-goal. Meanwhile, the low-level layer keeps (1 - a) of its effort for achieving the final goal, which yields a mixed overall loss.
[0, 1] as the proportion of how much effort the
4

---

Under review as a conference paper at ICLR 2020
Similarly, to learn the desired policy for the low-level layer, we optimize the discounted cumulative hybrid rewards, E EI 71((1 a)rf+1 + ar(+1) So. sgo, au]. We denote it as Qi 9, $9, a),
and further represent it by Qf (s, 9, 8g, a) and (8 g, 8g, a) as in Equation 6
Qi(3,g; 8g. a) E
((1
+ 00
t=
=(1 R
7' 7 = sg,00 = a]+
(6)
t=0
Y 180 =, 9.500 eg. 00
=(1 - a0) (s, (8, sg. + aaiiss
As result, to optimize Qi 9. 8g, a), one. With the policy gradient method, On, can be updated using Equation 7 and Equation g respec- tively (Mnih al., (016))
optimize (s, 9, sg. a) and Qj 9. sg, one by
8m, 5 Pir + Vor, log me., (als,g. (g) 9, a) (s.,,
(7)
Oz, r oN, + Vor, log To,, (als, g. (q) sg. a) - Ví' (g)).
(8)
Note that (7 sg) = E. s.g. g. sf) g. sg. a) by definition, therefore (s, g. sg) = 2% (s. sg) according to Equation [ Thus, the update of Oz, in Equation 7 for optimizing (f. g, 8g, a) is exactly the same as that in Equation 3 for optimizing (s, g, 8g). At last, we combine the two updates of on according Equation 6 Equation gives the final update.
Oz, c 0z, + Von, logto,, (als,g, **))*
(11a)(Qi (g. (g.g
(9)
a aQQ(( 9, a) (s()).
To conclude, we adopt Equation 4 5 and to update Or, One and On, for learning termis, g, sg).
Qi (s, 9, sg) and miaals,9 g, 8g) respectively.
2.4 NETWORK ARCHITECTURE
Feuture Roproserta
H
Tiga imAnt plaser)
656
Sub-goal
High-level
Ourent
Low-level
Move Forward Move Left
Decoder
Actininiliyy
Atenion Mark
Crtto
Vi
Goal
Samer to Segnwnstion
Concatmatin
Alant Enn Meas
To ritinationn Signa demsdr.....o p.sg)
Fully Layee
Concodittoona Lagaz
Dopth
Figure 2: Network architecture of our hierarchical reinforcement learning model.
Since the image captured by the robot's camera serves as the robot's current state, we adopt deep neu- ral networks to handle this high dimensional inputs and to approximate terml 8, g, sg) Q (s, g, sg) and Tr(a) s, g, sg). To update the model parameters according to Equation 4
5 and we still need
the value of (5 g). (s, sgs. a). (i g. sg. a) and V (s. g, sg). Here, we use deep neural networks to approximate (s, 9, sg), and estimate the remaining (Equation 10)
Ví (s = max @g(
(s. sg. =r (s,a, ,' 7U(g s'), Q g. ag =r (s, s', sg) + V V/ sg).
(10)
5

---

Under review as a conference paper at ICLR 2020
Fig. illustrates network architecture. We first extract multiple feature representations from the encoder-decoder network (shown in Fig. 2 upper right corne) for the image inputs. Our high- level network takes the outputs from the encoder as the feature representations of the robot's current view and the target object image accordingly. The two feature representations are then fed into two different fully connected layers respectively and the outputs are then concatenated into joint vector before attaching another fully connected layer to generate an embedding fusion. We feed the embedding fusion into one additional fully connected layer to approximate Qj (s, g, sg). For low-level model, we take the outputs of the two decoders (a semantic segmentation decoder and depth prediction decoder) as the feature representations of the robot's current view. Based on the goal (i.e. the target object) and the sub-goal proposed by the high-level network, we generate two attention masks using the semantic segmentation module. Note that if the sub-goal is proposed as the candidate non-goal-dre -o-o-----ddenn exploration, there is no sub-goal attention in hope that the robot can learn to explore. We take the two attention masks, as well as the depth map as the inputs to our low-level network, which shares similar architecture as network has three branches attached after the embedding fusion for approximating Ty T((ass 8, g, &g), V7 (s. 9. sg) and term(s, 9: sg) respectively. Each branch consists of two fully connected layers.
high-level network except that the low-level
5 and 9 to learn term(s, 9. sg). 2% (s. g. sg) and IT (als, g. sg). In ad- dition, to learn (7, g. sg), we update the estimated value towards the 1-step intrinsic return Ri r', a, , sg) + Vo,, (s', g. sg). As a result, the parameter O could then be updated as,
We follow Equation
Ou + 0/ Vo,, [R] - -Vo, (8, s.))
(11)
3 EXPERIMENTS
3.1 DATASET
2018). House3D con- We validate our framework on the simulation platform House3D (Wu et al. sists of rich indoor environments with diverse layouts for virtual robot to navigate. In each indoor environment, variety of objects are scattered at many locations, such as television, sofa, desk. While navigating, the robot has a first-person view RGB image as its observation. The simulator also provides the robot with the ground truth semantic segmentation and depth map corresponding to the RGB image. The RGB images, as well as the semantic segmentation and depth maps can be used the training data to learn the encoder-decoder network (shown in Fig. 2 upper right corner) for the feature representation extraction as we mentioned in Sec. Sec.2.4] In addition, the trained model, specifically the semantic segmentation prediction, can be used the robot's detection system.
To validate our proposed method in learning hierarchical policy for object search, we conduct the experiments in indoor environments where the objects' placements are in accordance with the real- world scenario. For example, the lelevision is placed close to the sofa, and is likely occluded by the sofa at many viewpoints. In such a way, to search the target object television, the robot could approach sofa first to increase the likelihood of seeing the television.
We consider discrete actions for the robot to navigate in this environment. Specifically, the robot moves forward / backward/ / left / right 0.2 meters, or rotates 90 degrees every time. It also discretizes each environment into a certain number of reachable locations, shown in Fig.
3.2 EXPERIMENTAL SETTING
We compare the following methods and variants:
RANDOM. At each time step, the robot ignores its observation and performs random action.
HIGH-LEVEL ONLY. We remove our low-level network and train the high-level network to predict the state action value (s, g, a) with the DQN method (Mnih et al. 2015).
LOW-LEVEL ONLY. Only the low-level network is trained to generate an action policy for object search without hierarchy. Since there is sub- aub-goal provided to the low-level network, we further block the sub-gu goal input channel. A recent work from Ye et al. (2018a) shows that it serves as a stronger baseline than Zhu et al. (0017).

---

Under review as a conference paper at ICLR 2020
OPTION-CRITIC (Bacon et al. 2017). This method aims build the hierarchy automatically. learns the desired policy by maximizing the discounted cumulative extrinsic rewards, while no intrinsic rewards are involved. When the high-level network proposes sub-goal (option), the corre- sponding option policy is learned by maximizing the extrinsic rewards and acting towards the goal. The option policy stops according the predicted termination signal. Here, the sub-goal doesn't t have much interpretability, it can be seen as a temporal abstraction.
HRL (Kulkarni al., 2016). Different from OPTION-CRITIC, this method works on both extrinsic rewards and intrinsic rewards. The high-level network maximizes the discounted cumulative ex- trinsic rewards reach the goal, while the low-level network maximizes the discounted cumulative intrinsic rewards to achieve the sub-goat that is proposed by the high-level network. The low-level network stops either when the sub-goal achieved, or predefined maximum number of steps have been performed.
HRL WITH STOP (Das et al. 2018b). This method is similar to HRL, while the low-level network has additional action stop in its action space. When the low-level network produces the stop action, it stops and returns the control to the high-level network. Otherwise, the low-level network continues until it has performed predefined maximum number of steps. Note that in Das et al. (2018b), the authors adopted an imitation learning method to warm-start the hierarchical policy learning. For fair comparison, we don't 't provide any imitation signals in our experiments, and learn the policy purely through the reinforcement learning method.
OUR METHOD follows See 2 To identify the role of the a value, compare multiple variants of our method, namely setting a = O, 0.2, 0.4, 0.6, 0.8, respectively, where the higher a value denotes the more effort the low-level network devotes achieving the sub-goal. It also worth to mention that when setting O, our method shares a similar idea with OPTION-CRITIC. However, our levelevel network still learns to approximate the state value V/ (s. g. sg) which is supervised by the intrinsic rewards. When setting a 1, our method is similar to HRL. Yet our low-level network learns a termination signal term(s, 9, sg). which is supervised by the extrinsic rewards.
Experiments are conducted under two settings:
SETTING-STSE (Singe Target in Single Environment). To compare the efficacy in learning the object search policy, we train each method to learn a policy for searching a specific target object. Without loss of generality, we randomly choose target object from a specific environment. We evaluate each method in terms of the network for searching the specific target object.
METTING-MTEE(Mutiinn Targets in Multiple Environments). To further evaluate, we train every method under different environments to search different target objects. To be specific, we randomly choose 24 target objects different environments (6 each) as our training data. We evaluate each method in terms of the network with the 24 different environment--arget object pairs being taken as the inputs randomly.
For each experiment, periodically increase the distance between the target object and the robot's 's starting position during training time which is also known as the curriculum learning paradigm. During testing time, we randomly choose 100 starting positions. We set the maximum number of all atomic action steps to 1000. For method HRL and HRL WITH STOP, we set the maximum number of each low-level steps to 50 to ensure the proposed sub-goals either it reaches the goal state (success case) or it runs out of the maximum number of all atomic action steps (failure case). We implement our algorithm using Tensorflow toolbox and conduct all the experiments with Nvidia V100 GPUs and 16 Intel Xeon ES-2680 v4 CPU cores.
be achieved. The robot stops
3.3 EXPERIMENTAL RESULTS AND DISCUSSION
Since we formulate the object search problem as maximizing the discounted cumulative extrinsic rewards, we take the Average discounted cumulative extrinsic Rewards (AR) as one of the evaluation metries, calculated by:
N Σ 1 (success))#ater #steps 100,
(12)
N
where 7 E (0, is the discount factor. From the perspective of the evaluation metric, it can also be seen as trade-off between the success rate metric and the average steps metric. With the higher
7

---

Under review as a conference paper at ICLR 2020
Top-down 2D Map
Top-down 2D Map
15
Random walk
Approach table chair
Approach sofa
Approach table
Approach player
Approach music player
OPTION-CRITIC (Bacon 2017
HRI.
Kulkarni
2016
Figure 3: Trajectories generated from OPTION-CRITTL and HRL for searching the target object music player from the same starting position under the SETTING-STSE.
value of 7, the average steps metric weighs more, and vice versa. In our experiments, we set 0.99.
In addition, we also report the following widely used evaluation metrics. Success Rate (SR). Average Steps over all successful cases (AS). Success weighted by inverse Path Length (SPL) (Anderson et 2018a), which calculated as N Σi= S: mar(t..p Here, S; is the binary indicator of success in episode i, 4 and are the lengths of the shortest path and the path actually taken by the robot. We adopt the number of the action steps as the path length. As result, SPL also trades-off' success rate against average steps.
Fig. reports the training performance (i.e. AR and SPL) of our method with different a values under SETTING-STIE -SSS.. From Fig. 4 we observe that the model with = 0.8 achieves much bet- ter performance, indicating that proper trade-off between achieving the goal and the sub-goal is needed for our hierarchical method to perform the object search well. When a 0, the model performs worst. It is because that the low-level layer needs to choose the actions that are suited for achieving the goal, and as a result the learning process still suffers from the sparse extrinsic rewards setting. When a = the model performs much better at the earlier training stage where the target object is closer robot's starting position. We explain it when generated by the low-level network are fully driven by the intrinsic rewards order to achieve the sub-goals. When the target object nearby, the proposed sub-goals are more likely be consistent with the final goal and thus achieving the sub-goals is helpful for achieving the goal. While the tar- get object is far away, the proposed sub-goals much more noisy. Then focus only achieving the sub-goals without keeping the final goal in mind may hurt the performance.
the navigation actions
a-C
0.4
0.8
0.35
40
0.2
0.6
a=1
average rewards
0.30
0.70
averag
20
0.15
0.10
10
0.05
0.00
0
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
iterations
# iterations
Figure 4: The average discounted cumulative extrinsic rewards (left) and average SPL (right) of our method with different values under SETTING-STSE.
8

---

Under review as a conference paper at ICLR 2020
Table 1: The performance of all methods under SETTING-STSE and SETTING-MTME.
SETTING-STSS
BETTING-MTME
Method
SR1
AS4
SPLt
ART
SR
AS4
SPLT
ART
ORACLE
1.00
26.74
1.00
77.72
1.00
22.42
1.00
81.39
RANDOM
0.16
357.56
0.01
3.91
0.23
372.65
0.01
4.07
HIGH-LEVEL ONLY
0.08
71.25
0.04
5.51
0.21
202.71
0.03
8.18
LOW-LEVEL ONLY
0.16
315.75
0.03
6.57
0.25
355.40
0.01
4.93
PPTION-CRITIC
0.25
388.28
0.02
4.74
0.28
429.21
0.01
4.37
HRL
0.58
359.52
0.08
11.03
0.35
340.34
0.05
10.00
HRL WITH STOP
0.66
338.33
0.13
18.02
0.36
370.37
0.04
5,80
Ours
= 0.8
0.77
289.56
0.20
23.34
0.34
208.32
0.07
12.96
We set a = 0.8 in the following experiments. Table [ shows the comparison of all methods under both settings. (Ieft) shows that under SETTING-STSE, our method (a 0.8) outperforms all previous methods in terms of SR. SPL, AR, and fairly in terms of AS since HIGHLEVEL ONLY method doesn't 't perform well in general observe that OPTION-CRITE doesn't 't perform well as other hierarchical methods (HRL, HRL WITH STOP and our method), indicating that the prior knowledge provided by human is vital in guiding the robot to perform the challenge tasks. Compared to HRL WITH STOP, HRI. doesn't stop the robot more valuable state timely and for which the performance worse. However, simply add stop action in the atomic action space is not much efficient when compared with our method. Under the more challenge SETTING-MTM method achieves higher SPL and AR that both trade-off the SR against AS,
its low SR, SPI. and AR suggest. Moreover,
also
though our method has a slightly lower SR and higher AS
from Table (right), which also fairly demonstrates the superiority our method.
We further depict sample qualitative results in Fig. Fig.D(ri (right) and Fig. which show that our method yields a more concise trajectory compare to other methods for the ROS task.
RELATED WORK
Our work is closely related to two major research thrusts: hierarchical reinforcement learning and target-driven visual navigation.
Hierarchical reinforcement learning. Previous work has studied hierarchical reinforcement learn- ing in many different ways. One is to come up with efficient methods to accelerate the learning process of the general hierarchical reinforcement learning scheme. As Nachum et al. (2018b), off- the authors introduced off-policy correction method. Levy al. (2017) and Levy al. (2018) proposed to use Hindsight Experience Replay to facilitate learning at multiple time scales. Though these methods' performance are impressive, they typically assume the sub-goal space for the higher level policy is subspace of the state space. However, in the ROS task, the RL system takes the image the state representation, these methods are not directly applicable since the higher layer can hardly propose an image as sub-goal for the lower layer to achieve.
Other methods designate separate sub-goal space for hierarchical reinforcement learning. For example, Kulkarni et al. (2016) defined the -ob- space in the space of entities and relations, such as the "reach" relation they used for their Atari game experiment. Sub-tasks and their relations were provided as inputs in Andreas et al. (2017) Sohn et al. (2018). Closer related to our work, Das et al. (2018b) adopted (exif-room, find-room, find-ohject, answer) as the sub-goal space to learn hicrarchical policy for the Embodied Question Answering task. For the same task, Gordon ct al. (2018) chose {navigate, scan, detect, manipulate, answer) as the possible sub-tasks, while the reinforcement learning methods were mainly applied for learning high-level policy. More recently, Gordon et al. (209)) integrated symbolic planning as an additional sub-task.
On the other side, attempts have been made to learn the sub-tasks automatically. These sub-tasks are referred to as temporal abstractions. Bacon et al. (2017) proposed the option-critic framework to autonomously discover the specified number of temporal abstractions. Osa al. (2019) learned

---

Under review as a conference paper at ICLR 2020
the temporal abstractions through advantage-wer weighted information maximization. Nachum et al. (2018a) addressed the uub-goal representation learning problem. With the learned representation, their hicrarchical policies are shown to approach the optimal performance within a bounded error.
Motivated by these works, we fully utilize the human specified sub-goal space to make the hicrarchy better interpretable. Meanwhile, to avoid being misled by the possibly imperfect human knowledge, we also leverage the benefits from the automatic temporal abstraction learning methods, which yields a hybrid one.
Target-driven visual navigation. Deep reinforcement learning has been studied extensively for the target-driven visual navigation task. Mirowski al. (2016) proposed learning depth prediction and action policy jointly for visual navigation. 'Zhu et al. (2017) introduced a target-driven deep reinforcement learning framework to enable a robot reach an image-specified location. The Em- bodied Question Answering or Interactive Question Answering task is introduced and addressed in
Das et al. (201 ( et al.] (2018b) Gordon al. (2018), Gordon al. (2019). Yu et al.. (2019)
and Wijmans et al. (2019), where the robot needs to navigate environment to answer an uncon- strained natural language question. [uupta et al. (2017). Ye et al. (2018b), Mousavian et al. (2018) and Ye et al. (2018a) focused on the robotic object search and/or object approaching task. Ander- sson et al. (2018b) Wang et al. (2018b) and Wang et al. (2018a) focused on natural language guided visual navigation. Among all of these cited work, most of them plan over the atomic actions for navigation (Mirowski et al. 2016; Zhu et al., 2017 Das et al., 2018a Wijmans et al.] 2219 Cupta
et al., 2017
Ye al.] [01 Mousavian et 2018 Ye al.) 2018a| Anderson et al. 201 8b
2018a) without hierarchical modeling, and thus lack interpretabilityy While Gordon al. (2018) and Gordon et al. (2019) studied hierarchical policies, they manually constructed the hierarchy with high-level and low-level controllers, and a standard reinforcement learning method applied on high-level layer decide which low-level controller invoke. Das al.] (2018b) also studied hierarchical reinforcement learning to generate explainable hierarchical policies. Along with these mentioned methods that build upon human specified sub-goal space, we further consider the likelihood that these manual sub-goals may not lead to the optimal steps towards the final goal, and we model it explicitly in our framework.
Wang et al.
More importantly, many of the previous works assume that the agent can access the full information of the environments during the training time, either by defining the reward function with the distance between the agent's current location and the target location (Mousavian et al., 2018 Wang et al, 2018b]a) a). and/or adopting shortest path as the supervised signal for pre- training (Das et al, 201 b Gordon al., 2018 Anderson et al., 2018b; Wang al. 2018a; Yu al., 2019). Nevertheless, for applications in real-world environments, collecting all the information is unarguably expensive and sometimes impossible. We would like to stress upon the point that our model does not assume
any environment information accessible even during the training process, which levels up the ROS' difficulty.
CONCLUSION AND FUTURE WORK
In this paper, we present novel two-layer hierarchical policy learning framework that builds on in- trinsic and extrinsic rewards. The framework fully utilizes the prior knowledge provided by human to make the hierarchy scheme interpretable. When the high-level layer plans over the human speci- fied sub-goal space to achieve the goal, the low-level layer plans over the atomic actions by taking both the sub-goal proposed by the high-level layer and the goal into the consideration. The frame- work is general and we validate it in the context of the object search task. The empirical experiments on 3D platform demonstrate the efficacy and efficiency of our proposed framework.
The current work also opens several avenues for future study, First, while we adopt fixed value in our experiments to indicate the importance of the sub-goals compared to the goal, the a value can be adaptively updated while approaching different sub- -ub-goals even be learned from common-sense knowledge. We intend to extend our work towards integrating top-down human knowledge together with the human specified sub-goal space facilitate the object search task with higher efficiency.
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. ACM, 2004.
Peter Anderson, Angel Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, al. On evaluation of embodied navigation agents. arXiv preprint arXiv: 1807::1177 2018a.
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE Con- ference on Computer Vision and Pattern Recognition, pp. 3674-3683, 2018b.
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 166-175. JMLR. org, 2017.
Kai Arulkumaran, Mare Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. A brief survey of deep reinforcement learning, arXiv preprint 1xXYv:1708.0866 2017.
Pierre-Lue Bacon, Jean Harb, and Doina Precup. The option-critic architecture. In AAAI. pp. 1726- 2017.
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In ECCV, 2018.
Yu Fan Chen, Michael Everett, Miao Liu, and Jonathan P How. Socially motion planning with deep reinforcement learning. In 2017 IEEE/RSJ International Conference on Intellgent Robots and Systems (IROS), pp. 1343-1350. IEEE, 2017.
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embod- ied question answering. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pp. 2054-2063, 2018a.
Abhishek Das, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Neural modular control for embodied question answering. arXiv preprint arXiv: 1810. / 181, 20188 8b.
Michael Everett, Yu Fan Chen, and Jonathan P How. Motion planning among dynamic, decision- making agents with deep reinforcement learning. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 3052-3059. IEEE, 2018.
Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering in interactive environments. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4089 4098, 2018.
Daniel Gordon, Dieter Fox, and Ali Farhadi. What should do now? marrying reinforcement learning and symbolic planning. arXiv preprint arXiv:1901.01492. 2019. 2019.
Shixiang Gu, Ethan Holly, Timothy Lillicrap, and Sergey Levine. Deep reinforcement learning for robotic manipulation with asynchronous off- off-policy updates. In 2017 IEEE International Confer- ence on Robotics and Automation (ICRA), pp. 1389-33996 IEEE, 2017.
Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cognitive mapping and planning for visual navigation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 261 6-6255,,, 2017.
Tejas D Kulkamn, Karthik Narasimhan, Ardavan Saeedi, and Josh Tenenbaum. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In Advances in neural information processing systems, pp. 3675-3683, 2016.
Hoang M Le, Nan Jiang, Alekh Agarwal, Miroslav Dudík, Yisong Yue, and Hal Daumé III. Hierar- chical imitation and reinforcement learning. arXiv preprint arXiv: 1803. 005000 2018.
11

---

Under review as a conference paper at ICLR 2020
Andrew Levy, Robert Platt, and Kate Saenko. arXiv:1712.00948. 2017.
Hierarchical actor-critie.
arXiv preprint
Andrew Levy, Robert Platt, and Kate Saenko. Hierarchical reinforcement learning with hindsight. arXiv preprint 1805:0805.081 2018.
Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukeuoglu, et Learning to navigate in complex environments. arXiv preprint arXiv:/6/1.0066 2016.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 51 518(7540):529, 2015.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937, 2016.
Arsalan Mousavian, Alexander Toshev, Marek Fiser, Jana Kosecka, and James Davidson. Visual representations for semantic target driven navigation. arXiv preprint arXiv: 06066, 2018.
Ofir Nachum, Shixiang Gu, Honglak Lee, and Sergey Levine. Near-optimal representation learning for hierarchical reinforcement learning. arXiv preprint arXiv:/810000/257, 2018a
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, pp. 3307-3317, 2018b.
information maximization. arXiv preprint 19X1v:1901.01366 2019. Takayuki Osa, Voot Tangkaratt, and Masashi Sugiyama. Hicrarchical reinforcement learning via
Ivaylo Popov, Nicolas Heess, Timothy Lillicrap, Roland Hafner, Gabriel Barth-Maron, Matej Ve- cerik, Thomas Lampe, Yuval Tassa, Tom Erez, and Martin Riedmiller. Data-efficient deep rein- forcement learning for dexterous manipulation. arXiv preprint arXhv:1704.03073 2017.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv: / 101117799000700 10087, 2017.
Sungryull Sohn, Junhyuk Oh, and Honglak Lee. Hierarchical reinforcement learning for zero-shot generalization with subtask dependencies. In Advances in Neural Information Processing Sys- tems, pp. 7156-7166, 2018.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised imi- tation learning for vision- vision-language navigation. arXiv preprint arXiv:/811.1000 2018a.
Xin Wang, Wenhan Xiong, Hongmin Wang, and William Yang Wang. Look before you leap: Bridg- ing model-free and model-based reinforcement learning for planned-ahead vision-and-lannuage navigation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 37-53, 2018b.
Erik Wijmans, Samyak Datta, Oleksandr Maksymets, Abhishek Das, Georgia Gkioxari, Stefan Lee, Irfan Essa, Devi Parikh, and Dhruv Batra. Embodied question answering in photorealistic en- vironments with point cloud perception. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6659-6668, 2019.
Yi Wu, Yuxin Wu, Georgia Gkioxari, and Yuandong Tian. Building generalizable agents with a realistic and rich 3d environment. arXiv preprint 1801v 0.2009209, 2018.
Xin Ye, Zhe Lin, Joon-Young Young Lee, Jianming Zhang, Shibin Zheng, and Yezhou Yang. Gaple: Generalizable approaching policy learning for robotic object searching in indoor environment. arXiv preprint arXiv: 088887 2018a.
12

---

Under review as a conference paper at ICLR 2020
Xin Ye, Zhe Lin, Haoxiang Li. Shibin Zheng, and Yezhou Yang. Active object perceiver: ionggnttion-uudder policy learning for object searching on mobile robots. In 2018 JEEE/RSJ In- ternational Conference on Intelligent Robots and Systems (IROS), pp. 6857-6863. IEEE, 2018b.
Licheng Yu, Xinlei Chen, Georgia Gkioxari, Mohit Bansal, Tamara L Berg, and Dhruv Batra. Multi- target embodied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.
2019.
Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), pp. 3357-3364. IEEE, 2017.
13

---

Under review as a conference paper at ICLR 2020
A NETWORK ARCHITECTURE AND TRAINING DETAILS
H
Tagget Ooper
Stake Action Valee (3
Sub-goal 0ft.a1
Highllevel
Cummit
Lov-leval
Decoder
Move Forward Move Left
Right
Action Pollty sg)
DH
Altrrian Mark
Criso
State Value Mosuarugp
Goal
Concatenatio
Segmanialin
Attent Mark
Fully Connecces Layors
Comeratarallagees
Degti Map
Figure 5: Network architecture of our hierarchical reinforcement learning model.
Feature representation network. The network is shown in Fig. 5 upper right corner. is trained to predict the semantic segmentation and depth map from a single RGB image. The network builds upon|Chen et al.](2210)) (2018) with an additional decoder branch to output the depth map. The two decoder branches share the the same architecture except that one decoder branch pipelines to a classification output layer for semantic segmentation and the other adopts a regression output layer for depth prediction. We adopt the cross entropy loss function for semantic segmentation and mean-square error loss function for depth prediction and train the network with the images sampled from 100 environments from House3D (Wu et al. 2018). The trained network is then deployed for extracting the feature representations and the semantic segmentation prediction is further adopted as the robot's detection system.
High-level network. The network is shown in Fig. upper part. The encoder of the feature rep- resentation network outputs a 256 dimensional feature for either target object image or the robot's current observation image. For each input stream, concatenate the features of 4 history frames and then project the 1024 dimensional vector down to a 512 dimensional vector. The two 512 di- mensional vectors from both input streams are then concatenated and aggregated into a singe 512 dimensional joint vector. The joint vector is further projected into 78 state action values. (78 is the number of the objects that being of our interest.)
Low-level network. The network is shown in Fig. lower part. We extract the two attention masks from the predicted semantic segmentation with the semantic label of the target object and the object specified in the sub-goal. The two attention masks and the predicted depth map are then resized to 10 by 10. For each input stream, we concatenate the features of 4 history frames and then project the 400 dimensional vector down to a 512 dimensional vector. The three 512 dimensional vectors from three input streams are then concatenated and aggregated into a single 512 dimensional joint vector. The joint vector is further passed through three branches. For all three branches, first project the 512 dimensional joint vector into a 20 dimensional vector. In the first branch, we further project the 20 dimensional vector into 6 atomic action policy outputs, i.e. probability over atomic actions. In the second branch, we project the 20 dimensional vector into a single state value output. In the third branch, project the 20 dimensional vector into single signal output termination, i.c. probability to terminate.
We train this hierarchical network with a RMSProp optimizer of learning rate 1 x 10-4, and reduce the learning rate by an order of magnitude every 1000 episodes.
B QUALITATIVE RESULTS
Fig. 6Jshows some qualitative results of our method. The robot can only access the first-person view RGB images. We show the top-down 2D map for better visualization. From Fig. [ that
14

---

Under review as a conference paper at ICLR 2020
the robot chooses to approach related object first before detecting the target object. Once the target object is detected, the robot then opts to approach the target object immediately.
approaching sofa
approaching picturn framm
First person View
Top-down 20 Map
First person View
Top-down 2D Map
approaching sofa
approaching frame
U
First person View
Top-down 2D Map
First par parson View
Top down 2D Map
approaching
approaching picture frams
approaching clock
approaching music
Pirst-pesoon View
Top-down 2D Map
Firs- son View
Top-down 2D Map
approeching
approaching
First person view
Top down 2D Map
First person view
Top down 20 Map
approaching music
approaching
(a)
(b)
15

---

Under review as a conference paper at ICLR 2020
down
down
appmaching fy stand
approsching by stand
V
y
First person View
Top down Map
First person kiew
Top down Map
aoprosching
approaching
First pertoperoon View
Top-down 20 Map
Fistoperaon View
Top-duae 20
approacting
Fandomm walking
person
approshinng Belovision
randem warkng
First person View
Top down Map
First person View
Top down Map
rpprosching
approsching
First serson View
Top down 20 Map
First person View
Top down Map
spproaching
spprosching
(c)
(d)
Figure 6: Trajectories generated by our method for the robot to search (a) music (b) music (c) television and (d) sofa.
16


Reviews:
[{"review_id": "SJx3B1Oucr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n------------------------------------------------------------------------------------\nRebuttal Response:\nThanks for the clarifications. Nevertheless, the rebuttal and the comments of the other reviewers did not convince me that this paper is ready for publication at ICLR and I keep my vote with weak reject. IMO this paper can be improved by either focussing more on the HRL part and performing simpler qualitative evaluations to highlight the HRL contribution OR by focussing completly on the robotics part by incorporating more classical robotics approaches and demonstrating their shortcommings within the experiments. \n\n------------------------------------------------------------------------------------\nSummary:\nThe paper proposes a hierarchical reinforcement learning scheme to search for objects specified by an image. The proposed learning approach is applied to a virtual house setting and compared against multiple baselines. \n\nI like that the authors do an extensive comparison of different baselines and compare their results. My main concern is the setup with the task, which seems quite artificial. Learning to search for objects using pure RL seems like neglecting all robotics research from the past 50 years. By now we can generate maps, planners and low-level control policies to navigate within these maps. Such approaches would be able to remember the objects location and just return to them and do not need to discover them 1000x times to remember them. Therefore, one would only need to learn an optimal search pattern. Therefore, I would like to see the proposed HRL approach in a more appropriate experiment or even more excitingly be combined with classical robotics. I think that this combination should be quite exciting. \n\nRegarding the HRL, could the authors please state their contributions in more detail? There is quite some work on subgoal generation within HRL. How does your work differ from these? \n\nCurrently, I am for borderline reject but I am happy to increase my rating during the rebuttal, when the authors clarify the motivation for the experiment and their contribution to HRL. \n\nMinor Comments: \nI think that the acronym is badly chosen. The term ROS is already famously coined within the robotics community for the Robot Operating System. Therefore, using this acronym for a robotics tasks is really confusing. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeFI71rcB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper proposes an intuitive 2-layer hierarchy for robotic object search. The high-level policy does subgoal selection, whereas the low-level layer handles explicit control. Notably, the low-level policy is trained to be aware of both the subgoal and the final goal. The authors conducted a series of ablations, demonstrating the value of training the low-level policy to be final goal-aware, and empirically demonstrated the strength of their method compared to other baselines.\n\nDecision: Weak Reject. The idea is intuitive and seems to be empirically successful (on some metrics). My primary concern is that the work appears incremental when compared to the baselines HRL and HRL with Stop. \n\nI think the acceptability of the paper is contingent on whether the tuning of alpha is considered a sufficiently significant contribution. The authors themselves noted that their method (alpha = 1) is similar to HRL---differing only in the introduction of a termination signal. This in and of itself suggests that the main contribution of the paper boils down to learning a suitable choice of alpha to manage the termination signal. \n\nI would also like to better understand the distinction between the author’s method versus HRL with Stop. Both methods have employ a low-level network capable of pre-emptive stopping. How, then, is the termination signal for HRL with Stop trained?\n\nIf the authors can convincingly demonstrate the novelty of the proposal to learn the terminal signal via extrinsic reward supervision, and if the other reviewers feel similarly convinced, then I would feel more comfortable re-evaluating my concerns about the significance of this work.\n\nI would also, in general, encourage a more thoughtful exposition of the results shown in Table 1. Can the authors posit/explain why, for example, High-Level Only performs so much better on AS than the other models, but so poorly on the other metrics? ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "Bke-0uyAtr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a hierarchical approach to perform robotic object search (ROS). \nThe idea is to use a high-level policy which produces subgoals and a low-level policy which produces atomic actions conditioned on both the subgoals and the true goal, and which is trained with a weighted sum of the original extrinsic reward and a reward for reaching the subgoal. Subgoals are consist of different objects in the field of view which the robot can choose to approach. \nThe approach is evaluated on the House3D dataset, where it is shown to perform well. \n\nRecommendation: weak reject. \n\nThis isn't a bad paper, but I'm not sure it will be of broad interest at ICLR.\nIt is very specific to ROS problem and House3D dataset and doesn't seem to propose a general algorithm which can be broadly applicable elsewhere. The mechanism for generating subgoals and training the low-level policy is very task dependent (subgoals are constrained to be objects in the field of view, the intrinsic reward for training the low level policy is dependent on the size of the bounding box of the object defining the subgoal). While this probably accounts for improved performance on the House3D dataset, I think the audience of ICLR would be more interested in a general approach which can be used in many different domains (even at the cost of performing less well on a specific domain than something more tailored). This paper may be a better fit for a robotics conference. \n\nAnother point concerning the experimental evaluation. Sparsity of the rewards is mentioned as a main motivation for the hierarchical approach. However, there are a number of methods which use exploration bonuses to address this issue (pseudocounts, random network distillation, ICM etc. [1, 2, 3]). At least one of these should be included as a baseline.\n\nSpecific comments:\n- using two letters for denote a single variable is confusing, since it seems like a product. I.e. using \"sg\" to denote a subgoal, \"at_t\" to denote area. Please use a single letter and add subscripts if necessary to disambiguate.\n- in the various equations, please use \"\\log\" instead of \"log\" so that it is not italicized.\n- bottom of page 4: \"Q-leaning\" -> \"Q-learning\"\n- page 2: \"way pf\" -> \"way of\"\n- please use more informative names for Settings A/B\n- First paragraph in Section 2: \"hierarchical policy for the robot to perform the object search, motivated by how human beings conduct object search\". Saying the method is similar to how humans behave is a fairly big claim that should be substantiated by appropriate references, or not made at all.  \n\n\nReferences:\n[1] https://arxiv.org/abs/1810.12894\n[2] https://arxiv.org/abs/1703.01310\n[3] https://arxiv.org/abs/1705.05363\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a two-level hierarchical reinforcement learning approach, applied to the problem of a robot searching for an object specified by an image.  The system incorporates a human-specified subgoal space, and learns low-level policies that balance the intrinsic and extrinsic rewards.  The method is tested in simulations against several baselines.

The reviewer discussion highlighted strengths and weaknesses of the paper.  One strength is the extensive comparisons with alternative approaches on this task.  The main weakness is the paper did not adequately distinguish between which aspects of the system were generic to HRL and which aspects are particular to robot object search.  The paper was not general enough to be understood as a generic HRL method. It was also ignoring much relevant background knowledge (robot mapping and navigation) if the paper is intended to be primarily about robot object search.  The paper did not convince the reviewers that the proposed method was desirable for either hierarchical reinforcement learning or for robot object search.

This paper is not ready for publication as the contribution was not sufficiently clear to the readers. 

Decision: Reject