Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: 
------------------------------------------------------------------------------------
Rebuttal Response:
Thanks for the clarifications. Nevertheless, the rebuttal and the comments of the other reviewers did not convince me that this paper is ready for publication at ICLR and I keep my vote with weak reject. IMO this paper can be improved by either focussing more on the HRL part and performing simpler qualitative evaluations to highlight the HRL contribution OR by focussing completly on the robotics part by incorporating more classical robotics approaches and demonstrating their shortcommings within the experiments. 

------------------------------------------------------------------------------------
Summary:
The paper proposes a hierarchical reinforcement learning scheme to search for objects specified by an image. The proposed learning approach is applied to a virtual house setting and compared against multiple baselines. 

I like that the authors do an extensive comparison of different baselines and compare their results. My main concern is the setup with the task, which seems quite artificial. Learning to search for objects using pure RL seems like neglecting all robotics research from the past 50 years. By now we can generate maps, planners and low-level control policies to navigate within these maps. Such approaches would be able to remember the objects location and just return to them and do not need to discover them 1000x times to remember them. Therefore, one would only need to learn an optimal search pattern. Therefore, I would like to see the proposed HRL approach in a more appropriate experiment or even more excitingly be combined with classical robotics. I think that this combination should be quite exciting. 

Regarding the HRL, could the authors please state their contributions in more detail? There is quite some work on subgoal generation within HRL. How does your work differ from these? 

Currently, I am for borderline reject but I am happy to increase my rating during the rebuttal, when the authors clarify the motivation for the experiment and their contribution to HRL. 

Minor Comments: 
I think that the acronym is badly chosen. The term ROS is already famously coined within the robotics community for the Robot Operating System. Therefore, using this acronym for a robotics tasks is really confusing. 

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Summary:
The paper proposes an intuitive 2-layer hierarchy for robotic object search. The high-level policy does subgoal selection, whereas the low-level layer handles explicit control. Notably, the low-level policy is trained to be aware of both the subgoal and the final goal. The authors conducted a series of ablations, demonstrating the value of training the low-level policy to be final goal-aware, and empirically demonstrated the strength of their method compared to other baselines.

Decision: Weak Reject. The idea is intuitive and seems to be empirically successful (on some metrics). My primary concern is that the work appears incremental when compared to the baselines HRL and HRL with Stop. 

I think the acceptability of the paper is contingent on whether the tuning of alpha is considered a sufficiently significant contribution. The authors themselves noted that their method (alpha = 1) is similar to HRL---differing only in the introduction of a termination signal. This in and of itself suggests that the main contribution of the paper boils down to learning a suitable choice of alpha to manage the termination signal. 

I would also like to better understand the distinction between the author’s method versus HRL with Stop. Both methods have employ a low-level network capable of pre-emptive stopping. How, then, is the termination signal for HRL with Stop trained?

If the authors can convincingly demonstrate the novelty of the proposal to learn the terminal signal via extrinsic reward supervision, and if the other reviewers feel similarly convinced, then I would feel more comfortable re-evaluating my concerns about the significance of this work.

I would also, in general, encourage a more thoughtful exposition of the results shown in Table 1. Can the authors posit/explain why, for example, High-Level Only performs so much better on AS than the other models, but so poorly on the other metrics? 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposed a hierarchical approach to perform robotic object search (ROS). 
The idea is to use a high-level policy which produces subgoals and a low-level policy which produces atomic actions conditioned on both the subgoals and the true goal, and which is trained with a weighted sum of the original extrinsic reward and a reward for reaching the subgoal. Subgoals are consist of different objects in the field of view which the robot can choose to approach. 
The approach is evaluated on the House3D dataset, where it is shown to perform well. 

Recommendation: weak reject. 

This isn't a bad paper, but I'm not sure it will be of broad interest at ICLR.
It is very specific to ROS problem and House3D dataset and doesn't seem to propose a general algorithm which can be broadly applicable elsewhere. The mechanism for generating subgoals and training the low-level policy is very task dependent (subgoals are constrained to be objects in the field of view, the intrinsic reward for training the low level policy is dependent on the size of the bounding box of the object defining the subgoal). While this probably accounts for improved performance on the House3D dataset, I think the audience of ICLR would be more interested in a general approach which can be used in many different domains (even at the cost of performing less well on a specific domain than something more tailored). This paper may be a better fit for a robotics conference. 

Another point concerning the experimental evaluation. Sparsity of the rewards is mentioned as a main motivation for the hierarchical approach. However, there are a number of methods which use exploration bonuses to address this issue (pseudocounts, random network distillation, ICM etc. [1, 2, 3]). At least one of these should be included as a baseline.

Specific comments:
- using two letters for denote a single variable is confusing, since it seems like a product. I.e. using "sg" to denote a subgoal, "at_t" to denote area. Please use a single letter and add subscripts if necessary to disambiguate.
- in the various equations, please use "\log" instead of "log" so that it is not italicized.
- bottom of page 4: "Q-leaning" -> "Q-learning"
- page 2: "way pf" -> "way of"
- please use more informative names for Settings A/B
- First paragraph in Section 2: "hierarchical policy for the robot to perform the object search, motivated by how human beings conduct object search". Saying the method is similar to how humans behave is a fairly big claim that should be substantiated by appropriate references, or not made at all.  


References:
[1] https://arxiv.org/abs/1810.12894
[2] https://arxiv.org/abs/1703.01310
[3] https://arxiv.org/abs/1705.05363

review_title: Official Blind Review #3

Meta-review: This paper introduces a two-level hierarchical reinforcement learning approach, applied to the problem of a robot searching for an object specified by an image.  The system incorporates a human-specified subgoal space, and learns low-level policies that balance the intrinsic and extrinsic rewards.  The method is tested in simulations against several baselines.

The reviewer discussion highlighted strengths and weaknesses of the paper.  One strength is the extensive comparisons with alternative approaches on this task.  The main weakness is the paper did not adequately distinguish between which aspects of the system were generic to HRL and which aspects are particular to robot object search.  The paper was not general enough to be understood as a generic HRL method. It was also ignoring much relevant background knowledge (robot mapping and navigation) if the paper is intended to be primarily about robot object search.  The paper did not convince the reviewers that the proposed method was desirable for either hierarchical reinforcement learning or for robot object search.

This paper is not ready for publication as the contribution was not sufficiently clear to the readers. 

Decision: Reject