Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper operates in a 2-agent setting, where one agent is a learning agent and the other agent is the assistant. The objective is for the learning agent to seek help from the assistant while solving a task, and the decision of what to ask the assistant and when to do so is made by learning an interaction policy using hierarchical reinforcement learning -- in particular using a variant of POMDPs. The assumption is that the learning agent's operational policy on the task environment is not changed and an additional interaction (hierarchical) policy is learned on top of the operational policy which dictates when and what to ask the assistant. The authors have empirically evaluated their approach on a (simulated) human-assisted navigation task, and show that the proposed method achieves close to seven times an increase in the success rate of solving the task compared to the policy obtained by learning without assistance. Additionally, the results also show that the frequency of seeking assistance was only 25 percent of all actions executed to solve the task. 
main_review: **Pros**:
* Writing and Presentation: The paper is well structured. All the graphs were well annotated.
* Modelling: The POMDP designed to obtain the interaction policy was succinct and rich enough to represent the assistive task. It presents a framework that can be extended further/utilized in several other multi-agent problems and opens up avenues for future research.
* Goal Stack and using multiple Sub-goals: The idea of introducing a goal stack and including sub-goals as a possible action for the interaction policy was very interesting. The generalization capabilities of the method on unseen environments also look promising.
* Limitations of the work are well addressed.

**Cons**:
* Experimental Evaluation: The experiments done on the Matterport3D simulator under three different conditions was exhaustive. However, it would have helped to know a few more details on the SUB action: how was the subgoal given to the agent during training -- particularly when the subgoal location was not adjacent /coincides with the agent's current location.  I did not fully understand this action as presented by the human during training. Experimental details were missing and given that the problem is heavily dependent on modeling the POMDP, it would have helped to know the details of the setup elaborately.

* Baselines: The rule-based baseline seemed to be particularly weak in the context of this work. Perhaps a slightly stronger baseline -- including some more domain knowledge could have led to interesting ablation studies.

* Typos in Notation: Typo on Page 4, where the subgoal function was first introduced -- the symbol used was \sigma_{A}.


**Additional Comments**:

I find the premise of the work very interesting and was curious to know if there were any thoughts on how the presented algorithm/model would perform (robustness of the technique to noise) in case there was noise induced in the feedback from the assistant -- like the assistant presented a noisy estimate of the current state description or a possible sub-goal?


**General Comments**:
Overall I liked the approach of learning interaction policies using the hierarchical RL and think that it presents a good starting point for future research in the direction of learning policies under assistance. The success rate on unseen tasks guided by sub-goals is particularly promising. There could have been some more clarity in the experimental setup (for training), and additional baselines used for evaluation. 

*Originality*:  Moderate

*Clarity*: Moderate

*Quality*: Good

*Significance*: Moderate to High
summary_of_the_review: The paper presents a good starting point for future research in the area and a model for application to several real-world problems such as assisted teaching (when to provide students help and how, for instance). However, the experimental section could be made clearer since the results presented look promising. An additional stronger baseline (though I understand this might be non-trivial to formulate) would have made the claims stand out. At the moment, both the baselines are weak (no assistance/rule-based assistance). Hence. I am inclined to reject the paper at the moment.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper formulates a hierarchical RL agent that learns to request information from human assistant, when task-critical information is missing. The assistant is assumed to be omniscent, present at all times, know the agent’s current state, know the agents goal state, and be able to accurately provide the requested information. On each step the agent has 5 types of actions: it can either issue one of the three information queries (request current state description, goal state description, subgoal description), or execute the highest value action according to its beliefs, or terminate execution. Such a framework can be applicable to situations where an RL agent is trained on a certain task, but deployed on a similar task with some information missing (for example, a missing object annotation). 
The framework is demonstrated on simulated human-assisted object finding task.  The paper shows that an agent equipped with a human assistant achieves an improved rate of successfully completing the task, compared to an unassisted agent. 


main_review: I am not convinced that this paper is ready for ICLR, it looks like this project is in an early stage with writing often unclear, results are presented without confidence intervals, in tables rather than figures. I am also not convinced of the strength of the contributions, even if the presentation was improved. The authors do a good job implementing existing algorithms in the context of the studied task, however the applicability of this framework seems to be very narrow. While reading this paper I kept wondering why is this a valuable research question, however I do not have a good answer.

A detailed list of my comments is below: 

1. The writing is often unclear -- for example:

 - Page 1: "On every step, the agent can ask for more information about its current state, the goal state, or a subgoal state..." "The agent learns when to convey an intent to the assistant..." 
this is confusing... so, does the agent convey intent, or ask questions? But the assistant is supposed to be omniscient, and always aware of the agent's intent?

 - Page 2:  states that agent has only a partial description of a goal state. So, does the uncertainty in this task reduce to uncertainty about the goal state? Toward the end of the paper I discovered that the uncertainty actually  comes from missing environment annotations. This needs to be clarified.

- Page 2: "The agent achieves this level of performance while issuing only 4.8 requests to the assistant on average, representing less than 1⁄4 of the total number of actions taken in a task execution." Why is this significant or relevant? The number of times the agent has to make requests is determined by how much information is missing in the evaluation task, compared to the task it was trained on -- which means that the exact numbers are arbitrary. Such references to arbitrary numbers happen in a few places through the paper.

2. Related work is very brief, and given in the end of the paper -- instead of at the beginning to motivate the current work. This makes it hard to evaluate the novelty of the contribution. 
The goal stack maintenance section could be moved to the supplement to make space for motivation of this work.

3. It appears that there are multiple goals at the same time in the goal stack - if so, how does the agent arbitrate between multiple goals? I expected that there should be only one goal, which can have subgoals, but this is not clear in the text.

4. A computational experiment is used to show that the ability to request information was useful to the agent. However, it seems obvious that an agent with less uncertainty will do better. Section 6 describes the model comparison as comparing the agent that requests assistance to an agent that takes random actions. However, with missing annotations a rational agent that can not request information should still be able to rationally search the graph until the object is found - which seems a more reasonable  performance benchmark.  Alternatively, the authors could compare the current framework to a state-of-the-art agent that can solve the same benchmark problem. Was the indoor object-finding task used in previous work, or as part of benchmark problem sets? 


5. Practical applications of this framework seem to be very limited, to a context of running RL agents with a near-perfect incomplete policy to evaluation tasks with missing annotations. 

6. What is the point of using the Matterport3D dataset, as a basis for environment graphs? This 3D nature of this dataset, or that it is made to resemble a living space, is not in any way leveraged by the current algorithm.

7. The data presented in tables would be better readable as a plot.

summary_of_the_review: I am not convinced that this paper is ready for ICLR, it looks like the project is in an early stage. The writing is unclear, results are presented in tables rather than figures -- and are mostly descriptions of model performance rather than model comparisons. I am also not convinced of the strength of the contributions, even if the presentation is improved. The authors do a good job implementing existing algorithms in the context of the given task, however the applicability of this framework is very narrow, to a context of running RL agents with a near-perfect incomplete policy to evaluation tasks with missing annotations. The paper claims that the agent that can request assistance from a human to reduce uncertainty performs better on the given task, than an agent that can not request assistance -- this claim is correct, however it also seems obvious.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a reinforcement learning framework where the agent is able to "ask for help" to receive from a human additional information to more easily solve the task. The paper is kinda Robotic Navigation-oriented by the way tasks and "additional information" about the environment are described, as well as in the empirical evaluation. It's uncertain if the paper presents any significant contribution due to major omissions on the related works list (explanation in the main review)
main_review: The authors present a "transfer learning" (TL) framework in which the agent is able to "ask for help" as one of the action options. While the authors propose an "unusual" modeling centered around POMDPs and robotic navigation (which could be good, bridging the gap between general TL for RL methods and this particular application), there has been a large amount of works with the same high-level idea in the last years. Yet, most of the related works are simply neglected and not even mentioned in the manuscript. 

The survey below summarizes dozens of related works that follow this same general idea, and not even the Survey itself is cited:

Silva, Felipe Leno, and Anna Helena Reali Costa. "A survey on transfer learning for multiagent reinforcement learning systems." Journal of Artificial Intelligence Research 64 (2019): 645-703.

In particular, works from the "action advising" area are very relevant to the present manuscript. The seminal Teacher-Student framework [1] introduced the idea of proactively "asking for help" when needed in the RL context. The RCMP advising method [2], based on Adhoc Advising [3], propose a more modern takeaway on how to compute the agent "uncertainty" (very relevant to the manuscript at hand). Works that focus on the specific points claimed as contributions in the manuscript also exist, such as considering as a MDP the problem of learning "when to ask for help" [4, 5], or adding a hierarchical component to task selection before the low-level resolution of the task [6].

[1] Torrey, Lisa, and Matthew Taylor. "Teaching on a budget: Agents advising agents in reinforcement learning." Proceedings of the 2013 international conference on Autonomous agents and multi-agent systems. 2013.

[2] Silva, Felipe Leno, et al. "Uncertainty-aware action advising for deep reinforcement learning agents." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 04. 2020.

[3] Silva, Felipe Leno, Ruben Glatt, and Anna Helena Reali Costa. "Simultaneously learning and advising in multiagent reinforcement learning." Proceedings of the 16th conference on autonomous agents and multiagent systems. 2017.

[4] Zimmer, Matthieu, Paolo Viappiani, and Paul Weng. "Teacher-student framework: a reinforcement learning approach." AAMAS Workshop Autonomous Robots and Multirobot Systems. 2014.

[5] Omidshafiei, Shayegan, et al. "Learning to teach in cooperative multiagent reinforcement learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33. No. 01. 2019.

[6] Kim, Dong-Ki, et al. "Learning hierarchical teaching in cooperative multiagent reinforcement learning." (2019).

I would expect that several of those related works were included in the empirical evaluation. Yet, none of them are even mentioned in the paper! At the very least the distinction between the contribution of the present manuscript and those related works should be discussed in deep, as well as an explanation of why those methods were not added to the empirical evaluation.

Due to the aforementioned major omission, I cannot judge the contribution of the paper.



-------------------
After Rebuttal
-------------------

Although I agree the method has its merits and contribution, the manuscript does a poor job in situating the proposed work in the current state-of-the-art literature. Most of the discussion period was devoted to explaining where the method differs from current literature, which should have been clear since the beginning by reading the paper. I am also not entirely convinced the method is that different from the action advising framework. Sure, the present manuscript chooses to extract more information from the advice by enforcing the advisor to be a human (more restrictions = more information to be extract from the advice at the cost of less flexibility), while most of those methods are very general and focus on information that exist in any RL environment (i.e., states and actions). Yet, the same general idea holds, and the present method is not the first one that extracted from advice more than action suggestions. Those similarities should be embraced and throughfully explained in the manuscript, as in my view the proposed method is a specialized advisor-advisee framework for this specific situation that matters most to the authors. Therefore, my grades are maintained so that the authors can improve the manuscript for the next submission.
summary_of_the_review: Paper investigates an interesting and relevant topic. However, there is a major omission of related works. Therefore, my recommendation is to reject the paper.
review_title: null

Meta-review: This paper is on the theme of active reinforcement learning with a human/assistant in the loop. Under partial observability, an agent acts as per an interaction policy that gathers state/goal information from the assistant, while an operational policy assumed pre-learnt in this paper executes low-level actions.  The reviews acknowledge the relevance of this topic and that the paper is well structured and coherently presented overall. However, there are unanimous concerns around experimental evaluation being unconvincing, lack of strong baselines and lack of thorough coverage of related work precluding an accurate assessment of claimed contributions. As such, the paper is not in a form that can be accepted at ICLR --  the authors are encouraged to revise their submission as per review feedback.
Decision: Reject