Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a framework for teaching agents using language advice. The framework consists of three stages: (i) grounding language advice to action, (ii) learning an unconditioned policy that mimics the advice-conditioned policy (iii) evaluating the unconditioned policy. Experiments are conducted in maze navigation environments. Results demonstrate the effectiveness of the framework compared with imitation learning and reinforcement learning.  
main_review: **Originality**: this work combines several pre-existing ideas, namely grounded language learning and policy distillation, in a novel way to enable teaching using language advice. It shares a lot of similarities with [6] (in hierarchical learning) and [3,4] (in learning to interpret advice) but the specific frameworks are not exactly the same. I would like to request comparison with these works in the paper and in the rebuttal. I also suggest reading [2] for a discussion on the drawbacks of reinforcement learning and imitation learning compared to a language-based learning framework. 

Missing related work:

[1] Learning Rewards from Linguistic Feedback. https://arxiv.org/pdf/2009.14715.pdf

[2] Interactive Learning from Activity Description. https://arxiv.org/pdf/2102.07024.pdf

[3] Vision-based navigation with language-based assistance via imitation learning with indirect intervention. https://arxiv.org/pdf/1812.04155.pdf

[4] Help, Anna! Visual navigation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning. https://arxiv.org/pdf/1909.01871.pdf

[5] Vision-and-Dialog Navigation. https://arxiv.org/pdf/1907.04957.pdf

[6] Hierarchical Imitation and Reinforcement Learning. https://arxiv.org/pdf/1803.00590.pdf

In fact, I strongly suggest the authors add more citations from the related work sections of these papers. 

**Quality**: 

The method is technically sound as it combines well-established methods. My concerns are mostly about evaluation:
- Using advice unit to measure communication efficiency oversimplifies comparison. A piece of language advice contains a lot of more information than a scalar reward. A scalar reward is represented by a 64-bit floating-point number in most machines; a 5-word advice in which each word is drawn from a 10^5-word vocabulary can be represented with log(10^5) * 5 ~ 80 bits; an action in a space consisting of 4 actions can be represented by only 2 bits. The bottom line is that communication efficiency of an advice should be based on the amount of information conveyed in the advice. A more sophisticated metric is to add the cognitive effort of humans when eliciting the advice but that may be too difficult to measure. Nevertheless, the advice unit metric in this work is overly naive and I do not advocate using it as a standard metric for later work to follow. 
- The results for imitation learning are absurdly low. Is it because the training set is too small? I request the authors to describe the dataset used for training with this baseline. I strongly suggest using "behavior cloning" instead of "imitation learning" because the latter is a much broader term. In fact, when the advice is an Action Advice, the approach is equivalent to DAgger [7], which is a form of imitation learning. I encourage the authors to clarify this point in the paper. 

[7] A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning. https://www.ri.cmu.edu/pub_files/2011/4/Ross-AISTATS11-NoRegret.pdf

**Clarity**: The paper is well-written and is easy to follow. However, I would like to suggest the authors merge the "bootstrapping with multi-level advice" with section 4.3 because they implement the same idea. In fact, this framework would be more general if the authors adapted a goal-conditioned RL framework (e.g. [8]). Then, this framework can be described as low-level policies being composed to form higher-level policies until the highest-level policies are learned. 

[8] Hindsight Experience Replay. https://arxiv.org/pdf/1707.01495.pdf

**Significance**: The paper studies an important direction: learning from language interaction, which enables non-expert humans to teach ML-based agents using their natural mode of communication. The presented framework is relatively general and can be potentially applied to a diverse range of tasks, especially that it captures the hierarchical structure of linguistically described tasks. 


limitations_and_societal_impact: The authors admit the limitations of the language and environments used in their experiments. The language is scripted and the environment is relatively simple (compared to the Matterport3D environment in [4]). I would like to point out several potential challenges when scaling this framework to more complex environments:
- Advice arises in contexts. In reality, we may not be able to ask humans to just "give me an advice" without grounding the question in the context of a task. Hence, it may make more sense to merge the grounding phase and the improvement phase as in [4], where humans give advice and also demonstrate the meaning of the advice at the same time so that the humans have contexts to give advice. 
- Giving advice too frequently could be tedious for humans. To reduce human effort, for example, [4] learns a policy that allows the agent to actively decide when it is necessary to ask humans for advice. 
- In complex environments, reinforcement learning could be too inefficient for language grounding. In that case, imitation learning could be used instead (as in [3, 4, 6]).
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a mechanism Coaching Augmented MDP for agents to learn from interactive expert feedback to avoid assumptions on reward functions and agent infrastructure. Here, structured advice is provided by experts to guide the agent learning process. A policy learning algorithm is designed for interpreting and acting on the advice to solve a problem. Experiments are conducted on discrete and continuous control problems to show how such mechanism outperforms traditional RL models.
main_review: Originality:
Though the proposed Coaching Augmented MDP is closely related to multi-task MDP and distillation, it still has novelty in processing more interactive feedback with sub-goals along with an un-conditional policy 

Quality
The paper proposes a framework for multi-task MDP with teacher in the loop using different forms of coaching functions called advice. They also propose an algorithm to perform exploration using these coaching signals while the actual task can be performed without these signals. The stage-wise learning helps to achieve this. The technical details of  these method are explained in detail in the paper. 
Experiments are conducted both in discrete and continuous control problems to verify that the proposed method outperforms traditional RL methods. The authors can clarify if the proposed mechanism is more time-consuming than the traditional RL techniques.

Clarity
The paper is well written and easy to follow. 

Significance:
The work is significant to the community as it introduces the paradigm of teacher in the loop RL which considers interactive feedback during learning. The work looks like a promising step towards a broader class of feedback loop during RL training. 

limitations_and_societal_impact: The paper points out that the teacher in the loop is not a real-time human which can be noisy and compromised which are scenarios that need to be considered. The proposed method is a work towards having autonomous systems operate with rich interactive teacher feedback instead of just reward signals
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper presents an interesting framework (CAMDPs) for teacher-in-the-loop-based RL, that takes into account rich interactive advice. The authors also show how the advice is interpreted in the Coaching augmented MDP. The framework comprises of 3 stages: grounding, improvement, and evaluation. In the grounding phase, the agent learns to interpret an 'advice', in the improvement phase the agent learns a policy using the advice and finally in the evaluation phase the learner deploys it's advice-independent policy. The claim is that the proposed pipeline helps a learner learn more efficiently, as it is able to account for richer forms of feedback. The method was tested on 3 rich RL environments: Baby AI, Ant Env, and Point Navigation Env, and evaluated based on its advice efficiency. The evaluations shpw that an agent learns quicker when presented with advice than when it is not.
main_review: **Pros**

* The paper tackles an ambitious problem of learning from high-level advice in teacher-in-the-loop-based RL systems. The problem formulation was clean, and the three steps of the framework were well explained. The idea of using bootstrapping to interpret multi-level advice was particularly interesting.
* The experimental evaluation carried out on the rich RL environments was promising. The experimental analysis on the 3 RL environments was exhaustive.



**Cons**

* Evaluation: While the evaluation on the 3 rich RL environments was done in terms of the advice efficiency metric, it would be interesting to see how this method compares to the existing methods such as imitation learning, in terms of their sample efficiency. Furthermore, it would also be interesting to investigate the effect of richer/ more structured advice grammar and analyze further if there was a way to determine the optimal sequence of different advice types within a task, to improve performance fastest. It would be great to see the effect of this framework on a more real setting: perhaps on teaching an agent to solve mathematical equations, which comprises a more challenging advice set.



**General Comments**
This paper opens avenues for analysis of such rich feedback-based RL systems. It presents an interesting framework to analyze the richness of advice-based feedback w.r.t task. The different advice types investigated, and the framework presented for their distillation into a policy is novel. The evaluation of such a framework on the simulated RL environments looks promising, and there is scope to explore more interesting/complex advice grammar.


*Originality*: Novel Approach

*Clarity*: Largely clear. 

*Quality*: Good

*Significance*: Important stepping stone to future work in this direction

limitations_and_societal_impact: The paper addresses its limitations of assuming that the provided advice by the teacher is always optimal. They have left incorporating noisy advice for future work. However, this work is still far from deployment to be concerned about its negative societal impacts.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a reinforcement learning approach where humans can 
provide different forms of advice to complete a task. The proposed
approach, called CAMDPs, follows a two-stage process: Grounding and
Improvement. 

During grounding the system learns how to interpret the advice. The
result of this phase is a surrogate policy. Grounding learns a mapping
from advice to actions using information from the true reward
function. Grounding can be seen as an ordinarily multitask RL, where
each task is a piece of advice, and the agent learns using normal RL
how to achieve the advice. 

Distillation is used to learn a policy using supervised learning and
bootstrap distillation uses the previously learned policies for
individual advice to learn a policy for a particular task.

During improvement the coach provides advice to solve a new task. A very
similar approach is followed as bootstrap distillation, a new task
is solved using advice followed by a distillation process.

It is expected for the learned policies to be general enough to be
applied in other, although similar, domains.

The proposed system is evaluated in three, relatively similar,
domains. As expected, it is experimentally shown that the proposed
system learns faster with advice.

main_review: Some positive aspects of this paper are that it can use different
types of advice and it is tested on different domains. It is also
positive that it can combine low-level advice and policies into
higher-level policies to solve more complex tasks.

On the negative side, the description of the method is not completely
clear and the proposed method seems to heavily depend on human
intervention in order to work. It also depends on some strong
assumptions in terms of how the advice is given, which makes it
difficult to assess how useful the proposed system will behave under
more realistic conditions. 


It is not clear how the sub-goals are related to higher goals. It
seems to be that the user is responsible for providing this, but needs
clarification. 

Also it is not clear how the pieces of advice are learned. Since more
abstract advice can be learned from more concrete advice, does it mean
that the system needs to first learn the less abstract advice (e.g.,
direction) before more abstract advises (e.g, cardinal and then
way-points) or can the advises be learned independently.

It is not clear how the proposed approach of combining low level
policies to learn more abstract policies compares to learning
options or other hierarchical learning approaches.

There are different types of advice (at different abstraction levels),
but in the evaluation phase they are treated equally. A more detailed
evaluation, considering these differences would certainly help to
understand the role of each type of advice.

At every state the teacher chooses the coaching function (if any) to
apply. This is currently hardwired, but it is not clear what was the
rationale to define this process (when to give advice and which type
of advice to use).

The authors should clarify the the types of advice is restricted to
those that suggest possible actions. For instance, a user may tell the
agent that what it is doing is wrong without providing any specific
hint of which action may be the right one.

It would be very helpful to give an example of one "typical" episode
with advice (e.g., when and what type of advice is given, how often, etc.)

It not clear how general is the approach. It is shown that the
learned advice can be used in larger versions of the same original
task. Can the advice be transferred between different tasks, e.g., from
BabyAI to Ant Navigation?

Also, the system assumes ideal conditions with perfect advice and no
noise. It is not clear how the proposed approach will behave under
more realistic conditions. It is easy to image scenarios where a user
gives wrong pieces of advises, with delays, or gives an advice that it
is too general from which the system is not able to learn, etc.

Small typos:
- a advice
- other We

----------
I have read the comments of the other reviewers and the authors' response.
I appreciate that the authors have performed further experiments to 
strengthen their paper and I think that the paper could be publish, 
however I still maintain my original acceptance score.
limitations_and_societal_impact: Yes the authors adequately addressed the limitations and positive societal impact
review_title: null

Meta-review: The reviewers were very impressed by the amount of additional material provided during the author response phase. The extensive replies addressed all major concerns of the reviewers. Incorporating all that in the paper will be a challenge though.
Decision: Accept (Poster)