Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper leverages the ideas of pedagogy and pragmatism for training goal-conditioned RL agents in multi-goal environments with demonstrations. Pedagogy refers to the ability of the teacher to provide non-ambiguous demonstrations to the learner, whereas pragmatism refers to the learner’s ability to discern which goal the teacher intended to demonstrate. The algorithm consists of two phases: 1) a goal-conditioned teacher policy is trained using off-policy trajectories. In addition to the reward from the environment, the teacher policy receives a reward for correctly inferring the intended goal of its own trajectory using Bayesian Goal Inference (BGI). 2) The teacher provides demonstrations to a new learner policy, which receives rewards from the environment and additional rewards for correctly inferring the intended goal of the teacher's demonstrations, and the intended goal of its own trajectories. The experimental results show that pedagogical teachers and pragmatic leaners are useful when learning from demonstrations.
strengths_and_weaknesses: ### Originality
The related work section is comprehensive, and I appreciate the authors breaking the different areas into different sections. This work seems to be a novel combination of pedagogy, pragmatism, BGI, and goal-conditioned RL in multi-goal environments.

### Quality
The experimental results were presented well and clearly addressed the contributions stated in the introduction. The authors do a great job of including specific information such as architecture, number of random seeds, mean, variance, etc. 

The questions that the experiments aim to answer seem to be 1) can the learner infer goals from the teacher, 2) can both the learner/teacher infer goals from their own trajectories, 3) can the learner reach all goals, and 4) can the learner both predict goals and reach them.

$GIA$ addresses 1, $OGIA$ addresses 2, $GRA$ addresses 3, and $GIA \times GRA$ addresses 4. The results for $GIA$, $GRA$, and $GIA \times GRA$ are included in Table 1, but I did not see the results for $OGIA$?

### Clarity & Significance
Overall, the paper is very well written and easy to follow. The authors motivated the work in an intuitive way, and also highlighted potential applications. The results seem to be significant, especially for areas like robotics where acquiring demonstrations can be expensive. Researchers can easily extend this approach to use pre-existing policies for the teachers, then further refine them to be pedagogical. 

* In Section 5.1, it is implied that the experiments are referring to Fetch Block Stacking, but I think this should be stated in the first sentence for clarity. E.g., "We verify that the pedagogical teacher can indeed better predict goals from its demonstrations compared to a naive teacher 'in the FBS environment'".
* I don't completely understand  how the Draw Two Balls environment is defined. If there are purple, orange, and pink balls, and two consecutive draws, how are there only three possible outcomes?
* Nits
    * I think the second line of Figure 5 should say "goal 2 (activation of sound 2)"
    * Line 85: remove the comma
    * Line 119: add comma "Finally,"

questions: * Is the teacher's ability to master all goals in the environment a requirement for the use of this algorithm? Were any experiments done to see if training both the student and teacher in a cooperative manner had better results (essentially combining phases 1&2 together)? 
* How is the first goal chosen in phases 1 and 2? Could strategically choosing the first goal impact the performance of the teacher and student?
* Do the authors think that increasing the number of demonstrations would make GPNN perform better compared with BGI?
limitations: The limitations were well addressed in Section 6.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: * This work focuses on the problem of learning from teacher/learner demonstrations and in particular focus on training a teacher/learner setup where goal ambiguity can be resolved to improve the quality of the demonstrations and to thus improve effective goal inference and task performance.  Multi-goal environments are utilized.
* The authors make use of concepts from cognitive science, pedagogy: optimization of teaching concepts; Pragmatism: resolution of ambiguities of intention from the teacher. 
* With this in mind the authors aim to tackle the problem of goal ambiguity: given that at least two goals are reachable in a given trajectory can reach at least two goals -> can't reliably predict pursued goal.  The authors aim to improve demonstrations by resolving goal ambiguity and learning good goal inference.
* Teacher/Learner share common goal / state / action spaces and use separate policies.  Communication between the two is facilitated via teacher demos, learner's inferred goal, teacher feedback on inference.

Training proceeds roughly as follows:

1. Teacher is pre-trained in the env to master/solve all goals with Goal Conditioned RL (GCRL)
2. Learner infers goal from teacher's demo using Bayesian goal inference (BGI), teacher provides feedback, learner rewarded for correct predictions

The teacher tries to infer its own goal (with BGI) from trajectory after reaching it.  It adds a "pedagogical" reward to the trajectory if so which aids in retaining high performance on the task.

Results measured over the Draw-Two-Balls (DTB) and Fetch-Block-Stacking (FBS) environments.  Four metrics are collected:

1. Goal inference accuracy (GIA) - learner can infer G from d
2. Own goal inference acc - teacher can infer G from its own demonstrations
3. Goal reaching acc (GRA)  - can learner reach all goals?
4. GIA x GRA - can the learner both infer its goal and reach it
Ambiguity Score

The main claim of the paper is that using a pedagogical and pragmatic teacher/learner setup to improve demonstrations through goal disambiguation will lead to to faster learning with fewer demonstrations.

strengths_and_weaknesses: **Strengths**:

* The authors aim to tackle an interesting problem of leveraging teach/learner demonstrations which are important in a variety of domains.  * The approach they take to disambiguate goals from existing demonstrations is very compelling and could lead to learning improvements in a variety of ways (signal boost, sample efficiency, data quality)
* Overall the approach presented in the paper is sound and the paper is clearly written for the most part although there may have been a bit more in the way of architectural detail.
* The authors provide useful context for their work in section 2 describing past relevant work in pedagogical demos, pragmatic inference and demonstrations in goal conditioned tasks.
* The training phases are detailed well with accompanying algorithms.
* The authors show that in the context of naive/simple v. pedagogical/pragmatic teacher/learner setups the pedagogical+pragmatic variant has a clear advantage in the evaluated tasks.


**Weaknesses**:

* In the  methods section some more explanation on how new goals are discovered could be helpful.  I'm a bit foggy on how the goals are actually selected.
* In section 4.1: A bit more explanation around the distinction of literal v pragmatic learners and naive v pedagogic teachers would be helpful.  Some basic details in the main paper would be helpful.  Same goes for the architectural details of the teacher/learner networks.  The training and interaction is explained however there are no clear architecture diagrams and not much explanation in the main paper.
* Results in the main paper are limited to a single task and while they show better sample efficiency in this case it is hard to understand the benefit realized more generally.  Further the 95% threshold for the 2x increase claimed seems somewhat hand picked (e.g. why not 80% or 90%?  These would reduce the sample efficiency significantly). 
* From the results it seems clear that Pedagogical/Pragmatic Teacher/Learner improve over Naive/Simple variants however it is less clear how well these improve more generally over other approaches.  Section 5.5 seems to address this over a limited set of baselines including Behavioural Cloning and SQIL but these don't seem to be particularly strong baselines (a couple of them don't even reach a substantial number of goals in the demos).  I believe more rigorous comparisons to baselines would be needed here in order to really justify the claims of this work.
 * In figure 4 there is a notable dip over the learner without demonstrations at roughly 6e5 learner steps.  It would be useful to explain what this might be, especially since it seems to be present for all seeds and only for this condition. It would be helpful to understand these gains in a broader context to understand the full benefit of the work here.  It also may have been useful to try with fewer than 100 demonstrations as performance seems to be mostly unaffected when reducing the number of demos from 1000.
* There are no results for DTS results in the main paper, including these could be useful.

Overall I think that this is a promising are of work but some more detail would help to explain the approach and the experiments/results would benefit from being more comprehensive to satisfy the claims.


[2022-07-27] EDIT:  I've added in the missing bit the authors referenced in their comment (I can't reply Officially yet, so adding here).  Apologies for that and thanks for flagging.
questions: 
* How can we ensure that all goals are mastered/solved/achieved by the teacher over the dataset?  What happens if we are only able to master some fraction?
* In alg. 1 are flags provided for "learner is pragmatic" and "teacher is pedagogical"?
* What do you think causes the accuracy dip for the learner curve with no demonstrations in figure 1?
* What computational overhead does the pre-training occur?  How does this compare to the phase involving learning from the demonstrations?

limitations: Limitations weren't explicitly addressed in the main paper to my knowledge.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This submission proposes a novel method to learn from demonstration. Inspired by human learning, the authors propose a Bayesian Goal Inference framework to avoid ambiguity in multi-goal learning. And use such a framework to benefit the learner's learning under the demonstrator's help.
strengths_and_weaknesses: I appreciate the novelty, and great effort put into the paper. Especially the illustration video in the supplementary material--- which helped a lot in understanding the work. 

However, there are several concerns I hope can be addressed according to the current stage of the presentation. 
1. motivation/ problem formalism: the authors failed to present the task in a clear enough way: e.g., why does the learner need to infer the goal when the goal is known?
2. missing baselines/ lack of empirical results.
3. lack of theoretical guarantee

questions: Major concerns:

1. How is the research motivated, why does the pedagogical demonstration and inference of goals important? In GCRL, demonstrations should contain the true goal state, then what is the purpose of inferring multiple potential goals? Real-world applications can be helpful in demonstrating the importance of the studied problem.

2. with those demonstration data, Hindsight Experience Replay methods can be applied to generate more successful goal-reaching experiences. I believe a comparison to HER can be helpful in evaluating the proposed method's performance. (e.g., in Table 3)

3. As for the precise GCRL algorithm the authors used, it was mentioned in line 104 by referring to [11], however, this [11] is a survey paper. More description of the learning algorithm itself should be at least referred to in this section.

4. during the learning of GCRL, the goal is known, then what is the purpose of inferring the goal?

5. the proposed method does not have a theoretical guarantee but is mostly of a shape as an empirical study.

minor:

1. in GCRL (practically, the robotics control tasks, e.g., [36]), the state space and goal space (with a mapping between state and goal space always assumed to be known [36]) are continuous, how will the proposed method tackle such situations where the goal set has an infinite cardinality?

2. Section 4.1 introduces several reward-shaping designs, is the proposed method sensitive to those choices? detailed ablation studies can be helpful to verify the proposed method.

3. 5 seeds are not enough for evaluating RL algorithms.

4. some presentations are ambiguous and need proofreading. e.g., line 100 'where learners are both...', I reckon there is only one learner?

I may misunderstand some parts of the work, and I'm happy to raise my score if the above concerns can be appropriately addressed.
limitations: 1. clear motivation of the research
2. solid guarantee of the proposed method
3. more empirical studies (#seeds and environments)
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This work proposes utilizing teacher-learner settings in Goal-conditioned Reinforcement Learning with concepts of pedagogy and pragmatics from other disciplines. In particular, the paper introduces the process of Bayesian Goal Inference (BGI), which is a reward-shaping method that encourages policies to learn behaviors that allow an observer to infer the goal given the trajectory. The paper uses BGI into two settings: to augment a teacher policy’s behaviors to have more disambiguation (“pedagogical teacher”), and to augment a learner to filter our teacher demonstrations and to predict it’s own goals (“pragmatic learner”). The paper applies this method to a toy setting and then a simulated robotic block-pushing setting, and show that pedagogy and pragmatics improve goal reaching and goal inference for both the teacher and the learner.
strengths_and_weaknesses: Strengths:
- The method is quite simple and easily usable with any GCRL + LfD method.
- Since the method implementation in this paper did depend on specific underlying decisions (ie. teacher-learner setup, BGI, SQIL), the ablations were useful to showing the method still works in other settings
- The method increases goal-reaching accuracy in addition to goal-inference accuracy

Weaknesses:
- The method is only tested on a fairly simple block-pushing environment. In more challenging environments (high-dimensional action and observation spaces, more ambiguity in the goal space), it is unclear how noisier BGI predictions will affect the method as it is used for reward-shaping
- Much of the benefit of the method arises from the learner’s utilization of BGI. While 5.4 studies a teacher-less setting, it’s unclear why it’s much more important to have a pragmatic learner than a pedagogical teacher. See questions below.

questions: - A lot of benefit results from going from literal learner => pragmatic learner, ie. the BGI pedagogical loss for reward shaping. Perhaps the fairest comparison for experiment 4) would be to include cumulative environment interactions (including how many the teacher used). Thus, it can be possible to tell whether the driving benefit results from on-policy updates to encourage the experience-collecting policy (whether it is teacher or learner) to have more disambiguated trajectories. Can you try an experiment where you give the learner (no demonstrations) as many on-policy updates w/ BGI as the cumulative teacher+learner on-policy update budget?
- I could not find information on how much ambiguity results from the environments, as measured by the Ambiguity Score. How percentage of trajectories in the Naive/Literal methods (without BGI) result in ambiguous policies? How does this scale to more complex tasks in higher-dimensional spaces such as learning from vision, where the BGI measure might be much noisier?
- How expensive is the BGI prediction, and how does it scale to continuous action spaces or continuous goal spaces? Given that the method relies on the learner BGI heavily, how does the method fare when goal inference becomes more challenging or a worse method is used? Some possible experiments might be to use GPNN instead of BGI in the main experiments, or an ablation increasing the number of goals.

limitations: It would be interesting to see how well the method scales to settings where ambiguity is much harder to quantify. In the paper, the authors illustrate the challenges of ambiguity with a three-block toy problem, but in the real world goal inference is often challenging even for experts. In such settings, goal inference auxiliary tasks (like BGI) may not help performance, and may even hurt occasionally.
review_title: null

Meta-review: metareview: After a strong rebuttal from the authors and an extensive discussion among the reviewers, I believe the paper's pros outweigh its cons and this paper will be a valuable contribution to NeurIPS. I recommend it for acceptance and encourage the authors to address the reviewers comments for the camera-ready version of the paper, especially regarding the weaknesses of empirical evaluation and differentiation against conventional GCRL.

confidence: Certain
Decision: Accept