Pragmatically Learning from Pedagogical Demonstrations in Multi-Goal Environments
Authors: Hugo Caselles-Dupré
Olivier Sigaud
Mohamed CHETOUANI
Venue: NeurIPS 2022 Accept (2022)
Keywords: learning from demonstration
multi-goal environments
pedagogical teaching
pragmatic reasoning

Abstract: Learning from demonstration methods usually leverage close to optimal demonstrations to accelerate training. By contrast, when demonstrating a task, human teachers deviate from optimal demonstrations and pedagogically modify their behavior by giving demonstrations that best disambiguate the goal they want to demonstrate. Analogously, human learners excel at pragmatically inferring the intent of the teacher, facilitating communication between the two agents. These mechanisms are critical in the few demonstrations regime, where inferring the goal is more difficult. In this paper, we implement pedagogy and pragmatism mechanisms by leveraging a Bayesian model of Goal Inference from demonstrations. We highlight the benefits of this model in multi-goal teacher-learner setups with two artificial agents that learn with goal-conditioned Reinforcement Learning. We show that combining BGI-agents (a pedagogical teacher and a pragmatic learner) results in faster learning and reduced goal ambiguity over standard learning from demonstrations, especially in the few demonstrations regime.

Pragmatically Learning from Pedagogical Demonstrations in Multi-Goal Environments
Hugo Caselles-Dupré, Olivier Sigaud, Mohamed Chetouani Sorbonne Université, CNRS, Institut des Systèmes Intelligents et de Robotique (ISIR)
Paris, France
casellesdupre hwg.ogmail.l..a................. com, olivier sigaud, sigaud, mohamed. chetouani@@iii. wwm.:iiiiiii...iii...mmmmmmmm fr
Abstract
Learning from demonstration methods usually leverage close to optimal demonstra- tions to accelerate training. By contrast, when demonstrating task, human teachers deviate from optimal demonstrations and pedagogically modify their behavior by giving demonstrations that best disambiguate the goal they want to demonstrate. Analogously, human learners excel at pragmatically inferring the intent of the teacher, facilitating communication between the two agents. These mechanisms are critical in the few demonstrations regime, where inferring the goal is more difficult. In this paper, we implement pedagogy and pragmatism mechanisms by leveraging Bayesian model of Goal Inference from demonstrations (BGI). We highlight the benefits of this model in multi-goal reacher-learne, setups with two artificial agents that learn with goal-conditioned Reinforcement Learning. We show that combining BGI-agents (a pedagogical teacher and pragmatic learner) results in faster learning and reduced goal ambiguity standard learning from demonstrations, especially in the few demonstrations regime. We provide the code for our experiments as well an illustrative video explaining our approach
Introduction
Imagine teacher-learner setup where colored blocks are lying on a table, the green and red blocks are close to each other and the blue block further away. Assume the teacher wants to demonstrate how to have the blue block close to the green one (see Fig. D. A naive teacher may move the blue block to the green one, but this would ambiguous: is the goal to put the blue block close to the green or the red one? By contrast, a pedagogical teacher would move the green block close to the blue one, hence away from the red one, resolving the ambiguity as illustrated in Fig. [
As this example shows, in real life, when teacher shows how to do something with demonstration, the agent receiving the demonstration does not have access to the intended goal of the demonstration: it must infer this goal. In many situations, single demonstration may be correctly interpreted as demonstration of a variety of goals we call this goal ambiguíty. pedagogy to help the learner infer the intended goal, and the learner may use pragmatism in order to increase its chance to infer the right goal. This is the situation we address in this paper.
that case, the teacher may use
To infer that demonstration facilitates learning, the pedagogical teacher may ask herself what goal she would infer she was the learner. Using her policy and Bayesian inference, she can determine the probability that she would infer the right goal given this demonstration, and assume the learner would do the same. Now, the learner has to interpret the demonstration infer the goal. Given the same inference mechanism as the teacher, a literal learner would perform this inference using her
praggmatteee
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

own policy. On top of that, pragmatic learer could first train her policy to predict her own goals from her own trajectories, increasing her likelihood correctly infer the teacher's goal.
Those mechanisms, pedagogy from the teacher and pragmatism from the learner, facilitate communi- cation between the two agents and thus improve learning by reducing ambiguity about the inferred goal. Pedagogy and pragmatism are concepts borrowed from cognitive science research. On the one hand, pedagogy is defined as the optimization of teaching concepts from examples [8 [9. On the other hand, pragmatism is property used to resolve ambiguities of intention interpretation from teaching signals, which can be language with for instance the rational speech act (RSA) 34 or actions with pedagogical demonstrations
38
Phose I: Teucher pre-truining
Phase 2: Learner trains with teacher's 's demonstrations and RL
Naive
Naive demonstration
Literal
Teacher
Infer the goal of the decoonstration and train to master all goals
Pre-traiee trnitri master all
Goal: Green
Bayreian Boal Inference
Inferred goul
Bayesian Goal Infermmee
Learner
Goal: Blue closn Drven
Pre-trained infer to better unimate goal via demonstrations
Train infor goals to betler interpret demonstrations
Pre-trained all goa
Pedagogical demonstration
Infer the goal the demonstration and Irain to masler all goals
Pedagogical
Pragmatic
Fig. I: Overview of our Macher-llearer setup. All goal inferences are performed using Bayesian Goal Inference, an inference method that computes the goal probability from a demonstration using the agent's 's policy. In the first phase, the naive teacher pre-trained to master all goals while the pedagogical teacher is additionally pre-trained to better infer goals from its own trajectories and effectively modifies its policy to produce demonstrations with less ambiguous goals. the second phase, the learner is trained using the teacher's demonstrations and goal-conditioned Reinforcement Learning. The literal learner infers goals from demonstrations and is rewarded for correct inferences while the pragmatic learner additionally trains itself to infer its goals from its trajectories and effectively modifying its policy to better infer goals from the teacher's demonstrations.
Such ideas have been leveraged into the general theory of "Inferential Social Learning" (ISL.) in order explain our abilities as learners and teachers to "interpret and generate evidence social contexts". ISL is characterized as an inference mechanism guided by an intuitive understanding of how people think, plan, and act. leverages Bayesian inference explain several key aspects involved in pedagogy and pragmatism: pedagogy can help avoid goal ambiguity 26 27 or accelerate goal inference from learner (i.e. legibility) [29]. and pragmatism can help better infer goals [4], or detect pedagogy (22), or generalize from induction 23 (what can I deduce from teaching signal?). In this paper, we show that using Bayesian inference for disambiguating goals (pedagogy) and better infer goals (pragmatism) can improve the learning capabilities of artificial agents. In particular, we show that this mechanism is critical to reduce ambiguity when the teacher performs few demonstrations, which may contribute to more feasible robot teaching in the future.
We present an approach for implementing inferential mechanisms to improve learning from demonstra- tions in an artificial teacher-learner setup, as illustrated in Fig. [ We use two multi-goal environments: simple one for illustrating our point, and a more complex simulated robotics block manipulation environment to show the robustness of our approach. In both environments, we create naive teachers trained to master all goals, and pedagogical teachers trained to provide demonstrations that best disambiguate between all goals and thus facilitate goal inference for learners. We then create literal learners, which learn from teacher demonstrations, and pragmatic learners, which adapt their policy to better infer goals from the teacher's demonstrations. The mechanism for implementing pedagogy and pragmatism an additional reward for correct goal inference of the agent's 's own trajectories. Goal inference is implemented using Bayesian model and the agent's 's policy.

---

We make the following contributions:
1. Inspired by the ISL. framework, we introduce pedagogical teaching and pragmatic learning mechanisms based on Bayesian Goal Inference and show that these mechanisms can be leveraged in a cooperative artificial teacher-learner training setup where both agents use goal-conditioned policies.
2. We show that pedagogy and pragmatism help reduce goal ambiguity and result in faster learning, which is especially helpful when the teacher performs fewer demonstrations.
2
Related work
Our work is related to several strongly connected areas.
Bayesian inference. Bayesian Inference was already used as a key mechanism for goal inference in the context of inverse planning 7 5 43 It uses the Bayes formula to compute the probabilities of goals given the actions and the policy. In our work we use it as tool to produce con-ambiguous demonstrations the teacher side and pragmatic inference on the learner side.
Pedagogical demonstrations. an attempt to explain how humans demonstrate tasks to each other, Ho et al. [29 28 26 introduce the Observer Belief MDP model and show that producing more pedagogical demonstrations results in better performance. However, their work does not investigate how this affects an artificial learner, nor the effect of pragmatism in the learner.
Legibility in Human-Robor Interaction. Similarly us, R3 32 to allow an observer (a human in their case) to more quickly and successfully understand a robot's goal from more legible trajectory. In work, Bayesian goal inference is employed in pedagogical teachers to allow them to generate less ambiguous goal demonstration using additional reward for correct goal inference on their own motion.
modify robot action sequences
Pragmatic inference. A long line of work in linguistics, natural language processing, and cognitive science has studied pragmatics: how linguistic meaning is affected by context and communicative goals 18, [7]. Another line of work on pragmatics, more related to our work, focuses on inferring meaning from action in context. It has been applied to Robotics 33 15) where pragmatic robot better infers the objective of teacher by considering its pedagogical intentions, in a multi-agent game theory context. In our case, are interested in how pragmatic learner can learn by itself to better infer the goals of a demonstrator, resulting in faster task learning, which is different from detecting if the teacher is pedagogical or not.
Demonstrations in goal-conditioned tasks. Our learning from demonstration mechanism is related to work learning from demonstrations in multi-goal environments. by contrast with the more common setup where the agent only learns one task. Most approaches targeting fewer demonstrations address the task setup I
Approaches using demonstrations for goal-conditioned tasks such as GAIL 25 DCRL 10 and CLIC [16] combine goal-conditioned RL and imitation learning with additional rewards. In work, also combine a goal-conditioned RI. algorithm (GANGSTR (3) and a slightly modified version of an algorithm learning both from demonstration and from the agent's experience (SQIL (24). The specificity of our pipeline is that can easily improve it with pedagogical demonstrations and pragmatic inference to accelerate training, but in principle all goal-conditioned RL algorithms using demonstrations could be improved in such way.
44
42
40)
3 Methods
We consider multi-goal environments with teacher-learner scenarios where the learner is both trained with teachers' demonstrations and using its own exploration. The teacher T and the leamer L are both represented by their respective goal-conditioned policies (Ty(-1 and (-1g)) These policies might be learned using any algorithm (multi arme bandits, RL, evolution strategies, etc.), here we use a goal-conditioned RL (GCRL) algorithm [[]] called GANGSTR 3 which we describe in our experimental setup. Our multi- malti-goal environments present goal ambiguity, as the same trajectory can simultaneously reach at least two goals (g), 92). Considering this hypothesis, it generally not possible to reliably predict the pursued goal from demonstration reaching both 91 and 92.

---

Both agents share common goal, state and action spaces. They communicate through teacher's demonstrations, learner's inferred goal and teacher feedback on this inference. To efficiently exploit these signals, we introduce a Bayesian Goal Inference (BGT) mechanism helping agents infer goals from demonstrations. The teacher uses to generate less ambiguous demonstrations (pedagogy), while the learner uses it to infer the teacher's goal from the demonstration (pragmatism). We refer to the pedagogical teacher and pragmatic learners as BGI-agents.
The training process is spread across two phases: 1) the teacher is pre-trained to master all goals in the environment using GCRL, and 2) the learner infers the goal from teacher's demonstration using Bayesian Goal Inference, the teacher provides feedback on the inference, and the learner is rewarded for correct predictions. The learner then attempts at reaching the inferred goal and iteratively improves its policy using GCRL.
Below formally define BGI, then define naive teachers and literal learners, which do not leverage the full benefits of BGI. Finally we introduce pedagogical teachers and pragmatic learners, which use BGI to facilitate communication with each other.
3.1 Bayesian Goal Inference in Teacher-Learner Interactions
We formally introduce the BGI mechanism used to infer goals from a goal-conditioned policy. In our work, 1) the teacher uses it to implement pedagogy, 2) the learner uses it to infer goals from the demonstrations of the teacher, and 3) the learner uses to implement pragmatism.
Inferring the Goal from a Demonstration In goal-conditioned Markov environment, the proba- bility of observing demonstration d ((s1,a)) ai), (8ny an)) given goal g and the goal-conditioned policy ग (-Ig) that generated it can be written 226 as:
g( = P((11, az), (Sn, an)lg)
II ((.)s))))) P(s)-1|s;;0) 0.) II [[a(as)s.)))
where ((5;41|8;; a.) as we consider deterministic environment.
Now, to infer goal given demonstration, by using Bayes' rule derive P(G the probability distribution over the goal space G given the demonstration:
n
P(G x P(dG) (PG)) II C).
For each goal g. the prior P(G) is uniform if not specified otherwise. Given P(G]d), an agent can infer the goal of a demonstration by either taking the most probable goal, sampling from the distribution. To perform this inference, the agent uses its own policy.
Learning Goal Inference from Own Trajectories When playing its policy, agent produces trajectories which we call demonstrations when they are produced for other agents. A GCRL agent can leverage the BGI mechanism its own trajectories improve its ability to infer goals. To do so, during training, the agent selects a goal, performs a trajectory, infers the goal from the trajectory, and rewards itself if the inference is correct. This reinforces the policy towards actions that lead to better goal inference. We use this to implement both pedagogy in the teacher and pragmatism in the learner.
3.2 Naive/Pedagogical Teacher and Literal/Pragmatic Learner training
We now present the two training phases in our teacher-learner setup. First, the teacher is pre-trained, and then it provides demonstrations for the learner train with. The two-phases training process presented in Algorithm

---

Algorithm Two-phases training of the teacher and the learner
1: PHASE I: Teacher pre-training (xr(-1g))
2: Initialize (-(g)) goal set Gr to {first goal} and pedagogical Boolean variable
3: for epoch = 1, do
4: Randomly sample goal g from goal set GT
5: Run policy (.(g)) obtain trajectory f = (171,, (8nx any rn)) and achieved goal go.
6:
if pedagogical Boolean = and = 9a then Infer goal 9 with BGI on t: if g g. add pedagogical reward to end
7:
8:
9: Add to replay buffer and if gu is new, add Ja to goal set GT
10: Update policy Ty with GCRL
11:
end for
1: PHASE 2: Learner (-1g)) training with Teacher's demonstrations
2: Initialize T (.1g). goal set GL to {first goal} and pragmatic Boolean variable
3: for epoch = 2, do
4: Teacher randomly samples goal ga from goal set Gr. and gives demo d by running गт (-19a)
5: Learner infers goal of d with BGT
6: if Id =
Ia (feedback provided by the teacher) then 7: Add d to learner's 's replay buffer
8:
end if
9: Run policy TT (-1ga) obtain trajectory t = ((51, (Sny One Tn)) and achieved goal
10:
if pragmatic Boolean 1 and Gd 9a then
11:
Infer own goal Io with BGI on t: if go = Ja. add pragmatic reward to
end if
13:
Add / to learner's replay buffer and if Jo is new, add Jo to goal set Gt.
14:
Update policy æT with GCRL.
15:
end for
Phase 1: Teacher pre-training. The teacher is pre- pre-trained before providing demonstrations to the learner. The naive teacher's policy is trained to master all goals: maintains a goal set Gr which adds newly encountered goals, samples goals from Gr and pursues them to collect trajectories added to a replay buffer. Finally, it applies GCRL for policy updates.
The pedagogical teacher is also trained to master all goals, but additionally trains itself to predict its own goals using BGI. When training, each time it successfully reaches goal, it takes its own trajectory and infers the pursued goal using BGI, effectively asking itself: "would a learner able to predict the goal from this demonstration?". If it correctly infers the goal from its trajectory, it rewards itself to reinforce its probability to select this trajectory. This is done with GCRL by adding "pedagogical reward" to the trajectory for which the teacher correctly infers its own goal. This biases policy learning towards finding demonstrations which avoid ambiguity in goal inference, while guaranteeing high performance on the actual task.
Phase 2: Training the Literal/Pragmatic Learner with Teacher's demonstrations. The literal
learner's 's noal-conditioned policy (-1g)) is trained to master all goals in the environment using teacher's demonstrations and GCRL.
At each iteration, the teacher samples a desired goal ga it wants the leamner to achieve. It then presents demonstration for the chosen goal to the learner. Using BGI with its own policy, the learner infers the goal ÿa from the teacher's 's demonstration. The teacher provides feedback, telling the learner ga = g. If the learner correctly inferred the goal of the teacher's demonstration (ga = 9a), the demonstration is added to the learner's 's replay buffer and used for further training. Then the learner plays its policy conditioned on the predicted goal Tz (-1g)) and obtains a trajectory resulting in an achieved goal 9a, which is added to the replay buffer. If 9u is achieved for the first time, then the teacher updates set of goals from which it can sample. Finally, GCRL improves the policy from its own trajectories and demonstrations drawn from the replay buffer. Note that we assume that the learner does not have access to the goal of the demonstration and thus must infer it, mimicking real-life situations where humans regularly have infer other people's 's goals [22].
5

---

A pragmatic learner improves over a literal learner by training its own policy to better infer the right goals from the teacher's demonstrations. If the learner is able infer its own goals using BGI, then it will better infer the teacher's goals. The pragmatic learner thus adds a "pragmatic reward" to its trajectories for which can correctly infer its own goals.
Experimental Setup
We use two environments as test-beds for experiments: simple one inspired from research in Developmental Psychology [22] called "Draw two balls" (DTB) that we initially used for illustrating approach, and more complex one called "Fetch block-stacking" (FBS) to test our approach more challenging domain [36]. Results on DTB are presented Appendix A and the main paper focuses FBS. Both DTB and FBS multi-god environments and present goal ambiguity.
4.1 Environment: Fetch Block Stacking
FBS a block-stacking environment with two Fetch robots (teacher and learner) equipped with robotic arms, see Fig. It is based on MuJoCo [4] and derived from the Fetch tasks (36].
The teacher and learner share goal/state/action spaces. Actions are 4-dimensional: 3D gripper velocities and grasping veloc- ity. Observations are the Cartesian and angular positions and velocities of the gripper and the three blocks. Following [2]. we adopt a semantic predicates representation where we add to the observation binary vector telling whether two blocks are close, and whether any block on top of any other. The agent uses these binary vectors as goals (with 35 possible goals including stacks and pyramids). Given a particular goal vec- tor configuration, the agent gets +1 reward for each pair of blocks (3 in total) if true predicates about the pair of blocks the same in the current vector and the goal vector. There several possibilities match the true predicates in the goal vector, which induces goal ambiguity. Details about the environment implementation provided in Appendix B.I
Tearher
Fig. 2: Fetch Block Stacking,
The training procedure and policy architecture the naive teacher are taken from GANGSTR (3 which already implements Ferch Block Stacking. The GANGSTR agent perceives the low-level geometric states and the high-level semantic configurations. Following
Naive/Pedagogical teachers implementation.
3 encode both object-centered and relational inductive biases in architecture. We model both the agents policies and critics as Message Passing Graph Neural Networks [19]. We consider graph of 3 nodes, each representing single object. All the nodes are interconnected. We consider the agent's 's body attributes as global features of the policy networks and both the agent's body attributes and the actions as global features of the critic networks.
The agent's goal-conditioned policy is trained using Soft Actor Critic (SAC) 24, staae-of-thh-aeee art RL algorithm, combined with Hindsight Experience Replay (HER) []. Additionally, the pedagogical teacher rewards itself with the "pedagogical reward" (1 here) when correctly infers the goal of its successful trajectories. It does so for each of its collected trajectories during training. All architecture, training and hyperparameters details are provided in Appendix B.2
Literal/P Peeral/Pragatic learners implementation. To implement literal and pragmatic learner training, we use a combination of GANGSTR combined with a slightly modified version of the Soft Q-mitaation Learning (SQIL) algorithm (37 Originally, SQIL rewards demonstrations to and experience to By contrast, we set the demonstration reward environment (3 here), while the reward of trajectories performed in the environment is unchanged. This allows the agent to learn from demonstrations and collected trajectories at the same time using SAC as the GCRL algorithm. As in the original paper, the percentage of demonstrations versus collected trajectories is set 0.5. On top that, the pragmatic learner adds the "pragmatic reward" (1 here) to its trajectories from which it can infer the goal.
the maximum reward obtainable in the

---

4.2 Metrics
We use four metrics for all of our experiments: 1) Goal Inference Accuracy (GIA): "is the learner able to correctly infer goals given demonstrations from teacher?". 2) Own Goal Inference Accuracy (OGIA): "is the agent (teacher or learner) able to correctly infer goals from its trajectories?", Goal Reaching Accuracy (GRA): "is the learner able to reach all goals in the environment?", and 4) the product of GIA and GRA (GIAXGRA), interpreted as the ability of the learner to predict its goal given demonstration from the teacher and then reach GIA and OGIA help us understand the pedagogy and pragmatism mechanism actually work, and computed using test set of 50 demonstrations per goal generated by the teacher for GIA and by the same agent being tested for OGIA (50 35 1750 demonstrations total), GRA allows us evaluate the performance of agents and is computed by testing the agent on each goal 50 times. All demonstrations are long enough to reach the goal (100 timesteps in our case). We provide means μ and standard deviations over 5 seeds and report statistical significance using two-tail Welch's 's t-test with null hypothesis μ1 P2, at level a 0.05 (noted by star markers in figures).
Finally, to compare pedagogical and naive demonstrations, we derive an Ambiguity Score: given two ambiguous goals that be achieved simultaneously and starting state, the ambiguous score is 1 if their associated demonstrations achieve the same goals (the demonstrations are thus ambiguous). and assumes that the demonstrator already knows how achieve the goals. When it is computed from several situations, it evaluates the degree of ambiguity in the demonstrations. For more details on this score please refer to Appendix B.2.6
0 otherwise. This
Experiments and Results
We first study learning results from Phase (naive/pedaagogical teachers training) and Phase (litera//pragmatic learners trained with teacher's demonstrations). We analyze the impact of demon- strations to accelerate learning, and then scrutinize two algorithmic choices in our approach: goal inference with BGT and learning from demonstrations with SQII..
5.1 Experiments on Phase Naive and Pedagogical Teachers
We verify that the pedagogical teacher can indeed better predict goals from its demonstrations compared to naive teacher in the FBS environment. Quantitatively, the pedagogical teacher achieves an OGIA of 15.6%%%22 + 0.1 while the naive teacher achieves 83.4% = 0.2. By training to infer its own goals from its demonstrations, it produces less ambiguous demonstrations. For qualitative evaluation of this phenomenon, we implement the example in the introduction. We compare demonstrations from common starting situation (red block close green block, blue block away) and for two different goals: "Put blue block close to red" and "Put all three blocks close together". As a result, the naive teacher provides the same demonstrations (putting the blue block close red), while the pedagogical teacher provides two different demonstrations to better disambiguate between the two goals (putting the red block close blue and putting the blue block close to red and green).
We compute the Ambiguity Score from 500 ambiguous situations sampled from list of ambiguous situations detailed in Appendix B.2.6 (two ambiguous goals and a starting state) and obtain an unequivocal result: 9% 1 for the pedagogical teacher vs 64% + for the naive teacher. Pedagogical demonstrations are thus 7 times less ambiguous than naive ones.
5.2 Experiments on Phase 2: Literal and Pragmatic Learners
We train pragmatic/fiteral learners with naive/pedagogical teachers and present our results in Fig. and [T In order to evaluate how the mechanisms would respond to training regimes with few demonstrations, we performed the experiments FBS with 1000, 500 and 100 demonstrations per goal, starting from random states.
7

---

1000 demonstrations per goal
500 demonstrations per goal
100 demonstrations per goal
Salve Progredts
Pasingrgiial
Reaching Rccurre
0.50
0.21
0.00
150
200
Timestegs (X10%)
Fig. 3: Global learner performance in the FBS environment (Goal Reaching Accuracy, GRA) with different numbers of demonstrations per goal. Table shows that the drop GRA under the few demonstrations regime (right) is mainly due to incorrect goal inference in the literal learner or with the naive teacher. Stars indicate significance (tested against naive+literal).
As presented in Fig. and Table is highly preferable to opt for BGI-agents (a pedagogical teacher with pragmatic learner). The benefits of adopting pedagogy and pragmatism are particularly promi- nent in the few demonstrations regime (100 per goal) where pedagogical+pragmeen performs 4 times better (going from 0.15 to 0.62) than the naive+literal one terms of GIAXGRA. A. Note that with less than 100 demonstrations per goal, none of the approaches able master all goals, even the hierarchy methods remains unchanged (pedagogy and pragmatism being better than naive and literal, see Fig. g in Appendix (.2.7). This is likely due SQIL forcing 50/50 split. of demonstrations and collected trajectories in the buffer.
combination
An analysis of the pragmatic learner policies reveals that they are better at predicting their own goals, as intended, with an average of 12.4% relative difference OGIA between a literal and a pragmatic learner. This, coupled with the effectiveness of pedagogical demonstrations shown in the previous section, helps the teacher and learner efficiently communicate and improves learning speed. The benefit of using pragmatism is prominent in the few demonstrations regime: the addition of pragmatism alone (red curve in Fig. 3 helps mastering all goals while pedagogy on its own (green curve in Fig. 3) is not enough do so. In additional experiments not reported here, verified that the value of the pedagogical reward and pragmatic reward does not affect performance.
Table 1: Goal Reaching Accuracy (GRA) and Goal Inference Accuracy (GIA) for the FBS environment.
Teacher Learner
GIA
GRA
GIAXGRA
With 100 demonstrations per goal
Naive + Literal
30.9 8.3%
50.2
0.15
Naive + Pragmatic
61.33 + 2.9%
84.8 4.1%
0.52
Pedagogical + Literal
49.6 + 6.6%
6 + 5.4%
0.31
Pedagogical + Pragmatic
69.2 + 4.2%
89.4 + 2.0%
0.62
With 500 demonstrations per goal
Naive + Literal
68.9 1.1%
90.6 + 1.4%
0.62
Naive + Pragmatic
76.3 + 0.4%
95.5 + 0.6%
0.73
Pedagogical + Literal
78.8 0.7%
91.5 + 1.1%
0.72
Pedagogical + Pragmatic
82.2 + 1.2%
96.8 + 0.6%
0.80
With 1000 demonstrations per goal
Naive Literal
75.3 0.9%
93.3 + 1.2%
0.70
Naive + Pragmatic
76.0 + 0.9%
96.6 0.4%
0.73
Pedagogical + Literal
80.0 1.6%
94.5 1.2%
0.76
Pedagogical + Pragmatic
84.7 + 1.8%
97.1 + 1.2%
0.82

---

5.3 Does using the teacher accelerate training for the learner?
In Phase of our setup, the learner is trained using teacher's demonstrations and RL on its own experience. But is the teacher useful? The
Learner (trained 1000 demonatrations)
Learner (trained w/ 500 demonstrations)
Learner (troined w/ 100 demonstrations)
answer lies in the training speed. We thus com- pare the training speed of pragmatic learners trained with 100, 500 and 1000 pedagogical demonstrations and their own collected expe- rience (using the same training procedure be- fore, with SQIL and GANGSTR), and a learner trained with the same architecture and training process, but no demonstration. We end up with a common result in the Learning from Demonstra- tions literature: the learner trains much faster (here roughly twice faster to obtain GRA of 90%) when it has access to demonstrations, as illustrated in Fig. This sanity check justifies training learners from demonstrations.
Goal Reaching Accuracy
Learner (trained without demonstrational
1.00
0.75
0.50
~Xx faster
to reach 90%
0.25
0.00
50
100
200
250
Timesteps (x10*)
Fig. Demonstrations double the learner's train- ing speed.
5.4 Could we use another goal inference method instead of Bayesian Goal Inference?
Table 2: Goal Inference Accuracy (GIA) of BGI and GPNN on FBS.
GIA for Pedagogical demonstrations
Nb of demos per goal
100 demos
500 demos
1000 demos
GPNN BGI
46.2%
70.0%
82.0%
69. 2%
82.2%
84.7%
GIA for Naive demonstrations
Nb of demos per goal
100 demos
500 demos
1000 demos
GPNN BGI
38.4%
67.1%
68.2%
61.3%
76.3%
76.0%
Our learner relies BGI using its own policy to infer the goals associated with the demonstrations of the teacher. Could we instead train a separate goal prediction module to learn how to achieve goal inference? We tested this by creating goal prediction neural network (GPNN) based on LSTM architecture (30 that inputs a demonstration and outputs a goal, and we trained it using the same demonstrations and goals as for the BGI case. In order to compare our BGI approach to this alternative, we trained GPNN on a different number of demonstrations per goal and compared the results separate test set. We performed this experiment Appendix B.2.4 Results from Table 2 show that BGI achieves higher GIA in all cases (naive or pedagogical demonstrations), and again performs especially well in the few demonstrations regime where the GPNN approach struggles because of too few training samples. However, note that the case of an infinite goal space, GPNN approach would be viable, while an approach based on BGI would not be straightforward to apply.
FBS and provide details in
5.5
What if we used another method to learn from demonstrations?
Our modified version of SQIL helps the learner leverage both demonstrations and the experience it collects. We justify the use of this method by comparing it to base- line approaches: learning only from demon- strations (B1), adding behavioural cloning on the demonstrations (B2) and the original version of SQIL (B3), All implementation
Table 3: Learning from demonstrations baselines.
Method
GIA (%)
GRA (%)
GIAXGRA
B1
5.3 + 1.5
6.2 0.7
0.00
B2
2.3 + 1.8
52.5 - 2.1
0.00
B3
59.1 2.4
73.8 - 1.7
0.43
Ours
69.2 = 4.2
+.04
0.62

---

details are provided in Appendix |.2.5 We perform the comparison using teacher's pedagogical demonstrations and pragmatism in the learner for all methods. B1 is not able learn, B2 is able to achieve reasonable GR A score but struggles to predict goals from demonstrations because the behavioural cloning updates interfere with the BGT mechanism by changing the action probabilities. B3 and our modified SQII. approach perform well, with a significant advantage for our approach. The original SQIL was designed for single goal environments and does not provide enough goal-reward associations, while our modification extends the approach to multi-goal environments by providing reward signal for the collected experience.
6 Conclusion
In this paper, building on the pedagogy and pragmatism concepts from Developmental Psychology, we have shown how learning from demonstration can benefit from Bayesian goal inference mechanism. Using BGT-agents over regular agents improves learning speed, resolves ambiguity in communication and globally improves performance, especially
the few demonstrations regime, In our work, the teacher's demonstration was insensitive to the learner's interpretation, as the teacher did not use a model the learner's policy to provide a demonstration tailored to that specific learner. Conversely, the learner did not have model of the teacher's policy which may help it interpret the goal the teacher wanted to convey. In the future, adding in each agent a model of the other agent's policy would result in closing the interaction loop and favoring the emergence of richer interaction mechanisms, such as both partners using signals to help the other update their model of each other during training. Besides, our approach was focused and improving goal inference for pragmatism. Pedagogy could be further improved with a curriculum of goals that would improve exploration (3. Pragmatism could be improved with pedagogy detection [22 and inductive generalization [23]. In the case of multiple teachers, the learner could also develop mechanism to evaluate teachers and select the most suited one [35 We leave all this for future work.
goal disambiguation for pedagogy
Finally, special trait of our learner is that it learns both from its own signals and from interaction with its teacher, in accordance with the guidelines [39]. While pedagogy and pragmatism are mechanisms that improve how agents can be taught, they suggest a number of other mechanisms to do so, which in combination to our approach could be beneficial such as language guided learning, internalization, motivation regulation and observational learning.
Acknowledgements
This work has received funding from European Union's Horizon 2020 ICT-48 research and innovation actions under grant agreements No 952026 (HumanE-Al-Ney) and No 765955 (ANIMATAS)
References
[1] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, page 1, 2004.
[2] Ahmed Akakzia, Cédric Colas, Pierre-Y Yves Oudeyer, Mohamed Chetouani, and Olivier Sigaud. Grounding language autonomously-acquioo skills via goal generation. In ICLR 2021-Ninth International Conference on Learning Representation, pages 1-21, 2021.
[3] Ahmed Akakzia, Olivier Serris, Cédric Colas, and Olivier Sigaud. Help me explore: Minimal social interventions for graph-based autotelic agents. arXiv preprint XXXi 2022.
[4] Marein Andrychowicz. Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbecl, and Wojeiech Zaremba. Hindsight experience replay. Advances in neural information processing systems, 30, 2017.
[5] Chris L. Baker, Noah D. Goodman, and Joshua B. Tenenbaum. Theory-based social goal inference. In Proceedings of the thirtieth annual conference of the cognitive science society, pages 1447-1452. Citeseer, 2008
10

---

[6] Chris L. Baker, Rebecca Saxe, and Joshua B. Tenenbaum. Action understanding as inverse planning. Cognition, 113(3):32--349 2009.
In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 29, 2007. [7] Chris L. Baker, Joshua B. Tenenbaum, and Rebecca R. Saxe. Goal inference as inverse planning.
[8] Daphna Buchsbaum, Alison Gopnik, Thomas L. Griffiths, and Patrick Shafto. Children's imitation of causal action sequences influenced by statistical and pedagogical evidence. Cognition, 120(3):331-340, 2011.
[9] Lucas P. Butler and Ellen M. Markman. Preschoolers use pedagogical cues to guide radical reorganization of category knowledge. Cognition, 130(1):1 2014.
[10] Théo Cachet, Christopher R. Dance, and Perez. Demonstration-ooniioonne reinforcement learning for few-shot imitation. International Conference on Machine Learning, pages 2376- 2387. PMLR, 2021.
[11] Cédric Colas, Tristan Karch, Olivier Sigaud, and Pierre-Y Yves Oudeyer. Intrinsically motivated goal-conditioned reinforcement learning: a short survey. arXiv preprint arXiv:2012.09830, 2020.
[12] Lisandro D Dalcin, Rodrigo R Paz, Pablo A Kler, and Alejandro Cosimo. Parallel distributed computing using python. Advances in Water Resources, 34(9):1 124-1139, 2011.
[13] Anca D. Dragan, Kenton C.T. Lee, and Siddhartha S. Srinivasa. Legibility and predictability of robot motion. 2013 8th ACM/IEEE International Conference on (III),, pages 301-308. IEEE, 2013.
Interaction
[14] Terje Falck- Ytter, Gustaf Gredebäck, and Claes Von Hofsten. Infants predict other people's action goals. Nature neuroscience, 9(7):878-87) 879, 2006.
[15] Jaime Fisac, Monica A Gates, Jessica Hamrick, Chang Liu, Dylan Hadfield-Menell, Malayandi Palaniappan, Dhruv Malik, S. Shankar Sastry, Thomas L., Griffiths, and Anca D. Dragan. Pragmatic-pedaaggrt value alignment. In Robotics Research, pages 49-57. Springer, 2020.
[16] Pierre Fournier, Cédric Colas, Mohamed Chetouani, and Olivier Sigaud. Clic: Curriculum learning and imitation for object control in nonrewarding environments. IEEE Transactions on Cognitive and Developmental Systems, 13(2):239-244 248, 2019.
[17] Michael C Frank and Noah D Goodman. Predicting pragmatic reasoning in language games. Science, 336(6084):998 9998,9,,)99998 2012.
[18) Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker- follower models for aiiioo-and-language navigation. Advances in Neural Information Process- ing Systems, 31, 2018.
[19] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. International conference on machine learning. pages 1263-1272. PMLR, 2017.
[20] Noah D. Goodman and Michael C. Frank. Pragmatic language interpretation as probabilistic inference. Trends in cognitive sciences, 20(1 (M(11)8188829 2016.
[21] Noah Goodman and Andreas Stuhlmüller. Knowledge and implicature: Modeling language understanding social cognition. Topics in cognitive science, 173:1133-184 2013.
[22] Hyowon Gweon. Inferential social learning: cognitive foundations of human social learning and teaching. Trends in Cognitive Sciences, 2021.
[23] Hyowon Gweon, Joshua B. Tenenbaum, and Laura E. Schulz. Infants consider both the sample and the sampling process in inductive generalization. Proceedings of the National Academy of Sciences, 107(20):00669077. 2010.

---

[24] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018.
[25] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[26] Mark K. Ho, Fiery Cushman, Michael I.. Littman, and Joseph L.. Austerweil. Communication in action: Planning and interpreting communicative demonstrations, 2019.
[27] Mark K Ho and Thomas L Griffiths. Cognitive science as a source of forward and inverse models of human decisions for robotics and control. Annual Review of Control, Robotics, and Autonomous Systems, 5, 2021.
[28] Mark K Ho. Michael L Littman, Fiery Cushman, and Joseph L Austerweil. Effectively learning from pedagogical demonstrations. In Proceedings of the annual conference of the cognitive science society, 2018.
[29] Mark K. Ho, Michael I.. Littman, James MacGlashan, Fiery Cushman, and Joseph L. Austerweil. Showing versus doing: Teaching by demonstration. In NeurIPS, 2016.
[30] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1735:1735 1997.
[31] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[32] Christina Lichtenthäler, Tamara Lorenz, and Alexandra Kirsch. Towards a legibility metric: How measure the perceived value robot. International Conference on Social Robotics, ICSR 2011, 2011.
[33] Smitha Milli and Anca D Dragan. Literal or pedagogic human? analyzing human model misspecification in objective learning. In Uncertainty in artificial intelligence, pages 925-934. PMLR, 2020.
[34] Stephen Neale. Paul Grice and the philosophy of language. Linguistics and philosophy, pages 509-559, 1992.
[35] Sao Mai Nguyen and Pierre Yves Oudeyer. Active choice of teachers, learning strategies and goals for a socially guided intrinsic motivation learner. Paladyn, 3(3):136-146, 2012.
[36] Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint 1802.0188. 2018.
[37] Siddharth Reddy, Anca D Dragan, and Sergey Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint 19X1v:1905.11/088 2019.
[38] Patrick Shafto, Noah D. Goodman, and Thomas Griffiths. rational account of pedagogical reasoning: Teaching by, and learning from, examples. Cognitive psychology, 71:55 89, 2014.
[39] Olivier Sigaud, Hugo Caselles- -uuléé, Cédric Colas, Ahmed Akakzia, Pierre- Yves Oudeyer, and Mohamed Chetouani. Towards teachable autonomous agents. arXiv preprint arXiv:2105.11977, 2021.
[40] Bradly C. Stadie, Siyan Zhao, Qiqi Xu, Bonnie Li, and Lunjun Zhang, One demonstration imitation learning. Advances neural information processing systems, 30, 2019.
[41] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEFF/RSJ international conference on intelligent robots and systems, pages 5026-5033. IEEE, 2012.
12

---

[42] Tianhao Zhang, Zoe McCarthy, Owen Jow, Dennis Lee, Xi Chen, Ken Goldberg. and Pieter Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleop- eration. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1028-5635.. IEFE, 2018.
[43] Tan Zuan, Jordyn L. Mann, Tom Silver, Joshua B. Tenenbaum, and Vikash K. Mans- inghka. Online bayesian goal inference for boundedly-rationn planning agents. arXiv preprint 2.Xv::2006.07322, 2020.
[44] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pages 1433-1438. Chicago, IL, USA, 2008.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section O
(c) Did you discuss any potential negative societal impacts of your work? [Yes] This work proposes a general framework for implementing pedagogy and pragmatism in learning from demonstrations, which does not have extra negative socictal impacts beyond learning from demonstrations.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material
URL.? [Yes]
(b) Did you specify all the training details (e.g.. data splits, hyperparameters, how they were chosen)? [Yes] See Appendix B
(c) Did you report error bars (c.g., with respect the random seed after running experi- ments multiple times)? [Yes] See all figures and tables.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix [B...3]
4. If you using existing assets (e.g., code, data, models) curating/releusing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "hhmnN3lPUvB", "reviewer": "Reviewer_4LZc", "summary": "This submission proposes a novel method to learn from demonstration. Inspired by human learning, the authors propose a Bayesian Goal Inference framework to avoid ambiguity in multi-goal learning. And use such a framework to benefit the learner's learning under the demonstrator's help.", "questions": "Major concerns:\n\n1. How is the research motivated, why does the pedagogical demonstration and inference of goals important? In GCRL, demonstrations should contain the true goal state, then what is the purpose of inferring multiple potential goals? Real-world applications can be helpful in demonstrating the importance of the studied problem.\n\n2. with those demonstration data, Hindsight Experience Replay methods can be applied to generate more successful goal-reaching experiences. I believe a comparison to HER can be helpful in evaluating the proposed method's performance. (e.g., in Table 3)\n\n3. As for the precise GCRL algorithm the authors used, it was mentioned in line 104 by referring to [11], however, this [11] is a survey paper. More description of the learning algorithm itself should be at least referred to in this section.\n\n4. during the learning of GCRL, the goal is known, then what is the purpose of inferring the goal?\n\n5. the proposed method does not have a theoretical guarantee but is mostly of a shape as an empirical study.\n\nminor:\n\n1. in GCRL (practically, the robotics control tasks, e.g., [36]), the state space and goal space (with a mapping between state and goal space always assumed to be known [36]) are continuous, how will the proposed method tackle such situations where the goal set has an infinite cardinality?\n\n2. Section 4.1 introduces several reward-shaping designs, is the proposed method sensitive to those choices? detailed ablation studies can be helpful to verify the proposed method.\n\n3. 5 seeds are not enough for evaluating RL algorithms.\n\n4. some presentations are ambiguous and need proofreading. e.g., line 100 'where learners are both...', I reckon there is only one learner?\n\nI may misunderstand some parts of the work, and I'm happy to raise my score if the above concerns can be appropriately addressed.", "limitations": "1. clear motivation of the research\n2. solid guarantee of the proposed method\n3. more empirical studies (#seeds and environments)", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I appreciate the novelty, and great effort put into the paper. Especially the illustration video in the supplementary material--- which helped a lot in understanding the work. \n\nHowever, there are several concerns I hope can be addressed according to the current stage of the presentation. \n1. motivation/ problem formalism: the authors failed to present the task in a clear enough way: e.g., why does the learner need to infer the goal when the goal is known?\n2. missing baselines/ lack of empirical results.\n3. lack of theoretical guarantee\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gMyGXvzyPsK", "reviewer": "Reviewer_yuXv", "summary": "* This work focuses on the problem of learning from teacher/learner demonstrations and in particular focus on training a teacher/learner setup where goal ambiguity can be resolved to improve the quality of the demonstrations and to thus improve effective goal inference and task performance.  Multi-goal environments are utilized.\n* The authors make use of concepts from cognitive science, pedagogy: optimization of teaching concepts; Pragmatism: resolution of ambiguities of intention from the teacher. \n* With this in mind the authors aim to tackle the problem of goal ambiguity: given that at least two goals are reachable in a given trajectory can reach at least two goals -> can't reliably predict pursued goal.  The authors aim to improve demonstrations by resolving goal ambiguity and learning good goal inference.\n* Teacher/Learner share common goal / state / action spaces and use separate policies.  Communication between the two is facilitated via teacher demos, learner's inferred goal, teacher feedback on inference.\n\nTraining proceeds roughly as follows:\n\n1. Teacher is pre-trained in the env to master/solve all goals with Goal Conditioned RL (GCRL)\n2. Learner infers goal from teacher's demo using Bayesian goal inference (BGI), teacher provides feedback, learner rewarded for correct predictions\n\nThe teacher tries to infer its own goal (with BGI) from trajectory after reaching it.  It adds a \"pedagogical\" reward to the trajectory if so which aids in retaining high performance on the task.\n\nResults measured over the Draw-Two-Balls (DTB) and Fetch-Block-Stacking (FBS) environments.  Four metrics are collected:\n\n1. Goal inference accuracy (GIA) - learner can infer G from d\n2. Own goal inference acc - teacher can infer G from its own demonstrations\n3. Goal reaching acc (GRA)  - can learner reach all goals?\n4. GIA x GRA - can the learner both infer its goal and reach it\nAmbiguity Score\n\nThe main claim of the paper is that using a pedagogical and pragmatic teacher/learner setup to improve demonstrations through goal disambiguation will lead to to faster learning with fewer demonstrations.\n", "questions": "\n* How can we ensure that all goals are mastered/solved/achieved by the teacher over the dataset?  What happens if we are only able to master some fraction?\n* In alg. 1 are flags provided for \"learner is pragmatic\" and \"teacher is pedagogical\"?\n* What do you think causes the accuracy dip for the learner curve with no demonstrations in figure 1?\n* What computational overhead does the pre-training occur?  How does this compare to the phase involving learning from the demonstrations?\n", "limitations": "Limitations weren't explicitly addressed in the main paper to my knowledge.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n* The authors aim to tackle an interesting problem of leveraging teach/learner demonstrations which are important in a variety of domains.  * The approach they take to disambiguate goals from existing demonstrations is very compelling and could lead to learning improvements in a variety of ways (signal boost, sample efficiency, data quality)\n* Overall the approach presented in the paper is sound and the paper is clearly written for the most part although there may have been a bit more in the way of architectural detail.\n* The authors provide useful context for their work in section 2 describing past relevant work in pedagogical demos, pragmatic inference and demonstrations in goal conditioned tasks.\n* The training phases are detailed well with accompanying algorithms.\n* The authors show that in the context of naive/simple v. pedagogical/pragmatic teacher/learner setups the pedagogical+pragmatic variant has a clear advantage in the evaluated tasks.\n\n\n**Weaknesses**:\n\n* In the  methods section some more explanation on how new goals are discovered could be helpful.  I'm a bit foggy on how the goals are actually selected.\n* In section 4.1: A bit more explanation around the distinction of literal v pragmatic learners and naive v pedagogic teachers would be helpful.  Some basic details in the main paper would be helpful.  Same goes for the architectural details of the teacher/learner networks.  The training and interaction is explained however there are no clear architecture diagrams and not much explanation in the main paper.\n* Results in the main paper are limited to a single task and while they show better sample efficiency in this case it is hard to understand the benefit realized more generally.  Further the 95% threshold for the 2x increase claimed seems somewhat hand picked (e.g. why not 80% or 90%?  These would reduce the sample efficiency significantly). \n* From the results it seems clear that Pedagogical/Pragmatic Teacher/Learner improve over Naive/Simple variants however it is less clear how well these improve more generally over other approaches.  Section 5.5 seems to address this over a limited set of baselines including Behavioural Cloning and SQIL but these don't seem to be particularly strong baselines (a couple of them don't even reach a substantial number of goals in the demos).  I believe more rigorous comparisons to baselines would be needed here in order to really justify the claims of this work.\n * In figure 4 there is a notable dip over the learner without demonstrations at roughly 6e5 learner steps.  It would be useful to explain what this might be, especially since it seems to be present for all seeds and only for this condition. It would be helpful to understand these gains in a broader context to understand the full benefit of the work here.  It also may have been useful to try with fewer than 100 demonstrations as performance seems to be mostly unaffected when reducing the number of demos from 1000.\n* There are no results for DTS results in the main paper, including these could be useful.\n\nOverall I think that this is a promising are of work but some more detail would help to explain the approach and the experiments/results would benefit from being more comprehensive to satisfy the claims.\n\n\n[2022-07-27] EDIT:  I've added in the missing bit the authors referenced in their comment (I can't reply Officially yet, so adding here).  Apologies for that and thanks for flagging.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Nx17r22J8Q", "reviewer": "Reviewer_3RPR", "summary": "This work proposes utilizing teacher-learner settings in Goal-conditioned Reinforcement Learning with concepts of pedagogy and pragmatics from other disciplines. In particular, the paper introduces the process of Bayesian Goal Inference (BGI), which is a reward-shaping method that encourages policies to learn behaviors that allow an observer to infer the goal given the trajectory. The paper uses BGI into two settings: to augment a teacher policy’s behaviors to have more disambiguation (“pedagogical teacher”), and to augment a learner to filter our teacher demonstrations and to predict it’s own goals (“pragmatic learner”). The paper applies this method to a toy setting and then a simulated robotic block-pushing setting, and show that pedagogy and pragmatics improve goal reaching and goal inference for both the teacher and the learner.", "questions": "- A lot of benefit results from going from literal learner => pragmatic learner, ie. the BGI pedagogical loss for reward shaping. Perhaps the fairest comparison for experiment 4) would be to include cumulative environment interactions (including how many the teacher used). Thus, it can be possible to tell whether the driving benefit results from on-policy updates to encourage the experience-collecting policy (whether it is teacher or learner) to have more disambiguated trajectories. Can you try an experiment where you give the learner (no demonstrations) as many on-policy updates w/ BGI as the cumulative teacher+learner on-policy update budget?\n- I could not find information on how much ambiguity results from the environments, as measured by the Ambiguity Score. How percentage of trajectories in the Naive/Literal methods (without BGI) result in ambiguous policies? How does this scale to more complex tasks in higher-dimensional spaces such as learning from vision, where the BGI measure might be much noisier?\n- How expensive is the BGI prediction, and how does it scale to continuous action spaces or continuous goal spaces? Given that the method relies on the learner BGI heavily, how does the method fare when goal inference becomes more challenging or a worse method is used? Some possible experiments might be to use GPNN instead of BGI in the main experiments, or an ablation increasing the number of goals.\n", "limitations": "It would be interesting to see how well the method scales to settings where ambiguity is much harder to quantify. In the paper, the authors illustrate the challenges of ambiguity with a three-block toy problem, but in the real world goal inference is often challenging even for experts. In such settings, goal inference auxiliary tasks (like BGI) may not help performance, and may even hurt occasionally. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The method is quite simple and easily usable with any GCRL + LfD method.\n- Since the method implementation in this paper did depend on specific underlying decisions (ie. teacher-learner setup, BGI, SQIL), the ablations were useful to showing the method still works in other settings\n- The method increases goal-reaching accuracy in addition to goal-inference accuracy\n\nWeaknesses:\n- The method is only tested on a fairly simple block-pushing environment. In more challenging environments (high-dimensional action and observation spaces, more ambiguity in the goal space), it is unclear how noisier BGI predictions will affect the method as it is used for reward-shaping\n- Much of the benefit of the method arises from the learner’s utilization of BGI. While 5.4 studies a teacher-less setting, it’s unclear why it’s much more important to have a pragmatic learner than a pedagogical teacher. See questions below.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Bmah8rnViHc", "reviewer": "Reviewer_ozr3", "summary": "This paper leverages the ideas of pedagogy and pragmatism for training goal-conditioned RL agents in multi-goal environments with demonstrations. Pedagogy refers to the ability of the teacher to provide non-ambiguous demonstrations to the learner, whereas pragmatism refers to the learner’s ability to discern which goal the teacher intended to demonstrate. The algorithm consists of two phases: 1) a goal-conditioned teacher policy is trained using off-policy trajectories. In addition to the reward from the environment, the teacher policy receives a reward for correctly inferring the intended goal of its own trajectory using Bayesian Goal Inference (BGI). 2) The teacher provides demonstrations to a new learner policy, which receives rewards from the environment and additional rewards for correctly inferring the intended goal of the teacher's demonstrations, and the intended goal of its own trajectories. The experimental results show that pedagogical teachers and pragmatic leaners are useful when learning from demonstrations.", "questions": "* Is the teacher's ability to master all goals in the environment a requirement for the use of this algorithm? Were any experiments done to see if training both the student and teacher in a cooperative manner had better results (essentially combining phases 1&2 together)? \n* How is the first goal chosen in phases 1 and 2? Could strategically choosing the first goal impact the performance of the teacher and student?\n* Do the authors think that increasing the number of demonstrations would make GPNN perform better compared with BGI?", "limitations": "The limitations were well addressed in Section 6.  ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Originality\nThe related work section is comprehensive, and I appreciate the authors breaking the different areas into different sections. This work seems to be a novel combination of pedagogy, pragmatism, BGI, and goal-conditioned RL in multi-goal environments.\n\n### Quality\nThe experimental results were presented well and clearly addressed the contributions stated in the introduction. The authors do a great job of including specific information such as architecture, number of random seeds, mean, variance, etc. \n\nThe questions that the experiments aim to answer seem to be 1) can the learner infer goals from the teacher, 2) can both the learner/teacher infer goals from their own trajectories, 3) can the learner reach all goals, and 4) can the learner both predict goals and reach them.\n\n$GIA$ addresses 1, $OGIA$ addresses 2, $GRA$ addresses 3, and $GIA \\times GRA$ addresses 4. The results for $GIA$, $GRA$, and $GIA \\times GRA$ are included in Table 1, but I did not see the results for $OGIA$?\n\n### Clarity & Significance\nOverall, the paper is very well written and easy to follow. The authors motivated the work in an intuitive way, and also highlighted potential applications. The results seem to be significant, especially for areas like robotics where acquiring demonstrations can be expensive. Researchers can easily extend this approach to use pre-existing policies for the teachers, then further refine them to be pedagogical. \n\n* In Section 5.1, it is implied that the experiments are referring to Fetch Block Stacking, but I think this should be stated in the first sentence for clarity. E.g., \"We verify that the pedagogical teacher can indeed better predict goals from its demonstrations compared to a naive teacher 'in the FBS environment'\".\n* I don't completely understand  how the Draw Two Balls environment is defined. If there are purple, orange, and pink balls, and two consecutive draws, how are there only three possible outcomes?\n* Nits\n    * I think the second line of Figure 5 should say \"goal 2 (activation of sound 2)\"\n    * Line 85: remove the comma\n    * Line 119: add comma \"Finally,\"\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept