Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes and validates a qualitative distinction between instructions and descriptions as ways of communicating partial information about a reward function.  Briefly, instructions are shown to be more useful over short horizons.  This is framed as a social learning problem, with a pragmatic listener; this resembles previous work on Inverse Reward Design.

This distinction is supported theoretically, and with synthetic and human experiments in a toy linear (contextual) bandit setting (see Figure 1).  The human experiments show strong but not perfect agreement with the theory, and there is a discussion of these results including this discrepancy, which is attributed to humans being more likely to convey literally true information as opposed to the information that would be most useful from a pragmatic point of view.

strengths_and_weaknesses: Strengths:
* The proposed distinction is insightful and useful, and well described and supported.
* The experiments and analysis are sensible and support the central point of the paper.
* Human experiments are a great addition.
* The experiments on inferring the horizon vs. assuming knowledge of the horizon were interesting and provide an important counter-point to Milli and Dragan [41].  



Weaknesses:
* The experimental setting is extremely simple and the work does not demonstrate any immediate practical utility.  A particular limitation with the author's mention is the need to ground natural language instructions and descriptions, which is not addressed in this work.  Overall, however, I think the work still does make a compelling case for further investigation, and the experiments contribute substantially to that case.
* I found the discussion of prior work on instructions misleading, and I think it needs to be more nuanced.  Instructions play a different role in this work versus, e.g., Badhanau et al. [11].  In that work (and other works that I have seen on instruction following), the instructions are meant to be a complete specification of the task/reward, and the method is meant to train a single agent which can be instructed to do different tasks.  In this work, the instruction is meant to be a partial specification of a single task, and the purpose of the method is for the learner/listener to make the best inference possible about the reward from that instruction.  
* I found the analysis lines 155-170 of Section 3.4 weak and unconvincing.  This establishes an equivalence between reward design, demonstrations, and instructions, but under fairly strong conditions.  These conditions should be discussed further, As it is unclear how broadly they apply and what happens when they do not apply.  It is also unclear how this analysis is meant to contribute to the central point of the paper.
* The approach is characterized as inverse reward design (IRD), but IRD is a more specific (albeit closely related) problem.
questions: Question: Why is “fixed effect of listener and random effects for each of the 2772 utterance-context pairs” used on line 295, but not (If I understand correctly) lines 266-268?


Suggestions:
* Line 62 needs references.
* In line 72, [9-14] should be placed after “infer”
* It should be clarified that this is a *contextual* bandit setting.
* There are typos on 164, 223, 262.
* In Table 1, It is not clear enough what the “future rewards from utterance” column refers to.  
* There should be more detail on the selection of $\beta_{S_1}$, to assure the reader that it is not (in some sense) cherry-picked.

limitations: Yes I believe they have.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper explores the idea of learning-from-teaching in an RL setting, where a teacher can provide either direct instructions or descriptions of a reward function.  The paper focuses on linear bandits, and introduces a sort of theory-of-mind framework whereby a learner can reason about a teacher's mental state. The paper validates their theoretical models with a human experiment, and show close agreement between human performance and the model's predictions.
strengths_and_weaknesses: I generally liked this paper. The framework is clear, subtle, interesting and potentially useful.

+ I liked the human experiments, and the validation of the model's predictions
+ The mathematical formulation of the model and the setting was clear
+ I appreciated the authors' attention to detail, and the careful buildup of ideas
+ The use of various statistical tests to support conclusions was tidy and well done
+ I especially enjoyed the bits where the agent reasoned explicitly about the speaker's unknown H. I think this sort of explicit reasoning about mental states needs to be more common in RL.
+ Section 4.3 was intriguing, I was glad to see it included

- The paper feels a bit contrived. I found it unsurprising that instructions were better for small H, and descriptions for long H; it almost feels like the problem setting was conceived to support this eventual conclusion. And so while I generally believe the claims in the paper, I have a hard time believing that this will scale much beyond this toy setting.  It always rubs me a /bit/ the wrong way when the story is much bigger than the result.
- Upon reflection, it was unclear to me what generalizable knowledge I should take with me after reading this paper, because if we eliminate the framing, the conclusions seem too closely tied to the specifics of the experimental setup.  Does this really help me in the non-bandit setting? Does this really help me build better teachers? Does this really say anything about building better agents? Does this really tell me when to prefer instructions vs. descriptions?  I appreciate that scientific knowledge must be built brick by brick, and so I am not unhappy with the contribution as it stands, but I do wish that that the ideas felt more generalizable.  (That said, the strengths of this paper outweigh its weaknesses, so I will argue for acceptance.)
questions: None
limitations: The authors do a good job of addressing limitations and ethical concerns.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper studies learning reward functions from natural language feedback. It proposes a linear bandit framework to model learning about the reward function by communicating with a teacher through natural language. The paper focuses on the distinction between _instructive_ feedback, i.e., saying which actions to take, and _descriptive_ feedback, i.e., describing which state to achieve. The main thesis is that instrutive feedback is preferred for short planning horizons and descriptive feedback is preferred for long planning horizons. The authors provide theoretical results to support this claim, as well as a behavioral experiment using humans giving and learning from natural language feedback.
strengths_and_weaknesses: **Strengths**
- The paper studies value alignment, a very important problem for designing robust AI systems. The paper succeeds in making this problem concrete enough for being relevant for practical applications.
- The conceptual insights about different types of language feedback are interesting and novel.
- The authors provide an extensive behavioral experiment with humans providing strong evidence for their hypothesis about instruction and description feedback.
- The proposed theoretical framework seems simple enough for providing meaningful theoretical insights, but general enough to extend to more complex types of interaction in the future. As such the paper could lead to interesting future work.
- The paper is very well-written and easy to understand.


**Weaknesses**
- I find the theoretical results not very surprising or insightful. To me Theorem 1 and to some extend Theorem 2 seem to follow quite directly from the problem setup (I did not check the proofs)
- The notion of planning over a time-horizon H is a bit artificial in the linear bandit setup, as it is not a full RL problem. I think the bandit setup is still interesting, but the paper should make it clearer that this is a repeated 1-step decision problem and there is no real planning involved.


questions: About Sec 3.3.: What's the difference between having the objective of the teacher be to induce true beliefs in the learning agents vs. causing the agent to achieve high utility? Can you elaborate more on choosing the latter rather than the former? Is there some fundamental trade-off between achieving high utility and having true beliefs? I'm guessing that these two goals are highly correlated, and I was surprised by this section which sounds like there is a trade-off.

**Minor suggestions:**

- It could be useful to expand the caption of Figure 1 to be easier to understand without the context of the main paper (for readers skimming the paper)
- Is Figure 1.B based on real humans or the theoretical model? Would be good to include this in the figure caption.
- In most of the paper you talk about "agents", but in the related work section you sometimes refer to "the robot". Are these the same, or are there parts that are specific to robotics? If not, it would be better to use consistent language.
limitations: The paper could highlight the limitations of choosing a bandit model more clearly. There is a clear trade-off between having a practically applicable model and a model that you can analyze theoretically, which usually motivated bandit models. It would be nice if the paper discussed this more explicitly.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors present a model of communication between teacher and
student in an action/reward setting (reinforcemnt learning), wherein
the message or communciation contents are either to do explicit
actions (do this or that) or more general descriptions of reward (what
aspects of the world or state tend to lead to positive/negative
rewards). They call it "instruction" (do specific actions) vs
"descriptions" (describing the reward function in more abstract or
general ways, which in the long run might be more effective). The
authors claim that the latter has been less explored in the research.
Through behavioral experiments, in specific limited settings (a linear
bandit setting), the authors show that humans (as teachers) tend to
tailor their communication appropriately, eg depending on the horizon
of the problem: if the human knows the horion of the task is long,
they will pick descriptive messages (but if the horizon say only one
step, they may just utter do this!). The algorithms they develop
("pragmatic agents") together with their experiments and technical
results, provide evidence that this kind of ("social") learning, in
particular learning the teacher/speaker's reward function, can be
achieved and is useful. For instance, the agent that are helped
initially by a human teacher, reach lower regret faster initially,
compared to a pure RL agent without this bias, and those with
descriptive (more general teaching) remain competitive with the pure
RL agent even after many episodes.  The authors conclude that
considering wider means of language use, rather than limiting to just
instructions, has promise for achieving value alignment, between
humans and artificial agents (in RL settings, etc).

strengths_and_weaknesses: Summary of strengths and weaknesses.

Strengths:

-- The novelty of the approach: can broaden the approaches to
teaching/communicating of best ways to behave (value alignment, etc).

-- The behavioral experiments shed some light on how humans, as
teachers or communicators of what to do, can behave. In addition there
are technical and empirical results.

-- Situating the work within the wider context is well done.

Weaknesses:

-- Captures a very limited window to language with a very limited
experimental set up (seems highly special case, or contrived).  Not
clear whether/how this might apply to real human-robot (or robot to
robot) interactions in practice.

-- The results regarding humans (the behavioral experiments) are not
 very suprising: humans do tailor their communication depending on
 context, and the context could be, in part, the horizon of the
 game/task (still, it is useful to show this in actual experiments,
 and agents that can use this to their benefit) (other relevant
 context includes: the level of the listener (which can depend on age,
 education level, ..), what information has already been conveyed, the
 situation both the speaker and listener are in, etc. etc.).

-- The experiments made a number of simplifying assumptions (eg, the
 concreteness of the communication, and as the authors state, ignoring
 the grounding problem).  It is not clear whether they tackle the most
 challenging problems of humon-robot (or robot to robot) interaction.

-- The clarity of the technical results, the formalism, could be
   improved (see below in 'Questions' section).

questions: I'll include misc comments, notes, suggestions, and questions here.

In line 48: "We find that short-horizon speakers (focused on a single,
known state) prefer instructions, while long-horizon speakers
(reasoning about future, unknown states) prefer descriptions.", it's
not that speakers are short-horizon, but how the problem is set up for
them (perhaps add a footnote to further clarify this).



footnote 5, 'available' what? in "However, as the number of available
increases,"

Section 3.1: In Pragmatics subsection ..

--> could it be removed? I am not sure if I saw its relevance to the
 rest of the paper (if you keep it, a few issues next).

--> (still in 3.1) Further describe/explain this and subsequent
formulae (this whole subsection is pretty complicated to decipher.. I
am thinking it's better to move it to after the following
subsections..  or remove.. ):

U(u;w^*)=ln P_{L0} (w^* |u)

-> line 103:  What is soft? in "(soft-)optimally"

-> Describe L1 more, say in "of interest emerges at the L1 level."
(a brief concrete example, perhaps?)

-> In fact, 3.2 was more clear and should come first (before 3.1),
especially the helpful statement that describes your particular
problem setting: "While the bandit problem is typically considered an
individual learning problem, we instead ask how an agent should learn
from a cooperative, knowledgeable partner. We formalize this social
learning problem by introducing a second agent: a speaker who knows
the true rewards w and the initial state s0 , and produces an
utterance u." (after 'partner' perhaps add '(speaker).' to make more
clear )

-> [probably a case of notation abuse or misuse] Note on line 122, the
 notation pi_L (a | u, s) is ambiguous (where pi is a policy), since
 the previous paragraph described policies as mapping states to
 action, while here pi_L is being applied to some conditional
 expression (action a given Speaker has uttered utterance u and given
 state s).. I am reading it as a policy that's a function of or is
 modified by, what the speaker has uttered (I think this is your
 intension), but this is ambigous.

-> In line ~132: "Rather than defining the speaker’s utility U (u; w)
 as Gricean informativeness [69] (i.e. inducing true beliefs, Section
 3.1), we suggest that a cooperative speaker should maximize the
 listener’s rewards, thus grounding utility in the listener’s
 subsequent actions.", this is not really a suggesetion, but happens
 to be the way you have set up the problem.  In this setting, indeed a
 cooperative speaker or teacher has to somehow teach the agent (of
 what actions or policy), and it is not surprising or unexpected the
 communication will take such a form (what specific actions to take,
 or more general information about the class of useful actions,
 depending on state or context).

--> I disagree with the subsection (section 3.3) title "Speakers as
 reward designers": speakers (or good teachers), convey (their
 understanding of) the reward function the best they can.. they don't
 'design' rewards (perhaps this title could be understood as the
 speaker trying to shape the reward function that is forming in the
 listener).

--> line 120 (and elsewhere), in 'We formalize this {\em social}
 learning problem', minor comment: to me, this appears to be a very
 specific case of 'social' learning. For instance, social learning,
 can take place by watching what others do (not just what they say),
 and often more than 2 agents are involved (the learner observes and
 learns from more than one 'teacher' and over several
 episodes). Perhaps further clarify, for instance, convert this line
 to: 'We formalize this type of social learning..'

--> [general comment, related to previous one] Why have one (a single)
  utterance only by the speaker? What about multiple rounds of
  interaction? does learning or the analysis become more complicated?
  Some discussion of that would be useful.

limitations: 
Yes, I believe so. The authors discuss potential negative societal impact.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The goal of this paper is considering a social learning setup to see how humans might communicate preferences over behaviours. The motivation for this is value alignment; making models understand what we want. It's difficult to specify desired in numerical reward functions, so this paper explores language as a means of communicating preferences.

The running example environment is foraging for mushrooms. This is formalised through linear bandits with a listener agent that takes actions (choosing a mushroom) and gets a reward based on a weighted linear combination of the features that make up the mushroom (e.g. spotted green). The listener agent takes actions at each time step in the horizon H. Each feature has a weight, and the combination determines the total reward. Stripes have a reward of -1, but green +2, making a striped green mushroom net-positive.
There's a speaker that produces utterances to maximise the reward of the listener. An utterance can be either an action (e.g. eat the striped green mushroom) or a description (stripes give -1 reward). The speaker produces one utterance per horizon H, so if the horizon is 1 the optimal utterance is to just point to the best action with an instruction, whereas if the horizon > 1, it's more efficient to choose a description that can be re-used at different states of the horizon.

*Literal listener*: interprets utterances literally, executes instructions directly (if utterance is "striped green mushroom", it eats that)
*Pragmatic speaker*: (soft-)optimally selects utterances to maximise reward of literal listener, this means that when the horizon is known the speaker just chooses the optimal utterance to maximise reward (instruction or particular description based on the horizon)
*Pragmatic listener*: pragmatic refers to there being some information outside of the literal utterance that influences how the utterance should be interpreted, and this information is unknown. In this setup, the information is the horizon. When the speaker says "take spotted red" or "spotted is +1" both utterances suggest taking the spotted red mushroom, but the former suggests the horizon is 1 and the latter > 1.

The contributions of the paper are presenting a formal model of cooperative social learning in a linear bandit setting, and showing that short-horizon speakers prefer instructions where long-horizon speakers prefer descriptions. The second contribution is showing that jointly inferring the speaker's horizon and reward function can mitigate the risk of model misspecification, and the third contribution is verifying the theoretical model with a behavioural experiment and showing that the model helps in a traditional RL setting.

strengths_and_weaknesses: **Strengths**
- Value alignment by understanding humans through language is an important problem
- Propose the setting of linear bandits with a social aspect (I can't judge whether this is novel due to lack of experience in theoretical RL)
- A behavioural experiment is done to verify the theoretical model and this shows (to some extent) that the theoretical model makes sense.
- The idea that learning from re-usable descriptions can help agents generalise, and these descriptions might be given in language form.

**Weaknesses***
**EDIT** this main weakness is addressed and was not a weakness but a misunderstanding on my part. Rating adjusted.
My main problem with this paper is that I take some issue with the claims about language, and therefore don't directly see the contribution towards learning from descriptions in linguistic form. The way I see it is as follows. What the authors call language in this work is (1) instructions that map 1-1 to actions where the grounding problem is assumed solved (e.g. the instruction "eat the spotted green mushroom" maps to the action "spotted green" and the model knows this a priori) and (2) descriptions that map 1-1 to features "spotted +1" maps directly to spotted mushrooms and their feature weight. These instructions/descriptions also give away the latent information about the horizon of the linear bandit game: if an instruction is used the horizon is short, if a description is used the horizon is longer. I don't think this qualifies as language (i.e. a symbolic reference system using syntax and symbols combined with context into meaning), because there's no notion of context-dependence, there's no notion of syntax, basically these symbols (e.g. instruction "spotted red") are just noisy indices that point directly 1-to-1 to something. In light of this, how I interpret the finding/contribution of this paper is that providing an agent in a linear bandit setup with noisy information about the reward function prior to the choice of action is useful in the early stages of learning where the agent is still very uncertain about the distribution of rewards. Additionally, this information might contain latent information about the potential rewards and jointly inferring this helps. This noisy information might be given in linguistic form, but I don't see how this paper makes a contribution towards actually giving it in linguistic form. I don't directly see how it tells me anything about how "descriptive language and [...] are powerful mechanisms for value alignment and learning" (which is what the authors summarise their result as). I would be happy to be convinced otherwise if the authors could tell me what I'm missing with regards to the formal description of the problem here, as it's certainly possible that I've missed something.
To me, the interesting challenge about learning from descriptions and the example the authors use to introduce their problem, namely learning in a situation where simply exploring through experience is too dangerous (eating the wrong mushroom is dangerous), is how to generalise from language instructions and descriptions (e.g., "don't eat green striped mushrooms but spotted mushrooms are tasty!") is precisely the grounding problem (how does this green striped mushroom relate to the new mushrooms I encounter in the forest?).
That said, the following statements did make me think this paper is about learning rewards directly from language:
- "Yet today, we lack computational models explaining such flexible and abstract language use. To address this challenge, we consider social learning in a linear bandit setting and ask how a human might communicate preferences over behaviors"
"flexible and abstract language use" and "how humans might communicate preferences" are overstated imo
- "Our findings suggest that social learning from a wider range of language—in particular, expanding the field’s present focus on instructions to include learning from descriptions—is a promising approach "
- "Finally, we conduct a behavioral experiment showing our models support strong reward inference from human-chosen utterances."
This makes me think humans will describe situations and the model can learn from those, whereas in reality in the behavioral experiment humans choose from a dropdown menu of the instructions/descriptions, again making it less "utterances" that are part of language and more indices in an English representation. 
- the claim that 99 participants "produced a total of 2772 utterances".
Production of utterances make me think the humans in the behavioral study did more than selecting from a dropdown menu.
questions: - I think what would help me with this paper is reframing it such that the finding is that providing an agent in a linear bandit setup with noisy information about the reward function prior to the choice of action is useful in the early stages of learning where the agent is still very uncertain about the distribution of rewards. (the claim that pragmatic inference helps I have less issues with in its current form), and that this noisy information could potentially be given in the form of language. Then acknowledging all the problems that come with learning directly from descriptions, like grounding, ambiguity, generalising to novel situations, etc, that need to be solved.

- Relatedly, I wonder if relaxing the assumption that grounding is solved would completely change the game. Again, I think the difficult and interesting problem is grounding language to states. I would like to see some discussion on this.

- When reading about the rational speech act and the recursive reasoning in section 3.1, I think an example of how this would work would be helpful and clarifying.

- line 223: "long-horizon speakers prefer instructions" typo: instructions should be descriptions?
limitations: To some extent, but I'd also like to read a bit of discussion about the limitation of using language to directly attach a reward to an action, as opposed the more general case where getting a reward by following language instructions might take multiple actions.
review_title: null

Meta-review: metareview: I thank the authors for their submission and active participation in the discussions. The paper investigates language instructions and descriptions as a way to teach a student RL agent. All reviewers unanimously agree that this is a solid paper worthy of acceptance. In particular, reviewers found the paper to be well motivated [vzCm] and well written [mi6j], tacklign an important problem [feLE]. The experiments are convincing [vzCm], and insightful [mi6j], and the method interesting [aJ8p] and novel [8vXv]. Thus, I am recommending acceptance of the paper and encourage the authors to further improve their paper based on the reviewer feedback.
confidence: Certain
Decision: Accept