How to talk so AI will learn: Instructions, descriptions, and autonomy
Authors: Theodore Sumers
Robert D. Hawkins
Mark K Ho
Thomas L. Griffiths
Dylan Hadfield-Menell
Venue: NeurIPS 2022 Accept (2022)
Keywords: value alignment
language
instructions
descriptions
pragmatics
social learning

Abstract: From the earliest years of our lives, humans use language to express our beliefs and desires. Being able to talk to artificial agents about our preferences would thus fulfill a central goal of value alignment. Yet today, we lack computational models explaining such language use. To address this challenge, we formalize learning from language in a contextual bandit setting and ask how a human might communicate preferences over behaviors. We study two distinct types of language: instructions, which provide information about the desired policy, and descriptions, which provide information about the reward function. We show that the agent's degree of autonomy determines which form of language is optimal: instructions are better in low-autonomy settings, but descriptions are better when the agent will need to act independently. We then define a pragmatic listener agent that robustly infers the speaker's reward function by reasoning about how the speaker expresses themselves. We validate our models with a behavioral experiment, demonstrating that (1) our speaker model predicts human behavior, and (2) our pragmatic listener successfully recovers humans' reward functions. Finally, we show that this form of social learning can integrate with and reduce regret in traditional reinforcement learning. We hope these insights facilitate a shift from developing agents that obey language to agents that learn from it.

How to talk so AI will learn:
Instructions, descriptions, and autonomy
Theodore Sumers
Robert D. Hawkins
Mark K. Ho
Computer Science Princeton University sumers@princeton. edu
Princeton Neuroscience Institute
Computer Science Princeton University
Princeton University
rdhawkins@princetoom edu
mho@princeton..o edu
Thomas L. Grifliths
Dylan Hadfield-Menell EECS, CSAIL MIT dhm@csail mit.edi
Computer Science, Psychology Princeton University t tomg@princetonm edu
Abstract
From the earliest years of our lives, humans use language to express our beliefs and desires. Being able to talk to artificial agents about our preferences would thus fulfill central goal of value alignment. Yet today, we lack computational models explaining such language use. To address this challenge, we formalize learning from language in a contextual bandit setting and ask how human might communicate preferences over behaviors. We study two distinct types of language: instructions, which provide information about the desired policy, and descriptions, which provide information about the reward function. We show that the agent's s degree of autonomy determines which form of language optimal: instructions are better in low-autonomy settings, but descriptions are better when the agent will need to act independently. We then define pragmatic listener agent that robustly infers the speaker's 's reward function by reasoning about how the speaker expresses themselves. We validate our models with behavioral experiment, demonstrating that (1) speaker model predicts human behavior, and (2) our pragmatic listener successfully recovers humans' reward functions. Finally, we show that this form of social learning can integrate with and reduce regret in traditional reinforcement learning. We hope these insights facilitate a shift from developing agents that obey language agents that learn from it.
Introduction
As artificial agents proliferate in society, aligning them with human values is increasingly important [1] 3.. But how can we build machines that understand what we want? Prior work has highlighted the difficulty of specifying
desires via numerical reward functions [3 Here, explore language as means to communicate them. While most previous work on language input to Al systems focuses on instructions (6-19), study instructions alongside more abstract, dexcriptive language 25) We examine how humans communicate about rewards and formalize learning from this input.
To consider how humans communicate about reward functions, imagine taking up mushroom foraging. How would you learn the rewards associated with different fungi (i.c. which are delicious and which d In such setting, learning from direct experience (26) is risky; most humans would seek to learn socially instead. So how might we learn reward functions from others? Prior work in reinforcement learning (RL) has examined number of social learning strategies, including passive inverse reinforcement learning (observe an expert pick mushrooms, then infer their reward
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

A
B
1/
#
instructions
descriptions
rewards
2
0
-2
Take spotted red
Spots aru +1
rewards w
P(wlj)) P(u)u)
3
utterance r
0
2
2
horizon H
-1
pragmatic listener
literal listener
speaker
state s
Figure 1: We formalize learning from language in contextual bandit setting. A: Linear rewards associated with features determine whether actions (mushrooms) are high or low reward. B: We consider two forms of language. Instructions tell the listener to take a specific action, while descriptions provide information about underlying reward function. A literal listener interprets utterances according to these fixed semanties. C: We introduce speaker model suggesting that humans use instructions and descriptions adaptively to maximize literal listener's expected rewards over some task horizon. We assume the start (i.e., the present) is known to both speaker and listener, but future states are unknown. the listener will act in. Short horizons thus formalize low autonomy settings where the agent receives frequent linguistic input. while long horizons are high autonomy. In order to learn from linguistic input, we define pragmatic listener which Bayesian inference recover the speaker's latent reward function.
horizon determines
total number of states
active preference learning (offer expert pairs of mushrooms, observe which one they eat, and infer their reward function 31).
function 27
28
However, few humans would rely on such indirect data if they had access to cooperative teacher 32
35
For example, an expert guiding a foraging trip might demonstrate 36 or verbally instruct
the learner to pick certain mushrooms, licensing stronger inferences. While such pedagogical actions have been useful for guiding RL agents [10 [7 [9], natural language affords richer, under-explored forms of teaching. For example, an expert teaching seminar might instead deseribe how to recognize edible or toxic mushrooms based on their features, thus providing highly generalized information.
To formalize this process, we present model of learning from language in a contextual bandit setting (Fig Th. We propose a speaker model that chooses utterances to maximize the listener's 's expected rewards over some task horizon. The horizon quantifies notions of autonomy described in previous the horizon short, the agent is closely supervised; at longer horizons, they are expected act more independently. We then analyze instructions (which provide a partial policy) and descriptions (which provide partial information about the reward function). We show that instructions optimal at short horizons, while descriptions are optimal at longer ones.
work 13 14 25
Second, consider how a listener might learn from such speaker. We define a pragmatic listener that infers the speaker's 's latent reward function based on their utterances. While prior work suggests pragmatic learning can be vulnerable to model siissppeciiiatiin (38), we show that jointly inferring the speaker's 's horizon and reward function can mitigate this risk.
Finally, we conduct online behavioral experiment which shows our models support strong reward inference. We integrate this social inference with traditional RL and find that it accelerates learning and reduces regret. Overall, results suggest that descriptive language and pragmatic inference powerful mechanisms for value alignment and learning !
2 Related work
Classic RL assumes that the reward function is given the agent 226
However, in practice,
motivating learning the reward function from social input Most social learning methods assume the expert is simply acting
is difficult to specify reward function to obtain desired behavior 3
'Code and data are available at /tttttps://giit
com/ t wtwes///ooooottttaaaa
2 "Anohher approach bypasses learning the reward function and allows the human to provide reward signal directly (39 40). This is challenging, however, people do typically provide classical reinforcement 41

---

optimally [2 28 42 but recent pragmatic methods 36 43 44 instead assume the expert is actively teaching.
Learning reward functions from observed actions. When the desired behavior is known (but the reward function is not), inverse reinforcement learning [IRI., 13
42 45 46 can be used to infer an expert's 's reward function from their actions. However, such approaches face fundamental issues with identifiability: observed behavior can often be explained by multiple reward functions.
14
27
28
One solution allows the agent to actively query the human [30 31 47 An alternative is to make
a stronger assumption: that the human is actively teaching [i.e. behaving pedagogically, 36
We next review work on learning rewards from language, then return to these methods.
Learning reward functions from instructions. Instructions use language to communicate specific actions or goals 6 Prior work has used instructions to shape reward functions [15 or learn language-conditioned reward model 12 An alternative approach uses instructions for value alignment. This paradigm assumes instructions reflect underlying preferences and uses them to infer this latent reward function [13 14]. Our work extends this to incorporate other forms of language: we ask when a human would use an instruction (vs. other forms of language), and-given an instruction (vs. other forms of language)- an agent should infer about the speaker's s reward function.
Learning reward functions from descriptions. Rather than expressing specific goals, reward- descriptive language encodes abstract information about preferences or the world. The education literature suggests such rich feedback is crucial for human learning 51] However, a relatively smaller body of work uses it for RL: by learning reward functions directly from existing bodies of text [20 or interactive, free-form language input [24] 25 Our work provides a formal model of such language in order to compare it with more typically studied instructions. Other related lines of work use language which describes agent behaviors. This language, whether externally provided [52]- 59] or internally generated [60-63], typically used to structure the task representation or guide exploration. In contrast, we study language describing task-relevant properties of the environment.
Learning reward functions from pedagogy. The preceding algorithms all assume that training examples (whether demonstrations or language) are generated by a human that is indifferent to the learning process. Recent work has begun to challenge this assumption by considering pedagogical settings 37. In particular, the Rational Speech Act framework [64 (65) builds on classic Gricean theory 666] to formulate optimal communication in terms of recursive Bayesian inference and decision-making. These ideas have been applied to a variety of language tasks including reference games (67 688), captioning [69]. and instruction following (70). Developing analogues of linguistic pragmatics in reinforcement learning - i.e., algorithms that assume data are intentionally designed to be informative
In particular, inverse reward design (IRD, ( uses pragmatic inference on numerical reward functions: rather than take the provided reward function literally, IRD quantifies uncertainty over the function mitigate alignment risk. Similarly. we apply pragmatic inference to reward-relater language. Applying IRD principles to reward-related language offers two primary benefits over its non-linguistic formulation. First, language is accessible for humans. While traditional IRD is useful when offline training RI. agents, inferring rewards from language would be broadly useful for real-world, real-time interactions with non-experts. Second, language can address future settings: speakers can refer to actions or features which are not physically present. Speakers can thus provide information about rare or hazardous possibilities (e.g. poisonous mushrooms) which lie outside the listener's experience. We next describe a model for such language use.
is an active area of research (36]]
38
43
44)
3 Background
In this work, we consider the problem of learning from language in a contextual bandit setting. This section introduces models of language use from cognitive science, then describes contextual bandits.
The Rational Speech Acts framework (RSA). Our theoretical approach is based on the Rational Speech Acts framework (RSA, 64 65 7LH73) ) a computational model of language understanding.
RSA uses theory-of-min mind reasoning to explain how human listeners derive meaning from words. begins by defining literal listener Lo which considers only the conventional (e.g., dictionary) definitions of words. These conventions represented by "lexicon" function L which maps
3

---

from utterances to world states w selection from finite set of utterances, choosing u according some utility function U(.). Finally, a pragmatic listener b,1 infers the intended meaning by inverting the speaker model:
RSA then assumes that a speaker S₁ makes noisy-optimal
PL. u) x C(u, w) P3, (u w) x (xp, U(u,w)) w)} PL, (w u) x Ps, (u w) P(w)
(1)
(2)
(3)
In Eq. Bs, > is an inverse temperature parameter. RSA models typically assume the speaker's objective is informativeness. Formally, the speaker tries to reduce the listener's is information-tteereeie uncertainty (78 over the true state of the world w*: (u, w²) = In Pr. (w" u). Explicitly modeling the speaker as making a rational choice from a set of possible utterances allows the pragmatic listener to enrich the literal meaning of the utterance, inferring additional information about the world.
To take an intuitive example, we consider the classic phenomenon of scalar implicature (9-81) Alice announces "I ate some of the cookies," typical listener Bob understands this "some but not all." RSA explains this as follows. First, the word "some" literally maps to any world state where Alice ate more than zero cookies (e.g. 1, 3... or all), while "all" maps only the world state where she ate all of them (Eq. [. Then, S Alice is presumed to choose a maximally-informall
utterance (i.e. one that specifies the world state as precisely as possible; Eq. 2). Finally, L1 Bob reasons as follows: if Alice had eaten all of the cookies, saying "I ate all of the cookies" would have been more precise and thus preferred. Because she chose not to say this, he can infer that her statement means "some but not all" (Eq. 3). In this work. we define a pragmatic L1 listener which uses RSA to infer the speaker's 's reward function. Crucially, such inferences are based on theory-of-mind and thus vulnerable to misspecification [38] if L1 Bob's model of S₁ Alice is wrong, his inferences will be wrong. We show how a carefully specified speaker model, incorporating both instructions and descriptions, allows our pragmatic listener to robustly infer a speaker's 's reward function.
Contextual bandits. Contextual bandits can be seen as one-step Markov Decision Processes 26 82
thus isolating the problem of learning a reward function from the problem of planning sequential actions maximize it. As such, they are a popular testbed for RL algorithms 83-877 Formally, we
define set of A possible actions. Actions are associated with a binary feature vector ø .A + (0. 1]
(e.g.. a mushroom may be green (or not), or striped (or not)). Following other work in IRL 4 27 42).
we assume rewards are linear function of these features (e.g., green mushrooms tend to be tasty):
R(a, w) w/ ((a)
so w is a vector that defines the value of each feature (see Fig. [A]. Each task consists of a sequence of H i.i.d. states. We refer to H as the horizon. At each time step t < H. the agent is presented with a state & consisting of a subset of possible actions: 8, S A (e.g., a patch containing a set of mushrooms). They choose an action a 8t according to their policy, T : S - A(A).
4 Formalizing learning from language in contextual bandits
While bandits are typically considered as an individual learning problem, we instead ask how an agent should learn socially from a cooperative, knowledgeable partner. We formalize this by introducing a second agent: a speaker who knows the true rewards w and the initial state 80, and produces an utterance u (Fig- [C). This utterance may affect the listener's 's policy; with slight abuse of notation, we denote this updated policy as म T u, 8). The listener then uses this policy to choose actions. The horizon H determines how many actions the listener will perform under this updated policy. Intuitively, H = 1 represents minimal autonomy (or maximal supervision, i.c. guided foraging), whereas H + oa is maximal autonomy (or minimal supervision, i.e. teaching the listener to forage independently). We first assume H is known to both listener and speaker, then relax this assumption. This framework exposes two interrelated problems. First, what should helpful speaker say? And second, how should the listener update their policy in light of this information?
'The lexicon can be seen as the grounding of language [5] Most RSA models, including this work, assume that groundings are mutually known. However, methods have been developed to learn them [24 (5) or allow for uncertainty [76] 77), and view integrating the problem of grounding as important future work.

---

A
norizon
Thus, horizon = 5
B
1.75
Instructions
O
3
0
Speaker Rewards
Descriptions
1.50
Utterance Probability
1.00
instruction Red Spotted
Description Spotted
1.25
0.75
Description - -Blue
Description
1.00
0.50
0.25
0.75
0.00
0.50
7
10
1
2
3
5 7 9
10
Speaker Horizon
Speaker Horizon H
Figure 2: The horizon formalizes the agent's degree of autonomy and strongly affects reward-maximizing horizon lengthens, the agent will act independently in more unknown future states. Bottom: simulating reward-maximizing speaker. At short horizons, the speaker is focused on the spotted red mushroom. As the horizon lengthens, they are less biased towards this particular state and provide averaged over start states) for a speaker with access only instructions only descriptions. Instructions outperform descriptions at short horizons, but descriptions offer much stronger generalization.
speaker's choice utterance. A: Top:
generally useful information about the reward function. B: Speaker rewards (Eq. 7
4.1 Speaker model
We apply the RSA framework (Section 3) RL setting by introducing new speaker objective: instead of Gricean informativeness 665 66 we assume that speakers seek to maximize the listener's rewards [88] Conceptually, this changes RSA's scope from epistemic to decision-theoretic.. Instead of reasoning about how utterances affect the listener's beliefs, such a speaker reasons about how they affect the listener's 's policy. When the state known, we define the present utility of an utterance as the expected reward from using the updated policy to choose action:
Ureesent (u 8, w)
Σ ( u, R),, w)
However, we are interested in settings where agents may need to act autonomously 13 24 25 must consider how well the policy generalizes unknown, future states. The future utility of an utterance with respect to distribution over states P(s) can be written as:
U (u w) (Presentu P(s)
SES
Because states are i.i.d. in the bandit setting, a speaker optimizing for a horizon H can be defined as linear combination of Eqs. 3 [
Us. (u w. s, H) UPresent + (H - 1)UUurr
1 Upresent (1 1
U
(7)
H
H
H
where H reduces to Eq. 5 and as H - oo reduces to Eq. At short horizons, speakers biased towards producing utterances that optimize the listener's s policy in the present state. As the horizon lengthens, the speaker expects the listener to behave more autonomously. Generalization becomes more important, and this bias is reduced.
4.2 Formalizing instructions
We now consider how utterances affect the listener's policy. Instructions map to specific actions or trajectories 13; in our work, "instruction" utterances correspond to the nine actions (Fig (B). Given an instruction, literal listener executes the corresponding action. If the action is not available, the listener acts randomly:
if $
L (a Uinstruction s)
(
if [u] E otherwise
(8)
where 6lwI(a) represents the meaning of u, evaluating one when utterance u grounds to a and zero otherwise. An instruction is partial policy: designates action to take in subset of states.
5

---

4.3 Formalizing descriptions
Rather than mapping to a specific action, descriptions provide information about the reward func- tion (24) [5 46]. Following [88], we model descriptions as providing the reward of a single feature, similar to feature queries [I] Descriptions are thus tuple: one-hot binary feature vector and a scalar value, (1 K, R). These are messages like (Blue, -2). In this work, we consider the set of features x 5 values in [-2, -1,0, 1, 2]. yielding 30 descriptive utterances. Formally, Lo "rules out" inconsistent hypotheses about reward weights w:
Lo(w Udeecription) xx )(w](w) P(w)
where ((w](ww) represents the meaning of u, evaluating to one when u is true of w and zero otherwise. In this work, we assume P(w) is uniform and there is no correlation between weights. The listener then marginalizes over possible reward functions to choose an action:
Lo (a
Udeeeriptionn 8) x exp Σ R(a, w) Lo(w x))}
(10)
where Br. is again an inverse temperature parameter
4.4 Comparing instructions and descriptions
Prior work suggests that humans use mix of instructions and descriptions 24 25 What modulates
this-wher should a rational speaker prefer instructions over descriptions? To explore the effects of horizon on utterance utility, bandit setting. We assume the listener begins with a uniform prior over reward weights throughout, and use states consisting of three unique actions (giving 84 possible states).
simulate a nearly- speaker (Bs, 10). Fig. IA shows
F shows how the speaker's 's choice of utterance varies as the horizon lengthens. At short horizons, the speaker optimizes for present rewards (Eq. 5) and chooses instructions and descriptions that target the "Spotted Red" action. At longer horizons, future rewards (Eq. () play a larger role, and the speaker blends the two objectives (Eq. 7 by describing the highly negative blue feature. Finally, at sufficiently long horizons, future rewards dominate and it settles on describing the green Teature-which is irrelevant to the start state, but the most important feature for generalization. To quantify how rational speakers should use instructions and descriptions, we repeat the task for all 84 start states using horizons ranging 1-10 and different utterance sets. Fig plots rewards for speakers with access to only instructions or descriptions, illustrating why this shift from instructions to descriptions occurs. Instructions outperform descriptions short horizons (achieving the theoretical maximum average reward of 1.75); as the horizon lengthens, however, descriptions generalize better Overall, as the horizon lengthens, speakers with access to both instructions and descriptions choose descriptions exclusively, producing highly generalizable information (see Appendix A).
4.5 Learning from utterances: a pragmatic listener
We now ask how the listener should learn from the speaker's 's utterance. Following RSA (Section 3 a pragmatic listener La inverts the speaker model to infer their latent reward function. This approach is closely related to inverse reward design [4: because the speaker maximizes rewards (Eq. 7 our pragmatic listener effectively treats utterances as proxy rewards [89 specified in language.
Known horizon. If the speaker's horizon H is known, we can write Lı listener as:
[ww 8, u, H) x (z w,, 8, P(w)
(11)
Given an instruction, L, infers the reward weights that would make such an instruction optimal [33 [4]: given description, L1 can recover information about features that were not mentioned [24] 25 The L, listener then chooses actions by substituting this posterior belief into Eq. 10 In practice,
'Due to the recursive structure of speaker and listener, simultaneously varying 8s, and 3r., introduces identifiability issues. We therefore fix Ir, = 3 throughout this work, and only vary As,- This implies that all speakers address the same listener, but do with different degrees of optimality.
Beeausee our states consist of only three actions, it is often possible to find a deseription that uniquely identifies the best action. This allows descriptions to perform nearly well as instructions at H = 1. However, the number of available actions increases, instructions become increasingly advantageous; Appendix A.

---

Take spotted rod ones
Mcan Rewards
Probabiiity
0.4
0.2
D
3
0.6
Mean Rewards
Spots are +1
Probability
0.4
O
Speaker Horizon H
Figure 3: Posteriors from latent-horizon pragmatic inference (F4. 12 for instructions and descriptions. Left: both utterances suggest the listener should take the spotted red mushroom in this state. Center: instruction (lop) suggests the speaker is short-horizon, while description (bottom) suggests they long-horizon. Right: this leads to different inference about the speaker's 's reward function (see Appendix for full posteriors).
however, the horizon not known, and so this approach not feasible for real- col-w applications. This model instead serves mheoretically-optinee baseline, and we next consider three practical ways of handling this uncertainty.
Assuming the horizon. Perhaps the most straightforward approach is to simply assume a speaker horizon. However, prior work has highlighted the risks of assuming human is actively trying to teach [38]. and suggests that the safest approach is to assume they are not. To test the effects of such specfiipcii,,,,nnn, in Section we define two pragmatic listener models which assume short (H 1) long (/l horizon. A pragmatic listener assuming И 1 will constrain inference: will assume the utterance reflects only the speaker's preference between actions in the present state, and thus generalize conservatively. In contrast, pragmatic listener assuming H > expects the utterance to generalize broadly, risking overfitting.
Inferring the horizon. To mitigate the risk of horizon mis-specification, we can instead assume the speaker's horizon unknown. Given an utterance, the latent horizon pragmatic listener jointly infers both their horizon and rewards, then marginalizes out the horizon:
Luw u) >S Si(u 8, H))((H)(ww)
(12)
Intuitively, because short-horizon speakers prefer instructions (and long-horizon speakers prefer descriptions, Fig. A), the aitent-horizon pragmatic listener speaker's 's horizon and determine the appropriate scope of generalization. To demonstrate this, we simulate pragmatic listener with uniform prior over H [1, 2, 4, 5, 10). Fig. shows example utterances and resulting inference about the speaker's rewards and latent horizon. Crucially, while both example utterances indicate preference for the spotted red mushroom, the description suggests the speaker is H > 1 and uniquely identifies the spotted feature as high-value.
the utterance type to infer the
Behavioral experiment
To validate our theoretical models, we collected behavioral dataset. Participants played the role of mushroom foraging guide and produced utterances for tourists to help them choose good mushrooms. We manipulated the speaker's horizon by varying tourists' itineraries: each tourist was shown visiting different visible mushroom patch, plus variable number of unknown future patches (0, 1, or 3, matching horizons 1, 2, and 4 respectively). In the following sections, we analyze the participants' choice of utterances, compare the resulting inference from different listener models, and show how this socially-learned information can accelerate traditional reinforcement learning
'This study was approved by the Princeton IRB. The full experiment can be viewed at https:// pragmatic-bandiee www.kkepp.....o
7

---

A
B
Empirical Results
Model Predictions
0.75
Percent Utterances
Future Rewards
0.50
0.75
0.25
Literal Listener
Pragmatic Known
Literal Listener
Pragmatic
Descriptions Instructions
Pragmatic- - H-1
Pragmatic
0.50
Pragmatic Hm4
Pragmatic- - Ho44
Pragmatic - Latent H
Pragmatic Latent H
0.00
2
i
4
Speaker Horizon H
Speaker Horizon H
Speaker Horizon
Figure 4: Results from behavioral experiment. Participants preferred instructions when H/ = and descriptions otherwise. B, left: empirical future rewards (Eq. 6 from human utterances. As the horizon lengthens, humans choose more generalizable utterances; our latent-horizon listener successfully recovers their reward function. Right: simulations using speaker model with Bs, = 3. All error bars show 95% CL.
5.1 Experiment setup
We recruited 119 participants the Prolific experimental platform (prolific.co) Participants were trained and tested on the game dynamics then advised total of 28 tourists. They were told to consider the tourists' itinerary and choose "the most helpful utterance" from drop-down menus allowing them to specify an instruction or description. Because the space of possible descriptions (30) larger than possible instructions (9), choosing description required more effort than choosing an instruction. To equalize this, we reduced the set of descriptions by removing neutral features ("red" and "solid") and the O value, yielding 4 x 4= 16 possible descriptive utterances. For the remainder of the paper, all pragmatic listeners assume the speaker chooses from this reduced utterance set, Participants received no feedback on how their utterances affected tourists' behaviors, ensuring they chose utterances according to their own sense of how to help. After screening out participants who failed comprehension or attention checks, we were left with 99 participants who produced a total of 2772 utterances. For more details on the data collection, see Appendix B.1.
We next this of utterances explore value alignment (whether our models can infer the speakers' reward function) and then integrated social and reinforcement learning (whether this socially-acquired information can reduce regret in traditional RL).
5.2 Inferring rewards: value alignment from language
Overall, our results validated our theoretical models. First, participants were sensitive to the horizon manipulation: there was
-iiigiiially shift in their utterance choices between horizons (for H = to 2, (26, 1 1856) = 336.1, p < .001; for H = 2 to 4, x x²6,,, 1833) 40.3, 3. .04). Almost all participants (96 out of 99) used mix of instructions and descriptions, favoring instructions at H and descriptions at H > 1 (Fig. A This led to lower literal future rewards when H 1, and higher future rewards at longer horizons ( (Fig 4 33, "Empirical Results"). To calibrate our pragmatic listeners, we tested Bs; E 10) and found that 3s, 3 optimized Known H and Latent H listeners (see Appendix B.3 for details). Simulating utterances produced by our speaker model and resulting pragmatic inference shows a close match to theoretical predictions (Fig. 4B, "Model Predictions" see Appendix B. for details).
Our pragmatic listeners offered muuisticaall-siinnficee improvements over the literal listener (Table Tp These gains were particularly large when the speaker has short horizon, matching our simulations. Somewhat surprisingly, our Latent H model (M = 0.94, SD = 0.39) outperformed all other models, including the Known H model (M = .93, SD = .40), by a significant albeit small margin (mean difference 0.01, paired-samples t-test ((2771) -4.18,p .001; see Appendix C. for other pairwise tests). This is particularly notable because underscores the inevitability of misspecification [3 even when experimentally controlled the horizon, participants did not perfectly follow our theory,

---

1.75
1.50
Pragmatic Latent H
Listener
Future rewards from utterance
Regret after
Literal Listener
25 actions
1.25
Regret
Individual Learner
Individual
12.14 14
1.00
Literal
.79
10.23
0.75
Prag Known H
.93
9.42
0.50
Prag H
.83
9.76
0.25
Prag H =
.91
9.67
0.00
Prag Latent H
.94***
9.55
10
20
Individual Learning Actions
Table 1: Results from human utterances. Higher rewards
Figure 5: Regret with and without social informa- tion from our behavioral experiment.
(Section 5.2 Eq. c and lower regret (Section 5.3) are better. Bolded results are significantly better.
leading our Latent H. model to outperform. We also confirmed the risks of assuming a horizon: the two fixed-horizon listeners (H- = 1 and H = 4) underperformed the Latent H listener.
Despite these successes, found notable discrepancy between theoretical predictions and empirical results, suggesting a possible future refinement. As described in [88, maxiiizig speakers agnostic the truth of an utterance may send messages with exaggerated values (e.g- preferring the false utterance (Spotted, +2) to the true utterance (Spotted, +1)). In our experiment, however, participants regularly chose the lower-reward, true utterance (Appendix B). Breaking out results by utterance type indicates that pragmatic gains come primarily on instructions (Appendix D). This suggests that our reward-design objective is in fact weak: participants tendency use true descriptions licenses stronger inference, which we return to in the discussion.
5.3 Reducing regret: integrating social and reinforcement learning
We explore how this form of social learning could augment traditional RL approaches. In particular, we take the inferences from our listeners the previous section and use them as a prior for reinforcement learning (or in the case of literal instructions, we integrate them into the learner's policy; see Appendix E for details on the experimental setup). We then use Thompson sampling [90] 92] to learn the reward function and compare regret the course of learning for five independent learning trials on each context-utterance pair. We report results for our listener agents, as well as an agent with social information (the "Individual" agent; Fig. 5 Table I
We find that social information can substantially reduce regret. Intriguingly, when comparing regret, Known H pragmatic listener now achieves the best results; this is likely due to the greater uncertainty inherent in the Latent inference. To confirm these differences are statistically significant, we use linear regression predict regret with fixed effect of listener and random effects for each of the 2772 utterance-context pairs, again comparing the Latent H listener to all other listeners. This confirms that the Known H listener achieves lower regret than the Latent H listener (3 - 12,, ((80400) = -2.14, = .03), and all other models suffer higher regret (see Appendix C).
Discussion
We formalized the challenge of using language to teach an agent to act on our behalf [3 14
84 in bandit setting, introducing the notion of a horizon to reflect the agent's degree of autonomy. We find instructions are optimal for www-autonmmy settings, while descriptions are optimal for high autonomy. This distinction allows pragmatic listener to jointly infer the speaker's 's horizon and reward function, reducing misspecification risk [38]. Our behavioral experiment supports our theory, demonstrating the benefit of learning from language for value alignment [+-3] and traditional RL [26).
We note several limitations and future directions. First, human participants preferred truthyful de- scriptions; this licenses stronger pragmatics [93] such as combining our model with epistemic objectives (64 65). Second, we assumed language groundings were known, but future work could incorporate uncertainty 76) [77 or learn them 24 25 75 Third, we studied a simple contextual ban- dit setting; sequential decision settings will require models of planning, and may consider additional language such as instructions at different levels of abstraction transition descriptions (94). Lastly,

---

we modeled a simple interaction consisting of single utterance. More naturalistic interactions would allow multiple utterances, bidirectional dialogue, interleaved action and communication 43
Our work can inform agent design by clarifying how and when different forms of linguistic input can be useful. Instruction following may be optimal when autonomy is unnecessary or preferences are non-Markovian [95]. However, if autonomy is desired [133] I3 14 38 43 44 agents should
be equipped to understand descriptions of the world or our preferences 22H255 Finally, pragmatic agents can infer whether preferences are local or general. This suggests that learning from wide range of language a promising approach for both value alignment and RL more broadly.
Acknowledgments and Disclosure of Funding
We thank Rachit Dubey, Karthik Narasimhan, and Carlos Correa for helpful discussions. TRS is supported by the NDSEG Fellowship Program and RDH is supported by the NSF (grant #191 #83).. This work was additionally supported by a John Templeton Foundation grant to TLG (#61454) and grant from the Hirji Wigglesworth Family Foundation to DHM.
References
[1] lason Gabriel. Artificial intelligence, values, and alignment. Minds and Machines, 30(3): 411-437, September 2020.
[2] Stuart Russell. Human Compatible: Artificial Intelligence and the Problem of Control. Viking, USA, 2019.
[3] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in Al safety. arXiv preprint arXiv:/606.0663 2016.
[4] Dylan MednelldMeneel,, Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca Dragan. Inverse reward design. In Advances in Neural Information Processing Systems, 2017.
[5] W Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (mis) design for autonomous driving. arXiv preprint arXiv: 1..4.339000 /3906, 2021.
[6] Terry Winograd. Understanding natural language. Cognitive Psychology, 3(1):1-191, 1972.
[7] Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R. Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In AAAI Conference on Artificial Intelligence, 2011.
[8] David Chen and Raymond Mooney. Learning to interpret natural language navigation instruc- tions from observations. In AAAI Conference on Artificial Intelligence, 2011.
[9] James MacGlashan, Monica -roman, Marie desJardins, Michael I. Littman, Smaranda Muresan, Shawn Squire, Stefanie Tellex. Dilip Arumugam, and Lei Yang. Grounding English commands reward functions. In Robotics: Science and Systems, 2015.
[10] Justin Fu, Anoop Korattikara, Sergey Levine, and Sergio Guadarrama. From language to goals: Inverse reinforcement learning for vision-based instruction following, In International Conference on Learning Representations, 2018
[11] Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Seyed Arian Hosseini, Pushmeet Kohli, and Edward Grefenstette. Learning to understand goal specifications by modelling reward. In International Conference on Learning Representations, 2019.
[12] Pratyusha Sharma, Balakumar Sundaralingam, Valts Blukis, Chris Paxton, Tucker Hermans, Antonio Torralba, Jacob Andreas, and Dieter Fox. Correcting robot plans with natural language feedback. arXiv preprint arXiv:2204.00 086, 2022.
[13) Hong Jun Jeon, Smitha Milli, and Anca Dragan. Reward-rational (implicit) choice: unifying formalism for reward learning. Advances in Neural Information Processing Systems, 2020.
[14] Smitha Milli, Dylan Hadfield-Menell, Anca Dragan, and Stuart Russell. Should robots be obedient? International Joint Conference on Artificial Intelligence, 2017.
[15] Richard Maclin and Jude Shavlik. Incorporating advice into agents that learn from reinforce- ments. In AAAI Conference Artificial Intelligence, 1994.
10

---

[16] Gregory Kuhlmann, Peter Stone, Raymond Mooney, and Jude Shavlik. Guiding reinforcement learner with natural language advice: Initial results in RoboCup soccer. AAAI Workshop Technical Report, 06 2004.
[17] Prasoon Goyal, Scott Nickum, and Raymond Mooney. Using natural language for reward shaping in reinforcement learning. In International Joint Conference on Artificial Intelligence. 2019.
[18] J Luketina, N Nardelli, G Farquhar, Foerster, J Andreas, Grefenstette, S Whiteson, and T Rocktäschel. A survey of reinforcement learning informed by natural language. In Interna- tional Joint Conference on Artifiical Intelligence, 2019.
[19] Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazit, and Cynthia Matuszek. Robots that use language. Annual Review of Control, Robotics, and Autonomous Systems, 3(1):25-55, 2020.
[20] SRK Branavan, David Silver, and Regina Barzilay. Learning to win by reading manuals in Monte-Carlo framework. Journal of Artificial Intelligence Research, 43:661-704, 2012.
[21] Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. Grounding language for transfer in deep reinforcement learning. Journal of Artificial Intelligence Research, 63:849-877 8744, 2018.
[22] Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. RTFM: Generalising to novel environment dynamies via reading. arXiv preprint arXiv: 2019.
[23] Austin W Hanjie, Victor Y Zhong, and Karthik Narasimhan. Grounding language to entities and dynamics for generalization in reinforcement learning. In International Conference on Machine Learning, pages 4051-4062. PMLR, 2021.
[24] Theodore Sumers, Mark K Ho, Robert D Hawkins, Karthik Narasimhan, and Thomas L Grif- fiths. Learning rewards from linguistic feedback. In AAAI Conference Artificial Intelligence, 2021.
[25] Jessy Lin, Daniel Fried, Dan Klein, and Anca Dragan. Inferring rewards from language in context. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, 2022.
[26] Richard S Sutton and Andrew G Barto. Reinforcement Learning: An Introduction. MIT press, 2018.
[27] Andrew Y Ng and Stuart J Russell. Algorithms for inverse reinforcement learning. In Interna- tional Conference on Machine Learning, 2000.
[28] Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning, 2004.
[29] Douglas Markant and Todd M Gureckis. Is it better to select or to receive? Learning via active and passive hypothesis testing. Journal of Experimental Psychology: General, 143(1):94, 2014.
[30] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems, 2017.
[31] Chandrayee Basu, Mukesh Singhal, and Anca D Dragan. Learning from richer human guidance: Augmenting comparison- based learning with feature queries. In 13th ACM/IEEE International Conference Human-Robot Interaction (HRI). IEEE, 2018.
[32] Natalia Vélez and Hyowon Gweon. Learning from other minds: An optimistic critique of reinforcement learning models of social learning. Current Opinion in Behavioral Sciences, 38: 1 110-115, 2021.
[33] Hyowon Gweon. Inferential social learning: Cognitive foundations of human social learning and teaching. Trends in Cognitive Sciences, 25(10):896991 2021.
[34] Pei Wang. Junqi Wang, Pushpi Paranamana, and Patrick Shafto. A mathematical theory of cooperative communication. Advances in Neural Information Processing Systems, 2020.
[35] Patrick Shafto, Noah D. Goodman, and Thomas I.. Griffiths. A rational account of pedagogical reasoning: Teaching by, and learning from, examples. Cognitive Psychology, 71:55 89, 2014.
[36] Mark K Ho, Michael I Littman, James MacGlashan, Fiery Cushman, and Joseph I. Austerweil. Showing versus Doing: Teaching by Demonstration. In Advances in Neural Information Processing Systems, 2016.
11

---

[37] Mark K. Ho, Fiery Cushman, Michael L. Littman, and Joseph L. Austerweil. Communication in action: Planning and interpreting communicative demonstrations. Journal of Experimental November 2021.
Psychology: General, 1 150(11) (:2211):2622220
[38] Smitha Milli and Anca D Dragan. Literal or pedagogic human? Analyzing human model misspecification in objective learning. In Uncertainty in Artificial Intelligence, 2020.
[39] W. Bradley Knox and Peter Stone. Interactively shaping agents via human reinforcement: The TAMER framework. In Proceedings of the Fifth International Conference on Knowledge Capture, page 9-16, New York, NY, USA, 2009. Association for Computing Machinery.
[40] James MacGlashan, Mark Ho, Robert Loftin, Bei Peng, Guan Wang, David L. Roberts, Matthew E. Taylor, and Michael L. Littman. Interactive learning from policy-dependent human feedback. In International Conference on Machine Learning. 2017.
[41] Mark K. Ho, Fiery Cushman, Michael L. Littman, and Joseph L. Austerweil. People teach with rewards and punishments as communication, not reinforcements. Journal of Experimental Psychology: General, 148(3):520-549, March 2019.
[42] Brian D. Ziebart, Andrew Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In AAAI Conference on Artificial Intelligence, 2008.
[43] Dylan Madfield-Menell Stuart Russell, Pieter Abbeel, and Anca Dragan. Cooperative inverse reinforcement learning. In Advances in Neural Information Processing Systems 29, 2016.
[44] Jaime F Fisac, Monica A Gates, Jessica Hamrick, Chang Liu, Dylan Medeill Malayandi Palaniappan, Dhruv Malik, S Shankar Sastry, Thomas L Griffiths, and Anca D Dragan. Pragmatic-pedaaggi value alignment. In Robotics Research, pages 49 47- Springer, 2020.
[45] Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In Interna- tional Joint Conference on Artificial Intelligence, 2007.
[46] Silvia Tulli, Francisco S. Melo, Ana Paiva, and Mohamed Chetouani. Learning from explana- tions with maximum likelihood inverse reinforcement learning. March 2022.
[47] Christian Daniel, Oliver Kroemer, Malte Viering, Jan Metz, and Jan Peters. Active reward learning with a novel acquisition function. Auton. Robots, 3963)3899445, October 2015.
[48] Dorsa Sadigh, Anea D Dragan, Shankar Sastry, and Sanjit Seshia. Active preference-based learning of reward functions. In Robotics: Science and Systems, 2017.
[49] Malayandi Palan, Gleb Shevchuk, Nicholas Charles Landolfi, and Dorsa Sadigh. Learning reward functions by integrating human demonstrations and preferences. In Robotics: Science and Systems, 2019.
[50] Valerie J. Shute. Focus on formative feedback. Review of Educational Research, 78(1):153-189 2008.
[51] A.A. Lipnevich and J.K. Smith. Effects of differential feedback students' examination performance. Journal of Experimental Psychology: Applied, 15(4):319-333, 2009.
[52] John Co-Reyes, Abhishek Gupta, Suvansh Sanjeev, Nick Altieri, John DeNero, Pieter Abbeel, and Sergey Levine. Meta-learning language-guider policy learning. In International Conference on Learning Representations, 2019.
[53] Pratyusha Sharma, Antonio Torralba, and Jacob Andreas. Skill induction and planning with latent language. arXiv preprint mrXiv:22/100//3 2021.
[54] Catherine Wong, Kevin M Ellis, Joshua Tenenbaum, and Jacob Andreas. Leveraging language to learn program abstractions and search heuristics. In International Conference on Machine Learning. 2021.
[55] Jacob Andreas, Dan Klein, and Sergey Levine. Learning with latent language. In NAACL, pages 2166-2179, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
[56] Valerie Chen, Abhinav Gupta, and Kenneth Marino. Ask your humans: Using human instruc- tions to improve generalization in reinforcement learning. In International Conference on Learning Representations, 2021.
[57] Khanh X Nguyen, Dipendra Misra, Robert Schapire, Miroslav Dudík, and Patrick Shafto. Interactive learning from activity description. International Conference on Machine Learning, 2021.
12

---

[58] Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. Language as an abstraction for hierarchical deep reinforcement learning. Advances in Neural Information Processing Systems, 2019.
[59] Allen Z. Ren, Bharat Govil, Tsung-Yen Yen Yang. Karthik Narasimhan, and Anirudha Majum- dar. Leveraging language for accelerated learning of tool manipulation. arXiv preprint arXiv:2206. 13074, 2022.
[60] Cédric Colas, Tristan Karch, Nicolas Lair, Michelichel Dussoux, Clément Moulin-Frier, Peter Dominey, and Pierre-Yvel Oudeyer. Language cognitive tool imagine goals in curiosity driven exploration. Advances in Neural Information Processing Systems, 2020.
[61] Geoffrey Cideron, Mathieu Seurin, Florian Strub, and Olivier Pictquin. Higher: Improving instruction following with hindsight generation for experience replay. In 2020 IEEE Symposium
Series on Computational Intelligence (SSCI), pages 225-232. IEEE, 2020.
[62] Jesse Mu, Victor Zhong, Roberta Raileanu, Minqi Jiang, Noah Goodman, Tim Rocktäschel, and Edward Grefenstette. Improving intrinsic exploration with language abstractions. arXiv preprint arXiv: 08938, 2022.
[63] Allison C Tam, Neil C Rabinowitz, Andrew K Lampinen, Nicholas A Roy, Stephanie CY Chan, DJ Strouse, Jane X Wang, Andrea Banino, and Felix Hill. Semantic exploration from language abstractions and pretrained representations. arXiv preprint arXiv:2204.0505 0000000000000.... 2022.
[64] Michael C Frank and Noah D Goodman. Predicting pragmatic reasoning in language games. Science, 336(6084):998-999) 2012.
[65] Noah D. Goodman and Michael C. Frank. Pragmatic language interpretation as probabilistic inference. Trends in Cognitive Sciences, 20(11):818 829, 2016.
[66] H. P. Grice. Logic and conversation. In Syntax and Semantics: Vol. 3: Speech Acts, pages
41 11-58. Academic Press, New York, 1975.
[67] Will Monroe, Robert XD Hawkins, Noah Goodman, and Christopher Potts. Colors in context:
A pragmatic neural model for grounded language understanding. Transactions the Association for Computational Linguistics, 5:325-338 2017.
[68] Jacob Andreas and Dan Klein. Reasoning about pragmatics with neural listeners and speakers. In Empirical Methods in Natural Language Processing, 2016.
[69] Allen Nie, Reuben Gorno,, and Christopher Potts. Pragmatic sesuessensitive image captioning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1934-1938, 2020.
[70] Daniel Fried, Jacob Andreas, and Dan Klein. Unified pragmatic models for generating and following instructions. In NAACL, June 2018.
[71] Erica J Yoon. Michael Henry Tessler, Noah D Goodman, and Michael C Frank. Polite speech emerges from competing social goals. Open Mind, 4:71-87, 2020.
[72] Justine T. Kao, Jean Y. Wu, Leon Bergen, and Noah D. Goodman. Nonliteral understanding of number words. Proceedings of the National Academy of Sciences, 12002-312002-2007 2014.
[73] Justine Kao, Leon Bergen, and Noah Goodman. Formalizing the pragmatics of metaphor under- standing. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 36, 2014.
[74] Stevan Harnad. The symbol grounding problem. Physica D: Nonlinear Phenomena, 42(1-3): 335-346, 1990.
[75] Raymond J Mooney. Learning to connect language and perception. In AAAI Conference on Artificial Intelligence, 2008
[76] Judith Degen, Robert D Hawkins, Caroline Graf, Elisa Kreiss, and Noah D Goodman. When redundancy is useful: A Bayesian approach to "overinformative" referring expressions. Psycho- logical Review, 127(4):591, 2020.
[77] Robert D Hawkins, Michael Franke, Michael C Frank, Adele E Goldberg, Kenny Smith, Thomas L. Griffiths, and Noah D Goodman. From partners to populations: A hierarchical Bayesian account of coordination and convention. Psychological Review, 2022.
[78] Thomas M Cover. Elements of Information Theory. John Wiley & Sons, 1999.
13

---

[79]
Stephen C Levinson.
Pragmatics. Cambridge university press, 1983.
[80] Stephen Levinson. Presumplive meanings: The theory of generalized conversational implica- fure. MIT press, 2000.
[81] Noah D Goodman and Andreas Stuhlmüller. Knowledge and implicature: Modeling language
understanding as social cognition. Topics in Cognitive Science, 5(1):173-184, 2013.
[82] Martin L Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, Inc., 1994.
[83] Tor Lattimore and Csaba Szepesvari. Bandit Algorithms. Cambridge University Press, 2020.
[84]
Kareem Amin, Nan Jiang, and Satinder Singh. Repeated inverse reinforcement learning. Advances in Neural Information Processing Systems, 2017.
[85] Carlos Riquelme, George Tucker, and Jasper Snoek. Deep Bayesian bandits showdown: An empirical comparison of Bayesian deep networks for Thompson sampling. In International Conference on Learning Representations, 2018.
[86] Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In International Conference Artificial Intelligence and Statistics, volume 15, 2011.
[87] Yasin Yadkor,, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. In Advances in Neural Information Processing Systems, volume 24, 2011. 1.
[88] Theodore Sumers, Robert Hawkins, Mark Ho, and Thomas Griffiths. Extending rational models of communication from beliefs to actions. In Proceedings of the Annual Conference of the Cognitive Science Society, 2021.
[89] Satinder Singh, Richard L Lewis, and Andrew G Barto. Where do rewards come from? In Proceedingss of the 3 st Annual Conference of the Cognitive Science Society, 2009.
[90] William Thompson. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika, 25(3-4):288 294, 1933.
[91] Daniel Russo and Benjamin Van Roy. Learning to optimize via posterior sampling. Mathematics of Operations Research, 39(4): 1 1211- 2014.
[92] Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. A tutorial on Thompson sampling. Foundations and Trends in Machine Learning, 11(1):1-96, 2018.
[93] Rohin Shah, Noah Gundotra, Pieter Abbeel, and Anca Dragan. On the feasibility of learning, rather than assuming, human biases for reward inference. In International Conference Machine Learning, volume 97, 2019.
[94] Anna N. Rafferty, Emma Brunskill, T. Griffiths, and Patrick Shafto. Faster teaching by POMDP planning. In AIED, 2011.
[95] David Abel, Will Dabney, Anna Harutyunyan, Mark K Ho, Michael Littman, Doina Precup, and Satinder Singh. On the expressivity of Markov reward, Advances in Neural Information Processing Systems, 34:7799-7812, 2021.
14


Reviews:
[{"review_id": "gK4QsvK8pO8", "reviewer": "Reviewer_aJ8p", "summary": "This paper explores the idea of learning-from-teaching in an RL setting, where a teacher can provide either direct instructions or descriptions of a reward function.  The paper focuses on linear bandits, and introduces a sort of theory-of-mind framework whereby a learner can reason about a teacher's mental state. The paper validates their theoretical models with a human experiment, and show close agreement between human performance and the model's predictions.", "questions": "None", "limitations": "The authors do a good job of addressing limitations and ethical concerns.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I generally liked this paper. The framework is clear, subtle, interesting and potentially useful.\n\n+ I liked the human experiments, and the validation of the model's predictions\n+ The mathematical formulation of the model and the setting was clear\n+ I appreciated the authors' attention to detail, and the careful buildup of ideas\n+ The use of various statistical tests to support conclusions was tidy and well done\n+ I especially enjoyed the bits where the agent reasoned explicitly about the speaker's unknown H. I think this sort of explicit reasoning about mental states needs to be more common in RL.\n+ Section 4.3 was intriguing, I was glad to see it included\n\n- The paper feels a bit contrived. I found it unsurprising that instructions were better for small H, and descriptions for long H; it almost feels like the problem setting was conceived to support this eventual conclusion. And so while I generally believe the claims in the paper, I have a hard time believing that this will scale much beyond this toy setting.  It always rubs me a /bit/ the wrong way when the story is much bigger than the result.\n- Upon reflection, it was unclear to me what generalizable knowledge I should take with me after reading this paper, because if we eliminate the framing, the conclusions seem too closely tied to the specifics of the experimental setup.  Does this really help me in the non-bandit setting? Does this really help me build better teachers? Does this really say anything about building better agents? Does this really tell me when to prefer instructions vs. descriptions?  I appreciate that scientific knowledge must be built brick by brick, and so I am not unhappy with the contribution as it stands, but I do wish that that the ideas felt more generalizable.  (That said, the strengths of this paper outweigh its weaknesses, so I will argue for acceptance.)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "avg4jtWZwoW", "reviewer": "Reviewer_vzCm", "summary": "This paper proposes and validates a qualitative distinction between instructions and descriptions as ways of communicating partial information about a reward function.  Briefly, instructions are shown to be more useful over short horizons.  This is framed as a social learning problem, with a pragmatic listener; this resembles previous work on Inverse Reward Design.\n\nThis distinction is supported theoretically, and with synthetic and human experiments in a toy linear (contextual) bandit setting (see Figure 1).  The human experiments show strong but not perfect agreement with the theory, and there is a discussion of these results including this discrepancy, which is attributed to humans being more likely to convey literally true information as opposed to the information that would be most useful from a pragmatic point of view.\n", "questions": "Question: Why is “fixed effect of listener and random effects for each of the 2772 utterance-context pairs” used on line 295, but not (If I understand correctly) lines 266-268?\n\n\nSuggestions:\n* Line 62 needs references.\n* In line 72, [9-14] should be placed after “infer”\n* It should be clarified that this is a *contextual* bandit setting.\n* There are typos on 164, 223, 262.\n* In Table 1, It is not clear enough what the “future rewards from utterance” column refers to.  \n* There should be more detail on the selection of $\\beta_{S_1}$, to assure the reader that it is not (in some sense) cherry-picked.\n", "limitations": "Yes I believe they have.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The proposed distinction is insightful and useful, and well described and supported.\n* The experiments and analysis are sensible and support the central point of the paper.\n* Human experiments are a great addition.\n* The experiments on inferring the horizon vs. assuming knowledge of the horizon were interesting and provide an important counter-point to Milli and Dragan [41].  \n\n\n\nWeaknesses:\n* The experimental setting is extremely simple and the work does not demonstrate any immediate practical utility.  A particular limitation with the author's mention is the need to ground natural language instructions and descriptions, which is not addressed in this work.  Overall, however, I think the work still does make a compelling case for further investigation, and the experiments contribute substantially to that case.\n* I found the discussion of prior work on instructions misleading, and I think it needs to be more nuanced.  Instructions play a different role in this work versus, e.g., Badhanau et al. [11].  In that work (and other works that I have seen on instruction following), the instructions are meant to be a complete specification of the task/reward, and the method is meant to train a single agent which can be instructed to do different tasks.  In this work, the instruction is meant to be a partial specification of a single task, and the purpose of the method is for the learner/listener to make the best inference possible about the reward from that instruction.  \n* I found the analysis lines 155-170 of Section 3.4 weak and unconvincing.  This establishes an equivalence between reward design, demonstrations, and instructions, but under fairly strong conditions.  These conditions should be discussed further, As it is unclear how broadly they apply and what happens when they do not apply.  It is also unclear how this analysis is meant to contribute to the central point of the paper.\n* The approach is characterized as inverse reward design (IRD), but IRD is a more specific (albeit closely related) problem.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JwcxH4inF98", "reviewer": "Reviewer_feLE", "summary": "The goal of this paper is considering a social learning setup to see how humans might communicate preferences over behaviours. The motivation for this is value alignment; making models understand what we want. It's difficult to specify desired in numerical reward functions, so this paper explores language as a means of communicating preferences.\n\nThe running example environment is foraging for mushrooms. This is formalised through linear bandits with a listener agent that takes actions (choosing a mushroom) and gets a reward based on a weighted linear combination of the features that make up the mushroom (e.g. spotted green). The listener agent takes actions at each time step in the horizon H. Each feature has a weight, and the combination determines the total reward. Stripes have a reward of -1, but green +2, making a striped green mushroom net-positive.\nThere's a speaker that produces utterances to maximise the reward of the listener. An utterance can be either an action (e.g. eat the striped green mushroom) or a description (stripes give -1 reward). The speaker produces one utterance per horizon H, so if the horizon is 1 the optimal utterance is to just point to the best action with an instruction, whereas if the horizon > 1, it's more efficient to choose a description that can be re-used at different states of the horizon.\n\n*Literal listener*: interprets utterances literally, executes instructions directly (if utterance is \"striped green mushroom\", it eats that)\n*Pragmatic speaker*: (soft-)optimally selects utterances to maximise reward of literal listener, this means that when the horizon is known the speaker just chooses the optimal utterance to maximise reward (instruction or particular description based on the horizon)\n*Pragmatic listener*: pragmatic refers to there being some information outside of the literal utterance that influences how the utterance should be interpreted, and this information is unknown. In this setup, the information is the horizon. When the speaker says \"take spotted red\" or \"spotted is +1\" both utterances suggest taking the spotted red mushroom, but the former suggests the horizon is 1 and the latter > 1.\n\nThe contributions of the paper are presenting a formal model of cooperative social learning in a linear bandit setting, and showing that short-horizon speakers prefer instructions where long-horizon speakers prefer descriptions. The second contribution is showing that jointly inferring the speaker's horizon and reward function can mitigate the risk of model misspecification, and the third contribution is verifying the theoretical model with a behavioural experiment and showing that the model helps in a traditional RL setting.\n", "questions": "- I think what would help me with this paper is reframing it such that the finding is that providing an agent in a linear bandit setup with noisy information about the reward function prior to the choice of action is useful in the early stages of learning where the agent is still very uncertain about the distribution of rewards. (the claim that pragmatic inference helps I have less issues with in its current form), and that this noisy information could potentially be given in the form of language. Then acknowledging all the problems that come with learning directly from descriptions, like grounding, ambiguity, generalising to novel situations, etc, that need to be solved.\n\n- Relatedly, I wonder if relaxing the assumption that grounding is solved would completely change the game. Again, I think the difficult and interesting problem is grounding language to states. I would like to see some discussion on this.\n\n- When reading about the rational speech act and the recursive reasoning in section 3.1, I think an example of how this would work would be helpful and clarifying.\n\n- line 223: \"long-horizon speakers prefer instructions\" typo: instructions should be descriptions?", "limitations": "To some extent, but I'd also like to read a bit of discussion about the limitation of using language to directly attach a reward to an action, as opposed the more general case where getting a reward by following language instructions might take multiple actions.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- Value alignment by understanding humans through language is an important problem\n- Propose the setting of linear bandits with a social aspect (I can't judge whether this is novel due to lack of experience in theoretical RL)\n- A behavioural experiment is done to verify the theoretical model and this shows (to some extent) that the theoretical model makes sense.\n- The idea that learning from re-usable descriptions can help agents generalise, and these descriptions might be given in language form.\n\n**Weaknesses***\n**EDIT** this main weakness is addressed and was not a weakness but a misunderstanding on my part. Rating adjusted.\nMy main problem with this paper is that I take some issue with the claims about language, and therefore don't directly see the contribution towards learning from descriptions in linguistic form. The way I see it is as follows. What the authors call language in this work is (1) instructions that map 1-1 to actions where the grounding problem is assumed solved (e.g. the instruction \"eat the spotted green mushroom\" maps to the action \"spotted green\" and the model knows this a priori) and (2) descriptions that map 1-1 to features \"spotted +1\" maps directly to spotted mushrooms and their feature weight. These instructions/descriptions also give away the latent information about the horizon of the linear bandit game: if an instruction is used the horizon is short, if a description is used the horizon is longer. I don't think this qualifies as language (i.e. a symbolic reference system using syntax and symbols combined with context into meaning), because there's no notion of context-dependence, there's no notion of syntax, basically these symbols (e.g. instruction \"spotted red\") are just noisy indices that point directly 1-to-1 to something. In light of this, how I interpret the finding/contribution of this paper is that providing an agent in a linear bandit setup with noisy information about the reward function prior to the choice of action is useful in the early stages of learning where the agent is still very uncertain about the distribution of rewards. Additionally, this information might contain latent information about the potential rewards and jointly inferring this helps. This noisy information might be given in linguistic form, but I don't see how this paper makes a contribution towards actually giving it in linguistic form. I don't directly see how it tells me anything about how \"descriptive language and [...] are powerful mechanisms for value alignment and learning\" (which is what the authors summarise their result as). I would be happy to be convinced otherwise if the authors could tell me what I'm missing with regards to the formal description of the problem here, as it's certainly possible that I've missed something.\nTo me, the interesting challenge about learning from descriptions and the example the authors use to introduce their problem, namely learning in a situation where simply exploring through experience is too dangerous (eating the wrong mushroom is dangerous), is how to generalise from language instructions and descriptions (e.g., \"don't eat green striped mushrooms but spotted mushrooms are tasty!\") is precisely the grounding problem (how does this green striped mushroom relate to the new mushrooms I encounter in the forest?).\nThat said, the following statements did make me think this paper is about learning rewards directly from language:\n- \"Yet today, we lack computational models explaining such flexible and abstract language use. To address this challenge, we consider social learning in a linear bandit setting and ask how a human might communicate preferences over behaviors\"\n\"flexible and abstract language use\" and \"how humans might communicate preferences\" are overstated imo\n- \"Our findings suggest that social learning from a wider range of language—in particular, expanding the field’s present focus on instructions to include learning from descriptions—is a promising approach \"\n- \"Finally, we conduct a behavioral experiment showing our models support strong reward inference from human-chosen utterances.\"\nThis makes me think humans will describe situations and the model can learn from those, whereas in reality in the behavioral experiment humans choose from a dropdown menu of the instructions/descriptions, again making it less \"utterances\" that are part of language and more indices in an English representation. \n- the claim that 99 participants \"produced a total of 2772 utterances\".\nProduction of utterances make me think the humans in the behavioral study did more than selecting from a dropdown menu.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9yMeIeSTq9O", "reviewer": "Reviewer_mi6j", "summary": "This paper studies learning reward functions from natural language feedback. It proposes a linear bandit framework to model learning about the reward function by communicating with a teacher through natural language. The paper focuses on the distinction between _instructive_ feedback, i.e., saying which actions to take, and _descriptive_ feedback, i.e., describing which state to achieve. The main thesis is that instrutive feedback is preferred for short planning horizons and descriptive feedback is preferred for long planning horizons. The authors provide theoretical results to support this claim, as well as a behavioral experiment using humans giving and learning from natural language feedback.", "questions": "About Sec 3.3.: What's the difference between having the objective of the teacher be to induce true beliefs in the learning agents vs. causing the agent to achieve high utility? Can you elaborate more on choosing the latter rather than the former? Is there some fundamental trade-off between achieving high utility and having true beliefs? I'm guessing that these two goals are highly correlated, and I was surprised by this section which sounds like there is a trade-off.\n\n**Minor suggestions:**\n\n- It could be useful to expand the caption of Figure 1 to be easier to understand without the context of the main paper (for readers skimming the paper)\n- Is Figure 1.B based on real humans or the theoretical model? Would be good to include this in the figure caption.\n- In most of the paper you talk about \"agents\", but in the related work section you sometimes refer to \"the robot\". Are these the same, or are there parts that are specific to robotics? If not, it would be better to use consistent language.", "limitations": "The paper could highlight the limitations of choosing a bandit model more clearly. There is a clear trade-off between having a practically applicable model and a model that you can analyze theoretically, which usually motivated bandit models. It would be nice if the paper discussed this more explicitly.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper studies value alignment, a very important problem for designing robust AI systems. The paper succeeds in making this problem concrete enough for being relevant for practical applications.\n- The conceptual insights about different types of language feedback are interesting and novel.\n- The authors provide an extensive behavioral experiment with humans providing strong evidence for their hypothesis about instruction and description feedback.\n- The proposed theoretical framework seems simple enough for providing meaningful theoretical insights, but general enough to extend to more complex types of interaction in the future. As such the paper could lead to interesting future work.\n- The paper is very well-written and easy to understand.\n\n\n**Weaknesses**\n- I find the theoretical results not very surprising or insightful. To me Theorem 1 and to some extend Theorem 2 seem to follow quite directly from the problem setup (I did not check the proofs)\n- The notion of planning over a time-horizon H is a bit artificial in the linear bandit setup, as it is not a full RL problem. I think the bandit setup is still interesting, but the paper should make it clearer that this is a repeated 1-step decision problem and there is no real planning involved.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "34yoLdlUnkj", "reviewer": "Reviewer_8vXv", "summary": "The authors present a model of communication between teacher and\nstudent in an action/reward setting (reinforcemnt learning), wherein\nthe message or communciation contents are either to do explicit\nactions (do this or that) or more general descriptions of reward (what\naspects of the world or state tend to lead to positive/negative\nrewards). They call it \"instruction\" (do specific actions) vs\n\"descriptions\" (describing the reward function in more abstract or\ngeneral ways, which in the long run might be more effective). The\nauthors claim that the latter has been less explored in the research.\nThrough behavioral experiments, in specific limited settings (a linear\nbandit setting), the authors show that humans (as teachers) tend to\ntailor their communication appropriately, eg depending on the horizon\nof the problem: if the human knows the horion of the task is long,\nthey will pick descriptive messages (but if the horizon say only one\nstep, they may just utter do this!). The algorithms they develop\n(\"pragmatic agents\") together with their experiments and technical\nresults, provide evidence that this kind of (\"social\") learning, in\nparticular learning the teacher/speaker's reward function, can be\nachieved and is useful. For instance, the agent that are helped\ninitially by a human teacher, reach lower regret faster initially,\ncompared to a pure RL agent without this bias, and those with\ndescriptive (more general teaching) remain competitive with the pure\nRL agent even after many episodes.  The authors conclude that\nconsidering wider means of language use, rather than limiting to just\ninstructions, has promise for achieving value alignment, between\nhumans and artificial agents (in RL settings, etc).\n", "questions": "I'll include misc comments, notes, suggestions, and questions here.\n\nIn line 48: \"We find that short-horizon speakers (focused on a single,\nknown state) prefer instructions, while long-horizon speakers\n(reasoning about future, unknown states) prefer descriptions.\", it's\nnot that speakers are short-horizon, but how the problem is set up for\nthem (perhaps add a footnote to further clarify this).\n\n\n\nfootnote 5, 'available' what? in \"However, as the number of available\nincreases,\"\n\nSection 3.1: In Pragmatics subsection ..\n\n--> could it be removed? I am not sure if I saw its relevance to the\n rest of the paper (if you keep it, a few issues next).\n\n--> (still in 3.1) Further describe/explain this and subsequent\nformulae (this whole subsection is pretty complicated to decipher.. I\nam thinking it's better to move it to after the following\nsubsections..  or remove.. ):\n\nU(u;w^*)=ln P_{L0} (w^* |u)\n\n-> line 103:  What is soft? in \"(soft-)optimally\"\n\n-> Describe L1 more, say in \"of interest emerges at the L1 level.\"\n(a brief concrete example, perhaps?)\n\n-> In fact, 3.2 was more clear and should come first (before 3.1),\nespecially the helpful statement that describes your particular\nproblem setting: \"While the bandit problem is typically considered an\nindividual learning problem, we instead ask how an agent should learn\nfrom a cooperative, knowledgeable partner. We formalize this social\nlearning problem by introducing a second agent: a speaker who knows\nthe true rewards w and the initial state s0 , and produces an\nutterance u.\" (after 'partner' perhaps add '(speaker).' to make more\nclear )\n\n-> [probably a case of notation abuse or misuse] Note on line 122, the\n notation pi_L (a | u, s) is ambiguous (where pi is a policy), since\n the previous paragraph described policies as mapping states to\n action, while here pi_L is being applied to some conditional\n expression (action a given Speaker has uttered utterance u and given\n state s).. I am reading it as a policy that's a function of or is\n modified by, what the speaker has uttered (I think this is your\n intension), but this is ambigous.\n\n-> In line ~132: \"Rather than defining the speaker’s utility U (u; w)\n as Gricean informativeness [69] (i.e. inducing true beliefs, Section\n 3.1), we suggest that a cooperative speaker should maximize the\n listener’s rewards, thus grounding utility in the listener’s\n subsequent actions.\", this is not really a suggesetion, but happens\n to be the way you have set up the problem.  In this setting, indeed a\n cooperative speaker or teacher has to somehow teach the agent (of\n what actions or policy), and it is not surprising or unexpected the\n communication will take such a form (what specific actions to take,\n or more general information about the class of useful actions,\n depending on state or context).\n\n--> I disagree with the subsection (section 3.3) title \"Speakers as\n reward designers\": speakers (or good teachers), convey (their\n understanding of) the reward function the best they can.. they don't\n 'design' rewards (perhaps this title could be understood as the\n speaker trying to shape the reward function that is forming in the\n listener).\n\n--> line 120 (and elsewhere), in 'We formalize this {\\em social}\n learning problem', minor comment: to me, this appears to be a very\n specific case of 'social' learning. For instance, social learning,\n can take place by watching what others do (not just what they say),\n and often more than 2 agents are involved (the learner observes and\n learns from more than one 'teacher' and over several\n episodes). Perhaps further clarify, for instance, convert this line\n to: 'We formalize this type of social learning..'\n\n--> [general comment, related to previous one] Why have one (a single)\n  utterance only by the speaker? What about multiple rounds of\n  interaction? does learning or the analysis become more complicated?\n  Some discussion of that would be useful.\n", "limitations": "\nYes, I believe so. The authors discuss potential negative societal impact.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Summary of strengths and weaknesses.\n\nStrengths:\n\n-- The novelty of the approach: can broaden the approaches to\nteaching/communicating of best ways to behave (value alignment, etc).\n\n-- The behavioral experiments shed some light on how humans, as\nteachers or communicators of what to do, can behave. In addition there\nare technical and empirical results.\n\n-- Situating the work within the wider context is well done.\n\nWeaknesses:\n\n-- Captures a very limited window to language with a very limited\nexperimental set up (seems highly special case, or contrived).  Not\nclear whether/how this might apply to real human-robot (or robot to\nrobot) interactions in practice.\n\n-- The results regarding humans (the behavioral experiments) are not\n very suprising: humans do tailor their communication depending on\n context, and the context could be, in part, the horizon of the\n game/task (still, it is useful to show this in actual experiments,\n and agents that can use this to their benefit) (other relevant\n context includes: the level of the listener (which can depend on age,\n education level, ..), what information has already been conveyed, the\n situation both the speaker and listener are in, etc. etc.).\n\n-- The experiments made a number of simplifying assumptions (eg, the\n concreteness of the communication, and as the authors state, ignoring\n the grounding problem).  It is not clear whether they tackle the most\n challenging problems of humon-robot (or robot to robot) interaction.\n\n-- The clarity of the technical results, the formalism, could be\n   improved (see below in 'Questions' section).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept