Reward Design with Language Models
Authors: Minae Kwon
Sang Michael Xie
Kalesha Bullard
Dorsa Sadigh
Venue: ICLR 2023 poster (2023)
Keywords: reward design
foundation models
gpt3
reward specification
reinforcement learning
human-ai interaction

Abstract: Reward design in reinforcement learning (RL) is challenging since specifying human notions of desired behavior may be difficult via reward functions or require many expert demonstrations. Can we instead cheaply design rewards using a natural language interface? This paper explores how to simplify reward design by using a large language model (LLM) such as GPT-3 as a proxy reward function, where the user provides a textual prompt containing a few examples (few-shot) or a description (zero-shot) of desired behavior. Our approach leverages this proxy reward function in an RL framework. Specifically, users specify a prompt once at the beginning of training. During training, the LLM evaluates an RL agent's behavior against the desired behavior described by the prompt and outputs a corresponding reward signal. The RL agent then uses this reward to update its behavior. We evaluate whether our approach can train agents aligned with user objectives in the Ultimatum Game, matrix games, and the DealOrNoDeal negotiation task. In all three tasks, we show that RL agents trained with our framework are well-aligned with the user's objectives and outperforms RL agents trained with reward functions learned via supervised learning. 

Published as a conference paper at ICLR 2023
REWARD DESIGN WITH LANGUAGE MODELS
Minae Kwon, Sang Michael Xie, Kalesha Bullard', Dorsa Sadigh
Stanford University, DeepMind fminae, xie, dorsa)@cs forssl@es.stannfrrd. stanford. edu, ksbullaad@@deepinddcccomm
ABSTRACT
Reward design in reinforcement learning (RL) is challenging since specifying human notions of desired behavior may be difficult via reward functions or require many expert demonstrations. Can we instead cheaply design rewards using natural language inter- face? This paper explores how simplify reward design by prompting a large language model (LLM) such as GPT-3 as proxy reward function, where the user provides a tex- tual prompt containing a few examples (few-shot) or description (zero-shot) of the de- sired behavior. Our approach leverages this proxy reward function in an RL framework. Specifically, users specify a prompt once at the beginning training. During training, the LLM evaluates RL agent's behavior against the desired behavior described by the prompt and outputs a corresponding reward signal. The RL agent then uses this reward update its behavior. We evaluate whether approach can train agents aligned with NODAL.
user objectives in the Ultimatum Game, matrix games, and the negotiation task. In all three tasks, we show that RL agents trained with our framework are well-aligned with the user's objectives and outperform RL agents trained with reward functions learned via supervised learning. Code and prompts can be found here.
INTRODUCTION
Autonomous agents are becoming increasingly capable with the rise of compute and data. This underscores the importance for human users to be able to control what policies the agents learn and ensure the policies are aligned with their objectives. For instance, imagine training an agent to represent users in salary negotiation. A working mother fighting for livable wage may want their agent to be stubborn whereas new hire looking to develop a good relationship with the company may want their agent to be more versatile.
Currently, users specify desired behaviors by 1) designing reward functions 2) providing large amounts of labeled data. Both approaches are challenging and impractical for different reasons. Designing reward func- tions is not an intuitive way to specify preferences. For instance, it isn't i straightforward how to write reward function for a ""veratile"" negotiator. Furthermore, designing reward functions that balance between different objectives also known the "reward design problem" is notoriously difficult because agents ceptible to reward hacking (Amodei et al., 2016; Hadfield-Menell et al., 2017). On the other hand, one can leam reward function from labeled examples. However, that is not possible with a single example; need large amounts of labeled data to capture the nuances of different users' preferences and objectives, which has shown be costly (Zhang et al., 2016). Additionally, both approaches do not generalize well new users who have different objectives we would have to re-design our reward functions or re-collect data.
Our aim is to create an easier way for users to communicate their preferences, where the interface is more intuitive than crafting reward function and where they can cheaply specify their preferences with no more than a few examples. To do this, we leverage large language models (LLMs) that are trained on internet-scale text data and have shown an impressive ability to learn in-context from few or zero examples (Brown et al., 2020). Our key insight that
The scale of data that LLMs have been trained on make them great in-context learners and also allows them to capture meaningful commonsense priors about human behavior. Given few examples should be able to provide an accurate instantiation of reward values on a new test example, allowing for easier generalization new objectives.
description demonstrating the user's objective, LLM
To this end, we explore how to prompt an LLM as a proxy reward function to train RL agents from user inputs. In our approach, the user specifies an objective with natural language prompt. Objectives can

---

Published as a conference paper at ICLR 2023
Training
Prompt [p)
Alice regotisting splii books, and balls.
(I)
Task discription (p:)
Feed prompt
LLM
LLM provides output
Allic proposse booket hatrt builedd
Bub propose: book (ha--1 bal-d) Allic gropose booker handd balle]
Agnoemant! ppponts
Example uver describing dbjactivo (venatill Ieduviar) (9x)
Construct prompe (p)
"No"
versatile nogotistns?
Yes, becouse she suggrsses sied diffcrentt proposals
(3) Convert int "(" using parse and use as reward signal
propose:
Episode doscribed string using parse (pu)
proppos
propose: booked ball-0
(5) Summarize episode outcome as string (Ps) using panser f
Agrorment!
Alice poines
Bob 15 points
Update agent (Alioo) weights eplonde
Quaetion
b Alize versaile negotiator?
RL Evalistioon
(1) Somple trajectory from
Evaluate whether trajectory satisfies user objective
Alie
agent in enviromment
Figure 1: Depiction of our framework the DEALORNODEAL negotiation task. A user provides example and explanation of desired negotiating behavior (e.g., versatility) before training. During training. (1) we provide the LLM with a task description, a user's description of their objective, an outcome of an episode that is converted to string. and a question asking if the outcome episode satisfies the user objective. (2-3) We then parse the LLM's respouse back into string and that the reward signal for the Alice the RL agent. (4) Alice updates their weights and rolls out new episode. (5) We parse the episode outcome sample trajectory from Alice and evaluate whether it is aligned with the user's ohjective.
string and continue training. During evaluation, we
be specified with few examples when they are difficult to define (such as "versatility") or as single phrase when they are well-known concepts (such as "Pareto-opiimality)). We use the prompt and the LLM to define reward function for training an RL agent. The LLM takes the user prompt and a trajectory from an RL episode as input and outputs a score (e.g.. "No" or "0") for whether the trajectory satisfies the user's objective, which we parse an integer reward for the RL agent (Figure 1).
There are two advantages to prompting LLMS as a proxy reward function: (1) we can leverage LLM's in-context learning abilities and prior knowledge on human behavior so that users only need to provide a handful of example desirable behaviors and (2) users can specify their preferences intuitively using language. On the other hand, a potential disadvantage is that it is unclear how much prompt design will be required for the LLM to reliably infer user intent (see Sec. 5 for discussion). The goal of this paper is to explore how well LLMs train objective-aligned agents by providing reward signals, and empirically examine whether we can do so with no more than a few examples. Our contributions are as follows:
We introduce the idea using LLMs as a proxy reward function. We propose general RI. training framework that leverages this proxy reward and is agnostic to the RI. algorithm used.
We show that LLM can more accurately train objective-aligned RL agents by an average of 35% compared the baseline. We use few-shot prompting for the Ultimatun Game and DEALORNODEAL negotiation task as well zero-shot prompting in Matrix Games.
We conduct pilot study with 10 human users. Users rate our agent to be significantly more aligned with their objective than an agent trained with a different one, pee.pee p
We provide further analysis quantifying the amount of user data required for our approach as well as the effect varying prompts has on the LLM's reward signal accuracy.
2 RELATED WORK
Using Language for Reward Shaping. Recent Reinforcement Learning from Human Feedback (RLHF) works Ouyang et al. (2022); Bai al. (2022) use LLMs as rewards by fine-tuning them on large amounts of user data. Our work does not fine-tune L.LMs but uses in-context learning from only handful of user data.
2

---

Published as a conference paper at ICLR 2023
Several works Goyal et al. (2019); Carta et al. (2022); Mirchandani et al. (2021) shape rewards by training an RL agent to lear and complete intermediate tasks guided by language. In contrast, our framework does not focus generating subtasks but leverages the in-context learning abilities of an LLM to determine whether an agent's policy satisfies the higher-level task.
RL and Foundation Models. We leverage large language models such as GPT-3 (Brown et al., 2020) to leam& a proxy reward function while avoiding the need for many expert demonstrations. Ahn et al. (2022); Huang al. (2022) use an LLM provide a plan which guides robot with reasonable/teasible actions towards a human goal (e.g., with enumerating subtasks). In contrast, our work is different in that we are using an LLM to identify if a behavior satisfies soard-to-ppeiff"" properties of a human's 's objective and also offers users more control over how they want their policy to be executed. In the vision domain, Parisi et al. (2022) used pre-trained vision models as a feature extractor for the learned policy, but not to design reward signal. In similar spirit of leveraging self-supervised pre-training to design a flexible reward function, Chen et al. (2021) use broad dataset of human videos and a small dataset of robot videos to train reward function, which improves generalization to new environments and tasks. The interface for the desired task is a video of the task to be completed, instead of text in our framework, and the domain is restricted to robot tasks. More related works can be found in Sec. A.2.
3 USING LLMS AS A REWARD SIGNAL
Our goal is use LLM as a proxy reward function train objective-aligned RL agents from user inputs. We formalize the task using a Markov Decision Process M= (S,.S.. where S is the state space (e.g.. in DEALORNODEAL, the space of representations of utterances the negotiation so far), A is the action space (c.g., set of all possible utterances), P:S x AAx S + [0.1] is the transition probability, and y is the discount factor. Traditionally the reward function maps states and actions to a real number R x A R. In our work, we use an LLM as proxy reward function that takes in text prompt and outputs string. We define A" to be the set of all strings, p A as our text prompt (input to the LLM) and the LLM as a function :LM A", As illustrated in Fig, I, the prompt P is concatenation of four components including string to describe the task 01 E EA* and a user-specified string that describes their objectives using examples or description, P2 E A". Additionally, we include textual description of states and actions from RL episode, Ps. using a parser f:S A"' Ps can describe the final state, final action, a trajectory, or any other representation of the episode. Finally, include question (a E A" that asks whether the RL agent's behavior, Ps. satisfies the user's objective, Pr. We define an additional parser g: A - [0.1} that maps the textual output of LLM to binary value. We use this as the reward signal. Our framework replaces the traditional reward function with proxy reward, LLM, and can be used with any RL training algorithm.
Our framework is depicted in Fig. Before training, a user specifies 02 which can be N examples describing their objective or a description of their objective using natural language. In Fig, I a user provides an example of their objective: versatile negotiating behavior. During training, we construct a prompt p by concatenating a description of the task, the user-specified examples/dsscrption an episode's 's outcome, and question asking if the outcome satisfies the objective. We (1) feed the prompt to the L.M,, (2) take its output, and (3) parse it into an integer using function g; we use a handcrafted, task-specific parser. We use the integer as the reward signal. (4) The RL agent then updates its weights and rolls out an episode. (5) We parse the episode outcome into string using f and continue training; we also instantiate f as handcrafted, task task-specifiic parser. To evaluate our framework, we sample a trajectory (e.g., a negotiation) from the agent and evaluate whether the trajectory is aligned with the user's objective (e.g., whether Alice demonstrated versatile negotiating behavior).
4 EXPERIMENTS
In this section we investigate three questions to determine the feasibility and efficacy of our approach: (QI) Can LLMS produce reward signals that are consistent with user objectives from a few examples (few-shot prompting)? (Q2) When objectives are well-known, can LLMs produce abjective-consister reward signals without any examples (zero-shot prompting)? (Q3) Can LLMSs provide objective-aligned reward signals from examples (few-shot prompting) in more complex, longer-horizon domains? We evaluate our approach on three tasks: the Ultimatum Game, 2-player Matrix Games, and the DEALORNODEAL negotiation task (Lewis et al., 2017). We address (QI) using the Ultimatum Game. We use Matrix Games to address (Q2) because it has well- solution concepts such as Pareto-optimality. The
3

---

Published as a conference paper at ICLR 2023
DEALORNODEAL negotiation task is a longer-horizon domain where the LLM rewards agents for negotiating in a user-specified style; we address (Q3) in this task.
In practice, we do not have access to ground truth user reward functions - this is the function that are trying to approximate. However, for most of our experiments, we assume access to the true reward by constructing user reward functions that humans have been shown to have inspired by prior work. We use the true rewards only to evaluate our framework's 's performance. Finally, we include a pilot user study where we evaluate agent performance when we do not have access to the ground truth reward. We use the 'text-davinci-022" GPT-3 model with temperature as our LLM and our results are reported across 3 random seeds. Details on how we trained RL agents for each task are in A.4. s
Evaluation Metrics We evaluate our approach using the following metrics across our tasks (task-specific metries are described within each subsection);
Labeling Accuracy. We construct ground-truth reward functions for each domain. We report the mean accuracy of predictions of the reward value during RL training with respect to the ground-truth reward functions. This assesses how effectively the LLM can produce reward signals that are consistent with the user's objective.
RI. Agent Accuracy. After RI. training, we evaluate the learned policy with respect to the ground truth reward functions. We report the mean accuracy of RI. agents.
Baselines.
SL (Few-shot baseline). A supervised learning (SL) model trained predict reward signals using the same examples given to the LLM in our framework. Examples are represented using structured non-text inputs, making an easier problem for the SL model. This baseline only applies tasks where we use few-shot prompting (Ultimatum Game, DEALOONNDDEA)). See A.5 for details on training and model architecture for each task.
No Objective (Zero-shot baseline). In our zero-shot task, Matrix Games, we do not use any examples so we do not use SL as a baseline. Instead, we use a No Objective baseline where we prompt the LLM without using the user's description of their objective isolate the effect the description has on the LLM's response.
RL trained with Ground Truth Reward Functions. RL agents trained with ground truth reward functions. We use this an oracle.
4.1 ULTIMATUM GAME: TRAINING OBJECTIVE-ALINNEE AGENTS WITH FEW-SHOT PROMPTING
When defining a precise objective is difficult, we can instead give a few examples of desired behavior. For instance, in a resource division game like the Ultimatum Game, it may be difficult for a user to specify the exact percentage (such as 32.4%) of resources they would be happy with receiving. Instead it could be easier for a user to give examples of splits that they would be happy with. We explore whether LLMs can produce reward signals that consistent with user objectives from few examples in the Ultimatun Game.
Task Description. The Ultimatuun Game consists of two players, a Proposer and a Responder. A sum of money is endowed to the Proposer and they must propose a way to split the endowment with the Responder. The Responder can accept or reject the proposed split. If the Responder accepts, players receive money as per the split; if the Responder rejects, then both players get nothing. We train an RL agent to play the Responder. The agent learns to reject proposals according to a user's preferences. The game consists of single timestep and our RL agents are trained using DQN for le4 steps.
Ground Truth User Objectives. A rational Responder would accept any proposal, even if is unfair because getting something is better than getting nothing (in fact, this is Nash Equilibrium of the game). However, prior work in behavioral economics shows that humans are willing to "punish" the Proposer by rejecting unfair proposals (Vavra et al., 2018). For instance, student may reject an unfair proposal only if she receives less than 30% of the endowment whereas a wealthier person may reject if they receive less than 60%. We experiment with the following preferences:
Low vs High Percentages. Users will reject proposals if they receive less than(30% (30%, 60%} of the endowment.
Low vs High Payoffs. Users will reject unfair proposals if they receive less than (S10, $1003. They accept unfair proposals otherwise.
4

---

Published as a conference paper at ICLR 2023
Labeling Accuracy
Low vs High %
Low High Payoffs
(10 examples, ours w/o expl)
(1 example, ours w. expp)))
(10 esamples, ours wio wip
(1 example, ours w. w.eexpl.
St.
-200%
-000%
<20%
<00%
<510
<5300
<510
<5300
RL Agent Accuracy example, expli)
esamples, w/o expl.)
examples, ours w/o expl)
exaamplle ouns w. expl))
SL
Oan
Uree
Reward
-5100
<$30 -5100
Figure 2:
Ultimatum Game, Few-shot. (Top) Accuracy of reward signals provided by LLM and SI. during RL training when prompted with/trained on 10 vs I example. (Bottom) Conresponding accuracy of RL agents after training. LLM is able maintain a high accuracy when prompted with single example followed by explanation. We do not provide figures Inequity Aversion because both LLM and urivially achieve perfect labeling and RL agent accuracy.
Inequity Aversion (Fehr & Schmidt (2010)). Users will reject proposals if they do not receive exactly 50% of the endowment.
Prompt Design. We describe user's objective using 10 examples of the Ultimatum Game An example consists of the proposed split, the Responder's 's action, and a "yes/no" label whether the Responder's s action was desirable undesirable. These examples do not have explanations and resemble traditional dataset used for supervised learning. We also experiment with using a single example followed by a short explanation. Importantly, we do not explicitly mention the user's ground truth objective in the prompt. See Fig. 10 in the Appendix for an example of both types prompts.
Design Procedure. We randomly generated 10 proposed splits used for our prompt and sampled one proposal from the set for our single-example case. For Low vs High Percentages and Low vs High Payoff', used the same set of proposals across variants (i.e., same proposals for (30% and 60%) and ($10, $100)), We also randomly generated 50 proposals used to evaluate the LLM. Due to limited resources when querying GPT-3, we query the model's responses the 50 evaluation splits in a batched manner and save them. We then use those responses as the reward signal.
4.1.1 RESULTS
Labeling Accuracy. We evaluated our approach on our test set of 50 proposals over 3 seeds; results are shown Fig. 2). When prompted with 10 examples without explanations, the LLM and SL perform similarly well (see Fig. 2, top row). This result is not surprising, given that the decision boundary for the binary decision tasks relatively simple learn with 10 training examples.
Instead, if we prompt the LLM with single example followed by explanation, it maintains a high accuracy whereas SL trained on the same, single example drops in accuracy. We did not use the explanation as part of input when training SL because it only takes non-textual inputs. This result highlights the advantage of using an LLM over a supervised learning model: they require far fewer examples because they can leam from explanations (Lampinen al., 2022). We find that explanations are critical, as removing explanations when prompting the LLM with a single example results in a drop in LLM labeling accuracy (avg. drop of 31.67%) and drop in RL agent accuracy (avg. drop of 28.8%).
RL Agent Accuracy. The accuracy of the trained RL agents mirror the labeling accuracy.
Summary. LLMs are efficient in-context learners. They able to provide reward signals that are consistent with a user's objectives from examples - even a single example with an explanation will suffice.
4.2 MATRIX GAMES: TRAINING OBJECTIVE- ALLINEED AGENTS WITH ZERO-SHOT PROMPTING
When objectives are well-known concepts such as Pareto-optimality, can we prompt the LIM without giving any examples? We hypothesize that well-known objectives are likely to be in-distribution for L.MMs, and thus LLMs may be able to produce seecctive-alinned reward signals from zero-shot prompting, Since
We do not display plats for Inequity Aversion hecause both LJM and SL. received perfect labeling and RI. agent accuracy.

---

Published as conference paper at ICLR 2023
Labeling Accuracy
Total Welfare
Equal Rewards
Rawlsian Fair
Pareto-Optimal
1.0
1.0
1.0
1.0
No Objective
Ours
0.5
0.5
0.5
0.5
Regpelar
Scrambled
Reggalat
Scrambled
Regpalir
Scrambled
Regallrr
Scrambledd
RL Agent Accuracy (Regular Order)
Total Welfare
Equal Rewards
Rawlsian Fair
Pareto-Optimal
0.0
0.0
0.D
0100
No Cbijetive
Ours
True Rewand
No Cbjpctive
Ours
True Reward
No Oljpctive
Qurs
True Reward
No Objective
Qurs
Itue Reward
Figure 3: Matrix Games, Zero-shot. (Top) Accuracy of reward signals provided by LLM and No Objective baseline during RL training. We report results for both regular and scrambled versions of matrix games. (Bottom) Accuracy of RL agents after training.
we do use examples, we do use SL baseline. Instead we use baseline No Objective where we do not mention any objectives and ask the LLM for reward signal (see example in Fig. 11 in the Appendix). This baseline evaluates whether the LLM can successfully apply its knowledge of each objective.
Task Description. We consider two-player normal- matrix games: Battle of the Sexes, Stag Hunt, Chicken, and Prisoner's Dilemma. Each matrix game has four joint outcomes (i.e., a tuple of joint actions and rewards) and we address pure strategies in this task. The game consists of single timestep and our RI. agents are trained using DQN for 500 steps.
Ground Truth User Objectives. Although (mixed) Nash Equilibria are traditional solution concepts for normal form matrix games, users may prefer solution for other properties. For instance, Prisoner's Dilemma, users may prefer both agents cooperate because they will maximize total welfare even though it is not a Pure Nash Equilibrium. We experiment with four well-known solution concepts (or objectives):
Total Welfare. Outcomes that achieve the greatest sum of player rewards.
Equality. Outcomes that result in equal rewards between players.
Rawlsian Fairness. Outcomes that maximize the minimum reward any player receives.
Pareto-optimalltyy Outcomes where the one of the corresponding rewards cannot be improved without lowering the other.
Prompt Design. Prompts for each solution concept are shown in Fig. 11 in the Appendix. Due to limited resources with querying GPT-3, we queried GPT-3 in batched manner and saved the corresponding labels to train our RL agents. Our prompt enumerates the outcomes of a matrix game and then asks the LLM for the outcome(s) that satisfy solution concept. We do not mention the name of the matrix game in the prompt. As Kojima al. (2022), we elicit intermediate reasoning steps by asking the LLM to "think step-by-step" and provide a definition of the solution concept. To prevent any bias the LLM may have towards the order which the outcomes of a matrix game are presented, we also randomly scramble associations between joint actions and rewards (example shown in Fig. 12 in the Appendix).
Design Procedure. We tuned the wording of our prompt (e.g., how to describe the matrix game, whether or use chain-of-thought prompting) on the Battle of the Sexes matrix game find prompt that gave us accurate results. During evaluation, we kept the structure of our prompt the same for all of the matrix games.
4.2.1 RESULTS
Labeling Accuracy. Given that each game can have many outcomes that satisfy solution concept, we report the LLM's accuracy if its response does not include any incorrect outcomes. If the LLM identifies any incorrect outcome, we report a score of O. The LLM produces more objective-aligned reward signals with zero-shot prompting by applying its knowledge of well-known objectives, improving the labeling accuracy over having no objective by 48% on average with a regular ordering of matrix game outcomes and 36% with scrambled order. Scrambling the order of matrix game outcomes in the prompt lowers accuracy for most solution concepts. We suspect that this is because the matrix games are well-known and likely to have been in the L.M''s training set, where each joint action is usually associated with particular

---

Published as a conference paper at ICLR 2023
payoffs. Scrambling the associations between joint actions and payoff's could make the matrix game more cut-of-distribution for the LLM, and thus lower accuracy.
RL Agent Accuracy. Compared to labeling accuracy, it is casier for the resulting RL agents to be accurate because they only need to learn one correct outcome, not all of them. Thus, LLMs that only identify one out of two correct outcomes can still train objective-aligned RL agents. Results are shown on the bottom row of Fig. 3. RL agents trained using rewards from the LLM receive perfect accuracy for Total Welfare and Equal- ity and 75% accuracy for the other two objectives. The baseline receives lower accuracy for all objectives.
Does the LLM Correctly Identify Each Objective? As part of our prompt, we ask the LLM to provide definition of the objective before reasoning about whether outcome satisfies the objective (see Fig. 11 in the Appendix). Table 2 in the Appendix shows how the LLM defines each objective zero-shot. The LLM is able successfully recall the definitions for each objective except for Rawlsian Fairness it gets it partially correct. However, the LLM varies in its ability to reason whether an outcome of game satisfies the objective, which explains why the LLM does not receive perfect labeling accuracy for all objectives.
Summary. An LLM is able to identify well-known objectives and provide objective-aligned reward signals in a zero-shot setting.
4.3 DEALORNODEAL: TRAINING OBJECTIVE- AGENTS MULTI-TIMESTIN TIMESTEP TASKS
We have shown that an LLM can provide objective-alignee reward signals in single-timestep tasks. In longer horizon tasks we must give trajectories instead of states as examples in our prompts. Longer prompts can be challenging because is less likely for an LLM have seen them during training. LLMs also have recency bias which makes it harder for them to remember context introduced earlier on (Zhao et al., 2021). Can an LLM provide anjective-aligned signals in longer horizon tasks? We investigate this question in the DEALORNODEAL negotiation task (Lewis et al., 2017).
Task Description. DEALORR An agent Alice must come to an agreement with her partner Bob on the allocation of a set of objects (books, hats, and balls). Agents are shown a context, which includes the counts of each item and their private utilities for each item. In the original task, agents get rewarded based the agreed upon split and their utilities. If Alice and Bob reach a disagreement, both agents get nothing. We train Alice using on-policy RL by negotiating against fixed partner model, which refer to as Bob. See Sec. A.4 for more details on the domain and training.
is a long-horizon task with a maximum length of 100 timesteps.
Ground Truth User Objectives. We train Alice negotiate in different styles. For this experiment, we assume we have access to precise definitions in order to evaluate our models. Importantly, we do not give definitions of each style the LLM,, only examples. We experiment with the following negotiation styles inspired by previous literature Sycara et al. (1997); Caputo et al. (2019):
Versatile. Alice does not suggest the same proposal more than once.
Push-Over. Alice gets less points than Bob.
Competitive. Alice gets more points than Bob.
Stubborn. Alice repeatedly suggests the same proposal.
Prompt Design. We describe user objectives using three examples. Each example contains negotiation between Alice and Bob, a question asking whether Alice negotiated in a particular style, and yes or no answer followed by a short explanation. For example, see Fig. 13 in the Appendix.
Design Procedure. To create example negotiations, we randomly sampled three negotiation contexts for each objective and trained an RL agent (using the original task reward, without the LLM) to negotiate against Bob in these contexts. We then sampled negotiations from the trained model. We also made sure all three sampled negotiations did not have the same ground truth label. We use a separate set of contexts when training RL agents with an LLM in the loop.
4.3.1 RESULTS
Labeling Accuracy. The top row of Fig. 4 shows that the LLM labels more accurately than SL except for Versatile. For Versatile, both models perform similarly because SL leams to overwhelmingly predict a negative label (avg of 96% negative predictions) and the RL agent showed more negative examples of Versatile behavior (avg. of 70% ground truth negative labels). However, the large portion of negative
7

---

Published as a conference paper at ICLR 2023
Labeling Accuracy
Versatile
Push-Over
Computiiiee
Stubbom
Avg. User Rating Alignment to Cornect Style
0.
00
3.
60
Chrr
Our
Our
Agegen Accuracy
Versatile
Pusb-Ovvr
Competilive
LG
Lo-
LD
10
7-
05-
05-
65-
Ager Teated Agent Tiik Oppostedille
Chans
Chons
Trir
Chana
Trur
Chrra
True
Rowand
Bunward
lonwad
Kenard
Figure DEALORNODEAL, Few-shot. (Top) Accuracy of reward signals provided by LLM and during RL training. (Bottom) Accuracy of RL agents after training. (Right) Pilot study results. Agents trained with the user's preferred style were rated as significantly more aligned than an agent trained with the opposite style p 0.001.
Advantage
Diversity
Agreementt Rate
Versatile
0.17::0911
0.99
1.98
Push-Over
-2.95
0.8210.26
1.0+0..0
Competitive
2.92:10.64
0.74
0.88:16.5 0.82=12.35
Stubbom
1.36:12224
0.52:10.1
Table 1:
Qualitative results describing negotiations produced by agents trained with L.M.
examples prevents the agent from learning correct behavior as shown in the Versatile plot on the bottom of Fig. 4); here, we get larger performance gap between the LLM and SL.
RL Agent Accuracy. Results are shown on bottom row of Fig. 4. LLM improves RL agent accuracy over SL by 46% on average. Our method approaches the performance of using the true reward; we under perform by an average of 4%. We remind readers that is possible to outperform an agent trained with the true reward - especially when the LLM's labeling accuracy is near- mear-perfect as is the case for Competitive and Stubborn -ue to reward hacking or stochasticity during training. For instance, agents trained with the true reward for Competitive end in more disagreements, leading 0 reward for both agents and lower accuracy.
Is There a Qualitative Difference in Styles? Example negotiations of agents trained with the LLM for each style shown in Fig. 9 in the Appendix. To measure qualitative differences among styles, we looked at average advantage (Alice's original task reward Bob's original task reward), diversity (percentage of Alice's utterances that are unique in a negotiation), and agreement rate (percentage of negotiations that end in agreement). The results in Table demonstrate qualitative differences in styles.
4.3.2 PILOT USER STUDY
We conduct a within-subject. pilot user study to determine whether our trained agents can meaningfully align themselves with different user objectives when do not have access to ground truth objectives and users evaluate agenl performance.
Method We asked N N=10 10 users select style in which they wanted their agent negotiate in. We gave them an option to choose from our existing styles (Versatile, Push-Over, Competitive, Stubborn), or come up with their own. Importantly, users did not know how we defined these styles. We then showed users a list of 10 example negotiations generated via selfplay using an RL agent trained with a greedy objective (no particular style). We asked users select 3 examples (1 positive, 1 negative, and 1 positive or negative) where Alice displayed positive or negative behavior of the user's 's chosen style. For each chosen example. we asked users whether Alice demonstrated their chosen style and asked them to provide "Yes/No" answer as well as short explanation. These examples corresponded to unknown, user-specific ground truth reward functions that we did not have access to. We then trained a negotiation agent by incorporating the user- user-proviin provided examples in the prompt described in Sec. 3. We also trained agent to negotiate in the opposite style by flipping the "Yes/No" labels in the user-provided explanations. We hypothesize that users should perceive significant difference between these two agents. To evaluate our trained agents, had both agents negotiate on a set of 10 test negotiations. For each test negotiation, we asked users to rate how well each agent aligned with their chosen style scale from (least aligned) to 5 (most aligned).
Results Agents trained with the correct style were significantly more aligned (avg. 1.2+1.22 than agents trained with the opposite style (avg. 1.56 + 1.05), < 0.001, Fig. 4 (right). Users varied in which
8

---

Published as a conference paper at ICLR 2023
styles they prefemre: 4 users chose styles such as Polite, Push-Over, Considerate and Compromising. 2 users chose Versatile, and 4 users chose styles such as Stubbom, Competitive and Ambitious. These results demonstrate that our framework can produce agents aligned with differently specified objectives by changing the examples in the prompt. Furthermore, these results suggest that our framework can be used when rewards are difficult to define and results are agents with humans.
Summary. Our framework can train objective-alinnet agents when ground truth rewards are not present in complex, longer-horizon tasks. Agents are able align the style in which they complete task as evaluated by automated metrics as well as human users.
5 ANALYSIS OF DATA EFFICIENCY & PROMPT DESIGN
We have shown that we can use an L.M as a proxy reward function to successfully train objective-aligned agents across different tasks. This is promising result because it represents important step in enabling human-compatible and value-aligned AI systems. In this section, 1) we further quantify how data efficient method is and 2) also analyze how robust LLM is to variations of prompt design.
1) How Data Data-efficient is Our Method? We quantify how much more user data a supervised learning baseline would need in order to achieve the same labeling accuracy the LLM. Results in the Ultimatum Game demonstrate that 10 labeled examples is enough for an SL model to reach comparable performance, whereas single labeled example is not sufficient. In this section, quantify the amount of data needed for DEALLORNODEAL because is an example of a task where the decision boundary is not as easy to learn as the Ultimatum Game. We train SL by adding additional blass-balanced, labeled examples the original three examples it was trained on. We plot the average labeling accuracy SL achieves when trained on increasingly larger amounts of examples well as the accuracy LLM achieves with three examples, shown in Fig. 6 in the Appendix. Results show that SL requires on the order of hundreds of more labeled examples in order to be comparably accurate as LLM.
2) How Much Does LLM's Labeling Accuracy Change When We Vary the Prompt? As with many approaches that use LLMMs, a limitation of our approach is that it requires prompt design. We attempt to quantify the effort required for designing prompts as well as determine the feasibility of using non- engineered prompts from humans. We analyze the effect of prompt variation on labeling accuracy in DEALORNODEAL for the Stubborn objective. We vary different parts of the user-specified prompt at a time: the keyword (i.e., replacing "Stubborn" with its synonyms), the example negotiations, and the explanations associated with each example. Fig. 7 provides summary of our results. See Sec. A.7 for the full results. Results illustrate that an LLM can be quite robust to different prompts - they all outperform SL. Furthermore, the quality of the explanation seems be the most important in determining LLM accuracy.
6 LIMITATIONS & FUTURE WORK
User Studies. This work takes a first step in determining whether we can use LLMs as proxy rewards. Given promising results, we plan on evaluating our approach with a larger user study.
Multimodal Foundation Models. Beyond language models, multimodal foundation models such as Flamingo (Alayrac et al., 2022) can enable us to provide more complex environment states to the foundation model through images or other modalities while preserving an intuitive language interface for specifying the user objective.
Non-Binary Rewards. Another limitation of our framework is that the LLM only specifies binary rewards. We plan on exploring how we can incorporate the likelihoods that LLMS produce for each word as non-binary reward signal.
ACKNOWLEDGMENTS
This work supported by NSF Award 1941722, 2125511, 2006388, AFOSR, DARPA YFA Award, ONR, and JP Morgan Faculty Award. We would also like to thank Karl Tuyls, Ian Gemp, Albert Gu, Siddharth Karamcheti, Kanishk Gandhi, and other reviewers for this paper.

---

Published as a conference paper at ICLR 2023
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian lchter, Alex Irpan, Eric Jang. Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as can, not as say: Grounding language in robotic affordances. arXiv, 2022.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring. Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong. Sina Samangooci, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock. Aida Nematzadeh, Sahand Sharifzadch, Mikolaj Binkowski. Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: a visual language model for few-shot learning. arXiv, 2022.
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565 2016.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint arXiv:2005.14166 2020.
Andrea Caputo, Oluremi B Ayoko, Nii Amoo, and Charlott Menke. The relationship between cultural values, cultural intelligence and negotiation styles. Journal of Business Research, 99:23-36, 2019.
Thomas Carta, Sylvain Lamprier, PiemreYYv Yves Oudeyer, and Olivier Sigaud. Eager: Asking and answering questions for automatic reward shaping in language-guide rl. arXiv preprint arXiv:2206.00 096744 2022.
Annie S. Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from "in-the-wild" human videos. ArXiv, abs/2103.168117 2021.
Paul Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences. Advances in neural information processing systems, 30, 2017.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv 2014.
Emst Fehr and Klaus M Schmidt. On inequity aversion: A reply to binmore and shaked. Journal of economic behuvior & organizution, 1311) 2010.
Prasoon Goyal, Scott Niekum, and Raymond J Mooney. Using natural language for reward shaping in reinforcement learning. arXiv preprini 1903.0290000022 2019.
Dylan Hadeield-Menell, Smitha Milli, Pieter Abbeel, Stuart J Russell, and Anca Dragan. Inverse reward design. Advances in neural information processing systems, 30, 2017.
He He, Derek Chen, Anusha Balakrishman, and Percy Liang. Decoupling strategy and generation in negotiation dialogues. In Empirical Methods in Natural Language Processing (EMNLP), 2018.
Wenlong Huang, P. Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In ICMI,, 2022.
10

---

Published as a conference paper at ICLR 2023
Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
W Bradley Knox, Alessandro Allievi, Holger Banzhaf, Felix Schmitt, and Peter Stone. Reward (mis) design for autonomous driving. arXiv preprint arXiv:2104. 13906, 2021.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv, 2022.
Minae Kwon, Siddharth Karamcheti, Mariano-Florentino Cuellar, and Dorsa Sadigh. Targeted data acquisition for evolving negotiation agents. In International Conference on Machine Learning, pp. 5894 5904. PMLR, 2021.
Andrew K Lampinen. Ishita Dasgupta, Stephanie CY Chan, Kory Matthewson, Michael Henry Tessler, Antonia Creswell, James L McClelland, Jane X Wang, and Felix Hill. Can language models learn from explanations in context? arXiv preprint arXiv:220.00332, 2022.
Mike Lewis, Denis Yarats, Yann N. Dauphin, Devi Parikh, and Dhruv Batra. Deal or no deal? end-to-end learning for negotiation dialogues. In Empirical Methods in Natural Language Processing (EMNLP), 2017.
Jessy Lin, Daniel Fried, Dan Klein, and Anca Dragan. Inferring rewards from language in context. arXiv preprint arXiv:2204.02515. 2022.
Suvir Mirchandani, Siddharth Karamcheti, and Dorsa Sadigh. Ella: Exploration through learned language abstraction. Advances in Neural Information Processing Systems, 01-29529-29544 2021.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.021555 2022.
Alexander Pan, Kush Bhatia, and Jacob Steinhardr. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.03:: 1035444, 2022.
Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Kumar Gupta. The unsurprising effectiveness of pre-trained vision models for control. In ICMI., 2022.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Emestus, and Noah Dormann. Stable-baselinee Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22(268):1 8, 2021. URL http: jmlr. org/ ppppps//pee///// 20/22/20-20 20-1364 html.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 201 1.
Dorsa Sadigh, Anca D. Dragan, S. Shankar Sastry, and Sanjit A. Seshia. Active preference- -assd learning of reward functions. In Proceedings of Roboties: Science and Systems (RSS), July 2017. doi: 10. 355607/RSSS2017.XII1
Katia Sycara, Daniel Zeng, et al. Benefits of learning in negotiation. In Proceedings of the AAAI National Conference on Artificial Intelligence. Menlo Park, California, pp. 36-41, 1997.
Peter Vavra, Luke J Chang, and Alan G Sanfey. Expectations in the ultimatum game: distinct effects of mean and variance of expected offers. Frontiers psychology, 9:992, 2018
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229-255 1992.
Jing Zhang, Xindong Wu, and Victor S Sheng. Learning from crowdsourced labeled data: survey. Artificial Intelligence Review, 46(4):543 576, 2016.
Tony Z. Zhao, Eric Wallace, Shi Feng. Dan Klein, and Sameer Singh. Calibrate before use: Improving few- shot performance of language models. In International Conference Machine Learning (ICML), 2021.
11

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1
SUMMARY
OF RESULTS: IS IS POSSIBLE TO USE LLM AS PROXY REWARD RL TRAINING?
Avg. Labeling Accurecy
Avg. RL Agent Accuracy
shro-shot Baseline (No Obj.)
Tew-shoo Baseline (SL)
Ours
sero-shot
Tew shot
Ours
True Reward
Baseline (No Obj,
Baseline (SL)
Ulltination Game
0.67 0.34
0.91 0.27
0.67 + 0.34
0.9 0.26
1.0 0.02
Matrix Games
0.19 #
0.61
0.54 0.29
0.88
1.0
DEALORNODEAL
0.9 0.22
0.33 + 0.42
0.8 + 0.33
0.84 0.27
Figure 5: Average Labeling and RL Agent Accuracy across the different objectives for cach task across 3 seeds.
We provide a summary of results in Fig. 5. The figure depicts the average Labeling and RL Agent Accuracy computed across the different user objectives for each task, across 3 seeds. Overall, our approach able produce more objective-alignel reward signals than our baselines. Our approach also able to produce objective-aligned policies that are close in accuracy to policies trained with the true reward.
A.2 MORE RELATED WORKS
Reward Design. Our framework addresses reward design- how to engineer rewards so that they align with our objectives (Amodei al., 2016). This is challenging because tasks often have conflicting objectives that a human must trade off (Pan et al., 2022). Misspecifying reward functions can lead to reward hacking, or the gaming of specified rewards. Reward hacking has appeared in various domains such as autonomous driving (Knox et al., 202 and game same-playing (lbarz et al., 2018). We hope to address these challenges by leveraging L.MS and making it easier for humans to specify their objectives.
Imitation Learning& & Preference-based Learning. Another method of specifying user objectives is to learn them from expert demonstrations (Ross et al., 2011) or preferences Sadigh et al. (2017). These techniques either assume access to large datasets (Christiano et al., 2017) place restrictive assumptions (such as linearity) about the reward function (Sadigh et al., 2017). Recent work attempts to learn reward functions from language instructions using pragmatic reasoning (Lin et al., 2022). In contrast, our work relies on an LLM's in-context learning abilities to provide a reward.
A.3 LLM DEFINITION OF OBJECTIVES IN THE MATRIX GAME
LLM Defns. Objectives
Total welfare is the sum of the rewards of both players. (V)
Equality of rewards is only possible if both players receive the same reward. (V) Rawlsian faimess is defined as the maxmin value of the game, which the minimum reward that the player could get assuming that the other player is maximizing their reward. (X)
An outcome is Pareto-optimal if there is no other outcome that would make one player better off without making the other player worse off. (V)
Table 2: Completion of cach sentence given by LLM in pink. LLM provides correct definitions for objectives except for Rawlsian Fairness, which is partially correct.
A.4 DETAILS ON RL ENVIRONMENTS AND TRAINING
Ultimatum Game The environment single horizon with discrete actions (i.e., accept or reject) and continuous observations (i.e., a proposed split). We train DQN agents using the Stable Baselines3 implementation for le4 timesteps with learning rate of 1c-4 across 3 seeds (Raffin et al., 2021). We instantiate our policy as ML. with the default parameters used in Stable Baselines3.
12

---

Published as a conference paper at ICLR 2023
Parser g. The parser g that transforms the LLM's output into an integer reward signal is defined using a handcrafted parser. When prompting the LLM, we structure the labels for each example to be in "Yes/No" form which enables the LLM to also reply using the same format. We are then able search for the "Yes" or "No" strings and parse them into a or O respectively. In the rare occasion that the L.M does not respond in this form, we skip the episode during RL training and omit the example from our evaluation.
Further Analysis on Performance of No Objective Baseline. The average LLM accuracy of a random baseline across the four matrix games are Welfare: 0.125, Equality: 0.125, Rawlsian Faimesss: 0.078, Pureto-optimalit:: 0.172. The No Objective baseline's 's performance is close to random as can be verified by comparing the random baseline results with Figure 3 (top row).* * Behaviorally, we observe that No Objective acts like a random baseline: the LLM often hallucinates matrix game rewards and also displays incoherent reasoning when selecting answers.
Matrix Game The environment is single horizon with discrete actions (i.e., one of the four joint actions) and no observations. We train DQN agents using the Stable Baselines3 implementation for 500 timesteps with a learning rate of le-4 across 3 seeds (Raffin et al., 2021). We instantiate our policy as a MLP with the default parameters used in Stable Baselines3.
Parser g. We parse the LLM's response by hand, since LLM output can be variable in zero-shot settings.
DRADORA. We use a version of the DEALOR environment used Kwon et al. (2021).
In the environment, the goal is for an agent A to come to an agreement with partner B on the allocation of a set of objects (books, hats, and balls). During each negotiation, agents receive a context, C = [i:u A..cal.cB = [iug]] detailing the count of each item i as well as their private utilities, Ug, Item counts and utilities are represented as vectors E {1 [}, and Ug E {O.,...100}$ and are sampled uniformly.
After receiving contexts CB.CB. an agent is randomly selected to begin the negotiation. Agents negotiate for T' time steps by exchanging coarse dialogue acts 2; at each time step 1 t S T (He et al., 2018). Rather than negotiate directly in natural language, where the generation problem is hard and result in degenerate dialogues (He et al., 2018), we use these dialogue acts instead to focus on learning diverse and interpretable strategies.
A dialogue act It is one of five actions: propose, insist, agree, disagree, or end. The propose and insist acts take allocations of items as arguments [0A; onl where 0A.OB E {1....,4}² (e.g. propose: books-1, hats-2, balls=1), When an agent selects end, the conversation terminates and each agent asked to make their final selection.
If agents agree on the final allocation of items, i.c., +A+ =i, agents are awarded points based on their private utilities, TA = U,A" OO = ugg--go '0.. If agents do not agree, they receive 0 points. Each agent's context is constrained so that the agent can receive maximum of 10 points.
Agents are first trained using supervised learning on a dataset of human- humman negotiations provided by (Lewis et al., 2017) to predict the next token. We use a learning rate of 1.0 and batch size of 16. We then fine-tune these agents using RL where they optimize the expected reward of each dialogue act using REINFORCE (Williams, 1992). Agents are trained on 250 contexts for 1 epoch with a learning rate of 0.1. We instantiate our policy with four GRUs (Chung et al., 2014). We closely follow the implementation outlined in (Kwon et al., 2021; Lewis et al., 2017), please refer to those papers for more training details.
Purser g. The parser y that transforms the LLM's output into an integer reward signal is defined using a handcrafted parser. When prompting the LLM, we structure the labels for each example to be in "Yes/No" form which enables the LLM to also reply using the same format. We are then able search for the "Yes" or "No" strings and parse them into or O respectively. In the rare occasion that the LIM does not respond in this form, we skip the episode during RL training and omit the example from our evaluation.
A.5 SL MODEL ARCHITECTURE AND TRAINING
Ultimatum Game SL is trained to predict binary labels for a batch of proposed splits. We implemented SL as multi-layer perceptron (MLP) network that consists of a single hidden layer with depth 32. We also use ReLU activations after our input and hidden layers. We trained the model on the same 10 examples we gave to LLM for 5 epochs with the Adam optimizer. We evaluate the model on the 50 heldout test examples and save the model with the best test accuracy. We show the training and test accuracy for each user objective below:
13

---

Published as a conference paper at ICLR 2023
Table 3: Training accuracy for SL on the Ultimatum Game.
30%
60%
$10
S100
Ineq. Aversion
Train Ace. (10 examples)
1.0
1.0
0.9
1.0
1.0
Train Acc. (1 example)
1.0
1.0
1.0
1.0
1.0
DEALORNODEAL. SL is trained predict binary labels given negotiation as input. We closely follow the implementation of SL model found in (Kwon et al., 2021; Lewis et al., 2017). A negotiation consists of context, coarse dialogue acts exchanged between Alice and Bob, and the outcome of the negotiation (the final split of items and whether agents agreed or disagreed). Please refer to Sec. A.4 for more details on the environment. We implement SL using MLP context encoder, MLP outcome encoder, and GRU ((Chung et al., 2014)) to process the coarse dialogue acts. The MLP encoders consist of an embedding layer followed by linear layer with a Tanh activation function; we use hidden size of 64 for the context encoder's s linear layer. We similarly embed each coarse dialogue act before feeding it into the GRU. We use a hidden size of 128 for the GRU. We train SL on the same 3 examples we use in our prompt for LLM. We train for maximum of 50 epochs using the Adam optimizer. SL received training accuracy of 100% for all of our objectives.
A.6 How DATA-EFFICIENT IS OUR METHON?
SL Labeling Accuracy When Trained with # Additional Labeled Examples
Versatile
Pusl-Over
Competitive
Stubborn
Ours
100
2001
300
100
300
300
100
300
400
200
Additionn Excempiis
Additional Examples
Additionll Esemples
Additional Foomples
Figure 6: SL requires on the order of hundreds of more labeled examples in order to be comparably accurate to the LLM.
A.7 How MUCH DOES LLM'S LABELING ACCURACY CHANGE WHEN WE VARY THE PROMPT?
Effect of Prompt Variation Labeling Accuracy for Stubborn (N+3, seeds)
SL
Ours
Vary Keyword
Vary Example Negotiations
Vary Explanations
0.58 + 0.48
0.97 + 0.16
0.91 + 0.28
0.93 + 0.28
0.79 + 0.33
Figure 7: On average, varying the prompt does not have large impact on the accuracy of the LLM.
We analyze the effect of varying prompts the LLM's labeling accuracy for the Stubborn negotiating style in DEALOONOODEAL. We vary prompts in three ways: we vary the keyword (i.e., replacing "Stubborn" with its synonyms), the example negotiations, and the explanations associated with each example.
When varying the keyword, we use the synonyms: "Headstrong". "Obstinate", and a less commonly used word, "Froward". To vary example negotiations, we randomly sample three new negotiations to have counterbalanced labels, all positive labels, or all negative labels. We vary the explanations by coming up with two plausible sets of explanations with the scenario where we give no explanations. Results are shown in Fig. 8. Overall, varying prompts do not have large impact on labeling accuracy they all outperform the baseline. However, the quality of explanation seems to have the largest impact on labeling accuracy.
might have given for each example. We also experiment
A.8 WHAT IS THE IMPORTANCE OF INCLUDING THE TASK DESCRIPTION, P1 IN THE PROMPT?
We experiment with removing P1, the task description in the Ultimatum Game with a single example followed by explanation. Performance increases slightly in LLM labeling accuracy (avg. of 8%) and RL agent accuracy (avg. of 9%). We run the same experiment in the Ultimatum game in the case of 10 examples with explanation. Performance drops slightly in L.M labeling accuracy (avg. 4.4%) and RI.
14

---

Published as a conference paper at ICLR 2023
Effect of Prompt Variation Labeling Accuracy for Stubborn (3 seeds)
Vary Keyword
Vary Example Negotiations
Vary Explanations
"Subboon (ours))
'Hoadstnng
"Chelinate""
"Troward
Counter balanoed
Poeltvv
Negalive
Qurs
Quality Expl.
Quality Expl.
Good
No
Figure 8: Varying prompts do not have large impact on labeling accuracy.
agent accuracy (avg. 5.3%). We conclude that p1 not conclusively influential in improving performance in few-shot settings.
A.9 EXPERIMENTING WITH SMALLER LLM SIZES
We experiment with GPT-2, 1.5B parameter model. We We find that GPT-2 underperforms GPT-3 both labeling (avg. 15%) and RL agent accuracy (avg. 49%). GPT-2 outperforms the SL baseline in labeling accuracy (avg. 24%). and slightly underperforms the SL baseline for RI. agent accuracy (avg. 2.7%). Results are averaged across styles and seeds. GPT-2 (1.5B) several orders smaller than GPT-3 (175B), and expect models larger than GPT-2 to close the gap with GPT-3's performance.
A.10 EXAMPLE NEGOTIATIONS
Context
Alice: boolefcounth valuer)) hate/countt value?) hell-fooutt3 valuet))
Bob book (counti) value:3) hat (countl value?)) ball (count3 vallest)
Versalile
Push-over
proposer HeemOe
proposer itemee lieml= itmm2e)
Bob propose: Ibemnh O itemi 1tem2-0
Bob prrppooe: stem Iteml stem2-0
Allie: proppows HemDee iomml00 Nem222
Aliie:
Bob
Agreement
Agaoment!
Alice:
2 points
110 points
Bob points
Compelitive
Stubborn
Allice proposee HemmO+ item1-0 itembe3
Alice: propose iumll- Hemn-] iemm21
propose: ireiet: HecrOe Hembee
Bob propose: Wem0 Iteml
Rem2
propose
iwmbe]
Bob : ngree
propose:
propose: Iteiied
Agrementt
propoee: iemm0=0 itemle
Alloe: 10 points
Alice: propose: item0-0 stemi- ibem2-1
Bob points
Bub propose: lemn-- iteml- itm2-0
Alice: propose iemm-- itemll item2e1
Bob temo Iteml item2
propose itmile
iwm224
:cisaggo
Disgpwment!
Alice (potential 0)
Bob (petential
Figure 9: Example negotiations after Alice is trained with reward signals from LLM in DEALORNODEAL. We illustrate qualitative differences how Alice negotiates for the same context. Bob is an agent that is trained with supervised leaming.
15

---

Published as a conference paper at ICLR 2023
A.11 EXAMPLE OF PROMPTS USED IN OUR EXPERIMENTS
A.11.1 ULTIMATUM GAME
Further Explanation of Our Prompt Selection Process. When constructing our explanations, we encourage the LLM to produce intermediate reasoning steps by using the "Let's think step by step" template used in Kojima al. (2022) which has been shown improve performance.
10 Evamples, No Explination
Example with Explaration
Task docripttiu
should split $10 and
Pt and I2 are playinng the U Ultimaaumm Came. Pt propases how they either accept coject. accepts, the done. aejets, tham partion get nothing
Pi and F2 are playing the Ultimatumm Came. P1 propuses how they should split $10 and either accept coject accepts, denw. sejecs, tham parties get nothing
Yeok dascijotion
proposes split $4.21 for and $5.79 decimblle wifishh sehurvion. No
propesee split for and 9002 desimble beburvion
rejected pursisee desinall?? tat's step by stegs should reject this otfer. Thansion, the
pummishes
difined
of
desinible?
12 eucuives 50.22 outcomee destrable.
Fl proposes a split $1.29 and $5.72 F2, rejected odiar.
Eproad onicesse
outcome is detined as where puntihes wilissh seturvion. the outeome desimible?? No
Fl proposes 2 split 59.21 for and 90.9 P2, rejected offer. A tisiimbble outoome is defined as one wheree purricass sollish behavior the outcome destralee of Let's think stop by steps
Exampies Ofjectie
Questine
Fl propeses a split of $9.78 for F1 and 90.22 for 12, 72 rejecied this oditr, A destrable cutcome is delined as one where 12 puntibes IT's wilissh behariior. Is the outcome desinable? Yes
|7 more esamples]
Ill propeses a split of 99 for 11ll and 90779 for 12, 12 relected this offer A deserable oubrome defired as when pumishes PT' welfish ietaavior la the outeome desmaile??
Qustin
Figure 10: An example of few-shot prompts used for the Ultimatum Game. We highlight the four parts of each prompt.
A.11.2 MATRIX GAMES
Further Explanation of Our Prompt Selection Process. We found that structuring the outcomes of the game multiple choice question improved performance. We also encouraged the LLM to produce intermediate reasoning steps by using the "Let's think step by step" template used in Kojima et al. (2022) which has been shown to improve performance.
Total Wilfiae
Equality
Rankinn Faimos
Ponttooptimallt
No Olryctive
We haver
We have tycrplayse where PI and cin cheoar
We have wher Pi and P2 can chuose these opeions Oytanee
We have tooeplayer whore PI and 12 can chersa these optione. Optices:
We have twoellaye whar Pl and choose of these options Optionsc
whem Pi and P2 choose
these Optinne
these options. Optics::
A. aettint (PT) and
A if actionn ((P1) and
A. wrthont(P)) and nctionn (P3) wo Pll gits reward of F2 geis reward 2f
ifaction ((1)) and
A 1f (Pt) and netion (P3) mo P1l gits noward al F2 ged roward of
(P3) so P1 pits noward
schiomt,PP3 so Pt gath reward
acthon so Pi gets reowand
2.2, F2 geis reward of 2.
of P2 gets reward ol
2. 12 gets reward of
B. actionn (P1) and
B. if acthond and
B. octionn (P1) and
B. if acttionn (PP) and
B. aetuin (Pa) and
Tisk
acson2/F2) so Pll gats roward
achissb/P2)) so Pl gets reward
actoon//2)) so Pll gats roward
2/thoe2/72) so Pl gots rowand
oction2/P2) no gets noweed of ges roward
dnermption
of ges roward
of 1, 12 gets resorrd of
of geis roward
1, 12 gets rescand of
C (PP)) and
C. seriont(P1) and
2ethoe and
of wettond and
(facttin and
o03on21F2) gess rewaed
action1/22) is gots reward
o0 geis roward
actioe is P1 gets reward
action) gois rowaed
3, gebs noward
of gits mecord of
of gets noward
3, 12 gots recond
3f gies noward
(ot action2177) get reward
action
oction((1))
octiond)
action2(11) gets reward
action2) gets reward
so gets reward D.
oction2 ges of
get
gets
geta of
P2 gats
Which optionto resuilt greatest wellare? Let's think Desrrpiee Total weltore of Ofjaitie
Which optiontst resull
Which optionto resulii Rawiisaan tair rewards? Let's think Rawleian tximess is
Which optiontst Pareto optimal? Let's think step
Which optioots) should select
Qassie
equality rewands? Let's thik stup by Equality rewards is
An onneome Pareto- optimal it
Figure 11: Examples of zero-shot prompts used for each objective, including the no-objective baseline, in the Matrix Games. Due to limited resources when querying GPT-3, we queried GPT-3 in batched manner and saved the corresponding lahels to train our RL. agents. Consequently, we do not have an Episode outcome in our prompts.
A.11. DEALORNODEAL
Further Explanation of Our Prompt Selection Process. We chose 3 counterbalanced examples from a training set of sample negotiations. This training set was generated via selfplay using an RL. agent trained
16

---

Published as a conference paper at ICLR 2023
Chicken Game (Regular Order)
Chickem Came (Serambled Order)
We have bwo-player game where 12 options.
two player game options.
F2 one
Optooss:
Options:
action ((P)) and action => Pl gets reward of reward 2.
actiion(()) (PI) and action => PI reward of reward
actionl((")) and actioni/'2) gets reward 12 reward 3.
actionl(P") action/ P1 gess
reward
action]([2) gets reward
gets reward gets reward
actioni(22) gets reward gets reward action2())) action2( gets reward of gats reward
action((11) action2/)2)
gets zeward
Which optonis) are Pareto- opreml- Let's think step by step:
Which optionis) are Pareto-optimal? Let's think think by step:
An outcome Pareto opetoolpttimll
An outcome Pareto-optimal
Examples of regular and scrambled outcomes for the Chicken Game (Matrix Game experiments). Scrambling creates new associations between joint actions and joint rewards. scramble outcomes order remove any bias L.L may have towards order which outcomes are normally presented.
Figure 12:
with greedy reward (no particular style). We chose complement examples with simple and succinet explanations.
17

---

Published as a conference paper at ICLR 2023
Tist devription
Alice and Bob regotiating how spiii lox hust, and bulls.
Alice bookenjeommt value he-fcountt valur-4) ballsfrountd valur Bob bookk(coont2 velueil) cute(connti values)) balleccounts valarcl)
Bob book-0 har-1 ball-2
peopose booke1 bated ball-d
proposer booked tullld
propoce book bat ball aggree
Apreement
points
points
stubborn negotiatoo?
Yes, becouse repestedly suggested some prepocel, booked hatel
Alice bookeejcount value=)) hut-fcountzl value 4j ball-joountd value Bob hook-(count2 volueds cotenco valuec) bolls(counts values))
Bobs proparer booke2 hatel holle]
Alice Slleepeope propueer bookel batel bailed
Bob agree
Examples of abjective
Agnecment
Alior points Bob 44 points
Is Alice a stubborn oegotiator?
Nos becouee ale did not reperiedly suggyst the proposal.
Alice booeetcounti value:)) (oruntt value ball foll-(orunt] valuc.)
Bob book toountl valuc hat (count) valuei)) ball (cnuntl valuci))
peopoee
proposs: Summll Him22-0
peopose:
propores
zagree
Agrement!
points
Bob :00 points
stubbeen negotiator?
Nis, becous ale did ros repestely suggeed the same propooo.
Alice bookesjeonnt values)) hes-ccontt valuer 1) ball-eountt volue
Rob -bvo--jrontt velue!) bubefenentt values)) bollsfooontt vahaes)
Fpisale cotcome
Bab proposet book Sook-Ohar-O0 har ball-3
Alice agnoe
Agreement
Alice points Bub points
Quottme { 1s Alice stubborn negotiaton?
Figure 13: Example of a prompt used for DEALORNODEAL.
18


Reviews:
[{"review_id": "mxp8FFEaIPL", "reviewer": "Reviewer_UKq1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper \"Reward Design with Language Models\" proposes to study to effectiveness of pre-trained large language models for designing rewards in RL tasks (tasks implying language). It shows that, somehow unsurprisingly given recent literature, leveraging these models to drive agents with textual prompts is possible in few-shot or zero-shot settings. ", "summary_of_the_review": "  - Could authors position their work w.r.t literature on prompt tuning, zero-shot learning with pre-trained LLMs (in particular, the last reference I give below looks very connected to the presented work), and also w.r.t. reward shaping with language? \n   - The global learning setting is not fully clear to me. How is defined or learned the function g that maps the textual outcome of the LLM to an int reward ? It is a language model too ? It is learned from binary true rewards ? If it is manually defined, how do authors manage variability of llm outputs ? (for expected answers such as yes or no, it is possible that the llm outputs something different right ? or do authors restrict its possible outcomes ?) \n   - In fig 8 right, the prompt finishes with \"let think step by step :\". So the LLM is supposed to propose a step by step forecasting of reasoning ?  \n   \n\n\n\n\nMultitask Prompted Training Enables Zero-Shot Task Generalization. ICLR 2022\n\nFinetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.\n\nElla: Exploration through learned language abstraction. In Advances in Neural Information Processing Systems (NeurIPS), 2021\n\nEAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL. CoRR abs/2206.09674 (2022)\n\nDo as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691. (2022)\n", "strength_and_weaknesses": "Strengths: \n- Using textual prompts to drive RL agents is an emerging topic with promising outcomes\n\n- Promising results for controling RL agents with language\n\nWeakness: \n    \n- Using pre-trained LLMs in zero-shot settings is clearly not new, many studies already exist on that topic. Not on the specific tasks authors suggest, but the transferability of their embedded knowledge with well specified prompts is well known. Prompt tuning is an emerging field following that observation. \n\n- Shaping rewards with LLMs is not new either, as many works recently proposed to leverage them to split difficult tasks into textual subgoals, or to drive agents by asking questions for instances, in a curriculum RL fashion. These works should at least be mentioned in the related work.   \n\n- Writing is not always very clear. Readers have to go in many various places again and again during reading to get things together. Very difficult to understand without appendix. Many settings not sufficiently formally specified to well analyse results (e.g., the supervised learning of SL, the definition of function g, etc.). \n\n- Environments look quite simple (short trajectories, few actions, deterministic transitions, full observability). It would be helpful to see more classical complex tasks of RL, in environments such as TextWorld, Alfred or BabyAI for instances. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "lpdR0xaQR3", "reviewer": "Reviewer_1rSQ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors in this paper study how to utilize large language models (LLM) as a proxy reward function for text based RL such as Ultimatum game, matrix game or the DealOrNoDeal negotiation tasks. The main idea is to consider to take a combined prompt $\\rho$ as the input of LLM, which consists of the task description, user provided example, converted outcome of an episode, and  a question asking if the outcome episode satisfies the user objective, and then use. the LLM's response as signals for reward.  ", "summary_of_the_review": "Overall I think the paper introduce an interesting approach for utilizing reward function for text based RL tasks, though the current approach is not quite general and can only applied  to short horizon based methods, and there is no user study with human evaluation to understand what is really going on here. ", "strength_and_weaknesses": "**Strength**\n- The idea is simple and intuitive, which is quite easy to understand. \n- The authors conducted detailed ablation study to show the effective of using LLM as Reward \n\n\n**Weakness** \n\n- I believe the approach can only be used for text based games, can not used for other more general tasks such as control tasks. \n- The authors only evaluate three tasks, and two of the are just 1 step tasks, and the left one is multi-step horizon based, but from the description it seems that the horizon is not that long. \n- Also, from the illustration figure, I feel the current approach may only deal with short horizons, since you will need to consider the whole trajectory as the prompt input to the LLM. \n- There is no user study. Since your tasks involves languages, I feel it should be necessary to conduct some user study with human evaluation, thus we can get some sense of what exactly happens here. \n- Also, for RL researchers, people would usually have the impression that  sparse reward would not actually work in practice, and require some more techniques to improve. Can authors explain why sparse reward provided by your approach actually work? What's the difference if we use a sparse intermediate reward function for training? Will that actually work? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iBEpTV2Zojt", "reviewer": "Reviewer_7igZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to use a pre-trained large language model to specify rewards for reinforcement learning. The model takes a description of the task and reward, and optionally a small set of demonstrations and outputs a binary reward for a new queried game state. Based on this reward, an RL agent is trained to complete the desired task and is compared to agents trained with the GT reward as well as a reward generated by training a supervised model on the task demonstrations. \nThrough experiments in different domains, authors show that: 1) The proposed model can represent rewards, and train successful RL agents from even a single example, 2) LLMs can use background knowledge and predict correct rewards from commonly known objectives, even in a zero-shot setting and 3) They can model rewards in long horizon tasks, provided demonstrations and explanations.  \n", "summary_of_the_review": "As stated in my strengths section, the paper addresses a very important problem with a solution that is simple and relatively general-purpose (safe for some of the cases mentioned in the limitations). The contributions are very clear and the design of the games and experiments show that using language models to specify rewards offers advantages in multiple ways. Moreover, as LLMs become a more pervasive building block in decision-making models, the approach has the potential to be extended and generalized as more research is done on this end (for instance through multi-modal foundation models to address more general states). As such, I think this is a valuable paper for this conference. The reason why I cannot give a strong accept is because I am missing experiments at the core of this paper goal: providing rewards where there is no way to specify them, evaluating them using human ratings.  Moreover, I would really like to clarify some of the details in the experimental results, as stated in the weakness section, experiments. This would help me convince that the proposed approach offers an effective way to specify rewards.\n\n[EDIT]: The authors have addressed all my concerns, clarifying experimental design and metrics, adding thorough ablation studies and a human experiment that proves the effectiveness of the approach under real scenarios with hard to specify rewards. I recommend this paper for acceptance.", "strength_and_weaknesses": "\n**Strengths:**\n\n- Clear and well written paper. The goal of the paper is clear and well motivated, backed by prior work evidence, the contributions of the paper are clearly stated, the method is also clearly written, even though a bit more clarity could be added into the particular example in figure 1. The experimental design, results and takeaways are clear as well.\n- The approach for reward specification is flexible, allowing both example-based specification, through a description of the desired task or a combination of the 2.\n- To my knowledge, this is the first approach using LLMs to specify rewards, it is thus a novel formulation and a promising one, as LLMs become a building block of applications in ML.\n- Good experimental design. While the tasks are relatively simple, they represent scenarios for which we care about specifying rewards, and prove the 3 main points that authors are claiming with the proposed metthod. \n- Very interesting to see that the LLM contains concepts such as pareto-optimality. It would be great to see if the LLM can combine multiple concepts at the same time, when possible, or negate concepts. I don't think it is a requirement for this paper but I would suggest adding it as follow up work. \n- As shown in 5.2, the method is relatively robust to different prompts, as long as an explanation for the reward is given, implying that the method should not necessarily require thorough prompt design.\n\n**Weaknesses:**\n\n- Given that one of the motivations in this work is to provide rewards in cases where it hard to engineer them, I think that this paper would strongly benefit from having a case where the reward cannot be formally described, and measure with a human study whether people prefer policies that come from rewards generated from the LLM vs policies that come from people playing successful games according to such reward. \n\n- Experiments:\n    - I am missing more results to understand the role of explanations vs few-shot. In experiment 1, what is the performance if we use a LLM with only one example and no explanation? Does the high performance come solely from explanation or it is also because the model has better capacity for few-short learning. On that line, what is the importance of \\rho1? Does the LLM perform the task worse when given few shot examples and no description of the task?\n\n    - I am missing more analysis and context on some of the results. In 4.2.1 what is the performance of a random baseline? Is there bias towards a given user objective when it is not given, or in the failure cases, or are the best case solutions random?\n\n\n\n- While this is also the case of works like Huang et al. 2022, a limitation of this approach is that it requires states and actions in the environment to be represented as tokens that the LLM has been trained on. This limitation is transparently addressed in the last section though, and since many tasks can actually be specified via language, I still think this work is a valuable contribution. A minor suggestion would be that besides commenting on multi-modal states, authors also mentioned non-token actions. This may be important for continuous control for instance, where we may not want aggressive motion in a robot arm.\n\n- See my comments on clarity.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "QwJ0Wb2FXP", "reviewer": "Reviewer_49Yr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes using LLMs to understand underlying reward functions given a few language examples, then demonstrate that RL agents trained with this LLM-provided sparse reward function nearly match those trained with true underlying reward functions on some simple tasks.", "summary_of_the_review": "I like this paper, and it demonstrates a well-written execution of a simple idea. The main issue that I have with this paper is the lack of experiments on smaller or more accessible LLMs. With that said, I still believe this paper should be accepted.", "strength_and_weaknesses": "## Strengths\n\n****************************************Writing and Clarity:****************************************\n\n- The writing is very intuitive and straightforward; the paper is easy to read.\n\n******************************Framing/Method:******************************\n\n- This is a great execution of a simple idea (simple being a good thing here). The authors tested reward design with LLM on a variety of environments and train an RL-agent in-tandem to prove that not only are the labels accurate with the LLM, but these labels allow RL agents to learn well on tasks compared to the true underlying reward functions.\n\n************************Experiments:************************\n\n- The authors have comprehensive experiments demonstrating one of the claims of the paper: that LLM’s can learn from far fewer examples due to all the data they’re trained on.\n\n## Weaknesses\n\n**Clarity:** \n\n- Some details regarding the games seem to be missing, for example what is the fixed Bob agent in DealOrNoDeal?\n\n**Experiments:** \n\n- Missing LLM size comparison: While using GPT-3 helps demonstrates the points the authors aim to make about the ability of LLMs to act as reward functions, GPT-3 `text-davinci-002` and even open source alternatives of the same size are prohibitive to most researchers either due to cost (GPT) or compute limitations (not enough VRAM to run inference). This also presents a problem because the experiments don’t really show that LLMs in general can help design reward functions, instead it shows that a 176B param GPT-3 can help design reward functions. Can the authors experiment with smaller alternatives? For example, at least GPT-3 `curie` (a 1-line code change for the authors), and/or the OPT family of open-sourced models (e.g. 350m, 1.3b, 6.7b - compute-matched with `curie`, 30b).\n\n******************************************Missing related work:******************************************\n\n- Using Natural Language to aid with reward design is a related subfield with some works, here is one for example: [https://www.ijcai.org/proceedings/2019/0331.pdf](https://www.ijcai.org/proceedings/2019/0331.pdf). This is a little bit related to the framing, as the authors should probably add a related works section explaining language as a proxy for rewards.\n\n****************************Minor details:****************************\n\n- With temperature 0 isn’t GPT-3 just producing the max-likelihood token? In that case I don’t believe top-p changes anything about the model’s outputs and therefore is irrelevant (Section 4 2nd paragraph).\n- Unfinished Appendix: A.5 and A.6 are not used and don’t have any text.\n- Typos:\n    - “They require far few examples” > “far fewer”\n    - “on the agreed upon the agreed upon split”", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster