Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors in this paper study how to utilize large language models (LLM) as a proxy reward function for text based RL such as Ultimatum game, matrix game or the DealOrNoDeal negotiation tasks. The main idea is to consider to take a combined prompt $\rho$ as the input of LLM, which consists of the task description, user provided example, converted outcome of an episode, and  a question asking if the outcome episode satisfies the user objective, and then use. the LLM's response as signals for reward.  
strength_and_weaknesses: **Strength**
- The idea is simple and intuitive, which is quite easy to understand. 
- The authors conducted detailed ablation study to show the effective of using LLM as Reward 


**Weakness** 

- I believe the approach can only be used for text based games, can not used for other more general tasks such as control tasks. 
- The authors only evaluate three tasks, and two of the are just 1 step tasks, and the left one is multi-step horizon based, but from the description it seems that the horizon is not that long. 
- Also, from the illustration figure, I feel the current approach may only deal with short horizons, since you will need to consider the whole trajectory as the prompt input to the LLM. 
- There is no user study. Since your tasks involves languages, I feel it should be necessary to conduct some user study with human evaluation, thus we can get some sense of what exactly happens here. 
- Also, for RL researchers, people would usually have the impression that  sparse reward would not actually work in practice, and require some more techniques to improve. Can authors explain why sparse reward provided by your approach actually work? What's the difference if we use a sparse intermediate reward function for training? Will that actually work? 
clarity,_quality,_novelty_and_reproducibility: **Originality**

The authors provide an interesting approach to solve text based RL tasks. As far as I know, this approach is new. 

**Clarity**

I feel the authors should clarify and explore the pros and cons of the approach, for both discussion and concrete experiments. It is not enough to show the limitations at the final conclusion section, since the the limitation of your approach I feel is some very fundamental problems to study and may help others to understand. 

**Quality**

Overall the paper is well written and the experiments shows the effectiveness of their approach on the limited domains. The quality can be further improved if the authors could answer my previous questions. 
summary_of_the_review: Overall I think the paper introduce an interesting approach for utilizing reward function for text based RL tasks, though the current approach is not quite general and can only applied  to short horizon based methods, and there is no user study with human evaluation to understand what is really going on here.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper "Reward Design with Language Models" proposes to study to effectiveness of pre-trained large language models for designing rewards in RL tasks (tasks implying language). It shows that, somehow unsurprisingly given recent literature, leveraging these models to drive agents with textual prompts is possible in few-shot or zero-shot settings. 
strength_and_weaknesses: Strengths: 
- Using textual prompts to drive RL agents is an emerging topic with promising outcomes

- Promising results for controling RL agents with language

Weakness: 
    
- Using pre-trained LLMs in zero-shot settings is clearly not new, many studies already exist on that topic. Not on the specific tasks authors suggest, but the transferability of their embedded knowledge with well specified prompts is well known. Prompt tuning is an emerging field following that observation. 

- Shaping rewards with LLMs is not new either, as many works recently proposed to leverage them to split difficult tasks into textual subgoals, or to drive agents by asking questions for instances, in a curriculum RL fashion. These works should at least be mentioned in the related work.   

- Writing is not always very clear. Readers have to go in many various places again and again during reading to get things together. Very difficult to understand without appendix. Many settings not sufficiently formally specified to well analyse results (e.g., the supervised learning of SL, the definition of function g, etc.). 

- Environments look quite simple (short trajectories, few actions, deterministic transitions, full observability). It would be helpful to see more classical complex tasks of RL, in environments such as TextWorld, Alfred or BabyAI for instances. 
clarity,_quality,_novelty_and_reproducibility: My major concern about this paper is its novelty. While it may be quite interesting for the community, I feel that its findings are not sufficient for the level of ICLR.   
summary_of_the_review:   - Could authors position their work w.r.t literature on prompt tuning, zero-shot learning with pre-trained LLMs (in particular, the last reference I give below looks very connected to the presented work), and also w.r.t. reward shaping with language? 
   - The global learning setting is not fully clear to me. How is defined or learned the function g that maps the textual outcome of the LLM to an int reward ? It is a language model too ? It is learned from binary true rewards ? If it is manually defined, how do authors manage variability of llm outputs ? (for expected answers such as yes or no, it is possible that the llm outputs something different right ? or do authors restrict its possible outcomes ?) 
   - In fig 8 right, the prompt finishes with "let think step by step :". So the LLM is supposed to propose a step by step forecasting of reasoning ?  
   




Multitask Prompted Training Enables Zero-Shot Task Generalization. ICLR 2022

Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652.

Ella: Exploration through learned language abstraction. In Advances in Neural Information Processing Systems (NeurIPS), 2021

EAGER: Asking and Answering Questions for Automatic Reward Shaping in Language-guided RL. CoRR abs/2206.09674 (2022)

Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691. (2022)
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes to use a pre-trained large language model to specify rewards for reinforcement learning. The model takes a description of the task and reward, and optionally a small set of demonstrations and outputs a binary reward for a new queried game state. Based on this reward, an RL agent is trained to complete the desired task and is compared to agents trained with the GT reward as well as a reward generated by training a supervised model on the task demonstrations. 
Through experiments in different domains, authors show that: 1) The proposed model can represent rewards, and train successful RL agents from even a single example, 2) LLMs can use background knowledge and predict correct rewards from commonly known objectives, even in a zero-shot setting and 3) They can model rewards in long horizon tasks, provided demonstrations and explanations.  

strength_and_weaknesses: 
**Strengths:**

- Clear and well written paper. The goal of the paper is clear and well motivated, backed by prior work evidence, the contributions of the paper are clearly stated, the method is also clearly written, even though a bit more clarity could be added into the particular example in figure 1. The experimental design, results and takeaways are clear as well.
- The approach for reward specification is flexible, allowing both example-based specification, through a description of the desired task or a combination of the 2.
- To my knowledge, this is the first approach using LLMs to specify rewards, it is thus a novel formulation and a promising one, as LLMs become a building block of applications in ML.
- Good experimental design. While the tasks are relatively simple, they represent scenarios for which we care about specifying rewards, and prove the 3 main points that authors are claiming with the proposed metthod. 
- Very interesting to see that the LLM contains concepts such as pareto-optimality. It would be great to see if the LLM can combine multiple concepts at the same time, when possible, or negate concepts. I don't think it is a requirement for this paper but I would suggest adding it as follow up work. 
- As shown in 5.2, the method is relatively robust to different prompts, as long as an explanation for the reward is given, implying that the method should not necessarily require thorough prompt design.

**Weaknesses:**

- Given that one of the motivations in this work is to provide rewards in cases where it hard to engineer them, I think that this paper would strongly benefit from having a case where the reward cannot be formally described, and measure with a human study whether people prefer policies that come from rewards generated from the LLM vs policies that come from people playing successful games according to such reward. 

- Experiments:
    - I am missing more results to understand the role of explanations vs few-shot. In experiment 1, what is the performance if we use a LLM with only one example and no explanation? Does the high performance come solely from explanation or it is also because the model has better capacity for few-short learning. On that line, what is the importance of \rho1? Does the LLM perform the task worse when given few shot examples and no description of the task?

    - I am missing more analysis and context on some of the results. In 4.2.1 what is the performance of a random baseline? Is there bias towards a given user objective when it is not given, or in the failure cases, or are the best case solutions random?



- While this is also the case of works like Huang et al. 2022, a limitation of this approach is that it requires states and actions in the environment to be represented as tokens that the LLM has been trained on. This limitation is transparently addressed in the last section though, and since many tasks can actually be specified via language, I still think this work is a valuable contribution. A minor suggestion would be that besides commenting on multi-modal states, authors also mentioned non-token actions. This may be important for continuous control for instance, where we may not want aggressive motion in a robot arm.

- See my comments on clarity.
clarity,_quality,_novelty_and_reproducibility: **Clarity:**
- Generally a very clear and well written paper, as stated in the strength sections. I like how each game is designed to test one of the hypothesis and the summary at the end of each experiment with an explanation of the findings.
- The example in Figure 1 could be improved. For instance you could replace item0... for books/hats/balls, and specify the total number, or how many go for Alice and How many for Bob.
- The sentence "we assume access to the true reward by constructing user reward functions that humans have been shown to have inspired by prior work", is a bit confusing. Are the rewards based on prior work? While this is further clarified in Sec 4.1 GT User objectives, an example of such reward here would be helpful.

- Clarification Question: In 4.2.1 the authors seem to be describing recall on correct outcomes (identifying correct outcome) but I imagine that they are actually measuring accuracy of outcomes? If so they should modify the text.
- Comment: What is the role of the description \rho1, when prompts in \rho2 are given.
- Minor:
    - Moving Figure 2 to page 5 would make the results a bit easier to read.
    - Because SL actually does not have the explanation, I would modify the title in Figure 2, and add explanation in the Ours bar.

**Novelty**:
- To my knowledge, this is the first approach using LLMs to specify rewards, it is thus a novel formulation and a promising one, as LLMs become a building block of applications in ML.

**Reproducibility**:
The domains and rewards are thoroughly explained in the appendix. More description of the process for selecting the right prompts, and especially the natural language explanations for the reward would be appreciated. It would also be valuable if authors released the dataset used for prompting the language model. As long as `text-davinci-002` remains available the results should be reproducible but given the importance of good explanations as per figure 6, it it important that authors share the dataset of explanations to ensure reproducibility.

summary_of_the_review: As stated in my strengths section, the paper addresses a very important problem with a solution that is simple and relatively general-purpose (safe for some of the cases mentioned in the limitations). The contributions are very clear and the design of the games and experiments show that using language models to specify rewards offers advantages in multiple ways. Moreover, as LLMs become a more pervasive building block in decision-making models, the approach has the potential to be extended and generalized as more research is done on this end (for instance through multi-modal foundation models to address more general states). As such, I think this is a valuable paper for this conference. The reason why I cannot give a strong accept is because I am missing experiments at the core of this paper goal: providing rewards where there is no way to specify them, evaluating them using human ratings.  Moreover, I would really like to clarify some of the details in the experimental results, as stated in the weakness section, experiments. This would help me convince that the proposed approach offers an effective way to specify rewards.

[EDIT]: The authors have addressed all my concerns, clarifying experimental design and metrics, adding thorough ablation studies and a human experiment that proves the effectiveness of the approach under real scenarios with hard to specify rewards. I recommend this paper for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes using LLMs to understand underlying reward functions given a few language examples, then demonstrate that RL agents trained with this LLM-provided sparse reward function nearly match those trained with true underlying reward functions on some simple tasks.
strength_and_weaknesses: ## Strengths

****************************************Writing and Clarity:****************************************

- The writing is very intuitive and straightforward; the paper is easy to read.

******************************Framing/Method:******************************

- This is a great execution of a simple idea (simple being a good thing here). The authors tested reward design with LLM on a variety of environments and train an RL-agent in-tandem to prove that not only are the labels accurate with the LLM, but these labels allow RL agents to learn well on tasks compared to the true underlying reward functions.

************************Experiments:************************

- The authors have comprehensive experiments demonstrating one of the claims of the paper: that LLM’s can learn from far fewer examples due to all the data they’re trained on.

## Weaknesses

**Clarity:** 

- Some details regarding the games seem to be missing, for example what is the fixed Bob agent in DealOrNoDeal?

**Experiments:** 

- Missing LLM size comparison: While using GPT-3 helps demonstrates the points the authors aim to make about the ability of LLMs to act as reward functions, GPT-3 `text-davinci-002` and even open source alternatives of the same size are prohibitive to most researchers either due to cost (GPT) or compute limitations (not enough VRAM to run inference). This also presents a problem because the experiments don’t really show that LLMs in general can help design reward functions, instead it shows that a 176B param GPT-3 can help design reward functions. Can the authors experiment with smaller alternatives? For example, at least GPT-3 `curie` (a 1-line code change for the authors), and/or the OPT family of open-sourced models (e.g. 350m, 1.3b, 6.7b - compute-matched with `curie`, 30b).

******************************************Missing related work:******************************************

- Using Natural Language to aid with reward design is a related subfield with some works, here is one for example: [https://www.ijcai.org/proceedings/2019/0331.pdf](https://www.ijcai.org/proceedings/2019/0331.pdf). This is a little bit related to the framing, as the authors should probably add a related works section explaining language as a proxy for rewards.

****************************Minor details:****************************

- With temperature 0 isn’t GPT-3 just producing the max-likelihood token? In that case I don’t believe top-p changes anything about the model’s outputs and therefore is irrelevant (Section 4 2nd paragraph).
- Unfinished Appendix: A.5 and A.6 are not used and don’t have any text.
- Typos:
    - “They require far few examples” > “far fewer”
    - “on the agreed upon the agreed upon split”
clarity,_quality,_novelty_and_reproducibility: Clarity: Very good
Quality: The paper is above-average to high quality.
Novelty: I believe this idea is novel. 
Reproducibility: Some RL training details are sparse, but LLM prompts are given. Somewhat reproducible.
summary_of_the_review: I like this paper, and it demonstrates a well-written execution of a simple idea. The main issue that I have with this paper is the lack of experiments on smaller or more accessible LLMs. With that said, I still believe this paper should be accepted.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper proposes to use a large language model (LLM) to design rewards to train reinforcement learning (RL) agents. It converts both the desired behavior and the agent's behavior into textual prompts, which are evaluated under the LLM to output a reward signal. The reward signal is then used to optimize the RL agent's behavior. The proposed approach can work well even only given a single or a few demonstrations of the desired behaviors. Experiments in several different tasks, including long-horizon ones, show that the proposed approach can train RL agents aligned with the user's objectives and outperforms RL agents trained with supervisedly-learned reward functions. 

Generally speaking, the reviewers consider the proposed approach to be simple, intuitive, straightforward, and well-motivated, and the presents experimental results on controlling RL agents with LLM to be promising. 

There are, however, some disagreements on novelty, clarity, practicality, and performance limitation.  

1) Novelty: Reviewer UKq1 does not consider the proposed approach to be that novel, as "somehow unsurprisingly given recent literature, leveraging these models to drive agents with textual prompts is possible in few-shot or zero-shot settings," and he/she feels that "its findings are not sufficient for the level of ICLR." 

2). Clarity: the paper could be difficult to read for people who are not familiar with the considered tasks and how to convert behaviors into textual prompts. 

3): Practicality: the proposed approach might not generalize well into the settings where behaviors cannot be easily concerted into textual prompts; it could be very difficult to appropriately design the g and f required in the proposed algorithm. 

4) Performance limitation: the zero-shot and few-short ability of an LLM is a limiting factor of the performance of the RL agent; if considering LLM as the teacher, the best the RL agent can do is probably performing as good as the teacher; it is unclear whether the RL agent can outperform the teacher by combing the LLM produced reward and its own policy learning. 



justification_for_why_not_higher_score: Given the remaining concerns of some reviewers and the AC, as listed in the meta review, it is more appropriate to accept the paper as a Poster.
justification_for_why_not_lower_score: Generally speaking, the reviewers consider the proposed approach to be simple, intuitive, straightforward, and well-motivated, and the presents experimental results on controlling RL agents with LLM to be promising.
Decision: Accept: poster