On the Modeling Capabilities of Large Language Models for Sequential Decision Making

## Abstract

Abstract Large pretrained models are showing increasingly better performance in reasoning and planning tasks across different modalities, opening the possibility to leverage them for complex sequential decision making problems. In this paper, we investigate the capabilities of Large Language Models (LLMs) for reinforcement learning (RL) across a diversity of interactive domains. We evaluate their ability to produce decision-making policies, either directly, by generating actions, or indirectly, by first generating reward models to train an agent with RL. Our results show that, even without task-specific fine-tuning, LLMs excel at reward modeling. In particular, crafting rewards through artificial intelligence (AI) feedback yields the most generally applicable approach and can enhance performance by improving credit assignment and exploration. Finally, in environments with unfamiliar dynamics, we explore how fine-tuning LLMs with synthetic data can significantly improve their reward modeling capabilities while mitigating catastrophic forgetting, further broadening their utility in sequential decision-making tasks.

## 1 Introduction

Large Language Models (LLMs) are generative models of natural language that
can produce accurate general and domain-specific knowledge (Singhal et al., ; Imani et al., ; Manigrasso et al., ; Liu et al., ), reason over long textual contexts (Reid et al., ), and generalize zero-shot (Kojima et al., ).
These capabilities suggest that LLMs might be well-suited for complex sequential decision-making problems, such as in embodied settings where an agent acts in an environment.
Recent research has begun exploring this potential, investigating how LLMs can serve as sources of intrinsic motivation (Wang et al., ; Klissarov et al., ), demonstrating world modeling capabilities (Lin et al., ; Liu et al., ), and for acting and/or planning directly in an environment (Wang et al., ; Padalkar et al., ; Zhang et al., ).

However, as the predominant paradigm for training LLMs is not inherently aligned with the challenges of sequential decision-making problems, such as active exploration, it is not obvious how to best bridge their capabilities to tackle such challenges in a general manner.
We study this problem through the lens of reinforcement learning (RL, Sutton & Barto, ), which formalizes how an agent interacts with an environment, receiving scalar rewards for each of its actions over a trajectory.
We examine the capabilities of LLMs to solve RL tasks by comparing how they model policies 1) directly by generating action tokens, to 2) indirectly through a reward model derived from the LLM to be used within an RL algorithm.
We perform a comprehensive evaluation on a diverse set of domains, including MiniWob (Liu et al., ), NetHack (Küttler et al., ), and Wordle (Lokshtanov & Subercaseaux, ), and MetaWorld (Yu et al., ). The environments we study present a variety of challenges, such as different action space granularities, observation modalities ranging from natural language to pixel data, and varying horizon lengths.

We first consider the off-the-shelf capabilities of LLMs for decision-making without updating them through additional gradient updates coming from the RL task.
We find that indirectly modeling policies by first extracting knowledge from LLMs in the form of a Bradley-Terry model (Bradley & Terry, ; Christiano et al., ) provides the best and most consistent performance across the environments we study.
We empirically analyze the various benefits, and limitations, provided by this approach, showing that it improves on long-standing challenges in RL problems, such as credit assignment and exploration.

Finally, while LLMs possess knowledge useful for many decision making tasks of interest, domains with complex or unfamiliar dynamics can significantly restrict their broader utility.
We explore how fine-tuning an LLM with domain-specific data can bridge this knowledge gap and study the effect of this procedure on the LLM’s previous knowledge, as measured through success on datasets like POPE (Yifan Li & Wen, ), GQA (Hudson & Manning, ), AI2D (Kembhavi et al., ) and MMMU (Yue et al., ).
Our investigation reveals that fine-tuning for indirect policy modeling mitigates catastrophic forgetting more effectively than direct policy modeling, offering a broadly applicable strategy for leveraging LLMs across diverse sequential decision-making tasks.

## 2 Using Language Models to Solve RL Tasks

We first introduce the types of RL problems as well as formalize the methodologies for using LLMs for RL tasks used in this work.

#### Reinforcement Learning.

An RL task can be defined through a Markov Decision Process (MDP, Puterman, ), which is composed of a state space $\mathcal{S}$, an action space $\mathcal{A}$, a transition function $p:\mathcal{S}\times\mathcal{A}\to\Delta(\mathcal{S})$ which describes the forward dynamics of the system, a reward function $r:\mathcal{S}\times\mathcal{A}\to\mathbb{R}$ and a discount factor $\gamma\in[0,1]$. Since it is often the case that the state is only partially observable, we also assume the environment emits an observation $o_{t}\sim p_{\mathcal{O}}:\mathcal{S}\to\Delta(\mathcal{O})$ from observation space $\mathcal{O}$.
A policy, or *actor*, is a probability distribution $\pi:\mathcal{S}\to\Delta(\mathcal{A})$ which describes the action to be taken at every step.
The objective of a rational actor is to maximize the expected cumulative rewards over horizon $H>0$,

$$ $\max_{\pi}\mathbb{E}[\sum_{t=0}^{H}\gamma^{t}r(s_{t},\pi(s_{t}))|s_{0}]=\max_{ \pi}\mathbb{E}_{s_{0}}[V^{\pi}(s_{0})],$ (1) $$

where the value function, $V^{\pi}(s)$, represents the expected discounted sum of rewards over the entire trajectory, re-weighted by the environment’s dynamics model, $p$, and the actor’s policy, $\pi$.

#### Large Language Models.

An LLM is a generative model of discrete random variables (i.e. tokens) conditioned on a history (i.e. context). The LLM models the data distribution autoregressively:

$$ $p(x_{t+1}|x_{1},..,x_{t})=\prod_{t^{\prime}=1}^{t}p(x_{t^{\prime}}|x_{<t^{ \prime}})\\ =\texttt{LLM}(x_{<t},l)$ (2) $$

where $x\in\mathcal{X}$ are token variables taken from a valid vocabulary. The suitability of LLMs for solving RL tasks without additional fine-tuning primarily hinges on the hypothesis that LLMs contain information – i.e., *knowledge* – about the underlying MDP, for instance, through the policy or reward function.
*How* that information is extracted depends on the data the LLM was trained on, the ability of the practitioner to properly prompt the model and interpret its responses to solve decision-making tasks.

### 2.1 Prompting

In this section, we describe the inputs, or *prompts*, to the LLM used in this work which allow to change the LLM’s output distribution to be useful for solving RL tasks.
All prompts in this work use 1) task specification using natural language as input to provide information about the MDP to the LLM as context and 2) episode history in order to address issues of partial-observability in some environments (similar to the Act-only baseline prompt found in Yao et al., ). We additionally use the following set of techniques,

- •
Chain of Thought. By prompting the LLM to provide a step-by-step reasoning process for its output, rather than just the final answer, we can help surface its internal decision-making and improve the resulting performance (Wei et al., ).
- •
In-Context Learning. To enhance the LLM’s ability to solve the task, example solutions (e.g., from expert policies) are provided for in-context learning (Brown et al., ), where solutions contain sequences of a combination of states, actions, and rewards.
- •
Self-Refinement. To further refine its output, the LLM is prompted to provide recursive criticism and improvement from its generated outputs. This general strategy knows many variants, such as feedback from an environment (Yao et al., ), self-critique (Zelikman et al., ), or self-reflection (Shinn et al., ). In this work, we use Recursive Criticism and Improvement (RCI, Kim et al., ) for its state-of-the-art performance on web agent domains and general applicability.
In its original form, the LLM is given a task description and generates a high-level plan. This plan is used along with the task description and current state to refine an action so that it is grounded in the current observation and the action space.

### 2.2 Policy Modeling Using LLMs

As shown in Equation 1, the goal of a decision making agent is to learn a high performing policy $\pi$. This can be done either by maximizing the expected cumulative rewards and directly modeling the policy parameters (Sutton et al., ; Kakade & Langford, ). Equivalently, this can be done indirectly by first modeling the parameters of the value function and applying a greedy operator, such as in Q-Learning (Watkins & Dayan, ). A similar separation between direct and indirect approaches can be useful to study the capabilities of LLMs to model RL policies.

#### Direct Policy Modeling.

The most straightforward way to obtain a policy using LLMs is for the LLM to generate tokens that will be directly interpreted as actions from the environment, $a\in\mathcal{A}$ (Yao et al., ; Shinn et al., ; Kim et al., ). To ensure the outputted actions adhere to the environment’s action set, the LLM output tokens can be projected back onto $\mathcal{A}$ using projection operator $\text{proj}(\cdot,\mathcal{A})$ (e.g., see Huang et al., ; Kim et al., , for examples of projection operators). A variety of prompting techniques can be combined to increase the ability of the LLM to act, without task-specific fine-tuning, as a policy, which we detail in Section 2.1. This direct policy method will be referred to in our experiments as LLM Policy.

#### Indirect Policy Modeling.

On the other hand, we can prompt the LLM to output tokens representing intermediate quantities that will then be used to learn a policy. For example, one can model the forward dynamics of the environment for planning (Liu et al., ), or an affordance model for action selection (Mullen Jr & Manocha, ).
In this work, we focus on the case where these intermediate quantities will be used to generate rewards – i.e., a reward model – which will then be maximized by an off-the-shelf RL policy.
In Section 2.3, we enumerate the different approaches for modeling reward functions with LLMs covered in our work.
(^†^†It is important to note that there exists many more ways in which we could indirectly model the policy. In Appendix A.4, we present in detail these possibilities and, in Figure 2(b), provide initial investigations that showcase their potential and limitations.)

In direct policy modeling experiments (LLM Policy), we found combining all of the prompting techniques in Section 2.1 to work the best, while for indirect modeling methods through reward we relied only on chain-of-thought prompting. Additional details, such specific prompt details and ablations on these choices are presented in the Appendix A.3.

### 2.3 Indirectly Modeling Policies through Reward Models

We consider a diversity of methods for modeling reward functions using LLMs, with a particular attention to methods that are applicable to a diversity of environments and modalities. We study the following set,

- •
Direct Scalar.
The LLM generates tokens that directly encode the reward (e.g., as a float or integer) given an observation (or a sequence of observations and actions). This reward is then given to the RL agent.
- •
AI Feedback  (Lee et al., ; Klissarov et al., ).
Ask the LLM to express a preference $y=\{1,2,\varnothing\}$ between two observations, $o_{1}$ and $o_{2}$, for the one showing the most progress towards a certain goal, or no preference if both observations are equally good. These labels can then be collected as a dataset of observation-preference tuples $\mathcal{D}_{\text{pref}}=\{(o_{1}^{(i)},o_{2}^{(i)},y^{(i)})\}_{i=1}^{M}$, which are then used to train a reward function modeled as,
$\begin{split}r_{\theta}=\operatorname*{arg\,min}_{\theta}\mathbb{E}_{(o_{1},o_
{2},y)\sim\mathcal{D}_{\text{pref}}}\bigg{[}&\mathbb{I}[y=1]\log P_{\theta}[o_
{1}\succ o_{2}]+\mathbb{I}[y=2]P_{\theta}[o_{2}\succ o_{1}]\\
&+\frac{1}{2}\mathbb{I}[y=\varnothing]\log\big{(}P_{\theta}[o_{1}\succ o_{2}]P
[o_{2}\succ o_{1}]\big{)}\bigg{]}\end{split}$
(3)
where $P_{\theta}[o_{1}\succ o_{2}]=\frac{e^{r_{\theta}(o_{1})}}{e^{r_{\theta}(o_{1})
}+e^{r_{\theta}(o_{2})}}$ the probability of preferring an observation to another, referred to as the Bradley-Terry model for preference learning (Bradley & Terry, ). The minimization of this equation is commonly done through binary cross-entropy.
- •
Reward as Code (Yu et al., ; Ma et al., ). Prompt the LLM to write code that will take as input a subset of symbolic features from the environment observations and will produce a scalar output representing the reward. When symbolic features are not available, these are constructed as in Venuto et al. ().
- •
Embedding-based (Rocamonde et al., ; Du et al., ). Instead of querying language tokens from the LLM, we can instead, for a given input, leverage the information encoded in its latent represention, or embeddings. These embeddings are used to calculate the cosine similarity with respect to the embeddings of natural language specification of a goal or a behaviour. The resulting similarity value is given as a reward to the agent.

Additional details, such specific prompts, are presented in the Appendix A.2.

## 3 Performance of Indirect and Direct Policy Models

Figure: Figure 1: AI feedback as the highest performance across different reward models derived from LLMs tested. AI feedback, which is a preference-based method for deriving a reward model from an LLM generally outperforms other methods.
Refer to caption: x1.png

Due to fundamentally different challenges between direct and indirect policy modeling approaches, conducting a fair comparison requires care.
For example, using the LLM directly as a policy requires grounding its outputs in the action space defined by the environment (Ahn et al., ; Huang et al., ). As the action space can vary
significantly between environments and attempting to solve this problem adds additional algorithm- or domain-specific complexities (e.g. by crafting skills, see Ahn et al. (); Wang et al. ()), we fix our experimental setting to the following

- 1.
Atomic actions. We only study approaches which can directly interface with the action space supported in the environment.
In other words, the action space is at least a subspace of the space of language generated by the LLM.
This allows for a more direct comparison across a variety of domains and study the relationship between an LLM’s knowledge and the fixed action space defined by the environment.
- 2.
No finetuning. In most of the paper we assume that LLMs are used without any gradient updates, i.e. without fine-tuning from the RL task, and evaluate their off-the-shelf capabilities.
In Section 5, we perform a preliminary study on the trade-offs between fine-tuning for direct and indirect policy modeling.

Figure: (a)
Refer to caption: x2.png

We investigate four separate domains, where each domain aims to highlight a specific capability of LLMs: 1) MiniWob-Hard, a subset of hard tasks from the full MiniWob suite, tests web interaction in observation/action spaces close to natural language, 2) Wordle measures reasoning and planning capabilities, 3) NetHack presents the difficulty of exploring open-ended environments under partial observability, long horizons and procedural scenarios, and 4) MetaWorld assesses the ability to control low-level, high-frequency actions in continuous space.
We provide a detailed description of each domain in Appendix A.1.

Direct policy modeling is done by querying the closed source GPT-4o model, whereas indirect policy modeling is done through the open source models of Llama 3 (Dubey et al., ), when environment observations consist of text, and PaliGemma (Beyer et al., ), when environment observation consist of pixel images. All results are averaged over 10 seeds with error bars indicating the standard error.

#### Indirect policy modeling through rewards.

We first present a comparison of the various indirect policy modeling approaches discussed in Section 2.3. In these experiments, the LLM generates a reward function which will be given to a RL agent for optimization, without access to any rewards coming from the environment. When learning policies through RL we do not perform any hyperparameter search and simply borrow the existing empirical setup for each domain, as detailed in Appendix A.1.

In Figure 1, we present the performance across domains as measured by the average success rate on all domains,
except for NetHack, where performance (the in-game score) is normalized by the highest recorded value.
Results show that AI feedback is the only method that successfully crafts rewards across all environments and
modalities (^†^†In Appendix A.6, we verify that AI feedback yields policies with performance on par with those optimized using human-designed environment rewards.).
On easier domains such as MiniWob-Hard, which consists of short episodes and limited scope of variations, the Direct Scalar method performs nearly as well as AI feedback.
However, the disparity between methods is much more pronounced on harder, open-ended tasks such as NetHack.
Out of all the methods, Embedding-based leads to the lowest performance.
Finally, the effectiveness of Reward as Code appears to be highly contingent on the availability of symbolic features for code processing.
In Appendix A.5, we further examine the assumptions—such
as access to functional knowledge of the environment—under which Reward as Code can achieve performance comparable to AI feedback

#### Direct vs indirect policy modeling.

We now compare the direct policy modeling method, LLM Policy, to the best performing indirect modeling method, AI feedback, reporting performance across the same set of domains.
Results in Figure 2(a) show that, despite the more complex prompting strategies and the use of a more capable closed source model, LLM Policy is unable to perform well in most environments, with the exception of MiniWob-Hard, where the performance is on-par with AI feedback.

A question emerging from these results is: what factors cause this significant performance disparity between direct and indirect policy models? One possible explanation is that LLMs, when directly queried for actions in an unfamiliar environment, may struggle to understand its dynamics (e.g., the transition function and action space). To test this hypothesis, we conduct the following experiment. We prompt the LLM to select between 1) a pair of candidate next observations given the current observation and action (probing knowledge of $p(o_{t+1}|a_{t},o_{\leq t})$), or 2) a pair of candidate actions given the next observation and current observation (probing knowledge of $p(a_{t}|o_{t+1},o_{\leq t})$). In each case, the pair contains the ground-truth and random sample. In this experiment, a $50\%$ accuracy corresponds to a random guess.

Results presented in Figure 2(b) show that the LLM performs relatively poorly on both of these tasks, indicating limited understanding of both the action space and the environment dynamics. This can potentially explain the limited performance of the LLM Policy approach on MiniWob-Hard, NetHack, and MetaWorld, while results on Wordle suggest that additional contributing factors are at play.

## 4 Analysis of AI Feedback for RL

Our results so far suggest that, without additional fine-tuning,
indirectly modeling policies by constructing reward functions through AI feedback is the most effective approach across the range of environments and modalities we studied.
In this section, we examine how rewards shaped by this method can assist RL agents in addressing core decision-making challenges, such as credit assignment and exploration. Through this analysis, we also emphasize the ways in which reward misspecification can unintentionally arise and severely impair performance.

Figure: Figure 3: Rewards learned through AI Feedback distribute rewards to key timesteps. By doing so, the problem of credit assignment, or learning from delayed rewards, is significantly reduced. Such distribution effectively shortens the horizon over which the RL algorithm must propagate credit through its update rule.
Refer to caption: x4.png

### 4.1 Credit Assignment

AI feedback-based rewards depend on the prompt used to capture preferences. In the experiments conducted so far, these prompts were designed to elicit preferences by emphasizing states that contribute to task progress (see prompts Appendix A.2). Additionally, a key aspect of our methodology involved presenting the LLM with observations sampled randomly within trajectories. This enabled querying preference for any observation in the environment, rather limiting the focus to final states - a distinction also known as process-based and outcome-based reward models (Uesato et al., ; Lightman et al., ). What are the resulting characteristics of the reward model under such choices?

#### Qualitative experiment

In Figure 3, we present the output of the AI feedback-based reward model over each timestep of an episode within a simple grid world environment. This task includes an agent, a key, a door, and a goal (Chevalier-Boisvert et al., ).
We notice that this reward model naturally captures the fact that picking up the key, as well as opening the locked door, are important steps towards the goal.
By propagating credit over such key moments in a trajectory, the LLM effectively shortens the horizon over which the RL algorithm must assign credit through temporal difference learning (Sutton & Barto, ).
This is manifested in Figure 3 where the agent learning through AI feedback reaches a high success rate in a fraction of the timesteps required by a similar agent learning from the environment feedback (which in this case is sparse reward of +1 for reaching the goal).

Figure: (a) Wordle
Refer to caption: x5.png

#### Quantitative experiment

In Figure 4, we present the correlation between the reward model derived from AI feedback and the value function of an RL agent across various levels of policy optimality. We observe that AI feedback generates reward functions with a stronger correlation to value functions obtained later in the training process compared to those from earlier stages. Additionally, this correlation is higher than that observed with the environment reward.
In the Wordle game, we generate, in code, a near-optimal policy and estimate its value function using Monte Carlo. We then compare it to the LLM-derived reward function find an almost perfect correlation.
These findings suggest that the reward models derived from AI feedback inherently encode aspects of high-quality value functions, which, when used as rewards for the RL agent, can substantially simplify the credit assignment process.
In Appendix A.7, we provide additional insights from the lens of heuristic-guided reinforcement learning (Cheng et al., ).

### 4.2 Exploration

Figure: Figure 5: By changing the prompt, LLMs can be steered to provide feedback that promotes exploration on NetHack. Additionally, to avoid degenerate solutions, preferences should be elicited in an online fashion and the reward function be non-Markovian.
Refer to caption: x8.png

In the previous section, we investigated how our standard prompting strategy can ease the problem of credit assignment in downstream RL tasks. This outcome stemmed from the specific preferences we requested from the LLM, that is, promoting task progress. However, to address different RL objectives, in particular the one of exploration, we may need to elicit alternative preferences.

Previously, Klissarov et al. () employed AI feedback to design an effective reward function for an agent operating in the open-ended environment of NetHack.
However, before applying this reward to the RL agent, the authors implemented the following transformation:

$$ $r(o_{t})\propto r_{AIF}(o_{t})/N(o_{t})^{\beta},$ (4) $$

where $r_{AIF}$ is the reward model obtained from AI feedback, $N(o_{t})$ denotes the number of times a particular observation $o_{t}$ was seen in an episode, and $\beta$ is a positive real-valued coefficient set to $3$.
The counting term was added to encourage exploration (Henaff et al., ), which is a key difficulty in NetHack. However, instantiating such a counting function proves difficult in many practical settings (Bellemare et al., ). Given the flexibility of natural language, can we alleviate the need for such a term and integrate the notion of exploration in the prompt itself?

In Figure 5, we demonstrate that this is indeed possible, leading to performance comparable when using count-based exploration by directly modifying the prompt used for preference elicitation. Specifically, when querying the LLM for preferences, we present it with a pair of sequences of observations (rather than a single observation) which provides crucial context. The prompt was also modified to steer the LLM towards avoiding low entropy sequences, i.e. sequences with repetitions (see Appendix A.2).

Our findings reveal two potential failure modes: the offline nature of the preference elicitation method and the assumption of a Markovian reward model.
Previous research has demonstrated that online preference querying can outperform offline methods when aligning LLMs (Bai et al., ; Touvron et al., ). In our experiments, offline elicitation led to a performance collapse, likely due to frequent RL policy updates during online learning.
Additionally, assuming a Markov reward model—where the current observation fully determines the reward—can lead to an equally poor performance, as complex tasks often require historical context beyond immediate observations (see Appendix A.8 for a full breakdown).

## 5 Beyond Zero-Shot Reward Modeling

So far, we have explored the ability of LLMs to model policies, directly and indirectly, without any fine-tuning.
However, in many cases the prior knowledge encoded in LLM might not contain the necessary information to do so successfully. In such instances, fine-tuning becomes an effective method for incorporating task-specific knowledge into the model.

Figure: (a) Fine-tuning for AI feedback
Refer to caption: x9.png

We consider the sweep-into task from MetaWorld, where AI feedback rewards lead to a success rate of only $15\%$. When measuring the perplexity score of the PaliGemma model on captions describing the pixel observations from the task, we obtain a value of 16.03. Both of these results indicate poor understanding and the necessity to adapt the model.

We therefore fine-tune PaliGemma on image-caption pairs annotated by GPT-4o and trained the model to predict the caption for a given image. Figure 6(a) shows significant gains in downstream RL performance after only a few fine-tuning epochs and as few as approximately $100$ image-caption pairs.
Moreover, Figure 6(a) shows how this procedure only marginally decreases performance of the LLM on the standard multi-modal reasoning benchmarks, such as POPE (Yifan Li & Wen, ), GQA (Hudson & Manning, ), AI2D (Kembhavi et al., ) and MMMU (Yue et al., ).
Surprisingly, performance on the AI2D benchmark *improves* as the number of task-specific fine-tuning epochs increases.

We contrast these findings with Figure 6(b), where we fine-tune PaliGemma with behaviour cloning on expert data on the same MetaWorld task. Similarly to RT-2 (Brohan et al., ), we overwrite the least frequent tokens with residual VQ-VAE codebooks (Szot et al., ). In this case, any significant increase of RL performance comes at the cost of catastrophically forgetting all previous knowledge. These results hint at an important trade-off: if preserving prior language reasoning knowledge is important, fine-tuning for AI feedback offers a viable approach. However, if maximizing downstream RL performance is the sole objective, directly fine-tuning for action selection can be more effective.

## 6 Discussion

In this paper, we explored two distinct approaches to leveraging LLMs for solving RL tasks: 1) directly, by modeling policies and 2) indirectly, by modeling rewards to be leveraged within a policy learning algorithm. Our results indicate that, without task-specific fine-tuning, current LLMs only show limited decision-making capabilities when directly generating actions. However, despite this limitation, LLMs are capable zero-shot reward modelers. In particular, when eliciting preferences to define rewards through the Bradley-Terry model, LLMs show strong performance across a wide range of domains presenting various challenges.

In cases where an LLM’s prior knowledge is not enough to obtain useful reward functions, we also investigated fine-tuning with task-specific data to bridge this gap.
Notably, fine-tuning to enhance reward modeling capabilities helps mitigate catastrophic forgetting, which is a crucial consideration for preserving the LLM’s general-purpose abilities
Maintaining these capabilities is essential for broad applicability to sequential decision-making tasks, including out-of-distribution tasks, and for supporting continued natural language interaction with users.

The reward modeling capabilities presented in this work offer potential solutions to challenges in RL.
First and foremost, LLM-derived reward models alleviate the need for human-designed reward functions, which are often complex and costly to develop.
Second, our empirical analysis reveals that AI-feedback based rewards produce dense functions which correlate positively with high-quality value functions.
Such reward functions can significantly reduce the difficulty of assigning credit by redistributing rewards across different steps within a trajectory.
Finally, distilling knowledge from LLMs into reward models opens new possibilities for applying RL in environments where simulators or symbolic features are unavailable—such as embodied AI agents interacting with humans.

Some notable limitations and caveats exist. For example, interacting with LLMs through natural language requires experimenting with various prompting techniques and specifications. However, this flexibility also enables the shaping of reward functions to incorporate valuable strategies (Knox et al., ), such as promoting exploration, which can further enhance the performance of RL agents.

## Appendix A Appendix

### A.1 Environment Details

In our experiments, we investigate tasks from four different domains: MiniWob (Liu et al., ), NetHack (Küttler et al., ), and Wordle (Lokshtanov & Subercaseaux, ), and MetaWorld (Yu et al., ). The observation space for all these environments is text, except fro MetaWorld which consists of RGB pixels.

In the MiniWob domain, we sample the subset of the five tasks on which state-of-the-art results are low. Specifically, we carry experiments on: click-tab-2-hard, click-checkboxes-soft, count-shape, tic-tac-toe and use-autocomplete. To learn RL policies from LLM-based rewards, we leverage the experimental setup of Shaw et al. (). In NetHack, we use the same environment and the same algorithmic setup as in Klissarov et al. (). In Wordle, we build on the code made available by Snell et al. () and use their proposed subset of 200 words from the official list of the game. Finally, in MetaWorld we study the same subset of environments presented in (Wang et al., ) consisting of drawer-open-v2, soccer-v2 and sweep-into-v2. Across all experiments where RL policies are learned, we use the original hyperparameter values defined in the respective experimental setups we are building upon.

### A.2 Details on Indirect Policy Modeling Through LLM-based Rewards

We use the following prompt templates to query the agent for AI feedback, Scalar Reward and Reward as Code across various environments. For the Embedding-based approach, we use calculate the cosine similarity between the representation, provided by a BERT (Devlin et al., ) sentence encoder (specifically the same paraphrase-MiniLM-L3-v2 model) when environments are text-based, and otherwise we use the CLIP encoder (Radford et al., ). The similarity is measured between the current observation and the same goal description contained in the each of the following prompts given for the other baselines.

Figure: List of Prompts 1

Figure: List of Prompts 2

Figure: List of Prompts 3

Figure: List of Prompts 4

Figure: List of Prompts 5

Figure: List of Prompts 6

Figure: List of Prompts 7

Figure: List of Prompts 8

Figure: List of Prompts 9

Figure: List of Prompts 10

Figure: List of Prompts 11

Figure: List of Prompts 12

### A.3 Details on Direct Policy Modeling

We present the exact prompts used to query GPT-4o for each of the domains we have considered. These are presented through Prompt 13, 15, 14 and 16.

Figure: Figure 7: Ablation on the set of prompting techniques used for direct policy modeling. The reported performance is averaged over all domains and tasks.
Refer to caption: x11.png

Additionally, in Figure 7, we ablate the prompting techniques used in our direct policy modeling approach. Results show that a combination of all prompting techniques presened in Section 2.1 works best.

Figure: List of Prompts 13

Figure: List of Prompts 14

Figure: List of Prompts 15

Figure: List of Prompts 16

### A.4 Additional Indirect Policy Modeling Methods

There are a number of other prompting methods for extracting information or *knowledge* from an LLM that may be relevant to solving RL tasks.

- •
Direct State Generation. The model generates tokens that will represent next states (or other-future-time states). This is similar to world modeling. The next state prediction can be conditioned on an action, or marginalized over a policy distribution.
- •
Action Preference. Ask the LLM to select, among two choices, the most likely action given previous and future observations.
- •
State Preference. Ask the LLM to select, among two choices, the most likely next state or observation conditioned on prior history and/or actions.

Many of the above could theoretically be used to construct a policy, yet a full implementation is out of scope from this paper due to the lack of available code-bases to build upon and we do not seek to build new algorithms from scratch. However, in Figure 2(b) we perform investigations into the capabilities of LLMs to perform Action Preference and State Preference. The results show that current LLMs struggle to achieve strong performance on any of these tasks. Additionally, in Table 1, we report the accuracy with which LLMs directly predicts the next observation (Direct State Generation), providing a probe into their direct world modeling capabilities.
Results show limited performance, except on MiniWob-Hard tasks, which are fully observable and encode deterministic transitions.

**Table 1: LLMs struggle to predict the next observation. We show the decreasing accuracy of the LLM to predict the next observation with increasing task complexity. LLMs are unable to generate pixel observations, which are used in MetaWorld.**
|  | Accuracy |
| --- | --- |
| MiniWob-Hard | $65\pm 11.4\%$ |
| Wordle | $28\pm 8.3\%$ |
| NetHack | $0.0\pm 0.0\%$ |
| MetaWorld | N/A |

### A.5 Ablating Reward as Code

**Table 2: AI Feedback performs on par with Reward as Code, without proprioceptive observations or expert demonstrations. To match AI Feedback performance on Metaworld, Reward as Code requires GPT-4o level knowledge, augmented with either in-context expert demonstrations or proprioceptive observations.**
| Reward as Code - RGB Observations |  |
| --- | --- |
| GPT-4o |  |
| w/o expert demonstration | $0\%\pm 1\%$ |
| with expert demonstration | $79\%\pm 7\%$ |
| Reward as Code - Proprioceptive Observations |  |
| Llama 3 70B |  |
| w/o background functional knowledge | $0\%\pm 1\%$ |
| with background functional knowledge | $10\%\pm 3\%$ |
| GPT-4o |  |
| w/o background functional knowledge | $5\%\pm 3\%$ |
| with background functional knowledge | $76\%\pm 6\%$ |
| AI Feedback - RGB Observations |  |
| PaliGemma | $72\%\pm 8\%$ |

In Table 2, we ablate the performance of the Reward as Code baseline across LLMs, observation spaces and additional assumptions. For pixel observations, we follow the methodology laid out in (Venuto et al., ), whereas for proprioceptive observations we follow the one from (Yu et al., ). Both methods heavily depend on access to a state-of-the-art, closed-source model to achieve performance comparable to that of AI Feedback, which uses the smaller, open-source model of Paligemma (Beyer et al., ). Additionally, each method requires expert demonstrations or specialized domain knowledge to guide the reward design process. While these assumptions may be viable in certain situations, such as in a controlled simulation environment, they can present significant practical challenges in more general contexts. In contrast, AI Feedback operates by simply comparing observations and reasoning using a chain-of-thought approach.

### A.6 Learning from Environment Rewards

In Figure 8, we compare the performance of an RL agent trained using a reward function derived from AI feedback with that of an agent trained on human-designed rewards across different environments. We observe that AI feedback achieves comparable results, with an average score of $89.93$ versus $86.3$ for the human-designed reward. The objective of this experiment is not to argue that LLM-based rewards consistently outperform human-crafted ones—since expert human knowledge can always be encoded into a reward function—but rather to contextualize the performance of LLM-based rewards. Notice that for MetaWorld we report the performance after fine-tuning the LLM as described in Section 5.

Figure: Figure 8: Comparison between the best performing LLM-based reward (AI Feedback) and human designed rewards for each domain.
Refer to caption: x12.png

### A.7 AI feedback and heuristic functions

While prior works have shown that rewards can be extracted from a language model (Brooks et al., ; Klissarov et al., ), it can be more generally thought of as encoding a heuristic function $h$. The function $h$ contains high-level, multi-step information about the MDP $M$. To extract it, one can solve the re-shaped MDP $\tilde{M}$ with $\tilde{r}(s_{t},a_{t})=r(s_{t},a_{t})+(1-\lambda)\gamma\mathbb{E}_{s_{t+1}|s_{
t},a_{t}}[h(s_{t+1})]$ and $\tilde{\gamma}=\lambda\gamma$ where $\lambda\in[0,1]$ Cheng et al. (). Solving $\tilde{M}$ yields a policy $\pi^{*}$ that is also optimal in $M$ - its value function’s bias can be shown to converge to $V^{*}$ in $M$ as a function of $||h-V^{*}||_{\infty}$.

Specifically, assume access to an initial dataset $\mathcal{D}_{0}$, from which a heuristic $h$ can be computed. In the reshaped MDP $\tilde{M}$, one can learn a new policy $\pi$ which optimizes $\tilde{r}$ with $\lambda\in[0,1]$. Equation 5 shows the performance difference lemma Kakade & Langford () as a function of true and reshaped MDP quantities:

$$ $\begin{split}\mathcal{L}(\pi,h)=&\mathbb{E}_{\mathcal{D}_{0}}[V^{*}(s)-V^{\pi} (s)]\\ =&c_{1}\color[rgb]{0,0.5,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.5,1} \pgfsys@color@cmyk@stroke{1}{0.50}{0}{0}\pgfsys@color@cmyk@fill{1}{0.50}{0}{0} \mathbb{E}_{\mathcal{D}_{0}}\bigg{[}\tilde{V}^{*}(s)-\tilde{V}^{\pi}(s)\bigg{] }\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0} \pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}+c_{2}\color[rgb]{ 0,0.88,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.88,0} \pgfsys@color@cmyk@stroke{0.91}{0}{0.88}{0.12}\pgfsys@color@cmyk@fill{0.91}{0} {0.88}{0.12}\mathbb{E}_{\mathcal{D}^{\pi}}\bigg{[}\tilde{V}^{*}(s)-\tilde{V}^{ \pi}(s)\bigg{]}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}+c_{3}\color[rgb]{ 1,0.04,0.61}\definecolor[named]{pgfstrokecolor}{rgb}{1,0.04,0.61} \pgfsys@color@cmyk@stroke{0}{0.96}{0.39}{0}\pgfsys@color@cmyk@fill{0}{0.96}{0. 39}{0}\mathbb{E}_{\mathcal{D}^{\pi}}\bigg{[}h(s^{\prime})-\tilde{V}^{*}(s^{ \prime})\bigg{]}\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{ 0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0},\end{split}$ (5) $$

where $c_{1},c_{2},c_{3}$ are non-negative constants. Minimizing $\mathcal{L}(\pi,h)$ with respect to $\pi$ and $h$ can be achieved by minimizing each individual term. In particular, the red term suggests that the heuristic $h$ has to be updated on data from $\mathcal{D}^{\pi}$ in order to not become "stale". This points out a shortcoming of existing LLM-as-critic algorithms, which sometimes fix $h$ after distilling the language model knowledge into it Klissarov et al. ()

These theoretical findings suggest, in particular, that heuristic $h$ (in our case, the Bradley-Terry preference model), has to be updated with on-policy samples, similarly to empirical results from Figure 5.

### A.8 Additional Considerations for Preference-based Reward Modeling

In Figure 9, we present the properties that were important to obtain effective exploration on NetHack, without the counting term shown in Equation 4.

Figure: Figure 9: Successful exploration on Nethack depends on both online preference elicitation and a non-Markovian reward function.
Refer to caption: x13.png

### A.9 In-Context Learning for Reward Modeling

In Figure 10, we present a variation on the Wordle game where the color code has been altered, which we refer to as Eldrow (reverse Wordle). Under this transformation, the off-the-shelf model provides feedback that correlates very poorly with the optimal value function. When we measure the perplexity of the LLM on a natural language description of the new rule set of Eldrow (see Appendix A) we obtain a value of $6.97$ which is higher than the one measured on the standard rule set of Wordle, with a value of $5.06$. Given that the difference in values is not very large, we leverage the simplest way for adapting the LLM: through in-context learning. As shown in Figure 10(b), by providing hints in the prompt about the new rule set, the LLM adapts its preferences and generates a Bradley-Terry model that recovers the correlation values we witnessed in 4.

Figure: (a) Before in-context learning
Refer to caption: x14.png

### A.10 LLMs as novelty detectors

We hypothesize that LLMs with long contexts can effectively act as novelty detectors. Within the scope of RL problems, this implies the ability to tell, for example, whether a sub-trajectory is contained in the replay buffer.

To test this, we query Gemini-1.5 Pro (Team et al., ) with a context video containing 500 frames of an agent exploring the bottom-left room (Figure 11-left) and a single frame sampled uniformly at random from a query episode which covers in the top-right room, center and bottom of the maze (Figure 11-middle). We ask the LLM to identify novel query states, i.e. states which are not seen in the context episode. We then train a direct predictor (3-layer MLP) to estimate the probability of any state on the grid to be novel with respect to the context (Figure 11-right). The language model correctly identifies the top-right portion of the trajectory to be novel, knowledge which could then be used to construct an intrinsic reward function.

Figure: Figure 11: LLMs can capture observation novelty. Given the context trajectory (red), and a single observation sampled uniformly at random from the query trajectory (blue), the LLM correctly identifies novel states that are seen in the query but not in the context (green).
Refer to caption: extracted/5905056/figures/exploration_minigrid.png

## Appendix B Additional Related Works

Large language models (LLMs) require additional adaptation for general-use language tasks (Christiano et al., ; Stiennon et al., ; Ouyang et al., ; Mialon et al., ).
Without additional context and/or fine-tuning, LLMs can generate misleading, harmful, or even nonsensical answers to queries or conversations with humans (Bai et al., ).
To modify their behavior, it is necessary to tune their prompts and/or fine-tune their outputs to ensure their output is desirable w.r.t. some set of linguistic tasks before deployment.
This at least if not more true in embodied settings, where real-world actions can have physical consequences, and methodologies for modifying LLM behavior in embodied settings more-or-less align with efforts in the language space.

#### Prompt tuning

Arguably the most common theme among techniques that modify LLM behavior in general is to change the prompt such that the distribution of LLM outputs better-fits a given desiderata on behavior.
Prompt-engineering can greatly align or calibrate an LLM, pretrained or no, to desired beneficial behavior (Christiano et al., ; Glaese et al., ; Bai et al., ), or even expose harmful or other unexpected behaviors.
Chain-of-thought (CoT, Wei et al., ) is an in-context method to either few-shot or zero-shot (Kojima et al., ) adjust an LLM’s outputs to generate more correct responses to question-and-answering tasks.
Further modifications to the prompt such as providing feedback from an environment (Yao et al., ), self-critique (Zelikman et al., ), or self-reflection (Shinn et al., ) can improve LLM performance in language as well as tasks that have an environment.
The biggest promise of in-context-based methods in RL is that somewhere within the large language model’s conditional distribution is the optimal policy for any given task (Brohan et al., ; Szot et al., ), an accurate world-explicit model (Lin et al., ), and/or a useful reward-model (Klissarov et al., ).
However, it is at best speculative as LLM’s are black box systems and prompt optimization is extremely difficult, and besides: systems built on this idea still must still overcome affordance mismatch (Ahn et al., ) and hallucinations (Zhang et al., ) to be useful for RL.

Querying model for feedback Another hypothesis is that LLMs contain knowledge relevant to tasks, and this knowledge can be extracted (Xu et al., ) in a way to train a policy that has desirable behavior (Huang et al., ).
RL AI Feedback (RLAIF Bai et al., ; Lee et al., ) is a scalable method akin to but without the practical issues that come paired with RL from Human Feedback (RLHF Christiano et al., ), the goal of which is to fine-tune an existing LLM to be more specific, accurate, innocuous, etc.
RLAIF trains a reward model on a dataset collected from an LLM’s preferences given a dataset of language responses from an LLM and a given set of queries, and this reward model is used to train a policy using RL, for example using PPO.
This process of extracting knowledge using preference data can also be directly used to train a policy without a reward model (Rafailov et al., ).

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. When reporting the final results in Fig 1, Fig 2, and Fig 6, do the authors control the number of interactions with the environment? I'm also interested in understanding how performance changes with an increasing number of interactions, as shown in the learning curves typically found in reinforcement learning (RL) papers.
soundness: 3
strengths: 1. The authors perform extensive experiments to evaluate the capabilities of large language models (LLMs) in sequential decision-making tasks. Specifically, they demonstrate that using LLMs to generate preference data and then train a reward model for RL training has the highest performance gain compared with other methods such as directly modeling policy by LLMs. This conclusion seems interesting and helpful for future works in designing RL agents incorporating LLMs.

2. The experiments in Fig. 3 and Fig. 4 clearly illustrate the effect of the rewards learned through the LLM Feedback.

3. It is interesting that the authors show that prompting engineering can steer LLMs for active exploration on the 
NetHack task, which approximates the count-based exploration bonus.
summary: In this paper, the authors study how Large Language Models (LLMs) can produce decision-making policies, either by generating actions or by creating reward models for reinforcement learning (RL). The authors use experimental results to reveal that LLMs excel at reward modeling, particularly when using AI feedback, and fine-tuning with synthetic data enhances performance in unfamiliar environments, helping prevent catastrophic forgetting.
weaknesses: 1. The related works section is very incomplete. The authors should discuss more recent works that study LLM decision-making problems and self-rewarding of LLMs such as [1]-[7]. The introduction of LLM for better reward design on RL is also studied in [8] and should be discussed carefully.

2. The authors compare direct and indirect policy modeling and use experiments to show that indirect policy modeling attains higher performance in multiple tasks. However, I am a little unconvinced about the experimental configuration, where the authors only query the GPT for the direct policy modeling but train an RL agent with multiple steps for the indirect policy modeling. It seems that the computation cost and the number of interactions with the environment are not controlled. Moreover, it has been shown in ([1]-[3]) that LLM can also serve as a critic model to give a numerical estimated value in the direct policy modeling. Maybe the authors should also add an experiment to analyze what would happen if the estimated reward from the indirect policy modeling were incorporated into the direct policy modeling. In this case, no RL training is involved and computation cost is easier to control.

[1] Zhou, Andy, et al. "Language agent tree search unifies reasoning acting and planning in language models." arXiv preprint arXiv:2310.04406 (2023).

[2] Sun, Haotian, et al. "Adaplanner: Adaptive planning from feedback with language models." Advances in Neural Information Processing Systems 36 (2024).

[3] Liu, Zhihan, et al. "Reason for future, act for now: A principled architecture for autonomous llm agents." Forty-first International Conference on Machine Learning. 2023.

[4] Huang, Jen-tse, et al. "How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments." arXiv preprint arXiv:2403.11807 (2024).

[5] Park, Chanwoo, et al. "Do llm agents have regret? a case study in online learning and games." arXiv preprint arXiv:2403.16843 (2024).

[6] Nottingham, Kolby, et al. "Do embodied agents dream of pixelated sheep: Embodied decision making using language guided world modelling." International Conference on Machine Learning. PMLR, 2023.

[7] Yuan, Weizhe, et al. "Self-rewarding language models." arXiv preprint arXiv:2401.10020 (2024).

[8] Kwon, Minae, et al. "Reward design with language models." arXiv preprint arXiv:2303.00001 (2023).

3. Minor typos: Line 762: "except fro".

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 3
questions: See weakness.
soundness: 1
strengths: 1. The paper is well-written and easy to follow.
2. The experiments are conducted on various environments.
summary: This paper studied the capabilities of Large Language Models (LLMs) for reinforcement learning (RL) in interactive decision-making tasks, by directly producing policies or indirectly generating rewards for policy optimization.
weaknesses: 1. The motivation of this paper is limited. One conclusion of the paper is that leveraging the feedback from the LLM to optimize the policy is better than using the LLM directly as a policy. However, since the latter is fine-tuning free, it is not surprising that the former is better. Even if the AI feedback is noisy, as long as the overall feedback is accurate, fine-tuning the policy against it can obtain some improvements.
2. The policy modeling and reward modeling abilities of LLMs in decision-making tasks have been widely studied in previous works [1,2]. No new methods are proposed in this paper, and the novelty is thus limited compared to these works.
3. The analysis is also not convincing. The authors claim that AI feedback helps better credit assignments. However, given only the training curves, it is not clear when and why credit is better assigned.

[1] Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models.\
[2] Ma et al., Eureka: Human-Level Reward Design via Coding Large Language Models.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - It seems that this work has some preliminary requirements for the LLMs to make it function well, e.g., chain-ot-thought (CoT), in-context learning, and self-refinement, as described in Section 2.1. How important are these components to the sequential decision-making capabilities of the LLMs? How do they affect the direct or indirect policy modeling of LLMs? 
- Under many real-world scenarios, we often expect a timely or frequent decision-making ability of the agent. When using LLMs directly or indirectly during the decision-making process, how can we guarantee that the decision is made timely yet effective?
- How much expert data do you use to fine-tune the PaliGemma model in Section 5? How about a different dataset quality, will there be a significant difference? I would expect a more in-depth discussion on fine-tuning LLMs here
soundness: 3
strengths: ## Pros

- The studied topic is very interesting. The authors explore the sequence modeling capability of large language models in the context of reinforcement learning, which can be of interest to a large number of researchers in the community
- The authors study either directly using LLM as the policy or leveraging LLM for decision-making and conduct numerous experiments
- The authors show that without task-specific fine-tuning, current LLMs only show limited decision-making capabilities when directly generating actions. Furthermore, the authors find that AI-feedback-based rewards produce dense functions that correlate positively with high-quality value functions. Such reward functions can significantly reduce the difficulty of assigning credit by redistributing rewards across different steps within a trajectory. Some of these observations and conclusions are interesting and can be useful for researchers, e.g., using LLM to directly output scalar reward signals can be surprisingly good on some tasks
- The results reported are averaged over 10 seeds, which is great considering the unstable nature of RL
summary: This paper explores the potential of large language models (LLMs) for tackling complex sequential decision-making problems in reinforcement learning (RL). The authors investigate two approaches: using LLMs to directly generate actions or indirectly generate reward models that guide RL agent training. Their findings demonstrate that even without specialized fine-tuning, LLMs excel at reward modeling. Notably, leveraging AI feedback to craft rewards emerges as a highly effective and generalizable strategy. This approach enhances performance by improving both credit assignment and exploration within the RL framework. Furthermore, the authors address the challenge of unfamiliar environments by demonstrating that fine-tuning LLMs with synthetic data significantly boosts their reward modeling capabilities. Importantly, this fine-tuning process effectively mitigates catastrophic forgetting, preserving the LLM's broad knowledge base.
weaknesses: ## Cons

- This paper does not propose any new method and looks more like an empirical paper that investigates the application of LLMs in either direct policy modeling or its capability of facilitating policy learning. Although I agree that such kind of paper is also of importance to the community, its technical novelty is still somewhat weak.
- Some conclusions derived by the authors are not surprising. For example, in environments with unfamiliar dynamics, fine-tuning LLMs with synthetic data can significantly improve their reward modeling capabilities while mitigating catastrophic forgetting. Many of the observations are actually known or acknowledged by many scholars and this paper seems to verify them through some designed experiments
- Another concern is that the conducted experiments may not be necessarily sufficient to back up the observations and conclusions in this paper. I list some of them below
  - the studied topics are limited and some selected topics are too broad to be covered in depth within a 10-page ICLR paper. The considered experimental settings are limited as described in Section 3 (lines 207-215), i.e., the authors only study approaches that can directly interface with the action space supported in the environment and assume that LLMs are used without any gradient update. When dealing with indirect policy learning, the authors only consider reward as code. Yet, there are many other options, e.g., LLMs to execute high-level plans for the agent, etc. Furthermore, the authors only consider a simplified version of reward as code without introducing some extra components or tricks as recent works do [1, 2, 3]. When using LLMs for exploration, the authors do not consider the possibility of letting LLMs write intrinsic reward functions, constructing intrinsic rewards [4, 5], etc.
  - the number of the base LLMs is limited. The authors used the closed-source GPT-4o model for direct policy modeling and the open-source Llama 3 for indirect policy modeling when environment observations consist of text, and PaliGemma when environment observations consist of pixel images. The number of LLMs is quite limited, making it hard to tell the significance and applicability of the observations and conclusions
  - the authors do not compare many baselines in this paper, e.g., the authors only consider the count-based exploration method in Section 4.2 (Figure 5). It would be better to compare against stronger baselines.

[1] Text2reward: Automated dense reward function generation for reinforcement learning

[2] Auto mc-reward: Automated dense reward design with large language models for minecraft

[3]  Eureka: Human-level reward design via coding large language models

[4] Guiding pretraining in reinforcement learning with large language models

[5] World Models with Hints of Large Language Models for Goal Achieving

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - The proposed method relies on consistent preference feedback from LLMs to construct reward signals, but doesn't adequately address cases of preference uncertainty. Consider a scenario where the LLM assigns similar preference probabilities to different states, or worse, provides inconsistent rankings when the same state pair is presented multiple times with slightly different prompts. 
	- So, what happens when the LLM expresses low confidence or contradictory preferences? 
	- Could inconsistent preferences lead to unstable reward signals that harm policy learning?
- The paper demonstrates the method primarily on environments where progress is relatively obvious (e.g., clear Wordle feedback, discrete game states in NetHack). However, many real-world tasks involve subtle, continuous progress where improvements may be hard to detect from observations. I have several concerns regard this problem:
	- How sensitive is the LLM's preference detection to small state changes?
	- Could the method miss important incremental progress by only detecting large, obvious changes?
- Sec 4.1 shows some evidence that AI feedback can help with credit assignment, the underlying mechanism isn't clear. In complex tasks, progress often results from a sequence of coordinated actions rather than single decisions. Can the method distinguish between critical and auxiliary actions when both contribute to the final outcome?
soundness: 3
strengths: - Comprehensive empirical evaluation across diverse domains.
- Interesting exploration of fine-tuning trade-offs between direct and indirect approaches.
- The paper is well written and easy to follow.
summary: The paper investigates how LLMs can be leveraged for RL, comparing direct policy generation versus indirect reward modeling approaches across diverse domains. The authors find that using LLMs to generate reward models, particularly through AI feedback preferences, yields better and more consistent performance compared to direct policy generation. They also explore fine-tuning approaches and analyze how LLM-based rewards can help address core RL challenges.
weaknesses: See the questions below.

Meta-review: The paper investigates how large language models can be integrated into reinforcement learning workflows, focusing on two approaches: direct policy generation and indirect reward modeling. Through experiments on diverse environments, the authors demonstrate that LLM-based reward modeling using AI feedback consistently outperforms direct policy modeling in both performance and sample efficiency. The paper further explores fine-tuning methods, revealing that fine-tuning for reward modeling mitigates catastrophic forgetting while improving reward quality.

**Strengths**

- The empirical evaluation is comprehensive (R1, R2, R3, R4).
- The empirical observation is interesting (R1, R3, R4).
- The paper is well-organized and easy to follow (R1, R2, R4).

**Weakness**

- Several reviewers note that the paper does not propose a new method, focusing instead on empirical analysis. Some conclusions, such as the benefits of fine-tuning with synthetic data, are unsurprising (R1, R2, R3).
- The handling of inconsistent or low-confidence preferences from LLMs and their sensitivity to subtle state changes remain underexplored (R4).
- The experiments lack detailed statistical analysis of critical phenomena (R2, R3).

The paper lacks significant methodological novelty, though its extensive empirical investigations and practical insights into leveraging LLMs for RL make it a valuable contribution to the field.

Decision: Accept (Poster)