On the Modeling Capabilities of Large Language Models for Sequential Decision Making
Authors: Martin Klissarov
R Devon Hjelm
Alexander T Toshev
Bogdan Mazoure
Venue: ICLR 2025 Poster (2025)
Keywords: reinforcement learning
large language models
ai agents
preference based learning
reward design
Primary area: reinforcement learning

Abstract: Large pretrained models are showing increasingly better performance in reasoning and planning tasks across different modalities, opening the possibility to leverage them for complex sequential decision making problems. In this paper, we investigate the capabilities of Large Language Models (LLMs) for reinforcement learning (RL) across a diversity of interactive domains. We evaluate their ability to produce decision-making policies, either directly, by generating actions, or indirectly, by first generating reward models to train an agent with RL. Our results show that, even without task-specific fine-tuning, LLMs excel at reward modeling. In particular, crafting rewards through artificial intelligence (AI) feedback yields the most generally applicable approach and can enhance performance by improving credit assignment and exploration. Finally, in environments with unfamiliar dynamics, we explore how fine-tuning LLMs with synthetic data can significantly improve their reward modeling capabilities while mitigating catastrophic forgetting, further broadening their utility in sequential decision-making tasks.

Published as a conference paper at ICLR 2025
ON THE MODELING CAPABILITIES OF LARGE LAN- GUAGE MODELS FOR SEQUENTIAL DECISION MAKING
Martin Klissarov* Mila, McGill University
Devon Hjelm Apple
Alexander Toshev Apple
Bogdan Mazoure Apple
ABSTRACT
Large pretrained models are showing increasingly better performance in reasoning and planning tasks across different modalities, opening the possibility to leverage them for complex sequential decision making problems. In this paper, we investi- gate the capabilities of Large Language Models (LLMs) for reinforcement learning (RL) across a diversity of interactive domains. We evaluate their ability to produce eeriiionnmmaknn policies, either directly, by generating actions, indirectly, by first generating reward models to train an agent with RL. Our results show that, even without -ssc-ipec fine-tuning, LLMs excel at reward modeling, In particu- lar, crafting rewards through artificial intelligence (AI) feedback yields the most generally applicable approach and can enhance performance by improving credit assignment and exploration. Finally, in environments with unfamiliar dynamics, we explore how fine-tuning LLMs with synthetic data can significantly improve their reward modeling capabilities while mitigating catastrophic forgetting, further broadening their utility in sequential decision decision-making tasks.
INTRODUCTION
Large Language Models (LL.M)) are generative models of natural language that can produce accurate general and domain knowledge (Singhal et al.) 2022; Imani et al. 2023 Manigrasso et al., 2024: Liu et al.) 2024a), reason over long textual contexts (Reid et al. 2024). and generalize zero- shot (Kojima al. (022). These capabilities suggest that LLMs might be well-suited for complex sequential decision decision-making problems, such as in embodied settings where an agent acts in an environment. Recent research has begun exploring this potential, investigating how LLMs can serve as sources of intrinsic motivation (Wang et al.) 2024 Klissarov et al.) (022)), demonstrating world modeling capabilities (Lin al. 20024] Liu et al. 2024b), and for acting and/or planning directly in an environment (Wang et al.) 2023 Padalkar et al. 2023 Zhang et al. 2024).
However, as the predominant paradigm for training LLMs not inherently aligned with the challenges of sequential Docision-making problems, such as active exploration, it is not obvious how to best bridge their capabilities to tackle such challenges in a general manner. We study this problem through the lens of reinforcement learning (RL., |Sutton& & Barto 2018), which formalizes how an agent interacts with an environment, receiving scalar rewards for each of its actions over a trajectory. We examine the capabilities of LLMs to solve RL tasks by comparing how they model policies 1) directly by generating action tokens, to 2) indirectly through a reward model derived from the LLM to be used within an RL algorithm. We perform comprehensive evaluation including MiniWob (Liu et al., 2018). NetHack (Küttler et al.] 2020), and Wordle (Lokshtanov & Subercaseaux 2022). and MetaWorls (Yu et al.) 2019). The environments we study present variety of challenges, such as different action space granularities, observation modalities ranging from natural language to pixel data, and varying horizon lengths.
diverse set of domains,
We first consider the off-the-shelf capabilities of LLMs for decision-making without updating them through additional gradient updates coming from the RL task. We find that indirectly modeling policies by first extracting knowledge from L.MMs in the form of a Bradley-Terry model (Bradley & Terry 1952 Christiano et al. (017) provides the best and most consistent performance across the environments we study. We empinicall analyze the various benefits, and limitations, provided by
Work done during Apple internship. Correspondence to: martin. kl klissarovemail wawprr/eemml.mmmmlll.ll ca.

---

Published as a conference paper at ICLR 2025
this approach, showing that it improves long-standing challenges in RL problems, such as credit assigmment and exploration.
Finally, while LLMs possess knowledge useful for many decision making tasks of interest, domains with complex or unfamiliar dynamics can significantly restrict their broader utility. We explore how fine-tuning an LLM with domain-specific data bridge this knowledge gap and study the effect of this procedure on the LLM's previous knowledge, as measured through success on datasets like POPE (Li et al. 2023b), GQA (Hudson & Manning 2019), A12D (Kembhavi al. 2016) and MMMU (Yue et al. 2024). Our investigation reveals that fine-tuning for indirect policy modeling mitigates catastrophic forgetting more effectively than direct policy modeling, offering a broadly applicable strategy for leveraging LLMs across diverse sequential decision crrision-mmaiigg making tasks.
USING LANGUAGE MODELS TO SOLVE RL TASKS
We first introduce the types of RL problems as well formalize the methodologies for using LLMs for RL tasks used in this work.
Reinforcement Learning. An RL task can be defined through a Markov Decision Process (MDP, Puterman 2014), which is composed of state space S, action space A, transition function P - (SS) which describes the forward dynamics of the system, a reward function r: r:8xA R and a discount factor 7 [0, 1]. Since it is often the case that the state only partially observable, we also assume the environment emits an observation or Po : S + A(O) from observation space O. A policy, or actor, is a probability distribution π S - (AA) which describes the action to be taken at every step. The objective of rational actor is to maximize the expected cumulative rewards over horizon H 0,
max E
Y rr( [s: max E. [V (su)),
t=
where the value function, (T(s), represents the expected discounted sum of rewards over the entire trajectory, re-weighted by the environment's dynamics model, p, and the actor's policy, т.
Large Language Models. An L.M a generative model of discrete random variables (i.e. tokens) conditioned on history (i.c. context). The LLM models the data distribution autoregressively:
p(1e4 [21, FT) II plarllere LLM(x
where E X' are token variables taken from a valid vocabulary. The suitability of LLMs for solving RL tasks without additional fine-tuning primarily hinges on the hypothesis that LLMs contain information - i... knowledge - about the underlying MDP, for instance, through the policy or reward function. How that information is extracted depends the data the LLM was trained on, the ability of the practitioner to properly prompt the model and interpret its responses to solve decision-makinn tasks.
2.1 PROMPTING
In this section, we describe the inputs, or prompts, to the LLM used in this work which allow change the LLM's output distribution to be useful for solving RL tasks. All prompts in this work 1) task specification using natural language input to provide information about the MDP to the LLM as context and 2) episode history in order to address issues of -ortrillobservabiliiy in some environments (similar to the Act-only baseline prompt found in Yao al. 2022). We additionally use the following set of techniques,
Chain of Thought. By prompting the LLM to provide step-by-step reasoning process for its output, rather than just the final answer, we can help surface its internal decision -eeiiiinnmmkiinggg and improve the resulting performance (Wei et al. 2022).
2

---

Published as a conference paper at ICLR 2025
In-Context Learning. To enhance the LLM's ability to solve the task, example solutions (e.g., from expert policies) are provided for in-context learning (Brown et al. 2020). where solutions contain sequences of combination of states, actions, and rewards.
Self-Refinement. To further refine its output, the LLM is prompted to provide recursive criticism and improvement from its generated outputs. This general strategy knows many variants, such as feedback from an environment Yao et al. 2022), self-critique (Zelikman) et al. 2022), or self-reflection (Shinn et al. 2023). ) In this work, we use Recursive Criticism and Improvement (RCI, Kim et al.) 2024) for its state-of-the-an performance on web agent domains and general applicability. In its original form, the LLM is given a task description and generates high-level plan. This plan is used along with the task description and current state to refine an action so that it is grounded in the current observation and the action space.
2.2 POLICY MODELING USING LLMS
As shown in Equation L. the goal of decision making agent is to learn a high performing policy и. This can be done either by maximizing the expected cumulative rewards and directly modeling the policy parameters (Sutton et al., 1999. Kakade & Langford 2002). Equivalently, this can be done indirectly by first modeling the parameters of the value function and applying greedy operator, such as in Q-Learning (Watkins & Dayan 1992). A similar separation between direct and indirect approaches can be useful to study the capabilities of LLMs to model RL policies.
Direct Policy Modeling. The most straightforward way to obtain a policy using LLMs is for the LLM to generate tokens that will be directly interpreted actions from the environment, E A (Yo et al. 2022 Shinn et al.] 2023 Kim et al.] 2024). To ensure the outputted actions adhere to the environment's action set, the LLM output tokens can be projected back onto A using projection operator proj(:- A) (e.g., see Huang et al. 2022 Kim et al.) 2024 for examples of projection operators). A variety of prompting techniques can be combined increase the ability of the L.MM to act, without task-specific fine-tuning, as policy, which we detail in Section 2. This direct policy method will be referred to in our experiments as LLM Policy.
Indirect Policy Modeling, On the other hand, we can prompt the LLM to output tokens representing intermediate quantities that will then be used to learn a policy. For example, one can model the forward dynamics of the environment for planning (Liu et al. 2024b) or an affordance model for action selection (Mullen Jr & Manocha 2024).
In this work, we focus the case where these intermediate quantities will be used to generate rewards - i.e., a reward model - which will then be maximized by an off-the-shelf RL policy. Section 2.3 we enumerate the different approaches for modeling reward functions with LLMs covered in our work. It is important to note that there exists many more ways in which we could indirectly model the policy. In Appendix A.4 we present in detail these possibilities and, in Figure 2b provide initial investigations that showcase their potential and limitations.
In direct policy modeling experiments (LLM Policy), we found combining all of the prompting techniques in Section [211 to work the best, while for indirect modeling methods through reward we relied only on chain-of-thouggh prompting. Additional details, such specific prompt details and ablations on these choices are presented in the Appendix A.3
2.3 INDIRECTLY MODELING POLICIES THROOGH REWARD MODELS
We consider a diversity of methods for modeling reward functions using LLMs, with a particular attention to methods that are applicable to a diversity of environments and modalities. We study the following set,
2023 The LLM generates tokens that directly encode the Direct Scalar. (Kwon et al.
reward (c.g., as float or integer) given an observation (or a sequence of observations and actions). This reward is then given the RL agent.
AI Feedback (Lee et al. 2023 Klissarov et al. 2024h). Ask the LLM to express preference y = [1, Ø between two observations, 0n and 02, for the one showing the most progress towards certain goal, preference if both observations are equally
3

---

Published as a conference paper at ICLR 2025
good. These labels then be collected as dataset of observation preefernneeeeeeeeeeee tuples y which are then used to train reward function modeled as, D Dpret {(o] 02
(4)
ro - arg min E (29)))
Ily - 1] log Polor > 02) + My - P) > o1l
1 + 2 - = Ø log (Polo1 > o2] -
where Polo1 > 02] (ellor)) the probability of preferring observation another,
referred to the Bradley-Terry model for preference learning (Bradley & Terry 1952). The
minimization of this equation is commonly done through binary cross-entropy.
Reward as Code (ii et al. (2023) ) Yu et al. (2023): ) Ma et al. (2023) Prompt the LLM to write code that will take as input a subset of symbolic features from the environment observations and will produce a scalar output representing the reward. The code defining the reward function is then updated throughout environment interactions as in Li et al. (2023a).
When symbolic features are not available, these are constructed as in Venuto et al. 2024).
Em Embedding--bseed (Rocamonde et al. 2023 Du al., 2023; Liu al., 2024b). Instead of querying language tokens from the LLM, we can instead, for a given input, leverage the information encoded in its latent represention, or embeddings. These embeddings are used to calculate the cosine similarity with respect to the embeddings of natural language specification of goal behaviour. The resulting similarity value is given as a reward to the agent.
Additional details, such specific prompts, are presented the Appendix A.2
PERFORMANCE OF INDIRECT AND DIRECT POLICY MODELS
100
Al Feedback Reward as Code
Normalized Performance
Embreddng-based
80
40
20
0.
MiniWob-Hard
Wordle
NetHack
MetaWorld
Figure I: AI feedback has the highest performance across different reward models derived from tested LLMs. Al feedback, which is preference-based method for deriving a reward model from an LLM, generally outperforms other methods.
Due to fundamentally different challenges between direct and indirect policy modeling approaches, conducting a fair comparison requires care. For example, using the I.L directly as a policy requires grounding its outputs the action space defined by the environment (Ahn al.. [002 Huang et al.. 2022). As the action space vary significantly between environments and attempting to solve this problem adds additional algorithm- or fomain-specifie complexities (e.g. by crafting skills, see (Ahn)
al. 2022
Wang et al. 2023)), fix experimental setting to the following
1. Atomic actions. We only study approaches which can directly interface with the action space supported in the environment. In other words, the action space is least subspace of the space of language generated by the LLM. This allows for a more direct comparison across variety of domains and study the relationship between an LLM's 's knowledge and the fixed action space defined by the environment.
2. No finetuning. In most of the paper we assume that LLMs are used without any gradient updates, i.e. without fine-runing from the RL task, and evaluate their off-the-sheiis capabilities. In Section 3] we perform a preliminary study on the trade-offs between fine- tuning for direct and indirect policy modeling.
4

---

Published as a conference paper at ICLR 2025
Normalized Performance
Binary Predicttion Accuracy
50
70
10.
Meratorid
40
MinWobb-ond Hand
Wordle
Netltack
MetaWorid
Action Granularity
(b)
(a)
Figure 2: a) Building reward model more-readily solves RL tasks than using LLM actor. M.L--policy only performs well in domains with coarse-grained actions while LLM feedback presents strong performance across the entire range of action granularities. h) LLMs have unreliable zero-shot understanding of the environment dynamics. While LLMs can be used to craft useful reward models, their failure direct policies may be explained by their poor understanding of the action space and the transition function.
We investigate four separate domains, where each domain aims to highlight a specific capability of LLMs: 1) MiniWob-Hard a subset of hard tasks from the full MiniWob suite, tests web interaction in observation/action spaces close to natural language, 2) Wordle measures reasoning and planning capabilities, 3) NetHack presents the difficulty of exploring open-ended environments under partial observability, long horizons and procedural scenarios, and 4) MetaWorld assesses the ability to control low-level, high-frequency actions continuous space. We provide a detailed description of each domain in Appendix A.
Direct policy modeling done by querying the closed source GPT-4o model, whereas indirect policy modeling is done through the open source models of Llama 3 (Dubey al. 2024), when environment observations consist of text, and PaliGemma (Beyer et al. 2024). when environment observation consist of pixel images. In the Appendix I. investigate larger set of LLMs, including Claude 3.5, Gemini Pro 1.5, Llama 3.2-V and Qwen 2.5 (Qwen Team 2024). All results are averaged over 10 seeds with error bars indicating the standard error.
Indirect policy modeling through rewards. We first present comparison of the various indirect policy modeling approaches discussed in Section 2.3] these experiments, the LLM generates a reward function which will be given RL agent for optimization, without access any rewards coming from the environment. When learning policies through RL we do not perform any hyper- parameter search and simply borrow the existing empirical setup for each domain, as detailed in Appendix A.I
In Figure we present the performance across domains measured by the average success rate on all domains, except for NetHack, where performance (the in- tn-game score) is normalized by the highest recorded value. Results show that AI feedback the only method that successfully crafts rewards across all environments and modalities] On easier domains such as MiniWob- Wob-Hard, which consists of short episodes and limited scope variations, the Direct Scalar method performs nearly as well as Al feedback. However, the disparity between methods is much more pronounced on harder, open-ended tasks such as NetHack. Out of all the methods, Embedding bassdd leads to the lowest performance. Finally, the effectiveness of Reward Code appcars be highly contingent on the availability of symbolic features for code processing. In Appendix A.5 we further examine the assumptions- such access functional knowledge of the environment under which Reward Code achieve performance comparable to Al feedback.
Direct vs indirect policy modeling. We now compare the direct policy modeling method, L.MM Policy, to the best performing indirect modeling method, Al feedback, reporting performance across the same set of domains. Results in Figure 2a show that, despite the more complex prompting
In Appendix A.6 verify that AT feedback yields policies with performance par with those optimized using human-designed environment rewards.
5

---

Published as a conference paper at ICLR 2025
strategies and the of more capable closed source model, LLM Policy is unable to perform well in most environments, with the exception of MiniWob-Harc where the performance on on-par with AI feedback. Given that the reward modeling baseline proceeds to fine-tuning the RL policy, may wonder if, for a similar amount of compute and environment samples, what performance could the I.LM Policy achieve by fine-tuning the LLM. In Appendix [A.13] also study this question, revealing that reward modeling is significantly more efficient. In Figure of of Appendix A2 we additionally investigate whether the reward obtained by the AI feedback method can be given the LLM Policy context, in order to improve the direct modeling performance. Results indicate that this additional information does not significantly change the performance.
A question emerging from these results is: what factors cause this significant performance disparity between direct and indirect policy models? One possible explanation is that L.LMs, when directly queried for actions in an unfamiliar environment, may struggle to understand its dynamics (e.g.. the transition function and action space). To test this hypothesis, we conduct the following experiment. We prompt the LLM to select between 1) a pair of candidate next observation given the current observation and action (probing knowledge p(o++1 (4, oge)), or 2) a pair of candidate actions given the next observation and current observation (probing knowledge of p(ar|ot+1 0ct)). In each case, the pair contains the ground- -oound-truth and random sample. In this experiment, a 50% accuracy corresponds to random guess.
Results presented in Figure 2h show that the LLM performs relatively poorly on both of these tasks, indicating limited understanding of both the action space and the environment dynamics. This can potentially explain the limited performance of the LLM Policy approach on MiniWob-Hard NetHack, and World, while results on Wordle suggest that additional contributing factors are at play.
ANALYSIS OF AI FEEDBACK FOR RL
Our results so far suggest that, without additional fine-tuning, indirectly modeling policies by constructing reward functions through AT feedback is the most effective approach across the range of environments and modalities we studied. In this section, we examine how rewards shaped by this method can assist RL agents in addressing core decision-making challenges, such credit assignment and exploration. Through this analysis, we also emphasize the ways in which reward misspecification can unintentionally arise and severely impair performance.
Doorkey-8x8
Reward Model Value
1.0
7.5-
5.0
Return
0.6
0.4
0.0
-25_
0.2
-5.0-
0.0
0.5
1.0
1.5
-7.-
Transitions
15
20
Time Step
Al Feedback
Environment Reward
Figure 3: Rewards learned through AI Feedback distribute rewards to key timesteps. By doing so, the problem of credit assignment, learning from delayed rewards, is significantly reduced. Such distribution effectively shortens the horizon over which the RL algorithm must propagate credit through its update rule.
4.1 CREDIT ASSIGNMENT
Setting aside performance of the direct policy method, we now turn our attention to why reward modeling with AI feedback is performing well. AI feedback-based rewards depend the prompt

---

Published as a conference paper at ICLR 2025
used to capture preferences. In the experiments conducted far, these prompts designed to elicit preferences by emphasizing states that contribute to task progress (see prompts Appendix A.2). Additionally, a key aspect of our methodology involved presenting the LLM with observations sampled randomly within trajectories. This enabled querying preference for any observation in the environment, rather limiting the focus to final states distinction also known as process-based and outcome-based reward models (Uesato et al. 2023 Lightman et al. 2023) What are the resulting characteristics of the reward model under such choices?
Qualitative experiment In Figure B] we present the output of the Al feedback- -bsee reward model over each timestep of an episode within a simple grid world environment. This task includes an agent, a key, a door, and a goal Shheallerr-Boiverr et al. (023). We notice that this reward model naturally captures the fact that picking up the key, as well as opening the locked door, are important steps towards the goal. By propagating credit over such key moments in trajectory, the LLM effectively shortens the horizon over which the RL algorithm must assign credit through temporal difference learning (Sutton & Barto 2018). This is manifested in Figure [3] where the agent learning through AI feedback reaches a high success rate in a fraction of the timesteps required by a similar agent learning from the environment feedback (which in this case is sparse reward of +1 for reaching the goal). The dense reward resulting from L.M feedback can be seen as a form of reward redistribution (Arjona- Medina et al. 2018), which is an established method for improving credit assignment.
1.0
0.6-
Surfiaan fand Comttax
0.8
Contier feed
0.8-
0.7
Contfer v.)
Correlation
0.6
Correlation
0.4
Correlation
8.5
c.8.
0.3-
0.4
0.4
0.21
0.1
Contere Feee Comborere
0.2-
61 0/2
1'
300m
0.0
200k
300k
400k
Policy IT degree optimality
Training Steps
Training Steps
(a) Wordle
(b) NetHack
(c) MetaWorld
Figure 4: LLM preferences correlate with value function preferences. The correlation between Bradley-Tere models trained from frozen LLM state preferences and value function preferences increases as the online policy improves in different domains.
Quantitative experiment In Figure 4 we present the correlation between the reward model derived from AI feedback and the value function of an RL agent across various levels of policy optimality. We observe that AI feedback generates reward functions with a stronger correlation to value functions obtained later in the training process compared to those from earlier stages. Additionally, this correlation is higher than that observed with the environment reward. In the Wordle game, we generate, in code, near-optimal policy and estimate its value function using Monte Carlo. We then compare the -L-M-deiieed reward function find an almost perfect correlation. These findings suggest that the reward models derived from AI feedback inherently encode aspects of high-quality value functions, which, when used as rewards for the RL agent, can substantially simplify the credit assignment process. In Appendix A.77 we provide additional insights from the lens of heuristic-guide reinforcement learning (Cheng et 202T).
4.2 EXPLORATION
In the previous section, investigated how our standard prompting strategy ease the problem of credit assignment in downstream RL tasks. This outcome stemmed from the specific preferences requested from the I.M,, that is, promoting task progress. However, to address different RL objectives, in particular the one of exploration, we may need to elicit alternative preferences.
Previously, Klissarov et al. (2024) employed AI feedback design an effective reward function for an agent operating in the open-ended environment of NetHack. However, before applying this reward the RL agent, the authors implemented the following transformation:
r(o:) TAAF (o:r(o:)/N(o)))
7

---

Published as a conference paper at ICLR 2025
where FAIF is the reward model obtained from Al feedback, [(o)) denotes the number of times a particular observation or was seen an episode, and a positive real-valued coefficient set to 3. The counting term added to encourage exploration (Henaff et al. 2022). which is key difficulty in NetHack. However, instantiating such a counting function proves difficult in many practical settings (Bellemare et al. 2016). Given the flexibility of natural language, can we alleviate the need for such a term and integrate the notion of exploration in the prompt itself?
In Figure we demonstrate that this is indeed possible, leading to performance comparable when using count- based exploration by directly modifying the prompt used for preference elicitation. Specifically, when querying the L.LM for preferences, we present it with a pair of sequences of observations (rather than a single observa- tion) which provides erucial context. The prompt was also modified to steer the LLM towards avoiding low entropy sequences, i.c. sequences with repetitions (see Appendix A.2)p.
Score
Normalized Performance
Dungeon Level
0.8
0.6
0.4
Online Preferences Non-Markovian Model
Offline Preferences Model
Our findings reveal two potential failure modes: the of- fline nature of the preference elicitation method and the assumption of a Markovian reward model. Previous re- search has demonstrated that online preference querying can outperform offline methods when aligning LLMs (Bai al., 2022; Touvron al.) 2023). In our experiments, offline elicitation led to performance collapse, likely due frequent RL policy updates during online learning. Ad- ditionally, assuming Markov reward model-where the current observation fully determines the reward-can lead to an equally poor performance, as complex tasks often require historical context beyond immediate observations (see Appendix A.8 for full breakdown).
Figure By changing the prompt, LLMs can be steered to provide feed- back that promotes exploration on NetHack. Additionally, to avoid degen- erate solutions, preferences should be elicited in online fashion and the re- ward function be non-Markovian.
5 BEYOND ZERO-SHOT REWARD MODELING
So far, we have explored the ability of LLMs to model policies, directly and indirectly, without any fine-tuning. However, in many cases the prior knowledge encoded in LLM might not contain the necessary information to do so successfully. In such instances, fine-tuning becomes an effective method for incorporating task-specific knowledge into the model.
We consider the sweep-into task from Meta World, where Al feedback rewards lead to a success rate of the PaliGemma model on captions describing
of only 15%. When measuring the perplexity the pixel observations from the task, we obtain a value of 16.03. Both of these results indicate poor understanding and the necessity adapt the model.
We therefore fine-tune PaliGemma on image-caption pairs annotated by GPT-4o and trained the model predict the caption for given image. Figure (aa shows significant gains in downstream RL performance after only few fine-tuning epochs and as few as approximately 100 image-caption pairs. Moreover, Figure 6a shows how this procedure only marginally decreases performance of the LLM on the standard mulli-moda reasoning benchmarks, such as POPE (Li et al. 2023b), GQA (Hudson Manning 2019). AI2D (Kembhavi al., (016)) and MMMU Yue al. 2024). Surprisingly, performance on the A12D benchmark improves the number of RL-specific fine-tuning epochs
increases.
We contrast these findings with Figure 6b| where we tune PaliGemma with behaviour cloning on expert data on the same Meta MetaWorld task. Similarly to RT-2 (Brohan et al. 2023), we overwrite the least frequent tokens with residual VQ-VAE codebooks (Szot et al. 2024). In this case, any significant increase of RL performance comes at the cost catastrophically forgetting all previous knowledge. These results hint at important trade-off: if preserving prior language reasoning knowledge is important, fine-tuning for Al feedback offers viable approach. However, maximizing downstream RL. performance the sole objective, directly fine-tuning for action selection can be more effective.
8

---

Published as a conference paper at ICLR 2025
POPE RL Parformance
GQA Performance
A12D Performance
MMMU Performance
547
0420
241
0.719
60/3
1330
64.0
0.58
3.39
608
134
RL Perfomance
RL Performance
RL Performance
RL Perforance
Off-the-sheif
Epoch
Epochs
10 Epochs
(a) Fine-tuning for Al feedback
POPE vs RL Parformance
GQA vs RL. Performance
A120 vs RL Performance
MMMU vs RL Porformance
60
0.25
MMMU
0.25
0.24
165
110
0.25
34
62
O4
02
Performance
Performance
Performance
Performancc
Off-he-sheiff
2000 Epochs
4000 Epochs
* 6000 Epochs
(b) Fine-tuning for direct policy modeling
Figure 6: Fine-tuning LLMs for AT feedback better preserves their prior knowledge. LLMs fine- tuned for AT feedback in (a) retain a higher portion of their original language reasoning knowledge than those fine-tuned for direct action selection in (b).
6 RELATED WORKS
Large language models (LLMs) require additional adaptation for general-use language tasks (Chris
tiano et al. 2017 Stiennon al. 2020 Ouyang et al. 2022 Mialon et al. 2023). Without additional context and/or menittning. LLMs can generate misleading. harmful, or even nonsensical answers to queries or conversations with humans (Bai et al.) 2022). To modify their behavior, it is necessary to tune their prompts and/or fine-tune their outputs to ensure their output is desirable w.r.t. some set of linguistic tasks before deployment. This at least if not more true in embodied settings, where real-world actions can have physical consequences, and methodologies for modifying LLM behavior in embodied settings more-or-less align with efforts in the language space.
Prompt tuning Arguably the most common theme among techniques that modify LLM behavior in general is to change the prompt such that the distribution of LLM outputs better-fits a given desiderata behavior. Prompt-engineering can greatly align or calibrate an LLM, pretrained or
no, to desired beneficial behavior (Christiano et al. 2017: Glaese et al., 2022 Bai et al. 2022). or even expose harmful or other unexpected behaviors. Chain-of-thoug
(CoT, Wei et al. 2022) is
an in-context method to either few-shot or zero-shot (Kojima et al.. 2022) adjust an LLM 's outputs tasks. Further modifications to the prompt such as providing feedback from an environment (Yao et al. (022), self-critique (Zelikman) al. 2022), or self-reflection (Shinn et al. 2023) can improve LLM performance in language as well as tasks that have an environment. The biggest promise of in-context based methods in RL is that somewhere within the LLM's s conditional distribution the optimal policy for any given task (Brohan et al. 2023 Szot et al. (023)), an accurate world-explicit model (Lin et al. 2024), and/or useful reward-model (Klissarov et al.) 2024). For example, [Sun et al.] (2023) has shown strong results on text-based domains such as web agents by using LLMs as policies. Other approaches combine different quantities from the RL problem within the same algorithm, for example by providing LLM
to generate more correct responses
crestion-and-answerrin
rewards/critiques as context an LLM acting a policy (Zhou et al. 2023 Liu al.
2024c). LLMSs
have also been shown to be particularly good at devising high-level plans that will guide an agent acting in the environment (Nottingham al. 2023)
Querying model for feedback Another hypothesis is that LLMs contain knowledge relevant to tasks, and this knowledge can be extracted (Xu et al. 2024) in way to train a policy that has desirable

---

Published as a conference paper at ICLR 2025
behavior (Huang al. 2022). Kwon et al. (2023) first studies the possibility of querying an LLM to design reward functions. In their method, the LLM is given a task description, examples and an objective specified by a user, and is used to outputs text that is mapped to binary value. Their results show that their approach is significantly more sample efficient compared to a supervised learning approach. The most successful reward modeling method in our paper is based on RL from AI Feedback (RLAIF Bai et al. 2022, Lee et al.) 2023), a scalable method akin to but without the practical issues that come paired with RL from Human Feedback (RLHF Christiaao et al. 2017). the goal of which is to fine-tune an existing LLM to be more specific, accurate, innocuous, etc. RLAIF trains a reward model on a dataset collected from an LLM's preferences given a dataset of language responses from an LLM and given set of queries. and this reward model is used to train a policy using RL, for example using PPO. This process of extracting knowledge using preference data can also be directly used to train policy without reward model, as in Direct Preference Optimization (Rafillov et al., 2024). A related line of work studies the possibility of LLMs crafting their own rewards to improve their capabilities. (Yuan et al. (2024) leverage DPO to improve an LLM's ss ability to for instruction following. Investigating this idea within the domain of sequential decision making is particularly promising.
7 DISCUSSION
In this paper, we explored two distinct approaches to leveraging LLMs for solving RL tasks: 1) directly, by modeling policies and 2) indirectly, by modeling rewards to be leveraged within a policy learning algorithm. Our results indicate that, without task-specific fine-tuning, current LLMs only show limited decision- makinn capabilities when directly generating actions. However, despite this limitation, LLMs are capable zero-shot reward modelers. In particular, when eliciting preferences to define rewards through the Grudley-Terr, model, LLMs show strong performance across a wide range of domains presenting various challenges.
In cases where an L.M's prior knowledge is not enough to obtain useful reward functions, we also investigated fine-tuning with task- data to bridge this gap. Notably, fine-tuning enhance reward modeling capabilities helps mitigate catastrophic forgetting, which is a crucial consideration for preserving the LLM's seneral-purpose abilities Maintaining these capabilities is essential for broad applicability to sequential decision- -maiing tasks, including out-of distriibbti tasks, and for supporting continued natural language interaction with users.
The reward modeling capabilities presented in this work offer potential solutions to challenges in RL. First and foremost, -.r--derivedd reward models alleviate the need for human-designed reward functions, which are often complex and costly to develop. Second, our empirical analysis reveals that Al-feedback based rewards produce dense functions which correlate positively with high-quality value functions. Such reward functions can significantly reduce the difficulty of assigning credit by redistributing rewards across different steps within a trajectory. Finally, distilling knowledge from LLMs into reward models opens new possibilities for applying RL in environments where simulators or symbolic features are unavailable- such as embodied AI agents interacting with humans. Additionally, when dealing with real-world scenarios, we may be concerned by the speed which a model may react to its environment. Direct policy modeling implies that the LLM is executed in the environment, which might limit the frequency with which it can queried. For example, the report from Black et al. (2024) emphasizes the necessity of using special techniques such as action chunking to achieve a 50 Hz control frequency. contrast, indirect policy modeling can distill an LLM's knowledge into smaller neural network, which would be queried much faster.
Some notable limitations and caveats exist. It is possible that by using LLM feedback design reward functions, we may obtain contradictory and inconsistent preferences, simply by virtue of LLMs having an imperfect understanding of the task at hand. In fact, we believe this is the main reason why, without fine-tuning, the PaliGiemaa model produces some unsuccessful RL policies on Meta World, as seen in Figure 6 If a particular task requires detecting subtle, incremental progress, it is entirely possible that the LLM might miss key milestones. Additionally, interacting with LLMs through natural language requires experimenting with various prompting techniques and specifications. However, this flexibility also enables the shaping of reward functions incorporate valuable strategies (Knox et al. 2013), such as promoting exploration, which can further enhance the performance of RI. agents.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, and Byron David et al. Do as can, not as say: Grounding language in robotic affordances. In Conference on Robot Learning, 2022. BRLhtt/p://app semanti www.chola...og//oouusiii........
Anonymous. Placeholder title, 2024. Under review.
José Antonio Arjona-Medina, Michael Gillhofer, Michael Widrich, Thomas Unterthiner, and Sepp Hochreiter. Rudder: Return decomposition for delayed rewards. In Neural Information Processing Systems, 2018. URL https:/t: api http:////aap.sssstssssccoooo semanticscholar www.oorggssD:eee...oo. 493306773
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprini arXiv:2212.000073 2022.
Mare G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Rémi Munos. Unifying count-based exploration and intrinsic motivation. In Neural Information Processing Systems, 2016. URL https:
Barg/CorpusID: 8310565.
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel M. Salz, Maxim Neumann, Ibrahim M. Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, Daniel Keysers, Skanda Koppula, Fangyu Liu, Adam Grycner, Alexey A. Gritsenko, Neil Houlsby, Manoj Kumar, Keran Rong, Julian Martin Eisenschlos, Rishabh Kabra, Matthias Bauer, Matko Bovsnjak, Xi Chen, Matthias Minderer, Paul Voigtlaender, Ioana Bica, Ivana Balazevic, Joan Puigcerver, Pinelopi Papalampidi, Olivier J. Hénaff, Xi Xiong, Radu Soricut, Jeremiah Harmsen, and Xiao- Qi Zhai. Paligemma: A versatile 3b vlm for transfer. 2024. URL :::
https://api. semanti cecttiiccchlaaacao
Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian lchter, Szymon Jakubczak, Tim Jones, Liyiming Ke, Sergey Levine, Adrian Li-Bell, Mohith Mothukuri, Suraj Nair, Karl Pertsch, Lucy Xiaoyang Shi, James Tanner, Quan Vuong, Anna Walling, Haohuan Wang, and Ury Zhilinsky. mn A vision-language- action flow model for general robot control, 2024. URL https::/aaxxv..rrg/aaa..
2410 24164
Ralph Allan Bradley and Milton E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39:324, 1952. URL seenaaiccorssmmtttcccccooao.rr org/ CorpusID: 125209808
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, and Krzysztof Choro- manski al. Rt-2: Vision-languagge-atiion models transfer web knowledge to robotic control, 2023. URL http/t//t////rtttt http:///xxxxvv.oogag/g/
15818
Ethan Brooks, Logan Walls, Richard L Lewis, and Satinder Singh. Large language models can implement policy iteration. Advances in Neural Information Processing Systems. 36, 2024.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Ma teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005. 14165, 2020. URL www.stissaanniiiieeiae.e
httpps///ap..smmmaticcchhho wow..oooss/D/:::111111111
Ching-An Cheng. Andrey Kolobov, and Adith Swaminathan. Heuristic Reuristic-guided reinforcement learnin Advances in Neural Information Processing Systems, 14:135500-3363, 2021.
Maxime Chevalier- Phevalier-Boiivert,, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro. and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal goal-oriented tasks. CoRR, abs/2306.1383. 2023.
11

---

Published as a conference paper at ICLR 2025
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. In North American Chapter of the Association for Computational Linguistics, 2019. URL httph://pp:ppp. semanticscholar. cww..aarurrrCCoustii 52967399
Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, P. Abbeel, Abhishek Gupta, and Jacob Andreas. Guiding pretraining in reinforcement learning with large language models. In International Conference on Machine Learning, 2023. URL http::t//s:/aat httpps///api.ssmtnissccoooo semanticscholar org/CorpusID: 256846700
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and Ahmad Al-Dahle et al. The Ilama 3 herd of models, 2024. U.L.Attpss//arxvvvorrg///// /ttp//bb//297.27333
Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209. 14375, 2022.
Mikael Henaff, Roberta Raileanu, Minqi Jiang, and Tim Rocktäschel. Exploration via elliptical episodic bonuses. NeurlPS, 2022.
Wenlong Huang, Picter Abbeel, Deepak Pathak, and Igor Mordatch. Language models zero-shot planners: Extracting actionable knowledge for embodied agents. In International conference on machine learning, pp. 91118-9445 PMLR, 2022.
Drew A Hudson and Christopher Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Shima Imani, Liang Du, and Harsh Shrivastava. Mathprompter: Mathematical reasoning using large language models. arXiv preprint arXiv:2303.05338, 2023.
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning, pp. 267-274, 2002.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pp- 235-251. Springer, 2016.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. Advances in Neural Information Processing Systems, 36, 2024.
Martin Klissarov, Pierluca D'Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. Motif: Intrinsic motivation from artificial intelligence feedback. In The Twelfth International Conference Learning Representations, 2024. URL
https://openrreviek met//forumiid==tBBTTTCCC
W. B. Knox, Peter Stone, and Cynthia Breazeal. Training robot via human feedback: A case study. In International Conference on Software Reuse, 2013. https:// wwwp.ssaaatiicchol....
org/CorpusID: 266033110
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199-22213, 2022.
Heinrich Küttler, Nantas Nardelli, Alexander H. Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The NetHack Learning Environment. In Proceedings of the Conference Neural Information Processing Systems (NeurIPS), 2020.
12

---

Published as a conference paper at ICLR 2025
Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. In The Eleventh International Conference on Learning Representations, 2023. URL
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, al. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2339.0002 2023.
Hao Li, Xue Yang. Zhaokai Wang, Xizhou Zhu, Jie Zhou, Yu Qiao, Xiaogang Wang, Hongsheng Li, Lewei Lu, and Jifeng Dai. Auto me-reward: Automated dense reward design with large language models for minecraft. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16426- 16435, 2023a. URL https://api. www.stiismcnnolteee..ee wwt.scchooa....oooooooo/.o. arggCCopusII
266210361
Yifan Li. Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large rision-languag models. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023b.
Hunter Lightman, Vineet Kosaraju, Yura Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. ArXiv, abs/2305.20050, 2023. URL htthhttt///ppp httpps//aap...mmmaiciccooooolol wrg. wergg/CorpusDD::: 258987659
Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca Dragan. Learning to model the world with language, 2024. URI htt.httpss//axiv..oom org/ abs 2308. 01399
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workfloo-guidee exploration. In International Conference on Learning Representations (ICLR), 2018. URL https::/arxiv..rrg abs// 1802. 08802
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36, 2024a.
Zeyuan Liu, Ziyu Huan, Xiyao Wang, Jiafei Lyu, Jian Tao, Xiu Li, Furong Huang, and Huazhe Xu. World models with hints of large language models for goal achieving. arXiv preprint arXiv:2406.07338 07381, 2024b.
Zhihan Liu, Hao Hu, Shenao Zhang, Hongyi Guo, Shuqi Ke, Boyi Liu, and Zhaoran Wang. Reason for future, act for now: A principled architecture for autonomous Ilm agents. In International Con- ference Machine Learning, 2024c. URL https: www.nticscholaattttt www.:laar..rg/oopuss11 272330199
Daniel Lokshtanov and Bernardo Subercaseaux. Wordle is np-hard. ArXiv, abs/2203.16713. 2022. 2022. semanticscholar.
URL https://api: http:://app.ssmnniiicchhol.aa htt.
Yecheng Jason Ma, William I. Liang, Guanzhi Wang, De-An Huang, Oshert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. ArXiv, Abs/2310.119311 2023. Uttpttps::/app ptatts::/app.sematticcchhh semanticscholat /ttp//oopusl::2243222220
Francesco Manigrasso, Stefan Schouten, Lia Morra, and Peter Bloem. Probing Ilms for logical reasoning. In International Conference on Neural-Symbolic Learning and Reasoning, pp. 257-278. Springer, 2024.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: a survey, 2023. URLphttps
www.ii.oorg/bb//230000 2788/ 07842
James F Mullen and Dinesh Manocha. Towards robots that know when they need help: Affordance- based uncertainty for large language model planners. arXiv preprint arXiv:2403. 198, 2024.
13

---

Published as a conference paper at ICLR 2025
Kolby Nottingham, Prithviraj Ammanabrolu, Alane Suhr, Yejin Choi, Hannaneh Hajishirzi, Sameer Singh, and Roy Fox. Do embodied agents dream of pixelated sheep?: Embodied decision making using language guided world modelling. In International Conference Machine Learning, 2023.
https:/t:://// http::///pp..ssattticchhho//o// semanticscholar ott.aaaiiiiiic..o..o.oooooroooor 256389514
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Abhishek Padalkar, Acorn Pooley, Ajinkya Jain, Alex Bewley, Alex Herzog, Alex Irpan, Alexander Khazatsky, Anant Rai, Anikait Singh, Anthony Brohan, et al. Open x-embodiment: Robotic learning datasets and rt-x models. arXiv preprint arXiv:2310.0883 2023.
Davide Paglieri, Bartiomiej Cupial, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Lukasz Kuciński, Lerrel Pinto, Rob Fergus, Jakob Nicolaus Foerster, Jack Parker- Parker-Holder, and Tim Rocktäschel. Balrog: Benchmarking agentic Ilm and vlm reasoning games, 2024. URL htt.:ps://aavv..rr/aaaassss/
13543
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL httph:ttp///// qwenllm g.t.t... wttbubbio/blooggwwee..... 5/
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. International Con- ference on Machine Learning, 2021. URL httts:://ppp. semanticscholar .rrg/CorpssDD 231591445
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model, 2024. URL
httpss//arxi..orr/bbs///// 18290
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac. Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055500 2024.
Juan Rocamonde, Victoriano Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision- language models are zero-shot reward models for reinforcement learning. arXiv preprint 10. 2023.
Peter Shaw, Mandar Joshi, James Cohan, Jonathan Berant, Panupong Pasupat, Hexiang Hu, Urvashi Khandelwal, Kenton Lee, and Kristina Toutanova. From pixels ui actions: Learning to follow instructions via graphical user interfaces. In Advances in Neural Information Processing Systems, 2023. URL luttps://aaxxii org/ /bo/ 00245
Noah Shinn, Federico Cassano, Beck Labash, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement (2023), arXiv preprint 0s.A112303.11366 2023.
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. arXiv preprint a3Xiv:22122131138, 2022.
Charles Burton Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language learning. ArXiv, abs/2206.11871, 2022. URL https://api. semanticscholat www.ooaarrrrrrrrrrooeoeee artg/CorpusD::24954400
14

---

Published as a conference paper at ICLR 2025
Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To cot or not to cot? chain-of- thought helps mainly math and symbolie reasoning, 2024. BtLhtttss///axiiv org/ abs/ 2409. 12183.
Nisan Stienmon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Haotian Sun, Yuchen Zhuang, Lingkai Kong, Bo Dai, and Chao Zhang. Adaplanner: Adaptive planning from feedback with language models. ArXiv, abs/2305. 2023. URL httpss
second edition, 2018. wwwreorrggppooo /ccC//oooo::::0944///
Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction. The MIT Press, http://mo....teeeeaaaeeeee/////a www.eeeewttteeeeeeeeeeeeeee book- 2nd. html
Richard S. Sutton, David A. McAllester, Satinder Singh, and Y. Mansour. Policy gradient methods for reinforcement learning with function approximation. In Neural Information Processing Systems, 1999. URL https:tp::// / //tp:/appp. api. semanticscholar. wwtp:/chhhoaaaaoooooooooooeo Ptrg//orpusID:: : 1211821
Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Rin Metcalf, Walter Talbott, Natalie Mackraz, R Devon Hjelm, and Alexander T Toshev. Large language models as generalizable policies for embodied tasks. In The Twelfth International Conference on Learning Representations, 2023.
Andrew Szot, Bogdan Mazoure, Harsh Agrawal, Devon Hjelm, Zsolt Kira, and Alexander Toshev, Grounding multimodal large language models in actions, arXiv preprint arXiv:2406. 07990, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint 2. 1805, 2023.
Hugo Touvron, Louis Martin, Kevin R. Stone. Peter Albert, Amjad Almahairi, and Yasmine Babaei et al. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307.09288, 2023. URL https://api. semanticscholar. wrw.orgggrroooo S 259950998
Jonathan Uesato, Nate Kushman, Ramana Kumar, H. Francis Song. Noah Yamamoto Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-based and outcome-based feedback, 2023. URL. /ttk/tsts////peeeeeiiee nww.eefornntrm
id
David Venuto, Sami Nur Islam, Martin Klissarov, Doina Precup, Sherry Yang. and Ankit Anand. Code as reward: Empowering reinforcement learning with vlms. ArXiv, abs/2402.04764, 2024.
URL https://api semanticscooolrrr wwwgeao.rrr////ee 2
267522976
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arX2v:2305.16291, 2023.
Yufei Wang. Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, and Zackory Erickson. RI-vim-f: Reinforcement learning from vision language foundation model feedback. arXiv preprint arXiv:2402.0368), 2024.
Christopher Watkins and Peter Dayan. Q-learning. Machine Learning, 8:279-292, 1992. URL wwwellarrrrrrrg/eeeeeeeeeeeeeeee artg/CorpusDD:22811133
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, F. Xia, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903. 2022. URL /tttps://pppi wwwanti www.laarrrggCoooussiii 246411221
Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2reward: Automated dense reward function generation for reinforcement leam- ing. ArXiv, abs/2309. 11.3099.14499 11489, 2023. URL https:: //api. semanticscholar. wwwphhhola..rrro////ee Arg/Corpus
262053612
15

---

Published as a conference paper at ICLR 2025
Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, Yang Wang, and Enhong Chen. Large language models for generative information extraction: A survey, 2024. hRtp:s//://aa www.g//////aa///e 2202s/22312... 17617
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, arXiv preprint arXiv:2210.06629, 2022.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference on Robot Learning (CoRL), 2019.
Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, Brian Ichter, Ted Xiao, Peng Xu, Andy Zeng, Tingnan Zhang, Nicolas Manfred Otto Heess, Dorsa Sadigh, Jie Tan, Yuval Tassa, and F. Xia. Language rewards for robotic skill synthesis. ArXiv, abs/2306.08647, 2023. URL httph: semanticscholaat wwwp/oorpssD:
www.aaawiitto.....oooooooooooooa 2591649006
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. ArXiv, abs/2401.10020, 2024. URL httpt::///ppp
semanticschol.a wwt httg//orpps.D:::: 267035293
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi musti-discipline multimodal under- standing and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 9556-9567, 2024.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances Neural Information Processing Systems, 35: 55:15476-1 14766-14888, 2022.
Shenao Zhang, Sirui Zheng, Shuqi Ke, Zhihan Liu, Wanxin Jin, Jianbo Yuan, Yingxiang Yang, Hongxia Yang, and Zhaoran Wang. How can Ilm guide rI? value-based approach. arXiv preprint arXiv:2402.1618/ 2024.
Andy Zhou, Kai Yan, Michal Shlapentokh-Rothmgn Haohan Wang, and Yu-Xiong Wang. Language agent search unifies reasoning acting and planning in language models. ArXiv, abs/2310.04466. 2023. URL htt.hhtt////// //api semanticschola ar wttpaa/rrrr/oooss//:22800000090
16

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1 ENVIRONMENT DETAILS
In our experiments, we investigate tasks from four different domains: MiniWob (Liu et al. (0018). NetHack (Küttler al. 2020), and Wordle (Lokshtanov Subercaseaux 2022), and Meta Merld (Yu
al. 2019). The observation space for all these environments is text, except fro Meta World which consists of RGB pixels.
In the MiniWob domain, we sample the subset of the five tasks which state-of-the-art results are low. Specifically, we carry experiments on: -lick-tab-i 1aab-2-hare 2-hard, beacck-chcckboee--ee count-shape, tic-tac-
and use -utocomplete. To learn RL policies from LLM-based rewards, we leverage the experimental setup of Shaw et al. (2023). In NetHack, we use the same environment and the same algorithmic setup as in Klissarov al. (2024). In Wordle, we build on the code made available by (Snell et al. 2022) and use their proposed subset of 200 words from the official list of the game. Finally, in Meta World we study the same subset of environments presented in (Wang et al. 2024) consisting of drawer- drawer-open-ve open-v2, soccer-v2 v2 and sweep sweep-into -ereep-int----eeee into-v2. Across all experiments where RL policies are learned, we use the original hyperparameter values defined in the respective experimental setups we are building upon.
A.2 DETAILS ON INDIRECT POLICY MODELING THRROGGH LLM- REWARDS
We the following prompt templates query the agent for AI feedback, Scalar Reward and Reward as Code across various environments. For the Embedding based approach, we use calculate the cosine similarity between the representation, provided by a BERT (Devlin al., 2019) sentence encoder (specifically the same otherwise we use the CL.IP encoder (Radford et al.) 202T). The similarity measured between the current observation and the same goal description contained in the each of the following prompts given for the other baselines.
Seiphrass--MnniA. -L3-2 model) when environments are text-based, and
MiniWob Prompt For Reward Modeling with Al feedback
will present you with two HTML descriptions irom interaction eavirenment.
ttask_deacription Write an analysis describing the semantics of each description strictly using information from the descriptions. Provide comparative analysis based on first principles. Finally, express a preference based on which description is the most likely to make some progress towards the goal, writing either ("best_descte dedeeriiiiiion: 1), ("best_descree deresptdescr;::ion" 2). You could also say ("best_dessripptionn None),
html atml_descriptionn:: (description_1)
html stml_description. (description_2)
Prompt
In Figure 7 we verify the importance of the chain-of- -fho-hghttthhtt prompting used for our Al feedback baseline. Results show no statistical difference in performance compared to using chain-of-thought. To understand this result, can refer to Sprague et al. (2024) which show that across 14 LLMs, chain-of-thee of-thought only significantly helps for mathematical and symbolic problems. We believe the way LLMs help the tasks studied in this paper (i.e., for generating feedback trajectories of experience) can be characterized as being part of the Commonsense and Knowledge categories from "Sprague et al. (2024), in which chain-of-thoughe does not improve performance.
17

---

Published as a conference paper at ICLR 2025
Wordle Prompt For Reward Modeling with Al feedback
will present you with two short ganeplay descriptions of Wordle, First, tell me about your knowledge of Wordle.
Mention the goal of wordle. Use the following information for Wordle states: black neans that the provided letter not present anywhere in the hidden word. yellow means that the provided letter is present somewhere in the hidden word, but not the correct position. green means that the provided letter is present in the hidden word exactly the correct position.
Then, write an analysis describing the semantics of each description strictly using information from the descriptions (which may be empty) and your knowledge of Wordle.
Provide comparative analysis based on first principles. Finally, express a preference based on which description is the most likely to make some progress towards the goal, writing either dbest_descripption: 1), ("bost_ddsccriptionn: 2). You could also say ("best_deseriptiion" None).
description_1: :description_1)
description_2: (description_2)
Prompt
NetHack Prompt For Reward Modeling with Al feedback
I will present you with two short gameplay descriptions of Nethack.
First, tell about your knowledge of NetHack.
Mention ion the goal of NetHack. Prefer agents that maximize the score in the game, for instance by killing monsters, collecting gold or going down the stairs in the dungeor.
Then, write an analysis describing the semantics of each description strictly using information from the descriptions (which may be empty) and your knowledge of NetHlckk
Provide comparative analysis based first principles.
Finally, express preference based which description is the most likely to nake some progress towards the goal, writing either 2).
("best_ cest_ddecripttinn.. 1), You could also say ("bost_description";; None),
description_1: [description_1]
description_2: ;description_2
Prompt
18

---

Published as a conference paper at ICLR 2025
NetHack Prompt For Online Reward Modeling with Al feedback
will present you with two short ganeplay descriptions of Nethack. First, tell me about your knowledge of NetHack.
Mention the goal of NetHack, Prefer agents that maximize the score in the game, for instance by killing monsters, collecting gold or going down the stairs in the dungeon.
Then, write analysis descriming the semantics each description strictly using information from the descriptions (which may be empty) and your knowledge of NetHack.
Provide comparative analysis based on first principles. Finally, express preference based on which description is the most likely make some progress towards the goal, writing either ("best_ ("best_descriae descriptiion 1), deeestt_dscritioon" 2).
You could also say ("best_descriptionn" None).
description_1; (description_1)
description_2: [description__]
Prompt
MetaWorld Prompt For Reward Modeling with Al feedback
Does the image satisfy (ourrent_task)?
image_l: limage_i) [1im_response]
Does the image satisfy (current_task)?
image_2 2: [image
[ilm response)
Prompt
A.3 DETAILS ON DIRECT POLICY MODELING
We present the exact prompts used to query GPT-4o for each of the domains we have considered.
These are presented through Prompt 13 15
and 16
Additionally, in Figure g we ablate the prompting techniques used in our direct policy modeling approach. Results show that combination of all prompting techniques presented in Section 2. works best.
A.4 ADDITIONAL INDIRECT POLICY MODELING METHODS
There are number of other prompting methods for extracting information or knowledge from LLM that may relevant to solving RL tasks.
Direct State Generation. The model generates tokens that will represent next states (or ather-future-tinn states). This is similar to world modeling. The next state prediction can be conditioned action, marginalized over policy distribution.
19

---

Published as a conference paper at ICLR 2025
Normalizedd Performance
20-
MiniWob-Hard
Wordle
NetHack
MetaWorld
Al Feedback
Al Feedback (no CoT)
Figure Ablation on the usefulness chain-of-thought in the Feedback baseline.
100
Chain-gf-Thught I--oontet Learning Salf- Salf-Bafmmment
Chain Dhin-of-Thought In-Context Learning
Thought
Normalized Performance
80
40
20
Figure 8: Ablation the set of prompting techniques used for direct policy modeling. The reported performance averaged over all domains and tasks.
20

---

Published as a conference paper at ICLR 2025
MiniWob Prompt For Reward Modeling with Scalar Reward
will present you with an HTML descriptions from web interaction environment.
"task_descriptionn
Write an analysis describing the semantics of the description strictly using information from the description.
Finally, output scalar value between and 5, with higher values correlation with progress towards the goal.
html_ html_description: (deacription)
Prompt
Wordle Prompt For Reward Modeling with Scalar Reward
I will present with gameplay description of Wordle.
First, tell me about your knowledge of Wordle.
Mention the goal of wordle. Use the following information for Wordle states: black means that the provided letter is not present anywhere that the provided letter is present
in the hidden word. yellow scmewhere in the hidden word, but not the correct position. green means that the provided letter is present in the hidden word exactly at the correct position.
Write an analysis describing the semantics of the descripticn strictly using information from the description. Finally, output scalar value between and 5, with higher values correlation with progress towards the goal. description: [description)
Prompt
High-level plans. Ask the LLM to generate a high-level plan. This can similar to direct state generation, but potentially relies some subset of features (for examples the inventory in MineCraft) and is typically temporally extended.
Action Preference. Ask the LLM to select, among two choices, the most likely action given previous and future observations.
State Preference. Ask the LLM to select, among two choices, the most likely next state observation conditioned on prior history and/or actions.
Many of the above could theoretically used to construct policy, yet full implementation out of scope from this paper due to the lack of available code-bases to build upon and we do not seek to build new algorithms from scratch. For instance, generating high-level plan can be constructed in various ways: ranging from in-context learning to hierarchical reinforcement learning agents, with many variables to be decided upon. For example, what the space over which plans are established? Is it the full observation space, subset of (c.g., in Minecraft, should it be pixel space the inventory)? ? What action space should be considered (c.g., web agents tasks should it low level mouse click and drag, human-crafted high levels
in many baselines?)? Additionally, the
21

---

Published as a conference paper at ICLR 2025
NetHack Prompt For Reward Modeling with Scalar Reward
will present you with gameplay description of Nethack.
First, tell me about your knowledge of NetHack.
Mention the goal of NetHack, Prefer agents that maximize the score in the game, for instance by killing monsters, collecting gold or going down the stairs in the dungeon.
Write analysis describing the semantics of the descriptien strictly using information from the description.
Finally, output scalar value between and 5, with higher values correlation with progress towards the goal. description: (description)
Prompt
MetaWorld Prompt For Reward Modeling with Scalar Reward
From to 5, how much does the image cchieve(currentt image: (image)
Prompt 9
MiniWob Prompt For Reward Modeling with Reward as Code
will present you with HTML descriptions from interaction environment.
jaakk_description
Write an analysis describing the semantics of the descriptions strictly using information from the descriptions.
Finally, write code that, when executed, will help make progress towards the goal.
html atml_descriptions: (descriptions)
Prompt 10
choice of these variables might not easily generalize across many environments, which is why decided not to focus on this direction. We do believe that this is particularly important direction for future work.
However, in Figure 2b we perform investigations into capabilities LLMs to perform Action Preference and State Preference. The results show that current LLMs struggle to achieve strong performance on any of these tasks. Additionally, Table I we report the accuracy with which LLMs directly predicts the next observation (Direct State Generation), providing probe into their direct world modeling capabilities. Results show limited performance, except on Mini Wob- Hard tasks, which are fully observable and encode deterministic transitions.
22

---

Published as a conference paper at ICLR 2025
Wordle Prompt For Reward Modeling with Reward as Code
will present you with gameplay descriptions Wordle.
First, tell me about your knowledge of Wordle.
Mention the goal of wordle. Use the following information for Wordle states: black neans that the provided letter not present anywhere in the hidden word. yellow means that the provided letter is present somewhere in the hidden word, but not at the correct position. green means that the provided letter is present in the hidden word exactly the correct position.
write an analysis describing the semantics of the descripticns strictly using information from the description. Finally, write code that, when executed, will help make progress towards the goal. descriptions: (descriptions)
Prompt 11
NetHack Prompt For Reward Modeling with Reward as Code
I will present you with gameplay descriptions of Nethack.
First, tell about your knowledge of NetHack.
Mentien the goal of NetHack. Prefer agents that maximize the score in the game, for instance by killing monsters, collecting gold or going down the stairs in the dungeor.
Write an analysis describing the semantics the descriptiens strictly using information from the descriptions. Finally, write a code that, when executed, will help make progress towards the goal. descriptions: (descriptions)
Prompt 12
A.5 ABLATING REWARD AS CODE
In Table ablate the performance of the Reward as Code baseline across LLMs, observation spaces and additional assumptions. For pixel observations, we follow the methodology laid out in
Accuracy
MiniWob-Hard
65 11.4%
Wordle
28 + 8.3%
NetHack
0.0 + 0.0%
MetaWorld
N/A
Table 1: LLMs struggle to predict the next observation. We show the decreasing accuracy of the LLM to predict the next observation with increasing task complexity. LLMs are unable to generate pixel observations, which are used in MetaWorld.
23

---

Published as a conference paper at ICLR 2025
MiniWob-Hard Prompt For Direct Policy Modeling
have an autononous computer control agent that can perform atomic instructions specifies by natural language to control computers. There are two types of instructions it can execute.
First, given the Instruction that matches the regular expression, "Atype. {1,
Second, given the instruction that matches the regular expression, "*clickxpp "nclickppahhls.2> is visible the webpage. The target of this instruction should be valid xpath.
it can click an HTML element with an xpath that
Below is the HTM code of the webpage where the agent should solve task.
httmi_observation)
Examples:
task: [example_task]
plan: [example_plan)
Current task: Enter an item that starts with Antiänd ends with dar Think step-by-step before answering, what is the current plan? [1lm_plan)
Repeat N times:
Find problems with this plan for the given task compared the example plans.
[ilm_criite cii__criticimm
Based on this, what the plan for the agent to complete the task?
Below is the HIML code of the webpage where the agent should solve task. hhtml_obaervattonn
Current task: Enter an item that starts with Äntiänd ends with dar Think step-by-step before answering, what is the current plan?
{11m plan]
Prompt 13
(Venuto et al. 2024), whereas for proprioceptive observations we follow the one from YYu al. 2023). Both methods heavily depend on access to state-of-the-art, closed-source model to achieve performance comparable to that Al Feedback, which uses the smaller, open-source model of Paligemma (Beyer et al.) 2024). Additionally, each method requires expert demonstrations or specialized domain knowledge to guide the reward design process. While these assumptions may be viable in certain situations, such in controlled simulation environment, they present significant practical challenges in more general contexts. In contrast, Al Feedback operates by simply comparing observations and reasoning using chain-of-thought approach.
24

---

Published as a conference paper at ICLR 2025
Wordle Prompt for Direct Policy Modeling
Let's play a gane of Wordle. You will have to guess the words and will give you the colors.
Use the following information for Wordle colors: black means that the provided letter is not present anywhere the hidden word.
yellow means that the provided letter is present somewhere in the hidden word, but not at the correct position.
green means that the provided letter is present in the hidden word exactly at the correct position.
You can choose among this list of words: (list_of_words)
Here are examples trajectories, containing past observations and actions, together with an appropriate action.
Example 1:
Trajectory: (example_traeestory)
Action: (example_action)
Example 2:
Irajectory: (example_trajectory
Action: (example_action)
Current trajectory:
(trajectory so so_far]
Think step-by-step before answering, what should be the current action? (llm_action)
Repeat N times:
Find problems with this action for the given task compared the example actions.
[imm Lim_criiii
#ased on this, what the action for the agent to make progress the task?
Current trajectory: (trajectory sorry___________e _arr
Think step-by-step before answering, what should the current action? [1lm_ lltionttion
Prompt 14
A.6 LEARNING FROM ENVIRONMENT REWARDS
In Figure we compare the performance an RL. agent trained using reward function derived from AI feedback with that of an agent trained on human-designe rewards across different environments.
observe that AI feedback achieves comparable results, with an average score of 89.93 versus 86.3 for the human-designed reward. The objective of this experiment is not to argue that LLM-based rewards consistently outperform numan-crafted ones ssi--since expert human knowledge can always be encoded into reward function but rather contextualize the performance of LLM-based rewards. Notice that for Meta World we report the performance after fine-tuning the LLM as described in Section 5
25

---

Published as a conference paper at ICLR 2025
NetHack Prompt for Direct Policy Modeling
Let's play the gane of NetHack.
First, tell me about your knowledge of NetHack. Montion the goal of NetHack.
Prefer maximizing the score in the game, for instance by killing monsters, collecting gold or going down the stairs in the dungeon.
Here examples sub-trajactories, containing past observations and actions, together with an appropriate action.
Example 1:
sub-Irajectory; texample_ssb-brrreeetttoy
Action: [example_action]
Example 2:
sub-Trajectory: texample_sub-ttrjjattoryy
Action: (cxample_action)
Current sub-trajectozy: (eub-trajectory_so__)
Think step-by-step before answering, what should be the current action? (11m_action)
Repeat N times:
Find problems with this action for the given task compared to the example actions.
_1lm_cri cil__criticism]
Based on this, what is the action for the agent to make progress the task?
Here is the current sub-trafectory, containing past observations and actions: (sub=trrajettory____ffrr
Think step-by-step before answering, what should the current action? (11m_action)
Prompt 15
A.7 AI FEEDBACK AND HEURISTIC FUNCTIONS
While prior works have shown that rewards can be extracted from a language model (Kwon et al. 2023 Brooks al. 2024 Klissarov et al. 2024). it can be more generally thought of as encoding heuristic function h. The function contains high-level, multi-step information about the MDP M. To extract
it, one can solve the re-shaped MDP M with F(st, a.) = r(st, a.) + (1 X) ls. [h(+++1] and 7 Xry where À E jo, 1] Cheng et al. (2021) Solving M yields policy '² that is also optimal in its value function's bias can he shown to converge to / M as a function of [| Vol...
Specifically, assume access the reshaped MDP M, can learn new policy which optimizes with À [0, 1]. Equation 5 shows the performance difference lemma Kakade & Langford (2002) as a function of true and reshaped MDP quantities:
initial dataset Do. from which heuristic h can be computed. In
26

---

Published as a conference paper at ICLR 2025
MetaWorld Prompt for Direct Policy Modeling
You are controlling robot for the following task:
[neta_world_task]
Here are examples sub-trajectories, containing past observations and actions, together with an appropriate action.
Example 1:
sub-Irajectory. /exmpple_suuttaaeettay
Action: (example_action
Example 2:
sub-Irajectory; jexampleessub-ttajjcctoory
Action: (example_action)
Current sub-trajectozy: (aub-trajectory_so_F
Think step-by-step before answering, what should be the current action? (llm_action)
Repeat N times:
#ind problems with this action for the gi ven compared to the example actions.
{1mm criticism)
Based on this, what the action the agent to make progress on the task?
Here is the current sub-trajectory, containing past observations and actions: (sub-trajectory_so___
Think step-by-step before answering, what should be the current action? (ilm_action)
Prompt 16
((m,h) =Ep, [E>,(V(s)) V=)(s]
=q (*(s)= - (T()) + og (*(()= - v*()) + cpr - (5)
where C1, C22 C3 are non-negative constants. Minimizing L(T, h) with respect to π and h can be achieved by minimizing each individual term. In particular, the red term suggests that the heuristic h has to be updated on data from 2= order to not become "stale". This points out shortcoming of existing LLM-as LLM-as-critie algorithms, which sometimes fix h after distilling the language model knowledge into it Klissarov et al. (2024)
These theoretical findings suggest, in particular, that heuristic h (in
the Bradley- Terry prefer- ence model), has to be updated with on-policy samples, similarly to empirical results from Figure 5
A.8 ADDITIONAL CONSIDERATION FOR PREFERENCE- BASED REWARD MODELING
In Figure I0 we present the properties that were important to obtain effective exploration on NetHack, without the counting term shown in Equation 4
27

---

Published as a conference paper at ICLR 2025
Reward as Code RGB Observations
GPT-4o
w/o expert demonstration
0% - 1% 79% = 7%
with expert demonstration
Reward Code Proprioceptive Observations
Llama 70B
w/o background functional knowledge with background functional knowledge
0% - 1% 10% = 3%
GPT-4o
w/o background functional knowledge with background functional knowledge
5% 3% 76% + 6%
Al Feedback RGB Observations
PaliGemma
72% + 8%
Table AI Feedback performs on par with Reward as Code, without proprioceptive observa- tions expert demonstrations. To match Al Feedback performance on Metaworld, Reward as Code requires GPT-4o level knowledge, augmented with either in-context expert demonstrations or proprioceptive observations.
100
Normalized Performance
MiniWob-Hard
Wordle
NetHack
MetaWorld
Al Feedback
Environment Reward
Figure 9: Comparison between the best performing LLM-based reward (AI Feedback) and human designed rewards for each domain.
Score
1.0
Normalized Performance
Scout
Dungeon Level
0.4
3.0
Preferences Non-Markovian Model & Markovian Model & Non-Markovian Model & Markovian Model
Offline
Figure 10: Successful exploration on Nethack depends on both online preference elicitation and Mar-Markovian reward function.
28

---

Published as a conference paper at ICLR 2025
A.9 IN-CONTEXT LEARNING FOR REWARD MODELING
In Figure II we present variation on the Wordle game where the color code has been altered, which we refer to as Eldrow (reverse Wordle). Under this transformation, the off-the- model provides feedback that correlates very poorly with the optimal value function. When we measure the perplexity of the LLM obtain a value of 6.97 which higher than the one measured on the standard rule set of Wordle, with value of 5.06. Given that the difference in values is not very large, we leverage the simplest way for adapting the LLM: through in in-context learning. As shown in Figure [ib by providing hints in the prompt about the new rule set, the LLM adapts its preferences and generates a Bradley- Terry model that recovers the correlation values we witnessed in
natural language description of the new rule set of Eldrow (see Appendix A we
0.2
1.0
Cor Va)
Corr(BTuw ve)
0.1
0.8
Correlation
0.0
Correlation
0.6
0.1
-0.2
-0.3
0.2
0.0
0.2
0.6 0.7 1.0
0.2 0.4 0.5 Policy degree of optimality
0.7 1.0
Policy π degree of optimality
(a) Before in-context learning
(b) Afler in-context learning
Figure 11: Al feedback can be adapted to novel settings through in-context learning. While the original LLM does poorly on Eldrow due to dutoof-distributinn, it manages to correct its feedback the task using in-context hints.
A.10 LLMS AS NOVELTY DETECTORS
We hypothesize that LLMs with long contexts can effectively act as novelty detectors. Within the scope of RL problems, this implies the ability to tell, for example, whether sub-trajectory is contained in the replay buffer.
To test this, query Gemini-1.5 Pro (Team et al., 2023) with context video containing 500 frames of an agent exploring the bottom-left room (Higure left) and a single frame sampled uniformly at random from query episode which covers in the top-right room, center and bottom of the maze (Figure middddee.. We ask the LLM to identify novel query states, i.e. states which are not seen in the context episode. We then train direct predictor (3-layer MLP) estimate the probability of any state on the grid to be novel with respect to the context (Figure 12 right). The language model correctly identifies the top-right portion of the trajectory to be novel, knowledge which could then be used to construct an intrinsic reward function.
A.11 INVESTIGATING A LARGER SET OF LLMS
In Figure 13 we investigate a larger set of LLMs for the LLM Policy method. We run experiments using Claude 3.5 Sonnet and Gemini Pro 1.5 across all environments. Results indicate that the performance of these frontiers models is generally the same, mirroring recent results from Paglieri ct al. (2024). In Figure [4] we extend the of LLM studied for the AI Feedback method, which is the best performing method from the reward modeling approaches. We investigate the Qwen 2.5 model on MiniWob- Wob-Hard, Wordle and NetHack (as they are text-based domains), and the Llama 3.2-V model on Meta World (pixel-based). Once again, the results show little difference in performance across most models, except for the comparison on the MetaWorld domain where the Llama 3.2-V model performs significantly better than PaliGemma.
29

---

Published as a conference paper at ICLR 2025
Context Trajectory
Query Trajectory
LLM-based Novelty Reward
Figure 12: LLMs can capture observation novelty. Given the context trajectory (red), and single observation sampled uniformly random from the query trajectory (blue), the LLM correctly identifies novel states that are seen in the query but not in the context (green).
162
Normalized Performance
MinlWbb-Haadd
Wordle
NetHlack
Metoooorld
EPii4o
Gemini Pro Exp-114
Claude 3.5 Summet
Figure 13: Comparison between different frontier closed-soures models as used within the LLM Policy method.
Normalized Performance
80
MiniWob-Hai
Wordle
NetHack
MetaWorld
Llama. 3
Qwen 2.5
PaliGemma
Liama 3.2-V
Figure 14: Comparison between the Llama model and Qwen 2.5 on the three text-based domains "MiniWob-Hard, Wordle and NetHack) as well comparison between the PaliGemma model and Llama 3.2-V V on the pixel-base domain (MetaWorld). We only notice significant difference in performance between PaliGemma and Llama 3.2-V, which also present quite different architectures and pre-training regimes.
A.12 REWARD AS CONTEXT FOR LLM POLICY
In Figure 15, we verify whether providing the LLM-based reward as additional context to the LLM Policy improve its performance.
A.13 LLM FINI-TUNING EXPERIMENTS
In this section, we investigate how we could fine-tune an LLM for direct policy modeling in the environments have studied. During this process, control the amount of samples seen during
30

---

Published as a conference paper at ICLR 2025
200
Normalized Performance
80
60
40-
20-
MiniWob-Hard
Wordle
NetHack
MetaWorld
LLM Policy
LLM Policy with Reward Information
Figure 15: Comparison between the LLM Policy described in Section 2.2 and a variation, LLM Policy with Reward Information, in which provide the LLM outputting actions in the environment with the reward defined from the Al Feedback method.
Enviromment Samples
Direct Policy (only RL)
Direct Policy (only SFT)
Direct Policy (RL after SFT)
Al Feedback
Success Rate
0
0.00
0.00
0.92
0.00
128k
(1.01
0.10
0.80
0.22
0.13
0.48
IM
0.09
0.87
0.65
0.72
Floating Point Operations (FLOPS)
0
0
3.65 x 10a7
6.9 x 10
128k
7.36 x 10
4.38 x 10
4.38 x 10
4.09 x 10
256k
1.46 10"7
5.11 x 10"7
5.11 10
8.109 10's
IM
7.12 10a?
1.09 x 10a8
1.09 10
3.145 10
Table 3: FLOPS for Different Policies and Environment Samples.
optimization and the amount of FLOPS used to fine-tune the LLM. Specifically, we fine-tune the direct policy method under three settings: only with online RL, only with SFT, and online RL training on an SFT-pretrained model. We compare these to the performance of the Al feedback method, which has shown to the best performing reward modeling method. For AI feedback, we calculate the FLOPS used by the RL algorithm, as well as any computation spent on the LLM for inference fine-tuning.
Results in Table 3 indicates that indirect policy modeling achieves comparable performance to direct policy modeling trained with SFT at fraction of the computational cost, i.e. two orders of magnitude lower. Policy modeling achieves higher success rate with million training samples, but is otherwise significantly less efficient than indirect policy modeling. Importantly, direct policy modeling with SFT relies on a large expert dataset to achieve good performance. On the other hand, direct policy modeling trained only with RL shows poor performance with 1M samples and high sensitivity to hyperparameters. This is line with Annonymoss) (2024) reporting low sample efficiency when training LLMs with online RL, without SFT, for sequential decision-making
31


Reviews:
[{"review_id": "LVmXIVJld6", "reviewer": "Reviewer_57NJ", "summary": "In this paper, the authors study how Large Language Models (LLMs) can produce decision-making policies, either by generating actions or by creating reward models for reinforcement learning (RL). The authors use experimental results to reveal that LLMs excel at reward modeling, particularly when using AI feedback, and fine-tuning with synthetic data enhances performance in unfamiliar environments, helping prevent catastrophic forgetting.", "questions": "1. When reporting the final results in Fig 1, Fig 2, and Fig 6, do the authors control the number of interactions with the environment? I'm also interested in understanding how performance changes with an increasing number of interactions, as shown in the learning curves typically found in reinforcement learning (RL) papers.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors perform extensive experiments to evaluate the capabilities of large language models (LLMs) in sequential decision-making tasks. Specifically, they demonstrate that using LLMs to generate preference data and then train a reward model for RL training has the highest performance gain compared with other methods such as directly modeling policy by LLMs. This conclusion seems interesting and helpful for future works in designing RL agents incorporating LLMs.\n\n2. The experiments in Fig. 3 and Fig. 4 clearly illustrate the effect of the rewards learned through the LLM Feedback.\n\n3. It is interesting that the authors show that prompting engineering can steer LLMs for active exploration on the \nNetHack task, which approximates the count-based exploration bonus.", "weaknesses": "1. The related works section is very incomplete. The authors should discuss more recent works that study LLM decision-making problems and self-rewarding of LLMs such as [1]-[7]. The introduction of LLM for better reward design on RL is also studied in [8] and should be discussed carefully.\n\n2. The authors compare direct and indirect policy modeling and use experiments to show that indirect policy modeling attains higher performance in multiple tasks. However, I am a little unconvinced about the experimental configuration, where the authors only query the GPT for the direct policy modeling but train an RL agent with multiple steps for the indirect policy modeling. It seems that the computation cost and the number of interactions with the environment are not controlled. Moreover, it has been shown in ([1]-[3]) that LLM can also serve as a critic model to give a numerical estimated value in the direct policy modeling. Maybe the authors should also add an experiment to analyze what would happen if the estimated reward from the indirect policy modeling were incorporated into the direct policy modeling. In this case, no RL training is involved and computation cost is easier to control.\n\n[1] Zhou, Andy, et al. \"Language agent tree search unifies reasoning acting and planning in language models.\" arXiv preprint arXiv:2310.04406 (2023).\n\n[2] Sun, Haotian, et al. \"Adaplanner: Adaptive planning from feedback with language models.\" Advances in Neural Information Processing Systems 36 (2024).\n\n[3] Liu, Zhihan, et al. \"Reason for future, act for now: A principled architecture for autonomous llm agents.\" Forty-first International Conference on Machine Learning. 2023.\n\n[4] Huang, Jen-tse, et al. \"How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments.\" arXiv preprint arXiv:2403.11807 (2024).\n\n[5] Park, Chanwoo, et al. \"Do llm agents have regret? a case study in online learning and games.\" arXiv preprint arXiv:2403.16843 (2024).\n\n[6] Nottingham, Kolby, et al. \"Do embodied agents dream of pixelated sheep: Embodied decision making using language guided world modelling.\" International Conference on Machine Learning. PMLR, 2023.\n\n[7] Yuan, Weizhe, et al. \"Self-rewarding language models.\" arXiv preprint arXiv:2401.10020 (2024).\n\n[8] Kwon, Minae, et al. \"Reward design with language models.\" arXiv preprint arXiv:2303.00001 (2023).\n\n3. Minor typos: Line 762: \"except fro\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "A8wa8k5j5F", "reviewer": "Reviewer_3Jq6", "summary": "This paper studied the capabilities of Large Language Models (LLMs) for reinforcement learning (RL) in interactive decision-making tasks, by directly producing policies or indirectly generating rewards for policy optimization.", "questions": "See weakness.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "1. The paper is well-written and easy to follow.\n2. The experiments are conducted on various environments.", "weaknesses": "1. The motivation of this paper is limited. One conclusion of the paper is that leveraging the feedback from the LLM to optimize the policy is better than using the LLM directly as a policy. However, since the latter is fine-tuning free, it is not surprising that the former is better. Even if the AI feedback is noisy, as long as the overall feedback is accurate, fine-tuning the policy against it can obtain some improvements.\n2. The policy modeling and reward modeling abilities of LLMs in decision-making tasks have been widely studied in previous works [1,2]. No new methods are proposed in this paper, and the novelty is thus limited compared to these works.\n3. The analysis is also not convincing. The authors claim that AI feedback helps better credit assignments. However, given only the training curves, it is not clear when and why credit is better assigned.\n\n[1] Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models.\\\n[2] Ma et al., Eureka: Human-Level Reward Design via Coding Large Language Models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FSOrrFGJtS", "reviewer": "Reviewer_8WYf", "summary": "This paper explores the potential of large language models (LLMs) for tackling complex sequential decision-making problems in reinforcement learning (RL). The authors investigate two approaches: using LLMs to directly generate actions or indirectly generate reward models that guide RL agent training. Their findings demonstrate that even without specialized fine-tuning, LLMs excel at reward modeling. Notably, leveraging AI feedback to craft rewards emerges as a highly effective and generalizable strategy. This approach enhances performance by improving both credit assignment and exploration within the RL framework. Furthermore, the authors address the challenge of unfamiliar environments by demonstrating that fine-tuning LLMs with synthetic data significantly boosts their reward modeling capabilities. Importantly, this fine-tuning process effectively mitigates catastrophic forgetting, preserving the LLM's broad knowledge base.", "questions": "- It seems that this work has some preliminary requirements for the LLMs to make it function well, e.g., chain-ot-thought (CoT), in-context learning, and self-refinement, as described in Section 2.1. How important are these components to the sequential decision-making capabilities of the LLMs? How do they affect the direct or indirect policy modeling of LLMs? \n- Under many real-world scenarios, we often expect a timely or frequent decision-making ability of the agent. When using LLMs directly or indirectly during the decision-making process, how can we guarantee that the decision is made timely yet effective?\n- How much expert data do you use to fine-tune the PaliGemma model in Section 5? How about a different dataset quality, will there be a significant difference? I would expect a more in-depth discussion on fine-tuning LLMs here", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "## Pros\n\n- The studied topic is very interesting. The authors explore the sequence modeling capability of large language models in the context of reinforcement learning, which can be of interest to a large number of researchers in the community\n- The authors study either directly using LLM as the policy or leveraging LLM for decision-making and conduct numerous experiments\n- The authors show that without task-specific fine-tuning, current LLMs only show limited decision-making capabilities when directly generating actions. Furthermore, the authors find that AI-feedback-based rewards produce dense functions that correlate positively with high-quality value functions. Such reward functions can significantly reduce the difficulty of assigning credit by redistributing rewards across different steps within a trajectory. Some of these observations and conclusions are interesting and can be useful for researchers, e.g., using LLM to directly output scalar reward signals can be surprisingly good on some tasks\n- The results reported are averaged over 10 seeds, which is great considering the unstable nature of RL", "weaknesses": "## Cons\n\n- This paper does not propose any new method and looks more like an empirical paper that investigates the application of LLMs in either direct policy modeling or its capability of facilitating policy learning. Although I agree that such kind of paper is also of importance to the community, its technical novelty is still somewhat weak.\n- Some conclusions derived by the authors are not surprising. For example, in environments with unfamiliar dynamics, fine-tuning LLMs with synthetic data can significantly improve their reward modeling capabilities while mitigating catastrophic forgetting. Many of the observations are actually known or acknowledged by many scholars and this paper seems to verify them through some designed experiments\n- Another concern is that the conducted experiments may not be necessarily sufficient to back up the observations and conclusions in this paper. I list some of them below\n  - the studied topics are limited and some selected topics are too broad to be covered in depth within a 10-page ICLR paper. The considered experimental settings are limited as described in Section 3 (lines 207-215), i.e., the authors only study approaches that can directly interface with the action space supported in the environment and assume that LLMs are used without any gradient update. When dealing with indirect policy learning, the authors only consider reward as code. Yet, there are many other options, e.g., LLMs to execute high-level plans for the agent, etc. Furthermore, the authors only consider a simplified version of reward as code without introducing some extra components or tricks as recent works do [1, 2, 3]. When using LLMs for exploration, the authors do not consider the possibility of letting LLMs write intrinsic reward functions, constructing intrinsic rewards [4, 5], etc.\n  - the number of the base LLMs is limited. The authors used the closed-source GPT-4o model for direct policy modeling and the open-source Llama 3 for indirect policy modeling when environment observations consist of text, and PaliGemma when environment observations consist of pixel images. The number of LLMs is quite limited, making it hard to tell the significance and applicability of the observations and conclusions\n  - the authors do not compare many baselines in this paper, e.g., the authors only consider the count-based exploration method in Section 4.2 (Figure 5). It would be better to compare against stronger baselines.\n\n[1] Text2reward: Automated dense reward function generation for reinforcement learning\n\n[2] Auto mc-reward: Automated dense reward design with large language models for minecraft\n\n[3]  Eureka: Human-level reward design via coding large language models\n\n[4] Guiding pretraining in reinforcement learning with large language models\n\n[5] World Models with Hints of Large Language Models for Goal Achieving", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MFgH6OUTLF", "reviewer": "Reviewer_9H9A", "summary": "The paper investigates how LLMs can be leveraged for RL, comparing direct policy generation versus indirect reward modeling approaches across diverse domains. The authors find that using LLMs to generate reward models, particularly through AI feedback preferences, yields better and more consistent performance compared to direct policy generation. They also explore fine-tuning approaches and analyze how LLM-based rewards can help address core RL challenges.", "questions": "- The proposed method relies on consistent preference feedback from LLMs to construct reward signals, but doesn't adequately address cases of preference uncertainty. Consider a scenario where the LLM assigns similar preference probabilities to different states, or worse, provides inconsistent rankings when the same state pair is presented multiple times with slightly different prompts. \n\t- So, what happens when the LLM expresses low confidence or contradictory preferences? \n\t- Could inconsistent preferences lead to unstable reward signals that harm policy learning?\n- The paper demonstrates the method primarily on environments where progress is relatively obvious (e.g., clear Wordle feedback, discrete game states in NetHack). However, many real-world tasks involve subtle, continuous progress where improvements may be hard to detect from observations. I have several concerns regard this problem:\n\t- How sensitive is the LLM's preference detection to small state changes?\n\t- Could the method miss important incremental progress by only detecting large, obvious changes?\n- Sec 4.1 shows some evidence that AI feedback can help with credit assignment, the underlying mechanism isn't clear. In complex tasks, progress often results from a sequence of coordinated actions rather than single decisions. Can the method distinguish between critical and auxiliary actions when both contribute to the final outcome?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Comprehensive empirical evaluation across diverse domains.\n- Interesting exploration of fine-tuning trade-offs between direct and indirect approaches.\n- The paper is well written and easy to follow.", "weaknesses": "See the questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)