On the Modeling Capabilities of Large Language Models for Sequential Decision Making

Authors: Martin Klissarov, R Devon Hjelm, Alexander T Toshev, Bogdan Mazoure
Primary area: reinforcement learning
Keywords: reinforcement learning, large language models, ai agents, preference based learning, reward design

Abstract:
Large pretrained models are showing increasingly better performance in reasoning and planning tasks across different modalities, opening the possibility to leverage them for complex sequential decision making problems. In this paper, we investigate the capabilities of Large Language Models (LLMs) for reinforcement learning (RL) across a diversity of interactive domains. We evaluate their ability to produce decision-making policies, either directly, by generating actions, or indirectly, by first generating reward models to train an agent with RL. Our results show that, even without task-specific fine-tuning, LLMs excel at reward modeling. In particular, crafting rewards through artificial intelligence (AI) feedback yields the most generally applicable approach and can enhance performance by improving credit assignment and exploration. Finally, in environments with unfamiliar dynamics, we explore how fine-tuning LLMs with synthetic data can significantly improve their reward modeling capabilities while mitigating catastrophic forgetting, further broadening their utility in sequential decision-making tasks.

Decision: accept (poster)
Reviewer scores: 5, 5, 6, 6

Reviews and discussion:
<thread>
creation date:
1730387774714

thread_type:
official_review

<thread_object>
creator: reviewer_9h9a

<content>
summary:
The paper investigates how LLMs can be leveraged for RL, comparing direct policy generation versus indirect reward modeling approaches across diverse domains. The authors find that using LLMs to generate reward models, particularly through AI feedback preferences, yields better and more consistent performance compared to direct policy generation. They also explore fine-tuning approaches and analyze how LLM-based rewards can help address core RL challenges.

strengths:
- Comprehensive empirical evaluation across diverse domains.
- Interesting exploration of fine-tuning trade-offs between direct and indirect approaches.
- The paper is well written and easy to follow.

weaknesses:
See the questions below.

questions:
- The proposed method relies on consistent preference feedback from LLMs to construct reward signals, but doesn't adequately address cases of preference uncertainty. Consider a scenario where the LLM assigns similar preference probabilities to different states, or worse, provides inconsistent rankings when the same state pair is presented multiple times with slightly different prompts. 
	- So, what happens when the LLM expresses low confidence or contradictory preferences? 
	- Could inconsistent preferences lead to unstable reward signals that harm policy learning?
- The paper demonstrates the method primarily on environments where progress is relatively obvious (e.g., clear Wordle feedback, discrete game states in NetHack). However, many real-world tasks involve subtle, continuous progress where improvements may be hard to detect from observations. I have several concerns regard this problem:
	- How sensitive is the LLM's preference detection to small state changes?
	- Could the method miss important incremental progress by only detecting large, obvious changes?
- Sec 4.1 shows some evidence that AI feedback can help with credit assignment, the underlying mechanism isn't clear. In complex tasks, progress often results from a sequence of coordinated actions rather than single decisions. Can the method distinguish between critical and auxiliary actions when both contribute to the final outcome?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730661279185

thread_type:
official_review

<thread_object>
creator: reviewer_8wyf

<content>
summary:
This paper explores the potential of large language models (LLMs) for tackling complex sequential decision-making problems in reinforcement learning (RL). The authors investigate two approaches: using LLMs to directly generate actions or indirectly generate reward models that guide RL agent training. Their findings demonstrate that even without specialized fine-tuning, LLMs excel at reward modeling. Notably, leveraging AI feedback to craft rewards emerges as a highly effective and generalizable strategy. This approach enhances performance by improving both credit assignment and exploration within the RL framework. Furthermore, the authors address the challenge of unfamiliar environments by demonstrating that fine-tuning LLMs with synthetic data significantly boosts their reward modeling capabilities. Importantly, this fine-tuning process effectively mitigates catastrophic forgetting, preserving the LLM's broad knowledge base.

strengths:
## Pros

- The studied topic is very interesting. The authors explore the sequence modeling capability of large language models in the context of reinforcement learning, which can be of interest to a large number of researchers in the community
- The authors study either directly using LLM as the policy or leveraging LLM for decision-making and conduct numerous experiments
- The authors show that without task-specific fine-tuning, current LLMs only show limited decision-making capabilities when directly generating actions. Furthermore, the authors find that AI-feedback-based rewards produce dense functions that correlate positively with high-quality value functions. Such reward functions can significantly reduce the difficulty of assigning credit by redistributing rewards across different steps within a trajectory. Some of these observations and conclusions are interesting and can be useful for researchers, e.g., using LLM to directly output scalar reward signals can be surprisingly good on some tasks
- The results reported are averaged over 10 seeds, which is great considering the unstable nature of RL

weaknesses:
## Cons

- This paper does not propose any new method and looks more like an empirical paper that investigates the application of LLMs in either direct policy modeling or its capability of facilitating policy learning. Although I agree that such kind of paper is also of importance to the community, its technical novelty is still somewhat weak.
- Some conclusions derived by the authors are not surprising. For example, in environments with unfamiliar dynamics, fine-tuning LLMs with synthetic data can significantly improve their reward modeling capabilities while mitigating catastrophic forgetting. Many of the observations are actually known or acknowledged by many scholars and this paper seems to verify them through some designed experiments
- Another concern is that the conducted experiments may not be necessarily sufficient to back up the observations and conclusions in this paper. I list some of them below
  - the studied topics are limited and some selected topics are too broad to be covered in depth within a 10-page ICLR paper. The considered experimental settings are limited as described in Section 3 (lines 207-215), i.e., the authors only study approaches that can directly interface with the action space supported in the environment and assume that LLMs are used without any gradient update. When dealing with indirect policy learning, the authors only consider reward as code. Yet, there are many other options, e.g., LLMs to execute high-level plans for the agent, etc. Furthermore, the authors only consider a simplified version of reward as code without introducing some extra components or tricks as recent works do [1, 2, 3]. When using LLMs for exploration, the authors do not consider the possibility of letting LLMs write intrinsic reward functions, constructing intrinsic rewards [4, 5], etc.
  - the number of the base LLMs is limited. The authors used the closed-source GPT-4o model for direct policy modeling and the open-source Llama 3 for indirect policy modeling when environment observations consist of text, and PaliGemma when environment observations consist of pixel images. The number of LLMs is quite limited, making it hard to tell the significance and applicability of the observations and conclusions
  - the authors do not compare many baselines in this paper, e.g., the authors only consider the count-based exploration method in Section 4.2 (Figure 5). It would be better to compare against stronger baselines.

[1] Text2reward: Automated dense reward function generation for reinforcement learning

[2] Auto mc-reward: Automated dense reward design with large language models for minecraft

[3]  Eureka: Human-level reward design via coding large language models

[4] Guiding pretraining in reinforcement learning with large language models

[5] World Models with Hints of Large Language Models for Goal Achieving

questions:
- It seems that this work has some preliminary requirements for the LLMs to make it function well, e.g., chain-ot-thought (CoT), in-context learning, and self-refinement, as described in Section 2.1. How important are these components to the sequential decision-making capabilities of the LLMs? How do they affect the direct or indirect policy modeling of LLMs? 
- Under many real-world scenarios, we often expect a timely or frequent decision-making ability of the agent. When using LLMs directly or indirectly during the decision-making process, how can we guarantee that the decision is made timely yet effective?
- How much expert data do you use to fine-tune the PaliGemma model in Section 5? How about a different dataset quality, will there be a significant difference? I would expect a more in-depth discussion on fine-tuning LLMs here
</content>
</thread_object>

</thread>
<thread>
creation date:
1730703612924

thread_type:
official_review

<thread_object>
creator: reviewer_3jq6

<content>
summary:
This paper studied the capabilities of Large Language Models (LLMs) for reinforcement learning (RL) in interactive decision-making tasks, by directly producing policies or indirectly generating rewards for policy optimization.

strengths:
1. The paper is well-written and easy to follow.
2. The experiments are conducted on various environments.

weaknesses:
1. The motivation of this paper is limited. One conclusion of the paper is that leveraging the feedback from the LLM to optimize the policy is better than using the LLM directly as a policy. However, since the latter is fine-tuning free, it is not surprising that the former is better. Even if the AI feedback is noisy, as long as the overall feedback is accurate, fine-tuning the policy against it can obtain some improvements.
2. The policy modeling and reward modeling abilities of LLMs in decision-making tasks have been widely studied in previous works [1,2]. No new methods are proposed in this paper, and the novelty is thus limited compared to these works.
3. The analysis is also not convincing. The authors claim that AI feedback helps better credit assignments. However, given only the training curves, it is not clear when and why credit is better assigned.

[1] Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models.\
[2] Ma et al., Eureka: Human-Level Reward Design via Coding Large Language Models.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730872868860

thread_type:
official_review

<thread_object>
creator: reviewer_57nj

<content>
summary:
In this paper, the authors study how Large Language Models (LLMs) can produce decision-making policies, either by generating actions or by creating reward models for reinforcement learning (RL). The authors use experimental results to reveal that LLMs excel at reward modeling, particularly when using AI feedback, and fine-tuning with synthetic data enhances performance in unfamiliar environments, helping prevent catastrophic forgetting.

strengths:
1. The authors perform extensive experiments to evaluate the capabilities of large language models (LLMs) in sequential decision-making tasks. Specifically, they demonstrate that using LLMs to generate preference data and then train a reward model for RL training has the highest performance gain compared with other methods such as directly modeling policy by LLMs. This conclusion seems interesting and helpful for future works in designing RL agents incorporating LLMs.

2. The experiments in Fig. 3 and Fig. 4 clearly illustrate the effect of the rewards learned through the LLM Feedback.

3. It is interesting that the authors show that prompting engineering can steer LLMs for active exploration on the 
NetHack task, which approximates the count-based exploration bonus.

weaknesses:
1. The related works section is very incomplete. The authors should discuss more recent works that study LLM decision-making problems and self-rewarding of LLMs such as [1]-[7]. The introduction of LLM for better reward design on RL is also studied in [8] and should be discussed carefully.

2. The authors compare direct and indirect policy modeling and use experiments to show that indirect policy modeling attains higher performance in multiple tasks. However, I am a little unconvinced about the experimental configuration, where the authors only query the GPT for the direct policy modeling but train an RL agent with multiple steps for the indirect policy modeling. It seems that the computation cost and the number of interactions with the environment are not controlled. Moreover, it has been shown in ([1]-[3]) that LLM can also serve as a critic model to give a numerical estimated value in the direct policy modeling. Maybe the authors should also add an experiment to analyze what would happen if the estimated reward from the indirect policy modeling were incorporated into the direct policy modeling. In this case, no RL training is involved and computation cost is easier to control.

[1] Zhou, Andy, et al. "Language agent tree search unifies reasoning acting and planning in language models." arXiv preprint arXiv:2310.04406 (2023).

[2] Sun, Haotian, et al. "Adaplanner: Adaptive planning from feedback with language models." Advances in Neural Information Processing Systems 36 (2024).

[3] Liu, Zhihan, et al. "Reason for future, act for now: A principled architecture for autonomous llm agents." Forty-first International Conference on Machine Learning. 2023.

[4] Huang, Jen-tse, et al. "How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments." arXiv preprint arXiv:2403.11807 (2024).

[5] Park, Chanwoo, et al. "Do llm agents have regret? a case study in online learning and games." arXiv preprint arXiv:2403.16843 (2024).

[6] Nottingham, Kolby, et al. "Do embodied agents dream of pixelated sheep: Embodied decision making using language guided world modelling." International Conference on Machine Learning. PMLR, 2023.

[7] Yuan, Weizhe, et al. "Self-rewarding language models." arXiv preprint arXiv:2401.10020 (2024).

[8] Kwon, Minae, et al. "Reward design with language models." arXiv preprint arXiv:2303.00001 (2023).

3. Minor typos: Line 762: "except fro".

questions:
1. When reporting the final results in Fig 1, Fig 2, and Fig 6, do the authors control the number of interactions with the environment? I'm also interested in understanding how performance changes with an increasing number of interactions, as shown in the learning curves typically found in reinforcement learning (RL) papers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732288744549

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to thank the reviewers for their time spent attending our submission. 

We would like to highlight the following two points, which are mentioned in the individual rebuttals.

**LLM Fine-tuning**

One of the goals of the paper is to understand how LLMs can represent the policy, reward or transition function. Our motivations for studying these capabilities without fine-tuning are: (1) training LLMs is computationally expensive and therefore only a handful of labs can afford the process (2) zero-shot or few-shot performance of non-finetuned LLMs on common reasoning benchmarks keeps increasing, meaning that our findings will be increasingly relevant with stronger models. Our results would therefore be of particular interest to a large group of readers. In Section 5, we then relax the assumption that the LLM is used without fine-tuning and present additional insights when the LLM is fine-tuned. 

We include the following results to give a complete picture of the LLM capabilities for decision-making. In the tables below, we directly compare the  performance of direct policy modeling and indirect policy modeling (AI Feedback) on one of the MetaWorld tasks. In these results, we control both for the amount of samples and the amount of FLOPS. We compute the FLOPs using the package [calflops](https://github.com/MrYxJ/calculate-flops.pytorch), whose results we found consistent with Jiang et al (2024). 

Specifically, we fine-tune the direct policy method under three settings: only with online RL, only with SFT, and online RL training on an SFT-pretrained model. We compare these baselines to the indirect policy modeling approach of AI Feedback that uses RL for learning.


Results indicate that indirect policy modeling achieves comparable performance to direct policy modeling trained with SFT at a fraction of the computational cost, i.e. two orders of magnitude lower. Policy modeling achieves a higher success rate with 1M training samples, but is otherwise significantly less efficient than indirect policy modeling. Importantly, direct policy modeling with SFT relies on a large expert dataset to achieve good performance. On the other hand, direct policy modeling trained only with RL shows poor performance with 1M samples and high sensitivity to hyperparameters. This is in line with recent work (under anonymous submission) reporting  low sample efficiency when training LLMs with online RL, *without* SFT, for sequential decision-making.


### Success rate
| Environment Samples | Direct Policy (only RL) | Direct Policy (only SFT) | Direct Policy (RL after SFT pretaining) | Indirect Policy |
| -------- | -------- | -------- | -------- | -------- |
| 0    | 0        | 0        | 0.92     | 0         |
| 128k | 0.01     | 0.10     | 0.80     |  0.22    |
| 256k | 0.05     | 0.13     | 0.63     | 0.48     |
| 1M | 0.09     | 0.87     | 0.65     | 0.72     |

### FLOPS
| Environment Samples | Direct Policy (only RL) | Direct Policy (RL after SFT pretaining) | Indirect Policy |
| -------- | -------- | -------- | -------- |
| 0 |  0    |  3.65x10^17    | 6.9x10^12     |
| 128k |  7.36x10^16    | 4.38x10^17     | 4.09×10^14     |
| 256k | 1.46x10^17     | 5.11x10^17     | 8.109×10^14     |
| 1M | 7.12x10^17     | 1.09x10^18     | 3.145x10^15     |


We will add these results in a revised version of paper and extend the comparison to the rest of the domains given enough time to run the experiments.


**Additional baselines and experiments**

To address the reviewers' points, we have conducted several additional experiments. We concisely present these results and conclusions here, leaving detailed discussions in the indivual rebuttals.

- In this [figure](https://imgur.com/a/822YgtT), we reran the LLM Policy baseline by including the LLM-based reward in-context. This did not significantly improve the performance.
- In this [figure](https://imgur.com/a/PWwUJfz), we improve the Reward as Code baseline through environment and LLM feedback.  This approach worked better on MiniWob-Hard, but not on other domains.
- We report results for [direct policy modeling](https://imgur.com/a/vlCNtMV) and [indirect policy modeling](https://imgur.com/a/DMu5dZ1) using a larger set of LLMs/VLMs, bringing the total to 7 models being investigated. The results are generally equivalent across models, except for AI feedback on MetaWorld.
- In this [figure](https://imgur.com/a/2m6odah) we investigate the importance of chain-of-thought for the AI feedback baseline. Removing CoT did not significantly alter the performance. 


These additional experiments provide a larger scope and expand the evidence supporting the conclusions of our paper, which remain unchanged.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper investigates how large language models can be integrated into reinforcement learning workflows, focusing on two approaches: direct policy generation and indirect reward modeling. Through experiments on diverse environments, the authors demonstrate that LLM-based reward modeling using AI feedback consistently outperforms direct policy modeling in both performance and sample efficiency. The paper further explores fine-tuning methods, revealing that fine-tuning for reward modeling mitigates catastrophic forgetting while improving reward quality.

**Strengths**

- The empirical evaluation is comprehensive (R1, R2, R3, R4).
- The empirical observation is interesting (R1, R3, R4).
- The paper is well-organized and easy to follow (R1, R2, R4).

**Weakness**

- Several reviewers note that the paper does not propose a new method, focusing instead on empirical analysis. Some conclusions, such as the benefits of fine-tuning with synthetic data, are unsurprising (R1, R2, R3).
- The handling of inconsistent or low-confidence preferences from LLMs and their sensitivity to subtle state changes remain underexplored (R4).
- The experiments lack detailed statistical analysis of critical phenomena (R2, R3).

The paper lacks significant methodological novelty, though its extensive empirical investigations and practical insights into leveraging LLMs for RL make it a valuable contribution to the field.