Efficient Reinforcement Learning with Large Language Model Priors

## Abstract

Abstract In sequential decision-making (SDM) tasks, methods like reinforcement learning (RL) and heuristic search have made notable advances in specific cases. However, they often require extensive exploration and face challenges in generalizing across diverse environments due to their limited grasp of the underlying decision dynamics. In contrast, large language models (LLMs) have recently emerged as powerful general-purpose tools, due to their capacity to maintain vast amounts of domain-specific knowledge. To harness this rich prior knowledge for efficiently solving complex SDM tasks, we propose treating LLMs as prior action distributions and integrating them into RL frameworks through Bayesian inference methods, making use of variational inference and direct posterior sampling. The proposed approaches facilitate the seamless incorporation of fixed LLM priors into both policy-based and value-based RL frameworks. Our experiments show that incorporating LLM-based action priors significantly reduces exploration and optimization complexity, substantially improving sample efficiency compared to traditional RL techniques, e.g., using LLM priors decreases the number of required samples by over 90% in offline learning scenarios.

## 1 Introduction

Many real-world tasks, including robotics (Polydoros & Nalpantidis, ; Brunke et al., ; Rana et al., ), autonomous driving (Naranjo et al., ; Song et al., ), human-AI dialogue (McTear, ; Li et al., ), and human-AI gaming (Yan et al., ; Granter et al., ), involve complex sequential decision-making (SDM) challenges. Traditional approaches to SDM, such as optimal control (Garcia et al., ), heuristic search (Świechowski et al., ) and reinforcement learning (RL) (Mnih, ), have seen substantial success. Notably, AlphaGo (Silver et al., ) and AlphaStar (Vinyals et al., ), both based on deep reinforcement learning (DRL), have achieved human-level proficiency in the games of Go and StarCraft II, respectively. However, these methods still suffer from high computational complexity, along with poor generalizability and limited applicability across diverse domains (Dulac-Arnold et al., ; Cobbe et al., ).

Recently, Large Language Models (LLMs) have emerged as effective tools for tackling diverse general-purpose tasks, such as in dialogue systems (Brooks et al., ), decision-making (Zhao et al., ), and mathematical reasoning (Imani et al., ). Their impressive performance across various domains is largely attributed to the vast amounts of human knowledge compressed during the pre-training phase on extensive corpora (Tucker & Tuckute, ). Inspired by how humans make decisions using existing knowledge while learning from new tasks, LLM-based agents are emerging as a promising approach for solving SDM tasks. For instance, researchers leverage human-crafted prompts to guide LLMs in making decisions directly (Zhang et al., ; Wang et al., ; Ma et al., ). While this approach relies heavily on the quality of the prompts and the LLMs’ inherent capabilities, another line of research aims to fine-tune LLMs with RL algorithms (RLFT) (Carta et al., ; Christianos et al., ; ; Zhou et al., ) to enhance their decision-making precision. A more detailed exploration of related work can be found in Section 5. However, these approaches are resource-intensive, as they require meticulously crafted human prompts and an expensive fine-tuning process for LLMs for each specific task.

Inspired by these, this work stands in the position of combining the advantages of leveraging rich domain knowledge with the strengths of reinforcement learning (RL), but in a different perspective to use fixed LLMs to enhance traditional RL.
The key insight is that while it may be challenging for an LLM to generate optimal plans at every state with limited experience, it can still offer suboptimal action proposals or significantly reduce the size of the action space to a manageable scope, thereby alleviating computational burdens. Building on this idea, we treat the LLM as an action prior distribution and incorporate it into Markov Decision Processes (MDPs) solving from a Bayesian inference perspective (Hu et al., ). The primary objective is to approximate the posterior action distribution that aligns with task goals through probabilistic inference approaches such as variational inference or direct posterior sampling. Building on these approaches, we derive the incorporation of LLMs into both policy-based and value-based RL frameworks in a simple yet logically sound manner. We conduct experiments on major benchmarks like ALFWorld and Overcooked, demonstrating that the sample efficiency of traditional RL algorithms, such as value-based DQN and policy-based PPO, can largely benefit from integrating LLM action priors.

In summary, our main contributions are three-fold:

- 1.
We present a unified framework for integrating Large Language Models (LLMs) as probabilistic priors into Markov decision-making frameworks. (Section 2)
- 2.
We practically implement the framework by leveraging LLMs as the refined action sampler for value-based online RL (Section 3.1), offline RL (Section 3.2) or behavior regularizer for policy-based RL (Section 3.3).
- 3.
Extensive experiments on ALFWorld and Overcooked demonstrate that our new framework can significantly boost sample efficiency compared with both pure RL and pure LLM baselines, and also bring in more robust and generalizable value function. (Section 4)

## 2 Formulation

We seek to utilize LLMs to solve complex SDM tasks from a Bayesian inference perspective. Although LLM struggles to directly generate optimal plans in sequential decision-making tasks, it can still provide valuable suggestions for possible sub-optimal actions with rich prior knowledgeHao et al. (); Zhang et al. (). Building on this insight, we treat the LLM as a reliable prior distribution over possible actions and analyze its role in solving MDPs from a Bayesian inference perspective, using traditional probabilistic tools such as variational inference and probabilistic sampling.

##### Markov Decision Process

A Markov Decision Process (MDP) can be described as a tuple $\langle\mathcal{S},\mathcal{A},P,r,\gamma\rangle$. $\mathcal{S}$ is the state space, and $\mathcal{A}$ is the finite action space. We consider the textual action space and state space here. Each state $s$ and each action $a$ is a sentence $s,a\in\mathcal{V}^{\infty}$, where $\mathcal{V}$ is the vocabulary set. $P:\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S})$ is the transition kernel, mapping the current state $s_{t}$ to the next state $s_{t+1}$ following the action $a_{t}$. $r:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is the reward function. $\gamma$ is the discount factor.

##### LLM as an action prior

It is challenging to directly solve textual decision-making tasks due to the lack of task-specific experience, yet powerful LLMs demonstrate the ability to generate reasonable action proposals Yao et al. (); Hao et al. (). To maximize the potential of LLMs in decision-making, we treat the LLM as an action sampler, denoted as $p_{\text{LLM}}(a|s_{t})$. Regarding the implementation details of sampling from LLM priors, we first generate a free-form output from the LLM, for example, a 7B LLM, which is then mapped to an executable action through a simple rule-based projection. This process is formally described as: $p_{\text{LLM}}(a|s_{t})=\sum_{o}p(a|o)\text{LLM}(o|s_{t})$, where $p(a|o)$ is a projection from the LLM output $o$ to action $a$. More detailed explanations of the LLM prior setting can be found in Appendix.

### 2.1 Variational Inference For Markov Decision Process

Similar to the Control as Inference framework proposed by (Levine, ), we introduce a binary optimality variable $\mathcal{O}$ to indicate the quality of a trajectory $\tau=\{s_{0},a_{0},r_{0},s_{1},...,s_{n}\}$, where $\mathcal{O}=1$ represents an optimal/successful trajectory, and $\mathcal{O}=0$ otherwise. Hence, the likelihood, positively related to the reward, can be written as $p(\mathcal{O}=1|\tau)\propto\exp\left(\sum_{t}\gamma^{t}r_{t}/\alpha\right)$, with $\alpha$ as the temperature parameter.
Our goal is to maximize the optimal marginal distribution. To do that, we first apply variational inference:

$$ $p(\mathcal{O}=1)\geq\mathbb{E}_{q(\tau)}\left[\log p(\mathcal{O}=1|\tau)\right ]-\text{KL}\left[q(\tau)\|p(\tau)\right]$ (1) $$

We define the prior $p(\tau)$ as the trajectory distribution, which follows the factorization: $p(\tau)=p(s_{0})\prod_{t}p(a_{t}|s_{t})P(s_{t+1}|s_{t},a_{t})$. Here, we use $p_{\text{LLM}}$ as the action prior $p$. The variational distribution $q(\tau)$ follows a similar factorization: $q(\tau)=p(s_{0})\prod_{t}q(a_{t}|s_{t})P(s_{t+1}|s_{t},a_{t})$. Substituting these factorizations back into Eq. 1, we obtain the step-wise objective as:

$$ $\arg\max_{\theta}\sum_{t}\mathbb{E}_{\pi_{\theta}}\left[\gamma^{t}r_{t}\right] -\alpha{KL}\left[\pi_{\theta}(a_{t}|s_{t})\|p_{LLM}(a_{t}|s_{t})\right],$ (2) $$

and we learn a language model policy $\pi_{\theta}$ as the variational distribution $q(a_{t}|s_{t})$, with parameter $\theta$.
More detailed derivation can be found in Appendix 8.1. Notice that Equation 2 is akin to formulation from (Wen et al., ; Rafailov et al., ; Zhang et al., ), where they fine-tune a language model $\pi_{\theta}$, applying KL regularization terms to preserve the model’s original language capabilities in RLHF settings or as an auxiliary reward for inference. In essence, from a probabilistic inference perspective, we can recover the same objective of these "policy-based" methods, which in turn provides us with practical approaches for implementation (Section 3).

However, fine-tuning language models in policy-based methods can be challenging in practice. To address this, we propose an alternative approach: direct posterior inference, which allows us to sample directly from the optimal posterior distribution $p(\tau|\mathcal{O}=1)$ for sequential decision-making.

### 2.2 Direct Posterior Inference for Markov Decision Process

Figure: Figure 1: An illustration of the process of approximate sampling from the intractable posterior: $q(a|s,\mathcal{O}=1)\propto p(\mathcal{O}=1|a,s_{t})p_{\text{LLM}}(a|s_{t})$, implemented by reweighting the action prior proposals according to $Q$-values, which act as likelihood estimates. In our experiments, the $Q$ function adopts BERT to encode textual state-action pairs and output a scalar value through an adaptor network.
Refer to caption: x1.png

Since the optimal posterior distribution over trajectory $\tau$ can be factorized into step-wise posterior described as: $p(\tau|\mathcal{O}=1)\propto p(s_{0})\prod_{t}p(a_{t}|s_{t},\mathcal{O}=1)P(s_
{t}|s_{t-1},a_{t-1})$, the learning procedure can be divided into two stages, modeling and inference of step-wise posterior $p(a|s_{t},\mathcal{O}=1)$. The modeling stage will point out and decompose the desirable distribution into more solvable probabilistic terms. The inference stage will give the desirable outputs using sampling-based techniques (Korbak et al., ).

##### Posterior Modeling

We first use Bayes’ rule to translate the intricate posterior into accessible terms:

$$ $p(a|s_{t},\mathcal{O}=1)=\frac{p(\mathcal{O}=1|a,s_{t})p(a|s_{t})}{p(\mathcal{ O}=1|s_{t})}\propto p(\mathcal{O}=1|a,s_{t})p(a|s_{t}),$ (3) $$

which is a combination of the likelihood $p(\mathcal{O}=1|a,s_{t})$ and the prior $p(a|s_{t})$. Here, we use the LLM as the prior action distribution $p(a|s_{t})\leftarrow p_{\text{LLM}}(a|s_{t})$. Recall that the optimality likelihood in MDPs is given by: $p(\mathcal{O}=1|a,s_{t})\propto\exp\left(\sum_{i=t}\gamma^{i-t}r_{i}\right)$, which corresponds to whether the goal state is achieved, similar to the concept of the Q function $Q(s_{t},a)$ in RL. In practice, methods such as MCTS or TD-learning can estimate the optimality likelihood $p(\mathcal{O}=1|s_{t},a)$

##### Action Inference

Having described the posterior as a product between LLM prior $p_{LLM}(a|s_{t})$ and estimate $Q$ function $Q^{\theta}(s_{t},a)$, we can implement the inference from the posterior distribution $p(a|s_{t},\mathcal{O}=1)$ by following these steps:

- •
Sample actions based on the prior policy $p_{\text{LLM}}$: Obtain $k$ action candidates for the current state $s_{t}$, denoted as: $\mathcal{C}^{k}(s_{t})=\{a_{1},a_{2},\cdots,a_{k}\}$, where $a_{i}\sim p_{\text{LLM}}(a|s_{t})$.
- •
Choose the next action from the candidates based on the estimated Q-values: Select an action $a^{*}$ according to the softmax probability distribution over the estimated Q-values. Specifically, $a^{*}\sim\operatorname{softmax}(Q^{\theta}(s_{t},a_{1})/\alpha,Q^{\theta}(s_{t
},a_{2})/\alpha,\dots,Q^{\theta}(s_{t},a_{k})/\alpha)$, where $\alpha$ is a hyper-parameter.

This sampling strategy is illustrated in Figure 1, where the action candidates are reweighted based on the Q-values.

###### Proposition 1 .

Denote that the above sampling strategy indeed follows a distribution of $q$. When $k\rightarrow\infty$, we have:

$$ $\displaystyle\lim_{k\rightarrow\infty}q(a|s_{t})$ $\displaystyle=p_{\text{LLM}}(a|s_{t}){\exp(Q^{\theta}(s,a)/\alpha)}/{\mathbb{E }_{a_{j}\sim p_{\text{LLM}}(\cdot|s)}\exp(Q^{\theta}(s,a_{j})/\alpha)}$ (4) $$

The limiting policy corresponds to the policy that optimizes the Q-values with a KL regularizer:

$$ $\displaystyle\lim_{k\rightarrow\infty}q(\cdot|s_{t})$ $\displaystyle=\arg\max_{\pi}\mathbb{E}_{\pi(a|s_{t})}[Q^{\theta}(s_{t},a)]- \alpha\text{KL}\left(\pi(\cdot|s_{t})\|p_{\text{LLM}}(\cdot|s_{t})\right)$ (5) $$

Then, the posterior sampling strategy is highly related to the solution of variational inference as shown in Eq. 2. Proof. Please see the appendix.

###### Proposition 1 .

## 3 Practical Implementation

In this section, we introduce the practical implementation of posterior approximation within robust RL frameworks. We describe a policy-based RL method with an LLM prior to solve the variational inference problem. In the meantime, as discussed earlier, optimality likelihood estimation is critical for direct posterior inference and is positively associated with Q-values in the MDP setting. We consider Q-value estimation with an LLM prior for value-based RL in both online and offline settings. Finally, we propose three simple yet efficient RL algorithms: one policy-based RL algorithm and two Q-learning variants.

We will first introduce the two Q-learning variants, which perform exploration and optimization within a narrowed yet reliable LLM prior action space, eliminating the need for fine-tuning the language model-based policy. Then, we will present a policy-based PPO variant that incorporates a KL loss with respect to the LLM prior. Detailed descriptions of classic Q-learning algorithms, DQN Mnih () and CQL Kumar et al. (), can be found in the Appendix.

### 3.1 Online Value-Based RL with LLM prior

The Q-value estimation is based on DQN (Mnih, ), a widely used value-based RL method, under the online setting. Unlike traditional DQN, which performs exploration and Bellman updates over the full action space $\mathcal{A}$, we propose a variant with an LLM prior, referred to as DQN-Prior. This variant limits these processes to the prior action space $\mathcal{C}^{k}\subseteq\mathcal{A}$, which is sampled from $p_{\text{LLM}}$. This prior action space represents a reduced and more rational subspace where clearly irrelevant actions may be filtered out in advance. Specifically, DQN-Prior alternates between the following steps:

##### Exploration

Roll out new episodes using the posterior inference strategy as described in Sec. 2.2, specifically:

- •
For a state $s_{t}$, sample the action prior space $\mathcal{C}^{k}(s_{t})=\{a_{1},a_{2},\cdots,a_{k}\},a_{i}\sim p_{\text{LLM}}(a
|s_{t})$.
- •
Then sample the action $a_{t}\sim\operatorname{softmax}\left({Q^{\theta}(s_{t},a_{1})/\alpha},\cdots,{
Q^{\theta}(s_{t},a_{k})/\alpha}\right)$.
- •
Apply the action $a_{t}$ to the environment, and obtain the next state $s_{t+1}\sim P(\cdot|s_{t},a_{t})$.
- •
Add the tuple to the reply buffer: $\mathcal{D}=\mathcal{D}\bigcup(s_{t},a_{t},s_{t+1})$.

##### Q-function Update

After expanding the replay buffer, we use TD-learning to update the Q-network. The loss function for the $i$-th iteration is given as:

$$ $\mathcal{L}_{i}(\theta)=\mathbb{E}_{(s,a,s^{\prime})\sim\mathcal{D}}\left[(Q^{ \theta_{i}}(s,a)-y_{i})^{2}\right],$ (6) $$

where $y_{i}=\mathcal{B}^{*}Q^{\theta_{i-1}}(s,a)=r(s,a)+\gamma\max_{{\color[rgb]{
1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}a^{\prime}\in\mathcal{C}^
{k}(s^{\prime})}}Q^{\theta_{i-1}}(s^{\prime},a^{\prime})$.
Compared to traditional DQN, the DQN-Prior performs exploration and applies the Bellman optimal operator within the LLM prior action space, as highlighted in red in Eq. 6.

To represent the Q-network, we introduce an adapter on top of the LLM embeddings and fit the adapter’s weights by minimizing the temporal difference loss. In our work, the LLM (i.e., BERT (Devlin, )) is used to encode state and action pairs and is frozen during optimization, as illustrated in Figure 1. Therefore, we do not propagate any gradients through the LLM to avoid expensive computational overhead.

### 3.2 Offline Value-Based RL with LLM Prior

This study also explores the feasibility of using gradient-free LLM priors for successful SDM based solely on offline datasets.
() have explored the use of offline RL for language generation tasks based on the SFT LLM from offline datasets. This approach assumes the entire token vocabulary as the action space, where each action corresponds to a single token, resulting in a vast action space with tens of thousands of possible actions. In our work, we consider a discrete action space where each action is a short phrase consisting of several tokens. For the SDM tasks we address, the action space is significantly smaller, with fewer than a hundred possible actions. Additionally, by leveraging an action prior, we can first obtain a tidy sub-action space $\mathcal{C}^{k}$, thereby narrowing the optimization space.

We apply this action prior within the CQL framework, a widely used offline RL approach, and refer to this variant as CQL-Prior. The loss function for the proposed CQL-Prior is given as:

$$ $\mathcal{L}_{i}(\theta)=\beta\mathbb{E}_{(s,a)\sim\mathcal{D}}\left[\log\sum_{ {\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}a^{\prime} \in\mathcal{C}^{k}(s)}}\exp(Q^{\theta_{i}}(s,a^{\prime}))-Q^{\theta_{i}}(s,a) \right]+\frac{1}{2}\mathbb{E}_{(s,a,s^{\prime})\sim\mathcal{D}}\left[(Q^{ \theta_{i}}(s,a)-y_{i})^{2}\right],$ (7) $$

where $y_{i}=r(s,a)+\gamma\max_{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor
}{rgb}{1,0,0}a^{\prime}\sim\mathcal{C}^{k}(s^{\prime})}}Q^{\theta_{i-1}}(s^{
\prime},a^{\prime})$ and $\beta$ is a hyper-parameter. The main difference from traditional CQL is that it restricts the overestimation of Q-values to the tidy action prior space and applies the Bellman optimal operator within this space, as highlighted in red.

### 3.3 Policy-based RL with LLM Prior

We implement variational inference for MDPs with LLMs as action priors using a policy-based RL framework. Specifically, we build upon GFlan (Carta et al., ), let the Flan-T5 small model (Rae et al., ) (with less than $1B$ parameters) as the action policy for leveraging pre-stored knowledge to solve complex SDMs and fine-tune it via Proximal Policy Optimization (PPO) (Schulman et al., ). Our variant, called GFlan-Prior, incorporates a KL regularizer into the PPO loss to optimize Eq. 2. During implementation, we sample $k=5$ examples from LLM prior distribution $p_{\text{LLM}}(a|s)$ to compute the approximated KL divergence.
Formally, for the $i$-th iteration, the policy $\pi_{\theta_{i}}$ is learned by optimizing:

$$ $\arg\max_{\theta}\mathbb{E}_{(s_{t},a_{t})\sim\pi_{\theta_{i-1}}}\text{CLIP} \left(\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{i-1}}(a_{t}|s_{t})}\right) \hat{A}_{t}+\gamma\mathcal{H}(\pi_{\theta}(\cdot|s_{t}))-\alpha\text{KL}[\pi_{ \theta}(\cdot|s_{t})\|\hat{p}_{\text{LLM}}(\cdot|s_{t})],$ (8) $$

where $A_{t}$ is the estimated advantage, $\gamma$ is the hyperparameter that encourages exploration, and $\hat{p}_{\text{LLM}}(\cdot|s_{t})$ is the discrete distribution constructed by $k$ action proposals from the LLM. ${\theta_{i-1}}$ represents the parameters of the action policy learned in the last iteration.

Figure: Figure 2: Results of comparison with online baselines. We plot the mean and standard error of the cumulative reward. For inference baselines, the reward is averaged over $20$ episodes. We plot the rewards averaged over the final third of the training processes for trainable baselines across five random seeds.
Refer to caption: x2.png

## 4 Experiments

In this section, we demonstrate the effectiveness of incorporating the LLM prior into the RL framework, including both policy-based and value-based algorithms in online and offline settings. LLM priors are used to reduce the exploration and optimization space or regulate the action policy’s behavior, thereby resulting in a more efficient RL framework. Further details on experimental settings and more ablation study results can be found in the Appendix.

### 4.1 Environments

We consider three environments:
ALFWorld (Shridhar et al., ) is a popular benchmark for examining LLM-based agents’ decision-making ability. This benchmark contains thousands of Textworld (Côté et al., ) games with the embodied engine. We focus solely on the text game part, where the action space consists of high-level plans such as "go to a room". The legal actions are finite but large, with the maximum possible admissible action space reaching up to 50, making it challenging to explore from scratch. It contains thousands of tasks, making testing the generalization performance on unseen tasks convenient. We consider two classes of ALFWorld tasks: ALFWorld(pick) and ALFWorld(examine).
Overcooked We use the partially observed text overcooked game(); the observation describes the position of visible items, and the agent should take a sequence of actions to deliver a dish. We consider two overcooked tasks: Overcooked(Tomato), deliver a dish of chopped tomato; Overcooked(Salad), deliver a salad containing chopped tomato and lettuce. The maximum possible action space is $8$. Besides the reward of $1$ for successfully delivering a dish, the textual Overcooked environment from () also provides dense reward signals.
Frozen Lake is a grid world game; the agent should move to the goal position while avoiding full into holes. There are four admissible actions: up, down, right, and left.

**Table 1: Results of offline algorithms. We consider two tasks, ALFWorld (Pick) and Overcooked (Salad), referred to as Pick and Salad for simplicity. To further compare the sample efficiency of the baselines, we use offline datasets of varying sizes for Overcooked (Salad), denoted as Salad ($N$), each containing approximately $N$ $(s,a,s^{\prime})$ tuples.**
| Baseline | Pick | Salad(1000) | Salad(4000) | Salad(8000) | Salad(12000) | Salad(24000) |
| --- | --- | --- | --- | --- | --- | --- |
| DQN | 0.27 | 0.14 | 0.31 | 0.32 | 0.32 | 0.32 |
| CQL | 0.03 | 0.32 | 0.32 | 0.32 | 0.32 | 1.33 |
| DQN-Prior | 0.49 | 0.98 | 0.81 | - | - | - |
| CQL-Prior | 0.80 | 1.01 | 1.19 | - | - | - |
| BC | 0.31 | 0.57 | 0.91 | - | - | - |

### 4.2 Baselines

We first test the ability of LLM Prior to solve SDMs in a zero-shot manner, without deliberately designed prompts. Next, we will introduce the value-based and policy-based RL algorithms:

Valued-Based RL: For the online setting, we use DQN and our DQN-Prior, which explores and updates the Q-function within the LLM prior space. For the offline setting, we compare the CQL, CQL-prior, and Behavior Clone(BC). Similarly, compared to CQL, CQL-Prior regulates and updates Q-values in the narrowed prior action space. We set $k=5$ for DQN-Prior and CQL-Prior. The BC utilizes the Flan-T5 small (Rae et al., ) as the action policy, and learns the action policy by optimizing: $\arg\min_{\pi}-\mathbb{E}_{(s,a)\sim\mathcal{D}}[\log{\pi(a|s)}].$

Policy-Based RL:
GFlan (Carta et al., ) lets the LLM Flan-T5 small as the action policy and finetunes it via PPO. (Schulman et al., ). GFlan-Prior adds a KL constraint between the optimized action policy and LLM prior action on the basis of GFlan, as shown in Eq. 8.

LLM Setting Due to the Qwen-1.5 series (Bai et al., ) containing a range of LLMs with different scales, we use Qwen-1.5 for our main experiments. Specifically, we use Qwen-1.5 7B (Bai et al., ) as the backbone of the LLM prior for most environments, except for the more complex Overcooked(Salad) task, where Qwen-1.5 14B is used.

### 4.3 Results Analysis

#### 4.3.1 Incorporating LLM Priors into Value-Based RL

##### The effectiveness of LLM Prior on Online Q-Learning

As shown in Figure 2, our DQN-Prior outperforms traditional online RL baselines such as DQN and GFlan by exploring and optimizing within a narrowed, more valuable prior action space. In complex ALFWorld tasks, DQN and GFlan struggle to bootstrap due to the large original action space, which can include up to 50 possible actions and the lack of dense reward incentives. In contrast, our DQN-Prior leverages the LLM’s domain knowledge to reduce the exploration space to just five suboptimal actions, significantly improving sample efficiency. Additionally, in the Overcooked (Salad) and Frozen Lake tasks, the LLM prior cannot directly solve these tasks by sampling one action per state. However, DQN-Prior still achieves success and demonstrates greater sample efficiency than traditional RL methods, despite the small action space (only 4 or 8 actions). This shows that the LLM prior can provide a suboptimal action proposal space, and conducting RL within this space significantly enhances sample efficiency.

**Table 2: Results on the generalization performance of posterior sampling on ALFWorld(Pick). We use Qwen-1.5 7B with $k=5$ to generate LLM prior action proposals for training the Q function via DQN-Prior and CQL-Prior. This LLM prior configuration is highlighted by (*).**
| Online (DQN-Prior) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Seen Tasks | Unseen Tasks |  |  |  |  |  |  |
|  | $k=1$ | $k=5$ | $k=10$ | $k=15$ | $k=1$ | $k=5$ | $k=10$ | $k=15$ |
| Qwen-1.5 4B | 0 0 | $0.35$ | $\bf 0.46$ | $0.35$ | 0 0 | $0.08$ | $0.25$ | $\bf 0.29$ |
| Qwen-1.5 7B | $0.19$ | $0.62^{\bf{*}}$ | $\bf 0.77$ | $0.65$ | $0.04$ | $0.38$ | $\bf 0.42$ | $\bf 0.42$ |
| LLaMa-3 8B | $0.04$ | $0.62$ | $\bf 0.73$ | $0.69$ | $0.13$ | $\bf 0.42$ | $0.38$ | $0.38$ |
| Qwen-1.5 14B | $0.19$ | $0.65$ | $\bf 0.77$ | $0.65$ | $0.16$ | $\bf 0.54$ | $\bf 0.54$ | $0.38$ |
| Qwen-1.5 32B | $0.35$ | $0.77$ | $\bf 0.81$ | $0.73$ | $0.33$ | $\bf 0.5$ | $\bf 0.5$ | $0.46$ |
| Offline (CQL-Prior) |  |  |  |  |  |  |  |  |
|  | Seen Tasks | Unseen Tasks |  |  |  |  |  |  |
|  | $k=1$ | $k=5$ | $k=10$ | $k=15$ | $k=1$ | $k=5$ | $k=10$ | $k=15$ |
| Qwen-1.5 4B | 0 0 | $0.27$ | $0.38$ | $\bf 0.62$ | $0.0$ | $0.17$ | $0.17$ | $\bf 0.38$ |
| Qwen-1.5 7B | $0.19$ | $0.80^{\bf{*}}$ | $\bf 0.81$ | $0.77$ | $0.04$ | $0.46$ | $\bf 0.50$ | $\bf 0.50$ |
| LLaMa-3 8B | $0.04$ | $0.85$ | $\bf 0.88$ | $0.81$ | $0.13$ | $\bf 0.54$ | $0.50$ | $\bf 0.54$ |
| Qwen-1.5 14B | $0.19$ | $0.73$ | $\bf 0.92$ | $0.73$ | $0.16$ | $0.54$ | $\bf 0.58$ | $0.54$ |
| Qwen-1.5 32B | $0.35$ | $0.81$ | $\bf 0.85$ | $0.69$ | $0.33$ | $\bf 0.63$ | $0.42$ | $0.50$ |

##### The effectiveness of LLM Prior on Offline Q-Learning

Table 1 illustrates the performance of baselines trained on offline datasets. A more detailed description of the offline datasets can be found in the Appendix. Table 1 shows that CQL-Prior outperforms all other baselines on the ALFWorld (Pick) and Overcooked (Salad) datasets. Although DQN-Prior lacks constraints on Q-values, it still demonstrates superior performance compared to DQN and CQL, which operate on the full action space. These results indicate that in offline RL, avoiding overestimation of Q-values and performing Bellman updates only within the suboptimal prior action space reduces optimization complexity, thereby improving sample efficiency.

Additionally, we provide further evidence to support this conclusion by testing the sample requirements for CQL and CQL-Prior. We find that traditional CQL, with fewer than $12000$ examples, fails to learn how to achieve the final goal, only managing to reach the subgoal of chopping the tomato and lettuce (with a reward of $0.4$). In contrast, CQL-Prior, with just around $1000$ examples, successfully learns how to achieve the final goal (with a reward of $1.0$) with high probability. Therefore, our CQL-Prior reduces the number of required samples by at least 90% compared to CQL on this task.

##### The generalization ability across unseen tasks and LLMs

Table 2 demonstrates the generalization ability of our posterior sampling framework by combining the Q-function, trained with Qwen-1.5 7B and $k=5$, with other LLMs and varying action proposal numbers ($k$) during inference. Even though the Q-network is trained with Qwen-1.5 7B and $k=5$, it can still effectively guide the decision-making process, i.e., action selection, for unseen LLMs of different scales and architectures. It is worth noting that, although Qwen-1.5 4B was originally unable to solve ALFWorld (Pick), it becomes capable of tackling such complex tasks by leveraging the Q-function trained with more powerful LLMs. This value-based RL approach, which utilizes an LLM prior, presents a promising opportunity to train an information-rich Q-network using larger LLMs while deploying only the Q-network and smaller LLMs on the client side. This method enhances reasoning speed and reduces deployment complexity. Furthermore, larger LLMs generally perform better during inference, as their action proposals are of higher quality. $k=1$ indicates the performance of the LLM on its own, without involving the Q-function. The setting of $k=5$ was chosen during training for querying efficiency, but it is not necessarily the optimal choice. We observed better test performance with $k=10$ on seen tasks, indicating that our method can generalize to a larger action space to some extent. In most cases, inference performance increases and then decreases as $k$ increases. This may occur because a larger action space is more likely to include the optimal action, but an excessively large action space can introduce unseen $(s,a)$ pairs during training, leading to out-of-distribution (OOD) problems in both online and offline settings.

Regarding generalization to unseen tasks, multiple LLM action proposals with $k>1$, guided by the Q-function, generally outperform the LLM’s zero-shot performance with $k=1$. This demonstrates the generalization capability of our posterior sampling framework, enabled by the powerful LLM as an action prior and the Q-function’s ability to compress interactive experiences.

Figure: (a) Results of policy-based RL with LLM prior
Refer to caption: x8.png

#### 4.3.2 Incorporating LLM Priors into Policy-Based RL

The Figure 3(a), 3(b), 3(c) show the results of policy-based RL algorithms. As shown in Figure 3(a), GFlan-Prior significantly outperforms GFlan and achieves performance comparable to DQN-Prior. Unlike GFlan, GFlan-Prior leverages the prior knowledge embedded in LLMs by aligning with the LLM prior distribution through the incorporation of a KL constraint, as shown in Equation 8. In our main results, we sample $k=5$ action proposals from the LLM prior to approximate the KL divergence. An ablation study on $k$ can be found in Figure 3(b) and Figure 4 in the Appendix. Similar to the results in Table 2, with $k=5$, we are more likely to sample the optimal action than with $k=1$, thereby better supervising the learning of the action policy.

The ablation study on the coefficient of the KL divergence is shown in Figure 3(c). A large $\alpha$ will cause the learned policy $\pi$ to closely follow the LLM prior proposals, while a small $\alpha$ encourages exploration. In DQN-Prior, the hyperparameter $\alpha$ regulates exploration by acting as the temperature in the softmax over Q, generating a Boltzmann distribution based on Q, as investigated in Figure 3(d). GFlan-Prior appears to be more sensitive to the hyperparameter on controlling exploration than DQN-Prior.

## 5 Related Work

This study explores the fusion of Bayesian inference and LLMs for decision-making tasks. Herein, we present related works on LLM-Agent for decision-making tasks and how previous research pursues this combination.

##### LLM-driven Agent

The idea of LLM-as-Agent has gone viral since the release of powerful large language models. Among all the works, some apply direct prompt engineering techniques using human-designed prompt and retrieval to harness LLMs for complex interactive tasks, such as ProAgent by (Zhang et al., ), Voyager by (Wang et al., ) and generative agent by (Park et al., ). Many researchers have also tried searching-based methods. (Yao et al., ) propose Tree-of-Thought (TOT) using BFS/DFS algorithm to search accurate decision sequences. (Hao et al., ) suggests using Monte-Carlo Tree Search to perform a more comprehensive search. These methods often fall into the framework of Proposer-Verifier (Snell et al., ) where the language proposes a number of possible sequences and the verifier picks desirable candidates. There are also a few works that relate LLM to conventional reinforcement learning settings and investigate how the superiority in language space helps RL agents’ capability in decision-making. For instance, (Brooks et al., ) uses LLM as a world model and performs policy iteration by in-context learning. GFlan proposed by (Carta et al., ) uses a learnable LLM as a probability likelihood estimator for all possible actions and incorporates it into actor-critic learning frameworks where a value function generates assessment and fine-tuning the LLM correspondingly. (Yan et al., ) also resort to an actor-critic learning paradigm but instead of fine-tuning the language model, they train a simple classifier that selects valuable outputs from a fixed language model (e.g. GPT3.5). (Zhang et al., ) use LLM as a behavior regularize and add it to the value estimate. (Wen et al., ) leverages entropy-regularized reinforcement learning to train LLM agents for verbal sequential decision-making tasks. Our work also falls into the category of LLM-driven RL agent. However, different from others, we see a large language model as an excellent action proposer and perform Q-learning directly on the reformed action sequence space, thereby achieving better learning efficiency with minimum modification.

##### LLM in Bayesian Sense

The probabilistic nature of transformer-based language models makes them well-suited for a Bayesian inference framework. As highlighted by (Korbak et al., ), the Bayesian approach offers a unified perspective on both modeling and inference. In in-context learning (ICL), some researchers argue for the Bayesian properties of ICL ((Jiang, ; Wang et al., ; Ye et al., )). Extending beyond ICL, (Yang & Klein, ) trains a classifier to approximate the likelihood function, framing conditional text generation as a Bayesian inference problem, with the LLM serving as the prior. (Hu et al., ) interprets Chain-of-Thought (CoT) reasoning in natural language processing (NLP) tasks, such as text infilling and sequence continuation, as probabilistic inference problems, using GFlowNet to fine-tune the model. In this context, the language model functions as a unified probabilistic generative process, capable of representing both prior and likelihood distributions. Similarly, (Zhao et al., ) adopts a Bayesian perspective, utilizing Sequential Monte Carlo (SMC) methods to generate undesired outcomes. (Gallego, ) frames LLM inference as sampling from a posterior distribution focused on high-reward regions, applying this approach to self-improvement tasks. Inspired by these works, we also formulate our agent framework within a Bayesian setting, exploring the use of LLMs as priors.

## 6 Conclusion

In this work, we aim to leverage the rich domain knowledge and strong reasoning abilities of LLMs to solve complex decision-making tasks while avoiding the costly prompt design and fine-tuning of large language models. Given their rich prior knowledge but lack of task-specific experience, we do not rely on LLMs to directly output optimal plans. Instead, we focus on simplifying their role to generate reliable action proposals. Thus, we treat the powerful LLM as an action prior distribution and, from a Bayesian inference perspective, analyze how to incorporate LLM priors into solving MDPs. We utilize variational inference and posterior sampling to achieve this, ultimately proposing both policy-based and value-based frameworks with LLM priors. LLM priors improve traditional RL frameworks by reducing the exploration space or regulating the behavior of the action policy according to suboptimal LLM action proposals. Experimental results demonstrate a significant improvement in sample efficiency by incorporating fixed LLM priors into RL frameworks in both online and offline settings. This work focuses solely on text-based games with finite action spaces. In future work, we will explore scenarios with free-form and infinite action spaces, and try to incorporate prompt-based approaches to further enhance the quality of LLM prior action proposals.

## 7 Description of Classic Q-learning Algorithm

We introduce the classic Q-learning algorithm in both offline and online settings.

##### Q-learning

Q-learning is a classic value-based RL algorithm, which iteratively applies the Bellman optimal operator to train the Q-function, formally written as $\mathcal{B}^{*}Q(s,a)=r(s,a)+\gamma\mathbb{E}_{s^{\prime}\sim P(\cdot|s,a)}
\left[\max_{a}^{\prime}Q(s^{\prime},a^{\prime})\right]$. After the optimal Q-function $Q^{*}$ learned, then the optimal action policy respect to the $Q^{*}$ is given as: $\pi^{*}(\cdot|s)=\arg\max_{a}Q^{*}(s,a)$. DQN (Mnih, ) is a well-known Q-learning algorithm, which can process various classes of the state information such as image and language, which learns a deep neural network $Q^{\theta}(s,a)\approx Q^{*}(s,a)$ to approximate $Q^{*}(s,a)$ and uses TD-learning to iteratively update the Q network, the loss function for the $i$-the iteration is given as:

$$ $\mathcal{L}_{i}(\theta)=\mathbb{E}_{(s,a,s^{\prime})\sim\mathcal{D}}\left[(Q^{ \theta_{i}}(s,a)-y_{i})^{2}\right],$ (9) $$

where $\mathcal{D}$ is the reply buffer, $y_{i}=\mathcal{B}^{*}Q^{\theta_{i-1}}(s,a)=r(s,a)+\gamma\max_{a^{\prime}\in
\mathcal{A}}Q^{\theta_{i-1}}(s^{\prime},a^{\prime})$. $\theta_{i-1}$ is the learned parameters of the last iteration and is frozen during the gradient descent for optimizing the loss function $\mathcal{L}_{i}$.

##### Conservative Q-learning

CQL (Kumar et al., ) framework is designed for offline reinforcement learning, where only offline data is available, but no further online exploration is accessible. Different from traditional Q-learning, CQL introduces an additional regularization term on top of the standard DQN framework or actor-critic framework(such as SAC (Haarnoja et al., )), ensuring that the learned Q-function lower-bounds the actual value. The lower-bounded Q-values help mitigate the overestimation problem commonly encountered with out-of-distribution (OOD) state-action pairs. The CQL loss on the top of value-based DQN is given as:

$$ $\mathcal{L}_{i}(\theta)=\beta\mathbb{E}_{(s,a)\sim\mathcal{D}}\left[\log\sum_{ a^{\prime}\in\mathcal{A}}\exp(Q^{\theta_{i}}(s,a^{\prime}))-Q^{\theta_{i}}(s,a )\right]+\frac{1}{2}\mathbb{E}_{(s,a,s^{\prime})\sim\mathcal{D}}\left[(Q^{ \theta_{i}}(s,a)-y_{i})^{2}\right],$ (10) $$

where $y_{i}=r(s,a)+\gamma\max_{a^{\prime}}Q^{\theta_{i-1}}(s^{\prime},a^{\prime})$ and $\beta$ is a hyper-parameter.

## 8 Proofs

### 8.1 Probabilistic Inference Derivation

In this section, we derive the detailed formulation for probabilistic inference for using LLM as a prior. This includes the policy-based method where we learn a parametrized policy function and the value-based method where we directly perform posterior inference.

##### Variational Inference

Assume that we have the optimality variable $\mathcal{O}$ indicating the quality of a trajectory $\tau=\{s_{0},a_{0},r_{0},s_{1},...,s_{n}\},p(\tau)=p(s_{0})\prod_{t}p(a_{t}|s_
{t})p(s_{t+1}|s_{t},a_{t})$. The likelihood function with temperature parameter $\alpha$ can be written as:

$$ $p(\mathcal{O}=1|\tau)=\exp\left(\sum_{t}\gamma^{t}r_{t}/\alpha\right)$ (11) $$

Our goal is to maximize the optimal marginal distribution formulated as follows:

$$ $\displaystyle p(\mathcal{O}=1)$ $\displaystyle=\int p(\mathcal{O}=1,\tau)d\tau$ (12) $\displaystyle=\int q(\tau)\frac{p(\mathcal{O}=1|\tau)p(\tau)}{q(\tau)}d\tau$ $$

where $q(\tau)$ is a parametrized variational distribution. Then, our variational inference objective can be written as:

$$ $\displaystyle\log p(\mathcal{O}=1)$ $\displaystyle\geq\int q(\tau)\log\left[p(\mathcal{O}=1|\tau)\frac{p(\tau)}{q( \tau)}\right]d\tau$ (13) $\displaystyle=\mathbb{E}_{q(\tau)}\left[\log p(\mathcal{O}=1|\tau)\right]- \textbf{KL}\left[q(\tau)|p(\tau)\right]$ $\displaystyle=\textbf{ELBO}$ $$

In the context of language models, we define prior $p(\tau)$ as trajectory distribution generated through a fixed LLM denoted as:

$$ $p_{LLM}(\tau)=p(s_{0})\prod_{t}p_{LLM}(a_{t}|s_{t})p(s_{t+1}|s_{t},a_{t})$ (14) $$

where $p_{LLM}(a_{t}|s_{t})$ is the probability likelihood of the generated output $a_{t}$ given input text $s_{t}$. The variational distribution $q(\tau)$ can also be decomposed in the same way as:

$$ $q(\tau)=p(s_{0})\prod_{t}\pi_{\theta}(a_{t}|s_{t})p(s_{t+1}|s_{t},a_{t})$ (15) $$

where the parametrized policy function $\pi_{\theta}$ can be a learnable language model with parameter $\theta$. Substituting these factorizations back to ELBO we have the step-wise objective function written as:

$$ $-\mathcal{L}=\sum_{t}\mathbb{E}_{\pi_{\theta}}\left[\gamma^{t}r_{t}\right]- \alpha\textbf{KL}\left[\pi_{\theta}(a_{t}|s_{t})\|p_{LLM}(a_{t}|s_{t})\right]$ (16) $$

$$ $\pi^{*}=\operatorname*{arg\,min}_{\theta}\mathcal{L}$ (17) $$

##### Direct Posterior Inference

Instead of learning a parametrized policy which might involve fine-tuning a language model, we can also try directly sampling from the posterior distribution formulated as:

$$ $\displaystyle p(\tau|\mathcal{O}=1)$ $\displaystyle=\frac{p(\mathcal{O}=1|\tau)p(\tau)}{p(\mathcal{O}=1)}$ (18) $\displaystyle\propto p(\mathcal{O}=1|\tau)p(\tau)$ $\displaystyle=\prod_{t}p(\mathcal{O}_{t}=1|s_{t},a_{t})p_{LLM}(a_{t}|s_{t})p(s _{t+1}|s_{t},a_{t})$ $$

in which we define $p(\tau)$ as in Equation 14. According to Control-as-Inference framework proposed by Levine (), we can formulate posterior inference as Soft Q-Learning where Q-values are updated using a modified soft Bellman equation:

$$ $\displaystyle Q^{\pi}(s_{t},a_{t})=r(s_{t},a_{t})+\gamma\mathbb{E}_{s_{t+1} \sim p(\cdot|s_{t},a_{t})}\left[V(s_{t+1})\right]$ (19) $\displaystyle V(s_{t})=\log\int\exp(Q(s_{t},a_{t})/\alpha)da_{t}$ $$

and the policy can be defined as a softmax function over the Q-values: $\pi(a|s)=\exp(Q^{\pi}(s,a)/\alpha)/\int_{a^{{}^{\prime}}}\exp(Q^{\pi}(s,a^{{}^
{\prime}})/\alpha)da^{{}^{\prime}}$, ensuring that actions with higher Q-values are more likely to be chosen. In our paper, however, due to the enormously large space of (textual) state-action pair, we also apply sampling approximation methods. Specifically, we leverage similar ideas from Fourati et al. () which sample a random subset of actions from the complete action space and seek the optimal within this subset. In our case, we rely on a fixed LLM to provide such action subspace and can further narrow down the space through repeated sampling (Brown et al. ()):

$$ $\displaystyle\pi(a|s)$ $\displaystyle=\frac{\mathbf{1}_{a\in p_{LLM}(\cdot|s)}\cdot\exp\left(Q^{\pi}(s ,a)/\alpha\right)}{\int_{a^{{}^{\prime}}}\mathbf{1}_{a\in p_{LLM}(\cdot|s)} \cdot\exp\left(Q^{\pi}(s,a^{{}^{\prime}})/\alpha\right)da^{{}^{\prime}}}$ (20) $\displaystyle\approx\mathbf{1}_{a\in\{a^{i}\}}\cdot\frac{1}{N}\frac{\exp(Q^{ \pi}(s,a^{i})/\alpha)}{\sum_{i}\exp(Q^{\pi}(s,a^{i})/\alpha)},a^{i}\sim p_{LLM }(a|s)$ $$

This means we restrict the Bellman backup to a specific action subspace that can potentially provide a refined set of actions based on context, largely reducing computational complexity and focusing on more relevant actions.

### 8.2 Proof of Proposition 1

See 1

###### Proof.

The proof mainly follows the (Li et al., ). Denote the above sampling strategy indeed follows a distribution $q$, and for each action $a$, the probability of $a$ is sampled from the distribution $q$ is given by:

$$ $\displaystyle q(a|s_{t})$ $\displaystyle=\mathbb{E}_{\{a_{1},\cdots,a_{k}\}\sim p_{\text{LLM}}(\cdot|s)} \left[\sum_{i=1}^{k}\mathbb{I}(a_{i}=a)\frac{\exp(Q^{\theta}(s_{t},a_{i})/ \alpha)}{\sum_{j=1}^{k}\exp(Q^{\theta}(s_{t},a_{j})/\alpha)}\right]$ (21) $\displaystyle=\mathbb{E}_{\{a_{1},\cdots,a_{k}\}\sim p_{\text{LLM}}(\cdot|s)} \left[\frac{\sum_{i=1}^{k}\mathbb{I}(a_{i}=a)}{\sum_{j=1}^{k}\exp(Q^{\theta}(s _{t},a_{j})/\alpha)}\right]\exp(Q^{\theta}(s_{t},a)/\alpha)$ $\displaystyle=\mathbb{E}_{\{a_{1},\cdots,a_{k}\}\sim p_{\text{LLM}}(\cdot|s_{t })}\left[\frac{\frac{1}{k}\sum_{i=1}^{k}\mathbb{I}(a_{i}=a)}{\frac{1}{k}\sum_{ j=1}^{k}\exp(Q^{\theta}(s_{t},a_{j})/\alpha)}\right]\exp(Q^{\theta}(s_{t},a)/\alpha)$ $$

According to the Law of Large Numbers, when $k\rightarrow\infty$, we have:

$$ $\displaystyle\lim_{k\rightarrow\infty}q(a|s_{t})$ $\displaystyle=\lim_{k\rightarrow\infty}\mathbb{E}_{\{a_{1},\cdots,a_{k}\}\sim p _{\text{LLM}}(\cdot|s_{t})}\left[\frac{\frac{1}{k}\sum_{i=1}^{k}\mathbb{I}(a_{ i}=a)}{\frac{1}{k}\sum_{j=1}^{k}\exp(Q^{\theta}(s_{t},a_{j})/\alpha)}\right] \exp(Q^{\theta}(s_{t},a)/\alpha)$ (22) $\displaystyle=\lim_{k\rightarrow\infty}\mathbb{E}_{\{a_{1},\cdots,a_{k}\}\sim p _{\text{LLM}}(\cdot|s_{t})}\left[\frac{p_{\text{LLM}}(a|s_{t})}{\mathbb{E}_{a_ {j}\sim p_{\text{LLM}}(\cdot|s_{t})}\exp(Q^{\theta}(s_{t},a_{j})/\alpha)} \right]\exp(Q^{\theta}(s_{t},a)/\alpha)$ $\displaystyle=p_{\text{LLM}}(a|s_{t})\frac{\exp(Q^{\theta}(s,a)/\alpha)}{ \mathbb{E}_{a_{j}\sim p_{\text{LLM}}(\cdot|s)}\exp(Q^{\theta}(s,a_{j})/\alpha)}$ $$

Following the proof process from the Appendix A.1. in Rafailov et al. (), we have:

$$ $\arg\max_{\pi}\mathbb{E}_{\pi(a|s_{t})}[Q^{\theta}(s_{t},a)]-\alpha\text{KL} \left(\pi(\cdot|s_{t})\|p_{\text{LLM}}(\cdot|s_{t})\right)=p_{\text{LLM}}(a|s_ {t})\frac{\exp(Q^{\theta}(s,a)/\alpha)}{\mathbb{E}_{a_{j}\sim p_{\text{LLM}}( \cdot|s)}\exp(Q^{\theta}(s,a_{j})/\alpha)}$ (23) $$

∎

Additionally, as shown in Sec. 2, the variational inference approach learns the optimal policy $\pi$ by maximising:

$$ $\arg\max_{\pi}\sum_{t}\mathbb{E}_{\pi}\left[\gamma^{t}r_{t}\right]-\alpha\, \text{KL}(\pi(a|s_{t})\|p_{\text{LLM}}(a|s_{t})).$ (24) $$

Introducing the occupancy measure $\rho$, $\rho(s)=\frac{1}{1-\gamma}\sum_{t=0}^{\infty}[\gamma^{t}\mathbb{P}(s_{t}=s|a
\sim\pi)]$, we have the following form objective respected to the Q-function

$$ $\arg\max_{\pi}\mathbb{E}_{\rho(s)}[\mathbb{E}_{a\sim\pi}[Q^{\pi}(s,a)]-\alpha \text{KL}(\pi(a|s_{t})\|p_{\text{LLM}}(a|s_{t}))]$ (25) $$

Combining Equations 23 and 25, we find that the variational inference approach and the direct posterior sampling yield similar solutions.

###### Proof.

## 9 Additional Experiment Details

### 9.1 Environments

We consider three environments:
ALFWorld We consider two classes of ALFWorld tasks: ALFWorld(pick) and ALFWorld(examine). For ALFWorld (Pick), we evaluate the online training baselines on 28 tasks, such as "put the cellphone on the armchair," and test the generalization ability on 26 unseen tasks. For ALFWorld (Examine), we use 11 tasks, such as "examine the laptop with the desk lamp." There are no auxiliary rewards, except for a reward of $1.0$ for reaching the final goal.
Overcooked We use the partially observed text overcooked game; the observation describes the position of visible items, and the agent should take a sequence of actions to deliver a dish. We consider two overcooked tasks: Overcooked(Tomato), deliver a dish of chopped tomato; Overcooked(Salad), deliver a salad containing chopped tomato and lettuce. The maximum possible action space is $8$. Besides the reward of $1$ for successfully delivering a dish, the textual Overcooked environment from also provides dense reward signals. The reward shaping is as follows: 0.2 for correctly chopping an ingredient, 1 terminal reward for successfully delivering the correct dish, $-0.1$ for delivering any incorrect item, and $-0.001$ for each time step.
Frozen Lake is a grid world game; the agent should move to the goal position while avoiding full into holes. There are four admissible actions: up, down, right, and left. The agent will receive a reward of $1$ for reaching the final goal and a penalty of $-1$ for falling into holes. The maximum steps for these three tasks are shown in Table 3.

**Table 3: Maximize horizon of environments.**
|  | ALFWorld(Pick) | ALFWolrd(Examine) | Cook(Tomato) | Cook(Salad) | Frozen Lake |
| --- | --- | --- | --- | --- | --- |
| Horizon | 60 | 60 | 30 | 50 | 20 |

### 9.2 LLM prior implementation

There are two ways to sample an action $a$ from the LLM prior $p_{\text{LLM}}(\cdot|s_{t})$. First, since the state and action can be described as text, and assuming the action $a$ consists of $k$ tokens, the probability of the LLM generating action $a$ is given by $p(a|s_{t})=\prod_{i=1}^{k}p_{\text{LLM}}(a_{i}|s_{t},a_{<i})$. Based on this probability, the first type of LLM prior computes a distribution over actions, denoted as $p_{\text{LLM}}^{\text{dist}}(a|s_{t})=\frac{\exp p(a|s_{t})}{\sum_{a^{\prime}}
\exp p(a^{\prime}|s_{t})}$. In contrast, the second approach involves sampling a free-form output from the LLM, which is then mapped to an executable action via a simple rule-based projection $\mathcal{P}$. We denote this LLM prior, which relies on mapping the LLM’s output, as $p_{\text{LLM}}^{\text{map}}$, where $a\sim p_{\text{LLM}}^{\text{map}}(\cdot|s_{t})\leftrightarrow o\sim\text{LLM}(
\cdot|s_{t}),a=\mathcal{P}(o)$.

We observed a performance gap between these two types of LLM action priors. For instance, in the ALFWorld pick & place task, $p_{\text{LLM}}^{\text{map}}$ achieved a winning rate of $19\%$, while $p_{\text{LLM}}^{\text{dist}}$ failed, with a winning rate of $0\%$, both using the LLM Qwen1.5-7B. Therefore, in this work, we implement $p_{\text{LLM}}^{\text{map}}$ as the LLM prior for thoughtfully exploiting the power of the LLM in SDM tasks, which we denote as $p_{\text{LLM}}$ for simplicity.
Examples of LLM’s output is given as:

### 9.3 Hyper-parameters

Our algorithms are trained on one machine with $2$ 40G A100. based on Pytorch-GPU 2.1 and cuda12.4. Table 6,7,8,9,10 reports the main hyper-parameters of our algorithms. For all CQL-based algorithms, we set the hyperparameter $\beta$ for regulating the Q-values, as shown in Eq. 7, as $5.0$.

##### Offline Datasets

The composition of offline datasets is shown in Table 11, which reports the number of $(s,a,s^{\prime})$ examples.

**Table 4: The hyperparameters on ALFWorld(Pick)**
| Baselines | Learning Rate | Epochs | Batch Size | Update Frequency | LLM | $\alpha$ |
| --- | --- | --- | --- | --- | --- | --- |
| DQN-Prior | 5e-4 | 4 | 128 | 5 | / | 0.01 |
| CQL-Prior | 5e-4 | 4 | 128 | 5 | / | 0.01 |
| GFlan-Prior | 1e-4 | 16 | 64 | 16 | Flan-T5 Small | 0.01 |

**Table 5: The hyperparameters on ALFWorld(Examine)**
| Baselines | Learning Rate | Epochs | Batch Size | Update Frequency | LLM | $\alpha$ |
| --- | --- | --- | --- | --- | --- | --- |
| DQN-Prior | 5e-4 | 4 | 128 | 10 | / | 0.01 |
| CQL-Prior | 5e-4 | 4 | 128 | 10 | / | 0.01 |
| GFlan-Prior | 1e-4 | 16 | 64 | 16 | Flan-T5 Small | 0.01 |

**Table 6: The hyperparameters on Overcooked(Tomato )**
| Baselines | Learning Rate | Batch Size | Update Frequency | LLM | $\alpha$ |
| --- | --- | --- | --- | --- | --- |
| DQN-Prior | 5e-4 | 128 | 5 | / | 0.1 |
| CQL-Prior | 5e-4 | 128 | 5 | / | 0.1 |
| GFlan-Prior | 1e-4 | 64 | 16 | Flan-T5 Small | 0.1 |

**Table 7: The hyperparameters on Overcooked(Salad)**
| Baselines | Learning Rate | Batch Size | Update Frequency | LLM | $\alpha$ |
| --- | --- | --- | --- | --- | --- |
| DQN-Prior | 5e-4 | 128 | 5 | / | 0.1 |
| CQL-Prior | 5e-4 | 128 | 5 | / | 0.1 |
| GFlan-Prior | 1e-4 | 64 | 16 | Flan-T5 Small | 0.1 |

**Table 8: The hyperparameters on Frozen Lake**
| Baselines | Learning Rate | Batch Size | Update Frequency | LLM | $\alpha$ |
| --- | --- | --- | --- | --- | --- |
| DQN-Prior | 5e-4 | 128 | 5 | / | 0.1 |

**Table 9: The hyperparameters on ALFWorld(Pick)**
| Baselines | Learning Rate | Batch Size | Update Frequency | LLM | $\alpha$ |
| --- | --- | --- | --- | --- | --- |
| DQN-Prior | 5e-4 | 128 | 10 | / | 0.01 |
| CQL-Prior | 5e-4 | 128 | 10 | / | 0.01 |
| GFlan-Prior | 1e-4 | 64 | 16 | Flan-T5 Small | 0.01 |

**Table 10: The hyperparameters on ALFWorld(Examine)**
| Baselines | Learning Rate | Batch Size | Update Frequency | LLM | $\alpha$ |
| --- | --- | --- | --- | --- | --- |
| DQN-Prior | 5e-4 | 128 | 10 | / | 0.01 |
| CQL-Prior | 5e-4 | 128 | 10 | / | 0.01 |
| GFlan-Prior | 1e-4 | 64 | 16 | Flan-T5 Small | 0.01 |

**Table 11: The composition of the offline dataset**
| Dataset | Total Examples | Good Examples | Bad Examples |
| --- | --- | --- | --- |
| ALFWorld(Pick) | 6572 | 6572 | 0 |
| Salad(1000) | 1186 | 574 | 612 |
| Salad(4000) | 3892 | 2209 | 1683 |
| Salad(8000) | 7571 | 4224 | 3347 |
| Salad(12000) | 11939 | 4004 | 7935 |
| Salad(24000) | 23869 | 7191 | 16678 |

### 9.4 Additional Results

Figure: Figure 4: The ablation of the number of action proposals $k$ used for approximating KL divergence between the required action policy and the LLM prior action distribution.
Refer to caption: x14.png

##### Additional ablation on k 𝑘 k italic_k for GFlan-Prior

As shown in Figure 4, when we sample $k=5$ action proposals from the LLM to approximate the KL divergence, it performs better than when using $k=1$. This is because, with more proposals, the optimal action is more likely to be included in the subset, thus better guiding the action policy.

## 10 Additional Related Works

##### Value-based Inference of LLM

Not limited to the pre-training phase, it has been recently investigated how scaling law applies at inference time for LLM. This is often referred to as searching over an enormously large output space with guidance from value estimation. For example, the naive case is beam search (citation) which selects the top K possible sequence based on cumulative probabilities as opposed to greedy search. The value estimate here is the likelihood of the sentence as predicted by the language model. (Wang et al., ) propose CoT-SC, which searches over reasoning paths and selects the most frequent answer. Tree-of-Thought (ToT, (Yao et al., )) adopts depth/breadth-first search. (Hao et al., ) introduce Reasoning-via-Planning (RAP) which uses Monte-Carlo Tree Search and value estimate from prompting LLM. TS-LLM designed by (Feng et al., ) guides MCTS with a learned value function conditioned on state and a learned Outcome-supervised Reward Model (ORM). (Li et al., ) proposes Q-probing to adapt a pre-trained language model to also maximize a tasks-specific reward function in code generation tasks. Furthermore, recent works investigate Process-supervised Reward Model (PRM, (Lightman et al., )) and apply it in LLM inference, (McAleese et al., ) searches over a step-wise critic model in code reviewing tasks. (Wang et al., ) learn a PRM in math reasoning and use it for LLM fine-tuning. (Snell et al., ) provides a detailed benchmark of different inference methods. (Zhang et al., ) represents the score as the probability of a single text token (e.g. ’Yes’ or ’No’) under the context and the prompt. Even though our proposed method can be seen as an example of value-based inference, we focus on a different perspective where we investigate how LLM helps conventional value-based reinforcement learning algorithms in complex environments. More discussion can be found in Section 2.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The experiments are only conducted on simple environments with discrete action spaces. What if continuous action space (e.g., offline RL on the D4RL benchmark)?
soundness: 2
strengths: 1. This paper is clearly written and easy to follow.
2. The considered "incorporating the background knowledge of LLM into RL" is an interesting topic.
summary: This paper considers treating LLMs as prior action distributions and integrating them into RL frameworks. Specifically, authors proposes to use LLM for exploration,  Q target calculation and imposing conservatism. Experiments on the ALFWorld and Overcooked environments illustrates the effectiveness of the proposed framework.
weaknesses: 1. There are no convergence guarantees of sampling using LLM as prior for Exploration and Q-function update. Moreover, there is no guarantee that learning the Q-function as shown in Equation (7) will give us a conservative Q-function.
2. If the LLM can not provide a good prior, constraining the learned policy as shown in Equation (8) will result in sub-optimal policies.
3. The RL process has to query the LLM each time we update the policy, which is time-consuming and computation-consuming.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: Questions and Suggestions:

- Should the proportionality sign on line 147 be equal to? I understand the sentence is correct in its current form.

- The $p$ is overloaded on lines 112-113, especially, please change the "$p_{LLM}$ as the action prior $p$" sentence where the latter $p$ is said to be not confused with $p$ used for probability of start state or action given a state in the definition of probability of trajectory.

- I am not sure by what is meant by "Denote that the above …" in Proposition 1. Please rewrite this for clarification.

- Line 215 - "reply buffer" should be changed to "replay buffer" 

- Is the gamma in equation 8 the same as the discount factor?

- Line 323 - "avoiding full into holes". Please rectify this.
soundness: 4
strengths: The paper is written very clearly with appropriate figures, well-formatted equations and more importantly lucid logical flow. The contributions of the paper are specified along with particular sections corresponding to individual contribution. Preliminaries provide enough details and are not overexplained. 

The core idea of the paper is pretty neat -- LLMs are trained on large amount of data and they have knowledge about tasks if not fine-grained controllability; this knowledge is useful to guide an RL training. 

I like how the value-based implementation is designed -- sampling a set of actions using LLM prior, taking Q estimates, sampling actions using their Q estimates. The method is well grounded in the theory where direct posterior sampling is shown to be proportional to LLM prior multiplied with soft-Q distribution. The idea being simple, can be plugged with both online and offline Q-learning approaches.

The versatility of the idea is shown by applying it to policy-based RL. The LLM prior guided learning takes form similar to KL-regularized RL, hence modifying PPO in a straightforward way would lead to easy access to prior knowledge. However, I do I have some contentions around the novelty of this part which is discussed in weakness section.


The choice of the experimental setup is also very apt. The environments are chosen so that LLMs can provide textual action descriptions which then can be executed through text to action conversion in the environment's processing.
summary: In this work, the authors propose idea of using large language models (LLMs) for providing prior information about execution of a task, which simplifies exploration in reinforcement learning (RL) and improves sample efficiency. The core idea is to treat LLMs as prior distribution over actions and then apply variational inference in conjunction with posterior sampling. The authors demonstrate this technique on both value-based and policy-based approaches. In value-based implementation, a subset of the action space is chosen for Q-learning instead of using entire action space. This subset selection is done using LLM prior. Similarly, in the policy-based implementation, a KL-penalty is introduced with respect to the prior LLM action distribution, but implemented differently from regular KL penalty in RL from human feedback (RLHF), by posterior sampling of multiple candidate actions. The experiments are conducted on text-based RL environments with large action spaces and multi-step horizon (note this is crucial for demonstrating the difference against RLHF techniques). Value-based baselines include DQN for online learning and CQL for offline learning. For policy-based baseline, PPO algorithm is compared against.

The results demonstrate the superiority of the approach in (1) overall returns, (2) generalizing beyond seen tasks, (3) sample efficiency.
weaknesses: Practicality of value based approaches: Although the value-based direct posterior sampling technique described in the paper is logically sound, I wonder about its practicality. Mainly, value-based approaches when applied to combinatorically large action spaces like text generation is practically infeasible. Hence, the paper's proposed method might be hard to apply beyond toy textual environments. Noticeably, these environments are designed to accept actions in a particular format or use rigorous action decoding, etc. to deal with the combinatoric complexity. Such environments are rare in reality.

Problems with choice of prior inducing LLMs: Two LLMs might produce semantically same action but different text. I am curious to know whether the Q-learning based approaches would assign similar values to both. In general, I feel that prior inducing LLM and its compatibility with the environment's action decoding plays major role when they are used to train RL policies in text-based sequential decision making. 

Novelty of the additional KL penalty in policy-based RL: ELBO-based RL approaches [1, 2] have been already established in the RL literature. The additional KL-penalty proposed in the current paper is based on the same. In single step decision making, the KL-penalty on an LLM prior would be exactly same as KL-penalty used along with supervised fine-tuned (SFT) policy. Thus, in that setting, the paper's technique lacks novelty. Now, coming to the multi-step decision making, unless the proposed direct posterior sampling is also applied on top of PPO, additional KL-penalty is equivalent to using a good enough prior policy (e.g. behavior cloned policy). Therefore, the novelty of the paper is limited in policy-based RL.

Combined together,  I find that paper provides great insights into a direct posterior sampling paradigm for using prior knowledge present in the LLMs, but the approach might have limited applicability beyond hand-designed textual environments, its applicability is limited by limitations of value-based approaches, and its novelty in policy-based RL is not significant.

---

confidence: 4
contribution: 1
initial_rating: 5
presentation: 2
questions: l.286 "We plot the rewards averaged over the final third of the training processes for trainable baselines" --> Evaluation of a model performance is not performed by conducting periodic performance measurements over a test set of tasks ? Could you elaborate on how performance is measured ?

In your policy-based version, which you present in section 3.3 as an extension of the work of Carta et al., 2023, I am not sure to understand what is trained is what is fixed. Are you, like in Carta et al, training all parameters of the LLM ? What is being trained by your modified PPO loss ? Would be good to have the policy-based version of figure 1 somewhere. My understanding after reading section 4.2 is that you have a large fixed pretrained model used to compute action priors, and you train another smaller Flan-T5 as the policy.

In your offline learning experiments, by looking at table 1 it looks like you are applying a vanilla DQN to offline datasets. It comes with the risk of overestimating values of actions that are not supported by the dataset. Would it make more sense to use value-based methods designed for offline learning, e.g. Implicit Q Learning from Kostrikov et al ?

In your offline learning results, I see that CQL underperforms wrt Behavioral Cloning. This is unusual: any ideas on why ?




Given the aforementioned questions and potential limitations of this paper, I will go for a weak reject, and look forward to the discussion to update my score.
soundness: 3
strengths: This paper tackles a very important topic: how to leverage LLMs into sequential decision-making tasks incorporating language.
I enjoyed reading it.
Performance improvements over considered baselines are impressive.
summary: Authors present a new way to train LLM-based agents in interactive sequential environments. Their approach is based on leveraging a potentially large frozen LLM to generate action subsets for encountered states. The action subset is then used to simplify the RL finetuning of a smaller LLM. Their approach can be adapted to value-based (giving a subset of actions for the LLM Q-network) and policy-based scenarios (constraining the LLM policy with a KL-div loss over the subset of actions). Authors showcase how their approach outperforms baselines in TextWorld and Overcooked, both in online and offline learning settings.
weaknesses: ### Value-based version

I am having trouble understanding how useful the value-based formulation is. What I mean is that, from my understanding, at inference time you need both the frozen LLM and the trained Q-network (which is a smaller LLM). This makes the approach quite expensive regarding memory. Regarding inference, it is also costly: the frozen LLM must provide k outputs before being able to use the Q-network LLM, for each step in the environment.
A good comparative study here to motivate this approach could be to have a baseline which discards the frozen-LLM. What happens in this scenario ? Is it what you call the DQN baseline ? Here, it would be interesting to see what happens if you use a larger LLM during training (something which induces a memory burden comparable to using a frozen LLM + a smaller trained Q-net LLM).

### Comparative study with GFlan from carta et al

The only baseline considered from the litterature is GFlan from carta et al. While I am not expert enough to assess whether this is sufficient, I have concerns about this comparative study. 
In carta et al, GFlan is evaluated on a text-based version of the BabyAI gridworld environment:

Why not featuring this environment in the study ?

Any intuitions on why GFlan is failing completely in ALFWorld (TextWorld) while it does maintain some performances in Overcooked ?






### Clarity

While the paper is mostly clear, I had trouble understanding the exact training and inference pipeline. The paper could benefit an update of figure 1 (along with providing the policy-based version of figure 1 somewhere).The right side of Figure 1 could be improved: in the current version, it is hard to visually understand that the Q function is used to weight each input actions and that these weights are used for proportional sampling of the next action. Also in figure 1 it is hard to understand that there are two pre-trained LLMs being used, one frozen generating an action subset, and a smaller one trained and used as a critic. I only understood that there was two pretrained LLMs being used in the DQN-Prior baseline at line 395 "... by combining the Q-function, trained with Qwen-1.5 7B". I would suggest making it clear early on.

l.97-99 "we first generate a free-form output from the LLM, for example, a 7B LLM, which is then mapped to an executable action through a simple rule-based projection." 
--> I would recommend moving at least part of what is discussed in the appendix into the main body of the paper, to be clearer about this. 
Related to this: I only understood that you repeat this output-to-action mapping multiple times at line 166-167. It would be clearer to mention it sooner in the paper.


### Minor / Typos

There are 3 acronym definitions in the abstract, which is a lot. Consider reducing this number, e.g. dropping SDM which is used only once.

l31. "human-AI dialogue (McTear, 2022; Li et al., 2019)," --> make sure to sort citations in ascending order wrt year. This error appears multiple times.

l. 232 --> (Snell et al.) missing year in citation.
 
l.104 "by (Levine, 2018)" --> use \cite not \citep when directly mentioning the citation in the sentence.

l.304 "can be found in the Appendix" -->  refer to the appropriate appendix section(s) rather than a general reference

l.323 "while avoiding full" --> falling

l.352 "Behavior Clone(BC)." --> Behavioral Cloning ?

Results-related figures could be moved closer to the result analysis in section 4.3 to simplify reading

l.404 "k = 1 indicates the performance of the LLM on its own, without involving the Q-function" 
--> Why calling this scenario k=1 and not k=0 ? you could theoretically use your frozen LLM to create a single action subset.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: If the authors can address any number of the points raised in the Weaknesses section, I would be willing to raise my score accordingly. I also have a few minor comments below:

- In Section 4.1, it would be helpful to give some more detail on the environments you use. The current descriptions are quite high level and not everyone may be familiar with these environments or have intuitions about them. Maybe include a figure with example observations/actions for each one - you still have half a page under the page limit so there should be space. 
- Some of the references are missing dates, for example (Snell et al), line 232. 
- This is a bit of a nitpick, but in Figure 1, is there a reason the OpenAI logo is on the LLM box? My understanding is that this paper does not use OpenAI’s LLMs, and instead uses open models (Qwen and Llama). I don’t think an institutional logo is necessary, but if you do want to include one, it would seem more fair to give credit to those that made the models that you use available.
soundness: 3
strengths: - the method is simple and makes intuitive sense. Its benefits are clearly demonstrated, although in fairly simple text-based environments. Although other works have proposed ways to incorporate LLM priors into RL systems, to my knowledge this particular approach is new. 
- the paper is overall clearly written and easy to understand. I have a few suggestions below, but they are relatively minor.
summary: This paper proposes an approach to leverage LLM priors for RL, both in the online and offline settings.
The setting is restricted to text-based observations. Specifically, they estimate a prior p(a|s) by sampling K actions from the LLM conditioned on the state, and use these to either restrict the Q-values to the actions with non-zero probability (for online or offline Q-learning based methods), or to enforce a KL penalty with respect to the policy’s distribution over actions. They evaluate on several text-based environments and show improvements in terms of sample efficiency over vanilla RL. Overall, the paper does a decent job of illustrating the benefits of the method within its scope, but the scope itself is quite limited (in terms of broad applicability and scalability - more on this below). Also, there are a number of alternate methods for incorporating LLM priors into RL systems, that are not discussed or compared to.

=====
Update after rebuttal: the authors have added several additional experiments which partially address the initial limitations I discussed. Although I still think the environments are on the simple side, and the method in its current form is difficult to scale to high-throughput settings, the results are convincing enough in demonstrating the approach's merits and give hints of wider applicability to meet the bar for acceptance.
weaknesses: - the method is limited to settings where the observation consists of text, which is a big limitation. One could imagine workarounds to apply this to images, where one first uses a pretrained captioning model to extract textual captions and then feed these to the LLM. However, it’s not clear if this would give enough granularity to produce a reasonable action prior and this would need to be checked experimentally. 
- the method, as I understand it, will have difficulty scaling to high-throughput RL settings because it requires a call to the LLM for each observation encountered by the agent (which is computationally expensive). This also limits its applicability to small tasks or offline settings where the LLM outputs can be precomputed in parallel and cached. Are there any ways to address this? For example, could you distill the LLM action prior into a lightweight model and use this instead?
- there are other ways to incorporate LLM priors into RL systems, namely through the reward function rather than the action distribution. See for example the references below [1, 2, 3, 4] (this is not a complete list). It is currently unclear what the pros/cons of the proposed method are compared to this other family of approaches. It would be helpful to include at least one method from this family as a comparison in the experiments. Also, these are not discussed in the related work section.  


[1] Eureka: https://arxiv.org/abs/2310.12931
[2] Language2Reward: https://arxiv.org/abs/2306.08647
[3] Motif: https://arxiv.org/abs/2310.00166
[4] Reward Design with LLMs: https://arxiv.org/abs/2303.00001

Meta-review: The authors propose to use LLM priors to guide policies in more efficient exploration. Specifically, they use Bayesian inference to integrate action distribution priors provided by an LLM and perform posterior sampling. They show how this can be done in both policy-based and value-based RL frameworks, and demonstrate improved sample efficiency in simple, discrete-action environments. 

Reviewers found the paper easy to follow and clearly written. The core idea is simple and neat, and performance improvements over the chosen baselines are impressive. However, there were some reviewer concerns about the simplicity of the environments chosen for the experiments, and that the framework does not allow for continuous action environments. There was also no discussion in related work of how this framework compares to providing LLM priors via a reward function. 

The authors provided additional experiments during the rebuttal phase that assuaged some reviewers' concerns. Given the nice idea, provision of both value-based and policy-based frameworks for incorporating prior action distributions, and alternative perspective from the more common reward-based framework, I vote to accept this paper.

Decision: Accept (Poster)