Large Language Models can Implement Policy Iteration
Authors: Ethan Brooks
Logan A Walls
Richard Lewis
Satinder Singh
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
In-Context Learning
Foundation Models
TL;DR: We demonstrate that a large language model is capable of implementing the policy iteration algorithm in order to solve simple reinforcement learning domains.

Abstract: In this work, we demonstrate a method for implementing policy iteration using a large language model. While the application of foundation models to RL has received considerable attention, most approaches rely on either (1) the curation of expert demonstrations (either through manual design or task-specific pretraining) or (2) adaptation to the task of interest using gradient methods (either fine-tuning or training of adapter layers). Both of these techniques have drawbacks. Collecting demonstrations is labor-intensive, and algorithms that rely on them do not outperform the experts from which the demonstrations were derived. All gradient techniques are inherently slow, sacrificing the “few-shot” quality that makes in-context learning attractive to begin with. Our method demonstrates that a large language model can be used to implement policy iteration using the machinery of in-context learning, enabling it to learn to perform RL tasks without expert demonstrations or gradients. Our approach iteratively updates the contents of the prompt from which it derives its policy through trial-and-error interaction with an RL environment. In order to eliminate the role of in-weights learning (on which approaches like Decision Transformer rely heavily), we demonstrate our method using Codex (M. Chen et al. 2021b), a language model with no prior knowledge of the domains on which we evaluate it.

Large Language Models Can Implement Policy Iteration
Ethan Brooks', Logan Walls', Richard L. Lewis", Satinder Singh CCmputer Science and Engineering, University of Michigan *Department of Psychology, University of Michigan Lactanbro,oogulll1 rickl, baveja) @umi ch. edu
Abstract
In this work, we demonstrate method for implementing policy iteration using large language model. While the application foundation models to Reinforce- ment Learning (RI.) has received considerable attention, most approaches rely on either (1) the curation of expert demonstrations (through manual design or task-specific pretraining) (2) adaptation to the task of interest using gradient methods (fine-tuning or training of adapter layers). Both of these techniques have drawbacks. Collecting demonstrations labor-intensive, and algorithms that rely on them do not outperform the experts from which the demonstrations were de- rived. Gradient techniques inherently slow, sacrificing the "few-shot" quality that makes in-context learning attractive to begin with. Our method demonstrates that large language model can be used to implement policy iteration using the machinery of in on-context learning, enabling it to learn to perform RL tasks without expert demonstrations or gradients. Our approach iteratively updates the contents of the prompt from which it derives its policy through trial-and-ere interaction with an RL environment. order to eliminate the role of in-weights learning (on which approaches like Decision Transformer (L. Chen et al. 2222]) rely heavily), we demonstrate our method using Codex (M. Chen et al. (0211)) a language model that was not trained to perform the tasks which we are evaluating.
Introduction
In many settings, models implemented using a transformer or recurrent architecture will improve their performance information accumulates in their context or memory. We refer this phenomenon as "in-context learning." Brown et al. demonstredtt technique for inducing form of learning by prompting large language model with small number of input/output exemplars. An interesting property of in-context learning in the case of large pre-trained models (or "foundation models") is that the models are not directly trained to optimize a meta-learning objective, but demonstrate an emergent capacity to generalize (or at least specialize) to diverse downstream task-distributions (Wei al. 2022b).
This remarkable capability has been studied in a variety of settings, including Reinforcement Learning (RL). However, most work in this area either (1) assumes access expert demonstrations collected
either from human experts (Huang et al. 20226 Baker et al. (022), or domain-specific pre-trained RL
agents (L. Chen et al. 2021 Lee al. 2022, Janner et al. 2021 Reed al. 2022, Xu al. (202)).
or (2) relies on gradient-based methods e.g. fine-tuning of the foundation models parameters as
whole (Lee al. 2022, Reed al. [202 Baker al. 2022) or newly training an adapter layer or
prefix vectors while keeping the original foundation models frozen (X. L. Li et al. 2021 A. K. Singh
al. 2022
Karimi Mahabadi al. 2022)
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

For sae's action in [A(]....A)]])
P
AED
BE THAN
LLM
LLM
LLM
1t A(1)) I) = = ,
LLM
LLM
LLM
Q A(ri)
Environment
Figure For each possible action A(1), .... A(n). the LLM generates rollout by alternately predicting transitions and selecting actions. Q-value estimates are discounted sums of rewards. The action chosen greedily with respect to Q-values. Both state/reward prediction and next action selection use trajectories from D to create prompts for the LLM. Changes to the content of 'D change the prompts that the LLM receives, allowing the model to improve its behavior over time.
Our work demonstrates approach in-context learning which relaxes these assumptions. Our method, In-Context Policy Iteration (ICPI), implements policy iteration using the prompt content, instead of the model parameters, the locus of learning, thereby avoiding gradient methods. Further- more, the use of policy iteration frees us from expert demonstrations because suboptimal prompts can be improved
the course of training.
We illustrate the method empirically on six small illustrative RL tasks- chain, distractor-chain, maze, mini-catch, mini-invaders, and point-mass- in which the method very quickly finds good policies. We also compare five pretrained Large Language Models (LLMs), including two different size models trained natural language OPT-30B and GPT-J-an and three different sizes of model trained on program code-twe two sizes of Codex as well as InCoder. On our six domains, we find that only the largest model (the vade-davinci-O0 variant of Codex) consistently demonstrates learning.
language-OPT-30B
Related Work
common application of foundation models to RL involves tasks that have language input, for example natural language instructions/goals (D. Garg et al. 2022] Hill et al. 2020) or text-based games (Peng et [021] I. Singh al. [021] Majumdar et al. 2020] Ammanabrolu et (222]). Another approach encodes RL trajectories into token sequences, and processes them with a foundation model, model representations as input to deep RL. architectures (S. Li al. 2022] Tarasov
2022
Tam et al. 2022). Finally, a recent set of approaches (which we will focus on in this Related Work section) treat RL as a sequence modeling problem and use the foundation models itself to predict states or actions. In this related work section, we will focus third set of recent approaches that treat reinforcement learning (RL) as sequence modeling problem and utilize foundation models for state prediction, action selection, and task completion. We will organize our survey of these approaches based on how they elicit these RL-relevant outputs from the foundation models. this respect the approaches fall under three broad categories: learning from demonstrations, specialization (via training or finetuning), and context manipulation (in-context learning).
2.1 Learning from demonstrations
Many recent sequence-bee based approaches to reinforcement learning use demonstrations that come either from human experts or pretrained RL agents. For example, Huang al. 2002b) use frozen LLM as a planner for everyday household tasks by constructing prefix from human-generated task instructions, and then using the L.LM generate instructions for new tasks. This work extended by Huang et al. 2022a Similarly, Ahn al. 2022 use value function that is trained on

---

human demonstrations to rank candidate actions produced by an LLM. Baker et al. 2022 use human demonstrations train the foundation model itself: they use video recordings of human Minecraft players to train foundation models that plays Minecraft. Works that rely on pretrained RI. agents include Janner et al. 202T who train a "Trajectory Transformer" to predict trajectory sequences in continuous control tasks by using trajectories generated by pretrained agents, and L. Chen et al. 2002 who use dataset of offline trajectories to train a "Decision Transformer" that predicts actions from state-action-reward sequences in RL environments like Atari Two approaches build on this method to improve generalization: Lee et al. [022 use trajectories generated by a DQN agent to train a single Decision Transformer that can play many Atari games, and Xu al. 2022 use combination of human and artificial trajectories to train a Decision Transformer that achieves few-shot generalization on continuous control tasks. Reed et al. 2022 take task- generality step farther and use datasets generated by pretrained agents to train a multi-modal agent that performs a wide array of RI. (e.g. Atari, continuous control) and Ron-RL (e.g. image captioning. chat) tasks.
Some of the above works include non-expert demonstrations as well. L. Chen et al. 202 include experiments with trajectories generated by random (as opposed to expert) policies. Lee et al. 2022 and Xu et al. 2022 also use datasets that include trajectories generated by partially trained agents in addition to fully trained agents. Like these works, our proposed method (ICPI) does not rely on expert demonstrations but we note two key differences between our approach and existing approaches. Firstly, ICPI only consumes self- gelfraenertted trajectories, it does not require any demonstrations (like L. Chen et al. [202T] with random trajectories. but unlike Lee et al. [022] Xu et al. [2022] and the other approaches reviewed above). Secondly, ICPI relies primarily
in in-context learning rather than in-weights learning to achieve generalization (like Xu et al. 2022, but unlike L.. Chen et al. 2021 & Lee et al. 2022). For discussion about in in-weights vs. in-context learning see Chan et al. 2022
2.2 Gradient-hased training & finetuning on RL tasks
Many approaches that use foundation models for RI. involve specifically training or fine-tuning on RL tasks. For example, Janner et al. 2021 L. Chen et al. 2021; Lee al. 2022 Xu et al. 2022] Baker et al. 2022] Reed et al. [2022] all use models that are trained from scratch on tasks of interest, and A. K. Singh al. 2022 Ahn et al. 2022] Huang et trainable components or adapters. In contrast, Huang et al. 2022bjuse use frozen foundation models for planning, without training or fine-tuning on RL tasks. Like Huang al. 2022b ICPI does not update the parameters of the foundation model, but relies on the frozen model's 's in-context learning abilities. However, ICPI gradually builds and improves the prompts within the space defined by the given fixed text-format for observations, actions, and rewards (in contrast to Huang et al. 2022b) which uses the frozen model select good prompts from a given fixed library of goal/plan descriptions).
combine frozen foundation models with
2.3 In-Context learning
Several recent papers have specifically studied in-context learning. Laskin et al. 2022 demonstrates an approach to performing in-context reinforcement learning by training a model on complete RL learning histories, demonstrating that the model actually distills the improvement operator of the source algorithm. Chan et al. 2022 and S. Garg et al. 2022 provide analyses of the properties that drive in-context learning, the first in the context of image classification, the second in the context of regression onto a continuous function. These papers identify various properties, including "burstiness," " model-size, and the sensitivity of in-context learning to small perturbations of the context. They propose a novel method that uses sensitivity as a proxy for model certainty. Some recent work has explored iterative forms of in-context learning, similar to our own. For example, Shinn et al. 2023 and Madaan et al. 2023 use iterative self-refin refineeeen to improve the outputs of large language model in natural language context. These approaches rely on the ability of the model to examine and critique its own outputs, rather than using policy iteration as our method does.
that in-context learning depends on. Y. Chen et al. [022] studies
3 Method
How can standard policy iteration make use of in-context learning? Policy iteration is either model- based-using world-model to plan future trajectories the environment or model-free-inne inferring value-estimates without explicit planning. Both methods can be realized with in-context learning.
3

---

Algorithm 2 Computing Q-values
1: function Q(o,, a, D)
2:
4- t
3:
o(1) = Oz
4: a(1)
5: repeat
D All samples come from the experience buffer D b balancing terminal and non non-terminal
Db ~ time-steps with action a!")
6:
b(8) LLM (Do, olw), ala))
7:
8:
D, ~ time-steps with action (1)) and termination ((x)
balancing reward
,(4) ~ LLM (D,,o! aa())
9:
balancing D. time-steps with action (1)) and termination ((u)
10:
d(*+1) ~ LLM (Do, o(*), a()))
11:
Da recent trajectories
( LLM (o(u+1), Da)
14:
+ u++ until ((4) is terminal
16:
return Σx=1 - k-
end function
We choose model-based learning because planned trajectories make the underlying logic of value- estimates explicit to our foundation model backbone, providing concrete instantiation a trajectory that realizes the values. This ties into recent work (Wei et al. 2022a Nye et al. 2021) demonstrating that "chains of thought" can significantly improve few-shot performance of foundation models.
Model-based RL requires two ingredients, rollout-policy used to act during planning and a world- model used to predict future rewards, terminations, and states. Since our approach avoids any mutation of the foundation model's 's parameters (this would require gradients), we must instead induce the rollout-policy and the world-model using in-context learning, i.e. by selecting appropriate prompts. We induce the rollout-policy by prompting the foundation model with trajectories drawn from the current (or recent) behavior policy (distinct from the rollout-policy). Similarly, we induce the world-model by prompting the foundation models with transitions drawn from the agent's history of experience. Note that our approach assumes access to some translation between the state-space of the environment and the medium (language, images, etc.) of the foundation models. This explains how an algorithm might plan and estimate values using a foundation model. It also explains how the rollout-policy approximately tracks the behavior policy.
How does the policy improve? When acting the environment (as opposed to planning), we choose the action that maximizes the estimated Q-value from the current state (see) Training Looplpseudocode, line 6). At time step I, the agent observes the state of the environment (denoted 8x) and executes action a, arg m (T()) (oc, a), where A (A(1), .A(n)) denotes the set of actions
Algorithm 1 Training Loop
1:
function TRAIN(enviromment)
2: initialize D
replay buffer containing full history of behavior
3:
while training do
4: On &- Reset environment.
5: while episode is not done do
6:
a é arg max, Q(o, a, D)
policy improvement
7:
0.41, b, + Execute in environment.
8:
1 1+1+1
9:
end while
D -- UUU (oos 4g. To, by o..... .......... Oz 0,, rz, 1t, 9+1)
D add trajectory buffer
11:
end while
12:
end function

---

available, denotes the policy of the agent at time step t, and Q(7)) denotes the Q-estimate for policy g. Taking the greedy (arg max) actions with respect to Q implements new and improved policy.
Computing Q-values This section provides details on the prompts that we use in our computation maintain buffer 'D of transitions experienced by the agent. To compute Q(oe)(on a) at time step t. in the o(T)) al?),
of Q-values (see Computing Q-values pseudocode & Figure D. During training,
real-world we rollout simulated trajectory o(2) Oz, a²)) r()), o(2), a(2), ,,(2)
r(T), o(T+1) by predicting, at each simulation time step u: reward r(") ~ LLM (Dr, ol"), (
termination b(a) LLM (Do; of"), a(")); observation of*+1) LLM (Do, ol",, a a(");; action (11) LLM (Da; ofw)). Termination b(a)) decides whether the simulated trajectory ends at step u.
The prompts D,. Do contain data sampled from the replay buffer. For each prompt, we choose some subset of replay buffer transitions, shuffle them, convert them to text (examples are provided in table 4.1) and clip the prompt at the 4000-token Codex context limit. We use the same method for Da. except that we use random trajectory subsequences.
In order to maximize the relevance of the prompt contents to the current inference we select transitions using the following criteria. Ds contains (ok, 0ke by) tuples such that ag equals (!"), the action for which the LLM must infer termination. D, contains (ok, ag T tuples, again constraining ag = a (w) but also constraining by j(() - that the tuple corresponds to a terminal time-step the LLM inferred 6(x) true, and to a non-terminal time-step if 6(m) false. For Do. the prompt includes (ox, 0g tuples with 0g (*) and by false (only non-terminal states need to be modelled).
We also maintain a balance of certain kinds of transitions in the prompt. For termination prediction, we balance terminal and non-terminal time-steps. Since non-terminal time-steps far outnumber terminal time-steps, this climinates a situation wherein the randomly sampled prompt time-steps entirely non-terminal, all but ensuring that the LLM will predict non-termination. Similarly, for reward prediction, we balance the number of time-steps corresponding to each reward value stored in D. In order to balance two collections of unequal size, we take the smaller and duplicate randomly chosen members until the sizes are equal.
In contrast to the other predictions, we condition the rollout policy on trajectory subsequences, not individual time-steps. Prompting with sequences better enables the foundation model to apprehend the logic behind policy. Trajectory subsequences consist of (OR, ax) pairs, randomly clipped from the most recent trajectories. More recent trajectories will, in general demonstrate higher performance, since they come from policies that have benefited from more rounds of improvement.
In contrast to the other predictions, we condition the rollout policy on trajectory subsequences, not individual time-steps. Prompting with sequences better enables the foundation model to apprehend the logic behind a policy. Trajectory subsequences consist of (ok: a*) pairs, randomly clipped from the c most recent trajectories. More recent trajectories will, in general demonstrate higher performance, since they come from policies that have benefited from more rounds of improvement.
Finally, the Q-value estimate is simply the discounted sum of rewards for the simulated episode. Given this description of Q-value estimation, we now return to the concept of policy improvement.
Policy-Improvement The arg max (line of Algorithm I drives policy improvement in ICPI. Critically this is not simply one-step improvement but mechanism that builds improvement on top of improvement. This occurs through a cycle in which the arg max improves behavior. The improved behavior is stored in the buffer D. and then used to condition the rollout policy. This improves the returns generated by the LLM during planning rollouts. These improved rollouts improve the Q-estimates for each action. Completing the cycle, this improves the actions chosen by the arg max. Because this process feeds into itself, it can drive improvement without bound until optimality is achieved.
Note that this process takes advantage of properties specific to in-context learning. In particular, relies on the assumption that the rollout policy, when prompted with trajectories drawn from a mixture of policies, will approximate something like an average of these policies. Given this assumption, the rollout policy will improve with the improvement of the mixture of policies from which its prompt-trajectories are drawn. This results in kind of rapid policy improvement that works without any use of gradients.
5

---

Algorithms
ICPI No ArgMax
Tabular Q Matching Model
Chain
Distractor Chain
Mini Catch
Mini Invaders
Point Mass
0.8
0.6
2
00.4
0.2
20
40
60
0
40
80 step
120
40
120
0
80
160
240
0
80 step
160
0
80
160
step
step
step
Figure Comparison of ICPI with three baselines, "No arg max", "Tabular Q." and "Nearest Neighbor." The y-axis depicts regret (normalized between and 1), computed relative to an optimal return with discount-factor of 0.8. The x-axis depicts time-steps during training. Error bars are standard errors from four seeds.
Prompt-Format The LLM cannot take non-linguistic prompts, so our algorithm assumes access to a textual representation of the environment of states, actions, terminations, and rewards- and some way to recover the original action, termination, and reward values from their textual representation (we do not attempt to recover states). Since our primary results use the Codex language model (see Table T. we use Python code to represent these values (examples are available Table 2 in the appendix).
In our experiments, we discovered that the LLM world-model was unable to reliably predict rewards, terminations, and next-states on some of the more difficult environments. We experimented with providing domain hints in the form of prompt formats that make explicit useful information -smilar to Chain of Thought Prompting (Wei et al. 2022a). For example, for the chain domain, the hint includes an explicit comparison (== !=) the current state with the goal state. Note that while
hints are provided in the initial context, the LLM must infer the hint content in rollouts generated from this context.
We use a consistent idiom for rewards and terminations, namely assert reward == x and assert done or assert not done. Some decisions had to made when representing states and actions. In general, we strove to use simple, idiomatic, concise Python. On the more challenging environments, we did search over several options for the choice of hint. For examples, see Table We anticipate that in the future, stronger foundation models will be increasingly robust to these decisions.
Experiments
We have three main goals in our experiments: (1) Demonstrate that the agent algorithm can in fact
quickly learn good policies, using pretrained LLMs, in set of six simple illustrative domains of increasing challenge; (2) provide evidence through an ablation that the poliey-improvement step- taking the arg over Q-values computed through LLM rollouts acceleraes learning; and (3) investigate the impact of using different LLMs (see Table T different sizes and trained on different data, particular, trained (mostly) natural language (GPT-3 and GPT-J) program code (Codex and InCoder). We next describe the six domains and their associated prompt formats, and then describe the experimental methodology and results.
4.1 Domains and prompt format
Chain. In this environment, the agent occupies an 8-state chain. The agent has three actions: Left. right, and try goal. The try goal action always terminates the episode, conferring reward of on state 4 (the goal state) and on all other states. Episodes also terminate after time-steps. States are represented as numbers from O to 7, in assert state == with the appropriate integer substituted for n. The actions are represented as functions () right O) and try try- (). For the hint, we simply indicate whether not the current state matches the goal state, 4.
Distractor Chain. This environment an 8-state chain, identical to the chain environment, except that the observation is pair of integers, the first indicating the true state of the agent and the second acting as a distractor which transitions randomly within (O...., 7). The agent must therefore learn to ignore the distractor integer and base its inferrences on the information contained in the first integer.

---

Aside from the addition of this distractor integer to the observation, all text representations and hints are identical to the chain environment.
Maze. The agent navigates a small 3 x 3 gridworld with obstacles. The agent can move up, down, left, or right. The episode terminates with a reward of 1 once the agent navigates to the goal grid, or with reward of O after 8 time-steps. This environment tests our algorithms capacity to handle 2-dimensional movement and obstacles, as well as a 4-action spaee. We represent the states as namedtuples - Cxx, y), with integers substituted for x and Similar to chain, the hint indicates whether or not the state corresponds to the goal state.
Mini Catch. The agent operates paddle to catch falling ball. The ball falls from a height of 5 units, descending one unit per time step. The paddle can stay in place (not move), or move left or right along the bottom of the 4-umit wide plane. The agent receives reward of for catching the ball and 0 for other time-steps. The episode ends when the ball's height reaches the paddle regardless of whether or not the paddle catches the ball. We chose this environment specifically to challenge the eetionnineerencc/rllllot--p policy component of our algorithm. Specifically, note that the success condition in Mini Catch allows the paddle to meander before moving under the ball-as long as it gets there on the final time-step. Successful trajectories that include movement away from the ball thus make good rollout policies more challenging to learn (i.e., elicit from the LLM via prompts). Again, we represent both the paddle and the ball as namedtuples C C(x, y) and we represent actions as methods of the paddle object: paddle. stay O. paddle 1eft (), and paddle right () For the hint, we call out the location of the paddle's 's z-position, the ball's x zz-position, the relation between these positions (which is larger than which, or whether they equal) and the ball's y-position. Table ii the appendix provides an example. We also include the text bww..alllle....ee.....oo.aaaelee descend () to account for the change in the ball's position between states.
Mini Invaders. The agent operates a ship that shoots down aliens which descend from the top of the screen. At the beginning of episode, two aliens spawn random location in two of four columns. The episode terminates when an alien reaches the ground (resulting in O reward) or when the ship shoots down both aliens (the agent receives reward per alien). The agent can move left, right, or shoot. This domain highlights ICPI's capacity to learn incrementally, rather than discovering an optimal policy through random exploration and then imitating that policy, which is how our "No arg max" baseline learns (see Comparison of ICPI with baseline algorithms). ICPI initially learns to shoot down one alien, and then builds on this good but suboptimal policy discover the better policy of shooting down both aliens. In contrast, random exploration takes much longer to discover the optimal policy and the "No arg max" baseline has only experienced one or two successful trajectories by the end of training.
We represent the ship by its namedtuple coordinate (C(x, y)) and the aliens list of these namedtuples. When an alien is shot down, we substitute None for the tuple, as in aliens == [C(x, y), None]. We add the text for in aliens: a.descend in order account for the change in the alien's position between states.
Point-Mass. A point-mass spawns at random position on continuous line between 66 and +6 with a velocity of O. The agent can either accelerate the point-mass (increase velocity by 1) or decelerate it (decrease the velocity by 1). The point-mass position changes by the amount of its velocity each timestep. The episode terminates with reward of once the point-mass is between -2 and +2 and its velocity is 0 once again. The episode also terminates after 8 time-steps. This domain tests the algorithm's 's ability to handle continuous states.
States represented as assert pos == p and vel == v. substituting floats rounded to two decimals for and v. The actions are accel (pos, vel) and decel (pos, vel). The hint indicates whether the success conditions are met, namely the relationship of pos -2 and +2 and the whether or not vel == O. The hint includes identification of the aliens' and the ship ' x x-positions as well as comparison between them.
4.2 Experiment Methodology and Results
Methodology and evaluation. For the results, we record the agent's regret over the course of training relative to an optimal policy computed with discount factor of 0.8. For all experiments
7

---

Ablations
ICPI No Hints c=16 No Constraints
No Balance
Chain
Distractor Chain
Maze
Mini Catch
Mini Invaders
Point Mass
0.8
0.6
2 0.44
0.2
20
40
60
80
0
40
80 step
120
40
80 step
120
0
80
160
240
0
80 step
160
0
80
160
step
step
step
Figure 3: Comparison of ICPI with ablations. The y-axis depicts regret (normalized between 0 and 1), computed relative to an optimal return with a discount-factor of 0.8. The z-axis depicts time-steps during training. Error bars are standard errors from four seeds.
8= (the number of most recent successful trajectories to include in the prompt). We did not have time for hyperparameter search and chose this number based on intuition. However the = 16 baseline demonstrates results when this hyperparameter is doubled. All results 4 seeds.
For both versions Codex, we used the OpenAl Beta under the API Terms of Use. For GPT-J (B. Wang et al. (021)) InCoder (Fried et al. 2022) and OPT-30B (Zhang et al. 2022), we used the open-source implementations from Huggingface Transformers (Wolf et al. 2020). each running
on one Nvidia A40 GPU. All language models use sampling temperature of 0.1. Code for
implementation available at /ttttss://gtthiii com/ ethanabrooks/icpl
Comparison of ICPI with baseline algorithms. We compare ICPI with three baselines (Fig.
The "No arg max" baseline learns a good policy through random exploration and then imitates this policy. This baseline assumes access a "success threshold" for each domain an undiscounted cumulative return greater than which a trajectory is considered successful. The action selection mechanism emulates ICPI's rollout policy: prompting the LLM with set of trajectories and eliciting an action as output. For this baseline, we only include trajectories in the prompt whose cumulative return exceeds the success threshold. Thus the policy improves as the number of successful trajectories in the prompt increases over time. Note that at the start of learning, the agent will have experienced too few successful trajectories to effectively populate the policy prompt. In order to facilitate exploration, we act randomly until the agent experiences 3 successes.
"Tabular Q" is a standard tabular Q-learning algorithm, which uses learning rate of 1.0 and opti- mistically initializes the Q-values to 1.0.
"Matching Model" is a baseline which uses the trajectory history instead of an LLM to perform modelling. This bascline searches the trajectory buffer for the most recent instance of the current state, and in the case of transition/rwwar//ermmnttt found, the model outputs the historical value (c.g. the reward associated with the state-action pair found in the buffer). If no match is found, the modelling rollout is terminated. Recall that ICPI breaks ties randomly during action selection so this will often lead to random action selection.
prediction, the current action. If match is
As our results demonstrate, only ICPI learns good policies on all domains. We attribute this advantage to ICPI's ability to generalize from its context to unseen states and state/action pairs (unlike "Tabular Q" and "Matching Model"). Unlike "No arg max" ICPI is able to learn progressively, improving the policy before experiencing good trajectories.
Ablation of ICPI components. With these experiments, we ablate those components of the algorithm which are not, in principle, essential to learning (Fig. 3). "No Hints" ablates the hints described the PPrompt- time-steps described in the Computing Q-values) paragraph (for example, D, is allowed to contain an unequal number of terminal and non-terminal time-steps). The "No Constraints" baseline removes the constraints on these time-steps described in the same paragraph. For example, D, is allowed to contain a mixture of terminal and non-terminal time-steps (regardless of the model's termination prediction). Finally, "c = 16" prompts the rollout policy with the last 16 trajectories (instead of the last 8, as ICPI). We find that while some ablations match ICPI's performance in several domains, none match its performance on all six.
paragraph. "No Balance" removes the balancing of different kinds of
8

---

Language Models
code-davinc--002 code-cushman-001
OPT-30B
InCoder GPT-J
Chain
Distractor Chain
Maze
Mini Catch
Mini Invaders
Point Mass
0.8
0.6
2
0..4
0.2
20
40
60
80
0
40
80 step
120
40
120
0
80
160
240
0
80
160
80
160
step
step
step
Figure Comparison of different language models used implement ICPI. The y-axis depicts regret (normalized between 0 and 1), computed relative to an optimal return with discount-factor of 0.8. The z-axis depicts time-steps of training. Error bars are standard errors from four seeds.
Comparison of Different Language Models. While our lab lacks the resources to do full study of scaling properties, we did compare several language models varying size (Fig. 4 See Table for details about these models. Both code setee-dvvini---n and code-cushman- -01 are variations of the Codex language model. The exact number of parameters in these models is proprietary according OpenAI, but M. Chen et al. 2021a describes Codex fine-tuned from GPT-3 (Brown al. 2020aa) which contains 185 billion parameters. As for the distinction between the variations, the OpenAl website describes Site-ccsshaa-O00 as "almost capable Davinci Codex, but slightly faster."
found that the rate learning and final performance of the smaller models fell significantly short of Codex all but the simplest domein, chain. Examining the trajectories generated by agents trained using these models, we noted that in several cases, they seemed to struggle to apprehend the underlying "logic" of successful trajectories, which hampered the ability of the rollout policy to produce good actions. Since these smaller models were not trained on identical data, we are unable to isolate the role of size in these results. However, the failure of all of these smaller models to learn suggests that size has some role to play in performance. We conjecture that larger models developed in the future may demonstrate comparable improvements in performance over Codex model.
Limitations ICPI can theoretically work on any control task with discrete actions, due to the guarantees associated with policy iteration. However, since our implementation uses Codex, the domains in our paper were limited by the ability to encode states as text and to fit these encodings in the model's context window. Moreover, Codex demonstrated a limited ability to predict transitions and actions in more complex domains. As sequence models mature, we anticipate that more domains will become tractable for ICPI. We also note that reliance on the proprietary OpenAPI API limits exact reproduction of these results.
Societal Impacts An extensive literature (Tamkin al. 2021 Abid et al. 2021 Liang al. 2021 Pan et al. 2023) has explored the possible positive and negative impacts of LLMs. Some of this work has explored mitigation strategies. In extending LLMs to RL, our work inherits these benefits and challenges. We highlight two concerns: the use of LLMs to spread misinformation and the detrimental carbon cost of training and using these models.
Conclusion
Our main contribution is method for implementing policy iteration algorithm using Large Language Models and the mechanism of in-context learning. The algorithm uses a foundation models as both world model and policy to compute Q-values via rollouts. Although we presented the method here as text-based, is general enough to be applied to any foundation models that works through prompting, including multi-modal models like Reed al. 2022 and Seo et al. 2022 In experiments we showed that the algorithm works in six illustrative domains imposing different challenges for ICPI, confirming the benefit of the LLM-rollout- bolsout-based policy improvement. While the empirical results are preliminary, we believe the approach provides an important new way to use LLMS that will increase in effectiveness the models themselves become more powerful.

---

Table 1: Pretrained Large Language Models (LLMs) Used in Experiments
Model
Parameters
Training data
"The Pile" (Leo Gao et al. 2020), an 825GB En- glish corpus incl. Wikipedia, GitHub, academic pubs
GPT-J (B. Wang al. 2021
6 billion
InCoder (Fried et al.
2022)
6.7 billion
159 GB of open-source StackOverflow code
180B tokens of predominantly English data in- cluding "The Pile" (Leo Gao al. 2020) and iPushShift.io Reddit" (Baumgartner et al. 2020
OPT-30B (Zhang al. 2022)
30 billion
Codex (M. Chen et al. 2021a
185 billion
179 GB of GitHub code
References
[1] Abubakar Abid et al. "Persistent anti-muslim bias in large language models". In: Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 2021, pp. 298-306.
[2] Michael Ahn et al. Do As Can, Not As I Say: Grounding Language in Robotic Affordances.
arXiv:2204.0169) [cs]. Apr. 2022. DOI: 10 48550 / arXiv 2204 01691
URL: http
//arxiv.org 2204 01691 (w16691vvviiiii (visited on 08/09/2022).
[3] Prithviraj Ammanabrolu et al. "Learning Knowledge Graph-based World Models of Tex- tual Environments". In: Advances in Neural Information Processing Systems. Vol. 34. Cur- ran Associates, Inc., 2021, pp. 3720-3731. URL: https http:///oocceeeeeeeeeeoee neurips. cc/
optp://0000/mae//f776oa529111111 Abstract.bbst..t........ htmllltttttmmll... (visited on 05/16/2022).
[4] Bowen Baker et al. Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos. arXiv:2206. 1795 [cs]. June 2022. DOI: [0 48550 arXiv 2206 11795
URL:
http://arxiv. orgorrggog.oo.. abs www.///.......... 11795 (visited 08/09/2022).
[5] Jason Baumgartner al. The Pushshift Reddit Dataset. Number: arXiv:2001
ar 7Xiv:2001.08473 [cs]. Jan. 2020. DOI: 10 48550 / arXiv 2001 08435
URL: http
//arxiv. 2001 08435| (visited 08/02/2022).
[6] Tom Brown et al. "I.anguage Models are Few-Shot Learners". In: Advances Neural Informa- tion Processing Systems. Vol. 33. Curran Associates, Inc., 2020, pp. 1877-1901. URI.: https
/ /papers nips cc / paper / 2020 / hash 1457:0066FF00000771111188a
Abstract. html (visited on 05/14/2022).
[7] Tom Brown al. "Language models few-shot learner".. In: Advances in neural information processing systems 33 (2020), pp. 1877-1901.
[8] Stephanie C. Y. Chan et al. Data Distributional Properties Drive Emergent In- In-Context Learning Transformers. arXiv:2205 05055 [cs]. May 2022. DOI: 10. 40550/8550400 arXiv. 2205 05055
URL: http::/arxii..../ http://aaxxvv..r/bbbbaaa 2205. 05055 (visited on 08/1 1/2022).
[9] Lili Chen et al. "Decision Transformer: Reinforcement Learning via Sequence Modeling". In:
arXi::2106.0/ [cs] (June 2021). URl: hal::htt:://an htap:://///xvvv.//aaa org/ abs /106. 01345 (visited
05/18/2022).
[10] Mark Chen et Evaluating Large Language Models Trained on Code. Tech. rep. arXiv:2107.03377 arXiv, July 2021. URL: http:///axi/i..orrtt hasgg//bb///a /bts/107. 03374 (visited 05/15/2022).
[11] Mark Chen al. "Evaluating large language models trained on code". arXiv preprint
arXiv:2107. 101.2200770337 (2021).
[12] Yanda Chen et al. On the Relation between Sensitivity and Accuracy in In-context Learning (visited on
arXiv:2209.074 107661 [cs]. Sept. 2022. URL: http::/arxiv www.aawwwwww...aa/a/aaaaaaa 09/28/2022).
10

---

[13] Daniel Fried et al. "InCoder: A Generative Model for Code Infilling and Synthesis". In: arXiv:2204. 059999 [cs] (Apr. 2022). URL: hoL::htt::/// Ut.: on 05/15/2022).
org/ hwwpg//ab//2224.. 25999 05999 (visited
[14] Leo Gao et al. "The Pile: An 800GB Dataset of Diverse Text for Language Modeling". In: arXiv:2 [cs] (Dec. 2020). URL:
wttp://www...../////........ (visited
05/18/2022).
[15] Divyansh Garg et al. "LISA: Learning Interpretable Skill Abstractions from Language". In:
arXiv:2203.0005. [cs] (Feb. 2022). URL:
http://wwwr.aab//.....o.o 00054 (visited
on 05/14/2022).
[16] Shivam Garg et al. What Can Transformers Learn In-Context? A Case Study of Simple Function Classes. arXiv:2208.0100 01066 [cs]. Aug. 2022. D01: 10. 48550/ arXiv. 2208. 01066 URL: http://aaret
org/ httgg/bs/222000 2208 01066 (visited on 09/28/2022).
[17] Felix Hill et al. "Human mnstractioo-Foolloviin with Deep Reinforcement Learnnnn via Transfer- Learning from Text". In: arXiv:2005.099: 09382 [cs] (May 2020). URL: http:://arat //axxvv..vvv/////a///i/ org/
(visited on 05/15/2022).
[18] Wenlong Huang et al. Inner Monologue: Embodied Reasoning through Planning with Lan-
guage Models, arXiv:2207 .05008 [es]. July 2022. DO1: 10 48550/ a20... 05608 URL:
http://arxiv org/ ab27 05608 (visited on 08/09/2022).
[19] Wenlong Huang al. "Lannuage Models Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents". In: arXiv:2201.07200 [es] (Mar. 2022). URI.: http
//arxiv.org/ abs abs /ttp//221...0 07207 (visited 05/14/2022).
[20] Michael Janner al. Offfine Reinforcement Learning as One Big Sequence Modeling Problem. ar arXiv:2106.02039 [cs]. Nov. 2021. DOT: 10 48550 / arXiv 2106 02039
URL: http
//axiivorg/aall 2206. 02039 (visited on 08/09/2022).
[21] Rabeeh Karimi Mahabadi et al. "Prompt-fres and Efficient Few-shot Learning with Language Models" In: Proceedings of the 60th Annual Meeting of the Association for Computational
Linguistics ( (Volume 1: Long Papers). Dublin, Ireland: Association for Computational Lin-
guistics. May 2022, pp. 3638-3652. DOI: 10 18653 / v1 2022. acl acl-long 254 URL:
https://aclaanthology wwwpo//2........oooo. acl sacc-long long. 254 (visited on 08/09/2022).
[22] Michael Laskin et al. "In-context reinforcement learning with algorithm distillation". In: arXiv
preprint arwiv:22iivi.0.... 14215 (2022).
[23] Kuang-Huei Lee et al. Multi-Gami Decision Transformers. arXiv:2205.1524/ [cs]. May 2022.
DO1: 10 48550 arxiv 2205 15241
URL: http: // arxiv org /bs 2205 15241
(visited on 08/09/2022).
[24] Shuang Li et al. "Pre-Trained Language Models for Interactive Decision-Making" In: arXiv:2202.01771 [cs] (Feb. 2022). URL: http:://arttt /ttpvv/xxvv.//// org/ abs 200//2202.000 01771 (vis- ited on 05/14/2022).
[25] Xiang Lisa Li et al. "Prefix-Tuning" Optimizing Continuous Prompts for Generation".. en. In:
(Jan. 2021). DO1: 10. 48540885/ak0 2101 00190
URL:
00190v1 (visited on 05/15/2022).
[26] Paul Pu Liang et al. "Towards understanding and mitigating social biases in language models".
In: International Conference on Machine Learning. PMLR. 2021, pp. 6565-6576.
[27]
Aman Madaan et al. "Self-refine: Iterative refinement with self feffffeedback... . In: arXiv preprint arXiv:2303. / (2023).
[28] Arjun Majumdar et al. "Improving Vision-and-Languun Navigation with Image-Texi Pairs from the Web". In: arxiv:2004. 14973 [cs] (May 2020). URL: http: arxiv. .rg abs/
2004. 14973 (visited on 05/14/2022).
[29] Maxwell Nye et al. Show Your Work: Scratchpads for Intermediate Computation with Language
Models. arXiv:21 12.00: 14 [cs]. Nov. 2021. DO1: 10 48550/ arXiv 2112. 00114
URL:
http://arxiv org/ / 1ttp//21111111111..1111111111111 (visited on 08/1 1/2022).
[30] Yikang Pan et al. "On the Risk of Misinformation Pollution with Large Language Models", In:
arXiv preprint 23..i::333..3666 (2023).
[31] Xiangyu Peng et al. "Inherently Explainable Reinforcement Learning in Natural Language", In:
arXiv:2112.08907 [cs] (Dec. 2021). URL: http::t/tat://i/ org/ htrgg/bbs/1122 08907 (visited on 05/14/2022).

---

[32] Scott Reed et al. A Generalist Agent. arXiv:2205.06175 [cs]. May 2022. DOI: 10. 48550/ arXiv. 200. 2205. URR: ItL::tttp://axiiv.co org/ atsp//bb..././11/1/ 06175 (visited on 08/09/2022).
[33] John Schulman et al. "Proximal policy optimization algorithms". In: arXiv preprint
vrXiv:/707.06349 (2017).
[34] Younggyo Seo al. HARP: Autoregressive Lalent Video Prediction with High-Fidelity Image
Generator. 01t1v:2229.0714444 [es]. Sept. 2022. URL: hall:http:/aaaae htttp://axxiv.ort org/ abs/ 2209 071443
(visited on 09/28/2022).
[35] Noah Shinn et al. Reflexion: Language Agents with Verbal Reinforcement Learning. 2023.
arXiv: 2303. 11366 [Cs.AI]
[36] Aaditya K. Singh et al. Know your audience: specializing grounded language models with the game of Dixit. Ai::::220 08349 [es]. June 2022. DOI: 10. 48850/ arXiv 2206 08349 URL:
http://arxiv .rg/ abaaaaaaa2222 08349 (visited on 08/09/2022).
[37]
Ishika Singh et al. "Pre-trained Language Models Prior Knowledge for Playing Text- Games". In: arXiv 107.0400 [cs] (Dec. 2021). URL. http / arxiv org/ /a/s//b///// 2107
08408 (visited on 05/14/2022).
[38] Allison C. Tam et al. "Semantic Exploration from Language Abstractions and Pretrained Representations". In: TrXivv2204.050883 [cs] (Apr. 2022). URL: httpp//arxii wwtpsiii.rrgg.oooooo..... abs/ 2204.
on 05/14/2022).
[39] Alex Tamkin et al. "Understanding the capabilities, limitations, and societal impact of large language models". In: arXiv preprint arXiv:2102.06 02.00003.22222 (2021).
[40] Denis Tarasov et al. "Prompts and Pre-Trained Language Models for Offline Reinforcement Learning". en. In: Mar. 2022. URL: httpp///w/.pggeeeeeeeeeeee//r//e (visited on 05/14/2022).
[41] Ben Wang et al. GPT-J-6B: A 6 Billion Parameter Autoogeeieee Language Model. May 2021. URL: https:::p/// httaa:://github www.eccom/kkikeeeee wttp:////owl///////a/aaaaaaaaaaa
[42] Jason Wei et "Chain of Thought Prompting Elicits Reasoning Large Language Models". In: 1rXiv:2201.11903 [cs] (Apr. 2022). URL: htt.htttp///xiivi org abs 2201 11903 (visited
on 05/15/202
[43] Jason Wei et al. Emergent Abilities of Large Language Models. ar 2ttpoddeo...Xxxi222266677/66// [cs]. June 2022. DOT: 10. 48550/ /ar0/vari.v//.. (visited on 08/11/2022).
URL.: http:::/aaarrvv.org/aa//////// wwtp://///aaa///////// 07682
[44] Thomas Wolf et al. "Transformers: State-of-the- Art Natural Language Processing", In: Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations: Online: Association for Computational I Linguisties, Oct. 2020, pp. 38-45. DOI: hrgg:.oog///00 2020. emnlp-
10. 15.1663vv1/2020 ett.:/20000mmmmmpppmmmmmmmmmm dem-s.eeoss..oooo e
demos 6
6 (visited on 01/22/2022).
[45] Mengdi Xu et al. Prompting Decision Transformer for Few-Shot Policy Generalization.
arXiv:2206.1 134999 [es]. June 2022. DOI: 10 48550 arXiv 2206 13499. URL: http
7/arxiv. orgg abs abbs/200 /2206.206622 13499 (visited 08/09/2022).
[46] Susan Zhang et al. OPT: Open Pre-trained Transformer Language Models.
_eprint:
2205.01068. 2022.
12

---

Appendix
A Example prompts for each domain
assert state -- and state 1- 4 left()
Chain
assert revard 0
assert not done
(6, and state (4,
Distractor
state left()
assert revard
assert not done
assert state == c(1=2, ==1)
0(i-1, (-0)
Maze
state, reward ()ft()
assert revard
assert not done
assert paddle - ((,, 0)
C(0,
and paddle.x == 2 and ball.x m-
Minl Catch
paddle.x ball.x
and ball.y ==
revard paddle left() ball (escend())
assert reward -- assert not
assert ship C(2, 0)
and aliens [C(3, 5), c(1, 5)
and (ship.x,
aliena [0].
Mini Invaders
allena [1].x)
and ship.z < aliens [0] ix
ship.x
[1]
thip.left( assert revard for in aliens:
d. assert not
assert pos -- -3.45 and vel 0.00 and pos and vel pos, vel decel (pos, vel) assert reward -- 0 not
Point-Man Mass
Table 2: This table provides example prompts for each domain, showcasing the text format and hints. Hints are highlighted in grey.
13

---

B Tabular Q-Learning Hyperparameter Sweeps
Tabular Q learning rate
0.01
0.05
0.1
0.25 0.5 1
Chain
Distractor Chain
Maze
Mini Catch
Mini Invaders
0.8
regret
0.6
0.4
WV
0.2
0
200
400
0
6,000
0
6,000 0
6,000
6,000
step
step
step
step
step
This figure presents learning curves using the standard Tabular Q-Learning algorithm. Given an appropriately chosen learning rate, this algorithm does converge, but this can take several thousand timesteps on all but the simplest domains.
Tabular exploration bonus
0.01 0.1
Chain
Distractor Chain
Maze
Mini Catch
Mini Invaders
0.8
0.6
0.4
0.2
200
400
6,000
6,000
6,000
6,000
step
step
step
step
step
This figure presents learning curves for Tabular Q-Learning with count-based exploration bonus. Actions chosen using the following formulation: arg max,' ((s, a') + k/(1 (s, where & the exploration bonus hyperparameter, assuming values {0, 0.01, 0.1, 1} in the graph, and [ss, d') is the visitation count for the state-action d').
C Proximal Policy Optimization Hyperparameter Sweeps
Chain
Distractor-chain
Maze
Mean return per episode
Mean return episode
1.0
Mean return per episode
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.2
0.0
0.0
0.0
Mini-catch
Point-mass
Space-invaders
Mean return per episode
Mean return episode
1.0
Mean return per episode
0.8
1.5
0.6
0.6
1.0
0.4
0.4
0.5
0.2
40,000
80,000
40,000
80.000
40,000
80.000
Steps
Steps
Steps
14

---

This figure presents learning curves for Proximal Policy Optimization (PPO) (Schulman et al. 2017).
deep RL algorithm trained using gradient descent. These runs search the following hyperparameters:
Number of Hidden Layers
1.2
Hidden Size
256, 512, 1024
Actor Learning Rate
0.001, 0.002, 0.005
Critic Learning Rate
0.0001, 0.0005, 0.001
As the graphs indicate, PPO takes several thousand steps to converge, even on simple domains.
D Random Prediction Ablation
This ablation studies the contribution that each prediction type makes to ICPI's ability to learn. To this end, for prediction type observation, termination, action, and reward that substitutes a random, valid prediction for the prediction made by the LLM. For example, for actions, we first sample random action from the action space of the environment and then apply the standard string formatting used in other parts of the algorithm. As figure indicates, on Chain, the simplest of our environments, none of the ablations are able learn.
ablation
Chain
Randomizes
None
0.6
Observation
0.4
Termination
0.2
Action
Reward
0
20
40
60
80
step
Figure 5: This figure presents learning curves for the random prediction ablation described above the Chain domain. The Y-axis represents regret and the X-axis represents timesteps in the environment. Each colored line refers to different prediction which was randomized. The red line is the original ICPI learning curve, included for reference.
E
Error Analysis
In this section we present examples of errors made by ICPI on each domain. each subsection, we will examine rollout generated by the Codex LLM that causes the agent to choose suboptimal behavior in the actual environment.
Chain
Here, we analyze an example which the rollouts incorrectly estimate equal values for all three actions. Each line in the code snippet represents actual output from the LLM, excluding comments which are added for clarity. The agent's state is 5 (recall that the goal state 4). For the left () action, the model generates the following rollout:
1- # the state left () a naw state in the result of the left () action
and
starting
aesert rexard --
assert state
now state state goal state
Here the rollout policy unexpectedly chooses nove right instesd of trying the goal.
atate right() done aesert rexard =-
assert state
Again the rollout policy outputs suboptinal action: reward sty_gosl(staate
assert done # try-goal alvays terninates an episode with revard
15

---

Note that the model outputs the action right () when the agent is in goal state 4, and then outputs action try-goal(). () As a result the rollout estimates value 0 for an action which has value ry under the optimal policy (one timestep away from a terminal state with reward 1).
The estimate for action try-goal () is accurate in state 5: atsert state -- 5 and state !- 4 * the current, starting state cry-goal(statt) asaert done aesert reward
However the rollout also underestimates the value of action right ():
sesert state and state !- * the current, starting state
state right ()
done
assert rezard ==
aesert state
reuard try-goal (atata) laft() sould be the optinal policy, not try-goal ().
assert done
rexard --
Distracter Chain
In this example the agent starts in state (2, 3). Recall that in this environment, the second digit in the tuple chosen randomly and does affect environment transitions or reward.
In the first of the three rollouts, the first action is left(). Since the goal is state (4, *), this action is already suboptimal. However, we that the agent continues to behave sub-optimally throughout the rest of the rollout by continuously choosing action left( () instead of the optimal action right O) This occurs because at this stage of training, the agent has mostly taken left () actions and result, the rollout policy prompt mostly contains left () actions.
assert state [2, state left()
[4, 3]
assert not dene
rexard --
assert state [1, 2] and state t= [4, 2] optinal action is r
left() ()
assert not done
assert rexard
state [0, 0] and state 1- [4, 0]
state left () The optimal action is right()
done
assert rezard ==
state = laft () # The optimal action is right()
assert not done
rexard --
assert state [0, and state [4, 0]
laft ()
The optinal action is right()
assert not done
aesert
assert state -
[0, o) and state t- [4, 0)
state left () The optinal action is right()
not dene
assert revard
[o,
[4,
state - () # completion
The rollout for the try-goall action is correct:
aesert state
revard try_gosl (state) assert assert rexard == completion
Here the initial action is optimal: right(). (). However, we once again see that the rollout policy is suboptimal because favors the left () action over the optimal right () action.
assert state == [2, 3] and state t= [4, 3]
state right not done assert rexard state
7]
16

---

state left()
assert not done
assert state -- 12, 4] and state !- [4, 4]
state
sesert not done
assert rexard
[1, and state [4, 7]
state laft ()
asgert reward ==
asaert state [o, and state [4, o]
state left ()
assert not done
rexard
assert state [o, 0] and state 1= [4, 6]
state left ()
asgert not done assert reward
[0, 0] and state [4, o]
state ()
This causes the model to inaccurately produce value estimate of for the right action.
Maze
Similar to the previous example, we study an example for maze early in training, when the rollout policy is still quite suboptimal. As result, the rollout policy does not discover the reward, causing the rollouts to estimate equal value for each of the four actions.
The rollouts all start at state (1, 1).
aesert state C(i-1, j-1) and state C(1-1, 5-0)
rexard left()
assert not done
aesert state -- C(i-1, j-0) and state 1- c(i-1, j-0)
state, rezard right() ()
assert not done
assert rexard
C(i-1, j-1) and state C(i-1, j-0)
state, resard (p
aesert rexard --
assert state C(1-0, j=1) and state C(1=1, (=0)
state, reward dovn
assert not dene
rexard ==
assert atate d(1=1, j=1) and state t= c(1=t, ==0)
state,
up
assert not done
assert reward
C(i-0, j-1) and state C(i-1, 5-0)
state, rexard davn()
done
assert revard ==
aesert state C(i-1, j-1) state C(i-1, j-0)
state, reward ()
assert state -- C(i-1, j-1) and state 1- C(i-1, j-0)
state, rexard doun
aesert not done
assert rexard
assert state C(i-2, j-1) state C(1-1, J-0)
state, rexard doun ()
assert
assert rexard ==
assert state C(1-2, j-1) and state c(1-1, (-0)
state, rexard left()
assert not dene
assert state -- C(i-2, j-0) and state 1- C(i-1, j-0)
state, rexard right()
assert not done
assert rexard
C(i-2, j-1) and state C(i-1, 5-0) state, rexard right( ()
17

---

rexard --
assert state o(1-2, j-2) and state !- c(1=1, )=0)
right()
assert not done assert
aesert state C(i-2, j-2) and state C(1-1, J-03
state, rexard right() ()
asaert atate c(i=1, j=1) and state 1= C(1=1, )=0)
state, rexard up
assert resard --
assert state C(i-0, j-1) state C(i-1, 5-0)
state, reward doun ()
assert dome
assert reward --
assert state C(i-1, j-1) and state C(i-1, j-0)
up
assert not done
assert
assert state - C(i-0, j-1) and state 1- C(im1, jm0)
state, rexard right()
sesert not done
assert rexard
state C(i-0, j-2)
C(1-1, J-0)
state, rexard right ()
assert
asgert reward ==
assert state C(1-0, j-2) and state C(1-1, J-0)
state, reward right() ()
assert not dene
aesert
assert state -- C(i-0, j-2) and state 1- C(i-1, j-0)
state,
right ()
assert state -- C(i-1, j-1) and state 1- C(i-1, j-0)
state, rexard right ()
assert not done
assert rexard
C(i-1, j-2) and state C(i-1, )-0)
state, resard right ()
assert rexard --
assert state C(i-1, j-2) state C(i-1, j-0)
state, regard doun
assert not dene
rexard
assert atate 0(i-2, j-2) and state t- c(1-1, (-0)
state,
right ()
assert not done
assert
assert atate =s O(1=2, j=2) and state t= c(1=1, |=0)
state, resard right() ()
assert resard --
assert state C(i-2, j-2) state C(i-1, j-0)
state, reward right() ()
assert done
assert revard --
assert state 0(1-2, 2-2) and state C(1-1, --0)
right()
18


Reviews:
[{"review_id": "Gk0VM8r22x", "reviewer": "Reviewer_6gCR", "summary": "The problem of using large-pretrained foundation models (LLMs) like codex as backbones for learning RL policies without gradient updates or expert demonstrations is addressed by the authors. They propose an algorithm called In-Context Policy Iteration (ICPI) to perform model-based RL policy iterations using a pretrained LLM to predict state/rewards as well as the next-action. \n\nTo make the environment prompt-ready, the task is translated into a text-based description. The authors focus on code-pre-trained backbones and consider translating to Python code in their experiments. They demonstrate that ICPI can learn to minimize regret over time in simple tabular environments such as maze and point-mass.", "questions": "1. The success of policy-iteration using replay-buffer enhanced prompting is likely influenced by the information content of the demonstration set $\\mathcal{D}$. It's worth noting that these trajectories may not be on-policy and are sourced from previous policy versions. It's important to investigate how the algorithm responds to the quality of the replay-buffer, including factors like the number of trajectories and whether failed examples are necessary.", "limitations": "The authors have acknowledged certain limitations of their work and discussed them thoroughly. One such limitation is that the ICPI method requires exhaustive unrolling of the action sequence for all actions, making it suitable only for domains with finite action space. Furthermore, it appears that only code-pre-trained models are capable of reliably performing ICPI, indicating that translating the MDP to an appropriate code format is crucial. Additionally, the method often relies on additional hints to ground the reward function/terminal states, hence a more comprehensive discussion on potential ways to mitigate this would be beneficial.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Some strengths of the work are outlined below:\n1. The proposed algorithm utilizes In-Context learning to implement policy-iteration with LLMs. This parameter-free approach has the potential to enhance the task-specific behavior of an LLM agent by designing a prompt-ready text-based interface for any MDP.\n2. The authors present a clear explanation of the algorithm for ICPI, including the necessary design choices. They also provide empirical evidence of policy improvement over time in six MDPs that are both simple and tabular.", "weaknesses": "1. Based on the results, it appears that the algorithm works efficiently when combined with code-pre-training. However, relying solely on text-based MDP descriptions may not yield consistent outcomes. It is uncertain whether this is due to the limitations in reasoning and in-context learning capabilities of current LLMs, or the difficulty in creating effective intermediate representations of the environment.\n2. To perform Q-value computation, the algorithm needs to unroll action sequences for all actions. Unfortunately, this greatly restricts the algorithm's usefulness to finite action-spaces.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> To perform Q-value computation, the algorithm needs to unroll action sequences for all actions. Unfortunately, this greatly restricts the algorithm's usefulness to finite action-spaces.\n\nYou are correct that we only demonstrated the method using discrete action domains. However we suggest two considerations. First, many existing state-of-the-art methods, like Dreamer-v2 or Muzero work primarily (or exclusively) on discrete-action domains, some of which (like Go or Minecraft) can be quite complex. Second, a variety of publications have demonstrated that continuous action domains can often be addressed through discretization (e.g. [Trajectory Transformer](https://arxiv.org/abs/2106.02039) and [RT-1: Robotics Transformer for Real-World Control at Scale](https://arxiv.org/abs/2212.06817)).\n\n> The success of policy-iteration using replay-buffer enhanced prompting is likely influenced by the information content of the demonstration set. It's worth noting that these trajectories may not be on-policy and are sourced from previous policy versions. It's important to investigate how the algorithm responds to the quality of the replay-buffer, including factors like the number of trajectories and whether failed examples are necessary.\n\nIt is true that the trajectories may not be strictly on-policy. First of all, we can provide no strict guarantees on the output of a pre-trained LLM. Second, some elements of the policy prompt are drawn from slightly older trajectories, which may be generated by an older policy. To mitigate this, we only draw the policy prompt from the c=8 most _recent trajectories_. In our “c=16” baseline, we investigate the result of extending the recency cutoff and allowing for trajectories generated by older policies. As Figure 3 indicates, the algorithm is still able to learn, though this modification hurts performance. As for failed examples, they are necessary for the world model component of the algorithm, but not the policy component, which in general benefits from the best demonstrations available.\n\n> Based on the results, it appears that the algorithm works efficiently when combined with code-pre-training. However, relying solely on text-based MDP descriptions may not yield consistent outcomes. It is uncertain whether this is due to the limitations in reasoning and in-context learning capabilities of current LLMs, or the difficulty in creating effective intermediate representations of the environment.\n\nPerhaps you  can offer some clarification here. It sounds like you are asking about whether any remaining sub-optimality is attributable to the LLM itself or to the text format that we chose. Is this correct?\n"}, {"review_id": "QI34lHTizr", "reviewer": "Reviewer_Yb53", "summary": "The paper demonstrates a way to implement policy iteration by means of in-context learning with a large language model. Authors suggest ICPI, a reinforcement learning algorithm that performs policy evaluation and policy improvement solely by encoding tokens and inferencing the model, without updating model parameters or adapters. They proceed by evaluating ICPI on a collection of simple RL problems, where each problem was manually converted to an LLM-digestible text representation. Authors show that certain ICPI configurations can achieve comparable or smaller regret than traditional RL algorithms, emphasizing the first few hundred steps. Finally, the paper analyzes the learned behavior produced by ICPI in the tested environments.", "questions": "\n\n> L185 - since our primary results use codex language model, we use python code to represent these values\n\nMinor question: was it an educated guess or does python outperform some alternative languages when used in this capacity?\n\n> six domains and their associated prompt formats\n\nMinor question: how robust is ICPI performance to the choice of prompts?\n\n\n> L21: improve … as information accumulated\n\nPossibly missing a verb, e.g. as information is accumulated / accumulates\n\n\n\n> Fig 4 caption: Error bars are standard errors from four seeds\n\nAssuming you use standard deviation, you may want to specify whether or not this is an adjusted estimate (i.e. whether or not you use Bessel’s correction). If you deliberately opt not to specify this, please feel free to ignore the question.\n", "limitations": "Authors properly address most of the main limitations of this work, as well as it's societal limitations.\n\nOne potentially missing limitation is the reproducibility of the main experiments. As it stands, ICPI relies on proprietary language models that can change without warning. To the best of my understanding, the current OpenAI policy allows them to alter the model behavior without warning users in advance, e.g. to patch a potential misuse. This could inadvertently affect the performance of ICPI, making it difficult to reproduce the main results in several years. While there is no guaranteed way to fix this issue, I would argue that it is worth acknowledging it.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The main strength of the paper is that it provides a new way to think about policy iteration from an LLM perspective. While ICPI follows the general PI framework, the way it does so certainly \"feels different\" than traditional (e.g. tabular) PI. Applying similar ideas to other RL areas  may inspire interesting future research.\n\nWhile the evaluation of ICPI is rather minimalistic, these experiments clearly showcase how the algorithm performs and explain the reported results in due detail.\n\nFinally, the paper is generally well-written and easy to follow. The presentation quality is high, both for text and the accompanying visualizations.", "weaknesses": "\n### 1. Other methods have drawbacks... so what about them?\n\nIn the abstract / intro, authors review existing LLM applications in RL and broadly group them in two categories: those based on expert demosntrations, and those that use gradient-based updates. As a motivation, they state that each of these methods have drawbacks, and the drawback of gradient-based methods is that they are slow (e.g. L9).\n\nTo the best of my knowledge, there is nothing in the paper that evaluates if ICPI is \"faster\" than equivalently tuned gradient-based methods. \nFurthermore, I have a hunch that ICPI scales worse than linearly with the number of demonstrations, i.e. 10000th episode is processed slower than the first one, due to the nature of Transformer attention.\n\nMy concern is that, since authors use speed as one of the motivating factors, this claim should be addressed somewhere in the paper. If ICPI is in fact slower than gradient-based methods, this should also be stated as a limitation. Please note that being slower does not discredit this work in any way -- but not exploring speed after using it as motivation does.\n\n### 2. Comparison to other in-context reinforcement learning methods\n\nThere are at least several prior works that also fall in the category of \"reinforcement learning in-context without updating model parameters\".\nOne prior work that fits this definition is [1], where the LLM-based agent \"reflects\" on it's performance in natural language, which is shown to improve performance. Another, similar work [2] tasks the large language model with providing feedback on model performance in natural language, which also leads to improved behavior. It is likely that there are more prior & concurrent papers that are also relevant.\n\n[1] https://arxiv.org/abs/2303.11366\n[2] https://arxiv.org/abs/2303.17651\n\nOn a surface level, it seems that the above two papers take a less formalistic stance on policy improvement, relying on instruction-following LLMs to do most of their work. However, they have similar requirements and solve similar problems. Both papers use text-based RL environments of similar complexity to the ones used in this submission.\n\nSince both these works were available for several months before submission, I would argue that this paper would benefit from comparing against them: both in terms of theoretical properties and in empirical regret on the same problem set.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> To the best of my knowledge, there is nothing in the paper that evaluates if ICPI is \"faster\" than equivalently tuned gradient-based methods. \n\nThank you for pointing out this omission. [This link](https://www.overleaf.com/read/ncjhbrxkvyrz) provides a demonstration of the performance of Proximal Policy Optimization (PPO) on the 5 domains in this paper. PPO is probably not the most sample-efficient RL algorithm, but as the results indicate, most runs do not converge in less than 20,000 time-steps. In contrast, ICPI converges in fewer than 250 timesteps on all 5 domains. We believe it is unlikely that another traditional deep RL method would diminish this number by several orders of magnitude. There are gradient-based metalearning methods, like RL^2 or MAML, which demonstrate learning on very small time scales, but these require a multitask domain, so it is unclear how to adapt them to our setting. Please let us know if there are other experiments we can run to address this concern.\n\n> Furthermore, I have a hunch that ICPI scales worse than linearly with the number of demonstrations, i.e. 10000th episode is processed slower than the first one, due to the nature of Transformer attention.\n\nIn general, we do not believe that the algorithm will scale with the number of episodes. The time-complexity of the algorithm is linear in a) the number of actions b) the expected length of an episode c) the size (especially the context window and layer count) of the foundation model. It is true that better policies may generate longer episodes for some environments, but episode length does not usually scale linearly with the optimality of the policy.\n\n> My concern is that, since authors use speed as one of the motivating factors, this claim should be addressed somewhere in the paper. If ICPI is in fact slower than gradient-based methods, this should also be stated as a limitation.\n\nAgain thank you for pointing this out. See our earlier comment and [results with PPO](https://www.overleaf.com/read/ncjhbrxkvyrz). Please let us know if other experimental results would better address this concern. \n\n> Minor question: was it an educated guess or does python outperform some alternative languages when used in this capacity?\n\nWe did not have the opportunity to compare against other languages due to the high cost of running these experiments. Our expectation would be that the LLM's performance would roughly scale with the prevalence of a given language in its dataset. We note that the Codex paper focuses its evaluations almost exclusively on python.\n\n> Minor question: how robust is ICPI performance to the choice of prompts?\n\nIn our ablation experiments, we do demonstrate several variations on the prompt selection mechanism proposed in the paper. Most demonstrate reasonable performance while underperforming the proposed method. Perhaps you are specifically interested in the wording of the prompt. Our experience during experimentation was that the language model is indifferent to details of punctuation or syntax, but sensitive to the semantics of the information present in the prompt, as demonstrated especially by our “no hints” ablation.\n\n> Possibly missing a verb, e.g. as information is accumulated / accumulates\n\nThank you for the correction. We will fix this in the final version.\n\n> Assuming you use standard deviation, you may want to specify whether or not this is an adjusted estimate (i.e. whether or not you use Bessel’s correction). If you deliberately opt not to specify this, please feel free to ignore the question.\n\n Our plots use the Pandas [“rolling.sem()” method](https://pandas.pydata.org/docs/reference/api/pandas.core.window.rolling.Rolling.sem.html) with a rolling window of 30.\n\n> One potentially missing limitation is the reproducibility of the main experiments. As it stands, ICPI relies on proprietary language models that can change without warning. To the best of my understanding, the current OpenAI policy allows them to alter the model behavior without warning users in advance, e.g. to patch a potential misuse.\n\nThis is a good point and one that we will include in the limitations section. It is worth noting that, since the submission of this paper, all newer models (the GPT-4 family) have only improved on the capabilities of the earlier codex model that we were using.\n"}, {"review_id": "EczZfPbC2f", "reviewer": "Reviewer_RKry", "summary": "The authors show how LMs can solve simple MDPs by implementing policy iteration in-context. They do that by constructing prompts that simulate policy rollouts using partial trajectories from an experience buffer as few-shot examples. They empirically validate their approach on six toy MDPs that test for specific failure modes.", "questions": "1. What roles does the choice of LM sampling temperature=0.1 play in the setup? Was temperature=1 causing too much hallucination when generating rollouts? Does the theoretical convergence of policy iteration still hold when Q values are estimated from a  biased (due to temperature) distribution of rollouts?\n2. What discount factor $\\gamma$ is used in experiments? Is it 0.8, the same as for the optimal policy relative to which regret is computed?", "limitations": "Limitations and societal impacts are discussed. One class of risks that could be discussed is those associated with combining powerful world models with optimisation pressure for agentic behaviour (e.g., Ngo et al., 2023).\n\nNgo et al., 2023. The alignment problem from a deep learning perspective", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. I find the result from the paper title impressive. It’s interesting from at least four different angles: (i) understanding the capabilities of LMs as general-purpose simulators, (ii) sketching the design space of hybrid systems combining RL with LMs, (iii) exploring prompting strategies for representing and rolling-out MDPs in token space, (iv) understanding the trade-offs between in-context learning and standard in-weights learning. The paper is a very original and solid contribution to all four.\n2. The paper is well-written and easy to follow. \n3. The experiments are pretty extensive, covering different MDPs (carefully designed to test specific failure modes), well-motivated ablations and different model families and sizes.", "weaknesses": "The weaknesses listed below are relatively minor and are greatly outweighed by strengths. The authors shouldn’t feel obliged to respond to them in detail.\n\n1. The authors attribute the relatively good performance of ICPI its “ability to generalize from its context to unseen states and state/action pairs (unlike “Tabular Q” and “Matching Model”)” (lines 296-298). How is that different from function approximation in more traditional RL? It could've been good to also compare ICPI with some baseline using function approximation; both a variant with a small MLP and with a linear head on top of an LLM could be interesting.\n2. The claim in the abstract that “algorithms that rely on [demonstrations] do not outperform the experts from which the demonstrations were derived” is too strong or too vague. It is possible in principle to generalise to rewards higher than those seen in the dataset and the DecisionTransformer paper demonstrated that in practice (in some cases; Fig. 4 in Chen et al.). \n3. The Related work section is very comprehensive but also somewhat repetitive. If the authors’ are short of space, I think it could be shortened without much harm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The authors attribute the relatively good performance of ICPI its “ability to generalize from its context to unseen states and state/action pairs (unlike “Tabular Q” and “Matching Model”)” (lines 296-298). How is that different from function approximation in more traditional RL?\n\nYes, this is a good question. Both our method and traditional RL function approximation methods take advantage of the generalization properties of deep neural networks. The primary difference is the data that these networks were trained on. For most RL methods, the policy and value networks are specialized to the environment or domain of interest, while our method relies on foundation models which are trained on a much larger and more general dataset. It is reasonable to expect that a traditional RL method will eventually outperform our method due to specialization, but it requires retraining on each new domain (while ICPI does not, since it can rely on the underlying foundation model for generalization).\n\n> Compare ICPI with some baseline using function approximation\n\nThank you for pointing this out. This is clearly an omission in the paper. [This link](https://www.overleaf.com/read/ncjhbrxkvyrz) provides a demonstration of the performance of Proximal Policy Optimization (PPO) on the 5 domains in this paper. PPO is probably not the most sample-efficient RL algorithm, but as the results indicate, most runs do not converge in less than 20,000 time-steps. In contrast, ICPI converges in fewer than 250 timesteps. It is unlikely that another traditional deep RL method would significantly diminish this number. There are metalearning methods, like RL^2 or MAML, which use function approximation and which can few-shot-learn a domain, but these require a multitask setting, so it is unclear how to adapt them to our setting. Please let us know if there are other experiments we can run to address this concern.\n\n> The claim in the abstract that “algorithms that rely on [demonstrations] do not outperform the experts from which the demonstrations were derived” is too strong or too vague.\n\nThis is valid. Perhaps we can state that “algorithms that rely exclusively on imitating demonstrations are inherently limited by the performance of the source expert.” In contrast, our method learns from new data generated by a continuously improving policy. We also draw attention to the fact that while Decision Transformer does outperform vanilla Behavior Cloning, it actually underperforms “Behavior Cloning 10%,” in which behavior cloning uses only the top 10% of trajectories. That can be thought of as similar to our “No ArgMax” which only includes trajectories that exceed a set performance threshold.\n\n> The Related work section is very comprehensive but also somewhat repetitive. If the authors’ are short of space, I think it could be shortened without much harm.\n\nThank you for the suggestion. Several reviewers commented on the related works and we intend to rewrite it.\n\n> What roles does the choice of LM sampling temperature=0.1 play in the setup? Was temperature=1 causing too much hallucination when generating rollouts? Does the theoretical convergence of policy iteration still hold when Q values are estimated from a biased (due to temperature) distribution of rollouts?\n\nEven with temperature=0, the sampling of Q estimates is still stochastic because the prompts are sampled randomly and shuffled. Our analysis relies on the observation that, given a prompt containing transitions sampled from a mixture of policies and ordered randomly, the optimal prediction for a language model will be a mixture of these policies. \n\n> What discount factor is used in experiments? Is it 0.8, the same as for the optimal policy relative to which regret is computed?\n\nYes to both questions.\n\n"}, {"review_id": "h616ObRQBj", "reviewer": "Reviewer_WHe7", "summary": "This paper formalizes a framework where language model can use its in-context learning capability to predict actions and Q-values, and the training occurs by updating the prompt with the latest history. As a result, the LM can perform reinforcement learning without gradient. The paper explored a few language models and a few different toy tasks and found that Codex can learn successfully in this environment that it has no prior knowledge. ", "questions": "N/A", "limitations": "The author properly listed their limitations.", "rating": 5, "confidence": 1, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "- The paper formalizes a framework where LLM can perform RL using its in-context learning capability. Such a framework might allow future works to connect LLM research to existing RL concepts to further improvement performance.\n- The empirical results look promising.", "weaknesses": "I'm not an expert in RL, so I am not confident in my judgement.\n\n- While the formalization seems nice, it is not particularly novel; the empirical results/datasets presented also seem toy and low-hanging fruits. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> While the formalization seems nice, it is not particularly novel; the empirical results/datasets presented also seem toy and low-hanging fruits.\n\nThe Policy Iteration algorithm itself is, of course, many decades old. What is novel is the in-context implementation of policy iteration using LLMs. Because of limited access to LLMs because of the associated cost we could only illustrate our algorithmic ideas on simple domains. However, our algorithm is applicable to more complex domains without modification and furthermore we believe that as LLMs become more and more widely available and there is further improvement in their quality and generalization abilities our method will scale and generalize accordingly. "}, {"review_id": "2xgKTTeF9N", "reviewer": "Reviewer_EN6r", "summary": "This paper examines the possibility of taking advantage of the ability of in-context learning to use large language models to implement policy iteration so that large language models can be used to solve reinforcement learning problems without querying the gradients. Experiments have been conducted on 5 artificial environments showing that the proposed method ICPI can indeed achieve decent performance across all considered environments. ", "questions": "Please see weakness.", "limitations": "The limitation that current large language models lack the ability to plan in complex domains accurately is discussed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Given the success of in-context learning of large language models, applying in-context learning to policy iteration is a well-motivated idea. The proposed ICPI is sufficiently different from the original policy iteration and the other recent attempts to combine the success of large language models with reinforcement learning techniques. In the experiments, 5 artificial environments are proposed and ICPI achieves better results than baselines in all environments. Detailed ablations are also provided to justify the importance of each component of ICPI.\n", "weaknesses": "The applications of ICPI are limited. Right now it can only be applied to handcrafted small environments like chain and mini catch. It is unclear if large language models can do complex planning in more practical tasks such as Atari and mujoco tasks. Additionally, the current formulation of ICPI can only handle discrete state and action space. Still, it would be interesting to see if large language models can be used to handle realistic continuous-action robotics tasks. It also seems that a curated hint is required for more complicated tasks like mini invaders and point mass.\n\nComparisons to other more recent methods are missing. To justify the advantage of ICPI, it would be interesting to see how ICPI compares to popular deep RL baselines (either state-based or text-based) such as PPO [1], SAC [2], and CQL [3]/IQL [4] since some recent works [5, 6] have shown that they can be used for large language models too. \n\nSome of the arguments are not supported. For example, in line 8-10, \"All gradient techniques are inherently slow, sacrificing the 'few-shot' quality that makes in-context learning attractive to begin with\" is not supported. ICPI also involves inferencing the entire trajectory to calculate the q function for each action, so it seems unclear whether it is faster than gradient-based techniques. A comparison of ICPI and gradient-based methods in terms of computation time is needed.\n\nThe writing can be improved. The related works and methodology sections can be more concise. There are some references typos and the paragraph from 160 - 164 is repeated.\n\n[1] Proximal Policy Optimization Algorithms, https://arxiv.org/abs/1707.06347\n\n[2] Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor, https://arxiv.org/abs/1801.01290\n\n[3] Conservative Q-Learning for Offline Reinforcement Learning, https://arxiv.org/abs/2006.04779\n\n[4] Offline Reinforcement Learning with Implicit Q-Learning, https://arxiv.org/abs/2110.06169\n\n[5] Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization, https://arxiv.org/abs/2210.01241 \n\n[6] Offline RL for Natural Language Generation with Implicit Language Q Learning, https://arxiv.org/abs/2206.11871", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The applications of ICPI are limited. Right now it can only be applied to handcrafted small environments like chain and mini catch. It is unclear if large language models can do complex planning in more practical tasks such as Atari and mujoco tasks. Additionally, the current formulation of ICPI can only handle discrete state and action space. Still, it would be interesting to see if large language models can be used to handle realistic continuous-action robotics tasks.\n\nThis is an important question. Some recent work has demonstrated promise in applying large language models to complex tasks. For example, [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://arxiv.org/abs/2305.16291) uses prompt engineering and hand-crafted state representations to elicit state-of-the-art policies from ChatGPT in the Minecraft domain, which is not continuous-action but still quite complex. A paper that was released very recently, [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://arxiv.org/pdf/2307.15818.pdf), actually trains a video-language model to control a robot (with continuous actions) using text outputs.\n\n> Comparisons to other more recent methods are missing. To justify the advantage of ICPI, it would be interesting to see how ICPI compares to popular deep RL baselines (either state-based or text-based) such as PPO [1], SAC [2], and CQL [3]/IQL [4] since some recent works [5, 6] have shown that they can be used for large language models too.\n\nThank you for directing our attention to these works. 5 and 6 are particularly interesting and will be integrated into our related works section. As for deep RL baselines, [this link](https://www.overleaf.com/read/ncjhbrxkvyrz) provides a demonstration of the performance of Proximal Policy Optimization (PPO) on the 5 domains in this paper. As the results indicate, most runs do not converge in less than 20,000 time-steps. ICPI converges in fewer than 250 timesteps on all 5 domains. We are working on generating results with SAC, but it is worth noting that SAC is not typically implemented for discrete action domains. As for CQL and IQL, offline reinforcement learning requires access to an offline dataset, so it is unclear how best to apply these algorithms to our domains. Please let us know how you would suggest approaching this.\n\n> Some of the arguments are not supported. For example, in line 8-10, \"All gradient techniques are inherently slow, sacrificing the 'few-shot' quality that makes in-context learning attractive to begin with\" is not supported. ICPI also involves inferencing the entire trajectory to calculate the q function for each action, so it seems unclear whether it is faster than gradient-based techniques. A comparison of ICPI and gradient-based methods in terms of computation time is needed.\n\nAgain thank you for pointing this out. See our earlier comment and [results with PPO](https://www.overleaf.com/read/ncjhbrxkvyrz). Your point about computation time is well taken. Computing the ICPI results was quite slow because OpenAI places rate-limit restrictions on the Codex API. Given access to an unrestricted API, one would expect much faster results. Intuitively, one can expect faster performance from ICPI because it leverages knowledge already present in the LLM which the neural network must learn from scratch.\n\n> The writing can be improved. The related works and methodology sections can be more concise. There are some references typos and the paragraph from 160 - 164 is repeated.\n\nWe appreciate the feedback and will correct these suggestions in the final version."}]

Author rebuttal:

Meta-review: This paper studies the use of neural language models for policy learning, finding that LLMs can learn to implement policy iteration in-context rather than relying on fine-tuning. All reviewers rated the paper positively, with at least one particularly enthusiastic.

The authors have promised some changes to the discussion of related work, as well as new experimental results, which should be incorporated into the final version of the paper. If there's room, it would also be nice to include some discussion of reviewer RKry's question about tradeoffs between sample efficiency (rollouts) and computational efficiency (FLOPs).
Decision: Accept (poster)