In-Context Policy Iteration
Authors: Ethan Brooks
Logan A Walls
Richard Lewis
Satinder Singh
Venue: Submitted to ICLR 2023 (2023)
Keywords: Reinforcement Learning
In-Context Learning
Foundation Models

Abstract: This work presents In-Context Policy Iteration, an algorithm for performing Reinforcement Learning (RL), in-context, using foundation models. While the application of foundation models to RL has received considerable attention, most approaches rely on either (1) the curation of expert demonstrations (either through manual design or task-specific pretraining) or (2) adaptation to the task of interest using gradient methods (either fine-tuning or training of adapter layers). Both of these techniques have drawbacks. Collecting demonstrations is labor-intensive, and algorithms that rely on them do not outperform the experts from which the demonstrations were derived. All gradient techniques are inherently slow, sacrificing the “few-shot” quality that made in-context learning attractive to begin with. In this work, we present an algorithm, ICPI, that learns to perform RL tasks without expert demonstrations or gradients. Instead we present a policy-iteration method in which the prompt content is the entire locus of learning. ICPI iteratively updates the contents of the prompt from which it derives its policy through trial-and-error interaction with an RL environment. In order to eliminate the role of in-weights learning (on which approaches like Decision Transformer rely heavily), we demonstrate our algorithm using Codex Chen et al. (2021b), a language model with no prior knowledge of the domains on which we evaluate it.

IN-CONTEXT POLICY ITERATION
Ethan Brooks', Logan Walls?, Richard L. Lewis,, Satinder Singh' ! 'Computer Science and Engineering, University of Michigan 2Department of Psychology. University of Michigan
Sethanbro,,ogwalllle rickl, back.bbaveja@eumicchcc edu
ABSTRACT
This work presents In-Conters Policy Iteration, algorithm for performing Re- inforcement Learning (RL), in-context, using foundation models. While the ap- plication of foundation models to RL has received considerable attention, most approaches rely on either (1) the curation of expert demonstrations (either through manual design or task-specific pretraining) or (2) adaptation to the task of interest using gradient methods (either fine-tuning or training of adapter layers). Both of these techniques have drawbacks. Collecting demonstrations is labor-intensive, and algorithms that rely on them do not outperform the experts from which the demonstrations were derived. All gradient techniques are inherently slow, sacrific- ing the "few-shot" quality that made in-context learning attractive to begin with. In this work, we present an algorithm, ICPI, that learns to perform RL tasks without expert demonstrations or gradients. Instead we present which the prompt content is the entire locus of learning. ICPI iteratively updates the contents of the prompt from which it derives its policy through trial-and-ere interaction with an RL environment. In order to eliminate the role of in-weights learning (on which approaches like Decision Transformer rely heavily), we demon- strate our algorithm using Codex (Chen et al. 20211), a language model with no prior knowledge of the domains which evaluate it.
volicy-iteration method in
INTRODUCTION
In-context learning describes the ability of sequence-prediction models to generalize to novel down-
stream tasks when prompted with a small number of exemplars (Lu et al. 2021 Brown et al. (020)). The introduction of the Transformer architecture (Vaswani et al. 2017) has significantly increased in- terest in this phenomenon, since this architecture demonstrates much stronger generalization capacity than its predecessors (Chan et al.) 2022). Another interesting property of in-context learning in the case of large pre-trained models (or "Foundation models"") is that the models are not directly trained to optimize a meta-learning objective, but demonstrate an emergent capacity to generalize (or at least specialize) diverse downstream task-distributions (Brown al. 2020
Wei al. 2022a).
A litany of existing work has explored methods for applying this remarkable capability to downstream tasks Reeated Work), including Reinforcement Learning (RL). Most work in this area either (1)
assumes access to expert demonstrations - collected either from human experts (Huang et al. 2022a
Baker et al.
2022). or domain domain-specific pre-trained RL agents (Chen et al.) 2021a] Lee al. [0022
Janner et al.
2021 Reed et al. [022] Xu et al. (022).- or (2) relies on gradient- based methods
e.g.
fine-tuning of the foundation models parameters as a whole (Lee et al. 2022 Reed et al. 2022
Baker et al.) 2022) or newly training an adapter layer or prefix vectors while keeping the original
foundation models frozen (Li & Liang] 2021: Singh et al. 2022 Karimi Mahabadi et al. 2022).
Our work presents algorithm, In-Context Policy Iteration (ICPI) which relaxes these assumptions. ICPI is a form of policy iteration in which the prompt content the locus of learning. Because method operates the prompt itself rather than the model parameters, gradient methods. Furthermore, the use of policy iteration frees us from expert demonstrations because suboptimal prompts can be improved over the course of training.
able to avoid
We illustrate the algorithm empirically on six small illustrative RL tasks- chain, distractor-chain, maze, mini-catch, mini--nvaders, and point-mass which the algorithm very quickly finds good

---

task For eaca oction in (Aii)....AAii)
P
P
LLM
LLM
LLM
Q(1i. A(1)) 7"'
3
3
LLM
LLM
LLM
(1..A(x)) = ~
Environment
the LLM generates rollout by alternately predicting transitions and selecting actions. Q-value estimates are discounted sums of rewards. The action chosen greedily with respect to Q-values. Both state/reward prediction and next action selection trajectories from D to create prompts for the LLM. Changes the content of D change the prompts that the LLM receives, allowing the model to improve its behavior over time.
Figure For each possible action
policies. We also compare five pretrained Large Language Models (LLMs), including two different size models trained on natural l -РР---В and GPT-J- and three different sizes of a model trained on program code-twe sizes of Codex as well InCoder. On our six domains, we find that only the largest model (the code-davinci-000 variant of Codex) consistently demonstrates learning.
RELATED WORK
A common application of foundation models to RL involves tasks that have language input, for example natural language instructions/goals (Garg et al. 2022a Hill et al. 2020) or text-based games (Peng et al. 202 Singh al. 2021 Majumdar et al. 2020 Ammanabrolu & Riedl 2021). Another approach encodes RL trajectories into token sequences, and processes them with a foundation model, model representations input deep RL architectures (Li al., 2022 Tarasov al. 2022 Tam et al.] 2022). Finally, a recent set of approaches treat RL. as sequence modeling problem and use the foundation models itself to predict states or actions. This section will focus this last category.
2.1 LEARNING FROM DEMONSTRATIONS
Many recent sequence- based approaches to reinforcement learning use demonstrations that come cither from human experts or pretrained RL agents. For example, Huang et al.) (2022a) use a frozen LLM planner for everyday household tasks by constructing prefix from sluman-generalie instructions, and then using the LLM to generate instructions for new tasks. This work is extended by Huang al. 2022bb. Similarly. Ahh al. 2022) use value function that is trained on human demonstrations to rank candidate actions produced by an LLM. Baker et al. (2022) use human demonstrations train the foundation model itself: they use video recordings of human Minecraft players to train foundation models that plays Minecraft. Works that rely on pretrained RL agents include Janner et al, (202I) who train "Trajectory Transformer" to predict trajectory sequences in continuous control tasks by using trajectories generated by pretrained agents, and Chen et al. (202 who use dataset of offline trajectories train "Decision Transformer" that predicts actions from iate-actionnrewart sequences in RL environments like Atari. Two approaches build on this method to improve generalization: Lee et al. (002) use trajectories generated by DQN agent to train a single Decision Transformer that can play many Atari games, and Xu et al. (2022) use combination of human and artificial trajectories to train a Decision Transformer that achieves few-shot generalization on continuous control tasks. Reed et al. (2022) take task-genen lusk-generality step farther and use datasets generated by pretrained agents to trai multi-modal agent that performs a wide array of RL (e.g. Atari, continuous control) and non-RI (e.g. image captioning, chat) tasks.
2

---

Some of the above works include non-expert demonstrations well. Chen al. (2021a) a include experiments with trajectories generated by random (as opposed to expert) policies. Lee et al. (2022) and Xu et al. 2022) also use datasets that include trajectories generated by partially trained agents in addition to fully trained agents. Like these works, our proposed method (ICPI) does not rely on expert demonstrations- but note key differences between
approach and existing approaches.
Firstly, ICPI only consumes self-generate trajectories, so it does not require any demonstrations (like Chen al. (2021a) with random trajectories, but unlike Lee al. (2022) , Xu al. (2022). and the other approaches reviewed above). Secondly, ICPI relies primarily on in in-context learning rather than in- in-weights learning to achieve generalization (like Xu et al. (2022), but unlike Chen al. (202
Lee et al. (2022), For discussion about in-weights vs. in-context learning see Chan et al. (2022).
2.2
GRADIENT-BASED TRAINING & FINETUNING ON RL TASKS
Most approaches involve training or fine-tuning foundation models RL tasks. For example,JJnnner Janner ctt al. (2021); [heen et al. (2021a);
lay; Lee et al. (2022); Xu et al. (2022); Baker et al. (2022); ; Reed et al.
(2022) all use models that trained from scratch tasks of interest, and [Singh et al. (2022): Ahn
et al. [2022 Huang et al. (2022h) combine frozen foundation models with trainable components adapters. In contrast, Huang al. (202aa) use frozen foundation models for planning, without training or fine-tuning on RL tasks. Like Huang et al. 2022a), ICPI does not update the parameters of the foundation model, but relies the frozen model's in-context learning abilities. However, ICPI gradually builds and improves the prompts within the space defined by the given fixed text-format for observations, actions, and rewards (in contrast Huang et al. (2022a), which uses the frozen model to select good prompts from a given fixed library of goal/plan descriptions).
2.3 IN-CONTEXT LEARNING
Several recent papers have specifically studied in-context learning. Min et al. (2022) demonstrates that I.LMs can learn in-context, even when the labels in the prompt are randomized, problemetizing the conventional understanding of in-context Icarning and showing that label distribution is more important than label correctness, Chan et al. (2022) and Garg et al. (022b) provide analyses of the properties that drive in-context learning, the first in the context of image classification, the second in the context of regression onto a continuous function. These papers identify various properties, including "burstiness." model-size, and model-architecttue, that in-context learning depends on. Chen et al. (2022) studies the sensitivity of in-context learning to small perturbations of the context. They propose a novel method that uses sensitivity as a proxy for model certainty.
ALGORITHM
How can standard policy iteration make use of in-context learning? Policy iteration is either model- based-using world-model to plan future trajectories the environment- foee--iree value-estimates without explicit planning. Both methods can be realized with in-context learning. We choose model-based learning because planned trajectories make the underlying logic of value-
Algorithm 1 Training Loop
1:
function TRAIN(enviromment)
2: initialize D
replay buffer containing full history of behavior
3:
while training do
4: 8o <- Reset environment.
5: while episode is not done do
6:
a; f- arg max, Q(s,, a, D)
& policy improvement
7:
St, b. Execute ar in environment.
8:
+ t++
9:
end while
10:
D + D U (So, an, Fo, bo, .i,...,
.a....,,y, St, at, Fe, be, 8141)
D add trajectory to buffer
11:
end while
12:
end function
3

---

Algorithm 2 Computing Q-values
1:
function Q(st, a, D)
2:
3:
s!
St
4: al
5: repeat
D All samples come from the experience buffer 'D balancing terminal and non-terminal
6:
Dp time-steps with action a
7:
b" LLM (D,, s", a²)
8:
D, ~ time-steps with action a" and termination b"
balancing reward
9:
r" LLM (D,,s", .a))
10:
D2 ~ time-steps with action a and termination b"
D no balancing
11:
g*+1 ~ LLM (Da, a*)
12:
Da recent trajectories
a LLM (s Pa)
14:
+ u++
until b" is terminal
16:
return Σk=1 7t
17:
end function
estimates explicit to our foundation model backbone, providing a concrete instantiation of a trajectory that realizes the values. This ties into recent work (Wei et al., 2022b Nye al. 2021 demonstrating that "chains of thought" can significantly improve few-shot performance of foundation models.
Model-based RL requires two ingredients, rollout-policy used act during planning and world- model used to predict future rewards, terminations, and states. Since our approach avoids any mutation of the foundation model's parameters (this would require gradients), we must instead induce the pollout-policy and the world-model using in-context learning, i.c. by selecting appropriate prompts. We induce the rollout-policy by prompting the foundation model with trajectories drawn from the current (or recent) behavior policy (distinct from the rollout-policy). Similarly, we induce the world-model by prompting the foundation models with transitions drawn from the agent's 's history of experience. Note that our approach assumes access to some translation between the state-space of the environment and the medium (language, images, etc.) of the foundation models. This explains how algorithm might plan and estimate values using foundation model. It also explains how the palluut-policy approximately tracks the behavior policy.
How does the policy improve? When acting in the environment (as opposed planning), choose the action that maximizes the estimated Q-value from the current state (see) Training pseuuuuudd,, line 6). At time step f,, the agent observes the state of the environment (denoted s,) and executes action a, = arg m QT (St, a). where A {A(1), ... A(nn) denotes the set of n actions available, गं denotes the policy of the agent at time step and Q" denotes the Q-estimate for policy π. Taking the greedy (arg max) actions with respect to Q implements new and improved policy.
Computing Q-values This section provides details on the prompts that we use in our computation of Q-values (see Computing Q-values pseudocode Figure T). During training. We maintain buffer 'D of transitions experienced by the agent. To compute Q (st, at time step t in the real-world we rollout simulated trajectory s² St, a = a, r.r s2,s" a², _....... ",...,"""" af r"','" (s', [T+ by predicting,
at each simulation time step reward ra ~ LLM (Dr, s" "" a));; termination b" ~ LLM (Dp, sa a²);; observation *+1 ~ LLM (Da, s", a²);; action g++1 LLM (D,,s)), Termination b" decides whether the simulated trajectory ends step u.
The prompts Dr, Db contain data sampled from the replay buffer. For each prompt, we choose some subset of replay buffer transitions, shuffle them, convert them to text (examples are provided in table 4. and clip the prompt at the 4000-token Codex context limit. We use the same method for D., except that use random trajectory subsequences.
In order to maximize the relevance of the prompt contents to the current inference we select transitions using the following criteria. D, contains (Sk, ag, ba) tuples such that as equals at, the action for
4

---

which the LLM must infer termination. D, contains (Sk, āg, āg, r3)) tuples, again constraining ag a" but also constraining by- b* that the tuple corresponds to a terminal time-step if the LLM inferred b" = true, and to a non-terminal time-step if ba false. For Dx. the prompt includes (Sk, S1+5++) tuples with a5 a" and by false (only non-terminal states need be modelled).
We also maintain a balance of certain kinds of transitions in the prompt. For termination prediction, balance terminal and non-terminal time-steps. Since non-terminal time-steps far outnumber terminal timessteps, this eliminates a situation wherein the randomly sampled prompt time-steps are entirely non-terminal, all but ensuring that the LLM will predict non-termination. Similarly, for reward prediction, we balance the number of time-steps corresponding to each reward value stored in D. In order to balance two collections of unequal size, we take the smaller and duplicate randomly chosen members until the sizes are equal.
In contrast to the other predictions, we condition the rollout policy on trajectory subsequences, not individual time-steps. Prompting with sequences better enables the foundation model to apprehend the logic behind a policy. Trajectory subsequences consist of (Sk; az.) pairs, randomly clipped from the e most recent trajectories. More recent trajectories will, in general demonstrate higher performance, since they come from policies that have benefited from more rounds of improvement.
Finally, the Q-value estimate is simply the discounted sum of rewards for the simulated episode. Given this description of Q-value estimation, we now return to the concept of policy improvement.
The arg max (line of Algorithm D drives policy improvement in ICPI. Critically this is not simply a one-step improvement as with Trajectory Transformer (Janner et al. 202T) but a mechanism that builds improvement on top of improvement. This occurs through a cycle in which the arg max improves behavior. The improved behavior is stored in the buffer D, and then used to condition the rollout policy. This improves the returns generated by the LLM during planning rollouts. These improved rollouts improve the Q-estimates for each action. Completing the cycle, this improves the actions chosen by the arg max. Because this process feeds into itself, it can drive improvement without bound until optimality is achieved.
Note that this process takes advantage of properties specific to in-context learning. In particular, relies the assumption that the rollout policy, when prompted with trajectories drawn from mixture of policies, will approximate something like an average of these policies. Given this assumption, the rollout policy will improve with the improvement of the mixture of policies from which its prompt-traecctories are drawn. This results in kind of rapid policy improvement that works without any use of gradients.
Prompt-Format The LLM cannot take non-linguistic prompts, so our algorithm assumes access a textual representation of the environment-ol states, actions, terminations, and rewards-and some way to recover the original action. termination, and reward values from their textual representation (we do not attempt to recover states). Since our primary results use the Codex language model (see Table 2
we use Python code to represent these values (examples are available Table
In our experiments, we discovered that the LLM world-model was unable to reliably predict rewards, terminations, and next-states on some of the more difficult environments. We experimented with providing domain hints in the form of prompt formats that make explicit useful information similar to Chain of Thought Prompting (Wei et al., 20226). For example, for the chain domain, the hint includes an explicit comparison (== or !=) of the current state with the goal state. Note that while hints are provided in the initial context, the LLM must infer the hint content in rollouts generated from this context.
We use a consistent idiom for rewards and terminations, namely assert reward == x and assert done or assert not done. Some decisions had to be made when representing states and actions. In general, we strove to use simple, idiomatic, concise Python. On the more challenging environments, did search over several options for the choice of hint. We anticipate that in the future, stronger foundation models will be increasingly robust to these decisions.
4 EXPERIMENTS
5

---

Algorithms
ICPI No ArgMax Tabular
Matching Model
Chain
Distractor Chain
Maze
Mini Catch
Mini Invaders
Point Mass
0.8
9066
20.4
0.2
20
40 step
60
80
0
40
80
120
0
40
80 step
120
0
80 160
240
0
80 step
160
0
80
160
step
step
step
Figure 2: Comparison of ICPI with three baselines, "No arg max". "Tabular Q." and "Nearest Neighbor:" The y-axis depicts regret (normalized between 0 and 1), computed relative to an optimal return with a discount-factur of 0.8. The r-axii depicts time-steps during training. Error bars are standard errors from four seeds.
Chain
state leftC
1=
We have three main goals in our experiments: (1) Demonstrate that the agent algorithm can in fact quickly learn good policies, using pretrained LLMs, in a set of six simple illustrative do- mains of increasing challenge; (2) provide evidence through an ablation that the gedicy-improvement step-taking the arg max over Q-values computed through LLM
assert reward
assert not done
Assert state (6, 3)
Distractor
and state 1- (1, 3)
state Ieft
assert not done
learning: and (3) investigate the impact of using different LLMs diffferent
(see Table 2 different sizes and trained different data, in particular, trained on (mostly) natural language (GPT-3 and GPT-J) vs. program code (Codex and InCoder). We next de- scribe the six domains and their associated prompt formats, and then describe the experimental methodology and results.
Teeee state C(i-2, jm1)
Maze
state, revord left()
State C(1=1, (:0)
assert reward ==
not
assert pacille -n C(2, 8)
and ball C(O, 4)
4.1 DOMAINS AND PROMPT FORMAT
Mini Catch
padile.x
ball.x
and padifle, ball.x
and hall y reward left(
Chain. In this environment, the agent occupies 8-state chain. The agent has three actions: Left, right, and try goal. The try goal action always terminates the episode, conferring a reward of 1 on state (the goal state) and 0 on all other states. Episodes also terminate after 8 time-steps. States are represented as numbers from 0 to 7, as in assert state == n, with the appropriate integer substituted for n. The actions are represented as functions left O) right (), and try_goal() (). For the hint, we simply indicate whether or not the current state matches the goal state, 4.
ball. descend
assert reward
assert not done
assert C(2, 9)
and allens [C(3, 5), 00, 5)3
and (ship.x, allens[o].xx
Mini Invaders
aliens[1].x0
3.
and ship.x < [0]
and ship.x > aliens[]]. 1.x
left()
assert reward
for a in aliens:
()
not
Distractor Chain. This environment is an 8-state chain, iden- tical to the chain environment, except that the observation is pair of integers, the first indicating the true state of the agent and the second acting distractor which transitions randomly within {0..... 7). The agent must therefore learn to ignore the distractor integer and base its inferrences contained in the first integer. Aside from the addition of this distractor integer to the observation, all text representations and hints are identical to the chain environment.
assert pas au -3, 46 and vel =s 0.88 and pas -2 and voll en pos, vel (pen, vel) assert reward
Point-Mas
assert not done
the information
Table I: Prompt formats. Basic prompt format in black, additional hint formats in grey (see $3 and 8..1 for further discussion).
Maze. The agent navigates a small x gridworld with obsta- cles. The agent can move up, down, left, right. The episode terminates with a reward of once the agent navigates to the goal grid, with a reward of 0 after 8 time-steps. This envi- ronment tests our algorithms capacity to handle 2-dimensional movement and obstacles, well a 4-action state-space. We represent the states as namedtuples C(x, y), with integers substituted for x and y. Similar to chain, the hint indicates whether or not the state corresponds to the goal state.

---

Mini Catch. The agent operates paddle to catch falling ball. The ball falls from height of 5 units, descending one unit per time step. The paddle can stay in place (not move), or move left or right along the bottom of the 4-umit wide plane. The agent receives reward of for catching the ball and 0 for other time-steps. The episode ends when the ball's height reaches the paddle regardless of whether not the paddle catches the ball. We chose this environment specifically to challenge the ritionninfcrencc/oollou----
component of our algorithm. Specifically, note that the success condition in Mini Catch allows the paddle meander before moving under the ball-as long as gets there on the final time-step. Successful trajectories that include movement away from the ball thus make a good rollout policies more challenging learn (i.e., elicit from the LLM via prompts).
Again, we represent both the paddle and the ball as namedtuples C(x, y) and we represent actions as methods of the paddle object: paddle. addle.stay() paddle. left(). and paddle. right (). For the hint, we call out the location of the paddle's x--position, the ball's zr-position, the relation between these positions (which larger than which, or whether they are equal) and the ball's y-position. Table [ provides example. We also include the text ball. (ell.descend() to account for the change in the ball's position between states.
Mini Invaders. The agent operates a ship that shoots down aliens which descend from the top of the screen. the beginning of an episode, two aliens spawn a random location in two of four columns. The episode terminates when an alien reaches the ground (resulting in O reward) or when the ship shoots down both aliens (the agent receives reward per alien). The agent can move left, right, shoot. This domain highlights ICPI's capacity to learn incrementally, rather than discovering an optimal policy through random exploration and then imitating that policy, which is how our "No arg max" baseline learns (see Comparison of ICPI with baseline algorithms). ICPI initially learns to shoot down one alien, and then builds on this good but suboptimal policy to discover the better policy of shooting down both aliens. In contrast, random exploration takes much longer to discover the optimal policy and the "No arg max" baseline has only experienced one or two successful trajectories by the end of training.
We represent the ship by its namedtuple coordinate (C(x, y)) and the aliens as a list of these nameduules. When alien is shot down, we substitute None for the tuple, in aliens == [C(x, y), NoneJ. We add the text for a in aliens: a. descend() in order to account for the change in the alien's 's position between states.
Point-Mass. A point-mass spawns at a random position on a continuous line between -6 and +6 with velocity of O. The agent can either accelerate the point-mass (increase velocity by 1) or decelerate it (decrease the velocity by 1). The point-mass position changes by the amount of its velocity each timestep. The episode terminates with reward of once the point-mass is between -2 and +2 and its velocity is 0 once again. The episode also terminates after 8 time-steps. This domain tests the algorithm's 's ability handle continuous states.
States are represented as assert pos == and vel == v. substituting floats rounded to two decimals for p and v. The actions are accel (pos, vel) and decel (pos, vel). The hint indicates whether the success conditions are met, namely the relationship of pos to - and +2 and the whether or not vel 0. The hint includes identification of the aliens' and the ship's 's x z--osstiions as well as a comparison between them.
4.2 EXPERIMENT METHODOLOGY AND RESULTS
Methodology and evaluation. For the results, we record the agent's 's regret over the course of training relative to an optimal policy computed with a discount factor of 0.8. For all experiments 8 (the number of most recent successful trajectories to include in the prompt). We did not have time for hyperparameter search and chose this number based on intuition, however the c = 16 baseline demonstrates results when this hyperparameter is doubled. All results use 4 seeds.
For both versions Codex, we used the OpenAl Beta under the API Terms of Use. For GPT-J (Wang & Komatsuzaki] 2021), InCoder (Fried et al. 2022) and OPT-30B (Zhang al. 2022). we used the open-source implementations from Huggingface Transformers (Wolf et al. (020). each running on one Nvidia A40 GPU, All language models sampling temperature of 0.1.
7

---

Ablations
ICPI No Hints cm16 No Constraints
No Balance
Chain
Distractor Chain
Maze
Mini Catch
Mini Invaders
Point Mass
0.8
006
20.4
0.2
20
40 step
60
80
0
40
80
120
0
40
80 step
120
0
80
160
240
0
80 step
160
0
80
160
step
step
step
Figure 3: Comparison of ICPI with ablations. The y-axis depicts regret (normalized between 0 and 1), computed relative to optimal return with discount-facton of 0.8. The z-axis depicts time-steps during training. Error bars are standard errors from four seeds.
Comparison of ICPI with baseline algorithms. We compare ICPI with three baselines (Fig. 2
The "No arg max" baseline learns a good policy through random exploration and then imitates this policy. This baseline assumes access "success threshold" for each domain cumulative return greater than which trajectory is considered successful. The action selection mechanism emulates ICPI's rollout policy: prompting the LLM with set of trajectories and eliciting an action output. For this baseline, we only include trajectories in the prompt whose cumulative return exceeds the success threshold. Thus the policy improves as the number of successful trajectories in the prompt increases over time. Note that at the start of learning, the agent will have experienced too few successful trajectories to effectively populate the policy prompt. In order to facilitate exploration, we act randomly until the agent experiences 3 successes.
undiscounted
"Tabular Q" is a standard tabular Q-learning algorithm, which uses a learning rate of 1.0 and opti- mistically initializes the Q-values to 1.0.
"Matching Model" is a baseline which uses the trajectory history instead of an LLM to perform modelling. This baseline searches the trajectory buffer for the most recent instance of the current state, and in the case of tranitiinn/ewaad/eemmnnaioo prediction, the current action. If a match is found, the model's outputs the historical value (e.g. the reward associated with the state-action pair found in the buffer). If no match is found, the modelling rollout is terminated. Recall that ICPI breaks ties randomly during action selection this will often lead to random action selection.
As our results demonstrate, only ICPI learns good policies on all domains. We attribute this advantage ICPI's ability generalize from its context unseen states and state/action pairs (unlike "Tabular Q" and "Matching Model"). Unlike "No arg max" ICPI is able to learn progressively, improving the policy before experiencing good trajectories.
Ablation of ICPI components. With these experiments, ablate those components of the algorithm which are not, in principle, essential to learning (Fig. 3). "No Hints" ablates the hints described the Prrmpt-Formaa paragraph. "No Balance" removes the balancing of different kinds of time-steps described in the Computing Q-values paragraph (for example, D, is allowed to contain an unequal number of terminal and con-terminal time-steps). The "No Constraints" baseline removes the constraints on these time-steps described in the same paragraph. For example, D, is allowed to contain a mixture of terminal and non-terminal time-steps (regardless of the model's termination prediction). Finally, "c = 16" prompts the rollout policy with the last 16 trajectories (instead of the last 8, as in ICPI). We find that while some ablations match ICPI's performance in several domains, none match its performance all six.
Comparison of Different Language Models. While our lab lacks the resources to do full study of scaling properties, we did compare several language models of varying size (Fig. 4). See Table 2 for details about these models. Both sode-davinci-ege and code-cushman-002 are variations of
the Codex language model. The exact number of parameters in these models is proprietary according to OpenAI, but Chen al. (2021b) describes Codex as fine-tuned from GPT-3 Gro-3/Brown al. (2020), which contains I85 billion parameters. As for the distinction between the variations, the OpenAl website describes cide-cushman-001 "almost as capable as Davinei Codex, but slightly faster."
8

---

Language Models
code-davinci-002 code-cushman-001
OPT-30B
InCoder GPT-J
Chain
Distractor Chain
Maze
Mini Catch
Mini Invaders
Point Mass
0.8
2 9.66
20.4
0.2
20
40
60
80
0
40
80
120
0
40
80 step
120
0
80
160
240
0
80 step
160
0
80
160
step
step
step
step
Figure 4: Comparison of ICPI with ablations. The y-axis depicts regret (normalized between 0 and I) computed relative to optimal return with discount-factor of 0.8. The z-axis depicts time-steps of training. Error bars are standard errors from four seeds.
Table 2: Pretrained Large Language Models (LLMs) Used in Experiments
Model
Parameters
Training data
GPT-J
Wang & Komatsuzaki
"The Pile" (Gao et al. 2020). an 825GB English corpus inel. Wikipedia, GitHub, academic pubs
6 billion
2021
InCoder Fried et al.
2022
6.7 billion
159 GB of open- StackOverflow code
180B tokens of predominantly English data including "The Pile" (Gao al.] (0200 and "PushShift.io Reddit" (Baumgartner et al. 2020)
ОPT-30B (Zhang et al.
2022)
30 billion
Codex
Chen et al.
2021b
185 billion
179 GB of GitHub code
We found that none of the smaller models were capable of demonstrating learning any domain but the simplest, chain. Examiniig the trajectories generated by agents trained using these models, we noted that in several cases, they seemed to struggle to apprehend the underlying "logic" of successful trajectories, which hampered the ability of the rollout policy to produce good actions. Since these smaller models were not trained on identical data, are unable to isolate the role of size in these results. However, the failure of all of these smaller models to learn suggests that size has some role to play in performance. We conjecture that larger models developed in the future may demonstrate comparable improvements in performance over our Codex model.
Limitations In principle, ICPI can work on any control task with a discrete state space. All that the method requires sequence prediction model (in the paper we use Codex) capable of inferring state transitions and action probabilities given a history of behavior. Such model will ensure that the rollouts (described in Computing Q-values) are unbiased monte-carlo estimates of value for the current policy. Given such estimates, the algorithm will implement policy iteration, method which proven to converge. The domains in our paper were limited by the abilities of Codex to accurately predict transitions and actions in more complex domains. As sequence models mature, anticipate that more domains will become tractable for ICPI.
CONCLUSION
Our main contribution gradient-free policy iteration algorithm, using foundation models solve RL tasks, in which the entire locus of learning is improved prompt content. The algorithm uses foundation models as both a world model and policy to compute Q-values via rollouts. Although we presented the algorithm here as text-based, the approach is quite general and could be applied to any foundation models that works through prompting, including multi-modal models like Reed et al.) (2022) and Seoo et al. (2022). In experiments we showed that the algorithm works in six illustrative domains imposing different challenges for ICPI, confirming the benefit of the LLM-rollout based policy improvement. While the empirical results are quite modest, we believe the approach provides an important new way to use LLMs that will increase in effectiveness as the models themselves become better, our size comparison study suggests.

---

REFERENCES
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho. Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang- Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Joreell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, and Mengyuan Yan. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances, April 2022. URL http: //axxiv.org/abs/222. 01691 arXiv:2204.01691 [cs].
Prithviraj Ammanabrolu and Mark Riedl. Learning Knowledge Graph-based World Models of Textual Environments. In Advances in Neural Information Processing Systems, volume 34, pp. 3720-3731. Curran Associates, Inc., 2021. URL htttps:// www.:eeddinnnnniii.ii..mmmm cctprppppccpppprr/
html
Bowen Baker, lige Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video PreTraining (VPT): Learning Act by Watching Unlabeled Online Videos, June 2022. URL arXiv:2206. 1195 [es].
11795
Jason Baumgartner, Savvas Zannettou, Brian Keegan, Megan Squire, and Jeremy Blackburn. The Pushshift Reddit Dataset, January 2020. URL /ttp::://aaxiii.oo////////// 08435 Number: arXiv:2001.08435 arXiv:2001.08435 [cs].
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse. Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Ben- jamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. In Advances in Neural Information Pro- cessing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. htt/https///ppptee html
nips. hccpppper/2200//ahh///// http://0000/aaa1115555770oooobb0 6b497756666666666666444444 Sbtbaac1bbbabbbbb
Stephanie C. Y. Chan, Adam Santoro, Andrew K. Lampinen, Jane X. Wang, Aaditya Singh, Pierre H. Richemond, Jay McClelland, and Felix Hill. Data Distributional Properties Drive Emergent In-Context Learning in Transformers, May 2022. URL http:://axxiv org abs/ 05055 arXiv:2205.00055 [es].
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision Transformer: Reinforcement Learning via Sequence Modeling. arXiv:21066 (.0.011444 01345 [cs], June 2021a. URL URL http:://arxiv.rrr/aa/222.... 01345)
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger. Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Merbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating Large Language Models Trained on Code. Technical Report arXiv:2107.03374 arXiv, July 2021b. URL http://arxiv http://wwwwrr///////////// 03374
Yanda Chen, Chen Zhao, Zhou Yu, Kathleen McKeown, and He He. On the Relation between Sensitivity and Accuracy in In-context Learning, September 2022. URL http:tttt//axi..rrr..... abs/2209. 07661 arXiv:2209.077 [es].
10

---

Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. InCoder: A Generative Model for Code Infilling and Synthesis. arXiv:2204.0555 [es), April 2022. URL
wwwpg///bb//......ggg 05999
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy, The Pile: An 800GB Dataset of Diverse Text for Language Modeling. arXiv:2101.00021 [cs], December 2020. URL
http://arxivv.orgaaa
80027
Divyansh Garg, Skanda Vaidyanath, Kuno Kim, Jiaming Song, and Stefano Ermon. LISA: Learning Interpretable Skill Abstractions from Language. 1xXiv:2203.0005: [es], February 2022a. URL 00054
Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What Can Transformers Learn In-Context? A Case Study of Simple Function Classes, August 2022b, URL http:///araii abs/2208. 01066 arXiv:2208. 010666 [es].
Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley. Human Instruction-Folllwiing with Deep Reinforcement Learning via Transfer-Leerning from Text. arXi::2005.09332 [cs], May 2020.
URL http:://rxxv httt:://arrxv.orr/aa//22555 09382
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents. arXiv:2201.07207 [es], March 200pab///220000007777
2022a. Uttt http:/awwwii...rga/////a/0//////
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner Monologue: Embodied Reasoning through Planning with Language Models, July 2022b. URL $207.05600 arXiv:2207.0560 [es].
Michael Janner, Qiyang Li, and Sergey Levine. Offline Reinforcement Learning as One Big Sequence Modeling Problem, November 2021. URL arXiv:2106.02039 [cs].
www.:/org.abb///o...eeg 02039
Rabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Lambert Mathias, Marzieh Saeidi, Veselin Stoyanov, and Majid Yazdani. Prompt-free and Efficient Few-shot Learning with Language Models. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3638-3652, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10. 10.18653/1120222ac--hb https://acllattioooo...g///////
URL
Kuang-Huel Lee, Ofir Nachum, Mengjiao Yang. Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, Ian Fischer, Eric Jang, Henryk Michalewski, and Igor Mordatch. Multi-Game De- cision Transformers, May 2022. URL http://////ax www. 2205. 15241 15X2 [cs].
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin Akyürek, Anima Anandkumar, Jacob Andreas, Igor Mordatch, Antonio Torralba, and Yuke Zhu. Pre-Trai Trained Language Models for Interactive Decision-Making arXi::220.001771 [cs], February 2022. URL http:://arrvv...g/aaa//////
Xiang Lisa Li and Percy Liang. Prefix-Tuning Optimizing Continuous Prompts for Generation. January 2021. doi: 10.48550/u2Xiv21111.000 URL httpl/ts/////xx..o////////////// .00190v1
Kevin Lu, Aditya Grover, Pieter Abbeel, and Igor Mordatch. Pretrained Transformers as Universal Computation Engines. 103iv:2103.05247 fes), March 2021. URI.htpp///axiv.orrr/aa/// 05247
Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, and Dhruv Batra. Im- proving Vision-and -ad-a--------eeee Navigation with Image- Pairs from the Web. arXiv:2004.1497] [cs), May 2020. URL htttp::/aaxxw.orr//abb...

---

Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? arXiv:2202.12837 [cs], February 2022.
org/ http//b//22282 12837
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show Your Work: Scratchpads for Intermediate Computation with Language Models, November 2021. URL htttp::/aaxxiv.ogg/aabs....
00114 arXiv:2112.001 14 [es].
Xiangyu Peng, Mark O. Riedl, and Prithviraj Ammanabrolu. Inherently Explainable Reinforcement Learning in Natural Language. arXiv:2 12.09907 [cs], December 2021. URL http://arxiv
org/abs/2112. 08907
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals. Mahyar Bordbar, and Nando de Freitas. A Generalist Agent, May 2022. URL arXiv:2205.06175 [cs].
06175
Younggyo Seo, Kimin Lee, Fangchen Liu, Stephen James, and Pieter Abbeel. HARP: Autoregressive
Latent Video Prediction with High-Fidelity Image Generator, September 2022. URL. http:
//arxiv.org/abs. 220gaa//////.e2ee 07143. arXiv:2209.0714. [es].
Aaditya K. Singh, David Ding, Andrew Saxe, Felix Hill, and Andrew K. Lampinen. Know your
audience: specializing grounded language models with the game of Dixit, June 2022. URL
http:///arxiv.org/abba 2233/
arXiv:2206.08349 [cs].
Ishika Singh, Gargi Singh, and Ashutosh Modi. Pre-trained Language Models as Prior Knowledge for Playing Text-based Games. arXiv:2107.0844 [cs), December 2021. URL http://arxiv
org/ a22/07 .84088
Allison C. Tam, Neil C. Rabinowitz, Andrew K. Lampinen, Nicholas A. Roy, Stephanie C. Y. Chan, D. J. Strouse, Jane X. Wang, Andrea Banino, and Felix Hill. Semantic Exploration from Language Abstractions and Pretrained Representations. arXiv:2204.05080 [cs], April 2022. URL
http://arxiw.org/abs/ 2204 220.000000
Denis Tarasov, Vladislav Kurenkov, and Sergey Kolesnikov. Prompts and Pre-Trainee Trained Language Models for Offline Reinforcement Learning. March 2022. URL
net/
forum?idd=pfATEENNWW
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need, December 2017. URL http: 03762 arXiv:1706.03766 [cs].
www.xiv.org/bbs///
Ben Wang and Aran Komatsuzaki. GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model, May 2021. URL http/pppr///ggt.ooooooo//oonnooo http:///esssttttasffrrrrrrrraaa
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent Abilities of Large Language Models, June 2022a. URL IRttp:///aax www.:voorrrroaao att//bbb/22626 07682 arXiv:2206.07682 [cs].
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. Chain of Thought Prompting Elicits Reasoning in Large Language Models. 220.:2200119900
[cs], April 2022b. URL htt:ht/p///attt http::///aaaaa.ooo/////////
wwwp:///g//b///////g/e 11903
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-Art Natural Language Processing. In Proceedings of the 2020 Conference on Empirical Meth- ods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.com hatpe000mmmlp--ee demol... URL https://acllnnthooogy..rgg/ org.////ooorrg/2200 emn emn1p-demos.6
12

---

Mengdi Xu, Yikang Shen, Shun Zhang. Yuchen Lu, Ding Zhao, Joshua B. Tenenbaum, and Chuang Gan. Prompting Decision Transformer for Few-Shot Policy Generalization, June 2022. URL
http://arkkv.orr//aa//2222 13499 arXiv:2206. 13499 [cs].
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang. and Luke Zettlemoyer. OPT: Open Pre-trained Transformer Language Models, 2022. eprint: 2205.01068.
13


Reviews:
[{"review_id": "o12Z7vw_ut", "reviewer": "Reviewer_8XAi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces In-Context Policy Iteration that learns to perform RL with foundation models without expert demonstrations or gradients", "summary_of_the_review": "The paper provides a simple idea of training RL with foundation models. But I'm not sure how previous related works conduct their experiments since the tasks seem very limited.", "strength_and_weaknesses": "Pros: (1) The paper is clearly written and easy to follow.\n\n(2) The proposed method is novel in my opinion.\n\nQuestions: (1) In Alg. 1, can the authors specify what are prompt \"parameters\"? Are these the only trainable parameters?\n\n(2) What will be the disadvantage of fine-tuning the foundation model parameters?\n\n(3) Can the method work on more general control tasks such as mujoco?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "egEszyoTVzh", "reviewer": "Reviewer_jeCL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces in-context policy iteration (ICPI) an algorithm that can perform model-based policy iteration by leveraging a pre-trained large language model (LLM). ICPI uses prompts to create both a world model and a rollout policy out of the same LM. Specifically, the world model prompt uses examples from a buffer of transitions from the real environment and the rollout policy. ICPI then uses Monte Carlo rollouts of the rollout policy in the world model to estimate a Q function that is used to choose actions in the real environment. The paper presents a variety of experiments on toy problems with small, finite action spaces and small state spaces (finite or easily discretized) demonstrating improvement over some baselines.  ", "summary_of_the_review": "Overall, I think the core idea presented in the paper is interesting and could be valuable to the community. However, I do have some concerns about why some things were done the way they were, so I am rating the paper as a weak reject for now. If the authors are able to resolve my questions, I would consider raising my score. ", "strength_and_weaknesses": "### Strengths\n\n1. The paper presents a clever way to leverage LLMs to perform RL entirely from in-context learning without any gradients. This is (to my knowledge) novel and seems like it could open up a new direction of research in using LLMs for RL.\n\n2. The paper is clearly written and experiments seem relatively thorough on the toy problems that were selected.\n\n### Weaknesses\n\n1. As the authors admit in the paper, the tasks presented are very small-scale. It is unclear to me what the key bottlenecks are in scaling this approach to larger tasks. It is clear that it is limited to finite action spaces and to the ability to encode the state space into simple Python code, but it does not seem too challenging to embed some more complicated environments in this format. The paper would benefit substantially from either (a) scaling up to harder tasks or (b) including a more detailed discussion of the bottlenecks that prevented scaling up in this project.\n\n2. The gains over the baselines were not always very clear and it was unclear how the baselines were chosen. On the first point, it would also be nice to see a more quantitative comparison to the baselines along with the regret curves. On the second, in particular, since all the domains are so small-scale it seems that it would be reasonable to include a more rigorous tabular baseline than Q-learning, for example a tabular model-based algorithm with more rigorous count-based exploration could potentially be a more fair comparison (especially since ICPI is using the LLM as a world model).\n\n3. There are no results that attempt to analyze what the algorithm is doing beyond plotting regret curves. It would be nice to either see (a) some more qualitative/exploratory results that explain what sorts of errors each of the considered methods makes in each task or (b) some theory about what is required from the LLM for ICPI to actually work. \n\n4. It is unclear whether how the \"hints\" fit into the standard RL framework and how they can be fairly compared against baselines that don't have access to them. This is not a major issue as the paper does provide the ablation without the hints, but it does seem like a potentially important caveat to the results that they required such hinting. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YQShCSBwhHw", "reviewer": "Reviewer_Kfri", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper implements an implementation of the policy iteration algorithm using trained LLM.\n", "summary_of_the_review": "I enjoyed the paper a lot. The presented utilized an LLM and, without any training, is able to solve a few RL tasks. However, I have serious doubts about its significance. What important research questions is being answered? Further,  very simple environments are treated and even then, some tricks are required (i.e. ejecting information about reward). It does not feel like the method would scale to anything bigger?\n\nI admit not being an expert in the field of LLMs and I'd be happy to reconsider my evaluation. ", "strength_and_weaknesses": "The strength of this paper lies in using an LLM without any training. This is also a weakness, at its present form, the paper feels not that significant. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Jq183erlvvP", "reviewer": "Reviewer_wJ8R", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose In-Context Policy Iteration, a Q-learning like algorithm that iteratively prompts large language models (LLM) with trajectories that have yielded better rewards and taps into the in-context learning abilities of LLMS to simulate policy rollouts. The proposed algorithm can self-improve without weight updates or expert demonstrations. \n\nAt its core, ICPI queries an LLM using past trajectory data (i.e. replay buffer) to obtain simulated rollouts that are used to estimate Q values, which are then used to take actions and augment the replay buffer. \n\nContributions:\n* The authors propose the ICPI algorithm, which enjoy the properties described above. \n* They evaluate it's performance on 6 toy tasks and report results on:\n  * a few sensible baselines, \n  * algorithm ablations, which involve some tips/tricks to get ICPI working, and \n  * the performance on ICPI using different core LLMs, including OpenAI's Codex. ", "summary_of_the_review": "The paper proposes ICPI, an algorithm that enables LLMS to learn to solve simple RL tasks without any expert demonstrations and weight updates. \n\nWhile there are concerns related to the broad applicability and scalability of this algorithm, I believe that the novelty and scientific value of the results justify acceptance. ", "strength_and_weaknesses": "**STRENGTHS**:\n* **No need for expert demonstrations:** The proposed algorithm doesn't require expert demonstrations, or any form of imitation learning. \n* **No weight updates:** There's no need to compute gradients through the LLMs during learning: only (discrete) prompts are refined over training to facilitate learning. \n* **Task difficulty appear well-chosen:** While relatively easy from an RL perspective, the six tasks presented in the paper appear to capture the right difficulty to test ICPI. Only Codex seems to get all tasks right, which gives useful signal as to what about the LLMs could be responsible for the reported performance. The chosen tasks are also in increasing complexity, and each display qualitatively different abilities. \n* **Nice variability of LLMs:** The authors seem to cover a sufficient range of publicly available LLMS. \n\n**WEAKNESSES:**\n* **Necessity of hints and tweaks:** It appears that for any sufficiently nontrivial environment, human-engineered hints are needed to get ICPI to work. The fact that some statistical rebalancing (of $D_r$ and $D_r$) also signal brittleness (which might potentially disappear as the LLMs get better).  \n* **Restricted to text domain:** Currently, ICPI is limited to domains where LLMs can exploit their pretrained knowledge (whether it be natural language or code). \n* **Issues regarding scaling:** The proposed method might have difficulty scaling to tasks with larger difficulty than those explored in the paper, as prompting LLMs with a large number of \"state-action-termination\" tuples when the dimensionality of these are high might become computationally challenging. \n\n**QUESTIONS TO THE AUTHORS:**\n* **Do LLMs play a unique role in this algoritm?** I wanted to ask further questions about the precise role LLMs play in the proposed algorithm. There doesn't appear to be an explicit pressure on the LLM to seek actions that yield higher returns (which could have been there if weight updates were allowed, or if it were prompted with expert trajectories, or perhaps the LLM were prompted with an explicit instruction to seek higher rewards). Instead, the algorithm seems to improve over time as the LLM is prompted with data that's increasingly coming from better (i.e. higher reward) rollouts. If this is the case, why would an LLM be necessary? Even if one randomly sampled termination, reward, next state and actions from $D_b$, $D_r$, $D_s$ and $D_a$ during rollouts, wouldn't learning still occur? What exactly is the benefit of using an LLM here? Perhaps the pretraining data helps pick better actions in a nontrivial way? \n* **Is Tabular Q tuned well?** All of the tasks seem simple enough that the Tabular Q algorithm should eventually learn the optimal policy (is this right?) However, that doesn't appear to be the case for certain environments. What seems to have gone wrong? Perhaps tabular-Q takes a bit longer to converge? Or perhaps the hyperparameters (most importantly the learning rate) aren't tuned separately for each task?\n* **Request for additional baselines and ablations:** At each rollout step, the LLM gets queried 4 times. Time permitting, could you please run an ablation where you systematically replace each one with a random sample from $D_x$ where $x \\in \\{b, r, s, a\\}$ while keeping the other three samples intact (i.e. they still use LLM)? I'm asking this to see if all of these queries are similarly important for the apparent superiority of ICPI over other baselines such as the \"matching model baseline\".\n* **Model outputs during the first few rollouts:** Since $D$ is empty at the beginning of training, it's not clear how the LLM should be prompted to get it to output sensible things. Is this every a problem? How do you address it? \n* **How exactly are $D_i sampled?** The text mentions that \"$D_b$ contains [...] tuples sampled randomly from the $D$\", Do you mind explaining how this exactly works? Is random sampling done uniformly? How many samples are picked? \n* **Further details on the Point-mass task:** Do you mind explaining what \"accel(pos, vel)\"does exactly?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject