Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose In-Context Policy Iteration, a Q-learning like algorithm that iteratively prompts large language models (LLM) with trajectories that have yielded better rewards and taps into the in-context learning abilities of LLMS to simulate policy rollouts. The proposed algorithm can self-improve without weight updates or expert demonstrations. 

At its core, ICPI queries an LLM using past trajectory data (i.e. replay buffer) to obtain simulated rollouts that are used to estimate Q values, which are then used to take actions and augment the replay buffer. 

Contributions:
* The authors propose the ICPI algorithm, which enjoy the properties described above. 
* They evaluate it's performance on 6 toy tasks and report results on:
  * a few sensible baselines, 
  * algorithm ablations, which involve some tips/tricks to get ICPI working, and 
  * the performance on ICPI using different core LLMs, including OpenAI's Codex. 
strength_and_weaknesses: **STRENGTHS**:
* **No need for expert demonstrations:** The proposed algorithm doesn't require expert demonstrations, or any form of imitation learning. 
* **No weight updates:** There's no need to compute gradients through the LLMs during learning: only (discrete) prompts are refined over training to facilitate learning. 
* **Task difficulty appear well-chosen:** While relatively easy from an RL perspective, the six tasks presented in the paper appear to capture the right difficulty to test ICPI. Only Codex seems to get all tasks right, which gives useful signal as to what about the LLMs could be responsible for the reported performance. The chosen tasks are also in increasing complexity, and each display qualitatively different abilities. 
* **Nice variability of LLMs:** The authors seem to cover a sufficient range of publicly available LLMS. 

**WEAKNESSES:**
* **Necessity of hints and tweaks:** It appears that for any sufficiently nontrivial environment, human-engineered hints are needed to get ICPI to work. The fact that some statistical rebalancing (of $D_r$ and $D_r$) also signal brittleness (which might potentially disappear as the LLMs get better).  
* **Restricted to text domain:** Currently, ICPI is limited to domains where LLMs can exploit their pretrained knowledge (whether it be natural language or code). 
* **Issues regarding scaling:** The proposed method might have difficulty scaling to tasks with larger difficulty than those explored in the paper, as prompting LLMs with a large number of "state-action-termination" tuples when the dimensionality of these are high might become computationally challenging. 

**QUESTIONS TO THE AUTHORS:**
* **Do LLMs play a unique role in this algoritm?** I wanted to ask further questions about the precise role LLMs play in the proposed algorithm. There doesn't appear to be an explicit pressure on the LLM to seek actions that yield higher returns (which could have been there if weight updates were allowed, or if it were prompted with expert trajectories, or perhaps the LLM were prompted with an explicit instruction to seek higher rewards). Instead, the algorithm seems to improve over time as the LLM is prompted with data that's increasingly coming from better (i.e. higher reward) rollouts. If this is the case, why would an LLM be necessary? Even if one randomly sampled termination, reward, next state and actions from $D_b$, $D_r$, $D_s$ and $D_a$ during rollouts, wouldn't learning still occur? What exactly is the benefit of using an LLM here? Perhaps the pretraining data helps pick better actions in a nontrivial way? 
* **Is Tabular Q tuned well?** All of the tasks seem simple enough that the Tabular Q algorithm should eventually learn the optimal policy (is this right?) However, that doesn't appear to be the case for certain environments. What seems to have gone wrong? Perhaps tabular-Q takes a bit longer to converge? Or perhaps the hyperparameters (most importantly the learning rate) aren't tuned separately for each task?
* **Request for additional baselines and ablations:** At each rollout step, the LLM gets queried 4 times. Time permitting, could you please run an ablation where you systematically replace each one with a random sample from $D_x$ where $x \in \{b, r, s, a\}$ while keeping the other three samples intact (i.e. they still use LLM)? I'm asking this to see if all of these queries are similarly important for the apparent superiority of ICPI over other baselines such as the "matching model baseline".
* **Model outputs during the first few rollouts:** Since $D$ is empty at the beginning of training, it's not clear how the LLM should be prompted to get it to output sensible things. Is this every a problem? How do you address it? 
* **How exactly are $D_i sampled?** The text mentions that "$D_b$ contains [...] tuples sampled randomly from the $D$", Do you mind explaining how this exactly works? Is random sampling done uniformly? How many samples are picked? 
* **Further details on the Point-mass task:** Do you mind explaining what "accel(pos, vel)"does exactly?

clarity,_quality,_novelty_and_reproducibility: **Clarity:** For the most part, the paper is written clearly and the motivation is laid out very well. Perhaps adding further details to the descriptions outlining how $D_b$, $D_r$, $D_s$ and $D_a$ are picked during rollouts would help. 

**Novelty:** While the paper doesn't represent a conceptual leap (i.e. the ingredients of ICLI were present at the type of publication), the execution and precise formulation appear to be novel. 

**Reproducibility:** While the proposed algorithm is clear overall, producing a faithful replication of the work seems difficult (i.e. how the $D_i$ prompts are constructed is not extremely clear, etc.). There's also no released codebase. 


**Typos:**
* Citep should be used when citing Codex in the abstract. 
* While described, $D_s$ is not mentioned in the second paragraph in the "Computing Q-values" section. 
* In the "prompt-format" section, the sentence "Note that while hints are provided hints in the initial context, [...]"
* In "comparison of ICPI with baseline algorithms:" "This baseline assumes access **to** a [...]". 



summary_of_the_review: The paper proposes ICPI, an algorithm that enables LLMS to learn to solve simple RL tasks without any expert demonstrations and weight updates. 

While there are concerns related to the broad applicability and scalability of this algorithm, I believe that the novelty and scientific value of the results justify acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper implements an implementation of the policy iteration algorithm using trained LLM.

strength_and_weaknesses: The strength of this paper lies in using an LLM without any training. This is also a weakness, at its present form, the paper feels not that significant. 

clarity,_quality,_novelty_and_reproducibility: The paper is clear, and of high quality, the presented method is correct.
summary_of_the_review: I enjoyed the paper a lot. The presented utilized an LLM and, without any training, is able to solve a few RL tasks. However, I have serious doubts about its significance. What important research questions is being answered? Further,  very simple environments are treated and even then, some tricks are required (i.e. ejecting information about reward). It does not feel like the method would scale to anything bigger?

I admit not being an expert in the field of LLMs and I'd be happy to reconsider my evaluation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper introduces in-context policy iteration (ICPI) an algorithm that can perform model-based policy iteration by leveraging a pre-trained large language model (LLM). ICPI uses prompts to create both a world model and a rollout policy out of the same LM. Specifically, the world model prompt uses examples from a buffer of transitions from the real environment and the rollout policy. ICPI then uses Monte Carlo rollouts of the rollout policy in the world model to estimate a Q function that is used to choose actions in the real environment. The paper presents a variety of experiments on toy problems with small, finite action spaces and small state spaces (finite or easily discretized) demonstrating improvement over some baselines.  
strength_and_weaknesses: ### Strengths

1. The paper presents a clever way to leverage LLMs to perform RL entirely from in-context learning without any gradients. This is (to my knowledge) novel and seems like it could open up a new direction of research in using LLMs for RL.

2. The paper is clearly written and experiments seem relatively thorough on the toy problems that were selected.

### Weaknesses

1. As the authors admit in the paper, the tasks presented are very small-scale. It is unclear to me what the key bottlenecks are in scaling this approach to larger tasks. It is clear that it is limited to finite action spaces and to the ability to encode the state space into simple Python code, but it does not seem too challenging to embed some more complicated environments in this format. The paper would benefit substantially from either (a) scaling up to harder tasks or (b) including a more detailed discussion of the bottlenecks that prevented scaling up in this project.

2. The gains over the baselines were not always very clear and it was unclear how the baselines were chosen. On the first point, it would also be nice to see a more quantitative comparison to the baselines along with the regret curves. On the second, in particular, since all the domains are so small-scale it seems that it would be reasonable to include a more rigorous tabular baseline than Q-learning, for example a tabular model-based algorithm with more rigorous count-based exploration could potentially be a more fair comparison (especially since ICPI is using the LLM as a world model).

3. There are no results that attempt to analyze what the algorithm is doing beyond plotting regret curves. It would be nice to either see (a) some more qualitative/exploratory results that explain what sorts of errors each of the considered methods makes in each task or (b) some theory about what is required from the LLM for ICPI to actually work. 

4. It is unclear whether how the "hints" fit into the standard RL framework and how they can be fairly compared against baselines that don't have access to them. This is not a major issue as the paper does provide the ablation without the hints, but it does seem like a potentially important caveat to the results that they required such hinting. 
clarity,_quality,_novelty_and_reproducibility: The paper was clear and well-written.

The quality of the experiments was good up to the issues raised above.

The algorithm is novel and interesting.

No code was provided so it is difficult to say how reproducible the results are. 
summary_of_the_review: Overall, I think the core idea presented in the paper is interesting and could be valuable to the community. However, I do have some concerns about why some things were done the way they were, so I am rating the paper as a weak reject for now. If the authors are able to resolve my questions, I would consider raising my score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces In-Context Policy Iteration that learns to perform RL with foundation models without expert demonstrations or gradients
strength_and_weaknesses: Pros: (1) The paper is clearly written and easy to follow.

(2) The proposed method is novel in my opinion.

Questions: (1) In Alg. 1, can the authors specify what are prompt "parameters"? Are these the only trainable parameters?

(2) What will be the disadvantage of fine-tuning the foundation model parameters?

(3) Can the method work on more general control tasks such as mujoco?
clarity,_quality,_novelty_and_reproducibility: The paper writing is clear and the novelty is clear.
summary_of_the_review: The paper provides a simple idea of training RL with foundation models. But I'm not sure how previous related works conduct their experiments since the tasks seem very limited.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The main contribution of this work lies in the proposed in-context policy iteration algorithm that exploits pretrained large-language models for solving small, simple RL tasks without needing expert demonstrations nor weight updates, which is interesting, new, and can potentially pave the way for future works.

After reviewing and responding to the authors' rebuttal, a major concern that is raised by all reviewers is that the empirical evaluation of the proposed approach is based on small simple RL tasks and it is unclear whether it can scale well to more complex tasks. We acknowledge that the authors have responded to this concern by saying that their approach can scale in principle, but the main culprit for this limitation is the large language model being used at this time. They believe that this concern will be resolved with the advancement of LLMs in the future. Their current work is simply to show its feasibility, albeit in small simple RL tasks. Such a response has drawn mixed feedback from the reviewers: On one hand, the authors have indeed demonstrated its feasibility through simple RL tasks. On the other hand, it is unclear whether their approach can indeed scale in practice.

We strongly encourage the authors to revise their work based on the reviewers' feedback.

Unfortunately, the authors have breached anonymity in their revised paper. Hence, we regrettably cannot recommend accepting this work.
justification_for_why_not_higher_score: See the aforementioned major concern and breach of anonymity.
justification_for_why_not_lower_score: N/A.
Decision: Reject