Reflexion: language agents with verbal reinforcement learning
Authors: Noah Shinn
Federico Cassano
Ashwin Gopinath
Karthik R Narasimhan
Shunyu Yao
Venue: NeurIPS 2023 poster (2023)
Keywords: language model
reasoning
decision making
programming
TL;DR: Reflexion is a framework that reinforces language agents by updating language rather than model weights.

Abstract: Large language models (LLMs) have been increasingly used to interact with external environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents to quickly and efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expensive model fine-tuning. We propose \emph{Reflexion}, a novel framework to reinforce language agents not by updating weights, but instead through linguistic feedback. Concretely, Reflexion agents verbally reflect on task feedback signals, then maintain their own reflective text in an episodic memory buffer to induce better decision-making in subsequent trials. Reflexion is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals, and obtains significant improvements over a baseline agent across diverse tasks (sequential decision-making, coding, language reasoning). For example, Reflexion achieves a 91\% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80\%. We also conduct ablation and analysis studies using different feedback signals, feedback incorporation methods, and agent types, and provide insights into how they affect performance. We release all code, demos, and datasets at \url{https://github.com/noahshinn024/reflexion}.

Reflexion: Language Agents with Verbal Reinforcement Learning
Noah Shinn Northeastern University noahshimo0240mmai.. com
Federico Cassano Northeastern University cassano www.rtheasterrr.nccmomm edu
Ashwin Gopinath MIT agopi aeppi.mi.eee
Karthik Narasimhan
Shunyu Yao
Princeton University karthiln@prinnetto... edu
Princeton University
shunyuy@princeton. edu
Abstract
Large language models (LLMs) have been increasingly used interact with exter- nal environments (e.g., games, compilers, APIs) as goal-driven agents. However, it remains challenging for these language agents quickly and efficiently learn from trial-and-on as traditional reinforcement learning methods require exten- sive training samples and expensive model fine-tuning. We propose Reflexion, a novel framework to reinforce language agents not by updating weights, but in- stead through linguistic feedback. Concretely, Reflexion agents verbally reflect task feedback signals, then maintain their own reflective text in episodic memory buffer to induce better becision-making in subsequent trials. Reflexion is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals, and obtains significant improvements over baseline agent across diverse tasks (sequential recissin-maainn coding, language reasoning). For example, Reflexion achieves a 91% pass@) @1 I accuracy on the HumanEval coding benchmark, surpassing the previ- ous state-of-the-ee GPT-4 that achieves 80%. We also conduct ablation and analysis studies using different feedback signals, feedback incorporation methods, and agent types, and provide insights into how they affect performance. We release all code, demos, and datasets huttpa://github... com http:////oooooaaaaaaim/////eeeee
Introduction
Recent works such ReAct 30], SayCan (. Toolformer 22]. HuggingGPT 23 generative agents [19], and WebGPT 17 have demonstrated the feasibility of autonomous decision-making agents that built top of large language model (LLM) core. These methods use LLMs to generate text and 'actions' that can be used in API calls and executed in an environment. Since they rely massive models with enormous number of parameters, such approaches have been so far limited to using in-context examples as way of teaching the agents, since more traditional optimization schemes like reinforcement learning with gradient descent require substantial amounts of compute and time.
In this paper, propose alternative approach called Refiexion that uses verbal reinforcement to help agents learn from prior failings. Reflexion converts binary or scalar feedback from the environment into verbal feedback in the form of textual summary, which is then added additional context for the LLM agent in the next episode. This self-reflective feedback acts as a 'semantic' gradient signal by providing the agent with concrete direction to improve upon, helping it learn from prior mistakes to perform better on the task. This is akin to how humans iteratively learn to
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

zaking You are in the riddle of zoor.
Programsing Zask: You are given Lis of two
Ressoning
Task:
what profession does John
(a) Task
[..e] Task: clean some pan and put
stzings I...] as opan 'C' oz cieas ')" parsotheses only Lood
Lenchaster and Alan Dean Fostez save 1h comson?
1t in countertop.
I...]
dcf metal, (10t:
Shink: [...] aoellst, jounndiat,
(6)
Action:tabe pani ccom atovaturneri
it Ei.comtt (7)
critic L..., novellat, screenvr-ter "exel comoon is
Trajectory
Obe: Wothinng happansal Lord
22.00 62.00nt 1"11 =e 51.00mee151)
Aation:oleen pan1 with sinkossin!
43 43.00nt 111 Tare]
nemills E5d soneeten
dbs:3othing happans, [oo]
tolata 100
Action: "coveliat, icto
[c]
Evaluation
Rale/IM Neuristic:
self-guoerated
they insornectly sare reltiple professione
(d)
Teael pani atoveoucnee [...]
up
open parentleses 29 equal [.o)
Reflection
paccatheses [xxe]
acoutate.y Loan Estano
pan
(o] Next
Solnn Landhes Alan Dean Raaer have contoo
Trajectory
[...) (bo: You put the pan
countartop 3.
Actica: "nove.lat"
Figure I: Reflexion works on decision-making programming 4.3 and reasoning 4.2 tasks.
accomplish complex tasks in a few-shot manner by reflecting on their previous failures in order to form an improved plan of attack for the next attempt. For example, in figure I a Reflexion agent learns optimize its own behavior to solve decision maaiii,, programming, and reasoning tasks through trial, error, and self-reflection.
Generating useful reflective feedhack is challenging since it requires good understanding of where the model made mistakes (i.e. the credit assignment problem (25)) as well as the ability to generate summary containing actionable insights for improvement. We explore three ways for doing this - simple binary environment feedback, pre-defined heuristics for common failure cases, and self-evaluation such binary classification using LLMs (decision-making) or self-written unit tests (programming). In all implementations, the evaluation signal is amplified to natural language experience summaries which can be stored in long- term memory.
Reflexion has several advantages compared to more traditional RL approaches like policy or value- based learning: 1) it is lightweight and doesn't t require finetuning the LLM, 2) it allows for more nuanced forms of feedback (e.g. targeted changes in actions), compared to scalar or vector rewards that are challenging to perform accurate credit assignment with, 3) it allows for more explicit and interpretable form of episodic memory over prior experiences, and 4) it provides more explicit hints for actions in future episodes. At the same time, does have the disadvantages of relying on the power of the LLM's self-evaluation capabilities (or heuristics) and not having a formal guarantee for success. However, as LLM capabilities improve, we only expect this paradigm get better over time.
We perform experiments on (1) decision-make trajectories, (2) reasoning tasks to test mnowledge-intensiin
tasks to test sequential action choices over long single-step generation improvement, and (3) programming tasks to teach the agent to effectively use external tools such as compilers and interpreters. Across all three types of tasks, we observe Reflexion agents better decision- makers, reasoners, and programmers. More concretely, Reflexion agents improve on decision-making AlfWorld [24] tasks over strong baseline approaches by an absolute 22% in 12 iterative learning steps, and on reasoning questions in HotPotQA (28) by 20%, and Python programming tasks on HvmanEval o by as much as 11%.
To summarize, our contributions are the following:
We propose Reflexion, a new paradigm for 'verbal' reinforcement that parameterizes a policy as an agent's memory encoding paired with choice of LLM parameters.
We explore this emergent property of self-reflection in LLMs and empirically show that self-reflection is extremely useful to learn complex tasks over a handful of trials.
We introduce LeetcodeHardGymm code gedereenetttonn RL gym environment consisting of 40 challenging Leetcode questions ('hard-level')) in 19 programming languages.
We show that Reflexion achieves improvements over strong baselines across several tasks, and achieves ffte--o-thheass results on various code generation benchmarks.

---

Related work reasoning and decision maaing
Approach
Self refine
Hidden
Decision making
Binary reward
Memory
constraints
Self-refine
1S
X
X
X
Beam search 27
X
Reflexion (ours)
Related work on programming
Approach
Test execution
Debugging
Self-generated
Multiple languages
Self-reflectin
Test execution
tests
AlphaCode 14
X
X
x
Code'
X
Sell-debugging 7
/
/
X
CodeRL 12
/
/
X
X
X
Reflexion (ours)
2 Related work
Reasoning and decision-making Self-Refine [.5 employs an iterative framework for self- refinement to autonomously improve generation through self-evaluation.. These self-evaluation and self-improvement steps are conditioned on given task constraints, such as "How can this genera- tion be written in more positive way". Self-Refine is effective but is limited single-generation reasoning tasks. Pryzant et al. [2 performs a similar semantic moompt-writing optimization, but is also limited to angle-generation tasks. Paul et al. 20 fine-tune critic models to provide intermediate feedback within trajectories to improve reasoning responses. Xie et al. 27 use stochastic beam search actions to perform more efficient decision-making search strategy which allows the agent to use foresight advantage due to its self-evaluation component. Yoran et al. (31] and Nair et al. [16] use decider models reason over several generations. Kim al. (TO) use retry pattern a fixed number of steps without an evaluation step. Goodman [9] perform a qualitative evaluation step that proposes optimizations to the previous generation. In this paper, we show that several of these concepts can be enhanced with self-reflection to build a persisting memory of self-reflective experiences which allows an agent identify its own errors and self-suggest lessons learn from its mistakes over time.
Programming Several past and recent works employ variations of test-driven development code debugging practices. AlphaCode [4] evaluates a set of generations on hidden test cases. CodeT (5] uses self- self-generated unit tests that are used score generated function implementations. Self-Debuggin (7] employs debugging component that is used to improve existing implementations given feedback from code execution environment. CodeRL 12 sets the problem in an RL frame- work using an actor-critic setup to debug programs given feedback from an execution environment. AlphaCode, Self- ooff-eebgging and CodeRL are effective fixing less-complex program bugs, but they rely upon ground truth test cases that invalidate pass@) I eligibility, and do not use self-reflection to bridge the gap between error identification and implementation improvement, CodeT does not access hidden test cases but does not implement self-learning step to improve code writing.
3 Reflexion: reinforcement via verbal reflection
We develop modular formulation for Reflexion, utilizing three distinct models: an Actor, denoted as Ma, which generates text and actions; an Evaluator model, represented by M.. that scores the outputs produced by M,; and Self-Reflection model, denoted M,r, which generates verbal reinforcement cues to assist the Actor in self-improvement. We provide detailed description of each of these models and subsequently elucidate their collaborative functioning within the Reflexion framework.
Actor The Actor is built upon a large language model (LLM) that is specifically prompted to generate the necessary text and actions conditioned on the state observations. Analogous traditional policy-based RI. setups, we sample an action or generation, dz, from the current policy at time I, receive an observation from the environment or, We explore various Actor models, including Chain of Thought (26) and ReAct [30]. These diverse generation models allow to explore different aspects

---

Algorithm Reinforcement via self-reflection
Agent
Initialize Actor, Evaluator, Self-Reflection:
Mas Me. Mar
Exterall leedback
(LM)
Initialize policy Th (s). {Maa mem)
Reflectvee
Generate initial trajectory using m
fbodback
toxt
Evaluate To using Mc
Evaluator (LM)
Experience (lang lerm memory)
Generate initial self-reflectin sry using M.r
Set mem < [sro]
Set /
Trajectory (short-term memory)
Actor (LM)
while M, not pass or / < max trials do
Generate T: (on, 00, ........ ais 0il using m
00,...
Evaluate т; using M,
Generate self-reflection sr; using Mar Append ST, to mem
Obs Reward
Environment
Action
Increment
end while return
Figure 2: (a) Diagram of Reflexion. (b) Reflexion reinforcement algorithm
of text and action generation within the Reflexion framework, providing valuable insights into their performance and effectiveness. In addition, we also add memory component mem that provides additional context to this agent. This adaption was inspired by Brooks et al. [), who suggest a policy iteration approach using in-context learning. Details on how this is populated are provided below.
Evaluator The Evaluator component of the Reflexion framework plays crucial role in assessing the quality of the generated outputs produced by the Actor. takes as input generated trajectory and computes reward score that reflects its performance within the given task context. Defining effective value and reward functions that apply semantic spaces is difficult, so we investigate several variants of the Evaluator model. For reasoning tasks, we explore reward functions based on exact match (EM) grading, ensuring that the generated output aligns closely with the expected solution. In decision-makinn makiin tasks, we employ pre-defined heuristic functions that are tailored to specific evaluation criteria. Additionally, we experiment with using a different instantiation of an LLM itself as Evaluator, generating rewards for decision decision-making and programming tasks. This multi-faccted approach to Evaluator design allows us to examine different strategies for scoring generated outputs, offering insights into their effectiveness and suitability across range of tasks.
LLM, plays crucial role in the Reflexion framework by generating verbal self-reflections to provide valuable feedback for future trials. Given sparse reward signal, such binary success status (success/fail), the current trajectory, and its persistent memory mem, the self-reflection model generates nuanced and specific feedback. This feedback, which is more informative than scalar rewards, is then stored in the agent's memory (mem). For instance, in a multi-step decision-making task, when the agent receives a failure signal, infer that specific action a; led to subsequent incorrect actions 0;+1 1 and 0,+2- The agent can then verbally state that it should have taken a different action, d/. which would have resulted in d'y4 and 2/42, and store this experience in its memory. In subsequent trials, the agent can leverage its past experiences to adapt its decision-making approach at time by choosing action a/. This iterative process of trial, error, self-reflection, and persisting memory enables the agent to rapidly improve its decision-making ability in various environments by utilizing informative feedback signals.
Self-reflection The Self-Reflection model instantiated
Memory Core components of the Reflexion process are the notion of short-term and long-term memory. At inference time, the Actor conditions its decisions on short and long-term memory, similar to the way that humans remember fine-grain recent details while also recalling distilled important experiences from long-term memory. In the RI. setup, the trajectory history serves as the short-term memory while outputs from the Reff-Reflection model are stored in long-term memory. These two memory components work together to provide context that is specific but also influenced by lessons

---

learned over several trials, which is key advantage of Reflexion agents over other LLM action choice works.
The Reflexion process Reflexion is formalized as an iterative optimization process in | In the first trial, the Actor produces trajectory to by interacting with the environment. The Evaluator then produces a score ru which is computed as Tz- (T)).). Ta is only a scalar reward for trial t that improves as task-specific performance increases. After the first trial, amplify ro to a feedback form that can be used for improvement by an LLM, the self-Reflection model analyzes the set of {To, ro] to produce a summary STo which is stored in the memory mem. Sr, is a verbal experience feedback for trial I. The Actor, Evaluator, and Self-Reflectiog models work together through trials in a loop until the Evaluator deems 7, to correct. As mentioned in the memory component of Reflexion is crucial to its effectiveness. After each trial I, sT,, is appended mem. In practice, we bound mem by a maximum number of stored experiences, Q (usually set to 1-3) to adhere to max context LLM limitations.
4 Experiments
We evaluate various natural language RL setups on decision-makinn makinng reasoning, and code generation tasks. Specifically, we challenge an agent to perform search- question answering HotPotQA 28], multi-step tasks in common household environments in AlfWorld (24), and code writing tasks in competition-like environments with interpreters and compilers in HumanEval [6), MBPP (2). and Herd,odeHard, a new benchmark. Most notably, Reflexion improves performance over strong baselines by 22% in AlfWorld, 20% in HotPotQA, and 11% on HumanEval.
4.1 Sequential decision making: ALFWorld
AlfWorld is a suite of text-based environments that challenge an agent to solve multi-step tasks in a variety of interactive environments based on TextWorld [8] Following Yao et al. [S0, we run the agent in 134 AlfWorld environments across six different tasks, including finding hidden objects (e.g., finding a spatula in a drawer), moving objects (e.g., moving a knife to the cutting board), and manipulating objects with other objects (e.g.. chilling a tomato in the fridge). We use ReAct (30 as the action generator as Yao et al. (30 has shown success in long trajectory decision- making using explicit intermediate thoughts. AlfWorld tasks naturally require a self-evaluation step as the environment can only signal if task is complete. To achieve fully autonomous behavior, we implement two self-evaluation techniques: natural language classification using an LLM and a hand-wrihh written heuristic. The heuristic is simple: if the agent executes the same action and receives the same response for more than cycles, or if the number of actions taken in the current environment exceeds 30 (inefficient planning), we self-reflect. In the baseline runs, if self-refiection is suggested, we skip the self-reflection process, reset the environment, and start a new trial. In the Reflexion runs, the agent uses self-reflection to find its mistake, update its memory, reset the environment, and start new trial. To avoid very long prompt windows that may exceed the maximum limit, we truncate the agent's memory to the last 3 self-reflections (experiences).
To avoid syntactic errors, we provide two specifcc few-shot trajectories to the agent. We use the same few-shot trajectory examples as Yao et al. [30] with GPT-3 for the LLM. AlfWorld tasks, ReAct few-shot prompts, and Reflexion examples are included in the appendix.
Results ReAct + Reflexion significantly outperforms ReAct by completing 130 out of 134 tasks using the simple heuristic to detect hallucinations and inefficient planning. Further, ReAct Reflexion learns to solve additional tasks by learning in 12 consecutive trials. In the ReAct-only approach, we see that performance increase halts between trials 6 and
Analysis A common error in baseline failed AlfWorld trajectories is when an agent thinks that it has possession of an item but does not actually have the item. The agent proceeds to execute several actions in a long trajectory and is not able to backtrack its actions to find the mistake. Reflexion eliminates almost all of these cases by using self-reflection to distill long, failed trajectories into relevant experiences that can are used as "self-hints" in the future. There are two main cases in which long-term memory helps an agent in AlfWorld: 1) An early mistake a long trajectory can be easily identified. The agent can suggest new action choice
new long-term plan. 2) There are too
5

---

(a) ALFWorld Success Rate
ALFWorld Success Rate
1.0
0.5
Proportion of Salved Environmen nts
ReAct only
ReAct only hallucination
Reflexton [Heuristic]
ReAct Inefficient planning
Reflexton (GPT)
Environments
ReAct Reflexion hallucination
0.9
0.4
RoAct Reflexton Inefficient planning
0.8
0.3
0.7
0.2
0.6
0.1
0.5
0.0
2
8
10
8
10
Trial Number
Trial Number
Figure 3: (a) AlfWorld performance across 134 tasks showing cumulative proportions of solved tasks using self-evaluation techniques of (Heuristic) and (GPT) for binary classification, (b) Classification of AlfWorld trajectories by reason of failure.
many surfaces/containers to check for an item. The agent can exploit its experience memory over suggests that the learning process
several trials thoroughly search room. In 3 the learning occurs over several experiences, meaning that the agent is successfully balancing cases and shown in the immediate spike in the improvement between the first two trials, then steady increase over the next Il trials to a near-perfect performance. On the other hand, 3 shows a ReAct-only agent converging hallucination rate of 22% with signs of long-term recovery.
4.2 Reasoning: HotpotQA
28 is Wikipedia- basedd dataset with 113k question -nd answer pairs that challenge agents to parse content and reason over several supporting documents. To test improvement in reasoning only ability, we implement Reflexion Q Q- A and Q, Cgt + A implementations, where Q is the question, Cgt is the ground truth context from the dataset, and A is the final answer. Since CoT' is not multi-step decision decision-making technique, we give Cyt. to the agent that we can isolate the reasoning behavior over large sections of the provided text. To test holistic question and answering ability, which requires reasoning and action choice, we implement Reflexion + ReAct (30] agent that retrieve relevant context using Wikipedia API and infer answers using step-by-step explicit thinking. For CoT implementations, we 6-shot prompting; for ReAct, prompting. All examples can be found in the appendix.
HotPotQA
of-iin-o--Thouhh (CoT) 226 for step-by-step
2-shot prompting, and for self-reflection
use 2-shot
Robustly evaluating natural language answers is long-standing problem in NLP. Therefore, between trials, we use exact match answer grading using the environment to give a binary success signal to the agent. After each trial, the self-reflection loop is employed to amplify the binary signal, similar to the decision- makiin setup 4.1 in AlfWorld with memory size of experiences.
0.8 (a) HotPotQA Success Rate
1.0
(b) HotPotQA CoT (GT)
HotPotQA Episodic Memory
CoT only
CoT (GT) only
Taaks
1.0
ReAct
CoT (GT) Reflexion
0.6
0.9
Reflexton
Proportion
0.7
Propertion
0.2
0.4
0.6
2 Trial Number
Trial Number
Trial Number
Figure 4: Chain-of- -hain-hf-Thought (CoT) and ReAct. Reflexion improves search, information retrieval,
and reasoning capabilities on 100 HotPotQA questions. (a) Reflexion ReAct Reflexion CoT (b)
Reflexion CoT (GT) for reasoning only (c) Reflexion vs episodic memory ablation.

---

Results Reflexion outperforms all baseline approaches by significant margins over several learning steps. Furthermore, ReAct-only, CoT-only, and CoT (GT)-only implementations fail to probabilisti- cally improve on any tasks, meaning that no failed tasks from the first trial from any of the baseline approaches were able to he solved in subsequent trials using a temperature of 0.7 In the Reflexion runs, we allowed the agent to gather experience and retry on failed tasks until it produced 3 consecutive failed attempts on the particular task. Naturally, the CoT (GT) achieved higher accuracy scores as was given access to the ground truth context of the question. Still, the CoT (GT) agent is unable to correctly infer the correct answer for 39% of the questions, but Reflexion helps the agent to correct its mistakes without access to the ground truth answer to improve its accuracy by 14%.
Analysis We perform an ablation experiment to isolate the advantage of the self-reflcctive step for reasoning using CoT (GT) the baseline approach 4 Recall that CoT (GT) uses Chain-of- Thouught reasoning with provided ground truth context, which tests reasoning ability over long contexts. Next, we add element of episodic memory (EPM) by including the most recent trajectory. For the Reflexion agent, we implement the standard self-reflection step as a final pass. Intuitively, we test the agent is iteratively learning more effectively by using verbal explanation using language written in the first person. 4 shows that self-reflection improves learning by an 8% absolute boost over the episodic memory learning advantage. This result supports the argument that refinement-only approaches are not as effective as self-reflection-gune -eudded refinement approaches.
4.3 Programming
We evaluate the baseline and Reflexinn approaches Python and Rust code writing on MBPP (2) HumanEval [6,, and Leetcode Harddyym our new dataset. MBPP and HumanEval measure function body generation accuracy given natural language descriptions. We a benchmark language compiler. MultiPL-E [4], to translate subsets of HumanEval and MBPP to the Rust language. MultiPL- E is a collection of small compilers that can be used to translate Python benchmark questions to 18 other languages. We include experiments for Rust code generation to demonstrate that Reflexion implementations for code generation are language-agnostic and can be used for interpreted and compiled languages. Lastly, we introduce a new benchmark, LeetcodeHardGGym, which is an interactive programming gym that contains 40 Leetcode hard-rated questions that have been released after October 8, 2022, which is the pre-training cutoff date of GPT-4 [8.
The task of programming presents unique opportunity such as self -eneerateed unit test suites. Thus, our Reflexion-based programming task implementation is Chain-of-Thouue prompting (26 to produce diverse, extensive tests with corresponding natural language descriptions. Then, we filter for syntactically valid test statements by attempting to construct valid abstract syntax tree (AST) for each proposed test. Finally, we sample n tests from the collection of generated unit tests to produce test suite T, denoted as (to, f1,...,
more grounded self-evaluation practices
eligible for pass@) @1 accuracy reporting. To generate test suite,
tx J. We set to a maximum of 6 unit tests. Aside from the unit test suite component, the setup for the learning loop for a Reflexion programming agent identical the reasoning and decision-mall makinng agents with max memory limit of I experience.
Benchmark + Language
Prev SOTA Pass @1
SOTA Pass @1
Reflexion Pass@)
HumanEval (PY)
65.8 (Code'T 5 + GPT-3355
80.1 (GPT-4)
91.0
HvaanEval (RS)
60.0 (GPT-4)
68.0
MBPP (PY)
67.7 7 (CodeT 5 + Codex 66)
80.1 (GPT-4)
77.1
MBPP (RS)
70.9 (GPT-4)
75.4
Leetcode Hard (PY)
7.5 (GPT-4)
15.0
Table 1: Pass 1 accuracy for various model-strratey--aaggaa combinations. The base strategy is a
single code generation sample. All iirructioo-baseee models follow zero-shot code generation.
Results Reflexion outperforms all baseline accuracies and sets new state-of-the-art standards on all benchmarks for Python and Rust except for MBPP Python [ performance of Reflexion on MBPP Python.
We further investigate the inferior
Analysis We acknowledge that self-reflecting onde-generation agents are bound to their ability to write diverse, comprehensive tests. Therefore, in the case in which the model generates a flaky test
7

---

Benchmark + Language
Base
Reflexion
TP
FN
FP
TN
HumanEval (PY)
0.80
0.91
0.99
0.40
0.01
0.60
MBPP (PY)
0.80
0.77
0.84
0.59
0.16
0.41
Eval (RS)
0.60
0.68
0.87
0.37
0.13
0.63
MBPP (RS)
0.71
0.75
0.84
0.51
0.16
0.49
Table 2: Overall accuracy and test generation performance for HumanEval and MBPP. For Rust, HumanEval is the hardest 50 problems from HumanEval Python translated to Rust with MultiPL.-E
[]. TP: unit tests pass, solution pass; FN: unit tests fail, solution pass; FP: unit tests pass, solution
fail; TN: unit tests fail, solution fail.
suite, it possible that all tests pass on an incorrect solution and lead a false positive label on code completion [II]. On the other hand, if the model produces an incorrectly written test suite, it possible for some of the tests to fail on a correct solution, leading to a self-reflection generation that is conditioned on a false negative code completion. Given the implementation of Reflexion, false negatives are preferred over false positives as the agent may be able to use self-reflection to identify the incorrect test(s) and prompt itself to keep the original code completion intact. On the other hand, an invalid test suite returns a false positive completion (all internal test cases pass but the implementation is incorrect), the agent will prematurely report invalid submission. In 2 various conditions are measured to analyze performance beyond pass @ accuracy. Previously, we displayed the inferior performance of Reflexion to the baseline GPT-4 MBPP Python. In 2 observe a notable discrepancy between the false positive labels produced by internal test execution, P(not pass@1 generation correct tests pass). That is, the probability that submission will fail given that it passes all unit tests. For HumanEval and MBPP Python, the baseline pass@) accuracies are relatively similar, 82% and 80%, respectively. However, the false positive test execution rate for MBPP Python is 16.3% while the rate for HumanEval Python is a mere 1.4%, leading to 91% overall accuracy
Approach
Test Generation
Self-reflection
Pass@1 1 (Ace)
Base model
False
False
0.60
Test generation omission
False
True
0.52
Sclf-reflection omission Reflexion
True
False
0.60
True
True
0.68
Table 3: Pass accuracy for various compromised approaches on the Reflexion approach using GPT-4 as the base model on HumanEval Rust 50 hardest problems
Ablation study We test the composite approach of Reflexion for test generation and self- reffrreflettin cooperation on subset of the 50 hardest HumanEval Rust problems. Our Rust compiler environment provides verbose error logs and helpful debugging hints, therefore serving as good playground for compromised approaches. First, we omit internal test generation and execution steps, which test the agent to self-reflect without guidance from current implementations. 3 shows an inferior 52% vs 60% (baseline) accuracy, which suggests that the agent is unable to determine if the current implementation is correct without unit tests. Therefore, the agent must participate in all iterations of the run without the option to return early, performing harmful edits to the implementation.
Next, we test self-reflection contribution by omitting the natural language explanation step following failed unit test suite evaluations. Intuitively, this challenges the agent to combine the tasks of error identification and implementation improvement across all failed unit tests. Interestingly, the compromised agent does not improve performance over the baseline run. We observe that the test generation and code compilation steps are able catch syntax and logic errors, but the implementation fixes do not reflect these indications. These empirical results suggest that several recent works that propose blind trial and error debugging techniques without self-reflection are ineffective on harder tasks such as writing complex programs in Rust.

---

Limitations
At its core, Reflexion is an optimization technique that uses natural language to do policy optimization. Policy optimization is a powerful approach to improve action choice through experience, but it may still succumb to non-optimal local minima solutions. In this study, we limit long-term memory to sliding window with maximum capacity, but we encourage future work extend the memory component of Reflexion with more advanced structures such as vector embedding databases or traditional SQL databases. Specific code generation, there many practical limitations to test- driven development in specifying accurate input-ouiput mappings such as non -etermermiiistic generator functions, impure functions that interact with APIs, functions that vary output according to hardware specifications, or functions that invoke parallel or concurrent behavior that may be difficult to predict.
6 Broader impact
Large language models are increasingly used to interact with external environments (e.g. the Internet, software, robotics, etc.) and humans. Our work has the potential of reinforcing and empowering these agents toward greater automation and work efficiency, but it also amplifies the risks when these agents were put into misuse. We believe that this direction of research will need more effort in safety and ethical considerations.
On the other hand, reinforcement learning has suffered from its black-box policy and optimiza- tion setups in which interpretability and alignment have been challenging. Our proposed "verbal" reinforcement learning might address some of the issues and turn autonomous agents more inter- pretable and diagnosable. For example, in the case of tool-usage that may be too hard for humans to understand, self-reflections could be monitored to ensure proper intent before using the tool.
7 Conclusion
In this work, we present Reflexion, an approach that leverages verbal reinforcement to teach agents learn from past mistakes. We empirically show that Reflexion agents significantly outperform currently widely-used decision- maasing approaches by utilizing self-reflection. In future work, Reflexion could be used employ more advanced techniques that have been thoroughly studied in traditional RL settings, such as value learning in natural language off-policy exploration techniques.
8
Reproducibility
We highly advise others to use isolated execution environments when running autonomous code writing experiments as the generated code not validated before execution.
References
[1] Ahn, M., Brohan, A., Brown, N., Chebotar, Y., Cortes, O., David, B., Finn, C., Gopalakrishnan, K., Hausman, K., Herzog, A., et al. (2022). Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691.
[2] Austin, J., Odena, A., Nye, M.. Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., al. (2021). Program synthesis with large language models. arXiv preprint 1oXiv:2/080077::
[3] Brooks, E., Walls, L., Lewis, R. I.,, and Singh, S. (2022). In-context policy iteration. arXiv preprint 2r.iv:22
[4] Cassano, F., Gouwar, J., Nguyen, D., Nguyen, S., Chipps-Costinn L., Pinekney, D.,, Yee, M.-H., Zi. Y., Anderson, C. J., Feldman, M. Q., Guha, A., Greenberg, M., and Jangda, A. (2022). Multipl-c: A scalable and extensible approach benchmarking neural code generation.
[5] Chen, B., Zhang, F., Nguyen, A., 7an, D., Lin, Z.,, Lou, J.-G., and Chen, (2022). Codet: Code generation with generated tests. arXiv preprint

---

[6] Chen, M., Tworek, J., Jun, H., Yuan, Q. Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., al. (2021). Evaluating large language models trained on code. arXiv preprint arXiv:2
[7] Chen, X., Lin, M., Schärli, N., and Zhou, D. (2023). Teaching large language models to self-debug. arXiv preprint arXiv:2304.05128
[8] Côté, M.-A., Kádár, A., Yuan, X., Kybartas, B., Barnes, T., Fine, E., Moore, J., Hausknecht, M., El Asri, L., Adada, M., al. (2019). Textworld: A learning enviromment for text-based games. In Computer Games: 7th Workshop, CGW 2018, Held in Conjunction with the 27th International Conference Artificial Intelligence, IJCAI 2018, Stockholm, Sweden, July 13, 2018, Revised Selected Papers pages 41-75. Springer.
[9] Goodman, N. (2023). Meta-prompt: A simple self-improving language agent. noahgood- man.substack.com
[10] Kim, G., Baldi, P., and McAleer, S. (2023). Language models solve computer tasks. arXiv preprint arXiv:2303. 17491.
[11] Lam, W., Winter, S., Wei, A., Xie, T., Marinov, D., and Bell, J. (2020). A large-scale longitudinal study of flaky tests. Proc. ACM Program. Lang., 4(OOPSLA).
[12] Le, H., Wang, Y., Gotmare, D., Savarese, S., and Hoi, S. C. H. (2022). Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances in Neural Information Processing Systems, 35:21314 21328.
[13] Li, R., Allal, L. B., Zi, Y., Muennighoff, N., Kocetkov, D., Mou, C., Marone, M., Akiki, C.,, Li, Chim, J., al. (2023). Starcoder: may the source be with you! arXiv preprint arXiv:2305.00 06161.
[14] Li, Y., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago, A., al. (2022). Competition-leve code generation with alphacode. Science, 378(6624):1092-11
[15] Madaan, A., Tandon, N., Gupta, P., Hallinan, S., Gao, L., Wiegreffe, S., Alon, U., Dziri, N., Prabhumoye, S., Yang. Y., et al. (2023). Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303. 17651.
[16] Nair, V., Schumacher, E., Tso, G., and Kannan, A. (2023). Dera: Enhancing large language model completions with Stalog-enablee resolving agents. arXiv preprint arXiv:2303.17077 17071.
[17] Nakano, R., Hilton, J., Balaji. S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W., et al. (2021). Webgpt: Browser arXiv preprint 22.xv:2112209332
browser-assisted assisted
with human feedback.
[18] OpenAl (2023). Gpt-4 technical report. ArXiv.
[19] Park, J. S., OrBrien, J. C., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. (2023). Generative agents: Interactive simulacra of human behavior. arXiv preprini arXiv:2304.03442
[20] Paul, D., Ismayilzada, M., Peyrard, M., Borges, B., Bosselut, A., West, R., and Faltings, B. (2023). Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304.01900
[21] Pryzant, R., Iter, D., Li. J., Lee, Y. T., Zhu, C., and Zeng, M. (2023). Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint XXXv:2305.03495
[22] Schick, T., Dwivedi- Yu, J., Dessi, R., Raileanu, R., Lomeli. M., Zettlemoyer, L., Cancedda, N., and Scialom, T. (2023). Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.0476)
[23] Shen, Y., Song, K., Tan, X., Li, D., Lu, W., and Zhuang, Y. (2023). Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580
10

---

[24] Shridhar, M., Yuan, X., Côté, M.-A., Bisk, Y., Trischler, A., and Hausknecht, M. (2021). ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings of the International Conference on Learning Representations (ICLR).
[25] Sutton, R. S. and Barto, A. G. (2018). Reinforcement Learning: An Introduction. The MIT Press, second edition.
[26] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. (2022). Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 1tt.::2201.119000 1/903.
[27] Xie. Y., Kawaguchi, K., Zhao, Y., Zhao, X., Kan, M.-Y., He, and Xie, Q. (2023). Decomposi- tion enhances reasoning via self-evaluation guided decoding. arXiv preprint 1rXiv:2305.00633
[28] Yang, Z., Qi. P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., and Manning, C. D. (2018). HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP).
[29] Yao, S., Chen, H., Yang. J., and Narasimhan, K. (preprint). Webshop: Towards scalable real-world web interaction with grounded language agents. ArXiv.
[30] Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. (2023). ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
[31] Yoran, O., Wolfson, T., Bogin, B., Katz, U.. Deutch, D., and Berant, J. (2023). Answering questions by meta-reasoning over multiple chains of thought. arXiv preprint arXiv:2304.1300 13007

---

A Evaluation with additional models
We further investigated the applicability of trial-and-error problem-solving with models of various strengths. We found that the ability to specify self-corrections is an emergent quality of stronger, larger models.
Approach Pass@1 accuracy (avg over trials)
Pass@ accuracy (std)
Baseline
0.26
0.00481
Reflexion
0.26
0.00305
Table @ accuracy on HumanEval Python using starchat-beta [13].
Model
Baseline accuracy
Reflexion accuracy
CoT (GT) + text-davinci-003
0.60
0.77
CoT (GT) + gpt-3.5-turbo
0.57
0.71
CoT (GT) + gpt-4
0.68
0.80
ReAct + text-davinci-003
0.30
0.55
ReAct gpt-3.5-turbo
0.26
0..88
ReAct + gpt-4
0.39
0.51
Table 5: Pass @ accuracy on 100 HotPotQA using various models.
12

---

B Decision-making
Environment: are the middle of a room. Looking quickly around you, you see bed a desk desk 1, drawer 6, drawer 5, drawer 4, drawer 3, a drawer 2, a drawer 1, a garbagecan 1, laundryhamper 1, safe a shelf 6, shelf a shelf 4, shelf shelf 2, and shelf 1.
Task: examine the mug with the desklamp.
Trial #1
think: To solve the task, need find and take mug, then find and use a desklamp.
> think: First need to find mug. A mug is more likely to appear in drawer (1-6), desk (1-2), shelf (1-6), garbagecan (1), laundryhamper (1). can check one by one, starting with drawer 1.
go to drawer
The drawer is closed.
> open drawer
You open the drawer 6. The drawer 6 is open. In it, you keychain 2.
go to desk
On the desk I, you see a creditcard a desklamp I, a laptop 2, a mug a pen I, and pencil 1.
think: Now find mug (I), Next, need to take it.
take mug from desk
You pick up the mug from the desk
> think: Now take mug (1). Next, need to find a desklamp. A desklamp is more likely to appear
in desk (1-2), sidetable (1-2), shelf (1-6), bed (I), drawer (1-6). can check one by one, starting with
desk 1.
go to desk
Nothing happens.
> go to desk
On the desk 2, you alarmelock 1, bowl 1, mug pencil 3, and pencil 2.
think: Now find a desklamp (1). Next, need to use it.
use desklamp 1
Nothing happens.
use desklamp
Nothing happens.
Status: Fail
Reflection: In this environment, my plan was to find mug then find and use a desklamp. However, the task says to examine the mug with the desklamp. should have looked for the desklamp first, then looked for the mug. I noticed that the desklamp was found on desk I. In the next trial, will go to desk 1, find the lamp, then look for the mug and examine with the desklamp.
Trial #2
go to desk
On the desk I, you see a creditcard 3, a desklamp I, a laptop 2, a mug 1, a pen 1, and pencil 1. think: To solve the task, need to find and take mug, then find and use a desklamp.
take mug from desk
You pick up the mug from the desk 1.
think:
To solve the task, I need find and take mug, then find and use desklamp.
use desklamp 1
You turn on the desklamp
Status: Success
Figure 5: [Top] An AlfWorld trajectory which the agent failed due to inefficient planning. In the reflection, the agent recognizes that it should have looked for the desklamp then the mug, not the mug then the desklamp. [Bottom] The agent is able correct reasoning trace and execute a sequence of actions in a concise manner.
13

---

B.1 WebShop Limitation
1n we briefly state that Reflexion struggles overcome local minima choices that require extremely creative behavior to escape. We observe this shortcoming in experiment on WebShop [29] WebShop is web-based problem-solving benchmark that tests agents to navigate an e-commerce website to locate and purchase products given requests from clients. We test two-shot ReAct + Reflexion agent in 100 environments. However, after only four trials, we terminate the runs as the agent does not show signs of improvement Further, the agent does not generate helpful, intuitive self-reflcctions after failed attempts. We conclude that Reflexion unable to solve tasks that require significant amount of diversity and exploration. In AlfWorld, the agent is able adequately explore new environments because the permissible actions can be seen in the observations. In HotPotQA, the agent faces similar WebShop search query task but is more successful the search space for Wikipedia articles is more diverse and requires less precise search queries. A common problem for e-commerce search engines is properly handling ambiguity in natural language search interpretations. Thus, WebShop presents a task that requires very diverse and unique behavior from a Reflexion agent.
WebShop Success Rate
0.50
Proportion of Solved Environments
ReAct only
0.45
ReAct + Reflexion
0.40
0.35
0.30
0.25
0.20
0.15
0.10
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Trial Number
Figure 6: Reflexion React performance WebShop across 100 customer shopping requests. ReAct + Reflexion fails to significantly outperform ReAct.
C Programming
Programming LLM calls require strict instructions to produce function bodies only, due to the extensive dialogue training of the LLMs. A few programming examples are reported below with instructions highlighted in blue and templates. See the full implementation at witttts: tittps://githb)
C.1 Programming function implementation example (HumanEval Python)
Sample function signature:
def minSubArraySun (nums):
Given an array of integers nums, find the múnimum sum of any non non-eme empty sub array of nums
Example minSubArraySun ([2, 3, 4, 1, 2, 4)) == 1

---

m minSubArraySun ([-1, -2, -3]) == -6
# n "
Sample function body implementation:
min_sum = float ('inf') for range (len (nums)):
current_sum = 0
for j in range (i, len (nums)) current_sum += nums [j] if current_ sum min_ sum min_sum current_sun sum
return min_sum
C.2 Reflexion Actor instruction
You are a Python writing assistant. You will be given your previous implementation of function, series of unit tests results, and your self-reflection on your previous implementation. Apply the necessary changes below by responding only with the improved body of the function. Do not include the signature in your response. The first line of your response should have 4 spaces of indentation so that it fits syntactically with the user provided signature. You will be given a few examples by the user.
Reflexion Actor generations follow the form: (Instruction)
(Function implementation)
(Unit test feedback)
(Self-reflection)
(Instruction for next function implmentation)
Full examples are shown in httppp//gggthao.oooo//aaaaaaaaap
C.3 Reflexion Self-reflection instruction and example
You are a Python writing assistant. You will be given your previous implementation of function, series of unit tests results, and your self-reflcction on your previous implementation. Apply the necessary changes below by responding only with the improved body of the function. Do not include the signature in your response. The first line of your response should have 4 spaces of indentation so that it fits syntactically with the user provided signature. You will be given a few examples by the user. Reflexion Self-Reflcction generations follow the form: (Instruction)
(Function implementation)
(Unit test feedback)
C.4 Reflexion programming no Sell-Reflection ablation example
Reflexion no Self-Reflection ablation Actor generations follow the form: (Instruction)
(Function implementation)
(Unit test feedback) refeetto))))))
(Instruction for next function implmentation)
C.5 Reflexion programming test generation ablation example
Reflexion no test generation ablation Actor generations follow the form: (Instruction)
15

---

(Function implementation)
(Unit-tes, test feedback)
(Self-reflection)
(Instruction for next function implmentation)
16

---

D Reasoning
D.1 Full example
Trial #1: Question: Grown-Ups starred the actor who was best known for which role on "'Allo ?
Trial #2: Question: Grown-Ups starred the actor who was best known for which role on "" "'Allo "Allo!"?
Thought 1: need to search Grown-Ups and "Allo "Allo!", find the actor who starred in Grown-Ups, then find the role he was best known for in "" Allo 'Allo!".
Thought I need to search the actor who was best known for role on "" "lllo 'Allo!" and find out what role they were best known for.
Action I: Search[Grown-Ups:
Action I: Search[Grown-Ups)
Observation 1: Grown-Ups
1980 British BBC television film devised and directed by Mike Leigh. It stars Lesley Manville, Philip Davis, Brenda Blethyn, Janine Duvitski, Lindsay Duncan and Sam Kelly. It was edited by Robin Sales and produced by Louis Marks for the BBC, and originally shown on BBC 2 on 28 November 1980.
Observation I: Grown-Ups is a 1980 British BBC television film devised and directed by Mike Leigh. It stars Lesley Manville, Philipp Davis, Brenda Blethyn, Jamine Duvitski, Lindsay Duncan and Sam Kelly. It was edited by Robin Sales and produced by Louis Marks for the BBC, and originally shown on BBC 2 28 November 1980.
Thought 2:
The paragraph does not mention the actor who was best known for a role on " AAllo Alloo". I need to search the actor's 's name instead. Action 2: Search (Sam Kelly]
Thought 2: Grown-Ups starred Lesley Manville, Philip Davis, Brenda Blethyn, Janine Duvitski, Lindsay Duncan and Sam Kelly. need to search "Allo "Allo!" and find which actor was best known for which role.
Observation 2: Roger Michael Kelly (19 Decem- ber 1943 14 June 2014), known by the stage name Sam Kelly, was an English actor who ap- peared in film, television, radio and theatre. He is best known for his roles as Captain Hans Geer- ing in Allo AAllo!, Warren in Porridge, Sam in On the Up, and Ted Liversidge in Barbara. Thought 3: Sam Kelly is best known for his role as Captain Hans Geering in " Allo 'Allo!", so the answer is Captain Hans Geering.
Action 2: Search("" "" Allo "]
Observation 2: Could not find Allo "]lo!"]. Similar: Allo 'Allo",, "List of 'Allo 'Allo! characters", 'Gorden Kaye', "Alloo" 'Allo! (series 2)", Al1ó, al16', 'Francesca Gonshaw', "'Allo 'Allo! (series 4)", "Kim Hartman', "" "Allo" 'Allo! (series 6)", 'Jeremy Lloyd']
Thought 3: To find the actor who was best known for which role on AAllo "Allo!", I can search Gorden Kaye.
Action 3: Finish (Captain Hans Geering] Observation 3: Answer is CORRECT
Action 3: Search[Gorden Kaye]
Observation 3: Gordon Irving Kaye (7 April 1941 23 January 2017), known profession- ally as Gorden Kaye, was an English actor, best known for playing womanising café owner René Artois in the television comedy series 'Allo "Allo!
Thought 4: Gorden Kaye was best known for playing womanising café owner René Artois in the television comedy series Allo 'Allo!.. So the answer is Rene Artois.
Action 4: Finish[Rens Artois]
Observation 4: Answer is INCORRECT
Reflection: searched the wrong title for the show, "'Allo Allo!", which resulted in no results.
should have searched the show's 's main character, Gorden Kaye, to find the role he was best known for in the show.
Figure 7: Two HotPotQA trials within the same environment and task. The Reflexion + ReAct agent uses self-reflection to determine a better search method for the next trial.
17

---

D.2 Chain-of-Thought + Reflexion
Trial #1: Question: What profession does John Lanchester and Alan Dean Foster have in com- mon?
Trial #2: Question: What profession does John Lanchester and Alan Dean Foster have in com- mon?
Thought I: Let's think step by step. John Lanch- ester is novelist, journalist, and critic. Alan Dean Foster a novelist and screenwriter. So the profession John Lanchester and Alan Dean Foster have in common is novelist and screen- writer.
Thought I: Let's think step by step. John Lanch- ester is novelist, journalist, and financial jour- nalist. Alan Dean Foster is a novelist and screen- writer. So the profession John Lanchester and Alan Dean Foster have in common novelist. Action 1: Finish[novelis
Action 1: Finish| novelist and screenwriter
Observation 1: Answer is CORRECT
Observation I: Answer is INCORRECT
Reflection: My reasoning for the profession John Lanchester and Alan Dean Foster have in common failed because incorrectly assumed that they both had the same profession. In the future, when attempting this question, should focus on better researching the individual backgrounds of both authors to ensure that am accurately identifying their professions. Additionally, should consider the possibility that they may have multiple professions common.
D.3 HotPotQA Chain-of-Thought (GT) + Reflexion
Context: The Battle of White Plains was battle in the New York and New Jersey campaign of the American Revolutionary War fought on October 28, 1776, near White Plains, New York. Following the retreat...
Trial #1: Question: What was a series of battles during the Revolutionary War, for control of New
Trial #2: Question: What was a series of battles during the Revolutionary War, for control of New York City and the state of New Jersey, fought on October 28, 1776 near White Plains, New York? Thought 1: The question is asking for series of battles during the Revolutionary War, for control of New York City and the state of New Jersey, fought on October 28, 1776 near White Plains, New York. Based on the context, know that this series of battles was the New York and New Jersey campaign.
York City and the state of New Jersey, fought on
October 28, 1776 near White Plains, New York? Thought 1: The question is asking about ries of battles during the Revolutionary War for control of New York City and the state of New Jersey, fought on October 28, 1776 near White Plains, New York. From the context, I know that this was the Battle of White Plains. Action I: Binish of White Plains] Observation 1: Answer is INCORRECT
Action 1: Finish (Th New York and New Jersey
campaign
Observation Answer CORRECT
Reflection:
Upon reflecting the incorrect answer provided, I realize that may not have provided
enough context to accurately answer the question. The question asked for series of battles, but I only provided the name of one battle. To better answer this question in the future, I will make sure to provide more context, such as the name of the campaign, which was the New York and New Jersey campaign, and the fact that it was series of battles. Additionally, will make sure to include the dates of the battles, as well as the locations, to ensure that my answer is comprehensive and accurate.
18

---

D.4 HotPotQA episodic memory (EPM) ablation prompts
D.4.1 (EPM) Chain-of-Thought Reflexion
Trial #1: Question: Which Jonny Craig and Pete Doherty has been a member of more bands
Trial #2: Question: Which Jonny Craig and Pete Doherty has been member of more bands
Let's think step by step. Jonny Craig has been member of six bands, including Emarosa, Dance Gavin Dance, and Slaves. Pete Doherty has been a member of seven bands, in- cluding The Libertines, Babyshambles, and The Puta Madres. So Pete Doherty has been mem- ber of more bands.
Thought I:
Thought I: Let's think step by step. will re- search the past and current bands of both musi- cians to ensure that I am accurately comparing their band memberships. After researching, I have found that Jonny Craig has been mem- ber of seven bands, including Emarosa, Dance Gavin Dance, and Slaves. Pete Doherty has been a member of seven bands, including The Lib- ertines, Babyshambles, and The Puta Madres. So Jonny Craig has been a member of more bands. Action 2: Finish[Jonny Craig]
Action 1: Finish[Pete Doherty]
Observation I: Answer is INCORRECT
Observation 2: Answer is CORRECT
Reflection: My reasoning for which musician has been member of more bands failed because I did not take into account that Jonny Craig has been member of more bands in the past, even though Pete Doherty is currently a member of more bands. In the future, when attempting this question, should focus on researching the past and current bands of both musicians to ensure that am accurately comparing their band memberships.
D.4.2 (EPM) Chain-of-Though (GT) + Reflexion
Context: Hari Bahadur Basnet is Nepalese politician. He is the head of the Foreign Relations Department of the Rastriya Janashakti Party. Basnet holds M.Sc. in Engineering.
A Master of Science (Latin: "Magister Scientiae" abbreviated MS, M.S., MSc, M.Sc., MSci, M.Sci., ScM, Sc.M., SciM or Sci.M.) is a master's s degree in the field of science...
Trial #1: Question: The head of the Foreign
Trial #2: Question: The head of the Foreign Re- lations Department of the Rastriya Janashakti Party holds a degree that can be abbreviated MS, M.S., or ScM, in what field?
Relations Department of the Rastriya Janashakti
Party holds a degree that can be abbreviated MS,
M.S., or ScM, in what field?
(Previous trajectory)
(Previous trajectory)
Thought 1: The question is asking what field of study the head of the Foreign Relations Department of the Rastriya Janashakti Party holds a degree in. From the context, I know that the degree is Master of Science degree, which is typically granted for studies in sciences, engineering, and medicine.
Thought 1: The question is asking what field of study the head of the Foreign Relations De- partment of the Rastriya Janashakti Party holds a degree in. From the context, I know that the degree is Master of Science degree, which is typically granted for studies in sciences, engi- neering, and medicine. However, the question is asking for the field of study, not the degree. Based on the context, I know that Hari Bahadur Basnet holds a M.Sc. in Engineering. Action 1: Finish [nniih/Enginerrin]]]
Action I: Finish[Sciences, Engineering, and Medicine
Observation 1: Answer is INCORRECT
Observation 1: Answer CORRECT For the first trial, misunderstood the question. I thought the question was asking for category of degrees, rather than specific degree. learned from my mistake and in the second trial, focused more closely the question understand that it was asking for specific field of study.
Reflection:
19


Reviews:
[{"review_id": "Mx2giYTkaj", "reviewer": "Reviewer_k1Ks", "summary": "The paper introduces Reflexion, a framework for in-context reasoning and decision-making for LMs. It allows the LM to iteratively attempt to solve a task and then reason about the results of its attempts and plan improved courses of action. The authors evaluate Reflexion on a diverse array of tasks (navigating and acting in text environments, question answering, code generation) and report impressive results, in some cases establishing new state-of-the-art. They also compare Reflexion to useful baselines and conduct instructive ablations.", "questions": "1. It's not entirely clear to me which parts of the algorithm happen within the \"main\" context window of Reflexion and which are encapsulated into separate, local context windows that are then discarded (my understanding is that this is the case, e.g., for unit test generation). I'd appreciate if the authors could make that more clear.\n2. How does Reflexion scale (down) with model size? Are GPT-4-level capabilities necessary for it to yield improvements or does it also work with `gpt3.5-turbo` and `text-davinci-003`?\n3. How was the error analysis in section 4.1 conducted, i.e. who classified failures as being due hallucination or due to insufficient planning?", "limitations": "Certain limitations and societal impacts are discussed. I think it could be good to complement that with a discussion of risks associated with making LMs prompted to be agents act autonomously by allowing them to set their own subgoals and to plan with rich world models (e.g., Perez et al., 2022; Ngo et al., 2023). This could have unknown failure modes if those agents are given unrestricted Internet access and are not supervised.\n\nPerez et al., 2022. Discovering Language Model Behaviors with Model-Written Evaluations\n\nNgo et al., 2023. The alignment problem from a deep learning perspective", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper proposes a sophisticated planning algorithm (composed of short-term and long-term-memory, gating mechanisms and feedback simulators) that can be run (almost) entirely in-context by an LM. I think this approach is very powerful and this paper does a good job showing its promise.\n2. The paper is well-written and easy to follow. I like the error analysis paragraph accompanying each section.\n3. I appreciate effort put into thorough benchmarking of code generation capabilities, including translating MBPP and HumanEval to Rust and introducing LeetcodeHardGym that prevents tests cases from having been leaked into GPT-4 data.\n4. The empirical results obtained by Reflexion are impressive, especially pass@1 for HumanEval.", "weaknesses": "1. It would be useful to compare Reflexion to baselines also in terms of inference efficiency. For instance, how many tokens are typically needed for a Reflexion run (vs e.g. ReAct or CoT) to reach an answer in a given task? Is the performance Reflexion bottlenecked by context window size and could we expect it to improve as context windows of LMs increase?\n2. Similarly, I am a bit disappointed that authors don’t report any error bars. I understand inference costs, but the authors could at least compute errors bars based on variance across problems (in code generation) or across tasks (in AlfWorld and HotPotQA). Alternatively, I’d rather see a less diverse task array than no error bars.\n3. Authors frequently compare Reflexion to human cognition (e.g., “This is akin to how humans iteratively learn  to accomplish complex tasks”, line 31-32; “similar to the way that humans remember fine-grain recent details while also recalling distilled important”, lines 141-143). I think these comparisons are overblown, not supported by cognitive science citations and potentially misleading. For instance, there are numerous dissimilarities between the kind of memory being used in Reflexion and human memory and the topic deserves a more careful discussion that just highlighting superficial similarities in passing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for your feedback!\n\n### 1. Token requirements and context\n\nReflexion computation scales linearly with respect to the number of episode attempts.\n    \nHowever, with regard to context sizes, Reflexion only keeps last N reflections in the context ($N=1$ for coding and $N=3$ for HotpotQA and AlfWorld), so the context length is not much longer than the base ReAct or CoT context length. \n\n### 2. Error bars\n\nWe were unable to run GPT experiments for many trials due to OpenAI API costs. See the **General Response (2)** for the error bars for StarCoder runs.\n- \"at least compute errors bars based on variance across problems (in code generation) or across tasks (in AlfWorld and HotPotQA)\": we are not sure what it means, we will be happy to provide any statistics based on existing results.\n\n### 3. References to human cognition\n\nThank you for the feedback. We will tone-down lines 31-32 and lines 141-143.\n\n### 4. Reflexion performance scaled down models\n\nPlease see the **General Response (2)**, where we show additional experiments with both stronger and weaker LLMs.\n\n### 5. Error classification for Alfworld\n\nError cases were classified by a simple heuristic: If a trajectory contained more than 30 actions, we classified the trial as poorly planned. If a trajectory contained less than 30 actions, but contained 3 consecutive identical action choices, then the trial was classified as a hallucination. This was implemented after our initial experiments as we found that agents were often unable to escape this repetitive action choice loop."}, {"review_id": "ah8y62DwGe", "reviewer": "Reviewer_V8mX", "summary": "The paper presents Reflexion, a framework to reinforce language agents by using language feedback. Reflexion agents verbally reflect on task feedback signals, then store their own reflective text in a memory buffer to improve their decision-making in future trials. Reflexion is flexible and effective across diverse tasks (sequential decision-making, coding, language reasoning). The paper also explores different types and sources of feedback signals, feedback incorporation methods, and agent types, and provides insights into how they affect performance.", "questions": "1. According to line 117, the evaluator is not necessarily an LLM, right? But it is pictured as “LM” in figure 2 which is a little bit confusing.\n2. In algorithm 1, $\\pi_\\theta$ is not relevant to trajectory $\\tau_{t-1}$, the short-term memory. But in figure 2 (a) and section 3, the Actor receives short-term memory as its input.\n3. How does short-term memory work on programming tasks? Is reflexion = self-debugging + CodeT?", "limitations": "The author has discussed the limitations well.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This paper proposes Reflexion, a new framework for “verbal” reinforcement that conditions a policy on the agent’s “memory” which is beneficial for the development of LLMs, reinforcement learning, reasoning, code generation, and so on.\n2. The results across diverse tasks including sequential decision-making, language reasoning, and programming are very promising with a large advantage.\n3. This paper proposes a new benchmark, LeetcodeHardGym, consisting of 40 challenging Leetcode questions in 19 different programming languages, which is beneficial for the program synthesis and code generation community.", "weaknesses": "1. Various LLMs (the GPT family, the vicuna family, and models with different sizes) should be evaluated to show reflexion’s generalization ability across models. Although reflexion performs well, it is important to analyze its border.\n2. Ablation study about long/short term memory. For example, in ALFWorld the baseline is react which resets the environment and starts a new trial whenever self-reflexion is suggested. I take it as “reflexion w/o long/short term memory”. Then, how about the performance of “reflexion w/o long-term memory (but w/ short-term memory)” and the performance of “reflexion w/o short-term memory (but w/ long-term memory)”?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the great comments!\n\n### 1. Other models\n\nPlease see **General Response (2)**, where we run additional experiments on closed and open source LLMs.\n\n### 2. Short vs long-term memory\n\nIn this work, we define short-term memory as the current trajectory and long-term memory as the reflections. It is not possible to omit short-term memory, and a long term memory ablation would be the baseline ReAct/CoT experiment. \n\n### 3. Evaluator\n\nYes, the Evaluator does not have to be an LLM. We will revise the figure.\n\n### 4. Policy\n\nYes, the policy is conditioned on the trajectory (and possibly other reflections). We will update the algorithm specification.\n\n### 5. Related work\n\nReflexion adds a self-reflection step and unit test generation step to Self-Debugging - and a self-debugging step to CodeT. So for coding Reflexion has important ideas of both self-debugging and CodeT, yet unlike self-debugging and CodeT, Reflexion is a more general framework that works for diverse tasks."}, {"review_id": "4UtbQvpFEm", "reviewer": "Reviewer_BKdQ", "summary": "This paper proposes a novel framework to do RL style learning through LLM. The idea is interesting. The authors have also done evaluation to show this technique/framework can work better than baseline methods w/ a few different tasks. ", "questions": "Will this framework also work well w/ VLMs? (if we also have vision/images as input)", "limitations": "See weakness section.\n", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The author proposed a novel framework to let LLM learn new tasks through trial and error. The framework seems general and can handle multiple tasks.", "weaknesses": "1. While the framework is general. The devil is in the details. It seems for each specific task, it requires different heuristic and tuning. Also different task may require different LLM/VLM as actor/evaluator/self-reflection.\n\n2. In terms of ablation, since this paper focuses on the framework, it seems more useful to ablate different framework level choices. i.e. is the self-reflection module really the key component in this framework, or can we bypass it by some other way, such as prompting?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n### 1. different heuristic/tuning/LLM/VLM for each task\n\n- In this work, there is no finetuning or VLM used. \n- We only used OpenAI GPT models as LLMs in the paper. In **General Response (2)**, we show it's also possible to use other LLMs, or use different LLMs for different roles (actor, evaluator, self-reflection). We believe such a flexibility is a benefit, rather than an issue.\n- Prompt design is task-specific to some degree, which is true for all prompting methods. However, Reflexion allows a principled way to design prompted roles (actor, evaluator, self-reflection) and their control flow. Also, the self-reflection prompt is mostly similar across different tasks. \n- In summary, as you said, \"the framework is general\" to solve diverse tasks because it could flexibly incorporate task heuristics with principled prompt role design and control flow. \n\n### 2. Necessity for self-reflection\n\n- We cannot \"bypass self-reflection by prompting\", because self-reflection is done via prompting in our work.\n- Ablating self-reflection is equivalent to ReAct/CoT baselines, which we reported in all tasks.\n\n### 3. Possible application with VLMs\n\nWe believe the idea of generating and conditioning on text reflections should be possible and useful for any language-conditioned policies, including ones using VLMs. This is an interesting question and a good future direction.\n"}, {"review_id": "2JGMMRlcSp", "reviewer": "Reviewer_U48e", "summary": "The paper introduces a novel framework, Reflexion, to improve the learning capabilities of language agents. Instead of traditional reinforcement learning methods, Reflexion uses linguistic feedback, where agents verbally reflect on task feedback and store these reflections in an episodic memory buffer to enhance decision-making. The authors demonstrate that Reflexion, which can incorporate various types and sources of feedback, outperforms baseline agents across diverse tasks, including text games, natural language question-answering, and code generation.", "questions": "1. Can you offer some statistics on the running time of the reflexion framework? As it seems that the optimization process is time-consuming.\n2. I am very curious about the comparison between reflexion and methods like self-refine and self-debugging.\n3. Can you offer some detailed analysis of the differences between your method (reflexion) and simple ReAct+self-refine?\n4. Is the quality of test cases generated in code generation tasks important? Are they guaranteed to be correct?\n", "limitations": "The authors have adequately introduced the limitations of their work. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a unique framework, Reflexion, that uses linguistic feedback for reinforcement learning in language agents. This is a significant departure from traditional methods that rely on weight updates.\n2. The paper offers a good explanation of their proposed methods and is easy to follow. Good visualizations are also included to help with reader understanding.\n3. Abundant experiments have been conducted across 3 different tasks, covering text games, question answering, and code generation. A comprehensive analysis has also shown the effectiveness of the reflexion method.", "weaknesses": "1. Although reflexion shows significant improvements over baselines on three different tasks, I still feel that reflexion shows limited generalization ability on different tasks. The key modules in the framework, including evaluator and self-reflection, have varying designs for different tasks. In other words, the specific design details may only work on one task and may fail on the other ones.\n2. The results for comparison in the question-answering task seem to be unfair. The reflexion framework leverages the environment binary reward of whether the answer is correct as the ground-truth answer or not. This leaks a little information from testing question-answer pairs. At least the ReAct baseline did not include such information.\n3. There are some related works mentioned in Section 2. However, there is no comparison between Reflexion and these methods in the experiment sections. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n### 1. the specific design details may only work on one task and may fail on the other ones\n\nPlease see the General Response.\n\n### 2. Accuracy comparisons\n\nYou are right. The Reflexion performance on HotpotQA is not meant as a SoTA claim. Rather, it is intended to show how performances can be improved with multiple rounds of reflections. ALFWorld and Coding results are fair comparisons. We will better clarify it in the paper.\n\n### 3. Comparison to other methods\n- Self-refine and self-debugging: self-refine is designed for text generation tasks, and self-debugging is designed for coding tasks. In contrast, Reflexion is designed for general agent tasks, which covers text generation, coding, and also embodied tasks (e.g. AlfWorld).\n- ReAct + self-refine: self-refine was not designed for acting tasks, cannot generated self-tests, and cannot incorporate environment or self-generated reward signals as Reflexion.\n- Experiments: In our programming ablation experiments, we omit various components, and these ablations are similar to baseline methods such as CodeT and Self-Debugging. Let us know if you think some other important baseline method is missing.\n\n### 4. Running time statistics\n\nWe evaluated `starchat-beta` on HumanEval Python - 161 problems (as mentioned in the General Response) and recorded wall time on 1 NVIDIA H100 GPU.\n\n*avg over 8 runs\nSimple: 00:25:18 (hh:mm:ss)\nReflexion: 01:17:06 (hh:mm:ss)\n\n### 5. Importance of tests\n\nYes, the quality of the tests in code generation is important and a bottleneck. In **Table 2**, we show that the high false positive rate of MBPP Python tests (relative to HumanEval Python) largely contributed to the inferior performance of Reflexion."}, {"review_id": "8MDoHwoI5M", "reviewer": "Reviewer_6XLB", "summary": "This paper presents a method by which LLMs can improve task performance through a form of learning in language: distilling unsuccessful attempts down to a \"lesson learned\" via a self-reflection process. The work shows that this idea leads to improved performance within a variety of task domains, and includes fairly thorough ablations, which I appreciated.\n\n", "questions": "See above for some suggestions.", "limitations": "Limitations are discussed", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "I find this to be a fairly satisfying paper overall. \n* The idea is compelling, and well executed.\n* The experiments demonstrate that the approach is beneficial across three fairly diverse LM application domains, which shows that it is not simply a narrow improvement.\n* The ablations are fairly thorough, and make the results more compelling.\n* The paper is generally well-written and presented.", "weaknesses": "I have a few suggestions on how the paper could be improved:\n\n1) I think this work could be more thoroughly integrated with the existing literature on learning from language feedback, both within LLMs (e.g. https://arxiv.org/abs/2204.14146 and https://arxiv.org/abs/2303.16749) and RL agents (https://proceedings.neurips.cc/paper_files/paper/2022/hash/0113ef4642264adc2e6924a3cbbdf532-Abstract-Conference.html and https://arxiv.org/abs/2112.03753). Another recent paper https://arxiv.org/abs/2304.03442 proposed a similar reflection approach, though with different motivation; it came out quite recently though so there is no expectation to refer to it.\n\n2) Evaluations on GPT-4 are challenging to interpret. We don't know what data the model was trained or tuned on, and even if the datasets used were generated after the pre-training cutoff, the model may have encountered them in the fine-tuning or RLHF stages. Indeed, OpenAI explicitly uses information submitted to their chat (at least) as training data; can we guarantee that no LeetCoder submitted the problems to GPT-4 to check their solutions? This problem is exacerbated if the benchmark has previously been evaluated against GPT-4, as it seems these problems have; there have been substantial concerns about GPT-4's coding performance evaluations being affected by data contamination. It would be best if the authors at least acknowledge these issues in the manuscript; ideally, they would also evaluate against an open language model for which the training process is more transparent, although undoubtedly such a model might not perform as well. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for your helpful suggestions!\n\n### 1. Related work\n\nThank you for pointing out these, which we will cite and discuss!\n\n### 2. Evaluation for closed-source large language models\n\n- This is a general issue for all works that use closed-source LLMs, and motivated us to create our own LeetcodeHardGym Benchmark, which was specifically filtered to only contain problems created later than the training data cutoff date of GPT-4.\n- Across all tasks, baseline methods based on GPT-4 are far from perfect, and Reflexion + GPT-4 significantly improves the performance, which indicates GPT-4 might not have memorized these tasks, and better prompting methods might be important.\n- It is also possible to use open-source LLMs for Reflexion, e.g. see **General Response (2)**."}]

Author rebuttal:

We appreciate all of the great feedback from our reviewers.

### 1. Related Work, Ablations, and Baselines

Reflexion is a general framework for reasoning, decision-making, and programming problems. A typical Reflexion loop consists of a generation, evaluation, and feedback function. Thus, the setup for the individual components are different, but similar across all tasks. To demonstrate the requirement for self-reflection, we precisely removed individual components of the loop to evaluate accuracy. On HotPotQA (Figure 4C), we show that having simple episodic memory (full past trajectories) in the context was not as effective as self-reflection instruction. On HumanEval Rust (Table 3), we show that removing the self-reflection step results in the same performance as the baseline accuracy. Many other related works such as Self-Debugging and CodeT lack at least one of: self-written tests, natural language reflections, or generalizability.

### 2. Different LLMs and Additional Experiments

- HotpotQA: We used `text-davinci-003` for the Actor and Self-Reflector. 
- Alfworld: We used `text-davinci-003` for the Actor, Evaluator, and Self-Reflector.
- Programming tasks: We used `gpt-4` for the Actor, Evaluator, and Self-Reflector.
- Additional experiment #1:
    - We ran HumanEval Python with `starchat-beta`, an open-source model that was pre-trained on a large open-source dataset of permissively licensed code, `The Stack`, as the Actor, Evaluator, and Reflector.
    - We found that low reflection generation quality prevented improvement in performance.
- Additional experiment #2:
    - We ran HotPotQA with `gpt-3.5-turbo` and `gpt-4` as the Actor, Evaluator, and Reflector to test Reflexion performance using stronger models.
    - It was not surprising that we found that Reflexion agents were able to learn with stronger models.

We ran two new experiments to evaluate Reflexion using `starchat-beta` on HumanEval, and `gpt-4` and `gpt-3.5-turbo` on HotPotQA.

|          |  HumanEval Pass@1  |
| -------- | ------- |
| Baseline | 0.26 (avg over 8 runs); SE = 0.00481 |
| Reflexion | 0.26 (avg over 8 runs); SE = 0.00305 |

StarChat is qualitatively and quantitatively not suitable for running a Reflexion loop. For example, given the following erroneous implementation:

```python
# Adds two numbers
def add(x, y):
  if x == 0:
    return 0
  else:
    return x + y
```

And feedback from the unit test suite:

```python
# passing tests
assert add(1, 2) == 3
assert add(3, 0) == 3

# failing tests
assert add(0, 2) == 2 # Output: 0
```

StarChat generates the following self-reflection which is not good:


    In this case, it appears as though there may have been an issue with scope or variable shadowing occurring within your function definition for add(). Specifically, in line 4 of your provided code snippet, where you define the local variable x with the value of 0, but then later reference the global variable x (which has no initial value assigned), it could potentially cause issues depending on how the rest of your function is written.

    To resolve this error, I would recommend explicitly defining a new variable called zero which takes on the value of 0 at the beginning of your function, like so:

    ```python
    def add(x,y):
        zero = 0 
        if x==0:
            return zero
        else: 
            return x+y
    ```

    This should ensure that the intended behavior of returning zero when x=0 remains intact while also preventing potential confusion around scoping/shadowing. Let me know if you need anything further!

On the other hand, we scaled-up the models for Reflexion evaluation on HotPotQA. We found that Reflexion enabled CoT and ReAct agents to improve performance.


| Model | Baseline | Reflexion |
| -------- | ------- | ------- |
| CoT (GT) + text-davinci-003 | 0.60 | 0.77 |
| CoT (GT) + gpt-3.5-turbo | 0.57 | 0.71 |
| CoT (GT) + gpt-4 | 0.68 | 0.80 |
| ReAct + text-davinci-003 | 0.3 | 0.55 |
| ReAct + gpt-3.5-turbo | 0.26 | 0.38 |
| ReAct + gpt-4 | 0.39 | 0.51 |

We believe that Reflexion will be strengthened as LLMs improve given that the ability to effectively self-reflect is crucial to performance.

Meta-review: In this paper, the authors proposed to treat LLM as an RL agent through linguistic feedback. Specifically, LLM agents reflect verbally on feedback signals and maintain a memory buffer to facilitate decision-making. The approach incorporates different types of feedback signals and can be applied to different types of tasks e.g. coding, language reasoning, and decision-making. 

All the authors appreciated the novelty of the proposed approach, especially in the short-term and long-term memory mechanism and natural feedback models, as well as the comprehensive experiments including a new benchmark LeetcodeHardGym. The results are significantly improved from baseline models, obtaining SOTA results in different types of tasks. The reviewers also suggested several ways to improve the paper, including a discussion of human cognition, and additional results from application to other LLMs besides OpenAI models. I hope the authors can incorporate some of these changes to improve their final paper.
Decision: Accept (poster)