ReAct: Synergizing Reasoning and Acting in Language Models
Authors: Shunyu Yao
Jeffrey Zhao
Dian Yu
Nan Du
Izhak Shafran
Karthik R Narasimhan
Yuan Cao
Venue: ICLR 2023 notable top 5% (2023)
Keywords: Language model
agent
reasoning
decision making

Abstract: While large language models (LLMs) have demonstrated impressive capabilities across tasks in language understanding and interactive decision making, their abilities for reasoning (e.g. chain-of-thought prompting) and acting (e.g. action plan generation) have primarily been studied as separate topics. In this paper, we explore the use of LLMs to generate both reasoning traces and task-specific actions in an interleaved manner, allowing for greater synergy between the two: reasoning traces help the model induce, track, and update action plans as well as handle exceptions, while actions allow it to interface with external sources, such as knowledge bases or environments, to gather additional information. We apply our approach, named ReAct, to a diverse set of language and decision making tasks and demonstrate its effectiveness over state-of-the-art baselines, as well as improved human interpretability and trustworthiness over methods without reasoning or acting components. Concretely, on question answering (HotpotQA) and fact verification (Fever), ReAct overcomes issues of hallucination and error propagation prevalent in chain-of-thought reasoning by interacting with a simple Wikipedia API, and generates human-like task-solving trajectories that are more interpretable than baselines without reasoning traces. On two interactive decision making benchmarks (ALFWorld and WebShop), ReAct outperforms imitation and reinforcement learning methods by an absolute success rate of 34% and 10% respectively, while being prompted with only one or two in-context examples.

Published as a conference paper at ICLR 2023
REACT: SYNERGIZING REASONING AND ACTING IN LANGUAGE MODELS
Shunyu Y Jeffrey Zhao?, Dian Yu², Nan Du", Izhak Shafran?, Karthik Narasimhan', Yuan Cao²
Department of Computer Science, Princeton University Googge Research, Brain team (shunnyy,, ka parthikni @princceee edu 2 jjeffreyzaao dianyu, dunan, izhak, yuancao / /google.com
ABSTRACT
While large language models (LLMs) have demonstrated impressive performance across tasks in language understanding and interactive decision making, their abilities for reasoning (e.g. chain-of-thought prompting) and acting (e.g. action plan generation) have primarily been studied as separate topics. In this paper, we explore the use of LLMs to generate both reasoning traces and task-specific actions in an interleaved manner, allowing for greater synergy between the two: reasoning traces help the model induce, track, and update action plans as well as handle exceptions, while actions allow it interface with and gather additional information from external sources such as knowledge bases or environments. We apply our approach, named ReAct, diverse set of language and decision making tasks and demonstrate its effectiveness over state- setie-of-the--es the-art baselines in addition to improved human interpretability and trustworthiness. Concretely, on question answering (HotpotQA) and fact verification (Fever), ReAct overcomes prevalent issues of hallucination and error propagation in chain-of-thought reasoning by interacting with a simple Wikipedia API, and generating human- like task-solving trajectories that are more interpretable than baselines without reasoning traces. Furthermore, on two interactive decision making benchmarks (ALFWorld and WebShop), ReAct outperforms imitation and reinforcement learning methods by absolute success rate of 34% and 10% respectively, while being prompted with only one or two in-context examples.
INTRODUCTION
A unique feature of human intelligence is the ability to seamlessly combine task-oriented actions with verbal reasoning (or inner speech, Jalech,,AAeessooo [Alderson- Day & Fernyhough, 2015), which has been theorized to play important role in human cognition for enabling self-regulation or strategization (Vygotsky) 1987 Luria 1965) Fernyhough 2010) and maintaining working memory (Baddeley 1992). Con- sider the example of cooking up dish in the kitchen. Between any two specific actions, reason in language in order to track progress ("now that everything is cut, should heat up the pot of water"), to handle exceptions me use soy sauce and pepper instead"), and to realize when external information is needed ("how do I prepare dough? Let me search on the Internet")). We may also act (open a cookbook read the recipe, open the fridge, check ingredients) to support the reasoning and to answer questions ("What dish can make right now?"). This tight synergy between "acting" and "reasoning" allows humans to learn new tasks quickly and perform robust decision making or reasoning, even under previously unseen circumstances facing information uncertainties.
may
adjust the plan according the situation ("I don't 't have salt, so let
Recent results have hinted at the possibility of combining verbal reasoning with interactive decision making autonomous systems. On one hand, properly prompted large language models (LLMMs) have demonstrated emergent capabilities to carry out several steps of reasoning traces to derive
Work during Google intersship. Projet page with code: https:://eeet wtw.cttth.mmiiiuuu....... whw.g.tttubiii.

---

Published as a conference paper at ICLR 2023
[3] Notspot
[Beason
Queation: Axide ceom the Apple Rezete, shat other device control the progran Apple Rezote saa originelly designed intaract
Thoughs
seurel. Apple Hesole und
peogras criginally cesigned
1: apple Remozee is renote control Incroduced In
October 2505 Apple originally desiqned co control the
Beandard
(1e) Act-Only
cunber paugean
Anzwer: 1Pod
Seapah(Apple Resote] obe The Apple Renota is penota centrol
Thought 2: Apple Renote was originally dessgned to control
Use 2sont Bow redia center prograr, need to pearch Fromt
Rid next and find shat other device can control It.
(Reason Only)
Act 2:
Seazch [Prent, Box]
thought: CSink
step. Rople Rendice
Aot (eaonh now]
Obs 2: Could nor find [Prent
Obe 2: could not (Frost Bov) Himilars ("Feoot Boy
Motorsporee",
Briginally designed hal
Reel. Stmilar:
Intarace with Apple TV
Thought Teant Rov found, need zearch
Apple TV be controkled
Act Search Bront Box luct
Bay
Aot 3:
Search Front BON roofraane
Roudia the unuwes La
Oba 3: Fxont s0x is d'scont Inued redia center
Ohe 3:
Front Row 16 discon-tnred vedia center enfrware
Lehone, 18od, and 1eod
Front Rov (aoi La cookanileed byyain Asull Benson or the keypaard funetian keys.. the aсaeer 1a hayboard Eunotion kayz. Finioh kephoond fonotion keyal
Thoughs 4:
Touch,
Anewer:
iPhone, :ad, 170d
Act. Pinishi
X
Alfooeld
Rotet (Heason Act)
You are Lt. the middle of roos, Locking quickly azound you, cabiines cabines cortee zachine 1, countertip atove burnne and toastaz 1, Your task
Act 1: [hink|Furat need to Find
siakat.....
likuly Lo appees cultineia (1nS1, countering 12-30,
Act 2: Go to cactnet
Fut pepper chaker draver.
dbs 2: do the caliinet Ly you soe a veau 2.
apeas
Act Go drawer Obe The dianet Ii clused.
[2x] Ast-Only
Act 6: Go countacten
obs 6: an the counter:op cop 3, you aee a apple L, bread I, peppas shakez 3, and vaza
Act 2: Open, cravez
Obe 21 You open the draver 1. The draser is open, In St, you see à dileheporge 2, and a sposn 1,
pegger zhaker
peopez abakes
courtestop countartop
Ast 8:
Shink flow I Find
shabez Bext, need
Act sinkauiin
18 sn/on drakes 1.
you distepange
Aot: 9) do to draner
apean
cbe 9:
Draker 16 olosed.
Act copperahaker froe sinkbasit
Act 10:
Open deaaes 1
Oba bassenas
10: You open Eraver
Act
aиpaarabaker
sinkbasto
X
pepper shaker
Figure 1: (1) Comparison of 4 prompting methods, (a) Standard. (b) offiin-o--houghh (CoT, Reason Only), (c) Act-only, and (d) ReAct (Reason+Act), solving a HotpotQA (Yang et al. 2018) question. (2) Comparison of (a) Act-only and (b) ReAct prompting solve an AlfWorld (Shridhar cet al. 2020b) game. In both domains, we omit in-context examples in the prompt, and only show task solving trajectories generated by the model (Act, Thought) and the environment (Obs).
answers from questions in arithmetic, commonsense, and symbolic reasoning tasks (Wei et al., 2022). However, this "chain-of-thoughe reasoning is a static black box, in that the model uses its own internal representations to generate thoughts and is not grounded in the external world, which limits its ability to reason reactively or update its knowledge. This can lead to issues like fact hallucination and error propagation over the reasoning process (Figure T (Ib)). On the other hand, recent work has explored the of pre-trained language models for planning and acting in interactive environments (Ahn et al.) 2022] Nakano et al. 2021 Yao et al. |020 Huang et al., 2022a), with focus on predicting actions via language priors. These approaches usually convert multi-modal observations into text, use a language model to generate domain-specific actions or plans, and then use a controller to choose or execute them. However, they do not employ language models to reason abstractly about high-level goals or maintain working memory to support acting, barring Huang al. (2022b) who perform a limited form of verbal reasoning to reiterate spatial facts about the current state. Beyond such simple embodied tasks to interact with few blocks, there have not been studies on how reasoning and acting can be combined in synergistic manner for general task solving, and if such a combination can bring systematic benefits compared to reasoning or acting alone.
In this work, we present ReAct, a general paradigm to combine reasoning and acting with language models for solving diverse language reasoning and decision making tasks (Figure [.. ReAct prompts LLMs to generate both verbal reasoning traces and actions pertaining to a task in an interleaved manner, which allows the model to perform dynamic reasoning create, maintain, and adjust high-level plans for acting (reason to act). while also interact with the external environments (e.g. Wikipedia) to incorporate additional information into reasoning (act to reason).
2

---

Published as a conference paper at ICLR 2023
We conduct empirical evaluations of ReAct and state-of-the-an baselines on four diverse benchmarks: question answering (HotPotQA, Yang et al., 2018), fact verification (Fever, Thomn et al.] (2018). text-based game (ALFWorld, [Shridhar et al. 2020b), and webpage navigation (WebShop, Yao eet al. 2022 For HotPotQA and Fever, with access to a Wikipedia API that the model can interact with, ReAct outperforms vanilla action generation models while being competitive with chain-of- thought reasoning (CoT) (Wei et al., (002)) The best approach overall is combination of ReAct and CoT that allows for the use of both internal knowledge and externally obtained information during reasoning. On ALFWorld and WebShop, two or even one-shot ReAct prompting is able to outperform imitation or reinforcement learning methods trained with 10 ~ 100h task instances, with an absolute improvement of 34% and 10% in success rates respectively. We also demonstrate the importance of sparse, versatile reasoning in decision making by showing consistent advantages over controlled baselines with actions only. Besides general applicability and performance boost, the combination of reasoning and acting also contributes to model interpretability, trustworthiness, and diagnosability across all domains, as humans can readily distinguish information from model's internal knowledge versus external environments, as well as inspect reasoning traces to understand the decision basis of model actions.
To summarize, our key contributions are the following: (1) we introduce ReAct, a novel prompt- based paradigm synergize reasoning and acting in language models for general task solving; (2) we perform extensive experiments across diverse benchmarks to showcase the advantage of ReAct in a few-shot learning setup over prior approaches that perform either reasoning or action generation in isolation; (3) we present systematic ablations and analysis to understand the importance of acting in reasoning tasks, and reasoning in interactive tasks; (4) we analyze the limitations of ReAct under the prompting setup (i.e. limited support of reasoning and acting behaviors), and perform initial finetuming experiments showing the potential of ReAct improve with additional training data. Scaling up ReAct to train and operate more tasks and combining it with complementary paradigms like reinforcement learning could further unlock the potential of large language models.
2 REACT: SYNERGIZING REASONING + ACTING
Consider a general setup of an agent interacting with an environment for task solving. At time step f, an agent receives an observation or E O from the environment and takes an action ae E A following some policy (a (ca), where q = (01, 41 0t-1, .,-1, 0+) is the context to the agent. Learning policy is challenging when the mapping c + is highly implicit and requires extensive computation. For example, the agent shown in Figure Icc) is unable to generate the correct final action (Act 4) to finish the QA task it requires complex reasoning over the trajectory context (Question, Act 1-3, Obs 1-3). Similarly, the agent shown in Figure | (2a) fails to comprehend from the context that sinkbasin 1 does not contain peppershaker 1, thus keep producing hallucinating actions.
The idea of ReAct is simple: space of language. An action à, E L in the language space, which we will refer to as a thought or a reasoning trace, does not affect the external environment, thus leading no observation feedback. Instead, a thought o aims to compose useful information by reasoning over the current context ct. and update the context 1 (ct, âz) to support future reasoning or acting. As shown in Figure [ there could be various types of useful thoughts, c.g. decomposing task goals and create action plans (2b, Act 1; 1d, Thought 1). injecting commonsense knowledge relevant to task solving (2b, Act 1), extracting important parts from observations (Id, Thought2, 4), track progress and transit action plans (2b, Act 8), handle exceptions and adjust action plans (Id, Thought 3), and so on.
augment the agent's action space Á -AUG L, where L is the
However, as the language space L is unlimited, learning in this augmented action space is difficult and requires strong language priors. In this paper, we mainly focus on the setup where a frozen large language model, PaLM-540B (Chowdhery et al. 2022 is prompted with few-shot in-context examples to generate both domain-specifte actions and free-form language thoughts for task solving (Figure [ (1d), (2b)). Each in-context example is human trajectory of actions, thoughts, and environment observations to solve a task instance (see Appendix C). For the tasks where reasoning is of primary importance (Figure T 1)), we alternate the generation of thoughts and actions that the task-solving trajectory consists of multiple noought-actionnobseree
steps. In contrast, for decision making tasks that potentially involve large number of actions (Figure (]),, thoughts only need to
We show some GPT-3 Brown al. 2020 results in Appendix A.1 which outperforms M-540B.
3

---

Published as a conference paper at ICLR 2023
appear sparsely in the most relevant positions of trajectory, we let the language model decide the asynchronous occurrence of thoughts and actions for itself.
Since decision making and reasoning capabilities are integrated into large language model, ReAct enjoys several unique features: A) Intuitive and easy to design: Designing ReAct prompts is straightforward as human annotators just type down their thoughts in language on top of their actions taken. No ad-hoc format choice, thought design, or example selection is used in this paper. We detail prompt design for each task in Sections Barooonsssnnddo Bnd 4 B) General and flexible: Due to the flexible thought space and thought-action occurrence format, ReAct works for diverse tasks with distinct action spaces and reasoning needs, including but not limited QA, fact verification, text game, and web navigation. C) Performant and robust: ReAct shows strong generalization to new task instances while learning solely from one to six in-context examples, consistently outperforming baselines with only reasoning or acting across different domains. We also show in Section adddiionnal benefits when finetuning is enabled, and in Section lhow ReAct performance is robust to prompt selections. D) Human aligned and controllable: ReAct promises an interpretable sequential decision making and reasoning process where humans can easily inspect reasoning and factual correctness. Moreover, humans can also control or correct the agent behavior the go by thought editing, as shown in Figure 5 in Section
3 BHOWLEDGE-INTENSSEE REASONING TASKS
We begin with knowledge-intensivy reasoning tasks like multi-hop question answering and fact verification. As shown in Figure dd), by interacting with a Wikipedia API, ReAct is able to retrieve information to support reasoning, while also use reasoning target what retrieve next, demonstrating synergy of reasoning and acting.
3.1 SETUP
Domains We consider two datasets challenging knowledge retrieval and reasoning: (1) Hot- PotQA (Yang et al., 2018), a multi-hop question answering benchmark that requires reasoning two more Wikipedia passages, and (2) FEVER (Thorne et al. 2018), fact verification benchmark where each claim is annotated SUPPORTS, REFUTES, or NOT ENOUGH INFO, based on if there exists a Wikipedia passage to verify the claim. In this work, operate in question-only setup for both tasks, where models only receive the question/claim as input without access to support paragraphs, and have to rely their internal knowledge retrieve knowledge via interacting with an external environment to support reasoning.
Action Space We design a simple Wikipedia web API with three types of actions to support interactive information retrieval: (1) search(entityl which returns the first 5 sentences from the corresponding entity wiki page if it exists, or else suggests top-5 similar entities from the Wikipedia search engine, (2) lookup[st st ing], which would return the next sentence in the page containing st wing, simulating Ctrl+F functionality on the browser. (3) finish(answer], which would finish the current task with answer. We note that this action space mostly can only retrieve a small part of a passage based on exact passage name, which is significantly weaker than state-of-the, art lexical or neural retrievers. The purpose to simulate how humans would interact with Wikipedia, and force models to retrieve via explicit reasoning in language.
3.2 METHODS
ReAct Prompting For HotpotQA and Fever, we randomly select 6 and cases" from the training set and manually compose ReAct-format trajectories to use as few-shot exemplars in the prompts. Similar Figure d)), each trajectory consists of multiple thought-action-observaii steps (i.e. dense thought), where free-form thoughts are used for various purposes. Specifically, we use a combination of thoughts that decompose questions ("I need search x, find y, then find z"), extract information from Wikipedia observations ("x was started in 1844", "The paragraph does not tell x"), perform commonsense ("xx not y, so z must instead be...") or arithmetic reasoning ('1844 < 1989"), ",99") guide
We find examples do not improve performance.
4

---

Published as a conference paper at ICLR 2023
HotpotQA (EM)
Fever (Acc)
Prompt Method
Standard
28.7
57.1
85.0
CoT
29.4
56.3
62.5
CoT-SC Wang
2022a
33.4
60.4
60.0
HoppoAA
Act
25.7
ReAct
27.4
60.9
55.0
CoT-5C RuAct
CoT-SC ReAct
34.2
64.6
ReAct-> CoT-SC
ReAct-+ CoT-SC
35.1
62.0
52.5
CoT
50.0
Supervised SoTA
67.5
89.5
28
10
15
Table 1: PaLM-540B prompting results on HotpotQA and Fever.
#CoT traals
#C.T-SC
Figure 2: PaLM-540B prompting results with respect to number CoT SC samples used.
"HoepoOOA
27.1, 28.9, 338 for Stondard, CoT,
GoT-5C in Wane et el Mi0iion
7au al.
2021
Lewis
2020
search reformulation ("maybe can search/look up x instead")), and synthesize the final answer ("...so the answer is x"). See Appendix C for more details.
Baselines We systematically ablate ReAct trajectories to build prompts for multiple baselines (with formats Figure I [a-lc)): (a) Standard prompting (Standard) rd), which removes all thoughts, actions, observations in ReAct trajectories.
(b) Chain-of-thouglle prompting (CoT) (Wei et al. 2022), which removes actions and observations and serve
reasoning baseline. We also build self-consistency baseline (CoT-SC) (Wang et al. 2022a b) by sampling 21 CcT trajectories with decoding temperature 0.7 during inference and adopting the majority answer, which is found to consistently boost performance over CoT. (c) Acting-only prompt (Act). which removes thoughts ReAct trajectories, loosely resembling how WebGPT (Nakano et al. (02T) interacts with the Internet to answer questions, though operates on different task and action space, and uses imitation and reinforcement learning instead prompting.
Combining Internal and External Knowledge As will be detail in Section 3.3 we observe that the problem solving process demonstrated by ReAct is more factual and grounded, whereas CoT more accurate in formulating reasoning structure but easily suffer from hallucinated facts or thoughts. We therefore propose to incorporate ReAct and CoT-SC, and let the model decide when switch the other method based the following heuristics: A) ReAct + CoT-SC: when ReAct fails to return an answer within given steps, back off to CoT SC. We set and steps for HotpotQA and FEVER respectively as we find more steps will not improve ReAct performance B) CoT-SC > ReAct: when the majority answer among n CoT-SC samples occurs less than n/2 times (i.e. internal knowledge might not support the task confidently), back off ReAct.
Finetuning Due to the challenge of manually annotating reasoning traces and actions at scale, we consider bootstraping approach similar to /Zelikaan al. (2022). using 3,000 trajectories with correct answers generated by ReAct (also for other baselines) to finetune smaller language models (PaLM-8/62B) to decode trajectories (all thoughts, actions, observations) conditioned on input questions/claims. More details are in Appendix B.T
3.3 RESULTS AND OBSERVATIONS
ReAct outperforms Act consistently Table Tsows HotpotQA and Fever results using PaLM- 540B as the base model with different prompting methods. We note that ReAct is better than Act on both tasks, demonstrating the value of reasoning guide acting, especially for synthesizing the final answer, as shown in Figure (Ic-d). Fine-tuning results alsso confirm the benefit of reasoning traces for more informed acting.
Of all trajectories with correct final answers, those with steps on HotpotQA and 5 steps FEVER only take up 0.84% and 1.33% respectively.
5

---

Published as a conference paper at ICLR 2023
Type
Definition
ReAct
CoT
Thie positive
Correct reasoning and facts
94%
86%
Success
False posilive
Hallacinated reasoning trace or facts
6%
14%
Reasoning error
Wrong reasoning trace (including failing to recover from ropetiive stops)
47%
16%
Failure
Seurch result ervor
Search return empty does contain useful information
23%
Hallucination
Hallacinated reasoning truce or facts
0%
56%
Label ambiguity
Right prediction but did match the label precisely
29%
28%
Table 2: Types of success and failure modes of ReAct and CoT on HotpotQA, well as their percentages in randomly selected examples studied by human.
ReAct vs. CoT On the other hand, ReAct outperforms CoT on Fever (60.9 vs. 56.3) and slightly lags behind CoT on HotpotQA (27.4 vs. 29.4). Fever claims for SUPPORTS/REFUTER might only differ by a slight amount (see Appendix [D.T), so acting to retrieve accurate and up-to-date knowledge is vital. To better understand the behavioral difference between ReAct and CoT on HotpotQA, we randomly sampled 50 trajectories with correct and incorrect answers (judged by EM) from ReAct and CoT respectively (thus 200 examples in total), and manually labeled their success and failure modes in Table Some key observations are as follows:
A) Hallucination is serious problem for COT, resulting in much higher false positive rate than ReAct (14% vs. 6%) in success mode, and make up its major failure mode (56%). In contrast, the problem solving trajectory of ReActis more grounded, fact-driven, and trustworthy, thanks to the access of an external knowledge base.
B) While interleaving reasoning, action and observation steps improves ReAct's grounded- ness and trustworthiness, such a structural constraint also reduces its flexibility in formulating reasoning steps, leading to more reasoning error rate than COT. we note that there is one frequent error pattern specific to ReAct, in which the model repetitively generates the previous thoughts and actions, and we categorize it as part of "reasoning error" as the model fails to reason about what the proper next action to take and jump out of the lootf]
C) For ReAct, successfully retrieving informative knowledge via search is critical. Non- informative search, which counts for 23% of the error cases, derails the model reasoning and gives a hard time to recover and reformulate thoughts. This is perhaps an expected trade-off between factuality and flexibility, which motivates our proposed strategies of combining two methods.
We provide examples for each success and failure modes in Appendix E. We also find some HotpotQA questions may contain outdated answer labels, see Figure 4 for example.
ReAct + CoT-SC perform best for prompting LLMS Also shown in Table the best prompting method on HotpotQA and Fever are ReAct + CoT-SC and CoT-SC - ReAct respectively. Furthermore, Figure 2 shows how different methods perform with respect to the number of CoT-SC samples used. While two ReAct + CoT-SC methods are advantageous at one task each, they both significantly and consistently outperform CoT-SC across different number of samples, reaching CoT-SC performance with 21 samples using merely 3-5 samples. These results indicate the value of properly combining model internal knowledge and external knowledge for reasoning tasks.
ReAct performs best for fine-tuning Figure 3 shows the scaling effect of four methods (Standard, CoT, Act, ReAct) on HotpotQA. With PaLM- prompting ReAct performs worst among four methods due to the difficulty to learn both reasoning and acting from in-context examples. However, when finetuned with just 3,000 examples, ReAct. becomes the best method among the four, with PaLM-8B finetuned ReAct outperforming all PaLM-62B prompting methods, and PaLM-62B finetuned ReAct outperforming all 540B prompting methods. In contrast, finetuning Standar rd or CoT is significantly worse than finetuning ReAct or Act for both PaLM- 8/62B, as the former essentially teaches models to memorize (potentially halluincated) knowledge facts, and the latter teaches models how to (reason and) act to access information from Wikipedia, more generalizable skill for knowledge reasoning. As all prompting methods are still significantly far from Nomain-spectio state-of-the-art approaches (Table I we believe finetuning with more Human-written data might be a better way to unleash the power of ReAct.
We suspect that this could he due to the sub-optimal greedy decoding procedure, and future work using hetter decoding (e.g. heam search) might help address this issue.

---

Published as a conference paper at ICLR 2023
learning prompt
leaming finetune
30
25
Method
Hotpot
Standard
15
CoT
Act
10
ReAct
8b
62b
540b
8b
62b
540b
Figure 3: Scaling results for prompting and finetuning on HotPotQA with ReAct (ours) and baselines.
DECISION MAKING TASKS
We also test ReAct on two language-based interactive decision-making tasks, ALFWorld and WebShop, both of which feature complex environments that require agents to act over long horizons with sparse rewards, warranting the need for reasoning to act and explore effectively.
ALFWorld ALFWorld (Shridhar et al. 2020b) (Figure 2)) a synthetic text-based game designed to align with the embodied ALFRED benchmark (Shridhar et al.) (2020a). It includes 6 types of tasks in which an agent needs to achieve high-level goal (e.g. examine paper under desklamp) by navigating and interacting with simulated household via text actions (e.g. go to coffeetable 1, take paper 2, use desklamp 1). A task instance can have more than 50 locations and take an expert policy more than 50 steps solve, thus challenging
agent to plan and track subgoals, as well as explore systematically (e.g. check all desks one by one for desklamp). In particular, one challenge built into ALFWorld is the need determine likely locations for common household items (e.g. desklamps will likely be on desks, shelfs, or dressers), making this environment a good fit for LLMs to exploit their pretrained commonsense knowledge. To prompt ReAct, we randomly annotate three trajectories from the training set for each task type, where each trajectory includes sparse thoughts that (1) decompose the goal, (2) track subgoal completion, (3) determine the next subgoal. and (4) reason via commonsense where to find an object and what to do with it. We show prompts used for ALFWorld in Appendix C.4, Following [Shridhar et al. (2000b), we evaluate on 134 unseen evaluation games in task-specific setup. For robustness, we construct 6 prompts for each task type through each permutation of 2 annotated trajectories from the 3 we annotate. Act prompts are constructed using the same trajectories, but without thoughts since task instances are randomly chosen from the training set, it favors neither ReAct nor Act and provides fair and controlled comparison test the importance of sparse thoughts. For baselines, we use BUTLER (Shridhar et al. 2020b), imitation learning agent trained on 10 expert trajectories for each task type
WehShop Can ReAct also interact with noisy real-world language environments for practical applications? We investigate WebShop (Yao et al., 2022). a recently proposed online shopping website environment with 1.18M real-world products and 12k human instructions. Unlike ALFWorld, Webshop contains a high variety of structured and unstructured texts (e.g. product titles, descriptions, and options crawled from Amazon), and requires an agent to purchase product based on user instruction (e.g." "I am looking for a nightstand with drawers. It should have a nickel finish, and priced lower than $140") through web interactions (e.g. search "nightstand drawers". choose buttons such "color: moderm-nickkl-wwiee or "back to search"). This task is evaluated by average score (percentage of desired attributes covered by the chosen product averaged across all episodes) and success rate (percentage of episodes where the chosen product satisfies all requirements) on 500 test instructions. We formulate Act prompts with actions to search, choose product, choose options, and buy, with ReAct prompts additionally reasoning to determine what to explore, when to buy, and what products options are relevant to the instruction. See Table 6 for an example prompt, and Table 10 for model predictions the Appendix. We compare an imitation learning (IL) method
Micheli & Fleuret 2021 finetuned GPT-2 model on 3553 task instances and achieved much improved nerformance than BUTT L.R. but is trained on all task types, thus not included baseline.
7

---

Published as a conference paper at ICLR 2023
Method
Pick
Clean
Heat
Cool
Look
Pick 2
All
Method
Score
SR
Act (bxxll())
88
42
74
67
72
41
45
Act ReAct
62.3
30.1
ReAct (ogg)
65
39
83
76
55
24
57
66.6
40.0
ReAct (e
92
58
96
86
78
41
71
IL
59.9
29.1
ReAct- TM (rrg)
59
60
55
23
24
48
IL++L
62.4
28.7
ReAct- IM oo
62
68
87
57
39
33
53
Human Expert
82.1
59.6
BUTLER, (hoel
33
26
70
76
17
12
22
BUTLER (boa 8)
46
39
74
100
22
24
37
Table 4: Score and suc- cess rate (SR) on Web- shop. IL/IL+RL taken
Table 3: AlfWorld task-specific success rates (%). BUTLER and BUTLERg results are from Table (f al. (2020b). All methods use greedy decoding, except that BUTLER uses beam search.
from Yao et al. (022)).
trained with 1,012 human annotated trajectories, and a imitation + reinforcement learning (IL + RL) method additionally trained with 10,587 training instructions.
Results ReAct outperforms Act on both ALFWorld (Table 3 and Webshop (Table On ALFWorld, the best ReAct trial achieves an average success rate of 71%, significantly outperforming the best Act (45%) and BUTLER (37%) trials. In fact, even the worse ReAct trial (48%) beats the best trial of both methods. Moreover, the advantage of ReAct over Act is consistent across six controlled trials, with relative performance gain ranging from 33% to 90% and averaging 62%. Qualitatively, we saw that, without any thoughts at all, Act fails correctly decompose goals into smaller subgoals, or loses track of the current state of the environment. Example trajectories comparing ReAct and Act can be found in Appendix D.2. and and Appendix [.2.22
On Webshop, one-shot Act prompting already performs on par with IL. and LL methods. With additional sparse reasoning, ReAct achieves significantly better performance, with an absolute 10% improvement over the previous best success rate. By checking examples, we find that ReAct is more likely to identify instruction- noisy observations and actions (e.g. "For 'space-savin saving ottoman bench for living room', the item has options '39x 1xx 11111111mm 18nnch' and 'blue' and seems good to buy."). However, existing methods are still far from the performance of expert humans (Table who perform significantly more product explorations and query re-formulations that are still challenging for prompting hased methods.
products and options by reasoning to bridge the gap between
On the value of internal reasoning vs. external feedback To our knowledge, ReAct is the first demonstration of combined reasoning and action using an LLM applied to an interactive environment within a closed-loop system. Perhaps the closest prior work Inner Monologue (IM), from Huang et al. (2022b), in which actions from an embodied agent are motivated by an eponymous "inner monologue". However, IM's "inner monologue" is limited to observations of the environment state and what needs to be completed by the agent for the goal to be satisfied. In contrast, the reasoning traces in ReAct for decision making is flexible and sparse, allowing diverse reasoning types (see Section 2) to be induced for different tasks.
To demonstrate the differences between ReAct and IM, and to highlight the importance internal reasoning vs. simple reactions to external feedback, ran ablation experiment using a thought pattern composed of IM-like dense external feedback. As can be seen in Table ReAct substantially outperforms IM-style prompting (ReAct-TM) (71 v3. overall success rate), with consistent advantages on five out of six tasks. Qualitatively, we observed that ReAct- IM often made mistakes in identifying when subgoals were finished, what the next subgoal should be, due lack of high- level goal decomposition. Additionally, many ReAct -IM IM trajectories struggled to determine where an item would likely be within the ALFWorld environment, due a lack of commonsense reasoning. Both shortcomings can be addressed in the ReAct. paradigm. More details about ReAct-In IM is in Appendix [B.2 An example prompt for ReAct-In IM can be found in Appendix C.4 and an example trajectory in Appendix D.2.3
5 RELATED WORK
Language model for reasoning Perhaps the most well- known work of using LLMs for reasoning Chain-of-Thoughh (CoTT) (Wei et al. 2022), which reveals the ability of LLMS to formulate their
8

---

Published as a conference paper at ICLR 2023
own "thinking procedure" for problem solving. Several follow-up works have since been performed,
including least-to-most prompting for solving complicated tasks (Zhou et al., (002)) zero-shot-
CoT (Kojima al. 2022), and reasoning with selffcoonsisency (Wang et al. 2022a). Recently,
(Madaan & Yazdanbakhsh 2022) systematically studied the formulation and structure of CoT,, and observed that the presence of symbols, patterns and texts is crucial to the effectiveness of COT. Other work has also been extended to more sophisticated reasoning architecture beyond simple prompting. For example Inferrion-Inference (Creswell al. 2022) divides the reasoning process into two steps of "selection" and "inference". STaR (Zelikman et al. 2022) bootstraps the reasoning process by finetuning the model on correct rationales generated by the model itself. Faithful reasoning (Creswell & Shanahan| 2022) decomposes multi-step reasoning into three steps, each performed by a dedicated LM respectively. Similar approaches like Scratchpad (Nye et al. 202T). which finetunes a LM on intermediate computation steps, also demonstrate improvement In contrast to these methods, ReAct performs more than just isolated, fixed reasoning, and integrates model actions and their corresponding observations into coherent stream of inputs for the model to reason more accurately and tackle tasks beyond reasoning (e.g. interactive decision making).
multi-step computation problems.
Language model for decision making The strong capability of LLMS has enabled them perform tasks beyond language generation, and it is becoming more popular to take advantage of LLMs as policy model for decision making, especially in interactive environments. WebGPT (Nakano et al.) 2021 uses an LM to interact with web browsers, navigate through web pages, and infer answers to complicated questions from ELI5 (Fan et al., (019). In comparison to ReAct, WebGPT does not explicitly model the thinking and reasoning procedure, instead rely expensive human feedback for reinforcement learning. In conversation modeling, chatbots like BlenderBot (Shuster et al. 2022b) and Sparrow (Glaese et al. 2022) and task-oriented dialogue systems like SimpleTOD (Hosseini-As) iet al.] 2020) also train LMs to make decision about API calls. Unlike ReAct, they do not explicitly
consider the reasoning procedure either, and also relies expensive datasets and human feedback collections for policy learning. In contrast, ReAct learns a policy in a much cheaper way, since the
decision making process only requires language description of the reasoning procedure 6
LLMS have also been increasingly employed in interactive and embodied environments for planning and decision making. Perhaps most relevant to ReAct in this respect are SayCan (Ahn al.) 2022) and Inner Monologue (Huang et al.) 2022b), which use LLMs for robotic action planning and decision making. In SayCan, LLMs were prompted to directly predict possible actions a robot can take, which is then reranked by an affordance model grounded on the visual environments for final prediction. Inner Monologue made further improvements by adding the eponymous "inner monologue", which implemented as injected feedback from the environment. To our knowledge, Inner Monologue is the first work that demonstrates such a closed-loop system, which ReAct builds on. However, we argue that Inner Monologue does not truly comprise of inner thoughts- - this is elaborated in Section [ We also note that leveraging language as semantically inputs in the process of interactive decision making has been shown to be successful under other settings (Abramson et al. 2020 Karamcheti et al. 2021 Huang et al.) 2022a Li et al., 2022). It is becoming more evident that with the help of fundamental cognitive mechanism will play a critical role in interaction and
LLMs, language
decision making. What is more, progress in LLMs has also inspired the development of versatile and
generalist agents like Reed et al. (202).
CONCLUSION
We have proposed ReAct a simple yet effective method for synergizing reasoning and acting in large language models. Through diverse set of experiments on multi-hop question-answeriigg fact checking, and interactive decision-making tasks, we show that ReAct leads to superior performance with interpretable decision traces. Despite the simplicity of our method, complex tasks with large action spaces require more demonstrations learn well, which unfortunately can easily go beyond the input length limit of in-context learning. We explore the fine-tuning approach on HotpotQA with initial promising results, but learning from more high-quality human annotations will be the desiderata to further improve the performance. Scaling up ReAct with multi-task training and combining it with complementary paradigms like reinforcement learning could result in stronger agents that further unlock the potential of LLMs for more applications.
"Humn feedback can also be incorporated in complementary manner but leave it for future work.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
We thank the support and feedback of many people from Google Brain team and Princeton NLP Group. This work was supported in part by the National Science Foundation under Grant No. 2107048. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.
REPRODUCIBILITY STATEMENT
Our main experiments done PaLM (Chowdhery et al. 2022), which is not an openly accessible model yet. To increase reproducibility, we have included all used prompts in Appendix [ additional experiments using GPT-3 (Brown et 2020) in Appendix A.. and associated GPT-3 ReAct
prompting code at httpt:tt//// /annnyooosuuu... 4open. see.nssceennn scwenceeeecccccccccccccccccccecs wttpee/sscciceeeeeeeeeee /ReAcct 22687
oltttpss//enonymousssu
ETHICS STATEMENT
ReAct prompts large language models to generate more human interpretable, diagnosable, and controllable task-solving trajectories than previous methods. However, hooking up a large language model with action space interact with external environments (e.g. the web, physical environ- ments) has potential dangers, e.g. looking up inappropriate or private information, or taking harmful actions in an environment. Our experiments minimize such risks by limiting the interactions to specific websites (Wikipedia or WebShop) that are free of private information, without any dangerous actions in the action space design (i.e. models cannot really buy products on WebShop the research benchmark, or edit Wikipedia). We believe researchers should be aware of such risks before designing more extensive experiments in the future.
REFERENCES
Josh Abramson, Arun Ahuja, lain Barr, Arthur Brussee, Federico Carnevale, Mary Cassin, Rachita Chhaparia, Stephen Clark, Bogdan Damoc, Andrew Dudzik, Petko Georgiev, Aurelia Guy, Tim Harley, Felix Hill, Alden Hung, Zachary Kenton, Jessica Landon, Timothy Lillicrap, Kory Mathew- son, Soña Mokrá, Alistair Muldal, Adam Santoro, Nikolay Savinov, Vikrant Varma, Greg Wayne, Duncan Williams, Nathaniel Wong, Chen Yan, and Rui Zhu. Imitating interactive intelligence, 2020. URL https::/ www.arxww...rgaaaa. org wws/////////////.aa/////////aa// 2012.00222022 0127756222
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov. Yuheng Kuang, Kuang- Huci Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng. Do as i can, not as say: Grounding language in robotic affordances, 2022. URL https://arxiy> wrg. 220.2.164....
Ben Alderson-Day and Charles Fernyhough. Inner speech: development, cognitive functions, phenomenology, and neurobiology. Psychological bulletin, 141(5):931, 2015.
Alan Baddeley. Working memory. Science, (35(5044):5566555) 1992.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neurul information processing systems, 33:1877-1901, 2020.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023111 2022.
Antonia Creswell and Murray Shanahan. Faithful reasoning using large language models, 2022. URL
https://arxiv. www./bgg.ss..oooooooo 2208 14271
10

---

Published as a conference paper at ICLR 2023
Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-inference: Exploiting large language models for interpretable logical reasoning. 2022. URL hellattpa://axxiv.. abs/
2205.09713
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3558 3567, Florence, Italy, July 2019. Association for Com- putational Linguistics. doi: 10. 10.18653/w1/P194 1346. URL httph::p//// http:::://aalaaahooooyoo
P1 P19-1346
Charles Femyhough. Vygotsky, luria, and the social brain. Self and social regulation: Social interaction and the development of social understanding and executive functions, pp. 56-79, 2010.
fxecutivefuncciins
Amelia Glaese, Nat McAleese, Maja Trebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Mari- beth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Hampbell-Gillinghan Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang. Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Soña Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, lason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, and Geoffrey Irving. Improving alignment of dialogue agents via targeted human judgements, 2022. URL httpsss//stoorrr.gooogggog////// www.leappss..oom.. com/ ceemm/depmiiii nd-med: a/DeepMi nd Com
Ehsan Hosseini-Asl,, Bryan McCann, Chien-Sheng Wu, Semih Yavuz, and Richard Socher. A simple language model for task-oriented dialogue. Advances in Neural Information Processing Systems, 33:20179-20191, 2020.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint (rXii:2201.07000 2022a.
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.056003 2022b.
Siddharth Karamcheti, Megha Srivastava, Percy Liang, and Dorsa Sadigh. Lila: Language-informed latent actions. In CoRL, pp. 1379-1390, 2021. URL
mlr www.press......e..mmmmm
#16l/karamchett222... html
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.1199 2022.
Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev. Internet- augmented language models through few-shot prompting for open-domain question answering. arXiv preprint 2000:v2203.05/ 15, 2022.
Patrick Lewis, Ethan Perez, Aleksandra Piktus. Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, al. Retrieval- -uggeeted genera- tion for Enowledge-ineensive nip tasks. Advances in Neural Information Processing Systems, 33: 9459 9474, 2020.
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clinton Wang, Linxi Fan, Tao Chen, De-An Huang. Ekin Akyürek, Anima Anandkumar, Jacob Andreas, Igor Mordatch, Antonio Torralba, and Yuke Zhu. Pre-trained language models for interactive decision-making, 2022. htRthttps
wtt.xxv..rr//bb/220000 01771
Aleksandr Romanovich Luria. Ls vygotsky and the problem of localization of functions. Neuropsy- chologia, 3(4):387-392, 1965.
Aman Madaan and Amir Yazdanbakhsh. Text and patterns: For effective chain of thought, it takes two to tango, 2022. URL httptttss://axiiiii org/ abs/2209, 07 6686
Vincent Micheli and François Fleuret. Language models are few-shot butlers. arXiv preprint arXiv:2/04.0799 2021.
11

---

Published as a conference paper at ICLR 2023
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback, 2021. URL httph::///ss //arxi/
org/abs/2112. 09332
Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski. Jacob Austin, David Bieber, David Dohan, Aitor Lewkowyez, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show your work: Scratchpads for intermediate computation with language models, 2021. URL
wttpp/////.bbbb//..///./// 211120012220111
Scott Reed, Konrad ZoIna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards. Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. generalist agent, 2022. URL https
7/ //arxiv. www.xi.wwwg/ab////aoaaaaa wttp:///ggg.b......... 06175
Mohit Shridhar, Jesse Thomason. Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10740-10749, 2020a.
Mohit Shridhar, Xingdi Yuan, Hausknecht, Alfworld: Aligning text and embodied environments for interactive learning, arXiv preprint arXiv:2010.03766 2020b.
Côté, Yonatan Bisk, Adam Trischler, and Matthew
Kurt Shuster, Mojtaba Komeili, Leonard Adolphs, Stephen Roller, Arthur Szlam, and Jason Weston. Language models that seek for knowledge: Modular search & generation for dialogue and prompt completion. arXiv preprint arXiv:2203. 13224, 2022a.
Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung. Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Naman Goyal, Arthur Szlam, Y-Lan Boureau, Melanie Kambadur, and Jason Weston. Blenderbot deployed conversational agent that continually learns to responsibly engage, 2022b. URL
2208 03188
James Thome, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: large-scale 2018.
dataset for fact extraction and verification. arXiv preprint
Lev S Vygotsky. Thinking and speech. The collected works of LS Vygotsky, 1:39-285, 1987.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang. Aakanksha Chowdh- ery, and Denny Zhou. Seff-consistency improves chain of thought reasoning in language models, 2022a. URL wtw.::tpps/..mmm http///xvvvv/vva///vv/ wttpp/wwwww.org////////////////.
11171
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, and Denny Zhou. Rationale-augmented ensembles in language models. arXiv preprint 0rXiv:2207.00747, 2022b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903 2022.
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv: met.::9099.0000 2018
Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. Keep CALM and explore: Language models for action generation text-based games. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8736-8755 8754, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10.18653/v /200022mm0000mmmmmee 704.
URL https: //acl antho www.oloo...ogg22// wwwp://.gg220..... emn 1amnnll-maiii 704
12

---

Published as a conference paper at ICLR 2023
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. arXiv preprint arXiv:2207.01206, 2022.
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning, 2022. hRtps::////xxx org/ abs 2203.14463 14465
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompitn enables complex reasoning in large language models, 2022. URL
10625
Yunchang Zhu, Liang Pang, Yanyan Lan, Huawei Shen, and Xueqi Cheng. Adaptive information seeking for open-domain question answering. arXiv preprint arX1v:2099.06747, 2021.
13

---

Published as a conference paper at ICLR 2023
A ADDITIONAL RESULTS
A.1 GPT-3 EXPERIMENTS
PaLM-540B
GPT-3
HotpotQA (exact match)
29.4
30.8
ALFWorld (success rate %)
70.9
78.4
Table 5: ReAct prompting results using PaLM-540B vs. GPT-3 (text-davinci-O0) greedy decoding). On HotpotQA, we randomly sample subset of 500 validation questions. On ALFWorld. use all 134 unseen validation task instances, and use the best prompt set according to PaLM-540B.
We run additional GPT-3 (Brown et al. 2020) experiments to confirm ReAct prompting performance is general across different large language models. As shown in Table 5 GPT-3 (text-davinci-002, greedy decoding) consistently outperforms PaLM-540B on HotpotQA and ALFWorld, possibly because it is finetuned with human instruction following. This indicates ReAct prompting is effective across different large language models different tasks. The code for these experiments are at https: deeracttmm.... github. 1o/
A.2 REACT OBTAINS UP-TO-DADE KNOWLEDGE ON HOTPOTQA
Rotepot
(1d) Rekot: (Reazon Act)
Queatioon: flww many rooms are in the hotel that le home the cieque du Heledd show Hystaze?
thought need asazch Cirque du Seleil abon Rysters,
number
Botpotgg Label: 664
Outdated
X
(1a) Standaed
(ie) Act-Only
Similurs ("Cisque soleil"
Anavez: 3,000
Aot 1: Search roue do Sotetl
Che 1:
cirope du dales: te
Cenodian entertaineent corpany..
Bearch Dtystre (dlique du Boleil)
(1b) Con (Besson Only)
Act 2:
Lookup looksp (apa)
Hystère iny-steer) 1a Held
theatre
thought:
think step
2:
No
zemuite
Tresegre Telard lobell and Castro
by step. The hotel that 15 Subuil ahov Pyolaze
Aot 3:
Search (Mpererre 10: rque do
thought Hystare hald in custon thastze the Creasure Ialand lotell and Casino. So need to Creasuee Ialand lotel and Cazina next find the of in the hotel.
Solell shoe)
Coold find.
Tressuza Ialand. Reosauee Baland has 2,385
Act 4:
Bearch Horaeze Bolague
Act 3: Search! Ersuzrate tolond Hobal and Cazano
Teoaauze roland Hotal and Capinoo Nith 2, zoun and 220 suices
Obg 3:
Mystare dey-steers
Ansuws: 7,
stx.
Lookup Teland
thought 4: Pressure Teland Forel and Castno has 2,881 roces
and 220 antiee. So the answer is 3,364,
lend without anawaс]
Act 4: 1,44
Op-to-date
Figure 4: Another example HotporQA question, where the original label is outdated. Only ReAct is able to obtain the up-to-date answer thanks to real-world web interaction plus reasoning.
During trajectory inspection, also find that sometimes ReAct does not agree with dataset labels as the labels themselves could be outdated. For example, as shown in Figure [4] the question asks about the size of hotel, which increased from the HotpotQA construction time. While Standard and CoT give wrong answers due to hallucination, Act. fails despite the access of real-world web interaction, due to a lack of reasoning to guide how interact with the Internet for QA. Only ReAct is able to retrieve up-to-date information from the Internet and provide a reasonable answer. Therefore, better incorporation of reasoning abilities might benefit recent Internet-auugmented language models (Nakano et al.
2021 Lazaridou et al. 2022 Shuster et al. 2022a) for up-to-date task solving.
A.3 HUMAN-IN-THE-LE BEHAVIOR CORRECTION ON ALFWORLD
We also explore numan-in-the-loow interaction with ReAct, to allow a human to inspect and edit ReAct's 's reasoning traces. Figure shows that by simply removing hallucinating sentence in Act 17 and adding some hints in Act 23, ReAct can be made to change its behavior drastically to align with these human thought edits and succeed in the task. From human perspective, solving such task becomes significantly easier, from typing tens of actions to only editing a couple of thoughts, which enables new forms of human-machine collaboration. We note that such policy edit on-the-go
14

---

Published as a conference paper at ICLR 2023
ALdNoeLd
(B) Dalct (Ream thought ediE)
You ace in the adddle of roon, looking quickly accund you, you see aimchate a. cabinat 6, cabsnat draver: od drawee drespet partopecan anto stell zeychain safe.
the
Loyobain and wanch 2.
ACt Bley Find Eleat kepetain DD reed
Your task
zeychatn tron ccapec
0be fou plos op the koyshain free the draver A.
Ralet
leyenatn is/en suts
Oba 14: Too open the diawer 6. The drowce to open, to tt, you
Obe 221 You pat the sayetain infon the axde L.
Act 17:
Saice Aie find the, ancons keyshain satne in deanee 4s.
Whick Bisw EiSS the Eseat kayzhain O3 Heae, need
seyehain
Act go stepas:
eauioagzan Lll safe Lt1, sidecable lit,
koyahais whiihhia Likels
skole HelR
from draver
soychain laptop
Act 22:
put beychain, lnfen sate
Adc Shink (aoe a Find the cecond koyshaia (3), max,, ased to take
Aot 23
Bhins [Si you the fivoo cogetaree in cafe
aceend knsehatte ses oliesttly by an deaece
safe
Obe 28: Rou pick
tsychain 5 rrom coesuer
koyshaia (2). Mia,,
thes dessame
Obe 24:
Ant 25
drewez open, Io
Ioom
aoooned tospoiale 1P1 cood
2.
Aet 28: go sate
Obe The made za open, Ma you aee baychets statoe and
La/on
Oba 28:
28:
tate teychain
Bothlee RaspaEan
Act 29 2e0 kavebaie 3 Lo/on sade
Cbe 20u the Sagozain lezon
Figure 5: A human-in-the-lopp behavior correction example with ReAct in AlfWorld. (a) ReAct trajectory fails due hallucinating thought (Act 17). (b) By human simply editing two thoughts (Act 17, 23), the ReAct trajectory produces desirable reasoning traces and actions and succeeds.
is difficult for Act and previous RL methods, as a human cannot change the model parameters, and changing few actions might not edit the rest of the model behavior. This paradigm is also more than human dialogue to update the goal or subgoal as in [Huang et al. (2022b) while editing ReAct thoughts can do these, it can also modify the model's internal belief, reasoning styles, or anything the flexible thought space supports, for better task solving. We believe this is an exciting direction for human alignment and leave more systematic study as future work.
B EXPERIMENT DETAILS
B.I HOTPOTQA FINETUNING DETAILS
For all finetuning we use a batch size of 64. On PaLM-8B, we finetune ReAct and Act methods for 4, 0000 steps and Standard and CoT methods for 2, 000 steps. On PaLM-62B, we finetune ReAct and Act methods for 4, 000 steps and St andard and CoT methods for 1, 000 steps. We find ReAct and Act methods generally benefit from more training steps (and more training data), while Standard and CoT methods degrade soon after finetuning.
B.2 ALFWORLD IM-STYLE DETAILS
For the IM-style ablation, the same expert trajectories used ReAct are reannotated with dense external feedback thoughts within these trajectories, that limit ReAct -MM to only think about (I) decomposing the current goal and (2) the current subgoal that needs to be completed. In particular, ReAct lacks thoughts that (1) determine when a subgoal is completed (2) determine what the next subgoal should be (3) inducing the LLM to refer to its internal pretraining knowledge to identify where items can be within the environment.
15

---

of the Published as a conference paper at ICLR 2023
C PROMPTS
C.I HOTPOTQA
Hotpol Prompts
D ginn
Queatiion
elevation range the that the Colcrado orogeny exleads into?
5,5 7,000
fon
Musician and satirist Allie Goerta wrote a song about the "The S: mpanns character Milhouse, who Matt Groening named after who? Richard Nixon
Anawer
Question
Which documentary is about Finnish rock groups, Adam Clayzon Powell or The Saiman Gesture? The Saimaa Gesture
Answer
Question
What professien does Nicholas and Elis Kazan have in common? director, screenwriter, actor
Anawer
Question
Which magazine was started first Arthur's s Nagazine or Pirst for Komen?
Answer
Arthur's Magazine
Question
Heze Pavel Drysoan and Leonld Levin known for the same type of work? Yes
Answer
Question
What is the elevation range for the area that the eastern sector of the Colerado orogeny exteads into? Seelch[Ce oregeay]
Action
Ubservation
the Colorado crogeny was an episode of mountain building (an orogeny) in colerado and aurrounding bookupleasters aector]
Aot lon Observatioon
High Plains and called High Plains refers to of two distinet land regions: Search [High Plains (United States)
(Resul:
eastern sector
the Central Plains orogeny.
Action
Seighh(Higgh Plains]
Observation
Action
Doservatioon
gubregion the Great plains. High Plains rise in elevation from acound 8,800 to 7,000 rt (550 to 2, 130 a) [3]
The High Plains
Action
Finish[L, 803 to 7,000 ft]
Question
Musician and satirist Allie Goertz chgracter Milhouse, who Matt Groaning named after who?
song about the "ihe Simpsons"
Action
Search [Mihouss]
Ooservation
Xilhouse Mussclini Van flouten is recurring character in the Fox sninated Simpsons
television
by Famela sayden created by Matt
Croening.
Action
Lookup[named after]
Observatioo
(8esult 1 Milhouse named after U.S. president Richard whose middle name was lhous.
Action
[RichaRRichard Nixon]
Question
which documentary
Finnish rock groups, Clayson Powell or
Saimaa Gesture?
Action
Search Clayton Powell]
Obgervation
could find [Adam Clayton Fewall). Similar: ('Adam clayton Powell
ITT', 'Seventh Avenue iManhattani', "Adam Clayton Powell Js. State orfice Building',
Washington Powell', (film)', 'Giancarlo Esposito'].
Powell', 'Adam Clayton Powell
Action Doservat lon 2
Sezrch clayton Adam Clayton Powell is 1985 American decumentary fi divected by kichard kilberg. The film is about the rise and fall of influentisl Arrican-Anari polistetan Adam playtor Powell [3) as part of the P38 series The American Experience. Finish[The
[film)]
laber red
Action
Gesture]
Question
profeasion does Nicholas Ray and Kazan have common? Bearch[Nicolla 3ayl
Action Oonervation
Nicholan Ray (born Raynend Nicholas Kienzla Jr., August 7, 1911 Juna 16, American film recter, screenwriter, the 1955 film Rebel Kithout Cause.
best for
tearch[Elis Kazaal
Ooservation 2 Elia Kazan was American film and thertre diracter, producer, screenwriter actor.
Continued on next page
16

---

Published as a conference paper at ICLR 2023
Holpot QA Prompis Continued from previous page
Action Question Action obgervatiion
Finishidireector, screenwriter, actor] Which magazine sterted first Arthur's Nagazine or Pirst for Kemen? beerch(Arthur" Magazine]
Arthur's Magazine (1844-1846)
American literary periodient publ ished in Phi ladelllapia in the 19th century.
Seerch([Pist
Nomen]
Ooservation 2 Pizst for Homen is a woman's s magazine published by Bazer Kedis Group in the USA. [1] magozine was started 1989.
Action
Finish(Arthur's Magazine]
Questiion
Pavel Uryso'n and Teontd Tevin known for the type of work? [Pavel Crysohn]
Actionn Coservation
Pavel Gemuilovich Urysohn (February 1898 August 1924) was Soviet mathematician who is best known for his contributiens in dimension theory. search (Leonid Levin]
Action
Observation
scientist. Leonid Anstolievich Levin is amviet-americen mathematicion and computer
Action
[ynish
Question
What i8 the elevation range for the area that the eastern secto" of the Colerado orogeny exteads inte?
Thought
Let's think step by step. The into the High Flains. High Flains rise in clevatien from around 1, to 7,000 Et, 1,600 to 7,000 ft
sector of Celorado orogeny extends
1,800 7,000
Answer
Question
Musician and satiriet Allle Goertz wrote song about the "The Simpaons* character Milhouse, who Matt Groening named aftez who? Let's think step by step. Milhouse named after U.S. president Richard Nixen, sc the answer is Richard Nixon. Richard Nixon
Thought
Question Which documentary about Finnish rock groupa, Adam alaoon Powell or The Saimia Gesture?
Thought
let's think step by step. Adam Clayton Powell (Eilml is documentary about African-smericnt politician, not Finnish reck groups. the documentary about Finnish rock groups must instead be The Salmas Gesture. The Saimas Gesture
Answer
Qon
What profession does Nicholas Ray and Elia Xazan have in common?
Thought
let's think step by step. Professions of Nicholas Ray directer, screenerites, and actor. Professions of Elis Kazen are directes, produces, screeneritar, and common la director, screenariter, and actor. director, screenwriter, actor
profession Nicholas Ray and glia Kazan have in
Ansver
Qun Thought
Which magazing ine was started first Arthur's Magazine or First for Komen? let's think step by step. Arthur's Magazine started. in 1641. First for Women was started in 1989. 1844 (Arthur's a Magazine) 1989 (First for Nomen), Arthur's Magazine Arthur's s Magazine
started first.
Anawer
Question Thought
Here Pavel Urysoan and Teonid Levin known for the same type of work? let's think step by step. Pavel Urysohn is a mathematician. Leonid Ievin is nathematician and computer scientist. Se Pavel Urysohn and Leonid Levin have the same type of work.
ReAce
Question
What is the elevation range for the ares that the eastern secto" of the colerado orogeny extends into? need search colorado orogeny, find the that the eastern seatir of the Colorado oregeny extenda into, then find the elevation range of the area.
Thought
Action obaervat
Sezrch (cclorsda oregeny]
The Calorado crogeny was an episade mountoin building (an orogeny)
Solorado and aurrounding areas.
Thought 2
It does not mention the eastern sector. So I need to look up eastern
Action Ooservation
Lookupleaste:n sector] (Resul: the Central Plains orogeny.
The eastern sector
High Plains and called
The eastern sector of Colorado orogeny extanda into the Righ Platna. So need to search ligh Plains and rind its elevation range. Action lHighh[High Plains] Ooservat.ion
Thought 3
distinct land regions need to instead search llih Plains (Uhited States). search [High plains (United States)
Plaina refers
Thought 4
Action
Continued on next page
17

---

Published as a conference paper at ICLR 2023
Holpot QA Prompis Continued from previous page
Opservat Ion
The High Plains are subregion of the Great Plains. From east to weat, the
ligh Plains rise in elevation from around 8,800 to 7,000 Et (550 to 2, m) [3]
Thought 5
High Plaina rise in clevation from around 800 7,000 the answer 1a 1,800 to 5,003 tt. Finish([1 803
Action
7,000
Question
Goertz wrate a song about the "the simpsons" cheracter Xilhouse, who Matt Groening named after who? the question simplifies to "the Simpsons" cheracter Milhouse is named after who. only nead search Mi Milhouse and find who is named after. [Kihoue
Musician and satirist
Thought 1
Act Coservation
Milhouse Mussclini llouten recurring character
animated
televisicn series The Simpsons voiced by Famela sayden and created by Matt Groening.
Thought 2
the paregraph does tell who Xilhouse is named after, maybe can look up "nemed ofter".
iot Ooservation 2
Tookup (named after]
(Result 1 / 1) Milhouse was named aflez U.S. president Richard N.xon, whose middle name was Milhous.
Thought 3
Xilhouse was named after U.S. president Sichard Nixon, the answer is Richard Nixon.
Action
Vinish(kichars Nixen]
Question
Which documentary
Finnish rock groupa, Clayson Powell The Saimea Gesture,
Sainaa Gesture?
Thought
need search documentary is about Finnish rock groups. Search[Adam Clayton Powell]
clayton Powell
which
Action
Observatioo
could not find [Adam Clayton Pewall). Similar: ('Adam clayton Powel: TTT', "Seventh Avenue iManhattani', rAdam Clayton Powel 1 Jr. State office Building', 'Isabel WashingLon Powell', (film)', ' 'Giancerlo Esposite'].
Powell', 'Adam Clayton Powell
Thought Action 2 Ooservationn
dccumentary, can search Clayton Foxell (film).
Sea Bearch[Aa (Adam Clayton Powel1 11 [fi]m]]
Adam Clayton Powell is 1985 American documentary film disected by Richard Kilbarg. The film about the and fall of influential as part of
policician Adam Claytor Powell Jr. [3] It waa later aired The American Experience.
Thought 3
Adam Clayton #owell (film) is a documentary about an africen-American politician, not Finnish rock groups. groupa must instead be The Saimaa Gesture.
the documentary about Finnish
Action
Finish[The Saimad Gesture]
Question
What profession does Nicholas Bay and Elia Kazan have in common? need search Nicholas Ray
Thought
Kazan, their professions, then find the profession they have in common. Search(Micho))
Action
Ray]
Ooservatioo Lon
Nicholas Ray (born Raynond Nicholas Klenzle Jr., August 1911 June 16, 1979) was an American film directer, screenwriter, and actor best known for the 1955 film Rebel without Cause.
Thought 2
Professiens of Nicholas Ray are director, screenwriter, and actor. seed search Elia Kazan next and find his professions. Secrch(Elia Kazan]
Action Doservat ton
Elia Kazan was American film and thestre rirector, producer, screeneriter and actor.
Thought 3
Professiens of Elia Razan are director, producer, screenwriter, and actor. profession Nichclas Ray and Eliz Kazen have screenwriter, and actor.
is cirector,
Action
Finish[diresterr, screenkriter, actor]
Question Thought 1
Which magazine was started first Arthur's Magazine or First for Komen? need to search Arthur's Magazine and Pirst for Nemen, and find which started first.
Action Ooservation
Search(Arthur's Magazina]
Arthur's Magazine (1814-1846) was an American literary periodica: published Philadelphia Arthur's Magazine was started in 1844. need to search First for Romen next.
century.
Thought 2
Action Observation 2
Sesrch[first for Nemeal
First for Women is a woman's a magazine published by Baser Nedia Group in the started in 1989.
U1A. magazine Fizst for women was started in 1989. 1844 (Arthur's Magazine) < 1939 (Fizat for Women), Arthor's Magazine was atarted first. Firtsh(Arthar's Magazine]
Thought 3
Act
Question Thought 1
Pavel Urysoin and Leonid Levin known for the type of work? need to search Pavel Uryschn and Leanid Levin, find thair typea of work, rind they the Secrch[Paval Urysohn]
Action
Continued next page
18

---

Published as a conference paper at ICLR 2023
Holpot QA Prompis Conlinued from previous page
Ooservation 1
Pavel Demuilovica Uzysohn (February 3, 1898 August 17, 1924) was Soviet nathematician who best known ror his contributions in dimenaion theory, Pavel Uzysoan is mathemetician. need to search Leonid Levin next and find its type of werk.
Thought
Action Coservat lon 2
Sezrch Levin]
seonid Anatolievioh Tevin 18 Boviet-Amerion mathematician and computer scientist.
Thought 3
Leonid Levin is a mathematicien and computer scientist. Bo Pavel Uzysoha Leonid Pinish[yes]
same type wark.
Action
19

---

Published as a conference paper at ICLR 2023
C.2 FEVER
FEVER Prompis
glos
Determine 15 there 1a Observat ion that SUPPORTS or REPUTES Claim, or LC there is NOT ENOUGH INFORXATION.
Claim
Nikolaj Coster-Kal dal worked with the Fox Broadcast Company, SUPPORTS
Claim Answer
Stranger Things is in Bloomington, Indiana. REFUTES
Besutiful reached NOT ENOUEH ONe0
the Billboard lott 2003.
Answer
Determine if there is Observation that SUPPORTS or REFUIES Claim, iE there is NOT ENOUGH INFORKATION.
Nikelaj Coster-Kalóau
Broadcasting Compony.
Act lon2
(N Coster-Xaldau)
Ooservation
Nikelaj Killiam Coster Kaldau (bozn 27 July 1970) is Danish actor and produces. de gradusted from the Denish Nationsl School of Performing Arts In Copenhagen in 1993, 1933,1) [1] and had his breakthrough zole in Denmark with film Nightwatch (1994).
played Caine Lanaister drema series Came of Thrones, for which he received two Primetime kmmy nominations
fantasy
Outstanding Supporting Actor Coster-Naldau has appeared in numerous filma in nis native Denmark and Scendinavia, including lleadhunters (2011) and Theusand Tines Good Night (2013). the U.S, his debut film role in the film slack Back (2001), playing Medal of Honor recipient Gary Gordon. [2] Re then played detective
Drama Series..
short-lived Fex television appeared in the 2009 sox television film Virtuslity, originally intended as pflot. [SUFFO()S
(2008),
Action
Claim Action Ooservation
sanagger Things is set in Bl cominggoo,, Tndiana. Search (Stranger Things!
stranger Things is an American science fiction horror drama television seriea created by the Duffer Brothers. Set the 1980s, primartly in the fictional town of llawkins, Indiana, the series centers number of mysteries and supernaturel events oczurring around the town and their impact ensemble of child and adult charasters. Finish (RFFUTES]
Act
Claim
Besutiful reached number two on the Billboard Hot 100 In 2303. Sesrch[[eeattfull] Could not find [seautiful), similer: ('Beautiful', 'gesutiful, Seautiful', 'A Reautiful Mind (film)', 'Reautifu" (Christina Aguilera song)', "Ie Ta Besutiful"].
Action
observation
Search [feautiful iChristina Aguilera song)
Doservation
"Beautifull 13 fourth studio album, Stripped (2002).
recorded by American singer Christina Agui for her
Action Ooservation
(sillbaad #et 100]
(Ressit 3) The song peaked at number two on the Billboard Not 100 in the United States, Finish[MOT SNOUGH INPO!
certified Gold for 003 unita shipped.
Action
neternine 1 there 1a Observation fon that SUPPORTS or REFUTES Cla'm, or 1f there is NOT ENOUGH INFORMATION.
Nikelaj Kaldau werked
Broadcasting Company. Nikelaj william Coster-kaldat appeared in the 2009 Fox television film Virtuality, so has SUPPORTS
Thought
the Fox Rroadoasting Company.
Answer
Claim
BLiangez Things is set in Bicomington, Indiana. Strangez Things Blocmington, indiana.
Thought
fictional town Hawkins, Indiana,
Continued on next page
20

---

Published as a conference paper at ICLR 2023
Fever Prompts Continued from previous page
#esutiful reached number
Billboard not 2301.
Thought
The song peaked at number two on the Bi Billboard Bot. 103 in the United States, but not sure if it was in 2003.
Anawer
ENOUCH
ReAce
Determine
observation SUPPORTS oz there is NOT ENOUCH INFORMATION.
Claim, if
Claim Thought
Nikelaj Coster- worked with the Fox Broadcasting Company. need search Nikolag Coster-Kaldau and find if he has worked with the Fox Brosdcasting Cempany.
Seerch (Nikolaj Coster-Naldau)
Ooservation 1 Nikelaj Killiam Coster-Kaldoi (born 27 culy 1970) is a Canish actor and producer. He graduated from the Danish National School of Performing Arts in Copenhagen in 1393, and had his breskthrough role in Denmark with the film Nightwatch (1994). ke pleyed caime Lannister in the asú fantasy droma series Came Thrones, for which he received Award nominaticns for Outstanding Supporting Acter in Drama Series.. coster-Naldau
Primetime Enmy
appeared numerous films in
native Denmark
Scandinavia, inclucing Headhunters (2011) and A Thousand Times Good V ght (2013). In the U.S, his debut film role was in the war film Black llakk Down (2001), playing Xedal of Honor recipient Gary Gordon. detective in the short-lived Fox television series New Amsterdam (2008), and appeared the 2009 bel selevis'on pilot.
le then played
Virtualityy originally *ntended
Thought
television virtuality", ahould have worked with the Fox Broadeasting Company, (SUFPORSS
Recause he "appeared
Action
Claim Thought 1
Stranger Things is Bloomington, should search for Btranger Things, and see if it is set in Bloumington, Tndiana.
Action
Sesach[[ Things]
Ooservation
Stranger Things is an American science fiction horror drame television series created the Duffer Brothers. Set the 1980s, primarily in
the fictional town of Hawkins, Indiana, the series centers on a number of their impact
systeries supernatural on an ensemble of child and adult characters. The observation says that Indiana", it is set in Bloemington. Finish [REFUTE]
occurring
the
Thought 2
set in a "fictional sown of Hawkins,
Action
Claim Thought 1
Desutiful resched number need to search the song Beautiful and find if it reached number two on the Billboard Bot 103 2003.
the Billboard Bot 2303.
Action Observation
Search Could not find [seautiful), similer: ('Beautiful', 'Besutiful, Seautiful', 'A Reautiful Mind (film)', 'Reautifu". (Christina Aguilera song)', "Iiif Ta Besutiful"].
Thought
suggestiens, should find the song.
"Beautiful (Cheistina Aguilera song)
Action Observation
Sesrchi[eeautfful iChristina Aguilera song)]
"Seautiful* is seng recorded by American singer Christins Aguilera for her fourth studio album, Stripped (2002).
Thought
need look up *3 1bboab Not 100" Lind if it zeached number Lwo on it in 2003,
mentioon lborbo,
Lookup [Billboard
100]
Coservation 3
(sesult / 3) The song peaked at number two en the gillboard sot 100 in the United States, where was certified Gold for 500, 003 units shipped. it says
Thought
the Billbeard hot 100, but if it wad in 2003. om not sura if this claim is true or not. Finish (NOT SNOUGH INFO]
pesked number
Actioo
21

---

Published as a conference paper at ICLR 2023
C.3 WEBSHOP
Table Prompts used for the Webshop experiments in Section 4 We use the prompt on the left for Act, and use the prompt on the right (add reasoning) for ReAct.
Instruction: would like 3 ounce bottle of bright citrus deodorant for sensitive skin, and price lower than 50.00 dollars
Act
ReAct
Action: search (3 bunce bright citrus deudorant sensitive skin]
Action: search ounce bright citrus deodorant sensitive skin
Observation:
Observation:
Back to Search
Back to Search
Page 1 (Total results: 50)
Page (Total results: 50)
[Nxx]]
[Next
[B08GGWRCJ]]
[R08GWRCCJ
Bright Citrus Deodorant by Earth Mamai Natural and Sale Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic
Bright Citrus Deodorant Earth Mama Natural and Sale for
Sensitive Skin, Pregnancy and Brenstfeeding, Contains Organic
Calendula 3-Ounce
Calendula 3-Ounce
$10.99
$10.99
[B078GTKXXY]
[R078GTKVXY]
Ginger Fresh Deodorant by Earth Mama Natural and Safe for
Ginger Fresh Deodorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy and Brenstfeeding, Contains Organic
Sensitive Skin, Pregnancy and Breastfeeding, Contains Organic
Calendula 3-Ounce
Calendula 3-Ounce
$10.99
$10.99
[B08KBV4XXN]
[B0KBBV4XXN]
Barrel and Oak Aluminun-Free Deodorant, Deodorant for Men Essential Oil- Ell-Baeed Sceni, 24-Hour Oder Protection. Cedar Patchouli Blend, Geatle Sensitive (Mountain Sage,
Barrel and Oak -Aluminum-Fro Deodorant, Deodorant for Men Essential Based Sceni, Ho-r Odar Protection, Cedar & Patchouli Blend, Gentle Seusitive Skin (Mountain Sage, 2.7 oz. 2-Pack)
2.7 oz, 2-Pack)
Action: click [B07SGWRCUJ]
Action: think B078GWWRU BO7SGTKVXY are bright citrus deodorant less then 50 dollars. can check B0780WRCU lirst. Observation: OK.
Observation:
Back Search]
Prev]
[assorted scents][bright citru]][calming lavender[[ginger
Action: click B078GWRCIJ
[fresh] simply non-scents]
Observation:
[travel set (4-pack)]]3 (pack 3-ounce
Back Search
pack)]
Prev'
Bright Deodorant by Earth
[assorted scents][hright citru]]calling lavender][ginger
Sensilive Skin, Pregnancy and Breastfedingg, Contains Organic
[fresh] simply scents)
Calendula 3-Qunce
size [travel set (4-pack]][3 ounce (pack of [)][3-ounce (2
Price: $10.99
-pack)]
Rating: N.A.
Bright Citrus Deodorant by Earth Mama Natural and Safe for Sensitive Skin, Pregnancy Breastfeeding, Organic
[Description]
[Features]
Calendula 3-Ounce
[Reviews] [Buy Now]
Price: $10.99
Rating: N.A.
[Description]
Action: click[bright citrus]
[Features]
Observation: You have clicked bright citrus.
[Reviews] [Buy Now]
click[3 ounce (pack of ()] Observation: You have clicked 3 ounee (pack of
Action:
Action: think[For 3 ounce bottle bright doodorant for sensitive skin, the item has options "hright citrus" and '3 ounce (pack of ()' and seems good to buy.] Observation: OK.
Action: click[Buy Now]
click[bright citrus] Observation: You have elicked bright citrus.
Action: click[3 (pack D)]
Observation: You have clicked ounce (pack of l).
Action: click[Bun Now]
22

---

Published as a conference paper at ICLR 2023
C.4 ALFWORLD
Table 7: An Act: prompt on the ALFWorld clean task. No thoughts are provided.
the middle 13, cabinet 12, cabiret 11, a cabinet 13, a cabinet 9, a net 8, cabinet cabinet 4, cabinet 3, cabinet 2, a cabinet 1, a coffeemsching 1, countertop 1, diningtable 1, a drawer 1, fridge 1, garbagecan 1, à micrewave 1, shelf 3, a shelf 2, à shelf 1, a sinkbasin 1, stoveburner 4, stovebaraer 3, stoveburaer 2, stoveburner 1, and toaster 1. Your task is to: put a clean lettuce in diningtable.
Looking quickly around you, you
cabinet
cabinet 6, cabinet
go fridge
The fridge 1 18 closed.
open fridge
The fridge is In it, see cup potato 3, and a potato 2.
You the fridge
go diningtable
on the diningtable 1, you see a apple 1, bread 1, a buttarknife 2, cup 2, fork knife 2, knife 1, ladle 1, lettuse peppershake:: 1, spatula 3, Lomato 2, and a tomato 1. letture from diningtsble You pick the lettuce from the ciningtable 1.
mug mug 1, 2,
> go to sinkbasin 1
apple 2, ladle 2, spoon 1, and tomato > clesn lettuce with sinkbas'r
the sinkbasin you
clear.
lettuce using the sinkoaein
> think: New I clean letture (1). Next, I seed to put it in/oa diningteble 1.
go to diningtable
On the diningtable 1, you see a apple 1, bread 1, butterknife 2, cup 2, fork knife 1, ladle 1, mug mug pan popperahaker 1,
knife apatula 3, a tonato 2, and tomate 1.
put lestuce in/on diningLable
You put the lettuce 1 in/on the diningtable 1.
23

---

Published as a conference paper at ICLR 2023
Table 8: An ReAct prompt on the ALFWorld clean task.
middle room. Looking quickly around you, you 13, cabinet 12, cabiret 11, cabinet 19, cabinet 9, casinet 8, cabinet a cabinet 6, cabinet 5, a cabinet 4, a cabinet 3, cabinet 2, a cabinet coffeemachine 1, countertop 1, diningtable 1, dzawer i, fridge 1, gerbagecan L, a microwave 1, a sheif 3, shelf 2, a shelf 4, a sinkbasin 1, a and toaster 1.
cabinet
stoveburrer 4, stovaburner 3, stoveburner 2, stoveburner Your task is to: put a clean lettuce in diningtable.
Lind and take lettuce, sinkhasin, then put it in diningtable.
think:
clean it with
Ox.
think: Firet seed to find lettuce, à lettuce is likely appear in fridge (1), diningteble (1), sinkbasin (1), stovaburaer (1-3), cabinet (1-13). check oR.
one, starting with fridge
go fridge
The fridge 1 is closed.
open fridge 1
You the fridge 1. The fridge is open. In it, you cup egg
potato 3, and a patato 2.
diningtable
On the diningtable 1, you see apple 1, pread 1, butterknite 2, cup 2, 2, knife peppershaker 1, spatula 3, temato 2, and a tomato 1.
knife 1, ledle 1, letture
mug mug pan 2,
think: Now I find lettuce (1). Next, need to take is.
08.
> take Lettuce from diningtable i
pick up lettuce from ciningtable
> think: Now I take lettuce (1). Next, I seed to to sinkbasin (1) and clean
ox.
> go to sinkbasin
on the sinkbasin you apple 2, ladle 2, spoon 1, and tomato
> clean lettuce with sinkbagin
lettuce using the sinkoaein > think: Now I cleen a letruce (1). Next, need to put is in/en diningtable 1.
> diningtable 1
on the diningtable 1, you see a apple 1, bread 1, butterknife 2, cup 2, fork knife
ladle 1, mug mug pan popperahaker 1, apatula 3, a tomato 2, and tomato 1.
knife
put lettuce in/oa diningtable
You put the lettuce 1 in/on the diningtable 1.
24

---

Published as a conference paper at ICLR 2023
Table 9: An ReAct IM prompt on the ALFWorld clean task.
middle room. Looking quickly around you, you 13, cabinet 12, cabiret 11, cabinet 19, cabinet 9, casinet 8, cabinet a cabinet 6, cabinet 5, cabinet 4, cabinet 3, cabinet 2, cabinet coffeemachine 1, countertop 1, diningtable 1, dzawer i, fridge 1, gerbagecan 1, microwave 1, a sheif 3, shelf 2, a shelf 4, a sinkbasin 1, stoveburrer 4, stovaburner 3, stoveburner 2, stoveburner Your task is to: put a clean lettuce in diningtable.
cabinet
and toaster 1.
Lind and take lettuce, clean it sinkhasir, then put it in diningtable.
think:
Ox.
think: First seed to find lettuse.
ex.
fridge
The tridge 1 is closed.
think: Firet
find lettuce.
ex.
open fridge 1
You the fridge 1. The fridge is open. In it, you cup egg
potato 3, and a potato 2.
think: First need find lettuse. OK.
go diningtable
on the diningtable 1, you see apple 1, bread 1, butterknife 2, 2, fork 2, a knife 2, a knife 1, ladle 1, lettuce 1, a mug 2, a mug 1, pan 2, peppershaker 1, spatuls 3, tomato 2, and > think: First need to find a lettuce.
1.
ox.
s take lettuce fzom diningtable
pick up letture
diningteble
> think: need to clean this lettuce (1) using sinkbasin 1,
CX.
sinkbasin
On the sinkaasin 1, you aea a apple 2, a ladle 2, spoon 1, and a tomato 3. this lettuce using sinkbasin sinkbasin
think:
CX.
lettuce
You clear the lettuce 1 using the sinkaasin 1,
> think: need to puz this cleaned lettuce (1) in/en diningtable 1.
0K.
> go to diningtable
diningiable you see apple bread butterknife cup Tozk 2, knife 2, a knife 1, a ladle 1, mug 2, a mug 1, a pan 2, peppershaker 1, spatuls 3, a tomato
tomate > think: need to this cleaned lettuce (1) ir/on diningtable 1.
ek.
put lettuse 1 in/on diningtable :
You put the lettuce 1 in/on the diningtable 1.
D TRAJECTORIES
D.1 FEVER TRAJECTORIES
We provide some trajectories produced by ReAct, CoTand Acton several randomly selected exam- ples from the FEVER dev below. Search results are omitted to space.
25

---

Published as a conference paper at ICLR 2023
Example 2491 (gt REFUTES)
ReAct
Claim:
Bermuda Trlangle loosely-derined region in the Paciric Ocean.
Thought 1:
I need Lo seazea Bermuda Triangle and find Ic it is in the Pacific Oceaa. Seareh(eermuse Triangle] seportedly haunted locations... The observation saya that Ocean, so it is not in the Pacific Ocean. Finish (RRFOTRR
Action L:
Doservation 1:
Thought:
in
part the North Atlartio
Act 2:
Ooservation 2:
Spisode rinished
Reb
Claim:
Sermuda Triangle loosely-decines region the Paciric (eerchd Triangle] Reportedly (REFUEE spisode finished
Action
observation
locations:....
Action 2:
Ooservation 2:
Claim:
Bermuds Triengle is loosely-definee region in the Pacific Ocean.. Let's think step by step. Bermuda Triangle lcosely-defined regior. in the Atlantic Ocean. The Pacl fie neean is not the Atlantic Ocean. REFUTES
Thought:
Answer:
Example 8908 (gt: SUPPORTS)
ReAct
Monenoke
Thought 1:
need to search princess Kenonoke and find if it is film. Search (Prinoesa Mononoke]
Action i
Doservat lon 1:
Princess Mononoke
Thought 2:
Frem the observation, it says that Princess Kononcke is film.
Action 2:
(SU
Ooservalion 2:
Episode finlshed
Claim:
Princess Monenoke is film. Search[Priacess Menonske] #rincess Nonenoke Finiah Finich (SUPPORT3)
Action 2:
observation 1:
lon 2:
spisode finished
Claim:
Princess Nonenoke is a film.
Thought:
Let's think step step. Frincess Xonenake movic, and movies #1 1ma
Answer:
SUPFORIS
Example 1951 (gt: REFUTES)
ReAct
Claim: Thought
of the American spece program. I need to search Soyuz and find if it was part of the American space program.
Soyuz
Action 2: Ooservation Thought 2:
Bearch[Soyus]
Soyuz is transliteration... It does not menticn the American space program, I need to search "American space program" to find if it was part of it. search (American space program] The space policy...
Action 2: Doservat ion 2:
Thought
mention Soyus,
it this claim la not.
Action 3:
Finish(NO7 DNOUGI] INFO]
observation
spisode
Claim:
Soyus was part of the American space program. [eaych(Soyus]
Action 2:
coservation 1:
Soyuz is transliteration... (ini ENOUGH INSU] spisode fintahed
Action 2:
Obgervation 2:
Eog
Soyuz was part
Amesican space program. Let's think step by step. Soyuz is Russian spacecraft. The American space program
Thought:
NASA and International Space Station. SUPFORTS
worked zogether
Answer:
26

---

Published as a conference paper at ICLR 2023
Example 3208 (gt: REFUTES)
ReAct
Claim:
Reign Over Me is American ri made in 2010
Thought
I need Lo seazea for Reign Ovez Ne and find if it is an American film made 2010.
Action 2: Doservat Thought 2:
search (Raign over Mc]
Reiggn over Me
The observation says that it is an American film made in 2007, so it is not made 2010.
Action 2: Observation 2:
FRRIUT(REPUTES]
Episode finished
Claim:
Reign Over Ne is American film made in 2010. Seaich(Keign Me] [SUP2RR spisode finished
Action 2:
Action 21 Observation 2:
Claim:
Reign Over Ke is American film made in 2010. Let's step by step. sirst, Reign Over it 1s. Second, was it made in 2010? Yes, it was. SUPFORTS
Thought:
film? Yes,
Answer:
D.2 ALFWORLD TRAJECTORIES
We provide an example trajectory of ReAct, Act, and ReAct IM on the same ALFWorld game, pick clean then place in recep-knife-Nonee -ot---oooo--oottoooo--///// trial T20190909 11044001111411111111111111111111 1105500011114446677111
D.2.1 REACT TRAJECTORY
ReAct is able correctly find the knife, clean it, and then place it on countertop.
27

---

Published as a conference paper at ICLR 2023
ReAct ALFWorld Injectory
You are in the middle of room. Leoking quickly around you, you see cabinet 6, cabinet 5, a cabinet 4, cabinet 3, a cabinet 2, cabinet 1, coffeemachine 1, countestop countertop 2, countertop
drawer 3, drawes 2, drawer fridge -, garbagecan 1, microwave i, shelf 3, a shelf 2, a shelf i, a sinkbasin stoveburner and toaster
stoveburner
stoveburner
stoveburner
Your Lask Lo: put clean
couatertop.
> think: To solve the task, need to find and take knife, then cleen is with sinkbasin, then put ox.
countertop.
> think: First I seed to find a knife. knife is more likely to appear in cobinet
(1-6), drawer (1-31, countertop (1-3), fridge (1), garbagaean (1), shelf (1-3),
sinkbasin (11, atoveburner (1-4), tosater (11, I can cteck one by one, starting with
cabinet
0x.
go
on the cabinet 1, you bowl 1.
> go to cabinet
closed.
> go to cabinet 3
cabinet 3, you see glassbettle
> cabinet 4
On the cabinet 4, you see mug 1,
cabinet 5
The cabirnt 5 i5 closad,
go cabinet
The cebinet is closed.
> go
The drawer 1 is closed.
> go drawer
The drawer 2 closed.
> go to drawer
closed.
> countertop 1
countertep 1, you see Lettuce 2, mug peppershaker spoon 2. > countertop
On the countertop 2, you see cug 1, dishaponge a glaasbot:le a knife 1, a
plate potsto 3, and statue
>
think: Now I find knife 11). Next, need to take it.
> take knife from countersop 2
pick up
countertop 2.
> think: Now I take knife (1). Next, need to sinkbasin (1) and clean it.
OX.
sinkbasin
On the sinkvaeia 1, you see a fork 3, lettuce 3, and spatula 2.
with sinkbasin
You clean the knife 1 using the sinkbasir. 1.
think: Now alean knifa (1), Next, I to put in/on countertop 1, ox.
> go to countertop
the countertep 1, you lettuce 2, mag peppershaker 1, and spoon 2.
> put knife 1 in/os countersop 1
put knife in/on countertop
D.2.2 ACT TRAJECTORY
Act is able to correctly find the knife, but tries to clean it using a sinkbasin before actually going to the sinkbasin. From there the model is unable to recover, and is stuck repeating the same sequence commands. With ReAct, an existing thought would have concretely identified that the knife had been taken, and that the next subgoal was go to sinkbasin clean it.
28

---

Published as a conference paper at ICLR 2023
Act ALFWorid Injjeciory
You are in the middle of a room. Leoking quickly around you, you see cabinet 6, cabinet 5, a cabinet 4, cabinet 3, a cabinet 2, cabinet 1, coffeemachine 1, countestop countertop 2, countertop
drawer 5, drawez 2, drawer fridge -, garbagecan 1, a microwave i, shelf 3, shelf 2, a shelf A, a sinkbasin stoveburner stoveburner and toaster
Stoveburner
stoveburner
Lo: put > countertop 1
Your
knife couatertop.
countertop 1, you see lettuce 2, mug a peppershaker
spoon 2.
countertop
on the countertep 2, you see a cup 1, dishaponge i, a glasshetzle 3, a knife 1, plate potato and statue
> take knite from countersop 2
pick up knife
countertop 2.
> countertop
the countertop 1, you see lettuce 2, mag poppershaker 1, spoon 2.
clean knife 1 with sinkbasin
Nothing happers.
countertop
On the countertep 3, you see bread 3, butteckniTe 2, cellphone 1, creditoard fork 2, houseplant tomato 1, and vase 2.
spatule 1, tomate tomato
take kri
countersop
Nothing happens.
> go to countert.pp 2
countertop 2, you cup dishaponge glaasbot:le 2, potato 3, and a statue 1.
take knife
countersop
Nothing hoppers.
go countertop
lettuce 2, mag peppershaker 1, and spoon 2. > clean knife 1 with sinkbaain
countertop 1, you
Nothing happens.
> countertop 3
butterknife cellphone 1, crediteard countertep 3, you see knife apstula 1, statue 3, tonato 3, tomato a fork 2, houseplant
2, a tonato 1, and vase 2,
> take knife from countersop
Nothing happens.
go countestop
on the countertep 2, you see cup 1, dishsponge 1, glassbet:le 3, a plate 2,
potato 3, and a statua
take knife from countersop 2
Nothing happens.
countertop
On the countertep 1, you see a lettuce 2, a mug 2, pepperahaker 1, and spooa 2. sinkbasin
Nothing happers.
go countertop
On the countertep 3,
bread butterknite cellphone 1, erediteard a fork 2, houseplant 1, a knife 2, a spetule 1, statue 3, a tomate 3, a tomato tonato and
2.
> take knife from countersop
Nothing happens.
>
countertop 2
On the countertop 2, you see cup 1, dishaponge 1, glaasbottle plate 2,
potato 3, and statue
> take knife from counter:op 2
Nothing happens.
> countertop
countertep 1, you see lettuce 2, mag peppershaker spoon 2. > clean knife 1. with sinkbasin 1
Nothing happens.
D.2.3 RRACT-IM TRAJECTORY
ReAct- - IM is able to correctly find the knife, but is unable to clean it well. The LLM then gets stuck trying to place this knife on countertop. One possible issue the incorrect thought think: I need to find clean knife. which seems to trick the LLM in believing that the knife is already cleaned. In ReAct, this sequence of subgoals would be concretely captured by each thought.
29

---

put the knifa in/on the countertop 1. Published as a conference paper at ICLR 2023
ReAct- 12 ALF trajeccyy
You are in the middle of room. Leoking quickly around you, you see cabinet 6, cabinet 5, a cabinet 4, cabinet 3, cabinet 2, cabinet 1, coffeemachine 1, countestop countertop 2, countertop
drawer 3, drawes 2, drawer fridge -r garbagecan 1, a microwave i, shelf 3, shelf 2, a shelf a sinkbasin stoveburner and toaster
stoveburner
stovehurner
stoveburner
Lo: put > think: To solve the task, need to find and take clean knife, then put it in countertop..
Your
knife couatertop.
ox.
> think: need to find a clean knife.
ok.
> go to countertop 1
lettuce 2, mug pepperehaker and spoon > think: need to find clean knife.
countertep 1, you
countertop
On the countertop 2, you sse a cup 1, dishaponge 1, a glasshotzle 3, a knife 1, a statue > think: need to find a clean knife.
potato
> take knife from countersop 2
countertop 2, think: need put this knife (1) in/on countertop
pick up the
OK.
countertop
On the countertep 1, you see lettuce 2, mug 2, a peppershaker 1, and spooa 2.
think:
need puz knife (1) countertop
OK
> put knife in/on countersop
> think: need to pot this kni (1) 'r/on countertop 1,
OX.
> put knife 1 in/on countersop 1
Nothing happens.
> think: need to this knife (1) 'r/on countertop 1,
OX.
put knife in/on countersop
Nothing happers,
think:
puz knife in/on countertop in/on countersop
cx.
> put
Nothing happens.
think: need to pu: this knite (1) Ln/on countertop 1. ox.
> put knife 1 ia/os countersop 1
Nothing happens.
> think: need to this knife (1) in/on countertop 1. ox
put knife 1 in/on countersop 1
Nothing happens.
think: need puz this knife (1) in/on countertop ox.
30

---

Published as a conference paper at ICLR 2023
D.3 WEBSHOP TRAJECTORIES
Table 10: Example trajectories for Webshop predicted by Act and ReAct. Compared to Act, ReAct uses reasoning to find products that satisfy all target attributes.
Instruction:
get me sixteen pack of apple cinnamon freeze dried banana chips, and price lower than 50.00 dollars
Act
ReAct
Action: search sixteen pack apple cinnamon freeze dried banana
Action: search sixteen pack apple cinnamon freeze dried benana chips]
chips]
Observation:
Observation:
Search
Back to Search
Page (Total resalts:
Page (Tolal results: 50) Next]
[Nex]]
[80061VVZEE]
[B0061VVZEE]
Brothers-Ali ALL-Natural Fruit Crisps, Strawberry Banana, 0.42
Brothers-ALL ALL-Naturl Fruit Crisps, Strawberry Banana, 0.42 Ounce (Pack of 100) $85.0
Ounce (Pack 100)
$85.0
[8096H2P6G2]
|B096H2P6G2
Moon Fruit Freeze Dried Fruit Snacks. Fruit Snacks for Kids
Moon Fruit Freeze Dried Fruit Snacks. Fruit Snacks for Kids (Variety Pack)
(Variety Pack)
$18.99
$18.99
[B092JLLYK]]
Nature's Turn Freeze-Dried Fruit Snacks Banana Crisps Perfoct For School Lunches On- Th-The-Go Snack No Sugar Added, Non GMO, Gluten Free, Nothing Artificial (0.53oz) Pack
Nature's Turn Froeze-Dried Fruit Snacks Banana Crisps - Perfect For School Lunches On-The-Ch Snack No Sugar Added, Non GMO, Gluten Free, Nothing Artificial (0.53mz) Pack $12.99
$12.99
Action: click [B066LIVEZE
think[R006| TVEZE strawherry hanana, not apple cinnamon. B096H2P6G2 is fruit snacks, not freeze dried banana chips.
Observation:
Back Search
BOW2JLLYK6 banana crisps, not apple cinnamoo. can check
Prev]
BO92JLLYK6 first.
dlvvo name [asion paan apple & cinnmmon]strawberng Istrawberry
Ohservation: OK.
banana] Brothers-Al AL.L-Natural Fruit Crisps, Straweerry Banana, Ounce (Pack of
Action: click [092JJLLYK]]
Price: $85.0
Observation:
Ruting:
BBak Search
[Description] [Features]
Prev]
flavor name [apple][apple
[Reviews]
[strawberrylIstee berry banana][sampler variety pack||mega variety pack][orchard variety pack] of [116]
[Buy Now]
size (0.53 (pack 6)](0.33 (pack 8)][0.53 ounce (pack
Action: click|Buy Now]
Snacks Crisps Perfect
For School Lunches or On-The-Go Snack No Sugar Added, Non
GMO, Gluten Froc, Nothing Artificial (0.53o2) 6-Pack
Price: $12.99
Rating: N.A.
[Description]
[Features]
[Reviews]
[Buy Now]
Action: think]Far
of apple cinnamon freeze dried bunanu chips, the item has options 'apple cinnamon' and '0.53 ounce (pack 167' and good buy.]
Ohservation: OK
Action: click[applc cinnamon]
Observation: You have clicked apple cinnamon.
Action: click[0.53 nunce (pack of 16) Observation: You clicked
(pack 16).
Action: click[ Buy Now]
Score: 0.125
Score: 1.0
31

---

Published as a conference paper at ICLR 2023
E MORE ANALYSIS
E. SUCCESS AND FAILURE MODES ANALYSIS
We provide some examples corresponding to the success and error mode analysis given in Sec. 3.3 Search results and non-representative steps omitted to reduce space.
Success: Thie positive
ReAcE Question:
Author David Chancfé collaborated with a Navy admiral served the ambasaader to the Unitad Kingdom under which President?
Thought
I need to search David Chanoff and find the U.S. Navy admiral he collaborated with.
Act lon ": Ooservation Action 2:
Search (havidd Chanoff]
David Chanoff
noted authoz...
(Kilci J. Crowe]
observation Thought 3:
william
is...
ailliam J. Crowe served as the ambassador to the United Kingdom under President 3111 Clinton. So the anawer is B111 Clincon. Finich(Bill clinton]
Action 3:
Doservat lon 3:
Episode finlshed
Question:
What was the name of the team that represented an Arizona und versetyty in the conference that covers bzoad expense of the western United States, including Arizons, Californis, and New Kexice?
Thought:
Let's think step by step. The name of the team that represented an Arizona western
conference that covers United States, including Arizona, Californiz, and New Nexico is Arizona State
university
expanse
Answer:
Arizona State Sun Devils
Success: False positive
ReAce
Question:
Did the board game San
About Time T need to search San Marco and About Time, and find which came out first.
first?
Thought 1:
Actiion 2:
Search Mazcol
daservation 1:
The was built in venide. San Marco is seatiere of Venice. need search About Time next. Search Time]
San Marco is one of the
Thought 2:
Action 2:
ioservation 2:
Could net find [About Timel. similar: [...]. About Time 2013 film, San Maree came first. Fintah Marco] Episode finished
Thought Action 3:
Ooseivation
Question:
Wes Vanderbilt University or Emory Jniversity founded first? Vandarbi university founded in 1873.
Thought:
let'a think step University was Sounded in 1836. 1873 (Vanderbilt University) 1836 (Bmory University), so Vanderbilt University Founded first. zmery Universicy
Answer:
Failure: Reasoning error
ReAct
Question:
what the middle of the actress who alongside Harvey Fierstein, Linda Hart, Dick tatessa, Clarke Thorell, Mary Bond navis, taura Bell Bundy, Matthew Morrison, Corey Reynolds, Mariasa Jaret Rinokur in Heirspray?
Thought
hart, Latessa, Mary sand Devis, Leura Bell Bundy, Marthew Kerrisen, Corey Raynalda, and Mariasa Jaret Winokur, find the actress that was caat alongside them in scirapray, then find its middle Searcn(Harvay Fierstein]
to search Harvey rierstein,
shorell,
Act lon ":
Observatioe ion Thought 2:
Harvey Forbes Pierstein
Harvey Fierstein is actor, playwright, and scresnsritez. need to search Linda Search (Iinda Hart)...
next...
lon 2:
Question:
clder Deany Let's think step by step. Danny Green was bern in 1987. James Worthy was born 1961. 1987 (Danny Green) (Jamas Northy), , Danny Green older.
Worthy?
Thought:
Answer:
Danny Green
32

---

Published as a conference paper at ICLR 2023
Failure: Search emorr
ReAch Question:
What does the goddess associated with the goddess Irigg conelats of what teles?
Thought 1:
need to search goddess frigg and find its associated goddess. search (goddeas frigg)
Action 1:
Coservat lon 1:
Could not find goodess frigg. Similar: l'Frige's 'Frigg and Freyja common origin hypothesis', "Preyja (section Relation Erigg other goddesses and rigures)', "Bir (redirect from Eir (goddesa)) (Bald="].
Thought 2:
I need to seazch Frigg instead...
Failure: Hallucination
Question:
What was the first year scientific journal published by an ozganization located Sronx was publ shed?
Thought:
Let's think step step. The Tirst year scientific journal published
an orgenization lecated in the Bronx was published was 1916.
Answer:
1916 (Label: 1908)
Failure: Label ambiguity
ReAce :uestion:
What nationalicy is the man who deve' oped the dictazor game? I need to searca dictator game, find the man who developed it, then Lind his netionslity.
Thought
Action 2: Obgervatioon
Search (dictstor game]
The dictator and economics...
is popular experimental instrument. in cocial paychology
Action 3: roservation fon 3:
Search (Amos Tversky]
Amos Nathan Tversky.....
Thought 4:
Amos Tversky is an Israeli cogaltive and nathematical psychologlat. So the who developed the dictator game is Israeli.
Action 4:
(israel) flabel: jarsel-Americen)
Question:
Burt Fuller from Stockton, CA appeared in which American Detective Comedy Drama television series created by Steve Franks?
Thought:
Kurt suller from Stockton, appeazed in #sych. #sych ia an American Detective comedy Drama talevision series created by
Let's think step
Answer:
Rsych flabel: 2sych is an American detective comedy drama)
33


Reviews:
[{"review_id": "XtB3cRz8J-f", "reviewer": "Reviewer_1Gpr", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new way of in-context prompting of language model, by combining chain-of-thought reasoning and action decision generation to guid the model generation. \nThe model performance are evaluated on several language and text-based games. ", "summary_of_the_review": "The proposed approach are well justified and evaluated. This is the first step towards a generalist agent that can think and decide the action, in interactive tasks such as open-ended question answering, dialogue, etc. It also show how to effectively incorporate knowledge query using action decision after reasoning. ", "strength_and_weaknesses": "Strength:\n\nThe proposed approach is well justified. The performance of the model is evaluated deeply, and the limitation of the approach are deeply investigated. Also, the combination of ReAct with COT is evaluated to boost the performance, which shed more lights into the limitation, and provide opportunities for future directions.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "RAT_SeKlfZL", "reviewer": "Reviewer_Kdgk", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes ReAct; a novel framework for prompting large language models (LLMs) on tasks that require explicit reasoning and/or acting in an environment. Driven by recent work in plugging in LLMs into the main loop of a reasoning problem (e.g., fact verification or multi-hop question answering), or embodied plan generation (requiring actions/partial observability over time), this work makes a notable observation that the existing ways we have of interacting with LLMs capable of in-context learning are insufficient.\n\nNamely, approaches that follow standard in-context prompting (example/answer) are not enough for any high-resolution reasoning tasks; similarly, chain-of-thought prompting is great for reasoning tasks, but when actions/observations are streaming in, there’s not a good way to incorporate that information in a structured fashion, allowing for downstream exploitation.\n\nWhere ReAct is different is in it’s fusion of chain-of-thought style reasoning with records of actions and observations from an external source; for example, one of the evaluations in this work is on HotPotQA — a multi-hop question answering dataset. The LLM in question (PaLM-540B) is enriched with an API that allows it to query targeted passages from Wikipedia. ReAct allows for a system to use the language model as a “notepad”, first noting down any “thoughts” or reasoning traces, and then using the LLM to predict concrete actions (e.g., Wikipedia queries), that are then paired with the corresponding environment observation (e.g., the retrieved Wikipedia passage). The LLM continues this (given only one-two ReAct few-shot formatted examples), until it comes up with the correct answer.\n\nOn evaluations spanning Fact Verification, Question Answering, and Plan Generation, this paper clearly demonstrates the superiority of the ReAct style prompting over alternative approaches like standard (prompt/example) approaches, and the competitive “chain-of-thought” prompting, especially by noting that chain-of-thought prompting has a high false positive rate, because it’s reasoning traces are more likely to propagate “hallucinated/false” information.", "summary_of_the_review": "Strong paper, thorough and insightful evaluation, with clear wins over all alternative methods for prompting language models. Evaluating on multiple tasks, and especially showing the weaknesses of prior approaches like chain-of-thought prompting (e.g., high false positive rate due to “hallucinating” facts) is an added bonus.\n\nI really like this work!\n", "strength_and_weaknesses": "This is a strong paper with a clear motivation, thorough evaluation, and strong results. I think ReAct-style prompting is a clear win for tasks involving reasoning and embodied actions, and would love to see it adopted.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JuK81nWrGv8", "reviewer": "Reviewer_xfQy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes ReAct, a method for prompting large language models in a way that enables them to both *reason* about problems as well as \"*act*\" in the world by producing predefined actions which have external effects (e.g. interact with a wikipedia API, or take some action in some simulated text-based environment). The method concretely looks like chain-of-thought prompting with actions: prompt questions contain not only example reasoning traces, but reasoning steps interleaved with concrete actions. This encourages the model to decode similar interleaved reason/act steps, with the results of the act steps then returned to the model during decoding.\n\nThe authors mostly focus on PaLM 540B, evaluated on both reasoning tasks (e.g. HotpotQA, FEVER) as well as text-based sequential decision making tasks (e.g. AlfWorld, Webshop). Results are reasonably convincing that models are indeed able to interleaving reasoning with a more structured action space, and that access to such an action space improves performance (though the precise benefits of ReAct over CoT are marginal at some points - see weaknesses). \n\nThe results in this paper are not particularly surprising - especially since gains over CoT are not necessarily a \"fair\" comparison, since we are comparing a model with access to a full wikipedia API to a model that has to generate closed-book reasoning traces and facts. Though of course, the primary goal of this paper is to propose *precisely how* to use such extra data (i.e. the wikipedia API), so it's nice to have empirical evidence that this fairly sensible idea (including structured actions in the prompt) improves performance, given sufficient model scale.", "summary_of_the_review": "In summary, the idea in this paper is quite sensible, and it's nice to see empirical results verifying the ability for language models to access structured APIs by literally just including example uses in the prompt. The results here are likely to be interesting to the community, even if the specific gains of ReAct over CoT are unclear at some points.", "strength_and_weaknesses": "## Strengths\n\n- A diverse set of tasks spanning both multistep QA style tasks (HotpotQA, FEVER) as well as sequential decision making tasks. It's cool to see both of these types of tasks unified under a single framework, and showing how LLMs can leverage structured action spaces to make headway on both of these tasks.\n- Quite interesting scaling results across different sized models, showing that ReAct really begins to shine with increased model scale (a similar finding to the original CoT paper).\n\n## Weaknesses\n\n- ReAct alone works fairly well for some tasks, but self-consistent Chain of Thought often outperforms ReAct alone, and leads to fairly dramatic increases in performance. The best models use a smart combination of both CoT and ReAct, using heuristics that likely maximize performance on the dataset(s) (see `ReAct -> CoT--SC` strategy), which makes the performance of ReAct by itself slightly less impressive (though I still think this is a useful contribution for the community)\n- This method relies heavily on prompting to be able to predefine the space of allowed API commands: there needs to be an example usage of each action in the prompt to make the model aware it can take the action. This is infeasible for many environments that do not have an action space that is sufficiently constrained to fit into a single prompt.\n- Inconsistent prompting format. The sequential decision making tasks have thoughts/inner monologues specifically encoded as an action that has no effect on the environment (e.g. `think[For 3 ounce bottle...]`, observation `OK`). This is the \"sparse reasoning\" that avoids cluttering the LLM history as authors claim on page 8. However, this is *differrent* from how thinking is formulated in the Section 3 (Knowledge Intensive Reasoning) where thoughts are *not* sparse and instead appear before every action. What's the moviation for the authors presenting slightly different methods here?\n- The decision making setting, as authors note, is not significantly different from Huang et al. (2022) and other uses of language models for decision making, though it's nice to see this approach unified with the multi-step reasoning tasks.\n- Lack of reproducibility (see Reproducibility section below).\n\n## Minor\n\n- Typo: bullet points page 5 - one should be \"CoT-SC -> React\", I think?\n- Is there a way to add self-consistency to ReAct, such that it samples multiple reasoning traces and actions and uses the majority action, for example? SC seems extremely effective for Chain of Thought tasks, and it seems like something similar could work here.\n- I don't think there are sufficient ethical concerns to flag this for ethics review, but authors should perhaps discuss dangers of hooking up a large language models to an API/action space with side effects. How might we prevent a model from looking up inappropriate/sensitive information or taking harmful actions in an environment?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%