Do Think Tags Really Help LLMs Plan?  A Critical Evaluation of ReAct-Style Prompting
Authors: Siddhant Bhambri
Mudit Verma
Subbarao Kambhampati
Venue: Accepted by TMLR ()

Abstract: The reasoning abilities of Large Language Models (LLMs) remain a topic of considerable interest and debate. Among the original papers arguing for emergent reasoning abilities of LLMs, ReAct became particularly popular by claiming to tease out LLM reasoning abilities with special prompting involving “interleaving reasoning trace with action execution". In this paper, we critically examine the claims of ReAct style prompting for planning and sequential decision-making problems. By introducing systematic variations to the input prompt, we perform a sensitivity analysis along the original claims of ReAct. Our experiments in AlfWorld and WebShop, domains that were used in the original ReAct work, show that the performance is minimally influenced by the interleaved reasoning trace or by the content of these generated reasoning traces. Instead, the performance of LLMs is primarily driven by the unreasonably high degree of similarity between input example tasks and queries, with shockingly little ability to generalize. In addition to raising questions on claims about reasoning abilities, this lack of generalization also implicitly forces the prompt designer to provide instance-specific examples, significantly increasing the cognitive burden on the human. Our empirical results show that the perceived reasoning abilities of LLMs stem from the exemplar-query similarity and approximate retrieval rather than any inherent reasoning abilities, thereby leading to severe lack of generalization beyond the few-shot examples given in the prompts. Our code and prompt settings can be found here on GitHub.

Published in Transactions on Machine Learning Research (05/2025)
Do Think Tags Really Help LLMs Plan? A Critical Evaluation of ReAct-Style Prompting
Siddhant Bhambri"
siddhanthammbr0asn..dl
School of Computing and AI Arizona State University
Mudit Verma*t
muditrema@asuu edu
School of Computing and AI Arizona State University
Subbarao Kambhampati
roo edu
School of Computing and Al Arizona State University
Reviewed on OpenReview: https: /ttps://openveee opw.revee......ee.eomee net/ forum? id- aFAMPSMNHR
Abstract
The reasoning abilities of Large Language Models (LLMs) remain a topic of considerable interest and debate. Among the original papers arguing for emergent reasoning abilities of LLMs, ReAct became particularly popular by claiming to tease out LLM reasoning abilities with special prompting involving "inlerleaving reasoming trace with aclion execulion'. In this paper, we critically examine the claims of ReAct-style prompting for planning and sequential decision-making problems. By introducing systematic variations to the input prompt, we perform a sensitivity analysis along the original claims of ReAct. Our experiments in AlfWorld and WebShop, domains that were used in the original ReAct work, show that the performance is minimally influenced by the interleaved reasoning trace or by the content of these generated reasoning traces. Instead, the performance of LLMs is primarily driven by the unreasonably high degree of similarity between input example tasks and queries, with shockingly little ability to generalize, In addition to raising questions on claims about reasoning abilities, this lack of generalization also implicitly forces the prompt designer to provide instance-specific examples, significantly inereasing the cognitive burden on the human. Our empirical results show that the perceived reasoning abilities of LLMs stem from the exemplar-query similarity and approximate retrieval rather than any inherent reasoning abilities, thereby leading severe lack of generalization beyond the few-shot examples given in the prompts. Our code and prompt settings be found here on GitHub
1 Introduction
Large Language Models (LLMs) have seen rapid advancements spocifically in Natural Language Processing and Understanding (NLP& & NLU). LLMs have unparalleled capabilities in text generation, sunmarization, translation, and question answering to name few (Bubeck et al. 2023). Motivated by these capabilities of LLMs, there has also boen a rush to look for other emergent abilitios- especially for reasoning and planning. A popular way to improve LLM performance on reasoning/ reasoning/plaaniig tasks has been in-context prompting or prompt engineering (Sahoo et al. 2024) to include instructions (Girayl 2023) syntax structure (Marvin et al. 2023). criticism and plan guidance with verification (Kambhampati et al. 2024), etc. Among these approaches, ReAct (Yo al. 2022b), presented at ICLR 2023, stands out for its claims to improve LLM
"Equal Contribution
Work done while PhD student ASU, Google DeepMind

---

Published in Transactions on Machine Learning Research (05/2025)
planning abilities through the use of reasoning traces interleaved with action execution given plan guidance. Given the conflicting reports about the planning abilities of LLMs in the literature Valmeckam et al. (2024). in this paper we are particularly interested in evaluating these claims.
In our initial experiments with ReAct for planning, we found that the system is overly dependent on a high degree of syntactic similarity between the example prompt and the query, and is extremely brittle to minor perturbations to the input prompt. For example, when provided with an explicit set of examples of pick-and-place- place-object task and asked to plan for pick-and- wawaaadd-pp-e-ttee tuoo--ets- task, it should be trivial to generalize the solution of the examples to the queried task. Unfortunately, even such minor variation to the original ReAct agent sctup disrupts its performance.
Given the seemingly widespread adoption of ReAct met (as of this writing, it has 2,411 citations), the brittleness we witnessed calls for a systematic study of the factors contributing to the performance of ReAct-basod LLM agents. Based on the claims of (Yao et al.] 2022b), we isolate three possible reasons for the claimed performance of ReAct framework: 1) the utility of interleaving reasoning trace during action execution, 2) the utility of providing plan guidance, and, 3) the significance of example prompt provided to the LLM.
In this work, we systematically evaluate the brittleness of ReAct by studying which potential factors contribute to its performance. This analysis is conducted by investigating the following research questions- RQI: Does the agent performance depend on interleaving reasoning trace with action execution? RQ2: How does the nature of the reasoning trace or guidance information affect the performance of LLM agents? RQ3: How does the similarity between the example (problem, solution)and the query (problem, ? ), which are present in the prompt, affect LLM agent performance?
We conduct extensive experiments on the AlfWorld and WebShop domain using various LLM models, including GUPT-3.5-turbl
1PPT-3.--instrct, GPT-4-0314((0L Variant), GPT-4-0663 (Latest Variant), GPT-4o, Claude- Opus and Llama 3.1-8B, Through our comprchensive empirical study, we answer cach of the rescarch questions above. (RQ1) Rather than being dependent on reasoning trace being interleaved with action execution, we find that LLM performance in fact improves when the reasoning trace is not interleaved with action execution. (RQ2) Morcover, providing weaker guidance placebo-guidance (where the text provides information about the task) has comparable performance to strong reasoning trace-basedy guidance. Answers to RQ1 and RQ2 suggest that the source of ReAct LLM agent performance is not the interleaving aspect or the content. of the reasoning trace. Finally, in (RQ3), we see that variations to the example prompt such that it belongs to a very similar task within the same domain, or just has a different goal or plan attributes than the queried problem; causes the performance of Aet--bad LLM agent to plummet, highlighting the lack of length and inter-task generalization.
This lack of generalization leads to significant brittleness with respect to prompt design. Depending the domain and task, prompt designers may provide abstract guidance, task-specific instructions within the same domain, analogical examples, or global reasoning traces. ReAct's inability to robustly adapt to such variations underscores its limitations in handling diverse input prompts, thereby necessitating a closer examination of its design and implementation.
Our findings highlight that the benefits of ReAct-style prompting are limited to cases where prompt engineers can curate highly instance-speeific examples. This may not scale for domains with a large number of problem
instance classes, and it places undue burden on prompt engineers to provide instance-specifie examples.
Finally, our experiments call into question claims of enhanced "emergent reasoning" of LLMs with prompt
engineering efforts; corroborating contemporary rescarch (Verma et al. 2024 Valmeekam et al. 2024 Stechly
et al. 20245 [11man 2023 Schaelfer et al., 2023 McCoy et al.] 2023 Stechly et al.] 2024a Sprague et. al.
2024
questioning the reasoning abilities of LLMs.
2

---

Published in Transactions on Machine Learning Research (05/2025)
ReAct
Interact with a housshold se solve task, Hleee are Evo examples.
<EXAMPL 1 of task QUERY>
You
middle of zoom ... <Task Description> need find and spraybottlo, then it toilet.
is put some spraybattle on toiler.
Act think solve the CK.
Act. 2: think Pirst, need to find a sprayhottles aprayhottle the is wore likely to agpear in cabinet (1-4), countertop (1), reilet (11, sinkoasia (1-2), gerbagecan (11,
can chack ona by one, atarting with cabinet
Obe 2: OK.
Act 3:
cabiret
Oba 3: On the eabinet 1, you can see cloth 1, sospboar 1, soepbottle
Interleaved Think tag
Reasoning Trace/ Guidance
Act think Now find sprayboctie (2), need take it.
6: OK.
Example Task Query Task
<EXINPLS of
QUERY>
<OUERY>
is the tack. You
middle of the Your tack
soapbottie toilet.
Figure 1:
An example of ReAct in AlfWorld. We highlight the main components of ReAct, i.e, Interleaved reasoning and acting, the reasoning trace / plan guidance and the example and query task.
2
Related Work
2.1
LLMs for Planning & Reasoning
Large Language Models have been shown to be successful in plethora of natural language tasks (ooco
ct al. 2023 Gilardi et al.] 2023 Zhu et al. 2023 Bubeck et al.) 2023 Bhattacharjee et al. 2023). However,
there are two schools of thought when it comes to utilizing off-the-shelf LLMs for planning and reasoning
tasks. Works such as Chain of Thought, ReAct, and others that followed (Wei et al. 2022 Yao et al. 2023
Long 2023 Yao et al. 2024 Besta et al., 2024 Fu et al. 2024 Aksitov et al.) 2023), have argued about the reasoning abilitics of LLMs by proposing prompting methods. On the other hand, (Valmeekam al.) Stechly al., 2024b) have refuted these claims by showing the inability of LLMs to solve deterministic
2024
planning and classical reasoning problems.
2.2 ReAct & Anthropomorphizing Reasoning
Lately, works such as ReAct, Reflexion, and their other variants Yao et al. 2022b Shinn et al. 2023 have argued on the prowess of LLMs' reasoning abilities text-based decision-making domains such as AlfWorld Shridhar et al. (2020) and WebShop Yao et al. (2022a). However, the use of generating reasoning traces and interleaving them with task-specific action execution, which is the central claim of ReAct, has been used in many other applications bosides these domains. Furthermore, there have been several extensions to ReAct that claim boost their generalization abilities across more domains including multi-modal domains Yang et al., 2023 Castrejon al. 2024), autonomous vehicles (Cui al., 20224), table question answering (Zhang et al. 2023), etc. While the effectiveness of ReAct is celebrated across different areas, these works only depend on anthropomorphization of LLMs (Min et al.) 2022 Peng et al. 2024 for using ReAct-basedd prompting with no justification on the source of improvement in performance. This motivates our work in investigating the components of ReAct with respect to sequential decision-making problems and analyzing the role each component plays.
3 Preliminaries
3.1 Domains
(Shridhar et al. 2020 is a synthetic text-based game built on top of a STRIPS-style PDDL domain description (Fikes Nilsson 1971). ReAct (Yao al.) 2022b) defines six tasks (or problem classes) within this domain namely Put, Clean, Heat, Cool, Examine, and PutTwo. Each problem class consists of several problem instances, such as pul spraybolile toilet (see Fig. is example instance of Put.
AlfWorld:
3

---

Published in Transactions on Machine Learning Research (05/2025)
class. Since AlfWorld is partially observable environment, each of these problem instances can be solved by navigating and interacting with the environment simulator via text actions. For example, this task can be solved by the following actions- go to cabinet 2, take spraybottle 2 from cabinet 2, go to toilet 1, put spraybottle 2 in/on mn toilet 1.
WebShop: Yaa et al. 2022a) is an online shopping website environment with 1.18M real-world products and 12K human instructions. The agent is provided with an initial human instruction (for example, "I am looking for nightstand with drawers. It should have nickel finish, and priced lower than $140"). The agent's task is to crawl the shopping environment using actions such as search 'nightstand drawers', choose 'white buttons', back to search, etc. For this work, we randomly sample 500 test instructions from the environment and evaluate the success rate of the agent's task completion.
3.2 ReAct
ReAct (Yao et al. 2022b) claims to increase LLM's performance on text-based planning tasks such as AlfWorld and WebShop primarily by augmenting the original action space of the agent with a think action. The think action tag provided by ReAct is claimed to comprise of Reasoning + Action trace that is provided in the solution for the example problems (exemplars) part of the prompt. During execution, the LLM is expected to generate think action tag for the queriod problem instance that is semantically similar to the one provided for the examples in the prompt.
In ReAct, the integration of the think tag within actions serves to expand the action space. This allows the language model (LLM) agent to exccute a think action, prompting an 'OK' response. Through analysis of example prompts in ReAct experiments, we identify various instances of the think action. Typically, it appears after stating the problem instance, reiterating the task, and providing problem-specific guidance. However, the authors offer no structured guidelines for its implementation, placement, or guidance. This observation aligns with feedback from the paper's 's reviewers (OpenReview 2024) citing inconsistenciis in the prompting format.
Location of THINK tag:
Content of THINK tag: In ReAct, the think action consistently provides the decision-making agent with muccess-oriented guidance for task completion. For instance, upon encountering a spraybottle, the prompt might. include: think: Now find a spraybottle (2). Next, I need to take it. This guidance exposes forthcoming actions and sub-tasks for the agent.
Few shot EXAMPLEs: In the AlfWorld domain (which is PDDL domain), ReAct authors Yao et al. 2022b) classify six problem classes or tasks: Put, Clean, Heat, Cool, Examine, PutTwo. Despite reprosenting different tasks, they share the same environment dynamics and action space, allowing for very similar execution trace. For instance, a Heat task might involve Putting an ilem into a microwave. In ReAct experiments, authors provide two example problem-solute pairs (referred to as exemplars in our work) before querying the LLM agent with a problem instance. Authors force ReAct agent to use examples and queries belonging to the same problem class without motivating this design decision. However, the queried problem may differ in objects locations from the exemplars.
4 Evaluating ReAct Prompting
The subsequent sub-sections explore the design of exemplar prompt variations investigate our research questions concerning the claims of ReAct. Each variation modifies the base ReAct prompt, and we use the AlfWorld domain as the running example for discussing these variations (see Appendix for WebShop prompts).
4.1 RQ1: Interleaving Thinking and Acting - Location of THINK tag
Does the agent performance depend on interleaving reasoning trace with action execution?
To answer this research question, we propose collating the guidance information contained within the multiple think tags present in the examples of the input prompt into single think tag appended after the example
4

---

Published in Transactions on Machine Learning Research (05/2025)
(RQ1 a) Exemplar-CoT
You in the middle of Your task la to put some spraybottle on toilet.
<Tesk Description>
Act 1: think To solve the task, I need to rind and Lake a sprayboitle, then put it on tollet. First, need to more likely appeat in cabinet (1-4), countertep (1), toilet (1), ainkbasin (1=2), garbagecan (1), I can check one by one, starting with cabinet 1,
find spraybottle. A spssybottle
New Once I find a epraybotile 40t, next I need to to take it. need put it in/on toilet (1).
Unce take spraybottle (2), abs 1: CK.
Act go cabinet
abs 2: Cn the cabinet i, can see cleta 1, soapboar 1, soapbottle 1.
(RQ1 b) Anonymized Exemplar-CoT
<EXAMPEE 1>
You are in the middle of rocm ... <resk Descriptien>
Your
put some spraybuttle on toilet. To
find take sprayoottle the object, then put it toilet the desired location, Pirst, I need to find opesybottie the object. A-opraybottle The object 1a more likely to apgear eobines (1-4) countessop the different locations. I check by one,
task,
thytetlet atarting with cobires the fivst location. e
th perbngecen
the object, next New Once I take essppopeotrie 1st the objers, next I need to put it in/on th the desired location. ox.
Once
Act 2: cabinet 1.
abs 2: on the csbinet i, you can see cloth 1, soapboar 1, a soapbottle 1.
(RQ2 a) Failure
(RQ2 c) Ordering
...
Act 3: open cebinet 2
Act 3: open cabinet
Oba You open the cabinet 2, The cabinet ia open. candle and spraybottle 2.
oper: esbinet
cabinet open. candle 1, and spraybottle 2.
In 1t, you
it, you
Aet ++
thins How
apssybottic
Act to thin Now New find agreybbetttl
Neus, nood
ifr
Act 4: put spraybottle 2 in/on tollet.
Next, need to take it.
Obs 4: Nothing happens.
think Next, need take spreybottle New find spraybettle 2.
(RQ2 b) Failure Explanation
(RQ2 d) Placebo Guidance
Act 3: open cabinet 2
Obs 3: You open the cabinet 2, The cabinet is open. candle
Act 3: open cabl
it, you
epraybottle tetttttt
A thint Noy
Oba 3: You oper. the cabinet 2, The cabinet 2 is open, In it, you see a candle 1, and a sprayboltle 2, det trink
Act 4: spraybollle 2 in/on Lollet.
opseybottic
oba 4: Kothing happers.
sext, need-eeeeee take its Act 4: think Take deep breadth and work on this problem step by step.
Act 5: think Nothing happens because do not have spcaybollle
Figure 2: Example of prompt variations considered for RQ1 and RQ2.
problem specified. This approach can be interpreted as Chain-of-Thouuht, (Kojima et al. 2022 Wei et al. 2022). where guidance information generated before action exccution.
Variation Exemplar-based CoT: AlfWorld is partially observable environment where an agent only observe objects after reaching that location. Hence, remove specific location and object identifiers to modify the think actions that are originally interleaved with other actions in the environment (sce RQ1 a in Figure 2p. Finally, we append all the think actions together at the beginning of the example problem. For example, we rewrite it as think: Now Unce I find a spraybottle (2), next I need to take it. Intuition: Proflem-specific guidance for a sequential decision-making agent can be given step-by-stop (as in ReAct) or all once. Note, that this variation is possible since AlfWorld is not dynamically changing environment in which case providing information the task will not be possible.
5

---

Published in Transactions on Machine Learning Research (05/2025)
Variation 2 Anonymized Exemplar-CoT: We take one step further and modify the think tag to remove references to specific locations and objects, making it more general (see RQ1 b in Figure 2) Similarly, in WebShop we anonymize specific item options as desired options. For example, think: To solve the task, I need to find and take a spraybottle the object, then put it on toilet the desired location. Intuition: Exemplars can be made more general by providing abstract guidance and exploiting LLMs ability to identify necessary semantic entity relations. This can be useful for a prompt design that can suitably encompass wider range of downstream tasks and mediate the need of providing problem-specific guidance in the think tags.
4.2 RQ2: Action Guidance by Thinking Content of THINK tag
How does the nature of the reasoning trace affect the performance of LLM?
ReAct claims to use reasoning trace the guidance information following the think tag. For instance, in ReAct (aao et al. (022b), thoughts are to (1) decompose the goal (2) track sub-goal completion (3) determine the next sub-goal and (4) reason via common-sense where to find and object and what to do with it. It is, however, unclear what the motivation to use these as the reasoning trace. The potential anthropomorphizztion of large language models (LLMs) may suggest that their thought processes are similar to the abstract plans humans make, and that they must be prompted in the same manner. However, is unclear why this assumption should hold true.
Variation 1 - Failure: We note that none of the examples used in ReAct prompting for any task consist of invalid actions. We inject two invalid actions in the execution trace: the first that attempts to execute the action pertinent to the lask, such as pul spraybollle 2 in/on loilet, when not possible and, second, executes some other invalid action (see RQ2 a in Figure 2 We include the expected simulator response, Nothing happens., when invalid actions are taken. Intuition: Reasoning trace can about what to do such as future sub-goals, or what not do such as mistakes in hindsight. This should be weaker guidance than in base ReAct as the exemplars do not point out what to do next.
Variation 2 Failure + Explanation: We place think actions after invalid actions injected in Failure Variation which consist of explanations for the failure such as, think: Nothing happens because I do not have a spraybottle 2 (sce RQ2 b in Figure Intuition: We can augment mistakes with explanations to avoid similar failures. This is a stronger guidance signal than Failure, however, the exemplars still not provide information on what to do next.
Variation 3 Guidance Ordering: LLMs are known be susceptible to minor syntactic per- turbations to imputs. We test whether it is true for guidance information given prompt as well (see RQ2 in Figure 2. We identify chain of sub-tasks in reasoning trace S + S2... S, and reverse it to be S₂ + Sn-1--.S For instance, think: Now I find spraybottle (2). Next, I need to take it. bocomes think: Next, I need to take the spraybottle (2). Now find a spraybottle (2). Intuition: LLM agent should be invariant to the reasoning trace syntax the semantic information preserved.
Placebo Guidance: It is unclear to what extent LLM agent uses the supposed helpful thoughts for the decision making task. In this variation we replace think tag guidance with a placebo thought that does not contain any task relevant information, such as think: Take a deep breath and work on this problem step by step. (see RQ2 d in Figure 2), but has been widely used as prompt engineering trick (Kojima et al. 2022). Intuition: According to claims of ReAct, we expect the performance to get worse when the guidance does not have any information useful for task success.
Variation 4
4.3 RQ3: EXAMPLES QUERY Similarity
How does the similarily between the prompt examples and the query problem affect LLM performance?
RQ3 investigates the role of example similarity to the query LLM agent's performance. Establishing problem similarity can be challenging, especially where minor variations to the problem can have varied interpretations (such as analogy to different task altogether). Our work explores this challenge in
6

---

Published in Transactions on Machine Learning Research (05/2025)
systematic way. During example prompt construction, prompt designers may use synonyms to refer to objects (Domain), come up with examples where the agent task is the same as query but the goals are different. (Instance), or provide optimal solutions as the examples (Optimal) preventing LLM to obtain information regarding exploration strategy. Furthermore, given that the domain has the same underlying action dynamics and that the lasks reuse several actions, prompt designers may choose to provide query specific example prompts (as base ReAct), provide one of a different task and one of the same task (One), provide both examples to be of a different task (Both), or take an exhaustive approach and provide one example of all tasks (Al1).
Variation 1 Synonyms (Domain): For this variation, we replace the object and location names in the example prompts with their synonyms. For example, sproybottle + aerosolboltle, eabinet + cupbourd, and, mierowase + oven. We make 36 such changes to object and location names across all the examples [ Intuition: Exemplar guidance maybe specified with alternate synonymous object and location names. Reasoning agents should be invariant to variable name substitution for closed world dynamics such as PDDL-based AlfWorld. When example-based guidance needs to be provided to a sequential decision making agent, examples may consist of synonymous object and location names. Hence, we expect to achieve similar task performance.
Variation 2 - Problem Level (Instance): We inject instance-level changes to the examples provided in the prompts. We change the goal location in exemplar problem to ensure that it does not match with any of the goal locations in query problem. We also add repetitive yet futile actions in the exemplar execution trace which does not effect the solution. Intuition: Ensuring a different goal location in exemplar from the queried problem is natural use-case. Moreover, exemplars may contain arbitrary exploration strategies such as action repetition (Sharma et al. 2017).
Variation 3- - Problem Level (Both, One, Al1): Recall that the environment dynamics for all the tasks are the same. In fact, several tasks subsume the use of our tasks such as Heat requires the agent to Put an food in the microwave. In general, all the tasks share a large portion of actions (such as exploring cabinets and locations, picking objects etc.). Motivated by how tight relationship of these tasks we come up with three variations. First, One, uses one exemplar of an arbitrarily picked task and the other exemplar of the same task as the query. Second, Both, uses both exemplars from an arbitrarily picked task. Finally, All, uses a total of six exemplars corresponding to each task under consideration Note, that while all other prompt variations use two in-context examples, the All variation uses six in-context examples. Remember, this includes the query task which is always present at the end in the input prompt. Intuition: With a very similar action execution trace (such as exploration, picking and placing objects) across tasks, and shared dynamics, I.M agent should be minimally affected by the use of exemplars of a different task.
Variation 4- Exploration Strategy (Optimal): As noted before, ReAct does not explain the choice of exemplars used. An important ingredient to the exemplars is the exploration strategy used. In this variation we provide exemplars which serendipitously take the optimal actions (as if the environment were fully observable) and therefore the example plan is the shortest possible. Intuition: Exploration strategy exposed in exemplars (that too for the same problem task) should not impact ReAct's performance if the LLM agent is reasoning instead of retrieval (or pattern matching).
5 Results
While the original ReAct experiments were carried out on PaLM (currently decommissioned), we reproduce their results with newer set of models. We use GPT-3.5- Turbo, Innt-st-- GPT-4, GPT-40, and Clando-Opus, which are all newer models than those benchmarked in RcAct Yaoo et al. 2022b Note, that despite using newer models, our results shed doubts on the reprodueibilityy and consistency across models of the original paper's results. As noted, we use the setup in (Yao et al., 2022b) for all our experiments. In AlfWorld, CPP.3.5-(Turb) Instruct) results are on 134 instances across six tasks, GPT-4/Claudee-put on 60 instances (10 for each task) due to cost considerations. In WebShop, GPTa.5(Turbo, Instruct), GPT-4o, and T.LAMA-31-88 results are on 500 samples, while GPT-4/ClaudeeOpuur are on 50 instances due to cost.
The object names location are unchanged for the problem query and subsequent interaction with the simulator.
7

---

Published in Transactions on Machine Learning Research (05/2025)
considerations. For each of the experiments, we include the Act baseline, originally used by ReAct, where the same examples are provided to prompt the LLMs but without any think tags in their execution traces. Naturally, in this case, the LLM is unable to produce a think tag for the queried problem, and thus, unable to utilize any supposed reasoning guidance given as part of the in-context examples. We include the lower and upper bound confidence intervals for all our experiments in the Appendix Section [3.
Table 1: Average Success % of LLM for RQI and RQ2 six AlfWorld tasks. The table uses color spectrum to represent the quality of the numbers. Cells shaded in green indicate the best performance compared to base ReAct, transitioning to red for the worst relative performance.
Model Prompt
Act
ReAct
RQ1
RQ2
CoT
Anon. CoT
Placebo
Order
Failure
Explanation
GPT-335-Turbo
34.3
27.6
46.6
41
30
28.3
43.3
41.6
GPT-33.-Instruut
44
50.7
61.9
50.7
41
42.5
47
44.7
GPT-4-0314 (Old)
23.3
43.3
33.3
36.6
50
36.6
GPT-4-0613 (Latest)
70.0
26.7
40.0
26.6
36.6
30
60
36.6
Claude-Opus
43.3
56.6
50
46.6
30
50
53.3
30
Table 2: Average Success % of LLM for RQI and RQ2 WebShop tasks. The table uses color spectrum to represent the quality of the numbers. Cells shaded in green indicate the best performance compared to base ReAct, transitioning to red for the worst relative performance.
Model / Prompt
Act
ReAct
RQ1
RQ2
CoT
Anon. CoT
Placebo
Failure
Explanation
GPT-3.5-Turbo
1.12
1.04
2.20
1.88
1.52
3.48
3.48
7.24
7.16
7.52
6.12
7.40
7.20
7.24
GPT-4-0613 (Latest)
8
4
8
8
8
8
GPT-4o
4.64
2.24
4.68
4.52
4.08
4.68
4.68
Claude-Opus
4
4
4
2
4
2
4
LLAMA-3.1-8B
1.44
3.16
3.28
3.92
2.04
1.20
2.16
5.1 Utility of Interleaving THINK tags (RQ1)
To recall, we create two prompt variations for RQI: CoT and Anon. COT. In CoT, we concatenate all the think tag content the beginning of the execution trace for the two examples. For Anon. CoT, we further anonymize the object and location names in the concatenated think tag.
From Table I (RQ1), note that the CoT and the Anon. CoT variations perform significantly better than base ReAct for all OpenAI models. Morcover, the performance dips slightly for Claude-Opus along these variations. This refutes ReAct's first claim on the importance of interleaving reasoning trace generation with action exccution. Even in the case of Claude where there slight dip in performance, the models seems performing at reasonably high success rate which questions the importance interleaved reasoning and action execution. We omit LLAMA-3.1-88 and GPT-4o (See B.2 for AlfWorld they achieve zero performance over baselines and all the variations. From Table 2 find similar pattern: CoT and Anon. COT variants perform closely better than the baseline ReAct. A surprising result consistent in both the domains is the performance of Act method. Act baseline is weaker only for two models - 5-Instruct, Claude-Opus for both the domains, which further questions the utility of using ReAct prompting.
5.1.1 RQ1 Failure Analysis
Figure shows a radar chart for four different LLMs, namely GPT-3.5-Turb. GPT--35-Insttin GPT-4, and Claude Opus, for the RQ1 ablations indicating the failure rate (in %) across the six tasks in the AlfWorld domain. In all but one case for Claude Opus, we note the largest shaded region in red indicating the highest failure rate for ReAct across five AlfWorld tasks as compared to the RQ1 prompt variations. This indicates

---

Published in Transactions on Machine Learning Research (05/2025)
GPT 3.5 Turbo
GPT 3.5 Instruct
GPT 4
Claude Opus
Put
Put
Put
Put
PutTwo
Clean PutTwo
Clean PutTwo
78
Clean Clean PutTwo
Clean
80
D0
50
Heat Examine
Heat
Heat
Heat
Examine
Examine
Examine
Cool
Cool
Cool Anonymized-Exempa
Cool
ReAct
Exemplar
The radar chart shows the failure rates of various LLMs with different ReAct-based prompt settings for RQ1 (Base React, Global, Anonymized) across six AlfWorld tasks (hexagon vertices). Higher values / larger shaded region indicate worse performance.
Figure 3: RQ1 Failure Analysis:
that there is no performance gain from interleaving think tags with actions in the prompt examples, rather has a contrary effect on majority of the tasks. CoT has the smallest average failure rate, followed by the Anonymized COT. We believe that this is because prompting LLMs with object and location-specific guidance as part of think tags in the in-context demonstrations provides more useful and targeted information for solving the downstream queried problem, which too has similar object and location identifiers the in-context. examples. We have also included direct comparison of the think lags generated by the ReAct framework and by the RQ1 prompt variations in Appendix Section B.4
5.2 Utility of THINK tag Content (RQ2)
To recall, we create four prompt variations for RQ2: Placebo, Failure, Explanation and Order. In Placebo, replace each think tag content with meaningless text. that has no relation to the task. For Failure and Explanation variations, we replace the think tag suggested actions to an action that docs not help solve the lask, along with an explanation for the same, respectively. In Order, switch the ordering of the sentences that are part of each individual think tag.
Recall that reasoning trace guidance pertains to the prospective actions or behaviors an agent should exccute (foresight guidance). This lype of guidance is more informative compared to other variations, such as hindsight. guidance, which focuses on past errors without providing future solution steps, and placebo guidance, which is entirely unrelated to the task. ReAct claims that reasoning trace is crucial for LLM agent performance, which would predict decline in performance with hindsight guidance and collapse with Placebo variation. Therefore, a practitioner would expect all the rows to be dark shade of red. In contrast, our findings in Table indicate that hindsight guidance (Failure, Explanation) actually improves the performance of the GPT family of models. The Claude-Opus model's performance remains stable with hindsight (Failure) guidance and declines with Placebo guidance. This refutes ReAct's claim that task-specific reasoning trace is the source of LLM performance. Our argument that LLM agent's 's performance is only slightly affected by the reasoning trace explains the indifference to Order perturbation as well. If the LLM is not utilizing the reasoning trace for decision making, change in ordering would not affect the agent's 's performance. Our arguments hold for the WebShop domain as well, where all of the variants perform closely or better than the baseline ReAct.
Finally, contrary to the general perception that better GPT models would improve over reasoning, we find that GPTd4-Oldd)s performance is the worst among GPT-X family further highlighting the brittleness of claims of ReAct. GPT-s-(Latest) performs similarly to GPT-4-(lld), except for the Act bascline which again shows the futility of ReAct prompting. In all our experiment settings, we note that LLMs replicate the exact.
9

---

Published in Transactions on Machine Learning Research (05/2025)
Table Average Success of LLM for RQ3 on six AlfWorld tasks. OC: Out of context limit. The table uses color spectrum to represent the quality of the numbers. Cells shaded in green indicate the best performance compared to base ReAct, transitioning to red for the worst relative performance.
RQ3
Model Prompt
Act
ReAct
Domain
Instance
Optimal
All
One
Both
GPT-3.5-Turbo
31.3
27.6
1.6
30
20.1
32
28.3
1.6
44
50.7
47.6
42.5
39.5
OC
17.9
5.2
GPT-4-0314 (Old)
23.3
13.3
23.3
50
23.3
16.6
GPT-4-0613 (Latest)
70.0
26.7
10.0
20.0
53.3
23.3
20
3.3
Claude-Opus 43.3
56.6
50
46.6
43.3
50
60
6.6
steps as shown for the examples in the prompts. Hence, they do not output what ReAct claims think tags if those tags are not present in the original prompt.
5.2.1 RQ2 Failure Analysis
Figure shows a radar chart for four different LLMs, namely GPT-3.5-Turbo, GPTT-3.--nntruct, GPT-4, and Claude Opus, for the RQ2 ablations indicating the failure rate (in %) across the six tasks in the AlfWorld domain. Similar Figure 3 we note the highest failure rate for ReAct across five AlfWorld tasks as compared to the other RQ2 prompt variations, with the smallest average failure rate for Failure prompt variation. We hypothesize that this is because providing actions that lead to failure in the in-context demonstrations assist in preventing similar lailure-leading actions for the queried problem. Moreover, providing failure information can be less cognitively demanding for prompt designers than providing an exact solution for the in-context examples, as required by the ReAct framework. We have also inchuded a direct comparison of the think tags generated by the ReAct framework and by the RQ2 prompt variations Appendix Section B.4
GPT Turbo
GPT 3.5 Instruct
GPT
Claude Opus
Put
Put
Put
Put
PutTwo
Clean PutTwo
Clean PutTwo
Clean PutTwo
Clean
50
25
25
25
Heat Examine
Heat
Heat
Heat
Examine
Examine
Examine
Cool
Cool
Cool
Cool
ReAct
Magic
Explanation
Failure
D Order
Figure 4: RQ2 Failure Analysis:
The radar chart shows the failure rates of various LLMs with different. ReAct-basee prompt settings for RQ2 (Base React, Magic, Failure, Failure+ DallureeExxllanttion Ordering) across six AlfWorld tasks (hexagon vertices). Higher values larger shaded region indicate worse performance.
5.3 Dependence on Literal Ceample-query Similarity (RQ3)
To recall, we create six prompt variations for RQ3: Domain, Instance, Optimal, All, One and Both. In Domain, we use synonymous object and location names. For Instance, we add futile but repelitive actions while in Optimal we only include the minimum number actions needed to reach to the goal the examples. For All, we include one example from each of the tasks. We change one example task be different from the same category as of the query task in One and change both the example tasks from another category Both.
10

---

Published in Transactions on Machine Learning Research (05/2025)
Intuitively, the similarity of Domain examples is closest with base ReAct, followed by Instance and Optimal variations. Finally, All contains an overload of information followed by One and Both which have the same action space but uses different tasks as exemplars. Recall that AlfWorld being PDDL domain has a shared environment dynamics across all tasks with up to 80% of actions shared across exccution traces. While ReAct does not investigale impact of varied exemplars, given the popular usage, one expects LLMs to be robust to such changes especially in a common-sense houschold domain. Table B shows the severe brittleness of ReAct based LLM agent to even minor variations (such as Domain, Instance). Specifically, performance of GPT--3.- Turbo and GPT-4 plummets for Domain. Clande-Opus which was more robust in RQ1, RQ2, is also impacted severely by Domain, Instance variations. Furthermore, the performance of all LLMs but GPT4 drops when we do not expose the exploration strategy and only provide Optimal exemplars.
Overloading the LLMs with more exemplars All does not impact its performance. We posit, this is because the query-task exemplar is still part of the large input prompt. Among the two exemplars, as provided in ReAct, when one of them of a different task (Dne) then the performance significantly reduces for LLMs. When both of the exemplars are of a different task then the performance collapses single digit success rates for all the models. This is key result of this work highlighting the severe dependence of LLMs on the similarity of the exemplars to the query task and the lack of generalization with ReAct. Through sensitivity analysis using our RQ3 variations, we could find parts of the input (the task similarity of the exemplar with query) which is the source of ReAct performance. Essentially, the LLM is mimicking/ pwwpriinn//eformingggg approximate retrieval from the context presented to it by trying to overfit on the information provided as part of the in-context demonstrations (or examples). Moreover, our results corroborate the line of research that questions
the inability of LLMs to reason or plan (Verma et al., 2024 Valmeekam et al., 2024 Stechly et al. 2024b
Ullman 2023
Schaeffer ct al. 2023: McCoy ct al. 2023
Stechly et al.) 2024a Sprague ct al. 2024
The reported success-rate from the ReAct paper Yao et al. (002bb) on the WebShop domain is 40%. Due to the absence of the exact queries used in the paper, we randomly sampled queries from the WebShop dataset comprising 12K records. This approach possibly resulted in the decoupling of any relationship between the exemplars and the queries. Referring to Table 2 it is evident that the performance of the WebShop ReAct agent significantly declined, reaching single digit percentages (as well as other variants). This mirrors the trends observed in the Both variant of the Alfworld in Table 3 further supporting our arguments.
5.3.1 RQ3 Failure Analysis
Figure shows a radar chart for four different LLMs, namely GPT-3.5-Turbo, GPT-3.5-Instrucl, GPT-4, and Claude Opus, for the RQ3 ablations indicating the failure rate (in %) across the six tasks in the AlfWorld domain. We note the highest failure rate for the Both variation across five AlfWorld tasks as comparod to the other RQ3 prompt variations, with the smallest average failure rale for Optimal prompt variation. These results further indicate the brittle dependence of ReAct prompting on the type of in-context demonstrations used for querying the LLM for downstream tasks. We have also included a direct comparison of the think tags generated by the ReAct framework and by the RQ3 prompt variations in Appendix Scction B.4 We also did preliminary study on the similarity (computed in terms of embedding distances) between the 134 prompting tasks with the originally provided examples, and the performance of ReAct. As can be seen from Figure t in Appendix Section B.6, as expected, the performance degrades significantly in the off-diagonal cases.
5.4 Length & Inter-task Generalization
Unrolling and Inter-task Similarity: We perform additional experiments to highlight the lack of length generalization where the query task is to essentially repeat the task in the exemplar (Unrolling). For instance, in AlfWorld, the exemplar is Put and the query is PutTwo to put two objects at given location. In this case, the LLM has to unroll the given advice and repeat exemplar task execution to solve the query. The success rate of SPPT-3.5-Instru. (the best performing GPT model in our experiments) drops down from 52% to 9%. Similarly, we experiment with a Inter-task Similarity variation where the exemplar task subsumes exccution of the query task. For instance, the Heat task requires the agent to pick and place object in the microwave (which is instantiation of Put task). One would expect that Heat is a good exemplar for Put, however, the
11

---

Published in Transactions on Machine Learning Research (05/2025)
GPT 3.5 Turbo
GPT 3.5 Instruct
GPT 4
Claude Opus
100
Put
100
Put
100
Put
100
Put
PutTwo
Clean Clean PutTwo
Clean PutTwo
Clean PutTwo
Clean
Heat
Heat
Heat Examine
Heat
Examine
Examine
Examine
Cool
Cool
Cool
Cool
ReAct
Domain
All
Instance
Both
One
0 Optimal
The radar chart shows the failure rates of various LLMs with different. ReAct-based prompt settings for RQ3 (Base React, Domain, Instance, All, Both, One) across six AlfWorld tasks (hexagon vertices). Higher values / Larger shaded region indicate worse performance.
Figure 5: RQ3 Failure Analysis:
performance of GPT-3.5-instruct model goes from 18% to 0% in this case. These results further underscore the brittleness and the need for instance-specific exemplars ReAct.
Exemplar CoT prompt from RQ1 + RQ3 variations: In this case, we Lest the main results of our work with the RQ1 CoT variation to identify whether our findings hold true there as well or not. That is, we test RQ3-One and RQ3-Both with Exemplar CoT prompt variation. For CPT-,.5-Turbo, we find that the average performance drops from 46.6% (RQ3- CоТ) as in Table [ to 28.3% in One and 10.4% in Both variation cases, and remains at 40.3% for All variation, thereby reaffirming hypothesis behind the lack of generalization in ReAct-based LLM prompting framework.
6 Discussion
Operationalizing 'think' actions by LLMs Given the free-form nature of think tag generation and arbitrary nature of their content (about sub-task, common-sense sense next steps etc.), checking whether the generated thoughts are in-fact reasonable is a challenging problem. For completeness, we find that 40% of the times after generation of lhink tag, subsequent environment action taken by the LLM was invalid (for GPT-3.5-instruct) in AlfWorld. It. is much higher ( (80% for GPT-3.5-Turbo, 90% for Claude-Haiku) for weaker LLM models. This further highlights the inability of LLMs to operationalize its generated think action, as also seen in (Roy et al.] 2024). From manual inspection we find that the typical think action would enlist all possible locations as next locations to visit for most of the tasks. As demonstrated in Section 5.2 the performance LLMs actually decreases when provided with foresight guidance, as seen with the base ReAct model.
Lack of Generalization in ReAct-Style Prompting: Recall, that ReAct claims an improved perfor- mance for text-based planning domains, namely- - AlfWorld and WebShop, where the presence think tag acts as guidance for the LIIM to generate the next set of actions during the I.M--nvvronmmett interaction. Through our sensitivity analysis, we dissect each component of ReAct-style prompting in critical effort to understand the factor that leads to the observed success rates in these domains. With variations the placement (RQ1) and content (RQ2) of the think tag, eliminate it as the primary cause of any improvement. Furthermore, slight variations in exemplar tasks (RQ3) lead to stark decline in success rate, clearly indicating the literal dependence of performance on the highly curated instance-specific examples provided by domain experts. While some works on In-Context Learning point out the impact of well-curated examples (Min) ct al. 2022 Peng al., 2024),, our work specifically highlights the severe lack of generalization and that exemplar-query similarity is the primary cause ReAct's performance, thereby rejecting contemporary belief that the heavy-lifting of LLM reasoning & planning is done through the think tag.
12

---

Published in Transactions on Machine Learning Research (05/2025)
Lack of Generalization to newer LLMs: ReAct uses the Act bascline (LLM-environment interaction without think tags) in their work to showcase improvements due to the presence of the proposed think tag. For AlfWorld, ReAct reports 45% success rate for Act baseline and 71% success rate for ReAct prompting using the PaLM model. For WebShop, ReAct reports 30.1% success rate for Act bascline and 40% success rate for ReAct prompling on PaLM. However, we note from our results on both domains that the Act bascline performs much better than ReAct for several LLMs, which questions the compatibility of ReAct to newer-age LLMs. ReAct performs worse with newer models as compared to the results they report on the currently decommissioned PaLM. Moreover, this further strengthens our claim that ReAct's performance can't be contributed to the presence of think tags, but is primarily due to the literal similarity between the in-context examples and the queried problem. This observation also questions the contemporary belief that such prompling strategies are generalizable throughout different. LLM families, including newer models.
We re-iterate our key result, given any LLM model, the success rates plummet with our RQ3 variations showing consistent pattern of literal dependence on the provided examples irrespoctive of the LLM. Morcover, the performance of all the LLMs remain quite high (if not better) when we vary the location and content. of the think tags. This highlights the need for higher rigor in agentic LLM experimentation and in-depth evaluation socking source of improvements.
7 Conclusion & Future Directions
ReAct based prompt engineering methods have been claimed improve planning abilitics of Large Language Models. In this study, we critically examine ReAct along three dimensions, informed by its claims and our hypotheses regarding its performance sources. Contrary to ReAct's claims, our findings reveal that its performance is neither due to interleaving reasoning trace and guidance information generation with action execution, nor due to the specific nature of the guidance information. Instead, we identify that the true source of LLM performance in sequential decision-making tasks, such as AlfWorld, is the high degree of literal similarity between exemplar problems (few-shot) and the query task, thereby leading to severe lack of generalization. Our findings caution against an uncritical adoption of ReAct-style frameworks for their putative abilities to enhance performance in domains requiring planning. To conclude, we believe that those results will be helpful for practitioners and future works, particularly when designing prompts for decision-making problems, and benefit from avoiding putting any efforts into constructing non-trivial problem-specific reasoning traces.
Broader Impact
With the surge in LLM-based applications, including those that may have critical consequences, it becomes imperative to study the robustness of LLMs and frameworks based on them. One such case in point is that of ReAct, which has gained popularity for its use across diverse domains. However, there has been little to no critical analysis to understand the robustness and generalizability in the ReAct framework. These include, but are not limited to, understanding the core reasons behind ReAct's performance, performance of ReAct on newer LLMs, robustness to slight deviations from the intended problems, etc. Our work presents a critical sensitivity analysis of the framework to answer these questions via comprehensive set of empirical results, thereby revealing the brittleness of the proposed framework along with the over-reliance on human-provided lask-specific examples. Ideally, the impet of providing generalization and robustness guarantees, if any, should fall on the authors of the proposed approach. This would allow future works to have a thorough understanding of various approaches and utilize them responsibly.
Acknowledgment
This rescarch is supported in part by ONR grant N0001423-1-2400 DARPA grant HR00112520015 and gifts from Qualcomm, J.P. Morgan and Amazon.
13

---

Published in Transactions on Machine Learning Research (05/2025)
References
Renat Aksitov, Sobhan Miryoosefi, Zonglin Li, Daliang Li, Sheila Babayan, Kavya Kopparapu, Zachary Fisher, Ruiqi Guo, Sushant Prakash, Pranesh Srinivasan, et al. Rest meets react: Self-improvement for multi-step reasoning Ilm agent. arXiv preprint arXiv:2812.10003, 2023.
Macicj Besta, Nils Blach, Ales Kubicck, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, et al. Graph of thoughts: Solving elaborate problems with large language models. In Proceedings the AAAI Conference on Artificial Intelligence, volume 38, pp. 17682 17690, 2024.
Amrita Bhattacharjee, Raha Moraffah, Joshua Garland, and Huan Liu. Towards Ilm-guided causal explain- ability for black-box text classifiers. arXiv preprint arXiv:2309.138 13340, 2023.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Loe, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
Lluis Castrejon, Thomas Mensink, Howard Zhou, Vittorio Perrari, Andre Araujo, and Jasper Uijlings. Hammr: Hierarchical multimodal react agents for generic vqa. arXiv preprint arXiv:2404.05463, 2024.
Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, and Ziran Wang. Receive, reason, and react: Drive as you say, with large language models in autonomous vehicles. IEEE Intelligent Transportation Systems Magazine, 2024.
Richard E Fikes and Nils J Nilsson. Strips: A new approach to the application of theorem proving to problem solving. Artificial intelligence, ((3-4):189-20) 208, 1971.
Dayuan Fu, Jianzhao Huang, Siyuan Lu,, Guanting Dong, Yejie Wang, Keqing He, and Weiran Xu. Preact: Predicting future in react enhances agent's planning ability. arXiv preprint arXiv:2402.11534 2024.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd workers for text-annotation tasks. Proccodings of the National Academy of Sciences, 120(30))c2300011120, 2023.
Louie Giray. Prompt engineering with chatgpt: guide for academic writers. Annals of biomedical engincering, 51(12):2629 2633, 2023.
Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Kaya Stechly, Mudit Verma, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy. Lims can't plan, but can help planning in Ilm-modulo frameworks. arXiv preprint arXiv:2402.0188 2024.
Jan Kocoń, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydio, Joanna Baran, Julita Bielaniewiez, Marcin Gruza, Arkadiusz Janz, Kamil Kanclerz, al. Chatgpt: Jack of all trades, master of none. Information Fusion, 99:101861, 2023.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Twasawa. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199 22213, 2022.
Jieyi Long. Large language model guided tree-of thouuhht arXiv preprint arXiv:2305.08991 2023.
Ggaliwango Marvin, Nakayiza Hellen, Daudi Jjingo, and Joyce Makatumba-Nabenia Prompt engineering in large language models. In International Conference on Data Intelligence and Cognitive Informatics, pp.
387 402. Springer, 2023.
R Thomas McCoy, Shunyu Yao, Dan Friedman, Matthew Hardy, and Thomas Griffiths. Embers of autoregression: Understanding large language models through the problem they are trained to solve. arXiv preprint arXiv:2309.1366 2023.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannanch Hajishirzi, and Luke Zettle- moyer. Rethinking the role of demonstrations: What makes in-context learning work? arXiv preprint arXiv:2202. 12837, 2022.
14

---

Published in Transactions on Machine Learning Research (05/2025)
OpenReview. ReAct: Synergizing Reasoning and Acting in Language Models. https://openreviev.nees forum?id-WE vluYUL-X 2024.
Keqin Peng, Liang Ding, Yancheng Yuan, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. Revisiting demonstration selection strategies in in-context learning. arXiv preprint arXiv:2401.12087 2024.
Shamik Roy, Sailik Sengupta, Daniele Bonadiman, Saab Mansour, and Arshit Gupta. Flap: Flow adhering planning with constrained docoding in llms. arXiv preprint 050.666666. 2024.
Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927, 2024.
Rylan Schaeffer, Kateryna Pistunova, Samar Khanna, Sarthak Consul, and Sanmi Koyejo. Invalid logic, equiv- alent gains: The bizarrenoss of reasoning in languago model prompting. arXiv preprint arXiv:2307.1057, 2023.
Sahil Sharma, Aravind Srinivas, and Balaraman Ravindran. Learning to repeat: Fine grained action repetition for deep reinforcement learning. arXiv preprint arXiv:1702.60044 2017.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.13366 2023.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.08768, 2020.
Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To cot or not cot? chain-of-thought helps mainly math and symbolic reasoning. arXiv preprint wxXiv2249.121888 2024.
Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. Chain of thoughtlessness: An analysis of cot in planning. arviv preprint arXiv:2405.04770 2024a.
Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. On the self-verification limitations of large language models on reasoning and planning tasks. arXiv preprint arXiv:2402.08115, 2024b.
Tomer Ullman. Large language models fail on trivial alterations to theory-of-mind tasks. arXiv preprint arXiv:2302.06399 2023.
Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems, 36, 2024.
Mudit Verma, Siddhant Bhambri, and Subbarao Kambhampati. Theory mind abilities large language models in human-robot interaction: An illusion? In Companion of the 2024 ACM/IEEE International Conference Human-Robot Interaction, pp. 36-45, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting clicits reasoning in large language models. Advances in noural information processing systems, 35:24824 24837, 2022.
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.118 11381, 2023.
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35: 20744-20757 2022a.
15

---

Published in Transactions on Machine Learning Research (05/2025)
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2022b.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024.
Yao Yao, Zuchao Li, and Hai Zhao. Beyond shain-of-thhugeee effective graph-of-thought reasoning in large language models. arXiv preprint arXiv:2305.16582, 2023.
Yunjia Zhang, Jordan Henkel, Avrilia Floratou, Joyce Cahoon, Shaleen Deep, and Jignesh M Patel. Reactable: Enhancing react for table question answering. arXiv preprint arXiv:2310.008L 2023.
Yiming Zhu, Peixian Zhang, Ehsan-Ul Haq, Pan Hui, and Gareth Tyson. Can chatgpt reproduce human- generated labels? a study of social computing tasks, arXiv preprint 1014: 2023.
16

---

Published in Transactions on Machine Learning Research (05/2025)
A Resources Used
In this work we leverage OpenAI API and Claude API for prompting the Language Models. We use gpt-4-0613 for GPT4, gpt-3. t-1t-----urbb00155, g tttr:---ttuooonnsrteee
and -ande---haiiu-22240000 for AlfWorld in April-May 2024. Results using GPT-4-0613 (for both AlfWorld and WebShop) and all the results on WebShop were completed in Sep 2024. As an estimate, for AlfWorld, ReAct and corresponding experiments use approximately 14M input tokens (due to repeated prompting after each action execution) and 150K output lokens for 134 problem instances used by ReAct.
B Additional Considerations
B.1 Performance of Claude-Haiku
We skip on mentioning the performance of Claude-Haiku, since it was not able to generate syntactically correct actions for any of the instances. We found that following our instruction to generate specific actions as in the exemplar was difficult. We improved the prompt to have specific instructions for generating actions (See D.3 ) but it did not yield any improvements for Claude-Haiku. However, the instruction did help with Claude Sounett and Claude-Opus. We find that Claude-Sonnet follows a similar pallern as GPT-3.5- (upt-uct-n as presented in our results, and decided to focus ourselves on the prongest/largees Claude model (Claude- Opudd for our evaluation.
B.2 Performance of GPT-4o and LLAMA-3.1 in AlfWorld
In our experiments, we found that GPT-4o obtained zero success-rate across all variations (including Act and ReAct and our proposed variations). Upon closer inspection, found that GPT-4o requires significant effort in instruction tuning for AlfWorld, specifically, it would start emitting justifications for why previously taken action was unsuccessful rather than generate syntactically accepted think tags and environment actions. Even for action generation, GPT-4o would pre-pend the actions with justifications, thereby expecting the users to write specific parsers. While stronger parsers maybe possible to implement, we hold GPT-4o to the same expectations as other models (which do not get the benefit of stronger parsers) and report our findings on a consistent evaluation.
L.L.MA3.1--8B obtains zero success-rate across all variations in AlfWorld well. We observe that the LLAMA model would generate incorrect actions and would repeat those actions exhausting the iteration budget.
B.3 Confidence Intervals
In this subsection, we present the experiment results on Alfworld and WebShop with the confidence intervals computed with 95% confidence level. The results have been discussed in Tables and
Table 4: Lower & Upper bound Success % of LLM for RQ1 and RQ2 on six AlfWorld tasks with 95% confidence interval. The table uses a color spectrum to represent the quality of the numbers. Cells shaded in green indicate the best performance compared to base ReAct, transitioning to red for the worst relative performance.
Model Prompt
Act
ReAct
RQ1
RQ2
CoT
Anon. CoT
Placebo
Order
Failure
Explanation
Ghu-
34.3
27.6
(36.8, 36.4)
(31.4, 50.6)
(21.0, 39.0)
(19.5, 37.1)
(33.6, 53.0)
(31.9, 51.3)
GPTT-355-nnstuct
44
50.7
(52.4, 71.4)
(40.9, 60.5)
(31.4, 50.6)
[32.8, 52.2)
(37.2, 56.8)
(35.0, 54.4)
GPT-4-0314 (Old)
23.3
(33.6, 53.0)
(24.1, 42.5)
(27.2, 46.0)
(21.0, 39.0)
(40.2, 59.8)
(27.2, 46.0)
GPT-4-0613 (Latest)
70.0
26.7
(30.4, 49.6)
(17.9, 35.3)
(27.2, 46.0)
(21.0, 39.0)
(50.4, 69.6)
(27.2, 46.0)
Claude-Opus
43.3
56.6
(40.2, 59.8)
(36.8, 56.4)
(21.0, 39.0)
[40.2, 59.8)
(43.5, 63.1)
(21.0, 39.0)
17

---

Published in Transactions on Machine Learning Research (05/2025)
Table Lower Upper bound Success % of LLM for RQI and RQ2 on WebShop tasks with 95% confidence interval. The table uses a color spectrum to represent the quality of the numbers. Cells shaded in green indicate the best performance compared to base ReAct, transitioning red for the worst relative performance.
Model Prompt
Act
ReAct
RQ1
RQ2
CoT
Anon. CoT
Placebo
Failure
Explanation
G.TT-35--Turb
1.12
1.04
(0, 5.1)
(0, 4.5)
(0, 3.9)
(0, 7.1)
[0, 7.1)
GPT-3.5-Instruc
7.24
7.16
(2.4, 12.7)
(1.4, 10.8)
(2.3, 12.5)
(2.1, 12.3)
(2.2, 12.3)
GPT-4-0613 (Latest)
4
(2.7, 13.3)
(2.7, 13.3)
(1.3, 10.7)
(2.7, 13.3)
(2.7, 13.3)
GPT-4o
4.64
2.24
(0.5, 8.8)
(0.4, 8.6)
(0.2, 8.0)
(0.5, 8.8)
(0.5, 8.8)
Clande-Opus
4
(0.2, 7.8)
(0, 4.7)
(0.2, 7.8)
(0, 4.7)
(0.2, 7.8)
LLAMA-3.1-SB
1.44
3.16
(0, 6.8)
(0.1, 7.7)
(0, 4.8)
(0, 3.3)
[0, 5.0)
Table 6: Lower Upper bound Success % of L.M for RQ3 on six Al/World tasks with 95% confidence interval. OC: Out of context limit. The table uses a color spectrum to represent the quality of the numbers.
Cells shaded in green indicate the best performance compared to base ReAct, transitioning to red for the worst relative performance.
RQ3
Model / Prompt
Act
ReAct
Domain
Instance
Optimal
All
One
Both
Turr-
34.3
27.6
(0, 4.1)
(21.0, 39.0)
(12.2, 28.0)
(22.9, 41.1)
(19.5, 37.1)
(0, 4.1)
GPTT-3.-Innstret
44
50.7
(37.8, 57.4)
(32.8, 52.2)
(29.9, 49.1)
OC
[10.4, 25.4)
(0.8, 9.6)
GPT-4-0314 (Old)
23.3
(6.6, 20.0)
(15.0, 31.6)
(40.2, 59.8)
(15.0, 31.6)
(9.3, 23.9)
(0, 0)
GPTT-4-0613 (Latest)
70.0
26.7
(4.1, 15.9)
(12.2, 27.8)
(43.5, 63.1)
(15.0, 31.6)
(12.2, 27.8)
(0, 6.8)
Claude-Opus
43.3
56.6
(40.2, 59.8)
(40.2, 59.8)
(33.6, 53.0)
(40.2, 59.8)50
(50.4, 69.6)
(1.7, 11.5)
B.4 Analysis of generated think tags
Example:
Your task find some apple and put it in sidetable.
think tags for example:
think: To solve the lask, need to find and take apple, then put it in sidetable.
> think: First need to find an apple. An apple is more likely to appear in fridges (1), diningtables (1-3), sidetables (1), countertops (1), sinkbasins (1), garbagecan (1). I can check one by onc, starting with fridge 1.
think: Now find apple (3). Next, need to take it.
think: Now find an apple (3). Next, need to put it in/on sidetable 1.
Query:
Your task is to: put pencil in shelf.
ReAct think tags:
lhink: To solve the lask, need to find and take a pencil, then put it in a shelf.
> think: First need to find a pencil. A pencil more likely to appear in a desk (1-2), drawer (1-6), shelf
(1-6). I can chock one by one, starting with desk
18

---

Published in Transactions on Machine Learning Research (05/2025)
think:
Now have the pencil (1). Next, need to put it in shelf.
think:
The shelf empty, so need to find another shelf to put the pencil in. will try shelf
Mhink: The shelf 2 already has a mug it, need to find another empty shelf to put the pencil in, will try shelf 3.
(multiple think tags leading to failure)
Whink: It appears that the pencil is not being accepted in any of the locations in the room. I may need to reconsider the task or find a different approach to solve it.
think: will try putting the pencil on the desk 2, since it the only location left that I haven't tried yet.
Result: FAILURE
RQ1: CoT variation think tags:
To solve the task, need to find and take a pencil, then put it in shelf. First need to find a pencil. A pencil is more likely to appear in desk (1-2), drawer (1-6), shelf (1-6). can check one by one, starting with desk 1. Once find a pencil, next need to take it. Once I take pencil, next need to put in/on shelf.
think:
Result: SUCCESS
RQI: Anonymous CoT variation think tags:
> think: To solve the task, I need to find and take the object, then put on the desired location. First need to find the object. The object is more likely lo appear in one of the different locations. can check one by one, starting with the first location. Once I find the object, next need to take Once take the object, next I need to put it in/on the desired location.
Result: SUCCESS
RQ2: Failure + Explanation variation think tags:
think: Nothing happens because am not shelf
think: Nothing happens because I already have pencil in shelf 1.
think: Nothing happens because am not at desk
Result: SUCCESS
RQ3: Both variation think tags:
> think: To solve the task, need to find a pencil and put in a shelf. A pencil is more likely to appear in a
drawer (1-6), desk (1-2), shelf (1-6), garbagecan (1), laundryhamper (1). I can check one by one, starting with drawer 1.
think: I didn't find pencil in drawer 1. I should check the other drawers. Next, I will check drawer 3.
19

---

Published in Transactions on Machine Learning Research (05/2025)
think: I found pencil (1) desk 1. Next, I need to put it in shelf. A shelf is more likely to appear in shelf (1-6). I can check one by one, starting with shelf 1.
lhink:
didn't find a place to put the pencil in shelf or shelf 2. Next, I will check shell 3.
think: It seems like there is no specific item in the room to place the pencil. may need to reconsider the task and find a different approach. Let me review the task again and see if there are any other clues or hints that may have missed.
think: It scems like there are no houschold members available assist me at the moment. I may need come back to this task later try different approach.
Result: FAILURE
RQ3: One variation think tags:
think: To solve the task, need to find and take a pencil, then put it a shelf. First, need to find a pencil. A pencil is more likely appear in desk (1-2), drawer (1-6), garbagecan (1), or safe (1). can check one by one, starting with desk 1.
think: Now have the pencil (1). Next, I need to find a shelf to put it in. A shelf is more likely to appear in shelf (1-6). can check one by one, starting with shelf 1.
think: It seems that shelf is empty. I should try putting the pencil in different shelf. Let me try shelf 2.
think: It appears that none of the shelves are the correct place to put the pencil. should reconsider where to place it. Let me try putting the pencil in the drawer instead.
think: It seems that none of the locations in the room are suitable for putting the peneil. I may need to reconsider the task and think of different approach. Let me try putting the pencil back the desk where found it originally.
think: It seems that the task cannot be completed in this room. I may need to reassess the instructions or sock further guidance successfully complete the task.
Result: FAILURE
B.5
Additional WebShop Experiment
We run an additional set of experiments with GPT-4o model on WebShop, as shown in Table on 500 sample problems. We once again note that the performance with the newer models as compared to the PaLM model originally used by ReAct is much worse. We also note a similar trend in the relative performance results with our prompt variations against the original ReAct performance, thereby emphasizing the brittleness of the ReAct. framework's s primary claims.
20

---

Published in Transactions on Machine Learning Research (05/2025)
B.6 Task Similarity analysis and Performance Comparison
We did preliminary study the similarity (computed in terms of embeddling distances using the 'MiniLM- L6-v2' model embeddings and computing the cosine similarity between them) between the 134 prompting tasks with the originally provided examples, and the performance of ReAct. As can be seen from Figure as expocted, the performance degrades significantly the off-dingonal cases.
0.35
Embedding Distance
y
Embedding Cistance
be.o
0.20
adc hase
per hwot then, place
ploe heat then place
020
pet cool then place look
220
6-6
pick
Bub
26.7
pack and plaase
mitk staan thae stacee
pet just shee sttecc
pick cort Migkk Hirls
pirk and peance
pree pldne
ged rRet (913
werk tare st
919
slik and place
Botk charr siave
plce hest aifance
pet ree tiai Wiris
fats
tats
Calogery
Catogery
Calapory
(a) Claude Opus
(b) GPT-4
(c) GPT-3.5-Intrrict
Figure 6: Performance Tradeoff with Task Description Similarity in AlfWorld
Table 7: Average Success % of GPT-4o for RQ1 and RQ2 on WebShop tasks. The table uses color spectrum to represent the quality of the numbers. Cells shaded in green indicate the best performance compared to base ReAct, transitioning to red for the worst relative performance.
Model / Prommt
Act
ReAct
RQ1
RQ2
CoT
Anon. CoT
Placebo
Failure
Explanation
GPT-40
8%
4%
9%
8%
7%
8%
5%
C Experiment Design
Each of the variations proposed along RQ1, RQ2 and RQ3 modifies the few-shot examples only. Remaining aspects such as the query problem or the interaction with the simulator is directly inherited from the ReAct base Yao et al. (202bb at publicly available at htttt:///gtteee http:///ggtuuuu....///o/aaa///// commyayyythhReleee Our code can be found the attached supplementary matorial.
Except All RQ3 variation, all other settings use the standard two exemplars for prompting the LLM. Depending the variation we change the content of the exemplar. Full prompts can be found in the attached supplementary code.
C.1 Running the experiments
In our experiments, according to the variation style we take the exemplar prompts and use the same exemplar prompts across the instances of the query task. Other than RQ3-Both, One we use the exemplar of the same task as the query as done in ReAct (and still find brittleness of ReAct). For RQ3 - Both, One we use exactly two exemplars but of a different task than query. Finally, RQ3-A11 is the only variation that provides six exemplars (instead of two) and we force the exemplar of the query-task to be appended at the end in the prompt. This was the best performing prompting strategy (on GPT-3.5- (urrbo)-Turbo) than when the query-task exemplar was placed the beginning, at position 4 (middle) and at the end.
21

---

Published in Transactions on Machine Learning Research (05/2025)
C.2 Hyperparameters
We temperature 0 for all of the GPT and Claude models and set max-tokens 100 which is borrowed from ReAct's hyperparameters. Rest of the parameters are kept to be default as specified in the respective model's API documentation.
D Example Prompts
The full list of curated variations can be found in the supplementary materials. However, for completeness reference the prompt used for base ReAct (as in Yao et al. 2022b)) and our variations along RQ1, RQ2 and RQ3 for the Put task.
D.1 Synonym Substitution mapping for Domain
We make the following substitutions to the object names locations in the exemplar prompt in the Domain variation. Note that these substitutions are done only to the exemplar, and the query problem and subsequent interaction with the AlfWorld simulator uses the original vocabulary mapping.
spraybottle -> aerosolbottle
cabinet -> cupboard
countertop -> worktop
sinkbasin -> sinkboul
toilet lavatory
toiletpaperhanger -> toiletpaperholder
towelholder -> towelrack
microwave -> oven shelf -> rack draver -> deskdrawer
stoveburner -> hob
diningtable table
garbagecan -> trashbin
fridge >> refrigerator
peppershaker >> pepperpot
room -> livingroom
bread breadloaf
pan fryingpan
pot saucepan
book -> notebook
creditcard -> amexcard
mirror -> lookingglass
dresser ~> chestofdrawers
sofa -> couch
cellphone mobilephone
coffeemachine -> coffeemaker
knife ~> kitchenknife
spatula -> turner
soapbottle -> liquidsoapdispenser
saltshaker -> saltpot
statue sculpture
vase flowerpot
dishsponge -> spongewipe
22

---

Published in Transactions on Machine Learning Research (05/2025)
desklamp -> tablelamp
sidetable -> nightstand
D.2
For A11, Both, One
All: We take the exemplar prompt for each task and concatenate it together.
Both We use the following mapping generated arbitrarily, to replace the exemplar prompt for the query task.
VARIATION MMAPIIG = 'put', 'examiin',
clean': 'cool',
'heat' 'put',
ccoll' : 'uuttwo'
'examine' clean',
putttwo 'heat'
One We use the first example from the above mapping, and the second example from the same task as the query.
D.3 Building the input prompt
We inherit ReAct's way of stiching together the exemplars and the query using the following prompt template as
Interact with household solve a task. Here are two examples. eg1 INSTRUCTION Here the task. (TASK)
eg2
In the case of GPT models INSTRUCTION is empty string. Claude Models gave 0% success rate (regardless of the model size, i.e. Haiku, Somnet and Opus) with the original ReAct prompt design. We identified that Claude required an explicit instruction to generate only the actions for the simulator. We added a task independent INSTRUCTION prompt as
INSTRUCTION Only provide one action at a time. Be concise, and do not provide any extra information. Always start with the action. For example, 'pick up apple' is correct, but 'I want to pick up the apple' is not."
Interestingly, GPT-3-Turbo performance dropped to 1% (averaged across 6 independent runs over 134 examples) with the inclusion of this INSTRUCTION tag. This further highlights how brittle prompt engineering is, and cautions us against misplacing our trust with ReAct in enhancing reasoning abilities of LLMs.
Note that, we did experiment with both prompt variations (with our explicit instruction and without) for Claude models and picked the best-performing variant. (which had the instructions) for reported results.
23

---

Published in Transactions on Machine Learning Research (05/2025)
D.4 RQ1
Prompt 1: rgl-amon-exempler
You are in the middle of room. Looking quickly around you, you see cabinet 4, a cabinet 3, cabinet 2, a cabinet 1, diningtable diningtable
coffermachine countertop 1, diningtable microwave sidetable I, sinkhasin 1, stoveburner 4, stoveburner 3, a stoveburner 2, stoveburner and toaster 1. Your task to: find some apple and put in sidetable. think: To solve the task, need to find and take the ohject, then object. The object is more likely to appear in one of the different locations. I can check one by one, starting with the first object, need to put in/on the desired location.
fridge 1, garbogecan
the desired loration. First need to find the
need location. Once find object, OK.
Once
go fridge
The fridge is closed.
> open fridge
You open the fridge I. The fridge open. In it, you letture mug potato and
> go to diningtable
the diningtable 1, you nothing.
go to diningtable 2
the diningtable 2, you
pincapple
to diningtable 3
On the diningtable: 3, you see bowl 2, a howl 1, a bread and a bread 1.
go sidetable
On the sidetable 1, you see 1, a lettuce 1, peppershaker 3, a potato 1, and saltahaker 1.
go countertop
On the countertop you see butterknife 1, dishsponge 1, and soaphottle 1.
go to sinkbasin
On the sinkbasin 1, see cup
mug 1, and potato 3.
go to garbagecan
you apple and egg > take apple from garbagecau
the gerbagecan
You pick up the apple from garbagecan
to siretahle
the sidetable you see cup 1, lettuce 1, peppershaker potato 1, and saltshaker L.
apple 3 in/on sidetable
You put the apple 3 in/on the sidetable
Prompt 2: rql-exemplar
You middle of room. Looking quickly around you, you cabinet cabinet cabinet cabinet coffeemachine 1, a countertop 1, a diningtable 3, diningtable 2, dininglable 1, a drawer 1, a fridge 1, garbagecan 1, a stoveburner
sinkbasin Your task is to: find some apple and put it in sidetahle.
microwave sidetable
stoveburner stoveburner stoveburner and toaster 1.
think: To solve the task, need find and take an apple, then put it in sidetable. First need to find an apple. An apple is likely appear in fridges diningtables (1-3), sidetables (1), countertops (1), sinkbasins garbagoran (1). ran check one by one, starting with fridge 1. Once I find an apple, next need to take it, Once I find an apple, next I need to put in/on sidetable.
OK.
go fridge
The fridge is closed.
>> open fridge
You open the fridge 1. The fridge open. In it, you letture mug potato and tomato
> go to diningtable
the diningtable 1, you nothing.
go to diningtable 2
the diningtable 2, you
pincapple
go to diningtable 3
On the diningtable 3, you see bowl 2, a howl 1, bread and a bread 1.
go sidetable
On the sidetable 1, you see cup 1, a lettuce 1, peppershaker 3, a potato 1, and saltahaker 1.
go countertop
On the countertop you see buttorknife 1, dishsponge 1, and soaphottle 1.
go to sinkbasin
On the sinkbasin 1, see cup
mug 1, and potato 3.
> go to garbageran
you apple and egg > take apple 3 from garbagecau
the gerbagecan
You pick up the apple from garbagecan
to sidetahle
the sidetable you see cup 1, lettuce 1, peppershaker potato 1, and saltshaker L. apple 3 in/on sidetable
You put the apple 3 in/ the sidetable
24

---

Published in Transactions on Machine Learning Research (05/2025)
Prompt 3: base-react
You are in the middle of room. Looking quickly around you cabinet 4, cabinet 3, cabinet 2, cabinet 1, coffeemachine a countertop 1, diningtable microwave sidetable 1, sinkbasin stoveburner stoveburner 3, stoveburner stoveburner and toaster 1. Your task to: find some apple and put in sidetable. think: Finst I need find apple. An apple
diningtable a diningtable a drawer 1, a fridge 1, garbagecan 1,
likely appear in fridges (1), diningtables (1-3), sidetables (1), countertops (1), sinkbasins (1), garbageran (1). I can check one by one, starting with fridge 1.
OK.
to fridge 1
The fridge closed. open fridge
You open the fridge 1. The fridge is open. In it, you see a letture 2, mug 2, a potato 2, and a tomato
go diningtable
On the diningtable 1, you nothing.
go to diningrable
On the diningtable 2, you pineapple 1.
go to diningtable 3
On the diningtable 3, you bowl 2, bowl bread and bread 1.
> go to sidetable
the sidetable you cup lettuce peppersbaker potato and saltshaker
to countertop
the countertop you see a butterknife 1, dishsponge 1, and soapbottle
go to sinkbasin 1
Ou the sinkbasin 1, you see cup 2, a egg 1, a mugg 1, and potato 3.
go garhageran
On the garbagecan 1, you see apple 3, and egg 3.
take apple from garbagecan
You pick up the apple 3 from the garbagecan 1.
think: Now find apple (3). Next, need to put it in/on sidetable OK.
> go to sidetable
the sidetable you cup lettuce peppershaker potato and saltshaker 1.
> put apple 3 in/on sidetable
You put the apple in/o the sidetable
25

---

Published in Transactions on Machine Learning Research (05/2025)
D.5 RQ2
Prompt 4: re2-Fxplanation
You are the middle of room. Looking quickly around you, you see a cabinet a cabinet 3, cabinet 2, a cabinet 1, coffermachine counteriop 1, diningtable 3, diningtable diningtable drawer fridge 1, garbagecan microwave I, a sidetahle I, sinkbasin a stoveburner 4, a stoveburner 3, stoveburner 2, a stoveburner and a toaster 1. find go to fridge 1
apple and put in sidetable. The fridge closed.
Your
close cabinet 2.
Nothing Happens.
think: Nothing happens because you not cabinet OK.
open fridge
You open the fridge 1. The fridge 1 is open. In it, you see lettuce 2, mug 2, a potato 2, and tomato 1.
go diningtable
On the diningtable 1, you nothing.
> go to diningtable
the diningtable 2, you pineapple 1.
> go to diningtable 3
the diningtable 3, you bowl 2, bowl bread bread
to sidetable
sidetable you cup
1, peppershaker a potato I, saltshaker
to countertop
On the counteroop 1, you see a butterknife 1, dishsponge 1, and soaphottle 1.
go sinkbasin
On the sinkbasin 1, you see cup 2, egg 1, a mug 1, and potato 3.
go garbagecan
On the garbagecan 1, you see apple 3, and ogg 3.
put apple in/on sidetable.
Nothing Happens.
> think: Nothing happens because do not have apple OK.
> take apple from garbagecau 1 You pick up the apple from the garbagecan
to sidetahle
the sidetable you see cup I, lettuce I, peppershaker a potato 1, saltshaker apple 3 in/on sidetable
You put the apple 3 in/ the sidetable
26

---

Published in Transactions on Machine Learning Research (05/2025)
Prompt 5: re2-ordering
You are in the middle of room. Looking quickly around you cabinet 4, cabinet 3, cabinet 2, cabinet 1, coffeemachine a countertop 1, diningtable diningtable diningtable a drawer 1, a fridge 1, garbagecan 1, a microwave sidetable 1, sinkbasin stoveburner stoveburner 3, stoveburner stoveburner and toaster 1. Your task to: find some apple and put in sidetable. think: To solve the OK.
need put apple sidetable, for which need find and take apple.
think: apple more likely appear sidetables (I); fridges (I), countertops (I), diningtables (1-3), garbagocan (1),
sinkbasins (1). First need to find an apple. can check one by one, starting with fridge L.
OK.
go to fridge 1
The fridge closed.
open fridge
You open the fridge 1. The fridge is open. In it, you see lettuce 2, mug 2, a potato 2, and tomato 1.
go to diningrable
On the diningtable 1, you nothing.
go to diningtable 2
On the diningtable 2, you pineapple
> go to diningtable 3
the diningtable 3, you
bowl 2, bowl bread bread
to sidetahle
the sidetable 1, you see cup 1, lettuce 1, peppershaker a potato 1, and saltshaker 1.
go to countertop
Ou the countertop 1, you see a butterknife 1, dishsponge 1, and soaphottle
go sinkbasin |
On the sinkbasin 1, you see cup 2, egg 1, a mug 1, and potato 3.
go garbagecan
On the garbagecan I, you see apple 3, and 3.
think: Next, need to take the Now find an apple (3).
OK.
> take apple from garbagecan
You pick up the apple from the garbageran
> think: Next, I need to put the apple in/on sidetable 1. Now find an apple (3).
OK.
go to sidetable
Oa the sidetable you see a cup 1, lettuce 1, a peppershaker 3, potato 1, and a saltshaker 1.
apple in/on sidetable
You put. the apple 3 in/on the sidetable
Prompt 6: re2-Placebo
You are the middle of a room. Looking quickly around you, you see a cabinet 4, a cabinet 3, cabinet 2, a cahinet I, coffeemachine countertop 1, diningtable microwave 1, a sidetable 1, sinkbasin 1, a stoveburner 4, a stoveburner 3, a stoveburner 2, a stoveburner and a toaster 1. some apple put sidetable. think: Take deep breath and work this problem step-by-step.
diningtable diningtable drawer fridge 1, garbagecan
OK.
go to fridge
The fridge 1 is closed.
open fridge
You open the fridge 1. The fridge is open. In it, you see lettuce 2, mug 2, a potato 2, and tomato 1.
go diningtable
On the diningtable 1, you nothing.
> go to diningtable
the diningtable 2, you pineapple 1.
> go to diningtable 3
the diningtable 3, you bowl 2, bowl bread bread
to sidetable
sidetable you cup
1, peppershaker a potato I, saltshaker
go to countertop
On the comnteroop 1, you see a butterknife 1, dishsponge 1, and soaphottle 1.
go sinkbasin
On the sinkbasin 1, you see cup 2, egg 1, a mug 1, and potato 3.
go garbagecan
On the garbagecan 1, you see apple 3, and egg 3. this problem step-by-step.
think: Take doep breath and OK.
> take apple from garbagecan
You up the apple from the garbagecan
> think: Take deep breath and work on this problem step-by-step. OK.
to sidetahle
the sidetable you see cup 1, lettuce I, a peppershaker potato 1, saltshaker L.
apple 3 in/on sidetable
You put the apple 3 in/ the sidetable
27

---

Published in Transactions on Machine Learning Research (05/2025)
Prompt 7: rq2-Failure
You are in the middle of room. Looking quickly around you cabinet 4, cabinet 3, cabinet 2, cabinet 1, coffeemachine a countertop 1, diningtable diningtable a diningtable a drawer 1, a fridge 1, garbagecan 1, microwave sidetable 1, sinkbasin stoveburner stoveburner 3, stoveburner stoveburner and toaster 1. Your task to: find some apple and put in sidetable. şo fridge
The fridge is closed.
close cabinet
Nothing Happens.
> open fridge
You open the fridge 1. The fridge open. In it, you lettuce mug patato and tomato
> go to diningtable
the diningtable you nothing.
to diningtable 2
the diningtable 2, you see pineapple
go to diningtable 3
the disiugtable 3, you see bowl 2, a bowl a bread and a bread
go sidetable
On the sidetable 1, you see cup 1, a lettuce I; peppershaker 3, a potato 1, and saltshaker I.
go countertop
On the countertop you sen a butterknife 1, dishsponge I, and soaphottle 1.
go to sinkbasin
On the sinkbasin
cup
mug 1, and potato 3.
go to garbagecan
the garbagoran 1, you apple 3, and ogg
> put apple 3 in/on sidetable.
Nothing Happeus.
take apple 3 from garbagecan
You pick up the apple from garbagecan
go to sidetable
On the sidetable 1, you see a cup 1, lettuce 1, a peppershaker 3, a potato 1, and saltshaker 1. apple in/on sidetable
You put the apple 3 in/on the sidetable
28

---

Published in Transactions on Machine Learning Research (05/2025)
D.6 RQ3
Prompt 8: rq3-Domain
You are in the middle of livingroom. Looking quickly around you, cupboard 1, coffeemaker a worktop 1, table 3, a table a table 1, a deskdrawer 1, a refrigerator 1, trashbin 1, a oven 1,
cupbnard 4, cuphoard 3, cupboard 2,
hob hob hob hob and toaster 1. Your task to: find some apple and put in nightstand. think: To solve the OK.
nightstand 1, sinkbowl
need find and take apple, then put in nightstand.
need find an apple. apple worktops (I), sinkbowls (1), trashbin (1). I can chock one by one, starting with refrigerator 1.
think:
likely appear refrigerators (1), tables (1-3), nightstands (1),
OK.
go refrigerator
The refrigerator is closed.
oballpointpen refrigerator
You oballpointpen the refrigerator 1. The refrigerator is oballpointpen. In it, see lettuce 2, mug 2, saucepanato 2, and
to table
Ou the table 1, you see nothing.
go table 2
On the table 2, you see pineapple 1.
go table
On the table 3, you see bowl 2, bowl a breadloaf 2, and a breadloaf 1.
go nightstand
On the nightatand you
1, lettuce 1, peppersaucepan saucepanato 1, and saltpot 1.
> go to warktop
the worktop you butterknife spongewipe
liquidsoepdispenser
> go to sinkbow]
the sinkbow! you cup 2, egg 1, mug saucepanato
to trashbin
egn think: Now find apple (3). Next, need to take it.
the trashhin 1, you see apple
take apple from trashbin
You up the apple from the trashbin
> think: Now find an apple (3). Next, I need to put it in/on nightstand 1. OK.
to nightstand
the nightatand you see cup 1, lettuce 1, peppersaucepan saucepanato 1, and saltpot 1.
apple 3 in/on nightstand
You put the apple 3 in/o the nightstand
29

---

Published in Transactions on Machine Learning Research (05/2025)
Prompt 9: rg3-Instance
You are in the middle of room. Looking quickly around you cabinet 4, cabinet 3, cabinet 2, cabinet 1, coffeemachine a countertop 1, diningtable stoveburner stoveburner 3, stoveburner stoveburner and toaster 1. Your task to: find some apple and put in sidetable. need find and take apple, then in sidetable.
diningtable a diningtable drawer 1, a fridge 1, garbagecan 1, a
microwave sidetable 1, sinkbasin
think: To solve the OK.
think:
need find apple. apple more likely appear fridges (1), diningtahles (1-3), sidetables (I),
countertops (1), sinkbasins (1), garbagecan (1). I can check one by one, starting with fridge 1.
OK.
go to fridge 1
The fridge closed.
open fridge
You open the fridge 1. The fridge is open. In it, you see lettuce 2, mug 2, a potato 2, and tomato 1.
go to diningrable
On the diningtable 1, you nothing.
go to diningtable 2
On the diningtable 2, you pineapple
> go to diningtable 3
the diningtable 3, you
bowl 2, bowl bread bread
to sidetahle
On the sidetable 1, you see cup 1, lettuce 1, peppershaker a potato 1, and saltshaker 1.
go to countertop
Ou the countertop 1, you see a butterknife 1, dishsponge 1, and soaphottle
go sinkbasin |
On the sinkbasin 1, you see cup 2, egg 1, a mug 1, and potato 3.
go diningtable
On the diningtable 2,
pineapple 1.
go to diningtable
On the diningtable 3, you bowl 2, bowl bread and bread 1.
go to sidetable
the sidetable you cup lettuce peppershaker potato and saltshaker I.
> go to countertop
the countertop you see butterknife 1, dishspouge 1, soapbottle
go to sinkbasin 1
On the sinkbasin 1, you see cup 2, a egg 1, a mug 1, and potato 3.
go to garbageran
On the garbagocan 1, you see apple 3, and ogg 3.
think: Now find apple (3). Next, need take
> take apple from garbagecau
You pick up the apple from the garbagecan
think: Now find an apple (3). Next, I need to put it in/on sidetable 1. OK.
go to sidetable
On the sidetable 1, you see a cup I, a lettuce I; peppershaker 3, potato I, and saltshaker I.
put apple in/on sidetable
You put the apple 3 in/on the sidetable 1.
Prompt 10: rq3-Optimal
You middle of room. Looking quickly around you, you cabinet cabinet 3, cabinct a coffeemachine countertop 1, diningtable 3, diningtable 2, diningtable 1, drawer a fridge 1, garbegecan 1, a microwave 1, a sidetable 1, sinkbasin a stoveburner a stoveburner 3, a stoveburner 2, a stoveburner 1, and a toaster 1. find > think: To solve the task, need to find and take an apple, then put in sidetable.
Your task
apple and put in sidetable.
OK.
> think: First I need to find an apple. An apple is more likely to appear in fridges (1), diningtables (1-3), sidetables (1),
countertops (1), sinkhasins (1); garbagoran (1). can chork one by one, starting with garhagoran
OK.
go to garbagecan
the garbagecan you apple 3, and egg > think: Now find an apple (3). Next, I need to take it.
take apple from garbagecan
You pick up the apple 3 from the garbageran 1.
think: Now find apple (3). Next, neod to put in/on sidetable
OK.
> go to sidetable
the sidetable you cup lettuce peppershaker potato saltshaker 1.
> put apple 3 in/on sidetable
You put the apple in/on the sidetable
30


Reviews:
[{"review_id": "ZUPC0H7Saw", "reviewer": "Reviewer_WeDX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The submission reveals an important and timely finding, questioning the role of reasoning in planning tasks. The studied problem and findings are critical and timely given the rise of agentic LLMs and reasoning LLMs.\n- The evaluated prompting strategies are comprehensive, including think token formatting, multi-turn-to-sing-turn reformatting, order reversal, anonymization, example similarity-based alteration, etc. In the LLM planning domain, such a systematic evaluation is needed, and even of more value than papers that propose a single strategy that only works in some selected scenarios.\n\nWeaknesses:\n- Evaluation is limited in terms of model selection and task selection. From model selection, the submission can be benefited from evaluation of recent reasoning-enhanced models such as OpenAI o1, o3, Google Gemini Flash Thinking 2.0, and DeepSeek r1. These models are augmented with reasoning processes through <think> tokens in general domains. It would be interesting to see whether they are still limited in generalization in planning scenarios. From the task selection, the submission is heavily based on the study in ReAct (Yao et al, 2022b). It would be interesting to see whether same conclusion holds for tasks beyond those in the original ReAct paper.\n\n- The submission can be improved in terms of better organization, result reproducibility, etc. See Requested Changes below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This submission studies prompting and chain-of-thought strategies in the ReAct (reasoning-action) framework for agentic LLM in simulated planning domains. Through comprehensive evaluation of recent models (GPT-3.5, GPT-4, Claude-Opus, Llama-3.1-8B) on two typical planning tasks (AlfWorld, WebShop), the submission challenges common beliefs that thinking or reasoning might not be the critical factor in some planning scenarios. Instead, the domain consistency of few-shot examples plays a bigger role.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Better organization: It could be better to merge the experiment setup with the results part so that we can introduce the experiment protocols and then the results for each RQ separately. In the current organization, readers have to jump between the two sections to understand each RQ.\n\n- Result reproducibility: For Average Success % numbers, reporting the number of tested instance and the corresponding confidence intervals would be good. Since the tested instances are small and diverse for different settings, we need to include confidence intervals to highlight numbers with potentially high variance and have an apple-to-apple comparison.\n\n- Discussion in Section 5.2: It seems that placing diverse reasoning traces improves the model performance in most cases. Does it imply that models can benefit from more diverse and complicated reasoning traces?\n\n- (Optional) It would be interesting to contact and then report the opinions of ReAct authors for these findings.", "broader_impact_concerns": "No concern that requires Broader Impact Statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IkmRoHJQ2P", "reviewer": "Reviewer_tvtd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper carefully designs prompt variations to examine ReAct’s capabilities.\n- A diverse set of models is evaluated using these variations, leading to insightful conclusions about potential factors influencing ReAct’s performance.\n\nWeaknesses:\n- While prompt design is a key factor, performance is influenced by multiple aspects, including model choice and evaluation benchmarks.\n- The experimental results support the claims within the given setup, but they may not be sufficient to fully justify ReAct’s effectiveness, especially in complex, real-world tasks beyond benchmarks like AlfWorld and WebShop.\n- The finding that exemplar-query similarity plays a crucial role is not surprising, as it aligns with common knowledge in the community.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores the impact of different prompt variations on the claims made in ReAct. The authors design three types of modifications:\n\n- Altering the format of interleaved thinking and acting, such as adopting a CoT-style approach.\n\n- Modifying reasoning traces, e.g., introducing invalid actions or reversing the guidance order.\n\n- Adjusting the similarity between exemplars and queries, such as using synonyms or selecting exemplars from different tasks.\n\nExperiments are conducted using various GPT models, Claude-Opus, and LLAMA-3.1-8B to evaluate Act, ReAct, and the proposed variations. The findings suggest that interleaving thinking and acting, as well as the context of reasoning traces, have minimal impact on performance. Instead, exemplar-query similarity is the primary influencing factor.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Simply strengthen the work in my view: Variation 4 - Exploration Strategy in Section 4.3 is hard to understand.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hq7MpuaxpD", "reviewer": "Reviewer_MVGU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strong Aspects:**\n\n1.  **Timely and Relevant Topic:** The paper addresses a highly relevant and debated topic: the actual reasoning and planning capabilities of LLMs, specifically focusing on the popular ReAct prompting method. Given the widespread adoption of ReAct (cited >2400 times), a critical evaluation is valuable.\n2.  **Systematic and Rigorous Evaluation:** The authors conduct a comprehensive sensitivity analysis, systematically isolating and testing the core claims of ReAct. They design specific variations targeting the interleaving aspect (RQ1), the content of reasoning (RQ2), and the exemplar-query similarity (RQ3). This methodical approach is a significant strength.\n3.  **Clear Research Questions:** The study is well-structured around three clear research questions (RQ1, RQ2, RQ3), making the investigation focused and easy to follow.\n4.  **Reproducibility Efforts:** The authors test their hypotheses across multiple modern LLMs (various GPT versions, Claude, Llama) and in the original domains used by ReAct (AlfWorld, WebShop), strengthening the generalizability of their findings and addressing potential model-specific artefacts. They also provide details on resources used and experiment design, including links to code (though availability wasn't verified).\n5.  **Compelling Results:** The findings consistently challenge the core tenets of ReAct across different models and domains. The strong dependence on exemplar-query similarity (RQ3) and the relatively minor impact of reasoning content/interleaving (RQ1, RQ2) provide compelling evidence against ReAct's claimed mechanism of action.\n6.  **Contextualization with Existing Literature:** The paper effectively positions its findings within the broader context of research on LLM reasoning, prompt engineering, and critiques of emergent abilities, citing relevant contemporary work.\n7.  **Clear Presentation:** The paper is generally well-written and clearly presents its methodology, results (using tables and radar charts effectively), and conclusions. The figures illustrating prompt variations (Fig 2) are helpful.\n\n**Weak Aspects / Requiring Attention:**\n\n1.  **\"Act\" Baseline Definition:** While the \"Act\" baseline is mentioned (no \"think\" tags), its performance relative to ReAct and other variations (especially in Tables 1 & 2) is crucial but sometimes understated in the text. The fact that \"Act\" often performs *better* than ReAct on newer models is a key finding that could be emphasized more as direct evidence against ReAct's utility.\n2.  **WebShop Results Interpretation:** The performance on WebShop is very low across *all* methods (single digits in Table 2). The authors attribute this to decoupling exemplars and queries due to random sampling. While plausible, this extremely low baseline makes it harder to draw strong conclusions about the *relative* differences between methods in this domain. The discussion could elaborate on why performance is so low and the implications for evaluating prompting methods when task success itself is rare.\n3.  **Analysis of Generated \"Think\" Content:** While RQ2 varies the *input* \"think\" content, the paper only briefly touches upon the *quality* or *validity* of the \"think\" content generated by the LLM itself during inference (Section 6). A deeper analysis of *what* the LLM generates when prompted with different variations (especially ReAct vs. CoT) could offer further insights into *why* performance varies. The observation that 40%-90% of subsequent actions after a `think` tag are invalid is interesting and could be explored further.\n4.  **Definition of \"Similarity\":** The concept of \"exemplar-query similarity\" is central to the paper's main conclusion (RQ3). While the variations (Domain, Instance, One, Both) operationalize different facets of similarity, the term itself remains somewhat qualitative. Could more quantitative measures of similarity (e.g., embedding distance, lexical overlap) be used to correlate with performance drops? This might further solidify the link between similarity and performance.\n5.  **Optimal Prompt Placement (Minor):** Appendix C.1 mentions testing different placements for the query-task exemplar in the `All` variation. While it's stated the end position was best, including this data point, even briefly in the appendix, might add completeness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper critically evaluates the effectiveness of ReAct-style prompting, specifically its claim that interleaving reasoning traces (\"think\" tags) with action execution improves the planning abilities of Large Language Models (LLMs). The authors conduct a sensitivity analysis by systematically varying aspects of the ReAct prompt in the AlfWorld and WebShop domains, using various LLMs including GPT-3.5, GPT-4, GPT-4o, Claude, and Llama 3.1-8B.\n\nThe key contributions and findings are:\n\n1.  **Interleaving reasoning is not the primary driver of performance:** Contrary to ReAct's claims, the study finds that LLM performance often *improves* when the reasoning trace is *not* interleaved with actions (RQ1). Chain-of-Thought (CoT) style prompting, where reasoning is done upfront, generally performs better or comparably to ReAct.\n2.  **Content of reasoning traces has minimal impact:** The specific content or structure of the \"think\" tags (e.g., providing foresight guidance vs. hindsight/failure explanation vs. placebo text) has a surprisingly small effect on performance (RQ2). Even placebo guidance performs comparably in many cases, suggesting the reasoning content itself might not be crucial.\n3.  **High dependence on exemplar-query similarity:** The study demonstrates a severe lack of generalization. Performance drastically drops when there are even minor variations between the few-shot examples in the prompt and the queried task (RQ3). This includes using synonyms for objects/locations, changing the goal instance slightly, or using examples from different but related tasks within the same domain. The LLMs appear to rely heavily on literal pattern matching or approximate retrieval from the provided examples rather than genuine reasoning or planning.\n4.  **Questioning emergent reasoning:** The results challenge the notion that ReAct successfully elicits robust reasoning and planning abilities. The observed performance seems largely attributable to the high similarity between prompts and queries, forcing prompt engineers to provide highly instance-specific examples, which limits scalability and practical applicability.\n5.  **Reproducibility and Consistency Issues:** The authors note discrepancies with original ReAct results (which used PaLM) when using newer models, and highlight inconsistencies in performance even across different versions of the same model family (e.g., GPT-4 variants). Performance of the simple \"Act\" baseline (no think tags) is often surprisingly strong, further questioning the added value of ReAct's core mechanism.\n\nIn essence, the paper argues that the perceived benefits of ReAct prompting stem primarily from the close similarity between few-shot examples and the target task (acting like a nearest-neighbor retrieval mechanism) rather than the interleaved reasoning traces themselves. This \"brittleness\" and lack of generalization call into question the claims of enhanced reasoning and planning capabilities fostered by the ReAct method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Here are proposed adjustments, categorized by importance:\n\n**Critical Adjustments:**\n\n1.  **None.** The paper's core methodology, results, and conclusions are sound and well-supported by the extensive experiments. The findings convincingly challenge the claims of ReAct based on the provided evidence.\n\n**Suggested Adjustments (to strengthen the work):**\n\n1.  **Elevate Discussion of \"Act\" Baseline Performance:** In Sections 5.1, 5.2, and the Discussion (Sec 6), more explicitly highlight and discuss the implications of the \"Act\" baseline often outperforming ReAct, especially with newer models (Tables 1 & 2). This directly contradicts ReAct's core premise and strengthens the paper's argument significantly. *[Strength: High - directly supports the main thesis]*\n2.  **Refine Interpretation of WebShop Results:** Acknowledge the extremely low absolute success rates in WebShop more directly in Section 5.3. Discuss the potential limitations this imposes on interpreting the *relative* differences between prompting strategies in this domain. Briefly speculate on *why* performance might be so degraded compared to the original ReAct paper (beyond just random sampling, e.g., model changes, API differences?). *[Strength: Medium - adds nuance and addresses potential reviewer concerns]*\n3.  **Expand Analysis of Generated Thoughts (Optional but valuable):** If feasible, briefly expand Section 6 ('Operationalizing ‘think’ actions') or add an appendix section with qualitative examples or quantitative analysis (beyond just invalid action rates) of the *content* generated within `think` tags by the LLMs under different conditions (e.g., ReAct vs. CoT vs. Placebo). Does the LLM generate meaningful reasoning, or just boilerplate text? This could provide deeper insight into the mechanism (or lack thereof). *[Strength: Medium - provides deeper mechanistic insight]*\n4.  **Quantify \"Similarity\" (Optional):** Consider adding a brief discussion or supplementary analysis exploring more quantitative measures of exemplar-query similarity (e.g., BLEU score, ROUGE, embedding cosine similarity between task descriptions or full traces) for the RQ3 variations (especially `Domain`, `One`, `Both`). Correlating these quantitative similarities with the observed performance drops could add another layer of evidence to the claim that literal similarity is the key driver. *[Strength: Low-Medium - adds rigor but may be complex to implement thoroughly]*\n5.  **Clarify Hyperparameters/Implementation Details:**\n    *   Ensure the referenced supplementary material/code is accessible and contains the full prompts and implementation details as stated (Appendix C).\n    *   Briefly state the number of examples used for each variation (e.g., \"two exemplars were used for all variations except 'All' which used six\"). This is implied but could be stated explicitly for clarity early in Section 4 or C. *[Strength: Low - improves clarity and reproducibility]*", "broader_impact_concerns": "The paper primarily focuses on a technical evaluation of a specific LLM prompting technique (ReAct) and its claimed effectiveness for reasoning and planning tasks. The core findings relate to the limitations of this technique and the nature of LLM performance (pattern matching vs. reasoning).\n\nBased on this focus, there do not appear to be immediate, direct ethical implications that would necessitate a dedicated Broader Impact Statement beyond what is standard for ML research. The work does not:\n\n*   Introduce new datasets containing sensitive information.\n*   Deploy models in high-stakes real-world scenarios.\n*   Directly address fairness, bias, or malicious use cases in a way that introduces novel risks.\n\nHowever, one could consider **indirect implications**:\n\n1.  **Over-reliance on LLMs for Critical Tasks:** By highlighting the brittleness and lack of true generalization/reasoning in current LLM prompting methods (even sophisticated ones like ReAct), the work implicitly cautions against deploying these systems in critical decision-making or planning scenarios where robustness and true understanding are required. This finding itself promotes responsible AI development by revealing limitations.\n2.  **Cost and Environmental Impact:** The experiments involve significant computation using large models (GPT-4, Claude Opus). While the authors provide token estimates (Appendix A), acknowledging the environmental and financial costs associated with such large-scale LLM experimentation is generally good practice in the field, though not typically requiring a full BIS unless the scale is exceptionally massive or introduces novel efficiency concerns.\n\nThese points are general considerations for much LLM research rather than specific ethical risks raised uniquely by this paper's methodology or findings. Therefore, a separate Broader Impact Statement is likely not strictly necessary, although a brief mention of the implications for responsible deployment given the observed lack of robustness could be considered for the conclusion or discussion.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The work addresses a timely and understudied issue, with carefully designed ablation studies. The conclusions are well-supported and challenge widely held assumptions.  

The scope is narrow (limited to ReAct’s original benchmarks), and broader generalization (e.g., newer reasoning-augmented models like O3, R1, Gemini Flash Thinking 2.0) remains unexplored. However, the authors’ focused analysis is sufficient for TMLR’s standards.  

The authors have already addressed key reviewer concerns (e.g., adding confidence intervals, clarifying examples, and broader impact statements). Minor revisions should ensure clarity in describing prompt variations and their implications.
Decision: Accept (poster)