Do Think Tags Really Help LLMs Plan? A Critical Evaluation of ReAct-Style Prompting

Authors: Mudit Verma, Siddhant Bhambri, Subbarao Kambhampati
Primary area: foundation or frontier models, including LLMs
Keywords: Large Language Models, ReAct, Sequential Decision-Making

Abstract:
The reasoning abilities of Large Language Models (LLMs) remain a topic of debate, which are critically tested in sequential decision-making problems. ReAct, a recently popular method has gained popularity for claiming to enhance LLM reasoning abilities while directly prompting them by $``\textit{interleaving reasoning trace with action execution}"$ in text-based planning domains such as AlfWorld and WebShop. However, given the different components of ReAct-style prompting, it remains unclear what the source of improvement in LLM performance is. In this paper, we critically examine the claims of ReAct-style prompting for sequential decision-making problems. By introducing systematic variations to the input prompt, we perform a sensitivity analysis along the original claims of ReAct. Contrary to these claims and common use-cases that utilize ReAct-style prompting, we find that the performance is minimally influenced by the interleaved reasoning trace or by the content of these generated reasoning traces. Instead, the performance of LLMs is primarily driven by the unreasonably high degree of similarity between input example tasks and queries, implicitly forcing the prompt designer to provide instance-specific examples which significantly increases the cognitive burden on the human. Our empirical results, on the same suite of domains as ReAct, show that the perceived reasoning abilities of LLMs stem from the exemplar-query similarity and approximate retrieval rather than any inherent reasoning abilities.

Decision: reject
Reviewer scores: 5, 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1729866743277

thread_type:
official_review

<thread_object>
creator: reviewer_5h7h

<content>
summary:
This paper examines the impact of different components of ReAct-style prompting for planning problems. The authors propose several kinds of alterations to the ReAct prompts, and find that 1) the LLM's performance is minimally influenced by the interleaved reasoning trace or the content of the reasoning trace in the prompt; 2) the performance heavily depends on a high degree of similarity between the demonstrations and queried problems. These findings indicate that LLM's reasoning/planning abilities are more fragile than normally perceived.

strengths:
This work studies a wide range of settings to understand how different components of the ReAct prompts affect the model's performance. The results are insightful for researchers and practitioners to better interpret what ReAct prompts are doing and the current LLMs' reasoning/planning performance.

weaknesses:
- **Incomplete analysis for supporting the claims**. The paper does not include enough details/analyses about what the LLMs' actual generations are like and their comparison with the (altered) demonstrations. Are LLMs following the changed demonstrations when addressing new problems, or are they still doing things in the original ReAct style? For example, is it possible that the LLMs are still doing interleaved reasoning and acting, even though the demonstrations are changed to have all thought steps together in the beginning? For cases where the performance drops a lot (e.g., the Domain, Instance variations), are these new errors caused by the model's decreased reasoning abilities, or simple mistakes around the surface-form symbols? Relatedly, the authors often make claims that are somewhat ambiguous on the target. For example, in lines 416-418: "...the performance of LLMs either improved or remained consistent when provided with weaker or irrelevant guidance information. This refutes ReAct’s claim that task-specific reasoning trace is the source of LLM agent performance". Is this "task-specific reasoning trace" the ones in demonstrations or those generated by the model? The results only show that LLMs don't need such reasoning traces in the demonstrations, but the LLMs could still generate good traces during inference.

- **Results are overall not surprising**. It is well known that LLMs are usually not "learning" how to perform the task from the demonstration examples, rather, the prompt mostly provides the overall format and some key anchors such as label/action space related to the test problems to shape the model generation [1, 2]. There are prior works showing that one doesn't even need to provide these explicit demonstrations for the model to work, e.g., just saying "think step by step" could elicit CoT behaviors of LLMs. It is also well-known that providing examples that are more similar to the queried problems brings better performance, and many prior efforts on demonstration selection are exactly about closing the gap between the demonstrations and queries, e.g., [3, 4]. LLMs, or ML models more broadly, generally suffer from distributional shifts which is one of the open research problems. Reporting this in some specific tasks/settings is not very significant in my view.

--

Citations
- [1] Min et al. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? EMNLP-22.
- [2] Wang et al. Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters. ACL-23.
- [3] Liu et al. What Makes Good In-Context Examples for GPT-3? DeeLIO-22.
- [4] Rubin et al. Learning To Retrieve Prompts for In-Context Learning. NAACL-22.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729883498607

thread_type:
official_review

<thread_object>
creator: reviewer_7upd

<content>
summary:
This paper examines how effective ReAct-style prompting is for improving Large Language Models (LLMs) in decision-making tasks. While ReAct is known for boosting LLM reasoning by combining reasoning and actions, this study questions where these improvements really come from. The authors conduct experiments in planning tasks in AlfWorld and WebShop to analyze the effects of reasoning traces, plan guidance, and how similar examples are to the queries.

Contributions: The paper tests ReAct's claims and finds that LLM performance doesn't significantly improve by manipulating reasoning with action execution. The results show that the main reason for LLM performance is the high similarity between example tasks and queries, rather than reasoning abilities from reasoning traces. A detailed sensitivity analysis shows that LLM performance drops when the example-query similarity is reduced, highlighting ReAct's fragility. The study challenges the belief that LLMs develop reasoning abilities through ReAct-style prompting, showing that success is mostly due to pattern matching or retrieving similar examples.

strengths:
Originality: The paper offers a new look at ReAct-style prompting by questioning if interleaving reasoning with actions truly improves LLM performance. It highlights example-query similarity as the main factor driving success, offering a different perspective.

Quality: The paper is well-structured, with clear experiments tested across different LLM models and domains. The use of varied tests and detailed analysis strengthens the reliability of the findings. The overall experiment setups are convincing and comprehensive.

Clarity: The paper is easy to understand and well-organized. Proposed ideas are reasonable, and tables and figures effectively explain the results, making the findings accessible to a wide audience.

Significance: The paper raises important questions about ReAct-style prompting, showing that example-query similarity may play a bigger role than reasoning in LLM performance. This insight could help guide future research and improve LLM prompting methods.

weaknesses:
While the paper offers important insights into the limitations of ReAct-style prompting, it doesn’t fully address whether these findings apply across different scenarios and domains. The study focuses on specific tasks in AlfWorld and WebShop, but it’s unclear how generalizable the results are to other environments or more complex tasks. For example, would the same reliance on example-query similarity hold in tasks with more diverse or less structured action spaces? The lack of broader applicability raises concerns about the scalability of the conclusions, making it hard to know if the findings can be generalized to all situations where ReAct prompting is used.

The study uses black-box LLMs without fully clarifying what matters more for performance—the ReAct-style prompting itself or the inherent capabilities of the LLMs. Since LLMs differ in architecture and training, it's difficult to separate the effects of the prompt from the underlying model’s abilities. This makes it unclear whether the performance issues are caused by the prompting technique or limitations within the models themselves. Ideally, more transparency in how the LLMs are interacting with the prompts and what exactly drives the observed behavior would help strengthen the conclusions.

The paper also lacks a strong technical contribution beyond critiquing ReAct. There is no obvious novel method or solution offered to address the identified weaknesses in ReAct-style prompting. While finding flaws in prompting techniques is important, the absence of a proposed solution limits the paper’s impact. Readers are left with an understanding of what is wrong but without a clear path to improve or resolve these issues. A more balanced approach would include recommendations or a framework for improving prompting techniques, which would provide more concrete value to readers.

questions:
Regarding scalability: Could you clarify how your findings apply to more complex environments or tasks beyond AlfWorld and WebShop? For instance, would the same dependency on example-query similarity hold in domains with less structured actions or greater variability?

Regarding justification of approaches: In your experiments, how much do you attribute the performance outcomes to the prompting method versus the underlying capabilities of the LLMs themselves?

Regarding workflow completeness: Did you explore how the size of the context window or the amount of training data provided to the LLMs impacts performance in your scenarios? While the analysis of ReAct’s limitations is thorough, do you have suggestions or possible remedies for reducing the reliance on example-query similarity?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730694839109

thread_type:
official_review

<thread_object>
creator: reviewer_c2ur

<content>
summary:
This paper explores the impact of different components of ReAct-style prompting on the final outcomes, showing that both interleaved reasoning traces and generated reasoning traces have minimal influence on results. In contrast, the similarity between examples has a much more significant effect.

strengths:
1. This paper provides a valuable exploration of how ReAct functions in decision-making tasks.

2. I highly appreciate the motivation for existing work, which holds great importance for the research community.

weaknesses:
1. The writing in the paper is difficult to follow. Prompts take up too much space in each paragraph, and some words are struck through without clear explanations. The font size in the figures is too small. I am confused about the names "GPT-3.5-instruct" and "GPT-3.5-Turbo" as the authors refer to "gpt-3.5-turbo-0125" and "gpt-3.5-turbo-instruct". I also struggled to understand what "Act" is in the tables 1 as it's hard to find the introduction of baselines. Overall, the paper is not ready for publication.

2. The examples provided, such as finding an item, do not seem to have sufficient changes in the environment. The authors suggest that giving LLMs a complete action plan at the outset (e.g., if A happens, do X; if B happens, do Y) is feasible, as shown in Figure 2. I disagree as many situations could change the environment, making it impossible to provide all potential scenarios upfront.

3. Regarding RQ1 and its two variants, I didn't find any intrinsic difference between these variants and React. I feel that the variants are just human-rewritten versions of React.  A more reasonable comparison would be to directly present the user’s query to the model and let it generate a similar plan prompt. The current setup, where the authors design prompts based on potential behaviors, seems unfair.

4. The conclusion that the "similarity between the example and the query" is important feels rather obvious. Since LLMs rarely encounter agentic structure data in their training, the significance of in-context learning is naturally high. This conclusion lacks sufficient insight for me.

questions:
1. In Table 1, the explanation for GPT-3.5 outperforming GPT-4o is “highlighting the brittleness of claims of ReAct.” I don’t quite understand this rationale, as it seems to imply that weaker prompting leads to more significant performance drops in stronger models. Shouldn’t stronger LLMs be more robust to suboptimal prompting? Could you provide some examples to clarify this counterintuitive result?

2. The Webshop dataset shows a low accuracy. I believe analyzing such low accuracy is not that meaningful. Other papers report much higher figures. Can you explain the reason?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730723106113

thread_type:
official_review

<thread_object>
creator: reviewer_gqdu

<content>
summary:
This paper studies into what makes ReAct-type of prompting works. It studies into 3 factors: (1) where the guidance is provided, (2) the different types and structure of this guidance (3) on varying the resemblance of example prompt to the queried problem.

strengths:
It conducts a very detailed sensitivity study into the effectiveness of ReAct prompting.

weaknesses:
Studies on prompt engineering and what parts of prompt work for final result has been studied for a long time. Various research on why and what types of intermediate thinking chains/in-context learning can work (such as "Rethinking the Role of Demonstrations") and on what models they work have been studied by many papers. However, no such papers are cited or discussed.

Many of the observational conclusions such as similarity to in-context examples and uselessness of thinking trace are also discussed by various papers, and thus renders this paper's conclusion not so exciting.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1732163042361

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal

comment:
We thank all the reviewers for their thoughtful reviews. We are gratified to note that all the reviewers acknowledge the fact that our systematic analysis raises significant questions about the claims in the ReAct paper. Some of the reviewers expressed reservations about the fact that we only point out the flaws in the claims made in the ReAct paper, and don’t ourselves provide solutions to rectify those flaws. We offer two rebuttals on this: first ReAct is not just any paper–but a rather influential paper (currently 1739 citations) whose claims have largely been accepted at face value. There are very recent papers [eg. 1-6] that repeat ReACT claims and claim to further extend them. So pointing out inherent flaws in ReAct is, we believe, an important addition to the science of LLM prompting. Second, the kind of claims ReAct makes are such that there is no actual way to  make them true, as has perhaps been shown by recent work evaluating the planning (in)capabilities of LLMs [7, 8]. We hope the reviewers take this view into consideration in evaluating the strength of our contribution. 


[1]Yao Yao, Zuchao Li, and Hai Zhao. Beyond chain-of-thought, effective graph-of-thought reasoning in large language models. arXiv preprint arXiv:2305.16582, 2023.

[2]Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366, 2023.

[3] Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381, 2023.

[4] Lluis Castrejon, Thomas Mensink, Howard Zhou, Vittorio Ferrari, Andre Araujo, and Jasper Uijlings. Hammr: Hierarchical multimodal react agents for generic vqa. arXiv preprint arXiv:2404.05465, 2024.

[5] Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, and Ziran Wang. Receive, reason, and react: Drive as you say, with large language models in autonomous vehicles. IEEE Intelligent Transportation Systems Magazine, 2024.

[6] Yunjia Zhang, Jordan Henkel, Avrilia Floratou, Joyce Cahoon, Shaleen Deep, and Jignesh M Patel. Reactable: Enhancing react for table question answering. arXiv preprint arXiv:2310.00815, 2023.

[7] Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models-a critical investigation. Advances in Neural Information Processing Systems, 36, 2024.

[8] Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. Chain of thoughtlessness: An analysis of cot in planning. arXiv preprint arXiv:2405.04776, 2024
</content>
</thread_object>

</thread>


Metareview: