ReAct: Synergizing Reasoning and Acting inLanguage Models
Berk Atil
04/03/2023

Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., & Cao, Y. (2022). React:
Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629.

Outline
●
●
●
●
●
●

Motivation
Methodology
Results on Knowledge Intensive Reasoning
Results on Decision Making
Human in the loop
Conclusion2

Motivation and Background● Prompting: Basically, we embed the task description that we wantfrom the model to solve into the input.
● Prompt Engineering: The focus is more on the prompts that weprovide to the model, trying to figure out a template/structure of aprompt that performs best.
● This can be used in both zero-shot and few-shot setting.
● [1] is one of the earliest work on prompting where they show thatfew-shot learning can provide good performance instead offine-tuning models.
[1] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners.
Advances in neural information processing systems, 33, 1877-1901.

3Motivation● Humans combine verbal reasoning with actions.
● Synergy between reasoning and acting● Emergent capability of language models [1].
○ Only internal knowledge, no update on knowledge○ Fact Hallucination● LMs can be used for planning actions in interactive environments[2]○ No reasoning, no action taking[1] Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., & Zhou, D. (2022). Chain of thought prompting elicits reasoning in large
language models. arXiv preprint arXiv:2201.11903.
[2] Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., ... & Fu, C. K. (2023, March). Do as i can, not as i say: Grounding language
in robotic affordances. In Conference on Robot Learning (pp. 287-318). PMLR.

4

5

Why Not Combining These Two?
● Models can reason to act to create and adjust plans● Also they can act to reason by interacting with external environments

6

ReAct●●●●●●Prompting based approach with PaLM [1]Enlarge the Action Space with Language Space
Sequence of “Thought”, “Action”, “Observation”
Easy to design
Flexible
Human aligned and controllable

7[1] Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., ... & Fiedel, N. (2022). Palm: Scaling language modeling with pathways. arXiv preprint
arXiv:2204.02311.

Knowledge Intensive Reasoning
●
●
●
●
●

Multi-hop Q&A, 6-shot
Fact Verification, 3-shot
3 actions: search, lookup, finish
Multiple thought-action-observation steps
ReAct provides grounded problem solving but chain-of–thought
provides more accurate reasoning○ Incorporate both and let model switch from one to the other.

8Results● Reasoning is important to guideacting● ReAct vs CoT● ReAct + CoT-SC [1] is the best[1] Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., & Zhou, D. (2022).
Rationale-augmented ensembles in language models. arXiv preprint
arXiv:2207.00747.

9Finetuning Results●10Decision Making● ALFWorld [1], synthetic text-based game○ 6 types of tasks that an agent needs to achieve a high-levelgoal.
○ Sparse thoughts/reasonings in prompts.
● WebShop [2], online shopping website environment.
○ Based on user instructions, it should buy a product.

[1] Shridhar, M., Yuan, X., Côté, M. A., Bisk, Y., Trischler, A., & Hausknecht, M. (2020). Alfworld: Aligning text and embodied environments forinteractive learning. arXiv preprint arXiv:2010.03768.
[2] Yao, S., Chen, H., Yang, J., & Narasimhan, K. (2022). Webshop: Towards scalable real-world web interaction with grounded language agents.
arXiv preprint arXiv:2207.01206.

11

Results on ALFWorld● It depends highly on the prompts● Sparse reasoning helps.
12Results on WebShop

13

Human in the loop correction● Edit and correct ReAct’s false reasonings

14Conclusion, Limitations and Future Work● Reasoning and action helps models reach to better conclusions.
● There is still room for improvement, especially for knowledge
intensive reasoning tasks it is behind the supervised methods● Prompt design affects the performance● When the action space is large the size of the incontext learning
grows quickly.
● More high quality labelled data for finetuning● Incorporation with Toolformer [1], a language model that caninteract with more tools such as calculator, calendar, translator etc.
[1] Schick, T., Dwivedi-Yu, J., Dessì, R., Raileanu, R., Lomeli, M., Zettlemoyer, L., ... & Scialom, T. (2023). Toolformer: Language
models can teach themselves to use tools. arXiv preprint arXiv:2302.04761.

15

Thanks!
● Any Questions?

16