ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent
Authors: name: Renat Aksitov; name: Sobhan Miryoosefi; name: Zonglin Li; name: Daliang Li; name: Sheila Babayan; name: Kavya Kopparapu; name: Zachary Fisher; name: Ruiqi Guo; name: Sushant Prakash; name: Pranesh Srinivasan; name: Manzil Zaheer; name: Felix Yu; name: Sanjiv Kumar
Published: 2023-12-15 18:20:15

Abstract: Answering complex natural language questions often necessitates multi-step
reasoning and integrating external information. Several systems have combined
knowledge retrieval with a large language model (LLM) to answer such questions.
These systems, however, suffer from various failure cases, and we cannot
directly train them end-to-end to fix such failures, as interaction with
external knowledge is non-differentiable. To address these deficiencies, we
define a ReAct-style LLM agent with the ability to reason and act upon external
knowledge. We further refine the agent through a ReST-like method that
iteratively trains on previous trajectories, employing growing-batch
reinforcement learning with AI feedback for continuous self-improvement and
self-distillation. Starting from a prompted large model and after just two
iterations of the algorithm, we can produce a fine-tuned small model that
achieves comparable performance on challenging compositional question-answering
benchmarks with two orders of magnitude fewer parameters.