SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Danyang Zhang
Lu Chen $^{1,2\dagger}$
Situo Zhang
Hongshen Xu
Zihan Zhao
Kai  $\mathbf{Y}\mathbf{u}^{1,2}$
$^{1}$ X-LANCE Lab, Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence, SJTU AI Institute Shanghai Jiao Tong University, Shanghai, China  $^{2}$ Suzhou Laboratory, Suzhou, China
{zhang-dy20,chenlusz,situozhang,xuhongshen,zhao_mengxin,kai.yu}@sjtu.edu.cn
Inspired by the insights in cognitive science with respect to human memory and reasoning mechanism, a novel evolvable LLM-based (Large Language Model) agent framework is proposed as REMEMBERER. By equipping the LLM with a long-term experience memory, REMEMBERER is capable of exploiting the experiences from the past episodes even for different task goals, which excels an LLM-based agent with fixed exemplars or equipped with a transient working memory. We further introduce Reinforcement Learning with Experience Memory (RLEM) to update the memory. Thus, the whole system can learn from the experiences of both success and failure, and evolve its capability without fine-tuning the parameters of the LLM. In this way, the proposed REMEMBERER constitutes a semi-parametric RL agent. Extensive experiments are conducted on two RL task sets to evaluate the proposed framework. The average results with different initialization and training sets exceed the prior SOTA by  $4\%$  and  $2\%$  for the success rate on two task sets and demonstrate the superiority and robustness of REMEMBERER. $^1$
Reasoning is remembering. As declared by Seifert et al. [1997], the episodic memory of the experiences from past episodes plays a crucial role in the complex decision-making processes of human [Suddendorf and Corballis, 2007]. By recollecting the experiences from past episodes, the human can learn from success to repeat it and learn from failure to avoid it. Similarly, an agent should optimize its policy for a decision-making task with the help of reminiscence of the interaction experiences. In this work, we primarily investigate how to utilize large language models (LLMs) as agents and equip them with interaction experiences to solve sequential decision-making tasks.
Despite the impressive performance of LLMs on many natural language processing (NLP) tasks [Wei et al., 2022, Kojima et al., 2022, Wang et al., 2022, Yao et al., 2022b], existing approaches still struggle to enable LLMs to effectively learn from interaction experiences. On the one hand, the most common approach for an agent to utilize the experiences is to fine-tune the model parameters through reinforcement learning (RL). However, it requires a considerable expenditure to deploy and fine-tune an LLM, which makes it difficult to apply task-aware RL to the LLM to preserve the experiences. On the other hand, recent work like Algorithm Distillation [Laskin et al., 2022] presents an in-context
$\dagger$  Lu Chen is the corresponding author.
<sup>1</sup>The codes are open-sourced at https://github.com/OpenDFM/Rememberer.
Different RL Task Goal
Experience Memory
(a) LLM-based agent with short-term goal-aware working memory
Figure 1: Comparison of the LLM-based agents with short-term working memory and long-term experience memory. The working memory stores only the historical information of the current episode  $(\mathcal{H})$ , while the experience memory stores the interaction experiences  $(\mathcal{E})$  permanently.
(b) LLM-based agent with long-term cross-goal experience memory
reinforcement learning by embedding the RL training trajectories into the input prompt of a pretrained decision transformer. This method manages to make use of past interaction experiences without model fine-tuning. However, existing LLMs suffer from a serious limitation of the input length to embed the whole experience. Hence, to better store a plethora of interaction histories and aid LLMs in learning during the interaction process, we introduce RLEM, i.e., Reinforcement Learning with Experience Memory, which accomplishes agent learning by updating the experience memory through the RL process, rather than modifying the model parameters.
An external experience memory is different from the existing work like Reflexion [Shinn et al., 2023] which combines the LLM with a short-term working memory. As depicted in Figure 1 (a), a working memory is tied to a specific task goal, and the stored histories cannot be leveraged in future episodes for different goals. This analogy can be drawn to the Random Access Memory (RAM) of a computer, where stored information is lost in the event of power removal. On the other side, learning from the successful or failed experiences stored in the memory is different from the existing work like Inner Monologue [Huang et al., 2022b], Corrective Re-Prompting [Raman et al., 2022], and DEPS [Wang et al., 2023b] that takes advantage of immediate failure feedback only once. Storing long-term experiences in a persistent memory gives an opportunity to discover the late failure and learn from the experiences in the past episodes even for different task goals (see Figure 1 (b)).
By combining RLEM with LLM, we propose REMEMBERER, an evolvable LLM-based agent framework for decision-making tasks. REMEMBERER can utilize the experiences stored in the memory selectively in accordance with the current interaction state to optimize the decision. Meanwhile, the experience memory can be updated through an RL process constantly. Such a joint system is regarded as a semi-parametric RL agent, which can evolve its ability through its interaction experiences analogically to a full-parametric system, however, without fine-tuning the LLM parameters. We evaluate REMEMBERER on two recent RL task sets with the promising performance of LLM-based agents, WebShop [Yao et al., 2022a] and WikiHow [Zhang et al., 2023]. The agent is trained on a few tasks and tested on some other tasks to check whether the experiences from different tasks can help the agent in the decision of the unseen episodes. REMEMBERER demonstrates a significant performance boost compared to both previous SOTA and our fixed-exemplar LLM baselines. Specifically, it achieves an average improvement of 2 points and 4 points on the Webshop and WikiHow tasks, respectively, compared to the SOTA models.
Our contributions are summarized as follows: 1) A new agent framework is proposed as REMEMBER for LLM to learn from the experiences of past episodes. The experiences are stored in an external persistent memory instead of fine-tuning the LLM parameters or forming an extremely long prompt. 2) We introduce RLEM, which updates experience memory through analogical RL training so that REMEMBER is capable of self-evolving. 3) REMEMBER manages to bypass the baseline
and the prior advanced performances and set up a new state of the art on two recent benchmarks, WebShop (+2 points on SOTA) and WikiHow (+4 points on SOTA).
LLM with external information External information is usually adopted to augment the LLM with the environment-grounded information, or to reduce the hallucination, or to unleash the ability to process longer context. Connecting with an external knowledge base is a common choice for question-answering and conversational tasks [Peng et al., 2023, Schick et al., 2023, Trivedi et al., 2022, Pan et al., 2022]. However, an external knowledge base is usually not directly corresponding to an RL environment and cannot provide environment-grounded assistance to an automatic agent. Meanwhile, the update to a mature knowledge base may not be in time for the instant interaction of the agent with the environment. In contrast, Schuurmans [2023] simulates a universal Turing machine with a RAM-augmented LLM and demonstrates the capability of a quickly-updatable working memory. Liang et al. [2023] and Zhong et al. [2023] adopt memory to store the conversational history and handle extremely long contexts. Relational database is leveraged to track states in a dynamic process by ChatDB [Hu et al., 2023]. Reflexion [Shinn et al., 2023] exploits a working memory to store experiences for a dedicated task to improve the performance of the agent through several trials. However, as illustrated in Figure 1, the histories stored in working memory cannot benefit the episode for different task goals. Instead, a long-term cross-goal experience memory should be considered. MemPrompt [Madaan et al., 2022] and Ret-LLM [Modarressi et al., 2023] adopt a persistent memory to store human feedback and remind the chatbot of the conversational knowledge and improve it continuously. Voyager [Wang et al., 2023a] designs a skill library to store the past learned skills as JavaScript functions. A simple text experience pool is adopted by GITM [Zhu et al., 2023] to store the successful trajectories for future referencing. Somewhat similar to GITM, REMEMBERER adopts a persistent environment-grounded experience memory to store the experiences and assist in future decision-making even for different task goal. However, instead of plain text records of successful trajectories, REMEMBERER uses a structured memory and designs a mechanism to task advantage of both successful and failed experiences. The experiences come from the interaction of the agent with the environment, and no human intervention is needed.
LLM learning from failure Learning from failure is one of the characteristic capabilities of human and turns to be an important topic for general artificial intelligence. Some work has explored the ability of the LLM to learn from its failure [Huang et al., 2022b, Raman et al., 2022, Wang et al., 2023b]. Nonetheless, most of such work takes advantage of immediate feedback from the environment and the correction is used only once. In practice, several late failures may be due to some early mistaken actions in an episode. Reflexion [Shinn et al., 2023] designs a heuristic function to detect late failure from the interaction history and stores the LLM-generated reflection in a working memory for use in the next trial. However, these reflections cannot be applied to different task goals. Madaan et al. [2022] stores the failure corrections for a long term, but relies on human feedback. In contrast, REMEMBERER adopts RL to learn from both late and immediate failure from the environment rewards without need for human feedback. Also, REMEMBERER enables the experiences to be reused in the future episode even for a different task goal with a long-term experience memory.
LLM for decision-making The powerful capability of LLM is exploited by recent work [Huang et al., 2022a, Raman et al., 2022, Mees et al., 2022, Chen et al., 2022, Ichter et al., 2022, Huang et al., 2022b, Liang et al., 2022] to generate better control plans for various robots and agents. Kim et al. [2023] and Zhang et al. [2023] design LLM-based agents for user interface (UI) interaction. ReAct [Yao et al., 2022b] combines the action decision with natural language reasoning and achieves a promising performance. To our best knowledge, This work is the first one that combines the LLM-based agent with RL algorithm to learn from the interaction experiences and achieve self-evolving.
The proposed REMEMBERer equips the LLM with an external experience memory to help it to learn from both successful and failed experiences. This is also the first work to combine the LLM-based agent with RL algorithm to improve the capability of the agent.
Figure 2: Pipeline of RLEM and architecture of REMEMBERER
3.1 RLEM pipeline
RLEM (Reinforcement Learning with Experience Memory) is proposed for an LLM-based agent to learn from its interaction experiences by updating an external persistent memory. The pipeline of RLEM and the architecture of REMEMBER agent are depicted in Figure 2. REMEMBER agent consists of two components: an LLM making decisions and an experience memory storing the interaction experiences. At the decision step, the LLM first takes an observation  $o_{t}$  from the environment. The observation  $o_{t}$  is then adopted to retrieve several related experiences from the connected experience memory according to some similarity functions. The experiences are represented as a group of observations  $O_{x}$ , actions  $A_{x}$ , and the corresponding  $Q$  value estimations  $Q_{x}$ . Here  $x$  denotes the index set of retrieved experiences and depends on the specific runtime observation  $o_{t}$ . Subsequently, LLM will decide the action  $a_{t}$  in accordance with  $o_{t}$ , the feedback from the last interaction (e.g., the reward  $r_{t-1}$ ), as well as the retrieved experiences  $(O_{x}, A_{x}, Q_{x})$ .  $a_{t}$  will be executed in the environment and the resulted reward  $r_{t}$  will be returned to the LLM as the feedback. And the transition tuple,  $(o_{t}, a_{t}, r_{t}, o_{t+1})$ , comprising the last observation, the taken action, the corresponding reward, and the new observation will be used to update the experience memory. The following subsections will detail the structure and updating policy of REMEMBER experience memory and the usage of the retrieved experiences.
3.2 Experience memory of REMEMBERER
The experience memory is one of the pivotal components of the proposed REMEMBER framework. It is adopted to store the interaction experiences, and the LLM is expected to benefit from the stored experiences in future decision-making. The memory can be regarded as a group of external parameters of the LLM-based agent. Such an agent is a semi-parametric system that can evolve through RL process. During the interaction, new experiences are added to the experience memory so that the overall system can attain a more capable interaction ability compared to the agents with just a fixed LLM
Task & Observ.	Action	Q Value
(g1, o1)	a1	q1
(g2, o2)	a2	q2
(g3, o3)	a3	q3
:	:	:
Figure 3: An example of the records stored in the proposed experience memory.
and fixed exemplars. This procedure can be considered analogous to the training stage of conventional parametric agents.
To be specific, the experience memory is designed as a table storing the task information, observation, action, and the corresponding  $Q$  value estimation. The  $Q$  value is the expectation of the accumulated future reward and gives an assessment of the value of the action candidate. Figure 3 depicts a demonstration of the proposed experience memory. There are two stages to build a practical REMEMBER agent with experience memory: initialization and training. The experience memory is supposed to be first initialized with some initial records before the training stage. The initial
records are necessary to inform the LLM of the format of the input and the output. Then, during the analogical training stage, the agent interacts with the environment to collect new experiences, and conducts off-policy learning [Sutton and Barto, 1999]. Particularly, given the task information  $g$  and the new transition  $(o_{t}, a_{t}, r_{t}, o_{t+1})$ , as a quadruple of the last observation, action, reward, and the new observation, a new estimation is calculated first according to the estimated Bellman optimality equation [Bellman, 1952] as
$$
Q ^ {\prime} (g, o _ {t}, a _ {t}) = r _ {t} + \gamma \max  _ {a} Q (g, o _ {t + 1}, a). \tag {1}
$$
Here max can be calculated from the actions already recorded for  $(g, o_{t+1})$  by considering the  $Q$  value of unrecorded actions 0, if the action space cannot be traversed, e.g., action space involving free-form language. Then a new record is inserted directly if there does not exist a record associated to  $(g, o_t, a_t)$  in the memory:
$$
Q \left(g, o _ {t}, a _ {t}\right) = Q ^ {\prime} \left(g, o _ {t}, a _ {t}\right). \tag {2}
$$
If  $(g, o_t, a_t)$  has been already inserted into the record, the recorded  $Q$  value estimation will be updated by Q-Learning [Watkins and Dayan, 1992]:
$$
Q \left(g, o _ {t}, a _ {t}\right) \leftarrow (1 - \alpha) Q \left(g, o _ {t}, a _ {t}\right) + \alpha Q ^ {\prime} \left(g, o _ {t}, a _ {t}\right). \tag {3}
$$
Here the learning rate,  $\alpha$ , is  $1 / N$  where  $N$  denotes the times this value is updated. As Equation 1 may lead to an inaccurate estimation owing to insufficient sampling out of few training steps of REMEMBERER,  $n$ -step bootstrapping [Mnih et al., 2016] is adopted to ameliorate this problem, which estimates  $Q^{\prime}$  by
$$
Q ^ {\prime} (g, o _ {t}, a _ {t}) = \sum_ {i = 0} ^ {n - 1} \gamma^ {i} r _ {t + i} + \gamma^ {n} \max  _ {a} Q (g, o _ {t + n}, a), \tag {4}
$$
where  $n$  is the steps to expand. The ablation study in Subsection 4.4 proves this perspective.
3.3 Usage of the experiences
In order to assist the LLM in making decisions, the stored experiences are adopted as dynamic exemplars for few-shot in-context learning. Given the task goal  $g$  and the current observation  $o_{t}$ , a similarity function  $f$  is used to calculate the similarity of  $(g,o_{t})$  with  $(g_{i},o_{i})$  from the memory.
$$
S _ {i} = f \left(\left(g, o _ {t}\right), \left(g _ {i}, o _ {i}\right)\right). \tag {5}
$$
Commonly, a similarity function  $f$  can be divided into two components, task similarity  $f_{g}$  and observation similarity  $f_{o}$ :
$$
S _ {i} = \lambda f _ {g} (g, g _ {i}) + (1 - \lambda) f _ {o} \left(o _ {t}, o _ {i}\right). \tag {6}
$$
The  $m$  records with the highest similarities are retrieved to form the exemplars in the prompt. The particular similarity function designed for each task set is detailed in Subsection 4.1.
The exemplar is supposed to demonstrate the format of the input and the output to the LLM. The input part usually comprises the task information and the observation, along with some interaction feedback or auxiliary information. The particular input format depends on the task domain and will be detailed in Subsection 4.1. The output part indicates the action decision. Specifically, we propose to present the action
Last 5 Actions:
search[3 ounce bottle bright citrus deodorant sensitive skin]
Observation:
Instruction: i would like a 3 ounce bottle ... [Back to Search]
[B078GWRC1J]
[B078GTKVXY]
[B08KBVJ4XN]
Available Actions:
Encouraged:
click[b078gwrc1j]  $\rightarrow$  1.0 b078gwrc1j and b078gtkxy are bright citrus deodorant less than 50 dollars. I can check b078gwrc1j first.
Discouraged:
click[b087wksr2g] -> 0.0 b087wksr2g is not the desired item.
Figure 4: An exemplar for WebShop task set [Yao et al., 2022a]. The input part is depicted in the upper box and the output part is depicted in the lower box. Action candidates are advised along with their  $Q$  value estimations and some optional extra information.
decisions in a form of "action advice" comprising both encouraged and discouraged actions rather than simply present an action to execute. This is motivated by the perspective that "reasoning is
remembering" to exploit both successful and failed experiences. To form the output part in the exemplar, the actions with the highest  $Q$  value estimations from the retrieved record are given as the encouraged actions, while the actions with poor  $Q$  value estimations (e.g., zero or negative estimations) are given as the discouraged actions. It is believed that the advice with high value expectations can lead the LLM to follow the past success, while the advice with poor expectations will teach the LLM to avoid a similar failure. A clear depiction of the exemplar format can be found in Figure 4. Prompted by such exemplars, the LLM will also predict both encouraged and discouraged actions and speculate their  $Q$  values given a new input. The predicted  $Q$  values are used to select the optimal action, to be specific, the encouraged action with the highest  $Q$  value speculation will be executed in the environment.
It is worth noting that REMEMBERER agent necessitates only a limited number of training steps to achieve a promising performance, which leads to a non-exhaustive action record within its memory. Consequently, instances may arise where there is only one action associated with a given context  $(g, o_t)$ , or the highest  $Q$  value remains deficient, or no sufficiently unfavorable action exists to discourage. In such cases, randomly sampled action advice is favored over encouraging an action with low expectations or discouraging an action with moderate expectations. Our ablation study in Subsection 4.4 sheds light on various strategies for generating advice in such scenarios.
4 Experiments & results
4.1 Experiment setup & implementation details
To assess the effectiveness of REMEMBERER, we evaluate it on two recent task sets with the promising performance of LLM-based agents: WebShop and WikiHow. All the experiments are conducted based on the OpenAI API of GPT-3.5 [Brown et al., 2020] text-davinci-  $003^{2}$
WebShop [Yao et al., 2022a] WebShop is a task set simulating a web store site. The agent is instructed to browse the site and shop for the target goods. The information of over 1M products is crawled from the Amazon store<sup>3</sup>. About 12K product requests are re-written by crowd laborers to generate more diverse instructions. A score between 0 and 1 will be rated after shopping by assessing the correspondence between the product and the instruction. We followed Shinn et al. [2023] and conducted our experiments on the first 100 tasks from the same shuffled task list released along with the task set. At each interaction step, the LLM takes the web page representation and a list of available actions as input. The task instruction is omitted, for there is always an instruction present on the top of the web page. As there are no intermediate rewards during the episode, only the last 5 performed actions serve as procedure feedback. Inspired by the chain-of-thought technique [Wei et al., 2022] and the ReAct mechanism [Yao et al., 2022b], the LLM is prompted to predict a reason for its decision as the extra information depicted in Figure 4. The representation of the web pages is simplified in the same way as ReAct. The task similarity  $f_{g}$  is calculated using the all-MiniLM-L12-v2 model from Sentence-Transformers [Reimers and Gurevych, 2019]. As it is noticed that the web pages in WebShop are instantiated from some templates, we categorize the web pages into four patterns and design a similarity lookup table to compute the observation similarity  $f_{o}$  according to the web page patterns. The details about the similarity table should be referred to in the supplementary. It is observed that most of the tasks end in 5 steps, thus we directly conduct a full-trajectory expanding while performing multi-step bootstrapping:
$$
Q ^ {\prime} \left(o _ {t}, a _ {t}\right) = \sum_ {\tau = t} ^ {T} \gamma^ {\tau - t} r _ {\tau}. \tag {7}
$$
WikiHow [Zhang et al., 2023] WikiHow is a task set based on the collaborative wiki app WikiHow<sup>4</sup> running on the interaction platform Mobile-Env [Zhang et al., 2023]. The task set contains amounts of navigation tasks. The target of the agent is to follow the instructions and navigate to the required page. Intermediate rewards and instructions may be triggered during the episode. We followed Zhang et al. [2023] and evaluated the proposed REMEMBERER on the "canonical subset" comprising 70
$^{2}$ https://openai.com/api/
<sup>3</sup>https://www.amazon.com/
<sup>4</sup>https://www.wikihow.com/Main-Page
Table 1: Results on WebShop. The result of the prior state of the art, ReAct [Yao et al., 2022b], is attained with the public implementation released by the original authors. The RL, IL, and  $\mathrm{IL + RL}$  results are retrieved directly from Yao et al. [2022a].
Method	Avg Score	Success Rate
LLM only	0.55	0.29
ReAct	0.66	0.36
RMMBR.	0.68	0.39
RL	0.55	0.18
IL	0.60	0.29
IL+RL	0.62	0.29
Table 2: Results on WikiHow. "Mobile-Env" indicates the prior result from Zhang et al. [2023]. "RMMBR. (A)" denotes the results by directly running the evaluation of REMEMBERER with a human-annotated experience memory.
Method	Avg Reward	Success Rate
LLM only	2.58	0.90
Mobile-Env	2.50	0.89
RMMBR.	2.63	0.93
RMMBR. (A)	2.56	0.91