SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 3: Results on WebShop with different exemplar combinations (initial experiences for RE-MEMBERER) and different training sets (for REMEMBERER).  $E_{i}$  denotes the different exemplar combinations, while  $S_{i}$  denotes the different training sets. The first line of each method shows the mean scores, and the second line shows the success rates.
Different (Initial) Exemplars	Different Training Sets
E0+S0	E1+S0	E2+S0	E0+S1	Avg	Std
ReAct	0.72	0.65	0.60	-	0.66	0.06
0.42	0.35	0.30	-	0.36	0.06
LLM only	0.52	0.54	0.59	-	0.55	0.04
0.26	0.28	0.32	-	0.29	0.03
RMMBR.	0.66	0.71	0.66	0.67	0.68	0.02
0.37	0.41	0.37	0.40	0.39	0.02
tasks. Specifically, the LLM is input with the task description, the screen representation, and the step instruction. The screen is represented in an HTML element sequence following Zhang et al. [2023]. Additionally, the last 5 performed actions along with the last reward are given to the LLM as the procedure feedback. As for the output, the LLM is prompted to print the HTML representation of the operated element as the extra information. This is expected to force the LLM to discover the relation between the element id and the certain element. The task similarity  $f_{g}$  designed for WikiHow is computed from the step instructions. It is noticed that the instructions follow some patterns, thus, we inspect the instructions and categorize them into six types. Then a similarity lookup table is designed according to the instruction types. The details should be referred to in the supplementary. The observation similarity  $f_{o}$  is computed based on the length of the longest common sequence of the HTML elements in the screen representation:
$$
f _ {o} \left(s c _ {1}, s c _ {2}\right) = \frac {l c s \left(s c _ {1} , s c _ {2}\right)}{\max  \left\{l e n \left(s c _ {1}\right) , l e n \left(s c _ {2}\right) \right\}}. \tag {8}
$$
The full-trajectory expanding is adopted, as most of the tasks will end in 5 steps as well.
4.2 Results on WebShop
REMEMBERER is applied to WebShop with 2-shot in-context learning. The experience memory is initialized with four annotated experiences of the decision step from one trajectory. The agent is trained for 3 epochs on a training set containing 10 different tasks outside the test sets used by Yao et al. [2022b] and Shinn et al. [2023]. To control the total expense and achieve bootstrapping, the succeeded tasks in the first epoch are excluded from training in the following two epochs. The trajectories exceeding 15 steps are considered to be failed, as most of the tasks can end in 5 steps. The main results are shown in Table 1. We used the public ReAct [Yao et al., 2022b] implementation released by the authors and run with text-davinci-003 instead of text-davince-002 in Yao et al. [2022b]. The run of ReAct shares the same trajectory as the exemplar with REMEMBERER. The "LLM only"
Table 4: Comparison of the number of annotated trajectories and steps of REMEMBERER and the IL baseline. The number of steps of the training set of IL is estimated according to the average human trajectory length on the test split as 11.3 in Yao et al. [2022a].
Method	#Trajectories	#Steps
IL	1,012	~11,436
REMEMBERER	1	4
Table 5: Comparison of the number of the tasks in the training set and the updating steps of RE-MEMBER with the IL and RL baselines. The number of the updating steps of IL is estimated from 10 epochs on 1,012 trajectories with an average trajectory length of 11.3.
Method	#Tasks	#Steps
RL	10,587	100,000
IL	-	~114,356
REMEMBERER	10	74
Table 6: Results on WikiHow with different exemplar combinations (initial experiences for REMEMBERER) and different training sets (for REMEMBERER).
Different (Initial) Exemplars	Different Training Sets
E0+S0	E1+S0	E2+S0	E0+S1	Avg	Std
LLM only	2.56	2.60	2.59	-	2.58	0.02
0.90	0.90	0.89	-	0.90	0.01
RMMBR.	2.63	2.63	2.59	2.66	2.63	0.03
0.93	0.91	0.90	0.97	0.93	0.03
baseline indicates a single LLM with 2 fixed exemplars sampled from the initial experiences of REMEMBERER. The average performance of REMEMBERER exceeds the baseline by a large extent and surpasses the prior state of the art, ReAct, as well. This proves the effectiveness of augmenting the LLM with an external evolvable experience memory. The proposed REMEMBERER also outperforms the RL, IL (imitation learning), and IL+RL baselines on both metrics.
In order to verify the robustness of REMEMBERER, experiments with different initial experience combinations or a different training set are conducted. The results are depicted in Table 3. The initial experience combination  $E_0$  denotes the certain trajectory adopted by the original implementation of ReAct while  $E_1$  and  $E_2$  are randomly sampled from  $S_0$ . It is observed that the proposed REMEMBERER can achieve better and more stable results with different initialization and training sets compared to ReAct. Thus, REMEMBERER can mitigate the workload to some extent to search for an optimal exemplar combination.
We compare the training efficiency of REMEMBERER with the conventional IL and RL methods in Table 4 and Table 5. In contrast to the IL, REMEMBERER requires quite few annotated samples to initialize the experience memory, while IL is in need of much more human annotations. REMEMBERER agent can be trained on only 10 tasks for 74 steps, while the RL and IL are expected to be trained for about 100 thousand steps to achieve an acceptable performance. Consequently, the proposed REMEMBERER offers a much more efficient way to build a practical agent agilely.
4.3 Results on WikiHow
REMEMBERER is applied to WikiHow with 2-shot in-context learning. The experience memory is initialized with two annotated experiences of the decision step. The agent is trained for 3 epochs on a training set containing 10 different tasks selected from WikiHow excluding the test tasks. Similar to the experiments on WebShop, the succeeded tasks in the first epoch are excluded from training in the following two epochs. As observed that most of the tasks require an interaction of less than 5 steps, the trajectory exceeding 15 steps will be regarded as failed. The main results are depicted in Table 2. The exemplars of the "LLM only" baseline are the initial experiences of REMEMBERER. The proposed REMEMBERER surpasses the baseline as well as the original result in Zhang et al. [2023]. In addition, 10 tasks are annotated to form an annotated experience memory. REMEMBERER agent with this annotated experience memory is evaluated without further training, and the result is denoted as "RMMBR. (A)" in the table. This result demonstrates that REMEMBERER is capable of
Table 7: Comparison of the average reward estimation of the full model and the ablation model without bootstrapping policy. The error is the absolute difference between the average reward estimation from the experience memory and the real training reward.
Task Set	Setting	Avg Reward Estimation	Avg Training Reward	Abs Error
WebShop	Full Model	0.86	0.84	0.02
w/o bootstrp.	0.62	0.84	0.22
WikiHow	Full Model	2.48	2.60	0.12
w/o bootstrp.	1.98	2.70	0.72
Table 8: Results of ablation study
Task Set	Setting	Avg Reward/Score	Success Rate
WebShop	Full Model	0.66	0.37
w/o bootstrp.	0.67	0.36
w/o random	0.65	0.37
WikiHow	Full Model	2.63	0.93
w/o bootstrp.	2.54	0.89
w/o random	2.64	0.90
w/o discouraged	2.48	0.81
w/o task sim. fg	2.63	0.94
w/o observ. sim. fo	2.47	0.87
exploiting expert experiences, which can be regarded as analogous to conventional imitation learning. Nevertheless, the annotated experiences may not offset the exact shortage of the particular LLM. In contrast, the RL training will have an opportunity to collect more specific experiences and gain a more promising performance.
The experiments with different initial experience combinations or a different training set are conducted on WikiHow as well, and the results are shown in Table 6. The proposed REMEMBERER achieves a consistent improvement compared to the baseline with fixed exemplars, which proves the effectiveness and robustness of REMEMBERER.
Several ablation studies are conducted to verify the design of REMEMBERER framework.
Ablation on  $n$ -step bootstrapping policy Ablation studies are conducted to verify the necessity of  $n$ -step bootstrapping policy to update the  $Q$  value estimations in the experience memory. As stated in Subsection 3.2, updating without bootstrapping may lead to inaccurate value estimations owing to few training steps to explore and exploit. In order to verify this perspective, an average reward estimation is calculated by averaging the sum of the maximum  $Q$  value and the history reward stored for each observation in the experience memory:
$$
\hat {R} = \frac {1}{M} \sum_ {i = 1} ^ {M} \left(R _ {h} \left(g _ {i}, o _ {i}\right) + \max  _ {a} Q \left(g _ {i}, o _ {i}, a\right)\right), \tag {9}
$$
where  $R_{h}$  denotes the total reward of the steps before  $(g_{i}, o_{i})$  on the trajectory and  $M$  is the size of the memory. The deduced average reward estimation  $\hat{R}$  is compared to the real training reward, and an absolute error is calculated in Table 7. It can be observed that the reward estimation from the experience memory trained without bootstrapping suffers a far greater error than that with bootstrapping. Meanwhile, the performance on the test set is demonstrated in Table 8. Although there is no apparent disparity in the final performance on the WebShop task set, a visible degradation is observed on WikiHow, which reveals the latent risk of a non-bootstrapping update.
Ablation on the advice generation strategy As stated in Subsection 3.3, owing to the nonexhaustive exploration in the brief training stage, there may be no suitable candidates for the action advice in the exemplars. For instance, there may be no actions recorded with a poor enough  $Q$  value estimation or no actions recorded as high-reward. Under this case, action advice can be generated with a randomly sampled action that is not in the record, or it can be given by directly encouraging the action with the highest  $Q$  value estimation and discouraging the action with the lowest estimation without regard to the certain value. These two strategies are compared in Table 8. As the results illustrate, the random plan appears to have a minor superiority over the non-random plan. This is attributed to that advice with improper value expectations will mislead the LLM to take wrong judgments about the true value of the available actions.
Additional experiments are conducted to investigate the necessity of the discourage actions in the output part of exemplars and the impact of similarity function components. Owing to limit of budgets, these experiments are only conducted on WikiHow task set.
Ablation on necessity of the discouraged actions The proposed output format "action advice" comprises both encouraged and discouraged actions. The discouraged actions are believed to help the LLM to avoid similar failures. Results in Table 8 prove necessity of the discouraged actions. Without access to the discouraged actions, the agent can only achieve a much poorer performance than the full model. In the case shown in the supplementary, it can be seen that there may not be proper actions to encourage in the retrieved experience. In such cases, the discouraged actions are especially crucial for the agent to prevent repeating similar mistakes.
Ablation on the similarity function As stated in Subsection 3.3, a similarity function is required to select related experiences from the memory. In experiments, the similarity is implemented as two components: task similarity  $f_{g}$  and observation similarity  $f_{o}$ . Ablation studies are conducted to draw a brief perspective on the impact of these two components. As shown in Table 8, removal of task similarity seems not to affect the performance remarkably, while removal of observation similarity causes a serious degradation. This may indicate that on these tasks, the tested LLM benefits more from experiences that have similar observations rather than similar instruction patterns. On the other side, the pattern-based task similarity for WikiHow introduced in Subsection 4.1 may be too coarse to cluster the experiences. During interaction, the agent may receive instructions of the same pattern (e.g., "access article ABC") while facing different types of observation (e.g., search result page or category page). The appropriate actions in two situations are also different. Removal of observation similarity will eliminate this difference in experience selection and results in misleading. Case study in the supplementary shows this perspective.
We introduce Reinforcement Learning with Experience Memory (RLEM) to aid the LLM in learning from its interaction experiences for decision-making tasks. A novel LLM-based agent framework called REMEMBERER is then designed with RLEM by equipping the LLM with a persistent experience memory and updating the memory with the RL algorithm. REMEMBERER agent is capable of exploiting the interaction experiences to improve its policy and gains a significant improvement compared to the baseline. Our experimental results demonstrate the superiority. Owing to the simplicity and effectiveness of REMEMBERER, we believe that this work provides a valuable perspective on designing evolvable LLM-based agents with RLEM.
The proposed REMEMBERER agent demonstrates strong superiority on the tested benchmarks. Nevertheless, it is wondered how this framework will be applied to the environments with more long-term episodes or with more extensive or visual-rich observations. Besides, it is observed that the performance of REMEMBERER will encounter quick saturation in training process. This may be due to the limited number of active exemplars. Further efforts are expected to be dedicated in to make the agent performance evolve continuously. Furthermore, as an early exploration, we didn't make use of complicated RL techniques. How recent advancement in RL domain works under RLEM is also an interesting problem.
This work is funded by the China NSFC Project (No.62106142 and No.62120106006) and Shanghai Municipal Science and Technology Major Project (2021SHZDZX0102).