Large Language Models Are Semi-Parametric Reinforcement Learning Agents
Authors: Danyang Zhang
Lu Chen
Situo Zhang
Hongshen Xu
Zihan Zhao
Kai Yu
Venue: NeurIPS 2023 poster (2023)
Keywords: Learning from Experiences
LLM
Reinforcement Learning
Decision Making
Experience Memory

Abstract: Inspired by the insights in cognitive science with respect to human memory and reasoning mechanism, a novel evolvable LLM-based (Large Language Model) agent framework is proposed as Rememberer. By equipping the LLM with a long-term experience memory, Rememberer is capable of exploiting the experiences from the past episodes even for different task goals, which excels an LLM-based agent with fixed exemplars or equipped with a transient working memory. We further introduce **R**einforcement **L**earning with **E**xperience **M**emory (**RLEM**) to update the memory. Thus, the whole system can learn from the experiences of both success and failure, and evolve its capability without fine-tuning the parameters of the LLM. In this way, the proposed Rememberer constitutes a semi-parametric RL agent. Extensive experiments are conducted on two RL task sets to evaluate the proposed framework. The average results with different initialization and training sets exceed the prior SOTA by 4% and 2% for the success rate on two task sets and demonstrate the superiority and robustness of Rememberer.

Large Language Models Are Semi-Parametric Reinforcement Learning Agents
Danyang Zhang'
Lu Chen 1.2
Situo Zhang'
Hongshen Xu
Zihan Zhao!
Kai Yu
X--LANCE Lab, Department of Computer Science and Engineering MoE Key Lab of Artificial Intelligence, SJTU AI Institute Shanghai Jiao Tong University, Shanghai, China Suzhou Laboratory, Suzhou, China
(zhang-dy20, chenlusz, www.nlusz,,sstuozzhheeeeeeeo www.:/ooooannn..oonssssae zzaaeoheennha....o cahha_mengxin..mm kai yu edw.e.ttuuuuu..... edu. cn
Abstract
Inspired by the insights in cognitive science with respect human memory and reasoning mechanism, novel evolvable LLM-based (Large Language Model) agent framework is proposed as REMEMBERER. By equipping the LLM with long- term experience memory, REMEMBERER is capable of exploiting the experiences from the past episodes even for different task goals, which excels an LLM-based agent with fixed exemplars equipped with transient working memory. We further introduce Reinforcement Learning with Experience Memory (RLEM) to update the memory. Thus, the whole system learn from the experiences of both success and failure, and evolve its capability without fine-tuning the parameters the LLM. In this way, the proposed REMEMBERER constitutes semi- semi- carimee RL agent. Extensive experiments are conducted on two RL. task sets to evaluate the proposed framework. The average results with different initialization and training sets exceed the prior SOTA by 4% and 2% for the success rate on two task sets and demonstrate the superiority and robustness of REMEMBERER
Introduction
Reasoning remembering. As declared by Seeifert al. [1997], the episodic memory of the experiences from past episodes plays a crucial role in the complex decision-making processes of human [Suddendorf' and Corballis 2007]. By recollecting the experiences from past episodes, the human can learn from success to repeat it and learn from failure to avoid it. Similarly, an agent should optimize its policy for decision miriiinnmmmaingggg task with the help of reminiscence of the interaction experiences. In this work, we primarily investigate how to utilize large language models (LLMs) as agents and equip them with interaction experiences solve sequential decision decision-making making tasks.
Despite the impressive performance of LLMS on many natural language processing (NLP) tasks (Wei et al., 2022 Kojima al., 2022 Wang al. 2022 Yao et al. 20226). existing approaches still struggle to enable LLMs to effectively learn from interaction experiences. On the one hand, the most common approach for agent utilize the experiences
fine-tune the model parameters through reinforcement learning (RL). However, it requires a considerable expenditure to deploy and fine-tune an LLM, which makes it difficult apply task-aware RL the LLM to preserve the experiences. On the other hand, recent work like Algorithm Distillation Laskin et al. 2022 presents an in in-context
Luu Chen is the corresponding author.
'The codes are open-sourced httppp//gggttabb.ooo///DD/////ee
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Task
Working Memory
Experience Memury
dt
01
az
d4
dt
0t
4t
H'
H'
0.2
H'
e
H
01
RAM 11:10101
RAM
RAM
(a) LLM-based agent with short-term goal awale working memory
(b) LLM--based agent with long-term cross-goal experience memory
Figure 1: Comparison of the LLM-based agents with short-term working memory and long- term experience memory. The working memory stores only the historical information of the current episode (H). while the experience memory stores the interaction experiences (E) permanently.
reinforcement learning by embedding the RL training trajectories into the input prompt of a pretrained decision transformer. This method manages to make use of past interaction experiences without model fine-tuning, However, existing LLMs suffer from serious limitation of the input length to embed the whole experience. Hence, to better store a plethora of interaction histories and aid LLMs in learning during the interaction process, we introduce RLEM, i.e., Reinforcement Learning with Experience Memory, which accomplishes agent learning by updating the experience memory through the RL process, rather than modifying the model parameters.
external experience memory different from the existing work like Reflexion (Shinn al. 2023] which combines the LLM with short-term working memory. As depicted in Figure | (a), working memory is tied to a specific task goal, and the stored histories cannot be leveraged future episodes for different goals. This analogy can be drawn to the Random Access Memory (RAM) of computer, where stored information is lost in the event of power removal. On the other side, learning from the successful or failed experiences stored in the memory is different from the existing work like Inner Monologue (Huang et al., 20226], Corrective Promppttinggg [Raman et al. 2022], and DEPS (Wang et al. 2023b] that takes advantage of immediate failure feedback only once. Storing long-term experiences
persistent memory gives an opportunity to discover the late failure and learn from the experiences in the past episodes even for different task goals (sce Figure (b)).
By combining RLEM with LLM, we propose REMEMBERER, an evolvable LLM-based agent framework for prcisionnmakingg tasks. REMEMBERER can utilize the experiences stored in the memory selectively in accordance with the current interaction state to optimize the decision. Meanwhile, the experience memory can be updated through an RL process constantly. Such a joint system is regarded as a semi- camimearaettri RL agent, which can evolve its ability through its interaction experiences analogically to full-parametric system, however, without fine-tuning the LLM parameters. We evaluate REMEMBERER on two recent RL task sets with the promising performance of LLM-based agents, WebShop Yao et al. 2022a] and WikiHow (Zhang et al.) 2023). The agent is trained on few tasks and tested on some other tasks to check whether the experiences from different tasks can help the agent in the decision of the unseen episodes. REMEMBERER demonstrates significant performance boost compared to both previous SOTA and our fixed-exent exeemee LLM baselines. Specifically, achieves an average improvement of 2 points and 4 points on the Webshop and WikiHow tasks, respectively, compared to the SOTA models.
Our contributions are summarized as follows: 1) A new agent framework is proposed as REMEM BERER for LLM to learn from the experiences of past episodes. The experiences are stored in an external persistent memory instead of fine-tuning the LLM parameters or forming an extremely long prompt. 2) We introduce RIEM,, which updates experience memory through analogical RL. training so that REMEMBERER is capable of self-evolving. 3) REMEMBERER manages to bypass the bascline

---

and the prior advanced performances and set up new state of the art on two recent benchmarks,
WebShop (+2 points on SOTA) and WikiHow (+4 points on SOTA).
2
Related work
LLM with external information External information is usually adopted to augment the LLM with the coroommmnn--roonder information, or reduce the hallucination, or to unleash the ability to process longer context. Connecting with an external knowledge base is a common choice for question-answernen
and conversational tasks [Peng et al.] 2023 Schick et al. 2023 Trivedi et al.) 2022 Pan et al. 2022]. However, an external knowledge base is usually not directly corresponding
to an RL environment and cannot provide environment- grounddd assistance to an automatic agent. Meanwhile, the update to a mature knowledge base may not be in time for the instant interaction of the agent with the environment. contrast, (Schruura,Ssuu [2023) simulates a universal Turing machine with a RAM-augmented LLM and demonstrates the capability of quickly-updatable working memory. Liang et al. [023) and Zhong et al. [2023] adopt memory to store the conversational history and handle extremely long contexts. Relational database is leveraged to track states in dynamic process by CBatDB Hu et al. 2023). J. Reflexion (Shinn et al. 2023] exploits a working memory to store experiences for dedicated task to improve the performance of the agent through several trials. However, as illustrated in Figure II the histories stored in working memory cannot benefit the episode for different task goals. Instead, a long-term cross-goal experience memory should be considered. MemPrompt [Madaan et al., 2022] and Ret-LLM (Modarressi et al. 2023) adopt a persistent memory to store human feedback and remind the chatbot of the conversational knowledge and improve it continuously. Voyager [Wang et al. [023a] designs a skill library to store the past learned skills as JavaSeript functions. A simple text experience pool is adopted by GITM (Zhu al. 2023 to store the successful trajectories for future referencing. Somewhat similar to GITM, REMEMBERER adopts persistent environmentt-roundee experience memory to store the experiences and assist in future decision-making even for different task goal. However, instead of plain text records of successful trajectories, REMEMBERER uses structured memory and designs a mechanism to task advantage of both successful and failed experiences. The experiences come from the interaction of the agent with the environment, and no human intervention is needed.
LLM learning from failure Learning from failure is one of the characteristic capabilities of human and turns to be an important topic for general artificial intelligence. Some work has explored the ability of the LLM to learn from its failure [Huang et al., 20226 Raman et al. 2022 Wang al.] 2023b]. Nonetheless, most of such work takes advantage of immediate feedback from the environment and the correction is used only once. In practice, several late failures may be due to some early mistaken actions in an episode. Reflexion [Shinn et al., 2023 designs a heuristic function to detect late failure from the interaction history and stores the LiM-genertted reflection in a working memory for use in the next trial. However, these reflections cannot be applied to different task goals. Madaan et al. [2022] stores the failure corrections for long term, but relies on human feedback. In contrast, REMEMBERER adopts RL to learn from both late and immediate failure from the environment rewards without need for human feedback. Also, REMEMBERER enables the experiences to be reused in the future episode even for a different task goal with a long-term experience memory.
LLM for decision-making
The powerful capability of LLM is exploited by recent work [Huang
ct al.
2022a
Raman et al. 2022 Mees et al. 2022 Chen et al. 2022 Ichter et al. 2022 Huang
ct al.
2022b Liang et al. 2022 to generate better control plans for various robots and agents. Kim
ct al.
2023 and Zhang et al. 2023 design LLM-based agents for user interface (UI) interaction.
ReAct Yao et al. 2022b) combines the action decision with natural language reasoning and achieves promising performance. To our best knowledge, This work is the first one that combines the LLM- based agent with RL algorithm to learn from the interaction experiences and achieve self-evolving.
The proposed REMEMBERER equips the LLM with an external experience memory to help it to learn from both successful and failed experiences. This is also the first work to combine the L.LM-based agent with RL algorithm to improve the capability of the agent.
3

---

REMEMBERER
Oj
2 OF
Ox Ar, Qu
dz
Environment
LLM
Experience
Memory
5
0i, 0t, Tz, 01+1
Figure 2: Pipeline of RLEM and architecture REMEMBERER
3 Method
3.1 RLEM pipeline
RLEM (Reinforcement Learning with Experience Memory) is proposed for an LLM-based agent to learn from its interaction experiences by updating external persistent memory. The pipeline of RLEM and the architecture of REMEMBERER agent are depicted Figure 2 REMEMBERER agent consists of two components: an LLM making decisions and an experience memory storing the interaction experiences. At the decision step, the LLM first takes an observation or from the environment. The observation o, is then adopted to retricve several related experiences from the connected experience memory according to some similarity functions. The experiences are represented group of observations O,, actions Ag, and the corresponding value estimations Qz- Here x denotes the index set of retrieved experiences and depends on the specific runtime observation 0N. Subsequently, LLM will decide the action a, in accordance with or, the feedback from the last interaction (e.g., the reward rr-1) ). as well as the retrieved experiences (O,,, Ay. Q..). a, will be executed in the environment and the resulted reward v, will be returned to the LLM the feedback. And the transition tuple, (Ot, a, Tt, 0t+1), comprising the last observation, the taken action, the corresponding reward, and the observation will be used to update the experience memory. The following subsections will detail the structure and updating policy of REMEMBERER experience memory and the usage of the retrieved experiences.
3.2 Experience memory of REMEMBERER
The experience memory is one of the pivotal components of the proposed REMEMBERER framework. It is adopted to store the interac- tion experiences, and the LLM is expected to benefit from the stored experiences in future decision-making The memory can be regarded as group of external parameters of the LLM- based agent. Such an agent semi- cami-paraeetric system that can evolve through RL process. Dur- ing the interaction, new experiences are added to the experience memory so that the overall sys- tem can attain a more capable interaction ability compared to the agents with just fixed LLM and fixed exemplars. This procedure can considered analogous to the training stage of conventional parametric agents.
Task & Obsv.
Action
Q Value
991, 01) aD
q1
(92, 02)
(22
Q2
(g3, 03) a3
q3
Figure 3: An example of the records stored in the proposed experience memory.
To be specific, the experience memory designed as table storing the task information, observation, action, and the corresponding Q value estimation. The Q value is the expectation of the accumulated future reward and gives an assessment of the value of the action candidate. Figure 3 depicts demonstration of the proposed experience memory. There are two stages to build practical REMEMBERER agent with experience memory: initialization and training. The experience memory is supposed to be first initialized with some initial records before the training stage. The initial

---

records are necessary to inform the LLM of the format of the input and the output. Then, during the analogical training stage, the agent interacts with the environment to collect new experiences, and conducts off off-policy learning [Sutton and Barto 1999]. Particularly, given the task information g and the new transition (Ot; at, rt, 0t+1), as a quadruple of the last observation, action, reward, and the
new observation, a new estimation is calculated first according to the estimated Bellman optimality equation [Bellman 1952 as
Q' Ot, ae) Ft 7 max Qgg: Or+1,
Here max can calculated from the actions already recorded for (g, 0t-1) by considering the Q value of unrecorded actions 0, if the action space cannot be traversed, e.g. action space involving free-form language. Then a new record inserted directly if there does not exist a record associated to (g. Oz, az) in the memory:
Q(g. or, ar) = Q' (g, or, at).
(2)
If (g, Ot, az) has been already inserted into the record, the recorded Q value estimation will be updated by Q-Learning [Watkins and Dayan 1992]:
Qgg, oc, 0t) + (1 a)Q Ot, ai) + aQ' o..
Here the learning rate, o, is 1/N where N denotes the times this value is updated. As Equation 1 may lead to an inaccurate estimation owing to insufficient sampling out of few training steps of REMEMBERER, n-step bootstrapping [Mnih et al. 2016 adopted ameliorate this problem, which estimates Q' by
R-1
Q' a.) =
- T'44 + y" max Q(g. Ottn, a),
where n is the steps to expand. The ablation study in Subsection 4.4 proves this perspective.
3.3 Usage of the experiences
In order to assist the LLM in making deci- sions, the stored experiences are adopted as dy- namic exemplars for few-shot in-context learn- ing. Given the task goal and the current ob- servation ot, similarity function f is used to calculate the similarity of (g, o.) with (gi, from the memory.
Last 5 Actions:
search(3 bottle bright cftrus deodorant
sansitive skin]
Observation:
Instruction:
would like ounce bottle
[Back to Search]
S: = s((g, ot). (9i,
(5)
[8878GNRC13]
[8678GTWVXY]
Commonly, similarity function f can be di- vided into two components, task similarity fa and observation similarity fo:
[в65KBV24XK]
Available Actions: back search
S, Af 9:) (1 A)fo(oe,0)) oz).
(6)
The m records with the highest similarities are retrieved to form the exemplars in the prompt. The particular similarity function designed for each task set detailed in Subsection 4.1
Encouraged:
b979gwrc1j beragtlivay are bright citrus deodorant less than dollars. check b078gure1j first.
D Discouraged:
The exemplar is supposed to demonstrate the format of the input and the output to the LLM. The input part usually comprises the task infor- mation and the observation, along with some interaction feedback or auxiliary information. The particular input format depends on the task domain and will be detailed in Subsection 4.1 The output part indicates the action decision. Specifically, we propose to present the action decisions in form of "action advice" comprising both encouraged and discouraged actions rather than simply present an action to execute. This is motivated by the perspective that "reasoning is
becc((b88 >> 0.0 basTwkor2g is not
destrad
Figure 4: An exemplar for WebShop task set (Yao et al. 2022a]. The input part is depicted in the upper box and the output part is depicted in the lower box. Action candidates are advised along with their Q value estimations and some optional extra information.
5

---

remembering" to exploit both successful and failed experiences. To form the output part in the exemplar, the actions with the highest Q value estimations from the retrieved record are given as the encouraged actions, while the actions with poor Q value estimations (e.g., zero or negative estimations) are given as the discouraged actions. expectations can lead the L.M to follow the past success, while the advice with poor expectations will teach the LLM to avoid a similar failure. A clear depiction of the exemplar format can be found in Figure ! Prompted by such exemplars, the LLM will also predict both encouraged and discouraged actions and speculate their Q values given new input. The predicted Q values are used to select the optimal action, to be specific, the encouraged action with the highest Q value speculation will be executed in the environment.
is believed that the advice with high value
It is worth noting that REMEMBERER agent necessitates only limited number of training steps to achieve a promising performance, which leads to a non-exhaustive action record within its memory. Consequently, instances may arise where there is only one action associated with a given context (g. ot), or the highest Q value remains deficient, or no sufficiently unfavorable action exists to discourage. In such cases, randomly sampled action advice is favored over encouraging an action with low expectations or discouraging an action with moderate expectations. Our ablation study in Subsection 4.4 sheds light on various strategies for generating advice in such scenarios.
4 Experiments & results
4.1 Experiment setup & implementation details
To assess the effectiivnness of REMEMBERER, we evaluate it on two recent task sets with the promising performance of LLM-based agents: WebShop and WikiHow. All the experiments are conducted based on the OpenAI API of GPT-3.5 [Brown et al. 2020 text-davinci-O00
WebShop Yao et al., 2022a] WebShop is a task set simulating a web store site. The agent is instructed to browse the site and shop for the target goods. The information of over 1M products is crawled from the Amazon stord About 12K product requests are re-written by crowd laborers to generate more diverse instructions. A score between 0 and 1 will be rated after shopping by assessing the correspondence between the product and the instruction. We followed [Shinn et al. (2023) and the first 100 tasks from the same shuffled task list released along with
conducted our experiments the task set. At each interaction step, the LLM takes the web page representation and a list of available actions input. The task instruction is omitted, for there is always instruction present on the top of the web page. As there are no intermediate rewards during the episode, only the last 5 performed actions serve as procedure feedhack. Inspired by the chain-of-thouge technique (Wei al. 2022 and the ReAct mechanism Yao et al.) 2022b), the LLM is prompted to predict a reason for its decision as the extra information depicted Figure 4 The representation of the web pages is simplified in the same way as ReAct. The task similarity fg is calculated using the MIl-MiniLMM-LI2-VV model from Transfforrrssssrrrs [Reimers and Gurevych, 2019]. As it is noticed that the web pages in WebShop are instantiated from some templates, we categorize the web pages into four patterns and design a similarity lookup table to compute the observation similarity fo according to the web page patterns. The details about the similarity table should be referred to in the supplementary. It is observed that most of the tasks end in 5 steps, thus we directly conduct full-trajectory expanding while performing multi-step bootstrapping:
Q'(or, 0c) = Σ 'r.
WikiHow is a task set based on the collaborative wiki app WikiHow rumning on the interaction platform Mobile-Env (Zhang et al. 2023). The task set contains amounts of navigation tasks. The target of the agent is to follow the instructions and navigate to the required page. Intermediate rewards and instructions may be triggered during the episode. We followed Zhang et al. (023) and evaluated the proposed REMEMBERER on the "canonical subset" comprising 70
WikiHow [Zhang et al. 2023
https:
httpa:// http//wwwaaaa.n..o/////// com/
https://www http://wwwwwkkii.e wikihow. www.:/coo//anniince Page

---

Table 1: Results on WebShop. The result of the prior state of the art, ReAct (ao et al. 2022b], is attained with the public implementation released by the original authors. The RI., II., and R..r- re-
Table 2: Results on WikiHow. "Mobile-Ene in-
dicates the prior result from Zhang et al. (2023). "RMMBR. (A)" denotes the results by directly running the evaluation of REMEMBERER with
sults are retrieved directly from Yao et al. 2022a
experience memory.
Method
Avg Score
Success Rate
Method
Avg Reward
Success Rate
LLM only
0.55
0.29
ReAct
0.66
0.36
LLM only
2.58
0.90
RMMBR.
0.68
0.39
Mobile-Env
2.50
0.89
RMMBR.
2.63
0.93
RL
0.55
0.18
IL
0.60
0.29
RMMBR. (A)
2.56
0.91
IL+RL
0.62
0.29
Table 3: Results on WehShop with different exemplar combinations (initial experiences for RE- MEMBERER) and different training sets (for REMEMBERER). E3 denotes the different exemplar combinations, while S, denotes the different training sets. The first line of each method shows the mean scores, and the second line shows the success rates.
Different (Initial) Exemplars
Different Training Sets
Eg So
E1 So
E2 + So
Eg S1
Avg
Std
0.72
0.65
0.60
0.66
0.06
ReAct
0.42
0.35
0.30
0.36
0.06
0.52
0.54
0.59
0.55
0.04
LLM only
0.26
0.28
0.32
0.29
0.03
0.66
0.71
0.66
0.67
0.68
0.02
RMMBR.
0.37
0.41
0.37
0.40
0.39
0.02
tasks. Specifically, the LLM is input with the task description, the screen representation, and the step instruction. The screen is represented in HTML element sequence following /Zhang al. [2033 Additionally, the last 5 performed actions along with the last reward are given to the I.I.M as the procedure feedback. As for the output, the LLM is prompted print the HTML representation of the operated element as the extra information. This is expected to force the LLM to discover the relation between the element id and the certain element. The task similarity f, designed for WikiHow is computed from the step instructions. It is noticed that the instructions follow some patterns, thus, we inspect the instructions and categorize them into six types. Then similarity lookup table is designed according to the instruction types. The details should be referred to in the supplementary. The observation similarity fo computed based on the length of the longest common sequence of the HTML elements in the screen representation:
les(sc1,800 fo(sc,, 502) max (lem(sa) 1.) (802))
(8)
The full-trajectory expanding is adopted, as most of the tasks will end 5 steps as well.
4.2 Results on WebShop
REMEMBERER is applied to WebShop with 2-shot in-context learning. The experience memory initialized with four annotated experiences of the decision step from one trajectory. The agent is trained for 3 epochs on training set containing 10 different tasks outside the test sets used by Yao al. [022b) and [Shinn et al. [223). To control the total expense and achieve bootstrapping. the succeeded tasks in the first epoch are excluded from training in the following two epochs. The trajectories exceeding 15 steps are considered to failed, as most of tasks can end in 5 steps.
The main results are shown in Table [ We used the public ReAct (aoo et al. (022b) implementation
released by the authors and run with 1ext-davinci-003 instead of text-davince- in Yao al.) [2022b]. The run of ReAct shares the same trajectory the exemplar with REMEMBERER. The "LLM only"
7

---

Table 5: Comparison the number of the tasks in the training set and the updating steps of RE- MEMBERER with the II. and RI. baselines. The number of the updating steps of II. is estimated from epochs on 1,012 trajectories with an av- erage trajectory length of 11.3.
Table 4: Comparison of the number of annotated trajectories and steps of REMEMBERER and the IL baseline. The number of steps of the training set of II. is estimated according to the average human trajectory length the test split as 1 11.3 in Yao et al. [202aa].
Method
#Tasks
#Steps
Method
# #Trajectories
#Steps
RL
10,587
100,000
IL REMEMBERER
1,012
-11, ~11,436
IL
-114,356
4
REMEMBERER
10
74
Table 6: Results on WikiHow with different exemplar combinations (initial experiences for REMEM-
BERER) and different training sets (for REMEMBERER).
Different (Initial) Exemplars
Different Training Sets
Eg So
E1 + Sp
E2 + Su
Eg + S1
Avg
Std
2.56
2.60
2.59
2.58
0.02
LLM only
0.90
0.90
0.89
0.90
0.01
2.63
2.63
2.59
2.66
2.63
0.03
RMMBR.
0.93
0.91
0.90
0.97
0.93
0.03
baseline indicates a single LLM with 2 fixed exemplars sampled from the initial experiences of REMEMBERER. The average performance of REMEMBERER exceeds the baseline by large extent and surpasses the prior state of the art, ReAct, as well. This proves the effectiveness of augmenting the LLM with external evolvable experience memory. The proposed REMEMBERER also outperforms the RI,, II (imitation learning). and IRI.I baselines on both metrics.
In order to verify the robustness of REMEMBERER, experiments with different initial experience combinations different training set are conducted. The results are depicted in Table The initial experience combination En denotes the certain trajectory adopted by the original implementation of ReAct while E, and Eg are randomly sampled from Sg. It is observed that the proposed RE- MEMBERER can achieve better and more stable results with different initialization and training sets compared to ReAct. Thus, REMEMBERER can mitigate the workload to some extent to search for an optimal exemplar combination.
We compare the training efficiency of REMEMBERER with the conventional IL and RL methods in Iable Ta In contrast to the II., REMEMBERER requires quite few annotated samples to ini- tialize the experience memory, while agent can be trained on only 10 tasks for 74 steps, while the RL and IL are expected to be trained for about 100 thousand steps to achieve an acceptable performance. Consequently, the proposed REMEMBERER offers a much more efficient way to build practical agent agilely.
in need of much more human annotations. REMEMBERER
4.3 Results on WikiHow
REMEMBERER is applied to WikiHow with 2-shot in-context learning. The experience memory is initialized with two annotated experiences of the decision step. The agent is trained for 3 epochs on training set containing 10 different tasks selected from WikiHow excluding the test tasks. Similar to the experiments on WebShop. the succeeded tasks in the first epoch are excluded from training in the following two epochs. As observed that most of the tasks require interaction of less than steps. the trajectory exceeding 15 steps will be regarded as failed. The main results are depicted in T2ble The exemplars of the "LLM only" baseline are the initial experiences of REMEMBERER, The proposed REMEMBERER surpasses the baseline as well as the original result in Zhang al. 2023 In addition, 10 tasks are annotated to form an annotated experience memory. REMEMBERER agent with this annotated experience memory is evaluated without further training, and the result denoted as "RMMBR. (A)" in the table. This result demonstrates that REMEMBERER is capable of

---

Table 7: Comparison of the average reward estimation of the full model and the ablation model without bootstrapping policy. The error is the absolute difference between the average reward estimation from the experience memory and the real training reward.
Task Set
Setting
Avg Reward Estimation
Avg Training Reward
Abs Error
Full Model
0.86
0.84
0.02
WebShop
w/o bootstrp.
0.62
0.84
0.22
Full Model
2.48
2.60
0.12
WikiHow
w/o bootstrp.
1.98
2.70
0.72
Table 8: Results of ablation study
Task Set
Setting
Avg Reward/Score
Success Rate
Full Model
0.66
0.37
WebShop
w/o bootstrp.
0.67
0.36
w/o random
0.65
0.37
Full Model
2.63
0.93
w/o bootstrp.
2.54
0.89
w/o random w/o discouraged
2.64
0.90
WikiHow
2.48
0.81
w/o task sim. fg w/o obsrv. sim. fo
2.63
0.94
2.47
0.87
exploiting expert experiences, which can be regarded as analogous conventional imitation learning. Nevertheless, the annotated experiences may not offset the exact shortage of the particular LLM. In contrast, the RL training will have an opportunity to collect more specific experiences and gain more promising performance.
The experiments with different initial experience combinations different training conducted on WikiHow as well, and the results are shown in Table The proposed REMEMBERER achieves consistent improvement compared the baseline with fixed exemplars, which proves the effectiveness and robustness of REMEMBERER.
4.4 Ablation study
Several ablation studies are conducted to verify the design of REMEMBERER framework.
Ablation on n-step bootstrapping policy Ablation studies are conducted verify the necessity of n-step bootstrapping policy to update the Q value estimations in the experience memory. As stated in Subsection 3.2] updating without bootstrapping may lead inaccurate value estimations owing to few training steps to explore and exploit. In order to verify this perspective, an average reward estimation calculated by averaging the sum of the maximum Q value and the history reward stored for each observation in the experience memory:
1 M Σ
R
(Ralgn oz) + max Q(g,, a),),
(9)
where R2 denotes the total reward of the steps before (gi, 0z) on the trajectory and M is the size of the memory. The deduced average reward estimation compared the real training reward, and an absolute error is calculated in 7 It can be observed that the reward estimation from experience memory trained without bootstrapping suffers a far greater error than that with bootstrapping. Meanwhile, the performance on the test set is demonstrated in Table 8 Although there no apparent disparity in the final performance on the WebShop task set, a visible degradation observed WikiHow, which reveals the latent risk of non-bootstrapping update.

---

Ablation on the advice generation strategy As stated in Subsection 3.3 owing to the non- exhaustive exploration the brief training stage, there may be no suitable candidates for the action advice in the exemplars. For instance, there may be no actions recorded with poor enough Q value estimation or no actions recorded as high-reward. Under this case, action advice can be generated with randomly sampled action that is not in the record, or it can be given by directly encouraging the action with the highest Q value estimation and discouraging the action with the lowest estimation without regard to the certain value. These two strategies are compared in Table g As the results illustrate, the random plan appears to have a minor superiority over the non-random plan. This is attributed to that advice with improper value expectations will mislead the LLM to take wrong judgments about the true value of the available actions.
Additional experiments are conducted to investigate the necessity of the discourage actions in the output part of exemplars and the impact of similarity function components. Owing to limit of budgets, these experiments are only conducted on WikiHow task set.
Ablation on necessity of the discouraged actions The proposed output format "action advice" comprises both encouraged and discouraged actions. The discouraged actions are believed to help the L.M to avoid similar failures. Results in Table [ prove necessity of the discouraged actions. Without access the discouraged actions, the agent can only achieve much poorer performance than the full model. In the case shown in the supplementary, it can be seen that there may not be proper actions to encourage in the retrieved experience. In such cases, the discouraged actions are especially crucial for the agent to prevent repeating similar mistakes.
Ablation on the similarity function As stated in Subsection 3.3 similarity function is required to select related experiences from the memory. In experiments, the similarity is implemented as two components: task similarity fa and observation similarity fo. Ablation studies are conducted to draw brief perspective on the impact of these two components. As shown in Table g removal of task similarity seems not to affect the performance remarkably, while removal of observation similarity causes serious degradation. This may indicate that on these tasks, the tested LLM benefits more from experiences that have similar observations rather than similar instruction patterns. On the other side, the pattern- task similarity for WikiHow introduced in Subsection 4. may be too coarse to cluster the experiences. During interaction, the agent may receive instructions of the same pattern (e.g.. "access article ABC") while facing different types of observation (e.g., search result page or category page). The appropriate actions in two situations are also different. Removal of observation similarity will eliminate this difference in experience selection and results in misleading. Case study in the supplementary shows this perspective.
5 Conclusion
We introduce Reinforcement Learning with Experience Memory (RLEM) to aid the LLM in learning from its interaction experiences for decision-making tasks. A novel LLM-based agent framework called REMEMBERER is then designed with RLEM by equipping the LLM with a persistent experi- ence memory and updating the memory with the RL algorithm. REMEMBERER agent is capable of exploiting the interaction experiences to improve its policy and gains significant improvement com- pared to the baseline. Our experimental results demonstrate the superiority. Owing to the simplicity and effectiveness of REMEMBERER, we believe that this work provides a valuable perspective on designing evolvable LLM-based agents with RLEM.
Limitations
The proposed REMEMBERER agent demonstrates strong superiority on the tested benchmarks. Nevertheless, it is wondered how this framework will be applied to the environments with more long-term episodes or with more extensive or visual-rich observations. Besides, is observed that the performance of REMEMBERER will encounter quick saturation in training process. This may be due to the limited number of active exemplars. Further efforts are expected to be dedicated in to make the agent performance evolve continuously. Furthermore, as early exploration, we didn't 't make use of complicated RI. techniques. How recent advancement in RI. domain works under RLEM also an interesting problem.
10

---

Acknowledgements
This work is funded by the China NSFC Project (No.62106142 and No.6212010600 and Shanghai Municipal Science and Technology Major Project (2021SHZDDZ00002),
References
Richard Bellman. On the theory of dynamic programming. Proceedings of the national Academy of Sciences, 38(8)7116-71, 1952.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models few-shot learners. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: An- mual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedinngg www.eediiiinnnn.nss.....m wtt.cccppppp/////0/00// www.:/4444--sttttat....... html
Boyuan Chen, Fei Xia, Brian Ichter, Kanishka Rao, Keerthana Gopalakrishnan, Michael Ryoo, Austin Stone, and Daniel Kappler. Open-vocabulary queryable scene representations for real world planning. CoRK, abs/2209 009.222999099744. 2022. doi: 44455500arXXi222290097777 URL httpss
7/doi. ordoi.or//ii 10. 48550/ 4.8500/ariiv 2209. 2
Chenxu Hu, Jie Fu, Chenzhuang Du, Simian Luo, Junbo Zhao, and Hang Zhao Chatdb: Augmenting Ilms with databases their symbolic memory. CoRR, abx/2306.0390 03901, 2023, doi: 10.48550/arXiv. 2306.03901. (RLLhhts:://do org/ 10. 48550/ axxiv 2306. 13806.033 03901
Wenlong Huang, Picter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero- shot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato, editors, International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Muchine Learning Reseurch, pages 9118-9147. PMLR, 2022a. URL https:tp::o http:::////ooceeegg..oemm mlr. press/ sitesssvv622// v122/huang222 html
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Tomas Jackson, Noah Brown, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner monologue: Embodied reasoning through planning with language models. In Karen Liu, Dana Kulic, and Jeffrey Ichnowski, editors, Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Research, pages 1769-1782. PMLR, 2022b. URL https://proceecingg.. mlr, press mansss//20/huuag2255 html
Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahn, Omar Cortes, Nicolas Sievers, Clayton Tan, Sichun Xu, Diego Reyes, Jarek Rettinghouse, Jornell Quiambao, Peter Pastor, Linda Luu, Kuang-Hus Huei Lee, Yuheng Kuang, Sally Jesmonth, Nikhil J. Joshi, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu. Do I can, not I say: Grounding language in robotic affordances. In Karen Liu, Dana Kulic, and Jeffrey Ichnowski, editors, Conference on Robot Learning, CoRL. 2022, 14-18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Research, pages 287-318. PMLR, 2022. /ttpstt/:///proeeediii v205/ichter23a. html
mlr mlr.pre .pees//
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. CoRR, abs/2303. 17491, 2023. doi: 10.448550/axxx.2203.174444 URL https:: //d::.org////000/0//// 48550/ www....ww..........aa 17491

---

Takeshi Kojima,
Gu. Machel Reid. Yutaka Matsuo, and Yusuke Large language models are zero-shot reasoners. In NeurIPS.
Shixiang Shane
Iwasawa.
copppprr..ppppp.....mmpp.p URL http: http://papers.... nips cc/ paper__ files/ wwwes/paper/eee/ 1eaper/2022/haae 1 oat.opa....c.lrotooooooa.aaaaaar htt.geoceeecoooooaaaacooooorrree
2022.
Michael Laskin, Luyu Wang. Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steigerwald, DJ Strouse, Steven Hansen, Angelos Filos, Ethan A. Brooks, Maxime Gazeau, Himanshu Sahni, Satinder Singh, and Volodymyr Mnih. In-context reinforcement learning with algorithm distillation. CoRR, abs/2210. 14215, 2022. doi: (0448500aa2ii.22210.1222 URL https::/doi..nnn 48550/ arxiv. 2210 14215
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. Code as policies: Language model programs for embodied control. CoRR. 0ttp2209.077533 2022. doi: (0.485500a2xi..220007755 URL /Rt//tt::///oi org/10 48550/ arXiv 20l9.2209.000 07753
Xinnian Liang, Bing Wang, Hui Huang, Shuangzhi Wu, Peihao Wu, Lu Lu, Zejun Ma, and Zhoujun Li. Unleashing infinite-length input capacity for large-scale language models with self-controlled memory system. CoRR, abs/2304. 13343, 2023. doi: 10.48550/r2xx1:334443333 URL https
//doi. 10. 405548550//0 20550//rXXi.. arxiv. 2304 13343
Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. Memory-assisted prompt editing to improve gpt-3 after deployment. arXiv preprint vrXiv:2201.06009 2022.
Oier Mees, Jessica Borja-Diaz, and Wolfram Burgard. Grounding language with visual affordances over unstructured data. CoRR, abs/2210.01911, 2022. doi: 10.88550arX...20000111. URL
httpps://oi. org/ 10. 48550/ arxiv. 2210. 01911
Volodymyr Mnih, Adrià Puigdomènech Badia, Mchdi Mirza, Alex Graves, Timothy P. Lillicrap. Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In Maria-Florina Balean and Kilian Q. Weinberger, editors, Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1928-1937. JMLR.org. 2016.
mlr. pwwps///8/mmm/i w4t///ihammmmnmmmmm hhmmllmm66m...mm...mmmmmm
Ali Modarressi, Ayyoob Imani, Mohsen Fayyaz, and Hinrich Schütze. RET-LLM: towards general read-write memory for large language models. CoRR, 10w./22055.1332... 2023, doi: 10.48550/arXiv. 2305. URL https:://toiii..o org/ 10. 48550/555/// arriv. 2305 14322
Xiaoman Pan, Wenlin Yao, Hongming Zhang. Dian Yu, Dong Yu, and Jianshu Chen. Knowledge-in context: Towards knowledgeable parammetiii language models. CoRR, abs/2210.16433 2022. URL htttps:///o...... org/ 10. 48550/ arXiv. 2210. 16433
doi: 10.48550//rxx1.2210..655
Baolin Peng, Michel Galley, Pengeheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. Check your facts and try again: Improving large language models with external knowledge and automated feedback. CORR, abs/2302. 12813, 2023. doi: 10.48550/arXiv.230222 12813. URL htttts:////tt www.../..00///aa.......ooo/ -8850//arxiv..... 2302. 12813
Shreyas Sundara Raman, Vanya Cohen, Eric Rosen, Ifrah Idrees, David Paulius, and Stefanie Tellex. Planning with large language models via corrective re-prompiing. CoRR, abs/2211 1. .09935, 2022. doi: 10.48550/arxiv.222 21.2222111919..1 1.09935. URL httpth:tttpsoooo. org/ 48050/8850//r/// arwiv. 2211 .09935
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong. China, November 3-7, 2019, pages 3980-3990. Association for Computational Linguistics, 2019. doi: 10.18653/v1/D19-11 1410. URL http::ttp////. //doi 10t.///110.663///1/100 1410
Timo Schick, Jane Dwivedi-YV Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. CORR, abs/2302.04761, 2023. doi: 10.48850/axxxv22302.47666 URL httpps
//doi org. 101 48550/ arxiv. www.xv..v00..... 2302.04761
12

---

Dale Schuurmans. Memory augmented large language models are computationally universal. CoRR. abs/2301 .04589, 2023. doi: 10.48550/arX3..330.12 arXivv. 2301 04589
|tttpss/://// org.//o...r////. 48550/
Colleen M Seifert, Andrea L Patalano, Kristian Hammond, and Timothy M Converse. Experience and expertise: The role memory in planning for opportunities. 1997.
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. CoRR, abs/2303.11366, 2023. doi: 10.48550k/aX1:2238.111666 URL
htttp:://di..rrg 10. 48550/ arXiv 2303. 11366
Thomas Suddendorf and Michael C Corballis. The evolution of foresight: What is mental time travel, and is it unique humans? Behavioral and brain sciences, 30(3):299-313, 2007.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. Robotica, 17(2): 229-235. 1999.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving re- trieval with tfain-of-thought reasoning for knowledge-intensive multi-step questions. CoRR. abs/2212.10500 2022. doi: 10448550/aXXx.2212.100999 URL https://doi. 0rg/0..88500/ 22Xiv.221..1555
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. CoRR, abs/2305.1622) 2023a. doi: 10.483550/22xi..2355.1 16291. https://://ddo arXiv.2305. 2305..6999111
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, and Denny Zhou. Self- consistency improves chain of thought reasoning in language models. CoRR, abs/2203. 11171, 2022. doi: 10.48550/arxiv.2203.. 11171. URL. httpt:ttt//:....... org/ 48550...55550.... arrvv. www.:rwwwii........mmmmmm 11171
Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. Describe, explain, plan and select: Interactive planning with large language models enables open- world multi-task agents. CoRR, abs/2302.01566 2023b. doi: 00.48550/rxxiw22322...553
48850/
arXii arXiv.2302.cm 01560
Christopher JCH Watkins and Peter Dayan. Q-learning. Machine learning, 8:279-292, 1992.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou, Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201. 1903, 2022. URL URL /ttt:ttts:/:/xxiitt org/ abs /2001 11903
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Wehshop: Towards scalable real-world web interaction with grounded language agents. CoRR, http/2277.01206 2022a. doi: 10.48550/arXiv.2207.02222
(RRLhhtts://dol/ http://60...000/10044466/////// 22...0022222
488500/8rXii
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. CoRR. abs/2210.03629, 2022b. doi: 100445500/axXiv2221003222
httphhts::///o...
10. 48550/4555a1i 2210. 03629
Danyang Zhang, Lu Chen, and Kai Yu. Mobile-Env: A universal platform for training and evaluation of mobile interaction. CoRR, abs/2305.08144, 2023. URL. httpp://arriiv.org/bbbl 2305
08144
Wanjun Zhong, Lianghong Guo, Qigi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. CoRR, abs/2305. 2023. doi: 10.48550/arXiv. 2305.102500 URI. httph:tt:://// //doi. org/10..e.00 48550/arXix 2305 10250
Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zhang, and Jifeng Dai. Ghost the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory. CoRR, abw/2305.11444, 2023. doi: 00.44555//xxi.22355.... 1714.. URL htttps://doi.rgg org/10 arg//0.48550/. 48550/ arxiv www.aa.ww..........a.a 17144
13


Reviews:
[{"review_id": "i50rcX4Jd2", "reviewer": "Reviewer_2hwN", "summary": "The paper proposes REMEMBERER, a novel framework for Large Language Models (LLMs) that employs a persistent experience memory and a Reinforcement Learning with Experience Memory (RLEM) mechanism. This setup aims to enable LLMs to learn from previous interaction experiences in decision-making tasks, improving their policies. The experience memory acts as an external repository, storing experiences from past episodes, which the LLM can exploit without the need for fine-tuning its parameters. The framework, tested on two recent Reinforcement Learning (RL) task sets, demonstrated an improvement in performance over previous state-of-the-art (SOTA) models.\n", "questions": "What is x in the subscript O_x in the methods section? \n", "limitations": "In the title, I think it should be “Large Language Models are Semi-Parametric Reinforcement Learning Agents”\n\n“Such an agent turns to be a semi-parametric system that can evolve through RL process” -> “Such an agent is a semi-parametric system that can evolve through an RL process”\n\nOverall, I think the paper can be cleaned up a lot with better English and notation.  ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper's primary strength lies in its innovative approach to RL using LLMs. The authors introduce an agent framework, REMEMBERER, that overcomes the typical limitations of existing approaches, such as the significant cost of fine-tuning LLMs. The proposal of RLEM, which updates the experience memory through RL training, enables the system to evolve its abilities in an efficient manner. Moreover, the empirical results validate the effectiveness of REMEMBERER, with it outperforming the SOTA models on two RL tasks. This paper is in line with a large number of recent papers that show that LLMs can be used to create agents. \n", "weaknesses": "While the results are encouraging, the paper doesn't delve deep enough into the impact of different configurations of the REMEMBERER system. It would be beneficial to explore how varying the size and management of the experience memory as well as the similarity functions affects system performance. In addition, the paper focuses on improvement against SOTA models but doesn't provide a comparative analysis against other similar approaches that utilize memory or experience in RL, such as the very recent Voyager paper (which is too recent to compare against, but an example of these types of papers). A weakness of this approach is that it requires using a reward function, when reward functions might be unavailable or hard to create in difficult computer tasks. In comparison to other RL algorithms, the approach seems constrained to text domains. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable review and advice.\n\n**Impact of memory size**: In practice, we didn't limit the capacity of the\nexperience memory, hence it can accommodate as many experiences as the hardware\nmemory allows. To have a perspective on the impact of actual memory size, we\nrefer you to the experiments in Sec. 4 in the supplementary. Fig. 3 in the\nsupplementary and Fig. 1 in the rebuttal PDF demonstrate the number of new\nexperiences and total experiences in the memory after each training epoch,\nrespectively. The results in Fig. 2 from the supplementary shows that the\nperformance improves while the memory size increases, and there is a\nsaturation. This saturation may be attributed to the limited number of active\nexemplars in the input prompt.\n\n**Impact of similarity function**:  We didn't include analysis of similarity\nfunction, as we do not regard this component as the core of the proposed\nRememberer framework. We mainly focus on building a general framework assisting\nLLM in exploiting its experience. The core is the experience memory and\nRL-based updating approach. The similarity function, in contrast, depends on\nthe specific task domain. However, we agree that the selection of similarity\nfunction may conduct obvious impact on the final performance. We add ablation\nstudy according to your advice on WikiHow. The results are shown below.\n\n|Setting|Avg Reward|Success Rate|\n|:--|:-:|:-:|\n|Full Model|2.63|0.93|\n|w/o Task Sim. $f_g$|2.63|0.94|\n|w/o Obsrv. Sim. $f_o$|2.47|0.87|\n\nIt is noticed that there are not significant difference between performances of\nthe full model and the ablation model without task similarity, while the\nperformance degrades if *observation similarity* is removed. This may indicate\nthat on these tasks, GPT model benefits more from experiences that have similar\nobservations rather than similar instruction patterns. (The task similarity for\nWikiHow is implemented according to the instruction pattern and may be too\ncoarse to distinguish different experiences.) In the results, we find that\nwithout observation similarity, Rememberer fails to adjust the retrieved\nexperiences according to the specific observation. *E.g.*, similar instructions to access an article may be emitted\nwhen the agent is on a search result page or a category\ndetail page. The actions to be taken are different according to underlying\npages the agent is on. In such cases, inappropriate experiences cannot bring\nvaluable enough guidance, which leads to a poorer performance. As a conclusion,\nit is worth seeking more appropriate and more effective similarity functions\nfor specific task domain. We will update our paper according to the new results\nand perspectives.\n\n**Comparison with other LLM-based agents utilizing memory or experience**: We\nhave also noticed that a group of related work has been released **after** the\nsubmission of NeurIPS (*e.g.*, Voyager [1] and GITM [2] at May 25, and ChatDB\n[3] at Jun 7). Voyager proposes to use codes as actions, and designs a skill\nlibrary to store the successful programs as skills.  GITM stores the past\nsuccessful action sequences in a text memory to assist an LLM planner in future\nplanning without further discriminating their values.  Then the stored\nexperiences are summarized by LLM to gain deeper insights into planning policy.\nChatDB leverages relational database to track states in a dynamic process. In\ncontrast, we store the experiences in a structured memory and learns their $Q$\nvalues to filters out the more valuable actions. In this way, we can combine RL\nwith LLM and design a semi-parametric RL agent.  Meanwhile, some work dedicated\nto long-term or knowledge-augmented conversational tasks also combines LLM with\nmemory (*e.g.*, MemoryBank [4], and Ret-LLM [5]). Here, the external memory is\nusually used to extend the context length the LLM can perceive. We are glad to\ninclude these comparisons in Related Work in future revision.\n\nAs the methods like Voyager and GITM are designed for the specific open-world\nenvironment MineDojo, it will take some effort to migrate them to our test\nbenches and will be difficult for us to introduce new experimental results\nduring rebuttal. Nonetheless, we find a number of valuable inspirations from\nthese work, *e.g.*, automatic curriculum & code as action from Voyager, and\nautomatic goal decomposition & experience summarization from GITM. We are\nworking to benefit from these ideas in our future work.\n\n**Generalizability**: We refer you to global reply 2 for this question.\n\n**Symbol $O_x$**: $x$ denotes the index set of the experiences retrieved from\nthe memory. We use symbol $x$ because the index set depends on the certain\nstate met during interaction and cannot be determined in advance and is deemed\n\"unknown\". We will update the notations and supplement explanations in later\nrevision.\n\n**About advice in \"Limitations\"**: Thanks for your kind opinion. We will update\nthe title and expressions in later revision according to your advice.\n\n* [1] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke\n  Zhu, Linxi Fan, Anima Anandkumar. Voyager: An Open-Ended Embodied Agent with\n  Large Language Models. arXiv:2305.16291.\n* [2] Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang,\n  Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zhang, Jifeng\n  Dai. Ghost in the Minecraft: Generally Capable Agents for Open-World\n  Environments via Large Language Models with Text-based Knowledge and Memory.\n  arXiv:2305.17144.\n* [3] Chenxu Hu, Jie Fu, Chenzhuang Du, Simian Luo, Junbo Zhao, Hang Zhao.\n  ChatDB: Augmenting LLMs with Databases as Their Symbolic Memory.\n  arXiv:2306.03901.\n* [4] Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, Yanlin Wang. MemoryBank:\n  Enhancing Large Language Models with Long-Term Memory. arXiv:2305.10250.\n* [5] Ali Modarressi, Ayyoob Imani, Mohsen Fayyaz, Hinrich Schütze. RET-LLM:\n  Towards a General Read-Write Memory for Large Language Models.\n  arXiv:2023.14322."}, {"review_id": "93GG0ctC47", "reviewer": "Reviewer_8w3X", "summary": "This paper introduces an interesting approach that harnesses the capabilities of large language models (LMs) to tackle reinforcement learning (RL) problems. The method involves estimating Q-functions using RL algorithms and providing advice to the LMs about actions with high and low Q-values. The expectation is that LMs will utilize their \"remembering\" abilities to effectively solve the problem. Empirical evaluation on two language-based reinforcement learning tasks demonstrate the enhanced performance achieved by the proposed method.", "questions": "1. Please respond to the questions raised in the weakness section.\n\n2. What is the architecture of the Q function and what is the reinforcement learning algorithm for learning the Q function?\n\n3. An ablation study where the LLM does not have the access to the discouraged actions is expected.\n\n4. Another critical issue, unrelated to the academic content of the paper, requires attention. It has been brought to attention that the usage of OpenAI's brand in figures, if without proper permission, is highly inappropriate and may constitute a violation.", "limitations": "The authors didn't discuss the limitations of the proposed method. The conclusion section summarizes the contribution of the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of combining a fixed large language model and a reinforcement learning component estimating action-value functions by presenting experience and advice in prompts is, as far as the reviewer is concerned, interesting.\n\n2. The paper exhibits a well-structured organization, featuring clear and coherent writing that is easily comprehensible. Additionally, the experiments provide compelling evidence of the advantages offered by the proposed method.", "weaknesses": "1. An essential assumption underlying the proposed method is that large language models (LMs) possess the capacity to \"reason\" by effectively recalling all instances of success and failure. Hence, it is crucial to engage in a discussion regarding the limitations of this assumption to establish a comprehensive understanding. Questions arise about the case where the observation space is extensive, and if LMs implicitly learn certain types of Q-functions. While the proposed method is simple yet seemingly effective, delving deeper into the understanding of simple methods often yields intriguing insights.\n\n2. The ablation studies are not sufficient, which would provide valuable insights and address specific questions.\n\n3. The applicability of the proposed method to tasks with observations in natural language is evident. However, it remains unclear how the LMs can be adapted to tackle problems involving image or vector-based observations. Further elucidation on this aspect is necessary to ensure a comprehensive understanding of the proposed method's adaptability and versatility across different observation types.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable review and advice.\n\n**About extensive observation space**: Extensive observation space may result\nin a much larger experience memory, which may require more scalable and more\nefficient approaches to store the experiences.  Meanwhile, we refer you to the\nexperiments in Sec. 4 in the supplementary.  As shown in the results, the\nperformance will improve along with increment of the number of experiences and\nthere is a saturation, which may be due to the limited number of exemplars. But\nin a more diverse observation space, the saturation may come later. Besides,\nexperience merging, observation shrinking (like in Synapse [1]), and forgetting\nmechanism (like in MemoryBank [2]) should help to control the memory size and\nexploit the experiences more effectively.  These perspectives will be studied\nin our future work.\n\n**About if LLM implicitly learns the certain $Q$ function**: We checked the Q\nvalues predicted by LLM during the test phase on WebShop. The average absolute\nerror is 0.417 (reward in WebShop is between 0 and 1), which indicates that *the\nLLM doesn't really learn the certain $Q$ function*. Nevertheless, as stated in\nglobal reply 1, the LLM is not expected to predict the accurate $Q$ value.\nInstead, we use the experience memory to implement a lookup-based $Q$ function\nand use RL to learn it. As for the LLM, *it is just expected to speculate which\naction is better (to be encouraged) and which one is worse (to be discouraged),\nas well as which action is the best among multiple encouraged actions*. Maybe\nour introduction to the \"output format\" in Sec. 3.2 confuse you.  We will\nrefine our expression in later revision.\n\n**Ablation study where LLM has no access to the discouraged actions**: We\nsupplemented experiments on WikiHow and the results are depicted below.\n\n| Setting         | Avg Reward | Success Rate |\n|:----------------|:---------:|:-------------:|\n| Full Model      |    2.63   |      0.93     |\n| w/o Discouraged |    2.48   |      0.81     |\n\nAfter removing the discouraged actions in the prompt, the performance seriously\ndegrades on WikiHow. We inspected the learned experience memories and the\nexemplars chosen by the agents. It is found that sometimes there are no proper\nactions to encourage in the retrieved experience. *In such cases, the\ndiscouraged actions will help the LLM to avoid several wrong attempts.  If the\ndiscouraged actions are then not accessible, the LLM will receive no valuable\nguidance from the experience.* This will lead to a poorer performance. Another\ninteresting discovery is that ablation model are more willing to imitate or\nrepeat the actions given in the action history in the aforementioned\ncircumstances rather than randomly explore. This mechanism is not clear to us\nby now.  We will update these new phenomena and perspectives in later revision.\n\n**Adaptability and versatility across different observation types**: We refer\nyou to global reply 2 for this question.\n\n**Architecture and learning algorithm for $Q$ function**: $Q$ function is\nimplemented with the experience memory as a lookup-based function. We implement\na basic Q-Learning algorithm to learn the $Q$ function.\n\n**About the figure and limitations**: Thanks for your kind reminder. We will\nupdate our figure and add Limitation section in later revision.\n\n* [1] Longtao Zheng, Rundong Wang, Bo An. Synapse: Leveraging Few-Shot\n  Exemplars for Human-Level Computer Control. arXiv:2306.07863.\n* [2] Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, Yanlin Wang. MemoryBank:\n  Enhancing Large Language Models with Long-Term Memory. arXiv:2305.10250."}, {"review_id": "Vqh4vREk6A", "reviewer": "Reviewer_dPzh", "summary": "This paper proposes a framework to combine RL w/ LLM using an offline Q-learning setting. An experience memory component is proposed to store past experience for estimating Q values. Evaluation on WikiHow and WebShop demonstrate the effectiveness of the proposed method and framework.", "questions": "1. It might be better to not use OpenAI logo to represent LLM in the figures.\n2. In Equation (1), how is max calculated?\n3. It is not clear how the reward is defined and can be obtained.\n4. There are a lot of RL issues people have overcome over years, such as over estimation of Q values, sparse reward, using replay buffer to provide iid data. It seems the proposed method will have all these issues. How did this paper tackle them? How do we guarantee that this method will converge?", "limitations": "It seems the proposed method of storing memory is not very scalable.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper proposes an interesting idea of combing LLM w/ RL. This is a relatively new field. The evaluation results on 2 tasks are strong.", "weaknesses": "This work seems very empirical. One question I have is how to generalize this method to other tasks. It will be good to see some theoretical analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable review and advice.\n\n**Generalizability**: For this question, we refer you to global reply 2.\n\n**$\\max$ in Eqn. 1**: This $\\max$ is calculated from the actions already\nrecorded for $(g, o_{t+1})$, as we cannot traverse all the possible actions\nwhen there are free-form languages. We will clarify this in later revision.\nActually, in the two tested task sets, the $Q$ value of the unrecorded actions\ncan be deemed to have a default value 0. Such a \"default\" value may have other\nbetter choices in different task domains.\n\n**How reward is defined and obtained**: The reward is defined by the underlying\ntask sets and obtained from the environments during interaction. As for\nWebShop, a score will be computed at the end as the returned reward according\nto the relevance between the reference product and the agent-bought product. As\nregards WikiHow, the agent is instructed to navigate several pages and a reward\nwill be given when the agent manages to reach one of the target pages.  The\ndetails about the reward definition should be referred to in the original paper\nof the task sets.\n\n**Classic RL issues**: This paper mainly focuses on how to improve and evolve\nLLM's capability with RL algorithm, but not on the issues of RL itself.\nTherefore, we only implement a basic Q-Learning method, ignoring a few issues\nin advanced RL domains. The methods aiming at solving these RL issues (*e.g.*,\nDouble Q-Learning [1], replay buffer [2], and diverse exploration methods) are\northogonal with this work, and will be straightforward to be embedded into\nRememberer. Besides, there are a few statements w.r.t. the specific concerns.\n\n* Over-estimation. As shown in Tab. 7 in the paper, over-estimation doesn't\n  constitute a serious problem for the two tested task sets. Double Q-Learning\n  is usually adopted for lookup-based $Q$ estimators to ameliorate\n  over-estimation by iteratively updating two $Q$ estimators. However, such an\n  iterative updating method may require more update steps to ensure an accurate\n  enough estimation than current Rememberer. If time permits, we will try to\n  supplement several results obtained with Double Q-Learning.\n* Sparse reward. On the two tested task sets, the strong base capability of LLM\n  guarantees trajectories that won't deviate far from the optimal policy. Thus,\n  sparsity of reward will not constitute a serious issue.\n* Convergence. We cannot guarantee mathematical convergence of Q-Learning on\n  the experience memory, as it is difficult for lots of seriously bad\n  observations and actions to be accessed owing to the strong base capability\n  of LLM, *i.e.*, insufficient traversal of observation & action spaces.\n  However, the experience memory is trained based on the strong capability of\n  LLM. Therefore, the training process shouldn't deviate far from the optimum.\n  Or in other word, the seriously bad states have already been implicitly\n  traversed in the pretraining stage of LLM, thus, convergence shouldn't be a\n  crucial issue. On the other hand, the experience memory serves as an\n  augmentation to a capable foundation model, thus, the mathematical\n  convergence is not as critical as for an RL model from scratch.  The current\n  version can already help the LLM well.\n\n**Scalability of the method to store experiences**: The powerful basic\ncapability of LLM makes a large-scale experience memory unnecessary. Meanwhile,\nthe limited number of exemplars in the prompt will lead to a performance\nsaturation when the size of experience memory increases continuously. We refer\nyou to the experiments in Sec. 4 in the supplementary for a deeper insight into\nthis argument. Besides, if there is truly a need for large-scale experience\nmemory, more scalable methods to store and retrieve the experiences (*e.g.*,\nvector database implemented with FAISS) can be surely adopted. The concrete\nimplementation of experience memory is not coupled with Rememberer framework.\n\n**About the figure**: Thanks for your kind reminder. We will update our figure\nin later revision.\n\n* [1] Hado van Hasselt. Double Q-learning. NeurIPS 2010.\n* [2] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel\n  Veness, Marc G. Bellemare, Alex Graves, Martin A. Riedmiller, Andreas\n  Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik,\n  Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg,\n  Demis Hassabis. Human-level control through deep reinforcement learning.\n  Nature 2015."}, {"review_id": "Ky9XbkxQYR", "reviewer": "Reviewer_4pvM", "summary": "The authors introduced Reinforcement Learning with Experience Memory (RLEM) to update the memory of the LLM agent, enabling it to evolve its capability without fine-tuning the parameters of the LLM. Extensive experiments were conducted on two RL task sets to evaluate the proposed framework. The experimental results demonstrate the superiority and robustness of Rememberer.", "questions": "1. Why use RL instead of other methods?\n2. If there are alternative methods to RL, should they be supplemented with comparisons in the experimental section?", "limitations": "Refer to weaknesses and questions", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The interesting perspective of exploration is a meaningful attempt to combine LLM with RL.\n2. The paper is well-written, easy to understand, and the illustrations are exquisite.\n3. The ample experiments demonstrate the effectiveness of the proposed method.", "weaknesses": "The discussion and introduction about whether to use RL are not sufficient. It is unclear why RL must be used instead of other methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your kind concern. First, we want to further clarify the necessity\nof updating the $Q$ values in the experience memory. Compared to the\ntraditional ICL (in-context learning) methods with labeled dynamic exemplars,\nthere will be both good and bad experiences in the memory of Rememberer. Hence,\n*it is necessary to discriminate between good and bad experiences through the\nlearned $Q$ values*. Given this premise, reinforcement learning is the most\nstraightforward method to update the recorded $Q$ values in POMDP\n(partially-observable Markov decision process) problems. And we didn't work out\nother alternatives.\n\nIf your concern is about other LLM-based agents utilizing memory or experience,\nwe are glad to have a quick comparison for you (Please note that most of these\nmethods are released after the submission deadline of NeurIPS, thus, we were unable\nto include the comparison into our paper). Voyager [1] proposes to use codes as\nactions, and designs a skill library to store the successful programs as\nskills.  GITM [2] stores the past successful action sequences in a text memory\nto assist an LLM planner in future planning without further discriminating\ntheir values.  Then the stored experiences are summarized by LLM to gain deeper\ninsights into planning policy. ChatDB [3] leverages relational database to\ntrack states in a dynamic process. In contrast, we store the experiences in a\nstructured memory and learns their $Q$ values to filters out the more valuable\nactions. In this way, we can combine RL with LLM and design a semi-parametric\nRL agent. We are glad to include these comparisons in Related Work in later\nrevision.\n\nAs it will take some certain efforts to migrate these methods to the test\nbenches in this paper, we may not introduce further new results during\nrebuttal.  Instead, we can draw a number of valuable inspirations from them and\nplan to include several new insights in our future work.\n\n* [1] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke\n  Zhu, Linxi Fan, Anima Anandkumar. Voyager: An Open-Ended Embodied Agent with\n  Large Language Models. arXiv:2305.16291.\n* [2] Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang,\n  Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, Yu Qiao, Zhaoxiang Zhang, Jifeng\n  Dai. Ghost in the Minecraft: Generally Capable Agents for Open-World\n  Environments via Large Language Models with Text-based Knowledge and Memory.\n  arXiv:2305.17144.\n* [3] Chenxu Hu, Jie Fu, Chenzhuang Du, Simian Luo, Junbo Zhao, Hang Zhao.\n  ChatDB: Augmenting LLMs with Databases as Their Symbolic Memory.\n  arXiv:2306.03901."}, {"review_id": "UMBEYOwDHj", "reviewer": "Reviewer_d9XL", "summary": "The paper proposes an LLM + RL architecture for text-based domains. The key idea is to run Q-learning on the side and augment LLM's prompt with Q values for available actions. The evaluation is done one WikiHow and WebShop.\n\nAcknowledging the rebuttal, I'm satisfied with authors responses and happy to increase the score.", "questions": "Would authors please address the points outlined in the weaknesses section.", "limitations": "The paper does not have limitations section", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper's core idea---adding q-values to LLM prompts---is interesting.", "weaknesses": "Clarity is the main issue with the paper. It is hard to read and hard to understand. Here are some of the issues that stand out:\n- In section 3.2 the Q-learning is introduced. It is not clear how exactly it is performed. Is function approximation used? If so then what is the network architecture. Is it tabular? If so then how come we are encountering exactly the same observations in the test set?\n- On line 163 the term \"flattening\" is used. This is not a common term in the literature, what is meant by it exactly? the eq.4 shgiws  bootstrapping for Q-value estimation. Is that what is meant?\n- There is a substantial amount of heuristics that is used to make the system work, but they are not well explained and their influence on the results is not well evaluated. For example, line 227-232 describes categorisation of the web pages into categories. Is this used for q learning? What happens if this categorisation is not used? Same with 243-249 for wikihow.\n- It is not clear whether the difference to the baselines is significant. For example on WebShop ReAct has avg score of 66.6 (from original paper) and the proposed method has 68. Is that a significant difference? Success rate is 40 for ReAct in the original paper, but is reported 36 in this paper. Where does the difference come from? Since the proposed method achieves the success rate of 38 this warrants a clarification.\n- Positioning in the introduction is adding to the confusion. The conversation of about using the external memory is misleading, as what is later proposed is Q-learning augmentation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your kind review.\n\n**About Q-Learning and $Q$ function architecture**: The $Q$ function is\nimplemented with the experience memory as a lookup table. Q-Learning is applied\nto the experience-memory-based lookup. Note that, we don't use this lookup to\npredict a $Q$ value for the new observation during inference. All the records\nin the lookup come from the training set and serve as candidate exemplars in\nthe input prompt to LLM. During the test, it is the LLM that will speculate a $Q$\nvalue for new observations and distinguish good actions from the bad ones. Here\nis a naive example:\n\nExperience Memory:\n\n| Task  | Observ. | Act.  | $Q$   |\n|-------|---------|-------|-------|\n| $g_0$ | $o_0$   | $a_0$ | $q_0$ |\n| $g_1$ | $o_1$   | $a_1$ | $q_1$ |\n\nAssume the new observation (& task) encountered during test is $(g, o)$. And\nseveral similar experiences will be retrieved from the memory, *e.g.*, $(g_0,\no_0)$ and $(g_1, o_1)$. Then the LLM will be fed:\n\n$$\ng_0, o_0: a_0 \\rightarrow q_0\n$$\n$$\ng_1, o_1: a_1 \\rightarrow q_1\n$$\n$$\ng, o:\n$$\n\nThe LLM is prompted to predict actions in format $a \\rightarrow q$ for $(g, o)$.\n\n**About \"flattening\"**: Eqn. 4 does show what is \"flattening\", which means we\nflatten/expand the preceding $n$ steps in the Bellman equation. We find that\nthis is usually called $n$-step Q-Learning or $n$-step bootstrapping [1, 2] and\nwe will update the expression in later revision accordingly.\n\n**About categorization in Sec. 4.1**: The categorization of pages in WebShop\nand instructions in WikiHow is used to calculate the similarity function\nintroduced in Sec. 3.3. The similarity function itself is not related with\nQ-Learning procedure, but is used to select relative exemplars in the input\nprompt to LLM for in-context learning. We will improve the corresponding\nexplanation and make it more clear in later revision.\n\n**About ReAct results**: The reported ReAct results in this paper are obtained\nwith *GPT LLM text-davinci-003* on 100 test tasks, while the results reported\nin the original paper of ReAct are obtained with *PaLM LLM* on 500 test tasks.\nThat's why the success rate reported in our Tab. 3 is **44**, while the success\nrate in the original paper is 40. The success rate 36 in Tab. 1 is an\n**average** result across three different exemplars (the result 44 takes exact\nthe same exemplar with the original paper of ReAct). You can also refer to the\naverage result 36 in \"Avg\" column in Tab. 3. Instead of significance of\nimprovement, we mainly claim the superior *robustness to the initial exemplars*\ncompared to ReAct, as shown by the results across different exemplars in Tab.\n3. It may take additional human labors and budget costs to find an optimal\nexemplar to guarantee the performance of the raw ReAct. In contrast, Rememberer\nholds a more stable performance across different initialization and can reduce\nthe required human labors to construct appropriate exemplars.\n\n**Relation between Q-Learning and experience memory**: For this question, we refer you to global reply 1. We use RL to assist\nLLM in exploiting the interaction experience, and the experience memory serves\nas *pivot* between RL algorithm and LLM. In practice, the experience memory is\nregarded as a tabular $Q$ function and is updated by Q-Learning.\n\n**About limitations**: Thanks for your kind reminder. We will add the section\nof Limitations in later revision.\n\n* [1] Watkins, Christopher John Cornish Hellaby. Learning from delayed rewards.\n  PhD thesis, University of Cambridge, England, 1989.\n* [2] Peng, Jing and Williams, Ronald J. Incremental multi-step q-learning.\n  Machine Learning 1996."}]

Author rebuttal:
Thanks to all the reviewers for kind review. We collect all the opinions and
give a reply to several common concerns here.

1. **Role of Q-Learning (and what the $Q$ function is)**: We'd like to further
   clarify our motivation and the role of Q-Learning in our proposed Rememberer
   approach. Our motivation is to make LLM utilize its experience and evolve
   its capability. A simple experience memory cannot label the good or bad
   experiences, thus, we introduce RL to learn the $Q$ values for the
   experiences, so that the good and bad experiences can be discrimianted and
   the memory-augmented system can evolve its ability.  In this system,
   Q-Learning is not directly performed onto LLM.  Instead, it is the
   *experience memory* which RL is applied to so as to avoid directly updating
   LLM parameters. *I.e.*, the experience memory constitutes a lookup-based $Q$
   function.  Actually, We don't expect that LLM can accurately predict the $Q$
   value. *LLM is just expected to be capable of discriminating between better
   and worse actions* with the help of exemplars from the experience memory.
2. **Generalizability**: The introduced RLEM framework and Rememberer approach
   are not intrinsically constrained to text domain. The working domain mainly
   depends on the underlying LLM, while Rememberer can work with any available
   LLM per se. *By replacing with a multimodal LLM like GPT-4 or Flamingo,
   Rememberer can also deal with multimodal observations. Besides, even a pure
   text LLM still has an opportunity to handle other modalities.* *E.g.*, some
   work like Socratic Models [1] leverages captioning model to enable a text
   LLM to handle visual inputs. In Voyager [2], the complex observation from
   Minecraft is summarized into texts. In this paper, WebShop and WikiHow are
   also multimodal tasks in fact. The GUIs from them are represented in a text
   format (See Sec. 1 in supplementary). For demonstration, we are conducting a
   simple experiment with Atari environment.  The results may not come up in
   time, however, we will at least describe our plan to represent an Atari
   observation in text format in later replies.

* [1] Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian
  Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael S. Ryoo, Vikas
  Sindhwani, Johnny Lee, Vincent Vanhoucke, Pete Florence. Socratic Models:
  Composing Zero-Shot Multimodal Reasoning with Language. ICLR 2023.
* [2] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke
  Zhu, Linxi Fan, Anima Anandkumar. Voyager: An Open-Ended Embodied Agent with
  Large Language Models. arXiv:2305.16291.

Meta-review: This paper proposes to combine reinforcement learning with large language models by augmenting language model prompts with Q values.  Several reviewers found the paper difficult to follow but others thought the positives, like its innovative core idea, outweigh the negatives.
Decision: Accept (poster)