LDM$^2$: A Large Decision Model Imitating Human Cognition with Dynamic Memory Enhancement
Authors: Xingjin Wang
Linjing Li
Daniel Dajun Zeng
Venue: EMNLP 2023 Findings (2023)
Keywords: Large Language Model
Decision Making
Memory Enhanced

Abstract: With the rapid development of large language models (LLMs), it is highly demanded that LLMs can be adopted to make decisions to enable the artificial general intelligence. 
Most approaches leverage manually crafted examples to prompt the LLMs to imitate the decision process of human. However, designing optimal prompts is difficult and the patterned prompts can hardly be generalized to more complex environments. 
In this paper, we propose a novel model named Large Decision Model with Memory (LDM$^2$), which leverages a dynamic memory mechanism to construct dynamic prompts, guiding the LLMs in making proper decisions according to the faced state.
LDM$^2$ consists of two stages: memory formation and memory refinement.
In the former stage, human behaviors are decomposed into state-action tuples utilizing the powerful summarizing ability of LLMs. Then, these tuples are stored in the memory, whose indices are generated by the LLMs, to facilitate the retrieval of the most relevant subset of memorized tuples based on the current state.
In the latter stage, our LDM$^2$ employs tree exploration to discover more suitable decision processes and enrich the memory by adding valuable state-action tuples.
The dynamic circle of exploration and memory enhancement provides LDM$^2$ a better understanding of the global environment.
Extensive experiments conducted in two interactive environments have shown that our LDM$^2$ outperforms the baselines in terms of both score and success rate, which demonstrates its effectiveness.

LDM A Large Decision Model Imitating Human Cognition with Dynamic Memory Enhancement
Xingjin Wang Linjing Li Daniel Dajun Zengl2
'School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, China 'eijjng Wenge Technology Co.,Ltd (wangxingjin2021, linjing. dajun. zeng zeng)@ia. ac. cn
Abstract
Tuplec::
Esamplet:
Soal comees.
Gosd you aca bertle torght shral
Pao: loarcs
With the rapid development of large language models (LLMs), it is highly demanded that LLMs can be adopted to make decisions to enable the artificial general intelligence. Most approaches leverage manually crafted exam- ples to prompt the LLMS imitate the decision process of human. However, designing optimal prompts is difficult and the patterned prompts can hardly be generalized to more complex en- vironments. In this paper, we propose a novel model named Large Decision Model with Mem- ory (LDMS), which leverages dynamic mem- ory mechanism to construct dynamic prompts, guiding the LLMs in making proper decisions according to the faced state. LDM2 consists of two stages: memory formation and mem- ory refinement. In the former stage, human behaviors are decomposed into state-action tu- ples utilizing the powerful summarizing ability of LLMs. Then, these tuples are stored in the memory, whose indices are generated hy the LLMs. to facilitate the retrieval of the most relevant subset of memorized tuples based on the current state. In the latter stage, our LDM2 employs tree exploration to discover more suit- able decision processes and enrich the memory by adding valuable state-action tuples. The dynamic circle of exploration and memory en- hancement provides LDM' a better understand- ing the global environment. Extensive ex- periments conducted in two interactive environ- ments have shown that our LDM2 outperforms the baselines in terms of both score and success rate, which demonstrates its effectiveness.
Authar une lright
Bicete Swarth
INTTOCARI: Bright Cinn Dentorent
Aatio (NREWaer1))
Kglos Clic (Jyy Now
Doat pless by stand..
Seall oradeacc cac-.
Ooseranoos
Medth lundll (omm)) (4mmm))
Choonation dieddd Actios diikk (hay Hov)
Farmpich
S Lange Language Model
hurrh
bunch
Anlee searc beace can stooee
(eakk potserson
Adtee (XPORTIGNN)
kaken Clis (rof)
Agle bocmeern
Dbserattr:: and bowet...
Anlis Clo
Figure I: Comparison between prompt with examples and prompt with state-action tuples.
including text/code generation (Chen et al., 2021), question answering (Zhou et al., 2022), and rea- soning (Wei et al., 2022b), to name a few. Be- sides NLP tasks, LLMs are also employed as pol- icy agents to accomplish decision-making tasks (Kim et al., 2022: Mialon et al., 2023). Nowadays, most approaches adopt the standard prompting paradigm which uses manually crafted in-context examples prompt the LLMs (Sanh al., 2022; Wei et al., 2022a) making decisions. However, the standard prompting is not suitable for decision- making tasks, as restricts the LLMs to merely imitate the provided examples, making the gener- ated decisions highly context-sensitive. Worse still, standard prompting cannot generate admissible de- cisions in complex environments (Liu et al., 2021; Dong et al., 2022). In certain cases, the LLMs may fall into confusion when new situations are greatly different from the examples. In more challeng- ing situations, even humans are unable to provide complete solution examples. Meanwhile, once the prompt examples are fixed, the LLMs can no longer
1 Introduction
The rapid development of large language models (LLMs) has led to remarkable revolution in the field of natural language processing (NLP). LLMs, such as Llama (Touvron et al., 2023), PaLM (Chowdh- ery et al., 2022), and GPT-4 (OpenAI, 2023), have achieved impressive results in variety of tasks,
*Corresponding author

---

learn from the feedback of the environment, thus cannot further improve the performance.
ods using fixed examples prompt, this advantage empowers the LLMs handle unseen or complex situations. Additionally, We conduct ablation ex- periments to evaluate the memory refinement mech- anism. The results demonstrate the effectiveness of adding highly rewarded state-action tuples into the memory. Our main contributions can be sum- marized as:
In this paper, we proposed Large Decision Model with Memory (LDM)), a framework that en- hances the standard LLMs with dynamic updating memory. The memory mechanism maintains the most valuable state-action tuples when imitating human decisions. As shown in Fig. 1, the prompts generated by the proposed LDM2 focus on pro- viding LLMs with sufficient information to guide decision making in the current situation, rather than relying solely on the simple examples in the entire process of decisions. In order to obtain sufficient state-action tuples in the memory, LDM2 incorpo- rates a memory formation stage that is analogous to the traditional imitation learning (Hussein et al., 2017). In this stage, we take human trajectories as training data and instruct the LLMs produce numerous standard state-action tuples including the task goals, observations, historical information, and actions. These standard state-action tuples are pre- served to form the initial memory. In the inference phrase, we retrieve the most similar state-action tuples from the memory with current observation and then construct the prompt inspire the LLMs dynamically. These retrieved state-action tuples I) inform the LLMs which actions would be taken by human in the current state and 2) help the LLMs understand the global environment.
We propose novel paradigm that leverages a two-stage memory mechanism to dynami- cally prompt the standard LLMs for decision- making tasks.
We make full use of the standard LLMs in the memory formation stage to produce state- action tuples and generate the corresponding indices.
We adopt tree exploration to generate potential decision processes and instruct the LLMs to identify the most valuable state-action tuples enhance the memory.
2 Related work
Our paper is closely related to the following three research directions: LLMs for recision-making feedback for LLMs, and memory and retrieval for LLMs. In this section, we briefly review the litera- ture on these research.
Differentiate from traditional imitation learning (Hussein al., 2017), LDM2 is equipped with dynamic memory refinement stage enhance the memory with valuable state-action tuples. First, We conduct tree exploration to generate all poten- tial decision processes and evaluate them according to the environment rewards. Then, we add the state- action tuple corresponding to the best decision pro- cess into the memory. This exploration explorationeevaluation adding circle mimics the traditional reinforcement learning framework (Arulkumaran al., 2017; Yao al., 2020). The refinement stage not only expands the action space of the LLMs, but also enable the LLMs to deal with new situations not covered by the initial memory.
LLMs for decision-making Powerful LLMs are able to act as policy models to make decisions in in- teractive environments. Li al. (2022) constructed a general framework for decision-making that uses LLMs to encode observations, goals, and history and then generate actions. Demonstration of exam- ple prompts and utilization of high-level function libraries are employed to explore innovative strate- gies (Huang et al., 2022a; Liang al., 2022: Wu et al., 2023; Nakano et al., 2021; Shen al., 2023). Prompting structure with pre-defined functions, be- haviors, and examples are leveraged ground LLMs to generate robotic actions (Ahn et al., 2022; Huang et al., 2022b; Vemprala et al., 2023). How- ever, these methods use manually crafted exam- ples to prompt the LLMs which results in decision- making in fixed direction. Our LDM leverages dynamic Mate-nctioon tuples prompt improve the effectiveness of decisions.
We evaluate the proposed LDM2 in two inter- active environments: WebShop and ALFworld. LDM2 outperforms the standard few shots prompt- ing methods and other methods prompted with ver- bal reasoning. We further analyze the successful examples in both tasks and find that LDM2 has a more diverse action space compared with meth-
Feedback for LLMs Recent techniques have emerged that focus on establishing closed-loop sys- tems which are capable of utilizing the scalar or

---

Human Trajectories
New Gool spant... New Observation:
Reat
Memory Formation
Fledron
Memory Refinement
dim
agent
Observation:
[earch]:
Tree Exploration
Anture Cao (8344501543)
Obseraation:
Gual: Eletron
Adian: Cliss.
I-I
Pat::
Observation:
[Uee]
New Action GonfMeence New Oiserpption
New Action 2: Conffiencc00
New Action N... Confdences83 New Obseraation N:
Goolt Food
Action: Click.
Pat spent
New Obsenollgn
Otnereation:
Asii Memury
Observatint
Paath Confidence
[Detalis]
Goal:
Actin
Part: sganb..
Observation:
Observatine
bot
Gratie Cuatter
Chatte
Action: Click.
Pvennt
Tuples
Port:
Pryuec
Phane
Obneraation:
I-lp
Datiiir Procene
Decision Proces
Decision
Ground Procss
Deshiion Propees
Valueble
Action CIiSs.
Tusles
Observation
[Detalis]
Action
Compare
Ooservaton::
Action: (lci..
Figure 2: Overview of the proposed LDM The formation of memory (left) takes human trajectories training data to produce standard state-action tuples (blue blocks) and then instructs the LLMs to generate the index by clustering data based the goals (green blocks) and observations (yellow blocks). The refinement of memory (right) leverages the tree exploration to produce potential action process and instruct the LLMs to find the most valuable state-action tuples. Finally, the valuable tuples are added into (red arrow) the corresponding memory batch.
textual feedback from environment or human to update the LLMs (Christiano et al., 2017; Ouyang al., 2022; Chen et al., 2022; Bai al., 2022). Madaan et al. (2023) and Pryzant et al. (2023) em- ploy an iterative framework for self-refinement to optimize the prompt of LLMs based on the feed- back of self-evaluation. REFINER (Paul et al., 2023) fine-tunes another critic model to provide intermediate feedback within trajectories to im- prove the reasoning response. ReAct (Yao et al., 2022b) prompts LLMs with both verbal reason- ing traces and actions which guides the models to perform dynamic reasoning according environ- mental feedback. Reflexion (Shinn et al., 2023) converts binary or scalar feedback from the envi- ronment into verbal feedback which is then added in the prompt of the next episode. Introspective Tips (Chen et al., 2023) learns tips from the ac- tion trajectories and environmental feedback to em- power the LLM agents with self-optimizing capa- bilities. These approaches mainly aim to leverage reward feedback to augment the prompt LLMs, however, LDM adds highly rewarded statc-action tuples into the memory which achieves dynamic learning ability.
Memory for LLMs Memory could store infor- mation perceived from the environment and lever- ages the recorded memories to facilitate future tions. Generative Agents (Park et al., 2023) main- tain a memory stream to record the experience including observations and behaviors. Reflexion (Shinn et al., 2023) stores experiential feedback natural language within a sliding window. Voyager (Wang et al., 2023) employs natural language de- scriptions to represent skills within the Minecraft game, which are directly stored in memory. Mem- ory Bank (Zhong al., 2023) encodes the mem- ory segment into embedding vector which could enhance memory retrieval and reading efficiency. Knowledge base is also used as the memory to retrieve relevant information and construct the task- related augmented prompts (He et al., 2022; Trivedi et al., 2022; Khattab et al., 2022). Our LDM2 also constructs memory to record vast state-action tu- ples, which forms the retrieval indices list through instructing the LLMs cluster different goals and observations.
Methodology
In this paper, we consider general setup of LLMs as policy models to accomplish decision- meriingn-mmking tasks an interactive environment. the follow-

---

ing, LLMA(-) denotes employing a LLM per- form the A function/operation.
assists the LLMs completing a whole decision pro- cess Py w.r.t. the goal g². We require the LLMs choose the optimal process as the final decision:
3.1 Problem Definition
as the training data, where each has a task We leverage N human decision trajectories {t', ......... ................................ tN trajectory t¹ = {oi,ai. : -r;
Pf = LLM choosse
(5)
where Pg is the decision process based on the batch memory Mp. b 1,2,.
goal g', T; is the length of this trajectory, of is the observation time step T, a; is the human action when faced with oz, 1 f T S Ti. The set T of trajectories can be further processed into a memory M consisting of standard state-action tuples:
3.2 Memory Formation
Previous imitation learning methods enable pol- icy agent to mimic expert behavior through updat- ing the parameters of language models. However, in the new prompt paradigm of LLMs (Liu et al., 2023), we need integrate human cognition into the context prompt while freezing the parameters of LLMs. Our LDM leverages the memory to store vast state-action tuples and constructs dynamic con- text prompt based on the current observation, which imitates the human decisions.
M 11 - Sis
(1) where hy (to be further elaborated in the next sub- section) represents historical information about the observations and actions before time step т. The memory M provides the LLMs with a sufficient set of state-action tuples to help generate proper ac- tions in various situations. Based on M, the LLMs can interactively explore the environment. Given new task goal g², the LLMs receive the current observation of at time step т, the historical infor- mation hiž and the context prompt pz to generate an action af:
Memory Structure and Format As introduced in the last subsection, the memory M consists of a large number of standard state-action tu- ples. This subsection depicts how to construct the memory M. Given human trajectory t' {or ai, .r, it can be divided into T₃ standard state-action tuples. In addition to the current observation of, the past decision process {of, ai. ... o of - and the task goal g' are the crucial factors that L.Mss must consider when making decision at time step т. However, the raw data of the past process is relatively long as the prompt context. Therefore, we instruct the LLMs to summarize them into the brief historical infor- mation:
df LLM SaMM,, Mis p.)
(2)
where the context prompt pt is subset of memory M that can be retrieved from M according to the task goal  and the current observation of:
b = LLMeetrieve oz).
(3)
In order to accomplish the goal g², the context prompt tells the LLMs about actions taken by hu- man in the current state, invoking the LLMs to comprehend the environment. Subsequently, the LLMs generate the complete decision process pi and get the final reward r:
hi LLM af) "f.....,,, of
1: at
1ig").
(6)
The instruction of this summary process require the LLMs to briefly describe the past experiences and assess the progress of tasks the current state. The historical information also provide LLMs agent with planning information, indicating the decision stage it has reached and assisting the agent in mak- ing proper decisions. The complete prompt of the summary process is listed in the appendix A.
pi = of of
(4)
Due to the context length, an intrinsic lim- itation of LLMs, we have to partition all those N trajectories into n batch data with size B, where N nB. Each batch data To [1b-1))++1 (66-1)B-2.... [B6), b 1,2, ., processed to form batch memory MG by the same procedure described above. Ac- cordingly, we construct n independent batch mem- ory M1,M2,'.. MMn. Each batch memory Md
To sum up, as indicted in Eq. (1), standard state- action tuple of human trajectory 1 time step т contains four elements: task goal g², agent history hi,, current observation of, and the current action a demonstrated Eq. (2). As shown in Fig. 2,

---

the blue blocks in the left represents the obtained state-action tuples from the human trajectories.
through splitting more leaf nodes at each parent nodes. For task goal g'. at each time step т, we instruct the LLMs to provide some possible actions based on the current observation and the memory M, and we prompt the LLMs to assign a confi- dence score to each action (the complete prompt is listed in appendix A).
Memory Index To efficiently store and retrieve tuples from the memory M, we construct the fol- lowing index system including two types of indices: goal index and observation index.
First, cluster the goals of different tasks in each batch memory form its goal index, which is achieved by instructing the LLMSs to generate high- level types of received information and classify each task goal to the corresponding type:
at c/. ...,, af d LLM(O Ny of pf), (9)
If the action distribution of retrieved state-action tu- ples in each node is highly concentrated, the LLMs will select the majority action and proceed to the next state. Otherwise, the LLMs will retain all admissible actions and explore subtree for each action. Meanwhile, we maintain the confidence of each exploration path, which is the product of confidence score of all nodes along the path. To avoid the exponential growth exploration paths, we only retain top-n (n=4) confidence paths for the the next step of exploration and prune the valueless exploration paths. Finally, we obtain top-n (n=4) decision processes and get the final rewards at the leaf nodes.
indexy LLMaluster (gh, 9h. 9h
where index Mp. b 1,2, y(b 1)BLE 2. ..... B. Then, we cluster the observations of each goal type by instructing the LLMs to classify all observations into a high-level type to form the observation index:
y is the goal index of batch memory and of is the goal of trajectory
index bk = LLM (ott
Zuke
(8)
where index is the observation index of goal type & in the batch memory M, and Zik denotes the total quantity of observations in goal type k. In the inference phrase, the LLMs agent firstly leverage the goal index to retrieve the similar tasks and then use the observation index to find the simi- lar situations with the current state, The complete prompt of the cluster process is listed in appendix A. As shown in Fig. 2, the green and yellow blocks are the classified data based on the goals and ob- servations. Compared with traditional clustering methods, LLMs based clustering could work with text-based input and generate text-based output in- stead of numerical data representations, which is more flexible and effective to capture complex se- mantic relationships in text-rich decision environ- ment.
Memory Enhancement The tree exploration generates a set of decision process Pi, Pj. with reward r1; r2, ... respectively. The process with the maximal reward is the best decision process P; and the process which is formed by instructing the best decision process based
LLMs to generate on the memory M is the ground decision process Pa If P have higher reward than Pj. we then instruct the LLMs to compare these processes and find the key decision step in the P Then we treat the subtrajectory after the key steps as the valuable data to enhance the memory:
(o af...... . . = LLLLLLopprr (Pi, Po).
(10)
where T" is the key step in the best decision process given by the LLMs.
3.3 Memory Refinement
The obtained valuable subtrajectory is converted into standard state-action tuples using the method depicted in the above sections. For each pair, we leverage the goal index and observation index to find the corresponding categories and directly add this tuples into this subset of the memory. Adding these new valuable state-action tuples into the mem- ory M changes the distribution of the action space. As shown in the right of Fig. 2, the L.MMs con- duct tree exploration to generate more decision processes. The best and ground processes in the
The above memory formation stage follows the imitation learning paradigm, which provides LDM2 a initial policy. To improve the policy dynamically, we adopt tree exploration, which mimics online reinforcement learning, to enhance the memory by adding the most valuable state-action tuples into M.
Tree Exploration We leverage the tree explo- ration to generate more possible decision processes

---

lea nodes are converted state-action tuples to enhance the memory M which marked by the red arrows.
Method
Score
SR
Standard
62.7
29.6
WebShop
62.4
28.7
ReAct
67.1
39.6
4 Experiment Setup
Reflexion
68.4
40.2
LDM
71.8
40.8
4.1 Tasks and Datasets
LDM In+Re
72.4
41.6
We evaluate LDM on two language-based interac- tive decision- maasinn tasks: ALFWorld and Web- Shop. Both are complex environments with various observations and actions that are difficult to be ad- dressed through fixed examples prompt.
Human Expert
82.
59.6
Table Score and success rate results on WebShop. The results of WenShop are from (Yao al., 2022a). The trial number of Reflexion is 4. The LDM,, f. is the result of initial memory and LDM is the result of refined memory.
WebShop WebShop (Yao et al., 2022a) is sim- ulated e-commerce website environment with real- world products and crowd-sourced text instructions. Given a text instruction specifying a product re- quirement, agent needs to navigate multiple types of webpages and make actions to find, cus- tomize, and purchase the required product. The performance is evaluated by average score and sucess rate, the former the percentage of desired attributes covered by the chosen product averaged across all episodes, and the latter is the percentage of episodes where the chosen product satisfics all requirements on 500 test instructions.
ReAct The prompts in the ReAct (Yao et al., 2022b) include not only the observations and ac- tions, but also verbal reasoning traces to guide the LLMs perform dynamic reasoning in the decision process.
Reflexion Reflexion (Shinn et al., 2023) is an extension of ReAct, which uses self-reflect to con- vert binary or scalar feedback from the environ- ment into verbal feedback and repeats the same task many times based on the reflection.
Imitation Learning BUTLER (Shridhar et al.. 2020) is an imitation learning agent for ALFWorld tasks trained on a large amount of human trajecto- ries. WebShop (Yao al., 2022a) finetunes mul- tiple language models learn how search and choose from various shopping processes.
ALFworld ALFworld (Shridhar et al., 2020) is suite of text-based environments which require the agent to accomplish multi-step tasks in variety interactive environments based on TerlWorld (Côté et ul., 2019). ALFworld includes six types of goals (e.g. picking specified objects and putting in designated place, examining objects by specific instructions and manipulating the objects through specific means). The agent needs to navigate and interact with a simulated household determine the actions. We conduct experiments on 134 test games, the result is scored by the success rate that is the percentage of episodes which achieves the given goals.
4.3 Training Setup
In the LDM memory formation stage, we use 500 human shopping trajectories to construct the Web- Shop training memory and 200 expert trajectories for each task in the ALFWorld to form the memory. We the batch size as 100 for WebShop and 50 for ALFWorld construct multiple independent batch memories for both tasks. In the LDM mem- ory refinement stage, we utilize 100 new instruc- tions to explore the WebShop environment and 10 new goals for each task to explore the ALFWorld environment.
4.2 Baselines
We compare LDM with two prompt-based ap- proaches using complete examples and traditional imitation learning methods trained with annotated trajectories. In all experiments, we employ GPT- 3.5 3. 5-turbo) as the workhorse LLM.
Results and Analyses
5.1 WebShop
Standard We directly leverage few-shot suc- cessful example decision processes as context to prompt the LLMs for both tasks.
shown by Tab. 1, our LDM2 outperforms all baselines in both score and success rate, which in- dicates the effectiveness of leveraging itate-action

---

Method
Pick
Clean
Heat
Cool
Look
Pick2
All
Standard
88
55
70
67
72
41
66
BUTLER,
33
26
70
76
17
12
22
BUTLER
46
39
74
100
22
24
37
ReAct
63
48
74
71
67
35
60
ReActhest
92
65
96
86
78
47
78
Reflexion
88
81
83
90
83
88
85
LDM
88
81
87
90
83
71
84
LDME
96
87
91
90
89
76
89
In-Re
Table 2: ALFWorld task-specific success (%)) BUTLER and BUTLER, results are from (Shridhar al., 2020). ReAct use two examples as prompt and Re Reteet is the best result in 6 prompts (Yao et al., 2022b). The trial number of Reflexion is 5. The LDMF, the result of initial memory and LDM is the result of refined memory.
when no option exists the current situation.
Analysis on the Initial Memory The result experiment on initial memory outperforms other baselines, which demonstrates the effectiveness of using initial memory to provide sufficient state- action tuples for the LLMs. State-action tuple ex- amples of the current situation assist the LLMs in understanding the patterns of the environment in- stead of merely imitating the actions of existing examples. As shown the upper left of Tab. 3, Re- Act first selects one of the most plausible product from the list of products, but the product detail is not fully aligned with the instruction. However, the LLMs still imitate the examples to purchase this The proposed LDM2 also select this product, but it clicks the "Prev" button to search another after finding that the detail is not matched. The reason that the retrieved memory in this state includes same situations in which the products are not the best choice, and click on "Prev" to find another product is illustrated in the examples.
70-
Score or Succass rate (%)
80
40-
Cal pee
30-
50-
20
WebShop
WorWo
Figure 3: The relationship between number of batch data and score or success rate.
tuples as dynamic prompts to instruct LLMs to make decisions. The traditional Imitation Learning method finetumes a language model with medium size, which results in poorer performance com- pared with the prompt-based method using L.L.s. Also, the poor performance of the standard prompt- ing method validates that fixed prompt of complete tasks,
Analysis of the Refined Memory The perfor- mance of model equipped with the refined memory better than the one with only the initial memory, which shows the effectiveness of updating the mem- ory by exploring more valuable decision processes. The nodes which generate multiple actions the tree exploration are mainly the products selection nodes. As shown in the upper right of Tab. 3, based on the initial memory, LDM2 selects the most plau- sible products, but these products do not include the specified options (size 11 women), which sults in lower reward. However, LDM2 with the refined memory explores more possible products and finds the product matching all options. In ad- dition, the tree exploration can expand and enrich
examples is not suitable for decision-makee Standard prompting methods may select products same order as demonstrated in the examples, instead of selecting the proper products according to the instructions. Additionally, they may gen- erate actions that similar to the examples but not executable in the current environment. Though ReAct adds dynamic reasoning into the prompt, may make incorrect reasoning when encountered with different situations. For instance, the ReAct prompts include thought process clicking corre- sponding attribute options, thus it may imitate the given reasoning process to click one option even

---

Instruction: I am looking for high power sound column subwoofer, uses bluetooth and also a 3d surround sound system, and price lower than
Instruction: I need khaki steel toe shoes size women, and price lower than 70.00 dollars.
650,00 dollars.
ReAct: Search [high power subwoofer... CCCcckk [..-RRQ] (-.. BBuueooohh Speaker subwoofer Portable Small 3D Surround... >CClickk [Buy
Initial: Search [khaki steel toe shoes size women]->Click [-.. [..-LHI]--Ciice [Khaki]]+Click [10.5 women men] (Size ... (--0005 women men] [11.5 women 9.5 menj [Buy
WehShop
RRwaad [0.5]
Now]->Reward [0.75]
Initial: Search [high power sound column subwnofer -I-Click [... RRQ]=Click {< Prev]->Click
Refined: Search [khaki steel toe shoes size women]-<Clik [-.. Q2KI-<Click [khaki]-<Click [11 women men| (Size 1.-[10.5 women 8.5 men] [1I women mea] .a) --CClk [Buy Now] -Reward [1.0]
..
.... TS6] (--- Bluctooth Speaker 60W High Power Sound Column Outdoor Subwoofer 3D Stereo Sur-
round Sound System ) C>>cc [Buy Now] RReaard
[1.0]
Goal: You are the middle of room, put two soapbar in garbagecan.
Goal: You are the middle of room, cool some mug and put in cabinet.
ReAct: go to cabinet I -opo cabinet 1->go to cabi- not 2-+go to cabinet go to cabinet 3-+go to cahi- net 4-te- cabinet 4-take soaphar from cabinet
Initial: go fridge 1 1- to shelf 1- to cabinet
I go to cabinet 2-+go to cabinet 3 3-+go to cabinet
4 go to cabinet 5- to cabinet 6 6 ReWard[O]
4-+- put soaphar in/on garbageean 1-.--g0 go to cabinet 1-+... Reward]0]
ALFWorld
Iniiial: go to cabinet cabinne cabinet I 1- to coun- tertop 1-take soapbar 1 from countertop 1- soapbar 1 in/on garbagecan 1->go to countertop 1->take soapbar from countertop 1->...ut soap- bar 3 in/on garbagecan I->Reward[1]
Refined: go to fridge -- to countertop T-take
mug 2 from countertop 1 1->go to fridge 1-+c0ol mug 2 with fridge 1 1- to cabinet 1->put mug in/on cabinet 1->Rewrd[[1]
Table 3: Sample result of WehShop and ALFWorld based the initial memory and refined memory.
the initial memory to help the model revise actions in case the state-action tuples in the initial memory are not sufficient.
shown in the bottom left of Tab. 3, ReAct takes first sopabar from cabinet and then falls into confu- sion, as it does not know where to find the second item. Based on the memory, the human experience teaches LDM2 to go to the most likely place (coun- tertop) instead of exploring all places, thus finds the items efficiently.
5.2 ALFWorld
According to Tab. 2, LDM2 also outperforms all baselines evaluated on tasks in ALFWorld. The prompt-based LLMs outperform the traditional deep learning method. In ALFWorld, finding the desired object and performing correct operations often involves many steps, which results in loss of the current state's tracking when received prompt with long examples. Despite its dynamic reasoning ability, ReAct still generates incorrect actions due to the long decision process and unseen situations.
Analysis on the Refined Memory The nodes split in the ALFWorld are the most possible selec- tion nodes. In the memory refinement stage, LDM2 generates some possible places to explore the en- vironment. LDM2 finds more convenient and fast ways to complete the goals and adds these tuples into the memory. Meanwhile, in some cases where there are no analogous situations to the test task in the memory, the tree exploration process can as- sist the LLMs in exploring the common appearing places of the unseen items. As shown in the bottom right of Tab. 3, LDM fails to find the desired item in the new environment based on the past human experience, but the tree exploration help find the item by providing more possible places.
Analysis on the Initial Memory The imitation learning has a better performance than other base- lines, which shows LDM2 can form a valid initial memory for ALFWorld tasks. In the memory for- mation stage, the LLMs not only cluster the house- hold items into many high-level types like furniture, kitchen ware, and electronic devices, but also clus- ter the observations as kitchen room, bathroom, and bedroom. The state-action tuples in different subsets of the memory guide LDM2 to go to the most likely place to find the desired object, take the desired object, manipulate the object correctly, and then put the object in the designated place. As
5.3 Analysis of Training Data
As the batch data represents human experience, we expect that the performance of LDM2 will increase as the data size increases. Thus we conduct experi- ment to find the relationship between score/success

---

rate and the size of the batch data.
sidered, but other methods must record the whole past experience which will increase as time goes by. The amount of inference tokens of our method and existing methods are roughly equal.
As shown by Fig. 3, the score/success rate in- creases in both tasks as the size of batch data in- creases, which demonstrates that more human tra- jectories can enhance the LLMs' ability to make more proper actions in the current state. In Web- Shop. more data means more types of products to help the LLMs learn what search or click. In ALFWorld, more data provides the LLMs more information about where the desired objects may appear and how to manipulate them correctly.
A cknooweegements
This work was supported in part by the Strategic Priority Research Program Chinese Academy Sciences under Grant #XDA27030100 and the Na- tional Natural Science Foundation of China under Grants #72293573, #72293575 and #62206282.
6 Conclusion
References
This paper proposed LDM2, which enhances the standard LLMs with dynamic updating memory to maintain the most valuable state-action tuples to imitate human decision. LDM consists of two stage: memory formation and memory refinement. In the formation stage, we take human trajectories training data and instruct the LLMs produce numerous standard state-action tuples. These stan- dard state-action tuples are preserved to form the initial memory. LDM is equipped with dynamic memory refinement stage to enhance the memory through adding valuable state-action tuples. We conduct tree exploration to generate all potential decision processes and add the state-action tuple corresponding to the higher reward decision pro- cess into the memory. Experiments on two inter- active environments illustrated that LDM2 outper- forms the standard few shots prompting methods and the ablation study verified the effectiveness of the memory formation and refinement mechanism.
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. 2022. Do as i can, not as i say: Ground- ing language in robotic affordances. arXiv preprint arXiv:2204.01691.
Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. 2017. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26-38.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini. Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
Boyuan Chen, Fei Xia, Brian Ichter, Kanishka Rao, Keerthana Gopalakrishnan, Michael S Ryoo, Austin Stone, and Daniel Kappler. 2022. Open-vocabulary queryable scene representations for real world plan- ning. arXiv preprint arXiv: arXiv:2209009874.
Limitations
Liting Chen, Lu Wang. Hang Dong. Yali Du. Jie Yan. Fangkai Yang. Shuang Li, Pu Zhao, Si Qin, Saravan Rajmohan, et al. 2023. Introspective tips: Large lan- guage model for in-context decision making. arXiv preprin arXiv:2305. 1598.
To fully leverage the capabilities of LDM2, we need to collect a certain amount of high-quality human trajectories, which may be difficult and infeasible in some environments. Hence, late the LLMs' own reasoning and understanding ability when having few data to interact with the environments.
need to stimu-
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0337..
Meanwhile, LDM2 bears a higher time cost com- pared with other standard prompting methods. In the inference phrase, LDM2 needs to retrieve rel- evant state-action tuples in the memory, which re- sults in a significant lower action generation speed. However, the computational cost of our method is comparable with other methods, because the prompt length of each timestep of our method is shorter and only the current state needs to be con-
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.022. 023111.
Paul Christiano, Jan Leike, Tom Brown, Miljan Mar- tic, Shane Legg, and Dario Amodei. 2017. Deep

---

reinforcement learning from human preferences, Ad- vances in neural information processing systems, 30.
programs for embodied control. arXiv preprint arXiv:2209.07753.
Mare-Alex Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla EI Asti, Mahmoud Adada, et al. 2019. Textworld: A learning environ- ment for text-based games. In Computer Games: 7th Workshop, CGW 2018, Held in Conjunction with the 27th International Conference on Artificial In- lelligence, IJCAI 2018, Stockholm, Sweden, July 13, 2018, Revised Selected Papers? 7, pages 41-75. Springer.
Jiachang Liu, Dinghan Shen. Yizhe Zhang. Bill Dolan,
Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiv preprimi arXiv:2101.0680
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig, 2023. Pre- train, prompt. and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint arXiv:2301.022:
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan. Luyu Gao. Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. et al. 2023, Self-refine: Iterative refinement with self-feedback. arXiv preprint arXiv:2303.1765.
Hangfeng He, Hongming Zhang, and Dan Roth. 2022. Rethinking with retrieval: Faithful large language model inference. arXiv preprint arXiv:2301.00300 0030033...000...
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christo- foros Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozière, Timo Schick, Jane Dwivedi- Yu, Asli Celikyilmaz, et 2023. Augmented language models: a survey. arXiv preprint arXiv:2302.0784.
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022a. Language models as zero- shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118-9147 PMLR.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders. et al. 2021. Webgpt: Browser-assisted question- answering with human feedback. arXiv preprint a22.:221
Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. 2022b. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint 207:02207.05608. 05608.
OpenAL. 2023. Gpt-4 technical report.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 25:27730-27744
Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. 2017. Imitation learning: sur- vey of learning methods. ACM Computing Surveys (CSUR), 1032):1-35.
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts and Matei Zaharia. 2022. Demonstrate-seant predict: Composing retrieval and language mod- els for knowledge-intensive nlp. arXiv preprint arXiv:2212.14024.
Joon Sung Park, Joseph O'Brrien, Carrie J Cai, Mered- Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442.
Minsoo Kim, Yconjoon Jung, Dohyeon Lee, and Seung- won Hwang. 2022. PLM-based world models for text-based games. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 1324-1341, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beat- riz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023. Refiner: Reasoning feedback on intermediate representations. arXiv preprint 2rXiv:2304.07904
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chen- guang Zhu, and Michael Zeng. 2023. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint 2305v2335..03495
Shuang Li, Xavier Puig, Chris Paxton, Yilun Du, Clin- ton Wang, Linxi Fan, Tao Chen, De-An Huang, Ekin al. 2022. Pre-
Akyürek, Anima Anandkumar, trained language models for interactive decision- making. Advances in Neural Information Processing Systems, 35:31199-3122
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja. et al. 2022. Multitask prompted training enables zero-shot task generalization. In ICLR 2022-Tenth International Conference on Learning Representa- lions.
Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian lchter, Pete Florence, and Andy Zeng. 2022. Code as policies: Language model

---

Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li. Weiming Lu, and Yueting Zhuang. 2023. Hugging- gpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303. 175.0
on Empirical Methods Natural Language Process- ing (EMNLP), pages 9736-8754, Online. Association for Computational Linguistics.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022b. React: Synergizing reasoning and acting in language models. arXiv preprini 2r 00.22220..33000 10.03629.
Noah Shimn, Beck Labash, and Ashwin Gopinath. 2023. Reflexion: an autonomous agent with dy- namic memory and self-reflection. arXiv preprint arXiv:2303. 11366.
Wanjun Zhong, Lianghong Guo, Qiqi Gao. He Ye, and Yanlin Wang. 2023. Memorybank: Enhancing large language models with long-term memory.
Mohit Shridhar, Xingdi Yuan. Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2020. Alfworld: Aligning text and em- bodied environments for interactive learning, arXiv preprint arxiv:2010.03768.
Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoe Le, and Chi. 2022. Least-to-most prompting enables complex reason- ing in large language models. arXiv preprint urXiv:2205. 10625.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro. Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1397.
Prompt Details
All the prompts used in this paper instruct the LLMs shown in Tab. 4.
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. Interleav-
ing retrieval with chain-of-thought reasoning for
B Example Process
wwwlledge-intensiye multi-step questions. arXiv preprint arXiv:2212.10009
Tab. and Tab. 6 provide an example decision pro- cess in the WebShop task produced by the proposed LDM2. Due to the long decision-making process, omit some parts of prompts.
Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. 2023. Chatgpt for robotics: Design principles and model abilities. Microsoft Auton. Syst. Robot. Res, 2:20.
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Man- dlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. 2023. Voyager: An open-ended embodied agent with large language models.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. 2022a. Finetuned language models are zero-shot learners. In International Con- ference Learning Representations.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou. 2022b. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201. 1903.
Chenfei Wu, Shengming Yin, Weizhen Qi, Xi- aodong Wang, Zecheng Tang, and Nan Duan. 2023. Visual chatgpt: Talking, drawing and edit- ing with visual foundation models. arXiv preprint arXiv:2303.0667.
Shunyu Yao, Howard Chen, John Yang. and Karthik Narasimhan. 2022a. Webshop: Towards scalable real-world web interaction with grounded language agents. In Advances in Neural Information Process- ing Systems.
Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. 2020. Keep CALM and ex- plore: Language models for action generation text- based games. Proceedings of the 2020 Conference

---

Evlaution:
I will give you task goal and agent past action process.
You should partition the goal into some subgoal and judge the past actions whether complete these subgoals.
The desired format is: subgoal 1:goal complete or in complete etc.
Do not give me explanation.
Summarization:
I will give you the past process and you should summarize the past process.
The desired format must be: Summary:
Do not give explanation.
Cluster Goals:
I will give you a few numbered task goals.
You need to help me classify these goals into some types. The number of each category should be almost average. The category must be high-level rype. The desired format is:
High-level Typel: type name [number] High-level Type2: type name [number] High-level Type3: type name [number] etc.
Cluster Observations:
I will give you few numbered observations.
You need to help me classify these observations into some types. The number of each category should be almost average. The category must be high-level type.
The desired format is:
High-level Typel: type name [number] High-level Type2: type name [number] High-level Type3: type name [number] etc.
Index Observations:
I will give you some numbered goal types and examples of this type. You should judge the new goal belongs to which type.
The desired format is:
[Type number]: reason Do not give me other information.
Index
I will give you some observation numbered types and examples of this type.
Observations:
You should judge the new observation belongs to which type.
The desired formal is:
(Type number]: reason
Do not give other information.
Action:
I give some numbered examples and a new observation.
You should imitate the actions in the example and give me the next action. give some numbered examples and a new observation.
Tree
Exploration:
You should imitate the actions in the example and give me some possible
next actions and the confidence of each action. All confidence should sum equal 1.
Compare:
I will give you two shopping process. Each process have some numbered step.
The first process is better than the second, Can you tell me the first number that two process different.
The desired formal is: Number: Reason.
Do not give any other information and strictly follow the format.
Table 4: Prompt used in the LDM2

---

Promptl:
1. Instruction: need a pair of shoes with rubber soles. remember get size seven and half womens, and price lower than 40.00 dollars.
Past:
Action: search[shoes, rubber soles, size 7.5, women's less than 40]
2. Instruction: can you find some carol wright mens lounge pants in 5x1? want the ones with a draw string closure that are charcoal colored, and price lower than 40.00 dollars
Past:
Action: search[5xl carol wright men's ss lounge pants charcoal color]
Instruction: need some black ankle strap flats that are in size wide, and price lower than 60.00 dollars
Past:
Action:
search[black ankle strap flats "size 9 wide"]
4. Instruction: need fleece jacket for the winter that is warm and gray, and price lower than 40.00
dollars
Past:
Action: scarch[winte fleece jacket gray warm]
Instruction: want extra large yellow loose fit shirt, and price lower than 40.00 dollars
Past:
Action: search(extra large yellow men's loose fit shirt]
Observationl: Instruction: Find me long lasting, moisture wicking, loose fit men's shorts with elastic waistband, quality materials, polyester cotton, short sleeve with color: black, and size: 4x-large, and price lower than 50.00 dollars.
Actionl: search[long lasting moisture wicking loose fit men's shorts with elastic waistband black 4a-lerge polyester cotton]
Prompt2:
1. Past:
Searched for 5XL Carol Wright men's 's lounge pants in charcoal color.
The interface is:
[Back to Search]
Page (Total results: 50)
Next >]
[08B6D39FM]
Carol Wright Gifts Men's Comfy Lounge Pant
14.99 17.99
[B075RCFF55]
Carol Wright Gifts Men's Fleece Lounge Pants by Cozee Corner
14.99 17.99
[B03LUVGVI
Carol Wright Gifts Women's Flats Comfortable Flats for Women Women's Dress Flats
21.99 to 32.99
[B08BI9VXVY]
Carol Wright Gifts Comfy Slip-On 21.99
Action: click (08b6d39fm)
Past: Searching for women's shoes with rubber soles in size 7.5 for less than 40.
The interface is:
Back to Search]
Page (Total results: 50)
Next >]

---

[B075IHYSBG]
ASICS Men's Gel-Lyte Gymnastics Shoes
35.34 to 77.99
[B09L5YX7IC]
Foggs Japan Slippers, Cozislides Original, CloudFeet Ultra-Soft Slippers, EVA Universal Quick- drying Thickened Non-slip Sandals 100.0
[B09PXXC33]]
ZSpzx Women's Fashion Sneakers Lace Up Casual Round Toe Arch Support Comfortable Low Top Sneakers Glitter Classic Leather Cushion Foam Comfort Loafers Dress Shoes Ladies Walking Flat Shoes 24.99 32.8
Action: click(b051lhy5pg
Observation2:
Past: Searched for black 4x-large polyester cotton loose fit men's 's shorts with elastic waistband that have long lasting moisture wicking.
The interface is:
Baac to Search]
Page (Total results: 50)
[Nxt >]
[B09QSZHRVM]
Yinimo Mens Gym Shorts Fashionable Loose Beach Drawstring Turkey Print Beach Cool Shorts
19.19
[09QQQ3336]]
HAUKLIE Men's Sports Waffle Ribbed Polo Shirts Summer Short Sleeve Cotton Muscle Quarter-Zip
Henley T-Shirt Tunics Tops
10.99
[B09NDSP2QR]
InterestPrint Deer and Stars Men's Loungewear Pajama Sets Long Sleeve Tee and Jogger Pant
43.59
Action2: click(B0QQ5ZHRVM]
Prompt3:
I.Past: Searched for black ankle strap flats in size wide and clicked on b09r2dhbd3. The interface is:
Back to Search]
[< Prev]
color [aa04-black] bro04-brown] khaki [aage-beige] [aa03-black] [aa02-yellow] [aa-01 beige] [aa-01 brown]
size [5.5] [6] [6.5] [6.5-7] [7] [7.5] [8] [8.5] [9] [9.5-10] [10] [10.5] [11] [11.5]
Women Flats Sandals for Summer,Fashion Casual Flat Shoes Solid Color Denim Zipper Sandals
Buckle Strap Open Toe Flip Flops
Price: 7.07 to 8.23
Rating: N.A.
Description]
Features]
Reviews]
[By Now]
Action: click [a003bbaack]
2.Past: Searched for black ankle strap flats in size wide, clicked on b09r2dhbd3, and then clicked on
aa03-black.

---

The interface is:
[Back to Search]
[< Prev]
color [aa04-black] [a004-brown] [aa04-khaki] [aa03-beige] [aa03-black] [aa02-yellow] (aa-01 beige]
[aa-01 brown]
size [5.5] 66] [6.5] 66.5-7 [7] [7.5] [8] [8.5] [9] [9.5-10] [10] [10.5] [11] [11.5]
Women Flats Sandals for Fashonnnashion Casual Flat Shoes Solid Color Denim Zipper Sandals
Buckle Strap Open Toe Flip Flops
Price: 7.07 to 8.23
Rating: N.A.
Description]
Features]
Reviews]
[Buy Now]
Action: click[9]
Observation3:
Past: Scarched for and clicked black 4x-large polyester cotton men's shorts with elastic waistband that have long lasting moisture wicking and loose fit.
The interface is:
[aack to Search] (< Prev] color [black] [blue] [red]
size [small] [medium] [Large] [x-large] [xx-large] [3x-large] [4x-large] [5x-large]
Yinimo Mens Gym Shorts Fashionable Loose Beach Drawstring Turkey Print Beach Cool Shorts
Price: 19.19
Rating: N.A.
[eecription]
Features]
Reviews]
[Buy Now]
Action3: click[black]
Prompt4:
2.Past: Searched for black ankle strap flats in size wide, clicked on b09r2dhbd3, and then clicked on aa03-black.
The interface is:
Back to Search]
[< Prev]
color [aa04-black] (ua04-brown) khaki] beige [aa03-black] [aa02-yellow] [aa-01 beige]
[aa-01 brown]
size [5.5] [6] [6.5] 66.5-7] [7] [7.5] [8] [8.5] [9] [9.5-10] [10] [10.5] [11] [11.5]
Women Flats Sandals for Summer.Fashion Casual Flat Shoes Solid Color Denim Zipper Sandals
Buckle Strap Open Toe Flip Flops
Price: 7.07 to 8.23
Rating: N.A.
Description]
Features]
Reviews]
[Buy Now]

---

Action: click[9]
3.Past: Searched for black ankle strap flats in size wide, clicked on b09r2dhbd3, then clicked on aa03-black and finally clicked on size
The interface is:
Back to Search]
k Prev]
color [aa04-black] [aa04-brown] kaa04- [aaOS-beige] [aa03-black] [aa02-yellow] (aa-01 beige]
[aa-01 brown]
size [5.5] [6] [6.5] [6.5-7] [7] [7.5] [8 [8.5] [9] [9.5-10] [10] [10.5] [11] [11.5]
Women Flats Sandals for Summer,Fashion Casual Flat Shoes Solid Color Denim Zipper Sandals
Buckle Strap Open Toe Flip Flops
Price: 7.07 to 8.23
Rating: N.A.
Description]
Features]
Reviews]
[Buy Now]
Action: click[buy now]
Observation4:
Past: Searched for and clicked on black, long-lasting, moisture-wicking, loose-fit men's 's shorts with
elastic waistband in 4X-large size made of polyester cotton. The interface is:
Baak to Search]
( Prev] color [black] [blue] [red]
size [small] [medium] [large] [x-large] [xx-large] [3x-large] [4x-large] [5x-large]
Yinimo Mens Gym Shorts Fashionable Loose Beach Drawstring Turkey Print Beach Cool Shorts
Price: 19.19
Rating: N.A.
Description]
Features]
Reviews]
[Buy Now]
You have clicked black.
Action4: largee
Prompt5:
1.Past: Past: Searched for black ankle strap flats in size 9 wide, clicked on b09r2dhhd3, then clicked on aa03-black and finally clicked on size 9.
The interface is:
[Bac to Search]
[< Prev]
color [aa04-black] [ro04-brown] [aa04-khaki] (aage-beige] [aa03-black] [aa02-yellow] [aa-01 beige] [aa-01 brown]
size [5.5] [6] [6.5] 66.57] [7] [7.5] [8] [8.5] [9] [9.5-10] [10] [10.5] [11] [11.5]
Women Flats Sandals for Hummer,Fashion Casual Flat Shoes Solid Color Denim Zipper Sandals
Buckle Strap Open Toe Flip Flops
Price: 7.07 to 8.23
Rating: N.A.

---

Description
Features]
Reviews
[Buy Now]
Action: click[buy now]
2.Past: Searched for 5XL Carol Wright men's lounge pants in charcoal color, clicked item code B08B6D39FM, selected size 5X-large, and chose the color charcoal. The interface is:
[Back Search]
K< Prev]
size [small] [medium] [large] [x-large] [xx-large] [3x-large] [4x-large] [5x-large]
color [black] [charcoal] [gray] [heather blue] [navy] [plaid] [olive]
Carol Wright Gifts Men's 's Comfy Lounge Pant
Price: 14.99 to 17.99
Rating: N.A.
[Deceription]
[Features]
[Reviews]
[Buy Now]
Action: click[buy now]
ObservationS:
Past: Searched for and clicked on black 4x-large polyester cotton moisture wicking loose fit men's shorts with elastic waistband (ASIN: B09Q5ZHRVM).
The interface is:
Back to Scarch]
{< Prev]
color [black] [blue] [red]
size [small] mediium] [large] [x-large] [xx-large] [3x-large] [4x-large] [5x-large]
Yinimo Mens Gym Shorts Fashionable Loose Beach Drawstring Turkey Print Beach Cool Shorts
Price: 19.19
Rating: N.A.
[Decrription]
[Feature]]
Reviews]]
[Buy Now]
You have clicked black.
You have clicked 4a-large
Action5: click[Buy Now]
Table 5: An Example decision process in the WebShop.

---

Goals: cool some apple and put diningtable.
You are in the middle of room. Looking quickly around you, you see cabinet 1, a cabinet 10, a cabinet 11, a cabinet 2, a cabinet 3. a cabinet 4, a cabinet 5, cabinet 6, a cabinet 7, cabinet 8, a cabinet 9, coffeemachine 1, countertop 1, countertop 2, diningtable 1, drawer 1, a drawer 2, a drawer 3, a fridge 1, garbagecan 1, microwave 1, sinkbasin 1, a stoveburner 1, stoveburner 2, stoveburner 3, stoveburner 4, and a toaster 1.
Prompil:
1. Observation: You are in the middle of room. Looking quickly around you, you nothing. Evaluation:Past: Subgoal I: Incomplete Subgoal 2: Cool Incomplete Subgoal 3: Incomplete Summary: Unknown. More information needed to provide a summary of the past process.
Action: go sinkbasin
Observation: You arrive at loc 12. On the sinkbasin I, you see a apple 2, a mug 2, and a potato 1. Evaluation: Past: Subgoal I: Take apples from somewhere Incomplete Subgoal 2: Cool apples Incomplete Subgoal 3: Put apples into dining table- Incomplete
Summary: Looked and went to sink basin
Action: take apple 2 from sinkbasin
3. Observation: You arrive loc 8. On the sinkbasin you egg mug 1, and potato 2. Evaluation.Past: Subgoal Take apple from somewhere Incomplete Subgoal 2: Cool apple Incomplete Subgoal 3: Put apple into countertop- Incomplete Summary: Looked and went to sink basin 1.
Action: go to microwave
4. Observation: You arrive at loc 10. The microwave is closed.
Evaluation:Past: Subgoal 1: Take apple from somewhere Incomplete Subgoal 2: Cool apple Incomplete Subgoal 3: Put apple into countertop Incomplete Summary: Looked and then went to sink basin and microwave 1.
Action: open microwave
Observation: You close the microwave
Evaluation: subgoal take apple from somewhere incomplete subgoal 2: cool apple incom- plete subgoal 3: put apple into countertop- incomplete
Summary: Looked and went to sink basin 1, then went to microwave 1, opened and closed it. Action: go to garbagecan
Observation1:
Observation: You in the middle of a room. Looking quickly around you, you see nothing.
Evaluation: Subgoal I: take apple from somewhere -Incomplete Subgoal 2: cool apple Incomplete Subgoal 3: put apple into countertop -Incomplet
Summary: Unknown. More information needed to provide a summary of the past process.
Action1: go to garbagecan
Prompt2:
1. Observation: You are in the middle of room. Looking quickly around you, you nothing, Evaluation:Pastt Subgoal I: Incomplete Subgoal 2: Cool Incomplete Subgoal 3: Incomplete Summary: Unknown. More information needed to provide summary of the past process.
Action: go sinkbasin 1
Observation: You arrive loc 12. On the sinkbasin 1, you see a apple 2, a mug 2, and potato Evaluation: Past: Subgoal 1: Take apples from somewhere Incomplete Subgoal 2: Cool apples Incomplete Subgoal 3: Put apples into dining table Incomplete
Summary: Looked and went to sink basin 1.
Action: take apple 2 from sinkbasin

---

Observation: You arrive loc On the sinkbasin you see a egg a mug 1, and potato 2. Evaluation.Passt Subgoal Take apple from somewhere Incomplete Subgoal 2: Cool apple Incomplete Subgoal 3: Put apple into countertop- Incomplete Summary: Looked and went to sink basin 1. Action: go to microwave 1
4. Observation: You arrive at loc 10. The microwave is closed.
Evaluation.Pastt Subgoal 1: Take apple from somewhere Incomplete Subgoal 2: Cool apple Incomplete Subgoal 3: Put apple into countertop Incomplete Summary: Looked and then went sink basin 1 and microwave 1.
Action: open microwave
Observation: You close the microwave 1.
Evaluation:Past. subgoal take apple from somewhere incomplete subgoal 2: cool apple incom- plete subgoal 3: put apple into countertop- incomplete
Summary: Looked and went to sink basin 1, then went to microwave I, opened and closed it. Action: go garbagecan
Observation2:
Observation: On the garbagecan you see a apple 1, and a tomato 1.
Evaluation: Subgoal I: take apple from somewhere -Incomplete Subgoal 2: cool apple -Incomplete Subgoal 3: put apple into countertop -Incomplete
Summary: Looked and then went to garbagecan 1.
Action2: take apple from garbagecan
Prompt3:
1. Observation: You pick up the apple 2 from the sinkbasin 1. Evaluation.Past: Subgoal 1:
Take apple from somewhere Complete Subgoal Cool apple Incomplete Subgoal 3: Put apple into dining table Incomplete Summary: Took apple from sinkbasin 1.
Action: go to fridge
2. Observation: You arrive at loc 16. The fridge is closed.
Evaluation:Past: Subgoal 1: Take apple from somewhere Complete Subgoal 2: Cool apple
Incomplete Subgoal 3: Put apple into dining table Incomplete
Summary: Looked, went to sinkbasin I, took apple 2 from sinkbasin 1, and went to fridge 1.
Action: cool apple with fridge
Observation: You pick up the apple from the garbagecan
Evaluation:Past: Subgoal 1: Take apple from somewhere Complete Subgoal Cool apple
Incomplete Subgoal 3: Put apple into countertop Incomplete Summary: Looked around, went to sinkbasin 1, went to microwave 1, opened and closed microwave
1, went to garbagecan 1, took apple from garbagecan 1.
Action: go to fridge
4. Observation: You arrive loc 1. The fridge is closed. Evaluation:Pastt Subgoal 1: Take apple from somewhere Complete Subgoal 2: Cool apple Incomplete Subgoal 3: Put apple into countertop- Incomplete
Summary: Looked around, went sinkbasin 1, went microwave 1, opened and closed microwave 1, went to garbagecan 1, took apple from garbagecan 1, and went to fridge 1.
Action: cool apple with fridge

---

Observation3:
Observation: You pick up the apple from the garbagecan 1.
Evaluation: Subgoal I: take apple from somewhere -complete Subgoal 2: cool apple -Incomplete Subgoal 3: put apple into countertop -Incomplete
Summary: Looked for apple, found it in garbage can 1.
Action3: go to fridge
Prompt4:
1. Observation: You pick up the apple 2 from the sinkbasin Evaluation:Past: Subgoal I:
Take apple from somewhere Complete Subgoal Cool apple Incomplete Subgoal 3: Put apple into dining table Incomplete Summary: Took apple from sinkbasin 1.
Action: go to fridge
2. Observation: You arrive at loc 16. The fridge is closed.
Evaluation.Past: Subgoal 1: Take apple from somewhere Complete Subgoal 2: Cool apple Incomplete Subgoal 3: Put apple into dining table Incomplete
Summary: Looked, went sinkbasin 1, took apple from sinkbasin and went fridge 1.
Action: cool apple with fridge
Observation: You pick up the apple from the garbagecan
Evaluation.Past. Subgoal I: Take apple from somewhere Complete Subgoal Cool apple Incomplete Subgoal 3: Put apple into countertop Incomplete Summary: Looked around, went to sinkbasin I, went to microwave 1, opened and closed microwave
1, went to garbagecan 1, took apple from garbagecan 1.
Action: go to fridge
4. Observation: You arrive loc 1. The fridge is closed. Evaluation:Past: Subgoal 1: Take apple from somewhere Complete Subgoal Cool apple Incomplete Subgoal 3: Put apple into countertop- Incomplete
Summary: Looked around, went to sinkbasin 1, went to microwave 1, opened and closed microwave 1, went to garbagecan 1, took apple from garbagecan 1, and went to fridge 1.
Action: cool apple with fridge
Observation4:
Observation: The fridge 1 closed.
Evaluation: Subgoal take apple from somewhere -complete Subgoal 2: cool apple Incomplete Subgoal 3: put apple into countertop IIncmpleee
Summary: Looked for apple, found it in garbage can and took it.
Action4: cool apple with fridge
Prompt5:
1. Observation: You cool the apple 2 using the fridge 1.
Evaluation:Past: subgoal 1: take apple 2 from somewhere complete subgoal 2: cool apple 2 complete subgoal 3: put apple 2 into diningtable incomplete
Summary: Looked for apple 2, took it from sinkbasin and cooled with fridge 1.
Action: go diningtable
Observation: You arrive at loc 15. the diningtable 1, you see a apple 3, a apple 4, a bowl 1, a bowl 2, butterknife 1, a egg 3, a fork 2, a fork 3, knife 2, knife 3, a lettuce 1, a lettuce 2, a lettuce 3, a spatula 3, and a spoon 2.

---

Evaluation.Past Subgoal 1: Take apple 2 from somewhere Complete Subgoal 2: Cool apple Complete Subgoal 3: Put apple into diningtable Complete Summary: Looked for apple 2. took it from sinkbasin 1, cooled in fridge 1, and placed on diningtable 1.
Action: put apple 2 in/on diningtable 1
3. Observation: You cool the apple using the fridge 1.
Evaluation:PPstt: Subgoal I: Take apple from somewhere Complete Subgoal 2: Cool apple Complete
Subgoal 3: Put apple into countertop Incomplete Summary: Looked for apple, went sinkbasin went to microwave 1, opened and closed microwave 1, went to garbagecan took apple from garbagecan went to fridge 1, cooled apple with fridge 1.
Action: go to countertop
4. Observation: You arrive at loc 15. On the countertop 1. you see a bowl 1, a bread 1, a butterknife 1, a cup 1, a dishsponge 1, a fork 2, a knife 2, a ladle 2, a pan 1, a potato 1, soapbottle 1, spatula 1, a spatula 2, spatula 3, and a spoon 1.
Evaluation:Pastt Subgoal I: Take apple from somewhere Complete Subgoal 2: Cool apple Complete Subgoal 3: Put apple into countertop Complete
Summary: Looked for an apple, found one in the garbage, cooled it in the fridge, and placed it on the countertop.
Action: put apple in/on countertop
ObservationS:
Observation: You cool the apple 1 using the fridge 1.
Evaluation: Subgoal I: take apple from somewhere -complete Subgoal 2: cool apple -complete Subgoal 3: put apple into countertop -Icompplee
Summary: Looked for apple 1, took it from garbage can 1, and cooled it with fridge
Action5: go to countertop
Prompt6:
1. Observation: You cool the apple 2 using the fridge 1.
Evaluation:Past: subgoal 1: take apple 2 from somewhere complete subgoal 2: cool apple 2 complete subgoal 3: put apple 2 into diningtable incomplete
Summary: Looked for apple 2, took it from sinkbasin 1, and cooled it with fridge 1.
Action: go diningtable
Observation: You arrive at loc 15. On the diningtable you see apple 3, apple 4, a bowl 1, a
bowl 2, butterknife a egg 3, a fork 2, fork 3, knife 2, knife 3, lettuce 1, lettuce 2, lettuce 3, a spatula 3, and a spoon 2.
Evaluation:Past. Subgoal 1: Take apple 2 from somewhere - Complete Subgoal 2: Cool apple 2 Complete Subgoal 3: Put apple into diningtable Complete Summary: Looked for apple 2, took it from sinkbasin 1, cooled it in fridge 1, and placed it on diningtable 1.
Action: put apple 2 in/on diningtable 1
3. Observation: You cool the apple using the fridge 1.
Evaluation:Past: Subgoal 1: Take apple from somewhere Complete Subgoal Cool apple Complete
Subgoal 3: Put apple into countertop Incomplete Summary: Looked for an apple, went to sinkbasin 1, went to microwave 1, opened and closed microwave 1, went to garbagecan 1, took apple from garbagecan went to fridge 1, cooled apple with fridge 1.
Action: go countertop

---

4. Observation: You arrive at loc 15. On the countertop 1, you see a bowl I, a bread 1, a butterknife 1,
cup I, dishsponge I, fork 2, knife 2, ladle 2, pan 1, potato I, soapbottle 1, a spatula 1, spatula 2, a spatula 3, and spoon 1.
Evaluation.Past: Subgoal 1: Take apple from somewhere Complete Subgoal 2: Cool apple- Complete Subgoal 3: Put apple into countertop Complete
Summary: Looked for an apple, found one in the garbage, cooled it in the fridge, and placed the countertop.
Action: put apple in/on countertop
Observation6:
Observation: You arrive at loc 15, On the countertop 1, you see bowl 1, bread 1, butterknife 1, cup dishsponge 1, fork 2, a knife 2, ladle 2, pan 1, potato soaphottle I, spatula 1, spatula 2, a spatula 3, and spoon 1.
Evaluation: Subgoal take apple from somewhere -complete Subgoal cool apple -complete Subgoal 3: put apple into countertop -complete
Summary: Looked for apple 1, took it from garbage can 1, cooled it fridge 1, and placed it on countertop 1.
Actionó: put apple in/on diningtable
Table 6: An Example decision process in the ALFWorld.


Reviews:
[{"review_id": "OEfB3HWblF", "reviewer": "Reviewer_X6GX", "summary": "", "questions": "1. Refer to Con(1), could you provide a more detailed cost analysis and comparison to existing methods?\n2. Refer to Con(2), did you have to collect human data? If so, you could give more details on how they are collected, who they are collected from, etc.\n\n3. The human behaviors are not exactly “training data” as no real training happens during the pipeline? Maybe replace this term to avoid confusion?\n4. I may have missed this, but I don’t see how the reward $r$ is calculated from the methodology section?\n5. The memory formation stage requires rewriting the human behaviors into a summary of state-action tuples with the GPT model. During this process, would there be any information loss? Are there any mistakes made during rewriting? I’d love to see an analysis on this aspect as well.\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful comments and suggestions.\n\n1.The number of training data of each task is not more than 200, thus it is not very difficult for LLMs to generate state-action tuples. The cost is medium and worth, because the agent can automatically learn from the given training data for different tasks instead of carefully designing task-specific prompts by hand. This process could also assist the agent to form a task-specific initial memory. \n\n2.Due to the context length constraint, we have to split the whole training data into several batches. Moreover, since the training data is not evenly distributed, each batch should be considered. Besides, the computational cost of this stage also depends on the number of training data, but the overall cost is not much.\n\n3.If there are many diverging actions in each node, the exploration paths will indeed grow exponentially. In our experiments, we find that the action distribution of retrieved state-action tuples in each node is highly concentrative and we instruct the LLM agent to generate the most occurred actions which are relatively few (only one in most cases). What’s more, nodes with several forks are mainly located in the upper layers of the exploration tree, and nodes in the lower layers are basically not needed to split. Hence, the exploration tree will not grow exponentially and the cost is considerable. Meanwhile, we consider adding an evaluation process in the tree exploration process which lets the LLM agent evaluate whether the current exploration direction is worth continuing. If the current exploration is valueless, the agent will prune the subtree from this node.\n4.\tThe number of human training data is not more than 200 which is not difficult to collect. In fact, there are plenty of available data collected in many scenarios. One innovation of our method is that we use these human data to automatically construct prompts. \n\nQ1: We can compare the time cost and computation cost of our method with existing methods. At each timestep, existing methods only need one inference to generate action but our method need to instruct LLMs to retrieve relevant memory and then generate action. In average, the number of LLM API calls of our method is four times than that of existing methods. However, the prompt length of each API call of our method is shorter because  only the current state needs to be considered, but existing methods must record the whole past experience which will increase as time goes by. The amount of inference tokens of our method and existing methods are roughly equal which is shown by the billing provided by OpenAI. All in all, the computation cost of our method is comparable with other methods, but our method bears a higher time cost. \n\nQ2: In our experiment, we directly use the human data collected by Yao et al. (2022a) for WebShop and Shridhar et al. (2020) for ALFworld. \n\nQ3: The reason why we refer the human behaviors as “training data” is that we feed these data to the LLM agents, and the agents learn from these data to form the memory that can be used to accomplish the task. The quality and number of the samples will affect the performance of learning. This process mimics the training process as the context of the memory is varying, though there is no parameter updating. \n\nQ4: We directly use the WebShop reward defined in Yao et al. (2022a) and the ALFworld reward defined in Shridhar et al. (2020). \n\nQ5: The state-action tuples consist of task goal g, history information ht, and current observation ot. The content length of these three should be roughly equal, otherwise the LLM agent may pay too much attention to one part and make unreasonable actions. We summarize the history information into short text with the same order of magnitude in length as the current observation, thus the agent pays equal attention to all parts in the tuples and considers them comprehensively. The summary process indeed loss some information, but the details in the history is not crucial and the LLMs agent can use the details in the current observation to help make decisions. The summarized history information assists the agent finding the similar experience and avoiding wrong actions appeared in the past.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The structure is innovative. The authors attempt to mimic the RL algorithm by designing different components that utilizes LLMs, such as dynamic memory, decoding (tree exploration in this case).\n2. The results show good improvement from the baselines.\n", "reasons_to_reject": "1. This algorithm may incur high costs. Firstly, the LLM needs to encode all training data (human behaviors) into state-action tuples. Secondly, due to the context length constraint, a decision process needs to be derived from each batch memory (M_b) in order to select the optimal process. Thirdly, the tree structure search may incur higher costs than normal decoding strategies as many paths need to be maintained simultaneously. Overall, even though the results are improved, it may not be worth it if the costs have increased too much.\n2. To provide better data, i.e. human behaviors, human responses need to be collected, which may again be very costly.\n", "questions_to_authors": "1. Refer to Con(1), could you provide a more detailed cost analysis and comparison to existing methods?\n2. Refer to Con(2), did you have to collect human data? If so, you could give more details on how they are collected, who they are collected from, etc.\n\n3. The human behaviors are not exactly “training data” as no real training happens during the pipeline? Maybe replace this term to avoid confusion?\n4. I may have missed this, but I don’t see how the reward $r$ is calculated from the methodology section?\n5. The memory formation stage requires rewriting the human behaviors into a summary of state-action tuples with the GPT model. During this process, would there be any information loss? Are there any mistakes made during rewriting? I’d love to see an analysis on this aspect as well.\n", "paper_topic_and_main_contributions": "This work proposes LDM^2, a large decision model with memory. It uses state-action tuples derived from human behaviors to form a dynamic memory storage. Then, it leverages relevant memory to guide the LLM in its decision process. It searches all potential decision processes with tree-exploration and evaluates them using environment rewards. Then, the best decision process is selected and added to dynamic memory. The main innovation is the exploration-evaluation-adding circle, which mimics reinforcement learning.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "the procedure in L338-342 resembles Zhao et al. 2023 (https://aclanthology.org/2023.acl-long.320/).\n"}, "extra_text": {}}, {"review_id": "7Tw93ryP6F", "reviewer": "Reviewer_gqCr", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful comments and suggestions. \n\n1.If there are numerous actions in each subnode, the exploration paths will indeed grow exponentially.  In our experiments, we find that the action distribution of retrieved state-action tuples in each node is highly concentrative and we instruct the LLM agent to generate the most occurred actions which are relatively few (only one in most cases). What’s more, nodes with several forks are mainly located in the upper layers of the exploration tree, and nodes in the lower layers are basically not needed to split. Hence, the exploration tree will not grow exponentially and the cost is considerable. Meanwhile, we consider adding an evaluation process in the tree exploration process which lets the LLM agent evaluate whether the current exploration direction is worth continuing. If the current exploration is valueless, the agent will prune the subtree from this node.\n\n2.Comparing with traditional clustering methods, our method has the following advantages. First, LLMs work with text-based input and generate text-based output, instead of numerical data representations, which is more flexible. Second, LLMs have the potential to capture complex semantic relationships between data points based on textual descriptions; LLMs consider not only quantitative features, but also textual context. Hence, using LLMs for clustering is more effective in text-rich domains where their language understanding capability can be harnessed effectively. What’s more, the LLM agent leverage its own in-parametric knowledge and cognitive ability to cluster the tuples, the clustering result is also beneficial to the LLM agent when faced with retrieving and understanding tasks in the future, although the result is not completely correct in the point of view of human being. The LLM agent is completely independent, it can act as versatile tools to accomplish sub-tasks and complete the whole decision-making process without resorting to other external tools.\n\n3.The GPT-4 API was released after the deadline of the submission phase, so we only used  GPT-3.5 in the experiment. We will provide the GPT-4 results in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The idea of LDM2 is quite reasonable. Intuitively, increasing the memory of the trajectories (candidate prompt components) helps get optimal prompts. However, larger is not always better. The main contribution of LDM2 lies in constructing a clustered memory and reward-based refinement, which dynamically updates the memory content towards a higher-reward state.\n\n- LDM2 constructs a tree exploration method to get all possible excellent state-action tuples updating the memory. In some cases, choosing which action are ambiguous due to the similarity of observation and historical information, so choosing multi-action is practical.", "reasons_to_reject": "- Exploring possible trajectories with tree structures helps find high-reward trajectories. However, if there are numerous actions to explore as well as in subnodes, it will be computation-consuming since the complexity is exponential.\n\n- In the Memory Formation phase, state-action tuples are clustered by LLM. But we all know that LLM is not fully accurate and fails to get the right answer. If LLM fails in the clustering phase, it may affect the continual process. So would it be better to substitute LLM with the traditional clustering method?\n\n- The experiments are all done with GPT-3.5, which is relatively weak. It is possible that with GPT4 and longer context windows, the proposed framework does not provide any noticeable improvement. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "Designing appropriate prompts for LLMs is difficult because of the lack of generalization in a complex environment. Merely imitating prompting examples, the traditional prompting method cannot adapt to situations where the environment content is distinctive from the examples.\n\nThis paper proposes establishing and maintaining a memory-based trajectory library that records state-action tuples and other information. The main idea is composed of two phrases: memory formation and memory refinement. In the memory formation phase, human trajectories are summarized into state-action tuples by LLMs and clustered with goal labels and observation labels. In the refinement phase, LDM2 explores the tree structure to retrieve the relevant information, and later LDM2 is finetuned by adding high-reward state-action tuples.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PuKfFd6s5A", "reviewer": "Reviewer_c7Lp", "summary": "", "questions": "1. What are the advantages of the proposed method compared to other memory-based frameworks for reinforce language agents, such as Reflexion [1]? Besides, it is necessary to add baselines about other state-of-the-art memory-based frameworks to demonstrate this. \n1. I was wondering will later valuable state-action tuples replace the prior ones, or is there a way to limit the size of the memory from being explosive? \n1. It seems that the chosen action depends on the history information of state-action tuples from the initial timestep. Is there a possibility of reducing the horizon of prompts for LLMs while maintaining the same level of performance to save the in-context learning budget?\n \n[1] Shinn, Noah, Beck Labash, and Ashwin Gopinath. \"Reflexion: an autonomous agent with dynamic memory and self-reflection.\" arXiv preprint arXiv:2303.11366 (2023). ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestions. \n\nActually, when the human data are insufficient, the performance will be affected, but our method could still work when there are only few trajectories (ten or less). In such cases, the existing methods just take these few examples as prompt which is fixed at each timestep, but our method splits these examples into state-action tuples based on the observation then constructs dynamic prompt in each timestep. \n\nThanks again for your valuable suggestion on the tree exploration. We consider adding an evaluation process in the tree exploration process which lets the LLMs agent evaluate whether the current exploration direction is worth continuing. If the current exploration is valueless, the agent will prune the subtree from there.  \n\nQ1: First, the stage is different. The memory of Reflexion and other existing methods mainly focus on recording the experience and reflection on the online exploration phase. But our method leverages human data to form an offline initial memory to make the LLM agent familiar with the task in advance and then refines the memory according to the online exploration. The initial memory is a crucial part of our method, it leverages human data to make the agent construct an initial world model. \nSecond, the information in the memory is different. The Reflexion and other existing methods record the past experience or reflections as the memory which are non-uniform or nonstandard, but our memory consists of standard state-action tuples (please refer to Section 3.2). which is easier to store, refine, and manipulate. Meanwhile, we cluster the standard data in the memory to form the index of data, which is beneficial for standard data operation (such as add, revise, and delete). Existing methods just directly add the nonstandard text into the memory and the resulting memory is hard to refine and retrieve. \nWe have cited the Reflexion in the related work. Reflexion is not a competitive approach, but it can be integrated into our method to further improve the performance. Reflexion uses self-reflect from the completed trajectory and repeats the same task again based on the reflection, which could be adapted to our method that leverage the self-reflection to make multiple decision trajectories on the same instance. We will conduct such experiment and report its results. \n\nQ2: In the current setting, we directly add the valuable state-action tuples into the corresponding sub-memory. We do not replace any data because they are different blocks of trajectories from different instances. In practice, if the initial memory is small, the exploration stage will produce a lot of valuable state-action tuples. However, when the memory size reaches a certain scale, the agent may accomplish most instances of certain task, and it can hardly find new valuable state-action tuples. If the human data is sufficient and the initial memory is large, the new valuable state-action tuples will be few and the memory size tends to be stable.\n\nQ3: The state-action tuples consist of goals, summarized history information, and the current observation. These three parts are equally important, the goal tells the agent the decision-making direction, the current observation teaches the agent how to act with similar observation, and the history information tells the agent what have done to get the current state. Different paths may reach the same state, but they are not the same state-action tuples. The history information could assist the agent finding the similar experience and avoiding wrong actions appeared in the past.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "In terms of novelty, this paper investigates an interesting problem, capitalizing on the remarkable summarization abilities of LLMs and dynamic prompting mechanisms to boost their decision-making performance. Unlike fixed or hand-crafted prompts, the proposed dynamic memory offers a natural and self-updating prompting approach, encouraging LLMs to improve their decisions based on environmental feedback. The experimental results demonstrate the effectiveness of the proposed method. \n \nIn terms of clarity, this paper is largely clear and concise. The problem setting and the ﻿two-stage memory mechanism are easy to follow. Moreover, the figures, in general, are clean and easy to understand. ", "reasons_to_reject": "As mentioned in the limitation section, I am also concerned about the applicability of the proposed method in other decision-making problems when the human trajectories or expert examples are insufficient. Besides, when the action space is larger, the tree exploration process could be time-consuming. Maybe some suggestions about how to integrate other tree search methods could help alleviate this issue.  ", "questions_to_authors": "1. What are the advantages of the proposed method compared to other memory-based frameworks for reinforce language agents, such as Reflexion [1]? Besides, it is necessary to add baselines about other state-of-the-art memory-based frameworks to demonstrate this. \n1. I was wondering will later valuable state-action tuples replace the prior ones, or is there a way to limit the size of the memory from being explosive? \n1. It seems that the chosen action depends on the history information of state-action tuples from the initial timestep. Is there a possibility of reducing the horizon of prompts for LLMs while maintaining the same level of performance to save the in-context learning budget?\n \n[1] Shinn, Noah, Beck Labash, and Ashwin Gopinath. \"Reflexion: an autonomous agent with dynamic memory and self-reflection.\" arXiv preprint arXiv:2303.11366 (2023). ", "paper_topic_and_main_contributions": "This paper proposes a dynamic memory mechanism to improve LLMs' decision-making with prompts. The proposed method consists of two stages: the memory formation stage, which stores human behaviors or expert trajectories as state-action tuples, and the memory refinement stage, which employs tree exploration to generate valuable decision processes and dynamically update the memory. The effectiveness of this approach is demonstrated in Webshop and ALFworld, where it is compared against standard prompting methods, ReAct and BUTLER.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are some grammatical errors and confusing sentences in this paper mainly because of inaccurate expressions. It is necessary for the authors to check the whole paper and correct them. Just some examples: \nLine 37: have->has \nLine 112: spaces->space \nLine 166: employed and Line 175: learned. Tenses in related work need to be unified. \nLine 200: it forms->which forms \nLine 220: the the-> the \nThe paragraph from Line 368 to Line 373 is confusing. \nLine 572: e->we? "}}]

Author rebuttal:

Meta-review: In complex test environments where the environment content is distinctive from the example environments, traditional prompting methods suffer from lack of generalization. 
This paper proposes establishing and maintaining a memory-based trajectory library that records state-action tuples and other information. The main idea is composed of two phrases: memory formation and memory refinement. In the memory formation phase, human trajectories are summarized into state-action tuples by LLMs and clustered with goal labels and observation labels. In the refinement phase, LDM2 explores the tree structure to retrieve the relevant information, and later LDM2 is finetuned by adding high-reward state-action tuples.

Pros:

Intuitive and interesting idea of using memory for dynamic prompting to boost decision making ability of LLMs
 
Unlike fixed or hand-crafted prompts, the proposed dynamic memory offers a natural and self-updating prompting approach, encouraging LLMs to improve their decisions based on environmental feedback.

The experimental results demonstrate the effectiveness of the proposed method, with a good improvement over baselines 

Cons:

In realistic applications of this, the computational cost can be quite high (as pointed out by most of the reviewers)


Again for many realistic applications, getting the expert human behavior trajectories might be quite difficult or  very expensive. Referring to the authors response on this (link), it is important to understand how effective the model would be if only a few example trajectories are provided for a test environment. Some more experimental analysis and ablations are needed to establish this. 


What are the advantages of the proposed method compared to other memory-based frameworks for reinforce language agents, such as Reflexion [1]? Besides, it is necessary to add baselines about other state-of-the-art memory-based frameworks to demonstrate this.
Decision: Accept-Findings