CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization
Authors: Bodhisattwa Prasad Majumder
Bhavana Dalvi Mishra
Peter Jansen
Oyvind Tafjord
Niket Tandon
Li Zhang
Chris Callison-Burch
Peter Clark
Venue: COLM (2024)
Keywords: Language Agents, Memory, Causal Abstraction, Continual Learning, Memory-augmented Agents, Task Adaptation, Text-based Simulator, Virtual Environment
TL;DR: We show language agents in continual learning setups can rapidly adapt and generalize to new tasks/environments by repeatedly refining causal abstractions of the world as a persistent memory without requiring parameter updates.

Abstract: Language agents have shown some ability to interact with an external environment, e.g., a virtual world such as ScienceWorld, to perform complex tasks, e.g.,  growing a plant, without the startup costs of reinforcement learning. While recent work, e.g., Reflexion, has demonstrated how such agents can also self-improve by adding a textual memory of ''hints'' learned from prior experience, such improvements have been limited both in size and scope. In contrast, our goal is a language agent that can robustly improve performance over time, including when both the task and environment are varied. Our approach is to have the agent learn a textual representation of how the world works (rather than just isolated hints), expressed as a memory of causal abstractions, to guide future decision-making. In experiments, we find CLIN is able to continually improve on repeated trials on the same task and environment, outperforming state-of-the-art reflective language agents like Reflexion by 23 points in ScienceWorld and 1.4 points in ALFWorld benchmarks. CLIN can also transfer its learning to new environments and tasks, enhancing performance by 21 points in ScienceWorld and 11 points in ALFWorld. This suggests that language agents with a textual causal memory can play a significant role in interactive environments, including being able to rapidly improve over time.

Published as a conference paper at COLM 2024
CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization
Bodhisattwa Prasad Majumder', Bhavana Dalvi Mishra',
Peter Jansen 1.2, Oyvind Tafjord', Niket Tandon', Li Zhang?, Chris-Callison Burch Peter Clark!
1 Allen Institute of AI
Universitty of Arizona University of Pennsylvania
Contact: {bodhisattwam, dhavanad)@alleen org
Project page: http://lllaai.ithuu.bb... co/ccin//n
Abstract
Language agents have shown some ability to interact with external envi- ronment, e-g., virtual world such as ScienceWorld, to perform complex tasks, e-g., growing plant, without the startup costs of reinforcement learn- ing. While recent work, e.g., Reflexion, has demonstrated how such agents also self-improve by adding textual memory of "hints" learned from prior experience, such improvements have been limited both in size and scope. In contrast, our goal is language agent that can robustly improve performance over time, including when both the task and environment are varied. Our approach is have the agent learn a textual representation of how the world works (rather than just isolated hints), expressed as a memory causal abstractions, to guide future decision-making In experi- ments, we find CLIN is able to continually improve on repeated trials on the same task and environment, outperforming language agents like Reflexion by 23 points in ScienceWorld and 1.4 points ALFWorld benchmarks. CLIN can also transfer its learning to new envi- ronments and tasks, enhancing performance by 21 points in ScienceWorld and 11 points in ALFWorld. This suggests that language agents with a tex- tual causal memory can play a significant role in interactive environments, including being able to rapidly improve over time.
crate-of-the-ar reflective
1 Introduction
Large language models (LLMs) have been increasingly used to interact with external vironments (e-g., simulated worlds) as goal-driven agents (Reed et al. 2022). However, has been challenging for these language agents efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expen- sive model fine-tuning (Chen et al. 2021 Ammanabrolu et al., 2020). More recently, new techniques have appeared in which an agent retlects on its own past experience solving task in particular environment, and generates language-based insights help retry the task, e.g., (Shinn et al.) 2023 Zhao et al., 2023). Such methods have the advantage of not requiring parameter updates (particularly with frozen LLMs). However, the style of such insights plays crucial role, and not all insights improve generalization performance. For example, a specific insight such as "I should go to desk 1 and find the lamp" (Shinn et al., 2023)
may have limited value (or even hurt) for different environment task.
Our goal is a system that will continually improve over time, both while attempting the same task in the same environment, and across different tasks and environments. Our approach builds on prior work on reflection in two ways: First, we conjecture that a specific style of insight will be useful, namely one that captures causal abstractions about agent's actions,

---

Published as a conference paper at COLM 2024
CREATION (Envl, Triall)
ADAPTATION (Envl, Trial2)
GENERALIZATION (for new Env3)
Task: Grow an orange
Task: Grow an orange
Goot: Find soods
Ooak: seeds
Memors:
Env1
Action:
Seing iichhen
Observation... (no seedsl...
find
THANT
THia?
Trinls
Action: to to the garden
to
Obear vation: (nn sewdla....
Observation you sne seeds
Actions the kitchen
Acsion: pick up seock...
Env2
Action: Pick up seeds
Observation: You see seeds
Action: more souds the pot... Trial completed successfully!
Gool: plant the seeds
Trial
Goal: Plant the seeds...
Triel failed!
MEMORY:
Going the kitchen
MEMORY:
MEMORY:
necessary to
Moving diffferen
Gorigg to may
Moving seeds the pot may
may be necessary to an abyact
necessary to find seads
nacassary for planting seads
Fig. 1: CLIN creates (Trial1) or adapts (Trial2+) a memory of causal abstractions help in future trials by reflecting on the last trial and current memory (Sec. 3.4.33..)A Adaptation). Here, reflecting on Triall, CLIN notes in memory that going to the kitchen helped with finding seeds, enabling it to find the seeds faster in Trial2. To further generalize in new environments, CLIN generates a summary ("meta- memory") of the best (starred) memories from each prior episode, here generating the generalization that moving to different rooms helps finding objects (Sec. 3.4 Generalization).
e.g., "opening doors may be necessary for movement between rooms". Causal abstractions can potentially help the agent decide which action to take in the future, and can be viewed as kind of action model learning (Arora et al. 2018), but placed in the modern context of language models. Second, we maintain these abstractions in a continually evolving, dynamic memory, which is regularly updated as the agent gains experience, allowing useful causal knowledge to persist (and unhelpful knowledge to be dropped) over time and between tasks and environments, as illustrated in Fig. [
We operationalize and evaluate this approach in a memory-augmented language agent called CLIN (continual learning from interactions). CLIN is an agent that operates in virtual, text-based environment (e.g. ScienceWorld (Wang et al. 2022), ALFWorld (Shridhar et al. 2021)) in which an agent is tasked with goals, e.g., boiling liquid or growing plant. We find that CLIN is able to rapidly learn about the environment and its action vocabulary and continually improve on repeated trials on the same task and environ- ment, outperforming state-of-the-art (SOTA) reflective language agents like Reflexion by 23 points in ScienceWorld. CLIN can also transfer its learning to new environments (or tasks), through continual memory updates and achieving 21 (20 for new tasks) points performance boost. Similarly, in ALFWorld, CLIN enhances its base performance by 11 points in unseen tasks/ environments. Our contributions are as follows:
We describe and evaluate CLIN, novel architecture for nonparametric agent learning. We show that using a dynamic, evolving memory over time, including over new tasks and environments, CLIN learns faster than the shorter-term memory approaches used in Reflexion and other memory-based agents, and with memories that generalizes better to new tasks and environments.
We show that a memory of causal abstractions (or "action models") helps agents learn over an extended period. While action models have been used in formal planning before, we are the first to apply this concept to language agents.
Overall, this work suggests that a dynamic memory, centered around causal knowledge, is promising way forward for agents built on frozen models to continually improve over time.
Related Work
Reinforcement Learning. There is a long literature of work on agents that can navigate complex environments. A common approach is to use reinforcement learning (RL), e.g., DRRN (He et al. 2015), KG-A2C (Ammanabrolu & Hausknecht 2020), CALM (aao et al.) 2020), where agents learn task over repeated trials. However, while effective, such agents typically require a large number of trials to learn and have trouble adapting to unexpected changes in the test environment. More recently, (Adaptive- Agent- Teant-TTaammmmmmmeeeee Team et al.
2

---

Published as a conference paper at COLM 2024
'Convert water into steam"
"Activating sink may necessary to obtain water, etc."
Task
CLIN
Simulator
BcienceWorid. ALFWorld)
Task
"Find the water" goal
"Go kitchen action
Memory
Controller
Executor
GAOGAO...GE GAO
trial so far (geal action observation-...
Memory Generator
Final trial sequence (GAO)* and final reward r,
End of trial (success/failuree
Fig. 2: In CLIN, controller takes the current task, memory, and the trial so far, to generate the next goal. The executor converts the goal to valid action. The simulator performs the action and returns an observation. Memory is updated at the end of each trial by the memory generator (Sec. 3.4
2023) demonstrated AdA, an agent that could rapidly adapt to open-ended novel 3D problems, using weta-reenforcement learning, essentially being able to change its policy the fly. However, AdA required vast amounts of pretraining, and was limited to the style of environments and problems seen in pretraining.
Language Agents. LLMs have provided new tool for building goal-directed agents Huang et al. 2022). Given linguistic description of the world, the LLM can be prompted to suggest the next actions to achieve goal requiring little training, e.g. SayCan (Ahn et al., 2022), ReAct (Yao et al., 2022), and SwiftSage (Lin al. 2023), which combines supervised agent and a deliberative agent together. However, while performing reasonably with little training data, such agents are unable to learn and adapt from experience.
Reflection and Memory. More recently, several language agents for interactive environ- ments have been developed that can reflect on their own performance, and create non- parameteric memory of language-based insights to help in new tasks via prompting (Brown) et al. 2020 Voyager (Wang et al. 2023) operates in the world of Minecraft, growing (code- based) skill library from rich leedback about the causes of its failures. Expel. (Zhao et al., 2023) acquired "insights" by reflecting on traces of its behavior (trajectories) in a training phrase, for later use, but did not operate in a continuous fashion. (Chen et al., 2023) learned a memory helpful "tips" by reflecting on prior trajectories, including comparing (human) expert and system trajectories. Rememberer (Zhang et al., 2023) learned "encouraged" and "discouraged" actions from prior experience, but did not generalize explicit reasons for those preferences. Finally, Reflexion (Shinn al. 2023 improved at task by reflecting on failed attempt, then learning specific retlections to help when retrying that same task (but not beyond). Our work is inspired by and goes beyond these, to create system that learns in a continually evolving setting, including across changing environments and tasks, using a memory of causal rules about how the world works.
expertand
More generally, others have found that a memory of useful learnings can be used to improve frozen LLM behavior, e.g., in QA (Dalvi et al. 2022 Tandon al. 2022 Madaan et al. 22023), or for modeling social behavior (Park et al. 2023). apply this to goal-directed agents.
Finally, we note that the content of experiential memory is also important. Specifically, CLIN learns a memory of causal abstractions, which can be seen as learning linguistic form of action model, descríbing the causal effects of actions. While there has been work in the planning community of learning action models in formal context (Arora et al. 2018 Aineto et al. 2018), CLIN loosely applies this idea the linguistic world of LLM agents.
3 Approach
3.1 Acting in the World
We follow the normal formalization for an agent performing actions in a partially observable environment, but add memory S as additional input for decision making. The memory
3

---

Published as a conference paper at COLM 2024
contains learned task/ enkk//omvion knowledge to help the agent make better decisions in the next trial, and is updated at the end of each trial (described shortly). At each time step f, given task m (e.g., "grow orange"), memory S, and the history of actions so far, the agent decides on its next goal g and action a in pursuit of that goal. In response, the environment returns the result of executing in the form of observation and reward This repeats until an end state is reached (such as completing, failing, or timing out). Thus at each step f, the history so far is T<t = {gi a; Higge and the agent's decision-making task at each time step can be described as:
m e + Ts+
81+1 + At+1
(1)
We describe the implementation of this shortly. The full sequence of steps Tend to end- state is called a trial. If the same task in is attempted in the same environment K times (resetting the environment each time), we call the collection of the K trials episode.
3.2 Continual Learning
The agent's goal is to maximize its reward (e.g,, completing the task) on new trials. Learning occurs by updating the dynamic memory S between (not during) trials, using memory update function that takes memories from old trials as input, and generates a new memory for the next trial. Note that the memory does not grow monotonically; it may drop previous memory items and add new ones. Also note it is not perfect; some memory items may be erroneous, and ideally be dropped or modified in subsequent iterations. Learning is continuous in the sense that each new memory generated from an ever-growing collection of previous memories. We define two classes of learning:
1. Within-episode learning ("adaptation") same task mı and environment e. After each trial k, a new memory Sx+1 is generated from the most recent trial history Te and final reward Tk, and memories from prior trials:
T + + { + Sk+1
(2)
Sk 11 is then used to retry the same task m, next trial.
2. Cross-episode learning ("generalization") new task Mnew or environment enew- Given new task or environment, an initial starting memory is generated using memories from other episodes. Specifically, we select the "best" (defined later) memory from each prior episode as inputs, and generate new memory for use in this unexplored task/ environment as output:
nnew + Cnew + { (Sbestr Fh: prior-episodes + Snew
(3)
Because this new memory generalizes prior memories, we also refer to it as a "meta- memory". Note that some generalizationsin Snew may be overly specific wrong. However, we see both net initial benefit in using Snew, and further task improvement in subsequent trials as Snew is refined (adaptation), described later in Sec.
3.3 CLIN: Agent Architecture
Our implementation, called CLIN, comprises three components for acting: the memory, controller, and an executor. Learning then occurs using a fourth module, a memory generator, generate an updated memory after each trial. These are illustrated in Fig. 2
Memory. CLIN's memory (S) is persistent, dynamic collection of NL sentences express- ing CLIN's current understanding of actions and their effects. Specifically, each sentence expresses a causal abstraction between actions, e.g., "opening the fridge is necessary to access apple juice", as well as negative learnings, e.g., "moving to another room does not contri bute to freezing mercury.". Such statements are learned from past experiences (described shortly). Their role is to help CLIN make better action choices (Eqn. [T.. Causal abstractions constitute CLIN's current understanding of the way the world behaves, and can be viewed as a modern version of action models used in formal planning, describing the effects of actions in the world (Arora et al. 2018).
4

---

Published as a conference paper at COLM 2024
Algorithm 1 CLIN
X (actions)
relation
(actions)
1:
procedure ADAPT(lask: m, Env: e, Memory: S):
Mamery off
Using the lighter
may be
to heat the
2: Initialize Memory: So
3: for k 1,...,Ka do:
ADAPTATION
4: Intialize Trial T
5: while < max. steps or task not complete do:
Uaing the lighter on metel p.oe.
shoulld be necessary
water the pott
the
6:
Controller (m, e, Tet, St-1)
and triat
boiling Devikinn
contribute
7:
Executor (gt, admissible actions)
8:
0; Simulator (Tet, at)
Tet+1 = Teı (gt, A,, 0t, Yt)
best
on the metal port
Solneer the pirtt
10:
Final reward n. =
11:
Sy = memory -geeerttr (1Sck), Te)
hwoz
the slove
necesssary
12:
end procedure
13:
procedure GHNERALIZH(Tas:: m, Env: e, past m² /é)
14:
15:
Smeta = meta-memors ( ((Serucill Fk), !II
(Serucia, = crucial -eercial-meoories (past m' /e))
Meda memory
Sourco 1stove fighncd contiareet
should be inod eSGARY
tO heat
16:
ADAPTATION(M, e, Smeta)
17:
end procedure
Fig. 3: (LEFT) CLIN'S continual learning algorithm. (RIGHT) Example causal abstractions.
Controller. each time step in a trial, the controller generates the next goal to pursue in service of the overall task m. In CLIN the controller is frozen LLM, whose prompt includes the current task m,, e-g,, "convert water into steam", selected statements from the current memory 8 (a list of sentences), and the trial so far (the sequence of goal-action-observatioo triples. It is prompted to output the next goal g1+1 to pursue, e.g., "find water". Note we only use selected statements from memory (rather than the whole memory), in order to avoid irrelevant knowledge distracting the controller. Selection is itself done with a separate query to the LLM, prompting it to list relevant memory items, with the full memory S and the task included in that prompt.
Executor. The role of the executor is convert the generated goal 8t 11 into a valid action 41+1 that can be executed in the environment in pursuit of that goal. In other words, it serves to map goals into the specific action space of the environment. Again a (frozen) LLM is used, whose prompt includes the goal 81+1 (from the controller, above), the trial so far, and all the possible actions that can be performed in the current state (provided by the environment, as standard practice in current generative agent research (Ahn et al., 2022 Yao et al. 2022
Lin et al. 2023 Park et al. 2023)). The list of possible actions is expressed as possible action templates and available objects that can instantiate them, rather than combinatorially
large enumeration of possible actions. The model is then prompted to generate a candidate action perform (see prompt in Fig. 6). Finally, CLIN checks this candidate action is one of the valid actions. If it is not, it finds the most similar valid action using the pre-trained embeddings from the settennce-trnfformme model (Reimers & Gurevych 2019). If the top-ranked valid action has similarity score greater than a threshold (here, 0.9, chosen as a hyperparameter) the action selected. Otherwise, we perform iterative refinement (Madaan et al. 2023) by suffixing the context with feedback stating the candidate action is not executable. This allows the executor to retry generation up to max number (here, 5).
Finally, upon executing the action 4,+1, CLIN receives partial next state, observation, from the environment and the reward (r) E [0,1], provided by the environment. A snapshot of full trial is given in lines 4-10 in Alg.
Note that CLIN does not make use of any gold data to identify goals and memories. Rather, expect CLIN to perform a balanced act of exeeoooationneeplloiaee
by interacting, learning, and adapting to unseen tasks or environments a key difference from few-shot generative
agents (Ahn et al. 2022 Yao et al. 2022 Lin et al. 2023 Park et al. 2023).
5

---

Published as a conference paper at COLM 2024
3.4 Continual Learning in CLIN
At the end of each trial (completion or failure), CLIN uses memory generator to create or update its memory. The memory generator (frozen) LLM prompted to reflect on the current trial and memory, and generate new memory of insights in the form of (English sentences expressing) useful causal abstractions.
To make the LLM generate causal abstractions, we special instructions in the prompt that ask the LLM to generate insights in particular templated syntax (see prompt in use
Fig. 7). To capture actions enabling desired changes and helpful state transitions, the template "x is NECESSARY to Y", and to capture contrastive examples of unsuitable actions and state transitions, we employ "X DOES NOT CONTRIBUTE to Y" (Fig. 3), where x, Y are related to actions. These abstractions are functionally analogous to hindsight al. 2017), obtained from CLIN'S past self-explorations. To allow the LLM to express uncertainty, we encourage it to use modifiers: "X may
experience replay (Andrychowiez,
for
moderate to high uncertainty, and "X should ... for low uncertainty (See Fig.
As described earlier, there are two kinds of memory update needed: (a) re-generating the memory when retrying the same task in the same environment (" ("adaptation") (b) re- generating the memory for a new task/ eekik/nmvirneettttt ("generalization") as we now describe.
Within-episode learning ("adaptation"), ") To update the memory after each trial within an episode (Eqn. 2 the memory generator is prompted with the most recent trial (a sequence of (gt, 4r, 0,) tuples and the final reward r ), and the memories from the three most recent trials SS
is then prompted to generate an updated memory S namely new list of semi-structured causal abstractions in the forms described above, for use in the next trial. Although we do not specify a maximum size for the memory, we observe that size of the generated memory (i.e., the number of causal abstractions generated) is far less than the number of actions executed in the trial, indicating the semory-geneeator additionally performs a saliency-based pruning to keep only important insights based on the success of the trial (final reward r2 for trial TTx)
Cross-episode learning (" ("generalization") Given a new task Iinew or environment enew, the memory generator is prompted to generate a suitable memory, generalizing from the best memories from previous tasks/ eevironmennss and suitable for this new situation (Eqn. a form of meta-learning (Hospedales et al. 2020).. Following the prioritized level replay scheme (Jiang et al., ), we choose the most successful trial per episode (based on the reward ri.) and retrieve memories abstracted from those trials with fixed archive of size 10, a hyperparameter. the environment is new, the prompt instructs the LLM to generate memory helpful "to solve the same task in new environment configuration", given the new task description. The prompt is designed to encourage the LLM to generate generic causal insights about the task, not tied to specific environmental details (Fig. g) Similarly, if the task is new, the prompt is modified accordingly (Fig.
Experimental Setup
Test-time adaptation and generalization via continual learning require a variety of complex tasks and environment configurations to allow agent to explore, learn latent causal insights from interactions, and exploit them in the future. We evaluate CLIN'S performance in two benchmarks: ScienceWorld (Wang et al. 2022) and ALFWorld (Shridhar et al., 2021). Both benchmarks consist of a text-based interactive environment requiring complex interactive reasoning processes to solve plethora of tasks. ScienceWorld focuses science- based tasks 2 spanning 18 task classes (e.g., thermodynamics, genetics, friction, etc.) with several variations (total 164). ALFWorld has 6 categories of tasks: Pick, Clean, Heat, Cool, Look, and Pick-two-items, with a total of 134 environments.
Thee reward is converted to NI. feedback using 7 simple rules, e-g., "if score >= and score 20 then leedback = "The agent performed poorly and made some progress but not enough to solve the task." " Worie tasks are grouped into Short (S), e.g., pick & place and Long (I.), e.g, grow plant, based on the # of gold actions.
6

---

Published as a conference paper at COLM 2024
100
80
ALForrld
ScienceWordd
ALFWorkd
BASE
Pellotoon
BAGE
ADAPT
80
72
86
Part SOTA
8133
60
Jug Roward
54
Tiak sutzom (K))
12
40
30
Pick: ADAPT
Pick: Reflexion
Plck2: ADAPT
Pick2: Rellexion
2
Tridk
Fig. 4: Rapid task adaptation with CLIN. (a) Example tasks with CLIN'S adaptation. For CLIN, Trial-O is BASE, Trial-4 is ADAPT. Comparison of CLIN with Reflexion Shinn et al. 2023) in (b) ScienceWorld and (c) ALFWorld. (More in Appendix C
ALFWorld is used in several previous approaches (Yao et al. 2022 Shinn et al. 2023). However, most tasks ALFWorld required small number of actions and limited atomic skills e.g,, pick, place, clean, etc. Our aim is show continual improvement in long and complex tasks. ScienceWorld entails tasks that require a median 37 steps, and each task can be tested in variety of different environments, each with its unique objects/ affordances.
CLIN-BASE. The standard base agent in previous work uses a few-shot ReAct prompt (aao et al. 2022), which also applies Reflexion. However, both in Science and ALFWorld, gold trajectories from similar tasks overlap more than 80% with the target task trajectory. To avoid reliance on these demonstration examples, in CLIN, we use zero-shot controller prompt (see in Fig. 6) as CLIN-BASE. We observe CLIN-BASE, powered by GPT-4 (Achiam al. 2023), superior on ScienceWorld (Fig. Figure and similarly performant on AlfWorld (Fig. 4b) than the one-shot ReAct agent. In our experiments, we use CLIN-BASH as the base agent for within- and cross-episode learning.
Results and Analysis
A. How well does CLIN perform within--epssode learning ("adaptation")?
We refer to this setup ADAPT. Here, we focus CLIN's ability to adapt to task by attempting it for several trials in the same environment configuration. Most importantly, CLIN initializes with empty memory at the beginning of the first trial and generates memory at the end of each trial. While the environment gets reset at the trial boundary, CLIN'S memory continues to be updated, capturing informative causal abstractions pertain- ing to both successful and failed actions. We compare with Reflexion (Shinn et al. 2023), representative SOTA memory-baset system that also reflects and learns from experience.
Results are shown in Fig, Tb (ScienceWorld) and Fig. 4 (ALFWorld), In both domains, after the trials, we see CLIN-ADAPT improves over time (+13.6 pts (all), compared its starting performance BASE), and also that the improvement is greater than that for Reflexion (+9.8 pts compared Reflexion's 's base performance, namely the non-memory ReAct). Fig. 4h also shows the learning curves for two tasks in ALFWorld. Again, see both Reflexion and CLIN-ADAPT improve over time, but again with greater improvement for CLIN-ADAPT (+16.7% / for Pick, +23.6% for Pick2) than Reflexion (+16.7%, +5.9% respectively).
B. How well does CLIN generalize for fixed task, across different environments?
In this setup, called (GEN-ENV), for task m, we CLIN for 10 different (train) environ- ment settings (with varying objects and starting locations) and then create meta-memories from its exploration to solve the same task in unseen (test) environment. If CLIN, in GEN-ENV does not successfully complete the task in the new environment, can continue learning and retrying that task. We refer to this setup as GEN-ADAPT(G+AA)
Here, we compare CLIN with RL methods DRRN (He et al. 2015), KG-A2C (Ammanabrolu & Hausknecht, 2020), and CALM (Yao et al. 2020) trained on all (large) training variations
7

---

Published as a conference paper at COLM 2024
RL Methods
Generative Language Agents
CLIN (ours)
Type
DRRN
KGA2C
CALM
SayCan
ReAct
Reflexion
BASE
GEN-ENV
G+A
S
22.1
19.1
5.2
36.5
37.6
49.9
54.7
58.3
71.0
11.2
6.2
1.9
29.2
21.5
28.9
42.5
47.1
68.0
16.7
3.6
32.9
29.6
39.4
48.6
52.7
Table 1: Generalization across unseen environments in ScienceWorld. Task-wise results in Table
Method
Pick
Clean
Heat
Cool
Look
Pick2
All
ReAct Reflexion
58.3
71.0
87.0
81.0
94.4
41.2
72.4
75,0
74.2
91.3
90.5
100.0
47.1
79.9
CLIN (ours)
GEN-ENV+A
83.3
77.4
87.0
95.2
100.0
58.8
83.6
GEN-TASK+A
79.2
74.2
91.3
90.5
100.0
64.7
82.8
Table 2: Generalization across unseen environments and tasks in ALFWorld
with simulator reward and Generative Language agents, SayCan (Ahn et al. 2022), ReAct (Yao et al. 2022), and Reflexion (Shinn et al. 2023), prompted with few-shot demonstrations.
Table compares CLIN with baselines that learn from training environmental variants for a task to improve its performance in a novel environment 3 Language agents (including CLin) that use NL feedback from the ScienceWorld (e.g,, "Door to the kitchen is closed")
perform significantly better compared to RL methods that purely rely on (sparse) numeric
rewards from the environment learn a policy.
We also observe that CLIN's meta-memory is overall advantageous, helping CLIN with a stronger start on unseen environ- ments (52.7, GEN-ENV all) compared with memory (BASE, 48.6). Note that advan- tage is not guaranteed, as for some tasks, the memories may be too specific and hurt, but there an overall net benefit. This gain persists through continued learning in these new environments, in which the meta-memory is refined, resulting in overall net gain of 20.9% (all) over base per- formance, substantially higher than for Re- flexion (+9.8% (all) compared with its no- learning version ReAct). Fig. 5 plots this improvement over time and also compares to improvement without meta-memory. showing persistent net improvement.
CLIN GEN-ENV
100
1.0
80
G+
0.8
60
0.6
ADAFT
G-A
40
0.4
20
Avg reward: ADAPT
0.2
Aug #steps: ADAPT
Aug reward:
Aug #steps:
0
0.0
0
100
ADAPT
80
GEN
60
40
0.4
20
Avg
0.2
Aug #steps: ADAPT
Aug reward:
Avg #steps:
0
2
3
0.0
2
Traiss
Trials
Similarly, for the ALFWorld, in the GEN- ENV setting, CLIN uses memories learned from earlier (different) environments. Ta- ble shows that CLIN generalizes better than Reflexion to new environments, namely from its base performance (same as ReAct) to 83.6% (+11.2%), +7.5% (Reflexion). The gains are likely smaller here because ALFworld is substantially simpler than ScienceWorke
CLIN in GEN-TASK
Fig. 5: Reward and #steps trends in (a) GEN-ENV
and (b) GEN-TASK for SeienceWorld.
C. How well does CLIN generalize across different tasks, in fixed environment?
This setup is referred to as GEN-TASK. For an environment e, we run CLIN to solve a task and then condense its learning to solve a novel task mn' in the environment e. If CLIN does not successfully complete the new task, it can continue learning and retrying that task. We refer to this setup GEN-ADAPT(G+AA We took all test examples where we have different task defined in the same environment configuration. (Adaptive-Agent -eam et al. 2023) suggests that transferring learning from a random task can be very hard; hence we
Baseline numbers are derived from Table in [Lin al. 2023
8

---

Published as a conference paper at COLM 2024
couple tasks that are related (revolve around overlapping task-critical objects/ pooccts/locatiin llcations such water, kitchen), such as boil and freeze to measure transfer learning from one to the other. This is novel setup where we do not have any off-the-shelf baselines. However, here, we compare against CLIN-BASE, a strong baseline.
Mirroring trends from GEN-ENV, CLIN demonstrates strong transfer learning to new tasks (Fig- 55) with +13% improvement over its BASE performance, being better at 38.8% of datapoints. The improvement comes from critical learnings about the environment (e.g., "apple juice is in the fridge", required for both boiling and freezing it), leading to improvement in previously low-performing tasks in both ADAPT and GEN-ENV setups.
For the ALFWorld benchmark, in the GEN-TASK setting, CLIN had access to memories from other tasks of same type. Fig. 2 showws that CLIN can generalize its learnings across tasks to improve its success rate by 10.4% % from its base performance (here same as ReAct).
D. Does CLIN become more efficient during generalization?
Yes. Fig. (b) shows that the transfer learning in GEN-ENv, GEN-TASK and G+A helps CL.N to solve the tasks with fewer @ achieve higher rewards.
E. Does a memory of causal abstractions help more than free-form advice ("hints")?
Yes. CLIN extracts causal abstractions structured around 'necessary' and 'does not contribute' rela- tions. To ablate this, we removed this constraint on generation and instead prompted CLIN to gener- ate free-form advice for future trials. We observed CLIN then no longer generated causal knowledge but instead generated helpful hints (see Fig. 13 In ScinnceWorld, the average reward drops by pts (in 10% cases than CLIN), and in ALFWorld, the success rate drops by 1.4% when using the unstructured memory, indicating the usefulness of causal abstractions, as shown in Table 3
CLIN in
Aavg score (L)
%ep.
drop. (1)
ScienceWorld ALFWorld
6.2
10.0
1.4
Table Ablation for causal memory
F. Do causal abstractions capture task- environment-specife information?
Causal abstractions in CLIN capture all necessary information about the world required as needed-1) environment: "Searching on sofa should be necessary to find the second keychain" " 2) task: "picking the keychain should be necessary to put it on the table", and 3) explorative sub-goals that not achieved yet: "Checking other locations like drawers and shelves may be necessary to finding the second CD."
G. How does CLIN recover from unhelpful memories?
While the final performance with memory is indicative of their ef- fectiveness, we performed addi- tional human evaluation of gen- erated memory insights to evalu- ate their correctness across trials.
ScienceWorld
ALFWorld
Insights
GEN-ENV
G+A
GEN-TASK
G+A
GEN
G+A
Total Corrcct
100
105
98
107
94
92
72.0%
91.4%
73.9%
91.1%
72.3%
90.1%
Table 4: Memory correctness for CL.N
For generalization setups, we ran-
domly 10 bask-environment combinations to evaluate the correctness of memories used in them, notably the meta-memory used for trial 0 (GEN) and memory adapted for the best trial (GEN-ADAPT) Two annotators rated the insights (cohen's к = 0.78) for correctness with reference to gold trajectories. Table 4 shows that some mema-ie may not be applicable initially; however, with adaptation, in later trials, the correctness of the memory insights significantly improves, leading to direct increase in task performance. However, CLIN is unlikely to generate an insight related to an unobserved activity, which could result in less useful insights and lower future performance in some cases. curiosity-driven agent could mitigate this issue (Burda et al. 2019), which we leave as future work.
4y steps in Fig. (b) are normalized between 0-1, being maximum #steps allowed for task.

---

Published as a conference paper at COLM 2024
6 Conclusion
Our goal is a system that can continually improve over time, both while rapidly adapting to a task by multiple retries and efficiently generalizing to novel tasks and environments. We propose CLIN, an architecture for language agents that constructs a persistent, dynamic memory of causal abstractions, refines it over time, and uses it to improve its performance on future tasks, achieving state-of-the-ane performance. Our evaluation suggests that this architecture is effective, and we hope can thus help future development of language agents.
Acknowledgement We sincerely thank Aristo team members Tushar Khot, Ashish Sabhar- wal, Shashank Gupta, Nathaniel Weir, Kyle Richardson, Jiangjie Chen, Archiki Prasad, and other members such as Faeze Brahman, Alexander Koller at the Allen Institute for Al for their generous feedback.
References
OpenAl Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-
Christo- pher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, An- drew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Benjamin Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Sim on Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Raphael Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Ryan Kiros, Matthew Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Adeola Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel P. Mossing, Tong Mu, Mira Murati, Oleg Murk, David ely, Ashvin Nair, Reichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Ouyang Long, Cullen O'Keefe, Jakub W. Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alexandre Passos, Mikhail Pavlov, Andrew Peng, Adam Perel- man, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Pondé de Oliveira Pinto, Michael Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario D. Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin D. Sokolowsky, Yang Song, Natalie Stau- dacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas A. Tezak,
10

---

Published as a conference paper at COLM 2024
Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cer'on on Uribe, Andrea Vallone, Arun Vijayvergyya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. GPT-4 technical report. 2023. URL https://api. wwt.pppsssmaaanccchsessseeeeeeee
org :
Adaptive-Agent-Tee
Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal M. P. Behba- hani, Avishkar Bhoopchand, Nathalie Bradley-Schmigg Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks-Thompson, Hannah Openshaw, Jack Parker-Holder, Shreyaan Pathak, Nicolas Perez Nieves, Nemanja Rakicevic, Tim Rocktäschel, Yannick Schroecker, Jakub Sygnowski, Karl Tuyls, Sarah York, Alexander Zacherl, and Lei M. Zhang. Human-timescale adaptation in an open-ended task space. In International Conference on Machine Learning, 2023. URL httpp://aapp www.nticscooollaaa..... org/ CorpusID:2559982774
Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Jayant Joshi, Ryan C. Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego M Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, F. Xia, Ted Xiao, Peng Xu, Sichun Xu, and Mengyuan Yan. Do as i can, not as say: Grounding language in robotic affordances. In Conference on Robot Learning, 2022. URL https //api.semanticachan
Diego Aineto, Sergio Jiménez, and Eva Onaindía. Learning strips action models with classical planning. In International Conference on Automated Planning and Scheduling, 2018. /atpss///p eeemanticschollr
:rrg/CorpusID:4400 49405569
Prithviraj Ammanabrolu and Matthew J. Hausknecht. Graph constrained reinforcement learning for natural language action spaces. In ICLR, 2020.
Prithviraj Ammanabrolu, Jack Urbanek, Margaret Li, Arthur Szlam, Tim Rocktaschel, and Jason Weston. How to motivate your dragon: Teaching goal-driven agents to speak and act in fantasy worlds. In North American Chapter of the Association for Computational Linguistics, 2020. URL /ttpss:/appii www.ssemmmcccooo..mo.m wwwpsechholaarroroooreeeeeeeeeee
2221253011
Marcin Andrychowicz, Dwight Crow, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welin- der, Bob McGrew, Joshua Tobin, P. Abbeel, and Wojeiech Zaremba. Hindsight experi- ence replay. ArXiv, 1ess/777700145, 2017. URL httpp:///pp www.etiiiccooll..ooo.. org/ (oorpusDD:353200:
Ankuj Arora, Humbert Fiorino, Damien Pellier, Marc Métivier, and Sylvie Pesty. A review of learning planning action models. The Knowledge Engineering Review, 33, 2018. URL
https://api http://///pp..mmatiiiceemmo wee.mtiicccooll...... httpa/rrrrrrrooooou1055553232330
Tom B. Brown et al. Language models are few-shot learners. In NeurIPS, 2020.
Yuri Burda, Harrison Edwards, Deepak Pathak, Amos J. Storkey, Trevor Darrell, and Alexei A. Efros. Large-scale study of curiosity-driven learning. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.nee, 2019. URL htt.aatpps///ppeeriiee.eeeeee
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, P. Abbeel, A. Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Neural Information Processing Systems, 2021. URL https
//api.ssmantiisccoll.. htt.://ccc......oo.ooopoaaaaaaa.
11

---

Published as a conference paper at COLM 2024
Liting Chen, Lu Wang, Hang Dong, Yali Du, Jie Yan, Fan Yang, Shuang Li, Pu Zhao, Si Qin, S. Rajmohan, Qingwei Lin, and Dongmei Zhang. Introspective tips: Large language model for in-context decision making. ArXiv, abs abs/2305.11598, 2023, URL
http:s///ap...mmmtttccccco..... wwwpsccholaa.ooooooooupreee 258823336
https://appl www.sssestntcoool.a......o
Bhavana Dalvi, Oyvind Tafjord, and Peter Clark. Towards teachable reasoning systems: Using dynamic memory of user feedback for continual system improvement. In EMNLP, 2022.
Ji He, Jianshu Chen, Xiaodong Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. Deep reinforcement learning with a natural language action space. arXiv: Artificial Intelligence, 2015. URL URtpttps::// httpss:/app...... www.nticscoolaattt..oooom wwwpsschhlaarrro/orrrreeeeee wwrg/CorppusID. 15986631
Timothy M. Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J. Storkey. Meta- learning in neural networks: A survey. IEEE Transactions Pattern Analysis and Machine Intelligence, 44:5149-5169, 2020. URL
http://app..emmmtccccccc.ocooooe org/
CorpusID:215744833
Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. Language mod- els as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pp. 9118-9147. PMLR, 2022.
Minqi Jiang, Michael Dennis, Jack Parker-Holder, Jakob Nicolaus Foerster, Edward Grefen- stette, and Tim Rocktaschel. Replay-guided adversarial environment design. In Neural Information Processing Systems, 2021. URL https://api. stt.pp..ssaacicccolla...
238408352
Bill Yuchen Lin, Yicheng Fu, Karina Yang, Prithviraj Ammanabrolu, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. ArXiv, abt.abb5...3900 2023.
URL https://api wemanticscholar www./orr///oriiii 258960143
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegr- effe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Sean Welleck, Bod- hisattwa Prasad Majumder, Shashank Gupta, Amir Yazdanbakhsh, and Peter Clark. Self-refine:
Iterative refinement with self-feedback. ArXiv, abs//303.17651, 2023. URL org htw.://aa..rrrrrruusiDDD::--../
Joon Sung Park, Joseph C O'Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2204.03442 2023.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley D. Edwards, Nicolas Man- fred Otto Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Trans. Mach. Learn. Res., 2022, 2022. URL semanticscholar.
sttp:://app.semmatiiccccoooo... ottp:aaiiiiiiioooooooooooooooooa II:
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese
bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https
//arxiv. org/ abs/ 10084
Noah Shinn, Beck Labash, and Ashwin Gopinath. Reflexion: an autonomous agent with dynamic memory and self-reflection. arXiv preprint arXiv:2303.11366 2023.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning. In Proceedings the International Conference on Learning
Representations (ICLR), URL https://aaxii:i wttp://a////0..03///66/3///
12

---

Published as a conference paper at COLM 2024
Niket Tandon, Aman Madaan, Peter Clark, and Yiming Yang. Memory-assisted prompt editing to improve GPT-3 after deployment. In ACL Workshop on Commonsense Representation and Reasoning (CSRR'22), 2022. (also arxiv:2001060009,
Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi (Jim) Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. ArXiv, abs/2305.16291, 2023. URL https://api semanticscholar. org/ wrtp/oorrpssi:::26887333333333
Ruoyao Wang, Peter Alexander Jansen, Mlex-Alex Côté, and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than 5th grader? In Conference Empirical Methods in Natural Language Processing, 2022. URL https:///pp.. htt.:t///app..ssaaattccoooooee semanticscholar
org/CorpusID : 247451124
Shunyu Yao, Rohan Rao, Matthew J. Hausknecht, and Karthik Narasimhan. Keep calm and explore: Language models for action generation in text-based games. ArXiv, abs/2010.02903, 2020. URL https://p//pp... http:/ppp..emmnn.ccccccoaar....r
:2211442:22 222142129
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. ArXiv, (bs/2210.03669, 2022.
semantccscoola.. wwwccholaa..og/ooopppsseee org//orppusDD:222772335
Danyang Zhang, Lu Chen, Situo Zhang, Hongshen Xu, Zihan Zhao, and Kai Yu. Large language models are semi-parametric reinforcement learning agents. In Neural
Information Processing Systems, 2023. URL https://api. www.ntiischollla............. org/
CorpusID:259145011
Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Liu, and Gao Huang. Expel: Llm agents are experiential learners. ArXiv, 0bbs/2308.10144 2023. URL https://apt
semanticscholar org/CorpusID:2810055
A
Appendix
A
CLIN prompts
Figures 6 to 9 are the complete prompts for next-action generation (controller + executor), memory-generator during ADAPT, GEN-ENV, and GEN-TASK.
B Example Memories
Example generated memory for ADAPT, GEN-ENV, and GEN-TASKsetups in Figures 10 to
12
C More results
Full results for CLIN outperforming Reflexion is in Table 5 For the ScienceWorld bench- mark, we exclude electricity tasks since they deviate from standard electrical conventions, prohibiting us from fairly using LLM agents. We choose the first 10 test variants for each 18 tasks selected. The full list of 18 tasks from the benchmark, with the number of test variants used in parentheses:
grow-plant (10), identify-liff-stagee (5), grow-fruit (10), messure-meltiigg-pin---noon substance (10), wewdeliinnggeeeiiiinnnnnnoneee
(10), cenmmisttyymiixxpiintteooooo color (9), freeze (9), lifespan-longest-lieed (10), saalnnedplaae--deeemmn-----e (10), boil (9), use-thermometer (10), mhimistry-mix (8), lifespan-shortestllive (10), find-plant (10), find- living-thing (10),
(4), sunudllianngeeeiis-kkoowwn (10), inclined-
taneeffritiin--mme---uuu (10).
13

---

Published as a conference paper at COLM 2024
Short tasks have oracle lengths less than 37 steps (median), and Long tasks have oracle lengths more than equal to 37 steps.
The map to the short names used for tasks in the paper:
Temp: use-thermometer,
Pick&Placee: find-plant, find-living-thing -tiing: Chemistry: chemistry-mix, seeeiistymmii-pintteeoodaaa
Lifes-
pan: lifespan-longest-lieell lifespan-shortest-liell Biology: identifyliff-stages-l iidettgggg--i,eiiff... life-stages-2, Boil; Freeze; Grow Plant, Grow Fruit; Force: padlnet-paae--deermin---gge
Friction: wtw.pppaaaaeiiiioonnee------eree plant,
Genetics: mendellan-geneticc-kooon
[System]: : You are an AI agent helping execute a science experiment in simulated environment with limited number objects and actions available at each step.
[User]:
Possible objects value OBJ take ): {objects_str)
Your next action should be in one of the following formats: Possible actions: (actions_str)
If say "Ambiiguuus request\", your action might multiple things. In that case, respond with the number corresponding to the action you want to take.
What action would you like to do next?
First, scan the (unordered) list of learnings, provided. Decide any the learnings are applicable given the last observation to make progress in this task. Then only selected learnings, any, construct rationale for picking the next action. If no Learning is selected, construct the rationale based on the last observation. Format your response follows:
Write 'I used learning id(s): ! as a comma separated list; the list can be empty if no learnings selected. Then, write $$3 followed by the rationale. Finally, write followed by the single next action you would like to take.
If you think you have completed the task, please write TASK_COMPLETT the next action.
If the task requires you to 'focus' on something (OBJ), please write FOCUS ON <OBJ> as the next action. FOCUS extremely critical action that can be only used the number times 'focus' mentioned in the task description. Using more than that inappropiately (such as wrong object) will terminate the session and the task will be rendered incomplete.
If you performed an action that requires waiting to see the effect, please write 'wait' as the next action.
Fig. Prompt for the Controller and the Executor
Superior BASE performance. Figure44 depicts a superior BASE performance for CLIN than the final performance of both ReAct and Reflexion despite using the same underlying
14

---

Published as a conference paper at COLM 2024
[System] You are an expert assistant.
[User]:
You given CURRENT TRACE, sequence of actions that agent made in world to accomplish task.
Task is detailed at the beginning.
For each action, there is a rationale why the agent made that action. There is an observation that provide details about the new state the world after each action was executed.
The CURRENT TRACE is accompanied by EVALUATION REPORT indicating the the attempt to the task.
You can also be provided with PREVIOUS LEARNINGS which are learninga from the previous attempts by the agent for the same task in the same environment/worle TASK indicates the task description. EPISODE indicates the number of previous attempts the task.
Generate summary of learning, numbered list, that will help the agent to successfully accomplish the SAME task AGAIN, in the SAME world.
Each numbered item in the summary can ONLY be of the form:
X MAY BE NECCESSARY to
SHOULD NECCESSARY to Y.
X MAY BE CONTRIBUTE
X DOES NOT CONTRIBUTE to Y.
(CURRENT TRACE)
Action:
Observation:
EVALUATION REPORT:
REWARD FINAL: 100. This The agent has performed exceptionally well and
DEWARD_FINAL:
successfully solved the task.
Summary of learning as a numbered list:
Fig. 7: Prompt for CLiN'S memory generator during ADAPT
15

---

Published as a conference paper at COLM 2024
[System] You are an expert assistant.
[User]: You are given collection of learning lists, that are derived from actions made by an agent and subsequent observations from world to accomplish TYPE TASKs.
these TASKs belong to a same TYPE (such as 'boiling') but they executed in different ENVIRONMENT configurations. A different ENVIRONMENT configuration means there are presence of different set of objects (lighter instead of a stove) that are critical for solving the TASK, presence of different set of distractor objects that are not useful for the TASK, different floor plan, etc.
learning list, the TASK description provided the beginning TASK:
Each learning list indicates list of learnings from the agent's best attempt to solve the TASK
Each learning list is associated with an EVALUATION REPORT indicated how sucessful the respective attempt was for solving the task.
Consider all learning lists and combine them numbered list, that will help the agent successfully accomplish NEW TASK related to the previous TASKs (such 'boliing') in ENVIRONMENT configuration that it has not seen before. The NEW TASK description will be provided.
aummary learnings, as
Each numbered item in the summary can ONLY be of the form:
MAY BE NECCESSARY
X SHOULD BE NECCESSARY to Y.
X MAY NOT CONTRIBUTE to Y.
X DOES NOT CONTRIBUTE to Y.
(PREVIOUS LEARNINGS)
TASK:
LEARNINGS:
EVALUATION REPORT:
REWARD_FING 100. This The agent has performed exceptionally well and
successfully solved the task.
NEW TASK:
Summary of learning as numbered list:
Fig. 8: Prompt for CLIN'S memory generator during GEN-ENV
16

---

REWARD_ : 100. This means: The agent has performed exceptionally well and Published as a conference paper at COLM 2024
[System] You are an expert assistant.
learnings, that are derived from actions made by agent and subsequent observations from world accomplish TASK ENVIRONMENT CONFIGURATION.
[User]: You may given
For the learning list, the TASK description is provided at the beginning as TASK:
The learnings are from the agent 's best attempt to solve the TASK.
The learning list is associated with EVALUATION REPORT indicated how sucessful the attempt was for solving the task.
Now, generate summary of learnings from the existing ones if provided, such that they will be useful to the NEW TASK in the SAME ENVIRONMENT CONFIGURATION. The NEW TASK may require different actions which are not captured in the given learnings but given learnings can used infer about the ENVIRONMENT CONFIGURATION. The NEW TASK description will given. PREVIOUS LEARNINGS says 'No learnings available', improvise learnings for the NEW TASK.
Each numbered item in the summary can ONLY be of the form:
X MAY BE NECCESSARY to Y.
X SHOULD VECCESSARY to Y.
MAY NOT CONTRIBUTE to Y.
X DOES NOT CONTRIBUTE
(PREVIOUS LEARNINGS]
TASK:
LEARNINGS:
EVALUATION REPORT:
successfully solved the task.
NEW TASK:
S Summary of learning as a numbered list:
Fig. 9: Prompt for CLiN'S memory generator during GEN-TASK
17

---

Published as a conference paper at COLM 2024
Task: Your task is to determine whether round seed shape is dominant or recessive trait in the pea plant. If the trait is dominant, focus on the blue box. the trait recessive, focus on the orange box.
Memory (ADAPT), after Trial5:
1. Looking around MAY BE NECESSARY to gather information about the environment.
2. Koving to the greenhouse SHOULD BE NECESSARY to find the pea seeds.
3. Picking the jar SHOULD BE NECESSARY to access the pea seeds.
4. Moving the round green pea seed to flover pot 2 SHOULD BE NECESSARY to grow the
pea plants.
Koving the wrinkly green pea seed flower pot SHOULD BE NECESSARY grow the pea plants.
6. Waiting SHOULD BE NECESSARY for the pea plants to grow and show their traits.
7. Looking at the adult pea plant in flower pots 2 and 3 MAY BE NECESSARY to observe the traits of the offspring.
8. Focusing on the blue box SHOULD BE NECESSARY to indicate the trait of the seed shape.
Continuously looking the same environment DOES NOT CONTRIBUTE to the task determining the dominant recessive trait in the pea plant.
10. Moving to the workshop DOES NOT CONTRIBUTE determining the dominant recessive trait in the pea plant.
11. Picking up the battery or the wires in the workshop DOES NOT CONTRIBUTE to finding the pea seeds.
12. Waiting for longer periods MAY BE NECESSARY to allow the pea plants to fully grow and show their traits.
Fig. 10: Memory generated after Trial5 in ADAPT for Genetics task in Science
18

---

Published as a conference paper at COLM 2024
Task: Your task is to determine which of the two inclined planes (aluminum, platinum) has the most friction. After completing your experiment, focus on the inclined plane with the most friction.
Meta-memory (CEN-ENV):
Koving to hallway SHOULD NECESSARY reach the workshop.
2. Noving to the workshop SHOULD BE NECESSARY to find the block.
3. Picking up the block SHOULD BE NECESSARY to move it to the inclined planes.
4. Placing the block on the first inclined plane (either aluminum platinum) SHOULD BE NECESSARY to measure the friction.
5. Activating the stopwatch SHOULD BE NECESSARY to time the experiment.
6. Waiting for a certain period MAY CONTRIBUTE to observing the friction effect. Deactivating the stopwatch SHOULD BE NECESSARY to stop timing experiment.
Koving the block the aecond inclined plane (either aluminum platinum) SHOULD BE NECESSARY compare the friction.
9. Activating the stopwatch again SHOULD BE NECESSARY to time the second part of
the experiment.
10. Waiting for certain period again MAY BE NECESSARY to observe the friction
effect.
11. Deactivating the stopwatch again SHOULD BE NECESSARY stop timing the experiment.
12. Focusing the inclined plane with the most friction SHOULD BE NECESSARY
conclude the experiment.
13. Repeating the experiment multiple times MAY BE NECESSARY for more accurate results.
14. Looking around in the initial room multiple times DOES NOT CONTRIBUTE to the task.
Moving block back and forth between two inclined planes DOES CONTRIBUTE the task.
Fig. 11: Meta-memory used in GEN-ENV for Friction task in ScienceWorld.
19

---

Published as a conference paper at COLM 2024
Task: task to freeze mercury. First, focus on the substance. Then, to change its state of matter.
actions that will
Meta-memory (CEN-TASK):
1. Looking around MAY BE NECESSARY to identify the available resources and the layout of the environment.
2. Moving to different rooms SHOULD NECESSARY to the tools materials needed change state of the substance.
like glass cups metal pots SHOULD NECESSARY contain the substance for changing its state.
Picking up
4. Focusing the substance SHOULD BE NECESSARY to understand its properties and
how to interact with it.
5. Picking the thermometer SHOULD BE NECESSARY to monitor the temperature of the substance.
Using the thermometer on the substance SHOULD NECESSARY to monitor the progreas the task.
7. Puring the substance into the container SHOULD BE NECESSARY prepare it for cooling.
8. Koving the container to cooling device SHOULD BE NECESSARY to cool the substance.
9. Waiting for a period of time after cooling the substance SHOULD BE NECESSARY to allow the substance to change state.
Repeatedly checking the temperature the substance SHOULD NECESSARY to monitor the progress of the task.
11. Activating the stove DOES NOT CONTRIBUTE to the task it does not progress the task.
12. Picking up unrelated items like a lighter DOES NOT CONTRIBUTE to the task as it does not progress the task.
13. Moving to unrelated rooms like the workshop DOES NOT CONTRIBUTE to the task as task.
it does not progress the kitchen MAY BE NECESSARY for the task speeds up the process of moving between rooms.
14. Teleporting
15. Using the thermometer multiple times the substance after it reaches freezing point DOES NOT CONTRIBUTE to the task as it does not progress the task.
Fig. 12: Meta-memory used in GEN-TASK for a Freeze task in World
20

---

Published as a conference paper at COLM 2024
Task: Your task is to make red paper.
Ablated memory (GEN-TASK):
1. Avoid repeating actions: The agent performed the "look around" action twice at the beginning of the task. This is unnecessary and wastes time. Once you have clear understanding of your surroundings, move on to the next step.
2. Be mindful of the order of actions: The agent moved to the workshop and read the instructions after looking around twice. It's better to first read the instructions understand the task requirements.
3. Use the correct containers for mixing: The agent used cup for mixing the ingredients, which was successful. Continue using suitable containers like cup or a bowl for mixing.
4. Be clear with your actions: The agent made an ambiguous request when trying to mix the cup containing red paint. Be specific with your actions to avoid confusion.
Avoid unnecessary movements: agent teleported the art studio after picking up the paper. Plan your actions carefully to avoid unnecessary movements. 6. Use the correct method combine ingredients: The agent successfully mixed the ingredients in the cup. Continue using suitable methods to combine the ingredients, such as using blender or a food processor.
7. Keep track of your actions: The agent moved the paper to the cup and then back to the inventory. This is unnecessary and wastes time. Keep track of your actions to avoid repeating them.
Always refer back the instructions: agent seemed forget the instructions make the red paper. Always refer back to the instructions ensure you are following the correct steps.
Fig. 13: Meta-memory generated for ablation GEN-ENV for Chemistry task in Science- World.
21

---

Published as a conference paper at COLM 2024
Generative L. Agents
CLIN (ours)
Task
Type
ReAct
Reflexion
BASE
ADAPT
Temp
S
7.2
5.9
25.2
14.3
Temp
6.1
28.6
53.2
51.8
Pick&Place
26.7
64.9
92.5
100.0
Pick&Place
S
53.3
16.4
55.0
100.0
Chemistry
51.0
70.4
44.5
44.4
Chemistry
58.9
70.7
56.7
56.7
Lifespan
100.0
85.0
100.0
Lifespan
67.5
84.4
70.0
90.0
Biology
S
8.0
8.0
10.0
8.0
Boil
L
3.5
4.2
7.0
15.2
Freeze
L
7.8
7.8
10.0
10.0
Grow Plant
L
7.3
10.2
11.1
GrowFruit
L
18.6
13.0
35.9
71.6
Biology
L
27.7
2.6
70.0
81.0
Force
L
40.5
50.6
53.5
100.0
Friction
L
44.0
100.0
56.5
72.5
Genetics
L
25.7
50.9
77.4
100.0
Genetics
L
16.8
23.7
62.3
92.6
S
37.6
49.9
54.7
62.8
L
21.5
28.9
42.5
61.6
All
29.6
39.4
48.6
62.2
Table 5: Comparing CLIN with baselines for adaptation in ScienceWorld
||
Methods
Generative Language
CLIN fours)
Task
Type
DRRN
KGAZC
CALM
SayCan
Reflexion
GEN-ENV
G+AA
25.2
15.7
1.0
PickePlacc
100.0
PickeePace
100.0
100.0
Chemistry
3.0
lifespan
lilespn
Biology
GrowPlant
Biology
77.4
Table 6: Generalization across unseen environments in ScienceWorld
LLM (here, gpt-4). We find we ablate for the controller module in CLIN, responsible for generating goal before outputting the next action, CLIN'S BASE performance drops in 44% cases. With an 18-point drop in average reward, the Abl-Contoller-BA. -Bas version of CLIN becomes equivalent to ReAct, the base agent for Reflexion, demonstrating the importance of the controller.
Type
#trials to success ())
%ep.
improv.
3.3
29.2
3.2
37.2
All
3.3
33.2
Table 7: CLIN'S ADAPT improvements in ScienceWorld
22

---

Published as a conference paper at COLM 2024
Type
GEN-TASK
G A
Aavg score
%ep.
Aavg score
%ep. improv.
improv.
S
14.6
40.0
4.9
5.7
L
10.3
36.7
9.2
15.6
13.0
38.8
6.5
9.3
Table 8: CLIN'S GEN-TASK improvements in ScienceWorld
23


Reviews:
[{"review_id": "QG3gDTjxKH", "reviewer": "Reviewer_w3HD", "summary": "This paper proposes a general learning agent based on LLM to learn to act and adapt to within environments and across environments. The proposed agent is composed of memories, controller, executor, and memory generator. The four components build a pipeline for producing information necessary for interaction. The authors also adopts a special prompting template and claim causal abstraction is learned. The experiment results show a large boost over reflexion agents.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n`W1. The pipeline is fairly complicated...Does the main boost come from new prompting template, executor design, etc.?`\n\nDespite having several components, CLIN is (we hope) somewhat of a minimalist extension on top of the standard language agent framework (Fig 2). We start with an executor similar to ReAct or Reflexion. We then found adding a separate controller that generates a goal for the next step (loosely analogous to ReAct’s “think” action) yields superior performance on ScienceWorld (Fig 4b) and similar performance on AlfWorld. We hypothesize that ReAct's few-shot demonstrations can often hurt performance; hence, our zero-shot base is a cleaner alternative where all performance comes from the cycles of goal generation and action execution.\n\nIn addition to the baseline boost from adding a controller and working zero-shot, we observe memory (the remaining component) helps improve over time (Fig 4), and with a rate of improvement higher than RaAct and Reflexion. Finally, and perhaps most interestingly, the ablation in Section 5E suggests that structuring memory around causal abstractions (rather than freeform advice) has also helped performance.\n\n`W2. Clarify the contributions and takeaways`\n\nYes, the two novel contributions are:\n(a) a continuously evolving memory (refined after each trial), rather than just a short-term retrieval from a prior trial (Reflexion), can help learning, including new tasks. We provide an architecture for implementing this.\n(b) Structuring memory around causal abstractions, rather than freeform advice (prior work), helps learning over an extended period\nWe will better clarify these in the paper.\n\n`W3. Prompting agents are, interesting, useful, but a bit handwavy`\n\nYes, indeed - the technology is a bit ad hoc and a sound statistical formulation for understanding prompting would be very helpful. However, our main goal is on the construction and use of non-parametric memory, with prompting as a mechanism for feeding those memories into new decisions. Despite the handwavyness, it appears to work well and achieves better generalization than finetuned models, making it interesting and significant for AI (e.g., a Behavior Cloning (Torabi et al., 2018) method presented in (Wang et al., 2022) was trained on 200K training examples to predict the next action. This method resulted in an average reward of 0.08 on the ScienceWorld benchmark). We hope the general prompting area achieves more rigor over time.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "+ The paper shows strong results over previous method (reflexion). The authors demonstrate the performance advantage across various setups.\n\n+ The paper is clearly written with structured flows.\n\n+ The proposed prompting template seems to be very useful\n\n+ The idea of four components borrows the structure from RL community and makes sense.", "reasons_to_reject": "+ The authors seem to have proposed a fairly complicated pipeline, making it difficult to understand which components actually contribute to the performance boost. It would be nice if the authors can thoroughly analyze, does the main boost come from new prompting template, executor design, etc.?\n\n+ Following the above point, the contribution and takeaways are not clear for readers\n\n+ Prompting agents are, interesting, useful, but a bit handwavy. I'm not sure how much of these tricks/methods can contribute in the long-term without sound statistical formulation for understanding and prompting LLMs.", "questions_to_authors": "see above.", "extra_scores": {}}, {"review_id": "n2T4hGImRr", "reviewer": "Reviewer_d8nz", "summary": "The work proposes CLIN, which writes (create/edit) to a textual long-term memory semantic knowledge about the domain and its causal abstractions. Compared to existing work like Reflexion or Voyager, the key difference is that CLIN learns across environments and tasks.\n\nOn ScienceWorld and ALFWorld, it shows performance gain over the base agent and compared to baselines like ReAct and Reflexion, across setups of performing same/different task/environments.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! \n\n`W1. The main limitation is that domains (especially ALFWorld) have regularities and are relatively synthetic. ... It would make the paper much stronger if some more noisy and realistic domains can be tested, e.g. Jericho/WebShop/WebArena/SWE-bench/etc. Even if the results are not good, I would still think the results make the paper stronger...`\n\nYes, ALFWorld is indeed very synthetic and regularized, with limited variation in both tasks and environments. However, Science World (where most of our effort lies) is at the other end of the spectrum, and is one of the more complex and varied text-based environments around at present, with tasks as varied as \"grow and orange\", \"convert a liquid into a gas\", etc.. Thus we hope we have at least covered two domain datapoints at these two extremes. The extensive experiments and ablations we performed on these two domains were at the limit of the scope of this work, both in terms of time and compute, and sufficient to demonstrate effectiveness and generality (at least across these different environments), but it would indeed be an interesting next step to explore generalization to other domains like WebArena.\n\n`W2. There are many math notations, I'd suggest some concrete examples to better illustrate the ideas`\n\nThanks for the pointer. We will add this to the updated version.\n\n`Q1. Equation 1: what's e? I guess environment but not defined in 3.1`\n\nYes, e is environment, as we define it in 3.2.\n\n`Q2. Table 1: Is SayCan basically \"Act\" in ReAct? i.e. \"Observation\" and \"Action\" turns? SayCan should have grounded visual parts, not sure if ScienceWorld has it.`\n\nIndeed, the SayCan baseline in ScienceWorld would be just an Act-equivalent of ReAct, because ScienceWorld does not provide visual information with observations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Long-term memory is a fundamental direction for language agents, and the paper makes solid contribution to this direction.\n\n- The idea makes sense and is tested in systematic setups across domains, environments, and tasks. \n\n- Findings are interesting, e.g. causal abstractions help more than free-form advice, the correlation of correctness and helpfulness of memories, etc. Some experiments and setups can serve as the basis for future research.", "reasons_to_reject": "- The main limitation is that domains (especially ALFWorld) have regularities and are relatively synthetic. Thus, it is not surprising that causal abstractions can be learned and applied, given the current reasoning capabilities of LLMs. It would make the paper much stronger if some more noisy and realistic domains can be tested, e.g. Jericho/WebShop/WebArena/SWE-bench/etc. Even if the results are not good, I would still think the results make the paper stronger, as it objectively reflects limitations of current methods and better point out future directions.\n\n\n- There are many math notations but the explanation is not most intuitive. I'd suggest some concrete examples to better illustrate the ideas.", "questions_to_authors": "- Equation 1: what's e? I guess environment but not defined in 3.1.\n\n- Table 1: Is SayCan basically \"Act\" in ReAct? i.e. \"Observation\" and \"Action\" turns? SayCan should have grounded visual parts, not sure if ScienceWorld has it.", "extra_scores": {}}, {"review_id": "a36si4vvgr", "reviewer": "Reviewer_Vfjj", "summary": "The paper introduces CLIN, a memory-augmented, large language model (LLM)-based agent capable of continuous learning from interactions with simulation environments. The memory system records actions paired with their rewards to evaluate whether certain actions are necessary for achieving the final goals. Results indicate that this approach achieves state-of-the-art performance on benchmarks such as ScienceWorld and ALFWorld, surpassing existing agent task planning methods like Reflexion, React, and Saycan.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! \n\n`W1. \"continual learning\" misleading?`\n\nYou are right. We are using \"continual learning\" in a slightly different sense, although the deviation isn't huge: In cross-episode learning (Sec 3.2), CLIN does learn from a sequence of __different__ tasks to perform on a new, previously unseen task, although they are all ScienceWorld related. To perform well on new tasks, CLIN does behave as if some contents are unobserved. It does this by selectively generalizing from prior episodes, effectively ignoring other prior knowledge.\n\nCLIN's memory mechanism is indeed quite different from continual parameter updates. However, the mechanism shouldn't be confused with the end result: CLIN's performance improves over changes in task and over time. In this sense, CLIN is an interesting alternative mechanism for performing a form of continual learning. Your comments are well-taken, and we will clarify these contrasts.\n\n`W2. How are rewards for given actions assigned?`\n\nLike many POMDP setups, CLIN only receives a reward at a trial's end. CLIN doesn't try to explicitly assign a numeric reward for individual actions; rather, the trials are stored, and when the memory is updated, CLIN relies on the LM to identify useful patterns in high-reward trials. In this way, the LM implicitly rewards some actions by generalizing them to memory.\n\n`W3. Experiment with hotpotQA, HumanEval?`\n\nFor cross-task learning to make sense, there should be __some__ potential for learning in prior trials to help in new trials. For QA tasks like hotpotQA, it is not clear that this happens, as each question is very different. \n\n`W4a. How the memory influences inference`\n\nMemory affects model inference through prompting. By including retrievals in the prompt, CLIN essentially expands the instructions to the controller, providing additional information to help it make its next decision.\n\n`W4b. How dynamic memory helps learning/adaptation to new tasks`\n\nThe memory helps adaptation by being continually updated after each trial, adding, modifying, and removing memories, ideally evolving to more helpful content (see ADAPT, Fig 5). Similarly, given a _new_ task, memory may help a little at the start (GEN, Fig 5), then as more trials are performed, it will again be updated, removing unhelpful content and evolving (G+A, Fig 5).\n\n`Q1. Memory depends on exploration?`\n\nYes indeed. CLIN's negative memories (X does not contribute to Y) help steer it into new explorations when it gets stuck.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The performance on benchmarks such as ScienceWorld and ALFWorld seems good, compared to other agent task planning methods like Reflexion, React, and Saycan.\n2. The paper is well organized, the demonstration of the method is clear.", "reasons_to_reject": "1. Misuse of the term continual learning: The term \"continual learning\" may be misleading. In machine learning, continual learning typically refers to a model's ability to learn from a sequence of different tasks, often requiring the model to behave as if not all contents were observed simultaneously. CLIN does not exhibit the characteristics of continual learning. The term \"learning\" as used here is brief and may be more accurately described as few-shot learning with trial-and-error iterations.\n2. The assignment of rewards for given actions are not specified clearly. It is also good to study the influence using different feedback signals, reward scoring methods such as in Reflexion.\n3. The authors should conduct experiments on more benchmark datasets such as hotpotQA, HumanEval.\n4. How and where does the dynamic memory mechanism influence the model inference, and why it can help continue learning and adaptation to new tasks? More ablation study and case study should be given.", "questions_to_authors": "The memory system depends on the agent's own exploration trails, so I wonder how the LLMs would affect the efficiency of the whole system? As I concern that if the agent cannot retrieve useful information or if the reward is consistently low for a bunch of trials, in the case that the agent cannot gain use knowledge of the world, what would happen.", "extra_scores": {}}, {"review_id": "X5FeRGa7P3", "reviewer": "Reviewer_8qBW", "summary": "This paper presents CLIN, a novel learning algorithm for LMM agents to reflect and improve on their collected experience by forming a memory of causal abstractions.  Expanding upon the _Reflexion_ algorithm, CLIN introduces dual memory updates to construct synergistic abstractions targeting two learning objectives, namely adaptation and generalization. Experimental validation conducted in two diverse textual environments, ALFWorld and Science World, underscores the efficacy of CLIN compared to Reflexion and other relevant baselines. The findings not only underscore the superiority of CLIN but also unveil insights into its underlying learning mechanisms.\n\nOverall, this paper tackles an important problem within the field of decision-making systems and contributes to our comprehension of LLM's capabilities to form novel forms of reasoning. While a more detailed exploration of the role of language as an internal tool for decision-making systems could enhance the paper's positioning, it is well-motivated, clear, and supported by thorough experimentation.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review!\n\n`W1. Positioning: Connecting the causal structure of memories with action models (Arora et al. 2018) is a relevant choice. Could include a brief discussion of previous approaches that use language as a means of abstraction for RL agents...approaches before LLM agents [1,2], and methodologies leveraging LLMs [3]. This perspective paper [4] could be pertinent too.`\n\nThis is a great point. We will include a discussion on the use of language as a form of abstraction in RL agents and cite the mentioned works; thank you for the pointers.\n\n`W2a. A minor limitation: With the immense action space of text games, the approach (like ReAct and Reflexion) only works if we can provide the set of actions available at each time step.`\n\nAgain this is a good point, we will add a note in the paper.\n\n`W2b. Experiments were only carried out using GPT-4 and could include other (maybe smaller) LLMs`\n\nThanks for pointing this out. We tried out with smaller LLMs but found they do not work at all in our setup, mostly because CLIN is a reasoning-based framework, which commonly requires a strong underlying LLM. Although we would like to have frameworks that work well with smaller LLMs, several previous reasoning frameworks also either require a strong proprietary model or find open-source LLMs less performant, similar to our findings. \n\n`Q1. I'm intrigued by the absence of algorithmic incentives for generating concise memories. LLMs often required constraints on their generation. An intuition on the underlying reasons for this phenomenon?`\n\nThis is a great observation, and indeed was a surprise to us also: Indeed, the LM could simply keep growing the memory indefinitely, and we would need to externally apply constraints. We do not have a good explanation for this beyond the notion that LM's appear to have a \"natural\" level of detail behind them when providing an answer (e.g., GPT will answer a question with a paragraph, not an essay), and we are leveraging that to maintain a loosely constant-size memory. One interesting future work would be to encourage the memory to be larger or smaller, either through prompting or post-hoc filtering, and measure the effect of that. \n\n`Q2. How often fuzzy matching occurs`\n\nWe observe the exact matching capability improves from GPT-3.5 to GPT-4.\n\n`Q3. What are the shaded areas in Figure 5?`\n\nThe shaded region indicates the standard deviation in rewards or #steps across all environments (GEN-ENV) or tasks (GEN-TASK).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is very pedagogical and the results demonstrate the benefit of the proposed approach.\n\n### Pedagogy\n\n- The paper is, overall, very well-written. I enjoyed reading it.\n- The paper does a great job of explaining the challenges of Scienceworld in terms of steps to achieve tasks, and convincingly justifies its relevance as a benchmark.\n- Providing all the prompts used for the experiments is a strength. It should be the standard of every contribution leveraging LLM agents.\n\n### Robust findings and thorough methodology\n\n- Unlike previous approaches such as Reflexion, the authors also propose to emancipate from the gold trajectories provided by Scienceworld and use a zero-shot controller.\n- The results section's organization in the form of questions offers clarity, allowing for a thorough assessment of CLIN's performance across diverse scenarios. It effectively isolates the impact of the two memory mechanisms and sheds light on optimal memory structures.", "reasons_to_reject": "### Positioning\n\n- Connecting the causal structure of memories with action models (Arorea et al 2018) is a relevant choice. However, the paper could include a brief discussion of previous approaches that use language as a means of abstraction for RL agents. This encompasses for instance approaches before LLM agents [1,2], as well as methodologies leveraging LLMs [3].\n- The following perspective paper [4] could be pertinent in the discussion of applying linguistic abstraction to goal-directed agents (highlighted by authors on page 3).\n\n### Method's limitation\n\n- A minor limitation of the method is that, as always with the immense action space of text games, the approach (like ReAct and Reflexion) only works if we can provide the set of actions available at each time step.\n- Experiments were only carried out using GPT-4 and could include other (maybe smaller) LLMs \n\n### References \n\n[1] Language as an Abstraction for Hierarchical Deep Reinforcement Learning, Yiding Jiang, Shixiang Gu, Kevin Murphy, Chelsea Finn, NeurIPS 2019\n\n[2] Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration. Cédric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Clément Moulin-Frier, Peter Ford Dominey, Pierre-Yves Oudeyer, NeurIPS 2020\n\n[3] Guiding Pretraining in Reinforcement Learning with Large Language Models, Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, Jacob Andreas, ICML 2023\n\n[4] Language and Culture Internalization for Human-Like Autotelic AI, Cédric Colas, Tristan Karch, Clément Moulin-Frier, Pierre-Yves Oudeyer, Nature Machine Intelligence", "questions_to_authors": "- I'm intrigued by the absence of algorithmic incentives for generating concise memories. LLMs often required constraints on their generation. Do you have an intuition on the underlying reasons for this phenomenon? The result section could investigate how often items are dropped in the memories.\n- It is very informative to explain the kind of fuzzy matching and steps used to retrieve an action when the LLM does not provide a clear candidate. I’m curious as to how often this kind of fuzzy matching occurs during interactions.\n- What are the shaded areas in figure 5?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces CLIN, a language agent that leverages a memory of "causal abstractions" (rather than freeform "advice") to adapt to new tasks and environments. CLIN extends previous reflective language agents like ReAct or Reflexion by using a continuously evolving memory that is refined after each trial. The agent demonstrates strong performance gains over baselines on the ScienceWorld and ALFWorld benchmarks, both in adapting to repeated trials and in generalizing to new tasks and environments.

The reviewers generally agree that the paper tackles an important problem and makes a solid contribution to the developing area of language agents. The paper is well-written, the method is clearly described, and the experiments are thorough, including ablations that provide insights into the key components contributing to CLIN's performance. The proposed CLIN agent demonstrates strong empirical performance and the ability to learn within the same task/environment and adapt to new tasks/environments. 

The authors have provided detailed and thoughtful responses to the reviewer comments in their rebuttal. They have offered to clarify the relationship of their approach to standard continual learning and to expand discussion of related work on linguistic abstractions in RL. While additional evaluations on more open-ended domains would be ideal, reviewers acknowledge that the experiments on ScienceWorld and ALFWorld already demonstrate effectiveness across rather different environment types. 

Overall, the strengths of the paper - the novel dynamic memory architecture, strong empirical performance, and clear presentation - outweigh the limitations pointed out by the reviewers, and I tend to side with the more positive reviews. The paper makes a valuable contribution to the field and opens up promising directions for future work on language agents with more advanced "meta-memory" capabilities.

[At least one review was discounted during the decision process due to quality]
Decision: Accept