CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalization

Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-23 05:52:06

Abstract: Language agents have shown some ability to interact with an external environment, e.g., a virtual world such as ScienceWorld, to perform complex tasks, e.g.,  growing a plant, without the startup costs of reinforcement learning. While recent work, e.g., Reflexion, has demonstrated how such agents can also self-improve by adding a textual memory of ''hints'' learned from prior experience, such improvements have been limited both in size and scope. In contrast, our goal is a language agent that can robustly improve performance over time, including when both the task and environment are varied. Our approach is to have the agent learn a textual representation of how the world works (rather than just isolated hints), expressed as a memory of causal abstractions, to guide future decision-making. In experiments, we find CLIN is able to continually improve on repeated trials on the same task and environment, outperforming state-of-the-art reflective language agents like Reflexion by 23 points in ScienceWorld and 1.4 points in ALFWorld benchmarks. CLIN can also transfer its learning to new environments and tasks, enhancing performance by 21 points in ScienceWorld and 11 points in ALFWorld. This suggests that language agents with a textual causal memory can play a significant role in interactive environments, including being able to rapidly improve over time.

Main paper: 
 introduction: 
 Large language models (LLMs) have been increasingly used to interact with external environments (e.g., simulated worlds) as goal-driven agents (Reed et al., 2022) . However, it has been challenging for these language agents to efficiently learn from trial-and-error as traditional reinforcement learning methods require extensive training samples and expensive model fine-tuning (Chen et al., 2021; Ammanabrolu et al., 2020) . More recently, new techniques have appeared in which an agent reflects on its own past experience solving a task in a particular environment, and generates languagebased insights to help it retry the task, e.g., Reflexion (Shinn et al., 2023) . Such methods have the advantage of not requiring parameter updates (particularly useful given the growing popularity of frozen language models). However, the style of such insights plays a crucial role in performance, and not all insights improve generalization performance. For example, a specific insight such as "In the next trial, I will go to desk 1 and find the lamp" (Shinn et al., 2023) may have limited value (or even hurt) for a different environment or task.Our goal is a system that will continually improve over time, both while attempting the same task in the same environment, and across different tasks and environments. Our approach builds on prior work on reflection in two ways: First, we conjecture that a specific style of insight will be useful, namely one that captures causal abstractions about agent's actions, e.g., "opening doors may be necessary for movement between rooms". Causal abstractions can potentially help the agent decide which action to take in the future, and can be viewed as a kind of action model learning (Arora et al., 2018) , but placed in the modern context of language models. Second, we maintain these abstractions in a continually evolving, dynamic memory, which is regularly updated as the agent gains experience, allowing useful causal knowledge to persist (and unhelpful knowledge to be dropped) over time and between tasks and environments, as illustrated in Figure 1 .Env2 Trial1 Trial2 Trial3 Trial1 Trial2 Trial3MEMORY:Moving to different rooms may be necessary to find an object Env3, Trial1Figure 1 : CLIN creates (Trial1) or adapts (Trial2+) a memory of causal abstractions to help in future trials by reflecting on the last trial and current memory. It does this using a suitably prompted LLM to generate the updated memory (Section 3.2). Here, reflecting on Trial1, CLIN notes in memory that going to the kitchen helped with finding seeds, enabling it to find the seeds faster in Trial2. From there, it also learns that moving the seeds to the pot helped plant the seeds. To further generalize across episodes (sequences of trials, right figure) for use in new environments, CLIN generates a summary ("meta-memory") of the best (starred) memories from each prior episode, here generating the generalization that moving to different rooms helps finding objects (Section 3.3)We operationalize and evaluate this approach in a memory-augmented language agent called CLIN (continual learning from interactions). CLIN is an agent that operates in ScienceWorld (Wang et al., 2022) , a virtual, text-based environment in which an agent is tasked with science-based goals, e.g., boiling a liquid, growing a plant. We find that CLIN is able to rapidly learn about the environment and its action vocabulary and continually improve on repeated trials on the same task and environment, outperforming state-of-the-art (SOTA) reflective language agents like Reflexion by 23 points. CLIN can also transfer its learning to new environments (or tasks), improving its zero-shot performance by 4 (13 for new tasks) points and can further improve performance through continual memory updates, enhancing performance by an additional 17 (7 for new tasks) points. Our contributions are as follows:• We describe and evaluate CLIN, an architecture for a novel nonparametric learning paradigm.We show using a dynamic, evolving memory over time, CLIN learns faster than the shortterm "reflect, use, then discard" approach used in Reflexion and other memory-based agents and generalizes better to new tasks and new environments, achieving state-of-the-art. • We show that memory of causal abstractions (or "action models") is effective at helping the agents learn over an extended period and for varying tasks and environments-first to apply in the modern context of language-based agents. • Overall, this work suggests that a dynamic memory, centered around causal knowledge, is a promising way forward for agents built on frozen models to continually improve over time. 
related work: 
 There is a long literature of work on agents that can navigate complex environments. A common approach is to use reinforcement learning (RL), e.g., DRRN (He et al., 2015) , KG-A2C (Ammanabrolu & Hausknecht, 2020) , CALM (Yao et al., 2020) , where agents learn a task over repeated trials. However, while effective, such agents typically require a large number of trials to learn and have trouble adapting to unexpected changes in the test environment. More recently, (Adaptive- Agent-Team et al., 2023) demonstrated AdA, an agent that could rapidly adapt to open-ended novel 3D problems, using meta-reinforcement learning, essentially being able to change its policy on the fly. However, AdA required vast amounts of pretraining, and this skill was still limited to the style of environments and problems seen in pretraining.the world and requiring little training, e.g., SayCan (Ahn et al., 2022) , ReAct (Yao et al., 2022) , and more recently SwiftSage (Lin et al., 2023) , which combines a supervised agent and a deliberative agent together. However, while performing reasonably with little training data, such agents are unable to learn and adapt from experience.Two recent systems have demonstrated how a frozen-model-based agent could improve at a task. Voyager (Wang et al., 2023) operates in the world of Minecraft, growing a (code-based) skill library from rich feedback of its failures. Reflexion (Shinn et al., 2023) improves at a task by reflecting on a failed attempt at that task and devise a new plan that accounted for that mistake, used in the subsequent prompt to retry the task. While Reflexion did not have a long-term memory, and its reflections were task-and environment-specific, e.g., "In the next trial, I will go to desk 1 and find the lamp.", we take inspiration from it to build an agent, CLIN, which continually maintains and adapts a long-term, persistent memory of reflections, useful across different trials, tasks, and environments.More generally, others have found that a memory of useful learnings can be used to improve frozen LLM behavior, e.g., in QA (Dalvi et al., 2022; Tandon et al., 2022; Madaan et al., 2023) , or for modeling social behavior (Park et al., 2023) . We apply this finding to goal-directed agents.Finally, we note that the content of experiential memory is also important. Specifically, CLIN learns a memory of causal abstractions, which can be seen as learning a linguistic form of action model, describing the causal effects of actions. While there has been substantial work in the planning community of learning action models in a fully formalized context (Arora et al., 2018; Aineto et al., 2018) , CLIN loosely applies this idea in the linguistic world of LLM agents. 
approach: 
 Problem Formulation. Sequential decision-making tasks require agents to repeatedly observe and then act in an environment until they accomplish a specific goal. At a high level, this can be accomplished by developing beliefs about the world, acting on the environment based on those beliefs, and then updating one's beliefs based on the observed outcome. Here, we investigate constructing an agent that can continually update its beliefs through interaction and observation while exploiting its past experience toward solving unseen parametric variations of tasks.Setup. We investigate our continual learning agents in simulated environments. Our environments are modeled in a high-fidelity text-based simulator (Wang et al., 2022) , where both actions and observations are expressed in natural language. Let's define the task space to be M, a collection of partially observable Markov Decision Processes (POMDPs) that can be executed in a set of environment configurations E. Each task m ∈ M has an initial state and a desired winning state, which vary depending on the environment e ∈ E.Our setup allows an agent to attempt a task several times; each time is denoted as a trial, T , which consists of a total of τ steps. Each step comprises an action by the agent (a), and in response, the simulator returns the result of that action in the form of an observation (o) and a reward (r). A collection of K trials is called an episode. The environment gets reset when the task reaches an end state (such as completing, failing, or timing out). In our continual learning setup, the agent retains its memory across trials/episodes, reaping the benefits of continued interaction with the environment.To act in the world, CLIN uses three modules: a memory, a controller, and an executor, illustrated in Figure 2 and which we now describe. Learning then occurs using a fourth module, a memory generator, to generate an updated memory after each trial.Memory. CLIN's memory (S) is a persistent, dynamic collection of NL sentences that express causal abstractions, generated by reflecting on past experiences in order to help the agent perform better in the future. This generation process is described shortly in Section 3.2. For example, having an explicit causal insight, "opening the fridge is necessary to access apple juice", learned from past experiences, can reduce the action search space for CLIN while looking for "apple juice" in the same environment in future trials. To aid in continual learning, the memory also captures mistakes made by the agent in previous trials, similar to reflective agents explored in recent work (Shinn et al., 2023; Park et al., 2023) , noting actions that failed to contribute to a task. A controller takes the current task, retrievals from memory, and the trial so far, to generate the next goal to achieve. The executor then converts this to a valid action to perform towards that goal. The simulator then performs the action and returns an observation of that action's effect. Memory is updated at the end of each trial by the memory generator (Section 3.2).Controller. The role of the controller is to generate the next goal to pursue in service of the task. In CLIN it is a frozen LLM, whose prompt includes the current task m, e.g., "convert water into steam", retrievals from the current memory S, and the trial so far (the sequence of goal-action-observation triples {g 1 , a 1 , o 1 , ..., g t , a t , o t }), and is prompted to output the next goal g t+1 to pursue, e.g., "find water". Memory items are retrieved using both the task instruction and the trial history. The controller first selects one or more memory items given the current state and if they are useful for the next action to progress in the task. After that, it appends the learning, if selected, in context to generate a goal, otherwise the goal is generated based on the trial history (see full prompt in Figure 6 ).Executor. The role of the executor is to convert the generated goal g t+1 into a valid action a t+1 that can be executed in the environment in pursuit of that goal. Again a (frozen) LLM is used, whose prompt includes the goal g t+1 , the trial so far, and all the possible actions that can be performed in the current state (provided by the simulator, as is standard practice in current generative agent research (Ahn et al., 2022; Yao et al., 2022; Lin et al., 2023; Park et al., 2023) ). The list of possible actions is expressed as possible action templates and available objects that can instantiate them, rather than a combinatorially large enumeration of possible actions. The model is then prompted to generate a candidate action to perform (see prompt in Figure 6 ). Finally, CLIN checks this candidate action is one of the valid actions. If it is not, it finds the most similar valid action using the pre-trained embeddings from the sentence-transformer model (Reimers & Gurevych, 2019) . If the top-ranked valid action has a similarity score greater than a threshold (here, 0.9, chosen as a hyperparameter), the action is selected. Otherwise, we perform iterative refinement (Madaan et al., 2023) by suffixing the context with feedback that the generated candidate action is not executable. This allows the executor to retry the generation for up to a maximum number of tries (here, 5).Finally, upon executing the action a t+1 , CLIN receives a partial next state, as an observation, from the simulator and the reward (r) ∈ [0, 1]. Rewards are nominally given by the simulator for achieving either major task subgoals (e.g., finding water, for the boiling task), or minor and optional subgoals (e.g., being in the kitchen, for the boiling task). Rewards are sparse and generally only supplied after the completion of a task subgoal. A snapshot of a full trial is given in lines 4-10 in Algorithm 1.Note that CLIN does not make use of any gold data to identify goals and memories. Rather, we expect CLIN to perform a balanced act of exploration-exploitation by interacting, learning, and adapting to unseen tasks or environment configurations-a key difference from few-shot generative agents by previous work (Ahn et al., 2022; Yao et al., 2022; Lin et al., 2023; Park et al., 2023) .At the end of each trial (completion or failure), CLIN uses a memory generator to create or update its memory. The memory generator is a (frozen) LLM prompted to reflect on the current trial and memory, and generate a new memory of insights in the form of (English sentences expressing) useful causal abstractions, as we now describe. Learning about state transitions is essential for sequential decision-making tasks (Mnih et al., 2013) , which can be manifested by knowing 1) actions enabling desired state transitions, 2) actions producing undesired or no change in states, and 3) state transitions that contribute to the task progress. To generate these kinds of knowledge, the generator is prompted to generate insights in a particular syntax (see prompt in Figure 7 ). To capture good actions enabling desired changes and helpful state transitions, we use the template "X is NECESSARY to Y", and to capture contrastive examples of unsuitable actions and state transitions, we employ "X DOES NOT CONTRIBUTE to Y", as depicted in Section 4.3, where X, Y are related to actions. These abstractions are functionally analogous to hindsight experience replay (Andrychowicz et al., 2017) , obtained from CLIN's past self-explorations.While useful insights can be abstracted from the trials, CLIN's exploration can be limited, especially in the early trials, given an incredibly large action space. Hence, incomplete exploration can pose varying degrees of uncertainty on extracted insights. To capture this, we also include a measure of uncertainty in each abstraction by either of the two linguistic variations in their surface forms: "X may ..." to denote moderate to high uncertainty, and "X should ..." to indicate low uncertainty (See Section 4.3). In the course of continual learning, as CLIN gathers experience, we expect the level of uncertainty to change depending on the frequency of their use and their fitness to the task.Updating Memory Across Trials. CLIN continually attempts to solve a task in an environment for multiple trials (in sum, an episode). To update the memory after each trial within an episode, the memory generator is prompted with the most recent trial (a sequence of (g t , a t , o t ) tuples and the final reward r k1 ), and the memories from the three most recent trials {S k-2 , S k-1 , S k }. It is then prompted to generate an updated memory S k+1 , namely a new list of semi-structured causal abstractions in the forms described above, for use in the next trial. Although we do not specify a maximum size for the memory, we observe that size of the generated memory (i.e., the number of causal abstractions generated) is far less than the number of actions executed in the trial, indicating the memory-generator additionally performs a saliency-based pruning to keep only important insights based on the success of the trial, as indicated by the final reward r k at the end of the trial T k .Updating memory based on past insights and the current trial to influence future trials for the same task in the current environment configuration during test-time adaptation. However, to generalize across tasks or environment configurations, the memory needs to contain more generalized causal abstractions than memories used across trials in an episode. We call this as meta-memory, abstracted across multiple episodes of solving different tasks in different environment configurations to be applicable in future episodes.Auto-curriculum Selection. Before we generate the meta-memory, it is important to choose memories extracted from the best trials from previous episodes because random sampling may not benefit CLIN in zero-shot generalization (Adaptive- Agent-Team et al., 2023) . Following the prioritized level replay scheme (Jiang et al., 2021) , we choose the most successful trial per episode and retrieve memories abstracted from those trials with a fixed archive of size 10, a hyperparameter.Generating Meta-Memory. The goal of the meta-memory is to help CLIN generalize to unseen tasks and/or environments. While we keep the format of the causal abstractions the same as memories generated across trials, the prompt to generate the meta-memory is different than those used for generating per-trial memory. When the new memory is to be used for the same task but in a different environment, the prompt instructs for a meta-memory helpful "to solve the same task in a new environment configuration" given the target task description with an expectation that metamemory abstractions will entail generic causal insights about the task irrespective of environment configurations (see Figure 8 ). Similarly, when the new memory is to be used for a different task, the prompt is modified accordingly to reflect this (Figure 9 ). Along with the target task description for better memory generation, each past memory selected to generate the meta-memory is attached to the final rewards for the associated trials, allowing the generator to combine insights across episodes and assign the levels of uncertainty using the evidence of success. 
results and analysis: 
 Experimental Setup. Test-time adaptation and generalization via continual learning require a variety of complex tasks and environment configurations to allow an agent to explore, learn latent causal insights from interactions, and exploit them in the future. We choose ScienceWorld (Wang et al., 2022) , a text-based interactive environment requiring complex interactive reasoning processes to solve a plethora of science-theory-based tasks spanning several diverse classes (e.g., thermodynamics, genetics, friction, etc.). The virtual space presents 10 sub-places: foundry, greenhouse, outside area, an art studio, workshop, kitchen, living room, bedroom, bathroom, and a hallway connecting inside rooms. The presence of several objects, their individual states, and action templates renders the search space intractable for any agent. ScienceWorld presents strikingly different environment configurations across task types, making it a rich testbed for evaluating adaptation and generalization. ScienceWorld tasks are partitioned into Short (S), e.g., pick & place and Long (L), e.g., grow plant, tasks based on the number of required actions to succeed.Here, we define our setups for zero-shot adaptation (ADAPT) and generalization (GEN-ENV and GEN-TASK) . For all setups, we test CLIN and competing baselines on 18 tasks (two task instances from 9 classes) in several environment configurations from the test split of the ScienceWorld benchmark resulting in a total of 164 task-environment combinations unless stated otherwise. We evaluate based on the final reward score provided by the ScienceWorld simulator.ADAPT: This setup focuses on CLIN's ability to adapt to a task by attempting it for several trials in the same environment configuration. Most importantly, CLIN initializes with an empty memory at the beginning of the first trial and generates memory at the end of each trial. While the environment gets reset at the trial boundary, CLIN's memory continues to be updated, capturing informative causal abstractions pertaining to both successful and failed actions. Here, we compare with Reflexion (Shinn et al., 2023) , a SOTA, however, CLIN differs from Reflexion by how the memory is abstracted.In this setup, we focus on CLIN's ability to transfer its learning from past experiences to solve tasks in an unseen environment. For a task m, we run CLIN for 10 different (train) environment settings (with varying objects and starting locations) and then create meta-memories from its exploration to solve the same task in an unseen (test) environment. Here, we compare CLIN with RL methods DRRN (He et al., 2015) , KG-A2C (Ammanabrolu & Hausknecht, 2020) , and CALM (Yao et al., 2020) trained on all (large) training variations with simulator reward and Generative Language agents, SayCan (Ahn et al., 2022) , ReAct (Yao et al., 2022) , and Reflexion (Shinn et al., 2023) , prompted with few-shot demonstrations. In this setup, we focus on CLIN's ability to transfer its learning from past experiences to solve a new task in the same environment. For an environment e, we run CLIN for to solve a task m and then condense its learning to solve a novel task m ′ in the environment e. We took all test examples where we have a different task defined in the same environment configuration.(Adaptive-Agent-Team et al., 2023) suggests that transferring learning from a random task can be very hard; hence we couple tasks that are related (revolve around overlapping task-critical objects/locations such water, kitchen), such as boil and freeze to measure transfer learning from one to the other. This is a novel setup where we do not have any off-the-shelf baselines. However, here, we compare against CLIN-BASE, a strong baseline agent.GEN-ADAPT (G+A): If CLIN, in GEN-ENV or GEN-TASK setting, does not successfully complete the new task, it can continue learning and retrying that task. We refer to this setup as GEN-ADAPT.CLIN can use any instruction-tuned LLM (Chung et al., 2022) as part of the controller, executor, and memory generator. In this paper, we use gpt-4, the same as our generative agent baselines.4.1 CLIN EXHIBITS RAPID TASK ADAPTATION Figure 4a demonstrates two example trends where CLIN learns from its own prior attempts (ADAPT) and gets better at solving a given task. Apart from length, the difficulty level of a task also depends on the environment configuration (hence, variance across environment configurations for each task). CLIN quickly adapts to a short task, Pick & Place, solving it in its 4th attempt, whereas for a longer task, Grow Fruit, it is not solved after 5th (max) attempts. Furthermore, Figure 4a depicts, CLIN becomes more efficient in later trials by solving the tasks with a lower number of (average) steps. Figure 4c shows an average number of attempts2 for CLIN to solve a task and % episodes per task where scores improved compared to its own first trial.Next, we compare CLIN with Reflexion, the reflective SOTA agent, in Figure 4b . CLIN already starts off with a stronger base performance (see discussion in 4.3), however, CLIN's relative improvement in ADAPT is significantly stronger than Reflexion's gain from its base agent ReAct. Furthermore, CLIN's relative improvement is higher for longer tasks. This can be attributed to CLIN's persistent memory, which gets refined over past trials, whereas Reflexion may fall short of collecting useful learnings from earlier trials as it only focuses on the current trial for its reflections (hence not longterm). Furthermore, CLIN accumulates both useful (for the task) and harmful (for the task) causal learnings, whereas Reflexion only learns from its mistakes, lacking comprehensive learning.Generalizing to new environments. Generalizing to new tasks. Mirroring trends from GEN-ENV, CLIN demonstrates strong transfer learning to new tasks with 13-point improvement over its BASE performance, being better at 38.8% of times (Figure 5c ). The improvement attributes to critical learning about the environment ("apple juice is in the fridge", required for both boiling and freezing it), leading to improvement in previously low-performing tasks in both ADAPT and GEN-ENV setups. This transfer learning in GEN-TASK and G+A helps CLIN to solve the tasks with a lesser number of steps4 and achieve higher rewards.Importance of memory structure. CLIN extracts causal abstractions structured around 'necessary' and 'does not contribute' relations. To ablate, we modified our memory generator to generate freeform advice for future trials, which, however, ended up generating generic insights without any causal abstractions (Figure 13 ). The average reward drops by 6 points (in 10% cases than CLIN) when using the unstructured memory, indicating the usefulness of causal abstractions, as shown in Figure 5d .Superior BASE performance. Figure 4 depicts a superior BASE performance for CLIN than the final performance of both ReAct and Reflexion despite using the same underlying LLM (here, gpt-4).We find if we ablate for the controller module in CLIN, responsible for generating a goal before outputting the next action, CLIN's BASE performance drops in 44% cases. With an 18 point drop in average reward (see Figure 5d ), Abl-Contoller-BASE version of CLIN becomes equivalent to ReAct, the base agent for Reflexion, demonstrating the importance of controller even in BASE setup.A qualitative example. Figure 3 depicts how memory items get refined during task adaptation and for generalization for a task boil. Env2 has a working stove, whereas in Env1, the stove is broken, but a lighter is available as an alternative. With a number of trials in these environments, CLIN learns how to use these two devices to generate heat. In an unseen environment with a broken stove, CLIN quickly receives a positive reward by using a lighter to heat a substance. While insights within an episode are often specific, e.g., "Using the lighter on the metal pot should be necessary to heat the water in the pot", CLIN compiles these insights for a new target environment (as meta-memory), e.g., "Using a heat source (stove, lighter) on the container should be necessary to heat a substance." Appendix B contains examples of memories generated during adaptation and generalization.Limitation: Lack of exploration. CLIN's learnings are dependent on its own past experience. If CLIN never explores a location in the environment or does not perform an action, an insight related to the unobserved activity can never be generated. Hence, exploration becomes important when task-critical location or action in unknown to CLIN from past trials. For example, in task of creating an orange paint, the agent is supposed to find red and yellow paints from the art studio. However, art studio is not visible when CLIN starts from location 'outside'. Unless the CLIN knows that there exists an art studio, it tries alternative method to create orange paints from other irrelevant objects (e.g., an orange) and remains unsuccessful. When a memory related art-studio appears from past exploration, CLIN is able to successfully complete the task. Similarly, in boil or freeze tasks, CLIN is unable to perform well which requires it to consistently measure the temperature of the substance to know its boiling/freezing point-an act it could never perform successfully in past trials resulting into less useful memory insights and subsequent lower performance in future trials.Limitation: Poor memory retrieval. For a task of boiling gallium, CLIN is supposed to use oven/blast furnace and not a stove. In the meta-memory for boiling tasks, there are two insights regarding the act of boiling: "Activating stove should be necessary to boil a substance" and "Using an alternative heat source (e.g., oven or fire pit) may be necessary if the initial heat source is insufficient." However, CLIN repeatedly retrieves the former and hence failing at the task despite performing other actions (e.g., finding gallium) correctly. This problem intensifies at the initial trial during generalization due to the presence of insights with varied initial conditions for them to be applied. This can be circumvented by improved memory representation, which we leave as a future work. 
conclusion: 
 Our goal is a system that can continually improve over time, both while rapidly adapting to a task by multiple retries and efficiently generalizing to novel tasks and environments. We propose CLIN, an architecture for language agents that constructs a persistent, dynamic memory of causal abstractions, refines it over time and uses it effectively to improve its performance on future tasks, achieving state-of-the-art performance. Our work systematically evaluates a novel nonparametric learning paradigm, promising never-ending learning abilities to frozen language agents.[System]: You are an expert assistant.[User]: You are given CURRENT TRACE, a sequence of actions that an agent made in a world to accomplish a task.Task is detailed at the beginning.For each action, there is a rationale why the agent made that action.There is an observation that provide details about the new state of the world after each action was executed.The CURRENT TRACE is accompanied by an EVALUATION REPORT indicating the success of the attempt to the task.You can also be provided with PREVIOUS LEARNINGS which are learnings from the previous attempts by the agent for the same task in the same environment/world. TASK indicates the task description. EPISODE indicates the number of previous attempts of the task.Generate a summary of learning, as a numbered list, that will help the agent to successfully accomplish the SAME task AGAIN, in the SAME world.Each numbered item in the summary can ONLY be of the form: X MAY BE NECCESSARY to Y. X SHOULD BE NECCESSARY to Y. X MAY BE CONTRIBUTE to Y. X DOES NOT CONTRIBUTE to Y.{CURRENT TRACE} Action: ... Observation: ... ... EVALUATION REPORT: REWARD_FINAL: 100. This means: The agent has performed exceptionally well and successfully solved the task.Summary of learning as a numbered list: [System]: You are an expert assistant.[User]: You are given a collection of learning lists, that are derived from actions made by an agent and subsequent observations from a world to accomplish a TYPE of TASKs.All of these TASKs belong to a same TYPE (such as 'boiling') but they are executed in different ENVIRONMENT configurations. A different ENVIRONMENT configuration means there are presence of a different set of objects (lighter instead of a stove) that are critical for solving the TASK, presence of a different set of distractor objects that are not useful for the TASK, a different floor plan, etc.For each learning list, the TASK description is provided at the beginning as TASK:Each learning list indicates a list of learnings from the agent's best attempt to solve the TASK.Each learning list is associated with an EVALUATION REPORT indicated how sucessful the respective attempt was for solving the task.Consider all learning lists and combine them in to a summary of learnings, as a numbered list, that will help the agent to successfully accomplish a NEW TASK related to the previous TASKs (such as 'boliing') in an ENVIRONMENT configuration that it has not seen before. The NEW TASK description will be provided. Task: Your task is to freeze mercury. First, focus on the substance. Then, take actions that will cause it to change its state of matter.Meta-memory (GEN-TASK):1. Looking around MAY BE NECESSARY to identify the available resources and the layout of the environment. 2. Moving to different rooms SHOULD BE NECESSARY to find the tools and materials needed to change the state of the substance.3. Picking up items like glass cups or metal pots SHOULD BE NECESSARY to contain the substance for changing its state. 4. Focusing on the substance SHOULD BE NECESSARY to understand its properties and how to interact with it. 5. Picking up the thermometer SHOULD BE NECESSARY to monitor the temperature of the substance.6. Using the thermometer on the substance SHOULD BE NECESSARY to monitor the progress of the task. 7. Puring the substance into the container SHOULD BE NECESSARY to prepare it for cooling. 8. Moving the container to a cooling device SHOULD BE NECESSARY to cool the substance. 9. Waiting for a period of time after cooling the substance SHOULD BE NECESSARY to allow the substance to change state. 10. Repeatedly checking the temperature of the substance SHOULD BE NECESSARY to monitor the progress of the task. 11. Activating the stove DOES NOT CONTRIBUTE to the task as it does not progress the task. 12. Picking up unrelated items like a lighter DOES NOT CONTRIBUTE to the task as it does not progress the task. 13. Moving to unrelated rooms like the workshop DOES NOT CONTRIBUTE to the task as it does not progress the task. 14. Teleporting to the kitchen MAY BE NECESSARY for the task as it speeds up the process of moving between rooms. 15. Using the thermometer multiple times on the substance after it reaches freezing point DOES NOT CONTRIBUTE to the task as it does not progress the task.Figure 12 : Meta-memory used in GEN-TASK for a Freeze task.Task: Your task is to make a red paper.Ablated memory (GEN-TASK):1. Avoid repeating actions: The agent performed the "look around" action twice at the beginning of the task. This is unnecessary and wastes time. Once you have a clear understanding of your surroundings, move on to the next step.2. Be mindful of the order of actions: The agent moved to the workshop and read the instructions after looking around twice. It's better to first read the instructions to understand the task requirements.3. Use the correct containers for mixing: The agent used a cup for mixing the ingredients, which was successful. Continue using suitable containers like a cup or a bowl for mixing. 4. Be clear with your actions: The agent made an ambiguous request when trying to mix the cup containing red paint. Be specific with your actions to avoid confusion. 5. Avoid unnecessary movements: The agent teleported to the art studio after picking up the paper. Plan your actions carefully to avoid unnecessary movements. 6. Use the correct method to combine ingredients: The agent successfully mixed the ingredients in the cup. Continue using suitable methods to combine the ingredients, such as using a blender or a food processor. 7. Keep track of your actions: The agent moved the paper to the cup and then back to the inventory. This is unnecessary and wastes time. Keep track of your actions to avoid repeating them. 8. Always refer back to the instructions: The agent seemed to forget the instructions to make the red paper. Always refer back to the instructions to ensure you are following the correct steps.  
 
 Appendix: Full results for CLIN outperforming Reflexion is in Table 3 . For ScienceWorld benchmark, we exclude electricity tasks since they deviate from standard electrical conventions, prohibiting us from fairly using LLM agents. We choose the first 10 test variants for each 18 tasks selected. The full list of 18 tasks from the benchmark, with the number of test variants used in parentheses:grow-plant (10), identify-life-stages-1 (5), grow-fruit (10), measure-melting-point-known-substance (10), mendelian-genetics-unknown-plant (10), chemistry-mix-paint-secondary-color (9), freeze (9), lifespan-longest-lived (10), inclined-plane-determine-angle (10), boil (9), use-thermometer (10), chemistry-mix (8), lifespan-shortest-lived (10), find-plant (10), find-living-thing (10), identify-lifestages-2 (4), mendelian-genetics-known-plant (10), inclined-plane-friction-named-surfaces (10).Short tasks have oracle lengths less than 37 steps (median), and Long tasks have oracle lengths more than equal to 37 steps.The map to the short names used for tasks in the paper:Temp: use-thermometer, measure-melting-point-known-substance; Pick&Place: find-plant, findliving-thing; Chemistry: chemistry-mix, chemistry-mix-paint-secondary-color; Lifespan: lifespanlongest-lived, lifespan-shortest-lived; Biology: identify-life-stages-1, identify-life-stages-2, Boil; Freeze; Grow Plant, Grow Fruit; Force: inclined-plane-determine-angle; Friction: inclined-planefriction-named-surfaces; Genetics: mendelian-genetics-known-plant, mendelian-genetics-unknownplant.Memory correctness While the final performance with memory is indicative of their effectiveness, we performed additional human evaluation of generated memory insights for correctness. For both the Gen-Env and Gen-Task setups, we randomly 10 task-environment combinations to evaluate the correctness of memories used in them, notably the meta-memory used for trial 0 (GEN) and memory adapted for the best trial (GEN-ADAPT). Two annotators rated all the insights (cohen's kappa 0.72) for correctness with reference to the respective gold trajectories. Table 2 shows that it is possible to have some meta-memories that are not applicable for a new task or environments at Trial 0; however, with adaptation, in later trials, the correctness of the memory insights significantly improves, leading to a direct increase in task performance. The reward is converted to NL feedback for a LLM using 7 simple rules, e.g., "if score >= 0 and score <then feedback = "The agent performed poorly and made some progress but not enough to solve the task."During ADAPT, CLIN tries up to 5 trials. If it solves a task with a score of 100, it stops retrying.Baseline numbers are derived from Table1in(Lin et al., 2023)# steps in Figure5(a),(b) are normalized between 0-1, 1 being maximum #steps allowed for a task. 


Decision: This paper introduces CLIN, a language agent that leverages a memory of "causal abstractions" (rather than freeform "advice") to adapt to new tasks and environments. CLIN extends previous reflective language agents like ReAct or Reflexion by using a continuously evolving memory that is refined after each trial. The agent demonstrates strong performance gains over baselines on the ScienceWorld and ALFWorld benchmarks, both in adapting to repeated trials and in generalizing to new tasks and environments.

The reviewers generally agree that the paper tackles an important problem and makes a solid contribution to the developing area of language agents. The paper is well-written, the method is clearly described, and the experiments are thorough, including ablations that provide insights into the key components contributing to CLIN's performance. The proposed CLIN agent demonstrates strong empirical performance and the ability to learn within the same task/environment and adapt to new tasks/environments. 

The authors have provided detailed and thoughtful responses to the reviewer comments in their rebuttal. They have offered to clarify the relationship of their approach to standard continual learning and to expand discussion of related work on linguistic abstractions in RL. While additional evaluations on more open-ended domains would be ideal, reviewers acknowledge that the experiments on ScienceWorld and ALFWorld already demonstrate effectiveness across rather different environment types. 

Overall, the strengths of the paper - the novel dynamic memory architecture, strong empirical performance, and clear presentation - outweigh the limitations pointed out by the reviewers, and I tend to side with the more positive reviews. The paper makes a valuable contribution to the field and opens up promising directions for future work on language agents with more advanced "meta-memory" capabilities.

[At least one review was discounted during the decision process due to quality]

Reviews: review: paper_summary: paper_summary: This paper proposes a general learning agent based on LLM to learn to act and adapt to within environments and across environments. The proposed agent is composed of memories, controller, executor, and memory generator. The four components build a pipeline for producing information necessary for interaction. The authors also adopts a special prompting template and claim causal abstraction is learned. The experiment results show a large boost over reflexion agents.
strength_weakness: strength_weakness: + The authors seem to have proposed a fairly complicated pipeline, making it difficult to understand which components actually contribute to the performance boost. It would be nice if the authors can thoroughly analyze, does the main boost come from new prompting template, executor design, etc.?

+ Following the above point, the contribution and takeaways are not clear for readers

+ Prompting agents are, interesting, useful, but a bit handwavy. I'm not sure how much of these tricks/methods can contribute in the long-term without sound statistical formulation for understanding and prompting LLMs.
questions: questions: see above.
paper_summary: paper_summary: The work proposes CLIN, which writes (create/edit) to a textual long-term memory semantic knowledge about the domain and its causal abstractions. Compared to existing work like Reflexion or Voyager, the key difference is that CLIN learns across environments and tasks.

On ScienceWorld and ALFWorld, it shows performance gain over the base agent and compared to baselines like ReAct and Reflexion, across setups of performing same/different task/environments.
strength_weakness: strength_weakness: - The main limitation is that domains (especially ALFWorld) have regularities and are relatively synthetic. Thus, it is not surprising that causal abstractions can be learned and applied, given the current reasoning capabilities of LLMs. It would make the paper much stronger if some more noisy and realistic domains can be tested, e.g. Jericho/WebShop/WebArena/SWE-bench/etc. Even if the results are not good, I would still think the results make the paper stronger, as it objectively reflects limitations of current methods and better point out future directions.


- There are many math notations but the explanation is not most intuitive. I'd suggest some concrete examples to better illustrate the ideas.
questions: questions: - Equation 1: what's e? I guess environment but not defined in 3.1.

- Table 1: Is SayCan basically "Act" in ReAct? i.e. "Observation" and "Action" turns? SayCan should have grounded visual parts, not sure if ScienceWorld has it.
paper_summary: paper_summary: The paper introduces CLIN, a memory-augmented, large language model (LLM)-based agent capable of continuous learning from interactions with simulation environments. The memory system records actions paired with their rewards to evaluate whether certain actions are necessary for achieving the final goals. Results indicate that this approach achieves state-of-the-art performance on benchmarks such as ScienceWorld and ALFWorld, surpassing existing agent task planning methods like Reflexion, React, and Saycan.
strength_weakness: strength_weakness: 1. Misuse of the term continual learning: The term "continual learning" may be misleading. In machine learning, continual learning typically refers to a model's ability to learn from a sequence of different tasks, often requiring the model to behave as if not all contents were observed simultaneously. CLIN does not exhibit the characteristics of continual learning. The term "learning" as used here is brief and may be more accurately described as few-shot learning with trial-and-error iterations.
2. The assignment of rewards for given actions are not specified clearly. It is also good to study the influence using different feedback signals, reward scoring methods such as in Reflexion.
3. The authors should conduct experiments on more benchmark datasets such as hotpotQA, HumanEval.
4. How and where does the dynamic memory mechanism influence the model inference, and why it can help continue learning and adaptation to new tasks? More ablation study and case study should be given.
questions: questions: The memory system depends on the agent's own exploration trails, so I wonder how the LLMs would affect the efficiency of the whole system? As I concern that if the agent cannot retrieve useful information or if the reward is consistently low for a bunch of trials, in the case that the agent cannot gain use knowledge of the world, what would happen.
paper_summary: paper_summary: This paper presents CLIN, a novel learning algorithm for LMM agents to reflect and improve on their collected experience by forming a memory of causal abstractions.  Expanding upon the _Reflexion_ algorithm, CLIN introduces dual memory updates to construct synergistic abstractions targeting two learning objectives, namely adaptation and generalization. Experimental validation conducted in two diverse textual environments, ALFWorld and Science World, underscores the efficacy of CLIN compared to Reflexion and other relevant baselines. The findings not only underscore the superiority of CLIN but also unveil insights into its underlying learning mechanisms.

Overall, this paper tackles an important problem within the field of decision-making systems and contributes to our comprehension of LLM's capabilities to form novel forms of reasoning. While a more detailed exploration of the role of language as an internal tool for decision-making systems could enhance the paper's positioning, it is well-motivated, clear, and supported by thorough experimentation.
strength_weakness: strength_weakness: ### Positioning

- Connecting the causal structure of memories with action models (Arorea et al 2018) is a relevant choice. However, the paper could include a brief discussion of previous approaches that use language as a means of abstraction for RL agents. This encompasses for instance approaches before LLM agents [1,2], as well as methodologies leveraging LLMs [3].
- The following perspective paper [4] could be pertinent in the discussion of applying linguistic abstraction to goal-directed agents (highlighted by authors on page 3).

### Method's limitation

- A minor limitation of the method is that, as always with the immense action space of text games, the approach (like ReAct and Reflexion) only works if we can provide the set of actions available at each time step.
- Experiments were only carried out using GPT-4 and could include other (maybe smaller) LLMs 

### References 

[1] Language as an Abstraction for Hierarchical Deep Reinforcement Learning, Yiding Jiang, Shixiang Gu, Kevin Murphy, Chelsea Finn, NeurIPS 2019

[2] Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration. Cédric Colas, Tristan Karch, Nicolas Lair, Jean-Michel Dussoux, Clément Moulin-Frier, Peter Ford Dominey, Pierre-Yves Oudeyer, NeurIPS 2020

[3] Guiding Pretraining in Reinforcement Learning with Large Language Models, Yuqing Du, Olivia Watkins, Zihan Wang, Cédric Colas, Trevor Darrell, Pieter Abbeel, Abhishek Gupta, Jacob Andreas, ICML 2023

[4] Language and Culture Internalization for Human-Like Autotelic AI, Cédric Colas, Tristan Karch, Clément Moulin-Frier, Pierre-Yves Oudeyer, Nature Machine Intelligence
questions: questions: - I'm intrigued by the absence of algorithmic incentives for generating concise memories. LLMs often required constraints on their generation. Do you have an intuition on the underlying reasons for this phenomenon? The result section could investigate how often items are dropped in the memories.
- It is very informative to explain the kind of fuzzy matching and steps used to retrieve an action when the LLM does not provide a clear candidate. I’m curious as to how often this kind of fuzzy matching occurs during interactions.
- What are the shaded areas in figure 5?

Comments: title: Following up.....
Thank you!
comment: Just to follow up, we hope we have addressed your concerns, and that you might consider increasing your score. Please let us know if you still have unresolved questions, and thank you for your work serving as a reviewer!
Dear reviewer,

Thank you for your reply. Please let us know if you have any questions in the meantime.

Thanks,\
Authors
Thanks for the rebuttal, i will keep my review as is