I am interested about this paper for embodied agents building long term memories and insights from previous experiences and how it can be applied to this open source agent system. I want to explore how I could potentially set this up. can you tell me how this could all work. if it is worth pursuing or not

2.2 LLM Long-term Memory
When making decisions, humans often recall past
cases to assist in decision-making. Due to the limited input length, the LLM Agent cannot receive infinite historical experiences. Therefore, efficiently
utilizing existing success/failure experiences becomes crucial. The LLM Long-term Memory is
designed to address this challenging issue (Zhao
et al., 2023; Wen et al., 2023; Majumder et al.,
2023; Qian et al., 2024). Currently, the LLM Agent
Memory operates in two modes: example memory
and insight memory. Example memory involves
manually crafting experience examples that were
successful in tasks. During usage, similar examples are retrieved based on the current task, using
methods such as vectors or BM25, to prompt the
large language model (Wang et al., 2023a; Wen
et al., 2023; Dong et al., 2024b; Song et al., 2023;
Zhong et al., 2023). Insight memory, on the other
hand, summarizes success/failure experiences into
insights through the LLM. When new tasks occur,
the insights are directly input as a part of the prompt
into the LLM for helping planning and decisionmaking. (Majumder et al., 2023; Zhao et al., 2023).
3 Method
Figures 2 and 3 illustrate our approach. Initially,
utilizing historical task data (train set), we employ
the task execution module to collect a sufficient
number of experiences. (§3.1) These experiences
are then subjected to the experience selector, which
identifies experiences/experience pairs suitable for
generating insights. (§3.2) Subsequently, the multiscale insights will be generated and stored in the
insight database. (§3.3) When a new task arises, we
retrieve relevant sights from the database based on
predefined rules. (§3.4) These insights, along with
task background, and user queries, are provided to
the task execution module to facilitate execution.
We refer to the process from experience collection
to insight generation as insight summarization, and
the subsequent insight selection and task execution
as insight utilization.
3.1 Experience Generation
As shown in Figure 2, we regard training data as
history tasks. For each history task, the executor leverages LLM to generate a plan based on
task background and user queries. Subsequently,
the robot employs first-order logic to decompose
the plan into atomic actions (e.g., moving forward,
picking up objects) and execute them in an environment. In some tasks or cases, the executor may
replan based on the environment feedback. Upon
completion, task background, user queries, agent’s
plans, environmental feedback, and execution results are stored as experiences for summarization.
Detailed information can be found in Appendix A.
3.2 Experience Selection
The selection of experiences is crucial in summarizing insights, as it determines the quality of insights
the model consolidates. As shown in Figure 3, our
Experience Selection employs two modes:
Success mode: We select experiences with successful execution results as the success mode experiences.
Pair mode: For each successful experience ss,
we identify a corresponding experience sf from the
unsuccessful experience database Sf by:
sf = argmax
s∈Sf
emb(s) · emb(ss)
p
||emb(s)||2||emb(ss)||2
(1)
Where emb is the embedding of the experience’s
user query and the (ss, sf ) is the final experience
pair in the pair mode.
These two types of selected experience (pair)
collections are subsequently preserved and utilized
as seed experience for insight summarization.
3.3 Multi-Scale Insight Generation
Multi-Scale Insight We categorize the insights
into several scales. For all tasks, we will generate general scale and subtask scale insights. If
the task provides a specific environment category
(for example, kitchen), we will also generate environment scale insights. General insight refers to
the knowledge required for all tasks, which should
be high-level. Environment insight pertains to the
knowledge needed in a specific environment, and
subtask insight involves the understanding of executing particular subtasks. The overall pipeline can
be seen in Figure 3’s Insight Generation module.
Insight Generation We initialize the insight
database to be empty. Whenever a seed experience merges, we select all insights in the order of
general, subtask.2
as a pool of candidate experience
for updating.
Subsequently, we prompt the LLM with templates containing the candidate insight, all experience information, and descriptions of 5 atomic
actions: adding, removing, editing, agreeing on
an insight, and moving an insight between scales,
requesting the LLM to update the insight database
through these atomic actions (Zhao et al., 2023).
For subtask insight, we also require the LLM to additionally generate a subtask name corresponding
to the insights. 3
After the LLM generation is complete, we update the insight database in the order of general,
environment (if have), and subtask, according to
the atomic actions.
Align with Expel, we also employ a scoring
mechanism in insight generation. Specifically, each
2
If there is a specific environment category in the task, we
will select environment and subtask insight that is consistent
with the experience’s environment category, and the order is
general, environment, and subtask
3The prompt of Insight Generation can be seen in Appendix C
insight receives an initial score of 2 when an "add"
or "move" action is executed, the score increases by
1 for an "agree" action, remains unchanged for an
"edit" action, and decreases by 1 for a "remove" action. An insight is discarded when its score reaches
zero.
3.4 Multi-Scale Insight Selection
Similar to the generation process, we use general
and subtask insights2
as candidate insights. For
subtask insights, we adopt two modes for further
selection:
Hashmap indexing: We extract all subtask
names from the subtask insight database, combine
them with user queries, and provide them to the
LLM, requiring the LLM to return all task names
related to the user query. Subsequently, we consider all insights under returned subtask names as
the subtask insights for this user query. The prompt
of hashmap subtask selection can be seen in Appendix D
Vector indexing: We compute the cosine similarity between all subtask insights and the user
query, selecting insights with at most 2000 tokens.4
Ultimately, we provide the different scales of
insights, and the user query to the task execution
module to accomplish the task.



AGENT SYSTEM (AGENT-Q)

1. Introduction
The recent advances in Large Language Models (LLMs) represent a significant leap in artificial
intelligence. Frontier models like ChatGPT (John Schulman et al., 2022), Gemini (Anil et al., 2023),
Opus (Anthropic, 2024), and LLaMA-3 (Touvron et al., 2023) demonstrate promising reasoning
capabilities that approach average human performance in a number of domains. These breakthroughs
have extended the utility of LLMs from traditional chat and text-based applications to more dynamic,
agentic roles, in which they do not just generate text but can take actions autonomously in a number
of environments including code and software engineering (Holt et al., 2024; Zhang et al., 2024d;
Jimenez et al., 2024; Yang et al., 2024), device control (Wang et al., 2024a; Zhang et al., 2023; Chen
and Li, 2024) and web applications (Hong et al., 2023; Deng et al., 2023; Zhou et al., 2024b; Lai
et al., 2024a; Gur et al., 2024) among others. However, despite these advancements, significant
challenges persist: LLMs still struggle to generalize effectively in interactive, multi-step environments,
since they are not native trained for such applications . This is true, even for some of the strongest
models of the current generation, such as GPT-4 (Achiam et al., 2023).
A growing literature on agentic formulation seeks to address these issues; however these works mostly
focus on building frameworks around prompt-based learning on existing models or limited fine-tuning
on static datasets, and are thus limited by the base models’ reasoning and decision making capabilities.
Reasoning and planning have indeed been highlighted as core challenges for current LLMs. Since
the seminal work on chain-of-thought reasoning (Wei et al., 2022), significant efforts have been
made to improve these capabilities via prompt-based strategies (Kojima et al., 2022; Wang et al.,
Corresponding author(s): <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>, <PRESIDIO_ANONYMIZED_EMAIL_ADDRESS>
Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents
Figure 1: We use Monte Carlo Tree Search (MCTS) to guide trajectory collection and iteratively
improve model performance using direct preference optimization (DPO). We begin on the left by
sampling a user query from the list of tasks in the dataset. We iteratively expand the search tree
using UCB1 as a heuristic to balance exploration and exploitation of different actions. We store the
accumulated reward obtained for each node in the tree, where in this image darker green indicates
higher reward and darker red indicates lower reward. To construct the preference dataset, we compute
a weighted score of the MCTS average Q-value and score generated by a feedback language model to
construct contrastive pairs for DPO. The policy is optimized and can be iteratively improved.
2023; Qiao et al., 2023; Yao et al., 2023a). While successful, these approaches are still bounded by
the base model’s performance. Another direction of research has explored fine-tuning approaches
(Zelikman et al., 2022; Pang et al., 2024), and more recently combining them with inference-time
search prompting (Yao et al., 2023a) to produce fine-grained feedback. Concurrent works (Xie et al.,
2024; Hwang et al., 2024; Zhang et al., 2024e; Tian et al., 2024) utilize the traces produced by search
algorithms and combine them with optimization approaches (Rafailov et al., 2023; Zelikman et al.,
2022) to achieve significant boost in capabilities, especially in mathematics problem solving and code
generation.
In this work we explore improving planning and reasoning capabilities of a web agent, which interacts
with a real world website. Our goal is to design an approach that allows the agent to improve with
autonomous experience and limited supervision. Indeed, prior works (Yao et al., 2023b; Zhang et al.,
2024c; Masterman et al., 2024; Sumers et al., 2024) have shown strong reasoning to be critical for
performance of autonomous agents, where challenges are even greater than during text generation,
as the model needs to further understand how its actions affect its environment. Towards this goal,
we introduce Agent Q—a novel approach that combines several key concepts in reasoning, search,
self-critique and reinforcement learning. Our method takes inspiration from Sutton’s The Bitter
Lesson on the power of general purpose methods that continue to scale with increased computation,
showing the significant benefits of combining search and learning.
Inspired by the success of search-based methods in prior game-playing settings (Silver et al., 2017a;
Brown and Sandholm, 2019; Gray et al., 2021) and mathematical reasoning (Yao et al., 2023a; Besta
et al., 2024), we deploy a Monte Carlo Tree Search (MCTS) based search routine over web pages to
guide agent exploration. Given the complexity of the environment, we use a base LLM for sampling
2
Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents
possible rationales and web actions to explore. While this simple search-strategy shows a meaningful
improvement in the success rate, it still struggles on long horizon tasks due to sparsity of environment
rewards. Indeed even a small mistake across the trajectory can cause the final agent output to be
wrong, creating significant credit assignment problems. To overcome this, we use AI feedback (Bai
et al., 2022) and self-criticism (Yuan et al., 2024) to further prompt the LLM to provide self-evaluation
feedback at each node, which serves as intermediate reward and helps guide the search steps. This
meaningfully improves the final agent success rate, but requires significant online interactions and
moreover the capability to rollback actions, which is not always possible in online realistic settings.
Such online autonomous search with little supervision on the web can result in a weak or unsafe
agent which can make many errors, resulting in risky behaviors in sensitive online settings like bank
transfers and sensitive information sharing.
To correct this, we use the traces generated by the search process to improve capabilities of the
model by learning from both the successful and unsuccessful trajectories with offline reinforcement
learning, utilizing the Direct Preference Optimization (DPO) algorithm. We create preferences over
different branches at the node level, which are scored using a mixture of the AI process feedback
rewards and the final success rate of the explored branch. We evaluate our approach on the simulated
WebShop benchmark (Yao et al., 2022)—a simulated e-commerce platform—as well as a real-world
reservations booking website. We utilize LLaMa 3-70B as the base model in our experiments. In the
WebShop environment, our approach consistently outperforms behavior cloning and reinforcement
learning fine-tuned baselines, and beats average human performance when equipped with the
capability to do online search.
In our real-world booking experiments, using our Agent Q framework we improve the model zero-shot
absolute success rate from 18.6% to 81.7% (a 340% relative increase), outperforming GPT-4’s
performance after a single day of autonomous data collection. When we equip Agent Q with online
search capability, our absolute success further improves to 95.4%. We believe that our approach
represents a significant step forward in the development of autonomous web agents through it’s
search and self-critique capabilities, setting a new benchmark for reliable multi-step decision-making
in interactive settings.