Agent Workflow Memory

Authors: Zhiruo Wang, Jiayuan Mao, Daniel Fried, Graham Neubig
Primary area: foundation or frontier models, including LLMs
Keywords: agent, web navigation, memory augmentation, online

Abstract:
Despite the potential of language model-based agents to solve real-world tasks such as web navigation, current methods still struggle with long-horizon tasks with complex action trajectories. In contrast, humans can flexibly solve complex tasks by learning reusable task workflows from past experiences and using them to guide future actions. To build agents that can similarly benefit from this process, we introduce Agent Workflow Memory (AWM), a method for inducing commonly reused routines, i.e., workflows, and selectively providing workflows to the agent to guide subsequent generations. AWM flexibly applies to both offline and online scenarios, where agents induce workflows from training examples beforehand or from test queries on the fly. We experiment on two major web navigation benchmarks -- Mind2Web and WebArena -- that collectively cover 1000+ tasks from 200+ domains across travel, shopping, and social media, among others. AWM substantially improves the baseline results by 24.6% and 51.1% relative success rate on Mind2Web and WebArena while reducing the number of steps taken to solve WebArena tasks successfully. Furthermore, online AWM robustly generalizes in cross-task, website, and domain evaluations, surpassing baselines from 8.9 to 14.0 absolute points as train-test task distribution gaps widen.

Decision: reject
Reviewer scores: 5, 6, 3, 5, 5

Reviews and discussion:
<thread>
creation date:
1730681359331

thread_type:
official_review

<thread_object>
creator: reviewer_fnjl

<content>
summary:
The authors propose an agent workflow memory that induces, augments and uses the most common functions as part of an existing library that the agent can take advantage of. This can be operated in both offline and online scenarios i.e. in the presence or absence of labeled samples. They evaluate the approach on two challenging  web navigation benchmarks mind2web and web arena and show that this simple and effective approach generalizes well across web navigation benchmarks.

strengths:
#### Strengths:
1. **IMemory Utilization:**
   - AWM's ability to integrate workflows into the agent’s memory, allowing for more sophisticated task handling and adaptation over time, is noteworthy. This is an advancement in making agents more autonomous and capable

2. **Empirical Validation:**
   - The empirical results showing AWM's performance boosts of 24.6% and 51.1% in relative success rates on the Mind2Web and WebArena benchmarks respectively are impressive. These results provide evidence of the workflow memory working to some level of competency.

3. **Generalization across Domains:**
   - The method's robustness across tasks, websites, and domains, particularly in environments where no annotated examples exist, highlights its potential for real-world applications and its adaptability.

weaknesses:
#### Areas for Improvement, please take this with a pinch of salt:
1. **Writing and Clarity:**
 - the paper needs a good amount of work on writing to meet the publication standards.  Can avoid unnecessary floating images and tables. Perhaps can save space by removing much of the unnecessary text and allowing natural space. More importantly, the overall story and clarity and motivation at places need improvement on why something is being discussed.

2. **Workflow Induction Clarity:**
   - The work could benefit from a more detailed explanation of the workflow induction process, particularly how the workflows are selected and refined during the induction phase. More insights into the criteria for choosing particular workflows over others would enhance the reader's understanding of the method's inner workings.

3. **Discussion on Limitations:**
   - While the paper discusses the success of the AWM model, a more thorough exploration of its limitations, especially in scenarios where the agent encounters completely novel tasks or domains not covered by the workflows, would provide a balanced view of its applicability.

4. **Comparison with State-of-the-Art Methods:**
   - Expanding the comparison with current state-of-the-art methods, not just in terms of task success rates but also considering factors like computational efficiency and scalability, would provide a clearer picture of standings.

questions:
In the feedback from fellow reviewers and the authors, I would appreciate a discussion or clarity on the terminology used in the paper. The approach described appears to be a knowledge-retaining technique, akin to the reuse of skills stored in a skills library for related tasks, as seen in other literature such as Voyager (Wang et al., 2023) or works on counterfactual reasoning. These sources typically refer to reused routines as "skills" rather than "workflows," and use "library" instead of "memory" for storage. It would be beneficial for the paper to acknowledge its inspirations from prior works, especially in the context of web navigation applications. This acknowledgment could help clarify the rationale behind the choice of terminology right from the start.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730791030585

thread_type:
official_review

<thread_object>
creator: reviewer_wfe1

<content>
summary:
This work proposes agent workflow memory as an approach to building web agents that accumulate knowledge from their experiences as workflows. Specifically, it converts each successful trajectory it obtains to a workflow, which consists of a workflow description and workflow trajectory. It stores those workflows on a per-website basis. For inference, the corresponding workflows are incorporated into the agent's input and used for solving the tasks. The authors conduct experiments on WebArena and Mind2Web and demonstrate that their approach outperforms the compared baseline methods.

strengths:
- Acquiring domain-specific knowledge in a re-usable form can be useful and the proposed approach is reasonable. Especially, the authors try to acquire shorter chunks of re-usable skills as workflows, which could be important for inceasing the utility of such workflows.
- Overall, the writing is easy to follow, especially with the well-visualized figures.
- They perform additional empirical analyses, such as the form of workflows and how useful those acquired workflows are. These kinds of analyses can be helpful in constructing a deeper understanding of the proposed approach.

weaknesses:
- The proposed approach accumulates experiences as workflows on a website basis (L196). While this can be effective in focusing on a specific set of websites, it also comes with a limitation in terms of generalization to different websites. This may also become a practical limitation, as there are myriad websites available and preparing various scenarios for most of them could be quite challenging. On benchmarks, this can be a factor that makes the empirical evaluation of the generalization capabilities of this approach difficult. Although this work tests the proposed approach in the online setting, existing benchmarks are based on pre-defined sets of websites.
- Although the online acquisition of workflows in a continual manner can be helpful in practice, comparing only the online version with the baseline approaches on the benchmark tasks can be unfair or misleading. Usually, getting information about other test tasks can bring noticeable improvement in the performance in most benchmarks, mainly because of the similarities in the tasks and the bigger data size. For a fairer comparison without the knowledge about the "held-out" test tasks, one can try gathering experiences (and thus workflows) without access to any of the tasks from the test set and evaluating on each of the test tasks without updating the set of workflows.
- In Table 4 for the Mind2Web results, the numbers for MindAct in the cross-domain split are the GPT-3.5 results, not the GPT-4 results.

questions:
Please take a look at the weaknesses section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730953439417

thread_type:
official_review

<thread_object>
creator: reviewer_nqfk

<content>
summary:
This paper introduces Agent Workflow Memory (AWM), a method for improving language model agents in complex, long-horizon tasks by learning and reusing task workflows. AWM flexibly supports both offline and online applications, showing strong improvements on web navigation benchmarks Mind2Web and WebArena. Results demonstrate notable gains in task success rates, efficiency, and cross-domain generalization.

strengths:
1. Using reusable routines from agent trajectories is a fresh and effective idea for complex task management.
2. Extensive evaluations, including ablation studies, validate AWM’s effectiveness across online and offline settings.
3. The concept of workflows is well-explained in Introduction, making AWM’s approach easy to understand.

weaknesses:
1. Figure 4 could be refined to better illustrate the workflow in the online setting.

2. In WebArena, if AWM leverages test set tasks to build workflows, it could cause data leakage and compromise fairness.

3. The paper does not show AWM’s sensitivity to task exploration order, which could affect its performance. For example, which task should be explored first? Task sequence could impact the quality of workflows induced—precisely, whether exploring tasks from difficult to easy (or vice versa) could degrade performance. Analyzing how task order affects workflow quality and overall results would provide valuable insight into AWM's robustness.

4. In Table 1, “# steps” is unclear. Reporting tokens or time would better reflect agent efficiency.

5. Section 5's description of “wrapping each workflow into a high-level function” needs more explanation and details.

6. In Table 4, MindAct consistently achieves a higher AF1 score than AWM. It would be helpful if the authors could explain this.

questions:
AWM’s high performance in domains like Maps deserves more detailed analysis to explain these gains, which can help to understand methods.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730990700240

thread_type:
official_review

<thread_object>
creator: reviewer_nwwu

<content>
summary:
The paper presents the Agent Workflow Memory method for extracting and using task workflows for web navigation task.
The proposed memory is a collection of subroutines each of which is comprised from a task description and a set of actions generated by the agent or taken from the training dataset.

AWM is implemented with GPT-3.5 and GPT-4 and evaluated on WebArena and Mind2Web datasets.

strengths:
- The proposed method enables online and offline variants of generating the task workflows to use both high-quality train data and past experiences of the agent.
- The authors conduct the ablation study and evaluations on two web navigation datasets.
- The paper is well-written and easy to follow.

weaknesses:
- The paper would benefit from the descriptions of how the agent was implemented and tested. The authors provide the model prompt and example workflow prompts in appendix, but considering that the agent is based on the closed-source models, the lack of implementation details impedes reproducibility.
- It is not clear how the workflows are extracted from memory to improve the task solving or the whole memory content is always provided as a part of the prompt.

questions:
- How many past experiences can be stored in memory?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732742748231

thread_type:
official_review

<thread_object>
creator: reviewer_1vxx

<content>
summary:
Paper describes an approach for agents to learn from past experiences using the notion of reusable task workflows and adapt to complex tasks. Overall, the paper is well written. However, establishing the novelty of the proposed method. By providing a clear differentiation from existing work would make this contribution’s impact more stronger.

strengths:
- Very flexible method which can sustain both offline and online cases.
- Strong improvement over benchmarks even those containing expertly labeled examples.
- Examination of workflow quality along with ablations is well written

weaknesses:
- The memory integration part is sub-optimal, where workflow integration is a simple addition. Using a more human-like mechanism such as attention based memory access or some sort of a weighted retrieval would be better.
- As I was reading the paper, hierarchical reinforcement learning came to mind. Having some analysis on how AWM compares with learning hierarchical policies implicitly would be helpful.

questions:
- How does AWM handle situations where the induced workflows are incorrect or suboptimal? 
- Why bother grouping actions into workflows instead of simply learning which action to take in each state?
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper develops an approach to summarize interactions with past web-agent-like environments to improve the performance on a given prompt. The essential approach consists of converting a successful trajectory (either obtained by the agent, or from the dataset) into a “workflow” for each website. For each website, the agent retrieves these workflows into its working memory for inference.

Based on comments of the reviewers, the authors have made a number of improvements to the manuscript during the rebuttal phase. The key concerns that are yet to be addressed however pertain to (i) clarity of the narrative that describes the approach with sufficient details and rigor, and (ii) ascertaining whether the approach unduly benefits from the correlations of tasks used to create workflows with new test tasks (and evaluating against appropriate baselines in this setting). Regarding the second point, it would be interesting if this approach can use workflows that are shared across websites and control for the number (or length) of these workflows explicitly. Regarding the first point, it is also important to use absolute success rate (as opposed to “relative improvement in the success rate” as the paper does at many places). The authors are also encouraged to compute error bars (e.g., via bootstrap) for the numerical results in the tables to ensure statistical significance of these results.