Title: Generalizing Experience for Language Agents with Hierarchical MetaFlows
Venue: NeurIPS 2025
Accepted: yes
Keywords: agent

workflow

large language models
Abstract: Recent efforts to employ large language models (LLMs) as agents have demonstrated promising results in a wide range of multi-step agent tasks. However, existing agents lack an effective experience reuse approach to leverage historical completed tasks. In this paper, we propose a novel experience reuse framework MetaFlowLLM, which constructs a hierarchical experience tree from historically completed tasks. Each node in this experience tree is presented as a MetaFlow which contains static execution workflow and subtask required by agents to complete dynamically. Then, we propose a Hierarchical MetaFlow Merging algorithm to construct the hierarchical experience tree. When accomplishing a new task, MetaFlowLLM can first retrieve the most relevant MetaFlow node from the experience tree and then execute it accordingly. To effectively generate valid MetaFlows from historical data, we further propose a reinforcement learning pipeline to train the MetaFlowGen. Extensive experimental results on AppWorld and WorkBench demonstrate that integrating with MetaFlowLLM, existing agents (e.g., ReAct, Reflexion) can gain substantial performance improvement with reducing execution costs. Notably, MetaFlowLLM achieves an average success rate improvement of 32.3% on AppWorld and 6.2% on WorkBench, respectively.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to reuse past experiences by constructing them into a hierarchical tree structure, and retrieving similar instances when solving future examples. This paper conducts SFT and RL training to enhance experience reusing performance. Empirical results on two common benchmarks demonstrate the effectiveness of the proposed MetaFlowLLM framework.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The proposed method is effective and achieves top performance on two well-known benchmarks, AppWorld and WorkBench, demonstrating the effectiveness of the proposed method.\n2. Coverage of both open-source (Qwen) and API-based (GPT) models, and showing improvements with both model families, demonstrating the general effectiveness of the proposed method. \n\nWeaknesses\n1. Missing baseline: Prior work [1] proposes inducing reusable, context-abstracted workflows from agent trajectories for later in-context reuse. The method, especially some key features highlighted in the proposed MetaFlow method, seems to overlap a lot with this prior work. It would further demonstrate the effectiveness of the proposed method if compared to AWM. Additionally, this work seems relevant enough to be added into the related work section.\n2. Ablation Study Needed: Necessity of Organizing Workflows in A Hierarhical Tree: while organizing data into tree structures sounds more complex, I wonder how effective this design is under empirical studies. More specifically, have you compared with a flat structure and directly retrieve from this pool of experiences? How does the performance compare to the more complex tree structure? How much more/less compute does the tree structure require?\n3. Ablation Study Needed: Showcasing the Necessity of SFT and RL Training Phrases: while the end result shows increases over the baselines, it is unclear how much the SFT training and RL training contribute to the final improvements, respectively. Ablation studies on (i) only doing SFT training, and (ii) only doing RL training, would provide more insights on why and how much these components improve the system.\n4. Clarification Needed: (i) on the definition of static and dynamic Nodes: I couldn’t find definitions of “static steps” and “dynamic steps” in the paper anyway, which makes the descriptions in the introduction and methodology sections more confusing (figure 1 does not help clarify these terms too); (ii) following the figure comment, some major figures to illustrate the method pipeline, e.g., Figure 1, I find it somewhat hard to understand, refining the figure a bit could be helpful.\n\n[1] Wang, Zora Zhiruo, et al. \"Agent workflow memory.\" ICML 2025.\n\nQUESTIONS:\n1. Efficiency Comparison: While figure 4 shows the improved efficiency with MetaFlow, when measuring the number of LLM and tool calls; these are only a portion of the computation caused during the process. I wonder have you calculated other costs, e.g., neural-based retrieval, additional costs because of lengthened contexts with retrieved instances, and how ReAct compare to ReAct+Metaflow?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MetaFlowLLM, a framework that enables language agents to learn from past experiences by building a hierarchical tree of reusable workflows. These workflows, called MetaFlows, combine static, executable steps with dynamic steps that require an LLM agent to complete. This approach leads to significant improvements in task success rates and efficiency on benchmarks like AppWorld and WorkBench.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The central idea of a hierarchical experience tree composed of hybrid static-dynamic workflows is a significant contribution. It presents an elegant middle ground between trajectory-based methods, which can be too specific, and guideline-based methods, which are often too abstract.\n2. The empirical results are impressive and demonstrate the effectiveness of the framework. The consistent and substantial performance gains shown across multiple base models and agent frameworks on two challenging benchmark support the paper's claims.\n\nWeaknesses:\n1. Figure 2, which serves as the main overview of the framework, is overloaded with information and difficult to interpret. The right-hand side in particular, which combines the training pipeline, hierarchical merging, and inference traversal, is too complex. The authors should consider splitting this into several smaller, more focused diagrams to improve readability.\n2. The paper lacks sufficient detail on the implementation of the baseline methods. How the \"Trajectory-based\" and \"Guideline-based\" baselines are implemented is not mentioned.\n3. The ablation study in Table 4 shows that reinforcement learning provides a very significant performance boost over just supervised fine-tuning. The key intuition behind how RL improves the performance requires further explanation.\n4. The analysis in Figure 4 convincingly shows a reduction in LLM API calls during task execution. However, the total cost of the MetaFlowLLM system also includes the offline training of MetaFlowGen and, more importantly, the online cost of the deep matching traversal, which itself uses an LLM as a discriminative model. The authors should provide a more holistic analysis of the cost-benefit trade-off, comparing the total inference cost (traversal + execution) of MetaFlowLLM against a standard ReAct agent.\n\nQUESTIONS:\nSee weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the challenge of leveraging existing experiences (i.e., trajectories) for solving sequential decision-making tasks. The authors point out the limitations of two existing approaches: trajectories provided as-is to guide solving new tasks might be too specific to the original tasks from the experiences, whereas trajectories summarized into higher-level guidelines may be too coarse-grained. Tackling this, the authors propose MetaFlowLLM with the notion of MetaFlows, which are nodes of the tree. In their framework, two similar (meta-)experiences can be merged to create their parent node, where the parent node contains a series of steps to solve the meta-task. Each of the steps can be either a (static) concrete workflow that is directly executable or a (dynamic) subtask that requires an LLM to solve differently depending on distinct task instances. For this merging process, they train a model named MetaFlowGen, using supervised fine-tuning using correctness-based rejection sampling followed by reinforcement learning fine-tuning with correctness and efficiency rewards. Incorporating the set of experiences, they repeat merging the two most similar nodes to construct the full experience tree. For inference, givne a task of interest, they find the node with the most alignment by evaluating node-level matching with LLM prompting. Empirically, they show that their framework, MetaFlowLLM, improves the Qwen2.5 (7B and 32B) and GPT-4o-mini agents in AppWorld and WorkBench.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The overall method is sound. Many of the components in the proposed framework are reasonably defined with fair design choices. For instance, allowing dynamic steps and interleaving static and dynamic steps could be an approach to find a good balance between fine-grainedness and coarse-grainedness.\n- Using the efficiency reward could be a reasonable way to tackle the granularity issue, which often arises when dealing with the abstraction of experiences (or trajectories). Even though each MetaFlow generation process does not take into account the global distribution of experiences or tasks, as the efficiency reward encourages keeping static steps, it mitigates the concern of obtaining overly coarse MetaWorkflows.\n- The authors present their experiments in AppWorld (coding tasks) and WorkBench (tool calling tasks) with the results that outperform the trajectory-based and guideline-based baselines in most cases.\n\nWeaknesses\n- Although the correctness reward looks like a fair measure for evaluating the newly constructed MetaFlow in terms of how well it covers all the existing task instances, the computation of the correctness reward seems quite costly. Especially, in order to compute a single correctness reward for a MetaFlow, that MetaFlow needs to be executed N times where N is the number of all the leaf nodes it has. The correctness reward computation is needed for both training of MetaFlowGen and constructing the experience tree. This may become a bottleneck if one attempts to scale up this framework to real-world problems.\n- On a related topic, I have some other concerns regarding the scalability of this work to more complex scenarios, such as ones with larger task spaces, where much more experiences are needed to provide enough guidance to agents. In this framework, identifying two similar experiences or MetaFlows and merging them to construct their parent MetaFlow (node) are the key steps of this framework. It means that the number of MetaFlow generation processes that need to be performed is (roughly, because it may also use the pure-agent node as a parent) proportional to the number of experiences. Combined with the first weakness I mentioned above, this may limit the scalability of the proposed approach. The number of experiences used in this work (237 for WorkBench and 90 for AppWorld) may not be big enough to demonstrate the scalability of the approach.\n- The quality of the constructed experience tree may be largely affected by the quality of the distance metric and how tasks are written. The construction of the experience tree is a greedy process of iteratively choosing and merging the two most similar MetaFlows. In this work, all-MiniLM-L6-v2 with the cosine similarity of task embeddings is used as the distance metric. all-MiniLM-L6-v2 is one of the popular choices for sentence embedding, but the similarity estimation based on task instruction embeddings may not be enough to fully encode complex behavior spaces.\n- The empirical results lack apples-to-apples comparison with the trajectory-based and guideline-based baseline approaches where stronger models are used as agents. For instance, the current Table only shows the performance of GPT-4o without any experience reuse methods.\n\nQUESTIONS:\n- I think expanding the experimental results with some of the stronger models as agents may strengthen the argument empirically in terms of comparing with the trajectory-based and guideline-based methods. Can you do similar empirical comparisons with GPT-4o or stronger models?\n- Regarding the scalability of the proposed approach, can you provide empirical measurements or trends of how the needed time and resources increase as the number of experiences increases? Other comments about the scalability would be appreciated, as well.\n\nMy score could be updated based on the response to the questions/suggestions above.\n\nLIMITATIONS:\nYes, regarding the potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper introduces a concept of experience tree, MetaFlow, that encodes trajectory-level information from past experiences. This helps agents to abstract and apply the right knowledge from past experience to adapt and generalize into new scenarios.

STRENGTHS AND WEAKNESSES:
Strengths:
1. Intuitive and effective mechanism for creating MetaFlow -- meta patterns from similar trajectories that can help generalize over unknown tasks.
2. Great performance across the board with two important benchmarks. 
3. Well-written paper; however, the figure could have been improved. 

Weaknesses: 
1. Unclear what it means "how to abstract MetaFlows given arbitrary two task execution trajectories" -- what are two trajectories? 
2. The method only works when you have reasonable examples for a type of metaflow.
3. I do not see a sample efficiency plot -- without it, it is hard to know if doing RL training in this case is cost-effective or not. 
4. I could not immediately find if there are experiments that actually learn from past trajectories rather than gold trajectories.

QUESTIONS:
I am curious how you would create the metaflow when the gold trajectories encode patterns that are not explicit? Would you need a stronger reasoning model?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-03