Venue: NAACL 2025

# xLAM: A Family of Large Action Models to Empower AI Agent Systems

Jianguo Zhang\({}^{*}\), Tian Lan\({}^{*}\), Ming Zhu\({}^{*}\), Zuxin Liu\({}^{*}\), Thai Hoang\({}^{*}\), Shirley Kokane\({}^{++}\), Weiran Yao\({}^{\dagger}\), Juntao Tan\({}^{\dagger}\), Zhiwei Liu, Yihao Feng, Juan Carlos Niebles,

Shelby Heinecke, Huan Wang, Silvio Savarese, Caiming Xiong Salesforce AI

Co-first Authors.Essential Contributors.

###### Abstract

Autonomous agents powered by large language models (LLMs) have attracted significant research interest. However, the open-source community faces many challenges in developing specialized models for agent tasks, driven by the scarcity of high-quality agent datasets and the absence of standard protocols in this area. We introduce **xLAM**, a series of large action models designed for AI agent tasks. The **xLAM** series includes five models with both dense and mixture-of-expert architectures, ranging from 1B to 8x22B parameters, trained using a scalable, flexible pipeline that unifies, augments, and synthesizes diverse datasets to enhance AI agents' generalizability and performance across varied environments. Our experimental results demonstrate that **xLAM** consistently delivers exceptional performance across multiple agent ability benchmarks, notably securing the 1st position on the Berkeley Function-Calling Leaderboard, outperforming GPT-4, Claude-3, and many other models in terms of tool use. By releasing the **xLAM** series, we aim to advance the performance of open-source LLMs for autonomous AI agents, potentially accelerating progress and democratizing access to high-performance models for agent tasks.

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 11583–11597

+
Footnote †: _Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 11583–11597

## 1 Introduction

The field of autonomous agents has witnessed significant advancements in recent years, with large language models (LLMs) playing a crucial role in enhancing agent capabilities across diverse tasks. Researchers have made substantial progress in developing sophisticated frameworks Hong et al. (2023); Team (2023); Wu et al. (2023); Xie et al. (2023) and specialized environments Deng et al. (2023); Yao et al. (2022); Zhou et al. (2023) to enhance agent capabilities, such as tool use Qin et al. (2024) and web browsing Zhou et al. (2023). Concurrently, comprehensive benchmarks like AgentBench Liu et al. (2023), ToolBench Qin et al. (2024), and AgentBoard Ma et al. (2024) have been established to rigorously assess agent performance in reasoning, planning, and multi-turn interactions.

While proprietary LLMs developed by industry leaders have demonstrated competitive performance in various agent tasks Anthropic (2024); OpenAI (2023); Reid et al. (2024); Touvron et al. (2023), the open-source community faces limited choices for specialized models in this domain. This scarcity stems from several challenges in adapting open-source LLMs to agent tasks, primarily due to the lack of comprehensive, high-quality datasets and the heterogeneity of existing data formats. These factors complicate the unification of diverse datasets and obstruct the learning of transferable knowledge across different agent tasks.

Recently, the agent research community has intensified efforts in open-source agent data processing and model training Qin et al. (2024); Chen et al. (2023); Xu et al. (2023); Patil et al. (2023); Zeng et al. (2023); Yin et al. (2023); Zhang et al. (2024). However, these works still face challenges in managing complex environments and generalizing to new scenarios, primarily due to limitations in the collected agent data. A major obstacle is the homogeneity of content and format in existing datasets, resulting in models that lack diversity across various tasks and struggle to adapt to new or slightly different data structures in practical applications. While previous efforts have attempted to design pipelines for unifying data, they typically cover only a few scenarios or lack flexibility in their unified formats. For instance, Lumos Yin et al. (2023) primarily addresses question answering, web agents, and mathematical tasks involving planning and grounding; while AgentOhana Zhang et al. (2024), despite encompassing a more diverse range of environments, lacks an extendable unified format to accommodate

new environments.

Moreover, open-source datasets often suffer from quality issues, such as incorrect agent outputs, hallucinated actions, and repeated interaction turns within trajectories (Zhang et al., 2024; Chen et al., 2024). The lack of detailed analysis and understanding of agent data further complicates these challenges, hindering the development of robust and versatile open-source agent models. Addressing them is crucial for advancing the field of open-source agent models and bridging the performance gap with proprietary LLMs in agent tasks.

In this work, we introduce and open-source **xLAM**, a series of powerful models with varying sizes. This diverse set is tailored for a variety of applications, with smaller models (1B and 7B) optimized for on-device deployment, while larger models (8x7B and 8x22B) are designed to tackle more challenging tasks. Alongside the model release, we offer several insights and lessons learned from our experience in agent model training:

* **Data Processing:** We highlight the importance of data unification and augmentation in enhancing dataset diversity and mitigating overfitting. Our developed data preprocess and augmentation pipeline significantly improves the generalizability of agent models across diverse environments.
* **Data Synthesis:** We showcase the impact of scalable, high-quality data synthesis on agent model performance. Our synthetic dataset enabled **xLAM** models to achieve 4 of the top 20 positions on the Berkeley Function Calling Leaderboard, including securing the top-1 spot, with smaller models achieving performance comparable to much larger counterparts, showing great potential in this direction.

We evaluate the **xLAM** series on public agent benchmarks, demonstrating exceptional performance across various agent tasks. By open-sourcing these models, we aim to advance open-source agent models and provide valuable insights into data processing and synthesis techniques, addressing key challenges in developing competitive alternatives to proprietary models.

## 2 Related Work

### LLM Agents

Recent advancements in LLMs have significantly enhanced their utility in various agent tasks. Several innovative prompt techniques have been developed to improve performance, including Chain of Thought Wei et al. (2022), ReACT Yao et al. (2023), and Reflection Shinn et al. (2023). Additionally, considerable efforts have been made to fine-tune open-sourced agent models for better capabilities Qin et al. (2024); Chen et al. (2023); Patil et al. (2023); Zeng et al. (2023); Zhang et al. (2024). These include enhancements in data collection and processing to facilitate effective agent learning Zeng et al. (2023); Li et al. (2023); Tang et al. (2023); Yin et al. (2023); Zhang et al. (2024); Chen et al. (2024), covering a range from simple question answering to more complex scenarios like web interactions, tool operations, reasoning, and planning. However, many of these agent frameworks still depend on proprietary models as their core engine to achieve optimal performance, revealing a substantial gap in the availability of high-quality open-source models for these tasks.

### Agent Benchmarks

A variety of benchmarks have been established to assess the abilities of LLM agents across diverse scenarios Yao et al. (2022); Qin et al. (2024); Liu et al. (2023); Ma et al. (2024); Huang et al. (2023); Liu et al. (2023); Wang et al. (2023); Liu et al. (2024); Du et al. (2024); Wang et al. (2024); Yan et al. (2024). Notably, AgentBench Liu et al. (2023), Mint-Bench Wang et al. (2023), and AgentBoard Ma et al. (2024) encompass environments ranging from code generation and games to web interactions and reasoning tasks. ToolBench Qin et al. (2024) specifically evaluates multi-turn reasoning and tool-usage abilities, while the Berkeley Function-Calling Leaderboard Yan et al. (2024) broadly assesses models' capabilities in function calling across various contexts.

## 3 Data Processing Pipeline

In this section, we discuss the data pipeline for training xLAM, including data unification, augmentation, quality verification, general instruction data synthesis, and preference data generation.

### Data Unification

Existing agent datasets are collected from diverse environments and designed in various formats, introducing noise and complicating data augmentation and verification. Models like NexusRaven Srinivasan et al. (2023), Gorilla-Openfunctions Cheng-Jie Ji et al. (2024), and 

AgentOhana (Zhang et al., 2024) have demonstrated superior performance in function-calling, suggesting that a well-defined, universal format could significantly enhance model performance. By standardizing the format of existing data, we can reduce noise and facilitate easier data augmentation and quality verification, leading to a more efficient and robust framework for model training and evaluation. Furthermore, a standardized format ensures consistency, simplifies model training, and enhances the model's ability to generalize across various benchmarks.

Function-calling formats form the basis for how models understand and execute tasks, motivating us to design our unified data format in a function-calling style. As illustrated in Figure 3, the unified format consists of several modules: task instruction, available tools, format instruction, few-shot examples, query, and steps. Specifically, the available tools define the agent's action space, and the format instruction specifies the output format the agent should follow when generating a response. In each step, the agent's output, the environment's feedback/execution results, and the user's follow-up input are organized into a dictionary. It's quite common for there to be purely conversational interactions between users and agents that don't trigger any APIs or receive corresponding observations. In these instances, the related entry values would simply remain empty.

This unified format is compatible with various environments and tasks, making our data processing pipeline adaptable to different datasets and scalable to large amounts of data. Moreover, the modularized design allows for fine-grained data augmentation and quality verification, which are essential in improving agent data quality. For example, by unifying all the available tools and tool calls, we can easily inspect for hallucination and function-call errors, and apply various augmentation techniques.

### Data Augmentation

Our data augmentation strategy focuses on improving the diversity of the data. It involves applying various transformations to the existing dataset, thereby generating new, synthetic data samples. The data unification step significantly simplifies the application of various augmentation techniques. A standardized data format ensures consistency and ease of implementation, allowing for more efficient augmentation processes. Specifically, the augmentation techniques we adopted can be categorized as prompt format augmentation and instructions-following augmentation.

**Prompt Format Augmentation:** Prompt format augmentation focuses on creating various prompt formats based on the structured, unified data format. The format augmentation can be further divided into two categories: 1) _Order Shuffling_. In the unified format, the available tools are provided in a list, and each tool contains the name, description, and parameters. To avoid model overfitting to the specific order of the tools, we randomly shuffle the original data set by adding a single number of objects from the underlying object. We then redefine the data set by adding a single number of objects from the underlying object. Finally, we redefine the data set by adding a single number of objects from the underlying object.

Figure 1: Overview of the data processing, training and evaluation of xLAM. We take the diagnostic feedback from the model evaluation results to iteratively improve the data quality.

 

fle the tool list. Furthermore, we also shuffle the order of the name, description, parameters, and within the parameters to present the information in different ways. We do the same thing within the tool_calls in each step. Additionally, we also shuffle the order of different sections of the input, including task instruction, tools, format instruction, few-shot examples etc. 2) _Concatenation Tokens_. Each training data point is a pair of input and output sequences. To convert the structured unified format to the training prompt, we use special tokens to concatenate different sections into one sequence. We create several different special token styles, including "[START/END OF QUERY]", "<query></query>", and plain text.

**Instruction-Following Augmentation:** It focuses on adding diversity to the instructions in order to improve the model's instruction-following capability. It involves rephrasing existing instructions and adding new instructions, without introducing inaccuracy and inconsistency. Therefore, verification of the new instructions is a crucial step for this type of augmentation. We employ two methods for instruction-following augmentation: 1) _Task Instruction Rephrasing_. We rephrase the task instructions using powerful LLMs to accommodate various input styles from users. To ensure the rephrased instructions still align with the original version, we verify them by prompting the LLMs with the rephrased instructions and check if the LLMs can still follow them and generate correct function calls. 2) _Format Instruction-Following_. In our unified format, the output format is a JSON string with thought and tool_calls. To avoid the model overfitting on JSON format and to enable the model to follow various output formats upon different format instructions, we prepare 15 different output formats along with their corresponding format instructions and format converters. The formats include JSON, XML, YAML, plain text, etc.

### Data Quality Verification

To further understand the data quality and to thoroughly investigate the sources of errors in the evaluation, we conduct a detailed analysis of the unified dataset. We identify a list of errors in the data using both rule-based and LLM-as-a-judge approaches.

**Undefined Function Call:** In function-calling, a list of available functions is provided, and the model should generate a function_call using one of the given functions. However, we found that in many cases, the predicted function_call is not from the given list. We match the predicted function with the given functions by comparing the function names and the list of parameter names. When the function_call name does not match any given functions, we refer to it as _Undefined Functions Invoked_. When the function name matches but the argument list contains undefined arguments, we refer to it as _Undefined Arguments Passed_. We also take into consideration optional parameters.

**Incorrect Argument Type:** Other than the error types mentioned above, we also observe that sometimes the model generates the correct argument's value, but in the wrong types. For example, when a parameter expects a [val1, val2, val3], the generated arguments is "[val1, val2, val3]", which is a string version of the list. When executing the function call, errors will occur due to incorrect data type. We identify trajectories containing the incorrect argument type error by comparing the parameter type in the available tools and the actual argument type. We also found that most argument type errors can be fixed by converting the arguments to the correct parameter types.

**Argument Hallucination:** Upon examining the unified dataset from public sources, we discovered that tool calls frequently include argument values not present in the user query or prior steps. This issue arises because much of this data is generated by LLMs, which are prone to hallucination, a common problem in LLM-generated content. We identified two types of hallucination: 1) the generated tool names or argument names do not appear in the provided tool and argument list; and 2) the argument values do not align with the user query or observations from previous steps. The first type of hallucination is straightforward to address by searching the generated tool call and argument names and matching them with the provided tool list, as they are all structured in JSON, making this process efficient. However, detecting the second type, where argument values are misaligned, is more challenging, as simple string matching is ineffective for complex queries and tasks. To tackle this, we use LLMs as judges to perform step-wise argument hallucination detection, detecting if there is a mismatch between the arguments and the intended query or prior observations.

**Low-Quality Reasoning and Planning:** We observe many data trajectories where the reasoning and planning steps are of low quality, which is a common issue in the outputs of many LLMs. To address this, we first filter out low 

quality data using rule-based methods informed by heuristics, then prompt models like Mixtral-8x22b-Instruct-v0.1 (Jiang et al., 2024) and DeepSeek-V2 (DeepSeek-AI, 2024) to evaluate both the overall trajectory and individual thought steps on the selected data. A portion of these rating results is then sampled and verified by humans. We also attempted to iterate on this process using specifically fine-tuned models.

### Data Synthesis

Based on our findings in Sec. 3.3, we observe that most of these publicly available datasets have several limitations. First, these datasets are often static, synthesized by weak models, limited in scope, and, more importantly, not verified by execution. Second, these datasets mainly focus on a single type of function-calling category, i.e., outputting a single function call based on the provided tools. However, real-world scenarios might consist of many other types of use cases, such as the parallel functionalities scenario Yan et al. (2024), where the user query contains multiple requests and the model should respond with concurrent function calls in parallel within a single response.

To address these two issues, we utilize a systematic data synthesis framework called APIGen Liu et al. (2024), to generate 50k verifiable data points based on a collection of 3,673 executable APIs. The key idea is a multi-stage verification process to ensure the accuracy and quality of the generated data. This process includes format verification, semantic verification and quality control as we developed in Sec. 3.3, along with execution verification. Together, these steps collectively help to identify and filter out low-quality data points, such as those with hallucination issues or inaccurate argument parameter values.

### Data Mixture

For supervised fine-tuning (SFT), our dataset combines training samples from three main sources: cleaned and augmented agent datasets, a synthetic function-calling dataset, and general instruction-tuning datasets. These sources are used to train the general xLAM models.

Specifically, to enhance the general instruction capability of xLAM, we integrate diverse instruction-tuning datasets from DialogStudio Zhang et al. (2023) and Data Provenance Longpre et al. (2023, 2024). This instruction data comprises 20% to 30% of our training set. To enhance the function-calling capability of xLAM-7b-fc-r and xLAM-1b-fc-r, we employ a targeted training approach, with 50% of their training data drawn from our high-quality synthetic function-calling dataset. The remaining 50% of the data is sampled from other tasks within our training set.

For Direct Preference Optimization (DPO) Rafailov et al. (2023), we prompt less powerful models to generate and rate responses for selected data from each source, then sample a subset for human verification. After adjustments to models and prompts, we classify the selected responses as rejected samples.

## 4 Model Training

### Modeling

We use a supervised fine-tuning (SFT) approach, further aligning model checkpoints with the DPO method, and leverage the robustness of our flexible data pipeline. Our training code is based on the HuggingFace Transformers and Accelerate librariesWolf et al. (2020); Gugger et al. (2022), as well as PyTorch FSDPZhao et al. (2023).

The fine-tuning of general xLAM models is conducted on Nvidia H100 GPUs. For SFT, we use a full fine-tuning framework that employs the fully sharded data parallel algorithm Zhao et al. (2023). In the case of xLAM-8x22b-r, we integrate LoRA Hu et al. (2021); Dettmers et al. (2023) to better preserve the model's original capacities and prevent catastrophic forgetting Liu et al. (2023). LoRA is also used for DPO alignment across all xLAM models. We utilize a total batch size of 128 and a learning rate that ranges from \(5\times 10^{-6}\), to \(5\times 10^{-5}\). Additionally, we use a cosine learning rate scheduler with 100 warm-up steps to optimize performance. Training times range from 6 to 40 hours.

### xLAM Model Series

We introduce a series of agent models tailored for different use cases. Our flagship model series, xLAM, is built upon the Mixrat Instruct Jiang et al. (2024) models and aims to achieve balanced performance across a diverse range of agent tasks, from complex multi-turn interactions to functionaling applications.

In addition to general xLAM models, we develop two specialized models for function-calling use cases, xLAM-7b-fc-r and xLAM-1b-fc-r, based on DeepSeek-Coder-7B-instruct-v1.5 and DeepSeek-Coder-1.3B-instruct, respectively Guo et al. ( 

2024). The smaller model sizes offer increased accessibility, allowing users to easily host them on a single GPU to address various function-calling tasks, ranging from simple user queries to parallel concurrent requests.

By offering a suite of models with varying sizes and specializations, the xLAM series caters to a wide range of user needs and computational resources, making powerful agent capabilities more accessible and adaptable to real-world applications.

## 5 Experiments

### Benchmarks

After considering the stability of environments and research budget limitations, we evaluate the performance of models across four rigorous benchmarks: Webishop Yao et al. (2022), ToolQuery Ma et al. (2024), ToolBench Qin et al. (2024), and the Berkeley Function-Calling Benchmark (BFCL) Yan et al. (2024). Each benchmark is designed to assess different aspects of model capabilities under a variety of settings and constraints.

**Webshop** is an interactive web environment designed to mimic online shopping experiences, testing an agent's ability to navigate and assist in e-commerce tasks. Webshop comprising approximately 250 test cases.

**ToolQuery** evaluates an agent's skills in using tools to retrieve and process information across domains. ToolQuery features 60 test cases across three distinct settings: Weather, Movie, and Academia. We use the testing configurations of AgentBoard Ma et al. (2024) with Success Rate and Progress Rate for both Webb shop and ToolQuery.

We also evaluate on **ToolQuery-Unified**, which is essentially ToolQuery but requires an agent to ingest the task instruction and tools following the augmented prompt format described in SS3.2 and likewise solve the task following the unified format. The purpose of testing agents in this setting is to assess their reasoning and tool-use abilities when evaluated on structured formats Tam et al. (2024).

**ToolBench** is developed for real-time evaluation of multi-turn reasoning and interactive capabilities via RapidAPI, includes around 1,000 test cases. It asks GPT-4 to evaluate Pass Rate of agent responses in both in-domain and out-of-domain settings, including unseen instructions with familiar tools, unseen tools within previously known categories, and entirely new categories of unseen tools. Due to space limit, we put testing results into Appendix A.2.

**Berkeley Function-Calling Benchmark** offers an extensive evaluation of an agent's ability to reason and execute function calls across various programming languages and application domains. With 2,000 test cases, it tests complex scenarios such as parallel and multiple function calls in languages including Java, JavaScript, and Python. The evaluation metrics include: Abstract Syntax Tree (AST) accuracy for non-executable test queries, executable accuracy by running the APIs to get the results, and relevance detection score which measures the agent's ability to distinguish non-relevant queries and provided tools. Since BFCL updates frequently, we used their stable V2 version, which has a cutoff date of September 19, 2024.

### Experimental Results

#### 5.2.1 Webshop and ToolQuery

**Webshop.** Table 1 presents detailed comparisons of state-of-the-art language and agent models in the Webshop and ToolQuery environments, illustrating the robust and strong performance of the xLAM models. In the Webshop environment, xLAM-7b-r not only achieves the highest Success Rate at 0.414, surpassing other general LLMs like GPT-4-0125-preview, GPT-4o-2024-0523, and Claude2, but also outperforms specialized agent models such as AgentOhana-8x7b and Lemur-70b. This shows xLAM's superior ability to navigate and execute tasks effectively in the web interactions.

**ToolQuery.** In the more complex and unseen ToolQuery environment, xLAM-8x7b-r and xLAM-8x22b-r also demonstrate high performance as shown in Table 1, ranking second with a Success Rate of 0.683. This shows a significant improvement over the baseline performance of Mixtral-8x7b-inst and Mixtral-8x22b-inst, which are 0.167 and 0.400, respectively. Notably, all three xLAM models surpass the Mixtral-8x22B-Instruct model. Despite Mixtral-8x22B-Instruct having a large number of parameters and specialized tuning for advanced functionalities such as function calling, reasoning, and complex tool usage, it falls short of the xLAM models' performance. Furthermore, same as other general LLMs, it lacks transparency regarding the data collection, unification processes, and other critical details, contrasting with the open source purposes provided for xLAM. These results show the efficacy of our proposed data unification and synthetic data pipeline.

**ToolQuery-Unified.** When the system prompt 

from ToolQuery is presented to the model in the unified format shown in Fig. 4, and the model is required to follow the provided format instructions to generate a structured output, we observe that xLAM's performances are more consistent compared to GPT models, as shown in Table 2. While GPT-4o's performance significantly degrades by 42% compared to ToolQuery, our best xLAM 8x22b model maintains comparable performance. This can be attributed to xLAM being trained on trajectories that adhere to the unified format, enabling it to perform consistently during inference. Concurrent research Tam et al. (2024) observed a similar decline in performance on reasoning tasks when LLMs are constrained to produce output in specific formats. Deeper analysis indicated that the degradation is more than just due to incorrectly formulated output in a specific format, but rather due to a drop in the reasoning ability of the model itself.

#### 5.2.2 Berkeley Function-Calling Benchmark

Table 3 presents the experimental results on the BFCL v2 benchmark (cutoff date 09/19/2024), which shows the exceptional performance of our xLAM model series in function-calling tasks. Notably, xLAM models secure four out of the top twenty positions, demonstrating the effectiveness of our data pipeline and training methodology across various model sizes.

Our flagship model, xLAM-8x22b-r, achieves the highest overall accuracy of 87.31%, surpassing all other models in the benchmark. This result validates the effectiveness of our data processing and model training pipeline in improving models' function-calling ability. Following closely, xLAM-8x7b-r ranks 6th, outperforming most prominent models including GPT-4o-mini and Claude-3.

The performance of our models demonstrates clear scaling with model size, a trend exemplified by xLAM-7b-r, which ranks 14th with an accuracy of 80.33%. This model outperforms several larger and more resource-intensive alternatives, including multiple GPT-4 and GPT-4o versions. Remarkably, our smallest model, xLAM-1b-fc-r, achieves a 32nd place ranking with an accuracy of 75.43%, surpassing much larger models like Claude-3-Opus (FC) and GPT-3.5-Turbo. This performance underscores the power of our data synthesis framework in producing high-quality, diverse datasets that enhance function-calling effectiveness even for smaller language models.

It is also worth noting that the BFCL v2 benchmark Yan et al. (2024) includes a live dataset released after our model training date. These fresh data are collected from real-world user queries that were entirely unseen by our models. Nevertheless, our models exhibit strong generalization capabilities in handling these real-world use cases. The consistently strong performance across our model

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & Success Rate & Academia & Movie & Weather \\ \hline
**xLAM-7b-r** & 0.466 (0.550) & 0.45 & 0.25 & 0.35 \\
**xLAM-8x7b-r** & 0.533 (0.683) & 0.45 & 0.40 & 0.45 \\
**xLAM-8x22b-r** & **0.733** (0.683) & 0.75 & 0.40 & 0.60 \\ \hline GPT+0125-preview & 0.566 (0.750) & 0.65 & 0.35 & 0.25 \\ GPF-no-2024@5-13 & 0.366 (0.633) & 0.45 & 0.20 & 0.25 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Testing results on ToolQuery-Unified. **Bold** and underline results denote the best result and the second best result for Success Rate, respectively. Values in brackets indicate corresponding performance on ToolQuery.

\begin{table}
\begin{tabular}{l c c c c} \hline \hline \multirow{2}{*}{LLM} & \multicolumn{2}{c}{Webshop} & \multicolumn{2}{c}{ToolQuery} \\ \cline{2-5}  & Success Rate & Progress Rate & Success Rate & Progress Rate \\ \hline
**xLAM-7b-r** & **0.414** & 0.767 & 0.550 & 0.674 \\
**xLAM-8x7b-r** & 0.410 & 0.763 & 0.683 & 0.745 \\
**xLAM-8x22b-r** & 0.390 & 0.763 & 0.683 & 0.758 \\ \hline GPT-4-0125-preview & 0.375 & 0.760 & **0.750** & 0.803 \\ GPT-4o-2024-0523 & 0.323 & 0.694 & 0.633 & 0.801 \\ AgentOhana=8x7b Zhang et al. (2024) & 0.331 & 0.737 & 0.533 & 0.766 \\ Claude2 & 0.378 & 0.746 & 0.483 & 0.735 \\ Mixxtl-8x22b-inst Jiang et al. (2024) & 0.383 & 0.739 & 0.400 & 0.740 \\ DeepSeek-6f7b-chat Bi et al. (2024) & 0.319 & 0.727 & 0.400 & 0.714 \\ GPT-3.5-Turbo-0125 & 0.323 & 0.749 & 0.367 & 0.545 \\ Llama3-70b-inst AlfaMeta (2024) & 0.299 & 0.746 & 0.367 & 0.526 \\ Lemur-70b-chatv1 Xu et al. (2023) & 0.116 & 0.718 & 0.283 & 0.720 \\ Mistral-8x7b-inst Jiang et al. (2024) & 0.222 & 0.766 & 0.167 & 0.654 \\ CodeLDA-3dab-inst Roziere et al. (2023) & 0.235 & 0.717 & 0.133 & 0.600 \\ Llama2-70b-chat Touvron et al. (2023) & 0.131 & 0.536 & 0.000 & 0.483 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Testing results on Webshop and ToolQuery. **Bold** and Underline results denote the best result and the second best result for Success Rate, respectively.

 

series, ranging from 8x22 billion to 1 billion parameters, demonstrates the scalability and versatility of our approach. This scalability is particularly noteworthy, as it enables strong results from compact models suitable for resource-constrained environments to large-scale models for more demanding applications. Furthermore, the ability of our smaller models to compete with much larger alternatives suggests significant potential for efficient deployment in various real-world scenarios.

### Ablation Study

We conducted an ablation study on the 7B models to measure the impact of various steps in our data pipeline. Three datasets were prepared for this analysis: raw data, augmented data, and augmented + cleaned data. The raw data represents the dataset before data unification, while the other two datasets are post-unification. Figure 2 presents the evaluation results of models trained on these three datasets. The metrics used for this evaluation are G1_instruction from ToolBench and success_rate from both Webbop and ToolQuery. The results indicate that augmented data consistently outperforms raw data across all metrics, with improvements of 2.3% on ToolBench, 5.8% on Webshop, and 18.3% on ToolQuery. Furthermore, the addition of data cleaning leads to a substantial performance increase on ToolQuery, with a further improvement of 23.4%. The results highlight the effectiveness of data augmentation and cleaning processes in the data pipeline.

## 6 Conclusion

This paper introduces xLAM series, a set of large action models for autonomous AI agents. Our models, ranging from 1B to 8x22B parameters, were trained with a scalable and flexible data pipeline that unifies, augments, and synthesizes diverse datasets. Evaluations show that xLAM models consistently perform exceptionally across various benchmarks. The insights we learned from training these models highlight the importance of rigorous data processing and the potential of data synthesis in developing capable AI agents. By releasing the xLAM series, we aim to democratize access to high-performance models for agent tasks, thereby accelerating progress in the field.

\begin{table}
\begin{tabular}{c|c c|c c c c|c c c|c c} \multirow{3}{*}{Rank} & \multicolumn{2}{c|}{Overall} & \multicolumn{3}{c|}{Abstract Syntax Tree (AST) Evaluation} & \multicolumn{3}{c|}{Evaluation by Execating APIs} & \multicolumn{3}{c}{Relevance Detection} \\  & Accuracy & Model & Simple & Multiple & Parallel & Parallel & Simple & Multiple & Parallel & Interleave & Relevance \\ \hline
**1** & **87.31** & **MAN822Bs/FCFC** & **72.79** & **86.37** & **87.13** & **84.75** & **98.57** & **94.00** & **92.00** & **185.00** & **74.96** & **97.56** \\
2 & 85.79 & GPT+0125-Preview Prompt & 78.82 & 88.44 & 91.00 & 83.75 & 99.00 & 96.00 & 82.00 & 80.00 & 61.35 & 97.56 \\
3 & 85.00 & GPT+140-Preview Prompt & 78.75 & 89.12 & 94.12 & 83.25 & 99.00 & 96.00 & 82.00 & 72.50 & 64.98 & 90.24 \\
4 & 84.74 & GPT+0163 Prompt & 78.76 & 85.46 & 91.75 & 82.67 & 98.29 & 96.00 & 86.00 & 70.00 & 75.57 & 82.93 \\
5 & 83.89 & GPT+140-D20409 Prepure & 80.47 & 88.81 & 88.12 & 84.25 & 99.00 & 96.00 & 80.00 & 77.50 & 61.82 & 82.93 \\ \cline{2-13}
**6** & **88.33** & **MAN822B/FCFC** & **73.32** & **80.09** & **72.00** & **82.50** & **92.57** & **96.00** & **90.00** & **72.57** & **72.52** & **92.65** \\ \hline
7 & 83.35 & GPT+012-mini20417s Preup & 75.88 & 81.64 & 85.12 & 79.42 & 98.29 & 94.00 & 82.00 & 77.50 & 79.20 & 80.49 \\
8 & 83.13 & GPT+02-d2045-03 (Dropout) & 76.18 & 86.01 & 92.12 & 81.00 & 98.00 & 94.00 & 76.00 & 72.50 & 77.44 & 78.05 \\
9 & 82.55 & Functionary-Medium-03.17 & 74.34 & 87.59 & 81.62 & 80.67 & 98.29 & 94.00 & 90.00 & 75.00 & 72.33 & 70.73 \\
10 & 81.78 & GPT+110-Preview PC (F) & 69.32 & 84.19 & 86.38 & 71.92 & 95.43 & 94.00 & 86.00 & 75.00 & 72.00 & 82.93 \\
11 & 81.59 & Lima++03-Intempt Prompt & 72.87 & 85.91 & 84.00 & 77.83 & 94.14 & 94.00 & 84.00 & 80.00 & 50.47 & 92.68 \\
12 & 80.88 & Claude-Jobs Propus (Prospent) & 76.65 & 87.47 & 73.88 & 75.17 & 98.57 & 94.00 & 82.00 & 75.00 & 56.15 & 85.37 \\
13 & 80.87 & GPT+0125-Preview PC (F) & 68.76 & 84.95 & 80.38 & 74.00 & 84.21 & 94.00 & 88.00 & 75.00 & 74.03 & 85.37 \\ \hline
**14** & **80.33** & **MAN822B/FCFC** & **60.98** & **84.00** & **63.00** & **79.17** & **75.71** & **94.00** & **92.00** & **80.00** & **72.88** & **92.65** \\
15 & 80.23 & Nemoto-34b-Hpn (FP) & 68.51 & 80.38 & 78.62 & 79.17 & 86.00 & 90.00 & 80.00 & 77.50 & 84.10 & 78.05 \\
16 & 80.21 & Foncianry-Small+v3.1 (FC) & 72.70 & 83.31 & 85.62 & 72.92 & 87.79 & 90.00 & 86.00 & 70.00 & 68.36 & 85.37 \\ \hline
**17** & **80.18** & **MAN822B/FCFC** & **70.92** & **78.27** & **73.58** & **88.50** & **95.21** & **90.00** & **88.00** & **72.57** & **97.50** & **84.09** \\
18 & 79.66 & mitral-large-2047 FC (FAC) & 81.01 & 87.42 & 94.00 & 83.50 & 98.29 & 92.00 & 86.00 & 77.50 & 93.4 & 100.00 \\
19 & 79.55 & GPT+202-d204-05-13 (FC) & 70.40 & 82.33 & 89.00 & 76.08 & 88.93 & 84.00 & 88.00 & 72.50 & 73.50 & 70.73 \\
**32** & **75.43** & **MAN81-Hp-FC** (FAC) & **64.63** & **72.33** & **64.90** & **61.42** & **80.21** & **92.00** & **86.00** & **75.00** & **66.05** & **97.56** \\
33 & 75.41 & GPT+3.75- turbo PC (F) & 69.79 & 83.58 & 71.88 & 68.83 & 95.14 & 88.00 & 86.00 & 57.50 & 35.83 & 97.56 \\
34 & 74.97 & Mittal-Nemeo-2407 GC Auto & 64.57 & 99.90 & 80.25 & 74.00 & 91.36 & 86.00 & 86.00 & 62.50 & 59.14 & 65.85 \\ \end{tabular}
\end{table}
Table 3: Performance comparison on BFCL-v2 leaderboard (cutoff date 09/19/2024). The rank is based on the overall accuracy, which is a weighted average of different evaluation categories. “FC” stands for function-calling mode in contrast to using a customized “prompt" to extract the function calls. The complete table can be found in the Appendix Table 5.

Figure 2: Ablation study for data augmentation and data quality verification (cleaning).

 



## 7 Limitations

Our data synthesis pipeline relies on large LLMs such as DeepSeek-V2 and Mixtral-8x22b-Inst. This dependence may result in extended data generation times when utilizing less powerful GPUs. Future work will focus on developing more efficient pipelines capable of operating effectively with both smaller and larger LLMs.

Additionally, due to the instability of some benchmarks and environments, along with the heavy reliance on API calls from commercial models such as OpenAI GPT-4, and constrained by limited research budgets, we face challenges in evaluating our models across even more benchmarks. Consequently, this may result in performance discrepancies on certain benchmarks. Moving forward, we aim to enhance our contributions to the research community by addressing instability and better maintaining and unifying existing benchmarks and environments.

## References

* A1@Meta (2024) Al@Meta. 2024. Llama 3 model card. External Links: Link Cited by: SS5.
* A. Anthropic (2024) The claude 3 model family: opus, sonnet, haiku. Claude-3 Model Card. External Links: Link Cited by: SS1.
* X. Bi, D. Chen, G. Chen, S. Chen, D. Dai, C. Deng, H. Ding, K. Dong, Q. Du, Z. Fu, et al. (2024) Deepseek llm: scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. Cited by: SS1.
* B. Chen, C. Shu, E. Shareghi, N. Collier, K. Narasimhan, and Shunyu Yao (2023) Fireact: toward language agent fine-tuning. arXiv preprint arXiv:2310.05915. Cited by: SS1.
* Zehui Chen, K. Liu, Q. Wang, W. Zhang, J. Liu, D. Lin, Kai Chen, and F. Zhao (2024) Agent-flan: designing data and methods of effective agent tuning for large language models. arXiv preprint arXiv:2403.12881. Cited by: SS1.
* Charlie Cheng-Jie Ji, H. Mao, F. Yan, S. Patil, T. Zhang, I. Stoica, and J. Gonzalez (2024) Gorilla openfunctions v2. In [https://gorilla.cs.berkeley.edu/blogs/7_open_functions_v2.html](https://gorilla.cs.berkeley.edu/blogs/7_open_functions_v2.html). Cited by: SS1.
* DeepSeek-AI (2024) Deepseek-v2: a strong, economical, and efficient mixture-of-experts language model. PreprintarXiv:2405.04434. Cited by: SS1.
* Xiang Deng, Y. Gu, B. Zheng, S. Chen, S. Stevens, B. Wang, H. Sun, and Yu Su (2023) Mind?web: towards a generalist agent for the web. arXiv preprint arXiv:2306.06070. Cited by: SS1.
* Tim Dettmers, Artidoro Pagnoni, A. Holtzman, and Luke Zettlemoyer (2023) Qlora: efficient finetuning of quantized llms. PreprintarXiv:2305.14314. Cited by: SS1.
* Yu Du, Fangyun Wei, and Hongyang Zhang (2024) Anytool: self-reflective, hierarchical agents for large-scale api calls. arXiv preprint arXiv:2402.04253. Cited by: SS1.
* Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan (2022) Accelerate: training and inference at scale made simple, efficient and adaptable. Note: [https://github.com/huggingface/accelerate](https://github.com/huggingface/accelerate) Cited by: SS1.
* D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. Li, et al. (2024) Deepsek-coder: When the large language model meets programming-the rise of code intelligence. arXiv preprint arXiv:2401.14196. Cited by: SS1.
* S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, C. Zhang, J. Wang, Z. Wang, S. Ka. Shiing Yau, Z. Lin, L. Zhou, C. Ran, L. Xiao, C. Wu, and J. Schnithuber (2023) MetagNet: meta programming for a multi-agent collaborative framework. PreprintarXiv:2308.00352. Cited by: SS1.
* Edward J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and Weizhu Chen (2021) Lora: low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. Cited by: SS1.
* Y. Huang, Jiawen Shi, Yuan Li, C. Fan, Siyuan Wu, Q. Zhang, Y. Liu, P. Zhou, Y. Wan, N. Zhenqiang Gong, et al. (2023) Metatool benchmark for large language models: deciding whether to use tools and which to use. arXiv preprint arXiv:2310.03128. Cited by: SS1.
* Albert Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, R. Savary, C. Bamford, D. Singh Chaplot, D. de las Casas, E. Bou Hanna, F. Bressand, et al. (2024) Mixtral of experts. arXiv preprint arXiv:2401.04088. Cited by: SS1.
* Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li (2023)Api-bank: a comprehensive benchmark for tool-augmented llns. In The 2023 Conference on Empirical Methods in Natural Language Processing, Cited by: SS1.
* Xiao Liu, Hao Yu, H. Zhang, Y. Xu, X. Lei, H. Lai, Y. Gu, H. Ding, K. Men, K. Yang, et al. (2023) Agentbench: evaluating lms as agents. arXiv preprint arXiv:2308.03688. Cited by: SS1.
* Zhiwei Liu, Weiran Yao, J. Zhang, L. Xue, S. Heinecke, R. Murthy, Y. Feng, Z. Chen, J. Carlos Niebles, D. Arpit, et al. (2023) Bolaa: benchmarking and orchestrating llvm-augmented autonomous agents. arXiv preprint arXiv:2308.05960. Cited by: SS1.



Zhiwei Liu, Weiran Yao, Jianguo Zhang, Liangwei Yang, Zuxin Liu, Juntao Tan, Prafulla K Choubey, Tian Lan, Jason Wu, Huan Wang, et al. 2024a. Agentlite: A lightweight library for building and advancing task-oriented llm agent system. _arXiv preprint arXiv:2402.15538_.
* Liu et al. (2024b) Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, Rithesh Murthy, Liangwei Yang, Silvio Savarese, Juan Carlos Niebles, Huan Wang, Shelby Heinecke, and Caiming Xiong. 2024b. Apigen: Automated pipeline for generating verifiable and diverse function-calling datasets. _arXiv preprint_.
* Liu et al. (2023c) Zuxin Liu, Jesse Zhang, Kavosh Asadi, Yao Liu, Ding Zhao, Shoham Sabach, and Rasool Fakoor. 2023c. Tail: Task-specific adapters for imitation learning with large pretrained models. _arXiv preprint arXiv:2310.05905_.
* Longpre et al. (2023) Shayne Longpre, Robert Mahari, Anthony Chen, Naana Obeng-Marnu, Damien Sileo, William Brannon, Niklas Muennighoff, Nathan Khazam, Jad Kabbara, Kartik Perisetla, et al. 2023. The data provenance initiative: A large scale audit of dataset licensing & attribution in ai. _arXiv preprint arXiv:2310.16787_.
* Longpre et al. (2024) Shayne Longpre, Robert Mahari, Ariel Lee, Campbell Lund, Hamidah Oderinwale, William Brannon, Nayan Saxena, Naana Obeng-Marnu, Tobin South, Cole Hunter, et al. 2024. Consent in crisis: The rapid decline of the ai data commons. _arXiv preprint arXiv:2407.14933_.
* Ma et al. (2024) Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. 2024. Agentboard: An analytical evaluation board of multi-turn llm agents. _arXiv preprint arXiv:2401.13178_.
* OpenAI (2023) OpenAI. 2023. Gpt-4 technical report. _ArXiv_.
* Patil et al. (2023) Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. Gorilla: Large language model connected with massive apis. _arXiv preprint arXiv:2305.15334_.
* Qin et al. (2024) Yujia Qin, Shihao Liang, Ying Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2024. ToollIm: Facilitating large language models to master 16000+ real-world apis. _ICLR_.
* Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. _arXiv preprint arXiv:2305.18290_.
* Reid et al. (2024) Mached Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittweiser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. _arXiv preprint arXiv:2403.05530_.
* Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Got, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jeremy Rapin, et al. 2023. Code Ilama: Open foundation models for code. _arXiv preprint arXiv:2308.12950_.
* Shinn et al. (2023) Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunya Yao. 2023. Reflexion: Language agents with verbal reinforcement learning. In _Thirty-severth Conference on Neural Information Processing Systems_.
* Srinivasan et al. (2023) Venkat Krishna Srinivasan, Zhen Dong, Banghua Zhu, Brian Yu, Damon Mosk-Aoyama, Kurt Keutzer, Jiantao Jiao, and Jian Zhang. 2023. Nexusraven: a commercially-permissive language model for function calling. In _NeurIPS 2023 Foundation Models for Decision Making Workshop_.
* Tam et al. (2024) Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. 2024. Let me speak freely? a study on the impact of format restrictions on performance of large language models. _arXiv preprint arXiv:2408.02442_.
* Tang et al. (2023) Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, and Le Sun. 2023. Toolalpaca: Generalized tool learning for language models with 3000 simulated cases. _arXiv preprint arXiv:2306.05301_.
* Team (2023) XAgent Team. 2023. Xagent: An autonomous agent for complex task solving.
* Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. LLama: Open and efficient foundation language models. _arXiv preprint arXiv:2302.13971_.
* Wang et al. (2024) Jize Wang, Zerun Ma, Yining Li, Songyang Zhang, Cailian Chen, Kai Chen, and Xinyi Le. 2024. Gta: A benchmark for general tool agents. _arXiv preprint arXiv:2407.08713_.
* Wang et al. (2023) Xingyao Wang, Zihan Wang, Jiatene Liu, Yangyi Chen, Lifan Yuan, Hao Peng, and Heng Ji. 2023. Mint: Evaluating llms in multi-turn interaction with tools and language feedback. _arXiv preprint arXiv:2309.10691_.
* Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. _Advances in neural information processing systems_, 35:24824-24837.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtovicz, et al. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations_, pages 38-45 

Qingyun Wu, Gagan Bansal, Ijeyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. Autoggen: Enabling next-gen llm applications via multi-agent conversation framework. _arXiv preprint arXiv:2308.08155_.
* Xie et al. (2023) Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Junning Zhao, Qian Liu, Che Liu, et al. 2023. Openagents: An open platform for language agents in the wild. _arXiv preprint arXiv:2310.10634_.
* Xu et al. (2023) Yiheng Xu, Hongjin Su, Chen Xing, Boyu Mi, Qian Liu, Weijia Shi, Binyuan Hui, Fan Zhou, Yitao Liu, Tianbo Xiao, et al. 2023. Lemur: Harmonizing natural language and code for language agents. _arXiv preprint arXiv:2310.06830_.
* Yan et al. (2024) Fanja Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Berkeley function calling leaderboard. [https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html](https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html).
* Yao et al. (2022) Shunya Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. _Advances in Neural Information Processing Systems_, 35:20744-20757.
* Yao et al. (2023) Shunya Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhik Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing reasoning and acting in language models. In _International Conference on Learning Representations (ICLR)_.
* Yin et al. (2023) Da Yin, Faeez Brahman, Abhilsha Ravichander, Khyathi Chandu, Kai-Wei Chang, Yejin Choi, and Bill Yuchen Lin. 2023. Lumos: Learning Agents with Unified Data, Modular Design, and OpenSource LLMs. _arXiv preprint arXiv:2311.05657_.
* Zeng et al. (2023) Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. 2023. Agenttuning: Enabling generalized agent abilities for lms. _arXiv preprint arXiv:2310.12823_.
* Zhang et al. (2024) Jianguo Zhang, Tian Lan, Rithesh Murthy, Zhiwei Liu, Weiran Yao, Juntao Tan, Thai Hoang, Liangwei Yang, Yihao Feng, Zuxin Liu, et al. 2024. Agentohana: Design unified data and training pipeline for effective agent learning. _arXiv preprint arXiv:2402.15506_.
* Zhang et al. (2023) Jianguo Zhang, Kun Qian, Zhiwei Liu, Shelby Heinecke, Rui Meng, Ye Liu, Zhou Yu, Silvio Savarese, and Caiming Xiong. 2023. Dialogstudio: Towards richest and most diverse unified dataset collection for conversational ai. _arXiv preprint arXiv:2307.10172_.
* Zhao et al. (2023a) Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. 2023a. Pytorch fsdp: Experiences on scaling fully sharded data parallel. _Preprint_, arXiv:2304.11277.
* Zhou et al. (2023b) Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Yonatan Bisk, Daniel Fried, Uri Alon, et al. 2023. Webarena: A realistic web environment for building autonomous agents. _arXiv preprint arXiv:2307.13854_.

 

Appendix

### Benchmarks

After considering the stability of environments and research budget limitations, we evaluate the performance of models across four rigorous benchmarks: Webb shop Yao et al. (2022), ToolQuery Ma et al. (2024), ToolBench Qin et al. (2024), and the Berkeley Function-Calling Benchmark Yan et al. (2024). Each benchmark is designed to assess different aspects of model capabilities under a variety of settings and constraints.

**Webshop** is an interactive web environment designed to mimic online shopping experiences, testing an agent's ability to navigate and assist in e-commerce tasks. Webshop comprising approximately 250 test cases.

**ToolQuery** evaluates an agent's skills in using tools to retrieve and process information across domains. ToolQuery features 60 test cases across three distinct settings: Weather, Movie, and Academia.

We use the testing configurations from AgentBoard Ma et al. (2024) for both Webshop and ToolQuery. These configurations assess overall performance using the Success Rate and evaluate progressive performance across interactive turns with the Progress Rate, with Success Rate being the more critic metric.

We additionally evaluate on **ToolQuery-Unified**, which is essentially ToolQuery but requires an agent to ingest the task instruction and tools following the augmented prompt format described in SS3.2 and likewise solve the task following the unified format. The purpose of testing agents in this setting is to assess their reasoning and tool-use abilities when evaluated on structured formats Tam et al. (2024).

**ToolBench** is developed for real-time evaluation of multi-turn reasoning and interactive capabilities via RapidAPI, and includes around 1,000 test cases. It uses Pass Rate as the metric, where the trajectory and final response are sent to GPT-4-0125-preview to determine whether the agent's final response successfully addresses the given user query. The evaluations cover both in-domain and out-of-domain settings, including unseen instructions with familiar tools, unseen tools within previously known categories, and entirely new categories of unseen tools.

**Berkeley Function-Calling Leaderboard Benchmark**Yan et al. (2024) provides a comprehensive evaluation framework for assessing an agent's capability to reason about and execute function calls across a variety of programming languages and application domains. The benchmark comprises over 2,200 test cases, challenging models with complex scenarios such as parallel and multiple function calls in languages like Java, JavaScript, and Python. The evaluation metrics include Abstract Syntax Tree (AST) accuracy for non-executable test queries, executable accuracy by running APIs to obtain results, and a relevance detection score that measures the agent's ability to distinguish non-relevant queries and provided tools.

Importantly, our evaluation utilizes the most stable BFCL v2 version, as of the cutoff date 09/19/2024. The v2 version introduces live function calls and real-world scenarios contributed by users, addressing issues such as data contamination, bias, and fairness by leveraging user-provided data. This updated dataset better reflects real-world distributions, characterized by a higher demand for selecting among multiple functions and a reduced demand for parallel function calls. For instance, our analysis indicates that in the v2 benchmark, the average number of available functions has doubled, while the average number of function calls has been halved compared to the non-live v1 data. It is important to note that all our models were trained prior to the release of the BFCL v2 live data.

### ToolBench

Table 4 presents the results on ToolBench, where xLAM models demonstrate impressive performance. They surpass both TooLIama-V2 and GPT-3.5-Turbo-0125 across all test settings. Moreover, xLAM models outperform AgentOhana-8x7b in scenarios involving unseen instructions and unseen tools, while achieving performance comparable to GPT-4-0125-preview in the unseen tools setting. These results show xLAM models' robust capabilities in multi-turn reasoning and complex tool usage, effectively handling both in-domain and out-of-domain tasks.

### Berkeley Function-Calling Leaderboard Benchmark

Table 5 is an extended version of Table 3. It provide a comprehensive table with the results of various models



* [10] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "A new approach to the detection of a single person's object in an image using deep neural networks," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 2378-2396, 2015.
* [11] M.-Y. Lin, R.-S. Huang, and S.-J. Zhu, "Deep residual learning for image recognition with deep convolutional neural networks," in _IEEE International Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 433-430, 2015.
* [12] D.-L. Lee, A.-M. Li, and L.-T. Wu, "Image classification by combining multiple layers of features from tiny images," in _Computer Vision and Pattern Recognition (CVPR)_, pp. 109-113, 2015.
* [13] H. Liu, X.-Q. Ma, and Z.-X. Sun, "Imagenet classification via deep convolutional neural networks," in _International Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 109-110, 2015.
* [14] P.-W. Lu, T.-B. Kim, and W.-D. Yuille, "Learning phrase representations from tiny images," in _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_, pp. 109-113, 2015.
* [15] G.-N. Li, K.-F. Chen, and Y.-C. Zhang, "Recurrent network attention models for semantic segmentation," in _Advances in Neural Information Processing Systems (IJASIP)_, vol. 10, no. 1, pp. 1-10, 2015.
* [16] E.-G. Le, B.-R. Fergus, I.-V. Van Derrike, and V.-E. Broxie, "Visualizing objects from tiny images," in _European Conference on Computer Vision (ECCV)_, pp. 1-10, 2015.
* [17] F.-K. Chen, Y.-C. Zhang, and Y.-C. Zhang, "The human visual representation model for semantic segmentation," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [18] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [19] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [20] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [21] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [22] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [23] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [24] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [25] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [26] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [27] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [28] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [29] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [30] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [31] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [32] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [33] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [34] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [35] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [36] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [37] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [38] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [39] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [40] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [41] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [42] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [43] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [44] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [45] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [46] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [47] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [48] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [49] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [50] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [51] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [52] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-10, 2015.
* [53] J.-P. Wang, C.-H. Chen, and Y.-C. Zhang, "Object recognition: A survey on the topic of interest," in _Proceedings of the 



**Prompt:**
[BEGIN OF TASK INSTRUCTION] Based on the previous context and API request history, generate an API request or a response as an AI assistant. [END OF TAK INSTRUCTION]

[BEGIN OF AVALABLE TOOLS]

[  {  "name": "get_fire_info",  "description": "Query the latest wildfire information",  "parameters": {  "location": {  "type": "string",  "description": "Location of the wildfire.",  "required": true,  },  "radius": {  "type": "number",  "description": "The radius (in miles) around the location.",  "required": false,  }  },...

]
[END OF AVAILABLE TOOLS]

[BEGIN OF FORMAT INSTRUCTION] Your output should be in the JSON format, which specifies a list of function calls. The example format is as follows. Please make sure the parameter type is correct. If no function call is needed, please make

 tool_calls an empty list "[]".  """

("thought": "the thought process, or an empty string", "tool_calls":  [("name": "api_name1", "arguments": ("argument1": "value1", "argument2":  "value2")]])  """

[END OF FORMAT INSTRUCTION]

[BEGIN OF QUERY] Can you give me the latest information on the wildfires occurring in California? [END OF QUERY]

[BEGIN OF HISTORY STEPS]

[  "thought": "Sure, what is the radius (in miles) around the location of the wildfire?",  "tool_calls": [],  "step_id": 1,  "next_observation": "",  "user_input": "User: Let me think... 50 miles."  },

]
[END OF HISTORY STEPS]

**Output:**

("thought": "", "tool_calls": [["name": "get_fire_info",  "arguments": ('location": "California", "radius": 50)]])

Figure 4: Example prompt and output for function-calling using xLAM.





\begin{table}
\begin{tabular}{|c||r c|c c r r|c c r|c c c|c c|} \hline \multirow{2}{*}{Rank} & \multirow{2}{*}{Overall} & \multicolumn{3}{c|}{Model} & \multicolumn{4}{c|}{Abstract Syntax Tree (AST) Evaluation} & \multicolumn{5}{c|}{Evaluation by Executing APIs} & \multicolumn{1}{c|}{Relevance Detection} \\ \cline{6-17}  & & Accuracy & & & Simple & Multiple & Parallel & \multicolumn{1}{c|}{Multiple} & Simple & Multiple & Parallel & \multicolumn{1}{c|}{Marlage} & \multicolumn{1}{c|}{Interleave} & Relevance \\ \hline
**1** & **87.31** & **GLANR@SE2DBC (FC)** & **72.79** & **86.37** & **87.13** & **84.75** & **89.57** & **94.00** & **92.00** & **85.00** & **74.96** & **97.56** \\ \hline
2 & 85.79 & GPT+0125-Preview (Prompt) & 78.82 & 88.44 & 91.00 & 83.75 & 99.00 & 96.00 & 82.00 & 80.00 & 61.35 & 97.56 \\
3 & 85.00 & GPT-4.1106-Preview (Prompt) & 78.75 & 89.12 & 94.12 & 83.25 & 99.00 & 96.00 & 82.00 & 72.50 & 64.98 & 90.24 \\
4 & 84.74 & GPT-4.0613 (Prompt) & 78.76 & 85.46 & 91.75 & 82.67 & 98.29 & 96.00 & 86.00 & 70.00 & 75.57 & 82.93 \\
5 & 83.89 & GPT-4 turbo-20.2409 (Prompt) & 80.47 & 88.81 & 88.12 & 84.25 & 99.00 & 96.00 & 80.00 & 77.50 & 61.82 & 82.93 \\ \hline
**6** & **83.83** & **SLANR@SE2DBC (FC)** & **73.12** & **86.09** & **71.00** & **82.50** & **92.57** & **94.00** & **90.00** & **77.50** & **72.35** & **92.68** \\ \hline
7 & 83.35 & GPT-4o mini-20240718 (Prompt) & 75.88 & 81.64 & 85.12 & 79.42 & 98.29 & 94.00 & 82.00 & 77.50 & 79.20 & 80.49 \\
8 & 83.13 & GPT-4o-2024-05-13 (Prompt) & 76.18 & 86.01 & 92.12 & 81.00 & 98.00 & 94.00 & 76.00 & 72.50 & 77.44 & 78.05 \\
9 & 82.55 & Functionary-Medium-v3.1 (FC) & 74.34 & 87.59 & 81.62 & 80.67 & 98.29 & 94.00 & 90.00 & 75.00 & 73.23 & 70.73 \\
10 & 81.78 & GPT-4i-1106-Preview (FC) & 69.32 & 84.19 & 86.38 & 71.92 & 95.43 & 94.00 & 86.00 & 75.00 & 72.70 & 82.93 \\
11 & 81.59 & Liamas-70B-Instruct (Prompt) & 72.87 & 85.91 & 84.00 & 77.83 & 94.14 & 94.00 & 84.00 & 80.00 & 50.47 & 92.68 \\
12 & 80.88 & Claude-3 Ops (Prompt) & 76.65 & 87.47 & 78.38 & 75.17 & 98.57 & 94.00 & 82.00 & 75.00 & 56.15 & 85.37 \\
13 & 80.87 & GPT-4.0125-Preview (FC) & 86.76 & 84.95 & 80.38 & 74.00 & 84.21 & 94.00 & 88.00 & 75.00 & 74.03 & 83.57 \\ \hline
**14** & **80.33** & **SLANR@SEDBC (FC)** & **69.35** & **84.00** & **63.00** & **79.17** & **75.71** & **94.00** & **92.00** & **80.00** & **72.88** & **92.68** \\ \hline
15 & 80.23 & Nemotron-3d0-inst (Prompt) & 68.51 & 80.38 & 78.62 & 79.17 & 86.00 & 90.00 & 80.00 & 77.50 & 84.10 & 78.05 \\
16 & 80.21 & Finucuirony-Small-v3.1 (FC) & 72.70 & 83.31 & 85.62 & 72.92 & 87.79 & 90.00 & 86.00 & 70.00 & 68.36 & 85.37 \\ \hline
**17** & **80.18** & **SLANR@SEDBC (FC)** & **78.52** & **78.22** & **73.38** & **68.50** & **95.21** & **99.00** & **80.00** & **77.50** & **79.54** & **80.49** \\ \hline
18 & 79.66 & mistral-large-2407 (FC Any) & 81.01 & 87.42 & 90.50 & 83.50 & 98.29 & 92.00 & 86.00 & 77.50 & 0.34 & 100.00 \\
19 & 79.55 & GPT-4o-2024-05-13 (FC) & 70.40 & 82.33 & 89.00 & 76.08 & 88.93 & 84.00 & 88.00 & 72.50 & 73.50 & 70.73 \\
20 & 79.25 & GPT-4o-min2024-07-18 (FC) & 67.83 & 80.16 & 85.38 & 77.17 & 83.21 & 92.00 & 82.00 & 70.00 & 71.83 & 82.93 \\
21 & 79.14 & Open Mitrist-8x2k2 prompt (Prompt) & 73.47 & 76.14 & 79.12 & 73.67 & 91.86 & 96.00 & 84.00 & 75.00 & 71.42 & 70.73 \\
22 & 79.10 & Gorilla-OpenFlunctions-v2 (FC) & 70.81 & 79.47 & 75.75 & 66.67 & 95.86 & 96.00 & 78.00 & 70.00 & 73.13 & 85.37 \\
23 & 79.09 & GPT-4 turbo-204-04-09 (FOOTNOTE:]Footnote [ENDFOOTNOTE] & 64.21 & 82.72 & 85.20 & 55.75 & 88.71 & 88.00 & 86.00 & 72.50 & 79.79 & 70.73 \\
24 & 78.96 & F