GOAT: A Training Framework for Goal-Oriented Agent with Tools

## Abstract

Abstract Large language models (LLMs) have recently been extended beyond traditional text generation to serve as interactive agents capable of using external tools based on user intent.
However, current LLM agents still show limited ability to handle goal-oriented queries, which require decomposing a high-level objective into multiple interdependent API calls with correct planning and execution.
Current approaches mainly rely on zero-shot evaluation due to the absence of training data. While proprietary closed-source models such as GPT-4 demonstrate strong reasoning abilities, smaller open-source models struggle to perform complex tool use effectively.
Thus, we propose a novel training framework GOAT , which enables fine-tuning of LLM agents in a human annotation-free setting. GOAT automatically constructs synthetic datasets of goal-oriented API execution tasks directly from given API documents, equipping models with the ability to reason over interdependent calls and generate coherent responses.
Through extensive experiments, we show that GOAT-trained agents achieve state-of-the-art performance across multiple existing goal-oriented benchmarks. In addition, we introduce GOATBench , a new goal-oriented API execution benchmark, and demonstrate that agents trained with GOAT also excel in this setting.
These results highlight GOAT as a practical path toward building robust open-source LLM agents capable of complex reasoning and tool use.

## 1 Introduction

Recent advances in large language models (LLMs) have led to remarkable progress across a wide range of natural language processing tasks (Zhao et al., 2025; Achiam et al., 2023).
Beyond their success in pure language understanding, a growing line of research explores a new paradigm, where LLMs function as agents that can actively interact with the external world by connecting to tools such as APIs to respond to user needs (Qin et al., 2024; Patil et al., 2023; Schick et al., 2023).
This paradigm, referred to as tool learning, highlights the shift from treating LLMs purely as text generators to enabling them to plan tasks, invoke external tools, and provide answers to user queries.

While prior literature on tool learning includes relatively simple tasks such as single-step queries (Patil et al., 2023; Schick et al., 2023) or fine-grained queries with explicitly specified instructions (Qin et al., 2023; Liu et al., 2024; Shen et al., 2023b), we focus on more realistic and challenging scenarios, which we term as goal-oriented tasks.
As illustrated in [Figure 1](https://arxiv.org/html/2510.12218v1#S1.F1), a goal-oriented user query provides only a high-level objective of the task rather than detailed step-by-step instructions, requiring the agent to break down the objective into intermediate steps, determine which APIs to call, and infer appropriate arguments for the selected API functions. Consequently, these tasks demand strong reasoning for task decomposition, long-horizon planning, and call realization that captures the interdependency between APIs, followed by their coordinated execution.

However, progress on such goal-oriented tasks has been limited by the lack of training data: constructing datasets that capture inter-API dependencies needs human annotation and is prohibitively costly at scale.
Consequently, current approach (Song et al., 2023) mainly rely on zero-shot evaluation, where models are expected to perform complex reasoning without task-specific supervision.
To address these challenges, we propose a novel training framework for Goal-Oriented Agent with Tools (GOAT), which enables fine-tuning LLM agents even in the absence of human-annotated data through a fully automatic synthetic data generation pipeline.
Our approach builds on the common practice that once a target set of APIs is specified, their function documentation is already available, which can be directly leveraged to construct training data.
Specifically, given API documentations of a target environment, GOAT first induces an API dependency graph through a refinement pipeline, retaining only feasible invocation relations between APIs.
From this graph, GOAT samples connected subgraphs, representing interdependent subtasks of a goal-oriented task.
These subgraphs are converted into training samples by instantiating and executing the corresponding APIs.
GOAT then generates a goal-oriented user query and a final response aligned with the resulting API sequence.
Finally, GOAT jointly fine-tunes the LLM and the retrieval model on the resulting dataset, equipping them with the ability to reason over interdependent APIs and produce coherent responses.

Furthermore, leveraging our data generation pipeline together with human labeling, we curate GOATBench, an evaluation benchmark for goal-oriented tasks.
Through extensive experiments on goal-oriented benchmarks including GOATBench, we show that GOAT enables open-source models to achieve state-of-the-art performance in multiple benchmarks, in some cases even surpassing certain closed-source models with strong reasoning capabilities. Our main contributions are as follows:

Figure: Figure 1: Goal-oriented API execution task. To solve a goal-oriented user query, the LLM agent performs step-by-step task planning, executes a sequence of interdependent API calls, and generates a natural language response. The figure illustrates the workflow where the user query is decomposed into subtasks, mapped to API calls, and each function call is executed by filling API arguments based on the outputs of previous responses, ultimately yielding a coherent answer.
Refer to caption: x1.png

- •
We propose GOAT, a novel training framework that automatically constructs goal-oriented API execution datasets from the target API documents without human annotation, enabling efficient domain-specific adaptation of LLM agents and strengthening their reasoning capabilities.
- •
Through extensive experiments on goal-oriented benchmarks, we demonstrate that agents trained with GOAT achieve state-of-the-art performance with open-source models.
- •
We further introduce GOATBench, a new evaluation benchmark for goal-oriented tasks, and confirm consistent performance gains of GOAT-trained agents on this benchmark as well.

## 2 Related Work

Task Formulations in Tool Learning Tool learning tasks can be categorized by the level of reasoning required between user queries and API calls.
In the simplest setting, a few APIs are directly provided in the prompt, so models learn only to operate within this restricted set in straightforward ways, making the task considerably easier and more constrained (Yao et al., 2023; Schick et al., 2023; Surís et al., 2023; Zhuang et al., 2023).
A more advanced formulation arises when many tools are available, requiring retrieval of the relevant one.
Among these, some tasks provide queries that explicitly specify each step-by-step instructions, leaving little need for planning and reasoning (Patil et al., 2023; Qin et al., 2023; Shen et al., 2023a; Yang et al., 2023; Shen et al., 2023b; Liu et al., 2024).
In contrast, the most challenging and realistic setting, which we target in this work, is the goal-oriented task, where user queries describe only high-level goals rather than explicit execution steps.
Here, the system must plan and execute a sequence of interdependent API calls to address the query holistically (Song et al., 2023; Li et al., 2023).
While closed-source models have achieved promising results in this setting, absence of training data makes boosting the performance of open-source models challenging.
Our work addresses this gap by introducing a training framework designed to handle realistic, goal-oriented tasks.

**Table 1: Comparison of existing synthetic training data generation works in tool learning. Our work uniquely generates synthetic training data specifically targeting goal-oriented tasks. *Note that although API-Bank provides a benchmark that includes goal-oriented tasks, its training data generation process does not target such queries.**
| Work | Real API | Fully Automatic | Scalable | API Call Dependency | Goal-oriented Query |
| --- | --- | --- | --- | --- | --- |
| ToolFormer (Schick et al., 2023) | ✗ | ✓ | ✗ | ✗ | ✗ |
| Gorilla (Patil et al., 2023) | ✓ | ✓ | ✓ | ✗ | ✗ |
| ToolLLM (Qin et al., 2023) | ✓ | ✓ | ✓ | ✗ | ✗ |
| API-Bank* (Li et al., 2023) | ✗ | ✓ | ✗ | ✗ | ✗ |
| TaskBench (Shen et al., 2023b) | ✓ | ✗ | ✗ | ✓ | ✗ |
| ToolFlow (Wang et al., 2024b) | ✓ | ✓ | ✓ | ✓ | ✗ |
| Magnet (Yin et al., 2025) | ✓ | ✓ | ✓ | ✓ | ✗ |
| ToolDial (Shim et al., 2025) | ✓ | ✓ | ✓ | ✓ | ✗ |
| Ours (GOAT) | ✓ | ✓ | ✓ | ✓ | ✓ |

Synthetic Training Data for Tool Learning High-quality training data is essential for developing LLM agents that can reliably use tools in real-world scenarios.
Earlier efforts generated single-API training data by first selecting an API function and then constructing a corresponding query–api call pair (Schick et al., 2023; Patil et al., 2023).
Subsequent work expanded to multi-API settings by randomly sampling multiple APIs, generating a combined query and the corresponding API call path, which resulted in parallel step-by-step instructions rather than high-level goals (Qin et al., 2023; Li et al., 2023).
Graph based approaches build API dependency graph to sample connected API sequences, but mainly to yield either fine-grained instructions (Shen et al., 2023b) or multi-turn dialogues (Wang et al., 2024b; Yin et al., 2025; Shim et al., 2025), with each node corresponding to a single API call.
The dependencies modeled in these graph-based frameworks are relatively weak, as they are derived from simple heuristics or algorithms without validating whether the connections are truly meaningful, and none of the above generate holistic goal-oriented queries that require agents to plan over multiple interdependent API calls.
In contrast, our approach automatically constructs such goal-oriented data, thereby addressing this gap and enabling effective training for realistic tool-use scenarios (see [Table 1](https://arxiv.org/html/2510.12218v1#S2.T1)).

## 3 A training framework for G oal- O riented A gent with T ools

We propose a novel human-annotation-free training framework for Goal-Oriented Agents with Tools (GOAT), motivated by the prohibitive cost of collecting manual annotations.
In contrast to prior works that primarily rely on zero-shot inference with pretrained LLMs alone, we begin with the practical observation that agents are typically deployed in specific domains with a fixed set of target APIs.
This makes it reasonable to assume that API documentation for these APIs is available beforehand.
Leveraging this assumption, our framework (i) automatically generates training samples from the available API documentation, and (ii) fine-tunes an LLM and retriever model on these samples to strengthen its goal-oriented reasoning capabilities for the target API environment.

### 3.1 Automatic Dataset Construction

A central challenge in training goal-oriented agents is the absence of manually annotated training datasets, which prevents models from acquiring task-specific supervision.
Our framework addresses this challenge by fully automatically constructing synthetic training data, thereby eliminating the need for costly human annotations.
It is important to note that intermediate API calls in goal-oriented tasks are inherently interdependent, as earlier calls are often executed to prepare inputs for later ones. In our framework, the generated synthetic data must therefore capture this interdependency across API calls. Such dependency information is typically implicit in API documentation, and we leverage this dependency cues during the data generation process.
Specifically, our method constructs goal-oriented API execution data through two main stages. First, given a set of API documents describing API functions, we build an API dependency graph that captures all possible ways in which the output of one API can serve as an input to another ([Section 3.1.1](https://arxiv.org/html/2510.12218v1#S3.SS1.SSS1)).
From the resulting graph, we extract connected subgraphs to create synthetic data points, each consisting of a goal-oriented user query $u$, a set of call units $\{(s,c,o)\}$—each consisting of a sub-query $s$, the corresponding API call $c$, and its output $o$—and the final response $r$ ([Section 3.1.2](https://arxiv.org/html/2510.12218v1#S3.SS1.SSS2)).

Figure: Figure 2: The overview of API dependency graph construction process. Given the API documents, each document is first parsed to extract function descriptions, which are then used to initialize a raw dependency graph in (a). This graph is progressively refined through three filtering steps (c)-(e), resulting in the final API dependency graph that captures reliable relations among APIs. The graphs shown under (b)-(e) illustrate how the API dependency graph evolves as it is progressively refined through each filtering step.
Refer to caption: x2.png

#### 3.1.1 API Dependency Graph Construction

Given a set of API documents $\{\text{doc}(n_{i})\}$ describing API functions $n_{i}$ with implicit dependency information, we formulate an API dependency graph $G=(\mathcal{V},\mathcal{E})$, where $\mathcal{V}=\{n_{i}\}$ represents the set of API functions and an edge in $\mathcal{E}=\{(n_{i},n_{j},k)\}$ indicates that the output of an API function $n_{i}$ can be used as the $k$-th argument in a subsequent call to $n_{j}$. Since a single output may serve multiple parameters for the same API function, $G$ is a multidigraph that allows multiple directed edges between the same pair of nodes.
The dependency graph captures the input–output dynamics across APIs, reflecting the execution flow and forming the basis for goal-oriented workflows.

To construct a correct dependency graph $G$, we leverage LLMs for their strong reasoning capabilities: a high-performing LLM generates candidate arguments for a source API, executes them, and verifies whether the resulting output can populate the parameters of a destination API. However, performing this procedure exhaustively across all input–output pairs is prohibitively expensive. To balance reliability and efficiency, we instead begin with an over-complete graph and progressively reduce LLM usage through a staged filtering pipeline: inexpensive coarse filters eliminate clearly incompatible edges, while only a shrinking subset is escalated to increasingly precise (and costlier) checks. The effect of each stage is detailed in [Section A.2](https://arxiv.org/html/2510.12218v1#A1.SS2).
The overall process is illustrated in [Figure 2](https://arxiv.org/html/2510.12218v1#S3.F2), and each stage is described in detail in the following.

API Document Parsing and Graph Initialization We begin by extracting the input and output specifications of each API function $n_{i}$ by parsing its corresponding documentation $\text{doc}(n_{i})$, which define the endpoints for edges in the graph (see [Figure 2](https://arxiv.org/html/2510.12218v1#S3.F2)a).
In particular, each specification includes the natural language description of parameters and output; we denote the description of the $k$-th input of function $n_{i}$ as $\text{In}(n_{i},k)$ and the output description of $n_{i}$ as $\text{Out}(n_{i})$.
The extraction is accomplished by prompting an LLM with the template described in [Section I.1](https://arxiv.org/html/2510.12218v1#A9.SS1), and the resulting structured representations are illustrated at the bottom of [Figure 2](https://arxiv.org/html/2510.12218v1#S3.F2)a and detailed further in [Section A.1](https://arxiv.org/html/2510.12218v1#A1.SS1).
Using these extracted specifications, we then construct an initial fully connected multidigraph by adding directed edges from the output of each API function to every input parameter of every other API function (see [Figure 2](https://arxiv.org/html/2510.12218v1#S3.F2)b). This over-complete graph tentatively connects all possible API pairs and serves as the starting point for subsequent filtering and refinement.

Filtering via Embedding Similarity To prune unlikely edges, we compare the descriptions of source API outputs and destination API input obtained from the previous step. For each edge $(n_{i},n_{j},k)$, we compute the cosine similarity between Sentence-BERT (SBERT) (Reimers & Gurevych, 2019) embeddings of $\text{In}(n_{j},k)$ and $\text{Out}(n_{i})$. Edges with similarity scores below a threshold $\tau$ are discarded. Note that since this method does not require invoking an LLM, it provides a very efficient way to filter out clearly incompatible pairs. At the same time, because more precise filtering will be applied in later stages, we set $\tau$ with a low threshold—favoring recall over precision—to avoid over-pruning and to retain most potentially valid connections. This filtering step is illustrated in Figure [2](https://arxiv.org/html/2510.12218v1#S3.F2)c.

Filtering via LLM As illustrated in Figure [2](https://arxiv.org/html/2510.12218v1#S3.F2)d, edges that survive the similarity filter are further examined leveraging the reasoning capability of LLMs with a single LLM call per edge. Given API documents ($\text{doc}(n_{i})$ and $\text{doc}(n_{j})$) and the descriptions ($\text{In}(n_{j},k)$ and $\text{Out}(n_{i})$), the LLM determines whether the output of $n_{i}$ can meaningfully populate the $k$-th input of $n_{j}$ using the prompt in [Section I.2](https://arxiv.org/html/2510.12218v1#A9.SS2). When the LLM judges the validity of edges, the LLM also generates a natural-language justification explaining why an edge is valid for edges deemed valid. We reuse these justifications later in data construction ([Section 3.1.2](https://arxiv.org/html/2510.12218v1#S3.SS1.SSS2)) to guide argument generation.
This filtering step checks for semantic plausibility based on descriptions, without grounding in actual values.

Filtering via API Call Execution As illustrated in Figure [2](https://arxiv.org/html/2510.12218v1#S3.F2)e, candidate edges are finally validated through execution with three LLM calls per edge. Unlike the previous semantic-only check, this step grounds edges in concrete values from real API executions. For each edge $(n_{i},n_{j},k)$, the LLM instantiates a call $c_{i}$ for $n_{i}$ by generating plausible arguments and execute it to obtain output $o_{i}$. It then constructs a destination API’s call $c_{j}$ for $n_{j}$, filling the $k$-th argument with content extracted from $o_{i}$, and finally verifies whether $c_{j}$ is coherent and executable as a continuation of $c_{i}$ ([Section I.3](https://arxiv.org/html/2510.12218v1#A9.SS3)).
By validating edges through real outputs, we eliminate connections that are semantically plausible but non-executable, ensuring that the final dependency graph represents reliable API workflows.

Figure: Figure 3: Overview of goal-oriented API execution data construction. The process involves (a) sampling connected API sequences, (b) generating API calls, outputs, and sub-queries, and (c) composing user queries and final responses.
Refer to caption: x3.png

#### 3.1.2 Goal-Oriented API Execution Data Construction

Once the dependency graph is finalized, we construct goal-oriented task samples where an user query leads through a sequence of interdependent API calls, to a final response. This process consists of three main steps. First, since the dependency graph is composed of all possible API relationships, we extract connected subgraphs and treat them as candidate API sequences. Second, for each sequence, we sequentially instantiate and execute each API functions by filling in their input arguments, executing them in order, and passing API outputs forward as inputs to subsequent calls. In doing so, we also generate sub-queries that explain each API call in natural language. Finally, based on the sub-queries and the outputs obtained from calling a API sequence, we construct a user query that captures the overall intent, and then generate a natural final response that interprets the outputs in their full context to address this query. The result is a dataset of samples where every component—from query to intermediate reasoning to final response—is grounded in real API executions.
The full process is illustrated in [Figure 3](https://arxiv.org/html/2510.12218v1#S3.F3).

Sampling Connected API Sequences As illustrated in [Figure 3](https://arxiv.org/html/2510.12218v1#S3.F3)a, we begin by extracting all possible connected subgraphs from $G$, with up to $L=4$ nodes, each of which serves as the basis of a task instance. If the subgraph contains cycles, we randomly break edges, producing an acyclic structure. The resulting acyclic multidigraph is then topologically sorted to produce an execution order $(n_{k_{1}},n_{k_{2}},\dots,n_{k_{L}})$ where $k_{l}$ represents the $l$-th node index among the $L$ nodes. This guarantees that APIs are invoked in a dependency-respecting sequence, forming a valid workflow.

API Call and Sub-Query Generation Following the predetermined sequence, each $n_{k_{\ell}}$ is instantiated into an API call $c_{k_{\ell}}$ by filling the required input arguments ([Section I.4](https://arxiv.org/html/2510.12218v1#A9.SS4)).
For each parameter, there are two cases: (i) when the subgraph contains an edge leading to it, where the argument should be obtained from the output $o_{k_{m}}$ of a prior API $(m<\ell)$, and (ii) when no such edge exists.
In the latter case, the LLM refers to the function’s specification in $\text{doc}(n_{k_{\ell}})$ to synthesize a plausible value.
In the former case, the result $o_{k_{m}}$ is provided, and the LLM extracts appropriate fields to populate the argument.
Since outputs are often complex and nested, non-trivial reasoning over the execution context is required; here, we prompt the LLM with the justification produced during the second filtering stage ([Section 3.1.1](https://arxiv.org/html/2510.12218v1#S3.SS1.SSS1)) to guide correct value selection.
Once all arguments are filled, the API call is executed to obtain $o_{k_{\ell}}$.
Alongside each API call, we generate a natural-language sub-query $s_{k_{\ell}}$ that explicitly corresponds to the operation performed by $c_{k_{\ell}}$.
This is constructed based on the API specification, the concrete arguments in API call $c_{k_{\ell}}$, and the execution context, ensuring that each sub-query reflects both the function being invoked and its role in the overall task flow ([Section I.5](https://arxiv.org/html/2510.12218v1#A9.SS5)).
By iteratively applying this process through the ordered sequence, we construct the full trajectory of call units $\{(s,c,o)\}$, where each element is grounded in real execution and contributes to a coherent, goal-oriented task. This process is depicted in Figure [3](https://arxiv.org/html/2510.12218v1#S3.F3)b.

User Query and Response Generation Figure [3](https://arxiv.org/html/2510.12218v1#S3.F3)c depicts the final stage, where once the full set of call units $\{(s,c,o)\}$ has been constructed, we generate a goal-oriented user query $u$ that encapsulates the overall task. The query is created by summarizing the sub-queries $\{s\}$ and abstracting the high-level objective that the entire API workflow is designed to achieve, as detailed in [Section I.6](https://arxiv.org/html/2510.12218v1#A9.SS6). Finally, we generate the final response $r$ corresponding to the user query $u$. For this, we provide the LLM with $u$ together with the full set of triplets ${(s,c,o)}$, as described in [Section I.7](https://arxiv.org/html/2510.12218v1#A9.SS7). This allows the model to produce a coherent answer that is grounded in the outputs of the composed API calls.

#### 3.1.3 Discussions

Reliable Supervision through Call-First Generation Although not targeting goal-oriented tasks, prior studies (Patil et al., 2023; Qin et al., 2023; Tang et al., 2023; Li et al., 2023) have explored using LLMs to construct synthetic data for agent training.
They generally follow an instruction-first strategy (Wang et al., 2023), where an LLM first generates a user query from API functions, and then produces API calls with arguments corresponding to that query.
However, this strategy has intrinsic limitations.
Converting a user query into API calls is precisely the capability we aim to improve in LLMs, thus training sets produced by LLMs themselves create a self-reinforcing loop:
the model continues to succeed on queries it already handles correctly, but fails to improve on cases where it was originally deficient.
This limitation persists even with human verification, as the surviving examples remain biased toward “easy” instances.
Consequently, instruction-first generation has primarily been used for distillation, transferring the knowledge of larger closed models into smaller open models.
In contrast, we adopt a call-first strategy.
Here, concrete API calls are instantiated directly from API document, executed to obtain outputs $(c,o)$, and then summarized into a natural language query.
This design is advantageous because generating a query from a complete API call is substantially easier for an LLM—requiring only abstraction over given inputs and outputs—than the inverse task of inferring API calls from natural language. By leveraging this easier direction of inference, we transform the summarization and abstraction strengths of LLMs into reliable supervision signals for the reverse direction: learning to map user queries into executable API workflows.

##### Benchmark Construction for Goal-Oriented Tasks.

Existing benchmarks for goal-oriented tasks, such as RestBench (Song et al., 2023) and API-Bank (Li et al., 2023), are built by human annotators who manually identify API relationships and construct corresponding queries and call paths. This costly process severely limits scalability and coverage, leaving existing benchmarks small in size and restricted to evaluation only.
In contrast, GOAT automatically captures dependencies across APIs and constructs goal-oriented data in a fully annotation-free manner.
This enables cost-effective collection of large-scale data suitable for training.
Moreover, with GOAT, evaluation benchmarks with new API sets can be built efficiently using human verification, which is substantially cheaper than full annotation.
We further compared a subset of GOAT-generated samples (after human verification) with fully human-annotated data from Song et al. (2023) and Li et al. (2023), and found no substantial difference (examples are in [Appendix C](https://arxiv.org/html/2510.12218v1#A3)).
Following this, we built GOATBench using the StableToolBench environment (Guo et al., 2024) with APIs from RapidAPI.
Details of the benchmark dataset construction and additional examples are provided in [Appendix D](https://arxiv.org/html/2510.12218v1#A4).

### 3.2 Agent Training

A central contribution of GOAT is that it turns a setting without any human-annotated training data into a learnable setup by automatically constructing synthetic supervision.
On top of this dataset, we train an LLM agent consisting of a language model and an API retriever.
The language model is instruction-tuned with supervision derived from the generated goal-oriented API execution data.
Specifically, the model learns to plan the correct sequence of calls and fill in their arguments, and it also receives supervision for generating the final natural-language response by aggregating the execution outputs.
To improve generalization beyond specific argument patterns, argument values are masked during training.
The retriever is trained using ground-truth query–API document pairs, enabling it to map user queries to the relevant API specifications.
Together, this framework enables effective learning of LLM agents tailored for domain-specific API tasks, where both planning and execution are grounded in real API behavior.

## 4 Experiments

### 4.1 Experiment Setup

We evaluate LLM Agents on goal-oriented API execution benchmarks where no human-annotated training data is available.
For each benchmarks, we use Llama-3-70B-Instruct (AI@Meta, 2024) to construct the corresponding synthetic training data within GOAT.
While filtering via embedding similarity, we set the hyperparameter $\tau$ to 0.2 for GOATBench and APIBank,
while a more conservative threshold of 0.05 is adopted for RestBench.
While training, the LLM is fine-tuned with Low-Rank Adaptation (LoRA) method (Hu et al., 2021) to enable parameter-efficient fine-tuning on API execution tasks.
For the retriever, we fine-tune a dense retrieval model based on the SBERT architecture (Reimers & Gurevych, 2019), specifically using the all-MiniLM-L6-v2 encoder using InfoNCE loss.
All hyperparameters and implementation details are provided in [Appendix B](https://arxiv.org/html/2510.12218v1#A2).

As a Baseline for prompting agents, we followed the decomposition-first method from prior work (Huang et al., 2024);
where given a user query, the agent retrieves the top-$k=5$ relevant API documents and predicts the entire sequence of API calls in a single step.
The planned sequence is executed iteratively, with each call incorporating outputs of previous ones, and the final answer is composed from the collected results.
This baseline was the strongest among those we tested and was thus chosen as our main baseline, with additional results for the others provided in [Appendix E](https://arxiv.org/html/2510.12218v1#A5).

### 4.2 Evaluation Benchmarks

RestBench (Song et al., 2023) RestBench is a human-generated benchmark that consists of two test sets built on real-world APIs from TMDB (movie database) and Spotify (music streaming). The evaluation is based on three metrics: (i) Success%, measured by human evaluation to assess whether the model result fulfills the user’s request; (ii) Correct Path%, which counts a case as correct if the gold API call path is contained as a subsequence within the model-generated path, where only the sequence of API functions matters and parameter values are disregarded; and (iii) $\Delta$ Solution Length, defined as the mean number of additional API calls inferred, measured only over successful cases, thereby reflecting the efficiency of the generated plan.

API-Bank (Li et al., 2023) API-Bank is constructed with human-implemented API functions in Python, where both the queries and their corresponding API call paths are manually annotated. It consists of three task sets, among which we focus on the Plan+Retrieve+Call set, as it uniquely reflects the goal-oriented query setting involving multi-API and multi-call reasoning. The official evaluation metrics include (i) Correctness, measured by the precision of API call responses, and (ii) ROUGE, computed between the model’s final response and the gold response. Since these metrics do not directly capture overall task success, we additionally report Success% and Correct Path%, following the definitions in RestBench.

GOATBench GOATBench is a human-verified benchmark on real APIs from RapidAPI, constructed through our data generation pipeline with additional human curation (see [Appendix D](https://arxiv.org/html/2510.12218v1#A4) for details).
Tasks are categorized into Single Tool and Inter Tool, depending on whether multiple APIs come from the same tool or from different tools.
For evaluation, we adopt three commonly used metrics:
(i) API Selection Accuracy (SA), measuring the Jaccard similarity between the predicted and ground-truth sets of API functions (Wang et al., 2024a);
(ii) API Invocation Accuracy (IA), similar to SA but requiring all arguments to match in addition to the function name; and
(iii) Success Rate (SR), a GPT-based evaluation metric that determines whether the final answer sufficiently and correctly solves the user query given the tool execution results (Qin et al., 2023).
For SR evaluation, we employ GPT-4.1 and provide the exact evaluation prompt in [Figure 21](https://arxiv.org/html/2510.12218v1#A11.F21).

**Table 2: Experiment results on RestBench. Closed-source and RestGPT results are reported numbers from original paper, shown here for reference. Metrics are Success%, Correct Path%, and $\Delta$ Solution Length. For Vicuna-13B, we additionally reproduced RestGPT using the released code and found substantially lower performance than reported (marked with * in the table).**
|  |  | Prompting | GOAT | TMDB | Spotify |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Backbone | Method | FT | Success% $\uparrow$ | CP% $\uparrow$ | $\Delta$ Len. $\downarrow$ | Success% $\uparrow$ | CP% $\uparrow$ | $\Delta$ Len. $\downarrow$ |
| Closed-source | text-davinci-003 | Baseline | - | 29.0 | 33.0 | +1.52 | 14.5 | 36.4 | +1.10 |
| text-davinci-003 | DEPS | - | 38.0 | 43.0 | +1.20 | 19.3 | 43.8 | +1.74 |  |
| text-davinci-003 | ReAct | - | 44.0 | 57.0 | +0.76 | 54.5 | 49.1 | +0.31 |  |
| text-davinci-003 | Reflexion | - | 52.0 | 59.0 | +1.37 | 59.6 | 61.4 | +1.68 |  |
| text-davinci-003 | RestGPT | - | 75.0 | 79.0 | +0.55 | 72.7 | 74.5 | +0.25 |  |
| Open-source | Llama2-13B | RestGPT | - | 0.0 | 0.0 | - | 0.0 | 0.0 | - |
| Llama2-13B | Baseline | ✗ | 0.0 | 0.0 | - | 3.5 | 7.0 | +0.00 |  |
| Llama2-13B | Baseline | ✓ | 7.0 | 13.0 | +0.71 | 28.1 | 28.1 | +0.44 |  |
| Vicuna-13B | RestGPT | - | 9.0 | 15.0 | +1.21 | 12.7 | 20.6 | +1.52 |  |
| Vicuna-13B | RestGPT* | - | 1.0 | 0.0 | +0.00 | 0.0 | 0.0 | - |  |
| Vicuna-13B | Baseline | ✗ | 0.0 | 0.0 | - | 0.0 | 0.0 | - |  |
|  | Vicuna-13B | Baseline | ✓ | 17.0 | 14.0 | +0.53 | 29.8 | 33.3 | +1.00 |

**Table 3: Experiment results on API-Bank. Performance of the API-Bank prompting method is reported numbers from original paper. Since the official inference code is unavailable, additional metrics (Success%, Correct Path%) could not be evaluated.**
|  | Prompting Method | Backbone | FT Method | Success% $\uparrow$ | CP% $\uparrow$ | Correctness% $\uparrow$ | ROUGE |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Closed-source | API-Bank | GPT-3 Davinci | - | - | - | 0.00 | 0.0156 |  |
| API-Bank | GPT-3.5-turbo | - | - | - | 22.00 | 0.3809 |  |  |
| API-Bank | GPT-4 | - | - | - | 70.00 | 0.4808 |  |  |
| Open-source | API-Bank | Alpaca-7B | - | - | - | 0.00 | 0.0860 |  |
| API-Bank | ChatGLM-6B | - | - | - | 0.00 | 0.1522 |  |  |
| API-Bank | Llama-7B | API-Bank | - | - | 20.00 | 0.3425 |  |  |
| Baseline | Llama-7B | - | 0.0 | 0.0 | 0.00 | 0.0048 |  |  |
| Baseline | Llama-7B | GOAT | 38.0 | 42.0 | 42.22 | 0.3173 |  |  |

**Table 4: Experiment results on GOATBench. Metrics are SA (Selection Accuracy), IA (Invocation Accuracy), and SR (Success Rate).**
|  |  |  |  | Inter Tool | Single Tool |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Prompting Method | Backbone | FT Method | SA | IA | SR | SA | IA | SR |
| Closed-source | Baseline | GPT-4.1 | - | 22.7 | 13.7 | 40.8 | 27.9 | 18.8 | 54.3 |
| Open-source | ReACT | Llama2-7B | - | 1.9 | 0.2 | 1.8 | 2.6 | 0.0 | 5.2 |
| ReACT | Llama2-7B | ToolLLM | 12.5 | 1.0 | 3.2 | 26.8 | 0.5 | 2.6 |  |
| ReACT | Llama2-7B | GOAT | 24.6 | 4.6 | 3.7 | 41.8 | 7.6 | 6.4 |  |
| Baseline | Llama3-8B | - | 10.4 | 3.3 | 4.1 | 18.6 | 6.0 | 7.1 |  |
| Baseline | Llama3-8B | GOAT | 59.0 | 26.1 | 14.4 | 68.9 | 35.6 | 24.5 |  |

**Table 5: Comparison across backbone models on GOATBench. We used Baseline as prompting method.**
| Backbone | FT | Inter Tool | Single Tool |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SA | IA | SR | SA | IA | SR |  |  |
| Qwen2-7B | ✗ | 18.9 | 6.2 | 2.5 | 29.0 | 6.1 | 6.4 |
| Qwen2-7B | ✓ | 39.8 | 15.8 | 3.0 | 46.2 | 14.1 | 7.2 |
| Llama3-8B | ✗ | 10.4 | 3.3 | 4.1 | 18.6 | 6.0 | 7.1 |
| Llama3-8B | ✓ | 59.0 | 26.1 | 14.4 | 68.9 | 35.6 | 24.5 |
| Llama3-70B | ✗ | 15.1 | 5.0 | 14.3 | 33.2 | 10.3 | 19.4 |
| Llama3-70B | ✓ | 55.7 | 18.9 | 20.2 | 62.8 | 18.3 | 39.0 |

### 4.3 Results

Results on RestBench [Table 2](https://arxiv.org/html/2510.12218v1#S4.T2) reports experimental results on RestBench. Without fine-tuning, closed-source models significantly outperform open-source models, reflecting the limited reasoning capabilities of the latter.
In fact, baseline results with open-source models are nearly zero in most cases, indicating complete failure.
This underscores the necessity of fine-tuning when building on-premise agents with open-source models.
Our GOAT training strategy yields clear and consistent improvements over these baselines.
Both Llama2-13B and Vicuna-13B achieve substantial gains after fine-tuning, demonstrating the effectiveness of our framework.
Remarkably, the fine-tuned models not only surpass the previously reported open-source state-of-the-art method, RestGPT, but in some cases even outperform the closed-source model text-davinci-003 from OpenAI.

Results on API-Bank [Table 3](https://arxiv.org/html/2510.12218v1#S4.T3) further demonstrates the effectiveness of GOAT through consistent gains on API-Bank.
Fine-tuning with GOAT improves performance over zero-shot Llama-7B and, in some cases, even surpasses closed-source models.
Compared to the original API-Bank training setup (Li et al., 2023)—which constructs synthetic API functions to train API call usage and relies on instruction-first data generation for multi-instruction queries—our method achieves substantially stronger results by aligning supervision more directly with the target task.
While our training method yields improvements across most evaluation metrics, ROUGE is slightly lower.
However, as shown in [Figure 11](https://arxiv.org/html/2510.12218v1#A5.F11), ROUGE is relatively less informative in this context, since LLMs may hallucinate lexically fluent responses that nevertheless bypass correct API execution.

Results on GOATBench As shown in [Table 4](https://arxiv.org/html/2510.12218v1#S4.T4), GOAT training achieves substantially higher performance compared to models without training.
We evaluate its effectiveness with an additional prompting method, ReACT (Yao et al., 2023), and find that GOAT consistently improves performance under both ReACT and the baseline prompting setup.
Notably, the gains hold not only over the non-fine-tuned model but also over the fine-tuned one on the instruction-first data generated by ToolLLM (Qin et al., 2023), operating with ReACT.
Although ToolLLM also generates synthetic data using the same APIs as GOATBench, its data construction process is not tailored to goal-oriented queries but instead emphasizes parallel multi-API queries with fine-grained instructions, resulting in poor performance on the goal-oriented evaluation.

Effects of GOAT with Diverse LLMs [Table 6](https://arxiv.org/html/2510.12218v1#S4.T6) further confirms that GOAT consistently boosts performance across different LLM backbones, including Qwen2-7B, Llama3-8B, and Llama3-70B. These results highlight the robustness and generality of our fine-tuning strategy.
Notably, even for Llama3-70B, where the same model was used both to generate training data and to perform fine-tuning, we still observe substantial gains.
As discussed in [Section 3.1.3](https://arxiv.org/html/2510.12218v1#S3.SS1.SSS3), this is because our call-first generation strategy provides reliable supervision: by starting from executable API paths and asking the model only to abstract them into natural-language queries, we avoid pitfalls of instruction-first methods and enable the model to benefit from supervision signals even when self-generated.

Results of Document API Retriever We conduct an ablation study to evaluate the retriever performance in isolation using recall metrics on GOATBench. [Table 6](https://arxiv.org/html/2510.12218v1#S4.T6) reports Recall@GT and Recall@5 when retrieving relevant API documents given the user query $u$. Recall measures how many of the retrieved documents are actually correct. Recall@GT evaluates this by retrieving the same number of documents as the ground-truth API calls for each data point.
The results show clear performance improvements through fine-tuning.

## 5 Conclusions

In this work, we present GOAT, a fully automatic training framework for equipping LLM agents with goal-oriented reasoning capabilities over interdependent APIs.
Unlike prior approaches that rely solely on zero-shot evaluation in the absence of training data, GOAT exploits existing API documentation to automatically construct synthetic supervision.
This allows open-source LLMs to be fine-tuned efficiently while retaining strong generalization across diverse goal-oriented scenarios.
Furthermore, leveraging GOAT pipeline, we curated GOATBench, a benchmark for evaluating goal-oriented tool use.
Extensive experiments on GOATBench and other benchmarks demonstrate that GOAT-trained agents not only achieve state-of-the-art performance among open-source models but also, in some cases, surpass closed-source systems with strong reasoning ability.
Overall, our results highlight GOAT as a practical and scalable path toward building robust open-source LLM agents that can reason over complex tool interactions and deliver accurate responses to high-level user goals.

## Ethics Statement

We acknowledge the importance of transparency and responsible use of publicly available resources in our research. All datasets, benchmarks, models, and APIs employed in this study were publicly released for academic research and were used in full compliance with their respective licenses and intended purposes.
Specifically, we used the Meta-Llama-3-8B-Instruct model under the Llama 3 Community License, the Sentence-BERT model under the Apache 2.0 License, and publicly available benchmarks such as RestGPT and StableToolBench, as well as the TMDB API under its non-commercial usage terms.
No proprietary or private data were used and this research does not involve human subjects or sensitive personal information.

## References

- Achiam et al. (2023)
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*, 2023.
- AI@Meta (2024)
AI@Meta.
Llama 3 model card.
*ArXiv*, 2024.
URL [https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md](https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md).
- Guo et al. (2024)
Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu.
Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models, 2024.
- Hu et al. (2021)
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.
Lora: Low-rank adaptation of large language models.
*ArXiv*, 2021.
URL [https://arxiv.org/abs/2106.09685](https://arxiv.org/abs/2106.09685).
- Huang et al. (2024)
Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen.
Understanding the planning of llm agents: A survey, 2024.
URL [https://arxiv.org/abs/2402.02716](https://arxiv.org/abs/2402.02716).
- Li et al. (2023)
Minghao Li, Feifan Song, Bowen Yu, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li.
Api-bank: A benchmark for tool-augmented llms, 2023.
- Liu et al. (2024)
Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang.
Agentbench: Evaluating llms as agents.
In *International Conference on Learning Representations (ICLR 2024) Poster*, 2024.
URL [https://iclr.cc/virtual/2024/poster/17388](https://iclr.cc/virtual/2024/poster/17388).
Poster.
- Patil et al. (2023)
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez.
Gorilla: Large language model connected with massive apis.
*ArXiv*, abs/2305.15334, 2023.
- Qin et al. (2023)
Yujia Qin, Shi Liang, Yining Ye, Kunlun Zhu, Lan Yan, Ya-Ting Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, M. Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun.
Toolllm: Facilitating large language models to master 16000+ real-world apis.
In *International Conference on Learning Representations*, 2023.
- Qin et al. (2024)
Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang, Tongshuang Wu, Heng Ji, Zhiyuan Liu, and Maosong Sun.
Tool learning with foundation models, 2024.
- Reimers & Gurevych (2019)
Nils Reimers and Iryna Gurevych.
Sentence-bert: Sentence embeddings using siamese bert-networks.
In *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing*. Association for Computational Linguistics, 11 2019.
URL [https://arxiv.org/abs/1908.10084](https://arxiv.org/abs/1908.10084).
- Schick et al. (2023)
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, M. Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.
Toolformer: Language models can teach themselves to use tools.
In *Neural Information Processing Systems*, 2023.
- Shen et al. (2023a)
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang.
Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face.
*Advances in Neural Information Processing Systems*, 36:38154–38180, 2023a.
- Shen et al. (2023b)
Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Y. Zhuang.
Taskbench: Benchmarking large language models for task automation.
In *arXiv.org*, 2023b.
- Shim et al. (2025)
Jeonghoon Shim, Gyuhyeon Seo, Cheongsu Lim, and Yohan Jo.
Tooldial: Multi-turn dialogue generation method for tool-augmented language models, 2025.
URL [https://arxiv.org/abs/2503.00564](https://arxiv.org/abs/2503.00564).
- Song et al. (2023)
Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang, Rong Yao, Ye Tian, and Sujian Li.
Restgpt: Connecting large language models with real-world restful apis.
In *arXiv.org*, 2023.
- Surís et al. (2023)
Dídac Surís, Sachit Menon, and Carl Vondrick.
Vipergpt: Visual inference via python execution for reasoning.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision*, pp. 11888–11898, 2023.
- Tang et al. (2023)
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun.
Toolalpaca: Generalized tool learning for language models with 3000 simulated cases.
*arXiv preprint arXiv:2306.05301*, 2023.
- Wang et al. (2024a)
Pei Wang, Yanan Wu, Noah Wang, Jiaheng Liu, Xiaoshuai Song, ZY Peng, Ken Deng, Chenchen Zhang, Junran Peng, Ge Zhang, et al.
Mtu-bench: A multi-granularity tool-use benchmark for large language models.
In *The Thirteenth International Conference on Learning Representations*, 2024a.
- Wang et al. (2023)
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.
Self-instruct: Aligning language models with self-generated instructions.
In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), *Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 13484–13508, Toronto, Canada, July 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.acl-long.754.
URL [https://aclanthology.org/2023.acl-long.754/](https://aclanthology.org/2023.acl-long.754/).
- Wang et al. (2024b)
Zezhong Wang, Xingshan Zeng, Weiwen Liu, Liangyou Li, Yasheng Wang, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong.
Toolflow: Boosting llm tool-calling through natural and coherent dialogue synthesis.
In *arXiv.org*, 2024b.
- Yang et al. (2023)
Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan.
Gpt4tools: Teaching large language model to use tools via self-instruction.
In *Advances in Neural Information Processing Systems 36 (NeurIPS 2023 Poster)*, 2023.
URL [https://neurips.cc/virtual/2023/poster/71060](https://neurips.cc/virtual/2023/poster/71060).
Poster.
- Yao et al. (2023)
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
ReAct: Synergizing reasoning and acting in language models.
In *International Conference on Learning Representations (ICLR)*, 2023.
- Yin et al. (2025)
Fan Yin, Zifeng Wang, I-Hung Hsu, Jun Yan, Ke Jiang, Yanfei Chen, Jindong Gu, Long T. Le, Kai-Wei Chang, Chen-Yu Lee, Hamid Palangi, and Tomas Pfister.
Magnet: Multi-turn tool-use data synthesis and distillation via graph translation, 2025.
URL [https://arxiv.org/abs/2503.07826](https://arxiv.org/abs/2503.07826).
- Zhao et al. (2025)
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen.
A survey of large language models, 2025.
URL [https://arxiv.org/abs/2303.18223](https://arxiv.org/abs/2303.18223).
- Zhuang et al. (2023)
Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang.
Toolqa: A dataset for llm question answering with external tools.
In *Advances in Neural Information Processing Systems 36 (NeurIPS 2023 Poster)*, 2023.
URL [https://neurips.cc/virtual/2023/poster/73466](https://neurips.cc/virtual/2023/poster/73466).
Poster.

## Appendix A API Dependency Graph Construction

### A.1 API Document Parsing Example

[Figure 4](https://arxiv.org/html/2510.12218v1#A0.F4) illustrates an example of how raw API document is converted into structured representations.
By prompting LLM as in [Figure 13](https://arxiv.org/html/2510.12218v1#A11.F13), we extract both the types and number of input parameters of each API function together with their semantic roles, as well as the semantic meaning of the returned output.

**Table 7: Precision and recall of each pruning stage.**
|  | Embedding Similarity | LLM | API Call Execution |
| --- | --- | --- | --- |
| Precision | 0.25 | 0.59 | 0.90 |
| Recall | 0.92 | 0.42 | 0.36 |

### A.2 Filtering Valid Dependency Edges

GOAT adapt three-step filtering process to obtain valid API dependency edges.
The precision and recall of each pruning stage are summarized in [Table 7](https://arxiv.org/html/2510.12218v1#A1.T7).
The first row represents the precision, while the second row corresponds to the recall, based on 500 data edges that we manually annotated.
The results demonstrate that our method successfully identifies valid edges $e=(n_{i},n_{j},k)$ while filtering out spurious connections, thereby enhancing the quality of the constructed graph.

### A.3 API Dependency Graph Example

[Figure 5](https://arxiv.org/html/2510.12218v1#A1.F5) shows example of constructed API dependency graph.

Figure: Figure 5: Example of Constructed API Dependency Graph from APIBank APIs.
Refer to caption: x5.png

## Appendix B Implementation Details

Fine-tuning of LLM model was performed on a single NVIDIA H100 GPU for 3 epochs on every experiment.
We adopted the Low-Rank Adaptation (LoRA) method Hu et al. (2021) with $r=8$, $\alpha=16$, and a dropout ratio of 0.05 to enable parameter-efficient fine-tuning on instruction-guided API execution tasks.
All experiment results reported in this paper are based on a single run without variation across random seeds.

## Appendix C GOAT-Generated Data Quality Analysis

As we discussed in  [Section 3.1.3](https://arxiv.org/html/2510.12218v1#S3.SS1.SSS3), GOAT enables benchmark curation with much less human effort than other fully human annotated benchmarks such as RestBench Song et al. (2023) and API-Bank Li et al. (2023).
[Figures 6](https://arxiv.org/html/2510.12218v1#A3.F6), [7](https://arxiv.org/html/2510.12218v1#A3.F7) and [8](https://arxiv.org/html/2510.12218v1#A3.F8) show qualitative comparsion between GOAT-generated data and fully human-generated data examples.

Figure: Figure 6: Comparison of Goat-Generated and Human-generated on TMDB.

Figure: Figure 7: Comparison of Goat-generated and Human-Generated on Spotify.

Figure: Figure 8: Comparison of Goat-generated and Human-Generated on APIBank.

Figure: Figure 9: Qualitative Example of GOATBench data.

## Appendix D GOATBench

GOATBench is a human-verified benchmark built on top of the GOAT framework.
It consists of 747 goal-oriented API execution tasks, where solving each task requires planning and invoking a sequence of interconnected APIs.
Among them, 372 tasks belong to the seen category and 375 to the unseen category, enabling evaluation across both familiar and novel API compositions.
While the seen set serves as the primary evaluation target, the unseen set provides an additional measure of generalization to previously unseen APIs.
This benchmark was constructed with the GOAT pipeline and subsequently verified and annotated by human experts.
Qualitative examples are shown in [Figure 9](https://arxiv.org/html/2510.12218v1#A3.F9).

The benchmark covers 291 tools and 829 APIs across four user-centric domains—financial, food, entertainment, and travel—with all APIs collected from RapidAPI Hub via StableToolBench Qin et al. (2023).
RapidAPI provides a hierarchical structure where each tool contains multiple APIs.
In addition, we implemented two global tools—compare and difference—that apply across all domains.
The compare tool evaluates numerical similarity or proximity between values, while the difference tool determines which value is greater, enabling comparison-based reasoning.
All LLM-based construction stages use the LLaMA-3-70B-Instruct model AI@Meta (2024), ensuring reproducibility and easy extension to other API sets.

The seen test set is created by sampling subgraphs from the same API dependency graphs as training, while modifying parameter values to ensure semantic variation under identical structures and call sequences.
This set is the main benchmark for measuring performance in in-domain settings.
The unseen test set includes tasks involving tools absent from training, serving as a secondary evaluation to test generalization.
Following Qin et al. (2023), we also categorize tasks into two types:
Single Tool, where multiple APIs under the same tool (i.e., a collection of related APIs from a single RapidAPI service) are composed to solve the task; and
Inter Tool, which requires composing APIs across different tools, testing the agent’s ability to reason over heterogeneous services and handle tool chaining.

**Table 8: Experiment results on seen test set. ID: Instruction Decomposer. GOAT FT: GOAT fine-tuned. We use Llama-3-8B-Instruct as backbone model. Metrics are SA (Selection Accuracy), IA (Invocation Accuracy), and SR (Success Rate).**
|  |  | Inter Tool | Single Tool |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Prompting Method | GOAT FT | SA | IA | SR | SA | IA | SR |
| ReACT | ✗ | 13.3 | 2.6 | 9.2 | 20.7 | 1.9 | 9.7 |
| ✓ | 51.9 | 20.3 | 13.4 | 57.6 | 30.9 | 18.0 |  |
| ReACT + ID | ✗ | 15.2 | 3.4 | 6.9 | 23.5 | 7.2 | 5.8 |
| ✓ | 37.1 | 14.9 | 8.3 | 42.6 | 30.7 | 8.4 |  |
| Global (Baseline) | ✗ | 10.4 | 3.3 | 4.1 | 18.6 | 6.0 | 7.1 |
| ✓ | 59.0 | 26.1 | 14.4 | 68.9 | 35.6 | 24.5 |  |
| Global + ID | ✗ | 19.0 | 11.9 | 9.2 | 35.6 | 26.0 | 12.9 |
| ✓ | 53.7 | 25.0 | 10.6 | 67.6 | 33.2 | 11.6 |  |

## Appendix E LLM Agent Prompting Baselines

We test our approach using following four LLM agent prompting baaselines designed to perform goal-oriented tasks. Each LLM agent design consists of four key components: API retrieval, API selection, API call generation, and final response generation. The agent designs differ in how and at what level the LLM performs planning.

ReACT (Yao et al., 2023) ReACT is a reflective agent that performs planning in an iterative manner.
Since a goal-oriented user query often requires a sequence of dependent API calls, ReACT can serve as the baseline planning method.
At each timestep, it jointly selects the next API to call and generates its arguments, conditioned on the full history of prior API calls and their outputs.
Given the user query $u$, $k$ potentially relevant documents are first retrieved using an external retriever.
The same set of the retrieved documents are then fed to the agent at each timestep.

Global Planner (Huang et al., 2024) Given $k$ relevant documents retrieved as in ReACT, this agent performs global planning by determining the entire sequence of API functions with a single prompt, using only the user query and the retrieved API documents.
This global planning strategy enables the agent to optimize the overall call sequence from a holistic perspective.
To reflect the outputs of the previous API calls,
we iteratively generate the API calls based on the planned API function sequence.
Then, API calls are executed iteratively, incorporating the previously executed API outputs.

ReACT + Instruction Decomposer This agent extends ReACT by incorporating iterative planning at the natural language level.
Starting from the user query, it generates one subinstruction at a time, conditioned on the original query, prior subinstructions, API calls, and their outputs.
For each subinstruction, it retrieves the $k$ most relevant API document and jointly performs API selection and call generation.
This process continues step-by-step, with each execution result guiding the generation of the next subinstruction.

Global Planner + Instruction Decomposer This agent also performs language-based subinstruction planning but returns global plans in a single shot.
For each subinstruction, it then retrieves $k$ relevant API documents, and jointly performs API selection and call generation in the given subinstruction order, reflecting the planned execution flow.

[Table 8](https://arxiv.org/html/2510.12218v1#A4.T8) compares the results of applying GOAT fine-tuning across different prompting methods.
We observe consistent performance improvements over all methods, demonstrating the broad effectiveness of our approach.
Among them, the decomposition-first method with a global planning strategy achieves the largest gains, and we therefore adopt it as our primary baseline for the main experiments.

Figure: Figure 10: Comparison of zero-shot inference result and GOAT fine-tuned inference result on RestBench.

Figure: Figure 11: Comparison of zero-shot inference result and GOAT fine-tuned inference result on API-Bank.

Figure: Figure 12: Comparison of zero-shot inference result and GOAT fine-tuned inference result on GOATBench.

## Appendix F Qualitative Results

Qualitative result examples on each benchmark are in [Figures 10](https://arxiv.org/html/2510.12218v1#A5.F10), [11](https://arxiv.org/html/2510.12218v1#A5.F11) and [12](https://arxiv.org/html/2510.12218v1#A5.F12).

## Appendix G Results on Unseen Test

**Table 9: Experiment results on unseen test set (Global agent). Full performance table is in [Appendix G](https://arxiv.org/html/2510.12218v1#A7).**
| Prompting Method | GOAT FT | Inter Tool | Single Tool |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| SA | IA | SR | SA | IA | SR |  |  |
| Baseline | ✗ | 11.1 | 5.2 | 11.5 | 19.5 | 9.1 | 27.1 |
| Baseline | ✓ | 42.0 | 16.7 | 19.4 | 37.2 | 26.1 | 34.6 |

Although the unseen setting is not the primary target of our benchmark, we observe that models trained on GOAT still exhibit strong generalization performance. As shown in Table [9](https://arxiv.org/html/2510.12218v1#A7.T9), fine-tuning both the LLM and retriever leads to consistent improvements across all task types. This demonstrates that task-aligned fine-tuning not only benefits in-domain execution but also improves robustness to previously unseen API combinations, though the gains are more limited compared to the seen setting.

**Table 10: Ablation results on different LLM fine-tuning methods with Global design. We use Llama-3-8B-Instruct model as backbone. All models use finetuned SBERT as the retriever.**
| LLM FT Method | Inter Tool | Single Tool |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| SA | IA | SR | SA | IA | SR |  |
| LoRA | 58.9 | 24.5 | 11.6 | 69.2 | 34.2 | 17.4 |
| LoRA + Self-Distill | 42.7 | 17.2 | 6.0 | 53.7 | 31.9 | 10.3 |
| LoRA + Masking Args | 59.0 | 26.1 | 14.4 | 68.9 | 35.6 | 24.5 |

## Appendix H Ablation Results on LLM Fine-tuning Methods

[Table 10](https://arxiv.org/html/2510.12218v1#A7.T10) compares different fine-tuning strategies designed to mitigate overfitting in the LLM. We observed that standard LoRA-based fine-tuning tends to cause the model to overfit to specific argument values seen during training, leading to reduced generalization. To address this issue, we explored two approaches: (1) self-distillation with soft targets from a pretrained model, and (2) masking argument tokens by setting their loss contributions to zero.
Our results show that the masking strategy is particularly effective, as it prevents memorization of argument values and encourages structural learning of API call formats.

## Appendix I Data Generation Prompts

### I.1 API Document Parsing Prompt

See [Figure 13](https://arxiv.org/html/2510.12218v1#A11.F13) for an example prompt.

### I.2 LLM Filtering Prompt

See [Figure 14](https://arxiv.org/html/2510.12218v1#A11.F14) for an example prompt.

### I.3 Actual Call Output Filtering Prompts

See [Figure 15](https://arxiv.org/html/2510.12218v1#A11.F15), [Figure 16](https://arxiv.org/html/2510.12218v1#A11.F16) for an example prompt.

### I.4 API Call Sequence Generation Prompt

See [Figure 17](https://arxiv.org/html/2510.12218v1#A11.F17) for an example prompt.

### I.5 Sub-instruction Generation Prompt

See [Figure 18](https://arxiv.org/html/2510.12218v1#A11.F18) for an example prompt.

### I.6 User Query Generation

See [Figure 19](https://arxiv.org/html/2510.12218v1#A11.F19) for an example prompt.

### I.7 Final Response Generation

See [Figure 20](https://arxiv.org/html/2510.12218v1#A11.F20) for an example prompt.

## Appendix J Success Rate Prompt

See [Figure 21](https://arxiv.org/html/2510.12218v1#A11.F21) for prompt used in SR evaluation on GOATBench.

## Appendix K Use of LLMs

We acknowledge that LLMs were used as writing assistants to improve grammar, clarity, and readability of the manuscript.

Figure: Figure 13: Prompt used for API document parsing.

Figure: Figure 14: Prompt used for filtering edges via LLM.

Figure: Figure 15: Prompt used for API call generation for each edges.

Figure: Figure 16: Prompt used for filtering edges via API Call Output.

Figure: Figure 17: API call sequence generation prompt.

Figure: Figure 18: Sub-instruction generation prompt.

Figure: Figure 19: User query generation prompt.

Figure: Figure 20: Final response generation prompt.

Figure: Figure 21: Prompt used for success rate metric.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: see weaknesses.
soundness: 3
strengths: 1. This paper proposes a comprehensive, multi-stage, and meticulous automatic data generation method for constructing goal-oriented API execution datasets from unlabeled API documentation. The entire process is carefully decomposed into multiple stages—embedding-based similarity filtering, LLM semantic verification, and real API call execution—significantly improving the reliability of the dependency graph while maintaining computational feasibility.
2. A novel benchmark is proposed: GOATBench is a human-validated, non-trivial benchmark set for goal-oriented tool use, covering multiple domains, and specifically designed to test the generalization ability of models on combinations of seen and unseen APIs.
summary: This paper proposes GOAT, a human annotation-free training framework for large language model (LLM) agents operating in goal-oriented tool-use scenarios, specifically for planning and executing interdependent API calls given only API documentation.  The approach consists of automatically generating synthetic datasets by parsing API docs, constructing and refining API dependency graphs, sampling realistic interdependent API task sequences, and fine-tuning LLM agents and retrievers on these synthetic tasks.  The paper introduces a new benchmark, GOATBench, and empirically demonstrates that models trained with GOAT achieve strong, often state-of-the-art, performance on multiple goal-oriented benchmarks, sometimes surpassing closed-source models.
weaknesses: 1. Additional scaling experiments could be included, and all current experiments appear to have been run only once without reporting standard deviations or confidence intervals, which limits statistical reliability.
2. Although GOATBench is described as a comprehensive goal-oriented benchmark, its construction pipeline is largely reused from the training process, raising potential risks of data leakage or overfitting to generation biases, particularly for the “seen” API compositions.
3. The introduction of GOATBench is a valuable contribution, and Appendix D provides useful implementation details. However, the paper does not sufficiently describe the data characteristics and distribution, nor does it relate GOATBench to existing benchmark efforts in this area. The authors are encouraged to enrich Appendix D with a more detailed comparison and statistical summary, referencing representative benchmark papers for context.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: Several questions and concerns are raised in "Weaknesses" part. I would be willing to change my recommendation according to the authors' response.
soundness: 2
strengths: **Originality**
This work demonstrates originality in two folds: (1) the authors proposes to generate training data of tool using sequences based on a refined API dependency Graph. The graph construction consists of three stages of filtering which ensures the correct dependencies in the graph. (2) GOAT achieves comparable performance with closed-source models in tool use by finetuning small-scale open-source models with synthetic data. 

**Quality & Clarity**
The framework is well-structured, with clear mathematical and procedural descriptions for the main components including API dependency Graph construction, Goal-Oriented API Execution Data Construction. Experiments cover three benchmarks (RestBench, API-Bank, GOATBench) and six LLMs, with both in-domain and out-of-domain evaluations are included. Examples on the synthetic dependency graph and prompt designs details are adequately illustrated in appendices.  

**Significance**
This work addresses a critical and practical problem —building cost-effective tool-use agents. The proposed method could enable adaptation of LLM agents to domain-specific APIs (e.g., financial, travel) at low cost without human annotations.
summary: This paper addresses the limitations of large language model (LLM) agents in handling goal-oriented tool-use tasks—specifically, the lack of annotated training data and poor performance of open-source models. The authors propose GOAT, a human-annotation-free training framework that automatically constructs synthetic datasets for goal-oriented API execution tasks from API documents. GOAT first builds a refined API dependency graph via a three-stage filtering pipeline (embedding similarity, LLM reasoning, and real API execution), then samples connected subgraphs to generate task samples (including user queries, interdependent API sequences, and final responses). Additionally, the authors introduce GOATBench, a new benchmark for goal-oriented tasks. Experiments show that GOAT-trained open-source models (e.g., Llama3-8B) achieve state-of-the-art performance on RestBench, API-Bank, and GOATBench, with some metrics even approaching closed-source models like text-davinci-003. The core contributions include: (1) an automatic data generation pipeline for goal-oriented tasks; (2) consistent performance gains across diverse LLMs and benchmarks; (3) the new GOATBench to facilitate future evaluation.
weaknesses: 1. As for the API Dependency Graph construction, unclear about the accuracy of the constructed graph node and edges since the authors proposed to enhance the correctness of the dependency by several strategies. Quantitative analysis on the correctness and noise of dependency graph is expected. 

2. The "call-first" strategy fills the API with plausible value at each filed as introduced in Section 3.1.3. However, enumerating all plausible values for every field of each API would constitute a huge mount of data sets. The authors used all the possible data points or they just select/utilize a small portion? There are no detailed discussions on this detail and no experiments to exam how does the selection influence the overall performance.

3. There could be obvious discrepancy between the synthetic training data and the real-world tool use cases. Connected subgraphs are collected to form the synthetic training data as introduced in Section 3.1, but in practice there could be tool calling trajectory on a subgraph with much more nodes (5+ steps v.s. 4 API nodes in this work) and even more some query could involve multiple subgraphs.   

4. Lack of comparison with recent closed-source models like GPT-4o or Claude 4 sonnet. Given that GPT-4o has strong zero-shot tool-use capabilities, omitting this comparison weakens the claim that GOAT-trained open-source models are "competitive with closed-source systems."

5. Missing technical details such as the cost of API calls, number of training data points for LLM fine-tuning and number of query-document pairs for SBERT training.

6. Section 3.1.3 seems like discussion on related works, then it would be better to move this paragraph to section 2.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: - Is the superiority of the “call-first” strategy empirically validated? The authors claim this approach avoids the self-reinforcing bias of instruction-first methods, but no ablation or comparative experiment is provided (e.g., training on instruction-first vs. call-first data using the same APIs and model).
- GOATBench’s “unseen” split only involves unseen tool compositions within the same domains (finance, food, entertainment, travel). Can the framework generalize to entirely new domains (e.g., healthcare or legal APIs) that were absent during training?
- Are the retriever (SBERT) and LLM (LoRA) jointly fine-tuned end-to-end, or trained separately? If trained in separate stages, could error propagation (e.g., incorrect API retrieval leading to failed planning) undermine overall performance?
soundness: 3
strengths: - The work targets the realistic and highly challenging setting of goal-oriented tool use, clearly identifying the key bottleneck for open-source models: the lack of training data for multi-step, interdependent API calls. This addresses a clear practical need.
- GOAT-trained agents consistently outperform zero-shot and existing fine-tuned baselines across multiple benchmarks, including RestBench, API-Bank, and the newly introduced GOATBench.
- The paper provides detailed prompt templates, hyperparameters, and data examples in the appendix, significantly supporting reproducibility.
summary: This paper proposes GOAT (Goal-Oriented Agent with Tools), a human annotation-free training framework designed to enhance the capability of open-source large language models (LLMs) on goal-oriented tool-use tasks. In such tasks, the model must autonomously decompose a high-level user goal, plan a multi-step API call chain, reason about inter-API parameter dependencies, and generate a final natural-language response.
weaknesses: - The method assumes that API documentation contains clear natural language descriptions of parameters and outputs. However, in practice, many real-world APIs—especially internal or niche services—have poor-quality, incomplete, or unstructured documentation. Under such conditions, the API dependency graph construction would likely fail. The authors do not evaluate robustness under low-quality or unstructured documentation.
- The framework supports only DAG-structured API sequences with up to 4 nodes, and cannot handle cyclic dependencies, conditional branching (e.g., if-else), or error recovery. Moreover, the inference process follows a static plan-then-execute paradigm and does not integrate iterative reasoning (e.g., ReAct-style reflection) or mechanisms for handling API call failures.
- The three-stage filtering pipeline—especially the API call execution validation step—involves numerous real API invocations and LLM inferences, which could incur significant time and monetary costs when scaling to large API sets.
- The definition of “goal-oriented” is ambiguous, and the boundary with multi-step instruction tasks is unclear. For example, generated queries like *“Can you help me find a hotel near the Flixbus stations in Paris?”* already imply a task structure (first locate stations, then find hotels), resembling implicit multi-step instructions. The distinction from prior work like API-Bank may thus reflect only a difference in phrasing granularity, rather than a fundamental innovation.

---

confidence: 5
contribution: 1
initial_rating: 2
presentation: 2
questions: Please solve the weakness provided.
soundness: 1
strengths: 1.  Sensible "Call-First" Design: The "call-first" generation strategy (Section 3.1.3) is a sound design choice. Generating a natural language query from a concrete, executable API path is a much more constrained and reliable task for an LLM than the reverse (generating correct API calls from a vague query), likely leading to higher-fidelity training data.

2.  Contribution of a New Benchmark: The introduction of GOATBench, a human-verified benchmark for goal-oriented tasks, is a useful contribution to the community, which currently lacks diverse and challenging evaluation sets for this specific problem.
summary: This paper identifies a challenge in developing tool-using LLM agents: open-source models struggle with "goal-oriented" queries that require complex planning and multiple, interdependent API calls. The authors attribute this to a lack of suitable training data.

To solve this, they propose GOAT (Goal-Oriented Agent with Tools), a training framework that automatically generates synthetic, goal-oriented training data from API documentation without human annotation. The core of GOAT is a three-stage pipeline:
1.  API Dependency Graph Construction: It parses API documents and builds an initial graph of all possible input-output connections.
2.  Graph Refinement: This graph is progressively filtered for feasibility using three steps: embedding similarity, LLM-based semantic checking, and finally, validation via actual API call execution.
3.  Data Generation: The framework samples connected subgraphs (valid API sequences) and uses a "call-first" strategy. It first executes the API sequence, then generates corresponding sub-queries, and finally generates a high-level user query and a final response based on the execution trace.

The authors demonstrate that fine-tuning open-source models on this GOAT-generated data improves performance on several goal-oriented benchmarks, including RestBench, API-Bank, and a new benchmark introduced in this paper, GOATBench.
weaknesses: 1.  Outdated and Weak Baselines in Key Comparisons: The experimental setup is undermined by the use of outdated models and weak baselines in key comparative tables.
    * In Table 2, the comparisons are against text-davinci-003 (a 2022 model) and open-source models like Llama2-13B and Vicuna-13B. These are not competitive backbones for a paper dated late 2025.
    * Similarly, Table 3's baselines (Alpaca-7B, Llama-7B) are obsolete.
    * The paper lacks a crucial comparison: training a model using data generated by another framework (like ToolFlow) and evaluating it on GOATBench, which would be a true test of GOAT's data generation superiority.
    
2.  Limited Methodological Novelty: The paper's core pipeline (build a dependency graph, filter it, sample paths, and generate data) is not new. The related work section and Table 1 already cite several highly similar, contemporaneous works (e.g., ToolFlow, Magnet, ToolDial) that also use graph-based methods to synthesize agent data. The 3-stage filtering pipeline (SBERT -> LLM check -> Execution) is a pragmatic engineering workflow, not a fundamental research contribution. The "call-first" strategy is also a known approach. The paper's primary contribution seems to be the application of this existing template to single-turn "goal-oriented" queries, which feels incremental.
  
3. The author didn't open-source GOATBench.