ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models
Authors: Jeonghoon Shim
Gyuhyeon Seo
Cheongsu Lim
Yohan Jo
Venue: ICLR 2025 Poster (2025)
Keywords: Tool-augmented language models
Primary area: datasets and benchmarks
TL;DR: A method for generating multi-turn dialogue data between the user and Tool-augmented language models.

Abstract: Tool-Augmented Language Models (TALMs) leverage external APIs to answer user queries across various domains. However, existing benchmark datasets for TALM research often feature simplistic dialogues that do not reflect real-world scenarios, such as the need for models to ask clarifying questions or proactively call additional APIs when essential information is missing. To address these limitations, we construct and release ToolDial, a dataset comprising 11,111 multi-turn dialogues, with an average of 8.95 turns per dialogue, based on APIs from RapidAPI. ToolDial has two key characteristics. First, the dialogues incorporate 16 user and system actions (e.g., request, clarify, fail inform) to capture the rich dynamics of real-world interactions. Second, we simulate dialogues where the system requests necessary information from the user based on API documentation and seeks additional APIs if the user fails to provide the required information. To facilitate this process, we introduce a method for generating an API graph that represents input and output compatibility between APIs. Using ToolDial, we evaluate a suite of language models on their ability to predict correct actions and extract input parameter values for API calls from the dialogue history. Modern language models achieve accuracy scores below 70\%, indicating substantial room for improvement. We provide a detailed analysis of the areas where these models fall short.

Published as a conference paper at ICLR 2025
TOOLDIAL: MULTI-TURN DIALOGUE GENERATION METHOD FOR TOOL-AUGMENTER LANGUAGE MODELS
Jeonghoon Shim', Gyuhyeon Seo', Cheongsu Lim", Yohan Jo
1 Graduate School of Data Science, Seoul National University 2 DDepartment of Industrial and Management Engineering, Korea University jhshim9 88 snu. ac kr
ABSTRACT
Tool-Augmented Language Models (TALMs) leverage external APIs answer user queries across various domains. However, existing benchmark datasets for TALM research often feature simplistic dialogues that do not reflect real-world scenarios, such as the need for models to ask clarifying questions or proactively call additional APIs when essential information missing. To address these lim- itations, we construct and release ToolDial, a dataset comprising 11,111 multi- turn dialogues, with an average of 8.95 turns per dialogue, based on APIs from RapidAPI. ToolDial has two key characteristics. First, the dialogues incorpo- rate 16 user and system actions (e.g.. "Request", "Clarify", "Fail inform") to capture the rich dynamics of real-world interactions. Second, we simulate dia- logues where the system requests necessary information from the user based on API documentation and seeks additional APIs if the user fails to provide the re- quired information. To facilitate this process, we introduce method for gener- ating an API graph that represents input and output compatibility between APIs. Using ToolDial, we evaluate a suite of language models on their ability to pre- dict correct actions and extract input parameter values for API calls from the di- alogue history. Modern language models achieve accuracy scores below 70%, indicating substantial room for improvement. We release our dataset and code https:: / tiggh..h.b.. cww /holi aabbToolDia
INTRODUCTION
A Tool-Augmented Language Model (TALM) is language model designed select and call appro- priate tools (usually APIs) while interacting with the user to answer the user's query. By leveraging external tools, the TALM can conduct complex tasks beyond its parametric knowledge and adapt its actions based on API results. Recent TALM benchmarks mostly feature single-turn interactions (Qin et al. 2023 Tang al., 2023) with a primary focus on improving tool selection and reasoning capabilities to address complex user queries within single turn. However, such interactions do not reflect world scenarios where the TALM should request additional information from the user or the user clarifies their intent. Even in studies that involve multi-turn interactions (Li et al.) 2023), dialogues tend to be short and limited to scenarios where the TALM asks the user for more details. The lack of richer datasets that reflect complex user-system interactions makes it difficult to accu- rately assess the ability of modern language models to handle challenging tool use scenarios in the wild, such as when the system identifies and requests information from the user based on available APIs, or when the user cannot provide requested information, requiring the model to call additional APIs to obtain the information.
To address this issue, we present a new dataset named ToolDial, which consists of multi-turn dia- logues between the user and TALM based on APIs from RapidAP[] The main focus of our dataset is to simulate dialogues where multiple APIs should be called in sequence (e.g., due to the user failing to provide information that is needed to call the main API) and where the user and the TALM can take diverse actions (16 total), such as clarifying the user's intent handling the user's failure
Corresponding author.
https://rapidapp com/hut /hub

---

Published as a conference paper at ICLR 2025
Graph Construction
Action Sequence Construction
Scenario Generation
<Dialogue Dats
Nive
Ober far
Ivart voutie Piforaiin
Cherr
Une de sysles witt..
Anggerr
Actir out
Systom Fmerd
API Chain usage Information
hem AAills part
Aal Anltooo
Gaodlyy
-deleo tystes Dookre atp goor
Figure 1: Overall structure of ToolDial. This represents the whole pipeline of our method.
to provide requested information. To that end, our data generation pipeline consists of four steps, as shown in Figure 11 First, to facilitate selecting two APIs that should be called in sequence, construct an API graph where nodes are APIs and edges between two APIs indicate that one API's output can be used input for the other API ($3.1). Second, to simulate rich dynamics between the user and TALM, we define 16 types of user and system actions informed by the literature of task-oriented dialogue systems and compile 23 plausible sequences of actions that are likely to oc- cur in dialogues (e.g.. Inform intent clear - Retriever call + Request- -> Fail inform) (:3.2) Third, to generate each dialogue, we select pair of APIs from the API graph and choose a sequence of actions that serves as skeleton. Based on this, we enrich the skeleton by incorporating additional dialogue state information for each turn, such as the input parameters of the APIs informed by the user ($.3) Fourth, we convert the augmented action sequence into natural utterances to complete a dialogue ($3.4). As a result, ToolDial contains 1 11,111 dialogues with an average 8.95 turns per dialogue.
Based on TooIDial, we designed three evaluation tasks to assess a suite of language models in their ability to use tool. Specifically, we evaluated their ability (I) to predict appropriate actions to progress toward answering the user query, (2) to choose the correct API and predict dialogue states (i.c., extracting user-informed values for API inputs), and (3) to generate responses faithful to API outputs. We found that GPT-based models struggle with dialogue state prediction, and their performance declines the dialogue length increases. Additionally, these models perform poorly at predicting next actions, particularly struggling with requesting input parameters and asking clar- ifying questions. For smaller Llama models, they generally underperform compared to GPT-based models, but fine-tuning on our dataset significantly improved the performance of each task. Notably, it led to substantial improvements in many actions that GPT models struggled with. Our experiments suggest that ToolDial can be a valuable resource for both assessing and improving TALMs in com- plex multi-turn interactions with users.
The main contributions of our work are summarized as follows:
We generate and release TooIDial, a dataset consisting of dialogues that reflect real-world interactions between the user and TALM, encompassing 16 user and system actions.
We present a framework for creating a large-scale and multi-turn turli-tturnun dialogue benchmark using API graph and GPT-4o with minimal human effort.
We provide insights into the abilities of various language models to answer user queries while interacting with the user across multiple turns and using external APIs.
2 RELATED WORKS
Tool Augmented Language Models cobpee]]oomprrss our dataset with existing benchmarks. Re- cent research on TALM has evolved toward investigating how to effectively select tools and deter- mine which reasoning steps are beneficial for solving complex problems (Yao et al.) 2023 Schick et al. 2023 Shen et al. 2023: Qin et al. 2023: Patil et al. 2023; Tang et al. 2023). Similar to our work, ToolNet (Liu et al. (024)) leverages an API graph, but this graph connects APIs that are called back-to-bart back in dialogues without considering the compatibility of the input and output of APIs. Most existing datasets contain single-tum dialogues between the user and TALM. For instance,
2

---

Published as a conference paper at ICLR 2025
Table Comparison between ToolDial and other TALM datasets. We derived the number of actions based on how many action types occur in each dataset with our action taxonomy as a reference.
Resource
ToolDial
ToolBench
API-Bank
ToolAIpaca
Real-world API?
Multi-turn Scenario?
Multi-tool Scenario?
sult Reasoning?
X
Situation Complexity?
X
X
X
Number of Actions
16
7
3
Number of Dialogues
11,111
188,304
6,860
4,889
Avg. Turn per Dialogue
8.95
2.84
TaskBench (Shen et al. 2024) attempted to construct graphs by matching API inputs and outputs and generating user queries that can be solved using API chains. However, they did not propose method for graph construction, and focused solely on inferring the sequence of APIs required to solve a user query in a single turn rather than through a multi-turn dialogue. Although API-Bank (Li et al. 2023) contains multi-turn interactions, the number of turns in each dialogue is limited (2.84 on average), and the interactions are relatively simplistic. ToolTalk (Farn Shin (023) also reflects some degree of multi- multi-turrn interactions (6.44 average), but it relies dialogue generation using human annotators, resulting in only a small amount of data (a total of 78 dialogues).
Task-Oriented Dialogue System A task-oriented dialogue (TOD) system is a goal-oriented di- alogue system that processes user queries, understands the intent, and provides answers based on database searches or tool calls. Representative datasets for TOD include MultWWOZ (Budzianowski al. 2020) and Schema- Gaineed Dialogue (SGD) (Rastogi al. 2020). MultiWOZ is multi-turn dialogue dataset generated by human annotators, which reflects the interactions between users and the system. Additionally, the annotations of dialogue states allow for the evaluation of system's 's ability to track dialogue states. Similarly, the SGD dataset features multi-turn interactions. Notably, the way SGD was generated shares similarities with our data generation method, particularly that an action sequence is chosen first for each dialogue, and then utterances are generated. However, unlike our work, the dialogues in SGD do not reflect difficult situations that a real-world tool agent may face, as SGD utilizes limited number of APIs and there to provide the necessary information for an API call. The literature on TOD offers useful concepts such dialogue state tracking (Jacqmin al. 2022) and rich taxonomies of user and system tions that occur in interactions with real-world agents. There have also been attempts to transfer TOD datasets into TALM-style data (Moghe et al. (0024). We designed the ToolDial dataset by ref- erencing representative benchmarks in TOD (e.g., the format of dialogue states in MultiWOZ and action types in SGD).
scenarios where the user fails
3 TOOLDIAL
The dialogues in ToolDial are generated to reflect complex interactions between the user and system in realistic situations involving chained API usage (i.c., the output of one API is used the input for another API). To achieve this, we follow four steps, shown in Figure I First, we construct an API graph by connecting the input and output entities of APIs ($3.T) This graph plays critical role in selecting interdependent APIs to be used for each dialogue. Second, we define 16 types of user and system actions to capture the complex dynamics interactions with tool agents. Based on these actions, we create 23 plausible action sequences that are likely to occur in dialogues ($3.2). Third, to generate a dialogue, we choose a pair of APIs from the API graph, select an action sequence, and augment it with concrete dialogue states that track the collection of input parameters for the APIs Lastly, we generate utterances that reflect the augmented action sequence using GPT-4o These processes carried out with minimal human effort.
(,3)
3.4
3

---

Published as a conference paper at ICLR 2025
Reasoning Step
Dialogue
Reasoning Step
Actior: Clur
Resporse
(hrught Berrawe broe. rlte veffeee
information for specific bealutbel
Morr home abordhigg
Chrogus dedee 477 bouraament
(nters Jecter Fahhwer
aunmal sansus?
Texs alloe issios poines for poists agalnnt rnooor
Frstes far thet Lopp vatinn))
(hhught westl Leggulioccaadiigg io
Freer est temonll bo
Adtone Requaet
Actire Inforn
Tromk ure, fowen 2 Thars fo win, cooser
Action: Requeet
Dalogue State (ssssonad touranentt
bournacivets for the lague
Sarel WiM.
Action Resporse
Requett
(loage proclett cou ott
(kcow the snarocti is di bur
Please chopee one of louguse
Ourega Susic pudeppeed (INE
lesido as colpooyy der
Adlon Requaet
t
Adton infors
-Oll lecal (Ct taru NDK) 1.2. name HCAA
Action: /hqquest
Action:
Dillgg State: (ntegeold
beakerbul curtgory to rabrivve
beskebbull cotggery da 204
An example dialogue from ToolDial. This illustrates the user and TALM actions for each turn, along with corresponding utterances. It also shows the reasoning steps TALM undergoes, including API calls and retriever calls, before asking or responding to the user.
Figure 2:
3.1 GRAPH CONSTRUCTION
Motivation To simulate dialogues where APIs should be called in sequence to fulfill the user's needs (e.g., the user fails provide a necessary argument for API, and thus the system should proactively find and run another API that can provide it), it is necessary to identify which API's output can be used as the input for another API (i.e., API chaining). To facilitate this, we construct an API graph where APIs from RapidAPI are represented as nodes, and two APIS are connected by an edge if one API's output can be used as the input for the other API. Eventually, this API graph will be used in dialogue generation, allowing us to easily select compatible APIs to be called in sequence.
Settings To determine whether to build an edge between two APIs, we used the names and de- scriptions of their input and output entities from the API documentation on RapidAPI. However, these input and output entities often had generic names (e.g., 'id'), and their descriptions did not sufficiently explain their meanings. To address this, we augmented the descriptions using GPT-4o- mini, incorporating the API documentation and instructions (A.I). To replace generic names with more descriptive and informative identifiers, we summarized the augmented description into a 5- to 7-word phrase. Additionally, we extracted up to 4 keywords from each API's description repre-
sent its functionality, ensuring that APIs from vastly different domains were not connected during edge construction ).2).
Edge Construction Using the keywords of APIs, along with the names and descriptions of their input and output entities, we established three criteria for constructing edges Edge based on their similarities. This process is formalized in Equation I
1, if emb(d,, d:) > la A emb(do + ko, di + ki) > 4 A LCS( ni) > 4
Edge
(1)
0, otherwise
where i and o represent the input and output entities, respectively. k, n, and d + k denote the description, keywords, name, and the concatenation of keywords and description, respectively. emb the embedding of a description obtained from the S-BERT model all-mpnet- -bse--22 (Reimers & Guuevycc][219)) LCS stands for the longest common subsequence (Hirschberg] 1977). i represents the threshold for each criterion. With the embedding similarity between d; and d, and the longest common subsequence similarity between n, and n., we aimed to match input and output entities that exactly correspond to each other. Furthermore, by considering the embedding similarity between d, + k; and d, + ko, ensured that entities from vastly different domains were not incorrectly matched. As a result, we constructed 4,857 edges from 500 million edge candidates (4.474 x 4,474 API pairs, with each pair averaging 25 edge candidates).
4

---

Published as a conference paper at ICLR 2025
Edge Evaluation To verify the edges in the constructed graph, we designed automated evalua- tion metric to classify whether each edge was valid (see the examples of mismatched edges in A.I.I. Directly calling the API would be the most reliable method for validating edges, but it requires substantial amount of time and cost and suffers from non-executable APIs in RapidAPI. To address this, we utilized StableToolBBench (Guo al. 2024). an API simulator based large language models. StableToolBench can generate API outputs similar to real API calls, allowing us to validate edges in similar way to actual API calls. However, MtableToolBench also has some issues; for ex- ample, the outputs of the same API have different formats upon multiple calls. We fixed such issues by augmenting StableToolBende with additional information from API documentation. We sam- pled 200 edges from our API graph and measured the Matthews Correlation Coefficient (Matthews) 1975) against human evaluations, which resulted in a score of 0.868. This score indicates a strong correlation between the evaluation metric and human judgment. For the 4,857 constructed edges, the precision (the proportion of valid edges among constructed edges) was 70.9%. Next, to estimate the number of missing edges, we measured Negative Predictive Value (the proportion of invalid edges among non-constructed edges). Since the graph contained too many unconstructed edges (i.e., no connection between APIs), sampled 5,501 pairs of input and output entities that nected. The NPV score was 95.0%, indicating that among the candidates that could form edges, the proportion missing was small. These results indicate that our constructed graph covers most valid edges at the expense of 30% invalid edges. For dialogue generation, we discarded the invalid edges in the subsequent steps.
not con-
3.2 ACTION SEQUENCES
Motivation In dialogue systems, an action refers to a dialogue act representing a specific behavior taken by the user or system during a conversation (e.g.. "request information". "deny suggestion", etc.). A taxonomy of user and system actions allows a dialogue system to manage dialogue flow effectively, by focusing on high-level behaviors before generating utterances and providing inter- pretability. We compile taxonomy that covers a wide range of actions occurring in user-system interactions so that the generated dialogues and trained systems reflect the complexity of the real world. To generate dialogue in the next step, we will first choose a plausible sequence of actions (i.e., dialogue flow) as a skeleton before generating utterances (a similar approach was adopted in SGD (Rastogi al. 2020
Definition of Actions We define a total of 16 actions that the user and system can take. User actions include three types of intent expressions: "Inform intent clear" (an unambiguous query that can specify the correct API), "Inform intent clear add" (an unambiguous query along with one additional input entity of the corresponding API), and "Inform intent vague" (an ambiguous query). Additionally, "Inform" and "Fail inform" refer to the success and failure, respectively, of providing an API's input entities requested by the system. With "Affirm" and "Negate", the user can accept or reject the system's 's suggestions.
System actions include "Request", which asks the user for information, and "Response", which provides an answer to the user's query. When the user's ' query is ambiguous, the system may take actions such as "Clarify" or "Suggest" to refine the query. We also define internal system actions such as "Retriever call"' and "Call", which occur during the TALM's reasoning steps. The "Retriever call" action retrieves the appropriate API, while "Call" executes the selected API once all input parameters have been obtained from the dialogue history (see the description of actions in A.3)
Action Sequences Based on the predefined actions, we define plausible action sequences (Fig- ure 3 ToolDial is created by combining API pairs from the API graph with action sequences. The types of combinable action sequences depend on whether the APIs in the pair require input parameters and the form of their outputs (c.g.. single value vs. a list of values).
For example, in Figure 2 the parameter "tournamentld" for the "LeagucHomeStandinng" API. Both APIs require input parame- ters, and "Category Tournaments" returns a list of "id's. In this case, the high-level action sequence is as follows:
API outputs "id",, which can serve as the input
Inform intent clear- + Retriever call - Request- + Fail inform - Retriever call + Request Inform Call- Request Inform Call Response.
5

---

Published as a conference paper at ICLR 2025
Inform Intent Clear
Suggest
Negate
Response Fail
Inform Intent Vague
Rabrewer score 05-05
Affirm
lspul pacesster
Inform Intent Clear Add
Rotriever call
Call
Response
Avellvar Jover than 0.
Fall
lipal perymeter
User Bye
Adllied
Clarily
Request
Inform
System
Figure 3: Action graph based on predefined user and system actions. This represents the whole multi turn interaction between user and TALM in our dataset.
There are three "Request" actions in this action sequence. The first one retrieves the input parameters needed to execute "LeaaueHoomSStaddnng"" the second executes "CategoryTounme Tournaments", and the third selects one "id" from the multiple IDs outputted by in Figure 2). If an API required input parameters or returned single value instead of list, there would be at most two "Request" actions, modifying the overall structure of the action sequence.
(see the 6th turn
We also construct different action sequences depending on whether the intent-informing action is "Inform intent clear" or "Inform intent vague". In the latter case, we further distinguish whether transitions into "Clarify" or "Suggest" action. Additionally, we design different action sequences based on the user's "Fail inform" action within the same API pair (see details in A.5). The complete set of rules governing action sequences visualized in Figure (see all types of action sequences A.
3.3 SCENARIO INSTRUCTION GENERATION
ToolDial is collection of task-oriented dialogues where the system utilizes appropriate APIs to achieve the user's goal. When necessary, the system retrieves suitable APIs through an API retriever and collects the required input parameters for API calls through multi-turn interactions. Generating such a dialogue involves simulating user query, defining dialogue states that specify the required input parameters for APIs provided by the user, and creating utterance instructions that guide utter- ance generation in the subsequent step.
User Query For each dialogue, randomly sampled either single API pair of connected APIs from the API graph. We also randomly sampled an action sequence to used in the dialogue. The next key step prompted GPT-4o with the names and documentation of the API(s) and instructed it to generate a user query that covers all the API(s). For example, given two APIs "search weather station (input: coordinates, output: weather station)" and "nearby weather station coordinate API (input: location name, output: coordinates)". GPT-4o generated the query "I'm going hiking week and would like to find a nearby weather station".. This query became the first user utterance, initiating the dialogue.
to generate user query relevant to the API(s). To accomplish this,
Dialogue State The dialogue state at any point dialogue specifies the API name the system aims to call, its input parameters, and the parameter values provided by the user. To generate dialogue given a user query and API(s), GPT-4o simulated concrete and plausible parameter values (e.g., "45", "264", and "NBA" in Figure 2 Dialogue states serve basis for generating utterances and as the ground-truth labels for dialogue state tracking (DST) evaluation ($4). The format of the dialogue state is specified in A.4
Scenario Instruction Based the dialogue states, construct instructions to guide GPT-4o in generating user and system utterances. These instructions are based on templates.
For instance, the instruction for the dialogue in Figure is follows:

---

Published as a conference paper at ICLR 2025
Table Overall statistics of ToolDial.
Table 3: Dialogue quality scores.
Metric
Value
Criterion
G-Eval
Humans
Train
8,859
Naturalness (1-3)
2.28
2.54
Validation
1,086
Coherence (1-3)
2.58
2.81
Test
1,166
Efficiency (1-3)
2.81
2.60
Total
11,111
Faithfulness (0-1)
0.90
0.95
# of turns
99,476
of turns per dialogue
8.95
Inform intent clear: the user utters pre-constructed query related with API LeagueHome- Standings and Gategory/Tournnne
(Retriever call) + Request: the system to ask the user for seasonld and tournamentld.
Fail inform: the user responds with seasonld 45 but fails to provide tournamentld.
(Retriever call) - Request: the system prompts the user for id.
Inform: the user responds with the requested information.
(Call) - Request: system asks the user for the name variable, to select one from multiple results.
Inform: the user responds with NBA.
(Call) + Response: the system responds based on the results of the call.
By prompting GPT-4o with these scenario instructions, we create multi-turn dialogue in which the user and system exchange utterances that align with the dialogue states to fulfill the user query (see details in A.8
3.4 DIALOGUE GENERATION
prompt GPT-4o with simple instructions, the scenario instruction (3.3), and the relationship between the two APIs in the API pair. Based on this guideline, GPT-4o generates each utterance of the user and the system that aligns with each turn's 's dialogue state (refer to the examples in Figure 2
Utterance Generation
Data Statistics Our dataset ToolDial contains 11 11,111 dialogues in English reflecting various sce- narios that can happen in the real world. The statistics of ToolDial shown in Table 2 ToolDial is constructed based on 23 types of action sequences and has an average of 8.95 turns per dialogue.
To assess the quality of our dataset, we sampled total 100 dialogues from action sequences and evaluated them using both G-Eval (Liu et al. 2023 and human annotator The evaluation criteria are as follows:
Data Quality
Naturalness (1-3): Are the dialogues natural interactions between the user and TALM?
Coherence (1-3): Are the user' and the TALM's utterances relevant to and coherent with the dialogue context?
Efficiency (1-3): Are the system's reasoning and actions to perform the user's request efficient and natural?
Faithfulness (True or False): Are the system's responses consistent with the output of the API call?
Table 3 presents the scores from G-Eval and human annotators. On average, G-Eval assigned high scores when evaluating the 100 sample dialogues across four criteria. The dialogues received partic- ularly high scores in Efficiency, indicating that the TALM efficiently performed the necessary steps to call APIs and collect information.
"Three Master's students majoring data science volunteered annotators. The authors not included.

---

Published as a conference paper at ICLR 2025
Table 4: Evaluation scores on three tasks, (w GT: ground-truth labels are included in the dialogue history, w/o GT: no ground-truth labels are provided)
Dialogue State Tracking
Action Prediction
Faithfulness
Model
GT
w/o GT
w GT
w/o GT
w/o GT
GPT-3.5-turbo
38.8
33.1
53.5
54.1
95.4
GPT-4o-mini
67.7
63.7
GPT-4-turbo
77.5
68.6
64.2
61.5
97.1
GPT-4o
81.4
67.8
57.6
63.7
96.7
BoeeLlama-7--Issttueee
47.2
28.9
35.7
30.0
81.7
Qwenn.55-Coder-7BBmme
48.9
34.2
55.8
46.8
93.9
Alama3--B-IIstruue
53.4
24.5
37.7
35.5
91.5
TD-Llama
92.7
72.2
77.5
91.0
88.4
Model Biases In ToolDial, we have leveraged several methods to mitigate GPT-4o's biases in dia- logue generation. When GPT-4o generates dialogues without any guidance, the resulting dialogues tend to be overly repetitive and monotonous. Specifically, certain types of APIs are dispropor- tionately preferred, and the actions performed by both the user and system lack variety, typically following simple "Inform intent Response" pattern. In ToolDial, we addressed this by creating dialogue data using 473 real-world APIs spanning 23 domains from RapidAPI ($3.I) and incorpo- rating 16 actions and 23 action sequences cover diverse scenarios ($3.2). Furthermore, for certain actions, GP'T-generated utterances tend to have overly consistent speaking styles. As a solution, predefined speaking styles for specific actions (A.7) and incorporated mechanism to randomly select from these predefined speaking styles during the scenario instruction generation ($3.3).
EXPERIMENTS
In these experiments, we designed evaluation tasks assess the capabilities that the TALM should possess when engaging in multi-turn interactions with users. The input to the model includes:
H,, (M,, S1. .1...... Un 8,), Rr, (F.,,
rn), Tn An, RS,, D,, DS,)
(2)
where Hi is the dialogue history up to the n-th tur,, and and s; are the utterances of the user and TALM the i-th turn. Rn represents the reasoning steps the TALM up to the n-th turn, where is the reasoning step in turn i. Fach reasoning step includes the thought 1, action A, retriever status RS, retrieved API documentation D from the retriever, and dialogue state DS the corresponding turn (see the formation of dialogue state and retriever status in A.4). The reasoning step of Figure 2 illustrates each component. We used Hi and R,, to predict DS and AA in each turn evaluate whether the model accurately captures the dialogue context, extracts the appropriate information, and takes the correct action. Additionally, we evaluated the last utterance 8n where An ="Response" in order assess the consistency between the model's response and the output of the API call.
4.1 EVALUATION TASKS
Dialogue State Tracking Dialogue state tracking (DST) evaluates the model's 's ability determine which API should be called based on the dialogue history, as well as the accuracy of the collected input parameter values. DST can be formalized
DS, 1,HHn_,, Rn-1, Un)
where DS, is the dialogue state of turn .M is the TALM's output, H.-1 and R. are the dialogue history and the TALM's reasoning steps up to tur n - 1. We evaluate a total of 6,746 annotated dialogue states within the test set. The evaluation checks whether the two dialogue states match completely after removing all special characters, converting to lowercase, and comparing API names, input parameters, and their corresponding values.
8

---

Published as a conference paper at ICLR 2025
Table 5: Fl score for each action in the action prediction task. This indicates that fine-tume tinettuning with our data supports the system in selecting appropriate actions in multi-turn conversations.
Response
Response fail
Request
Retriever call
Clarify
System
Suggest
Call
GPT-3,5-tubo
63.8
0.0
28.4
66.2
1.3
95.5
0.0
53.4
GPT-4o-mini
78.9
0.0
44.3
67.4
64.5
97.2
0.0
67.0
GPT-4-turbo
93.6
0.0
18.1
87.5
97.2
29,9
56.4
w GT
GPT-40
88.3
0.0
13.7
74.9
29.6
97.2
24.6
54.1
Llama3-8h 8b-Inst
46.4
0.0
8.5
23.7
0.0
99.8
14.0
44.4
TD-Llama
100.0
77.5
44.8
77.4
16.8
68.6
GPT-3.5-turbo
70.7
0.0
1.3
93.0
0.0
49.7
GPT-4o-mini
88.5
36.1
62.6
97.2
0.0
65.1
GPT-4-turbo
96.6
0.0
10.8
79.9
40.6
97.2
35.5
57.8
w/o
GPT-4o 95.8
0.0
14.3
81.2
38.6
97.2
46.1
62.0
Llama33-b-Inst
30.5
1.9
27.3
9.4
42.0
TD-Llama
98.2
99.1
78.4
94.5
99.8
100.0
99.9
86.9
The action prediction task involves selecting the next action to be taken based on the dialogue history and reasoning steps. For this task, the reasoning steps do not include ground- truth thought t, as it offers direct cue for which action to take. Action prediction is formalized as
Action Prediction
An = MMH (Rn-1 (n-1), un))
(4)
where An is the system action in πt. We evaluate total of 9,200 annotated actions within the test set. Each turn's true action and predicted action are converted to lowercase, and special characters are removed, Evaluation based on whether they match exactly.
Faithfulness We evaluate whether the final response of the TALM is grounded in the API call output, as generating responses faithful to API call results critical for tool agents. provide the TALM with dialogue history that includes the API call results and use G-Eval (Liu et al.) 2023) to assess whether the responses reflect the API call output. The evaluation method aligns with the faithfulness criterion outlined in the Dialogue Generation step ($3.4). We evaluate total of 943 system responses (removing "Response fail") within the test set. Following the same method as G-Eval, GPTT-o-mini model with temperature set above 0 evaluates each response for 10 times. The average score of the 10 results (all either 0 or 1) is used as the score.
4.2 EXPERIMENT SETTINGS
In the real world, the model is not provided with ground-truth actions or dialogue states the di- alogue history. Hence, evaluate models in two settings: "with GT (ground truth)" and "without GT". The latter is to see the upper bound performance of the models assuming that all prior pre- dictions correct. "With GT" uses the formulations in Equations and and "without GT" is formalized as
DS M(Hn-1, (Ra-1 u,),
Awost M(Hn-1. (R ( U(n-))) un) (5)
For the faithfulness task, we only conduct the experiment in the "without GT setting, the model generates the final turn response and no ground -oound-truth label exists in Hn-1 or R, All instruction prompts used in each task in A.13
As baseline models, we choose GPT-3.5-turbo, GPT-4o-mini. GPT-4-turbo, GPT-4o, CodeLlama- 7b-Instruct-hff 1wwen2.5-Coddrr7BBBnnnitt LLaMA3--B-iistruuct with the ToolDial dataset (TD-Llama) and conducted the same experiments. All experiments are conducted in a zero-shot setting, where only task-specific instructions are pro- vided without any additional few-shot samples.
and I.LaM33-B-iisttrue We also instruction-tui
4.3 RESULTS
The experiment results are summarized in Table
Dialogue State Tracking For the GPT-based models (rows 1-4), observed that the latest ver- sions outperform their predecessors,. Additionally, both closed-source and open-source LLMs scored lower in the "w/o GT setting compared to the "with GT" setting, expected. Instruct-tuning the

---

Published as a conference paper at ICLR 2025
Llama model (TD-Llama) our dataset (row 7) significantly enhances its performance in both set- tings, demonstrating the value of our dataset for training TALMs. Furthermore, we observed that accuracy decreases the number of turns increases (A.10). For TD-Llama, performance remains stable in the "with GT" setting even with longer turns. However, in the "w/o GT" setting, which better reflects real-world scenarios, performance declines the number of turns increases. This suggests that dialogue state tracking over multiple turns in real- settings remains a challenging task. A detailed error analysis of DST is provided in A.9
Action Prediction In the action prediction task, GPT models (rows 1-4) achieved an accuracy of around 60%, which suggests that there is significant room for improvement. On the other hand, Llama3-8B-Instruc received a much lower accuracy of around 35% indicating the difficulty in determining appropriate actions based on dialogue history. However, once fine-tuned on our dataset, TD-LIama (row 7) achieved an accuracy of 77.5% and 91.0% on with GT and w/o GT respectively, outperforming GPT models.
To better understand the models' performance across actions, Table shows the Fi-score for each action. Here, GPT models show relatively low scores for predicting actions like "Request", "Clar- ify". and "Suggest". This result is consistent with our observation that GPT-based models often rush to provide answers without collecting further information asking clarifying questions. These actions are essential in real-world interactions to serve the user's needs precisely and reduce hal- lucinations, and TD-Llama demonstrates improved performance on these actions. Another notable result is the low performance of GPT models on the "Response fail" action. When the user refuses to proceed with suggested API, the models often attempt to clarify the user's intent ("Clarify"") rather than acknowledging the failure and terminating the dialogue. While this move could be considered somewhat reasonable, violates the instruction provided in the prompt and may bother the user.
Faithfulness GPT models achieved over 90% accuracy in the faithfulness task. However, the performance of the smaller Llama-based models remains around 88.4%. This demonstrates that small language models are vulnerable to hallucination, and we need better methods for improving the faithfulness of these models.
Overall Performance To accurately resolve user's query in real-world settings, generating cor- rect reasoning trace (dialogue state, action) based on dialogue history and the user's most recent utterance is crucial at each turn. We evaluated the overall performance of the fine-tuned TD-Llama model in this context. We assessed whether the model correctly generated both the dialogue state and action after processing 5,213 user utterances in the test set. A result was marked true if both the action and dialogue state were accurately generated for each reasoning step; otherwise, it was marked as false. This evaluation yielded a performance score of 77.1%. Additionally, for 1,166 test dialogues, we measured the proportion of dialogues in which the reasoming trace was correctly gen- erated for all turns funss-rrmm the first to the last achieving an accuracy rate of approximately 28.3%. This suggests that there is significant room for improvement in overall performance.
CONCLUSION
In this work, we introduce ToolDial, a multi-turn dialogue dataset that reflects interactions between a user and the TALM in real-world scenarios. To generate realistic dialogues, we construct and em- ploy an API graph representing the interdependencies between APIs, aiming to simulate scenarios in which the TALM must call multiple APIs to obtain necessary information. Additionally, we de- fine 16 user and system actions to reflect the rich dynamics of tool-use conversations. To generate a dialogue, we first sample APIs and an action sequence as a skeleton. This skeleton is then aug- mented with dialogue states specific to the APIs and finally converted into utterances using GPT-4o. Our evaluation demonstrates that modern language models perform poorly in predicting appropriate actions and dialogue states in complex multi-turn interactions. We believe ToolDial can serve as a valuable resource for advancing the field of TALM.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported by the New Faculty Startup Fund and the Creative-Pionerrin Researchers Program through Scoul National University. was also supported by the National Research Founda- tion of Korea (NRF) grants RKS-2024-00333488
DS-2024-00414988 and the Institute of Informa- tion communications Technology Planning & Evaluation (IITP) under the Leading Generative Al grant, both funded by the Korea
Human Resources Development (ITTP-20255
government (MSIT, Ministry of Science and ICT).
REFERENCES
Pawel Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, lñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašié. Multiwoz a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling, 2020. URL
wttpaaiwwww..i.aaaaaeooooeoooere abs wttp:o//ggga..ooo..////. /bt.///a//111007 00278
Nicholas Farn and Richard Shin. Tooltalk: Evaluating tool-usage in conversational setting, 2023.
wtt..//r///////.11.1111.. 10775
Zhicheng Guo, Sijie Cheng, Hao Wang. Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. Stabletoolbench: Towards stable large-scale benchmarking on tool learning of large language models, 2024. URLhttppa www.aaiivvvv.oo org
Daniel Hirschberg. Algorithms for the longest common subsequence problem. J. ACM, 24
(4):664-675, October 1977. ISSN 0004-5411.
doi: 10.1145/322033.322044 URL https:
//doi.org/10.11110 1ttp://1..11/13220000 322044
Léo Jacqmin, Lina M. Rojas Barahona, and Benoit Favre. "do you follow me?": A survey of recent approaches in dialogue state tracking. In Oliver Lemon, Dilek Hakkani-Tur, Junyi Jessy Li, Arash Ashrafzadeh, Daniel Hernández Garcia, Malihe Alikhani, David Vandyke, and Ondřej Dušek (eds.), Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 336-350 Edinburgh, UK, September 2022. Association for Computational Lin- guistics. doi: 10. 10.18663/v1/2022.sgggll 1.33.. URL htt.https///// //toa//a/tttts//aaaaaaatoolooo.. Bigdial-1.33
Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. Api-bank: A comprehensive benchmark for augmented llms, 2023.
URL http://axvvvv.//////eee/eaaaa/la org/ abs/ 2abb/23000 2 08244
Xukun Liu, Zhiyuan Peng, Xiaoyuan Yi, Xing Xie, Lirong Xiang, Yuchen Liu, and Dongkuan Xu. Toolnet: Connecting large language models with massive tools via tool graph, 2024. URL
https: ptttps://ariiv. www. wwtp://gg///aaoooooooooooosses/s /adbs//433. 00839
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment, 2023. URL URL.ittpss///cxiv.orggg
2303. 66634
Brian W. Matthews. Comparison of the predicted and observed secondary structure of t4 phage lysozyme. Biochimica et biophysica acta, 405 2:442-51 51, 1975. URL htttps://apl.
semanticscholue ort.choola..ooro/oorroaaaaaa::aa
Nikita Moghe, Patrick Xia, Jacob Andreas, Jason Eisner, Benjamin Van Durme, and Harsh Jhamtani. Interpreting user requests in the context of natural language standing instructions, 2024. URL
https::/ org/ wwwp/////////////eoo/oe/eeeeeeee http://..//331111113
Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive apis, 2023. URL htRL:ttt/:// http:::///axvvvvorooo. 2305la///23330.. 2
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li. Zhiyuan Liu, and Maosong Sun. Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023. URLhtttss: ///xx/vvvvv//////////a//////a//a org/ abs////s//300 16789
11

---

Published as a conference paper at ICLR 2025
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. Towards scalable multi-domain conversational agents: The schema-guide dialogue dataset, 2020. URL
lhttss://arxii wttp.//////.....0000000
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks, 2019.
htwgg/bbs/999g 10084
Timo Schick, Jane Dwivedi- Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools, 2023.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hug- ginggpt: Solving ai tasks with chatgpt and its friends in huggingface. In Advances in Neural Information Processing Systems, 2023.
Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li. and Yueting Zhuang. Taskbench: Benchmarking large language models for task automation, 2024. /ttts:/:: //tp/vaarvvvvv//aa////a owg/abs.a......o 2311. 18760
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. Toolal- paca: Generalized tool learning for language models with 3000 simulated cases, 2023, URL
https://arxiv..org abs/ 2306. 05301
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models, 2023. URL https: :ttppss:///xvv..///
org/abs bog/abs/2210... 03629
12

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ENTITY DESCRIPTION GENERATION
Table 6: Prompt used to generate input entity
Prompt for generating input entity descriptions
System
You are an intelligent annotator. Your mission is to write the description of input parameters more specifically, referring to the given information.
Write as specifically as possible, referring to the given information. The new description should be based on the existing description but rewritten better reflect the content of the API description and API endpoint description than before. Just return the input and its description, not individual words. For example:
Category of the API: Data Description of the Category: APIs facilitate the seamless exchange of data between applications and databases, enabling developers to integrate functionalities securely and swiftly, API Name: YouTube Media Downloader API Description: A scraper API for YouTube search and download. Get videos, subtitles, comments without age or region limits (proxy URL supported). API Endpoint Name: Get Channel Details API Endpoint Description: This endpoint fetches details of a YouTube channel.
List of input parameters:
Input parameter name: channelld Description: Channel ID, custom URL name, or handle. @ is required prefix for channel handle.
Input parameter name: lang Description: Language code (ISO-639) for localized results. Defaults to en-US. Unsupported codes will fallback en-US. For this, you should return:
[ ["'hannelld", "The unique identifier for the YouTube channel, which can be the channel's ID, custom URL name, or a channel handle. When using a channel handle, ensure to prefix it with "@" (e.g., @c @channelnamee"" "".). ["lang", "The language code (ISO-639) used to specify the language for the localized results. If not provided, the default is 'en-US'. In case unsupported language code supplied, the results will revert to 'cn-US"".. Now, I'll give you another description. Follow the instructions, referring to the example.
Write as specifically as possible, referring to the given information. The new description should be based the existing description but written in way that better reflects the content of the API description and API endpoint description than before. Just return the input and its description, not individual words.
Table 7: Prompt used to generate output entity description
Prompt for generating output entity descriptions
System
You an intelligent annotator. Your mission is to write the description of the output components of an API endpoint, referring to the given information below.
For example:
Category of the API: Data Description of the Category: APIs facilitate the seamless exchange of data between applications and databases, enabling developers integrate functionalities securely and swiftly, API Name: YouTube Media Downloader API Description: A scraper API for YouTube search and download. Get videos, subtitles, comments without age or region limits (proxy URL supported). API Endpoint Name: Get Channel Details API Endpoint Description: This endpoint fetches details of a YouTube channel.
Based on the given description, write the description of the output component of this API endpoint. Write as specifically as possible. Do not generate examples for each component. The description should reflect closely as possible the description of the API and the API endpoint, someone seeing this API endpoint for the first time can understand exactly what the output compo- nent means. (A component separated with refers to the hierarchy of the schema. For example, avatar height refers to the height of the avatar.)
that even
13

---

Published as a conference paper at ICLR 2025
Output components: {{ "name": status", J. 'name": type" J. "name": id" }.{ "name": name" }.
"name": handle" }. 'name": description" }. "name": Veriiiie J. 'name": is isVriffiedArtist "name": subseriberCoumtTlet"" }. "name": videoCountText") }. 'name": viewCountText" }, "name": joinedDateTextt" ), 'name": country" }. 'name": links title" }. "name": links url"
J. { "name": avatar-url" J. 'name": avatar-width" J. { 'name": avatar- } For this example. you have return.
{{ "name": status", "description": Indicates whether the API call was successful. True means the call was successful, while False means it failed" }. { 'name": type". 'description": Specifies the type of Tubeube channel, such as 'User' or 'Brand', indicating the category of the channel".. }. { "name": id", 'description": The unique identifier assigned to the YouTube channel, which can be used to reference the channel in other API calls or services". }. { 'name": name", "description": The official name of the YouTube channel as displayed on the platform, which is set by the channel owner". J. { "name": handle", 'description": The unique handle of the YouTube channel, which often appears in the URL of the channel's 's page". }. { 'name": description", "description": A textual description provided by the channel owner that gives an overview of the channel's 's content, themes, and purpose". }. { 'name": is Verified" 'description": Indicates whether the YouTube channel is verified by YouTuub. A verified status signifies authenticity and is usually granted to public figures, brands, and popular content creators". ). "name": is AerifiedArttist" "description": Specifies if the YouTube channel is recognized as a verified artist's channel, which is special status for musicians and bands to highlight their official content". }.& { 'name": subscriberCoun(Texx numan-readalee representation of the number of subscribers the channel has, formatted for display purposes". }. 'name": videoCounTText", 'description": A human-readanl total number of videos uploaded by the channel, formatted for display purposes". ). "name": view- CountText", 'description": A human-readable representation of the total number of views across all videos on the channel, formatted for display purposes". ). 'name": A human- readable representation of the date when the YouTube channel was created, formatted for display purposes". }. { 'name": country", "description": The country where the YouTube channel is registered or primarily based, providing geographical context". ). 'name": links-title"" 'descrip- tion": The title of an external link provided by the channel, which can lead to the channel's 's social media profiles, websites, other related content". J. { 'name": links-url" 'description": The URL of an external link associated with the channel, which directs users to other online presences of the channel". }. { "name": avatar-urts "description": The URL of the channel's avatar image, which is the profile picture displayed on the channel's 's page". }. { 'name": avatar- tion": The width of the avatar image in pixels, providing information about the image dimensions". ). { "name": avatar-height" height". "description": The height of the avatar image in pixels, providing information about the image dimensions". }
'description": A
representation of the
"ateedDDteeext""" 'description":
'descrip-
Now, I'll give you another API endpoint description. Write the description of the output components and return the same format as the example. Just return the result, not individual words. Based on the given description, write the description of the output components of this API endpoint. Write as specifically as possible. Do not generate examples for each component. The description should reflect the API and the API endpoint as closely as possible, so that even someone seeing this API endpoint for the first time can understand exactly what the output component means. (A component refers to the hierarchy of the schema. For example, avatar-heigh refers to the separated with height of the avatar.)
Fill the < Your response>.
< response>>
A.2 KEYWORDS EXTRACTION
Table 8: Prompt used to extract keywords
Prompt for extracting keywords System
Extract the keywords from the given paragraph. Prioritize proper nouns first and nouns second, selecting up to 4 words that best describe the paragraph. Return the keywords in CSV format. Remember, the maximum is 4 words. Paragraph:
14

---

Published as a conference paper at ICLR 2025
A.3 USER AND SYSTEM ACTION LIST
Our work defines user actions and 8 system actions, which form the basis for conceptualizing interactions. Table and provide the names and descriptions of these actions.
Table 9: User action and description
User Action
Description
Inform intent clear
Say what one wants specifically.
Inform intent clear add
Say what one wants specifically with the information of
input parameter.
Inform intent vague
Say what wants vaguely.
Inform
Inform the requested information to system.
Fail inform
Fail to reply to system's 's request.
Affirm
Agree to the system's 's proposition.
Negate
Deny the system' s proposal.
User bye
Say thank you and goodbye to system.
Table 10: TALM action and description
System Action
Description
Request
Asks some information to user.
Response
Reply to user's ' request based on the result of API call.
Clarify
If user's query is vague,
re-ask to get intent specifically.
Suggest
Making suggestion for an unclear user's intent and ask-
ing whether satisfies the user.
Response fail
Notify the user that the system cannot execute the request
due to insufficient information.
System bye
System says goodbye to user politely.
Call
Call the API with collected information from user else
and don't reply to user yet.
Retriever call
Call the retriever find proper API satisfy user's
quests.
A.4 DIALOGUE STATE AND RETRIEVER STATUS ANNOTATION FORMAT
Our data is annotated with "retriever status" each turn. This indicates whether the retriever was called for each turn of the conversation, the APIs retrieved as a result, and their respective retriever scores. The actions that the TALM should take vary depending API with score of 0)6 or higher, the TALM asks the user for input parameters to call it. If the score between 0.5 and 0.6, the TALM suggests the retrieved API, and if the score lower, asks for clarification of the user's query. Format of retriever status can have three types described below:
the retriever score. If there is
When retriever is not called |Retriever status: false, Retrieved API: none}
Situation where the TALM needs to find the appropriate API to solve the user's query. [Retriever status: true, Retrieved API: (API 1: 0.65, AP12: 0.54, API3: 0.51...))
Situation that TALM needs to obtain an input parameter that the user has not provided. (Retriever status: true, Retrieved API: [Output component of source API procure target API's input parameter paraml + ]]]
Additionally, dataset is labeled with the dialogue state for each turn. The dialogue state includes the API that the TALM currently attempting to execute and the input parameter information col- lected for that API, based the dialogue history. The dialogue state has the following format:
15

---

Published as a conference paper at ICLR 2025
Case1
inforn
Request
Fall
"rame" provinnly
Rice ia June buti dantiennerl
leed anall
Rosut dit 12340
Calldit Aemach sbageey
gai 123404 same: JAA
CallAn2
Inform Inient
Request
Destwerr Call (AP1)
Resporee
Petrieer Goll (APH rentrnd
Gull APPI (awwi, akrijes
dectoarr
Fill inform
Inform
Inform
can't provid id
lied
the time?
Pet 12M4 manc-Mey MMN)
GARARO
Case2
yrwvkesdy
Figure 4: Possible cases of two action sequences according to perform types "Fail inform".
When there is confirmed API
{API confirmed: false, API status: none)
When the API confirmed
{API confirmed: true, API status: {API name: "API1", Required parameters: {paraml: "",
param2: "", Optional parameters: (param3: "}]}
When the API is confirmed and some input parameter information can be extracted from dialogue history
{API confirmed: true, API status: {API name: "API'", Required parameters: {paraml:
"valuel" ", param2: "), Optional parameters: {param3: "value3")]
A.5 VARIATION OF FAIL INFORM ACTION
User can perform "Fail inform" in two ways: either indicating they don't know one parameter while providing the rest, simply stating they don't know the missing parameter without further input. Figure 4 demonstrates the two ways.
A.6 COMPREHENSSIE ACTION SEQUENCES
Assuming that at most two APIs are called in dialogue, total of 23 action sequences are derived for data generation. Among these, 15 sequences involve two APIs, involve one API, and 1 involves a failure to call any APIs. The 15 sequences with two APIs are further categorized based on the type of action sequence request: either directly requesting input parameters from the user ("Request") or making an additional requesting to select an appropriate value from multiple results ("Request- multi").
Table 11: Action Sequences with two APIs
No. Action Sequence
'Inform intent vague', 'Retriever call', 'Suggest', 'Affirm', 'Reques'', 'Fail inform', 'Retriever call', 'Request', 'Inform', 'Call', 'Call', 'Response', "User bye", 'System bye'
'Inform intent vague', 'Retriever call', 'Suggest', 'Affirm', 'Request,, 'Fail inform', 'Retriever call', 'Call', 'Request', 'Infomm', 'Call', 'Response", 'User bye", 'System bye'
3 'Inform intent vague', 'Retriever call', 'Clarify', 'Inform intent clear', 'Retriever call', 'Reques', 'Fail inform', 'Retriever call', 'Call', 'Request,, 'Inform', 'Call', 'Response', "User bye', 'System bye"
4 'Inform intent vague', 'Retriever call', 'Clarify', 'Inform intent clear', 'Retriever call', 'Request', 'Fail inform' Retriever call', 'Request', 'Inform', 'Call', 'Request- multi', 'Inform', 'Call', 'Response,, 'User bye', 'System bye" 'Inform intent vague', 'Retriever call', 'Clarify'. 'Inform intent clear', 'Retriever call', 'Request', 'Fail inform' 'Retriever call', 'Request', IIffrm'', 'Call', 'Request', 'Inform', 'Call', 'Response', 'User bye', 'System bye'
16

---

Published as a conference paper at ICLR 2025
"Inform intent clear', 'Retriever call', 'Request', 'Fal inform', 'Retriever call', 'Call', 'Request-mult', 'Inform', 'Call', 'Response', 'User bye', 'System bye' "Inform intent clear', 'Retriever call', 'Request', 'Fail inform', 'Retriever call', 'Re- quest', 'Inform', 'Call', 'Request', 'Inform', 'Call', 'Response', 'User bye', 'System bye'
'Inform intent clear', "Retriever call', 'Request', 'Fail inform', 'Retriever call', 'Re- quest', 'Infrmm', 'Call', 'Call', 'Response', 'User bye', 'System bye'
"Inform intent clear', "Retriever call', 'Request', 'Fail inform', 'Retriever call', 'Call', 'Request', 'Inform', 'Call', 'Response', 'User bye', 'System bye' 'Inform intent vague", 'Retriever call', 'Suugestt,, 'Affirm', 'Request', 'Fail inform', "Retriever call'' 'Request', 'Inform', 'Call', 'Request-multi' 'Inform', 'Call', 'Re- sponse", 'User bye', 'System bye"
10
"Inform intent vague", 'Retriever call', 'Clarity', 'Inform intent clear', 'Retriever call', 'Request', 'Fail inform', 'Retriever call', 'Request', 'Inform', 'Call', 'Call', 'Response', 'User bye', 'System bye'
12 'Inform intent vague', 'Retriever call', 'Suggest', 'Affirm', 'Request,, 'Fail inform', 'Retriever call', 'Request', 'Inform', 'Call', 'Request', 'Inform', 'Call', 'Response", 'User bye', 'System bye'
13 'Inform intent clear', 'Retriever call', 'Request', 'Fail inform', 'Retriever call', 'Re- quest', 'Inform', 'Call', 'Request-mult', 'Inform', 'Call', 'Response', 'User bye', 'System bye'
14 "Inform intent vague', 'Retriever call', 'Clarify'. 'Inform intent clear', 'Retriever call', 'Request,, 'Fail inform', 'Retriever call', 'Call', 'Request-mult's 'Inform', 'Call', 'Response', 'User bye', 'System bye'
15 'Inform intent vague', 'Retriever call', 'Suggest', 'Affirm', 'Request', 'Fall inform', 'Retriever call', 'Call', 'Request-multi', 'Inform', 'Call', 'Response', 'User bye", "System bye'
Table 12: Action Sequences with one API
No. Action Sequence
'Inform intent vague', 'Retriever call', 'Clarify', 'Inform intent clear', 'Retriever call', 'Request', IIffrm'' 'Call', 'Response', 'User bye', 'System bye"
2 'Inform intent vague', 'Retriever call', "Clarify', 'Inform intent clear add', 'Retriever call', 'Call', 'Response' 'User bye', 'System bye"
3 'Inform intent vague', 'Retriever call', 'Suggest', 'Affirm', 'Request', 'Inform', 'Call', 'Response'. 'User bye', 'System bye'
'Inform intent vague', 'Retriever call', 'Clarify', 'Inform intent clear add', 'Retriever call', 'Rquues', 'Inform', 'Call', 'Response', 'Üser bye', "System bye" "Inform intent clear add', 'Retriever call', 'Request', 'Inform', 'Call', 'Response', User bye". 'System bye" 'Inform intent clear add', 'Retriever call', 'Call', 'Response', 'User bye', 'System bye'
7 'Inform intent clear', 'Retriever call', 'Request", 'Inform', Caall,, 'Response', 'User
bye', 'System bye'
Table 13: Action Sequence with failure
No. Action Sequence
"Inform intent vague', 'Retriever call', 'Suggest', 'Negate', 'Response fail'
A.7
UTTERANCE STYLE
We have defined several utterance styles for some actions to prevent GPT-4o from generating con- sistent speaking styles.
17

---

Published as a conference paper at ICLR 2025
User Action
Inform
Sure! ~ Ok Certainly!
Affirm
Yes, that works., That would be great., Sure, that sounds good., Yes, please pro- ceed.
-Negate
No, that's not what I meant, I'm good. Thank you though, Umm... that's what
I want
System Action
Request
* To call re, need ~. May I ask for mv, Please tell me
Clarify
Could you please provide more I'm not sure understand. Can you elarify ~, Could you explain that in more ~e, Can you clarify your
A.8 SCENARIO INSTRUCTION
We use detailed dialogue scenario instruction to ensure that the predefined interactions are accurately reflected in the dialogue data and that the correct entities are included in each utterance.
Table 14: Example of scenario instruction
Scenario prompt
User turn
-user action: Inform intent vague (Say what one wants vaguely.) -situation: User requests something from the system. User says "Can you provide detailed informa- tion about a city I plan to visit, including its geographical context and population data, so can find some highly-rated local businesses with good reviews and contact details nearby?"
System turn
-system action: Retriever call (Call the retriever to find the proper API to satisfy the user's request.) -ituationn The system, having received the user's query, calls the retriever to find an appropriate API. this turn, the system's 's thought is, "The user seems to have intent. I will call the retriever". Retriever status: retriever_call 'true'. retrieved 'Data loca-local.io lww..a.ssinesss....... www.bbuussiidaaae datssdaaaraded Nearby':
0.56445915, 'Data-loca business Pata-Search In Area': 0.5539355, 'Mapping
'Mapping-placee places Place
properties': 0.5367253, -peation-spott-SSSee Place and Maps Details': 0.5169816
places': 0.53351307, "Data setaa-serpwoww-oe Google
Dialogue state: coi.confirmed: 'false', api_status: 'none"
System turn
-system action: Suggest (Make a suggestion for an unclear user intent ask whether satisfies the user.) -situation: Since the user's query is unclear, no API with a retriever score higher than 0.6 has been found. However, several APIs have scores between 0.5 and 0.6. The system asks whether it would be appropriate to run Data-local business business_data- -eata-Search Search Nearby, which has the highest score among them, and retrieve the result. At this time, the system does mention the name of the API directly. Retriever status: retriever.cal call: 'false", retrieved appi: 'none' Dialogue state: api conffrmie:::: 'false', sti.status: 'none"
User turn
user action: Affirm (Agree to the system's 's proposition.)
18

---

Published as a conference paper at ICLR 2025
-situation:
User agrees with the system's 's proposition. User's speech should follow this format: "Yes, please proceed".
System turn
-system action: Request (Asks some information to user.) -ituuation: System asks user to..
A.9 DST ERROR ANALYSIS
Table 15: DST error analysis for GPT-based models
GPT-335-furbo
GPT-4o-mini
4PT-4-turbo
GPT-40
W GT
W/O GT
W GT
W/O GT
W
W/O GT
GT
W/O GT
# of Error
4128
4512
2781
2177
1515
2117
1257
2169
Generation Err
0
0
0
0
0
API Conf Err (GT (T)
1609
1841
1060
504
211
224
243
1607
API Conf Err (GT'=))
750
410
848
373
692
891
343
133
Format Err
532
502
153
0
0
74
0
531
Slot Err
1139
1674
508
912
430
774
443
22]
Value Err
561
630
398
823
498
626
495
221
Table 16: DST error analysis for (aamma3-Bb-inttuue and TD-Ilama
Llama3-8b-instuuc
TD-Ilama
W GT
W/O GT
W GT
W/O GT
# of Err
3138
5090
492
1873
Generation Er
0
260
1619
API Conf Eir (GT=1) T)
583
1014
30
API Conf Err =
723
923
Format Err
531
319
61
103
Slot Err
1101
2663
23
Value Err
846
1423
134
144
Tables 15 and
the error analysis results for each model on the dialogue state tracking (DST) task. We categorized the errors in DST as follows.
Generation Error: This occurs when the dialogue state dictionary is not generated all. API Confirmation Error (GT True): This error happens when the API is confirmed (api_.oonfieec--rruee) but is incorrectly predicted as not confirmed ampi.coonirmed--falls.
API Confirmation Error (GT = False): This error occurs when the API is not con- but the model incorrectly predicts it confirmed
firmed (api
fapi.cooffirmdd=truu
Format Error: This occurs when the dialogue state does not fully generate all fields such as api.confirme irmed, api.status, required parameters, and optional parameters.
Slot Error: When api-conff rei.coffrrmes is true, this error involves generating dialogue state that does not include all required and optional parameter slots as specified in the API documentation.
Value Error: This error involves incorrectyy extracting the slot's value from the dialogue history, with the following types:
Extracting Input Value from Multiple Result Error: This occurs when an appropriate value cannot selected from multiple results returned by the API output (as seen in turs 6 and 7 of Figure
19

---

Published as a conference paper at ICLR 2025
Inform Intent Add Error: This occurs when there is value within the query that could be used as an input parameter (Inform intent clear add), but the model fails to track it.
Other General Input Parameter Extraction Errors: Errors that occur in typical situations where the input parameter is extracted incorrectly.
The error analysis tables show error counts, where slot and value errors can overlap in single prediction, causing their sum to exceed the total errors. We also provide examples of DST errors as part of qualitative error analysis.
<xxmmplel. Extracting Input Value from Multiple Result Error>
Retriever status: 'Retriever :ttl'''true',, 'retrieved.nd apii: ['getPaymenntniitaiiollnnn--- tionSummary', "Output to procure input parameter uetr of gePaymeentniiiaiionssstttcii: end_to_emd www.eeeddddiiee iddentiiication']]
Call result:
[f end_to_ende and_iientificationn 2200://022222112244446/000020110
Ireeationddat_ttiee time': 2202-00--20114::00000,,,,
12011110-011109::000000
instructed_amount': "1500000.501]
creation
iitatioo_date_tinnee -erpstttedeeeetiooo.. date' 2000::0000200000000200000000000:
"2002-05-25110:15:000,
instructed amouuntttdddmuut' 1750000.00"
Thought: The API call "getsyymnthhiiiiiolltttttiiaauum need to ask the user to select one based on the 'requested execution date'...
returned multiple results. T
Message: Please tell the requested exccution date of the transaction you are interested in: 2021-10-01109::0::0""" or "2021-10-02200:000000""
Label: 'api name", "geePyyenntiiiaioootttttttoooo"" "trprruuuuttiorrrrreeeeeee parameters' ('uetr':"))... "oamee:'gun "ggaaymenttniiatoonnsstttoorrr required.ppramettee 'uetr' -XYZ/122 45678/2021-07-15"
Predict: name"
Example 1 is a situation where the output 'end to end identification' of API 'getPaymentInitia- In this turn,
tionInstructioSSummay can procure the 'uter' of setPayymenniiiiaoonnsstttiioiooi multiple 'end to end identification' values are returned, requiring request to the user to select one value and gather the uter value accordingly. However, it was observed that the model's prediction arbitrarily selected one of the results, which leads to generate wrong dialogue state.
<Example2. Inform Intent Add Error Error>
User: How do I create Basic Plan for recurring billing payments? System:
(retrieved createPlan from the retriever)
API docs:
('api_name"s "omme::''reaae "createPlan',
'input parameter name': 'name". deseriptionn: 'The name of the billing plan that is being created for the purpose of managing pay- ment schedules and billing cycles in the PayPal payment processing system.', .and other input parameter's name and descriptions)...
Message: To call the API to create Basic Plan, need the following information: accessToken description, paymenLDefiiitions, type, merchamtPreferencee, and sandbox.
Label: ('.pp name': 'createPlan', 'required parameters':
"aaceessToken""
'description':
20

---

Published as a conference paper at ICLR 2025
'paymennDefinttion':: 'name': 'Basic Plan', 'type': 'merchantPreferemees" "}. 'op-
sional_parameters" { ]]]]
Predict: {a.. name": "createPlan', 'required parameters': ('accessToken': ", 'description':
"paymentDefinttions"" 'name'; +, 'type": "merrchnntPrferrences"" "), 'optional parameters':
('sandbox': "}}}
Example is case where the input parameter 'name' required for executing the 'createPlan' API specified as the value 'Basic Plan' in user's query. Additionally, the system's 's request action message only inquires about input parameters other than 'name'. such a situation, the dialogue state should be generated with 'name' already populated as 'Basic Plan'. However, it was generated with 'name" left empty, resulting in this case being classified as an error.
A.10
DST ACCURACY BASED ON TURN LENGTH
DST
DST GT
DST Accurscy
2R-2 tate
Ukired-
Turns
Figure 5: DST Accuracy for each model as the number of dialogue turns increases.
A.11 REMOVING MISMATCH ERRORS
Blow examples shows the mismatch errors that occur during edge construction. There is a domain mismatch and an entity mismatch.
Domain mismatch
APT
Domain and Tools: Sports basketapi
API name: PaagueTTopPlayessPaayll
Entity name: tournamentld
Entity Description: The id the specific basketball tournament for which the top players in the playoffs are being retrieved.
APT
Domain and Tools: Sports baseballapi
API name: PlayeReeplaaSeeonSttiiiisss
Entity name: tournamentld
Entity Description: The of the specific baseball tournament for which the regular season statistics of a player are being requested.
Entity mismatch
APT
21

---

Published as a conference paper at ICLR 2025
Domain and Tools: Sports icehockeyapi
API name: PlyyereuulaSeeeonSStiiiiiss
Entity name: playerid
Entity Description: The unique identifier for a specific ice hockey player whose regular
season statisties are being requested.
API 2
Domain and Tools: Sports icehockeyapi
API name: LeaguePlayoffss TopPlayers
Entity name: seasonId
Entity Description: The id of the specific ice hockey season for which the top players being retrieved during the playoffs.
A.12 PROMPT FORMAT FOR THE EXPERIMENT
Table 18 presents the prompt format used the experiments conducted in our work. Both open- source and closed-source L.MSs utilized this format. DST involves predicting all dialogue states present in the format for each dialogue, while action prediction focuses on predicting all actions. In the case of action prediction, "thought" within the format are removed prior to the task. The W/O GT setting requires predicting the dialogue state and action for each turn using the dialogue history in the format without any dialogue states actions included in the reasoning steps (for DST and action prediction, respectively).
A.13 EVALUATION PROMPTS
release the prompts used our experiments. contaiin the prompt used for evaluating edges in graph construction ($3.1), Table [9 includes the prompt used for dialogue state tracking evaluation, Table(20prooidees the prompt used for action prediction evaluation, and presett
the prompt used for faithfulness evaluation. The prompt used in the overall performance task is detailed in the provided l
22

---

Published as a conference paper at ICLR 2025
Table 17: Prompt used to evaluate edges.
Edge Evaluation Prompt
System
Your task is to determine whether the source attribute in the response from the source API is compat- ible with the api input of the target API. Then, craft JSON formatted response that aligns with the expected output of the API, guided by the provided examples. For your judgment, we will provide descriptions tool description, API Documentation, source attribute and target attribute of both APIs.
The judgment is a two step process. In the first step, determine whether the two attributes are compatible based deep understanding of the source attribute and target attribute. Determine whether the source attribute and target attribute are compatible through attribute descriptions. The second step determine whether the input of the target API is compatible with the intent of the target API. If both steps are considered compatible, follow the Output format for True to output the result. not, follow the Output format for False output the result. Your responses adhere specific JSON structure, which is as follows:
Output format for True:
Ieerror:: "", "sesponse"; "<Your Response>")
Output format for False:
["error": "Invalid Edge Error", "response": "<Your_ "eooo__Rssposs>")
The response field should contain the content you formulate based on the API's functionality and the input provided. Ensure that your responses meaningful, directly addressing the API's intended functionality. the provided examples are mostly error messages or lack substantial content, use your judgment create relevant and accurate responses. The key is to maintain the JSON format's integrity while ensuring that your response is an accurate reflection of the API's intended output within the tool. Please note that your answer should not contain anything other than json format object, which should be parsable directly to json. Note that:
Your response should be around 100 to 200 words, containing rich information given the api input parameters. Keep Your answer short and simple.
Your response must be effective and have practical content.
If the api response example if null ineffective, ignore the example and give your inde- pendent response.
User
API Documentation:
source API Documentation JSON file target API Documentation JSON file source attribute: description of source attribute tource attribute: description of target attribute API Examples:
Example input 1: Example response
Example input 2: Example response 2
Example input 3: Example response 3
API Input:
Argument JSON string, e.g:
:
"tool_namer" "SQUAKE",
"api_name"" "Checkhealth", "tool_input", "(),
"strip": "filter"}
23

---

Published as a conference paper at ICLR 2025
Table 18: Prompt Format for the Experiment
Prompt format used in our experiments
Can you provide insights into which articles on NYTimes.com have been shared the most on social media platforms, particularly Facebook, to understand active engagement trends?
User:
System:
Thought:
The user seems to have intent. will call the retriever.
Action: retriever call
Retriever status: {retriever call: true, retrieved api: APl:: 0.7
-PI documentation: {api NYTimesAPI, (with required and optional parameter informa- tions)}
Thought: Based results of the retriever, need to call NYTimesAPI. To call it, I need to ask "share separe.type"aoo and "period" to the user.
Action: request
Retriever status: fretriever.call: false, retrieved none]
Message: Could you please provide the type of sharing on Facebook for articles and the time period for retrieving popular articles?
Dialogue State: <Current dialogue state>>
User: Certainly! The share type is Facebook and the period 7 days.
System:
Diallgue State: <Current dialogue state>>
- -Thought: Based on the user's response, now can call the API NYTimesAPI Action: call
24

---

Published as a conference paper at ICLR 2025
Table 19: Instruction prompt for the Dialogue State Tracking task
Dialogue state tracking task evaluation prompt System
Instruction: You will be given part of a dialogue between the user and the system. In this dialogue, the user is requesting information from the system, and the system will execute an API call to retrieve the necessary information.
Your task to output the appropriate dialogue state for the current turn, based on the dialogue provided.
System Rules:
1. The system selects the API with the highest score from among the APIs in the retriever status that have score of 0.6 or higher and are suitable for processing the user's query.
2. If no API has a score higher than 0.6, the system cannot confirm the API to call.
Dialogue state format:
Case 1. When the API has not been confirmed (if the retrieved API does not have score of 0.6 higher):
sep__confrreed' 'false", 'api_ _eei_status' 'none')
The API is not confirmed, so api confirmedd is set to false.
Therefore, api_status is 'none".
If api_confr is false, api must be 'none'.
Case 2. When the API is confirmed (if the retrieved API has score of 0.6 or higher); 'api_status': ('api_namel' (API1' ('paraml': " 'param2': (valuel"),
('api_confi rmed' rmed'' 'true',
requi red_pirre__parmeteer
('param3': +}}}}
The API is confirmed, so api.confirmed is set to true.
The api status contains the name of the API and the input parameter list needed for the API call. Any parameter whose value can be extracted from the dialogue history will have its value filled in.
The 'paraml', 'param2', and 'param3' in Case 2 are just example values. Do not use these parameters. Refer to the given API documentation on each turn.
The input parameters should always be determined by consulting the API documentation. Do not hallucinate them.
Now, part of the dialogue will be given. Just generate the dialogue state in the given format, without adding any extra words.
Dialogue:
(dialogue (dielogue_histozyy
25

---

Published as a conference paper at ICLR 2025
Table 20: Instruction prompt for the Action prediction task
Action prediction task evaluation prompt System
Instruction: You will be given part of dialogue between the user and the system. In this dialogue, the user is requesting information from the system, and the system will execute an API call to retrieve the necessary information.
Your task to predict the action that the system should take after the last utterance of the user. Read the dialogue history and return the one action that is most appropriate for the system take next. The actions that the system can take are as follows:
Request: Asks the user for some information.
Response: Replies to the user's request based on the result the API call.
specify their intent. If there is API in the most recently retrieved results with a score above 0.5, "Clarify" is required.
Clarify: If the user's query vague, re-ask the
Suggest: Makes a suggestion for an unclear user's intent and asks whether satisfies the user. If there is API in the most recently retrieved results with score above 0.5 but none exceeding 0.6, a "Suggest" action required.
Response fail: Notifies the user that the system cannot execute the request due to insuffi cient information.
System bye: Politely says goodbye to the user.
Call: Calls the API with the collected information from the user or other sources but does not reply the user yet.
Retriever call: Calls the retriever to find the proper API to satisfy the user's request, The system should call the retriever in the following two situations:
1. When the user specifies a requirement, and the system needs to search for an API to fulfill it.
When the user does provide the input parameters required for API call, and the system needs to search for another API to obtain those parameters.
Of the eight actions given, return only the that you think is most appropriate. Do not return any value other than the action provided above. Just return the action, not a single word more. Dialogue History:
[dialogue_hies history)
Table 21: Instruction prompt for the Faithfulness task
Faithfulness task evaluation prompt
System
Instruction: You will be given part of dialogue between the user and the system. In this dialogue, user is requesting information from the system, and the system will execute an API call to retrieve the necessary information. Your task is to generate response that satisfies the user's initial query based on the API call results provided the dialogue history. Dialogue History:
(dial aollooeehhhiec storry)
26


Reviews:
[{"review_id": "FyFdCfqTKY", "reviewer": "Reviewer_4sGc", "summary": "This work introduces a new dataset for tool calling in LLMs for a multi-turn setup. It consists of conversations where the API calls are dependent on the context in the previous turns as well as require the system to request/clarify more information from the user. The dataset has been generated. The benchmark is evaluated on three tasks - action prediction, dialogue state tracking, and correctness of the API call. Experiments are conducted on four GPT-based versions and one fine-tuned LLaMA.\n\nScore was updated from 6-> 8. See comments for the updated score", "questions": "Questions:\n1. What was the setting for the GPT-based experiments - zero-shot or in-context learning? Please add these details in the next version of the draft\n2. Could you explain why does GPT-4o-mini perform better without ground truth in Table 4?\nWhat are the domains of the examples in the dataset?\n\n\nSuggestions:\n\n1. Please mention the language of the dataset in the paper \n2. Please include qualitative examples from the dataset describing (i) examples after the data collection is complete and (ii) failures of respective LLMs\n3. Please report at least three significant digits when reporting the results\n4. As it is a multi-turn dataset, inclusion of task completion metric - when the model performs all the correct api calls at every turn it is marked as a complete task. See goal accuracy under https://aclanthology.org/2022.acl-short.35 \n5. Please include vanilla LLaMa results in Table 5. \n6. While GPT based results are not reproducible (based on API changes), it may be useful to release the predictions from those runs for future work to be built on this work.\n7. While it is exciting to benchmark the models on, non LLM based baselines such as BERT based/T5 based dialogue state tracking/action recognition should also be evaluated for completeness\n8. The dataset construction section is a little hard to understand. Perhaps a running example may make the section clear.\n9. [Future work] As API calls are similar to function calling in Python, it would be useful to also benchmark CodeLLaMa/finet-uned CodeLLaMa for the given benchmark\n\nMissing References:\n\nLéo Jacqmin, Lina M. Rojas Barahona, and Benoit Favre. 2022. “Do you follow me?”: A Survey of Recent Approaches in Dialogue State Tracking. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 336–350, Edinburgh, UK. Association for Computational Linguistics.\n\nSchick, Timo, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda and Thomas Scialom. “Toolformer: Language Models Can Teach Themselves to Use Tools.” ArXiv abs/2302.04761 (2023): n. pag.\n\n\nNikita Moghe, Patrick Xia, Jacob Andreas, Jason Eisner, Benjamin Van Durme, and Harsh Jhamtani. 2024. Interpreting User Requests in the Context of Natural Language Standing Instructions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4043–4060, Mexico City, Mexico. Association for Computational Linguistics.\n\nPlease see the related work of: \n\n\nQin, Yujia, Shi Liang, Yining Ye, Kunlun Zhu, Lan Yan, Ya-Ting Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Marc H. Gerstein, Dahai Li, Zhiyuan Liu and Maosong Sun. “ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs.” ArXiv abs/2307.16789 (2023): n. pag.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Moving to multi-turn use of tool calling is new and timely to the TALM literature.\n2. Instead of relying on one task for evaluation, the benchmark supports three smaller tasks to get a better understanding of the model's performance.", "weaknesses": "1. Non-LLM based baselines need to be included for completeness\n\n2. The dataset was constructed using an LLM (GPT4-o-mini) - perhaps some artifacts in generation can lead to undesirable effects . The authors back up the dataset quality through human evaluation as well, but perhaps a blind test prepared by human authors can also be helpful.\n\n3. The experiments in this work are still preliminary (see suggestions). Addition of more experimental details, clearly signifying the type of prompting baselines, inclusion of another open source LLM, experimenting with chain-of-thought like prompting (https://arxiv.org/abs/2403.04656) can also be useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MW04Zadasu", "reviewer": "Reviewer_qYBo", "summary": "The paper addresses the limitations of existing benchmarks for Tool-Augmented Language Models (TALMs), which often feature simplistic, single-turn dialogues that don't reflect real-world complexities. To bridge this gap, the authors introduce ToolDial, a dataset comprising 11,111 multi-turn dialogues with an average of 8.95 turns per dialogue, based on APIs from RapidAPI. ToolDial is designed to simulate rich user-system interactions by incorporating 16 types of user and system actions. The dataset also includes scenarios where the TALM must proactively seek additional APIs when the user fails to provide necessary information. To create ToolDial, the authors develop a method for generating an API graph that represents input and output compatibility between APIs, enabling sequential API calls. They then evaluate various language models on tasks like predicting correct actions, selecting appropriate APIs, and extracting input parameters from dialogue history. The results indicate that modern language models still have significant room for improvements.", "questions": "In Figure 2, the agent response contains symbolic terms from the API (seasonId and tournamentId). I am curious about the reasons for this design. Does this mean you are also showing the APIs to the users at the same time? This will make the interactions less natural. \n\nI also have a writing suggestion for the authors. Try to condense the paper and move some essential information from Appendix to the main paper. I am constantly scrolling to Appendix section when reading the paper.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors make appropriate citations and discussions to position their work and justify their contributions. \n\n2. This paper makes solid contributions to the dialogue community. Two of the most popular dialogue state tracking (user intent extractions) are MWOZ and SGD. Both of them are created over 4 years ago. This is a timely work to establish a new benchmark. It also highlights many-turn interactions which differentiates itself from other API benchmark (e.g. API bank)", "weaknesses": "1. Although I recognize the contributions this paper makes to advance the development of dialogue systems, I am not sure the proposed methodologies for creating the dataset are adequately novel or aligned with the values of ICLR. Maybe the authors can provide more evidence and arguments in the rebuttal.\n\n2. Lack of details about the dialogues and APIs: I would like to (1) see more descriptions of how complex the conversations are, as this is something the authors claim as a novelty, and (2) have a clear view of the APIs—what domains are you using and what types of arguments are you employing? Is there any fundamental difference between yours and the SGD?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WiPvbpj3Ja", "reviewer": "Reviewer_fu86", "summary": "This paper introduces ToolDial, a dataset specifically designed for Tool-Augmented Language Models (TALMs), focusing on simulating complex, real-world, multi-turn dialogues that involve external API usage. ToolDial consists of 11,111 dialogues, each structured to mirror realistic, multi-step user-system interactions where the model may need to request additional information or call relevant APIs in response to the user’s needs. The authors also propose a approach to dataset creation by using an API graph, which maps input-output compatibility between APIs. This structure enables TALMs to manage the sequence and dependencies of API calls within dialogues more effectively. \n\nThe paper evaluates various language models on ToolDial, providing valuable insights into their performance, particularly in handling multi-turn interactions requiring tool usage. This assessment highlights critical areas where TALMs could improve, notably in maintaining coherence and accuracy over extended dialogues.", "questions": "1. The concept of \"correctness\" is unclear. Clearer definitions and criteria for correctness could improve result interpretability. It is calculated at the session level, right?\n\n2. It is unclear how many test samples were used in Tables 4 and 5. The presence of integer scores is unusual, given that the test set consists of 1,166 samples. Including precise sample counts and calculation methods could enhance transparency.\n\n3. It remains uncertain if DST or action prediction are essential to achieving high correctness scores. It seems models could achieve high correctness even with low DST or action prediction accuracy. The authors are encouraged to conduct an ablation study to determine the necessity and impact of DST and action prediction on overall performance.\n\n4. The paper would benefit from a comparison with other works that utilize Tool graphs, such as “Taskbench: Benchmarking Large Language Models for Task Automation.”", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "(1) The ToolDial dataset addresses a gap in resources for TALMs, providing extensive and nuanced multi-turn dialogue interactions that are more reflective of real-world use cases.\n\n(2) Dataset generation leverages an API graph with GPT models, reducing the reliance on human-curated data and offering a scalable approach that may inspire similar future methodologies.\n\n(3) By evaluating multiple language models on ToolDial, the paper provides actionable insights into model performance, helping to identify specific challenges TALMs face in multi-turn, tool-augmented interactions.", "weaknesses": "1. Heavy reliance on GPT-generated dialogues introduces the risk of bias and synthetic errors inherent to the model, which could undermine the dataset’s realism.\n\n2. The paper lacks a detailed error analysis, missing an opportunity to discuss specific areas where TALMs underperform and how ToolDial might be used to address these issues.\n\n3. The evaluation process has potential reliability issues:\n(1) There are concerns that the upper bound results (when using ground truth) are lower than expected, especially with the TD-Llama model, which performed worse than anticipated.\n(2) TD-Llama (finetuned on Llama3-8b) scores lower than GPT-3.5-turbo in many aspect, a result that appears counterintuitive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)