ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models

Authors: Jeonghoon Shim, Gyuhyeon Seo, Cheongsu Lim, Yohan Jo
Primary area: datasets and benchmarks
Keywords: Tool-augmented language models

Abstract:
Tool-Augmented Language Models (TALMs) leverage external APIs to answer user queries across various domains. However, existing benchmark datasets for TALM research often feature simplistic dialogues that do not reflect real-world scenarios, such as the need for models to ask clarifying questions or proactively call additional APIs when essential information is missing. To address these limitations, we construct and release ToolDial, a dataset comprising 11,111 multi-turn dialogues, with an average of 8.95 turns per dialogue, based on APIs from RapidAPI. ToolDial has two key characteristics. First, the dialogues incorporate 16 user and system actions (e.g., request, clarify, fail inform) to capture the rich dynamics of real-world interactions. Second, we simulate dialogues where the system requests necessary information from the user based on API documentation and seeks additional APIs if the user fails to provide the required information. To facilitate this process, we introduce a method for generating an API graph that represents input and output compatibility between APIs. Using ToolDial, we evaluate a suite of language models on their ability to predict correct actions and extract input parameter values for API calls from the dialogue history. Modern language models achieve accuracy scores below 70\%, indicating substantial room for improvement. We provide a detailed analysis of the areas where these models fall short.

Decision: accept (poster)
Reviewer scores: 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730440152395

thread_type:
official_review

<thread_object>
creator: reviewer_fu86

<content>
summary:
This paper introduces ToolDial, a dataset specifically designed for Tool-Augmented Language Models (TALMs), focusing on simulating complex, real-world, multi-turn dialogues that involve external API usage. ToolDial consists of 11,111 dialogues, each structured to mirror realistic, multi-step user-system interactions where the model may need to request additional information or call relevant APIs in response to the user’s needs. The authors also propose a approach to dataset creation by using an API graph, which maps input-output compatibility between APIs. This structure enables TALMs to manage the sequence and dependencies of API calls within dialogues more effectively. 

The paper evaluates various language models on ToolDial, providing valuable insights into their performance, particularly in handling multi-turn interactions requiring tool usage. This assessment highlights critical areas where TALMs could improve, notably in maintaining coherence and accuracy over extended dialogues.

strengths:
(1) The ToolDial dataset addresses a gap in resources for TALMs, providing extensive and nuanced multi-turn dialogue interactions that are more reflective of real-world use cases.

(2) Dataset generation leverages an API graph with GPT models, reducing the reliance on human-curated data and offering a scalable approach that may inspire similar future methodologies.

(3) By evaluating multiple language models on ToolDial, the paper provides actionable insights into model performance, helping to identify specific challenges TALMs face in multi-turn, tool-augmented interactions.

weaknesses:
1. Heavy reliance on GPT-generated dialogues introduces the risk of bias and synthetic errors inherent to the model, which could undermine the dataset’s realism.

2. The paper lacks a detailed error analysis, missing an opportunity to discuss specific areas where TALMs underperform and how ToolDial might be used to address these issues.

3. The evaluation process has potential reliability issues:
(1) There are concerns that the upper bound results (when using ground truth) are lower than expected, especially with the TD-Llama model, which performed worse than anticipated.
(2) TD-Llama (finetuned on Llama3-8b) scores lower than GPT-3.5-turbo in many aspect, a result that appears counterintuitive.

questions:
1. The concept of "correctness" is unclear. Clearer definitions and criteria for correctness could improve result interpretability. It is calculated at the session level, right?

2. It is unclear how many test samples were used in Tables 4 and 5. The presence of integer scores is unusual, given that the test set consists of 1,166 samples. Including precise sample counts and calculation methods could enhance transparency.

3. It remains uncertain if DST or action prediction are essential to achieving high correctness scores. It seems models could achieve high correctness even with low DST or action prediction accuracy. The authors are encouraged to conduct an ablation study to determine the necessity and impact of DST and action prediction on overall performance.

4. The paper would benefit from a comparison with other works that utilize Tool graphs, such as “Taskbench: Benchmarking Large Language Models for Task Automation.”
</content>
</thread_object>

</thread>
<thread>
creation date:
1730682180313

thread_type:
official_review

<thread_object>
creator: reviewer_qybo

<content>
summary:
The paper addresses the limitations of existing benchmarks for Tool-Augmented Language Models (TALMs), which often feature simplistic, single-turn dialogues that don't reflect real-world complexities. To bridge this gap, the authors introduce ToolDial, a dataset comprising 11,111 multi-turn dialogues with an average of 8.95 turns per dialogue, based on APIs from RapidAPI. ToolDial is designed to simulate rich user-system interactions by incorporating 16 types of user and system actions. The dataset also includes scenarios where the TALM must proactively seek additional APIs when the user fails to provide necessary information. To create ToolDial, the authors develop a method for generating an API graph that represents input and output compatibility between APIs, enabling sequential API calls. They then evaluate various language models on tasks like predicting correct actions, selecting appropriate APIs, and extracting input parameters from dialogue history. The results indicate that modern language models still have significant room for improvements.

strengths:
1. The authors make appropriate citations and discussions to position their work and justify their contributions. 

2. This paper makes solid contributions to the dialogue community. Two of the most popular dialogue state tracking (user intent extractions) are MWOZ and SGD. Both of them are created over 4 years ago. This is a timely work to establish a new benchmark. It also highlights many-turn interactions which differentiates itself from other API benchmark (e.g. API bank)

weaknesses:
1. Although I recognize the contributions this paper makes to advance the development of dialogue systems, I am not sure the proposed methodologies for creating the dataset are adequately novel or aligned with the values of ICLR. Maybe the authors can provide more evidence and arguments in the rebuttal.

2. Lack of details about the dialogues and APIs: I would like to (1) see more descriptions of how complex the conversations are, as this is something the authors claim as a novelty, and (2) have a clear view of the APIs—what domains are you using and what types of arguments are you employing? Is there any fundamental difference between yours and the SGD?

questions:
In Figure 2, the agent response contains symbolic terms from the API (seasonId and tournamentId). I am curious about the reasons for this design. Does this mean you are also showing the APIs to the users at the same time? This will make the interactions less natural. 

I also have a writing suggestion for the authors. Try to condense the paper and move some essential information from Appendix to the main paper. I am constantly scrolling to Appendix section when reading the paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710639869

thread_type:
official_review

<thread_object>
creator: reviewer_4sgc

<content>
summary:
This work introduces a new dataset for tool calling in LLMs for a multi-turn setup. It consists of conversations where the API calls are dependent on the context in the previous turns as well as require the system to request/clarify more information from the user. The dataset has been generated. The benchmark is evaluated on three tasks - action prediction, dialogue state tracking, and correctness of the API call. Experiments are conducted on four GPT-based versions and one fine-tuned LLaMA.

Score was updated from 6-> 8. See comments for the updated score

strengths:
1. Moving to multi-turn use of tool calling is new and timely to the TALM literature.
2. Instead of relying on one task for evaluation, the benchmark supports three smaller tasks to get a better understanding of the model's performance.

weaknesses:
1. Non-LLM based baselines need to be included for completeness

2. The dataset was constructed using an LLM (GPT4-o-mini) - perhaps some artifacts in generation can lead to undesirable effects . The authors back up the dataset quality through human evaluation as well, but perhaps a blind test prepared by human authors can also be helpful.

3. The experiments in this work are still preliminary (see suggestions). Addition of more experimental details, clearly signifying the type of prompting baselines, inclusion of another open source LLM, experimenting with chain-of-thought like prompting (https://arxiv.org/abs/2403.04656) can also be useful.

questions:
Questions:
1. What was the setting for the GPT-based experiments - zero-shot or in-context learning? Please add these details in the next version of the draft
2. Could you explain why does GPT-4o-mini perform better without ground truth in Table 4?
What are the domains of the examples in the dataset?


Suggestions:

1. Please mention the language of the dataset in the paper 
2. Please include qualitative examples from the dataset describing (i) examples after the data collection is complete and (ii) failures of respective LLMs
3. Please report at least three significant digits when reporting the results
4. As it is a multi-turn dataset, inclusion of task completion metric - when the model performs all the correct api calls at every turn it is marked as a complete task. See goal accuracy under https://aclanthology.org/2022.acl-short.35 
5. Please include vanilla LLaMa results in Table 5. 
6. While GPT based results are not reproducible (based on API changes), it may be useful to release the predictions from those runs for future work to be built on this work.
7. While it is exciting to benchmark the models on, non LLM based baselines such as BERT based/T5 based dialogue state tracking/action recognition should also be evaluated for completeness
8. The dataset construction section is a little hard to understand. Perhaps a running example may make the section clear.
9. [Future work] As API calls are similar to function calling in Python, it would be useful to also benchmark CodeLLaMa/finet-uned CodeLLaMa for the given benchmark

Missing References:

Léo Jacqmin, Lina M. Rojas Barahona, and Benoit Favre. 2022. “Do you follow me?”: A Survey of Recent Approaches in Dialogue State Tracking. In Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 336–350, Edinburgh, UK. Association for Computational Linguistics.

Schick, Timo, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda and Thomas Scialom. “Toolformer: Language Models Can Teach Themselves to Use Tools.” ArXiv abs/2302.04761 (2023): n. pag.


Nikita Moghe, Patrick Xia, Jacob Andreas, Jason Eisner, Benjamin Van Durme, and Harsh Jhamtani. 2024. Interpreting User Requests in the Context of Natural Language Standing Instructions. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4043–4060, Mexico City, Mexico. Association for Computational Linguistics.

Please see the related work of: 


Qin, Yujia, Shi Liang, Yining Ye, Kunlun Zhu, Lan Yan, Ya-Ting Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Runchu Tian, Ruobing Xie, Jie Zhou, Marc H. Gerstein, Dahai Li, Zhiyuan Liu and Maosong Sun. “ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs.” ArXiv abs/2307.16789 (2023): n. pag.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732426687958

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Review Summary

comment:
Dear Area Chair and Reviewers,

We sincerely appreciate the time and effort you have dedicated to thoroughly reviewing our paper. The following summarizes the contributions of our work as highlighted by the reviewers:

1. The ToolDial dataset addresses a critical gap in TALM literature by providing extensive, nuanced multi-turn dialogue interactions that facilitate timely exploration of tool usage in real-world scenarios (Reviewers fu86, 4sGc).

2. The benchmark's smaller, focused tasks enable a comprehensive evaluation of various language models on ToolDial, offering actionable insights into the challenges TALMs face in multi-turn, tool-augmented interactions (Reviewers 4sGc, fu86).

3. The dataset is generated using an API graph with GPT models, reducing reliance on human-curated data and offering a scalable and inspiring approach that could inform future research (Reviewer fu86).

4. Our paper establishes a timely new benchmark that emphasizes multi-turn interactions, distinguishing it from older API benchmarks like API Bank, while addressing gaps left by aging datasets such as MWOZ and SGD. As a result, the paper makes solid contributions to the dialogue community (Reviewer qYBo).

The reviewers have also pointed out several weaknesses, which we have carefully addressed through detailed responses and supporting experiments and analyses. Those include internal biases in GPT during dialogue generation, additional baselines and error analysis, unclarity in the writing, and some missing related work. We have revised the paper to address these concerns and enhance the clarity.

Once again, we deeply appreciate the constructive feedback provided, which has greatly advanced our research.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732533489676

thread_type:
official_comment

<thread_object>
creator: area_chair_3v2x

<content>
title:
Action Required: Respond to Author Rebuttals - Nov 27

comment:
Dear ICLR Reviewers,

The author discussion phase is ending soon. Please promptly review and respond to author rebuttals for your assigned papers. Your engagement is critical for the decision-making process.

Deadlines:
- November 26: Last day for reviewers to ask questions to authors.
- November 27: Last day for authors to respond to reviewers.
- November 28 - December 10: Reviewer and area chair discussion phase.

Thank you for your timely attention to this matter.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces ToolDial, a new dataset for evaluating Tool-Augmented Language Models (TALMs) in multi-turn dialogue settings. The dataset contains 11,111 dialogues with an average of 8.95 turns, featuring complex interactions where API calls depend on context from previous turns and may require requesting additional information from users.

The reviewers value the timeliness and importance of addressing multi-turn tool calling capabilities, as well as the comprehensive evaluation across different tasks. However, reviewers are still a bit concerned about LLM-generated data quality,  evaluation metrics, and missing detailed error analysis.

Overall, I recommend acceptance given the paper's contribution to tool learning research.