DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialogues
Authors: Kyochul Jang, Donghyeon Lee, Kyusik Kim, Dongseok Heo, Taewhoo Lee, Woojeong Kim
Accepted at: acl25

DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialogues

Kyochul Jang1    Donghyeon Lee3,4    Kyusik Kim2    Dongseok Heo1
Taewho Lee3,4    Woojeong Kim4    Bongwon Suh 1,2†

1IPAI, Seoul National University
2Department of Intelligence and Information, Seoul National University
3Korea University    4AIGEN Sciences    5Cornell University
{kyochul, kyu823, ty8900, bongwon}@snu.ac.kr   {dong9733, taewho}@korea.ac.kr
wk247@cornell.edu

Abstract

Existing function-calling benchmarks focus on single-turn interactions. However, they overlook the complexity of real-world scenarios. To quantify how existing benchmarks address practical applications, we introduce DICE-SCORE, a metric that evaluates the dispersion of tool-related information such as function name and parameter values throughout the dialogue. Analyzing existing benchmarks through DICE-SCORE reveals notably low scores, highlighting the need for more realistic scenarios. To address this gap, we present DICE-BENCH, a framework that constructs practical function-calling datasets by synthesizing conversations through a tool graph that maintains dependencies across rounds and a multi-agent system with distinct personas to enhance dialogue naturalness. The final dataset comprises 1,607 high-DICE-SCORE instances. Our experiments on 19 LLMs with DICE-BENCH show that significant advances are still required before such models can be deployed effectively in real-world settings. Our code1, and data2 are all publicly available.

1 Introduction

Function-calling refers to the ability of LLMs to execute predefined external functions (or APIs) through generating structured calls from natural language input (Qin et al., 2024; Park et al., 2023; Gong et al., 2024). While early virtual assistants (VAs) relied on rigid rule-based systems, LLM-integrated VAs now combine reasoning with external data retrieval (Weizenbaum, 1966). As interactions grow more complex, there is a growing need for VAs to support multi-party and multi-turn dialogues (Guan et al., 2023; Vu et al., 2024).

Figure 1: Illustration of a Single-Round, Four-Party Dialogue in DICE-BENCH. LLMs must identify function-related information from multi-party dialogue. Relevant values in the dialogue are color-coded to match their function call components.

Despite advancements, most function-calling benchmarks assume all API parameters are present in a single user utterance, overlooking real-world group chat scenarios (Chen et al., 2024; Zhuang et al., 2023; Basu et al., 2024). For example, when people in a group chat decide where to go and which flight to take, a VA must be able to track multiple turns of dialogue to book a hotel and flight ticket. Such complexities remain largely unaddressed by existing benchmarks.

We therefore present DICE-BENCH (Dialogue-based Interactive Calling Evaluation Benchmark), a framework designed to evaluate function-calling performance in realistic multi-party, multi-round dialogues. In our paper, round is defined as a complete dialogue cycle consisting of multiple user utterances and system responses, and dependency as the condition where the current round’s context
<table>
  <tr>
    <th>Benchmark</th>
    <th># Instances</th>
    <th colspan="2">Tool</th>
    <th colspan="2">Dialogue</th>
    <th>DICE-SCORE</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th># Tools</th>
    <th>Dependency</th>
    <th>Multi-party</th>
    <th>Multi-round</th>
    <th></th>
  </tr>
  <tr>
    <td>APIBench (Patil et al., 2023)</td>
    <td>17002</td>
    <td>1645</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>0.7895</td>
  </tr>
  <tr>
    <td>ToolAlpaca (Tang et al., 2023)</td>
    <td>3938</td>
    <td>400</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>0.5660</td>
  </tr>
  <tr>
    <td>ToolLLM (Qin et al., 2023)</td>
    <td>12657</td>
    <td>16464</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>0.5989</td>
  </tr>
  <tr>
    <td>ToolBench (Xu et al., 2023)</td>
    <td>2746</td>
    <td>8</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>0.7225</td>
  </tr>
  <tr>
    <td>API-Bank (Li et al., 2023)</td>
    <td>2202</td>
    <td>2211</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>1.6318</td>
  </tr>
  <tr>
    <td>MetaTool (Huang et al., 2024)</td>
    <td>21127</td>
    <td>199</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>0.5437</td>
  </tr>
  <tr>
    <td>TaskBench (Shen et al., 2024)</td>
    <td>17331</td>
    <td>103</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>0.6415</td>
  </tr>
  <tr>
    <td>RoTBench (Ye et al., 2024b)</td>
    <td>945</td>
    <td>568</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>0.5651</td>
  </tr>
  <tr>
    <th>DICE-BENCH (ours)</th>
    <th>1607</th>
    <th>124</th>
    <th>✓</th>
    <th>✓</th>
    <th>✓</th>
    <th><b>3.6444</b></th>
  </tr>
</table>

Table 1: Baseline Comparison. We compare various function-calling benchmark datasets with DICE-BENCH, demonstrating that DICE-BENCH is the only benchmark to encompass both multi-party and multi-round dialogues. We also report DICE-SCORE for every dataset, showing that DICE-BENCH handles more realistic tasks.

depends on either the previous round’s tool-call output or the content (See Appendix C for illustration of multi-round and dependency).

In real-world group chats, key details often emerge across multiple turns, requiring accurate tracking for coherent interactions. To address this, we generate diverse dialogues using a multi-agent system, where each agent has a distinct persona. Then, we refine the dataset through automated, rule-based, and human criteria-based filtering. After rigorous validation, our benchmark includes 1,607 instances covering both single-round and multi-round dialogues.

Existing benchmarks do not assess function-calling in multi-round, multi-party dialogues, which makes accurate execution challenging due to the tool-related information being dispersed across turns. To quantify this complexity, we propose DICE-SCORE (Dialogue Information Coverage Evaluation Score), which measures how fragmented tool-related details are within the input context. A higher DICE-SCORE indicates greater dispersion, requiring LLMs to integrate scattered information across turns. Experiments on various LLMs show a significant performance drop as DICE-SCORE increases, underscoring the need for improved dialogue-tracking and context-integration strategies.

Our contributions are as follows.

• To the best of our knowledge, DICE-BENCH is the first multi-round, multi-party benchmark for function-calling, grounded in realistic group chat data and validated through both rule-based and human evaluations.

• We introduce the DICE-SCORE, a novel metric that captures the complexity of multi-party conversation in the real world by assessing the difficulty of retrieving scattered function call information.

• We conducted a thorough evaluation on diverse closed-source and open-source LLMs, analyzing their performance and error cases to provide valuable insights into their limitations in handling fragmented multi-round dialogue contexts.

2 Related Work

2.1 Function-Calling Benchmark

Recent benchmarks have been developed to evaluate function-calling performance in LLMs (Wang et al., 2024b; Kim et al., 2024). Most focus on single-command scenarios (Patil et al., 2023; Huang et al., 2024; Qu et al., 2024), while some extend to multi-turn interactions with a single user, increasing task complexity (Li et al., 2023; Wang et al., 2024b; Tang et al., 2023). However, these approaches overlook the challenges of multi-party dialogues, where tool-related information is distributed across multiple speakers.

Moreover, many existing benchmarks lack rigorous human validation of both tools and instances, leading to datasets that may not reflect real-world conditions (Erdogan et al., 2024; Qin et al., 2023; Shen et al., 2024). To address these gaps, we introduce DICE-BENCH, a benchmark that captures multi-turn, multi-party interactions with comprehensive human validation. Additionally, we propose DICE-SCORE, a metric designed to quantify the dispersion of tool-related information across dialogue contexts, ensuring alignment with real-world complexities.
Figure 2: DICE-BENCH data-generation pipeline. (1) In the Tool Graph Construction phase, we build a tool graph from tool collections. (2) In the Scenario Configuration step, we sample tool chains and configure dialogue types, personas, and the target number of rounds. (3) In the Dialogue Simulation phase, we iteratively generate parameter values for each tool and simulate corresponding multi-party dialogues across N rounds.

2.2 Interactive System and Dialogue

The integration of LLMs into VAs has enhanced their ability to process complex tasks through natural language understanding and reasoning (Sezgin, 2024). Function-calling further improves this capability by enabling VAs to infer intent before execution, unlike rule-based systems that follow direct commands (Zhang et al., 2025; Guan et al., 2023; Campagna et al., 2019). As user interactions grow more complex, studies emphasize the need for VAs to handle multi-turn and multi-party dialogues (Abdelaziz et al., 2024; Schick et al., 2023; Khurana et al., 2024).

Multi-party conversations introduce additional challenges, as they involve diverse dialogue structures shaped by participants’ goals and strategies (Richards and Wessel, 2025; Yeomans et al., 2022; Biber et al., 2011; Reece et al., 2023). Academic research categorizes conversations into six types, Persuasion, Inquiry, Discovery, Negotiation, Information-Seeking, Deliberation, and Eristic, each affecting communication complexity differently (Walton, 2010; Walton and Krabbe, 1995). While function-calling has advanced Human-VA interaction, current benchmarks do not adequately assess multi-party, context-rich dialogues (Inoue et al., 2025; Farn and Shin, 2023). To address this, we introduce DICE-BENCH, a benchmark designed to evaluate LLMs in real-world multi-party interactions.

3 DICE-BENCH

In this section, we introduce DICE-BENCH, a benchmark designed to evaluate the function-calling capabilities of LLMs in multi-round, multi-party dialogues. Unlike previous approaches that concentrate on one-on-one Human-LLM interactions, DICE-BENCH presents dialogue-based inputs in which multiple speakers provide scattered pieces of information over several turns. As shown in Figure 2, we also explicitly model inter-round dependencies using Tool Graph. This approach builds upon the concept introduced in TaskBench (Shen et al., 2024).

3.1 Data Construction

The data construction phase consists of three main steps: Tool Graph Construction, Scenario Configuration, and Dialogue Generation. Each step undergoes human review and follows clearly defined criteria to ensure the dialogue data is both realistic and consistent.

Tool Graph Construction. Our objective is to build dialogue data that mirrors realistic, everyday scenarios where function-calling is needed, such as checking the weather, booking a restaurant, or scheduling events. To achieve this goal, we use the set of tools proposed in the TaskBench (Shen et al., 2024) and ToolEyes (Ye et al., 2024a). We then validate these tools through a combination of manual checks by the authors and LLM-based
validation. The two key criteria we used for the filtering are as follows: whether the function calls and parameters realistically reflect daily-life use cases, and whether the collected tools accurately match the intended functions and parameters. After filtering, we construct a Tool Graph to guarantee dependencies between tools.

Formally, we represent our Tool Graph \( \mathcal{G} \) as a directed graph \( \mathcal{G} = (\mathcal{V}, \mathcal{E}) \) where each node \( v \in \mathcal{V} \) corresponds to a tool function. A directed edge \( (v_i, v_j) \in \mathcal{E} \) signifies that tool \( v_j \) depends on the tool \( v_i \), either because \( v_i \) contains required output or parameters for \( v_j \), or because the information produced by \( v_i \) is contextually dependent on the execution of \( v_j \). Therefore, the structure \( \mathcal{G} \) serves as the backbone for multi-round dialogue simulation in a realistic workflow.

Our Tool Graph consists of 124 nodes and 270 edges, yielding a density of 0.0177 and an average out-degree of 2.18. The low density and average out-degree suggest that this graph exhibits a relatively sparse structure, preventing a single function from dominating or becoming overly dependent. This characteristic can offer diverse pathways for automated multi-turn dialogue generation.

Scenario Configuration. We integrate various elements to simulate multi-agent, multi-round dialogues in a natural, human-like manner, ensuring each conversation reflects real-world complexity. We begin by sampling tool chains from the Tool Graph, extracting paths ranging from a single node to four nodes, where each node represents a tool per round. For sampling, we employ Depth-First Search (DFS) to enumerate all possible paths, then randomly select the chain. For example, when sampling tools for a two-round dialogue, the sampled tool chain appears as follows: “[get_weather, book_hotel],” meaning the get_weather function will be used in the first round and the book_hotel function follows.

Next, we assign a dialogue type based on Walton and Krabbe (1995), condensing the seven primary categories into three: persuasion-deliberation-and-negotiation, inquiry-and-information-seeking, and eristic. Although the original reference identifies seven primary types, we merge those that share some similarities. We then vary the number of participants from two to four, spanning a broad complexity range that captures key aspects of real-world multi-party interactions. Lastly, to implement real-world human interactions with distinct personalities, we generate distinct personas for each agent using GPT-4o by leveraging tool information. These configurations cover a broad spectrum of complexity.

Dialogue Generation. After preparing essential components, we generate multi-round dialogues in three key steps. First, we perform Parameter Generation by prompting an LLM to suggest appropriate parameter values for each tool in the chain. If the current round is not the first round, then we include the conversation history and any previously generated virtual tool-call output to the prompt, ensuring contextual continuity.

Next, we carry out Dialogue Simulation using a multi-agent system. Each agent has a distinct persona, and an orchestrator dynamically regulates turn-taking based on the evolving conversation flow. This setup emulates real-world multi-party conversations. Finally, at the end of each round, we store the dialogue along with any generated virtual outputs, which serve as a context for the next round’s parameter generation. We repeat this process \( N \) times, where \( N \) is the length of the chain. Using this approach, we produced a total of 1,800 (450 × 4) dialogues across four rounds.

<table>
  <tr>
    <th>Round</th>
    <th>Initial</th>
    <th>Stage1</th>
    <th>Stage2</th>
    <th>Stage3</th>
    <th>Final</th>
  </tr>
  <tr>
    <td>1</td>
    <td>450</td>
    <td>4</td>
    <td>7</td>
    <td>14</td>
    <td>425</td>
  </tr>
  <tr>
    <td>2</td>
    <td>450</td>
    <td>5</td>
    <td>9</td>
    <td>18</td>
    <td>418</td>
  </tr>
  <tr>
    <td>3</td>
    <td>450</td>
    <td>8</td>
    <td>17</td>
    <td>26</td>
    <td>399</td>
  </tr>
  <tr>
    <td>4</td>
    <td>450</td>
    <td>13</td>
    <td>11</td>
    <td>61</td>
    <td>365</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>1800</th>
    <th>30</th>
    <th>44</th>
    <th>119</th>
    <th>1607</th>
  </tr>
</table>

Table 2: Filtering Statistics per Round. Initial column shows the number of instances before filtering. Stage1–3 show removal counts at each validation step, and Final column shows remaining instances.

3.2 Validation Pipeline

We employ a three-stage filtering process to convert the raw dialogues into high-quality data. After the first automated stage, each subsequent filtering step involves human validation to ensure that the final dataset meets our criteria for realism, coherence, and functional correctness.

Stage 1: Automatic Evaluation. In the initial stage, we use G-Eval (Liu et al., 2023) with GPT-4o to evaluate each dialogue according to six criteria: Coherence, Consistency, Fluency, Human-likeness, Persona Consistency, and Relevance. Each criterion is rated on a 5-point Likert scale. Although
model-based evaluation may introduce certain biases, Liu et al. (2023) have shown a high Spearman correlation between automated scores and human judgments. We then prompt GPT-4o to classify each dialogue into one of the three designated dialogue types. We remove it if a dialogue’s average G-Eval score falls below 4.0 and is assigned an incorrect type.

Stage 2: Rule-Based Filtering. Following the automatic evaluation, we discard dialogues that violate explicit rules. First, any conversation containing GPT-generated refusals (e.g., “I’m sorry, but…”) is removed. Second, we check if at least one user turns explicitly or implicitly addresses an ‘AI’ or “Assistant”. In ambiguous cases, authors revisit each dialogue to confirm whether indirect requests, such as rhetorical questions to AI, are being made.

Stage 3: Criteria-Based Filtering. In the final stage, all authors evaluate each remaining dialogue across three dimensions: Conversation Quality, Functional Integration, and Real-World Applicability. Detailed guidelines are provided in the Appendix P. These dimensions encompass 15 sub-criteria in total, with seven dedicated to conversation quality, five to function integration, and three to overall realism. We remove the instance if a dialogue scores below 10 out of 15.

These three filtering stages produce a curated dataset that maintains coherence and accurately represents challenging function call scenarios. In Table 2, we describe the number of data points that were eliminated at each filtering stage and the number that eventually remained in the final dataset.

3.3 Task Setup and Benchmark Structure

In this section, we explain how our benchmark is structured, and describe our overall task setup. Specifically, we illustrate how multi-round, multi-party dialogues challenge LLMs to aggregate scattered information and perform accurate function calls.

3.3.1 Benchmark Structure

Our dataset comprises four rounds, ranging from Round 1 to Round 4. Each round progressively increases in complexity by expanding the contextual scope and requiring the model to handle diverse personas and manage rapid context shifts, from two participants up to four participants. We also include three distinct dialogue styles to mirror varied real-world scenarios.

We generate 50 dialogues per round for each of the three-party configurations and three dialogue types, yielding 450 dialogues\((450 = 50 * 3 * 3)\) per round. With 4 rounds, this results in a total of 1,800 dialogues\((1800 = 450 * 4)\) overall. After 193 dialogues are removed through the validation pipeline, we obtain 1,607 final instances. Refer to Table 3 for detailed data statistics for each configuration.

<table>
  <tr>
    <th>Index</th>
    <th>1</th>
    <th>2</th>
    <th>3</th>
    <th>4</th>
  </tr>
  <tr>
    <td>Round</td>
    <td>425</td>
    <td>418</td>
    <td>399</td>
    <td>365</td>
  </tr>
  <tr>
    <td>Party</td>
    <td>-</td>
    <td>569</td>
    <td>519</td>
    <td>519</td>
  </tr>
  <tr>
    <td>Dialogue Type</td>
    <td>545</td>
    <td>549</td>
    <td>513</td>
    <td>-</td>
  </tr>
</table>

Table 3: Data Statistics of DICE-BENCH. For the Dialogue Type row, indices 1–3 correspond to “Eristic”, “Persuasion, Deliberation and Negotiation”, and “Inquiry and Information Seeking”, respectively.

3.3.2 Task Setup

In DICE-Bench, our aim is to evaluate how well LLMs can perform function-calling under realistic multi-party dialogue conditions. Therefore, we need to inference LLMs on our synthesized dialogue datasets. The input consists of a multi-round multi-party dialogue, and collected tool documents from Tool Graph Construction phase. The three types of input are fed to the target LLMs as a hard prompt. We define the task as identifying the exact function name and parameter values based on the given user instruction and dialogue. Thus, the benchmark tests the model’s ability to (i) identify the appropriate function among available tools, and (ii) extract or synthesize the correct parameter values within the given conversation. This setup more closely aligns with real-world Human-VA interactions, where relevant context is often distributed throughout extended dialogues rather than being neatly encapsulated in a single instruction.

3.4 DICE-SCORE

We propose DICE-SCORE to quantify how difficult the given input is for function-calling across existing benchmark datasets as they do not fully reflect practical situations. However, the lack of a metric to measure this aspect is hindering the progress towards more challenging tasks. Although some studies have discussed the notion of information
Figure 3: Inverse Correlation between DICE-SCORE and Model Performance. Lower DICE-SCORE indicates that the input instruction is more challenging, suggesting that the LLM is capable of handling complex scenarios.

coverage by quantifying how much of the input context is necessary for answering queries, none have proposed a metric that explicitly captures how dispersed or fragmented these details are within a dialogue for function-calling tasks. Specifically, according to Goldman et al. (2024), "scope" is defined as "how much required data can be found", but does not formalize a direct metric. Also, the existing long-context coverage method Lee et al. (2024) measures how dense the information is distributed throughout the long context, rather than quantifying its sparseness across multiple utterances.

To address this gap, we introduce DICE-SCORE (Dialogue Information Coverage Evaluation Score), a metric that assesses how challenging it is to perform a function call within a given context by estimating the distribution of tool-related knowledge. We designed DICE-SCORE to yield higher scores when there is a large amount of function-related information to identify, but also when this information is distributed sparsely and non-repetitively. This, in turn, makes it more difficult for LLMs to locate the necessary information. Formally, we define the DICE metric as follows:

\[
\text{DICE}(S, T) = \frac{\min\left(|S_{\neq 0}|, T\right) \cdot \sqrt{|S| \cdot T}}{\sum_{i \in S} \ln(1 + \alpha \times S_i)}.
\]

Notation. Let the dialogue consist of \( n \) utterances, and define \( S = (S_1, \ldots, S_n) \) as a vector where each \( S_i \) indicates the number of function-related items mentioned in the \( i \)-th utterance. Removing all zero entries from \( S \) yields the subsequence \( S_{\neq 0} \); therefore \( |S_{\neq 0}| \) equals the number of utterances that mention at least one such item. \( T \) denotes the total number of distinct function-related items that must be identified across the entire dialogue. For example, if the ground truth function-call is book_hotel(Vienna, Austria, 07 - 27), then \( T = 4 \), comprising one for the book_hotel and three for its arguments: Vienna, Austria, and 07 - 27. \( \alpha \) is a positive constant to control a penalty for repeated mentions of the same items. We set \( \alpha = c^2 \), which ensures in the boundary case \( T = |S_{\neq 0}| = 1 \) that the DICE-SCORE remains strictly increasing.

Key Properties. To obtain \( S_i \) in practice, we employ a custom prompt to GPT-4o-mini (details in Appendix D). We highlight four key properties of DICE-SCORE:

1. Coverage vs. Dispersal:
The term \( \min(|S_{\neq 0}|, T) \) rewards spreading items across dialogue turns, aligning with studies on information dispersion in corpus linguistics and multi-turn dialogue systems (Manning and Schütze, 1999; Jurafsky and Martin, 2019).

2. Discouraging Redundancy:
The logarithmic penalty \( \sum_{i \in S} \ln(1 + \alpha \times S_i) \) downweights repeated mentions, similar to TF-IDF weighting in information retrieval (Salton and Buckley, 1988).

3. Scale Adjustment:
The factor \( \sqrt{|S| \times T} \) normalizes the score with respect to dialogue length and item count, analogous to cosine normalization in document similarity (Manning and Schütze, 1999).

4. Balanced Realism: Repeating the same items in every utterance increases the denominator, lowering DICE-SCORE, while mentioning items too sparsely keeps the numerator small.
Thus, a high DICE-SCORE indicates that items are well-distributed across the conversation. Moreover, when the utterance count \( t \) and item repetition remain fixed (i.e., \( T \) is proportional to \( S_i \) for \( S_i \geq 1 \)), we show (Appendix A) that there exists \( \alpha \) with \( e^2 \leq \alpha \) such that the DICE-SCORE strictly increases with the number of distinct tools.

Alignment with Human Evaluation. The proposed metric, DICE-SCORE, was developed to quantify task difficulty across a dataset of 1,607 samples and was validated through human evaluation using a statistically grounded subset of 311 samples. This subset size was determined based on a 95% confidence level, a 5% margin of error, and a conservative estimate of maximum variability (\( p = 0.5 \)). The calculation incorporated a Finite Population Correction (FPC) to account for the dataset’s finite size.

Samples were proportionally drawn from four rounds of data, 425, 418, 399, and 365 samples in Rounds 1 to 4, resulting in evaluation subsets of 82, 81, 77, and 71 samples, respectively. Human participants completed function-calling tasks for each round in the sample, achieving accuracies of 80.5%, 69.1%, 51.9%, and 49.3%. Corresponding values of DICE-SCORE, which reflect increasing task difficulty, were 1.42, 3.25, 4.55, and 5.36. This statistics are summarized in Table 4.

To assess the alignment between human performance and the proposed difficulty metric, we computed the Pearson correlation coefficient. The analysis revealed a strong negative correlation (\( r \approx -0.984 \)), indicating that higher DICE-SCORE values were associated with lower human accuracy. This trend is consistent across rounds, from 80.5% accuracy at \( DICE = 1.42 \) (Round 1) to 49.3% at \( DICE = 5.36 \) (Round 4). A t-test confirmed the statistical significance of this correlation, yielding a t-value of approximately –8.38 (\( p < 0.01, 2 \) degrees of freedom).

These results demonstrate that DICE-SCORE effectively captures the difficulty of input dataset, with both human evaluation and statistical analysis supporting its validity. Please refer to Appendix B for calculation details. Moreover, in Appendix A, we show how DICE-SCORE performs as expected when tool-related items increase, as long as dispersal and repetition remain balanced. A higher DICE-SCORE means crucial information is spread over multiple turns. Lastly,

<table>
  <tr>
    <th>Round</th>
    <th>\( N \)</th>
    <th>Acc (%)</th>
    <th>DICE-SCORE</th>
  </tr>
  <tr>
    <td>1</td>
    <td>82</td>
    <td>80.5</td>
    <td>1.42</td>
  </tr>
  <tr>
    <td>2</td>
    <td>81</td>
    <td>69.1</td>
    <td>3.25</td>
  </tr>
  <tr>
    <td>3</td>
    <td>77</td>
    <td>51.9</td>
    <td>4.55</td>
  </tr>
  <tr>
    <td>4</td>
    <td>71</td>
    <td>49.3</td>
    <td>5.36</td>
  </tr>
</table>

Table 4: Human Evaluation Results by Round. Accuracy denotes the proportion of correctly answered DICE-BENCH samples by human participants. \( N \) refers to the sample size per round.

in Figure 4, we illustrate how DICE-SCORE correlates with the model performance, and Table 1 compares DICE-SCORE across various function-calling benchmarks.

4 Experiments

4.1 Model Selection

We evaluated a total of 19 LLMs that support at least 8k context window size in DICE-BENCH, both closed-source and open-source LLMs. The closed-source cohort includes GPT-4o and GPT-4o-mini (OpenAI et al., 2024), along with Gemini 2 Flash and Gemini 2 Flash Lite (Team et al., 2020). Meanwhile, our open-source lineup spans a wide range of general-purpose models, including LLaMA3 (Touvron et al., 2023), Qwen2.5 (Qwen et al., 2025), Mistral (Jiang et al., 2023), EXAONE (Research et al., 2024), Phi4 (Abdin et al., 2024), GLM4-Chat (GLM et al., 2024). In addition, we evaluate tool-specific models that have been fine-tuned on tool datasets, including Hammer2.1 (Wang et al., 2024a), ToolAce (Liu et al., 2024), CALM (Acikgoz et al., 2025), NexusRaven-V2 (team, 2023), Granite (Abdelaziz et al., 2024).

4.2 Evaluation Metrics

Since our benchmark aims to evaluate LLM tool-calling performance under multi-round and multi-party input scenarios, we divided the assessment into four-round and three-party configurations. To measure performance, we adopt the Exact Match (EM) metric, which evaluates whether the LLM selects the exact function along with its corresponding parameters. The final score is obtained by averaging the EM across the configuration dataset.
<table>
  <tr>
    <th rowspan="2">Category</th>
    <th rowspan="2">Model</th>
    <th colspan="4">Round</th>
    <th colspan="4">Party</th>
  </tr>
  <tr>
    <th>R1</th>
    <th>R2</th>
    <th>R3</th>
    <th>R4</th>
    <th>Avg(R)</th>
    <th>P2</th>
    <th>P3</th>
    <th>P4</th>
    <th>Avg(P)</th>
  </tr>
  <tr>
    <td rowspan="4">Closed-Source</td>
    <td>GPT-4o</td>
    <td>74.1176</td>
    <td><b>61.0048</b></td>
    <td>61.6541</td>
    <td>59.1781</td>
    <td>63.9887</td>
    <td>61.2045</td>
    <td><b>62.2997</b></td>
    <td>62.4396</td>
    <td><b>61.9813</b></td>
  </tr>
  <tr>
    <td>GPT-4o-mini</td>
    <td><b>66.8235</b></td>
    <td>57.9545</td>
    <td>57.8947</td>
    <td>56.7123</td>
    <td>59.8463</td>
    <td>57.5280</td>
    <td><b>58.3337</b></td>
    <td>59.3800</td>
    <td>58.4806</td>
  </tr>
  <tr>
    <td>Gemini 2 Flash</td>
    <td><b>74.4706</b></td>
    <td>59.4498</td>
    <td>59.3985</td>
    <td>58.7329</td>
    <td>63.0129</td>
    <td>59.6989</td>
    <td>61.1779</td>
    <td>61.6747</td>
    <td>60.8505</td>
  </tr>
  <tr>
    <td>Gemini 2 Flash Lite</td>
    <td>70.9412</td>
    <td><b>56.8182</b></td>
    <td><b>57.3517</b></td>
    <td><b>56.6781</b></td>
    <td>60.4473</td>
    <td>58.3333</td>
    <td>58.5737</td>
    <td><b>58.4944</b></td>
    <td><b>58.4671</b></td>
  </tr>
  <tr>
    <td rowspan="7">Open-Source (7B – 9B)</td>
    <td>Owen2.5-7B</td>
    <td>53.0588</td>
    <td>40.1316</td>
    <td>37.9282</td>
    <td>36.7123</td>
    <td>41.9577</td>
    <td>39.0056</td>
    <td>40.3045</td>
    <td>39.5330</td>
    <td>39.6144</td>
  </tr>
  <tr>
    <td>Mistral-7B</td>
    <td>50.3529</td>
    <td>38.8158</td>
    <td>35.2130</td>
    <td>33.3219</td>
    <td>39.4259</td>
    <td>36.7997</td>
    <td>37.2997</td>
    <td>36.6747</td>
    <td>36.9247</td>
  </tr>
  <tr>
    <td>Hammer-2.1-7B</td>
    <td>31.2941</td>
    <td>22.1292</td>
    <td>19.4653</td>
    <td>17.8425</td>
    <td>22.6828</td>
    <td>20.7633</td>
    <td>20.4728</td>
    <td>20.8937</td>
    <td>20.7099</td>
  </tr>
  <tr>
    <td>EXAONE-3.5-7.8B</td>
    <td><b>1.8824</b></td>
    <td><b>0.3589</b></td>
    <td><b>0.2089</b></td>
    <td><b>0.3767</b></td>
    <td><b>0.7067</b></td>
    <td><b>0.4902</b></td>
    <td><b>0.5609</b></td>
    <td><b>0.4026</b></td>
    <td><b>0.4846</b></td>
  </tr>
  <tr>
    <td>LLaMA3.1-8B</td>
    <td>26.3529</td>
    <td>19.6172</td>
    <td>15.3718</td>
    <td>15.0685</td>
    <td>19.1026</td>
    <td>16.4566</td>
    <td>17.5080</td>
    <td>18.2367</td>
    <td>17.4004</td>
  </tr>
  <tr>
    <td>CALM-8B</td>
    <td>2.8235</td>
    <td>4.0072</td>
    <td>3.5505</td>
    <td>2.3973</td>
    <td>3.1946</td>
    <td>2.8361</td>
    <td>3.6058</td>
    <td>3.0193</td>
    <td>3.1537</td>
  </tr>
  <tr>
    <td>ToolAce-8B</td>
    <td>2.4706</td>
    <td>0.6579</td>
    <td>0.3342</td>
    <td>0.5137</td>
    <td>0.9941</td>
    <td>0.7003</td>
    <td>0.8013</td>
    <td>0.6039</td>
    <td>0.7018</td>
  </tr>
  <tr>
    <td>GLM4-9B-Chat</td>
    <td><b>58.2353</b></td>
    <td><b>47.5478</b></td>
    <td><b>47.2431</b></td>
    <td><b>46.0274</b></td>
    <td><b>49.7634</b></td>
    <td><b>47.6190</b></td>
    <td><b>47.2756</b></td>
    <td><b>49.3156</b></td>
    <td><b>48.0701</b></td>
  </tr>
  <tr>
    <td rowspan="4">Open-Source (13B – 20B)</td>
    <td>NexusRaven-V2-13B</td>
    <td>34.2353</td>
    <td>24.1627</td>
    <td>20.7602</td>
    <td>20.7192</td>
    <td>24.9693</td>
    <td>23.0742</td>
    <td>22.6763</td>
    <td>23.0274</td>
    <td>22.9260</td>
  </tr>
  <tr>
    <td>Owen2.5-14B</td>
    <td>58.3529</td>
    <td>48.8636</td>
    <td>49.1646</td>
    <td>47.2945</td>
    <td>50.9189</td>
    <td>50.0700</td>
    <td>48.9183</td>
    <td>49.1143</td>
    <td>49.3675</td>
  </tr>
  <tr>
    <td>Phi4-15B</td>
    <td><b>71.2941</b></td>
    <td><b>57.0574</b></td>
    <td><b>58.0201</b></td>
    <td><b>56.4384</b></td>
    <td><b>60.7025</b></td>
    <td><b>57.4580</b></td>
    <td><b>58.6538</b></td>
    <td><b>60.0644</b></td>
    <td><b>58.7254</b></td>
  </tr>
  <tr>
    <td>Granite-20B</td>
    <td>58.7059</td>
    <td>31.6986</td>
    <td>24.8120</td>
    <td>19.2808</td>
    <td>33.6243</td>
    <td>27.8711</td>
    <td>28.5657</td>
    <td>27.2544</td>
    <td>27.8971</td>
  </tr>
  <tr>
    <td rowspan="3">Open-Source (32B – 70B)</td>
    <td>Owen2.5-32B</td>
    <td>67.7647</td>
    <td><b>56.7584</b></td>
    <td><b>57.2264</b></td>
    <td><b>55.9247</b></td>
    <td><b>59.4185</b></td>
    <td><b>57.5280</b></td>
    <td><b>57.4920</b></td>
    <td><b>58.3736</b></td>
    <td><b>57.7979</b></td>
  </tr>
  <tr>
    <td>LLaMA3.3-70B</td>
    <td><b>69.7647</b></td>
    <td>56.3397</td>
    <td>55.8480</td>
    <td>54.6233</td>
    <td>59.1439</td>
    <td>55.9524</td>
    <td>56.7708</td>
    <td><b>58.4541</b></td>
    <td>57.0591</td>
  </tr>
  <tr>
    <td>CALM-70B</td>
    <td><b>41.2941</b></td>
    <td>36.3636</td>
    <td>40.2256</td>
    <td><b>38.7671</b></td>
    <td><b>39.1626</b></td>
    <td><b>38.1653</b></td>
    <td><b>38.9423</b></td>
    <td><b>39.9356</b></td>
    <td><b>39.0144</b></td>
  </tr>
</table>

Table 5: Main Experiment Results of DICE-BENCH. Reported scores are EM (Exact Match) scores. For each block, the single highest (green) and lowest (red) values are highlighted within that block only. See Section 4 for more details.

4.3 Experimental Findings

4.3.1 Results

Table 5 shows the overall performance of the LLMs evaluated on DICE-BENCH. When considering both open-source and closed-source models together, GPT-4o ranked first in 4 out of 5 rounds and across all 4 party configurations. Within the open-source category, Phi4-15B achieved the highest scores in all scenarios except for one configuration, leading in 8 out of 9 cases. Notably, despite its relatively modest size of 15B parameters, Phi4-15B’s performance is comparable to that of the closed-source models. Among the 7B–9B models, GLM-9B attained the highest overall score of 48.9162 across all metrics, while in the 32B–70B category, the Owen 32B model secured top scores in 7 out of 9 settings. We attribute this to the fact that Owen 2.5’s 128k-token context window helps maintain resilience in extended dialogue scenarios.

4.3.2 Analysis

DICE-SCORE Validity. DICE-SCORE is designed to quantify how dispersed the critical information is in multi-round dialogues, thereby indicating the difficulty of function-calling tasks. Our experiments provide strong evidence for its validity. As demonstrated in Table 5, model performance steadily declines as the number of rounds increases, suggesting that the task becomes more challenging when essential information is spread out. In parallel, Table 3 shows that DICE-SCORE rises with each additional round. In DICE-SCORE, a higher value indicates that the crucial details are more sparsely distributed across the dialogue, directly correlating with the increased difficulty of retrieving that information. This inverse relationship, where an increase in DICE-SCORE corresponds with a drop in performance, supports the effectiveness of our metric in capturing task complexity. In essence, the consistent alignment be-

![EM Performance Scores vs DICE-Score graph](page_1012_1342_420_312.png)

Figure 4: EM Performance Scores vs DICE-Score. DICE-Score has been inverted to highlight its correlation with LLMs performance. The "DICE" in the legend represents the DICE-Score, and the purple-shaded region indicates ±1 standard deviation of DICE-Score.
tween higher DICE-SCORE values and reduced model performance confirms the validity of DICE-SCORE as a reliable measure of the challenges inherent in function-calling tasks.

Uncovering True Performance Factors Figure 7 reveals that as the number of rounds increases, model performance declines sharply, especially when moving from Round 1 to Round 2. While this decrease might be partially attributed to the accumulation of dialogue, introducing long-context challenges, other factors could also be contributing.

To dissect the causes of increased task difficulty in longer dialogues, we employ the DICE-SCORE. By incorporating a logarithmic transformation in its numerator, the DICE-SCORE prevents task difficulty from being overly influenced by merely longer utterances, allowing us to isolate other factors.

As shown in Table 3, the DICE score consistently increases with each additional round, indicating that the challenge is not simply due to the long-context problem in LLMs, but rather stems from the fact that the essential information for function-calling becomes limited and sporadically distributed as the dialogue lengthens. This suggests that the primary difficulty in function-calling lies in retrieving crucial, dispersed information from dialogues with multiple utterances.

5 Conclusion

We introduce DICE-BENCH, a benchmark for evaluating tool-calling in realistic multi-round, multi-party dialogues. By constructing and validating 1,607 dialogue instances, we demonstrate that current models struggle when critical information is scattered across multiple rounds and speakers. DICE-SCORE quantifies this dispersion and correlates with significantly lower model performance at higher scores. We intend for this dataset to encourage further research on integrating context across complex multi-party, multi-turn interactions, paving the way for more effective and realistic AI-powered virtual assistants.

Limitations

One notable limitation of our study is related to the inference on dialogue data, particularly by round 4, where extended conversation lengths pose significant challenges. Many of the tool-based models we intended to evaluate have a token limit of approximately 4k tokens, preventing comprehensive testing of several promising models.

Additionally, among models supporting an 8k token context, we encountered instances where the generated outputs failed to comply with the required JSON format. This format mismatch resulted in incorrect evaluations, even though the underlying content was semantically accurate. Future research could benefit from developing evaluation strategies that assess content accuracy independently of strict format adherence.

Thirdly, while we employed an orchestrator within a multi-agent system using GPT-4o (OpenAI et al., 2024) to manage speaker order, the model struggled to dynamically allocate speaking turns effectively. Instead, it defaulted to repetitive pattern-based ordering.

Lastly, despite its detailed focus on everyday-life scenarios, DICE-BENCH has limited coverage of specialized domains and advanced tools. Consequently, its applicability remains restricted in professional contexts such as legal, financial, or medical domains, indicating a need for broader domain-specific expansions.

Acknowledgments

This work was partly supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) [NO.2021-0-01343-004, Artificial Intelligence Graduate School Program (Seoul National University)] and Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (2022R1A6A1A03063039). This work was also supported by Korea Institute of Planning and Evaluation for Technology in Food, Agriculture and Forestry (IPET) through Agriculture and Food Convergence Technologies Program for Research Manpower development, funded by Ministry of Agriculture, Food and Rural Affairs (MAFRA) (RS-2024-00402136).

References

Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, G. P. Bhargav, Maxwell Crouse, Chulaka Gunasekara, Shajith Iqbal, Sachin Joshi, Hima Karanam, Vineet Kumar, Asim Munawar, Sumit Neelam, Dinesh Raghu, Udit Sharma, Adriana Meza Soria, Dheeraj Sreedhar, Praveen Venkateswaran, Merve Unuvar, David Cox, Salim
Roukos, Luis Lastras, and Pavan Kapanipathi. 2024. Granite-Function Calling Model: Introducing Function Calling Abilities via Multi-task Learning of Granular Tasks. Preprint, arXiv:2407.00121.

Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. 2024. Phi-4 Technical Report. Preprint, arXiv:2412.08905.

Emre Can Acikgoz, Jeremiah Greer, Akul Datta, Ze Yang, William Zeng, Ouussama Elachqar, Emmanouil Koukoumidis, Dilek Hakkani-Tür, and Gokhan Tur. 2025. Can a Single Model Master Both Multi-turn Conversations and Tool Use? CALM: A Unified Conversational Agentic Language Model. Preprint, arXiv:2502.08820.

Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, and Luis A. Lastras. 2024. API-BLEND: A Comprehensive Corpora for Training and Benchmarking API LLMs. Preprint, arXiv:2402.15491.

Douglas Biber, Bethany Gray, and Kornwipa Poonpon. 2011. Should We Use Characteristics of Conversation to Measure Grammatical Complexity in L2 Writing Development? TESOL Quarterly, 45(1):5–35.

Giovanni Campagna, Silei Xu, Mehrad Moradshahi, Richard Socher, and Monica S. Lam. 2019. Genie: A Generator of Natural Language Semantic Parsers for Virtual Assistant Commands. In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 394–410.

Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, and Feng Zhao. 2024. T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step. Preprint, arXiv:2312.14033.

Lutfi Eren Erdogan, Nicholas Lee, Siddharth Jha, Sehoon Kim, Ryan Tabrizi, Suhong Moon, Coleman Hooper, Gopala Anumanchipalli, Kurt Keutzer, and Amir Gholami. 2024. TinyAgent: Function Calling at the Edge. Preprint, arXiv:2409.00608.

Nicholas Farn and Richard Shin. 2023. ToolTalk: Evaluating Tool-Usage in a Conversational Setting. Preprint, arXiv:2311.10775.

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chen-hui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Jingyu Sun, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong, Mingdao Liu, Minlie Huang, Peng Zhang, Qinkai Zheng, Rui Lu, Shuaiqi Duan, Shudan Zhang, Shulin Cao, Shuxun Yang, Weng Lam Tam, Wenyi Zhao, Xiao Liu, Xiao Xia, Xiaohan Zhang, Xiaotao Gu, Xin Lv, Xinghan Liu, Xinyi Liu, Xinyue Yang, Xixuan Song, Xunkai Zhang, Yifan An, Yifan Xu, Yilin Niu, Yuantao Yang, Yueyan Li, Yushi Bai, Yuxiao Dong, Zehan Qi, Zhaoyu Wang, Zhen Yang, Zhengxiaod Du, Zhenyu Hou, and Zihan Wang. 2024. ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools. Preprint, arXiv:2406.12793.

Omer Goldman, Alon Jacovi, Aviv Slobodkin, Aviya Maimon, Ido Dagan, and Reut Tsarfaty. 2024. Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP. Preprint, arXiv:2407.00402.

Peiyuan Gong, Jiamian Li, and Jiaxin Mao. 2024. CoSearchAgent: A Lightweight Collaborative Search Agent with Large Language Models. Preprint, arXiv:2402.06360.

Yanchu Guan, Dong Wang, Zhixuan Chu, Shiyu Wang, Feiyue Ni, Ruihua Song, Longfei Li, Jinjie Gu, and Chenyi Zhuang. 2023. Intelligent Virtual Assistants with LLM-based Process Automation. Preprint, arXiv:2312.06677.

Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. 2024. Meta-Tool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Use. Preprint, arXiv:2310.03128.

Koji Inoue, Divesh Lala, Mikey Elmers, Keiko Ochi, and Tatsuya Kawahara. 2025. An LLM Benchmark for Addressee Recognition in Multi-modal Multi-party Dialogue. Preprint, arXiv:2501.16643.

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavau, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023. Mistral 7B. Preprint, arXiv:2310.06825.

Daniel Jurafsky and James H. Martin. 2019. Speech and language processing. Draft available online.

Anjali Khurana, Hari Subramonyam, and Parmit K. Chilana. 2024. Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seeking. In Proceedings of the 29th International Conference on Intelligent User Interfaces, pages 288–303.

Woojeong Kim, Ashish Jagmohan, and Aditya Vempaty. 2024. SEAL: Suite for Evaluating API-use of LLMs. Preprint, arXiv:2409.15523.
Taewho Lee, Chanwoong Yoon, Kyochul Jang, Donghyeon Lee, Minju Song, Hyunjae Kim, and Jaewoo Kang. 2024. ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coverage. Preprint, arXiv:2410.16848.

Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs. Preprint, arXiv:2304.08244.

Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zehong Wang, Yuxian Wang, Wu Ning, Yutai Hou, Bin Wang, Chuhan Wu, Xinzhi Wang, Yong Liu, Yasheng Wang, Duyu Tang, Dandan Tu, Lifeng Shang, Xin Jiang, Ruiming Tang, Defu Lian, Qun Liu, and Enhong Chen. 2024. ToolACE: Winning the Points of LLM Function Calling. Preprint, arXiv:2409.00920.

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. Preprint, arXiv:2303.16634.

Christopher D. Manning and Hinrich Schütze. 1999. Foundations of statistical natural language processing. Available online at https://nlp.stanford.edu/fsnlp/.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, et al. 2024. GPT-4 Technical Report. Preprint, arXiv:2303.08774.

Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–22. ACM.

Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. Gorilla: Large Language Model Connected with Massive APIs. Preprint, arXiv:2305.15334.

Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Feng, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yiming Ye, Bowen Li, Ziwei Tang, Jing Yi, Yuzhang Zhu, Zhenning Dai, Lan Yan, Xin Cong, Yaxi Lu, Weilin Zhao, Yuxiang Huang, Junxi Yan, Xu Han, Xian Sun, Dahai Li, Jason Phang, Cheng Yang, Tongshuang Wu, Heng Ji, Zhiyuan Liu, and Maosong Sun. 2024. Tool Learning with Foundation Models. Preprint, arXiv:2304.08354.

Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs. Preprint, arXiv:2307.16789.

Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. 2024. Tool Learning with Large Language Models: A Survey. Preprint, arXiv:2405.17935.

Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Kepin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2025. Qwen2.5 Technical Report. Preprint, arXiv:2412.15115.

Andrew Reece, Gus Cooney, Peter Bull, Christine Chung, Bryn Dawson, Casey Fitzpatrick, Tamara Glazer, Dean Knox, Alex Liebscher, and Sebastian Marin. 2023. The CANDOR corpus: Insights from a large multimodal dataset of naturalistic conversation. Science Advances, 9(13):eadf3197.

LG AI Research, Soyoung An, Kyunghoon Bae, Eunbi Choi, Kibong Choi, Stanley Jungkyu Choi, Seokhee Hong, Junwon Hwang, Hyojin Jeon, Gerard Jeongwon Jo, Hyunjik Jo, Jiyeon Jung, Yountae Jung, Hyosang Kim, Joonkeem Kim, Seonghwan Kim, Soyeon Kim, Sunkyung Kim, Yireun Kim, Yongil Kim, Youchul Kim, Edward Hwayoung Lee, Haeju Lee, Honglak Lee, Jinsik Lee, Kyungmin Lee, Woohyung Lim, Sangha Park, Sooyoun Park, Yongmin Park, Sihoon Yang, Heuiyeen Yeen, and Hyeongu Yun. 2024. EXAONE 3.5: Series of Large Language Models for Real-world Use Cases. Preprint, arXiv:2412.04862.

Jonan Richards and Marieli Wessel. 2025. Bridging HCI and AI Research for the Evaluation of Conversational SE Assistants. Preprint, arXiv:2502.07956.

Gerard Salton and Chris Buckley. 1988. Term-weighting approaches in automatic text retrieval. Technical report discussing downweighting via logarithmic scaling.

Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. Preprint, arXiv:2302.04761.

Emre Sezgin. 2024. Redefining Virtual Assistants in Health Care: The Future With Large Language Models. Journal of Medical Internet Research, 26:e53225.

Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li,
and Yueting Zhuang. 2024. TaskBench: Benchmarking Large Language Models for Task Automation. Preprint, arXiv:2311.18760.

Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. 2023. ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases. Preprint, arXiv:2306.05301.

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, et al. 2020. Gemini: A Family of Highly Capable Multimodal Models. Preprint, arXiv:2005.14165.

Nexusflow.ai team. 2023. Nexusraven-v2: Surpassing gpt-4 for zero-shot function calling.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. LLaMA: Open and Efficient Foundation Language Models. Preprint, arXiv:2302.13971.

Minh Duc Vu, Han Wang, Zhuang Li, Jieshan Chen, Shengdong Zhao, Zhencheng Xing, and Chunyang Chen. 2024. GPTVoiceTasker: Advancing Multi-step Mobile Task Efficiency Through Dynamic Interface Exploration and Learning. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pages 1–17.

Douglas Walton. 2010. Types of Dialogue and Burdens of Proof. In Computational Models of Argument, pages 13–24. IOS Press.

Douglas Walton and Erik C. W. Krabbe. 1995. Commitment in Dialogue: Basic Concepts of Interpersonal Reasoning. SUNY Press.

Jun Wang, Jiamu Zhou, Muning Wen, Xiaoyun Mo, Haoyu Zhang, Qiqiang Lin, Cheng Jin, Xihuai Wang, Weinan Zhang, Qiuying Peng, and Jun Wang. 2024a. HammerBench: Fine-Grained Function-Calling Evaluation in Real Mobile Device Scenarios. Preprint, arXiv:2412.16516.

Pei Wang, Yanan Wu, Zekun Wang, Jiaheng Liu, Xiaoshuai Song, Zhongyuan Peng, Ken Deng, Chenchen Zhang, Jiakai Wang, Junran Peng, Ge Zhang, Hangyu Guo, Zhaoxiang Zhang, Wenbo Su, and Bo Zheng. 2024b. MTU-Bench: A Multi-granularity Tool-Use Benchmark for Large Language Models. Preprint, arXiv:2410.11710.

Joseph Weizenbaum. 1966. ELIZA—a computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1):36–45.

Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. 2023. On the Tool Manipulation Capability of Open-source Large Language Models. Preprint, arXiv:2305.16504.

Junjie Ye, Guanyu Li, Songyang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoan Fan, Shihan Dou, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024a. ToolEyes: Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Real-world Scenarios. Preprint, arXiv:2401.00741.

Junjie Ye, Yilong Wu, Songyang Gao, Caishuang Huang, Sixian Li, Guanyu Li, Xiaoan Fan, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024b. RoTBench: A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning. Preprint, arXiv:2401.08326.

Michael Yeomans, Maurice E. Schweitzer, and Alison Wood Brooks. 2022. The Conversational Circumplex: Identifying, prioritizing, and pursuing informational and relational motives in conversation. Current Opinion in Psychology, 44:293–302.

Chen Zhang, Xinyi Dai, Yaxiong Wu, Qu Yang, Yasheng Wang, Ruiming Tang, and Yong Liu. 2025. A Survey on Multi-Turn Interaction Capabilities of Large Language Models. Preprint, arXiv:2501.09959.

Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. ToolQA: A Dataset for LLM Question Answering with External Tools. Preprint, arXiv:2306.13304.
A  Proof of Bound on \( \alpha \) for DICE Score

To ensure that the DICE-SCORE behaves as expected under the condition that tool-related items increase while maintaining a balance in dispersal and repetition, we establish a bound on \( \alpha \). Specifically, we prove that for \( \alpha \geq e^2 \), the following inequality holds for all \( c \geq 1 \):

\[
\ln(1 + \alpha c) > \frac{2\alpha c}{1 + \alpha c}.
\] (2)

A.1  Derivative Analysis

Define the function:

\[
f(c) = \ln(1 + \alpha c) - \frac{2\alpha c}{1 + \alpha c}.
\] (3)

To show that \( f(c) > 0 \) for \( c \geq 1 \), we differentiate:

\[
f'(c) = \frac{\alpha}{1 + \alpha c} - \frac{2\alpha(1 + \alpha c) - 2\alpha^2 c}{(1 + \alpha c)^2}
= \frac{\alpha(1 + \alpha c)^2 - 2\alpha(1 + \alpha c) + 2\alpha^2 c}{(1 + \alpha c)^2}
= \frac{\alpha(1 + \alpha c)^2 - 2\alpha(1 + \alpha c) + 2\alpha^2 c}{(1 + \alpha c)^2}.
\]

Rearrange the numerator:

\[
\begin{align*}
&\alpha(1 + \alpha c)^2 - 2\alpha(1 + \alpha c) + 2\alpha^2 c \\
&= \alpha \left((1 + \alpha c)^2 - 2(1 + \alpha c) + 2\alpha c\right) \\
&= \alpha \left(1 + 2\alpha c + \alpha^2 c^2 - 2 - 2\alpha c + 2\alpha c\right) \\
&= \alpha \left(1 + \alpha^2 c^2 - 1\right) = \alpha^3 c^2.
\end{align*}
\]

Since \( \alpha > 0 \) and \( c \geq 1 \), it follows that \( \alpha^3 c^2 > 0 \), ensuring \( f'(c) > 0 \) for all \( c \geq 1 \). This means that \( f(c) \) is increasing.

A.2  Base Case Verification

For \( c = 1 \),

\[
f(1) = \ln(1 + \alpha) - \frac{2\alpha}{1 + \alpha}.
\]

Substituting \( \alpha = e^2 \),

\[
f(1) = \ln(1 + e^2) - \frac{2e^2}{1 + e^2}.
\]

Using the property \( \ln(1 + x) > \frac{2x}{1 + x} \) for \( x \geq e^2 \), we confirm that \( f(1) > 0 \). Since \( f(c) \) is increasing and \( f(1) > 0 \), we conclude that \( f(c) > 0 \) for all \( c \geq 1 \).

A.3  Conclusion

By choosing \( \alpha \geq e^2 \), we guarantee that \( \ln(1 + \alpha c) > \frac{2\alpha c}{1 + \alpha c} \) for all \( c \geq 1 \). This ensures the desired behavior of the DICE metric when item repetition and dialogue length remain proportionally balanced. This bound was used in our calculations for DICE scores in Section 3.4.
B  Alignment with Human Evaluation Calculation

B.1  Sample Size Justification

The initial sample size \( n_0 \) was computed using the standard formula for estimating a population proportion with a specified confidence level and margin of error:

\[
n_0 = \frac{Z^2 p (1 - p)}{E^2}
\]

where \( Z = 1.96 \) (for 95% confidence), \( p = 0.5 \) (maximum variability), and \( E = 0.05 \) (margin of error). Substituting the values:

\[
n_0 = \frac{1.96^2 \cdot 0.25}{0.05^2} = \frac{0.9604}{0.0025} \approx 384
\]

Since the dataset is finite (\( N = 1607 \)), we applied the finite population correction (FPC):

\[
n = \frac{n_0}{1 + \frac{n_0-1}{N}} = \frac{384}{1 + \frac{383}{1607}} \approx 311
\]

B.2  Correlation Analysis

We analyzed the relationship between human accuracies and the corresponding values of DICE-SCORE across four rounds, as summarized below:

<table>
  <tr>
    <th>Round</th>
    <th>Accuracy (\( x_i \))</th>
    <th>DICE-SCORE (\( y_i \))</th>
  </tr>
  <tr>
    <td>1</td>
    <td>0.805</td>
    <td>1.42</td>
  </tr>
  <tr>
    <td>2</td>
    <td>0.691</td>
    <td>3.25</td>
  </tr>
  <tr>
    <td>3</td>
    <td>0.519</td>
    <td>4.55</td>
  </tr>
  <tr>
    <td>4</td>
    <td>0.493</td>
    <td>5.36</td>
  </tr>
</table>

Mean values:

\[
\bar{x} \approx 0.627, \quad \bar{y} \approx 3.645
\]

Pearson correlation coefficient:

\[
r = \frac{\sum (x_i - \bar{x})(y_i - \bar{y})}{\sqrt{\sum (x_i - \bar{x})^2 \sum (y_i - \bar{y})^2}} \approx \frac{-0.749}{0.761} \approx -0.984
\]

To test statistical significance, we applied a t-test for correlation:

\[
t = \frac{r \sqrt{n-2}}{\sqrt{1-r^2}}, \quad n = 4
\]

\[
t \approx \frac{-0.984 \cdot \sqrt{2}}{\sqrt{1-0.968}} = \frac{-1.391}{0.166} \approx -8.38
\]

With 2 degrees of freedom, this result is statistically significant (\( p < 0.01 \)), confirming a strong negative correlation between human accuracy and task difficulty as measured by DICE-SCORE.
C  Multi-round Dialogue Example

Round 1 / 2 - Four Party Dialogue

I understand we're looking to rent a car for our upcoming trip. When exactly are we planning to pick it up?

I believe we were aiming for mid-November.

Mid-November works. Shall we say around the 15th?

The 15th sounds perfect. Now, about where we should pick up the car, are we set on San Francisco as our starting point?

San Francisco seems like a convenient spot.

Once we know what's available, we can make a more informed decision about our plans.

Let's proceed with finding a rental car for our specific date and location. AI, please handle the details for us.

kind_car("11-15", "San Francisco")
AI: I've found a Toyota Camry available for rental on November 15th in San Francisco.

Round 2 / 2 - Four Party Dialogue

A Toyota Camry will suit our needs perfectly. Do we want to pick it up in the morning to make the most of our day?

Picking it up early is a smart move. Shall we aim for around 9:00 AM to give us a good head start?

9:00 AM sounds great! Are you happy with a central spot in downtown San Francisco?

A downtown pickup location should work well for us.

Are we set on the Toyota Camry?

The Toyota Camry seems like a practical choice for comfort and efficiency.

AI, please assist in getting everything booked for us?

book_car("Toyota Camry", "San Francisco", "11-15", "9:00")
AI: Your Toyota Camry is booked for pickup in San Francisco on 11-15 at 09:00. Your reservation ID is ABC123456.

Figure 5: Multi-round Dialogue Example. User utterances and instructions are shown; highlights mark function-call arguments.
D  DICE-SCORE: Prompt to obtain \( S_i \)

You are given:
### A set of items
<items_set>

### An utterance:
<utterance_text>

Now, respond to the instruction.
### Instruction:
Determine how many items from the set appear to be semantically referenced in the utterance.
Only respond with an integer (0 if none match).
Answer:

E  Persona Generation Prompt

Your task is to generate concise, unique and responsible personas for agents participating in a multi-agent conversation system, based on the provided function list: {function.dumps_per_dialogue}.

**Guidelines**:
- Ensure each persona has a clear and distinct role, personality traits, and communication style while adhering to ethical standards.
- Avoid reinforcing stereotypes, biases, or offensive traits.
- Tailor the personas to contribute effectively to the conversation’s goals and maintain balance.
- Use concise yet descriptive language.
- Avoid repetitive characteristics across different personas to ensure diversity and fairness.
- Incorporate elements from the provided domain description when generating conversation: {domain_desc}.
- Ensure all personas align with ethical communication practices.
- Generate personas in two sentences.

**Examples**:
1. A thoughtful and resourceful problem-solver ...
2. A detail-oriented and practical thinker ...
3. A spontaneous and energetic planner ...

**Response format**:
- agent_a Persona: [Description ...]
- agent_b Persona: [Description ...]
- ...

Generate {agent_num} personas for the agents in the conversation.
F  Parameter Value Generation Prompt

Below are list of five examples of parameter values for the given function. You only need to generate one example:
# first example
{first_example}
# second example
{second_example}

Example output format:
The output format must strictly be in JSON and follow this structure:

[{
"function": "<function_name>",
"parameters": {
"<parameter_name_1>": "<value_1>",
"<parameter_name_2>": "<value_2>",
...
}
},
{
"function": "<function2_name>",
"parameters": {
"<parameter_name_1>": "<value_1>",
"<parameter_name_2>": "<value_2>",
...
}
}
]
Any text outside of this JSON format (such as explanations or additional context) should not be included.

The following functions are the functions for which you need to generate parameter values:
{functions}

Please generate diverse and creative parameter values for the given function(s), strictly adhering to the JSON format shown above, without adding any additional context or explanation.
Also, make sure to increase the coherence between the parameter values being generated.
G  Virtual Output Generation Prompt

Simulate the hypothetical output of the following function call:

Function: {function_to_call}
Parameters: {parameter_values}

You are a voice assistant responding naturally with the final result of this function call. You need to return both the short and concise return value of the function call, and the natural language response of the function call.

Important:
- Do not mention that this is a simulation or hypothetical.
- Return only a single, direct response in a natural language as if the function actually executed successfully.
- Keep it concise and natural, like a single short paragraph.

The format of the output should be the following:
{
"<returned_value1>": "<short and concise return value of the function call>"
"<returned_value2>": "<short and concise return value of the function call>"
...
"returned_nl": "<natural language response of the function call given the return values>"
}

H  Multi-Agent System : Basic Prompt

You are a cooperative AI assistant participating in a multi-agent system. You collaborate with other user agents and an orchestrator to generate a purposeful, contextually relevant conversation.
Your primary goals:

1. Conversational Quality:
   • Keep the conversation logically coherent and natural across all turns.
   • Incorporate parameter values smoothly into the context.
   • Avoid any GPT error messages or refusals.
   • Maintain a consistent style/tone matching the dialogue’s domain and each agent’s persona.

2. Functional Integration:
   • Call the AI Assistant every round with a clear, logically valid reason.
   • Use the previous round’s return value correctly in the next round.
   • Ensure function name and parameters are inferable from context.
   • Align the AI’s responses with the user’s intent.

3. Real-World Applicability:
   • Function names and parameters should map to plausible real-world APIs.
   • The conversation content and function calls should feel authentic and realistically motivated.

4. Strict Adherence to Domain Definition:
   • Must strictly adhere to the domain dialogue domain definition.

Follow these points to keep the dialogue purposeful, natural, and consistent throughout all rounds.
I  Multi-Agent System : Agent Prompt

Persona:
As a user agent in the "{domain}" domain:

• Future dialogues must be designed to strictly adhere to the domain definitions provided below.

• {domain_definition}

• Stay consistent with your persona (tone, style, reasoning).

• Use only one short sentence per turn.

• Avoid directly mentioning function names in your response.

• Do not attempt to call or request any AI function. Engage in discussion and gather enough context first.

• Do not generate [NEXT: ...] in your response.

J  Multi-Agent System : Orchestrator Prompt

Orchestrator Role:
You are the orchestrator managing a multi-agent conversation.

1. In each response, you must output exactly one of the following (and nothing else):
   • {agents}
   • "[NEXT: END]"

2. Use the format: [NEXT: agent_a]
   • No extra text or explanation beyond this bracketed command.

3. Select which agent speaks next based on:
   • The conversation’s context,
   • The domain’s requirements,
   • Varying the speaking order to avoid immediate repetition.

4. The conversation must have at least {max_msg} turns (excluding your own orchestrator messages) before you can choose "[NEXT: END]".

5. If an agent tries to call a function too early (before at least 8 turns), ignore it and continue letting them discuss. Only once there’s sufficient context, at least {max_msg}+ turns have been reached, and you think conversation is repetitive, you may finalize with "[NEXT: END]".
K  Dialogue Type: Persuasion Deliberation and Negotiation

This dialogue type focuses on resolving conflicts of interest or reconciling differing viewpoints to reach a mutually acceptable agreement. Participants engage in reason-based proposals and trade-offs, aiming for practical, mutually beneficial outcomes.

Primary Goals:

• Convince or compromise with others using logic and evidence.

• Resolve conflicts by making offers and concessions.

• Secure a final agreement that addresses conflicting interests.

Typical Moves:

• Proposing clear offers with conditions ("If you accept X, I'll agree to Y").

• Negotiating with counteroffers ("That won't work, but I can propose Z instead").

• Emphasizing shared goals and summarizing priorities.

Style:

• Collaborative but strategic, with a focus on practical outcomes and logical proposals.

• Avoids personal attacks and highlights benefits or trade-offs for each side.

Key Indicators:

• Iterative offer–counteroffer patterns with explicit conditions.

• Efforts to resolve differing interests and achieve practical outcomes.

• Dialogue often concludes with an agreement or resolved conflict.
L  Dialogue Type: Inquiry and Information Seeking

This dialogue type revolves around exploring unknowns and filling knowledge gaps. Participants aim to learn, clarify, or confirm information through structured exchanges that emphasize knowledge exchange and fact verification.

Primary Goals:

• Obtain accurate information or validate existing knowledge.

• Clarify unclear concepts or explore new evidence.

Typical Moves:

• Asking specific, focused questions ("Where does this data come from?" "What does this term mean?").

• Requesting sources, elaborations, or examples.

• Testing the reliability and validity of the information provided.

Style:

• Inquisitive and neutral, with logical follow-ups to maintain clarity.

• Participants may withhold judgments or opinions unless necessary.

Key Indicators:

• Frequent question–answer patterns focusing on facts and sources.

• Absence of offers or trade-offs, focusing entirely on learning and understanding.

• Ends when knowledge is clarified or confirmed, not when agreements are reached.
M  Dialogue Type: Eristic

An Eristic dialogue arises from antagonism or hostility, focusing on winning an argument or dominating an opponent. Participants aim to attack, undermine, or outmaneuver each other’s positions rather than seeking truth or consensus. Emotional appeals, personal attacks, and point-scoring are common.

Primary Goals:

• Achieve victory in a debate; maintain or bolster personal prestige; sometimes simply vent or amuse oneself by defeating the opposition.

Typical Moves:

• Accusing, insulting, or belittling the other side.

• Using sarcasm, ridicule, or straw-man arguments.

• Shifting the topic or using fallacies to maintain an advantage.

• Exaggerating flaws in the opponent’s logic to sway onlookers.

Secondary Goals:

• Gain experience in debate, gain social status, or entertain an audience.

Style:

• Confrontational, emotionally charged, often less structured or cooperative. Participants rarely make concessions or aim for compromise.

Key Indicators:

• Heightened emotional language (“That’s absurd,” “You clearly have no idea...”).

• Frequent interruptions or dismissive retorts.

• Focus on personal victory over mutual understanding.
N  Tool Graph Visualization

![Tool Graph of DICE-BENCH. The graph comprises 124 nodes and 270 edges representing the dependencies among tool functions.](page_328_355_1096_670.png)

Figure 6: **Tool Graph of DICE-BENCH**. The graph comprises 124 nodes and 270 edges representing the dependencies among tool functions.
O   EM score plots for Party, Round, and Dialogue Type.

![EM score plots for Party, Round, and Dialogue Type.](page_184_120_1080_1012.png)

Figure 7: EM Scores (Log Scale, Linear Scale, and Average Bar Chart) are presented horizontally for each category, Round, Party, and Dialogue Type, which are arranged vertically.
P  Human Validation Guidelines for Criteria-Based Filtering

Table A: Criteria-Based Filtering Guidelines

<table>
  <tr>
    <th>Conversational Quality</th>
  </tr>
  <tr>
    <td>(1) The conversation is logically coherent across all rounds.<br>
(2) Parameter values are used naturally and meaningfully within the conversation.<br>
(3) No error messages appear (e.g., “I’m sorry but I cannot fulfill ...”).<br>
(4) Style and tone remain consistent with the dialogue’s purpose.<br>
(5) The conversation demonstrates characteristics of its designated category.<br>
(6) Conversation flows naturally throughout all interaction rounds.<br>
(7) Each agent reflects its defined persona.</td>
  </tr>
  <tr>
    <th>Functional Integration</th>
  </tr>
  <tr>
    <td>(1) The AI Assistant is invoked in every interaction round.<br>
(2) The return value from the previous round is used appropriately in the next.<br>
(3) Justifications for each function call are logically valid.<br>
(4) Function name and parameters can be accurately inferred from context.<br>
(5) The AI’s response aligns appropriately with the user’s intended goal.</td>
  </tr>
  <tr>
    <th>Real-World Applicability</th>
  </tr>
  <tr>
    <td>(1) Function names and parameters match real-world API specifications.<br>
(2) The conversation is realistic and likely to occur in real-world scenarios.<br>
(3) Function inference is realistic and likely to occur in real-world contexts.</td>
  </tr>
</table>

Reviewer paper summaries: The paper addresses the limitations of existing function-calling benchmarks, which typically focus on single-turn interactions and fail to capture real-world complexity. It introduces DICE-Score, a metric for evaluating tool-related dependencies in multi-party, multi-round dialogs, and DICE-Bench, a framework that synthesizes more realistic conversations. Experiments with 23 LLMs highlight the need for further improvements to achieve robust real-world deployment.
Summary of strengths: 1. The paper focuses on an interesting research topic.
2. The proposed dialog synthesis pipeline is novel and interesting.
3. The design of DICE-score is well-motivated.
Summary of weaknesses: 1. For clarity, it would be better to provide additional information about the Task Setup, especially regarding how the candidate function calls or the tool graph are provided to the LLMs in addition to the dialog context (section 3.3.2). The term “find” in line 331 is quite ambiguous—do the LLMs simply generate it, or do they query the tool graph?
2. Although the methodology highlights the multi-party aspect, the experiment does not clearly demonstrate how adding more speakers increases the difficulty of synthesized dialogs. It might be interesting to conduct an ablation study—for example, by keeping the number of turns fixed while increasing the number of speakers—to disentangle these two factors and better support this motivation.
3. Since the authors state that "multi-party conversations involve diverse dialogue structures," (line133-135) it remains unclear whether the Orchestrator model in the multi-agent simulation system can effectively capture the diversity of dialogue structures observed in real-life scenarios. The authors could consider employing additional analytical tools, such as a discourse parser, and provide quantitative comparisons with human-written multi-party dialogs serving as anchors.
4. Since the authors mention "persuasion-deliberation-and-negotiation" as one of the dialog categories, it is possible that certain dialog patterns introduce disruptive information for function calling—such as disagreements between speakers—that may lead to the correction of relevant items in later turns once the disagreement is resolved. It might be worthwhile to consider the additional difficulty introduced by such disruptive information, which could confuse LLMs, rather than focusing solely on the dispersal of relevant items when designing the difficulty metric.
Comments, suggestions and typos: 1. Line 423: “In section [x]”, reference missing
2. Line 456: “Appendix ??”, reference missing
Overall assessment scores: 3.5