VoiceAgentBench: Are Voice Assistants ready for agentic tasks?

## Abstract

Abstract Large-scale Speech Language Models (SpeechLMs) have enabled voice assistants capable of understanding natural spoken queries and performing complex tasks. However, existing speech benchmarks primarily focus on isolated capabilities such as transcription, or question-answering, and do not systematically evaluate agentic scenarios encompassing multilingual and cultural understanding, as well as adversarial robustness. To address this, we introduce VoiceAgentBench , a comprehensive benchmark designed to evaluate SpeechLMs in realistic spoken agentic settings. It comprises over 5,500 synthetic spoken queries, including dialogues grounded in Indian context, covering single-tool invocations, multi-tool workflows, multi-turn interactions, and safety evaluations. The benchmark supports English, Hindi, and 5 other Indian languages, reflecting real-world linguistic and cultural diversity. We simulate speaker variability using a novel sampling algorithm that selects audios for TTS voice conversion based on its speaker embeddings, maximizing acoustic and speaker diversity. Our evaluation measures tool selection accuracy, structural consistency, and the correctness of tool invocations, including adversarial robustness. Our experiments reveal significant gaps in contextual tool orchestration tasks, Indic generalization, and adversarial robustness, exposing critical limitations of current SpeechLMs.

## 1 Introduction

Advancements in Large Language Models (LLMs) (touvron2023llama; grattafiori2024llama; abdin2025phi; guo2025deepseek; yang2025qwen3) have enabled the development of intelligent agents capable of reasoning (wei2022chain), planning (yao2023react), and executing complex, multi-step tasks through interaction with external tools (qin2024toolllm; patil2024gorilla) and databases (gao2024retrievalaugmentedgenerationlargelanguage). These agentic systems have shown strong performance on tasks such as code generation (rozière2024codellamaopenfoundation; DBLP:journals/corr/abs-2406-11931), document question answering (DBLP:journals/corr/abs-2410-18050), and interactive AI applications, highlighting their potential to automate sophisticated workflows. Most existing research, however, focuses on text-based interactions, overlooking speech as a natural and accessible modality. Extending agents to voice is critical for hands-free, conversational control in real-world applications. Current approaches typically depend on external automatic speech recognition (ASR) models, leaving open the fundamental question of how agents can directly process and respond to spoken input without an intermediate step. SpeechLMs(^1^11Although existing models and benchmarks cover multiple formats, including audio, speech, and music, our focus here is specifically on human speech tasks; henceforth we use the term SpeechLMs.) capable of instruction following and chat-style interactions (xu2025qwen25omnitechnicalreport; kimiteam2025kimiaudiotechnicalreport) pave the way toward such agents. By avoiding the sequential transcription overhead of ASR–LLM pipelines, SpeechLMs enable lower latency (Appendix [F](https://arxiv.org/html/2510.07978v2#A6)) and natural real-time interactions. They further demonstrate the ability to interpret spoken commands, maintain context across turns, and generate task-relevant responses in natural language, providing a strong foundation for voice assistants.

However, current evaluations of SpeechLMs primarily focus on individual tasks such as speech recognition, single-turn question answering and speech instruction following. Existing benchmarks overlook fundamental agentic capabilities essential for voice-based agents, including complex tool use, multi-turn interaction, and contextual decision-making, while providing limited multilingual coverage, particularly for Indic languages. As a result, there is a lack of standardized benchmarks that assess the ability of general-purpose voice agents to reason, plan, and execute complex agentic tasks in real-world settings. In this work, we introduce VoiceAgentBench (VAB),
a comprehensive agentic speech benchmark comprising over 5500 voice queries in 7 languages. VAB spans a wide range of tool-invocation tasks, from simple single-tool retrieval to the novel setting of orchestrating multiple dependent tools, as well as responding to adversarial queries. A significant portion of our benchmark queries are designed to reflect culturally grounded scenarios in the Indian context, enabling evaluation of contextual reasoning across diverse languages and cultural settings. To simulate realistic speaker variability, we introduce a diversity sampling method based on speaker embeddings for TTS voice conversion, producing a wide range of accents, speaking styles, and vocal characteristics. This ensures VoiceAgentBench captures the heterogeneity of real-world spoken interactions, making it an effective benchmark for evaluating SpeechLMs in multilingual, multicultural, and acoustically diverse settings. Our contributions could thus be summarized as follows:

- •
We present VoiceAgentBench, a benchmark of 5,500+ multilingual queries (English, Hindi, and 5 Indic languages). As far as we know, this is the first benchmark to evaluate agentic capabilities on speech. We will open-source VoiceAgentBench upon acceptance.
- •
We evaluate across diverse tool-invocation types (single/ multi-tool, multi-turn dialogue), including the novel setting of dependent tool orchestration, as well as adversarial safety.
- •
We propose a speaker embedding based sampling method for TTS voice conversion to simulate real-world diversity in accents, styles, and vocal characteristics.
- •
We evaluate state-of-the-art (SOTA) models in two settings: ASR-LLM pipelines and SpeechLMs, and reveal notable performance gaps in both, particularly in SpeechLMs.

Figure: Figure 1: Overview of the different agentic task categories in VoiceAgentBench, illustrating representative examples for each type of tool interaction, including single tool invocation, parallel and sequential tool use, multi-turn dialogue handling, and safety against harmful requests. The benchmark also supports multilingual capabilities, particularly for Indic languages.
Refer to caption: resources/placeholder/voiceagentbenchcategories.drawio.png

## 2 Related Work

LLM Agent Benchmarks. Interest in evaluating agentic LLMs has grown with advances in their reasoning and decision making capabilities. ToolBench (qin2024toolllm) evaluates models’ ability to invoke external APIs across diverse real-world tasks, while ToolQA (zhuang2023toolqa) assesses LLMs’ use of external tools for question answering via a scalable, automated dataset curation process. Berkeley Function Calling Leaderboard (BFCL) (patil2025the) emphasizes precise API generation across domains and robustness to both single and multiple function calls, and NESTful (basu2025nestfulbenchmarkevaluatingllms) focuses on nested sequences of API calls, where outputs of one call feed into the next. API-Bank and ToolTalk (li2023apibank; farn2023tooltalkevaluatingtoolusageconversational) target multi-turn, dialogue-driven tool-use scenarios, testing sequential API planning and interaction. Tau-bench (yao2025taubench) simulates dynamic conversations with domain-specific tools and policies to evaluate adherence to task rules. AgentHarmBench (andriushchenko2025agentharm) and DoomArena (boisvert2025doomarena) focus on safety and adversarial robustness, testing susceptibility to harmful or unsafe actions. Despite this progress for LLMs, no speech benchmark explicitly evaluates SpeechLMs in such realistic, agentic, and safety-critical settings, underscoring the need for specialized evaluation frameworks.

Speech Datasets and Benchmarks. Large-scale datasets such as LibriSpeech (7178964), CommonVoice (ardila-etal-2020-common), and MuST-C (di-gangi-etal-2019-must) have been foundational for advancing automatic speech recognition (ASR) and speech translation (AST). IndicST (11011192) and Lahaja (javed24_interspeech) extend these tasks to cover diverse Indic speech data. Evaluation suites like SUPERB (yang21c_interspeech) and SLUE (shon-etal-2023-slue) standardize the assessment of tasks such as intent classification, named entity recognition, and slot filling, with IndicSUPERB (javed2022indicsuperbspeechprocessinguniversal) further supporting Indic languages. However, these benchmarks primarily target simpler tasks like transcription, translation, NER and do not assess reasoning or decision-making over spoken content. To address this gap, recent work has begun exploring reasoning in the audio domain. Audio-CoT (ma2025audiocotexploringchainofthoughtreasoning) introduces chain-of-thought prompting for structured multistep inference on speech input, while MMAU (sakshi2025mmau) provides a large-scale benchmark of 10k audio clips covering 27 reasoning skills, such as temporal reasoning and causal inference, in speech, music, and environmental sounds. AIR-Bench (yang-etal-2024-air) and AudioBench (DBLP:journals/corr/abs-2406-16020) extend the scope to open-ended instruction following on various types of audio and speech, whereas VoiceBench (DBLP:journals/corr/abs-2410-17196) emphasizes robustness and generalization by converting text instruction into spoken form with real-world noise and speaker variation. More recently, SpeechR (yang2025speechrbenchmarkspeechreasoning) directly targets high-level reasoning on speech, focusing on logical deduction, and commonsense problem solving.
We also provide an extended discussion of related work on speech models in Appendix [B](https://arxiv.org/html/2510.07978v2#A2).

## 3 VoiceAgentBench

### 3.1 Overview

**Table 1: Statistics of VAB subsets across tasks, languages, and sources. VAB covers single, multiple, parallel, interdependent and dialogue based tool calls, and safety-focused evaluations.**
| Subset | Source Benchmark | Task | Language | Original<br>Functions | Original<br>Queries | Filtered<br>Functions | Indian Context<br>Queries | Average Duration<br>(sec) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Simple API | BFCL | Single Tool Parameter Filling | English | 370 | 400 | 151 | 142 | 4.50 |
| Hindi | 370 | – | 151 | 134 | 6.18 |  |  |  |
| 5 Indic | 370 | – | 151 | 710 | 7.32 |  |  |  |
| Multiple APIs | BFCL | Single Tool Retrieval + Param. Filling | English | 443 | 200 | 180 | 179 | 4.47 |
| Hindi | 443 | – | 180 | 177 | 6.03 |  |  |  |
| 5 Indic | 443 | – | 180 | 895 | 7.02 |  |  |  |
| Parallel Multi-APIs | BFCL | Parallel Tool Retrieval + Param. Filling | English | 458 | 200 | 246 | 125 | 10.67 |
| Hindi | 458 | – | 246 | 120 | 12.08 |  |  |  |
| 5 Indic | 458 | – | 246 | 625 | 14.80 |  |  |  |
| Dependent Multi-APIs | Novel | Interdependent Multi Tool Call | English | 21 | 40 | 21 | 40 | 4.53 |
| Hindi | 21 | 40 | 21 | 40 | 6.97 |  |  |  |
| 5 Indic | 21 | 200 | 21 | 200 | 7.10 |  |  |  |
| Level-2-API | API Bank | Dialogue-based Tool Call | English | 49 | 399 | 49 | 398 | 15.23 |
| Harmful | AgentHarmBench | Safety Evaluation via API Attacks | English | 76 | 176 | 76 | 80 | 28.13 |
| Hindi | 76 | – | 76 | 80 | 35.19 |  |  |  |
| 5 Indic | 76 | – | 76 | 400 | 32.04 |  |  |  |

VoiceAgentBench is a novel benchmark designed to evaluate the agentic capabilities for speech input in realistic spoken interaction scenarios. It comprises over 5,500 spoken queries synthetically generated using Text-to-Speech (TTS) engines, each paired with expected structured tool invocation or safety evaluation scenarios to enable rigorous assessment of core competencies required by real-world voice agents. As detailed in Table [1](https://arxiv.org/html/2510.07978v2#S3.T1), the benchmark spans six evaluation categories:

- •
Single Tool Call. Simple parameter filling on a spoken query given a tool
- •
Single Tool with retrieval. Selecting relevant tool from a tool list and parameter filling
- •
Parallel tool calls. Selecting and calling multiple independent tools from a tool list
- •
Dependent Tool calls. Selecting from a list of tools and making chained sequential tool calls where outputs of a tool call can feed into subsequent tool calls
- •
Dialog-Based Tool Invocation. Single tool calls based on multi-turn interactions
- •
Safety Evaluations. Rejecting adversarial queries and unsafe tool combinations

Each category in the benchmark is designed to isolate different agentic behaviours, enabling systematic evaluation of reasoning, retrieval, long-context, and tool orchestration capabilities. The evaluation framework further enhances interpretability by scoring each query along specific failure modes, including structured response generation, tool retrieval, and parameter filling. VAB further emphasizes indic multilingual generalization, covering English, Hindi, Bengali, Marathi, Tamil, Telugu, and Malayalam. 30% of the dataset consists of dialogues and queries contextualized in realistic Indian scenarios, to evaluate culturally grounded decision making. By combining structured evaluation targets, diverse linguistic coverage, and adversarial robustness testing, VAB fills a critical gap in the systematic evaluation of SpeechLMs’ real-world agentic competence. Table [5](https://arxiv.org/html/2510.07978v2#A1.T5) outlines the comparison of VAB with other key benchmarks.

### 3.2 Data Construction

In this section, we detail the construction of VoiceAgentBench, including the sourcing of tools, generation of culturally grounded queries and multi-turn dialogues, their conversion to speech using TTS with speaker diversity, and extension to multiple Indic languages; summarized in Fig [2](https://arxiv.org/html/2510.07978v2#S3.F2).

Figure: Figure 2: Pipeline for constructing VoiceAgentBench. We begin with seed tools, dialogues, and custom APIs for diverse agentic tasks. Indic grounding and TTS engine generate culturally contextualized speech queries, while diversity sampling ensures coverage across accents, and speakers. The final benchmark pairs user speech with tool context and model instructions.
Refer to caption: resources/placeholder/voiceagentbench_data.drawio.png

#### 3.2.1 Tool Sourcing

VAB sources tools to reflect core agentic capabilities like single-tool invocation, multi-tool orchestration, and safety evaluation, drawing from diverse sources to reflect realistic and varied domains.

Single Tool, Single Tool with Retrieval, and Parallel Tool Invocation. For these categories, we leverage BFCL (patil2025the), which provides well-structured functions for such cases. Single Tool Call directly adapts the simple tool subset from BFCL. The Single Tool with Retrieval category uses the multiple tool subset, requiring the model to select the most relevant tool and fill the arguments. The Parallel Tool category leverages the parallel multiple subset, where multiple independent tool calls are invoked simultaneously. In total, we incorporate 458 functions here.

Sequential Dependent Tools. While benchmarks like NESTful (basu2025nestfulbenchmarkevaluatingllms) focus on sequential tool invocation in specialized domains such as mathematics and coding, they do not capture the practical, everyday tasks expected of real-world voice assistants. These tasks often require chaining interdependent tools to complete workflows such as booking a cab, ordering food, or managing payments. To address this, we designed a set of 21 tools across three realistic agents: i) Cab booking, ii) Food ordering, and iii) Payment services. The toolsets are presented in Appendix [H](https://arxiv.org/html/2510.07978v2#A8).

Multi-Turn Dialogue Tools. For dialogue-based tool invocation, we adopt tools from the Level-2 subset of API-Bank (li2023apibank), which are designed to support Retrieval+Call based on multi-turn user interactions. Using this subset, we incorporate a total of 49 tools for this category, enabling evaluation of conversationally grounded, multi-turn tool-calling capabilities.

Safety Evaluation Tools. For evaluating the safety of agentic behavior, we utilize tools from the AgentHarm (andriushchenko2025agentharm). It encompasses 11 harm categories, including fraud, cybercrime, and harassment. These tasks are designed to assess whether models can refuse harmful agentic requests. By integrating these tools, we enable comprehensive evaluation of an agent’s robustness against adversarial and unsafe tool usage scenarios.

#### 3.2.2 Text Query & Dialogue Generation

To evaluate tool invocation across categories, we generate 2 types of inputs for each task in VAB: Source-native Queries and Indian context queries. Table [1](https://arxiv.org/html/2510.07978v2#S3.T1) reports the statistics across categories.

Source-native Queries. For categories with derived tools, we generate speech directly from the source queries identified during tool selection. These queries maintain the original intent and distribution of the datasets, providing continuity with prior evaluations while extending them to speech.

Indian Context Queries. To capture realistic, everyday scenarios reflective of voice assistant usage in India, we generate new queries and dialogues. This process relies on prompting LLMs with tool schemas and usage constraints, ensuring queries are both structurally valid and culturally grounded. Appendix [I](https://arxiv.org/html/2510.07978v2#A9) showcases examples across different agentic tasks categories.

i). Single Tool, Single Tool with Retrieval, and Parallel Tool Invocation: For Single Tool Invocation (SinTC), we filter 151 functions from BFCL’s simple subset and prompt Gemma3 27B to generate Indian-context queries requiring only parameter filling, producing paired Hindi/English tool calls. For Single Tool with Retrieval, we use 180 functions from BFCL’s multiple subset, clustered by domain, enrich candidate tool call list with tools from the same cluster to make retrieval more challenging and generate Indian-context queries using Gemma3 27B. For Parallel Tool Calling, we extend BFCL’s parallel subset by grouping co-invoked tools and prompting Gemma3 27B to generate Indian-context queries that demand multiple independent calls, with candidate lists further enriched with semantically and functionally related tools.

ii). Sequential Dependent Tool Calling (SeqDep): To capture realistic use cases, we design three service agents: cab booking, food ordering, and payments by creating seed data with available tools and expected outputs. Using GPT-4o-mini (hurst2024gpt), we generate queries conditioned on tool schemas and dependency chains, ensuring multiple interdependent calls (e.g., booking a cab after retrieving location coordinates or completing a food order with stored address and payment details). This yields 40 complex queries each in English and Hindi, paired with tool-call responses, capturing dependency-driven tasks beyond simple or parallel invocation.

v). Multi-Turn Dialog-Based Tool Calling: For the multi-turn dialogue category, we adapt 398 dialogues from API-Bank’s Level-2 subset by rewriting them in Indian context with GPT-4o-mini, preserving structure and final API correctness while updating responses for consistency, enabling evaluation of conversational grounding in realistic multi-turn interactions.

vi). Safety Evaluation: We adapt tasks from AgentHarm, preserving harmful intent but contextualizing user requests with Indian-specific entities. Requests are modified using Gemma3 27B to ensure they reflect realistic adversarial scenarios while maintaining the harmful category alignment.

#### 3.2.3 Model Instructions and Indic Multilingual Extension

To standardize behavior across models, we design category-specific system instructions that direct the model to produce tool calls strictly in Python syntax, following patil2025the. This prevents free-form or mixed response which cannot be deterministically parsed. To further anchor the model’s behavior, we provide a one-shot example in the instruction that clarifies the output format without imposing strong task-specific biases. For multilingual settings, we additionally instruct the model to generate tool calls exclusively in English, aligning with real-world usage scenarios. Additionally, for safety evaluations we append a refusal prompt in the instructions (examples in Appendix [J.1](https://arxiv.org/html/2510.07978v2#A10.SS1)).

We extend VAB to five Indic languages by translating the English Indian context queries into the target languages. For Malayalam, we employ Llama-3.3 70B, while for the remaining languages we use Gemma3 27B, this is based on human eval results shown in (2025bhasha). We complement this with a lightweight quality control pipeline that flags issues such as script mixing and unknown tokens. The validated queries are then fed into the diversity based TTS pipeline.

#### 3.2.4 Diversity Based TTS Generation

In synthetic speech generation, the absence of real speakers and natural voices necessitates methods that ensure diversity in the generated data, motivating new selection strategies to build robust and representative benchmarks. Following IndicSynth (sharma-etal-2025-indicsynth), which employed the VoxLingua107 ECAPA-TDNN model (Desplanques_2020), trained on diverse Indic languages and accents, to evaluate the linguistic authenticity of synthetic speech, we adopt ECAPA-TDNN embeddings for our diversity analysis. Adapting diversity principles from word embedding literature, we transfer these ideas to the audio domain using ECAPA-TDNN embeddings. Specifically, we ablate three strategies for selecting maximally diverse audio samples: i) Determinantal Point Processes (DPP) (wang2025diversity), ii) Farthest Point Sampling (FPS) adapted from PointNet++ (PointNet++), and iii) a Density-based Probabilistic Method (Appendix [C.1](https://arxiv.org/html/2510.07978v2#A3.SS1)). Diversity is quantified using the mean distance to the nearest selected point (yang2025measuringdatadiversityinstruction), a metric that captures coverage of the embedding space. Our evaluation (Appendix [C.2](https://arxiv.org/html/2510.07978v2#A3.SS2)) shows that FPS (Algorithm 1) consistently achieves the highest diversity scores on our dataset, establishing it as the most effective strategy. We conduct this ablation and sample final audios for voice conversion from IndicSuperb (javed2022indicsuperbspeechprocessinguniversal), to ensure Indic language coverage and gender-balanced diversity, and from IndicST (11011192), which collates English–Indic open-source audios.

| Algorithm 1 Diverse Audio Selection Using Farthest Point Sampling (FPS) |
| --- |
| Require: $A$ (set of audio samples), $M$ (desired subset size) |
| 1: procedure SELECTDIVERSEAUDIO($A$, $M$) |
| 2:  Extract embeddings $E=\{e_{1},e_{2},\ldots,e_{N}\}$ using ECAPA-TDNN |
| 3:  Compute distance matrix $D$ where $D(i,j)=||e_{i}-e_{j}||_{2}$ |
| 4:  Randomly select initial point $p_{0}$ and set $R=\{p_{0}\}$ |
| 5:  while $|R|<M$ do |
| 6:   for each $x\in A\setminus R$ do |
| 7:    $d(x)=\min_{r\in R}D(x,r)$ $\triangleright$ distance to nearest selected point |
| 8:   end for |
| 9:   $x^{*}=\arg\max_{x\in A\setminus R}d(x)$ $\triangleright$ select point farthest from current subset |
| 10:   $R=R\cup\{x^{*}\}$ |
| 11:  end while |
| 12:  return $R$ |
| 13: end procedure |

Text to Speech (TTS) Conversion. For English queries, speech is generated using ElevenLabs(^2^22[https://elevenlabs.io/](https://elevenlabs.io/))
and subsequently passed through Coqui-TTS(^3^33[https://github.com/coqui-ai/TTS](https://github.com/coqui-ai/TTS)) for voice conversion along with the sampled diverse audio. For Hindi and other Indian languages, we pass both the query and the sampled audio from diversity algorithm to Krutrim-TTS (^4^44[https://bit.ly/Krutrim-TTS](https://bit.ly/Krutrim-TTS)) which
handles both speech generation and voice conversion.

### 3.3 Evaluation Framework

Our evaluation framework is designed to assess voice assistants in a layered fashion, capturing complementary abilities across entity recognition, intent understanding, reasoning and robustness in all task categories. To capture failure cases and low performance across specific abilities, we use four evaluation metrics, listed below, with implementation details provided in the Appendix [G](https://arxiv.org/html/2510.07978v2#A7).

i) Tool Selection (TS): This checks if the correct tools are being called regardless of output format by doing an exact match on the expected tool names. It’s implemented through regex-based validation.

ii) Tool Call Structure (TCS) This evaluates if the tools follow the expected output format and schema. It’s applied only to correctly selected tools by validating against their Pydantic(^5^55[https://docs.pydantic.dev/latest/](https://docs.pydantic.dev/latest/))
model.

iii) Parameter Filling (PF) This evaluates whether the arguments generated for the selected tool align with the ground truth values. Since exact matching fails to capture valid semantic variations, we employ GPT-4o-mini as a judge to robustly assess faithfulness to the ground truth.

iv) Refusal Rate (RR). This is a model safety focused metric which checks if the system declines harmful or unsafe requests instead of executing them. We replicate the implementation in andriushchenko2025agentharm, using GPT-4o-mini as a semantic judge to classify each response.

## 4 Evaluations

### 4.1 Models

We evaluated 2 classes of speech-based systems on VAB: SpeechLMs and ASR-LLM pipelines.

SpeechLMs.
We benchmark 3 SOTA 7B SpeechLMs: (i) KimiAudio 7B (kimiteam2025kimiaudiotechnicalreport),
(ii) Qwen2.5-Omni 7B (xu2025qwen25omnitechnicalreport),
(iii) AudioFlamingo3 7B (ghosh2025audio).

ASR-LLMs. In this modular setup, user speech is first transcribed with Whisper v3 Large (Whisperv3), and the text is then passed to an LLM along with tools and instructions. We benchmark three strong LLMs: Qwen-3 8B (yang2025qwen3technicalreport), Gemma3 27B (gemmateam2025gemma3technicalreport), and LLaMA 3.3 70B (Llama3 70B), enabling comparison between this setup and SpeechLMs.

**Table 2: Performance comparison on the Indian-context queries. Evaluation of models across Single Tool Calling (SinTC), SinTC with Retrieval, Parallel Tool Calling, Sequential-Dependent Tool Calling (SeqDepTC), and Multi-turn Dialogue Tool Calling on English, Hindi, and Indic datasets. Metrics include TS, TCS, and PF (see Section [3.3](https://arxiv.org/html/2510.07978v2#S3.SS3) for definitions). For Indic subset we report average across all the 5 Indic languages. TS for Single Tool Calling is trivial, often yielding near-perfect scores. Best values are in bold, second best are underlined.**
| Model | Single Tool Calling | SinTC with Retrieval | Parallel Tool Calling | SeqDep Tool Calling | Multi-turn | Avg |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ | PF $\uparrow$ |
| English Subset |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Qwen2.5-Omni 7B | 100.00 | 2.11 | 1.41 | 90.5 | 0.00 | 0.00 | 73.47 | 0.40 | 0.00 | 55.00 | 5.00 | 5.00 | 80.30 | 2.02 | 2.02 | 1.69 |
| AudioFlamingo3 7B | 91.55 | 38.03 | 23.94 | 64.25 | 30.17 | 20.11 | 51.07 | 19.73 | 16.53 | 25.00 | 0.00 | 0.00 | - | - | - | 15.15 |
| KimiAudio 7B | 100.00 | 94.37 | 68.31 | 89.39 | 77.65 | 66.48 | 84.13 | 80.13 | 68.67 | 65.00 | 17.5 | 5.00 | 87.57 | 83.6 | 61.38 | 53.97 |
| Whisperv3-Qwen3 8B | 100.00 | 94.89 | 63.5 | 96.59 | 92.61 | 71.59 | 90.98 | 87.57 | 76.78 | 81.48 | 48.15 | 14.81 | 59.22 | 50.32 | 36.78 | 52.69 |
| Whisperv3-Gemma3 27B | 100.00 | 93.66 | 64.79 | 96.09 | 84.36 | 63.69 | 93.07 | 89.60 | 77.60 | 85.00 | 47.50 | 12.50 | 91.69 | 90.03 | 56.81 | 55.08 |
| Whisperv3-Llama3 70B | 100.00 | 94.37 | 62.68 | 97.77 | 90.5 | 72.07 | 88.93 | 85.33 | 74.93 | 82.5 | 42.5 | 10 | 97.73 | 93.43 | 61.62 | 56.26 |
| Hindi Subset |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Qwen2.5-Omni 7B | 100.00 | 0.00 | 0.00 | 79.10 | 0.00 | 0.00 | 72.64 | 0.00 | 0.00 | 41.03 | 0.00 | 0.00 | - | - | - | 0.00 |
| AudioFlamingo3 7B | 92.54 | 20.90 | 10.45 | 49.72 | 14.12 | 7.34 | 36.67 | 16.25 | 10.69 | 41.03 | 0.00 | 0.00 | - | - | - | 7.12 |
| KimiAudio 7B | 100.00 | 95.52 | 62.69 | 81.36 | 66.10 | 47.46 | 77.78 | 72.78 | 50.69 | 53.85 | 7.69 | 7.69 | - | - | - | 42.13 |
| Whisperv3-Qwen3 8B | 99.25 | 93.23 | 75.94 | 93.14 | 89.71 | 76.57 | 88.60 | 84.36 | 69.74 | 38.71 | 12.90 | 3.23 | - | - | - | 56.37 |
| Whisperv3-Gemma3 27B | 100.00 | 96.27 | 81.34 | 93.79 | 72.88 | 59.89 | 86.67 | 83.75 | 66.94 | 57.89 | 23.68 | 7.89 | - | - | - | 54.02 |
| Whisperv3-Llama3 70B | 100.00 | 95.52 | 76.87 | 92.66 | 87.01 | 73.45 | 89.72 | 86.81 | 75.42 | 60.53 | 36.84 | 7.89 | - | - | - | 58.41 |
| Indic Subset |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Qwen2.5-Omni 7B | 97.01 | 1.92 | 0.44 | 43.89 | 0.00 | 0.00 | 23.55 | 0.00 | 0.00 | 14.75 | 1.94 | 0.00 | - | - | - | 0.11 |
| AudioFlamingo3 7B | 90.41 | 27.74 | 5.15 | 23.77 | 7.55 | 1.65 | 23.56 | 9.19 | 1.18 | 27.30 | 0.00 | 0.00 | - | - | - | 1.99 |
| KimiAudio 7B | 99.40 | 94.22 | 40.32 | 61.82 | 50.67 | 25.62 | 60.35 | 53.89 | 34.24 | 28.94 | 2.56 | 1.04 | - | - | - | 25.30 |
| Whisperv3-Qwen3 8B | 97.86 | 92.80 | 41.30 | 81.70 | 78.74 | 40.97 | 61.37 | 58.23 | 37.64 | 31.21 | 8.49 | 1.84 | - | - | - | 30.34 |
| Whisperv3-Gemma3 27B | 91.25 | 85.89 | 41.23 | 67.75 | 61.10 | 37.60 | 64.38 | 61.50 | 43.01 | 31.59 | 9.77 | 3.64 | - | - | - | 31.36 |
| Whisperv3-Llama3 70B | 99.57 | 93.94 | 49.09 | 82.79 | 79.25 | 49.96 | 62.14 | 59.34 | 38.74 | 45.05 | 11.99 | 3.60 | - | - | - | 35.35 |

**Table 3: Performance comparison on source-native queries. Evaluation of models on Single Tool Calling (SinTC), SinTC with Retrieval, and Parallel Tool Calling. Metrics include TS, TCS, and PF (see Section [3.2.2](https://arxiv.org/html/2510.07978v2#S3.SS2.SSS2) for definitions). TS for Single Tool Calling is trivial, often yielding near-perfect scores. Best values are in bold, second best are underlined.**
| Model | Single Tool Calling | SinTC with Retrieval | Parallel Tool Calling | Avg |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ |  |
| Qwen2.5-Omni 7B | 99.75 | 1.25 | 1.25 | 100 | 0.00 | 0.00 | 96.22 | 0.87 | 0.29 | 0.51 |
| AudioFalmingo3 7B | 88.25 | 39.5 | 33 | 87.5 | 41.5 | 35 | 67.73 | 31.69 | 29.07 | 32.36 |
| KimiAudio 7B | 100.00 | 90.25 | 83.25 | 99 | 85.5 | 74.5 | 96.8 | 84.59 | 81.69 | 79.81 |
| Whisperv3-Qwen3 8B | 100.00 | 91.14 | 81.01 | 100.00 | 90.95 | 82.41 | 94.80 | 87.13 | 84.13 | 82.52 |
| Whisperv3-Gemma3 27B | 100.00 | 91.00 | 80.50 | 100.00 | 91.00 | 82.50 | 99.74 | 91.73 | 85.10 | 82.70 |
| Whisperv3-Llama3 70B | 100.00 | 90.50 | 81.25 | 100.00 | 91.00 | 85.50 | 99.74 | 90.22 | 86.68 | 84.48 |

**Table 4: Refusal rates (%) on the Safety subset. Evaluation on both Indian Context and Source-native with refusal prompt appended. The Indic subset reports the average across 5 Indic languages. Best scores are in bold, second best are underlined. Whisper v3 refers to Whisper-v3 large.**
| Model | English | Hindi | Indic Avg | Source-native |
| --- | --- | --- | --- | --- |
| Qwen2.5-Omni 7B | 18.75 | 6.67 | 4.31 | 20.69 |
| Audio-Flamingo-3 | 7.50 | 6.67 | 17.0 | 7.39 |
| KimiAudio 7B | 51.25 | 1.33 | 2.94 | 52.30 |
| Whisperv3-Qwen3 8B | 52.50 | 29.33 | 49.90 | 59.43 |
| Whisperv3-Gemma3 27B | 56.25 | 37.33 | 38.37 | 62.86 |
| Whisperv3-Llama3 70B | 42.5 | 58.75 | 44.75 | 35.43 |

### 4.2 Main Results

We present the primary results for English, Hindi, and the average across five additional Indic languages on VoiceAgentBench in Tables [2](https://arxiv.org/html/2510.07978v2#S4.T2), [3](https://arxiv.org/html/2510.07978v2#S4.T3) and [4](https://arxiv.org/html/2510.07978v2#S4.T4). Per-language extended results for all Indic languages are provided in Appendix [E](https://arxiv.org/html/2510.07978v2#A5).

SpeechLMs lag behind ASR-LLM setups.
There remains a significant performance gap between SpeechLMs and ASR–LLM pipelines across all tasks. Among SpeechLMs, KimiAudio 7B achieves the strongest results, performing comparably to ASR–LLM pipelines, while AudioFlamingo3 7B and Qwen2.5-Omni 7B lag substantially behind. Nevertheless, even KimiAudio 7B underperforms relative to an equivalently sized Whisper3–Qwen3 model on all English parameter-filling tasks, except in multi-turn settings. This gap increases in other languages. This is unsurprising, as Qwen3-8B and other LLMs have been extensively trained on agentic tasks, whereas most SpeechLM pipelines focus primarily on audio understanding and conversational objectives. KimiAudio 7B’s performance shows that SpeechLMs can approach ASR–LLM pipelines on agentic tasks. Given their lower Time Taken for First Token (TTFT) generation (Appendix [F](https://arxiv.org/html/2510.07978v2#A6)) and ability to leverage agentic context during speech decoding, further training on such tasks could substantially enhance their effectiveness and applications.

KimiAudio 7B establishes a strong lead among SpeechLMs.
Among SpeechLMs, KimiAudio 7B outperforms both Qwen2.5-Omni 7B and AudioFlamingo3 7B by a wide margin, achieving 53.9% PF accuracy on English benchmarks compared to 15.2% and 1.7%, respectively. AudioFlamingo3 7B shows limited reasoning in agentic settings, often failing tool identification but producing well-structured calls when successful. Qwen2.5-Omni 7B demonstrates stronger tool identification but struggles with schema compliance, reflecting limited exposure to structured supervision. Overall, KimiAudio 7B exhibits robust instruction-following and task performance despite being comparable in size to the others, suggesting that additional training of AudioFlamingo3 7B and Qwen2.5-Omni 7B on diverse, structured agentic tasks could significantly improve their performance.

Minimal Gaps Across ASR–LLM Pipelines. All ASR–LLM pipelines achieve broadly similar performance, with none surpassing 70% PF on any task, indicating substantial room for improvement. Whisperv3-Llama3 70B performs best overall, while Whisperv3-Gemma3 27B closely matches it in English and even outperforms it in parallel tool calling (77.6% vs. 74.9%). Whisperv3-Qwen3 8B also delivers comparable results (52.7% vs. 56.3% for Whisperv3-Llama3 70B) despite being significantly smaller, highlighting its suitability for agentic tasks.

Limited generalization on Indic languages. The average PF across languages is not directly comparable, as benchmark queries differ in content and difficulty. Nonetheless, while ASR–LLM pipelines maintain comparable or slightly improved performance when moving from English to Hindi, KimiAudio 7B’s PF drops from 54% in English to 42% in Hindi and further to 25% for other Indic languages. A similar decline is observed for Whisperv3-Qwen3 8B (53% to 30%). These results suggest that current SpeechLMs have been exposed to limited Indic data, underscoring the need for richer Indic supervision to preserve and enhance multilingual grounding.

Drop in Indian-context grounding. Both SpeechLMs and ASR-LLM pipelines show substantial degradation when moving from the Source-native benchmark (Table [3](https://arxiv.org/html/2510.07978v2#S4.T3)), which lacks cultural diversity, to the Indian-context set (Table [2](https://arxiv.org/html/2510.07978v2#S4.T2)). Across key tool-invocation categories, KimiAudio 7B’s average PF drops from 79.8% to 67.8%, and AudioFlamingo3 7B from 32.4% to 20.2%. Even the strongest ASR-LLM pipeline, Whisperv3-Llama3 70B, experiences reductions from 84.5% to 69.9% on average. In Single Tool Calling, KimiAudio 7B decreases from 83.3% to 68.3% and Whisperv3-Llama3 70B from 81.3% to 62.7%; in Parallel Tool Calling, KimiAudio 7B drops from 81.7% to 68.7% and Whisperv3-Llama3 70B from 86.7% to 74.9%. These results highlight that both end-to-end SpeechLMs and ASR-LLM pipelines struggle to maintain tool-invocation accuracy when queries require Indian-context grounding, revealing a clear gap in cultural robustness.

Sequential and dependent tool calling remains challenging. These tasks are the most difficult, with steep declines in PF scores across all models. Even the best ASR-LLM pipeline (Whisperv3-Qwen3 8B) achieves only 14.8% PF on English, while Whisperv3-Llama3 70B reaches just 7.9% on Hindi. These results emphasize the importance of evaluating multi-step and interdependent tool execution, which are essential in real-world agentic workflows. The inclusion of 21 practical tools in VoiceAgentBench highlights the benchmark’s ability to capture these complex challenges.

SpeechLMs lag behind on safety and refusal robustness. Safety evaluation reveals a stark gap between end-to-end SpeechLMs and ASR-LLM pipelines. Among SpeechLMs, KimiAudio 7B achieves a high refusal rate of 51.25% in English, but this drops sharply to 1.33% in Hindi and 2.94% on average across other Indic languages. Qwen2.5-Omni 7B shows a similar pattern, with 18.75% in English, 6.67% in Hindi, and 4.31% on the Indic average, while Audio-Flamingo-3 performs worst in English at 7.50%, though slightly better on Indic languages. In contrast, ASR-LLM such as Whisperv3-Gemma3 27B or Whisperv3-Qwen3 8B demonstrate substantial robustness, achieving up to 56.25% in English, 37.33% in Hindi, and 49.90% on the Indic average. These results highlight that SpeechLMs struggle to maintain consistent safety behavior across languages, emphasizing the need for stronger grounding and instruction-following to reliably refuse unsafe or harmful requests.

### 4.3 Ablation Studies & Analysis

Quantifying ASR-Induced Degradation in ASR-LLM Pipelines.
Given the relatively poorer performance of ASR–LLM pipelines in Indic settings, we examined how much of the degradation stems from ASR errors. We replaced Whisper outputs with the ground-truth transcripts and passed it to LLMs. This led to a jump of at least 24% in average PF scores across all non-Hindi Indic languages (see Table [6](https://arxiv.org/html/2510.07978v2#A4.T6) in Appendix), while the gains in English were smaller: 7–15% for Single Tool Calling and minimal for Parallel Tool Calling (Table [7](https://arxiv.org/html/2510.07978v2#A4.T7) in Appendix). These results indicate that Whisper’s weaker transcription in Indic languages is a major bottleneck, suggesting that stronger Indic ASR models could improve downstream performance.

One-Shot over Zero-Shot Instruction.
To evaluate the impact of one-shot examples in SpeechLMs, we remove it from KimiAudio 7B’s system prompt. This resulted in PF drops of atleast 10% for Parallel Tool Calling and SinTC with Retrieval (up to 17% for Hindi), while Single Tool Calling remained unaffected (0% English, 1.5% Hindi; Table [8](https://arxiv.org/html/2510.07978v2#A4.T8)), likely due to lower complexity.

Refusal Prompts Drive Safety, but Adversarial Hints Remain Challenging. In our safety subset, all queries include refusal prompts, and half contain harmful hints. Removing refusal prompts sharply lowers safety rates: KimiAudio 7B and Whisperv3-Qwen3 8B drop moderately, while Whisperv3-Gemma3 27B and Whisperv3-Llama3 70B fall fourfold (see Figure [4](https://arxiv.org/html/2510.07978v2#A4.F4) in Appendix). Adversarial hints further reduce refusal rates for all models to 35–40% (in Figure [5](https://arxiv.org/html/2510.07978v2#A4.F5) in Appendix), with Whisperv3-Gemma3 27B, Whisperv3-Qwen3 8B, and KimiAudio 7B outperforming Whisperv3-Llama3 70B on English queries. Performance is lower in Hindi, except for Whisperv3-Llama3 70B.

## 5 Limitations and Conclusion

Limitations. We acknowledge following limitations in our work: First, our evaluation does not include speech with background noise, and therefore we do not measure the noise impact on tool call invocation. Second, we do not extend evaluation for multi-turn dialogues for Indic languages, which is also critical for building general-purpose voice assistants. Third, due to prohibitive costs, we exclude closed-source voice assistant systems such as GPT-4o-audio and Gemini-2.5-Pro from our evaluation. Lastly, our study does not evaluate dynamic, real-time tool invocation with interactive user conversation, as explored in frameworks like yao2025taubench.

Conclusion. We introduce VoiceAgentBench with 5,500 synthetic spoken queries across English, Hindi, and five other Indian languages, providing a comprehensive benchmark for evaluating SpeechLMs in realistic agentic settings. Our experiments reveal substantial gaps in multi-tool orchestration, multi-turn dialogue, Indic language generalization, and adversarial robustness, highlighting critical limitations of current models. We hope this benchmark will drive the development of speech agents that are more capable, safe, and culturally inclusive.

## Ethics and Reproducibility Statement

##### Ethics Statement.

This work centers on the responsible creation of a benchmark for evaluating SpeechLMs in realistic spoken-agent settings, with a particular focus on multilingual and India-specific agentic queries. We employed strict filtering to minimize harmful or unsafe content, while recognizing that model outputs cannot be entirely controlled. All external datasets, tools, and resources are properly credited through citations, and no sensitive or personally identifiable information (PII) was collected. To encourage diversity, we designed a controlled pipeline for audio generation using a TTS engine suited to our tasks. Since no personal or medical data were involved, formal IRB approval was not required. At every stage, we aimed to advance robust speech agents while mitigating risks of bias and harm, releasing the benchmark to foster safe, multilingual, and culturally inclusive speech technologies.

##### Reproducibility Statement.

To ensure reproducibility, we will make all
artifacts publicly available, accompanied by comprehensive documentation. We carefully log experimental configurations, hyperparameters, and evaluation procedures so that results can be replicated with fidelity.

Appendix

## Appendix A VoiceAgentBench comparison with Other Agentic Benchmarks

Table [5](https://arxiv.org/html/2510.07978v2#A1.T5) contrasts VoiceAgentBench with existing text and speech agent benchmarks along nine key evaluation axes. Text-based datasets such as AgentHarm, APIBank, and BFCL focus primarily on tool invocation but and do not address cultural or multilingual grounding. While APIBank and BFCL include multiple tool calls and multi-turn dialogues, they do not evaluate sequentially dependent tool use, safety, or cross-lingual generalization. On the speech side, existing benchmarks remain limited in scope. VoiceBench targets safety in speech alignment but does not include tool usage, while AudioBench provides large-scale multilingual speech data without agentic tool-calling tasks. In contrast, VoiceAgentBench uniquely integrates all dimensions: it supports speech-based single, parallel, and sequential tool calls, multi-turn dialogues, safety evaluations, multilingual coverage, and cultural diversity. With 5,757 queries, it establishes the most comprehensive benchmark to date for evaluating speech-grounded tool-usage.

Table: Table 5: Comparison of text and speech benchmark across key agentic evaluation axes. VoiceAgentBench uniquely covers all dimensions, making it the most comprehensive benchmark for speech-grounded tool-using agents.

## Appendix B Additional Related Work: Speech Models

Early audio-language encoders, such as AudioCLIP (9747631) and CLAP (10095889), learn joint embeddings of speech and text, enabling tasks like cross-modal retrieval, keyword-based speech search, and basic classification. These models primarily focus on representation learning without complex reasoning or generative capabilities. Specialized speech models, including Whisper (radford2022robustspeechrecognitionlargescale), SALM (chen2024salm), and AudioPALM (rubenstein2023audiopalmlargelanguagemodel), excel in automatic speech recognition (ASR), speech-to-text translation, and speech understanding, enabling transcription, translation, and limited instruction following over speech inputs. Integrated multitask models such as AudioGPT (DBLP:conf/aaai/HuangLYSCYWHHLR24), WavLLM (DBLP:journals/corr/abs-2404-00656), LTU (gong2024listen), and SALMONN (tang2024salmonn) extend these capabilities to multi-turn dialogue, question answering, and instruction following by combining ASR, speech understanding, and LLM-based reasoning. Recent large audio-language models, including Qwen2-Audio (chu2024qwen2audiotechnicalreport), KimiAudio 7B (kimiteam2025kimiaudiotechnicalreport), Qwen2.5-Omni 7B (xu2025qwen25omnitechnicalreport), and Audio Flamingo 3 (ghosh2025audio), further enhance reasoning capabilities over speech, enabling long-form question answering, multi-step instruction execution, and chat-style conversation.

## Appendix C Diversity Methodologies

### C.1 Density-Based Probabilistic Method

The core idea of this method is to select sparsely populated points in the embedding space. We assign probability score to each point based on the number of nearest neighbors within a set radius and sample based on these scores.

In this method, we start with a set of audio samples from source dataset. Then, each audio sample is passed through an ECAPA-TDNN (Desplanques_2020) model trained on VoxLingua107 to generate fixed-dimensional embeddings that capture both speaker identity and acoustic features:

$$ $\mathbf{e}_{i}=f(a_{i}),\quad\mathbf{e}_{i}\in\mathbb{R}^{d}$ $$

where $f(\cdot)$ represents the embedding extraction function and $\mathbf{d}$ is the embedding dimension. These embeddings allow diversity to be analyzed in a structured and principled way.

Pairwise Euclidean distances between embeddings are calculated to measure similarity:

$$ $D(i,j)=||\mathbf{e}_{i}-\mathbf{e}_{j}||_{2}$ (1) $$

where smaller values indicate similar voices or acoustic conditions, and larger values indicate greater diversity. These distances form a distance matrix, capturing the relationships across the dataset.

A radius $r$ is then defined as the mean of all pairwise distances:

$$ $r=\frac{1}{N^{2}}\sum_{i=1}^{N}\sum_{j=1}^{N}D(i,j)$ (2) $$

where $N$ is the total number of audios. For each audio sample $i$, the neighbor count $n_{i}$ is computed by counting how many other samples lie within this radius:

$$ $n_{i}=\sum_{j=1}^{N}\mathbb{I}(D(i,j)\leq r)$ (3) $$

where $\mathbb{I}(\cdot)$ equals 1 when the distance is within the threshold and 0 otherwise.

- •
High $n_{i}$ $\rightarrow$ sample is in a dense cluster and likely redundant.
- •
Low $n_{i}$ $\rightarrow$ sample lies in a sparse region and contributes strongly to diversity.

The neighbor counts are transformed into diversity scores using a sigmoid-based inverse function to prioritize sparse samples:

$$ $s_{i}=\frac{1}{1+e^{k(n_{i}-\mu)}}$ (4) $$

where $\mu$ is the median neighbor count and $k$ controls the steepness of the sigmoid. Sparse samples with low $n_{i}$ receive higher scores, while dense cluster samples are penalized with lower scores.

These scores are normalized into a probability distribution:

$$ $P_{i}=\frac{s_{i}}{\sum_{j=1}^{N}s_{j}}$ (5) $$

This enables probabilistic selection, where diverse samples are more likely to be chosen but randomness is preserved to avoid bias toward extreme outliers.

### C.2 Comparision of Methodologies for selection of diverse audios

We evaluate three selection methods by choosing 20 audios from approximately 1,000 samples in English, Hindi, and five additional Indic languages, repeating the process 20 times. Selection is based on the mean distance to the nearest selected point. Our results indicate that Farthest Point Sampling (FPS) consistently achieves the highest mean distance compared to Density Based Sampling and DDP. For a representative sample, we visualize the mean distance and the corresponding t-SNE plots of the selected points across the three methods in Figure [3](https://arxiv.org/html/2510.07978v2#A3.F3).

Figure: (a) Density Based Sampling
Refer to caption: resources/placeholder/density_based_modified.png

## Appendix D Ablation Results

In this section, we report the ablation studies and analyses that complement the discussion in Section [4.3](https://arxiv.org/html/2510.07978v2#S4.SS3). We first quantify the effect of ASR errors on task performance. Tables [6](https://arxiv.org/html/2510.07978v2#A4.T6) and [7](https://arxiv.org/html/2510.07978v2#A4.T7) compare results obtained using ground-truth transcripts against WhisperV3-generated transcripts across three models (LLaMA-70B, Gemma3 27B, and Qwen3 8B), for both Indic and English subsets. The Difference rows highlight the degradation in accuracy attributable to ASR noise across tool selection, call structure, and parameter filling.

**Table 6: Ablation study: Impact of ASR errors on performance on Indic subset. Comparison of model performance using WhisperV3-generated transcripts versus ground-truth transcripts across LLaMA3.3-70B, Gemma3 27B, and Qwen3 8B. The Difference rows highlight performance degradation caused by ASR errors across Single Tool Calling, SinTC with Retrieval, and Parallel Tool Calling subsets.**
| Model | Single Tool Calling | SinTC with Retrieval | Parallel Tool Calling |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ |
| Qwen3 8B |  |  |  |  |  |  |  |  |  |
| Transcripts + Qwen3 8B | 100.00 | 94.17 | 72.09 | 93.95 | 90.17 | 65.04 | 89.25 | 85.99 | 72.96 |
| Whisperv3-Qwen3 8B | 97.86 | 92.80 | 41.30 | 81.70 | 78.74 | 40.97 | 61.37 | 58.23 | 37.64 |
| Difference ($\Delta$) | 2.14 | 1.37 | 30.79 | 12.25 | 11.43 | 24.07 | 27.89 | 27.76 | 35.32 |
| Gemma3 27B |  |  |  |  |  |  |  |  |  |
| Transcripts + Gemma3 27B | 100.00 | 94.37 | 79.58 | 92.55 | 81.75 | 71.14 | 90.18 | 86.49 | 76.62 |
| Whisperv3-Gemma3 27B | 91.25 | 85.89 | 41.23 | 67.75 | 61.10 | 37.60 | 64.38 | 61.50 | 43.01 |
| Difference ($\Delta$) | 8.75 | 8.47 | 38.34 | 24.80 | 20.65 | 33.54 | 25.80 | 24.99 | 33.62 |
| LLaMA-70B |  |  |  |  |  |  |  |  |  |
| Transcripts + Llama3.3-70B | 100.00 | 94.60 | 74.88 | 95.72 | 91.62 | 76.72 | 89.02 | 85.07 | 75.11 |
| Whisperv3-Llama3 70B | 99.57 | 93.94 | 49.09 | 82.79 | 79.25 | 49.96 | 62.14 | 59.34 | 38.74 |
| Difference ($\Delta$) | 0.43 | 0.66 | 25.80 | 12.93 | 12.37 | 26.76 | 26.88 | 25.72 | 36.37 |

**Table 7: Ablation study: Impact of ASR errors on performance on English subset. Comparison of model performance using WhisperV3-generated transcripts versus ground-truth transcripts across LLaMA3 70B, Gemma3 27B, and Qwen3 8B. The Difference rows highlight performance degradation caused by ASR errors across Single Tool Calling, SinTC with Retrieval, and Parallel Tool Calling subsets.**
| Model | Single Tool Calling | SinTC with Retrieval | Parallel Tool Calling |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ |
| LLaMA3 70B |  |  |  |  |  |  |  |  |  |
| Transcripts + Llama3 70B | 100.00 | 94.84 | 76.76 | 96.83 | 92.74 | 78.58 | 88.84 | 85.24 | 76.71 |
| Whisperv3-Llama3 70B | 100.00 | 94.37 | 62.68 | 97.77 | 90.50 | 72.07 | 88.93 | 85.33 | 74.93 |
| Difference ($\Delta$) | 0.00 | 0.47 | 14.08 | -0.94 | 2.24 | 6.51 | -0.09 | -0.09 | 1.78 |
| Gemma3 27B |  |  |  |  |  |  |  |  |  |
| Transcripts + Gemma3 27B | 100.00 | 94.37 | 80.52 | 93.48 | 85.10 | 74.12 | 90.93 | 87.60 | 78.02 |
| Whisperv3-Gemma3 27B | 100.00 | 93.66 | 64.79 | 96.09 | 84.36 | 63.69 | 93.07 | 89.60 | 77.60 |
| Difference ($\Delta$) | 0.00 | 0.71 | 15.73 | -2.61 | 0.74 | 10.43 | -2.14 | -2.00 | 0.42 |
| Qwen 8B |  |  |  |  |  |  |  |  |  |
| Transcripts + Qwen3 8B | 100.00 | 94.44 | 71.25 | 94.93 | 90.99 | 72.30 | 90.68 | 87.44 | 77.23 |
| Whisperv3-Qwen3 8B | 100.00 | 94.89 | 63.50 | 96.59 | 92.61 | 71.59 | 90.98 | 87.57 | 76.78 |
| Difference ($\Delta$) | 0.00 | -0.45 | 7.75 | -1.66 | -1.62 | 0.71 | -0.30 | -0.13 | 0.45 |

We next analyze the impact of few-shot prompting. Table [8](https://arxiv.org/html/2510.07978v2#A4.T8) reports results for KimiAudio 7B in zero-shot versus one-shot settings on English and Hindi subsets. These results illustrate the relative gains from a single demonstration compared to zero-shot prompting for the SpeechLMs, giving signficant boost to tool call structure and output response following.

**Table 8: Zero-Shot instruction results. We evaluate KimiAudio 7B on Single Tool Calling, Single Tool (SinTC) Calling with retrieval and Parallel Tool Calling in zero-shot and one-shot setting. Difference shows that Zero-shot leads to significant decrease in TCS and PF accuracy as compared to One-Shot.**
| Language | Single Tool Calling | SinTC with Retrieval | Parallel Tool Calling | Avg |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ | PF $\uparrow$ |
| English |  |  |  |  |  |  |  |  |  |  |
| Zero-Shot | 100 | 94.37 | 68.31 | 91.06 | 59.22 | 52.51 | 86.55 | 58.18 | 51.64 | 73.53 |
| One-Shot | 100 | 94.37 | 68.31 | 89.39 | 77.65 | 66.48 | 84.13 | 80.13 | 68.67 | 81.01 |
| Difference ($\Delta$) | 0 | 0 | 0 | -1.67 | 18.43 | 13.97 | -2.42 | 21.95 | 17.03 | 7.47 |
| Hindi |  |  |  |  |  |  |  |  |  |  |
| Zero-Shot | 100 | 95.52 | 64.18 | 83.05 | 40.68 | 30.51 | 85.66 | 54.34 | 40.00 | 65.99 |
| One-Shot | 100 | 95.52 | 62.69 | 81.36 | 66.10 | 47.46 | 77.78 | 72.78 | 50.69 | 72.7 |
| Difference ($\Delta$) | 0 | 0 | -1.49 | -1.66 | 25.42 | 16.95 | -7.88 | 18.44 | 10.69 | 6.72 |

Finally, we provide plots related to safety evaluation in different ablation settings. Figure [4](https://arxiv.org/html/2510.07978v2#A4.F4) shows a comparison of model performance with and without refusal prompts, while Figure [5](https://arxiv.org/html/2510.07978v2#A4.F5) compares performance when hints are included in the input versus when they are absent.

Figure: Figure 4: Comparison of Model performance with and without refusal prompts for Safety tasks.
Refer to caption: resources/placeholder/refusal_plot_modified.png

Figure: Figure 5: Comparison of Model performance with and without hint in the queries for Safety tasks.
Refer to caption: resources/placeholder/hint_not_hint_plot.png

## Appendix E Indic Multilingual Results

In this section, we present a detailed analysis of the evaluation results on the Indian-context subset of VoiceAgentBench across five Indic languages: Bengali, Malayalam, Marathi, Tamil, and Telugu. As shown in [9](https://arxiv.org/html/2510.07978v2#A5.T9).

**Table 9: In-detail performance comparison on the Indian-context set for Indic Languages. Evaluation of models across Single Tool Calling (SinTC), SinTC with Retrieval, Parallel Tool Calling, and Sequential-Dependent Tool Calling (SeqDepTC) on Bengali, Malayalam, Marathi, Tamil and Telugu. Metrics include TS, TCS, and PF (see Section [3.3](https://arxiv.org/html/2510.07978v2#S3.SS3) for definitions).**
| Model | Single Tool Calling | SinTC with Retrieval | Parallel Tool Calling | SeqDep Tool Calling |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ | TS $\uparrow$ | TCS $\uparrow$ | PF $\uparrow$ |
| Bengali Subset |  |  |  |  |  |  |  |  |  |  |  |  |
| AudioFlamingo3 7B | 91.37 | 23.74 | 5.76 | 29.07 | 8.14 | 1.16 | 18.79 | 8.62 | 0.28 | 28.95 | 0 | 0 |
| KimiAudio 7B | 100.00 | 94.96 | 33.81 | 58.72 | 50.58 | 20.93 | 60.03 | 52.97 | 29.94 | 44.74 | 2.63 | 2.63 |
| Qwen2.5-Omni 7B | 100.00 | 1.44 | 0.00 | 45.35 | 0.00 | 0.00 | 31.36 | 0.00 | 0.00 | 18.42 | 2.63 | 0 |
| Whisperv3-Gemma3 27B | 99.28 | 93.53 | 47.48 | 83.14 | 66.86 | 37.21 | 79.80 | 77.12 | 48.31 | 33.33 | 2.78 | 2.78 |
| Whisperv3-Llama3 70B | 98.56 | 92.81 | 43.17 | 81.98 | 78.49 | 41.86 | 76.41 | 73.73 | 42.09 | 89.61 | 3.73 | 2.41 |
| Whisperv3-Qwen3 8B | 99.23 | 94.62 | 33.85 | 77.30 | 74.85 | 35.58 | 75.30 | 71.52 | 41.21 | 73.07 | 0 | 0 |
| Malayalam Subset |  |  |  |  |  |  |  |  |  |  |  |  |
| AudioFlamingo3 7B | 90.84 | 32.06 | 5.34 | 24.42 | 6.40 | 2.33 | 21.65 | 7.26 | 1.14 | 2.56 | 0 | 0 |
| KimiAudio 7B | 98.47 | 93.89 | 40.46 | 58.14 | 53.49 | 26.74 | 63.25 | 56.13 | 36.75 | 5.13 | 2.56 | 0 |
| Qwen2.5-Omni 7B | 93.13 | 0.76 | 0.00 | 36.05 | 0.00 | 0.00 | 20.80 | 0.00 | 0.00 | 2.56 | 0 | 0 |
| Whisperv3-Gemma3 27B | 98.47 | 91.60 | 35.11 | 65.70 | 62.79 | 33.14 | 55.13 | 52.14 | 30.91 | 5.26 | 0 | 0 |
| Whisperv3-Llama3 70B | 100.00 | 93.89 | 35.11 | 62.21 | 59.30 | 29.07 | 48.01 | 45.44 | 26.92 | 38.59 | 0 | 0 |
| Whisperv3-Qwen3 8B | 93.16 | 88.03 | 29.91 | 63.58 | 61.73 | 23.46 | 50.16 | 47.88 | 27.94 | 19.55 | 0 | 0 |
| Marathi Subset |  |  |  |  |  |  |  |  |  |  |  |  |
| AudioFlamingo3 7B | 92.03 | 27.54 | 8.70 | 25.88 | 10.00 | 2.35 | 20.25 | 8.54 | 2.20 | 40 | 0 | 0 |
| KimiAudio 7B | 98.55 | 93.48 | 40.58 | 62.94 | 50.59 | 27.65 | 65.01 | 61.02 | 38.29 | 30 | 2.50 | 0 |
| Qwen2.5-Omni 7B | 100.00 | 2.90 | 0.72 | 55.88 | 0.00 | 0.00 | 41.87 | 0.00 | 0.00 | 17.5 | 0 | 0 |
| Whisperv3-Gemma3 27B | 100.00 | 93.48 | 55.80 | 88.24 | 78.24 | 53.53 | 77.82 | 73.97 | 58.40 | 57.5 | 27.5 | 10 |
| Whisperv3-Llama3 70B | 99.28 | 93.48 | 55.80 | 87.65 | 84.12 | 58.24 | 76.58 | 72.31 | 54.68 | 93.75 | 10.42 | 6.67 |
| Whisperv3-Qwen3 8B | 98.45 | 93.02 | 48.84 | 90.42 | 86.83 | 53.29 | 77.59 | 73.99 | 54.45 | 69.67 | 0 | 0 |
| Tamil Subset |  |  |  |  |  |  |  |  |  |  |  |  |
| AudioFlamingo3 7B | 85.51 | 25.36 | 3.62 | 18.60 | 5.23 | 0.58 | 27.78 | 10.97 | 1.11 | 34.21 | 0 | 0 |
| KimiAudio 7B | 100.00 | 94.93 | 40.58 | 58.14 | 45.93 | 22.09 | 52.92 | 47.64 | 29.03 | 28.95 | 0 | 0 |
| Qwen2.5-Omni 7B | 93.48 | 2.17 | 0.72 | 34.30 | 0.00 | 0.00 | 8.47 | 0.00 | 0.00 | 18.67 | 0 | 0 |
| Whisperv3-Gemma3 27B | 100.00 | 94.93 | 60.14 | 89.53 | 85.47 | 58.72 | 84.31 | 81.25 | 61.81 | 25 | 2.78 | 2.78 |
| Whisperv3-Llama3 70B | 100.00 | 94.93 | 52.90 | 90.70 | 87.21 | 59.30 | 80.14 | 77.50 | 56.53 | 90.61 | 0.66 | 0.66 |
| Whisperv3-Qwen3 8B | 99.25 | 94.78 | 45.52 | 90.91 | 87.27 | 45.45 | 79.24 | 76.46 | 50.88 | 84.61 | 0.74 | 0.74 |
| Telugu Subset |  |  |  |  |  |  |  |  |  |  |  |  |
| AudioFlamingo3 7B | 92.31 | 30.00 | 2.31 | 20.86 | 7.98 | 1.84 | 29.34 | 10.54 | 1.14 | 30.77 | 0 | 0 |
| KimiAudio 7B | 100.00 | 93.85 | 46.15 | 71.17 | 52.76 | 30.67 | 60.54 | 51.71 | 37.18 | 35.90 | 5.13 | 2.56 |
| Qwen2.5-Omni 7B | 98.46 | 2.31 | 0.77 | 47.85 | 0.00 | 0.00 | 15.24 | 0.00 | 0.00 | 20.51 | 5.13 | 0 |
| Whisperv3-Gemma3 27B | 58.47 | 55.93 | 7.63 | 12.16 | 12.16 | 5.41 | 24.85 | 23.03 | 15.61 | 36.84 | 15.79 | 2.63 |
| Whisperv3-Llama3 70B | 100.00 | 94.62 | 58.46 | 91.41 | 87.12 | 61.35 | 29.55 | 27.73 | 13.48 | 96.15 | 3.42 | 2.78 |
| Whisperv3-Qwen3 8B | 99.19 | 93.55 | 48.39 | 86.27 | 83.01 | 47.06 | 24.54 | 21.30 | 13.73 | 94.06 | 3.13 | 3.13 |

##### ASR-LLM Setup Dominance.

The results demonstrate a clear architectural advantage for ASR-LLM setups over end-to-end SpeechLMs. The three Whisper-based pipeline models achieve an average performance of 65.9% across all metrics and all categories, compared to just 27.7% for SpeechLMs representing a substantial 138% improvement. WhisperV3-Qwen3 8B emerges as the top performer with an average score of 66.9%, followed closely by Whisperv3-Gemma3 27B (65.8%) and Whisperv3-Llama3 70B (64.9%). In contrast, the best Speech LM, KimiAudio 7B, only achieves 48.9%, while AudioFlamingo3 7B and Qwen2.5-Omni 7B severely underperform at 16.3% and 17.8% respectively. This dramatic performance gap suggests that existing SpeechLM models have been trained on limited Indic multilingual data.

##### SpeechLMs Fail Catastrophically as Task Complexity Increases.

Speech Language Models show severe degradation as tasks become complex. In simple Single Tool scenarios, AudioFlamingo3 7B scores 91%-92% TS, KimiAudio 7B reaches 98%-100 TS%, and Qwen2.5-Omni 7B maintains 85%-100 TS%. Adding retrieval causes significant drops: KimiAudio 7B falls to 50%-60% TCS while Qwen2.5-Omni 7B falls below 10%. Parallel Tool Calling triggers complete failures: Qwen2.5-Omni 7B scores 0% TCS across multiple languages, AudioFlamingo3 7B struggles below 10% TCS, and KimiAudio 7B shows inconsistent performance. Sequential-Dependent tasks represent total breakdown: Qwen2.5-Omni 7B collapses to 18.42% TS and 0% PF, while other models exhibit unpredictable patterns unsuitable for reliable multi-step reasoning applications.

##### Sequential-Dependent Tool Calling Reveals Lowest PF Scores Across All Models.

PF scores in Sequential-Dependent Tool Calling show the most challenging scenario for all models, with consistently low scores indicating difficulties in maintaining context across dependent operations. SpeechLMs perform poorly: Qwen2.5-Omni 7B exhibits complete breakdown with 0% PF accuracy across most languages (only 0.74% in Tamil), AudioFlamingo3 7B ranges from 0-2.78% PF (peaking in Bengali), and KimiAudio 7B shows variable performance with 0% PF in most languages but modest scores in Tamil (2.78%) and Telugu (2.56%). Even ASR-LLM models struggle: WhisperV3-Qwen3 8B achieves the highest individual score of 10% PF in Marathi but drops to 0% in Bengali and Malayalam, while Whisperv3-Gemma3 27B and Whisperv3-Llama3 70B maintain modest ranges of 2.63-6.67% PF. This shows that both the types of models are not ready for this task.

## Appendix F Time Taken for First Token (TTFT) Generation: SpeechLM vs ASR-LLM

Traditional ASR-LLM setups typically adopt a two-stage pipeline in which an ASR model first transcribes the input speech, and the resulting text is subsequently processed by an LLM. While this modular design offers flexibility and ease of component substitution, it introduces additional computational overhead, resulting in substantially higher time-to-first-token (TTFT). In contrast, SpeechLMs employ end-to-end architectures that generate responses directly from speech, bypassing the intermediate transcription step and thereby reducing latency. Empirical measurements highlight this difference: When measured with a set of 100 queries of average duration 3.5 seconds, Qwen2.5-Omni 7B achieves a 90th percentile (p90) TTFT of approximately 40 ms on a single H100 GPU, whereas a pipeline combining Whisper-large-v3 with Qwen3 8B exhibits a p90 TTFT of around 800 ms under the same hardware conditions. This contrast underscores a fundamental trade-off: while ASR-LLM pipelines offer modularity and adaptability, their elevated latency constrains real-time deployment. In comparison, SpeechLMs are particularly well-suited for interactive speech systems and low-latency audio understanding tasks, where rapid response generation is critical.

## Appendix G Evaluation Framework Implementation

We detail the implementation of our evaluation framework along three dimensions: (i) Tool Selection, (ii) Tool Call Structure, and (iii) Parameter Filling. Each dimension is designed to assess model performance in a progressively layered manner. To illustrate these metrics, we also provide representative examples from our evaluation framework.

### G.1 Tool Selection

In tool selection, we evaluate whether the predicted function name exactly matches the gold reference. The resulting metric, function selection accuracy, reflects the model’s ability to identify the correct tool in scenarios where multiple APIs are available but only a subset is relevant. This also evaluate against hallucination of tool in model’s response even for Single Tool Calling tasks.

Here’s an example for Tool Selection Failure:

### G.2 Tool Call Structure

For each function, we automatically construct a Pydantic model from its JSON schema. The model’s predicted Python function call is first preprocessed to ensure it can be parsed; if this step fails, the output is flagged as a syntax mismatch. Otherwise, the preprocessed call is passed through the corresponding Pydantic model. Successful parsing indicates that the call is schema-adherent, which we measure as schema accuracy. If validation fails, it signals a structural mismatch, such as missing required fields, incorrect data types, or extraneous parameters.

Using Pydantic provides two main benefits: (i) strict enforcement of type and structural constraints, and (ii) reproducible, automated validation without relying on subjective or ad hoc checks.

Here’s an example for schema failure using Pydantic:

### G.3 Parameter Filling

Exact string matching is too rigid for parameter filling validation, since equivalent arguments may be expressed differently (e.g., “Connaught Place” vs. “CP, Delhi”) depending on the tool. To capture semantic correctness, we use a LLM as a judge. GPT-4o-mini is prompted with the query, gold answer, and predicted response, and asked to first reason step by step about whether the prediction aligns with the gold intent. After reasoning, it must return a binary judgment (correct/incorrect) on parameter fidelity. This design reduces spurious errors by ensuring the model grounds its verdict in explicit reasoning before committing to a score. We detail the meta judge prompt in Appendix [J.2](https://arxiv.org/html/2510.07978v2#A10.SS2).

Here’s an example for Parameter Filling Failure:

## Appendix H Custom Agent Tools

### H.1 Cab Agent

### H.2 Food Agent

### H.3 Payment Agent

## Appendix I VoiceAgentBench Examples

### I.1 Examples of Single, Single with Retrieval and Parallel Tool Calling

### I.2 Examples of Sequential Dependent Tool Calling

### I.3 Example of Multi Turn Dialogues Tool Calling

### I.4 Examples of Safety Evaluation

## Appendix J Prompts

### J.1 Model Instructions

### J.2 LLM as a Judge

## Appendix K Some Indic Multilingual Examples

## Appendix L Use of Large Language Models (LLMs)

### G.2 Tool Call Structure

For each function, we automatically construct a Pydantic model from its JSON schema. The model’s predicted Python function call is first preprocessed to ensure it can be parsed; if this step fails, the output is flagged as a syntax mismatch. Otherwise, the preprocessed call is passed through the corresponding Pydantic model. Successful parsing indicates that the call is schema-adherent, which we measure as schema accuracy. If validation fails, it signals a structural mismatch, such as missing required fields, incorrect data types, or extraneous parameters.

Using Pydantic provides two main benefits: (i) strict enforcement of type and structural constraints, and (ii) reproducible, automated validation without relying on subjective or ad hoc checks.

Here’s an example for schema failure using Pydantic:

### G.3 Parameter Filling

Exact string matching is too rigid for parameter filling validation, since equivalent arguments may be expressed differently (e.g., “Connaught Place” vs. “CP, Delhi”) depending on the tool. To capture semantic correctness, we use a LLM as a judge. GPT-4o-mini is prompted with the query, gold answer, and predicted response, and asked to first reason step by step about whether the prediction aligns with the gold intent. After reasoning, it must return a binary judgment (correct/incorrect) on parameter fidelity. This design reduces spurious errors by ensuring the model grounds its verdict in explicit reasoning before committing to a score. We detail the meta judge prompt in Appendix [J.2](https://arxiv.org/html/2510.07978v2#A10.SS2).

Here’s an example for Parameter Filling Failure:

## Appendix H Custom Agent Tools

### H.1 Cab Agent

### H.2 Food Agent

### H.3 Payment Agent

## Appendix I VoiceAgentBench Examples

### I.1 Examples of Single, Single with Retrieval and Parallel Tool Calling

### I.2 Examples of Sequential Dependent Tool Calling

### I.3 Example of Multi Turn Dialogues Tool Calling

### I.4 Examples of Safety Evaluation

## Appendix J Prompts

### J.1 Model Instructions

### J.2 LLM as a Judge

## Appendix K Some Indic Multilingual Examples

## Appendix L Use of Large Language Models (LLMs)

### G.3 Parameter Filling

Exact string matching is too rigid for parameter filling validation, since equivalent arguments may be expressed differently (e.g., “Connaught Place” vs. “CP, Delhi”) depending on the tool. To capture semantic correctness, we use a LLM as a judge. GPT-4o-mini is prompted with the query, gold answer, and predicted response, and asked to first reason step by step about whether the prediction aligns with the gold intent. After reasoning, it must return a binary judgment (correct/incorrect) on parameter fidelity. This design reduces spurious errors by ensuring the model grounds its verdict in explicit reasoning before committing to a score. We detail the meta judge prompt in Appendix [J.2](https://arxiv.org/html/2510.07978v2#A10.SS2).

Here’s an example for Parameter Filling Failure:

## Appendix H Custom Agent Tools

### H.1 Cab Agent

### H.2 Food Agent

### H.3 Payment Agent

## Appendix I VoiceAgentBench Examples

### I.1 Examples of Single, Single with Retrieval and Parallel Tool Calling

### I.2 Examples of Sequential Dependent Tool Calling

### I.3 Example of Multi Turn Dialogues Tool Calling

### I.4 Examples of Safety Evaluation

## Appendix J Prompts

### J.1 Model Instructions

### J.2 LLM as a Judge

## Appendix K Some Indic Multilingual Examples

## Appendix L Use of Large Language Models (LLMs)

## Appendix H Custom Agent Tools

### H.1 Cab Agent

### H.2 Food Agent

### H.3 Payment Agent

## Appendix I VoiceAgentBench Examples

### I.1 Examples of Single, Single with Retrieval and Parallel Tool Calling

### I.2 Examples of Sequential Dependent Tool Calling

### I.3 Example of Multi Turn Dialogues Tool Calling

### I.4 Examples of Safety Evaluation

## Appendix J Prompts

### J.1 Model Instructions

### J.2 LLM as a Judge

## Appendix K Some Indic Multilingual Examples

## Appendix L Use of Large Language Models (LLMs)

### H.2 Food Agent

### H.3 Payment Agent

## Appendix I VoiceAgentBench Examples

### I.1 Examples of Single, Single with Retrieval and Parallel Tool Calling

### I.2 Examples of Sequential Dependent Tool Calling

### I.3 Example of Multi Turn Dialogues Tool Calling

### I.4 Examples of Safety Evaluation

## Appendix J Prompts

### J.1 Model Instructions

### J.2 LLM as a Judge

## Appendix K Some Indic Multilingual Examples

## Appendix L Use of Large Language Models (LLMs)

### H.3 Payment Agent

## Appendix I VoiceAgentBench Examples

### I.1 Examples of Single, Single with Retrieval and Parallel Tool Calling

### I.2 Examples of Sequential Dependent Tool Calling

### I.3 Example of Multi Turn Dialogues Tool Calling

### I.4 Examples of Safety Evaluation

## Appendix J Prompts

### J.1 Model Instructions

### J.2 LLM as a Judge

## Appendix K Some Indic Multilingual Examples

## Appendix L Use of Large Language Models (LLMs)

## Appendix I VoiceAgentBench Examples

### I.1 Examples of Single, Single with Retrieval and Parallel Tool Calling

### I.2 Examples of Sequential Dependent Tool Calling

### I.3 Example of Multi Turn Dialogues Tool Calling

### I.4 Examples of Safety Evaluation

## Appendix J Prompts

### J.1 Model Instructions

### J.2 LLM as a Judge

## Appendix K Some Indic Multilingual Examples

## Appendix L Use of Large Language Models (LLMs)

### I.2 Examples of Sequential Dependent Tool Calling

### I.3 Example of Multi Turn Dialogues Tool Calling

### I.4 Examples of Safety Evaluation

## Appendix J Prompts

### J.1 Model Instructions

### J.2 LLM as a Judge

## Appendix K Some Indic Multilingual Examples

## Appendix L Use of Large Language Models (LLMs)

### I.3 Example of Multi Turn Dialogues Tool Calling

### I.4 Examples of Safety Evaluation

## Appendix J Prompts

### J.1 Model Instructions

### J.2 LLM as a Judge

## Appendix K Some Indic Multilingual Examples

## Appendix L Use of Large Language Models (LLMs)

### I.4 Examples of Safety Evaluation

## Appendix J Prompts

### J.1 Model Instructions

### J.2 LLM as a Judge

## Appendix K Some Indic Multilingual Examples

## Appendix L Use of Large Language Models (LLMs)

## Appendix J Prompts

### J.1 Model Instructions

### J.2 LLM as a Judge

## Appendix K Some Indic Multilingual Examples

## Appendix L Use of Large Language Models (LLMs)

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Has any portion of the GPT-generated ground truth been validated by humans?
- What is the agreement between human judges and GPT-4o-mini in evaluation?
- Why were multi-turn tasks excluded for Indic languages, and will they be added?
- Do you plan to include real or noisy speech in future versions of the benchmark?
- Can you provide per-language breakdowns of failure types?
soundness: 3
strengths: 1. Novel and timely: first benchmark for speech-based agentic reasoning and tool orchestration.  
2. Comprehensive task design: Includes sequential tool use, multi-tool planning, multi-turn dialogue, and adversarial safety tasks.  
3. Multilingual and culturally grounded: covers 7 languages, with 30% Indian-context queries, which has been rare in previous work.  
4. Acoustic diversity: uses Farthest Point Sampling (FPS) over TTS speaker embeddings to maximize voice variation.  
5. Granular evaluation metrics. Tool Selection (TS), Tool Call Structure (TCS), Parameter Filling (PF), and Refusal Rate (RR) allow fine-grained diagnostics.  
6. Extensive experiments and ablations, including error propagation from ASR, prompting strategies, and safety prompting.
summary: This paper introduces VoiceAgentBench (VAB), the first benchmark designed to evaluate agentic speech-based systems---i.e., voice assistants that can reason, call tools, and handle multi-step workflows. The benchmark includes 5,500+ synthetic spoken queries in 7 languages (English + 6 Indic), covering:

- Single-tool and multi-tool usage  
- Sequential dependent workflows  
- Multi-turn interactions  
- Safety and adversarial robustness  

Six models (SpeechLMs/ASR to LLM pipelines) are evaluated. Results highlight significant weaknesses in multilingual performance, tool reasoning, and safety alignment.
weaknesses: 1. All speech is synthetic (TTS), which limits realism and excludes natural speech variability, noise, and disfluencies.
2. Ground truth for sequential tool workflows is generated using GPT-4o-mini rather than human annotation, which may introduce hallucinations or inaccuracies.
3. Evaluation relies on GPT-4o-mini as a judge for parameter accuracy without human agreement studies or reproducibility checks.
4. Multi-turn dialogue is only evaluated in English and entirely missing for Indic languages.
5. Safety evaluation treats refusal as a binary outcome. And also does not seem to assess partial compliance, harmful outputs, or nuanced behavior.
6. No statistical significance testing, confidence intervals, or variance reporting; dataset construction and filtering steps lack transparency.

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 2
questions: It would be highly beneficial for the authors to include a more comprehensive set of examples or case studies from the dataset. This would provide greater transparency and a clearer understanding of the benchmark's tasks and data quality.
soundness: 2
strengths: 1. Voiceagentbench represents a timely and pioneering effort, being one of the first benchmarks dedicated to evaluating the agent capabilities of speech-based LLMs.

2. Although the methodology for dataset creation is not highly innovative, the resulting dataset, pipeline, and evaluation framework provide a contribution that will be useful to the research community.
summary: This work introduces Voiceagentbench, a new benchmark specifically designed to evaluate the agent-based (speech-to-text) functionalities of Speech LLMs. The benchmark comprises over 5,000 synthetic spoken queries, including dialogues grounded in Indian contexts, and is structured to cover single-tool invocations, multi-tool workflows, multi-turn interactions, and safety evaluations. The dataset construction follows a conventional pipeline: text prompts are sourced from existing text-based LLM agent benchmarks and then converted to speech via TTS synthesis. The authors evaluate several prominent models, including qwen2.5-omni, kimi-audio, and audioFalmingo3, alongside various cascaded speech-to-text LLM systems. The results indicate that, with the exception of kimi-audio, the other end-to-end models (qwen2.5-omni, audioFalmingo3) exhibit significant performance degradation on this benchmark.
weaknesses: 1. The dataset is heavily skewed towards Indian contexts, in both its textual content and synthesized audio. This poses a significant domain mismatch for mainstream models, which are predominantly trained on general English data. This specialization limits the benchmark's broader applicability and generalizability.
2.  Authors need to clarify the relationship between their work and existing research on voice agents. A more thorough discussion is required to situate this benchmark relative to recent advancements, such as[1][2][3][4]:
3. The benchmark's design appears to primarily target cascaded speech-to-text LLMs, rather than being optimized for evaluating true end-to-end speech agent models.
4. The exclusive use of synthetic data introduces a potential gap between the benchmark's evaluation scenarios and the complexities of real-world, spontaneous human-machine conversation.
5. Many of the conclusions drawn from the experiments offer limited insight. For instance, the poor performance of qwen2.5-omni (often scoring 0) is likely attributable to a simple lack of agent-specific training data, rather than a deeper flaw in the model's architecture. Such conclusions do not foster deeper investigation into the core challenges of speech agents.
6. The overall pipeline—dataset creation, model selection, and evaluation—is conventional, albeit representing a non-trivial amount of work. The benchmark overlooks several critical dimensions of agent performance that warrant further exploration, such as: The latency introduced by agent function calls. The potential degradation of the base model's core abilities when agent functionalities are integrated. The evaluation of complex, multi-agent collaborative scenarios.




[1]. Process-Supervised Reinforcement Learning for Interactive Multimodal Tool-Use Agents

[2]. Stream RAG: Instant and Accurate Spoken Dialogue Systems with Streaming Tool Usage

[3]. GPT-RealTime

[4]. Step-Audio 2 Technical Report

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 3
questions: Could the authors clarify the fundamental differences between the proposed speech agent benchmark and existing text agent benchmarks where queries are simply converted to speech using TTS? Specifically, what unique challenges or evaluation aspects are addressed by your benchmark that cannot be captured by applying TTS to text-based agent benchmarks?
soundness: 3
strengths: 1. The proposed evaluation framework is practical and effectively measures agentic capabilities of SpeechLMs in realistic scenarios.

2. The experiments are thorough, covering multiple languages, diverse tasks, and adversarial robustness, with clear and convincing results.

3. The paper is well-structured and clearly written, making the methodology and findings easy to understand.
summary: This paper introduces VoiceAgentBench, a novel and comprehensive benchmark designed to systematically evaluate the agentic capabilities of Speech Language Models (SpeechLMs) in realistic spoken interaction scenarios. Unlike existing benchmarks that focus on isolated tasks such as transcription or question answering, VoiceAgentBench assesses a wide range of agentic behaviors, including single and multi-tool invocation, multi-turn dialogue, and safety/adversarial robustness. The benchmark consists of over 5,500 synthetic spoken queries generated via TTS, covering English, Hindi, and five other Indic languages, with a significant portion contextualized in Indian cultural scenarios. The authors propose a diversity-based sampling method for TTS voice conversion to ensure coverage across accents and speaker characteristics, and they define a layered evaluation framework to measure tool selection, structural consistency, parameter filling, and refusal of unsafe requests. Overall, the work fills a critical gap in the evaluation of SpeechLMs for real-world voice assistant applications.
weaknesses: 1. As a benchmark, the evaluation in this paper is not sufficiently comprehensive; it mainly focuses on agent scenarios in the Indian context, which limits the generality of the assessment.

2. The paper’s originality is somewhat lacking. In the benchmark design, the authors do not clearly explain the fundamental differences between speech agents benchmark and text agents benchmark, nor the necessity of a dedicated speech agent benchmark. The three speech scenarios described seem to be subsets of text agent scenarios. Regarding the claimed innovation of diversity-based TTS generation, it is also unclear whether this approach is necessary, or if using a rich set of audio resources with a zero-shot TTS strategy could achieve similar results.

3. The presentation of the paper could be improved. For the many audio-related cases, a demo page may be a better choice than displaying cases across multiple pages.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: - Why was GPT-4o-mini chosen as the sole judge for PF/RR? Did you try alternative judges? Any evidence of judge–model bias?
- In ASR→LLM pipelines, did you test stronger Indic ASR? Can you quantify how much of the Indic gap remains after substituting better ASR models? This will provide better insights than directly using ground-truth transcriptions.
- Can you provide details in the one-shot settings?
soundness: 2
strengths: - The topic is quite timely in the era of AI agents.
- Well-scoped problem & broad coverage.
- Layered evaluation design. The split into Tool Selection, Tool-Call Structure, Parameter-Filling, and Refusal Rate gives interpretable failure modes, instead of a single pass/fail.
summary: This paper introduces VoiceAgentBench, a new benchmark for SpeechLLM agents. The benchmark covers various tool-calling scenarios and safety evaluations. It also covers several languages beyond English. The evaluation reveals significant limitations of current SpeechLLMs as agents, providing some insights into this direction.
weaknesses: - Synthetic speech only: Whether the synthesized speech is of good quality is unknown. There is also no consideration of real-world noisy environments.
- Lacking details of LLM-judge, such as decoding strategies and evaluation qualtiy (compared with human annotation).
- Metric design choices may bias outcomes. Tool Selection is an exact-match on function name; Tool-Call Structure hinges on Pydantic schema validation. These can over-penalize semantically correct outputs with minor format variance, while PF then depends on an LLM judge. A fuller error taxonomy or tolerance bands (e.g., argument similarity) beyond the judge would strengthen conclusions.
- The coverage of the evaluated models is quite restricted. The paper should include more models for a more comprehensive overview.
- No confidence intervals or significance tests for the results.


Decision: Reject