Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Models
Authors: Hanin atwany, Abdul Waheed, Bhiksha Raj, Rita Singh, Monojit Choudhury
Accepted at: acl25

Lost in Transcription, Found in Distribution Shift: Demystifying Hallucination in Speech Foundation Models

Hanin Atwany2*  Abdul Waheed1*  Rita Singh1  Monojit Choudhury2  Bhiksha Raj1
1Carnegie Mellon University   2MBZUAI

Abstract
Speech foundation models trained at a massive scale, both in terms of model and data size, result in robust systems capable of performing multiple speech tasks, including automatic speech recognition (ASR). These models transcend language and domain barriers, yet effectively measuring their performance remains a challenge. Traditional metrics like word error rate (WER) and character error rate (CER) are commonly used to evaluate ASR performance but often fail to reflect transcription quality in critical contexts, particularly when detecting fabricated outputs. This phenomenon, known as hallucination, is especially concerning in high-stakes domains such as healthcare, legal, and aviation, where errors can have severe consequences. In our work, we address this gap by investigating hallucination in ASR models. We examine how factors such as distribution shifts, model size, and model architecture influence the hallucination error rate (HER), a metric we introduce to quantify hallucinations. Our analysis of over 20 ASR models reveals three key insights: (1) High WERs can mask low hallucination rates, while low WERs may conceal dangerous hallucinations. (2) Synthetic noise, both adversarial and common perturbations like white noise, pitch shift, and time stretching, increase HER. (3) Distribution shift correlates strongly with HER (\( \alpha = 0.91 \)). Our findings highlight the importance of incorporating HER alongside traditional metrics like WER to better assess ASR model performance, particularly in high-stakes domains.

1 Introduction
Automatic Speech Recognition (ASR) systems have become fundamental to various applications, including personal assistants, automated customer service, and transcription tools used in fields such as education, healthcare, and law (Zhang et al., 2023; Adedeji et al., 2024). These systems have seen remarkable improvements in recent years (Ariaga et al., 2024; Radford et al., 2022; Communication et al., 2023), with state-of-the-art models demonstrating their capabilities across diverse datasets and languages (Shakhadri et al., 2025). However, the evaluation of ASR performance remains largely dependent on word and character error rate (WER/CER). The primary limitation of WER and CER is their dependence on token-level overlapping, which focuses on matching individual words or characters without considering the overall semantic aspect of the transcription. This could result in misleading evaluations, as a high WER/CER does not necessarily indicate poor outputs in all cases. In addition, these metrics fall short in capturing more subtle semantic failures which are not typically caught without human verification, such as hallucinations.

Hallucinations in ASR systems mirror perceptual experiences in neuroscience—plausible outputs generated without grounding in input stimuli (American Psychiatric Association et al., 2013; Zmigrod et al., 2016), deviating phonetically or semantically from source speech (Ji et al., 2023). Like natural neural perceptions, ASR hallucinations arise when models prioritize distributional patterns over fidelity to audio input, fabricating text unlinked to reference content (Hare, 2021). These errors are uniquely hazardous in high-stakes domains (Williamson and Prybutok, 2024), as they evade WER/CER detection while distorting meaning, similar to how clinical hallucinations disconnect from reality.

Hallucination in domains such as medical and legal can have serious consequences, including life-threatening outcomes and distorted testimonies or contracts, and may disproportionately affect marginalized groups (Xie et al., 2023; Vishwanath et al., 2024; Mujtaba et al., 2024; Sperber et al., 2020; Koenecke et al., 2024a). Hence, detecting
and mitigating hallucination is crucial for ensuring the reliability of ASR systems in sensitive environments.

The existing literature on hallucination detection in ASR models is confined to a single model (Frieske and Shi, 2024; Serai et al., 2021; Koenec et al., 2024a; Barański et al., 2025; Ji et al., 2023) or test setting (Kim et al., 2024a), highlighting a significant research gap for a systematic investigation across different supervision paradigms, test domains, and conditions.

In this work, we address this critical research gap and make the following key contributions:

• We conduct a thorough evaluation of ASR models across various setups, consisting of both synthetic and natural shifts from training to test distributions.

• We introduce an LLM-based error detection framework that classifies ASR outputs into different types of errors including hallucination errors through context-aware assessments.

• We provide an in-depth analysis of hallucination phenomena in ASR models, exploring the impact of domain-specific data, model architectures, and training paradigms, and offer valuable insights into the relationship between model size, type, and hallucination frequency.

• To validate our hallucination detection method, we compare our LLM-based hallucination detection pipeline with human evaluations and heuristic approaches, demonstrating that the LLM evaluation closely aligns with human judgments and other LLM-based assessments, unlike the heuristic-based approach.

The significance of this work lies in how it redefines the evaluation and improvement of ASR systems. By emphasizing hallucination detection, we aim to enhance the reliability of ASR models, particularly in domains where accuracy and precision are non-negotiable.

Outline. In Section 2, we review prior work related to ASR evaluation and hallucination detection. Section 3 outlines our proposed methodology. Section 4 presents our experimental setup. Finally, Section 5 discusses the results and implications of our findings, outlining directions for future research. We conclude our work in Section 6 and provide limitations in Section 7.

2 Related Work

The use of ASR systems in high-stakes domains, including healthcare (Afonja et al., 2024; Huh et al., 2023; Adedeji et al., 2024; Sunder et al., 2022), legal proceedings (Saadany et al., 2022; Garneau and Bolduc, 2024), and finance (Del Rio et al., 2021; Liao et al., 2023), has heightened the necessity for ensuring their robustness. Conventionally, the performance of these models is assessed using metrics such as WER and CER (Serai et al., 2022). Szymański et al. (2023); Sasindran et al. (2024) They show that when these metrics are used in isolation, they exhibit notable limitations.

Recent studies have extensively investigated hallucination in text generated by large language models (LLMs), identifying it as a prevalent phenomenon (Huang et al., 2025; Bai et al., 2024; Yao et al., 2023; Jiang et al., 2024; Maynez et al., 2020; Parikh et al., 2020; Ji et al., 2023; Mittal et al., 2024; Filippova, 2020). This issue has also been observed in audio foundation models (Sahoo et al., 2024). Furthermore, research suggests that pre-training language models for predictive accuracy inherently predisposes them to hallucination, even under ideal conditions where the training data is entirely factual (Kalai and Vempala, 2024).

However, few studies explore hallucination evaluation and detection in automatic speech recognition (ASR) systems, with most research focusing on Whisper, a semi-supervised model. For instance, Koenec et al. (2024b) analyzes the Aphasia dataset and reports that while Whisper’s overall hallucination rate is 1%, 40% of these hallucinations contain violent or harmful content. Similarly, Kim et al. (2024a) demonstrates that Whisper hallucinates at significantly higher rates under low signal-to-noise ratio (SNR) conditions, and observes a 20% increase in hallucinations at -4 dB and -2 dB SNRs. Prior work by Serai et al. (2021) proposes augmenting models with hallucinated transcripts to improve performance, while Frieske and Shi (2024) introduces a perturbation-based evaluation method using automatic metrics such as word error rate (WER), perplexity, and cosine similarity. Barański et al. (2025) develops a filtered Bag of Hallucinations (BoH) for detection, and reveals that hallucinations in Whisper correlate strongly with training data biases (e.g., phrases like “Thank you for watching” linked to YouTube content).

Despite these advances, existing studies remain
<table>
  <tr>
    <th>Model</th>
    <th>Reference</th>
    <th>Hypothesis</th>
    <th>WER</th>
    <th colspan="3">Hallucination</th>
    <th>Setting</th>
  </tr>
  <tr>
    <th></th>
    <th></th>
    <th></th>
    <th></th>
    <th>Heuristic</th>
    <th>Human</th>
    <th>LLM</th>
    <th></th>
  </tr>
  <tr>
    <td>whisper-small<br>hf-seamless-m4t-large</td>
    <td>hungry action hippos fruit probably i i had asthma</td>
    <td>Humm reaction in hippos fruit? The song is Erasmus.</td>
    <td>75<br>100</td>
    <td>F<br>F</td>
    <td>F<br>T</td>
    <td>F<br>T</td>
    <td>Home Environment<br>Primock57 (medical)</td>
  </tr>
  <tr>
    <td>hf-seamless-m4t-large</td>
    <td>it can’t be done</td>
    <td>I’m going to start with the first one.</td>
    <td>200</td>
    <td>F</td>
    <td>T</td>
    <td>T</td>
    <td>Superemecourt (Legal)</td>
  </tr>
  <tr>
    <td>whisper-medium</td>
    <td>patel para thirty eight page three hundred and fifty five</td>
    <td>How much is the tail?</td>
    <td>100</td>
    <td>F</td>
    <td>T</td>
    <td>T</td>
    <td>Legal</td>
  </tr>
  <tr>
    <td>hubert</td>
    <td>god i’m forty five</td>
    <td>god he whisper i’m forty five</td>
    <td>40</td>
    <td>F</td>
    <td>T</td>
    <td>T</td>
    <td>Primock57 (medical)</td>
  </tr>
  <tr>
    <td>hubert</td>
    <td>yeah</td>
    <td>in a handsome coat</td>
    <td>400</td>
    <td>F</td>
    <td>T</td>
    <td>T</td>
    <td>AMI (meetings)</td>
  </tr>
  <tr>
    <td>wav2vec</td>
    <td>today yeah</td>
    <td>she did yes</td>
    <td>150</td>
    <td>F</td>
    <td>T</td>
    <td>T</td>
    <td>Primock57 (medical)</td>
  </tr>
  <tr>
    <td>wav2vec2</td>
    <td>another dog secretary show</td>
    <td>i love her dog’s secretary show</td>
    <td>100</td>
    <td>F</td>
    <td>T</td>
    <td>T</td>
    <td>BERSt (Noisy)</td>
  </tr>
</table>

Table 1: Examples showing hallucination detection by different methods across domains and models.

limited in scope, focusing predominantly on semi-supervised models like Whisper. This highlights a critical gap: the lack of a comprehensive understanding of hallucinations across the full spectrum of supervision paradigms—from supervised to unsupervised models—and under domain shifts where test data distributions diverge sharply from training environments. Addressing these gaps is essential for developing robust ASR systems that maintain accuracy and faithfulness in diverse real-world applications, a challenge our work directly tackles by evaluating a wide range of models with diverse architectures, sizes, and training paradigms on synthetic and natural shifts.

3 Methodology

We examine transcription quality using the standard metrics WER and CER, and we assess the occurrence of hallucination errors to provide a comprehensive view of model performance. Our testing environment is characterized by both natural and synthetic distribution shifts. Furthermore, we investigate the deterioration of error rates when transitioning from a controlled source domain (LibriSpeech) to various target domains, with a particular focus on both WER and the Hallucination Error Rate (HER). In addition, we introduce noise to the input data and analyze its effect on error rate degradation. This multifaceted strategy offers valuable insights into the challenges encountered by ASR systems in real-world scenarios. We provide additional details about each step subsequently.

3.1 ASR Evaluation

We evaluate a broad range of ASR models under a zero-shot setting, using the default decoding parameters for each model. Standard preprocessing steps are applied prior to calculating the metrics, ensuring consistency in evaluation.

3.2 Hallucination Evaluation

In addition to conventional transcription errors, we also assess hallucination errors by using an LLM-based pipeline that classifies the errors produced by the ASR model. Specifically, we use GPT-4o mini to compare the ground truth transcription with the model outputs and ask the LLM to categorize them into different error types.

We conduct hallucination evaluation at two levels:

Coarse-grained: The model categorizes the output into one of three classes: Hallucination Error, Non-Hallucination, or No Error. For this evaluation, we provide two examples per category in the prompt.

Fine-grained: The model is asked to further refine the categorization by identifying specific error types, such as Hallucination Error, Language Error, Oscillation Error, Phonetic Error, and No Error. In this case, one example per category is provided in the prompt.

The prompts used for both coarse-grained and fine-grained evaluations are detailed in Appendix A.3 (Figure 5 and Figure 6, respectively). To quantify hallucination occurrences, we introduce the HER, defined as the ratio of hallucination errors to the total number of examples in the data.

3.3 Distribution Shifts and Quantification

We systematically evaluate ASR models under a variety of testing conditions, ranging from naturally occurring domain variations to scenarios involving both adversarial and non-adversarial perturbations. These conditions are designed to induce either natural or synthetic distribution shifts in the input speech.

Natural Shifts. These shifts arise from inherent
variations in data distributions, such as differences in accents, domain-specific content, background noise, and diverse speaking styles (Liu et al., 2021).
Synthetic Shifts. These shifts are artificially induced, encompassing simulated noise, goal-specific perturbations, and adversarial attacks (Fan et al., 2022). We design a comprehensive testing setup to identify the conditions under which ASR models are prone to hallucinate. Additional details about the datasets used in this study are provided in Section 4.1 and summarized in Table 10. We measure the extent of domain distribution shifts using the high-order metric Central Moment Discrepancy (CMD) (Zellinger et al., 2019; Kashyap et al., 2020), which assesses the discrepancy between two distributions. It is calculated as follows:

\[
\mathrm{CMD} = \frac{1}{L} \sum_{l=1}^{L} \| \mathbb{E}[h_l^s] - \mathbb{E}[h_l^t] \|_2^2,
\]

where \( L \) represents the number of layers, \( h_l^s \) and \( h_l^t \) denote the hidden representations for the source and target domains, respectively, and \( \mathbb{E}[\cdot] \) signifies the expectation.

3.4 Error Rate Degradation

Error Rate Degradation quantifies the decline in ASR performance when transitioning from a source domain (LibriSpeech) to a target domain, with degradation measured in two aspects: transcription errors and hallucination errors.

Word Error Rate Degradation (WERD). WERD is defined as the difference in WER between the target and source domains:

\[
\mathrm{WERD} = \mathrm{WER}_{\text{target}} - \mathrm{WER}_{\text{source}},
\]

where \( \mathrm{WER}_{\text{source}} \) and \( \mathrm{WER}_{\text{target}} \) denote the WER for the source and target domains, respectively.

Hallucination Error Rate Degradation (HERD).

HERD captures the increase in hallucination errors when moving from the source to the target domain:

\[
\mathrm{HERD} = \mathrm{HER}_{\text{target}} - \mathrm{HER}_{\text{source}},
\]

where \( \mathrm{HER}_{\text{source}} \) and \( \mathrm{HER}_{\text{target}} \) represent the hallucination error rates in the source and target domains, respectively.

Furthermore, the relationship between CMD and degradation rates is analyzed and visualized (see Figure 7) to understand how domain variations correlate with both transcription and hallucination errors.

4 Experiments

In this section, we present the experimental details of our work. We examine the models’ tendencies to produce hallucinated outputs, using LLM-based evaluation as described in 3. The results provide insights into the reliability of different ASR systems across various real-world and adversarial conditions.

4.1 Datasets

In our experiments, we use a diverse set of datasets representing various domains and testing conditions to evaluate the ASR systems under scenarios that differ from training data. To achieve this, we choose datasets from domains with a high likelihood of being unseen during training, ensuring a natural distributional shift. Additionally, we include datasets with synthetic perturbations, such as adversarial attacks, and common augmentation techniques like Gaussian noise addition, pitch shifting, and time stretching, to assess the model’s robustness under synthetic shift.

Domain Specific Datasets. To evaluate model performance under real-world conditions, we leverage datasets from diverse domains, ensuring a comprehensive assessment across various settings. These include legal proceedings: Supreme-Court-Speech¹, medical dialogues: Primock57 (Papadopoulos Korfiatis et al., 2022), meeting conversations: AMI (Consortium, n.d.), aviation communications: ATCOSim (Hofbauer et al., 2008), conversational speech: SLUE-VoxCeleb (Shon et al., 2022), home environments: BERS²,

¹https://huggingface.co/datasets/janaab/supreme-court-speech
²https://huggingface.co/datasets/macabdu19/BERS
and general speech corpora: LibriSpeech (Panayotov et al., 2015), GLOBE (Wang et al., 2024), and SPGHISpeech (Technologies, 2021), including noisy conditions (LibriSpeech_test_noise (Panayotov et al., 2015)). These datasets span a wide range of accents, recording conditions, and environments—from high-quality audiobooks to teleconferences and real-time simulations. This diversity ensures a robust evaluation of model performance across realistic and challenging scenarios, addressing the limitations of single-domain evaluations.

Perturbed Datasets. To simulate challenging acoustic conditions and evaluate WER and HER under adversarial scenarios, we apply various types of synthetic perturbations to speech inputs. These include an adversarial dataset featuring modified utterances with adversarial noise at varying radii (0.04 and 0.015) and Room Impulse Response (RIR) noise, primarily aimed at adversarially attacking the ASR models (Olivier and Raj, 2022). Additionally, we evaluate model robustness under challenging conditions by applying a range of general audio perturbations—including noise addition, time stretching, pitch shift, cross-lingual noise, distortion, echo, pub noise, and reverberation—to 1,000 randomly sampled audio clips from a mixture of domain-specific datasets. These perturbations are commonly used as augmentation techniques to simulate real-world variability and stress-test the models. We compare the performance on perturbed speech with that of non-perturbed speech to quantify the impact of these distortions. Comprehensive information about the datasets used in this study can be found in Appendix A.2.1 (Table 10).

4.2 Models

To comprehensively evaluate hallucination patterns in ASR systems, we select models that span diverse architectures, sizes, and training paradigms. This diversity enables systematic analysis of how these factors influence hallucination susceptibility. Specifically, we include:

• Encoder-only models: HuBERT (Hsu et al., 2021) and Wav2Vec2 (Baevski et al., 2020), which leverage self-supervised training to learn robust speech representations.

• Encoder-decoder models: Whisper (Radford et al., 2022) (10 variants), DistilWhisper (Gandhi et al., 2023) (4 variants), and SeamlessM4T (Communication et al., 2023) (2 variants) are designed for multilingual transcription, translation, and speech-to-text tasks. We also include Qwen2Audio (Chu et al., 2024) and SpeechLLM (Rajaa and Tushar), which are optimized for text generation and audio-language alignment. Additionally, we incorporate Canary (Harper et al.)3, which uses FastConformer (Rekesh et al., 2023) as its encoder and a transformer decoder.

In addition to these, we also include open-weight transducer-based models. Specifically, we add a model trained with transducer decoder loss (parakeet-tdt-1.1b) and a token-and-duration transducer (Xu et al., 2023) (parakeet-tdt-1.1b), both from the NVIDIA-NeMo ASR repository (Harper et al.). In summary, the selected models vary in size (39M to 7B parameters), depth (4 to 32 layers), and training paradigms (supervised, self-supervised, and semi-supervised). Full specifications, including architectural details, training data, and hyperparameters, are provided in Appendix A.2.2 (Table 11).

4.3 Experimental Setup

We utilize models and datasets sourced from Huggingface 4. All audio data is resampled to match the sampling rate required by the respective models. For each dataset, we randomly sample 1,000 examples from the test split to ensure a manageable and consistent experimental setup. Unless otherwise specified, we use the default decoding parameters for ASR evaluation. To analyze the data, we compute SONAR embeddings (Duquenne et al., 2023) for both speech and text. Additionally, we employ CMD based on prior work (Kashyap et al., 2020) to quantify domain shifts. All experiments are conducted on a single A100/H100 GPU. Prior to calculating WER and generating embeddings, we apply a basic English text normalizer 5 to ensure consistency in text preprocessing. For LLM evaluation, we perform greedy search decoding to ensure reproducible outputs.

4.4 Human Evaluation

We construct a human evaluation dataset by aggregating outputs from multiple models, filtering

3https://huggingface.co/nvidia/canary-1b
4https://huggingface.co/models, https://huggingface.co/datasets
5https://github.com/huggingface/transformers/blob/main/src/transformers/models/whisper/english_normalizer.py
samples with WER > 60 to focus on significant deviations. Hypotheses and references are constrained to 1–100 words for balance. To simulate synthetic hallucinations, we shuffle 50 hypotheses, introducing artificial errors (Stiff et al., 2019). The final dataset includes 500 samples, each reviewed by two independent annotators from a pool of 20. This framework ensures robust evaluation and reliable analysis of hallucination patterns across models.

5 Results

In our experiments, we use an LLM-based pipeline to classify ASR errors across various evaluation setups, validating it against human evaluation and heuristic baselines. We then explore the effects of natural and synthetic distribution shifts on error metrics, specifically examining how domain variations and input perturbations impact word and hallucination error rates. Additionally, we analyze the influence of model architecture and scale through a comparison of Whisper variants and other architectures. This approach provides insights into the complex interactions between error types, data conditions, and model characteristics. In this section, we present our findings.

5.1 Hallucination Error Detection

We assess the ASR outputs of large language models by classifying them into various error categories. Specifically, we use GPT-4o mini to identify the types of errors in ASR outputs. The evaluation process includes both coarse-grained and fine-grained error classifications, as detailed in Section 3.2. The prompts used for both evaluations are shown in Figures 5 and 6.

Verbalized Confidence Scores Accurate error classification alone does not always provide insight into the model’s own uncertainty or reliability. To address this, we also assess the self-reported confidence of the LLM in its classifications. Inspired by Yang et al. (2024) and other similar works (Tian et al., 2023; Xiong et al., 2024), we prompt GPT-4o-mini to provide explicit confidence scores (on a scale of 1–10) for each classification decision. This approach allows us to understand not just what the model predicts, but how confident it is in those predictions, offering an additional dimension for evaluating model performance and robustness. In this experiment, we take 500 examples, and after each classification, we simply ask the model in a multilurn conversation: “How confident are you about the classification on a scale of 1–10? Confidence:”. We report the resulting confidence scores in Tables 2 and 3 for fine-grained and coarse-grained evaluations, respectively.

The results show that GPT-4o-mini’s self-assessed confidence remains consistently high across most categories. In the fine-grained evaluation, the “Hallucination Error” (HE) and “Oscillation Error” (OE) classes achieve nearly perfect confidence scores, with median and 25%ile values of 10.0. The “No Error” (NE) predictions also exhibit strong confidence, with an overall average of 8.81 across fine-grained classes, highlighting robust certainty in identifying transcription errors.

<table>
  <tr>
    <th>Metric</th>
    <th>HE</th>
    <th>LE</th>
    <th>NE</th>
    <th>OE</th>
    <th>PE</th>
    <th>Overall</th>
  </tr>
  <tr>
    <td>Mean</td>
    <td>9.56</td>
    <td>7.33</td>
    <td>9.77</td>
    <td>10.0</td>
    <td>7.37</td>
    <td>8.81</td>
  </tr>
  <tr>
    <td>Median</td>
    <td>10.0</td>
    <td>7.0</td>
    <td>10.0</td>
    <td>10.0</td>
    <td>7.0</td>
    <td>8.80</td>
  </tr>
  <tr>
    <td>1%ile</td>
    <td>8.00</td>
    <td>7.00</td>
    <td>7.21</td>
    <td>10.0</td>
    <td>7.00</td>
    <td>7.84</td>
  </tr>
  <tr>
    <td>5%ile</td>
    <td>8.0</td>
    <td>7.0</td>
    <td>8.1</td>
    <td>10.0</td>
    <td>7.0</td>
    <td>8.02</td>
  </tr>
  <tr>
    <td>10%ile</td>
    <td>8.0</td>
    <td>7.0</td>
    <td>10.0</td>
    <td>10.0</td>
    <td>7.0</td>
    <td>8.40</td>
  </tr>
  <tr>
    <td>25%ile</td>
    <td>9.0</td>
    <td>7.0</td>
    <td>10.0</td>
    <td>10.0</td>
    <td>7.0</td>
    <td>8.60</td>
  </tr>
</table>

Table 2: Self-verbalized confidence scores (1–10) of the LLM classifier for each fine-grained error class—Hallucination Error (HE), Language Error (LE), No Error (NE), Oscillation Error (OE), and Phonetic Error (PE)—along with overall average scores.

In contrast, the “Language Error” (LE) and “Phonetic Error” (PE) classes show slightly lower confidence, with mean values around 7.3 and narrower interquartile ranges, reflecting more nuanced or ambiguous cases.

<table>
  <tr>
    <th>Metric</th>
    <th>HE</th>
    <th>NE</th>
    <th>NHE</th>
    <th>Overall</th>
  </tr>
  <tr>
    <td>Mean</td>
    <td>9.77</td>
    <td>9.20</td>
    <td>7.09</td>
    <td>8.69</td>
  </tr>
  <tr>
    <td>Median</td>
    <td>10.0</td>
    <td>10.0</td>
    <td>7.0</td>
    <td>9.00</td>
  </tr>
  <tr>
    <td>1%ile</td>
    <td>8.0</td>
    <td>7.0</td>
    <td>1.0</td>
    <td>5.33</td>
  </tr>
  <tr>
    <td>5%ile</td>
    <td>8.0</td>
    <td>7.95</td>
    <td>7.00</td>
    <td>7.65</td>
  </tr>
  <tr>
    <td>10%ile</td>
    <td>9.0</td>
    <td>8.0</td>
    <td>7.0</td>
    <td>8.00</td>
  </tr>
  <tr>
    <td>25%ile</td>
    <td>10.0</td>
    <td>8.0</td>
    <td>7.0</td>
    <td>8.33</td>
  </tr>
</table>

Table 3: Self-verbalized confidence scores (1–10) of the LLM classifier for each coarse-grained error class—Hallucination Error (HE), No Error (NE), and Non-Hallucination Error (NHE)—along with overall average scores.

The coarse-grained evaluation mirrors these trends, with “Hallucination Error” and “No Error” classes maintaining high confidence (mean \( \geq 9.2 \)), while the “Non-Hallucination Error” (NHE) class
has a lower mean of 7.09 and a notably low 1%ile of 1.0, indicating occasional uncertainty in ambiguous scenarios. Despite this, the overall average for the coarse-grained setting is 8.69, confirming the reliability of these scores.

Overall, these findings demonstrate that GPT-4o-mini not only provides reliable classification outputs but also accurately calibrates its confidence based on the error type. The high overall averages further validate the robustness of the LLM-based evaluation framework in discerning and categorizing transcription errors.

Coarse-grained and Fine-grained Evaluation. Figure 8 in the appendix illustrates the error distributions across coarse-grained and fine-grained hallucination categories. Our results demonstrate strong alignment between both levels, indicating consistent classification of hallucinations. Among non-hallucination errors, phonetic errors dominate across most datasets (see Appendix Table 15). However, in Primock57, language errors prevail, likely due to its specialized medical terminology. This aligns with (Ferrando et al., 2024), who emphasize language models’ struggles with domain-specific named entities. This is also reflected in the second example provided in Table 1.

Agreement with Human Evaluation and Heuristic Baseline. To validate our approach, we compare model-to-human and human-to-human agreement scores using a coarse-grained prompt. Our results demonstrate strong human-to-human raw agreement (0.71), indicating consistency. Additionally, we observe good agreement (0.6) between human annotations and GPT-4o-mini’s coarse-grained output, suggesting that the model aligns reasonably well with human judgments.

We further evaluate the agreement between human and model classifications against a heuristic baseline proposed by (Frieske and Shi, 2024). Their method is based on a cosine similarity threshold of 0.2, alongside an WER threshold of 30 and a Flan-T5 (Chung et al., 2024) perplexity threshold of 200. However, as shown in Table 4, this heuristic achieves significantly lower agreement scores: 0.1 with GPT-4o mini and 0.14 with Gemini-2.0-flash-001. These results highlight the limitations of purely heuristic-based approaches compared to our method, which better captures the more fine-grained aspects like hallucination.

<table>
  <tr>
    <th>Evaluation Pair</th>
    <th>Agreement Score (\( \uparrow \))</th>
  </tr>
  <tr>
    <td>Human - Human</td>
    <td>0.71 \( \pm \) 0.01</td>
  </tr>
  <tr>
    <td>Human - Heuristic</td>
    <td>0.00 \( \pm \) 0.00</td>
  </tr>
  <tr>
    <td>Human - GPT</td>
    <td>0.60 \( \pm \) 0.01</td>
  </tr>
  <tr>
    <td>Human - Gemini</td>
    <td>0.59 \( \pm \) 0.01</td>
  </tr>
  <tr>
    <td>GPT - Heuristic</td>
    <td>0.10 \( \pm \) 0.00</td>
  </tr>
  <tr>
    <td>GPT - Gemini</td>
    <td>0.78 \( \pm \) 0.01</td>
  </tr>
  <tr>
    <td>Heuristic - Gemini</td>
    <td>0.14 \( \pm \) 0.01</td>
  </tr>
</table>

Table 4: Agreement scores (\( \uparrow \)) between different hallucination evaluation methods. Here, *GPT-4o-mini* is referred to as GPT, and *Gemini-2.0-flash-001* is referred to as Gemini. All evaluations are coarse-grained.

5.2 Errors Under Distributional Shifts

We analyze how ASR performance degrades under distributional shifts—when models are exposed to domain conditions different from their training data or domains where they outperform the human baseline (referred to as the source domain)\footnote{Here, “domain” is loosely defined; for some models, the source or training data is unknown. In those cases, we treat the source domain as the data on which the error is nearly zero, such as LibriSpeech.}. For natural (domain-specific) shifts, we quantify the distribution shift and measure the resulting degradation in both WER and hallucination errors (HER), with results shown in Figure 2 and detailed breakdowns in Appendix A.4. For synthetic shifts (e.g., adversarial and random perturbations), we directly evaluate model robustness. In what follows, we describe these two shift types—natural and synthetic—separately.

Natural Shift Given that most ASR models now outperform the human baseline on the LibriSpeech clean test set, we consider LibriSpeech as the source domain. Other domain-specific datasets, such as Primock, SPGISpeech, GLOBE, and AMI, are therefore treated as the target domain. We compute the distribution shift as detailed in Section 3.3. We then measure the change (degradation) using Equation 2 and 3.

The \( \alpha \) is the correlation coefficient between error rate degradation and distribution shift. Figure 2 shows that both WER and HER degrade as we move from the source domain to different target domains, with considerable distribution shifts across various *whisper* models. This degradation exhibits a nearly linear positive correlation with the domain
<table>
  <tr>
    <th>Model</th>
    <th>SPGI</th>
    <th>BERSI</th>
    <th>ATCOSim</th>
    <th>ADV</th>
    <th>AMI</th>
    <th>SLU</th>
    <th>SNIPS</th>
    <th>SC</th>
    <th>GLOBE</th>
    <th>SALT</th>
    <th>LS_Noise</th>
    <th>LS</th>
    <th>Primack57</th>
  </tr>
  <tr>
    <td>whisper-large-v3</td>
    <td>3.40/4.0</td>
    <td>32.4/15.8</td>
    <td>65.3/17.6</td>
    <td>33.3/49.8</td>
    <td>23.4/10.1</td>
    <td>15.5/7.9</td>
    <td>8.2/9.5</td>
    <td>18.8/15.4</td>
    <td>3.4/2.5</td>
    <td>3.0/1.0</td>
    <td>2.6/0.4</td>
    <td>2.2/0.3</td>
    <td>19.2/4.5</td>
  </tr>
  <tr>
    <td>wav2vec2-large-sls-english</td>
    <td>19.6/10.1</td>
    <td>64.0/18.6</td>
    <td>63.0/19.6</td>
    <td>100.1/96.5</td>
    <td>53.0/22.7</td>
    <td>43.4/22.7</td>
    <td>12.4/12.1</td>
    <td>20.6/14.8</td>
    <td>27.0/10.3</td>
    <td>17.0/2.1</td>
    <td>9.0/0.6</td>
    <td>6.5/0.0</td>
    <td>47.9/13.7</td>
  </tr>
  <tr>
    <td>hf-seamless-m4-large</td>
    <td>15.8/10.4</td>
    <td>58.5/16.2</td>
    <td>61.5/17.7</td>
    <td>44.1/22.7</td>
    <td>17.5/10.2</td>
    <td>12.7/8.5</td>
    <td>8.1/6.5</td>
    <td>10.8/9.2</td>
    <td>4.1/3.8</td>
    <td>2.7/0.8</td>
    <td>1.5/0.2</td>
    <td>1.6/0.2</td>
    <td>47.2/12.4</td>
  </tr>
  <tr>
    <td>hf-seamless-m4-large-en</td>
    <td>11.5/4.8</td>
    <td>68.5/28.5</td>
    <td>121.4/57.4</td>
    <td>59.3/49.5</td>
    <td>127.3/54.4</td>
    <td>18.3/16.5</td>
    <td>10.8/9.2</td>
    <td>41.3/38.4</td>
    <td>27.9/28.7</td>
    <td>9.5/4.2</td>
    <td>10.9/5.2</td>
    <td>11.4/4.2</td>
    <td>41.7/17.1</td>
  </tr>
  <tr>
    <td>whisper-medium</td>
    <td>3.7/0.2</td>
    <td>34.5/16.2</td>
    <td>65.6/18.2</td>
    <td>42.9/63.1</td>
    <td>23.2/12.2</td>
    <td>17.4/8.7</td>
    <td>8.6/4.1</td>
    <td>18.7/14.4</td>
    <td>5.3/3.1</td>
    <td>5.0/3.7</td>
    <td>3.3/0.9</td>
    <td>3.1/0.4</td>
    <td>20.6/6.2</td>
  </tr>
  <tr>
    <td>distil-large-v2</td>
    <td>4.1/1.0</td>
    <td>38.0/16.0</td>
    <td>69.5/29.6</td>
    <td>45.6/64.3</td>
    <td>22.1/11.2</td>
    <td>16.0/7.2</td>
    <td>9.2/3.1</td>
    <td>18.9/14.3</td>
    <td>6.7/3.2</td>
    <td>5.2/1.0</td>
    <td>3.6/0.7</td>
    <td>3.4/0.5</td>
    <td>19.2/5.2</td>
  </tr>
  <tr>
    <td>hubert-large-l6960-ft</td>
    <td>12.4/1.4</td>
    <td>55.5/14.4</td>
    <td>50.4/19.1</td>
    <td>100.1/100.0</td>
    <td>44.4/29.5</td>
    <td>21.5/12.4</td>
    <td>12.6/1.2</td>
    <td>30.2/20.4</td>
    <td>23.4/7.1</td>
    <td>18.7/7.1</td>
    <td>3.6/1.3</td>
    <td>2.2/0.1</td>
    <td>32.2/12.0</td>
  </tr>
  <tr>
    <td>hubert-large-l6960-ft-en</td>
    <td>12.4/1.4</td>
    <td>55.5/14.4</td>
    <td>50.4/19.1</td>
    <td>100.1/100.0</td>
    <td>44.4/29.5</td>
    <td>21.5/12.4</td>
    <td>12.6/1.2</td>
    <td>30.2/20.4</td>
    <td>23.4/7.1</td>
    <td>18.7/7.1</td>
    <td>3.6/1.3</td>
    <td>2.2/0.1</td>
    <td>32.2/12.0</td>
  </tr>
  <tr>
    <td>distil-small-en</td>
    <td>4.6/0.6</td>
    <td>46.8/19.4</td>
    <td>77.0/41.6</td>
    <td>54.3/73.3</td>
    <td>24.2/10.0</td>
    <td>15.4/7.8</td>
    <td>11.3/2.3</td>
    <td>21.5/18.2</td>
    <td>11.7/8.1</td>
    <td>9.0/4.7</td>
    <td>4.1/0.9</td>
    <td>4.0/0.6</td>
    <td>21.4/6.7</td>
  </tr>
  <tr>
    <td>whisper-medium.en</td>
    <td>4.3/1.0</td>
    <td>34.2/18.8</td>
    <td>66.2/28.2</td>
    <td>43.3/60.4</td>
    <td>23.0/11.3</td>
    <td>19.4/9.5</td>
    <td>8.4/5.5</td>
    <td>21.3/15.4</td>
    <td>4.8/2.3</td>
    <td>5.7/3.3</td>
    <td>3.5/0.9</td>
    <td>3.1/0.4</td>
    <td>20.6/5.7</td>
  </tr>
  <tr>
    <td>whisper-small.en</td>
    <td>4.1/0.9</td>
    <td>38.7/19.5</td>
    <td>68.8/29.2</td>
    <td>50.9/72.9</td>
    <td>24.5/11.7</td>
    <td>20.8/10.7</td>
    <td>9.4/1.1</td>
    <td>20.9/16.4</td>
    <td>9.6/6.4</td>
    <td>7.2/5.8</td>
    <td>3.7/0.9</td>
    <td>3.6/0.5</td>
    <td>21.5/6.7</td>
  </tr>
  <tr>
    <td>hf-seamless-m4-medium</td>
    <td>13.2/5.7</td>
    <td>59.7/21.1</td>
    <td>52.7/19.0</td>
    <td>51.4/40.1</td>
    <td>57.0/41.3</td>
    <td>30.3/22.3</td>
    <td>8.8/3.1</td>
    <td>15.9/16.1</td>
    <td>6.4/2.3</td>
    <td>8.9/2.3</td>
    <td>3.7/0.5</td>
    <td>4.6/0.3</td>
    <td>46.1/26.3</td>
  </tr>
  <tr>
    <td>whisper-large</td>
    <td>3.8/0.6</td>
    <td>48.1/15.8</td>
    <td>67.5/18.5</td>
    <td>57.2/55.7</td>
    <td>22.6/12.9</td>
    <td>18.1/9.1</td>
    <td>8.5/1.0</td>
    <td>18.6/14.8</td>
    <td>4.2/2.5</td>
    <td>4.0/2.6</td>
    <td>3.1/0.7</td>
    <td>3.0/0.3</td>
    <td>20.0/5.7</td>
  </tr>
  <tr>
    <td>whisper-large-v2</td>
    <td>4.3/0.8</td>
    <td>34.1/17.9</td>
    <td>67.1/19.6</td>
    <td>38.9/57.6</td>
    <td>24.1/15.8</td>
    <td>24.1/11.2</td>
    <td>8.4/1.1</td>
    <td>23.6/15.5</td>
    <td>4.4/3.6</td>
    <td>3.2/1.0</td>
    <td>2.7/0.6</td>
    <td>3.0/0.3</td>
    <td>20.0/6.9</td>
  </tr>
  <tr>
    <td>whisper-large-v3-turbo</td>
    <td>3.4/0.4</td>
    <td>31.7/14.7</td>
    <td>66.2/18.7</td>
    <td>34.5/49.8</td>
    <td>23.8/10.0</td>
    <td>15.7/7.4</td>
    <td>7.8/9.5</td>
    <td>18.5/13.3</td>
    <td>3.9/1.7</td>
    <td>4.7/2.6</td>
    <td>2.7/0.5</td>
    <td>3.3/0.1</td>
    <td>20.0/6.0</td>
  </tr>
  <tr>
    <td>Qwen2-Audio-7B</td>
    <td>4.6/2.7</td>
    <td>36.3/15.6</td>
    <td>44.8/35.7</td>
    <td>31.7/36.7</td>
    <td>35.7/14.9</td>
    <td>47.4/21.2</td>
    <td>5.5/7.3</td>
    <td>35.3/37.1</td>
    <td>23.3/7.0</td>
    <td>5.9/5.8</td>
    <td>2.3/1.3</td>
    <td>2.0/0.7</td>
    <td>25.5/22.8</td>
  </tr>
</table>

Table 5: WER and coarse-grained HER across different models and datasets. The values are presented as WER/HER. The lowest HER for each dataset is highlighted in green. Abbreviations: SPGI (SPGISpeech), ATCOSIM (ATCOSim Corpus), ADV (Adversarial), AMI (AMI Corpus), SLU (SLUE-VoxCeleb), SC (Supreme-Court-Speech), SALT (SALT Multispeaker English), LS_Noise (LibriSpeech Test Noise), LS (LibriSpeech ASR Test).

![Three line plots showing degradation in word error rate WERD and hallucinated error rate HERD for whisper-large-v3, whisper-large-v3-turbo, and whisper-medium models, with color-coded correlation factors α.](page_1012_1042_420_246.png)

Figure 2: Degradation in word error rate WERD (blue, left y-axis) and hallucinated error rate HERD (green, right y-axis) w.r.t distribution shift (x-axis), measured using Central Moment Discrepancy (CMD) for three different models. The correlation factor, \( \alpha \), is represented by the color, which corresponds to the type of error. Each point on the line represents a new target domain.

shift.

Notably, the HER demonstrates a slightly stronger correlation with the shift compared to WER. This trend is consistent across all models, as illustrated in Appendix A.4 Figure 7. The ATCOSim dataset is an outlier, with artificially high WER due to its numerical content. For example, models generating digits (e.g., "23") instead of spoken forms ("two three") are heavily penalized, inflating WER without accurately reflecting transcription quality.

Synthetic Shift Under synthetic shift, we experiment with two configurations: (a) adversarial perturbations and (b) common perturbations. Our experiments reveal that adversarial attacks cause the most significant degradation in HER, with adversarial datasets showing the highest HER values across all models, exceeding the degradation observed under natural shift baselines (see Table 5). In contrast, random perturbations (such as white noise, pitch shifts, and time stretching) result in more moderate impacts. Notably, self- and semi-supervised models like whisper and seamless demonstrate consistent vulnerability to structured perturbations. For instance, pitch shifts and time stretching lead to a substantial increase of approximately (242%) in both WER and HER across these models, while white noise causes smaller degradations of approximately (142%). Interestingly, the supervised wav2vec2 model exhibits non-uniform behavior, where the impact on WER and HER is similar across all perturbations. Furthermore, it is noteworthy that HER increased by 50% in the wav2vec2 model, which is considerably less than the sharp increase observed in whisper and seamless models, highlighting a notable contrast in the robustness of these models to random synthetic shifts against more targeted shifts. We provide results for additional perturbations in Appendix A.4.2 Table 8.

5.3 Impact of Architecture and Scale

In this section, we analyze how model architecture and scale influence hallucination and transcrip-
tion errors. We specifically examine how different model types—encoder-only, encoder-decoder, and transducer-based—affect hallucination error rates, as well as how scaling within a model family (such as whisper) impacts overall performance.

Model Type  We hypothesize that model architecture plays a critical role in influencing error rates, with these effects further modulated by the scale and diversity of the training data. As shown in Appendix Table 14, our findings indicate that encoder-only models Wav2vec2-large-xlsr-53-english and Hubert-large-ls960-ft exhibit the lowest HER/WER ratios across multiple datasets, particularly when compared to models like Qwen2-Audio-7B and hf-seamless-m4t-large. This suggests that encoder-only models are less prone to hallucinations relative to their overall errors.

Model Size  In our experiments, we evaluate models of varying sizes where training data is fixed across different sizes (ie: whisper).

![HER and WER for various perturbations across four models.](page_180_120_600_370.png)

Figure 3: HER and WER for various perturbations across four models.

![HER (green, right y-axis) and WER (blue, left y-axis) averaged across all datasets. X-axis denotes models with different size in increasing order.](page_180_1040_600_370.png)

Figure 4: HER (green, right y-axis) and WER (blue, left y-axis) averaged across all datasets. X-axis denotes models with different size in increasing order.

More specifically, we select 14 models from the whisper family, ranging from the smallest whisper-tiny (39M parameters) to the largest whisper-large-v3 (1.5B parameters). We then calculate both WER and HER for each of these models, following the methodology outlined in Section 3. Our findings show that for smaller models, such as whisper-tiny and whisper-small, there is a significant increase in both WER and HER. While larger models such as whisper-medium and whisper-large show substantial improvements, as shown in Figure 4. However, this reduction is not linear. After a certain point, performance improvements in both metrics become less pronounced. This non-monotonic behavior is particularly evident when comparing models in the mid-range of parameter sizes, such as whisper-medium and whisper-large-v3-turbo, where the difference in performance becomes marginal despite the difference in model size. In conclusion, while larger models generally result in lower WER and HER, the benefits of scaling up model size diminish beyond a certain point, at least when it comes to more nuanced error types.

6 Conclusion

In our work, we introduce the Hallucination Error Rate (HER) as a crucial complement to traditional ASR evaluation metrics like WER, especially in high-risk applications where model reliability is critical. By developing a robust LLM-based hallucination detection framework, we present a comprehensive evaluation of ASR models across both synthetic and natural distribution shifts, highlighting the specific challenges ASR systems face under real-world conditions. Our findings emphasize the importance of incorporating HER into standard ASR evaluation practices, particularly for applications in safety-critical domains such as healthcare, legal, and aviation. Through detailed analysis, we show that traditional metrics like WER can mask significant hallucinations, emphasizing the need for more holistic evaluation methods. Our work lays the ground for future work in ASR model reliability, aiming to ensure that ASR systems not only produce accurate transcriptions but also avoid generating misleading, harmful, and unfaithful to input speech transcriptions. In future work, we plan to expand our evaluation to cover additional evaluation setups, ensuring a comprehensive set of assessments. Additionally, we aim to explore mitigation strategies, which are critical for enhancing the reliability of ASR systems.
7 Limitations

We explore the hallucination phenomenon in ASR systems focusing on the potential causes such as distribution shift, model types, and model size and architectures. While our work offers valuable insights into model behavior across different conditions, there are several limitations to consider.

Evaluation Datasets. In our study, we evaluate ASR models across multiple domains, including legal, medical, and conversational speech, ensuring a broad range of datasets not seen during training. However, it is possible that some of the datasets we treat as target domains may have been inadvertently exposed to the models during training. Furthermore, the lack of access to a diverse variety of domain-specific datasets limits our understanding of how these models will perform in more diverse or previously unseen domains, particularly those with limited or noisy data. While our focus on domain shifts is an important step, further research is needed to assess model performance in even more varied and challenging real-world environments.

Synthetic Noise and Perturbations. Our experiments also include synthetic noise and perturbations to evaluate model robustness. While this approach helps simulate real-world challenges, it does not capture all possible distortions that may occur in uncontrolled environments. Adversarial noise, pitch shifts, and time stretching are some of the perturbations we consider, but other potential real-world disruptions, such as cross-lingual noise or complex acoustic reverberations, are not fully explored.

Hallucination Detection. The detection of hallucinations in ASR systems, as measured by the Hallucination Error Rate (HER), is a key contribution of our study. However, our reliance on LLM-based classifiers introduces potential biases and variability. While we observe strong alignment with human judgments, the accuracy of these evaluations may be influenced by subjective interpretation, especially in edge cases where the boundaries between errors are unclear. Additionally, while LLM-based methods present a novel approach, their performance in low-resource settings or with models trained on limited data has not been fully explored. Furthermore, the use of proprietary models, such as those from OpenAI via API, introduces additional costs, which could limit the scalability of this approach.

8 Ethics Statement

Data Collection and Release. For this study, we rely on publicly available datasets from diverse domains to evaluate hallucinations in ASR systems. We ensure that the data used in our research is appropriately sourced, maintaining respect for copyright, license, and privacy regulations. Furthermore, we emphasize that the use of these datasets is strictly for academic purposes, aligned with the principles of fair use.

Intended Use. Our work aims to enhance the robustness of ASR systems, especially in high-stake domains where errors can have significant consequences. We believe our findings will encourage further research in hallucination detection, with particular attention to models’ performance in low-resource and critical domains such as healthcare and law. By introducing the Hallucination Error Rate (HER) as a complementary metric to traditional evaluation methods, we hope to inspire the development of more reliable and transparent ASR systems.

Potential Misuse and Bias. While our work provides valuable insights about hallucinations in ASR systems, we acknowledge that they could be misused if deployed in inappropriate contexts. Since these models are trained on a variety of data sources, there is the potential for them to generate biased or harmful content, especially if the training data contains any inherent biases. Moreover, hallucinations in ASR outputs, if undetected, can lead to severe consequences in critical applications such as legal, medical, and financial settings. We recommend careful deployment of these models, ensuring that they undergo rigorous bias mitigation and hallucination detection processes before being used in such domains.

References

Ayo Adedeji, Sarita Joshi, and Brendan Doohan. 2024. The sound of healthcare: Improving medical transcription asr accuracy with large language models. arXiv preprint arXiv:2402.07658.

Tejumade Afonja, Tobi Olatunji, Sewade Ogun, Naome A. Etori, Abraham Owodunni, and Moshood Yekini. 2024. Performant asr models for medical entities in accented speech. Preprint, arXiv:2406.12387.

DSM-TF American Psychiatric Association, DS American Psychiatric Association, et al. 2013. Diagnostic and statistical manual of mental disorders: DSM-5,
volume 5. American psychiatric association Washington, DC.

Carlos Arriaga, Alejandro Pozo, Javier Conde, and Alvaro Alonso. 2024. Evaluation of real-time transcriptions using end-to-end asr models. arXiv preprint arXiv:2409.05674.

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460.

Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2024. Hallucination of multimodal large language models: A survey. Preprint, arXiv:2404.18930.

Mateusz Barański, Jan Jasiński, Julitta Bartolewska, Stanisław Kacprzak, Marcin Witkowski, and Konrad Kowalczyk. 2025. Investigation of whisper asr hallucinations induced by non-speech audio. arXiv preprint arXiv:2501.11378.

Mateusz Barański, Jan Jasiński, Julitta Bartolewska, Stanisław Kacprzak, Marcin Witkowski, and Konrad Kowalczyk. 2025. Investigation of whisper asr hallucinations induced by non-speech audio. Preprint, arXiv:2501.11378.

Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. 2024. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759.

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yuxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53.

Seamless Communication, Loïc Barrault, Yu-An Chou, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klairer, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenzek, Ethan Ye, Bapi Akula, Peng-Jen Chen, Naji El Hachem, Brian Ellis, Gabriel Mejia Gonzalez, Justin Haehnem, Prangthip Hansanti, Russ Howes, Bernie Huang, Min-Jae Hwang, Hirofumi Inaguma, Somya Jain, Elahe Kalbasi, Amanda Kallet, Ilia Kulikov, Janice Lam, Daniel Li, Xutai Ma, Ruslan Mavlyutov, Benjamin Peloquin, Mohamed Ramadan, Abinesh Ramakrishnan, Anna Sun, Kevin Tran, Tuan Tran, Igor Tulanov, Vish Vogeti, Carleigh Wood, Yilin Yang, Bokai Yu, Pierre Andrews, Can Balioglu, Marta R. Costa-jussà, Onur Celebi, Maha Elbayad, Cynthia Gao, Francisco Guzmán, Justine Kao, Ann Lee, Alexandre Mourachko, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Paden Tomasello, Changhan Wang, Jeff Wang, and Skyler Wang. 2023. SeamlessM4t: Massively multilingual & multimodal machine translation. Preprint, arXiv:2308.11596.

The AMI Consortium. n.d. Ami corpus. Accessed: 2025-02-04.

Miguel Del Rio, Natalie Delworth, Ryan Westerman, Michelle Huang, Nishchal Bhandari, Joseph Palakapilly, Quinten McNamara, Joshua Dong, Piotr Zelasko, and Miguel Jetté. 2021. Earnings-21: A practical benchmark for asr in the wild. In Interspeech 2021, interspeech_2021, page 3465–3469. ISCA.

Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. 2023. Sonar: sentence-level multimodal and language-agnostic representations. arXiv e-prints, pages arXiv–2308.

Qi Fan, Mattia Segu, Yu-Wing Tai, Fisher Yu, Chi-Keung Tang, Bernt Schiele, and Dengxin Dai. 2022. Normalization perturbation: A simple domain generalization method for real-world domain shifts. arXiv preprint arXiv:2211.04393.

Javier Ferrando, Oscar Obeso, Senthooran Rajamanoharan, and Neel Nanda. 2024. Do i know this entity? knowledge awareness and hallucinations in language models. arXiv preprint arXiv:2411.14257.

Katja Filippova. 2020. Controlled hallucinations: Learning to generate faithfully from noisy data. arXiv preprint arXiv:2010.05873.

Rita Frieske and Bertram E Shi. 2024. Hallucinations in neural automatic speech recognition: Identifying errors and hallucinatory models. arXiv preprint arXiv:2401.01572.

Sanchit Gandhi, Patrick von Platen, and Alexander M. Rush. 2023. Distil-whisper: Robust knowledge distillation via large-scale pseudo labelling. Preprint, arXiv:2311.00430.

Nicolas Garneau and Olivier Bolduc. 2024. The state of commercial automatic french legal speech recognition systems and their impact on court reporters et al. Preprint, arXiv:2408.11940.

Stephanie M Hare. 2021. Hallucinations: A functional network model of how sensory representations become selected for conscious awareness in schizophrenia. Frontiers in Neuroscience, 15:733038.

Eric Harper, Somshubra Majumdar, Oleksii Kuchaiev, Li Jason, Yang Zhang, Evelina Bakhturina, Vahid Noroozi, Sandeep Subramanian, Koluguri Nithin, Huang Jocelyn, Fei Jia, Jagadeesh Balam, Xuesong Yang, Michal Livne, Yi Dong, Sean Naren, and Boris Ginsburg. NeMo: a toolkit for Conversational AI and Large Language Models.

Konrad Hofbauer, Stefan Petrik, and Horst Hering. 2008. The ATCOSIM corpus of non-prompted clean air traffic control speech. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08), Marrakech, Morocco. European Language Resources Association (ELRA).
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM transactions on audio, speech, and language processing, 29:3451–3460.

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1–55.

Jaeyoung Huh, Sangjoon Park, Jeong Eun Lee, and Jong Chul Ye. 2023. Improving medical speech-to-text accuracy with vision-language pre-training model. Preprint, arXiv:2303.00091.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38.

Xuhui Jiang, Yuxing Tian, Fengrui Hua, Chengjin Xu, Yuanzhuo Wang, and Jian Guo. 2024. A survey on large language model hallucination via a creativity perspective. Preprint, arXiv:2402.06647.

Adam Tauman Kalai and Santosh S Vempala. 2024. Calibrated language models must hallucinate. In Proceedings of the 56th Annual ACM Symposium on Theory of Computing, pages 160–171.

Abhinav Ramesh Kashyap, Devamanyu Hazarika, Min-Yen Kan, and Roger Zimmermann. 2020. Domain divergences: A survey and empirical analysis. arXiv preprint arXiv:2010.12198.

Seung-Eun Kim, Bronya R Chernyak, Olga Seleznova, Joseph Keshet, Matthew Goldrick, and Ann R Bradlow. 2024a. Automatic recognition of second language speech-in-noise. JASA Express Letters, 4(2).

Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2024b. Prometheus: Inducing fine-grained evaluation capability in language models. Preprint, arXiv:2310.08491.

Allison Koenecke, Anna Seo Gyeong Choi, Kate-lyn X. Mei, Hilke Schellmann, and Mona Sloane. 2024a. Careless whisper: Speech-to-text hallucination harms. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’24, page 1672–1681. ACM.

Allison Koenecke, Anna Seo Gyeong Choi, Kate-lyn X Mei, Hilke Schellmann, and Mona Sloane. 2024b. Careless whisper: Speech-to-text hallucination harms. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 1672–1681.

Feng-Ting Liao, Yung-Chieh Chan, Yi-Chang Chen, Chan-Jan Hsu, and Da-Shan Shiu. 2023. Zero-shot domain-sensitive speech recognition with prompt-conditioning fine-tuning. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1–8.

Jiashuo Liu, Zheyuan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu, and Peng Cui. 2021. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108.13624.

Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the middle: How language models use long contexts. Preprint, arXiv:2307.03172.

Rao Ma, Mengjie Qian, Mark Gales, and Katherine M Knill. 2023. Adapting an asr foundation model for spoken language assessment. In 9th Workshop on Speech and Language Technology in Education (SLaTE), slate_2023, page 104–108. ISCA.

Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. arXiv preprint arXiv:2005.00661.

Ashish Mittal, Rudra Murthy, Vishwajeet Kumar, and Riyaz Bhat. 2024. Towards understanding and mitigating the hallucinations in nlp and speech. In Proceedings of the 7th Joint International Conference on Data Science & Management of Data (11th ACM IKDD CODS and 29th COMAD), pages 489–492.

Dena Mujtaba, Nihar R Mahapatra, Megan Arney, J Scott Yaruss, Hope Gerlach-Houck, Caryn Herrings, and Jia Bin. 2024. Lost in transcription: Identifying and quantifying the accuracy biases of automatic speech recognition systems against disfluent speech. arXiv preprint arXiv:2405.06150.

Raphael Olivier and Bhiksha Raj. 2022. Recent improvements of asr models in the face of adversarial attacks. Interspeech.

Vassil Panayotov, Guogou Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An asr corpus based on public domain audio books. http://www.openslr.org/12/. Accessed: [Insert Date].

Alex Papadopoulos Korfiatis, Francesco Moramarco, Radmila Sarac, and Aleksandar Savkov. 2022. Pri-Mock57: A dataset of primary care mock consultations. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 588–598, Dublin, Ireland. Association for Computational Linguistics.

Ankur P Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. Totto: A controlled table-to-text generation dataset. arXiv preprint arXiv:2004.14373.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision. Preprint, arXiv:2212.04356.

Shangeth Rajaa and Abhinav Tushar. SpeechLLM: Multi-Modal LLM for Speech Understanding.

Dima Rekesh, Nithin Rao Koluguri, Samuel Kriman, Somshubra Majumdar, Vahid Noroozi, He Huang, Oleksii Hrinchuk, Krishna Puvada, Ankur Kumar, Jagadeesh Balam, and Boris Ginsburg. 2023. Fast conformer with linearly scalable attention for efficient speech recognition. Preprint, arXiv:2305.05084.

Hadeel Saadany, Catherine Breslin, Constantin Orăsan, and Sophie Walker. 2022. Better transcription of uk supreme court hearings. Preprint, arXiv:2211.17094.

Pranab Sahoo, Prabhash Meheria, Akash Ghosh, Sriparna Saha, Vinija Jain, and Aman Chadha. 2024. A comprehensive survey of hallucination in large language, image, video and audio foundation models. Preprint, arXiv:2405.09589.

Zitha Sasindran, Harsha Yelchuri, and TV Prabhakar. 2024. Semascore: a new evaluation metric for automatic speech recognition tasks. arXiv preprint arXiv:2401.07506.

Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haehnem, John Hoffman, Min-Jae Hwang, Hirofumi Inaguma, Christopher Klaiber, Ilia Kulikov, Pengwei Li, Daniel Licht, Jean Maillard, Ruslan Mavlyutov, Alice Rakotoarison, Kaushik Ram Sadagopan, Abinesh Ramanikrishnan, Tuan Tran, Guillaume Wenzek, Yilin Yang, Ethan Ye, Ivan Evtimov, Pierre Fernandez, Cynthia Gao, Prangthip Hansanti, Elahe Kalbassi, Amanda Kallet, Artyom Kozhevnikov, Gabriel Mejia, Robin San Roman, Christophe Touret, Corinne Wong, Carleigh Wood, Bokai Yu, Pierre Andrews, Can Balioglu, Peng-Jen Chen, Marta R. Costa-jussà, Maha Elbayad, Hongyu Gong, Francisco Guzmán, Kevin Heffernan, Somya Jain, Justine Kao, Ann Lee, Xutai Ma, Alex Mourachko, Benjamin Pelouquin, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyah Saleem, Holger Schwenk, Anna Sun, Paden Tomasello, Changhan Wang, Jeff Wang, Skyler Wang, and Mary Williamson. 2023. Seamless: Multilingual expressive and streaming speech translation.

Prashant Serai, Vishal Sunder, and Eric Fosler-Lussier. 2021. Hallucination of speech recognition errors with sequence to sequence learning. Preprint, arXiv:2103.12258.

Prashant Serai, Vishal Sunder, and Eric Fosler-Lussier. 2022. Hallucination of speech recognition errors with sequence to sequence learning. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 30:890–900.

Syed Abdul Gaffar Shakhadri, Kruthika KR, and Kartik Basavaraj Angadi. 2025. Samba-asr state-of-the-art speech recognition leveraging structured state-space models. arXiv preprint arXiv:2501.02832.

Suwon Shon, Ankita Pasad, Felix Wu, Pablo Brusco, Yoav Artzi, Karen Livescu, and Kyu J Han. 2022. Slue: New benchmark tasks for spoken language understanding evaluation on natural speech. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7927–7931. IEEE.

Matthias Sperber, Hendra Setiawan, Christian Gollan, Udhyakumar Nallasamy, and Matthias Paulik. 2020. Consistent transcription and translation of speech. Transactions of the Association for Computational Linguistics, 8:695–709.

Adam Stiff, Prashant Serai, and Eric Fosler-Lussier. 2019. Improving human-computer interaction in low-resource settings with text-to-phonetic data augmentation. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7320–7324. IEEE.

Vishal Sunder, Prashant Serai, and Eric Fosler-Lussier. 2022. Building an asr error robust spoken virtual patient system in a highly class-imbalanced scenario without speech data. Preprint, arXiv:2204.05183.

Piotr Szymański, Łukasz Augustyniak, Mikolaj Morzy, Adrian Szymczak, Krzysztof Surdyk, and Piotr Źelasko. 2023. Why aren’t we NER yet? artifacts of ASR errors in named entity recognition in spontaneous speech transcripts. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1746–1761, Toronto, Canada. Association for Computational Linguistics.

Bashar Talafha, Abdul Waheed, and Muhammad Abdul-Mageed. 2023. N-shot benchmarking of whisper on diverse arabic speech recognition. Preprint, arXiv:2306.02902.

Kensho Technologies. 2021. Spgispeech: A large-scale, high-quality dataset for speech recognition in financial earnings calls. https://datasets.kensho.com/datasets/spgispeech. Accessed: [Insert Date].

Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafaílov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. Preprint, arXiv:2305.14975.

Prathiksha Rumale Vishwanath, Simran Tiwari, Tejas Ganesh Naik, Sahil Gupta, Dung Ngoc Thai, Wenlong Zhao, SUNJAE KWON, Victor Ardulov, Karim Tarabishy, Andrew McCallum, and Wael Sal-loum. 2024. Faithfulness hallucination detection in healthcare AI. In Artificial Intelligence and Data Science for Healthcare: Bridging Data-Centric AI and People-Centric Healthcare.
Wenbin Wang, Yang Song, and Sanjay Jha. 2024. Globe: A high-quality english corpus with global accents for zero-shot speaker adaptive text-to-speech. arXiv preprint arXiv:2406.14875.

Steven M Williamson and Victor Prybutok. 2024. The era of artificial intelligence deception: Unraveling the complexities of false realities and emerging threats of misinformation. Information, 15(6):299.

Qianqian Xie, Edward Schenck, He Yang, Yong Chen, Yifan Peng, and Fei Wang. 2023. Faithful ai in medicine: A systematic review with large language models and beyond.

Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. Preprint, arXiv:2306.13063.

Hainan Xu, Fei Jia, Somshubra Majumdar, He Huang, Shinji Watanabe, and Boris Ginsburg. 2023. Efficient sequence transduction by jointly predicting tokens and durations. Preprint, arXiv:2304.06795.

Daniel Yang, Yao-Hung Hubert Tsai, and Makoto Yamada. 2024. On verbalized confidence scores for llms. Preprint, arXiv:2412.14737.

Jia-Yu Yao, Kun-Peng Ning, Zhen-Hui Liu, Mu-Nan Ning, Yu-Yang Liu, and Li Yuan. 2023. Llm lies: Hallucinations are not bugs, but features as adversarial examples. arXiv preprint arXiv:2310.01469.

Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschläger, and Susanne Saminger-Platz. 2019. Central moment discrepancy (cmd) for domain-invariant representation learning. Preprint, arXiv:1702.08811.

Jun Zhang, Jingyue Wu, Yiyi Qiu, Aigou Song, Weifeng Li, Xin Li, and Yecheng Liu. 2023. Intelligent speech technologies for transcription, disease diagnosis, and medical equipment interactive control in smart hospitals: A review. Computers in Biology and Medicine, 153:106517.

Leor Zmigrod, Jane R Garrison, Joseph Carr, and Jon S Simons. 2016. The neural mechanisms of hallucinations: a quantitative meta-analysis of neuroimaging studies. Neuroscience & Biobehavioral Reviews, 69:113–123.

A Appendix

A.1 Hallucination

Table 9 illustrates the key differences between phonetic errors and hallucinations in ASR outputs. In the given example, the reference sentence "It's hard to recognize speech" is misrecognized phonetically as "It's hard to wreck a nice beach", demonstrating a typical pronunciation-based error where acoustically similar words are confused. In contrast, the hallucination example shows the model generating "Now move to me" instead of the reference "There are more coming", introducing semantically unrelated content not supported by the input speech. This highlights that phonetic errors stem from acoustic-perceptual confusions, while hallucinations involve the model producing arbitrary, contextually inconsistent text. The distinction is crucial for analyzing ASR failures, as each type requires different mitigation strategies—improving acoustic modeling for phonetic errors versus constraining language generation for hallucinations.

A.2 Experiments

In this section, we provide more details about our experimental setup.

A.2.1 Datasets

We provide a brief description of the datasets we use in our experiment in Table 10.

A.2.2 Models

We provide a brief description of the models we use in our experiment in Table 11.

A.3 Prompts

We show the prompts for coarse-grained and fine-grained error classification in Figures 5 and 6, respectively.

A.4 Results

We provide a detailed analysis of the experimental results, focusing on the highest and lowest performing models across the study. To offer a comprehensive overview, we present Hallucination Error Rate (HER) and Word Error Rate (WER) across domain shifts for all models, as shown in 7.

Additionally, we include fine-grained error analysis in Table 13, which highlights the differences between coarse-grained and fine-grained error categorization.

We also calculate the HER to WER ratio and present the numbers in Table 14. Robust models
would exhibit a smaller gap between hallucination and non-hallucination errors.

We provide results for transducer-based models along with Canary-1B in Table 12.

Furthermore, we present the percentage of non-hallucination errors across datasets and models, categorizing them into Phonetic (P), Oscillation (O), and Language (L) errors in Table 15. This analysis provides deeper insights into the types of errors that are most frequent and their distribution across different dataset-model combinations.

We also highlight the overall distribution across all datasets and the robustness of both levels (coarse-grained and fine-grained) in correctly identifying hallucination in Figure 8. These results underscore the importance of considering both hallucination and non-hallucination errors when evaluating ASR systems, as well as the need for domain-specific adaptations to enhance robustness.

A.4.1 LLM Evaluation

We recognize that relying on GPT-4o for hallucination detection introduces the possibility that the evaluation model itself may produce incorrect labels, leading to false positives or negatives that could skew the assessment of ASR models. Given the length of our prompt (as shown in Figures 5 and 6), this also raises concerns about the “lost in the middle” effect (Liu et al., 2023), which can further increase the risk of hallucination during evaluation.

To mitigate these challenges, our evaluation framework restricts GPT-4o’s output to only the final classification decision, explicitly avoiding chain-of-thought generation that could introduce errors (Kim et al., 2024b). This is achieved through carefully designed prompts and a deterministic decoding strategy (temperature=0.0), ensuring consistent and reproducible outputs. Consequently, GPT-4o-mini consistently produces only the classification decision, minimizing the likelihood of introducing evaluation-induced hallucinations.

Our findings show strong alignment between GPT-4o-mini’s evaluations and human judgments, with minimal false positives or false negatives. While such errors are inevitable in any classifier short of a perfect oracle, they are substantially less frequent in our approach compared to heuristic-based methods, which often misclassify phonetic and oscillation errors as hallucinations. False negatives are particularly rare, and most false positives arise from phonetic or minor oscillation errors—very rarely (less than 1%) from semantically correct transcriptions.

Effect of Text Normalization  We also examine the impact of text normalization that we use to post-process the text for error rate calculation and hallucination detection. While normalization removes filler words and disfluencies—potentially important for applications requiring verbatim transcripts (Ma et al., 2023; Talafha et al., 2023)—we find that it has negligible impact on hallucination classification. We report the results in Table 6 and 7 for coarse-grained and fine-grained evaluations, respectively.

In our evaluation of 500 examples, we observe no transitions from “No Error” to “Hallucination Error,” and only about a 1% transition from “Hallucination Error” to “Non-Hallucination Error” across both coarse-grained and fine-grained evaluations.

<table>
  <tr>
    <th>From (Normalized)</th>
    <th>To (Orthographic)</th>
    <th>Count</th>
  </tr>
  <tr>
    <td>HE</td>
    <td>NE</td>
    <td>2</td>
  </tr>
  <tr>
    <td>HE</td>
    <td>NHE</td>
    <td>7</td>
  </tr>
  <tr>
    <td>NE</td>
    <td>NHE</td>
    <td>1</td>
  </tr>
  <tr>
    <td>NHE</td>
    <td>HE</td>
    <td>2</td>
  </tr>
</table>

Table 6: Transitions from normalized to orthographic transcriptions (coarse-grained) in our 500-example evaluation. Abbreviations: HE - Hallucination Error, NE - No Error, NHE - Non-Hallucination Error.

<table>
  <tr>
    <th>From (Normalized)</th>
    <th>To (Orthographic)</th>
    <th>Count</th>
  </tr>
  <tr>
    <td>HE</td>
    <td>NE</td>
    <td>1</td>
  </tr>
  <tr>
    <td>HE</td>
    <td>OE</td>
    <td>1</td>
  </tr>
  <tr>
    <td>HE</td>
    <td>PE</td>
    <td>5</td>
  </tr>
  <tr>
    <td>HE</td>
    <td>HE</td>
    <td>1</td>
  </tr>
  <tr>
    <td>OE</td>
    <td>HE</td>
    <td>1</td>
  </tr>
  <tr>
    <td>PE</td>
    <td>HE</td>
    <td>2</td>
  </tr>
  <tr>
    <td>PE</td>
    <td>LE</td>
    <td>1</td>
  </tr>
</table>

Table 7: Transitions from normalized to orthographic transcriptions (fine-grained) in our 500-example evaluation. Abbreviations: HE - Hallucination Error, NE - No Error, OE - Oscillation Error, PE - Phonetic Error, LE - Language Error.

The overall agreement scores of 0.9324 (coarse-grained) and 0.9495 (fine-grained) further underscore the robustness of our evaluation framework, confirming that normalization does not compromise the integrity of hallucination detection. These findings demonstrate that for our use case, nor-
malization preserves semantic fidelity without introducing artifacts that would affect hallucination classification, ensuring that our evaluation remains consistent and reliable even when “disfluencies” are removed.

Cost Analysis   We acknowledge that using proprietary models such as GPT-4o-mini incurs significant costs, which may limit scalability. To address this concern, we conduct a detailed cost analysis comparing our LLM-based evaluation framework to a human evaluation baseline, which serves as an upper bound for cost. This analysis scales to 1M segments, each approximately 20 seconds in duration, totaling roughly 5,000 hours.

For the LLM-based evaluation, the combined fine- and coarse-grained prompts take approximately 1,000 tokens per example (upper bound), with most input tokens cached. The error class outputs typically require fewer than five tokens on average. At the time of evaluation, the pricing is $0.075 per million input tokens and $0.600 per million output tokens. Scaling this setup to 1M segments:

• Input cost: 1,000M tokens \( \times \) $0.075/1M = $75.

• Output cost: 5M tokens \( \times \) $0.600/1M = $3.

The total upper-bound cost is therefore $78. With batch inference, we expect this could be reduced by 50%, making it even more cost-effective.

In contrast, our human evaluation baseline requires approximately 10 minutes to evaluate 50 examples. Scaling to 1M examples:

• Human evaluation time: (10 minutes / 50 examples) \( \times \) 1M = 200,000 minutes (\( \approx \) 3,333 hours).

• Human evaluation cost (at $10/hour): \( \approx \) $30,000.

Overall, this analysis demonstrates that our LLM-based evaluation framework is approximately 375 times cheaper than human evaluation, underscoring its scalability and cost-effectiveness for large-scale evaluation tasks.

A.4.2 Synthetic Shift
We provide results for the additional synthetic perturbations in Table 8.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">Cross-Lingual Noise</th>
    <th colspan="4">Distortion</th>
    <th colspan="4">Echo</th>
    <th colspan="4">Pub-Noise</th>
    <th colspan="4">Reverberation</th>
  </tr>
  <tr>
    <th>WER</th><th>CER</th><th>cHER</th><th>fHER</th>
    <th>WER</th><th>CER</th><th>cHER</th><th>fHER</th>
    <th>WER</th><th>CER</th><th>cHER</th><th>fHER</th>
    <th>WER</th><th>CER</th><th>cHER</th><th>fHER</th>
    <th>WER</th><th>CER</th><th>cHER</th><th>fHER</th>
  </tr>
  <tr>
    <td>distil-medium-en</td><td>25.95</td><td>18.06</td><td>14.60</td><td>14.30</td><td>20.89</td><td>14.26</td><td>8.50</td><td>7.80</td><td>32.42</td><td>20.74</td><td>20.90</td><td>19.00</td><td>41.49</td><td>28.10</td><td>30.00</td><td>28.20</td><td>23.98</td><td>16.00</td><td>11.00</td><td>10.60</td>
  </tr>
  <tr>
    <td>whisper-large-v3</td><td>22.93</td><td>16.43</td><td>12.30</td><td>10.90</td><td>17.71</td><td>12.79</td><td>6.40</td><td>6.40</td><td>32.74</td><td>24.14</td><td>12.80</td><td>12.10</td><td>36.31</td><td>27.24</td><td>23.60</td><td>23.10</td><td>21.21</td><td>15.16</td><td>8.30</td><td>9.20</td>
  </tr>
  <tr>
    <td>wav2vec2-large-sler-53-english</td><td>53.33</td><td>35.74</td><td>25.60</td><td>28.80</td><td>34.65</td><td>18.69</td><td>9.90</td><td>9.60</td><td>83.62</td><td>53.84</td><td>39.50</td><td>47.10</td><td>52.61</td><td>31.51</td><td>30.10</td><td>28.10</td><td>42.38</td><td>24.41</td><td>13.50</td><td>15.90</td>
  </tr>
</table>

Table 8: WER, CER, cHER, and fHER of different models evaluated across various perturbation scenarios: Cross-lingual Noise, Distortion, Echo, Pub Noise, and Reverberation. Abbreviations: WER/CER - Word/Character Error Rate; cHER - Coarse-grained Hallucination Error Rate; fHER - Fine-grained Hallucination Error Rate.

<table>
  <tr>
    <th>Error Type</th>
    <th>Reference</th>
    <th>Hypothesis</th>
  </tr>
  <tr>
    <td>Phonetic</td>
    <td>It’s hard to recognize speech</td>
    <td>It’s hard to wreck a nice beach</td>
  </tr>
  <tr>
    <td>Hallucination</td>
    <td>There are more coming</td>
    <td>Now move to me</td>
  </tr>
  <tr>
    <td>Language</td>
    <td>ripped ocean jumper</td>
    <td>Rips Ocean Jumper</td>
  </tr>
  <tr>
    <td>Oscillation</td>
    <td>oh so it’s uh yeah</td>
    <td>Oh, I see, I see, I see, I see, I see, I see,</td>
  </tr>
</table>

Table 9: Examples of different ASR error types, including phonetic confusions, hallucinations, language errors, and oscillation artifacts.

You are a classifier trained to detect and categorize specific transcription errors produced by a speech recognition system. The possible categories are:
1. Hallucination Error: The output contains fabricated, contradictory, or invented information that is not supported by the ground truth. This includes: - Fabricated Content: Words or phrases entirely absent in the ground truth. - Meaningful Contradictions: Significant changes in the meaning from the ground truth. - Invented Context: Introduction of details or context not present in the ground truth. - Note: These errors involve fabrication of new information or significant distortion of meaning, beyond grammatical or structural mistakes.
2. Non-Hallucination Error. Errors that do not involve fabrication or significant contradictions of the ground truth. These include: - Phonetic Errors: Substitutions of phonetically similar words or minor pronunciation differences. - Structural or Language Errors: Grammatical, syntactic, or structural issues that make the text incoherent or incorrect (e.g., incorrect verb tenses, subject-verb agreement problems, omissions, or insertions). - Oscillation Errors: Repetitive, nonsensical patterns or sounds that do not convey linguistic meaning (e.g., “ay ay ay ay”). - Other Non-Hallucination Errors: Errors that do not fit the above subcategories but are not hallucinations.
3. No Error: The generated output conveys the same meaning as the ground truth, even if the phrasing, grammar, or structure differs. Minor differences in wording, phrasing, or grammar that do not alter the intended meaning are acceptable.

Input Format:
Ground Truth: The original, accurate text provided.
Generated Output: The text produced by the speech recognition system.

Output Format: Classify the input text pairs into one of the following:
Non-Hallucination Error
Hallucination Error
No Error

Examples:
Example 1:
Ground Truth: “A millimeter roughly equals one twenty-fifth of an inch.”
Generated Output: “Miller made her roughly one twenty-liftths of an inch.”
Output: Non-Hallucination Error

Example 2:
Ground Truth: “Indeed, ah!”
Generated Output: “Ay ay indeed ay ay ay ay ay.”
Output: Non-Hallucination Error

Example 3:
Ground Truth: “Captain Lake did not look at all like a London dandy now.”
Generated Output: “Will you let Annabel ask her if she sees what it is you hold in your arms again?”
Output: Hallucination Error

Example 4:
Ground Truth: “The patient was advised to take paracetamol for fever and rest for two days.”
Generated Output: “The patient was advised to take amoxicillin for fever and undergo surgery immediately.”
Output: Hallucination Error

Example 5:
Ground Truth: “I need to book a flight to New York.”
Generated Output: “I need to book ticket to New York.”
Output: No Error

Example 6:
Ground Truth: “She went to the store yesterday.”
Generated Output: “She went to the shop yesterday.”
Output: No Error

Instruction: You must produce only the classification as the output. Do not include explanations, reasoning, or additional information.

Input: Ground Truth: "[ground_truth]" Generated Output: "[output]"
Output: [insert your classification here]

Figure 5: Coarsegrained error detection prompt. The task is to classify transcription errors produced by an ASR model into one of three categories: Non-Hallucination Error, Hallucination Error, or No Error.
<table>
  <tr>
    <th>Name</th>
    <th>Domain</th>
    <th>Recording Conditions</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>LibriSpeech</td>
    <td>Speech Recognition</td>
    <td>High-quality, read speech from audiobooks</td>
    <td>A corpus of approximately 1,000 hours of 16kHz read English speech, derived from LibriVox audiobooks, segmented and aligned for ASR tasks.</td>
  </tr>
  <tr>
    <td>GLOBE</td>
    <td>Accented Speech</td>
    <td>Close-talk microphone</td>
    <td>Contains utterances from 23,519 speakers and covers 164 accents worldwide, recorded in close-talk microphone conditions.</td>
  </tr>
  <tr>
    <td>Supreme-court</td>
    <td>Legal</td>
    <td>Diverse acoustic conditions (audiobooks, podcasts, YouTube)</td>
    <td>A multi-domain, multi-style speech recognition corpus incorporating diverse acoustic and linguistic conditions, sourced from audiobooks, podcasts, and YouTube.</td>
  </tr>
  <tr>
    <td>SPGISpeech</td>
    <td>Finance</td>
    <td>Corporate earnings calls (professional transcription)</td>
    <td>Contains 5,000 hours of professionally transcribed audio from corporate earnings calls, featuring both spontaneous and narrated speaking styles.</td>
  </tr>
  <tr>
    <td>Adversarial</td>
    <td>Synthetic</td>
    <td>Corporate earnings calls</td>
    <td>Includes multiple splits with utterances modified using adversarial noise of varying radii (0.04 and 0.015) and combined with Room Impulse Response (RIR) noise.</td>
  </tr>
  <tr>
    <td>AMI (IHM)</td>
    <td>Meetings</td>
    <td>Multi-device meeting environment</td>
    <td>The AMI Meeting Corpus is a 100-hour dataset of English meeting recordings, featuring multimodal data synchronized across various devices.</td>
  </tr>
  <tr>
    <td>SLUE - VoxCeleb</td>
    <td>Conversational</td>
    <td>YouTube video extracts (conversational)</td>
    <td>Consists of single-sided conversational voice snippets extracted from YouTube videos, originally designed for speaker recognition.</td>
  </tr>
  <tr>
    <td>Primock57</td>
    <td>Medical</td>
    <td>Mock consultations by clinicians</td>
    <td>Contains mock consultations conducted by seven clinicians and 57 actors posing as patients, representing a diverse range of ethnicities, accents, and ages.</td>
  </tr>
  <tr>
    <td>BERSt</td>
    <td>Home Environment</td>
    <td>Home recordings using smartphones</td>
    <td>A collection of speech data recorded in home environments using various smartphone microphones, with participants from diverse regions.</td>
  </tr>
  <tr>
    <td>ATCOSim</td>
    <td>Aviation</td>
    <td>Real-time air traffic control simulations</td>
    <td>A specialized database containing ten hours of English speech from ten non-native speakers, recorded during real-time air traffic control simulations.</td>
  </tr>
</table>

Table 10: Speech Datasets for ASR, categorized by domain, recording conditions, and description.
<table>
  <tr>
    <th>Model</th>
    <th>Type and Models</th>
    <th>Parameters</th>
    <th>Architecture</th>
    <th>Pre-Training Objective and Training Data</th>
  </tr>
  <tr>
    <td>wav2vec2</td>
    <td>(Baevski et al., 2020)<br>– wav2vec2-large-xlsr-53-english</td>
    <td>315M</td>
    <td>7-Conv (Kernel 10/3/3/3/3/2/2) + 24-Trans</td>
    <td>Self-supervised pre-training on raw audio via contrastive loss.<br>Training data: Common Voice 6.1 (53 languages).</td>
  </tr>
  <tr>
    <td>hubert</td>
    <td>(Hsu et al., 2021)<br>– hubert-large-ls960-ft</td>
    <td>316M</td>
    <td>7-Conv (Kernel 10/3/3/3/3/2/2) + 24-Trans</td>
    <td>Masked prediction pre-training.<br>Training data: Libri-Light (60k hours).</td>
  </tr>
  <tr>
    <td>seamless</td>
    <td>(Seamless Communication et al., 2023)<br>– hf-seamless-m4t-large<br>– hf-seamless-m4t-medium</td>
    <td>2.3B<br>1.2B</td>
    <td>UnitY2 (Enc-Dec + Text Decoder)</td>
    <td>Multilingual ASR/translation.<br>Training data: 443k hours of aligned speech-text (29 languages).</td>
  </tr>
  <tr>
    <td>speechlm</td>
    <td>(Rajaa and Tushar)<br>– speechlm-1.5B</td>
    <td>1.5B</td>
    <td>HubertX encoder + TinyLlama decoder</td>
    <td>Audio-text alignment via multi-task learning.<br>Training data: Proprietary ASR datasets.</td>
  </tr>
  <tr>
    <td>whisper</td>
    <td>(Radford et al., 2022)<br>– whisper-large-v3<br>– distil-large-v3<br>– whisper-large-v2<br>– whisper-large-v3-turbo<br>– distil-large-v2<br>– whisper-large<br>– whisper-tiny<br>– whisper-tiny.en<br>– whisper-medium<br>– whisper-medium.en<br>– distil-medium.en<br>– distil-small.en<br>– whisper-small<br>– whisper-small.en</td>
    <td>1.55B<br>756M<br>769M<br>244M<br>39M</td>
    <td>2-Conv (Kernel 3x3, stride 2) + 32-Trans (large)<br>2-Conv + 24-Trans (medium)<br>2-Conv + 12-Trans (small)</td>
    <td>Multilingual ASR/translation.<br>Pre-training: 680k hours of web-crawled audio.</td>
  </tr>
  <tr>
    <td>Qwen</td>
    <td>(Chu et al., 2024)<br>– Qwen2-Audio-7B</td>
    <td>7B</td>
    <td>Audio encoder + QwenLM decoder</td>
    <td>Multi-task pretraining (ASR, TTS, alignment).<br>Training data: 3M audio-text pairs.</td>
  </tr>
</table>

Table 11: Model architectures, parameters, and training details. Whisper variants include convolutional layers for spectrogram downsampling.

<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="4">BERSi</th>
    <th colspan="4">Primock57</th>
    <th colspan="4">Adversarial</th>
    <th colspan="4">Salt-multispeaker-eng</th>
    <th colspan="4">Supreme-court-speech</th>
  </tr>
  <tr>
    <th>WER</th>
    <th>CER</th>
    <th>cHER</th>
    <th>fHER</th>
    <th>WER</th>
    <th>CER</th>
    <th>cHER</th>
    <th>fHER</th>
    <th>WER</th>
    <th>CER</th>
    <th>cHER</th>
    <th>fHER</th>
    <th>WER</th>
    <th>CER</th>
    <th>cHER</th>
    <th>fHER</th>
  </tr>
  <tr>
    <td>canary-1b</td>
    <td>37.4</td>
    <td>37.4</td>
    <td>18.8</td>
    <td>14.7</td>
    <td>18.0</td>
    <td>17.2</td>
    <td>6.0</td>
    <td>6.6</td>
    <td>27.9</td>
    <td>27.5</td>
    <td>46.3</td>
    <td>41.6</td>
    <td>3.3</td>
    <td>3.3</td>
    <td>2.1</td>
    <td>1.6</td>
    <td>16.0</td>
    <td>15.7</td>
    <td>18.9</td>
    <td>21.3</td>
  </tr>
  <tr>
    <td>parakeet-rmt-1.1b</td>
    <td>26.1</td>
    <td>27.7</td>
    <td>16.4</td>
    <td>13.9</td>
    <td>17.0</td>
    <td>16.2</td>
    <td>5.7</td>
    <td>6.3</td>
    <td>17.5</td>
    <td>17.2</td>
    <td>29.8</td>
    <td>30.6</td>
    <td>3.1</td>
    <td>3.1</td>
    <td>1.6</td>
    <td>2.1</td>
    <td>15.5</td>
    <td>15.3</td>
    <td>17.4</td>
    <td>18.8</td>
  </tr>
  <tr>
    <td>parakeet-ldh-1.1b</td>
    <td>27.7</td>
    <td>28.6</td>
    <td>12.6</td>
    <td>11.1</td>
    <td>15.8</td>
    <td>15.0</td>
    <td>5.5</td>
    <td>6.4</td>
    <td>19.4</td>
    <td>19.1</td>
    <td>32.2</td>
    <td>30.2</td>
    <td>3.3</td>
    <td>3.3</td>
    <td>1.1</td>
    <td>1.6</td>
    <td>15.7</td>
    <td>15.5</td>
    <td>15.7</td>
    <td>19.1</td>
  </tr>
</table>

Table 12: WER, CER, cHER, and fHER for different models across datasets.
You are a classifier trained to detect and categorize specific transcription errors produced by a speech recognition system. The possible categories are:

1. Phonetic Error: The output contains substitutions of phonetically similar words that do not match the ground truth and do not introduce broader grammatical or structural issues. These errors typically involve misrecognition of similar-sounding words or minor pronunciation differences.
2. Oscillation Error: The output includes repetitive, non-synactical patterns of sounds that do not convey linguistic meaning (e.g., “ay ay ay ay”).
3. Hallucination Error: The output includes fabricated, contradictory, or irrelevant information that is not supported by the ground truth. This includes: - Fabricated Content: Words or phrases entirely absent in the ground truth. - Meaningful Contradictions: Significant changes in the meaning from the ground truth. - Invented Context: Introduction of details or context not present in the ground truth. - Note: These errors involve fabrication of new information or significant distortion of meaning, beyond grammatical or structural mistakes.
4. Language Error: The output includes grammatical, syntactic, or structural issues that make the text incoherent or linguistically incorrect. This category encompasses errors such as: - Incorrect verb tenses or subject-verb agreement problems. - Sentence fragments or incomplete structures. - Omissions or insertions of words that do not fabricate new context. - Incomplete sentences or phrases that do not convey the intended meaning as ground truth. - Note: Incomplete sentences or phrases are classified as Language Errors only when they do not fabricate new meaning or deviate from the intent of the ground truth.
5. No Error: The generated output conveys the same meaning as the ground truth, even if the phrasing, grammar, or structure differs. Minor differences in wording, phrasing, punctuation, or casing that do not alter the intended meaning are not considered errors. - Note: Minor omissions, such as missing articles, are acceptable as long as they do not change the meaning of the ground truth.

Input Format:
Ground Truth: The original, accurate text provided.
Generated Output: The text produced by the speech recognition system.

Output Format: Classify the input text pairs into one of the following:
Phonetic Error
Oscillation Error
Hallucination Error
Language Error
No Error

Examples:
Example 1:
Ground Truth: “A millimeter roughly equals one twenty-fifth of an inch.”
Generated Output: “Millimeter made her roughly one twenty-fifths of an inch.”
Output: Phonetic Error

Example 2:
Ground Truth: “I will go to New York City.”
Generated Output: “Ay ay ay ay ay ay ay.”
Output: Oscillation Error

Example 3:
Ground Truth: “Captain Lake did not look at all like a London dandy now.”
Generated Output: “Will let Annabel ask her if she sees what it is you hold in your arms again?”
Output: Hallucination Error

Example 4:
Ground Truth: “The cat is chasing the mouse.”
Generated Output: “The cat chased by the mouse.”
Output: Language Error

Example 5:
Ground Truth: “I need to book a flight to New York.”
Generated Output: “I need to book ticket to New York.”
Output: No Error

Your Task: Classify the input into one of the five categories.
Instruction: You must produce only the classification as the output. Do not include explanations, reasoning, or additional information.

Input: Ground Truth: "[ground_truth]" Generated Output: "[output]"
Output: ([insert your classification here])

Figure 6: Finegrained error detection prompt. The task is to classify transcription errors produced by an ASR model into one of five categories: Phonetic Error, Oscillation Error, Hallucination Error, Language Error, or No Error.

<table>
  <tr>
    <th>Model</th>
    <th>SPGI</th>
    <th>BERSI</th>
    <th>ATCOsim</th>
    <th>ADV</th>
    <th>AMI</th>
    <th>SLU</th>
    <th>SNIPS</th>
    <th>SC</th>
    <th>GLOBE</th>
    <th>SALT</th>
    <th>LS_Noise</th>
    <th>LS</th>
    <th>Primock57</th>
  </tr>
  <tr>
    <td>whisper-large-v3</td>
    <td>3.409</td>
    <td>32.413/2.8</td>
    <td>65.3/11.3</td>
    <td>33.347/1.3</td>
    <td>23.4/1.3</td>
    <td>15.5/13.4</td>
    <td>8.205/0.3</td>
    <td>18.8/0.4</td>
    <td>3.4/0.9</td>
    <td>3.0/0.1</td>
    <td>2.6/0.4</td>
    <td>2.2/0.5</td>
    <td>19.2/4.8</td>
  </tr>
  <tr>
    <td>wav2vec2-large-xlsr-53-english</td>
    <td>39.6/4.0</td>
    <td>41.0/3.8</td>
    <td>65.3/11.3</td>
    <td>10.4/0.7</td>
    <td>43.4/4.0</td>
    <td>12.4/0.9</td>
    <td>8.205/0.3</td>
    <td>18.8/0.4</td>
    <td>3.4/0.9</td>
    <td>3.0/0.1</td>
    <td>2.6/0.4</td>
    <td>2.2/0.5</td>
    <td>19.2/4.8</td>
  </tr>
  <tr>
    <td>speechlm-t5-large</td>
    <td>14.7/4.5</td>
    <td>58.9/5.5</td>
    <td>76.6/5.1</td>
    <td>16.2/1.4</td>
    <td>63.7/4.3</td>
    <td>17.5/2.5</td>
    <td>8.3/1.9</td>
    <td>10.8/2.8</td>
    <td>4.3/0.6</td>
    <td>2.7/0.2</td>
    <td>4.2/0.5</td>
    <td>4.2/0.5</td>
    <td>4.7/0.5</td>
    <td>18.9/5.8</td>
  </tr>
  <tr>
    <td>speechlm-t5-MB</td>
    <td>11.5/4.1</td>
    <td>68.5/1.8</td>
    <td>121.4/8.3</td>
    <td>95.3/9.5</td>
    <td>127.3/5.2</td>
    <td>83.8/19.3</td>
    <td>10.8/2.8</td>
    <td>4.3/0.6</td>
    <td>2.7/0.2</td>
    <td>4.2/0.5</td>
    <td>4.2/0.5</td>
    <td>4.7/0.5</td>
    <td>18.9/5.8</td>
  </tr>
  <tr>
    <td>whisper-medium</td>
    <td>3.7/1.1</td>
    <td>34.9/4.5</td>
    <td>65.6/14.4</td>
    <td>42.9/5.8</td>
    <td>23.2/1.4</td>
    <td>17.4/1.6</td>
    <td>8.6/1.1</td>
    <td>18.7/1.0</td>
    <td>5.3/0.9</td>
    <td>5.0/0.3</td>
    <td>3.3/0.8</td>
    <td>3.1/0.2</td>
    <td>20.6/5.9</td>
  </tr>
  <tr>
    <td>distil-large-v2</td>
    <td>4.1/0.8</td>
    <td>38.0/1.3</td>
    <td>69.9/2.4</td>
    <td>45.6/2.4</td>
    <td>22.1/1.1</td>
    <td>16.0/1.3</td>
    <td>9.2/0.8</td>
    <td>18.9/1.5</td>
    <td>6.7/0.3</td>
    <td>5.2/0.3</td>
    <td>3.6/0.3</td>
    <td>3.4/0.3</td>
    <td>19.2/5.9</td>
  </tr>
  <tr>
    <td>distil-large-256d-ft</td>
    <td>22.6/2.0</td>
    <td>58.9/1.3</td>
    <td>106.0/10.0</td>
    <td>10.8/1.1</td>
    <td>12.6/0.8</td>
    <td>12.6/0.8</td>
    <td>8.3/0.5</td>
    <td>10.8/2.4</td>
    <td>3.7/0.2</td>
    <td>2.6/0.2</td>
    <td>3.2/0.2</td>
    <td>2.5/0.1</td>
    <td>19.2/11.7</td>
  </tr>
  <tr>
    <td>distil-medium-en</td>
    <td>4.6/0.6</td>
    <td>39.9/1.8</td>
    <td>71.3/2.6</td>
    <td>45.8/5.8</td>
    <td>23.6/0.6</td>
    <td>15.6/1.3</td>
    <td>9.7/1.1</td>
    <td>20.0/1.5</td>
    <td>8.5/1.9</td>
    <td>7.4/1.1</td>
    <td>4.3/0.9</td>
    <td>4.2/0.5</td>
    <td>21.0/5.4</td>
  </tr>
  <tr>
    <td>distil-small-en</td>
    <td>4.6/1.0</td>
    <td>46.9/1.7</td>
    <td>77.0/3.2</td>
    <td>54.3/6.8</td>
    <td>24.2/0.9</td>
    <td>15.4/1.3</td>
    <td>11.3/1.7</td>
    <td>21.5/1.8</td>
    <td>11.7/1.6</td>
    <td>9.0/0.7</td>
    <td>4.1/0.8</td>
    <td>4.0/0.4</td>
    <td>21.4/6.8</td>
  </tr>
  <tr>
    <td>whisper-tiny-en</td>
    <td>4.3/0.5</td>
    <td>34.2/1.5</td>
    <td>61.6/2.4</td>
    <td>43.5/6.8</td>
    <td>23.0/1.1</td>
    <td>19.4/0.6</td>
    <td>8.4/1.1</td>
    <td>21.3/0.6</td>
    <td>4.8/1.7</td>
    <td>5.7/0.4</td>
    <td>3.5/0.7</td>
    <td>3.1/0.4</td>
    <td>20.6/6.0</td>
  </tr>
  <tr>
    <td>hf-seamless-mtl-small</td>
    <td>4.1/0.8</td>
    <td>38.7/1.3</td>
    <td>69.9/2.4</td>
    <td>45.6/2.4</td>
    <td>22.1/1.1</td>
    <td>16.0/1.3</td>
    <td>9.2/0.8</td>
    <td>18.9/1.5</td>
    <td>6.7/0.3</td>
    <td>5.2/0.3</td>
    <td>3.6/0.3</td>
    <td>3.4/0.3</td>
    <td>19.2/5.9</td>
  </tr>
  <tr>
    <td>speech5_asr</td>
    <td>25.8/2.1</td>
    <td>108.1/3.2</td>
    <td>81.7/5.7</td>
    <td>117.2/10.0</td>
    <td>46.2/5.2</td>
    <td>129.4/21.9</td>
    <td>24.6/7.8</td>
    <td>156.7/43.9</td>
    <td>60.1/54.1</td>
    <td>5.9/39.3</td>
    <td>13.9/41.1</td>
    <td>6.0/0.8</td>
    <td>53.9/41.2</td>
  </tr>
  <tr>
    <td>hf-seamless-mtl-medium</td>
    <td>13.2/5.1</td>
    <td>57.9/2.9</td>
    <td>52.7/4.0</td>
    <td>51.4/6.5</td>
    <td>57.0/4.1</td>
    <td>50.3/2.5</td>
    <td>8.8/1.6</td>
    <td>36.0/3.6</td>
    <td>15.9/4.2</td>
    <td>6.4/2.1</td>
    <td>8.9/3.0</td>
    <td>3.7/0.4</td>
    <td>46.1/24.5</td>
  </tr>
  <tr>
    <td>whisper-tiny</td>
    <td>8.8/4.3</td>
    <td>12.2/3.7</td>
    <td>36.3/0.3</td>
    <td>8.8/0.5</td>
    <td>40.3/2.5</td>
    <td>22.5/1.5</td>
    <td>15.6/4.7</td>
    <td>38.6/2.8</td>
    <td>14.7/4.6</td>
    <td>20.0/13.3</td>
    <td>10.8/6.7</td>
    <td>7.6/1.0</td>
    <td>32.8/15.8</td>
  </tr>
  <tr>
    <td>hf-seamless-mtl-large</td>
    <td>3.7/0.8</td>
    <td>34.2/1.5</td>
    <td>61.6/2.4</td>
    <td>43.5/6.8</td>
    <td>23.0/1.1</td>
    <td>19.4/0.6</td>
    <td>8.4/1.1</td>
    <td>21.3/0.6</td>
    <td>4.8/1.7</td>
    <td>5.7/0.4</td>
    <td>3.5/0.7</td>
    <td>3.1/0.4</td>
    <td>20.6/6.0</td>
  </tr>
  <tr>
    <td>whisper-large-v2</td>
    <td>4.3/0.9</td>
    <td>34.1/4.5</td>
    <td>67.1/1.5</td>
    <td>38.9/5.4</td>
    <td>24.1/4.3</td>
    <td>18.2/1.9</td>
    <td>8.4/0.5</td>
    <td>23.6/15.6</td>
    <td>4.4/0.9</td>
    <td>3.2/1.6</td>
    <td>2.7/0.6</td>
    <td>3.0/0.2</td>
    <td>20.0/6.4</td>
  </tr>
  <tr>
    <td>whisper-large-v3-turbo</td>
    <td>3.4/0.9</td>
    <td>31.7/1.1</td>
    <td>66.2/1.5</td>
    <td>34.5/4.1</td>
    <td>23.8/1.0</td>
    <td>15.7/1.4</td>
    <td>7.8/0.6</td>
    <td>18.5/1.4</td>
    <td>3.9/1.1</td>
    <td>4.7/1.6</td>
    <td>2.7/0.3</td>
    <td>2.5/0.5</td>
    <td>20.8/4.8</td>
  </tr>
  <tr>
    <td>whisper-tiny-en</td>
    <td>6.9/0.3</td>
    <td>75.4/2.5</td>
    <td>112.3/5.7</td>
    <td>80.1/8.7</td>
    <td>38.4/20.8</td>
    <td>19.4/15.1</td>
    <td>14.0/4.1</td>
    <td>38.0/28.8</td>
    <td>42.1/37.9</td>
    <td>19.4/12.6</td>
    <td>9.4/5.1</td>
    <td>6.1/0.8</td>
    <td>31.4/14.8</td>
  </tr>
  <tr>
    <td>distil-large-v3</td>
    <td>3.7/0.8</td>
    <td>33.7/1.3</td>
    <td>67.1/1.5</td>
    <td>38.9/5.4</td>
    <td>24.1/4.3</td>
    <td>18.2/1.9</td>
    <td>8.4/1.1</td>
    <td>21.3/0.6</td>
    <td>4.8/1.7</td>
    <td>5.7/0.4</td>
    <td>3.5/0.7</td>
    <td>3.1/0.4</td>
    <td>20.6/6.0</td>
  </tr>
  <tr>
    <td>distil-small</td>
    <td>4.3/0.2</td>
    <td>42.1/1.7</td>
    <td>73.4/2.6</td>
    <td>77.0/2.5</td>
    <td>39.8/1.7</td>
    <td>18.2/1.4</td>
    <td>9.6/1.4</td>
    <td>23.4/17.6</td>
    <td>10.0/4.4</td>
    <td>7.1/4.2</td>
    <td>4.3/0.4</td>
    <td>3.7/0.3</td>
    <td>22.1/7.4</td>
  </tr>
  <tr>
    <td>Qwen2-Audio-7B</td>
    <td>4.6/2.7</td>
    <td>36.5/1.6</td>
    <td>44.8/3.5</td>
    <td>31.7/4.6</td>
    <td>37.5/1.4</td>
    <td>47.4/3.2</td>
    <td>5.1/1.3</td>
    <td>33.3/0.7</td>
    <td>23.3/7.0</td>
    <td>5.9/5.8</td>
    <td>2.3/1.3</td>
    <td>2.0/0.7</td>
    <td>25.5/22.8</td>
  </tr>
  <tr>
    <td>seamless-msl-v2-large</td>
    <td>15.9/5.4</td>
    <td>55.2/5.8</td>
    <td>43.5/1.0</td>
    <td>50.5/6.7</td>
    <td>75.1/5.0</td>
    <td>49.5/2.1</td>
    <td>6.0/1.6</td>
    <td>34.7/2.8</td>
    <td>14.9/1.4</td>
    <td>5.3/1.6</td>
    <td>3.6/1.5</td>
    <td>2.7/0.4</td>
    <td>37.5/33.5</td>
  </tr>
</table>

Table 13: Character Error Rate (CER) and hallucination error rate (HER) across models and datasets. Values are presented as CER/HER.
Figure 7: Degradation in WER (blue, left y-axis) and HER (green, right y-axis) w.r.t distribution shift (x-axis), measured using Central Moment Discrepancy (CMD) for all models.
<table>
  <tr>
    <th>Model</th>
    <th>SP2G1 BERSL ATCOsim ADV AMI SLU SN IPS SC GLOBE SALT LS Noise LS Prime57</th>
    <th>all datasets</th>
  </tr>
  <tr>
    <td>whisper-large-v3</td>
    <td>0.12 0.49 0.27 1.49 0.43 0.51 0.11 0.82 0.74 0.34 0.16 0.14 0.23</td>
    <td></td>
  </tr>
  <tr>
    <td>wav2vec2-large-xlsr-53-english 0.05 0.29 0.31 0.96 0.43 0.14 0.10 0.47 0.38 0.12 0.07 0.00 0.29</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>hf-seamless-m4t-large 0.24 0.59 0.81 1.26 0.73 0.53 0.43 0.88 1.11 0.37 0.49 0.15 0.62</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>speekthlm-1.5B 0.42 0.56 0.47 0.99 0.43 0.20 0.27 0.93 1.03 0.44 0.48 0.37 0.41</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>whisper-medium 0.05 0.47 0.28 1.47 0.53 0.50 0.16 0.77 0.58 0.74 0.28 0.13 0.30</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>distil-large-v2 0.25 0.42 0.43 1.41 0.51 0.45 0.14 0.76 0.48 0.20 0.20 0.15 0.27</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>hubert-large-ls960-ft 0.11 0.24 0.22 0.91 0.66 0.11 0.10 0.68 0.30 0.20 0.37 0.05 0.37</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>distil-med1um.en 0.13 0.44 0.48 1.40 0.36 0.43 0.14 0.73 0.40 0.49 0.40 0.21 0.27</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>distil-small.en 0.13 0.41 0.54 1.35 0.33 0.51 0.20 0.85 0.60 0.53 0.22 0.15 0.30</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>whisper-meditum.en 0.23 0.51 0.34 1.40 0.49 0.20 0.18 0.72 0.47 0.64 0.26 0.23 0.33</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>whisper-small.en 0.22 0.51 0.42 1.43 0.38 0.51 0.21 0.87 0.67 0.89 0.24 0.08 0.31</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>hf-seamless-m4t-medium 0.33 0.56 0.96 1.31 0.77 0.50 0.24 0.87 1.01 0.33 0.26 0.13 0.57</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>whisper-tiny 0.41 0.34 0.58 1.02 0.65 0.55 0.35 0.82 0.93 0.68 0.66 0.22 0.51</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>whisper-large 0.16 0.33 0.28 1.50 0.57 0.50 0.12 0.80 0.59 0.65 0.23 0.03 0.28</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>whisper-large-x2 0.19 0.52 0.29 1.48 0.65 0.62 0.13 0.66 0.81 0.33 0.22 0.10 0.34</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>whisper-large+x3-turbo 0.12 0.46 0.28 1.44 0.42 0.47 0.12 0.75 0.44 0.55 0.19 0.16 0.23</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>whisper-tiny.en 0.35 0.45 0.53 1.08 0.56 0.55 0.35 0.75 0.97 0.62 0.53 0.18 0.47</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>distil-large-v3 0.03 0.41 0.37 1.45 0.44 0.40 0.11 0.78 0.43 0.32 0.27 0.14 0.23</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>whisper-small-en 0.14 0.46 0.40 0.98 0.36 0.47 0.14 0.74 0.64 0.67 0.21 0.08 0.34</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>Qwen2-Audio-7B 0.74 0.52 0.96 1.55 0.38 0.66 0.26 1.18 0.33 1.16 0.67 0.39 1.01</td>
    <td></td>
    <td></td>
  </tr>
  <tr>
    <td>seamless-m4t-v2-large 0.41 0.55 0.90 1.42 0.71 0.49 0.29 0.71 1.13 0.40 0.41 0.19 0.71</td>
    <td></td>
    <td></td>
  </tr>
</table>

Table 14: Comparison of HER/WER ratio across models for all datasets.

<table>
  <tr>
    <th colspan="6">Table 15: Non-Hallucination error analysis across various datasets and models.</th>
    <th colspan="8">The table shows the percentage of Phonetic (P), Oscillation (O), and Language (L) errors for each model evaluated on different datasets. Abbreviations.</th>
  </tr>
  <tr>
    <th>w – whisper, s – small, m – medium, l – large, t – turbo, dw – distil-whisper, sm4t – seamless, w2v2 – wav2vec2, spllm – SpeechLLM, Qwen2 – Q2A - Qwen2-Audio, SC - Supreme Court.</th>
    <td colspan="13"></td>
  </tr>
</table>
![Finegrained vs coarsegrained error rate distribution averaged across all models and datasets.](page_232_186_495_377.png)

Figure 8: Finegrained vs coarsegrained error rate distribution averaged across all models and datasets.

<table>
  <tr>
    <th>Attribute</th>
    <th>Value</th>
  </tr>
  <tr>
    <td><b>Reference</b></td>
    <td>lufthansa four three nine three descend to flight level two seven zero</td>
  </tr>
  <tr>
    <td><b>Transcription</b></td>
    <td>Lufthansa 4393, descent flight level 270.</td>
  </tr>
  <tr>
    <td><b>WER</b></td>
    <td>75.0</td>
  </tr>
  <tr>
    <td><b>Hallucination</b></td>
    <td>No Error</td>
  </tr>
</table>

Table 16: WER and error category labeled by LLMs for whisper-medium.