On the Robust Approximation of ASR Metrics
Authors: Abdul Waheed, Hanin atwany, Rita Singh, Bhiksha Raj
Accepted at: acl25

On the Robust Approximation of ASR Metrics

Abdul Waheed1    Hanin Atwany2    Rita Singh1    Bhiksha Raj1
1Carnegie Mellon University    2MBZUAI
{abdulw,bhiksha,rsingh}@cs.cmu.edu    hanin.atwany@mbzuai.ac.ae

Abstract

Recent advances in speech foundation models are largely driven by scaling both model size and data, enabling them to perform a wide range of tasks, including speech recognition. Traditionally, ASR models are evaluated using metrics like Word Error Rate (WER) and Character Error Rate (CER), which depend on ground truth labels. As a result of limited labeled data from diverse domains and testing conditions, the true generalization capabilities of these models beyond standard benchmarks remain unclear. Moreover, labeling data is both costly and time-consuming. To address this, we propose a novel label-free approach for approximating ASR performance metrics, eliminating the need for ground truth labels. Our method utilizes multimodal embeddings in a unified space for speech and transcription representations, combined with a high-quality proxy model to compute proxy metrics. These features are used to train a regression model to predict key ASR metrics like Word Error Rate (WER) and Character Error Rate (CER). We experiment with over 40 models across 14 datasets representing both standard and in-the-wild testing conditions. Our results show that we approximate the metrics within a single-digit absolute difference across all experimental configurations, outperforming the most recent baseline by more than 50%.

1 Introduction

Automatic Speech Recognition (ASR) models have made significant advancements in recent years, achieving near-human performance on several standard evaluation benchmarks (Radford et al., 2022; Seamless Communication et al., 2023; Communication et al., 2023; Harper et al., 2024, inter alia). These models are typically evaluated using metrics like Word Error Rate (WER) and Character Error Rate (CER) (Likhomanenko et al., 2020), which are essential for assessing model performance.

However, these metrics are dependent on ground truths, which are often scarce in resource-constrained environments, and human labeling is both costly and time-consuming. To mitigate this challenge, several reference-free evaluation methods are proposed (Yuksel et al., 2023b; Kalgaonkar et al., 2015; Swarup et al., 2019; Qiu et al., 2021; Del-Aguia et al., 2018; Raj et al., 2011). While these approaches eliminate the reliance on labeled data, they primarily offer relative assessments of transcription quality, rather than providing precise error counts or rates. As a result, their applicability in real-world scenarios, where actionable performance metrics are crucial for further model refinement and deployment, is limited.

Given the limitations of both methods, approximating ASR metrics has emerged as a promising alternative for label-free evaluation (Chowdhury and Ali, 2023; Sheshadri et al., 2021b; Ali and Renals, 2018). This approach typically involves training regression (Jalalvand et al., 2016) and/or classification models (Sheshadri et al., 2021a) on top of speech and text encoders. While this method offers a close approximation of error metrics, several important questions remain unresolved. Specifically, an approximation model trained on dataset sampled from \( D \) to predict ASR metrics for a source model \( M \) must be evaluated under diverse conditions: 1) on test data that is IID (independent and identically distributed) sampled from \( D \); 2) on out-of-distribution (OOD) data representing diverse domains and recording conditions; 3) on IID data, but transcription from a target model \( T \); and 4) on OOD data with transcriptions from a target model \( T \). Most prior works (Chowdhury and Ali, 2023; Sheshadri et al., 2021b) focus primarily on the first condition. Moreover, recent advancements in multimodal foundation models offer new opportunities to directly train regression models on unified speech and text embeddings.

To address these critical research gaps, we pro-
pose a novel framework for approximating the performance of a wide range of ASR models, both on standard benchmarks and in-the-wild scenarios. Specifically, we compute the similarity between speech and text embeddings in a unified space, capturing the semantic alignment between the two modalities. Additionally, we incorporate a high-quality reference model as a proxy, based on the intuition that agreement with a reliable proxy correlates with transcription quality, as shown in prior works (Waheed et al., 2025). These features are then used to train a regression model to predict key ASR metrics, such as WER, CER, and absolute word and character error counts.

In summary, our work represents one of the most comprehensive studies to date on approximating ASR metrics at scale, in terms of both data and model coverage. Our proposed approach serves as a reference-free evaluation particularly suited for label-scarce scenarios. Beyond evaluation, our method is especially valuable for tasks such as pseudo-labeling, where high-quality transcriptions are essential for downstream applications like knowledge distillation (Waheed et al., 2024; Gandhi et al., 2023).

Our contributions are as follows:

• We evaluate over 40 ASR models across 14 diverse evaluation setups, including both standard benchmarks and domain-specific, unseen conditions, followed by training regression models to approximate ASR metrics.

• We compare our approach with the most recent work on approximating ASR metrics and show over a 50% reduction in absolute difference against the strong baseline.

• We conduct a rigorous ablation study to analyze the impact of different experimental configurations, providing deeper insights into the robustness of our approach. Our findings show that our method is resilient to diverse evaluation setups and requires only a small amount of training data.

Outline. The remainder of this paper is organized as follows: Section 2 reviews related work. Section 3 presents our proposed methodology. Sections 4 and 5 detail our experimental setup, results, and ablation study, respectively. Section 6 concludes the paper and outlines future directions.

2 Related Work

Automatic speech recognition (ASR) has witnessed significant advancements in recent years, primarily due to the scaling of both data and model size (Radford et al., 2022; Communication et al., 2023). Transformer (Vaswani et al., 2023) based models, in particular, have significantly contributed to these developments by effectively capturing long-range dependencies and contextual nuances in speech, achieving state-of-the-art (SOTA) performance across diverse benchmarks (Kheddar et al., 2024; Dhanjal and Singh, 2024; Zimerman and Wolf, 2023). While traditional evaluation metrics like Word Error Rate (WER) and Character Error Rate (CER) are de-facto evaluation metrics in benchmarking ASR systems (Lin et al., 2021; Park et al., 2024), scenarios where ground truth transcriptions are unavailable have caught interest in reference-free ASR evaluation methods (Karbasi and Kolossa, 2022; Wang et al., 2024; Kuhn et al., 2024).

Reference-free ASR evaluation methods aim to estimate ASR performance without requiring ground truth transcriptions (Ospanov et al., 2024). Earlier approaches rely on heuristic features or metadata such as speaker demographics, background noise, and linguistic characteristics (Litman et al., 2000; Yoon et al., 2010), limiting their applicability across varied contexts. However, recent advancements focus on deep learning-based frameworks, such as convolutional neural networks (CNNs) (Elloumi et al., 2018) and contrastive learning methods (Yuksel et al., 2023a), to predict ASR quality directly from encoded speech and text. For instance, methods like NoRefER (Yuksel et al., 2023b) use Siamese architectures fine-tuned on ASR hypotheses, achieving high correlation with traditional metrics and improving WER by optimizing hypothesis ensembling (Park et al., 2024).

Efforts to approximate ASR metrics explore hybrid approaches that combine traditional and reference-free methods, such as leveraging word confidence scores, linguistic embeddings, or post-processing adaptations to estimate WER and CER without explicit references (Ali and Renals, 2020, 2018; Kuhn et al., 2024; Negri et al., 2014). However, these approaches often suffer from reliance on specific ASR models or domain characteristics, limiting their generalizability. Unlike existing methods, our work addresses these limitations by introducing a robust, model and data-agnostic frame-
work that evaluates ASR outputs across diverse datasets and configurations, emphasizing adaptability to unseen domains and variations.

3 Methodology

We present a scalable and robust method to approximate ASR performance metrics using multi-modal unified embeddings, proxy references, and regression models. The primary goal is to eliminate reliance on ground-truth labels, enabling performance evaluation in label-scarce scenarios. The pipeline consists of three components: representation similarity in a unified speech-text embedding space, agreement with a high-quality proxy reference, and a regression model trained on these features to predict ASR metrics. Our pipeline diagram is shown in Figure 1.

3.1 Similarity in Unified Representation Space

The foundation of our approach is the SONAR model (Duquenne et al., 2023), a state-of-the-art multimodal (speech-text) model trained to produce unified embeddings for both speech and text inputs. Let \( x_{\text{speech}} \) represent the input speech signal and \( x_{\text{text}} \) denote the corresponding ASR-generated transcription. SONAR maps these inputs to a shared embedding space, generating \( e_{\text{speech}} \) and \( e_{\text{text}} \):

\[
e_{\text{speech}} = f_{\text{SONAR}}(x_{\text{speech}}), \quad e_{\text{text}} = f_{\text{SONAR}}(x_{\text{text}})
\]

where \( f_{\text{SONAR}} \) represents the embedding model. The alignment between these embeddings is quantified using cosine similarity:

\[
\text{Similarity}(x_{\text{speech}}, x_{\text{text}}) = \frac{e_{\text{speech}} \cdot e_{\text{text}}}{||e_{\text{speech}}|| ||e_{\text{text}}||}
\]

The similarity metric serves as an indicator of transcription quality, with higher values suggest better alignment between speech and text representations.

3.2 Agreement with a Proxy Reference

To complement the similarity score, we utilize proxy references generated by a high-quality ASR model, denoted as \( x_{\text{proxy}} \). The comparison between the ASR-generated transcription \( x_{\text{text}} \) and the proxy reference \( x_{\text{proxy}} \) is quantified using Word Error Rate (pWER) and Character Error Rate (pCER) as defined in Appendix A.1.

These metrics assess transcription quality by comparing it with a reliable proxy reference, without using ground-truth labels at any stage. Proxy references are dynamically selected by profiling 41 models across datasets and ranking them by average performance. For each target ASR model, the reference is the highest-ranking model other than the target itself. For example, if whisper-large-v3 ranks highest, the reference for whisper will be the second-best model. This ensures the proxy reference is both relevant and reliable for evaluating the target model.

\[
\hat{y} = a\{\text{metrics}\}
\]

![High-level diagram for our framework. The proxy is an ASR model that takes input speech and generates a transcription. We use the output from the source model as a hypothesis, and the output from the proxy model as a reference, to calculate metrics like WER and CER (pmetrics), which we denote as pWER and pCER. We then use this, along with the the similarity between SONAR embeddings of the input speech and the hypothesis, to train the regression that gives approximated metrics (ametrics), e.g., aWER/aCER.](page_1016_682_377_377.png)

Figure 1: High-level diagram for our framework. The proxy is an ASR model that takes input speech and generates a transcription. We use the output from the source model as a hypothesis, and the output from the proxy model as a reference, to calculate metrics like WER and CER (pmetrics), which we denote as pWER and pCER. We then use this, along with the the similarity between SONAR embeddings of the input speech and the hypothesis, to train the regression that gives approximated metrics (ametrics), e.g., aWER/aCER.

3.3 Regression Model for Metric Prediction

The extracted features, including similarity scores and proxy metrics, are concatenated to form the input to a regression model. Let \( z = [\text{Similarity}, \text{pWER}/\text{pCER}] \) represent the feature vector. The regression model \( g \) estimates the ASR metrics \( \hat{y} \), denoted as aWER and/or aCER:

\[
\hat{y} = g(z)
\]

The regression model is an ensemble of Random
Forest, Gradient Boosting, and Histogram-based Gradient Boosting regressors. Each base model is fine-tuned via grid search for hyperparameter optimization. The ensemble is trained to minimize the mean absolute error between predicted and ground-truth metrics. Additionally, a ridge regression model with non-negativity constraints is included in the ensemble to ensure predictions remain within valid ranges. Additional details of our regression pipeline are provided in Section 4, with hyperparameter details in Appendix A.4.

3.4 Evaluation

We evaluate the regression model’s performance across four setups, including IID and OOD data and different model configurations. Specifically, we train our regression model on one ASR system (source) on one dataset and evaluate it on both IID and OOD data for the source and target models. We provide a detailed analysis of the distribution shift in Appendix A.5.

Let \( \mathcal{D}_{M,B} \) denote the 10 benchmark datasets, and \( \mathcal{D}_{M,W} \) represent the four in-the-wild datasets, as described in Section 4.1, where \( M \in \{S, T\} \) refers to either the source model \( S \) or the target model \( T \).

The regression model is trained on data \( \mathcal{D}_{S,B}^{\text{train}} \sim \mathcal{D}_{S,B} \) and evaluated on the IID test set \( \mathcal{D}_{S,B}^{\text{test-IID}} \sim \mathcal{D}_{S,B} \), consisting of 80% and 20% of the data, respectively. Additionally, the model is evaluated on \( \mathcal{D}_{T,B}^{\text{test-IID}} \), \( \mathcal{D}_{S,W} \), and \( \mathcal{D}_{T,W} \). Below, we detail the formulation of each evaluation setup.

Case 1: IID Evaluation (Source \( S \)) The regression model is trained on \( \mathcal{D}_{S,B}^{\text{train}} \) and evaluated on \( \mathcal{D}_{S,B}^{\text{test-IID}} \). Let \( x_1^S = f(s, o^S) \) represent the similarity between input speech \( s \) and the ASR output \( o^S \), and \( x_2^S = g(o^S, r) \) represent the agreement with the proxy reference \( r \), where \( o^S \) is the ASR output produced by the source model \( S \). The evaluation is formulated as:

\[
\mathcal{L}_{\text{IID}}^S = \mathbb{E}_{(x_1^S, x_2^S, y) \sim \mathcal{D}_{S,B}^{\text{test-IID}}} [\mathcal{L}(h(x_1^S, x_2^S), y)]
\] (4)

Case 2: IID Evaluation (Target \( T \)) The regression model trained on \( \mathcal{D}_{S,B}^{\text{train}} \) is evaluated on the IID test set \( \mathcal{D}_{T,B}^{\text{test-IID}} \). Let \( x_1^T = f(s, o^T) \) represent the similarity between input speech \( s \) and the ASR output \( o^T \), and \( x_2^T = g(o^T, r) \) represent the agreement with the proxy reference \( r \), where \( o^T \) is the ASR output produced by the target model \( T \). The evaluation is expressed as:

\[
\mathcal{L}_{\text{IID}}^T = \mathbb{E}_{(x_1^T, x_2^T, y) \sim \mathcal{D}_{T,B}^{\text{test-IID}}} [\mathcal{L}(h(x_1^T, x_2^T), y)]
\] (5)

Case 3: OOD Evaluation (Source \( S \)) The regression model trained on \( \mathcal{D}_{S,B}^{\text{train}} \) is evaluated on the out-of-distribution set \( \mathcal{D}_{S,W} \). Let \( x_1^S = f(s, o^S) \) represent the similarity between the input speech \( s \) and the ASR output \( o^S \), and \( x_2^S = g(o^S, r) \) represent the agreement with the proxy reference \( r \), where \( o^S \) is the ASR output produced by the source model \( S \). The evaluation is defined as:

\[
\mathcal{L}_{\text{OOD}}^S = \mathbb{E}_{(x_1^S, x_2^S, y) \sim \mathcal{D}_{S,W}} [\mathcal{L}(h(x_1^S, x_2^S), y)]
\] (6)

Case 4: OOD Evaluation (Target \( T \)) The regression model trained on \( \mathcal{D}_{S,B}^{\text{train}} \) is evaluated on the out-of-distribution set \( \mathcal{D}_{T,W} \), using the ASR output produced by the target model \( T \). Let \( x_1^T = f(s, o^T) \) represent the similarity between the input speech \( s \) and the ASR output \( o^T \), and \( x_2^T = g(o^T, r) \) represent the agreement with the proxy reference \( r \), where \( o^T \) is the ASR output produced by the target model \( T \). The evaluation is expressed as:

\[
\mathcal{L}_{\text{OOD}}^T = \mathbb{E}_{(x_1^T, x_2^T, y) \sim \mathcal{D}_{T,W}} [\mathcal{L}(h(x_1^T, x_2^T), y)]
\] (7)

Note. For computational feasibility, the primary experiments train the regression model on 9 out of the 10 datasets in \( \mathcal{D}_{S,B}^{\text{train}} \) and evaluate it on the remaining dataset, as well as on all four datasets in \( \mathcal{D}_{S,B}^{\text{OOD}} \). This process is repeated for each dataset in \( \mathcal{D}_{S,B}^{\text{train}} \), ensuring robust evaluation across various testing conditions. No examples from \( \mathcal{D}_{M,\text{OOD}} \) are used at any stage for training the regression model.

4 Experiments

In this section, we present the experimental setup to evaluate our ASR metrics approximation tool. We describe the datasets, models, and regression pipeline used in our experiments, highlighting the diversity of ASR systems and testing conditions.

4.1 Datasets

To evaluate the robustness and generalizability of our ASR metrics approximation tool, we use datasets sourced from multiple distributions, divided into two types: Standard Benchmark and Wild Challenge datasets. We describe these datasets below and provide additional details in Appendix A.2, Table 5.

Standard Benchmark Datasets. We include widely used datasets representing diverse domains and acoustic conditions. LibriSpeech (Panayotov et al., 2015) provides 1,000 hours of English read audiobooks, covering both clean and noisy
conditions. TED-LIUM (Rousseau et al., 2014) consists of TED talks from 2,000 speakers. Gigaspeech (Chen et al., 2021) spans audiobooks, podcasts, and YouTube, incorporating both read and spontaneous speech. SPGIspeech (Technologies, 2021) features 5,000 hours of earnings calls with a focus on orthographic accuracy. Common Voice (Ardila et al., 2020) is a multilingual, crowdsourced corpus with diverse accents. Earnings22 (Rio et al., 2022) provides 119 hours of accented, real-world earnings calls. Additional datasets include AMI (IHM) (Carletta et al., 2005), with 100 hours of English meeting recordings from non-native speakers, and People’s Speech (Galvez et al., 2021), emphasizing inclusivity and linguistic diversity. SLUE-VoXCeleb (Shon et al., 2022) contains conversational voice snippets, capturing diverse speaking styles and emotions.

Wild Datasets. The wild set focuses on real-world variability and challenging scenarios. Pri-mock57 (Papadopoulos Korfiatis et al., 2022) includes telemedicine consultations with diverse accents, ages, and scenarios, recorded by clinicians and actors. VoxPopuli Accented (Wang et al., 2021) contains multilingual speeches from European Parliament recordings, rich in named entities. AT-COSim (Jan van Doorn, 2023) features 10 hours of non-native English speech from air traffic control simulations with clean utterance-level transcriptions. Additionally, we include a noisy subset of LibriSpeech (Panayotov et al., 2015), which reflects challenging real-world conditions.

In addition to the above datasets, we also run a small experiment to assess the cross-lingual transferability of the trained regression model. Specifically, we train the model on English data and evaluate it on both German and English data, and vice versa, using the English and German splits from LibriSpeech (Panayotov et al., 2015) and Common Voice (Ardila et al., 2020). In our ablation study, we compare the trained regression model with a proxy reference. To broaden this evaluation, we expand the in-the-wild dataset to include four additional datasets: a privately collected Medical ASR dataset with clinical conversations; standard data with eight synthetic perturbations (white noise, time stretch, pitch shift, cross-lingual noise, reverberation, pub noise, echo, and distortion); noisy home recordings BERSt (Tuttösf et al., 2025); and CHiME-6(noisy subset) (Watanabe et al., 2020).

4.2 Models

We evaluate our approximation framework for a range of state-of-the-art ASR models, put into three categories based on their architecture and functionality. Below we describe the datasets and provide additional details in Appendix A.2 and in Table 6.

Encoder-Decoder Models. We include multiple encoder-decoder families of models capable of performing ASR tasks in a zero-shot setting. More specifically, we include whisper (Radford et al., 2023) and distil-whisper (Gandhi et al., 2023) models that perform really well across diverse testing settings. We also include seamless (Communication et al., 2023; Seamless Communication et al., 2023; Barrault et al., 2025), SpeechT5 (Ao et al., 2022) which are unified encoder-decoder frameworks for tasks such as ASR, speech synthesis, translation, and voice conversion. MMS (Pratap et al., 2023) supports hundreds of languages and excels in resource-constrained scenarios. Moonshine (2) (Jeffries et al., 2024), a lightweight and efficient model, is designed for edge deployments with strong performance. Additionally, we include speech language models like SpeechLLM (Rajaa and Tushar), which combine speech embeddings with language models to predict metadata such as speaker attributes, emotions, and accents, offering robust multimodal capabilities.

NeMo-ASR Models. We use multiple models from the NeMo-ASR (Gulati et al., 2020; Variani et al., 2020; Noroozi et al., 2024; Tang et al., 2023; Harper et al., 2024) toolkit by NVIDIA. We include models such as Canary and Parakeet, which use highly efficient speech encoders like Fast-Conformer (Rekesh et al., 2023). In addition to that, we use models based on various encoders and decoders (CTC, RNN-T, TDT, Conformer-CTC (Guo et al., 2021)). In our work, we evaluate 11 models from the NeMo-ASR toolkit.

Encoder-Only. We include self-supervised encoder-only models and their derivatives. Specifically, we use Wav2Vec2 (Schneider et al., 2019; Baevski et al., 2020), HuBERT (Hsu et al., 2021), and Data2Vec (Baevski et al., 2022).

4.3 Experimental Setup

We evaluate all models listed in Section 4.2 on 1000 examples sampled randomly from the test split of each dataset, as described in Section 4.1. Since all models are trained at a 16 kHz sampling rate, we (re)sample the speech accordingly. For ASR,
we use greedy decoding and all other parameters are default unless otherwise specified. We apply basic text post-processing \( ^1 \) before computing ASR metrics. We obtain all models from Huggingface Hub \( ^2 \) and implement the ASR pipeline using the Transformers (Wolf et al., 2020) library.

For multimodal embeddings, we use SONAR (Duquenne et al., 2023), a 1024-dimensional sentence-level multilingual model. Specifically, we utilize text_sonar_basic_encoder for text encoding and speech_sonar_basic_encoder for speech encoding.

The regression framework uses a stacking ensemble with base regressors and a final estimator. Hyperparameter tuning is performed with RandomizedSearchCV to minimize MAE. The model is trained on 9 benchmark datasets and evaluated on the remaining benchmark dataset and four in-the-wild datasets. This process is repeated for all 10 benchmark datasets. Additional details of the regression pipeline are provided in Section 3 and low-level details in Appendix A.4.1.

We conduct ASR experiments on a single A100/H100 GPU, while the regression model training runs on CPUs. Although ASR time and memory consumption depend on the model size, embedding extraction for 1000 audio-text pairs takes approximately one minute on a single consumer-grade GPU without parallelization or additional efficiency measures. Appendix A.4 provides further experimental setup details.

Baselines. Recent studies directly aligned with our approach are limited. For instance, eWER (Ali and Renals, 2018) and eWER2 (Ali and Renals, 2020) estimate error rates based on the input signal, which differs from our approach. In contrast, we incorporate the model’s output transcript into the error rate approximation function. The most closely related recent works are WERBERT (Sheshadri et al., 2021a) and eWER3 (Chowdhury and Ali, 2023), which share a similar pipeline. Both use encoders for text, speech, and other data, followed by a regression model trained in an end-to-end setting. Since eWER3 is the more recent of the two, we use it as our baseline. In eWER3, the speech encoder is wav2vec2 (Baevski et al., 2020), and the text encoder is roberta-base (Liu et al., 2019), with a regression model trained on top while both encoders remain frozen. Given the unavailability of public code or pretrained models for evaluation, we implement eWER3 with some modifications to ensure a fair comparison. Specifically, we extract features from both encoders and apply PCA for dimensionality reduction on each modality before training our regression pipeline. For both speech and text, we experiment with 32 and 64 PCA components (referred to as \( nc \) in Table 3).

5 Results

We conduct experiments using two dataset categories: standard benchmarks and in-the-wild, as described in Section 4.1. For each ASR model, a leave-one-out strategy is used, training the regression model on 9 benchmark datasets and testing it on the remaining benchmark dataset and all four in-the-wild datasets to ensure comprehensive evaluation on out-of-domain data. Additionally, in-domain testing is included in ablation studies, as detailed in Section 5.4. The regression model is trained to predict absolute error counts (word and character levels), which are normalized by the reference length to compute approximate error rates (\( aWER \) and \( aCER \)). We also train regression models to directly predict WER and CER. We provide results for fine-grained metrics in Table 12.

5.1 Evaluation on In-the-Wild Datasets

The wild datasets provide a realistic testbed for evaluating the regression model’s ability to approximate error rates under real-world conditions. As shown in Table 1, high-performing models, like canary-1b, demonstrate strong agreement between predicted and actual error rates. For example, on VP_Accented, canary-1b achieves mean absolute difference of 1.1%. On Primock57, the model shows robustness with a WER of 16.2% and an \( aWER \) of 13.4%, highlighting its effective generalization across diverse and domain-specific contexts.

For models like data2vec-audio-large-960h our approximation is pretty close to actual error rates with difference consistently under 2% on various datasets. For example, on LibriSpeech-test-noise, the model’s actual WER is 7.2% while the approximated \( aWER \) is 8.6%. Even on acoustically complex datasets like ATCOsim, where the WER is 44.0% and the \( aWER \) is 51.1%, the model exhibits a reasonable alignment between approximated and actual error rates.
In contrast, models with high actual error rates, such as mms-1b-ft102, show slightly larger deviations, particularly on datasets with challenging conditions. For instance, on ATCOsim, the WER is 93.4% and the aWER is 99.0%, resulting in a significant deviation of 5.6%, the highest observed across all in-the-wild datasets. Similarly, on Pri-mock57, where the WER is 70.2% and the aWER is 67.8%, the approximation also struggles to align due to the inherently high error rates. This highlights that extreme error cases often correspond to semantically nonsensical outputs, where the distinction between high and extremely high error rates becomes less relevant.

<table>
  <tr>
    <th>Model</th>
    <th>LS_Noise</th>
    <th>Primock57</th>
    <th>ATCOsim</th>
    <th>VP_Acc</th>
  </tr>
  <tr><td>w2v2-ls</td><td>8.8/10.2</td><td>32.8/35.6</td><td>43.0/49.5</td><td>20.4/26.4</td></tr>
  <tr><td>can-1b</td><td>4.1/6.4</td><td>16.2/13.4</td><td>30.4/35.5</td><td>23.2/12.1</td></tr>
  <tr><td>d2v-base</td><td>14.9/16.4</td><td>39.6/41.7</td><td>66.0/71.2</td><td>28.4/33.8</td></tr>
  <tr><td>d2v-large</td><td>7.2/8.6</td><td>28.3/30.7</td><td>44.0/51.1</td><td>21.4/26.5</td></tr>
  <tr><td>distil-l-v2</td><td>7.3/9.2</td><td>18.3/13.0</td><td>69.5/66.7</td><td>14.9/14.5</td></tr>
  <tr><td>distil-l-v3</td><td>6.1/8.3</td><td>18.4/12.9</td><td>69.0/63.6</td><td>14.8/14.0</td></tr>
  <tr><td>distil-s-en</td><td>9.1/10.6</td><td>19.3/14.7</td><td>74.9/69.1</td><td>14.6/14.7</td></tr>
  <tr><td>sm4t-l</td><td>11.2/12.3</td><td>41.7/37.8</td><td>75.0/82.5</td><td>29.3/19.9</td></tr>
  <tr><td>sm4t-m</td><td>14.9/15.6</td><td>44.1/39.7</td><td>54.6/60.4</td><td>30.5/22.5</td></tr>
  <tr><td>hub-l-1s-ft</td><td>7.3/8.8</td><td>29.5/32.0</td><td>50.4/56.9</td><td>21.4/26.6</td></tr>
  <tr><td>hub-xl-1s-ft</td><td>6.8/8.3</td><td>31.1/32.9</td><td>46.7/53.0</td><td>21.8/27.7</td></tr>
  <tr><td>mms-1b-a</td><td>9.5/11.1</td><td>36.2/34.4</td><td>63.4/71.8</td><td>29.9/23.8</td></tr>
  <tr><td>mms-1b-ft102</td><td>24.0/24.9</td><td>70.2/67.8</td><td>93.4/99.0</td><td>39.4/38.2</td></tr>
  <tr><td>moon-b</td><td>11.3/12.4</td><td>19.9/18.5</td><td>65.6/66.2</td><td>17.1/20.8</td></tr>
  <tr><td>moon-t</td><td>15.5/17.4</td><td>29.2/29.5</td><td>62.9/68.5</td><td>22.1/26.2</td></tr>
  <tr><td>par-ctc-0.6b</td><td>4.6/7.4</td><td>16.3/13.8</td><td>32.9/42.9</td><td>16.3/13.8</td></tr>
  <tr><td>par-ctc-1.1b</td><td>4.5/6.9</td><td>16.6/14.1</td><td>30.9/39.9</td><td>16.4/12.4</td></tr>
  <tr><td>par-rmt-0.6b</td><td>3.8/6.9</td><td>16.3/13.2</td><td>31.6/41.8</td><td>17.3/12.6</td></tr>
  <tr><td>par-rmt-1.1b</td><td>3.5/6.1</td><td>14.6/13.3</td><td>27.3/37.6</td><td>18.1/10.4</td></tr>
  <tr><td>par-tdt-1.1b</td><td>3.4/6.0</td><td>13.5/13.2</td><td>28.3/35.7</td><td>17.9/10.2</td></tr>
  <tr><td>pkt-ctc-110m</td><td>6.1/8.6</td><td>16.7/13.0</td><td>39.9/42.4</td><td>19.2/12.5</td></tr>
  <tr><td>sm4t-v2-l</td><td>7.2/8.4</td><td>34.6/31.7</td><td>52.4/57.6</td><td>33.8/24.5</td></tr>
  <tr><td>spchilm-1.5B</td><td>15.3/16.6</td><td>42.0/41.8</td><td>121.1/125.457</td><td>59.3/59.3</td></tr>
  <tr><td>spchilm-2B</td><td>13.9/15.6</td><td>39.4/40.3</td><td>60.6/64.1</td><td>39.2/44.1</td></tr>
  <tr><td>stt-cfc-l</td><td>5.8/6.8</td><td>16.1/17.6</td><td>35.9/38.0</td><td>18.6/11.5</td></tr>
  <tr><td>stt-cfc-s</td><td>9.7/11.2</td><td>22.2/24.6</td><td>43.7/47.7</td><td>16.4/15.6</td></tr>
  <tr><td>stt-fc-cfc-l</td><td>6.8/10.0</td><td>17.6/23.9</td><td>34.9/47.6</td><td>18.9/13.3</td></tr>
  <tr><td>stt-fc-tdl-l</td><td>6.0/8.8</td><td>17.0/20.6</td><td>34.5/46.5</td><td>21.1/15.1</td></tr>
  <tr><td>w2v2-960h</td><td>17.4/18.5</td><td>44.7/47.1</td><td>68.4/74.0</td><td>29.9/36.5</td></tr>
  <tr><td>w2v2-crelpos</td><td>5.9/7.4</td><td>28.5/30.3</td><td>47.2/54.0</td><td>22.4/26.7</td></tr>
  <tr><td>w2v2-crope</td><td>6.6/8.1</td><td>31.7/33.4</td><td>49.8/56.9</td><td>21.9/26.3</td></tr>
  <tr><td>w2v2-1960h</td><td>11.6/12.6</td><td>37.8/40.2</td><td>66.4/72.7</td><td>26.3/33.3</td></tr>
  <tr><td>w2v2-1lv60-s</td><td>7.8/9.4</td><td>33.1/35.5</td><td>40.5/48.8</td><td>19.3/24.9</td></tr>
  <tr><td>w2v2-1rlf-ls</td><td>10.0/11.5</td><td>32.2/34.6</td><td>48.9/55.7</td><td>22.0/28.6</td></tr>
  <tr><td>whisper-l</td><td>6.2/8.1</td><td>18.8/13.9</td><td>65.3/66.9</td><td>18.7/15.9</td></tr>
  <tr><td>whisper-l-v2</td><td>5.4/6.6</td><td>19.0/13.1</td><td>64.8/74.8</td><td>20.0/18.1</td></tr>
  <tr><td>whisper-l-v3</td><td>4.6/5.9</td><td>18.7/12.0</td><td>64.7/73.9</td><td>19.2/18.1</td></tr>
  <tr><td>whisper-l-v3-t</td><td>4.9/6.0</td><td>18.5/12.3</td><td>66.0/72.5</td><td>24.3/23.2</td></tr>
  <tr><td>whisper-m.en</td><td>6.5/7.9</td><td>19.5/14.0</td><td>66.2/73.8</td><td>27.6/26.4</td></tr>
  <tr><td>whisper-s.en</td><td>8.2/9.7</td><td>20.0/15.1</td><td>67.1/73.8</td><td>17.3/17.5</td></tr>
  <tr><td>whisper-tiny</td><td>18.5/20.7</td><td>30.0/26.6</td><td>97.6/102.5</td><td>29.8/33.2</td></tr>
</table>

Table 1: Actual and approximated WER (↓), separated by a slash, on out-of-distribution wild datasets. The regression model is trained independently for each ASR model on standard benchmarks, making the wild datasets out-of-distribution. See Table 11 for full names.

5.2 Evaluation on Benchmark Datasets

We summarize results on 10 standard benchmark datasets in Appendix A.6 Tables 13 and 14. Each table reports actual WER/CER alongside the approximated WER/CER (denoted by aWER/aCER).

Overall, models such as parakeet-tdt-1.1b and whisper-large-v3 show relatively small differences between WER and aWER, indicating reliable approximations. For instance, the actual WER for whisper-large-v3 on AMI_IHM is 19.0% compared to aWER 17.1%, 1.9% gap. Conversely, some challenging datasets (e.g., CV11 and Earnings22) reveal larger discrepancies for specific models, particularly those with higher overall error rates. For example, mms-1b-ft102 exhibits a wide WER/aWER gap on Earnings22, suggesting difficulty handling accented or domain-specific speech.

In general, high-performing ASR models demonstrate small WER–aWER gaps, indicating that it’s easy to approximate when error rates are low. However, models with higher WERs or faced with more acoustically or linguistically challenging test sets tend to show wider divergences. Despite these variations, most results remain within a reasonable margin, highlighting the robustness of our approximation model on diverse out-of-distribution data.

![Bar charts comparing actual and approximated WER for four models across standard benchmark datasets](page_1012_1342_484_312.png)

Figure 2: Actual and approximated WER for four models across standard benchmark.

These results underscore the critical role of model quality in achieving reliable approximations. The approximation framework remains effective for high-performing models, while deviations tend to increase in cases of semantically divergent or poorly structured outputs, reflecting the inherent challenges in approximating errors for low-performing systems.
5.3 Multilingual Evaluation
We train the regression model on English and evaluate it on English and German, and vice versa. We do this experiment with two models as source and proxy, namely seamless-m4t-v2-large and whisper-large-v3. We report the mean absolute difference between approximated and actual word error counts in Table 2.

<table>
  <tr>
    <th colspan="2">Source: seamless-m4t-v2-large</th>
    <th colspan="2">Proxy: whisper-large-v3</th>
  </tr>
  <tr>
    <th>Train/Test</th>
    <th>LS_De</th>
    <th>CV17_De</th>
    <th>LS_En</th>
    <th>CV17_En</th>
  </tr>
  <tr>
    <td>LS_De</td>
    <td>–</td>
    <td>2.16</td>
    <td>1.59</td>
    <td>1.98</td>
  </tr>
  <tr>
    <td>CV17_De</td>
    <td>1.93</td>
    <td>–</td>
    <td>0.50</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>LS_En</td>
    <td>1.60</td>
    <td>0.75</td>
    <td>–</td>
    <td>0.66</td>
  </tr>
  <tr>
    <td>CV17_En</td>
    <td>1.82</td>
    <td>0.66</td>
    <td>0.56</td>
    <td>–</td>
  </tr>
</table>

<table>
  <tr>
    <th colspan="2">Source: whisper-large-v3</th>
    <th colspan="2">Proxy: seamless-m4t-v2-large</th>
  </tr>
  <tr>
    <th>Train/Test</th>
    <th>LS_De</th>
    <th>CV17_De</th>
    <th>LS_En</th>
    <th>CV17_En</th>
  </tr>
  <tr>
    <td>LS_De</td>
    <td>–</td>
    <td>2.03</td>
    <td>1.11</td>
    <td>1.96</td>
  </tr>
  <tr>
    <td>CV17_De</td>
    <td>1.74</td>
    <td>–</td>
    <td>0.51</td>
    <td>0.76</td>
  </tr>
  <tr>
    <td>LS_En</td>
    <td>1.29</td>
    <td>1.50</td>
    <td>–</td>
    <td>1.42</td>
  </tr>
  <tr>
    <td>CV17_En</td>
    <td>2.29</td>
    <td>0.68</td>
    <td>1.69</td>
    <td>–</td>
  </tr>
</table>

Table 2: Cross-lingual mean absolute difference (\( \downarrow \)) between predicted and actual word error counts. Lower values mean better approximation.

We find that our framework demonstrates strong cross-lingual generalization: when trained on English data, the regression model maintains low absolute differences when evaluated on German datasets, and vice versa. This consistency across languages and datasets, using both seamless-m4t-v2-large and whisper-large-v3 as source-proxy pairs, underscores the robustness and language-agnostic nature of our approach. These results validate that our method can be effectively applied in multilingual settings without the need for language-specific adaptations.

5.4 Ablation
We conduct comprehensive ablation experiments to evaluate the robustness of the approximation model and the contributions of its individual components. Using the evaluation setup outlined in Section 3.4, we select data2vec-audio-base-960h as the source model (S) and wav2vec2-base-960h as the target model (T). The results are summarized in Table 3, where IID results correspond to Case-I 3.4, and \( D, M \), and \( D + M \) under OOD represent Case II 3.4, Case-III 3.4, and Case-IV 3.4, respectively. The reference model’s \( r \) value represents the average WER across all datasets. We include reference models with varying \( r \) values, such as whisper-large-v3 (\( r = 17.8 \)), whisper-medium.en (\( r = 20.1 \)), whisper-tiny (\( r = 33.4 \)), and mms-1b-fl102 (\( r = 51.0 \)).

The results in Table 3 demonstrate the importance of proxy references in improving the regression model’s performance. Training without proxy references (w/o PR) significantly increases the mean absolute error (MAE) across all conditions. For instance, the IID MAE increases from 1.03 (Base) to 3.13, and the OOD \( D + M \) MAE rises from 1.07 (Base) to 3.33, highlighting the essential role of proxy references in approximation.

Increasing the number of high-quality proxy references (\( MPR \)) further reduces errors. Under IID conditions, the MAE decreases from 1.00 with \( n = 2 \) to 0.93 with \( n = 5 \). Similarly, in OOD \( D + M \), the error drops from 1.06 (\( MPR, n = 2 \)) to 0.95 (\( MPR, n = 5 \)), demonstrating that multiple high-quality references enhance model robustness.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">IID</th>
    <th colspan="3">OOD</th>
  </tr>
  <tr>
    <th>D</th>
    <th>M</th>
    <th>D + M</th>
  </tr>
  <tr>
    <td>eWER3(nc=32)</td>
    <td>2.03<sup>0.07</sup></td>
    <td>2.09<sup>0.04</sup></td>
    <td>2.06<sup>0.03</sup></td>
    <td>2.12<sup>0.04</sup></td>
  </tr>
  <tr>
    <td>eWER3(nc=64)</td>
    <td>1.98<sup>0.06</sup></td>
    <td>2.07<sup>0.05</sup></td>
    <td>2.00<sup>0.04</sup></td>
    <td>2.09<sup>0.05</sup></td>
  </tr>
  <tr>
    <td>Base</td>
    <td>1.03<sup>0.03</sup></td>
    <td>1.05<sup>0.01</sup></td>
    <td>1.05<sup>0.02</sup></td>
    <td>1.07<sup>0.01</sup></td>
  </tr>
  <tr>
    <td>w/o S</td>
    <td>1.04<sup>0.03</sup></td>
    <td>1.05<sup>0.01</sup></td>
    <td>1.04<sup>0.03</sup></td>
    <td>1.05<sup>0.01</sup></td>
  </tr>
  <tr>
    <td>w/o PR</td>
    <td>3.13<sup>0.07</sup></td>
    <td>3.22<sup>0.02</sup></td>
    <td>3.23<sup>0.05</sup></td>
    <td>3.33<sup>0.02</sup></td>
  </tr>
  <tr>
    <td>w/ MPR (n=2)</td>
    <td>1.00<sup>0.02</sup></td>
    <td>1.04<sup>0.02</sup></td>
    <td>0.99<sup>0.02</sup></td>
    <td>1.06<sup>0.02</sup></td>
  </tr>
  <tr>
    <td>w/ MPR (n=3)</td>
    <td>0.96<sup>0.02</sup></td>
    <td>0.97<sup>0.01</sup></td>
    <td>0.95<sup>0.02</sup></td>
    <td>0.99<sup>0.01</sup></td>
  </tr>
  <tr>
    <td>w/ MPR (n=4)</td>
    <td>0.95<sup>0.02</sup></td>
    <td>0.96<sup>0.02</sup></td>
    <td>0.94<sup>0.02</sup></td>
    <td>0.98<sup>0.02</sup></td>
  </tr>
  <tr>
    <td>w/ MPR (n=5)</td>
    <td>0.93<sup>0.02</sup></td>
    <td>0.93<sup>0.01</sup></td>
    <td>0.92<sup>0.02</sup></td>
    <td>0.95<sup>0.01</sup></td>
  </tr>
  <tr>
    <td>w/MPR (n=10)</td>
    <td>0.90<sup>0.02</sup></td>
    <td>0.93<sup>0.01</sup></td>
    <td>0.88<sup>0.02</sup></td>
    <td>0.95<sup>0.01</sup></td>
  </tr>
  <tr>
    <td>w/MPR (n=20)</td>
    <td>0.89<sup>0.02</sup></td>
    <td>0.96<sup>0.02</sup></td>
    <td>0.87<sup>0.02</sup></td>
    <td>0.96<sup>0.02</sup></td>
  </tr>
  <tr>
    <td>w/mMPR (n=3)</td>
    <td>0.98<sup>0.02</sup></td>
    <td>0.96<sup>0.02</sup></td>
    <td>0.97<sup>0.02</sup></td>
    <td>0.98<sup>0.02</sup></td>
  </tr>
  <tr>
    <td>w/mMPR (n=5)</td>
    <td>0.94<sup>0.02</sup></td>
    <td>0.94<sup>0.01</sup></td>
    <td>0.93<sup>0.01</sup></td>
    <td>0.96<sup>0.02</sup></td>
  </tr>
  <tr>
    <td>w/mMPR (n=10)</td>
    <td>0.92<sup>0.02</sup></td>
    <td>0.94<sup>0.02</sup></td>
    <td>0.91<sup>0.02</sup></td>
    <td>0.96<sup>0.02</sup></td>
  </tr>
  <tr>
    <td>w/mMPR (n=20)</td>
    <td>1.04<sup>0.02</sup></td>
    <td>1.05<sup>0.01</sup></td>
    <td>1.02<sup>0.02</sup></td>
    <td>1.04<sup>0.01</sup></td>
  </tr>
  <tr>
    <td>Base (r=17.8)</td>
    <td>1.31<sup>0.04</sup></td>
    <td>1.44<sup>0.02</sup></td>
    <td>1.31<sup>0.04</sup></td>
    <td>1.40<sup>0.01</sup></td>
  </tr>
  <tr>
    <td>Base (r=20.1)</td>
    <td>1.36<sup>0.04</sup></td>
    <td>1.36<sup>0.01</sup></td>
    <td>1.34<sup>0.03</sup></td>
    <td>1.34<sup>0.01</sup></td>
  </tr>
  <tr>
    <td>Base (r=33.4)</td>
    <td>1.55<sup>0.04</sup></td>
    <td>1.66<sup>0.02</sup></td>
    <td>1.55<sup>0.04</sup></td>
    <td>1.63<sup>0.02</sup></td>
  </tr>
  <tr>
    <td>Base (r=51.0)</td>
    <td>2.03<sup>0.02</sup></td>
    <td>2.10<sup>0.01</sup></td>
    <td>2.08<sup>0.05</sup></td>
    <td>2.09<sup>0.01</sup></td>
  </tr>
  <tr>
    <td>w/o S (r=17.8)</td>
    <td>1.47<sup>0.04</sup></td>
    <td>1.56<sup>0.01</sup></td>
    <td>1.48<sup>0.04</sup></td>
    <td>1.54<sup>0.01</sup></td>
  </tr>
  <tr>
    <td>w/o S (r=20.1)</td>
    <td>1.55<sup>0.02</sup></td>
    <td>1.50<sup>0.01</sup></td>
    <td>1.55<sup>0.03</sup></td>
    <td>1.50<sup>0.01</sup></td>
  </tr>
  <tr>
    <td>w/o S (r=33.4)</td>
    <td>1.79<sup>0.07</sup></td>
    <td>1.89<sup>0.02</sup></td>
    <td>1.78<sup>0.06</sup></td>
    <td>1.82<sup>0.02</sup></td>
  </tr>
  <tr>
    <td>w/o S (r=51.0)</td>
    <td>2.23<sup>0.02</sup></td>
    <td>2.24<sup>0.01</sup></td>
    <td>2.28<sup>0.04</sup></td>
    <td>2.21<sup>0.01</sup></td>
  </tr>
</table>

Table 3: Mean absolute error (\( \downarrow \)) between predicted word error count and actual error count (in absolute terms) across different configurations. PR - Proxy Reference, S - Similarity, MPR - Multiple PR, D - Data, M - Model. The OOD results are averaged across four wild datasets. n is the number of proxy references. The r (\( \downarrow \)) value represents the average WER for proxy reference across 14 datasets. Superscript represents the standard deviation across five runs.
The quality of references, quantified by the \( r \)-value, also plays a critical role. For example, in IID conditions, the MAE increases from 1.31 for \( r = 17.8 \) to 2.03 for \( r = 51.0 \). A similar trend is observed in OOD \( D + M \), where the MAE rises from 1.40 (\( r = 17.8 \)) to 2.09 (\( r = 51.0 \)). The absence of similarity (w/o S) combined with low-quality proxies further degrades performance, underscoring the importance of both high-quality references and similarity measures. We provide character-level error count approximation in Appendix A.6 Table 10.

Scaling Training Data for Regression. To evaluate the impact of training data size on the regression model, we scale the data from 1K to 10K examples in increments of 1K. As shown in Figure 3, the model’s performance does not exhibit a clear trend with increasing training data size. Some datasets show slight improvements with more data; others show minimal improvement. This suggests that the regression model is largely agnostic to the size of the training data. In fact, it appears that a relatively small dataset of just 1,000 examples is sufficient to train a robust approximation model. This underscores the model’s ability to generalize effectively with limited data, making it an efficient choice for scenarios with constrained datasets.

![Mean absolute error (↓) between predicted and actual word error counts across varying training data sizes for the regression model.](page_180_1042_1087_377.png)

Figure 3: Mean absolute error (↓) between predicted and actual word error counts across varying training data sizes for the regression model. The model is trained on 10 standard benchmarks and evaluated on four in-the-wild test sets.

Direct Comparison with Proxy Reference. We evaluate our regression model and standard reference-free baseline that computes the word error rate between the target and a proxy hypothesis. We report the mean absolute difference and compare the regression model (OURS) with direct calculation with proxy (W PROXY) in Table 4.

<table>
  <tr>
    <th rowspan="2">Dataset</th>
    <th colspan="2">PR (r=33.4)</th>
    <th colspan="2">PR (r=51.0)</th>
    <th colspan="2">w/o PROXY</th>
  </tr>
  <tr>
    <th>w PROXY</th>
    <th>OURS</th>
    <th>w PROXY</th>
    <th>OURS</th>
    <th>OURS</th>
  </tr>
  <tr>
    <td>Primock57</td>
    <td>2.25</td>
    <td>1.54</td>
    <td>4.89</td>
    <td>2.63</td>
    <td>3.82</td>
  </tr>
  <tr>
    <td>ATCOSim</td>
    <td>5.35</td>
    <td>2.11</td>
    <td>3.56</td>
    <td>2.02</td>
    <td>2.64</td>
  </tr>
  <tr>
    <td>VP_Accented</td>
    <td>2.99</td>
    <td>2.09</td>
    <td>3.39</td>
    <td>2.83</td>
    <td>4.00</td>
  </tr>
  <tr>
    <td>LS_noise</td>
    <td>2.10</td>
    <td>1.58</td>
    <td>3.48</td>
    <td>1.42</td>
    <td>2.35</td>
  </tr>
  <tr>
    <td>BERS</td>
    <td>2.37</td>
    <td>1.25</td>
    <td>5.09</td>
    <td>2.47</td>
    <td>3.20</td>
  </tr>
  <tr>
    <td>Perturbed</td>
    <td>1.84</td>
    <td>1.34</td>
    <td>3.38</td>
    <td>1.75</td>
    <td>2.81</td>
  </tr>
  <tr>
    <td>Medical</td>
    <td>0.76</td>
    <td>0.95</td>
    <td>2.35</td>
    <td>1.33</td>
    <td>2.56</td>
  </tr>
  <tr>
    <td>CHM6-Noisy</td>
    <td>2.32</td>
    <td>1.43</td>
    <td>3.29</td>
    <td>1.93</td>
    <td>2.74</td>
  </tr>
  <tr>
    <td>Average</td>
    <td>2.50</td>
    <td>1.54</td>
    <td>3.88</td>
    <td>2.05</td>
    <td>3.02</td>
  </tr>
</table>

Table 4: Mean absolute difference (↓) between the predicted and actual word error counts for our regression models and the baseline direct comparison with proxy reference (W PROXY).

Our results show that the regression model consistently reduces the absolute difference compared to direct evaluation with the proxy across seven of the eight datasets, with the only exception being MEDICALASR. We also find that the regression model trained using only similarity features remains competitive. These findings demonstrate that our approach generalizes well across domains and provides more reliable reference-free ASR quality estimates than simply computing the error against a proxy reference.

6 Conclusion

We present a framework for approximating ASR metrics, demonstrating its effectiveness in generalizing to unseen, in-the-wild, and challenging conditions. Our results show that the model performs well with absolute error counts, consistently outperforming strong baseline, with error rates remaining relatively low. We show that our proposed method achieves consistent performance across 40 ASR models and 14 evaluation setups, including both standard benchmarks and domain-specific conditions. The trained regression model can be efficiently used to approximate ASR metrics, particularly in data-constrained environments, such as critical domains with limited labeled data. In summary, our work bridges the gap between theoretical advancements and real-world applications, paving the way for more reliable and scalable ASR systems. While in this work, we evaluate monolingual and cross-lingual generalization, future work will focus on extending this framework to support a multilingual setting and exploring language-agnostic ASR metric approximation.
7 Limitations

In this work, we introduced a framework for approximating ASR metrics, evaluated across various ASR models and datasets. Despite the promising results, there are several limitations to consider.

Evaluation. While our evaluation setup is comprehensive, consisting of over 40 models and 14 datasets representing various acoustic and linguistic conditions such as natural noise, dialects, and accents—far surpassing previous works—we have not explored more nuanced conditions such as gender, non-native speech, and approximation across various age groups. Additionally, while the framework has shown strong performance in approximating ASR metrics across multiple datasets, its generalization to highly diverse or extreme real-world conditions might still require further investigation.

Language. Additionally, the current evaluation focuses solely on monolingual and bilingual settings. Extending this framework to include multiple languages and rigorously testing it across diverse linguistic contexts represents a critical direction for future research.

Compute. Unlike previous works, our final approximator is a simple regression model that does not require GPUs to run, we do utilize a single GPU for multimodal embedding extraction, which could be performed on any consumer-grade GPU.

8 Ethics Statement

Data Collection and Release. The datasets used in our experiments consist of publicly available ASR data from both benchmark and in-the-wild sources, as detailed in Section 4.1. We ensure that the use of these datasets aligns with the principles of fair use, specifically in a non-commercial academic context or as specified in their original license. All datasets are openly accessible, and no private or confidential data is included in this work to the best of our knowledge.

Intended Use. By enabling the approximation of ASR performance metrics with minimal data, our work has the potential to impact applications in domains with limited data availability, such as healthcare, emergency response, and low-resource language research. We believe our approach will foster further research in scalable, low-cost ASR systems with comprehensive evaluation, benefiting industries and research areas that serve underrepresented or resource-limited populations.

Potential Misuse and Bias. While our regression model has demonstrated effectiveness in approximating ASR metrics, it is important to consider potential misuse and bias. Given that our model is trained on diverse datasets, including those with various linguistic, acoustic, and demographic variations, there is a risk that the model may inherit biases present in the data, particularly with respect to accents, dialects, and socio-linguistic factors. Additionally, as our model approximates error rates, it could be misused in applications where the approximation may not be sufficient for real-world critical tasks. We recommend cautious deployment and further evaluation in sensitive applications, especially those where fairness and accuracy are critical.

References

Ahmed Ali and Steve Renals. 2018. Word error rate estimation for speech recognition: e-WER. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 20–24, Melbourne, Australia. Association for Computational Linguistics.

Ahmed M. Ali and Steve Renals. 2020. Word error rate estimation without asr output: e-wer2. In Interspeech.

Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, and Furu Wei. 2022. Speecht5: Unified-modal encoder-decoder pre-training for spoken language processing. Preprint, arXiv:2110.07205.

Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. 2020. Common voice: A massively-multilingual speech corpus. https://commonvoice.mozilla.org/en/datasets. Accessed: [Insert Date].

Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. 2022. data2vec: A general framework for self-supervised learning in speech, vision and language. Preprint, arXiv:2202.03555.

Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Preprint, arXiv:2006.11477.

Loïc Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenzek, Ethan
Ye, Bapi Akula, Peng-Jen Chen, Naji El Hachem, Brian Ellis, Gabriel Mejía Gonzalez, Justin Haaheim, Prangthip Hansanti, Russ Howes, Bernie Huang, Min-Jae Hwang, Hirofumi Inaguma, Somya Jain, Elahe Kalbassi, Amanda Kallet, Ilia Kulikov, Janice Lam, Daniel Li, Xutai Ma, Ruslan Mavlyutov, Benjamin Peloquin, Mohamed Ramadan, Abinash Ramakrishnan, Anna Sun, Kevin Tran, Tuan Tran, Igor Tufanov, Vish Vogeti, Carleigh Wood, Yilin Yang, Bokai Yu, Pierre Andrews, Can Balioglu, Marta R. Costa-jussà, Onur Çelebi, Maha Elbayad, Cynthia Gao, Francisco Guzmán, Justine Kao, Ann Lee, Alexandre Mourachko, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyah Saleem, Holger Schwenk, Paden Tomasello, Changhan Wang, Jeff Wang, Skyler Wang, and SEAMLESS Communication Team. 2025. Joint speech and text machine translation for up to 100 languages. Nature, 637(8046):587–593.

Jean Carletta, Simone Ashby, Sebastien Bourban, Mike Flynn, Mael Guillemot, Thomas Hain, Jaroslav Kadlec, Vasilis Karaiskos, Wessel Kraaij, Melissa Kronenthal, et al. 2005. The ami meeting corpus: A pre-announcement. https://groups.inf.ed.ac.uk/ami/corpus/. Accessed: [Insert Date].

Guogou Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, et al. 2021. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio. arXiv preprint arXiv:2106.06909.

Shamnur Absar Chowdhury and Ahmed Ali. 2023. Multilingual word error rate estimation: e-wer3. Preprint, arXiv:2304.00649.

Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klairer, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenzek, Ethan Ye, Bapi Akula, Peng-Jen Chen, Naji El Hachem, Brian Ellis, Gabriel Mejía Gonzalez, Justin Haaheim, Prangthip Hansanti, Russ Howes, Bernie Huang, Min-Jae Hwang, Hirofumi Inaguma, Somya Jain, Elahe Kalbassi, Amanda Kallet, Ilia Kulikov, Janice Lam, Daniel Li, Xutai Ma, Ruslan Mavlyutov, Benjamin Peloquin, Mohamed Ramadan, Abinash Ramakrishnan, Anna Sun, Kevin Tran, Tuan Tran, Igor Tufanov, Vish Vogeti, Carleigh Wood, Yilin Yang, Bokai Yu, Pierre Andrews, Can Balioglu, Marta R. Costa-jussà, Onur Çelebi, Maha Elbayad, Cynthia Gao, Francisco Guzmán, Justine Kao, Ann Lee, Alexandre Mourachko, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyah Saleem, Holger Schwenk, Paden Tomasello, Changhan Wang, Jeff Wang, and Skyler Wang. 2023. Seamless4t: Massively multilingual & multimodal machine translation. Preprint, arXiv:2308.11596.

Miguel Ángel Del-Agua, Adrià Giménez, Albert Sanchis, Jorge Civera, and Alfons Juan. 2018. Speaker-adapted confidence measures for asr using deep bidirectional recurrent neural networks. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(7):1198–1206.

Amandeep Singh Dhanjal and Williamjeet Singh. 2024. A comprehensive survey on automatic speech recognition using neural networks. Multimedia Tools and Applications, 83(8):23367–23412.

Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. 2023. Sonar: sentence-level multimodal and language-agnostic representations. arXiv e-prints, pages arXiv–2308.

Zied Elloumi, Laurent Besacier, Olivier Galibert, Juliette Kahn, and Benjamin Lecouteux. 2018. Asr performance prediction on unseen broadcast programs using convolutional neural networks. Preprint, arXiv:1804.08477.

Daniel Galvez, Greg Diamos, Juan Ciro, Juan Felipe Cerón, Keith Achorn, Anjali Gopi, David Kanter, Maximilian Lam, Mark Mazumder, and Vijay Janapa Reddi. 2021. The people’s speech: A large-scale diverse english speech recognition dataset for commercial usage. Preprint, arXiv:2111.09344.

Sanchit Gandhi, Patrick von Platen, and Alexander M. Rush. 2023. Distil-whisper: Robust knowledge distillation via large-scale pseudo labelling. Preprint, arXiv:2311.00430.

Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. 2020. Conformer: Convolution-augmented transformer for speech recognition. Preprint, arXiv:2005.08100.

Pengcheng Guo, Xuankai Chang, Shinji Watanabe, and Lei Xie. 2021. Multi-speaker asr combining non-autoregressive conformer ctc and conditional speaker chain. Preprint, arXiv:2106.08595.

Eric Harper, Somshubra Majumdar, Oleksii Kuchaiev, Li Jason, Yang Zhang, Evelina Bakhturina, Vahid Noroozi, Sandeep Subramanian, Koluguri Nithin, Huang Jocelyn, Fei Jia, Jagadeesh Balam, Xuesong Yang, Michal Livne, Yi Dong, Sean Naren, and Boris Ginsburg. 2024. NeMo: a toolkit for Conversational AI and Large Language Models.

Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. Preprint, arXiv:2106.07447.

Shahab Jalalvand, Matteo Negri, Marco Turchi, José G. C. de Souza, Daniele Falavigna, and Mohammed R. H. Qwaider. 2016. TranscRater: a tool for automatic speech recognition quality estimation. In Proceedings of ACL-2016 System Demonstrations, pages 43–48, Berlin, Germany. Association for Computational Linguistics.
Jan van Doorn. 2023. atcosim (revision b5839d9).

Nat Jeffries, Evan King, Manjunath Kudlur, Guy Nicholson, James Wang, and Pete Warden. 2024. Moonshine: Speech recognition for live transcription and voice commands. Preprint, arXiv:2410.15608.

Kaustubh Kalgaonkar, Chaojun Liu, Yifan Gong, and Kaisheng Yao. 2015. Estimating confidence scores on asr results using recurrent neural networks. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4999–5003.

Mahdie Karbasi and Dorothea Kolossa. 2022. Asr-based speech intelligibility prediction: A review. Hearing Research, 426:108606.

Hamza Kheddar, Mustapha Hemis, and Yassine Himeur. 2024. Automatic speech recognition using advanced deep learning approaches: A survey. Information Fusion, page 102422.

Korbinian Kuhn, Verena Kersken, Benedikt Reuter, Niklas Egger, and Gottfried Zimmermann. 2024. Measuring the accuracy of automatic speech recognition solutions. ACM Transactions on Accessible Computing, 16(4):1–23.

Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Paden Tomasello, Jacob Kahn, Gilad Avidov, Ronan Collobert, and Gabriel Synnaeve. 2020. Rethinking evaluation in asr: Are our models robust enough? arXiv preprint arXiv:2010.11745.

Yu-Yi Lin, Wei-Zhong Zheng, Wei Chung Chu, Ji-Yan Han, Ying-Hsiu Hung, Guan-Min Ho, Chia-Yuan Chang, and Ying-Hui Lai. 2021. A speech command control-based recognition system for dysarthric patients based on deep learning technology. Applied Sciences, 11(6):2477.

Diane J. Litman, Julia B. Hirschberg, and Marc Swerts. 2000. Predicting automatic speech recognition performance using prosodic cues. In 1st Meeting of the North American Chapter of the Association for Computational Linguistics.

Yinhai Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692.

Matteo Negri, Marco Turchi, José G. C. de Souza, and Daniele Falavigna. 2014. Quality estimation for automatic speech recognition. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 1813–1823, Dublin, Ireland. Dublin City University and Association for Computational Linguistics.

Vahid Noroozi, Somshubra Majumdar, Ankur Kumar, Jagadeesh Balam, and Boris Ginsburg. 2024. Stateful conformer with cache-based inference for streaming automatic speech recognition. Preprint, arXiv:2312.17279.

Azim Ospanov, Jingwei Zhang, Mohammad Jalali, Xue-nan Cao, Andrej Bogdanov, and Farzan Farnia. 2024. Towards a scalable reference-free evaluation of generative models. arXiv preprint arXiv:2407.02961.

Vassil Panayotov, Guogou Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An asr corpus based on public domain audio books. http://www.openslr.org/12/. Accessed: [Insert Date].

Alex Papadopoulos Korfiatis, Francesco Moramarco, Radmila Sarae, and Aleksandar Savkov. 2022. PriMock57: A dataset of primary care mock consultations. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 588–598, Dublin, Ireland. Association for Computational Linguistics.

Chanho Park, Hyunsik Kang, and Thomas Hain. 2024. Character error rate estimation for automatic speech recognition of short utterances. In 2024 32nd European Signal Processing Conference (EUSIPCO), pages 131–135. IEEE.

Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. 2023. Scaling speech technology to 1,000+ languages. Preprint, arXiv:2305.13516.

David Qiu, Qiuqia Li, Yanzhang He, Yu Zhang, Bo Li, Liangliang Cao, Rohit Prabhavalkar, Deepti Bhatia, Wei Li, Ke Hu, Tara N. Sainath, and Ian McGraw. 2021. Learning word-level confidence for subword end-to-end asr. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6393–6397.

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision. Preprint, arXiv:2212.04356.

Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR.

Bhiksha Raj, Rita Singh, and James Baker. 2011. A paired test for recognizer selection with untranscribed data. In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5676–5679.

Shangeth Rajaa and Abhinav Tushar. SpeechLLM: Multi-Modal LLM for Speech Understanding.

Dima Rekesh, Nithin Rao Koluguri, Samuel Kriman, Somshubra Majumdar, Vahid Noroozi, He Huang, Oleksii Hrinchuk, Krishna Puvvada, Ankur Kumar, Jagadeesh Balam, and Boris Ginsburg. 2023. Fast conformer with linearly scalable attention for efficient speech recognition. Preprint, arXiv:2305.05084.
MD Rio, Peter Ha, Quinten McNamara, Corey Miller, and Shipra Chandra. 2022. Earnings-22: A practical benchmark for accents in the wild.

Anthony Rousseau, Paul Deléglise, and Yann Es-tève. 2014. Ted-lium 3: Twice as much data and corpus repartition for experiments on speaker adaptation. https://lium.univ-lemans.fr/ted-lium3/. Accessed: [Insert Date].

Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019. wav2vec: Unsupervised pre-training for speech recognition. Preprint, arXiv:1904.05862.

Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haasheim, John Hoffman, Min-Jae Hwang, Hirofumi Inaguma, Christopher Klaiber, Ilia Kulikov, Pengwei Li, Daniel Licht, Jean Maillard, Ruslan Mavlyutov, Alice Rako-toarison, Kaushik Ram Sadagopan, Abinesh Ramakrishnan, Tuan Tran, Guillaume Wenzek, Yilin Yang, Ethan Ye, Ivan Evtimov, Pierre Fernandez, Cynthia Gao, Prangthip Hansanti, Elahe Kalbassi, Amanda Kallet, Artyom Kozhevnikov, Gabriel Mejia, Robin San Roman, Christophe Touret, Corinne Wong, Carleigh Wood, Bokai Yu, Pierre Andrews, Can Balioglu, Peng-Jen Chen, Marta R. Costa-jussà, Maha Elbayad, Hongyu Gong, Francisco Guzmán, Kevin Heffernan, Somya Jain, Justine Kao, Ann Lee, Xutai Ma, Alex Mourachko, Benjamin Pelouquin, Juan Pino, Sravya Popuri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Anna Sun, Paden Tomasello, Changhan Wang, Jeff Wang, Skyler Wang, and Mary Williamson. 2023. Seamless: Multilingual expressive and streaming speech translation.

Akshay Krishna Sheshadri, Anvesh Rao Vijjini, and Sukhdeep Kharbanda. 2021a. WER-BERT: Automatic WER estimation with BERT in a balanced ordinal classification paradigm. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3661–3672, Online. Association for Computational Linguistics.

Akshay Krishna Sheshadri, Anvesh Rao Vijjini, and Sukhdeep Kharbanda. 2021b. Wer-bert: Automatic wer estimation with bert in a balanced ordinal classification paradigm. Preprint, arXiv:2101.05478.

Suwon Shon, Ankita Pasad, Felix Wu, Pablo Brusco, Yoav Artzi, Karen Livescu, and Kyu J Han. 2022. Slue: New benchmark tasks for spoken language understanding evaluation on natural speech. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7927–7931. IEEE.

Prakhar Swarup, Roland Maas, Sri Garimella, Sri Harish Mallidi, and Björn Hoffmeister. 2019. Improving asr confidence scores for alexa using acoustic and hypothesis embeddings.

Yun Tang, Anna Y. Sun, Hirofumi Inaguma, Xinyue Chen, Ning Dong, Xutai Ma, Paden D. Tomasello, and Juan Pino. 2023. Hybrid transducer and attention based encoder-decoder modeling for speech-to-text tasks. Preprint, arXiv:2305.03101.

Kensho Technologies. 2021. Spgispeech: A large-scale, high-quality dataset for speech recognition in financial earnings calls. https://datasets.kensho.com/datasets/spgispeech. Accessed: [Insert Date].

Paige Tuttosí, Mantaj Dhillon, Luna Sang, Shane Eastwood, Poorvi Bhatia, Quang Minh Dinh, Avni Kapoor, Yewon Jin, and Angelica Lim. 2025. Bersting at the screams: A benchmark for distanced, emotional and shouted speech recognition. Preprint, arXiv:2505.00059.

Ehsan Variani, David Rybach, Cyril Allauzen, and Michael Riley. 2020. Hybrid autoregressive transducer (hat). Preprint, arXiv:2003.07705.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention is all you need. Preprint, arXiv:1706.03762.

Abdul Waheed, Karima Kadaoui, and Muhammad Abdul-Mageed. 2024. To distill or not to distill? on the robustness of robust knowledge distillation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12603–12621, Bangkok, Thailand. Association for Computational Linguistics.

Abdul Waheed, Karima Kadaoui, Bhiksha Raj, and Muhammad Abdul-Mageed. 2025. udistil-whisper: Label-free data filtering for knowledge distillation in low-data regimes. Preprint, arXiv:2407.01257.

Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021. VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 993–1003, Online. Association for Computational Linguistics.

Haolan Wang, Amin Edraki, Wai-Yip Chan, Iván López-Espejo, and Jesper Jensen. 2024. No-reference speech intelligibility prediction leveraging a noisy-speech asr pre-trained model. In Proc. Interspeech 2024, pages 3849–3853.

Shinji Watanabe, Michael Mandel, Jon Barker, Emmanuel Vincent, Ashish Arora, Xuankai Chang, Sanjeev Khudanpur, Vimal Manohar, Daniel Povey, Desh Raj, et al. 2020. Chime-6 challenge: Tackling multi-speaker speech recognition for unsegmented recordings. In CHiME 2020-6th International Workshop on Speech Processing in Everyday Environments.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierrette Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Huggingface’s transformers: State-of-the-art natural language processing. Preprint, arXiv:1910.03771.

Su-Youn Yoon, Lei Chen, and Klaus Zechner. 2010. Predicting word accuracy for the automatic speech recognition of non-native speech. In Interspeech 2010, pages 773–776.

Kamer Ali Yuksel, Thiago Ferreira, Ahmet Gunduz, Mohamed El-Badrashiny, and Golara Javadi. 2023a. A reference-less quality metric for automatic speech recognition via contrastive-learning of a multi-language model with self-supervision. Preprint, arXiv:2306.13114.

Kamer Ali Yuksel, Thiago Ferreira, Golara Javadi, Mohamed El-Badrashiny, and Ahmet Gunduz. 2023b. Norefer: a referenceless quality metric for automatic speech recognition via semi-supervised language model fine-tuning with contrastive learning. Preprint, arXiv:2306.12577.

Itamar Zimerman and Lior Wolf. 2023. On the long range abilities of transformers. arXiv preprint arXiv:2311.16620.
A Appendix

A.1 Methodology

\[
\text{pWER}(x_{\text{text}}, x_{\text{proxy}}) = \frac{\text{EditDistance}(x_{\text{text}}, x_{\text{proxy}})}{\text{WordCount}(x_{\text{proxy}})}
\]
(8)

\[
\text{pCER}(x_{\text{text}}, x_{\text{proxy}}) = \frac{\text{EditDistance}(x_{\text{text}}, x_{\text{proxy}})}{\text{CharCount}(x_{\text{proxy}})}
\]
(9)

A.2 Datasets

To evaluate the robustness and generalizability of our ASR metrics approximation tool, data were sourced from multiple repositories, which we divided into two distinct groups: Standard Benchmark and Wild Challenge dataset.

A.2.1 Standard Benchmark Datasets

There are six datasets in total that fall under the benchmark group. These datasets are categorized based on their frequent use in ASR model training and their representation of commonly encountered domains in real-world applications.

LibriSpeech (Panayotov et al., 2015). prioritized speaker and content balance over explicit consideration of speech characteristics. It comprises approximately 1000 hours of English read audiobooks, with subsets featuring both clean and noisy speech conditions to simulate different acoustic environments. While the dataset covers diverse subject matter, its focus on formal, clear speech from public domain books means it lacks the natural variability of spontaneous speech, limiting its representation of conversational or informal dialogue.

TED-LIUM (Rousseau et al., 2014). contains TED Talks totaling 452 hours of English speech data from approximately 2,000 speakers, recorded in close-talk microphone conditions. The corpus features narrated speaking styles, capturing clear and articulate speech. While it provides non-orthographic transcriptions, lacking formatting such as capitalization and punctuation, it remains a valuable resource for training and benchmarking automatic speech recognition (ASR) models.

GigaSpeech (Chen et al., 2021). is a multi-domain, multi-style speech recognition corpus incorporating diverse acoustic and linguistic conditions. It sources audio from three primary domains: audiobooks, podcasts, and YouTube, covering a wide range of speaking styles, including both read and spontaneous speech. The dataset covers a broad spectrum of topics, such as arts, science, sports, and more, making it highly versatile for training robust speech recognition models.

SPGISpeech (Technologies, 2021). contains 5,000 hours of professionally transcribed audio from corporate earnings calls, featuring both spontaneous and narrated speaking styles. It emphasizes orthographic accuracy, providing fully formatted text with capitalization, punctuation, and denormalization of non-standard words.

Common Voice (Ardila et al., 2020). (a multilingual corpus of narrated prompts built through crowdsourcing. Recorded in teleconference conditions, the corpus features narrated speaking styles and emphasizes inclusivity by covering a wide range of accents and languages, including low-resource ones.

Earnings22 (Rio et al., 2022), is a 119-hour corpus of English-language earnings calls from global companies, designed to address the lack of real-world, accented speech data in ASR benchmarking

AMI (IHM) (Carletta et al., 2005). The AMI Meeting Corpus is a 100-hour dataset of English meeting recordings, featuring multimodal data synchronized across close-talking and far-field microphones, room-view and individual cameras, slide projectors, and whiteboards. It includes mostly non-native speakers recorded in three rooms with varying acoustics. Digital pens capture unsynchronized handwritten notes, supporting research in speech recognition, diarization, and multimodal interaction. Available under edinburghstr/ami, it is widely used for meeting analysis and speech processing studies.

People’s Speech (Galvez et al., 2021). Thousands of hours of labeled speech data collected from diverse speakers, covering a wide range of topics, accents, and speaking styles. The dataset emphasizes inclusivity and linguistic diversity, making it suitable for developing robust and generalized speech models. It is widely used in academic and industrial research to advance the state-of-the-art in automatic speech recognition (ASR) and other speech-related applications.

SLUE - VolxCeleb (Shon et al., 2022). consists of single-sided conversational voice snippets ex-
tracted from YouTube videos, originally designed for speaker recognition. The dataset represents natural, unscripted speech in diverse real-world settings, capturing a wide range of speaking styles, emotions, and acoustic conditions. Utterances containing slurs were excluded, and partial words were trimmed using a forced aligner to ensure clean, usable segments.

A.2.2 Wild Challenge Set

Primock57 (Papadopoulos Korfiatis et al., 2022). contains mock consultations conducted by seven clinicians and 57 actors posing as patients, representing a diverse range of ethnicities, accents, and ages. Each actor was provided with a detailed case card outlining a primary care scenario, such as urinary tract infections, cardiovascular issues, or mental health concerns, ensuring the conversations were realistic and clinically relevant. The consultations were recorded using telemedicine software, capturing separate audio channels for clinicians and patients, and transcribed by experienced professionals to ensure accuracy.

VoxPopuli Accented (Wang et al., 2021). is a comprehensive multilingual speech corpus derived from European Parliament event recordings. It includes audio, transcripts, and timestamps sourced directly from the official Parliament website. Due to its origin, the dataset features a rich collection of named entities, making it particularly suitable for tasks like Named Entity Recognition (NER).

ATCOsim (Jan van Doorn, 2023).is a specialized database containing ten hours of English speech from ten non-native speakers, recorded during real-time ATC simulations using close-talk headset microphones. It features orthographic transcriptions, speaker metadata, and session details. With a 32 kHz sampling frequency and 10,078 clean, utterance-level recordings.

A.3 Models

Whisper Models (Radford et al., 2023). is a transformer-based model that processes 80-dimensional log-mel filter bank features from 16 kHz audio, utilizing a 2D CNN stack followed by a transformer encoder-decoder architecture. Trained on a vast multilingual dataset of 680,000 hours, it incorporates timestamp tokens into its vocabulary and operates on 30-second audio windows during inference, auto-regressively generating text sequences while leveraging encoder outputs as context. Variants of Whisper, such as Distilled, Large, Base, and Medium, offer different trade-offs in model size and performance, catering to diverse computational and accuracy requirements.

Seamless Models (Communication et al., 2023; Seamless Communication et al., 2023; Barrault et al., 2025). is a cutting-edge multilingual and multitask model for speech and text translation. Built on the UnitY architecture, it uses w2v-BERT 2.0 for speech encoding and NLLB for text encoding, supporting nearly 100 languages. A text decoder handles ASR and translation, while a text-to-unit (T2U) model and multilingual HiFi-GAN vocoder generate speech. Leveraging SONAR embeddings and SeamlessAlign (443,000 hours of aligned speech/text data), it achieves SOTA results in ASR, speech-to-text, speech-to-speech, and text-to-text translation, excelling in low-resource languages. It introduces BLASER 2.0 for robust evaluation and outperforms competitors in noisy environments.

Nemo-ASR-Models (Gulati et al., 2020; Variani et al., 2020; Rekesh et al., 2023; Noroozi et al., 2024; Tang et al., 2023; Harper et al., 2024) We included several NVIDIA’s NeMo advanced automatic speech recognition (ASR) models, including Canary, Parakeet (110M, 0.6B, and 1.1b), Conformer-CTC, and Fast-Conformer, as each is designed for specific use cases and optimized for performance. Canary-1B is a state-of-the-art multilingual, multitask model featuring a FastConformer encoder and Transformer decoder. The Parakeet family includes models with a FastConformer encoder paired with different decoders: CTC, RNN-T, or TDT. Conformer-CTC is a non-autoregressive model based on the Conformer architecture, combining self-attention and convolution for global and local feature extraction. It uses CTC loss and a linear decoder, supporting both sub-word (BPE) and character-level encodings. While Fast-Conformer is an optimized version of the Conformer architecture, offering significant speed improvements (2.4x faster) with minimal quality degradation. It uses 8x depthwise convolutional subsampling and reduced kernel sizes for efficiency.

Wav2Vec2 Models (Schneider et al., 2019; Baevski et al., 2020). is a self-supervised pretrained model designed to process raw audio inputs and generate speech representations. The model ar-
chitecture consists of three key components: a convolutional feature encoder, a context network, and a quantization module. The convolutional feature encoder converts raw waveforms into latent representations, which are then processed by the context network a transformer based stack with 24 blocks, a hidden size of 1024, 16 attention heads, and a feed-forward dimension of 4096 to capture contextual information. The quantization module maps these latent representations to quantized forms.

HuBERT Models (Hsu et al., 2021), is a self-supervised learning framework designed for speech representation learning where CNN-encoded audio features are randomly masked. During training, the model predicts cluster assignments for masked regions of the input speech, forcing it to learn both acoustic and language models from continuous inputs.

Audio/Speech Language Models 1.5B and 2B (Rajaa and Tushar) is a multi-modal Language Model designed to analyze and predict metadata from a speaker’s turn in a conversation. It integrates a speech encoder to convert speech signals into meaningful embeddings, which are then processed alongside text instructions by TinyLlama-1.1B-Chat-v1.0 to generate predictions. The model accepts 16 KHz audio inputs and predicts metadata such as SpeechActivity, Transcript, Gender, Age, Accent, and Emotion.

SpeechT5 (Ao et al., 2022), unified modal framework capable of handling a wide range of tasks, including automatic speech recognition (ASR), speech synthesis, speech translation, voice conversion, speech enhancement, and speaker identification. Its audio post-net, which can incorporate speaker embeddings to enable prosody transfer, making it effective for tasks like voice conversion and speech synthesis. By leveraging its encoder-decoder architecture, SpeechT5 can generate high-quality mel-spectrograms from text input while preserving speaker-specific characteristics like emotion and gender.

A.4 Experiments

A.4.1 Regression Pipeline.

The regression framework is a stacking ensemble comprising multiple base regressors and a final estimator. We perform basic hyperparameter tuning using RandomizedSearchCV with 5-fold cross-validation, with the objective to minimize mean absolute error (MAE). The search explores key hyperparameters such as n_estimators, max_depth, learning_rate, and min_samples_split, balancing model complexity and generalization. We provide hyperparameter and other details in 7. The model is trained on 14 datasets divided into two groups: bench (10 standard benchmark datasets) and in-the-wild (4 diverse, real-world datasets). A leave-one-out strategy is applied to the bench set, where the model is trained on 9 datasets and evaluated on the remaining one. All trained models are also evaluated on the in-the-wild set, which remains isolated during training to assess out-of-domain generalization.

A.5 Domain Divergence and Phonetic Diversity Analysis

To quantify how our in-the-wild evaluation sets differ from the LibriSpeech-clean corpus that was used to fine-tune both data2vec-audio-base-960h (source) and wav2vec2-base-960h (target), we measure acoustic domain divergence with Central Moment Discrepancy (CMD) computed over SONAR speech embeddings and phonetic diversity with Total Vocabulary Overlap (TVO) calculated on the corresponding transcripts. We report these numbers in Tables 8 and 9.

CMD values confirm that LibriSpeech (Noise) remains acoustically close to the source domain because only artificial background sounds are added, whereas Primock57, VoxPopuli (Accented), and ATCOSIM exhibit substantial acoustic shifts. TVO scores tell a complementary story in the lexical space: LibriSpeech (Noise) preserves more than one-third of the vocabulary, while Primock57, VoxPopuli (Accented), and especially ATCOSIM share far less with LibriSpeech-clean. Together, these metrics justify the in-the-wild designation of our evaluation corpora and highlight the importance of robust models that generalize beyond clean, studio-quality speech.

A.6 Results

Fine-grained ASR Evaluation Metrics To assess whether our framework can also approximate fine-grained ASR evaluation metrics, we extend our experiments to cover insertions, deletions, and substitutions. This addition aims to evaluate the generalizability of our approach beyond overall word error rates and provide a more detailed analysis of ASR performance. We train the regression model under various configurations and examine its performance across both in-distribution (IID) and
out-of-distribution (OOD) conditions. We report the results in Table 12.

The results demonstrate that our regression model reliably approximates these fine-grained error counts. Specifically, the absolute differences remain low across substitutions, insertions, and deletions, even when evaluated on challenging out-of-distribution data or using transcriptions from models not seen during training. Furthermore, the similarity-only variant, which does not rely on proxy information during testing, remains competitive, highlighting the robustness and generalizability of the learned acoustic and semantic representations.
Figure 4: Actual and approximated word error rate across different models evaluated on four in-the-wild datasets.
Figure 5: Actual and approximated character error rate across different models evaluated on four in-the-wild datasets.
<table>
  <tr>
    <th>Name</th>
    <th>Type</th>
    <th>Description</th>
  </tr>
  <tr>
    <td>LibriSpeech</td>
    <td>Bench</td>
    <td>A corpus of approximately 1,000 hours of 16kHz read English speech, derived from LibriVox audiobooks, segmented and aligned for ASR tasks.</td>
  </tr>
  <tr>
    <td>TED-LIUM</td>
    <td>Bench</td>
    <td>Contains TED Talks totaling 452 hours of English speech data from approximately 2,000 speakers, recorded in close-talk microphone conditions.</td>
  </tr>
  <tr>
    <td>GigaSpeech</td>
    <td>Bench</td>
    <td>A multi-domain, multi-style speech recognition corpus incorporating diverse acoustic and linguistic conditions, sourced from audiobooks, podcasts, and YouTube.</td>
  </tr>
  <tr>
    <td>SPGISpeech</td>
    <td>Bench</td>
    <td>Contains 5,000 hours of professionally transcribed audio from corporate earnings calls, featuring both spontaneous and narrated speaking styles.</td>
  </tr>
  <tr>
    <td>Common Voice</td>
    <td>Bench</td>
    <td>A multilingual corpus of narrated prompts built through crowdsourcing, recorded in teleconference conditions, covering a wide range of accents and languages.</td>
  </tr>
  <tr>
    <td>Earnings22</td>
    <td>Bench</td>
    <td>A 119-hour corpus of English-language earnings calls from global companies, designed to address the lack of real-world, accented speech data in ASR benchmarking.</td>
  </tr>
  <tr>
    <td>AMI (IHM)</td>
    <td>Bench</td>
    <td>The AMI Meeting Corpus is a 100-hour dataset of English meeting recordings, featuring multimodal data synchronized across various devices.</td>
  </tr>
  <tr>
    <td>People’s Speech</td>
    <td>Bench</td>
    <td>Contains thousands of hours of labeled speech data collected from diverse speakers, covering a wide range of topics, accents, and speaking styles.</td>
  </tr>
  <tr>
    <td>SLUE - VoxCeleb</td>
    <td>Wild</td>
    <td>Consists of single-sided conversational voice snippets extracted from YouTube videos, originally designed for speaker recognition.</td>
  </tr>
  <tr>
    <td>Primock57</td>
    <td>Wild</td>
    <td>Contains mock consultations conducted by seven clinicians and 57 actors posing as patients, representing a diverse range of ethnicities, accents, and ages.</td>
  </tr>
  <tr>
    <td>VoxPopuli Accented</td>
    <td>Wild</td>
    <td>A comprehensive multilingual speech corpus derived from European Parliament event recordings, featuring a rich collection of named entities.</td>
  </tr>
  <tr>
    <td>ATCOsim</td>
    <td>Wild</td>
    <td>A specialized database containing ten hours of English speech from ten non-native speakers, recorded during real-time air traffic control simulations.</td>
  </tr>
</table>

Table 5: Overview of various ASR along with brief description.
<table>
  <tr>
    <th>Model Type and Models</th>
    <th>Description</th>
  </tr>
  <tr>
    <td><b>nemo_asr</b><br>
      – parakeet-ctc-1.1b<br>
      – parakeet-ctc-0.6b<br>
      – stt_en_conformer_ctc_large<br>
      – stt_en_fastconformer_ctc_large<br>
      – stt_en_conformer_ctc_small<br>
      – parakeet-tdt-1.1b<br>
      – parakeet-rmt-1.1b<br>
      – parakeet-rmt-0.6b<br>
      – stt_en_fastconformer_transducer_large<br>
      – parakeet-tdt_ctc-110m<br>
      – canary-1b<br>
      – speechbrain<br>
      – asr-wav2vec2-librispeech
    </td>
    <td>NVIDIA’s NeMo ASR models offer diverse architectures for speech-to-text applications. The Conformer-CTC model combines self-attention and convolutional operations, using Connectionist Temporal Classification (CTC) loss for efficient transcription. The Conformer-Transducer extends this by incorporating a Recurrent Neural Network Transducer (RNNT) decoder for autoregressive modeling. The Conformer-HAT variant separates label and blank score predictions, enhancing integration with external language models. For improved performance, the Fast-Conformer introduces depthwise convolutional subsampling, achieving approximately 2.4x faster encoding with minimal accuracy loss.<br><br>SpeechBrain provides robust models for ASR and speaker recognition.
    </td>
  </tr>
  <tr>
    <td><b>data2vec</b><br>
      – data2vec-audio-large-960h<br>
      – data2vec-audio-base-960h
    </td>
    <td>Data2Vec models by Facebook are designed for speech representation learning and ASR. These models use a unified learning framework for multiple modalities.
    </td>
  </tr>
  <tr>
    <td><b>wav2vec2</b><br>
      – wav2vec2-large-960h-lv60-self<br>
      – wav2vec2-large-robust-ft-libri-960h<br>
      – wav2vec2-large-960h<br>
      – wav2vec2-base-960h<br>
      – wav2vec2-conformer-rope-large-960h-ft<br>
      – wav2vec2-conformer-rel-pos-large-960h-ft
    </td>
    <td>Wav2Vec2 models leverage self-supervised learning on raw audio for ASR. With advanced configurations, these models provide high accuracy for diverse speech-to-text tasks.
    </td>
  </tr>
  <tr>
    <td><b>mms</b><br>
      – mms-1b-all<br>
      – mms-1b-fl102
    </td>
    <td>The Multilingual Speech (MMS) models by Facebook excel at speech recognition for multiple languages and accents.
    </td>
  </tr>
  <tr>
    <td><b>hubert</b><br>
      – hubert-xlarge-ls960-ft<br>
      – hubert-large-ls960-ft
    </td>
    <td>HuBERT models provide high-quality speech representations for ASR and other downstream speech tasks.
    </td>
  </tr>
  <tr>
    <td><b>seamless</b><br>
      – hf-seamless-m4t-large<br>
      – hf-seamless-m4t-medium<br>
      – seamless-m4t-v2-large
    </td>
    <td>Seamless models focus on multilingual transcription and translation, offering robust real-time speech processing solutions.
    </td>
  </tr>
  <tr>
    <td><b>speechllm</b><br>
      – speechllm-1.5B<br>
      – speechllm-2B
    </td>
    <td>SpeechLLM models are fine-tuned for ASR and text generation, leveraging billions of parameters for high performance.
    </td>
  </tr>
  <tr>
    <td><b>whisper</b><br>
      – whisper-large-v3<br>
      – distil-large-v3<br>
      – whisper-large-v2<br>
      – whisper-large-v3-turbo<br>
      – distil-large-v2<br>
      – whisper-large<br>
      – whisper-tiny<br>
      – whisper-medium.en<br>
      – distil-small.en<br>
      – whisper-small.en
    </td>
    <td>Whisper models by OpenAI provide state-of-the-art transcription and translation capabilities for multilingual ASR. These models range from tiny to large configurations.
    </td>
  </tr>
  <tr>
    <td><b>moonshine</b><br>
      – moonshine-base<br>
      – moonshine-tiny
    </td>
    <td>Moonshine models are lightweight and optimized for efficient ASR on edge devices with minimal computational resources.
    </td>
  </tr>
</table>

Table 6: Overview of various ASR along with brief description.
<table>
  <tr>
    <th>Model</th>
    <th>Hyperparameter</th>
    <th>Values</th>
  </tr>
  <tr>
    <td rowspan="4">Random Forest (RF)</td>
    <td>n_estimators</td>
    <td>{100, 200, 300, 500, 700, 1000}</td>
  </tr>
  <tr>
    <td>max_depth</td>
    <td>{5, 10, 15, 20, 25, 30}</td>
  </tr>
  <tr>
    <td>min_samples_split</td>
    <td>{2, 5, 10, 15, 20}</td>
  </tr>
  <tr>
    <td>min_samples_leaf</td>
    <td>{1, 2, 4, 8}</td>
  </tr>
  <tr>
    <td rowspan="4">Gradient Boosting (GBR)</td>
    <td>n_estimators</td>
    <td>{100, 200, 400, 600, 800}</td>
  </tr>
  <tr>
    <td>learning_rate</td>
    <td>{0.001, 0.01, 0.05, 0.1, 0.2}</td>
  </tr>
  <tr>
    <td>max_depth</td>
    <td>{3, 5, 7, 10}</td>
  </tr>
  <tr>
    <td>min_impurity_decrease</td>
    <td>{0.0, 0.001, 0.01, 0.1, 0.2}</td>
  </tr>
  <tr>
    <td rowspan="3">HistGradientBoosting (HGB)</td>
    <td>max_iter</td>
    <td>{100, 200, 300, 400, 500}</td>
  </tr>
  <tr>
    <td>learning_rate</td>
    <td>{0.001, 0.01, 0.05, 0.1, 0.2}</td>
  </tr>
  <tr>
    <td>max_depth</td>
    <td>{3, 5, 7, 10, 15}</td>
  </tr>
  <tr>
    <td>loss</td>
    <td>{Poisson}</td>
  </tr>
  <tr>
    <td>Ridge Regression (Final Estimator)</td>
    <td>alpha</td>
    <td>[1e-3, 1e-2, 0.1, 1, 10, 100, 1000]</td>
  </tr>
  <tr>
    <td></td>
    <td>positive</td>
    <td>{True}</td>
  </tr>
  <tr>
    <td>Pipeline</td>
    <td>passthrough</td>
    <td>{True}</td>
  </tr>
</table>

Table 7: Hyperparameter details for regression model.
<table>
  <tr>
    <th>Dataset</th>
    <th>CMD</th>
    <th>Interpretation</th>
  </tr>
  <tr>
    <td>LibriSpeech (Noise)</td>
    <td>0.062</td>
    <td>Low divergence</td>
  </tr>
  <tr>
    <td>Primock57</td>
    <td>0.329</td>
    <td>Significant divergence</td>
  </tr>
  <tr>
    <td>VoxPopuli (Accented)</td>
    <td>0.504</td>
    <td>High divergence</td>
  </tr>
  <tr>
    <td>ATCOSIM</td>
    <td>0.538</td>
    <td>High divergence</td>
  </tr>
</table>

Table 8: Acoustic domain divergence between LibriSpeech-clean and each evaluation set, measured with Central Moment Discrepancy (CMD).

<table>
  <tr>
    <th>Dataset</th>
    <th>TVO (%)</th>
    <th>Interpretation</th>
  </tr>
  <tr>
    <td>LibriSpeech (Noise)</td>
    <td>36.9</td>
    <td>High overlap</td>
  </tr>
  <tr>
    <td>VoxPopuli (Accented)</td>
    <td>20.3</td>
    <td>Low overlap</td>
  </tr>
  <tr>
    <td>Primock57</td>
    <td>14.3</td>
    <td>Low overlap</td>
  </tr>
  <tr>
    <td>ATCOSIM</td>
    <td>3.8</td>
    <td>Very low overlap</td>
  </tr>
</table>

Table 9: Total Vocabulary Overlap (TVO) between LibriSpeech-clean and each evaluation set.

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">IID</th>
    <th colspan="3">OOD</th>
  </tr>
  <tr>
    <th>D</th>
    <th>M</th>
    <th>D + M</th>
  </tr>
  <tr>
    <td>Base</td>
    <td></td>
    <td>3.79<sup>0.16</sup></td>
    <td>3.56<sup>0.06</sup></td>
    <td>3.76<sup>0.18</sup></td>
    <td>3.69<sup>0.06</sup></td>
  </tr>
  <tr>
    <td>w/o S</td>
    <td></td>
    <td>3.83<sup>0.14</sup></td>
    <td>3.65<sup>0.06</sup></td>
    <td>3.82<sup>0.16</sup></td>
    <td>3.73<sup>0.07</sup></td>
  </tr>
  <tr>
    <td>w/o PR</td>
    <td></td>
    <td>8.43<sup>0.28</sup></td>
    <td>8.36<sup>0.08</sup></td>
    <td>8.67<sup>0.24</sup></td>
    <td>8.66<sup>0.08</sup></td>
  </tr>
  <tr>
    <td>w/ MPR (n=2)</td>
    <td></td>
    <td>3.69<sup>0.14</sup></td>
    <td>3.57<sup>0.06</sup></td>
    <td>3.66<sup>0.17</sup></td>
    <td>3.69<sup>0.06</sup></td>
  </tr>
  <tr>
    <td>w/ MPR (n=3)</td>
    <td></td>
    <td>3.62<sup>0.13</sup></td>
    <td>3.44<sup>0.07</sup></td>
    <td>3.58<sup>0.15</sup></td>
    <td>3.56<sup>0.07</sup></td>
  </tr>
  <tr>
    <td>w/ MPR (n=4)</td>
    <td></td>
    <td>3.57<sup>0.13</sup></td>
    <td>3.40<sup>0.06</sup></td>
    <td>3.53<sup>0.13</sup></td>
    <td>3.52<sup>0.06</sup></td>
  </tr>
  <tr>
    <td>w/ MPR (n=5)</td>
    <td></td>
    <td>3.49<sup>0.13</sup></td>
    <td>3.37<sup>0.06</sup></td>
    <td>3.47<sup>0.12</sup></td>
    <td>3.49<sup>0.07</sup></td>
  </tr>
  <tr>
    <td>w/ mMMPR (n=3)</td>
    <td></td>
    <td>3.61<sup>0.15</sup></td>
    <td>3.40<sup>0.09</sup></td>
    <td>3.57<sup>0.13</sup></td>
    <td>3.51<sup>0.09</sup></td>
  </tr>
  <tr>
    <td>w/ mMMPR (n=5)</td>
    <td></td>
    <td>3.80<sup>0.15</sup></td>
    <td>3.47<sup>0.03</sup></td>
    <td>3.77<sup>0.13</sup></td>
    <td>3.56<sup>0.04</sup></td>
  </tr>
  <tr>
    <td>Base (r=11.9)</td>
    <td></td>
    <td>4.68<sup>0.17</sup></td>
    <td>5.16<sup>0.06</sup></td>
    <td>4.64<sup>0.16</sup></td>
    <td>5.06<sup>0.05</sup></td>
  </tr>
  <tr>
    <td>Base (r=14.0)</td>
    <td></td>
    <td>4.84<sup>0.18</sup></td>
    <td>4.88<sup>0.07</sup></td>
    <td>4.75<sup>0.17</sup></td>
    <td>4.77<sup>0.07</sup></td>
  </tr>
  <tr>
    <td>Base (r=20.2)</td>
    <td></td>
    <td>5.13<sup>0.12</sup></td>
    <td>5.38<sup>0.07</sup></td>
    <td>5.12<sup>0.10</sup></td>
    <td>5.30<sup>0.07</sup></td>
  </tr>
  <tr>
    <td>Base (r=23.5)</td>
    <td></td>
    <td>5.60<sup>0.13</sup></td>
    <td>6.12<sup>0.07</sup></td>
    <td>5.69<sup>0.21</sup></td>
    <td>6.03<sup>0.05</sup></td>
  </tr>
  <tr>
    <td>w/o S (r=11.9)</td>
    <td></td>
    <td>5.50<sup>0.21</sup></td>
    <td>5.84<sup>0.06</sup></td>
    <td>5.55<sup>0.21</sup></td>
    <td>5.65<sup>0.05</sup></td>
  </tr>
  <tr>
    <td>w/o S (r=14.0)</td>
    <td></td>
    <td>5.73<sup>0.12</sup></td>
    <td>5.50<sup>0.05</sup></td>
    <td>5.71<sup>0.13</sup></td>
    <td>5.37<sup>0.06</sup></td>
  </tr>
  <tr>
    <td>w/o S (r=20.2)</td>
    <td></td>
    <td>6.16<sup>0.18</sup></td>
    <td>6.24<sup>0.08</sup></td>
    <td>6.13<sup>0.10</sup></td>
    <td>5.97<sup>0.09</sup></td>
  </tr>
  <tr>
    <td>w/o S (r=23.5)</td>
    <td></td>
    <td>6.38<sup>0.09</sup></td>
    <td>6.77<sup>0.08</sup></td>
    <td>6.43<sup>0.16</sup></td>
    <td>6.58<sup>0.08</sup></td>
  </tr>
</table>

Table 10: Mean absolute error between predicted character error count and actual character error count (in absolute terms) across different configurations. R - Regression, C - Classification, PR - Proxy Reference, S - Silarity, MPR - Multiple PR. The OOD results are averaged across five wild datasets. Superscript represents the standard deviation across five runs.
<table>
  <tr>
    <th>Model</th>
    <th>LS_Noise</th>
    <th>Primock57</th>
    <th>Atcosim</th>
    <th>VP_accented</th>
  </tr>
  <tr><td>asr-wav2vec2-librispeech</td><td>4.2/5.8</td><td>17.2/20.8</td><td>18.8/21.9</td><td>9.8/14.0</td></tr>
  <tr><td>canary-1b</td><td>1.5/3.8</td><td>10.1/9.7</td><td>16.4/19.4</td><td>15.6/9.0</td></tr>
  <tr><td>data2vec-audio-base-960h</td><td>7.0/8.1</td><td>20.5/23.7</td><td>29.5/32.0</td><td>13.3/17.8</td></tr>
  <tr><td>data2vec-audio-large-960h</td><td>3.1/4.2</td><td>14.1/17.4</td><td>20.0/23.8</td><td>10.6/14.4</td></tr>
  <tr><td>distil-large-v2</td><td>3.5/5.2</td><td>11.5/9.2</td><td>49.5/41.8</td><td>10.2/9.4</td></tr>
  <tr><td>distil-large-v3</td><td>2.7/4.6</td><td>11.9/9.1</td><td>49.4/40.5</td><td>10.1/9.0</td></tr>
  <tr><td>distil-small.en</td><td>4.2/5.8</td><td>12.2/10.4</td><td>50.7/41.8</td><td>9.7/9.2</td></tr>
  <tr><td>hf-seamless-m4t-large</td><td>6.5/7.5</td><td>32.1/30.6</td><td>54.7/57.2</td><td>21.8/15.8</td></tr>
  <tr><td>hf-seamless-m4t-medium</td><td>9.4/10.1</td><td>34.4/32.7</td><td>35.5/37.9</td><td>23.1/17.9</td></tr>
  <tr><td>hubert-large-ls960-ft</td><td>3.0/4.2</td><td>14.4/17.4</td><td>21.3/25.0</td><td>10.0/14.3</td></tr>
  <tr><td>hubert-xlarge-ls960-ft</td><td>2.7/4.1</td><td>15.3/18.1</td><td>20.1/23.8</td><td>10.2/14.5</td></tr>
  <tr><td>mms-1b-all</td><td>3.6/4.8</td><td>19.5/19.1</td><td>27.2/31.8</td><td>17.0/12.6</td></tr>
  <tr><td>mms-1b-fl102</td><td>9.0/10.0</td><td>35.0/33.2</td><td>55.4/57.3</td><td>18.2/17.6</td></tr>
  <tr><td>moonshine-base</td><td>5.7/6.8</td><td>12.4/12.1</td><td>42.6/39.5</td><td>10.9/12.6</td></tr>
  <tr><td>moonshine-tiny</td><td>8.5/9.9</td><td>17.9/19.0</td><td>38.2/38.4</td><td>13.2/15.1</td></tr>
  <tr><td>parakeet-ctc-0.6b</td><td>1.7/3.7</td><td>10.1/9.9</td><td>16.2/22.7</td><td>9.7/9.0</td></tr>
  <tr><td>parakeet-ctc-1.1b</td><td>1.7/3.6</td><td>10.0/10.1</td><td>14.8/21.4</td><td>10.0/8.0</td></tr>
  <tr><td>parakeet-rmt-0.6b</td><td>1.3/3.4</td><td>10.1/9.4</td><td>16.9/24.1</td><td>10.9/8.8</td></tr>
  <tr><td>parakeet-rmt-1.1b</td><td>1.3/3.3</td><td>9.1/9.7</td><td>14.5/21.3</td><td>11.2/7.2</td></tr>
  <tr><td>parakeet-tdt-1.1b</td><td>1.1/3.1</td><td>8.2/9.4</td><td>14.0/20.0</td><td>10.9/6.8</td></tr>
  <tr><td>parakeet-tdf_ctc-110m</td><td>2.5/4.7</td><td>10.3/9.2</td><td>22.3/24.2</td><td>12.4/8.6</td></tr>
  <tr><td>seamless-m4t-v2-large</td><td>3.5/4.6</td><td>24.6/23.7</td><td>31.6/35.8</td><td>25.2/19.8</td></tr>
  <tr><td>speechlm-1.5B</td><td>9.9/11.2</td><td>30.1/31.4</td><td>85.4/88.7</td><td>47.3/49.0</td></tr>
  <tr><td>speechlm-2B</td><td>8.4/9.3</td><td>25.3/27.7</td><td>33.5/36.1</td><td>24.0/28.3</td></tr>
  <tr><td>stt_en_conformer_ctc_large</td><td>2.1/3.4</td><td>8.8/11.2</td><td>17.1/18.2</td><td>11.1/7.5</td></tr>
  <tr><td>stt_en_conformer_ctc_small</td><td>4.3/5.7</td><td>12.7/15.6</td><td>21.6/23.6</td><td>9.5/9.7</td></tr>
  <tr><td>stt_en_fastconformer_ctc_large</td><td>3.0/5.6</td><td>10.1/16.3</td><td>17.3/25.1</td><td>11.5/9.2</td></tr>
  <tr><td>stt_en_fastconformer_transducer_large</td><td>2.8/5.0</td><td>10.6/14.3</td><td>18.7/25.3</td><td>14.2/11.9</td></tr>
  <tr><td>wav2vec2-base-960h</td><td>7.9/9.1</td><td>23.3/26.7</td><td>30.3/33.2</td><td>13.7/18.9</td></tr>
  <tr><td>wav2vec2-conformer-rel-pos-large-960h-ft</td><td>2.6/3.8</td><td>14.7/17.4</td><td>21.0/24.5</td><td>11.2/14.7</td></tr>
  <tr><td>wav2vec2-conformer-rope-large-960h-ft</td><td>2.9/4.0</td><td>16.1/18.7</td><td>22.2/25.9</td><td>11.0/14.2</td></tr>
  <tr><td>wav2vec2-large-960h</td><td>5.1/6.3</td><td>19.1/22.4</td><td>28.8/31.8</td><td>12.2/17.4</td></tr>
  <tr><td>wav2vec2-large-960h-lv60-self</td><td>3.5/5.0</td><td>17.6/21.0</td><td>18.6/23.0</td><td>9.3/13.6</td></tr>
  <tr><td>wav2vec2-large-robust-ft-libri-960h</td><td>4.5/5.8</td><td>15.7/19.0</td><td>20.7/24.2</td><td>10.0/14.7</td></tr>
  <tr><td>whisper-large</td><td>2.9/4.2</td><td>13.7/10.6</td><td>49.3/47.5</td><td>13.7/11.9</td></tr>
  <tr><td>whisper-large-v2</td><td>2.6/3.8</td><td>15.3/12.5</td><td>48.6/51.5</td><td>15.3/14.2</td></tr>
  <tr><td>whisper-large-v3</td><td>2.0/3.3</td><td>12.3/8.7</td><td>48.9/48.3</td><td>14.3/13.6</td></tr>
  <tr><td>whisper-large-v3-turbo</td><td>2.0/3.2</td><td>12.4/8.8</td><td>48.3/49.9</td><td>19.7/19.0</td></tr>
  <tr><td>whisper-medium.en</td><td>3.3/4.3</td><td>13.1/10.5</td><td>49.1/49.2</td><td>23.8/20.6</td></tr>
  <tr><td>whisper-small.en</td><td>4.2/5.3</td><td>13.1/10.8</td><td>48.4/51.2</td><td>12.5/12.7</td></tr>
  <tr><td>whisper-tiny</td><td>9.8/11.3</td><td>19.3/18.2</td><td>60.8/63.0</td><td>21.0/22.3</td></tr>
</table>

Table 11: Actual and approximated CER (\( \downarrow \)), separated by a slash, on out-of-distribution wild datasets. The regression model is trained independently for each ASR model on standard benchmarks, making the wild datasets out-of-distribution.
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th rowspan="2">IID</th>
    <th colspan="3">OOD</th>
  </tr>
  <tr>
    <th>D</th>
    <th>M</th>
    <th>D + M</th>
  </tr>
  <tr>
    <td rowspan="6">Substitution</td>
    <td>Base</td>
    <td>0.76</td>
    <td>0.77</td>
    <td>0.77</td>
    <td>0.79</td>
  </tr>
  <tr>
    <td>w/o S</td>
    <td>0.76</td>
    <td>0.76</td>
    <td>0.77</td>
    <td>0.78</td>
  </tr>
  <tr>
    <td>w/o PR</td>
    <td>2.22</td>
    <td>2.28</td>
    <td>2.26</td>
    <td>2.36</td>
  </tr>
  <tr>
    <td>w/ MPR (n=5)</td>
    <td>0.65</td>
    <td>0.68</td>
    <td>0.66</td>
    <td>0.69</td>
  </tr>
  <tr>
    <td>w/ mMPR (n=5)</td>
    <td>0.68</td>
    <td>0.69</td>
    <td>0.69</td>
    <td>0.72</td>
  </tr>
  <tr>
    <td>Base (r=17.8)</td>
    <td>0.97</td>
    <td>1.09</td>
    <td>0.98</td>
    <td>1.11</td>
  </tr>
  <tr>
    <td>Base (r=51.0)</td>
    <td>1.41</td>
    <td>1.39</td>
    <td>1.45</td>
    <td>1.42</td>
  </tr>
  <tr>
    <td rowspan="6">Insertion</td>
    <td>Base</td>
    <td>0.59</td>
    <td>0.58</td>
    <td>0.61</td>
    <td>0.59</td>
  </tr>
  <tr>
    <td>w/o S</td>
    <td>0.58</td>
    <td>0.56</td>
    <td>0.60</td>
    <td>0.57</td>
  </tr>
  <tr>
    <td>w/o PR</td>
    <td>0.86</td>
    <td>0.94</td>
    <td>0.86</td>
    <td>0.95</td>
  </tr>
  <tr>
    <td>w/ MPR (n=5)</td>
    <td>0.54</td>
    <td>0.55</td>
    <td>0.55</td>
    <td>0.57</td>
  </tr>
  <tr>
    <td>w/ mMPR (n=5)</td>
    <td>0.56</td>
    <td>0.56</td>
    <td>0.58</td>
    <td>0.57</td>
  </tr>
  <tr>
    <td>Base (r=17.8)</td>
    <td>0.63</td>
    <td>0.80</td>
    <td>0.65</td>
    <td>0.82</td>
  </tr>
  <tr>
    <td>Base (r=51.0)</td>
    <td>0.69</td>
    <td>0.76</td>
    <td>0.71</td>
    <td>0.79</td>
  </tr>
  <tr>
    <td rowspan="6">Deletion</td>
    <td>Base</td>
    <td>0.66</td>
    <td>0.63</td>
    <td>0.67</td>
    <td>0.64</td>
  </tr>
  <tr>
    <td>w/o S</td>
    <td>0.69</td>
    <td>0.60</td>
    <td>0.69</td>
    <td>0.60</td>
  </tr>
  <tr>
    <td>w/o PR</td>
    <td>1.01</td>
    <td>1.09</td>
    <td>1.06</td>
    <td>1.15</td>
  </tr>
  <tr>
    <td>w/ MPR (n=5)</td>
    <td>0.60</td>
    <td>0.56</td>
    <td>0.60</td>
    <td>0.56</td>
  </tr>
  <tr>
    <td>w/ mMPR (n=5)</td>
    <td>0.61</td>
    <td>0.57</td>
    <td>0.61</td>
    <td>0.58</td>
  </tr>
  <tr>
    <td>Base (r=17.8)</td>
    <td>0.76</td>
    <td>0.82</td>
    <td>0.78</td>
    <td>0.85</td>
  </tr>
  <tr>
    <td>Base (r=51.0)</td>
    <td>0.92</td>
    <td>1.05</td>
    <td>0.97</td>
    <td>1.09</td>
  </tr>
</table>

Table 12: Mean absolute deviation between predicted and true counts for substitutions, insertions, and deletions. Columns: IID (source in-distribution), D (source out-of-distribution), M (target in-distribution), D+M (target out-of-distribution). Lower values indicate better approximation.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="2">AMI_HM</th>
    <th colspan="2">CV11</th>
    <th colspan="2">Earnings22</th>
    <th colspan="2">Gigaspeech</th>
    <th colspan="2">LibriSpeech_clean</th>
  </tr>
  <tr>
    <th>WER/αWER</th><th>CER/αCER</th>
    <th>WER/αWER</th><th>CER/αCER</th>
    <th>WER/αWER</th><th>CER/αCER</th>
    <th>WER/αWER</th><th>CER/αCER</th>
    <th>WER/αWER</th><th>CER/αCER</th>
  </tr>
  <tr><td>ase.wav2vec2-librispeech</td><td>28.43/0.5</td><td>13.8/17.6</td><td>25.09/2.7</td><td>11.7/15.0</td><td>37.3/3.3</td><td>21.3/16.1</td><td>16.6/16.5</td><td>6.9/7.4</td><td>1.8/3.8</td><td>0.5/2.2</td></tr>
  <tr><td>camary-lb</td><td>15.4/17.6</td><td>9.2/12.7</td><td>8.7/14.2</td><td>4.1/8.5</td><td>21.8/16.0</td><td>15.8/9.1</td><td>11.1/6.9</td><td>5.4/3.4</td><td>1.5/7.5</td><td>0.5/3.5</td></tr>
  <tr><td>data2vec-audio-base-960h</td><td>39.9/40.4</td><td>19.9/21.5</td><td>37.8/42.3</td><td>18.3/21.7</td><td>50.8/48.6</td><td>28.0/25.0</td><td>23.8/23.5</td><td>10.1/10.8</td><td>2.8/10.0</td><td>0.9/16.1</td></tr>
  <tr><td>distil-large-v2</td><td>18.4/16.1</td><td>11.0/12.7</td><td>18.2/17.6</td><td>8.7/10.6</td><td>37.3/3.2</td><td>21.3/16.1</td><td>17.1/16.7</td><td>7.1/8.0</td><td>1.8/3.8</td><td>0.5/2.2</td></tr>
  <tr><td>distil-large-v2</td><td>17.8/16.8</td><td>11.2/11.5</td><td>14.2/19.7</td><td>7.1/10.6</td><td>19.3/20.0</td><td>12.5/13.7</td><td>12.8/8.2</td><td>7.1/5.4</td><td>3.4/6.7</td><td>1.5/4.2</td></tr>
  <tr><td>distil-large-v3</td><td>18.5/17.3</td><td>11.6/11.7</td><td>13.7/19.4</td><td>6.6/10.3</td><td>18.4/19.8</td><td>12.1/13.0</td><td>12.2/7.9</td><td>6.9/5.3</td><td>2.0/6.6</td><td>1.2/4.1</td></tr>
  <tr><td>distil-small-en</td><td>18.5/18.4</td><td>11.1/12.6</td><td>18.5/23.1</td><td>9.4/12.5</td><td>21.2/21.4</td><td>13.6/14.7</td><td>13.1/16.6</td><td>7.3/5.7</td><td>3.7/7.6</td><td>1.6/5.4</td></tr>
  <tr><td>hubert-large-m6-large</td><td>36.5/39.9</td><td>19.3/21.5</td><td>35.3/32.3</td><td>18.3/20.1</td><td>40.1/36.7</td><td>24.4/21.8</td><td>24.1/21.7</td><td>13.8/16.7</td><td>3.3/28.5</td><td>1.5/27.2</td></tr>
  <tr><td>hf-scanless-m4-medium</td><td>40.6/37.2</td><td>29.5/28.9</td><td>11.3/14.3</td><td>6.0/7.4</td><td>33.7/15.9</td><td>23.9/26.4</td><td>30.2/28.1</td><td>22.3/21.7</td><td>3.8/5.3</td><td>1.6/29.9</td></tr>
  <tr><td>hubert-large-b960-ft</td><td>31.1/33.6</td><td>15.2/19.8</td><td>24.1/28.8</td><td>10.6/13.6</td><td>37.6/34.4</td><td>20.6/16.3</td><td>19.3/18.3</td><td>8.1/7.8</td><td>2.1/3.7</td><td>0.6/1.6</td></tr>
  <tr><td>hubert-large-b960-ft</td><td>31.1/34.6</td><td>15.0/18.2</td><td>24.1/28.7</td><td>10.5/13.9</td><td>37.6/34.9</td><td>20.4/17.4</td><td>18.1/17.4</td><td>7.3/7.8</td><td>2.0/3.8</td><td>0.6/1.7</td></tr>
  <tr><td>hubert-large-b960-ft</td><td>37.0/36.2</td><td>17.0/20.8</td><td>32.7/25.8</td><td>8.5/10.8</td><td>34.1/30.6</td><td>19.1/18.9</td><td>19.4/18.9</td><td>8.4/6.5</td><td>4.2/2.9</td><td>1.3/2.7</td></tr>
  <tr><td>mms-1b-f102</td><td>75.4/73.3</td><td>35.3/31.9</td><td>42.6/45.3</td><td>17.8/19.9</td><td>50.6/52.3</td><td>24.2/26.5</td><td>37.2/35.7</td><td>15.7/15.2</td><td>1.5/17.3</td><td>5.1/5.9</td></tr>
  <tr><td>moonshine-base</td><td>15.6/24.7</td><td>9.4/16.7</td><td>20.8/25.4</td><td>10.8/13.8</td><td>24.3/25.6</td><td>15.9/16.6</td><td>14.2/10.4</td><td>8.1/6.8</td><td>3.4/6.3</td><td>1.3/3.7</td></tr>
  <tr><td>moonshine-tiny</td><td>21.5/25.3</td><td>12.8/16.7</td><td>26.1/31.7</td><td>14.1/17.3</td><td>31.2/27.2</td><td>19.7/20.2</td><td>16.0/16.1</td><td>9.4/7.5</td><td>4.5/10.1</td><td>1.8/4.2</td></tr>
  <tr><td>pulse-ctc-1.0b</td><td>17.0/21.1</td><td>9.0/11.6</td><td>10.7/21.1</td><td>5.0/11.8</td><td>24.0/19.6</td><td>13.0/18.6</td><td>12.0/15.6</td><td>6.0/5.2</td><td>2.0/5.1</td><td>0.7/2.5</td></tr>
  <tr><td>parakeet-ctc-1.1b</td><td>15.7/21.4</td><td>9.0/15.3</td><td>10.5/20.1</td><td>5.2/11.0</td><td>24.0/17.7</td><td>16.6/10.7</td><td>12.7/2.9</td><td>6.2/5.0</td><td>1.8/5.4</td><td>0.5/2.6</td></tr>
  <tr><td>parakeet-mm-0.6b</td><td>18.8/24.0</td><td>11.7/17.9</td><td>8.5/19.9</td><td>4.2/10.8</td><td>25.1/18.7</td><td>17.5/11.5</td><td>11.7/9.0</td><td>6.2/5.4</td><td>1.8/5.5</td><td>0.6/3.2</td></tr>
  <tr><td>parakeet-mm-1.1b</td><td>18.6/25.3</td><td>11.7/17.2</td><td>6.9/19.6</td><td>3.4/10.5</td><td>25.7/17.9</td><td>18.4/11.4</td><td>11.3/8.4</td><td>6.0/5.5</td><td>1.5/5.0</td><td>0.5/3.3</td></tr>
  <tr><td>parakeet-ctc-1.1b</td><td>17.1/23.5</td><td>9.1/13.6</td><td>12.0/19.6</td><td>4.2/10.4</td><td>24.5/16.6</td><td>17.6/10.8</td><td>10.8/8.8</td><td>4.4/6.7</td><td>1.8/10.0</td><td>0.4/2.9</td></tr>
  <tr><td>parakeet-ctc-1.1b</td><td>18.5/18.8</td><td>10.7/13.6</td><td>12.7/17.7</td><td>6.9/10.1</td><td>22.4/14.8</td><td>15.7/9.2</td><td>12.6/8.2</td><td>6.2/5.0</td><td>2.6/6.7</td><td>0.9/3.8</td></tr>
  <tr><td>seamless-mmt-v2-large</td><td>43.0/42.3</td><td>30.2/20.0</td><td>8.2/12.3</td><td>8.9/6.3</td><td>47.3/47.4</td><td>33.7/33.9</td><td>25.7/23.2</td><td>18.1/17.2</td><td>2.7/4.4</td><td>1.0/2.5</td></tr>
  <tr><td>speechml-1.5B</td><td>67.7/69.9</td><td>51.5/55.1</td><td>18.5/22.7</td><td>10.7/12.2</td><td>50.1/48.2</td><td>38.3/35.4</td><td>27.5/26.0</td><td>18.1/18.1</td><td>10.5/11.1</td><td>3.7/9.2</td></tr>
  <tr><td>speechml-2B</td><td>86.8/86.6</td><td>52.1/52.8</td><td>20.8/28.2</td><td>16.5/18.3</td><td>47.3/44.0</td><td>24.2/24.6</td><td>13.3/18.8</td><td>7.0/9.3</td><td>4.5/8.8</td><td>4.5/8.8</td></tr>
  <tr><td>st_en_conformer_ctc_large</td><td>15.3/19.9</td><td>7.9/13.4</td><td>10.4/14.5</td><td>4.7/8.0</td><td>24.8/20.0</td><td>16.4/10.7</td><td>13.2/10.6</td><td>5.9/5.5</td><td>2.2/3.7</td><td>0.7/2.4</td></tr>
  <tr><td>st_en_conformer_ctc_small</td><td>21.2/24.4</td><td>11.2/15.4</td><td>19.1/24.1</td><td>8.9/12.2</td><td>29.3/25.3</td><td>19.0/14.1</td><td>15.5/14.9</td><td>7.2/7.7</td><td>3.9/5.4</td><td>1.4/3.1</td></tr>
  <tr><td>st_en_tasconformer_ctc_large</td><td>20.3/24.0</td><td>11.7/15.5</td><td>9.9/13.9</td><td>4.6/10.2</td><td>27.3/21.5</td><td>18.3/11.3</td><td>14.5/14.7</td><td>7.2/8.2</td><td>1.9/5.2</td><td>0.7/2.8</td></tr>
  <tr><td>st_en_tasconformer_ctc_large</td><td>19.8/23.0</td><td>11.3/14.1</td><td>8.3/10.8</td><td>3.8/11.0</td><td>28.1/25.8</td><td>18.3/11.2</td><td>13.2/10.2</td><td>7.2/8.8</td><td>1.8/3.9</td><td>0.6/2.6</td></tr>
  <tr><td>wav2vec2-base-960h</td><td>37.9/38.7</td><td>18.7/21.9</td><td>40.6/45.7</td><td>19.5/22.8</td><td>51.1/48.6</td><td>28.2/25.4</td><td>26.2/26.6</td><td>11.7/12.2</td><td>3.7/4.5</td><td>1.1/1.9</td></tr>
  <tr><td>wav2vec2-conformer-red-pos-large-960h-ft</td><td>35.0/38.7</td><td>18.5/24.1</td><td>23.7/28.0</td><td>10.7/13.7</td><td>38.4/36.2</td><td>21.7/17.6</td><td>18.5/17.2</td><td>8.5/7.9</td><td>1.6/3.3</td><td>0.5/1.5</td></tr>
  <tr><td>wav2vec2-conformer-rope-large-960h-ft</td><td>34.0/36.6</td><td>18.0/22.2</td><td>23.6/28.5</td><td>11.6/15.0</td><td>36.9/33.9</td><td>25.1/16.7</td><td>17.9/17.7</td><td>7.8/7.7</td><td>1.8/3.0</td><td>0.5/1.6</td></tr>
  <tr><td>wav2vec2-large-960h-bf64-self</td><td>29.1/31.5</td><td>15.9/19.5</td><td>23.1/28.8</td><td>11.0/15.0</td><td>36.7/32.5</td><td>20.8/15.7</td><td>17.6/12.7</td><td>7.5/8.0</td><td>1.7/3.5</td><td>0.5/1.9</td></tr>
  <tr><td>wav2vec2-large-robust-ft-libri-960h</td><td>30.5/33.9</td><td>13.6/19.2</td><td>25.0/29.3</td><td>10.7/13.8</td><td>37.1/33.5</td><td>20.5/15.7</td><td>18.0/17.6</td><td>7.1/7.7</td><td>2.8/4.3</td><td>0.8/2.1</td></tr>
  <tr><td>whisper-large</td><td>18.5/18.8</td><td>12.3/13.0</td><td>13.0/18.0</td><td>6.6/9.3</td><td>18.2/20.3</td><td>12.3/14.9</td><td>12.7/7.7</td><td>7.1/5.1</td><td>2.8/5.1</td><td>1.4/3.5</td></tr>
  <tr><td>whisper-large-v2</td><td>18.6/17.1</td><td>12.3/12.0</td><td>13.0/15.5</td><td>4.9/6.9</td><td>18.2/20.7</td><td>12.1/14.9</td><td>12.5/7.3</td><td>7.2/4.9</td><td>2.2/4.0</td><td>0.9/2.9</td></tr>
  <tr><td>whisper-large-v3</td><td>19.0/17.1</td><td>12.3/12.0</td><td>9.9/14.5</td><td>4.9/6.9</td><td>18.2/20.7</td><td>12.1/14.9</td><td>12.5/7.3</td><td>7.2/4.9</td><td>2.2/4.0</td><td>0.9/2.9</td></tr>
  <tr><td>whisper-large-v3-turbo</td><td>19.0/17.5</td><td>12.3/11.6</td><td>6.3/8.1</td><td>18.2/11.1</td><td>12.9/15.6</td><td>12.6/8.6</td><td>7.1/4.6</td><td>2.4/4.4</td><td>1.1/2.5</td><td>1.1/2.5</td></tr>
  <tr><td>whisper-medium-en</td><td>20.8/18.3</td><td>13.7/14.5</td><td>14.3/17.8</td><td>9.2/9.2</td><td>20.1/22.6</td><td>13.3/16.5</td><td>12.8/7.6</td><td>7.6/5.5</td><td>3.3/5.4</td><td>1.8/3.5</td></tr>
  <tr><td>whisper-small-en</td><td>18.7/18.9</td><td>12.3/12.2</td><td>12.6/12.0</td><td>9.2/10.1</td><td>20.1/22.9</td><td>13.2/13.3</td><td>7.1/5.1</td><td>3.3/5.6</td><td>1.4/3.0</td><td>1.4/3.0</td></tr>
  <tr><td>whisper-tiny</td><td>26.7/25.1</td><td>16.7/17.0</td><td>33.5/40.3</td><td>17.7/20.9</td><td>33.8/35.4</td><td>22.0/25.4</td><td>20.6/18.2</td><td>12.0/11.0</td><td>7.9/11.2</td><td>3.4/9.1</td></tr>
</table>

Table 13: Actual and approximated WER and CER, separated by a slash, across five standard datasets. The regression model is trained on nine datasets and tested on one, with this process repeated for all datasets, ensuring that the test data is always out-of-distribution.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th colspan="3">peoples_speech</th>
    <th colspan="3">slue_voxceleb</th>
    <th colspan="3">spgsspeech_S</th>
    <th colspan="3">tedium-dev-test</th>
    <th colspan="3">vospopuli_en</th>
  </tr>
  <tr>
    <th>WER@WER</th><th>CER@CER</th><th>WER/CER</th>
    <th>WER@WER</th><th>CER@CER</th><th>WER/CER</th>
    <th>WER@WER</th><th>CER@CER</th><th>WER/CER</th>
    <th>WER@WER</th><th>CER@CER</th><th>WER/CER</th>
    <th>WER@WER</th><th>CER@CER</th><th>WER/CER</th>
  </tr>
  <tr><td>ase-wave2vec-libspeech</td><td>15.6/0.29</td><td>19.8/17.7</td><td>0.82/0.47</td><td>19.5/20.4</td><td>9.8/12.2</td><td>1.1/1.22</td><td>4.8/4.7</td><td>10.3/11.11</td><td>5.2/5.8</td><td>1.4/1.326</td><td>6.6/5.1</td></tr>
  <tr><td>camfly-1b</td><td>14.6/2.25</td><td>11.1/15.2</td><td>1.4/19.11</td><td>10.8/8.2</td><td>2.6/7.67</td><td>2.03/9</td><td>7.9/7.6</td><td>9.0/0.5</td><td>6.4/4.9</td><td>3.9/4.1</td></tr>
  <tr><td>data2vec-audio-base-96fb</td><td>43.4/3.86</td><td>42.4/0.28</td><td>26.1/27.6</td><td>13.0/15.5</td><td>19.2/19.8</td><td>8.27/9</td><td>13.6/14.2</td><td>6.36/6.44</td><td>18.0/17.5</td><td>8.5/7.1</td></tr>
  <tr><td>data2vec-audio-large-96fb</td><td>51.5/9.1</td><td>50.9/7.76</td><td>23.5/23.9</td><td>21.5/12.7</td><td>16.6/15.9</td><td>12.2/21.7</td><td>13.8/9.4</td><td>18.4/5.2</td><td>14.2/9.6</td><td>6.6/8.5</td></tr>
  <tr><td>distil-large-v2</td><td>17.4/21.8</td><td>12.2/14.1</td><td>16.0/10.8</td><td>11.4/7.4</td><td>3.7/7.6</td><td>1.8/4.5</td><td>10.4/6.5</td><td>8.8/5.4</td><td>9.5/8.2</td><td>8.5/4.6</td></tr>
  <tr><td>distil-large-v3</td><td>17.4/21.6</td><td>12.4/13.8</td><td>14.4/10.0</td><td>10.3/8.6</td><td>3.6/7.4</td><td>1.8/4.5</td><td>10.7/9.2</td><td>8.6/5.7</td><td>9.3/6.7</td><td>8.5/4.1</td></tr>
  <tr><td>distil-small-en</td><td>19.1/0.23</td><td>13.3/13.4</td><td>15.9/11.4</td><td>11.7/7.8</td><td>4.0/7.9</td><td>1.9/4.7</td><td>10.8/8.8</td><td>9.1/5.4</td><td>10.7/6.4</td><td>6.4/4.1</td></tr>
  <tr><td>hubert-large-imodel-large</td><td>39.8/3.65</td><td>35.3/10.3</td><td>13.7/28.8</td><td>16.6/18.7</td><td>11.0/11.0</td><td>19.1/14.5</td><td>15.4/4.4</td><td>18.0/5.5</td><td>6.1/0.6</td><td>5.1/0.6</td></tr>
  <tr><td>hf-senseless-m4l-medium</td><td>43.6/4.57</td><td>33.6/4.32</td><td>20.9/47.4</td><td>43.2/40.3</td><td>19.1/15.5</td><td>8.8/1.04</td><td>27.0/12.6</td><td>21.3/0.00</td><td>8.8/1.54</td><td>5.5/4.4</td></tr>
  <tr><td>hubert-large-ls90b-ft</td><td>34.1/3.13</td><td>18.8/17.7</td><td>20.8/22.0</td><td>10.1/12.3</td><td>11.6/12.4</td><td>4.9/4.6</td><td>11.0/12.0</td><td>5.3/3.54</td><td>15/01.3.6</td><td>6.9/5.1</td></tr>
  <tr><td>hubert-large-ls90b-ft</td><td>35.5/3.51</td><td>19.5/0.10</td><td>20.3/22.1</td><td>9.9/12.3</td><td>11.9/12.3</td><td>4.8/4.5</td><td>10.1/11.7</td><td>4.2/5.1</td><td>14.5/12.1</td><td>6.7/5.3</td></tr>
  <tr><td>mosix-tt-1a-ft</td><td>32.2/6.06</td><td>183.0/17</td><td>29.3/26.3</td><td>16.0/13.4</td><td>40.0/26.5</td><td>13.3/9.3</td><td>19.9/3.5</td><td>8.8/8.36</td><td>4.4/3.1</td></tr>
  <tr><td>mms-1b-ft102</td><td>52.4/57.2</td><td>26.0/25.5</td><td>51.7/46.7</td><td>26.1/23.2</td><td>19.1/29.2</td><td>5.9/8.6</td><td>29.7/0.3</td><td>16.9/2.9</td><td>22.8/20.9</td><td>8.2/1.5</td></tr>
  <tr><td>monosmile-base</td><td>26.4/2.66</td><td>18.17/7.5</td><td>17.0/13.8</td><td>11.6/9.1</td><td>6.4/7.5</td><td>1.8/5.3</td><td>2.8/7.0</td><td>1.4/1.1</td><td>11.7/9.1</td><td>8.6/7.2</td></tr>
  <tr><td>moonshine-tiny</td><td>31.8/2.24</td><td>24.7/3.09</td><td>19.8/18.2</td><td>13.2/11.6</td><td>9.1/9.9</td><td>0.9/4.2</td><td>3.0/13.9</td><td>2.3/0.8</td><td>4.7/2.9</td><td>3.2/7.2</td></tr>
  <tr><td>parakeet-ctc-0.6b</td><td>24.2/10.7</td><td>19.6/6.9</td><td>13.0/13.0</td><td>8.7/6.8</td><td>2.5/1.63</td><td>0.84</td><td>7.6/2.2</td><td>2.48/3.6</td><td>7.0/3.7</td><td>4.1/1.97</td></tr>
  <tr><td>parakeet-ctc-1.1b</td><td>20.7/18.1</td><td>13.1/01.8</td><td>13.0/11.6</td><td>8.7/8.3</td><td>1.6/6.7</td><td>0.7/1.3</td><td>4.7/1.8</td><td>2.04/4</td><td>3.2/6.6</td><td>3.93/3.3</td></tr>
  <tr><td>parakeet-mmt-0.6b</td><td>26.1/7.9</td><td>15.40/1.2</td><td>14.5/11.6</td><td>10.0/8.6</td><td>4.9/7.3</td><td>2.9/3.7</td><td>5.0/8.9</td><td>3.05/4</td><td>6.0/4.7</td><td>5.9/3.7</td></tr>
  <tr><td>parakeet-mmt-1.1b</td><td>23.3/24.7</td><td>16.5/11.8</td><td>14.1/18.0</td><td>9.9/3.89</td><td>4.5/5.7</td><td>2/4.7</td><td>5.27/7.7</td><td>3.54/5</td><td>5.6/0.56</td><td>3.47/8.2</td></tr>
  <tr><td>parakeet-cct-c-1.1b</td><td>24.5/57.9</td><td>16.7/11.6</td><td>15.1/12.9</td><td>10.4/9.7</td><td>9.3/10.2</td><td>1.4/3.21</td><td>24.4/12.5</td><td>5.5/3.4</td><td>10.3/1.1</td></tr>
  <tr><td>parakeet-dtt-etc-110m</td><td>16.6/21.4</td><td>11.5/11.5</td><td>15.3/11.5</td><td>10.7/8.3</td><td>3.8/7.3</td><td>2.2/4.0</td><td>5.26/6.2</td><td>3.31/4</td><td>7.5/6.2</td><td>4.6/3.1</td></tr>
  <tr><td>seamless-mmt-v2-large</td><td>35.0/36.3</td><td>25.9/24.9</td><td>45.1/43.2</td><td>35.9/34.7</td><td>11.7/13.6</td><td>7.69/8.7</td><td>26.5/5.5</td><td>21.0/9.11</td><td>8.0/7.5</td><td>5.7/5.0</td></tr>
  <tr><td>speechlen-1.5b</td><td>41.5/114.4</td><td>32.7/52.0</td><td>60.3/66.08</td><td>46.5/8.6</td><td>10.6/10.9</td><td>6.25/70</td><td>19.4/17.6</td><td>14.1/119.1</td><td>30.9/29.9</td><td>22.0/12.1</td></tr>
  <tr><td>wav2vec-2b</td><td>25.9/0.34</td><td>28.8/158.8</td><td>28.2/177.8</td><td>28.5/143.8</td><td>14.1/14.35</td><td>18.9/8.7</td><td>12.4/25.3</td><td>8.28/7.</td><td>18.5/15.5</td></tr>
  <tr><td>st_en_conformer_etc_large</td><td>24.2/21.5</td><td>15.2/13.0</td><td>16.2/14.7</td><td>7.6/9.4</td><td>7/9.73</td><td>4.1/4.3</td><td>5.97/7</td><td>3.31/4</td><td>8.95/3.5</td><td>3.9/2.7</td></tr>
  <tr><td>st_en_conformer_etc_small</td><td>31.3/26.9</td><td>19.0/13.7</td><td>16.4/17.8</td><td>19.6/11.7</td><td>10.0/9.4</td><td>5.1/4.1</td><td>8.09/9</td><td>5.40/5.23</td><td>8.9/7.3</td><td>5.0/3.8</td></tr>
  <tr><td>st_en_tacotron_conformer_etc_large</td><td>26.3/20.14</td><td>18.1/8.23</td><td>15.4/14.0</td><td>9.0/6.4</td><td>3.7/8.4</td><td>1.5/7.8</td><td>16.7/3</td><td>6.3/18.4</td><td>6.8/3.5</td><td>8.8/3.3</td></tr>
  <tr><td>st_edem-conformer_invadet-large</td><td>26.7/9.34</td><td>18.5/13.8</td><td>15.5/13.5</td><td>13.2/11.1</td><td>6.0/7.9</td><td>2.1/1.3</td><td>4.0/9.0</td><td>2.4/2.4</td><td>8.6/3.9</td><td>4.1/2.8</td></tr>
  <tr><td>wav2vec-base-96fb</td><td>44.7/40.1</td><td>24.5/20.6</td><td>27.3/78.7</td><td>13.5/15.7</td><td>21.5/24.2</td><td>8.98/7</td><td>18.3/14.7</td><td>6.16/6</td><td>20.95/14.6</td><td>9.17/8.6</td></tr>
  <tr><td>wav2vec2-conformer-rd-pos-large-96fb-ft</td><td>37.4/34.7</td><td>21.9/17.9</td><td>20.3/23.0</td><td>10.6/13.4</td><td>12.1/12.2</td><td>5.26/4</td><td>11.7/12.3</td><td>6.66/6</td><td>14.81/13.1</td><td>6.96/5.6</td></tr>
  <tr><td>wav2vec2-conformer-rope-large-96fb-ft</td><td>39.5/35.7</td><td>20.4/52.3</td><td>20.6/22.1</td><td>10.7/14.2</td><td>11.7/12.5</td><td>5.14/8</td><td>10.0/13.8</td><td>5.86/8.2</td><td>14.51/4.2</td><td>6.94/5.4</td></tr>
  <tr><td>wav2vec2-large-96fb-b6bfscf</td><td>32.29/4.2</td><td>18.7/15.7</td><td>20.8/28.0</td><td>10.7/12.9</td><td>10.4/12.2</td><td>4/3.47</td><td>9.51/9.0</td><td>4.21/9</td><td>13.1/5.1</td><td>6.4/5.2</td></tr>
  <tr><td>wav2vec2-large-robust-ft-libri-96fb</td><td>36.2/32.6</td><td>19.2/8.4</td><td>20.9/23.0</td><td>9.1/6.25</td><td>11.3/18.5</td><td>5.04/8</td><td>10.6/11.9</td><td>4.8/6.5</td><td>15.41/0.4</td><td>7.0/0.8</td></tr>
  <tr><td>whisper-lm*2</td><td>31.2/2.7</td><td>24.0/23.8</td><td>17.6/16.2</td><td>13.1/15.9</td><td>3.7/14.4</td><td>2.1/14</td><td>19.43/15.4</td><td>15.03/3.1</td><td>0.38/5.63</td><td>5.82/0</td></tr>
  <tr><td>whisper-large-na2</td><td>30.7/6.6</td><td>18.5/2.8</td><td>13.9/4.1</td><td>8.3/8.24</td><td>1.17/3.6</td><td>2.0/2.6</td><td>24.31/2.1</td><td>19.41/4.4</td><td>7.68/4.5</td><td>5.33/2.8</td></tr>
  <tr><td>whisper-large-v3</td><td>20.4/27.5</td><td>15.6/11.9</td><td>15.6/11.9</td><td>11.8/8.7</td><td>3.26/3</td><td>1.74/0.5</td><td>10.58/3</td><td>8.7/5.5</td><td>11.8/8.6</td><td>8.3/6.1</td></tr>
  <tr><td>whisper-large-v3-srhub</td><td>16.0/32.7</td><td>12.0/16.8</td><td>15.3/11.9</td><td>11.5/9.0</td><td>3.16/3</td><td>1.79/1.8</td><td>9.43/2.1</td><td>8.55/5</td><td>13.71/11</td><td>9.8/7.9</td></tr>
  <tr><td>whisper-large-en</td><td>20.1/25.2</td><td>15.3/0.08</td><td>21.2/16.1</td><td>16.6/13.2</td><td>4.2/7</td><td>2.52/0</td><td>17.3/41.6</td><td>18.7/14.0</td><td>9.76/0</td><td>3.7/3.3</td></tr>
  <tr><td>whisper-medium</td><td>21.9/31.1</td><td>15.4/4.89</td><td>14.3/11.1</td><td>9.8/5.49</td><td>2.07/5</td><td>1/0.75</td><td>10.9/6.1</td><td>15.9/2.8</td><td>4.38/1.8</td><td>6.71/3.2</td></tr>
  <tr><td>whisper-small-en</td><td>30.1/31.9</td><td>21.7/21.5</td><td>24.0/20.8</td><td>17.5/14.4</td><td>8.1/11.9</td><td>3.9/8.7</td><td>17.6/15.3</td><td>13.09/5</td><td>13.2/11.2</td><td>7.4/6.3</td></tr>
</table>

Table 14: Actual and approximated WER and CER, separated by a forward slash, across five standard datasets. The regression model is trained on nine datasets and tested on one, with this process repeated for all datasets, ensuring that the test data is always out-of-distribution.

Reviewer paper summaries: The paper proposes a method to approximate word error rate (WER) and character error rate (CER) without requiring ground-truth transcripts. The approach includes:

- A multimodal embedding model to measure similarity between audio and text hypothesis.
- A proxy reference generated by a strong ASR model to obtain proxy WER/CER features.
- A regression model (an ensemble of regressors) trained on these features to predict actual WER/CER.

The authors evaluate over 40 ASR models on 14 datasets covering both standard benchmarks and challenging “in-the-wild” scenarios. They compare their approach to a recent baseline (eWER3) and show substantially lower error in approximating WER/CER. They also conduct ablations on the number/quality of proxy references and the size of training data for the regression model.

Overall, the work demonstrates that high-quality approximations of WER/CER are achievable without labeled reference transcripts, especially useful for domains where ground truth is scarce or expensive.
Summary of strengths: - The paper evaluates a wide range of ASR systems across many datasets, including both standard benchmarks and “in-the-wild” ones, which strengthens the claims.
- The ablation shows that even a relatively small set of examples (∼1k) can suffice to train the regression model, which is promising for low-resource or domain-specific settings.
- The paper compares against eWER3 and demonstrates an improvement.
- It's clear about its goals, provides relevant methodological details, and outlines results logically. The ablations (e.g., number/quality of references, effect of similarity embeddings, etc.) offer additional insights.
- The approach can be used to monitor or filter ASR outputs in unlabeled settings. This could be valuable for iterative data selection, pseudo-labeling, or real-time system evaluation where references cannot be collected.
Summary of weaknesses: - If the proxy model is consistently better than the primary ASR system, the natural question arises: Why not deploy the proxy directly in production? The authors acknowledge this partially by noting cost or domain mismatch, but the paper could use more explicit discussion of when the “approximation + cheaper model” paradigm is truly preferred over “just use the better model.”
- Additionally, for extremely resource-constrained settings, running a second large ASR for the proxy might not be feasible. Further elaboration on real-time or large-scale inference scenarios would strengthen the paper.
- The paper primarily evaluates English ASR systems. Although the approach is conceptually language-agnostic, more discussion (or experiments) on how it might extend to low-resource languages or multilingual speech domains would be beneficial.
- While the authors show that the regression pipeline can run on CPUs, the paper could better highlight the trade-offs involved in extracting embeddings (GPU usage) plus running an additional proxy model. This might not be trivial at scale.
Comments, suggestions and typos: - Could you elaborate on real-world deployment scenarios in which you wouldn’t simply replace the target ASR with the stronger proxy ASR, but instead rely on your metric approximation approach?
- You discuss a ranking-based selection of the proxy reference. Is there any risk that the “best” proxy on your training set might degrade in new, unseen domains? If so, do you envision re-ranking or adaptive selection?
- For large-scale or streaming applications, how might we limit the computational overhead of extracting SONAR embeddings and running a second ASR pass?
- Information overload: The tables and figures contain a large amount of information, so much so that it's difficult to interpret them. Is it possible to present this information in a more focused way?

Typos
------
- 532 - “model’” -> “model’s”
Overall assessment scores: 4
Metareview score: 3.5
Metareview: This paper proposes a regression-based framework for reference-free ASR evaluation using multi-modal embeddings. The approach is intuitive, and the authors provide comprehensive comparisons across ASR systems and datasets. Many concerns from the reviewers—such as dataset selection and practicality of proxy use—were addressed in the rebuttal with additional results and clarifications. While some questions remain, the core idea is sound and potentially impactful, especially for low-resource scenarios.
Summary of reasons to publish: * The use of a multi-modal embedder for evaluating ASR quality is well-motivated and technically sound.
* The regression approach based on surrogate references is reasonable and aligns with the reference-free goal.
* The empirical study is comprehensive, covering a wide range of ASR systems and datasets.
* The method shows promise for low-resource scenarios, as demonstrated in the ablation study.
* The paper is generally well-structured and easy to follow.