Speech Robust Bench: A Robustness Benchmark For Speech Recognition

## Abstract

Abstract As Automatic Speech Recognition (ASR) models become ever more pervasive, it is important to ensure that they make reliable predictions under corruptions present in the physical and digital world.
We propose Speech Robust Bench ( SRB ), a comprehensive benchmark for evaluating the robustness of ASR models to diverse corruptions. SRB is composed of 114 input perturbations which simulate an heterogeneous range of corruptions that ASR models may encounter when deployed in the wild. We use SRB to evaluate the robustness of several state-of-the-art ASR models and observe that model size and certain modeling choices such as the use of discrete representations, or self-training appear to be conducive to robustness. We extend this analysis to measure the robustness of ASR models on data from various demographic subgroups, namely English and Spanish speakers, and males and females. Our results revealed noticeable disparities in the model’s robustness across subgroups. We believe that SRB will significantly facilitate future research towards robust ASR models, by making it easier to conduct comprehensive and comparable robustness evaluations.

## 1 Introduction

As novel ML models continue to be developed and deployed at an ever-increasing rate, it has become crucial to ensure that their robustness to corruptions arising from a myriad of sources, including the environment, sensing apparatus, and even malicious actors is well understood. To this end, prior works have developed comprehensive robustness benchmarks, particularly for vision [, , , ] and natural language processing models [, ], that evaluate several aspects of a model’s performance under a variety of input perturbations. These benchmarks have proven to be invaluable to the advancement of research into more robust models because (1) they unify robustness evaluations, thus enabling meaningful comparisons across models and allowing progress to be accurately tracked, and (2) they make it easier for researchers to comprehensively evaluate the robustness of their models by aggregating a diverse and representative set of perturbations in a single benchmark.

While several robustness benchmark datasets exist for Automatic Speech Recognition (ASR) models [, , , , , , , , , , , ], none of the currently existing ones is in any sense comprehensive: each benchmark measures the model’s robustness w.r.t. to one or a few specific types of perturbations, thereby putting the onus on model developers to find and collect all the relevant benchmarks to evaluate their model comprehensively. This has often resulted in model developers evaluating their models on disparate benchmarks  [, , , ], which makes it hard to reliably compare performance and robustness across models. Recently, Huggingface Open ASR Leaderboard [] has sought to unify ASR model evaluations by developing a benchmark consisting of several real-world speech datasets. Although evaluating models on exclusively natural data may accurately reflect their real-world performance, it is generally not informative about the specific types of perturbations the models are weak or strong against, because the noise sources present in these datasets are not controlled or even fully known. For example, the crowdsourced recordings in Common Voice [] contain a variety of distortions including sensor noise from low-quality equipment, background noise, and mispronunciation by non-native speakers.
Additionally, crucial digital perturbations like special effects, computer-generated speech, and adversarial examples, to which humans are usually invariant, are largely overlooked by existing benchmarks.

In this paper, we propose Speech Robust Bench (SRB), a benchmark for comprehensively evaluating the robustness of ASR models to input perturbations and corruptions. SRB is designed to address the aforementioned major shortcomings of existing ASR robustness benchmarks, i.e., that (1) they are often specialized and thus are not individually comprehensive, (2) even taken together, they overlook important perturbation types. SRB addresses these shortcomings by curating a large comprehensive bank of 114 perturbations and inherently noisy data sources.
The perturbation bank includes distortions and variations caused by the environment, recording equipment, special effects, computer-generated speech, and significantly, adversarial attacks, which are often overlooked by existing benchmarks. When selecting inherently noisy data sources, we take special care to ensure that the sources of variation in each data source are limited and known, so that the models’ specific weaknesses can be identified. These sources contain speech with non-native English accents and inter-personal conversations, giving rise to articulatory and lexical variability in speech.

To highlight the need for and the benefits of doing systematic and comprehensive robustness assessment, we evaluate the robustness of several popular ASR models (§4.1) using SRB. We observe that Whisper [] is the most robust on average among the models we tested, even outperforming the more recent Canary []. We conduct further analyses to disentangle the effects of model and training data size, revealing that larger models tend to be more robust than smaller models, even if the latter are trained on significantly more data. We further extend our analysis by evaluating the models’ robustness for the various population sub-groups, namely, English and non-English (Spanish) speakers, and male and female speakers. We find that significant disparities exist across these sub-groups, thus identifying issues with fairness of the models and highlighting areas where future work could provide clear improvements. Besides pinpointing robustness issues, this demonstrates the utility of SRB for fairness evaluations that consider robustness disparities across models as well.

To summarize we make the following contributions:

- •
We present SRB, a comprehensive robustness benchmark for ASR models, enabling direct and easy comparison of robustness evaluations between models to facilitate progress.
- •
We demonstrate the use of SRB by conducting a fine-grained robustness analysis for several popular models. We extend our analysis by using SRB to uncover disparities in the robustness of ASR for various sub-groups of speakers. This highlights the broad utility of this benchmark to the field of trustworthy AI.
- •
To facilitate out-of-the-box robustness evaluations for the research community, we have publicly released a large dataset ([https://huggingface.co/datasets/mshah1/speech_robust_bench](https://huggingface.co/datasets/mshah1/speech_robust_bench)) containing perturbed versions of LibriSpeech [], TEDLIUM [] and Spanish Multilingual LibriSpeech [] test sets, and accented speech from common voice and segemented near- and far-field audios from CHiME-6 [] and AMI[].
- •
We open source our code with clear documentation to allow for easy reproducibility and extensibility.([https://github.com/ahmedshah1494/speech_robust_bench](https://github.com/ahmedshah1494/speech_robust_bench))

## 2 Related Work

### 2.1 Robust Automatic Speech Recognition

Several techniques have been proposed for making Automatic Speech Recognition (ASR) models robust to input perturbations, such as noise and other signal corruptions [].
We can divide these techniques into two high-level categories: i) model-based and ii) feature-based.
Model-based techniques modify the models to make them more robust.
Examples of such approaches include adapting pre-trained models [, ], de-noising the audio before processing [, ], and training ASR models on noisy data [].
Since model-based strategies generally require access to noisy data [], they are most effective if the sources of noise, and/or the exact environment in which the ASR model will be deployed in are known, and one can gather data to represent them.
Feature-based approaches, on the other hand, involve developing handcrafted features, invariant to noise and corruptions in the signal [].
Several of these features are inspired by biological audition [, , ], while others use signal processing techniques []. Generally, these methods are designed to extract the components of the audio signal salient for speech production and perception, while discarding irrelevant components []. Consequently, they do not require precise knowledge of the environment and noise distributions.
Recently, however, handcrafted features have fallen out of favor, and have been replaced by features learned via end-to-end training of deep learning models on large amounts of data [, , , ].
Proponents of these techniques posit that models trained on larger datasets become more robust. Our evaluations in § 4 reveal that there are several input perturbations against which smaller models trained on less data outperform larger models trained on more data.

### 2.2 Adversarial Robustness

Adversarial perturbations can change the response of a model when added to their inputs, but are either imperceptible to humans or perceptually and semantically irrelevant enough to be ignored by them [, ]. Adversarially perturbed inputs are known as adversarial attacks. They can be targeted (aiming to change a prediction to a specific incorrect class), or un-targeted (aiming to change a prediction to any incorrect class []).
The design of adversarial attacks is determined by the level of knowledge the attacker is assumed to have about the target model. Attacks that assume full knowledge of the target model’s architecture and weights (white-box threat model) often use gradient-based optimization techniques [, , , , ].
Attackers who do not have any knowledge of the target model’s architecture and only have query access to it (black-box threat model) typically use gradient-free optimization methods [, , , , , ].
An intriguing property of adversarial perturbations is that they transfer between models [], and inputs [, ], i.e. perturbations designed for one model/input may be effective against others as well.
Our SRB includes two types of white box adversarial attacks; those that generate perturbations: 1) specific to each input [], 2) that cause models to mis-transcribe multiple inputs [].

### 2.3 Robustness Benchmarks for Speech

Several robustness benchmark datasets exist for Automatic Speech Recognition (ASR) models [, , , , , , , , , , , , , , , ]. Unfortunately, these benchmarks do not comprehensively evaluate model robustness. While past works such as [] did indeed propose benchmarks containing diverse perturbations, many of the recently proposed benchmarks measure the model’s robustness to one or a few specific types of perturbations. For example, some datasets [, , , ] focus exclusively on reverberant speech, while others focus only on multi-speaker scenarios [, ], environmental noise [, , , ], or accented speech [, ]. In this situation, the onus is on model developers to collect all relevant benchmarks and comprehensively evaluate their model.
Furthermore, certain types of digital perturbations, like special effects, computer-generated speech, and adversarial examples, which humans are usually invariant to (and thus ASR models are expected to be as well) are largely overlooked by existing benchmarks. To facilitate future research, SRB combines several benchmark datasets containing known noise sources to allow researchers to comprehensively evaluate the robustness of their models using a single benchmark.

In the absence of comprehensive benchmarks, prior works have tried to evaluate ASR model robustness in various ways.
For example, several works have used subsets of existing benchmarks to evaluate the model robustness [, , ], while others [, , ] have evaluated the robustness of models by computing their transcription accuracy on multiple natural speech datasets.
A key shortcoming of these approaches is that the choice of benchmarks and datasets varies across studies, making it difficult to compare results.
Recently, Huggingface Open ASR Leaderboard [] has sought to mitigate this issue of inconsistent evaluation setups by developing a benchmark comprising several real-world speech datasets.
Although evaluating models only on naturally noisy data may accurately reflect their real-world performance, it may not inform about the specific type of perturbations the models are weak against, because noise sources present in these datasets are not controlled or even fully known.
For example, the crowdsourced recordings in Common Voice [] contain a variety of distortions including sensor noise from low-quality equipment, background noise, and mispronunciation by non-native speakers.
If a model struggles with Common Voice, identifying the specific types of noise it is sensitive to is not a straightforward task.

By evaluating models on a diverse set of known perturbations and noise sources, including adversarial attacks, SRB complements and extends benchmarks, like [], that measure average case real-world performance and, thus, allows the robustness of ASR models to be comprehensively evaluated.

## 3 Speech Robust Bench

Figure: Figure 1: An illustration of the processes involved in using our benchmark to evaluate the robustness of ASR models.
Refer to caption: figures/eval-pipeline.png

At a high level, Speech Robust Bench (SRB) consists of four components:
(1) Clean datasets, and a curated selection of inherently noisy datasets containing limited and known sources of noises and variations that are difficult to simulate;
(2) Large bank of perturbations applicable on audio data;
(3) Transcription extraction from ASR models and analysis;
(4) Set of metrics for quantifying ASR model robustness, and facilitating comparisons among models.
The basic workflow of SRB is the following (see Fig. 1):
first, recordings from the clean dataset(s) (§ 3.1) are perturbed by a large bank of perturbations (§ 3.2), resulting in several perturbed versions of each recording. Next, these perturbed recordings, the original clean recordings, and the recordings with inherent noise are transcribed using the target ASR model. The predicted and reference transcripts are compared, and the accuracy and robustness of each model under each type of noise are captured with various metrics (§ 3.3). To account for the relative differences in the level of distortion introduced by different perturbations, we estimate speech quality scores using appropriate models and use them to normalize the metrics.

### 3.1 Dataset

We use clean speech from Librispeech [] and TEDLIUM []. Librispeech contains professional recordings of audio books, and thus, has properly enunciated and cleanly recorded speech.
Meanwhile, TEDLIUM contains professional recordings of TED talks and provides a degree of lexical and phonetic diversity that may be lacking in Librispeech. We chose these datasets due to their popularity in the ASR literature, however, any dataset containing utterances and reference transcripts would also be compatible with SRB. In addition to clean speech, SRB also uses inherently noisy speech containing specific types of perturbations that are hard to simulate, but are likely to be encountered by ASR models in the real-world. Specifically, SRB uses accented speech from Common Voice [], and inter-personal conversations and far-field recordings from CHiME-6 [] and AMI [].

### 3.2 Perturbations

The taxonomy of the perturbations used in our benchmark is presented in Fig. 2. The perturbations are of two broad types: (1) non-adversarial and (2) adversarial.
The non-adversarial perturbations fall under one of four categories:
(1.1) Inter-personal communication gathered in social settings using near-field and far-field microphones. This setting is representative of the environments in which voice-enabled smart devices are expected to operate. The data for this category are drawn from the ChiME [] and AMI [] corpora.
(1.2) Environmental effects, such as white noise, environmental noise, spatial acoustics (room impulse response and echo)
(1.3) Digital augmentations produced by standard audio processing operations, and special effects.
The audio processing operations included amplitude gain, resampling, lowpass, and highpass filtering. The special effects included bass gain, treble gain, tempo increase, tempo decrease, speed increase, speed decrease, pitch increase, pitch decrease, chorus, tremolo, and phaser.
All special effects are applied using SoX.
(^1^11https://sourceforge.net/projects/sox/)
(1.4) Speech Variations are represented by speech from non-American speakers, and speech generated by a TTS model []. Since users from a variety of ethnic backgrounds may interact with ASR models, they must be robust to a variety of accents. With the rise of generative AI, robustness to synthetic speech will also be important.

Figure: Figure 2: Taxonomy of perturbations currently used in SRB.
Refer to caption: figures/taxonomy.png

The adversarial perturbations we use are untargeted adversarial attacks whose objective is to prevent the correct transcription of input speech. We use two types of adversarial attacks: (2.1) utterance-specific and (2.2) utterance-agnostic attacks. The utterance-specific attack optimizes a perturbation for a given speech recording, such that a given model maximally mistranscribes. The utterance-agnostic attack works similarly but the perturbation is jointly optimized over many speech recordings.
Thus, this perturbation can affect several speech recordings, even those that were not used during the optimization. This represents a more realistic scenario where a malicious actor tries to mount a denial-of-service style attack against an ASR model by introducing utterance-agnostic perturbation at some point in the transcription pipeline.

Most perturbations, except for those categorized as informal speech, or speech variations, are applied at 4 levels of severity, which are modulated by adjusting the perturbation parameters. Details about the perturbations and their parameters are presented in Appendix A.

### 3.3 Metrics

SRB measures the utility of the model with the widely used Word Error Rate. WER is computed as the word-level edit distance between the reference and the predicted transcripts, normalized by the length of the reference (see Appendix B for formal definitions). To measure the robustness of the model to input perturbations, we use WER Degradation (WERD), computed as $WER(\mathcal{X}_{p})-WER(\mathcal{X})$, where $\mathcal{X}$ contains clean audio, while the audio in $\mathcal{X}_{p}$ is perturbed by perturbation $p$. If $p$ is synthetically applied, both $\mathcal{X}$ and $\mathcal{X}_{p}$ contain the same utterances, otherwise $\mathcal{X}$ contains clean audios from LibriSpeech and TEDLIUM.

When aggregating metrics (WER/WERD) over multiple perturbations, we follow the practice of [] and divide the metric by a measure of difficulty, i.e., by the (estimated) speech quality degradation.
This adds weight to errors on “easy” corruptions (less quality degradation) and under weighs errors on “harder” corruptions (more quality degradation) when computing averages. We refer to the difficulty normalized versions of WER/WERD as Normalized WER/WERD (NWER/NWERD).
We estimate speech quality using DNSMOS [] and PESQ[], which are models of human judgments of speech quality and predict Mean Opinion Scores (MOS) []. PESQ uses various signal processing methods to predict MOS, while DNSMOS uses DNNs to do the same. To compute speech quality degradation we compute PESQ and DNSMOS for each clean and noisy recording multiplied by -1 (lower values indicate less degradation).
Since we are only interested in the relative degradation between perturbations, we normalize the scores to have mean 50 and standard deviation 25.

## 4 Evaluation

We evaluate several recent and popular ASR DNNs (§4.1) using SRB and analyze the results to uncover fine-grained differences in their robustness to non-adversarial and adversarial input perturbations. We further extend our analysis by measuring ASR model robustness for various sub-groups, namely English speech and non-English (Spanish) speech, and male and female speakers. Prior works [, ] observe that there is a disparity in transcription accuracy between subgroups.
Our analysis augments these observations by showing that inter-group disparities in robustness may also exist, thus demonstrating the utility of SRB in the broader field of trustworthy AI.

### 4.1 Models

For English speech, we evaluate Whisper[] variants: large-v2 (wsp-lg), medium (wsp-md), small (wsp-sm), base (wsp-bs), and tiny (wsp-tn, wsp-tn.en), Wav2Vec-2.0 [] variants: base (w2v2-bs), large (w2v2-lg), self-trained large [] (w2v2-lg-slf), and Robust Wav2Vec [], HuBERT [] variants: large (hubt-lg) and XL (hubt-xl), Nvidia Canary [] (cnry-1b), MMS [] (mms-1b), Speech-T5 [] (spch-t5) and DeepSpeech [] (ds).
For Spanish speech, we evaluate on the mono-lingual Wav2Vec base Spanish [] (w2v2-bs-es), Wav2Vec XLSR Spanish [] (w2v2-lg-es), wsp-lg, wsp-bs, wsp-tn, and mms-1b. We used the Huggingface implementations for all the models except cnry-1b ([https://huggingface.co/nvidia/canary-1b](https://huggingface.co/nvidia/canary-1b)) and ds ([https://github.com/SeanNaren/deepspeech.pytorch](https://github.com/SeanNaren/deepspeech.pytorch)). More details about the models are in Table 3 in the Appendix.

Figure: (a)
Refer to caption: figures/avg_wer_ls_td.png

### 4.2 Accuracy on Clean Data

Figures 3(a) and 3(d) show the WER of the models on clean English and Spanish, respectively. We see that cnry-1b is the most accurate model on both languages, followed by w2v2-lg-slf on English and wsp-lg on Spanish.

### 4.3 Robustness of English ASR Models to Non-Adversarial Perturbations

Fig. 3(b) presents the NWERD for the English ASR models under non-adversarial noises and perturbations.
We observe that wsp-lg emerges as the most robust model, followed by wsp-md and cnry-1b. Despite having the lowest overall WERD, wsp-lg is outperformed by wsp-md and cnry-1b on white noise, environmental noise, and special effects (see Appendix D for more detailed results). The massively multilingual mms-1b has the highest NWERD in social scenarios and far-field recordings category, indicating that it is unsuited for applications like voice assistants and smart home set-ups. We also notice that Wav2Vec-2.0 based models also struggle with the “speech variations” indicating that they are not suitable when users have diverse accents.
Surprisingly, spch-t5 is the least robust model, and performs even worse than the much smaller and older ds. Similarly, wsp-md outperforms the newer and larger cnry-1b on almost all corruption categories.

Figure: Figure 4: WERD of cnry-1b, wsp-md and wsp-sm on perturbations in the spatial acoustics and special effects categories.
Refer to caption: figures/cnry_spatial_special.png

Upon closer examination, we see that cnry-1b has higher NWERD than its peers (wsp-md and wsp-sm) for two categories of perturbations: spatial acoustics and special effects. Upon looking at the WERD on each perturbation within these categories (Fig. 4) we find that cnry-1b is more sensitive than its peers to echo, real room impulse responses, and speed and pitch modifications. This analysis demonstrates that SRB can evaluate the robustness of ASR models at multiple ganularities and can pinpoint the weaknesses of a given model. Detailed results can be found in Figs. 7 and 9 in the appendix.

Takeaway: cnry-1b is the most accurate, but not most robust; Wav2Vec variants struggle against speech variations and social settings, thus, may not be suitable when users have diverse accents.

### 4.4 Robustness of English ASR Models to Adversarial Perturbations

Fig. 3(c) shows the WERD of the models on utterance specific and utterance agnostic adversarial attacks. We observe that hubt-lg and hubt-xl emerge as the most robust models to utterance-specific adversarial attacks, followed by other models based on Wav2Vec-2.0 large, Whisper, and Canary.
Interestingly, Wav2Vec models that exhibited mediocre robustness to non-adversarial perturbations are robust to utterance-specific attacks, while Whisper models and Canary, which were the most robust on non-adversarial perturbations, struggle on utterance specific attacks. We also note that most Wav2Vec models are considerably more robust to attacks against TEDLIUM than against LibriSpeech, and the opposite is true for whisper and Canary models.
Under utterance-agnostic attacks, the best-performing models are mms-1b, wsp-bs, and wsp-sm. It is interesting to note that the smaller variants of Whisper limit the generalizability across utterances of the adversarial perturbations to a greater extent than their larger counterparts.

Takeaway: Models that are most robust to non-adversarial perturbations, are mediocre against adversarial perturbations; cnry-1b is highly vulnerable to utterance specific attacks.

### 4.5 Robustness vs. Model Attributes

To determine if the prevailing practice of training DNNs with more parameters on larger datasets is yielding improvements in robustness, we use robust linear regression to fit a line to WERD vs. number of model parameters/size of the training data for the candidate models in Figs. 5(a) and 5(b), respectively. Increasing model size is correlated with improved robustness (lower WERD).To further isolate the impact of the model size we plot the NWERD of models from the same family in Fig. 5(c), which have similar architectures and training datasets. We note that larger models are more robust in the Whisper and Wav2Vec-2.0 families, but, surprisingly, not in the HuBert family. In contrast, increasing training data appears to have only a minor influence on robustness (Fig. 5(b)).

Takeaway: Larger models tend to be more robust, while smaller models, even if they are trained on large datasets, are less robust. This runs somewhat counter to the prevailing wisdom [, ].

Figure: (a)
Refer to caption: figures/model_params.png

### 4.6 Robustness for Population Sub-Groups

In the preceding analysis, we considered robustness aggregated over the entire population (i.e., dataset). However, populations are generally not homogeneous, and, thus, the robustness of the model may differ on various population sub-groups. Prior works have commonly analyzed sub-group fairness of ASR models by comparing the overall WER for each sub-group on a benchmark dataset []. It is possible that models that are fair on average, may not be fair under certain conditions.
In the following, we use SRB to uncover and analyze the disparities in the models’ robustness across four sub-groups: English and Spanish speech, and male and female speakers. We find that disparities indeed exist, with multi-lingual models generally being more robust for English than Spanish (Fig. 5(d)), and most models being less robust for females than males.

#### 4.6.1 Robustness on Non-English (Spanish) Speech

The clean accuracy of the Spanish ASR models is presented in Fig. 3(d), and the NWERD under non-adversarial perturbations for each model is shown in Fig. 3(e). We observe, that the two most accurate models, cnry-1b and wsp-lg, are also the two most robust models, but mms-1b, which had fairly high WER, is the third most robust model in terms of NWER. We also notice a general trend that all the models, except cnry-1b, struggle against “spatial acoustics” category, which is rather worrying since the influence of spatial acoustics is ubiquitous in the real-world.

Considering adversarial attacks (Fig. 3(f)), we observe that Wav2Vec models exhibit better robustness than Whisper and Canary under both utterance-agnostic and utterance-specific perturbations. This is similar to the trends observed in English speech (Fig. 3(c)), and indicates that perhaps learned front-ends may be more adversarial robust than the spectrograms used by Whisper and Canary. Detailed results can be found in Fig. 8 in the appendix.

We also compare the robustness exhibited by multi-lingual models, wsp-lg, wsp-bs, cnry-1b and mms-1b on English and Spanish. The WERD of these models on both languages is presented in Fig. 5(d). We observe that Whisper models achieve lower WERD on English speech than on Spanish on almost all perturbation categories, while cnry-1b and mms-1b achieve similar WERD on some categories. We also note that the difference in WERD on some common perturbation categories, like environmental noise, and spatial acoustics, is much greater for wsp-lg than for cnry-1b.

Takeaway: Multilingual models are more robust on English than Spanish; cnry-1b and wsp-lg most robust on both languages; adversarial robustness results follow the same trend as English.

#### 4.6.2 Robustness and Gender

To measure the disparity in prediction accuracy across genders (males/females), we compute the log of the ratio of the WERs of the ASR model on female and male speakers. We call this measure the Log WER Ratio (LWERR). A positive value of LWERR indicates that the model is biased against females and a negative value indicates that the model is biased against males.

The LWERR for each dataset is shown in Fig. 6. We note that, on average, the models are biased against females on LibriSpeech and Spanish Multilingual Librispeech (MLS-ES), and against males on TEDLIUM, Common Voice and AMI. The bias is most prominent in MLS-ES, where cnry-1b seems to be yielding the highest disparities among genders. We also note that adversarial perturbations cause the WER of wsp-lg to increase significantly more for females than males in LibriSpeech. This is interesting because adversarial perturbations do not target a specific part of the spectrum and thus should not impact one gender more than the other.

Takeaway: Models are more robust for males on some datasets, and females on other datasets suggesting that used data require further examination; adversarial attacks increase WER of Whisper variants for females more than males; multilingual models, particularly cnry-1b, are more biased against females when transcribing Spanish.

Figure: Figure 6: Log WER Ratio for various TEDLIUM, LibriSpeech, Common Voice (CV) and AMI, and Spanish Multilingual Librispeech (MLS (ES)). WERs are averaged across severity levels and individual augmentations within each category before computing the Log WER Ratio.
Refer to caption: figures/gender_fairness_all.png

## 5 Limitations

Despite the comprehensive design of SRB, there are limitations to consider. Firstly, while SRB includes a diverse set of datasets, distortions and adversarial attacks, it may not encompass all possible real-world scenarios. Additionally, the benchmark has been tested for now on English and Spanish languages, which may not capture the robustness challenges faced by ASR models in other languages and dialects.
While these limitations might potentially affect the generalizability of the results obtained with SRB, its extensibility facilitates the incorporation of novel approaches for quantifying ASR robustness.

## 6 Conclusion

We propose SRB, a comprehensive benchmark designed to standardize the robustness evaluations of Automatic Speech Recognition (ASR) models. To the best of our knowledge, SRB is the first ASR robustness benchmark that contains long-form speech, and expands the set of distortions to include special effects, speaker attribute modification and adversarial attacks.
We demonstrate the utility of SRB in evaluating the robustness of ASR models via several concrete examples, as well as its potential to facilitate evaluations of other aspects of trustworthy AI, like fairness.
We believe that SRB will enable rigorous robustness evaluations of ASR models in a highly standardized manner, allowing easy comparisons between existing and new approaches. To further facilitate robustness evaluations for researchers and model developers, we release transformed test sets in English and Spanish. We anticipate that this will make robustness evaluations more prevalent and encourage model developers to consider robustness as a key metric for improvement.

## Appendix A Perturbation Generation/Application Procedure

Below, we provide further details the perturbations that make up Speech Robust Bench. Table 1 shows the parameters for each perturbation and Table 2 shows the normalized DNSMOS and PESQ scores for each perturbation.

**Table 1: The parameters defining the various severity levels of the perturbations used in the proposed benchmark.**
| Category | Perturbation | Sev 1 | Sev 2 | Sev 3 | Sev 4 |
| --- | --- | --- | --- | --- | --- |
|  | Gaussian Noise | 30 dB | 20 dB | 10 dB | 0 dB |
| Environment | Environmental Noise | 30 dB | 20 dB | 10 dB | 0 dB |
| Music | 30 dB | 20 dB | 10 dB | 0 dB |  |
| Crosstalk | 30 dB | 20 dB | 10 dB | 0 dB |  |
| Spatial Acoustics | RIR | 0.27s | 0.58s | 0.99s | 1.33s |
| Real RIR | 9.1 | 7.1 | 4.1 | 1.8 |  |
| Echo (delay) | 125 ms | 250 ms | 500 ms | 1000 ms |  |
| Special Effects | Bass (gain) | 20 | 30 | 40 | 50 |
| Treble (gain) | 10 | 23 | 36 | 50 |  |
| Phaser (decay) | 0.3 s | 0.5 s | 0.7 s | 0.9 s |  |
| tempo-up | 1.25x | 1.5x | 1.75x | 2x |  |
| tempo-down | 0.875x | 0.75 | 0.625x | 0.5x |  |
| Speed-up | 1.25x | 1.5x | 1.75x | 2x |  |
| Slow-down | 0.875x | 0.75 | 0.625x | 0.5x |  |
| Pitch Step-up | 0.25 oct | 0.5 oct | 0.75 oct | 1 oct |  |
| Pitch Step-down | 0.25 oct | 0.5 oct | 0.75 oct | 1 oct |  |
| Chorus (delay) | 30 | 50 | 70 | 90 |  |
| tremolo (depth) | 50 | 66 | 83 | 100 |  |
| Audio Processing | Resampling | 0.75x | 0.5x | 0.25x | 0.125x |
| Gain (factor) | 10x | 20x | 30x | 40x |  |
| Low-pass filter | 4 kHz | 2833 kHz | 1666 kHz | 500 kHz |  |
| High-pass filter | 500 kHz | 1333 kHz | 2166 kHz | 3000 kHz |  |
| Adversarial | PGD Attack | 40 dB | 30 dB | 20 dB | 10dB |
| Utterance Agnostic Attack | 40 dB | 30 dB | 20 dB | 10dB |  |

**Table 2: Normalized DNSMOS and PESQ score for each perturbation.**
| Metric $\rightarrow$ | AVG | normalized DNSMOS | normalized PESQ |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Perturbation / Severity | 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 | 1 | 2 | 3 | 4 |
| accent | 31.6 | - | - | - | 31.6 | - | - | - | - | - | - | - |
| bass | 18.3 | 23.0 | 35.0 | 55.2 | 22.5 | 27.4 | 38.0 | 56.7 | 11.6 | 14.5 | 30.5 | 54.6 |
| chorus | 39.1 | 48.2 | 54.4 | 55.9 | 29.8 | 39.7 | 47.1 | 48.2 | 63.7 | 71.1 | 74.2 | 75.7 |
| crosstalk | 22.3 | 38.2 | 52.3 | 59.1 | 23.3 | 31.6 | 36.7 | 39.4 | 22.4 | 46.2 | 70.0 | 81.4 |
| echo | 54.1 | 53.4 | 52.8 | 50.6 | 39.0 | 36.3 | 36.0 | 34.9 | 71.2 | 72.6 | 71.7 | 67.9 |
| env noise | 50.5 | 61.5 | 76.0 | 88.5 | 51.8 | 59.2 | 72.7 | 91.4 | 39.7 | 58.7 | 76.6 | 84.1 |
| env noise esc50 | 26.1 | 40.9 | 57.5 | 72.8 | 37.7 | 43.7 | 53.3 | 71.3 | 20.3 | 43.2 | 65.9 | 79.7 |
| env noise musan | 24.3 | 42.3 | 62.1 | 75.4 | 24.8 | 37.2 | 55.3 | 71.2 | 25.0 | 48.7 | 70.3 | 80.9 |
| env noise wham | 22.4 | 45.4 | 73.2 | 92.2 | 22.3 | 39.7 | 70.5 | 99.5 | 23.3 | 52.0 | 76.5 | 85.2 |
| gain | 50.0 | 68.9 | 76.6 | 80.7 | 45.1 | 65.9 | 76.4 | 82.6 | 61.3 | 75.9 | 79.8 | 81.6 |
| gnoise | 52.4 | 75.6 | 90.5 | 100.9 | 70.3 | 87.5 | 101.6 | 117.3 | 42.2 | 69.1 | 82.8 | 86.1 |
| highpass | 40.2 | 55.4 | 67.5 | 77.5 | 34.3 | 43.4 | 64.9 | 81.0 | 46.2 | 68.3 | 71.5 | 74.8 |
| itw ff | 100.1 | - | - | - | 100.1 | - | - | - | - | - | - | - |
| itw ff ami | 83.9 | - | - | - | 83.9 | - | - | - | - | - | - | - |
| itw nf | 78.1 | - | - | - | 78.1 | - | - | - | - | - | - | - |
| itw nf ami | 35.8 | - | - | - | 35.8 | - | - | - | - | - | - | - |
| lowpass | 33.1 | 37.1 | 50.8 | 78.0 | 47.3 | 46.7 | 62.5 | 97.9 | 20.5 | 29.3 | 40.5 | 58.4 |
| music | 22.3 | 43.1 | 65.8 | 78.9 | 25.5 | 41.4 | 62.2 | 76.7 | 20.2 | 45.4 | 70.0 | 81.7 |
| phaser | 15.0 | 32.3 | 59.8 | 79.5 | 21.4 | 34.7 | 58.1 | 76.7 | 10.9 | 32.0 | 63.5 | 83.4 |
| pitch down | 60.9 | 67.3 | 53.4 | 83.3 | 38.0 | 49.2 | 65.5 | 80.8 | 85.7 | 86.3 | 55.5 | 86.6 |
| pitch up | 58.0 | 61.2 | 64.3 | 65.1 | 32.2 | 38.1 | 44.1 | 45.9 | 85.7 | 86.2 | 86.3 | 86.3 |
| real rir | 38.7 | 53.8 | 68.9 | 84.2 | 34.2 | 44.8 | 59.9 | 87.1 | 43.2 | 62.7 | 77.9 | 81.3 |
| resample | 14.4 | 27.3 | 49.0 | 63.3 | 23.8 | 42.1 | 61.5 | 75.5 | 7.0 | 18.3 | 38.2 | 52.5 |
| rir | 50.3 | 63.1 | 68.3 | 68.0 | 40.4 | 56.3 | 64.3 | 64.7 | 64.9 | 74.5 | 78.0 | 78.0 |
| slowdown | 50.7 | 57.0 | 64.3 | 72.7 | 17.6 | 29.7 | 43.6 | 66.7 | 85.7 | 85.9 | 85.8 | 86.2 |
| speedup | 51.5 | 58.7 | 66.3 | 72.9 | 21.7 | 36.1 | 50.9 | 63.7 | 83.6 | 83.4 | 83.1 | 82.8 |
| tempo down | 48.7 | 51.8 | 54.5 | 50.1 | 18.4 | 21.6 | 26.7 | 35.1 | 81.3 | 84.5 | 85.1 | 85.5 |
| tempo up | 50.2 | 57.1 | 63.2 | 69.7 | 24.3 | 34.6 | 46.1 | 58.3 | 78.9 | 82.1 | 82.3 | 82.3 |
| treble | 11.6 | 21.6 | 40.5 | 62.4 | 19.4 | 29.7 | 43.2 | 60.6 | 2.4 | 12.4 | 42.3 | 72.4 |
| tremolo | 16.8 | 29.0 | 59.0 | 98.7 | 23.1 | 37.2 | 72.1 | 111.4 | 9.9 | 17.8 | 37.1 | 75.5 |
| universal adv | 10.3 | - | - | - | 11.7 | - | - | - | 8.8 | - | - | - |
| voice conversion vctk | 49.6 | - | - | - | 15.7 | - | - | - | 83.4 | - | - | - |

##### Gaussian Noise:

A noise vector of the same length as the audio signal is sample from a standard normal distribution, scaled such that its magnitude corresponds to a specific SNR, and then added to the audio signal.
We use torchaudio.function.add_noise to add the noise to the speech at a given SNR.

##### Environmental Noise:

We use the recordings of environmental noises from the test/eval subsets of ESC-50  [], MS-SNSD [], MUSAN [] and WHAM []. We create a separate perturbed version of the clean data using each of these noise datasets. To do so, for each test utterance we sample a random environmental noise and add it to the audio signal at the specified SNR. We clip the noise if it is longer than the speech, and repeat it if it is shorter than the speech. We use torchaudio.function.add_noise to add the noise to the speech at a given SNR.

##### Room Impulse Response:

The simulated and real RIRs from [] are applied to clean recordings. As a measure of intensity, RT60 is estimated for the simulated RIRs using Sabine’s formula with the room dimensions and absorption coefficient provided in the dataset. For the real RIRs, we compute the SRMR [] using the implementation from [https://github.com/aliutkus/speechmetrics/tree/master](https://github.com/aliutkus/speechmetrics/tree/master). The severity is defined in increasing RT60s for the synthetic RIRs, and decreasing SRMR for the real RIRs.
Table 1 shows the average RT60/SRMS in each severity level. During evaluation, a random RIR having the given severity level is sampled for each test recording.

##### Resampling, Speed, Pitch, and Gain Perturbations:

The resampling speed, pitch, and gain perturbations were applied using the Resample Speed, PitchShift and Vol transforms from torchaudio.

##### Other special effects:

These effects are applied via SoX filters of the same name. We used torchaudio.sox_effects.apply_effects_tensor to apply these filters to the audio. The args for each filter are as follows:

- •
echo 0.8 0.9 <delay> 0.3
- •
phaser 0.6 0.8 3 <decay> 2 "-t"
- •
Tempo <factor> 30
- •
sinc <lo-freq>
- •
sinc 0-<hi-freq>
- •
tremolo 20 <depth>
- •
treble <gain>
- •
bass <gain>
- •
chorus 0.9 0.9 <delay> 0.4 0.25 2 -t {<delay>+10} 0.3 0.4 2 -s

##### Voice Conversion

We use use YourTTS [] from Coqui.ai(^2^22[https://github.com/coqui-ai/TTS](https://github.com/coqui-ai/TTS)) to synthesize audio from textual transcripts in a given speaker’s style. The transcripts from the test clean subset of LibriSpeech are used. The target speakers are drawn from the VCTK corpus [], which contains accented speech from 12 accents. For each transcript a random speaker is chosen to synthesize the audio.

##### Untargeted Adversarial Attack:

The adversarial perturbations are computed using an untargeted PGD adversarial attack. The attack is computed as follows. First, the maximum possible L2 norm of the noise is determined by solving the equation for SNR for the norm of the noise.

$$ SNR $\displaystyle=20\log_{10}\left(\frac{||x||_{2}}{||\delta||_{2}}\right)$ (1) $\displaystyle\epsilon_{\text{SNR}}$ $\displaystyle=||\delta||_{2}=10^{-\frac{\text{SNR}}{20}}||x||_{2},$ (2) $$

where $\delta$ is the noise, $x$ is the audio signal and SNR is the upper bound on the SNR in the final signal. Then, we follow the approach of [] and optimize $\delta$ using Projected Gradient Descent (PGD) to maximize the divergence between the true and predicted transcriptions.

### A.1 Crosstalk and Music

We use crosstalk and music audios from MUSAN []. We use torchaudio.function.add_noise to add the noise to the speech at a given SNR.

### A.2 Accents

We select a subset of audios from the test set of Common Voice 17. The selected audios satisfied the following criteria: (1) the speaker’s accent must be present in the metadata, (2) the accent must not be American, (3) should be clean. The last criterion is satisfied if the DNSMOS [] score of the recording is at least 3.4. The resulting subset contains 640 recordings. The most popular accent in this set is South Asian (India, Pakistan, Sri Lanka) (2̃5%), followed by British English (2̃5%).

### A.3 Inter-Personal Communications

We CHiME-6 [] and AMI [] to obtain recordings of people in social scenarios (dinner party and meetings). In both these datasets, the speakers are recorded through lapel microphone and a room microphone resulting in near and far field recordings. We use both types of recordings and show separate results for them. We remove recordings that contain less than three words since they are often fillers.

##### Untargeted Utterance Agnostic Adversarial Attack:

We use the method of [] to compute utterance agnostic adversarial perturbations. Concretely, we compute a single perturbation vector that can cause the accuracy of the model on several (ideally all) inputs to degrade severely. The main difference between the universal attack and the PGD attack is that the latter computes a perturbation vector for each input, whereas the former computes a single perturbation that is expected to successfully attack any input to the model. Once we compute the perturbation we add it to the audio signal at the specified SNR.

We use the robust_speech package [] to generate adversarial perturbations.

$$ $\displaystyle\delta$ $\displaystyle=\max_{\hat{\delta}:||\hat{\delta}||_{2}\leq\epsilon_{\text{SNR}}}D(h_{\hat{\delta}},r)$ (3) $$

## Appendix B Additional definitions

##### Word Error Rate:

As noted in the main text, we use word error rate (WER) as a basic measure for quantifying performance of the models.
Following the common practice from ASR literature, the WER is computed as the word-level edit distance between the reference and the predicted transcripts, normalized by the length of the reference. The edit distance is computed as the total number of word substitutions, deletions, and additions required to transform the reference transcript into the predicted transcript. We remove all punctuation from both the reference and predicted transcripts, and convert to lower case before computing WER.

When WER is computed over multiple pairs of predicted and reference transcripts, it is common practice to sum the number of substitutions, deletions, and additions for all the pairs, and divide by the sum of the lengths of the reference transcripts. Formally, this can be written as:

$$ $\displaystyle WER^{M}(\mathcal{X},\mathcal{R}):=100\frac{\sum_{x\in\mathcal{X},r\in\mathcal{R}}ED(M(x),r)}{\sum_{r\in\mathcal{R}}|r|}\%,$ (4) $$

where $ED$ computes the edit distance.

For quantifying differences between (binary) genders,
we measure the disparity in prediction accuracy across males and females by the Log WER Ratio (LWERR). Formally,

$$ $LWERR:=\log_{2}\frac{WER^{M}(\mathcal{X}_{f},\mathcal{R}_{f})}{WER^{M}(\mathcal{X}_{m},\mathcal{R}_{m})},$ (5) $$

where the $(\mathcal{X}_{f},\mathcal{R}_{f})$ and $(\mathcal{X}_{m},\mathcal{R}_{m})$ represent the subsets of utterances by females and males respectively.

##### Fairness through robustness:

In this work, we use SRB to conduct a fairness assessment based on robustness disparities across population subgroups (English vs. Spanish speech; male vs. female speakers). Most of the state of the art quantifies fairness in terms of predictive performance disparities. This occurs in the domain of fairness in ASR systems [, , ] and similar prevalence is found in other domains [, ]. However, previous work in the domain of ASR also considered robustness disparities as an alternative fairness notion [].

Moreover, we argue that considering the dimension of robustness could give better sense of the expected disparities that could be observed when deployed in the wild, in the presence of diverse noisy conditions.

## Appendix C Models

Table 3 provides a summary of the models used in our evaluations. The model names correspond to the names of their pretrained checkpoints in the Huggingface library ([https://huggingface.co/models](https://huggingface.co/models)). The abbreviations of these names are in the parentheses after them. Some of the unilingual models are pre-trained on multilingual data but are fine-tuned on only one language and thus can not transcribe any other language. Multilingual models have been pre-trained and fine-tuned on multiple languages so the same DNN can transcribe several languages. The WER of multilingual models is presented as English/Spanish.

**Table 3: Models used in our evaluations.**
|  |  |  | # Params (M) | Data (hrs) | WER |
| --- | --- | --- | --- | --- | --- |
| Language | Model | model (abrv) |  |  |  |
| EN | canary-1b [] | cnry-1b | 1,000.0 | 85,000 | 6.0 |
| deepspeech[] | ds | 86.0 | 960 | 26.5 |  |
| hubert-large-ls960-ft[] | hubt-lg | 317.0 | 60,000 | 8.4 |  |
| hubert-xlarge-ls960-ft[] | hubt-xl | 964.0 | 60,000 | 8.4 |  |
| mms-1b-fl102 [] | mms-1b | 964.6 | 55,000 | 22.8 |  |
| speecht5 asr[] | spch-t5 | 154.6 | 960 | 22.1 |  |
| wav2vec2-base-960h [] | w2v2-bs | 95.0 | 960 | 11.3 |  |
| wav2vec2-large-960h [] | w2v2-lg | 317.0 | 960 | 9.7 |  |
| wav2vec2-large-960h-lv60-self [] | w2v2-lg-slf | 317.0 | 60,000 | 7.7 |  |
| wav2vec2-large-robust-ft-libri-960h [] | w2v2-lg-rob | 317.0 | 63,000 | 8.9 |  |
| whisper-base [] | wsp-bs | 74.0 | 680,000 | 9.6 |  |
| whisper-base.en [] | wsp-bs.en | 74.0 | 563,000 | 5.1 |  |
| whisper-large-v2 [] | wsp-lg | 1,550.0 | 680,000 | 8.0 |  |
| whisper-medium [] | wsp-md | 769.0 | 680,000 | 7.9 |  |
| whisper-medium.en [] | wsp-md.en | 769.0 | 563,000 | 4.1 |  |
| whisper-small [] | wsp-sm | 244.0 | 680,000 | 8.3 |  |
| whisper-small.en [] | wsp-sm.en | 244.0 | 563,000 | 4.0 |  |
| whisper-tiny [] | wsp-tn | 39.0 | 680,000 | 11.3 |  |
| whisper-tiny.en [] | wsp-tn.en | 39.0 | 563,000 | 10.1 |  |
| ES | canary-1b [] | cnry-1b | 1,000.0 | 85,000 | 3.2 |
| mms-1b-fl102 [] | mms-1b | 964.6 | 55,000 | 15.7 |  |
| wav2vec2-base-10k-voxpopuli-ft-es [] | w2v2-bs-es | 94.4 | 10,116 | 25.7 |  |
| wav2vec2-large-xlsr-53-spanish [] | w2v2-lg-es | 315.4 | 54,350 | 6.8 |  |
| whisper-base [] | wsp-bs | 74.0 | 680,000 | 14.8 |  |
| whisper-large-v2 [] | wsp-lg | 1,550.0 | 680,000 | 5.8 |  |
| whisper-tiny [] | wsp-tn | 39.0 | 680,000 | 23.3 |  |

## Appendix D Fine Grained Analyses

The following figures present fine-grained analyses of robustness. These figures may be referenced by the main text but were not included in the main body in the interest of space. Figures 7, and 8 present the breakdown by perturbation of the robustness of models on English and Spanish, respectively. Figure 9 presents a breakdown of robustness by severity.

Figure: Figure 7: NWERD of English models on different augmentations, averaged over all severities.
Refer to caption: figures/nwerd_all_augs.png

Figure: Figure 8: NWERD of Spanish models on different augmentations, averaged over all severities.
Refer to caption: figures/nwerd_all_augs_es.png

Figure: Figure 9: NWERD on English data as the severity of the augmentation is increased.
Refer to caption: figures/sev_nwerd.png

Figure: (a)
Refer to caption: figures/lwerr_en.png

## Appendix E Compute Resources

The experiments were performed on the Bridges-2 cluster at the Pittsburgh Supercomputing Center. This cluster contains 200 32G and 16G Nvidia V-100, which were used for these experiments.

## Appendix F Dataset Licenses

The licenses of each of the considered datasets are described in Table 4.

**Table 4: Licenses of each of the considered datasets in SRB**
| Dataset | License |
| --- | --- |
| LibriSpeech | CC-BY-4.0 |
| Multilingual Librispeech | CC BY 4.0 |
| TEDLIUM | CC-BY-NC-ND 3.0 |
| AMI | CC-BY-4.0 |
| Common Voice | CC0-1.0 |
| CHiME | CC BY-SA 4.0 |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: 1. Can the authors clarify, for each domain, what dataset(s) are used, specify their statistic (min/max/avg length, male/female, total number of utterances), exactly how a model is evaluated (WER, WERD, NWERD with speech quality weights from...) , and how the data is perturbed, if applicable?
2. Can the authors clarify whether Spanish data is included in the benchmark, or whether this is a separate analysis?
3. Can the authors clarify how weights for the DeepSpeech were obtained?
soundness: 3
strengths: ### originality 

This work is combines multiple existing datasets to build a comprehensive analysis tool for ASR models. The authors place their work fairly into existing literature.

### quality 

The writing is mostly error-free and clear to understand. The authors analyzed relevant, contemporary ASR models. Nice-to-have but not required models to include could've been WavLM and, perhaps, OWSM.        

### clarity 

The authors use a familiar section lay-out for their paper. I also appreciate the take-away summaries in Section 4.    

### significance

I think this work can contribute to a better evaluation of ASR models. It is significant to the speech community to make it is easy for researchers to evaluate on data other than Librispeech test-clean and test-other.
summary: The authors propose a comprehensive benchmark for automatic speech recognition (ASR) models. The benchmark aims to cover a large variety of real-world data-quality challenges for deployed ASR models.

To benchmark on clean speech, the authors use the Librispeech and TEDLIUM datasets.  
To benchmark on inter-personal speech, the authors use the CHiME-6 dataset.  
To benchmark on far-field speech, the authors use the AMI dataset.  
To benchmark on accented English, the authors use the Mozilla CommonVoice dataset.  
To benchmark on text-to-speech audio, the authors generate data using the model XTTS.  
To benchmark against adversarial attacks, the authors use utterance-specific and utterance-agnostic attacks.  
To benchmark against environmental effects, the authors add 1) white noise, 2) environmental noise, or 3) simulate echo and RIR.  
To benchmark against digital augmentations, the authors modify the audio data using most effects available in SoX.

The authors use the word-error-rate (WER) as the metric for clean speech. The metric for adversarial attacks is WER degredation (WERD). For all others, the authors use difficulty-normalized WER or WERD. The normalization is done based on a weight, which is an approximated speech quality given by a neural network.    

The authors apply their benchmark on popular ASR models and their variants, including Whisper, wav2vec 2.0, HuBERT, and Canary.
The following observations are made:
- Canary has the best performance on clean speech
- Whisper is robust against digital augmentations, echo and RIR, while Canary is not.
- Wav2vec 2.0 is robust against adversarial attacks. 
- There is a positive correlation between average robustness and amount of model parameters

Furthermore, the authors did an analysis on Spanish datasets with models fine-tuned (single-or multi-lingually) on Spanish. They find that multi-lingual models are more robust on English than Spanish.

Finally, the authors analyzed performance differences between male and female speakers. They observe it is data dependent.
weaknesses: ### Benchmark protocol is unclear

Section 3 is missing important details for reproducing this work. I think practitioners cannot replicate the benchmark protocol by reading the paper as-is. It is critical for the usefulness of a benchmark that it can be independently reproduced. I have the following remarks:

1. I find the usage of the word 'Perturbations' unclear. I see how adversarial attacks, environmental effects, and digital augmentations can be classified as perturbations of the audio signal. However, I do not see how accented speech, computer generated speech, and inter-personal communication, can be seen as a perturbation. I would prefer the usage of "domain" over "perturbation". My suggestion would be to change the taxonomy (Figure 2) to include the domains clean/professional, social gathering, accented, text-to-speech, noise (with subgroups environment, digital augmentation), and adversarial.    
2.  It follows from 1 that is unclear to me exactly which dataset is used in which setting of the benchmark. I made a best guess in the summary above. For example, is environmental noise, or adversarial attacks, used only on Librispeech and TEDLIUM? Moreover, which split of Librispeech should be used? Only test-clean, or also test-other? Another example, which version of Mozilla CommonVoice is used? Do you filter out any data from CommonVoice if it is not labeled as 'accented'? I think the paper would be greatly improved by having a subsubsection for each domain/perturbation in Section 3.2, which list all details needed to generate the (exact) test set(s).  
3. I praise the authors for sharing an (omited) link to perturbed versions of the datasets. I think this benchmark should require to use these exact perturbed versions in the noise domain, otherwise comparisons are unfair. The paper currently does not touch upon this topic.
4. Section 3.2 does not include any information on the generation of adversarial perturbations. These details should be in the main text of the paper.
5. There is no information on which text is used to generate the text-to-speech audio. Moreover, it would be desirable for these generated audio files to be shared so everyone can test on the same data. 
6. Section 3.1 does not mention any Spanish datasets. Moreover, it is missing datasets for the noise domain (WHAM!, MUSAN, etc)
7. It is unclear to me whether this benchmark includes Spanish data, or whether this is simply an extra analysis from the authors. 
8. For reproducing the benchmark, it is required to have the speech perception weights of each utterance to calculate the NWER(D). 

### Confusing details in analysis

Section 4.1 mentions the use of DeepSpeech (which does not have open weights?) and speech-t5, these are not used in Table 1, only Figure 7. Moreover, Section 4.4 mentions the use of 4 more models, it would be more clear to include these in Section 4.1. 

It is unclear when WER or WERD should be used. From Table 1 I assume that you use NWERD for e.g., accent and text-to-speech data. I don't see how one can have an $X$ and $X_p$ audio sequence to calculate the WERD in these scenarios. 

### Figures and tables are difficult to read

I find Figure 4, 5 and 7 too small to read. As the page limit is 10, there should be enough room to increase the size without exceeding the page limit.  

### Editorial remarks

* I would advice the authors to use \citep instead of \cite. Currently, the reference style (other than in section 2.2) makes the text hard to parse. 
* Figure 3's space exceeds the bottom margin, and indents the texts on page 7 unnecessarily. 
* Table 1 and Table 2 have a different format. I would recommend the use of the booktabs package and no usage of vertical lines. 
* Appendix A is empty.
* Table 4 and Table 6 exceeds the page margin. 
* The citation of Canary (Elena Rastogueva) is incorrectly formatted, is missing data, and exceeds the page margin. 

#### Typos
- ln 41: is comphe.. -> are comphre...
- ln 107:  them -> those
- ln 161:  "others" does not fit here (Radford et al is in both)
- ln 346:  than -> compared to
- ln 472 (and others): the term accuracy is specific to classification, 'quality' is more appropriate to describe ASR predictions.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: See Weaknesses section
soundness: 3
strengths: * The experiments are comprehensive. It considers a couple state-of-the-art ASR systems, and it analyzes adversarial/non-adversarial perturbations.
* It analyzed the correlation between different models in terms of robustness.  
* It analyzed the robustness of speakers from different sub-groups (accent, gender).
summary: This paper proposes a new benchmark data and pipeline to measure the ASR robustness under different perturbations. The proposed work has 4 components: 1. clean datasets and noise sources; 2. bank of perturbations; 3.ASR transcription extractions; 4. metric computations. Compared to existing benchmark pipeline, the authors claimed that “the proposed pipeline combines several benchmark datasets containing known noise sources to allow researchers to comprehensively evaluate the robustness of their models using a single benchmark”. During experiments, the authors evaluated a couple state-of-the-art ASR systems on the proposed benchmark, and conduct a series of analysis on robustness.
weaknesses: Although the analyzes are very comprehensive, the contributions/novelty are not clearly stated. According to the quoted claim above, it feels like the most significant contribution compared to prior work is having more noise sources? Please explain more about the contributions/novelty here. It would be helpful to have a table to compare between the proposed approaches with baselines on the diffs from the 4 modules.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 2
questions: Listed in the weaknesses part
soundness: 2
strengths: 1. The paper conducts extensive and detailed analysis to evaluate the robustness of recent ASR models. The Takeaways help develop robust ASR models.
summary: This paper proposes SRB, a benchmark to evaluate the robustness of Automatic Speech Recognition (ASR) models. Extensive evaluations are conducted on several recent and popular ASR DNNs to analyze their robustness to non-adversarial and adversarial input perturbations,  and various sub-groups, namely English speech and non-English (Spanish) speech, and male and female speakers. From the above analysis, several conclusions are drawn.
weaknesses: 1. While I appreciate the efforts to evaluate the robustness of recent ASR models, the paper offers limited novelty. I suggest that the authors compare their benchmarks to existing robustness metrics or frameworks in the ASR field. 
2. To evaluate the robustness of ASR models, the current framework adds noise to clean speech data to evaluate the performance. How do recent ASR models perform in real-world noisy data? In other words, how can we evaluate the robustness of ASR models to real-world noisy data?
3. The paper lacks a discussion on strategies for improving robustness. It would be great if the authors could provide some sights or suggestions to improve the robustness.
2. The experimental sections of the paper need reorganization. I can not see a clear logic for the experimental parts.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: The reviewer does not have explicit further questions apart from above commetns.
soundness: 3
strengths: 1. The dataset description is very detailed and the related pipeline has been open-sourced, which is a decent contribution to the speech community.
2. The models covered in the field has been quite at the level of state-of-the-art.
3. The paper has answer to the critical question - "does the improvement come from additional variants from data or simply just MORE data?" in section 4.4, which is rare and good.
summary: This paper proposes Speech Robust Bench, a benchmark framework for testing speech recognition models . It collects almost all speech variation techniques available in the field, with close connection with real-time application scenarios. Results on
weaknesses: 1. The reviewers think the presentation of the paper is good, but has room for improvement. Some of the extensive statistics and information (e.g. about attacks) can be enlisted or illustrated in the main text, rather than extensive text description in both main text and appendix; Some resulting figures (e.g. Figure 5) are hard to read; Also there are minor formatting issues.
2. One limitation of this work is lacking comparison or having initial attempts with some hybrid audio processing pipeline, especially based on its growing popularity on speech processing fields. In fact, applying noise and perturbation has been a long-time practice since Kaldi era.
3. It would be good if the authors can correspond the related earlier works onto the methods enlisted and highlight the differences and potential variants the authors covered.
4. The authors highlighted in the paper about multi-lingual scenario but in fact, only English and Spanish are covered, and the models are mostly mono-lingual. The authors may either remove such statement or want to conduct additional experiments using other languages and multi-lingual models for ASR (e.g. Google T5). This is not major problem though.

Meta-review: This work introduces a new benchmark dataset and pipeline designed to evaluate ASR robustness under various scenarios. The proposed framework consists of four key components: (1) clean datasets and noise sources, (2) a bank of perturbations, (3) ASR transcription extraction, and (4) metric computation. The benchmark aims to cover a large variety of real-world data-quality challenges for deployed ASR models.

The key authors' claim is that "the proposed pipeline integrates multiple benchmark datasets with known noise sources, enabling researchers to comprehensively assess the robustness of their models using a single unified benchmark."

Key strengths of the work are: (i) The experiments are thorough, evaluating several state-of-the-art ASR systems and examining both adversarial and non-adversarial perturbations,  (ii) The study explores the correlation between different models in terms of robustness and investigates the robustness of speakers from various sub-groups, including accent and gender, and (iii)  the authors conducts extensive and detailed analysis to evaluate the robustness of recent ASR models. 

Key weakness is that multilingual scenario is not fully covered.

The proposed benchmark is different enough from what already available, and it gives researchers to comprehensively assess the robustness of their models using a single unified benchmark. The paper is well written.

Decision: Accept (Poster)