Speech Robust Bench: A Robustness Benchmark For Speech Recognition
Authors: Muhammad A Shah
David Solans Noguero
Mikko A. Heikkilä
Bhiksha Raj
Nicolas Kourtellis
Venue: ICLR 2025 Poster (2025)
Keywords: robustness
automatic speech recognition
benchmark
adversarial
Primary area: datasets and benchmarks

Abstract: As Automatic Speech Recognition (ASR) models become ever more pervasive, it is important to ensure that they make reliable predictions under corruptions present in the physical and digital world. We propose  Speech Robust Bench (SRB), a comprehensive benchmark for evaluating the robustness of ASR models to diverse corruptions. SRB is composed of 114 input perturbations which simulate an heterogeneous range of corruptions that ASR models may encounter when deployed in the wild. We use SRB to evaluate the robustness of several state-of-the-art ASR models and observe that model size and certain modeling choices such as the use of discrete representations, or self-training appear to be conducive to robustness. We extend this analysis to measure the robustness of ASR models on data from various demographic subgroups, namely English and Spanish speakers, and males and females. Our results revealed noticeable disparities in the model's robustness across subgroups. We believe that SRB will significantly facilitate future research towards robust ASR models, by making it easier to conduct comprehensive and comparable robustness evaluations.

Published as a conference paper at ICLR 2025
SPEECH ROBUST BENCH: A ROBUSTNESS BENCH- MARK FOR SPEECH RECOGNITION
Muhammad A. Shah David Solans Mikko A. Heikkili Bhiksha Raj' Nicolas Kourtellis
ABSTRACT
As Automatic Speech Recognition (ASR) models become ever more pervasive, is important to ensure that they make reliable predictions under corruptions present in the physical and digital world. We propose Speech Robust Bench (SRB), a comprehensive benchmark for evaluating the robustness of ASR models to diverse corruptions. SRB is composed of 114 challenging speech recognition scenarios which largely cover the range of corruptions that ASR models may encounter when deployed the wild. We use SRB to evaluate the robustness of several state-of-the- art ASR models and observe that model size and certain modeling choices such as the use of discrete representations, or self-training appear to conducive to robustness. We extend this analysis to measure the robustness of ASR models on data from various demographic subgroups, namely English and Spanish speakers, and males and females. Our results revealed noticeable disparities in the model's s robustness across subgroups. We believe that SRB will significantly facilitate future research towards robust ASR models, by making easier conduct comprehensive and comparable robustness evaluations.
INTRODUCTION
As novel ML models continue to be developed and deployed at an ever-increasing rate, it has become crucial to ensure their robustness to challenging real-world scenarios, where corruptions arising from a myriad of sources, including the environment, sensing apparatus, and even malicious actors are present. To this end, prior works have developed comprehensive robustness benchmarks, particularly for vision (Hendrycks Dietterich 2019: Hendrycks et al. 2021a b Croce et al.) 2020) and natural language processing models (Wang et al. 2021a| 2022b), that evaluate model's 's performance under variety of challenging scenarios. These benchmarks have proven to be invaluable to the advancement of research into more robust models because (1) they unify robustness evaluations, thus enabling meaningful comparisons across models and allowing progress to be accurately tracked, and (2) they make it easier for researchers to comprehensively evaluate the robustness of their models by aggregating a diverse and representative set of scenarios, and methods of simulating them, in a single benchmark.
While several robustness benchmark datasets exist for Automatic Speech Recognition (ASR) mod- els (Barker al. 2017 Kraaij et al., [2005] Wichemn al. 2019 Reddy al.) [020 Cosentino et al.) 2020
Hershey et al. 2016 Chen et al. 2020: Snyder et al. 2015; Kinoshita et al. 2013 Ko et al. 2017 Nakamura et al.) 2000 JJubb et al. 20090), none of the currently existing ones are in any sense
comprehensive, because each benchmark measures the model's robustness w.r.t. to one or a few specific types of corruptions or scenarios, which puts the onus on model developers to find and collect all the relevant benchmarks to evaluate their model comprehensively, This has often resulted in model developers evaluating their models on disparate benchmarks (Radford et al., 2023] Wen et al.) 2016))
Chen et al. 2022; Likhomanenko et al. 20200), which makes it hard to reliably compare performance
and robustness across models. Recently, Huggingface Open ASR Leaderboard (Srivastav et al.
2023] has sought unify ASR model evaluations by developing benchmark consisting of several real-world speech datasets. Although evaluating models on exclusively natural data may accurately reflect average case real-world performance, it is generally not informative about the specific types of corruptions the models are weak against, because the noise sources present in these datasets are not
* email: mshah1 1 @ @cmu.edu Camegie Mellon University. Telefonica Research, University of Helsinki; M. Shah did this work during internship at Telefonica Research; N. Kourtellis is now Keysight Technologies.

---

Published as a conference paper at ICLR 2025
controlled even fully known. For example, the crowdsourced recordings in Common Voice (Ardila jet al., 2019) contain a variety of distortions including sensor noise from low-quality equipment, background noise, and mispronunciation by non-native speakers. Furthermore, digital perturbations like special effects, immputer-generated speech, and adversarial examples, that may be prevalent in digital content are largely overlooked by existing benchmarks.
In this paper, we propose Speech Robust Bench (SRB), a benchmark for comprehensively
evaluating the robustness of ASR models to input perturbations and corruptions. SRB is designed to address the aforementioned major shortcomings of existing ASR robustness benchmarks, i.e., that (1) they are often specialized and thus are not individually comprehensive. (2) even taken together, they overlook important challenging scenarios, like special effects and adversarial attacks, and (3) may not reveal the specific weaknesses of the models. SRB addresses these shortcomings by evaluating ASR models under a comprehensive set of challenging scenarios, using recordings that are either are recorded under specific scenarios, and thus are inherently "noisy", or recordings that are digitally perturbed to simulate the various scenarios. SRB uses real recordings of accented speech and inter- personal conversations to evaluate robustness to articulatory and lexical variability. We take care to ensure that the recordings are clean and do not have any other corruption that may confound the results. To digitally simulate challenging scenarios, we curate a large comprehensive bank of 114 perturbations that represent common distortions arising from the environment, recording equipment, special effects, computer-generated speech, and adversarial attacks that are often overlooked by existing benchmarks.
To highlight the need for and the benefits of doing systematic and comprehensive robustness assess- ment, we evaluate the robustness of several popular ASR models ($4.1) using SRB. We observe that Whisper (Radford et al. 2023) is the most robust on average among the models we tested, even outperforming the more recent Canary (VVIDIA). We conduct further analyses disentangle the effects of model and training data size, revealing that larger models tend to be more robust than smaller models, even the latter are trained on significantly more data. We further extend our analysis by evaluating the models' robustness for the various population sub-groups, namely, English and non-English (Spanish) speakers, and male and female speakers. We find that significant disparities exist across these sub-groups, thus identifying issues with fairness of the models and highlighting areas where future work could provide clear improvements. Besides pinpointing robustness issues, this demonstrates the utility of SRB for fairness evaluations that consider robustness disparities across models as well.
To summarize make the following contributions:
We present SRP, comprehensive robustness benchmark for ASR models, enabling direct and casy comparison of robustness evaluations between models to facilitate progress. We demonstrate the use of SRB by conducting fine-graines robustness analysis for several popular models. We extend our analysis by using SRB to uncover disparities in the robust- ness of ASR for various sub-groups of speakers. This highlights the broad utility of this benchmark to the field of trustworthy AI.
robustness evaluations for the community, we have publicly re- leased large dataset Danaaaiiinnnnniinng containing perturbed versions of I LibriSpeech (Panayotov et al. (2015) test-clean, Spanish, and French and German test sets of Multilingual LibriSSpeech (Pratap) et al.] (2020), as well as accented speech from common voice, and segemented near- and far-field audios from CHiME-6 (Reddy et al. 2020) and (MI et al. 2005).
To facilitate out-of-the-bul
We release our code with clear documentation to enable reproducibility and extensibility.
RELATED WORK
2.1 ROBUST AUTOMATIC SPEECH RECOGNITION
Several techniques have been proposed for making Automatic Speech Recognition (ASR) models robust to input perturbations, such as noise and other signal corruptions (Li et al. 2014). We can divide these techniques into two high-level categories: i) model-based and ii) feature-based. Model-
code https: http:///thhhttcccom/mmeaaaaaaaa0 hoa_
data:
2

---

Published as a conference paper at ICLR 2025
based techniques modify the models to make them more robust. Examples of such approaches include adapting pre-traing vre-trained models Yu et al., 2009; Juang & Rahim 1996). de-noising the audio before processing [Mohammadiha et al. 2013 Wilson et al., 2008), and training ASR models on noisy data (Likhomanenko et al., 2020). Since model-based strategies generally require access to noisy data (Li et 20144, they most effective if the sources of noise, and/or the exact environment in which the ASR model will be deployed in are known, and one can gather data to represent those. Feature-based approaches, on the other hand, involve developing handcrafted features, invariant to noise and corruptions in the signal (Li et al. 2014). Several of these features are inspired by biological audition (Kim & Stem/ 2016 Hermansky et al.
1991; Hermansky & Sharma 1998). while others use signal processing techniques (Li et al.] 2014). Generally, these methods are designed to extract
the components of the audio signal salient for speech production and perception, while discarding irrelevant components (Stern & Morgan 2012). Consequently, they do not require precise knowledge of the environment and noise distributions. Recently, however, handcrafted features have fallen out of favor, and have been replaced by features learned via end-to-end training of deep learning models on large amounts of data (Baevski et al. [220 Hss et al. 2021a Likhomanenko et al. [0200] Radford et al. 2023). Proponents of these techniques posit that models trained on larger datasets become more robust. Our evaluations in & reveal that there are several input perturbations against which smaller models trained on less data outperform larger models trained on more data.
2.2 ADVERSARIAL ROBUSTNESS
Adversarial perturbations can change the response of a model when added to their inputs, but are either imperceptible humans perceptually and semantically irrelevant enough to be ignored by them (Szegedy et al. 2014] Goodfellow et al. 2014). Adversarially perturbed inputs are known as adversarial attacks. They can be targeted (aiming change a prediction to a specific incorrect class), or un-targeted (aiming to change a prediction to any incorrect class (Akhtar et al. 202I)). The design of adversarial attacks determined by the level of knowledge the attacker assumed have about the target model. Attacks that assume full knowledge of the target model's architecture and weights whxite-box threat model) often use gradient-basee based optimization techniques (Szegedy et al. 2014]
Goodfellow et al. 2014 Madry et al. 2018 Laidlaw et al. 2021: Akhtar et al. (021). Attackers who do not have any Enowledge of the target model's architecture and only have query access to it (black-box threat model) typically use gradient-free optimization methods (Wang et al. 2022a
Andriushchenko et al. [2220] Wicker et al.) 2018 Chen et al.) 2017 Zhao et al.) [2020] Vo et al. 2022). An intriguing property of adversarial perturbations is that they transfer between models (Papernot et al. 2016), and inputs (Akhtar et al. 2021, Neekhara et al. 2019). i.e. perturbations designed for one model/input may be effective against others as well. Our SRB includes two types of white box adversarial attacks; those that generate perturbations: 1) specific to each input (Madry et al. 2018).
2) that cause models to mis-transcribe multiple inputs (Neekhara et al. 2019).
2.3 ROBUSTNESS BENCHMARKS FOR SPEECH
While several robustness benchmark datasets exist for ASR models, they, unfortunately, suffer from three major shorteomings that make it difficult to perform robustness evaluations in way that is comprehensive, sufficiently fine-grained, and comparable across models.
Firstly, find that existing benchmarks do not comprehensively evaluate robustness. While past works such as (Pearce & Picone] (202)) did indeed propose benchmarks containing diverse perturba- tions, they were limited relatively simple data unsuitable for modern ASR models. However, many recently proposed benchmarks measure the robustness in one or a few specific types of scenarios. For example. some datasets (Kinoshita et al.) 2013 Nakamura et al. 2000) Jeub et al., 2009 Ko et al. 2017) focus exclusively on reverberant speech, while others focus only on mult--peakkr sce- narios Kraaij et al. |2055 Baarkrr et al. (017]) environmental noise (Snyder et al. 2015 Reddy et al. 2020; Wichern et 2019 Hirsch & Pearce, 2000), or accented speech (Lander 2022] Shi et al. 2021. Furthermore, certain types of digital perturbations, like special effects, computer-generated speech, and adversarial examples, which humans are usually invariant to (and thus ASR models are expected to be as well) are largely overlooked by existing benchmarks. SRB evaluates robustness under wide range of scenarios, including those represented in prior works, and novel ones that have not yet received due attention, thus enabling comprehensive robustness evaluations via a single benchmark.
3

---

Published as a conference paper at ICLR 2025
Secondly, since there are several robustness benchmarks that researchers can choose from, the onus is on model developers to collect all relevant benchmarks and comprehensively evaluate their model. This has often resulted in model developers evaluating their models disparate benchmarks
(Radford et al. 2023 Wen et al. 2016 Chen et al. 2022 Likhomanenko et al. 2020), which makes hard to reliably compare performance and robustness across models. For example, several works
have used subsets of existing benchmarks to evaluate the model robustness (Radford et al. [023
Wen et al. 2016 Chen et al. 2022), or have evaluated the robustness of models by computing their transcription accuracy on multiple natural speech datasets (Likhomanenko et al. 2020] Radford al.. 2023 Hsu et al., (021b). SRB covers wide range of challenging ASR scenarios in a single
benchmark, it obviates the need to select from various benchmarks, and, thus, can unify robusiness evaluations across studies and make them compurable.
Finally, robustness benchmarks that try to mitigate the above two shortcomings are often too coarse to reveal the specific scenarios and corruptions the model(s) struggle against. For example, Huggingface Open ASR Leaderboard (OAL, Srivastav al. 2023) evaluates models on several real-world speech datasets, which may accurately reflect their average-case real-world performance, it may not inform about the specific type of perturbations the models are weak against. This is because noise sources present in these datasets are not controlled or even fully known. For example, the crowdsourced recordings in Common Voice (Ardila et al. 2019) contain variety of distortions including sensor noise from low-quality equipment, background noise, and mispronunciation by non-native speakers. If model struggles with Common Voice, identifying the specific types of noise is not a straightforward task. In the design of SRE, we have ensured that the sources of noise (and variance, in general) in each utterance are limited and known, which allows users to pinpoint specific scenarios and/or perturbations under which their models struggle (e.g. Fig. andd $
sensitive to
SPEECH ROBUST BENCH
Scenarto Stmubriio
Trincriation
Metric Computation
Speech Robust Bench (SRB) evaluates the
robustness of ASR models by a three-step pro-
Calimater
cess consisting of (1) scenario simulation, (2) transcription, and (3) metrics computation, as shown in Fig. first, various challenging speech recognition scenarios are simulated by applying a large bank of synthetic perturbations to clean speech datasets ($ 3.I. as well as by using in- herently noisy speech datasets with limited and known sources of real noise and variations that are difficult to simulate. Next, the perturbed recordings, the original clean recordings, and the recordings with inherent noise are transcribed using the target ASR model. Finally, the predicted and reference transcripts are compared, and the accuracy and robustness of each model in each setting is captured with various metrics ($ 3.2). To account for the differences in the level of difficulty between scenarios, we also estimate speech quality scores using appropriate models and use them to calculate normalized metrics.
Inherently
WER
NIVER
Noby
Murid
Datcseeja
WERD
NIVERO
Modell Zoo
Speech Cuseity
Estimation Modiees
An illustration of the processes involved in using our benchmark to evaluate the robustness of ASR models.
Figure I:
3.1 SCENARIO SIMULATION
The various speech recognition scenarios sim- ulated by SRB are taxonomized in Fig. 2 and can be divided into six high-level categories, namely (1) clean speech, (2) social gatherings, (3) speech variations, (4) environmental effects, (5) digital augmentations. and (6) adversarial at- tacks. The scenarios are described briefly below, while more details are given in Appendix A
Ozan
Socaa
Speech Vetelooa
Envronmental Offiets
Digiaal Augmanmettom
Advoranial Afbeos
Spawch
cathecrg
heeelwe
Audio procouing
Ulteaace
Nche
Agyostic
gerented
Computer
Acousties
mansing prh))
Specik
Unsrance
Nemcs caos
While nake
Figure 2: Taxonomy of scenarios currently repre- sented in SRB. Scenarios in dashed boxes have real- world recordings, while scenarios in solid boxes are simulated by digitally adding perturbations.
(1) Clean speech:
SRB uses clean speech for two purposes: to benchmark the baseline accu- racy of ASR models, and to simulate various challenging scenarios by perturbing it. Clean
4

---

Published as a conference paper at ICLR 2025
speech is drawn from Librispeech (Panayotov al. 2015) test-clean, TEDLIUM (Hernandez et al., 2018) release 3 test and MultiLingual LibriSpeech (MLS) (Pratap et al. (020) test. Librippecch con- tains professional recordings of English audio books. Meanwhile, TEDLIUM contains professional recordings of English TED talks and provides lexical and phonetic diversity which LibriSpeech may lack. To increase the applicability of SRB to non-English and multi-lingual models we also include Spanish speech from MLS, which contains professionally recorded audio books in several languages.
(2) Social Gatherings: The ability transcribe speech from semi-formal or informal settings, as well as far-field audio is useful for models used in meeting rooms, smart homes, and even subtitle generation, thus SRB includes English speech from dinner parties and meetings recorded by (2.1) near- and (2.2) far-field mics from CHiME-6 (Barker et al. 2017) and AMI (Kraaij et al. (005).
(3) Speech Variations: ASR models must remain accurate under variations in pronunciation and prosody to serve diverse speakers. We therefore include (3.1) clean accented speecl from English and Spanish subsets of Common Voice 17 (CV17, et al. 2019) in SR3. To provide additional prosodic variability and to represent the increasing pervasiveness of generative AI, we also include synthetic speech generated by YourTTS (Casanova et al. (2022) (English) and Bark|Suno| (b) (Spanish) from transcripts of the three clean datasets from scenario (1) in the voices of English and Spanish speakers from VCTK Yamagishi et al. 2019) and Bark Speaker Library (v2. Suno a). respectively.
The following scenarios involve synthetic perturbation of all three clean datasets from scenario (I).
(4) Environmental Effects: While noisy real speech datasets like Common Voice (Ardila et al. 2019)
and Switchboard (Godfrey et al. 1992) exist, the noise in them is not controlled or even known. Thus in SRB, we perturb clean speech to simulate (4.1) environmental noise, and (4.2) spattal acoustics. Coneretely, we add real environmental noise from ESC-50 (Piczak 2015), , MS-SNSD (Reddy et al.,
2019), MUSAN (Snyder et al. 2015) and WHAM! (Wichern et al. 2019) at Signal-to-
Ratios (SNR) of 10, 20, 30 and 40 dB. To simulate spatial acoustics, we add echo via SoXfI and simulate Room Impulse Response (RIR) via convolution with real and simulated RIRs from Ko et al. (2017).
Digital media often undergoes processing and contains special effects, which are therefore included in SRB. Specifically, we include standard audio processing operations like amplitude gain, resampling, lowpass, and highpass filtering (2.2) special effects like bass gain, treble gain, tempo increase, tempo decrease, speed increase, speed decrease, pitch increase, pitch decrease, chorus, tremolo, and phaser, and (2.3) Gaussian white noise.
(5) Digital Augmentations:
(6) Adversarial Attacks: Models used in high-stakes settings are prime targets for adversaries and thus must resist attempts to compromise their accuracy. We use two types of adversarial attacks in SRB: (2.1) utterance-specific and (2.2) aiterance-agnostie attacks. The utterance- -peecffcc attack searches for a perturbation 8, for a given speech recording x, such that a given model maximally mistranscribes it. To find 8, we follow Madry et al. (2018) and use projected gradient descent to solve ma L(M yz)) where is a differentiable loss function, like CTC-Loss, between the model's 's output Mxx) and the true transcript y", with E [10, 40). The utterance-agnostic attack is similar to the utterance-specific attack, except 5 is optimized over held-out set, dey instead of each test utterance. This represents a more realistic scenario where an attacker tries to mount denial-of-service attack against an ASR model by introducing utterance-agnostic perturbation at some point in the transcription pipeline. We use the method of Neekhara et al. (2019) to find 5 : Sece....SRR( 2) S E [10, 40) such that CER( (æ: + 8|æ E ( > (see Alg. where de is the dev split of LibriSpeech, TEDLIUM and MLS.
Note of Extensibility and Usage:
We have released our source code with instructions for recon- structing the data in SRB, and reproducing the results of this paper ( SRB can easily be extended to other languages and speech datasets using the provided scripts for extracting accented speech from any language in CV17, and for simulating scenarios 3.2-6 on any speech recording or dataset.
We have also publicly released the data for all the above scenarios, except adversarial attacks and perturbed TEDLIUM recordings, on Huggingface Hub (see footnote []. We made these exceptions because TEDLIUM's license prohibits the distribution of derivatives, and the adversarial attacks must be computed separately for each target ASR model. To the extent possible, we encourage users to evaluate their models on the publicly released data to ensure reproducibility.
AAceett annotations (excluding US English), and a DNSMOS score > 3.4 Reddy et al. 2020
AAaiilalle from
5

---

Published as a conference paper at ICLR 2025
3.2 METRICS
SRB measures the utility of the model with the widely used Word Error Rate. WER is computed as the word-level edit distance between the reference and the predicted transcripts, normalized by the length of the reference (see Appendix [ for formal definitions). To measure the robustness of the model under challenging scenarios, we use WER Degradation (WERD), computed as WER((,) W F. ( where A' and X, are datasets containing clean speech and speech from scenario s, respectively. For scenarios (1)-(3.1) (see $3.1). ). X₂ is an inherently noisy dataset, and X will be LibriSpeech for English and Multi-Lingual LibriSpeech for Spanish. For scenarios (3.2)-(6), X is a clean dataset, and X, is a perturbed version of X.
When aggregating metrics (WER/WERD) over multiple scenarios, we follow the practice of Hendryyks & Dietterich (2019) and divide the metric by a measure of difficulty, ie., by the (estimated) speech quality degradation. This adds weight to emrors on "easy" scenarios (less qual- ity degradation) and underweights errors on "harder" scenarios (more quality degradation) when computing averages. We refer to the difficulty normalized versions of WER/WERD as Normalized WER/WERD (NWER/NWERD). We estimate speech quality using DNSMOS (Reddy et al.) 2019) and PESQ et al., 2001 Miao Wang & ananda seelan 2022). which are models of human judg- ments of speech quality and predict Mean Opinion Scores (MOS, Rec 2018). PESQ uses various signal processing methods to predict MOS, while DNSMOS uses DNNs do the same. To compute speech quality degradation we compute PESQ and DNSMOS for each clean and noisy recording multiplied by- 11 (lower values indicate less degradation). Since we are only interested in the relative degradation between scenarios, we normalize the scores to have mean 50 and standard deviation 25.
Note on usage:
We use NWERD for non-adversaiial scenarios (1-5) but WERD for adversarial attacks because adversarial attacks are model-specific and thus DNSMOS/PESQ scores for adversarially perturbed audio will be different for each model, which will lead to a different normalization during NWERD computation and make comparisons difficult.
EVALUATION
We evaluate several recent ASR DNNs ($4.1) using SRB and analyze the results uncover fine- grained differences in their robustness in various challenging scenarios. We further extend our analysis by measuring ASR model robustness for various sub-groups, namely English speech and non-English (Spanish) speech, and male and female speakers. Prior works (Liu et al. 2022 Veliche &
Fung 2023) observe that there is disparity in transcription quality between subgroups. Our analysis
augments these observations by showing that shorr disparities in robustness may also exist, thus demonstrating the utility of SRB the broader field of trustworthy Al.
4.1 MODELS
For English, we evaluate Whisper (Radford al.
2023 large-v2, base, medium, small, and tiny (wsp-
md..bsmd,ssm.o Wav2Vec-2.0 (Baevski et al. 2020) base, large, self-trained large (Xu et al. 2021
and Robust Wav2Vec (Likhomanenko al.) 2020) w22.2-bs..l..gg-fff.gg
HuBERT (Hsu et al.
2021a) large and XI (hubt-llg.xl)). Nvidia Canary (VVIDIA) (cnry-1b), Nvidia Parakeet RNN-T and CTC (VIDIA)) with 0.6B and I.IB parameters (prkt-rnnt-(0 6, 1. 1.1b,, prkt-ctc-(0.66 (.. MMS (Pratap et al. 2020) (mms-1b).. Sp (Ao al. 2022) (spch-t5), DeepSpeech (Amodei ct al. 2016) (ds), and Speechbrain (Ravanelli et al. (024)) models with Conformer encoders, and transformer and RNN-T decoders. For Spanish speech, we evaluate mono-lingual Wav2Vec base Spanish (Wagg et al. 2021b) (w2v2-bs-es). Wav2Vec XLSR Spanish (Conneau et al., 2020) (w2v2-Ig-es), wsp-llg.bos ts. and mms-1b. We used the Huggingface implementations where available, except ds (htpps: /ttt:s://gtiin /tt/tggtbbbb.////////////t com/ SweaSSeaaeeeeeeeeeeeeeeeeeeeeeee pytorc)) More
details about the models are in Table 5
4.2 ROBUSTNESS OF ASR MODELS
Table I presents the utility and robustness of a subset of English and Spanish ASR models under non-adversarial and adversarial scenarios. The subset was selected to exclude small and/or less accurate models. The results of the excluded models, however, are used in 4.3

---

Published as a conference paper at ICLR 2025
clean
accent
noise (env)
noise (White)
social
social (NF)
sputial
synth speech
AVG
Adv
proc
(FF)
(UA)
(US)
Lang
Model
(WER)
(NWERD)
(WERD)
wsp-lg
3.1
2.8
40.9
prkt-rnnt-Ii lb
5.9
11.6
8.7
4.2
1.6
6.4
48.3
39.8
11.8
3.0
15.0
10.9
69.
40.0
wsp-md
7.9
12.4
27.8
2.8
3.3
3.5
41.8
35.5
5.2
15.4
3.7
56.6
30.1
prkt-cte-I.Ib
6.0
16.9
10.3
4.2
9.6
44.6
35.0
13.5
5.9
15.9
71.2
39.8
EN
enry-Ib
6.0
13.9
18.2
4,4
2.7
15.0
45.7
36.7
15.3
5.7
17.5
14.8
61.7
38.3
w2V2-Ig-shh
30.7
17.6
15.2
24.0
hubr-xl
8.4
38.9
29.1
15.5
16.2
20.8
69.5
71.2
26.4
13.5
33.5
13.9
36.8
25.3
wsp-bs
9.6
30.1
88.8
8.7
22.5
63.9
47.8
17.9
12.6
33.6
88.5
45.6
w2v2-Ig
9.7
60.6
39.5
19.0
26.0
24.1
77.3
79.9
37.3
17.7
42.4
16.6
31.1
23.8
cary-1b
3,2
699.9
21.7
17.4
7.6
30.5
36.3
28.6
120.3
26.1
84.3
55.2
wsp-ig
5.8
6.8
19,4
12.3
5.5
9.2
5.5
24.6
11.9
13.7
65.0
39.4
w2v2-1g-es
31.6
31.8
30.1
20.5
40.7
89.0
104.1
49.7
33.9
71.0
52.4
wsp-bs
14.8
62.0
133.2
43.1
25.8
60.4
58.1
87.8
67.2
19.5
159.5
89.5
mms-lb
15.7
27.9
49.9
30.6
w2v2-bs-es
25.7
45.6
55.9
44.9
25.0
58.4
103.3
152.7
69.4
10.0
33.8
21.9
Table 1: The utility and robustness of selected English and Spanish models (see Table 7 for more results). Utility is measured by WER of the models on clean speech. Robustness is measured by the NWERD on aon-adversarially perturbed speech and WERD on adversarially perturbed speech. Adv (UA) refers to utterance agnostic attacks, while Adv (US) refers to utterance specific ones. The metries are averaged over all datasets, perturbations, and severities in each category.
4.2.1 ROBUSTNESS NON-ADVERSARIA SCENARIOS
English Models:In terms of average NWERD, we observe that wsp-ig emerges as the most robust model for non-adversarial scenarios, followed by mikt-rrnt-1.1b and wsp-md. Interestingly, cnry-1b, which is the top model on the Open ASR Leaderboard (OAL,, (Sriaataa et al, 2023), ranks 5th on SRB. This result highlights the fact that SRB provides more rigorous assessment of model's robustness than existing benchmarks like OAL. We also see that SRB reveals subtle weaknesses and strengths of various models.
For instance, we see that wsp-ig and wsp-md are significantly more robust to special effects (sFX) and spatial acoustics than other models, including cnry-1b.
To identify the specific types of sFX and spatial acous- tie perturbations against which cnry-1b lacks robustness, we plot the WERD on each perturbation within these cate- gories (Fig. 3 we find that cnry- 1b much more sensitive than its peers to echo, real room impulse responses, and speed and pitch modifications. This analysis demonstrates that SRB evaluate the robustness of ASR models at multiple granularities and can pinpoint the weaknesses of given model. Detailed results can be found in Figs. and I1 in the appendix. Given that no data augmentation, other than SpecAugment (Park al. 2019). was used to train Whisper (Radford et al. 2023). this indicates that Whisper was trained on data that may have included digital media like music or movie soundtracks, and speech recorded in diverse acoustic environments settings that may not sufficiently represented in public data sources. Curating such diverse datasets is promising direction for future work.
wagemd wap-ig cnry-1b
prkt-mmt--1.1
WERD
20
pitch down
memolo
ahaser
penn Thanu
real
ptch
Spatial
Figure 3: WERD of cnry- wsp-md and wsp-ig perturbations in the spa- tial acoustics and special effects cate- gories.
We also note that despite being pre-trained on 60K hours of speech, Wav2Vec and Hubert models severely lack robustness. Particularly concerning is their weak performance on accented speech, social scenarios and spatial acoustics, which models very likely to encounter in the real world.
Takeaways: (1) Despite topping the Open ASR Leaderboard, cnry- is significantly less robust than wsp-lg, which is ranked 10 OAL. cnry- 1b particularly lacks robustness to special effects and spatial acoustics. (2) Wav2 Vec variants struggle against accented speech and social settings, thus, may not be suitable when users have diverse accents.
7

---

Published as a conference paper at ICLR 2025
Spanish Models:We observe that wsp-ig is the most robust model against non non-adversarial perturba- tions by some margin. We notice that all models, except wsp-lg, struggle against accented speech and yield high NWERS. cnry-1b is particularly, weak against accented speech with an NWERD of 700% (WERD=205%) Apart from accented speech, cnry-1b b is quite robust on all other categories of non-adversarial perturbations. mms-1b is also fairly robust and, unlike other models, its NWERD does not vary erratically from one category to another.
4.2.2 ROBUSTNESS ADVERSARIAL SCENARIOS
English models: We observe that w2v2-1g achieves the lowest WERD and thus is the most robust model against utterance-speciff adversarial attacks. Interestingly. while Wav2Vee models exhibited mediocre robustness to non-adversarial perturbations, they are more robust to utterance-specific attacks, than Whisper, Canary, and Parakeet, which were the most robust on non-adversarial pertur- bations. We also note from Fig. 8 (in Appendix) that most Wav2Vee models are considerably more robust to attacks against TEDLIUM than against LibriSpeech, and the opposite is true for whisper and Canary models. Under utterance-agmostic attacks, the most robust models are mms- b,, wsp-bs, and wsp-sm. It is interesting note that the smaller variants of Whisper limit the generalizability across utterances of the adversarial perturbations to a greater extent than their larger counterparts.
Wav2Vec models are most robust to adversarial attacks; Models that are most robust to non-adversarial perturbations, are mediocre against adversarial perlurbations; Canary and Parakeet models are highly vulnerable to ufterance specific atlacks.
Takeaway:
Spanish models:On Spanish, w2v2-bs-es is the most adversarially robust model. Generally, we observe that Wav2Vee models exhibit better robustness than Whisper and Canary under both utterance- agnostic and utterance-specific perturbations. This is similar to the trends observed in English speech (Fig. Sch. Detailed results can be found in Fig. in the appendix.
Takeaway: General trends similar to English but WERD is higher when Spanish speech is attacked.
4.3 CORRELATES OF ROBUSTNESS
To glean insights that can inform future work, we have conducted the following analysis to model attributes that yield robust models. Specifically, we examine the impact of model size, architecture and accuracy, as well as training dataset size on robustness.
To determine if the prevailing practice of training DNNs with more parameters on larger datasets yielding improvements in robustness, we use robust linear regression to fit a line to WERD vs. number of model parameters/size of the training data for the candidate models in Figs. 5a
and 5b respectively. Increasing model size is correlated with improved robustness (lower WERD).
To further isolate the impact of the model size we plot the NWERD of models from the same family in Fig. 4
Model Family
wav2vec2
which have similar architectures and training datasets. We
WERD
hubert
note that larger models are more robust in the Whisper, Parakeet and 2ec2 families, but, surprisingly, not in the HuBert family.
30
whisper parakeet Pert, Type
20
10
non-adversarial
Next, we consider the model architectures. The architec- tures of the models used in this paper can be divided in to three categories: sequence-to-sequene
10a
10
adversarial
Model Params
(seq2seq) models like Whisper and Canary, encoder only models trained with CTC loss (Graves et al.] 2006) like the Wav2Vee fam- ily, and RNN-T models which are capable of streaming such as some variants of Parakeet. From Fig. 5e we see that in terms of non adversarial robustness RNN-T models outperform seq2seq and CTC models, but in terms of adversarial robustness CTC models achieve the lowest WERD.
Figure 4: NWERD lineplot with non- adversarial and adversarial perturbations, three families of models.
We also measure the robustness-utility trade-off by plotting WERD and NWERD for adversarial and aon-aaversarial perturbations, respectively, against WER on clean data in Figs. 5e and We observe that in both cases the relationship is positive, i.e. more accurate models tend to be more robust, however, the relationship between WERD on adversarial perturbations and clean WER is much weaker.
8

---

Published as a conference paper at ICLR 2025
ran de
72ge
Model harmss
Taining Dota tris Ied
WeR
(b)
(d)
Figure 5: (a & b) WERD for all models with robust regression fitted line on non-ad adversarial (blue) and adversarial (orange) perturbations, plotted against (a) number of parameters, and (b) hours of training data. (c % d) WERD and NWER on adversarial and non-adversarial perturbations are plotted against WER to illustrate the robustness-utin utilliyy trade off. Pareto optimal points are highlighted. (e) Boxplot of WERD for models having various architectures.
Finally, we measure the impact of training data size and, find that increasing training data appears to have only minor influence on robustness (Fig. 5b
Takeaway: (1) Larger models tend to be more robust, while smaller models, even if they are trained on large datasets, are less robust. This runs somewhat counter to the prevailing wisdom (Radford jet al. 2023 Likhomanenko al. 2020). (2) CTC models are more robust than seq2seq models to adversarial attack, but less robust than seq2seq and RNN-T models on non -ovvevreraa perturba- tions. (3) Utility and robustness positively correlated, but correlation is weaker for adversarial robustness.
4.4 DISPARITY IN ROBUSTNESS ACROSS POPULATION SUB-GROUPS
In the preceding analysis, we considered robustness aggregated over the entire population (i.e., dataset). However, populations are generally not homogeneous, and, thus, the robustness of the model may differ on various population sub-groups. Prior works have commonly analyzed sub-group fairness of ASR models by comparing the overall WER for each sub-group on a benchmark dataset (Koenecke et al. 2020). It is possible that models that are fair on average, may not be fair under certain conditions. In the following, we use SRB to uncover and analyze the disparities in the models' robustness across four sub-groups: English and Spanish speech, and male and female speakers. We find that disparities indeed exist, with multi- multiilinggual models generally being more robust for English than Spanish (Fig. 6). and most models being less robust for females than males.
4.4.1 DISPARITY IN ROBUSTNESS ACROSS LANGUAGES IN MULTI-LINGUAL, MODELS
We compare the robustness exhibited by multi-lingus models, wsp-Ig, wsp-bs, cnry-1b and mms-1b English and Spanish. The WERD of these models on both languages is presented in Fig. We observe that Whisper models achieve lower WERD on English speech than on Spanish on almost all perturbation categories, while c and mms- achieve similar WERD on some categories. We also note that the difference in WERD on some common perturbation categories, like environmental noise, and spatial acoustics, is much greater for wsp-ig than for cb.
1056
arry-b
WERD
mms-10 1b
wsp bs
20
env noise white noise spatial acoustics special effects audio processing
adversarial
Figure 6: Comparing the robustness of multi-lingual on English (solid) and Spanish (hatched).
Takeaway: Multilingual models are robust English than Spanish: cnry-1b and wsp-ig most robust on both lan- guages; adversarial robustness results follow the same trend as English.
4.4.2 DISPARITY IN ROBUSTNESS ACROSS GENDERS
To measure the disparity in transcription quality across genders (males/females), we compute the log of the ratio of the WERs of the ASR model on female and male speakers. We call this measure the Log WER Ratio (LWERR). A positive value of LWERR indicates that the model is biased against females and negative value indicates that the model is biased against males.

---

Published as a conference paper at ICLR 2025
WER Rato
0.8
1.0
TEDLIUM
IhriSpeech
CV + AMI
MLS (FS)
social (FF)
0.0
(NF)
Ady (UA)
0.0
Les
-0-2
audio processing
-0.6
nuise
Drcke
St 1.2 20 aud
ue bybzA
fi ZAzIA
squdc
10
Parthon
Gyddsm
pu doa
40 By
01 sun
ST-Au
spatial acoustics
special effects synthetic speech
mm
white
accent
Figure 7: Log WER Ratio for various TEDLIUM, LibriSpeech, Common Voice (CV) and AMI, and Spanish Multilingual Librispeech (MLS (ES)). WERs are averaged across severity levels and individual augmentations within each category before computing the Log WER Ratio.
The LWERR for each dataset is shown in Fig. 7 We note that, on average, the models are biased against females LibriSpeech and Spanish Multilingual Librispeech (MLS-ES), and against males on TEDLIUM, Common Voice and AMI. The bias is most prominent in MLS-ES, where cnry- 1b seems to be yielding the highest disparities among genders. We also note that adversarial perturbations cause the WER of wsp-ig to increase significantly more for females than males LibriSpeech. This interesting because adversarial perturbations do target a specific part of the spectrum and thus should not impact one gender more than the other.
Takeaway: Models are more robust for males on some datasels, and females on other datasets suggesting that used data require further examination; adversarial attacks increase WER of Whisper variants for females more than males; multilingual models, particularty cnry-1b, are more biased against females when transcribing Spanish.
5
LIMITATIONS
Despite the comprehensive design of SRB, there are limitations to consider. Firstly, while SRB includes diverse of datasets, distortions and adversarial attacks, may not encompass all possible real-world scenarios. Additionally, the benchmark has been tested for now English and Spanish languages, which may not capture the robustness challenges faced by ASR models in other languages and dialects. While these limitations potentially affect the generalizability of the results obtained with SRB, it extensibility allows users to easily incorporate additional datasets and models.
CONCLUSION
We propose SRB, a comprehensive benchmark designed to standardize the robustness evaluations of Automatic Speech Recognition (ASR) models. We demonstrate the utility of SRB in evaluating the robustness ASR models via several concrete examples, as well as its potential to facilitate evaluations of other aspects of trustworthy AI, like faimess. We believe that SRB will enable rigorous robustness evaluations of ASR models in a highly standardized manner, allowing easy comparisons between existing and new approaches. To further facilitate robustness evaluations for researchers and model developers, we release transformed test sets in English and Spanish. We anticipate that this will make robustness evaluations more prevalent and encourage model developers consider robustness as a key metric for improvement.
7 ACKNOWLEDGMENTS
Funded by the European Union's Horizon 2020 RIA ELOQUENCE project (Grant Agreement No. 101 135916), by the Spanish Project 6G-RIEMANN (Grant Agreement No. 2022/0005420), and partially also by the European Lighthouse Safe and Secure AI (ELSA) from the European Union's Horizon Europe program under grant agreement No 101070617. Views and opinions expressed however those of the author(s) only and do not necessarily reflect those of the European Union or European Commission- EU. Neither the European Union nor the granting authority can be held responsible for them.
10

---

Published as a conference paper at ICLR 2025
Funded by the European Union
REFERENCES
Naveed Akhtar, Ajmal Mian, Navid Kardan, and Mubarak Shah. Advances in adversarial attacks and defenses in computer vision: A survey. IEEE Access, 9:155161-1555 2021.
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech 2: End-to- end speech recognition in english and mandarin. In International conference on machine learning, pp. 173-182. PMLR, 2016.
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein, Square attack: query-efficient black-box adversarial attack via random search. In European Conference on Computer Vision, pp. 484-501. Springer, 2020.
Junyi Ao, Rui Wang. Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, Zhihua Wei, Yao Qian, Jinyu Li, and Furu Wei. SpeechTS: snified-modas encoder- decoder pre-training for spoken language processing. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 7723-5738, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 118888//1122222acl-100 393. BtLhttps::/ httttt::s//caaaatholooooo
org 2022 acl-longl..... 393
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massively- multilingual speech corpus. arXiv preprint arXiv:191..06677 2019.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: A framework for supe-super learning of speech representations. Advances in neural information processing systems, 33: 21212244414466 2020.
Jon P Barker, Ricard Marxer, Emmanuel Vincent, and Shinji Watanabe. The chime challenges: Robust speech recognition in everyday environments. New era for robust speech recognition: Exploiting deep learning, pp. 327-344, 2017.
Edresson Casanova, Julian Weber, Christopher D Shulby, Armaldo Candido Junior, Eren Gölge, and Moacir A Ponti. Yourtts: Towards zero-shot multi-speakeer tts and zero-shot voice conversion for everyone. In International Conference Machine Learning, pp. 2709-2727 2720. PMLR, 2022.
Chen Chen, Nana Hou, Yuchen Hu, Shashank Shirol, and Eng Siong Chng. Noise-robust speech recognition with 10 minutes unparalleled in-domain data. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4298 4302. IEFE, 2022.
Pin-Yu Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks deep neural networks without training substitute models. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pp. 15-26, 2017.
Zhuo Chen, Takuya Yoshioka, Liang Lu, Tianyan Zhou, Zhong Meng, Yi Luo, Jian Wu, Xiong Xiao, and Jinyu Li. Continuous speech separation: Dataset and analysis. In ICASSP 2020-2020 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 7284-7288. IEEE, 2020.
Alexis Conneau, Alexei Baevski, Ronan Collobert, AbdeIrahman Mohamed, and Michael Auli, Unsupervised cross-lingual representation learning for speech recognition. arXiv preprint arXiv:2006. 13979, 2020.
Joris Cosentino, Manuel Pariente, Samuele Comell, Antoine Deleforge, and Emmanuel Vin- cent. Librimix: An open-source dataset for generalizable speech separation. arXiv preprint arXiv:2005. 1262, 2020.
11

---

Published as a conference paper at ICLR 2025
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag. Edoardo Debenedetti, Nicolas Flam- marion, Mung Chiang, Prateek Mittal, and Matthias Hein, Robustbench: a standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670. 2020.
John J Godfrey, Edward C Holliman, and Jane McDaniel. Switchboard: Telephone speech corpus for research and development. In Acoustics, speech, and signal processing, ieee international conference on, volume 1, pp. 517-520. IEEE Computer Society, 1992.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples, 2014.
Alex Graves, Santiago Fernnndez, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference Machine learning, pp. 369-376, 2006.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv: 122611 2019.
Dan Hendryeks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of -ut-of-distriuttion generalization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. $340-8349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adver- sarial examples. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15262-15271, 2021b.
Hynek Hermansky and Sangita Sharma. Traps-classffiers of temporal patterns. In Fifth International Conference on Spoken Language Processing. 1998.
Hynek Hermansky, Nelson Morgan, Aruna Bayya, and Phil Kohn. Rasta-plp speech analysis. In Proc. IEEE Int't I Conf. Acoustics, speech and signal processing, volume I, pp. 121-124, 1991.
François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Esteve. Ted-lium 3: Twice as much data and corpus repartition for experiments on speaker adaptation. In Speech and Computer: 20th International Conference, SPECOM 2018, Leipzig, Germany, September 18-22, 2018, Proceedings 20, pp. 198-208. Springer, 2018.
John R Hershey, Zhuo Chen, Jonathan Le Roux, and Shinji Watanabe. Deep clustering: Discriminative embeddings for segmentation and separation. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 31-35. IEEE, 2016.
Hans-Günter Hirsch and David Pearce. The aurora experimental framework for the performance evaluation of speech recognition systems under noisy conditions. In ASR2000-Automaan speech recognition: challenges for the new Millenium ISCA tutorial and research workshop (ITRW), 2000.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. JEEE/ACM Transactions Audio, Speech, and Language Processing, 29:3451-3460, 2021a.
Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski, Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Jacob Kahn, Ann Lee, Ronan Collobert, Gabriel Synnaeve, et al. Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training. arXiv preprini arXiv:2104.011 2021b.
Marco Jeub, Magnus Schafer, and Peter Vary. A binaural room impulse response database for the evaluation of dereverberation algorithms. In 2009 16th International Conference Digital Signal Processing, pp. 1 1-5. IEEE, 2009.
Biing Hwang Juang and Mazin G Rahim. Signal bias removal by maximum likelihood estimation for robust telephone speech recognition. IEEE Transactions on Speech and Audio Processing, 4(1):19, 1996.
12

---

Published as a conference paper at ICLR 2025
Jeff Larson Julia Angwin. Machine Bias. URL machine-bias -i
Chanwoo Kim and Richard M Stern. Power-normalized cepstral coefficients (pncc) for robust speech recognition. JEEE/ACM Transactions on audio, speech, and language processing, 24(7): 1315-1329, 2016.
Keisuke Kinoshita, Mare Delcroix, Takuya Yoshioka, Tomohiro Nakatani, Emanuel Habets, Reinhold Habb-Ummach, Volker Leutnant, Armin Sehr, Walter Kellermann, Roland Maas, et al. The reverb challenge: A common evaluation framework for dereverberation and recognition of reverberant speech. In 2013 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, pp. 1-4. TEEE, 2013.
Tom Ko, Vijayaditya Peddinti, Daniel Povey, Michael Seltzer, and Sanjeev Khudanpur, study on data augmentation of reverberant speech for robust speech recognition. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5220-5224. IEEE, 2017.
Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R Rickford, Dan Jurafsky, and Sharad Goel. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 1 117(14):7844-689, 2020.
Wessel Kraaij. Thomas Hain, Mike Lincoln, and Wilfried Post. The ami meeting corpus. In Proc. International Conference on Methods and Techniques in Behavioral Reseurch, 2005.
Cassidy Laidlaw, Sahil Singla, and Soheil Feizi. Perceptual adversarial robustness: Defense against unseen threat models. In International Conference on Learning Representations, 2021. URL
https://openreview. net forum? fecummid==FFwaa WBosccGcJJ
T Lander. CSLU: Foreign Accented English Release 1.2, 2022. URL TRL[htttp:://b..org//1 5683/8P2/K7EQTE
Jinyu Li, Li Deng, Yifan Gong, and Reinhold Haeb-Umbach. An overview of noise-robust automatic speech recognition. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 22(4):
745-777, 2014.
Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Paden Tomasello, Jacob Kahn, Gilad Avidov, Ronan Collobert, and Gabriel Synnaeve. Rethinking evaluation in asr: Are our models robust enough? arXiv preprint arXiv:2010.11745: 2020.
Chunxi Liu, Michael Picheny, Leda San, Pooja Chitkara, Alex Xiao, Xiaohui Zhang, Mark Chou, Andres Alvarado, Caner Hazirbas, and Yatharth Saraf. Towards measuring fairness in speech recognition: Casual conversations dataset transcriptions. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6162-6166. IEEE, 2022.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Rafael G. Dantas Miao Wang, Christoph Boeddeker and ananda seelan. Pesg (perceptual evaluation of speech quality) wrapper for python users, May 2022. URL htt.ttss///oi..og////10// Zzeoodo. 6549559
Nasser Mohammadiha, Paris Smaragdis, and Ame Leijon. Supervised and unsupervised speech enhancement using nonnegative matrix factorization. IEEE Transactions audio, speech, and language processing, 21010))214-21151 2013.
Satoshi Nakamura, Kazuo Hiyane, Futoshi Asano, Takanobu Nishiura, and Takeshi Yamada. Acous- tical sound database in real environments for sound scene understanding and hands-free speech recognition. In LREC, pp. 965-968, 2000.
Vedant Nanda, Samuel Dooley, Sahil Singla, Soheil Feizi, and John P Dickerson. Faimess through robustness: Investigating robustness disparity in deep learning. In Proceedings of the 2021 ACM Conference Fairness, Accountability and Transparency, pp. 466-477, 2021.
13

---

Published as a conference paper at ICLR 2025
Paarth Neekhara, Shehzeen Hussain, Prakhar Pandey, Shlomo Dubnov, Julian McAuley, and Farinaz Koushanfar. Universal adversarial perturbations for speech recognition systems. arXiv preprint arXiv:1905.03828. 2019.
NVIDIA. NeMo Documentation. user Snerr-gtee /tt.i///aactteeeeetttttta/////aa
www.coooooooo/e/..ooooo nemo- -etm//eeo---rwortttee framework/
Raphael Olivier and Bhiksha Raj. Recent improvements of asr models in the face of adversarial attacks. Interspeech, 2022. http//ttppp:/xxx..ooo///////
16536
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based on public domain audio books. 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5206-5210 5210. IEEE, 2015.
Nicolas Papernot, Patrick McDaniel, and Ian Goodfellow. Transferability in machine learning: from phenomena to black-box attacks using adversarial samples. arXiv preprint 2016.
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc Le. Specaugment: A simple data augmentation method for automatic speech recognition. arXiv preprint arXiv:1904.08770 2019.
David Pearce and J Picone. Aurora working group: Dsr front end Ivesr evaluation au/384/02. Inst. for Signal Inform. Process., Mississippi State Univ. Tech. Rep, 2002.
Karol J. Piczak. ESC: Dataset for Environmental Sound Classification. In Proceedings of the 23rd Annual ACM Conference on Multimedia, pp. 1015-1018. ACM Press, 2015. ISBN 978-1-4503- 3459-4. doi: 10. 10.1145/2733333 URL http: //l acm. wwwp/eitt.tt..ooooo.ooo.o....ttt cfm?
Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. Mls: A large-scale multilingual dataset for speech research. arXiv preprint arXiv:2012.03411,, 2020.
Vineel Pratap. Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarane Zarandi, et al. Scaling speech technology to 1,000+ languages. arXiv preprint arXiv: 230.::3355..1166, 2023.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, pp. 28492-28518. PMLR, 2023.
Mirco Ravanelli, Titouan Parcollet, Adel Moumen, Sylvain de Langen, Cem Subakan, Peter Plantinga, Yingzhi Wang, Poonch Mousavi, Luca Della Libera, Artem Ploujnikov, Francesco Paissan, Davide Borra, Salah Zaiem, Zeyu Zhao, Shucong Zhang, Georgios Karakasidis, Sung-Lin Yeh, Pierre Champion, Aku Rouhe, Rudolf Braun, Florian Mai, Juan Aallaga-Gomeet Seyed Mahed Mousavi, Andreas Nautsch, Xuechen Liu, Sangeet Sagar, Jarod Duret, Salima Mdhaffar, Gaelle Laperriere, Mickael Rouvier, Renato De Mori, and Yannick Esteve. Open-source conversational ai with SpeechBrain 1.0, 2024. URL httttttt///aaee
org/ httg:/bb/2240005555
ITUT Rec. P. 808, subjective evaluation of speech quality with crowdsourcing approach. ITU-T, Geneva, 2018
Chandan KA Reddy, Ebrahim Beyrami, Jamie Pool, Ross Cutler, Sriram Srinivasan, and Johannes Gehrke. A scalable noisy speech dataset and online subjective test framework. Proc. Interspeech 2019, pp. 1820, 2019.
Chandan KAA Reddy, Vishak Gopal, Ross Cutler, Ebrahim Beyrami, Roger Cheng, Harishchandra Dubey, Sergiy Matusevych, Robert Aichner, Ashkan Aazami, Sebastian Braun, et al. The in- terspeech 2020 deep noise suppression challenge: Datasets, subjective testing framework, and challenge results. In INTERSPEECH. 2020.
14

---

Published as a conference paper at ICLR 2025
Antony W Rix, John G Beerrndss Michael P Hollier, and Andries P Hekstra. Perceptual evaluation of speech quality (pesq)-a new method for speech quality assessment telephone networks and codees. In 2001 IEEE international conference acoustics, speech, and signal processing. Proceedings (Cat. No. 01CH37221), volume 2, pp. 749-752. IEEE, 2001.
Jo ao Santos and Tiago H Falk. Updating the srmr-ci metric for improved intelligibility prediction for cochlear implant users. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 22(12)2197-2220 2014.
Xian Shi, Fan Yu, Yizhou Lu, Yuhao Liang, Qiangze Feng, Daliang Wang, Yanmin Qian, and Lei Xie. The accented english speech recognition challenge 2020: open datasets, tracks, baselines, results and methods. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6918-6922. IEEE, 2021.
David Snyder, Guoguo Chen, and Daniel Povey. Musan: A music, speech, and noise corpus. arXiv preprint arXiv: 15 1111500044444 2015.
David Solans Noguero, Diana Ramírez-Cifueee
Esteban Andrés Ríssola, and Ana Freire. Gender bias when using artificial intelligence to assess anorexia nervosa on social media: data-driven study. Journal of Medical Internet Research, 25:e45188 2023.
Vaibhav Srivastav, Somshubra Majumdar, Nithin Koluguri, Adel Moumen, Sanchit Gandhi, et al.
Open automatic speech recognition leaderboard. htttps:/hhugee ngw.ce.c....oo co/ wtwpaacc..cccceeaaaeeeeeeees
iif-audic/opent
leaderboard. 2023.
Richard M Stern and Nelson Morgan. Hearing believing: Biologically inspired methods for robust
automatic speech recognition. IEFE signal processing magazine, 29(6):3 2012.
Suno.
Bark
speaker
library
(v2).
https:// suno ai
bc67 (cc.ccff776600000033366676533333 a. Accessed: 2024-11-23.
Suno. http://.p/////abbbb..c/////////.
b. Accessed: 2024-11-23.
Christian Szegedy, Wojeiech Zaremba, llya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014. URL http:://///ai
org/abs 1312.6622 6199
Irina-Elena Veliche and Pascale Fung. Improving fairness and robustness in end-to-end speech recognition through unsupervised clustering. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023.
Viet Quoc Vo, Ehsan Abbasnejad, and Damith C Ranasinghe. Query efficient decision based sparse attacks against black-box deep learning models. arXiv preprint arXiv:2202.00091 2022.
Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. Adversarial glue: A multi-task benchmark for robustness evaluation of language models. arXiv preprint (tt::2/11.02800 2021a.
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. Voxpopuli: large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. arXiv preprint arXiv:2101.003390 2021b.
Chenxu Wang, Ming Zhang, Jinjing Zhao, and Xiaohui Kuang. Black-box adversarial attacks on deep neural networks: A survey. In 2022 4th International Conference Data Intelligence and Security (ICDIS), pp. 88-93. IEEE, 2022a.
Xuezhi Wang, Haohan Wang, and Diyi Yang. Measure and improve robustness in nlp models: A survey. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4589,4586, 2022b.
Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning structured sparsity in deep neural networks. In NIPS, 2016.
15

---

Published as a conference paper at ICLR 2025
Gordon Wichern, Joe Antognini, Michael Flynn, Licheng Richard Zhu, Emmett McQuinn, Dwight Crow, Ethan Manilow, and Jonathan Le Roux. Wham!: Extending speech separation to noisy environments. arXiv preprint arXiv: 1tt1::99070116600 06000 2019.
Matthew Wicker, Xiaowei Huang, and Marta Kwiatkowska. Peature-guider black- box safety testing of deep neural networks. In Tools and Algorithms for the Construction and Analysis of Systems: 24th International Conference, TACAS 2018, Held as Part of the European Joint Conferences on Theory and Pructice of Software, ETAPS 2018, Thessaloniki, Greece, April 14-20, 2018, Proceedings, Part I 24, pp. 408 426. Springer, 2018.
Kevin W Wilson, Bhiksha Raj. Paris Smaragdis, and Ajay Divakaran. Speech denoising using nonnegative matrix factorization with priors. 2008 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 4029 4032. IEEE, 2008
Qiantong Xu, Alexei Bacvski, Tatiana Likhomanenko, Paden Tomasello, Alexis Conneau, Ronan Collobert, Gabriel Synnaeve, and Michael Auli. Self-training and pre-training are complementary for speech recognition. In ICASSP 202 1021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3030-3034. IEEE, 2021.
Junichi Yamagishi, Christophe Veaux, and Kirsten MacDonald. CSTR VCTK Corpus: English multi-speaker corpus for CSTR voice cloning toolkit (version 0.92), 2019.
Kai Yu, Mark Gales, and Philip Woodland. Unsupervised adaptation with discriminative mapping transforms. IEEE Transactions on Audio, Speech, and Language Processing, 17(4):714-723, 2009,
Pu Zhao, Pin- Yu Chen, Siyue Wang, and Xue Lin. Towards query-efficient black-box adversary with zeroth-order natural gradient descent. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 6909-6916, 2020.
A PERTURBATION GENERATTIO//APPICCATOO PROCEDURE
Below, we provide further details the perturbations that make up Speech Robust Bench. Table shows the parameters for each perturbation and Table 3 shows the normalized DNSMOS and PESQ scores for each perturbation.
A noise vector of the same length as the audio signal is sample from standard normal distribution, scaled such that its magnitude corresponds to a specific SNR, and then added to the audio signal. We use torchaudio.. function. _dd__odid noise to add the noise to the speech at a given SNR.
GAUSSIAN NOISE:
ENVIRONMENTAN NOISE: We use the recordings of environmental noises from the test/eval subsets of ESC-50 (Piczak] 2015), MS-SNSD (Reddy et al.] 2019). MUSAN (Snyder et al.) 2015) and WHAM (Wichern al. 2019). We create separate perturbed version of the clean data using each of these noise datasets. To do so, for each test utterance we sample a random environmental noise and add it the audio signal at the specified SNR. We clip the noise if is longer than the speech, and repeat it if it is shorter than the speech. We use torchaudic function. add no hoise to add the noise the speech at given SNR.
ROOM IMPULSE RESPONSE: The simulated and real RIRs from (Ko et al. 2017 are applied to clean recordings. As a measure of intensity, RT60 is estimated for the simulated RIRs using Sabine's 's formula with the room dimensions and absorption coefficient provided in the dataset. For the real RIRs, we compute the SRMR (Santos Falk 2014) using the implementation from https 7/gi thub. com/ ut kus sww.cheeteeeeeeee i cs / tree/ ma mateer) The severity is defined in increasing RT60s for the synthetic RIRs, and decreasing SRMR for the real RIRs. Table 2 shows the average RT60/SRMS in each severity level. During evaluation, random RIR having the given severity level is sampled for each test recording.
RESAMPLING, SPEED, PITCH, AND GAIN PERTURBATIONS: gain perturbations were applied using the Resample Speed, PitchShift and Vol transforms from torchaudio.
The resampling speed, pitch, and
16

---

Published as a conference paper at ICLR 2025
Category
Perturbation
Sev
Sev 2
Sev
Sev 4
Gaussian Noise
30 dB
20 dB
10 dB
0 dB
Environmental Noise
dB
20 dB
10 dB
0 dB
Environment
Music
30 dB
20 dB
10 dB
0 dB
Crosstalk
dB
20 dB
10 dB
0 dB
RIR
0.27s
0.58s
0.99s
1.33s
Spatial Acoustics
Real RIR
9.1
7.1
4.1
1.8
Echo (delay)
125 ms
250 ms
500 ms
1000 ms
Bass (gain)
20
30
40
50
Treble (gain)
10
23
36
Phaser (decay)
0.3
0.5
0.7
0.9
tempo-up
1.25x
1.5x
1.75x
2x
lempo-down
0,875x
0.75
0.625x
0.5x
Special Effects
Speed-up
1,25x
1.5x
1.75x
2x
Slow-down
0.875x
0.75
0.625x
0.5x
Pitch Step-up
0.25 oct
0.5 oct
0.75 oct
1 Oct
Pitch Step-down
0.25 oct
0.5 oct
0.75 oct
Chorus (delay)
30
50
70
90
tremolo (depth)
50
66
83
100
Resampling Gain (factor)
0.75x
0.5x
0.25x
0. 0.125x
10x
20x
30x
40x
Audio Processing
Low-pass filter
kHz
2833 kHz
1666 kHz
500 kHz
High-pass filter
500 kHz
1333 kHz
2166 kHz
3000 kHz
PGD Attack
40 dB
30 dB
20 dB
10dB
Adversarial
Utterance Agnostic Attack
40 dB
30 dB
20 dB
10dB
Table 2: The parameters defining the various severity levels of the perturbations used in the proposed benchmark.
OTHER SPECIAL EFFECTS: These effects are applied via SoX filters of the same name. We used torchaudio sox_ www__fffetss__./ apppl_effectssteeeeeeeee to apply these filters to the audio. The args for each filter are follows:
echo 0.8 0. <delay> 0.3
phaser 0.6 0.8 3 <decay> 2 =-t"
Tempo <factor> 30
sinc
<lo-freq>
sinc 0 O-<hi- --ihiffreq>>
tremolo 20 <depth>
treble <gain>
bass <gain>
chorus 0.9 0.9 <delay> 0.4 0.25 2 -t (<delay>+10) 0.3 0.4 2
-8
VOICE CONVERSION We use use YourTTS (Casanova et al. audio from textual transcripts in a given speaker's 's style. The transcripts from the test clean subset of LibriSpeech are used. The target speakers are drawn from the VCTK corpus (Yamagishi et al. 2019). which contains accented speech from 12 accents. For each transcript random speaker is chosen to synthesize the audio.
2022) from Coqui to synthesize
https://githhbb wttp://..ooooouuuooeeeoo coquiouuuu...o
17

---

Published as a conference paper at ICLR 2025
Metric
AVG
normalized DNSMOS
normalized PESQ
Scenario
clean
23.1
23.1
accent (en)
33.1
33.1
accent (es)
29.3
29.3
social (chime, FF)
102.2
102.2
social (ami, FF)
85.9
85.9
social (chime, NF)
80.1
80.1
social (ami, NF)
37.3
37.3
AugmentattoonSeverity
2
3
bass
19.1
23.9
36.0
56.4
23.9
28.9
39.6
58.4
11.4
14.3
30.3
54.6
chorus
40.1
49.3
55.5
57.0
31.3
41.3
48.7
49.9
63.8
71.2
74.3
75.8
crosstalk
22.9
38.9
53.1
59.9
24.7
33.1
38.2
41.0
22.2
46.1
70.1
81.6
echo
54.9
54.2
53.6
51.4
40.6
37.8
37.6
36.5
71.3
72.7
71.7
67.9
env noise (MS-SNSD)
51.4
62.4
77.0
89.6
53.5
61.0
74.6
93.5
39.6
58.7
76.7
84.2
env noise (ESC50)
26.7
41.7
58.4
73.8
39.2
45.3
55.0
73.2
20.0
43.1
66.0
79.8
noise (MUSAN)
24.9
43.0
63.0
76.4
26.3
38.7
57.1
73.1
24.8
48.7
70.4
81.0
cnv noise (WHAM)
23.0
46.2
74.2
93.3
23.7
41.2
72.3
101.7
23.1
52.0
76.6
85.4
gain
50.8
69.9
77.6
81.8
46.7
67.8
78.3
84.6
61.3
76.0
80.0
81.7
gaussian noise
53.2
76.6
91.7
82.7
72.1
89.5
103.8
119.6
42.1
69.2
83.0
66.0
highpass
40.9
56.2
68.5
78.5
35.8
45.1
66.7
83.0
46.1
68.3
71.6
74.9
lowpass
33.8
37.9
51.6
79.1
48.9
48.4
64.3
100.1
20.3
29.2
40.4
58.4
music
22.9
43.8
66.7
79.9
26.9
43.0
64.0
78.6
20.0
45.3
70.1
81.9
phaser
15.6
32.9
60.7
80.6
22.8
36.2
59.8
78.6
10.6
31.9
63.6
83.5
pitch down
61.8
68.2
63.8
84.4
39.6
50.9
67.3
82.8
85.9
86.5
68.1
86.7
pitch up
58.9
62.1
65.2
66.0
33.7
39.6
45.7
47.5
85.9
86.4
86.5
86.4
real rir
39.4
54.6
69.9
85.3
35.7
46.5
61.7
89.2
43.1
62.7
78.0
81.4
resample
14.9
28.0
49.9
64.2
25.2
43.7
63.3
77.4
6.7
18.0
38.1
52.5
rir
51.1
64.0
69.3
68.9
42.0
58.1
66.2
66.5
65.0
74.6
78.1
78.1
slowdown
51.4
57.8
65.2
74.0
18.9
31.2
45.3
68.6
85.9
86.1
86.0
86.3
speedup
52.2
59.6
67.2
73.8
23.1
37.6
52.6
65.5
83.7
83.6
83.3
83.0
tempo down
49.4
52.6
55.3
51.2
19.8
23.0
28.2
36.7
81.4
84.7
85.3
85.6
tempo up
51.0
57.9
64.1
70.6
25.7
36.1
47.8
60.1
79.0
82.2
82.4
82.4
treble
12.4
22.4
41.6
63.6
20.8
31.2
44.8
62.3
2.1
12.1
42.2
72.5
tremolo synthetic (es, Bark)
17.7
29.9
60.2
100.2
24.5
38.8
73.9
113.7
9.6
17.6
36.9
75.6
30.7
30.7
synthetic (en, yourTTS)
50.3
17.1
83.6
Table 3: Normalized DNSMOS and PESQ score for each perturbation.
Name
Subset
Hours
Utterances
Speakers
Male/Female
LibriSpeech TEDLIUM 3
test-clean
5.4
2620
20/20
test
3.76
1155
16
10/6
Multi-Lingual LibriSpeech (es)
test
2385
10/10
CHiME-6
eval
5.25
13000
AMI
test
7.35
13168
16
8/8
ESC-50
2.78
2000
MUSAN
108.5
2016
WHAM!
noise-test
9
3000
MS-SNSD
noise-test
0.7
51
Table 4: Distributional statistics of speech (top) and noise (bottom) datasets used in SRB.
CROSSTALK AND MUSIC We use crosstalk and music audios from MUSAN (Snyder et al,, 2015). add the noise the speech at given SNR.
We use torchaudi o. functi
18

---

Published as a conference paper at ICLR 2025
ACCENTS We select subset of audios from the test set of Common Voice 17. The selected audios satisfied the following criteria: (1) the speaker's s accent must be present in the metadata, (2) the accent must not be American, (3) should be clean. The last criterion is satisfied if the DNSMOS (Reddy et al. 2020) score of the recording is at least 3.4. The resulting subset contains 640 recordings. The most popular accent in this set is South Asian (India, Pakistan, Sri Lanka) (25%), followed by British English (25%).
PERERRLL COMMUNICATIONS We CHiME-6 (Barker et al. 2017) and AMI (Kraaij et al. 2005 to obtain recordings of people in social scenarios (dinner party and meetings). In both these datasets, the speakers are recorded through lapel microphone and a room microphone resulting in near and far field recordings. We use both types of recordings and show separate results for them. We remove recordings that contain less than three words since they are often fillers.
UTTARANCE SPECIFIC ADVERSARIAL ATTACK: The utterance specific adversarial perturbations are computed using the untargeted PGD adversarial attack implemented in robust_speech package Olivierr & Raj 2022). The attack is computed follows. First, the maximum possible L2 norm of the noise is determined by solving the equation for SNR for the norm of the noise as follows.
|
SNR = 2000
|10
12
<SNR |8112
10 SNR
||01/2,
where 5 is the noise, x is the audio signal and SNR is the upper bound the SNR in the final signal. Then, we follow the approach of (Madry et al. (2018) and optimize using Projected Gradient Descent (PGD) to maximize the divergence between the true and predicted transcriptions. Formally stated, the attack performs the following optimization:
LM(x 8,y);
max [b1b|l,
where LM is the loss function used to train the ASR model, M, such as CTCLoss or NLLLoss.
We use the method of (Neekhara et al. 2019); implemented in robust cobust_speecll speech package (Olivier Raj] 2022), compute utterance agnostic adversarial perturbations. The main difference between the universal attack and the PGD attack is that the latter computes perturbation vector for each input, whereas the former computes a single perturbation that is expected to successfully attack any input to the model.
UTTERANCE AGNOSTIC ADVERSARIAL ATTACK:
Formally, given ASR model, M, and development speech dataset, X let X (a + 6|æ X be the same dataset under additive perturbation ö, and let M (Xrder) and M (Xyder be the transcripts predicted by M for dev and X The utterance agnostic attack uses PGD to optimize d such that || least t, i.e. CER- (Atde (xdeer) 2 t (using the notation from B}. Similar to the utterance- specific attack, the value of is determined by the maximum allowable SNR using (2. except that to norms are used instead of l2 norms. The full algorithm is described in Algorith
and the Character-Frror Rate (CER) (see $ B) between M( and M (x
Once we compute the perturbation we add it to the test audios (Attstt) at the specified SNR using torchaud io..oo.... funct on _od__oni_iioooooooooooooo se. For LibriSpeech, we use 500 utterances from test-dev as de and test-clean X For TEDLIUM, For Multi-Lingual LibriSpeech, we use 500 utterances from the dev set in the relevant language as deev and the full test set of the same language as Iesst
use the full dev and test sets as d and P
B ADDITIONAL DEFINITIONS
As noted in the main text, use word error rate (WER) as a basic measure for quantifying performance of the models. Following the common practice from ASR literature, the WER is computed as the word-level edit distance between the reference and the predicted transcripts, normalized by the length of the reference. The edit distance is computed as the total number of word substitutions, deletions, and additions required to transform the reference transcript into the predicted
WORD ERROR RATE:
19

---

Published as a conference paper at ICLR 2025
Algorithm Utterance Agnostic Attack Algorithm
Require:
Speech data Aedev and, ASR model M and its loss function, Lu (e.g. CTCLoss, NLLLoss, etc.), allowed SNR s, learning rate, a, max epochs Cmars max iterations per sample imaz. target attack success rate, tar, target Character-Erre Rate (CER) (see B teer
procedure CER(a, b)
return EditDistance(a, b)/len(b) end procedure
and are character sequences
procedure SUCCESSRATE(2)
return Ezex ([CER(M(x++))) - > Lerr]
end procedure
procedure SNRTONORM(Z, SNR)
return 10
20 ||ا||
end procedure
de SNRTT x
while Success Rale < ler and e < Emaz do for (ar, y) dev do
is the audio, y is the transcript
+ 0
while C > forr and i < imax do asiin(V,n..s... |r|p LMs + u +
A, - A +
--it1
end while
U t clipe +
end for e e end while
20

---

Published as a conference paper at ICLR 2025
transcript. We remove all punctuation from both the reference and predicted transcripts, and convert to lower case before computing WER.
When WER is computed over multiple pairs of predicted and reference transcripts, it is common practice to sum the number of substitutions, deletions, and additions for all the pairs, and divide by the sum of the lengths of the reference transcripts. Formally, this can be written
WER (x, R) = 100 EEEX ED(M %
(4)
ΣrER |r]
where ED computes the edit distance.
The Character Error Rate (CER) can also be defined similarly, by using the character-level edit distance in the above equation.
For quantifying differences between (binary) genders, we measure the disparity in prediction accuracy across males and females by the Log WER Ratio (LWERR). Formally,
WERM(X, RRy)
LWERR logz
(5)
W WERM (Kms Rm)
where the (Xy, R,) and (Xm; Rm) represent the subsets utterances by females and males respec- tively.
SRB conduct fairness assessment based on robustness disparities across population subgroups (English vs. Spanish speech; male vs. female speakers). Most of the state of the art quantifies fairness in terms of predictive performance disparities. This occurs in the domain of fairness in ASR systems (Liu et al., 2022, Veliche &
FAIRNESS THROOGH ROBUSTNESS: In this work,
Fung 2023] Koenecke Koenekee al. 2020) and similar prevalence is found in other domains (Julia Angwin
Solans Noguero et al.) (023). However, previous work in the domain of ASR also considered
robustness disparities as an alternative fairness notion (Nanda al. (021).
Moreover, we argue that considering the dimension of robustness could give better sense of the expected disparities that could be observed when deployed in the wild, in the presence of diverse noisy conditions.
C MODELS
Table providde a summary of the models used in our evaluations. The model names correspond to the names of their pretrained checkpoints in the Huggingface library co / ///meoelss) s)). The abbreviations of these names are in the parentheses after them. Some of the unilingual models are pre-trained on multilingual data but are fine-tuned on only one language and thus can not transcribe any other language. Multilingual models have been pre-trained and fine-tuned on multiple languages so the same DNN can transeribe several languages. The WER of multilingual models is presented as English/Spanish.
ce
D FINE GRAINED ANALYSES
The following figures present fine-graine fine-grained analyses of robustness. These figures may be referenced by the main text but an overview of the accuracy and robustness of the various models. Figures [) and I0 present the breakdown by perturbation of the robustness models on English and Spanish, respectively. Figure presents breakdown of robustness by severity.
included in the main body in the interest of space. Figure 8 provides
COMPUTE RESOURCES
The experiments were performed on the Bridges-2 cluster at the Pittsburgh Supercomputing Center. This cluster contains 200 32G and 16G Nvidia V-100, which used for these experiments.
21

---

Published as a conference paper at ICLR 2025
Lang
Model
Abrv.
Params (M)
Data (hrs)
WER
canary- (VIIDIA)
enry-lb
1,000.0
85,000
6.0
parakect-etc-O.6b (
prkt-cte-0.6b
600.0
64,000.0
6.1
parakcet-ctc-I.Ib )
prkt-cte-1.Ib
1,100.0
64,000.0
6.0
parakeet-mntt-..6 (VVIDIA
prkt-rant-0.6b
600.0
64,000.0
6.0
parakeet-rmnt-l.bb (NVIDIA)
prkt-rnnt-1.1b
1,100.0
64,000.0
5,9
decpsppeech(Ammode et al. 2016
86.0
960
26.5
hubert-large-11960-ti///116 Hsu et al. 2021a
hubt-lg
317.0
60,000
8.4
hubert-xlarge-Is9900 Hsu al. 2021a)
hubt-xl
964.0
60,000
8.4
mms-Ib-fil02 (Pratap ct 2023)
mms-lb
964.6
55,000
22.8
EN
speecht5 asr(Aö ct al.
2022)
spch-t5
154.6
960
22.1
wav2vec2-base--66h Baevski al.
2020
w2v2-bs
95.0
960
11.3
way2vac2-large-960h Baevski et al.
2020)
w2v2-1g
317.0
960
9.7
(Xu el
2021
w2v2-Ig-slf
317.0
60,000
7.7
wav2xec2-innee--buusttiiiii---
(Hsu et al. 2021b
w2v2-Ig-rob
317.0
63,000
8.9
whisper-base (Radford et al. 2023)
wsp-bs
74.0
680,000
9.6
whisper-base.en (Radtord ct al. 2023
wsp-bs.en
74.0
563,000
5.1
whisper-large-v2 [Radtord et 2023
wsp-lg
1,550.0
680,000
8.0
whisper-medium (Radford et al. 2023
wsp-md
769.0
680,000
7.9
whisper-medium..co (Radford et 2023
wsp-md.en
769.0
563,000
4.1
whisper-small (Radford ct al. 202.3
wsp-sm
244.0
680,000
8.3
whisper-small.cn (Radlord el al. 2023
wsp-sm.en
244.0
563,000
4.0
whisper-tiny (Radford et al. 2025
wsp-tn
39.0
680,000
11.3
whisper-tiny.en (Radford et al. 2023
wsp-tn.en
39.0
563,000
10.1
canary-1b (NVIDIA)
cnry-lb
1,000,0
85,000
3.2
mms-Ib-fll02 (Pratap et al. 2023)
mms-lb
964.6
55,000
15.7
(Wagg al. 202
w2v2-bs-es
94.4
10,116
25.7
ES
(Conneau et al.
2020)
w2v2-Ig-es
315.4
54,350
6.8
whisper-base (Radford et al. 2023)
wsp-bs
74.0
680,000
14.8
whisper-large-v2 (Radford el al.
2023
wsp-lg
1,550.0
680,000
5.8
whisper-tiny (Radford et al. 2023)
wsp-tn
39.0
680,000
23.3
Table 5: Models used in our evaluations.
Dataset
License
LibriSpeech
CC-BY-4.0
Multilingual Librispeech
CC BY 4.0
TEDLIUM
CC-BY-NC-ND 3.0
AMI Common Voice CHiME
CC-BY-4.0
CCO-1.0
CC BY-SA 4.0
Table 6: Licenses of each of the considered datasets in SRB
F DATASET LICENSES
The licenses of each of the considered datasets described in Table 0
22

---

Published as a conference paper at ICLR 2025
LbrfSpeech
audo procesoing
nolo S0c al (FF)
UberSpacc
100
soc of (NF)
WERR
spotal acoustics
special effects
speech variution
white no
(a)
utterance agraste
utterance specific
7.00
1775
prosessing
250
1.301
200
13N
offects
88
0.50
10
WSg bs
WBR bs
100
200
(d)
(e)
Figure The accuracy and robustness of English (top) and Spanish (bottom) ASR models on clean and perturbed data. Accuracy is measured by WER of the models clean speech (a & d). Robustness is measured by the NWERD on non-adversarially perturbed speech (b & e) and WERD adversarially perturbed speech (c & f). The markers indicate the dataset in (a & c), and the perturbation category in (b & e). The x axes are in ascending order of the values on the y axes.
calegory model (abre)
clem WER
audio proe
(white)
social
syoth speach
AVG
lang disaset
NWERD
WERD
anry-1b
du MLS DU
wap-lg
7.3
14.3
2.6
5.5
4.1
5.3
6.5
mms-lb
14.6
21.6
8.1
12.0
14.3
18.9
15.9
prst-mut-1.1b
5.1
ony--b
1.7
12.3
3.0
2.7
12.7
9.3
4.2
7,4
7.7
61.7
34.7
piit-met-01.6
1.5
11.2
3.4
3.0
6.0
94
38
6.1
w2x2-Ig-4tt
55
prtt-te-n-b
7.
pitt-mte1.Ilb
2.0
6.2
3.0
33
6.4
9.1
44
5,4
hube-d
2.1
18.2
11.1
1633
13.3
14.7
5.5
13.2
26.7
$4,4
40,5
9.4
74
w2v2-lg-obb
2.6
18.1
102
15.1
1344
169
59
13.6
45.8
61.0
53.4
sheenfmr-rnnt
2.7
15.7
17.2
9.6
204
84
13.5
wZv2-Ig
86
LibriSpeech
w2-2-hh
18.9
364
20.3
96
29.5
41.5
wog-md
3.9
19.1
2.4
33
2.1
38
53
6.0
1.7
$1.9
26.8
wop-lg
3.9
6.9
2.0
35
16
30
4.0
3.5
3.4
43.6
23.5
39
wop-md.an
09
wap-sm
4.5
27.9
40
54
45
6.5
7.0
9.2
8.5
71.1
39.7
Seg-bb.er
5.1
12.4
17.9
11.2
8.1
wap-hs
8.5
wap- in on
6.4
$7.2
86
258
92
22,0
90.2
49.8
spch-85
7.2
32,6
49
263
27.3
67.9
64.3
44.7
$2.4
1064
79.4
sherden
7.2
11.5
$3.9
22.7
wap
15.1
28.9
62.9
mb
15.4
16.3
S.5
70
11.1
11.9
89
10.1
6.2
44.6
25.4
enry-
10.2
15.3
3.1
29
11.7
15
7.5
21.9
21.9
wep-mi
4.3
03
wop lg
12.1
12.5
25
26
2.
40
044
4,0
63.5
36.3
wop sm
12.3
32.0
2.3
49
5.1
56
166
8.6
2.2
69.8
36.0
wop-hs
15.3
67.6
S
88
19.2
KT
4.1
19.0
80.3
41.0
w2x2-Ig
99
wop-in.en
50.9
72
29.3
2
51.7
71,2
TEDLIUM
hube lg
wop-tn
14.4
14.7
59.7
11.5
12.1
ISA
29.9
17.5
162
4.3
9.1
22,9 1.3
13.9
110.9
7.3
$6.1
hube-sl
w2v2 Ig rob
15.2
24.4
8.7
157
18.7
19.4
92
16.0
17,9
9.4
w2v2
16.5
31.2
12.9
24,7
18.8
25.5
92
20.4
4.2
17.9
11.0
whv2-hw
18.8
37.0
163
300
23.3
32.6
95
24,8
7.1
19.9
13.5
spch-45
48.
31.8
150
27.5
48.1
21.7
31.8
90.6
61.2
ds
38.0
40.1
161
243
30,0
48.7
73
27.8
15.3
$1.5
33,4
31.7
139
22.8
23

---

Published as a conference paper at ICLR 2025
hute-ig
hube
5L1
27.6
39.8
mms- lb
71.
55.2
63.2
prit-etc-..68
prat
prat mitt-mat-- 0.65
37.0
16.7
26.9
prit mant-1.b
35.9
170
26.4
enfmr
35.4
sb-cofut--rott
sbordia
91.2
91.2
spch 45
304.3
221.9
263.1
w2v2--bs
w2x2--gg
w2v2 Ig rob
51.1
24.7
37,9
w2v2-1g-4tr
50.6
26.2
38.4
wop-lg
29.3
wop-md
29.2
15.8
22.5
wsp
31.4
17.0
24.2
wap-tn
wsp-t.n
19.5
cary-lb
33.6
28.8
42.2
90.6
$4.0
87.3
mms-1b
126.7
95.8
112.7
páit-etc-O 0.00
36.6
27.4
42.9
pfkt-man--1.6h
priz- mat---l
35.9
27.3
41.5
<b-cnfir
92.5
74.3
83.3
sh-cnffur-rnt
spch-a5
527.7
388.7
458.2
W2V22-bs
88.7
64.6
76.7
wov2-1g
wisi2-lg-roh
w2v2-1g-sir
81.6
52.0
66.8
wog--i
83.4
35.5
59.4
wop-lg
wop-md
22.8
wop-um
59.3
24.9
42.1
98.6
46.3
72.4
wop-tn
onry-Ib
93.4
93.4
hubt-lg
14.1
14.1
lubr-al
mmss1b
pitt-tt--.6b
5.8
5.8
pktt-te-1.Il
5.6
5.5
prlt-mat1..1b
3.8
sh-cnfur
20.3
20.3
19.1
spchaa5
wOv be
25.4
25.4
w3v2-1g
20.0
20.0
w2v2-1p Ig
w2x2-1p-slf
13.5
wog-bs
10.0
10.0
lg
3.7
3.7
wap
wop-sm
wap in
13.9
13.9
12.7
12.7
onr--Ib
wap Ig
5.8
74
4,8
20
6.7
13.7
65.0
39,4
w2x2-Ig-es
6.8
21.2
14.7
22.1
19.0
26.3
20.6
33.9
71.0
52.4
MLS-ES
wap
22.7
27.3
32.5
38.3
19.5
159.5
89.5
wep-in
124.3
45.0
$2.3
$7.9
41.8
43.1
156.5
w2v2 bo
25.7
32.4
20.4
24.2
24.1
32.4
26.7
10.0
33.8
21.9
lb
205.0
205.0
w2v2 bo
13.3
CVsss
w2v2-1g
9.3
9.3
wop lg
wop-tt in
30.6
30.5
lb
6.1
15.2
52
73
13.0
1000
10.1
MI.S-FR
wap
7.7
15.5
3.5
83
5.6
5.7
7.7
wap bs
25.0
124.8
16.7
64.4
47.9
25.9
51.9
Table 7: Accuracy and robustness of ASR models on all datasets
24

---

Published as a conference paper at ICLR 2025
sugmentation Pgg
sugmentacion
sugmentation gain
augmentation Highpass
augmentation lowpass
25
0.4
2.-
1.--
100
0.3
0.
066
augmentation resomple
ougmentarion crosstalk
2ygmensaton www.notecugmeettaaoo www_nelse_ssoeeeeeeeeee retbo_csttgmeetatton ewy_netsc_miin muson
0.33
0.30
0.3
0.15
02
0.15
0.50
010
0.25
0.05
0.
0.00
0.0
0.00
0.00
augmentation enyy nolse wham
outientation music
augmentatir tationtation-n bw
ougmentation Ew.ff_cm
sugmentat Be
0.25
2.5
0.5--
0.2
0.30
1.0
0.36
0.5
0:30
*.smenttation
1.0
0.8
0.8
0.15
0.4
0.10
0.7
0.7
0.05
a p.gmentstion chons
Nug sigmentation phaser
augmentst phch_down
augmenttt on phch.pp
sugmentatiion sindoom
0.5
0.4
0.5
0.6
0.4
04
0.1
0.2
0.1
02
0.2
00
0.
0.0
0.0
3.0
augmentation speecup
a.gmentation tempoodoon
sugmenncttio terrcoo
augmencation treble
augmentation tremooo
0.05
0.20
0.04
0.15
0.00
0.03
0.00
0.22
0.04
0.
0.0
0.00
aspnettattion
axgmentaation convensng w
0.30
(abrr)
(abry)
0.20
0.4
0.2
0.05
model (abry)
model (abex)
midell tabry)
Figure 9: NWERD of English models on different augmentations, averaged over all severities.
25

---

Published as a conference paper at ICLR 2025
sugmentatton augmentation uriversa Ladv
augmentation
0.5
0.33
0.1
aigmentatinn
Augnertation
1.--
30
1.0
12
0.6
0.8
15-
0.6-
0.4
33
02
8:55
augmentation my.nooee_wham sugmentation musi
sugmentation etho
sugmentation
sugmentation bass
0.200
2,5
8:71
8.55
8.100
8127
1.5
D.C
8.10
0.100
0.022
1.
0.15
0.4
8.000
011
0.023
0.0
0.30
0.000
0.0
0.00
0.0
ougientaation choras
sugmentatin phaser
augmentation ptth down
sugmentarion ptthhup
sugmentation on slowdown
3.4
2.0
2.8
15
3.5
2-
1.0
044
0.0
(.00
ajsmentationn
sugmentztion
2.0
8.12
0:25
0.05
0.00
6:20
0:55
0.00
0.02
s.04
40
22
sugmentation gnotsc
model (abrv)
labn))
model (abiv)
modal (oorv)
1200
model (abev)
Figure 10: NWERD of Spanish models on different augmentations, averaged over all severities.
severity
severity
severity -
severity
severity
2.5
0.5
0.6
0.30
0.25
0.6
0.5
1.2
1.0-
WWERD
2.0
0.4
6.8
2.5
0.15
0.3
2.0
0.10
0.5
0.1
0.2
0.0
0.0
0.00
0.0
0.0
model (sbrr)
modal (obr)
madel (bDV)
made (abri
model (abrv)
Figure 11: NWERD on English data as the severity of the augmentation is increased.
26

---

Published as a conference paper at ICLR 2025
51300
6125
5.00
0,000
0.00
io mo
ap
mocdel (alre)
model (abrv)
(b)
Figure 12: Log WER Ratio between male and female speakers from Librispeech (English) (a) and Spanish Multilingual Librispeech (b).
27


Reviews:
[{"review_id": "wlLgldCrMB", "reviewer": "Reviewer_4pNh", "summary": "The authors propose a comprehensive benchmark for automatic speech recognition (ASR) models. The benchmark aims to cover a large variety of real-world data-quality challenges for deployed ASR models.\n\nTo benchmark on clean speech, the authors use the Librispeech and TEDLIUM datasets.  \nTo benchmark on inter-personal speech, the authors use the CHiME-6 dataset.  \nTo benchmark on far-field speech, the authors use the AMI dataset.  \nTo benchmark on accented English, the authors use the Mozilla CommonVoice dataset.  \nTo benchmark on text-to-speech audio, the authors generate data using the model XTTS.  \nTo benchmark against adversarial attacks, the authors use utterance-specific and utterance-agnostic attacks.  \nTo benchmark against environmental effects, the authors add 1) white noise, 2) environmental noise, or 3) simulate echo and RIR.  \nTo benchmark against digital augmentations, the authors modify the audio data using most effects available in SoX.\n\nThe authors use the word-error-rate (WER) as the metric for clean speech. The metric for adversarial attacks is WER degredation (WERD). For all others, the authors use difficulty-normalized WER or WERD. The normalization is done based on a weight, which is an approximated speech quality given by a neural network.    \n\nThe authors apply their benchmark on popular ASR models and their variants, including Whisper, wav2vec 2.0, HuBERT, and Canary.\nThe following observations are made:\n- Canary has the best performance on clean speech\n- Whisper is robust against digital augmentations, echo and RIR, while Canary is not.\n- Wav2vec 2.0 is robust against adversarial attacks. \n- There is a positive correlation between average robustness and amount of model parameters\n\nFurthermore, the authors did an analysis on Spanish datasets with models fine-tuned (single-or multi-lingually) on Spanish. They find that multi-lingual models are more robust on English than Spanish.\n\nFinally, the authors analyzed performance differences between male and female speakers. They observe it is data dependent.", "questions": "1. Can the authors clarify, for each domain, what dataset(s) are used, specify their statistic (min/max/avg length, male/female, total number of utterances), exactly how a model is evaluated (WER, WERD, NWERD with speech quality weights from...) , and how the data is perturbed, if applicable?\n2. Can the authors clarify whether Spanish data is included in the benchmark, or whether this is a separate analysis?\n3. Can the authors clarify how weights for the DeepSpeech were obtained?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "### originality \n\nThis work is combines multiple existing datasets to build a comprehensive analysis tool for ASR models. The authors place their work fairly into existing literature.\n\n### quality \n\nThe writing is mostly error-free and clear to understand. The authors analyzed relevant, contemporary ASR models. Nice-to-have but not required models to include could've been WavLM and, perhaps, OWSM.        \n\n### clarity \n\nThe authors use a familiar section lay-out for their paper. I also appreciate the take-away summaries in Section 4.    \n\n### significance\n\nI think this work can contribute to a better evaluation of ASR models. It is significant to the speech community to make it is easy for researchers to evaluate on data other than Librispeech test-clean and test-other.", "weaknesses": "### Benchmark protocol is unclear\n\nSection 3 is missing important details for reproducing this work. I think practitioners cannot replicate the benchmark protocol by reading the paper as-is. It is critical for the usefulness of a benchmark that it can be independently reproduced. I have the following remarks:\n\n1. I find the usage of the word 'Perturbations' unclear. I see how adversarial attacks, environmental effects, and digital augmentations can be classified as perturbations of the audio signal. However, I do not see how accented speech, computer generated speech, and inter-personal communication, can be seen as a perturbation. I would prefer the usage of \"domain\" over \"perturbation\". My suggestion would be to change the taxonomy (Figure 2) to include the domains clean/professional, social gathering, accented, text-to-speech, noise (with subgroups environment, digital augmentation), and adversarial.    \n2.  It follows from 1 that is unclear to me exactly which dataset is used in which setting of the benchmark. I made a best guess in the summary above. For example, is environmental noise, or adversarial attacks, used only on Librispeech and TEDLIUM? Moreover, which split of Librispeech should be used? Only test-clean, or also test-other? Another example, which version of Mozilla CommonVoice is used? Do you filter out any data from CommonVoice if it is not labeled as 'accented'? I think the paper would be greatly improved by having a subsubsection for each domain/perturbation in Section 3.2, which list all details needed to generate the (exact) test set(s).  \n3. I praise the authors for sharing an (omited) link to perturbed versions of the datasets. I think this benchmark should require to use these exact perturbed versions in the noise domain, otherwise comparisons are unfair. The paper currently does not touch upon this topic.\n4. Section 3.2 does not include any information on the generation of adversarial perturbations. These details should be in the main text of the paper.\n5. There is no information on which text is used to generate the text-to-speech audio. Moreover, it would be desirable for these generated audio files to be shared so everyone can test on the same data. \n6. Section 3.1 does not mention any Spanish datasets. Moreover, it is missing datasets for the noise domain (WHAM!, MUSAN, etc)\n7. It is unclear to me whether this benchmark includes Spanish data, or whether this is simply an extra analysis from the authors. \n8. For reproducing the benchmark, it is required to have the speech perception weights of each utterance to calculate the NWER(D). \n\n### Confusing details in analysis\n\nSection 4.1 mentions the use of DeepSpeech (which does not have open weights?) and speech-t5, these are not used in Table 1, only Figure 7. Moreover, Section 4.4 mentions the use of 4 more models, it would be more clear to include these in Section 4.1. \n\nIt is unclear when WER or WERD should be used. From Table 1 I assume that you use NWERD for e.g., accent and text-to-speech data. I don't see how one can have an $X$ and $X_p$ audio sequence to calculate the WERD in these scenarios. \n\n### Figures and tables are difficult to read\n\nI find Figure 4, 5 and 7 too small to read. As the page limit is 10, there should be enough room to increase the size without exceeding the page limit.  \n\n### Editorial remarks\n\n* I would advice the authors to use \\citep instead of \\cite. Currently, the reference style (other than in section 2.2) makes the text hard to parse. \n* Figure 3's space exceeds the bottom margin, and indents the texts on page 7 unnecessarily. \n* Table 1 and Table 2 have a different format. I would recommend the use of the booktabs package and no usage of vertical lines. \n* Appendix A is empty.\n* Table 4 and Table 6 exceeds the page margin. \n* The citation of Canary (Elena Rastogueva) is incorrectly formatted, is missing data, and exceeds the page margin. \n\n#### Typos\n- ln 41: is comphe.. -> are comphre...\n- ln 107:  them -> those\n- ln 161:  \"others\" does not fit here (Radford et al is in both)\n- ln 346:  than -> compared to\n- ln 472 (and others): the term accuracy is specific to classification, 'quality' is more appropriate to describe ASR predictions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9COYHwwuOV", "reviewer": "Reviewer_UsXF", "summary": "This paper proposes a new benchmark data and pipeline to measure the ASR robustness under different perturbations. The proposed work has 4 components: 1. clean datasets and noise sources; 2. bank of perturbations; 3.ASR transcription extractions; 4. metric computations. Compared to existing benchmark pipeline, the authors claimed that “the proposed pipeline combines several benchmark datasets containing known noise sources to allow researchers to comprehensively evaluate the robustness of their models using a single benchmark”. During experiments, the authors evaluated a couple state-of-the-art ASR systems on the proposed benchmark, and conduct a series of analysis on robustness.", "questions": "See Weaknesses section", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The experiments are comprehensive. It considers a couple state-of-the-art ASR systems, and it analyzes adversarial/non-adversarial perturbations.\n* It analyzed the correlation between different models in terms of robustness.  \n* It analyzed the robustness of speakers from different sub-groups (accent, gender).", "weaknesses": "Although the analyzes are very comprehensive, the contributions/novelty are not clearly stated. According to the quoted claim above, it feels like the most significant contribution compared to prior work is having more noise sources? Please explain more about the contributions/novelty here. It would be helpful to have a table to compare between the proposed approaches with baselines on the diffs from the 4 modules.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4S7jddCvEZ", "reviewer": "Reviewer_R2oE", "summary": "This paper proposes SRB, a benchmark to evaluate the robustness of Automatic Speech Recognition (ASR) models. Extensive evaluations are conducted on several recent and popular ASR DNNs to analyze their robustness to non-adversarial and adversarial input perturbations,  and various sub-groups, namely English speech and non-English (Spanish) speech, and male and female speakers. From the above analysis, several conclusions are drawn.", "questions": "Listed in the weaknesses part", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper conducts extensive and detailed analysis to evaluate the robustness of recent ASR models. The Takeaways help develop robust ASR models.", "weaknesses": "1. While I appreciate the efforts to evaluate the robustness of recent ASR models, the paper offers limited novelty. I suggest that the authors compare their benchmarks to existing robustness metrics or frameworks in the ASR field. \n2. To evaluate the robustness of ASR models, the current framework adds noise to clean speech data to evaluate the performance. How do recent ASR models perform in real-world noisy data? In other words, how can we evaluate the robustness of ASR models to real-world noisy data?\n3. The paper lacks a discussion on strategies for improving robustness. It would be great if the authors could provide some sights or suggestions to improve the robustness.\n2. The experimental sections of the paper need reorganization. I can not see a clear logic for the experimental parts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cz1ZzbhpId", "reviewer": "Reviewer_Dmh5", "summary": "This paper proposes Speech Robust Bench, a benchmark framework for testing speech recognition models . It collects almost all speech variation techniques available in the field, with close connection with real-time application scenarios. Results on", "questions": "The reviewer does not have explicit further questions apart from above commetns.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The dataset description is very detailed and the related pipeline has been open-sourced, which is a decent contribution to the speech community.\n2. The models covered in the field has been quite at the level of state-of-the-art.\n3. The paper has answer to the critical question - \"does the improvement come from additional variants from data or simply just MORE data?\" in section 4.4, which is rare and good.", "weaknesses": "1. The reviewers think the presentation of the paper is good, but has room for improvement. Some of the extensive statistics and information (e.g. about attacks) can be enlisted or illustrated in the main text, rather than extensive text description in both main text and appendix; Some resulting figures (e.g. Figure 5) are hard to read; Also there are minor formatting issues.\n2. One limitation of this work is lacking comparison or having initial attempts with some hybrid audio processing pipeline, especially based on its growing popularity on speech processing fields. In fact, applying noise and perturbation has been a long-time practice since Kaldi era.\n3. It would be good if the authors can correspond the related earlier works onto the methods enlisted and highlight the differences and potential variants the authors covered.\n4. The authors highlighted in the paper about multi-lingual scenario but in fact, only English and Spanish are covered, and the models are mostly mono-lingual. The authors may either remove such statement or want to conduct additional experiments using other languages and multi-lingual models for ASR (e.g. Google T5). This is not major problem though.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)