RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Robust Automatic Speech Recognition
Over the years, several techniques have been proposed for making Automatic Speech Recognition (ASR) models robust to input perturbations, such as noise and other signal corruptions (Li et al., 2014 ) . We can divide these techniques into two high-level categories: i) model-based and ii) feature-based. i) Model-based techniques modify the models to make them more robust. Examples of such approaches include adapting pre-trained models (Yu et al., 2009 ; Juang & Rahim, 1996 ) , denoising the audio before processing (Mohammadiha et al., 2013 ; Wilson et al., 2008 ) , and training ASR models on noisy data (Likhomanenko et al., 2020 ) . Since model based strategies generally require access to noisy data (Li et al., 2014 ) , they are most effective if the sources of noise, and/or the exact environment in which the ASR model will be deployed in are known, and one can gather data to represent them. ii) Feature-based approaches, on the other hand, involve developing handcrafted features that are invariant to noise and corruptions in the signal (Li et al., 2014 ) . Several of these features are inspired by biological audition (Kim & Stern, 2016 ; Hermansky et al., 1991 ; Hermansky & Sharma, 1998 ) , while others use signal processing techniques (Li et al., 2014 ) . Generally, these methods are designed to extract the components of the audio signal salient for speech production and perception, while discarding irrelevant components (Stern & Morgan, 2012 ) . Consequently, they do not require precise knowledge of the environment and noise distributions. Recently, however, handcrafted features have fallen out of favor, and have been replaced by features learned via end-to-end training of deep learning models on large amounts of data (Baevski et al., 2020 ; Hsu et al., 2021a ; Likhomanenko et al., 2020 ; Radford et al., 2023 ) . Proponents of these techniques posit that models trained on larger datasets become more robust. Our evaluations in § 4 reveal that there are several input perturbations against which smaller models trained on less data outperform larger models trained on more data.
Adversarial Robustness
Adversarial perturbations are perturbations that can change the response of a model when added to their inputs, but are either imperceptible to humans or perceptually and semantically irrelevant enough to be ignored by them (Szegedy et al., 2014 ; Goodfellow et al., 2014 ) . Adversarially perturbed inputs are known as adversarial attacks . They can be targeted (aiming to change a prediction to a specific incorrect class), or un-targeted (aiming to change a prediction to any incorrect class, Akhtar et al. 2021 ). The design of adversarial attacks is determined by the level of knowledge the attacker is assumed to have about the target model. Attacks that assume full knowledge of the target model’s architecture and weights ( white-box threat model) often use gradient-based optimization techniques (Szegedy et al., 2014 ; Goodfellow et al., 2014 ; Madry et al., 2018 ; Laidlaw et al., 2021 ; Akhtar et al., 2021 ) .
Attackers who do not have any knowledge of the target models architecture and only have query access to it ( black-box threat model) typically use gradient-free optimization methods (Wang et al., 2022a ; Andriushchenko et al., 2020 ; Wicker et al., 2018 ; Chen et al., 2017 ; Zhao et al., 2020 ; Vo et al., 2022 ) . An intriguing property of adversarial perturbations is that they transfer between models (Papernot et al., 2016 ) , and inputs (Akhtar et al., 2021 ; Neekhara et al., 2019 ) . Our SRB includes two types of white box adversarial attacks: those that generate perturbations specific to each input (Madry et al., 2018 ) , and those that generate perturbations that cause models to mis-transcribe multiple inputs (Neekhara et al., 2019 ) .
Robustness Benchmarks
Robustness benchmarks have unified robustness evaluations and enabled fair comparisons between various models and robustness enhancing techniques in several domains, including vision, Natural Language Processing (NLP) and ASR.
In the domain of vision, Croce et al. ( 2020 ) propose an adversarial robustness benchmark and leaderboard based on AutoAttack (Croce & Hein, 2020 ) , while Hendrycks & Dietterich ( 2019 ) and Hendrycks et al. ( 2021a ) propose benchmarks and metrics for measuring the robustness of image recognition models to non-adversarial perturbations. In the domain of NLP, Wang et al. ( 2021a ) propose a benchmark to evaluate models under various lexical and semantic perturbations, such as typos, distractors, and word replacements. However, in the domain of ASR, there is a lack of similar comprehensive benchmarks that measure the robustness of ASR models on diverse corruptions. The current benchmarks are often specialized to one (or few) types of perturbations such as reverberation (Nakamura et al., 2000 ; Kinoshita et al., 2013 ; Jeub et al., 2009 ) , environmental noise (Barker et al., 2018 ; Piczak, 2015 ) , and accented speech (Lander, 2022 ; Shi et al., 2021 ) . While there is some initial work in developing more comprehensive benchmarks (Hirsch & Pearce, 2000 ) , it is limited to relatively simple data consisting of spoken sequences of digits, which lack the complexity of long sentences that modern ASR models are expected to transcribe.
Some recent works (Radford et al., 2023 ; Likhomanenko et al., 2020 ) have evaluated the robustness of ASR models by computing transcription accuracy on several speech datasets. We find that this evaluation method is too coarse to pinpoint the strengths and weaknesses of the models; for example, Likhomanenko et al. ( 2020 ) present the word error rates for three settings: clean, noisy, and extreme, for each dataset, from which it is difficult to determine the kinds of noise or distortions the model has difficulties with. Furthermore, most robustness evaluations of ASR models do not consider adversarial robustness, and thus neglect to highlight potential security vulnerabilities of the models.
Overview
At a high level our benchmark consists of two components: (1) a bank of perturbations that are representative of various scenarios that the model could encounter when deployed in the wild (§ 3.3 ), and (2) a set of metrics for quantifying the robustness of the ASR models and facilitating comparisons among them (§ 3.5 ). Given an ASR model (§ 3.4 ) to be evaluated and dataset consisting of audio utterances and reference transcripts (§ 3.2 ), the procedure for evaluating the robustness of the target ASR model using our benchmark involves the following three steps, as illustrated in Figure 1 . First, the input audio is perturbed using a comprehensive bank of perturbations (§ 3.3 ). In the case of deterministic perturbations only a single perturbed audio is produced, while in the case of randomized perturbations we obtain several perturbed audios that are used to test the stability of the model’s prediction. Then the perturbed audio is transcribed by the target ASR model. Since some of our metrics are normalized by the error of a baseline ASR model (see § 3.5 ), transcripts from the baseline model are also obtained. Finally, the reference transcript and the predicted transcripts from the baseline and target ASR models are used to compute the metrics described in § 3.5 .
Dataset
We have designed SRB to be largely agnostic to the evaluation data to make it more broadly applicable across various speech recognition domains. In principle, SRB can be used with any dataset that contains utterances and reference transcripts, however, we recommend using datasets with high-quality clean audio and accurate transcripts so that pre-existing corruptions in the dataset do not confound the robustness metrics obtained from the benchmark. For this reason, our evaluation in § 4 uses clean speech from Librispeech (Panayotov et al., 2015 ) .
Perturbations
The taxonomy of the perturbations used in our benchmark is presented in Figure 2 . The perturbations are of two broad types: 1) non-adversarial and 2) adversarial.
The non-adversarial perturbations fall under one of three categories: 1.i) common corruptions, such as white noise, environmental noise, and room impulse response, that may be introduced by the environment or by other sources, 1.ii) semantically irrelevant perturbations such as special effects that may be encountered in digital media, and 1.iii) speaker attribute perturbation, such as voice/accent transformations.
The adversarial perturbations fall under two categories: specific and general. 2.i) General adversarial perturbations are designed to be agnostic to the utterance, to the model, or to both. 2.ii) Specific adversarial perturbations, on the other hand, are crafted to cause a specific model to mistranscribe a specific utterance, and, in general, they are not expected to be very effective on other models and utterances.

Each perturbation is applied at 4 levels of severity, which is modulated by adjusting the parameters and attributes of the perturbation. The perturbations and their parameters and attributes are discussed in detail in Appendix A .
Models and Transcription
Once perturbed, the utterances are transcribed by the (target) ASR model being evaluated, as well as a baseline model. The transcripts from the baseline model are needed to compute the metrics described in § 3.5 . In SRB we use Deepspeech (Amodei et al., 2016 ) as baseline. Indeed, in principle, any model other than the target model(s) can be used. We recommend using a model that does not exhibit counter-intuitive behavior, such as unusually high error rate on relatively mild corruptions or lower error rate on more severe corruptions, to make results easier to understand.
Metrics
Similar to (Hendrycks & Dietterich, 2019 ) , SRB measures robustness along two dimensions: utility and stability of the model’s prediction under input perturbations. SRB measures the utility of the model with the widely used Word Error Rate (WER, see § 3.5 ). When aggregating WER over multiple perturbations we follow the practice of (Hendrycks & Dietterich, 2019 ) and normalize the WER of the target model by the WER of a baseline model. Doing so penalizes errors on “easy” corruptions more than errors on “harder” corruptions. This normalized metric is called Normalized WER (NWER, see § 3.5 ). Meanwhile, SRB measures the prediction stability of the model by computing the variance in the WER caused by corrupting the signal with multiple corruption samples drawn from the same distribution. We call this metric WER Variance (WERV, see § 3.5 ).
. Word Error Rate (WER)
Following the common practice from ASR literature, the WER is computed as the word-level edit distance between the reference and the predicted transcripts, normalized by the length of the reference. The edit distance is computed as the total number of word substitutions, deletions, and additions required to transform the reference transcript into the predicted transcript.
When WER is computed over multiple pairs of predicted and reference transcripts, it is common practice to treat all the predicted transcripts as one long text segment \mathcal{X} , and likewise consider all the reference transcripts as a single long text segment \mathcal{R} . This means that the number of substitutions, deletions, and additions for all the pairs are summed, and divided by the sum of the lengths of the reference transcripts. Formally, this can be written as

 WER^{M}(\mathcal{X},\mathcal{R}):=100\frac{\sum_{x\in\mathcal{X},r\in\mathcal{R}}ED(M(x),r)}{\sum_{r\in\mathcal{R}}|r|}, (1)

where ED computes the edit distance.
. Normalized Word Error Rate (NWER)
Since our benchmark consists of several corruptions, each at several levels of severity, it is intractable to compare the WER of each model for each corruption-severity pair. We therefore require an aggregate metric to summarize the WER for all the pairs. Since all the corruption-severity pairs do not present the same level of difficulty, simply summing or averaging the WERs over all the pairs will not present an accurate picture of the model’s robustness. Ideally, we should penalize errors on easier corruption-severity pairs more than errors on harder pairs.

To incorporate the difficulty of the various corruption-severity pairs, we follow the approach of (Hendrycks & Dietterich, 2019 ) , and compute the Normalized Word Error Rate (NWER). NWER is computed as a weighted sum of the WERs corresponding to the various corruption-severity pairs, where the weight is the inverse of the difficulty of the corruption-severity pairs. Following (Hendrycks & Dietterich, 2019 ) , we use the WER of a baseline model ( M_{\text{base}} ) as an estimate of the difficulty. This way errors on easier corruption-severity pairs are penalized more heavily than errors on more difficult corruption-severity pairs, when the difficulty is measured by the baseline model performance. Thus, NWER can be formally defined as

 NWER^{M}_{c,s}(\mathcal{X},\mathcal{R}):=\frac{WER^{M}_{c,s}(\mathcal{X},\mathcal{R})}{WER^{M_{base}}_{c,s}(\mathcal{X},\mathcal{R})}, (2)

where WER^{M}_{c,s} and WER^{M_{\text{base}}}_{c,s} are the WERs of target and baseline models under corruption c at severity s .
. Word Error Rate Variance (WERV)
To evaluate the stability of a target model’s predictions with corrupted inputs, we compute the variance over the model’s predictions on repeated random noise sampled from a fixed distribution. We differ from Hendrycks & Dietterich ( 2019 ) who use the co-called flip rate, since their metric is only compatible with class predictions, not with sequence predictions as we have in ASR.
Instead, we propose to compute the variance in the WER over repeated random samples of corruption. Similar in aim to (Hendrycks & Dietterich, 2019 ) , this yields a metric that measures how much the model’s predictions fluctuate in presence of noise.

Concretely, given a set of utterances \mathcal{X}=\{x_{1},...,x_{N}\} , we corrupt each utterance, x_{i} , with K random corruption samples to obtain \tilde{x}_{i}^{(1)},...,\tilde{x}_{i}^{(K)} . We organize the corrupted samples into K datasets, \tilde{\mathcal{X}}^{(1)},...,\tilde{\mathcal{X}}{(K)} , where {\tilde{\mathcal{X}}^{(k)}=\{x_{1}^{(k)},...,x_{N}^{(k)}\}} .
WERV can now be computed as:

WERV(\tilde{\mathcal{X}}^{(1:k)},\mathcal{R}):=Var_{k:1\rightarrow K}\left[WER(\tilde{\mathcal{X}}^{(k)},r_{i})\right]\vspace{-8mm} (3)

where r_{i} is the reference transcript for utterance x_{i} .
Evaluation
We evaluate several recent and popular ASR DNNs (see Table 1 ) using SRB and analyze the results to uncover fine-grained differences in their robustness to non-adversarial and adversarial input perturbations. Our analysis is two-pronged. First, we focus on the primary goal of SRB , which is to measure the robustness of the models to various input perturbations. As a case study, we use SRB to evaluate the prediction accuracy and stability of the English ASR models as different perturbations are introduced. As mentioned in § 3.2 , we use English speech from Librispeech (Panayotov et al., 2015 ) . We extend our analysis by studying the relationship between robustness and various model characteristics, such as the number of parameters and size of the training data. Having considered robustness averaged over the entire population (represented in the dataset), we extend our analysis to the robustness of the models for various population sub-groups, namely English speech, non-English (Spanish) speech, male speakers, and female speakers. Prior works (Liu et al., 2022 ; Veliche & Fung, 2023 ) observe that there is disparity in transcription accuracy between subgroups.
Our analysis augments these observations by showing that inter-group disparities in robustness may also exist, and thus demonstrating the utility of SRB in the broader field of trustworthy AI. The Spanish speech is obtained from the Spanish test set in Multi-Lingual Librispeech (Pratap et al., 2020 ) .
Robustness of English ASR Models
We evaluate the robustness of English DNNs from Table 1 on the non-adversarial and adversarial perturbations in SRB . Figures 3(a) and 3(b) present aggregate metrics showing overall robustness of the ASR models along the two dimensions of utility, which is measured by NWER, and stability, which is measured by WERV (see § 3.5 for details on metrics).
In terms of both utility and stability, we note that wsp-lg is the most robust model on average, followed by hubt-lg . This high-level analysis gives us an overview of the robustness of many popular state-of-the-art ASR models. Next, we conduct more fine-grained analyses to better determine under which conditions the models’ performance deteriorates.
. Robustness to Non-Adversarial Perturbations
We first consider non-adversarial perturbations. In the following, we measure the robustness of the candidate models under each perturbation and at each severity level to identify potential failure modes.
. Robustness to Adversarial Perturbations
We evaluate the English-only DNNs from Table 1 on the untargeted utterance-agnostic perturbations and untargeted specific perturbations. Figure 10 in Appendix B shows the WER of the DNNs on the two perturbations as well as their average. In terms of average WER w2v2-lg-slf emerges as the most robust model followed by wsp-lg . However, looking at the breakdown by perturbation types, we find that while wsp-lg is almost equally susceptible to general and specific perturbations, w2v2-lg-slf is much more vulnerable to specific perturbations, but significantly less vulnerable to general perturbations. Therefore, under black- or grey-box scenarios, w2v2-lg-slf may be a better choice, while under the white-box threat model wsp-lg may be more robust. Considering the three DNNs that are based on Wav2Vec-2 Large: hubt-lg , w2v2-lg-slf and w2v2-lg-rob , we note that hubt-lg achieves the lowest WER against specific adversarial perturbations, while w2v2-lg-slf achieves the lowest WER against general adversarial perturbations. Meanwhile w2v2-lg-rob performs similarly as w2v2-bs .
We note that: (1) the self-training used in w2v2-lg-slf may induce robustness to utterance-agnostic adversarial attacks, (2) the discrete representations of hubt-lg may induce robustness to specific adversarial attacks, and (3) diversifying training data has little to no effect on adversarial robustness.
Finally, wsp-tn.en is highly vulnerable to specific adversarial attacks.
Robustness vs. Model Attributes
To determine if the prevailing practice of training DNNs with more parameters on larger datasets is yielding improvements in robustness, we plot NWER against the number of model parameters and the size of the training data of all the candidate models. These plots are shown in Figures 4(a) and 4(b) . We note that increasing model size is correlated with improved robustness (lower NWER), however, the effect is more inconsistent and weaker for adversarial perturbations. To further isolate the impact of the model size we control the architecture and training data and plot the NWER of models from the same family in Figure 4(c) , which have similar architectures and training datasets. We note that larger models are more robust in the Whisper and Wav2Vec-2.0 families, but, surprisingly, not in the HuBert family. On the other hand, increasing training data appears to have only a minor influence on robustness.

Larger models tend to be more robust, while smaller models, even if they are trained on large datasets, are less robust.
Robustness for Population Sub-Groups
In the preceding analysis, we considered robustness aggregated over the entire population (i.e. dataset). However, populations are generally not homogeneous, and, thus, the robustness of the model may differ on various population sub-groups. Prior works have commonly analyzed sub-group fairness of ASR models by comparing the overall WER for each sub-group on a benchmark dataset (Koenecke et al., 2020 ) . It is possible that models that are fair on average , may not be fair under certain conditions. In the following, we use SRB to uncover and analyze the disparities in the models’ in robustness across four sub-groups: English and Spanish speakers, and males and females. We find that disparities indeed exist, with multi-lingual models generally being more robust for English than Spanish (Figure 5(b) ), and most models being less robust for females than males (Figure 12 ).
. Robustness on Non-English (Spanish) Speech
The overall NWER of the Spanish ASR models under non-adversarial perturbations is presented in Figure 5(a) , and the NWER for each perturbation is presented in Figure 11 in Appendix B . We observe that w2v2-lg-es is the most robust model, followed by wsp-lg and mms . It is interesting that despite having 33% more parameters and being trained on 10 \times more data, wsp-lg is outperformed by w2v2-lg-es , thus indicating that simply scaling the model and training data is not sufficient to achieve robustness, particularly in the multi-lingual setting . Extending this analysis further, we measure the degradation in WER of wsp-lg on each perturbation when transcribing English and Spanish, and plot the results in Figure 5(b) . We observe that while the degradation is greater for Spanish under all perturbations, wsp-lg struggles particularly on RIR.

wsp-lg is not the most robust model on Spanish, and struggles on simple perturbations like RIR.
. Robustness and Gender
To measure the disparity in prediction accuracy across genders (males/females), we compute the log of the ratio of the WERs of the ASR model on female and male speakers. We call this measure the Log WER Ratio (LWERR) and formally it can be written as

LWERR:=\log_{2}\frac{WER^{M}(\mathcal{X}_{f},\mathcal{R}_{f})}{WER^{M}(\mathcal{X}_{m},\mathcal{R}_{m})}, (4)

where the (\mathcal{X}_{f},\mathcal{R}_{f}) and (\mathcal{X}_{m},\mathcal{R}_{m}) represent the subsets of utterances by females and males respectively. A positive value of LWERR indicates that the model is biased against females and a negative value indicates that the model is biased against males.

The average LWERR values for English and non-English (Spanish) models are shown in Figure 12 and a breakdown by perturbations is shown in Figure 6 . We note that, on average, the models are biased against females, and the bias is exacerbated when transcribing Spanish. From Figure 6(a) we observe that on clean data the English models are generally weakly biased against females, with ds being the most biased and hubt-lg the least. wsp-lg is an exception in that it is slightly biased against males. The bias of the models diminishes significantly under environmental and Gaussian noise, but, interestingly it is exacerbated (against females) under room impulse response augmentation, and phasing. Perturbations that mask or shift the spectrogram, such as filtering, resampling, and pitch and speech modulation, have somewhat expected results – perturbations that shift increase the pitch or mask higher frequencies tend to increase WER for females, and vice-versa. Interestingly, we observe that utterance-agnostic adversarial attacks biased 4/7 models against female speakers, with wsp-lg exhibiting the most significant increase in bias.

The bias against female speakers is more pronounced in the Spanish ASR models. Unlike English models, the bias does not diminish under noise indicating that the Spanish models have considerably less robustness against noise for female speakers. The trends for the other utterances remain the same as the ones observed in English models, except the increase in bias due to the addition of room impulse response is much greater.

To summarize, using our benchmark to conduct a fairness analysis reveals that (1) certain perturbations, like room impulse response and adversarial attacks disproportionately degrade the performance of models for female speakers, and (2) the disparity across genders is greater in Spanish models and input perturbations tend to further exacerbate this disparity.