Introduction
Robustness to input perturbations is a highly desirable attribute in Machine Learning (ML) models. When deployed in the real world, ML models are likely to encounter noise and corruptions arising from a myriad of sources, including the environment, sensing apparatus, and even malicious actors. The ability of the models to counteract these sources of noise and continue to make accurate predictions has significant implications for their safety, security, and reliability.

As novel ML models continue to be developed and deployed at an ever-increasing rate, it has become all the more important to ensure that their robustness is well understood. To this end, prior works have developed robustness benchmarks that evaluate various aspects of a model’s performance under a variety of input perturbations. These benchmarks have proven to be invaluable to the advancement of research into more robust models because (1) they enable meaningful comparisons across existing and new models, which allows progress to be accurately tracked, and (2) make it easier for researchers to evaluate the robustness of their models, thereby reducing the barrier to entry into robustness research. While robustness benchmarks have been proposed in prior works for vision (Hendrycks & Dietterich, 2019 ; Hendrycks et al., 2021a , b ; Croce et al., 2020 ) and natural language processing (Wang et al., 2021a , 2022b ) tasks, the development of such benchmarks for Automatic Speech Recognition (ASR) models has received little attention beyond benchmarks based on simple digit sequence recognition (Hirsch & Pearce, 2000 ) .

In the absence of standardized benchmarks, prior works have tried to evaluate the robustness of ASR models in various different ways. For example, several works have used a combination of distortions from various datasets to perturb the input audio (Radford et al., 2023 ; Wen et al., 2016 ; Chen et al., 2022 ) , however, since the choice of dataset varies among studies, their results are not comparable. Other, more recent, works (Likhomanenko et al., 2020 ; Radford et al., 2023 ; Hsu et al., 2021b ) have evaluated the robustness of models by computing their transcription accuracy on multiple speech datasets that may have perturbations from the real world. Since the type of perturbations present in these datasets is not controlled and is unknown, this evaluation method does not inform about the
specific type of perturbations the models may be weak against.
Moreover, prior works rarely evaluate their proposed models against adversarial attacks (unless they are proposing defenses), and thus neglect to highlight potential security vulnerabilities of the models.

In this paper, we propose Speech Robust Bench (SRB) , a standardized robustness benchmark for ASR models. Following the design of successful robustness benchmarks for image recognition (Hendrycks & Dietterich, 2019 ; Croce et al., 2020 ) , SRB is composed of two parts: (1) a comprehensive bank of perturbations, and (2) a set of robustness metrics. The perturbations bank contains a comprehensive set of perturbations that represent common distortions arising from the environment or equipment, variations in speaker attributes, semantic preserving special effects found in digital media, and adversarial attacks. Meanwhile, the metrics we propose, following the methodology of (Hendrycks & Dietterich, 2019 ) , measure two aspects of robustness: the transcription accuracy of the models, as well as the stability of the predicted transcripts under randomized perturbations.

To highlight the need for and the benefits of doing systematic robustness assessment, we evaluate the robustness of several popular ASR models using SRB , and compare their robustness. We observe that while Whisper (Radford et al., 2023 ) is the most robust on average among the models we tested, it is outperformed by other, smaller, models on several perturbations. Further analyses reveal that larger models tend to be more robust than smaller models, even if the latter are trained on significantly more data. We further extend our analysis by evaluating the models’ robustness for the various population sub-groups, namely, English and non-English (Spanish) speakers, and male and female speakers. We find that significant disparities exist across these sub-groups, thus identifying areas where future work could provide improvements, and demonstrating the utility of SRB in fairness evaluations as well.

To summarize we make the following contributions:

We present SRB , a robustness benchmark for ASR models, which can result in directly comparable robustness evaluations and facilitate progress.

We open source our code 1 1 1 https://github.com/ahmedshah1494/speech_robust_bench/tree/release with clear documentation of existing use cases and support easy extensibility.

We release perturbed versions of the Librispeech test set, and the test set of the Spanish subset of Multi-Lingual Librispeech to facilitate out-of-the-box robustness evaluations for the research community.

We demonstrate the use of SRB by conducting a fine-grained robustness analysis for several popular models. We extend our analysis by using SRB to uncover disparities in the robustness of ASR for various sub-groups of speakers. This highlights the broad utility of such benchmarks to the field of trustworthy AI.