RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Conclusion
In this paper we have proposed, SRB , a comprehensive benchmark designed to standardize the robustness evaluations of ASR models. To the best of our knowledge, SRB , is the first ASR robustness benchmark that contains long-form speech, and expands the set of distortions to include special effects, speaker attribute modification and adversarial attacks. We present case studies to demonstrate the utility of SRB in evaluating the robustness of ASR models, as well as its potential to facilitate evaluations of other aspects of trustworthy AI, particularly fairness. Our analysis reveals that while the latest large models, trained on vast quantities of data, are more robust than their counterparts on average , under certain types of perturbations they are outperformed by smaller models trained on lesser data. Furthermore, our analysis reveals that model size generally has a greater impact on robustness than the size of the training dataset, which runs somewhat counter to the prevailing wisdom espoused by recent works (Radford et al., 2023 ; Likhomanenko et al., 2020 ) . We also use SRB to measure the disparities in the robustness of ASR models on data generated by different population subgroups, namely speakers’ language and gender. We find that multi-lingual models are generally more robust for English speech than non-English (Spanish) speech. We also find that under certain perturbations models may be more robust for one gender than the other. We believe that SRB will enable rigorous robustness evaluations of ASR models in a highly standardized manner that will allow comparisons between existing and new approaches and thus facilitate progress tracking. Furthermore, we also release transformed test sets in English and Spanish to facilitate robustness evaluations for researchers and model developers. We believe that this will make robustness evaluations more prevalent and encourage model developers to consider robustness as a key metric to improve.
Impact statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Acknowledgments
The execution of this work was funded by the European Union’s Horizon 2020 RIA SPATIAL project (Grant Agreement No. 101021808), RIA ELOQUENCE project (Grant Agreement No. 101135916) and from the Spanish Project 6G-RIEMANN (Grant Agreement No. 2022/0005420). The authors bear the sole responsibility for the content presented in this paper, and any interpretations or conclusions drawn from it do not reflect the official position of the funding agencies.
Figure 1 : An illustration of the processes involved in using our benchmark to evaluate the robustness of ASR models.; An illustration of the processes involved in using our benchmark to evaluate the robustness of ASR models.
Figure 2 : Taxonomy of perturbations currently used in SRB .; Taxonomy of perturbations currently used in SRB .
(a); (b); Overall utility and stability of English ASR models. Panel (a) shows the NWER for English ASR models under non-adversarial and adversarial perturbations, averaged across all perturbations and severities. The green markers and line show the average WER across the two perturbation types. Panel (b) shows the variance in WER (WERV) caused by the addition of random Gaussian noise and randomly selected environmental noise at fixed SNR of 10 dB.
(a); (b); (c); NWER on non-adversarial and adversarial perturbations plotted against the number of parameters (a & c) and hours of training data used to train the DNN (b). Figures (a) and (b) plot all the models, while (c) plots three families of models (indicated by the color). The models within the family share similar architectures, and training datasets.
(a); (b); (a)The average NWER of Spanish only and Multi-Lingual models on Spanish utterances, and (b) The average degradation (across severities) in the WER of wsp-lg caused by various perturbations (x-axis).
(a) English; (b) Spanish; Log WER Ratio for (a) English, (b) Spanish models under various perturbations. WERs are averaged across severity levels before computing the Log WER Ratio.
Figure 7 : NWER of the models on different augmentations, averaged over all severities.; NWER of the models on different augmentations, averaged over all severities.
Figure 8 : NWER as the severity of the augmentation is increased.; NWER as the severity of the augmentation is increased.
Figure 9 : The WERV of various models under Gaussian noise and environmental noise of SNR 10 dB; The WERV of various models under Gaussian noise and environmental noise of SNR 10 dB
Figure 10 : The WER of the various models under general (blue) and specific (orange) adversarial perturbations. The green markers and line show the average WER on both perturbation types.; The WER of the various models under general (blue) and specific (orange) adversarial perturbations. The green markers and line show the average WER on both perturbation types.
Figure 11 : The NWER of Spanish only and Multi-Lingual models on Spanish utterances under various perturbations.; The NWER of Spanish only and Multi-Lingual models on Spanish utterances under various perturbations.
(a); (b); Log WER Ratio between male and female speakers from Librispeech (English) (a) and Spanish Multilingual Librispeech (b).
Models used in our evaluations. The model names correspond to the names of their pretrained checkpoints in the Huggingface library ( https://huggingface.co/models ). The abbreviations of these names are in the parentheses after them. Some of the unilingual models are pre-trained on multilingual data but are fine-tuned on only one language and thus can not transcribe any other language. Multilingual models have been pre-trained and fine-tuned on multiple languages so the same DNN can transcribe several languages. The WER of multilingual models is presented as English/Spanish
The parameters defining the various severity levels of the perturbations used in the proposed benchmark.