Pre-trained Speech Processing Models Contain Human-Like Biases that Propagate to Speech Emotion Recognition
Authors: Isaac Slaughter
Craig Greenberg
Reva Schwartz
Aylin Caliskan
Venue: EMNLP 2023 Findings (2023)
Keywords: AI bias
speech processing
embeddings
representation learning
bias propagation

Abstract: Previous work has established that a person's demographics and speech style affect how well speech processing models perform for them. But where does this bias come from? 
In this work, we present the Speech Embedding Association Test (SpEAT), a method for detecting bias in one type of model used for many speech tasks: pre-trained models. The SpEAT is inspired by word embedding association tests in natural language processing, which quantify intrinsic bias in a model's representations of different concepts, such as race or valence—something's pleasantness or unpleasantness—and capture the extent to which a model trained on large-scale socio-cultural data has learned human-like biases. Using the SpEAT, we test for six types of bias in 16 English speech models (including 4 models also trained on multilingual data), which come from the wav2vec 2.0, HuBERT, WavLM, and Whisper model families. We find that 14 or more models reveal positive valence (pleasantness) associations with abled people over disabled people, with European-Americans over African-Americans, with females over males, with U.S. accented speakers over non-U.S. accented speakers, and with younger people over older people. Beyond establishing that pre-trained speech models contain these biases, we also show that they can have real world effects. We compare biases found in pre-trained models to biases in downstream models adapted to the task of Speech Emotion Recognition (SER) and find that in 66 of the 96 tests performed (69\%), the group that is more associated with positive valence as indicated by the SpEAT also tends to be predicted as speaking with higher valence by the downstream model. Our work provides evidence that, like text and image-based models, pre-trained speech based-models frequently learn human-like biases when trained on large-scale socio-cultural datasets. Our work also shows that bias found in pre-trained models can propagate to the downstream task of SER.

Pre-trained Speech Processing Models Contain Human-Like Biases that Propagate to Speech Emotion Recognition
Isaac Slaughter', Craig Greenberg', Reva Schwartz', Aylin Caliskan'
'University of Washington, 2NIST
edu, grreiigrrgeeeerrr.rg reva. wwwaaaaaaaaattttittt..rg.
Abstract
Introduction
Recent approaches many speech tasks rely on pre-trained models: large models trained to learn speech representations (multi-dimensional matri- ces referred to as embeddings) from large-scale corpora, which can be adapted to a variety of tasks (Feng and Chaspari, 2020; Niu et al., 2020; Yang et al., 2021). In computer vision and natural lan- guage processing, methods called Embedding As- sociation Tests (EATs) have been used to evaluate biases in the ways pre-trained models represent cial groups, allowing researchers identify bias early in the machine learning pipeline, before it propagates to downstream tasks (Wolfe et al., 2023; Wolfe and Caliskan, 2022a,b; Steed and Caliskan, 2021; Guo and Caliskan, 2021; Toney- Toney-Wails and Caliskan, 2021; Caliskan et al., 2016). In this paper, present the first intrinsic association and bias evaluation method for pre-trained models in speech processing: the Speech Embedding Association Test (SpEAT)-
Previous work has established that person'ss s demographics and speech style affect how well speech processing models perform for them. where does this bias come from? this work, present the Speech Embedding Asso- ciation Test (SpEAT), a method for detecting bias in one type of model used for many speech tasks: pre-trained models. The SpEAT is in- spired by word embedding association tests in natural language processing, which quantify intrinsic bias in model's 's representations of different concepts, such as race or valence- something's ' pleasantness unpleasantness and capture the extent to which model trained on large-scale veco-cultural data has learned human-like biases. Using the SpEAT, we test for six types of bias in 16 English speech mod- els (including 4 models also trained on multilin- gual data), which come from the wav2vec 2.0, HuBERT, WavLM, and Whisper model fami- lies. We find that 14 more models reveal positive valence (pleasantness) associations with abled people over disabled people, with European- Americans over African-Americam with females over males, with U.S. accented speakers over n.n-U.S. accented speakers, and with younger people over older people. Beyond establishing that pre-trained speech models con- tain these biases, also show that they can have real world effects. We compare biases found in pre-trained models to biases in down- stream models adapted to the task of Speech Emotion Recognition (SER) and find that in 66 of the 96 tests performed (69%), the group that is more associated with positive valence as indi- cated by the SpEAT also tends to be predicted speaking with higher valence by the down- stream model. Our work provides evidence that, like text and image-based models, pre- trained speech based-model models frequently learn human-like like biases when trained on large-scale socio-culural datasets. Our work also shows that bias found in pre-trained models prop- agate to the downstream task of SER.
A SpEAT measures bias related to two social groups in a speech model, and produces an ef- feet size d, which when positive indicates that the speech model favors the social group that humans also tend to favor. We evaluate the SpEAT first by testing whether it reveals positive effect sizes, (congruent with human stereotypes). for biases re- lated twelve social groups. The types of bias that we study have been documented large popu- lations, and all involve associations with valence, a term often used in psychology literature to de- scribe emotions. Valence is frequently equated with "pleasantness" or "pleasure," where positive valence indicates something plcasant and negative valence indicates something unpleasant (Russell, 1980; Morgan, 2019; Nielen et al., 2009). We study associations with valence due its role as a pri- mary dimension of affect, strong signal speech and language, and determinant of how people form

---

As negative valence is associated with anger and sadness (Morgan, 2019), this could potentially re- sult in speech from one group of people being translated as more frequently angry than speech from another in a speech translation system that considered emotion when performing translations, or speech from one group being treated as more frequently sad than speech from another in diag- nostic model used in mental health setting, two suggested applications for SER models (El Ayadi et al., 201 1). Beyond showing that biases found in pre-trained models can propagate, our work adds to the growing body of evidence showing bias in attitudes (Barrett, 2006; Sharot and Garrett, 2016). Using the SpEAT, we find that 15 of 16 models we test show bias for U.S. accented speakers over non-U.S. accented speakers (where relative to non- U.S. accented speakers, U.S. accented speakers are more associated with positive valence than nega- tive valence ), 15 of 16 models show bias for young speakers over old speakers, 14 of 16 models show bias for female speakers over male speakers, 14 of 16 models show bias for European-Ament American speakers over African-American speakers, and 14 of 16 models show bias for abled speakers over disabled speakers. Our results indicate that, as with models trained in other modalities, models trained on large corpora of speech data also learn human- like biases.
Automated Emotion Recognition (AER), one of many issues that have been raised concerning this area of research (Mohammad. 2022).
Finally. we provide an approach for studying how the number of stimuli (sample size) used in an EAT can change its results. EAT's performed in other modalities have used fixed numbers of stimuli to represent social groups and concepts when mea- suring bias. To evaluate how an EAT score changes when differing sample sizes are used, we calculate bootstrap estimates of the Standard Error (SE) at different sample sizes (Hesterberg, 201 1). The SE measures how much a statistic would change if it were calculated repeatedly based on new data, and lower SE values indicate that there is less uncer- tainty associated with the statistic. We find that the SE of the SpEAT decreases sharply as the number of stimuli used to represent social groups increases, for example that increasing the number of stimuli from to 10 can lead to an increase in precision by a factor of two, Our results show that the number of stimuli used in an EAT can have a large effect on the uncertainty of its effect size d.2
To understand the potential impact of these bi- ases, we evaluate whether results found with the SpEAT are indicative of downstream effects. We do so by considering the task of Speech Emo- tion Recognition (SER), which predicts emotions based on speech (Mohammad, 2022). We compare SpEAT scores for two social groups to disparities in a downstream model's predictions of valence for speech from the groups. We find that social groups that have positive associations with valence in pre- trained models also tend to be predicted by down- stream SER models as more positive in valence: For 66 of 96 (69%) SpEATs performed, a down- stream SER model tends to predict speech from the social group favored in the SpEAT as more posi- tively valenced than speech from the other group considered.
Background and Related Work
We now survey related work on pre-trained models in speech processing, as well as work measuring bias in speech models and models in other modali- ties.
Pre-Tree Trained Models in Speech Processing Pre- trained models initially learn from large quanti- ties of unstructured data, for example corpora con- taining hundreds of thousands of hours of speech, and can then be adapted using smaller structured datascts to a variety of specific tasks (Bommasani et al., 2021). Pre-trained models used for speech processing often train on speech samples that are of variable lengths (Liu et al., 2022a). After initial pre- training, downstream model for specifie task may then be created based the pre-trained model, using transfer learning. This is often done using fine-tuning, whereby slight architectural changes are made to the model before it is retrained on a smaller dataset, or feature extraction, which treats the pre-trained model as a pre-processing step for downstream task-specific model (Peters et al., 2019). Pre-trained models are often transformer based, and with feature extraction user extracts numerical representations of input data, (also called
'Previous work studying propagation between intrinsic bias in models and bias in how they perform downstream has suggested that upstream hias is more likely to be indicative of downstream bias if the biases are conceptually similar (Steed et al. 2022; Goldfarb- Tarrant et al., 2021). Because the SpEAT relates to associations with valence, we chonse SER as a related downstream task.
"Code for this work is available at https//githh..ommssacchhhhhhhhhp

---

embeddings), after each layer in the neural network. The user then employs these embeddings as input data when training downstream model for new task (Yang et al., 2021; Peters et al., 2019).
2019), image encoders (Steed and Caliskan, 2021), and multimodal vision-language models (Wolfe and Caliskan, 2022a). These tests have shown that large-scale socio-cultural data, such as text and im- ages, can be a source of implicit associations and biases.
Bias in Speech Models Previous research measur- ing bias in speech processing models largely stud- ies differences in performance on specific speech tasks, for data sourced from people of differing so- cial groups. Social group-based performance com- parisons exist for Automated Speech Recognition (ASR) (Tatman, 2017; Tatman and Kasten, 2017; Koenecke et al., 2020; Feng et al., 2021; Liu et al.. 2022b; Riviere et al., 2021), Speaker Verification or Speaker Identification (SID) (Hutiri and Ding, 2022: Fenu et al., 2021, 2020; Fenu and Marras,
3 Methodology
Building on EATs, the SpEAT measures how a model's embeddings of stimuli representing two target concepts, (for example female and male), relate to its embeddings of stimuli representing two attribute concepts (for example positive and negative valence). After extracting embeddings for stimuli corresponding to the four different concepts from speech model, the SpEAT is carried out by measuring the relative cosine similarities between the speech models' embeddings of samples repre- senting the four concepts. Let X and Y be the sets of embeddings representing the target concepts, for example embeddings derived from speech samples from female (X) and male (Y) speakers respec- tively, and AA and B be the set of embeddings repre- senting the attribute concepts, such as embeddings derived from speech samples rated as being posi- tive (A) and negative (B) in valence, respectively. Similar to other EAT', we present an effect size metric, the SpEAT d, which shows which of the target sets, X or Y, is relatively more similar to the first attribute set, .A, than to the second, B. As EAT's were originally based on IAT's, the SpEAT d is based on the IAT D. adaptation of Cohen's that uses unified standard deviation rather than pooled standard deviation (Greenwald et al., 2003). Values of Cohen's d of 0.20, 0.50 and 0.80 were originally introduced as being small, medium, and large (Cohen, 1977).
2022; Chen et al., 2022b; Meng et al., 2022), as well as number of other speech tasks (Meng et al., 2022: Hutiri et al., 2023). Differences in model performance based the gender (Tatman, 2017; Tatman and Kasten, 2017: Chen al., 20226; Feng et al., 2021; Liu et al., 2022b; Hutiri and Ding, 2022: Fenu et al., 2020, 2021; Fenu and Marras, 2022; Riviere et al., 2021), dialect (Tatman, 2017;
Tatman and Kasten, 2017), race (Koenecke et al.. 2020; Tatman and Kasten, 2017; Chen et al., 2022b;
Riviere et al., 2021), age (Fenu et al., 2020, 2021), city (Koenecke al., 2020), nationality (Hutiri and Ding, 2022), and native language (Feng et al., 2021) of the speaker have been tested. While these works have established that some speech systems perform worse for people of different social groups (depending on the system and group), pre-trained models have only been studied by testing for per- formance differences across speech styles in down- stream models, by Meng al. (2022).
Embedding Association Tests EATs are methods for quantifying bias in representation learning mod- els, first introduced by Caliskan et al. (2016, 2017). EAT' were originally adapted from Implicit As- sociation Tests (IATs), extensively validated tests used for indirectly measuring implicit associations and bias in humans (Greenwald et al., 1998, 2022). EAT' measure the association of two sets of target concepts (e.g., European American and African American) with two sets of attributes (e.g., posi- tively valenced and negatively valenced stimuli). EAT'S often use positive and negative valence as attributes, due to valence's role in belief forma- tion. EATS were originally used for measuring bi- ases in word embeddings, but have since been used to study biases in sentence encoders (May et al.,
To calculate the SpEAT d, as for past EATs, first the mean cosine similarity between a target embed- ding w and each embedding E A is calculated, followed by the mean cosine similarity between B. The difference between these
w and each means is the relative association between each em- bedding w C X Y and and B:
s(w, A, B) = meangeAcos(w,,) a) mconse 6)
For example, if were the embedding of speech sample from male speaker, this differ- ence in means s would measure how much more associated the sample was with positive valence than negative valence. These values for each of

---

the target stimuli are then aggregated to construct the effect size measurement, d. The SpEAT d is given below, and shows how much closer the em- beddings in X are to than B, relative to the embeddings in Y. The effect size is the same as that used by Caliskan et al. (2017) and Steed and Caliskan (2021):
sentence encoders, however, we elect apply the aggregation method they propose for ELMo (Peters et al., 2018) to all models we test. For each stimuli, this method involves taking the mean embedding within a layer, then summing these averaged em- beddings together. We elect to use this aggrega- tion method because the methods that May et al. (2019) propose for other models only use embed- dings from a single layer in a model, which discards large amount of information that is sometimes used when adapting pre-trained speech models to downstream tasks and can potentially contain bias (Yang et al., 2021; Chen et al., 2022a). We do test alternative strategies for extracting embeddings in Appendix D, however.
d meanzex5(I; A, B) meangeysyy, A, B) std dew dedddewexxuxs(u A. B)
Before cosine distances can be compared, how- ever, embeddings of target and attribute stimuli need to be extracted. Pre-trained speech mod- els, like some of the sentence encoders studied by May et al. (2019) with the Sentence Embed- ding Association Test (SEAT), create dynamically sized embeddings depending on the length of the sequence fed to the model as input (Radford et al., 2022; Hsu et al., 2021; Chen et al., 2022a; Baevski et al., 2020). Because embeddings are compared using cosine similarity in EAT, which requires operands to have the same dimensions, this means that raw embeddings cannot be used directly. An- other relevant feature of the pre-trained speech pro- cessing models we consider, also similar to many models tested with the SEAT, is that these models are multi-layered, with different layers providing information that is relevant to different downstream tasks (Radford et al., 2022; Hsu et al., 2021; Chen et al., 2022a; Baevski et al., 2020).
4 Data
We now describe the pre-trained models we focus on, their training data, and relevant features of the stimuli that we use for evaluating these models. Pre- Tre-Trained Models and Their Training Data To study pre-trained models, we chose model families that either widely used or achieve state-of-the art performance on relevant speech benchmarks. We use multiple models from each family, rather than merely the largest model, as not clear that models from the same family encode the same biases, and multiple models from a family are of- ten used by practitioners not necessarily just the largest one. We use 16 models from these families: three from wav2vec 2.0, three from HuBERT, three from WavLM, and seven from Whisper. Further details concerning the models we use appear in Appendix B.
Recent approaches for adapting pre-trained mod- els to specific tasks based on feature extraction rely on embeddings from all layers in the pre-trained model, rather than just a single layer (Yang et al., 2021; Peters et al., 2018, 2019). Intuitively, embed- dings from internal layers may contain information not relevant to the task the model was pre-trained for, but important for the task the model is being adapted to. In speech, for example, embeddings from later layers in HuBERT and WavLM mod- els have been shown to more useful for granular tasks like ASR than embeddings from earlier layers, while for more speaker-centric tasks such as SID, embeddings from earlier in the models have been shown to be useful as well (Chen et al., 2022a). Be- cause of the similarities in the models they study, as well as in the way that embeddings from the mod- els are used, the SpEAT follows the SEAT in how handles variable-sized, multi-layered embeddings (May et al., 2019). While the SEAT uses differing methods for aggregating embeddings for different
Of these models, wav2vec 2.0, HuBERT, and WavLM use Self-Supervised Learning (SSL. and train exclusively on speech, while models from the Whisper family, which use Weakly Supervised Learning (WSL), train on speech paired with tran- scripts. The wav2vec 2.0, HuBERT, and WavLM models we use are trained on either Librispeech (Panayotov et al., 2015) or Libri-Light (Kahn et al., 2020), speech datasets derived from readings of au- dio books recorded by volunteers as part of the Lib- ri Vox project. In addition to the Libri-Light train- ing set, some WavLM models were also trained on additional data: 10,000 hours of audio from the GigaSpeech corpus (Chen al., 2021), which is derived from podcasts, YouTube, and audiobooks, as well as 24,000 hours of audio from the VoxPop- uli corpus (Wang et al., 2021a), which is derived

---

Mean IAT n
Target Concepts X and Y
Corresponding LAT
Data Source
Speech Content
Viaual IAT:
Abled and Disabled
(Nasek et 2007)
0.45
(Rudricz al., 2012)
Read Phrases
60
Eusopean-Americaan and
Visual IAT:
(Pitt al., 2007),
Extemporancous Speech from
0.37
60
African-America
(Nosek et al., 2007)
(Kendall and Farrington, 2021)
Sociolinguistic Interviews
Audio LAT:
Female and Male
0.31
(Weinherger, 2015)
Read Paragraph
60
(Mitchell et al., 2011)
Human
Audio LAT:
0.33
(Weinherger, 2015),
57
Syuthesized
(Mitchell 2011)
Syuthesized Paragraphi
Audio IAT: (Pantos and
U.S. and Foreign
(Weinberger, 2015)
Paragraph
Perkins, 2012)
Viaual IAT:
Young and Old
0.49
(Weinberger, 2015)
Read Paragraph
58
(Nosek et 2007)
IATs used for evaluating whether the SpEAT captures human-like biases in pre-trained speech models. and Y indicate the larget concepts, Data Source identifies the source of the speech data used for representing target concepts, and N indicates the number of stimuli in each target group. Mean IAT D indicates the mean IAT effect size measuring bius in humans, where a positive IAT D value indicates that X concept is more associated with positive valence than the concept in humans.
Table I:
from European Parliament events. The Whisper models we use are either trained on multilingual corpus, which consists of 680,000 hours of audio paired with transcripts in 97 distinct languages, or on the subset of the multilingual dataset which is in English.
EAT's, we evaluate the SpEAT by testing for biases that have been found in humans using IATs and comparing the results in speech models to those found in humans (Table 1). We do so using two types of IATs: 1) foundational tests in the IAT liter- ature, which were originally performed entirely using visual stimuli, and 2) audio IAT', which have explicitly established biases related to speech styles in humans (Mitchell et al., 2011; Pantos and Perkins, 2012). For visual IATs, we consider tests performed by Nosek et al. (2007) due to the large number of test participants (more than 20,000 peo- ple each). We focus on tests that use positive and negative valence as attribute concepts and that use target concepts for which there are speaking style differences between social groups, (for example we adapt an IAT related to age bias, but do not adapt the test related to weight bias). For audio IATs, we consider work from Mitchell et al. (201 1) and Pan- tos and Perkins (2012). To our knowledge, these are the only reer-re publications performing LAT's that both use speech in Standard American English (SAE) and use positive and negative va- lence as attribute concepts.
Weber (2021) performed an analysis of users who both had contributed to LibriVox and had cat- alog names that could be associated with gender, and identified 45.3% female and 54.7% as male. Beyond this we are not aware of any information documenting the extent to which different speech styles are represented in the dataset, such as in- formation on speaking styles. We note however that similar speech datasets have relied on U.S. speech. For example, of speech in the Common Voice English (7.0) dataset (Ardila et al., 2020) la- beled by accent, about 47% was identified as U.S. English, compared to the next most frequent ac- cent, England English, which made up about 16% of labeled speech (Markl, 2022).4 The Pcople's Speech dataset (Galvez et al., 2021), as another example, is identified by its authors as being made up primarily of American English. We hypothesize that U.S. English is also more prominent in the datasets that were used for training the models we consider, and following best practices for LAT's that stimuli should be familiar to the entity being tested (Greenwald et al., 2022), we choose to center our work on U.S. English speech. We include results for U.K.-based tests in Appendix G however.
Because Nosck et al. (2007) originally repre- sented concepts visually, and because audio IAT's from Mitchell et al. (2011) and Pantos and Perkins (2012) used short samples which may not fully capture variations in accents (for example record- ings of speakers saying individual words), we adapt these tests by using alternate stimuli to represent the intended concepts. 5 To do so we follow guid-
Target and Attribute Stimuli Following previous
'Transcripts in the multilingual dataset are either in the same language as speech or in English.
'For audio LATs, in addition to SpEAT results for tests based on the adapted stimuli, we also include results based on the original stimuli that were tested on human subjects in Appendix R. We include an alternate set of stimuli for the Abled and Disabled test as well in Appendix E, which
"Common Voice has been used for training similar speech models such as XISRR (Conneau et al., 2020) and UniSpeech (Wang et al., 2021b).

---

Abled and Disabled
SpEAT Model
EA and AA
Meon SpEAT d
Female and Male
Menn LNI
Human and Synthesized
U.S. and Foreign
Young and Old
-0.5
0.5
Figure I: Each gray square corresponds to hias test using embeddings from single pre-trained model. Pink square refers mean across all models, and blue triangle refers to mean LAT D for bias measured in humans. A positive SpEAT d indicates that in the model's emheddings, speech samples from the first target group (c-g, abled speakers or European-Amen Amerriaan (EA) speakers) relatively more associated with positive valence over negative valence than samples from the second target group are (e.g. disabled speakers or speech model tends to favor the group that humans also tend to favor. 14 or more (out of 16) models show bias results that are congruent with biases found in humans regarding ability, race, gender, age, and accent.
(AA) speakers). A SpEAT d with the same sign as the IAT D indicates that the
ance from Greenwald et (2022) by selecting stimuli that are easily distinguishable. All stimuli that we use are U.S. English (excluding the stim- uli used to represent the concept of foreign accent). To ensure that differences between the stimuli rep- resenting the target concepts in a test are only due differences in these concepts, and not due to another factor, we ensure that speech samples be- tween the two target groups are matched. Stimuli are matched on gender as well as approximate age, with the exception of the female and male test, which is not matched on gender, the young and old test, which is not matched on age, the human and synthesized test, which is not matched on age, and the abled and disabled test (for which age infor- mation was not made available for speakers). All tests, other than the test comparing speech from European-American speakers and from African- American speakers (which uses extemporaneous speech) are also matched in speech content, and involve phrases, sentences, or paragraphs that are repeated across conditions. Further details on the target stimuli chosen, well as on the matching procedures used, can be found in Appendix C.7
For attribute stimuli, we use speech from the Morgan Emotional Speech Set (MESS) (Morgan, 2019), a corpus of 1,800 semantically neutral recordings made by six Caucasian actors, aged 19- 21. The Morgan Emotional Speech Set is the only publicly-vailable corpus of which we are aware that contains SAE speech rated on valence. Voice actors were native speakers of American English, evenly split on gender, and were asked read sentences while portraying one of four categorical emotions happy, sad, angry, or calm. The record- ings were then rated on valence by 10 listeners who were aged 19-28 and evenly split on gender. To ensure that differences in association with positive and negative valence not due differences in the speakers whose speech is used to represent the valenced concepts, we ensure that each speaker is represented equally in each valence pole. We se- lect the 10 recordings from each speaker that have the highest average valence ratings to represent positively valenced speech, and the 10 recordings from each speaker that have the lowest average valence ratings to represent negatively valenced
than speech content from other. We pass transcripts the speech from European- Amerrccan and African- AmrrrcnnAmeeiann speakers through five commonly used sentiment analysis tools (Barbieri et al., 2020, 2022; Loureiro et al., 2022: Hutto and Gilbert, 2014; Elias, 2022), then use a Welch's unequal vari- ances t-test evaluate whether the sentiment scores differ by race on average (Delacre et al., 2017; West, 2021). Each sentiment analysis tool gives three scores (negative sentiment, neutral sentiment, and positive sentiment); VADER also gives a compound score. Using the typical p = 0.05 threshold, across all 16 scores output by these models, only one signifi- cant difference is found: the positive sentiment score given by the model from Loureiro et al. (2022) (p = 0.02). In this case, however, the social group that is scored higher on positive sentiment for semantics is not the one that tends to be favored by the speech processing models.
represents the target concepts using short phrases. "Synthesized speech is generated with Microsoft S.PI which contains the synthetic voices originally employed for evaluating bias humans. We this generator in order ensure that the biases we measure in speech models relate humans.
the same concepts as biases Each voice is Microsoft SAPI 4.0 labeled with a gender or gendered name that the voice is intended to represent, we use these labels to ensure that the set speech samples we use is balanced on gender. As of submission. code for the generator is available at https://githuh..c haatthhh..oom//// Com/TETYYSS PASAP144
compare
"To ensure that differences in associations with valence in the European-Amne Camooean-American American and African-American test are not due to differences in semantics, we test whether the speech content from one social group tends to show more positive valence

---

Abled and Disabled
Cahen's
EA and AA
Modhl
Female and Male
Meon SpEAT d' for Upsrzesm
Human and Synthesized
Modrls
U.S. and Foreign Young and Old
Cahen's
Dowzstieom
Figure 2: Comparison of bias found in pre-trained model using SpEAT bias found in downstream SER model. Gray circles corresoond to bias test using individual downstream SER model, and yellow circles show mean bias across downstream SER models. In 66 the (69%) tests perforned, the group more associated with positive valence by the pre-trained model is also more associated with positive valence by the downstream SER model.
speech. This results 60 samples representing positive valence and 60 samples representing nega- tive valence.
LAT D values given in Table 1, All IAT D values that we compare to are positive, and range between 0.31 and 0.49. Of the tests we perform, 79 out of 96 (82%) have positive SpEAT d values, reflecting the same biases recorded in society, Results vary depending on the test, however: only 6 of 16 mod- els show bias against synthetic voices, while 14 more models show stereotype-congruen biases for all other tests.
5 Evaluation and Results
We provide three evaluations of the SpEAT. First, we test whether it captures human-like biases in speech models by comparing results found in pre- trained models using the SpEAT to results found in humans using IATs. Second, as the SpEAT de- signed for pre-trained models that are often adapted to other tasks, we also evaluate the method by test- ing whether these biases propagate, and whether biases found in models using the SpEAT are in- dicative of biases in downstream models. Third, we measure how much uncertainty is associated with the SpEAT effect size by studying how its SE varies when different numbers of stimuli are used represent target concepts.
There does appear to be slight model family effect, with models in the same family having some- what of a tendency toward similar SpEAT d values concerning given social group. Performing a two-way ANOVA with the SpEAT value as de- pendent variable and social group and model family as independent variables, we find that the interac- tion between model family and social group is just below the typical p=0.05 significance threshold, (p=0.048). The main effect of model family is not significant, however, suggesting that it is not the case that SpEAT d values for a family tend be located in the same area across all social groups. For example, Whisper SpEAT d values tend to be similar to one another in the AfrrrcnnAmeriann and European-Americat test, but Whisper SpEAT d values for the African-Amern American and European- American test do not tend to be similar to Whisper SpEAT d values from the Young and Old test. Bias Propagation to Speech Emotion Recogni- tion We further evaluate the SpEAT by testing whether SpEAT d values for a pre-trained model are related to bias in the downstream task of SER, which involves predicting emotions associated with speech (Mohammad, 2022). Using an architecture from Yang et al. (2021) designed for adapting pre-
Comparison to Human Biases Similar to other EATs, our first evaluation of the SpEAT is to test whether it detects biases in speech models using ground truth data from human biases (Caliskan et al., 2017; Steed and Caliskan, 2021), We com- pare the direction of the SpEAT to the direction of the mean IAT D among humans for number of well documented biases to check for stereotype- congruence. For each speech model discussed in Section 4, we calculate the SpEAT d for each bias in Table All IAT D values we compare to are positive, meaning that humans showed more of an association between the first target group and positive valence than between the second target group and positive valence. SpEAT d values for the 16 models we test are displayed in Figure 1, If the SpEAT were able to find human-like biases in pre-trainn trained models in speech, we would expect SpEAT d values to be identical in sign to the mean
8 *Previous EATs have also reported p-values for a related Null Hypothesis Significance Test, which we also report in Appendix A.

---

trained models downstream tasks, we adapt each of the 16 pre-trained models to predicting valence in the full MESS dataset (consisting of 1,800 sam- ples) which is the only publicly available corpus of SAE labeled for valence of which we are aware. We train three separate downstream models for each pre-trained model. 9 We then predict the valence of speech stimuli that were introduced in Table 1, for example stimuli from abled and disabled speakers, and calculate Cohen's d for the predicted valences for each pair of target concepts. We consider dif- ferences in standardized mean predicted valence between the two groups to be a bias due to the matched nature of the stimuli.
1.00
Standard Error
0.75
0.50
0.25
0.00
10
20
30
40
50
60
Number of Stimuli per Target Group
Figure 3: SE for SpEAT d when different numbers of stimuli are used to represent each target concept. Lower SE indicates more precise estimate, Each point represents a single bootstrap estimate based on a single number of target stimuli, type of bias. and speech model. Orange line indicates average SE each number of stimuli (across types of bias and speech models).
As shown in Figure 2, we find that SpEAT d values favoring one target group tend to align with speech from the favored target group being pre- dicted as higher in valence: In 66 of the 96 (69%) tests performed, the group more associated with positive valence by the pre-trained model is also more associated with positive valence by the down- stream SER model. We also note that in all tests excluding that related to human and synthesized speech, the average Cohen's d across all 16 speech models is in the same direction as the average SpEAT across the models.
2011). For each bias test and speech model, we calculate multiple bootstrap estimates of the SE, using different numbers of target stimuli for each. This allows us to study how the SpEAT's uncer- tainty changes when more or less stimuli are used to represent the target concepts. Each SE estimate is based on 10,000 bootstrap resamplings. Results in Figure 3 show that the SE decreases sharply as the sample size increases: the average standard er- when two stimuli are used per target group is 0.75, the average standard error when 10 stimuli are used 0.33, and the average standard error when 60 stimuli are used 0.12.
Uncertainty of SpEAT d Our final evaluation of the SpEAT involves quantifying the uncertainty as- sociated with SpEAT values when different num- bers of stimuli are used to represent target concepts, in order to evaluate the extent to which SpEAT ef- feet sizes might differ if they were recalculated based on similar datasets. We do so by estimating the SE for each SpEAT d value at varying sample sizes. The SE of statistic measures how much the statistic will vary if is calculated repeatedly based on new data. Lower values indicate that the statis- tic will vary less, and that there less associated uncertainty. We estimate the SE using bootstrap- ping (Hesterberg, 1), where we sample with replacement from the original dataset, repeatedly calculate the statistic, then calculate the sample standard deviation of these bootstrapped statistics. We perform resampling match the original sam- pling strategy used for each dataset, (for example, the abled and disabled test where each individual stimuli from a disabled speaker was matched with an individual stimuli from an abled speaker saying the same phrase, we resample by pair) (Hesterberg,
Discussion
The results in Figure show that pre-trained speech processing models trained on large corpora of speech data, like those trained on large corpora of image or text data, can easily learn human-like bi- ases related to gender, race, ability, age, and accent. Work in computer vision has suggested that biases detected by EATs may arise from co-occurrence in the training data (Steed and Caliskan, 2021). and we hypothesize that co-occurrence may con- tribute here as well, for example if speakers who use one speech style tend to more frequently ex- press positive valence than speakers who use other speech styles in the training set. We also hypothe- size that the wide dispersion in the test comparing human and synthesized speech may be related frequency as it is not clear that synthesized speech is contained in any of the pre- pre-training corpora that the models we consider were trained on.
"Full details on the training process for downstream mod- els is provided in Appendix F.

---

Beyond just identifying that biases can be learned from data in new modality, however, this work also identifies a new means of representing target concepts. In using speaking styles to rep- resent concepts, rather than names or words as in text-based models, we' ve shown that not only do models demonstrate bias based on language con- tent, but also based on how, and by whom, it is delivered. The data used for training models in computer vision, natural language processing, and speech processing inherently multi-faceted. does not connect to social groups in just one way. This data is produced by people who belong to cer- tain social groups; it may be about people of certain social groups; it may be recorded by people of cer- tain social groups; and it may also have meaning to people of certain social groups. Our results sug- gest that, at least in speech processing, models are able to learn associations beyond just content, and raises the question of whether there are differential associations related to the provenance of text and image data as well. Would text of similar meaning, but written by people of different social groups, have different associations with valence?
stream task. Considering these works together, our results add more evidence that there can be a rela- tionship between upstream and downstream bias when the two biases are related in concept.
Our work calls attention to the need for detailed study of bias mitigation techniques in pre-traines trained speech processing models. While we are not aware of any technical solutions for mitigating implicit bias in pre-trained speech models, work by Meng et al. (2022) has tested the effect of the makeup of the training set on performance biases observed in downstream tasks. Their results suggest that al- tering dataset composition is likely not complete solution to bias in pre-trained models, however. is possible that methods that have previously been proposed for mitigating bias in other modali- ties may be useful for managing the bias that has been shown in speech. Strategies that have been explored in computer vision and natural language processing include oversampling the training set to make it more representative of social groups; adversarial training, whereby in learning task the model also learns to explicitly ignore demo- graphic information; and embedding modification, whereby the gcometry of the embeddings is altered in an attempt to remove information related to de- mographics (Wang et al., 2020; Sun et al., 2019).
Beyond adding further evidence for bias in AER (Kiritchenko and Mohammad, 2018), our exper- iment with SER models also adds evidence that biases in pre-trained models can propagate down- stream. The biases that we test for in upstream and downstream models are connected in that they both involve valence-in the upstream model the SpEAT tests for associations with valence, while in the downstream model we measure bias using standardized differences in predicted valence be- tween speech samples from people of differing so- cial groups (which have been matched using the procedure in Section 4). Studying the relationship between intrinsic bias in embeddings measured us- ing an EAT and performance bias in downstream tasks using a variety of performance metrics, tasks, and models, Goldfarb-Tarme Tarrant al. (2021) find that the relationship between the two biases is mostly uncorrelated. They do, however, find that one of two models they test shows a slightly positive rela- tionship between intrinsic bias and bias in concep- tually related downstream task. Work by Steed et al. (2022) similarly finds slight connection between upstream and downstream bias in one of the two tasks they test, and Ladhak et al. (2023) find that when considering name- nationatto bias, abstrac- tive upstream models can pass bias on to a down-
7 Conclusion
In this paper we contributed the SpEAT, a novel method for measuring bias in pre-trained speech processing models. We used the SpEAT to study state-of-the-art pre-trained speech models which are likely to have wide impact on variety of speech processing tasks. We tested for six cate- gories of bias in 16 tre-trained models, and found consistent biases in how speech samples from speakers of different ages, races, genders, and abil- ities were associated with speech samples rated on valence by human listeners. We also showed that biases detected with the SpEAT propagated to the downstream task of SER, as a pre-trained model displaying bias toward group was correlated with bias in downstream models adapted to SER. These findings imply, first, that association-based biases are present in larger number of modalities than had previously been known, and second, that up- stream bias may propagate to downstream bias, when the two biases are connected.

---

8 Limitations
of the speaker, but is at best only able capture the emotion being conveyed, or the emotion being perceived by the annotator (Mohammad, 2022). Furthermore, even well defined and accurate SER runs the risk of setting and enforcing norms for how emotions should be expressed, being used for emotional manipulation, and degrading privacy and autonomy.
In designing our work, we elected to focus on state- of-the-art and popular speech processing models, and to validate the SpEAT using biases relating to speech styles that have previously been doc- umented in humans with IATs. The biases studied concern social characteristics with wide applicabilityy such as race, gender, age, and abil- ity, however they do not capture the full extent of biases that may be contained in speech models. While the SpEAT does allow for the future study of biases related to a wider set of social groups, more consideration and data will be needed for adapting the test quantify biases related to, for example, wider range of gender identities ethnicities.
A Acknowledgments
We would like to thank Hari Iyer, along with the anonymous reviewers, for their comments and sug- gestions. This work was supported by the U.S. National Institute of Standards and Technology (NIST) Grant 60NANB23D194, Any opinions, findings, and conclusions recommendations pressed in this material are those of the authors and do not necessarily reflect those of NIST. This work was facilitated in part through the use of advanced computational, storage, and networking infrastruc- ture provided by the Hyak supercomputer system and funded by the STF at the University of Wash- ington.
Our results may be impacted by the 120 speech samples that were used to represent positive and negative valence, which, while balanced on gender, was provided entirely by six younger Caucasian speakers and rated entirely by 10 younger listeners whose race information was not given (Morgan, 2019). As positive and negative valence were both represented by speech from the same speakers (and rated by the same listeners), it is at least clear that race and age-based associations are not due to dif- ferences in the races and ages of speakers used for each valence pole. While we are not aware a publicly-available dataset of SAE with the neces- sary information to evaluate whether the SpEAT d is affected by the demographics of people used to create the attribute stimuli, we look forward to future work in this direction.
Disclaimer
These results presented this paper are not to be construed or represented as endorsements of any participant's system, methods, or commercial product, or as official findings on the part of NIST or the U.S. Government.
References
9 Ethics Statement
Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gre- gor Weher. 2020. Common Voice: A massively- multilingual speech corpus. In Proceedings of the Twelfth Language Resources and Evaluation Confer- ence, pages 4218 4222, Marseille, France. European Language Resources Association.
The central focus of this work is evaluating the extent to which pre-trained speech processing mod- els contain intrinsic biases and if these biases may propagate downstream. We recognize that metrics are not solutions in and of themselves, and can even be used to propagate bias if deployed in a harm- ful manner (for example practitioners measure bias and then purposefully sclect biased models for downstream applications). We believe, however, that detecting and measuring bias plays an impor- tant role in raising awareness, avoiding harmful impacts, and studying the science of bias.
Jimmy Lei Ba. Jamie Ryan Kiros, and Geoffrey E. Hin- ton. 2016. Layer Normalization.
Alexei Baevski, Yuhao Zhou, AbdeIrahman Mohamed, and Michael Auli. 2020. wav2vee 2.0: A framework for self-supervised learning of speech representations. In Advances in Neural Information Processing Sys- tems, volume 33, pages 12449-12460. Curran Asso- ciates, Inc.
While we study the connection between bias in a pre-trained model and bias after the model has been adapted to SER, we would like to note that bias is just one of many potential issues with SER. SER may purport to capture the inner state
Francesco Barbieri, Jose Camacho-Collados, Luis Es- pinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In Findings of the Association

---

for Computational Linguistics: EMNLP 2020. pages (644-1650, Online. Association for Computational Linguisties.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. 2018. Deep clustering for unsuper- vised learning of visual features. In Proceedings of the European Conference on Computer Vision (ECCV).
Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collados. 2022. XIM-T: Multilingual language models in Twitter for sentiment analysis and beyond. In Proceedings of the Thirteenth Lan- guage Resources and Evaluation Conference, pages 258 266. Marseille, France. European Language Re- sources Association.
Guoguo Chen, Shuzhou Chai, Guan-Bo Bon--Bo Wang. Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khudanpur, Shinji Watamabe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang- Zhao You, and Zhiyong Yan. 2021. GigaSpeech: ASR corpus with 10,000
Lisa Feldman Barrett. 2006. Valence is a basic build- ing block of emotional life. Journal of Research in Personality, 40(1):35-55, 55.
An evolving, hours of transcribed audio. In Proc. Interspeech 2021, pages 3670-3674.
Sanyuan Chen. Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. 2022a. WavLM: Large-scale self-superviedd pre-training for full stack speech processing. IEEE Journal of Se- lected Topics in Signal Processing, 16(6):1505-16
Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson. Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn. Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang. Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Kr- ishna, Rohith Kuditipudi, Ananya Kumar, Faisal Lad- hak, Mina Lee, Tony Lee, Jure Leskovee, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchan- dani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Pa- padimitriou. Joon Sung Park, Chris Piech. Eva Porte- lance. Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang. William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang. 2021. On the Opportunities and Risks of Foundation Models.
Xingyu Chen, Zhengxiong Li. Srirangaraj Setlur, and Wenyao Xu. 2022b. Exploring racial and gender disparities in voice biometrics. Scientific Reports, 12(1):3723.
Jacob Cohen. 1977. Statistical Power Analysis for the Behavioral Sciences. Elsevier Science & Technol- ogy.
Alexis Conneau, Alexei Baevski, Ronan Collobert, Ab- delrahman Mohamed, and Michael Auli. 2020. Un- supervised cross-lingual representation learning for speech recognition.
Marie Delacre, Daniël Lakens, and Christophe Leys. 2017. Why psychologists should by default use Welch's 's t-test instead of Student's t-test. Interna- tional Review of Social Psychology, 30(1):92-101.
Moataz El Ayadi, Mohamed S. Kamel, and Fakhri Kar- ray. 2011. Survey speech emotion recognition: Features, classification schemes, and databases. Pat- tern Recognition, 44(3):572-587.
Seethal Elias. 2022. Sentiment analysis generic dataset.
Kexin Feng and Theodora Chaspari. 2020. A review of generalizable transfer learning in automatic emotion recognition. Frontiers in Computer Science, 2.
Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2016. Semantics derived automatically from language corpora necessarily contain human biases. CoRR, abs/1608.00187, 33.
Siyuan Feng, Olya Kudina, Bence Mark Halpem, and Odette Scharenborg. 2021. Quantifying bias in auto- matic speech recognition.
Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 18666334)1183-186
Gianni Fenu and Mirko Marras. 2022. Demographic
faimess in multimodal biometrics: A comparative analysis on audio- speaker recognition systems. Procedia Computer Science, 198:249-254.

---

Gianni Fenu, Mirko Marras, Giacomo Medda, and Gia- como Meloni. 2021. Fair voice biometrics: Impact of demographic imbalance on group faimess in speaker recogmition. In Proc. Interspeech 2021, pages 1892- 1896.
Tim Hesterberg. 2011. Bootstrap. WIRES Computa-
tional Staristics, 3(6):497-526.
Wei-Ning Hsu, Benjamin Bolte. Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdel- rahman Mohamed. 2021. HuBERT: Self-supervised speech representation learning by masked prediction of hidden units. JEEE/ACM Trans. Audio, Speech and Lang. Proc.. 29:3451-3460
Gianni Fenu, Giacomo Medda, Mirko Marras, and Gia- como Meloni. 2020. Improving fairness in speaker recognition. In Proceedings of the 2020 European Symposium on Software Engineering, pages 129-136, New York, NY, USA. Association for Computing Ma- chinery.
Wiebke Toussaint Hutiri and Aaron Yi Ding. 2022. Bias in automated speaker recognition. In 2022 ACM Conference on Fairness, Accountability. and Trans- parency, pages 230-247, New York, NY, USA. Asso- ciation for Computing Machinery.
Daniel Galvez, Greg Diamos, Juan Manuel Ciro Tor- res, Juan Felipe Cerón, Keith Achorn, Anjali Gopi, David Kanter, Max Lam, Mark Mazumder, and Vi- jay Janapa Reddi. 2021. The people's (ell speech: large-scale diverse english speech recognition dataset for commercial usage. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1).
Wiebke Toussaint Hutiri, Aaron Yi Ding, Fahim Kawsar, and Akhil Mathur. 2023. Tiny, always-on and fragile: Bias propagation through design choices in on-device machine learning workflows. ACM Trans. Softw. Eng- Methodol.
Seraphina Golrfarb-Tarrant, Rebecca Marchant, Ri- cardo Muñoz Sánchez, Mugdha Pandya, and Adam Lopez. 2021. Intrinsic bias metries do not correlate with application bias. Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the I 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers). pages 1926-1940, Online. Association for Computational Linguistics.
C. Hutto and Eric Gilbert. 2014. VADER: A parsi- monious rule-based model for sentiment analysis of social media text. In Proceedings of the Interna- rional AAAI Conference on Web and Social Media, volume 8, pages 216-225.
J. Kahn, M. Riviere, W. Zheng, E. Kharitonov, Xu, P. E. Mazare, J. Karadayi, V. Liptchinsky, R. Col- lobert, C. Fuegen, T. Likhomanenko, G. Synnaeve. A. Joulin, A. Mohamed, and E. Dupoux. 2020. Libri- Light: A benchmark for ASR with limited no supervision. In ICASSP 2020- 2020 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), ICASSP 2020- 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), [S.L]: :. IEEE.
Anthony G. Greenwald, Miguel Brendl, Huajian Cai, Dario Cvencek. John F. Dovidío, Malte Friese. Adam Hahn, Eric Hehman, Wilhelm Hofmann. Sean Hughes, Tan Hussey, Christian Jordan, Teri A. Kirby, Calvin K. Lai, Jonas W. B. Lang, Kristen P. Lind- gren, Dominika Maison, Brian D. Ostafin, James Rae, Kate A. Ratliff, Adriaan Spruyt, and Reinout W. Wiers. 2022. Best research practices for using the im- plicit association test. Behavior Research Methods, 54(3): 1113)))161-11800
Tyler Kendall and Charlie Farrington, 2021. The Corpus of Regional African American Language (Version 2021.07).
Heejin Kim, Mark Hasegawa-Johnson, Adrienne Perl- man, Jon Gunderson, Thomas Huang, Kenneth Watkin, and Simone Frame. 2008. Dysarthric Speech Database for Universal Access Research. In Inter- speech, pages 1741-1744.
Anthony G. Greenwald, Debbie E. Meghee, and Jordan L. K. Schwartz. 1998. Mcasuring individual differ- ences in implicit cognition: The implicit association test. Journal of Personality and Social Psychology, 74(6):1464-144)
Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization.
Anthony G. Greenwald, Brian A. Nosek, and Mahzarin R. Banaji. 2003. Understanding and using the implicit association test: I. An improved scoring algorithm. Journal of personality and social psychol- ogy. 85()).
Svetlana Kiritchenko and Saif Mohammad. 2018. Ex- amining gender and race bias in two hundred senti- ment analysis systems. In Proceedings of the Seventh Joint Conference on Lexical and Computational Se- mantics, pages 43-53. Association for Computational Linguistics.
Wei Guo and Aylin Caliskan. 2021. Detecting emergent intersectional biases: Contextualized word embed- dings contain a distribution of human-like biases. In Proceedings of the 2021 AAAI/ACM Conference on AI, Fthics, and Society, pages 122-133, New York, NY, USA. Association for Computing Machinery.
Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R. Rickford, Dan Jurafsky, and Sharad Goel. 2020. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 1 117(14):76847688
Dan Hendrycks and Kevin Gimpel. 2016. Gaussian Error Linear Units (GELUs).

---

Faisal Ladhak, Esin Durmus, Mirac Suzgun, Tianyi Zhang. Dan Jurafsky, Kathleen McKeown, and Tat- sunori Hashimoto. 2023. When do pre-training bi- ases propagate to downstream tasks? A case study in text summarization. In Proceedings of the 17th Conference of the European Chapter of the Associa- tion for Computational Linguistics, pages 3206-3219. Association for Computational Linguistics.
of synthesized speech support new HCI model of impression management. Computers in Human Be- havior, 27(1):402-412.
Saif M. Mohammad, 2022. Ethics sheet for automatic emotion recognition and sentiment analysis. Compu- national Linguistics, 48(2):239-278.
Shae D. Morgan. 2019. Categorical and dimensional ratings of emotional speech: Behavioral findings from the Morgan Emotional Speech Set. Jour- nal of Speech, Language, and Hearing Research, 62(11 (2(11): 4029.
Norman J. Lass, Dennis M. Ruscello, and Judith A. Lakawicz. 1988. Listeners' perceptions of nonspeech characteristics of normal and dysarthric children. Journal of Communication Disorders, 2155):885- 391.
M.M.A. Nielen. DJJ. Heslenfeld, K. Heinen, J.W. Van Strien, M.P. Witter, C. Jonker, and D.J. Veltman. 2009. Distinct brain systems underlie the processing of valence and arousal of affective pictures. Brain and Cognition, 71(3):387-396
Stephanie Lindemann. 2003. Koreans, Chinese or In- dians? Attitudes and idcologies about non-native English speakers in the United States. Journal of Sociolinguistics, 7:348-364. 364.
Shuteng Niu, Yongxin Liu, Jian Wang, and Houbing Song. 2020. A decade survey of transfer learning (2010-2020). IEEE Transactions on Artificial Intelli- gence, 12))1111-16.
Shuo Liu, Adria Ragolt,, Emilia Parada- Cabaleiro, Kun Qian, Xin Jing, Alexander Kathan, Bin Hu, and Bjöm W. Schuller. 2022a. Audio self- supervised learning: A survey. Patterss 3(12).
Brian A. Nosek, Frederick L. Smyth, Jeffrey J. Hansen, Thierry Devos, Nicole M. Lindner, Kate A. Ran- ganath, Colin Tucker Smith, Kristina R. Olson, Dolly Chugh, Anthony G. Greenwald, and Mahzarin R. Ba- naji. 2007. Pervasiveness and correlates of implicit attitudes and stereotypes. European Review of Social Psychology, 18(1):36-88.
Zhe Liu, Riin--Eleaa Veliche, and Fuchun Peng. 2022b. Model-based approach for measuring the fairness in ASR. In ICASSP 2022 2022 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 6532-6536.
Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose Camacho-collades 2022. TimeLMs: Diachronic language models from Twitter. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics: System Demonstrations, pages 251 1-60, Dublin, Ire- land. Association for Computational Linguistics.
H. O'Reilly, D. Lundqvist, D. Pigat, K. Baron, S. Friden- son, S. Tal, N. Meir, S. Berggren, A. Lassalle, O. Golan, S. Bolte, S. Piana, C. Rotman, P. Co- letta, E. Marchi, I. Davies, N. Sullings, A. Baranger, C. Gauvain, B. Schuller, S. Newman, A. Camurri, P. Robinson, and S. Baron-Cohen. 2012. The EU- Emotion Stimulus Set.
Nina Markl. 2022. Mind the data gap(s): Investigating power in speech and language datasets. In Proceed- ings of the Second Workshop on Language Technol- ogy for Equality, Diversity and Inclusion, pages 1-12, Dublin, Ireland. Association for Computational Lin- guistics.
Helen O' O'Reilly, Delia Pigat, Shimrit Fridenson, Steve Berggren, Shahar Tal, Ofer Golan, Sven Bölte. Simon Baron-Cohen, and Daniel Lundqvist. 2016. The EU- Emotion Stimulus Set: A validation study. Behavior Research Methods, 48(2):567-53 -576.
Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2079 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 622-628, Minneapolis, Min- nesota. Association for Computational Linguistics.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An ASR corpus based public domain audio books. In 2015 IEÉE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206-5210.
Andrew J. Pantos and Andrew W. Perkins. 2012. Mea- suring implicit and explicit attitudes toward foreign accented speech. Journal of Language and Social Psychology, 32.1))3220.
Yen Meng, Yi-Hui Chou, Andy T. Liu, and Hung-yi Lee. 2022. Don't it speak too fast: The impact of data bias on self- suff-supervieed speech models. In ICASSP 2022 2022 IEEE International Conference on Acoustics. Speech and Signal Processing (ICASSP), pages 3258- 3262. Institute of Electrical and Electronics Engi- neers.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018 Deep contextualized word repre- sentations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computa- tional Linguistics.
Wade J. Mitchell, Chin Chang Ho, Himalaya Patel, and Karl F. MacDorman. 2011. Does social desirabil- ity bias favor humans? Explicit-impliit evaluations

---

Matthew Peters, Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? Adapting pretrained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepLANLP-2019) pages 7-14, Florence, Italy. Association for Computational Linguistics.
processing: Literature review. In Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pages 1630-1640, Florence, Italy. Association for Computational Linguisties.
Rachael Tatman. 2017. Gender and dialect bias in YouTube's 's automatic captions. In Proceedings of the First ACI. Workshop on Ethics in Natural Language Processing, pages 53-59, Valencia, Spain. Associa- tion for Computational Linguistics.
Mark A. Piu, Laura Dilley, Keith Johnson, Scott Kies- ling, William Raymond, Elizabeth Hume, and Eric 2007. Buckeye Corpus of Conversa- tional Speech (2nd release).
Rachael Tatman and Conner Kasten. 2017. Effects of talker dialect, gender & race on accuracy of Bing speech and YouTube automatic captions. In Proc. Interspeech 2017, pages 934-938.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine Mcleavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak su- pervision.
Morgane Riviere, Jade Copet, and Gabriel Synnaeve. 2021. ASR4REAL: An extended benchmark for speech models.
Autumn Toney- and Aylin Caliskan. 2021. Val- Norm quantifies semantics to reveal consistent va- lence biases across languages and over centuries. In Proceedings of the 2021 Conference on Empir- icul Methods in Natural Language Processing, pages 7203-7218, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Carlos Romero-Rivas, Charlotte Morgan, and Thomas Collier. 2021. Accentism on trial: Categoriza- tion/stereotypngg and implicit biases predict harsher sentences for foreign-accented defendants. Journal of Language and Social Psychology, (1(2):191-208
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc
Frank Rudzicz, Aravind Kumar Namasivayam, and Talya Wolff. 2012. The TORGO database of acoustic and articulatory speech from speakers with dysarthria. Language Resources and Evaluation, 46:523-541.
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 202 la. VoxPop- uli: A large-scale multilingual speech corpus for rep- resentation learning. semi-supervised learning and interpretation. Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 993-1003, Online. Association for Computational Linguistics.
James A. Russell. 1980. A circumplex model of af- fect. Journal of Personality and Social Psychology, 1966))1111-11777 161-1178. 188. Publisher: American Psychologi- cal Association Place: US.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. 2019, wav2vec: Unsupervised pre- training for speech recognition. In Proc. Interspeech 2019, pages 3465-3469.
Tali Sharot and Neil Garrett. 2016. Forming beliefs: Why valence matters. Trends in Cognitive Sciences, 20(1)335-33.
Chengyi Wang. Yu Wu, Yao Qian, Kenichi Kumatani. Shujie Liu, Furu Wei, Michael 7eng, and Xuedong Huang. 2021b. UniSpeech: Unified speech repre- sentation learning with labeled and unlabeled data. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10937 10947. PMLR
Ryan Steed and Aylin Caliskan. 2021. Image represen- tations learned with unsupervised pre-traine pre-training con- tain human-like like biases. Proceedings of the 2021 ACM Conference on Fairness, Accountabiliry, and Transparency, pages 701-713, New York, NY, USA. Association for Computing Machinery.
Ryan Steed, Swetasudha Panda, Ari Kobren, and Michael Wick. 2022. Upstream mitigation is not all you need: Testing the bias transfer hypothesis in pre-trained language models. Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 3524-3542, Dublin, Ireland. Association for Compu- tational Linguistics.
Zeyu Wang. Klint Qinami, Ioannis Christos Karakozis. Kyle Genova, Prem Nair, Kenji Hata, and Olga Rus- sakovsky. 2020. Towards fairness in visual recog- nition: Effective strategies for bias mitigation. Proceedings of the IEFF/CVE Conference on Com- puter Vision and Pattern Recognition (CVPR).
Millicent Weber. 2021. "Reading" the public domain: Narrating and listening to librivox audiobooks. Book History, 24( 24(1):209-243
Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang, Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Belding, Kai-Wei Chang, and William Yang Wang- 2019. Mitigating gender bias in natural language
Steven Weinberger. 2015. Speech Accent Archive.

---

Robert M. West. 2021. Best practice in statisties: Use the Welch t-test when testing the difference be- tween two groups. Annals of clinical biochemistry, 58(4):267-269.
A Statistical Significance of SpEAT'
Along with an effect size metric d, many prior works concerning EATs also include results from Null Hypothesis Significance Test (NHST) (Caliskan et al., 2017; Steed and Caliskan, 2021; May al., 2019). This NHST tests the null hypoth- esis that the EAT for model and set of concepts equal to (or equivalently that stimuli represent- ing the first target concept and stimuli representing the second target concept are equal terms of their relative distances to stimuli representing the two attribute concepts). The NHST tests this null hy- pothesis against the alternate hypothesis that the EAT d is greater than 0 for model and set of concepts.
Robert Wolfe and Aylin Caliskan. 2022a. American == White in multimodal language-and-image AI. In
Proceedings of the 2022 AAAT/ACM Conference on
AI, Ethies, and Society, AIES '22. pages 800-811 -12. New York, NY, USA. Association for Computing Machinery.
Robert Wolfe and Aylin Caliskan. 2022b. VAST: The Valence-Assessing Semantics Test for contextual- izing language models. Proceedings of the AAAI Conference Artificial Intelligence, 36(10); 1477- 11485.
Robert Wolfe, Yiwei Yang. Bill Howe, and Aylin Caliskan. 2023, Contrastive language-vision AI mod- els pretrained on web-scraped multimodal data ex- hibit sexual objectification bias. Proceedings of the 2023 ACM Conference on Fairness, Accountabil- ity, and Transparency, pages 1174-1185, Association for Computing Machinery.
We include results from this NHST in Figure 4. We find that 63 of 96 tests (66%) are significant at the a = 0.01 level. Because performing such a large number of NHSTs unison increases the probability of accidentally rejecting true null hy- pothesis at least once, we also perform a Bonferroni correction for multiple comparisons. Bonferroni corrections make the threshold required to claim significance in individual tests more strict, lower- ing the probability of accidentally rejecting a true null hypothesis in any individual test. We find that 46 of the 63 tests (73%) that were significant at the a= 0.01 level remain significant after the correc- tion.
Shu-wen Yang. Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi. Xuankai Chang, Guan- Ting Lin, Tzu-Hsien Huang. Wei-Cheng Tseng, Ko- tik Loe, Da-Rong Liu. Zili Huang, Shuyan Dong, Shang- Ween Li, Shinji Watanabe, Abdelrahman Mo- hamed, and Hung-yi Lee. 2021. SUPERB: Speech Processing Universal PRRformance Benchmark. In Proc. Interspeech 2021, pages 1194 1198.
B Further Details of Models
Speech
Below, we provide further details concerning the pre-trained speech models that we evaluate bias in.
B.1 wav2vec 2.0
We test wav2vec 2.0 (Baevski et al., 2020), speech representation learning framework originally pub- lished in 2020, due to its wide usage. As of submis- sion in June of 2023, wav2vec 2.0-based models make up seven of the ten most downloaded ASR models over the last month in HuggingFace's 's trans- formers library. In the wav2vec 2.0 family, we use the non-finetune con-finetuned wav2vec 2.0 Base and wav2vec 2.0 Large (both trained on Librispeech (Panayotov et al., 2015)), as well as the non-finetuned wav 2vec 2.0 Large (LV-60), (which was trained Libri- Light (Kahn et al., 2020)). During the pre-training phase, wav2vec 2.0 models are trained using a con- trastive loss to identify continuous sequences audio. The model takes wav files as input, splits

---

Significaat Correction
Abled and Disabled-
Siguificam at 0.01 Ouly
EA and AA
Not Signifiennt
Female and Male
Human and Syuthesized
U.S. and Foreign
Young and Old
lubert base Jest
hubert Largee litik
hubert stralorge lttil
ward large I6Gok
wae2ree2. bnsec
warDhee2 Jarge, letei)
way In. haner
way base plus
wavim barge
w whispor. Jaso, en
whispor. whisper. hase Jarge to whisper, medliu an whisper moetium
whisper. small en
whisper small
Figure 4: Statistical significance of SpEAT's. We find that 63 of 96 tests (66%) are significant at the a = 0.01 level. Of the tests significant the 0.01 level, 46 (73%) are also significant after Bonferroni correction for multiple comparisons.
them into a sequence of chunks, embeds each of the chunks in latent space using a set of feature encoding layers, then masks some elements in the sequence of latent representations. The latent rep- resentations are contextualized using set of se- quence encoding layers, and the model is then tasked with identifying the correct masked chunk of sound from set of random distractor samples, using the chunks that came prior in the sequence as context.
et al., 2015) test-other and test-clean datasets, mea- sure the Word Error Rate (WER) for a clean speech set and a noisy speech set, respectively. (WER being the mean edit distance, at the word level, between predicted and true transcriptions). In the HuBERT family we use all available pre-trained models that were not finetuned: Base, Large, and Extra Large. The Base version of HuBERT trained using Librispeech, while the larger versions were trained using Ligri-Light. addition to its performance on LibriSpeech, HuBERT Large is currently ranked only beneath WavLM models the public SUPERB Challenge, which measures speech representation models on how well they adapt to a battery of downstream tasks, including ASR, ST, and speaker diarization." 10
Architecturally, the feature encoding portion of the model, which operates on each element in the sequence of audio chunks individually, contains seven blocks. Each block contains a temporal convolution step, a layer normalization step, and a Gaussian Error Linear Unit (GELU) activation, which use 512 channels, strides of (5,2,2,2,2,2,22) and kernel widths of (10,3,3,3,3,2,22. The sequence encoder portion contains transformer blocks 24 blocks in the largest version of the model, each with 16 attention heads. We take embeddings from the end of each transformer block (as well as from before the first transformer block).
HuBERT inherits much of its architecture from prior unsupervised speech models, but also incor- porates pseudo-labeling component adopted from the vision model DeepCluster (Caron et al., 2018). Like its predecessors way2vec (Schneider et al., 2019) and way2vec 2.0 (Baevski et al., 2020), Hu- BERT contains a feature encoder portion, for learn- ing individual representations of sequenced snip- pets of sound, followed by sequence encoder portion, used to build contextualized representa- tions from the individual representations. Rather than try to predict masked representations directly, HuBERT's loss involves the prediction of psuedo- labels, which it creates itself, using k-means clus- tering. Unlike wav2vec and wav2vee 2.0 models,
B.2 HuBERT
Hidden-Unin BERT (Hsu et al., 2021) (HuBERT) is a current open source state-of-the-an framework for learning representations of speech data. A fine-tuned HuBERT model achieves the first-best and third-best score of all open source models on two widely-used benchmarks for measuring per- formance in ASR, according to Papers With Code. These benchmarks, the LibriSpeech (Panayotov
1hehe leaderboard can be found at
http::/ssneehhsccchuuuk..ooooooe www..uur....g.lcccbbrr..ddo

---

HuBERT uses cross-entropy loss for pre-training, predict the psuedo-label that the next snippet be- longs to. It alternates between generating clusters and predicting those clusters.
B.4 Whisper
The final model family we test is Whisper (Rad- ford et al.. 2022). While Whisper does not outper- form wav2vec 2.0 on the LibriSpeech test-clean benchmark, it does outperform wav2vec 2.0 at tran- scribing wide array of other datasets, implying Whisper may be more robust. Whisper models can be used directly for ST, spoken language identifi- cation, voice activity detection, and multilingual ASR. In the Whisper model family, we use the Base, Small, Medium, and Large versions, as well as any English-only variants used for these models, We use the 1.0 versions of these models, which were the only versions available when our experi- ments were run.
Each layer in the feature encoder blocks of Hu- BERT consists of a temporal convolution, a layer normalization (Ba et al., 2016), and GELU acti- vation (Hendrycks and Gimpel, 2016). There are 7 layers in the feature encoder portion, each of which has 512 channels. The sequence encoder portion of HuBERT models contain between 12 and 48 transformer blocks and between and 16 attention heads, depending on the model size (Hsu et al., 2021). During pre-training a projection layer is applied to the output from the sequence encoder portion of the model, as well as a code embedding layer. For extracting representations from HuBERT models, we take embeddings from the end of each transformer block, relying the embedding traction implementation published with the original paper.
A difference between Whisper and the other model families tested is that Whisper uses a sequence-to-seqqenec encoder-decoder trans- former architecture. Log-mel spectrograms are first created from audio inputs, which are then fed into convolutional layers and a GELU activation layer (Hendrycks and Gimpel, 2016), before being passed on to between four and 32 transformer coder blocks (Vaswani al., 2017), depending on the model size. The decoder portion of the model contains between four and 32 transformer decoder blocks as well. We take embeddings from after each of the transformer encoder blocks. We test both the multilingual and English-only variants of the Base, Small, and Medium versions, as well as the multilingual Large version (which does not have an English-Only variant)."
B.3 WayLM
WavLM (Chen et al., 2022a) is another family of state-of-the-ard representation learning models for speech data. As of this paper's submission WavLM is ranked first the public SUPERB Challenge, implying that its pre-trained weights adapt well wide variety of speech tasks. In the WavLM family, we also use all available versions: Base, Base+, and Large. WavLM Base uses Ligri-Light for training, however the WavLM Base+ and Large versions extend the training data to GigaSpeech and VoxPopuli. WavLM is based on HuBERT, but uses different training setup and slightly different architecture. In addition to the extended dataset, WavLM models also employ data pre-processing technique whereby 20 percent of pre-training in- puts are mixed with other noise, while their labels from the k-means portion of the model are left untouched, to make the model more robust. Archi- tecturally, WavLM models involve gated relative position biases that are incorporated into their trans- former blocks, whereas HuBERT models use con- volutional relative position embeddings. We take embeddings from the end of each transformer block (as well from before the first transformer block). We use the embedding extraction implementation published with the original paper.
C Further Details on Target Stimuli
We now provide details the stimuli used for representing target concepts.
C.1 Abled and Disabled Speakers
We adapt the test performed by Nosek et al. (2007), originally using images representing abled and dis- abled people, to speech. We elect to represent dis- ability and ability using speech from dysarthric and non-dysarthric speakers, respectively. Dysarthria is speech condition caused by muscle weakness, that is considered be noticeably low in valence to human listeners (Lass al., 1988). While dysarthric speakers do not represent all disabled people, IAT stimuli should be easily distinguish- able across target groups (Greenwald et al., 2022),
For clarity, despite using the multilingual versions of the models, we only use English stimuli for our tests.

---

and we choose dysarthria as one relevant speech condition, for which speech data is readily avail- able. We leave testing of other forms of disability that manifest in speech to future work.
data from two separate datasets do so: The first is the Buckeye Corpus (Pitt et al., 2007), which contains speech entirely from Caucasian speakers, and the second is the Corpus of Regional African American Language (CORAAL) (Kendall and Far- ringion, 2021), which contains speech entirely from speakers of African-American Language. Both datasets consist of sociolinguistic interviews of variable length, recorded on high quality equip- ment.
We use speech from the TORGO Database to represent speech from disabled and abled speakers. The database contains speech samples from eight dysarthric and seven con-dysarthric speakers, who are approximately equal in mean age across con- ditions. Speakers were given a variety of prompts for different stimuli, including single words, sen- tences designed to elicit accent differences, well images used elicit natural and unrestricted sentences. In order to capture speech style differ- ences, we use speech samples that contain two or more words. We match samples gender and speech content across conditions, but are unable to match in age, as ages of individual speakers were not included with the dataset.
We start with the full Buckeye Corpus, which was published in 2007, as well with all CORAAL components that were recorded after the year 2000 and available in October of 2022. We process the datasets by isolating speech from the interview sub- jects, removing any non-speech noises, and remov- ing any clips from speakers who are younger than 18 years of age. We match by speaker gender, in- terviewer gender, and speaker age. (The Buckeye Corpus does not contain an exact age, only infor- mation whether the speaker is younger than 30 or older than 40, so calculate the same feature in CORAAL and use this as our age feature.)
To test the extent to which results from the TORGO Database extend to other datasets, we also carry out the SpLAAT using speech from the Univer- sal Access Speech (UASpeech) dataset (Kim et al., 2008) and present the results in Appendix E. We use the most recently cleaned version of UASpeech, which was processed in 2020 using noisereduce. UASpeech consists of recordings of words read by dysarthric and non-dysarthrie speakers. The UASpeech set is the largest dataset of which we are aware that contains matched English speech from dysarthric and non-dysarthric speakers, how- ever because recordings in the UASpeech are of single words phrases, rather than complete sen- tences, which may not fully capture accent differ- ences across conditions, we elect only present results from the TORGO database in the main body of the paper. Speakers were paired across condi- tions based on approximate age in UASpeech (be- ing within three years in age of each other), as well as by gender. As the corpus contains more than 100,000 recordings in total, we elect to sample the number of recordings to be on par with the sample sizes used for EATS performed in other modalities. We randomly sample five words per speaker pair, which gives 55 samples represent each group.
Like UASpeech, CORAAL and Buckeye both contain many more recordings than have been used in EATS in other modalities, to conserve sources we again sample the datasets. We match by gender and approximate age, then take balanced stratified sample, (by speaker age and speaker gen- der), to ensure that no age or gender groups are over or under represented in our sample, and the SpEAT scores do not pertain to speakers of any specifc age gender group more than another. We take 15 samples per cell, which gives us 60 sam- ples from speakers of each ethnicity. We perform a paired samples t-test ensure that audio clips do not differ in length on average, order to establish that any potential differences association are not simply due to differences in audio clip length. A t-test is appropriate due to the sample size of the dataset, and paired samples test is required due to the matched relationship between the audio clips. We do not find evidence that clips between the two groups differ in length on average (t(59) -0.12, p = 0.90), which implies that differences in asso- ciation are not attributable to differences in length of the input audio clips.
C.2 European-American Speakers and African-American Speakers
C.3 Female and Male Speakers
We also adapt the test by Nosek et al. (2007) of the concepts of European- American and African- American to the speech modality. We take speech
We also adapt audio IATs from (Mitchell et al., 201 1) for comparing female and male speech. We

---

use speech from the Speech Accent Archive (SAA), a corpus containing over 2,800 samples of English speech from unique speakers, as well as metadata on the speakers' ages (Weinberger, 2015). Data in the archive consists of high quality recordings of speakers reading an identical elicitation para- graph, and range between 10 and 80 seconds long. While the SAA does not contain label indicating whether speech is in SAE, it does contain informa- tion on speakers' places of birth, and in order to ensure that the speech we use is representative of SAE, (which was used for the original tests per- formed on humans), we limit our sample to speak- ers whose place of birth is in the United States, (Speakers must also have English listed as their na- tive language.) We match by age across conditions.
online generator. In addition samples created using the default speeds provided by the generator, we also create fast and slow versions of each voice, where the fast version has a speed parameter 1 1.25 times the default value, and the slow version has a speed parameter 0.75 times the default. These syn- thetic samples do not have associated ages, but do have associated genders. which we are able to use for matching the synthetically generated samples to speech from the SAA. (Because human speakers in the original IAT were speakers of U.S. English, we filter to only include speakers whose native lan- guage is English and who were born in the United States.)
To test whether results from the SAA general- ize to other datasets, we also perform the human speaker and synthesized speech test using the same stimuli as (Mitchell et al., 2011). We present these results in Appendix F. Stimuli are the same samples consisting of single words or phrases as those used in Appendix C.3, however now split by whether the sample contains speech from human speaker or from synthetic voice, rather than by gender. We test 64 samples per target group.
To test the extent to which results from the SAA generalize to other datasets, we also perform the female and male test using the speech samples that were originally shown to humans in the audio IATs performed by Mitchell et al. (2011). We present these results in Appendix E. The data for these tests come from four distinct human speakers and four distinct synthesized voices: two human female speakers, two human male speakers, two synthetic female voices, and two synthetic male voices. The synthetic voices come from Microsoft and Read- Please. Each speaker or synthetic voice is used to generate speech from the same 16 short neu- tral phrases, such as "candle holder" or "cardboard box." We test 64 samples per target group.
C.5 United States and Foreign Accented Speakers
We also adapt the audio LATs performed by Pan- and Perkins (2012), which compare English language speech from a native speaker of United States English to speech from a native speaker of Korean. Korcan was originally chosen to repre- sent the concept of foreign accent in order to min- imize the associations that listeners might have with stereotypes, as prior work had found that Ko- rean accents were difficult to identify in the United States (Lindemann, 2003). We use speech from the SAA represent both U.S. and Foreign speakers, and use similar filtering criteria to those used by Pantos and Perkins (2012) to select speakers. We use speakers whose native language is English and whose place of birth is in the United States to rep- resent U.S. accented speech, and speakers whose native language is Korean and whose place of birth is in Korca to represent Korean accented speech. Samples were matched on age and gender across conditions.
C.4 Human Speakers and Synthesized Speech
We also adapt the test from Mitchell et al. 12 comparing human and synthesized speech using speech from the SAA. Text to Speech technol- ogy has changed significantly since Mitchell et al. (2011) carried out IATs using synthetic voices, and to ensure that the synthetic voices that we use are similar to the synthetic voices that were orig- inally played for humans, we use similar Text to Speech systems to those used originally. The orig- inal voices used were Microsoft Mary, Microsoft Mike, as well as the ReadPlease Male and Female voices. These voices are included in Microsoft Speech API 4.0 (SAP14.0), from which we also include 15 other synthetic voices in our tests. We generate speech using the same elicitation text as that used for human speakers in the SAA, using an
To evaluate whether results from the SAA ex- tend to other datasets, we validate the test results
As of submission, code for the generator is available at https//giitab.co///EiYYYSSS

---

Layer Agg.
Sum
Min
Max
First
Second
Q!
Q2
Q3
Penultimate
Last
Mean
0.82
0.72
0.70
0.52
0.69
0.80
0.92
0.88
0.76
0.73
Temporal
Min
0.66
0.66
0.71
0.61
0.80
0.77
0.74
0.65
0.64
0.57
Agg.
Max
0.66
0.56
0.68
0.47
0.70
0.82
0.69
0.65
0.68
0.59
Table 2: Proportion SpEAT ds are positive, and therefore congruent with biases that tend to be found humans, when different strategies used for aggregating raw embeddings. Temporal Agg. refers to the first step of ieoegaation-ummmiiiiin
across the variably-sized sequence dimension of the raw embeddings (whose length depends the duration of the input speech sample). Layer Agg. refers to the second step aggregattg across Transformer layers. For aggregation across layers, aggregation strategies other than sum, min, and max involve taking only embeddings from an individual layer, for example Q2 refers to selecting embeddings from the median layer in the model (e.g., layer 24 in the 48 layer HuBERT XI.). Layer aggregation strategies that involve selecting from single layer are denoted in italics.
using the original stimuli played humans. The original tests use speech samples read by two sim- ilarly aged male actors in their native accents as target stimuli. The actors read the same eight neu- tral phrases, for example "at this point", and the phrases are repeated three times for human listen- ers. We use the exact samples shown to humans, (i.e. those that repeat the phrases three times).
there is some variability between strategies. Taking the mean across the temporal dimension tends result in more alignment with human biases than
taking the min or max, for example, and selecting embeddings from early to middle layers also tends to result in more alignment with human biases.
E SpEAT Results Using Alternate Stimuli Sets
C.6 Young and Old Speakers
As described in Appendix C, we also perform EATs using alternate stimuli, to test the extent which the results we find would generalize to other datasets. The SpEATs comparing U.S. and For- eign speech using the stimuli originally used by Pantos and Perkins (2012) are opposite in mean from the tests comparing U.S. and foreign speech using speech from the SAA, however other tests are similar in mean when comparing original to alternate stimuli sets. We hypothesize that this may be related to the number of stimuli used by Pantos and Perkins (2012) represent each target concept-8 stimuli, rather than the more than 50 stimuli that were used for all other tests performed- or related to the fact that speech samples are from two speakers (one to represent U.S. speech and one to represent Foreign speech), and contain single words or phrases repeated three times, rather than lengthy and fluid samples such as that contained in the SAA.
We also adapt the IATs performed in (Nosek et al.. 2007) comparing the concepts of young and old. We use speech from the SAA, and limit ourselves to speakers whose birth place is in the United States. We take the oldest and youngest speakers to repre- sent the concepts of young and old. To ensure that our results are not more applicable to male or to female speakers, we balance the samples based on gender. We take 58 speakers per target group (29 young male speakers, 29 young female speakers, and so on). The young category consists of speak- ers between ages 18 and 19, while the old category consists of speakers between ages 57 and 93,
D Other Embedding Aggregation Strategies
Similar to work from May et al. (2019), who test various sequence and layer aggregation in text- based models, we evaluate whether our results would have changed had a different strategy been used for summarizing embeddings, (other than us- ing the mean across the temporal dimension then the sum across layers). The proportion of SpEAT ds that are positive are shown Table All but one of the aggregation strategies result in more positive SpEAT values than negative, however
F Downstream Speech Emotion Recognition Models
Here we describe the training process for the mod- els used for the downstream task of SER, which is taken from Yang et al. (2021). After extraction

---

Abled and Disabled (TORGO)
Indivdd. Model
Moaa SpEAT
Abled and Disabled (UASpeach)
Moan IAT
Female and Male (SAA)
Female and Male (Mitchell et al.)
Human and Syathesized (SAA)
Human and Syuthesized (Mitchell es al.)
U.S. and Foreiga (SAA)
U.S. and Foreign (Pantos and Perkins)
1.5 -1.0 0.5
0.0
0.5
1.0 1.5
Figure 5: SpEAT values showing the extent to which results extend to other datasets that can represent the same concepts. Speech samples in original datasets (TORGO and SAA) longer and include more words, and were hypothesized to better capture differences in speaking style. Three of four tests using alternae stimuli show similar mean SpEAT values to original tests, while the fourth (which uses a small number of samples, only includes speech from single speaker to represent each target concepts, and contains speech samples consisting of single words phrases repeated three times) docs not.
from the upstream pre-trained model, each embed- ding starts as shape (N Layers, Seq Len, Embd Dim). The first dimension (N Layers) corresponds to the number of layers in the pre-trained model. The second dimension (Seq Len) corresponds to the length of the input speech sample. The third dimension (Embd Dim) corresponds to the width of the pre-trained model. The downstream model first takes weighted average over the first dimension of the embeddings, where the weights are tuned in the process of training the downstream model, al- lowing the downstream model to give more weight to layers that are more relevant to valence predic- tion. The downstream model then makes a first linear projection to a fixed size of 256, before tak- ing unweighted average over the audio sequence, and then second linear projection down to sin- gle number. We train the downstream model us- ing mean squared error loss, Adam optimization (Kingma and Ba, 2014), and for a maximum of 20,000 steps. These hyperparameters, as well as the model's 's architecture, are based on those used for the emotion recognition task by Yang et al. (2021), although we decrease the total number of steps, as valence datasets are significantly smaller than those used originally, and switch MSE loss, we are predicting continuous ratings of valence rather than categorical emotions. We train three downstream models for each pre-trained model, one using a learning rate of 10-3 one using a learn-
ing rate of 10-4, and one using learning rate of 10 We then use predictions from all downstream models when calculating Cohen's d.
G Embedding Association Tests for U.K. English
To test the extent to which our results extend be- yond U.S. English, we also perform tests using U.K. English.
G.1 Stimuli
We now describe the stimuli used for tests concern- ing U.K. English.
G.1.1 Target Stimuli
British and Foreign Accented Speakers We adapt the audio IAT performed in (Romero-Rivas et al., 2021) comparing speech from native speaker of British Received (BR) English with speech from a speaker of Spanish accented En- glish. The original test uses speech samples read by two female actors, one using BR English, and the other using Spanish accented English. The ac- tors each read 16 neutrally-valenced phrases, for example "table." Some participants in the original IAT receive versions of the audio stimuli with back- ground noise added, while others received stimuli free of background noise, giving two versions of each sample. We therefore test 32 samples for each group.

---

British and Foreign
Indivdd. Model
Moaan SpEAT
Mean IAT D
Young and Old (U.K.)
-1.5
-1.0
1.5
Figure 6: SpEAT d values showing that results do not appear to generalize U.K.-accented speech. The overall mean IAT D value for the British and Foreign test was not provided by Romero- Rivas et al. 12 however they calculate D value related British accents 0.60 and D value related Foreign accents -0.46.
Young and Old Speakers We also perform a test of age for U.K. English, and use similar tar- get dataset as for U.S. English, sourced from the SAA (Weinberger, 2015). There are 82 speech samples in the dataset from speakers born in the United Kingdom. A tradeoff needed to be made be- tween larger group sizes, which can provide more precision in tests, and smaller group sizes, which provide better representations of the categories of young and old. The value of 12 is within typical ranges for sample sizes of past EAT's The young category consists of speakers between ages 18 and 21 while speakers in the old category are between ages 38 and 72. As in the U.S. dataset, we balance our sample on gender.
each of the valence poles, giving us 54 recordings for positive valence and 54 recordings for negative valence. To establish that any potential differences in association between gender and valence mea- sured by the SpEAT are not due how the samples were rated by humans, we ensure that human rat- ings of valence not differ based on the speaker's gender. We perform a Welch's unequal variances (-test to evaluate whether the ratings of valence differ by gender on average. A t-test is appropri- ate due to the sample size of the dataset, and we use Welch's t-test rather than Student's t-test due to its robustness (Delacre al., 2017; West, 2021). We do not find evidence that clips from male and female speakers are rated differently on valence on average (f(437) = 0.81, implying that any differences in association with valence between gender are not due to differences in how the audio clips were rated by humans. We also establish that potential differences in associ- ation between age and valence measured by the SpEAT are not due to how the samples were rated by humans. We test whether age has bivariate relationship with valence using a simple linear re- gression model fit using ordinary least squares. A regression is appropriate due to the continuous na- ture of both variables. We do not find evidence for a relationship between speaker age and valence (1(438)
0.42).
G.1.2 Attribute Stimuli: EU-Emotion Stimulus Set
When performing the SpEAT using stimuli based U.K. accents, use EU-Emotion Stimulus Set (EUESS) rather than the MESS represent pos- itive and negative valence (O'R (O'Reilly et al., 2016; O' O'Reilly et al., 2012). The EUESS is a corpus of 695 recordings made by actors and rated by listen- ers in the United Kingdom. Actors were speakers of British English, and were asked to read sets of sentences each portraying
of twenty categor- ical emotional states. The corpus contains both 255 semantically emotional sentences and 440 se- mantically neutral sentences, of which we only use the semantically neutral sentences, in order to cen- ter our work on acoustics. (Semantically neutral sentences were those that were possible to read in multiple emotional states, for example "T knew it would happen.") A minimum of 20 listeners pro- vided ratings of valence for each recording. Actors varied in age and gender, and we again sample positive and negative valence within speaker to min- imize the influence of speaker-buse based confounders. We take three recordings from each speaker for
-0.00, p 0.37), implying that differ- ences in association with valence by age not due differences how the audio clips were rated by humans. Analysis does not indicate any depar- tures from the lincar regression model, implying that it is not inappropriate for the data. A residual plot for the model berween age and valence in the EUESS is shown in Figure 7. Based on the lack of pattern in the local average line (in red), we do not evidence for nonlinear relationship between the independent and dependent variables. We also do not see evidence for heteroscedasticity in the

---

Residuals vs Fitted
Residnals
2.76
2.78
2.80 Fitted values
2.82
Figure 7: Residual plot for regression between age and valence in EUESS dataset, which contains speech from speakers U.K. English rated on valence. We do not evidence for heteroscedasticity for non-linearity, suggesting that assumptions from the regression model are not violated.
residuals. Due to the sample size (440), an assump- tion of normality is not necessary for use of the t-test, and we therefore do not perform test for normality of the residuals.
G.2 Test Results
G.2.1 British and Foreign Accented Speakers
10 of 16 models show a positive SpEAT d value. We note the British and Foreign Accented samples are based on only 16 unique recordings from each the two unique speakers, and for this reason may not fully representative of the intended concepts. The locality of the speech (being from a British speaker) may also affect the results.
G.2.2 Young and Old Speakers
Only two models show bias against speech from older speakers from the United Kingdom. We note that due to the data available in the SAA, the U.K. test only involves twelve speech samples to repre- sent each concept. We also note that the speech samples used to represent older speakers in the U.K. also have much lower age cutoff than older speak- ers in the United States, which may mean the target concepts are not as differentiated from each other.


Reviews:
[{"review_id": "W3DURC6j1n", "reviewer": "Reviewer_b7KZ", "summary": "", "questions": "A. Do the authors have intuitions on why model predictions seem to have similar biases to humans? I would have thought that some of the bias may be because of labelling by annotators, but it appears that models pre-trained in a self-supervised manner (without any labels) carry this bias as well.\n\nB. Could you please clarify why summing the activations across multiple layers is useful/necessary when works like Chen et. al. and Pasad et. al. [1] have identified methods to estimate layers that may correlate with valence or emotion?\n\nC. There are multiple categories of large-scale speech models today - did the authors find any commonalities in the type and extent of bias in models (a) trained on the same data or (b) trained in the same manner? Some conclusions in this respect would be useful for future work. \n\n[1] Comparative layer-wise analysis of self-supervised speech models, Ankita Pasad, Bowen Shi and Karen Livescu, ICASSP 2023", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer b7KZ,\n\nThank you for your feedback. We have included in line responses to your comments and questions below.\n\n***\n\n\n> A. The method used to generate embeddings, i.e., temporal averaging per layer and summing across layers is quite possibly error-ridden. As the authors have pointed out, different layers of large-scale pre-trained models are more useful to particular tasks, and summing activations across different layers may result in less discriminative features.\n\nWhile we note that the method we use for generating embeddings is taken from a prior EAT [1], we thought this concern might also be addressed by evaluating the extent to which our results would have changed had different aggregation strategies been used. Across layers, we tested aggregating by using either the sum, min, or max—or by simply selecting embeddings from the first layer, second layer, first quartile layer, median layer, third quartile layer, penultimate layer, or last layer. Across the temporal dimension, we aggregated either by taking the mean, min, or max. \n\nIn the paper, we report that when using the sum (layers)/mean (temporal) to generate embeddings, 82% of SpEAT $d$s are positive. When considering all 30 aggregation strategies, we find that 69% percent of the 2,880 tests show positive SpEAT $d$s. In the paper we show that when using the sum/mean to generate embeddings, the SpEAT $d$ averaged across the 16 models is in the same direction as the related IAT $D$ from human experiments for 5 of the 6 social groups (83%) considered. We in fact find that considering all 30 aggregation strategies and 6 social groups, exactly 150 of 180 (83%) average SpEAT $d$ values are in the same direction as the related IAT $D$. We also find that there are more positive SpEAT $d$ values than negative when using all but one aggregation strategy. If our work is accepted, we will include these results in the final manuscript’s appendix. We are also happy to answer further questions concerning this experiment. \n\n***\n\n\n> B. Further, SpEAT evaluations appear to be done with relatively small sample sizes, which is likely suboptimal. In many cases, this appears to be due to the availability of data, but this remains a challenge unaddressed.\n\nWe use between 57 and 60 stimuli to represent each social group (or valence pole), resulting in between 234 and 240 stimuli being used in each test. These sample sizes are larger than those used in any test performed either by Caliskan et al. when introducing the Word Embedding Association Test [3] or by Steed and Caliskan when introducing the Image Embedding Association Test [2].\nRegardless of this context, however, we do not think that larger sample sizes are necessary to support the claims made in our work. This is because of the way that we are using the stimuli, which is to estimate, first, the SpEAT $d$ and, second, an effect size concerning the difference in predicted valence for speech coming from two groups of people. \n\nWe see smaller sample sizes as suboptimal mainly in that they will result in less precise estimates of these statistics. In our work estimating the Standard Error (SE) of the SpEAT $d$, however, we found that across all tests and speech models, the average estimated SE was 0.14 when 50 samples were used to represent target concepts, and the maximum estimated SE for any condition was 0.20 when 50 samples were used, (with both mean and maximum SE monotonically decreasing as the number of samples increased beyond 50). We see these SEs as already fairly small relative to the size of the SpEAT $d$ values estimated (whose average magnitude was 0.70 across all sets of stimuli and models), and therefore do not think that, had the sample size been increased, the uncertainty associated with the SpEAT $d$ would have depreciated by a meaningful amount. We have not estimated SEs for Cohen's $d$, but do not anticipate the results to have been significantly different had we done so.\n\nWith that said, we recognize that larger sample sizes may come along with a different method for generating stimuli (i.e., using speech samples from other sources), and that testing alternative methods of generating stimuli may indeed lead to interesting results. We think that using the SpEAT with stimuli from a variety of different sources may be a promising direction for future work.\n\n\n***\n\n\n> A. Do the authors have intuitions on why model predictions seem to have similar biases to humans? I would have thought that some of the bias may be because of labelling by annotators, but it appears that models pre-trained in a self-supervised manner (without any labels) carry this bias as well.\n\nWork in other modalities [2] has suggested that co-occurrence may be related to EAT results. In the case of the speech data, this may occur if people of one social group tend to more frequently speak with positive valence than people of another social group. We think that co-occurrence in the pre-training data might be a good place to start if one were interested in investigating how these results arise. \n\n***\n> C. There are multiple categories of large-scale speech models today - did the authors find any commonalities in the type and extent of bias in models (a) trained on the same data or (b) trained in the same manner? Some conclusions in this respect would be useful for future work.\n\nWe did notice a slight model family effect, with models in the same family seeming to have somewhat of a tendency toward similar SpEAT $d$ values concerning a given social group. We performed a two-way ANOVA to confirm this, using the SpEAT $d$ value as the dependent variable and the social group and model family as the independent variables. The interaction between model family and social group was just below the typical p=0.05 significance threshold, (p=0.048), suggesting, as suspected, that there may be a slight effect of model family regarding a social group. The main effect of model family was not significant, however, suggesting that it is not the case that SpEAT $d$ values for a family tend to be located in the same area across all social groups. For example, Whisper SpEAT $d$ values tend to be similar to one another in the African-American and European-American test, but Whisper SpEAT $d$ values for the African-American and European-American test do not tend to be similar to Whisper SpEAT $d$ values from the Young and Old test. With that said, model family is related to both the way the models were trained and the training dataset used, and we don’t believe that the information we have available allows for strong conclusions regarding these factors. \n\n***\n\nThank you again for your time and effort. We will review the introduction for readability if our work is accepted.\n\n*** \n[1] May, C., Wang, A., Bordia, S., Bowman, S. R., & Rudinger, R. (2019). On Measuring Social Biases in Sentence Encoders. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 622–628. https://doi.org/10.18653/v1/N19-1063\n\n[2] Steed, R., & Caliskan, A. (2021). Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 701–713. https://doi.org/10.1145/3442188.3445932\n\n[3] Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334), 183–186. https://doi.org/10.1126/science.aal4230", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. This paper focuses on biases of pre-trained models - an important topic for the speech community to study and actively pursue as large-scale models are being used and deployed in real-world scenarios. \n\nB. This paper compares speech model biases to human biases which is interesting and novel in application if not methodology to the best of my knowledge. \n\nC. The paper is detailed in describing experimental goals, experiments, and results and tests a broad variety of speech models including Wav2Vec variants, Whisper variants, and HUBERT variants.", "reasons_to_reject": "A. The method used to generate embeddings, i.e., temporal averaging per layer and summing across layers is quite possibly error-ridden. As the authors have pointed out, different layers of large-scale pre-trained models are more useful to particular tasks, and summing activations across different layers may result in less discriminative features. \n\nB. Further, SpEAT evaluations appear to be done with relatively small sample sizes, which is likely suboptimal. In many cases, this appears to be due to the availability of data, but this remains a challenge unaddressed. \n\nEDIT:\n\nThe authors have provided a convincing justification that addresses part B.", "questions_to_authors": "A. Do the authors have intuitions on why model predictions seem to have similar biases to humans? I would have thought that some of the bias may be because of labelling by annotators, but it appears that models pre-trained in a self-supervised manner (without any labels) carry this bias as well.\n\nB. Could you please clarify why summing the activations across multiple layers is useful/necessary when works like Chen et. al. and Pasad et. al. [1] have identified methods to estimate layers that may correlate with valence or emotion?\n\nC. There are multiple categories of large-scale speech models today - did the authors find any commonalities in the type and extent of bias in models (a) trained on the same data or (b) trained in the same manner? Some conclusions in this respect would be useful for future work. \n\n[1] Comparative layer-wise analysis of self-supervised speech models, Ankita Pasad, Bowen Shi and Karen Livescu, ICASSP 2023", "paper_topic_and_main_contributions": "This paper addresses the question - do large pre-trained speech models carry markers of bias similar to humans? The authors extend the technique of Embedding Association Tests (EAT) to speech forming SpEAT to first (a) validate whether prediction of valence (used for Affect prediction or SER) differs across 6 social groups, and (b) finally examine the impact of this bias on Speech Emotion Recognition. \n\n\nPrior work in diagnosing bias from speech models has focused on reporting performance differences across data representing different social groups, and in reporting downstream performance of fine-tuned large-scale pre-trained speech models on these different groups. This work in contrast uses pre-trained model embeddings directly and compares them to human biases obtained through IATs. The authors consider activations from multiple layers in the large-scale model, compute temporal averages within every layer, and then sum across layers to obtain the embedding they test with. \n\n\nThe authors verify (a) whether the pretrained speech models carry similar biases to humans by comparing SpEAT d values to IAT test D values from previous work, (b) whether the biases propagate when fine-tuning for downstream Speech Emotion Recognition, and (c) how much SpEAT values vary when changing the sample size of the stimuli (participants ). The authors find that many of the models carry biases similar to humans, and that biases propagate to downstream SER models.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "The introduction is highly technical and hard to read for a person unfamiliar with EATs - at a high level, I suggest maybe going through it and simplifying it where possible for non-expert readers."}, "extra_text": {}}, {"review_id": "meE8nxcZG2", "reviewer": "Reviewer_QLo4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer QLo4,\n\n\nThank you for your review. We have tried to clarify the concepts and arguments that you raised concerns about here, and if our work is accepted we will use some of the additional space for the same purpose.\n\n\n***\n\n\n> (3) argued more cogently for the relationship between \"valence\" (allegedly descriptive of how a person feels) and \"pleasantness\" (supposedly descriptive of how a person makes another person feel)\n\n\nWe apologize for not defining valence more clearly in our work’s introduction. Valence is a concept used for categorizing emotions, and is in fact often defined directly in psychology literature as “pleasantness” or “pleasure,” or used interchangeably with these terms [4-6]. Valence is a part of many dimensional models of emotion, which consider emotions to be composed of different (sometimes independent) dimensions. For example, the circumplex model [4] categorizes emotions using valence and arousal. This model considers excitement to have positive valence and positive arousal, contentment to have positive valence and negative arousal, distress to have negative valence and positive arousal, and so on. We have used the terms “valence” and “pleasantness” interchangeably, and apologize for the confusion this has caused. If our work is accepted we will plan on both defining valence more clearly at the outset and removing references to “pleasantness” that are not used for defining valence.\n\n\n***\n\n\n> (2) included a conceptual description of how that bias is computed\n\n\nThe SpEAT considers where in a model’s embedding space two concepts related to demographics, such as European-American and African-American, are located, relative to the concepts of positive and negative valence. A SpEAT involves passing stimuli representing these four concepts into a speech model, and retrieving the respective embeddings. The SpEAT is computed by first calculating how close embeddings representing one demographic concept are to positive valence relative to negative valence, then comparing this to how close embeddings representing the other demographic concept are to positive and negative valence. The SpEAT gives an effect size $d$, which when large in magnitude indicates that one group tends to be closer than the other group to positive valence relative to negative valence. A positive SpEAT $d$ indicates that the group that is closer to positive valence is also the group that tends to be more associated with positive valence by humans.\n\n\n***\n\n\n> (1) provided a single, highlighted definition of the (authors') operationalized definition of bias \n\n\nBias is a complex concept with many, often contradictory, ways of being operationalized [2], and we apologize for not describing how the SpEAT relates to bias more clearly. We consider a large SpEAT $d$ magnitude to be a bias, although it is not the only bias we consider in the paper. The other biases we consider are bias in the downstream task of Speech Emotion Recognition (SER), and bias in humans measured using Implicit Association Tests (IATs). In the downstream task of SER, we consider a disparity in predicted valence between matched speech samples coming from two demographic groups to be a bias, (which we will discuss in more detail in the following in line comment). In humans we consider a large IAT $D$ magnitude to be a bias.\n\n\nWe view Embedding Association Tests (EATs) as measurements of bias in fact because of the relationship between distances in embedding spaces and other types of bias. Distance between embeddings has been found to be associated with biases in society, both in our work as well as in related work in Natural Language Processing and Computer Vision. In related work, it has been found that distances between embeddings for models trained on text from different time periods reflect social conditions from those time periods [3], and that distances between embeddings reflect real world statistics, like the proportion of people holding a certain job who are male versus female [1]. In our work, we find that SpEAT $d$ values tend to be in the same direction as average IAT $D$ values. We also find that the SpEAT is associated with disparities in predicted valence in downstream SER models. \n\n\n***\n\n\n> (4) argued more cogently for the relationship between \"valence\" in a downstream SER system and \"bias\" as it is found in earlier portions of this article\n\n\nAs noted above, we consider a disparity in predicted valence between speech samples coming from two demographic groups to be a bias in the downstream task of SER. The focus of this portion of our work is on how biases found in pre-trained models with the SpEAT may propagate to downstream tasks. To study this, we train downstream models (one for each pre-trained speech processing model) to predict the valence of speech samples, a form of SER. We then use these models to predict the valence of speech samples coming from people with the demographic characteristics that we studied in the pre-trained models. For example, we predict the valence of speech coming from African-American speakers and European-American speakers. \n\n\nWhen testing for bias, the samples that we predict the valence of consist either of speech read in a controlled environment or extemporaneous speech taken from socio-linguistic interviews. In addition to being collected in similar fashions, the clips were also often matched on gender and approximate age, as we describe in Section 4 of the paper. We consider disparities in predicted valence to be a bias because of the matched nature between the clips across the demographic groups studied.\n\n\n***\n\n\nIn addition to the requested clarifications, which we think will improve the readability of the paper, we’d also like to thank you for your attention to detail regarding grammar. We will review the paper again for grammatical errors if it is accepted. \n\n\n***\n\n\n[1] Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334), 183–186. https://doi.org/10.1126/science.aal4230 \n\n\n[2] Blodgett, S. L., Barocas, S., Daumé III, H., & Wallach, H. (2020). Language (Technology) is Power: A Critical Survey of “Bias” in NLP. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 5454–5476. https://doi.org/10.18653/v1/2020.acl-main.485\n\n\n[3] Garg, N., Schiebinger, L., Jurafsky, D., & Zou, J. (2018). Word embeddings quantify 100 years of gender and ethnic stereotypes. Proceedings of the National Academy of Sciences of the United States of America, 115(16), E3635–E3644. https://doi.org/10.1073/pnas.1720347115\n\n\n[4] Russell, J. A. (1980). A circumplex model of affect. Journal of Personality and Social Psychology, 39(6), 1161–1178. https://doi.org/10.1037/h0077714\n\n\n[5] Nielen, M. M. A., Heslenfeld, D. J., Heinen, K., van Strien, J. W., Witter, M. P., Jonker, C., & Veltman, D. J. (2009). Distinct brain systems underlie the processing of valence and arousal of affective pictures. Brain and Cognition, 71(3), 387–396. https://doi.org/https://doi.org/10.1016/j.bandc.2009.05.007\n\n\n[6] Morgan, S. D. (2019). Categorical and Dimensional Ratings of Emotional Speech: Behavioral Findings From the Morgan Emotional Speech Set. Journal of Speech, Language, and Hearing Research, 62(11), 4015–4029. https://doi.org/10.1044/2019_JSLHR-S-19-0144", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The treatment of bias for speech models is novel.", "reasons_to_reject": "The treatment of bias is likely exotic for most readers. The current manuscript misses some opportunities for making the related concepts plain and crisp. The article would likely be much improved if it: (1) provided a single, highlighted definition of the (authors') operationalized definition of bias; (2) included a conceptual description of how that bias is computed; (3) argued more cogently for the relationship between \"valence\" (allegedly descriptive of how a person feels) and \"pleasantness\" (supposedly descriptive of how a person makes another person feel); and (4) argued more cogently for the relationship between \"valence\" in a downstream SER system and \"bias\" as it is found in earlier portions of this article. Several of these concepts/attempts are provided loosely, scattered throughout the article, or made/hinted at only in parentheses.\n\nLines 596 and 606 mention \"(speaking) style\" -- this appears late in the article, is not defined, and thereby retains an unclear relationship with the authors' arguments.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The article describes a new method for detecting bias in pre-trained speech models, demonstrates using this method that many existing state-of-the-art models exhibit bias, and demonstrates that downstream systems relying on such models' output also exhibit bias.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "105: \"positively valence\"?\n413: agrammatical\n573: \"SE varying\"?\n643: \"relationship\" is \"uncorrelated\"?\n"}, "extra_text": {}}, {"review_id": "c4t1c4XwjT", "reviewer": "Reviewer_FQKD", "summary": "", "questions": "   If you using public models, it would be useful to disclose the actual models you used. For example, could you please provide the exact links or identifiers for the wav2vec2.0 models you used? This would greatly enhance the transparency of your study and make it easier for readers to understand your methodology and reproduce your results. Additionally, as certain models like Whisper Large have version 1 and an updated version 2, providing the specific versions used can help avoid confusion.\n  \n  Do you think the lack of synthesized speech in pre-training models could explain the difference between upstream and downstream models for synthesized speech (as shown in figure 2). Are you considering experimenting with models pre-trained on a significant synthesized speech data? Might this change the bias behaviors depicted in figures 1 and 2. Furthermore, could using synthesizing speech data (based on actual speech) to minimize biases in pre-training models create a significant bias between synthesized and human speech, while reducing other biases? I'd appreciate your views on this.\n", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer FQKD, \n\n\nThank you for your comments and suggestions. We provide in line responses below.\n\n\n***\n\n\n> It did not become completely clear to me how much the SpEAT adds on the previously proposed EAT methods.\n\n\nThe aim of our work was to study bias in speech processing models, and we chose to do so using the EAT, a method that has been studied extensively in the Natural Language Processing and Computer Vision communities. We saw making methodological improvements to EATs as mostly out of scope. Once embeddings have been extracted from a speech model, the SpEAT in fact does not differ in process from a form of the Sentence Embedding Association Test. We see our primary additions to the EAT literature as 1) introducing an EAT for models in a new modality, 2) showing that EATs generalize across modalities and architectures, and 3) adding evidence that biases found with EATs can propagate to downstream tasks.\n\n\nWe note one methodological contribution, however, which is that we have shown how bootstrapping can be used to estimate statistical properties of the EAT, such as the Standard Error. We hope that this can be built upon by others in the future, for example for constructing confidence intervals for EAT $d$ values, or for estimating the sample size that would be necessary to achieve a confidence interval of a certain size. \n\n\n***\n\n\n>The statistical significance of the results are somewhat difficult to assess.\n\n\nIn light of this comment, we ran hypothesis tests for each of the individual SpEATs performed. Prior works concerning EATs have typically touched on statistical significance using a permutation-based Null Hypothesis Significance Test (NHST) [1,2,3]. This NHST tests the null hypothesis that the EAT $d$ in a specific model is equal to 0 for the specific set of concepts against the alternative hypothesis that the EAT $d$ is larger than 0.\n\n\nWe perform this test for each of the 6 sets of concepts in each of the 16 speech models, and find that 63 of 96 tests (66%) are significant at the 0.01 level. Of the tests significant at the 0.01 level, 46 (73%) are also significant after a Bonferroni correction for multiple comparisons. We will include these results in the appendix of our work, if it is accepted. We are also happy to answer further questions concerning these tests. \n\n\n***\n\n\n> If you using public models, it would be useful to disclose the actual models you used. For example, could you please provide the exact links or identifiers for the wav2vec2.0 models you used? This would greatly enhance the transparency of your study and make it easier for readers to understand your methodology and reproduce your results. Additionally, as certain models like Whisper Large have version 1 and an updated version 2, providing the specific versions used can help avoid confusion.\n\n\nWe apologize for the lack of detail. For the Whisper family we use the 1.0 versions of the models, and for the wav2vec 2.0 model family we use the non fine-tuned versions of the models. For the wav2vec2.0 family, the model versions are provided on lines 1228-1233 in the appendix of our submission. For the HuBERT family, the versions are provided on lines 1273-1275. For the WavLM family, the versions are provided on lines 1328-1329. Finally, for the Whisper family, the versions are provided on lines 1355-1358. \n\n\nWe will add the missing information to future versions of our work, which we believe should be sufficient for finding the models that we used. In order to facilitate replication, however, we have also included links to the wav2vec 2.0, WavLM, and HuBERT models used below. We will include these links in the main README for the code released with the project as well, if our work is accepted. We have not added links for the Whisper models, as these are downloaded automatically by the scripts that we have provided.\n\n\n* [wav2vec 2.0 Base (No finetuning/Librispeech)](https://github.com/facebookresearch/fairseq/blob/4db264940f281a6f47558d17387b1455d4abd8d9/examples/wav2vec/README.md)\n* [wav2vec 2.0 Large (No finetuning/Librispeech)](https://github.com/facebookresearch/fairseq/blob/4db264940f281a6f47558d17387b1455d4abd8d9/examples/wav2vec/README.md)\n* [wav2vec 2.0 Large (No finetuning/Libri-Light)](https://github.com/facebookresearch/fairseq/blob/4db264940f281a6f47558d17387b1455d4abd8d9/examples/wav2vec/README.md)\n* [HuBERT Base (No finetuning/Librispeech)](https://github.com/facebookresearch/fairseq/tree/4db264940f281a6f47558d17387b1455d4abd8d9/examples/hubert)\n* [HuBERT Large (No finetuning/Libri-Light)](https://github.com/facebookresearch/fairseq/tree/4db264940f281a6f47558d17387b1455d4abd8d9/examples/hubert)\n* [HuBERT Extra Large (No finetuning/Libri-Light)](https://github.com/facebookresearch/fairseq/tree/4db264940f281a6f47558d17387b1455d4abd8d9/examples/hubert)\n* [WavLM Base](https://github.com/microsoft/unilm/tree/f4695ed0244a275201fff00bee495f76670fbe70/wavlm)\n* [WavLM Base+](https://github.com/microsoft/unilm/tree/f4695ed0244a275201fff00bee495f76670fbe70/wavlm)\n* [WavLM Large](https://github.com/microsoft/unilm/tree/f4695ed0244a275201fff00bee495f76670fbe70/wavlm)\n\n\n***\n\n\n> Do you think the lack of synthesized speech in pre-training models could explain the difference between upstream and downstream models for synthesized speech (as shown in figure 2). Are you considering experimenting with models pre-trained on a significant synthesized speech data? Might this change the bias behaviors depicted in figures 1 and 2. Furthermore, could using synthesizing speech data (based on actual speech) to minimize biases in pre-training models create a significant bias between synthesized and human speech, while reducing other biases? I'd appreciate your views on this.\n\n\nThe lack of synthesized speech in the pre-training datasets used by these speech models may explain some results in the experiments comparing human and synthesized speech. It’s difficult to say how synthetic data may change biases, however, or how biases relate to one another, especially without better information on the dialectal and demographic characteristics of the speech that make up common speech datasets. The quality of synthesized speech has been improving though, and if newer speech processing models are trained with synthetic speech, we may see interesting developments in the future.\n\n\n***\n\n\nThank you for raising the concerns above, as well for the grammar and style suggestions. We will address these in the final manuscript if the paper is accepted, and will also make another pass at reviewing the paper ourselves. \n\n\n*** \n\n[1] May, C., Wang, A., Bordia, S., Bowman, S. R., & Rudinger, R. (2019). On Measuring Social Biases in Sentence Encoders. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 622–628. https://doi.org/10.18653/v1/N19-1063\n\n[2] Steed, R., & Caliskan, A. (2021). Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 701–713. https://doi.org/10.1145/3442188.3445932\n\n[3] Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334), 183–186. https://doi.org/10.1126/science.aal4230", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "    The paper introduces a new method called Speech Embedding Association Test (speEAT) for detecting biases in pre-trained models. This is a noteworthy contribution as the method can be applied to a broad range of widely-adopted model architectures. Moreover, it is important to note that this research addresses an under-explored area in the field.\n\n    The results of this paper also raise awareness about the social biases within speech models. While the paper primarily focuses on SER, it also raises ethical discussion about the necessity of developing unbiased pre-trained speech models in the future.\n\n    The code for SpeEAT method will also be published, which could server as a good foundation to develope more sophisticated bias detection method in the future.\n", "reasons_to_reject": "It did not become completely clear to me how much the SpEAT adds on the previously proposed EAT methods. \n\nThe statistical significance of the results are somewhat difficult to assess.\n", "questions_to_authors": "   If you using public models, it would be useful to disclose the actual models you used. For example, could you please provide the exact links or identifiers for the wav2vec2.0 models you used? This would greatly enhance the transparency of your study and make it easier for readers to understand your methodology and reproduce your results. Additionally, as certain models like Whisper Large have version 1 and an updated version 2, providing the specific versions used can help avoid confusion.\n  \n  Do you think the lack of synthesized speech in pre-training models could explain the difference between upstream and downstream models for synthesized speech (as shown in figure 2). Are you considering experimenting with models pre-trained on a significant synthesized speech data? Might this change the bias behaviors depicted in figures 1 and 2. Furthermore, could using synthesizing speech data (based on actual speech) to minimize biases in pre-training models create a significant bias between synthesized and human speech, while reducing other biases? I'd appreciate your views on this.\n", "paper_topic_and_main_contributions": "The paper introduces a novel method to identify biases (disability, race, gender, accent, age) in pre-trained speech models. Experiments are presented including popular speech models: wav2vec 2.0, HuBERT, WavLM, Whisper. The paper also proves that the pre-trained speech models learn the biases available in the large training dataset and can propagate those to the downstream task of Speech Emotion Recognition (SER).\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The Paper \"GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audio\" was published in  interspeech 2021 (10.21437/Interspeech.2021-1965). It's better to cite that instead of arXiv\n\nThe citation \"202\" and \"2023c\" in page 14 point to the same reference, probably an error\n", "Typos_Grammar_Style_and_Presentation_Improvements": "There are a few very long sentences: E.g. On page 5: \"Stimuli are matched on gender as well as approximate age, with the exception of the female and male test, which is not matched on gender, the young and old test, which is not matched on age, the human and synthesized test, which is not matched on age, 6 and the abled and disabled test (for which age information was not made available for speakers).\"\n\nOn page 7: There might be a missing word in this sentences: \"Work in computer vision has suggested that biases detected by EATs may arise from co-occurrence in the training data (Steed and Caliskan, 2021), and we hypothesize that co-occurrence may contribute here as well, for example if speakers who use a ??? speech style tends to more frequently express positive valence than others in the training set.\"\n\nOn page 14 (two \"a\"): embeds each of the chunks in latent space using a a set of feature encoding layers \n\nOn page 19 (two \"are\"): There are are 82 speech samples in the dataset \n\nWorfe et al., 2023 reference should at least capitalize the word \"AI\" (Contrastive Language-Vision {AI} Models Pretrained on Web-Scraped Multimodal Data Exhibit Sexual Objectification Bias)\n\nThere are a few more errors in capitalize the proper noun in reference (like english instead of English), please check that.\n "}}, {"review_id": "fcABPAi1n0", "reviewer": "Reviewer_PR28", "summary": "", "questions": "A: how many seeds have been used to train the downstream models?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer PR28,\n\n\nThank you for your comments and questions. We have addressed these in line below. \n\n\n***\n\n\n> The paper does not consider that such models have been found to also pick up linguistic knowledge similar to large Language Models (Wagner, Johannes, et al. \"Dawn of the transformer era in speech emotion recognition: closing the valence gap.\" IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).). This may be partly relevant here, as (if I understand correctly) the speakers in the European-American vs. African-American dataset do not all say the same text, but deliver individual answers to interview questions.\n\nBecause embeddings for speech from African-American speakers tend to be closer to negative valence in our tests of pre-trained models (and tend to be predicted as lower in valence by downstream models) we think that if linguistic differences were to explain the results in the AA/EA tests, the linguistic content of the speech sourced from African-American speakers would need to be lower in valence than the linguistic content of the speech sourced from European-American speakers. To evaluate this, we passed the transcripts of the speech from the AA/EA test through five popular sentiment analysis tools [1-5]. \n\nIn all but one tool we used, the transcripts from African-American speakers in fact have *higher average positive sentiment* and *lower average negative sentiment* than transcripts from European-American speakers. The other tool (VADER) showed the transcripts from African-American speakers as higher in *both* average positive sentiment and average negative sentiment, but outputs a compound score combining positive, negative, and neutral sentiment, which was higher for the transcripts of African-American speakers. \n\nRegardless of these results, however, we think it is worth noting that in all other tests the content of the speech is matched across demographic groups. \n\n***\n> It is not clarified if several seeds have been used for the fine-tuning process. The paper reads like there was only one seed, which would be a bit of a limitation, as oftentimes, finetuning results are heavily seed-dependent.\n\nFor the results reported in the paper, we do in fact only use one seed when training downstream models to predict valence. In early experiments, however, we performed light hyperparameter tuning by training downstream models using the learning rates 1e-3, 1e-4, and 1e-5. We only discuss the models fit using the 1e-4 learning rate in the paper, in order to follow Yang et al.’s [6] method for adapting pre-trained models to downstream tasks as closely as possible. We think this question raises a good point regarding the reliability of the results when only 16 downstream models are used, however, and for this reason chose to report on the other models below. For the most part, we do not see large performance disparities between the models fit with different learning rates. Excluding one downstream model, which was fit using a learning rate of 1e-5 on embeddings from the Libri-Light version of wav2vec 2.0 Large and had a negative $R^2$, the average $R^2$ values for the 1e-3, 1e-4 and 1e-5 models were 0.91, 0.88, 0.89, respectively. We will plan on switching to discussing results from all three models in our paper, if it is accepted. \n\nWhen considering the additional models, we do not find that our results change significantly. In the paper, we report that when using only downstream models trained with the 1e-4 learning rate, in “65 of the 96 (68%) tests performed, the group more associated with positive valence by the pre-trained model is also more associated with positive valence by the downstream SER model.” Similarly, when using predictions from all three downstream models, there are 66 of 96 (69%) of tests where the group more associated with positive valence by the pre-trained model is also more associated with positive valence by the downstream SER models. \n\nIn the SER experiments we measured bias in the downstream task using Cohen’s $d$, an effect size metric summarizing the difference in mean predicted valence between the two social groups. We do not find that including the predictions from the models trained with other learning rates results in Cohen’s $d$ values that we see as substantially different. When we compare predictions from all three downstream models to predictions from only the 1e-4 model, we find that the average absolute difference in Cohen’s $d$ is 0.13. This is relative to an average Cohen’s $d$ magnitude of 0.79 when using the 1e-4 models or 0.71 when using the 1e-3, 1e-4, and 1e-5 models together. In the paper we state that “in all tests excluding that related to human and synthesized speech, the average Cohen’s $d$ across all 16 speech models is in the same direction as the average SpEAT $d$ across the models.“ This does not change when including the additional downstream models. \n\n***\n\nThank you for your suggestions and concerns, including those raised regarding style. We agree that a portion of the ethics statement (namely, the second paragraph) would be better placed in the discussion section, and will plan on making this change, along with the others you suggest, if our work is accepted.\n\n***\n\n[1] Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, and Jose Camacho-collados. 2022. TimeLMs: Diachronic Language Models from Twitter. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 251–260, Dublin, Ireland. Association for Computational Linguistics.\n\n[2] Francesco Barbieri, Luis Espinosa Anke, and Jose Camacho-Collados. 2022. XLM-T: Multilingual Language Models in Twitter for Sentiment Analysis and Beyond. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 258–266, Marseille, France. European Language Resources Association.\n\n[3] Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1644–1650, Online. Association for Computational Linguistics.\n\n[4] [Seethal/sentiment_analysis_generic_dataset · Hugging Face](https://huggingface.co/Seethal/sentiment_analysis_generic_dataset)\n\n[5] Hutto, C., & Gilbert, E. (2014). VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text. Proceedings of the International AAAI Conference on Web and Social Media, 8(1), 216–225. https://doi.org/10.1609/icwsm.v8i1.14550\n\n[6] Yang, S., Chi, P.-H., Chuang, Y.-S., Lai, C.-I. J., Lakhotia, K., Lin, Y. Y., Liu, A. T., Shi, J., Chang, X., Lin, G.-T., Huang, T.-H., Tseng, W.-C., Lee, K., Liu, D.-R., Huang, Z., Dong, S., Li, S.-W., Watanabe, S., Mohamed, A., & Lee, H. (2021). SUPERB: Speech Processing Universal PERformance Benchmark. Proc. Interspeech 2021, 1194–1198. https://doi.org/10.21437/Interspeech.2021-1775", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper addresses a relevant research question\n- It provides a notably large number of experiments which cover different pretrained models and different kinds of biases\n- The reported experiments seem to be designed very carefully (selection of speech samples etc.)\n- The paper reads well and is well-structured", "reasons_to_reject": "(EDIT: the questions have been sufficiently addressed in the authors' response)\n\n- The paper does not consider that such models have been found to also pick up linguistic knowledge similar to large Language Models (Wagner, Johannes, et al. \"Dawn of the transformer era in speech emotion recognition: closing the valence gap.\" IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).). This may be partly relevant here, as (if I understand correctly) the speakers in the European-American vs. African-American dataset do not all say the same text, but deliver individual answers to interview questions.\n\n- It is not clarified if several seeds have been used for the fine-tuning process. The paper reads like there was only one seed, which would be a bit of a limitation, as oftentimes, finetuning results are heavily seed-dependent.", "questions_to_authors": "A: how many seeds have been used to train the downstream models?", "paper_topic_and_main_contributions": "The paper proposes a test (SpEAT) to measure bias for/against certain demographics in pretrained speech models such as WavLM. SpEAT is based on comparing embeddings of speech by the demographic groups in question to embeddings of speech that is deemed of low and high valence, respectively. The results show that oftentimes, the models actually show bias comparable to bias reported for humans. Moreover, the authors provide evidence that these biases are often propagated to the downstream taks of speech emotion recognition.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The Ethics Statement (9) sounds more like a Discussion/Conclusion in large parts, e.g. when suggesting approaches to tackle biases. I suggest to put most of these considerations into the main part of the paper in case of acceptance.\n\n- Equations (line 257, line 269) should be put in a equation environment in LaTex.\n\n- 261-262: clarifying that A represents pleasantness and B unpleasantness in this example would help\n- 343: appears -> appear\n- 589: wording a bit unclear here: \"can easily learn\" - it is clear that these models *can* learn such biases, the question here seems rather whether the pretrained models *did in fact* adopt human biases during pretraining?"}}]

Author rebuttal:

Meta-review: The paper proposes a way to quantify an intrinsic bias in the pre-trained model and shows the observation of a human-like bias in the pre-trained model and downstream as well. The authors addressed several questions and concerns from reviewers and showed potential evidence to strengthen the paper upon final acceptance.
Decision: Accept-Findings