SlowGuess/ACL_2020_OCR huggingface dataset:
We specifically report results for binary gender (male, female), two-class religion (Christianity, Islam) and three-class race (white, Hispanic, and Asian), directly following Garg et al. (2018). We study bias with respect to target word lists of professions  $\mathcal{N}_{\text{prof}}$  and adjectives  $\mathcal{N}_{\text{adj}}$ . These results are by no means intended to be comprehensive with regards to the breadth of bias socially and only address a restricted subset of social biases which notably does not include intersectional biases. The types of biases being evaluated for are taken with respect to specific word lists (which are sometimes subjective albeit being peer-reviewed) that serve as exemplars and definitions of bias are grounded in the norms of the United States. All word lists are provided in Appendix B and are sourced in §A.6.
Layer-wise Bias Trends. In Figure 2, we report layer-wise bias across all (attribute, definition) pairs. We clearly observe that for every social attribute, there is a great deal of variation across the layers in the quantified amount of bias for a fixed bias estimator. Further, while we are not surprised that different bias measures for the same social attribute and the same layer assign different absolute scores, we observe that they also do not agree in relative judgments. For gender, we observe that the bias estimated by the definition of Manzini et al. (2019) steadily increases before peaking at the penultimate layer and slightly decreasing thereafter. In contrast, under bias
we see a distribution with two peaks corresponding to layers at the start or end of the pretrained model with less bias within the intermediary layers. For estimating the same quantity, bias is mostly uniform across the layers. Similarly, in looking at the religious bias, we see similar inconsistencies with the bias increasing monotonically from layers 2 through 8 under bias, decreasing monotonically under bias, and remaining roughly constant under bias. In general, while the choice of  $\mathcal{N}$  (and the choice of  $\mathcal{A}_i$  for gender) does affect the absolute bias estimates, the relative trends across layers are fairly robust to these choices for a specific definition.
Consequences. Taken together, our analysis suggests a concerning state of affairs regarding bias quantification measures for (static) word embeddings. In particular, while estimates are seemingly stable to some types of choices regarding word lists, bias scores for a particular word embedding are tightly related to the definition being used and existing bias measures are markedly inconsistent with each other. We find this has important consequences beyond understanding the social biases in our representations. Concretely, we argue that without certainty regarding the extent to which embeddings are biased, it is impossible to properly interpret the meaningfulness of debiasing procedures (Bolukbasi et al., 2016; Zhao et al., 2018a,b; Sun et al., 2019) as we cannot reliably estimate the bias in the embeddings both before and after the procedure. This is further compounded with the existing evidence that current intrinsic measures of social bias may not handle geometric behavior such as clustering (Gonen and Goldberg, 2019).
Cross-Model Bias Trends. In light of the above, next we compare bias estimates across different pretrained models in Table 3. Given the conflicting scores assigned by different definitions, we retain all definitions along with all social attributes in this comparison. However, we only consider target words given by  $\mathcal{N}_{\text{prof}}$  due to the aforementioned stability (and for visual clarity) with results for  $\mathcal{N}_{\text{adj}}$  appearing in Table 8. Since we do not preprocess or normalize embeddings, the scores using bias are incomparable (and may be improper to compare in the layer-wise case) as
<sup>5</sup>We slightly modify the definition of Manzini et al. (2019) by (a) using cosine similarity where they use cosine distance and (b) inserting absolute values around each term in the mean over  $\mathcal{N}$ . We make these changes to introduce consistency with the other definitions and to permit comparison.
Figure 2: Layer-wise bias of distilled BERT-12 embeddings for  $f = \mathrm{mean}$ ,  $g = \mathrm{mean}$ ,  $N = 100\mathrm{K}$ .
B, P	GE, P	GC, P	Gender M, P	GE	GC	M	Race M	GE	Religion GC	M
Word2Vec	0.0503	0.1758	0.075	0.2403	0.1569	0.0677	0.2163	0.0672	0.0907	0.053	0.14
GloVe	0.0801	0.3534	0.0736	0.1964	0.357	0.0734	0.1557	0.1171	0.2699	0.0702	0.0756
BERT-12	0.0736	0.3725	0.0307	0.3186	0.2868	0.0254	0.3163	0.2575	1.2349	0.0604	0.2955
BERT-24	0.0515	0.6418	0.0462	0.234	0.4674	0.0379	0.2284	0.1956	0.6476	0.0379	0.2316
GPT2-12	0.4933	25.8743	0.0182	0.6464	2.0771	0.0062	0.7426	0.6532	4.5282	0.0153	0.776
GPT2-24	0.6871	40.1423	0.0141	0.8514	2.3244	0.0026	0.9019	0.8564	8.9528	0.0075	0.9081
RoBERTa-12	0.0412	0.2923	0.0081	0.8546	0.2077	0.0057	0.8551	0.8244	0.4356	0.0111	0.844
RoBERTa-24	0.0459	0.3771	0.0089	0.7879	0.2611	0.0064	0.783	0.7479	0.5905	0.0144	0.7636
XLNet-12	0.0838	1.0954	0.0608	0.3374	0.6661	0.042	0.34	0.2792	0.8537	0.0523	0.318
XLNet-24	0.0647	0.7644	0.0407	0.381	0.459	0.0268	0.373	0.328	0.8009	0.0505	0.368
DistilBERT-6	0.0504	0.5435	0.0375	0.3182	0.3343	0.0271	0.3185	0.2786	0.8128	0.0437	0.3106
Table 3: Social bias encoded within different pretrained models with respect to a set of professions  $\mathcal{N}_{\text {prof }}$ . Parameters are discussed in the supplement. Lowest bias in a particular column is denoted in bold.
they are sensitive to the absolute norms of the embeddings. Further, we note that bias may not be a reliable indicator since the first principal component explains less than  $35\%$  of the variance for the majority of distilled embedding (Zhao et al. (2019a) show similar findings for ELMo). For bias and bias, we find that all distilled static embeddings have substantially higher scores under bias but generally lower scores under bias GARG-COS when compared to Word2Vec and GloVe. Interestingly, we see that under bias both GPT-2 and RoBERTa embeddings consistently get high scores when compared to other distilled embeddings but under bias they are deemed the least biased.
Data alone does not determine bias. Comparing the results for BERT-12 and BERT-24 (full layer-wise results for BERT-24 appear in Figure 11) reveals that bias trends for BERT-12 and BERT-24 are starkly different for any fixed
bias measure. What this indicates is the bias observed in contextualized models is not strictly determined by the training data (as these models share the same training data as do all other 12 and 24 model pairs) and must also be a function of the architecture, training procedure, and/or random initialization.
Takeaways. Ultimately, given the aforementioned issues regarding the reliability of bias measures, it is difficult to arrive at clear consensus of the how the bias encoded compares between our distilled representations and prior static embeddings. What our analysis does resolutely reveals is a pronounced and likely problematic effect of existing bias definitions on the resulting bias estimates.
Contextualized  $\rightarrow$  Static. Recently, Akbik et al. (2019) introduced an approach that gradually aggregates representations during training to accumulate global information and demonstrated improvements over only contextualized representations for NER. May et al. (2019) instead syntheti
6 When we normalized using the Euclidean norm, we found the relative results to reliably coincide with those for bias which is consistent with Garg et al. (2018).
cally construct a single semantically-bleached sentence which is fed into a sentence encoder to yield a static representation. In doing so, they introduce SEAT as a means for studying biases in sentence encoders by applying WEAT (Caliskan et al., 2017) to the resulting static representations. This approach appears inappropriate for quantifying bias in sentence encoders<sup>7</sup> as sentence encoders are trained on semantically-meaningful sentences and semantically-bleached constructions are not representative of this distribution and their templates heavily rely on deictic expressions which are difficult to adapt for certain syntactic categories such as verbs (as required for SIMVERB3500 especially). Given these concerns, our reduction method may be preferable for use in estimation of bias in contextualized representations. Due to the fact that we use mean-pooling, our approach may lend itself to interpretations of the bias in a model on average across contexts.
Ethayarajh (2019) considers a similar method to ours where pooling is replaced by PCA. While this work demonstrated contextualized representations are highly contextual, our work naturally explores the complementary problem of what value can be extracted from the static analogue of these representations.
Bias. Social bias in NLP has been primarily evaluated in three ways: (a) using geometric similarity between embeddings (Bolukbasi et al., 2016; Garg et al., 2018; Manzini et al., 2019), (b) adapting psychological association tests (Caliskan et al., 2017; May et al., 2019), and (c) considering downstream behavior (Zhao et al., 2017, 2018a, 2019a; Stanovsky et al., 2019). Our bias evaluation is in the style of (a) and we consider multi-class social bias in the lens of gender, race, and religion whereas prior work has centered on binary gender. Additionally, while most prior work has discussed the static embedding setting, recent work has considered sentence encoders and contextualized models. Zhao et al. (2019a) consider gender bias in ELMo when applied to coreference systems and Kurita et al. (2019) extend these results by leveraging the masked language modeling objective of BERT. Similarly, Basta et al. (2019) considers intrinsic gender bias in ELMo via gender-swapped
sentences. When compared to these approaches, we study a broader class of biases under more than one bias definition and consider more than one model. Further, while many of these approaches generally neglect reporting bias values for different layers of the model, we show this is crucial as bias is not uniformly distributed throughout model layers and practitioners often do not use the last layer of deep Transformer models (Liu et al., 2019a; Zhang et al., 2020; Zhao et al., 2019b).<sup>9</sup>
7 Future Directions
Our work furnishes multiple insights about pretrained contextualized models that suggest changes (subword pooling, layer choice, beneficial variance reduction via averaging across contexts) to improve downstream performance. Recent models have combined static and dynamic embeddings (Peters et al., 2018; Bommasani et al., 2019; Akbik et al., 2019) and our representations may also support drop-in improvements in these settings.
While not central to our goals, we discovered that our static embeddings substantially outperform Word2Vec and GloVe under intrinsic evaluation. Future research may consider downstream gains as improved static embeddings are critical for resource-constrained settings and may help address environmental concerns in NLP (Strubell et al., 2019), machine learning (Canziani et al., 2016), and the broader AI community (Schwartz et al., 2019). Future research could explore weighting schema in the averaging process analogous to SIF (Arora et al., 2017) for sentence representations computed via averaging (Wieting et al., 2016).
The generality of the proxy analysis method implies that other interpretability methods for static embeddings can also be considered. Further, post-processing approaches beyond analysis/interpretability such as dimensionality reduction may be particularly intriguing given that this is often challenging to perform within large multilayered networks like BERT (Sanh et al., 2019) but has been successfully demonstrated for static embeddings (Nunes and Antunes, 2018; Mu and Viswanath, 2018; Raunak et al., 2019).
Future work may revisit the choice of the corpus  $\mathcal{D}$  from which contexts are drawn. For downstream use, setting  $\mathcal{D}$  to be the target domain may serve as a lightweight domain adaptation strategy similar to findings for averaged word representations for
The authors also identified several empirical concerns that draw the meaningfulness of this method into question.
${}^{8}$  Sun et al. (2019) provides a taxonomy of the work towards understanding gender bias within NLP.
9This is the only layer studied in Kurita et al. (2019).
out-of-domain settings (Wieting et al., 2016).
8 Discussion and Open Problems
While our work demonstrates that contextualized representations retain substantial representational power even when reduced to be noncontextual, it is unclear what information is lost. After all, contextualized representations have been so effective precisely because they are tremendously contextual (Ethayarajh, 2019). As such, the validity of treating the resulting static embeddings as reliable proxies for the original contextualized model still remains open.
On the other hand, human language processing has often been conjectured to have both context-dependent and context-independent properties (Barsalou, 1982; Rubio-Fernández, 2008; Depraetere, 2014, 2019). Given this divide, our approach may provide an alternative mechanism for clarifying how these two properties interact in the computational setting from both an interpretability standpoint (i.e. comparing results for analyses on the static embeddings and the original contextualized representations) and a downstream standpoint (i.e. comparing downstream performance for models initialized using the static embeddings and the original contextualized representations). However, the precise relationship between the role of context in human language processing and computational language processing remains unclear.
Theoretical explanation for the behavior we observe in two settings is also needed. First, it is unclear why learning contextualized representations and then reducing them to static embeddings drastically outperforms directly learning static embeddings. In particular, the GloVe embeddings we use are learned using 6 billion tokens whereas the BERT representations were trained on roughly half as much data (3.3 billion tokens). Perhaps the behavior is reminiscent of the benefits of modelling in higher dimensional settings temporarily as is seen in other domains (e.g. the kernel trick and Mercer's theorem for learning non-linear classifiers using inner product methods): begin by recasting the problem in a more expressive space (contextualized representations) and then project/reduce to the original space (static embeddings). Second, the reason for the benefits of the variance reduction that we observe are unclear. Given that best-performing mechanism is to average over many contexts, it may be that approaching the asymptotic mean of
the distribution across contexts is desirable/helps combat the anisotropy that exists in the original contextualized space (Ethayarajh, 2019).
In this work, we consider how methods developed for analyzing static embeddings can be re-purposed for understanding contextualized representations. We introduce simple and effective procedures for converting from contextualized representations to static word embeddings. When applied to pretrained models like BERT, we find the resulting embeddings are useful proxies that provide insights into the pretrained model while simultaneously outperforming Word2Vec and GloVe substantially under intrinsic evaluation. We further study the extent to which various social biases (gender, race, religion) are encoded, employing several different quantification schemas. Our large-scale analysis reveals that bias is encoded disparately across different popular pretrained models and different model layers. Our findings also have significant implications with respect to the reliability of existing protocols for estimating bias in word embeddings.
10 Reproducibility
All data, code and visualizations are made publicly available. $^{10}$  Further details are explicitly and comprehensively reported in Appendix A.
We thank Ge Gao, Marty van Schijndel, Forrest Davis, and members of the Mozilla DeepSpeech and Cornell NLP groups for their valuable advice. We especially thank the reviewers and area chairs for their articulate and constructive feedback.
Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşa, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and WordNet-based approaches. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 19-27, Boulder, Colorado. Association for Computational Linguistics.
Alan Akbik, Tanja Bergmann, and Roland Vollgraf. 2019. Pooled contextualized embeddings for named
$^{10}$ https://github.com/rishibommasani/Contextual2Static
entity recognition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 724-728, Minneapolis, Minnesota. Association for Computational Linguistics.
Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In International Conference on Learning Representations.
Jimmy Ba and Rich Caruana. 2014. Do deep nets really need to be deep? In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 27, pages 2654-2662. Curran Associates, Inc.
Solon Barocas, Kate Crawford, Aaron Shapiro, and Hanna Wallach. 2017. The problem with bias: from allocative to representational harms in machine learning. *Special Interest Group for Computing, Information and Society (SIGCIS)*.
Lawrence W Barsalou. 1982. Context-independent and context-dependent information in concepts. Memory & cognition, 10(1):82-93.
Christine Basta, Marta R. Costa-jussà, and Noe Casas. 2019. Evaluating the underlying gender bias in contextualized word embeddings. In Proceedings of the First Workshop on Gender Bias in Natural Language Processing, pages 33-39, Florence, Italy. Association for Computational Linguistics.
Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. J. Mach. Learn. Res., 3:1137-1155.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 4349-4357. Curran Associates, Inc.
Rishi Bommasani, Arzoo Katiyar, and Claire Cardie. 2019. SPARSE: Structured prediction using argument-relative structured encoding. In Proceedings of the Third Workshop on Structured Prediction for NLP, pages 13-17, Minneapolis, Minnesota. Association for Computational Linguistics.
Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183-186.
Alfredo Canziani, Adam Paszke, and Eugenio Culurciello. 2016. An analysis of deep neural network models for practical applications. CoRR, abs/1605.07678.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286, Florence, Italy. Association for Computational Linguistics.
Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, ICML '08, pages 160-167, New York, NY, USA. ACM.
Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. J. Mach. Learn. Res., 12:2493-2537.
Ryan Cotterell and Hinrich Schütze. 2015. Morphological word-embeddings. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1287-1292, Denver, Colorado. Association for Computational Linguistics.
Ilse Depraetere. 2014. Modals and lexically-regulated saturation. Journal of Pragmatics, 71:160-177.
Ilse Depraetere. 2019. Meaning in context and contextual meaning: A perspective on the semantics-pragmatics interface applied to modal verbs. Anglophonia, 28.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55–65, Hong Kong, China. Association for Computational Linguistics.
Allyson Ettinger. 2020. What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34-48.