SlowGuess/ACL_2020_OCR huggingface dataset:
rb724@cornell.edu
Kelly Davis
kdavis@mozilla.com
Claire Cardie
cardie@cs.cornell.edu
Contextualized representations (e.g. ELMo, BERT) have become the default pretrained representations for downstream NLP applications. In some settings, this transition has rendered their static embedding predecessors (e.g. Word2Vec, GloVe) obsolete. As a side-effect, we observe that older interpretability methods for static embeddings — while more mature than those available for their dynamic counterparts — are underutilized in studying newer contextualized representations. Consequently, we introduce simple and fully general methods for converting from contextualized representations to static lookup-table embeddings which we apply to 5 popular pretrained models and 9 sets of pretrained weights. Our analysis of the resulting static embeddings notably reveals that pooling over many contexts significantly improves representational quality under intrinsic evaluation. Complementary to analyzing representational quality, we consider social biases encoded in pretrained representations with respect to gender, race/ethnicity, and religion and find that bias is encoded disparately across pretrained models and internal layers even for models that share the same training data. Concerningly, we find dramatic inconsistencies between social bias estimators for word embeddings.
Word embeddings (Bengio et al., 2003; Collobert and Weston, 2008; Collobert et al., 2011) have been a hallmark of modern natural language processing (NLP) for many years. Embedding methods have been broadly applied and have experienced parallel and complementary innovations alongside neural network methods for NLP. Advances in embedding quality in part have come from integrating additional information such as syntax (Levy and Goldberg, 2014a; Li et al., 2017), morphology (Cotterell and Schütze, 2015), subwords (Bojanowski
et al., 2017), subcharacters (Stratos, 2017; Yu et al., 2017) and, most recently, context (Peters et al., 2018; Devlin et al., 2019). Due to their tremendous representational power, pretrained contextualized representations, in particular, have seen widespread adoption across myriad subareas of NLP.
The recent dominance of pretrained contextualized representations such as ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019) has served as the impetus for exciting and diverse interpretability research: Liu et al. (2019a); Tenney et al. (2019a) study what is learned across the layers of these models, Tenney et al. (2019b); Ethayarajh (2019) consider what is learned from context, Clark et al. (2019); Michel et al. (2019) look at specific attention heads, Hewitt and Manning (2019); Ettinger (2020) address linguistic understanding such as syntax and negation, and Wallace et al. (2019); Tan and Celis (2019) address ethical concerns such as security (adversarial robustness) and social bias. In fact, the neologism BERTology was coined specifically to describe this flurry of interpretability research.<sup>1</sup> While these works have provided nuanced fine-grained analyses by creating new interpretability schema/techniques, we instead take an alternate approach of trying to re-purpose methods developed for analyzing static word embeddings.
In order to employ static embedding interpretability methods to contextualized representations, we begin by proposing a simple strategy for converting from contextualized representations to static embeddings. Crucially, our method is fully general and assumes only that the contextualized model maps word sequences to vector sequences. Given this generality, we apply our method to 9 popular pretrained contextualized representations. The resulting static embeddings serve as proxies for the original contextualized model.
<sup>1</sup>We direct interested readers to a more complete survey of this work from Rogers et al. (2020).
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4758-4781 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
We initially examine the representational quality of these embeddings under intrinsic evaluation. Our evaluation produces several insights regarding layer-wise lexical semantic understanding and representational variation in contextualized representations. Importantly, our analyses suggest constructive improvements to potentially improve downstream practices in using contextualized models. Simultaneously, we find that our static embeddings substantially outperform Word2Vec and GloVe and therefore suggests our method serves the dual purpose of being a lightweight mechanism for generating static embeddings that track with advances in contextualized representations. Since static embeddings have significant advantages with respect to speed, computational resources, and ease of use, these results have important implications for resource-constrained settings (Shen et al., 2019), environmental concerns (Strubell et al., 2019), and the broader accessibility of NLP technologies.
Alongside more developed methods for embedding analysis, the static embedding setting is also equipped with a richer body of work regarding social bias. In this sense, we view understanding the encoded social bias in representations as a societally critical special-case of interpretability research. We employ methods for identifying and quantifying gender, racial/ethnic, and religious bias (Bolukbasi et al., 2016; Garg et al., 2018; Manzini et al., 2019) to our static embeddings. These experiments not only shed light on the properties of our static embeddings for downstream use but can also serve as a proxy for understanding latent biases in the original pretrained contextual representations. We find that biases in different models and across different layers are quite disparate; this has important consequences on model and layer selection for downstream use. Further, for two sets of pretrained weights learned on the same training data, we find that bias patterns still remain fairly distinct. Most surprisingly, our large-scale evaluation makes clear that existing bias estimators are dramatically inconsistent with each other.
In order to use a contextualized model like BERT to compute a single context-agnostic representation for a given word  $w$ , we define two operations.
The first is subword pooling: the application of a pooling mechanism over the  $k$  subword representations generated for  $w$  in context  $c$  in order to compute a single representation for  $w$  in  $c$ , i.e.  $\{\mathbf{w}_c^1, \ldots, \mathbf{w}_c^k\} \mapsto \mathbf{w}_c$ . Beyond this, we define context combination to be the mapping from representations  $\mathbf{w}_{c_1}, \ldots, \mathbf{w}_{c_n}$  of  $w$  in different contexts  $c_1, \ldots, c_n$  to a single static embedding  $\mathbf{w}$  that is agnostic of context.
Subword Pooling. The tokenization procedure for BERT can be decomposed into two steps: performing a simple word-level tokenization and then potentially deconstructing a word into multiple subwords, yielding  $w^{1}, \ldots, w^{k}$  such that  $\text{cat}(w^{1}, \ldots, w^{k}) = w$  where  $\text{cat}(\cdot)$  indicates concatenation. Then, every layer of the model computes vectors  $\mathbf{w}_{c}^{1}, \ldots, \mathbf{w}_{c}^{k}$ . Given these vectors, we consider four pooling mechanisms to compute  $\mathbf{w}_{c}$ :
$$
\mathbf {w} _ {c} = f \left(\mathbf {w} _ {c} ^ {1}, \dots , \mathbf {w} _ {c} ^ {k}\right)
$$
$$
f \in \{\min , \max , \text {m e a n}, \text {l a s t} \}
$$
$\min (\cdot), \max (\cdot)$  are element-wise min/max pooling, mean  $(\cdot)$  is the arithmetic mean and last  $(\cdot)$  indicates selecting the last vector,  $\mathbf{w}_c^k$
Context Combination. Next, we describe two approaches for specifying contexts  $c_{1}, \ldots, c_{n}$  and combining the associated representations  $\mathbf{W}_{c_{1}}, \ldots, \mathbf{W}_{c_{n}}$ .
- Decontextualized: For a word  $w$ , we use a single context  $c_{1} = w$ . That is, we feed the single word  $w$  into the pretrained model and use the outputted vector as the representation of  $w$  (applying subword pooling if the word is split into multiple subwords).
- Aggregated: Since the Decontextualized strategy presents an unnatural input to the pretrained encoder, which likely never encountered  $w$  in isolation, we instead aggregate representations of  $w$  across multiple contexts. In particular, we sample  $n$  sentences from a text corpus  $\mathcal{D}$  (see §A.2) each of which contains the word  $w$ , and compute the vectors  $\mathbf{w}_{c_1}, \ldots, \mathbf{w}_{c_n}$ . Then, we apply a pooling strategy to yield a single representation that aggregates representations across contexts:
$$
\mathbf {w} = g \left(\mathbf {w} _ {c _ {1}}, \dots , \mathbf {w} _ {c _ {n}}\right); g \in \{\min , \max , \text {m e a n} \}
$$
We begin by verifying that the resulting static embeddings that we derive retain their representational
2A humanist's outlook on the (in)accessibility of BERT: https://tedunderwood.com/2019/07/15/do-humanists-need-bert/
strength, to some extent. We take this step to ensure that properties we observe of the static embeddings can be attributed to, and are consistent with, the original contextualized representations. Inspired by concerns with probing methods/diagnostic classifiers (Liu et al., 2019a; Hewitt and Liang, 2019) regarding whether learning can be attributed to the classifier and not the underlying representation, we employ an exceptionally simple parameter-free method for converting from contextualized to static representations to ensure that any properties observed in the latter are not introduced via this process.
When evaluating static embedding performance, we consider Word2Vec (Mikolov et al., 2013) and GloVe (Pennington et al., 2014) embeddings as baselines since they have been the most prominent pretrained static embeddings for several years. Similarly, we begin with BERT as the contextualized model as it is currently the most prominent in downstream use among the growing number of alternatives. We provide identical analyses for 4 other contextualized model architectures (GPT-2 (Radford et al., 2019), XLNet (Yang et al., 2019), RoBERTa (Liu et al., 2019b), DistilBERT (Sanh et al., 2019)) and, in total, 9 sets of pretrained weights. All models, weights, and naming conventions used are enumerated in Appendix C and Table 9. Additional representation quality results appear in Tables 4-7 and Figures 4-10. We primarily report results for bert-base-uncased; further results for bert-large-uncased appear in Figure 3.
4 Representation Quality
4.1 Evaluation Details
To assess the representational quality of our static embeddings, we evaluate on several word similarity and word relatedness datasets.3 We consider 4 such datasets: RG65 (Rubenstein and Goodenough, 1965), WS353 (Agirre et al., 2009), SIMLEX999 (Hill et al., 2015) and SIMVERB3500 (Gerz et al., 2016) (see §A.4 for more details). Taken together, these datasets contain 4917 examples and specify a vocabulary  $\mathcal{V}$  of 2005 unique words. Each example is a pair of words  $(w_{1}, w_{2})$  with a gold-standard annotation (provided by one or more humans) of the semantic similarity or relatedness between  $w_{1}$  and  $w_{2}$ . A word embedding is evaluated by the relative correctness of its ranking
Model	N	RG65	WS353	SIMLEX999	SIMVERB3500
Word2Vec	-	0.6787	0.6838	0.4420	0.3636
GloVe	-	0.6873	0.6073	0.3705	0.2271
BERT-12 (1)	500K	0.7206	0.7038	0.5019	0.3550
BERT-24 (1)	500K	0.7367	0.7074	0.5114	0.3687
BERT-24 (6)	500K	0.7494	0.7282	0.5116	0.4062
BERT-12	10K	0.5167 (1)	0.6833 (1)	0.4573 (1)	0.3043 (1)
BERT-12	100K	0.6980 (1)	0.7023 (1)	0.5007 (3)	0.3494 (3)
BERT-12	500K	0.7262 (2)	0.7038 (1)	0.5115 (3)	0.3853 (4)
BERT-12	1M	0.7242 (1)	0.7048 (1)	0.5134 (3)	0.3948 (4)
BERT-24	100K	0.7749 (2)	0.7179 (6)	0.5044 (1)	0.3686 (9)
BERT-24	500K	0.7643 (2)	0.7282 (6)	0.5116 (6)	0.4146 (10)
BERT-24	1M	0.7768 (2)	0.7301 (6)	0.5244 (15)	0.4280 (10)
Table 1: Performance of distilled BERT embeddings.  $f$  and  $g$  are set to mean and (#) indicates the layer the embeddings are distilled from. Bold indicates best performance for a given dataset of embeddings depicted.
Model	RG65	WS353	SIMLEX999	SIMVERB3500
BERT-12	0.6980 (1)	0.7023 (1)	0.5007 (3)	0.3494 (3)
BERT-24	0.7749 (2)	0.7179 (6)	0.5044 (1)	0.3686 (9)
GPT2-12	0.5156 (1)	0.6396 (0)	0.4547 (2)	0.3128 (6)
GPT2-24	0.5328 (1)	0.6830 (0)	0.4505 (3)	0.3056 (0)
RoBERTa-12	0.6597 (0)	0.6915 (0)	0.5098 (0)	0.4206 (0)
RoBERTa-24	0.7087 (7)	0.6563 (6)	0.4959 (0)	0.3802 (0)
XLNet-12	0.6239 (1)	0.6629 (0)	0.5185 (1)	0.4044 (3)
XLNet-24	0.6522 (3)	0.7021 (3)	0.5503 (6)	0.4545 (3)
DistillBERT-6	0.7245 (1)	0.7164 (1)	0.5077 (0)	0.3207 (1)
Table 2: Performance of static embeddings from different pretrained models.  $f$  and  $g$  are set to mean,  $N = {100}\mathrm{\;K}$  ,and (#) indicates the layer the embeddings are distilled from. Bold indicates best performance for a given dataset of embeddings depicted.
of the similarity/relatedness of all examples in a dataset with respect to the gold-standard ranking using the Spearman  $\rho$  coefficient. Embedding predictions are computed using cosine similarity.
Pooling Strategy. In Figure 1, we show the performance on all 4 datasets for the resulting static embeddings. For embeddings computed using the Aggregated strategy, representations are aggregated over  $N = 100\mathrm{K}$  sentences where  $N$  is the number of total contexts for all words (§A.5). Across all four datasets, we see that  $g =$  mean is the best-performing pooling mechanism within the Aggregated strategy and also outperforms the Decontextualized strategy by a substantial margin. Fixing  $g =$  mean, we further observe that mean pooling at the subword level also performs best (the dark green dashed line in all plots). We further find that this trend consistently holds across pretrained models.
Number of Contexts. In Table 1, we see that performance for both BERT-12 and BERT-24 steadily increases across all datasets with increas
3Concerns with this decision are discussed in  $\S A.3$
Figure 1: Layer-wise performance of distilled BERT-12 embeddings for all pairs  $(f, g)$  with  $N = 100\mathbf{K}$ .
ing  $N$ ; this trend holds for the other 7 pretrained models. In particular, in the largest setting with  $N = 1\mathrm{M}$ , the BERT-24 embeddings distilled from the best-performing layer for each dataset drastically outperform both Word2Vec and GloVe. However, this can be seen as an unfair comparison given that we are selecting specific layers for specific datasets. As the middle band of Table 1 shows, we can fix a particular layer for all datasets and still outperform both Word2Vec and GloVe on all datasets.
Relationship between N and model layer. In
Figure 1, there is a clear preference towards the first quarter of the model's layers (layers 0-3) with a sharp drop-off in performance immediately thereafter. A similar preference for the first quarter of the model is observed in models with a different number of layers (Figure 3, Figure 10). Given that our intrinsic evaluation is centered on lexical semantic understanding, this appears to be largely consistent with the findings of Liu et al. (2019a); Tenney et al. (2019a) regarding where lexical semantic information is best encoded in pretrained contextualized models. However, as we pool over a larger number of contexts, Table 1 reveals an interesting relationship between  $N$  and the best-performing layer. The best-performing layer monotonically (with a single exception) shifts to be later and later within the pretrained model. Since the later layers did not perform better for smaller values of  $N$ , these layers demonstrate greater variance with respect to the layer-wise distributional mean and reducing this variance improves performance. Since later layers of the
model are generally preferred by downstream practitioners (Zhang et al., 2020), our findings suggest that downstream performance could be further improved by considering variance reduction as we suggest; Ethayarajh (2019) also provides concrete evidence of the tremendous variance in the later layers of deep pretrained contextualized models.
Cross-Model Results. Remarkably, we find that most tendencies we observe generalize well to all other pretrained models we study (specifically the optimality of  $f =$  mean,  $g =$  mean, the improved performance for larger  $N$ , and the layer-wise tendencies with respect to  $N$ ). This is particularly noteworthy given that several works have found that different contextualized models pattern substantially differently (Liu et al., 2019a; Ethayarajh, 2019).
In Table 2, we summarize the performance of all models we studied. All of the models considered were introduced during a similar time period and have comparable properties in terms of downstream performance. In spite of this, we observe that their static analogues perform radically differently. For example, several do not reliably outperform Word2Vec and GloVe despite outperforming Word2vec and GloVe reliably in downstream evaluation. Future work may consider whether the reduction to static embeddings affects different models differently and whether this is reflective of the quality of context-agnostic lexical semantics from other types of linguistic knowledge (e.g. context modelling, syntactic understanding, and semantic composition). In general, these results
<sup>4</sup>Shi et al. (2019) concurrently propose a different ap-
proach with similar motivations.
provide further evidence to suggest that linguistic understanding captured by different pretrained weights may be substantially different, even for models with near-identical Transformer (Vaswani et al., 2017) architectures.
Somewhat surprisingly, in Table 2, DistilBert-6 outperforms BERT-12 on three out of the four datasets despite being distilled (Ba and Caruana, 2014; Hinton et al., 2015) from BERT-12. Analogously, RoBERTa, which was introduced as a direct improvement over BERT, does not reliably outperform the corresponding BERT models.
5 Bias
Bias is a complex and highly relevant topic in developing representations and models in NLP and ML. In this context, we study the social bias encoded within our static word representations as a proxy for understanding biases of the source contextualized representations. As Kate Crawford argued for in her NIPS 2017 keynote, while studying individual models is important given that specific models may propagate, accentuate, or diminish biases in different ways, studying the representations that serve as the starting point and that are shared across models (which are used for possibly different tasks) allows for more generalizable understanding of bias (Barocas et al., 2017).
In this work, we simultaneously consider multiple axes of social bias (i.e. gender, race, and religion) and multiple proposed methods for computationally quantifying these biases. We do so precisely because we find that existing NLP literature has primarily prioritized gender (which may be a technically easier setting and is starkly incomplete in terms of social biases of interest). Further, as we will show, different computational specifications of bias that evaluate the same underlying social phenomena yield markedly different results. As a direct consequence, we strongly caution that the results must be taken with respect to the definitions of bias being applied. Further, we note that an embedding which receives low bias scores cannot be assumed to be (nearly) unbiased. Instead, it satisfies the significantly weaker condition that under existing definitions the embedding exhibits low bias and perhaps additional (more nuanced) definitions are needed.
5.1 Definitions
Bolukbasi et al. (2016) introduced a measure of
gender bias which assumes access to a set  $\mathcal{P} = \{(m_1, f_1), \ldots, (m_n, f_n)\}$  of (male, female) word pairs where  $m_i$  and  $f_i$  only differ in gender (e.g. 'men' and 'women'). They compute a gender direction  $\mathbf{g}$ :
$$
\mathbf {g} = \operatorname {P C A} \big ([ \mathbf {m} _ {1} - \mathbf {f} _ {1}, \dots , \mathbf {m} _ {n} - \mathbf {f} _ {n} ] \big) [ 0 ]
$$
where [0] indicates the first principal component.
Then, given a set  $\mathcal{N}$  of target words that we are interested in evaluating the bias with respect to, Bolukbasi et al. (2016) specifies the bias as:
$$
\operatorname *{bias}_{\substack{\text{BOLUKBASI}}}\left(\mathcal{N}\right) = \operatorname *{mean}_{w\in \mathcal{N}}\left|\cos \left(\mathbf{w},\mathbf{g}\right)\right|
$$
This definition is only inherently applicable to binary bias settings, i.e. where there are exactly two protected classes. Multi-class generalizations are difficult to realize since constructing  $\mathcal{P}$  requires aligned  $k$ -tuples whose entries only differ in the underlying social attribute and this becomes increasingly challenging for increasing  $k$ . Further, this definition assumes the first principal component explains a large fraction of the observed variance.
Garg et al. (2018) introduced a different definition that is not restricted to gender and assumes access to sets  $\mathcal{A}_1 = \{m_1, \dots, m_n\}$  and  $\mathcal{A}_2 = \{f_1, \dots, f_{n'}\}$  of representative words for each of the two protected classes. For each class,  $\mu_i = \max_{w \in \mathcal{A}_i} w$  is computed. Garg et al. (2018) computes the bias in two ways:
$$
\underset {\text {G A R G - E U C}} {\operatorname {b i a s}} (\mathcal {N}) = \underset {w \in \mathcal {N}} {\operatorname {m e a n}} \| \mathbf {w} - \mu_ {1} \| _ {2} - \| \mathbf {w} - \mu_ {2} \| _ {2}
$$
$$
\operatorname *{bias}_{\substack{\text{GARG - COS}}}\left(\mathcal{N}\right) = \operatorname *{mean}_{w\in \mathcal{N}}\cos \left(\mathbf{w},\mu_{1}\right) - \cos \left(\mathbf{w},\mu_{2}\right)
$$
Compared to the definition of Bolukbasi et al. (2016), these definitions may be more general as constructing  $\mathcal{P}$  is strictly more difficult than constructing  $\mathcal{A}_1, \mathcal{A}_2$  (as  $\mathcal{P}$  can always be split into two such sets but the reverse is not generally true) and Garg et al. (2018)'s definition does not rely on the first principal component explaining a large fraction of the variance. However, unlike the first definition, Garg et al. (2018) computes the bias in favor of/against a specific class (meaning if  $\mathcal{N} = \{\text{'programmer', 'homemaker'}\}$  and 'programmer' was equally male-biased as 'homemaker' was female-biased, then under the definition of Garg et al. (2018), there would be no bias in aggregate). To permit comparison, we insert absolute values around each term in the mean over  $\mathcal{N}$ .
Manzini et al. (2019) introduced a definition for quantifying multi-class bias which assumes access
to sets of representative words  $\mathcal{A}_1, \ldots, \mathcal{A}_k^5$ :
$$
\underset {\text {M A N Z I N I}} {\text {b i a s}} (\mathcal {N}) = \underset {w \in \mathcal {N}} {\text {m e a n}} \underset {i \in \{1, \ldots , k \}} {\text {m e a n}} \underset {a \in \mathcal {A} _ {i}} {\text {m e a n}} \cos (\mathbf {w}, \mathbf {a})
$$
Inspired by the results of Nissim et al. (2020), in this work we transparently report social bias in existing static embeddings as well as the embeddings we produce. In particular, we exhaustively report the measured bias for all 3542 valid (pretrained model, layer, social attribute, bias definition, target word list) 5-tuples — all possible combinations of static embeddings and bias measures considered. The results for models beyond BERT appear in Figures 11-18.