SlowGuess/ACL_2020_OCR huggingface dataset:
John M. Wu*1 Yonatan Belinkov*12  
Hassan Sajjad³ Nadir Durrani³ Fahim Dalvi³ James Glass¹
<sup>1</sup>MIT Computer Science and Artificial Intelligence Laboratory, Cambridge, MA, USA
$^{2}$ Harvard John A. Paulson School of Engineering and Applied Sciences, Cambridge, MA, USA
$^{3}$ Qatar Computing Research Institute, HBKU Research Complex, Doha 5825, Qatar
{johnmwu, belinkov, glass}@csail.mit.edu
{hsajjad,ndurrani,faimaduddin}@qf.org.qa
This paper investigates contextual word representation models from the lens of similarity analysis. Given a collection of trained models, we measure the similarity of their internal representations and attention. Critically, these models come from vastly different architectures. We use existing and novel similarity measures that aim to gauge the level of localization of information in the deep models, and facilitate the investigation of which design factors affect model similarity, without requiring any external linguistic annotation. The analysis reveals that models within the same family are more similar to one another, as may be expected. Surprisingly, different architectures have rather similar representations, but different individual neurons. We also observed differences in information localization in lower and higher layers and found that higher layers are more affected by fine-tuning on downstream tasks.
Contextual word representations such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2019) have led to impressive improvements in a variety of tasks. With this progress in breaking the state of the art, interest in the community has expanded to analyzing such models in an effort to illuminate their inner workings. A number of studies have analyzed the internal representations in such models and attempted to assess what linguistic properties they capture. A prominent methodology for this is to train supervised classifiers based on the models' learned representations, and predict various linguistic properties. For instance, Liu et al. (2019a) train such classifiers on 16 linguistic tasks, including part-of-speech tagging, chunking, named
entity recognition, and others. Such an approach may reveal how well representations from different models, and model layers, capture different properties. This approach, known as analysis by probing classifiers, has been used in numerous other studies (Belinkov and Glass, 2019).
While the above approach yields compelling insights, its applicability is constrained by the availability of linguistic annotations. In addition, comparisons of different models are indirect, via the probing accuracy, making it difficult to comment on the similarities and differences of different models. In this paper, we develop complementary methods for analyzing contextual word representations based on their inter- and intra-similarity. While this similarity analysis does not tell us absolute facts about a model, it allows comparing representations without subscribing to one type of information. We consider several kinds of similarity measures based on different levels of localization/distributivity of information: from neuron-level pairwise comparisons of individual neurons to representation-level comparisons of full word representations. We also explore similarity measures based on models' attention weights, in the case of Transformer models (Vaswani et al., 2017). This approach enables us to ask questions such as: Do different models behave similarly on the same inputs? Which design choices determine whether models behave similarly or differently? Are certain model components more similar than others across architectures? Is the information in a given model more or less localized (encoded in individual components) compared to other models?
$^{2}$ Hinton (1984) defines a localist representation as one using one computing element for each represented entity. In a language model, this definition would depend on what linguistic concepts we deem important, and is thus somewhat arbitrary. We develop a measure that aims to capture this notion of localization without recourse to a specific set of linguistic properties.
1The code is available at https://github.com/johnmwu/contextual-corr-analysis.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4638-4655
We choose a collection of pre-trained models that aim to capture diverse aspects of modeling choices, including the building blocks (Recurrent Networks, Transformers), language modeling objective (unidirectional, bidirectional, masked, permutation-based), and model depth (from 3 to 24 layers). More specifically, we experiment with variants of ELMo, BERT, GPT (Radford et al., 2018), GPT2 (Radford et al., 2019), and XLNet (Yang et al., 2019). Notably, we use the same methods to investigate the effect that fine-tuning on downstream tasks has on the model similarities.
Our analysis yields the following insights:
- Different architectures may have similar representations, but different individual neurons. Models within the same family are more similar to one another in terms of both their neurons and full representations.
- Lower layers are more similar than higher layers across architectures.
- Higher layers have more localized representations than lower layers.
- Higher layers are more affected by fine-tuning than lower layers, in terms of their representations and attentions, and thus are less similar to the higher layers of pre-trained models.
- Fine-tuning affects the localization of information, causing high layers to be less localized.
Finally, we show how the similarity analysis can motivate a simple technique for efficient finetuning, where freezing the bottom layers of models still maintains comparable performance to finetuning the full network, while reducing the finetuning time.
The most common approach for analyzing neural network models in general, and contextual word representations in particular, is by probing classifiers (Ettinger et al., 2016; Belinkov et al., 2017; Adi et al., 2017; Conneau et al., 2018; Hupkes et al., 2018), where a classifier is trained on a corpus of linguistic annotations using representations from the model under investigation. For example, Liu et al. (2019a) used this methodology for investigating the representations of contextual word representations on 16 linguistic tasks. One limitation of this approach is that it requires specifying
linguistic tasks of interest and obtaining suitable annotations. This potentially limits the applicability of the approach.
An orthogonal analysis method relies on similarities between model representations. Bau et al. (2019) used this approach to analyze the role of individual neurons in neural machine translation. They found that individual neurons are important and interpretable. However, their work was limited to a certain kind of architecture (specifically, a recurrent one). In contrast, we compare models of various architectures and objective functions.
Other work used similarity measures to study learning dynamics in language models by comparing checkpoints of recurrent language models (Morcos et al., 2018), or a language model and a part-of-speech tagger (Saphra and Lopez, 2019). Our work adopts a similar approach, but explores a range of similarity measures over different contextual word representation models.
Questions of localization and distributivity of information have been under investigation for a long time in the connectionist cognitive science literature (Page, 2000; Bowers, 2002; Gayler and Levy, 2011). While neural language representations are thought to be densely distributed, several recent studies have pointed out the importance of individual neurons (Qian et al., 2016; Shi et al., 2016; Radford et al., 2017; Lakretz et al., 2019; Bau et al., 2019; Dalvi et al., 2019; Baan et al., 2019). Our study contributes to this line of work by designing measures of localization and distributivity of information in a collection of models. Such measures may facilitate incorporating neuron interactions in new training objectives (Li et al., 2020).
3 Similarity Measures
We present five groups of similarity measures, each capturing a different similarity notion. Consider a collection of  $M$  models  $\{f^{(m)}\}_{m = 1}^{M}$ , yielding word representations  $\mathbf{h}_l^{(m)}$  and potentially attention weights  $\alpha_{l}^{(m)}$  at each layer  $l$ . Let  $k$  index neurons  $\mathbf{h}_l^{(m)}[k]$  or attention heads  $\alpha_{l}^{(m)}[k]$ .  $\mathbf{h}_l^{(m)}[k]$ ,  $\alpha_{l}^{(m)}[k]$  are real (resp. matrix) valued, ranging over words (resp. sentences) in a corpus. Our similarity measures are of the form  $\mathrm{sim}(\mathbf{h}_l^{(m)},\mathbf{h}_{l'}^{(m')})$  or  $\mathrm{sim}(\pmb {\alpha}_l^{(m)},\pmb {\alpha}_{l'}^{(m')})$ , that is, they find similarities between layers. We present the full mathematical details in appendix A.
3.1 Neuron-level similarity
A neuron-level similarity measure captures similarity between pairs of individual neurons. We consider one such measure, neuronsim, following Bau et al. (2019). For every neuron  $k$  in layer  $l$ , neuronsim finds the maximum correlation between it and another neuron in another layer  $l'$ . Then, it averages over neurons in layer  $l$ . This measure aims to capture localization of information. It is high when two layers have pairs of neurons with similar behavior. This is far more likely when the models have local, rather than distributed representations, because for distributed representations to have similar pairs of neurons the information must be distributed similarly.
3.2 Mixed neuron-representation similarity
A mixed neuron-representation similarity measure captures a similarity between a neuron in one model with a layer in another. We consider one such measure, mixedsim: for every neuron  $k$  in layer  $l$ , regress to it from all neurons in layer  $l'$  and measure the quality of fit. Then, average over neurons in  $l$ . It is possible that some information is localized in one layer but distributed in another layer. mixedsim captures such a phenomenon.
3.3 Representation-level similarity
A representation-level measure finds correlations between a full model (or layer) simultaneously. We consider three such measures: two based on canonical correlation analysis (CCA), namely singular vector CCA (svsim; Raghu et al. 2017) and projection weighted CCA (pwsim; Marcos et al. 2018), in addition to linear centered kernel alignment (ckasim; Kornblith et al. 2019). These measures emphasize distributivity of information—if two layers behave similarly over all of their neurons, the similarity will be higher, even if no individual neuron has a similar matching pair or is represented well by all neurons in the other layer.
Other representation-level similarity measures may be useful, such as representation similarity analysis (RSA; Kriegeskorte et al. 2008), which
has been used to analyze neural network representations (Bouchacourt and Baroni, 2018; Chrupała and Alishahi, 2019; Chrupała, 2019), or other variants of CCA, such as deep CCA (Andrew et al., 2013). We leave the explorations of such measures to future work.
3.4 Attention-level similarity
Previous work analyzing network similarity has mostly focused on representation-based similarities (Morcos et al., 2018; Saphra and Lopez, 2019; Voita et al., 2019a). Here we consider similarity based on attention weights in Transformer models.
Analogous to a neuron-level similarity measure, an attention-level similarity measure finds the most "correlated" other attention head. We consider three methods to correlate heads, based on the norm of two attention matrices  $\alpha_{l}^{(m)}[k]$ ,  $\alpha_{l'}^{(m')} [k']$ , their Pearson correlation, and their Jensen-Shannon divergence. We then average over heads  $k$  in layer  $l$ , as before. These measures are similar to neuronsim in that they emphasize localization of information—if two layers have pairs of heads that are very similar in their behavior, the similarity will be higher.
3.5 Distributed attention-level similarity
We consider parallels of the representation-level similarity. To compare the entire attention heads in two layers, we concatenate all weights from all heads in one layer to get an attention representation. That is, we obtain attention representations  $\alpha_{l}^{(m)}[h]$ , a random variable ranging over pairs of words in the same sentence, such that  $\alpha_{l,(i,j)}^{(m)}[h]$  is a scalar value. It is a matrix where the first axis is indexed by word pairs, and the second by heads. We flatten these matrices and use svsim, pwsim, and ckasim as above for comparing these attention representations. These measures should be high when the entire set of heads in one layer is similar to the set of heads in another layer.
Models We choose a collection of pre-trained models that aim to capture diverse aspects of modeling choices, including the building blocks (RNNs, Transformers), language modeling objective (unidirectional, bidirectional, masked, permutation-based), and model depth (from 3 to 24 layers).
5Other recent work has used the Jensen-Shannon divergence to measure distances between attention heads (Clark et al., 2019; Jain and Wallace, 2019).
<sup>3</sup>In this and other measures that allowed it, we also experimented with averaging just the top  $k$  neurons (or canonical correlations, in Section 3.3 measures) in case most of the layer is noise. Heatmaps are in the online repository. We did not notice major differences.
4We also experimented with the RBF variant, which is computationally demanding. We found similar patterns in preliminary experiments, so we focus on the linear variant.
(a)neuronsim
(b) ckasim
Figure 1: Similarity heatmaps of layers in various models under neuron- and representation-level similarities.
ELMo variants We use the original ELMo (Peters et al., 2018a), a bidirectional RNN model with two hidden layers, as well as two variants - a deeper and larger 4-layer model and a Transformer-equivalent variant (Peters et al., 2018b).
GPT variants We use both the original OpenAI Transformer (GPT; Radford et al. 2018) and its successor GPT2 (Radford et al., 2019), in the small and medium model sizes. These are all unidirectional Transformer LMs.
BERT We use BERT-base/large (12/24 layers; Devlin et al. 2019): Transformer LMs trained with a masked LM objective function. $^6$
XLNet We use XLNet-base/large (12/24 layers; Yang et al. 2019). Both are Transformer LM with a permutation-based objective function.
Data For analyzing the models, we run them on the Penn Treebank development set (Marcus et al., 1993), following the setup taken by Liu et al. (2019a) in their probing classifier experiments. We collect representations and attention weights from each layer in each model for computing the similarity measures. We obtain representations for models used in Liu et al. (2019a) from their implementation and use the transformers library (Wolf et al., 2019) to extract other representations. We aggregate sub-word representations by taking the representation of the last sub-word, following Liu et al. (2019a), and sub-word attentions by summing up at
tention to sub-words and averaging attention from sub-words, following Clark et al. (2019), which guarantees that the attention from each word sums to one.
5 Similarity of Pre-trained Models
5.1 Neuron and representation levels
Figure 1 shows heatmaps of similarities between layers of different models, according to neuronsim and ckasim. Heatmaps for the other measures are provided in Appendix B. The heatmaps reveal the following insights.
Different architectures may have similar representations, but different individual neurons Comparing the heatmaps, the most striking distinction is that neuronsim induces a distinctly block-diagonal heatmap, reflecting high intra-model similarities and low inter-model similarities. As neuronsim is computed by finding pairs of very similar neurons, this means that within a model, different layers have similar individual neurons, but across models, neurons are very different. In contrast, ckasim- show fairly significant similarities across models (high values off the main diagonal), indicating that different models generate similar representations. The most similar cross-model similarities are found by mixedsim (Figure 8d in Appendix B), which suggests that individual neurons in one model may be well represented by a linear combination of neurons in another layer. The other representation-level similarities (ckasim, svsim, and pwsim), also show cross-model similarities, albeit to a lesser extent.
6BERT is also trained with a next sentence prediction objective, although this may be redundant (Liu et al., 2019b). 7As suggested by a reviewer, we verified that the results are consistent when using another dataset (Appendix B.1).
Models within the same family are more similar The heatmaps show greater similarity within a model than across models (bright diagonal). Different models sharing the same architecture and objective function, but different depths, also exhibit substantial representation-level similarities – for instance, compare BERT-base and BERT-large or ELMo-original and ELMo-4-layers, under ckasim (Figure 1b). The Transformer-ELMo presents an instructive case, as it shares ELMo's bidirectional objective function but with Transformers rather than RNNs. Its layers are mostly similar to themselves and the other ELMo models, but also to GPT, more so than to BERT or XLNet, which use masked and permutation language modeling objectives, respectively. Thus it seems that the objective has a considerable impact on representation similarity.[8]
The fact that models within the same family are more similar to each other supports the choice of Saphra and Lopez (2019) to use models of similar architecture when probing models via similarity measures across tasks. A possible confounder is that models within the same family are trained on the same data, but cross-family models are trained on different data. It is difficult to control for this given the computational demands of training such models and the current practice in the community of training models on ever increasing sizes of data, rather than a standard fixed dataset. However, Figure 2 shows similarity heatmaps of layers from pre-trained and randomly initialized models using ckasim, exhibiting high intra-model similarities, as before. Interestingly, models within the same family (either GPT2 or XLNet) are more similar than across families, even with random models, indicating that intrinsic aspects of models in a given family make them similar, regardless of the training data or process. As may be expected, in most cases, the similarity between random and pretrained models is small. One exception is the vertical bands in the lower triangle, which indicate that the bottom layers of trained models are similar to many layers of random models. This may be due to random models merely transferring information from bottom to top, without meaningful processing.
Figure 2: ckasim similarity heatmap of layers in base and random models.
Still, it may explain why random models sometimes generate useful features (Wieting and Kiela, 2019). Meanwhile, as pointed out by a reviewer, lower layers converge faster, leaving them closer to their initial random state (Raghu et al., 2017; Shwartz-Ziv and Tishby, 2017).
Lower layers are more similar across architectures The representation-level heatmaps (Figure 1) all exhibit horizontal stripes at lower layers, especially with ckasim, indicating that lower layers are more similar than higher layers when comparing across models. This pattern can be explained by lower layers being closer to the input, which is always the same words. A similar observation has been made for vision networks (Raghu et al., 2017). Voita et al. (2019a) found a similar pattern comparing Transformer models with different objective functions.
Adjacent layers are more similar All heatmaps in Figure 1 exhibit a very bright diagonal and bright lines slightly off the main diagonal, indicating that adjacent layers are more similar. This is even true when comparing layers of different models (notice the diagonal nature of BERT-base vs. BERT-large in Figure 1b), indicating that layers at the same relative depth are more similar than layers at different relative depths. A similar pattern was found in vision networks (Kornblith et al., 2019). Some patterns are unexpected. For instance, comparing
11Raghu et al. (2017) also used svsim to study recurrent language models, showing that lower layers converge faster. Although they have not looked at cross-model comparisons, faster convergence may be consistent with fewer changes during training, which can explain why lower layers are more similar across architectures.
Voita et al. (2019a) found that differences in the training objective result in more different representations (according to pwsim) than differences in random initialization.
<sup>9</sup>We thank a reviewer for pointing out this connection.
10Relatedly, Morcos et al. (2018) found similar CCA coefficients in representations from recurrent language models trained on different datasets.
XLNet with the BERT models, it appears that lower layers of XLNet are more similar to higher layers of BERT. We speculate that this is an artifact of the permutation-based objective in XLNet.
We found corroborating evidence for this observation in ongoing parallel work, where we compare BERT and XLNet at different layers through word(Liu et al., 2019a) and sentence-level tasks (Wang et al., 2019): while BERT requires mostly features from higher layers to achieve state-of-the-art results, in XLNet lower and middle layers suffice.
Higher layers are more localized than lower ones The different similarity measures capture different levels of localization vs. distributivity of information. neuronsim captures cases of localized information, where pairs of neurons in different layers behave similarly. svsim captures cases of distributed information, where the full layer representation is similar. To quantify these differences, we compute the average similarity according to each measure when comparing each layer to all other layers. In effect, we take the column-wise mean of each heatmap. We do this separately for svsim as the distributed measure and neuronsim as the localized measure, and we subtract the svsim means from the neuronsim means. This results in a measure of localization per layer. Figure 3 shows the results.
In all models, the localization score mostly increases with layers, indicating that information tends to become more localized at higher layers.[12] This pattern is quite consistent, but may be surprising given prior observations on lower layers capturing phenomena that operate at a local context (Tenney et al., 2019), which presumably require fewer neurons. However, this pattern is in line with observations made by Ethayarajh (2019), who reported that upper layers of pre-trained models produce more context-specific representations. There appears to be a correspondence between our localization score and Ethayarajh's context-specificity score, which is based on the cosine similarity of representations of the same word in different contexts. Thus, more localized representations are also more context-specific. A direct comparison between context-specificity and localization may be fruitful avenue for future work.
Some models seem less localized than others,
Figure 3: Localization score of various model layers.
especially the ELMo variants, although this may be confounded by their being shallower models. BERT and XLNet models first decrease in localization and then increase. Interestingly, XLNet's localization score decreases towards the end, suggesting that its top layer representations are less context-specific.
5.2 Attention level