SlowGuess/ACL_2020_OCR huggingface dataset:
Ravid Shwartz-Ziv and Naftali Tishby. 2017. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810.
Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Chris Manning. 2014. A gold standard dependency corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 2897-2904, Reykjavik, Iceland. European Language Resources Association (ELRA).
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP, pages 1631-1642.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Elena Voita, Rico Sennrich, and Ivan Titov. 2019a. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4395-4405, Hong Kong, China. Association for Computational Linguistics.
Elena Voita, David Talbot, Fedor Moiseev, Rico Senrich, and Ivan Titov. 2019b. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797-5808, Florence, Italy. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations.
John Wieting and Douwe Kiela. 2019. No training required: Exploring random encoders for sentence classification. In International Conference on Learning Representations.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. HuggingFace's Transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 5753-5763. Curran Associates, Inc.
A Mathematical Details of Similarity Measures
We assume a fixed corpus with  $W = \sum_{i} W_{i}$  total words, and  $W^{(2)} = \sum_{i} W_{i}^{2}$  total pairs. Here  $W_{i}$  is the number of words in sentence  $i$ .
A representational layer  $\mathbf{h}_l^{(m)}$  may be seen as a  $W\times N_{m}$  matrix, where  $N_{m}$  is the number of neurons (per layer) in model  $m$ . A single neuron  $\mathbf{h}_l^{(m)}[k]$  (really  $\mathbf{h}_l^{(m)}[:,k]$ ) is a  $W\times 1$  column vector.
An attention head  $\alpha_{l}^{(m)}[k]$  may be seen as a random variable ranging over sentences  $s_i$  and taking matrix values  $\alpha_{l}^{(m)}[k](s_{i})\in \mathbb{R}^{t_{i}\times t_{i}}$ ,  $t_i = \mathrm{len}(s_i)$ .
A.1 Neuron-level similarity
For a given neuron  $\mathbf{h}_l^{(m)}[k]$ , we define
$$
\begin{array}{l} \operatorname {n e u r o n s i m} \left(\mathbf {h} _ {l} ^ {(m)} [ k ], \mathbf {h} _ {l ^ {\prime}} ^ {(m ^ {\prime})}\right) = \\ \max  _ {k ^ {\prime}} \left| \rho \left(\mathbf {h} _ {l ^ {\prime}} ^ {(m ^ {\prime})} [ k ^ {\prime} ], \mathbf {h} _ {l} ^ {(m)} [ k ]\right) \right| \\ \end{array}
$$
as the maximum correlation between it and another neuron in some layer (Bau et al., 2019). Here  $\rho$  is the Pearson correlation. This naturally gives rise to an aggregate measure at the layer level:
$$
\operatorname {n e u r o n s i m} \left(\mathbf {h} _ {l} ^ {(m)}, \mathbf {h} _ {l ^ {\prime}} ^ {(m ^ {\prime})}\right) =
$$
$$
\frac {1}{N _ {m}} \sum_ {k} \mathrm {n e u r o n s i m} (\mathbf {h} _ {l} ^ {(m)} [ k ], \mathbf {h} _ {l ^ {\prime}} ^ {(m ^ {\prime})})
$$
A.2 Mixed neuron-representation similarity
$$
\begin{array}{l} \operatorname {m i x e d s i m} \left(\mathbf {h} _ {l} ^ {(m)} [ k ], \mathbf {h} _ {l ^ {\prime}} ^ {(m ^ {\prime})}\right) := \\ \operatorname {l s t s q} \left(\mathbf {h} _ {l ^ {\prime}} ^ {(m ^ {\prime})}, \mathbf {h} _ {l} ^ {(m)} [ k ]\right). r \\ \end{array}
$$
where  $\mathbf{r}$  is the r-value associated with the regression, the norm of the prediction divided by the norm of the regressand. As before, this is extended to the layer level:
$$
\operatorname {m i x e d s i m} \left(\mathbf {h} _ {l} ^ {(m)}, \mathbf {h} _ {l ^ {\prime}} ^ {(m ^ {\prime})}\right) =
$$
$$
\frac {1}{N _ {m}} \sum_ {k} \operatorname {m i x e d s i m} (\mathbf {h} _ {l} ^ {(m)} [ k ], \mathbf {h} _ {l ^ {\prime}} ^ {(m ^ {\prime})})
$$
A.3 Representation-level similarity
In the following, let  $\mathbf{Z}$  denote a column centering transformation. For a given matrix  $\mathbf{A}$ , the sum of each column in  $\mathbf{Z}\mathbf{A}$  is zero.
SVCCA Given two layers
$$
\mathbf {X}, \mathbf {Y} = \mathbf {Z h} _ {l _ {x}} ^ {(m _ {x})}, \mathbf {Z h} _ {l _ {y}} ^ {(m _ {y})}
$$
we compute the truncated principal components
$$
\mathbf {X} ^ {\prime}, \mathbf {Y} ^ {\prime} = \mathbf {U _ {x}} [:,: l _ {x} ], \mathbf {U _ {y}} [:,: l _ {y} ]
$$
where  $\mathbf{U}_{\mathbf{x}}$  are the left singular vectors of  $\mathbf{X}$ , and  $l_{x}$  is the index required to account for  $99\%$  of the variance.  $\mathbf{U}_{\mathbf{y}}$  and  $l_{y}$  are defined analogously. The SVCCA correlations,  $\rho_{SVCCA}$ , are defined as:
$$
\mathbf {u}, \rho_ {S V C C A}, \mathbf {v} = \operatorname {S V D} \left(\mathbf {X} ^ {\prime T} \mathbf {Y} ^ {\prime}\right)
$$
The SVCCA similarity,  $\operatorname{svsim}(\mathbf{h}_{l_x}^{(m_x)},\mathbf{h}_{l_y}^{(m_y)})$  is the mean of  $\rho_{SVCCA}$ .
PWCCA Identical to SVCCA, except the computation of similarity is a weighted mean. Using the same notation as above, we define canonical vectors,
$$
\begin{array}{l} \mathbf {H} _ {\mathbf {X}} := \mathbf {X} ^ {\prime} \mathbf {u} \\ \mathbf {H} _ {\mathbf {Y}} := \mathbf {Y} ^ {\prime} \mathbf {v} \\ \end{array}
$$
We define alignments
$$
\begin{array}{l} \mathbf {A} _ {\mathbf {X}} := \operatorname {a b s} \left(\mathbf {H} _ {\mathbf {X}} ^ {\mathrm {T}} \mathbf {X}\right) \\ \mathbf {A} _ {\mathbf {Y}} := \operatorname {a b s} \left(\mathbf {H} _ {\mathbf {Y}} ^ {\mathrm {T}} \mathbf {Y}\right) \\ \end{array}
$$
where abs is the element-wise absolute value. The weights are
$$
\alpha_ {x} := \operatorname {w e i g h t s} (\mathbf {A} _ {\mathbf {X}} \mathbf {1}), \quad \alpha_ {y} := \operatorname {w e i g h t s} (\mathbf {A} _ {\mathbf {Y}} \mathbf {1})
$$
where  $\mathbf{1}$  is the column vector of all ones, and weights normalize a vector to sum to 1. The PWCCA similarity is
$$
\begin{array}{l} \operatorname {p w s i m} (\mathbf {h} _ {l _ {x}} ^ {(m _ {x})}, \mathbf {h} _ {l _ {y}} ^ {(m _ {y})}) := \alpha_ {x} ^ {T} \rho_ {S V C C A} \\ \operatorname {p w s i m} \left(\mathbf {h} _ {l _ {y}} ^ {(m _ {y})}, \mathbf {h} _ {l _ {x}} ^ {(m _ {x})}\right) := \alpha_ {y} ^ {T} \rho_ {S V C C A} \\ \end{array}
$$
It is asymmetric.
CKA We use the same notation as above. Given two layers,
the CKA similarity is
$$
\operatorname {c k a s i m} \left(\mathbf {h} _ {l _ {x}} ^ {\left(m _ {x}\right)}, \mathbf {h} _ {l _ {y}} ^ {\left(m _ {y}\right)}\right) := \frac {\left\| \mathbf {X} ^ {T} \mathbf {Y} \right\| ^ {2}}{\left\| \mathbf {X} ^ {T} \mathbf {X} \right\| \left\| \mathbf {Y} ^ {T} \mathbf {Y} \right\|}
$$
where  $\| \cdot \|$  is the Frobenius norm. It is symmetric.
A.4 Attention-level similarity
$$
\begin{array}{l} \operatorname {a t t n s i m} \left(\boldsymbol {\alpha} _ {l} ^ {(m)} [ k ], \boldsymbol {\alpha} _ {l ^ {\prime}} ^ {(m ^ {\prime})}\right) = \\ \max _ {k ^ {\prime}} \left[ \operatorname {S i m} \left(\boldsymbol {\alpha} _ {l ^ {\prime}} ^ {(m ^ {\prime})} [ k ^ {\prime} ], \boldsymbol {\alpha} _ {l} ^ {(m)} [ k ]\right) \right] \\ \end{array}
$$
We consider three such values of Sim.
- Matrix norm: for each sentence  $s_i$ , compute the Frobenius norm  $\left\| \alpha_{l'}^{(m')} [h'] (s_i) - \alpha_l^{(m)} [h] (s_i) \right\|$ . Then average over sentences in the corpus.
- Pearson correlation: for every word  $x_{i}$ , compare the attention distributions the two heads
induce from  $x_{i}$  to all words under Pearson correlation:  $\rho\left(\alpha_{l',i}^{(m')} [h'], \alpha_{l,i}^{(m)} [h]\right)$ . Then average over words in the corpus.
- Jensen-Shannon divergence: for every word  $x_{i}$ , compare the attention distributions under Jensen-Shannon divergence:  $\frac{1}{2}\mathrm{KL}(\alpha_{l',i}^{(m')}[h']\parallel \beta) + \frac{1}{2}\mathrm{KL}(\alpha_{l,i}^{(m)}[h]\parallel \beta)$ , where KL is the KL-divergence and  $\beta$  is the average of the two attention distributions. Then average of words in the corpus.
As before, this gives rise to aggregate measures at the layer level by averaging over heads  $h$ .
B Additional Representation-level Similarity Heatmaps
Figure 8 shows additional representation-level similarity heatmaps.
B.1 Effect of Data Used for Similarity Measures
The majority of the experiments reported in the paper are using the Penn Treebank for calculating the similarity measures. Here we show that the results are consistent when using a different dataset, namely the Universal Dependencies English Web Treebank (Silveira et al., 2014). We repeat the experiment reported in Section 5.1. The resulting heatmaps, shown in Figure 9, are highly similar to those generated using the Penn Treebank, shown in Figure 8.
C Additional Attention-level Similarity Heatmaps
Figure 10 shows additional attention-level similarity heatmaps.
D Efficient Fine-tuning
The analysis results showed that lower layers of the models go through limited changes during fin-tuning compared to higher layers. We use this insight to improve the efficiency of the fine-tuning process. In standard fine-tuning, back-propagation is done on the full network. We hypothesize that we can reduce the number of these operations by freezing the lower layers of the model since they are the least affected during the fine-tuning process. We experiment with freezing top and bottom layers of the network during the fine-tuning process. Different from prior work (Raghu et al., 2017; Felbo
Froze	SST-2	MNLI	QNLI	QQP
0	92.43	84.05	91.40	91.00
BERT	Top 4	91.86	82.86	91.09	90.97
Bot. 4	92.43	84.16	91.85	90.86
Top 6	91.97	82.53	90.13	90.61
Bot. 6	93.00	84.00	91.80	90.71
0	93.92	85.97	90.35	90.55
XLNet	Top 4	92.89	85.55	87.96	90.92
Bot. 4	93.12	86.04	90.65	89.36
Top 6	93.12	84.84	87.88	90.75
Bot. 6	93.92	85.64	90.99	89.02
Table 1: Freezing top/bottom 4/6 layers of BERT and XLNet during fine-tuning.
et al., 2017; Howard and Ruder, 2018), we freeze the selected layers for the complete fine-tuning process in contrast to freezing various layers for a fraction of the training time. We use the default parameters settings provided in the Transformer library (Wolf et al., 2019): batch size  $= 8$ , learning rate  $= 5e^{-5}$ , Adam optimizer with epsilon  $= 1e^{-8}$  and number of epochs  $= 3$ .
Table 1 presents the results on BERT and XLNet. On all of the tasks except QQP, freezing the bottom layers resulted in better performance than freezing the top layers. One interesting observation is that as we increase the number of bottom layers for freezing to six, the performance marginally degrades while saving a lot more computation. Surprisingly, on SST-2 and QNLI, freezing the bottom six layers resulted in better or equal performance than not freezing any layers of both models. With freezing the bottom six layers, one can save backpropagation computation by more than  $50\%$ .
(a) ckasim
(c) pwsim
(b)svsim
(d) mixedsim
Figure 8: Similarity heatmaps of layers in various models under different representation-level similarity measures.
(b) ckasim
(d) pwsim
Figure 9: Similarity heatmaps of layers in various models under neuron-level and representation-level similarity measures, using the English Web Treebank corpus.
(c)svsim
(e) mixedsim
(a) Matrix norm
(b) Jensen-Shannon
(c) Pearson
(d)svsim
(e) pwsim
(f) ckasim
Figure 10: Similarity heatmaps of layers in various models under different attention-level similarity measures.