SlowGuess/NAACL_2021_OCR huggingface dataset:
Figure 3: Here we juxtapose the minimum distances between clusters and the best classifier accuracy for all 12 layers. The horizontal axis is the layer index of  $\mathrm{BERT}_{\mathrm{base,cased}}$ ; the left vertical axis is the best classifier accuracy and the right vertical axis is the minimum distance between all pairs of clusters.
separates them and compute its margin. The distance we seek is twice the margin. For a given representation, we are interested in the minimum distance across all pairs of clusters with different labels.
5.2.1 Minimum Distance of Different Layers
Higher layers usually have larger  $\epsilon$ -version spaces. Different layers of BERT play different roles when encoding linguistic information (Tenney et al., 2019). To investigate the geometry of different layers of BERT, we apply DIRECTPROBE to each layer of  $\mathrm{BERT}_{\mathrm{base,cased}}$  for all five tasks. Then, we computed the minimum distances among all pairs of clusters with different labels. By comparing the minimum distances of different layers, we answer the question: how do different layers of BERT differ in their representations for a task?
Figure 3 shows the results on all tasks. In each subplot, the horizontal axis is the layer index. For each layer, the blue circles (left vertical axis) is the best classifier accuracy, and the red triangles (right vertical axis) is the minimum distance described above. We observe that both best classifier accuracy and minimum distance show similar trends across different layers: first increasing, then decreasing. It shows that minimum distance correlates with the best performance for an embedding space, though it is not a simple linear relation. Another interesting observation is the decreasing performance and minimum distance of higher layers, which is also corroborated by Ethayarajh (2019) and Liu et al. (2019a).
Task	Min Distance	Best Acc
SS-role	original	0.778	77.51
fine-tuned	4.231	81.62
SS-function	original	0.333	86.13
fine-tuned	2.686	88.4
POS	original	0.301	93.59
fine-tuned	0.7696	95.95
SR	original	0.421	86.85
fine-tuned	4.734	90.03
DEP	original	0.345	91.52
fine-tuned	1.075	94.82
5.2.2 Impact of Fine-tuning
Fine-tuning expands the  $\epsilon$ -version space. Past work (Peters et al., 2019; Arase and Tsujii, 2019; Merchant et al., 2020) has shown that fine-tuning pre-trained models on a specific task improves performance, and fine-tuning is now the de facto procedure for using contextualized embeddings. In this experiment, we try to understand why fine-tuning can improve performance. Without training classifiers, we answer the question: What changes in the embedding space after fine-tuning?
We conduct the experiments described in §5.2.1 on the last layer of  $\mathrm{BERT}_{\mathrm{base,cased}}$  before and after fine-tuning for all tasks. Table 2 shows the results. We see that after fine-tuning, both the best classifier accuracy and minimum distance show a big boost. It means that fine-tuning pushes the clusters away from each other in the representation space, which results in a larger  $\epsilon$ -version space. As we discussed in §5.2, a larger  $\epsilon$ -version space admits more good classifiers and allows for better generalization.
5.2.3 Label Confusion
Small distances between clusters can confuse a classifier. By comparing the distances between clusters, we can answer the question: Which labels for a task are more confusable?
We compute the distances between all the pairs of labels based on the last layer of  $\mathrm{BERT}_{\mathrm{base,cased}}$ . Based on an even split of the distances, we partition all label pairs into three bins: small, medium, and large. For each task, we use the predictions of the best classifier to compute the number of mis
Table 2: The best performance and the minimum distances between all pairs of clusters of the last layer of  $\mathrm{BERT}_{\mathrm{base,cased}}$  before and after fine-tuning.
Task	Small Distance	Medium Distance	Large Distance
SS-role	97.17% (555)	2.83% (392)	0% (88)
SS-func	96.88% (324)	3.12% (401)	0% (55)
POS	99.19% (102)	0.81% (18)	0% (16)
SR	93.20% (20)	6.80% (20)	0% (5)
DEP	99.97% (928)	0.03% (103)	0% (50)
Table 3: Error distribution based on different distance bins. The number of label pairs in each bin is shown in the parentheses.
classified label pairs for each bin. For example, if the clusters associated with the part of speech tags ADV and ADJ are close to each other, and the best classifier misclassified ADV as ADJ, we put this error pair into the bin of small distance. The distribution of all errors is shown in Table 3. This table shows that a large majority of the misclassified labels are concentrated in the small distance bin. For example, in the supersense role task (SS-role),  $97.17\%$  of the errors happened in small distance bin. The number of label pairs of each bin is shown in the parentheses. Table 3 shows that small distances between clusters indeed confuse a classifier and we can detect it without training classifiers.
5.3 By-product: A Parameter-free Classifier
Figure 4: Comparison between the best classifier accuracy, intra-accuracy, and 1-kNN accuracy. The X-axis is different representation models. BB: BERT<sub>base,cased</sub>, BL: BERT<sub>large,cased</sub>, RB: RoBERT<sub>base</sub>, RL: RoBERT<sub>large</sub>, E: ELMo. The Pearson correlation coefficient between best classifier accuracy and intra-accuracy is shown in the parentheses alongside each task title. This figure is best viewed in color.
We can predict the expected performance of the best classifier. Any  $h \in V_{\epsilon}(\mathcal{H}, E, D)$  is a predictor for the task  $D$  on the representation  $E$ . As a by-product of the clusters from DIRECTPROBE,
For all tasks,  $\mathrm{BERT}_{\mathrm{base,cased}}$  space (last layer) is linearly separable. So, the number of label pairs equals the number of cluster pairs.
we can define a predictor. The prediction strategy is simple: for a test example, we assign it to its closest cluster.<sup>7</sup> Indeed, if the label of the cluster is the true label of the test point, then we know that there exists some classifier that allows this example to be correctly labeled. We can verify the label every test point and compute the aggregate accuracy to serve as an indicator of the generalization ability of the representation at hand. We call this accuracy the intra-accuracy. In other words, without training classifiers, we can answer the question: given a representation, what is the expected performance of the best classifier for a task?
Figure 4 compares the best classifier accuracy, and the intra-accuracy of the last layer of different embeddings. Because our assignment strategy is similar to nearest neighbor classification (1-kNN), which assigns the unlabelled test point to its closest labeled point, the figure also compares to the 1-kNN accuracy.
First, we observe that intra-accuracy always outperforms the simple 1-kNN classifier, showing that DIRECTPROBE can use more information from the representation space. Second, we see that the intra-accuracy is close to the best accuracy for some tasks (Supersense tasks and POS tagging). Moreover, all the pearson correlation coefficients between best accuracy and intra-accuracy (showed in the parentheses alongside each task title) suggest a high linear correlation between best classifier accuracy and intra-accuracy. That is, the intra-accuracy can be a good predictor of the best classifier accuracy for a representation. From this, we argue that intra-accuracy can be interpreted as a benchmark accuracy of a given representation without actually training classifiers.
5.4 Case Study: Identifying Difficult Examples
The distances between a test point and all the clusters from the training set can not only be used to predict the label but also can be used to identify difficult examples as per a given representation. Doing so could lead to re-annotation of the data, and perhaps lead to cleaner data, or to improved embeddings. Using the supersense role task, we show a randomly chosen example of a mismatch
between the annotated label and the  $\mathrm{BERT}_{\mathrm{base,cased}}$  neighborhood:
...our new mobile number is ...
The data labels the word our as GESTALT, while the embedding places it in the neighborhood of POSSESSOR. The annotation guidelines for these labels (Schneider et al., 2017) notes that GESTALT is a supercategory of POSSESSOR. The latter is specifically used to identify cases where the possessed item is alienable and has monetary value. From this definition, we see that though the annotated label is GESTALT, it could arguably also be a POSSESSOR if phone numbers are construed as alienable possessions that have monetary value. Importantly, it is unclear whether  $\mathrm{BERT}_{\mathrm{base,cased}}$  makes this distinction. Other examples we examined required similarly nuanced analysis. This example shows DIRECTPROBE can be used to identify examples in datasets that are potentially mislabeled, or at least, require further discussion.
In addition to the classifier based probes described in the rest of the paper, a complementary line of work focuses on probing the representations using a behavior-based methodology. Controlled test sets (Senel et al., 2018; Jastrzebski et al., 2017) are designed and errors are analyzed to reverse-engineer what information can be encoded by the model (e.g., Marvin and Linzen, 2018; Ravichander et al., 2021; Wu et al., 2020). Another line of work probes the space by "opening up" the representation space or the model (e.g., Michel et al., 2019; Voita et al., 2019). There are some efforts to inspect the space from a geometric perspective (e.g., Ethayarajh, 2019; Mimno and Thompson, 2017). Our work extends this line of work to connect the geometric structure of embedding space with classifier performance without actually training a classifier.
Recent work (Pimentel et al., 2020b; Voita and Titov, 2020; Zhu and Rudzicz, 2020) probe representations from an information theoretic perspective. These efforts still need a probability distribution  $p(y|x)$  from a trained classifier. In §5.3, we use clusters to predict labels. In the same vein, the conditional probability  $p(y|x)$  can be obtained by treating the negative distances between the test point  $x$  and all clusters as predicted scores and normalizing via softmax. Our formalization can
To find the distance between the convex hull of a cluster and a test point, we find a max-margin separating hyperplane by training a linear SVM that separates the point from the cluster. The distance is twice the distance between the hyperplane and the test point.
fit into the information theoretic analysis and yet avoid training a classifier.
Our analysis and experiments open new directions for further research:
Novel pre-training target: The analysis presented here informs us that larger distance between clusters can improve classifiers. This could guide loss function design when pre-training representations.
Quality of a representation: In this paper, we focus on the accuracy of a representation. We could seek to measure other properties (e.g., complexity) or proxies for them. These analytical approaches can be applied to the  $\epsilon$ -version space to further analyze the quality of the representation space.
Theory of representation: Learning theory, e.g. VC-theory (Vapnik, 2013), describes the learnability of classifiers; representation learning lacks of such theoretical analysis. The ideas explored in this work ( $\epsilon$ -version spaces, distances between clusters being critical) could serve as a foundation for an analogous theory of representations.
In this work, we ask the question: what makes a representation good for a task? We answer it by developing DIRECTPROBE, a heuristic approach builds upon hierarchical clustering to approximate the  $\epsilon$ -version space. Via experiments with several contextualized embeddings and linguistic tasks, we showed that DIRECTPROBE can help us understand the geometry of the embedding space and ascertain when a representation can successfully be employed for a task.
We thank the members of the Utah NLP group and Nathan Schneider for discussions and valuable insights, and reviewers for their helpful feedback. We also thank the support of NSF grants #1801446 (SATC) and #1822877 (Cyberlearning).
Roee Aharoni and Yoav Goldberg. 2020. Unsupervised domain clusters in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7747-7763, Online. Association for Computational Linguistics.
Guillaume Alain and Yoshua Bengio. 2017. Understanding intermediate layers using linear classifier probes. In International Conference on Learning Representations.
Yuki Arase and Jun'ichi Tsujii. 2019. Transfer finetuning: A BERT case study. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5393-5404, Hong Kong, China. Association for Computational Linguistics.
Y Bengio, A Courville, and P Vincent. 2013. Representation learning: a review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798.
Ann Bies, Justin Mott, Colin Warner, and Seth Kulick. 2012. English web treebank. Linguistic Data Consortium, Philadelphia, PA.
Chih-Chung Chang and Chih-Jen Lin. 2011. Libsvm: A library for support vector machines. ACM transactions on intelligent systems and technology (TIST), 2(3):1-27.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single  $\$ \& !\#$ * vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126-2136, Melbourne, Australia. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Steffen Eger, Andreas Rückle, and Iryna Gurevych. 2019. Pitfalls in the evaluation of sentence embeddings. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 55–60, Florence, Italy. Association for Computational Linguistics.
Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of BERT, ELMo, and GPT-2 embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 55–65, Hong Kong, China. Association for Computational Linguistics.
Emily Goodwin, Koustuv Sinha, and Timothy J. O'Donnell. 2020. Probing linguistic systematicity. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1958-1969, Online. Association for Computational Linguistics.
Vikram Gupta, Haoyue Shi, Kevin Gimpel, and Mrrmaya Sachan. 2020. Clustering contextualized representations of text for unsupervised syntax induction. arXiv preprint arXiv:2010.12784.
LLC Gurobi Optimization. 2020. Gurobi optimizer reference manual.
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Seaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. SemEval-2010 task 8: Multiway classification of semantic relations between pairs of nominals. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 33-38, Uppsala, Sweden. Association for Computational Linguistics.
John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733-2743, Hong Kong, China. Association for Computational Linguistics.
Stanisław Jastrzebski, Damian Lesniak, and Wojciech Marian Czarnecki. 2017. How to evaluate word embeddings? on importance of data efficiency and simple supervised tasks. arXiv preprint arXiv:1702.02170.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651-3657, Florence, Italy. Association for Computational Linguistics.
Nora Kassner and Hinrich Schütze. 2020. Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7811-7818, Online. Association for Computational Linguistics.
Najoung Kim, Roma Patel, Adam Poliak, Patrick Xia, Alex Wang, Tom McCoy, Ian Tenney, Alexis Ross, Tal Linzen, Benjamin Van Durme, Samuel R. Bowman, and Ellie Pavlick. 2019. Probing what different NLP tasks teach machines about function word comprehension. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM* 2019), pages 235-249, Minneapolis, Minnesota. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of BERT. In Proceedings of the 2019 Conference on
Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4365-4374, Hong Kong, China. Association for Computational Linguistics.
Katarzyna Krasnowska-Kieras and Alina Wróblewska. 2019. Empirical linguistic study of sentence embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5729-5739, Florence, Italy. Association for Computational Linguistics.
Artur Kulmizev, Vinit Ravishankar, Mostafa Abdou, and Joakim Nivre. 2020. Do neural language models show preferences for syntactic formalisms? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4077-4091, Online. Association for Computational Linguistics.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1073-1094, Minneapolis, Minnesota. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192-1202, Brussels, Belgium. Association for Computational Linguistics.
Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. 2020. What happens to BERT embeddings during fine-tuning? In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 33-44, Online. Association for Computational Linguistics.
Julian Michael, Jan A Botha, and Ian Tenney. 2020. Asking without telling: Exploring latent ontologies in contextual representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6792-6812.
Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 14014-14024.
David Mimno and Laure Thompson. 2017. The strange geometry of skip-gram with negative sampling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2873-2878, Copenhagen, Denmark. Association for Computational Linguistics.
Tom M Mitchell. 1982. Generalization as search. Artificial intelligence, 18(2):203-226.
Joakim Nivre, Marie-Catherine De Marneffé, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan McDonald, Slav Petrov, Sampa Pyysalo, Natalia Silveira, et al. 2016. Universal dependencies v1: A multilingual treebank collection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 1659-1666.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830.
Christian S Perone, Roberto Silveira, and Thomas S Paula. 2018. Evaluation of sentence embeddings in downstream and linguistic probing tasks. arXiv preprint arXiv:1806.06259.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Matthew E. Peters, Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? adapting pretrained representations to diverse tasks. In Proceedings of the 4th Workshop on Representation Learning for NLP (RepL4NLP-2019), pages 7-14, Florence, Italy. Association for Computational Linguistics.
Tiago Pimentel, Naomi Saphra, Adina Williams, and Ryan Cotterell. 2020a. Pareto probing: Trading off accuracy for complexity. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3138-3153, Online. Association for Computational Linguistics.
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay, Ran Zmigrod, Adina Williams, and Ryan Cotterell. 2020b. Information-theoretic probing for linguistic structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4609-4622, Online. Association for Computational Linguistics.