SlowGuess/NAACL_2021_OCR huggingface dataset:
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5231-5247, Online. Association for Computational Linguistics.
Abhilasha Ravichander, Yonatan Belinkov, and Edward H. Hovy. 2021. Probing the probing paradigm: Does probing accuracy entail task relevance? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19-23, 2021, pages 3363-3377. Association for Computational Linguistics.
Nils Reimers, Benjamin Schiller, Tilman Beck, Johannes Daxenberger, Christian Stab, and Iryna Gurevych. 2019. Classification and clustering of arguments with contextualized word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 567-578, Florence, Italy. Association for Computational Linguistics.
Nathan Schneider, Jena D Hwang, Archna Bhatia, Vivek Srikumar, Na-Rae Han, Tim O'Gorman, Sarah R Moeller, Omri Abend, Adi Shalev, Austin Blodgett, et al. 2017. Adposition and case supersenses v2. 5: Guidelines for english. arXiv preprint arXiv:1704.02134.
Nathan Schneider, Jena D. Hwang, Vivek Srikumar, Jakob Prange, Austin Blodgett, Sarah R. Moeller, Aviram Stern, Adi Bitan, and Omri Abend. 2018. Comprehensive supersense disambiguation of English prepositions and possessives. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 185-196, Melbourne, Australia. Association for Computational Linguistics.
Lütfi Kerem Şenel, Ihsan Utlu, Veysel Üçesoy, Aykut Koc, and Tolga Cukur. 2018. Semantic structure and interpretability of word embeddings. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(10):1769-1779.
Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. 2020. oLMpics-on what language model pre-training captures. Transactions of the Association for Computational Linguistics, 8:743-758.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
Vladimir Vapnik. 2013. The nature of statistical learning theory. Springer science & business media.
Elena Voita, David Talbot, Fedor Moiseev, Rico Senrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797-5808, Florence, Italy. Association for Computational Linguistics.
Elena Voita and Ivan Titov. 2020. Information-theoretic probing with minimum description length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 183-196, Online. Association for Computational Linguistics.
Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do NLP models know numbers? probing numeracy in embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5307-5315, Hong Kong, China. Association for Computational Linguistics.
William F Whitney, Min Jae Song, David Brandfonbrener, Jaan Altosaar, and Kyunghyun Cho. 2020. Evaluating representations by the complexity of learning low-loss predictors. arXiv preprint arXiv:2009.07368.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierrick Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. 2020. Perturbed masking: Parameter-free probing for analyzing and interpreting BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4166-4176, Online. Association for Computational Linguistics.
Mengzhou Xia, Antonios Anastasopoulos, Ruochen Xu, Yiming Yang, and Graham Neubig. 2020. Predicting performance for natural language processing tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8625-8646, Online. Association for Computational Linguistics.
Yadollah Yaghoobzadeh, Katharina Kann, T. J. Hazen, Eneko Agirre, and Hinrich Schütze. 2019. Probing for semantic classes: Diagnosing the meaning content of word embeddings. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5740-5753, Florence, Italy. Association for Computational Linguistics.
Zining Zhu and Frank Rudzicz. 2020. An information theoretic view on selecting linguistic probes. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9251-9262, Online. Association for Computational Linguistics.
A DIRECTPROBE in Practice
In practice, we apply several techniques to speed up the probing process. Firstly, we add two caching strategies:
Black List Suppose we know two clusters  $A$  and  $B$  can not be merged, then  $A'$  and  $B'$  cannot be merged either if  $A \subseteq A'$  and  $B \subseteq B'$ .
White List Suppose we know two clusters  $A$  and  $B$  can be merged, then  $A'$  and  $B'$  can be merged too if  $A' \subseteq A$  and  $B' \subseteq B$ .
These two observations allow us to cache previous decisions in checking whether two clusters overlap. Applying these caching strategies can help us avoid unnecessary checking for overlap, which is time-consuming.
Secondly, instead of merging from the start to the end and checking at every step, we directly keep merging to the end without any overlap checking. After we arrived at the end, we start checking backwards to see if there is an overlap. If final clusters have overlaps, we find the step of the first error, correct it and keep merging. Merging to the end can also help us avoid plenty of unnecessary checking because, in most representations, the first error usually happens only in the final few merges. Algorithm 2 shows the whole algorithm.
Algorithm 2 DIRECTPROBE in Practice
Input: A dataset  $D$  with labeled examples  $(x_{i},y_{i})$ .
Output: A set of clusters of points  $\mathcal{C}$ .
1: Initialize  $C_i$  as the cluster for each  $(x_i, y_i) \in D$
2:  $\mathcal{C} = \{C_0,C_1,\dots \}$
3: Keep merging the closest pairs who have the same label in  $\mathcal{C}$  without overlapping checking.
4: if There are overlappings in  $\mathcal{C}$  then
5: Find the step  $k$  that first overlap happens.
6: Rebuild  $\mathcal{C}$  from step  $k - 1$
7: Keeping merging the closest pairs who have the same label in  $\mathcal{C}$  with overlapping checking.
8: end if
9: return  $\mathcal{C}$
Table 4 shows the runtime comparison between DirectProbe and training classifiers.
B Classifier Training Details
We train 3 different kinds of classifiers in order to find the best one: Logistic Regression, a one
DirectProbe	Classifier
Clustering	Predict	CV	Train-Test
SS-role	1 min	7 min	1.5 hours	1 hour
SS-func	1.5 min	5.5 min	1 hour	1 hour
POS	14 min	43 min	3.5 hours	2 hours
SR	10 min	22 min	50 min	1 hour
DEP	3 hours	3 hours	20 hours	11 hours
Table 4: The comparison of running time between DIRECTPROBE and training classifiers using the setting described in Appendix B. The time is computed on the  $\mathrm{BERT}_{\mathrm{base,cased}}$  space. DIRECTPROBE and classifiers run on the same machine.
Representations	#Parameters	Dimensions
BERTbase,cased	110M	768
BERTlarge,cased	340M	1024
RoBERTabase	125M	768
RoBERTalarge	355M	1024
ELMo	93.6M	1024
Table 5: Statistics of the five representations in our experiments.
layer neural network with hidden layer size of (32, 64, 128, 256) and two-layers neural network with hidden layer sizes of (32, 64, 128, 256)  $\times$  (32, 64, 128, 256). All neural networks use ReLU as the activation function and optimized by Adam (Kingma and Ba, 2015). The cross-validation process chooses the weight of each regularizer of each classifier. The weight range is from  $10^{-7}$  to  $10^{0}$ . We set the maximum iterations to be 1000. After choosing the best hyperparameters, each specific classifier is trained 10 times with different initializations.
C Summary of Representations
Table 5 summarizes all the representations in our experiments.
D Summary of Tasks
In this work, we conduct experiments on 5 tasks, which is designed to cover different usages of representations. Table 6 summarizes these tasks.
E Classifier Results v.s. DIRECTPROBE Results
Table 8 summarizes the results of the best classifier for each representation and task. The meanings of each column are as follows:
- Embedding: The name of representation.
- Best classification: The accuracy of the best classifier among 21 different classifiers. Each classifier run 10 times with different starting points. The final accuracy is the average accuracy of these 10 runs.
- Intra-accuracy: The prediction accuracy by assigning test examples to their closest clusters.
- Type: The type of the best classifier. e.g. (256,) means one-layer neural network with hidden size 256 and (256,128) means two-layer neural network with hidden sizes 256 and 128 respectively.
- Parameters: The number of parameters of the best classifier.
- Linear SVM: Training accuracy of a linear SVM classifier.
- #Clusters: The number of final clusters after probing.
F Detailed Classification Results
Table 7 shows the detailed classification results on Supersense-role task using the same settings described in Appendix B. In this table, we record the difference between the minimum and maximum test accuracy of the 10 runs of each classifier. The maximum difference for each representation is highlighted by the underlines. The best performance of each representation is highlighted by the bold numbers. From this table, we observe: (i) the difference between different runs of the same classifier can be as large as  $4 - 7\%$ , which can not be ignored; (ii) different representation requires different model architecture to achieve its best performance.
G Cluster Number Revision
We discovered a bug in the implementation of DirectProBE which causes the merging to stop early while the remaining clusters are still mergeable. The main paper (Table 1 and Table 8) has been updated to report the correct results. Table 9 shows the original results.
This bug does not change the natural of the linearity of datasets and representations. All the findings from original experiments remain the same.
Task	#Training	#Test	Token-based	Span-based	Pair-wise	Semantic	Syntax
Supersense-role	4282	457	✓	✓
Supersense-function	4282	457	✓	✓
POS	16,860	4323	✓	✓
Dependency Relation	42,081	4846	✓	✓
Semantic Relation	8000	2717	✓	✓	✓
Table 6: Statistics of the five tasks with their different characteristics.
Model Family	Specific Model	BERTbase,cased	BERTlarge,cased	RoBERTa,base	RoBERTalarge	ELMo
Min	Max	Diff	Min	Max	Diff	Min	Max	Diff	Min	Max	Diff	Min	Max	Diff
Linear	LR	75.93	76.59	0.66	72.21	72.43	0.22	79.65	80.09	0.44	77.46	77.90	0.44	76.37	77.02	0.65
One-layer	(32,)	75.27	78.56	3.29	70.24	73.09	2.85	77.9	79.87	1.97	74.84	78.34	3.5	74.4	75.93	1.53
(64,)	76.37	78.56	2.19	71.33	72.87	1.54	78.56	80.74	2.18	76.59	77.68	1.09	73.74	75.93	2.19
(128,)	75.71	77.9	2.19	70.46	73.3	2.84	77.9	80.53	2.63	75.71	77.46	1.75	73.52	76.15	2.63
(256,)	75.27	78.12	2.85	70.68	73.96	3.28	78.34	80.96	2.62	74.84	77.9	3.06	73.96	76.81	2.85
Two-layers	(32,32)	71.99	76.81	4.82	67.4	70.68	3.28	75.71	79.43	3.72	70.68	74.62	3.94	70.24	73.96	3.72
(32,64)	73.3	75.49	2.19	68.49	73.3	4.81	76.15	78.99	2.84	72.21	76.37	4.16	71.33	76.81	5.48
(32,128)	73.74	75.27	1.53	67.61	72.65	5.04	76.15	80.09	3.94	72.65	76.81	4.16	70.02	74.4	4.38
(32,256)	73.52	76.15	2.63	67.18	70.68	3.5	74.18	77.46	3.28	71.77	74.84	3.07	72.43	74.62	2.19
(64,32)	73.52	76.59	3.07	67.61	72.87	5.26	75.93	78.56	2.63	71.33	75.93	4.6	71.99	75.49	3.5
(64,64)	72.87	76.59	3.72	68.05	71.77	3.72	76.81	79.21	2.4	73.74	75.49	1.75	73.52	75.49	1.97
(64,128)	73.09	76.81	3.72	69.37	72.43	3.06	76.81	79.21	2.4	72.65	75.49	2.84	72.87	75.27	2.4
(64,256)	73.96	76.59	2.63	68.49	71.55	3.06	76.59	78.77	2.18	73.52	76.37	2.85	73.3	76.59	3.29
(128,32)	72.65	77.24	4.59	70.02	72.43	2.41	76.59	80.53	3.94	73.74	75.93	2.19	72.43	75.93	3.5
(128,64)	73.74	77.02	3.28	68.93	73.3	4.37	76.59	79.87	3.28	72.65	75.93	3.28	72.21	76.37	4.16
(128,128)	74.84	77.24	2.4	69.37	72.43	3.06	77.46	79.21	1.75	72.87	75.71	2.84	73.52	76.15	2.63
(128,256)	73.74	77.24	3.5	64.33	72.21	7.88	77.02	80.74	3.72	72.87	76.15	3.28	73.09	76.59	3.5
(256,32)	73.74	78.12	4.38	68.05	73.3	5.25	77.68	80.31	2.63	73.74	75.93	2.19	72.87	75.71	2.84
(256,64)	75.49	78.77	3.28	68.93	72.43	3.5	77.68	80.09	2.41	73.74	75.71	1.97	72.87	76.15	3.28
(256,128)	75.27	78.77	3.5	70.46	73.96	3.5	77.9	80.74	2.84	73.74	77.02	3.28	73.96	76.37	2.41
(256,256)	75.71	78.34	2.63	66.74	73.52	6.78	78.12	81.18	3.06	74.84	76.37	1.53	73.3	75.71	2.41
Table 7: Classification results on Supersense role task. Each specific classifier is trained ten times with different initializations. We record the minimum and maximum performance of these ten runs. Diff is the difference between the minimum and maximum performance. Bold number highlights the best performance of each representation. Underlines highlights the maximum difference between ten runs. See Appendix F for a discussion.
Task	Embedding	Best classification	Intra-accuracy	Type	Parameters	Linear SVM	#cluster
SS-role	BERTbase,cased	77.48 ± 0.92	76.58	(256,128)	235,264	100	46
BERTlarge,cased	72.25 ± 0.09	71.55	linear	47,104	100	46
RoBERTa,based	79.85 ± 0.15	78.56	linear	35,328	100	46
RoBERTa large	77.7 ± 0.18	74.18	linear	47,104	100	46
ELMo	76.7 ± 0.24	74.18	linear	47,104	100	46
SS-function	BERTbase,cased	86.3 ± 0.44	86.21	(128,)	103,424	100	40
BERTlarge,cased	82.84 ± 0.73	80.53	(256,)	272,384	100	40
RoBERTa,based	89.87 ± 0.51	88.18	(256,)	206,848	100	40
RoBERTa large	87.72 ± 0.56	84.03	(256,128)	300,032	100	40
ELMo	86.87 ± 0.31	84.03	linear	40,960	100	40
POS	BERTbase,cased	94.11 ± 0.14	93.59	(256,256)	266,496	100	17
BERTlarge,cased	89.54 ± 0.34	87.69	(128,256)	168,192	100	17
RoBERTa,based	95 ± 0.09	94.26	(128,256)	135,424	100	17
RoBERTa large	94.25 ± 0.19	92.92	(128,)	133,248	99.97	23
ELMo	95.08 ± 0.1	94.93	(256,)	266,496	100	17
Semantic Relation	BERTbase,cased	86.43 ± 0.28	80.27	(256,)	395,776	100	10
BERTlarge,cased	84.71 ± 0.27	76.41	(128,256)	297,472	100	10
RoBERTa,based	85.55 ± 0.28	78.58	(256,32)	401,728	100	10
RoBERTa large	85.04 ± 0.23	76.26	(256,)	526,848	100	10
ELMo	83.47 ± 0.28	78.06	(128,128)	279,808	100	10
Dependency	BERTbase,cased	91.52 ± 0.25	87.49	(256,)	405,248	100	47
BERTlarge,cased	88.9 ± 0.38	83.06	(256,)	536,320	100	47
RoBERTa,based	92.24 ± 0.03	87.99	linear	72,192	100	47
RoBERTa large	91.18 ± 0.04	85.06	linear	96,256	100	47
ELMo	92.21 ± 0.25	88.05	(256,)	536,320	100	47
Table 8: Intra-accuracy results on 5 tasks, compared against the best classifier results. See Appendix E for details about each column.
This bug only affects the number of clusters when the representation is non-linear for a given task.
Task	Embedding	Best classification	Intra-accuracy	Type	Parameters	Linear SVM	#cluster
SS-role	BERTbase,cased	77.48 ± 0.92	76.58	(256,128)	235,264	100	46
BERTlarge,cased	72.25 ± 0.09	71.55	linear	47,104	100	46
RoBERTa,base	79.85 ± 0.15	78.56	linear	35,328	100	46
RoBERTa large	77.7 ± 0.18	74.18	linear	47,104	100	46
ELMo	76.7 ± 0.24	74.18	linear	47,104	100	46
SS FUNC	BERTbase,cased	86.3 ± 0.44	86.21	(128,)	103,424	100	40
BERTlarge,cased	82.84 ± 0.73	80.53	(256,)	272,384	100	40
RoBERTa,base	89.87 ± 0.51	88.18	(256,)	206,848	100	40
RoBERTa large	87.72 ± 0.56	84.03	(256,128)	300,032	100	40
ELMo	86.87 ± 0.31	84.03	linear	40,960	100	40
POS	BERTbase,cased	94.11 ± 0.14	93.59	(256,256)	266,496	100	17
BERTlarge,cased	89.54 ± 0.34	87.69	(128,256)	168,192	100	17
RoBERTa,base	95 ± 0.09	94.26	(128,256)	135,424	100	17
RoBERTa large	94.25 ± 0.19	92.92	(128,)	133,248	99.97	1487
ELMo	95.08 ± 0.1	94.93	(256,)	266,496	100	17
Semantic Relation	BERTbase,cased	86.43 ± 0.28	80.27	(256,)	395,776	100	10
BERTlarge,cased	84.71 ± 0.27	76.41	(128,256)	297,472	100	10
RoBERTa,base	85.55 ± 0.28	78.58	(256,32)	401,728	100	10
RoBERTa large	85.04 ± 0.23	76.26	(256,)	526,848	100	10
ELMo	83.47 ± 0.28	78.06	(128,128)	279,808	100	10
Dependency	BERTbase,cased	91.52 ± 0.25	87.49	(256,)	405,248	100	47
BERTlarge,cased	88.9 ± 0.38	83.06	(256,)	536,320	100	47
RoBERTa,base	92.24 ± 0.03	87.99	linear	72,192	100	47
RoBERTa large	91.18 ± 0.04	85.06	linear	96,256	100	47
ELMo	92.21 ± 0.25	88.05	(256,)	536,320	100	47
Table 9: Original intra-accuracy results on 5 tasks, compared against the best classifier results. These results were in the original version of the paper before we found a bug in the implementation of DIRECTPROBE. The updated results are in Table 8. See Appendix E for details about each column.