SlowGuess/NAACL_2022_OCR huggingface dataset:
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129-4138, Minneapolis, Minnesota. Association for Computational Linguistics.
J. D. Hunter. 2007. Matplotlib: A 2d graphics environment. Computing in Science & Engineering, 9(3):90-95.
Kiyoung Kim. 2020. Pretrained language models for korean. https://github.com/kiyoungkim1/LMkor.
John Koutsikakis, Ilias Chalkidis, Prodromos Malakasiotis, and Ion Androutsopoulos. 2020. Greek-bert: The greeks visiting sesame street. In 11th Hellenic Conference on Artificial Intelligence, SETN 2020, page 110-117, New York, NY, USA. Association for Computing Machinery.
Artur Kulmizev, Vinit Ravishankar, Mostafa Abdou, and Joakim Nivre. 2020. Do neural language models show preferences for syntactic formalisms? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4077-4091, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations.
Martin Malmsten, Love Börjeson, and Chris Haffenden. 2020. Playing with words at the national library of Sweden – making a swedish bert.
Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, Claudia Bedini, Núria Bertomeu Castelló, and Jungmee Lee. 2013a. Universal Dependency annotation for multilingual parsing. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 92-97, Sofia, Bulgaria. Association for Computational Linguistics.
Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, Claudia Bedini, Núria Bertomeu Castelló, and Jungmee Lee. 2013b. Universal Dependency annotation for multilingual parsing. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers),
pages 92-97, Sofia, Bulgaria. Association for Computational Linguistics.
Max Müller-Eberstein, Rob van der Goot, and Barbara Plank. 2022. Probing for labeled dependency trees. Computing Research Repository, arxiv:2203.12971. Version 1.
Cuong Nguyen, Tal Hassner, Matthias Seeger, and Cedric Archambeau. 2020. Leep: A new measure to evaluate transferability of learned representations. In International Conference on Machine Learning, pages 7294-7305. PMLR.
Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Junseong Kim, Yongsook Song, Taehwan Oh, Joohong Lee, Juhyun Oh, Sungwon Lyu, Younghoon Jeong, Inkwon Lee, Sangwoo Seo, Dongjun Lee, Hyunwoo Kim, Myeonghwa Lee, Seongbo Jang, Seungwon Do, Sunkyoung Kim, Kyungtae Lim, Jongwon Lee, Kyumin Park, Jamin Shin, Seonghyun Kim, Lucy Park, Alice Oh, Jungwoo Ha, and Kyunghyun Cho. 2021. Klue: Korean language understanding evaluation.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc.
Sampo Pyysalo, Jenna Kanerva, Anna Missilä, Veronika Laippala, and Filip Ginter. 2015. Universal Dependencies for Finnish. In Proceedings of the 20th Nordic Conference of Computational Linguistics (NODALIDA 2015), pages 163-172, Vilnius, Lithuania. Linköping University Electronic Press, Sweden.
Ali Safaya, Moutasem Abdullatif, and Deniz Yuret. 2020. KUISAIL at SemEval-2020 task 12: BERTCNN for offensive speech identification in social media. In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 2054–2059, Barcelona (online). International Committee for Computational Linguistics.
Sber Devices. 2021. ruRoBERTa-large. https://huggingface.co/sberbank-ai/ruRoberta-large. Accessed 4th January, 2022.
Amit Seker, Elron Bandel, Dan Bareket, Idan Brusilovsky, Refael Shaked Greenfeld, and Reut Tsarfaty. 2021. Alephbert: A hebrew large pre-trained language model to start-off your hebrew nlp application with.
Mo Shen, Ryan McDonald, Daniel Zeman, and Peng Qi. 2016. UD_Chinese-GSD. https://
github.com/UniversalDependencies/UD_Chinese-GSD.
Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Chris Manning. 2014. A gold standard dependency corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 2897-2904, Reykjavik, Iceland. European Language Resources Association (ELRA).
Pranaydeep Singh, Gorik Rutten, and Els Lefever. 2021. A pilot study for BERT language modelling and morphological analysis for ancient and medieval Greek. In Proceedings of the 5th Joint SIGHUM Workshop on Computational Linguistics for Cultural Heritage, Social Sciences, Humanities and Literature, pages 128-137, Punta Cana, Dominican Republic (online). Association for Computational Linguistics.
Aaron Smith, Miryam de Lhoneux, Sara Stymne, and Joakim Nivre. 2018. An investigation of the interactions between pre-trained word embeddings, character models and POS tags in dependency parsing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2711-2720, Brussels, Belgium. Association for Computational Linguistics.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3645-3650, Florence, Italy. Association for Computational Linguistics.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations.
Rob van der Goot, Ahmet Üstün, Alan Ramponi, Ibrahim Sharaf, and Barbara Plank. 2021. Massive choice, ample tasks (MaChAmp): A toolkit for multitask learning in NLP. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 176-197, Online. Association for Computational Linguistics.
Sebastiano Vigna. 2015. A weighted correlation index for rankings with ties. In Proceedings of the 24th international conference on World Wide Web, pages 1166-1176.
Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. Multilingual is not enough: BERT for finnish. CoRR, abs/1912.07076.
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stefan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C J Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cirmrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antonio H. Ribeiro, Fabian Pedregosa, Paul van Mulbregt, and SciPy 1.0 Contributors. 2020. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272.
Elena Voita and Ivan Titov. 2020. Information-theoretic probing with minimum description length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 183-196, Online. Association for Computational Linguistics.
Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. 2021. Logme: Practical assessment of pre-trained models for transfer learning. In International Conference on Machine Learning, pages 12133-12143. PMLR.
Daniel Zeman, Joakim Nivre, Mitchell Abrams, Elia Ackermann, Noëmi Aepli, Hamid Aghaei, Željko Agić, Amir Ahmadi, Lars Ahrenberg, Chika Kennedy Ajede, Gabrielé Aleksandraviciţu, Ika Alfina, Lene Antonsen, Katya Aplonova, Angelina Aquino, Carolina Aragon, Maria Jesus Aranzabe, Bilge Nas Arican, Hórunn Arnardóttir, Gashaw Arutie, Jessica Naraiswari Arwidarasti, Masayuki Asahara,
Deniz Baran Aslan, Luma Ateyah, Furkan Atmaca, Mohammed Attia, Aitziber Atutxa, Liesbeth Augustinus, Elena Badmaeva, Keerthana Balasubramani, Miguel Ballesteros, Esha Banerjee, Sebastian Bank, Verginica Barbu Mititelu, Starkadur Barkarson, Rodolfo Basile, Victoria Basmov, Colin Batchelor, John Bauer, Seyyit Talha Bedir, Kepa Bengoetxea, Gozde Berk, Yevgeni Berzak, Irshad Ahmad Bhat, Riyadh Ahmad Bhat, Erica Biagetti, Eckhard Bick, Agne Bielinskiene, Kristin Bjarnadottir, Rogier Blokland, Victoria Bobicev, Loic Boizou, Emanuel Borges Völker, Carl Börstell, Cristina Bosco, Gosse Bouma, Sam Bowman, Adriane Boyd, Anouck Braggaar, Kristina Brokaite, Aljoscha Burchardt, Marie Candito, Bernard Caron, Gauthier Caron, Lauren Cassidy, Tatiana Cavalcanti, Gülsen Cebiroğlu Eryigit, Flavio Massimiliano Cecchini, Giuseppe G. A. Celano, Slavomir Čéplö, Neslihan Cesur, Savas Cetin, Özlem Çetinoglu, Fabricio Chalub, Shweta Chauhan, Ethan Chi, Taishi Chika, Yongseok Cho, Jinho Choi, Jayeol Chun, Juyeon Chung, Alessandra T. Cignarella, Silvie Cinkova, Aurélie Collomb, Cağri Çoltekin, Miriam Connor, Marine Courtin, Mihaela Cristescu, Philemon Daniel, Elizabeth Davidson, Marie-Catherine de Marneffe, Valeria de Paiva, Mehmet Oguz Derin, Elvis de Souza, Arantza Díaz de Ilarraza, Carly Dickerson, Arawinda Dinakaramani, Elisa Di Nuovo, Bamba Dione, Peter Dirix, Kaja Dobrovoljc, Timothy Dozat, Kira Droganova, Puneet Dwivedi, Hanne Eckhoff, Sandra Eiche, Marhaba Eli, Ali Elkahky, Binyam Ephrem, Olga Erina, Tomaz Erjavec, Aline Etienne, Wograine Evelyn, Sidney Facundes, Richard Farkas, Jannatul Ferdaousi, Marília Fernanda, Hector Fernandez Alcalde, Jennifer Foster, Cláudia Freitas, Kazunori Fujita, Katarina Gajdoshova, Daniel Galbraith, Marcos Garcia, Moe Gärdenfors, Sebastian Garza, Fabricio Ferraz Gerardi, Kim Gerdes, Filip Ginter, Gustavo Godoy, Iakes Goenaga, Koldo Gojenola, Memduh Gökirmak, Yoav Goldberg, Xavier Gömez Guinovart, Berta González Saavedra, Bernadeta Griciuté, Matias Grioni, Loïc Grobol, Normunds Gržitis, Bruno Guillaume, Céline Guillot-Barbance, Tunga Güngör, Nizar Habash, Hinrik Hafsteinsson, Jan Hajic, Jan Hajic jr., Mika Hämäläinen, Linh Hà Myhä-Na-Rae Han, Muhammad Yudistira Hanifmuti, Sam Hardwick, Kim Harris, Dag Haug, Johannes Heinecke, Oliver Hellwig, Felix Hennig, Barbora Hladka-Jaroslava Hlaváčová, Florinel Hociung,Petter Hohle,Eva Huber,Jena Hwang,Takumi Ikeda,Anton Karl Ingason,Radu Ion,Elena Irimia,Olajide Ishola Kaoru Ito,Siratun Jannat,Tomás Jelinek,Apoorva Jha Anders Johannsen,Hildur Jónsdóttir Fredrik Jorgensen,Markus Juutinen,Sarveswaran K,Hüner KasikaraAndre Kaasen,Nadezhda Kabaeva,Sylvain Kahane,Hiroshi KanayamaJenna Kanerva Neslihan KaraBoris KatzTolga Kayadelen Jessica Kenney,Václava Kettnerová,Jesse Kirchner Elena Klementieva,Elena KlyachkoArne Köhn Abdullatif Köksal,Kamil Kopacewicz,Timo Korkiakangas,Mehmet Köse,Natalia Kotsyba,Jolanta Kovalevskaite,Simon Krek Parameswari Krishna
murthy, Sandra Kübler, Oğuzhan Kuyrukcu, Asli Kuzgun, Sookyoung Kwak, Veronika Laippala, Lucia Lam, Lorenzo Lambertino, Tatiana Lando, Septina Dian Larasati, Alexei Lavrentiev, John Lee, Phuong Lê Hùng, Alessandro Lenci, Saran Lertpradit, Herman Leung, Maria Levina, Cheuk Ying Li, Josie Li, Keying Li, Yuan Li, KyungTae Lim, Bruna Lima Padovani, Krister Lindén, Nikola Ljubesic, Olga Loginova, Stefano Lusito, Andry Luthfi, Mikko Luukko, Olga Lyashevskaya, Teresa Lynn, Vivien Macketanz, Menel Mahamdi, Jean Maillard, Aibek Makazhanov, Michael Mandl, Christopher Manning, Ruli Manurung, Büşra Marşan, Catalina Márández, David Mareček, Katrin Marheinecke, Héctor Martínez Alonso, Lorena Martín-Rodriguez, André Martins, Jan Mašek, Hiroshi Matsuda, Yuji Matsumoto, Alessandro Mazzei, Ryan McDonald, Sarah McGuinness, Gustavo Mendonça, Tatiana Merzhevich, Niko Miekka, Karina Mischenkova, Margarita Misirpashayeva, Anna Missilä, Catalin Mittelu, Maria Mitrofan, Yusuke Miyao, Amir Hossein Mojiri Foroushani, Judit Molnár, Amirsaeid Moloodi, Simonetta Montemagni, Amir More, Laura Moreno Romero, Giovanni Moretti, Keiko Sophie Mori, Shinsuke Mori, Tomohiko Morioka, Shigeki Moro, Bjartur Mortensen, Bohdan Moskalevskyi, Kadri Muischnek, Robert Munro, Yugo Murawaki, Kaili Müürisep, Pinkey Nainwani, Mariam Nakhle, Juan Ignacio Navarro Horniacek, Anna Nedoluzhko, Gunta Nespore-Berzkalne, Manuela Nevaci, Luong Nguyen Thi, Huyen Nguyen Thi Minh, Yoshihiro Nikaido, Vitaly Nikolaev, Rattima Nitisaroj, Alireza Nourian, Hanna Nurmi, Stina Ojala, Atul Kr. Ojha, Adedayo Olóökun, Mai Omura, Emeka Onwuegbuzia, Petya Osenova, Robert Ostling, Lilja Øvrelid, Şaziye Betül Özatesç, Merve Özçelik, Arzucan Özgün Baskı Öztürk Başaran, Hyunj Hayley Park, Niko Partanen, Elena Pascual, Marco Passarotti, Agnieszka Patejuk, Guilherme PaulinPassos, Angelika Peljak-Lapinska, Siyao Peng. Cenel-Augusto Perez, Natalia Perkova, Guy Perrier, Slav Petrov, Daria Petrova, Jason Phelan Jussi Piitulainen, Tommi A Pirinen, Emily Pittler Barbara Plank Thierry Poibeau Larisa Ponomareva Martin Popel Lauma Pretkalnina,Sophie Prévost Prokopis Prokopidis Adam Przepiörkowski Tiina Puolakainen,Sampo Pyysalo,Peng Qi Andriela Raabis,Alexandre Rademaker,Mizanur Rahman Taraka RamaLoganathan Ramasamy Carlos Ramisch,Fam Rashel Mohammad Sadegh Rasooli,Vinit Ravishankar,Livy Real,Petru Rebeja Siva Reddy Mathilde Regnault Georg Rehm Ivan Riabov Michael Rießler,Erika Rimkute Larissa Rinaldi Laura Rituma Putri Rizqiyah Luisa Rocha Eirikur Rögnvaldsson Mykhailo Romanenko Rudolf Rosa Valentin Rosca Davide Rovati Olga Rudina Jack Rueter Kristjan Rúnarsson Shoval Sadde Pegah Safari Benoit SagotAleksi Sahala Shadi Saleh Alessio Salomoni Tanja Samardžić Stephanie Samson Manuela Sanguinetti Ezgi Saniyar,Dage Sarg Baiba Saulite Yanin Sawanakunanon Shefali Saxena Kevin Scannell Salvatore Scarlata Nathan Schneider Sebastian Schuster Lane Schwartz
Djame Seddah, Wolfgang Seeker, Mojgan Seraji, Syeda Shahzadi, Mo Shen, Atsuko Shimada, Hiroyuki Shirasu, Yana Shishkina, Muh Shohibussirri, Dmitry Sichinava, Janine Siewert, Einar Freyr Sigursson, Aline Silveira, Natalia Silveira, Maria Simi, Radu Simonescu, Katalin Simko, Maria Smikova, Kiril Simov, Maria Skachedubova, Aaron Smith, Isabela Soares-Bastos, Shafi Sourov, Carolyn Spadine, Rachele Sprugnoli, Steinhor Steingrímsson, Antonio Stella, Milan Straka, Emmett Strickland, Jana Strnadová, Alane Suhr, Yogi Lesmana Sulestio, Umut Sulubacak, Shingo Suzuki, Zsolt Szántó, Chiihiro Taguchi, Dima Taji, Yuta Takahashi, Fabio Tamburini, Mary Ann C. Tan, Takaaki Tanaka, Dipta Tanaya, Samson Tella, Isabelle Tellier, Marinella Testori, Guillaume Thomas, Liisi Torga, Marsida Toska, Trond Trosterud, Anna Trukhina, Reut Tsarfaty, Utku Türk, Francis Tyers, Sumire Uematsu, Roman Untilov, Zdenka Uresova, Larraitz Uria, Hans Uszkoreit, Andrius Utka, Sowmya Vajjala, Rob van der Goot, Martine Vanhove, Daniel van Niekerk, Gertjan van Noord, Viktor Varga, Eric Villemonte de la Clergerie, Veronika Vincze, Natalia Vlasova, Aya Wakasa, Joel C. Wallenberg, Lars Wallin, Abigail Walsh, Jing Xian Wang, Jonathan North Washington, Maximilan Wendt, Paul Widmer, Sri Hartati Wijono, Seyi Williams, Mats Wiren, Christian Wittern, Tsegay Woldemariam, Tak-sum Wong, Alina Wróblewska, Mary Yako, Kayo Yamashita, Naoki Yamazaki, Chunxiao Yan, Koichi Yasuoka, Marat M. Yavrumyan, Arife Betül Yenice, Olcay Taner Yildiz, Zhuoran Yu, Arlisa Yuliawati, Zdenek Žabokrtsky, Shorouq Zahra, Amir Zeldes, He Zhou, Hanzhi Zhu, Anna Zhuravleva, and Rayan Ziane. 2021. Universal dependencies 2.9. LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University.
A Treebanks
TARGET	LANG	FAMILY	SIZE
AR-PADT	Arabic	Afro-Asiatic	7.6k
EN-EWT	English	Indo-European	16.6k
FI-TDT	Finnish	Uralic	15.1k
GRC-PROIEL	Ancient Greek	Indo-European	17.1k
HE-HTB	Hebrew	Afro-Asiatic	6.2k
KO-GSD	Korean	Korean	6.3k
RU-GSD	Russian	Indo-European	5k
SV-Talbanken	Swedish	Indo-European	6.0k
ZH-GSD	Chinese	Sino-Tibetan	5.0k
Table 2: Target Treebanks based on Smith et al. (2018) with language family (FAMILY) and total number of sentences (SIZE).
Table 2 lists the nine target treebanks based on the set by Smith et al. (2018): AR-PADT (Hajic et al., 2009), EN-EWT (Silveira et al., 2014), FI
TDT (Pyysalo et al., 2015), GRC-PROIEL (Eckhoff et al., 2018), HE-HTB (McDonald et al., 2013a), KO-GSD (Chun et al., 2018), RU-GSD (McDonald et al., 2013b), SV-Talbanken (McDonald et al., 2013a), ZH-GSD (Shen et al., 2016). We use these treebanks as provided in Universal Dependencies v2.9 (Zeman et al., 2021). DEPPROBE and BAP are trained on each target's respective training split and are evaluated on the development split as this work aims to analyze general performance patterns instead of state-of-the-art performance.
DEPPROBE is implemented in PyTorch v1.9.0 (Paszke et al., 2019) and uses language models from the Transformers library v4.13.0 and the associated Model Hub (Wolf et al., 2020). Following the structural probe by Hewitt and Manning (2019), each token which is split by the LM encoder into multiple subwords is mean-pooled. Similarly, we follow the original hyperparameter settings and set the structural subspace dimensionality to  $b = 128$  and use embeddings from the middle layer of each LM (Hewitt and Manning, 2019; Tenney et al., 2019; Fayyaz et al., 2021). The structural loss is computed based on the absolute difference of the Euclidean distance between transformed word embeddings and the number of edges separating the words in the gold tree (see Hewitt and Manning, 2019 for details). The relational loss is computed using cross entropy between the logits and gold head-child relation. Optimization uses AdamW (Loshchilov and Hutter, 2018) with a learning rate of  $10^{-3}$  which is reduced by a factor of 10 each time the loss plateaus. Early stopping is applied after three epochs without improvement and a maximum of 30 total epochs. With the only trainable parameters being the matrices  $B$  and  $L$ , the model's footprint ranges between 51k and 190k parameters.
BAP For the biaffine attention parser (Dozat and Manning, 2017) we use the implementation in the MaChAmp framework v0.3 (van der Goot et al., 2021) with the default training schedule and hyperparameters. The number of trainable parameters depends on the LM encoder's size and ranges between 14M and 583M.
Analyses For our analyses in Sections 3 and 4 we further make use of numpy v1.21.0 (Harris et al., 2020), SciPy v1.7.0 (Virtanen et al., 2020) and Matplotlib v3.4.3 (Hunter, 2007).
Training Details Models are trained on an NVIDIA A100 GPU with 40GBs of VRAM and an AMD Epyc 7662 CPU. BAP requires around 1 h ( $\pm$  30 min). DEPPROBE can be trained in around 15 min ( $\pm$  5 min) with the embedding forward operation being most computationally expensive. The models use batches of size 32 and are initialized using the random seeds 692, 710 and 932.
Reproducibility In order to ensure reproducibility and comparability with future work, we release our code and token-level predictions at https://personads.me/x/naacl-2022-code.
Tables 3-11 list exact LAS and standard deviations for each experiment in Section 3's Figure 2 in addition to the HuggingFace Model Hub IDs of the LMs used in each of the 46 setups as well as their number of layers, embedding dimensionality  $d$  and total number of parameters. In addition, Figure 5 shows UUAS for all setups, equivalent to only probing structurally (Hewitt and Manning, 2019) for unlabeled, undirected dependency trees.
Figure 5: UUAS of DEPPROBE in relation to BAP across nine language targets (dev) using language-specific and multilingual LM encoders of different architecture types.
MODELS	SOURCE	LAYERS	EMB d	PARAMS	BAP	DEPPROBE
bert-base-multilingual-cased	Devlin et al. (2019)	12	768	178M	83.5±0.2	54.8±0.6
xlm-roberta-base	Conneau et al. (2020)	12	768	278M	85.2±0.1	57.2±0.1
google/rembert	Chung et al. (2021)	32	1152	576M	85.4±0.2	20.7±0.1
aubmindlab/bert-base-arabertv02	Antoun et al. (2020)	12	768	135M	85.8±0.1	59.0±0.1
asafaya/bert-base-arabic	Safaya et al. (2020)	12	768	111M	84.9±0.1	57.0±0.2
Table 3: LAS on AR-PADT (Dev) using BAP and DEPPROBE with different LMs (± standard deviation).
MODELS	SOURCE	LAYERS	EMB d	PARAMS	BAP	DEPPROBE
bert-base-multilingual-cased	Devlin et al. (2019)	12	768	178M	90.0±0.1	64.5±0.3
xlm-roberta-base	Conneau et al. (2020)	12	768	278M	91.7±0.2	64.8±0.1
google/rembert	Chung et al. (2021)	32	1152	576M	92.2±0.0	41.6±0.3
bert-base-uncased	Devlin et al. (2019)	12	768	109M	91.2±0.1	63.4±0.3
roberta-large	Liu et al. (2019)	24	1024	355M	92.3±0.2	59.9±0.2
Table 4: LAS on EN-EWT (Dev) using BAP and DEPPROBE with different LMs (± standard deviation).
MODELS	SOURCE	LAYERS	EMB d	PARAMS	BAP	DEPPROBE
bert-base-multilingual-cased	Devlin et al. (2019)	12	768	178M	89.1±0.2	54.5±0.4
xlm-roberta-base	Conneau et al. (2020)	12	768	278M	92.4±0.1	62.4±0.2
google/rembert	Chung et al. (2021)	32	1152	576M	93.1±0.1	30.8±0.1
TurkuNLP/bert-base-finnish-uncased-v1	Virtanen et al. (2019)	12	768	125M	93.4±0.1	68.9±0.3
TurkuNLP/bert-base-finnish-cased-v1	Virtanen et al. (2019)	12	768	125M	93.4±0.1	67.5±0.4
Table 5: LAS on FI-TDT (Dev) using BAP and DEPPROBE with different LMs (± standard deviation).
MODELS	SOURCE	LAYERS	EMB d	PARAMS	BAP	DEPPROBE
bert-base-multilingual-cased	Devlin et al. (2019)	12	768	178M	73.1±0.1	41.6±0.5
xlm-roberta-base	Conneau et al. (2020)	12	768	278M	85.0±0.2	51.1±0.2
google/rembert	Chung et al. (2021)	32	1152	576M	87.7±0.1	15.3±0.1
pranaydeeps/Ancient-Greek-BERT	Singh et al. (2021)	12	768	113M	87.3±0.1	60.0±0.0
nlpaeb/bert-base-greek-uncased-v1	Koutsikakis et al. (2020)	12	768	113M	84.6±0.3	53.9±0.1
Table 6: LAS on GRC-PROIEL (Dev) using BAP and DEPPROBE with different LMs (± standard deviation).
MODELS	SOURCE	LAYERS	EMB d	PARAMS	BAP	DEPPROBE
bert-base-multilingual-cased	Devlin et al. (2019)	12	768	178M	86.7±0.2	60.2±0.6
xlm-roberta-base	Conneau et al. (2020)	12	768	278M	88.8±0.1	59.2±0.3
google/rembert	Chung et al. (2021)	32	1152	576M	90.5±0.1	11.6±0.4
onlplab/alephbert-base	Seker et al. (2021)	12	768	126M	89.6±0.1	61.4±0.2
Table 7: LAS on HE-HTB (Dev) using BAP and DEPPROBE with different LMs (± standard deviation).
MODELS	SOURCE	LAYERS	EMB d	PARAMS	BAP	DEPPROBE
bert-base-multilingual-cased	Devlin et al. (2019)	12	768	178M	83.8±0.2	46.6±0.2
xlm-roberta-base	Conneau et al. (2020)	12	768	278M	86.1±0.1	49.4±0.3
google/rembert	Chung et al. (2021)	32	1152	576M	86.1±0.2	15.9±0.3
klue/bert-base	Park et al. (2021)	12	768	111M	86.8±0.0	51.0±0.1
klue/roberta-large	Park et al. (2021)	24	1024	337M	88.1±0.3	48.8±0.5
kykim/bert-kor-base	Kim (2020)	12	768	118M	86.8±0.1	46.9±0.4
Table 8: LAS on KO-GSD (Dev) using BAP and DEPPROBE with different LMs (± standard deviation).
MODELS	SOURCE	LAYERS	EMB d	PARAMS	BAP	DEPPROBE
bert-base-multilingual-cased	Devlin et al. (2019)	12	768	178M	89.1±0.1	60.7±0.1
xlm-roberta-base	Conneau et al. (2020)	12	768	278M	90.0±0.2	59.9±1.1
google/rembert	Chung et al. (2021)	32	1152	576M	90.8±0.0	26.0±0.2
cointegrated/rubert-tiny	Dale (2021)	3	312	11M	76.7±0.1	41.5±0.6
sberbank-ai/ruRoberta-large	Sber Devices (2021)	24	1024	355M	90.3±0.3	63.2±0.4
blinoff/roberta-base-russian-v0	Blinov (2021)	12	768	124M	75.8±0.0	15.6±0.2