SlowGuess/ACL_2020_OCR huggingface dataset:
in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2733-2743, Hong Kong, China. Association for Computational Linguistics.
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129-4138, Minneapolis, Minnesota. Association for Computational Linguistics.
Stephen C. Levinson. 2000. *Presumptive Meanings: The Theory of Generalized Conversational Implicature*. MIT Press.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1073-1094, Minneapolis, Minnesota. Association for Computational Linguistics.
Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340-2353, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Sieb G. Nooteboom, Fred Weerman, and F. N. K. Wijnen. 2002. Storage and Computation in the Language Faculty. Springer Science & Business Media.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018a. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018b. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1499-1509, Brussels, Belgium. Association for Computational Linguistics.
Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 2089-2096, Istanbul, Turkey. European Language Resources Association (ELRA).
Gözde Gül Sahin, Clara Vania, Ilia Kuznetsov, and Iryna Gurevych. 2019. LINSPECTOR: Multilingual probing tasks for word representations. CoRR, abs/1903.09442.
Naomi Saphra and Adam Lopez. 2019. Understanding learning dynamics of language models with SVCCA. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3257-3267, Minneapolis, Minnesota. Association for Computational Linguistics.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? Probing for sentence structure in contextualized word representations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Elena Voita and Ivan Titov. 2020. Information-theoretic probing with minimum description length. arXiv preprint arXiv:2003.12298.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. 2020. A theory of usable information under computational constraints. In International Conference on Learning Representations.
Daniel Zeman, Joakim Nivre, Mitchell Abrams, Noemi Aepli, Zeljko Agic, Lars Ahrenberg, Gabriele Aleksandraviciuté, Lene Antonsen, Katya Aplonova, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, Aitziber Atutxa, Liesbeth Augustinus, Elena Badmaeva, Miguel Ballesteros, Esha Banerjee, Sebastian Bank, Verginica Barbu Mititelu, Victoria Basmov, Colin Batchelor, John Bauer, Sandra Bellato, Kepa Bengoetxea, Yevgeni Berzak, Irshad Ahmad Bhat, Riyaz Ahmad Bhat, Erica Biagetti, Eckhard Bick, Agne Bielinskiené, Rogier Blokland, Victoria Bobicev, Loic Boizou, Emanuel Borges Völker, Carl Borstell, Cristina Bosco, Gosse Bouma, Sam Bowman, Adriane Boyd, Kristina Brokaite, Aljoscha Burchardt, Marie Candito, Bernard Caron, Gauthier Caron, Tatiana Cavalcanti, Gülsen Cebiroğlu Eryigit, Flavio Massimiliano Cecchini, Giuseppe G. A. Celano, Slavomír Čéplö, Savas Cetin, Fabricio Chalub, Jinho Choi, Yongseok Cho, Jayeol Chun, Alessandra T. Cignarella, Silvie Cinková, Aurélie Collomb, Çagri Çöltekin, Miriam Connor, Marine Courtin, Elizabeth Davidson, Marie-Catherine de Marneffe, Valeria de Paiva, Elvis de Souza, Arantza Díaz de Ilarraza, Carly Dickerson, Bamba Dione, Peter Dirix, Kaja Dobrovoljc,
Timothy Dozat, Kira Droganova, Puneet Dwivedi, Hanne Eckhoff, Marhaba Eli, Ali Elkahky, Binyam Ephrem, Olga Erina, Tomaz Erjavec, Aline Etienne, Wograine Evelyn, Richard Farkas, Hector Fernandez Alcalde, Jennifer Foster, Cláudia Freitas, Kazunori Fujita, Katarina Gajdošová, Daniel Galbraith, Marcos Garcia, Moa Gärdenfors, Sebastian Garza, Kim Gerdes, Filip Ginter, lakes Goenaga, Koldo Gojenola, Memduh Gökirmak, Yoav Goldberg, Xavier Gómez Guinovart, Berta González Saavedra, Bernadeta Griciūte, Matias Grioni, Normunds Grūzītis, Bruno Guillaume, Céline Guillot-Barbance, Nizar Habash, Jan Hajic, Jan Hajic jr., Mika Hämäläinen, Linh Hā My, Na-Rae Han, Kim Harris, Dag Haug, Johannes Heinecke, Felix Hennig, Barbora Hladká, Jaroslava Hlaváčová, Florinel Hociung, Petter Hohle, Jena Hwang, Takumi Ikeda, Radu Ion, Elena Irimia, Olajíde Ishola, Tomás Jelinek, Anders Johannsen, Fredrik Jørgensen, Markus Juutinen, Hūner Kaspikara, Andre Kaasen, Nadezhda Kabaeva, Sylvain Kahane, Hiroshi Kanayama, Jenna Kanerva, Boris Katz, Tolga Kayadelen, Jessica Kenney, Václava Kettnerova, Jesse Kirchner, Elena Klementieva, Arne Köhn, Kamil Kopacewicz, Natalia Kotsyba, Jolanta Kovalevskaité, Simon Krek, Sookyoung Kwak, Veronika Laippala, Lorenzo Lambertino, Lucia Lam, Tatiana Lando, Septina Dian Larasati Alexei Lavrentiev, John Lee, PhErlEang Lé Hong Alessandro Lenci, Saran Lertpradit, Herman Leung Cheuk Ying Li, Josie Li, Keying Li, KyungTae Lim, Maria Livina, Yuan Li, Nikola Ljubesić, Olga Loginova, Olga Lyashevskaya, Teresa Lynn, Vivien Macketanz, Aibek Makazhanov, Michael Mandl Christopher Manning, Ruli Manurung, Catalina Maranduc, David Mareček, Katrin Marheinecke, Héctor Martínez Alonso, André Martins, Jan Mašek Yuji Matsumoto, Ryan McDonald, Sarah McGuinness Gustavo Mendonça, Niko Miekka Margarita Misirpashayeva, Anna Missilă,Cătalin Mitelulu, Maria Mitrofan, Yusuke Miyao, Simonetta Montemagni, Amir More, Laura Moreno Romero Keiko Sophie Mori, Tomohiko Morioka Shinsuke Mori Shigeki Moro Bjartur Mortensen Bohdan Moskalevskyi Kadri Muischnek Robert Munro Yugo Murawaki Kaili Mūürisep Pinkey Nainwani Juan Ignacio Navarro Horniacek Anna Nedoluzhko Gunta Nešpore-Bérzkalne,LAErEang Nguyen Thi Huyen Nguyen Thi Minh Yoshihiro Nikaido Vitaly Nikolaev Rattima Nitisaroj Hanna Nurmi Stina Ojala Atul Kr. Ojha AdedayoI'Al Olioukun Mai Omura Petya Osenova Robert OstlingLilja Øvrelid,Niko Partanen,Elena Pascual Marco Passarotti Agnieszka Patejuk Guilherme Paulino-Passos Angelika Peljak-Lapińska Siyao Peng,Cenel-Augusto Perez,Guy Perrier Daria Petrova Slav Petrov Jason Phelan Jussi Piitulainen Tommi A Pirinen Emily Pitler Barbara Plank Thierry Poibeau Larisa Ponomareva Martin Popel Lauma Pretkalnina Sophie Prévost Prokopis Prokopidis Adam Przepiórkowski Tiina Puolakainen,Sampo Pyysalo Peng Qi Andriela Raïbis Alexandre Rademaker Loganathan Ra
masamy, Taraka Rama, Carlos Ramisch, Vinit Ravishankar, Livy Real, Siva Reddy, Georg Rehm, Ivan Riabov, Michael Rießler, Erika Rimkute, Larissa Rinaldi, Laura Rituma, Luisa Rocha, Mykhailo Romanenko, Rudolf Rosa, Davide Rovati, Valentin Rosca, Olga Rudina, Jack Rueter, Shoval Sadde, Benoit Sagot, Shadi Saleh, Alessio Salomoni, Tanja Samardžić, Stephanie Samson, Manuela Sanguinetti, Dage Särg, Baiba Saulite, Yanin Sawanakunanon, Nathan Schneider, Sebastian Schuster, Djame Seddah, Wolfgang Seeker, Mojgan Seraji, Mo Shen, Atsuko Shimada, Hiroyuki Shirasu, Muh Shohibussirri, Dmitry Sichinava, Aline Silveira, Natalia Silveira, Maria Simi, Radu Simionescu, Katalin Simko, María Šimková, Kiril Simov, Aaron Smith, Isabela Soares-Bastos, Carolyn Spadine, Antonio Stella, Milan Straka, Jana Strnadóva, Alane Suhr, Umut Sulubacak, Shingo Suzuki, Zsolt Szántó, Dima Taji, Yuta Takahashi, Fabio Tamburini, Takaaki Tanaka, Isabelle Tellier, Guillaume Thomas, Lisi Torga, Trond Trosterud, Anna Trukhina, Reut Tsarfaty, Francis Tyers, Sumire Uematsu, Zdenka Urešová, Larraitz Uria, Hans Uszkoreit, Andrius Utka, Sowmya Vajjala, Daniel van Niekerk, Gertjan van Noord, Viktor Varga, Eric Villemonte de la Clergerie, Veronika Vincze, Lars Wallin, Abigail Walsh, Jing Xian Wang, Jonathan North Washington, Maximilan Wendt, Seyi Williams, Mats Wiren, Christian Wittern, Tsegay Woldemariam, Tak-sum Wong, Alina Wróblewska, Mary Yako, Naoki Yamazaki, Chunxiao Yan, Koichi Yasuoka, Marat M. Yavrumyan, Zhuoran Yu, Zdeněk Žabokrtský, Amir Zeldes, Manying Zhang, and Hanzhi Zhu. 2019. Universal dependencies 2.5. LINDAT/CLARIAHCZ digital library at the Institute of Formal and Applied Linguistics (UFAL), Faculty of Mathematics and Physics, Charles University.
Kelly Zhang and Samuel Bowman. 2018. Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task analysis. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 359-361, Brussels, Belgium. Association for Computational Linguistics.
A Variational Bounds
Theorem 2. The estimation error between  $\mathcal{G}_{q_{\theta}}(T,R,\mathbf{e})$  and the true gain can be upper- and lower-bounded by two distinct Kullback-Leibler divergences.
Proof. We first find the error given by our estimate, which is a difference between two KL divergences—as shown in eq. (22) in Figure 1. Making use of this error, we trivially find an upper-bound on the estimation error as
$$
\begin{array}{l} \Delta = \mathrm {K L} _ {q _ {\theta_ {1}}} (T \mid R) - \mathrm {K L} _ {q _ {\theta_ {2}}} (T, \mathbf {c} (R)) \tag {23} \\ \leq \operatorname {K L} _ {q \theta_ {1}} (T \mid R) \\ \end{array}
$$
which follows since KL divergences are never negative. Analogously, we find a lower-bound as
$$
\begin{array}{l} \Delta = \mathrm {K L} _ {q _ {\theta_ {1}}} (T \mid R) - \mathrm {K L} _ {q _ {\theta_ {2}}} (T, \mathbf {c} (R)) \tag {24} \\ \geq - \mathrm {K L} _ {q _ {\theta 2}} (T, \mathbf {c} (R)) \\ \end{array}
$$
B Further Results
In this section, we present accuracies for the models trained using BERT, fastText and one-hot embeddings, and the full results on random embeddings. These random embeddings are generated once before the task, at the type level, and kept fixed without training. Table 3 shows that both BERT and fastText present high accuracies at POS labeling in all languages, except Tamil and Marathi. One-hot and random results are considerably worse, as expected, since they could not do more than take random guesses (e.g. guessing the most frequent label in the training test) in any word which was not seen during training. Table 4 presents similar results for dependency labeling, although accuracies for this task are considerably lower.
These tables also show how ambiguous the linguistic task is given the word types  $(\mathrm{H}(T \mid \mathrm{id}(R)))$ . These values were calculated using a plug-in estimator on the treebanks—which are known to underestimate entropies when used in undersampled regimes (Archer et al., 2014)—so they should not be considered as good approximations. Even so, we can see that most of the analysed languages are not very ambiguous with respect to POS labeling, and that there is a large variability of uncertainty across languages with respect to both tasks.
$$
\begin{array}{l} \mathcal {G} (T, R, \mathbf {e}) := \mathrm {H} (T; \mathbf {c} (R)) - \mathrm {H} (T \mid R) \tag {22} \\ = \mathrm {H} _ {q _ {\theta 2}} (T \mid \mathbf {c} (R)) - \underset {\mathbf {r} \sim p (\cdot)} {\mathbb {E}} \operatorname {K L} (p (\cdot \mid \mathbf {c} (\mathbf {r})) \mid | q _ {\theta 2} (\cdot \mid \mathbf {c} (\mathbf {r}))) \\ - \operatorname {H} _ {q _ {\boldsymbol {\theta} 1}} (T \mid R) + \underset {\mathbf {r} \sim p (\cdot)} {\mathbb {E}} \operatorname {K L} (p (\cdot \mid \mathbf {r}) | | q _ {\boldsymbol {\theta} 1} (\cdot \mid \mathbf {r})) \\ = \mathrm {H} _ {q _ {\theta 2}} (T \mid \mathbf {c} (R)) - \mathrm {K L} _ {q _ {\theta 2}} (T, \mathbf {c} (R)) - \mathrm {H} _ {q _ {\theta 1}} (T \mid R) + \mathrm {K L} _ {q _ {\theta 1}} (T \mid R) \\ = \mathrm {H} _ {q _ {\theta 2}} (T \mid \mathbf {c} (R)) - \mathrm {H} _ {q _ {\theta 1}} (T \mid R) + \mathrm {K L} _ {q _ {\theta 1}} (T \mid R) - \mathrm {K L} _ {q _ {\theta 2}} (T, \mathbf {c} (R)) \\ = \underbrace {\mathcal {G} _ {q _ {\boldsymbol {\theta}}} (T , R , \mathbf {e})} _ {\text {e s t i m a t e d g a i n}} + \underbrace {\mathrm {K L} _ {q _ {\boldsymbol {\theta} 1}} (T \mid R) - \mathrm {K L} _ {q _ {\boldsymbol {\theta} 2}} (T , \mathbf {c} (R))} _ {\text {e s t i m a t i o n e r r o r}} \\ \end{array}
$$
Figure 1: Derivation of the estimation error.
Language	accuracies	base entropies	random
BERT	fastText	one-hot	random	H(T)	H(T | id(R))	H(T | c(R))	G(T, R, c)
Basque	0.92	0.93	0.82	0.82	3.17	0.13	0.83	0.48 (15.0%)
Czech	0.98	0.98	0.91	0.87	3.33	0.06	0.57	0.47 (14.0%)
English	0.95	0.90	0.85	0.83	3.61	0.26	0.72	0.48 (13.4%)
Finnish	0.95	0.96	0.82	0.81	3.17	0.06	0.87	0.62 (19.6%)
Indonesian	0.92	0.92	0.86	0.84	3.24	0.16	0.68	0.30 ( 9.2%)
Korean	0.92	0.85	0.73	0.70	3.04	0.14	1.33	1.01 (33.1%)
Marathi	0.83	0.79	0.68	0.69	3.17	0.48	1.43	0.67 (21.1%)
Tamil	0.88	0.89	0.64	0.68	3.15	0.09	1.41	0.82 (26.2%)
Telugu	0.91	0.92	0.78	0.82	2.73	0.07	0.86	0.44 (16.2%)
Turkish	0.92	0.95	0.79	0.80	3.03	0.08	0.81	0.45 (14.7%)
Urdu	0.92	0.91	0.88	0.87	3.23	0.29	0.59	0.27 ( 8.3%)
Table 3: Accuracies of the models trained on BERT, fastText, one-hot and random embeddings for the POS tagging task.
Language	accuracies	base entropies	random
BERT	fastText	one-hot	random	H(T)	H(T | id(R))	H(T | c(R))	G(T, R, c)
Basque	0.87	0.83	0.71	0.65	4.03	0.55	1.71	1.08 (26.9%)
Czech	0.91	0.88	0.80	0.68	4.24	0.78	1.58	1.16 (27.3%)
English	0.91	0.78	0.72	0.68	4.48	1.01	1.61	1.16 (25.8%)
Finnish	0.87	0.85	0.65	0.56	4.42	0.52	2.21	1.59 (36.1%)
Indonesian	0.85	0.76	0.69	0.64	4.16	0.83	1.76	0.99 (23.9%)
Korean	0.92	0.84	0.68	0.56	4.17	0.35	2.08	1.68 (40.4%)
Marathi	0.75	0.70	0.61	0.62	4.01	0.81	2.12	0.73 (18.2%)
Tamil	0.76	0.74	0.51	0.54	3.78	0.31	2.32	1.15 (30.5%)
Telugu	0.80	0.78	0.67	0.69	3.64	0.31	1.96	0.88 (24.1%)
Turkish	0.77	0.75	0.59	0.54	3.95	0.54	2.14	1.02 (25.8%)
Urdu	0.87	0.80	0.76	0.73	3.83	1.02	1.26	0.63 (16.4%)
Table 4: Accuracies of the models trained on BERT, fastText, one-hot and random embeddings for the dependency labeling task.