SlowGuess/NAACL_2021_OCR huggingface dataset:
Natalia Silveira, Timothy Dozat, Marie-Catherine de Marneffe, Samuel Bowman, Miriam Connor, John Bauer, and Chris Manning. 2014. A gold standard dependency corpus for English. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pages 2897-2904, Reykjavik, Iceland. European Language Resources Association (ELRA).
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. What do you learn from context? probing for sentence structure in contextualized word representations. In International Conference on Learning Representations.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Causal mediation analysis for interpreting neural nlp: The case of gender bias.
Alex Wang, Ian F. Tenney, Yada Pruksachatkun, Katherin Yu, Jan Hula, Patrick Xia, Raghu Pappagari, Shuning Jin, R. Thomas McCoy, Roma Patel, Yinghui Huang, Jason Phang, Edouard Grave, Najoung Kim, Phu Mon Htut, Thibault F'evry, Berlin Chen, Nikita Nangia, Haokun Liu, Anhad Mohananey, Shikha Bordia, Ellie Pavlick, and Samuel R. Bowman. 2019. jiant 1.0: A software toolkit for research on general-purpose text understanding models. http://jiant.info/.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23.
Peng Xu, Jaeho Kang, Michael Ringgaard, and Franz Och. 2009. Using a dependency parser to improve smt for subject-object-verb languages. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, NAACL '09, page 245-253, USA. Association for Computational Linguistics.
Jason Yosinski, Jeff Clune, Anh Mai Nguyen, Thomas J. Fuchs, and Hod Lipson. 2015. Understanding neural networks through deep visualization. CoRR, abs/1506.06579.
A.1 Additional Example of the Extreme Case
We show another example of a task-pair that, under certain distributions of context lengths, exhibits similar behavior to that observed in the edge case described in §3.1.1 (figure 7).
Figure 7:  $\mathbb{E}_{layer}$  of SRL and Non-term. for different context length ranges (4 left blue and yellow pairs), and their  $\mathbb{E}_{layer}$  when all instances of SRL are of context length  $l\in [0,2]$  and all those of Non-term. are of context length  $l\geq 9$  (rightmost green and red pair). While for every context length range, SRL's  $\mathbb{E}_{layer}$  is bigger than that of Non-term., for some context length distribution that order may be reversed.
A.2 Context Length Distribution
A lot of our work deals with possible context length distributions, normalizing distribution, and accounting for the distribution. We provide here the actual distributions which are the underlying property controlling the seen effects. We provide data on the percentage of examples in each context length range for each task (figure 8).
Figure 8: Percentage of examples as a function of context length range, for each of the 7 tasks (see legend).
A.3 NDE vs. Unmediated Difference for All Task-Pairs
For every task-pair, we compare the unmediated  $\mathbb{E}_{layer}$  difference with the pair's NDE. Figure 9 presents this comparison for each task-pair, with
the distribution of one of the pair's tasks being applied in the NDE calculations, for each task-pair.
A.4 Extreme  $\mathbb{E}_{layer}$  Differences
Based on figure 6, we compute the extreme  $\mathbb{E}_{layer}$  differences of each task-pair. Namely, for each such pair, we juxtapose the difference between the maximal possible  $\mathbb{E}_{layer}$  of the first task and the minimal  $\mathbb{E}_{layer}$  of the second one with the opposite case (the difference between the minimal possible  $\mathbb{E}_{layer}$  of the first task and the maximal  $\mathbb{E}_{layer}$  of the second one). Our results can be seen in figure 10.
Figure 9: Difference between unmediated  $\mathbb{E}_{layer}$  and NDE for every task-pair. The employed context length distributions (as part of the NDE calculations) are, from left to right, of NER, SRL, Dep., Non-term., SRL, Co-ref., Dep., Non-term., SRL, Non-term., SPR, SRL, SPR, SPR, Non-term., SRL, RC, NER, Non-term., Dep. and SRL.
Figure 10: Difference between the minimal possible expected layer of the left task and the maximal possible expected layer of the right task (blue - see legend), and vice-versa (yellow - see legend), for every task-pair.