SlowGuess/NAACL_2021_OCR huggingface dataset:
Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601, Florence, Italy. Association for Computational Linguistics.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142-147.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. In NeuRIPS.
Elena Voita and Ivan Titov. 2020. Information-theoretic probing with minimum description length.
Thomas Wolf, Julien Chaumont, Lysandre Debut, Victor Sanh, Clement Delangue, Anthony Moi, Pierric Cistac, Morgan Funtowicz, Joe Davison, Sam Shleifer, Remi Louf, Patrick von Platen, Tim Rault, Yacine Jernite, Teven Le Scao, Sylvain Gugger, Julien Plu, Clara Ma, Canwei Shen, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Daniel Zeman, Martin Popel, Milan Straka, Jan Hajic, Joakim Nivre, Filip Ginter, Juhani Luotolahti, Sampo Pyysalo, Slav Petrov, Martin Potthast, Francis Tyers, Elena Badmaeva, Memduh Gokirmak, Anna Nedoluzhko, Silvie Cinkova, Jan Hajic jr., Jaroslava Hlavacová, Václava Kettnerová, Zdenka Urešová, Jenna Kanerva, Stina Ojala, Anna Missilä, Christopher D. Manning, Sebastian Schuster, Siva Reddy, Dima Taji, Nizar Habash, Herman Leung, Marie-Catherine de Marneffe, Manuela Sanguinetti, Maria Simi, Hiroshi Kanayama, Valeria de Paiva, Kira Droganova, Héctor Martínez Alonso, Cagrī Çöltekin, Umut Sulubacak, Hans Uszkoreit, Vivien Macketanz, Aljoscha Burchardt, Kim Harris, Katrin Marheinecke, Georg Rehm, Tolga Kayadelen, Mohammed Attia, Ali Elkahky, Zhuoran Yu, Emily Pitler, Saran Lertpradit, Michael Mandl, Jesse Kirchner, Hector Fernandez Alcalde, Jana Strnadóva, Esha Banerjee, Ruli Manurung, Antonio Stella, Atsuko Shimada, Sookyoung Kwak, Gustavo Mendonça, Tatiana Lando, Rattima Nitisaroj, and Josie Li. 2017. CoNLL 2017 shared task: Multilingual parsing from raw text to Universal Dependencies. In Proceedings of the CoNLL 2017 Shared Task: Multilingual Parsing from Raw Text to Universal Dependencies, pages 1-19, Vancouver, Canada. Association for Computational Linguistics.
Kelly Zhang and Samuel Bowman. 2018. Language modeling teaches you more than translation does: Lessons learned through auxiliary syntactic task
analysis. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 359-361, Brussels, Belgium. Association for Computational Linguistics.
Mengjie Zhao, Tao Lin, Fei Mi, Martin Jaggi, and Hinrich Schütze. 2020. Masking as an efficient alternative to finetuning for pretrained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2226-2241, Online. Association for Computational Linguistics.
The mask parameters are optimized using Adam with  $\beta = (0.9, 0.999)$ ,  $\epsilon = 1 \times 10^{-8}$ , and learning rate 0.2 with linear warmup for the first  $10\%$  of the data. The classification head parameters are also optimized using Adam with the same hyperparameters and warmup, except with learning rate  $5 \times 10^{-5}$ . The MLP-1 and fine-tuning baselines are also optimized using Adam with the same hyperparameters, warmup, and learning rate  $5 \times 10^{-5}$ . We train for 30 epochs for all tasks.
A.2 Varying Regularization Strength
Table 2 shows probing accuracies for  $\lambda_{\mathrm{max}} \in (1,5,25,125)$ . Our method is consistently more selective than MLP-1 across the various values of  $\lambda_{\mathrm{max}}$ , except for  $\lambda_{\mathrm{max}} = 125$ , which seems to require too much sparsity.
A.3 Reproducibility Checklist
Experiments were run in Google Colab using a single 12GB NVIDIA Tesla K80 GPU. For each task, one run of fine-tuning took about half an hour. We used the transformers implementation of the bert-base-uncased model (Wolf et al., 2020; Devlin et al., 2019), which has 12 layers, 768 hidden dimension, 12 heads, and 110M parameters. As data, we used the dev (2002 examples) and train (12541 examples) splits of the English universal dependencies dataset (Zeman et al., 2017), and the test (3235 examples) and train (13862 examples) splits of the CoNLL 2003 NER shared task (Tjong Kim Sang and De Meulder, 2003).
Reset all	Reset encoder	Pre-trained
Part-of-speech Tagging
Subnetworks
λmax=1	71.53	87.53	93.39
λmax=5	70.45	87.20	92.41
λmax=25	68.65	86.23	90.66
λmax=125	66.39	86.10	84.86
MLP-1	69.16	86.53	90.25
Fine-tuning	84.42	86.47	95.69
Dependency Parsing
Subnetworks
λmax=1	39.84	54.31	86.86
λmax=5	43.36	54.93	85.99
λmax=25	41.43	54.41	83.12
λmax=125	43.07	53.70	74.49
MLP-1	42.81	54.09	76.65
Fine-tuning	74.48	79.10	89.93
Named Entity Recognition (NER)
Subnetworks
λmax=1	30.83	68.09	87.94
λmax=5	22.92	65.18	84.48
λmax=25	3.41	57.82	72.26
λmax=125	2.04	57.56	50.67
MLP-1	53.25	69.35	84.80
Fine-tuning	70.08	81.80	93.68
Table 2: Subnetwork probing accuracies while varying regularization strength.