SlowGuess/NAACL_2022_OCR huggingface dataset:
Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2021. Amnesic probing: Behavioral explanation with amnesic counterfactuals. Transactions of the Association for Computational Linguistics, 9:160-175.
Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. 2021. CausaLM: Causal Model Explanation Through Counterfactual Language Models. Computational Linguistics, 47(2):333-386.
Atticus Geiger, Hanson Lu, Thomas Icard, and Christopher Potts. 2021a. Causal abstractions of neural networks. In Advances in Neural Information Processing Systems, volume 34, pages 9574-9586. Curran Associates, Inc.
Atticus Geiger, Kyle Richardson, and Christopher Potts. 2020. Neural natural language inference models partially embed theories of lexical entailment and negation. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 163-173, Online. Association for Computational Linguistics.
Atticus Geiger, Zhengxuan Wu, Hanson Lu, Josh Rozner, Elisa Kreiss, Thomas Icard, Noah D. Goodman, and Christopher Potts. 2021b. Inducing causal structure for interpretable neural networks. ArXiv:2112.00826.
Mario Giulianielli, Marco Del Tredici, and Raquel Fernandez. 2020. Analysing lexical semantic change with contextualised word representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3960-3973, Online. Association for Computational Linguistics.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
Xiaqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. TinyBERT: Distilling BERT for natural language understanding. arXiv:1909.10351.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. arXiv:1609.07843.
Judea Pearl. 2009. Causality: Models, Reasoning and Inference, 2nd edition. Cambridge University Press, USA.
Reid Pryzant, Dallas Card, Dan Jurafsky, Victor Veitch, and Dhanya Sridhar. 2021. Causal effects of linguistic properties. In Proceedings of the 2021 Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, pages 4095-4109, Online. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Guy Rotman, Amir Feder, and Roi Reichart. 2021. Model compression for domain adaptation through causal effect estimation. arXiv:2101.07086.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. arXiv:1910.01108.
Peter Spirtes, Clark N Glymour, and Richard Scheines. 2001. Causation, Prediction, and Search, 2nd edition. MIT Press.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4323-4332, Hong Kong, China. Association for Computational Linguistics.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142-147.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Causal mediation analysis for interpreting neural NLP: The case of gender bias. arXiv:2004.12265.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
A.1 Standard Distillation Objectives
In our setting, our teacher model  $\mathcal{T}$  is a BERT model, and our student model  $S$  is a shallower BERT model with fewer layers.
Assume that we randomly draw a training example  $(\mathbf{x}_1,\mathbf{y}_1)\in \mathcal{D}$ , where  $\mathbf{x}_1$  is the input to our models and  $\mathbf{y}_1$  is the corresponding ground truth (the token prediction at each masked position). We denote the model predictions (output logits) as  $\mathcal{T}(\mathbf{x}_1)$  and  $\mathcal{S}(\mathbf{x}_1)$ . Additionally, we denote the contextualized representation for tokens for  $\mathbf{x}_1$  at the last layer as  $\mathrm{BERT}_{\mathcal{T}}(\mathbf{x}_1)$  and  $\mathrm{BERT}_{\mathcal{S}}(\mathbf{x}_1)$ .
We adopt the three standard distillation objectives of Sanh et al. (2019):
$\mathcal{L}_{\mathrm{MLM}}$  The masked language modeling loss of the student model calculated over all examples using the cross-entropy loss as follows:
$$
\sum_ {\left\{\mathbf {x} _ {1}, \mathbf {y} _ {1} \right\} \in \mathcal {D}} \operatorname {C E} \left(\mathcal {S} \left(\mathbf {x} _ {1}\right), \mathbf {y} _ {1}\right) \tag {4}
$$
$\mathcal{L}_{\mathrm{CE}}$  Following Hinton et al. (2015), the smoothed cross-entropy loss measuring the divergence between the student and teacher outputs as follows:
$$
\sum_ {\mathbf {x} _ {1} \in \mathcal {D}} \operatorname {C E} _ {\mathrm {S}} \left(\mathcal {S} \left(\mathbf {x} _ {1}\right), \mathcal {T} \left(\mathbf {x} _ {1}\right)\right) \tag {5}
$$
$\mathcal{L}_{\mathrm{Cos}}$  The cosine embedding loss defined in terms of the final hidden states of the teacher and the student as follows:
$$
\sum_ {\mathbf {x} _ {1} \in \mathcal {D}} \operatorname {C o s} \left(\operatorname {B E R T} _ {\mathcal {S}} \left(\mathbf {x} _ {1}\right), \operatorname {B E R T} _ {\mathcal {T}} \left(\mathbf {x} _ {1}\right)\right) \tag {6}
$$
As a result, comparing to standard DistilBERT, DIITO essentially adds a new type of objective by pushing the student model to become a causal abstraction of the teacher model.
A.2 Causal Distillation Objectives
In addition to our causal loss  $\mathcal{L}_{\mathrm{CE}}^{\mathrm{DIITO}}$ , we also propose a new loss  $\mathcal{L}_{\mathrm{Cos}}^{\mathrm{DIITO}}$  which is identical to  $\mathcal{L}_{\mathrm{Cos}}$  with interchange interventions. In this section, we provide a formal definition for  $\mathcal{L}_{\mathrm{Cos}}^{\mathrm{DIITO}}$ .
We denote our teacher and student models as  $\mathcal{T}$  and  $S$  respectively. Using the notational conventions from Section 3, we use  $\mathbf{N}_{\mathcal{T}}^{\mathbf{y}}$  and  $\mathbf{N}_S^{\mathbf{y}}$  to represent the neurons corresponding to the final
output for each model. Likewise, we use  $\mathbf{N}_{\mathcal{T}}^{L_{\mathcal{T}}}$  and  $\mathbf{N}_{\mathcal{S}}^{L_{\mathcal{S}}}$  to represent the neurons representing contextualized representation for each token after the final BERT layer.
Assuming we randomly sample a pair of examples from a training dataset  $(\mathbf{x}_1,\mathbf{y}_1),(\mathbf{x}_2,\mathbf{y}_2)\in \mathcal{D}$  we can then rewrite our causal loss  $\mathcal{L}_{\mathrm{CE}}^{\mathrm{DIITO}}$  by rearranging Eqn. 2 and Eqn. 3 as follows:
$$
\sum_ {\mathbf {x} _ {1}, \mathbf {x} _ {2} \in \mathcal {D}} \operatorname {C E} _ {\mathrm {S}} \binom {\text {G E T V A L S} \left(\mathcal {M} _ {\mathbf {S}} ^ {\mathbf {x} _ {1}}, \mathbf {x} _ {2}, \mathbf {N} _ {\mathcal {S}} ^ {\mathbf {y}}\right),} {\text {G E T V A L S} \left(\mathcal {M} _ {\mathbf {T}} ^ {\mathbf {x} _ {1}}, \mathbf {x} _ {2}, \mathbf {N} _ {\mathcal {T}} ^ {\mathbf {y}}\right)} \tag {7}
$$
where  $\mathcal{M}_{\mathrm{S}}^{\mathbf{x}_i}$  and  $\mathcal{M}_{\mathrm{T}}^{\mathbf{x}_i}$  are derived as in Eqn. 1 for each model respectively. Crucially, Eqn. 7 can be regarded as the causal form of the standard smoothed cross-entropy loss with interchange intervention. Likewise, we can further define the  $\mathcal{L}_{\mathrm{Cos}}^{\mathrm{DIITO}}$  as:
$$
\sum_ {\mathbf {x} _ {1}, \mathbf {x} _ {2} \in \mathcal {D}} \cos \binom {\text {G E T V A L S} \left(\mathcal {M} _ {\mathbf {S}} ^ {\mathbf {x} _ {1}}, \mathbf {x} _ {2}, \mathbf {N} _ {\mathcal {S}} ^ {L \mathcal {S}}\right),} {\text {G E T V A L S} \left(\mathcal {M} _ {\mathbf {T}} ^ {\mathbf {x} _ {1}}, \mathbf {x} _ {2}, \mathbf {N} _ {\mathcal {T}} ^ {L \mathcal {T}}\right)} \tag {8}
$$
with adjusted interchange alignments for  $\mathbf{N}_{\mathcal{T}}^{L\mathcal{T}}$  and  $\mathbf{N}_S^{L_S}$
A.3 Evaluation Set-up
GLUE We fine-tune for 25 epochs for the smaller datasets (RTE and CoLA) and 3 epochs for the others. Following Devlin et al. (2019) and Sanh et al. (2019), we use Matthew's Correlation for CoLA, F1 for MRPC and QQP, Spearman correlation for STS-B, and accuracy for all the other tasks in GLUE.
To foster reproducibility and provide a fair comparison between methods, we distill BERT for each condition with three distinct random seeds. We then fine-tune each model with five distinct random seeds. Consequently, we report results aggregated from three distinct runs for the language modeling task, and 15 distinct runs for others.
Named Entity Recognition We follow the experimental set-up in the Hugging Face (Wolf et al., 2020) repository for evaluation for the CoNLL-2003 Named Entity Recognition task (Tjong Kim Sang and De Meulder, 2003). For fine-tuning, we set the learning rate to  $5e^{-5}$  with an effective batch size of 32 for three epochs.
2For DistilBERT performance in Table 1 on CoNLL2003, we evaluate with a publicly available model downloaded from https://huggingface.co/delpart/distilbert-base-uncased-finetuned-ner.
Model	Layers	Pretraining
Tokens	General Language Understanding Evaluation (GLUE)
CoLA	MNLI	MRPC	QNLI	QQP	RTE	SST-2	STS-B
BERTBASE (Devlin et al., 2019) (Wikipedia+BookCorpus)	12	3.3B	56.30	84.70	88.60	91.80	89.60	69.30	92.70	89.00
DistilBERT (Sanh et al., 2019) (Wikipedia+BookCorpus)	6	3.3B	51.30	82.10	87.50	89.20	88.50	59.90	91.30	86.90
DistilBERT (WikiText)	3	0.1B	22.78	71.55	82.51	82.12	82.16	55.43	86.47	56.33
DIITOMIDDLE (WikiText)	3	0.1B	23.21	72.97	82.81	83.15	82.83	55.98	86.52	66.93
DIITOLATE (WikiText)	3	0.1B	24.12	72.80	82.16	82.88	82.85	57.29	87.31	62.65
DIITOFULL (WikiText)	3	0.1B	25.01	72.85	82.71	83.05	82.85	55.37	86.92	66.15
DistilBERT (WikiText)	6	0.1B	40.43	78.95	87.45	84.76	84.96	60.10	89.38	80.40
DIITOMIDDLE (WikiText)	6	0.1B	43.97	79.47	87.57	85.45	85.21	60.72	89.97	81.33
DIITOLATE (WikiText)	6	0.1B	43.93	79.49	87.70	85.79	85.22	60.14	90.31	81.79
DIITOFULL (WikiText)	6	0.1B	43.43	79.66	88.17	85.57	85.28	59.95	90.01	81.26
DIITOTFULL+Random (WikiText)	6	0.1B	44.27	79.70	88.06	85.63	85.34	60.89	89.76	81.08
DIITOTFULL+Masked (WikiText)	6	0.1B	43.39	79.63	87.88	85.61	85.30	61.06	89.97	81.58
DIITOTFULL+LCOsDIITOTFULL(WikiText)	6	0.1B	45.17	79.68	88.18	85.83	85.31	60.94	90.32	81.69
Table 2: Model performance results on the development sets of the GLUE benchmark. The GLUE score is the averaged performance scores across 15 distinct runs with precision aligned for a fair comparison. Following the evaluation for BERT (Devlin et al., 2019), we exclude WNLI for evaluation.
Question Answering We use the experimental set-up of Sanh et al. (2019) for evaluation on SQuAD v1.1 (Rajpurkar et al., 2016). For fin-tuning, we set the learning rate to  $3e^{-5}$  with an effective batch size of 48 for two epochs. We set the stride to 128.