SlowGuess/NAACL_2022_OCR huggingface dataset:
We also found that the QA model used representations of syntax causally, contrary to prior findings, through a series of similar causal analysis experiments using syntactically-ambiguous inputs. The QA model is a BERT-based model fine-tuned on a question-answering task to map from context and a question to a continuous span of the context that answered the question (Rajpurkar et al., 2016).
We performed experiments using depth- and distance-based probes, using dropout values at increments of 0.1 from 0 to 0.9. We used three test suites for analyzing the causal use of syntax in the QA model: "Coordination", "Relative Clause" (RC), and a "Noun Phrase/Verb Phrase" (NP/VP) suite. The Coordination suite consisted of 256 prompts with coordination ambiguity like, "I saw the men and the women were tall. Who was tall?" The RC suite consisted of 193 prompts with attachment ambiguity of a relative clause like, "I saw the women and the men who were tall. Who was tall?" The NP/VP suite consisted of 256 prompts like, "The girl saw the boy with the telescope. Who had the telescope?" Prompts were designed such that answers were dictated by syntactic interpretations.
Findings for the Coordination suite are plotted
QA Model Causal Effect on Coordination Suite by Layer
Figure 5: Causal effects for the QA model using depth- (top row) and distance- (bottom row) based probes with dropout of 0 (left column) or 0.4 (right column) on the Coordination corpus to counterfactual loss 0.05. Dropout probes produce more stable and larger effect sizes. Means and standard deviations over 5 trials plotted.
Figure 6: Mean causal effects when using depth-based probes for the QA test suites. Smaller counterfactual losses and higher dropout rates typically induced greater effects, although the scale of the effects varied by suite (note different axis scales). Means and standard deviations over 5 trials.
in Figure 5. On the  $y$  axis, we plotted the model's prediction of words in the first noun phrase (NP1) starting the answer. Correct causal use of representations of syntax would move the red line (corresponding to parses indicating NP1) above the original outputs, in green, and the blue line (for the other parse) below.
Unlike prior art, we found evidence that QA models use representations of syntax causally. In the left column of Figure 5, we found similar results to prior art: using standard depth-based probes produced noisy results, and distance-based probes had a small effect. (In fact, this effect size shrank if we only trained the distance probe for 30 epochs, as in prior art, instead of the 100 epochs we used, indicating the importance of well-trained probes.) In contrast to the standard probes, the dropout probes, plotted in the right column, revealed much larger effects of syntactic interventions.
More systematic analysis for all dropout rates, using distance and depth-based probes for all 3 test suites confirmed these trends. We plotted the aggregate metrics for all suites using depth probes
in Figure 6. The causal effects were smaller in the RC and NP/VP suites than in the Coordination suite, indicating that the model may have learned a weaker causal link for these syntactic relations. Nevertheless, all suites demonstrate the importance of using dropout in probes: without dropout (solid black curve), the causal effects were smaller than for any positive dropout rate.
We note briefly that the causal effects uncovered by dropout probes may not be solely attributed to dropout probes performing better at their parsing task. In fact, adding dropout worsened probe performance according to typical probe performance metrics (Appendix E).
4.2.3 NLI Model
Lastly, we performed similar causal analysis on the NLI and NLI-HANS models and, in contrast to the Mask and QA models, we found no evidence for the causal use of syntax using any of our probes for either model. The NLI model was finetuned on just the MNLI corpus, and the NLI-HANS model was finetuned with both the MNLI and HANS corpora,
Figure 7: Using dropout probes over a range of dropout values (different curves) and counterfactual stopping losses improved model performance, and dropout typically improved performance. Medians and quartiles plotted over 5 trials.
based on code from Gao et al. (2021). The NLI model had a test set accuracy of  $86\%$ , and the NLI-HANS model had test set accuracy of  $93\%$ .
We used a test suite based on the Coordination suite already introduced in this work: an example prompt was "The person saw the keys in the cabinets which are green. The keys are green." The models had to classify these inputs among three classes of entailment, contradiction, or neutrality.
Ultimately, we failed to find any evidence that either the NLI or the NLI-HANS model used syntactic information causally. The models always predicted entailment for all prompts, whether using original embeddings or counterfactuals generated for different parses. We used distance probes with dropout values from 0 to 0.9 and created counterfactuals for losses from 0.05 to 0.3 and never observed a shift in predicted probability mass of more than  $1\%$  when using counterfactuals. Unfortunately, this suggests that simply augmenting the MNLI dataset with HANS may not be enough to produce a model that uses syntactic information causally.
4.3 Boosting Performance with Probes
Earlier, we demonstrated that the QA model causally used representations of syntax for predictions; here, we showed that we could improve QA model performance at test time by "injecting" syntactic information into embeddings. Because prior art had not found that QA models used syntax causally, such interventions were not previously pursued, as far as we are aware.
We designed a new, syntactically challenging "Intervene" test suite of 288 prompts for the QA model. Example prompts are "The person saw the keys by the cabinet which was green. What was green?" and "The person saw the keys by the cabinet which were green. What was green?" An
swering correctly ("the cabinet" first and "the keys" second) depends upon using noun-verb agreement. We used template-generated parse trees for each sentence and distance probes to create counterfactual embeddings for each sentence at layer 4 of the QA model. Layer 4 was chosen based on performance on a validation dataset (Appendix C).
We passed the original and counterfactual embeddings through the QA model and measured performance on a test suite. F1 performance is plotted in Figure 7; exact match metrics had similar trends. Typically, higher-dropout probes improved performance more, although the highest-dropout probes deteriorated for the lowest counterfactual losses. We hypothesize that this deterioration corresponded to generating out-of-distribution embeddings, but this topic warrants further study.
Lastly, we performed a similar experiment using the NLI and NLI-HANS models using 486 prompts drawn from the HANS dataset like "The doctor near the actor danced. The actor danced" (McCoy et al., 2019). The NLI model achieved  $50\%$  accuracy (always predicting entailment) and the NLI-HANS model achieved  $99\%$  accuracy. Neither model's accuracy changed significantly when using counterfactuals with the correct parse for the first sentence, yet again indicating that these models may not use representations of syntax causally.
5 Contributions and Conclusion
In this work, we designed and evaluated "dropout probes," a new neural probing architecture for generating useful causal analysis of trained language models. Our technical contribution — adding a dropout layer before probes — was inspired by a theory of redundant syntactic encodings in models. Our results fit within three categories: we showed that 1) models encoded syntactic information redundantly, 2) dropout probes, unlike standard probes, revealed that QA models used syntactic representations causally, and 3) by injecting syntactic information at test time in syntactically-challenging domains, we could increase model performance without retraining.
Despite our step towards better understanding of pretrained models, future work remains. Natural extensions include studying pretrained models beyond those considered in this work, further research into redundancy in embeddings, more investigation into inserting symbolic knowledge into neural representations, and new methods for training models
to respond appropriately to interventions.
6 Ethical and Broader Impacts
While the majority of this paper details the technical contributions of our work, here, we briefly consider some of the possible consequences of our findings based on transparency and causal modeling.
Fundamentally, we believe that causal analysis of models is a powerful tool towards more ethical AI. Our dropout probes enable better inspection of models, providing possible mechanisms for regulators, ethicists, and even the general public to better understand AI systems with which they interact. By injecting information into models at test time, as demonstrated in Section 4.3, we provide another mechanism for people to control model behavior. Thus, our tool may reinforce values of transparency and value-alignment in AI, contingent upon access to the model for probing.
While we hope that our probing mechanism will be used for good, misuse of the tool is certainly possible. In particular, the very causal rules that our tool uncovers may be used to reinforce biases. For example, people may attempt to argue that a gender bias exhibited by a model are evidence of the "correctness" of that bias. We urge readers to remember that models likely reflect biases present in human-generated data and certainly not "true" stereotypes.
We also note that the transparency benefits of our technique are not universally accessible. Training a single probe on a single layer took approximately 2 minutes on an NVIDIA GeForce 3080; generating counterfactuals took approximately 1 second per counterfactual on similar hardware. Although these operations individually are relatively lightweight (and certainly less computationally intensive than finetuning a whole model), systematic evaluation of models for many layers, multiple probes, and many counterfactuals is more challenging. Furthermore, all analysis assumes access to the internals of the pretrained model itself.
Lastly, while our work is limited to diagnosis of existing models, we hope that it will enable important future research in causally-motivated models. We hope to ultimately develop models that blend causal rules based on human guidance with emergent learned patterns from data. Our work can complement such research by certifying that models have indeed learned the desired rules.
TE acknowledges support from the GEM consortium and the National Science Foundation Graduate Research Fellowship under Grant No. 1745302.
Guillaume Alain and Yoshua Bengio. 2017. Understanding intermediate layers using linear classifier probes. *ArXiv*, abs/1610.01644.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeshwar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and Devon Hjelm. 2018. Mutual information neural estimation. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 531-540. PMLR.
Yonatan Belinkov. 2021. Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics, pages 1-13.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Noam Chomsky. 1965. Aspects of the Theory of Syntax. MIT press.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. 2018. What you can cram into a single  $\$ \& !\#$ * vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2126-2136, Melbourne, Australia. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yanai Elazar, Shauli Ravfogel, Alon Jacovi, and Yoav Goldberg. 2020. Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals. arXiv eprints, page arXiv:2006.00995.
Amir Feder, Nadav Oved, Uri Shalit, and Roi Reichart. 2021. CausaLM: Causal model explanation through counterfactual language models. Computational Linguistics, 47(2):333-386.
Yang Gao, Nicolò Colombo, and Wei Wang. 2021. Adapting by pruning: A case study on BERT. CoRR, abs/2105.03343.
Mario Giulianielli, Jack Harding, Florian Mohnert, Dieuwke Hupkes, and Willem Zuidema. 2018. Under the hood: Using diagnostic classifiers to investigate and improve how language models track agreement information. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 240-248, Brussels, Belgium. Association for Computational Linguistics.
Rowan Hall Maudslay, Josef Valvoda, Tiago Pimentel, Adina Williams, and Ryan Cotterell. 2020. A tale of a probe and a parser. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7389-7395, Online. Association for Computational Linguistics.
John Hewitt and Christopher D Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129-4138.
Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2020. Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521-535.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330.
Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428-3448, Florence, Italy. Association for Computational Linguistics.
Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. 2018. Stress test evaluation for natural language inference. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2340-2353, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Judea Pearl and Dana Mackenzie. 2018. The Book of Why: The New Science of Cause and Effect, 1st edition. Basic Books, Inc., USA.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv e-prints, page arXiv:1606.05250.
Shauli Ravfogel, Yanai Elazar, Hila Gonen, Michael Twiton, and Yoav Goldberg. 2020. Null it out: Guarding protected attributes by iterative nullspace projection. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7237-7256, Online. Association for Computational Linguistics.
Emily Reif, Ann Yuan, Martin Wattenberg, Fernanda B Viegas, Andy Coenen, Adam Pearce, and Been Kim. 2019. Visualizing and measuring the geometry of bert. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Ivan Sanchez, Jeff Mitchell, and Sebastian Riedel. 2018. Behavior analysis of NLI models: Uncovering the influence of three factors on robustness. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1975-1985, New Orleans, Louisiana. Association for Computational Linguistics.
Mycal Tucker, Peng Qian, and Roger Levy. 2021. What if this modified that? syntactic interventions with counterfactual embeddings. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 862-875, Online. Association for Computational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. CoRR, abs/1910.03771.
A MINE Details
The Mutual Information Neural Estimator technique works by training a neural net to compute and maximize a lower bound on mutual information between two random variables (Belghazi et al., 2018). We describe the intuition of the technique, as well as our implementation, in this section; we refer readers to the full paper for theoretical analysis of MINE.
The mutual information between two variables is defined via the KL Divergence between the joint distribution of the variables and the product of their marginals:  $I(X,Y) = D_{KL}(P(XY)||P(X)P(Y))$ . For notational simplicity, we describe the joint distribution as  $P$  and the product of the marginals as  $Q$ . Let us further state that  $P$  and  $Q$  define outputs that are jointly in  $R^{D}$ .
A lower-bound for the KL divergence is as follows, setting  $F$  as any class of functions that maps from  $R^D$  to  $R$ :
$$
D _ {K L} (P | | Q) \geq \sup  _ {T \in F} \mathbb {E} _ {P} [ T ] - (\mathbb {E} _ {Q} [ e ^ {T} ]) \quad (1)
$$
In other words, one can lower bound the mutual information by finding a function,  $T$ , that maximizes the difference between the two terms in Equation 1. Belghazi et al. (2018) do so with functions parametrized as a neural net that maps from the concatenation of two inputs (one for each random variable) to a single-valued output. Training the neural net is conducted to maximize the value described by Equation 1.
In our experiments, we create neural networks with separate, linear layers of size 64 for each input. The embeddings from those two layers are concatenated, passed through two 1024-dimensional layers with ReLU activations, and then passed through a linear layer with a single output. We thus mapdp from the two inputs to a single, real-valued output.
Training was performed using batch size 32 over 50 epochs, at which point the mutual information estimates appeared to have converged.
B Test Suite Creation
Here, we specify the details of the test suites used to evaluate models for reproducibility.
The Mask model Coordination test suite comprised sentences like "The man saw the girl and the dog [MASK] tall." More generally, sentences followed the following template: "The NN1 V the NN2 and the NN3 [MASK] ADJ." We created all sentences by iterating through the combinations of the words described in Table 2. This generated 243 sentences, and each sentence was associated with 2 parses: one described as a conjunction of sentences (e.g., "(The man saw the girl) and (the dog [MASK] tall.)") and one as a single sentence with a conjunction of noun phrases (e.g., "The man saw (the girl and the dog) [MASK] tall").
Category	Words
NN1	man, woman, child
NN2	boy, building, cat
NN3	dog, girl, truck
V	saw, feared, heard
ADJ	tall, falling, orange
Table 2: Words used for sentence generation in the Mask Coordination test suite.
The mask model NP/Z test suite comprised sentences like, "When the dog scratched the vet [MASK] ran." More generally, sentences followed the following template: "When the NN1 V1 the NN2 [MASK] V2." Each sentence was associated with two parses, favoring either adverbs (e.g., "When the dog scratched the vet quickly ran" or nouns, "When the dog scratch the vet she ran"). We used the word tuples described in Table 3, inspired by prior art, to generate 150 sentences.
The QA model Coordination test suite comprised prompts like "Who was tall? The happy stranger saw the angry men and the angry women were tall." More generally, the prompts followed the following template: "Who was ADJ1? The ADJ2 NN1 V the ADJ3 NN3 and the ADJ4 NN4 were ADJ1." We created 256 prompts by iterating through combinations of the words in Table 4. "None" adjectives were excluded from the text.
The QA model NP/VP suite comprised prompts like "Who had the telescope? The girl saw the boy with the telescope." The prompts followed the following template: "Who had the NN1? The ADJ1 NN2 ADV V the ADJ2 NN3 with the ADJ3 NN4." In this suite, the choice of V and NN4 was tightly coupled - one may see with a telescope but not see with a stick, for example. Table 5 details the combinations of words used to fill out the template, including V-NN4 pairs. Overall, we generated 256 prompts.
The QA model RC suite comprised prompts like "Who was desperate? The women and the men who were desperate bribed the politician." The prompts followed the following template: "Who was ADJ1? The ADJ2 NN1 and the ADJ3 NN2 who were ADJ1 V the NN3." We generated 192 example prompts by iterating over combinations of the words listed in Table 6, excluding sentences in which NN1 and NN2 or ADJ2 and ADJ3 would have been the same.
The Intervention suite for the QA model com
NN1	V1	NN2	V2
(dog/child)	(scratched/bit)	(vet/girl/boy)	(ran/screamed/smiled)
author	wrote	book	grew
(doctor/professor)	lectured	student	listened
(girls/boys)	raced	(kids/children)	(watched/cheered)
(people/spectators)	watched	(show/movie)	(stopped/paused)
(lawyers/judges)	(studied/considered)	case	(languished/proceeded)
(people/viewers)	(notice/spot)	actor	(departs/stays)
(band/conventions)	left	(hotel/stalls)	closed
Table 3: Words used for sentence generation in the Mask NP/Z test suite.
Category	Words
ADJ1	tall, short
ADJ2	happy, None
ADJ3	angry, None
ADJ4	angry, None
NN1	stranger, child
NN2	men, women
NN3	women, men
V	saw, believed
Table 4: Words used for sentence generation in the QA Coordination test suite.
Category	Words
V - NN4	(saw, telescope), (poked, stick)
ADJ1	tall, None
ADJ2	short, None
ADJ3	special, None
NN1	man, woman
NN2	boy, girl
prised prompts like "What was green? The human saw the keys by the cabinet which were green." More generally, prompts were created via the following template: "What was ADJ1? The NN1 V the NN2 by the NN3 which was/were ADJ1." By changing the plurality of NN2 or NN3 and replacing "was" with "were," the correct answer should change. Overall, we generated 288 sentences by iterating over all combinations of the words listed in Table 7, such that exactly one of NN1 and NN2 was plural at a time.
In the intervention experiments in Section 4.3, we performed interventions at layer 4, based on results of a validation study shown below. We reported
Table 5: Words used for sentence generation in the QA NP/VP test suite.
Category	Words
ADJ1	corrupt, desperate
ADJ2	tall, smart, rich
ADJ3	tall, smart, rich
NN1	men, women
NN2	men, women
NN3	judge, politician
Table 6: Words used for sentence generation in the QA RC test suite.
Category	Words
ADJ1	green, large, dirty
NN1	human, stranger, child
NN2	key, keys, gadget, gadgets
NN3	cabinet, cabinets, vase, vases
Table 7: Words used for sentence generation in the QA intervention experiments.