SlowGuess/NAACL_2022_OCR huggingface dataset:
Mycal Tucker
mycal@mit.edu
Tiwalayo Eisape
eisape@mit.edu
Peng Qian
pqian@mit.edu
julie_a_shah@csail.mit.edu
Recent causal probing literature reveals when language models and syntactic probes use similar representations. Such techniques may yield "false negative" causality results: models may use representations of syntax, but probes may have learned to use redundant encodings of the same syntactic information. We demonstrate that models do encode syntactic information redundantly and introduce a new probe design that guides probes to consider all syntactic information present in embeddings. Using these probes, we find evidence for the use of syntax in models where prior methods did not, allowing us to boost model performance by injecting syntactic information into representations.
Figure 1: In a 2D embedding space, a model might redundantly encode syntactic representations of a sentence like "the girl saw the boy with the telescope." Redundant encodings could cause misalignment between the model's decision boundary (blue) and a probe's (red). We introduce dropout probes (green) to use all informative dimensions.
Recent large neural models like BERT and GPT-3 exhibit impressive performance on a large variety of linguistic tasks, from sentiment analysis to question-answering (Devlin et al., 2019; Brown et al., 2020). Given the models' impressive performance, but also their complexity, researchers have developed tools to understand what patterns models have learned. In probing literature, researchers develop "probes:" models designed to extract information from the representations of trained models (Linzen et al., 2016; Conneau et al., 2018; Hall Maudslay et al., 2020). For example, Hewitt and Manning (2019) demonstrated that one can train accurate linear classifiers to predict syntactic structure from BERT or ELMO embeddings. These probes reveal what information is present in model embeddings but not how or if models use that information (Belinkov, 2021).
To address this gap, new research in causal analysis seeks to understand how aspects of models' representations affect their behavior (Elazar et al., 2020; Ravfogel et al., 2020; Giulianelli et al., 2018; Tucker et al., 2021; Feder et al., 2021). Typically, these techniques create counterfactual representations that differ from the original according to some
property (e.g., syntactic interpretation of the sentence). Researchers then compare outputs when using original and counterfactual embeddings to assess whether a property encoded in the representation is causally related to model behavior.
Unfortunately, negative results — wherein researchers report that models do not appear to use a property causally — are difficult to interpret. Such failures can be attributed to a model truly not using the property (true negatives), or to a failure of the technique (false negatives). For example, as depicted in Figure 1, if a language model encodes syntactic information redundantly (here illustrated in two-dimensions), the model and probe may differentiate among parses along orthogonal dimensions. When creating counterfactual representations with such probes, researchers could incorrectly conclude that the model does not use syntactic information.
In this work, we present new evidence for the causal use of syntactic representations on task performance in BERT, using newly-designed probes that take into account the potential redundancy in a model's internal representation. First, we find evidence for representational redundancy in BERT-based models. Based on these findings, we propose
Human Language Technologies, pages 5393 - 5408
a new probe design that encourages the probe to use all relevant representations of syntax in model embeddings. These probes are then used to assess if language models use representations of syntax causally, and, unlike prior art, we find that some fine-tuned models do exhibit signatures of causal use of syntactic information. Lastly, having found that these models causally use representations of syntax, we used our probes to boost a question-answering model's performance by "injecting" syntactic information at test time. $^{1}$
2.1 Language Model Probing
Probing literature seeks to expose learned patterns of a neural language model by training small neural networks to map from model representations to human-interpretable properties (Alain and Bengio, 2017; Conneau et al., 2018; Reif et al., 2019). For example, Hewitt and Manning (2019) propose single-layered neural nets that map from embeddings to syntactic representations of sentences. Such probing methods are correlative rather than causal because they depict what information is present in representations instead of how that information is used (Hall Maudslay et al., 2020; Pearl and Mackenzie, 2018). Understanding when language models use structural information causally is an important question given the central role structure appears to play in human understanding of natural language (Chomsky, 1965). In this work, we perform causal analysis by combining causal methods with a new probe design.
2.2 Causal Analysis of Language Models
Recently, researchers have begun applying causal analysis to language models to understand if and how they use human-interpretable properties in their decision making. While direct text manipulations are sometimes possible (e.g., modifying "The man works as a..." to "The woman works as a ..."), several methods rely on constructing counterfactual representations to measure model behavior (Kaushik et al., 2020; Ravfogel et al., 2020). Prior art has often found that standard models learn undesirable causal relationships by encoding unwanted biases or by not learning to rely upon syntactic principles (Feder et al., 2021; Elazar et al., 2020).
Our work is most closely related to Tucker et al. (2021), so we explain their technical approach here. Tucker et al. (2021) train non-linear structural probes (based on those designed by Hewitt and Manning (2019)) to predict aspects of a sentence's syntactic structure from model embeddings. That is, a probe  $p$  maps from an embedding,  $z$ , to a representation of syntax,  $s$ . Trained probes are used to create counterfactual embeddings,  $z'$ , by updating  $z'$  from  $z$  via gradient descent to minimize a loss function,  $L$ , evaluated on the probe's output and a desired output based on an alternative syntactic interpretation,  $s'$ :  $\nabla_{z'} L(p(z'), s')$ . Intuitively, these  $z'$  are meant to represent "what  $z$  would have been if the structure of the sentence were  $s'$ ". Using suites of syntactically ambiguous sentences, the authors measured how a model's outputs differed when using  $z'$  generated from different syntactic interpretations.
While Tucker et al. (2021) find that a pretrained BERT model does use representations of syntax causally (i.e., model outputs change when using different syntactic interpretations), the authors find that a BERT model fine-tuned on a question-answering task does not show similar behavior. Identifying causal mechanisms in models is important not only for fairness and robustness measures, but also for improving model performance. In specific cases of subject-verb agreement, Giulianielli et al. (2018) found that changing representations of a subject's plurality affected the plurality of verbs predicted by an LSTM.
In this work, we use the gradient-descent method proposed by Tucker et al. (2021), but we use a new probe design. We identify several cases in which their method fails to uncover a causal relationship, whereas ours does. Furthermore, compared to Giulianielli et al. (2018), we use more general representations of syntax instead of only plurality.
Here, we identify a limitation of prior causal probing art in which redundant information in embeddings could lead to probes and models using different representations of the same information, which in turn could lead to uninformative causal analysis results. We propose a new probe architecture that addresses this limitation by encouraging probes to use all sources of information in embeddings.
1Code at https://github.com/mycal-tucker/mlm_dropout_probes
(a)
"...the keys by the cabinet which are green."
(b)
Figure 2: If a model encodes the dependency structure of a sentence twice its embedding, a probe,  $p$ , may learn to ignore one copy of the information (indicated by learned weight 0) and only use the other (via learned weight 1) to predict  $s$  (a). In such cases, the gradient of  $s$  with respect to the embedding (dashed orange) only flows from one of the copies, so only that copy will be updated in counterfactual embeddings (b). However, by introducing a dropout layer that masks random inputs to the probe, dropout probes learn to use all informative parts of embeddings, which distributes the gradient across the whole embedding (c).
3.1 Limitations from Redundancy
We show by example how prior art in causal probing may fail to reveal causal uses of syntactic information in language models. Here, we use a simplified example; in later experiments we demonstrate that trained models exhibit similar phenomena.
In neural network probing literature, a probe,  $p_{\theta}$ , is a neural network parametrized by weights,  $\theta$ , that maps from representations,  $z$ , to a predicted property,  $\hat{s}$ :  $\hat{s} = p_{\theta}(z)$ . Hewitt and Manning (2019) define two types of structural probes that map from  $z$  to representations of a sentence's syntax. The "depth" probe predicts words' depths in a parse tree; the "distance" probe predicts the distance between pairs of words in a parse tree. In this paper, we assume  $s$  refers to syntactic information, but probing techniques are general. Given a corpus comprising  $(z,s)$  pairs, probes are trained using supervised learning to minimize some loss.
Suppose that there exists a trained model,  $M$ ;  $M_{k-}$  (the first  $k$  layers of  $M$ ) encodes an input,  $x$ , into an embedding  $z$ . The layers of  $M$  after  $k$ , dubbed  $M_{k+}$ , produce a prediction,  $\hat{y}$ , from  $z$ . For the purposes of this example, we state that  $M$  uses syntactic information, and specifically that  $z$  is informative of the syntactic structure of  $x$ .
Let us assume that the dependency structure of  $x$  may be represented by within a vector,  $z_{dep}$ , and that  $M_{k-}$  produces embeddings,  $z$ , which are two identical copies of  $z_{dep}$ . Using pythonic notation,  $z = [z_{dep}] + [z_{dep}]$ . Thus,  $z$  contains syntactic information and, when we state that  $M$  "uses" syntactic information, we formally mean that  $\nabla_{z_{dep}} M_{k+}(z) \neq 0$ .
Building upon this example, let us label the two copies of  $z_{dep}$  as  $z_{dep^1}$  and  $z_{dep^2}$ , although the two vectors remain identical. If we train a probe to predict syntactic forms from  $z$ , it may arbitrarily
learn to use any aspects of  $z$  that are informative of its prediction,  $s$ . Let us say that the probe learns to use only  $z_{dep^2}$ , again defined as  $\nabla_{z_{dep^2}}p(z)\neq 0$ . However,  $M_{k + }$  may only use  $z_{dep^1}$ : the copy that the probe does not use.
We claim that this example, while simplified, demonstrates a potential scenario in which causal probing techniques could return a false negative. Specifically, if one generates counterfactual embeddings,  $z'$ , by changing  $z$  according to the activations that change the probe's outputs, only  $z_{dep^2}$  will change. Because  $M_{k+}$  uses only  $z_{dep^1}$  for predictions, the model's output will not change. This example is depicted in Figure 2. Ultimately, without considering the redundancy in a model's internal representation, prior methods will fail to uncover the fact that  $M$  actually does use representations of syntax causally.
3.2 Dropout Probes
In this section, we propose a neural probe architecture to address the limitations of prior art by encouraging probes to use all syntactic information present in  $z$ . The desired behavior is depicted in Figure 2c: if the probe uses all activations that are informative of syntax, that will necessarily be a superset of the activations that the model uses for downstream processing (if the model uses syntax). Therefore, when generating counterfactual embeddings using such probes, every activation encoding syntactic information would be updated, which in turn would change the model's output.
Our approach was inspired by an idea of creating a mixture of probes, each trained to use a different masked subset of activations in  $z$ . The full set of such probes would have to learn to use all activations in  $z$  that are informative of  $s$ . One may approximate creating such a set by introducing a
dropout layer as the first layer to a single probe. At training time, the dropout layer masks a random subset of the input; the mask itself changes with every training batch. We dub such probes "dropout probes." This probe design, and our resulting findings when using them, are the main contributions of our work. We note that adding a dropout layer to probes introduces a new hyperparameter but, in experiments, we found consistent results over a wide range of positive dropout values.
Here, we report the results from three experiments establishing the benefits of dropout probes. First, we found evidence supporting our hypothesis of redundantly-encoded syntactic information by calculating the mutual information between various activations in trained networks. Second, we compared dropout probes to standard probes in a set of syntactically-ambiguous test domains. We found that our method revealed evidence supporting the causal use of syntax in models where other methods did not (Tucker et al., 2021). Lastly, given our findings that models used syntax causally, we demonstrated how one could "inject" syntactic information into models to improve performance in syntactically-challenging tasks.
Experiments were conducted on four models, all based on huggingface's bert-base-uncased (Wolf et al., 2019). The Mask model was the original model, trained on a masked language modeling task and next-sentence prediction (Devlin et al., 2019). The QA model was fine-tuned on the Stanford Question Answering Dataset 2.0 (Rajpurkar et al., 2016).<sup>2</sup> Lastly, we trained two models, dubbed NLI and NLI-HANS, that were finetuned on the Multi-Genre Natural Language Inference dataset or that dataset augmented with the Heuristic Analysis for NLI Systems (HANS) dataset, respectively (Williams et al., 2018; McCoy et al., 2019).
The Mask model was used to compare our method to Tucker et al. (2021), who found that such models used syntactic information causally. The QA model was used to study a finetuned model; prior art did not find evidence of causal use. Lastly, the NLI models were used because Natural Language Inference is recognized as a difficult linguistic task that models appear to "cheat" on by leveraging spurious correlations in datasets (McCoy et al.,
I(Z1,D)	I(Z2,D)	I(Z,D)
Mask	2.2	2.6	2.7
QA	2.7	2.8	2.8
NLI	2.3	2.7	2.8
Table 1: The mean in nats of  $I(Z, D)$  is less than  $I(Z_{1}, D) + I(Z_{2}, D)$ , indicating that information about  $D$  is redundantly encoded in embeddings. Standard deviation under 0.2 for all values over 5 trials.
2019; Naik et al., 2018; Sanchez et al., 2018).
4.1 Measuring Redundancy in Embeddings
First, we found that language models redundantly encoded syntactic information in their embeddings, which motivated using dropout probes.
We used a technique from prior art, Mutual Information Neural Estimator (MINE), which is a neural-network based approach for estimating the mutual information between two random variables (Belghazi et al., 2018). It does so by computing a lower bound of mutual information and training a neural network to maximize that value. This provides a conservative but tight estimate of mutual information. We refer readers to Appendix A for further details of our implementation.
We defined four random variables of interest. The first,  $D$ , was the depth of each word in a sentence's parse tree; in other words, the labels used to train depth probes in prior literature (Hewitt and Manning, 2019). The second random variable,  $Z$ , was the 768-dimensional embeddings generated by a language model for each token in an input sentence. Lastly, the third and fourth random variables  $(Z_{1}$  and  $Z_{2})$  corresponded to the first and second halves of  $Z$  for each token. That is, these variables comprised the starting and ending 384 units for each token's embedding. By measuring the mutual information between different pairs of these variables, one may formalize our redundancy hypothesis into the following test:  $I(Z,D) < I(Z_{1},D) + I(Z_{2},D)$ . Intuitively, if the test holds, there is shared syntactic information between  $Z_{1}$  and  $Z_{2}$ .
We trained a MINE neural network on the first 5000 examples from the Penn TreeBank to estimate mutual information between random variables (Marcus et al., 1993). Embeddings were taken from the fourth layer of the MASK, QA, and NLI models, although they may be generated elsewhere. Our results are presented in Table 1. For all models,  $I(D,Z) < I(D,Z_1) + I(D,Z_2)$ ; i.e., one gains
2The QA model was downloaded from huggingface model repository under "twmkn9/bert-base-uncased-squad2"
Mask Model Likelihood of Plural Candidates in Coordination Suite Dropout 0 Dropout 0.4
Figure 3: Mean and standard deviation probabilities over 5 trials for plural candidates using the original embeddings (green) or counterfactual embeddings favoring plural (dashed red) or singular (solid blue) parses. Counterfactual embeddings generated by both depth- and distance-based probes caused the greatest shift in model predictions.
little to no information for predicting  $D$  from the full  $Z$  instead of from just  $Z_{1}$  or just  $Z_{2}$ . This is evidence of redundant syntactic information in  $Z$ .
In these experiments using MINE, we demonstrated how  $Z_{1}$  and  $Z_{2}$  could be defined as the subsets of redundant activations depicted in Figure 2. One could define other  $Z_{1}$  and  $Z_{2}$  to better characterize redundancy; here, we merely claim that at least some redundancy is present in the model embeddings.
4.2 Ambiguity Suite Experiments
The prior section established that language models encode syntactic information redundantly; here, we showed that dropout probes overcame the challenges introduced by this redundancy by better aligning with models' true causal usage of syntax. We compared dropout probes to the probes used in prior art via counterfactual experiments inspired by those used by Tucker et al. (2021).
We trained both distance- and depth-based probes, the two types of syntactic probes proposed by Hewitt and Manning (2019). We trained a new probe for each layer of each model, conducting 5 trials with random seeds 0 through 4. All probes were implemented as 3-layer, non-linear neural nets that mapped from model embeddings (of dimension 768) through 2 ReLU layers of dimension 1024, to a final layer to predict a word's depth or distance in the parse tree from other words. Probes were trained for up to 100 epochs, with early stopping based on validation set loss, using the Penn TreeBank dataset (Marcus et al., 1993). We found that this produced more accurate probes than prior art, which capped training at 30 epochs, and that these resulting probes did better than prior reported results, even without using dropout. Each probe was prefixed by a dropout layer with a parameter,
$\alpha$ , that specified the proportion of inputs that were masked before being fed to the probe. By setting  $\alpha = 0$ , we recreated prior art of standard probes. We additionally investigated positive values of  $\alpha$  to measure the benefit of dropout. Counterfactual embeddings were created via gradient descent through trained probes (with dropout disabled), as in prior art (Tucker et al., 2021). That is, new embeddings,  $z'$ , were generated to decrease the loss between  $p(z')$  and a desired parse. We called this loss the counterfactual loss.
In these experiments, we reported two types of results. First, we visualized the effect of interventions, by layer, for a particular dropout rate and counterfactual loss. This revealed that, typically, earlier layers in models were more susceptible to interventions. Second, we devised an aggregate metric for the average difference, across all layers, in model outputs for counterfactuals generated with different parses. This showed how lower counterfactual losses (i.e., more interventions) and higher dropout typically revealed larger effects.
Additionally, we note that the probes were trained to parse single sentences, but two of the models (QA and NLI) accepted two sentences as inputs. For both models, counterfactual embeddings were creating by only updating the syntactically-ambiguous sentence and then concatenating it to the unaltered other embeddings.
4.2.1 Masked Language Model
In testing the Mask model, we largely reproduced patterns in prior results that such models use representations of syntax causally, although we found new results with dropout depth probes. We tested the model with ambiguity test suites inspired by the Coordination and NP/Z suites from Tucker et al. (2021). For example, in the Coordination suite,
Figure 4: For the Coordination (left) and NP/Z (right) suites, interventions to a lower counterfactual loss ( $x$  axis) and with higher-dropout probes (different curves) revealed the greatest causal effects. Means and standard errors.
one sentence reads, "The man saw the girl and the dog [MASK] tall." One may plausibly insert either a plural or singular noun in the masked location, depending upon the syntactic interpretation of the sentence. We generated sentences using a template-based method; details of the prompts (and all prompts in this work) are included in Appendix B.
The results of passing  $z'$  generated from different parses in the Coordination suite through the rest of the Mask model are plotted in Figure 3. The three plotted lines correspond to the model output using the normal embeddings (green), using  $z'$  generated according to a parse favoring plural verbs (red dashed), or using  $z'$  generated using parses implying singular verbs (blue solid). The  $y$  axis corresponds to the probability the model assigned to words implying a plural interpretation ("were," "are," and "as") fitting in the masked location, normalized by the sum of probabilities assigned to those plural words or singular words ("was" and "is"). If the Mask model uses syntactic representations correctly, counterfactuals from plural parses should increase the probability of plural words.
We indeed found that effect, although it is clearest when using dropout probes. The causal effects using standard probes are plotted in the left column; we reproduced the findings from prior art that distance-based probes create the desired effect, but depth-based probes had little to no effect. Conversely, when using dropout probes with  $\alpha = 0.4$  (right column), we found much larger effects.
Averaging across all layers, we also measured the mean difference in output when using counterfactual embeddings generated according to different parses. Intuitively, this generated a single number that captured the average difference between the red and blue lines in the plots in Figure 3.
For a range of dropout values and counterfactual losses, we plotted the mean causal effect for the Coordination and NP/Z suites in Figure 4, using distance probes. For a given counterfactual loss, using higher dropout probes produced larger effects. In addition, lower counterfactual losses (corresponding to more gradient steps) induced greater effects. These trends also held true for depth-based probes (Appendix D). Overall, using the Mask model, we recreated prior art and found new evidence that models also use a depth-based representation of syntax.
4.2.2 QA Model