Predicting Inductive Biases of Pre-Trained Models
Authors: Charles Lovering
Rohan Jha
Tal Linzen
Ellie Pavlick
Venue: ICLR 2021 Poster (2021)
Keywords: information-theoretical probing
probing
challenge sets
natural language processing

Abstract: Most current NLP systems are based on a pre-train-then-fine-tune paradigm, in which a large neural network is first trained in a self-supervised way designed to encourage the network to extract broadly-useful linguistic features, and then fine-tuned for a specific task of interest. Recent work attempts to understand why this recipe works and explain when it fails. Currently, such analyses have produced two sets of apparently-contradictory results. Work that analyzes the representations that result from pre-training (via "probing classifiers") finds evidence that rich features of linguistic structure can be decoded with high accuracy, but work that analyzes model behavior after fine-tuning (via "challenge sets") indicates that decisions are often not based on such structure but rather on spurious heuristics specific to the training set. In this work, we test the hypothesis that the extent to which a feature influences a model's decisions can be predicted using a combination of two factors: The feature's "extractability" after pre-training (measured using information-theoretic probing techniques), and the "evidence" available during fine-tuning (defined as the feature's co-occurrence rate with the label). In experiments with both synthetic and natural language data, we find strong evidence (statistically significant correlations) supporting this hypothesis.

Published as a conference paper at ICLR 2021
PREDICTING INDUCTIVE BIASES OF TRE-TRAINED MODELS
Charles Lovering', Rohan Jha!, Tal Linzen', Ellie Pavlick"
Brown University
Department of Computer Science
(cherles.loverinnge rohan.jha@alume
ellie.pavlien brown. edu
8 New York University
Department of Linguistics and Center for Data Science
Linzen@nyu.edi
ABSTRACT
Most current NLP systems are based on a which large neural network is first trained in self-supervised way designed encourage the network to extract broadly-useful linguistic features, and then fine- tuned for specific task of interest. Recent work attempts to understand why this recipe works and explain when it fails. Currently. such analyses have produced two sets of opparenly--oontaadictor results. Work that analyzes the representa- tions that result from pre-training (via "probing classifiers") finds evidence that rich features linguistic structure can be decoded with high accuracy, but work that analyzes model behavior after fine-tuning (via "challenge sets") indicates that decisions are often not based on such structure but rather on spurious heuristics specific to the training set. In this work, test the hypothesis that the extent which feature influences a model's decisions can be predicted using a combina- tion of two factors: The feature's exfructability after pre-training (measured using tefeooaiioo-theooetiic probing techniques), and the evidence available during fine- tuning (defined as the feature's 's co-occurrence rate with the label). In experiments with both synthetic and naturalistic data, we find strong evidence (statistically sig- nificant correlations) supporting this hypothesis.
paradigm, in
INTRODUCTION
Large pre-trained language models (LMs) (Devlin et al.) 2019 Raffel et al. 2020 Brown et al. 2020) have demonstrated impressive empirical success range of benchmark NLP tasks. How- ever, analyses have shown that such models are easily fooled when tested on distributions that differ from those they were trained on, suggesting they often "right for the wrong reasons" (McCoy jet al. 2019). Recent research which attempts to understand why such models behave in this way has primarily made use of analysis techniques: probing classifiers (Adi et al., [017]] Hupkes) ct al. 2018). which measure whether or not a given feature is encoded by representation, and chal- lenge sety Cooper al. 1996. Linzen al. 2016 Rudinger al. 2018). which measure whether model behavior in practice is consistent with use of a given feature. The results obtained via these two techniques currently suggest different conclusions about how well pre-trained representations encode language. Work based on probing classifiers has consistently found evidence that models contain rich information about syntactic structure (Hewitt& & Manning. 2019 Bau al. 2019 Ten- ney et al.) 2019a), while work using challenge sets has frequently revealed that models built top of these representations do not behave as though they have access to such rich features, rather they fail in trivial ways (Dasgupta et al. 2018 Glockner al. 2018 Naik et al. 2018).
In this work, we attempt to link these two contrasting views of feature representations. We assume the standard recipe in NLP, in which linguistic representations first derived from large-scale self- supervised pre-training intended to encode broadly-usetu linguistic features, and then are adapted for task of interest via transfer learning. or fine-tuning, on task-specific dataset. We test the

---

Published as a conference paper at ICLR 2021
hypothesis that the extent to which fine-tuned model uses given feature can be explained function of two metrics: The extractability of the feature after pre-training (as measured by probing classifiers) and the evidence available during fine-tuning (defined as the rate of co-occurrence with the label). We first show results on a synthetic task, and second using state-of-the-art pre-trained be viewed
LMs on language data. Our results suggest that probing classifiers the pre-trained representation's 's inductive biases: The more extractable a feature is after pre-training, the less statistical evidence is required in order for the model to adopt the feature during fine-tuning.
measure of
Contribution. This work establishes a relationship between two widely-used techniques for ana- lyzing LMs. Currently, the question of how models' internal representations (measured by probing classifiers) influence model behavior (measured by challenge sets) remains open (Belinkov & Glass) 2019 Belinkov et al. 2020). Understanding the connection between these two measurement tech- niques can enable more principled evaluation of and control over neural NLP models.
SETUP AND TERMINOLOGY
2.1 FORMULATION
Our motivation comes from McCoy et al. (2019), which demonstrated that, when fine-tuned natural language inference task (Williams et al., 2018 MNLI), a model based on a state-of-the-ar pre-trained LM (Devlin et al. 2019 BERT) categorically fails test examples which defy the expectation of a "Texical overlap heuristic". For example, the model assumes that the sentence "the lawyer followed the judge" entails "the judge followed the lawyer" purely because all the words in the latter appear in the former. While this heuristic is statistically favorable given the model's training data, not infallible. Specifically, McCoy et al. (2019) report that 90% of the training examples containing lexical overlap had the label "entailment" , but the remaining 10% did not. Moreover, the results of recent studies based on probing classifiers suggest that more robust features are extractable with high reliability from BERT representations. For example, given the example "the lawyer followed the judge"|" judge followed the lawyer", if the model can represent that "lawyer" is the agent of "follow" in the first sentence, but the patient in the second, then the model should conclude that the sentences have different meanings. Such semantic role information can be recovered at > 90% accuracy from BERT embeddings (Tenney al. 2019b). Thus, the question is: Why would a model prefer a weak feature over a stronger one, if both features are extractable from the model's representations and justified by the model's ''s training data?
bbttracting over details, we distill the basic NLP task setting described above into the following, to be formalized in the Section 2.2 We assume binary sequence classification task where larget feature f perfectly predicts the label (c.g., the label is 1 iff t holds). Here, represents fcatures which actually determine the label by definition, e.g., whether one sentence semantically entails another. Additionally, there exists spurious feature that frequently co-occurs with in training but is not guaranteed to generalize outside of the training set. Here, (often called a "heuristic" or "bias" elsewhere in the literature) corresponds to features like lexical overlap, which are predictive of the label in some datasets but are not guaranteed to generalize.
Assumptions. In this work, we assume there is a single and single s; in practice there may be many s features. Still, our definition of feature accommodates multiple spurious or target features. In fact, some of our spurious features already encompass multiple features: the lexical feature, for example, is combination of several individual-wort features because it holds if one of a set of words is in the sentence. This type of spurious feature is common real datasets: E.g., the hypothesisonnly baseline in NLI is a disjunction of lexical features (with semantically unrelated words like "no", "sleeping", etc.) (Poliak al., 2018b Gururangan et al. 2018).
We assume that s and & frequently co-occur, but that only occurs in isolation. This assumption reflects realistic NLP task settings since datasets always contain some heuristics, e.g., lexical cues, cultural biases, artifacts from crowdsourcing (Gururangan et al. 2018). Thus, our experiments focus on manipulating the occurrence of s alone, but not alone: This means giving the model evidence against relying on s. This is in line with prior applied work that attempts to influence model behavior by increasing the evidence against s during training (Elkahky et al. 2018 Zmigrod et al. 2019 Min et al. 2020).
2

---

Published as a conference paper at ICLR 2021
2.2 DEFINITIONS
Let X' be the set of all sentences and S be the space of all sentence-label pairs (2,1) v) E X x {0,1); We use D S to denote a particular training sample drawn from S. We define two types of binary features: larget (1) and spurious (x). Each is a function from sentences x E X' to a binary label {0, 13 that indicates whether the feature holds.
Target and spurious features. The target feature is such that there exists some function f (0, + {0, 13 such that V(z,z) v) S. f((x)) y. In other words, the label can always be perfectly predicted given the value of t A feature is spurious if it is not target feature.
Partitions of S. To facilitate analysis, we partition S in four regions (Figure Tp. We define S--only to be the set of examples in which the spurious feature occurs alone (without the target). Similarly, St is the set of examples in which the target occurs without the spurious feature. Shoth and Swither are analogous. For clarity, we sometimes drop the S. notation (e.g., s-only in place of Sa-coly).
Sooth = {(1. [(x) 1 s(x) 1
S
s-only x x
Smeither = {xx,y) [(x) A s(x) 0}
f-only
both
x
St-nly = {zz,)) [(()) = 1A s()) 0}
x
Ss-only = (1x,)) ((x)) = 0 <()) 1} neither
x
Figure I: We partition datasets into four sections, defined by the features (spurious and/or target) that hold. We sample training datasets D, which provide varying amounts of evidence against the spurious feature, in the form of s-only examples. In the illustration above, the s-only rate is 2 TO 0.2, i.e., 20% of examples in D provide evidence that s alone should not be used to predict y.
Evidence from Spurious-Only Examples. We are interested in spurious features which are highly correlated with the target during training. Given a training sample D and features and t, we define the s-only example rate as the evidence against the use of as a predictor of y. Concretely, s-only rate- a DD) the proportion of training examples in which s occurs without t (and y = 0).
Use of Spurious Feature. If a model has falsely learned that the spurious feature alone is pre- dictive of the label, will have a high error rate when classifying examples for which holds but t does not. We define the s-only error to be the classifier's s eттоr on examples from Sa-only. When relevant, t-only error. both error, and neither error are defined analogously. In this work, "feature use" is model's predictions consistency with that feature; we are not making causal argument.
Extractability of a Feature. We want to compare features in terms of how extractable they are given representation. For example, given a sentence embedding, it may be possible to predict multiple features with high accuracy, e.g., whether the word "dog" occurs, and also whether the word "dog" occurs as the subject of the verb "run". ". However, detecting the former will no doubt be an easier task than detecting the latter. We use the prequential minimum description length (MDL) Rissanen (977) -first used by Voita & Titov (2020) for probing-to quantify this intuitive difference MDL is an information- -theorreicc metric that measures how accurately a feature can be decoded and the amount of effort required to decode it. Formally, MDL measures the number of bits required to communicate the labels given the representations. Conceptually, MDL can be understood measure of the area under the loss curve: If feature is highly extractable, model trained to detect that feature will converge quickly to high accuracy, resulting in low MDL. Computing MDL. requires repeatedly training a model over a dataset labeled by the feature in question. To compute MDL(s), we train a classifier (without freezing any parameters) to differentiate Sa.unty vs. Speithrrs
and similarly compute MDL(t). See Voita & Titov (2020) for additional details on MDL
Without loss of generality, we define in our datasets s.t. f(x) = y. Vz, E S. We do this to iron out the
case where t outputs the opposite value of y.
We observe similar overall trends when using alternative metric based validation loss (Appendix
Note that our reported MDL is higher in some cases than that given by the uniform code (the number of
sentences that being encoded). The MDL is computed sum of the costs of transmitting successively
3

---

Published as a conference paper at ICLR 2021
2.3 HYPOTHESIS
Stated using the above-defined terminology, our hypothesis is that a model's ss use of the target feature modulated by two factors: The relative extractability of the target feature (compared to the spu- rious feature s). and the evidence from s-only examples provided by the training data. In particular, we expect that higher extractability of (relative to s), measured by MDL(s)/MDDL.) will yield models that achieve better performance despite less training evidence.
EXPERIMENTS WITH SYNTHETIC DATA
Since it is often difficult to fully decouple the target feature from competing spurious features in practice, we first use synthetic data in order to test our hypothesis in clean setting. We use simple classifier with an embedding layer, I-layer LSTM, and an MLP with hidden layer with tanh activation. We use synthetic sentence classification task with k-length sequences of numbers as input and binary labels as output. We use a symbolic vocabulary with the integers 0... V 1. We fix k 10 and [V] 50K. We begin with initial training set of 200K, evenly split between examples from Sooth and Swetthrr Then, varied across runs, we manipulate the evidence against the spurious feature (i.e., the s-only rate) by replacing percentage p of the initial data with examples from S--only for {0%, 0.1%, 1%, 5%, 10%, 20%, 50%). Test and validation sets consist of 1,000 examples each from Shoth Steither, St.only. S.-only. In all experiments, be the presence of the symbol 2. We consider several different target features (Table TT, intended vary their extractability. Table containss MDL metries for each feature (computed on training sets of 200K, averaged over 3 random seeds). We see some gradation of feature extractability, but having more features with wider variation would help solidify our results
set the spurious feature to
Target Feature
Description
MDL(s)
MDL(()
Rel. MDL
Example
contains-
occurs sequence
0.36
0.29
1.259
prefix-dupll Sequence begins with duplicate
0.42
175.74
0.002
5
11
12
2
adj-dupl
Adjacent duplicate in seq.
0.37
242.20
0.001
11
12
3
first-last
First number equals last number
0.37
397.64
0.001
11.
12
Table 1: Instantiations of the target feature in our synthetic experiments. The spurious feature is always the presence of the symbol 2. Features are intended to differ in how hard they are for an LSTM to detect given sequential input (measured by MDL per (2.2 reported in k-bits).
Figure showws model performance as a function of s-only rate for each of the four features described above. Here, performance is reported using error rate (lower is better) each partition (S (S..only- St.only. Shats Sneither) separately. We are primarily interested in whether the relative extractability of the target feature (compared to the spurious feature) predicts model performance. We indeed see a fairly clear relationship between the relative extractability mance, at every level of training evidence (s-only rate). For example, when é is no less extractable than s (i.e., contains ns--), the model achieves zero error at an s-only rate of 0.001, meaning learns that t alone predicts the label despite having only handful of examples that support this inference. In contrast, when & is harder to extract than s (e.g., first-ll -aast), the model fails to make this inference, even when large portion of training examples provide evidence supporting it.
(M))L()))(( and model perfor-
4 EXPERIMENTS WITH NATURALISTIE DATA
We investigate whether the same trend holds for language models fine-tuned with naturalistic data, e.g., grammar-generated English sentences. To do this, we fine-tune models for the linguistic ceptability task, a simple sequence classification task as defined in Warstadt & Bowman 2019))
longer blocks, using classifiers that are trained previously transmitted data. The high MDL's result of overfitting by classifiers that are trained on limited data-and therefore, the classifiers have worse compression performance than the uniform baseline.
'Note, all models are ultimately able to learn to detect f (achieve high test accuracy) on the both partition, but the f-only partition.
4

---

We fo- Published as a conference paper at ICLR 2021
Spurious (s-only) Error
Target (t-only) Error
Neither Error
Both Error
1.00
0.75
0.50
0.25
0.00
0.001
0.0
DHNN 0
0.001
0oo
0,001 001
Goo
0.001
Evidence from
Evidence from
Evidence from
Evidence from
Spurious-only Examples (s-only example rate)
Spurious-only Examples (s-only example rate)
Spurious-only Examples (s-only example rate)
Spurious-onij Examples (s-only example rate)
Rel. Extractibility
0.001 first-last
0.001 adj-dupl
0.002 prefix-dupl
1.295 contains-1
Figure 2: Results Synthetic Data. Error each partition of the test set, function of s-only rate. model that has learned to use the target feature alone to predict the label will achieve zero across all partitions. s-only and f-only error reach 0 quickly when is easy extract (i.e., the relative extractability is 1). However, when is harder to extract than (rel. extractability performance lags until evidence from s-only examples is quite strong.
in which the goal is differentiate grammatical sentences from ungrammatical ones. We focus on acceptability judgments since formal linguistic theory guides how we define the target features, and recent work in computational linguistics shows that neural language models can sensitive to spurious features in this task (Marvin & Linzen 2018 Warstadt et al. 2020a).
4.1 DATA
We design a series of simple natural language grammars that generate a variety feature pairs (s, t), which we expect will exhibit different levels of relative extractability (MDL(s)/MDLL(()) cus on three syntactic phenomena (described below). In each case, we consider the target feature t to be whether given instance of the phenomenon obeys the expected syntactic rules. We then introduce several spurious features which we deliberately correlate with the positive label during fine-tuning. The Subject- Verb Agreement (SVA) construction requires detecting whether the verb agrees in number with its subject, e.g., "the girls playing" " is acceptable while "the girls is play- ing" is not. In general, recognizing agreement requires some representation of hierarchical syntax, since subjects may be separated from their verbs by arbitrarily long clauses. We introduce four spu- rious features: 1) lexical, grammatical sentences begin with specific lexical items (e.g., "often "); 2) length, grammatical sentences are longer; recent-noun, verbs grammatical sentences agree with the immediately preceding noun (in addition to their subject); and 4) plural, verbs in grammatical sentences are preceded by singular nouns as opposed to plural ones.
The Negative Polarity Items (NPI) construction requires detecting whether negative polarity item (e.g., "any", "evve") grammatical in a given context, e.g., "no girl ever played" acceptable while "a girl ever played" is not. In general, NPIs are only licensed in contexts that fall within the scope of downward entailing operator (such as negation). We again consider four types spurious features: 1) lexical, in which grammatical sentences always include
of set of lexical items ("no" " and "not") :); 2) length (as above); 3) plural, in which each noun in a grammatical sentence is singular, opposed plural: and tense, in which grammatical sentences are in present tense.
Some verbs (e.g. "recognize" ") require direct object. However, in the right syntactic contexts (i.e., when in the correct syntactic relation with wh-word), the object position can be empty, creating what known as a "gap". E.g., "I know what you recognized " acceptable while "I know that you recognized is not. The Filler-Gap Dependencies (GAP) construction requires detecting
5

---

Published as a conference paper at ICLR 2021
whether sentence containing a gap is grammatical. For our GAP tasks, we again consider four spurious features (lexical, length, plural, and tense), defined similarly to above.
Target
Spurious
Example
Subject agrees
N before V
[botk] The piano teachers of the lawyer wound the handyman.
with verb
is singular
[s-only] *The piano teachers of the lawyer wounds the handyman.
NPT in down. -entailing context
Contains
[both]
No
student who was wrong ever resigned.
negation word
[s-only] *The student who was not wrong ever resigned.
Correct filler-gap dependency
Main verb
[both]
knew
what recognized yesterday.
in past lense
[s-only] knew
what he recognized someone yesterday.
Table 2: Examples of features used to generate fine-tuning sets with target/spurious features of varying extractability scores. Top examples show a case which and s both occur and the sentence is acceptable, and bottom examples show a case in which occurs without and the sentence is unacceptable. Only s highlighted since is often defined over the structure of the sentence (see text) and thus difficult to localize to few tokens. Table 9 in the Appendix has neither examples.
The templates above (and slight variants) result in 20 distinct fine-tuning datasets, over which we perform our analyses (see Appendix for details). Table shows several examples. For the purposes of this paper, we are interested only in the relative extractability of vs. 8 given the pre-trained representation; we don't 't intend to make general claims about the linguistic phenomena per se. Thus, we do not focus on the details of the features themselves, but rather consider each template as generating one data point, i.e., an (s, () pair representing particular level of relative extractability.
4.2
SETUP
We evaluate T5, BERT. RoBERTa, GPT-2 and an LSTM with GloVe embeddings (Raffel et al, 2020 Devlin et al. 2019 Liu et al. 2019;; Radford et al. 2019 Pennington et al., 2014 hp Both T5 and BERT learn to perform well over the whole test set, whereas the GloVe model struggles with many of the tasks. We expect that this is because contextualized pre-training encodes certain syntactic features which let the models better leverage small training sets (Warstadt & Bowman begin with an initial training set of 2000 examples, evenly split between both and neither, and then introduce s-only examples at rates of 0%. 0.1%, 1%. 5%, 10%, 20%, and 50%, using three random seeds each. Test and validation sets consist of 1000 examples each from Sboth Sweiherr S anyy In the natural language setting, it is often difficult to generate t-only examples, and thus we cannot compute extractability of the target feature t by training classifier to distinguish St-onty from random subset of Speither, as we did in Section 3 Therefore, we estimate MDL by training a classifier to distinguish between examples from Sa-only and examples from Shat. Using the simulated data from Section 3 we confirm that both methods (Sa-anly vs. Shoth and Se vs. Sucither) produce similar estimates of MDL(t)) (see Appendix). Per model, we filter out feature pairs for which the model could not achieve at least 90% accuracy on each probing task in isolation
2020). Again,
4.3 RESULTS
For each (s, t) feature pair, we plot the use of the spurious feature (s-only error) as a function of the evidence against the spurious feature seen in training (s- (s-only example rate) We expect to the same trend we observed in our synthetic data, i.e., the more extractable the target feature / is relative to the spurious feature s, the less evidence the model will require before preferring over s. To quantify this trend, we compute correlations between 1) the relative extractability of f. compared to s and 2) the test F-score averaged across all rates and partitions of the data capturing how readily the model uses (i.e., makes predictions consistent with the use of) the target feature.
In pilot studics, we found that standard BOW and CNN-based models were unable solve the tasks. "This control does not impact results: Appendix A.1
7 "See Appendix both error and neither error; both are stable and low in general.
Initiallyy, we used more complicated metrie based on the example rate required for the model to solve the test Both report similar trends and correlations. For posterity, we include details in the Appendix A.2

---

Published as a conference paper at ICLR 2021
Absolute
Relative (1 to 8)
Target
Spurious
Ratio
Difference
BERT RoBERTa
-0.72*
0.65*
0.79*
0.96*
-0.26
0.8*
0.83*
0.81*
T5
-0.91
0.04
0.57*
0.73*
GPT2
-0.56*
0.55*
0,73*
0.78*
GloVe
0.12
0.44
0.14
0.10
(a) Spearman's 's p: MDL vs. Average Test F-score
BERT
RoBERTa
T5
GPT-2
GloVe
Average F-Score
0.8
0.7
0-979
79
p -3
57
--. 73
14
0.01
1.00
2.00
0.01
1.00
2.00
0.01
1.00
2.00
1.00
2.00
1.00
2.00
Relative extractibility of target feature (MDL(S)/MDL(())
(b) Logistic regression plots of Average F-score vs. relative extractability s, via (MDL(s)) (MDL()))MD((()) (D).
Figure 3: Relative Extractability Correlates with Target Feature Use. In (a) we show the Spear- man's y between the test F-Score vs measures of extractability of the (s, () pairs; indicates signifi- cance. Relative extractability, whether ratio (MDL(s)/MDL((U difference (MDL(s)-MDL(())) explains learning behavior better than absolute extractability of either feature.
Figure shows these correlations and associated scatter plots. We can see that relative extractability is strongly correlated with average test F-score (Figure Bap, showing high correlations for both BERT (p 0.79) and T5 (p 0.57). That is, the more extractable the model requires before preferring t, performing better across all partitions. This relationship holds regardless of whether relative extractability is computed using a ratio of MDL scores or an absolute difference. We also see that, in most cases, the relative extractability explains the model's behavior better than does the extractability
relative to 8, the less evidence
s or t alone. For GloVe there is little variation in model behavior: For most of the 1 1/20 pairs which the model is able to learn the task, it requires an s-only example rate of 0.5. Thus, the correlations are weak, but qualitative results appear steady (Figure in Appendix Ab, following the pattern that when is easier extract than i,, more evidence is required to stop using
Figure shows the performance curves for BERT and T5 (with others the in Appendix A). i.e., use of the spurious feature (s-only error) as function of the evidence from s-only examples seen in different 8, feature pair, and each data
training (s-only example rate). Each line corresponds point is the test performance on a dataset with a given s-only example rate (which varies along the x-axis.) For pairs with high MDL ratios (i.e., when t actually easier to extract than s), the model learns to solve the task "the right way" even when the training data provides no incentive to do so: That is, in such cases, the models' decisions do not appear to depend on the spurious feature s even when s and the target feature t perfectly co-occur in the fine-tuning data.
Figure shows that T5 (compared to BERT) requires more data to perform well. This may be because we fine-tuned T5 with linear classification head, rather than the text-only output on which was pre-trained, We made this decision 1) because we had trouble training T5 in the original manner, and 2) using a linear classification head was consistent with the other model architectures.
DISCUSSION
Our experimental results provide support for our hypothesis: The relative extractability of features given input representation (as measured by information-theoretie probing tech-
7

---

Published as a conference paper at ICLR 2021
bert
Use of Spurious Feature (s-only erron)
1.00
5. feature pairs
0.75
0.50
0.25
0.00
0.001
0.01
0.05 0.1
0.2
0.5
Relative Extractability of Target Feature (MDLis)/MDLIt)
Evidence from Spurious-only Examples (s-only example rate)
t5
Use of Spurious Feature
1.00
5, f.atuue pairs
(s-only error)
0.75
0.50
0.25
0.00
0.001
0.01
0.05 0.1 0.2 0.5
Relative Extractability Target Feature (MDL(S)/DL((t)
Evidence from Spurious-only Examples (s-only example rate)
Figure 4: Learning Curves for BERT & T5. Curves show use of spurious feature (s-only accuracy) as a function of training evidence (s-only rate). Each line represents one (s, pair (described in (4.1p. Pairs vary in the relative extractability of t vs. (measured by the ratio (sLLL))MDL(() and summarized in the bar chart). When much harder to extract relative to s (lower ratios), the classifier requires much more statistical evidence during training (higher s-only rate) in order to achieve low error. find similar patterns GPT2 and RoBERTa; see Appendix A for all results.
niques) is predictive of the decisions trained model will make in practice. In particular, we see evidence that models will tend use imperfect features that are more readily extractable over perfectly predictive features that are harder to extract. This insight is highly related to prior work which has shown, e.g., that neural networks learn "easy" examples before they learn "hard" exam- ples (Mangalam & Prabhu 2019). Our findings additionally connect to new probing techniques which have received significant attention in NLP but have yet to connected to explanations or predictions about state-of-the-art art models' decisions in practice.
Fine-tuning may not uncover new features. The models are capable of learning both the s and experiments show that if the relative extractibility is highly skewed,
t features in isolation, one feature may hide the other a fine-tuned model may not use the harder-to-extract feature. This suggests pattern that seems intuitive but is in fact non-trivial: If one classifier does not pick up on a feature readily enough, another classifier (or, rather, the same classifier trained with different data) may not be sensitive that feature all. This has ramifications for how view fine-tuning, which is generally considered to be beneficial because it allows models to learn new, task-relevant features. Our findings suggest that if the needed feature is already extractable-enough after pretraining. fine-tuning may not have the desired effect.
Probing classifiers can be viewed as measures of pre-trained representation's inductive bi- ases. Analysis with probing classifiers has primarily focused on whether important linguistic fea- tures can be decoded from representations at seater-than-baseliin rates, but there has been little insight about what it would mean for a representations' encoding of a feature to be "sufficient". Based on these experiments, we argue that a feature is "sufficienll"" encoded it is as available to the model are surface features of the text. For example, if fine-tuned model can access features about a word's semantic role as easily as it can access features about that word's lexical identity, the model may need little (or no) explicit training signal to prefer decision rule based on the former structural feature. The desire for models with such behavior motivates the development of architec- tures with explicit inductive biases (e.g., TreeRNNs). Evidence that similar generalization behavior
8

---

Published as a conference paper at ICLR 2021
can result from pre-trained representations has exciting implications for those interested in sample
efficiency and bognitvelly-plasibbe language learning (Warstadt & Bowman 2020 Linzen 2020).
We note that this work has not established that the relationship between extractability and feature use is causal. This could he explored using intermediate task training (Pruksachatkun et al. (020) in
order to influence the extractability of features prior to fine-tuning for the target task; e.g.. Merchant et al. (2020) suggests fine-tuning on parsing might improve the extractability of syntactic features.
6 RELATED WORK
Significant prior work analyzes the representations and behavior of pre-trained LMs. Work using probing classifiers (Veldhoen al.) 2016 Adi et al., 2017 Conneau et al. 2018 Hupkes al. 2018)) suggests that such models capture a wide range of relevant linguistic phenomena (Hewitt & Man-
ning 2019 Bau et al. 2019 Dalvi al., 2019; Tenney et al. 2019a b). Similar techniques include
attention maps/visualization (Voita et al. 2019; Serrano & Smith 2019), and relational similarity analyses (Chrupata & Alishahi 2019). A parallel line of work uses challenge sets to understand model behavior in practice. Some works construct evaluation sets to analyze weaknesses in the decision procedures of neural NLP models (Jia & Liang 20176] Gloockner al. 2018] Dasgupta
iet al. [0188] Gururangan et al.] 2018 Poliak et al. | [20 Elkahky et al. 2018 Ettinger et al. 2016
Linzen al. [2016] [sabelle al., 2017 Naik et 2018| Jia Liang 2017a Linzen et 2016
Goldberg, [019 and others). Others use such datasets to improve models' handling of linguistic
features (Min et al., 2020. Poliak et al.) 2018a| Liu et al., 2019a), or to mitigate biases (Zmigrod
et al.
et al.] 2019 Hall Maudslay et al. 2019: Lu et al. 2020). Nie et al. (2020)
and Kaushik et al. (2020) explore augmenting training sets with numan-in-the-loo, methods.
Our work is related to work on generalization of neural NLP models. Geiger et al. (2019) dis- cusses ways in which evaluation tasks should be sensitive to models' inductive biases and Warstadt & BBowman 2020) discusses the ability of language model pre-training to encode such inductive biases. Work on data augmentation (Elkahky et al. 2018, Min et al., 2020 Zmigrod et al. 2019) is relevant, the approach relies the assumption that altering the training data distribution (analo- gous to what we call s-only rate in our work) will improve model behavior in practice. [Kodner &
Gupta (2020); [haa al. (2020) discuss concerns about ways in which such approaches can be coun-
terproductive, by introducing new artifacts. Work on adversarial robustness (Ribeiro et al. 2018
lyyer et al. 2018 HHiih et al. 2019; Jia al. 2019 Alzantot al. 2018) Hsieh et al.) 2019 Iyyas et al. [019 Madry et al. 2017; Athalye et al., 2018 is also relevant, as it relates to the influence of dataset artifacts on models' decisions. still larger body of work studies feature representation and generalization in neural networks outside of NLP. Ma.galam & Prabhu (2019) show that neural net- works learn "eaay" examples (as defined by shallow machine learning model performance) before they learn "hard" examples. Zhhng et al. (2016) and Arpit al. (2017) show that neural networks which are capable of memorizing noise nonetheless acheive good generalization performance, sug- gesting that such models might have an inherent preference to learn more general features. Finally, ongoing theoretical work characterizes the ability of over- wver-parameterized networks to generalize in terms of complexity (Neyshabur et al., 2019) and implicit regularization (Blanc al. 2020).
Concurrent work (Warstadt et al., 2020b) also investigates the inductive biases of large pre-trained models (RoBERTa), in particular, they ask when (at what amount of pre-training data) such models shift from a surface feature (what we call spurious features) to a linguistic feature (what we call a target feature). In our work, we focus on how to predict which of these two biases characterize the model (via relative MDL).
7 CONCLUSION
This work bears on an open question in NLP, namely, the question of how models' internal repre- sentations (as measured by probing classifiers) influence model behavior (as measured by challenge sets). We find that the feature extractability can be viewed as an inductive bias: the more extractable a feature is after pre-training the less statistical evidence is required in order for the model to adopt the feature during fine-tuning. Understanding the connection between these two measurement tech- niques can enable more principled evaluation of and control over neural NI. models.
9

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGEMENTS
We would like to thank Michael Littman for helpful suggestions on how to better present our findings and Ian Tenney for insightful comments on previous draft of this work. We also want to thank our reviewers were their detailed and helpful comments. This work is supported by DARPA under grant number HR001 1990064. This research was conducted using computational resources and services at the Center for Computation and Visualization, Brown University.
REFERENCES
Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained Anal- ysis of Sentence Embeddings Using Auxiliary Prediction Tasks. In International Conference on Learning Represeniations (ICLR), 2017.
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. Generating natural language adversarial examples. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pp. 2890-2896, Brussels. Bel- gium, Oetober-Novemble 2018. Association for Computational Linguisties. doi: 10. 1865/v1v D18-1316. (t..ttt:::: www. ac aclweb.
1121181115 1316
Devansh Arpit, Stanislaw Jastrzkebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxin- der S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A closer look at memorization in deep networks. In Proceedings of the 34th International Conference on Machine Learming Volume 70, ICML 17, pp. 233-242, JMLR.org, 2017. URL http::ttp//ll...... htt.//lacc.mmmm /ttp:/aa..oo..cmmmdd333..38...8 3305406
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of se- curity: Circumventing defenses to adversarial examples. In International Conference on Machine Learning, pp. 274-283. PMIR, 2018
Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi, and James Glass. Identifying and controlling important neurons in neural machine translation. In International Conference on Learning Representations, 2019. URLhttps::/ /trtps:://peeneeveew..ro htw.e/wwweee.ooooeeeeeeeeee forum?
id Hd-H1z-PsR5XX
Yonatan Belinkov and James Glass. Analysis Methods in Neural Language Processing: Survey. Transactions of the Association for Computational Linguistics, 7:49-72. March 2019, doi: 10.
1162/tacl 00254. URL. httthtts:://wwww http://wwwwwww..b..ooo.o.o...o
ac web. wwt.://www............ooooo/////
Yonatan Belinkov, Sebastian Gehrmann, and Ellie Pavlick. Interpretability and analysis in neural NLP. In Proceedings of the 58th Annual Meeting of the Association for Computational Lingnis- fies: Tutorial Abstracts, pp. 1-5, Online, July 2020. Association for Computational Linguistics. ac ubb.bpp:///www....bbbbb.oooo hww.://wwww...r/aatoohoooooo
doi: 10. 1106653//11200000.clltttttdall
Guy Blane, Neha Gupta, Gregory Valiant, and Paul Valiant. Implicit regularization for deep neural networks driven by an 483-513. PMLR, 2020.
like process. In Conference on learning theory, pp-
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL ht 1457c0d6b cbw. h11e4418f08cc11100
neurips.......n....oe.....iii.s. http:///w.n.naoccc////////////// pd
10

---

Published as a conference paper at ICLR 2021
Grzegorz Chrupata and Afra Alishahi. Correlating neural and symbolic representations of language. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 2952-2962, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/ v1/P19- 1283.
URI. https:::p........... www. ac1 web. www.rggaaanoooggggegoooooo 9-1833
9-
Alexis Conneau, German Kruszewski, Guillaume Lample, Loic Barrault, and Marco Baroni. What you can cram into a single S&!#* vector: Probing sentence embeddings for linguistic proper- ties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguis- tics (Volume 1: Long Papers). pp. 2126-2136 Melbourne, Australia, July 2018. Association for Computational
Linguutttt Linguistics. doi: 10.18653/v 1/P18- 19--1198. URLhttps: httpp::://wwwww. http://wwwwwwwwbbb../o/.. aww.eb. org/
anthology/FIB-11E
Robin Cooper, Dick Crouch, Jan van Eijck, Chris Fox, Josef van Genabith, Jan Jaspars, Hans Kamp, David Milward, Manfred Pinkal, Massimo Poesio, Steve Pulman, Ted Briscoe, Holger Maier, and Karsten Konrad. Using the framework. Technical report, The FraCaS Consortium, 1996.
Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, D. Anthony Bau, and James Glass. What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models. In Proceedings of the Third AAAI Conference on Artificial Intelligence (AAAI), January 2019.
Ishita Dasgupta, Demi Guo, Andreas Stuhlmüller, Samuel J Gershman, and Noah D Goodman. Evaluating compositionality in sentence embeddings. arXiv preprint 18X2v:800..04300 2018.
Jacob Devlin, Ming-Wa Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 417 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-144 1423. https::: 7/ //wwwwwc www.wb..wb..... wwtp:aawwww............a........
org wwzpg/anthoooyy/Mls--a
Ali Elkahky, Kellie Webster, Daniel Andor, and Emily Pitler. A challenge set and methods for
noun-verb ambiguity. In Proceedings of the 2018 Conference on Empirical Methods in Natural
Language Processing, pp. 2562-2572, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.11655/11/D18-1277, URL htttt:t///// www. web.
Allyson Ettinger, Ahmed Elgohary. and Philip Resnik. Probing for semantic evidence of compo- sition by means of simple classification tasks. In Proceedings of the 1st Workshop on Evaluat- ing Vector-Space Representations for NLP. pp. 134-139. Association for Computational Linguis- tics, 2016. doi: 10.18653/1//116-2522 URL http: /ttp:://www... ac www.wwggbb..g..ooooo/oooooo W1 6-524
WA Falcon. Pytorch lightning. GütHub. Note: http://sihaaaccoom/////oeelllggn lightnngg 3. 2019.
Atticus Geiger, Ignacio Cases, Lauri Karttunen, and Christopher Potts. Posing fair generaliza- tion tasks for natural language inference. In Proceedings of the 2019 Conference Empir- ical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNI.P-JCCIPP) pp. 4484 4494, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-110 1456. URL
ht aittps:/ http://wwwwccccc www.aaclweb. www.lwewb.rrgeee........ www.eeborrr/aatooogo///...
1456
Max Glockner, Vered Shwartz, and Yoav Goldberg. Breaking NLII Systems with Sentences that Require Simple Lexical Inferences. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 650-655, Association for Computa- tional Linguistics, 2018. URL http:://t///alllee wwweebborrg/aaae hwt.ee/....gg.aaaooogg//ee-----.
Yoav Goldberg. Assessing bert's syntactic abilities. arXiv preprint arXiv:1001.05287 2019.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. Annotation artifacts in natural language inference data. In Proceedings of
11

---

Published as a conference paper at ICLR 2021
the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 2 (Short Papers), pp. 107-112, New Orleans, Louisiana, June 2018. Association for Computational Linguisties. doi: 10.18653/v1/118-2000
URL https: / /www a orabb.rgg/anne wwt..r/aggolgggg//t....o.ooo 8 8-2017
Rowan Hall Maudslay, Hila Gonen, Ryan Cotterell, and Simone Teufel. It's all in the name: Mitigating gender bias with name-based counterfactual data substitution. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- Natural Language Processing (EMNLP-IJCNLA) pp. 5266- 5274, Hong Kong, China, November 2019, Association for Computational Linguistics. doi: 10.18653/v 1L118633/1/DI1--11 1/309-1991100. URL https:: www wwwwwb.wwee.... http://wwwww.w.g....cc.. http:///aa.atoooogo//---------/.
lernational Joint Conference
John Hewitt and Christopher D. Manning, A structural probe for finding syntax in word representa- tions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Shori Pa- pers), pp. 4129-4138, Minneapolis, Minnesota, June 2019. Association for Computational Lin- guistics. doi: 10.18653/1//119-44199 URL ht wwtp://wwwwww.//..aooooooooooooa
N1 M119-1419
Jack Hoeksema. On the natural history of negative polarity items. Linguistic Analysis, 38(1):3, 2008.
Yu-Lun Hsich, Minhao Cheng, Da- Da-Cheng Juan, Wei Wei, Wen-Lian Hsu, and Cho-Jui Hsich. On the robustness of self-attentive models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1520-1529, 2019.
Dieuwke Hupkes, Sara Veldhoen, and Willem Zuidema. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. Journal of Artificial Intelligence Research, 61:907-926. 2018.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. In Advances in Neural Information Processing Systems, pp. 125-136, 2019.
Pierre Isabelle, Colin Cherry, and George Foster. A Challenge Set Approach to Evaluating Machine Translation. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pp. 2486-2496. Association for Computational Linguistics, 2017. URI. acl welweb.cco /rg/ antho wachooooyyl logy/olyy//l//
http://
Mohit lyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. Adversarial example gener- ation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Confer- ence of the North American Chupter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume (Long Pupers), pp. 1875-1885, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 1118653/11/18811111 URL
https: / www.wb.ww.c...... http://..wwwb..og///atttto...o./ (11001000
Rohan Jha, Charles Lovering, and Ellie Pavlick. When does data augmentation help generalization in nip? arXiv preprini arXiv:2004 150000004.5112.. 2020.
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp- 2031-2031. Association for Computational Linguistics, 2017a. URL (tRLhttt://acliei org/
anthology/D177 1215
Robin Jia and Percy Liang. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2021-2033 Association for Computational Linguistics, 2017b.
Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. Certified robustness to adversarial word substitutions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4129-4144 Hong Kong, China. November 2019. Association for Com- putational Linguistics. doi: 10.18653/v1/D199 1423. URL https: / www. aclweb. org/
bnthologyyDD991422
12

---

Published as a conference paper at ICLR 2021
Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. Learning the difference that makes differ- ence with coumterfactually-aagmeennt data. In International Conference on Learning Represen- tations, 2020. URL
net/ htt.:evwnen....rrmiioosssssssaa
Jordan Kodner and Nitish Gupta. Overestimation of syntactic representation in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics. pp. 1757-1762, Online, July 2020. Association for Computational Linguistics. doi:
http:/3//v1/222.aclmmma. 160. URL https: www. ac www.wb.ww.......ccc...........e. htw.:e/wwrr./aaoooooo////// 160
Tal Linzen. How can we accelerate progress towards human-like linguistic generalization? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. Online, July 2020. Association for Computational Linguisties. doi: 10.18653/v1/
2020.acl main.465. URL httph:tp////c. http://wwwwwlww...cc... wtt.://wwww.b.//aaao..a.....aaaa hwt.://wwwe..orr//aaaaoooo////// aac..acc0ccc........... aacl-man...
465
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies. Transactions of the Association for Computational Linguistics, 4:521-535, 2016. Bttt:htt://aclt wwrp///wwborgg/aae....... http://ahhhhooo/oooolll/1
Nelson F. Liu, Roy Schwartz, and Noah A. Smith. Inoculation by fine-tuning: A method for analyz- ing challenge datasets. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2171-2179, Minncapolis, Minnesota, June 2019a. Association for Com- putational Linguistics. doi: 10.18653/v1/N199 1225. URL https: www www.wb.orwwbbogo
anthol 9othh19999/9911122222 1225
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907) 11622, 2019b.
Kaiji Lu, Piotr Mardziel, Fangjing Wu, Preetam Amancharla, and Anupam Datta. Gender bias in neural natural language processing. In Logic, Language, and Security, pp. 189-202. Springer, 2020.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint urXiv:1706.0608. 2017.
Karttikeya Mangalam and Vinay Uday Prabhu. Do deep neural networks learn shallow Iearnable examples first? 2019.
Rebecca Marvin and Tal Linzen. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1192-1202, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi:
10.18653/v1/D18 1151. URL http:::/t////t. www. aclweb. org/ arthology/D18-111
Tom McCoy, Ellie Pavlick, and Tal Linzen. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the As-
sociation for Computational Linguistiex, pp. 1428-3448 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 18.18653/v 1/119-13/11111144 URL https: www. aclweb.
org/antheloogy/119-1 1334
Amil Merchant, Elahe Rahimtoroghi, Ellie Pavlick, and Ian Tenney. What happens to bert embed- dings during fine-tuning? 2020.
Junghyun Min, R. Thomas McCoy, Dipanjan Das, Emily Pitler, and Tal Linzen. Syntactic data aug- mentation increases robustness to inference heuristics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Seattle, Washington, July 2020. Association for Computational Linguistics.
13

---

Published as a conference paper at ICLR 2021
Aakanksha Naik, Abhilasha Ravichander, Norman Sadeh, Carolyn Rose, and Graham Neubig. Stress Test Evaluation for Natural Language Inference. In Proceedings of the 27th International Conference on Computational Linguistics, pp. 1340-2355, Association for Computational Lin- www..www........................ hwwpa//www....nnnoeeeeooeee.eeee
guistics, 2018.
petstp://aclwbbi
Behnam Neyshabur, Zhiyuan Li, Srinadh Bhojanapalli, Yann LeCun, and Nathan Srebro. The role of over-parametrizztiion in generalization of neural networks. In International Conference on Learn- ing Representations, 2019.
hnttps://opeereeee
net/ htt.:/orrmmmiid-yfffa.c...
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adver- sarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4885-4901, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2000 m URL
https:/ htts://wwww.cco www.wwwwwwww..... org wwt.oo///nnnooooooooooooooooooo
main. 441,
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Adam Poliak, Aparajita Haldar, Rachel Rudinger, J. Edward Hu, Ellie Pavlick, Aaron Steven White, and Benjamin Van Durme. Collecting diverse natural language inference problems for sentence representation evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 67-81, Brussels. Belgium, NetoberrNovember 2018a. 8a. Association for Computational Linguistics. doi: 10. 10.18653/w1/D188 1007. URL https: // //www. acw.lllllss.... www.ebb.rgggnthhooogg.ooo 8-1007
Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. Hypothesis only baselines natural language inference. In Proceedings of the Seventh Joins Conference Lexical and Computational Semantics, pp. 180-191, New Orleans, Louisiana, June 201 8b. Association for Computational Linguistics. doi: 10. 1881188/111118222222 URL https: www. ac www. org/ antho
Yada Pruksachatkun, Jason Phang. Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. Intermediate-tast transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 5231-5247, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/0020..u a.1.0alll-m. nel-main.467. URL
https::/wwwt http:////www..wwwe.bbbb. awwweb. org www.://ee....o/attoooooo//////// http//annhooooooooeeeaea acl main. 467.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 188):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http:
7/ //jmlr. hww.l/.rggpppers////e
074 html
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Semantically equivalent adversarial rules for debugging nlp models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics |Volume 1: Long Papers), pp. 856-865, 2018.
Jorma Rissanen. Modeling by shortest data description. Automatica, 14(5):465 471, 1978.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chap- of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 8-14, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18 2002. htthttts: / /www acwweb. org/ antgg//nhology// N18-2002]
Sofia Serrano and Noah A. Smith. attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pp. 2931-2951, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/P19 1282. URL https:
7/ 7/www. a web. wwwpaa/ww...gggaaaoooooo/l//---.
14

---

Published as a conference paper at ICLR 2021
Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 4593-4601, Florence, Italy, July 2019a. Association for Computational Linguistics. doi: 10. 18653/v1/P19- 1452. URL httpt:t/t///.. htt:://www... /www. ac web. /wtp.//wwwwwwwww/////aattooaeeta
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Sam Bowman, Dipanjan Das, and Ellie Pavlick. What do you learn from context? Probing for sentence structure in contextualized word representations. In Interna- tional Conference on Learning Representations, 2019b. URL
net/
Darumiid-SJ2ScnRonn
Sara Veldhoen, Dieuwke Hupkes, and Willem Zuidema. Diagnostic Classifiers: Revealing how Neural Networks Process Hierarchical Structure. In CEUR Workshop Proceedings, 2016.
Elena Voita and Ivan Titov. Information-theerreie probing with minimum description length. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 183-196, Online, November 2020. Association for Computational Linguisties. doi: 10. 10.118653w1/2220.cmm
URL https:://///.. http///wwww..coo www.wa.wwwb..omm harzg//nthlogy//
2020. emn 1
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pp. 5797-5808, Florence, URL
Italy, July 2019. Association for Computational Linguistics. doi:
https: www. www.acllebb www.wwwwwww..................... httpg//nnolooyy///.1 Pt11//99911100
Alex Warstadt and Samuel R Bowman. Grammatical analysis of pretrained sentence encoders with acceptability judgments. arXiv preprint arXiv: 191100339 2019.
Alex Warstadt and Samuel R. Bowman. Can neural networks acquire a structural bias from raw linguistic data?, 2020.
Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Fu Wang, and Samuel R Bowman. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational I Linguistics, 8:377-392, 2020a.
Alex Warstadt, Yian Zhang, Xiaocheng Li, Haokun Liu, and Samuel R. Bowman. Learning which features matter: RoBERTa acquires preference for linguistic generalizations (eventually). In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 217-235, Online, November 2020b. Association for Computational Linguistics. hotooyyy/
doi: 306653/vw/222..mm///. m6....6. URL https:
2020. em:0.emnnii 1p main.16
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, Volume 1 (Long Papers), pp. 1 1122,122, New Orleans, Louisiana, June 2018 Association for Computational Linguistics. doi: 10.18655/v1/118-110000 URL htthhttp::/ www aclweb. ht///athhoooyy/eee.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jemiit,, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. As- sociation for Computational Linguistics. URL https::///// www. aclweb. arg/anthology/
2020. emn 1p demos...
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. CoRR,
2016. URL http:
15

---

Published as a conference paper at ICLR 2021
Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai- Wei Chang. Gender bias in coreference resolution: Evaluation and debiasing methods. Proceedings of the 2018 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Techmologies, Volume (Short Papers), pp- 15-20, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/11N11--200 URL https: / wwwp.aacleeb.org/aatboo...oooo
2003
Jieyu Zhao. Tianlu Wang. Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai- Wei Chang. Gender bias contextualized word embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pp. 629-634, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 10. 10.18633/1//1191004,, URL https:
7/ /ww..acwwwb.b..m wwtp:///nnnntooooool-----0--
Ran Zmigrod, Sebastian J. Mielke, Hanna Wallach, and Ryan Cotterell. Counterfactual data aug- mentation for mitigating gender stereotypes in languages with rich morphology. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1651-1661, Flo- rence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 10.18653/v1/P19-1110 URL https: / www. aclwebs org/ httpbb/oor//ccooooo///---1111
16

---

Published as a conference paper at ICLR 2021
PREDICTING INDUCTIVE BIASES OF PRE-TRAINED MODELS
A ADDITIONAL RESULTS
10 show additional results for all models over all partitions (both accuracy, neither accuracy, and F sccore). These charts appear at the end of the Appendix.
Figure
Details on the MDL statistics are available in Table 3
MDL(() GloVe
BERT
GPT2
RoBERTa
T5
gap_base lengbahh.ennth
443
614
2921
21
537
gap_base, bex__l
223
337
1815
17
323
gap base plural
299
346
1664
14
401
gap base tense
278
417
1962
16
373
gap hard length
421
479
2845
18
493
rap_hard_lexien
292
341
1846
17
321
gap_hard
116
77
946
182
gap.hard.plura.
322
440
1862
15
387
gap gap.hard_teen hense
354
353
1685
15
332
300
240
952
16
273
npi lexiell
225
345
1249
12
245
mpi-plural
292
1267
15
282
npi tense
220
258
999
13
286
sva. base agreement
113
463
1581
12
149
lexical
150
629
5872
15
154
sva_base_plure
5122
166
sva_hard_agreemen
173
579
1582
14
176
sva hard length
183
708
1466
17
197
hard lexical
181
684
1467
16
189
hard plural
192
698
1449
18
201
MDL(s)
gap. base lenghh
103
gap base lexical
297
425
452
12
241
eap.base.plure
3421
3794
1906
45
1924
gap.base_tenss
159
141
710
9
235
eap_hard_lengee
42
46
98
gap hard lexical
313
410
12
246
gap hard none
144
85
957
9
179
gap hard plural
3200
3161
3987
37
1795
lense
131
131
694
198
nei.length
499
297
nei_
48
475
6 123
npi-plural
15
22
1165
6
125
npi_tense
14
18
435
5
150
www_bass__ggrcmmee
305
348
1879
16
198
sva hase lexical
162
sva hase plural
77
199
535
11
174
sva hard agreement
319
835
1314
15
230
sva_hard_enngt sva.hard_lexical
94
28
465
8
103
891
sva_hard_plural
168
695
14
256
Table 3: Summary of extractability (MDL in bits) for t and for each template and each model.
A.1 BEYOND ACCURACY?
For the transformer models, for 18/20 feature pairs, the models are able to solve all the spurious and target features in isolation during probing. (They do solve the test set in all cases-its that two
17

---

Published as a conference paper at ICLR 2021
Absolute
Relative (1 to 8)
Target
Spurious
Ratio
Difference
BERT RoBERTa
-0.51
0.72*
0.83*
0.95*
-0.36
0.84*
0.87
0.85*
T5
-0.57*
0.26
0.67*
0.78*
GPT2
-0.5*
0.66*
0,8*
0.83*
GloVe
0.24
0.43
0.3
0.27
Table 4: Spearman's p: MDL vs. Average Test F-score: Correlations when Probing Accuracy is Not Controlled.
of the spurious features ended up being very difficult for the models.) During the reviews, we did control for the cases where the model did not solve the probing task. These 2 extraneous points accentuate the lineplot curves, but do not change the character of the results (nor much adjust the correlations). In the paper. now, we control for accuracy by filtering out these cases. With or without this control, the accuracy provides no predictive power about the inductive biases. We present the correlations without filtering for these cases for consistency with the reviews (Table 4 above); we believe it is important control for these cases because they could have acted giveaways, where even accuracy might have worked.
A.2 ALTERNATE METRIC: S-RATE*
We initially used different metric when computing the correlations to compact the lineplots. Rather than using the average test performance, looked at the evidence required for the model to solve the test set. Both of these metrics conceptually capture what we are interested in, but the new one (simply averaging test performance) is much easier to understand, and captures the performance across all partitions. Here we report the correlations with this evidence required metric instead, which we called s-rate*. Specifically, at which the fine-tuned model achieves essentially perfect performance (F-score > 0.99) (see Figure Sap. Intuitively, s-rate* is the (observed) minimum amount of evidence from which the model infer that alone is predictive of the label. See Table for the results.
defined be: s-rate* is the lowest s-only example rate
Absolute Target
Relative (t to s)
Spurious
Ratio
Diff
F-Scare
1.0
BERT RoBERTa
0.68*
-0.63*
-0.79*
-0.81
3-rate
0,04
-0,69*
-0.71
-0,69*
Threshold: 0.99
0.8
T5
0.81
-0.03
-0.55*
-0.65*
GPT2
-0.11
-0.24
-0.29
-0.32
0.001
0.01
0.05
0.5
GloVe 0.29
-0.38
-0.48
-0.48
Spurious-onlyy Ex. Rate
(a) Evidence Required: s-rate*
(b) Using Evidence Required (s-rate*) instead Average score. The correlations are negative instead of positive: As the extractibility increases, less evidence is required for the model perform well.
A.3 RESULTS USING AUC INSTEAD OF MDL
See Table 5 A metric similar to MDL for capturing the same intuition is the area under the valida- tion loss curve (AUC). This metric is highly related to online MDL. in computation.
B IMPLEMENTATION DETAILS & REPRODUCIBLITY
Our code is available at:
18

---

Published as a conference paper at ICLR 2021
Absolute
Relative (1 to 8)
Target
Spurious
Ratio
Diff.
BERT RoBERTa
-0.61*
0.61
0.71 *
0.67*
-0.12
0.67*
0.75*
0.64*
T5
-0.88*
0.37
0.84
0,95*
GPT2
-0.34
0,52*
0,64
0.67
GloVe
-0.5
-0.63*
-0.23
-0.14
Table 5: AUC. Each column presents the Spearman correlation between the given measure and the s-only rate which the model first acquired the target feature. * 'indicates a significant correlation. A feature is considered acquired when its test-performance is above 0.99 F-score. All (s, pairs (20 of 20 each) were eventually learned by T5, BERT, GPT2, and RoBERTa; 11 of 20 (s, t) pairs were learned by the GloVe Ve-beed LSTM.
There two major parts to this project in terms of reproducibility: (1) the data and (2) the model implementations. We describe the templates for the data below in Appendix - the full details are the project source. For the transformer models, we use Hugging Face for the implementations and access to the pre-trained embeddings (Wolf et al. (020). We use Py Lightning to organize the training code (Falcon 2019). We fix all hyperparameters, which are reported in Table
We want to call attention to BERT requiring much less data than T5 to capture our target features. At face value, it seems that BERT requires much less data than T5 to capture our target features. However, we are wary about making such strong claims. Something to consider here (noted in the Appendix
that for we used a linear model rather than formatting the task in text (which is how T5 is trained). We made this decision (1) because we had trouble training T5 in this purely textual manner, and (2) using a linear classification head over two classes consistent with the other model architectures. Again, GPT2 and RoBERTa performed between the performance of BERT and T5 may be due to how we trained T5.
par with BERT, so the difference
C MEASURING EXTRACTABILITY INDIRECTLY
We measure the MDL. for with both and s-only examples. In the simulated setting compare this approach with measuring the MDL. directly (t-only vs neither). See Table 7 for MDL results. The ordering of the feature's difficulty holds across the two methods.
D TEMPLATES FOR NATURALISTIC DATA
Each template corresponds to combination of target features, grammars, and spurious features (the target and spurious features are discussed in Section 4.1). See Table 8 for a complete list of templates. See Table for further details about the templates that are used for each of the the target features. Complete details about implementation of these templates (and all data) will be released upon acceptance.
E WHY EXACTLY IS IT HARD TO GENERATE T-ONLY EXAMPLES?
Target features may be unavoidably linked to spurious ones. For example, for a Negative Polarity Item to be licensed (perhaps smoothing over some intricacies) the NPI ("any", "all", etc) must be a downward entailing context. These downward entailing contexts are created by triggers, e.g., negative word like "no" or "not" quantifier like "some". Linguists who study the problem have assembled a list of such triggers (see Hoeksemal (2008)). Arguably, one cannot write down correct example of NLP licensing that doesn't contain one of these memorizable triggers. Thus, we cannot train or test models on correct examples of NPI usage while simultaneously preventing from having access trigger-specific features.
19

---

Published as a conference paper at ICLR 2021
Hyperparameter
Value
Hyperparameters
random seed
1,
batch size cumulative mdl block sizes (%)
128
0.1, 0.1, 0.2, 0.4, 0.8, 1.6, 3.05, 6.25, 12.5, 25
s-only rates (%)
0.1, 5, 10, 20, 50
TS Hyperparameters
model keycode
t5-base
model architecture
T5Model
warmup
linear (default values)
optimizer
AdamW
pooler
last hidden state
Ir
2e-5
batch size
128
classifier head
Linear
BERT/ROBERTUGGPT
Hyperpurameters
model keycode
bert-base-uncased
model architecure
BertForSequenceCaasifttttt
warmup
cosine (default values)
optimizer
Adam
pooler
first hidden state
Ir
2e-5
batch size
128
classifier head
MLP
LSTM (Toy and GloVe) Hyperparameters
optimizer
Adam
pooler
last hidden state
Ir
lc-3
batch size
128
classifier head
MLP with 1 hidden-layer
tanh activation
hidden size
300
Table 6: Hyper and System Parameters. We use Hugging Face for the underlying model imple- mentations.
Feature
Probe
MDL in k-bits
adj-dupl
(s-only vs both)
242.19
(I-only vs neither)
248.68
contains-1
(s-only vs both)
0.29
(f-only neither)
0.34
first-last
(s-only vs both)
397.64
(f-only neither)
607.46
prpf
(s-only vs both)
175.74
(t-only neither)
193.92
Table 7: We measure the target MDL directly on the toy data, where we can access the target feature. Recall that in natural conditions of spurious features), so we used a dataset comprising both and s-only examples. In the simulated setting, approach for measuring the target extractability indirectly by using both and s-only examples reports results similar to those when directly using target and neither. These values are in kbits.
generate examples with the target feature (free
20

---

Published as a conference paper at ICLR 2021
Target feature
Spurious features
grammar
#
SVA
closest-noun, length', lexical, plurality length, lexical, plurality, tense length, lexical, plurality, tense
base, nested 7 base
NPI
4
GAP
base, islands"
9
Length is not applicable for the base template We also run the islands template without additional spurious features.
Table 8: List of templates that are used in Section
These are discussed in greater detail below.
Target
Spurious
Example
Subject agrees
before
[neither] * The piano teachers of the lawyers wounds the handyman.
with verb
is singular
NPI in down. entaiiling context
Contains negation word
[neither]
The student who wrong ever resigned.
Correct filler-gap dependency
Main verb is in past tense
[neither]
know
that he recognized
yesterday.
Table 9: neither examples for Table
Subcase
Template
Example
NPI V
No Np- ever V. Det Np V.
No girl who was sad ever resigned. Some student who no girl ever fol- lowed ran.
NPI V
NPI X
The N, ever V.
*The man who some lawyer hated ever traveled.
NPI X
The Np-ever
*The boy who ever smiled shouted.
SVA V
The Nf of the (N, of the y" N₂ Vi2 the N3.
The piano teacher of the lawyers wounds the handyman.
SVA X
The NI of the (N; of the )* N2 Vi" the Ng.
*The piano teacher of the lawyers wound the handyman. Whether or not noun and verb agree is given by whether their super- scripts match.
GAP
(N; V/ that) N1 V1 who (N₂ Vi
know who he believed.
that)* (N, V.).
GAP V GAP X
I know that he believed them. (N; V2 that)= N1 V1 who (N2 Vi
(N, V: that) 1+ N2.
#I know who he believed them.
that)* N₁ Vi2 N₂.
GAP X
(N, V; that) N₁ Vi.
know that he believed. "I know who he knew the lawyers that she believed.
ISL X
Ni Vi who N₂ V/2 N₂ that N₂ Vg.
Table 10: List of templates that are used in Section These do not include the spurious features which are discussed in Section 4.I For NPI, each N_ represents a noun phrase. Np-ney is valid after negation (might contain polarity item). Np is valid after determiner (cannot contain an unlicensed polarity item). Ny-ever is not valid after a determiner (contains an unlicensed polarity item). These phrases have complex nesting behavior and become arbitrary long. In addition. a sentence might consist of multiple independent clauses, each of which is given by one of these templates. For SVA, the base templates do not have the additional nouns in the starred parentheticals, while the nested templates have zero more. For GAP, the harder set of templates include ISL (island) examples an additional s-only example that force the model to not violate (one specific type) of island constraint. For complete details and lexicons see the source.
21

---

Published as a conference paper at ICLR 2021
Similar to the NPI example, it's not possible (to our knowledge) to construct target-only examples for filler-gap since construction requires a wh-word and syntactic gap; thus, we can't It create a positively labeled, grammatical sentence that exhibits Filler Gap without these elements.
In summary, target-only examples may add new spurious features (as with NPD, or be impossible to construct because the presence of the target feature implies the presence of the spurious feature (as with filler gaps). Still, our setup permits the MDL to be computed directly with target-only examples, and so, in cases where feasible create target-only examples (e.g. the Subject- Verb Agreement templates), would have bolstered our argument to do so.
f.taeeeure pairs
0.015
0.010
Both
0.005
0.000
0,001
0.01
0.05
0.1
0.2
0.5
Relative Extractability of Target Feature (MDL(s)/MDL())
Evidence from Spurious-only Examples (s-only example rate)
Neither Error
0.03
0.02
0.00
0.001
0.01
0.05 0.1
0.2
Evidence from Spurious-only Examples (s-only example rate)
Spurious (s-only) Error
1.0
0.8
0.6
0.2
0.0
0.2
Evidence from Spurious-only Examples (s-only example rate)
1.0
Test S-ore
0.9
0.8
0.7
0.6
0.001
0.01
0.05 0.1
0.2
Evidence from Spurious-only Examples (s-only example rate)
Figure 6: T5.
22

---

Published as a conference paper at ICLR 2021
0.0025
0.0020
Both Error
5, feature
0.0015
0.0010
0.0005
0.0000
0.001
0.01
0.05 0.1
0.2
0.5
Relative Extractability Target Feature (MDL(s)/MDL(()
Evidence from Spurious-only Examples (s-only example rate)
0.006
0.004
Neither
0.002
0.000
0.001
0.01
0.05 0.1
0.2
0.5
Evidence from Spurious-only Examples
(s-only example rate)
Spurious (5-only) Error
1.0
0.4
0.2
0.0
0.001
0.01
0.05
0.1
0.2
0.5
Evidence from Spurious-only Examples (s-only example rate)
1.0
Test F-Score
0,001
0.01
0.05
0.1
0.2
0.5
Evidence from Spurious-only Examples (s-only example rate)
Figure 7: BERT.
F MDL ISSUES: OVERFITTING IN THE SYNTHETIC EXPERIMENTS
We found that the MDL exceeds the uniform code length in some of the synthetic experiments. We found that this occurs because the model overfits on the small carly-block sizes. See Figure
23

---

Published as a conference paper at ICLR 2021
feature pairs
0.6
Both Error
0.4
0.2
0.0
0.0
0.5
1.0
1.5
0.001
0.01
0.05 0.1
0.2
0.5
Relative Extractability of Target Feature MML(()/MDL(()
Evidence from Spurious-only Examples (s-only example rate)
0.08
Neither Erro
0.06
0.02
0.00
0.001
0.01
0.05 0.1
0.2
0.5
Evidence from Spurious-only Examples (s-only example rate)
1.0
Spurious (s-only) Error
0.8
0.6
0.4
0.2
0.0
0.001
0.01
0.05 0.1
0.2
0.5
Evidence from Spurlous-only Examples (s-only example rate)
1.0
0.9
Test F-Scrr
0.8
0.7
0.6
0.001
0.01
0.05 0.1
0.2
0.5
Evidence from Spurious-only Examples (s-only example rate)
Figure 8: GloVe.
24

---

Published as a conference paper at ICLR 2021
0.006
s,tfeature pairs
0.004
Both
0.002
0.000
0.001
0.01
0.05 0.1
0.2
0.5
Relative Extractability of Target Feature (MDL(S)/MDL(()
Evidence from Spurious-only Examples (s-only example rate)
0.020
0.015
Nelther
0.010
0.005
0.000
0.001
0.01
0.05
0.1
0.2
0.5
Evidence from Spurious-only Examples (s-only example rate)
Spurious (s-only) Error
0.8
0.6
0.4
0.2
0,001
0.01
0.05 0.1
0.2
0.5
Evidence from Spurious-only Examples (s-only example rate)
1.0
F-Scrr
0.9
0.8
0.7
0.6
0.001
0.01
0.05 0.1
0.2
0.5
Evidence from Spurious-anly Examples (s-only example rate)
Figure 9: GPT2.
25

---

Published as a conference paper at ICLR 2021
0.3
s.tfeature pairs
Both Error
0.2
0.0
0.001
0.01
0.05
0.1
0.2
0.5
Relative Extractability of Target Feature (MDL(S)/MDL(()
Evidence from Spurious-only Examples (s-only example rate)
0.015
Nelther
0.010
0.005
0.000
0.001
0.01
0.05
0.1
0.2
0.5
Evidence from Spurious-only Examples (s-only example rate)
Spurious (s-only) Error
0.8
0.6
0.4
0.2
0.001
0.01
0.05
0.1
0.2
0.5
Evidence from Spurious-only Examples (s-only example rate)
1.0
Test F-Sorre
0.9
0.8
0.7
0.6
0.001
0.01
0.05 0.1
0.2
0.5
Evidence from Spurious-only Examples (s-only example rate)
Figure 10: RoBERTa.
26

---

Published as a conference paper at ICLR 2021
Models Overfit on Early Blocks
105
10
10
Loss
10-1
prop
toy_
10-3
toy_5 5
toy_2 2
10-5
toy_3
uniform code
103
104
105
Block Length
Figure 11: Overfitting on the Synthetic Tasks.
27


Reviews:
[{"review_id": "wjiGejIhgG", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper aims to bridge the gap between model interpretation using probing and model's use of spurious features. They show that the findings of MDL with respect to a feature correlate with the extractability of the feature, given the evidence of representing the feature is available in the training data. The results are presented using both synthetic and natural language data.\n\nI really like the premise of the paper, which is connecting the research on the linguistic learning of a model with the presence of important and spurious features in the data.\n\nOne issue I have with the work is the simplistic assumptions that are likely to be different in the real-world data. Real-world data may have various spurious features and it is possible that not one feature alone is playing a role in pushing the model to rely on spurious features. It can be a combination of spurious features plus the relative presence of important features. It is hard to imagine how this method will scale to real-world datasets. I would like the authors to comment on it.\n\nMoreover, the findings are quite expected. In general, the probing methods including MDL were mainly aimed at analyzing the linguistic learning of the representations. In that case, MDL is scoring the representations with respect to various linguistic properties. Here, the authors are using MDL to look at how input features are represented in the model. Statistically, MDL is likely to look at the same things as the model is looking at since both of them are based on the same training data and input features. Please comment on this, in case I misunderstood the point.\n\n\nMinor comments:\n- what is the reason for low performance when using Glove?", "extra_scores": {}, "extra_text": {}}, {"review_id": "bzqPfJ0T1rN", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n# Summary\nThis paper studies the relationship between extractability of features from pre-trained representations and how much a fine-tuned model uses that feature. The extractability of features is measured by the minimum description length of a probing classifier trained to detect the feature from the pre-trained representations (using the online code version of Voita and Titov). The degree to which a fine-tuned model uses the feature is measured by the amount of evidence required for a model to tease apart spurious from non-spurious features (called \"target\" features). Evidence here means examples where a spurious feature occurs but a non-spurious feature does not occur. When there are many such examples (high spurious-only rate), it is easier for a model to reject the spurious feature and learn to rely on the target feature. The \"degree to which a fine-tuned model uses a feature\" is defined as the minimal spurious-only rate at which the model can accomplish the task. \n\nThe paper has two kinds of experiments, on synthetic and more natural data. The synthetic data are sequences of symbols where the task is to identify simple properties like occurrence or repetition of symbols. The experiments are set up such that varying rates of spurious-only examples are presented during training, providing increasing amounts of evidence against the spurious feature (presence of the symbol 2) and in favor of the target feature. The target feature is identical to the label, that is, it is 1 when the example corresponds to the label and 0 otherwise. The paper reports extractability of the spurious and target features via the MDL of a probing classifier. The metric of interest is the relative MDL, where higher means the feature is more extractable. When the features are more extractable, less evidence is required for the model to reject spurious features. With less extractable features, more evidence is required. \n\nThe natural language examples are made with acceptability judgements of examples generated by grammars for three linguistic phenomena (subject-verb agreement, negative polarity items, and filler gap dependencies). Here again the setting is similar, modulu a tweak on how to calculate extractability. The main result here is high (negative) correlation between extractability and evidence required for rejecting the spurious feature. \n\n# Main comments\n1. This paper fills an important gap in the NLP interpretability literature that has recently been a cause of concern in the community. On the one hand, probing classifiers tell us something about the existence (and more recently the extractability) of properties in pre-trained models' representations. But they do not tell us whether a model uses those properties. One the other hand, many challenge sets and test suites tell us whether a model can successfully perform a task requiring some linguistic property. The paper aims to connect these two aspects, and it does so quite convincingly, although I have some reservations below. \n2. The experimental setup is well designed. The use of synthetic data allows a fairly clean setup where spurious and non-spurious features are distinct and simple. The experiments of training with increasing amount of spurious-only examples are instructive. \n3. The natural language examples are important as they go beyond synthetic data and closer to a naturalistic scenario. However, these are still templatic sentences and synthetic in a sense. I wouldn't call these naturalistic examples. Ideally, experiments on naturally occurring data would be more convincing. Or, at the very least a discussion of this issue should be made. \n4. The paper makes use of recent advances in interpretability work, including information-theoretic probing, and draws connections to a broad range of related work. \n5. The assumption that the extent to which a model uses a feature can be measured by the spurious-only error rate (at some spurious-only occurrence rate) is questionable in my opinion. In a very clean setting like the synthetic data, I could maybe accept it. But, \"using\" is in fact a causal concept, while a causal mechanism has not been demonstrated. The paper alludes to this point in the discussion, but I think the discussion around this point should be expanded, and the strong claims should be rephrased or modulated. \n \n\n# Questions and other comments\n1. The paper makes the assumption that the target feature t and the label are the same. I am not convinced about the \"without loss of generality\" claim. In practice, it is not easy to isolate a feature t that is identical with the label. How would this assumption affect the generalization of the approach to more realistic scenarios? \n2. The task is a binary classification task. The features holding is also binary, that is either a feature holds (1) or not (0). But, suppose the label is 0, then the t feature is also 0, meaning it does not hold. This seems contrary to what is meant. This could be a confusion on my part. \n\n## Synthetic data\n3. Why is MDL computed by training a classifier to distinguish s-only from neither, and not from some other part of S? \n4. Footnote 3 is concerning - Aren't MDLs higher than a uniform code meaningless? \n5. The classifier is not so simple (LSTM + 1-layer MLP). Why is that? How does the identity of the classifier affects the results? \n6. The both-error subplot in figure 2 shows a slight increase in error rate with large s-only rate. Does this mean that the model has (falsely) learned to reject the example when s is in it? That is, it has learned another spurious feature, just in the other direction, instead of learning to rely on t. \n7. A similar pattern is found in the t-only error subplot. There, even with high s-only rate, the models don't classify t-only examples correctly. I wonder why this plot is different from the s-only error plot, as this shows a directional behavior. Some discussion of this would be useful. \n8. There seems to be a stark contrast between the contains-1 feature and the other three, both in terms of MDL and in figure 2. Is it possible to show a more gradual behavior between the two extremes? \n\n## Natural language examples\n9. Why are the training sets so small? How does this affect MDL numbers and their validity? Apropos footnote 1. \n10. Why exactly is it hard to generate t-only examples? The appendix is indeed helpful in making sure the MDL(t) calculation method is legit, but more clarification around this issue would be good.  \n11. Here, s-only error is used as \"the use of the spurious feature\", but this is only one aspect in which a model may make use of s. It may be that a model makes more complicated use of s, when s is found in combination with t. The discussion touches upon this point by acknowledging that the work does not establish a causal relationship between extractability and feature use. I'd go even further and say that \"feature use\" should be defined in causal terms. \n12. What is the performance (F-score) for determining s-rate*? is that the performance on s-only examples? on other examples? Why the shift to F-score now?\n13. Discuss y-axis differences in figure 3c. BERT needs much less evidence than (some cases of) GloVe and T5. How does that impact the analysis?\n14. The term \"learning curves\" for figure 4 is confusing: those aren't results during training, right? They are results after training, each time with a different rate of s-only examples. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "CsE0g3JfN-x", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After reading author responses:\n\nThank you to the authors for your detailed responses. With regard to the highlighted implication that \"the harder feature can be obscured completely by a spurious one; i.e., there are settings in which the model just won't adopt the harder feature at all\" --  to clarify, while my phrasing may not have made this apparent, I was assuming this implication in my interpretation of the results. So my impression of the finding is not changed substantially by the author response. However, I do want to give appropriate acknowledgment of the value of explicitly testing/confirming intuitive explanations of model behaviors, and it is clear that other reviewers find value in the contribution, so I am bumping my score up a bit.\n\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nThis paper addresses a seeming contradiction between findings that indicate encoding of linguistic information in models' internal representations, and findings that show models not to use more sophisticated linguistic information during fine-tuning on downstream tasks. The paper hypothesizes that a model's use of a given feature can be explained as a function of extractability of the feature in combination with the amount of evidence for that feature's predictive reliability. The authors test on toy, non-language data as well as natural language data, and find support for their hypothesis. \n\nAll in all I think this is a reasonably clear and well-written paper, with a concrete and intuitive hypothesis. My main concern is that the motivating issue is a bit of a strawman, in that the posited explanation was fairly obvious as a means of reconciling the \"contradiction\" raised at the start of the paper. I can't speak for the rest of the community, and it may be that this is something that people have found puzzling -- but speaking for myself I can say that I haven't at any point considered the highlighted \"contradiction\" to be a contradiction, having simply assumed something like the explanation hypothesized in this paper. Now, there is of course value in providing concrete evidence supporting intuitive assumptions made by the community. However, as the authors point out,  related intuitions have already been supported by, e.g., evidence that models will more readily pick up on \"easy\" examples over \"difficult\" examples. So it's not clear to me that the paper is making a sufficiently novel, surprising contribution at present.\n\nI think one way in which these findings would be more compelling would be if the measure of extractability were defined independently of empirical classifier sensitivity. As it is, the experiments are seemingly demonstrating that the more readily a classifier is able to pick up on a given feature, the more readily another classifier will use that feature during learning.  I have to assume that this will strike most readers as obvious. However, if extractability/MDL were measured independently of classification performance, then we would presumably learn some interesting and valuable things about what determines extractability for these models.\n\nSmaller notes:\n\nThe two sets of experiments are described as \"synthetic\" versus \"natural language\" -- but if I'm understanding correctly, the natural language examples are generated synthetically. If this is correct, then the current framing of the distinction is misleading.\n\nFigure 2 is difficult to interpret, and the placement of the legend is odd-looking and confusing. Fig 3 is also pretty difficult to extract information from -- generally presentation of information could be made clearer for the reader.\n\nThe wording on p3 can be taken to imply that MDL was introduced by Voita & Titov (2020). I would recommend rephrasing and/or also citing earlier MDL references.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the features extracted by the pre-trained language model and how fine-tuning makes use of these features. The paper is well-motivated by two lines of research in the NLP area -- 1) probing approaches for understanding the features extracted in the pre-training model, 2) model behavior analysis that shows models take shortcuts for making predictions. The paper provides a comprehensive study to bridge the gaps between these two lines of discussion. 

All the reviewers agree the paper has strong merits and concerns have been addressed.  

Decision: Accept (Poster)