PREDICTING INDUCTIVE BIASES OF PRE-TRAINED MODELS
Year: 2021

Abstract:
Most current NLP systems are based on a pre-train-then-fine-tune paradigm, in which a large neural network is first trained in a self-supervised way designed to encourage the network to extract broadly-useful linguistic features, and then finetuned for a specific task of interest. Recent work attempts to understand why this recipe works and explain when it fails. Currently, such analyses have produced two sets of apparently-contradictory results. Work that analyzes the representations that result from pre-training (via “probing classifiers”) finds evidence that rich features of linguistic structure can be decoded with high accuracy, but work that analyzes model behavior after fine-tuning (via “challenge sets”) indicates that decisions are often not based on such structure but rather on spurious heuristics specific to the training set. In this work, we test the hypothesis that the extent to which a feature influences a model’s decisions can be predicted using a combination of two factors: The feature’s extractability after pre-training (measured using information-theoretic probing techniques), and the evidence available during finetuning (defined as the feature’s co-occurrence rate with the label). In experiments with both synthetic and naturalistic data, we find strong evidence (statistically significant correlations) supporting this hypothesis.

heading: 1 INTRODUCTION
text: Large pre-trained language models (LMs) (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020) have demonstrated impressive empirical success on a range of benchmark NLP tasks. However, analyses have shown that such models are easily fooled when tested on distributions that differ from those they were trained on, suggesting they are often “right for the wrong reasons” (McCoy et al., 2019). Recent research which attempts to understand why such models behave in this way has primarily made use of two analysis techniques: probing classifiers (Adi et al., 2017; Hupkes et al., 2018), which measure whether or not a given feature is encoded by a representation, and challenge sets (Cooper et al., 1996; Linzen et al., 2016; Rudinger et al., 2018), which measure whether model behavior in practice is consistent with use of a given feature. The results obtained via these two techniques currently suggest different conclusions about how well pre-trained representations encode language. Work based on probing classifiers has consistently found evidence that models contain rich information about syntactic structure (Hewitt & Manning, 2019; Bau et al., 2019; Tenney et al., 2019a), while work using challenge sets has frequently revealed that models built on top of these representations do not behave as though they have access to such rich features, rather they fail in trivial ways (Dasgupta et al., 2018; Glockner et al., 2018; Naik et al., 2018).
In this work, we attempt to link these two contrasting views of feature representations. We assume the standard recipe in NLP, in which linguistic representations are first derived from large-scale selfsupervised pre-training intended to encode broadly-useful linguistic features, and then are adapted for a task of interest via transfer learning, or fine-tuning, on a task-specific dataset. We test the
hypothesis that the extent to which a fine-tuned model uses a given feature can be explained as a function of two metrics: The extractability of the feature after pre-training (as measured by probing classifiers) and the evidence available during fine-tuning (defined as the rate of co-occurrence with the label). We first show results on a synthetic task, and second using state-of-the-art pre-trained LMs on language data. Our results suggest that probing classifiers can be viewed as a measure of the pre-trained representation’s inductive biases: The more extractable a feature is after pre-training, the less statistical evidence is required in order for the model to adopt the feature during fine-tuning.
Contribution. This work establishes a relationship between two widely-used techniques for analyzing LMs. Currently, the question of how models’ internal representations (measured by probing classifiers) influence model behavior (measured by challenge sets) remains open (Belinkov & Glass, 2019; Belinkov et al., 2020). Understanding the connection between these two measurement techniques can enable more principled evaluation of and control over neural NLP models.

heading: 2 SETUP AND TERMINOLOGY

heading: 2.1 FORMULATION
text: Our motivation comes from McCoy et al. (2019), which demonstrated that, when fine-tuned on a natural language inference task (Williams et al., 2018, MNLI), a model based on a state-of-the-art pre-trained LM (Devlin et al., 2019, BERT) categorically fails on test examples which defy the expectation of a “lexical overlap heuristic”. For example, the model assumes that the sentence “the lawyer followed the judge” entails “the judge followed the lawyer” purely because all the words in the latter appear in the former. While this heuristic is statistically favorable given the model’s training data, it is not infallible. Specifically, McCoy et al. (2019) report that 90% of the training examples containing lexical overlap had the label “entailment”, but the remaining 10% did not. Moreover, the results of recent studies based on probing classifiers suggest that more robust features are extractable with high reliability from BERT representations. For example, given the example “the lawyer followed the judge”/“the judge followed the lawyer”, if the model can represent that “lawyer” is the agent of “follow” in the first sentence, but is the patient in the second, then the model should conclude that the sentences have different meanings. Such semantic role information can be recovered at > 90% accuracy from BERT embeddings (Tenney et al., 2019b). Thus, the question is: Why would a model prefer a weak feature over a stronger one, if both features are extractable from the model’s representations and justified by the model’s training data?
Abstracting over details, we distill the basic NLP task setting described above into the following, to be formalized in the Section 2.2. We assume a binary sequence classification task where a target feature t perfectly predicts the label (e.g., the label is 1 iff t holds). Here, t represents features which actually determine the label by definition, e.g., whether one sentence semantically entails another. Additionally, there exists a spurious feature s that frequently co-occurs with t in training but is not guaranteed to generalize outside of the training set. Here, s (often called a “heuristic” or “bias” elsewhere in the literature) corresponds to features like lexical overlap, which are predictive of the label in some datasets but are not guaranteed to generalize.
Assumptions. In this work, we assume there is a single t and a single s; in practice there may be many s features. Still, our definition of a feature accommodates multiple spurious or target features. In fact, some of our spurious features already encompass multiple features: the lexical feature, for example, is a combination of several individual-word features because it holds if one of a set of words is in the sentence. This type of spurious feature is common in real datasets: E.g., the hypothesis-only baseline in NLI is a disjunction of lexical features (with semantically unrelated words like “no”, “sleeping”, etc.) (Poliak et al., 2018b; Gururangan et al., 2018).
We assume that s and t frequently co-occur, but that only s occurs in isolation. This assumption reflects realistic NLP task settings since datasets always contain some heuristics, e.g., lexical cues, cultural biases, or artifacts from crowdsourcing (Gururangan et al., 2018). Thus, our experiments focus on manipulating the occurrence of s alone, but not t alone: This means giving the model evidence against relying on s. This is in line with prior applied work that attempts to influence model behavior by increasing the evidence against s during training (Elkahky et al., 2018; Zmigrod et al., 2019; Min et al., 2020).
Published as a conference paper at ICLR 2021

heading: 2.2 DEFINITIONS
text: Let X be the set of all sentences and S be the space of all sentence-label pairs (x, y) ∈ X × {0, 1}. We use D ⊂ S to denote a particular training sample drawn from S. We define two types of binary features: target (t) and spurious (s). Each is a function from sentences x ∈ X to a binary label {0, 1} that indicates whether the feature holds.
Target and spurious features. The target feature t is such that there exists some function f : {0, 1} → {0, 1} such that ∀(x, y) ∈ S, f(t(x)) = y. In other words, the label can always be perfectly predicted given the value of t.1 A feature s is spurious if it is not a target feature.
Partitions of S. To facilitate analysis, we partition S in four regions (Figure 1). We define Ss-only to be the set of examples in which the spurious feature occurs alone (without the target). Similarly, St-only is the set of examples in which the target occurs without the spurious feature. Sboth and Sneither are analogous. For clarity, we sometimes drop the S∗ notation (e.g., s-only in place of Ss-only).
t-only both s-only
neither
S D
(a) (b)
Sboth “ tpx, yq | tpxq “ 1 ^ spxq “ 1u Sneither “ tpx, yq | tpxq “ 0 ^ spxq “ 0u St-only “ tpx, yq | tpxq “ 1 ^ spxq “ 0u Ss-only “ tpx, yq | tpxq “ 0 ^ spxq “ 1u
<latexit sha1_base64="bSiqwQANSN306jL/eI6aFqdfpQw=">AAA00nicnVtbc9zKcaYcx3EYxzlOHv2CCiX7nBTFInWYil9cZd5JcXlfXqSzOiosthcLETdhZrGk4E1VKi9+8Evyg/I//F/8kJ4ZDLoHC/LEYZVE9PfNNOby9XQvFhzmcSTk+vqfXvzor3781z/5m5/+7fLf/ezvf/4PX/3iH29ENi0CuA6yOCvuhr6AOErhWkYyhru8AD8ZxnA7vN9R/G0JhYiytC8fc/iQ+GEajaPAlwh9/OrPHv5cfawGEh5kkVQvh5mcvJzPf/XbQfX1w+rjN4PV3w9W5dcP3/x2YzCDUQieMMbcGwyW271TiOQEim4H69zBercD+fJ1lsaPPzyAJ/qLZ/qvtyaw/PGrlfW1df3jLV5s1BcrS/XP+cdfvPmfwSgLpgmkMoh9Ib7bWM/lh8ovZBTEMF8eTAXkfnDvh/AdXqZ+AuJDpbdp7r1CZOSNswL/pdLTKO9R+YkQj8kQWya+nIg2p8Au7rupHP/mQxWl+VRCGpgbjaexJzNP7bk3igoIZPyIF35QRDhWL5j4hR9IVIZzl2GW3Ut/KNCHt68HWiQ+OvJRT25LNQqZZbFuGqACxfICX4ixmHfO0AHFdDiOwrnnzCnVd4bkQzXF/7s6rHrjOPPlfOG2/vBhNSz8fBIFDy4rs1GWZhJao5phYzWCp2bQuRFtUO3p/2W2z4/w/+N9efmV+vFO9269k63+obe7t390etQ/Oju98jTVOYNV63Z1iOu7/Mo78Yt7T6BS8GQQXjbGbc3NtdJsAWMoiigNlaxGURkJ2wyXblooeaQwC7Ik8dNRNUAwhrGcV9UAEu/rHl5/M58vtAkwkqCwrXa01dWuiMJJ4+xSGV2tZJbbNv0s72qBx5vMEttoW1sL7ep5+7aZ/1SLoW0xfKpFYFsET7UY2RYj1QK34RBnF6sZer6H7VXYwhiP95GHa5O4PvBagfPvNj6gl+HYW9mYa0F4+3pTzK7hoQCrXpzNoHitQnVteYAu9bLCeGWjMhv47wO0Ku2gqzsON5J+vKbPBSHxyFN7L9SOIW887luP+22PmpazzN5z5U19V+HZRh7OqDbe2B6fp/6Iuqx8u7K50G216WOvvuWuNvV0royqn10OVL4ZfB0Cznp0OLALYnpf2d5XHb0vbS99JM+yJsrWmoUxdxd6ZZoYfGJp2g4nBUDbJfO38u2iR1o15vvbRd9+6gFugurcsWTw2czZNnl60o6faZ5DoZOGcbNXu9nrcrPlFf6M1r3l7PXr136ZRSNvKtTRFI29PBMiwnRlXOexj6FT+396dCof5hhJHXNUjOlet/mLJ1k72mkc7fygI5xzipWKOoNNW2F8aLgZEUrF0tbV69dPygRH58dhhvl/knTMEzkzuqbRsxNlrhZmumVdbXW4soK398NJNL6ePwz6TqetH+y0sKg5Hlr1zJn6FGqGq66e2xTTv63e86b/udvfzrS5AY5aXT854FpwEMVKrLG6wAMdG6ir2h/WPFmhaX1leH1ZN0BqmFQb7WwjCwyEeTVQmT/w42q33aD042jEG3w011hVG2q+4BKE7O6gmXkzI8iFynG5iOIsrfPTJbrIEq/0i6guLrW+QfqVqeZNAVi9HCD0cm6Xs2jRPjFDlxkSE7hMQMzIZUbEgMsAMWOXGRMTukxIzMRlJsRELhMR88llPhFz7zL3xMQuE8+1jIvEiwRGLH4wHD2qw87s4Kr3aSqkhyXxr6WnPqqgHB/VyeNsjJfUvlPXd0p3zVwmIyZ3mZyYzy7zmZjCZQpihMsIYqTLSGKmLjMlpnSZkpiZy8yIeXCZB2IeXeaRmC8u82VuyjwbAJiZs+Z4L+sgqUwoDccsbJpx4ydsHSW2hbYZzzgODwlmsVEGBLPAKEcEs6gogWAWEuWYYBYPZUgwC4ZyQjCLhHJKMAuD8hPBLAbKe4JZAJQxwTGDE4ITBrOF5iucEczEXOYEMyWXnwlmMi4LgpmGS0Gw4JtKsOxeEy7dkmCm23JGMBNt+UAwU2z5SDCTa/mFYKvVPfwIjoW+/rRXdOgWjOg6z2Uwyus8mcHIr/NsBqPBztMZjBA7z2cwauw8ocFIsvOMBqPLzlMauSfPaTAK7Typwci086wGo9X2aW25xOUSzj15EoORbudZDEa/nacxGBF3nsdglNx5IoORc+eZDEbTnacyGGF3nstg1N15MoOReOfZDEbnnaczGLF3ns9gFP/0CY2xUERBU6EkWxQfWxQ2yTbB2wzeIXiHwbsE7zJ4j+A9Bu8TvM/gA4IPGHxI8CGDjwg+YvBbgt8y+JjgYwb3CO4x+ITgEwafEnzK4DOCzxh8TvA5gy8IvmDwJcGXDL4i+IrBfYL7DL4m+JrBNwTfMPiW4FsG3xF8x+B3BL9j8HuC3z99vLqiA6M6ptEtpl8tPcZtc27H5XY4t+tyu5zbc7k9zu273D7nDlzugHOHLnfIuSOXO+LcW5d7y7ljlzvmXM/lepw7cbkTzp263CnnzlzujHPnLnfOuQuXu+Dcpctdcu7K5a4413e5PueuXe6aczcud8O5W5e75dydy91x7p3LvePce5ezsr/hJUT5BfTnCPzsut70LbMUKvt51mLJ1ECDhJJGUxMr3K2HyxpmyNAgVIfoKgQRqj507YEI1RxlPRKqNHSdgQjVF7q6QISqCl1TIEK1hK4kEKEKQtcPiFDdoKsGRKha0LUCIjFbB4NQZaDrAkRStn4GoSpA1wCIUO7XmR8Ryvg63yNCeV5neUQEW3CDUE4v621hm1IahPK3zt6IUNbWORsRytU6UyNCGVrnZ0S6qlG3DC39OJ+o/da/GwWWw1ocWhcWpI9a9GSipmI/GY5UD3NBRJZAqHD9m2AtSSVHC6BDRPB/gkQUJqqr/k2wFW4t2mYiVcXHXymxWgvFGpCFQh2xSVVKoNZCgY7JQnGGZKEwJ2ThcNlYUZCfyEIx3rO1qZQIm5lXSoDWwsVkq4jiy9iSVEp01kLRfSYLBVewlaqU0JoFqpTIrIULzZYZBVaSheKakYXCeiALRfVIFgrqy7z+zgvz7IPBdY5FnVFu1ZkVEcqoOp8iQnlUZ1FEKHvq3IkI5UydMRGhTKnzJCKUH3V2RISyos6JiFAu1JkQEcqAOv8hQnlPZz1EKNvpXIcI5Tid4RChzKbzGiKUz3Q2Q4SymM5hiFDu0pkLEcpYOl8hQnlKZylEKDvp3IQI5SSdkRChTKTzECKUf3T2QYSyjs45iFCu0ZkGkfdsBykvDHlaSM4n9UE8wCu2ejb0FdOrw7+ZXB3DirsycaxV1IdUqO/ydyGI/QJQVJMtdQLhHU2xJ8aRelQKaZCNojREZ/40VogYN9fJvBLqKe8VyKccDLN49ENuhg9zDML2k9pU6O8ITd6s/emn1PXUpKkvU8HUL7ctRvqXOxajCJC7FqMYkHsWoyiQ+xajOJAHFqNIkIcWo1iQRxajaJBvLUbxII8tRhEhexajmJAnFqOokKcWo7iQZxajyJDnFqPYkBcWo+iQlxaj+JBXFqMIkX2LUYzIa4tRlMgbi1GcyFuLUaTIO4tRrMh3FqNoke8tZioyFPKBeinBsKH9nBs4HzfCbQaTLsIdBpM0wl0GkzrCPQaTQMJ9BpNGwgMGk0zCQwaTUsIjBpNYwrcMJr2ExwwmyYQ9BpNqwhMGk3DCUwaTdsIzBpN8wnMGk4LCCwaTiMJLBpOOwisGk5TCPoNJTeE1g0lQ4Q2DSVPhLYNJVuEdg0lZ4TsGk7jC9wy2FT8ebXWpJpqnKEMmLrFNKGlL7BBK0hK7hGplvfJ29TcZUwGe7wmQHt46hpG3t+oNIfAVLieR8GbZNB4hhBZ4Qn/vgbXktPDU6zRZjI7U+y7wkGNtqb/MtV+p79MdSZ3igFASpzgklLQpjgglaYq3hJIyxTGhJEzRI5R0KU4IJVmKU0JJleKMUBKlOCeUNCkuCCVJiktCSZHiilASpOgTSnoU14SSHMUNoaRGcUsoiVHcEUpaFO8IJSmK94Q2j1xSrPtAf4TwzcOWuggEqgB6bvGvysMtslCq22ShRHfIQmnukoWH3R5ZKKJ9slA8B2ShaA7JQrEckYUieUsWiuOYLBRFjywUwwlZKIJTsnDzz8jCTT8nCzf7gizc5EuycHOvyMJN7ZOFm3lNFm7iDVm4ebdk4abdkYWb9Y4s3KT37H51pVVXWWrLgG+ZNBUXHikqfvX7lBjEBl31ZpGcZFPpYbnjqZf7cijcggioInKqofr2stGAbrhQCIIul6BVL4EumKBVMYEumaBVM4EumqBVNYEum6BVN4EunKBVOYEunaBVO4EunqBVPYEun6BVP4EuoKBVQYEuoaBVQ4EuoqBVRYEuo6BVR4EupKBVSYEupaBVS4EupqBVTYEup6BVT4EuqKBVUYEuqaBVU4EuqqBVVYEuq6BVV4EurKBVWYEuraBVW4EurqBVXYEur6BVX4EusIBVWPhJAVOOLKbgTdMRFPGjemlp5EvfCyGFArONsiOBSh9OVepxZZurpvMq/1gNiqTShk58yiskeVREmPKc/s27g8NHne70ayDqJpgfW77tGyITX+IndfcWTstz3vJ83jWYJBtB/NxEdINmJsZauE/d6Py5RrmM4hHULQfaaEbf9MBjQmbBxBfq1Wd/KjP9CQoKZ4StF1hz06YZY91lcQAjcNoZs6NdgQQeOradMVELgX6G5jaO/Tz2A5g3b9T0akC9bV1fu8vr9t+bNxlvrz2QnmAv7fTa7OWc5/bWqZnkbI1bZFzM7XM3lyggnDdP0tpUIGmOyorGERRt1yIby8R/oJYWaLfDZJHpl5jMQ7ZFL3k8VbP/op4EuOxxb87fYDruLWzgjV/QCJTR9i/xl1/g3hcZa3m1sAE7WUm0MpRAb7N4XPiJeiA1mWUFFqjCfxTey973b16qt3f0nw1MU/MWqshx/4V+e+zlAOKYtbEPRF9525gAMeRT9d8jxjsk6i02VQUbp6y1eoU0m4Y6Z+qiOJKwqt2LzBtloNzNovsoh1Hkr7VeQc6KJFYP7+dV7/v1eQeZpaC4jS5OznS/N11crpi8g9Fa6H0/iNKxfGyHTu4X6uEwHhu+CpYrwLNW+CF4UeqlWV3QS3hY83YmmVDLk6kCMJh4u/jZN4VfC0/9FcTasvM45yxXp3NW/AtqvAj1APD3YFVdPddQnZOmIV51u9RqxWb6/yda9FFQffWGXwxy4A8xzuJsNizAv2/NXk7zGExhE/tpGIN5QVBf4v0/frWy0f77lsWLmzdrG5tr/3qxufK77fpvX3669Mulf176emlj6d+Wfrd0uHS+dL0UvBi++MOL/3rx35v9zS+b/7H5n6bpj17Uff5pyfnZ/OP/AsWXYDg=</latexit>
Figure 1: We partition datasets into four sections, defined by the features (spurious and/or target) that hold. We sample training datasets D, which provide varying amounts of evidence against the spurious feature, in the form of s-only examples. In the illustration above, the s-only rate is 210 = 0.2, i.e., 20% of examples in D provide evidence that s alone should not be used to predict y.
Evidence from Spurious-Only Examples. We are interested in spurious features which are highly correlated with the target during training. Given a training sample D and features s and t, we define the s-only example rate as the evidence against the use of s as a predictor of y. Concretely, s-only rate = |Ds-only| / |D|, the proportion of training examples in which s occurs without t (and y = 0).
Use of Spurious Feature. If a model has falsely learned that the spurious feature s alone is predictive of the label, it will have a high error rate when classifying examples for which s holds but t does not. We define the s-only error to be the classifier’s error on examples from Ss-only. When relevant, t-only error, both error, and neither error are defined analogously. In this work, “feature use” is a model’s predictions consistency with that feature; we are not making a causal argument.
Extractability of a Feature. We want to compare features in terms of how extractable they are given a representation. For example, given a sentence embedding, it may be possible to predict multiple features with high accuracy, e.g., whether the word “dog” occurs, and also whether the word “dog” occurs as the subject of the verb “run”. However, detecting the former will no doubt be an easier task than detecting the latter. We use the prequential minimum description length (MDL) Rissanen (1978)–first used by Voita & Titov (2020) for probing–to quantify this intuitive difference.2 MDL is an information-theoretic metric that measures how accurately a feature can be decoded and the amount of effort required to decode it. Formally, MDL measures the number of bits required to communicate the labels given the representations. Conceptually, MDL can be understood as a measure of the area under the loss curve: If a feature is highly extractable, a model trained to detect that feature will converge quickly to high accuracy, resulting in a low MDL. Computing MDL requires repeatedly training a model over a dataset labeled by the feature in question. To compute MDL(s), we train a classifier (without freezing any parameters) to differentiate Ss-only vs. Sneither, and similarly compute MDL(t). See Voita & Titov (2020) for additional details on MDL.3
1Without loss of generality, we define t in our datasets s.t. t(x) = y,∀x, y ∈ S. We do this to iron out the case where t outputs the opposite value of y.
2We observe similar overall trends when using an alternative metric based on validation loss (Appendix A.3).
3Note that our reported MDL is higher in some cases than that given by the uniform code (the number of sentences that are being encoded). The MDL is computed as a sum of the costs of transmitting successively

heading: 2.3 HYPOTHESIS
text: Stated using the above-defined terminology, our hypothesis is that a model’s use of the target feature is modulated by two factors: The relative extractability of the target feature t (compared to the spurious feature s), and the evidence from s-only examples provided by the training data. In particular, we expect that higher extractability of t (relative to s), measured by MDL(s)/MDL(t), will yield models that achieve better performance despite less training evidence.

heading: 3 EXPERIMENTS WITH SYNTHETIC DATA
text: Since it is often difficult to fully decouple the target feature from competing spurious features in practice, we first use synthetic data in order to test our hypothesis in a clean setting. We use a simple classifier with an embedding layer, a 1-layer LSTM, and an MLP with 1 hidden layer with tanh activation. We use a synthetic sentence classification task with k-length sequences of numbers as input and binary labels as output. We use a symbolic vocabulary V with the integers 0 . . . |V | − 1. We fix k = 10 and |V | = 50K. We begin with an initial training set of 200K, evenly split between examples from Sboth and Sneither. Then, varied across runs, we manipulate the evidence against the spurious feature (i.e., the s-only rate) by replacing a percentage p of the initial data with examples from Ss-only for p ∈ {0%, 0.1%, 1%, 5%, 10%, 20%, 50%}. Test and validation sets consist of 1,000 examples each from Sboth, Sneither, St-only, Ss-only. In all experiments, we set the spurious feature s to be the presence of the symbol 2. We consider several different target features t (Table 1), intended to vary in their extractability. Table 1 contains MDL metrics for each feature (computed on training sets of 200K, averaged over 3 random seeds). We see some gradation of feature extractability, but having more features with wider variation would help solidify our results.4
Figure 2 shows model performance as a function of s-only rate for each of the four features described above. Here, performance is reported using error rate (lower is better) on each partition (Ss-only, St-only, Sboth, Sneither) separately. We are primarily interested in whether the relative extractability of the target feature (compared to the spurious feature) predicts model performance. We indeed see a fairly clear relationship between the relative extractability (MDL(s) / MDL(t)) and model performance, at every level of training evidence (s-only rate). For example, when t is no less extractable than s (i.e., contains-1), the model achieves zero error at an s-only rate of 0.001, meaning it learns that t alone predicts the label despite having only a handful of examples that support this inference. In contrast, when t is harder to extract than s (e.g., first-last), the model fails to make this inference, even when a large portion of training examples provide evidence supporting it.

heading: 4 EXPERIMENTS WITH NATURALISTIC DATA
text: We investigate whether the same trend holds for language models fine-tuned with naturalistic data, e.g., grammar-generated English sentences. To do this, we fine-tune models for the linguistic acceptability task, a simple sequence classification task as defined in Warstadt & Bowman (2019),
longer blocks, using classifiers that are trained on previously transmitted data. The high MDL’s are a result of overfitting by classifiers that are trained on limited data–and therefore, the classifiers have worse compression performance than the uniform baseline.
4Note, all models are ultimately able to learn to detect t (achieve high test accuracy) on the both partition, but not on the t-only partition.
in which the goal is to differentiate grammatical sentences from ungrammatical ones. We focus on acceptability judgments since formal linguistic theory guides how we define the target features, and recent work in computational linguistics shows that neural language models can be sensitive to spurious features in this task (Marvin & Linzen, 2018; Warstadt et al., 2020a).

heading: 4.1 DATA
text: We design a series of simple natural language grammars that generate a variety of feature pairs (s, t), which we expect will exhibit different levels of relative extractability (MDL(s) / MDL(t)). We focus on three syntactic phenomena (described below). In each case, we consider the target feature t to be whether a given instance of the phenomenon obeys the expected syntactic rules. We then introduce several spurious features s which we deliberately correlate with the positive label during fine-tuning. The Subject-Verb Agreement (SVA) construction requires detecting whether the verb agrees in number with its subject, e.g., “the girls are playing” is acceptable while “the girls is playing” is not. In general, recognizing agreement requires some representation of hierarchical syntax, since subjects may be separated from their verbs by arbitrarily long clauses. We introduce four spurious features: 1) lexical, grammatical sentences begin with specific lexical items (e.g., “often”); 2) length, grammatical sentences are longer; 3) recent-noun, verbs in grammatical sentences agree with the immediately preceding noun (in addition to their subject); and 4) plural, verbs in grammatical sentences are preceded by singular nouns as opposed to plural ones.
The Negative Polarity Items (NPI) construction requires detecting whether a negative polarity item (e.g., “any”, “ever”) is grammatical in a given context, e.g., “no girl ever played” is acceptable while “a girl ever played” is not. In general, NPIs are only licensed in contexts that fall within the scope of a downward entailing operator (such as negation). We again consider four types of spurious features: 1) lexical, in which grammatical sentences always include one of a set of lexical items (“no” and “not”); 2) length (as above); 3) plural, in which each noun in a grammatical sentence is singular, as opposed to plural; and 4) tense, in which grammatical sentences are in present tense.
Some verbs (e.g. “recognize”) require a direct object. However, in the right syntactic contexts (i.e., when in the correct syntactic relation with a wh-word), the object position can be empty, creating what is known as a “gap”. E.g., “I know what you recognized ” is acceptable while “I know that you recognized ” is not. The Filler-Gap Dependencies (GAP) construction requires detecting
whether a sentence containing a gap is grammatical. For our GAP tasks, we again consider four spurious features (lexical, length, plural, and tense), defined similarly to above.
The templates above (and slight variants) result in 20 distinct fine-tuning datasets, over which we perform our analyses (see Appendix for details). Table 2 shows several examples. For the purposes of this paper, we are interested only in the relative extractability of t vs. s given the pre-trained representation; we don’t intend to make general claims about the linguistic phenomena per se. Thus, we do not focus on the details of the features themselves, but rather consider each template as generating one data point, i.e., an (s, t) pair representing a particular level of relative extractability.

heading: 4.2 SETUP
text: We evaluate T5, BERT, RoBERTa, GPT-2 and an LSTM with GloVe embeddings (Raffel et al., 2020; Devlin et al., 2019; Liu et al., 2019b; Radford et al., 2019; Pennington et al., 2014).5 Both T5 and BERT learn to perform well over the whole test set, whereas the GloVe model struggles with many of the tasks. We expect that this is because contextualized pre-training encodes certain syntactic features which let the models better leverage small training sets (Warstadt & Bowman, 2020). Again, we begin with an initial training set of 2000 examples, evenly split between both and neither, and then introduce s-only examples at rates of 0%, 0.1%, 1%, 5%, 10%, 20%, and 50%, using three random seeds each. Test and validation sets consist of 1000 examples each from Sboth, Sneither, Ss-only. In the natural language setting, it is often difficult to generate t-only examples, and thus we cannot compute extractability of the target feature t by training a classifier to distinguish St-only from a random subset of Sneither, as we did in Section 3. Therefore, we estimate MDL by training a classifier to distinguish between examples from Ss-only and examples from Sboth. Using the simulated data from Section 3, we confirm that both methods (Ss-only vs. Sboth and St-only vs. Sneither) produce similar estimates of MDL(t) (see Appendix). Per model, we filter out feature pairs for which the model could not achieve at least 90% accuracy on each probing task in isolation.6

heading: 4.3 RESULTS
text: For each (s, t) feature pair, we plot the use of the spurious feature (s-only error) as a function of the evidence against the spurious feature seen in training (s-only example rate).7 We expect to see the same trend we observed in our synthetic data, i.e., the more extractable the target feature t is relative to the spurious feature s, the less evidence the model will require before preferring t over s. To quantify this trend, we compute correlations between 1) the relative extractability of t compared to s and 2) the test F-score averaged across all rates and partitions of the data8, capturing how readily the model uses (i.e., makes predictions consistent with the use of) the target feature.
5In pilot studies, we found that standard BOW and CNN-based models were unable solve the tasks. 6This control does not impact results: Appendix A.1. 7See Appendix for both error and neither error; both are stable and low in general. 8Initially, we used a more complicated metric based on the s example rate required for the model to solve
the test set. Both report similar trends and correlations. For posterity, we include details in the Appendix A.2.
Figure 3 shows these correlations and associated scatter plots. We can see that relative extractability is strongly correlated with average test F-score (Figure 3a), showing high correlations for both BERT (ρ = 0.79) and T5 (ρ = 0.57). That is, the more extractable t is relative to s, the less evidence the model requires before preferring t, performing better across all partitions. This relationship holds regardless of whether relative extractability is computed using a ratio of MDL scores or an absolute difference. We also see that, in most cases, the relative extractability explains the model’s behavior better than does the extractability of s or t alone. For GloVe there is little variation in model behavior: For most of the 11/20 pairs on which the model is able to learn the task, it requires an s-only example rate of 0.5. Thus, the correlations are weak, but qualitative results appear steady (Figure 8 in Appendix A), following the pattern that when s is easier to extract than t, more evidence is required to stop using s.
Figure 4 shows the performance curves for BERT and T5 (with others the in Appendix A), i.e., use of the spurious feature (s-only error) as a function of the evidence from s-only examples seen in training (s-only example rate). Each line corresponds to a different s, t feature pair, and each data point is the test performance on a dataset with a given s-only example rate (which varies along the x-axis.) For pairs with high MDL ratios (i.e., when t is actually easier to extract than s), the model learns to solve the task “the right way” even when the training data provides no incentive to do so: That is, in such cases, the models’ decisions do not appear to depend on the spurious feature s even when s and the target feature t perfectly co-occur in the fine-tuning data.
Figure 4 shows that T5 (compared to BERT) requires more data to perform well. This may be because we fine-tuned T5 with a linear classification head, rather than the text-only output on which it was pre-trained. We made this decision 1) because we had trouble training T5 in the original manner, and 2) using a linear classification head was consistent with the other model architectures.

heading: 5 DISCUSSION
text: Our experimental results provide support for our hypothesis: The relative extractability of features given an input representation (as measured by information-theoretic probing tech-
niques) is predictive of the decisions a trained model will make in practice. In particular, we see evidence that models will tend to use imperfect features that are more readily extractable over perfectly predictive features that are harder to extract. This insight is highly related to prior work which has shown, e.g., that neural networks learn “easy” examples before they learn “hard” examples (Mangalam & Prabhu, 2019). Our findings additionally connect to new probing techniques which have received significant attention in NLP but have yet to be connected to explanations of or predictions about state-of-the-art models’ decisions in practice.
Fine-tuning may not uncover new features. The models are capable of learning both the s and t features in isolation, so our experiments show that if the relative extractibility is highly skewed, one feature may hide the other – a fine-tuned model may not use the harder-to-extract feature. This suggests a pattern that seems intuitive but is in fact non-trivial: If one classifier does not pick up on a feature readily enough, another classifier (or, rather, the same classifier trained with different data) may not be sensitive to that feature at all. This has ramifications for how we view fine-tuning, which is generally considered to be beneficial because it allows models to learn new, task-relevant features. Our findings suggest that if the needed feature is not already extractable-enough after pretraining, fine-tuning may not have the desired effect.
Probing classifiers can be viewed as measures of a pre-trained representation’s inductive biases. Analysis with probing classifiers has primarily focused on whether important linguistic features can be decoded from representations at better-than-baseline rates, but there has been little insight about what it would mean for a representations’ encoding of a feature to be “sufficient”. Based on these experiments, we argue that a feature is “sufficiently” encoded if it is as available to the model as are surface features of the text. For example, if a fine-tuned model can access features about a word’s semantic role as easily as it can access features about that word’s lexical identity, the model may need little (or no) explicit training signal to prefer a decision rule based on the former structural feature. The desire for models with such behavior motivates the development of architectures with explicit inductive biases (e.g., TreeRNNs). Evidence that similar generalization behavior
can result from pre-trained representations has exciting implications for those interested in sample efficiency and cognitively-plausible language learning (Warstadt & Bowman, 2020; Linzen, 2020). We note that this work has not established that the relationship between extractability and feature use is causal. This could be explored using intermediate task training (Pruksachatkun et al., 2020) in order to influence the extractability of features prior to fine-tuning for the target task; e.g., Merchant et al. (2020) suggests fine-tuning on parsing might improve the extractability of syntactic features.

heading: 6 RELATED WORK
text: Significant prior work analyzes the representations and behavior of pre-trained LMs. Work using probing classifiers (Veldhoen et al., 2016; Adi et al., 2017; Conneau et al., 2018; Hupkes et al., 2018) suggests that such models capture a wide range of relevant linguistic phenomena (Hewitt & Manning, 2019; Bau et al., 2019; Dalvi et al., 2019; Tenney et al., 2019a;b). Similar techniques include attention maps/visualizations (Voita et al., 2019; Serrano & Smith, 2019), and relational similarity analyses (Chrupała & Alishahi, 2019). A parallel line of work uses challenge sets to understand model behavior in practice. Some works construct evaluation sets to analyze weaknesses in the decision procedures of neural NLP models (Jia & Liang, 2017b; Glockner et al., 2018; Dasgupta et al., 2018; Gururangan et al., 2018; Poliak et al., 2018b; Elkahky et al., 2018; Ettinger et al., 2016; Linzen et al., 2016; Isabelle et al., 2017; Naik et al., 2018; Jia & Liang, 2017a; Linzen et al., 2016; Goldberg, 2019, and others). Others use such datasets to improve models’ handling of linguistic features (Min et al., 2020; Poliak et al., 2018a; Liu et al., 2019a), or to mitigate biases (Zmigrod et al., 2019; Zhao et al., 2018; 2019; Hall Maudslay et al., 2019; Lu et al., 2020). Nie et al. (2020) and Kaushik et al. (2020) explore augmenting training sets with human-in-the-loop methods.
Our work is related to work on generalization of neural NLP models. Geiger et al. (2019) discusses ways in which evaluation tasks should be sensitive to models’ inductive biases and Warstadt & Bowman (2020) discusses the ability of language model pre-training to encode such inductive biases. Work on data augmentation (Elkahky et al., 2018; Min et al., 2020; Zmigrod et al., 2019) is relevant, as the approach relies on the assumption that altering the training data distribution (analogous to what we call s-only rate in our work) will improve model behavior in practice. Kodner & Gupta (2020); Jha et al. (2020) discuss concerns about ways in which such approaches can be counterproductive, by introducing new artifacts. Work on adversarial robustness (Ribeiro et al., 2018; Iyyer et al., 2018; Hsieh et al., 2019; Jia et al., 2019; Alzantot et al., 2018; Hsieh et al., 2019; Ilyas et al., 2019; Madry et al., 2017; Athalye et al., 2018) is also relevant, as it relates to the influence of dataset artifacts on models’ decisions. A still larger body of work studies feature representation and generalization in neural networks outside of NLP. Mangalam & Prabhu (2019) show that neural networks learn “easy” examples (as defined by shallow machine learning model performance) before they learn “hard” examples. Zhang et al. (2016) and Arpit et al. (2017) show that neural networks which are capable of memorizing noise nonetheless acheive good generalization performance, suggesting that such models might have an inherent preference to learn more general features. Finally, ongoing theoretical work characterizes the ability of over-parameterized networks to generalize in terms of complexity (Neyshabur et al., 2019) and implicit regularization (Blanc et al., 2020).
Concurrent work (Warstadt et al., 2020b) also investigates the inductive biases of large pre-trained models (RoBERTa), in particular, they ask when (at what amount of pre-training data) such models shift from a surface feature (what we call spurious features) to a linguistic feature (what we call a target feature). In our work, we focus on how to predict which of these two biases characterize the model (via relative MDL).

heading: 7 CONCLUSION
text: This work bears on an open question in NLP, namely, the question of how models’ internal representations (as measured by probing classifiers) influence model behavior (as measured by challenge sets). We find that the feature extractability can be viewed as an inductive bias: the more extractable a feature is after pre-training, the less statistical evidence is required in order for the model to adopt the feature during fine-tuning. Understanding the connection between these two measurement techniques can enable more principled evaluation of and control over neural NLP models.

heading: ACKNOWLEDGEMENTS
text: We would like to thank Michael Littman for helpful suggestions on how to better present our findings and Ian Tenney for insightful comments on a previous draft of this work. We also want to thank our reviewers were their detailed and helpful comments. This work is supported by DARPA under grant number HR00111990064. This research was conducted using computational resources and services at the Center for Computation and Visualization, Brown University.

heading: A ADDITIONAL RESULTS
text: Figure 6, 7, 8, 9, 10 show additional results for all models over all partitions (both accuracy, neither accuracy, and F-score). These charts appear at the end of the Appendix.
Details on the MDL statistics are available in Table 3.
A.1 BEYOND ACCURACY?
For the transformer models, for 18/20 feature pairs, the models are able to solve all the spurious and target features in isolation during probing. (They do solve the test set in all cases–its that two
of the spurious features ended up being very difficult for the models.) During the reviews, we did not control for the cases where the model did not solve the probing task. These 2 extraneous points accentuate the lineplot curves, but do not change the character of the results (nor much adjust the correlations). In the paper, now, we control for accuracy by filtering out these cases. With or without this control, the accuracy provides no predictive power about the inductive biases. We present the correlations without filtering for these cases for consistency with the reviews (Table 4 above); we believe it is important to control for these cases because they could have acted as giveaways, where even accuracy might have worked.
A.2 ALTERNATE METRIC: s-RATE?
We initially used a different metric when computing the correlations to compact the lineplots. Rather than using the average test performance, we looked at the evidence required for the model to solve the test set. Both of these metrics conceptually capture what we are interested in, but the new one (simply averaging test performance) is much easier to understand, and captures the performance across all partitions. Here we report the correlations with this evidence required metric instead, which we called s-rate?. Specifically, we defined it to be: s-rate? is the lowest s-only example rate at which the fine-tuned model achieves essentially perfect performance (F-score> 0.99) (see Figure 5a). Intuitively, s-rate? is the (observed) minimum amount of evidence from which the model infer that t alone is predictive of the label. See Table 5b for the results.
s-rate★ Threshold: 0.99
(a) Evidence Required: s-rate?
Absolute Relative (t to s) Target Spurious Ratio Diff
BERT 0.68* -0.63* -0.79* -0.81* RoBERTa 0.04 -0.69* -0.71* -0.69* T5 0.81* -0.03 -0.55* -0.65* GPT2 -0.11 -0.24 -0.29 -0.32 GloVe 0.29 -0.38 -0.48 -0.48
(b) Using Evidence Required (s-rate?) instead of Average Fscore. The correlations are negative instead of positive: As the extractibility increases, less evidence is required for the model to perform well.
A.3 RESULTS USING AUC INSTEAD OF MDL
See Table 5. A metric similar to MDL for capturing the same intuition is the area under the validation loss curve (AUC). This metric is highly related to online MDL in computation.
B IMPLEMENTATION DETAILS & REPRODUCIBLITY
Our code is available at: https://github.com/cjlovering/predicting-inductive-biases.
There are two major parts to this project in terms of reproducibility: (1) the data and (2) the model implementations. We describe the templates for the data below in Appendix D – the full details are in the project source. For the transformer models, we use Hugging Face for the implementations and access to the pre-trained embeddings (Wolf et al., 2020). We use PyTorch Lightning to organize the training code (Falcon, 2019). We fix all hyperparameters, which are reported in Table 6.
We want to call attention to BERT requiring much less data than T5 to capture our target features. At face value, it seems that BERT requires much less data than T5 to capture our target features. However, we are wary about making such strong claims. Something to consider here (noted in the Appendix B is that for T5 we used a linear model rather than formatting the task in text (which is how T5 is trained). We made this decision (1) because we had trouble training T5 in this purely textual manner, and (2) using a linear classification head over two classes is consistent with the other model architectures. Again, GPT2 and RoBERTa performed on par with BERT, so the difference between the performance of BERT and T5 may be due to how we trained T5.

heading: C MEASURING EXTRACTABILITY INDIRECTLY
text: We measure the MDL for t with both and s-only examples. In the simulated setting we can compare this approach with measuring the MDL directly (t-only vs neither). See Table 7 for MDL results. The ordering of the feature’s difficulty holds across the two methods.

heading: D TEMPLATES FOR NATURALISTIC DATA
text: Each template corresponds to a combination of target features, grammars, and spurious features (the target and spurious features are discussed in Section 4.1). See Table 8 for a complete list of templates. See Table 10 for further details about the templates that are used for each of the the target features. Complete details about implementation of these templates (and all data) will be released upon acceptance.

heading: E WHY EXACTLY IS IT HARD TO GENERATE T-ONLY EXAMPLES?
text: Target features may be unavoidably linked to spurious ones. For example, for a Negative Polarity Item to be licensed (perhaps smoothing over some intricacies) the NPI (“any”, “all”, etc) must be a downward entailing context. These downward entailing contexts are created by triggers, e.g., if a negative word like “no” or “not” or a quantifier like “some”. Linguists who study the problem have assembled a list of such triggers (see Hoeksema (2008)). Arguably, one cannot write down a correct example of NLP licensing that doesn’t contain one of these memorizable triggers. Thus, we cannot train or test models on correct examples of NPI usage while simultaneously preventing it from having access to trigger-specific features.
Similar to the NPI example, it’s not possible (to our knowledge) to construct target-only examples for filler-gap since construction requires a wh-word and syntactic gap; thus, we can’t create a positively labeled, grammatical sentence that exhibits a Filler Gap without these elements.
In summary, target-only examples may add new spurious features (as with NPI), or be impossible to construct because the presence of the target feature implies the presence of the spurious feature (as with filler gaps). Still, our setup permits the MDL to be computed directly with target-only examples, and so, in cases where it is feasible to create target-only examples (e.g. the Subject-Verb Agreement templates), it would have bolstered our argument to do so.

heading: F MDL ISSUES: OVERFITTING IN THE SYNTHETIC EXPERIMENTS
text: We found that the MDL exceeds the uniform code length in some of the synthetic experiments. We found that this occurs because the model overfits on the small early-block sizes. See Figure 11.

Review summaries:
The paper aims to bridge the gap between model interpretation using probing and model's use of spurious features. They show that the findings of MDL with respect to a feature correlate with the extractability of the feature, given the evidence of representing the feature is available in the training data. The results are presented using both synthetic and natural language data.

This paper studies the relationship between extractability of features from pre-trained representations and how much a fine-tuned model uses that feature. The extractability of features is measured by the minimum description length of a probing classifier trained to detect the feature from the pre-trained representations (using the online code version of Voita and Titov). The degree to which a fine-tuned model uses the feature is measured by the amount of evidence required for a model to tease apart spurious from non-spurious features (called "target" features). Evidence here means examples where a spurious feature occurs but a non-spurious feature does not occur. When there are many such examples (high spurious-only rate), it is easier for a model to reject the spurious feature and learn to rely on the target feature. The "degree to which a fine-tuned model uses a feature" is defined as the minimal spurious-only rate at which the model can accomplish the task.