Predicting Inductive Biases of Pre-Trained Models

1 INTRODUCTION . Large pre-trained language models ( LMs ) ( Devlin et al. , 2019 ; Raffel et al. , 2020 ; Brown et al. , 2020 ) have demonstrated impressive empirical success on a range of benchmark NLP tasks . However , analyses have shown that such models are easily fooled when tested on distributions that differ from those they were trained on , suggesting they are often “ right for the wrong reasons ” ( McCoy et al. , 2019 ) . Recent research which attempts to understand why such models behave in this way has primarily made use of two analysis techniques : probing classifiers ( Adi et al. , 2017 ; Hupkes et al. , 2018 ) , which measure whether or not a given feature is encoded by a representation , and challenge sets ( Cooper et al. , 1996 ; Linzen et al. , 2016 ; Rudinger et al. , 2018 ) , which measure whether model behavior in practice is consistent with use of a given feature . The results obtained via these two techniques currently suggest different conclusions about how well pre-trained representations encode language . Work based on probing classifiers has consistently found evidence that models contain rich information about syntactic structure ( Hewitt & Manning , 2019 ; Bau et al. , 2019 ; Tenney et al. , 2019a ) , while work using challenge sets has frequently revealed that models built on top of these representations do not behave as though they have access to such rich features , rather they fail in trivial ways ( Dasgupta et al. , 2018 ; Glockner et al. , 2018 ; Naik et al. , 2018 ) . In this work , we attempt to link these two contrasting views of feature representations . We assume the standard recipe in NLP , in which linguistic representations are first derived from large-scale selfsupervised pre-training intended to encode broadly-useful linguistic features , and then are adapted for a task of interest via transfer learning , or fine-tuning , on a task-specific dataset . We test the hypothesis that the extent to which a fine-tuned model uses a given feature can be explained as a function of two metrics : The extractability of the feature after pre-training ( as measured by probing classifiers ) and the evidence available during fine-tuning ( defined as the rate of co-occurrence with the label ) . We first show results on a synthetic task , and second using state-of-the-art pre-trained LMs on language data . Our results suggest that probing classifiers can be viewed as a measure of the pre-trained representation ’ s inductive biases : The more extractable a feature is after pre-training , the less statistical evidence is required in order for the model to adopt the feature during fine-tuning . Contribution . This work establishes a relationship between two widely-used techniques for analyzing LMs . Currently , the question of how models ’ internal representations ( measured by probing classifiers ) influence model behavior ( measured by challenge sets ) remains open ( Belinkov & Glass , 2019 ; Belinkov et al. , 2020 ) . Understanding the connection between these two measurement techniques can enable more principled evaluation of and control over neural NLP models . 2 SETUP AND TERMINOLOGY . 2.1 FORMULATION . Our motivation comes from McCoy et al . ( 2019 ) , which demonstrated that , when fine-tuned on a natural language inference task ( Williams et al. , 2018 , MNLI ) , a model based on a state-of-the-art pre-trained LM ( Devlin et al. , 2019 , BERT ) categorically fails on test examples which defy the expectation of a “ lexical overlap heuristic ” . For example , the model assumes that the sentence “ the lawyer followed the judge ” entails “ the judge followed the lawyer ” purely because all the words in the latter appear in the former . While this heuristic is statistically favorable given the model ’ s training data , it is not infallible . Specifically , McCoy et al . ( 2019 ) report that 90 % of the training examples containing lexical overlap had the label “ entailment ” , but the remaining 10 % did not . Moreover , the results of recent studies based on probing classifiers suggest that more robust features are extractable with high reliability from BERT representations . For example , given the example “ the lawyer followed the judge ” / “ the judge followed the lawyer ” , if the model can represent that “ lawyer ” is the agent of “ follow ” in the first sentence , but is the patient in the second , then the model should conclude that the sentences have different meanings . Such semantic role information can be recovered at > 90 % accuracy from BERT embeddings ( Tenney et al. , 2019b ) . Thus , the question is : Why would a model prefer a weak feature over a stronger one , if both features are extractable from the model ’ s representations and justified by the model ’ s training data ? Abstracting over details , we distill the basic NLP task setting described above into the following , to be formalized in the Section 2.2 . We assume a binary sequence classification task where a target feature t perfectly predicts the label ( e.g. , the label is 1 iff t holds ) . Here , t represents features which actually determine the label by definition , e.g. , whether one sentence semantically entails another . Additionally , there exists a spurious feature s that frequently co-occurs with t in training but is not guaranteed to generalize outside of the training set . Here , s ( often called a “ heuristic ” or “ bias ” elsewhere in the literature ) corresponds to features like lexical overlap , which are predictive of the label in some datasets but are not guaranteed to generalize . Assumptions . In this work , we assume there is a single t and a single s ; in practice there may be many s features . Still , our definition of a feature accommodates multiple spurious or target features . In fact , some of our spurious features already encompass multiple features : the lexical feature , for example , is a combination of several individual-word features because it holds if one of a set of words is in the sentence . This type of spurious feature is common in real datasets : E.g. , the hypothesis-only baseline in NLI is a disjunction of lexical features ( with semantically unrelated words like “ no ” , “ sleeping ” , etc . ) ( Poliak et al. , 2018b ; Gururangan et al. , 2018 ) . We assume that s and t frequently co-occur , but that only s occurs in isolation . This assumption reflects realistic NLP task settings since datasets always contain some heuristics , e.g. , lexical cues , cultural biases , or artifacts from crowdsourcing ( Gururangan et al. , 2018 ) . Thus , our experiments focus on manipulating the occurrence of s alone , but not t alone : This means giving the model evidence against relying on s. This is in line with prior applied work that attempts to influence model behavior by increasing the evidence against s during training ( Elkahky et al. , 2018 ; Zmigrod et al. , 2019 ; Min et al. , 2020 ) . Published as a conference paper at ICLR 2021 2.2 DEFINITIONS . Let X be the set of all sentences and S be the space of all sentence-label pairs ( x , y ) ∈ X × { 0 , 1 } . We use D ⊂ S to denote a particular training sample drawn from S. We define two types of binary features : target ( t ) and spurious ( s ) . Each is a function from sentences x ∈ X to a binary label { 0 , 1 } that indicates whether the feature holds . Target and spurious features . The target feature t is such that there exists some function f : { 0 , 1 } → { 0 , 1 } such that ∀ ( x , y ) ∈ S , f ( t ( x ) ) = y . In other words , the label can always be perfectly predicted given the value of t.1 A feature s is spurious if it is not a target feature . Partitions of S. To facilitate analysis , we partition S in four regions ( Figure 1 ) . We define Ss-only to be the set of examples in which the spurious feature occurs alone ( without the target ) . Similarly , St-only is the set of examples in which the target occurs without the spurious feature . Sboth and Sneither are analogous . For clarity , we sometimes drop the S∗ notation ( e.g. , s-only in place of Ss-only ) . t-only both s-only neither S D ( a ) ( b ) Sboth “ tpx , yq | tpxq “ 1 ^ spxq “ 1u Sneither “ tpx , yq | tpxq “ 0 ^ spxq “ 0u St-only “ tpx , yq | tpxq “ 1 ^ spxq “ 0u Ss-only “ tpx , yq | tpxq “ 0 ^ spxq “ 1u < latexit sha1_base64= '' bSiqwQANSN306jL/eI6aFqdfpQw= '' > AAA00nicnVtbc9zKcaYcx3EYxzlOHv2CCiX7nBTFInWYil9cZd5JcXlfXqSzOiosthcLETdhZrGk4E1VKi9+8Evyg/I//F/8kJ4ZDLoHC/LEYZVE9PfNNOby9XQvFhzmcSTk+vqfXvzor3781z/5m5/+7fLf/ezvf/4PX/3iH29ENi0CuA6yOCvuhr6AOErhWkYyhru8AD8ZxnA7vN9R/G0JhYiytC8fc/iQ+GEajaPAlwh9/OrPHv5cfawGEh5kkVQvh5mcvJzPf/XbQfX1w+rjN4PV3w9W5dcP3/x2YzCDUQieMMbcGwyW271TiOQEim4H69zBercD+fJ1lsaPPzyAJ/qLZ/qvtyaw/PGrlfW1df3jLV5s1BcrS/XP+cdfvPmfwSgLpgmkMoh9Ib7bWM/lh8ovZBTEMF8eTAXkfnDvh/AdXqZ+AuJDpbdp7r1CZOSNswL/pdLTKO9R+YkQj8kQWya+nIg2p8Au7rupHP/mQxWl+VRCGpgbjaexJzNP7bk3igoIZPyIF35QRDhWL5j4hR9IVIZzl2GW3Ut/KNCHt68HWiQ+OvJRT25LNQqZZbFuGqACxfICX4ixmHfO0AHFdDiOwrnnzCnVd4bkQzXF/7s6rHrjOPPlfOG2/vBhNSz8fBIFDy4rs1GWZhJao5phYzWCp2bQuRFtUO3p/2W2z4/w/+N9efmV+vFO9269k63+obe7t390etQ/Oju98jTVOYNV63Z1iOu7/Mo78Yt7T6BS8GQQXjbGbc3NtdJsAWMoiigNlaxGURkJ2wyXblooeaQwC7Ik8dNRNUAwhrGcV9UAEu/rHl5/M58vtAkwkqCwrXa01dWuiMJJ4+xSGV2tZJbbNv0s72qBx5vMEttoW1sL7ep5+7aZ/1SLoW0xfKpFYFsET7UY2RYj1QK34RBnF6sZer6H7VXYwhiP95GHa5O4PvBagfPvNj6gl+HYW9mYa0F4+3pTzK7hoQCrXpzNoHitQnVteYAu9bLCeGWjMhv47wO0Ku2gqzsON5J+vKbPBSHxyFN7L9SOIW887luP+22PmpazzN5z5U19V+HZRh7OqDbe2B6fp/6Iuqx8u7K50G216WOvvuWuNvV0royqn10OVL4ZfB0Cznp0OLALYnpf2d5XHb0vbS99JM+yJsrWmoUxdxd6ZZoYfGJp2g4nBUDbJfO38u2iR1o15vvbRd9+6gFugurcsWTw2czZNnl60o6faZ5DoZOGcbNXu9nrcrPlFf6M1r3l7PXr136ZRSNvKtTRFI29PBMiwnRlXOexj6FT+396dCof5hhJHXNUjOlet/mLJ1k72mkc7fygI5xzipWKOoNNW2F8aLgZEUrF0tbV69dPygRH58dhhvl/knTMEzkzuqbRsxNlrhZmumVdbXW4soK398NJNL6ePwz6TqetH+y0sKg5Hlr1zJn6FGqGq66e2xTTv63e86b/udvfzrS5AY5aXT854FpwEMVKrLG6wAMdG6ir2h/WPFmhaX1leH1ZN0BqmFQb7WwjCwyEeTVQmT/w42q33aD042jEG3w011hVG2q+4BKE7O6gmXkzI8iFynG5iOIsrfPTJbrIEq/0i6guLrW+QfqVqeZNAVi9HCD0cm6Xs2jRPjFDlxkSE7hMQMzIZUbEgMsAMWOXGRMTukxIzMRlJsRELhMR88llPhFz7zL3xMQuE8+1jIvEiwRGLH4wHD2qw87s4Kr3aSqkhyXxr6WnPqqgHB/VyeNsjJfUvlPXd0p3zVwmIyZ3mZyYzy7zmZjCZQpihMsIYqTLSGKmLjMlpnSZkpiZy8yIeXCZB2IeXeaRmC8u82VuyjwbAJiZs+Z4L+sgqUwoDccsbJpx4ydsHSW2hbYZzzgODwlmsVEGBLPAKEcEs6gogWAWEuWYYBYPZUgwC4ZyQjCLhHJKMAuD8hPBLAbKe4JZAJQxwTGDE4ITBrOF5iucEczEXOYEMyWXnwlmMi4LgpmGS0Gw4JtKsOxeEy7dkmCm23JGMBNt+UAwU2z5SDCTa/mFYKvVPfwIjoW+/rRXdOgWjOg6z2Uwyus8mcHIr/NsBqPBztMZjBA7z2cwauw8ocFIsvOMBqPLzlMauSfPaTAK7Typwci086wGo9X2aW25xOUSzj15EoORbudZDEa/nacxGBF3nsdglNx5IoORc+eZDEbTnacyGGF3nstg1N15MoOReOfZDEbnnaczGLF3ns9gFP/0CY2xUERBU6EkWxQfWxQ2yTbB2wzeIXiHwbsE7zJ4j+A9Bu8TvM/gA4IPGHxI8CGDjwg+YvBbgt8y+JjgYwb3CO4x+ITgEwafEnzK4DOCzxh8TvA5gy8IvmDwJcGXDL4i+IrBfYL7DL4m+JrBNwTfMPiW4FsG3xF8x+B3BL9j8HuC3z99vLqiA6M6ptEtpl8tPcZtc27H5XY4t+tyu5zbc7k9zu273D7nDlzugHOHLnfIuSOXO+LcW5d7y7ljlzvmXM/lepw7cbkTzp263CnnzlzujHPnLnfOuQuXu+Dcpctdcu7K5a4413e5PueuXe6aczcud8O5W5e75dydy91x7p3LvePce5ezsr/hJUT5BfTnCPzsut70LbMUKvt51mLJ1ECDhJJGUxMr3K2HyxpmyNAgVIfoKgQRqj507YEI1RxlPRKqNHSdgQjVF7q6QISqCl1TIEK1hK4kEKEKQtcPiFDdoKsGRKha0LUCIjFbB4NQZaDrAkRStn4GoSpA1wCIUO7XmR8Ryvg63yNCeV5neUQEW3CDUE4v621hm1IahPK3zt6IUNbWORsRytU6UyNCGVrnZ0S6qlG3DC39OJ+o/da/GwWWw1ocWhcWpI9a9GSipmI/GY5UD3NBRJZAqHD9m2AtSSVHC6BDRPB/gkQUJqqr/k2wFW4t2mYiVcXHXymxWgvFGpCFQh2xSVVKoNZCgY7JQnGGZKEwJ2ThcNlYUZCfyEIx3rO1qZQIm5lXSoDWwsVkq4jiy9iSVEp01kLRfSYLBVewlaqU0JoFqpTIrIULzZYZBVaSheKakYXCeiALRfVIFgrqy7z+zgvz7IPBdY5FnVFu1ZkVEcqoOp8iQnlUZ1FEKHvq3IkI5UydMRGhTKnzJCKUH3V2RISyos6JiFAu1JkQEcqAOv8hQnlPZz1EKNvpXIcI5Tid4RChzKbzGiKUz3Q2Q4SymM5hiFDu0pkLEcpYOl8hQnlKZylEKDvp3IQI5SSdkRChTKTzECKUf3T2QYSyjs45iFCu0ZkGkfdsBykvDHlaSM4n9UE8wCu2ejb0FdOrw7+ZXB3DirsycaxV1IdUqO/ydyGI/QJQVJMtdQLhHU2xJ8aRelQKaZCNojREZ/40VogYN9fJvBLqKe8VyKccDLN49ENuhg9zDML2k9pU6O8ITd6s/emn1PXUpKkvU8HUL7ctRvqXOxajCJC7FqMYkHsWoyiQ+xajOJAHFqNIkIcWo1iQRxajaJBvLUbxII8tRhEhexajmJAnFqOokKcWo7iQZxajyJDnFqPYkBcWo+iQlxaj+JBXFqMIkX2LUYzIa4tRlMgbi1GcyFuLUaTIO4tRrMh3FqNoke8tZioyFPKBeinBsKH9nBs4HzfCbQaTLsIdBpM0wl0GkzrCPQaTQMJ9BpNGwgMGk0zCQwaTUsIjBpNYwrcMJr2ExwwmyYQ9BpNqwhMGk3DCUwaTdsIzBpN8wnMGk4LCCwaTiMJLBpOOwisGk5TCPoNJTeE1g0lQ4Q2DSVPhLYNJVuEdg0lZ4TsGk7jC9wy2FT8ebXWpJpqnKEMmLrFNKGlL7BBK0hK7hGplvfJ29TcZUwGe7wmQHt46hpG3t+oNIfAVLieR8GbZNB4hhBZ4Qn/vgbXktPDU6zRZjI7U+y7wkGNtqb/MtV+p79MdSZ3igFASpzgklLQpjgglaYq3hJIyxTGhJEzRI5R0KU4IJVmKU0JJleKMUBKlOCeUNCkuCCVJiktCSZHiilASpOgTSnoU14SSHMUNoaRGcUsoiVHcEUpaFO8IJSmK94Q2j1xSrPtAf4TwzcOWuggEqgB6bvGvysMtslCq22ShRHfIQmnukoWH3R5ZKKJ9slA8B2ShaA7JQrEckYUieUsWiuOYLBRFjywUwwlZKIJTsnDzz8jCTT8nCzf7gizc5EuycHOvyMJN7ZOFm3lNFm7iDVm4ebdk4abdkYWb9Y4s3KT37H51pVVXWWrLgG+ZNBUXHikqfvX7lBjEBl31ZpGcZFPpYbnjqZf7cijcggioInKqofr2stGAbrhQCIIul6BVL4EumKBVMYEumaBVM4EumqBVNYEum6BVN4EunKBVOYEunaBVO4EunqBVPYEun6BVP4EuoKBVQYEuoaBVQ4EuoqBVRYEuo6BVR4EupKBVSYEupaBVS4EupqBVTYEup6BVT4EuqKBVUYEuqaBVU4EuqqBVVYEuq6BVV4EurKBVWYEuraBVW4EurqBVXYEur6BVX4EusIBVWPhJAVOOLKbgTdMRFPGjemlp5EvfCyGFArONsiOBSh9OVepxZZurpvMq/1gNiqTShk58yiskeVREmPKc/s27g8NHne70ayDqJpgfW77tGyITX+IndfcWTstz3vJ83jWYJBtB/NxEdINmJsZauE/d6Py5RrmM4hHULQfaaEbf9MBjQmbBxBfq1Wd/KjP9CQoKZ4StF1hz06YZY91lcQAjcNoZs6NdgQQeOradMVELgX6G5jaO/Tz2A5g3b9T0akC9bV1fu8vr9t+bNxlvrz2QnmAv7fTa7OWc5/bWqZnkbI1bZFzM7XM3lyggnDdP0tpUIGmOyorGERRt1yIby8R/oJYWaLfDZJHpl5jMQ7ZFL3k8VbP/op4EuOxxb87fYDruLWzgjV/QCJTR9i/xl1/g3hcZa3m1sAE7WUm0MpRAb7N4XPiJeiA1mWUFFqjCfxTey973b16qt3f0nw1MU/MWqshx/4V+e+zlAOKYtbEPRF9525gAMeRT9d8jxjsk6i02VQUbp6y1eoU0m4Y6Z+qiOJKwqt2LzBtloNzNovsoh1Hkr7VeQc6KJFYP7+dV7/v1eQeZpaC4jS5OznS/N11crpi8g9Fa6H0/iNKxfGyHTu4X6uEwHhu+CpYrwLNW+CF4UeqlWV3QS3hY83YmmVDLk6kCMJh4u/jZN4VfC0/9FcTasvM45yxXp3NW/AtqvAj1APD3YFVdPddQnZOmIV51u9RqxWb6/yda9FFQffWGXwxy4A8xzuJsNizAv2/NXk7zGExhE/tpGIN5QVBf4v0/frWy0f77lsWLmzdrG5tr/3qxufK77fpvX3669Mulf176emlj6d+Wfrd0uHS+dL0UvBi++MOL/3rx35v9zS+b/7H5n6bpj17Uff5pyfnZ/OP/AsWXYDg= < /latexit > Figure 1 : We partition datasets into four sections , defined by the features ( spurious and/or target ) that hold . We sample training datasets D , which provide varying amounts of evidence against the spurious feature , in the form of s-only examples . In the illustration above , the s-only rate is 210 = 0.2 , i.e. , 20 % of examples in D provide evidence that s alone should not be used to predict y . Evidence from Spurious-Only Examples . We are interested in spurious features which are highly correlated with the target during training . Given a training sample D and features s and t , we define the s-only example rate as the evidence against the use of s as a predictor of y . Concretely , s-only rate = |Ds-only| / |D| , the proportion of training examples in which s occurs without t ( and y = 0 ) . Use of Spurious Feature . If a model has falsely learned that the spurious feature s alone is predictive of the label , it will have a high error rate when classifying examples for which s holds but t does not . We define the s-only error to be the classifier ’ s error on examples from Ss-only . When relevant , t-only error , both error , and neither error are defined analogously . In this work , “ feature use ” is a model ’ s predictions consistency with that feature ; we are not making a causal argument . Extractability of a Feature . We want to compare features in terms of how extractable they are given a representation . For example , given a sentence embedding , it may be possible to predict multiple features with high accuracy , e.g. , whether the word “ dog ” occurs , and also whether the word “ dog ” occurs as the subject of the verb “ run ” . However , detecting the former will no doubt be an easier task than detecting the latter . We use the prequential minimum description length ( MDL ) Rissanen ( 1978 ) –first used by Voita & Titov ( 2020 ) for probing–to quantify this intuitive difference.2 MDL is an information-theoretic metric that measures how accurately a feature can be decoded and the amount of effort required to decode it . Formally , MDL measures the number of bits required to communicate the labels given the representations . Conceptually , MDL can be understood as a measure of the area under the loss curve : If a feature is highly extractable , a model trained to detect that feature will converge quickly to high accuracy , resulting in a low MDL . Computing MDL requires repeatedly training a model over a dataset labeled by the feature in question . To compute MDL ( s ) , we train a classifier ( without freezing any parameters ) to differentiate Ss-only vs. Sneither , and similarly compute MDL ( t ) . See Voita & Titov ( 2020 ) for additional details on MDL.3 1Without loss of generality , we define t in our datasets s.t . t ( x ) = y , ∀x , y ∈ S. We do this to iron out the case where t outputs the opposite value of y . 2We observe similar overall trends when using an alternative metric based on validation loss ( Appendix A.3 ) . 3Note that our reported MDL is higher in some cases than that given by the uniform code ( the number of sentences that are being encoded ) . The MDL is computed as a sum of the costs of transmitting successively

Summary: This paper studies the relationship between extractability of features from pre-trained representations and how much a fine-tuned model uses that feature. The extractability of features is measured by the minimum description length of a probing classifier trained to detect the feature from the pre-trained representations (using the online code version of Voita and Titov). The degree to which a fine-tuned model uses the feature is measured by the amount of evidence required for a model to tease apart spurious from non-spurious features (called "target" features). Evidence here means examples where a spurious feature occurs but a non-spurious feature does not occur. When there are many such examples (high spurious-only rate), it is easier for a model to reject the spurious feature and learn to rely on the target feature. The "degree to which a fine-tuned model uses a feature" is defined as the minimal spurious-only rate at which the model can accomplish the task.