Predicting Inductive Biases of Pre-Trained Models

1 INTRODUCTION . Large pre-trained language models ( LMs ) ( Devlin et al. , 2019 ; Raffel et al. , 2020 ; Brown et al. , 2020 ) have demonstrated impressive empirical success on a range of benchmark NLP tasks . However , analyses have shown that such models are easily fooled when tested on distributions that differ from those they were trained on , suggesting they are often “ right for the wrong reasons ” ( McCoy et al. , 2019 ) . Recent research which attempts to understand why such models behave in this way has primarily made use of two analysis techniques : probing classifiers ( Adi et al. , 2017 ; Hupkes et al. , 2018 ) , which measure whether or not a given feature is encoded by a representation , and challenge sets ( Cooper et al. , 1996 ; Linzen et al. , 2016 ; Rudinger et al. , 2018 ) , which measure whether model behavior in practice is consistent with use of a given feature . The results obtained via these two techniques currently suggest different conclusions about how well pre-trained representations encode language . Work based on probing classifiers has consistently found evidence that models contain rich information about syntactic structure ( Hewitt & Manning , 2019 ; Bau et al. , 2019 ; Tenney et al. , 2019a ) , while work using challenge sets has frequently revealed that models built on top of these representations do not behave as though they have access to such rich features , rather they fail in trivial ways ( Dasgupta et al. , 2018 ; Glockner et al. , 2018 ; Naik et al. , 2018 ) . In this work , we attempt to link these two contrasting views of feature representations . We assume the standard recipe in NLP , in which linguistic representations are first derived from large-scale selfsupervised pre-training intended to encode broadly-useful linguistic features , and then are adapted for a task of interest via transfer learning , or fine-tuning , on a task-specific dataset . We test the hypothesis that the extent to which a fine-tuned model uses a given feature can be explained as a function of two metrics : The extractability of the feature after pre-training ( as measured by probing classifiers ) and the evidence available during fine-tuning ( defined as the rate of co-occurrence with the label ) . We first show results on a synthetic task , and second using state-of-the-art pre-trained LMs on language data . Our results suggest that probing classifiers can be viewed as a measure of the pre-trained representation ’ s inductive biases : The more extractable a feature is after pre-training , the less statistical evidence is required in order for the model to adopt the feature during fine-tuning . Contribution . This work establishes a relationship between two widely-used techniques for analyzing LMs . Currently , the question of how models ’ internal representations ( measured by probing classifiers ) influence model behavior ( measured by challenge sets ) remains open ( Belinkov & Glass , 2019 ; Belinkov et al. , 2020 ) . Understanding the connection between these two measurement techniques can enable more principled evaluation of and control over neural NLP models . 2 SETUP AND TERMINOLOGY . 2.1 FORMULATION . Our motivation comes from McCoy et al . ( 2019 ) , which demonstrated that , when fine-tuned on a natural language inference task ( Williams et al. , 2018 , MNLI ) , a model based on a state-of-the-art pre-trained LM ( Devlin et al. , 2019 , BERT ) categorically fails on test examples which defy the expectation of a “ lexical overlap heuristic ” . For example , the model assumes that the sentence “ the lawyer followed the judge ” entails “ the judge followed the lawyer ” purely because all the words in the latter appear in the former . While this heuristic is statistically favorable given the model ’ s training data , it is not infallible . Specifically , McCoy et al . ( 2019 ) report that 90 % of the training examples containing lexical overlap had the label “ entailment ” , but the remaining 10 % did not . Moreover , the results of recent studies based on probing classifiers suggest that more robust features are extractable with high reliability from BERT representations . For example , given the example “ the lawyer followed the judge ” / “ the judge followed the lawyer ” , if the model can represent that “ lawyer ” is the agent of “ follow ” in the first sentence , but is the patient in the second , then the model should conclude that the sentences have different meanings . Such semantic role information can be recovered at > 90 % accuracy from BERT embeddings ( Tenney et al. , 2019b ) . Thus , the question is : Why would a model prefer a weak feature over a stronger one , if both features are extractable from the model ’ s representations and justified by the model ’ s training data ? Abstracting over details , we distill the basic NLP task setting described above into the following , to be formalized in the Section 2.2 . We assume a binary sequence classification task where a target feature t perfectly predicts the label ( e.g. , the label is 1 iff t holds ) . Here , t represents features which actually determine the label by definition , e.g. , whether one sentence semantically entails another . Additionally , there exists a spurious feature s that frequently co-occurs with t in training but is not guaranteed to generalize outside of the training set . Here , s ( often called a “ heuristic ” or “ bias ” elsewhere in the literature ) corresponds to features like lexical overlap , which are predictive of the label in some datasets but are not guaranteed to generalize . Assumptions . In this work , we assume there is a single t and a single s ; in practice there may be many s features . Still , our definition of a feature accommodates multiple spurious or target features . In fact , some of our spurious features already encompass multiple features : the lexical feature , for example , is a combination of several individual-word features because it holds if one of a set of words is in the sentence . This type of spurious feature is common in real datasets : E.g. , the hypothesis-only baseline in NLI is a disjunction of lexical features ( with semantically unrelated words like “ no ” , “ sleeping ” , etc . ) ( Poliak et al. , 2018b ; Gururangan et al. , 2018 ) . We assume that s and t frequently co-occur , but that only s occurs in isolation . This assumption reflects realistic NLP task settings since datasets always contain some heuristics , e.g. , lexical cues , cultural biases , or artifacts from crowdsourcing ( Gururangan et al. , 2018 ) . Thus , our experiments focus on manipulating the occurrence of s alone , but not t alone : This means giving the model evidence against relying on s. This is in line with prior applied work that attempts to influence model behavior by increasing the evidence against s during training ( Elkahky et al. , 2018 ; Zmigrod et al. , 2019 ; Min et al. , 2020 ) . Published as a conference paper at ICLR 2021 2.2 DEFINITIONS . Let X be the set of all sentences and S be the space of all sentence-label pairs ( x , y ) ∈ X × { 0 , 1 } . We use D ⊂ S to denote a particular training sample drawn from S. We define two types of binary features : target ( t ) and spurious ( s ) . Each is a function from sentences x ∈ X to a binary label { 0 , 1 } that indicates whether the feature holds . Target and spurious features . The target feature t is such that there exists some function f : { 0 , 1 } → { 0 , 1 } such that ∀ ( x , y ) ∈ S , f ( t ( x ) ) = y . In other words , the label can always be perfectly predicted given the value of t.1 A feature s is spurious if it is not a target feature . Partitions of S. To facilitate analysis , we partition S in four regions ( Figure 1 ) . We define Ss-only to be the set of examples in which the spurious feature occurs alone ( without the target ) . Similarly , St-only is the set of examples in which the target occurs without the spurious feature . Sboth and Sneither are analogous . For clarity , we sometimes drop the S∗ notation ( e.g. , s-only in place of Ss-only ) . t-only both s-only neither S D ( a ) ( b ) Sboth “ tpx , yq | tpxq “ 1 ^ spxq “ 1u Sneither “ tpx , yq | tpxq “ 0 ^ spxq “ 0u St-only “ tpx , yq | tpxq “ 1 ^ spxq “ 0u Ss-only “ tpx , yq | tpxq “ 0 ^ spxq “ 1u < latexit sha1_base64= '' bSiqwQANSN306jL/eI6aFqdfpQw= '' > AAA00nicnVtbc9zKcaYcx3EYxzlOHv2CCiX7nBTFInWYil9cZd5JcXlfXqSzOiosthcLETdhZrGk4E1VKi9+8Evyg/I//F/8kJ4ZDLoHC/LEYZVE9PfNNOby9XQvFhzmcSTk+vqfXvzor3781z/5m5/+7fLf/ezvf/4PX/3iH29ENi0CuA6yOCvuhr6AOErhWkYyhru8AD8ZxnA7vN9R/G0JhYiytC8fc/iQ+GEajaPAlwh9/OrPHv5cfawGEh5kkVQvh5mcvJzPf/XbQfX1w+rjN4PV3w9W5dcP3/x2YzCDUQieMMbcGwyW271TiOQEim4H69zBercD+fJ1lsaPPzyAJ/qLZ/qvtyaw/PGrlfW1df3jLV5s1BcrS/XP+cdfvPmfwSgLpgmkMoh9Ib7bWM/lh8ovZBTEMF8eTAXkfnDvh/AdXqZ+AuJDpbdp7r1CZOSNswL/pdLTKO9R+YkQj8kQWya+nIg2p8Au7rupHP/mQxWl+VRCGpgbjaexJzNP7bk3igoIZPyIF35QRDhWL5j4hR9IVIZzl2GW3Ut/KNCHt68HWiQ+OvJRT25LNQqZZbFuGqACxfICX4ixmHfO0AHFdDiOwrnnzCnVd4bkQzXF/7s6rHrjOPPlfOG2/vBhNSz8fBIFDy4rs1GWZhJao5phYzWCp2bQuRFtUO3p/2W2z4/w/+N9efmV+vFO9269k63+obe7t390etQ/Oju98jTVOYNV63Z1iOu7/Mo78Yt7T6BS8GQQXjbGbc3NtdJsAWMoiigNlaxGURkJ2wyXblooeaQwC7Ik8dNRNUAwhrGcV9UAEu/rHl5/M58vtAkwkqCwrXa01dWuiMJJ4+xSGV2tZJbbNv0s72qBx5vMEttoW1sL7ep5+7aZ/1SLoW0xfKpFYFsET7UY2RYj1QK34RBnF6sZer6H7VXYwhiP95GHa5O4PvBagfPvNj6gl+HYW9mYa0F4+3pTzK7hoQCrXpzNoHitQnVteYAu9bLCeGWjMhv47wO0Ku2gqzsON5J+vKbPBSHxyFN7L9SOIW887luP+22PmpazzN5z5U19V+HZRh7OqDbe2B6fp/6Iuqx8u7K50G216WOvvuWuNvV0royqn10OVL4ZfB0Cznp0OLALYnpf2d5XHb0vbS99JM+yJsrWmoUxdxd6ZZoYfGJp2g4nBUDbJfO38u2iR1o15vvbRd9+6gFugurcsWTw2czZNnl60o6faZ5DoZOGcbNXu9nrcrPlFf6M1r3l7PXr136ZRSNvKtTRFI29PBMiwnRlXOexj6FT+396dCof5hhJHXNUjOlet/mLJ1k72mkc7fygI5xzipWKOoNNW2F8aLgZEUrF0tbV69dPygRH58dhhvl/knTMEzkzuqbRsxNlrhZmumVdbXW4soK398NJNL6ePwz6TqetH+y0sKg5Hlr1zJn6FGqGq66e2xTTv63e86b/udvfzrS5AY5aXT854FpwEMVKrLG6wAMdG6ir2h/WPFmhaX1leH1ZN0BqmFQb7WwjCwyEeTVQmT/w42q33aD042jEG3w011hVG2q+4BKE7O6gmXkzI8iFynG5iOIsrfPTJbrIEq/0i6guLrW+QfqVqeZNAVi9HCD0cm6Xs2jRPjFDlxkSE7hMQMzIZUbEgMsAMWOXGRMTukxIzMRlJsRELhMR88llPhFz7zL3xMQuE8+1jIvEiwRGLH4wHD2qw87s4Kr3aSqkhyXxr6WnPqqgHB/VyeNsjJfUvlPXd0p3zVwmIyZ3mZyYzy7zmZjCZQpihMsIYqTLSGKmLjMlpnSZkpiZy8yIeXCZB2IeXeaRmC8u82VuyjwbAJiZs+Z4L+sgqUwoDccsbJpx4ydsHSW2hbYZzzgODwlmsVEGBLPAKEcEs6gogWAWEuWYYBYPZUgwC4ZyQjCLhHJKMAuD8hPBLAbKe4JZAJQxwTGDE4ITBrOF5iucEczEXOYEMyWXnwlmMi4LgpmGS0Gw4JtKsOxeEy7dkmCm23JGMBNt+UAwU2z5SDCTa/mFYKvVPfwIjoW+/rRXdOgWjOg6z2Uwyus8mcHIr/NsBqPBztMZjBA7z2cwauw8ocFIsvOMBqPLzlMauSfPaTAK7Typwci086wGo9X2aW25xOUSzj15EoORbudZDEa/nacxGBF3nsdglNx5IoORc+eZDEbTnacyGGF3nstg1N15MoOReOfZDEbnnaczGLF3ns9gFP/0CY2xUERBU6EkWxQfWxQ2yTbB2wzeIXiHwbsE7zJ4j+A9Bu8TvM/gA4IPGHxI8CGDjwg+YvBbgt8y+JjgYwb3CO4x+ITgEwafEnzK4DOCzxh8TvA5gy8IvmDwJcGXDL4i+IrBfYL7DL4m+JrBNwTfMPiW4FsG3xF8x+B3BL9j8HuC3z99vLqiA6M6ptEtpl8tPcZtc27H5XY4t+tyu5zbc7k9zu273D7nDlzugHOHLnfIuSOXO+LcW5d7y7ljlzvmXM/lepw7cbkTzp263CnnzlzujHPnLnfOuQuXu+Dcpctdcu7K5a4413e5PueuXe6aczcud8O5W5e75dydy91x7p3LvePce5ezsr/hJUT5BfTnCPzsut70LbMUKvt51mLJ1ECDhJJGUxMr3K2HyxpmyNAgVIfoKgQRqj507YEI1RxlPRKqNHSdgQjVF7q6QISqCl1TIEK1hK4kEKEKQtcPiFDdoKsGRKha0LUCIjFbB4NQZaDrAkRStn4GoSpA1wCIUO7XmR8Ryvg63yNCeV5neUQEW3CDUE4v621hm1IahPK3zt6IUNbWORsRytU6UyNCGVrnZ0S6qlG3DC39OJ+o/da/GwWWw1ocWhcWpI9a9GSipmI/GY5UD3NBRJZAqHD9m2AtSSVHC6BDRPB/gkQUJqqr/k2wFW4t2mYiVcXHXymxWgvFGpCFQh2xSVVKoNZCgY7JQnGGZKEwJ2ThcNlYUZCfyEIx3rO1qZQIm5lXSoDWwsVkq4jiy9iSVEp01kLRfSYLBVewlaqU0JoFqpTIrIULzZYZBVaSheKakYXCeiALRfVIFgrqy7z+zgvz7IPBdY5FnVFu1ZkVEcqoOp8iQnlUZ1FEKHvq3IkI5UydMRGhTKnzJCKUH3V2RISyos6JiFAu1JkQEcqAOv8hQnlPZz1EKNvpXIcI5Tid4RChzKbzGiKUz3Q2Q4SymM5hiFDu0pkLEcpYOl8hQnlKZylEKDvp3IQI5SSdkRChTKTzECKUf3T2QYSyjs45iFCu0ZkGkfdsBykvDHlaSM4n9UE8wCu2ejb0FdOrw7+ZXB3DirsycaxV1IdUqO/ydyGI/QJQVJMtdQLhHU2xJ8aRelQKaZCNojREZ/40VogYN9fJvBLqKe8VyKccDLN49ENuhg9zDML2k9pU6O8ITd6s/emn1PXUpKkvU8HUL7ctRvqXOxajCJC7FqMYkHsWoyiQ+xajOJAHFqNIkIcWo1iQRxajaJBvLUbxII8tRhEhexajmJAnFqOokKcWo7iQZxajyJDnFqPYkBcWo+iQlxaj+JBXFqMIkX2LUYzIa4tRlMgbi1GcyFuLUaTIO4tRrMh3FqNoke8tZioyFPKBeinBsKH9nBs4HzfCbQaTLsIdBpM0wl0GkzrCPQaTQMJ9BpNGwgMGk0zCQwaTUsIjBpNYwrcMJr2ExwwmyYQ9BpNqwhMGk3DCUwaTdsIzBpN8wnMGk4LCCwaTiMJLBpOOwisGk5TCPoNJTeE1g0lQ4Q2DSVPhLYNJVuEdg0lZ4TsGk7jC9wy2FT8ebXWpJpqnKEMmLrFNKGlL7BBK0hK7hGplvfJ29TcZUwGe7wmQHt46hpG3t+oNIfAVLieR8GbZNB4hhBZ4Qn/vgbXktPDU6zRZjI7U+y7wkGNtqb/MtV+p79MdSZ3igFASpzgklLQpjgglaYq3hJIyxTGhJEzRI5R0KU4IJVmKU0JJleKMUBKlOCeUNCkuCCVJiktCSZHiilASpOgTSnoU14SSHMUNoaRGcUsoiVHcEUpaFO8IJSmK94Q2j1xSrPtAf4TwzcOWuggEqgB6bvGvysMtslCq22ShRHfIQmnukoWH3R5ZKKJ9slA8B2ShaA7JQrEckYUieUsWiuOYLBRFjywUwwlZKIJTsnDzz8jCTT8nCzf7gizc5EuycHOvyMJN7ZOFm3lNFm7iDVm4ebdk4abdkYWb9Y4s3KT37H51pVVXWWrLgG+ZNBUXHikqfvX7lBjEBl31ZpGcZFPpYbnjqZf7cijcggioInKqofr2stGAbrhQCIIul6BVL4EumKBVMYEumaBVM4EumqBVNYEum6BVN4EunKBVOYEunaBVO4EunqBVPYEun6BVP4EuoKBVQYEuoaBVQ4EuoqBVRYEuo6BVR4EupKBVSYEupaBVS4EupqBVTYEup6BVT4EuqKBVUYEuqaBVU4EuqqBVVYEuq6BVV4EurKBVWYEuraBVW4EurqBVXYEur6BVX4EusIBVWPhJAVOOLKbgTdMRFPGjemlp5EvfCyGFArONsiOBSh9OVepxZZurpvMq/1gNiqTShk58yiskeVREmPKc/s27g8NHne70ayDqJpgfW77tGyITX+IndfcWTstz3vJ83jWYJBtB/NxEdINmJsZauE/d6Py5RrmM4hHULQfaaEbf9MBjQmbBxBfq1Wd/KjP9CQoKZ4StF1hz06YZY91lcQAjcNoZs6NdgQQeOradMVELgX6G5jaO/Tz2A5g3b9T0akC9bV1fu8vr9t+bNxlvrz2QnmAv7fTa7OWc5/bWqZnkbI1bZFzM7XM3lyggnDdP0tpUIGmOyorGERRt1yIby8R/oJYWaLfDZJHpl5jMQ7ZFL3k8VbP/op4EuOxxb87fYDruLWzgjV/QCJTR9i/xl1/g3hcZa3m1sAE7WUm0MpRAb7N4XPiJeiA1mWUFFqjCfxTey973b16qt3f0nw1MU/MWqshx/4V+e+zlAOKYtbEPRF9525gAMeRT9d8jxjsk6i02VQUbp6y1eoU0m4Y6Z+qiOJKwqt2LzBtloNzNovsoh1Hkr7VeQc6KJFYP7+dV7/v1eQeZpaC4jS5OznS/N11crpi8g9Fa6H0/iNKxfGyHTu4X6uEwHhu+CpYrwLNW+CF4UeqlWV3QS3hY83YmmVDLk6kCMJh4u/jZN4VfC0/9FcTasvM45yxXp3NW/AtqvAj1APD3YFVdPddQnZOmIV51u9RqxWb6/yda9FFQffWGXwxy4A8xzuJsNizAv2/NXk7zGExhE/tpGIN5QVBf4v0/frWy0f77lsWLmzdrG5tr/3qxufK77fpvX3669Mulf176emlj6d+Wfrd0uHS+dL0UvBi++MOL/3rx35v9zS+b/7H5n6bpj17Uff5pyfnZ/OP/AsWXYDg= < /latexit > Figure 1 : We partition datasets into four sections , defined by the features ( spurious and/or target ) that hold . We sample training datasets D , which provide varying amounts of evidence against the spurious feature , in the form of s-only examples . In the illustration above , the s-only rate is 210 = 0.2 , i.e. , 20 % of examples in D provide evidence that s alone should not be used to predict y . Evidence from Spurious-Only Examples . We are interested in spurious features which are highly correlated with the target during training . Given a training sample D and features s and t , we define the s-only example rate as the evidence against the use of s as a predictor of y . Concretely , s-only rate = |Ds-only| / |D| , the proportion of training examples in which s occurs without t ( and y = 0 ) . Use of Spurious Feature . If a model has falsely learned that the spurious feature s alone is predictive of the label , it will have a high error rate when classifying examples for which s holds but t does not . We define the s-only error to be the classifier ’ s error on examples from Ss-only . When relevant , t-only error , both error , and neither error are defined analogously . In this work , “ feature use ” is a model ’ s predictions consistency with that feature ; we are not making a causal argument . Extractability of a Feature . We want to compare features in terms of how extractable they are given a representation . For example , given a sentence embedding , it may be possible to predict multiple features with high accuracy , e.g. , whether the word “ dog ” occurs , and also whether the word “ dog ” occurs as the subject of the verb “ run ” . However , detecting the former will no doubt be an easier task than detecting the latter . We use the prequential minimum description length ( MDL ) Rissanen ( 1978 ) –first used by Voita & Titov ( 2020 ) for probing–to quantify this intuitive difference.2 MDL is an information-theoretic metric that measures how accurately a feature can be decoded and the amount of effort required to decode it . Formally , MDL measures the number of bits required to communicate the labels given the representations . Conceptually , MDL can be understood as a measure of the area under the loss curve : If a feature is highly extractable , a model trained to detect that feature will converge quickly to high accuracy , resulting in a low MDL . Computing MDL requires repeatedly training a model over a dataset labeled by the feature in question . To compute MDL ( s ) , we train a classifier ( without freezing any parameters ) to differentiate Ss-only vs. Sneither , and similarly compute MDL ( t ) . See Voita & Titov ( 2020 ) for additional details on MDL.3 1Without loss of generality , we define t in our datasets s.t . t ( x ) = y , ∀x , y ∈ S. We do this to iron out the case where t outputs the opposite value of y . 2We observe similar overall trends when using an alternative metric based on validation loss ( Appendix A.3 ) . 3Note that our reported MDL is higher in some cases than that given by the uniform code ( the number of sentences that are being encoded ) . The MDL is computed as a sum of the costs of transmitting successively

Summary: The paper aims to bridge the gap between model interpretation using probing and model's use of spurious features. They show that the findings of MDL with respect to a feature correlate with the extractability of the feature, given the evidence of representing the feature is available in the training data. The results are presented using both synthetic and natural language data.