Understanding the failure modes of out-of-distribution generalization

1 INTRODUCTION . A machine learning model in the wild ( e.g. , a self-driving car ) must be prepared to make sense of its surroundings in rare conditions that may not have been well-represented in its training set . This could range from conditions such as mild glitches in the camera to strange weather conditions . This out-of-distribution ( OoD ) generalization problem has been extensively studied within the framework of the domain generalization setting ( Blanchard et al. , 2011 ; Muandet et al. , 2013 ) . Here , the classifier has access to training data sourced from multiple “ domains ” or distributions , but no data from test domains . By observing the various kinds of shifts exhibited by the training domains , we want the classifier can learn to be robust to such shifts . The simplest approach to domain generalization is based on the Empirical Risk Minimization ( ERM ) principle ( Vapnik , 1998 ) : pool the data from all the training domains ( ignoring the “ domain label ” on each point ) and train a classifier by gradient descent to minimize the average loss on this pooled dataset . Alternatively , many recent studies ( Ganin et al. , 2016 ; Arjovsky et al. , 2019 ; Sagawa et al. , 2020a ) have focused on designing more sophisticated algorithms that do utilize the domain label on the datapoints e.g. , by enforcing certain representational invariances across domains . A basic premise behind pursuing such sophisticated techniques , as emphasized by Arjovsky et al . ( 2019 ) , is the empirical observation that ERM-based gradient-descent-training ( or for convenience , just ERM ) fails in a characteristic way . As a standard illustration , consider a cow-camel classification task ( Beery et al. , 2018 ) where the background happens to be spuriously correlated with the label in a particular manner only during training — say , most cows are found against a grassy background and most camels against a sandy one . Then , during test-time , if the correlation is completely flipped ( i.e. , all cows in deserts , and all camels in meadows ) , one would observe that the ∗Work performed in part while Vaishnavh Nagarajan was interning at Blueshift , Alphabet . 1Code is available at https : //github.com/google-research/OOD-failures accuracy of ERM drops drastically . Evidently , ERM , in its unrestrained attempt at fitting the data , indiscriminately relies on all kinds of informative features , including unreliable spurious features like the background . However , an algorithm that carefully uses domain label information can hope to identify and rely purely on invariant features ( or “ core ” features ( Sagawa et al. , 2020b ) ) . While the above narrative is an oft-stated motivation behind developing sophisticated OoD generalization algorithms , there is little formal explanation as to why ERM fails in this characteristic way . Existing works ( Sagawa et al. , 2020b ; Tsipras et al. , 2019 ; Arjovsky et al. , 2019 ; Shah et al. , 2020 ) provide valuable answers to this question through concrete theoretical examples ; however , their examples critically rely on certain factors to make the task difficult enough for ERM to rely on the spurious features . For instance , many of these examples have invariant features that are only partially predictive of the label ( see Fig 1a ) . Surprisingly though , ERM relies on spurious features even in much easier-to-learn tasks where these complicating factors are absent — such as in tasks with fully predictive invariant features e.g. , Fig 1c or the Waterbirds/CelebA examples in Sagawa et al . ( 2020a ) or for that matter , in any real-world situation where the object shape perfectly determines the label . This failure in easy-to-learn tasks , as we argue later , is not straightforward to explain ( see Fig 1b for brief idea ) . This evidently implies that there must exist factors more general and fundamental than those known so far , that cause ERM to fail . Our goal in this work is to uncover these fundamental factors behind the failure of ERM . The hope is that this will provide a vital foundation for future work to reason about OoD generalization . Indeed , recent empirical work ( Gulrajani & Lopez-Paz , 2020 ) has questioned whether existing alternatives necessarily outperform ERM on OoD tasks ; however , due to a lack of theory , it is not clear how to hypothesize about when/why one algorithm would outperform another here . Through our theoretical study , future work can hope to be better positioned to precisely identify the key missing components in these algorithms , and bridge these gaps to better solve the OoD generalization problem . Our contributions . To identify the most fundamental factors causing OoD failure , our strategy is to ( a ) study tasks that are “ easy ” to succeed at , and ( b ) to demonstrate that ERM relies on spurious features despite how easy the tasks are . More concretely : 1 . We formulate a set of constraints on how our tasks must be designed so that they are easy to succeed at ( e.g. , the invariant feature must be fully predictive of the label ) . Notably , this class of easy-to-learn tasks provides both a theoretical test-bed for reasoning about OoD generalization and also a simplified empirical test-bed . In particular , this class encompasses simplified MNIST and CIFAR10-based classification tasks where we establish empirical failure of ERM . 2 . We identify two complementary mechanisms of failure of ERM that arise from how spurious correlations induce two kinds of skews in the data : one that is geometric and the other statistical . In particular , we theoretically isolate these failure modes by studying linear classifiers trained by gradient descent ( on logistic/exponential loss ) and its infinite-time-trained equivalent , the maxmargin classifier ( Soudry et al. , 2018 ; Ji & Telgarsky , 2018 ) on the easy-to-learn-tasks . 3 . We also show that in any easy-to-learn task that does not have these geometric or statistical skews , these models do not rely on the spurious features . This suggests that these skews are not only a sufficient but also a necessary factor for failure of these models in easy-to-learn tasks . 4 . To empirically demonstrate the generality of our theoretical insights , we ( a ) experimentally validate these skews in a range of MNIST and CIFAR10-based tasks and ( b ) demonstrate their effects on fully-connected networks ( FNNs ) and ResNets . We also identify and explain failure in scenarios where standard notions of spurious correlations do not apply ( see Fig 1d ) . We perform similar experiments on a non-image classification task in App E . 2 RELATED WORK . Spurious correlations . Empirical work has shown that deep networks find superficial ways to predict the label , such as by relying on the background ( Beery et al. , 2018 ; Ribeiro et al. , 2016 ) or other kinds of shortcuts ( McCoy et al. , 2019 ; Geirhos et al. , 2020 ) . Such behavior is of practical concern because accuracy can deteriorate under shifts in those features ( Rosenfeld et al. , 2018 ; Hendrycks & Dietterich , 2019 ) . It can also lead to unfair biases and poor performance on minority groups ( Dixon et al. , 2018 ; Zhao et al. , 2017 ; Sagawa et al. , 2020b ) . Understanding failure of ERM . While the fact that ERM relies on spurious correlations has become empirical folk wisdom , only a few studies have made efforts to carefully model this . Broadly , there are two kinds of existing models that explain this phenomenon . One existing model is to imagine that both the invariant and the spurious features are only partially predictive of the label ( Tsipras et al. , 2019 ; Sagawa et al. , 2020b ; Arjovsky et al. , 2019 ; Ilyas et al. , 2019 ; Khani & Liang , 2020 ) , as a result of which the classifier that maximizes accuracy can not ignore the spurious feature ( see Fig 1a ) . The other existing model is based on the “ simplicity bias ” of gradient-descent based deep network training ( Rahaman et al. , 2018 ; Neyshabur et al. , 2015 ; Kalimeris et al. , 2019 ; Arpit et al. , 2017 ; Xu et al. , 2019 ; des Combes et al. , 2018 ) . In particular , this model typically assumes that both the invariant and spurious features are fully predictive of the label , but crucially posits that the spurious features are simpler to learn ( e.g. , more linear ) than the invariant features , and therefore gradient descent prefers to use them ( Shah et al. , 2020 ; Nar et al. , 2019 ; Hermann & Lampinen , 2020 ) . While both these models offer simple-to-understand and useful explanations for why classifiers may use spurious correlations , we provide a more fundamental explanation . In particular , we empirically and theoretically demonstrate how ERM can rely on the spurious feature even in much easier tasks where these explanations would fall apart : these are tasks where unlike in the first model ( a ) the invariant feature is fully predictive and unlike in the second model , ( b ) the invariant feature corresponds to a simple linear boundary and ( c ) the spurious feature is not fully predictive of the label . Further , we go beyond the max-margin settings analyzed in these works to analyze the dynamics of finite-time gradient-descent trained classifier on logistic loss . We would also like to point the reader to concurrent work of Khani & Liang ( 2021 ) that has proposed a different model addressing the above points . While their model sheds insight into the role of overparameterization in the context of spurious features ( and our results are agnostic to that ) , their model also requires the spurious feature to be “ dependent ” on the invariant feature , an assumption we don ’ t require ( see Sec 3 ) . Algorithms for OoD generalization . Due to the empirical shortcomings of ERM , a wide range of sophisticated algorithms have been developed for domain generalization . The most popular strategy is to learn useful features while constraining them to have similar distributions across domains ( Ganin et al. , 2016 ; Li et al. , 2018b ; Albuquerque et al. , 2020 ) . Other works constrain these features in a way that one can learn a classifier that is simultaneously optimal across all domains ( Peters et al. , 2016 ; Arjovsky et al. , 2019 ; Krueger et al. , 2020 ) . As discussed in Gulrajani & Lopez-Paz ( 2020 ) , there are also many other existing non-ERM based methods , including that of meta-learning ( Li et al. , 2018a ) , parameter-sharing ( Sagawa et al. , 2020a ) and data augmentation ( Zhang et al. , 2018 ) . Through their extensive empirical survey of many of the above algorithms , Gulrajani & Lopez-Paz ( 2020 ) suggest that ERM may be just as competitive as the state-of-the-art . But we must emphasize that this doesn ’ t vindicate ERM of its failures but rather indicates that we may be yet to develop a substantial improvement over ERM . 3 EASY-TO-LEARN DOMAIN GENERALIZATION TASKS . Below , we first set up the basic domain generalization setting and the idea of ERM . Then , in Section 3.1 , we formulate a class of domain-generalization tasks that are in many aspects “ easy ” for the learner ( such as fully informative invariant features ) – what exactly makes a task “ easy ” will be discussed in Section 3.1 ) . This discussion sets the ground for the later sections to show how ERM can fail even in these easy tasks , which will help uncover the fundamental factors behind its failure . Notations . Consider an input ( vector ) space X and label space Y ∈ { −1 , 1 } . For any distribution D over X × Y , let pD ( · ) denote its probability density function ( PDF ) . Let H denote a class of classifiers h : X → R. Let the error of h on D be denoted as LD ( h ) : = E ( x , y ) ∼D [ h ( x ) · y < 0 ] . The domain generalization setting and ERM . In the domain generalization setting , one considers an underlying class D of data distributions over X ×Y corresponding to different possible domains . The learner is given training data collected from multiple distributions from D. For an ERM-based learner in particular , the training data will be pooled together , so we can model the data as coming from a single ( pooled ) distribution Dtrain , which for simplicity , can be assumed to belong to D. Given this data , the learner outputs a hypothesis ĥ ∈ H that is tested on a new distribution Dtest picked from D. This can be potentially modeled by assuming that all test and training distributions are drawn from a common hyper-distribution over D. However , this assumption becomes pointless in most practical settings where the training domains are not more than three to four in number ( e.g. , PACS ( Asadi et al. , 2019 ) , VLCS ( Fang et al. , 2013 ) ) , and therefore hardly representative of any hyper-distribution . Here , the problem becomes as hard as ensuring good performance on a worst-case test-distribution without any hyper-distribution assumption ; this boils down to minimizing maxD∈D LD ( ĥ ) . Indeed , most works have studied the worst-case setting , both theoretically ( Sagawa et al. , 2020b ) and empirically ( Arjovsky et al. , 2019 ; Sagawa et al. , 2020a ) . Similarly , for this work , we focus on the worst-case setting and define the optimal target function h ? to be h ? = arg minh∈H maxD∈D LD ( h ) . Then , we define the features that this “ robust ” classifier uses as invariant features Xinv ( e.g. , the shape of the object ) , and the rest as spurious features Xsp ( e.g. , the background ) . To formalize this , we assume that there exists a mapping Φ : Xinv×Xsp → X such that each D ∈ D is induced by a distribution over Xinv × Xsp ( so we can denote any x as Φ ( xinv , xsp ) ) . With an abuse of notation we will use pD ( · ) to also denote the PDF of the distribution over Xinv×Xsp . Then , the fact that Xsp are features that h ? does not rely on , is mathematically stated as : ∀xinv and ∀xsp 6= x′sp , h ? ( Φ ( xinv , xsp ) ) = h ? ( Φ ( xinv , x′sp ) ) . Finally , we note that , to make this learning problem tractable , one has to impose further restrictions ; we ’ ll provide more details on those when we discuss the class of easy-to-learn domain generalization tasks in Sec 3.1 . Empirical failure of ERM . To guide us in constructing the easy-to-learn tasks , let us ground our study in a concrete empirical setup where an ERM-based linear classifier shows OoD failure . Specifically , consider the following Binary-MNIST based task , where the first five digits and the remaining five digits form the two classes . First , we let Φ be the identity mapping , and so x = ( xinv , xsp ) . Then , we let xinv be a random ReLU features representation of the MNIST digit i.e. , if xraw represents the MNIST image , then xinv = ReLU ( Wxraw ) where W is a matrix with Gaussian entries . We make this representation sufficiently high-dimensional so that the data becomes linearly separable . Next , we let the spurious feature take values in { +B , −B } for some B > 0 , imitating the two possible background colors in the camel-cow dataset . Finally , on Dtrain , for any y , we pick the image xinv from the corresponding class and independently set the “ background color ” xsp so that there is some spurious correlation i.e. , PrDtrain [ xsp · y > 0 ] > 0.5 . During test time however , we flip this correlation around so that PrDtest [ xsp · y > 0 ] = 0.0 . In this task , we observe in Fig 2a ( shown later under Sec 4 ) that as we vary the train-time spurious correlation from none ( PrDtrain [ xsp · y > 0 ] = 0.5 ) to its maximum ( PrDtrain [ xsp · y > 0 ] = 1.0 ) , the OoD accuracy of a max-margin classifier progressively deteriorates . ( We present similar results for a CIFAR10 setting , and all experiment details in App C.1 . ) Our goal is now to theoretically demonstrate why ERM fails this way ( or equivalently , why it relies on the spurious feature ) even in tasks as “ easy-to-learn ” as these .

Summary: The paper studies generalization under distribution shift, and tries to answer the question: why do ERM-based classifiers learn to rely on "spurious" features? They present a class of distributions called "easy-to-learn" that rules out several explanations given in recent work and isolates the spurious correlation phenomenon in the simplest possible setting. Even on "easy-to-learn" distributions, linear models obtained from ERM use spurious features owing to either the dynamics of gradient descent trained on separable data (very slow convergence to the max-margin classifier) or a certain geometric skew in the data.