Overparameterisation and worst-case generalisation: friend or foe?

1 INTRODUCTION . Overparameterised neural networks have demonstrated the remarkable ability to perfectly fit training samples , while still generalising to unseen test samples ( Zhang et al. , 2017 ; Neyshabur et al. , 2019 ; Nakkiran et al. , 2020 ) . However , several recent works have revealed that overparameterised models ’ good average performance does not translate to good worst-case performance ( Buolamwini & Gebru , 2018 ; Hashimoto et al. , 2018 ; Sagawa et al. , 2020a ; b ) . In particular , the test performance of such models may be poor on certain subgroups that are under-represented in the training data . Worse still , such degradation can be exacerbated as model complexity increases . This indicates the unsuitability of such models in ensuring fairness across subgroups , a topical concern given the growing societal uses of machine learning ( Dwork et al. , 2012 ; Hardt et al. , 2016 ; Buolamwini & Gebru , 2018 ) . Why does overparameterisation induce such unfavourable bias , and how can one correct for it ? Sagawa et al . ( 2020a ) demonstrated how such models may fit to spurious correlations that explain under-represented samples , which can generalise poorly . Sagawa et al . ( 2020b ) further posited that overparameterised models have an inductive bias towards memorising labels for as few samples as possible , which are invariably those from under-represented subgroups . To mitigate such bias , existing approaches include subsampling majority subgroups ( Sagawa et al. , 2020b ) , and modifying the training objective ( Sagawa et al. , 2020a ; Nam et al. , 2020 ; Zhang et al. , 2020 ; Goel et al. , 2020 ) . This suggests two important points regarding overparameterised models ’ performance : ( a ) with standard training , increasing model complexity exacerbates degradation on rare subgroups ; ( b ) controlling this degradation may require alternate training objectives or procedures . In this paper , we establish that while overparameterised models are biased against under-represented examples , in certain settings , such bias may be easily corrected via post-hoc processing of the model outputs . Specifically , such models ’ bias can be largely restricted to their classification layers , and manifest as structured shifts in predictions for rare subgroups . We thus show how two simple techniques applied to the model outputs — classifier retraining based on the learned representations , and correction of the classification threshold — can help overparameterised models improve worstsubgroup performance over underparameterised counterparts . Consequently , even with standard training , overparameterised models can learn sufficient information to model rare subgroups . To make the above concrete , Figure 1 plots a histogram of model predictions for a synthetic dataset from Sagawa et al . ( 2020b ) ( cf . §2 ) . The data comprises four subgroups generated from combinations ( y , a ( x ) ) of labels y ∈ { ±1 } and a feature a ( x ) ∈ { ±1 } . Most samples ( x , y ) have y = a ( x ) , and so these comprise two dominant subgroups within the positive and negative samples . We train an overparameterised linear model , yielding logits f±1 ( x ) . We then plot the decision scores f+1 ( x ) − f−1 ( x ) , which are expected to be > 0 iff y = +1 . Strikingly , there is a distinct separation amongst the subgroup scores : e.g. , samples with y = +1 , a ( x ) = −1 have systematically lower scores than those with y = +1 , a ( x ) = +1 . Consequently , the model incurs a significant error rate on rare subgroups . The structured nature of the separation implies suggests to post-hoc shift the scores to align the distributions ; this markedly improves performance on the rare subgroups ( Figure 1b ) . Scope and contributions . The primary aim of this work is furthering the understanding of the behaviour of overparameterised models , rather than proposing new techniques . Indeed , the post-hoc correction techniques we employ have been well-studied in the related problem setting of longtail learning or learning under class imbalance ( He & Garcia , 2009 ; Buda et al. , 2017 ; Van Horn & Perona , 2017 ) . Several works have demonstrated that the representations learned by standard networks contain sufficient information to distinguish between dominant and rare labels ( Liu et al. , 2019 ; Zhang et al. , 2019 ; Kang et al. , 2020 ; Menon et al. , 2020 ) . Similar techniques are also common the fairness literature ( Hardt et al. , 2016 ; Chzhen et al. , 2019 ) . However , it is not a-priori clear whether such techniques are effective for overparameterised models , whose ability to perfectly fit the training labels can thwart otherwise effective approaches ( Sagawa et al. , 2020a ) . Existing techniques for improving the worst-subgroup error of overparameterised models involve altering the inputs to the model ( Sagawa et al. , 2020b ) , or the training objective ( Sagawa et al. , 2020a ) . By contrast , the techniques we study alter the outputs of a standard network , trained to minimise the softmax cross-entropy on the entire data . Our findings illustrate that such models do not necessarily require bespoke training modifications to perform well on rare subgroups : even with standard training , overparameterised models can ( in certain settings ) learn useful information about rare subgroups . In summary , our contributions are : ( i ) we demonstrate that , in certain settings , overparameterised models ’ poor performance on underrepresented subgroups is the result of a structured bias in the classification layer ( cf . §3 ) ; ( ii ) we show that two simple post-hoc correction procedures ( cf . §4 ) can mitigate the above bias , and thus significantly reduce their worst-subgroup error ( cf . §5 ) . 2 BACKGROUND AND SETTING . Suppose we have a labelled training sample S = { ( xi , yi ) } ni=1 ∈ ( X × Y ) n , for instance space X ⊂ Rd and label space Y . One typically assumes S is an i.i.d . draw from some unknown distribution P ( x , y ) . Further , suppose each ( x , y ) has an associated group membership g ( x , y ) ∈ G , with G . = |G| . This induces G data subgroups , with a prior P ( g ) and conditional distributions P ( x , y | g ) . Following Sagawa et al . ( 2020a ; b ) , we consider groups g ( x , y ) = ( y , a ( x ) ) , where a ( x ) ∈ R is some attribute within x . We assume a ( x ) is fully specified during train and test time ; while not always realistic , such an assumption has precedent in the fairness literature ( Lipton et al. , 2018 ) . The standard goal in classification is to learn a classifier h : X→ Y that minimises the average error Lavg ( h ) . = E g E x , y|g [ ` 01 ( y , h ( x ) ) ] , where ` 01 ( y , h ( x ) ) = Jy 6= h ( x ) K is the 0-1 loss . Typically , one constructs h ( x ) = argmaxy fy ( x ) , where f ( x ) ∈ RY comprises real-valued logits , as learned by empirical risk minimisation ( ERM ) : minf∈F 1 n ∑n i=1 ` ( yi , f ( xi ) ) . Here , ` is a surrogate loss such as the softmax cross-entropy , and F is a function class , such as a neural networks with a fixed architecture . A network is overparametrised if it can perfectly fit the training labels , and thus drive the training error to zero . Remarkably — and in apparent contrast to orthodox statistical wisdom — this does not come at the expense of generalisation on test samples ( Zhang et al. , 2017 ; Belkin et al. , 2019 ; Nakkiran et al. , 2020 ) . This apparent power comes at a price , however . Let us define the worst-subgroup error as Lmax ( h ) . = max g∈G E x , y|g [ ` 01 ( y , h ( x ) ) ] , ( 1 ) i.e. , the worst-case error over all data subgroups . Prior work ( Sagawa et al. , 2020a ; b ) established that for overparameterised models , the worst-subgroup training error can go to zero ( since the model can fit all samples ) , but the worst-subgroup test error can devolve to that of random guessing ( since the model can fit spurious correlations for rare subgroups ) . Further , the degree of degradation can increase with the model complexity . This indicates that the naïve use of overparametrised models may be at odds with ensuring fairness across data subgroups , a core concern in moden applications of machine learning ( Calders & Verwer , 2010 ; Dwork et al. , 2012 ; Hardt et al. , 2016 ; Zafar et al. , 2017 ) . There are several potential strategies to cope with this . One is to perform distributionally robust optimisation ( Hashimoto et al. , 2018 ; Mohri et al. , 2019 ; Sagawa et al. , 2020a ) , and minimise : LDRO ( h ) . = max g∈G [ E x , y|g [ ` ( y , f ( x ) ) ] + Ωg ( f ) ] , where Ωg is some per-group regulariser . In settings where P ( g ) is non-uniform , Sagawa et al . ( 2020a ) proposed to set Ωg ( f ) ≡ 1√ng , where ng is the number of training samples with group g. Alternatively , one can reweight samples to upweight the contribution of rarer groups and minimise : LRW ( h ) . = ∑ g∈G wg · E x , y|g [ ` ( y , f ( x ) ) ] , ( 2 ) where , e.g. , wg = P ( g ) leads to the standard average error , while wg = 1 implicitly upweights rare subgroups . While intuitive , Sagawa et al . ( 2020b ) established that such an approach is also subject to poor worst-subgroup performance , owing to a broader issue with using importance weighting in conjunction with neural networks ( Wen et al. , 2014 ; Byrd & Lipton , 2019 ) . Sagawa et al . ( 2020b ) established that one can achieve good performance by instead subsampling dominant groups , an operation equivalent in expectation to minimising LRW ( h ) with wg = 1 . Recent developments in the mitigation of worst-subgroup errors include Nam et al . ( 2020 ) ; Zhang et al . ( 2020 ) ; Goel et al . ( 2020 ) . In the sequel , we shall make extensive use of three datasets from Sagawa et al . ( 2020a ; b ) , each of which involve binary labels y ∈ Y and a binary attribute a ( x ) ∈ A : ( i ) synth , a synthetic dataset where X ⊂ R200 , Y = { ±1 } , and A ∈ { ±1 } . ( ii ) waterbirds , a dataset of bird images with Y = { land bird , water bird } corresponding to the bird type , and A = { land background , water background } corresponding to the background . ( iii ) celebA , a dataset of celebrity images with Y = { blond , dark } corresponding to individuals ’ hair colour , and A = { male , female } . For each dataset , we construct four subgroups g ( x , y ) = ( y , a ( x ) ) , with two such subgroups being under-represented . On synth and waterbirds , these correspond to subgroups with y 6= a ( x ) , while on celebA , these corespond to the subgroups { ( blond , male ) } and { ( dark , female ) } . Owing to the rarity of certain subgroups , it is intuitively easy for an overparameterised network to learn to predict a ( x ) rather than y , and memorise spurious patterns to predict the rare subgroups . To train overparameterised models , we follow the setup of Sagawa et al . ( 2020a ; b ) , which we briefly summarise . For celebA and waterbirds , we use a ResNet-50 , which can attain perfect training accuracy . For synth , we train a weakly regularised ( λ = 10−16 ) logistic regression model on a fixed representation Φ constructed as follows : for fixed m , we construct Φ ( x ) = ReLU ( V x ) , where V ∈ Rm×200 is a random Gaussian matrix with normalised rows . Overparameterised models consistently demonstrate a significant gap between the average and worst-subgroup error : e.g . ( see Figure 4 ) , on synth , the model achieves 91 % average accuracy , but 36 % worst-subgroup accuracy .

Summary: The paper builds upon prior work that shows that overparameterized networks learned by ERM can have poor worst-case performance over pre-defined groups. Specifically, the paper demonstrates that this result is not necessarily due to overparameterized learning poor representations for rare subgroups, but rather mis-calibration in the classification layer that can be addressed with two simple correct techniques: thresholding and re-training the classification layer. They show improvements over ERM in worst-case subgroup error.