Tilted Empirical Risk Minimization

1 INTRODUCTION . Many statistical estimation procedures rely on the concept of empirical risk minimization ( ERM ) , in which the parameter of interest , θPΘĎRd , is estimated by minimizing an average loss over the data : Rpθq : “ 1 N ÿ iPrNs fpxi ; θq . ( 1 ) While ERM is widely used and has nice statistical properties , it can perform poorly in situations where average performance is not an appropriate surrogate for the problem of interest . Significant research has thus been devoted to developing alternatives to traditional ERM for diverse applications , such as learning in the presence of noisy/corrupted data ( Jiang et al. , 2018 ; Khetan et al. , 2018 ) , performing classification with imbalanced data ( Lin et al. , 2017 ; Malisiewicz et al. , 2011 ) , ensuring that subgroups within a population are treated fairly ( Hashimoto et al. , 2018 ; Samadi et al. , 2018 ) , or developing solutions with favorable out-of-sample performance ( Namkoong & Duchi , 2017 ) . In this paper , we suggest that deficiencies in ERM can be flexibly addressed via a unified framework , tilted empirical risk minimization ( TERM ) . TERM encompasses a family of objectives , parameterized by a real-valued hyperparameter , t. For t P Rz0 , the t-tilted loss ( TERM objective ) is given by : rRpt ; θq : “ 1 t log ˆ 1 N ÿ iPrNs etfpxi ; θq ˙ . ( 2 ) TERM generalizes ERM as the 0-tilted loss recovers the average loss , i.e. , rRp0 , θq “ Rpθq.1 It also recovers other popular alternatives such as the max-loss ( tÑ ` 8 ) and min-loss ( tÑ´8 ) ( Lemma 2 ) . For tą0 , the objective is a common form of exponential smoothing , used to approximate the max ( Kort & Bertsekas , 1972 ; Pee & Royset , 2011 ) . Variants of tilting have been studied in several contexts , ˚Equal contribution . 1 rRp0 ; θq is defined in ( 14 ) via the continuous extension of Rpt ; θq . including robust regression ( Wang et al. , 2013 ) ptă0q , importance sampling ( Wainwright et al. , 2005 ) , sequential decision making ( Howard & Matheson , 1972 ; Nass et al. , 2019 ) , and large deviations theory ( Beirami et al. , 2018 ) . However , despite the rich history of tilted objectives , they have not seen widespread use in machine learning . In this work , we aim to bridge this gap by : ( i ) rigorously studying the objective in a general form , and ( ii ) exploring its utility for a number of ML applications . Surprisingly , we find that this simple extension to ERM is competitive for a wide range of problems . To highlight how the TERM objective can help with issues such as outliers or imbalanced classes , we discuss three motivating examples below , which are illustrated in Figure 1 . ( a ) Point estimation : As a first example , consider determining a point estimate from a set of samples that contain some outliers . We plot an example 2D dataset in Figure 1a , with data centered at ( 1,1 ) . Using traditional ERM ( i.e. , TERM with t “ 0 ) recovers the sample mean , which can be biased towards outlier data . By setting t ă 0 , TERM can suppress outliers by reducing the relative impact of the largest losses ( i.e. , points that are far from the estimate ) in ( 2 ) . A specific value of t ă 0 can in fact approximately recover the geometric median , as the objective in ( 2 ) can be viewed as approximately optimizing specific loss quantiles ( a connection which we make explicit in Section 2 ) . In contrast , if these ‘ outlier ’ points are important to estimate , setting t ą 0 will push the solution towards a point that aims to minimize variance , as we prove more rigorously in Section 2 , Theorem 4 . ( b ) Linear regression : A similar interpretation holds for the case of linear regression ( Figure 2b ) . As tÑ ´8 , TERM finds a line of best while ignoring outliers . However , this solution may not be preferred if we have reason to believe that these ‘ outliers ’ should not be ignored . As tÑ ` 8 , TERM recovers the min-max solution , which aims to minimize the worst loss , thus ensuring the model is a reasonable fit for all samples ( at the expense of possibly being a worse fit for many ) . Similar criteria have been used , e.g. , in defining notions of fairness ( Hashimoto et al. , 2018 ; Samadi et al. , 2018 ) . We explore several use-cases involving robust regression and fairness in more detail in Section 5 . ( c ) Logistic regression : Finally , we consider a binary classification problem using logistic regression ( Figure 2c ) . For t P R , the TERM solution varies from the nearest cluster center ( tÑ´8 ) , to the logistic regression classifier ( t “ 0 ) , towards a classifier that magnifies the misclassified data ( tÑ ` 8 ) . We note that it is common to modify logistic regression classifiers by adjusting the decision threshold from 0.5 , which is equivalent to moving the intercept of the decision boundary . This is fundamentally different than what is offered by TERM ( where the slope is changing ) . As we show in Section 5 , this added flexibility affords TERM with competitive performance on a number of classification problems , such as those involving noisy data , class imbalance , or a combination of the two . Contributions . In this work , we explore TERM as a simple , unified framework to flexibly address various challenges with empirical risk minimization . We first analyze the objective and its solutions , showcasing the behavior of TERM with varying t ( Section 2 ) . Our analysis provides novel connections between tilted objectives and superquantile methods . We develop efficient methods for solving TERM ( Section 4 ) , and show via numerous case studies that TERM is competitive with existing , problemspecific state-of-the-art solutions ( Section 5 ) . We also extend TERM to handle compound issues , such as the simultaneous existence of noisy samples and imbalanced classes ( Section 3 ) . Our results demonstrate the effectiveness and versatility of tilted objectives in machine learning . 2 TERM : PROPERTIES & INTERPRETATIONS . To better understand the performance of the t-tilted losses in ( 2 ) , we provide several interpretations of the TERM solutions , leaving the full statements of theorems and proofs to the appendix . We make no distributional assumptions on the data , and study properties of TERM under the assumption that the loss function forms a generalized linear model , e.g. , L2 loss and logistic loss ( Appendix D ) . However , we also obtain favorable empirical results using TERM with other objectives such as deep neural networks and PCA in Section 5 , motivating the extension of our theory beyond GLMs in future work . General properties . We begin by noting several general properties of the TERM objective ( 2 ) . Given a smooth fpx ; θq , the t-tilted loss is smooth for all finite t ( Lemma 4 ) . If fpx ; θq is strongly convex , the t-tilted loss is strongly convex for t ą 0 ( Lemma 5 ) . We visualize the solutions to TERM for a toy problem in Figure 2 , which allows us to illustrate several special cases of the general framework . As discussed in Section 1 , TERM can recover traditional ERM ( t “ 0 ) , the max-loss ( tÑ ` 8 ) , and the min-loss ( tÑ´8 ) . As we demonstrate in Section 5 , providing a smooth tradeoff between these specific losses can be beneficial for a number of practical use-cases— both in terms of the resulting solution and the difficulty of solving the problem itself . Interestingly , we additionally show that the TERM solution can be viewed as a smooth approximation to a superquantile method , which aims to minimize quantiles of losses such as the median loss . In Figure 2 , it is clear to see why this may be beneficial , as the median loss ( orange ) can be highly non-smooth in practice . We make these rough connections more explicit via the interpretations below . ( Interpretation 1 ) Re-weighting samples to magnify/suppress outliers . As discussed via the toy examples in Section 1 , the TERM objective can be tuned ( using t ) to magnify or suppress the influence of outliers . We make this notion rigorous by exploring the gradient of the t-tilted loss in order to reason about the solutions to the objective defined in ( 2 ) . Lemma 1 ( Tilted gradient , proof in Appendix B ) . For a smooth loss function fpx ; θq , ∇θ rRpt ; θq “ ÿ iPrNs wipt ; θq∇θfpxi ; θq , where wipt ; θq : “ etfpxi ; θq ř jPrNse tfpxj ; θq “ 1 N etpfpxi ; θq´ rRpt ; θqq . ( 3 ) From this , we can observe that the tilted gradient is a weighted average of the gradients of the original individual losses , where each data point is weighted exponentially proportional to the value of its loss . Note that t “ 0 recovers the uniform weighting associated with ERM , i.e. , wipt ; θq “ 1 { N . For positive t , it magnifies the outliers—samples with large losses—by assigning more weight to them , and for negative t , it suppresses the outliers by assigning less weight to them . ( Interpretation 2 ) Tradeoff between average-loss and min/max-loss . To put Interpretation 1 in context and understand the limits of TERM , a benefit of the framework is that it offers a continuum of solutions between the min and max losses . Indeed , for positive values of t , TERM enables a smooth tradeoff between the average-loss and max-loss ( as we demonstrate in Figure 10 , Appendix I ) . Hence , TERM can selectively improve the worst-performing losses by paying a penalty on average performance , thus promoting a notion of uniformity or fairness ( Hashimoto et al. , 2018 ) . On the other hand , for negative t , the solutions achieve a smooth tradeoff between average-loss and min-loss , which can have the benefit of focusing on the ‘ best ’ losses , or ignoring outliers ( Theorem 3 , Appendix D ) . Theorem ( Formal statement and proof in Appendix D , Theorem 3 ) . Let θ̆ptq be the minimizer of rRpt ; θq , referred to as t-tilted solution . Then , for t ą 0 , max-loss , pRpθ̆ptqq , is non-increasing with t while the average loss , Rpθ̆ptqq , is non-decreasing with t. ( Interpretation 3 ) Empirical bias/variance tradeoff . Another key property of the TERM solutions is that the empirical variance of the loss across all samples decreases as t increases ( Theorem 4 ) . Hence , by increasing t , it is possible to trade off between optimizing the average loss vs. reducing variance , allowing the solutions to potentially achieve a better bias-variance tradeoff for generalization ( Bennett , 1962 ; Hoeffding , 1994 ; Maurer & Pontil , 2009 ) ( Figure 10 , Appendix I ) . We use this property to achieve better generalization in classification in Section 5 . We also prove that the cosine similarity between the loss vector and the all-ones vector monotonically increases with t ( Theorem 5 ) , which shows that larger t promotes a more uniform performance across all losses and can have implications for fairness defined as representation disparity ( Hashimoto et al. , 2018 ) ( Section 5.2 ) . Theorem ( Formal statement and proof in Appendix D , Theorem 4 ) . Let fpθq : “ pfpx1 ; θqq , . . . , fpxN ; θqq be the loss vector for parameter θ . Then , the variance of the vector fpθ̆ptqq is non-increasing with t while its average , i.e. , Rpθ̆ptqq , is non-decreasing with t. ( Interpretation 4 ) Approximate superquantile method . Finally , we show that TERM is related to superquantile-based objectives , which aim to minimize specific quantiles of the individual losses that exceed a certain value ( Rockafellar et al. , 2000 ) . For example , optimizing for 90 % of the individual losses ( ignoring the worst-performing 10 % ) could be a more reasonable practical objective than the pessimistic min-max objective . Another common application of this is to use the median in contrast to the mean in the presence of noisy outliers . As we discuss in Appendix G , superquantile methods can be reinterpreted as minimizing the k-loss , defined as the k-th smallest loss of N ( i.e. , 1-loss is the min-loss , N -loss is the max-loss , pN´1q { 2-loss is the median-loss ) . While minimizing the k-loss is more desirable than ERM in many applications , the k-loss is non-smooth ( and generally non-convex ) , and is challenging to solve for large-scale problems ( Jin et al. , 2020 ; Nouiehed et al. , 2019b ) . Theorem ( Formal statement and proof in Appendix G , Theorem 10 ) . The quantile of the losses that exceed a given value is upper bounded by a smooth function of the TERM objective . Further , the t-tilted solutions are good approximate solutions of the superquantile ( k-loss ) optimization .

Summary: This work analyzes the LogSumExp aggregated loss (named tiled empirical risk minimization, or TERM, in the paper). It provides several general properties of the loss, such as its relation to min/avg/max-loss, and interpretations of different trade-offs. Empirically, it is shown that TERM can be applied to a diverse set of problems, including robust optimization, fairness and generalization.