Tilted Empirical Risk Minimization

1 INTRODUCTION . Many statistical estimation procedures rely on the concept of empirical risk minimization ( ERM ) , in which the parameter of interest , θPΘĎRd , is estimated by minimizing an average loss over the data : Rpθq : “ 1 N ÿ iPrNs fpxi ; θq . ( 1 ) While ERM is widely used and has nice statistical properties , it can perform poorly in situations where average performance is not an appropriate surrogate for the problem of interest . Significant research has thus been devoted to developing alternatives to traditional ERM for diverse applications , such as learning in the presence of noisy/corrupted data ( Jiang et al. , 2018 ; Khetan et al. , 2018 ) , performing classification with imbalanced data ( Lin et al. , 2017 ; Malisiewicz et al. , 2011 ) , ensuring that subgroups within a population are treated fairly ( Hashimoto et al. , 2018 ; Samadi et al. , 2018 ) , or developing solutions with favorable out-of-sample performance ( Namkoong & Duchi , 2017 ) . In this paper , we suggest that deficiencies in ERM can be flexibly addressed via a unified framework , tilted empirical risk minimization ( TERM ) . TERM encompasses a family of objectives , parameterized by a real-valued hyperparameter , t. For t P Rz0 , the t-tilted loss ( TERM objective ) is given by : rRpt ; θq : “ 1 t log ˆ 1 N ÿ iPrNs etfpxi ; θq ˙ . ( 2 ) TERM generalizes ERM as the 0-tilted loss recovers the average loss , i.e. , rRp0 , θq “ Rpθq.1 It also recovers other popular alternatives such as the max-loss ( tÑ ` 8 ) and min-loss ( tÑ´8 ) ( Lemma 2 ) . For tą0 , the objective is a common form of exponential smoothing , used to approximate the max ( Kort & Bertsekas , 1972 ; Pee & Royset , 2011 ) . Variants of tilting have been studied in several contexts , ˚Equal contribution . 1 rRp0 ; θq is defined in ( 14 ) via the continuous extension of Rpt ; θq . including robust regression ( Wang et al. , 2013 ) ptă0q , importance sampling ( Wainwright et al. , 2005 ) , sequential decision making ( Howard & Matheson , 1972 ; Nass et al. , 2019 ) , and large deviations theory ( Beirami et al. , 2018 ) . However , despite the rich history of tilted objectives , they have not seen widespread use in machine learning . In this work , we aim to bridge this gap by : ( i ) rigorously studying the objective in a general form , and ( ii ) exploring its utility for a number of ML applications . Surprisingly , we find that this simple extension to ERM is competitive for a wide range of problems . To highlight how the TERM objective can help with issues such as outliers or imbalanced classes , we discuss three motivating examples below , which are illustrated in Figure 1 . ( a ) Point estimation : As a first example , consider determining a point estimate from a set of samples that contain some outliers . We plot an example 2D dataset in Figure 1a , with data centered at ( 1,1 ) . Using traditional ERM ( i.e. , TERM with t “ 0 ) recovers the sample mean , which can be biased towards outlier data . By setting t ă 0 , TERM can suppress outliers by reducing the relative impact of the largest losses ( i.e. , points that are far from the estimate ) in ( 2 ) . A specific value of t ă 0 can in fact approximately recover the geometric median , as the objective in ( 2 ) can be viewed as approximately optimizing specific loss quantiles ( a connection which we make explicit in Section 2 ) . In contrast , if these ‘ outlier ’ points are important to estimate , setting t ą 0 will push the solution towards a point that aims to minimize variance , as we prove more rigorously in Section 2 , Theorem 4 . ( b ) Linear regression : A similar interpretation holds for the case of linear regression ( Figure 2b ) . As tÑ ´8 , TERM finds a line of best while ignoring outliers . However , this solution may not be preferred if we have reason to believe that these ‘ outliers ’ should not be ignored . As tÑ ` 8 , TERM recovers the min-max solution , which aims to minimize the worst loss , thus ensuring the model is a reasonable fit for all samples ( at the expense of possibly being a worse fit for many ) . Similar criteria have been used , e.g. , in defining notions of fairness ( Hashimoto et al. , 2018 ; Samadi et al. , 2018 ) . We explore several use-cases involving robust regression and fairness in more detail in Section 5 . ( c ) Logistic regression : Finally , we consider a binary classification problem using logistic regression ( Figure 2c ) . For t P R , the TERM solution varies from the nearest cluster center ( tÑ´8 ) , to the logistic regression classifier ( t “ 0 ) , towards a classifier that magnifies the misclassified data ( tÑ ` 8 ) . We note that it is common to modify logistic regression classifiers by adjusting the decision threshold from 0.5 , which is equivalent to moving the intercept of the decision boundary . This is fundamentally different than what is offered by TERM ( where the slope is changing ) . As we show in Section 5 , this added flexibility affords TERM with competitive performance on a number of classification problems , such as those involving noisy data , class imbalance , or a combination of the two . Contributions . In this work , we explore TERM as a simple , unified framework to flexibly address various challenges with empirical risk minimization . We first analyze the objective and its solutions , showcasing the behavior of TERM with varying t ( Section 2 ) . Our analysis provides novel connections between tilted objectives and superquantile methods . We develop efficient methods for solving TERM ( Section 4 ) , and show via numerous case studies that TERM is competitive with existing , problemspecific state-of-the-art solutions ( Section 5 ) . We also extend TERM to handle compound issues , such as the simultaneous existence of noisy samples and imbalanced classes ( Section 3 ) . Our results demonstrate the effectiveness and versatility of tilted objectives in machine learning . 2 TERM : PROPERTIES & INTERPRETATIONS . To better understand the performance of the t-tilted losses in ( 2 ) , we provide several interpretations of the TERM solutions , leaving the full statements of theorems and proofs to the appendix . We make no distributional assumptions on the data , and study properties of TERM under the assumption that the loss function forms a generalized linear model , e.g. , L2 loss and logistic loss ( Appendix D ) . However , we also obtain favorable empirical results using TERM with other objectives such as deep neural networks and PCA in Section 5 , motivating the extension of our theory beyond GLMs in future work . General properties . We begin by noting several general properties of the TERM objective ( 2 ) . Given a smooth fpx ; θq , the t-tilted loss is smooth for all finite t ( Lemma 4 ) . If fpx ; θq is strongly convex , the t-tilted loss is strongly convex for t ą 0 ( Lemma 5 ) . We visualize the solutions to TERM for a toy problem in Figure 2 , which allows us to illustrate several special cases of the general framework . As discussed in Section 1 , TERM can recover traditional ERM ( t “ 0 ) , the max-loss ( tÑ ` 8 ) , and the min-loss ( tÑ´8 ) . As we demonstrate in Section 5 , providing a smooth tradeoff between these specific losses can be beneficial for a number of practical use-cases— both in terms of the resulting solution and the difficulty of solving the problem itself . Interestingly , we additionally show that the TERM solution can be viewed as a smooth approximation to a superquantile method , which aims to minimize quantiles of losses such as the median loss . In Figure 2 , it is clear to see why this may be beneficial , as the median loss ( orange ) can be highly non-smooth in practice . We make these rough connections more explicit via the interpretations below . ( Interpretation 1 ) Re-weighting samples to magnify/suppress outliers . As discussed via the toy examples in Section 1 , the TERM objective can be tuned ( using t ) to magnify or suppress the influence of outliers . We make this notion rigorous by exploring the gradient of the t-tilted loss in order to reason about the solutions to the objective defined in ( 2 ) . Lemma 1 ( Tilted gradient , proof in Appendix B ) . For a smooth loss function fpx ; θq , ∇θ rRpt ; θq “ ÿ iPrNs wipt ; θq∇θfpxi ; θq , where wipt ; θq : “ etfpxi ; θq ř jPrNse tfpxj ; θq “ 1 N etpfpxi ; θq´ rRpt ; θqq . ( 3 ) From this , we can observe that the tilted gradient is a weighted average of the gradients of the original individual losses , where each data point is weighted exponentially proportional to the value of its loss . Note that t “ 0 recovers the uniform weighting associated with ERM , i.e. , wipt ; θq “ 1 { N . For positive t , it magnifies the outliers—samples with large losses—by assigning more weight to them , and for negative t , it suppresses the outliers by assigning less weight to them . ( Interpretation 2 ) Tradeoff between average-loss and min/max-loss . To put Interpretation 1 in context and understand the limits of TERM , a benefit of the framework is that it offers a continuum of solutions between the min and max losses . Indeed , for positive values of t , TERM enables a smooth tradeoff between the average-loss and max-loss ( as we demonstrate in Figure 10 , Appendix I ) . Hence , TERM can selectively improve the worst-performing losses by paying a penalty on average performance , thus promoting a notion of uniformity or fairness ( Hashimoto et al. , 2018 ) . On the other hand , for negative t , the solutions achieve a smooth tradeoff between average-loss and min-loss , which can have the benefit of focusing on the ‘ best ’ losses , or ignoring outliers ( Theorem 3 , Appendix D ) . Theorem ( Formal statement and proof in Appendix D , Theorem 3 ) . Let θ̆ptq be the minimizer of rRpt ; θq , referred to as t-tilted solution . Then , for t ą 0 , max-loss , pRpθ̆ptqq , is non-increasing with t while the average loss , Rpθ̆ptqq , is non-decreasing with t. ( Interpretation 3 ) Empirical bias/variance tradeoff . Another key property of the TERM solutions is that the empirical variance of the loss across all samples decreases as t increases ( Theorem 4 ) . Hence , by increasing t , it is possible to trade off between optimizing the average loss vs. reducing variance , allowing the solutions to potentially achieve a better bias-variance tradeoff for generalization ( Bennett , 1962 ; Hoeffding , 1994 ; Maurer & Pontil , 2009 ) ( Figure 10 , Appendix I ) . We use this property to achieve better generalization in classification in Section 5 . We also prove that the cosine similarity between the loss vector and the all-ones vector monotonically increases with t ( Theorem 5 ) , which shows that larger t promotes a more uniform performance across all losses and can have implications for fairness defined as representation disparity ( Hashimoto et al. , 2018 ) ( Section 5.2 ) . Theorem ( Formal statement and proof in Appendix D , Theorem 4 ) . Let fpθq : “ pfpx1 ; θqq , . . . , fpxN ; θqq be the loss vector for parameter θ . Then , the variance of the vector fpθ̆ptqq is non-increasing with t while its average , i.e. , Rpθ̆ptqq , is non-decreasing with t. ( Interpretation 4 ) Approximate superquantile method . Finally , we show that TERM is related to superquantile-based objectives , which aim to minimize specific quantiles of the individual losses that exceed a certain value ( Rockafellar et al. , 2000 ) . For example , optimizing for 90 % of the individual losses ( ignoring the worst-performing 10 % ) could be a more reasonable practical objective than the pessimistic min-max objective . Another common application of this is to use the median in contrast to the mean in the presence of noisy outliers . As we discuss in Appendix G , superquantile methods can be reinterpreted as minimizing the k-loss , defined as the k-th smallest loss of N ( i.e. , 1-loss is the min-loss , N -loss is the max-loss , pN´1q { 2-loss is the median-loss ) . While minimizing the k-loss is more desirable than ERM in many applications , the k-loss is non-smooth ( and generally non-convex ) , and is challenging to solve for large-scale problems ( Jin et al. , 2020 ; Nouiehed et al. , 2019b ) . Theorem ( Formal statement and proof in Appendix G , Theorem 10 ) . The quantile of the losses that exceed a given value is upper bounded by a smooth function of the TERM objective . Further , the t-tilted solutions are good approximate solutions of the superquantile ( k-loss ) optimization .

Summary: This paper considers a unified framework named TERM for addressing a bunch of problems arising in the simple averaged empirical minimization. By leveraging the key hyper-parameter t in the TERM loss, it can recover the original average loss and approximate robust loss, min/max loss, and the superquantile loss, etc. The authors also propose gradient-based optimization algorithms for solving the TERM problem.