Metric-Optimized Example Weights

1 INTRODUCTION . In machine learning applications , each training example is usually weighted equally during training . Uniform weighting delivers satisfactory performance when the noise is homoscedastic , training examples follow the same distribution as the test data , and when the training loss matches the test metric . However , these requirements are often violated in real-world applications . For example , the training loss does not always correlate sufficiently with the test metric we care about , such as revenue impact or some fairness metrics . This discrepancy between the loss and metric can lead to inferior test performance ( Cortes & Mohri , 2004 ; Perlich et al. , 2003 ; Davis & Goadrich , 2006 ) . There are many proposals to address some of these issues , as we review in Section 2 , but none of them address all of them at once . In this paper , we propose a framework to learn a weighting function , which estimates training example weights that produce a machine learned model with optimal test metrics . Our proposal , Metric-Optimized Example Weights ( MOEW ) , solves the aforementioned issues simultaneously and is suitable for any standard or customized test metrics . To use MOEW , we need a small set of labeled validation examples that are identically distributed as test examples . By learning a weighting function , MOEW effectively rescales the loss of each training example , and reshapes the total loss function such that its optima better match the optima of the testing metric . As an illustrative example , Figure 1 shows a simulated non-IID toy dataset and our learned example weighting . The ground truth decision boundary is the diagonal line from the upper left to lower right corners . The feature values in the training and validation/test data follow a beta distribution : specifically , in the training data , ( x1 , x2 ) ∼ ( β ( 2 , 1 ) , β ( 2 , 1 ) ) , whereas in the validation/test data , ( x1 , x2 ) ∼ ( β ( 1 , 2 ) , β ( 1 , 2 ) ) . The goal is to maximize precision at 95 % recall on the test distribution . With uniform example weights during training , we get 20.8 % precision at 95 % recall on the test data . With importance weighting , we get 21.8 % precision at 95 % recall . With MOEW , we obtain 23.2 % precision at 95 % recall ( the Bayes classifier achieves around 25.0 % precision at 95 % recall ) . Comparing Figures 1c to 1d , one can see that MOEW learned to upweight the negative training examples , and upweights examples closer to the center . 2 RELATED WORK . Our proposal simultaneously addresses three potential issues : non-constant noise levels across training examples , non-identical distributions between training and test examples ( a.k.a . covariate shift or domain shift ) , and training and test objectives mismatch . Several existing methods summarized below address some of these issues , but none of them provide a method to address all of them at once . Maximum likelihood based inference is commonly used to address the issue with non-constant noise levels in the training data . In addition , several common machine learning techniques , such as ensemble and graph-based methods , are found to be robust to binary label noise ; see Frenay & Verleysen ( 2014 ) for a survey of these methods . To address the issue of covariate shift , classical approaches include propensity score matching ( Rosenbaum & Rubin , 1983 ; 1985 ; Lunceford & Davidian , 2004 ) or importance weighting ( Robins et al. , 1994 ; Shimodaira , 2000 ; Sugiyama et al. , 2008 ; Kanamori et al. , 2009 ) . These techniques can also be adapted for model selection tasks ( Sugiyama et al. , 2007 ) . In addition , Bickel et al . ( 2009 ) proposed a discriminative approach of learning under covariate shift . To address the issue of loss/metric mismatch , researchers have formulated several plug-in approaches and surrogate loss functions . These approaches can be used to optimize some non-decomposable ranking based metrics , such as AUC ( Ferri et al. , 2002 ; Yan et al. , 2003 ; Cortes & Mohri , 2004 ; Freund et al. , 2003 ; Herschtal & Raskutti , 2004 ; Rudin & Schapire , 2009 ; Zhao et al. , 2011 ) , F score ( Joachims , 2005 ; Jansche , 2005 ; Dembczynski et al. , 2013 ) , and other ranking metrics ( Joachims , 2005 ; Burges et al. , 2007 ; Yue et al. , 2007 ; Eban et al. , 2017 ; Narasimhan et al. , 2014 ; Kar et al. , 2014 ; 2015 ; Narasimhan et al. , 2015 ) . Our work differs in that the test metric can be a black box , and we adapt for it by learning a corresponding weighting function over the examples . 3 METRIC ADAPTIVE WEIGHT OPTIMIZATION . We propose learning an example weighting function , trained on validation scores for different example weightings . Our proposal is suitable for any customized metrics . 3.1 OVERVIEW . Let T and V denote the sets of training and validation examples , respectively . We assume V is drawn IID with the test set , but that T and V may not be IID . Consider a classifier or regressor , h ( x ; θ ) ∈ H for x ∈ RD , which is parameterized by θ . Let y ∈ R be the label of the data , and ŷ ∈ R be the predicted score . The optimal θ∗ is obtained by optimizing a weighted loss function L ( ŷ , y ) : θ∗ ( α ) = argmin θ ∑ j∈T w ( xj , yj ; α ) L ( h ( xj ; θ ) , yj ) , ( 1 ) where α parameterizes the example weighting function w. Our goal is to learn an optimal example weighting function w ( detailed below ) . Let MV ( θ∗ ( α ) ) be the metric of interest evaluated for the model h with parameters θ∗ ( α ) on the validation dataset V . Without loss of generality , we assume a larger validation metric is more desirable . We propose learning the example weighting function w ( x , y ; α∗ ) ∈ W , such that α∗ = argmax α MV ( θ ∗ ( α ) ) . ( 2 ) That is , we propose finding the optimal parameters α∗ for the example weighting function w ( x , y ; α∗ ) such that the model trained with this example weighting function achieves the best validation score . To simplify the notation , we note that in this paper , the θ parameters always depend on the weight parameters α and henceforth we simplify the notation θ ( α ) to θ . The metric MV ( θ ) as a function of θ and α is likely non-convex and non-differentiable , which makes it hard to directly optimize MV ( θ ) through e.g. , SGD . Instead , we adopt an iterative algorithm to optimize for θ∗ and α∗ , which is detailed in Algorithm 1 . Specifically , we start with a random sample of K choices of weight parameters , α0 = { α01 , . . . , α0K } . For each of the randomly generated weight parameters α0l , 1 ≤ l ≤ K , we solve equation 1 to obtain the K corresponding model parameters θ0l , and use those to compute the K corresponding validation metrics , MV ( θ0l ) . Then , based on the batch of K weight parameters and validation metrics , { ( α01 , MV ( θ01 ) ) , . . . , ( α0K , MV ( θ0K ) ) } , we determine a new set of K weight parameter candidates , α1 , and repeat this process for a pre-specified number of iterations , B . At the end , we choose the candidate α that produced the best validation metric . Algorithm 1 Get optimal α∗ and θ∗ 1 : procedure GET OPTIMAL α∗ AND θ∗ 2 : T ← training data 3 : V ← validation data 4 : B ∈ N+ ← number of batches of weight parameters 5 : α0 = { α01 , . . . , α0K } ← randomly generated initial batch of weight parameters 6 : for i = 0 , 1 , . . . , B − 1 do 7 : parfor l ∈ { 1 , . . . , K } do 8 : θil ← argminθ ∑ j∈T w ( xj , yj ; α i l ) L ( h ( xj ; θ ) , yj ) 9 : end parfor 10 : si ← { ( αi1 , MV ( θi1 ) ) , . . . , ( αiK , MV ( θiK ) ) } 11 : αi+1 ← GetCandidateAlphas ( s0 , . . . , si ) 12 : end for 13 : b∗ , k∗ ← argmaxb∈ { 0 , ... , B−1 } , k∈ { 1 , ... , K } MV ( θbk ) 14 : α∗ , θ∗ ← αb∗k∗ , θb ∗ k∗ 15 : end procedure In the following subsections , we describe the function classW of the weight model , and the subroutine used in Algorithm 1 : GetCandidateAlphas . 3.2 FUNCTION CLASS FOR THE EXAMPLE WEIGHTING MODEL . Recall that the final optimal α∗ is taken to be the best out of B ×K samples of α ’ s . To ensure a sufficient coverage of the weight parameter space for a small number of B ×K validation samples , and to avoid overfitting to the validation data , we found it best to use a function classW with a small number of parameters . There are many reasonable strategies for definingW . In this paper , we chose the functional form : w ( x , y ; α ) = c π ( y ) σ ( z ( x , y ) > α ) , ( 3 ) where c ∈ R+ is a constant that normalizes the weights over ( a batch from ) the training set T , and σ ( z ( x , y ) > α ) is a sigmoid transformation of a linear function of a low-dimensional embedding z of ( x , y ) . In the experiments discussed in Section 4 , we used the standard importance function π ( y ) = pV ( y ) pT ( y ) , where pV ( y ) and pT ( y ) denote the probability density function of y in the validation and training data , respectively . In practice , w can take many other forms , and π ( y ) can be substituted with any baseline weighting function , which can be considered as an initialization of MOEW . While there are many possible ways to form a low-dimensional embedding z of ( x , y ) , we choose to use an autoencoder ( Hinton & Salakhutdinov , 2006 ) to form an embedding , as it can be applied to a wide range of problems . Specifically , to train the autoencoder z ( x , y ; ψ∗ ) , we minimize the weighted sum of the reconstruction loss for x and y : ψ∗ = argmin ψ ∑ j∈T λLx ( z ( xj , yj ; ψ ) , xj ) + ( 1− λ ) Ly ( z ( xj , yj ; ψ ) , yj ) , ( 4 ) where Lx is an appropriate loss for the feature vector x and Ly is an appropriate loss for the label y . The hyperparameter λ in equation 4 is used to adjust the relative importance of features and the label in the embedding : using λ = 0 is similar to weighting based solely on the value of the label . For all the experiments in this paper we used a fixed λ = 0.5 , but it may need tuning for some problems .

Summary: The authors propose to optimize a black-box (validation/test) metric by learning to re-weight the training examples. The weights are calculated from a linear model on an auto-encoder-computed embedding, and the parameters of the linear model is found by an Gaussian-Process-Regression-(UCB)-guided global optimization procedure. Experimental results demonstrate that the learnt weights outputs uniform weights.