When False Positive is Intolerant: End-to-End  Optimization with Low FPR for Multipartite Ranking

1 Introduction . The multipartite ranking is a multi-class extension of bipartite ranking , aiming to sort a dataset with multiple discrete labels in proper order . Different from general multi-class classification , the categories in multipartite ranking are arranged according to a certain attribute . In the past decade , multipartite ranking is studied in a variety of application scenarios in the field of computer vision , including age estimation [ 33 ] , monocular depth estimation [ 12 ] and aesthetic visual analysis [ 32 ] , etc . Previous literature [ 13 , 29 , 35 , 37 , 38 ] uses Area Under the ROC Curve ( AUC ) as the evaluation metric for multipartite ranking . Intuitively , Hanley & McNeil [ 18 ] prove that AUC measures the probability expectation that positive instances are sorted higher than negative instances . This metric ∗Corresponding authors . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . is less sensitive to class imbalance than other metrics like accuracy [ 41 ] . Beside AUC , other common used metrics include accuracy and mean square error ( MSE ) ( see Sec . 2 ) . However , do these metrics always reflect the model performance in different scenarios properly ? This problem has more significance under False Positive Rate ( FPR ) sensitive application scenarios , i.e. , predictions leading to high false positive rates are intolerant . In these cases , the metrics mentioned above can not properly measure the desired performance , especially when the expected FPR is low . One example of such scenarios is medical diagnosis [ 24 , 25 ] . Considering diseased cases as negatives , more patients are missed if FPR is higher . Another example of churn in the telecommunications industry is provided by Mozer et al. [ 31 ] . Based on the above consideration , our interest is to optimize the True Positive Rate ( TPR ) at a fixed False Positive Rate ( FPR ) for multipartite ranking , denoted as TPR @ FPR . For practical reasons described above , the chosen FPR is in general small . Despite the practical advantages , it is infeasible to optimize such a metric in an end-to-end manner . To be concrete , the main challenges are as follows : ( C1 ) TPR @ FPR is a constrained combinatorial optimization problem , where the gradients of the objective function are not directly available even when proper surrogate losses are utilized . ( C2 ) The loss function could not be expressed as a sum of independent instance-wise terms , making the stochastic optimization unavailable . To solve these problems , we propose a novel method named Cross-Batch Approximation for Multipartite Ranking ( CBA-MR ) . In a nutshell , the main contributions of this paper are summarized as follows : • We consider a new evaluation metric TPR @ FPR for multipartite ranking in FPR sensitive scenarios . This metric focuses on model performance with a low FPR , which is consistent with practical requirements . • To optimize the TPR @ FPR metric , we propose Cross-Batch Approximation for Multipartite Ranking . The main idea is to relax the tricky hard constraint on FPR as a probability-based loss function . Specifically , we propose a random-walk model to measure the possibility that a negative instance will affect short-term FPR value . On top of this model , we propose a differentiable loss function that automatically focuses on hard negative instances . • Motivated by the slow feature drift phenomenon , we introduce a nonparametric cross-batch cache module to approximate the ranking of negative examples rapidly . An upper bound of the approximation error is provided to exhibit the feasibility of approximation . 2 Related Work . Multipartite Ranking & Ordinal Regression . Existing methods on ordinal regression can be roughly divided into two technical routes [ 16 ] : classification-based approaches and threshold-based approaches . Classification-based approaches transform the ordinal regression problem into the general classification problem . The most common seen transformation is F & H [ 11 ] , which decomposes the M classes original regression into M − 1 binary problems , i.e. , for each k ∈ { 1 , · · · , M − 1 } , whether the rank of the category of an instance is larger than k ? This method has been applied in age estimation [ 33 ] and depth estimation [ 12 ] . Lin and Li [ 23 ] define an extended binary example x ( k ) = ( x , k ) , y ( k ) = 2I [ y < k ] − 1 for each example ( x , y ) and sub-problem k ∈ { 1 , · · · , M − 1 } , and learns a single binary classifier on the extended data . Most of the above methods use manually extracted features and classifiers based on support vector machine ( SVM ) [ 2 ] , while the neural network is first adopted by Cheng et al . [ 4 ] , and further developed in [ 9 , 27 , 1 ] . The basic idea of threshold-based approaches is to sort the samples firstly , and then determine the categories of samples by thresholds . Multipartite ranking and threshold-based ordinal regression are essentially similar , since the purpose of both problems is to learn scores for instances indicating the ranking . The main difference between these two problems is that multipartite ranking only requires to sort the queries , while ordinal regression needs to further seek discrete labels . Uematsu & Lee [ 38 ] prove that some solutions to ordinal regression [ 22 , 6 , 30 ] are special cases of multipartite ranking . Along this technical route , Chu and Ghahramani [ 5 ] assume that the latent function from the input space to the output space is a Gaussian process , and solve the optimal parameters through a multilayer perceptron ( MLP ) . Liu et al . [ 28 ] further extend a non-conjugate likelihood to estimate parameters of the Gaussian process in a stochastic mini-batch manner . Quevedo et al . [ 35 ] further develop F & H [ 11 ] , exploring another decomposition based on decision directed acyclic graph . Other early literature modifies SVM or ensemble models from classification problems . Chu and Keerthi propose SVOR [ 6 ] , which learns a linear transformation , and classifies the samples into corresponding categories according to the distance to the discriminant plane . Lin and Li et al . [ 22 ] propose an ensemble model structure based on large-margin bounds . Deep learning is also applied along this technical route [ 26 ] . Although various methods have been proposed for ordinal regression or multipartite ranking , these methods still leave the FPR sensitive scenarios remaining , since they focus on metrics like AUC , accuracy or MSE . To fill this gap , we consider a new metric TPR @ FPR that focuses on low FPRs , and propose a novel framework to optimize it in an end-to-end manner . Constrained Optimization on Classification . Optimizing the TPR @ FPR metric can be viewed as a constrained optimization problem . Existing literature provides solutions to some constrained optimization problems . For example , to maximize precision at the top of a ranked list ( P @ K ) , Kar et al . [ 21 ] propose surrogate functions for this problem . However , this work fails when constraints depend on the ground truth labels . To solve this kind of constraints in a special case , i.e. , linear models , Goh et al . [ 15 ] propose to use ramp penalty to accurately quantify costs . For non-linear models , Eban et al . [ 10 ] introduce Lagrange multiplier to transform the problem into a min-max problem , and solve it with iterative SGD . Mackey et al . [ 29 ] use kernel quantile estimator to approximate the constraint . At first glance , TPR @ FPR can be solved similarly to these two methods , but the convergence can not be guaranteed when the model is non-linear or even non-convex , and the mini-batch-based Lagrange multiplier has a large update variance . Moreover , the method of Mackey et al . [ 29 ] will lead to training collapse when the excepted FPR is low . Recent work [ 19 , 20 ] proposes to apply deep neural networks to learn a surrogate function for black-box metrics . However , this route also requires gradients of the metric . In short , these existing methods can not solve our problem through simple modifications . In this work , to solve TPR @ FPR optimization problem , we propose an end-to-end framework based on cross-batch approximation . 3 Methodology . In this section , we will present the details of our proposed framework . In Sec . 3.1 , we provide a formal definition of the multipartite ranking problem and the TPR @ FPR metric . In Sec . 3.2 , we present an overview of our proposed method . Finally , in Sec . 3.3 and Sec . 3.4 , we describe the details of the cross-batch ranking estimation and the cross-batch cache module , respectively . 3.1 Problem Settings and Metrics . Notations . Consider a dataset with N instances and M categories : D = { ( xi , yi ) |xi ∈ X , yi ∈ Y } Ni=1 , where X is the input image space and Y = { r1 , r2 , · · · , rM } is the corresponding category space . In multipartite ranking , the categories in Y satisfy an ordered ranking r1 ≺ r2 ≺ · · · ≺ rM . Generally , the objective is to learn a score function f : X → R , such that the instances with label ri are assigned higher scores than those with label rj , whenever ri � rj . Problem Decomposition . As explained in Sec . 1 , we decompose the multipartite ranking problem into M−1 bipartite ranking sub-problems . The k-th sub-problem is to perform a bipartite ranking with D+k = { ( xi , yi ) ∈ D|yi � rk } being the set of positive instance , and D−k = { ( xi , yi ) ∈ D|yi � rk } being that of the negative instance . For the sake of convenience , we denote the i-th instance from D+k and its corresponding score as xk+i and sk+i respectively , and denote xk−i , sk−i as those for the instances from D−k . Given such a decomposition , we then define the TPR @ FPR metric based on the performance of the bipartite sub-problems : Definition 1 . Given a bipartite ranking dataset with a positive subset D+ = { x+i } i and a negative subset D− = { x−i } i , the TPR at a fixed FPR α ( TPR @ FPR ) for a scoring function fθ is defined as TPR @ FPR ( D+ , D− , fθ , τ ( α ) ) = 1 |D+| |D+|� i=1 I [ s+i > τ ( α ) ] , ( 1 ) where the indicator function I [ x ] equals to 1 if x is true and 0 otherwise . s+i , s − i refer to fθ ( x + i ) , fθ ( x − i ) , respectively . τ ( α ) is the minimum classification threshold ensuring that the FPR is smaller than α . Mathematically , this could be expressed as : τ ( α ) = min τ̂ τ̂ s.t . 1 |D−| |D−|� i=1 I [ s−i > τ̂ ] ≤ α . ( 2 ) Based on our decomposition scheme , TPR @ FPR for multipartite ranking can be naturally expressed as a weighted average of TPR @ FPR for bipartite ranking defined as Def . 2 . Definition 2 . Given a multipartite ranking dataset as defined above , the True Positive Rate at a fixed False Positive Rate ( TPR @ FPR ) for a scoring function fθ is defined as TPR @ FPR ( D , fθ , α ) = 1 �M−1 k=1 λk M−1� k=1 λkTPR @ FPR ( D+k , D−k , fθ , τk ( α ) ) , ( 3 ) where λk is the weight of the k-th sub-problem . Given the definitions above , our goal is to maximize the TPR @ FPR ( D , fθ , α ) . It is abbreviated as TPR @ FPR in the rest of this paper if it does not lead to ambiguity . As mentioned in Sec . 1 , optimizing such a complicated metric is non-trivial especially in the end-toend training regime . In the rest of this section , we will propose an efficient method to optimize the metric approximately .

Summary: In this paper, the authors studied a particular metric for M-class multipartite ranking, with the particular motivation of FPR-sensitive applications in mind. Given that the combinatorial objective is hard to solve exactly, a series of relaxations are introduced / assumptions brought in for the algorithm to be amenable to mini-batch gradient updates. The high-level approach taken is to turn the problem into M-1 bipartite ranking sub-problems, after which a probability-based differentiable surrogate loss function are used and a cache module are leveraged to efficiently approximate the ranking of examples. Numerical experiments are also conducted on a series of real-world datasets.