When False Positive is Intolerant: End-to-End  Optimization with Low FPR for Multipartite Ranking
Authors: Peisong Wen
Qianqian Xu
Zhiyong Yang
Yuan He
Qingming Huang
Venue: NeurIPS 2021 Poster (2021)
Keywords: multipartite ranking
computer vision
deep learning
constrained metric

Abstract: Multipartite ranking is a basic task in machine learning, where the Area Under the receiver operating characteristics Curve (AUC) is generally applied as the evaluation metric. Despite that AUC reflects the overall performance of the model, it is inconsistent with the expected performance in some application scenarios, where only a low False Positive Rate (FPR) is meaningful. To leverage high performance under low FPRs, we consider an alternative metric for multipartite ranking evaluating the True Positive Rate (TPR) at a given FPR, denoted as TPR@FPR. Unfortunately, the key challenge of direct  TPR@FPR optimization is two-fold: \textbf{a)} the original objective function is not differentiable, making gradient backpropagation impossible; \textbf{b)} the loss function could not be written as a sum of independent instance-wise terms, making mini-batch based optimization infeasible. 
     To address these issues, we propose a novel framework on top of the deep learning framework named \textit{Cross-Batch Approximation for Multipartite Ranking (CBA-MR)}. In face of \textbf{a)},  we propose a differentiable surrogate optimization problem where the instances having a short-time effect on FPR are rendered with different weights based on the random walk hypothesis. To tackle \textbf{b)}, we propose a fast ranking estimation method, where the full-batch loss evaluation is replaced by a delayed update scheme with the help of an embedding cache. Finally, experimental results on four real-world benchmarks are provided to demonstrate the effectiveness of the proposed method.

When False Positive is Intolerant: End-to-End Optimization with Low FPR for Multipartite Ranking
Peisong Wen 1.2
Qiangian Xul
Zhiyong Yang² Qingming Huang 1,2,4,5
Yuan He
Key Lab of Intell. Info. Process., Inst. of Comput. Tech., CAS School of Computer Science and Tech., University of Chinese Academy of Sciences
Allibaba Group
BDKM, University of Chinese Academy of Sciences
Peng Cheng Laboratory
(wenpeisong20z, augeelsoongdoo,uuiiigiiaaaaaaat aug.ae/aiiiiiiiinicii.i.....
(yangzhiyong21, Huchas ac cn
byyuam.yyaalibaba-ime
Abstract
Multipartite ranking is basic task in machine learning, where the Area Under the receiver operating characteristies Curve (AUC) is generally applied as the evaluation metric. Despite that AUC reflects the overall performance of the model, it is inconsistent with the expected performance in some application scenarios, where only low False Positive Rate (FPR) is meaningful. To leverage high perfor- mance under low FPRs, we consider an alternative metric for multipartite ranking evaluating the True Positive Rate (TPR) at given FPR, denoted as TPR @ FPR. Unfortunately, the key challenge of direct TPR @ FPR optimization is two-fold: a) the original objective function not differentiable, making gradient backpropaga- tion impossible; b) the loss function could not be written as a sum of independent instance- wise terms, making mini-batch based optimization infeasible. To address these issues, propose novel framework on top of the deep learning frame- work named Cross-Batch Approximation for Multipartite Ranking ( (CBA-MR), In face of a), we propose differentiable surrogate optimization problem where the instances having short-time effect on FPR are rendered with different weights based on the random walk hypothesis. To tackle b), propose fast ranking estimation method, where the full-batch loss evaluation is replaced by a delayed update scheme with the help of embedding cache. Finally, experimental results on four real-world benchmarks are provided to demonstrate the effectiveness of the proposed method.
Introduction
The multipartite ranking is a multi-class extension of bipartite ranking, aiming to sort dataset with multiple discrete labels in proper order. Different from general multi-class classification, the categories in multipartite ranking are arranged according to a certain attribute. In the past decade, multipartite ranking is studied in variety of application scenarios in the field of computer vision, including age estimation [33, monocular depth estimation [12] and aesthetic visual analysis [32], etc.
Previous literature [13, 29, 35, 37, 38] uses Area Under the ROC Curve (AUC) the evaluation metric for multipartite ranking. Intuitively, Hanley & McNeil [18] prove that AUC measures the probability expectation that positive instances are sorted higher than negative instances. This metric
"Corresonning authors.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

is less sensitive to class imbalance than other metrics like accuracy [41]. Beside AUC, other common used metrics include accuracy and mean square error (MSE) (see Sec. 2).
However, do these metrics always reflect the model performance in different scenarios properly?
This problem has more significance under False Positive Rate (FPR) sensitive application scenarios, i.e., predictions leading to high false positive rates are intolerant. In these cases, the metrics mentioned above cannot properly measure the desired performance, especially when the expected FPR is low. One example of such scenarios is medical diagnosis [24, 25J. Considering diseased cases as negatives, more patients missed if FPR is higher. Another example of churn in the telecommunications industry is provided by Mozer et al.
Based on the above consideration, our interest is to optimize the True Positive Rate (TPR) at fixed False Positive Rate (FPR) for multipartite ranking, denoted as TPR @ FPR. For practical reasons described above, the chosen FPR is in general small. Despite the practical advantages, it is infeasible to optimize such a metric in an end-to-end manner. To be concrete, the main challenges are follows:
(C1) TPR @ FPR is a constrained combinatorial optimization problem, where the gradients of the objective function are not directly available even when proper surrogate losses are utilized.
(C2) The loss function could not be expressed as sum of independent instance-wise terms, making the stochastic optimization unavailable.
To solve these problems, we propose a novel method named Gross-Batch Approximation for Mul- ripartite Ranking (CBA-MR). In nutshell, the main contributions of this paper are summarized as follows:
We consider a new evaluation metric TPR @ FPR for multipartite ranking FPR sensitive scenarios. This metric focuses on model performance with a low FPR, which is consistent with practical requirements
To optimize the TPR @ FPR metric, we propose Cross-Batch Approximation for Multipartite Ranking. The main idea is to relax the tricky hard constraint on FPR as a probability- -based loss function. Specifically, a negative instance will affect short-term FPR value. On top of this model, we propose a differentiable loss function that automatically focuses hard negative instances.
propose random- walk model to measure the possibility that
Motivated by the slow feature drift phenomenon, we introduce nonparametric cross- batch cache module to approximate the ranking of negative examples rapidly. An upper bound of the approximation error is provided to exhibit the feasibility of approximation.
2 Related Work
Multipartite Ranking & Ordinal Regression. Existing methods on ordinal regression can be roughly divided into two technical routes [16]: dlassification-basee approaches and threshold-based approaches. Classification-basedd approaches transform the ordinal regression problem into the general classification problem. The most common seen transformation is F&H [1]]. which decomposes the M classes original regression into M - 1 binary problems, i.e., for each E {I. .M- 1}. whether the rank of the category of an instance is larger than k?? This method has been applied in age estimation[33 [33] and depth estimation [12]. Lin and Li [23] define an extended binary example (k) (ar, k), (k) = 27 < k] - for each example (ær, y) and sub-problem & E {1, M - 1), and learns single binary classifier on the extended data. Most of the above methods use manually extracted features and classifiers based on support vector machine (SVM) [2], while the neural network is first adopted by Cheng et al. [4], and further developed in [9, 27, 1].
The basic idea of threshold-basee approaches is to sort the samples firstly, and then determine the categories of samples by thresholds. Multipartite ranking and threshold- based ordinal regression are essentially similar, since the purpose of both problems ranking. The main difference between these two problems is that multipartite ranking only requires to sort the queries, while ordinal regression needs to further seek discrete labels. Uematsu Lee [38] prove that some solutions to ordinal regression [22, 6, 30] are special cases of multipartite ranking. Along this technical route, Chu and Ghahramani [5] assume that the latent function from the input space to the output space is a Gaussian process, and solve the optimal parameters through a multilayer
learn scores for instances indicating the

---

perceptron (MLP). Liu ef al. [28] further extend non-conjugate likelihood to estimate parameters of the Gaussian process in stochastic mini-batch manner. Quevedo er al.[35] further develop F&H[ 1], exploring another decomposition based on decision directed acyelic graph. Other early literature modifies SVM or ensemble models from classification problems. Chu and Keerthi propose SVOR [6], which learns a linear transformation, and classifies the samples into corresponding categories according to the distance to the discriminant plane. Lin and Li et al. [22] propose an ensemble model structure based on large-margin bounds. Deep learning is also applied along this technical route [26].
Although various methods have been proposed for ordinal regression or multipartite ranking, these methods still leave the FPR sensitive scenarios remaining, since they focus on metrics like AUC, accuracy or MSE. To fill this gap. we consider new metric TPR @ FPR that focuses on low FPRs, and propose novel framework to optimize it in an end-to-end manner.
Constrained Optimization Classification. Optimizing the TPR @ FPR metric be viewed as a constrained optimization problem. Existing literature provides solutions to some constrained optimization problems. For example, to maximize precision the top of ranked list (P@ K), Kar et al][21] propose surrogate functions for this problem. However, this work fails when constraints depend on the ground truth labels. To solve this kind of constraints a special case, i.e., linear models, Goh et al.[15] propose to use ramp penalty to accurately quantify costs. For non-linear models, Eban et al. [10] introduce Lagrange multiplier to transform the problem into min-man min-mmx problem, and solve it with iterative SGD. Mackey et al. [29] use kernel quantile estimator to approximate the constraint. At first glance, TPR @ FPR can be solved similarly these two methods, but the convergence cannot be guaranteed when the model is non-linear or even non-convex, and the mini-batch-baser Lagrange multiplier has a large update variance. Moreover, the method of Mackey et al. [29] will lead to training collapse when the excepted FPR is low. Recent work [19, 20] proposes to apply deep neural networks to learn a surrogate function for black-box metrics. However, this route also requires gradients of the metric.
In short, these existing methods cannot solve our problem through simple modifications. In this work, to solve TPR @ FPR optimization problem, we propose an end-to-end framework based on cross-batch approximation.
3 Methodology
In this section, we will present the details of our proposed framework. In Sec. 3.1, we provide a formal definition of the multipartite ranking problem and the TPR@ FPR metric. Sec. 3.2, we present an overview of our proposed method. Finally, in Sec. 3.3 and Sec. 3.4, we describe the details of the cross-batch ranking estimation and the cross-batch cache module, respectively.
3.1 Problem Settings and Metrics
Notations. Consider a dataset with N instances and M categories: D = (0i,,;)) E X, E V where X' is the input image space and y (r1, r2, (2,. ... TM} is the corresponding category space. In multipartite ranking, the categories in ) satisfy an ordered ranking T1 < <22 ..... < FM- Generally, the objective to learn a score function f X - IR such that the instances with label r; are assigned higher scores than those with label rj. whenever r; > Tj.
Problem Decomposition. As explained in Sec. 1, we decompose the multipartite ranking problem into M - bipartite ranking sub- aob-problems. The k--th sub-problem is to perform a bipartite ranking with DE (z,,, 9:) E D Dly; > r being the set of positive instance, and D = { 9/) Dlys - rk} being that of the negative instance. For the sake of convenience, we denote the i-th instance from Dr and its corresponding score as x and 8;
k+ respectively, and denote x;
as those for the
instances from DT--
Given such decomposition, we then define the TPR @TPR metric based on the performance of the bipartite sub- aaobbrolsems:
3

---

Iapat Iranges (x)
Feature Extractor (s((?))
Weighted Loss
Fimhedklngg (c)
Simolaritses
An overview of the proposed method. After extracting embeddings for input images, update the cache block, and estimate the rankings instances in caches. Then, the rankings of instances in the current mini-batch are estimated with similarities in caches. Finally, the rankings transformed into probabilities for weighting the loss function.
Figure 1:
Definition 1. Given a bipartite ranking dataset with a positive subset DD (xx }s and a negative ) for scoring function so is defined as subset D {5 )a, the TPR at fixed FPR a (TPR@ FPR)
g Σ
TPR@FPR(D* ,D, (o,
nst > >(a)],
DD+
where the indicator function [[x]] equals to 1 if x is true and O otherwise. ,5 refer to fo(x)), fo( fo(x;)
respectively. 7 T(a) is the minimum classification threshold ensuring that the FPR
smaller than a. Mathematically, this could be expressed as:
T(a) = min
s.t.
[ss Σ
S a.
(2)
D
Based on our decomposition scheme, TPR@FPR for multipartite ranking can be naturally expressed as weighted average of TPR @RR for bipartite ranking defined as Def. 2.
Definition 2. Given multipartite ranking dataset defined above, the True Positive Rate at a fixed False Positive Rate @TPR@ FPR) scoring function fe is defined as
M
TPR@FFR(D,fo.o)
TTRRFFFRR fo,
(3)
EkST
M-1
Ale
where A is the weight of the k-th sub-problem.
Given the definitions above, goal is to maximize the TPR TPR@FPR(D.fo... is abbreviated as TPR @ FPR in the rest of this paper if it does not lead to ambiguity.
As mentioned in Sec. 1, optimizing such a complicated metric is non-trivial especially in the end-to- end training regime. In the rest of this section, we will propose an efficient method to optimize the metric approximately.
3.2 Overview
An overview of our proposed method is shown in Fig. 1. In this paper, implement the scoring function fo as a deep neural network. Specifically, we first extract an embedding E Rdd from an input image æ with deep model Jæ-e I where & is the set of parameters. Afterward, learn the ranking and f.)-) = (-). Then our goal is to learn that approximately maximizes @
E IR with linear transformation
Rd: w Denote {Φ, w)
Recall the main challenges summarized in Sec. 1. To address (C1), we propose a cross-batch ranking estimation module make the gradient backpropagation feasible. Meanwhile, to address (C2), we propose cross-batch caching module to approximate the full-batch update in a few iterations.

---

3.3 Cross-Batch Ranking Estimation
Firstly, we consider a reformulation of the objective to remove the constraints. Here, we denote (() as the i-th largest score of the negative instances in the observed dataset D such that
(PPE))
Obviously (a) = k- ([a)) IT) and thus the objective function could then be reformulated as an unconstrained problem:
M-1 A
A D
min 0
Co,1 =
fo. (s,
sk
(Ta12,
(4)
Ek=1
where f (x) equals 1 if x: is negative and otherwise, and [.] is the ceiling function. Nonetheless, is easy to see that the 0-1 loss fo.1 here is not differentiable, and thus directly optimizing the original problem is NP-hard. To handle this issue, the 0-1 loss could be replaced by differentiable surrogate function l such as logistic loss, exponential loss or hinge loss [14]. Among them, we simply implement with logistic loss, i.e. {(x) log(1 + exp(-x)), which leads to the following objective:
M-1
min
L
E(s k+
(TODDE
(5)
kk=1
Unfortunately, even with a proper surrogate loss, there still two main challenges optimize the is non-trivial since the gradient backpropagation
objective: 1) optimizing the order statistic s ([a|P, (1)
is impossible without a full-batch scan; and 2) s ([a/D] is the only negative instance that explicitly appears in the loss function, making information from other negative instances totally ignored.
To address this issue, we turn to seek proper approximated objective function where all available negative instances are included. Note the objective function in Eq. (5) essentially tries to punish hard negative instances. In this sense, we expect the approximated objective function to focus on the top [a|PE] hard negatives instead of merely & k- "[aa/D]1)) To do this, we introduce a set of probabilistic
weights to attend to the hard negative instances. Since the model parameters are constantly changing the training process goes on, the score ranking also enjoys a dynamic nature. Hence, Our main idea is to consider all the negative examples that may reach the top [a ranking list within the potential candidates of the hard negatives. Formally, given the negative sample set A, we denote the probability that the score of a negative example æ reaches the top [[] ranking list within time window P(x; A). Then come a dynamic approximated problem as:
short period of training iterations
|Di p
min
L
Σ
E(s
k+
-=
Ta|D [T]-
(6)
M-1 k=
d
D
D
M
ID DD
min
L =
A
Σ
((s k+
M-1 Σ
Ak
D
k=
1=1
Here we reweight different loss terms by P(a ;D²) to focus on the potential hard negatives.
The problem then becomes how to estimate P( :D ). To discuss the approximation guarantec from Eq. (5) to Eq. (7), we provide a sufficient condition under which the objectives are consistent with the performance increasing. Here consider sub sub-problem and ignore the subscript k. Denote Ri as the empirical risk of TPR @ FPR (as defined Eq. (5)), and Ré as the empirical risk of the approximated version (as defined in Eq. (7)), i.e.,
5

---

R..
Σ
((oT 85),
(8)
D
m7 EDI
R DD+
Σ
l(s P ;D-),
(9)
ED
we provide sufficient condition for R RR in the following theorem (see Appendix A.l for the proof and the simulation experiment).
Theorem 1. Given a probability estimation P(: R + (0, and a surrogate loss func-
tion E()), denote lúj- l(s )j P(a;: ; ; ;. [a|D-11, Dj = =D /a and
Ig D+ Dzz denote ED+ [a] as the empirical expectation of over D+, and similarly for
B -DE;
=D; [z]. Ex eIalz]; denote = Var a! ED+ se ED-1:; 11, ,, (s s,,)/o, //,6,,
(s Σ 2jDo)/o. Assume
Σ Pj
2 2DED
We have RS, + R$ when a - the following conditions are met:
(a) E is convex, monotonically decreasing, ((0) > O, and (L)2 - C" 22 0:
(b) Σ 8ps ZjED-
(c) in1 Bo] < 0, where
A, = (ID- - (ext [el/1])
(10)
B. = (1 P2) Ex+
eD+ (1ool/BB (B/DBBB-DD/BC ED, (2)/11)0.
(11)
Based on Thm. 1, we provide an instantiation of P(a" A- DD) in the reset of this subsection. Our key idea here to measure such possibilities with the randomness of the relative ranking changes. Moreover, by a hypothesis test results listed in Appendix B, we observe that short-term ranking changes will tend stabilize at white noises as the model training becomes stable with statistical significance. Next, we will propose a basic assumption to capture such phenomenon. Concretely, denote a subset of the dataset as A S D. and the descending rank of an example E under parameters O as R(x; A, - [[f((z) < fo(x)),, the assumption is described as follows: Assumption 1. Denote the parameters timestamp as 0.. relatively short period of time T, the score ranking on certain set A obeys random walk. Equivalently, the difference sequence x;; A, 0.+1)- R(x; A, Ot),. is a white noise sequence, i.e., its expectation is and for any k 2 2, the k-order correlation with i is O.
Based on the above assumption, the probability P(æ; AA) can be viewed as the probability of random walk from R(x; A, 0) to [a] or vice versa from [a]|A]] to R(:; A, 0). When tends to convergence, can be seen that only three events occur each time: the rank of a specific instance decreases by 1, increases by 1, remains the same. In order to facilitate the analysis, we consider the following continuous form of the assumption.
Assumption 2. In relatively short period of time T, the normalized score ranking R(æ; A, /)/( of a specific instance & on a certain set A takes step per time, where three conditions are possible: 1) R( x; A, 0,+)))/ R(x; A, 0,)/ A| As with probability p/2; or 3) R(x; 02+ A1) |AA| = R(x: A.))/ A with probability p. Here, 0 < < 1.
R(x; A, 02) B,)/|A ||| + As with probability p/2; 2) R(x; A, Ou+Ae)/
Finally, based on Asmp. 2, the following proposition shows that P( A) enjoys a simple expression (see Appendix A. for proof):
Proposition Assume At + O, As + O, and 2 = BAt, the probability that the ranking of x reaches the rank [a]A[]] after period of time T is as follows:
1
(R(x: A, 0)/ a)2
PFx;; Rd
exp
(12)
V
20²

---

where o = v pP3T/22
According to Prop. 1, given the current ranking of negative examples, the probability that negative instance reaches the top ax [ ranking list in few iterations can be approximated with a Gaussian process. Moreover, for P(æ;:)) A), the only dependence on time T is from o, tunable hyperparameter. In this way, one can perform gradient backpropagation on top of the proposed loss function, as long proper estimation of rank R(x; A, 0) is available.
3.4 Cross-Bateh Cache
Despite that P(æ : D in Eq. (7) can be approximated with Eq. (12), the loss function is still not a sum of independent instance-wise terms, since the ranks are not independent of other instances. Moreover, re-calculating the ranking under the current parameter after each update will significantly slow down the training process, which motivates us to propose rapid estimation for the score ranks.
The main computational bottleneck is the frequent update of the full-batch feature embeddings from input images, where historical embeddings are usually considered as out-of-date once the update is finished. However, recent literature [3, 39] on deep metric learning shows that the embeddings generated in the past iterations are still valuable. Wang et al.[39] observe an interesting phenomenon called slow drift, Le., the embedding drift of the same sample within few steps will decrease the training is stable, thus the past embeddings can approximate the embeddings under the current model parameters.
Inspired by this phenomenon, we adopt a caching mechanism to perform a delayed embedding update. Concretely, for each category ls {I..... .M), ,, we up cache M₁ {é, (3(2x,)(x,,, 34)
D, 9: k), and Ma = U Mj, where Ø is the model parameters in past iterations. To balance the per-category sample size, the size of the cache for each category limited Ne. At each iteration, the oldest embeddings from M2 are popped out from the cache, and the newly generated embeddings are pushed into the corresponding cache.
Given mini-batch C 'D, we generate embeddings for instances in B, and then update the caches as described above. Afterward, the ranks of instance scores under the current parameters A estimated with the cache. Specifically, f(( ; ; 0), the rank of at- D is approximated by the ranks of embeddings the corresponding cache M₁ according the feature similarities ay:
i( ; ,) = Eajem; aj M, 0),
(13)
0g = exp(=> k- &,)i\/ Eerem; exp(->(e) ,t)),
(14)
where 7 is a hyperparameter, (:--) refers to inner product, and f((a; M₁ 0) is local estimation of R(x,; Di 0) over the cache list Mj. Afterward, P(æf;; D2) is obtained with Eq. (12):
1
(Ri(z);: Di M00M0MMMMM - a)²
)
P(af; Dw
exp
(15)
V2To
20
where a is tunable hyperparameter.
Moreover, we find that using Eq. (15) alone will lead significant fluctuations, since the magnitude of (P(x/-: : DF) might vary greatly across different mini-batches. Therefore, we use the normalized form to unify the magnitude of IP )
P( DT)
Pæ - //
Σ
P(x,;DE))
(16)
a E, nB
To ensure the approximation is acceptable, we have an upper bound for the error of ranking estimation Prop. 2. (See Appendix A.2 for proof); Proposition Assume that 9+ is L-Lipschitz continuous on Φ. and | 2112 S c, the error of ranking estimation on A is
RR A,0) R(I; <
Σ I > ww" (go(x) 90 (x. (xx w||l
(17)
iEA
where 0 = ( w).

---

Table 1: Statistics of three datasets.
Dataset
Training samples
Test / Val samples Categories
Application
DR
24,587
5,271
medical diagnosis
FTCe
16,591
3,556
age estimation
NSFW
38,822
8,321
content review
AgePrediction 152,295
32,639
7 age estimation
Prop. 2 shows that R(;:; A, 0) provides reasonable approximation of R(æ; A, 0), if the predictions of instances are sufficiently scattered. Therefore, we can estimate the ranks in M by R(æ: M 0), which can be obtained at ignorable cost of O(MN, (d + log( (N)) time complexity.
Together with cross-batch ranking estimation, we decompose the objective into instance independent items and optimize it in mini-batch manner. In this way, the two main challenges mentioned in the introduction are solved, and the TPR @ FPR metric can be approximately optimized by mini-batch gradient descent. We present a summary of the detailed process in Appendix C.
4 Experiments
4.1 Datasets
The experiments are conducted on four benchmarks, where the downstream applications include medical diagnosis, age estimation and content review. Each dataset is split into training set, validation set and set at a ratio 0.7 0.15 0.15. The statistics of these datasets are provided in Tab. 1.
Diabetic Retinopathy Detection (DR) 2 Diabetic retinopathy is complication of diabetes, which is a main cause of blindness the working-agg agge population. The Diabetic Retinopathy Detection dataset contains 35,126 optical images of the retinas obtained from patients, each image is labeled as one of 5 levels: O-No DR, 1-Mild DR, 2-Moderate DR, 3-Severe DR and P-Poliiferative DR.
UTKFace ? The UTKFace [42] dataset is popular benchmark for ordinal regression. This dataset contains 23,707 face images with ages ranging from 0 to 16 years old. Following previous work [8], divide the labels into 7 ordinal groups according to ages: baby (0-3), child (4-12), teenager (13-19), young adult (20-30), adult (31-45), middle-aged (46-60) and senior (elder than 61). We use the aligned and cropped faces provided officially.
NSFW 4, The NSFW dataset is public available dataset for Not Suitable For Work images detection. After filtering the broken links, we select the following three types of images from this dataset: O-neutral, I-sexy, 2-porn, which have a total of 55,459 images.
AgePrediction The AgePrediction dataset is large scale facial age prediction dataset. This dataset contains 217k face images. The age groups are split as in UTKFace dataset.
4.2 Implementation details
Network architecture. The feature extractor is implemented with ReXNet200 [17], the state-of-the- art CNN on image classification tasks, which is initialized with the pretrained model on ImageNet [36]. The backbone takes an image of size 224 x 224 x as input, which is normalized by subtracting the mean and dividing the standard deviation ImageNet, and outputs 2560-d embedding.
Optimization strategy. the training stage, we apply data augmentation for data preprocessing, in which the augmentation policies are searched in ImageNet with autt-auumeenttion [7]. The objective function is optimized with the stochastic gradient descent (SGD) optimizer with a momentum 0.9, and the batch size is set 64. The learning rate is initialized as 0.01, and decays by 0.1 per 30
"nttps://wwwe http::////wkk.ggleee. kaggle. balggll..co////aateeeeeee http:///aaaeeiiceeeioeppeeeeeeee hetneeeteerrrettttttteeeeettott detection. Licensed MIT.
"https://susang:
cachuu..io/UUKe. io.UTFaaee Licensed Data files C Original Authors.
'nttpa://github. hhttpe://gitaab..mmmaaee/to////e sararaa.aaaaaaaaaaaeeeeeeee Licensed MIT:
data 'https://ww htttp::/www..aaggee hagglee..o//rriiarenti/gggg prediction.

---

Table 2: Quantitative results of proposed method and the competitors. TPR@FPR (%) and AUC (%) are reported. The best and the second best results are highlighted in soft red and soft blue, respectively.
Datasets
DR
UTKFace &PR
NSFW
AgePrediction
@
TPR@FR
Metrics
AUC
AUC
AUC
0.003
0.01
0.03
0.003
0.01
0.03
0,003
0.01
0.03
0.003
0.01
0.03
CE
35.2
47.4
61.0
84.4
33.1
55.8
80.9
94.6
53.6
71.8
96.5
96.9
11.3
38.6
53.1
NNRank
32.8
45.3
61.3
85.4
35.7
59.7
75.7
94.5
54.8
71.3
96.7
97.5
F&H [1]]
36.0
48.4
63.5
85,8
37.5
63.0
80.9
95.7
52.2
71.9
96.3
97.4
11.0
36.4
48.9
CR-DDAG [35]
21.2
41.3
53.4
84.2
28.7
57.8
77.1
93.7
23.3
63.4
95.4
96.1
PR-DDAG [35]
25.3
45.7
60.3
85.0
24.9
60.1
82.5
96.0
49.6
71.0
96.6
97.1
SoftLabel [9]
36.8
48.2
64.3
85.9
39.5
63.2
82.0
96.0
38.8
71.9
96.9
97.6
11.0
37.5
52.1
F&1l+ + AUC (UI
35.0
48.5
63.0
85.6
42.1
64.5
80.5
95.6
54.6
71.2
95.8
97.3
12.4
39.6
53.8
Ehan's ' [10]
36.0
48.3
59.8
85.4
38.0
63.4
78.8
95.4
56.3
71.7
97.3
97.7
11.7
39.3
54.3
Ours
39.1
50,9
64.9
86.3
44.9
66.0
82,9
95.9
58.1
74.2
97.7
98.0
16.0
40.2
56.2
epochs. We also utilize the cosine schedule to adjust the learning rate. We also warm up the training for 5 epochs with the learning rate increasing from 10 The maximum number of epochs is 70, and the models that perform best the validation set saved for performance evaluation. The hyperparameters are set as follows: = 40 (Eq. (14)), a = 0.5 (Eq. (15)), N: is set to 512 for the DR dataset and the AgePrediction dataset, and 1024 for UTKFace and NSFW.
Evaluation metrics. Aiming at maximizing the TPR FPR metrics as shown in Eq. (3), we evaluate the TPR @ FPR with = 0.003, 0.01, 0.03, respectively. Without prior knowledge, we consider all sub-problems as equally important, so 0 set to
Competitors. To validate the effectiveness of our proposed method, we compare with seven methods: Cross-Entropy (CE), NNRank [4], F&H [1]], SoftLabel [9], F&H + AUC, DDAG [35] and Eban's method [10]. More descriptions on competitors are provided in Appendix D.
Environments. We implement method and the competitors with modified timm 6 [40], and all the experiments are conducted on Ubuntu 16.04, Pytorch 1.7.0 [34], with a single NVIDIA RTX 3090 GPU.
4.3 Results and comparison
The quantitative results of the involved datasets are summarized in Tab. 2. It can be seen that our proposed method outperforms most of the involved methods on all datasets. Notice that on the dataset DR, with = 0.003, 0.01, 0.03, the proposed method achieves higher TPR than the best competitor's with 2.3%, 2.4% and 0.6%, respectively. Moreover, we have the following observations from the experimental results: 1) Although the effectiveness of the competitors with regard to metrics like AUC, accuracy or mean square error has been verified the literature, under more than half the groups, these methods only achieve comparable performance to the baseline (CE). This phenomenon verifies that previous methods ignore the performance with a fixed low FPR, and this work necessary: 2) In the case of lower FPR, our method can bring more significant improvements, which is in line with the practical requirements: when more work assigned to models instead of manpower, thresholds should be set to ensure that FPR is smaller reduce the negative effects of model misjudgment. Therefore, TPR at a lower FPR is the performance when models are applied to a wider range, which more instructive.
4.4 Sensitivity Analysis
There are two main hyperparameters in our proposed method, namely, N₂ to control the size of cache, and a dependent on the random walk assumption. In this sub-section, we analyze how these hyperparameters affect the model performance. We train models with 2d grid search N₂ e (64, 128, 256, 512, 1024, 2048} and E {0.1, 0.3, 0.5, 0.7, 1.0, 1.5, 2.0}. For each combination, we evaluate performance with 0.003, 0.01, 0.03. The experiments are conducted on the DR dataset.
http://ghhbb..oo//rrrttaaan/rops model. Licensed Apache 2.0.

---

2048
2048
2048
size cache
1024
size of cache
1024
size of cache
1024
512
612
512
128
128
84-
84-
10
20
30
40
40
45
50
50
66
65
TPR@FPR
TPR@FRR
TPR@FP
2-
2-
1.5
1.5
1.5
0.7
0.7
0.5
0.5
0.5
0.3
0.3
0.3
0.1
20
40
45
50
50
56
60
05
TPR@FPR
TPR@FPR
TPR@FPR
0.003
0.01
0.03
Figure 2: Sensitive analysis on N₂ and a. Best viewed in color.
The results for Ne are shown in the first row of Fig. 2. We can observe that too large or too small cache size will reduce model performance. This due to a small cache uses less historical information, while cache module that is too large will lead to a larger feature drift. We also observe that for a = 0.003, No has less impact on the results.
The second row of Fig. 2 exhibits the results for from which we can see that extremely small significantly reduces model performance. Directly optimizing the original problem can viewed as + O, which verifies the conclusion we draw that directly optimizing the original is infeasible.
5 Limitations
Although the TPR FPR metric shows practical significance and our proposed method performs well on this metric, they still have some limitations as follows.
Require sufficient clear test data. Since we focus on low FPRs, it requires sufficient clear data for testing. Otherwise, take 0.01 example, if the number of negative examples is less than 100, only one false positive is allowed, which makes the metric sensitive to noises.
Focus only on computer vision tasks. The basic framework of our proposed method can be generalized to general multipartite ranking applications, but we focus on computer vision tasks in our experiments currently. This is because computer vision tasks are more diverse and challenging, and it also one of the main areas of deep learning application, which more suitable for our framework. We will consider extending it to other areas in future work.
6 Summary
In this paper, we study the TPR at a fixed FPR for multipartite ranking. From the practical perspective, we demonstrate the necessity of this metric some scenarios sensitive FPRs. Since TPR @ FPR is a constrained optimization problem which is hard to solve in an end-to-end manner, we propose a novel method CBA-CR approximately optimize this problem. On one hand, to tackle the challenge that the gradients of most negative examples are not available, we propose relaxed form according to a random walk hypothesis.
the other hand, to decompose the objective into sum of independent instance-win wise terms, we approximate the rankings by caching features in the past iterations. Experiments on three real-world benchmarks are provided show the effectiveness of our proposed method.
10

---

Acknowledgments
This work was supported in part by the National Key R&D Program of China under Grant 2018AAA0102003, in part by National Natural Science Foundation of China: 61931008, 61620106009, 61836002 and 61976202, in part by the Fundamental Research Funds for the Central Universities, in part by Youth Innovation Promotion Association CAS, in part by the Strategic Priority Research Program of Chinese Academy of Sciences, Grant No. XDB28000000, and in part by Alibaba Group through Alibaba Research Fellowship Program.
References
[1] Axel Berg. Magnus Oskarsson, and Mark O'Connor. Deep ordinal regression with label diversity. In International Conference on Pattern Recognition (ICPR), pages 2740 27470 IEEE, 2021.
[2] Dong Cao. Zhen Lei, Zhiwei Zhang, Jun Feng, and Stan Z Li. Human age estimation using ranking svm. In Chinese Conference on Biometric Recognition, pages 324-331. Springer, 2012.
[3] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.00 04297. 2020.
Jianlin Cheng, Zheng Wang, and Gianluca Pollastri. A neural network approach ordinal regression. In IEEE International Joint Conference on Neural Networks, pages 1279-1288 JEEE, 2008
[5] Wei Chu, Zoubin Ghahramani, and Christopher KI Williams. Gaussian processes for ordinal regression. Journal of Machine Learning Research, 6(7), 2005.
Wei Chu and Sathiya Keerthi. Support vector ordinal regression. Neural Computation, 19(3):792-815, 2007.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation strategies from data. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 113-123, 2019.
[8] Abhijit Das, Antitza Dantcheva, and Francois Bremond. Mitigating bias in gender, age and ethnicity classification: multi-task convolution neural network approach. In European Conference on Computer Vision Workshops, pages 0-0, 2018.
[9] Raul Diaz and Amit Marathe. Soft labels for ordinal regression. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4738-4747, 2019.
[10] Elad Eban, Mariano Schain, Alan Mackey, Ariel Gordon, Ryan Rifkin, and Gal Elidan. Scalable learning of non-decomposable objectives. In Artificial Intelligence and Statisticy, pages 832-840. PMLR, 2017.
[11] Eihe Frank and Mark Hall. A simple approach to ordinal classification. In European Conference on Machine Learning, pages 145-156. Springer, 2001.
[12] Huan Fu, Mingming Gong, Chaohui Wang. Kayhan Batmanghelich, and Dacheng Tao. Deep ordinal regression network for monocular depth estimation. In IEEE Conference Computer Vision and Pattern Recognition, pages 2002-2011, 1, 2018.
[13] Johannes Fürnkranz, Eyke Hüllermeier, and Stijn Vanderlooy. Binary decomposition methods for multipar- tite ranking. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 359-374, Springer, 2009.
[14] Wei Gao and Zhi- Hua Zhou. the consistency auc pairwise optimization. In International Joint Conferences on Artificial Intelligence, pages 939-945, 2015.
[15] Gabriel Goh, Andrew Cotter, Maya Gupta, and Michael Friedlander. datasct constraints. In Advances in Neural Information Processing Systems, pages 2233-233,, 2016.
Satisfying real-world goals with
[16] Pedro Antonio Gutiérrez, Maria Prtiz-Ortiz, Javier Sanchez-Monedero, Francisco Temandez-Navaree and Cesar Elervas-Martines Ordinal regression methods: survey and experimental study. IEEE Transactions on Knowledge and Data Engineering, 2871): 2015.
[17] Dongyoon Han, Sangdoo Yun, Byeongho Heo, and YoungJoon Yoo. Rexnet: Diminishing representational bottleneck on convolutional neural network. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
11

---

[18] James A Hanley and Barbara McNeil. The meaning and use of the area under receiver operating characteristic (roc) curve. Radiology, 143(1):29-3,, 1982.
[19] Chen Huang, Shuangfei Zhai, Walter Talbott, Miguel Bautista Martin, Shih- Yu Sun, Carlos Guestrin, and Josh Susskind. Addressing the loss- mismatch with adaptive loss alignment. In International Conference on Machine Learning, pages 2891-29000 PMLR, 2019.
[20] Qijia Jiang, Olaoluwa Adigun, Harikrishna Narasimhan, Mahdi Milani Fard, and Maya Gupta. Optimizing black-box metries with adaptive surrogates. International Conference on Machine Learning, pages 4784 4793. PMLR, 2020.
[21] Purushottam Kar, Harikrishna Narasimhan, and Prateck Jain. Surrogate functions for maximizing precision at the top. In International Conference on Machine Learning, pages 189-198. PMLR, 2015.
[22] Hsuan- Tien Lin and Ling Li. Large-margin thresholded ensembles for ordinal regression: Theory and practice. In International Conference on Algorithmic Learning Theory, pages 319-333. Springer, 2006.
[23] Hsuan- Lin and Ling Li. Reduction from cost- cost-sensitive ordinal ranking to weighted binary classification. Neural Computation, 24 24(5):1329-1367 2012.
[24] Xiaofeng Liu, Xu Han, Yukai Qiao, Yi Ge, Site Li, and Jun Lu. Unimodal-unifore constrained wasserstein training for medical diagnosis. In IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
[25] Xiaofeng Liu, Yang Zou, Yuhang Song. Chao Yang, Jane You, and BV Vijaya Kumar. Ordinal regression with neuron stick- beekkhhhaaking for medical diagnosis. In European Conference on Computer Vision (ECCV) Workshops, pages 0-0, 2018.
[26] Yanzhu Liu, Adams Wai-Kin Kong, and Chi Keong Goh. Deep ordinal regression based on data relationship for small datasets. In International Joint Conferences on Artificial Intelligence, pages 2372-2377, 2017.
[27] Yanzhu Liu, Adams Wai Kin Kong, and Chi Keong Goh. A constrained deep neural network for ordinal regression. In IEEE Conference on Computer Vision and Puttern Recognition, pages 831-839 2018.
[28] Yanzhu Liu, Fan Wang, and Adams Wai Kin Kong. Probabilistic deep ordinal regression based on gaussian processes. In JEEE/CVF International Conference on Computer Vision, pages 5301-5309 2019.
[29] Alan Mackey, Xiyang Luo, and Elad Eban. Constrained classification and ranking via quantiles. arXiv preprint arXiv: 800600000000..00000 2018.
[30] Peter McCullagh. Regression models for ordinal data. Journal of the Royal Statistical Society: Series B (Methodological), 42(2):109-127, 1980.
[31] Michael C Mozer, Robert Dodier, Michael D Colagrosso, César Guerra-Salcedon Salcedo, and Richard Wolniewicz. Prodding the roc curve: constrained optimization of classifier performance. In Advances in Neural Information Processing Systems, pages 1409-144400 1415, 2001.
[32] Naila Murray, Luca Marchesotti, and Florent Perronnin. Ava: A large-scale database for aesthetic visual analysis. In IEEE Conference Computer Vision and Pottern Recognition, pages 2408 2415. IEEE, 2012.
[33] Zhenxing Niu, Mo Zhou, Le Wang, Xinbo Gao, and Gang Hua. Ordinal regression with multiple output cnn for age estimation. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4920-4928, 2016.
[34] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems, 32:8026-8033 80377 2019.
[35] José Ramón Quevedo, Elena Montanés, Oscar Luaces, and Juan José Del Coz. Adapting decision dags for multipartite ranking. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 115-130. Springer, 2010.
[36] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Audrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recogaition challenge. International Journal of Computer Vision, 115(3):211-22) 2015.
[37] Wanli Shi, Bin Gu, Xiang Li, and Heng Huang. Quadruply stochastic gradient method for large scale non- linear semi- suni-supereieed ordinal regression aue optimization. In AAAI Conference on Artificial Intelligence. volume 34, pages 5734-5741, 2020.
12

---

[38] Kazuki Uematsu and Yoonkyung Lee. Statistical optimality in multipartite ranking and ordinal regression. IEEE Transactions on Puttern Analysis and Machine Intelligence, 3755):080-1009, 2014.
[39] Xun Wang, Haozhi Zhang, Weilin Huang, and Matthew R Scott. Cross-batch memory for embedding learning. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 6388 6397, 2020.
[40] Ross Wightman. Pytorch image models. https: http::///gtaab...cmmrrrrggaaaaar www.r//wigtaaaaapptttoooooooo 2019.
models.
[41] Zhiyong Yang. Taohong Zhang, Jingcheng Lu, Dezheng 7hang. and Dorothy Kalui. Optimizing area under the roc curve via extreme learning machines. showledge-Basse Systems, 130:74-89, 2017.
[42] Zhifei Zhang, Yang Song, and Hairong Qi. Age progressionrregresionn by conditional adversarial autoen- coder. In IEEE Conference Computer Vision and Pottern Recognition. IEEE, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sec. 5,
(c) Did you discuss any potential negative societal impacts of your work? [No] We haven't tt found any negative societal impact of our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yss]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. 4.2.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] Because it would be too computationally expensive.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Sec. 4.2.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material or as URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're you're using/curating? [No] Assets we used are open source.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "k7BdSX4h94V", "reviewer": "Reviewer_EgXj", "summary": "In this paper, the authors studied a particular metric for M-class multipartite ranking, with the particular motivation of FPR-sensitive applications in mind. Given that the combinatorial objective is hard to solve exactly, a series of relaxations are introduced / assumptions brought in for the algorithm to be amenable to mini-batch gradient updates. The high-level approach taken is to turn the problem into M-1 bipartite ranking sub-problems, after which a probability-based differentiable surrogate loss function are used and a cache module are leveraged to efficiently approximate the ranking of examples. Numerical experiments are also conducted on a series of real-world datasets.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " While the assumptions made in the paper sound reasonable to me, it would be good (and important) to have an approximation guarantee on the metric of interest in my opinion - it is unclear to me what can be said about Definition 2 (eqn 3) after the whole sequence of relaxations. It would also be interesting to trace out how much each level of approximation improve the performance / runtime on real data (going from (5) to (6) for example, and similarly for the ones in section 3.4). The relevant literature on ranking is adequately surveyed and the claims do look right.\n\nThe experimental results are shown, but it seems that some of the margin against other baselines are not that significant and makes me wonder what is the runtime comparison for those methods. On the other hand, from a more practical perspective, do the authors find other ways to estimate the probability weighting improving the performance in practice, beyond the random walk hypothesis made? These seem like questions that would be nice to investigate further.\n\nMinor typos: line 73: firstly -> first", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "XnbeDrm0MXn", "reviewer": "Reviewer_e74h", "summary": "In this paper, the authors aim to optimize the True Positive Rate (TPR) at a fixed False Positive Rate (FPR) for multipartite ranking, denoted as TPR@FPR. They considered a new evaluation metric TPR@FPR for multipartite ranking in FPR sensitive scenarios. This metric focuses on model performance with a low FPR, which is consistent with practical requirements. To optimize this metric, they propose cross-bnatch approximation for multipartite ranking. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In this paper, the authors aim to optimize the True Positive Rate (TPR) at a fixed False Positive Rate (FPR) for multipartite ranking, denoted as TPR@FPR. They considered a new evaluation metric TPR@FPR for multipartite ranking in FPR sensitive scenarios. This metric focuses on model performance with a low FPR, which is consistent with practical requirements. To optimize this metric, they propose cross-batch approximation for multipartite ranking. They have conducted extensive experiments on three real-world benchmarks to show the effectiveness of the proposed method. I have the following concerns.\n(1) The proposed algorithm can achieve more improvement under false positive rate sensitive application scenarios. What if we apply the proposed algorithm to other scenarios? If the data is black to the end-users, how to determine what algorithm to use?\n(2) The three datasets used in this paper are small-scale. Is it possible to apply the proposed algorithm to large-scale dataset?\n(3) How the feature extractor is used in this paper is not clear. Did the authors fix all the layers of the feature extractor, or did the authors tune some layers of the feature extractor?\n(4) The proposed algorithm focused on improving TPR@FPR. The reviewer is very interested to see what the performance will be if we evaluate in terms of traditional evaluation metrics.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "ToM5aF3EL2Y", "reviewer": "Reviewer_spF1", "summary": "The authors propose to optimize an interesting metric named TPR@FPR in multipartite ranking. This work approximates the original constraint on FPR to a probability-based loss function and speeds up the ranking estimation with embedding caches. With the above techniques, it is available to optimize TPR@FPR for deep models in an end-to-end manner.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strengths of this paper are:\n1) TPR@FPR is a useful metric in practical applications like biometrics, as FPR < \\alpha mirrors the security requirements of those systems.\n2) It is non-trivial to derive a suitable objective function to optimize TPR@FPR, since the quantile constraints are tricky for SGD. I like in particular how the proposed method avoids this difficult problem by proposing an arguably simple learning objective, which makes it possible to optimize TPR@FPR in deep models.\n3) The effectiveness of the method is verified by experiments in different applications, and the experiment detail is clear. The sensitivity analysis helps to select hyperparameters.\n4) The limitations are clearly discussed.\n \nMy two major concerns with the paper are:\n1) It is mentioned in line 280 that the hyperparameters \\sigma and Nc are tuned on the validation set, do other baselines have access to the validation set? If not, the comparison might be unfair.\n2) The proposed method requires more memory to store the embeddings, could you please provide the analysis on space cost? Although the time complexity is present in line 245, it would be clearer if the exact time consuming is shown.\n \nOther minor issues:\n1) Equ.(8) is used to estimate ranking on a mini-batch (in line 230), but in my understanding, it should be on the whole training set.\n2) Equ.(9) uses the inner product to measure the similarity, are the embeddings are normalized? It’s unreasonable if not.\n3) A typo in line 280: “with learning increasing..” -> “with learning rate increasing..”.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "6MsFnFEVgDq", "reviewer": "Reviewer_H8zB", "summary": "In many practical applications such as medical diagnosis, the machine learning models are often required to has a low FPR. Motivated by this, the paper proposes a novel framework to optimize the true positive rate at a fixed false positive rate (TPR@FPR) for multipartite ranking. In their framework, the author proposes a surrogate optimization problem and a rank estimation technique to directly optimize TPR@FPR.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1.This paper is well-written with formal derivation and the overall motivation is clear. \n\n2. I appreciate that the optimization for TPR@FPR is challenging, especially when one has to use it to train a deep model in an end-to-end manner. \n\n3. It’s interesting to see that the proposed framework overcomes this with a series of approximation schemes. Moreover, the observation that the short time dependence on historical updates can be reduced to white noises also helps to establish a simple yet effective probability model.\n  \n4. I find some minor issues in the paper:\n  - In line 81, “multilayer perceptron (MAP)” should be “multilayer perceptron (MLP)”;\n  - In line 84, the verb is missing in “Other early literature SVM”.\n \nOverall, this paper presents a solid work, which I really enjoy reading. I recommend accepting this paper.\n \nQuestions:\n- There is a significant gap between the experimental results on DR and DR-flipped, even with the same \\alpha. What’s the main difference between these two versions?\n- What is the specific meaning of “practical meaning of this dataset” in line 259?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: TPR@FPR is a useful metric in many important practical applications (e.g. biometrics). The paper made a valuable contribution on the challenging problem of optimizing TPR@FPR for multipartite ranking.   On the top of deep learning models,  it proposed a novel framework by introducing a differentiable surrogate and a fast ranking estimation method to optimize the non-decomposable objective function. Overall, the problem is well motivated and the proposed framework/algorithms for optimizing TPR@FPR are novel and interesting. 
Decision: Accept (Poster)