Estimating the Arc Length of the Optimal ROC Curve and Lower Bounding the Maximal AUC
Authors: Song Liu
Venue: NeurIPS 2022 Accept (2022)
Keywords: ROC Curve
f-divergence
density ratio estimation
AUC maximization

Abstract: In this paper, we show the arc length of the optimal ROC curve is an $f$-divergence. By leveraging this result, we express the arc length using a variational objective and estimate it accurately using positive and negative samples. We show this estimator has a non-parametric convergence rate $O_p(n^{-\beta/4})$ ($\beta \in (0,1]$ depends on the smoothness). Using the same technique, we show the surface area sandwiched between the optimal ROC curve and the diagonal can be expressed via a similar variational objective. These new insights lead to a novel two-step classification procedure that maximizes an approximate lower bound of the maximal AUC.  Experiments on CIFAR-10 datasets show the proposed two-step procedure achieves good AUC performance in imbalanced binary classification tasks.

Estimating the Arc Length of the Optimal ROC Curve and Lower Bounding the Maximal AUC
Song Liu" School of Mathematics University of Bristol Bristol, BS8 IUG, UK song. Long.liu@bristol.com ac. uk
Abstract
In this paper, we show the are length of the optimal ROC curve is an f-divergence. By leveraging this result, we express the are length using variational objective and estimate it accurately using positive and negative samples. We show this estimator has a non-parametric convergence rate Op(n 8/4) (3 E (0, 1] depends on the show the surface area between the
smoothness). Using the same technique, optimal ROC curve and the diagonal can be expressed via similar variational ob- jective. These new insights lead novel classification procedure that maximizes an approximate lower bound of the maximal AUC. Experiments on CIFAR-10 datasets show the proposed two-step procedure achieves good AUC performance in imbalanced binary classification tasks.
1 Introduction
The study of Receiver operating characteristic (ROC) curves has a long history in medicine [26]. psychology [155] and radiology [16, 13]. In machine learning, ROC curves have been primarily used to analyze the performance of different classification algorithms [8, 9]. Indeed, the Area Under the Curve (AUC) encodes classifier's ranking accuracy, making it preferable performance metric for imbalanced class classification [8, 31. In recent years, ROC curves have also been used in comparing two distributions and achieved promising results. Examples include analyzing the mode collapsing issue of Generative Adversarial nets (GAN) [25], and diagnosing the performance of an amortized Markov Chain Monte Carlo [18].
In applications that require computing statistical discrepancy between distributions (c.g. GAN [14]
Variational Inference (VT) [1)). f-divergences widely used discrepancy measures. The family of f-divergences includes Kullback-Leeibee divergence [23] and Total Variation distance. It has been shown that f-divergences, generally, can be expressed via variational objectives and efficiently approximated from empirical samples [29, 30].
Since the ROC curves are used as performance metrics in many two sample applications. are they in any way related to /-divergences? For example, can AUC be an f-divergence between positive and negative data distributions given some classification score function? An earlier investigation proves that the answer is no when the score function is the likelihood ratio [31]. Nonetheless, this result inspired us look for f-divergence from other geometries of the ROC curve.
In this paper, we show that, when using the likelihood ratio score, a novel f-divergence arises from the are length of the corresponding ROC curve. By leveraging this result, we can express the are length using a variational objective and approximate it using only samples from two distributions.
WWbbpage: https::/aallmodelssrrwronggg net
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

03
E3
0.5
65
rPR
(a) different distributions, long ROC (b) different distributions, long ROC (c) same distribution, short ROC
Figure/ 1: ROC curves generated for one dimensional datasets using the identity classification score function (x) = æ. Notice that the are length of ROC curves seem to be a good indication on how
different the positive (red) and negative (blue) data distributions are.
We show this are length estimator is also consistent estimator to the arctangent of likelihood ratio and has a non non-parametric convergence rate (n -n/4), where 3 E (0, depends on the smoothness of the true arctangent likelihood ratio. Moreover, by parameterizing the ROC curves of positive and negative mixtures distributions, the surface area between the optimal ROC curve and the diagonal can be expressed via similar variational objective. With the help of our arctangent ratio estimator, can approximately maximize a lower bound to this surface area. We point out the similarity between this lower bound maximization and the classic AUC maximization [3]. We show our approximated optimal score achieves comparable performance to state of the art AUC maximizer in an imbalanced classification problem on CIFAR-10 dataset.
2 Background
2.1 An Illustrative Example
ROC curves are frequently used to visualize binary classification performance, and we often rely on the Area Under the Curve (AUC) as numerical metric for selecting good classifier. this section, we highlight an often overlooked ROC geometry: arc length. We illustrate its potential as good discrepancy measure between positive and negative data distributions.
Let us consider the one-dimensional distributions and the ROC curves in Figure 1. In this example, ROC curves are generated using the identity score function t(x) =x.
In both (a) and (b), P+ and p- are quite different, and thus the discrepancies between positive and negative distributions should be large in both cases. In (c), the densities p- and p. the same and consequently their discrepancy should smaller than that of both (a) and (b). Further, notice
that both (a) and (b) have long ROC curves (= 2) while (c) has a shorter one (#= v2). This example suggests that the more similar p+ and p- are, the shorter the ROC curve is.
the are length will not reflect any discrepancy between data distributions at all. This observation inspires the following questions: Why the are length in this example good at telling the differences between two data distributions? Are there other score functions whose ROC are lengths are also good discrepancy measures? What are the practical applications of studying the are length of ROC? the following sections, we study the are length of ROC curve under probabilistic framework and provide answers to these questions.
However, we can see that ((x) = x is special choice: If t(x)
2.2 ROC Curve in Probabilistic Setting
Suppose we have positive and negative datasets X := (= and X_ := {x, y drawn from two distributions P+ and PP_ respectively. These distributions have respective probability density functions p+ (x) and p=(x) that are both defined on the domain X' Rd. A classification score function (score function for short) takes a sample æ as input and outputs a real-valued score. Suppose have score function ((x) E R. We classify as positive if ((x) т, where is threshold.
Let F4 and F_ denote the Cumulative Distribution Functions (CDFs) of ((xx)) and ((x respectively. Then can define the following quantities:

---

False Positive Rate (FPR) at threshold r. (T) :
True Positive Rate (TPR) threshold [r) = 1+()++
The ROC curve score function t: the graph of function (+)F where & [0, 1].
The above definition of ROC curve requires F- to be strictly increasing. In this paper. we assume F+, F- to be both strictly increasing. Obviously, both F+ and F- depend on the choice of score function t.
2.3 Arc Length of ROC Curve
Due to the strict monotonicity of F+ and P_, F+ and F_ form bijective parameterization of the ROC curve in the sense that each point on this ROC curve can be written as (F-((r). F- (To)) for unique To I.. Using the line integral formula, the are length of an ROC curve for fixed score function can be expressed using the derivatives of F_ and F+:
ROC(1) = L V [=) + [0. S VI+())++5((794)
where f+(r) and (T) are the density functions of t(x+) and t(x=) respectively. Although (1) an elementary result, has been seldom discussed the ROC literature. Authors in [5, 6] have proposed a performance metric computed over the "ROC hypersurface" and (1) is used to justify such a metric in binary classification setting.
Using (1), we can confirm simple geometric fact whose proof can be found in Appendix A:
Proposition 1. ROČ(t) e [v2, for all
This result reflects the geometric observation that any monotone curve (such as ROC curve) starts and ends at two opposite corners of the ROC space [0, 1]2 has an are length between v2 and 2.
f-divergences Arising from ROC Are Length
3.1 f-divergence of Score Distributions
Among many discrepancies measures, foresss,,ddwve f-divergence has been widely used in many applications.
Definition 1. Let p and 9 be densities of two continuous distributions. An f-divergence is defined as:
(plg)) = Eg 9 p(x)
where g is convex and lower-semicomminuous satisfying 9(1) = 0.
Now let us slightly rewrite (I). Assuming f- strictly positive (in which case F_ is strictly increasing), we can write
(T(T) (-)
+1 =2.=EB-F 1 (4(77)
ROC(!) V2= Bj -,
T-
(2)
where ((8) V -1- Equation (2) yields the first important result of this paper: ROČ(()-v3
an /-divergence between score densities f+ and since g(s) is convex function and g(1)
This result confirms, for any given t, ROC(t) is good discrepancy for measuring positive and negative scores (i.e., score distributions). It also explains why the ROC are length in Figure is good discrepancy measure: Given t(x) = x. the score distributions are same as the data distributions in each plot. The length of ROCs in Figure 1 plots are f-divergences of the score distributions hence are also f f-divergences of data distributions.
Although ROČ(()- V is an f-divergence of score distributions, it is not an f-divergence of the positive and negative data distributions for general f. The choice ((x) = x in the toy example does not have simple analogues for higher dimensional datasets. Are there choices of / such that the are lengths of their ROC curves good discrepancy measures on data distribution? In what follows,
show when is bijective transform of the likelihood ratio (=(=) ROC(t) encodes the differences (m)
between p+ (x) and p- (xx) in the form of an fddivergence between p+ (æ) and (x)).

---

3.2 f-divergences of Data Distributions
Using the law of the unconscious statistician, express ROC(!) in terms of expectation (())))) (+()))
2
with respect to the negative data density p-(x): Ep-
+1. Consider a special family of
(+ p- (m) where is any strictly increasing function. Due the Neyman- Pearson lemma [27]. ROC((() has the highest TPR at any FPR level. Geometrically speaking, they dominate all other ROC cuves in an ROC plot and have the maximal AUC. Hence, we refer to t" as the optimal score and ROC(t*) the optimal ROC curve. For convenience, we denote the ROC(t*) as ROC which reads "rock star". can be shown that
score functions: (a)))
Jen (1) =t' (aza) P+ (x))dz
(+(()))))
P4 (200)
bA
(3)
f-(t((zo)
P+(m) (a)
=t (zen)
p- (x)d
p-(x)))
where the second equality holds due to
Ia a(a))d
Jp Marjda - (C), when
o(æ) M(m)
= C, Væ E D. When
(()) = x; (3) expresses a known result [7], and is often given in plain English as "the density ratio of the likelihood ratio score is the likelihood ratio itself"
Finally, ROC takes an elegant form free from or y: (xx
(=))) (m)
2
1. Equivalently,
p+(x))
ROC - /2 = Fr (2)
(4)
(x)
We can see that the same f-divergence arises from computing the are length of ROC However, unlike the f-divergence given in (2), (4) is an f-divergence between data distributions, not score use the the optimal scores, the arc length of the optimal
distributions. It shows that as long ROC can indeed reflect the differences between data distributions. From now on, we will refer to ROC - 22 as the ROC divergence. To the best of our knowledge, (4) has not been presented in literature before.
By definition, the ROC divergence is symmetric. Moreover, as a result Proposition 1, the ROC divergence is upper bounded by V2 and lower bounded by 0. Some geometric properties of ROC* can be found in Section B.
Estimating the Are Length of ROC*
4.1 A Variational Objective
To numerically approximate the are length using samples alone, we leverage that ROC* v22 is an f- f-divergence. Utilizing Fenchel's 's duality [20], authors [29] show that an f-divergence D (p+ (p-) has variational representation:
p+ (x)
da = / p (ar { u(xr).
P+ (x)
Do(p+lp-)= = p- (x)g
- g' [u(ar)] da p- (ax)
p (x)
= sup / p+ (=))))((
p- (2))g' [u(x)]dæ,
where g' is the convex conjugate of g and the supremum taken over all measurable functions. [0, oo] thus g has convex conjugate g'(z') -v - z²2 v2.+ x' e [0, 1]. Rewriting ROC 22 using the above variational representation and dropping the /2, obtain:
In the case of the ROC divergence, g(2) = +I= v2 and
ROC sup Ep, [u(=)) E, [V1 ==2(=)))) =E[0,1]

---

We reparameterize u(x) = sin[v(>)), where v E [0, [T2]:
ROC sup Ep, sin sin[e(()] + E cos(v(a))
(5)
eE[U,a/2]
Differentiating the objective (5) for and setting the derivative zero, we can see the supremum is attained at v² atan PL, In other words, the optimal v* is the arctangent likelihood ratio function.
is also interesting to see how v" is visualized in the ROC plot. We see the tangent of ROC" FPR level 8o E [0, 1] is
0.9
FFF+ (+(F="(80)) p+ (20) f- (-(F=-(so) p- (xx)'
(6)
TPR
0.8
ROC
where zp is any point in X' that satisfies the equality (((())) (ma) F=1(sp) (6) is a known result [9]. p-
0.6
other words, V* atan ft the slope angle of ROC
See Figure 2 for visualization of the tangent of ROC expressed by the likelihood ratio.
0
0.1
0.2
0.3
0.4
0.5
FPR
Morcover, using (5), we can obtain a relationship between
Figure ROC* and its tangent marked
ROC* and the total variation distance between P+ and P_ (denoted as TV(++, P-)).
by vector (1, p (ma)
(1 (scaled to fit).
P+ = (11, 1),p- = W(-1,1).
Proposition 2. IV.++.P-) has lower and upper bound expressed via ROC"
2 ROC - 2-1 a²
max aE(0,1]
arccos(()) - ar)sin(a)
< TV(P+,P-)) ROC 11
Proof of this proposition be found in Section C. This proposition justifies that ROC* is valid measure of the discrepancy between P+ and P_. We compare this bound with other known TV
bounds in Section 6.1.
4.2 A Tractable Objective for Estimating atan P+
To use (5) in practice, we need to find an appropriate function class F. We can simply restrict e to bounded (parametric/non parametric) function class and solve the sample version of (5):
sin(v((
Σ cos(v(())))
max DE / TE
In practice, enforcing the boundedness [0, r/2 over X' is difficult. We can relax (7) by only enforcing the boundedness constraint of on the sample dataset X4 X_X.
For example, by letting F be Reproducing Kernel Hilbert Space (RKHS) [33], can translate (7) into the following optimization problem:
n Σ = argmin {(v) + |1llle ((v) = n
n- sin (,.x(())))))) - n Σ
cos(u, ((x7))
VEM
2
s.t: (v, (()) E (o. π -
Vx E X
(8)
where H is RKHS with positive definite kernel (()xx") (v(()), s(x')) || ||x is the RKHS norm and 2 || is the regularization term. The optimizer (0, ((xr) is an estimation of u² (a:). the arctangent of the likelihood ratio. (8) strictly convex optimization and thus, if solution exists, it must be unique.
Instead of modelling atan Pt, we can opt for modelling the log likelihood ratio log R=, However, this modelling choice results an non-convex optimization thus presents extra challenges in the theoretical analysis. Details can be found in Section K.
p-

---

4.3 Finite Sample Guarantee
We show that the solution of (8), (6. s(z)) converges to the true arctangent likelihood ratio (or its projection onto H) as the number of samples goes to infinity. Below are few regularity conditions.
Assumption 1. There exists a unique e² H. such that E V.V.((v*)] = and (o", (((x) E [0, =/2] holds for all z E X.
sufficient condition the above condition is specified in the following proposition.
Proposition 3. If there exists unique v' E H. such that (0(x))))) atan (-(#) then Assump- lion/ I holds.
p (x)
The proof can be found in Appendix F. Proposition 3 states model is correctly specified and identifiable then Assumption 1 holds. is possible that there exists a E H satisfying E[V,,((")) = which does not meet the boundedness constraint [0, x x/2]. ] In this paper we only consider situations where Assumption 1 holds, which includes all situations where the model is correctly specified and some situations where the model is misspecified.
{u ;== ). such that Vo E HH",Va E xx+ U X-, (v, ((x)) E (0, 5) holds with high probability. The sequence 8nain is monotonically decreasing as mmin grows infimity.
Assumption 2.
Lel Dmmin min(n+, n.). There exists a subspace ==
Hillu v I S
Assumption states all e within a vicinity of v² are in the interior of (8)'s feasible region with high probability. The following proposition gives sufficient condition under which Assumption 2 holds.
Proposition 4. Suppose [(((())) I 1. If our model is correctly specified as described in Proposition
3 and Vo E X, alan
(+) E [R1, Ral, for some Rí and R2 such that 2 > R2 > Rı > 0, then
p-
there exists N > such that Assumption holds when Tmmin > N.
The proof can be found in Appendix D. Since we use RKHS as the estimator function class, our final assumption is that v* should be reasonably smooth. In previous works such an assumption depends on the decay of the integral operator's eigenvalues [36, 10]. In this paper, we measure the smoothness using the range space technique which has been recently adopted in [11, 35]. We define
Σ. == E(V? = Ep, [sin(v, ((x)) (p(x,) :) @ plx, -)] Ep_ [cos(v, (p(x)) (p(x, :) @ p(x, )],
where @ denotes the outer product. Given vo E H. Ee, is an integral operator on H. and
E u = Ep, (sin ((z)) ((xx,) :) u(=)]] + Ep_ [cos(vo, ()(x)) ((xx) -) u(x)).
By definition E., is positive, self-adjoint operator, in the sense that (u, E., u) 2 0, (u, E., u} (Eno u, v), Ve, u E H. Moreover, some algebra shows that Eng is also bounded and compact operator. See Section G for more details.
Next, we assume the true arctangent ratio function (or its projection) in the range space of Ege.
Assumption 3. Let R(E..) denote the range space of E.. There exists O < 3 S 1, v* E R(E where C is the fraction power of a compact, positive and self-adjoint operator C.
Note that the larger / is, the smoother the functions in the range space are. More discussions on the range space assumption can be found in Section 4.2, [35]. Now we are ready to state our theorem:
Theorem 1 (Convergence Rate of 0). Suppose Assumptions 1, and hold and è exists. If [|<(() 22 < 1 and
T
K
4
8
1/4 mini
974 T'min
max (B+, B-)²
where B4 = 11 (E AI) Ep, [g(x)]]lm
B_ = 11 (E.- + -1) Ep [=(x))]l and T 2
1, K > 0 are constants that do not depend on Rmins then there exists constant N > 0 such that when Pinin N, ||e 0² 11 (n (8/44
The proof can be found at Appendix E. Theorem I shows that, under mild assumptions, 0 is indeed a
good estimator for alan
(=)
Some discussions comparing our results with convergence results
(m)

---

an optimal score that gives rise to ROC", our estimator may have some interesting applications such outlier detection [19] or Neyman-Pearson classification [37]. We will defer discussions on those applications in future works. In the next section, we employ our aretangent likelihood ratio estimator an application of lower bounding the maximal AUC.
proved by Nguyen al. [28] can be found in Section M. Since atan
(1
Approximately Lower Bounding the Maximal AUC
Finding a score function t that approximately maximizes AUC is an important task in binary classifi- cation. Let us denote the AUC of ROC(t) as AUC(t). It can be seen that
AUC(t) = / (F= E E, (f(x+) 2 t(x_)))
[0.1]
Due to the Neyman-Pearson lemma, ROC" has the maximum AUC among all ROC curves. Denote the AUC of ROC" as AUC Consider the following inequalities:
AUC sup Ep Ep, [1 (Mxx+) 2 1(x (x))))
sup Ep Ep, [Lxx 1, ))]
tET
(R)
where L(a,b) b) is continuous and concave lower bound of the indicator function (aa b). Due to
the Neyman-Pearson lemma, the supremum of (i) is only attained when t(xx 7 p+ (m) where
p-
a strictly increasing function. Replacing the expectations in (ii) with sample averages yields the optimization problem of AUC maximization [3, 12]:
max REJ
Σ (, i1j=1
(10)
The objective above is also referred to Wilcoxon- Manton-Mann---iaa Mann-Whitne Whitney statistie [16]. Therefore, see that AUC maximization is a procedure that approximates an optimal score function by maximizing lower bound of AUC".
Now, we show a different way of lower bounding AUC" with the help of atan P (ac) We have
(ax)
seen that how (F (T), F+ (r)) parameterizes an ROC curve in Section 2.3. fact, the area between ROC* and the diagonal line from (0,0) to (1, 1) can be similarly parameterized by considering ROC curves of positive and negative mixture score distributions.
Let F and F denote CDFs of any optimal score. For 55). we can define CDFs of a-mixtures of Fü and F as follows
0.4
f
03
-1=)()==(-)))((( - +
= (1-0))
0.1
Then, FPR (F (T, a)) and TPR (F; a)) for these a- mixtures can be defined accordingly. We visualize densi- ties of (a) and F (a) for different a in Figure 3.
0
Further, we can see that the 2-dimensional coordinate a(T,a) == (F (T, a), (T, parameterizes the area between ROC and the diagonal line in [0, 1J2:
Figure
Densities (T and [2 a) for E [5.
When fixing and varying т, the coordinates give rise to smooth curve in ROC space from [0, 0] to [1, 1].
When = 0. such curve is ROC*.
When a .5, such a curve is the diagonal line.
When fixing To and varying the coordinates produce straight line segment connect- ing (F. 0), Fr (To. 0)) and (F (To. .5), F FI (5).
7

---

0.8
015
0.6
0.4
(Prw, LPP. 1)
(Pr0.1.PIL 9]
Figure 4: Left: (F"(T, (T(, parameterizes the surface between ROC and the diagonal line in [0, 1J2. This plot is created by setting p+ 1), ())1, and (*)x) 2 log (+
p-
a. Middle: Our parameterization "mesh" divides AUC into surface elements (small patches on the plot). Right: Wilcoxon-Mann-Wine
statistic divides AUC into histogram bars.
The left plot in Figure 4 visualizes this paramcterization. Now the surface area sandwiched between ROC* and the diagonal line can computed using a surface integration:
f /
AUC -5
ddom(T) Joo.5) [0, orr(T,a) dodt,
(11)
where x denotes the cross product. After some algebra and applying the Fenchel duality technique in Section 4.1, we prove that AUC can be expressed as the supremum of variational objective similar to (5):
Proposition 5. AUC*
V2A 2
+
Ep+ w atan (x) =E(0,x/2]
w [atan p+ (xx
A: sup
sin sinfo(=)] + Ep-
cos cos(e(x))]
(12)
p (x)
p p=(x)
where w(TT) == sin(r + + [FI - FL (T)). The supremum of (12) is attained v² atan PL. F-
The proof can be found in Appendix H in the supplementary material. A lower bound of A can be obtained by restricting v to function class.
Evaluating w requires us to evaluate atan (a)
(=)) Fi and F which are not readily available. However,
Section 4.3 shows that the empirical estimator (8) a consistent estimator of atan (+(x)
under mild
(w)
conditions. Therefore, we propose the following two-step procedure to approximately lowerbound A:
Algorithm 1 Two-step Procedure for Approximately Lower Bounding A
1. Obtain f(x:) == (6, σ(x)) using (8). Approximate F; and F" using F_ and F_ which are empirical CDFs of ((x++) and [((==).
2. Optimize the empirical version of (12) by restricting to a feasible function class and plugging estimates obtained the earlier step, i.e.,
n
1 n_ n.
i argmax vE(0.a/ /2 n4
Σ ú [((xtt]] sin(u(x;t))
Σ d [((m]] cos(u((2:) )
(13)
where @(7) = sin(T + 7) -F
Note that (13) is nothing a weighted sample objective (7). Thus, can easily optimized by the algorithm that solves weighted version of (7) given the approximated weights in the first step. In

---

automobile
0.988
0.985
0.87
0.99
994
0.905
0.80
0.96
0
0.985
0.975
30
100
50
100
50
100
50
100
50
100
0.68
dog
999
frog
0.680
0.980
ship
0.975
0.594
0.86
0.992
Tacootop
0
0.985
Logidic
0.88
0.50
0.88
0.800
100
50
100
90
100
100
50
100
Figure 6: Testing AUC of one-versus-the-rest classification CIFAR-10 dataset.
practice, we simply run the solver for (8) twice: The first time we run it without weights then run again with weights w [((x))) calculated from the first run.
p- Since the above algorithm also approximates an optimal score (atan P_ by maximizing an approxi-
mated lower bound of AUC", it is natural to wonder how the maximizer of (13) would perform in AUC maximization tasks. the next section, we show that our two-step algorithm achieves promising AUC performance compared to state of the art AUC maximizer.
Computing AUC* using (11) different from using Wilcoxon-Mani Mannoonnn-n---- statistic (i.e., (13)): Our approach divides the space between ROC" and the diagonal into small surface elements and then adds them up. Wilcoxon-Mann Maroooon-Man-Whhineey statistic adds up all histogram bars, which TPRs at different FPR levels. Our approach requires F+ and F_ to be differentiable with respect means the score distributions cannot be discrete. However, Wilcoxon-Man Manno-MM-----neeyyy Whitney can compute the AUC of discrete score distributions without problem. This difference is visualized in the middle and right plots of Figure 4.
which
Experiments
6.1 Numerical Comparison of Divergences and TV Bounds
In this experiment, we numerically compare the ROC di-
vergence, the upper and lower bound in Proposition 2 with several other divergences and some known bounds of TV
(ruscaled)
ROC
ROC
in Figure In this numerical simulation, P+ A0 1)
and p. = (8, 1). We plot ROC divergence, Jensen Shan- non divergence, Wasserstein distance and TV between p+ and p_ as 5 grows from 0 to 5. We can see the (rescaled) ROC divergence closely resembles TV. When > 1.45, the upper bound given in Proposition is the tightest among known TV upper bounds [2, 4] (Pinsker's upper- bound, Bretagnolle & Huber's upper bound, Le Cam's upper bound). This suggests that combining our upper- bound with existing bounds may produce even tighter bound for TV.
Pinsker
(og2)
Wasserstain
0.5
2
5
Figure 5: Comparison various diver- gences and bounds of TV.
Imbalanced Classification on CIFAR-10
In this section, we test if the i obtained in our two-step procedure (13) indeed a good score function in terms of AUC imbalanced classification tasks. We use widely known image classification dataset CIFAR-10 [22]. The performance is compared with an AUC maximizer which maximizes the empirical lower bound in (10) and vanilla logistic regression classifier. We set the surrogate loss L(a, b) ;= - -(1 (a b))² in the AUC maximizer, as

---

suggested in [12]. All methods use linear models with no regularization terms since our models are simple and we have sufficient samples. Particularly, the and ē in our two-step algorithm is obtained using (8) by setting pf(x) = x and À The AUC maximization (AUC max) implemented using SPAUC method [24].
Instead of using the raw features, extract 50 dimensional bounded features by training residual network [17] on the training dataset using the 10-class cross entropy loss. The structure of the network is included in the supplementary material. After obtaining features,
construct datasets for 10 different one-versus nee-versus-the- rest classification tasks. For a single task, we pick a class and obtain X by randomly sampling from this class n times in the training set. Similarly, X_ is obtained by randomly sampling from the rest of the classes n... times. In our experiments, we set n = 24, 48, 72, 96, 120 and fix n- 1000 to create imbalanced positive and negative datasets. We all three methods and obtain the corresponding score functions, For each class, we repeat the experiment 96 times using different random samples. We use the testing and training split provided by the dataset itself.
Our experiments can be seen as a transfer learning task which reuses predictive features trained for a multi-class classifier for one-versus, teesssttherssss binary classification tasks.
The average AUCs computed on the testing dataset and their standard errors over 96 runs over different + sample sizes are shown in Figure 6. Our method has approximately equal performance with the AUC maximizer despite not directly maximizing the AUC. This observation indicates that è can be a good score function in AUC maximization tasks. Both of the methods significantly outperform vanilla logistic regression.
6.3 Discussions on Computational Complexity
Without loss of generality, assume 7.., п n. The naive caclulation of objective (10) has a computational complexity Onn²) since we evaluate the loss function L at each pair of samples. However, authors [21] have shown that the objective function (10) can also be computed with (nn (n))))) complexity for hinge loss (and decomposable loss functions). A recent work [39] simplifies the computation of (10) for the squared loss function L with an unbiased estimate. Suppose t(x) = (v, æ), then the negative objective of (10) is an unbiased estimate of
Var p+ (1v, xx) Varp- [u) + 2(v, E, [x:] - Epp [a]) + (v, Ep [x] [=])?
(14)
After we approximating (14) with empirical terms, the computation can be done with a complexity On)). When implemented in an online fashion, it has computational complexity of one datum.
In comparison, the objective (7) and (13) are summation of sin / cos(v(z)) evaluated each datum, so computing the objective/gradient has a computational complexity O(n). Computing A, and F_ requires sorting our dataset, which has an average complexity O(n log n). However, once our datasets are sorted, F+ (f(za) m4.
_ where i is the index of ((xo) in the sorted set (f(x;))
7 Conclusions
In this paper, show that novel f-divergence arises from the are length of the optimal ROC curve. The are length can be accurately estimated from positive and negative samples using a variational expression. It is also an estimator for atan p-/p- and has a convergence rate Op (n-4)) Finally, show that the area between the optimal ROC curve and the diagonal can be parameterized using similar variational objective. It leads to a two-step procedure that approximately lower bounds the maximal AUC which achieves a promising result in AUC maximization tasks.
Acknowledgments and Disclosure of Funding
The author would like to thank Prof. Peter Flach and Dr. Hao Song for their helpful discussions. The author would like to thank four anonymous reviewers for their insightful comments. In particular, we thank Reviewer WSBr and the Area Chair for pointing out the computational complexity inaccuracies in our initial version.
10

---

References
[1] D. M. Blei, A. Kucukelbir, and J. D. McAuliffe. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518)8339-77,, 2017.
[2] C. Canonne. A short note inequality between kl and arXiv:2202.071188, 2022
[3] C. Cortes and M. Mohri. Auc optimization vs. enror rate minimization. In Advances Neural Information Processing Systems 16 (NeurIPS 2003), volume 16, 2003.
[4] L. Devroye, A. Mehrabian, and T. Reddad. The total variation distance between high- dimensional gaussians. 1riiv:88000080// 2018.
[5] D. C. Edwards and C. E. Metz. A utility-based performance metric for roc analysis of n-class classification tasks. In Medical Imaging 2007: Image Perception, Observer Performance, and Technology Assessment, volume 6515, pages 21 30. SPIE, 2007.
[6] D. C. Edwards and C. E. Metz. Optimality of utility-based performance metric for roc analysis. In Medical Imaging 2008: Image Perception, Observer Performance, and Technology Assessment, volume 6917, pages 122 127. SPIE, 2008.
[7] S. Eguchi and J. Copas. A class of logistic-type discriminant functions. Biometrika, 89(1):1-22, 2002.
[8] Fawcett. An introduction to roc analysis. Pattern Recognition Letters, 27(8):861-874, 2006.
[9] A. Flach. Roc analysis. In Encyclopedia of machine learning and data mining, pages 1-8. Springer, 2016.
[10] K. Fukumizu. Exponential manifold by reproducing kernel Hilbert spaces, page 291-306. Cambridge University Press, 2009.
[11] K. Fukumizu, L. Song, and A. Gretton. Kernel bayes' rule: Bayesian inference with positive definite kernels. The Journal of Machine Learning Research, 14(1):375333733 2013.
[12] W. Gao, R. Jin, S. Zhu, and Z-H Zhou. One-pass auc optimization. In Proceedings of the 30th International Conference on Machine Learning (ICML 2013), volume 28, pages 906 (669914,,5 2013.
[13] D. J. Goodenough, K. Rossmann, and L. B Lusted. Radiographic applications of receiver operating characteristic (roc) curves. Radiology, 110(1):89-95, 1974.
[14] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Orair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in Neural Information Processing Systems 27 (NeurIPS 2014), pages 2672-2680, 2014.
[15] D. M. Green and J. A. Swets. Signal detection theory and psychophysics, volume 1. Wiley New York, 1966.
[16] A. Hanley and B. McNeil. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology, 143(1):29-36, 1982.
[17] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778. 2016.
[18] J. Hermans, V. Begy, and G. Louppe. Likelihood-free MCMC with amortized approximate ratio estimators. In Proceedings of the 37th International Conference on Machine Learning (ICML2020), volume 119, pages 4239-4248. 2020.
[19] Hido, Y. Tsuboi, H. Kashima, M. Sugiyama, and T. Kanamori. Statistical outlier detection using direct density ratio estimation. Knowledge and information systems, 26(2):309-336, 2011.
[20] J.B. Huriart-Urruty and C. Lemaréchal. Fundamentals of convex analysis. Springer Science Business Media, 2004.

---

[21] T. Joachims. A support vector method for multivariate performance measures. In Proceedings of the 22nd international conference on Machine learning (ICML), pages 377-384, 2005.
[22] A. Krizhevsky. Learning multiple layers of features from tiny images. 2009. techincal report.
[23] S. Kullback and R. A. Leibler. On information and sufficiency. Annals of Mathematical Statistics, 22:79- 86, 1951.
[24] Y. Lei and Y. Ying. Stochastic proximal auc maximization. Journal of Machine Learning Research, 22(61):: 2021.
[25] Z. Lin, A. Khetan, G. Fanti, and S. Oh. Pacgan: The power of two samples in generative adversarial networks. In Advances in Neural Information Processing Systems 31 (NeurIPS 2018), volume 31, 2018.
[26] L. B. Lusted. Signal detectability and medical decision-making. Science, 171(3997) 1971.
[27] J. Neyman and E. S. Pearson. On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of Mathematical or Physical Character, 231:289-320 -377, 1933.
[28] X. Nguyen, M. Wainwright, and M. I. Jordan. Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. In Advances in Neural Information Processing Systems 20, pages 1089-1096. 2008.
[29] X. Nguyen, M. J. Wainwright, and M. I. Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56 (11):5847-5861, 2010.
[30] S. Nowozin, Cseke, and R. Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Advances in Neural Information Processing Systems 29 (NeurIPS 2016), pages 271 271-279, 2016.
[31] M. D. Reid and R. C. Williamson. Information, divergence and risk for binary experiments. Journal of Machine Learning Research, 12(22):731-817 2011.
[32] L. Rosasco, M. Belkin, and E. De Vito. On learning with integral operators. Journal of Machine Learning Research, 11(2), 2010.
[33] B. Scholkopf and A. J. Smola. Learning with kernels: support vector machines, regularization, optimization, and beyond. MIT press, 2001.
[34] B. K. Sriperumbudur, K. Fukumizu, A. Gretton, B. Schölkopf, and G. RG Lanckriet. On the empirical estimation of integral probability metries. Electronic Journal of Statistics, 1550-1599, 2012.
[35] B. Sriperumbudur, K. Fukumizu, A. Gretton, A. Hyvärinen, and Kumar. Density estimation in infinite dimensional exponential families. Journal of Machine Learning Research, 18(57):
1-59, 2017.
[36] I. Steinwart, D. R. Hush, and C. Scovel. Optimal rates for regularized least squares regression. In Proceedings of Conference on Learning Theory (COLT) 2009, pages 79-93, 2009.
[37] X. Tong. A plug-in approach to neyman-pearson classification. Journal of Machine Learning Research, 14(56):301130400 2013.
[38] M. J. Wainwright, High--Diensional Statistics: A Non-Asymptotic Viewpoint Cambridge University Press. 2019.
[39] Y. Ying, L. Wen, and S. Lyu. Stochastic online auc maximization. In Advances in Neural Information Processing Systems 29 (NeurIPS 2016), volume 29, 2016.
12

---

Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
[Yes] In introduction, we discussed the background/usages ROC curves, f-divergence estimation and provided full paragraph summary of the contribution of this paper. We give the same contribution rundown in the abstract as well.
(b) Did you describe the limitations of your work?
[Ye]] In Section 5, we included brief discussion comparing the proposed maximal AUC lower bounding approach vs. the classic Wilcoxon-Mand Manoo--an-hnneeeyyy Whitney statistic.
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results?
Yes] See Section 4.3
(b) Did you include complete proofs of all theoretical results?
[Yes] See Appendix
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)?
[Yes] Full instructions how to reproduce our experiments is provided in the supple- mentary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] See Section 6.2
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)?
[Ye]] See Section 6.2
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)?
[N/A] The Computation time is not compared this paper.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 6.2
(b) Did you mention the license of the assets?
[No] The authors of CIFAR-10 10 do not specify a license for the dataset.
(c) Did you include any new assets either in the supplemental material or as a URL?
[No] We did not create any new asset in this research.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating?
[No] We did not collect any data ourselves.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content?
[N/A] CIFAR-10 dataset we use in our experiment does not contain any identifiable human information.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable?
[N/A]
13

---

(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable?
[N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation?
[N/A]
14


Reviews:
[{"review_id": "c2mWu7eURmz", "reviewer": "Reviewer_zcmh", "summary": "This paper discusses f-divergence with arc length of ROC curve.  The authors find that several papers use ROC curves to compare two distributions.  And then try to answer the reason why arc length of ROC curve could be a type of f-divergence. The authors build the connection between the arc length of ROC curve and f-divergence, and then derive the algorithm to estimate the arc length of ROC curve. The experiments on Cifar-10 show that the score function obtained from 2-step procedure produced promising results.", "questions": "It would be great if the author could address the questions in weakness and comments.", "limitations": "none", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The paper provides a good presentation to the paper.\n\n2. The paper starts with a question of using ROC curves to compare two distributions, and then answers this question step by step. The deviations in the main body make sense. \n\n\nWeakness:\n1. Although the authors have convinced me that the arc length of ROC curve is a type of f-divergence similar to KL divergence, total variational distance, the relationship with them are not discussed well. I only find proposition 3 discuss the relationship with total variational distance. More relations are appreciated to be provided. In addition, in the experiments, the authors only show the rationality of the arc length of ROC curve, the discussion and comparison with other f-divergences are missed here.\n\n2. Although the paper is of a good presentation, still has some minor types, like line 30 the font of f should be mathematical way, line 32, there has one more space between [28] and ‘.’.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "c2BBG4T8FII", "reviewer": "Reviewer_WSBr", "summary": "This paper propose a novel method that utilizes arc length of the AUROC curve to approximate AUROC itself. Then, propose to optimize the arc length of the curve with some theoretical justifications for the relationship with AUROC. ", "questions": "Please refer to the weakness part above. I like the new idea, but don't think it is ready to be published.", "limitations": "I don't see any limitations for societal impact.", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The idea is novel and interesting. The paper provides some illustrations for the idea. \n\nWeakness: \n\ni) Optimizing arc length is not consistent with optimizing AUROC itself. As the examples shown in Figure 1, people could have the same optimized arc length, but have different optimized AUROC (subfigure a and b). Or, people could have the same optimized AUROC but have different optimized arc length (subfigure b and c).\n\nii) Although the idea is novel as an alternative way to optimizing AUROC, the paper lack some justifications for the advantages over the main-stream AUC optimization methods based on surrogate loss functions. (Gao, Wei, et al. \"One-pass AUC optimization.\" International conference on machine learning. PMLR, 2013.;  Ying, Yiming, Longyin Wen, and Siwei Lyu. \"Stochastic online AUC maximization.\" Advances in neural information processing systems 29 (2016).;  Liu, Mingrui, et al. \"Fast stochastic AUC maximization with $ o (1/n) $-convergence rate.\" International Conference on Machine Learning. PMLR, 2018. etc.)\n\niii) The experiments are insufficient. Only CIFAR10 is conducted for the experiment. And only logistic regression and a pairwise squared loss optimization method are considered as comparisons. There is no deep learning experiments. Moreover, even based on the current experimental results, the proposed method doesn't outperform the pairwise loss optimization. \n\niv) There are some short claims for the computational efficiency of proposed method, but there is no justifications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_dVWybjPaNd", "reviewer": "Reviewer_2jFB", "summary": "f-divergences can be written as variational objectives, typically parameterised in terms of a density-ratio estimator between 'positive' and 'negative' distributions. This paper identifies a new equivalence between a particular kind of f-divergence and the arc length of the `optimal AUC curve' (i.e. the best possible AUC curve amongst all classification scoring-functions). Convergence guarantees are given when the  estimated (arctan) log-density-ratio belongs to an RKHS. Finally, this estimator is used to (approximately) lower bound the optimal AUC, which is an important quantity in unbalanced binary classification settings, as shown in the experiments on Cifar-10.", "questions": "Could the authors comment on the added value of their theoretical convergence results over that of Nguyen et al. ? (2007, \"Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization). A superficial glance at that paper suggests that they prove a rather similar result for *arbitrary* f-divergences.\n\nIf I am correct that your results are similar to the prior work (but perhaps involve small modifications), then this should be made clear in the exposition.\n\nRight now, I have assigned a score of 'fair' to the contributions of this paper. I would be willing to raise this if these concerns regarding theoretical novelty turn out to be wrong.\n\n\nMinor points\n\nline 40: the term 'surface area' is unclear here.  How does it differ from the usual AUC? needs clarifiying.\nline 73-74: Readers may be used to thinking of FPR & TPR as fixed scalars (rather than functions of a threshold). I think it could help to give these functions arguments and perhaps refer to such arguments as `thresholds'.\n\nminor correction: line 40: `maximual' --> `maximal'", "limitations": "The authors are very upfront about certain limitations (e.g. the fact that they require differentiabiltity of the score function w.r.t its input, unlike prior work).\n\nOne issue is that their `lower bound' on the AUC does not appear to truly be a lower-bound (since they have to replace certain terms with approximations). This limitation could be made clearer, since right now, the conclusion implies otherwise.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- It's an interesting, non-obvious finding that the ROC curve can be connected to variational f-divergence representations. Also, it takes some ingenuity to consider the arc-length rather than the more standard AUC.\n\n- As far as I can tell, the theoretical results look correct (although I have concerns about novelty - see below)\n\n- The experiment is interesting and shows some promise (but I have concerns - see below).\n\nWeaknesses\n\n- I have questions about the novelty of the theoretical results. See Questions below.\n\n- I think it is *very* important to report the wallclock times for the different methods used in the experiments. I understand that the proposed method has better asymptotic complexity, but I want a sense of how big those hidden constant factors are.\n\n- Section 4.1 (line 146 onwards) & Section 4.2 were hard to follow. In equation 5, the variational formula is reparameterised in term of an angle v \\in [0, \\pi/2]. At this point in the exposition, it is quite unclear why such a reparameterisation is a good idea; it is more standard to parameterise the variational formla in terms of the log-density-ratio, which belongs to [0, \\infty]. A whole page later, we discover that the log-density-ratio approach (combined with a linearity assumption) is theoretically less convenient because of nonconvexity. I think that this nonconvexity issue could be mentioned *before* Equation 5, so the reader understands where the argument is going.\n\n- Section 5 (lines 232-256) is very math-notation-heavy, making it difficult to read. More plain-english explanations would greatly improve clarity. For instance, many explanations could be stated in terms of the lines in Figure 2, rather than using math.\n\n- the abstract is rather vague, and I struggled to understand it (In contrast, the introduction was quite clear). I recommend clarifying what is meant by `score function', since it has multiple meanings (e.g. the gradient of log-likelihood w.r.t parameters/data).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6YJSYdbNikH", "reviewer": "Reviewer_RsuF", "summary": "This paper builds a connection between the ROC curve of testing two distribution $p_{+}(\\boldsymbol{x})$,  $p_{-}(\\boldsymbol{x})$ and the \"distance\" between these two distributions. In particular, the length of the optimal ROC curve corresponds to a type of $f$-divergence between $p_{+}(\\boldsymbol{x})$ and $p_{-}(\\boldsymbol{x})$. This $f$-divergence can be estimated utilizing its variational formulation. The optimal solution of the equivalent variational problem is $\\arctan[\\frac{p_{+}(\\boldsymbol{x})}{p_{-}(\\boldsymbol{x})}]$. Therefore, by solving the empirical version of this variational problem, an estimate of $\\arctan[\\frac{p_{+}(\\boldsymbol{x})}{p_{-}(\\boldsymbol{x})}]$ can be obtained.", "questions": "1. For the two-step procedure stated in Line 264, why do we need the second step? The goal is to estimate $\\arctan[\\frac{p_{+}(\\boldsymbol{x})}{p_{-}(\\boldsymbol{x})}]$, which has been done in Step 1.\n\n2. It turns out that the optimal solution of the variational problem (Line 141) is a monotone transform of likelihood function. Can the author comment on under which choice of $g$, this is the case?\n\n3. Is there a typo in Eq. (1): Should it be $[\\partial_{\\tau}\\tilde{F}_{-}(\\tau)]^2$ ?\n", "limitations": "NA", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The fact that the length of optimal ROC curve corresponds to a $f$-divergence is an interesting new result. Also the presentation of this paper is clear and fun to read.\nWeakness: I think the condition in Proposition 4, which requires that $\\arctan[\\frac{p_{+}(\\boldsymbol{x})}{p_{-}(\\boldsymbol{x})}] \\in \\mathcal{H}$ is a little stringent. In practice, to reduce computational complexity, the class $\\mathcal{H}$ may not be very rich. It is better to discuss what will happen when there exist some approximation error of $\\arctan[\\frac{p_{+}(\\boldsymbol{x})}{p_{-}(\\boldsymbol{x})}]$. Also I think it will be better if more numerical experiments are provided, e.g., comparison with other methods of choosing score function $t(x)$ (if any) and experiments in other datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept