Characterising the Area Under the Curve Loss Function Landscape
Authors: Maximilian Paul Niroomand
Conor T Cafolla
John William Roger Morgan
David John Wales
Venue: ICLR 2022 Submitted (2022)
Keywords: loss function landscape
loss function
AUC
area under the curve
alternative loss functions
loss function visualisation

Abstract:  One of the most common metrics to evaluate neural network classifiers is the
area under the receiver operating characteristic curve (AUC). However, 
optimisation of the AUC as the loss function during network
training is not a standard procedure. Here we compare minimising the cross-entropy (CE) loss
and optimising the AUC directly. In particular, we analyse the loss function
landscape (LFL) of approximate AUC (appAUC) loss functions to discover
the organisation of this solution space. We discuss various surrogates for AUC approximation and show their differences.
We find that the characteristics of the appAUC landscape are significantly
different from the CE landscape. The approximate AUC loss function improves
testing AUC, and the appAUC landscape has substantially more minima, but
these minima are less robust, with larger average Hessian eigenvalues. We provide a theoretical foundation to explain these results.
To generalise our results, we lastly provide an overview of how the
LFL can help to guide loss function analysis and selection. 

Under review as a conference paper at ICLR 2022
CHARACTERISING THE AREA UNDER THE CURVE LOSS FUNCTION LANDSCAPE
Anonymous authors Paper under double-bling review
ABSTRACT
One the most common metrics to evaluate neural network classifiers is the area under the receiver operating characteristic curve (AUC). However, optimisation of the AUC as the loss function during network training is not a standard procedure. Here we compare minimising the cross-entropy (CE) loss and optimising the AUC directly. In particular, we analyse the loss function landscape (LFL) of approximate AUC (appAUC) loss functions to discover the organisation of this solution space. We discuss various surrogates for AUC approximation and show their differences. We find that the characteristies of the appAUC landscape are significantly different from the CE landscape. The approximate AUC loss function improves testing AUC, and the appAUC: landscape has substantially more minima, but these minima are less robust, with larger average Hessian eigenvalues. We provide theoretical foundation to explain these results. To generalise our results, we lastly provide an overview of how the LFL can help guide loss function analysis and selection.
INTRODUCTION
The area under the curve of the receiver operating characteristic curve (AUC) is commonly used method to evaluate the accuracy and reliability of a neural network classifier. However, for mathematical reasons, the AUC cannot be used as as the loss function to be minimised during neural network training (Menon & Elkan, 2011). Instead, other functions such as cross-entropy are commonly employed. The stepwise nature of the AUC function is the reason that it is non- differentiable and hence cannot simply be optimised. However, the AUC can be approximated using surrogate losses such as the sigmoid function. This approach can lead to formulation equivalent to the Soft-AUC described by Calders & Jaroszewicz/ (2007), which is referred to here appAUC. We find it intuitive to optimise function that as close as possible to the one used to evaluate the model. Our main contributions in this paper are therefore:
To understand the use of different approximation the AUC as the loss function employed in training of neural network
To explore the organisation of the appAUC landscape and compare it to 'standard' cross- entropy landscape
A theoretical foundation of the differences between appAUC and cross-entropy landscapes
To outline how a loss function landscape analysis can be useful for loss function selection
To better understand the advantages and disadvantages the approximated AUC loss function, we study the functional space, commonly referred to as LFL, using tools from the theoretical study of energy landscapes in molecular and condensed matter systems (Wales) 2003). The usefulness of the energy landscape approach has previously been demonstrated in the context of neural network LFLs (Ballard et al. 2017). We will employ methods from this approach to gain insights about geometric features of the LFL, including the number of minima, their curvatures, and their connectivity. By repeatedly surveying large parts of the LFL, we are, with high probability, able to find the true global minimum. Additionally, due to Metropolis criterion in the global optimisation approach described below, we do not get stuck in a local minimum, and explore the full LFL, hence learning more about the functional surface. Instead of single minimum, aim to find large number of minima that,

---

Under review as a conference paper at ICLR 2022
together with transition states, provide a faithful coarse- boars--griined representation of the loss function landscape. We believe that this approach will yield valuable insights into the use of appAUC as a loss function in neural networks.
Various interesting questions arise from the use of an appAUC loss function. Besides a comparison of properties between landscapes, and the effects of hyperparameter changes, interested in the differences between appAUC and CE landscapes. Both loss functions, for the same neural network architecture, address the learning problem of finding a mapping f from input data to class label. Does this common foundation imply that minima of the CE loss function are also minima of the appAUC function, or are they at least very similar to each other? We will show below that this condition does not hold, and explain why. Furthermore, to the best of our knowledge, there has been no previous research into the functional properties of AUC surrogates. We believe that quantifying inherent, geometric properties of loss functions will provide more fundamental understanding of the applicability of particular loss functions to distinct machine learning problems.
especially
1.1 RELATED WORK
This contribution lies at the intersection of three different areas of research: general understanding and study of loss functions, the appAUC loss function particular, and the study of loss function landscapes to understand neural networks. Optimising the AUC as a loss function has been considered before (Cortes & Mohri 2004). It has been shown that testing AUC is improved when a loss function is chosen that is closer to the true AUC function (Yan et al. 2003). Nonetheless, optimising loss functions that approximate the AUC is rarely considered. One reason for this situation may be that the computational complexity is formally O(N"), or specifically in the binary classification case employed here, O(NPNN) where Np are the positive data points and NN the negative ones. Other reasons include the non-convexity and perceived complexity of the appAUC landscape, and importantly the fact that it has zero derivatives almost everywhere (Ghanbari & Scheinberg 201 8). The loss function is one of the most critical choices in the design of a neural network, because is the element that underlies the entire learning procedure. At first glance, it may appear that all loss functions solve the same problem for given neural network, which led Rosasco et al. (2004) to question the practical differences between alternative loss functions. Yet. it is widely accepted that different loss functions allow us to optimise for specific properties (Janocha & Czarnecki 2017). By studying the appAUC LFL, we can quantitatively address these important questions.
The topology of the loss function has been a topic of interest for over 20 years (Hochreiter & Schmidhuber 1997). Understanding the loss function for a given neural network can help establish the fundamental interpretation of the "black-box', as such networks are often described (Li et al. 2017). However, key problem with studying the LFL is the large associated computational cost. Unlike the standard machine learning approach, where only a single minimum is located, the LFL approach attempts sample representative set of minima, which may often exceed 10,000 solutions (Ballard et al., 2017). Computing the true number of minima for a loss function would require enhanced sampling techniques (Wales) 2013 Martiniani et al. 2016).
Recently, others have looked into the I.FL of overparameterised neural networks (Cooper 2018) and shown that various properties can be exploited to improve accuracy and, importantly, robustness of neural networks (Baldassi et al. 2020: Chaudhari et al., 2019)) Insights from the LFL have also been used to understand initialisation methods and their relative success (Fort& & Scherlis 2019). However, to the best of our knowledge, there has been no research so far into interpreting individual loss functions from the underlying LFL. In this work, we aim to address this knowledge gap, and show that studying the LFL can provide important insights into our understanding and design of neural networks and the associated loss functions.
METHODS
2.1 NEURAL NETWORK
For this initial survey, we consider neural networks with a single hidden layer. We denote the set of data points as D (X, c), containing N := [D] elements. For given classification problem with C classes, a data point E D has input features x and a known output class, cd. We use a nonlinear tanh activation function and convert output values yı, E C into softmax probabilities by
2

---

Under review as a conference paper at ICLR 2022
Pa (W:x") explys (W:x*)| Σラ= (xply)(W;;"" (W:x"). where W denotes the weight vector contain- ing all weights of the neural network. As a reference to compare with our appAUC loss function, we use a cross-cntropy (CE) loss function, defined as:
CE(W; x)
1 N
Σ
In(p (W;x") XW?
The XW2 in equation 1 represents L2 regularisation term, which eliminates Hessian Eigen- values and counteracts overfitting. We fix = 10-3 for fair comparison between loss functions.
2.2 AREA UNDER THE CURVE
To evaluate model such as the described above, it is standard practice consider the receiver operating characteristic (ROC) and calculate the area under the curve (AUC) (Hastie et al. 2009). For given classifier, the ROC is plot of the true positive ratio, T(P), against the false positive ratio, F(P). These quantities are defined by
T(W:X,P) Σ 8(c - () (1000(/(::0)) (W:x*) P)
(2)
Ea=1 8(c4 1)
N F(W:X,P)= Ed=n(1 8(c² 1))0(p1 (W; (Wix*) PP
(3)
Ex=1 5(cd 1)
where 6(c - 11 1) is the Dirac delta function, and Opp P) is the Heaviside step function, defined as
-(c² 1 11
1 if 2 if P
@(1 P) P)=
(4)
if +
and P is parameter, which data point belongs to class (p1). Note that the choice of class as positive reading is arbitrary, and for a multi-class system, any class may be chosen.
a cutoff probability for the neural network to predict that a given
The ROC curve is defined by T and F as P varies from O to 1. For a perfect classifier, the ROC would simply be horizontal line would then be 1. The AUC is therefore measure of how close the model is to a perfect classifier. Formally, the AUC as a function of network weights W, parameterised by the data X, given by
1 (i.e. P + 0, T(P) 1), and the area under the curve
AUC- = S/ T(W;X, (FWW;XXP) Np.N
1 ΣΣ (W; (W;x") (5)
where p labels positive data points (class 1), and labels negative data points (not in class I).
2.3 APPROXIMATED AUC: APPAUC LOSS FUNCTION
Most optimisation routines benefit from analytical derivatives of the function to be optimised. The function defined in equation] has smooth analytical derivatives, but equation 5 does not, because of the discontinuous step function. However, in similar way to other approaches (e.g. (Calders & Jaroszewicz 2007)), one can replace the discontinuous & function with approximate, smooth, analytical surrogate function, which can then be differentiated and optimised. We write
AUC(W;X) R X) = Np.N ΣΣ 1+ exp(-3(pr
1 (1:0:p((::::)) Pı (W))")))
(6)
where the Heaviside step function has been replaced with a smooth sigmoid function:
(x)) + +==) = 1 exp(-Bz)
with parameter 33 that is discussed in detail in the Appendix. An important consideration for surrogate AUC loss functions is that they do not change the optimal solution when replacing the step function. Such surrogates are referred to as AUC-consistent (Agarwal, 2014). Charoenphakdee et al. (2019) show that sigmoid is an AUC-consistent surrogate. The appAUC loss function in equation 6 is now differentiable and can hence be used for optimisation, where minimise the negative form of equation 6 such that a minimum (i.e. lower loss) is a good solution of higher AUC. For reference, we have included the analytical first and second derivatives for Equation 6 in the Appendix.
3

---

Under review as a conference paper at ICLR 2022
2.3.1 OTHER SURROGATE LOSS FUNCTIONS
There exist many possible ways to approximate
sigmold
the AUC. Besides the sigmoid 7, we also show results for the popular (Gao & Zhou 2015)
lingee
exp
hinge {(2) = max (0, - 2). exponential (2)
square
exp(-22, quadratic ((z) (1 2)² and tanh surrogates (Fig. D. where z denotes the differ- in probability between true and false class Le. Pr- Py. Note that for the sigmoid in Figure 40 (Appendix). It important note that all but the hinge loss are AUC-consistent (Gao & Zhou 2015 Charocuphakdee et al. 2019). AUC-consistency implies that the optimal solution remains unchanged with the surrogate loss replac- ing the true AUC. These different losses can all be used as surrogates because they include pairwise comparison of positive and negative case probabilities as for the AUC.
[
0-
-1.
-1.0
-0.5
0.0
0.5
1.0
3.5
Figure 1: Surrogate loss functions.
2.4 OPTIMISATION ROUTINES
Surveying the LFL for a neural network requires methods to locate minima of the landscape and connect them via transition states, defined as index one saddles (Murell & Laidler 1968). We perform global optimisation using the basin-hopping method, which uses Metropolis criterion to
avoid getting stuck local minima (L. & Scheraga 1987; Wales Doye 1997). Further details included in the Appendix. We use modified quasi-Newton L-BFGS minimiser (Nocedal 1980). Connectivity between minima is defined by transition states using doubly-nudged (Trygubenko & Wales 2004a|b) elastic band (Henkelman & following (Munro & Wales 1999 al. (0144). These methods are implemented in the GMIN (Wales) a). OPTIM (Wales) and PATHSAMPLE (Wales] programs, which are available for use under the GNU GPL.
regarding landscape exploration using basin basinnhopping
Jónsson [2000] Henkelman et al. (000) approach and hybrid-eigenvectte
2.5 DISCONNECTIVITY GRAPHS
To visualise the -igediimnsiooal loss function landscape, we employ disconnectivity graphs (Becker & Karplus 1997 Wales al. 1998), which provide characteristic representation of the organisation for a nigh-dimensional surface. The vertical axis corresponds to increasing loss value, highlighting the barrier heights between local minima, and the horizontal arrangement chosen to avoid crossings between the branches. In molecular science the vertical axis corresponds to potential or free energy, so the loss value plays the role the energy in the LFL. Any node at which two or more branches split up can be understood as the minimum energy (loss) required for the system to move between the sets of minima associated with the respective branches. Such points correspond to the energies of transition states, discretised at regular set of threshold values. Each leaf (tip) of the graph corresponds to minimum of the LFL; the lower the leaf is, lower loss value. The lowest-lying leaf is the global minimum. Disconnectivity graphs provide a concise and intuitive way to visualise complex topologies, and preserve a faithful representation of the barriers on the landscape.
2.6 DATASET
The LFL is not only a function of the weights, but also of the data, as shown in equation We will present our analysis of the AUC landscape mostly using the synthetic spiral dataset, introduced by Lang & Witbrock (1888. Spiral data remains one of the most popular synthetic datasets due to its high degree of non-linear separability. We add small uniform noise term increase the complexity of the problem and with an 80-20 train-test split. We acknowledge the limitations of applicability of synthetic dataset and therefore supplement our results using real-world dataset of fraudulent credit card transactions (Le Borgne& & Bontempi 2004). The dataset is highly imbalanced with only 10% of all datapoints classified fraudulent transaction.
4

---

Under review as a conference paper at ICLR 2022
3 RESULTS
In this section, we highlight three distinct points of interest. Firstly, we visualise and outline key differences between the AUC and CE landscapes. Secondly, present extended quantitative and geometric analysis of different AUC surrogates. Thirdly, we provide a comparison between minima of the AUC and CE landscape in terms of their classification properties.
3.1 LFL PROPERTIES
The loss function landscapes for LUC-approximated and CE loss functions look substantially different (Fig. () We performed landscape exploration using PATHSAMPLE until no new minima were found. For computational reasons, we only compare sigmoid appAUC with CE, and show below that for shorter timescales, sigmoid looks similar to the other surrogates. In particular, we find that the CE landscape is much more convex or funnelled than the appAUC landscape, with larger uphill barriers. In molecular science, this structure is associated with self-organising systems where relaxation to the global minimum relatively efficient (Wales 2003).
I
I 001
(a) appAUC landscape
(b) CE landscape
Figure 2: Discomnectivity graphs showing characteristic LFLs for sigmoid and CE. The vertical axes of the landscapes are not directly comparable, as they correspond different loss functions.
Table I: Characteristic summary statistics for LFLs after exhaustive landscape exploration.
AUC
LPPHE
Loss function
minima
best
μ
# transition states
Sigmoid
13,948
0.81
0.7
0.04
25.957 3,606
-93
7.2
1,903
0.77
0.66
0.03
-110
4.4
In Table [ we report some summary statistics characterise the two different LFLs numerically. The last column reports the mean of the product of positive Hessian Eigenvalues (LPPHE) for all minima of the landscape. This measure of basin geometry provides insight into the local curvature

---

Under review as a conference paper at ICLR 2022
and hence the 'flatness' of minimum (Verpoort al. 2020) and the volume of the corresponding basin attraction (Mezey 1987 Wales 2003). The smaller the LPPHE, the flatter the local basin. minima and transition states than
Table shows that the appAUC landscape has substantially the CE landscape. Further, the mean LPPHE of all appAUC minima larger than for the CE minima.
3.2 AUC SURROGATES
Table shows a comparison of the different appAUC surrogates (Section 2.3.1 after a week of landscape exploration. We find that these functions behave similarly, with comparable AUCs found and more minima than the CE landscape. The mean LPPHEs vary, the higher values belonging to the stepwise AUC, namely sigmoid and tanh. They
the two functions that are true approximations are all nonetheless larger or equal to the CE LFL in Table ! The disconnectivity graphs for these alternative APPAUC functions in Figure 2 the sense that they appear very wide (referred to as "glassy" in molecular sciences) and much less single-funnelled than the CE L.F...
shown in Figure 3 They are all similar to the apPAUC landscape
-0.537
-0.482
0.972
0.586
0..42
0.887
-0.634
-0.602
0.803
-0.682
-0.661
0.719
0.7330
-0.721
0.635
-0.778
-0.780
0.551
(a) Sigmoid
(b) Tanh
(c) Hinge
0.256
0.154
0.8
0.117
0.072
0.75
0.002
00.100
0.7
AUC
-.160
0.093
0.65
0.299
0.175
0.6
0.5
0.448
0.257
(d) Quad
(e) Exp
Figure 3: Disconnectivity graphs showing the LFLs for the alternative surrogate AUC loss functions. The minima coloured by testing AUC. Note that for sigmoid and tanh, the energy/loss value directly interpreted as (negative) AUC.

---

Under review as a conference paper at ICLR 2022
Table 2: Summary statistics for LFLs with alternative appAUC functions and CE comparison.
AUC
LPPHE
Loss function
# minima
best
o
transition states
p
Sigmoid
4,202
0.78
0.67
0.04
4,264
-99
Hinge
6,086
0.74
0.64
0.03
5,822
-109
12
Tanh
3,562
0.79
0.68
0.05
3,000
-94
14
Quad
3,606
0.77
0.65
0.04
3,464
-103
11
Exp
2,563
0.77
0.64
0.03
2,515
-110
12
3.3 LOSS-AUC CORRELATION
Below, show the correlation between loss value (energy) and AUC for test data for the different surrogate losses (Figure 4 In all five cases, there a significant negative correlation. Tanh has the most negative Pearson correlation coefficient of -0.83, followed by sigmoid and hinge.
Quad
0.6
0.5-
-0.7
-0.5
0.6
0.7
1.0
-04
-0.2
0.0
0.
0.4
-0.2
-0.1
0.0
0.1
Laci
Lacn valua
Lass
Lonevalze
Figure 4: Loss vs AUC correlation for the five different AUC surrogate functions. Minima at lower values of loss generally correspond to higher AUC values.
3.4 APPAUC MINIMA IN THE CE LFL.
Another point of interest is understand the relationship between CE loss value and testing AUC for different LFLs. In Figure 5 we use sigmoid to represent appAUC. We see that the negative correlation between CE loss and AUC much stronger for minima of the CE than for the appAUC landscape. Hence, for most minima of the appAUC landscape, the CE loss values are relatively high, even though some of them may in fact have higher AUC that the CE minima. This result raises the question of how good diagnostic the CE loss is for the testing AUC. The insights into the position of appAUC minima within the CE LFL can be combined with the previous results on appAUC minima and the minimum they map to in the CE LFL. Above, we have identified that there exist many appAUC minima that are not minima of the CE LFL and thus have higher CE loss value, but have comparable testing AUC. In general, while the best appAUC minima have reasonably low CE loss, the mean loss 1.2 is nearly twice as high as for CE minima. The median of 0.8 is slightly lower, indicating few very high loss minima, which lie in the lower right hand corner of Figure
Lendscape:
appAUC
0.7
0.6
0.4
0.0
2.5
7,0
7.5
10.0
12.5
15.0
17.6
CE
loss
Figure 5: Correlation between CE loss and testing AUC for minima from CE and sigmoid LFLs.
7

---

Under review as a conference paper at ICLR 2022
3.5 CREDIT CARD DATASET
We perform the same analysis on the imbalanced, real-world credit card dataset. Optimising AUC on such datasets expected to particularly useful, because the AUC measure invariant to class imbalance. The uppAUC landscape has more, better minima, and larger LPPHEs (Table As above, the CE landscape single funnelled while appAUC landscape more glassy, similar figure
Table Summary statistics for credit card fraud LFLs
AUC
LPPHE
Loss function
# minima
best
transition states
11
sigmoid
3,448
0.99
0.96
0.02
2343
-772
38
CE
1,721
0.97
0.96
0.01
1584
-858
DISCUSSION
We observe that the appAUC and CE LFLs have rather different characteristics, emphasising the point that different loss functions are truly different. In this section, we discuss and explain these observations, both empirically and theoretically. We explain why an analysis of the LFL can offer unique insights into the understanding of loss functions. Different characteristies of such a function may be important, and studying the LFL can help to choose an appropriate loss function. We also provide comparison of the different AUC surrogates.
4.1 COMPARISON Loss FUNCTION LANDSCAPES
As the disconnectivity graphs in Figure 2 show, the CE landscape is more funnelled, and there are fewer minima. However, not only does the number of minima vary, the LPPHE also indicates that the minima appear rather different. Flatter minima (smaller LPPHE) are sometimes considered advantageous, because they may be more robust to perturbations in the weights (Hochreiter & Schmidhuber 1997), and therefore potentially also to noise in the training data. The increased robustness is explained by a lower deviation from the loss value of the global minimum for small deviations in training data model hyperparameters in flatter landscapes where the gradients small. This feature may constitute a geometrically-insiied argument against using the appAUC landscape if appAUC minima are less robust. The data suggest that average the CE minima relatively flat, which may translate to increased robustness to perturbations of the weights.
The location of appAUC minima within the CE LFL raises several new questions. Figure 5 illlstrrtes the weakly negative correlation between CE loss and testing AUC for appAUC minima, much weaker than for CE LFL minima. This result initially seems intuitive, as we are looking the CE LFL. The fact that the correlation between testing AUC and loss is more strongly negative for points of the CE loss function means that the CE loss works well the minima its own LFL. However, there also exist points in the CE LFI. that are not minima (but minima of the appAUC) with higher loss and also higher AUC. Hence, under the assumption that appAUC the more 'accurate' loss function, this analysis highlights some weaknesses of the CE LFL, namely not identifying many better, correct solutions. A mathematical foundation of the limitations of CE as opposed to appAUC given in the next section.
4.2 THEORETICAL INTERPRETATION
To explain our empirical results, we can provide theoretical basis for some of the observations made. Notably, the substantially higher number minima of the appAUC LFL is explained by mathematical properties of the solution space for CE in relation to appAUC loss generally. In fact, the solutions to
CE loss are subset of the solutions to the appAUC loss (Menon & Williamson 2016). The optimal solution of appAUC is any function that has a strictly monotonic relationship with P(y 1|z) (aao & Zhou 2015). On the other hand, the optimal solution of CE loss is P(y = 1 1|z). Thus, a space of solutions for appAUC is infinitely larger than CE loss and therefore it should have more minima. The
8

---

Under review as a conference paper at ICLR 2022
CE loss with Softmax belongs to the class of 'strictly proper composite" losses (Reid & Williamson 2010) which estimates p(y = [ In statistics, this is also referred to as 'proper scoring rule' (Buja et al. 2005). Obviously, Ply = 1|z) has strictly monotonic relationship with itself and hence must be optimal w.r.t. AUC, however this relationship must not hold the other way, optimising AUC must not necessarily be optimal w.r.t. P(y = [ Furthermore, the same line of argument also allows us to explain why appAUC minima are not necessarily close to CE minima in Euclidean space. Minimising appAUC does not give any additional incentive find solutions of the kind P(y = 112²) compared to any other solutions which may have a strictly monotonic relationship with P(y = 1
4.3 APPAUC vs CE LOSS FUNCTION
The appAUC landscape we have considered has approximately 8 times as many minima as the corresponding CE landscape. The global minimum for appAUC achieves better testing AUC, and the mean AUC is also higher, yet there are also few especially poor minima the AUC landscape that are not found for the CE case. For practical applications the existence of solutions that are poor classifiers is probably not a concern, because finding low-lying minima is relatively straightforward. However, computation of the appAUC as done here is of order ((N2), where N is the number of data points, which is much more intensive that the (N) requirement associated with CE. Additionally, minima of the CE LFL seem to be more robust. In summary, exploiting the appAUC directly is more attractive if computational cost is not problem, especially if the loss function landscape can be studied extensively. For machine learning applications involving large deep networks, where only single minimum is identified, and where computational cost is a major concern, the CE landscape may be the best choice. This result strengthens the case for using CE as a 'standard' loss function, while also highlighting its limitations, and illustrating how using different loss functions can be beneficial.
4.4 AUC SURROGATES
We observe that the LFLs for all AUC surrogates look roughly similar, and distinctly different to the CE LFL., providing a strong argument for a fundamental geometric difference between these loss functions that has not been shown before. We also observe that the hinge loss has substantially more minima than all other surrogates, yet the testing AUCs are much worse. This may be explained by the fact that hinge is not AUC consistent which means that the hinge functional space does not contain the AUC-optimal solution for this problem. Quad, tanh and sigmoid have broadly similar characteristics with small deviations in the LPPHE while the exponential does slightly worse in terms of AUC but has very flat minima, similar to the CE landscape. Hence, using an exponential AUC surrogate may provide more robust solutions than other surrogates. If the main objective however is to maximise testing AUC, tanh seems to be the best choice, as it not only has the highest AUC, but also the strongest inverse correlation between energy and AUC, Both tanh and sigmoid are directly approximating the stepwise AUC which explains the strongly negative correlations, while both exponential and squared exponential function behave broadly similar as expected.
4.5 CONCLUSIONS
Loss functions essential to guide the learning process in neural networks. We have provided detailed analysis of the approximate AUC landscape based on global exploration of tractable but realistic examples. In this contribution, we have proposed new method to understand and select loss function, based on geometric properties of the LFL. We have shown that LFLs for apPAUC and the standard CE LFL are qualitatively and quantitatively different. We have also provided detailed comparison of relevant LFL features between AUC surrogates and shown that the AUC-inconsistent hinge has much lower AUC than the other surrogates (but more minima) and that the exp surrogate has wide, robust minima, similar to the CE LFL. It is outlined that optimising the AUC improves testing AUC, but CE minima have on average smaller LPPHEs, which may make them more robust. In general, we observe tradeoff between robustness and testing AUC for all the loss functions. Furthermore, it must be noted that the greater computational cost of optimising the AUC directly is likely to be the main drawback of this approach. A quantitative analysis of geometric features of the LFL such as by my-miimmmm AUC, mmeryy-testingg AUC correlation, number of minima (landscape convexity) and LPPHE i.e catchment basin volumes which is connected to minima robustness allows insights into the strengths and weaknesses of loss functions. Hence, this analysis provides a valuable tool guide loss function selection for machine learning applications.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Shivani Agarwal. Surrogate regret bounds for bipartite ranking via strongly proper losses, The Journal of Machine Learning Research, 15(1):1653-1674 2014.
Carlo Baldassi, Fabrizio Pittorino, and Riccardo Zecchina. Shaping the learning landscape in neural networks around wide flat minima. Proceedings of the National Academy of Sciences, 117(1): 161-170, 2020.
Andrew J Ballard, Ritankar Das, Stefano Martiniani, Dhagash Mehta, Levent Sagun, Jacob D Stevenson, and David J Wales. Energy landscapes for machine learning. Physical Chemistry Chemical Physics, 19(20): 12585-12603, 2017.
Oren M Becker and Martin Karplus. The topology of multidimensional potential energy surfaces: Theory and application to peptide structure and kinetics. The Journal of chemical physics, 106(4): 1495-1517, 1997.
Andreas Buja, Werner Stuetzle, and Yi Shen. Loss functions for binary class probability estimation and classification: Structure and applications. Working draft, November, 3, 2005.
Toon Calders and Szymon Jaroszewicz. Efficient auc optimization for classification. In European Conference on Principles of Data Mining and Knowledge Discovery, pp. 42-53. Springer, 2007.
Nontawat Charoenphakdee, Jongyeong Lee, and Masashi Sugiyama. On symmetric losses for learning from corrupted labels. In International Conference on Machine Learning, pp. 961-970.
PMLR, 2019.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12): 2019.
Yaim Cooper. The loss landscape of overparameterized neural networks. arXiv preprint arXiv: 1000, 2018.
Corinna Cortes and Mchryar Mohri. Auc optimization vs. error rate minimization. Advances in neural information processing systems, 16(16):313-320, 2004.
Stanislav Fort and Adam Scherlis. The goldilocks zone: Towards better understanding of neural network loss landscapes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3574-3581, 2019.
Wei Gao and Hui Zhou. On the consistency of auc pairwise optimization. In Twenty-Fourth International Joint Conference Artificial Intelligence, 2015.
Hiva Ghanbari and Katya Scheinberg. Directly and efficiently optimizing prediction error and auc of linear classifiers. arXiv preprint 1rXiv:1802.02533 2018.
Hastie, R. Tibshirani, and J. Friedman. The Elements of Statistical Learning. Springer, New York, 2009.
Graeme Henkelman and Hannes Jónsson. Improved tangent estimate in the nudged elastic band method for finding minimum energy paths and saddle points. The Journal of Chemical Physics, 113((2)99978-999) -9907, December 2000.
Graeme Henkelman, Blas P. Uberuaga, and Hannes Jónsson. A climbing image nudged clastic band method for finding saddle points and minimum energy paths. The Journal of Chemical Physics, 113(22):9901-93) 9904,, December 2000.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1):1-42, 1997.
Katarzyna Janocha and Wojeiech Marian Czamecki. On loss functions for deep neural networks in classification. arXiv preprint arXiv: 1702 2017.
10

---

Under review as a conference paper at ICLR 2022
Kevin Lang and Michael Witbrock. Learning to tell two spirals apart. In Proceedings of the 1988 connectionist models summer school, pp. 52-59. San Matco, 1988. Issue: 1989.
Yann-A Le Borgne and Gianluca Bontempi. Machine learning for credit card fraud detection setection-practical handbook. ACM SIGKDD explorations newsletter, 6(1):1-6, 6, 2004.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. arXiv preprint arXiv:1712.099133,22 2017.
and H. A. Scheraga. Monte Carlo-minimization approach to the multiple-minima problem in protein folding. Proceedings of the National Academy of Sciences, (M(19)6661166111 October 1987.
Stefano Martiniani, K. Julian Schrenk, Jacob D. Stevenson, David J. Wales, and Daan Frenkel. Turning intractable counting into sampling: Computing the configurational entropy of three-dimensional jammed packings. Phys. Rev. E, 93:012906, 2016.
Aditya Krishna Menon and Charles Elkan. Link prediction via matrix factorization. Joint european conference machine learning and knowledge discovery in databases, pp. 437-452. Springer, 2011.
Aditya Krishna Menon and Robert C Williamson. Bipartite ranking: risk-theoretic perspective. The Journal of Machine Learning Research, 17(1):67666886) 2016.
P. G. Mezey, Potential Energy Hypersurfaces. Elsevier, Amsterdam, 1987.
Lindsey J. Munro and David J. Wales. Defect migration in crystalline silicon. Physical Review B, 59 (6):3969-3980, February 1999.
Murrell and J. Laidler. Symmetries of activated complexes. Trans. Faraday, Soc., 64:371-377, 1968.
Jorge Nocedal. Updating quasi-Newton matrices with limited storage. Mathematics of Computation, 35(151):773-773. September 1980.
Mark Reid and Robert C Williamson. Composite binary losses. The Journal of Machine Learning Research, 11:2387-2421 2010.
Lorenzo Rosasco, Ernesto De Vito, Andrea Caponnetto, Michele Piana, and Alessandro Verri. Are loss functions all the same? Neural computation, 16(5):1063-1076 2004.
Semen A. Trygubenko and David Wales. Analysis of cooperativity and localization for atomic rearrangements. The Journal of Chemical Physics, (1(11))6689-6697) October 2004a.
Semen A. Trygubenko and David J. Wales. A doubly nudged elastic band method for finding transition states. The Journal of Chemical Physics, 120(5):2082-0044 February 2004b.
Philipp C Verpoort, David Wales, et al. Archetypallanddscaee for deep neural networks. Proceedings of the National Academy of Sciences, 117(36)21857-21864 2020.
D. J. Wales. Energy Landscapes. Cambridge University Press, Cambridge, 2003.
David Wales. GMIN: program for basin-hopping global optimisation, basin-sampling, and parallel tempering, a.
David J. Wales. OPTIM: A program for geometry optimisation and pathway calculations,
David J. Wales. PATHSAMPLE: A program for generating connected stationary point databases and extracting global kinetics,
David J. Wales. Surveying a complex potential energy landscape: Overcoming broken ergodicity using basin-sampling. Chem. Phys. Letr. 584:1 9,, 2013.
David J. Wales and Jonathan P. K. Doye. Global Optimization by Basin-Hopping and the Lowest Energy Structures of Lennard-Jones Clusters Containing up to 110 Atoms. The Journal of Physical Chemistry A, (28))5551111-55 July 1997.

---

Under review as a conference paper at ICLR 2022
David J Wales, Mark A Miller, and Tiffany R Walsh. Archetypal energy landscapes. Nature, 394 (6695):758-777) -60,,, 1998.
Lian Yan, Robert H Dodier, Michael Mozer, and Richard H Wolniewicz. Optimizing classifier performance via approximation to the wilcoxon-mamn-whinng statistic. In Proceedings the 20th international conference on machine learning (icml-03), pp. 848-855, 2003.
Yi Zeng, Penghao Xiao, and Graeme Henkelman. Unification of algorithms for minimum mode optimization. The Journal of Chemical Physics, 140(4):0441I5 15, January
Jannurry 2014.
12

---

Under review as a conference paper at ICLR 2022
5 APPENDIX
5.1 CASINNG OPTIMISATION
Surveying the L.L. for neural network requires methods to locate minima of the landscape and connect them via transition states, defined as index one saddles (Murrell & Laidler 1968). We perform global optimisation using the basin-hopping method, which uses Metropolis criterion to avoid getting stuck in local minima (Li & Scheraga 1987 Wales & Doye 1997). For local minimisation we use a modified quasi- muasi-Newton L-BFGS (Limited -mmmory Broyden Fletcher Goldfarrb Goldfarb-Shamoo) algorithm (Nocedal 1980). After local basin of attraction (local minimum) is found via L-BFGS optimisation, "basin-hopping" jump is performed to some other point in the LFL. This jump always accepted if the energy, here loss value, is lower than the current minimum. If the loss value of the new point is higher, it is accepted with probability
AE TpT
x exp
where A is the difference loss value between the current minimum and the new point, ky the Boltzmann constant and T a fictitious temperature. Intuitively, if the energy difference between the old and new points is large, the move is less likely to accepted. A minimum is characteristed as such if RMS norm of gradient vector) convergence criterion at threshold of 10 is reached.
5.2
HYPERPARAMETER ABLATION STUDY
5.2.
SIGMOID
The relevant hyperparameter the sigmoid function is in the exponent (see Equation 7). For larger 3, the function becomes more stepwise, i... a better approximation the true AUC, with O(z),
lim (2: 8-000
where again (=) denotes the sigmoid and ((2) the Heaviside step function. We interested in the impact of the hyperparameter 3 on the appAUC LFI. Tigure/Bjsoww the effect on the loss function landscape of increasing 3 (on log scale). Clearly, both the number of minima in the LFL and the AUC of the best minimum increase with 3. The AAUC plot in Figure showws difference between the approximate AUC value, i.e. the value of the loss function, and the true AUC. for all minima found at specific B. This value is inversely proportional to 3 before it plateaus off median AUC difference of around 0.04. This median absolute difference a relevant metric because it shows how closely the appAUC approximates the true AUC value. The rightmost plot in Figure shows a measure of the computational cost associated with optimising the appAUC loss function for different values of B. The number of optimisation steps for each iteration of the L-BFGS optimiser grows roughly linearly with 3.
shows the median absolute
AUC
# minima
AAUC
L-BFGS steps
100
0.30
200
1000
0.05
0.63
0-
0.00
10
100
1000
100
10000
6d8
1000
6101
19
B
Figure 6: AUC the best minimum, number of minima of the LFL, difference between appAUC loss value and true AUC, and number of steps per optimiser masinnhopping run for increasing values of hyperparameter B in the sigmoid function. The horizontal axes are plotted on a log-scale for visualisation purposes. The results are smoothed using 2nd-order Savitzky-Golay filter.
The difference between the true AUC and the approximated AUC inversely proportional to 3, as expected, because for larger 3, the AUC approximation becomes more stepwise and hence more
13

---

Under review as a conference paper at ICLR 2022
exact. We find that for 3 50, the median difference goes to around 0.04, which means it is reasonably good approximation of the true AUC. We also observe an increase in best AUC, as well as the number of minima of the LFL with increasing B. All of these observations suggest that larger values of /3 are better, but increased performance comes at an increased computational cost. The reason for this increase is that the numher of optimisation steps in the L.-BFGS routine increases for larger 3. Most likely, this result can be explained by the flatter appAUC function at large / around local minima. In flatter regions, convergence to a true minimum usually requires more optimisation steps because the function is more anharmonic. To conclude, we suggest value for 3 of around 40. For higher values, the AUC does not improve significantly, while the computational cost, measured by the number of optimisation steps, increases linearly. Lastly, the distance between appAUC and CE minima for the number of optimisation steps, as opposed Euclidean distance, produces a larger tail in the plot due to the nature of the L--BGS optimiser. The step size is determined by the L--FGS formulation, and for a small step size there will be substantially more steps for the same Euclidean distance. We expect the number of steps to grow when the LFL is more locally anharmonic.
5.2.2 EXPONENTIAL
The same analysis is done with the exponential surrogate function, {(z) exp(-32). in Figure 7 Similar conclusions above are drawn for L-BFGS optimiser steps. However, the number of minima found and the best AUC are found to decrease with increasing B. Larger values of / here cause the exponential function more closely resemble a step for > 0, but the function doesn' 't then plateau for < 0 as nicely, which is perhaps the cause of the worse AUCs. It does not make sense to calculate AUC here, as exp( -32) does not truly attempt to "approximate" the actual step-function AUC. From this conclude that is the best parameter to use for the exponential function.
AUC
# minimu
L-BFGS stops
140
0.75
1500
100
8
9
8
Figure 7: AUC of the best minimum, number of minima of the LFL and number of steps per optimiser basin-hopping run for increasing values of hyperparameter 3 in the Exponential function.
5.3 EQUALITY IN 5
The AUC is defined by the following integral:
AUC(W;X) / T(W;X, P) d),;
(10)
Substituting in T and F (equations) 2 and 3 gives
AUC(W;X) XX: 1.' En 1x 5(c (W: P)
5(c" - ()))(pi(WW))) (W:x P) (2r(1- 1 1- -1)
Exd(ct-1
(11)
We may take outside of the integral any factors that do not depend on the parameter P. We note also that the sums En 6(cd -1) and Ex 1 5(c"" 1) are simply Np (the number of positive cases) and NN (the number of negative cases), respectively, and that the sums in the numerator, reduce to taking the sum over all positive points, with p. and all negative points, with d' n
AUC(W; X: NpNN ΣΣ
@ (W: P) dO(pi (W::)) - P).
(12)
Because the parameter P is continuous, can write
(0pp((W P) - 5(p1 ((p((W;*") P) dP.
(13)
14

---

Under review as a conference paper at ICLR 2022
The limits of the integral must be changed: when P 1, 0 and when P 0, 1, so
So J :
AUC(W:X) Np ΣΣ @p (W: - (W:x(W: dP. (14)
Finally, we integrate over this O-function, using the property J f(x)8(x - 2o) dr = f(vo), to obtain equation
AUC(W; X) Np ΣΣ Opi (W; PI (W;x)))
(15)
Note that the minus sign has disappeared, because the integral limits were flipped.
5.4 DERIVATIVES
Equation (6) can now be differentiated and treated as a function that can be optimised, just like
equation (I). For completeness and comparison, the first and second derivatives of both equations [ and are given. For compactness, the arguments each function are omitted, but superscript is
given if the function depends specific data point:
Ped (W:x*) + Patt
- př) - oP..
(16)
Finally, the derivatives given in component form (e.g. aCE/oW is vector of length Nw. the number weights), with labels μ,, v... denoting the relevant parts of the weight vector, W.
The loss function:
N CE
In(p
AW2
(17)
N
d=1
aCE
N
Opt
+ 2A
(18)
Ouc
N
Jul
d=1
O
1
Ppd
Op Op ( Ow ow,
2A8
(19)
Pd
Ow,dw,
The approximate AUC function (note L2 regularisation term has been added, to combat overfitting):
A =
ΣΣ
P.u + 2
(20)
Np.N
P n
04
1
ΣΣ
(1-0(1-0"))
Opp Sww,
Öpf
(21)
owp
NpNN
oww
Opp Ow,
Opī
Opp
0pi
ΣΣ NpNN
B 82OD. oP. 20
Dw, Ow
Ow,
Ow,
P
p2p
82p
+ B.P. (1 g...)
+ 2A6 pM
(22)
Dw Dww
Finally, for both the loss function and approximate AUC function, the derivatives of p must be evaluated (f is placeholder label cover both cases, cd 1). Here separate the four different groups of weights. These groups are: input-hidden weights (denoted as w(2)), hidden-output (w("), bias to hidden nodes (h) and bias to output nodes (o.
First derivatives:
15

---

Under review as a conference paper at ICLR 2022
Opp
Opp
14
(23)
(25)
Buobo
Duob
Opg Tuhh
ppp.
0pp
Opp
(5/))2
(24)
(26)
ss awobo
pwp
awhb
where
Nin
Nin
1y = tanh
w +
W
uk 2k
(2) zd
5 = sech
bh +
w
uk
(27)
k=
and
1/ 8zj Pj
(28)
Second derivatives:
SPp
p
ppp
dw 1 Swub
p
A d
Ep
A
P/ 20D
(29)
Seo Owbo
(1)
Oe p
dwobo
(33)
C
(1) Pv
Pyp
Ppp
(5/)) Σ
(34)
Bwith in
Ow (1) aw
owboowbe
(30)
p%
p
p
du (2) Owoo P
x
Owpb Ow
(35)
(od)2
(1) (1)
w ji
Bubb
P'p
p'p
1==1
d
(36)
Dup Dwph
(2)
Shiph Duph
2p td
w i/2
A
x (s (31)
044
ppp
x
(37)
Oreja
(2) 2m
(1)
a pd dw
Pp dwph Qubb
+ (spp)(s0)
Oww Oubb
(1)
p2p
ppp
(38)
(32)
Supu Owp
Sww Dwhh
(2)
16


Reviews:
[{"review_id": "xR-GLrB-ns", "reviewer": "Reviewer_Q2LC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n-- The authors have an intriguing idea of analyzing prediction models through their loss landscapes with two different losses (a standard CE loss and a proposed loss that is an approximation of the AUC). This is an interesting idea for a regime to draw conclusions.\n-- The experimental setup in 3.4 is interesting, as it suggests that there are learned models with a high CE value but a good AUC that the appAUC-trained models can find. However, this experiment is not studied any further. It would be interesting to explore those models specifically to understand how they can have high CE but have a good AUC, in terms of the effect on actual points. Are these unstable, spiky solutions? Are they overfitting, such that slightly different test sets would yield significantly worse results? This hinted at interesting conclusions but left them under-explored.\nWeaknesses:\n-- This paper assumes the AUC is a perfect measurement of the ultimate goal of training these models. While acknowledging that the traditional CE loss is only an approximation of the testing metric AUC, it ignores that the testing metric AUC may only be an approximation of the actual goal in our prediction.\n-- The datasets considered are too limited. Not only is the main dataset synthetic, but there is only one synthetic dataset and one real dataset. The results are not convincing as a result of the limited experimental setting. The real credit card data is given merely a couple sentences total, meaning the paper is almost exclusively analyzing a single synthetic dataset. This raises many questions, including how it would perform on data of other modalities, as well as whether the results would even replicate on additional synthetic or simple real datasets. The single synthetic dataset is also never shown, leaving the reader unclear about how these two models differ in their learned classifications in any way other than in terms of their AUC.\n-- As briefly mentioned in the related work, this optimization is still very similar to standard prediction losses. It throws extra terms in such that an incorrect prediction contributes to making the loss go up in multiple terms, but it is still moving in the same direction. The added terms make it a more complicated, less stable formula. While studying the slightly different landscape is interesting, ultimately it is not practically useful as a loss function.\n-- The loss compared to is a simple CE loss, but the data trained on is unbalanced. While the appAUC accounts for the class imbalance, the CE does not. There are very basic balancing techniques that could be used (e.g. proportional sampling) that would be used in practice for the CE model, and I wonder if those would eliminate the minor improvement in test AUC obtained from the appAUC loss. This is just one example of the much bigger problem that the types of models analyzed are far too minimal, with a single CE model compared to the proposed model.\n-- Many of the figures are unclear. Despite briefly defining disconnectivity graphs, their implications are not explained. In fact, many conclusions are just summarily stated from informally looking at these graphs (e.g. “the CE landscape is much more funnelled”). This statement is vague and qualitative and its implications are unclear.\n-- The correlation between the loss and the true AUC is not especially strong (section 3.3), which is concerning, considering this is the fundamental assumption made in designing the loss.\n", "summary_of_the_paper": "The authors propose to minimize approximations of the AUC during training directly, rather than the usual training objectives which are then evaluated with AUC after training.\n", "summary_of_the_review": "The model is not convincingly valuable, either as one for study or for practical use. Almost exclusively using a single synthetic dataset with a brief paragraph on a small real dataset is insufficient to draw any conclusions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ZzFFUojXjM1", "reviewer": "Reviewer_ifdg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n1. authors compared the minimization of CE versus the direct minimization of AUC and its surrogates. \n2. authors find a way to visualize the comparisons, which is quite intuitive (Figure 2)\n3. authors correlate these losses to demosntrate the relevance between these losses. \n4. it is not difficult to understand the effectiveness of surrogates of AUC and the tightness of CE to approximate AUC\n\nCons\n1. the experiment settings are not well addressed. AUC is mostly used for recommendation and advertising. Using such datasets might be more convincing.\n2. Some Belkin's work show that there is not so many difference between least square and CE or hinger loss in over-parameterized learning regiem. I am wondering the whether the discussion on the AUC could be included as a special case of their works?\n\n[1] Classification vs regression in overparameterized regimes: Does the loss function matter?\n[2] Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification", "summary_of_the_paper": "Authors did empirical studies to visualize and investigate the landscape of AUC (area under curve) loss. AUC is cirtical for a wide range of applications, such as recommendation and online advertising. Thus it is important.", "summary_of_the_review": "It is a solid work, though the connections to [1][2] are not clear.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NLzUrGrByTO", "reviewer": "Reviewer_hg7P", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well presented and is easy to follow. My main concern about this paper is its contribution. Frankly speaking, from my view, the contribution made in this paper is quite limited, especially given that the approximate AUC is not proposed by the authors. It is true that the authors did observe something from numerical experiments. However, I'm afraid that these observations cannot be articulated as real contributions. In particular, the authors stated that they \"provide a theoretical foundation to explain\" the observations they made, which, in my view, is overclaimed. ", "summary_of_the_paper": "The paper studied the AUC loss by investigating its approximates and by comparing it with the cross-entropy loss. The landscapes of the optimization based on approximate AUC and based on the cross-entropy loss are also studied. Numerical experiments are conducted.  ", "summary_of_the_review": "See above. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "9MucHfLbfsA", "reviewer": "Reviewer_qWyh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Writing: In general, this paper is well-written and easy to follow.\n\nNovelty: The significance and novelty of this paper is rather limited for several reasons.\n- The approximate surrogate of AUC examined in this paper has actually been proposed in previous research. Given that this paper only gives some empirical validations on an existing method without many innovations in algorithm design or theoretical analysis, I am not sure this work is significant enough to match the ICLR’s standard. \n- AppAUC was really an old metric. As far as I am concerned, there are some recently proposed surrogate functions for AUC maximization (see the Relevance part). These methods are not considered in this paper, which largely affects the significance of this work.\n\nRelevance: This paper missed several literatures on AUC maximization which are closely related with this work. \n- As commented above, besides appAUC, there are actually many other literatures investigating deep AUC maximization [1,2]. Some of them have also proposed new surrogate function of AUC. These surrogates should be discussed and compared with appAUC.\n- As a very relevant research field, online AUC maximization also aims to directly optimize the AUC metric through some computationally efficient surrogate functions for AUC [3]. I would strongly recommend the author to consider this field in the Related Work section.\n\n[1] Large-scale Robust Deep AUC Maximization: A New Surrogate Loss and Empirical Studies on Medical Image Classification.\n[2] Stochastic auc maximization with deep neural networks.\n[3] Stochastic online AUC maximization.", "summary_of_the_paper": "This paper studies AUC maximization in classification tasks, which aims to directly optimize the AUC metric instead of minimizing traditional classification losses such as the cross-entropy loss. The main contribution of this paper is an empirical study of an existing AUC maximization method termed approximate AUC. Specifically, this paper compares the landscape of the approximate AUC surrogate loss with that of the cross-entropy loss. The empirical results gives an explanation on why AUC maximization can achieve higher test AUC compared to the conventional method that directly minimizes the cross-entropy loss. ", "summary_of_the_review": "Overall, the novelty of this paper is rather limited and the relevance with previous work is not clear enough, hence I would vote for rejection. I would encourage the authors to consider more recent research on deep AUC maximization, and possibly elaborate a more systematic empirical study on the new surrogate AUC methods. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "3tByD9-5-tm", "reviewer": "Reviewer_yYns", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Some approximated AUC loss functions show better performance than standard learning via cross-entropy. However, most cases are about empirical results.\nGetting theoretical evidence about the advantages of those loss functions is a promising idea.\n\nHowever, the evaluation part of the paper is not satisfied standard requirements. First, the paper only considers a simple architecture of a neural network, using one linear layer. It could be irrelevant because the existing models have a much more difficult architecture. Second, just two datasets (synthetic and real ones) are used. Further conclusions could be non-reliable.\n\nAlso, it is mentioned that hinge loss is not AUC-consistent, but it is used in the SOTA AUC-M loss that is top-1 in the CheXpert competition.", "summary_of_the_paper": "The paper compares using the cross-entropy and approximated AUC loss functions in neural network training with AUC evaluation. The authors analyze loss function landscapes via solution space and minima properties.", "summary_of_the_review": "The authors suggest a useful way to analyze loss function selection, but experiments are not persuasive.\nI recommend expanding the evaluation part in terms of the number of datasets and real deep learning models with a sufficient number of layers.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper analyses the loss landscape induced by AUC loss. Reviewers found critical issues with the paper, and the Authors have not provided feedback. As such I have to recommend rejecting the paper. I thank the Authors for submitting the paper to the ICLR conference. I hope the reviews will be helpful in improving the paper.
Decision: Reject