Year: 2021

Abstract:
One of the most common metrics to evaluate neural network classifiers is the area under the receiver operating characteristic curve (AUC). However, optimisation of the AUC as the loss function during network training is not a standard procedure. Here we compare minimising the cross-entropy (CE) loss and optimising the AUC directly. In particular, we analyse the loss function landscape (LFL) of approximate AUC (appAUC) loss functions to discover the organisation of this solution space. We discuss various surrogates for AUC approximation and show their differences. We find that the characteristics of the appAUC landscape are significantly different from the CE landscape. The approximate AUC loss function improves testing AUC, and the appAUC landscape has substantially more minima, but these minima are less robust, with larger average Hessian eigenvalues. We provide a theoretical foundation to explain these results. To generalise our results, we lastly provide an overview of how the LFL can help to guide loss function analysis and selection.

heading: 1 INTRODUCTION
text: The area under the curve of the receiver operating characteristic curve (AUC) is a commonly used method to evaluate the accuracy and reliability of a neural network classifier. However, for mathematical reasons, the AUC cannot be used as as the loss function to be minimised during neural network training (Menon & Elkan, 2011). Instead, other functions such as cross-entropy are commonly employed. The stepwise nature of the AUC function is the reason that it is nondifferentiable and hence cannot simply be optimised. However, the AUC can be approximated using surrogate losses such as the sigmoid function. This approach can lead to a formulation equivalent to the soft-AUC described by Calders & Jaroszewicz (2007), which is referred to here as appAUC. We find it intuitive to optimise a function that it as close as possible to the one used to evaluate the model. Our main contributions in this paper are therefore:
• To understand the use of different approximation to the AUC as the loss function employed in training of a neural network • To explore the organisation of the appAUC landscape and compare it to a ‘standard’ cross-
entropy landscape • A theoretical foundation of the differences between appAUC and cross-entropy landscapes • To outline how a loss function landscape analysis can be useful for loss function selection
To better understand the advantages and disadvantages of the approximated AUC loss function, we study the functional space, commonly referred to as LFL, using tools from the theoretical study of energy landscapes in molecular and condensed matter systems (Wales, 2003). The usefulness of the energy landscape approach has previously been demonstrated in the context of neural network LFLs (Ballard et al., 2017). We will employ methods from this approach to gain insights about geometric features of the LFL, including the number of minima, their curvatures, and their connectivity. By repeatedly surveying large parts of the LFL, we are, with high probability, able to find the true global minimum. Additionally, due to a Metropolis criterion in the global optimisation approach described below, we do not get stuck in a local minimum, and explore the full LFL, hence learning more about the functional surface. Instead of a single minimum, we aim to find a large number of minima that,
together with transition states, provide a faithful coarse-grained representation of the loss function landscape. We believe that this approach will yield valuable insights into the use of appAUC as a loss function in neural networks.
Various interesting questions arise from the use of an appAUC loss function. Besides a comparison of properties between landscapes, and the effects of hyperparameter changes, we are especially interested in the differences between appAUC and CE landscapes. Both loss functions, for the same neural network architecture, address the learning problem of finding a mapping f from input data to class label. Does this common foundation imply that minima of the CE loss function are also minima of the appAUC function, or are they at least very similar to each other? We will show below that this condition does not hold, and explain why. Furthermore, to the best of our knowledge, there has been no previous research into the functional properties of AUC surrogates. We believe that quantifying inherent, geometric properties of loss functions will provide a more fundamental understanding of the applicability of particular loss functions to distinct machine learning problems.

heading: 1.1 RELATED WORK
text: This contribution lies at the intersection of three different areas of research: general understanding and study of loss functions, the appAUC loss function in particular, and the study of loss function landscapes to understand neural networks. Optimising the AUC as a loss function has been considered before (Cortes & Mohri, 2004). It has been shown that testing AUC is improved when a loss function is chosen that is closer to the true AUC function (Yan et al., 2003). Nonetheless, optimising loss functions that approximate the AUC is rarely considered. One reason for this situation may be that the computational complexity is formally O(N2), or specifically in the binary classification case employed here, O(NPNN ) where NP are the positive data points and NN the negative ones. Other reasons include the non-convexity and perceived complexity of the appAUC landscape, and importantly the fact that it has zero derivatives almost everywhere (Ghanbari & Scheinberg, 2018). The loss function is one of the most critical choices in the design of a neural network, because it is the element that underlies the entire learning procedure. At first glance, it may appear that all loss functions solve the same problem for a given neural network, which led Rosasco et al. (2004) to question the practical differences between alternative loss functions. Yet, it is widely accepted that different loss functions allow us to optimise for specific properties (Janocha & Czarnecki, 2017). By studying the appAUC LFL, we can quantitatively address these important questions.
The topology of the loss function has been a topic of interest for over 20 years (Hochreiter & Schmidhuber, 1997). Understanding the loss function for a given neural network can help to establish the fundamental interpretation of the ‘black-box’, as such networks are often described (Li et al., 2017). However, a key problem with studying the LFL is the large associated computational cost. Unlike the standard machine learning approach, where only a single minimum is located, the LFL approach attempts to sample a representative set of minima, which may often exceed 10,000 solutions (Ballard et al., 2017). Computing the true number of minima for a loss function would require enhanced sampling techniques (Wales, 2013; Martiniani et al., 2016).
Recently, others have looked into the LFL of overparameterised neural networks (Cooper, 2018) and shown that various properties can be exploited to improve accuracy and, importantly, robustness of neural networks (Baldassi et al., 2020; Chaudhari et al., 2019). Insights from the LFL have also been used to understand initialisation methods and their relative success (Fort & Scherlis, 2019). However, to the best of our knowledge, there has been no research so far into interpreting individual loss functions from the underlying LFL. In this work, we aim to address this knowledge gap, and show that studying the LFL can provide important insights into our understanding and design of neural networks and the associated loss functions.

heading: 2 METHODS

heading: 2.1 NEURAL NETWORK
text: For this initial survey, we consider neural networks with a single hidden layer. We denote the set of data points as D = (X, c), containing N := |D| elements. For a given classification problem with C classes, a data point d ∈ D has input features xd, and a known output class, cd. We use a nonlinear tanh activation function and convert output values yi, i ∈ C into softmax probabilities by
pi(W;x d) = exp[yi(W;x d)]/ ∑C j=1 exp[yj(W;x
d)], where W denotes the weight vector containing all weights of the neural network. As a reference to compare with our appAUC loss function, we use a cross-entropy (CE) loss function, defined as:
CE(W;X) = − 1 N N∑ d=1 ln(pcd(W;x d)) + λW2. (1)
The λW2 in equation 1 represents an L2 regularisation term, which eliminates zero Hessian Eigenvalues and counteracts overfitting. We fix λ = 10−5 for a fair comparison between loss functions.

heading: 2.2 AREA UNDER THE CURVE
text: To evaluate a model such as the one described above, it is standard practice to consider the receiver operating characteristic (ROC) and calculate the area under the curve (AUC) (Hastie et al., 2009). For a given classifier, the ROC is a plot of the true positive ratio, T (P ), against the false positive ratio, F (P ). These quantities are defined by
T (W;X, P ) =
∑N d=1 δ(c
d − 1)Θ(p1(W;xd)− P )∑N d=1 δ(c d − 1) , (2)
F (W;X, P ) = ∑N d=1(1− δ(cd − 1))Θ(p1(W;xd)− P )∑N
d=1 1− δ(cd − 1) , (3)
where δ(cd − 1) is the Dirac delta function, and Θ(p1 − P ) is the Heaviside step function, defined as δ(cd − 1) = {
1 if cd = 1, 0 if cd 6= 1, Θ(p1 − P ) = { 1 if p1 ≥ P, 0 if p1 < P,
(4)
and P is a parameter, which acts as a cutoff probability for the neural network to predict that a given data point belongs to class 1 (p1). Note that the choice of class 1 as a positive reading is arbitrary, and for a multi-class system, any class may be chosen.
The ROC curve is defined by T and F as P varies from 0 to 1. For a perfect classifier, the ROC would simply be a horizontal line at T = 1 (i.e. ∀ P 6= 0, T (P ) = 1), and the area under the curve would then be 1. The AUC is therefore a measure of how close the model is to a perfect classifier. Formally, the AUC as a function of network weights W, parameterised by the data X, is given by
AUC = ∫ 1 0 T (W;X, P ) dF (W;X, P ) = 1 NPNN ∑ p ∑ n Θ(p1(W;x p)− p1(W;xn)) (5)
where p labels positive data points (class 1), and n labels negative data points (not in class 1).

heading: 2.3 APPROXIMATED AUC: APPAUC LOSS FUNCTION
text: Most optimisation routines benefit from analytical derivatives of the function to be optimised. The function defined in equation 1 has smooth analytical derivatives, but equation 5 does not, because of the discontinuous step function. However, in a similar way to other approaches (e.g. (Calders & Jaroszewicz, 2007)), one can replace the discontinuous Θ function with an approximate, smooth, analytical surrogate function, which can then be differentiated and optimised. We write
AUC(W;X) ≈ A(W;X) ≡ 1 NPNN ∑ p ∑ n
1
1 + exp(−β(p1(W;xp)− p1(W;xn))) , (6)
where the Heaviside step function has been replaced with a smooth sigmoid function:
Θ(z)→ σ(z) ≡ 1 1 + exp(−βz) , (7)
with parameter β that is discussed in detail in the Appendix. An important consideration for surrogate AUC loss functions is that they do not change the optimal solution when replacing the step function. Such surrogates are referred to as AUC-consistent (Agarwal, 2014). Charoenphakdee et al. (2019) show that sigmoid is an AUC-consistent surrogate. The appAUC loss function in equation 6 is now differentiable and can hence be used for optimisation, where we minimise the negative form of equation 6 such that a minimum (i.e. lower loss) is a good solution of higher AUC. For reference, we have included the analytical first and second derivatives for Equation 6 in the Appendix.

heading: 2.3.1 OTHER SURROGATE LOSS FUNCTIONS
text: There exist many possible ways to approximate the AUC. Besides the sigmoid (Eq.7), we also show results for the popular (Gao & Zhou, 2015) hinge `(z) = max(0, 1− z), exponential `(z) = exp(−z), quadratic `(z) = (1 − z)2 and tanh surrogates (Fig. 1), where z denotes the difference in probability between true and false class i.e. PT − PN . Note that for the sigmoid in Figure 1, we use β = 40 (Appendix). It is important to note that all but the hinge loss are AUC-consistent (Gao & Zhou, 2015; Charoenphakdee et al., 2019). AUC-consistency implies that the optimal solution remains unchanged with the surrogate loss replacing the true AUC. These different losses can all
be used as surrogates because they include a pairwise comparison of positive and negative case probabilities as for the AUC.

heading: 2.4 OPTIMISATION ROUTINES
text: Surveying the LFL for a neural network requires methods to locate minima of the landscape and connect them via transition states, defined as index one saddles (Murrell & Laidler, 1968). We perform global optimisation using the basin-hopping method, which uses a Metropolis criterion to avoid getting stuck in local minima (Li & Scheraga, 1987; Wales & Doye, 1997). Further details regarding landscape exploration using basin-hopping are included in the Appendix. We use a modified quasi-Newton L-BFGS minimiser (Nocedal, 1980). Connectivity between minima is defined by transition states using a doubly-nudged (Trygubenko & Wales, 2004a;b) elastic band (Henkelman & Jónsson, 2000; Henkelman et al., 2000) approach and hybrid-eigenvector following (Munro & Wales, 1999; Zeng et al., 2014). These methods are implemented in the GMIN (Wales, a), OPTIM (Wales, b) and PATHSAMPLE (Wales, c) programs, which are available for use under the GNU GPL.

heading: 2.5 DISCONNECTIVITY GRAPHS
text: To visualise the high-dimensional loss function landscape, we employ disconnectivity graphs (Becker & Karplus, 1997; Wales et al., 1998), which provide a characteristic representation of the organisation for a high-dimensional surface. The vertical axis corresponds to increasing loss value, highlighting the barrier heights between local minima, and the horizontal arrangement is chosen to avoid crossings between the branches. In molecular science the vertical axis corresponds to potential or free energy, so the loss value plays the role of the energy in the LFL. Any node at which two or more branches split up can be understood as the minimum energy (loss) required for the system to move between the sets of minima associated with the respective branches. Such points correspond to the energies of transition states, discretised at a regular set of threshold values. Each leaf (tip) of the graph corresponds to a minimum of the LFL; the lower the leaf is, the lower its loss value. The lowest-lying leaf is the global minimum. Disconnectivity graphs provide a concise and intuitive way to visualise complex topologies, and preserve a faithful representation of the barriers on the landscape.

heading: 2.6 DATASET
text: The LFL is not only a function of the weights, but also of the data, as shown in equation 6. We will present our analysis of the AUC landscape mostly using the synthetic spiral dataset, introduced by Lang & Witbrock (1988). Spiral data remains one of the most popular synthetic datasets due to its high degree of non-linear separability. We add a small uniform noise term to increase the complexity of the problem and with an 80-20 train-test split. We acknowledge the limitations of applicability of a synthetic dataset and therefore supplement our results using a real-world dataset of fraudulent credit card transactions (Le Borgne & Bontempi, 2004). The dataset is highly imbalanced with only 10% of all datapoints classified as fraudulent transaction.

heading: 3 RESULTS
text: In this section, we highlight three distinct points of interest. Firstly, we visualise and outline key differences between the AUC and CE landscapes. Secondly, we present an extended quantitative and geometric analysis of different AUC surrogates. Thirdly, we provide a comparison between minima of the AUC and CE landscape in terms of their classification properties.

heading: 3.1 LFL PROPERTIES
text: The loss function landscapes for AUC-approximated and CE loss functions look substantially different (Fig. 2). We performed landscape exploration using PATHSAMPLE until no new minima were found. For computational reasons, we only compare sigmoid appAUC with CE, and show below that for shorter timescales, sigmoid looks similar to the other surrogates. In particular, we find that the CE landscape is much more convex or funnelled than the appAUC landscape, with larger uphill barriers. In molecular science, this structure is associated with self-organising systems where relaxation to the global minimum is relatively efficient (Wales, 2003).
In Table 1 we report some summary statistics to characterise the two different LFLs numerically. The last column reports the mean of the log-product of positive Hessian Eigenvalues (LPPHE) for all minima of the landscape. This measure of basin geometry provides insight into the local curvature
and hence the ‘flatness’ of a minimum (Verpoort et al., 2020) and the volume of the corresponding basin of attraction (Mezey, 1987; Wales, 2003). The smaller the LPPHE, the flatter the local basin. Table 1 shows that the appAUC landscape has substantially more minima and transition states than the CE landscape. Further, the mean LPPHE of all appAUC minima is larger than for the CE minima.

heading: 3.2 AUC SURROGATES
text: Table 2 shows a comparison of the different appAUC surrogates (Section 2.3.1) after a week of landscape exploration. We find that these functions behave similarly, with comparable AUCs found and more minima than the CE landscape. The mean LPPHEs vary, the higher values belonging to the two functions that are true approximations to the stepwise AUC, namely sigmoid and tanh. They are all nonetheless larger or equal to the CE LFL in Table 1. The disconnectivity graphs for these alternative appAUC functions are shown in Figure 3. They are all similar to the appAUC landscape in Figure 2 in the sense that they appear very wide (referred to as “glassy” in molecular sciences) and much less single-funnelled than the CE LFL.

heading: 3.3 LOSS-AUC CORRELATION
text: Below, we show the correlation between loss value (energy) and AUC for test data for the different surrogate losses (Figure 4). In all five cases, there is a significant negative correlation. Tanh has the most negative Pearson correlation coefficient of -0.83, followed by sigmoid and hinge.
3.4 APPAUC MINIMA IN THE CE LFL
Another point of interest is to understand the relationship between CE loss value and testing AUC for different LFLs. In Figure 5 we use sigmoid to represent appAUC. We see that the negative correlation between CE loss and AUC is much stronger for minima of the CE than for the appAUC landscape. Hence, for most minima of the appAUC landscape, the CE loss values are relatively high, even though some of them may in fact have a higher AUC that the CE minima. This result raises the question of how good a diagnostic the CE loss is for the testing AUC. The insights into the position of appAUC minima within the CE LFL can be combined with the previous results on appAUC minima and the minimum they map to in the CE LFL. Above, we have identified that there exist many appAUC minima that are not minima of the CE LFL and thus have a higher CE loss value, but have comparable testing AUC. In general, while the best appAUC minima have reasonably low
CE loss, the mean loss of 1.2 is nearly twice as high as for CE minima. The median of 0.8 is slightly lower, indicating a few very high loss minima, which lie in the lower right hand corner of Figure 5.

heading: 3.5 CREDIT CARD DATASET
text: We perform the same analysis on the imbalanced, real-world credit card dataset. Optimising AUC on such datasets is expected to particularly useful, because the AUC as a measure is invariant to class imbalance. The appAUC landscape has more, better minima, and larger LPPHEs (Table 3). As above, the CE landscape is single funnelled while appAUC landscape is more glassy, similar to figure 2.

heading: 4 DISCUSSION
text: We observe that the appAUC and CE LFLs have rather different characteristics, emphasising the point that different loss functions are truly different. In this section, we discuss and explain these observations, both empirically and theoretically. We explain why an analysis of the LFL can offer unique insights into the understanding of loss functions. Different characteristics of such a function may be important, and studying the LFL can help to choose an appropriate loss function. We also provide a comparison of the different AUC surrogates.

heading: 4.1 COMPARISON OF LOSS FUNCTION LANDSCAPES
text: As the disconnectivity graphs in Figure 2 show, the CE landscape is more funnelled, and there are fewer minima. However, not only does the number of minima vary, the LPPHE also indicates that the minima appear rather different. Flatter minima (smaller LPPHE) are sometimes considered advantageous, because they may be more robust to perturbations in the weights (Hochreiter & Schmidhuber, 1997), and therefore potentially also to noise in the training data. The increased robustness is explained by a lower deviation from the loss value of the global minimum for small deviations in training data or model hyperparameters in flatter landscapes where the gradients are small. This feature may constitute a geometrically-inspired argument against using the appAUC landscape if appAUC minima are less robust. The data suggest that on average the CE minima are relatively flat, which may translate to increased robustness to perturbations of the weights.
The location of appAUC minima within the CE LFL raises several new questions. Figure 5 illustrates the weakly negative correlation between CE loss and testing AUC for appAUC minima, much weaker than for CE LFL minima. This result initially seems intuitive, as we are looking at the CE LFL. The fact that the correlation between testing AUC and loss is more strongly negative for points of the CE loss function means that the CE loss works well for the minima in its own LFL. However, there also exist points in the CE LFL that are not minima (but minima of the appAUC) with higher loss and also higher AUC. Hence, under the assumption that appAUC is the more ‘accurate’ loss function, this analysis highlights some weaknesses of the CE LFL, namely not identifying as many better, correct solutions. A mathematical foundation of the limitations of CE as opposed to appAUC is given in the next section.

heading: 4.2 THEORETICAL INTERPRETATION
text: To explain our empirical results, we can provide a theoretical basis for some of the observations made. Notably, the substantially higher number of minima of the appAUC LFL is explained by mathematical properties of the solution space for CE in relation to appAUC loss generally. In fact, the solutions to CE loss are a subset of the solutions to the appAUC loss (Menon & Williamson, 2016). The optimal solution of appAUC is any function that has a strictly monotonic relationship with P (y = 1|x) (Gao & Zhou, 2015). On the other hand, the optimal solution of CE loss is P (y = 1|x). Thus, a space of solutions for appAUC is infinitely larger than CE loss and therefore it should have more minima. The
CE loss with Softmax belongs to the class of ‘strictly proper composite’ losses (Reid & Williamson, 2010) which estimates P (y = 1|x). In statistics, this is also referred to as ‘proper scoring rule’ (Buja et al., 2005). Obviously, P (y = 1|x) has a strictly monotonic relationship with itself and hence must be optimal w.r.t. AUC, however this relationship must not hold the other way, optimising AUC must not necessarily be optimal w.r.t. P (y = 1|x). Furthermore, the same line of argument also allows us to explain why appAUC minima are not necessarily close to CE minima in Euclidean space. Minimising appAUC does not give any additional incentive to find solutions of the kind P (y = 1|x) compared to any other solutions which may have a strictly monotonic relationship with P (y = 1|x).

heading: 4.3 APPAUC VS CE LOSS FUNCTION
text: The appAUC landscape we have considered has approximately 8 times as many minima as the corresponding CE landscape. The global minimum for appAUC achieves better testing AUC, and the mean AUC is also higher, yet there are also a few especially poor minima in the AUC landscape that are not found for the CE case. For practical applications the existence of solutions that are poor classifiers is probably not a concern, because finding low-lying minima is relatively straightforward. However, computation of the appAUC as done here is of orderO(N2), whereN is the number of data points, which is much more intensive that the O(N) requirement associated with CE. Additionally, minima of the CE LFL seem to be more robust. In summary, exploiting the appAUC directly is more attractive if computational cost is not a problem, especially if the loss function landscape can be studied extensively. For machine learning applications involving large deep networks, where only a single minimum is identified, and where computational cost is a major concern, the CE landscape may be the best choice. This result strengthens the case for using CE as a ‘standard‘ loss function, while also highlighting its limitations, and illustrating how using different loss functions can be beneficial.

heading: 4.4 AUC SURROGATES
text: We observe that the LFLs for all AUC surrogates look roughly similar, and distinctly different to the CE LFL, providing a strong argument for a fundamental geometric difference between these loss functions that has not been shown before. We also observe that the hinge loss has substantially more minima than all other surrogates, yet the testing AUCs are much worse. This may be explained by the fact that hinge is not AUC consistent which means that the hinge functional space does not contain the AUC-optimal solution for this problem. Quad, tanh and sigmoid have broadly similar characteristics with small deviations in the LPPHE while the exponential does slightly worse in terms of AUC but has very flat minima, similar to the CE landscape. Hence, using an exponential AUC surrogate may provide more robust solutions than other surrogates. If the main objective however is to maximise testing AUC, tanh seems to be the best choice, as it not only has the highest AUC, but also the strongest inverse correlation between energy and AUC. Both tanh and sigmoid are directly approximating the stepwise AUC which explains the strongly negative correlations, while both exponential and squared exponential function behave broadly similar as expected.

heading: 4.5 CONCLUSIONS
text: Loss functions are essential to guide the learning process in neural networks. We have provided a detailed analysis of the approximate AUC landscape based on global exploration of tractable but realistic examples. In this contribution, we have proposed a new method to understand and select a loss function, based on geometric properties of the LFL. We have shown that LFLs for appAUC and the standard CE LFL are qualitatively and quantitatively different. We have also provided a detailed comparison of relevant LFL features between AUC surrogates and shown that the AUC-inconsistent hinge has much lower AUC than the other surrogates (but more minima) and that the exp surrogate has wide, robust minima, similar to the CE LFL. It is outlined that optimising the AUC improves testing AUC, but CE minima have on average smaller LPPHEs, which may make them more robust. In general, we observe a tradeoff between robustness and testing AUC for all the loss functions. Furthermore, it must be noted that the greater computational cost of optimising the AUC directly is likely to be the main drawback of this approach. A quantitative analysis of geometric features of the LFL such as by-minimum AUC, energy-testing AUC correlation, number of minima (landscape convexity) and LPPHE i.e. catchment basin volumes which is connected to minima robustness allows insights into the strengths and weaknesses of loss functions. Hence, this analysis provides a valuable tool to guide loss function selection for machine learning applications.

heading: 5 APPENDIX

heading: 5.1 BASIN-HOPPING OPTIMISATION
text: Surveying the LFL for a neural network requires methods to locate minima of the landscape and connect them via transition states, defined as index one saddles (Murrell & Laidler, 1968). We perform global optimisation using the basin-hopping method, which uses a Metropolis criterion to avoid getting stuck in local minima (Li & Scheraga, 1987; Wales & Doye, 1997). For local minimisation we use a modified quasi-Newton L-BFGS (Limited-memory Broyden–Fletcher–Goldfarb–Shanno) algorithm (Nocedal, 1980). After a local basin of attraction (local minimum) is found via L-BFGS optimisation, a ’basin-hopping’ jump is performed to some other point in the LFL. This jump is always accepted if the energy, here loss value, is lower than the current minimum. If the loss value of the new point is higher, it is accepted with probability
P ∝ exp ( −∆Ẽ kBT ) (8)
where ∆Ẽ is the difference in loss value between the current minimum and the new point, kB the Boltzmann constant and T a fictitious temperature. Intuitively, if the energy difference between the old and new points is large, the move is less likely to be accepted. A minimum is characteristed as such if a RMS norm of gradient vector) convergence criterion at a threshold of 10−10 is reached.

heading: 5.2 HYPERPARAMETER ABLATION STUDY

heading: 5.2.1 SIGMOID
text: The relevant hyperparameter in the sigmoid function is β in the exponent (see Equation 7). For larger β, the function becomes more stepwise, i.e. a better approximation to the true AUC, with
lim β→∞
σ(z;β)→ Θ(z), (9)
where again σ(z) denotes the sigmoid and Θ(z) the Heaviside step function. We are interested in the impact of the hyperparameter β on the appAUC LFL. Figure 6 shows the effect on the loss function landscape of increasing β (on a log scale). Clearly, both the number of minima in the LFL and the AUC of the best minimum increase with β. The ∆AUC plot in Figure 6 shows the median absolute difference between the approximate AUC value, i.e. the value of the loss function, and the true AUC, for all minima found at a specific β. This value is inversely proportional to β before it plateaus off at a median AUC difference of around 0.04. This median absolute difference is a relevant metric because it shows how closely the appAUC approximates the true AUC value. The rightmost plot in Figure 6 shows a measure of the computational cost associated with optimising the appAUC loss function for different values of β. The number of optimisation steps for each iteration of the L-BFGS optimiser grows roughly linearly with β.
The difference between the true AUC and the approximated AUC is inversely proportional to β, as expected, because for larger β, the AUC approximation becomes more stepwise and hence more
exact. We find that for β > 50, the median difference goes to around 0.04, which means it is a reasonably good approximation of the true AUC. We also observe an increase in best AUC, as well as the number of minima of the LFL with increasing β. All of these observations suggest that larger values of β are better, but increased performance comes at an increased computational cost. The reason for this increase is that the number of optimisation steps in the L-BFGS routine increases for larger β. Most likely, this result can be explained by the flatter appAUC function at large β around local minima. In flatter regions, convergence to a true minimum usually requires more optimisation steps because the function is more anharmonic. To conclude, we suggest a value for β of around 40. For higher values, the AUC does not improve significantly, while the computational cost, measured by the number of optimisation steps, increases linearly. Lastly, the distance between appAUC and CE minima for the number of optimisation steps, as opposed to Euclidean distance, produces a larger tail in the plot due to the nature of the L-BFGS optimiser. The step size is determined by the L-BFGS formulation, and for a small step size there will be substantially more steps for the same Euclidean distance. We expect the number of steps to grow when the LFL is more locally anharmonic.

heading: 5.2.2 EXPONENTIAL
text: The same analysis is done with the exponential surrogate function, `(z) = exp(−βz), in Figure 7. Similar conclusions to above are drawn for L-BFGS optimiser steps. However, the number of minima found and the best AUC are found to decrease with increasing β. Larger values of β here cause the exponential function to more closely resemble a step for z > 0, but the function doesn’t then plateau for z < 0 as nicely, which is perhaps the cause of the worse AUCs. It does not make sense to calculate ∆AUC here, as exp(−βz) does not truly attempt to “approximate” the actual step-function AUC. From this we conclude that β = 1 is the best parameter to use for the exponential function.

heading: 5.3 EQUALITY IN EQ.5
text: The AUC is defined by the following integral: AUC(W;X) = ∫ 1 0 T (W;X, P ) dF (W;X, P ). (10)
Substituting in T and F (equations 2 and 3) gives AUC(W;X) = ∫ 1 0 ∑ d δ(c d − 1)Θ(p1(W;xd)− P )∑ d δ(c d − 1) d (∑ d′(1− δ(cd ′ − 1))Θ(p1(W;xd ′ )− P )∑ d′ 1− δ(cd ′ − 1) ) . (11) We may take outside of the integral any factors that do not depend on the parameter P . We note also that the sums ∑ d δ(c d − 1) and ∑ d′ 1− δ(cd
′ − 1) are simply NP (the number of positive cases) and NN (the number of negative cases), respectively, and that the sums in the numerator, reduce to taking the sum over all positive points, with d = p, and all negative points, with d′ = n:
AUC(W;X) = 1
NPNN ∑ p ∑ n ∫ 1 0 Θ(p1(W;x p)− P ) dΘ(p1(W;xn)− P ). (12)
Because the parameter P is continuous, we can write
dΘ(p1(W;xn)− P ) = −δ(p1(W;xn)− P ) dP. (13)
The limits of the integral must be changed: when P = 1,Θ = 0 and when P = 0,Θ = 1, so∫ 1 0 → ∫ 0 1 or − ∫ 1 0 :
AUC(W;X) = 1
NPNN ∑ p ∑ n ∫ 1 0 Θ(p1(W;x p)− P ) δ(p1(W;xn)− P ) dP. (14)
Finally, we integrate over this δ-function, using the property ∫ f(x)δ(x− x0) dx = f(x0), to obtain equation 5:
AUC(W;X) = 1
NPNN ∑ p ∑ n Θ(p1(W;x p)− p1(W;xn)). (15)
Note that the minus sign has disappeared, because the integral limits were flipped.

heading: 5.4 DERIVATIVES
text: Equation (6) can now be differentiated and treated as a function that can be optimised, just like equation (1). For completeness and comparison, the first and second derivatives of both equations 1 and 6 are given. For compactness, the arguments of each function are omitted, but a superscript is given if the function depends on a specific data point:
pcd(W;x d)→ pdcd , σ(p p 1 − pn1)→ σp,n. (16)
Finally, the derivatives are given in component form (e.g. ∂CE/∂W is a vector of length Nw, the number of weights), with labels µ, ν... denoting the relevant parts of the weight vector, W.
The loss function:
CE = − 1 N N∑ d=1 ln(pdcd) + λW 2 (17)
∂CE ∂wµ = − 1 N N∑ d=1 1 pd cd ∂pdcd ∂wµ + 2λwµ (18)
∂2CE
∂wµ∂wν = − 1 N N∑ d=1
( 1
pd cd
∂2pdcd
∂wµ∂wν − 1
(pd cd
)2 ∂pdcd ∂wµ ∂pdcd ∂wν
) + 2λδµν (19)
The approximate AUC function (note an L2 regularisation term has been added, to combat overfitting):
A = 1
NPNN ∑ p ∑ n σp,n + λW2 (20)
∂A
∂wµ =
1
NPNN ∑ p ∑ n
[ βσp,n(1− σp,n) ( ∂pp1 ∂wµ − ∂p n 1 ∂wµ )] + 2λwµ (21)
∂2A
∂wµ∂wν =
1
NPNN ∑ p ∑ n
[ β2σp,n(1− σp,n)(1− 2σp,n) ( ∂pp1 ∂wµ − ∂p n 1 ∂wµ )( ∂pp1 ∂wν − ∂p n 1 ∂wν )
+ βσp,n(1− σp,n) (
∂2pp1 ∂wµ∂wν − ∂ 2pn1 ∂wµ∂wν
)] + 2λδµν (22)
Finally, for both the loss function and approximate AUC function, the derivatives of pd` must be evaluated (` is a placeholder label to cover both cases, cd or 1). Here we separate the four different groups of weights. These groups are: input-hidden weights (denoted as w(2)), hidden-output (w(1)), bias to hidden nodes (wbh) and bias to output nodes (wbo).
First derivatives:
∂pd` ∂wboµ = pd`∆ d `µ (23)
∂pd` ∂wbhµ = (sdµ) 2 C∑ i=1 w (1) iµ ∂pd` ∂wboi
(24)
∂pd`
∂w (1) µν = tdν ∂pd` ∂wboµ
(25)
∂pd`
∂w (2) µν = xdν ∂pd` ∂wbhµ
(26)
where
tdν ≡ tanh ( wbhν + Nin∑ k=1 w (2) νk x d k ) sdν ≡ sech ( wbhν + Nin∑ k=1 w (2) νk x d k ) (27)
and ∆dij ≡ δij − pdj (28)
Second derivatives:
∂2pd` ∂wboµ ∂w bo ρ = pd` (∆ d `ρ∆ d `µ − pdµ∆dµρ) (29)
∂2pd` ∂wbhµ ∂w bo ρ = (sdµ) 2 C∑ i=1 w (1) iµ ∂2pd` ∂wboi ∂w bo ρ
(30)
∂2pd` ∂wbhµ ∂w bh ρ =
[ (sdρ) 2 C∑
i,j=1
w (1) iµ w (1) jρ ∂2pd` ∂wboi ∂w bo j
− 2δµρtdµpd` C∑ i=1 w (1) iµ ∆ d `i ] × (sdµ)2 (31)
∂2pd`
∂w (1) µν ∂wbhρ
= tdν ∂2pd`
∂wbhρ ∂w bo µ
+ δνρp d ` (s d ρ) 2∆d`µ
(32)
∂2pd`
∂w (1) µν ∂wboρ
= tdν ∂2pd`
∂wboµ ∂w bo ρ
(33)
∂2pd`
∂w (1) µν ∂w (1) ρσ
= tdνt d σ ∂2pd` ∂wboµ ∂w bo ρ
(34)
∂2pd`
∂w (2) µν ∂wboρ
= xdν ∂2pd`
∂wbhµ ∂w bo ρ
(35)
∂2pd`
∂w (2) µν ∂wbhρ
= xdν ∂2pd`
∂wbhµ ∂w bh ρ
(36)
∂2pd`
∂w (2) µν ∂w (1) ρσ
= xdν ∂2pd`
∂w (1) ρσ ∂wbhµ
(37)
∂2pd`
∂w (2) µν ∂w (2) ρσ
= xdσ ∂2pd`
∂w (2) µν ∂wbhρ
(38)

Review summaries:
This paper studies AUC maximization in classification tasks, which aims to directly optimize the AUC metric instead of minimizing traditional classification losses such as the cross-entropy loss. The main contribution of this paper is an empirical study of an existing AUC maximization method termed approximate AUC. Specifically, this paper compares the landscape of the approximate AUC surrogate loss with that of the cross-entropy loss. The empirical results gives an explanation on why AUC maximization can achieve higher test AUC compared to the conventional method that directly minimizes the cross-entropy loss. 

The paper studied the AUC loss by investigating its approximates and by comparing it with the cross-entropy loss. The landscapes of the optimization based on approximate AUC and based on the cross-entropy loss are also studied. Numerical experiments are conducted.  

Authors did empirical studies to visualize and investigate the landscape of AUC (area under curve) loss. AUC is cirtical for a wide range of applications, such as recommendation and online advertising. Thus it is important.