Characterising the Area Under the Curve Loss Function Landscape

1 INTRODUCTION . The area under the curve of the receiver operating characteristic curve ( AUC ) is a commonly used method to evaluate the accuracy and reliability of a neural network classifier . However , for mathematical reasons , the AUC can not be used as as the loss function to be minimised during neural network training ( Menon & Elkan , 2011 ) . Instead , other functions such as cross-entropy are commonly employed . The stepwise nature of the AUC function is the reason that it is nondifferentiable and hence can not simply be optimised . However , the AUC can be approximated using surrogate losses such as the sigmoid function . This approach can lead to a formulation equivalent to the soft-AUC described by Calders & Jaroszewicz ( 2007 ) , which is referred to here as appAUC . We find it intuitive to optimise a function that it as close as possible to the one used to evaluate the model . Our main contributions in this paper are therefore : • To understand the use of different approximation to the AUC as the loss function employed in training of a neural network • To explore the organisation of the appAUC landscape and compare it to a ‘ standard ’ cross- entropy landscape • A theoretical foundation of the differences between appAUC and cross-entropy landscapes • To outline how a loss function landscape analysis can be useful for loss function selection To better understand the advantages and disadvantages of the approximated AUC loss function , we study the functional space , commonly referred to as LFL , using tools from the theoretical study of energy landscapes in molecular and condensed matter systems ( Wales , 2003 ) . The usefulness of the energy landscape approach has previously been demonstrated in the context of neural network LFLs ( Ballard et al. , 2017 ) . We will employ methods from this approach to gain insights about geometric features of the LFL , including the number of minima , their curvatures , and their connectivity . By repeatedly surveying large parts of the LFL , we are , with high probability , able to find the true global minimum . Additionally , due to a Metropolis criterion in the global optimisation approach described below , we do not get stuck in a local minimum , and explore the full LFL , hence learning more about the functional surface . Instead of a single minimum , we aim to find a large number of minima that , together with transition states , provide a faithful coarse-grained representation of the loss function landscape . We believe that this approach will yield valuable insights into the use of appAUC as a loss function in neural networks . Various interesting questions arise from the use of an appAUC loss function . Besides a comparison of properties between landscapes , and the effects of hyperparameter changes , we are especially interested in the differences between appAUC and CE landscapes . Both loss functions , for the same neural network architecture , address the learning problem of finding a mapping f from input data to class label . Does this common foundation imply that minima of the CE loss function are also minima of the appAUC function , or are they at least very similar to each other ? We will show below that this condition does not hold , and explain why . Furthermore , to the best of our knowledge , there has been no previous research into the functional properties of AUC surrogates . We believe that quantifying inherent , geometric properties of loss functions will provide a more fundamental understanding of the applicability of particular loss functions to distinct machine learning problems . 1.1 RELATED WORK . This contribution lies at the intersection of three different areas of research : general understanding and study of loss functions , the appAUC loss function in particular , and the study of loss function landscapes to understand neural networks . Optimising the AUC as a loss function has been considered before ( Cortes & Mohri , 2004 ) . It has been shown that testing AUC is improved when a loss function is chosen that is closer to the true AUC function ( Yan et al. , 2003 ) . Nonetheless , optimising loss functions that approximate the AUC is rarely considered . One reason for this situation may be that the computational complexity is formally O ( N2 ) , or specifically in the binary classification case employed here , O ( NPNN ) where NP are the positive data points and NN the negative ones . Other reasons include the non-convexity and perceived complexity of the appAUC landscape , and importantly the fact that it has zero derivatives almost everywhere ( Ghanbari & Scheinberg , 2018 ) . The loss function is one of the most critical choices in the design of a neural network , because it is the element that underlies the entire learning procedure . At first glance , it may appear that all loss functions solve the same problem for a given neural network , which led Rosasco et al . ( 2004 ) to question the practical differences between alternative loss functions . Yet , it is widely accepted that different loss functions allow us to optimise for specific properties ( Janocha & Czarnecki , 2017 ) . By studying the appAUC LFL , we can quantitatively address these important questions . The topology of the loss function has been a topic of interest for over 20 years ( Hochreiter & Schmidhuber , 1997 ) . Understanding the loss function for a given neural network can help to establish the fundamental interpretation of the ‘ black-box ’ , as such networks are often described ( Li et al. , 2017 ) . However , a key problem with studying the LFL is the large associated computational cost . Unlike the standard machine learning approach , where only a single minimum is located , the LFL approach attempts to sample a representative set of minima , which may often exceed 10,000 solutions ( Ballard et al. , 2017 ) . Computing the true number of minima for a loss function would require enhanced sampling techniques ( Wales , 2013 ; Martiniani et al. , 2016 ) . Recently , others have looked into the LFL of overparameterised neural networks ( Cooper , 2018 ) and shown that various properties can be exploited to improve accuracy and , importantly , robustness of neural networks ( Baldassi et al. , 2020 ; Chaudhari et al. , 2019 ) . Insights from the LFL have also been used to understand initialisation methods and their relative success ( Fort & Scherlis , 2019 ) . However , to the best of our knowledge , there has been no research so far into interpreting individual loss functions from the underlying LFL . In this work , we aim to address this knowledge gap , and show that studying the LFL can provide important insights into our understanding and design of neural networks and the associated loss functions . 2 METHODS . 2.1 NEURAL NETWORK . For this initial survey , we consider neural networks with a single hidden layer . We denote the set of data points as D = ( X , c ) , containing N : = |D| elements . For a given classification problem with C classes , a data point d ∈ D has input features xd , and a known output class , cd . We use a nonlinear tanh activation function and convert output values yi , i ∈ C into softmax probabilities by pi ( W ; x d ) = exp [ yi ( W ; x d ) ] / ∑C j=1 exp [ yj ( W ; x d ) ] , where W denotes the weight vector containing all weights of the neural network . As a reference to compare with our appAUC loss function , we use a cross-entropy ( CE ) loss function , defined as : CE ( W ; X ) = − 1 N N∑ d=1 ln ( pcd ( W ; x d ) ) + λW2 . ( 1 ) The λW2 in equation 1 represents an L2 regularisation term , which eliminates zero Hessian Eigenvalues and counteracts overfitting . We fix λ = 10−5 for a fair comparison between loss functions . 2.2 AREA UNDER THE CURVE . To evaluate a model such as the one described above , it is standard practice to consider the receiver operating characteristic ( ROC ) and calculate the area under the curve ( AUC ) ( Hastie et al. , 2009 ) . For a given classifier , the ROC is a plot of the true positive ratio , T ( P ) , against the false positive ratio , F ( P ) . These quantities are defined by T ( W ; X , P ) = ∑N d=1 δ ( c d − 1 ) Θ ( p1 ( W ; xd ) − P ) ∑N d=1 δ ( c d − 1 ) , ( 2 ) F ( W ; X , P ) = ∑N d=1 ( 1− δ ( cd − 1 ) ) Θ ( p1 ( W ; xd ) − P ) ∑N d=1 1− δ ( cd − 1 ) , ( 3 ) where δ ( cd − 1 ) is the Dirac delta function , and Θ ( p1 − P ) is the Heaviside step function , defined as δ ( cd − 1 ) = { 1 if cd = 1 , 0 if cd 6= 1 , Θ ( p1 − P ) = { 1 if p1 ≥ P , 0 if p1 < P , ( 4 ) and P is a parameter , which acts as a cutoff probability for the neural network to predict that a given data point belongs to class 1 ( p1 ) . Note that the choice of class 1 as a positive reading is arbitrary , and for a multi-class system , any class may be chosen . The ROC curve is defined by T and F as P varies from 0 to 1 . For a perfect classifier , the ROC would simply be a horizontal line at T = 1 ( i.e . ∀ P 6= 0 , T ( P ) = 1 ) , and the area under the curve would then be 1 . The AUC is therefore a measure of how close the model is to a perfect classifier . Formally , the AUC as a function of network weights W , parameterised by the data X , is given by AUC = ∫ 1 0 T ( W ; X , P ) dF ( W ; X , P ) = 1 NPNN ∑ p ∑ n Θ ( p1 ( W ; x p ) − p1 ( W ; xn ) ) ( 5 ) where p labels positive data points ( class 1 ) , and n labels negative data points ( not in class 1 ) . 2.3 APPROXIMATED AUC : APPAUC LOSS FUNCTION . Most optimisation routines benefit from analytical derivatives of the function to be optimised . The function defined in equation 1 has smooth analytical derivatives , but equation 5 does not , because of the discontinuous step function . However , in a similar way to other approaches ( e.g . ( Calders & Jaroszewicz , 2007 ) ) , one can replace the discontinuous Θ function with an approximate , smooth , analytical surrogate function , which can then be differentiated and optimised . We write AUC ( W ; X ) ≈ A ( W ; X ) ≡ 1 NPNN ∑ p ∑ n 1 1 + exp ( −β ( p1 ( W ; xp ) − p1 ( W ; xn ) ) ) , ( 6 ) where the Heaviside step function has been replaced with a smooth sigmoid function : Θ ( z ) → σ ( z ) ≡ 1 1 + exp ( −βz ) , ( 7 ) with parameter β that is discussed in detail in the Appendix . An important consideration for surrogate AUC loss functions is that they do not change the optimal solution when replacing the step function . Such surrogates are referred to as AUC-consistent ( Agarwal , 2014 ) . Charoenphakdee et al . ( 2019 ) show that sigmoid is an AUC-consistent surrogate . The appAUC loss function in equation 6 is now differentiable and can hence be used for optimisation , where we minimise the negative form of equation 6 such that a minimum ( i.e . lower loss ) is a good solution of higher AUC . For reference , we have included the analytical first and second derivatives for Equation 6 in the Appendix .

Summary: This paper studies AUC maximization in classification tasks, which aims to directly optimize the AUC metric instead of minimizing traditional classification losses such as the cross-entropy loss. The main contribution of this paper is an empirical study of an existing AUC maximization method termed approximate AUC. Specifically, this paper compares the landscape of the approximate AUC surrogate loss with that of the cross-entropy loss. The empirical results gives an explanation on why AUC maximization can achieve higher test AUC compared to the conventional method that directly minimizes the cross-entropy loss.