Characterising the Area Under the Curve Loss Function Landscape

1 INTRODUCTION . The area under the curve of the receiver operating characteristic curve ( AUC ) is a commonly used method to evaluate the accuracy and reliability of a neural network classifier . However , for mathematical reasons , the AUC can not be used as as the loss function to be minimised during neural network training ( Menon & Elkan , 2011 ) . Instead , other functions such as cross-entropy are commonly employed . The stepwise nature of the AUC function is the reason that it is nondifferentiable and hence can not simply be optimised . However , the AUC can be approximated using surrogate losses such as the sigmoid function . This approach can lead to a formulation equivalent to the soft-AUC described by Calders & Jaroszewicz ( 2007 ) , which is referred to here as appAUC . We find it intuitive to optimise a function that it as close as possible to the one used to evaluate the model . Our main contributions in this paper are therefore : • To understand the use of different approximation to the AUC as the loss function employed in training of a neural network • To explore the organisation of the appAUC landscape and compare it to a ‘ standard ’ cross- entropy landscape • A theoretical foundation of the differences between appAUC and cross-entropy landscapes • To outline how a loss function landscape analysis can be useful for loss function selection To better understand the advantages and disadvantages of the approximated AUC loss function , we study the functional space , commonly referred to as LFL , using tools from the theoretical study of energy landscapes in molecular and condensed matter systems ( Wales , 2003 ) . The usefulness of the energy landscape approach has previously been demonstrated in the context of neural network LFLs ( Ballard et al. , 2017 ) . We will employ methods from this approach to gain insights about geometric features of the LFL , including the number of minima , their curvatures , and their connectivity . By repeatedly surveying large parts of the LFL , we are , with high probability , able to find the true global minimum . Additionally , due to a Metropolis criterion in the global optimisation approach described below , we do not get stuck in a local minimum , and explore the full LFL , hence learning more about the functional surface . Instead of a single minimum , we aim to find a large number of minima that , together with transition states , provide a faithful coarse-grained representation of the loss function landscape . We believe that this approach will yield valuable insights into the use of appAUC as a loss function in neural networks . Various interesting questions arise from the use of an appAUC loss function . Besides a comparison of properties between landscapes , and the effects of hyperparameter changes , we are especially interested in the differences between appAUC and CE landscapes . Both loss functions , for the same neural network architecture , address the learning problem of finding a mapping f from input data to class label . Does this common foundation imply that minima of the CE loss function are also minima of the appAUC function , or are they at least very similar to each other ? We will show below that this condition does not hold , and explain why . Furthermore , to the best of our knowledge , there has been no previous research into the functional properties of AUC surrogates . We believe that quantifying inherent , geometric properties of loss functions will provide a more fundamental understanding of the applicability of particular loss functions to distinct machine learning problems . 1.1 RELATED WORK . This contribution lies at the intersection of three different areas of research : general understanding and study of loss functions , the appAUC loss function in particular , and the study of loss function landscapes to understand neural networks . Optimising the AUC as a loss function has been considered before ( Cortes & Mohri , 2004 ) . It has been shown that testing AUC is improved when a loss function is chosen that is closer to the true AUC function ( Yan et al. , 2003 ) . Nonetheless , optimising loss functions that approximate the AUC is rarely considered . One reason for this situation may be that the computational complexity is formally O ( N2 ) , or specifically in the binary classification case employed here , O ( NPNN ) where NP are the positive data points and NN the negative ones . Other reasons include the non-convexity and perceived complexity of the appAUC landscape , and importantly the fact that it has zero derivatives almost everywhere ( Ghanbari & Scheinberg , 2018 ) . The loss function is one of the most critical choices in the design of a neural network , because it is the element that underlies the entire learning procedure . At first glance , it may appear that all loss functions solve the same problem for a given neural network , which led Rosasco et al . ( 2004 ) to question the practical differences between alternative loss functions . Yet , it is widely accepted that different loss functions allow us to optimise for specific properties ( Janocha & Czarnecki , 2017 ) . By studying the appAUC LFL , we can quantitatively address these important questions . The topology of the loss function has been a topic of interest for over 20 years ( Hochreiter & Schmidhuber , 1997 ) . Understanding the loss function for a given neural network can help to establish the fundamental interpretation of the ‘ black-box ’ , as such networks are often described ( Li et al. , 2017 ) . However , a key problem with studying the LFL is the large associated computational cost . Unlike the standard machine learning approach , where only a single minimum is located , the LFL approach attempts to sample a representative set of minima , which may often exceed 10,000 solutions ( Ballard et al. , 2017 ) . Computing the true number of minima for a loss function would require enhanced sampling techniques ( Wales , 2013 ; Martiniani et al. , 2016 ) . Recently , others have looked into the LFL of overparameterised neural networks ( Cooper , 2018 ) and shown that various properties can be exploited to improve accuracy and , importantly , robustness of neural networks ( Baldassi et al. , 2020 ; Chaudhari et al. , 2019 ) . Insights from the LFL have also been used to understand initialisation methods and their relative success ( Fort & Scherlis , 2019 ) . However , to the best of our knowledge , there has been no research so far into interpreting individual loss functions from the underlying LFL . In this work , we aim to address this knowledge gap , and show that studying the LFL can provide important insights into our understanding and design of neural networks and the associated loss functions . 2 METHODS . 2.1 NEURAL NETWORK . For this initial survey , we consider neural networks with a single hidden layer . We denote the set of data points as D = ( X , c ) , containing N : = |D| elements . For a given classification problem with C classes , a data point d ∈ D has input features xd , and a known output class , cd . We use a nonlinear tanh activation function and convert output values yi , i ∈ C into softmax probabilities by pi ( W ; x d ) = exp [ yi ( W ; x d ) ] / ∑C j=1 exp [ yj ( W ; x d ) ] , where W denotes the weight vector containing all weights of the neural network . As a reference to compare with our appAUC loss function , we use a cross-entropy ( CE ) loss function , defined as : CE ( W ; X ) = − 1 N N∑ d=1 ln ( pcd ( W ; x d ) ) + λW2 . ( 1 ) The λW2 in equation 1 represents an L2 regularisation term , which eliminates zero Hessian Eigenvalues and counteracts overfitting . We fix λ = 10−5 for a fair comparison between loss functions . 2.2 AREA UNDER THE CURVE . To evaluate a model such as the one described above , it is standard practice to consider the receiver operating characteristic ( ROC ) and calculate the area under the curve ( AUC ) ( Hastie et al. , 2009 ) . For a given classifier , the ROC is a plot of the true positive ratio , T ( P ) , against the false positive ratio , F ( P ) . These quantities are defined by T ( W ; X , P ) = ∑N d=1 δ ( c d − 1 ) Θ ( p1 ( W ; xd ) − P ) ∑N d=1 δ ( c d − 1 ) , ( 2 ) F ( W ; X , P ) = ∑N d=1 ( 1− δ ( cd − 1 ) ) Θ ( p1 ( W ; xd ) − P ) ∑N d=1 1− δ ( cd − 1 ) , ( 3 ) where δ ( cd − 1 ) is the Dirac delta function , and Θ ( p1 − P ) is the Heaviside step function , defined as δ ( cd − 1 ) = { 1 if cd = 1 , 0 if cd 6= 1 , Θ ( p1 − P ) = { 1 if p1 ≥ P , 0 if p1 < P , ( 4 ) and P is a parameter , which acts as a cutoff probability for the neural network to predict that a given data point belongs to class 1 ( p1 ) . Note that the choice of class 1 as a positive reading is arbitrary , and for a multi-class system , any class may be chosen . The ROC curve is defined by T and F as P varies from 0 to 1 . For a perfect classifier , the ROC would simply be a horizontal line at T = 1 ( i.e . ∀ P 6= 0 , T ( P ) = 1 ) , and the area under the curve would then be 1 . The AUC is therefore a measure of how close the model is to a perfect classifier . Formally , the AUC as a function of network weights W , parameterised by the data X , is given by AUC = ∫ 1 0 T ( W ; X , P ) dF ( W ; X , P ) = 1 NPNN ∑ p ∑ n Θ ( p1 ( W ; x p ) − p1 ( W ; xn ) ) ( 5 ) where p labels positive data points ( class 1 ) , and n labels negative data points ( not in class 1 ) . 2.3 APPROXIMATED AUC : APPAUC LOSS FUNCTION . Most optimisation routines benefit from analytical derivatives of the function to be optimised . The function defined in equation 1 has smooth analytical derivatives , but equation 5 does not , because of the discontinuous step function . However , in a similar way to other approaches ( e.g . ( Calders & Jaroszewicz , 2007 ) ) , one can replace the discontinuous Θ function with an approximate , smooth , analytical surrogate function , which can then be differentiated and optimised . We write AUC ( W ; X ) ≈ A ( W ; X ) ≡ 1 NPNN ∑ p ∑ n 1 1 + exp ( −β ( p1 ( W ; xp ) − p1 ( W ; xn ) ) ) , ( 6 ) where the Heaviside step function has been replaced with a smooth sigmoid function : Θ ( z ) → σ ( z ) ≡ 1 1 + exp ( −βz ) , ( 7 ) with parameter β that is discussed in detail in the Appendix . An important consideration for surrogate AUC loss functions is that they do not change the optimal solution when replacing the step function . Such surrogates are referred to as AUC-consistent ( Agarwal , 2014 ) . Charoenphakdee et al . ( 2019 ) show that sigmoid is an AUC-consistent surrogate . The appAUC loss function in equation 6 is now differentiable and can hence be used for optimisation , where we minimise the negative form of equation 6 such that a minimum ( i.e . lower loss ) is a good solution of higher AUC . For reference , we have included the analytical first and second derivatives for Equation 6 in the Appendix .

Summary: The paper studied the AUC loss by investigating its approximates and by comparing it with the cross-entropy loss. The landscapes of the optimization based on approximate AUC and based on the cross-entropy loss are also studied. Numerical experiments are conducted.