On the Capacity and Superposition of Minima in Neural Network Loss Function Landscapes

1 INTRODUCTION . Deep learning with neural networks ( NNs ) is a high-dimensional , non-convex optimisation problem for a loss function landscape ( LFL ) . The coordinates of a minimum in the LFL are a set of weights for the machine learning model and a locally optimal solution to the learning problem , and these terms will therefore be used interchangeably throughout . It follows that the coordinates of the global minimum of the LFL are the weights that produce the lowest possible value of the loss function for the training data . The aim of machine learning is usually for the model to find a set of weights that fit the training data , but also generalise well to unseen testing data . Our approach extends this view . Instead of looking at just one minimum of the LFL , we are interested in the expressive power of multiple minima . To analyse how different minima extract and process information from the input data , we survey numerous low-lying minima of the LFL . Here , we employ tools from the energy landscape approach ( Wales , 2003 ) to gain new insight into machine learning LFLs ( Ballard et al. , 2017 ) . We note that the concept of a minimum is somewhat abstract in machine learning landscapes compared to molecular systems . While in a molecular energy landscape only minima provide valid configurations for a stable molecule , this restriction does not apply to LFLs for machine learning . In fact , some low-lying non-minima will have a smaller loss value and higher classification accuracy than a high-lying minimum . Here , we are interested in developing a better understanding of the capacity of diverse minima of the LFL , and showing that by combining the expressive power of different minima , we can build a better classifier . The compact form of this predictor provides a balance between accuracy and efficiency as required in applications where evaluation is a computational bottleneck . 1.1 BACKGROUND . Machine learning models are structurally limited in the amount of data they can fit : their capacity is finite . The most commonly known measure of capacity is perhaps the Vapnik–Chervonenkis ( VC ) dimension ( Vapnik & Chervonenkis , 1971 ; Vapnik et al. , 1994 ) . The higher the VC dimension , the more complex are the data can be fitted . More rigorously , VC dimension is defined as the largest cardinality of a set of data points that the NN can shatter ( for our purpose , shatter means classify correctly ) . Thus , the weights of an underparameterised model ( i.e . fewer parameters than training data points ) may be incapable of fitting the entire test data set , but instead fit just parts of it . The approach we employ to study the expressive power of combinations of individual minima is a variation of ensemble learning , where the results of multiple different predictors are combined to improve the overall accuracy of an approximation problem ( Dong et al. , 2020 ) . The idea of combining multiple sources of information , specifically the output predictions of multiple classifiers , has been considered for over two decades ( Breiman , 1996 ; Hashem , 1997 ; Jin & Lu , 2009 ) . Two of the most important questions in ensemble learning are : which classifiers to consider , and how to combine the individual predictions ( Wang , 2008 ) . For a detailed review see Kuncheva ( 2014 ) . 1.2 MOTIVATION . In this contribution , we are interested in quantitatively and systematically characterising a cornerstone of ensemble learning , namely classifier diversity . Logically , ensemble learning works if different classifiers extract different information from the input data or process it differently ( Melville & Mooney , 2005 ; Zaidi et al. , 2020 ) . In the present work , the classifiers in question correspond to local minima of a reference neural network . We aim to visualise diversity of minima from the corresponding LFL and show how to select a few of them to produce a compact yet more accurate classification . We will show that different minima of the LFL successfully classify distinct subsets of the entire input dataset . Hence different local minima specialise in distinct parts of the test dataset , which we believe has not been shown before . In summary , our main contributions are : • Showing that different local minima specialise in distinct subsets of the input • MLSUP , a proof-of-concept method that exploits minima diversity to improve classification results for complex problems • An interpretation of the limitation of single-minima models and visualisation of the differ- ences between minima • Novel insights into the symmetry properties of minima in neural network LFLs 2 SUPERPOSITION OF MACHINE LEARNING SOLUTIONS : MLSUP . We observe that different local minima of a reference neural network extract different information from the input data and that combining just a few examples can improve classification significantly . To study this effect , we employ a modified stacking approach where multiple minima from the same classifier are combined , rather than multiple classifiers . We do not obtain these minima by different random initialisation but rather from sampling solutions from the LFL . This approach provides insight into the functional landscape and a deeper understanding of LFL minima . To answer the second important question in ensemble learning design , we employ a second neural network to select one of the local minima for a given input data item . This idea is related to previous theory , Jordan & Jacobs ( 1994 ) where a gating network chooses which classifier to apply to some problem ( Shazeer et al. , 2017 ; McGill & Perona , 2017 ) . We call our method MLSUP , to denote a superposition of machine learning solutions ( local minima of the LFL ) . We describe MLSUP in a four step process ( Figure 1 ) . The first step involves characterising local minimaM by exploring the loss function landscape during training . Next , we choose a subset of minimaM′ ⊆M and evaluate eachm ∈M′ for every training datapoint , which reveals how well each of them can classify specific data items ( step 2 in Figure 1 ) . A detailed discussion of how a few minima are selected for combination is included below . The superposition of chosen minima is done by training a second , meta-network ( classifier 2 , i.e . step 3 in Figure 1 ) to learn which of the m ∈M′ minima is best suited to classify a specific input datapoint . Thus , the second network learns to apply different minima to classify different types of input data , as shown in Step 4 of Figure 1 . A pseudocode version of MLSUP is provided in the Appendix . 3 MODEL . We consider a classification problem for C classes with a single hidden layer , as we are specifically interested in underparameterised networks . For some data D = ( X , c ) , the inputs are denoted X = { x1 , . . . , xN } , where N is the number of data points in the training or testing set , which are denoted as Xtrain and Xtest respectively . The correct label for some data point d is defined as cd . We use tanh as the nonlinear activation function to the hidden layer , since it has continuous derivatives , which we require for optimisation . Outputs at node yi are converted to softmax probabilities pi ( W ; X ) = exp ( yi ) / exp ( ∑ j e yj ) , where W denotes the vector containing all weights . During training , we minimise a loss function L ( W ; X ) with respect to these weights . We use a cross-entropy loss function L ( W ; X ) = − 1 N N∑ d=1 ln pcd ( W ; X ) + λW2 ( 1 ) where cd is the correct class for some data item xd . A L2 regularisation term λW2 is added to eliminate zero Hessian eigenvalues , which by Noether ’ s theorem arise as a consequence of continuous symmetries in the loss function when additively shifting all the output bias weights ( Ballard et al. , 2017 ) . We find λ = 10−5 to be appropriate for the tests considered ; our conclusions are largely insensitive to λ . This setup is used for the neural networks in steps 1 , 2 and 4 of Figure 1 . 3.1 DEFINING THE META-NETWORK LOSS FUNCTION . For step 3 of Figure 1 , the loss function is different from equation 1 . This section describes Classifier 2 from Figure 1 , which is distinct from the one used for steps 1 , 2 and 4 . For Classifier 2 , we are not interested in learning cd , i.e . the correct output class , but rather the best local minimum to classify some input data item d , which is defined by the highest corresponding probability : bd = arg maxm mpcd ∀ m ∈M′ ( 2 ) for data item d , where m is one set of weights . This formulation changes the loss function to L ( W̃ ; X ) = − 1 N N∑ d=1 ln pbd ( W̃ ; X ) + λW̃2 . ( 3 ) with W̃ representing the weights for network 2 . We evaluate the classification predictions of our model using the area under the receiver operating characteristic curve ( ROC-AUC ) ( Fawcett , 2006 ) . The change in loss function also impacts the way we calculate the AUC for Step 4 in Figure 1 . In the usual case , the AUC is given as AUC = ∫ 1 0 T ( P ) dF ( P ) ( 4 ) with the true positive rate for outcome number one , T ( P ) , and the false positive rate , F ( P ) , T ( P ) = ∑Ndata d δ ( c d − 1 ) Θ ( p1 − P ) ∑Ndata d δ ( c d − 1 ) F ( P ) = ∑Ndata d [ 1− δ ( cd − 1 ) ] Θ ( p1 − P ) ∑Ndata d 1− δ ( cd − 1 ) ( 5 ) where δ ( cd − 1 ) is the Dirac delta function , and Θ ( p1 − P ) the Heaviside step function , defined as δ ( cd − 1 ) = { 1 if cd = 1 , 0 if cd 6= 1 Θ ( p1 − P ) = { 1 if p1 ≥ P , 0 if p1 < P ( 6 ) However , we now have |M ′| possibilities . Thus , we must evaluate the AUC using the minimum sd that is chosen by Classifier 2 from Figure 1 T ( P ) = ∑Ndata d δ ( c d − 1 ) Θ ( psd1 − P ) ∑Ndata d δ ( c d − 1 ) F ( P ) = ∑Ndata d [ 1− δ ( cd − 1 ) ] Θ ( ps d 1 − P ) ∑Ndata d 1− δ ( cd − 1 ) ( 7 ) 3.2 OPTIMISATION ROUTINE . To survey the loss function landscape we employ methods from the energy landscape approach , which has been widely used to study molecular and condensed matter systems in the physical sciences ( Wales , 2003 ) . Specifically , global optimisation is performed using the basin-hopping method ( Li & Scheraga , 1987 ; Wales & Doye , 1997 ) with a customised quasi-Newton L-BFGS ( Nocedal , 1980 ) optimiser . More information is included in the Appendix . Candidates for transition states are obtained using a doubly-nudged ( Trygubenko & Wales , 2004a ; b ) elastic band ( Henkelman & Jónsson , 2000 ; Henkelman et al. , 2000 ) approach and accurately refined by hybrid-eigenvector following ( Munro & Wales , 1999 ; Zeng et al. , 2014 ) . The routines are implemented in the GMIN GMI , OPTIM OPT and PATHSAMPLE PAT programs , which are available under the GNU General Public License .

Summary: ## Summary This paper proposes a new ensemble learning based on neural networks.  First, A basin-hopping method is used to sample diverse minima of the empirical risk landscape. Then, a meta-network is trained to predict which minimum performs the best for specific input. This meta-network maps the input sample to the index set of minima. This can be viewed as an adaptive ensemble method, in the sense that for different input samples, the evaluated models are different.