Year: 2021

Abstract:
Minima of the loss function landscape of a neural network are locally optimal sets of weights that extract and process information from the input data to make outcome predictions. In underparameterised networks, the capacity of the weights may be insufficient to fit all the relevant information. We demonstrate that different local minima specialise in certain aspects of the learning problem, and process the input information differently. This effect can be exploited using a meta-network in which the predictive power from multiple minima of the LFL is combined to produce a better classifier. With this approach, we can increase the area under the receiver operating characteristic curve (AUC) by around 20% for a complex learning problem. We propose a theoretical basis for combining minima and show how a meta-network can be trained to select the representative that is used for classification of a specific data item. Finally, we present an analysis of symmetryequivalent solutions to machine learning problems, which provides a systematic means to improve the efficiency of this approach.

heading: 1 INTRODUCTION
text: Deep learning with neural networks (NNs) is a high-dimensional, non-convex optimisation problem for a loss function landscape (LFL). The coordinates of a minimum in the LFL are a set of weights for the machine learning model and a locally optimal solution to the learning problem, and these terms will therefore be used interchangeably throughout. It follows that the coordinates of the global minimum of the LFL are the weights that produce the lowest possible value of the loss function for the training data. The aim of machine learning is usually for the model to find a set of weights that fit the training data, but also generalise well to unseen testing data. Our approach extends this view. Instead of looking at just one minimum of the LFL, we are interested in the expressive power of multiple minima. To analyse how different minima extract and process information from the input data, we survey numerous low-lying minima of the LFL. Here, we employ tools from the energy landscape approach (Wales, 2003) to gain new insight into machine learning LFLs (Ballard et al., 2017). We note that the concept of a minimum is somewhat abstract in machine learning landscapes compared to molecular systems. While in a molecular energy landscape only minima provide valid configurations for a stable molecule, this restriction does not apply to LFLs for machine learning. In fact, some low-lying non-minima will have a smaller loss value and higher classification accuracy than a high-lying minimum. Here, we are interested in developing a better understanding of the capacity of diverse minima of the LFL, and showing that by combining the expressive power of different minima, we can build a better classifier. The compact form of this predictor provides a balance between accuracy and efficiency as required in applications where evaluation is a computational bottleneck.

heading: 1.1 BACKGROUND
text: Machine learning models are structurally limited in the amount of data they can fit: their capacity is finite. The most commonly known measure of capacity is perhaps the Vapnik–Chervonenkis (VC) dimension (Vapnik & Chervonenkis, 1971; Vapnik et al., 1994). The higher the VC dimension, the more complex are the data can be fitted. More rigorously, VC dimension is defined as the largest cardinality of a set of data points that the NN can shatter (for our purpose, shatter means classify correctly). Thus, the weights of an underparameterised model (i.e. fewer parameters than training data points) may be incapable of fitting the entire test data set, but instead fit just parts of it.
The approach we employ to study the expressive power of combinations of individual minima is a variation of ensemble learning, where the results of multiple different predictors are combined to improve the overall accuracy of an approximation problem (Dong et al., 2020). The idea of combining multiple sources of information, specifically the output predictions of multiple classifiers, has been considered for over two decades (Breiman, 1996; Hashem, 1997; Jin & Lu, 2009). Two of the most important questions in ensemble learning are: which classifiers to consider, and how to combine the individual predictions (Wang, 2008). For a detailed review see Kuncheva (2014).

heading: 1.2 MOTIVATION
text: In this contribution, we are interested in quantitatively and systematically characterising a cornerstone of ensemble learning, namely classifier diversity. Logically, ensemble learning works if different classifiers extract different information from the input data or process it differently (Melville & Mooney, 2005; Zaidi et al., 2020). In the present work, the classifiers in question correspond to local minima of a reference neural network. We aim to visualise diversity of minima from the corresponding LFL and show how to select a few of them to produce a compact yet more accurate classification. We will show that different minima of the LFL successfully classify distinct subsets of the entire input dataset. Hence different local minima specialise in distinct parts of the test dataset, which we believe has not been shown before.
In summary, our main contributions are:
• Showing that different local minima specialise in distinct subsets of the input • MLSUP, a proof-of-concept method that exploits minima diversity to improve classification
results for complex problems • An interpretation of the limitation of single-minima models and visualisation of the differ-
ences between minima • Novel insights into the symmetry properties of minima in neural network LFLs

heading: 2 SUPERPOSITION OF MACHINE LEARNING SOLUTIONS: MLSUP
text: We observe that different local minima of a reference neural network extract different information from the input data and that combining just a few examples can improve classification significantly. To study this effect, we employ a modified stacking approach where multiple minima from the same classifier are combined, rather than multiple classifiers. We do not obtain these minima by different random initialisation but rather from sampling solutions from the LFL. This approach provides insight into the functional landscape and a deeper understanding of LFL minima. To answer the second important question in ensemble learning design, we employ a second neural network to select one of the local minima for a given input data item. This idea is related to previous theory, Jordan & Jacobs (1994) where a gating network chooses which classifier to apply to some problem (Shazeer et al., 2017; McGill & Perona, 2017). We call our method MLSUP, to denote a superposition of machine learning solutions (local minima of the LFL). We describe MLSUP in a four step process (Figure 1). The first step involves characterising local minimaM by exploring the loss function landscape during training. Next, we choose a subset of minimaM′ ⊆M and evaluate eachm ∈M′ for every training datapoint, which reveals how well each of them can classify specific data items (step 2 in Figure 1). A detailed discussion of how a few minima are selected for combination is included below. The superposition of chosen minima is done by training a second, meta-network (classifier 2, i.e. step 3 in Figure 1) to learn which of the m ∈M′ minima is best suited to classify a specific input datapoint. Thus, the second network learns to apply different minima to classify different types of input data, as shown in Step 4 of Figure 1. A pseudocode version of MLSUP is provided in the Appendix.

heading: 3 MODEL
text: We consider a classification problem for C classes with a single hidden layer, as we are specifically interested in underparameterised networks. For some data D = (X, c), the inputs are denoted X = {x1, . . . , xN}, where N is the number of data points in the training or testing set, which are denoted as Xtrain and Xtest respectively. The correct label for some data point d is defined as cd. We
use tanh as the nonlinear activation function to the hidden layer, since it has continuous derivatives, which we require for optimisation. Outputs at node yi are converted to softmax probabilities pi(W; X) = exp(yi)/ exp( ∑ j e
yj ), where W denotes the vector containing all weights. During training, we minimise a loss function L(W; X) with respect to these weights. We use a cross-entropy loss function
L(W; X) = − 1 N N∑ d=1 ln pcd(W; X) + λW2 (1)
where cd is the correct class for some data item xd. A L2 regularisation term λW2 is added to eliminate zero Hessian eigenvalues, which by Noether’s theorem arise as a consequence of continuous symmetries in the loss function when additively shifting all the output bias weights (Ballard et al., 2017). We find λ = 10−5 to be appropriate for the tests considered; our conclusions are largely insensitive to λ. This setup is used for the neural networks in steps 1, 2 and 4 of Figure 1.

heading: 3.1 DEFINING THE META-NETWORK LOSS FUNCTION
text: For step 3 of Figure 1, the loss function is different from equation 1. This section describes Classifier 2 from Figure 1, which is distinct from the one used for steps 1, 2 and 4. For Classifier 2, we are not interested in learning cd, i.e. the correct output class, but rather the best local minimum to classify some input data item d, which is defined by the highest corresponding probability:
bd = arg maxm mpcd ∀ m ∈M′ (2) for data item d, where m is one set of weights. This formulation changes the loss function to
L(W̃; X) = − 1 N N∑ d=1 ln pbd(W̃; X) + λW̃2. (3)
with W̃ representing the weights for network 2. We evaluate the classification predictions of our model using the area under the receiver operating characteristic curve (ROC-AUC) (Fawcett, 2006). The change in loss function also impacts the way we calculate the AUC for Step 4 in Figure 1. In the usual case, the AUC is given as
AUC = ∫ 1 0 T (P )dF (P ) (4)
with the true positive rate for outcome number one, T (P ), and the false positive rate, F (P ),
T (P ) =
∑Ndata d δ(c
d − 1)Θ(p1 − P )∑Ndata d δ(c d − 1) F (P ) =
∑Ndata d [1− δ(cd − 1)]Θ(p1 − P )∑Ndata
d 1− δ(cd − 1) (5)
where δ(cd − 1) is the Dirac delta function, and Θ(p1 − P ) the Heaviside step function, defined as δ(cd − 1) = {
1 if cd = 1, 0 if cd 6= 1 Θ(p1 − P ) = { 1 if p1 ≥ P, 0 if p1 < P
(6)
However, we now have |M ′| possibilities. Thus, we must evaluate the AUC using the minimum sd that is chosen by Classifier 2 from Figure 1
T (P ) =
∑Ndata d δ(c
d − 1)Θ(psd1 − P )∑Ndata d δ(c d − 1) F (P ) =
∑Ndata d [1− δ(cd − 1)]Θ(ps d
1 − P )∑Ndata d 1− δ(cd − 1)
(7)

heading: 3.2 OPTIMISATION ROUTINE
text: To survey the loss function landscape we employ methods from the energy landscape approach, which has been widely used to study molecular and condensed matter systems in the physical sciences (Wales, 2003). Specifically, global optimisation is performed using the basin-hopping method (Li & Scheraga, 1987; Wales & Doye, 1997) with a customised quasi-Newton L-BFGS (Nocedal, 1980) optimiser. More information is included in the Appendix. Candidates for transition states are obtained using a doubly-nudged (Trygubenko & Wales, 2004a;b) elastic band (Henkelman & Jónsson, 2000; Henkelman et al., 2000) approach and accurately refined by hybrid-eigenvector following (Munro & Wales, 1999; Zeng et al., 2014). The routines are implemented in the GMIN GMI, OPTIM OPT and PATHSAMPLE PAT programs, which are available under the GNU General Public License.

heading: 4 MINIMA SELECTION
text: The problem analogous to classifier selection in ensemble learning for MLSUP is minima selection. Here our motivation is to identify a small number of minima that significantly enhance the accuracy of the predictions, to produce a compact representation suitable for use in large-scale simulations that are typical in physical science applications. Hence we need to identify complementary minima that can be combined to improve the classification. We compare two methods for minima selection: one based on the theory of a thermodynamic analogue for the LFL, the other on more abstract machine learning concepts. We will also explain why simply picking minima that are ‘far away’ in Euclidean distance can be ineffective.

heading: 4.1 EUCLIDEAN DISTANCE
text: The Euclidean distance between two weight vectors is simply |w1 −w2| = √
(w1 − w2)2. However, minima ‘far away’ from each other in Euclidean space may not extract different information from the training data. In fact, there are symmetry-related minima, potentially distant in Euclidean space, that for the same input data will return exactly the same loss value. We refer to these solutions as permutational invariants or isomers, as for molecular systems. This sort of problem was considered in Dinh et al. (2017), where redundant minima are shown to exist for non-negative homogeneous nonlinear functions, such as the rectifier function φrect(x) = max(x, 0). However, these results do not apply here because tanh is not homogeneous. Instead, let G denote the permutation group for some set of weights w, i.e. all permutations allowed such that for the same input, they return the same
output. By shuffling hidden nodes (Brea et al., 2019), and the fact that tanh is an odd function, we can show that for each minimum, there exist
|G| = H∏ l=1 (nl!× 2nl) or |G| = H−1∏ l=1 (nl!× 2nl)× nH ! (8)
degenerate solutions for each minimum, depending on whether the number of hidden layers H is odd or even, respectively, where nl is the number of nodes in hidden layer l. Because tanh is an odd function, we can change the sign of the weights before and after each hidden node and each combination of any number of nodes for an odd number of hidden layers. This result increases the per-layer number of degenerate minima from n! to n! × 2n, i.e. for each node swap, any nodes in each element of the power set of the nodes P(n) could be multiplied by a factor of −1 with no effect on the output. Thus, randomly picking some ‘far away’ minima might just return permutationally invariant solutions, which does not lead to any performance improvement in MLSUP. For further interest, we have included in the Appendix an example of four such permutational invariants and a pairwise distance matrix of permutational isomers of one minimum. In conclusion, a large Euclidean distance for minima selection is neither a necessary, nor a sufficient selection condition.

heading: 4.2 MISCLASSIFICATION SPACE
text: Instead of computing the distance between two sets of weights in Euclidean space, we can compute a metric in misclassification space. The rationale behind this approach is that minima with a high pairwise misclassification rate will be complementary, and therefore good candidates for MLSUP. We denote a vector containing the predicted output class for each data item i ∈ X for some minimum m as c(m)i , and define the set Ξ = {n | c (1) n 6= c(2)n }, which contains the indices of all elements that are classified differently by the two minima. To obtain a misclassification ratio Λ from this set, we simply represent the cardinality of Ξ as a fraction of the number of data points Λ = |Ξ|/N , i.e. the number of misclassified items divided by the total number of items. If we compute Λ for all m ∈M, where M again denotes the set of all known minima, we obtain a symmetric matrix of pairwise differences in classification space. We can then train MLSUP on selected minima that are distant in classification space. Below, we will abbreviate the Maximum Misclassification Distance to the Best Minimum as MMDBM. A more theoretical basis for weighting classifiers according to correlation of errors is given in Masegosa et al. (2020).
4.3 HEAT CAPACITY (CV )
Building on results for molecular systems, we can compute a theoretical analogue of the heat capacity (CV ) (Wales, 2017). The heat capacity reports upon the changing occupation probabilities of local minima as the temperature changes, which depends on both the relative loss function value for the minimum, and a local density of states (the analogue of entropy) in weight space. By computing the partial derivatives of the occupation probabilities with respect to temperature, we can identify the minima with the largest rate of change of probability, which theory shows make the largest contribution to the heat capacity.(Wales, 2017) Peaks in the heat capacity correspond to changes in occupation between local minima with different loss function values and entropy. Thus, in physical systems, the minima with the largest positive and negative partial derivative around a peak generally have qualitatively different properties. Peaks in CV therefore tell us where to find local minima with complementary properties (Wales, 2017), which may provide good candidates for combinations in a new classifier. This reasoning as the original motivation for the MLSUP procedure, since the heat capacity analysis is computationally efficient and physically insightful.

heading: 4.4 DATA
text: To test MLSUP, we chose to study the spiral data problem (Lang & Witbrock, 1988). An example can be seen in Figure 4. This problem is considered to be relatively difficult because of its high degree of non-linear separability. We further increase the difficulty by adding a small uniform noise term to each datapoint. Our models are trained on 3,200 data points and tested on 800. To show that the MLSUP approach is generalisable, we report results for other datasets in the Appendix.

heading: 5 RESULTS
text: In this section, we report general results for MLSUP and specifically the impact on the AUC for different minima selection methods and different numbers of minima combined in MLSUP. We benchmark the MLSUP AUC in four ways. First, we compare it to the best individual AUC of the minima that we choose for MLSUP. Second, we compare it to the best AUC of the non-MLSUP system (classifier 1 in Figure 1) from which the MLSUP minima are selected. Third, in order to evaluate MLSUP, we compare its AUC with the best possible AUC achievable from combining the selected n minima. To compute this theoretical maximum AUC (TMA), for each datapoint we select the maximum Softmax probability of the correct class amongst all minima considered. The TMA gives an upper bound for the accuracy of MLSUP. Thus, to evaluate the quality of Classifier 2 alone, an AUC achieved with MLSUP should be compared against the TMA. Lastly, we compare it with a simple majority vote (with ties considered as correctly classified).
5.1 MLSUP EFFECT
A substantial increase in the AUC can be achieved with MLSUP. For the original network (Classifier 1 in Figure 1), we found 7,842 minima and a maximum AUC value of 0.76. Using MLSUP with two of these minima (6436 and 656), we obtain an AUC of 0.91. By combining four minima, the AUC increases to 0.94 (Table 1). The weights for these four minima are shown in Figure 2. There is no apparent relationship between these four sets, yet they are all minima of the original LFL and have similar AUCs. We also find that MLSUP outperforms a ma-
jority vote and include results for other benchmark datasets, higher-dimensional problems and MLSUP hyperparameters in the Appendix.

heading: 5.2 MINIMA SELECTION
text: distance in Euclidean and misclassification space is only 0.06, which demonstrates that Euclidean distance alone is not useful for minima selection. Additional details can be found in the Appendix.

heading: 5.2.1 MISCLASSIFICATION SPACE
text: The distance between minima in misclassification space is the best minima selection method for an MLSUP meta-network based on two minima. By combining the best individual AUC minimum (656) with the minimum furthest away from it in misclassification space (6436), we obtain an AUC value of 0.91 (MMDBM). We note that the individual AUC values of both these minima were reasonably high already, at 0.74 and 0.76. The best possible AUC that MLSUP could have achieved, if for each data item the better minimum had been chosen for this system (TMA) is 0.97 (Table 1).
5.2.2 HEAT CAPACITY
The heat capacity (CV ) curve for the LFL has two peaks, a small one at low T and a larger peak at T ≈ 3×10−3 (Fig. 3). Such aCV curve in a molecular system would indicate a solid-solid phase transition at low temperature (first peak), and a solid-liquid transition (melting) at higher temperature (second peak) (Wales, 2003). In the neural network case, these peaks can be understood in terms of transitions between two energetically/entropically different sets of minima. By combining the minima that make the greatest contributions to CV at the larger peak (Fig.3) in MLSUP, we improve the AUC to 0.86 (Table 1). For the minima that produce the smaller peak, the improvement is similar, with an AUC value of 0.88 (Table 1). The TMA of MLSUP is 0.98 for the first and 0.97 for the second peak.

heading: 6 DISCUSSION
text: Combining different minima in one classifier can substantially improve the ROC-AUC for a complex problem like spiral data. This result strongly suggests that different minima extract different information from the input and/or process it in a different way. Our results shed new light on the LFLs of complex problems. Even the global minimum is limited in the amount of information it can extract and process. To maximise the efficiency of MLSUP, we therefore aim to identify complementary high-AUC minima for which the intersection of correctly classified datapoints is small. This selection translates to a large distance in misclassification space. Such minima are ideal candidates for MLSUP.
6.0.1 COMPUTATIONAL RESOURCES
Training MLSUP for four minima is fast and hence does not constitute a bottleneck. All tasks were run on six-core dual Xeon X5650 (2.6GHz) nodes with 24Gb RAM per node. Using just one core, MLSUP is trained within a few hours. The initial analytical landscape exploration requires more subtle considerations. Due to the nature of the LFL (theoretically infinitely large, yet implicitly bounded by L2-regularisation), more minima can generally be found. For reference, finding 7,842 minima took us around 92 hours. The computing time required for landscape exploration depends on system size, number of data points, and various parameters in the optimisation routine, but also the structure of the LFL, which is specific to a given problem. Landscape exploration can be parallelised, so there is scope to speed up the calculations significantly in the future.

heading: 6.1 APPLICATIONS FOR MLSUP
text: We emphasise that our main objective in the present contribution is to improve our understanding of the LFL and investigate whether the MLSUP approach might be useful. In particular, we have investigated the underlying assumptions concerning the information extracted by different minima for underparameterised networks. For larger, overparameterised networks, a single minimum may suffices to achieve near-perfect AUC. However, evaluating a complex classifier could become a computational bottleneck in large-scale simulations of physical systems. Hence, a compact MLSUP representation could be useful for such applications. For example, we have an application where predicted values of specific molecular properties are required at every step in a molecular simulation (Vendruscolo & Dobson, 2005; Dedmon et al., 2005; Lindorff-Larsen et al., 2005; Clore & Schwieters, 2006; Cavalli et al., 2007). The properties depend on the atomic coordinates, which vary throughout the simulation, and we need analytical derivatives of the properties (outputs) with respect to the coordinates (inputs). A compact but computationally efficient prediction engine, such as MLSUP, is needed to prevent these computations from becoming a bottleneck in the simulations.

heading: 6.2 MINIMA SELECTION
text: Selecting the correct minima for MLSUP is a key problem. Even when combining just two out of |M| minima, there are (|M| 2 ) possible combinations. While a choice guided by Euclidean distance may be appealing, this approach is ineffective due to low misclassification distance or permutationally equivalent solutions. Hence, we have analysed two other methods, one based on a theoretical analogue of the heat capacity, and the other on misclassification space. Combining minima of high misclassification distance achieves the best results. However, evaluating the misclassification space is relatively expensive and scales O(n2), where at each step, all data points must be evaluated for one of n minima. In contrast, the analysis of CV scales linearly as O(n). Additionally, a high misclassification rate between two individually poor minima is unlikely to substantially improve results, as the confidence in each minimum would be low, and a superposition would not necessarily improve this situation. We have not observed any such issues with the CV approach. The superior results of CV as opposed to random initialisation are striking, and demonstrate how a methodology developed for molecular energy landscapes can provide new insight into machine learning LFLs.

heading: 6.3 INTERPRETATION OF DIFFERENCES BETWEEN MINIMA
text: Figure 4 provides an explanation for the substantial improvements that are achieved in MLSUP. While Figure 2 shows that the individual minima are very different from each other, it does not show how these differences are propagated to classification decisions. Figure 4 shows that individual minima are good at predicting specific ranges of the input data, and only in combination do they manage to classify points all around the spiral. Note that the Softmax probability cannot generally be read as true probability because it lacks uncertainty quantification (Sensoy et al., 2018). However, it can be used here as a visualisation proxy due to the well-behaved nature of the synthetic data. We see that the number of large points per minimum correlates with the AUC. Minima 6436 and 656 have AUCs of 0.74 and 0.76, respectively. In the right panel of Figure 4, large parts of the MLSUP predictions are red or blue, corresponding to these two minima. In contrast, minimum 2428, which only seems to be able to accurately predict datapoints from Class 1 in the upper right quadrant, has a lower AUC (0.66). Another feature clearly illustrated in Figure 4 is the input-sign dependence of the minima. In particular, minima 1223 and 2428 only have high confidence for datapoints that lie in one of the four segments. While 2428 is good at classifying Class 1 datapoints with positive sign for both inputs, minimum 1223 works best for classifying Class 0 datapoints with positive x and negative y. Thus, if MLSUP receives an input data point where both coordinates are positive (and perhaps at least one of them large), it is likely to classify it using the weights from minimum 2428.
Combining the above results with Figure 2, we see that minimum 6436 has strongly positive weights going into the upper output node O0 and strongly negative weights into O1, while the opposite holds for minimum 656. Indeed, Figure 4 confirms that 6436 is very good at classifying data belonging to output class 0, while 656 is very good at classifying data belonging to output class 1. We therefore conclude that different minima truly ‘specialise’ in parts of the input data, thus providing the foundations for the MLSUP approach. Visualising the results we provides an intuitive way to confirm that different minima process the same information in a different way. This observation raises various
questions around initialisation methods and interpretability of ML models. Instead of considering model interpretability, it may be more appropriate to discuss the interpretability of different minima, as various solutions may extract, and hence focus on, completely different information from the input data. The overparameterisation of neural networks (Belkin et al., 2019) may explain why different minima achieve such good results individually. However, to explain neural network decision making, understanding that different minima process the input differently seems a crucial first step.

heading: 6.4 CONCLUSIONS
text: Different sets of weights associated with different minima of the loss function landscape extract different information from the input data, or process it differently. By training a meta-network to decide which minimum should be applied for any specific input data point, we can substantially improve classification accuracy. These results explain why different solutions to the same learning problem, obtained with different initialisation methods, have varying degrees of accuracy. Furthermore, we show that if the capacity of the network is limited, individual minima specialise in parts of the network. In future work, it will be interesting to consider these results in the light of mathematical measures of network capacity, such as VC-dimensions. In addition to this insight, we also provide an analysis of redundant minima which, although they might appear to be different, return the same loss value for some input. Such symmetries make minima selection more challenging. Simple measures, such as the Euclidean distance in weight space, are insufficient due to low correlation with misclassification distance. To solve this problem, we can exploit thermodynamic analogues for the LFL by analogy to molecular systems. In particular, we find that defining a heat capacity analogue for the LFL provides an efficient and direct approach for locating complementary local minima. Computing the heat capacity, and identifying the minima that make the largest contributions to the peaks, is a promising method for constructing a compact but relatively accurate classifier in the MLSUP approach. This representation may prove useful for applications where evaluation of the classifier is a potential bottleneck, as for simulations in the physical sciences, where predictions of molecular properties are required at every step.Vendruscolo & Dobson (2005); Dedmon et al. (2005); Lindorff-Larsen et al. (2005); Clore & Schwieters (2006); Cavalli et al. (2007) The analogy to physical systems provides further motivation for pursing the energy landscape view of machine learning loss function landscapes.

heading: 7 APPENDIX

heading: 7.1 FURTHER EXPERIMENTS

heading: 7.2 PERMUTATIONAL INVARIANTS

heading: 7.2.1 EXAMPLES
text: Here we provide additional insight into the issue of permutationally equivalent sets of weights.
1-2-3 1-3-2 2-1-3 2-3-1 3-1-2 3-2-1
1-2-3
1-3-2
2-1-3
2-3-1
3-1-2
3-2-1 0
10
20
30
40
50
60
E u clid ean d istan ce
Supplementary Figure 5: Euclidean distance between 48 permutationally equivalent local minima of the global minimum for a particular LFL.
Supplementary Figure 6: An example of four very different looking minima that are actually permutationally equivalent, i.e. redundant solutions. For any input, these four networks will all return exactly the same loss value.

heading: 7.3 COMPARISON OF EUCLIDEAN AND MISCLASSIFICATION SPACE
text: This section presents further evidence that the Euclidean distance alone is insufficient for minima selection. Figure 7 shows that only a very weak correlation exits between Euclidean and misclassification distance. Large misclassification distance can be observed roughly as often for points of large Euclidean distance as it is observed for points close together in the parameter space defined by the weights. Given the good results of misclassification distance for minima selection, the low correlation coefficient indicates that Euclidean distance alone is not useful.
0 10 20 30 40 50 60
Euclidean distance
0
10
20
30
40
50
60
70
M is
cl as
si fi
ca ti
on d
is ta
n ce
0.0
0.2
0.4
0.6
0.8
1.0
L og relative d en sity
Supplementary Figure 7: Euclidean vs Misclassification distance for each of the ( n 2 ) combinations of n minima of the spiral data LFL. The Pearson correlation coefficient for Euclidean distance vs misclassification is 0.06.

heading: 7.4 MLSUP ALGORITHM
text: The MLSUP algorithm can be represented in pseudocode as:
Algorithm 1 Train MLSUP Require: D = (X, c)
Sample LFL, find |M| minima Pick some |M′| minima candidates Evaluate each minimum m ∈ |M′| on training data Train second nn to learn arg maxm mpcd for d ∈ D do
Evaluate first network, query second network for which minimum to use end for
7.5 CHOOSING m′ MINIMA BY PAIRWISE MISCLASSIFICATION DISTANCE
One may wish to compute MLSUP with m′ minima, chosen to have the maximum misclassification distance from another. This is a combinatorial optimisation problem, analogous to having m cities with pairwise distances and trying to find the m′ cities that are furthest from each other. This problem can be solved by quadratic programming with convex relaxation. Let D be an upper triangular matrix with element Dij denoting the misclassification distance from point i to point j, where i < j. The square matrix D has 0 diagonal. Denote x as a m-dimensional binary vector, which contains the value 1 at the position of a chosen minimum and 0 elsewhere. The objective is then to maximise x>Dx. In practice, the values of x will be relaxed so they need not be discrete, and the optimisation procedure becomes
max y
y>Dy subject to 0 ≤ yi ≤ 1 ∀i, 1 · y = m′ (9)
The final task is then to discretise y back to x such that y>Dy is very close to x>Dx. We have achieved this mapping by simply setting the m′ largest elements of y to 1 and all others to 0, but other methods of approximation may work equally well or even better.

heading: 7.6 BASIN-HOPPING GLOBAL OPTIMISATION
text: The first step in surveying the LFL for a neural network is to locate the global minimum. We perform global optimisation using the basin-hopping method, which uses a Metropolis criterion to avoid getting stuck in local minima (Li & Scheraga, 1987; Wales & Doye, 1997). For local minimisation we use a modified quasi-Newton L-BFGS (Limited-memory Broyden–Fletcher–Goldfarb–Shanno) algorithm (Nocedal, 1980). After a local basin of attraction (local minimum) is found via L-BFGS optimisation, a ’basin-hopping’ jump is performed to some other point in the LFL. This jump is always accepted if the energy, here loss value, is lower than the current minimum. If the loss value of the new point is higher, it is accepted with probability
P ∝ exp ( −∆Ẽ kBT ) (10)
where ∆Ẽ is the difference in loss value between the current minimum and the new point, kB the Boltzmann constant and T a fictitious temperature. Intuitively, if the energy difference between the old and new points is large, the move is less likely to be accepted. A minimum is characterised as such if a RMS norm of gradient vector) convergence criterion at a threshold of 10−10 is reached. Each basin-hopping run naturally produces a sampling of low-lying minima as it progresses, in addition to the global minimum.

Review summaries:
The paper proposes a method to select a subset of local minima of loss function landscape and combine the minima using a second meta-network which learns to select a set of weights for each input data point. The subset of local minima is selected based on two criteria, minima that are distant in classification space and peak heat capacity. The combination of selected minima achieves better classification accuracy than the best individual and minima selected based on Euclidean distance. 

This paper studies a form of ensembling of under-parameterized networks. They show that different local minima specialize distinct subset of inputs, and an algorithm called MLSUP can be used to combine them to improve the prediction. The method is empirical tested with a 1-hidden-layer network on a 2D synthetic classification dataset (and in the appendix an artificially corrupted UCI Iris flower task).

## Summary This paper proposes a new ensemble learning based on neural networks.  First, A basin-hopping method is used to sample diverse minima of the empirical risk landscape. Then, a meta-network is trained to predict which minimum performs the best for specific input. This meta-network maps the input sample to the index set of minima. This can be viewed as an adaptive ensemble method, in the sense that for different input samples, the evaluated models are different.