On the Capacity and Superposition of Minima in Neural Network Loss Function Landscapes
Authors: Maximilian Paul Niroomand
John William Roger Morgan
Conor T Cafolla
David John Wales
Venue: ICLR 2022 Submitted (2022)
Keywords: ensemble learning
interpretability
loss function landscape
theoretical chemistry

Abstract: Minima of the loss function landscape of a neural network are locally optimal sets of
weights that extract and process information from the input data to make outcome predictions. 
In underparameterised networks, the capacity of the weights may be insufficient to fit all the relevant information. 
We demonstrate that different local minima specialise in certain aspects of the learning problem, and process the input
information differently. This effect can be exploited using a meta-network in
which the predictive power from multiple minima of the LFL is combined to produce a better
classifier. With this approach, we can increase the area under the receiver operating characteristic curve
(AUC) by around $20\%$ for a complex learning problem. 
We propose a theoretical basis for combining minima and show how a meta-network can
be trained to select the representative that is used for classification of a
specific data item. Finally, we present an analysis of symmetry-equivalent
solutions to machine learning problems, which provides a systematic means to improve the
efficiency of this approach.

Under review as a conference paper at ICLR 2022
ON THE CAPACITY AND SUPERPOSITION OF MINIMA IN NEURAL NETWORK Loss FUNCTION LANDSCAPES
Anonymous authors Paper under double-blind review
ABSTRACT
Minima of the loss function landscape of a neural network are locally optimal sets of weights that extract and process information from the input data make outcome predictions. In underparameterised networks, the capacity of the weights may be insufficient to fit all the relevant information. We demonstrate that different local minima specialise in certain aspects of the learning problem, and process the input information differently. This effect can be exploited using a meta-network in which the predictive power from multiple minima of the LFL is combined to produce a better classifier. With this approach, we can increase the area under the receiver operating characteristic curve (AUC) by around 20% for a complex learning problem. We propose a theoretical basis for combining minima and show how meta-network can be trained to select the representative that is used for classification of a specific data item. Finally, we present an analysis of symmetry- equivalent solutions machine learning problems, which provides systematic means to improve the efficiency of this approach.
INTRODUCTION
Deep learning with neural networks (NNs) is high-dimensional, non-convex optimisation problem for a loss function landscape (LFL). The coordinates of minimum in the LFL are a set of weights for the machine learning model and a locally optimal solution to the learning problem, and these terms will therefore be used interchangeably throughout. It follows that the coordinates of the global minimum of the LFL are the weights that produce the lowest possible value of the loss function for the training data. The aim of machine learning is usually for the model find set of weights that fit the training data, but also generalise well to unseen testing data. Our approach extends this view. Instead of looking at just one minimum of the LFI., we are interested in the expressive power of multiple minima. To analyse how different minima extract and process information from the input data, we survey numerous low-lying minima of the LFL. Here, we employ tools from the energy landscape approach (Wales) 2003) to gain new insight into machine learning LFLs (Ballard et al.) 2017). We note that the concept of minimum is somewhat abstract machine learning landscapes compared to molecular systems. While in a molecular energy landscape only minima provide valid configurations for a stable molecule, this restriction does not apply to LFLs for machine learning. In fact, some low-lying non- non-minima will have smaller loss value and higher classification accuracy than a high-lying minimum. Here, we are interested in developing a better understanding of the capacity of diverse minima of the LFL, and showing that by combining the expressive power of different minima, we can build a better classifier. The compact form of this predictor provides a balance between accuracy and efficiency required in applications where evaluation is computational bottleneck.
1.1 BACKGROUND
Machine learning models are structurally limited in the amount of data they can fit: their capacity is finite. The most commonly known measure of capacity is perhaps the Vapnik-Chervonee dimension (Vapnik& & Chervonenkis 1971 Vapnik et al. 1994). The higher the VC dimension, the more complex are the data can be fitted. More rigorously, yy, VC dimension is defined as the largest cardinality of set of data points that the NN can shatter (for our purpose, shatter means classify correctly). Thus, the weights of an underparameterisedd model (i.e. fewer parameters than training data points) may be incapable of fitting the entire test data set, but instead fit just parts of it.
(VC)

---

Under review as a conference paper at ICLR 2022
The approach we employ study the expressive power of combinations of individual minima is variation of ensemble learning, where the results of multiple different predictors are combined to improve the overall accuracy of an approximation problem (Dong al.] 2020). The idea of combining multiple sources of information, specifically the output predictions of multiple classifiers, has been considered for over two decades (Breiman 1996; Hashem 1997 Jin & Lu 2009). Two of the most important questions in ensemble learning are: which classifiers to consider, and how to combine the individual predictions WWan 2008). For a detailed review see Kuncheva (2014).
1.2 MOTIVATION
In this contribution, we are interested in quantitatively and systematically characterising cornerstone of ensemble learning, namely classifier diversity. Logically, ensemble learning works if different classifiers extract different information from the input data process it differently (Melville & Mooney] 2005| Zaidi et al. 2020). In the present work, the classifiers in question correspond to local minima of reference neural network. We aim visualise diversity of minima from the corresponding LFL and show how to select few of them to produce a compact yet more accurate classification. We will show that different minima of the LFL successfully classify distinct subsets of the entire input dataset. Hence different local minima specialise in distinct parts of the test dataset, which we believe has not been shown before.
In summary, our main contributions are:
Showing that different local minima specialise in distinct subsets of the input
MLSUP, proof-of-concept method that exploits minima diversity improve classification results for complex problems
An interpretation of the limitation of single-minima models and visualisation of the differ- ences between minima
Novel insights into the symmetry properties of minima in neural network LFLs
2 SUPERPOSITION OF MACHINE LEARNING SOLUTIONS: MLSUP
We observe that different local minima of a reference neural network extract different information from the input data and that combining just a few examples can improve classification significantly. To study this effect, we employ a modified stacking approach where multiple minima from the same classifier are combined, rather than multiple classifiers. We do not obtain these minima by different random initialisation but rather from sampling solutions from the LFL. This approach provides insight into the functional landscape and a deeper understanding of LFL minima. To answer the second important question in ensemble learning design, we employ a second neural network select of the local minima for a given input data item. This idea is related to previous theory, theory,JJrdan [Jordan & Jacobs (1994) where gating network chooses which classifier to apply some problem (Shazeer al. McGill McGill & Perona 2017). We call our method MLSUP, to denote a superposition of machine learning solutions (local minima of the LFL). We describe MLSUP in a four step process (Figure I The first step involves characterising local minima M by exploring the loss function landscape during training. Next, we choose subset minima M' S . and evaluate each m E M' for every training datapoint, which reveals how well each of them can classify specific data items (step 2 in Figure D. A detailed discussion of how a few minima are selected for combination is included below. The superposition of chosen minima is done by training second, meta-network (classifier 2, i.c. step 3 in Figure| Dh to learn which of the m E MM minima is best suited to classify a specific input datapoint. Thus, the second network learns apply different minima classify different types of input data, as shown in Step 4 of Figure| A pseudocode version of MLSUP is provided in the Appendix.
3 MODEL.
We consider a classification problem for C' classes with single hidden layer, as we are specifically interested in underparameterrised networks. For some data D (X. the inputs are denoted
= {x] xx.,,"..x""" xN., where N is the number of data points in the training or testing set, which are
denoted X and X test respectively. The correct label for some data point d is defined as da We
2

---

Under review as a conference paper at ICLR 2022
Evalunte velected solations from Classifey
Eval mi
Probubility the cocrect ourpur "u" for each
Xiris
tnin
Classifier
indiviidual weights
Find [.M] solutions
M
Each element of
Fxail .ace)
M
weights
Classifier
"P,,
Lean
as item
nearal network on
clasification problem
2: Foshuate the accuracy of selected minima
Xast
Evaluator
3: Train sccond classutter to leam which set weights works best foe which input data
with associated weights
Classiffer determines which set of weights to use for each individual inpur item these weights, inpur classified again
Text the added probern
incorporating classi-
Figure 1: Diagrammatic representation of the MLSUP learning procedure. Ellipsoid boxes contain the true outputs, which are compared to the predicted output for training/evaluation purposes.
use tanh as the nonlinear activation function to the hidden layer, since has continuous derivatives, which require for optimisation. Outputs at node Yi are converted softmax probabilities P: p:(W; X) = exp(ıı) exp(E, ed>),, where W denotes the vector containing all weights. During training, we minimise loss function L(W:X) with respect to these weights. We use cross-entropy loss function
N
L(W:X) = N Σ
In p (W:X) + 2
(1)
where cd is the correct class for some data item x². A L2 regularisation term 2 is added to eliminate zero Hessian eigenvalues, which by Noether's theorem arise as a consequence continuous symmetries in the loss function when additively shifting all the output bias weights (Ballard et al.) 2017). We find À = 10-5 to be appropriate for the tests considered; our conclusions are largely insensitive to A. This setup is used for the neural networks in steps 1, 2 and 4 of Figure [
3.1 DEFINING THE META-NETWORK LOSS FUNCTION
This section describes Classifier 2 from Figure II which is distinct from the one used for steps 1, and For Classifier 2, we interested in learning c¹ i.e. the correct output class, but rather the best local minimum to classify some input data item d, which is defined by the highest corresponding probability:
For step 3 of Figure I the loss function is different from equation]] 1
5d = arg max "re V m E M'
(2)
for data item d, where m is set of weights. This formulation changes the loss function to
N
X): = N
Σ
In Psa (:
2
(3)
d=1
3

---

Under review as a conference paper at ICLR 2022
with W representing the weights for network 2. We evaluate the classification predictions of our model using the area under the receiver operating characteristic curve (ROC-AUC) (Fawcett) 2006).
The change in loss function also impacts the way calculate the AUC for Step 4 in Figure] In the usual case, the AUC given as
AUC = -f. T(P)d((P)
(4)
with the true positive rate for outcome number one, T(P), and the false positive rate, F(P),
T(P) Ed Nass 5(c² 1)0(pi-1 - P)
F(P) Ed - 5(c² 1)] - P)
(5)
Ed
Ld 1 - 8(c -
where 8(c² - 1) is the Dirac delta function, and O(p1 P) the Heaviside step function, defined as
5(c - 1) {
if 1, if #1
{
if pı P if pf < P
-
(p1 P)
(6)
However, we now have |M' possibilities. Thus, we must evaluate the AUC using the minimum sd that is chosen by Classifier from Figure
T(P) = ENam 8(e 1)0(pi PP)
F(P) = ENan [1 8(c² (p) - PP)
(7)
ENum 8(ca 1)
Σ 8(c²² - 1)
3.2 OPTIMISATION ROUTINE
To survey the loss function landscape we employ methods from the energy landscape approach, which has been widely used study molecular and condensed matter systems in the physical sciences Waless 2003). Specifically, global optimisation is performed using the basin-hopping method (Li &
Scheraga 1987 Wales Doye 1997) with a customised quasi-Newton L-BFGS (Nocedal, 1980)
optimisr. More information is included in the Appendix. Candidates for transition states are obtained using a doubly-nudged (Trygubenko
Wales 20044a b) elastic band (Henkelman Jónsson, 2000)
Henkelman et al. (2000) approach and accurately refined by eibred-eigenvector following (Munro & Wales Zeng et al. 2014). The routines are implemented in the GMIN GMI OPTIM OPTIMOOPTanm OPT]and
PATHSAMPLE PAT programs, which are available under the GNU General Public License.
MINIMA SELECTION
The problem analogous to classifier selection ensemble learning for MISUP is minima selection. Here our motivation is identify a small number of minima that significantly enhance the accuracy of the predictions, to produce a compact representation suitable for use in large-scale simulations that typical in physical science applications. Hence we need to identify complementary minima that can be combined to improve the classification. We compare two methods for minima selection: one based on the theory of thermodynamic analogue for the LFL, the other on more abstract machine learning concepts. We will also explain why simply picking minima that are 'far away" in Euclidean distance can ineffective.
4.1 EUCLIDEAN DISTANCE
The Euclidean distance between two weight vectors is simply |w1- -wl =((w minima 'far away' from each other in Euclidean space may not extract different information from the training data. In fact, there are symmetry-related minima, potentially distant in Euclidean space, that for the same input data will return exactly the same loss value. We refer to these solutions as permutational invariants or isomers, as for molecular systems. This sort of problem was considered in Dinh et al. (2017). where redundant minima are shown to exist for non-negative homogeneous nonlinear functions, such the rectifier function (rect(z)) max(æ; (z. 0). However, these results do not apply here because tanh is not homogencous. Instead, let & denote the permutation group for some set of weights w. i.e. all permutations allowed such that for the same input, they return the same
W2) However,
4

---

Under review as a conference paper at ICLR 2022
output. By shuffling hidden nodes (Brea et al. 2019), and the fact that tanh is an odd function, can show that for each minimum, there exist
H II (mgl x 2nd) or |9| II (nd 2n) x ng!
H-1
|C]
(8)
degenerate solutions for each minimum, depending whether the number of hidden layers H is odd or even, respectively, where ny is the number of nodes in hidden layer 1. Because tanh is an odd function, we can change the sign of the weights before and after each hidden node and each combination of any number of nodes for an odd number of hidden layers. This result increases the per-layer number of degenerate minima from n!! to n! 2n, i.e. for each node swap, any nodes in each element of the power set of the nodes (n(n) could be multiplied by a factor of 1 with no effect on the output. Thus, randomly picking some 'far away' minima might just return permutationally invariant solutions, which does not lead to any performance improvement in MLSUP. For further interest, we have included in the Appendix an example of four such permutational invariants and a pairwise distance matrix of permutational isomers of one minimum. In conclusion, a large Euclidean distance for minima selection neither a necessary, nor a sufficient selection condition.
4.2 MISCLASSIFICATION SPACE
Instead of computing the distance between two sets of weights in Euclidean space, we can compute a metric in misclassification space. The rationale behind this approach is that minima with a high pairwise misclassification rate will be complementary. and therefore good candidates for MLSUP. We denote vector containing the predicted output class for each data item i E X for some minimum m {n2 c A c(²)), J. which contains the indices of all elements that are
as cf cm)) and define the set classified differently by the two minima. To obtain misclassification ratio A from this set, we simply represent the cardinality of E as a fraction of the number of data points A = |E|/N, i.e. the number of misclassified items divided by the total number of items. If we compute A for all m E .M, where M again denotes the set of all known minima, we obtain symmetric matrix of pairwise differences in classification space. We can then train MLSUP selected minima that distant in classification space. Below, we will abbreviate the Maximum Misclassification Distance to the Best Minimum MMDBM. A more theoretical basis for weighting classifiers according correlation of errors is given in Masegosa et al. (2020).
4.3 HEAT CAPACITY (Cv)
Building on results for molecular systems, we can compute a theoretical analogue of the heat capacity (Cv.) (Wales) 2017). The heat capacity reports upon the changing occupation probabilities of local minima as the temperature changes, which depends on both the relative loss function value for the minimum, and local density of states (the analogue of entropy) in weight space. By computing the partial derivatives of the occupation probabilities with respect to temperature, we can identify the minima with the largest rate of change of probability, which theory shows make the largest contribution to the heat capacity. (Wales) 2017) Peaks in the heat capacity correspond to changes in occupation between local minima with different loss function values and entropy. Thus, physical systems, the minima with the largest positive and negative partial derivative around peak generally have qualitatively different properties. Peaks Cv therefore tell us where to find local minima with complementary properties (Wales) 2017), which may provide good candidates for combinations in new classifier. This reasoning as the original motivation for the MLSUP procedure, since the heat capacity analysis is computationally efficient and physically insightful.
4.4 DATA
To test MLSUP, we chose to study the spiral data problem (Lang & Witbrock| 1988). An example can be seen in Figure This problem is considered to be relatively difficult because of its high degree of non-linear separability. We further increase the difficulty by adding small uniform noise term to each datapoint. Our models are trained on 3,200 data points and tested on 800. To show that the MUS approach is generalisable, we report results for other datasets in the Appendix.
5

---

Under review as a conference paper at ICLR 2022
5 RESULTS
In this section, we report general results for MLSUP and specifically the impact the AUC for different minima selection methods and different numbers minima combined in MLSUP. We benchmark the MLSUP AUC in four ways. First, compare it to the best individual AUC of the minima that we choose for MLSUP. Second, we compare it to the best AUC of the non-ML-SUP system (classifier 1 in Figure ! from which the MISUP minima evaluate MLSUP, we compare its AUC with the best possible AUC achievable from combining the selected minima. To compute this thcoretical maximum AUC (TMA), for each datapoint we select the maximum Softmax probability of the correct class amongst all minima considered. The TMA gives an upper bound for the accuracy of MLSUP. Thus, evaluate the quality of Classifier alone, an AUC achieved with MLSUP should be compared against the TMA. Lastly, we compare it with simple majority vote (with ties considered as correctly classified).
selected. Third, in order to
5.1 MLSUP EFFECT
A substantial increase in the AUC can be achieved with MLSUP. For the original network (Classifier 1 in Fig- [), found 7,842 min- ima and a maximum AUC value of 0.76. Using ML- SUP with two of these min- ima (6436 and 656), we obtain an AUC of 0.91. By combining four minima, the AUC increases to 0.94 (Table D The weights for these four minima are shown in Figure There apparent relationship between these four sets, yet they are all minima of the original LFL and have simi- lar AUCs. We also find that MLSUP outperforms ma- jority vote and include results for other benchmark datasets, higher-dimensional problems and MLSUP hyperparameters in the Appendix.
Mintmm:: 1251 ADO: 07
Munmenn: 2428 odt
Miniman: 6136 ATC: 3.74
Misiume 06 ALC 0776
Figure 2:
Four of weights, shown for small network. All four minima have very different sets of weights, as evidenced by the colours. However, their AUC values for the test data are similar.
Table 1: AUC improvements for different MLSUP settings
AUC
Selection method
# minima
Best individual
MLSUP
TMA
Majority vote
Cy peak
0.70
0.88
0.98
0.76
Cy peak 2
2
0.70
0.86
0.97
0.73
MMDBM
0.76
0.91
0.97
0.78
MMDBM & Cy peak 1
4
0.76
0.93
0.99
0.86
MMDBM & Cv peak 2
0.76
0.94
0.99
0.85
Max Euclidean distance
2
0.65
0.67
0.68
0.66
5.2 MINIMA SELECTION
Table clearly shows why minima selection routine is necessary. Choosing the two minima with the largest Euclidean distance, we obtain an AUC of 0.67, substantially lower than for either of the systematic minima selection methods. Furthermore, the Pearson correlation coefficient between

---

Under review as a conference paper at ICLR 2022
distance in Euclidean and misclassification space is only 0.06, which demonstrates that Euclidean distance alone not useful for minima selection. Additional details can be found in the Appendix.
5.2.1 MISCLASSIFICATTIN SPACE
The distance between minima in misclassification space is the best minima selection method for an MLSUP meta-network based two minima. By combining the best individual AUC minimum (656) with the minimum furthest away from it in misclassification space (6436), we obtain an AUC value of 0.91 (MMDBM). We note that the individual AUC values of both these minima were reasonably high already, at 0.74 and 0.76. The best possible AUC that MISUUP could have achieved, if for each data item the better minimum had been chosen for this system (TMA) 0.97 (Table
5.2.2 HEAT CAPACITY
The heat capacity (Cv) curve for the LFL has two peaks, a small one at low T and larger peak at 7 s 3 x 10-3 (Fig. 3 Such Cv curve in molecular system would in- dicate a solid-solid phase transition at low temperature (first peak), and solid-liquid transition (melting) at higher temperature (second peak) (Wales] (003). In the neural network case, these peaks can be under- stood in terms transitions between two pnergetically/entroicaalll different sets of minima. By combining the minima that make the greatest contributions to Cy at the larger peak (Fig 3) in MLSUP, we im- prove the AUC 0.86 (Table D For the minima that produce the smaller peak, the improvement is similar, with AUC value of 0.88 (Table D. The TMA of MLSUP is 0.98 for the first and 0.97 for the second peak.
20-
10
14
12-
0,002 0.004
0.000
T
Figure Heat capacity (Cv) curve for ML landscape. As the temperature is increased, the occupation prob- abilities change and more minima become accessible. Cv is calculated using an analogue of normal mode analysis for each minimum (Bogdan et 2006).
DISCUSSION
Combining different minima in one classifier can substantially improve ROC-AUC for a complex problem like spiral data. This result strongly suggests that different minima extract different informa- tion from the input and/or process it in a different way. Our results shed new light on the LFLs of complex problems. Even the global minimum is limited in the amount of information it can extract and process. To maximise the efficiency of MISUP, we therefore aim to identify complementary high-AUC minima for which the intersection of correctly classified datapoints is small. This selection translates to a large distance in misclassification space, Such minima are ideal candidates for MLSUP
6.0.1 COMPUTATIONAL RESOURCES
Training MLSUP for four minima is fast and hence does not constitute a bottleneck. All tasks were run on six-core dual Xeon X5650 (2.6GHz) nodes with 24Gb RAM per node. Using just one core, MLSUP is trained within few hours. The initial analytical landscape exploration requires more subtle considerations. Due to the nature the LFL (theoretically infinitely large, yet implicitly bounded by L2-regularisation), more minima minima took us around 92 hours. The computing time required for landscape exploration depends on system size, number of data points, and various parameters in the optimisation routine, but also the structure of the LFL, which is specific to given problem. Landscape exploration can be parallelised, there is scope speed up the calculations significantly in the future.
generally be found. For reference, finding 7,842
7

---

Under review as a conference paper at ICLR 2022
6.1 APPLICATIONS FOR MLSUP
We emphasise that our main objective in the present contribution is to improve our understanding of the LFL and investigate whether the MLSUP approach might be useful. In particular, we have investigated the underlying assumptions concerning the information extracted by different minima for underparameterised networks. For larger, overparameterised networks, a single minimum may suffices to achieve pear-perfect AUC. However, evaluating complex classifier could become computational bottleneck in large-scale simulations of physical systems. Hence, a compact MLSUP representation could be useful for such applications. For example, we have application where predicted values of specific molecular properties are required at every step in a molecular simulation (Vendruscolo & Dobson 2005 Dedmon et al. 2005 - Larsen et al.) (205) [Clore & Schwieters 2006] Cavalli et al. 2007). ). The properties depend on the atomic coordinates, which vary throughout the simulation, and we need analytical derivatives of the properties (outputs) with respect to the coordinates (inputs). A compact but computationally efficient prediction engine, such as MLSUP, is needed to prevent these computations from becoming a bottleneck in the simulations.
6.2 MINIMA SELECTION
Selecting the correct minima for MLSUP is a key problem. Even when combining just two out of [M]] minima, there are ((Mi) possible combinations. While a choice guided by Euclidean distance may be appealing. this approach is ineffective due to low misclassification distance permutationally equivalent solutions. Hence, we have analysed two other methods, one based on a theoretical analogue of the heat capacity, and the other on misclassification space. Combining minima of high misclassification distance achieves the best results. However, evaluating the misclassification space relatively expensive and scales Onn)). where at each step, all data points must be evaluated for one of n minima. In contrast, the analysis of Cv scales linearly as O(n). Additionally, a high misclassification rate between two individually poor minima is unlikely to substantially improve results, as the confidence in each minimum would be low, and a superposition would not necessarily improve this situation. We have not observed any such issues with the Cy approach. The superior results of Cv as opposed to random initialisation striking, and demonstrate how a methodology developed for molecular energy landscapes can provide new insight into machine learning LFLs.
6.3 INTERPRETATION OF DIFFERENCES BETWEEN MINIMA
Figure provides an explanation for the substantial improvements that are achieved in MLSUP. While Figure shows that the individual minima are very different from each other, it does not show how these differences are propagated to classification decisions. F are good at predicting specific ranges of the input data, and only in combination do they manage to classify points all around the spiral. Note that the Softmax probability cannot generally be read as true probability because it lacks uncertainty quantification (Sensoy et al. 2018). However, it can be used here as visualisation proxy due to the well-behaved nature of the synthetic data. We see that the number of large points per minimum correlates with the AUC. Minima 6436 and 656 have AUCs of 0.74 and 0.76, respectively. In the right panel of Figure 4 large parts of the MISUP predictions are red or blue, corresponding to these two minima. In contrast, minimum 2428, which only seems to be able to accurately predict datapoints from Class 1 in the upper right quadrant, has a lower AUC (0.66). Another feature clearly illustrated in Figure is the input-sign dependence of the minima. In particular, minima 1223 and 2428 only have high confidence for datapoints that lie in one of the four segments. While 2428 is good at classifying Class datapoints with positive sign for both inputs, minimum 1223 works best for classifying Class 0 datapoints with positive x and negative y. Thus, MISUP receives an input data point where both coordinates are positive (and perhaps at least one of them large), it likely to classify using the weights from minimum 2428.
that individual minima
Combining the above results with Figure 2 we see that minimum 6436 has strongly positive weights going into the upper output node O, and strongly negative weights into O₁, while the opposite holds for minimum 656. Indeed, Figure 4 confirms that 6436 is very good at classifying data belonging to output class O, while 656 is very good at classifying data belonging to output class 1. We therefore conclude that different minima truly 'specialise' in parts of the input data, thus providing the foundations for the MLSUP approach. Visualising the results we provides an intuitive way to confirm that different minima process the same information in different way. This observation raises various
8

---

Under review as a conference paper at ICLR 2022
Test data
MLSUP prediction of tedd data
Minimurs:
Classe
obe Soter
Set wae 4
B636
636
the
Figure 4: Contribution of each of four selected minima (right) to the MLSUP prediction of binary test data (left). Marker shape indicates the label, colour of the respective minimum, and size the Softmax probability that the respective minimum obtains for the correct class of this datapoint (scaled quadratically). Only points that are classified correctly by the respective minimum are included.
questions around initialisation methods and interpretability of ML models. Instead of considering model interpretability, may be more appropriate to discuss the interpretability different minima, as various solutions may extract, and hence focus on, completely different information from the input data. The overparamcterisation of neural networks (Belkin et al. 2019) may explain why different minima achieve such good results individually, However, to explain neural network decision making, understanding that different minima process the input differently seems a crucial first step.
6.4 CONCLUSIONS
Different sets of weights associated with different minima of the loss function landscape extract different information from the input data, or process it differently. By training a meta-network to decide which minimum should be applied for any specific input data point, we can substantially improve classification accuracy. These results explain why different solutions to the same learning problem, obtained with different initialisation methods, have varying degrees of accuracy. Further- more, we show that if the capacity of the network is limited, individual minima specialise in parts of the network. In future work. it will be interesting to consider these results in the light of mathematical measures of network capacity, such as VC-dimensions.
In addition to this insight, we also provide an analysis of redundant minima which, although they might appear to be different, return the same loss value for some input. Such symmetries make minima selection more challenging. Simple measures, such as the Euclidean distance in weight space, insufficient due to low correlation with misclassification distance. To solve this problem, we can exploit thermodynamic analogues for the LFL by analogy to molecular systems. In particular, we find that defining a heat capacity analogue for the LFL provides an efficient and direct approach for locating complementary local minima. Computing the heat capacity, and identifying the minima that make the largest contributions to the peaks, a promising method for constructing a compact but relatively accurate classifier in the ML.UP approach. This representation may prove useful for applications where evaluation of the classifier a potential bottleneck, as for simulations in the physical sciences, where predictions of molecular properties are required every step Vendruscolo & Dobson (2005); Dedmon et al. (2005): Lindorff- Larsen et al. (2005); Clore & Schwieters (2006); Cavalli et al. (2007) The analogy to physical systems provides further motivation for pursing the energy landscape view of machine learning loss function landscapes.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
GMIN: A program for basin- -osping global optimisation, basin-sampling, and parallel tempering. http://wwwwwales..h.aa..aa htmml.
OPTTM:
program for geometry optimisation and pathway calculations. http://www-
PATHSAMPLE: A program for generating connected stationary point databases and extracting global www.aals.cc..aa.a...//s/ss///eee
kinetics.
Andrew J. Ballard, Ritankar Das, Stefano Martiniani, Dhagash Mehta, Levent Sagun, Jacob D. Stevenson, and David J. Wales. Energy landscapes for machine learning. Physical Chemistry Chemical Physics, 19(20): 12585-12603, 2017.
Mikhail Belkin, Daniel Hsu. Siyuan Ma, and Soumik Mandal. Reconciling modern machine-learning practice and the classical varianar trade-off. Proceedings of the National Academy of Sciences, 15139)):5584-15554- August 2019.
Tetyana V. Bogdan, David J. Wales, and Florent Calvo. Equilibrium thermodynamics from basin- sampling. The Journal of Chemical Physics, 124(4):04410,, January 2006.
Johanni Brea, Berfin Simsek, Bernd Illing, and Wulfram Gerstner. Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape. arXi::1907.00 1907.0291 0291/ [cs, staf], 2019. arXiv: 1907.02911.
Leo Breiman. Bagging predictors. Machine Leurning, 24(2): August 1996.
Andrea Cavalli, Xavier Salvatella, Christopher M. Dobson, and Michele Vendruscolo. Protein structure determination from nmr chemical shifts. Proc. Natl. Acad. Sei. USA, 104(23):9615-9622 2007. doi: 10. 10773/paa.00610311111
G. Marius Clore and Charles D. Schwieters. Concordance of residual dipolar couplings, backbone order parameters and crystallographic b-factors for small alpha/beta protein: A unified picture of high probability, fast atomic motions proteins. J. Mol. Biol., 355(5):879 8867 2006. ISSN 0022-2836. doi: https://doi.org/100 0001.
Matthew M. Dedmon, Kresten Lindorff-Larsen John Christodoulou, Michele Vendruscolo, and Christopher M. Dobson. Mapping long-range interactions in alpha-synuclein using spin-label nmr and ensemble molecular dynamics simulations. J. Am. Chem. Soc., 127(2):476-477. 2005. doi: 10. 10.1021/ja044834
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp Minima Can Generalize For Deep Nets. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp- 1019-1028. PMLR, August 2017.
Xibin Dong, Zhiwen Yu. Wenming Cao, Yifan Shi, and Qianli Ma. A survey ensemble learning. Frontiers of Computer Science, 14(2):241-258, April 2020.
Tom Fawcett. An introduction to ROC analysis. Pattern Recognition Letters, 27(8):661-874, June 2006.
Sherif Hashem. Optimal Linear Combinations of Neural Networks. Neural Networks, 10(4))599-614, June 1997.
Graeme Henkelman and Hannes Jónsson. Improved tangent estimate in the nudged elastic band method for finding minimum energy paths and saddle points. The Journal of Chemical Physics, 113(22):9978-999) December 2000.
Graeme Henkelman, Blas P. Uberuaga, and Hannes Jónsson. A climbing image nudged elastic band method for finding saddle points and minimum energy paths. The Journal of Chemical Physics, 113(22)9901-999) December 2000.
10

---

Under review as a conference paper at ICLR 2022
Hua Jin and Ying Lu. The optimal linear combination of multiple predictors under the generalized linear models. Statistics & Probability Letters, 79(22):321-2322) November 2009.
Michael I Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):1812214 1994.
Ludmila I. Kuncheva. Combining Puttern Classifiers: Methods and Algorithms. John Wiley & Sons, Inc., Hoboken, NJ, USA, September 2014. ISBN 978-1-118-91455 978-1-118-31523-1
Kevin Lang and Michael Witbrock. Learning to tell spirals apart. In Proceedings of the 1988 connectionist models summer school, pp. 52-59. San Matco, 1988. Issue: 1989.
7. Li and H. A. Scheraga. Monte Carlo-minimization approach to the multiple-minima problem in protein folding. Proceedings of the National Academy of Sciences, (((1))6661-6611) October 1987.
Kresten Lindorff-Larsen. Robert B. Best, Mark A. DePristo, Christopher M. Dobson, and Michele Vendruscolo. Simultaneous determination of protein structure and dynamics. Nature, 433(7022): 128-132, 2005. doi: 10. 10338//aturr003199
Andrés R Masegosa, Stephan Sloth Lorenzen, Christian Igel, and Yevgeny Seldin. Second order pac-bayesian bounds for the weighted majority vote. In NeurlPS, 2020.
Mason McGill and Pietro Perona. Deciding how to decide: Dynamic routing in artificial neural networks. In International Conference on Machine Learning, pp. 2363-2372. PMLR, 2017.
Prem Melville and Raymond J Mooney, Creating diversity in ensembles using artificial data. Infor- mation Fusion, 6(1):99-111, 2005.
Lindsey J. Munro and David Wales. Defect migration in crystalline silicon. Physical Review B, 59 (6):3969-3980, February 1999.
Jorge Nocedal, Updating quasi-Newton matrices with limited storage. Mathematics of Computation, 35(151):773-773. September 1980.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning quantify classification uncertainty. arXiv preprint 1rXXi:1806.00766 2018.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gate mixture-of-eepprts layer. arXiv preprint /70iv 06538, 2017.
Semen A. Trygubenko and David J. Wales. Analysis of cooperativity and localization for atomic rearrangements. The Journal of Chemical Physics, 121(64))66996667, October 2004a.
Semen A. Trygubenko and David J. Wales. A doubly nudged elastic band method for finding transition states. The Journal of Chemical Physics, 120(5):0082-2094 February 2004b.
V. N. Vapnik and A. Ya. Chervonenkis. On the Uniform Convergence of Relative Frequencies of Events Their Probabilities. Theory of Probability & Its Applications, 16(2):264 280, January 1971.
Vladimir Vapnik, Esther Levin, and Yann Le Cun. Measuring the VC-Dimension of a Learning Machine. Neural Computation, 6(5):851 8766,177,,661 September 1994.
Michele Vendruscolo and Christopher M. Dobson. Towards complete descriptions of the free-energy landscapes of proteins. Phil. Trans. Roy. Soc. A, 363(1827))433 452, 2005. doi: 10. 10.1098/rsta. 2004.1501.
David J. Wales. Energy landscapes. Cambridge molecular science. Cambridge University Press, Cambridge, UK; New York, 2003. ISBN 978-0-521-81415-7
David Wales. Decoding heat capacity features from the energy landscape. PHYSICAL REVIEW E. pp. 6, 2017.
11

---

Under review as a conference paper at ICLR 2022
David J. Wales and Jonathan P. K. Doye. Global Optimization by Basin Basin-Hopping Hopping and the Lowest Energy Structures of Lennard-Jones Clusters Containing up to 110 Atoms. The Journal of Physical Chemistry A, 101(28)5511-5511 July 1997.
Wenjia Wang. Some fundamental issues in ensemble methods. In 2008 IEEE International Joint Conference on Neural Networks (IEEE World Congress on Computational Intelligence), pp. 2243- 2250, Hong Kong, China, June 2008, IEEE. ISBN 978-1-4244-18200 1820-6.
Sheheryar Zaidi, Arber Zela, Thomas Elsken, Chris Holmes, Frank Hutter, and Yee Whye Teh. Neural ensemble search for performant and calibrated predictions. arXiv preprint arXiv:2006.08573, 2020.
Yi Zeng, Penghao Xiao, and Graeme Henkelman. Unification of algorithms for minimum mode optimization. The Journal of Chemical Physics, 140(4):04415, 15, January 2014.
12

---

Under review as a conference paper at ICLR 2022
7 APPENDIX
7.1 FURTHER EXPERIMENTS
Table[[include some additional results that reinforce our conclusions. Importantly, we also want to test MLSUP other datasets to provide evidence that the performance increase is not specific to the spiral dataset. We first considered the well known Iris flower dataset? and, to make it more difficult, mislabel 45% of the training samples. The second additional test for MLSUP was for the checkerboard dataset, another synthetic, nonlinear problem. Similar to Iris, we achieve significant improvements using MLSUP. When the number of minima is increased substantially (e.g. to 20) the AUC only increases slightly. Training MI.UUP for 20 minima takes around 10 times as long as for four minima using the same setup above. These results again suggest that MLSUP can be effective for small number of local minima, with the potential to provide useful accuracy at a low computational cost. This balance of accuracy and efficiency could be valuable in applications where reasonable predictions are required without significant overhead.
Table 2: AUC improvements by MLSUP
AUC
Description
# minima
Best individual
MLSUP
TMA
Large hidden layer (n=100)
2
0.99
0.99
1
Small training set N=400
2
0.62
0.78
0.86
5 MLSUP minima
5
0.76
0.94
0.99
20 MLSUP minima
20
0.76
0.97
100 randomly chosen (mean)
2
0.74 (0.67)
0.7-0.89 (0.78)
0.98
100 randomly chosen (mean)
0.76 (0.67)
0.78-0.93 (0.84)
0.99
Mislabelled Iris flower dataset Checkerboard
2
0.85
0.96
0.99
2
0.68
0.84
0.96
7.2
PERMUTATIONAL INVARIANTS
7.2.
EXAMPLES
Here we provide additional insight into the issue of permutationally equivalent sets of weights.
60
1-23
50
1-3-
Euclidean distance
40
2-1-
30
2-3-1
20
3-1-2
10
3-2-1
0
1-2-3
1-3-2
2-1-3
2-3-1
3-1-2
3-2-1
Supplementary Figure 5: Euclidean distance between 48 permutationally equivalent local minima of the global minimum for a particular LFL.
13

---

Under review as a conference paper at ICLR 2022
Figure 5 shows how distant permutationally equivalent local minima be in Euclidean space. The system contains a single hidden layer with n nodes. Shuffling the nodes leads to n! permutations and within each permutation, the weights going in and out of any combination of the hidden nodes can be multiplied by -1. Thus, for each shuffle-permutttion there are an additional 2n 8 isomers, producing 48 in total. In misclassification space, the distance between all these 48 minima would be O, as in Figure which shows how far apart these equivalent solutions can appear be.
Minirrmm:
Miniram:
Minimamm: 21
Minimam::
Weight valne
-15
Supplementary Figure 6: An example of four very different looking minima that are actually permutationally equivalent, i.c. redundant solutions. For any input, these four networks will all return exactly the same loss value.
7.3 COMPARISON EUCLIDEAN AND MISCLASSIFICATION SPACE
This section presents further evidence that the Euclidean distance alone is insufficient for minima selection. Figure 7 shows that only a very weak correlation exits between Euclidean and misclas- sification distance. Large misclassification distance can be observed roughly often for points of large Euclidean distance as it is observed for points close together the parameter space defined by the weights. Given the good results of misclassification distance for minima selection, the low correlation coefficient indicates that Euclidean distance alone is not useful.
14

---

Under review as a conference paper at ICLR 2022
70
1.0
Misclassification distance
0.8
Log density
-0.6
10
0.4
30
0.2
0.0
Euclidean distance
Supplementary Figure 7: Euclidean Misclassification distance for each of the (*) combinations of n minima of the spiral data LFL. The Pearson correlation coefficient for Euclidean distance vs misclassification is 0.06.
7.4 MLSUP ALGORITHM
The MISUP algorithm be represented in pseudocode as:
Algorithm Train MLSUP
(X, c) Require: D
Sample LFL, find IM minima
Pick some [MM') minima candidates
Evaluate each minimum m [M'| on training data
Train second nn learn arg max
"p
for d E 'D do
Evaluate first network, query second network for which minimum to use end for
7.5 CHOOSING mn MINIMA BY PAIRWISE MISCLASSIFICATION DISTANCE
One may wish to compute MLSUP with m' minima, chosen to have the maximum misclassification distance from another. This is a combinatorial optimisation problem, analogous to having m cities with pairwise distances and trying find the m' cities that are furthest from each other. This problem can be solved by quadratic programming with convex relaxation. Let D be an upper triangular matrix with element D.y denoting the misclassification distance from point to point j. where < j. The square matrix D has 0 diagonal. Denote as m-dimensional binary vector, which contains the value at the position of a chosen minimum and 0 elsewhere. The objective then maximise Dr. In practice, the values of will be relaxed so they need not be discrete, and the optimisation procedure becomes
max y Dy subject to 0 y4
Vi, 1 1-y=m m
The final task is then to discretise back to such that y Dy is very close to x: Dz. We have achieved this mapping by simply setting the m' largest elements of y to and all others to 0, but other methods of approximation may work equally well even better.
15

---

Under review as a conference paper at ICLR 2022
7.6 HASINGHOPPING GLOBAL OPTIMISATION
The first step in surveying the LFL for a neural network is to locate the global minimum. We perform global optimisation using the basin-hopping method, which
Metropolis criterion to avoid getting stuck in local minima (Li & Scheraga 1987, Wales & Doye 1997). For local minimisation Sere--Goooaaabbaaaanaaa algorithm (Nocedal 1980). After a local basin of attraction (local minimum) is found via L-BFGS optimisation, "basin-hopping" jump is performed to some other point in the LFL. This jump always accepted if the energy, here loss value, is lower than the current minimum. If the loss value of the new point is higher, it is accepted with probability
we use modified quasi-Newton L-BFGS (Limited memoory Broyden-Fletee
AE kBT
P exp
(10)
where AE is the difference in loss value between the current minimum and the new point, kg the Boltzmann constant and fictitious temperature. Intuitively, if the energy difference between the old and new points is large, the move is less likely be accepted. A minimum is characterised as such if a RMS norm of gradient vector) convergence criterion at a threshold of 10-10 is reached. Each basin-hopping run naturally produces sampling of low-lying minima it progresses, addition the global minimum.
16


Reviews:
[{"review_id": "q14EiUPS4Bj", "reviewer": "Reviewer_yRSp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n\n## Pros\nThe motivation looks reasonable to me. \n\n## Cons\nThe paper is poorly written and I find it hard to follow the authors' ideas. The mathematical notations look very strange to me. For instance, the authors use $d$ to denote the input; in Eq. (2), the left superscript denotes the variables to be optimized. Moreover, I do not understand the so-called heat capacity in Section 4.3 and Section 5.2.2. The authors should have provided detailed formulations and explanations in the appendix. \n\nThe proposed methods make sense only for under-parameterized networks. However, deep neural networks often work in the over-parameterized regime in modern applications. I would like to know if the proposed under-parameterized ensemble method outperforms well-tuned over-parameterized nets or not. In addition, the authors did not provide comparisons to other deep ensemble methods, e.g., Snapshot Ensembles: Train 1, get M for free (https://arxiv.org/abs/1704.00109). Therefore, it is not clear about the effectiveness of the proposed method.\n\n\n## Other comments\n\n1. For the abstrct,\n    - LFL is introduced without providing the full name. \n    - \"by 20% for a complex learning problem\". I do not think that the spiral dataset is complex given the application of modern deep learning. In particular, this paper aims to propose efficient applicable methods instead of theoretical analysis. \n    - \"We propose a theoretical basis ...\" I did not see where the theoretical study is provided in this paper.\n\n2. For Section 1.2:\n   - MLSUP is first used without providing the full name. What is a \"proof-of-concept method\"?\n\n3. For Section 2:\n    - \"We employ a modified stacking approach where multiple minima from the same classifier are combined, rather than multiple classifiers\". I think that the combination of multiple minima has been extensively exploited in the deep learning community, e.g., the paper mentioned above. \n    - What is a \"functional landscape\"?\n4. For Section 3, Eq. (1), The mathematical formulation of the reguarlized risk seems wrong to me. $\\mathbf{W}^2$ is not defined. $p_{c^d}(W;X)\\to p_{c^d}(W;x_d)$.\n\n5. Eq. (2) and Eq. (3) are strange. \n\n6. I do not understand the purpose of Section 4.1. That the Euclidean distance is not a good metric to measure the difference in the high dimensional regime is well-known. I do not see any extra insight from this section. \n\n7. In Algorithm 1, for the second steps, how do you pick $|\\mathcal{M}'|$?\n\n", "summary_of_the_paper": "## Summary\nThis paper proposes a new ensemble learning based on neural networks.  First, A basin-hopping method is used to sample diverse minima of the empirical risk landscape. Then, a meta-network is trained to predict which minimum performs the best for specific input. This meta-network maps the input sample to the index set of minima. This can be viewed as an adaptive ensemble method, in the sense that for different input samples, the evaluated models are different. ", "summary_of_the_review": "See above", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VdHu-_lpri", "reviewer": "Reviewer_saVg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- I like the general idea of having a companion network to choose the best network out of an ensemble.\n- The two methods to select diverse minima are interesting and outperform the baselines.\n- The results of the proposed method on the spiral dataset seem rather encouraging, but lack further support.\n- The analysis of heat capacity looks like an interesting bridge with physics, but is poorly presented.\n\nWeaknesses\n- The spiral dataset considered is a very basic machine learning dataset, and is far from challenging enough to qualify as a proper deep learning task. The same goes for the Iris dataset used in the appendix. I understand that this dataset is convenient for visualisation of MLSUP (Fig. 4), and that not everyone can train on ImageNet due to computational constraints, but MNIST or CIFAR10 are a minimum to validate the method proposed. \n- The paper lacks comparison with concurrent ensembling methods, such as snapshot ensembles, stochastic weight averaging, etc.\n- The paper lacks clarity in parts. The physics jargon used in parts of the manuscript (heat capacity etc) is difficult to parse, I believe, for the machine learning audience. Sentences like “By computing the partial derivatives of the occupation probabilities with respect to temperature” need to be clarified or better, formalised mathematically. The Applications section 6.1 is also difficult to understand and does not add much value to the paper since the applications are not presented in the manuscript itself.\n- The experiments lack description : what network is used (width, activations, depth etc) ? What is the optimiser ? etc.\n- As acknowledged by the authors, MLSUP is effective for underparametrized networks, which specialise to parts of the data. However, Tab. 2 of the Appendix shows that MLSUP does not help for overparametrized networks. This is quite a strong limitation in my opinion, given that most networks used in practice are overparametrized, and should be stated more explicitly in the main text.\n\nComments :\n- The paper is framed as a theoretical exploration of loss landscapes, but seems to me more applied, as most of the content is devoted to the MLSUP method. Perhaps the authors should change the title accordingly.\n- “We use tanh as the nonlinear activation function to the hidden layer, since it has continuous derivatives, which we require for optimisation” : why ? This needs to be clarified.\n- “Different local minima specialise in distinct parts of the test dataset, which we believe has not been shown before” : such broad claims should be avoided. Obviously, different minima do not perform exactly the same on different parts of test data. The extent to which they specialise entirely depends on the dataset used : the one considers here is very specific and favours strong specialization. The question of ensemble-of-specialist networks has also been studied extensively, see for example the Multiple Choice Learning line of literature [1,2,3].\n[1] Guzman-Rivera et al 2012, Multiple Choice Learning: Learning to Produce Multiple Structured Outputs\n[2] Lee et al 2016, Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles \n[3] Lee et al 2017, Confident Multiple Choice Learning \n\nIn the abstract : \n- “that extract and process information from the input data to make outcome predictions” : this isn’t necessary\n- “We demonstrate that different local minima specialise in certain aspects of the learning problem, and process the input information differently” : too vague\n- “for a complex learning problem” : too vague\n- “multiple minima of the LFL” : LFL needs to be defined\n", "summary_of_the_paper": "This paper introduces a method for combining the expressive power of different minima of a given loss landscape, by using an auxiliary meta-network to select which minima is best suited to classify a given data point. They also introduce two methods to select diverse sets of weights : one based on misclassification distance, the other based on a heat capacity method from statistical physics.", "summary_of_the_review": "Overall, I think this paper presents an interesting set of ideas, but strongly lacks empirical support. The experimental section is poorly explained and is too simplistic to support the method presented (the only dataset considered in the main text is a spiral dataset). Also, the paper is difficult to parse in parts, not because of mathematical formalism but because of the use of physics jargon with a lack of clarifications. \nTherefore, in the current state, I would not recommend publication, but strongly encourage the authors to consolidate this interesting  research direction with more extensive experiments.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "GkgnBAP59_C", "reviewer": "Reviewer_XRAh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper studies an interesting topic of classifier diversity in ensemble learning. However, the presentation is a bit hard to follow and the empirical evidence are not strong enough to support the claims. Please see below for detailed comments:\n\n1. The writing is a bit sloppy and sometimes hard to follow. For example, in Equation (1), the right hand side \"X\" should probably be \"X^d\"? In Equation (3), tilda(W) is denoted as \"the weights for network 2\", but I had a hard time figuring out how is \"network 2\" constructed. Is it just a one layer linear classifier? What are the inputs and outputs? How is tilda(W) used?\n\n2. Empirical results are weak: Comparison to previous ensemble learning baselines are missing. In the main text, we only see an experiments on a 2D synthetic dataset. While synthetic dataset could be quite useful to illustrate the intuitions, it could also be misleading as the behaviors in high dimensional data could be quite different. So using this synthetic example as the main experiments makes the paper less convincing. \n\n3. This paper focus on tiny, under-parameterized networks, making it less clear how relevant the results presented here are to the literature. This paper argues that the rationale is that evaluating a large overparameterized networks could be a computational bottleneck. However, evaluating multiple under-parameterized networks with an ensembling method could also lead to extra computation overhead. It would be useful to back this argument up with an empirical comparison between single overparameterized models and ensemble of underparameterized models, for both performance and computation costs.\n", "summary_of_the_paper": "This paper studies a form of ensembling of under-parameterized networks. They show that different local minima specialize distinct subset of inputs, and an algorithm called MLSUP can be used to combine them to improve the prediction. The method is empirical tested with a 1-hidden-layer network on a 2D synthetic classification dataset (and in the appendix an artificially corrupted UCI Iris flower task).", "summary_of_the_review": "I found the presentation could be improved, and the experiment results are too weak to support the paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5EyW0ZfEG0n", "reviewer": "Reviewer_DYC3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper draws inspiration from physics and studies minima of loss landscape. They show that different local minima perform well on distinct subsets of data. It’s natural to wish to combine the strengths of these local minima. How to select and combine the local minima? The authors propose a framework and selection criteria. The recommended selection criteria are interesting. The authors share insights on why these criteria are better than the alternative approach based on Euclidean distance. I think the insights are useful. \n\nI feel it would be helpful if the authors can provide more details on how they compute the heat capacity. In section 4.3 the method description seems a bit vague. In the experiment, 7,842 minima were found,  then two were selected. The big gap may need some discussion. In the introduction, the authors mentioned low-lying and high-lying minima, and that low-lying non minima may be better than high-lying minima. There’s no further discussion in the rest of the paper, making parts of the intro seemingly disconnected.  \n\nOverall, although the proposed methods are interesting, the presentation and discussion need improvements.\n", "summary_of_the_paper": "The paper proposes a method to select a subset of local minima of loss function landscape and combine the minima using a second meta-network which learns to select a set of weights for each input data point. The subset of local minima is selected based on two criteria, minima that are distant in classification space and peak heat capacity. The combination of selected minima achieves better classification accuracy than the best individual and minima selected based on Euclidean distance.\n", "summary_of_the_review": "The authors adopt methods from physics and investigate local minima of  loss landscape. The methods are interesting and insights are useful. However, the current presentation is unsatisfactory and needs major improvements.  Sections of the paper seem disconnected or repetitive, and the methods sections lack details. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that the paper is below the acceptance threshold and the authors did not respond to the reviews.
In summary, this is a clear reject
Decision: Reject