Density of states in neural networks: an in-depth exploration of learning in parameter space
Authors: Margherita Mele
Roberto Menichetti
Alessandro Ingrosso
Raffaello Potestio
Venue: Accepted by TMLR ()

Abstract: Learning in neural networks critically hinges on the intricate geometry of the loss landscape associated with a given task. Traditionally, most research has focused on finding specific weight configurations that minimize the loss. In this work, born from the cross-fertilization of machine learning and theoretical soft matter physics, we introduce a novel approach to examine the weight space across all loss values. Employing the Wang-Landau enhanced sampling algorithm, we explore the neural network density of states -- the number of network parameter configurations that produce a given loss value -- and analyze how it depends on specific features of the training set. Using both real-world and synthetic data, we quantitatively elucidate the relation between data structure and network density of states across different sizes and depths of binary-state networks. This work presents and illustrates a novel, informative analysis method that aims at paving the way for a better understanding of the interplay between structured data and the networks that process, learn, and generate them.

Published in Transactions on Machine Learning Research (02/2025)
Density of states in neural networks: an in-depth exploration of learning in parameter space
Margherita Mele
mergherita. @uminti.
Physics Department, University of Trento, via Sommarive, 14 1-38123 Trento, Italy
ANEN-TIFPE Trento Institute for Fundamental Physics and Applications, 1-38123 Trento, Italy
Roberto Menichetti
mbberto.menncchett@@@ann..
Physics Department, University of Trento, via Sommarive, 14 1-38128 Trento, Italy INFN-TIFPA, Trento Institute for Fundamental Physics and Applications, 1-38123 Trento, Italy
Alessandro Ingrosso
alessandro alessandro.nggoossoooo wngrosso Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
Raffaello Potestio
Selfaelllo.oteestee Potitoioointttii @anitn.
Physics Department, Universily of Trento, via Sommorive, 14 1-38123 Trendo, Itayy
INFN-TIFPA, Trento Instätute for Fundamental Physics and Applicntions, T-38123 Trento, Itaby
Reviewed on OpenReview: https: // openreview. net/ forum? id= BLDtWLFKAm
Abstract
Learning in neural networks critically hinges on the intricate geometry of the loss landscape associated with given task. Traditionally, most research has focused finding specific weight configurations that minimize the loss. In this work, born from the cross- coss-fertilization of machine learning and theoretical soft matter physics, we introduce novel approach to examine the weight space across all loss values. Employing the Wang-Landau enhanced sampling algorithm, we explore the neural network density of states the number of net- work parameter configurations that produce given loss value and analyze how it depends on specific features of the training set. Using both real-world and synthetic data, we quan- titatively elucidate the relation between data structure and network density of states across different sizes and depths of binary-state networks. This work presents and illustrates novel, informative analysis method that aims at paving the way for better understanding of the interplay between structured data and the networks that process, learn, and generate them.
1 Introduction
The history of machine learning is deeply intertwined with that of modern statistical mechanics. Since the early days of neural networks, both algorithm development and theoretical investigation have leveraged concepts and tools derived from the equilibrium statistical physics of disordered systems.
A prominent example is Gardner's pioneering work (Gardner & Derrida 1988 Gardner 1988), which framed the problem learning the weights a perceptron the simplest single- lingle-layer neural network as that of an oquilibrium system in the presence of disorder inducod by random inputs and outputs. Virtually everything that followed, from optimization techniques for training network the understanding of its generalization capabilities, has been addressed by the statistical physics community using the Gardner framework.

---

Published in Transactions on Machine Learning Research (02/2025)
This long-lasting approach has been highly successful in characterizing the ability of simple neural net- work architectures to both memorize random input-output associations and generalize from previous exam- ples (Gardner& & Derrida 1989 Seung et al. 1992 Watkin et al.
1993 Sompolinsky et al. 1990). Extensions Franz & Parisi 1995) can also connect
of the formalism used to study metastable states in complex systems
the local geometry of the loss landscape (particularly its flatness)
to the generalization abilities of solutions
found by actual learning procedures (Baldassi et al. 2015; 2016b; 2021), even when the dynamics of such algorithms does not necessarily satisfy detailed balance (Baldassi et al. 2016a).
The theoretical investigation of neural networks has greatly benefited from the mathematical toolbox devel- oped in the context of spin glass theory (Mézard et al.) 1987 Engel & Van den Broeck, 2001 Charbonneau et al. 2023). These methods, however, classically relied on simplifying assumptions such as the thermoo dynamic limit or well-defined, mathematically tractable input data, generated as independent, identically distributed (i.i.d.) variables. Thanks to these simplifications, researchers obtained a plethora of exact or approximate results that shed light on the inner workings of neural networks. A strong limitation, however, is the fact that real data are typically finite-sized and structured, thus defying the founding assumptions upon which those results are built.
More recently, novel approaches have been developed that explicitly incorporate data structure into the the- ory of deep neural net net works, e.g., by using the second-order moments of input data (Opper & Winther 2001
Shinzato & Kabashima 2008: Ingrosso 2021j, or employing tasks involving effective Gaussian models and
mixtures of Gaussians [Gerace el al. 2024 Mignacco et al. 2020 Gerace et al. 2021 Loureiro et al. 202 4a by 2023). Recent work has studied the importance of higher-ordes input statistics for feature learning in neural
networks (Ingrosso & Goldt 2022), but the impact of such properties on the dynamics of gradient-based
learning, and more generally network behavior, is only starting to be investigated systematicaaly (Fischer
ct al. 2022 Refinetti et al. 2023 Belrose et al., 2024).
The approaches mentioned thus far very often focus on the solvability of a learning problem, that is, the identification of those configurations of network weights that minimize the loss function. Phrasing this task in physical terms, one explores a weight, configuration space equipped with a canonical probability measure, and investigates the properties of the ground states by varying the number and features of the inputs and controlling for their statistics.
In summary, a great deal of work has been carried out insofar that mainly assumes infinite, unstructured input data and focuses on tiny fraction of the weight configuration space that is, the part that minimizes the loss; in comparison, much less has been done to understand how finite-sized networks behave in the presence of structured inputs. A large body of information might nonetheless be retrieved through the st study of such networks not only in terms of their solutions but also investigating the learning task globally in configuration space, by analyzing how the entire loss spectrum, defined on all possible weight configurations, is affected by given task.
In this work, we address this problem by introducing a method to efficiently explore the entire density of states (DoS) of the loss function of a neural network. We achieve this by leveraging the Wang-Landan Landau (WL) algorithm, a powerful enhanced sampling method used to compute the density of states of physical systems
and access exponentially suppressed regions of the configuration space (Wang & Landau 2001a b Yin &
Landau 2012 Vogel et al., [014 Farris & Landau 2021). By adaptively estimating the microcanonical ent.royy of the loss (that is, the logarithm of the density of states), the WL approach allows us to uniformly explore the loss spectrum, enabling thermodynamic characterization of neural network and its behavior
for given training datasct.
We show how the statistics and geometrical structure of realistic input data have a direct and measurable impact on the density of states of the loss function; we then highlight how specific properties can be repro- duced by simple synthetic datasets explicitly controlled by few intuitive parameters, and thereby rationalize the relationship between data structure and loss density of states. In perspective, we propose that the ap- proach here illustrated can be employed in a twofold manner, either to analyze large datasets of annotated instances, or to design networks to perform better on particular problems.
2

---

Published in Transactions on Machine Learning Research (02/2025)
The paper is organized as follows. In the Methods section, we introduce the problem of learning in single- and one-hidden-layer neural networks focusing on the case of discrete synaptic weights, and describe the dataset used in our simulations; we then provide brief introduction to Wang-Landau sampling. In the ensuing Results section, we first use the WL technique on real datasets, and then introduce a number of simpler synthetic classification problems. In the Discussion and Conclusions section, we discuss our findings and highlight potential generalizations of the approach.
2 Methods
2.1 Neural Network Architectures
A single-layer binary neural network, also known as perceptron, operates by constructing a hyperplane within the input space to achieve classification, Mathematically, it implements a mapping from an N-dimensional input vector & E IR to a binary output using the function o(f:W) =ggnWW E), where W is a (synaptic) weight vector.
In this work, we focus on networks with binary weights W E (-1,1}%. Given set of P aN patterns [ and their corresponding labels o# E {-1,,1),, the perceptron learning problem consists in finding an appropriate weight vector W satisfying o(W,"") = ṭt for all μ. i.. a perceptron that correctly classifies its inputs according to the prescribed labels.
The first step in formulating statistical mechanical theory for perceptron learning is to define an energy function Ew E (W) Σn= (--o"W g*), where ((x) is the Heaviside step function, counting the number of incorrect input-output associations produced by perceptron with a given weight vector W.
Here, the term eneryy is borrowed from the analogy to physical spin systems, where each spin can be in one of two states (up or down), analogous to binary weights W; E (-1,1) in our neural network. The energy in these systems represents the degree of misalignment among the spins. Similarly, in our model the energy represents the number of misclassificationss, and minimizing this energy corresponds to reducing these classification errors. A perfect configuration where all patterns are correctly classified corresponds to the "ground state" of the system, akin to achieving perfect alignment in ferromagnetic spin system. Furthermore, due to the binary nature of the weights and the use of the sign activation function, the energy satisfies the relation E (W) = P E(W) for any set of weights W and its negation W = -W. Consequently, the amount of states for energy levels equidistant from the center of the spectrum will be identical by construction.
The learning objective can be formalized as that of finding a ground state configuration, or solution, i.c. a weight vector W such that the energy E(W) is minimized. In a simplified scenario with random inputs and outputs, the probability of finding zero energy solution decreases as the number of input patterns increases, i.e. the entropy of weight vectors with zero energy approaches zero critical capacity. In such a setting, the application of replica and cavity methods borrowed from the statistical mechanics of spin glasses has been contral in studying the storage capacity and generalization propertics of neural networks (Gardner & Derrida 1988 Krauth & Mezard 1987 Mézard et al., 1987 Engel & Van den Charbonneau et al. 2023).
P/N grows up to a
Broeck 2001
Training a perceptron involves adjusting the weights to minimize classification errors. When constrained to binary weights, the problem becomes significantly more challenging than the naatinuous-weighte counterpart. The use of cavity methods has been instrumental in developing powerful algorithms able to solve typical
problem instances [Braunstein & Zecchina 2006 Baldassi et al. 2007) almost up to the critical perceptron capacity.
Multi-layer architectures extend the capabilities of a perceptron to non-linear classification in input space by stacking many of these simple units in consecutive layers. In this work, we will analyze one-hidden- layer architectures, comprising an N-dimensional input layer, a hidden layer with Nu neurons, and an No dimeieeniinnal output layer for generic multi-class classification. An input vector is initially mapped to hidden layer via the weight matrix W E (-1,11)MxN yiclding hy = =gn(E, WhS)) for the k-th hidden neuron's activation. Subsequently, the hidden layer activations h (h+ No are linearly mapped to the
3

---

Published in Transactions on Machine Learning Research (02/2025)
output layer using W2 E (-1,1) The final output is computed by @W2h), where & can be either the sign (when =) 1) or the argmax function, respectively for binary or multi-class classification.
For a set of P = a.N patterns { and their corresponding labels ot, we define the energy associated with weight matrices W and W as E (W", W²) = En- (1 5 [o" (W2sgn (W), with 8 a Kronecker delta function. Again, the learning task involves finding appropriate weight matrices Wi and W such that the output matches the labels for all patterns, minimizing the energy E (w²' .,/) to zero.
Symmetries also exist in the one-hidden-layer network, but binary and multi-class classifications must be considered separately, For binary classification, the system displays a symmetry with respect the middle of the energetic spectrum, as it was the case for the perceptron: if both weight matrices W and W are simullaneuusy negated, the energy function remains unchanged, so that the configurations (W1, W2) and (-W!, WW3) produce the same energy value, E (W1, W²) E (-WW,, -W?). In contrast, negating only one matrix results in energy value that is symmetric around the middle of the energetic spectrum:
E(-W", >) = E(W), (WW] = P - E(W), W²). This symmetry arises directly from the binary nature of the sign output activation function, and it is lost when an argmax function is used in the case of multiple
labels.
2.2 Datasets
In this work, address the learning problem for single and one-hiddeen-laer networks in different scenarios involving both benchmark datasets and synthetically generated tasks.
The first sconario involves real-world datascts. The classification task is performed for various sizes of the training set a = P/N, where a is gradually increased from 0.1 to 1.0 by randomly adding examples from both classes. As result, for a given total number of inputs P, there might be a discrepancy in the number of examples between the two classos (Po F Pi): this condition is commonly referred to as "unbalanced" dataset. By contrast, a "balanced" situation entails equally populated classes (Po P1). In the main text, we show results obtained on the classic MNIST dataset (TeCun et al., 1998). focusing specifically on binary classification problems involving only the first two classes (0 and 1). Additional classification problems on the MNIST dataset are reported in the supporting material (see Supplementary Section V), along with results obtained using MaSSI (Xiao et al. 20170.
In the second scenario, we use a series of approximations of an original real-world dataset that progressively capture higher-order moments of each class. In particular, construct Gaussian clones of the datasets, by sampling inputs from a mixture of Gaussians each capturing the mean vector and covariance matrix of the images in that class. We call this configuration GM (Gaussian mixture), where inputs possess the correct mean and covariance per class but where all higher-order cumulants are zero. We also compare the results on a simplified isotropic Gaussian mixture (MisoGM), where only the mean of each class has been preserved and the pixel-wise variance of the two classes equal to the geometric mean of the real variances.
Finally, the last two scenarios involve symthelic datasets. We gather hereafter some basic information on the gata-gene process; all technical details can be found in the supplementary material (see Supplementary Section 1). In the first case, each class represents isotropic Gaussian distribution of points in N dimensions. Two controlling parameters are employed: the inter-class separation, defined as the norm of the difference between the mean vectors of the two classes, and the angle between these two mean vectors. In the second case, we consider a perceptron learning problem with randomly generated data in a teacher-student setup. Each input component 4 is drawn i.i.d. at random with probability 1/2 in {-1,1}, and the outputs 6º are generated using a teacher vector WT as o² sgn(Wr {"), implying that data points are linearly separable by construction as the solution Wy always exists.
2.3 Wang Landau Algorithm
The estimation of the density of states 2(E) E(n 8(Ew E), namely the number of all internal configurations of a network with energy E, poses a significant challenge. In an unbiased random walk, i.c., stochastically changing the states of the network and accepting all moves regardless of the energy values
4

---

Published in Transactions on Machine Learning Research (02/2025)
obtained, the histogram of the energies sampled along the simulation converges up to normalization factor - to the density of states ()E)) in the limit of very long walk. However, achieving such an extensive exploration over the whole energy spectrum is extremely difficult with our current computational resources, given the extremely large number of weight configurations. For example, a perceptron with 100 inputs with binary weights has 2100 = 1.3 x 1000 possible configurations; by assuming a previously unvisited state is sampled on average every nanosecond, would take more than 1013 years to explore the entire configuration space.
The sampling protocol proposed by Wang and Landau (WL) offers an efficient approach to estimating the density of states of system (Wang & Landau 2001a b Shell et al. 2002 Barash et al.] 2017). More specifically, in WL sampling S2(E) is self-consistently determined through a series of random walks in eneryy space, in which the transition to a new configuration W with energy E is only accepted with a probability that is proportional to the reciprocal of the density of states (2))). Critically, the system's DoS is a priori unknown; the power of the WL protocol resides in its ability to reconstruct (?(E) via a sequence == 0,.., K of non-equilibrium Monte Carlo (MC) simulations that provide increasingly accurate approximations to the exact result.
Given the discrete spectrum of possible energy values E, the main ingredients of the WL iterative scheme are, respectively: (i) the runming MC estimate of the density of states ((F); (ii) the rumning histogram of visited energy levels at iteration k, Hs(E); and (iii) the multiplicative modification factor fa governing the overall convergence of the algorithm. For computational purposes, to fit large numbers into double precision variables is convenient to work with the entropy S(E) = In[[(E),, further replacing the multiplicative control parameter f& with an additive one F& == In(fx)- Another key ingredient of WL simulations is the proposal probability governing the way in which the space of weights is explored. In essence, this rule serves as the mechanism employed by the algorithm to propose new state starting from the current one. In this work, we relied on combination of global and local moves (see Supplementary Section II).
Let us now describe the first step (k = 0) of the self-consistent scheme, the following iterations being simply repetitions of this process in which the modification factor Fo is replaced by its updated version Fs. as detailed below. As stated previously,
the beginning of the workflow ((E) a priorí unknown. A commonly employed guess is to set (?E) 1, cquivalently S(E) 0, for all energies E. The histogram is set to zero, HE(E)= O, and the control parameter to 1, Fo 1. Subsequently, a series of MC moves is performed in which consecutive transitions between two network states W and W', respectively, with energy E = Ew and E" = Ew,, are proposed by flipping a subset of the weight vector components. Each of such moves accepted with probability:
A(W + w') = min 1, !(E)
((EE))
= =min ex, (eexxpp(((EE)))))) -(S((()-
(1)
Equation implies that if ((F') < ((F),, the state W with energy E' is accepted; otherwise it is accepted with a probability ((E)/((E)) We note that if the exact density of states was employed in Equation the associated Markov chain would, after an initial transient, generate configurations distributed as I/{(E), hence resulting in a flat histogram (E)). Departures from this flat histogram condition are associated with deviations of the running density of states from the true one.
If the move W + W" is accepted, the running histogram Ho(E') and entropy S(E') are updated according to
Ho(E') Ho(E') 1, S(F') S(F') Fo-
(2)
In case of rejection, replace E' with E the previous equations. By gradually increasing the entropy of the energy states that were previously observed and a consequence of Equation 1 it follows that, during its
5

---

Published in Transactions on Machine Learning Research (02/2025)
initial phases, the WL scheme tends to "push away" the sampling from already visited regions of the weight space, thus significantly boosting its exploration compared to randomly drawing network configurations. Critically, this initial random walk in energy space is continued until the histogram Ho(L) becomes "flat"; at this stage, the rumning density of states matches the true value with accuracy proportional to Fu = - =1 Subsequently, iteratively, the modification factor is reduced following the rule Fx+1 = Fx/2 (/x+1 = VTx,, the histogram is reset to zero, (E) = O, and the next random walk in energy starts, the latter being interrupted when the flatness of Hx+1 is achieved. This procedure repeated for several steps until the modification factor is smaller than a predefined final value Fend In fend.
The Wang-Landau (WL) scheme produces a density of states compliant with the true value Balne-witee with an accuracy lower-bounded by fend up to an overall multiplicative constant Cg. In our specific application, given the finite and predetermined boundaries of the energy range and the discrete nature of the system, is possible to determine the global constant through simulations that cover the entire energy spectrum. More specifically, it is sufficient to ensure that the area under the DoS curve equals the number of possible configurations available to the system. For single-layer perceptron with N neurons, this is 2%; for one-hidden layer network with N imput neurons, Nn hidden neurons, and No output neurons, this is 2 NN A validation of the Wang-Landau sampling and a convergence time scaling analysis can be found in the supplementary material (see Supplementary Section III).
3 Results
Deep learning rescarchers are usually interested in the minimizers of the loss function. In this work, we broaden this perspective to the whole loss range, investigating how the number of parameter configurations associated with each value of the loss - that is, the densily of states - depends on the properties of the network and its training set. Several interconnected factors, among which network architecture, activation functions, and statistical structure of the input, play a critical role in shaping the loss DoS. Rationalizing this interplay bears twofold consequences: on the one hand, it provides insight into the properties of the dataset itself; on the other hand, it can help us in understanding the inner working of task-optimized neural networks. In the following, report the results of our work in one of these directions, our main goal being to elucidate how data structure impacts the density of states.
In the first part of this section, we apply our method to a benchmark dataset commonly employed in supervised learning. We examine both a perceptron and a one-hidden layer architecture solving classification tasks. A drawback in the theoretical analysis of real-world datasets is that the probability distribution underlying the input data is generally not known: to make sense of our results, thus turn to analyze synthetic datascts whose generative model is controllable with a small set of paramcters, recovering the same phenomenooogy observed in the case of realistic inpul data.
3.1
DoS analysis of benchmark tasks
We start off addressing the problem of computing the DoS for simple networks applied to both random and structured input data. We analyze both a binary, single-layer (SL) perceptron and a network with one hidden layer with varying size. The latter architecture is employed for binary classification as well as multi-class labeling.
Figure 1 shows the DoS for the aforementioned architcetures (panel (a)) and classification tasks, demon- strating how various factors modify its properties. Panels (b) and (c) show the DoS in perceptron solving a binary classification task on both random (Fiuure 1b) and real data (FFigure lc). We also show the results obtained in one-hidden layer neural network with an increasing number of neurons N₁, in the hidden layer,
for binary [Fiuure I.d) and multi-class classification on the MNIST dataset Figure 1e)
Panels (b), (c), and (d) illustrate that, in the case of binary classification, the DoS curves are symmetric with respect the mid-energy value (E = 0.5) irrespoctive of the network architecture. As anticipated, the symmetry is induced by the sign activation function in conjunction with the binary nature of the weights, i.c.
W = (-1,1) Such symmetry is lost. in the presence of multi-class classification with an argmax function
6

---

Published in Transactions on Machine Learning Research (02/2025)
b
Single-layer perceptron
N
P/N = 0.1
N=251
= 784
P=
N-501
P-784
W;
egn
Wi
8
DoS
One-hidden layer network
3-
Binary classilication
N
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
W
A
W
ENERGY/P
ENERGY/P
sgn
We
N= 100
12
N 100
140
=
200
N, 15
10
Multiple classification
No
N. 51
N
N
No = 101
W
DoS
DO5
W
A
WA
Argmna
0!!
2
My - 20
Wi
A
W
ON
0.0 0.2
0.4
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
ENERGY/P
ENERGY/P
Figure 1: Comparative analysis of the density of states (DoS) in classification tasks of random or structured data across different network architectures and input dimensions. (a) Schematic representations of the net- work architectures: single-layer perceptron (top), and onc-hidden layer network for both binary classification (middle) and multiple classifications (bottom), illustrating the input layer (N), hidden layer (N₂), and out- put layer (N.), along with their respective weight connections. (b) Density of states for binary classification of random input data, in teacher-student setup, using single-layer perceptron. binary classification of the first two classes of the MNIST dataset with N = 784 input dimensions by single- layer perceptron, illustrating results for different learning complexities (P/N = 1.0 and P/N 0.1). (d) Density of states for binary classification of the first two classes of the MNIST dataset with N 100 input dimensions, with a fixed data size P = 140 and varying network architectures, including single-layer (SL) perceptron and one-hidden layer nelwork with different numbers of hidden layer neurons of states for multi-class classification of the MNIST dataset with 100 input dimensions, showing results for varying numbers of hidden neurons (Nr) in the onehiidden layer network.
(c) Density of states for
(N₂). (e) Density

---

Published in Transactions on Machine Learning Research (02/2025)
(sce Figure 1 e), where instead the DoS peaks at a value of the energy corresponding to the typical error - 1/CC obtained when guessing C' possible classes with uniform probability.
We now compare the characteristics of the DoS curves for binary classification via a perceptron in two distinct scenarios: i..d. data labeled by a teacher network Figure 1b) and structured data, exemplified by the classification of the first two MNIST classes (Figure l(c). Both curves exhibit a maximum at the central energy value (E' 0.5), indicating that most network configurations incorrectly label half of the input data. However, the shape of the distribution is markedly affected by the type of data used in the binary classification. For i..d. data, the density of states follows Gaussian distribution centered at 0.5 (see Supplementary Section IV). For structured data, the DoS decrcases almost lincarly away from the center of the energy range.
The non- non-Gaussian shape of the DoS is maintained even when the network architecture is modified by adding a hidden layer (Figure [c.. Specifically, we used a onc-hidden layer network with varying inner layer sizes to perform binary classilication of the first. two MNIST classes (reshaped into N = 100 dimensions, i.e., 10 x 10 instead of the original 28 x 28). We note that the general properties of the curve, symmetries and shape, are preserved. The DoS obtained with perceptron, taken as reference, is different from the one obtained with dataset of larger-sized images; in particular, this curve has a sharper central peak, from which a slower decrease is observed while moving towards lower and higher energy values. This notwithstanding, as the number of hidden layers is increased, the DoS curve gradually converges to a shape qualitatively much more similar to the one observed in the case of Figure 1 lb (SL perceptron, MINST dataset with images of size 28 x 28).
Taken together, these findings suggest that the shape of the DoS is inffuenced by the architecture of the network as well as by the structure of the input data, in a nontrivial interplay between the two. In order to better understand the origin of specific properties of the DoS, we thus modulated some features of the underlying data and investigated the relationship these bear with the shape of the resulting curve.
Firstly, we studied the effect of class imbalance on MNIST dataset, Lee. the discrepancy between the number of data points {Po, Pi} in the two classes, while keeping the total number of points p = P1 + Po fixed. To assoss the impact of class imbalance on the dataset, the DoS curves of the binary classification were computed for different values of P1, thus including scenarios where class 1 was either predominant (P1 > Po) or in the minority (Pi <. Po). Figure 2 shows that as class imbalance is increased, the DoS peak shifts away from the center of the spectrum.
Irrespective of which class is predominant, we observe the appearance of two peaks that shift from the center as the imbalance increases, due to the symmetry of the DoS curve. When P1 is significantly greater than Po (green curves), the DoS shows pronounced peaks away from the center, indicating higher concentration of states at low /oihh energy values. As P1 approaches P/2 (red curve) from above, the DoS curves become more centralized, indicating more balanced distribution of states across the energy spectrum. When P. is less than P/2 (and thus less than Po, blue curves), a pattern similar, albeit mirrored, to the first case emerges, with the peaks moving away from the center as P1 is decreased.
This behavior can be understood by noting that the two symmetrical peaks in the DoS correspond to distinct sets of solutions arising from the class imbalance. The low-energy peak represents configurations where the decision boundary optimally classifies the majority class, while the minority class contributes most of the errors (and vice versa for the high-energy peak). This duality is natural consequence of the binary weights of the perceptron and the definition of energy, which symmetrically measures the misclassifications of the two classes.
The shifting of the peaks described above is clearly reflected in a scatter plot Figure 2) that correlates the position of the low-energy maximum of the DoS curve with the degree of unbalancing, computed as by - PP/Pl The plot shows results for two distinct values of dataset size, a = 0.1 (squares) and a = 1 (diamonds). As the degree of imbalance increases, the peak energy shifts proportionally, thereby providing quantifiable mcasure of the impact of class distríbution on the density of states.
These results show how class imbalance plays crucial role in shaping the DoS, in a way that is further dependent on the structure of the input data; most importantly, class imbalance does not change the location
8

---

Published in Transactions on Machine Learning Research (02/2025)
b
C
P1 > P
P1 =
P: =57
PIN=0.1
P/N=1
P1 = 21
P
PI Po
P1
P - 39
Pi
2.5
P/N 78/784
0.5
P1 =
2.0
PEAK ENERGY
1.5
0.3
Do5
1.0
0.2
Pi Po
0.5
0.0
0.0
0.2
0.4
1.0
0.0
0.4
ENERGY/P
|0.5 Ps/PI
Density of states analysis for binary classification of MNIST digits (0 and 1) under various class imbalances. (a) Schematic representation of class imbalance. In the top section of the panel, P1 Po. there more elements in class (orange) than in class 0 (blue). In the middle section, P: Po, the classes balanced. In the bottom section, Pi < Po, there are fewer elements in class than in class 0. The total mumber of points Pi Po = P is fixed. (b) Density of states at a fixed learning complexity of P/N = 78/784 resulting in MNIST binary classification for different class imbalances. The legend indicates the number of clements class 1 (P:). Perfect class balance is achieved when P1 = Po = P/2 = 39 (red curve). Larger deviations from this value indicate greater class imbalance. Blue curves represent predominance of class 0, while green curves represent a predominance of class 1; the lighter the color the greater the class imbalance. (c) Peak energy values plotted against the absolute difference |0.5 Pi Pi/Pl, showing the correlation between peak energy and class inbalance. This correlation does not depend predominant: blue points indicate class ( is predominant, and green points indicate class 1 is predominant. Results are shown for two values of learning complexity: P/N = 0.1 (squares) and P/N = 1.0 (diamonds).
Figure 2:
which class is
of the DoS peak for randomly distributed imput: observe such behavior only in the presence of structured data. This conclusion is supported by further analyzes we carried out on the MNIST and Fashion-MNIST datasets (see Supplementary Figure 4).
3.2
DoS analysis of synthetic tasks
To further elucidate the factors impacting the density of states, studiod simple classification problems synthelic datasets, designed in such a way that systematic study can be carried out over a small number of control parameters.
3.2.1 DoS in a classification problem with data from a mixture of isotropic Gaussians
Here, we focus on the DoS obtained from input data sampled from Ganssian probability densitics and study its changes we modify specific parameters of these distributions. We construct binary classification task where the different classes consist of isotropic clouds of points in tigh-dimensional space (we set N 41). The distributions for each class are Gaussians with zero mean and unit variance (further details can be found in supplementary material Section I). We systematically vary the inter-class distance Aμ, defined as the distance between the mean points of the distributions, as well as the angle 0 between the mean vectors.
Figure b shows the DoS curves obtained by varying the value of Aμ, the simple case of 180 We expect that a large value of Ap is associated to rather trivial classification tasks, since the two groups of points neatly distinet and, consequently, great number of planes in input space successfully separate
9

---

Published in Transactions on Machine Learning Research (02/2025)
them. In contrast, Au close zero corresponds to overlapping clouds: in this latter case, no plane likely exists that allows a distinction between them.
This expectation is indeed confirmed by the behaviour of the density of states: the latter is bell-shaped, with a peak at the center of the spectrum when Ap is small, implying that the vast majority of network parameter configurations wrongly classify the inputs 50% of the times. As Âμ increases, the DoS broadens significantly: for sufficiently high values of Ay the DoS develops two distinct peaks at the lowest and highest energy values, implying that most parameter sets correspond to classification planes that correctly label the data (or, alternatively, that make the wrong labeling all the times, which is equivalent to perfect labeling up to sign).
In Figure 3/c we show the position of the DoS peaks as a function of Ap (0 = 180"). The orange curve represents the balanced case (Pi = Po). while the grey points correspond to scenarios with class imbalance (Pi * Po). In this context, class imbalance does not affect the location of the DoS peaks: the trend is indood consistent for both balanced and unbalanced scenarios.
We then extend our analysis to synthetic dataset where both the angle and inter-class distance are varied systematically; we focus in particular on the two different regimes of small and large inter-class distances. Results are summarized in Figure 3 For large inter-class distances (e.g. = 2) and low angles (e.g. 0 = 18)), the DoS curves exhibit several properties observed in real datasets (Figure 2]:): the curves lose their typical bell shape, with the location of the maximum density being controlled by class imbalance. When P1 P/2- = 30, the density of states (red curve) displays a single peak at the center of the energy spectrum. As P1 deviates from P/2, with larger values marked in green and smaller values in blue, the peak shifts away from the center. The peak location approaches the tail of the spectrum P1 deviates further from the balanced case.
The location of the maximum density of states reveals two distinct regimes based on the angle between the mcan vectors: > 90 and 0 < 90% In the first regime, class imbalance docs not affect the peak location of the DoS curve. In contrast, for smaller angles, it plays a crucial role in determining the position of the peak, which shifts systematically towards the edges as the imbalance increases. The influence of class imbalance the peak location is diminished when the inter-class separation is small. For small Auu, the maximum of the DoS remains centered in the energy spectrum, regardless of the angle between the mean vectors.
The results discussed thus far demonstrate that synthetic datasets have considerable power in replicating the behavior observed in real-world cases. This provides valuable insights in the way neural networks carry out classification tasks: particularly, in the context of binary labeling involving MNIST data, we observe that the angle between the inter-class mean vectors generally falls below 90º (see Supplementary Figure 5.a); such observation aligns with the results obtained on synthetic data, where class imbalance significantly influences the location of the peak of the DoS curves only for small angles, as it is the case for MNIST data.
3.2.2 DoS of a classification problem with mislabeled data
Label noise can affect the training performance and robustness of neural networks. Here we investigate how the DoS is affected by the presence of mislabeling, i.e., the attribution of a given data point to the wrong class, which we can easily model in a synthetic dataset by switching the label of subsets of points generated from a mixture of Gaussians.
Specifically, we analyze a setup where some input patterns in the dataset are mislabeled. Fach class is an isotropic Gaussian cloud of points, with the two mcan vectors identifying the classes as antiparalicl (0 = 180"); one cloud has a mean vector m and is labeled as +1, while the other cloud has a mean vector -m and is labeled as -1. We vary both the inter-class separation Au and the number of mislabeled elements L. To achieve the latter, a set of elements L is chosen at random, and their labels are reversed so that points from the cloud with mean m are labeled as -1, and/or points from the cloud with mean -m are labeled as +1. The parameter I. ranges from 0, indicating all points are correctly labeled, to P/2, representing the maximum degree of disorder where half of the points are mislabeled.
Figure 4 shows that, as in previous scenarios, the peak of the DoS remains centrally located for any L value when the classes are close to each other (Ap- 1). However, as the inter-class separation increases, the
10

---

Published in Transactions on Machine Learning Research (02/2025)
b
Inter-class separation
Aμ 0.2
4==1.0
Au 0.5
Au 1.5
Small Ay
Big Aw
Au 0.7
60/41
1.0
PEAK ENERGY
Inter-class mean angle
0.6
0 < 90'
91'
0.4
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.25
0.50
0.75
1.00 Aμ
1.25
1.50
ENERGY/P
d
P. 10
40
Pi
=0
P: 50
Pi 10
P5=50
30
Pi
20
Pi 30
P.
PIN 60/41
Au = 2.0
1.0 x 90"
1.0 9 90
12
18"
0.8
10
PEAK ENERGY
PEAK ENERGY
0.6
0.6
Do5
1 11111
0.2
0.2
0.0
0.2
0.8
0.3 0.7
0.3
0.7
ENERGY/P
Aμ
Au
Figure Density of states analysis for binary classification of synthelic datasets studying the impact of inter-class separation (Aμ) and inter-class mean angle (0). The angle between the two mean vectors uniformly ranges from 180 to 18 (a) Pictorial representations of inter-class separation (Ap) at the top and inter-class angle (0) at the bottom. (b) Density of States at fixed learning complexity of P/N = 60/41 obtained in the binary classification for multiple inter-class separations Aμ and fixed inter-class angle (0 180%) for balanced classes. The legend indicates that lighter colors correspond to larger inter-class distances. (c) Peak energy values plotted against the inter-class distance Ap at = 180 for balanced classes (orange) and imbalanced classes (gray). The plotted values represent the average from 20 independent replicas of the system in the same configuration, with associated statistical deviations. (d) DoS at fixed learning complexity of P/N 60/41 obtained in the binary classification for large inter-class separations (Aμ= 2.0) and small inter-class mean angle (Ø 189) for different class imbalances. The legend indicates the number of elements in class 1 (P:). Perfect class balance is achieved when Pi Po P/2 = 30 (red curve). Larger deviations from this value indicate greater class imbalance. Blue curves represent a predominance of class 0, while green curves represent a predominance class 1; lighter colors indicate greater class imbalance. (e) Peak energy values plotted against the inter-class distance (Ap) for different values of class imbalance as reported in the legend. Red points denote balanced classes. Blue and green points indicate unbalancod classes with predominance of class 0 and 1, respectively; lighter colors indicate greater imbalance. The two panels differ by the value of the inter-class mean angle, with 90" on the left and 0 > 90% the right.
11

---

Published in Transactions on Machine Learning Research (02/202))
b
L 0
L=0
L-25
L=0
L=
+1
L=5
L-30
L=5
L-25
L=15
L=10
L=15
L=30
m
P/N 60/41
Au 1.3
12
L = 5
PEAK ENERGY
0.8
+1
m
0.6
Do5
0
1
0.4
P/2
0.2
+1
0.0
0.00
0.25
0.75 ENERGY/P
0.50
1.00
0.2
0.6
0.8 Aμ
1.0
1.2
Figure 4: Density of states analysis for binary classification of synthetic datasets examining the impact of mislabeled elements at different inter-class separations Au. (a) Pictorial representation of feature mislabeling for different numbers of mislabeled elements L. Each class is represented by an isotropic cloud of points, with one class having mean vector m labeled +1 (blue) and the other class with mean vector -m labeled as -1 (orange). The total number of points remains constant across different values of L. In the absence of feature noise (i.e., = 0), there are an equal number of points in each class. Mislabeling is introduced by randomly selecting L elements and flipping their labels. For = 0, all points are correctly labeled, while for increasing (e.g. = and I. P/2), a portion of points from the cloud labeled +1 are relabeled as -1, and and a portion of points from the cloud labeled -1 are relabeled as +1, leading to label noise. (b) Density of states at fixed lcarning complexity P/N 60/41 and inter-class separation (Ap 1.3), obtained in the binary classification for various numbers of mislabeled elements (L). The legend indicates that lighter colors correspond to higher number of mislabeled elements (greater L). (c) Peak energy values plotted against the inter-class distance for different numbers of misclassified elements. The plotted values in the range Ap E [0.8, 1.3] represent the average from 10 independent replicas the system in the same configuration, with associated statistical deviations.
introduction of mislabeled elements causes the peaks to shift. Notably, in the absence of mislabeling, the peaks typically appear at the tails the spectrum for configurations with large Ap and 0 = 180"
As the level of mislabeling increases, the peaks move progressively toward the center of the spectrum, converging an energy of approximately 0.5. This behavior reflects the increasing randomness of the classification task: with more mislabeled data, the network encounters a loss landscape where a large fraction of weight. configurations yield similar levels of classification error, tending towards the limit of random classification. In this scenario, the density of states becomes dominated by configurations that provide no clear decision boundary betwoen the classes. This transition aligns with the expectation that a hcavily mislabeled dataset effectively loses its structure, rendering classification equivalent to chance.
This shift in peak position due to mislabeling mirrors the behavior observed with class imbalance low values of the angles between mean vectors Fiiurre dd) highlighting the complex interplay between data characteristics and the resulting density of states. The progression of the DoS toward central energy point emphasizes the sensitivity of neural network performance to label noise and underscores the importance of
ensuring dataset quality in real-world applications (Relinetti et al. 2023 Belrose et al. 2024
12

---

Published in Transactions on Machine Learning Research (02/2025)
b
P1=10
P==50
P1= =20
P==
P1=30
41=40
P1=70
90
0.9
0.9
GM
2isoGM
80
0.8
0.8
70
0.7
0.7
Angle (deg)
60
PEAK ENERGY
PEAK ENERGY
0.6
50
0.5
0.5
40
0.4
0.3
0.3
20
0.2
0.2
10
0.1
0.1
2
3
5 Class
7
8
0.1
0.2
0.3
1.5
2.0
0.1
0.2
0.3
1.5
2.0
4u
Aμ
Figure 5: Density of states analysis for binary classification of Gaussian clones of MNIST data, investigating the impact of high-order statistics on DoS properties. (a) Angle between the mean vector of class and the mean vectors of other classes in the MNIST dataset, as indicated on the x-axis. (b) Peak location of the DoS plotted against the inter-class separation distance Ap for different values of class imbalance, as shown in the legend. Results are obtained from 9 different binary classífication ion problems, comparing MNIST class against each of the other MNIST classes. Red points indicate balanced classes, while blue and green points represent unbalancod classes, with lighter colors corresponding to greater imbalance. The two pancls differ in the approximation used for the covariance matrix: Gaussian mixture (GM, left) and isotropic Gaussian mixture (2isoGM, right.) clones.
3.2.3 DoS in a classification problem with data from Gaussian clones of real-world datasets
An interesting technique to understand the relevance of input structure on learning the construction of hierarchy of clones that only retain some statistical properties of the original real-world data.
In the light of this, we constructed synthetic dataset where each class is a Gaussian clone of the original MNIST data, but where we can adjust the inter-class distances (see Supplementary Section I for further details). By doing so, we can reproduce the behavior observed in the previous sections. This is true whether we employ the real correlation matrix (GM clone) or a diagonal covariance matrix (2ISOgm). In both cases, the DoS curves show properties that closcly mirror those obscrved with the synthetic datasets discussed Section 3.2.I. partieularly how the peak location is influenced by class imbalance for small angles bet ween the mean vectors (see Figure 5). The specific structure of the covariance matrices does not seem to play role in determining the peak location, which is mostly influenced by the difference in the mean vectors.
4 Discussion and conclusions
In this work, introduced a general approach to measure the density of states in a neural network learning problem. Albeit common practice in the field of statistical mechanics and soft matter physics (Kim et all 2006 Stelter & Keyes] 2019 Kim et al.) 2012 Tsai et al. 2007 Menichetti et al. 2021), the use of enhanced sampling techniques is only recently starting to be discovered in the context of machine learning in general, and neural networks in particular [Liu et al. 2023).
Here, for the first time to the best of our knowledge, we make use of one of these methods to thoroughly characterize the loss spectrum of simple neural architectures variety of learning tasks. It is crucial note that such method, at variance with classic average-case analytical methods, be profitably employed
13

---

Published in Transactions on Machine Learning Research (02/2025)
finite-size single instances, thus representing complementary and promising approach to novel class of problems and questions. Specifically, we used the Wang-Tandau algorithm to elucidate the impact of several factors on the DoS, focusing on the differential effects of input statistics, linear separability, and class imbalance. The last point is particularly interesting also in view of an increasing interest in the effect of class imbalance on learning and implicit biases in deep neural nelworks (Francazi et al. 2024 2023). Specifically, we addressed the effect of class imbalance on the entire spectrum of the training loss.
In this first application of our methodology concentrated our efforts on the analysis of fully connected, one- (hidden)-laye -layer networks; the natural generalization steps will be the study of more complex, multilayered structures, which would shed light on the impact on the DoS of weight sharing (e.g. in convolutional layers), recurrent computation (e.g. riqueece-to-sequees interactions (attention layers in transformer architectures). While a generalization of our technique to deeper networks conceptually straightforward, a key issue in the usage of this protocol is, however, its scalability (sce Figure 2 of the Supplementary Matorial for a benchmark of the convergence time): in our context, this means that its application deep overparamet rized neural networks would entail high computational cost, deriving from the exploration of large configuration space that grows exponentially with network size. We note that, in this work, we relied on the original, "plain vanilla" implementation of the WL algorithm due to its simplicity and suitability as a proof of concept of our approach (Wang & Landau 2001a b): the price to pay for algorit hmicc clarity and simplicity, though, is efficiency, especially on deep overparametrized tasks. This notwithstandinng the scalability challenge can be addressed through multiple strategies. Among them, we mention the parallelization across both training patterns and energy windows utilizing GPU resources (Yin & Landau 2012 Vogel et al., 2014 Farris & Landaul 2021), ; as well more advanced implementations of the basic WL algorithm making use, for example, of multi-level sampling (Vogel et al.] 2013), replica exchange (Vogel et al. 2014), and energy-biasing (Valentim et al., 2015), which in the realm of soft and condensed matter physics have been shown to siguificantly enhance the method's sampling efficiency and scalability. In addition, (Liu et al. 2023) demonstrates the merits of using optimized WL. implementations in the context of neural networks: in this work, in fact, the authors employ the method to explore the inpul space network, and they manage to improve efficiency and scalability by leveraging approxinations such as entropy interpolation and gradient discretization. Appropriate optimizations of the WL algorithm can thus offer promising strategies to enhance the scope of our approach and its application to larger and deeper network architectures, which is the subject of further ongoing work.
tasks in recurrent neural networks), and multiplicative
Furthermore, in this work we focused on networks with discrete synaptic weights: nonetheless, in the context of soft and condensed matter physics, WL techniques have been shown to be applicable to problems in continuous space see for instance their use in molecular simulations (Shell et al. 2002; Zhou al.; 2006; Yan & de Pablo, 2003). The extension of the WL algorithm to continuous weight neural networks does not modily the overall conceptual framework, except that smarter techniques are needed to generate the proposal moves one employs to navigate the weight space. On the other hand, in contrast to the discrete case, such moves can rely on the differentiable nature of the loss landscape, so that strategies akin to, e.g., stochastic gradient descent, can be employed to generate new weight configurations. Further significant help in this endeavor can come from enhanced sampling techniques that are commonly employed in the field of molecular simulations (Hénin et al. 2022).
Overall, the improvements and extensions of our methodology discussed insofar open a window towards its real-world application in modern neural network models. Our method could in fact e-g. contribute to explore the mapping between the weight space and function space descriptions, in line with recent advances on lincarized models in Bayesian networks (Roy et al. 2024]: this could facilitate deeper understanding of how weight configurations relate to functional outputs, complementing the insights gained from density of states analyses. Furthermore, the WL. approach could provide valuable insights into selecting the best network architecture for a given problem by analyzing the DoS and the dynamics of learning algorithms as weights traverse configurations of different entropy. It is indeed well known that gradient- based algorithms do not sample the loss landscape uniformly but rather possess inductive biases (Farnia et al. 2020 Nakkiran ct al. 2019 Rahaman et al.) 2019) that can also affect the entire learning dynamics (Refinetti et al. 2023) Belrose el al.] 2024]: analyzing the dynamics of optimization algorithus from the perspective of loss densily is thus an interesting avenue for future work.
14

---

Published in Transactions on Machine Learning Research (02/2025)
Acknowledgments
RP and AI conceived the study and proposed the method. All authors contributed the analysis and interpretation of the data. All authors drafted the paper, reviewed the results, and approved the final version of the manuscript. The authors thank Luca Tubiana and Camilla Spreti for critical reading and insightful comments. RP acknowledges support from ICSC Centro Nazionale di Ricerca in HPC, Big Data and Quantum Computing, funded by the European Union under NextGenerationEU. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those the European Union or The European Research Executive Agency. Neither the European Union nor the granting authority can be held responsible for them. RP and MM acknowledge support from Fondazione Cassa Rurale di Trento through the project SENTINEL. RP and RM acknowledge support from Fondazione CARITRO through the project COMMODORE (#20260).
Data availability
The raw data associated with this work are froely available on GitHub at https://github. com/ Dotestiolab/NaNDo. along with a Python code to reproduce the analysis presented in the work.
Supplementary Material
The supplementary material file provides further insight and data that support the findings of this paper. It includes:
1. Synthetic Dataset Generation: An expansion of the main text's Methods section with further technical details on how the syuthetic dalasets used in the study have been generated.
2. Wang-Landau (WL) Sampling Procedure: An in-depth description of the WL sampling method for estimating the density of states (DoS), including a step-by-step pseudocode.
3. Validation and Scaling Analysis: A discussion of WT. sampling accuracy comparing entropy estimates with exact values, and an analysis of how convergence time scales with network size and task complexity.
4. Gaussian Fit: Analysis of the DoS distribution for binary classification using single-layer perceptron, assessing Gaussian characteristics in specific data scenarios.
5. FashionnMMNIST Analysis: Study of how class imbalance affects the DoS in Fashion-MNIST binary classification, highlighting shifts in peak locations as imbalance varies.
References
Carlo Baldassi, Alfredo Braunstein, Nicolas Brunel, and Riccardo Zecchina. Efficient supervised learning in networks with binary synapses. Proceedings of the National Academy of Sciences, 104(26):11079- 11084, June 2007. doi: 10.1073 2..1073/ppaa.00003211010. URL htttts:://///i www.//wwwwp........./.../....... mtgg/oi//bbs/1..077/ 0700324104
Carlo Baldassi, Alessandro Ingrosso, Carlo Lucibello, Luca Saglietti, and Riccardo Zecchina. Subdominant dense clusters allow for simple learning and high computational performance in neural networks with discrete synapses. Physical Review Letters, 115(12):128101, September 2015. doi: 10. 10.1103/PhysRevLet.. 128111 URL http:httt/////// aps. www//oo/oooooooooooooooooo/o.oot 1103/PhysReviett..t 115. 128101
Carlo Baldassi, Christian Borgs, Jemnifer Tour Chayes, Alessandro Ingrosso, Carlo Lucibello, Luca Saglietti, and Riccardo Zecchina. Unreasonable effectiveness of learning neural networks: From accessible states and robust ensembles basic algorithmic schemes. Proceedings of the National Academy of Sciences, 113 E465 November 2016a. doi: 10.1078/pmas.10011081111 URL https: //www/ pnas org./oir// abs/ 10. 10733/ma.. 1608103113
15

---

Published in Transactions on Machine Learning Research (02/2025)
Carlo Baldassi, Alessandro Ingrosso, Carlo Lucibello, Luca Saglietti, and Riccardo Zecchina. Local entropy measure for sampling solutions in constraint satisfaction problems. Journal of Statislical Mechanics: Theory and Experiment, 2026(1) February 2016b. doi: 10. 00.008///T42--54455///////////00
URL
https: /x. doi. www.oo........... 1088/ L0A.///7425448/20001111111 21122/446///11//222333330
Carlo Baldassi, Clarissa Lauditi, Enrico Maria Malatesta, Gabriele Perugini, and Riccardo Zecchina. Un- veiling the structure of wide flat minima in neural networks. Physical Review Letters, 127(27):278301, December 2021. doi: PhysRevLett. 127. 278301
URL. https://link aps apss.org/do//10.10000
Lev Yu Barash, Maria Alexandrovna Fadoeva, and Lev Nikolaevich Shchur. Control of accuracy in the Wang- Landau algorithm. Physical Review E, 9644):043307, October 2017. doi: Sete33/PyySSeee....000
URL atttp:://inme http://iinnkaas..ooo aps. wtt.:/110///hee/../.
043307
Nora Belrose, Quintin Pope, Lucia Quirke, Alex Mallen, and Xiaoli Fern. Neural networks learn statistics of increasing complexity. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 3382-3409. PMLR, July 2024. URL https:// proceedings mlr. wwtps//www//eeeeeeeeeeeeeeeeeeee
html
Alfredo Braunstein and Riccardo Zeechina. Learning by message passing in networks of discrete synapses. Seetelyyeeeeett.....0... URL https:
Physical Review Letters, 96(3):03020)) Jamuary 2006. doi:
//link.aps www.ppppp...pppppp........oo.... hwtpeoogg//oooiie /o1/10. 11t.//hyysReeLeett..... wt.pettttt...00000...... 030201
Patrick Charbonneau, Enzo Marinari, Giorgio Parisi, Federico Ricci-tersenghi, Gabriele Sicuro, Francesco Zamponi, and Mare Mezard. Spin Glass Theory and Far Beyond: Replica Symmetry Breaking after 40
Years. World Scientific, August 2023. doi: 10. 10.1142/13341. URL https:/ https:///www.wwrr httpps//wwwwwwwoeeccciirri/////e wwwp:owwodsiiiiiiicccoooo
doi/abs/10. 10ta//1..110 0ete//.0.1122/111
Andreas Engel and Chris Van den Broeck. Statistical mechanics of learning. Cambridge Univer-
sity Press, 2001. doi: 101007/CHH097711390665555 URL https://wwz www.brriggg..oggooorroooooooooo
Farzan Farnia, Jesse M. Zhang, and David N. Tse. fourier-based approach to generalization and optimiza- tion in deep learning. IEEE Journal on Selected Areas in Information Theory, ((1):145 156, March 2020. doi:
2020.2983192 URL httpps:/hseeeppooree 1eee http://ww....e.eee.aaa/oa//orr/a
Alfred C.K. Farris and David Paul Landau. Replica exchange wang landau sampling of long hp model sequences. Physica A: Statistical Mechanícs and its Applications, 1697188777) May 2021. doi: https://doi.orr//ii 10.
www.saa.222.1122.. 125777 URL
com/science/
articleepp1/003784771100000
Kirsten Fischer, Alexandre René, Christian Keup, Moritz Layer, David Dahmen, and Moritz Helias. De- composing neural networks as mappings of correlation functions. Physical Review Research, 4(4):043143, November 2022. doi: 10.1103 hwtpseeeeRReeear.....001... URL https: aps wwwp/oo/wo///o.ooooootoooooooott 1103/ PhysRevResearch. 4. 043143
Emanuele Francazi, Marco Baity-Jesi, and Aurelien Lucchi. A theoretical analysis of the learning dynamics under class imbalance. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 10285 10322. PMLR, July 2023. URL https: //proceedings mlr. pwwps/vv22222 proess/v202/franczz22233 htt////raaaaaziaaaae html
Emanuele Francazi, Aurelien Lucchi, and Marco Baity-Jesi. Initial guessing bias: How untrained networks favor some classes. In Proceedings of the 41th International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 13783-13839. PMLR, July 2024. URL https: html
//proceedings.nl mlr. prroceedingg.m...reess/////
Silvio Franz and Giorgio Parisi. Recipes for metastable states in spin glasses. Journal de Physique 1, 5(11): 1401-1415, November 1995. doi: 10.1051/jp111199521 URL httppp///oa..orr///.....//.ppppp
16

---

Published in Transactions on Machine Learning Research (02/2025)
Elizabeth Gardner. The space of interactions in neural network models. Journal of Physics A: Mathematical and General, (1)) January 1988. doi: 10. 00.108880000-4477/21111111 /030. URL https://dx doi org/ 10. 00.1008/0305-44702211110
Elizabeth Gardner and Bernard Derrida. Optimal storage properties of neural network models. Journal of Physics A: Mathematical and General, 21(1):271, January 1988. doi:
URL
https://áx: doi. org/10. 1088/00883388 01:300-4477/2111//00
Elizabeth Gardner and Bernard Derrida. Three unfinished works on the optimal storage capacity of networks. Journal of Physics A: Mathematical and General, 22(1)):1983, June 1989. doi: /000080000054440/22] 12/004. URL httpt:://x.......// www.:ddii.ogg.//.1.1 org/10..//. 101.1100000054770222221111
Federica Gerace, Brumo Loureiro, Florent Krzakala, Mare Mezard, and Lenka Zdeborová. Generalisation
error in learning with random features and the hidden manifold model*. Journal of Statistical Mechanics:
Theory and Experiment, (12)(12):24 December 2021. doi:
/ac3ae6. URL https:
7/dx. 10tg///..1///14225544////
Federica Gerace, Florent Krzakala, Bruno Loureiro, Ludovic Stephan, and Lenka Zdeborová. Gaussian
universality of perceptrons with random labels. Physical Review E, 109(3))034305, March 2024. doi:
10.1108/PiysReee.1000.....
URL htttps://ink: aps. wpw..org//oooo 11t.p.....////ee..e......
034305
Jérôme Hénin, Tony Lelièvre, Michael R. Shirts, Omar Valsson, and Lucie Delemotte. Enhanced sampling methods for molecular dynamics simulations Jarticle v1.0]. Living Journal of Computational Molecular
Science, 4(1):15 83, December 2022. doi: 10.33011 10.33011/livecooo /liiellvvss..... 4.1.155,, URL http::://iiveeeomggurraaall
org/ htg/index.cr http:://pww....aaaaaaaa////////a
Alessandro Ingrosso. Optimal learning with excitatory and inhibitory synapses.
PLOS Compulational wwtp://o.a.................. URL httpe://doi.org/10
Biology, 16( (1)): 24, December 2021. doi:
1371/journal pcbi. 1008536
Alessandro Ingrosso and Sebastian Goldt. Data-driven emergence of convolutional structure in neural net-
works. Proceedings of the National Academy of Sciences, 119(40)::2200854111, September 2022. doi:
10.1073/pnas.2200 2201 854119. URL htttps///w// http///wwwpas..../////////m pnas. wttp:pppwwo.ooop////iiiieee httpsa////.////b////..///////ps 2201854119
Jaegil Kim, John Edward Straub, and Thomas Keyes. Statisticalltemperaauur Monte Carlo and molecular
dynamics algorithms. Physical Review Lellers, 97(5):050601, August 2006. doi: 10.1108/PhysReev.ee..97 050601. URL https://link: http:////innk..s.org///e//
1101 http://eeeeeeeeeee..... 050660
Jaegil Kim, John Edward Straub, and Tom Keyes. Replica exchange statistical temperature molecular dynamics algorithm. The Journal of Physical Chemistry B, 116((9):8646 8653, April 2012. doi: 10.1021/
jp300366j. URL httphhttp///oo. org/ 10. 1011//000000////30333666
Wolfgang Krauth and Marc Mezard. Learning algorithms with optimal stability in neural networks. Journal of Physics A: Mothematical and Genenal, 20(11):1745, August 1987. doi: 10. ///////0000040////1/1 URL httph:ttpp:sstt doi. wtwpd//..rrr....111/000000000070 00011000000000-770/0000000000000
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Ctadient-based learning applied to doc- ument recognition. Proceedings of the IEFE, 86(11):2278 2324, November 1998. doi: 10.1109/5.726791.
URL http:::///eeeeeplrreeeeeee htt.:///e...e..eeeeo/c/comme///a orgg/aocument//26779
Weitang Liu, Yi-Zhuang You, Ying Wai Li, and Jingbo Shang. Gradient-based wang-landan algorithm: a novel sampler for output distribution of neural networks over the input space. In Proceedings of the 40th International Conference on Machine Learning, number 929 in Proceedings of Machine Learning Research, pp. 22338 22351. PMLR, July 2023.
Bruno Loureiro, Cédric Gerbelot, Maria Refinetti, Gabriele Sicuro, and Florent Krzakala. Fluctuations, bias, variance and ensemble of Icarners: exact asymptotics for convex losses high-dimension* Journal of Statistical Mechanics: Theory and Experiment, 2023(11): November 2023. doi: 10.1088/ 00.108/1742--4666 ad0221. URI //dx. //t.x.ddi...o. http://p/e/..cooooooo///////////
17

---

Published in Transactions on Machine Learning Research (02/2025)
Bruno Loureiro, Cédric Gerbelot, Hugo Cui, Sebastian Goldt, Florent Krzakala, Mare Mézard, and Lenka Zdeborová. Learning curves of generie features maps for realistic datasets with a teacher-student model. In Proceedings of the 35th International Conference on Neural Information Processing Systems, vol- ume 34, pp. 18137-18151. Curran Associates Inc., June 2024a. URL URLpttps:: hhttps:///d acm. orgg 5555/ 0440261.3541644
Bruno Lourciro, Gabriele Sicuro, Cedric Gerbelot, Alessandro Pacco, Florent Krzakala, and Lenka Zde- borová. Learning gaussian mixtures with generalized linear models: Precise asymptotics in high- dimensions. In Proceedings of the 35th International Conference on Neural Information Processing Sys- tems, volume 34, pp. 10144 10157. Curran Associates Inc., June 2024b. URL https: acm. ota.doorg/ooo 10. 10.5555535402255 3541037
Roberto Menichetti, Marco Giulini, and Raffaello Potestio. A journey through mapping space: characterising the statistical and metric properties of reduced representations of macromolecules. The European Physical Journal B, 94(204): 26, October 2021. doi: 10.1140/eppji 10.114//eejb/s1000100000000001 URL https://doi.cno org/ 10.
Marc Mézard, Giorgio Parisi, and Miguel Angel Virasoro. Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications. World Scientific, November 1987. doi: 10.1142/0271. URL
https: /oo11/acieecii....comm//////////
Francesca Mignacco, Florent Krzakala, Yue Lu, Pierfrancesco Urbani, and Lenka Zdeborova. The role of regularization in classification of high-dimensional noisy Gaussian mixture. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, html
pp. 6874 6883. PMLR, July 2020. URL
mlr
Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang, Benjamin L. Edelman, Fred Zhang, and Boaz Barak. SGD on neural networks learns functions of increasing complexity, In Proceedings of the 33rd International Conference on Neural Information Processing Systems, volume 32, pp. 3491-3501. Curran Associates Inc., December 2019. URL https://t/ti acm. www.aaa.w....o....oo..o. /55p////11005555544222277 34446601
Manfred Opper and Ole Winther. Adaptive and self- self-averaging thoules--nndesoo-palme mean-field theory for probabilistic modeling. Physical Review E, 64:056131, October 2001. doi: 10.1 F..1103/PhySeev..6.... E.6.000611.... URL https://link: aps org..rrg//olo 1103 www.o//yy/eeee.ee.o 6461311.05611111
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Proceedings of the 36th International Conference Machine Leurning, volume 97 of Proceedings of Muchine Learning Resewrch, pp. PMLR, June 2019. URL https://proceeddini
mlr, wrwss/eevvvv //t./raaaaaan/... html
Maria Refinetti, Alessandro Ingrosso, and Sebastian Goldt. Neural networks trained with SGD learn distribu- tions of increasing complexity. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 19843-28866) PMI.R, July 2023. URI. html
https://proceeding mlr. www..rrressev2222 pttess/v222/efffetttttt
Aritra Roy, Lewis D. Smith, and Andrew G. Wilson. Reparameterization invariance in approximate bayesian inference, In Proceedings of the 38th International Conference on Neural Information Processing Systems. Curran Associates Inc., December 2024. URL http://ps///arrrrp....////t/////
Hyunjunc Sobastian Scung, Haim Sompolinsky, and Naftali Tishby. Statistical mechanics of learning from examples. Physical Review A, 68()):6056 6091, April 1992. doi:
A................66..... URL https:
//link //link.aps.org wtwppppp../oooooo..... 1103 www.:////Pyeeee......o.. 45.
Myers Scott Shell, Pablo G Debenodetti, and Athanassios Panagiotopoulos. Generalization of the Wang- Landau method for off-lattice simulations. Physical Review E, 66(5) November 2002. doi: 10. 1 T.0//PhysReeF..6.055777 URL httpss///inkk walnnkapss.oo wpp...or//aao//o..11////p//apapp
056703
18

---

Published in Transactions on Machine Learning Research (02/2025)
Takashi Shinzato and Yoshiyuki Kabashima. Perceptron capacity revisited: classification ability for cor- related patterns. Journal of Physies A: Mathemaationl and Theoretioal, 41(32)3324013, July 2008. doi: (00.188//TT5--811/44///////22/
URL httppp///g....ooooocc.....////11
Haim Sompolinsky, Naftali Tishby, and Hyunjune Sehastian Seung. Learning from examples in large neural networks. Physical Review Letters, (5(13):1683-16883 September 1990. doi: 10. 2.1100/PhysReeIeet.......
URL https://likkkk.... aps. org. http://oo/1.... 1103 100//hysseeeeet...... 1683
David Stelter and Tom Keyes. Simulation of fluid/gel phase equilibrium in lipid vesicles. Soft Matter,
15(40):8102 8112, September 2019. 098300854C
doi:
URL http://dx. doi wtw.:addoooorgrgg..ooooo http://dx.doi..o//t....o.
Shan-Ho Tsai, Fugao Wang, and David Paul Landau. Critical endpoint behavior an asymmetric ising model: Application of Wang-Landau sampling to calculate the donsity of states. Physicaa Review E, 75 (6):061108, June 2007. doi: 11113//PyyBexe..5.0000 URL https://link: aps wwtgg/////...o// 1103/ PhysRevE.75.06110
Alexandra Valentim, Julio Rocha, Shan-Ho Tsai, Ying Wai Li, Markus Eisenbach, Carlos E Fiore, and David Landau. Exploring replica-exchange wang-landau sampling in higher-dimensional parameter space. Journal of Physics: Conference Series, 640(1):012006, September 2015. doi:
65122//6666////0//22000066 640/1/012006. URL https:///dx: www.::.oo...rg//.////.. 000.00/00000...11.02222266666611
Thomas Vogel, Ying Wai Li, Thomas Wüst, and David P. Landau. Generic, hicrarchical framework for massively parallel wang-landau sampling. Physical Review Letters, 110(21):210003, May 2013. doi: 10.
hutp/PlyssteeLeet.....0.000
URL https://lini aps htgp//ooo/....110///yeeeeeeeeeet 110.221611111001000
Thomas Vogel, Ying Wai Li, Thomas Wüst, and David Paul Landau. Scalable replica-exchanne framework for wang-landau sampling. Physical Review E, 90(2):023302, August 2014. doi: 10.1 1..1108/PyyRRov..00..22222
URL https:://ikk. aps org//oi//1/ 1ww111101Phh/ 1100/PhysRevE.99.... 023302
Fugao Wang and David Paul Landau. Determining the density of states for classical statistical models: A random walk algorithm to produce a flat histogram. Physical Review E, 65(5))056101, October 2001a. doi: F.1103//hysRee.......555
URL htttp::/linkk httpp/://iinna.p.oogg/aaaaa//// wtwppppo../oooo....ooooooooooooo 1003000.111..000 Www.110///hhhysee 64. 056101
Fugao Wang and David Paul Landau. Efficient, multiple-range random walk algorithm to calculate the density of states. Physical Review Letters, 86(10)22050, March 2001b. doi: 10.1103/ 10.113/PPyssselltlttt 060t..0.
URL https: //link org.
www./o/gg./////oooooo/ooo/.oooo/ /tod/ooi//// 1103 www.///hyseeeeet.tc.. 86. 2050
Timothy L. H. Watkin, Albrecht Rau, and Michael Biehl. The statistical mechanics of learning rule.
Reviews of Modern Physics, 65(2):499-556) April 1993. doi: 10.
URL https:
7/1ink. //link.aps.o org.ppp.....pppppppp.......o.... wwwg///g//oo//oooooooooooo /to/10.111111/// 1100..11000 1103 / RewoodPhss....o 65.65..444 499
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mmist: novel image dataset for benchmarking
machine learning algorithms. Arx abs/ abs/1708.07777, September 2017. doi: 11.48550/rXiiv.170..00
URL attps://arxivv org/ abs/ 1000 07747v2
Qiliang Yan and Juan J. de Pablo. Fast calculation of the density of states of a fluid by monte carlo simulations. Physical Review Letlers, 90(3):035701 Janury 2003. doi: 10.1108/PhysReeeeet..000 URL https://linm: aps. http//oi/110/ 1103 wtw.:///aayyeeeeeteee.....
Junqi Yin and David Paul Landau. Massively parallel wang landau sampling on multiple gpus. Compuler Physics Communications, 18368);1568 1573, August 2012, doi: http:///....../00...0/.........p
Chenggang Zhou, Thomas C. Schulthess, Stefan Torbrügge, and David Paul Landau. Lang-Landa algo- rithm for continuous models and joint density of states. Physical Review Lellers, 2006. doi: 10.1088/PaysRerLeett....22 96. 120201
March URL https:: homps://1imt //link. aps org. h/gg/doi/10..o 1003/hyyseeLeett.
19


Reviews:
[{"review_id": "iXNbWfYdsk", "reviewer": "Reviewer_Fd7m", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The problem that the paper studies is interesting and quite relevant to modern deep learning.\n2. Several empirical studies are carried out, highlighting how DoS varies across multiple factors.\n\nWeaknesses:\n1. The paper focuses on networks with binary weights and simple architecture, which are clearly different from modern neural networks. While I believe it is still interesting and some insights may carry over, it is a limitation.\n2. The vanilla Wang-Landau sampling algorithm as presented in the paper has high computational complexity, both with respect to network size and problem complexity, thus is in general not scalable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies the density of states (DoS) of neural networks given a specific loss, which may or may not be the minimum. The paper focuses on neural networks with binary weights, and employs the Wang-Landau algorithm to draw samples while constructing estimates of the densities. Using the approach, the authors conducted experiments, investigating the loss landscape of simple neural networks. The authors highlighted the effects of different factors on the DoS, including the structure of data and network, class imbalances, inter-class statistics and mislabeling. The authors highlighted that the proposed approach is general, and is in principle applicable to larger neural networks, though the computational cost would become too large.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The empirical studies carried out in the paper is interesting, and the claims are generally justified. However, in abstract, the authors did explicitly state \"a novel, computationally efficient approach to examine the weight space across all loss values\". However, as acknowledged by the authors themselves, the computational complexity becomes too large as either the network size or the problem complexity becomes too large. Thus, I think the statement is slightly misleading, and might need some rewordings.\n\n(More like comments) It might be interesting to note that there are papers studying the function space landscape of modern neural networks, e.g. [1]. The referenced paper took a differential geometric perspective, though indeed did not study the exact densities. It might also be worth noting that previous research has also employed Wang-Landau algorithm in deep learning; I was able to find [2].\n\n1. Reparameterization invariance in approximate Bayesian inference, Roy et al., NeurIPS 2024\n2. Gradient-based Wang-Landau Algorithm: A Novel Sampler for Output Distribution of Neural Networks over the Input Space, Liu et al., ICML 2023", "broader_impact_concerns": "The work is largely theoretical, thus not applicable.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "eDoedq9eaA", "reviewer": "Reviewer_7kEe", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The use of the Wang-Landau algorithm to analyze the entire loss spectrum of neural networks is novel and provides insights beyond traditional loss minimization approaches. \n- The paper investigates multiple factors influencing the DoS, such as class imbalance, data structure, and network architecture, offering a thorough exploration of the topic.\n- The study highlights how the statistical properties of datasets, including class balance and inter-class separation, directly affect the density of states, providing actionable insights for dataset design. \n- By validating results on both real-world datasets and controlled synthetic tasks, the work bridges theoretical exploration and practical relevance. The insights into DoS offer practical applications for designing neural network architectures and understanding the dynamics of learning algorithms, particularly in structured datasets.\n\n\nWeaknesses:\n\n- The Wang-Landau algorithm is computationally expensive for large neural networks, which limits the method's immediate applicability to real-world deep learning models.\n- The study primarily analyzes single-layer and one-hidden-layer networks with binary weights, which may reduce the relevance of findings to modern, deeper architectures commonly used in practice. While the method is validated on small-scale tasks, its behavior and computational feasibility for larger networks with continuous weights remain unexplored.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a novel method to analyze the loss landscape of neural networks by examining the density of states (DoS) -- the number of weight configurations corresponding to each loss value. Employing the Wang-Landau enhanced sampling algorithm, the authors explore the entire loss spectrum rather than focusing solely on loss minimizers. The work reveals how properties of structured datasets influence the DoS, using both real-world datasets (e.g., MNIST) and synthetic data for validation. Key findings include the impact of network architecture, class imbalance, and data structure on the shape and symmetries of the DoS, with potential applications in understanding network behavior and guiding architecture design.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would appreciate more discussion on how the WL algorithm can be adapted for networks with continuous weights, as this would bridge the gap to real-world applications.\n\nLess importantly, some empirical results demonstrating how the WL algorithm scales to larger neural networks and datasets may help convince readers the framework’s practical applicability.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Uh2DYUJrcA", "reviewer": "Reviewer_K1yk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- This paper is very well written and accessible for a researcher outside of this niche\n- The results shown are extensive and consider a range of conditions under which the network solution space can be compromised due to dataset issues\n\n**Weaknesses**\n- Though benchmark datasets were used, the application to binary networks and use of MNIST limits the interpretability of the results to real-world models and tasks\n- The conditions investigated in the results section were extensive, but were not always discussed in a way which could provide the reader maximum intuition. Observations of state density variation could have been combined with some more discussion.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work provides a novel method for the determination of the density of states, in terms of number of parameters which achieve a particular loss value, in a neural network model. This state density is determined by an efficient sampling method (the Wang-Landau sampling algorithm) and is applied to binary neural networks for single and multi-class classification. One benefit for this method is its application beyond simulated data to benchmark datasets. The application of this method explains and explores the impact of a number of adverse conditions including dataset bias, class scarcity, class overlap and more. Each of these conditions is then explored via the lens of density of states, allowing an understanding of the impact upon solution space.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Ideally, sections 3.1 and 3.2.2 would benefit with a little more discussion beyond simple observations of the results. As an example, for Section 3.1, the existence of two symmetric peaks in the DoS can be linked to the existence of sets of solutions in which the class with a larger number of samples is more or less easily classified (decision boundary is easier to find which minimises/maximises energy). Similarly for Section 3.2.2, more and more mislabelled data tends toward a random classification task with a DoS focussed upon a 0.5 energy.\n\n- Section 3.2.3 discusses results which are in Supplementary Material, however these figures are not visible without opening up an additional document. I would request that either the relevant figures are brought into the main text, or that this section be integrated into an existing section (and not its own subsection). Currently it holds too prominent a place for the results to be unavailable in the main text.\n\nThough this is not my area of expertise, I see this work as thought provoking and with some potential interest from the community.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers agree that the authors responded adequately to their comments and that the paper is ready for publication.
Decision: Accept (poster)