Sufficient is better than optimal for training neural networks

Domain: physics
Published: 2025-12-04
Editorial outcome: major_revision

Abstract
The array of neural network training techniques that invoke optimization but rely on ad hoc modification for validity suggest that optimization-based training is misguided. Shortcomings of optimization-based training are brought to strong relief by overfitting, where naive optimization produces spurious outcomes. Here, we introduce simmering, a physics-based method that trains neural networks to generate “good enough” weights and biases, paradoxically outperforming leading optimization-based approaches. Instead of optimizing, simmering systematically samples non-optimal weights and biases to generate an ensemble that provides sufficient representations of the underlying phenomenon. Simmering corrects neural networks that are overfit by optimization, and produces more generalizable predictions if deployed from the outset compared to other overfitting mitigation methods. Our results question optimization as a paradigm for training transformers, and feed-forward and convolutional neural networks. We leverage information-geometric arguments to point to the existence of classes of sufficient-training algorithms that do not take optimization as their starting point.

Introduction
Although neural networks’ universal estimation capability1–4 allows them to represent many complex data relationships5, that capability makes training generalizable networks challenging. The over-parameterization that supports the universal capabilities of neural networks nonetheless gives key advantages over other estimators in settings where complex data produce a training loss landscape that is non-convex and has many local minima6,7. Yet noise in training data can misdirect the parameter estimation process towards an overspecified representation that accurately respects idiosyncrasies in training data, but that severely limits generalizability8–10. This accuracy-generalizability discord is exacerbated by optimization-based training methods, which are overly effective at exploiting universal estimation capacity to achieve minimized-loss representations of training data.

The danger of combining the excessive expressiveness of a neural network and discrepant data with optimization is brought to particular relief by overfitting11–15. Overfit neural networks are inevitable when an over-parameterized architecture is combined with an efficient optimization algorithm6 (e.g., Adam16). Optimization yields high-complexity networks that generalize poorly because optimization-based training cannot distinguish between the ground truth and the noise in the data during training. Even in large- and augmented-data regimes, with correspondingly advanced architectures such as ResNet17 and Transformers18, modern networks can memorize noise or random labels with negligible training error while still exhibiting poor generalization,19 indicating that overfitting is a fundamental learning challenge that transcends dataset and model size19–21. Attempts to mitigate overfitting, e.g., early stopping22, bagging23, boosting24, dropout21, all account for data uncertainty by incorporating deviations from empirical error minimization into training. However, the effectiveness of most overfitting mitigation techniques relies on the data distribution satisfying specific assumptions12, and is thus problem dependent. Nonetheless, the success of avoiding overfitting via increased training loss12 suggests that more generalizable representations of ground truth are near-optimal rather than optimal11. Thus, training paradigms that are founded on an alternate premise, e.g., sufficiency rather than optimality, could produce non-overfit, generalizable estimators while still benefiting from the expressive capacity of neural networks.

Results
Sufficient training by simmering
Existing, optimization-based training algorithms that work to mitigate overfitting are engineered to avoid optimizing the empirical error because optimized sets of weights and biases do not reproduce ground truth in generic problems. The fact that generalizable representations of ground truth do not optimize the empirical error suggests the need to systematically explore non-optimal configurations. Exploring non-optimal configurations in generic problems, where it is not known a priori how training data depart from ground truth, motivates generating minimally biased deviations from optimality. Information theory suggests32 employing a generating function that is the Pareto-Laplace transform30 of the training loss
Z(β,D) = dNx e−βL(x,D), (1)
where x is the set of neural network parameters (weights and biases), x = (w, b), N is the total number of neural network parameters, L(x,D) is the loss function evaluated over the training data D, and β is the Laplace transform variable. Z is a generating function for sufficiently trained networks, and generates networks that minimize training loss in the limit β→∞.

We use Eq. (1) to generate sufficiently trained networks algorithmically by identifying Z(β,D) as a partition function in statistical mechanics30. The training algorithm (see “Methods”) treats Z(β,D) as the thermal, configuration space integral of a system of classical particles representing the weights and biases of the network, with each particle’s 1D motion driven by an interaction potential energy determined by the training loss. In this representation we take T=1/β as the system temperature. Without loss of generality, we lift this configuration space to a phase space by augmenting each weight and bias with an auxiliary, canonically conjugate momentum and a canonical, non-relativistic kinetic energy. These momenta and kinetic energy impart the system with auxiliary dynamics.

This dynamical approach entails gradient forces on weights and biases that can be computed via backpropagation. We thermalize the dynamics to reproduce the distribution Eq. (1). We operationalize the thermalization via numerical integration of the equations of motion of the neural network coupled to a Nosé-Hoover chain thermostat, which we implement (see Supplementary Methods) by symplectic integration33,34. For portability, and to facilitate use for problems beyond those we study in detail below, we implement the algorithm in Python and model the neural network in TensorFlow35, leveraging TensorFlow’s autodifferentiation to compute the gradient forces that drive training dynamics. An open-source implementation of our approach is available at Ref. 31.

Equation (1) serves as a generating function for sufficiently trained networks for finite β=1/T. The key to obtaining sufficient training in the dynamical approach we use here is to maintain the auxiliary dynamics at a small but finite temperature, or simmer, so that the network systematically explores near-optimal configurations of weights and biases. These near-optimal configurations need not individually improve the network accuracy for test data because they belong to ensembles of networks, and various methods36,37 can be applied to the ensembles to extract more generalizable representations. We give general arguments in “Methods” that this supremacy of sufficient training over optimal training is a generic feature of the family of methods we introduce here.

To facilitate comparison between sufficient- and optimal training methods, we first deploy simmering to reduce overfitting in networks overfit by Adam. Figure 1a gives an example of this retrofitting procedure in the case of a standard curve fitting problem. Figure 1b shows a set of training and test data that are generated by adding noise to a sinusoidal signal (green line). With these training data, we train the parameters of a fully connected feedforward network using Adam. Figure 1a shows the evolution of the loss, with a clear divergence of the training and test loss during the Adam training stage. Figure 1b shows that the Adam-generated fit discernibly deviates from the true signal.

To correct this deviation, we apply simmering, taking the overfit, Adam-generated network as the initial condition. We introduce stepwise increases in temperature (gray line, Fig. 1a) from T=0 to T=0.05 (taking T to be measured in units of loss). Simmering generates ensembles of sufficiently trained networks at finite T which we then aggregate to construct a retrofitted representation of the underlying signal. Figure 1c shows that simmering has reduced the discrepancies that were present between the Adam-produced fit and the original signal. Figure 1d shows that a simmering-generated ensemble of sufficiently trained networks at T=0.05 generates an aggregated fit that is virtually indistinguishable from the original sinusoidal signal.

We carried out an analogous retrofitting procedure on a set of similar problems. Figure 1e shows retrofitting results for classification problems, where, in all tested cases, applying simmering to retrofit overfit networks results in improved classification accuracy on test data. Figure 1f shows results for the application of simmering to retrofit regression problems, where simmering reduces the residual of the fit for test data compared with overfit, Adam-produced networks.

Ab initio sufficient training
Figure 1 demonstrated several applications of simmering to retrofit networks that are susceptible to overfitting by conventional, optimization-based training. These results raise the question of whether optimization-based training is necessary or whether ab initio implementations of sufficient training could avoid overfitting without any need for an optimized initial condition.

Figure 2 shows results from sufficiently trained neural networks in which simmering was deployed from the outset, without the need for optimization. Figure 2a shows results for classification, and Fig. 2b shows results for regression. It is important to note that because simmering yields an ensemble of networks, like other ensemble learning approaches37, it can be used to generate prediction uncertainty estimates that mitigate the artificial precision that arises from singular, optimization-generated solutions. These uncertainty estimates are shown in Fig. 2c–f. A key advantage of simmering is that, by exploring the maximum entropy distribution of weights and biases, it makes minimally biased assumptions about the training data error distribution and thus yields a minimally biased prediction uncertainty estimate.

Employing a minimally biased model of training data error allows ab initio simmering to outperform other commonly used overfitting avoidance methods in two different learning tasks: image classification with a convolutional neural network and language translation with a Transformer. Figure 3 shows that, for both learning tasks, ab initio simmering achieves a higher test data accuracy than standard implementations of early stopping and dropout. Consistent with ensemble learning approaches, ab initio simmering outperforms singular optimized models treated with early stopping (c.f. rectangular maroon marker vs. round green markers in Figure 3). For CIFAR-10, Simmering had both the most accurate ensemble prediction and the most significant ensembling improvement.

These comparisons highlight the role of simmering’s distinct minimally-biased ensemble generation in efficiently producing generalizable predictions.

Methods
Sufficient training is a class of approaches that addresses the generalizability limitations of optimal training by systematically sampling neural network parameters with near-optimal loss. Sufficient training samples ensembles of near-optimal neural networks from the maximum entropy distribution32, which makes a minimally biased assumption about the deviation from ground truth present in the training data. This minimally biased ensemble can then be aggregated to produce generalizable predictions37. Starting from entropy maximization, we show how sufficient training generically uses the geometry of the loss landscape to deviate from optimal training loss during sampling, and how this deviation is controlled by the temperature T. For additional information about our notation, see Supplementary Methods.

A neural network’s training loss landscape is specified by its parameters x, training dataset D, and loss function evaluated over the training data, L(x,D) where we will drop the vector notation on x hereafter. The neural network parameters x are distributed according to the probability density p(x|D) that maximizes the entropy
S(x,D) = dNx p(x|D) ln(p(x|D)) − β dNx p(x|D)L(x,D) − λ dNx p(x|D) − 1 ,
(2)
where β=1/T is a Lagrange multiplier that enforces the moment constraint, and λ=lnZ is a Lagrange multiplier that enforces the normalization of p(x|D). The p(x|D) value that maximizes Eq. (2) is
p(x|D) = e−βL(x,D) / Z(β,D) , (3)
where Z(β,D) is the partition function we are familiar with from statistical physics,
Z(β,D) = dNx e−βL(x,D) , (4)
describing the partition of probabilities of states (parameter sets), weighted by their training loss value. We can thus also interpret Eq. (4) as the Pareto-Laplace transform30 of the loss, with β controlling the severity of penalization of high-loss parameter sets (recovering only the optimal solution in the limit β→∞). Sampling states (parameter sets) from the canonical ensemble (Eq. (3)), or approximating the integral in Eq. (4), yields an ensemble of individually non-optimal parameter sets (corresponding to an ensemble of models) that can be used for uncertainty analysis and ensemble prediction.

The Pareto-Laplace filter provides one of many possible mechanisms for generating ensembles of models at non-minimal loss. However, the ensemble generated by this filter makes minimally biased assumptions about the noise distribution in the data, and is thus advantageous in settings where additional information is not available about the ground truth. When other information about the ground truth is available (e.g., the physics of the data), an extension of the current approach via physics-informed neural networks45 is likely to provide minimally biased sufficient-training methods as well.

Information geometric framing
To facilitate analysis of the partition function defined in Eq. (4), we define a set of collective variables θ(x,D) (for which we will drop the vector notation) that are a function of the neural network parameters and the training data. We can rewrite Eq. (4) in terms of the collective coordinates θ by first inserting the identity operator
Z(β,D) = dnθ dNx e−βL(x,D)δn(θ(x,D) − θ) , (5)
and then rewriting Eq. (5) only in terms of θ
Z(β,D) = dnθ e−βF(θ,D) , (6)
where
e−βF(θ,D) = dNx e−βL(x,D)δn(θ(x,D) − θ) : (7)
The effective free energy, F(θ,D), specifies the probability density p(θ|D) of sampling a particular value of θ. Up to an overall constant,
p(θ|D) / e−βF(θ,D) , (8)
and this probability can be computed via the mean-value theorem
e−βF(θ,D) = e−βL(θ,D) Ω(θ,D) , (9)
where Ω(θ,D) is the volume of the n-dimensional hypersurface of x along which θ(x,D) is constant, < > denotes the mean on that surface (also called an ensemble average), and L is evaluated over the training dataset D. Typically, we refer to Ω(θ,D) it in terms of the entropy S(θ,D) (reparameterized Eq. (2)) through the relation S=lnΩ. Additionally, since θ is a set of collective variables, we can assume that it is slowly varying over Ω(θ,D) such that <e−βL> ≈ e−β<L>. Therefore, taking L(θ,D) as the loss from the ensemble average in Eq. (9) and replacing T=1/β, we get the relation for the free energy F(θ,D)
F(θ,D) = L(θ,D) − TS(θ,D) : (10)
The most probable θ values (and thereby, the most probable neural network parameter combinations) are those that maximize p(θ|D) (Eq. (8)) and thus minimize F(θ,D) (Eq. (10)). In the T→0 limit, we recover optimization, where minimizing F(θ,D) minimizes the loss L(θ,D). However, at any finite temperature T>0, the free energy-minimizing θ does not minimize L(θ,D) (or L(x,D)) because an entropic force −T∂ θS(θ,D) continuously drives θ away from minimal loss. This entropic drive away from minimal loss is generic because there are generally more ways for a system (e.g., a neural network) to have non-minimal loss than minimal loss (neural networks that do not exhibit this property could be trivially trained by randomly selecting weights and biases). This entropic force is key to the effectiveness of sufficient training – by incorporating finite-temperature dynamics into parameter updates, entropic forces systematically and continuously drive the learning trajectory away from loss-minimizing and subsequently overfit parameters.

The collective coordinate representation of F also allows us to more clearly see how β filters the loss landscape during sampling, via techniques from information geometry41. In practice, this filtration occurs automatically and generically.

To do this analysis, consider the Fisher information metric (FIM), which is defined as
gμν (β,D) = − ∂2(βF(θ,D)) / ∂θμ ∂θν : (11)
For the present purposes, Eq. (11) can be computed in physics terms as a generalized susceptibility according to
gμν (β,D) = 1 / Z dnθ e−βF(θ,D) ∂2(βF(θ,D)) / ∂θμ ∂θν , (12)
where Z is defined in Eq. (6). The FIM has been interpreted in the work of Ref. 41 as an object that can be used to identify key order parameters that describe data–model relationships. Ref. 41 showed that the spectrum of gμν yielded sloppy modes that describe parameters that are loosely constrained by data, as well as stiff modes that are highly constrained. This mode classification provides a useful lens for interpreting the following analysis of the effect of temperature on the probability distribution of F(θ,D) (and subsequently, the neural network parameters sampled) in the context of the Pareto-Laplace filter (Eqs. (4–10)).

Consider the parameter space near a free energy minimum such that ∂ θF=0. For simplicity, take the minimum at θ=0 (which we can do without loss of generality by making a coordinate transformation). Taylor-expanding the FIM (Eq. (12)) near the minimum yields
gμν (β,D) ≈ 1 / Z e−βF(0,D) dnθ e − 1/2βθ α θ γ ∂2Fα(θ,D) / ∂θμ ∂θν | θ=0 , (13)
and
≈ 1 / Z e−βF(0,D) dnθ e − 1/2βθ α θ γ ∂2Fα(θ,D) / ∂θμ ∂θν | θ=0 : (14)
For sufficiently large β (or low T=1/β), these expressions can be integrated using a saddle point approximation, which yields
gμν (β,D) ≈ β ∂2F(θ,D) / ∂θμ ∂θν | θ=0 : (15)
The FIM describes the relative proximity of θ to the free energy minimum at θ=0 as a function of β=1/T. The squared distance in parameter space is
ds2 / T 1 X λ j dλ2 j , (16)
where λ are the eigenvalues of the Hessian of F(θ,D) that describe the curvature of F along different modes in parameter space. Distances (Eq. (16)) in a finite-temperature system diverge for small T, except along directions corresponding to sloppy modes for which the corresponding eigenvalues λj →0. Simmering and other sufficient training implementations exploit this effect of temperature on distances in parameter space to systematically sample families of loss-equivalent networks located along these sloppy modes. In Supplementary Fig. 3, we provide a comparison of the learning trajectory of simmering with that of Adam for the noisy sine problem in Fig. 1a. This trajectory visualization shows that simmering identifies and travels along sloppy modes in parameter space, whereas Adam traverses the space perpendicular to loss function contour lines.

This information geometric framing presented here also motivates the effectiveness of both ab initio simmering and retrofitting at overfitting in neural networks. The effect of T on the Hessian is analogous to the effect of the regularization strength parameter, typically denoted λ, on the eigenvalues of the Hessian of the training loss when using L2-norm regularization.46 Using finite-temperature parameter sampling, both ab initio simmering and retrofitting rescale distances in parameter space in the same way as L2-regularization, which has already been shown to reduce overfitting46.

Prediction uncertainty quantification with simmering. Since sufficient training identifies a distribution of parameter sets (and thus, a distribution of predictions) at a given β=1/T, simmering and other sufficient training can be used for prediction uncertainty quantification. In a generic supervised learning task, we aim to compare the ground truth based on training data D = {z, z , ..., z } and targets {t, t , ..., t }. We use a neural network to model our estimation of ground truth y(z0; θ) dependent on an unseen input z0 and collective coordinates θ(x,D) (where we will drop the vector notation as in the information-geometric framing). The probability density of predicting a target value t0 based on an unseen input z0 is
p(t0|z0,D) = dnθ p(t0|z0,θ)p(θ|D) : (17)
According to Bayes’ rule, the posterior distribution p(θ|D) in Eq. (17) is
p(θ|D) / p(D|θ)p(θ) , (18)
and where p(θ|D) is the canonical distribution of θ at β=1/T=147. If we generalize the Bayesian posterior distribution equation for a general T, we define the probability distribution from Eq. (10). Thus, the uncertainty quantification from Eq. (8, 9), with F(θ,D) can be thought of as a generalization of that of Bayesian inference. Comparison between Eq. (8) and Eq. (18) identifies S(θ,D) as the regularizer on the collective coordinates θ, and the temperature T as the regularization strength.

As in the information-geometric framing presented, we consider the region of the parameter space near a free energy minimum, and for convenience, this minimum occurs at θ=0. We can expand F(θ,D) near this minimum as
F(θ,D) ≈ F(0,D) + 1/2 β gαγ (β,D)| θ=0 θ α θ γ , (19)
Using Eq. (8) and Eq. (15) and (19) to relate ∂2Fα(θ,D) / ∂θμ ∂θν to gαγ(β,D) for large β (small T). Using Eq. (20) we can write
p(t0|z0,D) / dnθ p(t0|z0,θ)e− 1/2gαγ(β,D)| θ=0 θ α θ γ : (20)
Eq. (20) shows that for a generic training dataset D, simmering reduces distances in parameter space, allowing for sampling of near-optimal θ values that can elucidate features of p(t0|z0,θ).

In the particular case where the training data targets have additive Gaussian noise ϵ ∼ N(0, σ2t), and we choose mean-squared error (MSE) as the loss function, the prediction uncertainty distribution is structurally similar to that of Bayesian inference with noise injection, but generalized to any β=1/T. However, this equivalence between noise injection and sufficient training is a special case; in general, sufficient training does not assume a form for the noise distribution. Given this training data, target noise distribution, and loss function choice, p(t0|z0,θ) becomes
p(t0|z0,θ) = 1 / 2π σ2t 1/2 e − 1/2σ2t (y(z0;θ) − t0)2 : (21)
Then, we can expand y(z0;θ) around the free energy-minimizing θ=0 as
y(z0;θ) ≈ y(z0;0) + ∂y(z0;θ) / ∂θ α | θ=0 θ α : (22)
Using Eq. (21) and the Taylor expansion of y(z0;θ) in Eq. (22), we can evaluate Eq. (20) to be
p(t0|z0,D) / e−(t0 − y(z0;0))2 / 2σ2 , (23)
where σ2 = σ2t + gαγ(β,D)−1| θ=0 ∂y(z0;θ) / ∂θ α | θ=0 ∂y(z0;θ) / ∂θ γ | θ=0 . In Bayesian inference, σ2 = σ2t + 1/β, where β is the regularization strength parameter, typically denoted λ, which has already been shown to reduce overfitting46. In sufficient training, the variance equation generalizes to any positive non-zero inverse temperature β. The variance σ2 of the prediction uncertainty distribution has two terms: the variance introduced by uncertainty in data target measurements, σ2t, and a variance-like quantity defined by the sensitivity of the free energy and the model predictions to changes in θ. The overall effect of temperature on the second variance term is to reduce the FIM, gαγ, thereby increasing the second variance term contribution, but the exact way in which changing gαγ(β,D) affects ∂ θy is problem-dependent.

Parameters sampling mechanics
The probability density in Eq. (3) describes the distribution of states of a mechanical system in thermal equilibrium with a heat bath. There exist many thermostat algorithms in molecular dynamics28 that control the temperature of a system in order to simulate thermal equilibrium and sample from this distribution of states. Thus, if we model our neural network parameters as a system of particles, we can use existing constant-temperature simulation methods to generate canonical ensembles of parameters (corresponding to ensembles of models).

To employ molecular dynamics techniques in a neural network problem, we treat the neural network parameters as a system of one-dimensional particles in an interaction potential. The magnitude of each weight and bias defines the position of each particle in the physical system, and the loss function acts as the system potential. The negative gradient of the training loss function acts as a force on the system of particles that push them towards a minimum of the loss function. If we also define a conjugate momentum for each particle, we can integrate the equations of motion for the physical system and iteratively generate sets of weights and biases. In the absence of other forces, this integration would yield a set of weights and biases that predict with increasing accuracy on the training data, and as such is analogous to the result of any traditional momentum gradient-descent training algorithm. Generically, a thermostat algorithm adds another force on the particles of the system to simulate particle interactions with a heat bath (a thermal noise source).

Nosé-Hoover chain thermostat
A thermostat, in a molecular dynamics context, is an algorithm that control the temperature of a physical system28. In simmering, we use a Nosé-Hoover chain (NHC) thermostat, but other thermostats can also be employed to achieve constant-temperature conditions in other implementations of sufficient training. The NHC thermostat samples from the canonical ensemble by introducing an interaction between the neural network parameters and a chain of massive virtual particles.28 The first particle in the chain exchanges energy with the system of neural network parameters, and the rest of the chain only interacts with their neighboring chain particles. Each virtual particle has a position and a conjugate momentum that are computed iteratively along with those of the system of real particles.

Given a set of N neural network parameters, we define a set of positions x={xi}, associated momenta {pi} and masses {mi}. Using this set of quantities, we can model a system of one-dimensional particles in a potential defined by a loss function L(x,D) that depends on the neural network parameter positions (weights and biases) x and the training dataset D. This physical system is also interacting with an NHC of length NHC, where each chain particle also has its own mass Qk, positions and momentum pk. Henceforth, the neural network parameters will be referred to as the real particles, to contrast with the virtual particles of the NHC.

The particles’ equations of motion are derived from the NHC Hamiltonian50,51. For simplicity of notation, we will henceforth set mi=1, Qk=1 ∀i,k and describe the equations of motion in terms of the positions and velocities, rather than the positions and momenta. The real particles’ positions evolve according to33
x_ i(t) = vi(t) (24)
v_ i(t) = a(t) − vs1(t)vi(t) , (25)
where the i subscript denotes real particle quantities, and the s subscript denotes a quantity affiliated with the kth virtual particle. The vs1(t)vi(t) term from the NHC in Eq. (25) distinguishes simmering from gradient-based optimization. For example, in momentum gradient descent, Eq. (25) would have vs1(t)=const.52 In contrast, vs1(t) in NHC dynamics can take on both positive and negative values over time.

The virtual particle dynamics are given by
s_ k(t) = vk(t) (26)
v_ s1(t) = as1(t) − vs1(t)vs2(t) (27)
v_ sk(t) = ask(t) − vsk(t)vsk+1(t) , (28)
with the accelerations for all particles a(t), as1(t), ask(t) defined as
a(t) = − 1 ∇ L(x1(t),...,xN(t)) (29)
as1(t) = 1 Q1 ( mi vi2(t) − NTtarget ) (30)
ask(t) = 1 Qk ( Qk−1 vk−12(t) − Ttarget ) . (31)
The acceleration of the real particles in Eq. (29) is proportional to the negative gradient of the loss with respect to the weights and biases in the neural network.

Given Eqs. (24–31), the trajectories of the real and virtual particles, and thus the evolution of neural network weights and biases over training time, can be determined via numerical integration. Further details on implementation and model architecture are given in Supplementary Methods.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "** What are the noteworthy results?\nThe work proposed and validated an idea of \"sufficient training\": It demonstrates better generalization performance by\ntraining neural networks to get \"good enough\" (near-optimal, higher training loss) solutions, instead of adopting the usual\noptimal ones on training set (i.e. lowest training loss).\nThe work achieves \"sufficient training\" using a physics motivated optimizer: it models neural network parameters as particles\nin a simulated physical molecular system and the loss function as the system potential. The negative gradient of the loss\n(obtained by automatic differentiation, same as backpropagation) pushes the \"particles\" (network parameters) towards a\nminimum of the loss function, similar to a force on the particles. A Nosé-Hoover chain (NHC) thermostat is used to control the\ntemperature of this physical system by introducing virtual particles that interact with the network parameters. The NHC\nallows generating networks (and form ensembles) at any target temperature and thus control the level of fitting to the training\nset.\nThe idea of \"sufficient training\" is validated on two regimes: 1. retrofitting a pretrained neural network and reducing its\noverfitting on training set. This starts with a pretrained model at temperature 0 and gradually increases temperature. 2. \"Ab\ninitio\" training: applying the above described sufficient training optimizer from the start of training. The system is trained at a\nfixed target temperature throughout training. The authors show that in both regimes, the proposed sufficient training optimizer\nbrings benefits to the generalization performance compared to the baseline Adam optimizer.\nThe result is validated on two tasks on a few datasets: 1. classification: MNIST, HIGGS, IRIS datasets 2. Regression: fitting a\nnoisy sine curve, Auto-MPG (S) single-variate regression, Auto-MPG (M) multivariate regression.\n** Will the work be of significance to the field and related fields? How does it compare to the established literature? If the\nwork is not original, please provide relevant references.\nThe general topic of sufficient training may be potentially interesting. However, I believe there are a few more things to be\naddressed in order to fully convince the general machine learning community.\n1. Most people agree that after some level of training on a small dataset, neural networks can overfit and test errors can\nincrease. The standard approach around overfitting in the current big data era is to use larger datasets and more data\naugmentation. Therefore, the significance of this approach (or any approach mitigating overfitting) depends on the problem\ndomain. In general, overfitting seems not a significant problem in most big data domains. For demonstrating your approach,\nit may be useful to find some domains where it is always in a small data regime and training data cannot be obtained and/or\naugmented easily (like some specific astronomy, biology, medical domains).\n2. The proposed sufficient training, although motivated by physical systems, is essentially an approach that tradeoff training\nperformances with test performances. A baseline comparison to other simpler approaches may be needed to establish the\nnecessity of your less simple physics formulations. For example, how does some simple noise injection schemes to\nparameters or combinations of early stopped models work compared to your formulation?\nTo make significant contribution in machine learning, it may benefit from closely investigating the problem and identifying the\nkey cause of overfitting instead of only proposing a large physics-based formulation and claim that it helps mitigating\noverfitting. This may require simplifying your system and identifying minimal components that make it work, which will\nprovide more insight to machine learning researchers who do not share a physics background/interest.\n** Does the work support the conclusions and claims, or is additional evidence needed?\nAs discussed above, I believe the comparison with similar techniques or simpler alternative is lacking. It is not yet\ndemonstrated that this physics-motivated formulation is really necessary. Not enough simpler alternatives are discussed and\ncompared.\nSome claims in the main text and abstract are a bit exaggerated. The author seems to suggest that \"sufficient training\" is\ndifferent from optimizing. A more reasonable view is that \"sufficient training\" is just another way of optimizing, since your\napproach is essentially a different type of optimizer, which rely on gradients computed by automatic differentiation via\nbackpropagation. The way particles are pushed by forces of gradients in your system is essentially optimizing. The tone of\n\"creating a new paradigm different from optimization\" needs to be soften to avoid misleading if you still rely on\nbackpropagation/automatic differentiation.\n** Are there any flaws in the data analysis, interpretation and conclusions? Do these prohibit publication or require revision?\nOne thing I notice is that the results of your approach are all obtained with ensemble models. It is expected that ensemble\nmodels can benefit generalization, it may be fairer to compare with baselines in a non-ensemble fashion if possible. Or you\ncan establish an ensemble baseline using popular ensemble techniques.\nThis ensemble vs. single model comparison issue need to be addressed in order to support the claim that your approach is\nbetter than baseline.\n** Is the methodology sound? Does the work meet the expected standards in your field\nSome general comments about the direction/approach are given in the above section about significance.\nRegarding details and execution, I feel that the datasets used are on a simple/toy or uncommon side. MNIST, sine fitting,\nIRIS, etc. are all relatively toy datasets in machine learning nowadays. CIFAR10 and CIFAR100 have the same format as\nMNIST and are less \"toy\" --- it is pretty standard in the field to try on them. Optionally, if you can obtain some results on\nadvanced tasks like modern NLP and ImageNet, it would make your work much more solid.\n** Is there enough detail provided in the methods for the work to be reproduced?\nI believe there is enough detail. Also the code is provided.", "reviewer_2": "In this manuscript, the authors present a novel algorithm, termed \"simmering,\" for training neural networks. By leveraging\nNose-Hoover chain thermostats, a concept originating from molecular dynamics, the proposed approach aims to address the\npersistent challenge of overfitting in deep learning. While the idea is intriguing and has potential, the current work lacks\nsufficient experimental evidence to fully support the claims made.\nThe experiments conducted primarily utilize synthetic data or relatively simple datasets, including MNIST, HIGGS, and IRIS,\nin conjunction with baseline architectures such as LeNet-5. While these initial results provide some insight into the potential\nof the proposed method, they are insufficient to establish its robustness and generalizability. To strengthen the conclusions, I\nstrongly recommend that the authors extend their evaluation to more complex and realistic datasets, such as ImageNet, and\nincorporate benchmark neural network architectures, such as Transformer or ResNet-101.\nFurthermore, the manuscript would benefit from a more thorough comparison of the proposed method against existing\nregularization techniques commonly used in deep learning, particularly in mitigating overfitting. Examples include weight\ndecay, dropout, and early stopping. Such comparisons are essential to position the proposed approach within the existing\nbody of work and to demonstrate its advantages or limitations.\nFinally, the clarity of the methodological description should be improved. As it stands, the explanation of the proposed\napproach is not sufficiently detailed to enable reproducibility. I recommend the authors provide a more comprehensive and\nprecise account of the algorithm, including its implementation details and overall pipeline.\nIn summary, while the manuscript introduces a potentially promising approach, significant revisions are needed to\nsubstantiate the claims and enhance the clarity and reproducibility of the work. With these improvements, the study could\nmake a more meaningful contribution to the field."}, "author_response": "ManuscriptID: NCOMMS-25-24798-T\nTitle: Simmering: sufficientisbetterthanoptimalfortrainingneuralnetworks\nAuthors: IrinaBabayan,HazhirAliahmadi,GregvanAnders\nResponsetoRefereeReports\nWe thank both Referees for their constructive feedback on the manuscript. The Referees made a number\nof suggestions thatprompted us to significantly extend our analysis and to clarify the presentation of our ap\nproach. We believethe new analyses (of CIFAR-10 and a transformer-based translation task) better situate\nour approach in themachine learning literature.\nThereportsarereproducedbelowintheirentirety. Thequotationsareverbatimandindentedinblue,sans-\nserif font. In some places we quote text directly from the revised manuscript using red, serif font; text is\nreproducedverbatimexceptinplaceswhereitinvolvesreferencenumbers,which,sincethisreportdoesnot\nappearwiththecontextofthefullreferencelist,wehavegivenauthor-datepairsinstead.\nInaddition,wecorrectedsomeminorspellingandgrammaticalerrors. Wealsomadesubstantialclarifica-\ntionstotheDiscussionandbeginningoftheMethods.\nFollowingtheresponsetotheReferees, weappendahighlightedversionofthemanuscriptthatindicates\nthechangeswehavemadeinresponsetotheReferees’suggestions,andformattingissues.\nWehopethatyouwillfindtherevisedmanuscriptsuitableforpublicationinNatureCommunications.\nReport of Referee 1\nFromthereport:\n**Whatarethenoteworthyresults?\nThe work proposed and validated an idea of \"sufficient training\": It demonstrates better gen-\neralization performance by training neural networks to get \"good enough\" (near-optimal, higher\ntraining loss) solutions, instead of adopting the usual optimal ones on training set (i.e. lowest\ntrainingloss).\nTheworkachieves\"sufficienttraining\"usingaphysicsmotivatedoptimizer: itmodelsneuralnet-\nworkparametersasparticlesinasimulatedphysicalmolecularsystemandthelossfunctionas\nthe system potential. The negative gradient of the loss (obtained by automatic differentiation,\nsame as backpropagation) pushes the \"particles\" (network parameters) towards a minimum of\nthe loss function, similar to a force on the particles. A Nosé-Hoover chain (NHC) thermostat is\nusedtocontrolthetemperatureofthisphysicalsystembyintroducingvirtualparticlesthatinteract\nwiththenetworkparameters. TheNHCallowsgeneratingnetworks(andformensembles)atany\ntargettemperatureandthuscontroltheleveloffittingtothetrainingset.\nnetworkandreducingitsoverfittingontrainingset. Thisstartswithapretrainedmodelattemper-\nature0andgraduallyincreasestemperature. 2. \"Abinitio\"training: applyingtheabovedescribed\nsufficienttrainingoptimizerfromthestartoftraining. Thesystemistrainedatafixedtargettem-\nperature throughout training. The authors show that in both regimes, the proposed sufficient\ntraining optimizer brings benefits to the generalization performance compared to the baseline\nAdamoptimizer.\nThe result is validated on two tasks on a few datasets: 1. classification: MNIST, HIGGS, IRIS\ndatasets2. Regression: fittinganoisysinecurve,Auto-MPG(S)single-variateregression,Auto-\nMPG(M)multivariateregression.\nWethanktheRefereefortheircarefulexaminationofourresults.\nFromtheReport:\n** Will the work be of significance to the field and related fields? How does it compare to the\nestablishedliterature? Iftheworkisnotoriginal,pleaseproviderelevantreferences.\nThe general topic of sufficient training may be potentially interesting. However, I believe there\nare a few more things to be addressed in order to fully convince the general machine learning\ncommunity.\n1. Most people agree that after some level of training on a small dataset, neural networks can\noverfitandtesterrorscanincrease. Thestandardapproacharoundoverfittinginthecurrentbig\ndataeraistouselargerdatasetsandmoredataaugmentation. Therefore,thesignificanceofthis\napproach (or any approach mitigating overfitting) depends on the problem domain. In general,\noverfitting seems not a significant problem in most big data domains. For demonstrating your\napproach, it may be useful to find some domains where it is always in a small data regime and\ntrainingdatacannotbeobtainedand/oraugmentedeasily(likesomespecificastronomy,biology,\nmedicaldomains).\nWethanktheRefereeforthisvaluableopportunitytodiscusstherelevanceandapplicabilityofsufficient\ntrainingintheeraofbigdata.\nWewillstartbyaddressingtherelevanceofalternativetrainingparadigms,likesufficienttraining,inlarge\ndata regimes. Although the proportionality between a model’s size (in this case, the number of trainable\nneural network parameters) and the training dataset size plays a key role in overfitting risk, both computa-\ntionalandtheoreticaladvancementshaveallowedmodelcapacitygrowthtokeeppacewithdatasetgrowth.\nState-of-the-artarchitecturessuchasResNet,Transformer-basedmodelslikeBERTandGPT,andlargecon-\nvolutionalnetworkshavebeendevelopedwithdegreesoffreedominthehundredsofmillionstobillionsto\naddress real-world problems in vision, language, and speech. Nonetheless, overfitting remains a persistent\nobstacle to generalizability even in these “big data” regimes. For example, the “double descent” training\nphenomenon shows that increasing model capacity beyond what is necessary for interpolation can initially\nworsen unseen data prediction error before performance improvements occur, (Belkin et al., 2019) under-\nscoringthattheclassicaloverfittingunseendataloss“peak”persistsinmodernsettings. Thegeneralization\nadvantageobservedbyusingdropoutwhenlearningdatasetslikeImageNetorCIFAR-10indicatesthateven\nlargemodelslearningfromlargedatasetscanbenefitsignificantlyfromregularizationtocombatoverfitting.\n(Srivastavaetal.,2014)However,evenunderregularization,moderndeepnetworkshavebeenshowntohave\nthecapacitytoperfectlymemorizerandomlabelsandnoise,(Zhangetal.,2017)demonstratingtheirmassive\noverfitting capacity. Overall, overfitting continues to manifest when employing optimization, even in “big\ndata”regimes.\noverfitting mitigation technique aimed at combating the potential mismatch between model capacity and\ndatasetsize. WhileDAcanreducemodelvarianceandtherebyimprovegeneralizability,itisnotgenerically\nsuitableforlearningcomplexdata. Bymakingthedatacovariancemoreisotropic,DAcaneraseinformative\ndatastructuresinhigh-dimensionalfeaturespacesandthusincreasemodelbias,particularlywhenappliedtoo\naggressivelyortooverparameterizedmodels.Certainaugmentationscanalsointroducetrain–testdistribution\nshifts,harmingperformance,especiallyinregressiontasks. (Linetal.,2024)Moreover,althoughDAtends\ntoimproveamodel’saverageclassificationaccuracy,thiscancomeattheexpenseofsubstantialdrops(upto\n20%)inindividualclassaccuracy,asobservedonImageNet. (Kirichenkoetal.,2023)Finally,whileDAhas\nshownclearbenefitsfornon-pretrainedmodelsandinsmall-dataregimes,recentwork(Longpreetal.,2020)\nhas found that these accuracy gains often do not translate to modern pretrained Transformer architectures,\nwheretask-agnosticDAfailstoconsistentlyimproveperformance.\nOverall, overfitting is not a training phenomenon that is exclusive to small-data domains. Furthermore,\nemployingdataaugmentationdoesnotconsistentlyimprovegeneralizability,particularlywhenlearninghigh-\ndimensionaldataorwithmodernarchitectures.\nWethanktheRefereeforraisingthisimportantpoint;accordingly,wehaveaddedthefollowingsectionto\ntheintroduction:\nEveninlarge-andaugmented-dataregimes,withcorrespondinglyadvancedarchitecturessuchasResNet\n(Heetal.,2016)andTransformers,(Vaswanietal.,2017)modernnetworkscanmemorizenoiseorrandom\nlabelswithnegligibletrainingerrorwhilestillexhibitingpoorgeneralization,(Zhangetal.,2017)indicating\nthat overfitting is a fundamental learning challenge that transcends model and dataset size. (Zhang et al.,\n2017;Belkinetal.,2019;Srivastavaetal.,2014)\nFromtheReport:\n2. The proposed sufficient training, although motivated by physical systems, is essentially an\napproach thattradeoff training performances with testperformances. Abaseline comparisonto\nothersimplerapproachesmaybeneededtoestablishthenecessityofyourlesssimplephysics\nformulations. For example, how does some simple noise injection schemes to parameters or\ncombinationsofearlystoppedmodelsworkcomparedtoyourformulation?\ntheRefereepresentsavaluableopportunityforcomparisonwithotheroverfittingavoidancemethods.\nWe will start by addressing the comparison with noise injection. Generically, one cannot know a priori\nwhat noise process has affected the input and target data in the training set, unless additional information\nabout the ground truth is available. Thus, the performance of noise injection is highly domain- and task-\nspecific. We recognize that noise injection can nonetheless approximate the noise distribution of a dataset\neffectively. If the target data of a training dataset has additive Gaussian noise, and the neural network loss\nfunctionischosentobemean-squarederror,thenthepredictionuncertaintydistributionofsufficienttraining\nisageneralizationofthatofBayesianinferencewithnoiseinjection,asBayesianinferenceisaspecialcase\nof maximum entropy distribution sampling (at T = 1). (Neal, 1993) We have expanded our description in\nMainMethodstohighlighttherelationshipbetweennoiseinjectionandsufficienttrainingbutnotethatthis\ncorrespondence is specific to a particular noise distribution and loss function choice and is not generically\npresent. TheMainMethodsnowreads:\n...IntheparticularcasewherethetrainingdatatargetshaveadditiveGaussiannoiseϵ∼N(0,σ2),andwe\nt\nchoosemean-squarederror(MSE)asthelossfunction,thepredictionuncertaintydistributionisstructurally\n3\nsimilar to that of Bayesian inference with noise injection, but generalized to any β = 1/T. However, this\nequivalence between noise injection and sufficient training is a special case; in general sufficient training\ndoesnotassumeaformforthenoisedistribution. Giventhistrainingdatatargetnoisedistributionandloss\nfunctionchoice,p(t′|z′,θ)becomes\n(cid:18) (cid:19)1\np(t′|z′,θ)= 2π 1 σ2 2 e − 2σ 1 t 2 (y(z′;θ)−t′)2 . (21)\nt\nThen,wecanexpandy(z′;θ)aroundthefreeenergy-minimizingθ =0as\ny(z′;θ)≈y(z′;0)+ ∂y ∂ (z θ ′;θ) (cid:12) (cid:12) (cid:12) (cid:12) θ α . (22)\nα θ=0\nUsingEquation(21)andtheTaylor-expansionofy(z′;θ)inEquation(22),wecanevaluateEquation(20)to\nbe\np(t′|z′,D)∝e−(t′−y\n2\n(\nσ\nz\n2\n′;0))2\n(23)\nwhereσ2 =σ t 2+g αγ (β,D)−1 (cid:12) (cid:12) θ=0 ∂y ∂ ( θ z α ′;θ) (cid:12) (cid:12) (cid:12) θ=0 ∂y ∂ (z θγ ′;θ) (cid:12) (cid:12) (cid:12) θ=0 .Innoiseinjection(Matsuoka,May-June/1992;\nWright, 1999), β is not a parameter, but is implicitly set to 1.(Neal, 1993) However, in sufficient training,\nthevarianceequationgeneralizestoanypositivenon-zeroinversetemperatureβ. Thevarianceσ2ofthepre-\ndictionuncertaintydistributionhastwoterms: thevarianceintroducedbyuncertaintyindatatargetmeasure-\nments,σ2,andavariance-likequantitydefinedbythesensitivityofthefreeenergyandthemodelpredictions\nt\nto changes in θ. The overall effect of temperature on the second variance term is to reduce the FIM, g ,\nαγ\nthereby increasing the second variance term contribution, but the exact way in which changing g (β,D)\nαγ\naffects∂ yisproblem-dependent.\nθ\nWehopethatthischangeclarifieshowsufficienttrainingrelatestonoiseinjectionincertaintrainingprob-\nlems.\nNext,weaddressthesuggestiontocomparesufficienttrainingtoacollectionofearlystoppedmodels. We\nthank the Referee for this suggestion, and have added a comparison between ab initio simmering and both\nasingularearlystoppedmodelandacombinationofindependentlytrainedearlystoppedmodels,aswellas\nwith dropout, trained on the CIFAR-10 dataset and the Portuguese-English TED talk transcript translation\ndataset.(Yeetal.,2018)Therevisedmanuscript,intheResultssection,nowreads:\nEarlystopping(single) 71.90 39.62(67epochs)\nEarlystopping(ensemble) 73.81 41.42(53.1epochs–average)\nDropout 75.53 39.86(60epochs)\nAbinitiosimmering 82.05 43.46(21epochs)\nTable1: Simmeringoutperformsearlystopping,ensembledearlystoppinganddropoutontheCIFAR-\n10(Krizhevsky&Hinton,2009)andPortuguese-English(PT-EN)TEDtalktranscripttranslation(Ye\netal.,2018)datasets. FortrainingontheCIFAR-10dataset,allfourlearningcasesemployedtheConvNet\narchitecture(Krizhevsky,Sutskever,etal.,2012),thestandardtrain/testsplit.Forthedropoutcase,weapplied\nthedropoutratesreportedinRef.(Srivastavaetal.,2014). WeusedTensorFlow’simplementationsofearly\nstopping and dropout were, paired with stochastic gradient descent(Martín Abadi et al., 2015) to optimize\ntheearlystoppedanddropout-equippedmodels. Theearlystoppingensembleconsistsof100independently\ntrainedearlystoppedmodels,wheretheaveragetrainingtimewas14.56epochs. Wechosethesingleearly\nstopped model by its proximity to the average accuracy of the ensemble members (71.60%). Dropout and\nabinitiosimmeringeachtrainedfor20epochs,andweincludedthemodelscorrespondingtothelast2000\nweightupdatesinthesimmeringensemble.Atthetestingstage,sinceithasnoexplicitensembletoaggregate,\nwe took dropout’s “inference mode” prediction to be its ensemble prediction. (Hara et al., 2016) We used\nmajorityvotingtoaggregatetheearlystoppingandsimmeringensembles. Forlearningthetranslationtask,\nwetrainedareducedversion(describedinSupplementaryMethods)oftheTransformerarchitecturepresented\nin Ref. (Vaswani et al., 2017) with a pre-trained BERT tokenizer from TensorFlow’s model database, and\nassessed translation accuracy via teacher-forced token prediction accuracy. We used the same batch size\nand fixed learning rate for all cases, and used the Adam optimizer(Kingma & Ba, 2017) for all the non-\nsimmeringtrainingscenarios. Theearlystoppedensembleconsistsof10independentlytrainedmodels,with\nanaveragetrainingtimeof53.1epochs,aggregatedwithmajorityvotingasforCIFAR-10. Thesingleearly\nstopped model is presented as a characteristic example of the ensemble members. Based on the average\nearly stopped training time, we trained a model with dropout for 60 epochs and determined its “ensemble\ntestaccuracy”inthesamewayasforCIFAR-10. Wefoundthatthesimmeringensembleexceededthetest\naccuracyofallothercasesinonly21trainingepochs,withamajority-votedensemblepredictionderivedfrom\nitslast200sampledmodels. Thus,forequivalenttrainingtime,abinitiosimmeringproducesmoreaccurate\npredictionsthanbothsingle-modelandensembledoverfittingmitigationtechniquesfortheCIFAR-10dataset.\nHowever, simmering can also both accelerate and exceed the accuracy of other overfitting techniques in a\nnaturallanguageprocessingtasklikelanguagetranslation.\nEmployingaminimallybiasedmodeloftrainingdataerrorallowsabinitiosimmeringtooutperformother\ncommonly used overfitting avoidance methods in two different learning tasks: image classification with a\nconvolutional neural network and language translation with a Transformer. Table 1 shows that, for both\nlearningtasks,initiosimmeringachievesahighertestdataaccuracythanstandardimplementationsofearly\nstoppinganddropout. Consistentwithensemblelearningapproaches,abinitiosimmeringoutperformsasin-\ngularoptimizedmodeltreatedwithearlystopping. ForCIFAR-10,givenanequaltrainingtimeof20epochs,\nab initio simmering also outperforms two other ensemble learning methods: ensembled early stopping, an\nensembleofindependentlyoptimizedneuralnetworks,anddropout,anensembleofsimultaneously-trained\nthinnedneuralnetworks.Wefindthatsimmeringyieldsatestaccuracyofover82%in20epochs,whereasno\nothermethodexceeded76%testaccuracyonthesamebasis. Forthetranslationproblem,simmeringoutper-\nformsallthreeoverfittingmitigationmethodswithashortertrainingtime. Usingfixed, equivalentlearning\nrates,simmeringexceededtheaccuracyofbothdropoutandensembledearlystoppinginasmallfractionof\nthetrainingtime(21epochsforsimmering,versus60epochsfordropoutandanaverageof53.1epochsfor\nensembled early stopping). These comparisons highlight the role of simmering’s distinct minimally-biased\nensemblegenerationinefficientlyproducinggeneralizablepredictions.\nBeyond generalizable prediction, simmering also yields prediction uncertainty estimates that reflect data\n5\ndifficulty. Figure 3showshowthethreeensemblelearningapproachespredictontwodifferentCIFAR-10\ntestsamples(Fig.3a–b). Thoughensembleuncertaintyestimatescanbeextractedeitherdirectlyorindirectly\n(see Supplementary Methods) from all three ensemble learning approaches, the ensemble generated by ab\ninitio simmering allows for uncertainty estimates that are proportional to image difficulty as shown in Fig.\n3c–d.Thus,themaximumentropydistributionsamplingthatsupportssimmeringprovidesbothgeneralizable\npredictionsandadditionalinsightintotheneuralnetwork’slearningprocess.\nFigure 3: Ab initio simmering correctly classifies CIFAR-10 test images correctly while also yielding\nconfidenceestimatesthatcorrespondtoimagedifficulty. Earlystopping(greenbars),dropout(bluebars)\nand ab initio simmering (pink bars) ensembles predict an “easy” image of an automobile (panel b), with a\nclear foreground and with the automobile in full view, with similarly high confidence (high proportion of\nensemblevotes). However,whentaskedwithclassifyinga“difficult”testimageofatruck(panela),which\npresents the truck at close range and with low contrast with the background, early stopping remains nearly\nequally confident (panel c, green bar). In contrast, simmering predicts the correct class, but with lower\nconfidence, reflecting the difficulty of identifying the truck in the image (see Supplementary Methods for\nimagedifficultyquantification). Simmering’ssecond-mostvotedpredictionbearsrelationtothecorrectclass\n(“automobile”and“truck”havesimilarimagefeatures). Thoughdropoutalsopredictsuncertainly,itpredicts\nthe correct label with near-equivalent confidence to most other classes. Thus, ab initio simmering predicts\nimagesofvaryingdifficultyaccuratelywithconfidenceproportionaltoimagedifficulty.\nWethanktheRefereeforthissuggestionandhopethatourrevisionclarifiesthedifferenceinperformance\nbetweenthementionedoverfittingavoidanceschemesandsufficienttraining.\nFromtheReport:\nproblem and identifying the key cause of overfitting instead of only proposing a large physics-\nbasedformulationandclaimthatithelpsmitigatingoverfitting.\nThismayrequiresimplifyingyoursystemandidentifyingminimalcomponentsthatmakeitwork,\nwhich will provide more insight to machine learning researchers who do not share a physics\nbackground/interest.\nWethanktheRefereefortheopportunitytoclarifythepresentationofsufficienttraininginourmanuscript.\nFirst,weaddressthequestionraisedbytheRefereeabouthowourapproachmitigatesoverfitting.\nIn the introduction of the manuscript, we highlight that one of the main advantages of neural networks\nis their universal estimation capability, which is supported by their high-dimensional (over-parameterized)\nparameterspaces. Inproblemswithnoisydata,auniversalestimatorlikeaneuralnetworkcanproduceeither\nageneralizablerepresentation,thathashighertrainingerrorbutpredictsunseendatamoreaccurately,andan\noverspecifiedone,whichhaslowertrainingerrorbutpoortestperformance. Anoptimizationalgorithmwill\nalwaysseekthelower-errorrepresentationofthedata,andwillalwaysbeoverspecifiedifsufficientlyeffective\nattraversingthelosslandscape. Otherworksreferencedintheintroduction((Livnietal.,2014;Ruder,2017;\nGhojogh & Crowley, 2023)) also identify overfitting as a symptom of treating over-parameterized neural\nnetworkswithoptimization. Thus,asdescribedintheResultssection“Sufficienttrainingbysimmering”,to\navoidoverfittingentirely,wecanseeknon-optimalneuralnetworkparametersfromtheoutset.\nThemechanismofoverfittingavoidancecanbeshownintwoways: viathetrainingdynamicsandvialoss\nlandscapecurvatureanalysis. WehaveexpandedbothdescriptionsinMethods. Wehighlightthekeyfeatures\nofsufficienttrainingthatsupportoverfittingavoidanceintheseextendeddescriptions.\nTheextendeddescriptionofthetrainingdynamicsinMainMethodsnowreads:\nTheparticles’equationsofmotionarederivedfromtheNHCHamiltonian.(Martynaetal.,1996;Tucker-\nmanetal.,1992)Forsimplicityofnotation,wewillhenceforthsetm = 1,Q = 1∀i,k anddescribethe\ni k\nequationsofmotionintermsofthepositionsandvelocities,ratherthanthepositionsandmomenta. Thereal\nparticles’positionsevolveaccordingto(Jang&Voth,1997)\nx˙ (t)=v (t) (24)\ni i\nv˙ (t)=a (t)−v (t)v (t), (25)\ni i s1 i\nwheretheisubscriptdenotesrealparticlequantities, andthes subscriptdenotesaquantityaffiliatedwith\nk\nthe kth virtual particle. The v (t)v (t) term from the NHC in Equation (25) distinguishes simmering\ns1 i\nfromgradient-basedoptimization. Forexample, inmomentumgradientdescent, Equation(25)wouldhave\nv (t) = const. (França et al., 2020) In contrast, v (t) in NHC dynamics can take on both positive and\ns1 s1\nnegativevaluesovertime.\nThevirtualparticledynamicsaregivenby\ns˙ (t)=v (t) (26)\nk sk\nv˙ (t)=a (t)−v (t)v (t) (27)\ns1 s1 s2 s1\nv˙ (t)=a (t)−v (t)v (t), (28)\nsk sk sk+1 sk\n7\nwiththeaccelerationsforallparticlesa (t),a (t),a (t)definedas\ni s1 sk\n1\na (t)=− ∇ L(x (t),...,x (t)) (29)\ni m xi 1 N\ni\na (t)= 1 (cid:0)(cid:88) m v2(t)−NT (cid:1) (30)\ns1 Q i i target\n1\ni\na (t)= 1 (cid:0) Q v2 (t)−T (cid:1) . (31)\nsk Q k−1 k−1 target\nk\nThekeytakeawayofthisimplementationofsufficienttrainingistheintroductionofadynamicalcoefficient\nofv (t),v (t),thatvariesovertime. Thisway,theaccelerationsoftheneuralnetworkweightsandbiases\ni s1\nareneverentirelydeterminedbythegradientofthelossandareinsteadcontinuouslydriventonon-optimal\nlossbythedynamicsinv˙ ,whichdependonthetemperatureT.\ns1\nAnother interpretation of simmering’s overfitting avoidance is through loss landscape curvature analysis\nanalogous to mathematical arguments that support L -norm regularization. We have extended the existing\n2\ncomparisonintheMainMethods,whichnowreads,\nTheFIMdescribestherelative“proximity”ofθtothefreeenergyminimumθ =0inparameterspaceasa\nfunctionofβ =1/T. Thesquareddisplacementds2betweenθandθ =0is\n1 (cid:88)\nds2 ∝ λ dλ2 , (16)\nT j j\nj\nwhere λ are the eigenvalues of the Hessian of F(θ,D) that describe the curvature of F along different\nj\nmodes in parameter space. Distances (Equation (16)) in a finite-temperature system diverge for small T,\nexceptalongdirectionscorrespondingto“sloppymodes”forwhichthecorrespondingeigenvaluesλ → 0.\nj\nSimmering and other sufficient training implementations exploit this effect of temperature on distances in\nparameter space to systematically sample families of loss-equivalent networks located along these sloppy\nmodes. ...\n...This information geometric framing presented here also motivates the effectiveness of both ab initio\nsimmering and retrofitting at reducing overfitting in a neural network. The effect of T on the Hessian of\nF(θ,D)(showninEquation(16))isanalogoustotheeffectoftheregularizationstrengthparameter,typically\ndenoted λ, on the eigenvalues of the Hessian of the training loss in L -norm regularization.(Hastie et al.,\n2\n2009)Usingfinite-temperatureparametersampling,bothabinitiosimmeringandretrofittingrescaledistances\ninparameterspaceinthesamemannerasL -normregularization,whichhasalreadybeenshowntoreduce\n2\noverfitting.(Hastieetal.,2009)\nWe hope that the dynamical angle and comparison with the mathematical arguments supporting another\nwell-knownoverfittingmitigationtechniqueclarifythekeycomponentsofsufficienttrainingthatdistinguish\nitfromoptimizationandthusgenericallyavoidoverfitting.\nFromtheReport:\n8\n**Doestheworksupporttheconclusionsandclaims,orisadditionalevidenceneeded?\nAs discussed above, I believe the comparison with similar techniques or simpler alternative is\nlacking. Itisnotyetdemonstratedthatthisphysics-motivatedformulationisreallynecessary. Not\nenoughsimpleralternativesarediscussedandcompared.\nWehaverevisedthemanuscripttoincludeacomparisonwithasingularoptimizedmodel,aswellastwo\nensemble approaches, on the CIFAR-10 dataset and the Portuguese-English TED talk transcript translation\ndataset.(Yeetal.,2018). WeappreciatetheReferee’spromptforcomparison,andagreethatthiscomparison\nwithotheroverfittingmitigationmethodsbettermotivatesthenecessityofsufficienttrainingapproaches.The\nrevisedResultsnowread:\nMethod CIFAR-10TestAccuracy(%) PT-ENTranslationTestAccuracy(%)\nEarlystopping(single) 71.90 39.62(67epochs)\nEarlystopping(ensemble) 73.81 41.42(53.1epochs–average)\nDropout 75.53 39.86(60epochs)\nAbinitiosimmering 82.05 43.46(21epochs)\nTable1: Simmeringoutperformsearlystopping,ensembledearlystoppinganddropoutontheCIFAR-\n10(Krizhevsky&Hinton,2009)andPortuguese-English(PT-EN)TEDtalktranscripttranslation(Ye\netal.,2018)datasets. FortrainingontheCIFAR-10dataset,allfourlearningcasesemployedtheConvNet\narchitecture(Krizhevsky,Sutskever,etal.,2012),thestandardtrain/testsplit.Forthedropoutcase,weapplied\nthedropoutratesreportedinRef.(Srivastavaetal.,2014). WeusedTensorFlow’simplementationsofearly\nstopping and dropout were, paired with stochastic gradient descent(Martín Abadi et al., 2015) to optimize\ntheearlystoppedanddropout-equippedmodels. Theearlystoppingensembleconsistsof100independently\ntrainedearlystoppedmodels,wheretheaveragetrainingtimewas14.56epochs. Wechosethesingleearly\nstopped model by its proximity to the average accuracy of the ensemble members (71.60%). Dropout and\nabinitiosimmeringeachtrainedfor20epochs,andweincludedthemodelscorrespondingtothelast2000\nweightupdatesinthesimmeringensemble.Atthetestingstage,sinceithasnoexplicitensembletoaggregate,\nwe took dropout’s “inference mode” prediction to be its ensemble prediction. (Hara et al., 2016) We used\nmajorityvotingtoaggregatetheearlystoppingandsimmeringensembles. Forlearningthetranslationtask,\nwetrainedareducedversion(describedinSupplementaryMethods)oftheTransformerarchitecturepresented\nin Ref. (Vaswani et al., 2017) with a pre-trained BERT tokenizer from TensorFlow’s model database, and\nassessed translation accuracy via teacher-forced token prediction accuracy. We used the same batch size\nand fixed learning rate for all cases, and used the Adam optimizer(Kingma & Ba, 2017) for all the non-\nsimmeringtrainingscenarios. Theearlystoppedensembleconsistsof10independentlytrainedmodels,with\nanaveragetrainingtimeof53.1epochs,aggregatedwithmajorityvotingasforCIFAR-10. Thesingleearly\nstopped model is presented as a characteristic example of the ensemble members. Based on the average\nearly stopped training time, we trained a model with dropout for 60 epochs and determined its “ensemble\ntestaccuracy”inthesamewayasforCIFAR-10. Wefoundthatthesimmeringensembleexceededthetest\naccuracyofallothercasesinonly21trainingepochs,withamajority-votedensemblepredictionderivedfrom\nitslast200sampledmodels. Thus,forequivalenttrainingtime,abinitiosimmeringproducesmoreaccurate\npredictionsthanbothsingle-modelandensembledoverfittingmitigationtechniquesfortheCIFAR-10dataset.\nHowever, simmering can also both accelerate and exceed the accuracy of other overfitting techniques in a\nnaturallanguageprocessingtasklikelanguagetranslation.\nEmployingaminimallybiasedmodeloftrainingdataerrorallowsabinitiosimmeringtooutperformother\ncommonly used overfitting avoidance methods in two different learning tasks: image classification with a\nconvolutional neural network and language translation with a Transformer. Table 1 shows that, for both\nlearningtasks,initiosimmeringachievesahighertestdataaccuracythanstandardimplementationsofearly\nstoppinganddropout. Consistentwithensemblelearningapproaches,abinitiosimmeringoutperformsasin-\ngularoptimizedmodeltreatedwithearlystopping. ForCIFAR-10,givenanequaltrainingtimeof20epochs,\nab initio simmering also outperforms two other ensemble learning methods: ensembled early stopping, an\nensembleofindependentlyoptimizedneuralnetworks,anddropout,anensembleofsimultaneously-trained\nthinnedneuralnetworks.Wefindthatsimmeringyieldsatestaccuracyofover82%in20epochs,whereasno\nothermethodexceeded76%testaccuracyonthesamebasis. Forthetranslationproblem,simmeringoutper-\nformsallthreeoverfittingmitigationmethodswithashortertrainingtime. Usingfixed, equivalentlearning\nrates,simmeringexceededtheaccuracyofbothdropoutandensembledearlystoppinginasmallfractionof\nthetrainingtime(21epochsforsimmering,versus60epochsfordropoutandanaverageof53.1epochsfor\nensembled early stopping). These comparisons highlight the role of simmering’s distinct minimally-biased\nensemblegenerationinefficientlyproducinggeneralizablepredictions.\nBeyond generalizable prediction, simmering also yields prediction uncertainty estimates that reflect data\n9\ndifficulty. Figure 3showshowthethreeensemblelearningapproachespredictontwodifferentCIFAR-10\ntestsamples(Fig.3a–b). Thoughensembleuncertaintyestimatescanbeextractedeitherdirectlyorindirectly\n(see Supplementary Methods) from all three ensemble learning approaches, the ensemble generated by ab\ninitio simmering allows for uncertainty estimates that are proportional to image difficulty as shown in Fig.\n3c–d.Thus,themaximumentropydistributionsamplingthatsupportssimmeringprovidesbothgeneralizable\npredictionsandadditionalinsightintotheneuralnetwork’slearningprocess.\nFigure 3: Ab initio simmering correctly classifies CIFAR-10 test images correctly while also yielding\nconfidenceestimatesthatcorrespondtoimagedifficulty. Earlystopping(greenbars),dropout(bluebars)\nand ab initio simmering (pink bars) ensembles predict an “easy” image of an automobile (panel b), with a\nclear foreground and with the automobile in full view, with similarly high confidence (high proportion of\nensemblevotes). However,whentaskedwithclassifyinga“difficult”testimageofatruck(panela),which\npresents the truck at close range and with low contrast with the background, early stopping remains nearly\nequally confident (panel c, green bar). In contrast, simmering predicts the correct class, but with lower\nconfidence, reflecting the difficulty of identifying the truck in the image (see Supplementary Methods for\nimagedifficultyquantification). Simmering’ssecond-mostvotedpredictionbearsrelationtothecorrectclass\n(“automobile”and“truck”havesimilarimagefeatures). Thoughdropoutalsopredictsuncertainly,itpredicts\nthe correct label with near-equivalent confidence to most other classes. Thus, ab initio simmering predicts\nimagesofvaryingdifficultyaccuratelywithconfidenceproportionaltoimagedifficulty.\nduration,andachievesthehighesttestaccuracyinafractionofthetrainingtime,givenequalandfixedlearn-\ning rates, for the language translation task. Additionally, Fig. 3 shows that, beyond making generalizable\npredictions,theensemblepredictiondistributionreflectsqualitativesimilaritiesbetweenclassesandtheim-\nage difficulty. We hope that this revised comparison highlights that sufficient training is both accurate and\ninformativeincomparisontowhattheRefereedescribesassimplerapproaches.\nTheRefereeraisesaninterestingquestionregardingtherelationshipbetweenourapproachandoptimiza-\ntion. This issue is raised in three locations in the review. We believe it is most straightforward to discuss\ntheseremarkstogetherasagroup.\nTheRefereestates:\nSomeclaimsinthemaintextandabstractareabitexaggerated. Theauthorseemstosuggest\nthat \"sufficient training\" is different from optimizing. A more reasonable view is that \"sufficient\ntraining\" is just another way of optimizing, since your approach is essentially a different type of\noptimizer,whichrelyongradientscomputedbyautomaticdifferentiationviabackpropagation.\nFurther,theRefereeremarks:\nThewayparticlesarepushedbyforcesofgradientsinyoursystemisessentiallyoptimizing.\nFinallytheRefereenotes:\nThe tone of \"creating a new paradigm different from optimization\" needs to be soften to avoid\nmisleadingifyoustillrelyonbackpropagation/automaticdifferentiation.\nThroughtheseremarks,theRefereeraisesanimportantdistinction. Weappreciatetheopportunitytoclarify.\nAsGoodfellow,BengioandCourvillestateinChapter6oftheirtextbookDeepLearning: “Thetermback-\npropagationisoftenmisunderstoodasmeaningthewholelearningalgorithmformultilayerneuralnetworks.\nActually, back-propagation refers only to the method for computing the gradient, while another algorithm,\nsuch as stochastic gradient descent, is used to perform learning using this gradient.” The referee is correct\nthat we use backpropagation/automatic differentiation to compute gradients but instead of using them for\noptimization,weusegradientsforfiltration.\nIn our filter-based training approach, the fact that backpropagation/automatic differentiation for neural\nnetworks has already been implemented in TensorFlow makes it particularly convenient for computing the\nforcesweusetoimplementthefilter.\nOnewaytoidentifythe distinction betweenourapproachandoptimizationisbyreferringto theintegral\ntransformwepresentinmaintextEquation(1):\n11\n(cid:90)\nZ(β,D)= dNxe−βL(x,D) (1)\nMathematically,Z playstheroleofageneratingfunctionthatactsasalow-passfilterforloss(L)ratherthan\nasanoptimizer. Fromthisgeneratingfunction,onecouldinspecttheeffectiveforcesonmodelparameters,\ne.g.,intermsofcollectivecoordinatesθ,whichhavetheform\n1\n∇L(θ,D)− ∇S(θ,D),\nβ\nwherethesecondtermisaneffectiveentropicforce. Simulatingtheeffectoftheseeffectiveforcesonmodel\nparametersallowssufficienttrainingtosamplethesetsofmodelparametersthatcontributetoZ.\nTheseentropicforcescausesufficienttrainingtoexhibitdistincttrainingdynamicsfromoptimization. To\nhelpclarifythisdistinction,wehaverevisedthecorrespondingMethodssection,whichnowreads\nTheparticles’equationsofmotionarederivedfromtheNHCHamiltonian.(Martynaetal.,1996;Tucker-\nmanetal.,1992)Forsimplicityofnotation,wewillhenceforthsetm = 1,Q = 1∀i,k anddescribethe\ni k\nequationsofmotionintermsofthepositionsandvelocities,ratherthanthepositionsandmomenta. Thereal\nparticles’positionsevolveaccordingto(Jang&Voth,1997)\nx˙ (t)=v (t) (24)\ni i\nv˙ (t)=a (t)−v (t)v (t), (25)\ni i s1 i\nwheretheisubscriptdenotesrealparticlequantities, andthes subscriptdenotesaquantityaffiliatedwith\nk\nthe kth virtual particle. The v (t)v (t) term from the NHC in Equation (25) distinguishes simmering\ns1 i\nfromgradient-basedoptimization. Forexample, inmomentumgradientdescent, Equation(25)wouldhave\nv (t) = const. (França et al., 2020) In contrast, v (t) in NHC dynamics can take on both positive and\ns1 s1\nnegativevaluesovertime.\nThevirtualparticledynamicsaregivenby\ns˙ (t)=v (t) (26)\nk sk\nv˙ (t)=a (t)−v (t)v (t) (27)\ns1 s1 s2 s1\nv˙ (t)=a (t)−v (t)v (t), (28)\nsk sk sk+1 sk\nwiththeaccelerationsforallparticlesa (t),a (t),a (t)definedas\ni s1 sk\n1\na (t)=− ∇ L(x (t),...,x (t)) (29)\ni m xi 1 N\ni\na (t)= 1 (cid:0)(cid:88) m v2(t)−NT (cid:1) (30)\ns1 Q i i target\n1\ni\na (t)= 1 (cid:0) Q v2 (t)−T (cid:1) . (31)\nsk Q k−1 k−1 target\nk\nTheaccelerationoftherealparticlesinEquation(29)isproportionaltothenegativegradientofthelosswith\nrespecttotheweightsandbiasesintheneuralnetwork.\nNote that the factor of v in the expression for v˙ (t) marks a crucial distinction from optimization. In\ns1 i\ngradient-based optimization, e.g., momentum gradient descent, the v factor is a positive constant. Over\ns1\ntraining time, such a constant has a diminishing effect on the dynamics of the optimizer. However, for the\ns1\nofthisfactorpreventsourapproachfromoptimizing.\nItisalsopossibletoillustratethedistinctionbetweenoptimizationandourapproach(Equations(24)–(31))\nbyexaminingtrainingdynamics. Wegiveanexampleofthedynamicsofouralgorithminthelosslandscape\ninanewSupplementaryFig.1. TherevisedSupplementaryInformationnowreads:\nFigure 1: Simmering (panel b), an implementation of sufficient training, produces a distinct training\ntrajectory(blackcurve,panelb)fromtheAdamoptimizer(panela). TheAdamoptimizer(blackcurve,\npanela)traversesthetraininglosslandscape(colouredcontoursinpanela)perpendiculartothelosscontours,\nas expected in gradient-based optimization. In contrast, simmering travels along equivalent-loss (coloured\ncontours in panel b) directions in parameter space, aligning with the information-geometric interpretation\nof sufficient training presented in Main Methods. The Adam trajectory (panel a) corresponds to the Adam\nportionofMainFig.1a,andthesimmeringtrajectoryisa100-epochsliceofthesimmeringsamplingshown\nin Fig. 1a. The loss landscape and training trajectories have been projected onto the principal components\n(PCs)oftheAdamandsimmeringtrainingtrajectoriesinpanelsaandbrespectively.\nSupplementaryFigure 1,generatedusingthelossvisualizationmethodofRef.(Lietal.,2018),showsthe\ndifference in training dynamics between Adam,(Kingma & Ba, 2017) a gradient-based optimizer, and sim-\nmering,animplementationofsufficienttraining. TheAdamoptimizertravelsperpendiculartothelosscon-\ntours,wherassimmeringtravelsparalleltothedirectionofnear-equivalentloss.Thefirstprincipalcomponent\n(PC)ofAdamdescribes90.44%ofitstrajectoryvarianceinparameterspace, andthesecondPCdescribes\nanother6.12%. Forsimmering, thefirstandsecondPCsdescribe99.9428%and0.0324%oftheparameter\nspacetrajectoryrespectively. Inthelattercase,alargerproportionofthetrainingtrajectoryisdescribedby\nthe first PC. Since this PC points along the direction of equivalent loss in parameter space, Supplementary\nFig. 1b shows that sufficient training automatically identifies and samples along sloppy (equivalent-loss)\nmodesinmodelparameterspaceasdescribedinMainMethods.\nThisvisualizationshowsthatourapproachtraversesthelosslandscapealongthedirectionoftheprincipal\ncomponent, while undergoing transverse oscillation. These transverse oscillations are a clear signal that\ndistinguishesthebehaviourofourapproachfromanoptimizationalgorithmthatisconstructedtomigrateto\natraininglossminimumandhalt.\nOnecouldrecoveraformofoptimizationfromourapproachbymovingtotheβ → ∞limitofEquation\n(1). We do not do that. Instead, we specifically maintain finite β = 1/T during sampling. There is also a\nsenseinwhichonecouldconstructadistributionofparametersρ(x)thatoptimallyrepresentstheactionof\nthe filter at finite β. However, it is important to note that that approach represents constructing a sufficient\n13\nrepresentationofanon-optimalquantity(inthiscontext,non-minimaltrainingloss),whichdiffersstrongly\nin spirit from optimizing (which seeks the minimal training loss representation). Even so, instead of doing\nthat,wesimplysamplefromthefilterinEquation(1),employinggradientsasaconvenientwaytogenerate\na sample of a set of parameters that contributes to Z. Although our implementation of sufficient training\nemploys gradients, other implementations could use other methods to sample from the filter, e.g., by using\nrandomwalks.\nInaddition,wefurtherdiscussthemotivationforthedifferencesbetweenourapproachandoptimizationin\ntherevisedDiscussion.\n** Are there any flaws in the data analysis, interpretation and conclusions? Do these prohibit\npublicationorrequirerevision?\nOne thing I notice is that the results of your approach are all obtained with ensemble models.\nItisexpectedthatensemblemodelscanbenefitgeneralization,itmaybefairertocomparewith\nbaselinesinanon-ensemblefashionifpossible.Oryoucanestablishanensemblebaselineusing\npopularensembletechniques.\nThisensemblevs.singlemodelcomparisonissueneedtobeaddressedinordertosupportthe\nclaimthatyourapproachisbetterthanbaseline.\nWethanktheRefereeforthissuggestion.Itistruethatensembledpredictionbenefitsgeneralization,andwe\nagreethatbothanon-ensembleandensemblebaselinecanclarifythegeneralizabilityadvantageofsufficient\ntraining as an ensemble generation method. The revised manuscript provides the following comparison to\nasingular,earlystoppedmodelandtwoensembleapproachesfortwolearningproblems(CIFAR-10image\nclassificationwithaconvolutionalneuralnetworkandlanguagetranslationwithaTransformer):\nEarlystopping(single) 71.90 39.62(67epochs)\nEarlystopping(ensemble) 73.81 41.42(53.1epochs–average)\nDropout 75.53 39.86(60epochs)\nAbinitiosimmering 82.05 43.46(21epochs)\nTable1: Simmeringoutperformsearlystopping,ensembledearlystoppinganddropoutontheCIFAR-\n10(Krizhevsky&Hinton,2009)andPortuguese-English(PT-EN)TEDtalktranscripttranslation(Ye\netal.,2018)datasets. FortrainingontheCIFAR-10dataset,allfourlearningcasesemployedtheConvNet\narchitecture(Krizhevsky,Sutskever,etal.,2012),thestandardtrain/testsplit.Forthedropoutcase,weapplied\nthedropoutratesreportedinRef.(Srivastavaetal.,2014). WeusedTensorFlow’simplementationsofearly\nstopping and dropout were, paired with stochastic gradient descent(Martín Abadi et al., 2015) to optimize\ntheearlystoppedanddropout-equippedmodels. Theearlystoppingensembleconsistsof100independently\ntrainedearlystoppedmodels,wheretheaveragetrainingtimewas14.56epochs. Wechosethesingleearly\nstopped model by its proximity to the average accuracy of the ensemble members (71.60%). Dropout and\nabinitiosimmeringeachtrainedfor20epochs,andweincludedthemodelscorrespondingtothelast2000\nweightupdatesinthesimmeringensemble.Atthetestingstage,sinceithasnoexplicitensembletoaggregate,\nwe took dropout’s “inference mode” prediction to be its ensemble prediction. (Hara et al., 2016) We used\nmajorityvotingtoaggregatetheearlystoppingandsimmeringensembles. Forlearningthetranslationtask,\nwetrainedareducedversion(describedinSupplementaryMethods)oftheTransformerarchitecturepresented\nin Ref. (Vaswani et al., 2017) with a pre-trained BERT tokenizer from TensorFlow’s model database, and\nassessed translation accuracy via teacher-forced token prediction accuracy. We used the same batch size\nand fixed learning rate for all cases, and used the Adam optimizer(Kingma & Ba, 2017) for all the non-\nsimmeringtrainingscenarios. Theearlystoppedensembleconsistsof10independentlytrainedmodels,with\nanaveragetrainingtimeof53.1epochs,aggregatedwithmajorityvotingasforCIFAR-10. Thesingleearly\nstopped model is presented as a characteristic example of the ensemble members. Based on the average\nearly stopped training time, we trained a model with dropout for 60 epochs and determined its “ensemble\ntestaccuracy”inthesamewayasforCIFAR-10. Wefoundthatthesimmeringensembleexceededthetest\naccuracyofallothercasesinonly21trainingepochs,withamajority-votedensemblepredictionderivedfrom\nitslast200sampledmodels. Thus,forequivalenttrainingtime,abinitiosimmeringproducesmoreaccurate\npredictionsthanbothsingle-modelandensembledoverfittingmitigationtechniquesfortheCIFAR-10dataset.\nHowever, simmering can also both accelerate and exceed the accuracy of other overfitting techniques in a\nnaturallanguageprocessingtasklikelanguagetranslation.\nEmployingaminimallybiasedmodeloftrainingdataerrorallowsabinitiosimmeringtooutperformother\ncommonly used overfitting avoidance methods in two different learning tasks: image classification with a\nconvolutional neural network and language translation with a Transformer. Table 1 shows that, for both\nlearningtasks,initiosimmeringachievesahighertestdataaccuracythanstandardimplementationsofearly\nstoppinganddropout. Consistentwithensemblelearningapproaches,abinitiosimmeringoutperformsasin-\ngularoptimizedmodeltreatedwithearlystopping. ForCIFAR-10,givenanequaltrainingtimeof20epochs,\nab initio simmering also outperforms two other ensemble learning methods: ensembled early stopping, an\nensembleofindependentlyoptimizedneuralnetworks,anddropout,anensembleofsimultaneously-trained\n15\nthinnedneuralnetworks.Wefindthatsimmeringyieldsatestaccuracyofover82%in20epochs,whereasno\nothermethodexceeded76%testaccuracyonthesamebasis. Forthetranslationproblem,simmeringoutper-\nformsallthreeoverfittingmitigationmethodswithashortertrainingtime. Usingfixed, equivalentlearning\nrates,simmeringexceededtheaccuracyofbothdropoutandensembledearlystoppinginasmallfractionof\nthetrainingtime(21epochsforsimmering,versus60epochsfordropoutandanaverageof53.1epochsfor\nensembled early stopping). These comparisons highlight the role of simmering’s distinct minimally-biased\nensemblegenerationinefficientlyproducinggeneralizablepredictions.\nBeyond generalizable prediction, simmering also yields prediction uncertainty estimates that reflect data\ndifficulty. Figure 3showshowthethreeensemblelearningapproachespredictontwodifferentCIFAR-10\ntestsamples(Fig.3a–b). Thoughensembleuncertaintyestimatescanbeextractedeitherdirectlyorindirectly\n(see Supplementary Methods) from all three ensemble learning approaches, the ensemble generated by ab\ninitio simmering allows for uncertainty estimates that are proportional to image difficulty as shown in Fig.\n3c–d.Thus,themaximumentropydistributionsamplingthatsupportssimmeringprovidesbothgeneralizable\npredictionsandadditionalinsightintotheneuralnetwork’slearningprocess.\nFigure 3: Ab initio simmering correctly classifies CIFAR-10 test images correctly while also yielding\nconfidenceestimatesthatcorrespondtoimagedifficulty. Earlystopping(greenbars),dropout(bluebars)\nand ab initio simmering (pink bars) ensembles predict an “easy” image of an automobile (panel b), with a\nclear foreground and with the automobile in full view, with similarly high confidence (high proportion of\nensemblevotes). However,whentaskedwithclassifyinga“difficult”testimageofatruck(panela),which\npresents the truck at close range and with low contrast with the background, early stopping remains nearly\nequally confident (panel c, green bar). In contrast, simmering predicts the correct class, but with lower\nconfidence, reflecting the difficulty of identifying the truck in the image (see Supplementary Methods for\nimagedifficultyquantification). Simmering’ssecond-mostvotedpredictionbearsrelationtothecorrectclass\n(“automobile”and“truck”havesimilarimagefeatures). Thoughdropoutalsopredictsuncertainly,itpredicts\nthe correct label with near-equivalent confidence to most other classes. Thus, ab initio simmering predicts\nimagesofvaryingdifficultyaccuratelywithconfidenceproportionaltoimagedifficulty.\nTable1showsthatsimmeringoutperformsbothasingular,optimizedmodelandtwootherensemblepre-\ndictionapproaches.Thiscomparisonmotivatestheuseofminimally-biasedensemblegenerationinsufficient\ntrainingoverbothsingle-modelpredictionandotherimplicitandexplicitensemblecollectionmethods.\n16\n**Isthemethodologysound? Doestheworkmeettheexpectedstandardsinyourfield\nSomegeneralcommentsaboutthedirection/approacharegivenintheabovesectionaboutsig-\nnificance.\nRegardingdetailsandexecution,Ifeelthatthedatasetsusedareonasimple/toyoruncommon\nside. MNIST,sinefitting,IRIS,etc.areallrelativelytoydatasetsinmachinelearningnowadays.\nCIFAR10andCIFAR100havethesameformatasMNISTandareless\"toy\"—itisprettystandard\ninthefieldtotryonthem.Optionally,ifyoucanobtainsomeresultsonadvancedtaskslikemodern\nNLPandImageNet,itwouldmakeyourworkmuchmoresolid.\nTherefereeraisesaninterestingpointaboutthesetofdatasetschosentoillustratetheapproach. Weagree\nthat testing the method on one of the CIFAR datasets is useful for contextualizing the performance of the\nmethod against a standard benchmark, and have thus revised our manuscript to include the performance of\nab initio simmering on the CIFAR-10 dataset. We have also included an example of ab initio simmering\nperformance on a natural language processing task with a Transformer. The revised Results section now\nreads:\nMethod CIFAR-10TestAccuracy(%) PT-ENTranslationTestAccuracy(%)\nEarlystopping(single) 71.90 39.62(67epochs)\nEarlystopping(ensemble) 73.81 41.42(53.1epochs–average)\nDropout 75.53 39.86(60epochs)\nAbinitiosimmering 82.05 43.46(21epochs)\nTable1: Simmeringoutperformsearlystopping,ensembledearlystoppinganddropoutontheCIFAR-\n10(Krizhevsky&Hinton,2009)andPortuguese-English(PT-EN)TEDtalktranscripttranslation(Ye\netal.,2018)datasets. FortrainingontheCIFAR-10dataset,allfourlearningcasesemployedtheConvNet\narchitecture(Krizhevsky,Sutskever,etal.,2012),thestandardtrain/testsplit.Forthedropoutcase,weapplied\nthedropoutratesreportedinRef.(Srivastavaetal.,2014). WeusedTensorFlow’simplementationsofearly\nstopping and dropout were, paired with stochastic gradient descent(Martín Abadi et al., 2015) to optimize\ntheearlystoppedanddropout-equippedmodels. Theearlystoppingensembleconsistsof100independently\ntrainedearlystoppedmodels,wheretheaveragetrainingtimewas14.56epochs. Wechosethesingleearly\nstopped model by its proximity to the average accuracy of the ensemble members (71.60%). Dropout and\nabinitiosimmeringeachtrainedfor20epochs,andweincludedthemodelscorrespondingtothelast2000\nweightupdatesinthesimmeringensemble.Atthetestingstage,sinceithasnoexplicitensembletoaggregate,\nwe took dropout’s “inference mode” prediction to be its ensemble prediction. (Hara et al., 2016) We used\nmajorityvotingtoaggregatetheearlystoppingandsimmeringensembles. Forlearningthetranslationtask,\nwetrainedareducedversion(describedinSupplementaryMethods)oftheTransformerarchitecturepresented\nin Ref. (Vaswani et al., 2017) with a pre-trained BERT tokenizer from TensorFlow’s model database, and\nassessed translation accuracy via teacher-forced token prediction accuracy. We used the same batch size\nand fixed learning rate for all cases, and used the Adam optimizer(Kingma & Ba, 2017) for all the non-\nsimmeringtrainingscenarios. Theearlystoppedensembleconsistsof10independentlytrainedmodels,with\nanaveragetrainingtimeof53.1epochs,aggregatedwithmajorityvotingasforCIFAR-10. Thesingleearly\nstopped model is presented as a characteristic example of the ensemble members. Based on the average\nearly stopped training time, we trained a model with dropout for 60 epochs and determined its “ensemble\ntestaccuracy”inthesamewayasforCIFAR-10. Wefoundthatthesimmeringensembleexceededthetest\naccuracyofallothercasesinonly21trainingepochs,withamajority-votedensemblepredictionderivedfrom\nitslast200sampledmodels. Thus,forequivalenttrainingtime,abinitiosimmeringproducesmoreaccurate\npredictionsthanbothsingle-modelandensembledoverfittingmitigationtechniquesfortheCIFAR-10dataset.\nHowever, simmering can also both accelerate and exceed the accuracy of other overfitting techniques in a\nnaturallanguageprocessingtasklikelanguagetranslation.\ncommonly used overfitting avoidance methods in two different learning tasks: image classification with a\nconvolutional neural network and language translation with a Transformer. Table 1 shows that, for both\nlearningtasks,initiosimmeringachievesahighertestdataaccuracythanstandardimplementationsofearly\nstoppinganddropout. Consistentwithensemblelearningapproaches,abinitiosimmeringoutperformsasin-\ngularoptimizedmodeltreatedwithearlystopping. ForCIFAR-10,givenanequaltrainingtimeof20epochs,\nab initio simmering also outperforms two other ensemble learning methods: ensembled early stopping, an\nensembleofindependentlyoptimizedneuralnetworks,anddropout,anensembleofsimultaneously-trained\nthinnedneuralnetworks.Wefindthatsimmeringyieldsatestaccuracyofover82%in20epochs,whereasno\nothermethodexceeded76%testaccuracyonthesamebasis. Forthetranslationproblem,simmeringoutper-\nformsallthreeoverfittingmitigationmethodswithashortertrainingtime. Usingfixed, equivalentlearning\nrates,simmeringexceededtheaccuracyofbothdropoutandensembledearlystoppinginasmallfractionof\nthetrainingtime(21epochsforsimmering,versus60epochsfordropoutandanaverageof53.1epochsfor\nensembled early stopping). These comparisons highlight the role of simmering’s distinct minimally-biased\nensemblegenerationinefficientlyproducinggeneralizablepredictions.\n**Isthereenoughdetailprovidedinthemethodsfortheworktobereproduced?\nIbelievethereisenoughdetail. Alsothecodeisprovided.\nreferee#1(Remarksoncodeavailability):\nThe code has a README file and runs on default settings via one command line without any\nproblem. The code consists of a demo of applying the proposed approach on learning a noisy\nsinecurve.\nWe thank the Referee for taking the time to verify the reproducibility of our work by testing the code we\nprovided. Wehaverevisedtheopen-sourceimplementationtobeademonstrationofabinitiosimmeringon\ntheCIFAR-10dataset,whichthisrefereenotedisastandardbenchmarkinmachinelearning.\nWethanktheRefereefortheircarefulexaminationofthework.WebelievethatReferee’ssuggestionshave\nstimulatedchangesintherevisedmanuscriptthatsubstantiallyimproveitspresentation.\nReport of Referee 2\nInthismanuscript,theauthorspresentanovelalgorithm,termed\"simmering,\"fortrainingneural\nnetworks. By leveraging Nose-Hoover chain thermostats, a concept originating from molecular\ndynamics,theproposedapproachaimstoaddressthepersistentchallengeofoverfittingindeep\nlearning. While the idea is intriguing and has potential, the current work lacks sufficient experi-\nmentalevidencetofullysupporttheclaimsmade.\nThe experiments conducted primarily utilize synthetic data or relatively simple datasets, includ-\ningMNIST,HIGGS,andIRIS,inconjunctionwithbaselinearchitecturessuchasLeNet-5. While\nthese initial results provide some insight into the potential of the proposed method, they are in-\nsufficienttoestablishitsrobustnessandgeneralizability. Tostrengthentheconclusions,Istrongly\nrecommendthattheauthorsextendtheirevaluationtomorecomplexandrealisticdatasets,such\nasImageNet, andincorporatebenchmarkneuralnetworkarchitectures, suchasTransformeror\nResNet-101.\nmance on more complex architectures and datasets helps contextualize the utility of our approach for con-\ntemporaryuse.WehaveextendedourResultstoincludeacomparisonbetweenabinitiosimmeringandother\noverfitting mitigation techniques. The two learning tasks presented are: training on the CIFAR-10 dataset\nwith the ConvNet architecture, and performing Portuguese-English translation of TED talk transcripts (Ye\netal.,2018)withaTransformer. ThesetupofthecomparisonsisdescribedinSupplementaryMethods,and\ntheResultssectionnowreads\nMethod CIFAR-10TestAccuracy(%) PT-ENTranslationTestAccuracy(%)\nEarlystopping(single) 71.90 39.62(67epochs)\nEarlystopping(ensemble) 73.81 41.42(53.1epochs–average)\nDropout 75.53 39.86(60epochs)\nAbinitiosimmering 82.05 43.46(21epochs)\nTable1: Simmeringoutperformsearlystopping,ensembledearlystoppinganddropoutontheCIFAR-\n10(Krizhevsky&Hinton,2009)andPortuguese-English(PT-EN)TEDtalktranscripttranslation(Ye\netal.,2018)datasets. FortrainingontheCIFAR-10dataset,allfourlearningcasesemployedtheConvNet\narchitecture(Krizhevsky,Sutskever,etal.,2012),thestandardtrain/testsplit.Forthedropoutcase,weapplied\nthedropoutratesreportedinRef.(Srivastavaetal.,2014). WeusedTensorFlow’simplementationsofearly\nstopping and dropout were, paired with stochastic gradient descent(Martín Abadi et al., 2015) to optimize\ntheearlystoppedanddropout-equippedmodels. Theearlystoppingensembleconsistsof100independently\ntrainedearlystoppedmodels,wheretheaveragetrainingtimewas14.56epochs. Wechosethesingleearly\nstopped model by its proximity to the average accuracy of the ensemble members (71.60%). Dropout and\nabinitiosimmeringeachtrainedfor20epochs,andweincludedthemodelscorrespondingtothelast2000\nweightupdatesinthesimmeringensemble.Atthetestingstage,sinceithasnoexplicitensembletoaggregate,\nwe took dropout’s “inference mode” prediction to be its ensemble prediction. (Hara et al., 2016) We used\nmajorityvotingtoaggregatetheearlystoppingandsimmeringensembles. Forlearningthetranslationtask,\nwetrainedareducedversion(describedinSupplementaryMethods)oftheTransformerarchitecturepresented\nin Ref. (Vaswani et al., 2017) with a pre-trained BERT tokenizer from TensorFlow’s model database, and\nassessed translation accuracy via teacher-forced token prediction accuracy. We used the same batch size\nand fixed learning rate for all cases, and used the Adam optimizer(Kingma & Ba, 2017) for all the non-\nsimmeringtrainingscenarios. Theearlystoppedensembleconsistsof10independentlytrainedmodels,with\nanaveragetrainingtimeof53.1epochs,aggregatedwithmajorityvotingasforCIFAR-10. Thesingleearly\nstopped model is presented as a characteristic example of the ensemble members. Based on the average\nearly stopped training time, we trained a model with dropout for 60 epochs and determined its “ensemble\ntestaccuracy”inthesamewayasforCIFAR-10. Wefoundthatthesimmeringensembleexceededthetest\naccuracyofallothercasesinonly21trainingepochs,withamajority-votedensemblepredictionderivedfrom\nitslast200sampledmodels. Thus,forequivalenttrainingtime,abinitiosimmeringproducesmoreaccurate\npredictionsthanbothsingle-modelandensembledoverfittingmitigationtechniquesfortheCIFAR-10dataset.\nHowever, simmering can also both accelerate and exceed the accuracy of other overfitting techniques in a\nnaturallanguageprocessingtasklikelanguagetranslation.\nEmployingaminimallybiasedmodeloftrainingdataerrorallowsabinitiosimmeringtooutperformother\ncommonly used overfitting avoidance methods in two different learning tasks: image classification with a\nconvolutional neural network and language translation with a Transformer. Table 1 shows that, for both\nlearningtasks,initiosimmeringachievesahighertestdataaccuracythanstandardimplementationsofearly\nstoppinganddropout. Consistentwithensemblelearningapproaches,abinitiosimmeringoutperformsasin-\ngularoptimizedmodeltreatedwithearlystopping. ForCIFAR-10,givenanequaltrainingtimeof20epochs,\nab initio simmering also outperforms two other ensemble learning methods: ensembled early stopping, an\nensembleofindependentlyoptimizedneuralnetworks,anddropout,anensembleofsimultaneously-trained\n19\nthinnedneuralnetworks.Wefindthatsimmeringyieldsatestaccuracyofover82%in20epochs,whereasno\nothermethodexceeded76%testaccuracyonthesamebasis. Forthetranslationproblem,simmeringoutper-\nformsallthreeoverfittingmitigationmethodswithashortertrainingtime. Usingfixed, equivalentlearning\nrates,simmeringexceededtheaccuracyofbothdropoutandensembledearlystoppinginasmallfractionof\nthetrainingtime(21epochsforsimmering,versus60epochsfordropoutandanaverageof53.1epochsfor\nensembled early stopping). These comparisons highlight the role of simmering’s distinct minimally-biased\nensemblegenerationinefficientlyproducinggeneralizablepredictions.\nBeyond generalizable prediction, simmering also yields prediction uncertainty estimates that reflect data\ndifficulty. Figure 3showshowthethreeensemblelearningapproachespredictontwodifferentCIFAR-10\ntestsamples(Fig.3a–b). Thoughensembleuncertaintyestimatescanbeextractedeitherdirectlyorindirectly\n(see Supplementary Methods) from all three ensemble learning approaches, the ensemble generated by ab\ninitio simmering allows for uncertainty estimates that are proportional to image difficulty as shown in Fig.\n3c–d.Thus,themaximumentropydistributionsamplingthatsupportssimmeringprovidesbothgeneralizable\npredictionsandadditionalinsightintotheneuralnetwork’slearningprocess.\nconfidenceestimatesthatcorrespondtoimagedifficulty. Earlystopping(greenbars),dropout(bluebars)\nand ab initio simmering (pink bars) ensembles predict an “easy” image of an automobile (panel b), with a\nclear foreground and with the automobile in full view, with similarly high confidence (high proportion of\nensemblevotes). However,whentaskedwithclassifyinga“difficult”testimageofatruck(panela),which\npresents the truck at close range and with low contrast with the background, early stopping remains nearly\nequally confident (panel c, green bar). In contrast, simmering predicts the correct class, but with lower\nconfidence, reflecting the difficulty of identifying the truck in the image (see Supplementary Methods for\nimagedifficultyquantification). Simmering’ssecond-mostvotedpredictionbearsrelationtothecorrectclass\n(“automobile”and“truck”havesimilarimagefeatures). Thoughdropoutalsopredictsuncertainly,itpredicts\nthe correct label with near-equivalent confidence to most other classes. Thus, ab initio simmering predicts\nimagesofvaryingdifficultyaccuratelywithconfidenceproportionaltoimagedifficulty.\nForbothtrainingtasks, whichemploydifferentcomplexarchitecturesandtrainondistinctdatatypes, ab\ninitiosimmeringcanachieveahighertestdataaccuracyineitherequivalentorlesstrainingtime.\nWe note that the for the purposes of a constructing a direct, head-to-head comparison of Transformer\ntraining,wehaveusedfixed,identicallearningratesaswellasidenticalbatchsizes,andtrain/testsplitsfor\nalllearningmethods.\nFurthermore, the manuscript would benefit from a more thorough comparison of the proposed\nmethod against existing regularization techniques commonly used in deep learning, particularly\n21\ninmitigatingoverfitting. Examplesincludeweightdecay,dropout,andearlystopping. Suchcom-\nparisonsareessentialtopositiontheproposedapproachwithintheexistingbodyofworkandto\ndemonstrateitsadvantagesorlimitations.\nWethanktheRefereeforthissuggestion.TherevisedResultsnowincludescomparisonstodropoutandearly\nstoppingfortrainingonCIFAR-10andalanguagetranslationtask:\nMethod CIFAR-10TestAccuracy(%) PT-ENTranslationTestAccuracy(%)\nEarlystopping(single) 71.90 39.62(67epochs)\nEarlystopping(ensemble) 73.81 41.42(53.1epochs–average)\nDropout 75.53 39.86(60epochs)\nAbinitiosimmering 82.05 43.46(21epochs)\nTable1: Simmeringoutperformsearlystopping,ensembledearlystoppinganddropoutontheCIFAR-\n10(Krizhevsky&Hinton,2009)andPortuguese-English(PT-EN)TEDtalktranscripttranslation(Ye\netal.,2018)datasets. FortrainingontheCIFAR-10dataset,allfourlearningcasesemployedtheConvNet\narchitecture(Krizhevsky,Sutskever,etal.,2012),thestandardtrain/testsplit.Forthedropoutcase,weapplied\nthedropoutratesreportedinRef.(Srivastavaetal.,2014). WeusedTensorFlow’simplementationsofearly\nstopping and dropout were, paired with stochastic gradient descent(Martín Abadi et al., 2015) to optimize\ntheearlystoppedanddropout-equippedmodels. Theearlystoppingensembleconsistsof100independently\ntrainedearlystoppedmodels,wheretheaveragetrainingtimewas14.56epochs. Wechosethesingleearly\nstopped model by its proximity to the average accuracy of the ensemble members (71.60%). Dropout and\nabinitiosimmeringeachtrainedfor20epochs,andweincludedthemodelscorrespondingtothelast2000\nweightupdatesinthesimmeringensemble.Atthetestingstage,sinceithasnoexplicitensembletoaggregate,\nwe took dropout’s “inference mode” prediction to be its ensemble prediction. (Hara et al., 2016) We used\nmajorityvotingtoaggregatetheearlystoppingandsimmeringensembles. Forlearningthetranslationtask,\nwetrainedareducedversion(describedinSupplementaryMethods)oftheTransformerarchitecturepresented\nin Ref. (Vaswani et al., 2017) with a pre-trained BERT tokenizer from TensorFlow’s model database, and\nassessed translation accuracy via teacher-forced token prediction accuracy. We used the same batch size\nand fixed learning rate for all cases, and used the Adam optimizer(Kingma & Ba, 2017) for all the non-\nsimmeringtrainingscenarios. Theearlystoppedensembleconsistsof10independentlytrainedmodels,with\nanaveragetrainingtimeof53.1epochs,aggregatedwithmajorityvotingasforCIFAR-10. Thesingleearly\nstopped model is presented as a characteristic example of the ensemble members. Based on the average\nearly stopped training time, we trained a model with dropout for 60 epochs and determined its “ensemble\ntestaccuracy”inthesamewayasforCIFAR-10. Wefoundthatthesimmeringensembleexceededthetest\naccuracyofallothercasesinonly21trainingepochs,withamajority-votedensemblepredictionderivedfrom\nitslast200sampledmodels. Thus,forequivalenttrainingtime,abinitiosimmeringproducesmoreaccurate\npredictionsthanbothsingle-modelandensembledoverfittingmitigationtechniquesfortheCIFAR-10dataset.\nHowever, simmering can also both accelerate and exceed the accuracy of other overfitting techniques in a\nnaturallanguageprocessingtasklikelanguagetranslation.\nEmployingaminimallybiasedmodeloftrainingdataerrorallowsabinitiosimmeringtooutperformother\ncommonly used overfitting avoidance methods in two different learning tasks: image classification with a\nconvolutional neural network and language translation with a Transformer. Table 1 shows that, for both\nlearningtasks,initiosimmeringachievesahighertestdataaccuracythanstandardimplementationsofearly\nstoppinganddropout. Consistentwithensemblelearningapproaches,abinitiosimmeringoutperformsasin-\ngularoptimizedmodeltreatedwithearlystopping. ForCIFAR-10,givenanequaltrainingtimeof20epochs,\nab initio simmering also outperforms two other ensemble learning methods: ensembled early stopping, an\nensembleofindependentlyoptimizedneuralnetworks,anddropout,anensembleofsimultaneously-trained\n22\nthinnedneuralnetworks.Wefindthatsimmeringyieldsatestaccuracyofover82%in20epochs,whereasno\nothermethodexceeded76%testaccuracyonthesamebasis. Forthetranslationproblem,simmeringoutper-\nformsallthreeoverfittingmitigationmethodswithashortertrainingtime. Usingfixed, equivalentlearning\nrates,simmeringexceededtheaccuracyofbothdropoutandensembledearlystoppinginasmallfractionof\nthetrainingtime(21epochsforsimmering,versus60epochsfordropoutandanaverageof53.1epochsfor\nensembled early stopping). These comparisons highlight the role of simmering’s distinct minimally-biased\nensemblegenerationinefficientlyproducinggeneralizablepredictions.\nWehavealsoextendedthisanalysistoincludeacomparisonofensemblepredictiondistributionsgenerated\nbyensembledearlystopping,dropoutandabinitiosimmering:\nBeyond generalizable prediction, simmering also yields prediction uncertainty estimates that reflect data\ndifficulty. Figure 3showshowthethreeensemblelearningapproachespredictontwodifferentCIFAR-10\ntestsamples(Fig.3a–b). Thoughensembleuncertaintyestimatescanbeextractedeitherdirectlyorindirectly\n(see Supplementary Methods) from all three ensemble learning approaches, the ensemble generated by ab\ninitio simmering allows for uncertainty estimates that are proportional to image difficulty as shown in Fig.\n3c–d.Thus,themaximumentropydistributionsamplingthatsupportssimmeringprovidesbothgeneralizable\npredictionsandadditionalinsightintotheneuralnetwork’slearningprocess.\nconfidenceestimatesthatcorrespondtoimagedifficulty. Earlystopping(greenbars),dropout(bluebars)\nand ab initio simmering (pink bars) ensembles predict an “easy” image of an automobile (panel b), with a\nclear foreground and with the automobile in full view, with similarly high confidence (high proportion of\nensemblevotes). However,whentaskedwithclassifyinga“difficult”testimageofatruck(panela),which\npresents the truck at close range and with low contrast with the background, early stopping remains nearly\nequally confident (panel c, green bar). In contrast, simmering predicts the correct class, but with lower\nconfidence, reflecting the difficulty of identifying the truck in the image (see Supplementary Methods for\nimagedifficultyquantification). Simmering’ssecond-mostvotedpredictionbearsrelationtothecorrectclass\n(“automobile”and“truck”havesimilarimagefeatures). Thoughdropoutalsopredictsuncertainly,itpredicts\nthe correct label with near-equivalent confidence to most other classes. Thus, ab initio simmering predicts\nimagesofvaryingdifficultyaccuratelywithconfidenceproportionaltoimagedifficulty.\nWebelievethatthecomparisonpresentedinTable1andFig. 3demonstratestheaccuracyandprediction\nuncertainty quantification advantages of simmering over both singular-model and ensembled predictions in\ntwodifferenttrainingsettings.\nFinally,theclarityofthemethodologicaldescriptionshouldbeimproved. Asitstands,theexpla-\nnationoftheproposedapproachisnotsufficientlydetailedtoenablereproducibility. Irecommend\nthe authors provide a more comprehensive and precise account of the algorithm, including its\nimplementationdetailsandoverallpipeline.\nods,whichnowreads:\nTheparticles’equationsofmotionarederivedfromtheNHCHamiltonian.(Martynaetal.,1996;Tucker-\nmanetal.,1992)Forsimplicityofnotation,wewillhenceforthsetm = 1,Q = 1∀i,k anddescribethe\ni k\nequationsofmotionintermsofthepositionsandvelocities,ratherthanthepositionsandmomenta. Thereal\nparticles’positionsevolveaccordingto(Jang&Voth,1997)\nx˙ (t)=v (t) (24)\ni i\nv˙ (t)=a (t)−v (t)v (t), (25)\ni i s1 i\nwheretheisubscriptdenotesrealparticlequantities, andthes subscriptdenotesaquantityaffiliatedwith\nk\nthe kth virtual particle. The v (t)v (t) term from the NHC in Equation (25) distinguishes simmering\ns1 i\nfromgradient-basedoptimization. Forexample, inmomentumgradientdescent, Equation(25)wouldhave\nv (t) = const. (França et al., 2020) In contrast, v (t) in NHC dynamics can take on both positive and\ns1 s1\nnegativevaluesovertime.\nThevirtualparticledynamicsaregivenby\ns˙ (t)=v (t) (26)\nk sk\nv˙ (t)=a (t)−v (t)v (t) (27)\ns1 s1 s2 s1\nv˙ (t)=a (t)−v (t)v (t), (28)\nsk sk sk+1 sk\nwiththeaccelerationsforallparticlesa (t),a (t),a (t)definedas\ni s1 sk\n1\na (t)=− ∇ L(x (t),...,x (t)) (29)\ni m xi 1 N\ni\na (t)= 1 (cid:0)(cid:88) m v2(t)−NT (cid:1) (30)\ns1 Q i i target\n1\ni\na (t)= 1 (cid:0) Q v2 (t)−T (cid:1) . (31)\nsk Q k−1 k−1 target\nk\nTheaccelerationoftherealparticlesinEquation(29)isproportionaltothenegativegradientofthelosswith\nrespecttotheweightsandbiasesintheneuralnetwork.\nWepresentthediscretizationschemeweusedtonumericallyintegrateEquations(26)–(28)inSupplemen-\ntaryMethods. Wehaverevisedthisdescription,whichreads:\nTo numerically integrate the particle positions over time, we discretize Main Equations 24–28 using a\nVerletintegratorforpositionandvelocityforboththerealandvirtualparticles.Verletintegrationisusedhere\nbecause it preserves Hamilton’s equations.(Jang & Voth, 1997; Yamamoto, 2006) The resulting discretized\n25\nequationsofmotionare\n∆t\nx (t+∆t/2)=x (t)+ v (t) (32)\ni i 2 i\n∆t\ns (t+∆t/2)=s (t)+ v (t) (33)\n2k 2k 2 s2k\n∆t\nv s2k−1 (t+∆t/2)=v s2k−1 (t)e\n−\n2\n∆tvs2k\n+ 2 a s2k−1 (t)e\n−\n4\n∆tvs2k,\n(34)\nv i (t+∆t)=v i (t)e−∆tvs1 (t+∆t/2)+∆ta i (t+∆t/2)e−1 2 ∆tvs1 (t+∆t/2) (35)\ns (t+∆t)=s (t)+∆tv (t+∆t/2) (36)\n2k−1 2k−1 s2k−1\nv s2k (t+∆t)=v s2k (t)e−∆tvs2k+1 (t+∆t/2)+∆ta s2k (t+∆t/2)e− 2 1∆tvs2k+1 (t+∆t/2), (37)\n∆t\nx (t+∆t)=x (t+∆t/2)+ v (t+∆t) (38)\ni i 2 i\n∆t\ns (t+∆t)=s (t+∆t/2)+ v (t+∆t) (39)\n2k 2k 2 s2k\n∆t\nv s2k−1 (t+∆t)=v s2k−1 (t+∆t/2)e− 2 1∆tvs2k (t+∆t)+ 2 a s2k−1 (t+∆t)e− 4 1∆tvs2k (t+∆t), (40)\nwhere a ,a ,a are defined in Main Equations 29–31. Given this numerical integration scheme, an\ni 1 sk,k̸=1\nensemble of neural networks can be generated by selecting an appropriate learning rate ∆t and iteratively\ncomputingx(t+∆t).\nWealsonotethattheexperimentalconditionscorrespondingtoalltestcasesandcomparisonspresentedin\nthemanuscriptaredocumentedforreproductionintheSupplementaryMethods.\nWehavealsoupdatedtheopen-sourceimplementationprovidedatRef.(Babayanetal., 2025)topresent\na demonstration of training on the CIFAR-10 dataset with ab inito simmering. We made this modification\nbecauseCIFAR-10isastandardbenchmarkdatasetfortestingmachinelearningmodelperformance,andits\nfamiliaritytothemachinelearningcommunityfacilitatesunderstandingofourtrainingapproach. Wehope\nthat the extended implementation details in Methods and Supplementary Methods, as well as the updated\nopen-sourceimplementation,clarifytheimplementationofsimmeringusedinthismanuscript.\nIn summary, while the manuscript introduces a potentially promising approach, significant revi-\nsions are needed to substantiate the claims and enhance the clarity and reproducibility of the\nwork. With these improvements, the study could make a more meaningful contribution to the\nfield.\nWe thank the Referee for the suggestions to better contextualize and describe our work. We believe that\nthe addition of larger convolutional neural network and Transformer training problems, the comparison to\ndropout,earlystopping,andotherensemblemethods,andtheimprovedarticulationofourmethods,substan-\ntiallyimprovethemanuscript.\n26"}, {"round": 2, "reviewer_comments": {"reviewer_1": "Most of my questions are reasonably well addressed by the authors' response. The newly added experiments on CIFAR-10 with ConvNets and NLP with Transformers make the work more solid. Yet some more details are needed for those experiments: 1. for all settings, both training and test performances should be shown, preferably in curves, as functions of training epochs. 2. did you use any data augmentation with CIFAR-10?\nRegarding the new Figure 3 of model confidence. It would be more convincing if some statistics are shown with a larger sample size, instead of a single example. One example is good for illustration purpose but it is not sufficient to draw a conclusion on such a property.", "reviewer_2": "The paper proposes a “sufficient training” method called Simmering to mitigate overfitting in deep learning. It uses a Nosé–\nHoover Chain (NHC) thermostat to sample parameter configurations and aggregates predictions via ensemble. While\napplying NHC to deep learning is novel, the core gains against overfitting largely stem from ensembling, which is already\nwell studied.\nSimmering requires forward and backward passes over many parameter configurations, incurring substantial computational\nand time costs that scale poorly for large models. However, the paper provides no quantitative analysis or comparison of\ncomputational complexity, training time, or resource usage, and experiments are mostly limited to small-scale settings,\nmaking it difficult to assess real-world performance and scalability.\nThe authors claim that “ab initio” Simmering from random initialization outperforms baselines, but do not provide\ncorresponding training/validation loss curves, leaving the evidence insufficient. The assertion in Supplementary Fig. 1 that\nAdam and Simmering follow distinct training trajectories is also questionable: the Adam trajectory reflects training from\nscratch and is expected to move normal to loss contours; the Simmering trajectory is a 100-epoch slice taken from the\nAdam run after near-convergence, where oscillations around a local minimum are typical. Thus, differences between Fig. 1a and\nFig. 1b are likely due to different training phases rather than algorithmic mechanics. Moreover, the images in Fig. 3 are very\nlow resolution, making human recognition difficult and weakening the persuasive power of the comparison.\nThe writing also needs improvement: the paper contains many mathematical symbols and formulas, but the exposition is not\nwell organized and the notation is unclear, failing to present an intuitive algorithmic pipeline and hindering reproducibility. I\nrecommend adding a complete training pipeline description (including hyperparameters, temperature/learning-rate\nschedules, sampling and ensembling strategy), providing complexity and resource-cost comparisons, releasing\nconvergence curves for from-scratch training, and supplying higher-quality visualizations to substantiate the claims."}, "author_response": "ManuscriptID: NCOMMS-25-24798-A\nTitle: Simmering: sufficientisbetterthanoptimalfortrainingneuralnetworks\nAuthors: IrinaBabayan,HazhirAliahmadi,GregvanAnders\nResponsetoRefereeReports\nThank you for your communication and efforts in editing and collating these reviews. We also thank both\nReferees for their constructive feedback on the manuscript. The Referees made a number of suggestions\nthat improve the clarity and reproducibility of our results. We believe the addition of training performance\ncurves, computational complexity analysis, algorithm pseudocode and additional training procedure details\nsignificantlyimprovethemanuscript’spresentation.\nThereportsarereproducedbelowintheirentirety. Thequotationsareverbatimandindentedinblue,sans-\nserif font. In some places we quote text directly from the revised manuscript using red, serif font; text is\nreproducedverbatimexceptinplaceswhereitinvolvesreferencenumbers,which,sincethisreportdoesnot\nappearwiththecontextofthefullreferencelist,wehavegivenauthor-datepairsinstead.\nInaddition,wecorrectedsomeminorspellingandgrammaticalerrors.\nFollowingtheresponsetotheReferees, weappendahighlightedversionofthemanuscriptthatindicates\nthechangeswehavemadeinresponsetotheReferees’suggestionsandformattingissues.\nWehopethatyouwillfindtherevisedmanuscriptsuitableforpublicationinNatureCommunications.\nReport of Referee 1\nMostofmyquestionsarereasonablywelladdressedbytheauthors’response. Thenewlyadded\nexperimentsonCIFAR-10withConvNetsandNLPwithTransformersmaketheworkmoresolid.\nYetsomemoredetailsareneededforthoseexperiments:\n1. forallsettings,bothtrainingandtestperformancesshouldbeshown,preferablyincurves,\nas\nfunctionsoftrainingepochs.\nWethanktheRefereeforthissuggestion.Weagreethattheadditionoftrainingandtestperformancecurves\nclarifiesthedifferenceintrainingdynamicsbetweenthecomparedmethods. WehaveaddedSupplementary\nFigures1and2,tocomplementTable1inMainResults,whichdepicttrainingperformanceasafunctionof\ntrainingepochs. TheSupplementaryFiguressectionnowreads:\nSupplementaryFigure 1showsthetrainingperformancecurvesthataccompanytheCIFAR-10\ntestaccuracyresultspresentedinMainTable1. Simmering(SupplementaryFig.1a)andearly\nstopping (Supplementary Fig. 1b) converge to similar individual-model training and validation\ndata accuracies, and their ensemble predictions are aggregated in the same manner (majority\nMainTable1,showthatsimmering(panela)producesboththehighestensembleaccuracy(redrectan\ngularmarker,allpanels,correspondingtoCIFAR-10resultsinMainTable1)andthemostsignificant\nensemblingadvantage(roundvs. rectangularmarkersinpanela)comparedtoearlystopping(round\nvs. rectangular markers in panel b), and dropout (round vs. rectangular markers in panel c). In all\npanels, lines denote the individual model accuracy averaged over all weight updates of a given epoch, and\nroundmarkersdenotetheperformanceofindividualmodelsthatareincludedintheultimateensemble.\nvoting). However, the early stopped ensemble aggregate prediction accuracy is approximately\nthe average accuracy of its constituent models (round vs. rectangular markers in Supplemen-\ntaryFig.1b),whereasthesimmeringensemblesignificantlyimprovesonitsconstituentmodels’\naccuracies (round vs. rectangular markers in Supplementary Fig. 1a). The model trained with\ndropout(SupplementaryFig.1c)outperformstheearlystoppingensemblebutdoesnotexceed\ntheensembledtestaccuracyofsimmeringforequivalenttrainingtime. Thoughallthreemodel\nensembleshavesimilarindividual-modelepoch-averagedvalidationandtestaccuracies,theen-\nsemblecollectedviasimmeringyieldsthehighestensembledpredictionaccuracy.\nSupplementary Figure 2 shows the accuracy convergence curves that accompany the PT-EN\ntranslation test accuracy results (rectangular markers in Supplementary Fig. 2) shown in Main\nTable 1. Given an identical, fixed learning rate, simmering converges to a higher individual-\nmodeltrainingandvalidationaccuracythanmodelstrainedwithearlystoppinganddropoutin\nless than half of the number of epochs. As for CIFAR-10, simmering also displays the great-\nestensemblingadvantagerelativetothetestaccuracyofensemblemembers: thetestaccuracy\nof the early stopped ensemble is lower than its most accurate constituent model (green round\nmarkervs. greenrectangularmarkerinSupplementaryFig.2),whereasthesimmeringensemble\naccuracyisseveralpercentagepointshigherthanthemostaccurateconstituentmodel. Dropout\nconvergesthemostslowlyoutofthethreemethodstested(redvs. bluerectangularmarkersin\nSupplementaryFig.2). Overall,givenidenticallearningrates,simmeringconvergestoahigher\nensembledtestaccuracythanbothanensembleofearlystoppedmodelsandamodeltrainedwith\ndropoutinafractionofthenumberoftrainingepochs.\nshowninMainTable1, showthatsimmering(panela)producesboththehighestensembleaccuracy\n(red rectangular marker, all panels, corresponding to Main Table 1) and the most significant ensem-\nblingadvantage(roundvs. rectangularmarkersinpanela)inlessthanhalfofthetrainingtime(grey\nvs.colouredline,andredvs.greenorbluerectangularmarkerinpanelsbandc)comparedtotraining\nwith an equivalent learning rate with early stopping (panel b), and dropout (panel c). In all panels,\nlinesdenoteindividual-modelaccuracyaveragedoverallweightupdatesofagivenepoch,androundmark-\nersdenotetheperformanceofindividualmodelsthatareincludedintheultimateensemble. Thesimmering\nepoch-averagedvalidationcurveandensembledtestaccuracy(greylineandredrectangularmarker,respec-\ntively,inpanelsbandc)areprovidedforconvergencetimescaleandperformancecomparison.\n2. didyouuseanydataaugmentationwithCIFAR-10?\nWe appreciate the opportunity to clarify this aspect of the comparison presented in Table 1. We did not\nusedataaugmentationwithanyofthemethodsshowninthistable. Allofthedetails,e.g.,batchsize,model\narchitecture, hyperparameter choices, pertaining to the setup of each training experiment are described in\nSupplementaryMethods.\nRegardingthenewFigure3ofmodelconfidence. Itwouldbemoreconvincingifsomestatistics\nare shown with a larger sample size, instead of a single example. One example is good for\nillustrationpurpose[sic]butitisnotsufficienttodrawaconclusiononsuchaproperty.\nThe Referee raises an important point about the model confidence example we show in Main Figure 3.\nWeagreethatstatisticsforalargersamplesizewouldsubstantiatethepresenceofthepropertyweillustrate\nin the figure. The challenge with collecting statistics to measure model confidence dependence on image\ndifficulty is that there is currently no consensus on a single measure, computed from a data sample, that\neffectively and universally encapsulates “image difficulty”. For example, there is poor agreement between\n“easy” and “difficult” images based on pixel-intensity standard deviation, Shannon entropy, and intra-class\ncosine similarity. Defining an appropriate measure, and comparing the performance of different training\nmethodsusingthismeasure, isthebeyondthescopeofthisinvestigationandweintendtoreportonthisin\nfuturework.\ninthemaintext. TheMainResultssectionnowreads\nFigure 3: Ab initio simmering correctly classifies CIFAR-10 test images. Panel b shows an image with\nhighpixel-intensitystandarddeviationandlowShannonentropy,whichdepictsanautomobile,withaclear\nforeground and with the automobile in full view. Early stopping (green bars), dropout (blue bars) and ab\ninitio simmering (pink bars) ensembles yield predictions with similarly high confidence (high proportion\nof ensemble votes). In contrast, panel a shows an image with low pixel-intensity standard deviation and\nhigh Shannon entropy, which depicts a truck at close range with low background contrast. Early stopping\nremainsnearlyequallyconfident(panelc,greenbar)whereassimmeringpredictsthecorrectclasswithlower\nconfidence,aligningwiththedifficultyofidentifyingthetruckintheimage. Simmering’ssecond-mostvoted\npredictionbearsrelationtothecorrectclass(“automobile”and“truck”havesimilarimagefeatures). Though\ndropoutalsopredictsuncertainly,itpredictsthecorrectlabelwithnear-equivalentconfidencetomostother\nclasses. Thus,abinitiosimmeringpredictsimagesofvaryingdifficultyaccuratelywithconfidencethatcan\nreflectimagedifficulty.\n...\nBeyondgeneralizableprediction,simmeringalsoyieldspredictionuncertaintyestimatesthatcan\nreflectdatadifficulty. Figure 3showshowthethreeensemblelearningapproachespredicttwo\ndifferent CIFAR-10 test samples (Fig. 3a–b) chosen based on relative image difficulty (pixel-\nintensitystandarddeviationandShannonentropy). Thoughensembleuncertaintyestimatescan\nbeextractedeitherdirectlyorindirectly(seeSupplementaryMethods)fromallthreeensemble\nlearningapproaches,theensemblegeneratedbyabinitio simmeringallowsforuncertaintyesti-\nmates that reflect relative image difficulty as shown in Fig. 3c–d. Thus, the maximum entropy\ndistributionsamplingthatsupportssimmeringprovidesgeneralizablepredictionsandmayoffer\nadditionalinsightintotheneuralnetwork’slearningprocess.\nstimulatedchangesintherevisedmanuscriptthatsubstantiallyimproveitspresentation.\nReport of Referee 2\nThepaperproposesa“sufficienttraining”methodcalledSimmeringtomitigateoverfittingindeep\nlearning. It uses a Nosé–Hoover Chain (NHC) thermostat to sample parameter configurations\nandaggregatespredictionsviaensemble[sic]. WhileapplyingNHCtodeeplearningisnovel,the\ncoregainsagainstoverfittinglargelystemfromensembling,whichisalreadywellstudied.\nWe thank the Referee for noting that our approach provides gains against overfitting. The Referee is also\ncorrectthatourapproachusesensembles,butthatisnottheprimaryoriginoftheadvantageofsimmering.\nThemanuscriptdescribesseveralwaysinwhichsimmeringdiffersfromoptimization-basedtraining. One\nkeydifferenceisinthewayitgeneratesanensemble. Asasufficienttrainingmethod, simmeringdoesnot\nattempttogenerateoptimalweightsandbiases. Insteaditsystematicallygeneratesnon-optimalweightsand\nbiases. Thiscontrastswith,e.g.,ensemblesofearly-stoppedmodelswhere,priortostopping,thealgorithm\nseekstoidentifyoptimalweightsandbiases. Thedistinctionbetweensystematicallygeneratingnon-optimal\nweightsandbiasesandsearchingforoptimalweightsandbiasesandisafundamentaldifferencebetweenthe\napproachwepresenthereandallotherensemble-basedmethodsweareawareof.\nWealsohaveaddedtwofigurestotheSIwhich,inadditiontodepictingtheaccuracyconvergence,showthe\nensemblingadvantagerelativetoindividualmodelaccuracyofeachensemblingmethod. TheSupplementary\nFiguressectionnowreads\nSupplementaryFigure 1showsthetrainingperformancecurvesthataccompanytheCIFAR-10\ntest accuracy presented in Main Table 1. Simmering (Supplementary Fig. 1a) and early stop-\nping(SupplementaryFig.1b)convergetosimilarindividual-modeltrainingandvalidationdata\naccuracies,andtheirensemblepredictionsareaggregatedinthesamemanner(majorityvoting).\nHowever, theearlystoppedensembleaggregatepredictionaccuracyisapproximatelytheaver-\nage accuracy of its constituent models (round vs. rectangular markers in Supplementary Fig.\n1b),whereasthesimmeringensemblesignificantlyimprovesonitsconstituentmodels’\naccuracies(roundvs. rectangularmarkersinSupplementaryFig.1a). Themodeltrainedwithdropout\n(Supplementary Fig. 1c) outperforms the early stopping ensemble but does not exceed the en-\nsembledtestaccuracyofsimmeringforequivalenttrainingtime. Thoughallthreemodelensem-\nbleshavesimilarindividual-modelepoch-averagedvalidationandtestaccuracies,theensemble\ncollectedviasimmeringyieldsthehighestensembledpredictionaccuracy.\nSupplementary Figure 2 shows the accuracy convergence curves that accompany the PT-EN\ntranslationtestaccuracy(rectangularmarkersinSupplementaryFig.2)showninMainTable1.\nGivenanidentical,fixedlearningrate,simmeringconvergestoahigherindividual-modeltrain-\ningandvalidationaccuracythanmodelstrainedwithearlystoppinganddropoutinlessthanhalf\nofthenumberofepochs. AsforCIFAR-10,simmeringalsodisplaysthegreatestensemblingad-\nvantagerelativetothetestaccuracyofensemblemembers:thetestaccuracyoftheearlystopped\nensembleislowerthanitsmostaccurateconstituentmodel(greenroundmarkervs. greenrect-\nangularmarkerinSupplementaryFig.2),whereasthesimmeringensembleaccuracyisseveral\nTable1,showthatsimmering(panela)producesboththehighestensembleaccuracy(redrectangular\nmarker,allpanels,correspondingtoCIFAR-10inMainTable1)andthemostsignificantensembling\nadvantage(roundvs. rectangularmarkersinpanela)comparedtoearlystopping(roundvs. rectan\ngularmarkersinpanelb),anddropout(roundvs. rectangularmarkersinpanelc). Inallpanels,lines\ndenotetheindividualmodelaccuracyaveragedoverallweightupdatesofagivenepoch,androundmarkers\ndenotetheperformanceofindividualmodelsthatareincludedintheultimateensemble.\npercentagepointshigherthanthemostaccurateconstituentmodel. Dropoutconvergesthemost\nslowlyoutofthethreemethodstested(redvs. bluerectangularmarkersinSupplementaryFig.\n2). Overall, givenidenticallearningrates, simmeringconvergestoahigherensembledtestac-\ncuracy than both an ensemble of early stopped models and a model trained with dropout in a\nfractionofthenumberoftrainingepochs.\nin Main Table 1, show that simmering (panel a) produces both the highest ensemble accuracy (red\nrectangular marker, all panels, corresponding to Main Table 1) and the most significant ensembling\nadvantage (round vs. rectangular markers in panel a) in less than half of the training time (grey\nvs.colouredline, andred vs. green orbluerectangularmarker inpanelsbandc) comparedtotraining\nwith an equivalent learning rate with early stopping (panel b), and dropout (panel c). In all panels,\nlinesdenoteindividual-modelaccuracyaveragedoverallweightupdatesofagivenepoch,androundmark-\nersdenotetheperformanceofindividualmodelsthatareincludedintheultimateensemble. Thesimmering\nepoch-averagedvalidationcurveandensembledtestaccuracy(greylineandredrectangularmarker,respec-\ntively,inpanelsbandc)areprovidedforconvergencetimescaleandperformancecomparison.\nachieve a similar ensembled accuracy to the early stopped ensemble and dropout given similar individual\nmodeltestaccuracy. Instead,wefindthatthesimmeringensembleyieldsboththehighestensembletestac-\ncuracyandthelargestindividualmodelvs. ensembleperformanceimprovement,highlightingthedifference\nbetweentheensemblingwithsimmeringandwithoptimization-basedmethods.\nSimmeringrequiresforwardandbackwardpassesovermanyparameterconfigurations,incurring\nsubstantialcomputationalandtimecoststhatscalepoorlyforlargemodels. However,thepaper\nprovides no quantitative analysis or comparison of computational complexity, training time, or\nresourceusage,andexperimentsaremostlylimitedtosmall-scalesettings,makingitdifficultto\nassessreal-worldperformanceandscalability.\nWe thank the Referee for suggesting this clarification. We have now added the following computational\ncomplexityanalysistoSI:\nSimmeringhasaper-iterationcomplexitycomparabletostandardfirst-ordermethods(e.g.,SGD\nwithmomentum),withtimecomplexityO(N). Theparameterupdatesinvolveonlyelementary\noperations(exponentiationandmultiplication)onahandfulofvariablesandasingleN-length\ndotproduct. Thealgorithmstoresonemomentumvalueperneuralnetworkparameter(similar\ntomomentum-basedoptimizers)and2×N variablesforthethermostatchain(e.g.,2×10\nNHC\nvariablesforthetestedproblems). ThisiscomparabletoalgorithmslikeSGDwithmomentum\norAdam(whichstoreoneortwoextratensorsofsizeN). Therefore, memoryusagescalesas\nO(N),whichisofessentiallythesameorderasstandardtraining. Moreover,theextracompu-\ntationsforthermostatvariableintegrationareafixedsmallcost,independentofmodelsize. No\nlargeauxiliarydatastructuresareneededbeyondwhattypicaloptimizersuse.\nIn practice, training time depends not only on per-iteration cost (discussed above) but also on\nthenumberofiterationsrequiredtoreachatargetperformance. Traditionalensemblemethods\nlike ensembled early stopping, bagging, or boosting, require an independent training run per\nensemble member prior to aggregation. In contrast, since simmering samples near-optimal so-\nlutionsratherthanseekingasinglelossminimum,itcangenerateanentireensembleimplicitly\nwithin a single training run. This reduces the overall training time significantly compared to\ntraditionalensemblemethods,particularlywhenconvergenceistime-consuming,e.g.,forlarge\nTransformers.\nIn addition, we note that simmering requires the same number of forward/backward passes as standard\ngradient-basedoptimizationtraining. Forexample,forthedefaultparametersinthesimmeringopen-source\ncode (Babayan et al., 2025), initialization and training to achieve a test accuracy of 78.5% on CIFAR-10\nrequires59secondsofwalltimeonaworkstationwithanNVIDIAGeForceRTX4060GPU,whereasob-\ntainingasimilarlevelofaccuracyusingAdamanddropoutonthesamehardwarerequires89secondsofwall\ntime. WealsonotethatoursimmeringimplementationisinpurePython,andweexpectthatitsperformance\ncouldbefurtherimprovediftheintegrationstepswereimplementedinC++/CUDA.\nHowever, we also note that our aim here is not to identify or produce the most optimal implementation\nof simmering. Instead our aim is to demonstrate that sufficient training is a viable approach for generating\ngeneralizable neural networks and that even a naive implementation of simmering is comparable to other\nleadingmethods.\ndonotprovidecorrespondingtraining/validationlosscurves,leavingtheevidenceinsufficient.\nWethanktheRefereeforthissuggestion. Weagreethattheadditionofthetraining/validationperformance\ncurvesclarifiesthedifferenceintrainingdynamicsbetweenabinitiosimmeringandtheothermethodsshown\ninMainTable1. WehaveaddedtwonewfigurestoSI,SupplementaryFigures1and2,tocomplementMain\nTable1inMainResults. TheSupplementaryFiguressectionnowreads\nFigure 1: The training convergence curves corresponding to the CIFAR-10 training , shown in Main\nTable 1, show that simmering (panel a) produces both the highest ensemble accuracy (red rectangu-\nlar marker, all panels, corresponding to CIFAR-10 results in Main Table 1) and the most significant\nensemblingadvantage(roundvs. rectangularmarkersinpanela)comparedtoearlystopping(round\nvs. rectangular markers in panel b), and dropout (round vs. rectangular markers in panel c). In all\npanels, lines denote the individual model accuracy averaged over all weight updates of a given epoch, and\nroundmarkersdenotetheperformanceofindividualmodelsthatareincludedintheultimateensemble.\nSupplementaryFigure 1showsthetrainingperformancecurvesthataccompanytheCIFAR-10\ntestaccuracyresultspresentedinMainTable1. Simmering(SupplementaryFig.1a)andearly\nstopping (Supplementary Fig. 1b) converge to similar individual-model training and validation\ndata accuracies, and their ensemble predictions are aggregated in the same manner (majority\nvoting). However, the early stopped ensemble aggregate prediction accuracy is approximately\nthe average accuracy of its constituent models (round vs. rectangular markers in Supplemen-\ntaryFig.1b),whereasthesimmeringensemblesignificantlyimprovesonitsconstituentmodels’\naccuracies (round vs. rectangular markers in Supplementary Fig. 1a). The model trained with\ndropout(SupplementaryFig.1c)outperformstheearlystoppingensemblebutdoesnotexceed\ntheensembledtestaccuracyofsimmeringforequivalenttrainingtime. Thoughallthreemodel\nensembleshavesimilarindividual-modelepoch-averagedvalidationandtestaccuracies,theen-\nsemblecollectedviasimmeringyieldsthehighestensembledpredictionaccuracy.\nSupplementary Figure 2 shows the accuracy convergence curves that accompany the PT-EN\ntranslation test accuracy results (rectangular markers in Supplementary Fig. 2) shown in Main\nTable 1. Given an identical, fixed learning rate, simmering converges to a higher individual-\nmodeltrainingandvalidationaccuracythanmodelstrainedwithearlystoppinganddropoutin\nin Main Table 1, show that simmering (panel a) produces both the highest ensemble accuracy (red\nrectangular marker, all panels, corresponding to Main Table 1) and the most significant ensembling\nadvantage (round vs. rectangular markers in panel a) in less than half of the training time (grey\nvs.colouredline, andred vs. green orbluerectangularmarker inpanelsbandc) comparedtotraining\nwith an equivalent learning rate with early stopping (panel b), and dropout (panel c). In all panels,\nlinesdenoteindividual-modelaccuracyaveragedoverallweightupdatesofagivenepoch,androundmark-\nersdenotetheperformanceofindividualmodelsthatareincludedintheultimateensemble. Thesimmering\nepoch-averagedvalidationcurveandensembledtestaccuracy(greylineandredrectangularmarker,respec-\ntively,inpanelsbandc)areprovidedforconvergencetimescaleandperformancecomparison.\nless than half of the number of epochs. As for CIFAR-10, simmering also displays the great-\nestensemblingadvantagerelativetothetestaccuracyofensemblemembers: thetestaccuracy\nof the early stopped ensemble is lower than its most accurate constituent model (green round\nmarkervs. greenrectangularmarkerinSupplementaryFig.2),whereasthesimmeringensemble\naccuracyisseveralpercentagepointshigherthanthemostaccurateconstituentmodel. Dropout\nconvergesthemostslowlyoutofthethreemethodstested(redvs. bluerectangularmarkersin\nSupplementaryFig.2). Overall,givenidenticallearningrates,simmeringconvergestoahigher\nensembledtestaccuracythanbothanensembleofearlystoppedmodelsandamodeltrainedwith\ndropoutinafractionofthenumberoftrainingepochs.\nThe assertion in Supplementary Fig. 1 that Adam and Simmering follow distinct training trajec-\ntories is also questionable: the Adam trajectory reflects training from scratch and is expected\nto move normal to loss contours; the Simmering trajectory is a 100-epoch slice taken from the\nAdamrunafternear-convergence,whereoscillationsaroundalocalminimumaretypical. Thus,\ndifferences between Fig. 1a and Fig. 1b are likely due to different training phases rather than\nalgorithmicmechanics.\nThe Referee is correct that the Adam and simmering trajectories are taken from different training regimes.\nHowever, that is by necessity, since optimization-based training approaches like Adam do not exhibit the\nformofoscillationsthatoccurinsimmering. Hencethereexistsnocomparableregimetoshow.\ntheoscillationsthattheRefereedescribeoccurbecausefinitedifferencingpreventsalgorithmsfromreaching\nexactconvergence. Theoscillationsthatoccurinsimmeringarenotafinitedifferencingartifact. Despitethe\napparent similarity, the oscillations in simmering are the manifestation of the physical analogue of thermal\noscillations.Thermaloscillationsoccurinnaturewherethereisnofinitetimestep(innaturetheseoscillations\ndetermineheatcapacity),andtheywouldoccurinsimmeringeventhetimeintegrationwasexact. Thissame\neffectispresentintensofthousandsofpapersthatemploymoleculardynamicssimulationinconventional\nphysics,chemistry,biological,andmaterialssciencecontexts.\nThethermalnatureofthefluctuationsinsimmeringisalsoreadilyapparentintheretrofittingresultsshown\ninmaintextFig.1. Atthescaleoftheplots,fluctationsarenotvisibleintheinitialsimmeringperiodatlow\ntemperaturebutareclearlydiscernibleathighertemperature. Iftheoscillationswerethesamenumericalar-\ntifactthatoccursinoptimization,theywouldnotexhibitthischaracteristictemperature-dependentbehaviour\nthatispresentinreamsofmoleculardynamicsliterature.\nWeclarifiedthisintheSItextassociatedwithSupplementaryFigure3asfollows:\nFigure 3,generatedusingthelossvisualizationmethodofRef.(Lietal.,2018),showsthedif-\nferenceintrainingdynamicsbetweenAdam,(Kingma&Ba, 2017)agradient-basedoptimizer,\nand simmering, an implementation of sufficient training. The Adam optimizer travels perpen-\ndiculartothelosscontours,wherassimmeringtravelsparalleltothedirectionofnear-equivalent\nloss. Thefirstprincipalcomponent(PC)ofAdamdescribes90.44%ofitstrajectoryvariancein\nparameterspace,andthesecondPCdescribesanother6.12%. Forsimmering,thefirstandsec-\nondPCsdescribe99.9428%and0.0324%oftheparameterspacetrajectoryrespectively. Inthe\nlattercase,alargerproportionofthetrainingtrajectoryisdescribedbythefirstPC.SincethisPC\npointsalongthedirectionofequivalentlossinparameterspace, SupplementaryFig. 3bshows\nthatsufficienttrainingautomaticallyidentifiesandsamplesalongsloppy(equivalent-loss)modes\ninmodelparameterspaceasdescribedinMainMethods. Notethatoptimization-basedtraining\ncanalsoexhibitaformofoscillationthatresultsfromfinitedifferencingartifacts,whereassim-\nmeringisdesignedtoexhibitthermaloscillationsinstead.Thesethermaloscillationsareacritical\nfeaturethatsimmeringleveragestosampleanensembleofmodels.\nMoreover, the images in Fig. 3 are very low resolution, making human recognition difficult and\nweakeningthepersuasivepowerofthecomparison.\nWeappreciatetheopportunitytomotivateourchoiceofillustrativeexamplepresentedinMainFigure3.\nThe resolution of the visualization we show exactly matches the resolution of the images in the CIFAR-\n10dataset,whichconsistsof32x32x3-pixelimages.(Krizhevsky&Hinton,2009)Weoptedtogeneratethis\nillustrativeexamplewiththisdatasetbecauseitwasrecommendedbybothRefereesinthefirstsetofrevisions\nasafamiliarandeffectivebenchmarkforlearningmethodperformance.\nThe writing also needs improvement: the paper contains many mathematical symbols and for-\nmulas, but the exposition is not well organized and the notation is unclear, failing to present an\nintuitivealgorithmicpipelineandhinderingreproducibility.\ntermsandnotationthatmostdirectlyfacilitatecomparisonwithsimilaralgorithmsthatappearinwell-known\nliteratureonthermostatsinothercontexts. Choosingnotationthatfacilitatesthisformofreproducibilityhas\nthesecondaryadvantagethatitprovidesthemoststraightforwardwayofrepresentingalgorithmsofthistype.\n(Which is not coincidental because the notation was developed in other literature to do just that.) We also\nnote that molecular dynamics algorithms were some of the first algorithms that were developed for digital\ncomputersinthe1950s,andourpresentationofthealgorithmfollowsdisciplinaryconventionsthatevolved\ntoensurethereproducibilityofscientificalgorithmsoverthecourseofseveraldecades.\nInaddition,wenotethatitispossibletoresolveanyquestionsthatreadersmayencounterwithunfamiliar\nnotation by referring to/comparing with the source code we provide. For example, as noted by one of the\nRefereesinthefirstroundofreview,itispossibletoreproduceresultsfromthisinvestigationusingaone-line\ncommandwiththecodeweprovide. Ourmostrecentcodereleaseprovidesthisreproducibilitycapabilityfor\nthe CIFAR-10 dataset. We describe the simmering hyperparameters and other training details, e.g., model\narchitecture,datapre-processing,lossfunction,batchsize,andnumberofepochs,foreachtrainingscenario\nwe presented, in the Supplementary Methods. These descriptions, paired with the open-source code we\nprovided,wouldenablereaderstodirectlyreproduceanyofthespecifictrainingproblemswepresentinthis\nwork.\nTofurtherfacilitatereproducibilitywehaveaddedapseudocodedescriptionofthealgorithmthatisanalo-\ngoustorepresentationofAdaminRef. (Kingma&Ba,2017),accompaniedbyatrainingproceduredescrip-\ntion.\nTherevisedSupplementaryMethodssectionnowreads\nAlgorithm 1 describes the implementation of simmering used to generate the results presented\ninthiswork. Therearetwomain“phases”ofsimmering: theequilibrationphase,andthesam-\npling phase. Both phases use the same algorithm (Algorithm 1), but during sampling, model\nparameters and/or predictions are saved for offline ensemble prediction. The main simmering\nhyperparameters are the thermostat temperature T, the particle mass m, and the learning rate\n∆t. ThethermostattemperatureT canbeeitheraconstantoraschedulethatdependsontrain-\ning time t, but note that the system must be equilibrated during sampling to sample from the\ncanonicalensemble. Forourtrainingexperiments,weusedthesamemassforallrealandvirtual\nparticles, i.e., m = Q ∀ i,k, based on Ref. (Martyna et al., 1992). Generally, decreasing the\ni k\nmassmakesthealgorithmmoresensitivetothelossgradientandthuscanaccelerateequilibra-\ntion. The learning rate ∆t can be a schedule, i.e., depend on time t, but must be the same for\nall particles (real and virtual). We also note that we used a NHC length of 10 for all experi-\nments. Tocommenceequilibration,providetheinitialweightsandbiasesoftheneuralnetwork\nasx . Then,duringtraining,beforeexecutingline20ofthefor-loopinAlgorithm1,setthe\nall,0\nneuralnetworkparameterstox ,andusebackpropagationtocomputethegradientofthe\nall,t+1/2\nloss ∇ L. The system is equilibrated once T is fluctuating around T. The precision with\nx real\nwhich the thermostat algorithm will fluctuate about T depends on the neural network parame-\nterspacedimensionality, soexpect that T willfluctuatewitha largeramplitudeforsmaller\nreal\nneuralnetworks.\nOnceequilibrationiscomplete,thesamealgorithmisreusedforsampling. Theonlydifference\nin execution is that the model parameters and/or predictions (depending on the neural network\nanddatasetsizes)generatedatthelossgradientcomputationstage(line18ofAlgorithm1)are\nsavedforofflineensembleanalysis. Thereisnoprescriptionforsamplingfrequency. However,\nsome options for sampling frequency include saving every iteration’s predictions, saving the\npredictionsatafixediterationinterval, orsavingrandomly. Ifsamplingatsomefixedinterval,\n12\ncheck that the samples are uncorrelated at that interval. After the desired number of samples\nhave been collected, the set of models and/or predictions can be aggregated or analyzed as a\ndistributionofsolutionsinthewaysdescribedinthemaintext.\nfunctionLwhichhasparametersx,basedontheNHCthermostatdynamicsintegrationinRef.(Jang&Voth,\n1997). The first subscript of each variable indexes the particles (i for real particles and k for virtual),\nand\nthesecondsubscriptindicatesthetimestepoftheassociatedvariable. NotethatsincetheNHCparticlesare\nintegrated differently based on their chain position (even or odd), the virtual particle index takes on values\nk =1,...,N /2.Unlessotherwiseindicated(e.g.,withaparticleindexofall),operationsareperformed\nNHC\nelement-wise for all elements in the given vector(s). If no initialization is specified, assume that a vector’s\ninitialvalueis0forallelements.\nRequire: T >0: Thermostattemperature\nRequire: ∆t: Learningrate\nRequire: L(x): ObjectivefunctionLwithparametersx\nRequire: x : Initialparametervector/initialrealparticlepositions,ofdimensionN\nall,0\nRequire: m: Realparticlemassvector,ofdimensionN\nRequire: t : Numberofiterations\nmax\nRequire: Q: NHCparticlemassvector,ofdimensionN\nNHC\n1: functionPOSITION_FORWARD(a,b,dt) ▷Integratorforpositions(x i ,s 2k ,s 2k−1 ).\n2: returna+dt·b\n3: functionVELOCITY_FORWARD(a,b,c,dt) ▷Integratorforvelocities(v i ,v s2k ,v s2k−1 ).\n4: returnaexp{(−dt·c)}+dt·b·exp{(−dt/2·c)}\n5\n6\n:\n:\nfunc\nr\nt\ne\ni\nt\no\nu\nn\nrn\nA\n(cid:0)\nC\n(cid:80)\nCE\nj\nL\nm\nER\nj\nA\nv\nT\nj 2\nI\n−\nON\nN\n_S\n·\n(v\n(cid:1)\n,\n/\nm\nQ 1\n,T,N,Q 1 ) ▷AccelerationofthefirstNHCparticle(a s1 ).\n7: functionACCELERATION_SK(v,T,b,c) ▷AccelerationofallotherNHCparticles(a s2k ,a s2k−1̸=1 ).\n8: return(c·v2−T)/b\n9: functionREAL_T(v,m) ▷Forcomputingrealparticlesystemtemperature.\n10: return( (cid:80) j m j v j 2)/N\n(cid:113)\n11: v i,0 ← m T ▷Initializevelocityvectorelements.\n12: T real ←T ▷Definecurrentrealsystemtemperature.\n13: t←0 ▷Initializetimestep.\n14: whilet<t max do\n15: x i,t+1/2 ← POSITION_FORWARD(x i,t ,v i,t ,∆t/2) ▷Firsthalf-stepupdatetox i .\n16: s 2k,t+1/2 ← POSITION_FORWARD(s 2k,t ,v s2k,t ,∆t/2) ▷Firsthalf-stepupdatetos 2k .\n17: a s1,t ← ACCELERATION_S1(v all,t ,m all ,T,N,Q 1 )\n18: a s2k−1̸=1,t ← ACCELERATION_SK(v s2(k−1),t ,T,Q 2k−1 ,Q 2(k−1) )\n19: v s2k−1,t+1/2 ← VELOCITY_FORWARD(v s2k−1,t ,a s2k−1,t ,v s2k,t ,∆t/2) ▷Firsthalf-stepupdateto\nv .\ns2k−1\n20: a i,t+1/2 ←−∇ xi L(x all,t+1/2 )/m i ▷Uselossgradienthere!\n21: v i,t+1 ← VELOCITY_FORWARD(v i,t ,a i,t+1/2 ,v s1,t+1/2 ,∆t) ▷Full-stepupdatetov i .\n22: s 2k−1,t+1 ← POSITION_FORWARD(s 2k−1,t ,v s2k−1,t+1/2 ,∆t) ▷Full-stepupdatetos 2k−1 .\n23: a s2k,t+1/2 ← ACCELERATION_SK(v s2k−1,t+1/2 ,T,Q 2k ,Q 2k−1 )\n24: v s2k̸=Nc ,t+1 ← VELOCITY_FORWARD(v s2k,t ,a s2k,t+1/2 ,v s2k+1,t+1/2 ,∆t) ▷Full-stepupdateto\nv .\ns2k\n25: v sNc ,t+1 ← VELOCITY_FORWARD(v sNc ,t ,a sNc ,t+1/2 ,0,∆t) ▷Full-stepupdatetov sNc .\n26: x i,t+1 ← POSITION_FORWARD(x i,t+1/2 ,v i,t+1 ,∆t/2) ▷Lasthalf-stepupdatetox i .\n27: s 2k,t+1 ← POSITION_FORWARD(s 2k,t+1/2 ,v s2k,t+1 ,∆t/2) ▷Lasthalf-stepupdatetos 2k .\n28: a s1,t+1 ← ACCELERATION_S1(v all,t+1 ,m all ,T,N,Q 1 )\n29: a s2k−1̸=1,t+1 ← ACCELERATION_SK(v s2(k−1),t+1 ,T,Q 2k−1̸=1 ,Q 2(k−1) )\n30: v s2k−1,t+1 ← VELOCITY_FORWARD(v s2k−1,t+1/2 ,a s2k−1,t+1 ,v s2k,t+1 ,∆t/2) ▷Lasthalf-step\nupdatetov .\ns2k−1\n31: T real ← REAL_T(v all,t+1 ,m all ) ▷Optional: TrackrealsystemTthroughoutsampling.\n32: t←t+1\nrateschedules,samplingandensemblingstrategy),providingcomplexityandresource-costcom-\nparisons, releasing convergence curves for from-scratch training, and supplying higher-quality\nvisualizationstosubstantiatetheclaims.\nWethanktheRefereeforthesuggestionstobetterpresentourwork. Webelievethattheadditionofconver-\ngencecurves,complexityanalysisandadditionalimplementationdetailspromotereproducibilityandbetter\ncontextualizeourmethod’sperformance.\n15"}, {"round": 3, "reviewer_comments": {"reviewer_1": "The results seem overall interesting. A few things to improve:\n1. I feel that Supplementary Figure 1 and 2 are much more informative than Table 1. Perhaps replace table 1 with those\nfigures will make the main paper more convincing.\n2. From Supplementary Figure 1, it seems that Simmering produces more \"diverse\" models that works better in an ensemble\nthan traditional early stopping. Although this may be beyond the scope of current paper, it may be interesting to explore more\nalong this direction.\n3. In Supplementary Figure 2, Simmering converges much faster than the baseline method on Transformers, where it only\nuses 20 epochs where the baseline takes 60 epochs to reach the same level of performance. If this result is true, it could be\nmore interesting the rest of the paper combined. Why is this convergence speedup not explored more in the paper? I find it\nhard to believe that it converges so much faster.\n4. Main text Figure 3 about classification confidence, on the other hand, is much less interesting since it contains merely one\nillustrative example. It is hard to conclude anything from one example. If a more statistical measure is beyond the scope the\npaper, I suggest reducing the emphasis on this, by perhaps moving this one to the supplementary materials and add the\nSupplementary Figure 1 and 2 to the main text, together with related analyses.\n(Remarks on code availability)\nI looked at the original code in the initial review and it ran without problem. I have NOT yet checked the latest code about\nCIFAR10 and transformers (if any).", "reviewer_2": "My main concerns have been addressed, but I have one further suggestion. Although this is an interdisciplinary work, the\nproposed method is ultimately aimed at training neural network models. Therefore, the notations and pseudocode in the\npaper should follow the conventions of neural network training algorithms, rather than those of thermodynamics. This will\nenable researchers in the artificial intelligence field to better understand and utilize the proposed method.\n(Remarks on code availability)\nNA"}, "author_response": "ManuscriptID: NCOMMS-25-24798-C\nTitle: Simmering: sufficientisbetterthanoptimalfortrainingneuralnetworks\nAuthors: IrinaBabayan,HazhirAliahmadi,GregvanAnders\nResponsetoRefereeReports\nWethankbothRefereesfortheirfeedbackonthemanuscript. TheRefereesmadeanumberofsuggestions\nregardingresultsandmethodspresentationthatimprovetheimpactandinterpretabilityofourresults.\nThereportsarereproducedbelowintheirentirety. Thequotationsareverbatimandindentedinblue,sans-\nserif font. In some places we quote text directly from the revised manuscript using red, serif font; text is\nreproducedverbatimexceptinplaceswhereitinvolvesreferencenumbers,forwhich,sincethisreportdoes\nnotappearwiththecontextofthefullreferencelist,wehavegivenauthor-datepairsinstead.\nReport of Referee 1\nTheresultsseemoverallinteresting. Afewthingstoimprove:\n1. I feel that Supplementary Figure 1 and 2 are much more informative than Table 1. Perhaps\nreplacetable1withthosefigureswillmakethemainpapermoreconvincing.\nWe thank the Referee for this suggestion, and agree that the contents of Supplementary Figures 1 and 2\nbetterdisplaytheuniqueadvantagesofsimmering.\nWe have replaced Table 1 with a new figure, Main Figure 3, that compares the ensemble and individual\nmodel performances on test data for each of the three overfitting mitigation methods, and for both training\ntaskstested. AsindicatedbythisReferee’snextcomment, thesearethefeaturesoftheconvergencecurves\nthatmostclearlyhighlightthedifferencebetweensimmeringandothermethods.WehaveleftSupplementary\nFigures1and2astheyareinSupplementaryInformation,sothefullaccuracyconvergencecurvesarestill\navailableinthemanuscript.\nTheMainResultssectionnowreads,\nEmployingaminimallybiasedmodeloftrainingdataerrorallowsabinitiosimmeringtooutper-\nformothercommonlyusedoverfittingavoidancemethodsintwodifferentlearningtasks: image\nclassificationwithaconvolutionalneuralnetworkandlanguagetranslationwithaTransformer.\nFig. 3 shows that, for both learning tasks, ab initio simmering achieves a higher test data ac-\ncuracythanstandardimplementationsofearlystoppinganddropout. Consistentwithensemble\nlearning approaches, ab initio simmering outperforms singular optimized models treated with\nearlystopping(rectangularmaroonmarkervs. roundgreenmarkersinFig.3). ForCIFAR-10,\ngivenanequaltrainingtimeof20epochs,abinitiosimmeringalsooutperformstwootherensem-\nblelearningmethods:ensembledearlystopping,anensembleofindependentlyoptimizedneural\nnetworks,anddropout,anensembleofsimultaneously-trainedthinnedneuralnetworks.Wefind\n1\nthat simmering yields a test accuracy of over 82% in 20 epochs, whereas no other method ex-\nceeded76%testaccuracyonthesamebasis. Inadditiontoyieldingthemostaccurateensemble\npredictions, we find that the simmering ensemble yields the most pronounced improvement in\naccuracyafterensembleaggregation,despiteitsensemblemembershavingsimilaraccuraciesto\nthoseoftheearlystoppedensemble. Thisdifferenceinensemblingeffectivenessindicatesthat\nthe simmering ensemble, consisting of sufficient rather than optimal representations, uniquely\nensuresgeneralizableaggregatepredictions. InSupplementaryInformation,wealsopresentex-\namplesofpredictionuncertaintyanalysisonCIFAR-10images,analogoustoFigure??c-d,that\ndisplaythedifferenceinensemblepredictiondistributionbetweenthethreeoverfittingmitigation\nmethodsinFig.3a.\nForthetranslationproblem,simmeringoutperformsallthreeoverfittingmitigationmethodswith\nashortertrainingtime. Usingfixed,equivalentlearningrates,simmeringexceededtheaccuracy\nofbothdropoutandensembledearlystoppinginasmallfractionofthetrainingtime(21epochs\nforsimmering,versus60epochsfordropoutandanaverageof53.1epochsforensembledearly\nstopping). AsforCIFAR-10,simmeringhadboththemostaccurateensemblepredictionandthe\nmostsignificantensemblingimprovement. Thesecomparisonshighlighttheroleofsimmering’s\ndistinctminimally-biasedensemblegenerationinefficientlyproducinggeneralizablepredictions.\n10(Krizhevsky&Hinton,2009)(panela)andPortuguese-English(PT-EN)TEDtalktranscripttrans-\nlation(Yeetal.,2018)(panelb)datasets. Forbothproblems,simmering’sensembleprediction(rectangular\nmarker) achieves both the highest accuracy and the most significant ensembling improvement (rectangular\nmarker vs. round markers), with the latter indicating that the advantage of simmering extends beyond just\ntheactofensembling. Incontrast,theearlystoppedensembleaccuracy(rectangularmarker)doesnotexceed\nthatofitsensemblemembers(roundmarkers)forbothtrainingtasks. FortrainingontheCIFAR-10dataset,\nallfourlearningcasesemployedtheConvNetarchitecture(Krizhevsky,Sutskever,etal.,2012),andthestan-\ndard train/test split. For the dropout case, we applied the dropout rates reported in Ref. (Srivastava et al.,\n2014).WeusedTensorFlow’simplementationsofearlystoppinganddropout,pairedwithstochasticgradient\ndescent(Martín Abadi et al., 2015) to optimize the early stopped and dropout-equipped models. The early\nstopping ensemble consists of 100 independently trained early stopped models, where the average training\ntime was 14.56 epochs. Dropout and ab initio simmering each trained for 20 epochs, and we included the\nmodelscorrespondingtothelast2000weightupdatesinthesimmeringensemble. Atthetestingstage,since\nithasnoexplicitensembletoaggregate, wetookdropout’s“inferencemode”predictiontobeitsensemble\nprediction.(Haraetal.,2016)Weusedmajorityvotingtoaggregatetheearlystoppingandsimmeringensembles.\nForlearningthetranslationtask,wetrainedareducedversion(describedinSupplementaryMethods)of\ntheTransformerarchitecturepresentedinRef.(Vaswanietal.,2017)withapre-trainedBERTtokenizerfrom\nTensorFlow’s model database, and assessed translation accuracy via teacher-forced token prediction accu-\nracy. Weusedthesamebatchsizeandfixedlearningrateforallcases,andusedtheAdamoptimizer(Kingma\n& Ba, 2017) for all the non-simmering training scenarios. The early stopping ensemble consists of 10 in-\ndependentlytrainedmodels, withanaveragetrainingtimeof53.1epochs, aggregatedwithmajorityvoting\nas for CIFAR-10. Based on the average early stopped training time, we trained a model with dropout for\n60 epochs and determined its “ensemble test accuracy” in the same way as for CIFAR-10. We found that\nthe simmering ensemble exceeded the test accuracy of all other cases in only 21 training epochs, with a\nmajority-votedensemblepredictionderivedfrom200modelssampledduringthelastepoch. Fullaccuracy\nconvergence curves for both training tasks are shown in Supplementary Figures 1–2, and additional details\non comparison implementation are described in Supplementary Methods. For equivalent training time, ab\ninitiosimmeringproducesmoreaccuratepredictionsthanotherensembledoverfittingmitigationtechniques\nfortheCIFAR-10dataset. However,simmeringcanalsobothacceleratetrainingandexceedtheaccuracyof\notheroverfittingtechniquesinanaturallanguageprocessingtasklikelanguagetranslation.\n3\n2. FromSupplementaryFigure1,itseemsthatSimmeringproducesmore\"diverse\"modelsthat\nworks better in an ensemble than traditional early stopping. Although this may be beyond the\nscopeofcurrentpaper,itmaybeinterestingtoexploremorealongthisdirection.\nTheRefereehasdrawnattentiontoaveryimportantaspectofourwork,andweappreciatetheopportunity\ntoelaborate. Thefactthatsimmeringproducesamore“diverse”ensemblethanachievablebyindependently\noptimized,early-stoppedmodelsisnotacoincidence–simmeringleveragesinformationtheorytogenerate\nthemostrepresentativeensemblepossibleatagiventemperature.\nThe aim of this work is to present a generically generalizable neural network training method. Other\nmethodsforimprovinggeneralizabilityarecase-dependentbecausetheyimposeadditionalrestrictionsonthe\nparameter distribution, e.g., L -norm regularization encourages smaller-magnitude parameters, which may\n2\nnot be a generically suitable restriction on parameter magnitude for all contexts. In contrast, the principle\nof maximum entropy (Jaynes, 1957) generates the minimally biased parameter distribution, i.e. is most\nuncertainabouttheparameterdistribution,subjecttoallavailableknowledgeabouttheparameters.Sincethis\ndistributionisthemostuncertainwaytomodeltheneuralnetworkparameters,itisalsotheleastrestrictiveon\nwhatparametervaluesaresampled,andthusyieldsthemostdiverserepresentationoftheavailableknowledge\nabouttheprocessbeingmodelled.\nTo ensure that this important property of simmering is conveyed strongly in the paper, in addition to the\nexisting description of entropy maximization in Main Methods, we have replaced Main Table 1 with Main\nFigure3whichdisplaystheeffectofthisdiverseensemblegeneration.Seeresponsetothepreviouscomment\nfortheextractsfromthemanuscriptthataddressthis.\n3. In Supplementary Figure 2, Simmering converges much faster than the baseline method on\nTransformers, where it only uses 20 epochs where the baseline takes 60 epochs to reach the\nsamelevelofperformance. Ifthisresultistrue,itcouldbemoreinterestingtherestofthepaper\ncombined. Why is this convergence speedup not explored more in the paper? I find it hard to\nbelievethatitconvergessomuchfaster.\nTheRefereeraisesaninterestingpointaboutourTransformertrainingcomparison.\nThegoalofthisworkistopresentagenericallygeneralizabletrainingmethod. Assuch, thecomparison\nbetweendifferentoverfittingmitigationmethodsisintendedtohighlightthedistinctivefeaturesofsimmering\nthatfacilitategeneralizability.\nSince accelerating convergence was not the aim of this work, we elected not to focus on the effect of\nsimmeringontherateofconvergencewhenpresentingtheseresults. However,weseethattemperaturecan\npositivelyaffecttherateofconvergence,andweexpecttoexplorethisfurtherinsubsequentwork.\n4. MaintextFigure3aboutclassificationconfidence,ontheotherhand,ismuchlessinteresting\nsinceitcontainsmerelyoneillustrativeexample.Itishardtoconcludeanythingfromoneexample.\nIfamorestatisticalmeasureisbeyondthescopethepaper,Isuggestreducingtheemphasison\nthis, by perhaps moving this one to the supplementary materials and add the Supplementary\nFigure1and2tothemaintext,togetherwithrelatedanalyses.\nple. WehavemovedthefiguretotheSupplementaryInformation(nowSupplementaryFigure3).\n(Remarksoncodeavailability):\nIlookedattheoriginalcodeintheinitialreviewanditranwithoutproblem.IhaveNOTyetchecked\nthelatestcodeaboutCIFAR10andtransformers(ifany).\nWeappreciatetheReferee’sinitialreviewofourcode. Wewouldliketonotethatsincethatinitialreview,\nthe only changes to the code were substitutions of the neural network architecture, dataset, and simmering\nhyperparameterstocorrespondtotheCIFAR-10trainingproblem(Figure3)ratherthanthenoisysinecurve\nproblem(Figure1). Thecurrentversionofthecodecanberunwithoutmodificationtotrainaneuralnetwork\nonCIFAR-10.\nWethanktheRefereefortheircarefulexaminationofthework.WebelievethatReferee’ssuggestionshave\npromotedchangesintherevisedmanuscriptthatsubstantiallyimproveitspresentation.\nReport of Referee 2\nMymainconcernshavebeenaddressed,butIhaveonefurthersuggestion. Althoughthisisan\ninterdisciplinarywork,theproposedmethodisultimatelyaimedattrainingneuralnetworkmodels.\nTherefore, the notations and pseudocode in the paper should follow the conventions of neural\nnetworktrainingalgorithms,ratherthanthoseofthermodynamics. Thiswillenableresearchersin\ntheartificialintelligencefieldtobetterunderstandandutilizetheproposedmethod.\nWethanktheRefereefortheirsuggestionforourmethodpresentation. TheRefereeiscorrectthatsomeof\nthenotationweusedmaybeunfamiliartosomemembersoftheartificialintelligenceresearchcommunity.\nHowever, in our reading, the equivalent terminology (in entirely artificial intelligence convention) is not\nemployed consistently in analogous contexts. Using physics notation allows us to present our method in\na cohesive way. However, to aid in the interpretability of our methodology, we have prepared a table that\ndescribes each physics symbol we employ in both Main Methods and Supplementary Methods, alongside\noneormoremachinelearning(ML)analogues.\nWehavealsomodifiedthe“TrainingProcedure”oftheSupplementaryInformationtoprovideadditional\ncontextforthephysics-motivatedaspectsofthetrainingprocedure.\nWerefertotheadditionofthetableinMainMethods:\n...Starting from entropy maximization, we show how sufficient training generically uses the\ngeometryofthelosslandscapetodeviatefromoptimaltraininglossduringsampling,andhow\nthisdeviationiscontrolledbythetemperatureT. Foradditionalinformationaboutournotation,\nseeSupplementaryMethods.\nx∈RN Fullparametervector(weightsand θ(trainableparameters)\nbiases)\nL(x,D) TraininglossevaluatedondatasetD L(θ;D)\nT =1/β Temperature Temperature(Hintonetal.,2015;\nWenzeletal.,2020)\np(x|D) Canonicaldistributionoverparameters Temperedposterior(Wenzeletal.,\n2020)overweights\nθ(x,D)∈Rn Collectivevariables/collective Latentparameter;low-dimensional\ncoordinates embeddingofmodels(Kingma&\nWelling,2022)\ne−βF(θ,D) Effectivefreeenergy Energyinlatentparameterspace\np(θ |D) Distributionovercollectivevariables Temperedposterior(Wenzeletal.,\n2020)overlatentparameters\nx , v , m Realparticleposition,velocity,mass Parametervalueθ ,momentumm\ni i i t t\n(particleindexi) (Kingma&Ba,2017),per-paramscale\ns , v Virtualthermostatparticleposition, Auxiliarythermostatstatesξ,\nk sk\nvelocity(chainindexk) momentumP(Dingetal.,2014)\na (t) Real-particleacceleration Preconditioned/scaledgradient\ni\na (t) Firstthermostatacceleration Temperaturecontrolterm\ns1\na (t) Chainthermostataccelerations(k≥2) Coupledcontrolterms\nsk\nQ Thermostatmasses Hyperparameterscontrollingresponse\nk\ntime\nN LengthoftheNosé–Hooverchain Thermostatdepth\nNHC\nT Targetthermostattemperature Set-pointfortemperature\ntarget\nT Measuredkinetictemperature EmpiricalTemperature/Scaled\nreal\nMomentumVariance\n∆t Integrationtimestep/learningrate Stepsizeη\n(cid:112)\nv ← T/m Velocityinitialization Momentuminitializationconsistent\ni,0 i\nwithT (Dingetal.,2014)\nt, t Iterationindexandmaximumiterations Trainingstepk/totalstepsK\nmax\nTable1: SymbolsusedtodescribesimmeringanditsimplementationinMethodsandSupplementary\nMethods,andtheiranaloguesinmachinelearning(ML)contexts.\nTable 1 describes the notation used in Main Methods and Supplementary Methods. For each\nsymbol,wedescribeitsphysicalmeaning(instatisticalphysicsliterature)anditsequivalentor\nanalogousnotationinmachinelearning(ML)literature.\n...\nAlgorithm 1 describes the implementation of simmering used to generate the results presented\ninthiswork. Therearetwomain“stages”ofsimmering: theequilibrationstage,andthemodel\nsampling stage. The purpose of this algorithm is to sample models from the canonical dis-\ntribution of models at a given temperature, and the equilibration stage ensures that the model\nparameterdistributionreachesthermalequilibrium,i.e.,isdistributedaccordingtothecanonical\ndistribution,beforemodelsaresampled. Themainsimmeringhyperparametersarethethermo-\nstat temperature T, the particle mass m, and the learning rate ∆t. The thermostat temperature\nT can be either a constant or a schedule that depends on training time t. Model sampling can\ncommence once the average parameter temperature T over time is approximately equal to\nreal\nthethermostattemperatureT. Notethattheprecisionwithwhichtheneuralnetworkparameter\ntemperature will fluctuate about T depends on the neural network parameter space dimension-\nality, so expect that T will fluctuate with a larger amplitude for neural networks with less\nreal\nparameters. Forourtrainingexperiments, allrealandvirtualparticleshadthesamemass, i.e.,\nm = Q ∀ i,k, based on Ref. (Martyna et al., 1992). Generally, decreasing the mass makes\ni k\ntheparameterupdatesmoresensitivetothelossgradient(seemdependenceinline20)andthus\ncanaccelerateequilibration. Thelearningrate∆tisfixedandidenticalforallparticles(neural\nnetworkparametersandNHCparticles). WealsonotethatweusedaNHClengthof10forall\nexperiments.\nTocommenceequilibration,providetheinitialweightsandbiasesoftheneuralnetworkasx ,\nall,0\nandproceedwithAlgorithm1. Duringtraining, beforeexecutingline20ofthefor-loopinAl-\ngorithm1,settheneuralnetworkparameterstox ,andusebackpropagationtocompute\nall,t+1/2\nthegradientoftheloss∇ L. Onceequilibrationiscomplete,reusethesamealgorithmforsam-\nx\nplingmodels. Theonlydifferenceinexecutionisthatthemodelparametersand/orpredictions\ngenerated at the loss gradient computation stage (line 18 of Algorithm 1) are saved for offline\nensembleanalysis. Thereisnoprescriptionforsamplingfrequency. However,someoptionsfor\nsamplingfrequencyincludesavingeveryiteration’spredictions,savingthepredictionsatafixed\niterationinterval,orsavingrandomly. Ifsamplingatsomefixedinterval,checkthatthesamples\nare uncorrelated at that interval. After the desired number of samples have been collected, the\nsetofmodelsand/orpredictionscanbeaggregatedoranalyzedasadistributionofsolutionsin\nthewaysdescribedinthemaintext.\nfunctionLwhichhasparametersx,basedontheNHCthermostatdynamicsintegrationinRef.(Jang&Voth,\n1997). Inthisintegrationscheme,particlepositionsareintegratedoveronehalf-step,i.e.,halfofthelearning\nrate,atatime,andcorrespondingvelocitiesareintegratedforwardbyafulltime-stepatonce. Theobjective\nfunction parameters (e.g., neural network parameters) are treated as the positions of N one-dimensional\nparticleswhichevolveovertimeaccordingtoSupplementaryEquations1–9. Inthisalgorithmdescription,\nthefirstsubscriptofeachvariableindexestheparameters(ifortheith parameter, k forthekth evenorodd\nNHC variable), and the second subscript indicates the timestep of the associated variable. Note that since\ntheNHCparticlesareintegrateddifferentlybasedontheirchainposition(evenorodd), thevirtualparticle\nindextakesonvaluesk = 1,...,N /2. Unlessotherwiseindicated(e.g., withaparticleindexofall),\nNHC\noperationsareperformedelement-wiseforallelementsinthegivenvector(s). Ifnoinitializationisspecified,\nassumethatavector’sinitialvalueis0forallelements.\nRequire: T >0: Thermostattemperature\nRequire: ∆t: Learningrate\nRequire: L(x): ObjectivefunctionLwithparametersx\nRequire: x : Initialneuralnetworkparametervector,ofdimensionN\nall,0\nRequire: m: Neuralnetworkparametermassvector,ofdimensionN\nRequire: t : Numberofiterations\nmax\nRequire: Q: NHCparticlemassvector,ofdimensionN\nNHC\n1: functionPOSITION_FORWARD(a,b,dt) ▷Integratorforparametersx i andpositionss 2k ,s 2k−1 .\n2: returna+dt·b\n3: functionVELOCITY_FORWARD(a,b,c,dt) ▷Integratorforvelocities(v i ,v s2k ,v s2k−1 ).\n4: returnaexp{(−dt·c)}+dt·b·exp{(−dt/2·c)}\n5\n6\n:\n:\nfunc\nr\nt\ne\ni\nt\no\nu\nn\nrn\nA\n(cid:0)\nC\n(cid:80)\nCE\nj\nL\nm\nER\nj\nA\nv\nT\nj 2\nI\n−\nON\nN\n_S\n·\n(v\n(cid:1)\n,\n/\nm\nQ 1\n,T,N,Q 1 ) ▷AccelerationofthefirstNHCparticle(a s1 ).\n7: functionACCELERATION_SK(v,T,b,c) ▷AccelerationofallotherNHCparticles(a s2k ,a s2k−1̸=1 ).\n8: return(c·v2−T)/b\n9: functionREAL_T(v,m) ▷Forcomputingtheparametertemperature.\n10: return( (cid:80) j m j v j 2)/N\n(cid:113)\n11: v i,0 ← m T ▷Initializevelocityvectorelements.\n12: T real ←T ▷Defineinitialparametertemperature.\n13: t←0 ▷Initializetimestep.\n14: whilet<t max do\n15: x i,t+1/2 ← POSITION_FORWARD(x i,t ,v i,t ,∆t/2) ▷Firsthalf-stepupdatetox i .\n16: s 2k,t+1/2 ← POSITION_FORWARD(s 2k,t ,v s2k,t ,∆t/2) ▷Firsthalf-stepupdatetos 2k .\n17: a s1,t ← ACCELERATION_S1(v all,t ,m all ,T,N,Q 1 )\n18: a s2k−1̸=1,t ← ACCELERATION_SK(v s2(k−1),t ,T,Q 2k−1 ,Q 2(k−1) )\n19: v s2k−1,t+1/2 ← VELOCITY_FORWARD(v s2k−1,t ,a s2k−1,t ,v s2k,t ,∆t/2) ▷Firsthalf-stepupdateto\nv .\ns2k−1\n20: a i,t+1/2 ←−∇ xi L(x all,t+1/2 )/m i ▷Uselossgradienthere!\n21: v i,t+1 ← VELOCITY_FORWARD(v i,t ,a i,t+1/2 ,v s1,t+1/2 ,∆t) ▷Full-stepupdatetov i .\n22: s 2k−1,t+1 ← POSITION_FORWARD(s 2k−1,t ,v s2k−1,t+1/2 ,∆t) ▷Full-stepupdatetos 2k−1 .\n23: a s2k,t+1/2 ← ACCELERATION_SK(v s2k−1,t+1/2 ,T,Q 2k ,Q 2k−1 )\n24: v s2k̸=Nc ,t+1 ← VELOCITY_FORWARD(v s2k,t ,a s2k,t+1/2 ,v s2k+1,t+1/2 ,∆t) ▷Full-stepupdateto\nv .\ns2k\n25: v sNc ,t+1 ← VELOCITY_FORWARD(v sNc ,t ,a sNc ,t+1/2 ,0,∆t) ▷Full-stepupdatetov sNc .\n26: x i,t+1 ← POSITION_FORWARD(x i,t+1/2 ,v i,t+1 ,∆t/2) ▷Lasthalf-stepupdatetox i .\n27: s 2k,t+1 ← POSITION_FORWARD(s 2k,t+1/2 ,v s2k,t+1 ,∆t/2) ▷Lasthalf-stepupdatetos 2k .\n28: a s1,t+1 ← ACCELERATION_S1(v all,t+1 ,m all ,T,N,Q 1 )\n29: a s2k−1̸=1,t+1 ← ACCELERATION_SK(v s2(k−1),t+1 ,T,Q 2k−1̸=1 ,Q 2(k−1) )\n30: v s2k−1,t+1 ← VELOCITY_FORWARD(v s2k−1,t+1/2 ,a s2k−1,t+1 ,v s2k,t+1 ,∆t/2) ▷Lasthalf-step\nupdatetov .\ns2k−1 8\n31: T real ← REAL_T(v all,t+1 ,m all ) ▷Optional: Trackparametertemperaturethroughoutsampling.\n32: t←t+1\n(Remarksoncodeavailability):\nNA\nWethanktheRefereefortheirsuggestionregardingmethodpresentation. Webelievetheresultingchanges\nallowourworktobemoreeasilyinterpretedbymembersoftheartificialintelligencecommunity.\nTitle: Simmering: sufficientisbetterthanoptimalfortrainingneuralnetworks\nAuthors: IrinaBabayan,HazhirAliahmadi,GregvanAnders\nResponsetoRefereeReports\n,\nThank you for your communication and efforts in editing and collating these reviews. We also thank both\nRefereesfortheirfeedbackonthemanuscript.\nThe reports are reproduced below in their entirety. The quotations are verbatim and indented in blue,\nsans-seriffont.\nWehopethatyouwillfindthemanuscriptsuitableforpublicationinNatureCommunications.\nReport of Referee 1\nMy suggestions and concerns have been addressed in the latest manuscript. The results pre-\nsentedinthepaperareinterestingandconvincingtome.\n(Remarksoncodeavailability):\nItwouldbeniceiftheauthorcanprovidethesourcecodeforthetransformerbasedNLPexperi-\nment.\nWethanktheRefereefortheircarefulexaminationofthework,andappreciatetheirinterestinthetransformer-\nbasedNLPexperiment.\nFor the open-source implementation of simmering, we opted to present the CIFAR-10 training problem\nas the use case, because it is feasible to run on hardware that is generally accessible, whereas transformer\ntrainingrequiresspecializedequipment. Nonetheless,thecodethatwehavepublishedcanbeappliedtoNLP\ntasksbyreplacingtheneuralnetworkarchitecture,simmeringhyperparametervaluesanddatasetusedinthe\nsamplecode. InSupplementaryMethods(sectionComparisonWithOtherOverfittingPreventionMethods),\nwedescribethetransformerarchitecture,simmeringhyperparameters,andtrainingprocedure(e.g.,batching,\nnumberofepochs)neededtoreproduceourNLPresults. InthisSupplementaryMethodssection,andinthe\nmain text, we also link to the open-source dataset we used for training. We believe that the description we\nprovidedissufficienttoreproducethetransformer-basedexperiment.\nReport of Referee 2\nMyconcernshavebeenaddressed\nWethankbothRefereesagainfortheirfeedbackoncurrentandpreviousversionsofthemanuscript.\n1"}]

Decision letter