Soft ascent-descent as a stable and flexible alternative to flooding
Authors: Matthew J. Holland
Kosuke Nakatani
Venue: NeurIPS 2024 poster (2024)
Keywords: Flooding
classification
ascent-descent
Primary area: learning_theory
TL;DR: We propose and study a learning algorithm that encourages training losses to concentrate around a pre-fixed threshold.

Abstract: As a heuristic for improving test accuracy in classification, the "flooding" method proposed by Ishida et al. (2020) sets a threshold for the average surrogate loss at training time; above the threshold, gradient descent is run as usual, but below the threshold, a switch to gradient *ascent* is made. While setting the threshold is non-trivial and is usually done with validation data, this simple technique has proved remarkably effective in terms of accuracy. On the other hand, what if we are also interested in other metrics such as model complexity or average surrogate loss at test time? As an attempt to achieve better overall performance with less fine-tuning, we propose a softened, pointwise mechanism called SoftAD (soft ascent-descent) that downweights points on the borderline, limits the effects of outliers, and retains the ascent-descent effect of flooding, with no additional computational overhead. We contrast formal stationarity guarantees with those for flooding, and empirically demonstrate how SoftAD can realize classification accuracy competitive with flooding (and the more expensive alternative SAM) while enjoying a much smaller loss generalization gap and model norm.

Soft ascent-descent as a stable and flexible alternative to flooding
Matthew Holland*
Kosuke Nakatani Osaka University
Osaka University
Abstract
As a heuristic for improving test accuracy in classification, the "flooding" method proposed by [Ihida et al. (2020) sets threshold for the average surrogate loss at training time; above the threshold, gradient descent is run as usual, but below the threshold, switch gradient ascent is made. While setting the threshold non- trivial and is usually done with validation data, this simple technique has proved remarkably effective in terms of accuracy. On the other hand, what if we are also interested in other metrics such as model complexity or average surrogate loss at test time? As an attempt achieve better overall performance with less fine-tuning, we propose a softened, pointwise mechanism called SoftAD (soft ascent-descent) that downweights points on the borderline, limits the effects of outliers, and retains the ascent-descent effect of flooding, with no additional computational overhead. We contrast formal stationarity guarantees with those for flooding, and empirically demonstrate how SoftAD can realize classification accuracy competitive with flooding (and the more expensive alternative SAM) while enjoying a much smaller loss generalization gap and model norm.
Introduction
Modern machine learning makes use of sophisticated models that are trained through optimization of non-convex objective functions, which typically admit numerous local minima that make for natural candidates when taken at face value. While many such candidates are indeed essentially "optimal" from the viewpoint of classification error rates or other average losses incurred training time, these often turn out to be highly sub-optimal in terms of performance at test time. It goes without saying that understanding and closing this gap is the problem of "generalization" that underlies most machine learning research (Jiang et al. 2020 Dziugaite et al. 2020: Johnson and Zhang 2023).
When are faced with multiple candidates which essentially optimal and thus indistinguishable in terms of some "base" objective function (e.g.. the average loss) at training time, one of best-known heuristics for identifying good candidates is that of the "landscape" "geometry" of the base objective in a neighborhood around each candidate. Roughly speaking, one expects that candidates in regions which are in some sense "flat" (often said to be less "sharp"") tend to perform better at test time. Strictly speaking, flatness is not necessary for generalization (Dinh et al.) 2017). but our intuition can often be empirically verified be correct, as good generalization regularly observed in flat regions where the eigenvalues of the Hessian are mostly concentrated near zero (Chaudhari al. 2017). The spectral density of the Hessian can in principle be used to evaluate sharpness, and has well-known links to norms that can be used for explicit regularization (Karakida et al., 2011)). but for large-scale neural network training in practice, first-order approximations have shown the greatest utility. In particular, the sharpness-awar minimization (SAM) algorithm of Foret et al. (2021), extended for scale invariance by [won et al. (202T) and later captured as a special case of the gradient norm penalization (GNP) scheme of Zhao et al. (2022), has shown state-of-the-an
Cooresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

performance on a variety of deep learning tasks. All of these first-order procedures can be cast as (forward) difteeedfferenee approximations of the curvature (Karakida et al. 2023). requiring at least double the computational cost of vanilla gradient descent (GD) at each iteration.
As alternative approach, the "flooding" technique of Ishida al. 2020) is worthy of attention for surprising improvements in test accuracy despite its apparent simplicity. Flooding is done as follows: fix threshold before training and run vanilla GD until the average loss goes below 0, and while below this threshold, run gradient ascent rather than descent (see $2. for details). Flooding appeared before SAM in the literature, but near the threshold, flooding can iterate between optimizing the empirical risk and the squared gradient norm (penalizing sharpness), establishing the former as an inexpensive alternative the latter. On the other hand, it is not at all obvious how the flooding threshold 0 should be set given a particular data distribution and model class, and at present there is no methodology for settings which are "optimal" or at least "sufficient" from the viewpoint of test accuracy. More importantly, what if we are interested in performance criteria going beyond that of classification accuracy? Flooding just says "make the average loss as close to as possible," " and we hypothesize that this requirement is too weak to encourage low model complexity and/or good generalization in terms of losses, while also keeping test accuracy high.
In this work, we investigate the validity of this hypothesis, and consider the impact of making stronger requirement, namely to ask the algorithm to "make sure the loss distribution is well- concentrated near 6." We show in $ that this can be implemented by introducing a smooth wrapper, applicable to any loss, which penalizes both per-perfonnngg and ander--erforming examples in per-point fashion, instead of applying a hard threshold to the whole (mini-)bateh as in flooding. We call this proposed procedure "soft ascent-decent" (SoftAD), and provide detailed comparison with the flooding technique, highlighting the smoothness of SoftAD with implications in terms of formal stationarity guarantees, and emphasize how our mechanism leads to update directions that qualitatively distinct from those used in flooding. Through rigorous empirical tests using both simulated and real-world benchmark classification datasets, featuring neural networks both large and small, we discover that compared with ERM, SAM, and flooding, the proposed SoftAD achieves far and away the smallest generalization error in terms of the base loss, while maintaining competitive accuracy and small model norms, without any explicit regularization.
Before diving into the main results just described, we introduce notation and background concepts in $2 SoftAD is introduced in $ where we make basic empirical comparisons to flooding in $ and contrast formal guarantees stationarity for these two methods in $3.2 Our main empirical test results are given in $ with discussion and concluding remarks wrapping up the paper in $5 All detailed proofs and supplementary empirical results are relegated to the appendix.
2 Background
To begin, we formulate performance metrics characterizing the problem of interest. Let W C Rd parameterize our hypothesis class, let Z denote the set to which individual data points a belong. Let Z represent random (test) data point with distribution f over Z. For classification, where our data takes the form Z (X, Y) and for each w e W we let Yww) denote the predicted label given X, the traditional performance metric of interest is the error probability at test time, denoted by
(()) = P (V(()) * }
(1)
When we refer to test accuracy, we mean the probability of correct prediction, namely 1 Eww). Even when high accuracy is desired, it is standard to make use of computationally congenial surrogate loss functions for training. Let é: Rd x Z + R denote a generic loss function to be used for training. For example, if (x., y) represents (input, label) pairs, then typical choice of & would be the cross-entropy loss. While the model is left abstract in our notation, note that for non-linear models such neural networks, the mapping w + E(w; =) may be non-convex and non-smooth over W. As with the error probability (T). the expected loss (or "risk")
R = Ep. E(w: Z)
is also an important indicator of classifier performance. Both and 2 are ideal quantities since y is unknown and is not observed at training time. We assume the learning algorithm has access to an independent sample of n observations from μ. denoted by Z, = (Zi...., ,,,) for

---

convenience. Traditional machine learning algorithms are driven by the empirical risk, denoted here by R,, (w) = (1/n) Ei=1 l(w; Z,), in that they seek (local) minima of R In this paper. we use the term empirical risk minimization (ERM) to refer to algorithms that directly apply optimizer to Ra. Under sophisticated models, the usual ERM objective (-) tends to admit complex landscape. As discussed in O numerous alternatives to ERM have been proposed, with the aim of minimizing E() more reliably; next we take a closer look at one such technique, called "flooding."
2.1 Flooding
The basic intuition underlying the proposal of Ishida et al. (2020) is that while minimizing (-) may be sufficient for maximizing the training accuracy, it need be necessary. and from the perspective of optimizing E-)), it may be worth it to sacrifice (.) and even (.).). Fixing threshold 0 e R, the "flooded" objective is
(w: 0) = 0 + (w) 8)
(3)
This objective can be implemented simple wrapper around typical loss functions, to which off-the-shelf gradient-based optimizers can be applied; running vanilla sub-gradient descent yields sequence (W1, .... generated using the update
Wi+1 = Wy - a sign (wxx) 0) (wr)
(4)
for all 2 1, where sign(x) = 2/1an for all x * and sign(0) = O, and a > is a fixed step size. The update ( characterizes what we call the Flooding algorithm. From the above definitions, Fn(w;; 0) is minimal and only RR (w) = 0. That is, Flooding seeks out w such that the training loss distribution induced by (e(w; ...,.... ... [(w; Zn)) has mean which is close to 0; nothing more, nothing less.
2.2 Links sharpness
Assuming for now that the loss is differentiable, it has been appreciated for some time that the distribution of the loss gradients ((( Z) can convey important information about generalization (Zhang et al. (2020), and in particular the role of gradient regularization, both implicit and explicit, is receiving significant attention (Barrett and Dherin 2021 (mmth al. 2021 ( As concrete example of explicit regularization, consider modifying the ERM objective using the squared Euclidean norm
as
R₂ (w; A) = (w) + [w))l"")
(5)
where À 2 O controls the degree of penalization. one is to minimize this objective in w directly using gradient descent, this involves computing
VŘ, (w; A) (R, + AV (w) (w),
and thus doing matrix multiplication using a x d Hessian V²,, an unattractive proposition when d is large. A linear approximation to the expensive term can be obtained via
VR, (w + au) (R,(w) = V²R,,()) (u)
where u E Rd is arbitrary and |a| small; see Zhao al. (2022 $3.3) for example. Applying this to approximate VR, (w; A). we have
VR, (V + AVR, (w)) VR, (w)) R VR, (w; A).
The iterative update directions used by the SAM algorithm of al. (2021) are captured by setting X, offering a nice link between loss-based based sharpness control and gradient regularization. The
extension of SAM GNP (Zhao et al. 2022) can be expressed by an analogous derivation, replacing
'Trroughout this paper, all gradients are taken with respect to w + f(w; z), assumed to exist on an open set containing W for all x E Z. It should however be noted that gradients taken with respect to parts of the data have been used in objective function design for years; see Drucker and Le Cun (192)).

---

the squared norm 11-113 in (5) with ||-II. Using updates of the form given in with a > 0 is called "forward" difte-difference (FD) approach to explicit gradient regularization (GR) (henceforth, FD-GR), and clearly requires two gradient calls per update, 3 Better precision is available using "centered" FD, at the cost of additional gradient calls (Karakida et al., 2023). How does this all relate to Flooding? In repeating the update { once the empirical risk R,, goes below 0 and the algorithm switches from ascent to descent, it is straightforward to show conditions where this leads to iteration between minimizing R,, and (, We give more details in $B.1
3 Soft ascent-descent
With the context of 2 in place, we consider making two qualitative changes to the Flooding update described as follows.
1. Pointwise thresholds: Invert the order of applying (-) and sign(.), i.e., do summation over data points after per-loss truncation.
2. Soft truncation: (increasing) function e(-) satisfying (0)
Replace the hard threshold sign(.) with continuous, bounded, monotonic
The reason for making the thresholds pointwise is to allow the algorithm to view "ascent" and "descent" from the perspective of individual losses (rather than bundled up in Ra). making it possible to utilize sum of both ascent and descent update directions soft truncation using e plays a key role. Keeping & bounded limits the impact of errant loss values, while the other assumptions allow for both ascent and descent, with "borderline" points near the threshold given less weight. Written explicitly as an empirical objective function, we use
To To make this sum weighted sum, the
1 Σ
(w: 0) = 0 + TI
ple(w; Z,) - -0)
where once again E R is a fixed threshold, and we set p(x) = v/² +1-11111 on ()) in ( yields the update
11-11.1 1. Running vanilla GD
W++1 = We
a n
(k(w; Z.) - 8) V) Z.),
(8)
and 0 is once again a fixed step size. For convenience, with = (or SoftAD for short) to refer to the algorithm implied by the iterative
we use soft
update (8) Note that there is nothing particularly special about choice of p here; it is just simple algebraic function whose derivative also takes a simple form; note that the function e resulting from our choice of satisfies the desired properties of continuity, boundedness, monotonicity, and
OF Note that for each point being summed over, Eww; Z,) > 0 implies descent while Eww;; Z:) < 0 iimplis ascent, and borderline points with f(w;; Z.) s 0 have a smaller relative impact. In contrast with Flooding, SoftAD requires that the training loss distribution induced by (f(w; Zi)..., (ww; Zn)) be well-concentrated around 0, where the degree of dispersion is measured in symmetric fashion using p.
((0) = p(())
Remark I (Comparison with other variants of Flooding). During the review phase for this work, we were made aware of another recent related work by Xie et al. (2022). Their proposed method is known as iFlood, and it is essentially middle ground between our proposal above and the original Flooding procedure. They use pointwise thresholds as we do in SoftAD, but retain the hard ascent-descent switch in Flooding, More concisely, replacing value |z]] yields the iFlood update. We have added empirical test results for iFlood to complement our original experiments at the end of $2.2 Another very recent variant is AdaFlood (Bae et al.) 2023), which sets the threshold level individually for each point based on "difficulty" as evaluated using an auxiliary model.
soft truncator o(x) in (8) with the absolute
In the current example, one call at w. and another call at w + RVR
"This cannot be achieved by taking mini-batch of size when using Flooding, since the number of gradients summed over always equals the number losses averaged when checking the ascent/descent condition.
'There are many other possible candidates, but this is typical "smooth Huber function." also known in the literature as pseudo-Hube Charbonnier, and L1-L2 (Barron 2019)

---

2,500
1.000
Floed
2.225
0.778
1.950
0.556
0.333
1.0
0.300
0.778
0.025
1.000
2]. The two remaining figures show plots of the graphs of (z) (dashed black line) and (((((=) 0)/a) (xr for the range of values, with colors corresponding modified values of (middle plot; 0 0.5 fixed) and (right-most plot; o = 1.0 fixed) respectively. Thick dotted lines are @ sign, thin solid lines are Ø p'.
Figure 1: The left-most figure simply plots the graph of =x) over
Deterministic comparison (Xo = =2.0, 0.5, 0.0))
Iterates (xp)
Function values (fix))
0.0
2.0
GD
GD
SoftAD Flood
SoftAD
Flood
-1.0
www.
www.tsd
Www
-2.0
20
40
60
80
100
20
40
60
80
100
Figure 2: Gradient descent on the quadratic example from Figure The horizontal axis denotes iteration number, and we plot sequences iterates (x+) and function values (z:)) vanilla gradient descent, with "Flood" and "SoftAD" corresponding to and (8) respectively. Step size is 0.1.
) for each method. Here "GD" denotes
Remark 2 (Difference from OCE-like criteria). At first glance. objective S (w; 0) in (7) may appear similar to the criteria used OCE risk minimization (Lee et al. 2020 Li et al. 202 and some varieties of DRO (Duchi and Namkoong 2021). Aside from the obvious difference that is fixed, rather than optimized alongside w, the critical difference here that our p(-) is not monotonic. Losses which are too large and too small are both penalized. It is precisely this bi-directional property that allows for switching between ascent and descent; this is impossible to achieve with monotonic OCE and DRO risks (Holland 2023 Holland and Tanabe 2023 Hu et al. 2023 Royset 2024). This bi-directional criterion can also be used as a straightforward method to provably avoid unintentional "collapse" into ERM solutions (Holland 2024).
3.1 Initial comparison with Flooding
To develop some intuition for our SoftAD (8) and the Flooding update (4). we carry out a few illustrative numerical experiments. To start, let us consider simple, non-stochastic example one dimension. Letting f R + R be some differentiable function, we consider how the transformed gradient ((((x) 0) behaves under & sign and p'. In Figure | we give numerical example using quadratic function. The softened nature of the transformed gradients used in SoftAD clear when compared with the hard switching mechanism underlying the Flooding update. In Figure we continue the quadratic function example, looking at sequences (I1, I2,,.2 generated based on the Flooding and SoftAD procedures. That is, instead points from which to compute losses, we have just one "loss." namely f(x) x2/2. Both procedures realize effective "flood level" of sorts (i.e., buffer around the loss minimizer),, as expected, the Flooding procedure tends to be far more "jagged" in its trajectory.

---

Flooding
SoftAD
Figure 3: Left: randomly sample n points (black dots) from the Gaussian distribution, zero mean, zero correlations, with standard deviation 2v2 in each coordinate. The two candidates are denoted by square-shaped points (red and green), and the minimizer of R. is given by gold star. Center: The Flooding updates (colored arrows) via (4) each candidate. Right: Analogous SoftAD update vectors (8) with per-point transformed gradients (semi-transparent arrows) for reference. Throughout, we have fixed 1.5 minl Ru (w) and 0.75.
Finally, simple example to illustrate how the per-point soft thresholding of SoftAD leads to distinct gradient-based update directions when compared to the Flooding strategy. Here we consider a dataset of n data points 21...., 2n E R.2, and use the squared Euclidean norm as a loss, i.e., [(w; 2) = ||ue- - =lI?. This is natural extension of the quadratic example in the previous paragraph, to multiple points and two dimensions. In Figure we look at two candidate points, and compute the Flooding and SoftAD update directions that arise each candidate under randomly generated dataset. We can clearly see how the Flooding update plunges directly towards the minimizer of Ra, unless it is too close (given threshold 0), in which case it goes in the opposite direction. In contrast, the SoftAD update composed of per-poiin point update directions, some which attract toward the minimum, and some which repel from the minimum. with borderline points down-weighted (shorter update arrows). Since the final update averages over these, movement both toward and away from the minimum is clearly "softened" when compared with the Flooding updates.
3.2 Comparison of convergence properties
With Flooding and SoftAD methods mind, next we consider concrete conditions under which stochastic cradient-based learning algorithms can be given guarantees of (approximate) stationarity. Throughout this section, we assume that the loss w + É(w; 2) is locally Lipschitz on W for each E Z, but convexity will not be used. Let us assume for simplicity that (Z1, Ze.... is sequence of independent random variables with distribution μa, the same as our test data point ~ μ.
Starting with SoftAD, we are interested in procedures fuelled by stochastic gradients of the form
(w(w) = <(((w; Z,) - 0) (ww; Z,)
for all integers 2 and W, with threshold 0 E fixed in advance, and p' before. Recalling the empirical SoftAD objective S,, in (7). the underlying population objective is
(,(w) = + E,, p(l(w; Z) -)).
(10)
By design, do not expect SoftAD approach stationary point of the original Ry, Note that under mild assumptions on the data distribution, we have an unbiased estimator with E, G,(w)= VS, (w). suggesting stationarity in terms of (-) natural goal. Assuming the losses Lz-smooth in expectation, one can readily confirm that the objective (10) is LAD-shooth, with
LAD = E,, sup [[: 2) + Le.
(11)
LweW
more detailed derivation given in BB.5 Assuming that second-order moments are finite uniform sense over W ensures that LAD < and naturally implies pointwise variance bounds, allowing us to seek out stationarity guarantees using combination of gradient norm control and momentum in the fashion of Cutkosky and Mehta 2021).

---

but rather Proposition 3 (Stationarity for SoftAD, smooth case). Starting with an arbitrary W. update using W = W/ MMell where M, == 6M,- + (1 ()Gr(w)) for t > 1, with Mo == 0 and Gr(-) = G,(-) min 2/ll))))))l)))) taking each gradient G+(-) based on ② Assuming we make T - 1 updates. set the momentum parameter to b = 1 1/vT, the norm threshold to VLLAD Le)/(1- - b), and the step size to 1/T3/4 The stationarity of this sequence (Wi, W2,...) assumed be in W, in terms of the modified objective (10) can be bounded by
T T [w,(( $ T1/4
1
(Sy(wi) (Satt)))S,koe Sp( +
3LAD
+ LAD - (1+
2
factor Cs- = 10 10 + +4/1og (3 1, with probability no less than 8 over the draw of (Z1, Zz....).
using
For comparison, next consider stationarity of the Flooding algorithm in the same context of smooth losses. The argument used in Proposition critically depends on the smoothness of the underlying objective (TO). Unfortunately, this smoothness cannot be leveraged when we consider the population objective underlying the Flooding procedure, namely the function
+ (w) 0).
(12)
Further complicating things is the fact that stochastic gradients of the form 9 with o(-) replaced by sign(.) do not yield unbiased sub-gradient estimates for 12). but rather for an upper bound 0 0+E,, (w:;; Z) 0l that follows via Jensen's ss inequality. A lack of smoothness means we cannot establish stationarity in terms of ( nor the upper bound just given, but possible using a smoothed approximation (the Moreau envelope) of this bound:
(w=))== = inf vEW
0 E, Z) - 0 +
1 23
w/l?
(13)
The parameter >> controls the degree of smoothness. The objective in (33 can be linked to "gradients" in (9) with & = sign, and leveraging the Lipschitz continuity of along with a sufficiently smooth loss, it is possible to show that this non-smooth objective satisfies a weak form of convexity, and using the techniques of Davis and Drusvyatskiy (2019) it is possible to show that stochastic gradient algorithms enjoy stationarity guarantees, albeit not in terms of the objective 12 the smoothed upper bound (13).
Proposition 4 (Stationarity for Flooding, smooth case). Letting W be closed and convex, take an initial point wi E W. and make T updates using W1+1 Il [w aGe (w.)], where Hwll denotes projection to W. and each Gx(-) is computed using (9) with Φ = sign. Assuming the loss w + E(u; 2) is I, -smooth on for all 2, and taking a step size of
a²
L.)' using A such that A Fu()))
inf F,,(w,
WEW
with LAD and Le as in M the expected squared stationarity in terms of the smoothed upper bound 13 at smoothness level B 1/(2L)) can be controlled as
1 T
2L 2LA Le)A
T
M E V
(w:))1
<
T
with expectation taken over the draw of (Zi, Zz....)
Remark 5 (Comparing rates and assumptions). Considering the preceding Propositions 3 and one common point is that learning algorithms based on both the SoftAD and Flooding gradients (of mini-batch size 1) can be shown to be approximately stationary in terms of functions of a similar form (i.e., (10) and (12), differing only in how they measure deviations from the threshold 6. The rates of decrease (as function of T) are essentially the same, noting that the bounds in Proposition are in terms of squared norms. That said, a lack of smoothness means the Flooding guarantees only hold for a smoothed variant, plus they require a stronger form of smoothness in the loss (over all vs. in expectation). In addition, the SoftAD guarantees hold with high probability over the data sample, and can be readily strengthened hold for an individual iterate (instead of summing over T iterates), using for example the technique of Cutkosky and Mchta 2021 Thm. 3).
7

---

Gaussian
Sinusoid
Spiral
CIFAR-I0
CIFAR-I00
Fashion
SVHN
ERM
0.080
0.150
0.297
3.265
7.603
0.801
0.762
Flood
0.011
0.058
0.119
1.239
3.114
0.436
0.436
SAM
0.024
0.096
0.154
1.512
3.672
0.639
0.493
SoftAD
0.004
0.016
0.087
1.168
2.701
0.422
0.362
Table Generalization gap (test training) for averaaee cross entropy loss after final epoch.
4 Empirical study
In this section, we apply the proposed SoftAD procedure to a variety of classification tasks using neural network models, leading to losses that are non-convex and non-smooth, Our goal here is to compare and contrast the behavior and performance (accuracy, average loss, model norm) of SoftAD with three natural alternatives: ERM, Flooding, and SAM 6
4.1 Overview of experiments
Our core experiments are centered around re-creating the tests done by Ishida et al. (2020) $4. 1, $4.2) and Foret al. (202T) $3.1) to include all four methods of interest. There are two main parts: simulation- -aaed tests and real benchmark- -bbsed tests. We briefly describe the setup of each below.
Non-linear binary classification on the plane We use three synthetic data-generators ('two Gaussians," " "sinusoid," and "spiral," see Figure 7 to create a dataset on the 2D plane that is not linearly separable, but separable using relatively simple non-linear models. We treat the underlying model as unknown, and approximate it using a shallow feedforward neural network. All four methods of interest (ERM, Flooding, SAM, and SoftAD) are driven by the Adam optimizer, with the cross-entropy loss used as the base loss function. Complete experimental details are provided in C
Image classification from scratch Our second set of experiments utilizes four well-known bench- mark datasets for multi-class image classification. Compared to the synthetic experiments, the classification task is more difficult (much larger inputs, variation within classes, more classes), and so we utilize more sophisticated neural network models to tackle the classification task. That said, as the sub-section title indicates, this training is done "from scratch," i.e., no pre-trained models are used. The datasets we use are all standard benchmarks in the machine learning community: CIFAR- 10, CIFAR-I00, 100, MNshi and SVHN. Model choice essentially mirrors that of Ishida et al. (2020) $4.2). For FashionMNIST, forward neural network with one hidden layer. For SVHN, we use ResNet-18 as implemented in torchvision. models, without any pre-trained weights. Finally, for both CIFAR-10 10 and CIFAR- 100, we use ResNet-34 (again in torchvision models) without pre-training. For the optimizer, we use vanilla SGD with fixed step size. Full details are given in C22 in the appendix.
flatten each image into vector, and
simple feed-
4.2 Main findings
Uniformly small loss generalization gap One of the most lucid results we obtained is that SoftAD shows the smallest loss generalization gap of all the methods studied, across all models and datasets used. In [ we show the gaps incurred under each dataset. More precisely, for each trial and each epoch, we compute the average cross-entropy loss on test and training (less validation) datasets, and then respective average both of these over all trials. The difference of these two values (i.e., riial-aeeagee test loss minus tvial-averaged training loss) after the final epoch of training is the value shown in each cell of the table.
Balance of accuracy and loss real data In the previous paragraph noted that SoftAD has superior loss gaps, but this is not much to celebrate if performance in terms of the key metrics of interest (i.e., test loss and test accuracy) is poor. In Figures 5 we show the trajectory of loss and accuracy (for both training and test data) over epochs run (averaged over trials). All four methods
'To re-create all of the numerical test results and figures from this paper. source code and Jupyter notebooks are available public GitHub repository: httpp::/ //gtthub.bbtttt// http//eebbacckaar//ooo--eeeeeeee
8

---

Loss (cifar10)
Loss (cifar
10.
3.0
8
1.0
Accuracy (cifar10)
Accuracy (cifar100)
0.4
0.3
ERM
ERM
0.4
Flood
Flood
0.2
SoftAD
SAM
0.1
SoftAD
SAM
0.0
0
100
200
300
400
500
100
200
300
400
500
Figure 4: Trajectories over epochs for average test loss (top row) and test accuracy (bottom row). Horizontal axis is epoch number. Columns associated with the CIFAR-10 and CIFAR-100 datasets (left right).
Loss (fashionmnist)
Loss (svhn)
1.5
1.0
0.5
0.0
Accuracy (fashionmnist)
Accuracy (svhn)
0.90
0.6
0.85
Flood
0.4
Flood
0.80
SoftAD
0.2
0.75
300
400
500
400
500
Figure Analogous to Figure but with FashionMNIST and SVHN datasets.
are comparable in terms of accuracy, with SAM (at double the gradient cost) coming out slightly ahead. On the other hand, there is significant divergence between the different methods in terms of test loss. For each dataset, SoftAD achieves superior test loss, often converging faster than any of the other methods; this may be natural by-product of the fact that SoftAD is designed ensure losses are well-concentraaee around threshold 6, instead just asking that their mean get close to (as in Flooding). While there is clearly major difference between ERM and the other three methods, the stable nature the "double descent" in SoftAD is quite stark compared with Flooding and SAM.
Uniformly smaller model norms We do not do any explicit model regularization (e.g., L2 norm penalization) in our experiments here, and we only use fixed step-size parameters for Adam and SGD, so as we run for many iterations, the norm of the model weight parameters tends to grow. While this property holds across all methods tested here, we find that under all datasets and models tested, SoftAD uniformly results in the smallest model norm; see Figure 6 for trajectories over epochs for each benchmark dataset. The "Model norm" " values plotted here are the L2 norm of all the model parameters (neural network weights) concatenated into single vector, and these norm values are averaged over trials. Completely analogous trends hold for the simulated datasets as well.
Trends in hyperparameter selection Aside from ERM, the three key methods of interest (Flooding, SAM, SoftAD) each have one hyperparameter. Flooding and SoftAD have the threshold described

---

Model norm (cifar10)
Model norm (cifar100)
250
200
180
160
150
100
200
300
400
500
0
100
200
300
400
500
Model norm (fashionmnist)
Model norm (svhn)
125
300
100
75
200
100
200
300
400
500
0
100
200
300
400
500
Figure 6: Model norm trajectories over epochs for each dataset in Figures
Gaussian
Sinusoid
Spiral
CIFAR-I0
CIFAR-106 100
Fashion
SVHN
Flood
0.05 (0.06)
0.05 (0.05)
0.05 (0.06)
0.05 (0.05)
0.01
0.01
0.03 (0005)
SAM
0.36 (0.19)
0.05 (0.05)
0.05 (0.06)
0.36 (0.19)
0.5 (0)
0.32 (0.6))
0.28 (0.20)
SoftAD
0.08 (0.08)
0.05 (0.06)
0.05 (0.06)
0.08 (0.05)
0.22 (0.14)
0.03 (0.4)
0.13 (0.10)
Table 2:
Hyperparameters selected by validation for each method (averaged over trials). Flooding and SoftAD have threshold 0; SAM has radius parameter. Standard deviation (over trials) is given in small-text parentheses.
in $ and SAM has the radius parameter (denoted by "p" in the original paper). In all tests, we select representative candidate for each method with hyperparameters by using validation data held out from the training data, and in Table we show the average and standard deviation of the validation-bal based hyperparameters over randomized trials (see $Ç for exact hyperparameter grid values). One clear take-away from this table is that the "best" value of (in terms of accuracy) for SoftAD tends to be larger than that for Flooding, and this trend uniform across all datasets, both simulated and real. In particular for the real benchmark datasets. it is interesting to note that while larger threshold (applied to loss distribution) is selected for SoftAD, the resulting test loss value achieved is actually smaller/better than that achieved by Flooding (top row of Figures 5).
5 Limitations and concluding remarks
While previous work had already shown that it is possible to sacrifice performance in terms of losses improve accuracy, the nature of that tradeoff was left totally unexplored, and in $ we put forward the hypothesis that simply asking the empirical loss mean to get close to a non-zero threshold 0, in Flooding, would not be enough to realize competitive tradeoff over varied learning tasks (datasets, models). Our main take-away is that we have empirical evidence that the slightly stronger requirement of "losses well-concentrated around 6" (implemented as SoftAD) can result in an appealing balance of average test loss and accuracy, with the added benefit of a strong (implicit) regularization effect, likely due to the soft dampening effect on borderline points. A more formal theoretical understanding of this regularization effect is of interest, as are empirical studies going far beyond the limited choice of loss functions used here. Our biggest limitation is that the question of "how to set the threshold 8?" still remains without an answer. Any meaningful answer will likely require some user judgement regarding tradeoffs between performance metrics. One potential first approach would be to leverage recent techniques for estimating the Bayes error (Ishida et al. 2023) combined with existing surrogate theory (Bartlett et al., (006) reverse-engineer a loss threshold given a user-specified "tolerable" drop in accuracy, for example.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by JST PRESTO (grant number JPMJPR21C6) and a grant from the SECOM Science and Technology Foundation.
References
Arora, S., Ge, R., Neyshabur, B., and Zhang, Y. (2018). Stronger generalization bounds for deep nets via a compression approach. In Proceedings of the 35th International Conference on Machine Learning (ICML), volume 80 of Proceedings of Machine Learning Research, pages
Bae, W., Ren, Y., Ahmed, M. O., Tung, F., Sutherland, D. J., and Oliveira, G. L. (2023). AdaFlood: Adaptive flood regularization. arXiv preprint arXtv:2311.02891.
Baker, A. (2022). Simplicity, In Zalta, E. N., editor, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Summer 2022 edition.
Barrett, D. G. and Dherin, (2021). Implicit gradient regularization. In The 9th International Conference on Learning Representations (ICLR).
Barron, J. I. (2019). A general and adaptive robust loss function. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages
Bartlett L. (1998). The sample complexity of pattern classification with neural networks: the size of the weights is more important than the size of the network. IEEE Transactions on Information Theory, 44(2)5255-536.
Bartlett, P. L.,, Jordan, M. I., and McAuliffe, J. D. (2006). Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138-155
Chaudhari, P., Choromanska, A., Soatto, S., LeCun, Y., Baldassi, C., Borgs, C., Chayes, J., Sagun, L., and Zecchina, R. (2017). Entropy-SGD: Biasing gradient descent into wide valleys. In International Conference on Learning Representations.
Claeskens, G. and Hjort, N. L. (2008). Model Selection and Model Averaging. Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press.
Cutkosky, A. and Mehta, H. (2021). High-probability bounds for non-convex stochastic optimization with heavy tails. Advances in Neural Information Processing Systems 34 (NeurIPS 2027), pages 4883-4895.
Cutkosky, A., Mehta, H., and Orabona, F. (2023). Optimal stochastic non-smooth non-convex optimization through online-t--oon-conve conversion. arXiv preprint 2rXin:2302.037755v.
Davis, D. and Drusvyatskiy, (2019). Stochastic model-based minimization of weakly convex functions. SIAM Journal on Optimization, 29(1):207-23
Denker, and LeCun, (1990). Transforming neural-net output levels to probability distributions. In Advances in Neural Information Processing Systems 3 (NIPS 1990), volume 3.
Devroye, L.,, Györfi, I.., and Lugosi, G. (1996). A Probabilistic Theory of Pattern Recognition, volume 31 of Stochastic Modelling and Applied Probability. Springer.
Dinh, L., Pascanu, R., Bengio, S.. and Bengio, Y. (2017). Sharp minima can generalize for deep nets. In Proceedings of the 34th International Conference on Machine Learning (ICML), volume 70 of Proceedings of Machine Learning Research, pages 1019-1028.
Drucker, H. and Le Cun, Y. (1992). Improving generalization performance using double backpropa- gation. IEEE Transactions Neural Networks, 3(6))991-997.
Drusvyatskiy, D. and Paquette, (2019). Efficiency of minimizing compositions of convex functions and smooth maps. Mathematical Programming, 178:503-558,

---

Duchi, J. C. and Namkoong, H. (2021). Learning models with uniform performance via distribution- ally robust optimization. The Annals of Statisties, 49(3):138--1406
Dziugaite, G. K., Drouin, A., Neal, B., Rajkumar, N., Caballero, E.., Wang. L., Mitliagkas, I., and Roy, D. M. (2020). In search of robust measures of generalization. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), pages 1
Efron, B. and Hinkley, D. V. (1978). Assessing the accuracy of the maximum likelihood estimator: Observed versus expected Fisher information. Biometrika, 65(3):457-483
Feldman, V. (2016). Generalization of ERM in stochastic convex optimization: The dimension strikes back. In Advances in Neural Information Processing Systems 29 (NIPS 2016), pages 3576-3584.
Flaxman, A. D., Kalai, A. I., and McMahan, H. B. (2004). Online convex optimization in the bandit setting: gradient descent without gradient. arXiv preprint arXiv:cs/0408007v.
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. (2021). Sharpness-awart minimization for efficiently improving generalization. In International Conference on Learning Representations.
Goodfellow, I. J.. Vinyals, O., and Saxe, A. M. (2014), Qualitatively characterizing neural network optimization problems. arXiv preprint arXiv:1412.654:
Grünwald, P. D. (2007). The Minimum Description Length Principle. MIT Press.
Hinton, G. E. and van Camp, D. (1993). Keeping the neural networks simple by minimizing the description length of the weights. Proceedings of the 6th Annual Conference on Computational Learning Theory, pages 5-13.
Hochreiter, S. and Schmidhuber, J. (1994). Simplifying neural nets by discovering flat minima. In Advances in Neural Information Processing Systems 7 (NIPS 1994).
Hochreiter, S. and Schmidhuber, J. (1997). Flat minima. Neural Computation, 9(1):1-42.
Holland, M. J. (2022). Learning with risks based on M-location. Machine Learning, 111:4679-4475 4718.
Holland, M. J. (2023). Flexible risk design using bi-directional dispersion. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics (AISTATS), volume 206 of Proceedings of Machine Learning Research, pages 1586-1623.
Holland, M. J. (2024). Criterion collapse and loss distribution control. In Proceedings of the 41st International Conference Machine Learning (ICML), volume 235 of Proceedings of Machine Learning Research, pages 18547-18567.
Holland, M. J. and Tanabe, K. (2023). A survey of learning criteria going beyond the usual risk. Journal of Artificial Intelligence Research, 73:781-821.
Hu, S., Wang, X., and Lyu, S. (2023). Rank-based decomposable losses in machine learning: survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45.
Ishida, T., Yamane, I., Charoenphakdee, N., Niu, G., and Sugiyama, M. (2023). Is the performance of my deep network too good to be true? A direct approach to estimating the Bayes error in binary classification. In The I /th International Conference on Learning Representations (ICLR).
Ishida, T., Yamane, I., Sakai, T., Niu, G., and Sugiyama, M. (2020). Do we need zero training loss after achieving zero training error? In Proceedings of the 37th International Conference on Machine Learning (ICMI.), volume 119 of Proceedings of Machine Learning Research, pages 4604-4614.
Jia, 7. and Su, H. (2020). Information-theoree local minima characterization and regularization. In Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research, pages 4773-4783
Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. (2020). Fantastic generalization measures and where to find them. In International Conference Learning Representations.
12

---

Johnson, R. and Zhang, T. (2023). Inconsistency, instability, and generalization gap of deep neural network training. Advances in Neural Information Processing Systems 36 (NeurIPS 2023). pages 9479-9505
Karakida, R., Akaho, S., and Amari, S.-i. (2019). Universal statistics of Fisher information in deep neural networks: Mean field approach. In 22nd International Conference on Artificial Intelligence and Statistics (AISTATS), volume 89 of Proceedings of Machine Learning Research, pages 1032-1041.
Karakida. R.,. Takase, T., Hayase, T., and Osawa, K. (2023). Understanding gradient regularization in deep learning: Efficient finite-difference computation and implicit bias. In Proceedings of the 40th International Conference Machine Learning (ICML), volume 202 of Proceedings of Machine Learning Research, pages 15809-1588 -18277.
Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, P. T. P. (2017). On batge-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations.
Kullback, S. (1968). Information Theory and Statistics. Dover.
Kwon, J., Kim, J., Park, H., and Choi, I. K. (2021). ASAM: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pages 5905-5914.
Lee, J., Park, S., and Shin, J. (2020). Learning bounds for risk-sensitive learning. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), pages 13867-13879.
Li, H., Xu, Z., Taylor, G., Studer, C., and Goldstein, T. (2018). Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems 31 (NIPS 2018), pages 3576-3584.
Li, T., Beiramii A., Sanjabi, M., and Smith, V. (2021). Tilted empirical risk minimization. In The 9th International Conference on Learning Representations (ICLR).
MacKay, D. J. C. (1992). A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3):448-472.
Montavon, G., Orr, B., and Müller, K.-R., editors (2012). Neural Networks: Tricks of the Trade, volume 7700 of Lecture Notes in Computer Science. Springer, 2nd edition.
Nesterov, Y. and Spokoiny, V. (2017). Random gradient- minimization of convex functions. Foundations of Computational Mathematics, 17(2)527-5566
Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. (2017). Exploring generalization in deep learning. In Advances in Neural Information Processing Systems 30 (NIPS 2017).
Royset, O. (2024), Risk-adaptive approaches to stochastic optimization: A survey, arXiv preprint arX1v:2212.00856v3.
Schmidt, R. M., Schneider, F., and Hennig, P. (2021). Descending through a crowded valley benchmarking deep learning optimizers. In Proceedings of the 38th International Conference on Machine Learning (ICML), volume 139 of Proceedings of Machine Learning Research, pages 9367-9376.
Sejnowski, T. J. (2020). The unreasonable effectiveness of deep learning in artificial intelligence. Proceedings of the National Academy of Sciences, 1 117745):00333-00038
Shalev-Shwartz, S., Shamir, O., Srebro, N., and Sridharan, K. (2010). Learnability, stability and uniform convergence. Journal of Machine Learning Research, 11:2635-2670.
Smith, S. I.., Dherin, B., Barrett, D., and De, S. (2021). On the origin of implicit regularization in stochastic gradient descent. In International Conference Learning Representations.
13

---

Sun, K. and Nielsen, F. (2021). A geometric modeling of Occam's 's razor in deep learning. arXiv preprint 1riv:1905.1102771
Wu, L., Zhu, Z., and E, W. (2017). Towards understanding generalization deep learning: Perspective of loss landscapes. arXiv preprini arXiv: 10239.
Xie, Y., Wang, Z.., Li, Y., Zhang, C., Zhou, J., and Ding, B. (2022). iFlood: A stable and effective regularizer. In International Conference on Learning Representations.
Xie, Z., Sato, I., and Sugiyama, M. (2020). diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. arXiv preprint urXiv:2002.03493
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, (2017). Understanding deep learning requires rethinking generalization. In International Conference on Learning Representations.
Zhang, J., Karimireddy, S. P., Veit, A., Kim, S., Reddi, S., Kumar, S., and Sra, S. (2020). Why are adaptive methods good for attention models? In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), pages 15393315393.
Zhao, Y., Zhang, H., and Hu, X. (2022). Penalizing gradient norm for efficiently improving general- ization in deep learning. In Proceedings of the 39th International Conference on Machine Learning (ICML), volume 162 of Proceedings of Machine Learning Research, pages 26982-26992.
14

---

A
Bibliographic notes
In this section, we provide additional references intended complement those in the main body of the paper.
A.1 Broad overview
The following questions succinctly summarize key aspects of the problem of generalization
QP. What properties at training time are reliable indicators of performance at test time?
QA. How can we efficiently find candidates with such desirable properties?
The easy answer to these questions is. of course, "it depends." " There is no fixed procedure that can guarantee arbitrarily good performance on all statistical learning problems, even if restricted binary classification tasks [ A more subtle answer involves characterizing the problems on which abstract learning algorithms such empirical risk minimization (ERM) yield tight bounds on tractable criteria of interest (e.g., the expected loss) Even more difficult is refining our understanding of the learning problems on which concrete algorithms used in practice can be reliably expected perform well 10
For conceptual grounding, we make use of the two questions, QP (the "property" question) and QA (the "algorithmm" question), particularly within the context of non-linear models such as neural networks. Broadly speaking, in the machine learning literature over the past three decades, most answers to the property question QP come in the form of quantifying some notion of "simplicity," property of candidate w. It goes without saying that the underlying heuristic is that all else equal (at training time), "complex" candidate seems intuitively less likely perform well test time [T] As for the algorithm question, there are numerous "workhorse" procedures of machine learning that are computationally convenient, have high-quality software available, and tend to generalize very well in practice, providing a partial answer to QA. That said, the design principles underlying these procedures are often only very loosely related to the properties that satisfy QP With this in mind, a large body of research can be understood as trying to develop new connections between answers to QP and QA, either through post hoc analysis using existing concepts, or by introducing new properties and deriving algorithms in a more unified fashion.
A.2 Notions of model complexity
Model complexity is concept that has long history in the context of statistical model selection (Claeskens and Hjori 2008), with well-established ties to information theory (Kullback) 1968).
Assuming the quality of "ftt" is measured using negative log-likelihood, the second derivative of this objective function (Hessian matrix in multii-dimesioonll case) is known as the Fisher information (matrix) In the context of neural networks, it is common to use their outputs to model probabilities DDnker and LeCun 1990), and thus at least conceptually, much of the existing statistical methodology be inherited. For early work in the context of backprop-driven neural networks, |MacKay| (1992) looks at designing objective criteria for comparing and choosing between models (including norm regularization parameters). MacKay introduces form of Bayesian "evidence" for candidate models using a Gaussian approximation that requires evaluating the (inverse) Hessian of the base objective function. More generally, the Hessian describes the curvature of the objective function, and is closely related to geometric notions of "flat regions" on the surface induced by the objective function (Goodfellow et al. 2014 Li et al. 2018
Thhee questions are inspired by the concise and lucid problem setting of Jia and Su 2020
*A lucid explanation is given by Devroye et al. 1996 Ch. 1).
'Even this refined problem is far from trivial; see Shalev-Shwartz, et al. 2010 and Feldman 2016).
1vve the critical question of which core optimizer to use does not have clear-cut answer on standard benchmark datasets (Schmidi et al. (00T)) With additional options such dropout, batch normalization, and all manners of data augmentation, it is not surprising that the practitioner often takes a trill-and-eroor approach.
"In sense this is just human nature, not specific to machine learning (Baker 2022
IIn the context of training machine learning models both big and small, "Goodhart's law" sug- some extent probably good thing http:s://opmas..corrrresaaa
gests that this gap Y2 Since the data is random, so is the Fisher information; some authors call this the observed Fisher information. in contrast with the expected Fisher information (Efron and Hinkley 1978).
15

---

The notion of model complexity has also played an central role in statistical learning theory. It has long been known that even when the number of parameters far outnumber the number of training samples, a small weight norm can be used to guarantee off-sample generalization for empirical risk minimizers (Bartleti 1998). Of course, due to the high expressive power of neural network models, even with strong weight regularization it is possible to perfectly fit random labels (Zhang et al., 2017), leading to a gap between the test error (chance level) and training error (zero) that is methodologically unsatisfactory. This has motivated variety of new approaches to measure off-sample generalization (Jiang et al. 2020), as well as to quantify model complexity, such as the degree to which a model can be meaningfully compressed (Arora et al. 2018
The notion of "flat minima" is seen in the early work of Hochreiter and Schmidhuber (1994) 1997).
which considers both how to measure sharpness, and heuristics for actually finding candidates in flat regions. The basic underlying notion that of measuring "volume," namely the idea that a "flat" point is one from which we need to go far in most (if not all) directions for the objective function to increase a certain fixed amount. See more recent work by Wu al. (007)) for related notions of volume in this context. These notions of sharpness are intimately related to properties of the Hessian matrix of the underlying objective function, even when the loss is not based on negative log-likelihood, and an active line of research is centered around the eigenvalue distribution of this Hessian. See for example Chaudhari et al. (0017) and Karakida et al. (019)) for representative work. For sufficiently "regular" models, the determinant of the Fisher information matrix plays central role in the complexity term used to implement the minimum description length (MDL) principle (Grünwald (007);; see also early work from Hinton and van Camp (1993) and more recent work by Jia and Su (2020) in the context of neural networks. More generally, however, many neural networks do not satisfy these regularity conditions, and new technical innovations based on the Fisher information have been explored to bridge this gap in recent years (Sun and Nielsen 2021).
A.3 Algorithms that generalize well
The empirical effectiveness of deep learning goes well beyond what we would expect based purely on learning theoretical insights (Sejnowskil 2020). This success is driven by a handful of workhorse stochastic gradient-bascd solvers (Schmidt et al. 2021), often coupled with explicit norm-based regularization and a number of techniques used to stabilize learning and effectively constrain the model candidate which is selected by the learning algorithm 14 A rich line of research has developed over the past decade looking at why a certain algorithmic "recipe" tends to generalize well. The tendency for stochastic gradient descent to "escape" from regions near undesirable critical points is one key theme; see Xie et al. (2020) for example. For influential work on relating sharpness, mini- batch size, and (weight) norms to off-sample generalization, see Keskar et al. (217)) and Neyshabur eet al.] (2117). In both papers, the notion of measuring sharpness by a worst-case perturbation appears, and this is pursued even further by Foret et al. (2021) in the well-known sharpness-aware minimization (SAM) algorithm, and extensions due to Kwon et al.] (202T) and /hhao et al. (2022). These algorithms, as well as the Fisher information a forward-difference implementation of explicit gradient regularization (using squared Euclidean norm), and recent work from Karakida et al. (2023) compares this approach with that of direct back-propagatic approach. [Brrett and Dherin (02T)) look at both implicit and explicit gradient regularization. The implicit side contrasts the path of "continuous" gradient- based updates with the "discrete" " updates made in practice (continuuus/discret with respect to time), saying that the discrete updates, even when computed based on an unregularized objective function, tend to move closer to the (continuous) path of a regularized objective, where regularization is in terms of the (squared) gradient norms. Inspired by this finding, they also consider explicit GR in the same way; see also Smith et al. (202T).
procedure of Jia and Su (2020), all involve
14These include early stopping. modifying mini-batch size, dropout, batch normalization, stochastic depth. data augmentation, and "mixup" (mixed sample augmentations) among others. See also Montavon et al. 2002 for techniques that were established in the decades before the current wave of deep learning.
16

---

B Technical appendix
B.1 More details on Flooding and sharpness
When the empirical risk goes below the threshold 6, the Flooding update 4 attempts to push it back up above 0. Consider the case in which this occurs in a single step, i.c. the situation in which at some step f, the pair of sequential iterates (we, Wx+1) satisfy the following:
R (wi) and R, (w++1)) 0.
(14)
When condition 14 holds, some basic algebra immediately shows us that running two iterations of yields the equality
the Flooding update
W2+2 We a² [Rnnwu + a (VRn VR, (wx)) (
(15)
a
telling that the result is equivalent running one iteration of FD descent with step size a on the GR penalty (-) at wt, using the forward FD approximation described earlier in $2.2 To the best of our knowledge, this link was first highlighted by Karakida al. (2023 $5.1). In sense, this is a natural complement to the strategy employed in (6); instead of tackling the GR objective R. (w; A) in (5) directly, the Flooding algorithm can iterate back and forth between optimizing the empirical risk and the squared gradient norm. The GR effect is thus constrained to regions in W with 0-small empirical risk, but all updates outside this region enjoy the same per-step computational complexity as vanilla GD.
B.2 Non-smooth loss setting
All of the analysis in $3.2 relies heavily on smoothness of the underlying loss function. Here we consider the case in which the loss itself may not even be differentiable. All we ask is that the losses be L-Lipschitz on W in expectation, i.e., E,, Z) L(w2; Z)I I L [y w2ll for all w1, W2 E W. As an explicit objective function, we start with S, as given (10), but with the understanding that P can actually be any 1-Lipschitz function, capturing the two special cases of interest, namely p(x: = V 1 1 for SoftAD and p(x) = |ar| for Flooding. Shifting our focus to function values (rather than gradients), we will also need to assume a second moment bound
Ep (0 + p(e(w;; Z) - 0))² < <:00
(16)
that holds over W. With these basic assumptions in place, stationarity guarantees are available via a smooth approximation of Sp.
Proposition (Stationarity, non-smooth case). Choosing an initial value w1 E W, run the algorithm described in Proposition 3 but re-defining the core gradients used for updating as
d G,(w) - - pll(w ( p0tlwwt++ru::3
U)
where (U1, U2....) is a sequence of independent vectors sampled uniformly at random from the unit sphere, and r > O sets the smoothing radius. In addition, the norm threshold is set as = / /dL/((1 - b)r) (with b unchanged). Stationarity of the resulting sequence (W1, W2,...), assumed be in W, can be controlled with probability 6 as
1 1 (Iv,,((rr)) TIAI (R,((u)) (,T+1)) +
3dL 2d 2r VV
T
where (wrrr (w rV)] is the r-smoothed approximation of the objective Su with V distributed uniformly over the unit ball. Probability is taken over the random draw of (Zı,Za,... and (U1, .2.. and the confidence factor Cs matches that given in Proposition 3
Remark (Stationarity in the original objective). When the original objective Sy) is sufficiently well- behaved, e.g., differentiable almost everywhere and Lipschitz, then stationarity guarantees in terms of the r-smoothed objective r)):)) can be easily translated into analogous guarantees for (-). In addition, recent work by Cutkosky et al. (2023) shows how a modified algorithmic approach can be used to achieve faster rates under such congenial (but still non-convex and non-smooth) conditions.
17

---

B.3 Additional proofs
Proof of Proposition 3 The machinery of Cutkosky and Mehta (202T) Thm. 2) gives us the ability to control the stationarity of sequences generated using the described procedure (norm- clomm-ciipping.... momentum, normalization), just assuming the "raw" stochastic gradients (here, G2) are unbiased estimators of a smooth function. As such, we just need to ensure the assumptions underlying their Theorem (henceforth, CHT2) are met; the key points have already been described in the main text, so we just fill in the details here. The "unbiased estimator" property we refer to means that we want
Ep G:(w) = (w,
(17)
to hold for all w E W. Fortunately, this holds under very weak assumptions; the running assumption that LAD < oo is more than sufficient B3 smooth in the sense that
In addition, finite LAD also implies that the objective (10 is
VS, (w2)l) S LAD||R w2ll
(18)
for any W1, Wy W; this is proved in SB.5 In addition, uniform second moment bounds naturally imply pointwise bounds, so we have
Ve 2)1l & E,
sup VV((;; Z)1?² S LAD - Le
(19)
LwW
for each w E W. Taken with the construction of sequence (W1, W...... in the hypothesis, the properties (17) (9) ensure all the basic requirements of CHT2 are met (with their "p" at 2). Noting that we assume W C Rd using the standard norm and inner product on Euclidean space, the Banach space generality in CHT2 is not needed (their "C" and "p" can be fixed and 2 respectively). For reference, the complete upper bound implied by CHT2 is
Sy (w1) - (WT++)
aLLD + 2
2b LAD - Le 20oLAD + (1-b) - Le)Cs (20)
Ta
(1 - bjT
b)
where for readability the coefficient in the right-most summand is defined by
C = /8) + 4v/log3T/88) + 1.
We have simplified all the terms in CHT2 involving max (1, loT (3T) since as long T 0 and 0 < 5 <11, 1, we trivially have 37/e 2 1 >> 5 and thus log(3T/6 (5) 2 1. Furthermore, their free parameters "b" (different from our b) and "s" are both set to 1, without loss of generality. Plugging in
our settings of and to the bound in (0) and bounding (1 - 1/vT) < for readability yields the desired upper bound.
Proof of Proposition Here we leverage the projected sub- gub-gradient analysis done by Davis and Drusvyatskiy (2019), in particular their Theorem 3 (henceforth, DDT3). The core of their argument relies upon weak convexity property held by rather large class of composite functions, namely compositions of the form f = h g. where is smooth and h is both convex and Lipschitz. Considering the non- non-smooth objective function
(21) it can be taken as a compound function by writing E,, f(w; Z) with f(w; 2) = (w; 2)), where g(w; 2) = f(w; 2) and h(z) 0 + I 0. Fixing 2 E Z for now, clearly h is 1-Lipschitz and convex. By assumption, we have that g(::2) is -moohh and locally Lipschitz. Then, using standard arguments, it is straightforward to show that f(::2) is Li-weakly convex T6 Since the weak convexity parameter Li does not depend the arbitrary choice of 2, it follows that the function (2 is Li-weakly convex. In the setting of DDT3, their "f(-)" is + (,.: Z) 0). and their "X" is W here. The bound on the expected squared stochastic gradient norms (their "[²") our LAD L, just as in the proof of Proposition 3 Finally, the key "unbiased estimator" property in this case deals with sub-differentials, namely we require that
w --+ 0 + E,, ((:: Z) 0),
E, (w) Ô E,,l((w:; Z) 01
(22)
15 For a more general result, see Holland 2023
Lem. 2) for example.
16 See for example Drusvyatskiy and Paquette 2019 Lem. 4.2) and Holland 2022
Prop. 8).
18

---

for all w E W. Fortunately this basic property holds under very weak assumptions that are trivially satisfied when LAD is finite 17 With these facts in place, we simple apply DDT3, in particular their inequality (3.5), with their "p" corresponding to our 1. here, and their "<px" corresponding to our (1)). with "X" as our B. The desired bound follows by applying their result to the specified procedure over T - updates (instead of their T updates).
L
To begin, under the assumptions given, the objective Sy clearly inherits the Lipschitz property that the losses have in expectation; since is 1-Lipschitz, we have
Proof of Proposition 6
(w1) S (w2) _ Epl(((1:2
f(w2; Z) - L|[ w2ll
(23)
We proceed by using standard technique for function smoothing If we let V be random vector distributed over the unit ball {i E R |||||| E 1}, then regardless of whether S.,.) is differentiable or not, one can obtain a smooth approximation by averaging over random r-length perturbations, namely
18
(, = E [S, + rV)].
(24)
critical property of the function given in 24 is that differentiable and its gradient can be represented explicitly in terms of the function it trying smooth, namely have
d r
VS (w: r)
(w
rUJU]
[(0 p(e(w rU) U]
(25)
for any choice > 0 and w W. where uniformly distributed on the unit sphere {ar E Rd
||a||| = 11 (Flaxman et al. 2004 Lem. 1) 19 This means that Lipschitz properties on the original function translate smoothness properties for the new function. Making this more explicit, using the equality 25 note that for any choice of W1, w2 E W, we have
(wi;))) (S,(wr))) E (wn + + Sy(w2 -U))
Taking norms and using the Lipschitz property (3 of S,,, we observe that
VSJ(wur) (S,
E||U||S
rU) Sn + rU)
r
dT.
w wz2
and thus have that the smoothed function S. (3r) is (dL -msooth over W. This means the function is analogous to the objective function (10) used in Proposition 3 gradients taking the form
except with unbiased stochastic
G. Gw)w = d (0 ( p(((w + rUU; - 8)) U,
(26)
for 2 where each U, independent copy of U from 25 From this point, the remainder of the
proof is basically identical to that of Proposition 3 the only remaining changes are the smoothness factor and the second moment bound. For the former, impacts the norm clipping radius y. For the latter, since we are assuming w, + rU, E W for each t, and using the bound (16), we have
use dL./r in place of LAD. which also
(
2
EIJU,|I p(e(w;; - < (9) 2
B|]G((w)]l
< sup
wEW
Plugging in these two remaining modified factors to the bounds obtained in Proposition 3 yyields the desired result.
See for example Holland 22022
Prop. 14).
18 See for example Flaxman et al. 2004 Nesterov and Spokoiny (011)).
Not to be confused with V in 24). which is uniform the unit ball.
19

---

B.4 Gradient of GR objective
With w (Wi..... wa) R we will frequently use dz to denote partial derivatives taken with respect wy, i.e., for differentiable function f I2 R, we write
(5 = lim J(wI,..., Wj + a, a...., wa) - f(w)
(27)
|a|-
With the above notation in place, note that basic with analogous definitions for all j I.....d. calculus gives us
i VR. (w) Σ 02 (O, = Σ (O)Rn(w)) (0,0&,,u)))
Am
As such, the gradient takes the form
(w))l 2 (V (VR,(W))
where RRR, denotes the d x d Hessian matrix of R,,, and (w) is taken column vector (a
x matrix) for the purpose this multiplication.
B.5
Smoothness check
Let us consider a simple, non-stochastic example one dimension. Letting / differentiable function, consider how the transformed gradient ((f(x)
be some 0) fa behaves under sign and (xx) x/V2² + As we have seen visually Figure | the soft threshold of SoftAD makes possible to have Lipschitz gradients, which impacts iterative optimization procedures. For arbitrary values 2 and z2, taking the difference of transformed gradients using arbitrary ø. we can write
((f(x1) - (a(n))-0((()))(((())0000-)))))0)
= Note that even if [2] 22 for some arbitrarily small æ > 0, if for example the threshold is such that f(x)) < 0 < f(22). then under Φ = sign, the difference multiplying f' f'(z1) cannot be arbitrarily small, even if Lipschitz. On the other hand, such a property follows easily when p', since o' itself is 1-Lipschitz.
(28) f'(11) + o(f(22) - -((
Returning to more general learning setup, let denote the modified gradients concisely as g(w; 2) = ((((w;)))=))))) =)) With random variable Z ~ μa, taking any two points w1, W2 E W, based on the equality (2)). the normed difference of the gradient expectations can be bounded
E g(1; Z) E, ww Z) s B1 + B2
with Bj and B2 defined as
B E V (w w1; ( Z) 0) ((l(w'z Z) 0)
Bg = E,, (((({(); - V[ vé(w>; Z)||-
Bounding each of these terms is trivial when the functions and á are smooth enough. First, note that if Φ La- -ipichhctt and is a convex subset of R., we have
B1 $ Ls E, V Z) [[ Z) E(w2; Z)
< La || w2l / Ve Z)ll sup VVlaa + a)w2; Z)
0<ca<1
< Lo | w2
E
sup [[: Z)[l2
LwE
noting that the second inequality uses the mean value theorem differentiable ((::2), applied pointwise in z E Z, and the last inequality uses convexity W. This bounds B,. Moving on to B2, note that if [o(xr) is bounded by B& and the losses Le-smooth in expectation, we have
Bg < Bo V Z) Viww>; Z)
< BoLellwi W2ll.
Taking these new bounds together, have
E,, g(w:; Z) E,, (wzz Z) < Lo E,
sup V[(w; 2)12 LwW
+ Bobe |mll - w2ll.
namely Lipschitz property in expectation for the modified gradients.
20

---

Figure 7: Synthetic dataset examples. From left to right: "two Gaussians," "sinusoid," and "spiral."
C Empirical appendix
Here provide additional details and results related to the empirical tests described in $
Software and hardware All of the experiments done in this section have been implemented using PyTorch 2, utilizing three machines each using single- GPU implementation, i.c., there is no parallelization across multiple machines or GPUs. Two units are equipped with an NVIDIA A100 (80GB), and the remaining machine uses NVIDIA RTX 6000 Ada. We use the MLLfow library for storing and retrieving metrics and experiment details. Our coding of SAM follows that of David Samuel (https:/ h/gps::///ttubb com/ davda54/sam which is the PyTorrc implementation acknowledged in the original SAM paper of Foret et al. (2021
C.1 Non-linear binary classification on the plane
Data The three types of syntheric data that we generate here differ chiefly in the degree of non- linearity; see Figure 7 for an example. The "two Gaussians" dataset is almost linearly separable, save for some overlap of the two distributions. The "sinusoid" data is separated by simple curve, casily approximated by a low-order polynomial, but the curves the "spiral" data are a bit more complicated. Exact implementation details, plus historical references, are given by Ishida et al. (2020) $4.1). For each trial, we generate training and validation data of size 100, and test data size 20000. All methods see the same data in each trial.
Model For each dataset, we use the same model, namely a simple feedforward neural network with four hidden layers, 500 units per layer, using batch normalization and Rel.U activations at each layer
Algorithms In line with the experiments we are trying to replicate, all methods (ERM, Flooding, SAM, and SoftAD) are driven by the Adam optimizer, using fixed learning rate of 0.001, with no momentum or weight decay. All methods use the multi-class logistic loss as their base loss (i.c., nn. CrossEntropyLoss
PyTorch), and are run for 500 epochs. We use mini-batch size of 50 here, but key trends remain the same for full-batch (of size 100) runs.
Hyperparameter selection ERM has no hyperparameters, but all the other methods have one each. Flooding and SoftAD both have the threshold parameter 0 seen in $21-3 and SAM has radius parameter (denoted "p" in the original paper). For each of these methods, in each trial, we select from a grid of 40 points spaced linearly between 0.01 and 2.0. Selection is based on classification accuracy on validation data.
C.2 Image classification from scratch
Our second set of experiments utilizes four classification. Compared to the synthetic experiments done in SC.T the classification task is more difficult (much larger inputs, variation within classes, more classes), and so we utilize more sophis- ticated neural network models to tackle the classification task. That said, as the sub-section title indicates, this training is done "from scratch," i.e., no pre-trained models are used.
benchmark datasets for multi-class image
Ishida et al. 2020) say they use "five- heiee-hidde--laye layer feedforward neural network." but looking at their public code, the number of hidden layers (i.e., number of linear transformations excluding that of the output layer) is actually four.
21

---

Data The datasets we use are all standard benchmarks in the machine learning community: CIFAR- 10, CIFAR-I00, FashionMNIST, and SVHN. All of these datasets are collected using classes defined in the torchvision datasets module, with raw training/test splits left as-is with default settings. As such, across all trials the test set is constant, but in each trial we randomly select 80% of the raw training data to be used for actual training, with the remaining 20% used for validation. We normalize all pixel values in the image data to the unit interval [0, 1]: this is done separately for training, validation, and testing data.
Models Unlike the previous sub-section, here we use different models for different data sets. Model choice essentially mirrors that of Ishida et al. (2020] $4.2). For FashionMNIST,, we flatten each image into a vector, and use simple feedforward neural network composed of single hidden layer with 1000 units, batch normalization, and ReLU activation before the output transformation. For SVHN, we use ResNet-18 as implemented in torchvision midio..mmms.,......mooooo models, without any pre-trained weights. Finally, for both CIFAR-10 10 and CIFAR-100, use ResNet-34 (again in torchvision models) without pre-training. Both of the ResNet models used do not flatten the images, but rather take each RGB image as-is.
Algorithms Just as in cross entropy loss, and run for 500 epochs. However, instead of Adam the base optimizer, here use vanilla SGD with a fixed step size of 0.1, and momentum parameter 0.9. For all datasets, we use mini-batch size of 200. All these settings match the experimental setup of Ishida al. (2020) $4.2) 21
we are testing ERM, Flooding, SAM, and SoftAD. Again we use the
Hyperparameter selection Once again select hyperparameters for Flooding, SoftAD, and SAM from a grid of candidate values, such that the classification accuracy on validation data is maximized. Unlike jhweeer here we use different grids for each method. For Flooding, we follow the setup of the original paper, choosing from ten values: {0.01, 0.02, 0.1}. For SAM, once again we follow the original paper (their $3.1), which for analogous tests utilized the {0.01, 0.02, 0.05, 0.1, 0.2, 0.5). Finally, for SoftAD we match the set size used by Flooding (i.c., ten) by taking the union of (0.15, 0.25, 0.35, 0.75) and the set used by SAM.
21 Batch size is not given in the original Flooding paper, but the size of 200 was confirmed by means of private communication with the authors.
22

---

Comparison with iFlood As mentioned in Remark [ during the review phase of this work, the iFlood method of et (2022) was brought to our attention, and have rum additional tests analogous to those described in the preceding paragraphs, but this time comparing ERM, iFlood, and TO and Table in that order.
SoftAD. The results are given below Table 3 Figures
CIFAR-10
CIFAR-100
Fashion
SVHN
ERM
3.252
7.695
0.806
0.726
iFlood
1.468
2.558
0.273
0.404
SoftAD
1.072
2.696
0.463
0.397
Table Generalization gap (test training) for trial averaged cross entropy after final epoch.
Loss (cifar10)
Loss (cifar100)
Accuracy (cifar10)
Accuracy (cifar100)
0.3
ERM
0.2
ERM
iFlood
SoftAD
SoftAD
0.0
100
200
300
400
500
100
200
300
400
Figure Trajectories over epochs for average test loss (top row) and test accuracy (hottom row). Horizontal axis is epoch number. Columns are associated with the CIFAR-10 and CIFAR-100 datasets (left to right).
Loss (fashionmnist)
Loss (svhn)
2.0
0.5
0.0
Accuracy (fashionmnist)
Accuracy (svhn)
0.90
0.8
0.6
0.85
ERM
0.4
ERM
0.80
iFlood
iFlood
SoftAD
0.2
SoftAD
0.75
0
100
200
300
400
500
100
200
300
400
500
Figure 9: Analogous to Figure
8 hut with FashionMNIST and SVHN datasets.
23

---

Model norm (cifar10)
Model norm (cifar100)
200
180
200
160
150
100
500
300
400
500
Model norm (fashionmnist)
Model norm (svhn)
250
60
150
40
200
500
400
Figure Model norm trajectories over epochs each dataset Figures
CIFAR-10 10
CIFAR-100
Fashion
SVHN
iFood
0.04 (0.05)
0.06 (0.04)
0.15 0.
0.10 (0:05)
SoftAD
0.12 (0.06)
0.34 (0:28)
0.01
0.12 (0.12)
Table 4: Hyperparameters selected by validation for each method (averaged trials). Flooding and SoftAD
have threshold 0; SAM has radius parameter. Standard deviation (over trials) given in small-text parentheses.
24

---

C.3 Linear binary classification
In Figure II we compare ERM with Flood and SoftAD run with a common threshold level of 0 = 0.25, using the "two Gaussians" and "sinusoid" data described in $ and simple linear model, i.e., a feed-forward neural network with no hidden layers. Training and test sizes match those described in $ Even with very simple linear model, it clear that SoftAD can be used to achieve competitive accuracy much larger loss levels. Note that the case of "sinusoid," the average loss does not reach the threshold 0, and thus Flooding is identical to ERM. These basic trends hold over range of thresholds and re-scaling parameters a (i.e., using <((x 0)) with a o 1). These trends are captured by the heatmaps given in Figure 12 where for each setting of 0 (for SoftAD and Flooding) and a (for SoftAD only), we generate a fresh dataset. Clearly taking the threshold level far too high leads to arbitrarily bad performance, but below certain level, similar performance is observed over a wide range of values. function of 6, the test accuracy drops in much sharper manner
interesting to note how while test loss changes rather
predictable continuous fashion when 0 is set too high in the case of SoftAD, whereas this drop is smoother in the case of Flooding. That said, these trends only within the confines of this very simple linear model example using full batch, and tend to change (even with the same model) as we modify the mini-batch size.
Linear binary classification (gaussian, trial 0)
Average CE loss
Accuracy
1.0
0.6
0.9
0.4
0.8
0.3
ERM
0.7
Flood
Soft-AD
100
200
150
Linear binary classification (sinusoid, trial 0)
Average CE loss
Accuracy
0.95
0.6
0.90
0.5
0.85
0.4
0.80
ERM
0.3
Flood
0.75
Ao
50
100
150
200
50
100
150
200
Figure 11: Average entropy loss and accuracy epochs (full batch) for each method.
25

---

gaussian(test loss)
gaussianttest accuracy)
03
03
0.5-
0.5-
04
04
86
144
-0.
-0.
igma
sinusoid(test loss]
sinusoid/test accuracy)
SeRAa
fonding
30
1.3
1.0
033
0.3
0.6
0.5
16
-32
-02
4.1
0.3
0003 6375
L12 2008 2d0
2300
5435 3.20 6375
L.00
2353
2205
Figure 12: Test loss and accuracy heatmaps for Flooding and SoftAD, depending on threshold level (denoted "theta") and scaling parameter (denoted "sigma").
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We are very clear that if our interest is purely in improving the test accuracy, the existing Flooding algorithm tends to work quite well. Our contributions are set in the context of learning tasks where accuracy is important, but average loss and/or model norms are also of importance; our new method SoftAD is shown to achieve an appealing balance between these metrics compared with Flooding (and ERM/SAM)), as claim. Furthermore, we make no claims that our method can circumvent the issue of how to set the threshold level 0; we discuss in | this is a broad question that underlies both Flooding and SoftAD, and the pursuit of an answer is stated as future work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
As discussed above under "Claims," " we emphasize the strong points of our method (smoothness, stability, good balance across loss/accuracy/nomms) but are completely open about the fact that our method, just as with Flooding, still has free parameter 0 whose setting by no means trivial (though empirically, simple validation is shown work well).
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
27

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All assumptions are stated clearly, either in the main text or in the body of propositions/theorrmss, and complete proofs are given in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All experimental results can be recovered using code and Jupyter notebooks that we have already prepared a GitHub repository.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
28

---

(c) If the contribution is a new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: See the above answer; code is provided on GitHub (after the review phase). As for data, all datasets are public, clearly described, and accessible via the outlets we have mentioned.
Guidelines:
The answer NA means that paper does not include experiments requiring code. Please see the NeurlPS code and data submission guidelines (tttps://niis.cco)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https (Coooooodimsssiooooo
/ //nips.ccs wttpccccpppppuppi.eeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In the main body of the paper we give a broad summary, but complete details are given in $C a fact that clearly mention in the main body.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
29

---

Answer: [Yes]
Justification: For results with relatively large discrepancy between methods (namely hyper- parameter selection results in Table 2). give standard deviation (over randomized trials) in addition to averages.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: At the start of $ we describe the hardware used in experimental setup. Actual computation time is not key factor in our results.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
30

---

10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out, For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Datasets and software are all credited, both within the paper itself and within our code made available GitHub.
31

---

Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The only relevant new "assets" are code for re-creating the experiments, and this all well-documented on our GitHub repository.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
32

---

Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
33


Reviews:
[{"review_id": "4bp1CUpAqb", "reviewer": "Reviewer_55bG", "summary": "The paper proposes a new optimization technique, Soft Ascent-Descent (SoftAD), aimed at enhancing the stability and performance of machine learning models. The method balances loss minimization and model complexity, positioning itself as an improvement over existing techniques like Flooding and SAM. The paper provides a theoretical foundation, empirical studies across multiple datasets, and a comparison of convergence properties with existing methods, claiming superior stability and often better test loss outcomes.", "questions": "1.\tHow do you suggest optimizing the threshold parameter θ in practice? Are there any heuristics or automated methods that can be employed?\n\n2.\tHow does SoftAD perform on larger datasets and models, such as those used in natural language processing or large-scale image recognition tasks?\n\n3.\tBesides θ, are there other critical hyperparameters that significantly affect the performance of SoftAD? How sensitive is the method to these parameters?\n\n4.    Can you provide examples or case studies where SoftAD has been successfully applied in real-world scenarios?", "limitations": "Refer to Weaknesses", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper discusses SoftAD designed to balance minimizing loss and controlling model complexity. This method is compared to existing optimization techniques like Flooding and SAM and is shown to provide better stability and often superior test loss outcomes. The theoretical foundation of SoftAD is robust, with clear mathematical formulations and comparisons to existing methods. Empirical validation includes extensive experiments on multiple datasets, such as CIFAR-10, CIFAR-100, FashionMNIST, and SVHN, demonstrating the effectiveness of SoftAD in practical applications. The paper offers a comprehensive analysis of model norms and loss trajectories, underscoring the practical benefits of the method. Additionally, the document is well-structured, featuring clear explanations, figures, and tables that effectively support its claims and contributions.", "weaknesses": "1.\tThe SoftAD method may be more complex to implement compared to traditional methods, which could limit its adoption. Implementing SoftAD involves managing both ascent and descent phases, which adds to the complexity. This requires additional code for conditionally switching between ascent and descent based on the loss value and the threshold θ.\nThe complexity is implied by the need to incorporate equations (3) and (4) for the ascent and descent phases into the optimization loop.\n\n2.\tThe method's performance depends on the setting of the threshold parameter θ, which can be non-trivial to optimize. The threshold θ determines when the algorithm switches between ascent and descent phases, directly impacting the performance. Choosing an inappropriate θ can lead to suboptimal results.\n\n3.\tWhile the experiments are thorough, additional datasets and real-world applications could further validate the robustness of SoftAD. The experiments are conducted on CIFAR-10, CIFAR-100, FashionMNIST, and SVHN. While these datasets are standard benchmarks, they do not cover the full range of possible applications, especially large-scale or domain-specific datasets.\nThe scope of the experiments is detailed in Section 4, where the datasets used are listed.\n\n4.\tThe paper does not provide a detailed analysis of how sensitive the performance is to hyperparameters other than θ. Changes in other hyperparameters can significantly affect the performance of optimization algorithms. The focus on θ without detailed analysis of other hyperparameters is evident in Section 5, where the experiments primarily vary θ.\n5.\tThe paper acknowledges the difficulty in setting the threshold θ but does not provide a concrete solution for it, which might be a crucial limitation for practical implementation.\t\t\n6.\tWhile SoftAD shows better performance in terms of test loss and model norms, it is not always superior in terms of test accuracy when compared to SAM, as noted in the empirical results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time in carrying out this review. Below, we respond to the questions raised.\n\n---\n\n> 1. How do you suggest optimizing the threshold parameter $\\\\theta$ in practice? Are there any heuristics or automated methods that can be employed?\n\nThe ideal setting in practice will inevitably depend on how we want to measure \"good performance,\" but if good performance means a small average value of both surrogate loss and 0-1 loss, then much like the perturbation radius for SAM and the analogous threshold $\\\\theta$ in Flooding, a small value (usually between 0.01 and 0.25) tends to be a safe bet. In our tests, we used validation accuracy for $\\\\theta$ selection, and we find that all else equal, more complex tasks (more classes, more extraneous features) tend to lead to a larger value of $\\\\theta$ being chosen, e.g., larger values for CIFAR-100 and SVHN compared with CIFAR-10 and FashionMNIST.\n\nIn terms of possible directions for setting the threshold $\\\\theta$ in a wide variety of learning settings, one natural approach is to simply run vanilla ERM once, and set $\\\\theta$ to the mean + standard deviation of the training losses incurred by ERM, noting that even with $\\\\theta = 0$ (or whatever the minimum possible loss value is), our SoftAD does not collapse into ERM. A modified version of this approach for binary classification, which is slightly more theoretically motivated, is to [incorporate an estimate of the Bayes error](https://openreview.net/forum?id=FZdJQgy05rz), which can then be leveraged to select an early stopping time for the ERM-based approach just mentioned. This Bayes error estimation work comes from the same group that originally proposed Flooding, but it appears they have not integrated the two methods.\n\n> 2. How does SoftAD perform on larger datasets and models, such as those used in natural language processing or large-scale image recognition tasks?\n\nWe have not tested SoftAD on NLP datasets yet, but we have run analogous tests using ResNet classifiers trained on a variety of subsets of ImageNet, and the trends observed were essentially the same as those observed in the current manuscript. We can definitely add these results as supplementary material to the current manuscript.\n\n> 3. Besides $\\\\theta$, are there other critical hyperparameters that significantly affect the performance of SoftAD?\n\nThe only hyperparameter inherent to SoftAD as formulated in this paper is $\\\\theta$. Changes to learning rate, weight decay, and momentum hyperparameters of the SGD optimizer used naturally can impact performance, but in our exploratory analysis the change looks to be essentially uniform across the different methods considered.\n\n> 4. Can you provide examples or case studies where SoftAD has been successfully applied in real-world scenarios?\n\nOne recent application that we have been directly involved in is to cybersecurity datasets (e.g., [CIC-MalMem-2022](https://www.unb.ca/cic/datasets/malmem-2022.html)) with a distinct imbalance between benign examples and attack examples, and a lot of diversity within certain types of attacks. We applied SoftAD in combination with distributionally robust optimization (DRO; to put more weight on the tails) to achieve better balanced performance across imbalanced data types, without knowing which example falls into which type of attack, a cost-efficient and type-agnostic alternative to methods like [SharpDRO](https://github.com/zhuohuangai/SharpDRO/tree/main), which we found to work quite effectively (results have not yet been published, in review currently). While specific applications are not the focus of this paper, we can definitely add a remark highlighting the potential practical applications based on our experience thus far.\n\n---\n\n__Comment regarding code complexity:__\n\nPlease let us make a quick remark regarding the following comment raised by the reviewer in the \"weaknesses\" field.\n\n> Implementing SoftAD involves managing both ascent and descent phases, which adds to the complexity. This requires additional code for conditionally switching between ascent and descent based on the loss value and the threshold $\\\\theta$.\n\nWhile admittedly this may not be immediate from the equations in our paper, in fact the implementation of SoftAD is essentially no more complicated than vanilla ERM implemented with SGD. All we do is pass the losses through a wrapper function, which is differentiable and works smoothly with autograd. The switch between ascent and descent is automatic. For anonymity purposes, we have kept our GitHub repository of code anonymous, but here is a simple example of how it would work for training using PyTorch (the `criterion` here corresponds to equation (7) in the paper):\n\n```\noutputs = model(inputs)\nlosses = loss_function(outputs, labels)\ncriterion = theta + rho(losses-theta).mean()\noptimizer.zero_grad()\ncriterion.backward()\noptimizer.step()\n```\n\nApologies for the long response, but we feel the simplicity of implementation is an important point we hope the reviewer can appreciate. At the very least, the implementation is definitely a lot simpler than SAM, and uses half the number of gradients per step. We can definitely add a remark in the paper to ensure this is crystal-clear to readers. If you have any additional questions that would help you raise your score or confidence, please do not hesitate to ask.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yeztJ3717f", "reviewer": "Reviewer_JmzP", "summary": "This paper proposes a method called SoftAD (soft ascent-descent) which aims to improve the \"flooding\" method. Specifically, it downweights points on the borderline, limits the effects of outliers, and retains the ascent-descent effect of flooding, with no additional computational overhead. Stationarity guarantees of these two methods are provided. Empirical results illustrate that SoftAD can have comparable classification accuracy with flooding while enjoying a smaller loss generalization gap and model norm.", "questions": "Please give the situations where we are interested in other metrics such as model complexity or average surrogate loss at test time.", "limitations": "Please see the Weaknesses part.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Overall, this paper is well written.\n2. The experimental results demonstrate the merit of the proposed method SoftAD over two baselines (i.e., flooding and SAM). Specifically, SoftAD has competitive classification accuracy with these two methods while enjoying a smaller loss generalization gap and model norm.", "weaknesses": "1. The motivation might be confusing. As the authors claimed, there are situations where we are interested in other metrics such as model complexity or average surrogate loss at test time. I can not imagine these situations except the generalization analyses of classification accuracy might involve these factors.\n\n2. The proposed method is heuristic which lacks theoretical support. More specifically, generalization analyses are missing to support the claims.\n\n3. Although it is good to give the optimization convergence guarantees of SoftAD and flooding, they can be not directly comparable because one is high probability bound and another is expectation bound.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for their time in reading the paper and writing the review. Below, we respond to the points raised.\n\n> Please give the situations where we are interested in other metrics such as model complexity or average surrogate loss at test time.\n\nThis is an important point, thanks. While there are numerous possibilities, here are a few that have arisen in our own work.\n\n- Training calibrated classifiers (e.g., [Zhao et al. 2021](https://proceedings.neurips.cc/paper/2021/hash/bbc92a647199b832ec90d7cf57074e9e-Abstract.html), [Berta et al. 2024](https://proceedings.mlr.press/v238/berta24a.html)). Using SoftAD to make a small sacrifice to performance at training time without the choppy flooding updates can be quite effective in terms of realizing a good tradeoff between accuracy and calibration at test time, by ensuring the test-time average surrogate loss (cross-entropy, typically) is at a desirable level. The basic principles underlying this point are another big topic; we are working on a separate paper in this direction.\n\n- Learning under fairness constraints, where fairness is measured in terms of dispersion/variance in performance between sensitive sub-populations (e.g., [Hashimoto et al. 2018](https://proceedings.mlr.press/v80/hashimoto18a.html), [Williamson and Menon 2019](https://proceedings.mlr.press/v97/williamson19a.html)). One standard approach is to introduce convex approximations which put more weight on worst-case examples and can approximate the mean + standard deviation of the surrogate losses, such as conditional value-at-risk (CVaR) or distributionally robust optimization (DRO). These methods, however, are extremely sensitive to outliers, and by design discard a lot of data in order to emphasize the worst-case examples. In contrast, SoftAD doesn't discard any data, but instead by design it encourages the base losses to be well-concentrated around $\\\\theta$, which has a direct impact in terms of fairness-as-dispersion. This is also a big topic on its own, which we are currently working on.\n\n\n> Although it is good to give the optimization convergence guarantees of SoftAD and flooding, they can be not directly comparable because one is high probability bound and another is expectation bound.\n\nOur intention with these two results was to emphasize how with all else equal, using standard analytical techniques, SoftAD can readily be given high-probability guarantees on the test-time objective function of interest, whereas flooding only gets in-expectation guarantees on an approximation of that objective function. In-expectation guarantees can be easily converted into high-probability guarantees using Markov's inequality, but they will be exponentially worse than those for SoftAD in terms of dependence on the confidence level; we can definitely make this explicit in the paper by adding a remark with discussion.\n\n\n> generalization analyses are missing to support the claims.\n\nPlease let us emphasize that the \"convergence properties\" we prove in section 3.2 are not simply convergence guarantees in terms of the objective used at training time, but rather are results given in terms of the population objective (i.e., the objective at test time). In Proposition 2, stationarity is given in terms of the SoftAD risk function at test time defined in equation (10). These are guarantees in terms of the test-time objective for an algorithm run using training-time data, so we have intended them to be formal insights in terms of off-sample generalization. If the reviewer has any additional questions on this point that would help them raise their score or confidence level, we are happy to respond; this is an important point characterizing our formal results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bnelj8BeQd", "reviewer": "Reviewer_yxTo", "summary": "The flooding method sets a threshold for the average surrogate loss within a mini-batch during training: if above it do gradient descent as usual but if below it switch to gradient ascent. This paper updates the flooding method in 2 ways: 1) it invert the order of the sign function and the aggregation making it a pointwise method rather than an average method and 2) it replaces the sign function with a continuous, bounded, monotonic function. It provides numerical results to demonstrate the behavior of the new proposed method in Section 3.1 and show theoretical results to compare the convergence properties of the new method with flooding in Section 3.2. Finally, experiments show that the proposed method have similar accuracy compared with flooding/SAM while having much lower generalization gap and smaller model norms (despite not using any explicit shrinkage methods).", "questions": "(I wrote my main suggestions in the \"Weaknesses section\". Here I provide some other comments/questions.)\n\n- Having a low generalization gap and better surrogate loss seems to be one strong benefit of the approach. I wonder if the paper can check calibration (such as ECE) which is often correlated with lower test losses.\n- It is interesting that softAD allows a higher theta threshold. Is it possible to provide some discussions on why the proposed method allows a higher theta?", "limitations": "I believe there is no immediate negative societal impact of this work. The paper discuss that hyper-parameter selection remains and provide future directions to remove this aspect.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The soft truncation is a nice idea to improve the flooding method. The behavior shown in Fig. 1 & 2 is intuitive.\n- Stationarity guarantees are provided in the theoretical studies and the intuition is provided in Remark 4.\n- Experiments show softAD has the same level of test accuracy (SAM may be slightly better but at the double the gradient cost) while achieving better level of surrogate test loss. Furthermore, the generalization gap is much smaller and model norms are usually smaller than other baselines. The flood level comparisons are interesting: the softAD allows for a larger flood level.\n- The discussions about links to sharpness and OCE-like criteria are interesting and informative.\n- The story, organization, and presentation is clear and easy to read. There are a lot of informative information in the appendix as well.", "weaknesses": "- A similar pointwise idea was proposed in \"iFlood: A Stable and Effective Regularizer\" (ICLR 2022). It would be helpful to have some discussions about iFlood and explain if there are any differences in the pointwise idea.\n- The experiments are informative, clearly showing the benefits of the approach. However, given that the pointwise idea is already proposed, as an ablation study, it would be important to further check which parts are more effective: the pointwise component or the soft truncation component. If the paper can show that the soft truncation is important, that would make the paper's contribution clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time and constructive feedback provided in the review. Below, we respond to the points raised.\n\n__Regarding iFlood (ICLR 2022):__\n\nThank you very much for this reference. We were completely unaware of this work, but it is a very nice paper, and as the reviewer points out, it is essentially a midpoint between our proposed SoftAD and the original flooding procedure, using pointwise aggregation, but without the smoothing effect that aids in truncating outliers and down-weighting borderline points.\n\nWe commit to adding this reference, with discussion, to our paper, and we also commit to adding tests in our setup using iFlood to investigate the effective impact of smoothing on the performance metrics studied in our experiments. The experiments will not conclude by August 6th, but we fully commit to adding such results for a camera-ready version.\n\n---\n\n__Other comments:__\n\n> Having a low generalization gap and better surrogate loss seems to be one strong benefit of the approach. I wonder if the paper can check calibration (such as ECE) which is often correlated with lower test losses.\n\nThanks for this very natural question, this is a point we are indeed quite interested in, as mentioned above in the response to Reviewer JmzP. We do have some nascent test results in this direction already, and while an in-depth study is going to be the topic of a future paper, we can certainly add some early results to this paper to complement the insights obtained in terms of test loss performance.\n\n> It is interesting that softAD allows a higher theta threshold. Is it possible to provide some discussions on why the proposed method allows a higher theta?\n\nYes, we can definitely discuss this in the paper in more detail, plus provide some supplementary figures. In essentially all cases, when we select $\\\\theta$ based on validation set accuracy, we find that SoftAD ends up with a larger value of $\\\\theta$ than Flooding. Intuitively, we interpret this as resulting from the fact that SoftAD is qualitatively asking for something quite different from Flooding, namely asking for \"good concentration around $\\\\theta$\", instead of just \"average loss value close to $\\\\theta$.\" After a sufficient amount of training, losses often become quite asymmetrically distributed, and it can be infeasible to achieve high concentration at a threshold $\\\\theta$ too close to the minimal possible value. This property is most directly due to the point-wise form of the SoftAD objective, and we believe the iFlood method would also show similar properties when compared with Flooding in our experimental setup; we will add such results.\n\nThanks again for the constructive feedback and overall positive review. If there are any additional points you would like to discuss, please feel free to let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WnhazbqxUT", "reviewer": "Reviewer_bEZy", "summary": "Flooding method is a previous method which aims at improving the generalization performance by making the average loss equal to a given threshold. This paper makes two changes of the 'flooding' method: 1. Rather than making the average loss equal to a given threshold, they make the point-wise loss equal to a given threshold. 2. To make them equal, previous work uses the absolute function to measure the difference, this work uses a softer function. Experiments are conducted to support their method.", "questions": "None", "limitations": "None", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper is easy to follow. The organization of this paper is clear.\nThe proposed link between flooding and sharpness, as shown in 2.2, is interesting and insightful. It explains why flooding method works from the flatness perspective.", "weaknesses": "Although the generalization gap of the proposed method is smaller (shown in Table 1), the accuracy is not larger than SAM (shown in Figure 4). What are the advantages of the proposed method compared with SAM?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for taking the time to read our paper, write a review, and for the positive outlook on our work. Below, we respond to the question you raised.\n\n> Although the generalization gap of the proposed method is smaller (shown in Table 1), the accuracy is not larger than SAM (shown in Figure 4). What are the advantages of the proposed method compared with SAM?\n\nIf accuracy at test time is the only metric of interest, and computational cost is not a concern, then SAM is perfectly fine. On the other hand, our tests highlight how with a slight drop in accuracy, SoftAD can achieve big gains in terms of other metrics such as the test-time average surrogate loss, generalization gap, and model norm size. In addition, the implementation of SoftAD is very simple, and computational cost is half of that of SAM (one gradient computed by backprop, instead of two). As such, when multiple performance metrics are of interest, and we are interested in achieving good *tradeoffs* between these metrics, then we believe the main results of this paper are encouraging initial evidence for the utility of SoftAD.\n\nThanks again for the feedback. If you have any additional questions, we are happy to respond at any time.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces SoftAD, a novel optimization method aimed at balancing loss minimization and model complexity control. It is well-supported by a strong theoretical foundation and extensive empirical validation across several datasets, demonstrating its effectiveness compared to existing methods like Flooding and SAM. The paper is well-organized, with clear explanations and visual aids that support its claims. 
The authors have carefully and appropriately responded to the reviewers' concerns at the time of submission in the rebuttal phase and have provided reasonable responses. In conclusion, the paper would make a significant contribution to the field and is appropriate for acceptance.
Decision: Accept (poster)