Sassha: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
Authors: Dahun Shin
Dongyeop Lee
Jinseok Chung
Namhoon Lee
Venue: ICML 2025 poster (2025)
Keywords: deep learning
second-order optimization
sharpness minimization
Primary area: deep_learning->algorithms
TL;DR: We introduce Sassha, a novel second-order optimization method that improves generalization by reducing solution sharpness.

Abstract: Approximate second-order optimization methods often exhibit poorer generalization compared to first-order approaches. 
In this work, we look into this issue through the lens of the loss landscape and find that existing second-order methods tend to converge to sharper minima compared to SGD.
In response, we propose Sassha, a novel second-order method designed to enhance generalization by explicitly reducing sharpness of the solution, while stabilizing the computation of approximate Hessians along the optimization trajectory.
In fact, this sharpness minimization scheme is crafted also to accommodate lazy Hessian updates, so as to secure efficiency besides flatness.
To validate its effectiveness, we conduct a wide range of standard deep learning experiments where Sassha demonstrates its outstanding generalization performance that is comparable to, and mostly better than, other methods.
We provide a comprehensive set of analyses including convergence, robustness, stability, efficiency, and cost.

SASSHA: Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
Dahun Shin 1 Dongyeop Lee I Jinseok Chung'
Namhoon Lee
Abstract
Approximate second-order optimization methods often exhibit poorer generalization compared to first-order approaches. In this work, we look into this issue through the lens of the loss landscape and find that existing second-order methods tend converge sharper minima compared to SGD. In response, propose SASSII,, novel second- order method designed to enhance generalization by explicitly reducing sharpness of the solution, while stabilizing the computation of approximate Hessians along the optimization trajectory. fact, this sharpness minimization scheme crafted also to accommodate lazy Hessian updates, so secure efficiency besides flatness. To validate its effectiveness, we conduct wide range of stan- dard deep learning experiments where SASSHA demonstrates its outstanding generalization per- formance that comparable to, and mostly better than, other methods. We provide comprehensive set of analyses including convergence, robustness, stability, efficiency, and cost.
Figure 1. Motivating toy example (a mixture of bivariate Gaussian densities). SASSHA converges flat minimum unlike others.
While the precise understanding is still under investigation, many studies have suggested a strong correlation between the flatness of minima and their generalization capabilitiess (Keskar et al., 2017), spurring the development of optimiza- tion techniques aimed at inducing flat minima (Chaudhari et al., 2017; Izmailov et al., 2018; Foret et al., 2021; Orvieto et al., 2022). Inspired by this, we raise an important question in this work: what type of minima do second-order methods converge to, and is there any potential for improving their generalization performance based on that?
1. Introduction
To answer these questions, we first measure the sharpness of different second-order methods using diverse metrics, suggesting that they converge to significantly sharper min- ima compared to stochastic gradient descent (SGD). Then, we propose SASSHA- Sharphess- aware Adaptive Second- order optimization with Stable Hessian Approximation- designed enhance the generalization of approximate second-order methods by explicitly reducing sharpness (see Figure 1 for the basic results).
Approximate second-order methods have recently gained surge of interest due their potential accelerate the large-scale training process with minimal computational and memory overhead (Yao et al., 2021; Liu et al., 2024; Gupta et al., 2018). However, studies also suggest that these methods may undermine generalization, trying to identify underlying factors behind this loss (Wilson et al., 2017: Zhou al., 2020: Zou al., 2022). For instance, Amari al. (2021) shows that preconditioning hinders achieving the optimal bias for population risk, and Wadia et al. (2021) points to negative effect of whitening data.
SASSHA incorporates a sharpness minimization scheme similar to SAM (Foret et al., 2021) into the second-order optimization framework, in which the Hessian diagonal is estimated. Such estimates, however, can become numeri- cally unstable when enforcing the sharpness reduction pro- cess. To increase stability while preserving the benefits of reduced sharpness, we make a series of well-engineered design choices based on principles studied in the literature. This not only smoothly adjusts underestimated curvature, but also enables efficient reuse of previously computed Hes-
Equal contribution 'POSTECH. Correspondence to:
Dahun Shin <dahunshin www.@@ppoosotrrr...p. Dongyeop Lee <dylee23@ [yylle23@pstecchacc.. postech.ac.kr.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
2017), averaging model weights (Izmailov et al., 2018), ex- We extensively evaluate the effectiivneess of SASSHA across diverse vision and natural language tasks. Our results reveal that SASSHA consistently achieves flatter minima and at- tains stronger generalization performance, all compared to existing practical second-order methods, and interestingly, to first-order methods including SGD, AdamW, and SAM. Furthermore, we provide an array of additional analyses to comprehensively study SASSHA including convergence, robustness, stability, efficiency, and cost.
sians, resulting in stable and efficient algorithm.
plicitly regularizing sharpness by solving a min-max prob- lem (Foret et al., 2021), and injecting anti-correlatd noise (Orvieto et al., 2022), to name few. In particular, the sharpness-aware minimization (SAM) (Foret et al., 2021) has attracted significant attention for its strong generaliza- tion performance across various domains (Chen et al., 2022; Bahri et al., 2022: Qu et al., 2022) and its robustness to label noise (Baek al., 2024). Nevertheless, to our knowledge, the sharpness minimization scheme has not been studied to enable second-order methods to find flat minima as of yet.
2. Related Works
3. Practical Second-order Optimizers Converge to Sharp Minima
Second-order optimization for deep learning. First- order methods such as SGD are popular optimization meth- ods for deep learning due to their low per-iteration cost and good generalization performance (Hardt et al., 2016). However, these methods have two major drawbacks: slow convergence under ill- ill-conditioned landscapes and high sen- sitivity to hyperparameter choices such as learning rate (De- meniconi& & Chawla, 2020). Adaptive methods (Duchi et al., 2011; Hinton et al., 2012; Kingma& Ba, 2015) propose us- ing empirical Fisher-type preconditioning to alleviate these issues, though recent studies suggest their insufficiency to do so (Kunstner et al., 2019). This has led to recent inter- est in developing approximate second-orde -oderr methods such as Hessian-Free Inexact Newton methods (Martens et al., 2010; Kiros, 2013), stochastic quasi-Newton methods (Byrd et al., 2016; Gower et al., 2016), Gauss-Newton methods (Schraudolph, 2002; Botev al., 2017), natural gradient methods (Amari et al., 2000), and Kronecker-factored ap- proximations (Martens & Grosse, 2015; Goldfarb et al., 2020). However, these approaches still incur non-trivial memory and computational costs, or are difficult to paral- lelize, limiting their applicability to large-scale problems such as deep learning. This has driven growing interest in developing more scalable and efficient second-order ap- proaches, particularly through diagonal scaling methods (Bottou et al., 201 8; Yao et al., 2021; Liu et al., 2024), to better accommodate large-scale deep learning scenarios.
In this section, we investigate the sharpness of minima ob- tained by approximate second-order methods and their gen- eralization properties. We posit that poor generalization of second-order methods reported in the literature (Amari et al., 2021: Wadia al.. 2021) can potentially be attributed to sharpness of their solutions.
We employ four metrics frequently used in the literature: maximum eigenvalue of the Hessian, the trace of Hes- sian, gradient-direction sharpness, and average sharpness (Hochreiter & Schmidhuber, 1997; Jastrzębski et al., 2018; Xie et al., 2020; Du et al., 2022h; Chen et al., 2022). The first two, denoted Xmaax (ff and tr(H), are often used as standard mathematical measures for the worst-case and the average curvature computed using the power iteration method and the Hutchinson trace estimation, respectively. The other two measures, óggg and ò Laug, assess sharpness based on the loss difference under perturbations. &Lgr eval- uates sharpness in the gradient direction and is computed as L(x* + +VL((**) V( (I) L(x"). & computes the average loss difference over Gaussian random perturba- tions, expressed ( [<<x")>>>>>> + pz/ pz/ /||]]]]] L(x")). Here we choose p = 0.1 for the scale of the perturbation.
With these, we measure the sharpness of the minima found by three approximate second-order methods designed for deep learning; Sophia-H (Liu et al., 2024), AdaHessian (Yao et al., 2021). and Shampoo (Gupta al., 2018), and compare them with SASSHA as well as SGD for reference. We also compute the validation loss and accuracy to see any correlation between sharpness and generalization of these solutions. The results are presented in Table 1.
Sharpness minimization for generalization. The rela- tionship between the geometry of the loss landscape and the generalization ability of neural networks was first discussed in the work of Hochreiter & Schmidhuber (1994), and the interest in this subject has persisted over time. Expanding on this foundation, subsequent studies have explored the impact of flat regions on generalization both empirically and theoretically (Hochreiter & Schmidhuber, 1997; Keskar et al., 2017; Dziugaite& & Roy, 2017: Neyshabur et al., 2017; Dinh et al., 2017; Jiang et al., 2020). Motivated by this, various approaches have been proposed to achieve flat min- ima such as regularizing local entropy (Chaudhari et al.,
We observe that existing second-order methods produce solutions with significantly higher sharpness compared to SASSHA in all sharpness metrics, which also correlates well with their generalization. We also provide a visualization of the loss landscape for the found solutions, where we find that the solutions obtained by second-order methods are indeed much sharper than that of SASSHA (Figure 2).
2

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
Table Sharpness measurements of the solutions found by different optimizers and their generalization for ResNet-32 CIFAR-I00
Approximate second-order methods tend to yield highly sharp solutions and poor generalization compared to SASSHA. We provide more results other workloads Appendix where the same trend holds.
Sharpness
Generalization
Amaz (H)
tr(H)x10 Xx10
&ggad
8 10
Lival
AcCyat (%)
SGD
265
=.290=0.300
0.703=0.132
1.310
1.260=0.001
69.32=00.19
AdaHessian
11992
46,94 17.00
4.119 =1136
12.50
1.377
68.06.0.22
Sophia-H
22797 10807
68. 15.15 220.19
=3.00
19.19+6.3800
1,463 00.0222
67. 67.76
Shampoo
436374499017
6823 #6617
73.27 12.51
+9337789
1.386
64.08 20.46
SASSHA
107 107.440.00
1.870
0.238
00.888
0.650
0.961 10.005
0.900000005
72. 14.14.00.10
10
10
20
-011 ms
10
-01
10
05
-0.05
6.05
3.20
0.05
-0.05
0.05
0.10
3,55
-0.05
5.05
0.19
605
-0.03
0.05
0.10
5.05
(a) AdaHessian
(b) Sophia-H
(c) Shampoo
(d) SASSHA
Figure 2. Visualization of the found solutions along the directions of the dominant eigenvectors.
4. Method
function:
In the previous section, observe that the generalization performance of approximate second-order algorithms anti- correlates with the sharpness of their solutions. Based on this, we introduce SASSHA-a novel adaptive second-order method designed to improve generalization by reducing sharpness without adversely impacting the Hessian.
F((z) = f(+
Vi(aa)
(3)
[V((2)))
which shifts the point of the approximately highest function value within the neighhorhood to the current iterate.
With this sharpness-penalizer objective, proceed to make second-order Taylor approximation:
4.1. Sharpness-aware Second-order Optimization
1++1 = arg min J (xx) + V (xr)) (x x..))
We consider a min-max problem, similar to Keskar et al. (2017); Foret et al. (2021), to minimize sharpness. This defined minimizing the objective within the entire p-ball neighborhood:
+ (x x4 H. (3+)) (x x.t),
where Hl denotes the Hessian of fc. Using the first-order optimality condition, we derive the basis update rule for our sharpness-awe aware second-order optimization:
min max f(x + cc,
(1)
ZERd ||r
T++1 = Ft He (z.) Vi (xx)
Based on this, we construct our sharpness minimization technique for second-order optimization as follows. We first follow a similar procedure as Foret et al. (2021) by solving for on the first-order approximation of the objective, which exactly solves the dual norm problem as follows:
I2 H (z, + + V Vf (I,
(4)
where H denotes the Hessian of the original objective f.
Practical second-order methods must rely on approximately estimated Hessians (i.e., H + H) since the exact computa- tion is prohibitively expensive for large-scale problems. We choose to employ the diagonal approximation via Hutchin- son's 's method. However, as we will show in our analysis (Section 6.2), we find that these estimates can become nu- merically unstable during the sharpness reduction process,
Vf(x))
c = arg max f(x)) + TT Vf((z)) = P (
We plug this back to yield the following perturbed objective
3

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
as it penalizes Hessian entries close to zero. This can lead to fatal underestimation of the diagonal Hessian compared to scenarios without sharpness minimization, significantly disrupting training. We propose a stable Hessian approxi- mation to address these issues in the following sections.
between bias and variance of the population risk, thereby improving generalization. We specifically adopt a 1/2 (i.e., square root), as it has consistently demonstrated robust performance across various scenarios (Amari et al., 2021; Kingma & Ba, 2015).
4.2. Improving Stability
Avoiding critical points. A well- limitation of ap- plying second-order optimization to deep learning objectives is the risk of convergence to saddle points or local maxima. To mitigate this, we attend to the prior works of Becker et al. (1988); Yao et al. (2021) and employ the absolute function to conservatively adjust the negative entries of the diagonal Hessian to be positive, i.e.
Alleviating divergence. Approximate second-order meth- ods can yield overly large steps when their diagonal Hessian estimations underestimate the curvature (Dauphin et al., 2015). However, this instability seems to be more present under sharpness minimization, presumably due to smaller top Hessian eigenvalue A (Agarwala & Dauphin, 2023; Shin et al., 2025) yielding smaller diagonal entries:
A
> Hale,e
(5)
|ual = le, Heel S max ||
10 Ho|
where e; is the jth standard basis vector. This tendency toward zero intensifies numerical instability during Hessian inversion, increasing the risk of training failures.
where Hü and e, are the 7th diagonal entry of the approx- imate diagonal Hessian and the 7th standard basis vector, respectively. Here, the basic idea is to maintain the direc- tionality of the gradient by flipping the sign of the negative entries in the Hessian, preserving the original magnitude of its values. This allows our method not only to take descent steps along directions of originally negative curvature, but also to preserve the relative scale specifically among the negative elements of the Hessian. As result, it mitigates the risk of convergence to the undesired critical points. We empirically validate the effectiveness of this approach in Appendix F.2.
A conventional approach to mitigating this involves damping or clipping, which, while stabilizing it reasonably well, re- quires carefully tuning their additional hyperparameters. In- stead, we propose square rooting the Hessian (ie.. 1/2) which effectively mitigates instability and allows improved generalization performance over other alternatives without additional hyperparameters. We present empirical validation of this in Section 6.2 and Appendix F.I.
Its benefits can be understood from two perspectives. First, the square root preserves the relative scale between each element of the Hessian while smoothly increasing the mag- nitude of the near-zero diagonal Hessian entries in the de- nominator (i.e., h < vh if 0 < h < 1). This property is particularly valuable when the sharpness minimization is underway, where the overall Hessian values tend to be small. In such cases, even small differences between Hes- sian elements may carry nontrivial curvature information. square-rooting can help retain this relative structure while also mitigating numerical instability caused by underesti- mated curvature. In contrast, both damping and clipping operate by entirely shifting abruptly replacing Hessian values based on predefined and fixed threshold criterion. As a result, when the Hessian is generally small due to sharpness minimization, informative dimensions may fall below the threshold, removing potentially critical variations and hence deteriorating the quality of preconditioning. This behavior can also make the method more sensitive to the choice of the threshold hyperparameter.
4.3. Improving Efficiency via Lazy Hessian Update
While the diagonal Hessian approximation can significantly reduce computations, it still requires at least twice as much backpropagation compared to first-order methods. Here we attempt to further alleviate this by lazily computing the Hessian every steps:
D. Be (1-)))) +G] if 1 mod & = D
Be)
herwiss
where D; and ß3 are the moving average of the Hessian and its hyperparameter, respectively. This reduces the overhead from additional Hessian computation by 1/k: We set k = 10 for all experiments in this work unless stated otherwise.
However, extensive Hessian reusing will lead to significant performance degradation since it would no longer accurately reflect the current curvature (Doikov et al., 2023). Interest- ingly, S SASSHA is quite resilient against prolonged reusing, keeping its performance relatively high over longer Hessian reusing compared to other approximate second-ord -order meth- ods. Our investigation reveals that along the trajectory of
Second, it can further be interpreted as geometric interpo- lation between the identity matrix and the preconditioning matrix [ which, as theoretically analyzed in (Amari et al., 2021), provide natural mechanism for balancing
4

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
SASSHA, the Hessian tends change less frequently than existing alternatives. We hypothesize that the introduction of sharpness minimization plays an integral role in this phe-
Algorithm 1 SASSHA algorithm
1: Input: Initial parameter Zo, learning rate ( J. moving av- erage parameters B1, 92, Hessian update interval k, weight decay parameter
nomenon by biasing the optimization path toward regions
with lower curvature change, allowing the prior Hessian to remain relevant over more extended steps. We provide a
2: Set m- = 0. D-1
3: for f 1 to T do
4:
gt VfB(x+)
detailed analysis of the lazy Hessian updates Section 6.3.
5:
G p9./|19:|l
6:
g: V/a(Ir +
4.4. Algorithm
7:
= 31m-11
Bi)
8:
m²/(1-88 -i)
The exact steps of SASSHA outlined in Algorithm 1. We also compare SASSIIA with other adaptive and second-order methods in detail in Appendix B. where one can see the exact differences between these sophisticated methods.
9:
if fodd k: then
10:
H. = Hx + +2)
b Section 4.1
11:
D. 32
B2) Feel
12:
D. /D./ 135)
b Section 4.2
13:
else
14:
DI-1
b. Section 4.3
4.5. Convergence Analysis
15:
end if
16:
D, Mi da
In this section, we present a standard convergence analysis of SASSHA under the following assumptions.
17:
end for
Assumption 4.1. The function is bounded from below,
i.c., f = inf, f(x) > -00.
on this, we demonstrate that SASSHA requires minimum to possess a certain level of flatness and Hessian uniformity to settle in it, whereas approximate second-order optimizers do not necessarily require such restriction, thus allowing to stay in much sharper minima.
Assumption 4.2. The function f is twice differentiable,
convex, and B-smooth. That is, 0 3 42 3... B.
Assumption 4.3. The gradient V (are) is nonzero for finite number of iterations, i.e., V (xx) + for all t e n...
Consider general optimizer .;-1 I₁ G( &x) with randomness induced by i.i.d. variables &, that are indepen- dent from the iterate z₁. Also, let us assumee that the minima possess a fixed point x= such that fx(( = O for any &.
Under these assumptions, we derive a descent inequality for [aree) by leveraging Adam-like proof techniques from al. (2023) to handle the diagonal Hessian and employing smoothness -ased bounds to account for the perturbation step based on analyses of Khanh et al. (2024). Now we give the convergence results as follows:
With these, we define the linear stability of the fixed point x* as follows:
Theorem 4.4. Under Assumptions 4. 4.1.4.3, given any initial point Zo E Rd, let {xr} be generated by the update rule SASSHA Equation (8) with step sizes me and perturbation radii Pt satisfying Et le 00, Ei= Ti < 00, Eespin pêne < co. Then, we have lim inf, [[x(2)]] 00
Definition 4.5. (Linear stability). A fixed point xt is lin- early stable for the optimizer G if there exists a constant C such that
E[ 1 < Cl x" 11?, for all t 6> 0
211|
This preliminary result indicates that any limit point of SASSHA is a stationary point of f, ensuring progress to- wards optimal solutions. We refer to Appendix C for the full proof details.
under the linearized dynamics near xt: 2141 by VG - x²), ), i.e., when it does not deviate infinitely far from x".
4.6. Flatness Analysis
Here, this linear dynamic arises when o, has approached sufficiently close to x= such that the loss becomes approxi- mately quadratic.
To understand how SASSHA can end up in flatter minima as observed in Section 3, we attend to linear stability analysis. Originally developed to explain similar behavior in SGD (Wu et al., 2018) 8) and also later extended to SAM (Shin et al., 2025), this framework suggests that an optimizer does not settle in every minimum it approaches, but instead escapes unless it encounters one that satisfies specific stability condi- tions conditinns that can vary between optimizers. Based
Under this framework, we present the necessary conditions of the linearly stable minima of SASSHA in the following corollary.
Corollary 4.6. Assume without loss of generality that x² 0. Then, the linearly stable fixed point x of SASSHA satisfy
5

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
the following necessury conditions:
We begin by comparing the generalization performance of adaptive second-order methods to that of first-order meth- ods. Across all settings, adaptive second-order methods consistently exhibit lower accuracy than their first-order counterparts. This observation aligns with previous stud- ies indicating that second-order optimization often result in poorer generalization compared to first-order approaches. In contrast, SASSHA, benefiting from sharpness minimiza- tion, consistently demonstrates superior generalization per- formance, outperforming both first-order and second-order methods in every setting. Particularly, SASSHA is up to 4% more effective than the best-performing adaptive or second-order methods (e.g., WRN-28-10 10, ViT-s-32).. In ad- dition, SASSHA continually surpasses SGD and AdamW by approximately 0.3% to 3%, even when these methods are trained for twice as many epochs. Further details on these experiments are provided in Appendix H.
2x
e
Sa(1+ + pa)
5
05
82
1p²
2npe
6²
2
2 V 0 5
0
2
S
2rppp
P
1/²
where Amaz (B (B) and Sk. Amaz ([E
)//g) are the sharpness and the non non-uniformity of the stochastic Hessian measured with the k-th moment, re- spectively.
A detailed proof is provided in Appendix D.
These results indicate that SASSHA escapes from minima unless they satisfies both low sharpness and uniformly distributed Hessian moments, with the conditions becom- ing much stricter with larger p and 1). In comparison, standard approximate second-order methods of the form Tt-1 ztt (ax)) with P z H-1 remain stable without such conditions, as shown by Wu et al. (2018):
Interestingly, SASSHA also outperforms SAM. Since first- order methods typically exhibit superior generalization per- formance compared to second-orde order methods, it might be intuitive to expect SAM to surpass SASSHA if the two are viewed merely as the outcomes of applying sharpness min- imization first-order and second-order methods, respec- tively. However, the results conffict with this intuition. We attribute this to the careful design choices made in SASSHA, stabilizing Hessian approximation under sharpness mini- mization, so as to unleash the potential of the second-order method, leading to its outstanding performance. As a sup- port, we show that naively incorporating SAM into other second-order methods does not yield these favorable results in Appendix I. We also make more comparisons with SAM in Section 5.3.
2
Amax ( (P- = 17
allowing convergence to minima of any sharpness provided
I 2.
5. Evaluations
In this section, we demonstrate that SASSHA can indeed improve upon existing second-order methods available for standard deep learning tasks. We also show that SASSHA performs competitively to the first-order baseline meth- ods. Specifically, SASSHA is compared to AdaHessian (Yao et al., 2021), Sophia-H (Liu et al., 2024), Shampoo (Gupta et al., 2018), SGD, AdamW (Loshchilov & Hut- ter. 2018), and SAM (Foret al., 2021) on a diverse set of both vision and language tasks. We emphasize that we perform extensive hyperparameter search to rigorously tune all optimizers and ensure fair comparisons. We pro- vide the details of experiment settings to reproduce our results in Appendix E. The code to reproduce all results reported in this work is made available for download at https: / //ttbb. www/Loo-postech/// Sassha.
5.2. Language Modeling
Recent studies have shown the potential of second-order methods for pretraining language models. Here, we first evaluate how SASSHA performs on this task. Specifically, we train GPT1-mini, a scaled-down variant of GPT1 (Rad- ford et al., 2019), on Wikitext-2 dataset (Merity et al., 2022) using various methods including SASSHA and compare their results (see the left of Table 3). Our results show that SASSHA achieves the lowest perplexity among all meth- ods including Sophia-H (Liu et al., 2024), recent method that is designed specifically for language modeling tasks and sets state of the art, which highlights generality in addition to the numerical advantage of SASSHA. We further evaluate S SASSHA for GPT2 and provide results in Appendix J.
5.1. Image Classification
We first evaluate SASSIIA for image classification on CIFAR-10, CIFAR-100, and ImageNet. We train various models of the ResNet family (He al., 2016; Zagoruyko & Komodakis, 2016) and an efficient variant of Vision Trans- former (Beyer et al., 2022). We adhere to standard inception- style data augmentations during training instead of making use of advanced data augmentation techniques VDevs & Taylor, 2017) or regularization methods (Gastaldi, 2017). Results are presented in Table and Figure 3.
We also extend our evaluation to finetuning tasks. Specifi- cally, we finetune SqUeeZeBERT (landola et al., 2020) for di- verse tasks in the GLUE benchmark (Wang et al., 2018). The results are on the right side of Table 3. It shows that SASSHA compares competitively to other second-order methods. No-
6

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
Table Image classification results of various optimization methods in terms of final validation accuracy (mean++std). SASSHA consistently outperforms the other methods for all workloads. means omitted due to excessive computational requirements.
CIFAR-10
CIFAR-100
ImageNet
Category
Method
ResNet-20
ResNet-32
ResNet-32
WRN-28-10
ResNet-50
ViT-s-32
SGD
92.03
92.6944.00
69. 19.19
80.06
75,58
62.90
AdamW
92.04
92.42.0.113
68.78.0.22
10.35
75.38 20.08
66.166..115
First-order
SAM SGD
92.85 1:007
93.89 10
71.99 0.200
83.14
76.36.0.18
64.54
SAM AdamW
+::029
93.45.0024
71.15
$2.88420.31
76.35
68.31
AdaHessian
92.00
92.48
68.06.2.22
76.92
73.64
66.42
Sophia-H
91.81 +0.27
91.9944..0
67. 67.7640..37
79.35 +0.24
72.06.0.40
62.44_0.96
Second-order
Shampoo
88.55
90.23
64.08=0.06
74.06
SASSHA
92.98 +0.05 05
94.09
=0.21
72.14
83.54
76.43=0.18
=0.18
69.20
=0.30
CIFAR-10/ / ResNet-32
CIFAR-100 / WRN-28-10
ImageNet/ / ResNet-50
94
Saseha
75
SGOM
80
Adallessian
Accuracy (%)
92
Adamnt
Accuracy (%)
Accuracy (%)
70
Sophia-H
70
90
Shampoo
65
88
60
Sassha
60
SGDM
Savsha
Adalesssan
SCON
86
AdamiN
55
Adamly
50
Sophll
Shremppo
Sophls
84
50
50
100
150
50
100 Epoch
150
200
40
60
80
Epoch
Epoch
Figure 3. Validation accuracy curves along the training trajectory. We also provide loss curves in Appendix G.
Table 3. Language finetuning and pertraining results for various optimizers. For finetuning, SASSHA achieves hetter results than Adam W and AdaHessian and compares competitively with Sophia-H. For pretraining, SASSHA achieves the lowest perplexity among all optimizers.
Pretrain/ GPTi-mini
Fimetune/ SQeereBERT
Wikitext-2
SST-2
MRPC
STS-B cosE.
QQP
MNLI mat/n. mat/n.mat
QNLI Arc
RTE Acc
Ferplexity
Acc
Acc F1
S/B
AdamW
175.00 79.00
90 29
84. 31.50 / 1888800 90
88.3 3es34
/88 88
89. 39.92annss / 86.38
81 81.22
/82 m2
69 29.93
68. 26.86
SAM
158.00 06
90.52:0.39
25 / 87 20
88.3 38.38...
/88 g0
90 10::26:0.0
86 86.90
81. 31.56:2014
/82,46.0
90.38 20.3820.00
68 83.
AdaHessiin
407.0 00 Lo.30
80 90.64..0 10.1
2.00 85.26 3.00
2..0 /86.46 1.00
90.37 an 97.07 1o.05
81.33 10
18.08 oom
89.94 89.94.0.1
71.00.1.01
Sophin.H H
157. 00..
90.44:s.46
85. 78 28.07 / 80. 00.90
88. 17 17.1.07
88. 53 2.133
90.70 / 87, g7.00
$1.77 a0.18 / /8.366.022
90. 12
20.76.1.41
SASSHA
.0.15
90.44.0.0s
.n.2
99..13...00
$8.72
m..m
89.10
90.91.0.02
10.00
$1.81gnas
/8171.n11
80.85
72.08...53
tably, it also outperforms Adam AdamW-often the method of choice for training language models- nearly all tasks.
to SAM to see whether it compares to SASSHA. The results are presented in Table 4. We find that SAM still underper- forms SASSHA, even though it is given more budgets of training iterations over data or wall-clock time. Further- more, we also compare SASSHA to more advanced variants of SAM including ASAM (Kwon et al., 2021) and GSAM (Zhuang et al., 2022), showing that SASSHA performs com- petitively even to these methods (Appendix K). Notably, however, these variants of SAM require lot more hyperpa- rameter tuning to be compared.
5.3. Comparison to SAM
So far, we have seen that SASSHA outperforms second-order methods quite consistently on both vision and language tasks. Interestingly, we also find that SASSHA often im- proves upon SAM. In particular, it appears that the gain is larger for the Transformer-based architectures, i.e., ViT results in Table 2 or GPT/BERT results in Table 3.
We suspect that this may be due to the robustness of SASSHA to the block heterogeneity inherent in Transformer archi-
To further investigate these findings, we conducted addi- tional experiments. First, we allocate more training budgets
7

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
Table Comparison between SASSHA and SAM with more train- ing budgets for the ViT-s-32/ / ImageNet workload.
Table 5. Validation accuracy measured for ResNet-32/CIFAR--I at different levels of noise. SASSHA shows the best robustness.
Epoch
Time (s)
Accuracy (%)
Noise level
Method
0%
20%
40%
60%
SAM SGD
180
220,852
65. 10.63
SAM Adamm
180
234,374
68. 706
SGD
69.32-g0.19
62.18
55. 55.78
45.53
SAM sGD
71.99
65.53 g0.11
61.2010.17
51.9310.47
SASSHA
90
123,948
69.195
AdaHessian
68. 08.06:2..2
63.06.0.25
58.37 58.3720.13
46.02.
Sophia-H
67.76.0.37
=2.34
56.54...2s
45.37 20.27
Shampoo
64.08.0. 46
58.85 0.66
53.82.0.7
42.91.:0.00
tectures, where the Hessian spectrum varies significantly across different blocks. This characteristic is known to make SGD perform worse than adaptive methods like Adam on Transformer-base based models (Zhang et al., 2024). Since information via precon-
SASSHA
72.14=0.11
66.78
61.97
$3,98
10t
Sanera Nu-sot
10
Sports
Sagtt
hedor
Ind-+1i1
3 g1o
100
100
SASSHA leverages second-ordd ditioning gradients, it has the potential to address the ill- conditioned nature of Transformers more effectively than SAM with first-order methods.
10
10%
150
100
100
200
10
100
200
300
0.00
0.65
0.10
a.5
Step
D value
(a) Train loss
(b) Update size
(c) D distribution
6. Further Analysis
Figure 4. Effects of square-root measured for 3/-CIAA-
100; D is set to be either [H] 1/22 for SASSHA [H] for No-Sqrt. Sharpness minimization drives the diagonal Hessian entries move towards zero, causing divergence, The square-root in SASSHA helps counteract this effect, stabilizing the training process.
6.1. Robustness
Noisily labeled training data can critically degrade gener- alization performance (Natarajan et al., 2013). To evaluate how SASSHA generalizes under these practical conditions, we randomly corrupt certain fractions of the training data and compare the validation performances between differ- ent methods. The results show that SASSHA outperforms other methods across all noise levels with minimal accuracy degradation (Table 5). Additionally, we also observe the same trend on CIFAR-10 (Table 21).
around step 200 (Figure 4a). To look into it further, we also measure the update sizes along the trajectory (Figure 4b). The results show that matches well with the loss curves, suggesting that the training failure is somehow due to taking too large steps.
It turns out that this problem stemss from the preconditioning matrix D being too small; i.e., the distribution of diagonal entries in the preconditioning matrix gradually shifts toward zero values (Figure 4c); as a result, D-1 becomes too large, creating large steps. This progressive increase in near-zero diagonal Hessian entries is precisely due to the sharpness minimization scheme that we introduced; it penalizes the Hessian cigenspectrum to yield flat solutions, yet it could also make training unstable if taken naively. By including square-root, the preconditioner are less situated near zero, effectively suppressing the risk of large updates, thereby stabilizing the training process. We validate this further by showing its superiority to other alternatives including damping and clipping in Appendix F.I.
Interestingly, SASSHA surpasses SAM (Foret et al., 2021), which is known to be one of the most robust techniques against label noise (Back al., 2024). We hypothesize that its robustness stems from the complementary benefits of the sharpness- minimization scheme and second-order methods. Specifically, SAM enhances robustness by adversarially perturbing the parameters and giving more importance to clean data during optimization, making the model more resistant to label noise (Foret et al., 2021; Back et al., 2024). Also, recent research indicates that second-order methods are robust to label noise due to preconditioning that reduces the variance in the population risk (Amari et al., 2021).
pharpness-minniizzaionn
6.2. Stability
We also provide an ablation analysis for the absolute-value function in Appendix F.2, which demonstrates that it in- creases the stability of SASSHA in tandem with square-root.
To show the effect of the square-root function on stabilizing the training process, we run SASSHA without the square-root (No-Sqrt), repeatedly for multiple times with different random seeds. As result, we find that the training diverges most of the time. A failure case is depicted in Figure 4.
6.3. Efficiency
Here we show the effectiveness of lazy Hessian updates in SASSHA The results are shown in Figure 5. At first, we see that SASSHA maintains its performance even at = 100,
At first, we find that the level of training loss for So-Sqrt much higher than that of S ASSHA, and also, it spikes up
8

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
Saetha
AdaHessian
Saphia+
Sophis-H
Table 6. Average wall-clock time per epoch (3) and the thcoretical cost of different methods. SASSHA can be an effective alternative to existing methods for enhanced generalization performance.
Maxt (A (p.)) Ailll
Aaguaay
70
MAsee Hin- solli-
_0a-
10
10
10
10a-
Method
Descent
Sisepsos
Cost
Howian
Total
RosNe32
CIEARIO
WRN25-10
COTAR100
VIT-omall
ImageNett
0
50
100 Epach
160
AdamWy
0.00
1:00
5.03
59.29
976.56
H update interval
(x10%)
SAM AdaHessian
162
192
0 U19
2.2
9.16
115.46
1302.06
160
0.20
1 18V2
4.00
33.75
296.63
2489.07
(a) Lazy Hessian
(b) Hl change
(c) Local sensitivity
SASSEA
166
1:30
011 RVB
2.
12.00
142.06
1377.00
M-SASSIAA
64.12
1065.40
Figure 5. Effect of lazy Hessian for DosNet-32/CIFAR--T
SASSHA stays within the region where the Hessian varies small.
Furthermore, we leverage a momentum of gradients in the perturbation step to reduce the cost. This variant M- SASSHA requires only 1.3GCs with minimal decrease in per- formance. Notably, M-SASSHA still outperforms standard first-order methods like SGD and Adam (Appendix M).
indicating that it is extremely robust to lazy Hessian updates (Figure 5a). We also measure the difference between the cur- rent and previous Hessians to validate lazy Hessian updates more directly (Figure 5b). The result shows that SASSHA keeps the changes in Hessian to be small, and much smaller than other methods, indicating its advantage of robust reuse, and hence, computational efficiency.
To verify, we measure the average wall-clock times and present the results in Table 6. First, one can see that the theoretical cost is reflected well on the actual cost; i.e., the time measurements scales proportionally roughly well with respect to the total cost. More importantly, this result in- dicates the potential of SASSHA for performance-crin applications. Considering its well-balanced cost, and that it has been challenging to employ second-order methods effi- ciently for large-scale tasks without sacrificing performance, SASSHA can be reasonable addition to the lineup.
We attribute this robustness the sharpness minimization scheme incorporated in SASSHA, which can potentially bias optimization toward the region of low curvature sensitivity. To verify, we define local Hessian sensitivity as follows:
8 6 ( A (x)
(
i
max (0,1
x + P
110112
7. Conclusion
Le., it measures the maximum change in Hessian induced from normalized random perturbations. A smaller Hessian sensitivity would suggest reduced variability in the loss curvature, leading to greater relevance of the current Hessian for subsequent optimization steps. We find that SASSHA is far less sensitive compared to other methods (Figure 5c).
In this work, we focus on addressing the issue of poor gener- alization in approximate second-order methods. Our empiri- cal analysis indicates that this limitation may be attributed to their tendency to converge sharp minima, which are known to correlate with weaker generalization performance. To this end, we propose a new method called SASSHA that stably minimizes sharpness within the framework of second- order optimization. SASSHA converges to flat solutions and achieves state-of-the-an performance within this class. S ASSHA also performs competitively to widely-used first- order, adaptive, and sharpness-aware methods. SASSHA achieves this efficiently through lazy Hessian updates, to which it is robust, and does so without requiring extra hy- perparameter tuming. Moreover, SASSHA exhibits strong resilience to label noise. All of these are rigorously assessed with extensive experiments.
6.4. Cost
Second-ordes methods can be highly costly. In this section, we discuss the computational cost of SASSIAA and reveal its competitiveness to other methods.
SASSHA requires one gradient computation (GC) in the sharpness minimization step, one Hessian-vector product (HVP) for diagonal Hessian computation, and an additional GC in the descent step. That is, a total of 2GCS and HHV are required. However, with lazy Hessian updates, the num- ber of HVPs reduces drastically to 1/k. With 10 as the default value used in this work, this scales down to 0.1HVPS.
Nonetheless, there are still many limitations to be addressed to further improve this work. Some examples may include, but are not limited to, extending experiments to various models and different data of extreme scales, as well as de- veloping theoretical properties such as convergence rate, generalization bound, and implicit bias, all to more rigor- ously confirm the value of SASSHA Seeing it as an exciting opportunity, we plan to investigate further in future work.
It turns out that this is critical to the utility of SASSHA, because HHVP is known to take about x3 the computation time of 1GC in practice (Dagrćou et al., 2024). Compared to conventional second-order methods (1GC + 1HVP ~ 4GCs), the cost of SASSHA can roughly be a half of that (2.3GCs). is also comparable to standard SAM variants (2GCs).

---

Sharpness-aware Adaptiv Second-order Optimization with Stable Hessian Approximation
Acknowledgement
Botev, A., Ritter, H., and Barber, D. Practical gauss-newton optimisation for deep learning. ICMI., 2017.
This work was partly supported by the Institute of Infor- mation & communications Technology Planning & Evalua- tion (IITP) grant funded by the Korean government (MSIT) (ITTP-2019-0-0) 0001--0-0190,, Artificial Intelligence Graduate School Program (POSTECH), RS-2024-00338144 140, Development of learning and utilization technology to reflect sustain- ability of generative language models and up-to-dateness over time), and the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT) 0122022RIFIA1064550 RS-2023-0021066,, RS-2023- 00265444).
Bottou, L.,, Curtis, F. E., and Nocedal, Optimization methods for large-scale machine learning. SIAM Review, 2018.
Byrd, R. H., Hansen, S. L.,, Nocedal, J., and Singer, stochastic quasi-newton method for large-scale optimiza- tion. SIAM Journal Optimization, 2016.
Chaudhari, P., Choromanska, A., Soatto, S., LeCun, Y., Bal- dassi, C., Borgs, C., Chayes, J., Sagun, L., and Zecchina, R. Entropy-SGD: Biasing gradient descent into wide valleys. ICLR, 2017.
Impact statement
Chen, X., Hsich, C.-.,, and Gong, When vision trans- formers outperform resnets without pre-training strong data augmentations. ICLR, 2022.
This paper contributes to advancing second-order optimiza- tion, with potential implications for both theoretical insights and practical applications. While our work does not present immediate concerns warranting specific emphasis, we rec- ognize that progress in this field may have broader socictal impacts. We remain committed to engaging in discussions on the broader implications of our research should the need arise in the future.
Dagréou, M., Ablin, P., Vaiter, S., and Moreau, T. How compute hessian-vector products? The Third Blogpost Track at ICLR, 2024.
Dauphin, Y., De Vries. H., and Bengio, Y. Equilibrated adap- tive learning rates for non-convex optimization. NeurlPS, 2015.
References
Demeniconi, C. and Chawla, N. Second-order optimization for non-convex machine learning: an empirical study, Society for Industrial and Applied Mathematics, 2020.
Agarwala, A. and Dauphin, Y. Sam operates far from home: cigenvalue regularization as a dynamical phenomenon. ICML, 2023.
Deieis T. and Taylor, G. W. Improved regularization of convolutional neural networks with cutout. arXiv, 2017.
Amari, S.-i., Park, H., and Fukumizu, K. Adaptive method of realizing natural gradient learning for multilayer per- ceptrons. Neural computation, 2000.
Dinh, L.., Pascanu, R., Bengio, S., and Bengio, Y. Sharp
minima can generalize for deep nets. ICML, 2017.
Amari, S.-i., Ba, J., Grosse, B., Li, X., Nitanda, A., Suzuki, T., Wu, D., and J. When does preconditioning help hurt generalization? ICLR, 2021.
Doikov, N., Chayti, E. M., and Jaggi, M. Second-order optimization with lazy hessians. ICML, 2023.
Du, J., Yan, H., Feng, J., Zhou, J. T., Zhen, L.., Goh, R. S. M., and Tan, Efficient sharpness-aware minimization for improved training of neural networks. ICLR, 2022a.
Back, C., Kolter, J. Z., and Raghunathan, A. Why is SAM robust to label noise? ICLR, 2024.
Du, J., Zhou, D., Feng. J., Tan, V., and Zhou, J. T. Sharpness- aware training for free. NeurlPS, 2022b.
Bahri, D., Mobahi, H., and Tay, Y. Sharpness-awar mini- mization improves language model generalization. ACL, 2022.
Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. JMLR, 2011.
Becker, M., Altrock, F., and Risse, B. Momentum-sam: Sharpness aware minimization without computational overhead. arXiv, 2024.
Dziugaite, G. K. and Roy, D. M. Computing nonvacuous generalization bounds for deep (stochastic) neural net- works with many more parameters than training data. UAI, 2017.
Becker, S., Le Cun, Y., et al. Improving the convergence of propappropn learning with second order methods. CMSS, 1988.
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B. Sharpness-awe aware minimization for efficiently improving generalization. ICLR, 2021.
Beyer, L., Zhai, X., and Kolesnikov, A. Better plain vit baselines for imagenet- 1k. arXiv, 2022.
10

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang. J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leaby, C. The Pile: An 800gb dataset of diverse text for language modeling. arXiv, 2020.
Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. Fantastic generalization measures and where to find them. ICLR, 2020.
Karimireddy, S. P., Rebjock, Q. Stich, S., and Jaggi, M. Error feedback fixes signsgd and other gradient compres- sion schemes. ICML, 2019.
Gastaldi, X. Shake-shake shake regularization. arXiv, 2017.
Goldfarb, D., Ren, Y., and Bahamou, A. Practical quasi-newton methods for training deep neural networks. NeurlPS, 2020.
Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, T. P. On large-batch training for deep learning: Generalization gap and sharp minima. ICLR, 2017.
Gomes, D. M., Zhang, Y. Belilovsky, E., Wolf, G., and Hosseini, M. S. Adafisher: Adaptive second order opti- mization via fisher information. arXiv, 2024.
Khanh, P. D., Luong, H.-C., Mordukhovich, S., and Tran, D. B. Fundamental convergence analysis of sharpness- aware minimization. NeurlPS, 2024.
Gower, R., Goldfarb, D., and Richtárik, P. Stochastic block bfgs: Squeezing more curvature out of data. ICMI., 2016.
Kingma, D. and Ba, Adam: A method for stochastic optimization. ICLR, 2015.
Gupta, V., Koren, T., and Singer, Y. Shampoo: Precondi- tioned stochastic tensor optimization. In ICLR, 2018.
Kiros, R. Training neural networks with stochastic hessian- free optimization. arXiv, 2013.
Hardt, M., Recht, B., and Singer, Y. Train faster, generalize better: Stability of stochastic gradient descent. ICMI,, 2016.
Kunstner, F., Hennig, P., and Balles, L.. Limitations of the empirical fisher approximation for natural gradient descent. NewrIPS, 32, 2019.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. CVPR, 2016.
Kwon, J., Kim, J., Park, H., and Choi, I. K. Asam: Adaptive sharpness- aware aware minimization for invariinvrr learn- ing of deep neural networks. ICML, 2021.
Hinton, G., Srivastava, N., and Swersky, K. Neural networks for machine learning lecture 6a overview of mini-batch gradient descent. Coursera Lecture slides https:/cllass. 2012.
Li, H., Rakhlin, A., and Jadbabaic, A. Convergence of adam under relaxed assumptions. NeurIPS, 2023.
coursera. 0atpeeurrhnets-202200000//
Liu, H., Li, Z., Hall, D., Liang, P., and Ma, T. Sophia: scalable stochastic second-order optimizer for language model pre-training ICLR, 2024.
Hochreiter, S. and Schmidhuber, J. Simplifying neural nets by discovering flat minima. NeurIPS, 1994.
Hochreiter, S. and Schmidhuber, J. Flat minima. Neural Computation, 1997.
Liu, Y., Mai, S., Chen, X., Hsieh, C.-J., and You, To- wards efficient and scalable sharpness- minimiza- tion. CVPR, 2022.
Hutchinson, M. A stochastic estimator of the trace of the influence matrix for laplacian smoothing splines. Com- munications in Statistics Simulation and Computation, 1989.
Loshchilov, and Hutter, Decoupled weight decay regu- larization. ICLR, 2018.
Martens, J. and Grosse, R. Optimizing neural networks with kronecker-factored approximate curvature. ICML, 2015.
landola, F. Shaw, A., Krishna, R., and Keutzer, K. Squeeze- bert: What can computer vision teach nlp about efficient neural networks? SustaiNLP: Workshop on Simple and Efficient Natural Language Processing, 2020.
Martens, J. et al. Deep learning via hessian-free optimiza- tion. ICML, 2010.
Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. ICLR, 2022.
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. UAI, 2018.
Mi, P., Shen, L., Ren, T., Zhou, Y., Sun, X., Ji, R., and
Tao, D. Make sharpness-ave aware minimization stronger: sparsified perturbation approach. NeurIPS, 2022.
Jastrzębski, S., Kenton, Z., Ballas, N., Fischer, A., Bengio, Y., and Storkey, A. On the relation between the sharpest directions of dnn loss and the sgd step length. ICLR, 2018.
Natarajan, N., Dhillon, I. S., Ravikumar, P. K., and Tewari,
Learning with noisy labels. NeurIPS, 2013.

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
Neyshabur, B., Bhojanapalli, S., Mcallester, D., and Srebro, N. Exploring generalization in deep learning. NeurlPS, 2017.
Yao, Z.,, Gholami, A., Keutzer, K., and Mahoney, M. W. Py- hessian: Neural networks through the lens of the hessian. IEEE BigData, 2020.
Orvieto, A., Kersting, H., Proske, F., Bach, F., and Lucchi, A. Anticorrelated noise injection for improved general- ization. ICML, 2022.
Yao, Z.,, Gholami, A., Shen, S., Keutzer, K., and Mahoney, M. W. Adahessian: An adaptive second order optimizer for machine learning. AAAI, 2021.
Zagoruyko. S. and Komodakis, N. Wide residual networks. BMVC, 2016.
Qu, Z., Li, X., Duan, R., Liu, Y. Tang, B., and Lu, Z. Generalized federated learning via sharpness aware mini- mization. ICMI, 2022.
Zhang, Y., Chen, C.,, Ding, T., Li, 7., Sun, R., and Luo, 7.- Q. Why transformers need adam: A hessian perspective. NeurIPS, 2024.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Zhou, P., Feng, J., Ma. C., Xiong, C., Hoi, S. C. H., et al. Towards theoretically understanding why sgd generalizes better than adam in deep learning. NeurIPS, 2020.
Roosta-Khorasane F. and Ascher, U. Improved bounds on sample size for implicit matrix trace estimators. FoCM, 2014.
Zhuang. J., Gong, B., Yuan, L.., Cui, Y., Adam, H., Dvomek, N. C., sekhar tatikonda, Duncan, J., and Liu, T. Surro- gate gap minimization improves sharpness-aware training. ICLR, 2022.
Schraudolph, N. N. Fast curvature matrix-vector products for second-order gradient descent. Neural computation, 2002.
Zou, D., Cao, Y., Li, Y., and Gu, Q. Understanding the generalization of adam in learning neural networks with proper regularization. ICLR, 2022.
Shin, S., Lee, D., Andriushchenko, M., and Lee, N. Critical influence of overparamettrization on sharpness-aware minimization. UAI, 2025.
Wadia, N., Duckworth, D., Schoenholz, S. S., Dyer, E., and Sohl-Dickstein, Whitening and second order optimiza- tion both make information in the dataset unusable during training. and can reduce or prevent generalization. ICML, 2021.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Glue: A multi-task benchmark and analy- sis platform for natural language understanding. ICLR, 2018.
Wilson, A. C., Roelofs, Stern, M., Srebro, N., and Recht, B. The marginal value of adaptive gradient methods in machine learning. NeurlPS, 2017.
Wolf, T,, Debut, I.., Sanh, V., Chaumond, J., Delangue, C.,
Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jermite, Y., Plu, J., Xu, C., Scao, T. I.,, Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Huggingface's transformers: State-of-the-art natural language processing. arXiv, 2020.
Wu, L., Ma, C., et al. How sgd selects the global minima in perepaarmeterieed learning: A dynamical stability perspective. NeurIPS, 31, 2018.
Xie, Z., Sato, I., and Sugiyama, M. diffusion theory for deep learning dynamics: Stochastic gradient descent exponentially favors flat minima. ICLR, 2020.
12

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
A. Sharpness Measurements for Other Settings
Table 7. Sharpness measurements of the solutions found by seven different optimizers and their generalization on CIFAR-10/10 and Wikitext-2. Approximate second-order methods tend to yield highly sharp solutions and poor generalization compared SGD; SASSHA and M-SASSHA effectively recover this. Here, we measure sharpness in terms maximum Hessian eigenvalue Xmor (M), trace of Hessian tr( tr()),, gradient-direction sharpness g and average sharpness &ggg along with generalization using validation loss Leaf and accuracy Acceat.
Sharpness
Generalization
Aman (H)
H1) x x0
ggar
Lval
AcCyal
CIFAR-10 10
SGD
107=4.370
1.380
0.840
0.690
0.295
92,03
SAM
58 12.90
10.00
0.171 10.038
0.461 0.461= 00,240
0.119 1.0002
92.85 $0.07
AdaHessian
23048 220932
189.5 189.5_22006
4.538 4.538 L.664
198.7 -98.0=206.0
0.260=0.000
92,00 :0.00
ResNer20
Sophia-H
3606
31.24 226628
6.120 2.120
18.11 =1.000
0.316 0.316.40.002
91.81 +0.27
Shampoo
647066 1119944
3900 #100
166.3 1:48.0
2177189.
=1628993
0.38 10002
88.55 28.55
M-SASSHA
129
1.580
1,551 +0.684
1.025 1.025_0.300
-0.360
0.234_20.00
92.36
SASSHA
78 =5 5,00
0.860 00000
0.184 053
0.388
00.704
0.209 10.001
92.98 1:0.08
SGD>
56
0.800
0.560 10.219
0.196
0.309
92.69
SAM
42.000
0.580
0.107 +0.00
-0.351
00000
93.89 #0.13
AdaHessian
1746
17.06_10.22 10.24
4.599+1.710
5.518
0.27840.000
92,48 92.48.0.115 10.15
ResNet32
Sophia-H
7167
18.82 5,000
9.399 283
7.915 23.397
0.394 0.394_10.010
91.99 ::0.08
Shampoo
717553 #9312
717553
4523 2229.7
162. +:12.22
105322
82246
0.348 40.08
90.23 90.23.00.4 40.24
M-SASSHA
283:10.00
3,960=0.110 10.100
2.986
1.300
11 100010
93.18 10.30
SASSHA
47 1888
0.590
0.136+:0.019
0.714
00.072
94.09 #:0.4
CIFAR-100
SGD
265.25.00
7.290
0.703
1.310
1.260.0.001
69.32
SAM
12.00
2.630
0.266.40.025
-0.619
-0.61990.594
20.604
0.512410.016
71.99 400.0
AdaHessian
1199225779
46.94 46.94_11..0
4.119 1111::133
12.50 66008
1.377
68.06
ResNet32
Sophia-H
22797 10087
2:2011
8.130+3..00 #3.082
19.19 66388
1.463
67.76
Shampno
436374490177
6823 6664.7
73.27 1:2.51
49307489
-59979794
1.386:0.010
64.08 1:0
M-SASSHA
382
8.750
2.391 1.0. 425
2.260
1.067
70.93
SASSHA
107 40.000
1.870
0.238 +0.088
0.650
-0.800
0.961 40.000
72.14
SGD
1841
0.660 20.000
1,984 400506
-0.0077
0.820
80.06
SAM
9.0.8
0.230 20.000
0.841 1.0.081
0.024=0.041
0.648 10.00
83.14
AdaHessian
35119
139.5
6.745 1..932
19.727 2727-22.400
1.005
76.92
WRN28-10
Sophia-H
3419.33200
13.57-13.300
5,073 1:0.26
0.067=0.054
0.866_10.000
79.35 79.35.:0.24
Shampoo
102129-16772)
1459
483.0 +
98.558 123.11
1.173.0.058
74.06
M-SASSHA
2257 +248.0
30.40
30.40
4.770
4.599 4.0.003
0.301 -0.47
0.757
81.53
SASSHA
84.3.150
2.030 1.010
4.540 1.0 122.122
0.007
20.199
1..55.0.022
83.54..0.08
Wikitext-2
AdamW
836113.00
31.61
20.43 43.433
1.642
7
5.072.10.013
175.06
AdaHessian
114132
46.36 26.885
0.289 0.289.to.187 10.177
9
7.23 1.0003
407.69 20.20
Mini-GPT1
Sophia-H
319414.00
55.17
0.824
13_11
5.07744.0014
157.60+0.0r
M-SASSHA
145 =1250
13.23 117.19
0.379 +0.275
3-1
5,259 10.000
125.01 10.21
SASSHA
79 79
14.50
0.221 +0.023
3
4.808
122.40
13

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
B. Algorithm Comparison
Table Comparison of various optimization algorithms in terms of gradient momentum Me, diagonal preconditioning matrix D., and method-speeific operations U()). Here gt, He are the stochastic gradient and the Hessian estimation respectively, and f1, 82 denotes the momentum hyperparameters for the gradient and estimated Hessian. Bias correction bo(-) compensates for initialization biases in the gradient and Hessian momentum variables due to zero initialization.
2t+1 = Ft 17 UD,
m
U(z)
SGD with momentum
8ym + (1 B1)g:
I
2
Stochastic Newton
9t
(1ra))
2
Adam (Kingma & Ba, 2015))
8,m, + (1-B
Bon 1 + (1 (1) diag(ogg))
bc(=)
AdaHessian (Yao et al., 2021)
V 82m-1 + (1- -12) 17, (zx)²
bo(z)
Sophia-H (Liu et al., 2024)
B20, + (32) : (xx)
every steps
clip(2)
SASSHA (Ours)
8,m m + (1- (01-))) B1) ge(2r + +
V 110 + (1 82)
Hl(x, + ef) every steps
bc(#)
In this section, we compare our algorithm with other adaptive and approximate second-order methods designed for deep learning to better illustrate our contributions within concurrent literature. We present a detailed comparison of each methods in Table 8.
Adam (Kingma& & Ba, 2015) is an adaptive method popular among practitioners, which rescales the learning rate for each parameter dimension by dividing by the square root of the moving average of squared gradients. This adaptive learning rate effectively adjusts the gradient (momentum) at each descent step, accelerating convergence and improving update stability. Although Adam is not explicitly second-order method, its process is related to second-order methods as can be viewed as preconditioning via a diagonal approximation of the empirical Fisher information matrix. Adam (Loshchilov & Hutter, 2018) proposes to improve Adam by decoupling the weight decay from the update rule for better generalization. This is also shown to be effective in most approximate second-order methods, thus employed in all subsequently mentioned algorithms.
AdaHessian (Yao et al., 2021) is one of the earliest approximate second-order optimization methods tailored for deep learning. To reduce the prohibitive cost of computing the Hessian, it uses Hutchinson's 's method (Hutchinson, 1989; Roosta- Khooraann & Ascher, 2014) to estimate a diagonal Hessian approximation H. and applies a moving average to reduce variance the estimation. The authors also propose spatial averaging of the Hessian estimate, denoted as (EI( which involves averaging the diagonal element within filter of convolution layer for filter-wise gradient scaling. Sophia (Liu et al., 2024) is an approximate second-order method specifically designed for language model pretraining. Its primary feature is the use of the clipping mechanism clip(2) = max{ (min(,, (z, p): -p} with a predefined threshold to control the worst-case update size resulting from errorneuss diagonal Hessian estimates preconditioning. Additionally, a hard adjustment is applied to each Hessian entry, substituting negative and very small values with constant such as H/)) max (he, e} to prevent convergence to saddle points and mitigate numerical instability. Furthermore, Sophia also incorporates lazy Hessian updates to enhance computational efficiency. This works without significant performance degradation as the clipping technique and hard adjustment prevent a rapid change of the Hessian, keeping the previous Hessian relevant over more extended steps.
Our method, SASSHA, adds a perturbation E before computing the gradient and diagonal Hessian estimation penalize sharpness during training for promoting improved generalization previously explored in the literature. However, naive second-order optimization under sharpness minimization can be numerically unstable. because, when the overall curvature is small due to sharpness reduction, Hessian underestimation can cause the optimization to yield extremely large update. To address this issue, we introduce two simple techniques to Hessian estimates: the square root and the absolute function. The square root smoothly adjusts underestimated curvature while preserving the relative scale among Hessian entries. The absolute function enforces Hessian estimates to be comitpositive
approach that, our knowledge, has not been
14

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
definite while maintaining their magnitude. This not only prevents convergence to saddle points local maxima, but also allows the square root operate Hessian entries retaining the original Hessian magnitude. Together, the combination of sharpness minimization and Hessian stabilization enables efficient reuse of previously computed Hessians, resulting in stable and efficient algorithm.
C. Convergence Analysis of SASSHA
In this section, we provide preliminary convergence analysis results. Based on the well-established analyses of Li et al. (2023); Khanh et al. (2024), we further investigate the complexities arising from preconditioned perturbed gradients.
Assumption C.1. The function :R I is convex, f-smooth, and bounded from below, i.e., == inf, f(x) -0, Additionally, the gradient V fzz) is non-zero for finite number of iterations, i.e., Vf(xx) o 0 for all E 21. n..
Assumption C.2. Step sizes 7i, and perturbation radii Pt assumed to satisfy the following conditions:
Σ
00
Σ m
00,
<
00,
Σ t=1
p < 00.
Remark C.3.
The following notations will be used throughout
1. 9: (z+) denotes the gradient of f iteration t.
2. The intermediate points and the difference between the gradients are defined as
gt
c == Ie + Pt
g1+ = ((((++)) 6, = : 9t-
3. For u, E IR operations such as VT, |u| and at well the symbols 3 and 2, are applied element-wise.
Remark C.4, The update rule for the iterates is given by
11
1
91+3'
(8)
V ( ))1 +
where diag extracts the diagonal elements of matrix as a vector, or constructs a diagonal matrix from vector, and is small constant introduced prevent division by zero. Define h4 as
h
V diag(V" (V2(((++))))) +) +e
then the following hold
1. From the convexity and B-smoothness of f. the diagonal elements of V f2) are bounded within the interval [0, 3].
i.e.,
0 < V [V (.,) = e, 2 (()ee 8B
where is the i-th standard basis vector in Rdd.
2. The term h, is bounded as
17t
3 h4 1
BB
+c
15

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
Remark C.5. For the matrix representation
1. Denoting H, == diag(h;) (4)), the matrix bounds for H, are given by
?p
M1 I, H1
13
(9)
VB+C
where / is the identity matrix.
2. Using the matrix notation Hz, the update for the iterates is expressed
2++1 zt High+
Remark C.6. From the 3-smoothness of f. 8, is bounded by
(s()) | S B|| + Pt f xell = Bpt--
(10)
Lemma C.7 (Descent Lemma). Under Assumption C.I and Assumption C.2, for given B and e, there exists T such 2
for Vt 2 T, ip satisfies 1p min (B(vB+c)* 43 For such 2 T. the following inequality holds
f(x(+1) S f(x))
Tl
|1 + Tt 11 11.
(11)
2(v3 +e)
Proof. We begin by applying the 8-smoothness of
8 (0411 S (xx) (gt, -xa) + || xrl
8
f (x,) (gt, He + +,,) +
H1( + 81)1
2
S f (xx) 9.
T Hig. +
1
9t
Hig:
+
_
a T
11,8,
-
3
[| + 8t)1
20
2
S (x+) - (1
1
Tl
||
πlt
|6,118
8 m
119e + 6ell
2a
v
B + e
e
2 e
f fx)) - (1
1
|| 1 +
1 8 2
B t
(11 + 11ō [|ō: )
2a
vB +
E
f (z+) - (n((11
3 2
2 + nl 2e
a
+
B Tht
2 11&.111.
2a
/ +
The second inequality follows from Young's inequality, the third inequality is obtained from Equation (9), and the last
inequality is simplified using the property ||a + b12² < 2||a||2 + 2|bb/ By setting a 3. we get
= =f - m 2
3 nt Ilgeel 2 + t
3
+
+
vB +
4e
Since N/ + 0, hH E N such that m < min
63()3+e) 43 J. this gives 3/2 (att) - -3 2 2(v3tc)
2
3 and
+ < 1 4r c
which implies
< fxx) 2(V3
11
|1 12 +
11 2h 1|
Theorem C.S. Under Assumption C.I and Assumption C.2, given any initial point To E Rd,, let {xr} be generated by
Equation (8). Then, holds that lim inf ||g:1| 0.
16

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation Proof. From Lemma C.7 and Equation (10), have the bound
f(x+11) S f(x))
1 2(v3++
||,||
f(1)))
7
|
8 1t.
2(v3
By rearranging the terms, we obtain the following
7) 2(vB +
|9 1 f(x)) - f(1++1) T1t B3P.
For any M T, we have
M
M
32 M
Σ T=
no 119 11 $
Σ (f(x)) f(x(+1)) + Σ
p.n
2(v3
t=t
T=T
82
M
= [(MM+1) +
t=T
8
S f((T)
inf ((x) + LEN
Σ
pên-
As + co, the series Σt=T me | converges. Now, assume for contradiction that lim inf I19ell * O. This means there exists some ç > 0 and N 2 such that [ge]] 2 & for 2 N. Consequently, we have
nelloell 22
n 00,
N
which is contradiction. Therefore, lim inf || = 0.
17

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
D. Linear stability analysis on SASSHA
In this section, we provide the detailed proof of Section 4.6,
We begin by considering the minimization of the training error
1 f(x) - Σ fi()) i=1
via general second-order optimization method:
I++1 = 24 - P(x(: {t) [c) -1 (axe
(12)
where & is a i.i.d. random variable independent of 24. For SASSHAA, the update is specified
Te+1
It
VJe (xx + (V)e
(13)
diag fe, p ffa (x,)) +e
where diag extracts the diagonal elements of matrix as a vector, or constructs a diagonal matrix from vector, and is small constant introduced prevent division by zero.
We now derive the linear stability conditions for the fixed points of SASSHA
Definition D.1. (Fixed point). A point is called a fixed point of the stochastic dynamics (12) for any &, we have [fg(=*)
Definition D.2. (Linearized SASSHA). Let x² be the fixed point of interest and assume f(x) = 0. Consider the quadratic approximation of near x": fe(x) sJ 1 (x xx)) Hg(r SASSHA given by
x*) with Hçg V (x*). The corresponding linearized
T++1 It
c HE, (8x -x))
(14)
diag Hę,) +
where ã. = I4 + *.H.H. (xe - )²) is the linearized perturbed point.
For brevity, we define
de. / (digg(He.) (e.)) +
Then, the update (14) can be rewritten as
Fe+1 = Dt - 1I dg.
3 Hz. (āx - xx),
(15)
Remark D.3. In neighborhood of x", the inverse scaling vector is uniformly upper bounded by
1 e
(16)
ds.
where 1 is the vector all ones.
Definition D.4. (Linear stability). Consider fixed point x" of linearized stochastic dynamic such as (14). We say that x² is linearly stable if there exists constant C' such that
B[|a, =*1 < Cl x² for all 0
Theorem D.5. Assume without loss of generality that x= = 0. Then, the fixed point x* of SASSHA is linearly stable if:
Amaz (II !1 HH nP
2
2 2mpc
EHH
H²)
2n p
EHS - H
ppp² (EH - H²)) 1
2
22
2
18

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
Proof. Our goal is obtain bound of the form E|j Cl for some constant C. We begin by substituting (15)
into E
+++1 1 and proceed to expand the terms as follows:
- H (1 + (l..)) (1 17 diag(
E
||
[x]]
x4 E
(1 diag(
Hgg (1 + H2.)) It
dec
der
(16)
(I - H4. (I + p pH&,)) T (1 -1 Hs. (I + H I
1
1
S E
2n He, (I + p Hg,)
1
x, E
Hgg (I p Hgg, ) 21
2
21 H₂,
2np H
2n p H +
pp"p²
2, E
H
H 1 Dt
e
é²
è
2y H +
" 2npe
2
ppp H
z
H
H
?
2
2mpe
2mpp
p2pp²
(EH H!²))
(EH - 11")) +
(E - -14))
2
2
2
1t (11 ! H
70 21 1
2mpe (EEl 112)) 2n p (B H² + x²p²
(B 111) xt.
22
Since for any x and any matrix A the inequality x T Ax S Amaz (A) ==| with Amaz (A) denoting the maximum eigenvalue of A, holds true, applying this inequality and taking the total expectation yields the following:
1/2 2npe
21pp 2
p2p² (EH -H!) E|
B| 1| S Ammax (1-
- "P H2)2+
(EH)
Hd) +
[
Hd)
c
Recursively applying this bound gives
E||aa|12 = Amax ( (I-"" " H²)
(T-"H-"²H+) + 1p 2npe
(EH? -H2 +
2n
(EH Hd) + pp
(EH H1) E B|ooll
e
2²
e²
2
Here, we can see that x² is linearly stable if
Amaz (1--" 77 c
nP 2 2 + e²
1/2 2mpc
EH22
H +
2nppp p
E
H +
p2pp²
EE H4) $ $1.
E1
22
2
19

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
E. Experiment Setting
Here, we describe our experiment settings in detail. We evaluate SASSIIA against AdaHessian (Yao et al., 2021), Sophia-H (Liu et al., 2024), Shampoo (Gupta et al., 2018), SGD, AdamW (Loshchilov & Hutter, 2018), and SAM (Foret al., 2021) across a diverse set of vision and language tasks. Across all evaluations except for language finetuning, we set lazy Hessian update interval to k = 10 for SASSHA In fact, Sophia-H also supports lazy Hessian updates, but Liu et al. (2024) reports that it achieves the best performance when k- = 1, without lazy updating. Since our goal is to demonstrate that SASSHA exhibits better generalization than existing approximate second-order methods, we compare it with Sophia-H without lazy Hessian updating k= = 1, ensuring that the algorithm is assessed under its optimal configuration.
E.1. Image Classification
We trained ResNet-20 and ResNet-32 on the CIFAR datasets for 160 epochs and Wide-ResNet28-10 10 for 200 epochs. Only standard inception-style data augmentations, such as random cropping and horizontal flipping, were applied, without any additional regularization techniques or extra augmentations. We used standard cross-entropy without label smoothing as a loss function. Also, we adopted a multi-step decay learning rate schedule. Specifically, for ResNet-20 and ResNet-32, the learning rate was decayed by a factor of 0.1 at epochs 80 and 120. For Wide-ResNet28- 10, the learning rate was decayed by factor of 0.2 at epochs 60, 120 and 160. The exponential moving average hyperparameters were set to 31 = 0.9 and 82 = 0.999. All experiments were conducted with a batch size of 256. The hyperparameter search space for each method is detailed in Table 9.
CIFAR
Method
SASSHA
M- SASSHA
Adalleessian
Sophia II
Adam / /SGD
SAM
shampou
Learning Rate
fo3.0.5, 0.03, 0.03..015) 0001)]
(03,0.15, 0.1, 0,000 0.05, 0.015, 0.01, 0.003, 0.001.0..000 0.0000 0.0001
Weight Deeay
1o03, Io-,, Se-5, le-5, 50-6, te-s]
Pestorbation radius p
{01,015,02,025)
001,0203,000003
fom Low.OSSLLLISS. (scagtess)
Clipping-lhnssliolld
Demping
le- 2.3.480.
Hessian Updatte Interval &
10
10
leaming cite schalule
Multi-steg daeay
Table 9, Hyperparameter search space for CIFAR datasets
ImageNet We trained ResNet-50 and plain Vision Transformer (plain ViT) (Beyer al., 2022) for 90 epochs. Remarkably, plain ViT converges in just 90 epochs on ImageNet, attaining performance comparable to the original ViT trained for 300 epochs (Beyer et al., 2022). This faster convergence allows us to efficiently assess whether SASSHA can enhance the generalization in ViT architectures. Consistent with our CIFAR training settings, we applied only standard inception-style data augmentations and used standard cross-entropp as a loss function. For ResNet-50, we adopted a multi-step decay learning rate schedule, reducing the learning rate by a factor of 0.1 at epochs 30 and 60. However, AdaHessian could not be trained with a multi-step decay schedule; therefore, as recommended by Yao et al. (2021), we employed a plateau decay schedule instead. For Vision Transformer training, following Chen et al. (2022), we used a cosine learning rate schedule with 8-epoch warm-up phase. Additionally, the exponential moving average hyperparameters 81 and B2 were set to 0.9 and 0.999 respectively. We used batch size of 256 for ResNet50 and 1024 for VTT. The hyperparameter search spaces for each methods used during training on the ImageNet dataset are detailed in Table 10.
E.2. Language
Language Pretraining Following the training settings introduced in Gomes et al. (2024), we conducted experiments on a mini GPT-1 model using the Wikitext-2 dataset. This scaled-down version of GPT-1 maintains essential modeling capabilities while reducing computational demands. We trained the model with three methods: SASSHA, M- SASHA,, and Sophia-H. The hyperparameter tuning spaces for these methods are summarized in Table 11. For other methods not listed in the table, we directly reported the results from Gomes et al. (2024).
20

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
SASSHA
AdaHessioo
Sophia-H
Adam
Leurning
0..........5
(00.03.0..5
0.15
0.2, 0.02 0.000.0000
Weight Decay
(1e-3,5c-4, 1o-4,5-55 1e-5)
Pertuchation radius
(0..0.15.0022255))
(...0.2.......))
0...11...2........
Hliping--hnsshule
0.1,0.05, 0.005, 0.101, 0.0005, 0.0001
Hessian Update Interval &
10
10
Table 10. Hyperparameter search space for ImageNet
methods
SASSHA/ / M-SASSHA
Sophia-H
SAM
Learning Rate
0.15,, 0.1, 0.03, o.nn a 0.003, 0.0015]
1e-2, Se-4, 1e4, 50-,
11-2, 1o-3,
le4,
166)
Weight Decay
1e:(1.2.3...56667.....
Perturbation radius
2,,31,2,3,4.5
Chipping-thresholld
1e-1,,0-,, S0-2, 5o-4, 10.4)
Hessian Update Interval
Epochs
30
Table 11. Hyperparameter search space for language pretraining
Language Finetuning We utilized a pretrained SqueezeBERT (landola et al., 2020) from the HuggingFace Hub (Wolf et al., 2020). We set the batch size to 16, the maximum sequence length to 512, and the dropout rate to 0. The number of training epochs varied depending on the specific GLUE task: epochs for MNLI, QQP, QNLI, and SST-2: 10 epochs for STS-B, MRPC, and RTE. Additionally, We adopted polynomial learning rate decay scheduler. The detailed hyperparameter search spaces presented in Table 12.
methods
SASSHA M-SASSHA
Sophia-H
AdaHessian
Adam
SAM
Learingg Rate
1o-(1,
2,5, 6,
Weight Decay
10-(1,
4, 7,88
Perturhation radius
1e: (2,3,4,5)
te (1.21,333.5566667..38
Clipping threshold
O 1.0.0 o 0.006, 0.001 0006 00000
Hossian Update Interval &
Table 12. Hyperparameter search space language finetuning
E.3. Label Noise
We introduced label noise by randomly corrupting a fraction of the training data at rates of 20%, 40%, and 60%. Using this setup, we trained ResNet-32 for 160 epochs with a batch size 256. We adopted a multi-step decay learning rate schedule, reducing the learning rate by a factor of 0.1 at epochs 80 and 120. The specific hyperparameters explored during these experiments are detailed in Table 13.
21

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
Methods
SASSHA
M-SASSHIA
Sophiz-Ht H
AdaHessian
SAM
SGD
Learning Rate
0.3,0.15.011,00003 0.000.00.1,00,,...... 0.015, 0.01 (0003,,00001..0 0.0015, 0.001
Weight Decay
II-3, S0-4, 1e-5, So-6, Te-6
Perturbation radius
(0.25,02,015.0.1
{ (0.8, 003.0.0003
0.3. 0.25, 0.15,,0..,11... 0.1, 0.1,0.05,,0022. 0.0000011,,00000000 0.011, 0.011,00002,..000
Clipping-treeshild
Hessian Update Interval k
10
Table 13. Hyperparameter search space for label noise experiments
F. More Ablations
F.I Square Root Function
Table 14. Comparison of square root against damping and clipping.
CIFAR-10
CIFAR-100
ResNet-20
ResNet-32
ResNet-32
Clipping
92.78
93.80
69.47 20:20
Damping
92.74 92.74
93.68
71.27 10.43
Square root (SASSHA)
92.98 =0.05
94.09 +.099
72.14
We conduct an ablation study to support our use of the square rooted preconditioner in SASSHA, comparing other alternatives to stabilize the preconditioner such as damping clipping. We search damping and clipping hyperparameters over (10-1, 10-6, 10-8 10-12) and 10.1, 0.05, 0.01, 0.005, 0.001, 0.0005, 0.0001}. respectively. We note that the square- root employed in SASSHA does not require such extensive hyperparamter search. The results are presented in Table 14.
Our experiments demonstrate that the square rooted preconditioner achieves higher validation accuracy than those with damping or clipping, even with a three times smaller hyperparameter search budget. We provide two possible explanations for this observation. First, taking square root preserves the relative scale among Hessian elements while smoothly amplifying 1). This property is particularly valuable during sharpness
near-zero entries in the denominator (i.e., h vh if 0 minimization, where the overall magnitude of the Hessian components tends to be small. In such cases, even small differences between Hessian elements may carry nontrivial curvature information. Applying Square root can help retain this relative structure while also reducing numerical instability caused by underestimated curvature. In contrast, both damping and clipping modify the Hessian by entirely shifting or abruptly replacing values based on a predefined and fixed threshold criterion. As a result, when the Hessian is generally small due to sharpness minimization, informative dimensions may fall below the threshold, removing potentially critical variations and hence deteriorating the quality of preconditioning. This behavior can also increase the sensitivity the choice of the threshold hyperparameter. Second, square-rooted preconditioner can be interpreted as the result of geometric interpolation between the identity matrix I and H" This interpolation has been demonstrated to enable selecting an optimal preconditioner that balances the bias and the variance of the population risk, thereby minimizing generalization error (Amari et al., 2021). In general, a = 1/2 (i.e., square root) has consistently shown moderate performance across various scenarios (Amari et al., 2021; Duchi et al., 2011; 1; Kingma & Ba, 2015).
22

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
F.2. Absolute Value Function
We observe how the absolute function influences the train- ing process to avoid convergence to critical solution that could result in sub-optimal performance. We train ResNet- 32 on CIFAR-100 using SASSHA without the absolute function (bos) and compare the resulting training loss to that of the original SASSHA We also plot the Hessian eigenspectrum of the found solution via the Lanczos algo- rithm (Yao et al., 2020) to determine whether the found solution corresponds to minimum or saddle point. The results are illustrated in Figure 6. We can sec that with- out the absolute function, the training loss converges to a sub-optimat solution, where the prevalent negative values in the diagonal Hessian distribution indicate it as a saddle point. This shows the necessity of the absolute function for preventing convergence to these critical regions.
5
Sassha No-Abs
Sassha
109
Abs
4-
Density
10-2
10-1
2-
10-6
100 Epoch
150
110
10
Eigenvalue
(a) Train loss
(b) Hess. eigenspectrum
Figure 6. Effect the absolute function on the training loss and the Hessian eigenspectrum of the found solution of SASSHA ResNet- 32/CIFAR-10. Without absolute function, SASSHA converges sub-optimal saddle point.
G. Validation Loss Curve for Vision Task
CIFAR-10 ResNet-20
CIFAR-10 ResNet-32
CIFAR-100 ResNet-32
1.0
1.0
3.0
Saecha
Sanaha
Gendaa
SCDM
SCOM
SCOM
0.8
AdaMessian
0.8
AdaHtsssian
2.5
Alimosion
AdamiN
Adamwv
Loss
Shampoo
Saphlos
108
0.6
Sophio H Shampoo
2.0
Stampoo
0.4
0.4
1.5
0.2
0.2
1.0
50
100
150
50
100
150
50
100
150
Epoch
Epach
Epoch
CIFAR-100 / WRN-28-10
ImageNet/ / ResNet-50
ImageNet ViT-s-32
2.5
3.0
3.0
Sacoha
S0UM
2.0
Adallessan Sophieht
2.5
Asaliamtan Soptrait
Loss
Loss
2.0
Loss
1.5
2.0
1.5
1.0
1.5
0.5
50
100
200
20
40
60
80
Epoch
Epoch
Epoch
Figure 7. Validation loss curve of SASSHA, SGD, AdaHessian, AdamW, and Sophia-H on various image classification models and tasks. SASSHA outperforms first-order and second-order baseline methods.
The experimental results in Figure 7 demonstrate better generalization capability of SASSHA over the related methods. Across all datasets and model architectures, our method consistently achieves the lowest validation loss, indicative of its enhanced ability generalize from training to validation data effectively. This robust performance of SASSHA underscores its potential leading optimization method for various deep learning applications, particularly in image classification.
23

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
H. Comparison with First-order Baselines with Given More Training Budget than SASSHA
We train SGD and AdamW for twice as many epochs as SASSIA and compare their final validation accuracies. The results are presented in Table 15. Despite this extended training budget, these first-order methods fall short of the performance attained with SASSHA, demonstrating their limited effectiveness compared to SASSHA We attribute this outcome SASSHA reaching flatter and better generalizing solution along with stable preconditioning, which together enables consistent outperformance over first-order baselines.
Table 15. Performance comparison of SASSHA against SGD and Adam with twice the epoch allocation. SASSHA achieves better results with significantly fewer epochs.
RN20- CIFAR-I0
RN32 CIFAR-10
RN32 - CIFAR-100
WRN28 100
RN50 ImageNet
ViT_s ImageNer
Ace (epoch)
Ace (epoch)
Ace (epoch)
Ace (epoch)
Acc (epoch)
Acc (epoch)
SGD
92.62 (320e)
93.43 (320e)
69.93 (320e)
80.50 (400e)
75.90 (180e)
63.64 (180e)
A
92.55 (320e)
92.97 (320e)
69.50 (320e)
79.46 (400e)
75.57 (180e)
66.97 (180e)
SASSHA
92.98 (160e)
94.09 (160e)
72.14 (160e)
83.54 (200e)
76.43 (90e)
69.20 (90e)
I. Effectiveness of Stable Hessian Approximations in SASSHA
Table 16. Results of Sophia-H with sharpness minimization.
CIFAR-10
CIFAR-100
ResNet-20
ResNet-32
ResNet-32
WRN-28-10
SAM
92.85
93.89
13.13
40.29
83.14 83.1440.13
Sophia-H H (with SAM)
92.53
93.59
20.31
71.31
80.15
SASSHA
92.98 10.05
94.09
1..24
72.14
10.16
83.54 10.08
We demonstrate limited benefit from naively combining SAM with existing approximate second-order methods without the carefully designed stabilization strategies of SASSHA. Precisely, we compare the validation accuracy of S SASSHA with a simple combination of SAM and Sophia, denoted as Sophia-H (with SAM). We provide results in Table 16.
We observe that Sophia-H H (with SAM) performs worse than SAM, whereas SASSHA outperforms both methods, validating the effectiveness of the design choices made in SASSHA We attribute this to the reduced compatibility of Sophia-H with SAM compared to SASSHA First, Sophia clipping destroys the relative scale between individual elements of the Hessian. This may be particularly problematic when using SAM, where the overall Hessian values tend to be small. In such cases, even very small differences between Hessian elements may carry nontrivial curvature information. However, Sophia clipping abruptly replaces small or negative Hessian values based on predefined and fixed threshold criterion. As a result, when the Hessian is generally small due to SAM, informative dimensions may fall below the threshold, removing potentially critical variations and thereby deteriorating the quality of preconditioning. This situation also raises the sensitivity to hyperparameters like the clipping threshold and makes the optimization process more dependent on careful tuning. Conversely, the stable Hessian approximation in SASSHA, incorporating the absolute function and square rooting, preserves the relative scale among the Hessian entries by smoothly adjusting their magnitudes.
In addition, the use of sophia clipping results in Sophia partially performing signSGD over subset of parameters (Liu et al., 2024), which may lead to suboptimal convergence in typical situations (Karimireddy al., 2019).
24

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
J. Additional Language Modeling Experiments
Table 17. GPT2 pretraining results. SASSHA achieves similar performance to state-of-the-art methods.
In this section, we provide additional language modeling experiments. We train GPT2-small for 50k iterations on OpenWebText (Gao et al., 2020), us- ing various optimization methods for comparison. For AdamW and Sophia- G, we adopt the best hyperparameter configurations reported by (Liu et al., 2024), and set p 0.1 for all SAM-related methods. Due to limited com- putational resources, all experiments were run with single random seed. The results are presented in Table 17. We observe that SASSIIA achieves performance comparable to related methods.
Method
Loss
Ferplexily
AdamW
2.9622
19.353
Sophia-G
2.9307
18,751
SAM Adoo
2.9558
19.196
Sophia-CS G (with SAM)
2.9319
18,773
SASSHA
2.9445
19.015
K. Comparison with Advanced SAM Variants
Thus far, our primary focus has centered on validating the effectiveness of SASSHA in the context of approximate second- order optimization. While this remains the principal objective of our study, here we additionally compare SASSHA with advanced SAM variants (i.e. ASAM (Kwon et al., 2021), GSAM (Zhuang et al., 2022)) to prove that SASSIA is sensible approach. We also evaluate G-SASSHA (S ASSSA with surrogate gap guided sharpness from (Zhuang et al., 2022)) for fair comparison. The results are represented in Table 18.
Table 18. SASSHA v.s. advanced SAM variants in Image classification.
CIFAR-10
CIFAR- 100
ImageNet
ResNet-20
ResNet-32
ResNet-32
WRN-28-10
ResNet-50
ViT-s-32
ASAM
92.96-0.25
93.85 10.
72.02.00.28
83.3920.00
76.54
68.2644.30
GSAM
92.72.0.09
20..6
72.10
83.21
76.45
69.60.n.16
SASSHA
92.98 20.05
94.09
72.14.0.16
83.5410.08
76.43.10.1
69.20
10.200...00
G-SASSHA
92.94
94.15 #0.12
72.18 20.62
83.56
76.66 =0.23
$9.67+0.14
We find that SASSHA competitive with these advanced SAM variants. However, we note clearly that those SAM variants require considerably more hyperparameter tuning to achieve generalization performance comparable to SASSHA For example, GSAM introduces an additional hyperparameter a, demanding much tuning effort as tuning p. Similarly, ASAM, noted by its authors, typically necessitates exploring a broader p range, its appropriate value is approximately 10 times larger than that of SAM. In our setup, tuning GSAM and ASAM involved 4.5x ~ 15.75x and 3x ~ 8x larger search grids compared to SASSHA, respectively. We provide detailed setup and hyperparameter search space below.
Setup and Search space. For ResNet, we use SGD as the base methods for ASAM and GSAM, while for ViT, AdamW with gradient clipping set to 1.0 serves as the base methods. For all models, typical cross entropy loss is used (not label-smoothing cross entropy), and the best learning rate and weight decay of the base methods selected in experiments with ASAM and GSAM. All algorithms are evaluated with constant p (without scheduling). For learning rate schedule, we apply multi-step decay with a decay rate of 0.1 for ResNet on CIFAR, and use cosine learning rate decay with 8 warm-up epochs for ViTs.
ResNeuCIIFR
ResNet/ImagoNet
ViT/ImageNee
{0.01, 0.05, 0.1.0.5..000 0.2, 0.25 0.3, 0.35, 0.4) f0.01, 0.1, 0.15, 0.2,0.25,,000 0.3)
003,00220033040,00,,000
(0.01, 0.1,0.15, 0.2, 0.33
00.01,0.05, 0.05, 0.1, 0.15,.002) 0.2)
0..0,3
Table 19. Hyperparameter search space for GSAM and G-SASSHA
25

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
ResNet/CIFAR
ImageNet
Table 20. Hyperparameter search space for ASAM
L. Additional Label Noise Experiments
Table 21. Robustness label noise. Here measure the validation accuracy under various levels of label noise using ResNect-2 trained on CIFAR-100 CIFAR-10, SASSHA shows much robust performance under label noise.
CIFAR-10
CIFAR-100
Noise level
0%
20%
40%
60%
0%
20%
40%
60%
SGD
02.69
89.91 10.87
87.26.0.40
82.72
69.32
62.18
55.78
45.53.0.78
SAM SGD
93.89=0.ur
92.27 10.14
90.11
85.79 +0.300
71.99
65.53.20.11
61.20=a0.1
51.93:00447
AdaHessian
92.48_0.11
90.11 #0.01
86.88
83. 83.25
68. 68.06
63.06=0025
58.37 10.13
46.02:1.96
Sophia-H
91.99 =0.08
89. 40.93
87.30
82. 48.183
67. 77. #0.37
62.34
56.54_0.24
45.37 #0.27
Shampoo
90.23
88. 48.1440.29
85.1540.61
81.16+0030
64.08
58.85
53.82
42.91
SASSHA
94.09 =0.24
92.49 -0.11
90.29 :0011
86.50.0..0
72.14 :0.16
66.78 00.47
06.78:00.47
61.97 =0.27
53.98 20.37
M. M-SASSHA: Efficient Perturbation
Having explored techniques to reduce the computational cost second-ordes methods, here we consider employing techniques to alleviate the additional gradient computation in sharpness-minin minimizaaiiii..n Prior works have suggested different ways to reduce this computational overhead including infrequent computations (Liu et al., 2022), use of sparse perturbations (Mi et al., 2022), computing with selective weight and data (Du et al., 2022a). In particular, employ the approaches of Becker al. (2024), which uses the normalized negative momentum as the perturbation:
74-1
c = P 11m-11
(17)
which entirely eliminates the need for additional gradient computation with similar generalization improvement as the original SAM. We call this low-computation alternative as M-SASSHA and evaluate this across vision, language, and label noise tasks, did in the main sections. The results presented in Tables 22 to 24, respectively.
Despite having computational cost comparable to first-order methods like SGD and Adam, and significantly lower than approximate second-order methods, M-SASSHA demonstrates superior performance over both first-order and second-order approaches. In image classification, M-SASSIIA proves more effective than the best-performing approximate second-order methods by 2% on CIFAR-100 with ResNet-32 and by 2.5% with ResNet-50, while also exceeding AdamW by approximately 1.6% on VIT. For language pretraining. it attains test perplexity that is 22 points lower than the second- beet performinig Sophia-H H and outperforms AdamW nearly all language tasks. Lastly, M-SASSHA surpasses other methods across all noise levels, proving highly resilient in the presence of extreme label noise. These results reaffirm the effectiveness and consistency of our well-engineered design choices, which enable the stable integration of efficient sharpness minimization into second-order optimization while retaining its benefits.
26

---

Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation
Table 22. M-SASSHA v.s. baselines image classification. M-SASSHA shows superior performance.
CIFAR-10
CIFAR-100
ImageNet
Category
Method
ResNet-20
ResNet-32
ResNet-32
WRN-28-10
ResNet-50
ViT-s-32
SGD Adam W
42.33
92.6940.06
69.32
80.06
75.58+0.05
62.90
First-order
40.04
92.42+0.13
68.78
79.09
75.38
66.46
AdaHessian
92.00
92.48400115
68,00 68.0640.22
76.92
73.6440.16
66.42 4..22
Sophia-H
91.81 ::022
91.99 11.09 10.08
67.70 67.76 0..37
79.35-11.24
72.06 10.49
12.44
Second-order
Shampoo
88.55
90.23
64.08 64.08
74.06 +1.08
M-SASSHA
92.36 -0.23
93.18 #0.30
70.93 +0.21
81.53 =0.27
76.00 0.004
68.04 =0.14
Table 23. M-SASSHA v.s. baselines in language tasks. For pretraining, M-SASSHA achieves the lowest perplexity among all methods. For finetuning, M-SASSHA performs better than AdamW and compares competitively with Sophia-H.
Pretrain/ GPii-mini
Finetune SqpesBBERT
Wikitext-2 Perplexity
SST-2
MRPC Acc
STS-B
QQP Apc
MNLI
QNLI
RTE
Acc
S/V
mat.
A.c
AdhinW
173.06
90:29ga s
88 34.344 is 888.48.0m
80.00 2o. 86 88
82.26
20.93
68 95
Adallessian
407.00 89
89.6 19.64
79.74
/ 88.5.2 3.00
88.08
poos 19.0000 00 L.0
90.37
/ 187 an
61,33; .o
/ 82.08 Lo.se
89 90.94,m io.94
71.00 1:.1
Sophins
187.80
20.44.0.4
16.78
27. 88.53
90.70...oo 47
81.77
88.36.0.23
90.12.0.14
20.76
M-SASSHA
125.01
90 332332...4 as
87.092
990599 2151
88.37 28.37
88.46
90.78 os /8761
81.4220 1o /81 94
80 8.g...
70:40.0
Table 24. Robustness label noise. Here the validation accuracy under various levels of label noise using ResNet-32 trained on CIFAR-100 and CIFAR-10. M-SASSHA shows much robust performance under label noise.
CIFAR-10
CIFAR-100
Noise level
0%
20%
40%
60%
0%
20%
40%
60%
SGD
92.69
89.91
87.2640.40
82. 12.72.11.69
69.32.t0.19
62.18
55.78-20.5
45.53
AdaHessian
92.48
90.11 10.01
86.88.10.44
83. $3.225.m 83.05
68.06....22
63.06
58.37 13.13
46.02.1.96
Sophia-H
91.99
89.93 89.93.0..1
10.01
87.30.10.51
82. 78. 11.43
67. 67.76_10.37
10.37
62.34 62.344...4 10.47
56.54
1..28
45.37 10.27
Shampoo
90.23
88. 14 10.29
85.15
81.16 81.1610.30
64.08 1..46
58.85 1..66
53.82
0.771
42.91
10.99
M-SASSHA
93.18 20.23
91.27 =0.31
88.85
0..31
85.17 20.24
70.93 +:021
66.10 :0.26
61.13 0..28
52.45 20.34
27


Reviews:
[{"review_id": "enr7EAwo4z", "reviewer": "Reviewer_sYMy", "summary": "The paper introduces SASSHA, a second-order optimization method designed to enhance generalization by explicitly reducing the sharpness of minima through a sharpness-aware framework, while stabilizing Hessian approximations via techniques like square-rooting and absolute value transformations. It incorporates lazy Hessian updates to maintain efficiency and demonstrates robustness across diverse tasks, including image classification and language modeling. Empirical results show SASSHA consistently outperforms existing first- and second-order methods, achieving flatter minima and superior generalization performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate the reviewer taking the time to engage with our work. We address the specific points below and would be happy to clarify any remaining concerns.\n\n---\n\n**Theory for improved generalization**\n\nThe relationship between flatness and generalization is theoretically well-established, and many prior studies have shown that flat solutions correlate with better generalization performance [1-3].\nWe are currently developing a theory to prove the improvement of Sassha's generalization. Precisely, we have obtained the result that Sassha finds a flat solution using linear stability analysis [4] (which we will add in the final paper) and plan to use this result and existing flatness-based generalization bounds [3] to establish a generalization theory for Sassha. \nIf you have any other suggestions, please let us know—we will try our best to incorporate them.\n\n---\n\n**Convergence analysis for non-convex neural networks**\n\nThis is indeed a valid limitation. However, convergence analyses under such assumptions are quite standard in optimization literature [6-8]. Moreover, convergence analysis under more realistic conditions, such as non-smoothness, remains a challenge and is being actively studied even for standard deep learning optimizers like SGD and Adam [9].\n\nNonetheless, we intend to analyze convergence properties of Sassha in non-convex settings. To achieve this, we plan to leverage frameworks for preconditioned optimizers that rely on Hessian eigenstructure analysis [10] and convergence analyses under mild assumptions in stochastic settings [11].\n\n---\n\n**Scalability & Large-scale evaluations**\n\nThe scalability of second-order optimization is a valid concern. However, despite such limitations, the broader community consensus is that the benefits provided by second-order methods—such as preconditioning—are potentially significant, and substantial collective efforts have made its computational complexities comparable to first-order methods[6, 12] ($O(n^3) \\rightarrow O(n)$ in time complexity, and $O(n^2) \\rightarrow O(n)$ in memory complexity).\n\nNevertheless, more generally, it is quite natural that leveraging more information for performance improvement can entail increased computations, which is understood as a cost-performance tradeoff. Perhaps more importantly, what matters would be whether or not the tradeoff is reasonable compared to existing alternatives, and precisely in that sense, Sassha sets a solid stand among other second-order baselines.\n\nAlso, we hope that the reviewer understands that the reason we have not evaluated billion-parameter scale models is not due to a fundamental limitation of second-order methods, but rather to enormous resource requirements to train models of such scale that we are unable to afford in our environments. In order to train an 8.3B parameter model using Adam, for instance, NVIDIA has employed 512 V100 GPUs (~16 TiB) [14]. \n\n---\n\n**Hyperparameters**\n\nThank you for sharing your concern. However, we would like to note that the results for Sassha were obtained using values within standard ranges commonly adopted in prior work [4, 13, 17], and Sassha demonstrates strong performance without excessive tuning. We kindly refer you to Appendix D for detailed information on the hyperparameter search spaces across all experimental settings.\n\nAlso, regarding the Hessian update interval, we have already shown that Sassha is extremely robust to this hyperparameter, as demonstrated in Fig. 5(a). Please find a more detailed discussion in Section 6.3.\n\n---\n\n**Source code**\n\nWe have prepared a source code that reproduces all results presented in the paper, and we plan to release it alongside the camera-ready version.\n\n&nbsp;\n\n**Reference** \n\n[1] Jiang et al., Fantastic generalization measures and where to find them \\\n[2] H Petzka et al., Relative Flatness and Generalization \\\n[3] Foret et al., Sharpness-Aware Minimization for Efficiently Improving Generalization \\\n[4] Wu et al., how sgd selects the global minima in over-parameterized learning: a dynamical stability perspective \\\n[5] Shin et al., Critical Influence of Overparameterization on Sharpness-aware Minimization \\\n[6] Liu et al., Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training \\\n[7] Bottou et al., Optimization Methods for Large-Scale Machine Learning \\\n[8] Reddi et al., On the Convergence of Adam and Beyond \\\n[9] Li, Rakhlin & Jadbabaie, Convergence of Adam Under Relaxed Assumptions \\\n[10] Doikov et al., Spectral Preconditioning for Gradient Methods on Graded Non-convex Functions \\\n[11]  He et al., Convergence of Adam for Non-convex Objectives: Relaxed Hyperparameters and Non-ergodic Case \\\n[12] Yao et al., ADAHESSIAN: An adaptive second order optimization for machine learning \\\n[13] Gupta et al., Shampoo: Preconditioned Stochastic Tensor Optimization, ICML, 2018. \\\n[14] Shoeybi et al., Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism", "claims_and_evidence": "In this paper, authors propose a novel second-order method designed to enhance generalization by explicitly reducing sharpness of the solution. There lacks of therotical analysis on why this mathod enhance generalization.", "methods_and_evaluation_criteria": "While SASSHA demonstrates efficiency improvements (e.g., lazy Hessian updates), its evaluation focuses on small-sized models (e.g., ResNets, ViT-s, GPT1-mini). The computational and memory demands of second-order methods like SASSHA may still hinder scalability to modern billion-parameter architectures or distributed training scenarios, which are common in large-scale language/vision models.", "theoretical_claims": "The convergence analysis (Section 4.5) assumes convexity and smoothness, which are unrealistic for non-convex neural networks. No theoretical guarantees connect sharpness minimization to generalization in practical settings.", "experimental_designs_or_analyses": "A more comprehensive ablation study on key hyper-parameters—such as learning rate and perturbation radius—is critical. These parameters likely have a significant impact on model performance, and their sensitivity could undermine the method's robustness if not rigorously analyzed. Without understanding how variations in these hyper-parameters affect results, the practicality and reliability of the approach in real-world scenarios remain questionable.", "relation_to_broader_scientific_literature": "The authors present a novel stabilization framework for Hessian approximations to mitigate loss landscape sharpness.", "essential_references_not_discussed": "N.A.", "other_strengths_and_weaknesses": "weakness.\n\nThough SASSHA reduces tuning compared to methods like SAM, it still introduces new hyperparameters (e.g., perturbation radius, Hessian update interval). The paper acknowledges that lazy updates require careful balancing to avoid performance degradation, suggesting sensitivity to configuration choices that could complicate real-world deployment.", "other_comments_or_suggestions": "Publicly releasing the full source code, accompanied by detailed implementation guidelines and hyper-parameter configurations, is essential to ensure transparency and reproducibility.", "questions_for_authors": "N.A.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sK3HpNV4qA", "reviewer": "Reviewer_Cfv3", "summary": "This paper introduces SASSHA (Sharpness-aware Adaptive Second-order Optimization with Stable Hessian Approximation), a novel second-order optimization method designed to improve generalization performance. The authors investigate why approximate second-order methods tend to generalize poorly compared to first-order approaches, finding they converge to sharper minima. SASSHA addresses this by incorporating a sharpness minimization scheme similar to SAM into a second-order framework while stabilizing Hessian approximation with square-rooting and absolute value operations. The method also enables efficient lazy Hessian updates, reducing computational costs. Across image classification (CIFAR, ImageNet) and language tasks (pretraining and finetuning), SASSHA consistently achieves flatter minima and superior generalization compared to existing second-order methods and often outperforms first-order approaches including SGD and SAM.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for thoroughly reviewing our work and giving us insightful and constructive feedback. While we address the raised questions below, we would be keen to engage in any further discussion as needed.\n\n---\n\n> “How does SASSHA's performance scale to larger models and datasets beyond those presented?”\n\nWe appreciate the reviewer’s suggestion. We provide the validation performance of Sassha on larger models (ViT-B-32) below.\n\n|VIT base|ImageNet|\n|-|-|\n|Metric|val. acc.|\n|AdamW|66.90|\n|SAM(AdamW)|69.18|\n|AdaHessian|66.96|\n|Sophia-H|64.26|\n|Sassha|**69.82**|\n\nWe observe that Sassha outperforms both first- and second-order baselines. We will include this in the updated paper.\nAdditionally, we plan to include larger-scale models such as ViT-L and GPT2-small in the camera-ready version.\n\nNote on configuration. For ViT-B, we used the same hyperparameters from ViT-S due to the limited time frame of the rebuttal.\n\n---\n\n> “Is there a theoretical explanation for why combining sharpness awareness with second-order information works better than either approach alone?”\n\nWe believe that the strong performance of SASSHA stems from the complementary benefits of combining sharpness-awareness and second-order information. Specifically:\n- The flatness induced by sharpness minimization has been shown—both theoretically and empirically—to improve generalization [1–5], and\n- The effectiveness of preconditioning based on second-order information in adapting to the ill-conditioned geometry of deep learning has been well established in theory [6–9]. More recently, it has also been shown that optimal preconditioners can potentially accelerate the decrease of the population risk [10].\n\nThese advantages appear to act synergistically. In contrast, using either technique in isolation may introduce certain limitations (e.g., sharpness minimization alone may struggle with ill-conditioned geometry, while second-order methods alone may converge to sharp minima).\n\n---\n\n> “Additional tuning parameter $\\rho$ (perturbation radius) compared to pure second-order methods” as a weakness\n\nWhile it is true that Sassha requires $\\rho$, this might not necessarily be a fair criticism since (1) pure second-order methods are not applicable for deep learning, and (2) approximate second-order methods require (i.e., the fair baselines for the purpose of this work) their own hyperparameters to mitigate various issues such as training instability; for instance, AdaHessian requires the spatial averaging block size, Sophia requires a clipping threshold, and Shampoo requires a damping factor. It is also worth noting that Sassha is found to be generally robust to the range of $\\rho$ values commonly used for SAM.\n\n\n&nbsp;\n\n**Reference**\n\n[1] Jiang et al., Fantastic generalization measures and where to find them, ICML 2019. \\\n[2] Tsuzuku et al., Normalized flat minima: Exploring scale invariant definition of flat minima for neural networks using pac-bayesian analysis, ICML, 2020. \\\n[3] H Petzka et al., Relative Flatness and Generalization,  NeurIPS, 2021. \\\n[4] Foret et al., Sharpness-Aware Minimization for Efficiently Improving Generalization, ICLR, 2021. \\\n[5] Orvieto et al., Anticorrelated Noise Injection for Improved Generalization, ICML, 2022. \\\n[6] Boyd et al., Convex Optimization, Cambridge University Press, 2004. \\\n[7] Nocedal et al., Numerical Optimization, Springer, 2006. \\\n[8] Bottou et al., Optimization Methods for Large-scale machine learning, SIAM Review, 2018. \\\n[9] Jiang et al., How Does Adaptive Optimization Impact Local Neural Network Geometry?, NeurIPS, 2023. \\\n[10] Amari et al., When Does Preconditioning Help or Hurt Generalization?, ICLR, 2021.", "claims_and_evidence": "The claims are well-supported through extensive empirical evidence. The authors clearly demonstrate that:\n\n- Second-order methods converge to sharper minima (Table 1, Fig. 2)\n- SASSHA achieves flatter minima across different sharpness metrics\n- SASSHA consistently outperforms other methods in generalization (Tables 2, 3)\n- Square-rooting and absolute value operations stabilize training (Figs. 4, 7)\n- SASSHA is robust to label noise (Table 5)\n- Lazy Hessian updates are effective due to lower Hessian sensitivity (Fig. 5)\n\nThe theoretical convergence analysis (Theorem 4.4) provides a sound foundation for the approach.", "methods_and_evaluation_criteria": "The experimental methodology is thorough and appropriate for evaluating optimization algorithms. The authors:\n\n- Compare against relevant state-of-the-art methods (AdaHessian, Sophia-H, Shampoo, SGD, AdamW, SAM)\n- Use diverse tasks (image classification, language modeling, finetuning)\n- Conduct extensive hyperparameter tuning for fair comparison\n- Evaluate across multiple metrics (validation accuracy, loss, sharpness metrics)\n- Analyze robustness, stability, efficiency, and computational cost\n- Provide ablation studies for each component of their method", "theoretical_claims": "I reviewed the convergence analysis in Section 4.5 and the expanded proof in Appendix C. The convergence theorem (Theorem 4.4) follows standard optimization theory for adaptive methods, incorporating both the effects of the perturbation and diagonal Hessian preconditioner. The proof correctly uses smoothness conditions and perturbation bounds. The theoretical analysis is sound within the given assumptions.", "experimental_designs_or_analyses": "The experimental design is robust and thoughtfully constructed:\n\n- Multiple dataset sizes (CIFAR-10/100, ImageNet) and domains (vision, language)\n- Several model architectures (ResNet variants, WideResNet, Vision Transformer, GPT1-mini, SqueezeBERT)\n- Comprehensive hyperparameter tuning detailed in Appendix D\n- Multiple random seeds with standard deviations reported\n- Investigation of label noise robustness\n- Detailed ablation studies for each component", "relation_to_broader_scientific_literature": "The paper effectively positions its contributions within the optimization literature. It builds upon:\n\n- Sharpness-aware minimization (SAM) for generalization\n- Approximate second-order methods (AdaHessian, Sophia-H)\n- Stable Hessian approximation techniques\n- Insights about flat minima and generalization\n\nThe authors clearly differentiate their approach from existing methods and provide comprehensive comparisons.", "essential_references_not_discussed": "The paper has a thorough literature review covering most relevant work. The authors mention recent second-order methods and sharpness-aware optimization approaches comprehensively.", "other_strengths_and_weaknesses": "Strengths:\n\n- Novel combination of sharpness minimization with second-order methods\n- Strong empirical results across diverse tasks\n- Effective stabilization techniques for Hessian approximation\n- Computational efficiency through lazy Hessian updates\n- Thorough analysis and ablation studies\n- Robustness to label noise\n\nWeaknesses:\n\n- Limited theoretical justification for why square-rooting works better than alternatives\n- Mostly empirical validation of design choices\n- Additional tuning parameter $\\rho$ (perturbation radius) compared to pure second-order methods\n- Performance gains on language tasks are more modest than for vision tasks", "other_comments_or_suggestions": "- The figures effectively visualize the differences in loss landscapes\n- The stability analysis provides valuable insights for practitioners\n- The M-SASSHA variant offers an efficient alternative with competitive performance", "questions_for_authors": "1- How does SASSHA's performance scale to larger models and datasets beyond those presented? Have you tried applying it to very large language models or transformer-based vision models?\n\n2- The paper shows SASSHA outperforms SAM in most settings. Is there a theoretical explanation for why combining sharpness awareness with second-order information works better than either approach alone?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qgsv3vVNZ2", "reviewer": "Reviewer_Wdix", "summary": "This paper compares the sharpness and generalisation of solutions found by second-order vs first-order optimisers, observing worse generalisation and larger sharpness of second-order methods.\nTo rescue test performance of second-order optimisers, it proposes an optimization method combining (diagonal) second-order optimization with sharpness-aware minimisation, obtaining better generalisation than first-order (including adaptive) methods.\nA few crucial design choices are made to stabilise training, such as taking the absolute value and square root of the diagonal of the Hessian.\nIt is shown that the Hessian changes during training more slowly with respect to other optimisers, allowing infrequent Hessian updates, that are computationally convenient. \nThe method is tested on image classification and language modelling.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We’re sincerely grateful for the reviewer’s thoughtful engagement and recognition of our contribution. It was encouraging and helped us further refine the work. We provide our responses below and welcome further discussion.\n\n---\n\n**Justification for design choices**\n\nThank you for your comment. We believe square-rooting can be potentially more effective than clipping or damping for two main reasons.\nFirst, square-rooting preserves the relative scale between each element of the Hessian. This property is particularly valuable when the sharpness minimization is underway, where the overall Hessian values tend to be small. In such cases, even small differences between Hessian elements may carry nontrivial curvature information. Square-rooting can help retain this relative structure while also mitigating numerical instability caused by underestimated curvature. In contrast, both clipping and damping operate by abruptly replacing Hessian values based on a predefined and fixed threshold criterion. As a result, when the Hessian is generally small due to sharpness minimization, informative dimensions may fall below the threshold,  removing potentially critical variations and hence deteriorating the quality of preconditioning. This behavior can also make the method more sensitive to hyperparameter choices.\n\nSecond, square-rooting can further be interpreted as a geometric interpolation between a Hessian-based preconditioner and the identity matrix, which, as theoretically analyzed in [1], provides a natural mechanism for balancing between bias and variance. \n\nAdditionally, clipping mechanism in Sophia has been shown to behave like signSGD on certain parameters, which prior studies [2,3] suggest may lead to suboptimal performance depending on the architecture.\n\nWhile developing a precise account behind the benefit remains a challenge, we hope the reviewer sees our contributions in this work given that the proposed method is rigorously evaluated across diverse under fair settings and sets a new state of the art. \n\n---\n\n**Comparison with SAM” section**\n\nWe apologize for the confusion by the current wording of the section.\n\nFirst, the main goal of Section 5.3 is to provide a controlled comparison between Sassha and SAM, demonstrating that Sassha is competitive with—or even outperforms—SAM. In line with this objective, we included an experiment in which SAM is given significantly more training iterations (x2) to show that Sassha still maintains superior performance, even under conditions that are favorable to SAM.\n\nThe addressing of block Hessian heterogeneity was not intended as a central claim of Section 5.3, but rather as one plausible hypothesis —based on prior literature [4,5]—for why Sassha may perform better than SAM on Transformer-based architectures. We did not mean to position this as the primary claim for the observed performance difference.\n\nWe appreciate the valuable feedback and will revise the writing in the final version.\n\n---\n\n**On reported performances**\n\n> “SAM gets 79.1 acc with ResNet50 on ImageNet, while the authors report 76.3”\n\nThis discrepancy is primarily due to the significantly larger number of training epochs used in [6] compared to ours (400 epochs in [6] vs. 90 in our setting). Additionally, we find differences in the experimental setup—such as the learning rate scheduler (cosine in [6] vs. multi-step in ours) and the use of label smoothing in [6]—this complicates a direct comparison of final accuracy.\n\n> “Sam gets 83.7 acc with WRN-28-10 on CIFAR100 ... However, the authors report 82.9”\n\nFirst, we would like to clarify that the SAM (with SGD as the base optimizer) result reported in our paper is 83.14%, not 82.9% as mentioned. Additionally, we were unable to find a reference for the 83.7% accuracy cited by the reviewer. If possible, we would appreciate it if you could share the source. In our own investigation, we found that the performance numbers reported in reference [7] are consistent with our results.\n\nMost importantly, we want to emphasize that all methods in our study were evaluated under an identical experimental setup to ensure fair comparison. Furthermore, we have prepared full configurations and code to reproduce all reported results, which will be released alongside the camera-ready version.\n\n&nbsp;\n\n**Reference**\n\n[1] Amari et al., When Does Preconditioning Help or Hurt Generalization? \\\n[2] Liu et al., Sophia: A Scalable Stochastic Second-order Optimizer for Language Model Pre-training \\\n[3] Karimireddy et al., Error feedback fixes sign sgd and other gradient compression schemes \\\n[4] Zhang et al., Why Transformers Need Adam: A Hessian Perspective \\\n[5] Ormaniec et al., What does it mean to be a transformer? insights from a theoretical Hessian analysis \\\n[6] Foret et al., Sharpness-Aware Minimization for Efficiently Improving Generalization \\\n[7] Wu et al., CR-SAM: Curvature Regularized Sharpness-Aware Minimization", "claims_and_evidence": "The debate on whether second-order methods generalise is ongoing, and this paper provides a significant contribution.\n\nThe combination of second-order optimization with sharpness-aware minimisation seems novel.\n\nEmpirical results are compelling.\n\nSome of the design choices are not well justified, it is really not clear why square rooting should be better than damping and/or clipping.\n\nThe section “Comparison with SAM” is not quite convincing in general. The authors claim that SASSHA is more robust to block heterogeneity inherent in Transformer architectures. Then, they show that SAM underperforms SASSHA, even with more training iterations. I don’t see how this result relates with the claim.", "methods_and_evaluation_criteria": "SAM gets 79.1 accuracy with ResNet50 on ImageNet, while the authors report 76.3. Is it because of differences in data augmentation?\n\nSam also gets 83.7 accuracy with WRN-28-10 on CIFAR-100, using basic augmentation, so that should be the same used by the authors. However, the authors report 82.9. While the difference is small, it would reverse the claim that SASSHA, which gets 83.5, has a better performance.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "OK", "relation_to_broader_scientific_literature": "OK", "essential_references_not_discussed": "OK", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LeA8w5tNFf", "reviewer": "Reviewer_VPX6", "summary": "Combine diag hessian estimate optimizer with SAM/M-SAM", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate the reviewer’s feedback. While we address the reviewer’s specific comments below, we would be keen to engage in any further discussion.\n\n---\n\n> “ViTs should be optimized with Adam or Shampoo not with SGD”\n\nThere seems to be some confusion. We have already provided results for ViT trained with AdamW in Table 2.\n\n---\n\n> “a very baisic resnet can get 94%”\n\nWe would appreciate it if you could provide a specific reference for the claimed 94% performance of a basic ResNet. The paper we are referencing [1] reports results similar to ours.\n\n---\n\n> “This paper is basically just a different version of Flat Sophia” [2]\n\nThank you for providing a pointer. With all due respect, however, we disagree with this assessment and find it to be quite stretched, if not unfair. \n\nFirst, Sassha operates very differently from Flat-Sophia, notably in its sharpness minimization approach [3,4] and second-order techniques (see Appendix B), and such differences are known to yield distinct optimization behavior [5].\n\nMore importantly, however, Sassha is supported by a comprehensive empirical and theoretical study that offers insights and presented as a form of research paper, rendering potentially greater value in a significantly more reliable fashion compared to the suggested pointer, which seems to be a code repository in an idea level without providing any numerical result  [2].\n\nWe sincerely hope that the reviewer sees the value we created in this work to architect a generally well-performing method for standard deep learning settings.\n\n\n---\n\n> “In RL many people have combined SAM and 2nd order opts to boost neuro plasticity. This paper is not novel in this.”\n\nThank you for your comment. We would appreciate it if you could point us to specific references. This would allow us to more accurately compare under fair settings and discuss their relations to our work.\n\nIn the meantime, we hope that the reviewer sees the contributions we made in this work given that the method is rigorously evaluated across diverse standard settings to set a new state of the art, and that leveraging general ideas (i.e., sharpness minimization and second-order optimization) should not necessarily be used as a ground for criticism. \n\n\n\n&nbsp;\n\n**Reference**\n\n[1] Yao et al., ADAHESSIAN: An Adaptive Second Order Optimization for Machine Learning, AAAI, 2021.\\\n[2] https://github.com/evanatyourservice/flat-sophia. \\\n[3] Wang et al., Improving Generalization and Convergence by Enhancing Implicit Regularization, NeurIPS 2024.\\\n[4] Foret et al., Sharpness-Aware Minimization for Efficiently Improving Generalization, ICLR, 2021.\\\n[5] Dauphin et al., Neglected Hessian component explains mysteries in sharpness regularization., NeurIPS 2024.", "claims_and_evidence": "The paper is written clearly. It combines an adaptive(adamized) diagonal 2nd order optimizer with M-SAM. This results in an empirical boost over other optimizers that.", "methods_and_evaluation_criteria": "For imagenet ViT SAM is not SoTA. One should be using Shampoo or Adam. Also SGD is not good for optimizing transformers so the baselines look a bit skewed.", "theoretical_claims": "NA", "experimental_designs_or_analyses": "The experiment baselines did not always make sense. For example, ViTs should be optimized with Adam or Shampoo not with SGD. So that experiment is a bit moot. Also a very baisic resnet can get 94%", "relation_to_broader_scientific_literature": "This paper is basically just a different version of Flat Sophia proposed last year https://github.com/evanatyourservice/flat-sophia\n\nIn RL many people have combined SAM and 2nd order opts to boost neuro plasticity. This paper is not novel in this.", "essential_references_not_discussed": "This paper is basically just a different version of Flat Sophia proposed last year https://github.com/evanatyourservice/flat-sophia", "other_strengths_and_weaknesses": "I am very happy the authors are using Momentum SAM which is basically just SAM but without the extra back prop. https://arxiv.org/abs/2401.12033\n\nI also very much like that they provided the sharpness comparison of the momentum/full sam version of the optimizer. \n\nI dont think combining M-SAM and diagonal 2nd order is novel enough.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a second-order optimizer termed Sassha based on sharpness-aware minimization (SAM).
The author's show how to tackle SAM's min-max problem with curvature information and highlight additional challenges to stabilize the method due to the interaction of driving curvature to zero (flatness), while simultaneously using its inverse to pre-condition the gradient.
Empirically, the method is verified to find flatter minima than other second-order methods.

The paper's quality is good in terms of novelty, experimental design/depth, and presentation:

- The majority of reviewers found combining second-order optimization with SAM to be novel and I agree with this assessment.
- The paper is well written, figures and tables are clean and legible
- Most reviewers agreed that the paper contains extensive experiments, demonstrating that the proposed method reaches flatter minima than second-order competitors, justifying individual design decisions, such as square-rooting the curvature estimate to stabilize the algorithm, and providing sensitivity analyses to various hyper-parameters.
- During the discussion, the authors presented concrete plans for addressing criticisms expressed by the reviewers, such as strengthening the theoretical connections between flatness and generalization. One bigger concern brought up by one of the reviewers were the theoretical motivations for square-rooting the curvature that were rather vague. I believe the authors should follow the suggestions of Reviewer Wdix and justify this decision purely empirically.

After including the comments brought up by the reviewers, this paper is in good shape and I recommend acceptance.
Decision: Accept (poster)