When Flatness Does (Not) Guarantee Adversarial Robustness
Authors: Nils Philipp Walter
Linara Adilova
Jilles Vreeken
Michael Kamp
Venue: ICLR 2026 Poster (2026)
Keywords: Flatness
Adversarial Robustness
Primary area: learning theory

Abstract: Despite their empirical success, neural networks remain vulnerable to small, adversarial perturbations. A longstanding hypothesis suggests that flat minima, regions of low curvature in the loss landscape, offer increased robustness. While intuitive, this connection has remained largely informal and incomplete. By rigorously formalizing the relationship, we show this intuition is only partially correct: flatness implies *local* but not *global* adversarial robustness. To arrive at this result, we first derive a closed-form expression for relative flatness in the penultimate layer, and then show we can use this to constrain the variation of the loss in input space. This allows us to formally analyze the adversarial robustness of the entire network. We then show that to maintain robustness beyond a local neighborhood, the loss needs to curve *sharply* away from the data manifold.
We validate our theoretical predictions empirically across architectures and datasets, uncovering the geometric structure that governs adversarial vulnerability, and linking flatness to model confidence: adversarial examples often lie in large, flat regions where the model is confidently wrong. Our results challenge simplified views of flatness and provide a nuanced understanding of its role in robustness.

Published as a conference paper at ICLR 2026
WHEN FLATNESS DOES (NoT) GUARANTEE ADVERSARIAL ROBUSTNESS
Nils Philipp Walter
Linara Adilova
CISPA Helmholtz Center
Research Center Trustworthy Data Science and Security
for Information Security
TU Dortmund
nils. Salleeree ieterriisppe .ci.pa.de..e
1 linara. linara.
Jilles Vreeken
Michael Kamp
CISPA Helmholtz Center for Information Security vreekenêci spa. de
Lamarr Institute, Technical University Dortmund & Institute for Al in Medicine, University Hospital Essen mi chael., kampåuk-essen
ABSTRACT
Despite their empirical success, neural networks remain vulnerable to small, ad- versarial perturbations. longstanding hypothesis suggests that flat minima, re- gions of low curvature in the loss landscape, offer increased robustness. While intuitive, this connection has remained largely informal and incomplete. By rigor- ously formalizing the relationship, we show this intuition is only partially correct: flatness implies local but not global adversarial robustness. To arrive this result, we first derive a closed-form expression for relative flatness in the penultimate layer, and then show we can use this to constrain the variation of the loss in input space. This allows to formally analyze the adversarial robustness of the entire network. We then show that to maintain robustness beyond local neighborhood, the loss needs to curve sharply away from the data manifold. We validate our the- oretical predictions empirically across architectures and datasets, uncovering the geometric structure that governs adversarial vulnerability, and linking flatness model confidence: adversarial examples often lie in large, flat regions where the model is confidently wrong. Our results challenge simplified views of flatness and provide nuanced understanding of its role in robustness.
INTRODUCTION
Despite their success across wide range of tasks, neural networks remain notoriously brittle under adversarial perturbations. Small, often imperceptible changes to the input can dramatically alter the prediction of model. Understanding the structural properties that contribute to this vulnerability central to building more robust systems. One property that has long attracted attention is the flatness of the loss surface. Earlier work suggested that flatter minima correlate with better gen- eralization (Hochreiter & Schmidhuber] 1994 Jiang et al., 2019), however, the universality of this link remains an open question (Andriushchenko et al. 2023). Flatness also emerged indicator for adversarial robustness (Wu et al.) (200)]: a model whose loss landscape is locally flat in parameter space might resist small perturbations in input space. At first glance. this appears to be disconnected, since adversarial examples concern the change of the loss with respect to the input, while flatness quantifies the change with respect to the weights. Still, simple settings, one can observe a connection. For linear model and a given training sample we have
potential
((((((a + 8x)), v)) ((f(ww + wox), v) <(S((w wo)z), v) ((J(wr))))) v) = (([(w)) v)
The last step follows from the loss surface being flat. That is, by definition the loss does not change under small perturbations. This informal derivation suggests that input perturbations can, in simple cases, be reinterpreted as weight perturbations. However, oversimplifies the nonlinear structure of neural networks. In this work, we resolve this disconnect through a formal theoretical analysis. The main challenge lies in accounting for nonlinearity, which fundamentally alters how perturbations

---

Published as a conference paper at ICLR 2026
Parameter
Feature
Input Space
B
Figure 1: Flatness shapes robustness through the network. Flatness in parameter space (left)
implies a minimum robustness radius feature space (center), which maps back to warped but bounded regions in input space (right). This illustrates how local flatness in the weight space of the penultimate layer translates into localized but non- mon-unffrm robustness in the input space.
propagate through the network. To overcome this, we extend the theoretical results by Petzka et al. 202T who related carefully defined notion of flatness in single layer, based the trace of the hessian Tr(H). ). that binds the robustness to perturbations in feature space. Here, we show that their theoretical framework can be extended to relate the flatness of the loss surface adversarial robust- ness. Intuitively, we show that under aneory-friendly conditions, flatness in feature space constrains how much the loss can change input space. This relationship illustrated in Figure [ point-wise flatness, measured at individual inputs for a fixed parametrization, corresponds to spherical regions in feature space where the loss changes slowly. These map back to regions in input space, potentially warped or elliptical, where the loss also varies gradually, including under adversarial perturbations.
We then make this intuition precise. To this end, we derive a closed-fone form expression for relative flatness in the penultimate layer for the cross-entropy loss. This enables to characterize how local curvature in parameter space influences the variation of the loss in feature space. By bounding the third derivative of the loss, show that flatness at the penultimate layer induces lower bound the radius within which the loss remains approximately constant in feature space. Through the local Lipschitz continuity of the feature extractor, these spherical regions propagate back into the input space, where they become warped and anisotropic but retain their minimal safe diameter. Crucially, the size of these robust regions is dictated directly by the relative flatness at the penultimate layer.
Our derivations and experiments confirm suneesuse relationship between flatness and robust- ness; yet they expose its limitations. The theoretical picture is appealing but incomplete. While is tempting to believe that flatter model is automatically more robust, our results show that this can be misleading. Flatness ensures local robustness around individual points where it measured, but it does not characterize the structure of the loss landscape globally all the data. In particular, observe that first-order adversarial attacks tend to move examples from initially sharp regions into much flatter ones. That is, adversarial examples often settle in flat basins of the loss, which we call the Uncanny Valley (cf. Fig. 4.. To understand this behavior, we revisit our closed-form expression for relative flatness in the penultimate layer. This expression reveals tight connection between local geometry and model confidence: flatness tends to emerge in regions where the model is highly confident. In turn, this helps explain why adversarial examples being incorrect, they often reside in regions of low curvature and high confidence. This local flat- ness backpropagates through the network, influencing sensitivity in earlier layers. This theoretical lens enables us to rigorously invalidate overly simplified claims of the form: Flat minima result in adversarially more robust networks. In summary, we make the following contributions:
appear deceptively despiee
(i) We establish a theoretical connection between relative flatness and adversarial robustness and empirically validate these results.
(ii) We derive a closed-form expression for relative flatness in the penultimate layer, revealing its link confidence and its influence on network sensitivity.
(iii) We introduce and analyze the Uncanny Valley, showing that adversarial examples often lie in flat, vast, high-confidence regions despite being misclassified.
These results provide a clearer understanding of how flatness and adversarial robustness interact. While flatness offers meaningful local guarantees, our theory reveals structural blind spots that still permit the existence adversarial examples. This bridges parameter-space geometry and input- space robustness through both theoretical analysis and empirical evidence.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Adversarial Examples Szegedy ct al. (2014) introduced the notion of adversarial samples. An ad- versarial sample is a subtle perturbation of benign input that remains imperceptible humans but causes a model to make incorrect predictions. Numerous methods for crafting adversarial examples have been developed. Early approaches include the Fast Gradient Sign Method (FGSM) (Goodfel- low et al. 2015), followed by more powerful and widely adopted techniques such as Projected Gra- dient Descent (PGD) (Madry et al. 2017) and the C&W (Carlini & Wagner 2017). Further work ex- panded attack methods (Papernot et al. 2016; Kurakin et al. [017 | Narodytska & Kasiviswanathan
2017 Brown al., 2018 Alaifari et al. 2018 Andriushchenko al. 2020; Croce & Hein| 2020,
2021 typically relying on gradient-based optimization. In general, all deep learning based models,
including large language models are vulnerable input manipulations (Li al., 2020 Zou et al.)
2023
Wei et al., 2024). Because of many established attacks and smoothness of the input space we
here focus on the image domain.
Flatness for Generalization & Robustness The notion of flat minima introduced by Hochre- iter & Schmidhuber (1994). who argued that such solutions correspond to simpler models with bet- ter generalization. This view gained empirical support from Keskar et al. (2017), who showed that small-batch training tends to converge to flatter minima with lower test error, while large-batch train- ing leads to sharper, less generalizable solutions. Subsequent studies linked generalization sharp- ness metrics such as the Hessian trace and maximum eigenvalue (Jiang et al. 2017]. These insights inspired a range of optimization techniques aimed biasing training towards flatter regions of the loss surface. Methods like Entropy- SSD (Chaudhari et al., 2019), Stochastic Weight Averaging (SWA)lzmailoo) Minimization (SAM) (Foret et al. 2021) explicitly or implicitly minimize sharpness and have been shown to improve generalization across various architectures and datasets.
2019; Neyshabur et al.)
al. 2018), SWAD (Cha et al., 2021). and Sharpness-
The relation between flatness and generalization remains topic of debate, however, Dinh et al. (2017) showed that sharpness measured via the loss Hessian is not invariant under reparameterization, un- dermining raw curvature-based metrics. Kaur et al. (2022) and Andriushchenko et al. (2023) found that sharpness often reflects training hyperparameters more than generalization quality, and may not reliably predict out-of-distribution performance. To address these issues, reparameterizzation invariant definitions such as adaptive sharpness (Kwon et al., 202T) and relative flatness (Petzka et al. T) were proposed, with the latter also providing theoretical generalization guarantees. Flatness has also been studied in the context of adversarial robustness. (Stutz et al. (2021) analyzed curvature under fy-bounded perturbations and observed that flatter input-space regions tend to cor- relate with robustness. Subsequent works showed that robust models often lie in flatter regions of the weight landscape (Zhao et al., 2020), and that sharpness awaree aware training alone can enhance
robustness even without adversarial examples (Wu et al., 2020 Wei et al. 2023
3 PRELIMINARIES
We assume a distribution D over an input space X' and a label space y with corresponding proba- bility density function P(X, Y)= P(Y XXP( We consider models f X' y from model class F, and loss functions I y x y + R+.
3.1 CLASSIC DEFINITION OF ADVERSARIAL EXAMPLES AND ROBUSTNESS
An adversarial example is a minimal perturbation r" added to a sample such that E = x' + 7" is misclassified. Formally, it is defined as an optimization problem.
Definition (Prediction-change adversarial example (Szegedy et al., 2014)). Let f Rm {1. k) be classifier, z E [0, 1lTh and E [k] with = f(x) the predicted class. Then
arg min ||r||2 s.t. f(x+r) and x xtre [0, 1)mh
PER
and the perturbed sample & x + r is called an adversarial example.
Conversely, a classifier is said to be adversarially robust at a sample x its predictions remain constant under small perturbations, i.e., there exists no small perturbation such that the label flips.
3

---

Published as a conference paper at ICLR 2026
Definition 2 (Prediction-changg pointwise robustness). A classifier f is said to be ô-robust at X if E BB(z), f(E) = f()) where B5(x) = {5 E X x E1l2 S 6} denotes the closed ball of radius 6 around x.
Definition 3 (Prediction-change dataset robustness). Given a dataset S C X x Y, classifier I is b-robust over if 6-robust at every (x,, y) according to Definition! 2
3.2 LLSS-BASHE DEFINITIONS OF ADVERSARIAL EXAMPLES AND ROBUSTNESS
In this work, connect the geometry of the loss surface with adversarial robustness. The classical definition of adversarial examples (Definition D only incorporates the loss implicitly through the prediction flip, making unsuitable for directly analyzing how robustness relates to the structure of the loss landscape. To address this, we adopt straight-forward modification.
Definition 4 (Loss-change adversarial example ). Let f R 7 {l....,kt .k be a classifier, x e [0, 1]mh, and 1 f(x) the predicted class. Let E: (1,....Ak) h)) . k- 1R, be loss function, and let e > be a threshold. Then
r- = arg min rERm
||r||2 s.I.
E([(x + -)),)))(((()))))))))))))))
and a atre [0,1]
and the perturbed sample & = r is called a (6, -adverrrrra example, where ó ||r||2-
Instead of requiring change in prediction, we require the loss to increase by more than a threshold This retains the core idea of adversarial examples i.c. how small input perturbations have a mali- cious effect on the model, while capturing how perturbations in the input space translate to changes in the loss surface, rather than relying solely on discrete prediction flips. This definition provides a generalization of the classic definition. Any adversarial example from Definition [ tion 4 for some e > O, and by using a conservative >> log(k) for cross-entropy loss smaller than ensure prediction-flip, which recovers the classical criterion. We make this precise in Lemma 10 Analogous to Definition 2 we define robustness against adversarial loss changes. Note that in the low-confidence regime (loss > In 2), flatness is not informative on adversarial robustness, since predictions can flip without changing the loss.
satisfies Defini-
In 2,
Definition (Loss-change pointwise robustness). Let f X' - y be a classifier, é y - RR loss function, and (I, y) E X x Y sample with ((f(x), y) In Given thresholds ó > O (perturbation radius) and > 0 (loss tolerance), we say that f is (6, e)-robust at (z, y) if Ag E
Bz(x)) ((f)))((())))) ((f(x)) y) S
In this setting, robustness requires that small perturbations do not substantially increase the loss. This makes robustness continuous property, where the degree of robustness is controlled by the
choice of c. A smaller c enforces stricter stability, while a larger one allows more tolerance. For example, perturbation may not flip the predicted label but can still reduce the model's confidence, leading to a higher loss. The classical definition overlooks such cases, but the loss-based view captures such vulnerabilities naturally. We now extend the pointwise definition to the dataset level.
Definition 6
dataset robustness). Given dataset S C X x V, a classifier f is (6, c)- robust over S if it is (6, rr-buuttt at every (æ. y) E S according to Definition 5
is easy to see that by choosing e. (S:6) min (e (((((()))) ( ((f(2), v),, one recovers the prediction-fij criterion. This expression captures the smallest loss increase over all 5-perturbations that change the prediction. With this choice, no prediction flip will incur. This loss-based formulation is strictly stronger, though, as it requires small loss under perturbation even for high-confidence Alternatively, allowing to vary per sample recovers the original pointwise definition exactly and offers greater flexibility. For ease of analysis, we focus on the stronger version with a fixed global c.
a stricter, uniform standard across the dataset.
4 WHAT IS FLATNESS MADE OF?
Flatness measures aim to quantify how sensitive a neural network's loss is to perturbations in pa- rameters. Empirical approaches estimate flatness heuristically by applying random perturbations to model weights and observing the resulting loss variations (Keskar et al., 2017). Analytical methods quantify curvature (Hochreiter & Schmidhuber 1994), using the trace eigenvalue spectrum of
4

---

Published as a conference paper at ICLR 2026
the Hessian of the loss. Both approaches, however, expensive to compute in high- high-dimensional settings and offer limited support for analytical reasoning especially eigenvalue- measures, which are harder to manipulate symbolically and less amenable to theoretical analysis.
Among Hcssian-based metrics, the trace is more practical and often used in theoretical analysis due to its simpler structure. However, the raw trace is not invariant to reparameterization and thus does not reliably correlate with generalization (Dinh et al. 2017). To address these limitations, Pet- zka et al. (2021) proposed reepatameterizaaion--vvaaaa flatness measure, relative flatness, derived from a theoretical decomposition of the generalization gap. The measure combines the trace of the Hessian with the norm of the weights in single layer. They show that, under mild technical assumptions, low relative flatness implies a small generalization gap. Importantly, they informally argue that it sufficient to compute this quantity at any individual layer, particular the penulti- mate layer, avoiding the need to analyze the full network. Below, we provide a formal justification for this claim. Empirically, relative flatness also correlates strongly with generalization performance, making it both theoretically grounded and practically informative. Formally, given a network de- composed into a feature extractor Ø : X + IR mapping to the penultimate layer w E RRxdd and a classifier g applying softmax, i.e., fz) g (wo(2)) relative sharpness is defined as,
KTr (r>(w) == (wll2I"((HW,
(1)
where Tr denotes the trace, and H is the Hessian of the loss computed on dataset S wrt w:
ge
H(w, 5) = SS|
Σ
e (o(ww(z)))
(2)
Ow,Ow,
(2.y)ES
s.jE(km)
This simplified measure (Petzka al., 2019) upper bounds full relative flatness (Lemma A.2 Han iet al. (025). Counter-intuitively, this measure is small when the loss surface is flat, since in that case the trace of the Hessian is small. Similarly, a large value of (w) indicates that the loss surface is sharp. To avoid confusion, we therefore call RTy (w) relative sharpness.
To understand the geometric structure of relative sharpness and enable the theoretical analysis in Section 4, we need an explicit formula for the Hessian. We therefore derive a closed-fon expression of the Hessian for the cross-entropy loss, which reveals the underlying geometry and serves as the foundation for our theoretical analysis. The full derivation is provided in Appendix C.4 For single example (ær, y), defining the softmax predictions as ý = g g(w((x)), our derived expression is
H(w, {(x, v)}) = (diag "g" @ o0²
(3)
yielding simple analytical formula,
k
d
(w) = [|w|laTT(H( ((, (())))))) = ||l|| Σ 9z (1 0,) j=1
(4)
Thus, the geometry as measured by relative sharpness consists of three intuitive components: the network's confidence in terms of softmax output (captured by 0 11--0 -;)), the scale of the feature representation (of)), and the weight magnitude ([]w]]2)) Although this derivation only applies to the penultimate layer, we will demonstrate that this geometry propagates to earlier layers. This analytical form is key bounding adversarial susceptibility using relative sharpness in Section 5
Why the last layer is sufficient For any recitified affine neural network composed of linear, convolutional and BatchNorm layers, curvature propagates cleanly through the architecture. Let fi(x)) = ReLU(W,zz) denote the l-th layer, with pre-activations a' W,r''11 Define the gradient and Hessian of the loss l with respect to these pre-activations as g' = at àé/Oa' /d' and H' = 8 { Since ReLU has zero second derivative almost everywhere, and each affine map introduces no addi- tional curvature, the chain rule yields the recurrence
Hi-1 = W,T D' H'
where D' diag({La>>)) masks the active units. The Hessian block with respect to the weights of layer / takes the coonecker-factoree form
Hw, (at- - (D'H'D).
5

---

Published as a conference paper at ICLR 2026
If the final-layer Hessian vanishes, do all earlier activation- and weight-space blocks. Therefore, the curvature profile of the entire network is fully determined by the last layer, and any sharpness measure based on the Hessian attains its maximum there. We formalize this result in Appendix C.4
This structure reveals that all curvature in hidden layers inherited from the output layer. In partic- ular, if HL 0, then by induction all downstream Hessians also vanish. Even when H & 0, the conditioning of every earlier block is shaped by it. As a result, any sharpness measure based on the Hessian strongly influenced by the final layer and its immediate input, justifying our focus on the penultimate layer in what follows. We formalize this result in Appendix B
5 CONNECTING RELATIVE SHARPNESS TO ADVERSARIAL ROBUSTNESS
In this section we make explicit how relative sharpness influences adversarial robustness. To achieve this, we first connect input perturbations to changes in the output, then relate these changes feature space to parameter perturbations, i.c., relative sharpness. Second, use Taylor expansion around adversarial examples to bound the loss increase under such perturbations. Finally, we derive the robustness bound that reveals the role of relative sharpness. This precisely shows how relative sharpness influences the relationship between input perturbations and loss variations, though the resulting bound is not intended as practically tight guarantees. As the derivations technical, present the key ideas and proof sketches here and defer full details to the appendix.
Connecting input to feature space We first formalize how input-space perturbations propagate into feature-space representations. This is essential since neural networks transform perturbations nonlinearly, complicating the imput-to-paramect parameter relationship. Formally, consider a model f(z) = g( ((w((z)) with feature extractor o. classifier 9, and weight matrix Such model nonlinearly relates input perturbations to those in feature space (Petzka et al.] 2021). Specifically, we represent the adversarial representation (b{) a perturbation of the clean representation ((x)).
Lemma 7. Let f = ((wó(z)) be model with ø L-Lipschitz and [[((x)] 2 т, and E, E X with || - æ|| I 6, then there exists > 0 with A S Lör-1, such that ((E) = Φ(x) + AAΦ(1), where A is orthogonal matrix.
The proof is provided in Appx. Thus in the weights w of the representation layer, for which we use the linearity argument from Petzka et al. (2021) Since can express Φ{E) as <(x) + AAØ(z), we can then use the linearity of the representation layer to relate the perturbation of input to a perturbation weights. That is,
relate the adversarial example & to perturbations
{(f(E). v) = ((((w(())) 3) = ((((w((()) + AAo(z))) y) = ((g((w + AwA))(z).))
This means that we can express an adversarial example as a suitable perturbation of the weights w of the representation layer, where the magnitude is bounded by A < L8r-1
Bounding loss increase Next, we bound the loss difference between & and æ. For case of notation, we define f(w + Aw.) == ((g((w AwA))(z)).)))) The Taylor expansion of f(w + AwA) at w gives the following
{(w + Aw) = ((w) + AAwA, V Vwl(w)) +
2 (wA, R2(w, A)
where H((w) the Hessian of ((w). If we now maximize over all A with ||-A|| S 1, follows that = ni (w). Therefore, we have
A, H((W) = Ir
A NT, (w) + R2(w,
(e(f()))) - =
(5)
The remainder depends the partial third derivatives of the loss. We show in Appdx. C.2 that for feature extractor ø that is I.-Lipschitz, it can be bound by 133 With this we can bound the difference between the loss suffered on an adversarial example & and the loss on clean example for a converged model as follows.
This does hold for transformer since attention can introduce curvature.

---

Published as a conference paper at ICLR 2026
Proposition 8. For (r.y) X x y with ||||| < 1 for all E X, model f(æ) o(wo(x)) a minimum w E R mm with o L- L-Lipschitz and [[
| 2 7, and the cross-entropy loss ((w) =
e(g(wó(z)),)) v) of f on (z,y), holds for all E X with 2æ El2 6 that
83
{(f(E) - ([f(2)) v)
L L' 2,2
kmL
24r
We defer the proof to Appx. C.2 This analytical result clarifies explicitly how relative sharpness controls loss sensitivity perturbations.
Bounding adversarial robustness We now derive an explicit expression for the maximum pertur- bation radius 5 within which the loss increase remains bounded by given threshold Solving the Taylor-based bound for ó yields cubic equation with both linear and quadratic terms. While this equation does not admit clean closed-form solution in general, we isolate and present the dominant terms to illustrate the key dependencies:
Proposition 9. [Informal] For a dataset S C X x y with |||||| I for all x E X, model f(z) = g(w((z))
minimum E 12 wrt. S, with o L-Lipschitz and [[(())] 2 r, and the cross-entropy loss f(w) = {(((w(())) v) of f on (x, y), d being the L2-distance, and e >> 0, is (c, -rouuutstt against adversarial examples with
kkm L (w)
24c
8 x
lim 8 8= L NT- (W) ++
(w)
kmL3
Hence decreasing sharpness enlarges the certified radius, yet the bound does not blow up; it satu- rates at the value above.
The formal version and full proof are provided in Appendix C.3 Proposition 9 precisely reveals how relative sharpness constrains the extent to which the loss can change under input perturba- tions. Beyond its analytical role, it aligns with geometric intuition: flatter models in the sense of lower (y)( together with smooth feature maps, exhibit locally stable behavior. Interestingly, the explicit formula for relative sharpness in Equation 4 reveals that sharpness decreases as model confidence increases. When the predicted class probability approaches one, the corresponding term (1 új) vanishes, reducing the overall sharpness. This implies that highly confident predictions are, at least locally, more robust to input perturbations than uncertain ones. This stands in contrast to the common claim that high- sigh-coofidence predictions are cause for adversarial susceptibility. Instead, analysis suggests that high-confidence regions may actually promote local stability by flattening the loss surface. We revisit and empirically validate this observation in Section
6 FROM THEORY TO PRACTICE
In this section, we empirically investigate how relative sharpness influences adversarial robustness. To test the theoretical predictions from Section 4, exploit a key insight: by scaling the penultimate layer weights w by a factor s i.c. w, = sw, we can directly control the loss surface curvature without retraining- Smaller scaling factors yield sharper networks, while larger values produce flatter ones. This method allows us to precisely control sharpness in systematic and reproducible manner, in contrast to SAM-based training approaches.
We design three experiments to examine different aspects of the sharpness-robustness connection. First, we examine whether the curvature properties the penultimate layer propagate throughout the network, validating our derivations in Section 4 and our focus on this layer as representative of the global loss landscupe. Second, we test our theoretical insight from Proposition 9 by examining whether networks with reduced sharpness demonstrate smaller loss changes when subjected to ad- versarial perturbations, directly validating whether flatter networks exhibit improved stability under input perturbations. Additionally, we show that adversarial training significantly increases the guar- antecable radius. Third, we investigate potential limitations of using relative sharpness as a global robustness and correctness indicator given its close connection to model confidence.
General setting. We evaluate our theoretical predictions on standard architectures and datasets. We train 18 (He al. 2016), W2deResNet-28-4 (Zagoruyko & Komodakis 2016).
7

---

Published as a conference paper at ICLR 2026
DenseNet- 121 (Huang al. 2017), and VGG-11 with BatchNorm (Simonyan & Zisserman 2014 on both CIFAR-I0 10 and CIFAR-IO0. All models are trained using stochastic gradient descent for 100 epochs with an initial learning rate of 0.1, cosine learning rate scheduling, and weight decay of 10-1. For clarity of presentation, we focus our main analysis on ResNet-18 trained on CIFAR-10. Our results are consistent across all architectures and datasets; hence report on this standard combination. Complete results for other architectures and CIFAR-100 are provided in Appendix D
6.1 SHARPNESS PREDICTS LOss SENSITIVITY UNDER ADVERSARIAL PERTURBATIONS
We now empirically test our core claim that relative sharpness governs local loss sensitivity and shapes adversarial robustness. By scaling the network post-hoe, we isolate the effect of sharpness while keeping all model parameters fixed, enabling controlled investigation of its role. While flatten- ing the network does not eliminate adversarial examples entirely, significantly affects how rapidly the loss increases in their vicinity. Specifically, we examine how local sharpness influences the magnitude of loss increase as defined in Definition 4
Setup. To this end, generate adversarial examples using a weaker attack PGD-L2 (25 steps, c 0 step size a = 0.001). This attack incrementally increases perturbations by 0.001 per iteration, achieving robust test accuracy of 90.33% on the original model. We evaluate scaled networks with scaling factors E {0.25, 0.5, 1, 2.5, 5, 10, 50) along the trajectories of these attacks, measuring how the loss evolves. Results are summarized in Figure with subset of scaling factors presented for clarity. Comprehensive results for all scaling values are provided in the appendix.
Loss Increase Analysis. Figure(a the histogram of loss increases between clean examples 2o and final adversarial iterates I25- As predicted by Equation 5 increasing the scale factor (flattening the network) substantially reduces observed loss increases, approaching zero for large scaling values. This occurs because flattening creates locally stable regions or basins-around inputs.
Sharpness Creates Basins. Figure 2b illustratees this phenomenon for a representative example. As we increase the sclaing s. the loss remains nearly flat over progressively larger distances along the attack trajectory. For small scaling factors (e.g., = 0.5), the loss increases steadily from the first iteration, while for large factors (e.g., 50), the loss stays almost constant throughout the entire trajectory until a sharp take-off point where it suddenly strongly increases, which marks the bound- ary of the basin. Specifically, flatter loss surfaces, derived in Proposition 9. can progressively guarantee larger adversarial robustness radii for given e, but only up to the take-off point, which represents the effective limit of sharpness when exclusively enforced by reducing tr(H).
Per Sample Basin. The location of the take-off point varies across samples, as shown in Figure 2c In other words, the width of the basin is sample-dependen and consequently, so is the guarantecable radius. Following conventional wisdom in the field, we observe that basin width strongly correlates with the sharpness measured at the clean input-specifically, flatter networks (lower sharpness) yield broader basins. This relationship is empirically confirmed in Figure 2d This observation aligns with the established intuition that reduced sharpness increases robustness, though within radius that remains insufficient to prevent label flips.
Normalized Loss
60
25
Trequency
-1,500
0.25
0.8
0.6
40
Basin widhh
20
1,000
0.4
20
10
500
0
0-
0
5
10
15
20
25
10 15
20
25
100 200
Loss increase
lterations
Iierations
Relative Sharpness00
(a)
(b)
(C)
(d)
Figure 2: Loss geometry. (a) We report the distribution of the loss increase for varying scaling
values. (b) We show for one example how the basin formes as we increase the scaling. We use the normalized loss to plot all in one axis. (c) We show examples of samples exhibiting different basin widths. (d) We plot the sharpness measured at the test data (s 1) and the basin width.
8

---

Published as a conference paper at ICLR 2026
6.2 ADVERSARIAL TRAINING INCREASES BASIN WIDTH
We also run the same evaluation as in Section 6.1 with an adversarially trained ResNet. For training, use standard adversarial training with PGD- (10 steps, e 8/255, step size a = 2/255). Given that this will result in a more adversarially robust model, we use a stronger attack as in the previous section, namely PGD-&2 (50 steps. 0.5, step size a = 0.01). This attack explores radius 20x bigger radius than compared to the weaker attack. We present the results in Figure
In general, we observe trends that are consistent with all other models and datasets. Specifically, as we scale the model, the loss increase diminishes, and we observe a similar scaling pattern and correlation between sharpness@0 and basin width. The primary distinction that the basin width is significantly larger (approx. 20x) because we measure distance in iterations. Consequently, the classification remains constant within a larger radius, even though the local curvature measured by Tr(H) remains comparable (e.g., 0.6 clean vs. 1.0 adversarial). As shown in Equation 4 the small disparity stems from the fact that adversarially trained networks are significantly less confident.
Normalized Loss
50-
50
Trequency
11,000
0.25
0.8
40-
Basin width
40
0.6
30-
500
0.4
20
20-
0
10
20
30
10
50
5
10
15
20
25
200
600
Loss increase
Iterations
lierations
Relative Sharpness000
(a)
(b)
(c)
(d)
Figure 3: Loss geometry for adversarially trained RESNET-IE 18 (trained via PGD-L 6 8/255)
evaluated under a PGD-&z attack (E = 0.5, a 0.01). (a) We report the distribution of the loss increase for varying scaling values. (b) We show for one example how the basin forms as we increase the scaling. We use the normalized loss to plot all one axis. (c) We show examples of samples exhibiting different basin widths. (d) We plot the sharpness measured at the test data (s = 1).
6.3 SHARPNESS CAN BE DECEIVING
In the previous experiments, we validated our theoretical derivations showing that relative sharpness governs local loss sensitivity, with flatter networks exhibiting limited loss vari- ation under perturbations. We now turn a more nuanced question. The flatness literature carries implicit hope that geometric properties can linked to correctness-thal flatter minima naturally correspond to regions where the model gen- cralizes better and makes accurate predictions. This hope is the implicit motivation of seeking flatter minima. To investigate this, we measure relative sharpness (the same holds for com- mon classification losses, see Lemma [ Lemma 12p along PGD- attack trajectories (10 steps, 8/255, a 2/255) examine the evolving geometry.
Rolative sharpnos
40
30
0.6
20
0.1
0.2
10
0-
0
1
3
5
7
11
Attack iteration
Figure 4: Uncanny Valley, On the PGD-trajectory, sharpness quickly peaks and then decreases zero, while loss keeps increasing.
We plot the results in Fig 4 which reveals a striking phenomenon: as the adversarial attack pro- gresses, sharpness initially iereasses as expected. then, however, dramatically decreases near- zero values while the loss platcaus. This demonstrates that adversarial examples settle in exception- ally flat regions. Our closed-form expression for relative sharpness explains this uncanny valley; in high-confidence regions, confidence dominates the geometry, creating flat valleys where gradients nearly vanish. This challenges the hope that flatness indicates correctness-thes regions are geomet- rically stable yet correspond to confident misclassifications. The sharp peak in relative sharpness marks the decision boundary, where model uncertainty creates maximum curvature. The adversarial attack traverses this ridge before descending into flat, high-confidence regions the wrong side. This reveals how confidence shapes the loss surface geometry-shaee at decision boundaries, flat in confident regions-regardlens of correctness.

---

Published as a conference paper at ICLR 2026
7 DISCUSSION & CONCLUSION
In this work, we formalized the long-suspected connection between flatness and adversarial ro- bustmess, revealing both the promise and fundamental limitations of sharpness as a mediator for robustness. Our analysis bridges parameter-space and input-space perspectives through rigorous theoretical derivations and extensive empirical validation. Traditional prediction-flip definitions of adversarial examples proved insufficient for this analysis, so we introduced a loss-based formulation (Definition 4) that enables direct analysis of how perturbations propagate through the loss landscape. This approach connects the geometric properties at the penultimate layer to adversarial vulnerability in input space. We extended (Petzka al., 202T framework to the adversarial setting, deriving plicit bounds (Proposition 9) that show how relative flatness constrains the adversarial perturbation radius on the points measured on. The robustness guarantee scales as e /1/3 /KTT (w)1/8, provid- ing rigorous characterization of parameter-space geometry influences adversarial robustness.
Our empirical investigations confirmed these theoretical predictions while exposing the myopic na- ture of flatness-based robustness. Through controlled experiments, we showed that while flatter networks exhibit larger stability basins around individual inputs, these basins remain insufficient for practical adversarial robustness. The basins have finite width with sharp boundaries that act as "slingshots," " rapidly increasing the loss once crossed. We observe a string phenomenon; starting from relatively flat regions, attacks traverse increasingly sharp areas before descending into ceptionally flat valleys. The peak corresponds to the decision boundary where predictions flip i.e maximal uncertainty. Beyond lie broad broad flat regions, termed the Uncanny Valley, where models are maximally confident yet wrong. This shows that confidence manifests in the geometry of the loss surface and that sharpness can be deceiving.
Our decision to analyze the network in two parts parts-feature extractor and classifier- proved crucial for understanding the true nature of flatness in neural networks. By focusing our theoretical analy- sis on the penultimate layer, we could isolate how classifier confidence dominates standard flatness measures. This decomposition revealed that Tr(H))-basd metrics conflate two different properties: the geometric stability of learned representations and the confidence of the final classifier. When measuring flatness over the entire network, high confidence in the classifier can mask the true sen- sitivity of the feature extractor. Our approach demonstrates that meaningful robustness evaluation requires separately assessing these components. While the penultimate layer analysis might initially seem like limitation, it actually enabled us uncover how confidence effects propagate through the network and shape the global loss landscape.
In conclusion, by extending relative flatness to the adversarial setting, have derived explicit bounds that confirm flatness does influence robustness-but only through local mechanisms that cannot extend globally. Our work demonstrates that flatness provides robustness within local basins whose size is fundamentally constrained by the confidence-flatness coupling. As long as flatness measures at the penultimate layer are dominated by the confidence term ú, - Úz). they cannot distinguish between true geometric robustness and mere confident predictions. This helps explain why optimizing for flatness, while beneficial, has proven insufficient for achieving strong adversarial robustness. While Tr(H)-based flatness works as local robustness indicator, achieving global robustness requires developing new notions of flatness that meaningfully separate the contributions of the feature extractor from classifier confidence.
REFERENCES
Rima Alaifari, Giovanni S Alberti, and Tandri Gauksson. Adef: an iterative algorithm to construct adversarial deformations. In International Conference on Learning Representations, 2018.
Jean- Cean-Baptiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli. Are labels required for improving adversarial robustness? Advances in Neural Information Processing Systems, 32, 2019.
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square at- tack: A query-efficient black-box adversarial attack via random search. In European Conference on Computer Vision, pp. 484-501, 2020.
10

---

Published as a conference paper at ICLR 2026
Maksym Andriushchenko. Francesco Croce, Maximilian Müller, Matthias Hein, and Nicolas Flam- marion. A modern look at the relationship between sharpness and generalization. In International Conference on Machine Learning, pp. 840-902, PMLR, 2023.
Sue Becker and Yann LeCun. Improving the convergence of back-propagation learning with sec- ond-order methods. David Touretzky, Geoffrey Hinton, and Terrence Sejnowski (eds.), Proceedings of the 1988 Connectionist Models Summer School, pp. 29 29-37, San Francisco, CA, 1989. Morgan Kaufmann.
Christopher M. Bishop. Exact calculation the hessian matrix for the multilayer perceptron. Neural Computation, 4(4):494-501, 1992. doi: 10.1162/neco.199 (ttp://eco.19924.4444. 1992.4.4.494,
Tom B. Brown, Dandelion Mané, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial Patch, May 2018. NIPS 2017, Long Beach, CA, USA.
Qi-Zhi Cai, Chang Liu, and Dawn Song. Curriculum adversarial training. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pp.
2018.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pp. 39-57. IEEE, 2017.
Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. Advances in neural information processing systems, 32, 2019.
Junbum Cha, Sanghyuk Chun, Kyungjae Lee, Han-Cheol Cho, Seunghyun Park, Yunsung Lee, and Sungrae Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Infor- mation Processing Systems, 34:22405-22418 2021.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019(12): 124018, 2019.
Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing. In International Conference Machine Learning, pp. 1310-1320, 2019.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free free attacks. In International conference on machine learning, pp. 2206- 2216. PMLR, 2020.
Francesco Croce and Matthias Hein. Mind the box: l_lappdd for sparse adversarial attacks on image classifiers. International Conference on Machine Learning, pp. 2201-2211. PMLR, 2021.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In International Conference on Machine Learning, pp. 1019-1028. PMLR, 2017.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur, Sharpness-awand minimiza- tion for efficiently improving generalization. In International Conference on Learning Represen- fations, 2020.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. In 9th Int. Conf. on Learning Representations (ICLR), 2021.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. ICLR, 2015.
Sven Gowal, Sylvestre-Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A Mann. Improving robustness using generated data. Advances in neural information processing systems, 34:4218 4233, 2021.
Rajdeep Haldar, Yue Xing, and Qifan Song. Effect of ambient-intrine dimension gap on adversarial vulnerability. In International Conference on Artificial Intelligence and Statistics, pp. 1090-1098, PMLR, 2024.
11

---

Published as a conference paper at ICLR 2026
Ting Han, Linara Adilova, Henning Petzka, Jens Kleesiek, and Michael Kamp. Flatness is necessary, neural collapse not: Rethinking generalization via grokking. In Advances Neural Information Processing Systems, 2025.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Matthias Hein and Maksym Andriushchenko. Formal guarantees the robustness of classifier against adversarial manipulation. In Advances in Neural Information Processing Systems, pp. 2266-224 2276, 2017.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learn- ing can improve model robustness and uncertainty. Advances in neural information processing systems, 32, 2019.
Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. Advances in neural information processing systems, 7, 1994.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4705 2017.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint a8X 2018.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantas- tic generalization measures and where to find them. In International Conference on Learning Representations, 2019.
Sckitoshi Kanai, Masanori Yamada, Hiroshi Takahashi, Yuki Yamanaka, and Yasutoshi Ida. Rela- tionship between nonsmoothness in adversarial training, constraints of attacks, and flatness in the input space. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Simran Kaur, Jeremy Cohen, and Zachary Chase Lipton. On the maximum hessian eigenvalue and generalization. In / Can't Believe It's Not Better Workshop: Understanding Deep Learning Through Empirical Falsification, 2022.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batd batch training for deep learning: Generalization gap and sharp minima. In Proceedings of the 5th International Conference on Learning Representations (ICLR), 2017.
Nupur Kumari, Mayank Singh, Abhishek Sinha, Harshitha Machiraju, Balaji Krishnamurthy, and Vineeth Balasubramanian. Harnessing the vulnerability of latent layers in adversarially trained models. In Proceedings of the 28th International Joint Conference Artificial Intelligence, pp- 2779-2785, 2019.
Alexey Kurakin, lan J Goodfellow, and Samy Bengio. Adversarial machine learning scale. In Proceedings of the International Conference on Learning Representations, 2017.
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. Asam: Adaptive sharpness-
aware minimization for scale-invariant learning of deep neural networks. In International Con- ference Machine Learning, pp. 5905-5914, PMLR, 2021.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. Bert-attack: Adversarial attack against bert using bert. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 6193-6202, 2020.
Tengyuan Liang, Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metrie, ge- ometry, and complexity of neural networks. In The 22nd international conference on artificial intelligence and statisfics, pp. 888-896. PMLR, 2019.
12

---

Published as a conference paper at ICLR 2026
Youwei Liang and Dong Huang. Large norms of cmn layers do not hurt adversarial robustness. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 8565-8573, 2021.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. arXiv preprint arXiv:1706.06000 060833 2017.
James Martens and Roger Grosse. Optimizing neural networks with kronecker- approximate curvature. In Proceedings of the 32nd International Conference on Machine Learning (ICML), 2015.
Odelia Melamed, Gilad Yehudai, and Gal Vardi. Adversarial examples exist in two-layer relu net- works for low dimensional linear subspaces. Advances in Neural Information Processing Systems, 36:5028-500 2023.
Seyed-Mohsen Moosavi-Dezfooli; Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard. Ro- bustness via curvature regularization, and vice versa. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9078-9086 2019.
Preetum Nakkiran. Adversarial robustness may be at odds with simplicity. arXiv preprint arXiv: 2019.
Nina Narodytska and Shiva Kasiviswanathan. Simple Black-Box Adversarial Attacks on Deep Neu- ral Networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1310-1318 18, Honolulu, HI, USA, July 2017. IEEE. ISBN 978-1-5386-0733-6 doi: 10. 10.11099CVPR...217..72
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring general- ization deep learning. Advances neural information processing systems, 30, 2017.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In 2016 IEEE European sympo- sium on security and privacy &EuroS&P),, pp. 372-387. IEEE, 2016.
Julien Perolat, Mateusz Malinowski, Bilal Piot, and Olivier Pietquin. Playing the Game of Universal Adversarial Perturbations, September 2018. arXiv:18099 109.0780 [cs, stat].
Henning Petzka, Linara Adilova, Michael Kamp, and Cristian Sminchisescu. A reparameterization invariant flatness measure for deep neural networks. In Science meets Engineering of Deep Learn- ing workshop at NewrIPS, 2019.
Henning Petzka, Michael Kamp, Linara Adilova, Cristian Sminchiseseu, and Mario Boley, Relative flatness and generalization. Advances in neural information processing systems, 1820-424-8, 2021.
Rahul Rade and Seyed-Mohsen Moosavi-Dezffe Dezzooll. Helper-based adversarial training: Reducing excessive margin to achieve better accuracy vs. robustness trade-off. In ICMI. 202/ Workshop on Adversarial Machine Learning, 2021.
Aditi Raghunathan, Sang Michael Xie, Fanny Yang, John Duchi, and Percy Liang. Understanding and mitigating the tradeoff between robustness and accuracy. In International Conference on Machine Learning, pp. 7909-7919, PMLR, 2020.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International conference on machine learning, pp. 8093-8100 8104. PMLR, 2020.
Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. Ad- vances in neural information processing systems, 32, 2019.
Ludwig Schmidt, Shibani Santurkar, Dimitris Tsipras, Kunal Talwar, and Aleksander Madry. Ad- versarially robust generalization requires more data. Advances in neural information processing systems, 31, 2018
13

---

Published as a conference paper at ICLR 2026
Vikash Sehwag, Shiqi Wang, Prateck Mittal, and Suman Jana. Hydra: Pruning adversarially robust neural networks. Advances in Neural Information Processing Systems, 33:19655- 19666, 2020.
Ali Shafahi, Mahyar Najbb,, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Ad- vances in neural information processing systems, 32, 2019.
Ali Shafahi, Mahyar Najibi, Zheng Xu, John Dickerson, Larry S Davis, and Tom Goldstein. Uni- versal adversarial training. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 5636-5643, 2020.
Adi Shamir, Odelia Melamed, and Oriel BenShmuel. The dimpled manifold model of adversarial examples in machine learning. arXiv preprint arXiv:2106. 10151, 2021.
Carl-Johann Simon-Gabriel, Yann Ollivier, Leon Bottou, Bernhard Schölkopf, and David Lopez- Paz. First-order adversarial vulnerability of neural networks and input dimension. In International conference on machine learning, pp. 5809-5811111 5817. PMLR, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. Internattional Conference on Machine Learning,
2014.
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. In Inter- national Conference on Learning Representations, 2018.
David Stutz, Matthias Hein, and Bernt Schiele. Disentangling adversarial robustness and general- ization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6976-6987, 2019.
David Stutz, Matthias Hein, and Bernt Schiele. Relating Adversarially Robust Generalization to Flat Minima. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp- 7787-7797, Montreal, QC, Canada, October 2021. IEEE.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. In International Conference on Learning Representations, 2018.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. International Conference on Learning Representa- tions, 2018
Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Lipschitz-margin training: Scalable certi- fication of perturbation invariance for deep neural networks. Advances in neural information processing systems, 31, 2018.
Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Normalized flat minima: Exploring scale in- variant definition of flat minima for neural networks using nac-bayesian analysis. In International Conference Machine Learning, pp. 9636-9647. PMLR, 2020.
Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does Ilm safety training fail? Advances in Neural Information Processing Systems, 36, 2024.
Zeming Wei, Jingyu Zhu, and Yihao Zhang. Sharpness-aware minimization alone can improve adversarial robustness. arXiv preprint trXiv:2305.05392 2023.
Dongxian WU, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust gener- alization. Advances in neural information processing systems, 33:2958- 29699, 2020.
Jia Xu, Yiming Li, Yong Jiang, and Shu- Xia. Adversarial defense via local flatness regulariza- tion. In 2020 JEEE International Conference on Image Processing (ICIP), pp. 1196-2200. IEEE, 2020.
14

---

Published as a conference paper at ICLR 2026
Yuelin Xu and Xiao Zhang. Understanding adversarially robust generalization via weight-curvature index. arXiv preprint arXiv:2410.07719, 2024.
Hao Xuan, Bokai Yang, and Xingyu Li. Exploring the impact of temperature scaling in softmax for classification and adversarial robustness. arXiv preprint arXiv:2502.20664, 2025.
Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Russ Salakhutdinov, and Kamalika Chaud- huri. A closer look at accuracy vs. robustness. Advances in neural information processing systems, 33:8588 8601, 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. British Machine Vision Confer- ence, abs/ (00s//605.07144 46, 2016.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In International Conference on Learning Rep- resentations, 2017.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, pp. 7472-7482. PMLR, 2019.
Wenjia Zhang, Yikai Zhang, Xiaoling Hu, Mayank Goswami, Chao Chen, and Dimitris N Metaxas. A manifold view of adversarial risk. In International Conference Artificial Intelligence and Statistics, pp. 11 11598-1 1614. PMLR, 2022.
Sijia Zhao, Tianyu Pang, Changyou Du, Yinpeng Dong, Hang Su, and Jun Zhu Bridging mode connectivity in loss landscapes and adversarial robustness. arXiv preprint 0tt::20005000660) 2020.
Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307. 15043, 2023.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A GENERALIZED DEFINITION OF ADVERSARIAL EXAMPLES
Lemma 10. Let f: X y be a classifier, 1: yx x x - + a loss function (e.g., cross-entropy), and (I, 4) E X x3 a sample with [(x) = y. Suppose & Bx(x)) is a classical adversarial example, i.e, ((E + y. Then there exists e such that
I((()) v) 1(f(xr))) >6
i.e., & an adversarial loss change according to Definition 4
Conversely, if satisfies a separation condition such that for all incorrect predictions f(E) # y we have
((f(E), y) 2 log(k),
then any adversarial loss change with threshold log(k) implies f(E) + y.
Proof. Since f(x = y, we have that the loss at is small, typically I(f(x)) y) 0 for standard losses like cross-entropy. If g satisfies f({) * y. then under standard classification losses, ((f(E), large, typically ((f(E), v) O. Thus, the difference
I(f(E). y) ((f(x),) 3) ((f(E)))
is positive and large. Choosing any satisfying 0 < e < ((S(E), v) [(f(x),)) ensures the loss- change condition is satisfied.
Conversely, assume that for incorrect predictions ((E) * we have I(S(E).)) > log(k), as is the case under cross-entropy loss when the model assigns uniform or lower probability to the correct class. If ((f(E), y) [(f(x),) > for some > log(k), then necessarily ((f(E), 3) > >>> log(k), implying that f(E) # y, because correct predictions would have loss much smaller than log(k). Li
B THE PENULTIMATE LAYER SHAPES THE HESSIAN
We consider a feedforward neural network of the form l(fz fL-1 o.... fi (x)), where each layer defined recursively by fu) ReLU(Waa). For each layer we define the pre-activation af Wix'-1 and the post-activation x' = ReLU(aa). Let g' = Df/0a' E R and H' = @ee0(aa) e mm denote the gradient and Hessian with respect to the pre-activations. diagonal gating matrix D' diag(1
define the binary which captures the ReL.U derivative.
To compute derivatives efficiently through the network, we apply the second-order chain rule for ccalar-valued functions. For composition y g(u(x)), where R", u(x) E IR"", and y E I. the Hessian satisfies
Oy
H2 = J + Σ
Hu,ks Ouk
where J, = 00/0u/ and Hu, Oux Applying this to the ReLU network, observe that all affine transformations a' = Wiz'=1 are linear in their imput, the second-order term E Hu,k vanishes. Similarly, the nonlinearity f(u) = ReLU(u) has second derivative zero almost every- where, curvature is introduced at the nonlincarity either.
Under these conditions, the gradient and Hessian propagate layer by layer as
-'-1= = W
This recurrence propagates curvature information purely through the affine structure and the ReLU activation mask. To compute the exact Hessian with respect the weights of layer I, we vectorize the weight matrix Wi columnwise as wy = vec Wi) E - Differentiating the affine map a' == W with respect to wi yields the Jacobian J, = 1-1 s Im,- Applying the second-order chain rule with respect to then yields the exact weight Hessian block
Hw, J,H ( - 00 (D'
16

---

Published as a conference paper at ICLR 2026
The full backward recursion thus proceeds follows. Starting from the output layer, where g" and H are obtained analytically from the loss function, we iterate backward for / = L, L ...... 1, computing at each layer the gradient g the aetivation-space Hessian H'-1, and the weight-space Hessian block Hw, as
- W
H1- - w''''''''''',,
Hw: = (a @ (D'H''').
This derivation follows the second-order backpropagation framework originally introduced by Becker & LeCun (1889) and Bishop (1992). The Kronecker-factoree form of the weight-space
Hessian is the foundation for scalable second-order optimisers such as K-FAC Martens & Grosse 2013). In the ReLU case, the absence of second-order contributions from the nonlinearity implies that all curvature originates at the loss layer and propagates backward linearly through the active part of the network.
Let H ) denote the Hessian with respect to the pre-activations at the final layer. At differentiable local minimum, the gradient vanishes and the Hessian positive-semidefnnic H > 0. In particular, all eigenvalues A of H- are non-negative. If in addition the trace vanishes, tr( (H") )) ) = 0, then the sum of the eigenvalues is zero. Since each >₂ 2 0, it follows that X = 0 for all
We now consider the spectral decomposition of 14, which exists because the matrix symmetric:
H= QAQ", A diag(Ai, ...,, An);
where Q is orthogonal and contains the eigenvalues of H4, As shown above, A = 0, so the decomposition becomes
H³ QOQ" = 0.
Hence, if HL ž 0 and tr(H)) = 0, then H 0.
For ReLU hidden layer the second derivative of the activation vanishes almost everywhere, so
the curvature transfer matrix reduces to B; D' with D' diag(("'>o)) The second-order back-propagation recurrence is then
H = W
Setting | = T. and using 4 0 yields H = 0. Re-applying the same identity inductively gives HL-2 0...., H 0; thus every activation-space block of the exact Hessian is zero.
The weight-space block for layer factorises as
Hw, = (a' & (D'H'D').
Since D' once the upstream H² has been shown vanish, each Hw, is also identically zero. Hence, away from the seeru kink set of the ReLU, the entire Hessian of the network collapses whenever the trace the output-layer Hessian vanishes local optimiser.
Lemma 11. (Vanishing of polynomial sharpness scores] Let W = (Waff W Wepr) be the parameters of a feedforward relu network whose logits are WT. @W..p- (z) and whose loss is soft-max cross-entropy. For any scale factor a > define the logit-scaling re-parameterisation
((a) (a Was, Wrepr).
Assume the sharpness score S (W, (W)) + Rzo satisfies the polynomial growth bound S(W, H) & C (1 + [[Wll=) (1+1111))*
(6)
for constants C > 0, >>>0>00 0, s 0, where H(W) = Vly is the exact Hessian. Then lim (()))), (a))) =. 00300
Proof. Fix one training example and let k be its correct class. Write the margins A; 2k 2j for j f l and Amin = minjœ4 Aj > O. After scaling the logits to az the soft-max probabilities are Py (a(a) (oftmax,(22). For every #
az;
(a) = Σ, easi S aad S -admin
(1)
17

---

Published as a conference paper at ICLR 2026
The logit-space Hessian for cross-entropy is Hee diag(p) -pp", , so its trace is tr Hee(a) - PP - -Pa) +
ΣpS(( -1,(1-p)) -p)1)
jyk
Using px = Pj and S 11- - Pj S we get
tr He) - Σ
Pj +
Σ
Pj 2
Pj(a) S 2(K -1) 1) -aamin
-aAmin =: De admm
j*k
JZk
jzk
Since ||Heel| S VK tr Heer
[a)ll & De a for D' = DVK.
(3)
Given the derivations above in Appendix B that is the whole hessian is a linear function of the penultimate layer, there exists D" > D' such that
(a) I D eamma
Only the classifier block is scaled, hence
[[(a)llr= a + ||repr 11? = O(a).
(5)
Insert (4) and (5) into equation
(W(()), H(a)) s C1++ Aa)" ( (1 + D" S C' e -aadmm
for constants A,C' The exponential decay dominates the polynomial growth, implying S(W(a), (()) 0 as
Lemma 12. |Confidence-collapse for losses with at most quadratic poles] Let E Rd be the input to the classifier, W IR xxd the classifier, 2 = Wx x the logits, and = softmax(z) the class probabilities. Fix twice- sifce-differentiable loss E(p. y) and write
8 (p. Op.
h₂(p)
1= K.
Assume there is M >> O such that for every p in the open simplex A =k1:=(pp (pe (0, }}
EmP=
hy(p) pr S M,
max(h,(p)1 M
(A)
Denote Hw (W y). Then, along any sequence of finite weight matrices for which the model becomes confident (Pg 1-),
Hw 0.
Remark. The boundary point = Cy (true one-hot) not reached with finite weights; would require Zy 2j 00. The lemma therefore treats the limit Py 71 inside the open simplex, where all derivatives are well defined.
Proof. Let Wx E K and = softmax(2) E A- The Jacobian of the softmax is
J = V.P = diag(p) pp E k
Write r.T for the ith row of J, so = p.(e: - p). Then |r = (pī 2p, + 1()1))) S pł Let
9i at/ap; and h, @/p/pp? The full second-order chain rule gives
H. == Vol(p()), v) = J diagg()) +
Σ
9. 2 = H (
18

---

Published as a conference paper at ICLR 2026
We first bound |
p. Since diag(h) is diagonal, have
K
hero
1|r:112-
Therefore,
hh 1elll1:11 112 + |
i
For i * y, assumption (A) gives |h,]
M, so h 1b:l11r:11
Mp For y, compute
|ryl1 = - p13 = p"
(1 (1-Pソ)) py-
+ Σ
p3
jØy
By assumption (A), [,y S M/r we have
M
1hyll1rg11 v
Py
(1 po) (
Σ p M (1
- pb)2 + Σ D3
piy
Since p3 Pj and En Pj = py, it follows that
Σ 2 S Pn,
so we conclude
h 1111111222 < M(1 ppy) M(1 Py).
Since 13 Pj and Enjn Dj
Py, we conclude
|H s M Σ + M(1 pb) S 2M py).
Next, bound || p|p. The second derivative of softmax satisfies /Pppll Cip; entrywise for some fixed constant C2 (K), hence
V2piile I C2ps.
Therefore,
K For all standard losses (cross-entropy, focal, squared-error, etc.), the product 19:|p; is bounded as Py + 17. Thus there exists G > such that
K Σ
H
9: V Pipll S C2
Σ gglp:
|| S G(1 - Ppy)-
Combining both bounds, we have
|H I || IF ||
S (2M + G) Pv) + as Py + 17.
Finally, since Wx, the Hessian with respect to the weights is Hw (xx & Hg, and
l[ ||
||a F [Hg ||
|| |g + 0.
19

---

Published as a conference paper at ICLR 2026
B.1 MORE CONTEXT TO LEMMA 12
Assumption (A) is easiest to interpret in terms of how fast the loss can blow up when probability goes O. It allows at most a 3 singularity remain bounded. That restriction may look arbitrary, yet the entries in Table [ show it is satisfied by almost every loss routinely used for hard-label classification: negative log- log-iikel,ooo, focal loss with y 2 1, the Brier score, and the (hard-label) KL divergence. For these objectives a perfectly confident model (py - 1) is well-behaved stationary point: the weight-space Hessian collapses to zero as stated in Lemma 12
only for the true class y; all other curvatures must
The only common exceptions are losses that deliberately assign positive target mass to more than one class. Label smoothing and KL to soft target inject factor qu //pp into the second derivative for every class with qi > 0; the same problem appears reverse KL, DrL (pllg) = =. Pi log(ps/q3 whose curvature is h and Lemma 12 no longer applies i.c. the proof is not valid anymore. In this particular case; this is not problem since ey is anyways not local minima and hence not of interest to make statements about robustness and generalization. Yet this is an interesting result which could explain the training dynamics of teach-student optimisation approaches.
1/p. Because such terms diverge when p; + O, Assumption (A) fails
Hinge-style or margin losses defined on the logits rather than on the probabilities fall outside the scope of the lemma because they are either twice differentiable do not depend on p all.
Loss function
hy (p)
hgzp(p)
Assumption (A)
Cross-entropy L --logg0 Py
0
Focal (1 -P,y) log Py
Opp 2+7
0
Vif 21
Brier / MSE Σ, - 5zy)
2
2
KL hard label é (KI ll p)
0
Py
KL to soft target e DKL(q || p)
4y
21
(some Qj > 0)
P3
Reverse KL to soft target e = DKL(p (1
0
Pv
Table 1: Curvature terms (p) for common loss functions. Assumption (A) is violated when any (p)p) becomes unbounded on the simplex. We give the derivations in Section B.2
B.2 ADDITIONAL DERIVATIVES FOR TABLE
8 Exact curvature expressions. Let h:(p) āyg (p,y) be the second derivative of the loss w.r.t
class i in the probability simplex.
Focal loss.
&tocal (p. v:?) -(1 (111-y)) py)" log Py-
Only Py is involved, so h2 for j # y. Let Py:
27 2² Py
hy(p) = -2 -7(7 (((7-11)o0)) 1)log
+
Then,
(p) = - |-> 1-) 1)p9 loggy + 2yp, + 0
This remains bounded as Pv + if and only if 7 2 1.
KL divergence to soft target.
(kL(p. q) = Σ>
qi log di
Σ qi log ps + const,
=
pi
20

---

Published as a conference paper at ICLR 2026
(p)
Pf
If any qj > 0, then hp(p) diverges Pj - 0, and assumption (A) is violated.
C PROOFS OF THEORETICAL RESULTS
In this section, we provide the proof' for the theoretical results in this paper.
C.I PROOF OF LEMMA 7
For convenience, we restate the lemma.
Lemma 7. Ler f = g(wob())) be a model with á L--Lipschitz and [|((.)]] 2 r, and E.x E with ||g ælI S 6, then there exists A >> 0 with A < Lör=1, such that ((E) x(x) + AAg(1), where is an orthogonal matrix.
Proof. It follows from the proof in Thm. 5 in Petzka et al. 2021 that can represent any vector E X as U = w + AAw for some vector w E X, A E R and A an orthogonal matrix. Then
((5) = ((x) AΦx()) + (0(E) - ((x)) = ((Ap(=))
#사 - ([(((E) -
11(0(x + A'A'z) +(x))1 (AA(()))
SLA
*사 < LA)I((A(()))
- LA
r
A arth.
The result follows from || - A <
C.2 PROOF OF PROPOSITION g
For convenience, we restate the proposition.
Proposition 8. For (x. X x y with ||ж||| 1 for all E X, model f(z) g(wo(x)) a minimum w E IR mm with & L-Lipschitz and [[o(z) 2 т, and the cross-entropy loss [(w) =
((g(w)(z)) v) of on (z,y), holds for all E X with | El2 < å that
8²
83
- - (((()),)))
L niy (w) + 24. kmL
2r
Proof. The remainder R₂ in Eq. 5 is
L ow NwwBww,ww. i.j.k
R2(w, A) S sup sup
(x + cAh)
heam cE(0.1) |
where E Rd with d = km is the vectorization of E B We now bound this remainder. Using the representation of the loss Hessian from Eq. 4 we can write the partial third derivatives in the remainder
ge (x)= Σ ONJEIN a.b.0E(m)
-1,bo(10o3 01) (1.=1 0,)) (()
dw,
where ÿ = f(x). Under the assumption that is L-Lipschitz, for all X. |||||| S and observing that Loels 9o = 1 we can bound this term by
Ee
Ow.üCi Bww (x) < kmL.
(7)
21

---

Published as a conference paper at ICLR 2026
The terms k follow from the sum all rows and columns of w, and the factor 4-1 follows from the fact that the predictions á sum up to 1. The factor L can be derived as follows.
Φ(x); S [|0(x):;1 = [1(b(a)); (00): + (00).1
(8)
< [1(((()); 60))) + |
(9)
I L|| 001 + |0(0):]l
(ó is L-Lipschitz)
(10)
- I. + |lx(0))] S Co(o)
(||=) s 1)
(11)
(12)
where Coco) == maz, mazz|lo()).. (0):) For Relu networks without bias Coro) is 0 and empirically for networks with bias term, it is very small i.e. x 1. Therefore L' = L + Coro) = L, in particular since for most realistic neural networks L is large. For simplicity, we subsitute L'.
Inserting this bound into Eq. 5 yields
A +
A 1
(E(())) - & V
kmL
2
3! 4
The result follows from setting A < Lör-1
C.3 PROOF OF PROPOSITION 9
Proposition 12. For a dataset S C X
with ||||| S f all X, model f(z) = g(wo(z))
at minimum w E Rm wrt. S, with o L-Lipschitz and [[ø(x)] 2 r. and loss function {(w) = [(((w((z)) y) of on (z,z), d being the L2-distance, and > 0, f is (c, robustt against adversarial examples with
1r313m333 1.9 + 27c
27 1o
662
m 1
+
27 NiT (w)6
n7n (w))
8 km L2 27E
27 6
6662
24
2T (w)
27 KiT (w)
27k7, (w)
2rkmT2
72k7,(W)
where (w) is the relative flatness of f wrt. That is,
8 e
(w)) + rkmL2
(w)
Proof. From Prop. it follows that we achieve (e, A, robbusstsss where
2
A
NTy (w) +
kmL
2
24
First, we need to solve this cubic equation for A. For that, we substitute and get
3.
and (..
21
0 = aA 3 + 6A = 3
3 + a 8
b
by subsituting
a and B f. We use the depressed cubic form A 1 and get
0
(
a
+
(
a
2
B
3
3
?
3
2a
a
0
3
Pat
1-
la
B
3
9
2
2
0
3
3
3
27
22

---

Published as a conference paper at ICLR 2026
a² with p and 9 formula.
2a 27
B get the form 0 = t + pt + q for which apply Cardano's
9
2
q²
p
t
+
+
2
9
4
Resubstituting p, 9, a, yields
q
pd
1
a
q²
1
a
a
2
271
2n
9
93
76
4
272
86
412
27h
Substituting this in the solution for i then gives
a
2a
2
a²c
2
2763
26
2766
462
276
2a
22
a3e E
+
+
27h3
2b
27h6
46
27b4
Substituting a, and = + of yields
m L + 27e
27 mm 6 1
2²
24ek
L 67
A
+
27k (w)
27 KTr (w)
kT,(W)²
27mig,
8k3m3 L 27E
27 k.sh m 2
2²
24 m 1
+
27k7,, (w)
27 1
RiT (w)6
NTr
(w)2
27ng,(w) (w))
27k
2kmLL
72
(w)
Finally, from Lemma
we have A < Lôr-1, 1, so that
3 L 27e
27 6 1
2 $9
r
8 >
+
T
27k4, (w)
27 th
KiT, (w)6
KTT (w)2
27n 1 (w)²
$
8233m3T + 27E
2 1
2
21ck3m311 3 1,9
+
27n
27 e (w)S
(w)2
27n
(w)
2rkmL²
72k (w)
wm
81 k3 L + +7e
27 3 m L3
24, ck m Lx
27L3Mg, (w)
27 NTy (w)6
L (VN,(WW)
27 (w) (w)a
8r km 1 + 27c
27 6 m L
24,3 kk 111 1
+
2 n.,, (w)
27 not (w)
"nf,((W)'
27n (w)+
2rkmL² 72k 2
(w)
C.4 DERIVATION OF HESSIAN AND THIRD DERIVATIVE
Let & RD denote the embedding of the feature extractor and W Kxm, where denote the weights of the k-th neuron as W.. The output layer is given by the softmax function O = softmax(Wo) RRK More precisely the softmax is given by.
23

---

Published as a conference paper at ICLR 2026
1/k exp(wxd)
Σ j== exp(w,0)
For simplicity, omit the bias term. The one- hot encoded ground truth is given by The derivative of the loss function wrt. the weight vector wj can computed as
aL(y.0) Ow,
- -0;
Second derivative
@T 0)
Own Ow, Owr
0
Ow 2
Owi
① Owr The last equation follows from y being independent of wy. Next, we do case analysis on / j.
210
-1= = j): In (1), use the quotient rule and in (2) definition of softmax.
① 2 exp(w,ō) S
Omn 200
ow, E exp(w
(exp(w;)) Ek exp(wre))oc' - e exp(wx (1)
(K exp(
exp(ww3) Σ exp(wxe)
exp(w,))²
(x=rexp(uxdd))
Eh= exp(wx
= (0) - 83 D00 Rmxm (2)
(1 / j): Again quotient rule, but the left side vanishes.
a Our
0,0 -000,00 e mm
Then have
@T(y.0) 0) Owi Ow,
in (n(1((=))) - 01)ØOT E mx
The hessian is then given by
H(L:W)(y,B) (diag(i) -ū @ des E ER Km
Third derivative
First, rewrite
OL 1)
(n( - 01)00" = Gr "= - 000
Oun dwy
Then define operator II Ra x R R Rnxmxo II (2, 2e We now compute
aL(y.0) 2)
Ur l p= 010,00"
Ow, Owr Ow
Dw.
Again make CA =1
24

---

Published as a conference paper at ICLR 2026
(1= j):
8
Ow. (ir00 - "x
U1 of OΦ
Ow.
Ow:
Bo [1==]]] (1)) 11(0, Φ, 6) 2(00(1|a=1) in) 11(0, o, 0))
-oo( [o=1] (1) IT ④ 0,
. *1 5):
0
= -
in (1r T in( Owo
Owo
Owo
Úūūūo |0=1| -i1) [I( 0, Φ) タi [== -g)) [I((2) o.
[o=1|
[0=1] - (1)) + û1 ((-= - 11 E mmxmxm 110880 (1
1ū [o=1] in) + in in (==0] ú5)1م,.0=11.8 [1, 0, @
D ADDITIONAL RESULTS
D.I RESULTS FOR OTHER ARCHITECTCRES AND DATASETS
We provide the remaining results in Figure 7 to Figure 12 We see the same trend as observed in the main paper. The only difference is that for models trained CIFAR-100 the basin width correlates not as strong with the sharpness O compared to models trained CIFAR-10.
0.5
2.5
Normalized Loss
50
Frequency
1,500
0.25
0.8
40
Busin witth
20-
1,000
0.4
20
500
0.2
10-
0
0-
10
15
20
25
5
10
15
20
25
200 300
Loss increase
Iterations
Iterations
Relative Sharpness@00
(a)
(b)
(c)
(d)
Figure 5: Loss geometry for VGG11.(a) We report the distribution of the loss increase for varying scaling values. (b) We show for one example how the basin formes as we increase the scaling, We use the normalized loss to plot in one axis. (c) We show examples samples exhibiting different basin widths. (d) We plot the sharpness measured at the test data (s 1) and the basin width.
0.5 1 25
10
Normlizzdd Loss
50-
Frequency
1,500
0.25
0.8
40
Basin wäthh
1,000
0.6
30
15
500
10
0.2
10-
0
2
10
20
25
2k
4k
6k
8k
Loss increase
Iterations
lerrations
Relative Sharpnesx@0
(d)
Figure 6: Loss geometry for VGG11 on (00,)- We report the distribution of the loss in- crease for varying scaling values. (b) We show for one example how the basin formes as we increase the scaling. We use the normalized loss to plot all in one axis. (c) We show examples of samples exhibiting different basin widths. (d) We plot the sharpness measured at the test data (s 1) and the basin width.
25

---

Published as a conference paper at ICLR 2026
0.5
-
50
50
Trequency
1,500
0.25
Nonmalized suullzzdd
0.8
40
Busin with
20-
1,000
0.4
20
500
10
0
0-
10
15
20
25
5
10
15
20
25
100
200
Loss increase
Lterations
lierations
Sharpness
(a)
(b)
(c)
(d)
Figure Loss geometry for RESNET- (a) We report the distribution of the loss increase for varying scaling values. (b) We show for example how the basin formes scaling. We use the normalized loss to plot all in one axis. (c) We show examples samples exhibiting different basin widths. (d) We plot the sharpness measured at the test data (s 1) and the basin width.
increase the
Normalized Lows
50-
25
Frequency
1,500
40
Basin widlth
1,000
0.6
30
15-
0.4
20
10
500
0-
15
20 25
10
20 25
2k 4k
Loss increase
Iterations
Iterations
Relative Sharpness
(a)
(b)
(c)
(d)
Figure 8: Loss geometry for RESNET-18 on CIFAR-100. We report the distribution of the loss
increase for varying scaling values. (b) We show for one example how the basin formes as we increase the scaling. We use the normalized loss to plot all in one axis. (c) We show examples of samples exhibiting different basin widths. (d) We plot the sharpness measured at the test data 1) and the basin width.
0.5
25
Nommaltzed Loss
50-
25
Frequency
1,500
0.25
0.8
40
Basin widhh
1,000
0.6
30
15
20
500
10
0.2
10
0
10
20
15
25
100
200
300
Loss
Iterations
Iterations
Relative Sharpnes:00
(b)
(c)
(d)
Figure 9: Loss geometry for WIDERESNEE-28-44.02 We report the distribution of the loss increase for varying scaling values. (b) We show for one example how the basin formes as we increase the scaling. We use the normalized loss to plot all in one axis. (c) We show examples of samples exhibiting different basin widths. (d) We plot the sharpness measured at the test data (s = 1) and the basin width.
D.2 FLATNESS CAN INDUCE GRADIENT MASKING
To test whether the geometry backpropagates to the input, need to compute the Hessian with respect to the input, which is prohibitively expensive. Instead, we take an actionable route: the loss surface is truly flat, then first-order attacks must fail, as without curvature the optimization cannot succeed ie. they become unattackable. We attack ResNet- 18 models with standard PGD-L (10 steps, 8/255, step size a 2/255) while varying the penultimate layer scaling factor
26

---

Published as a conference paper at ICLR 2026
0.5
50
Normalized suullzzdd Loss
50
Trequency
1,500
0.25
0.8
40-
Basin widhh
20
1,000
0.4
20
600
10
0-
10
15
20
25
10
15
20
25
4k
6k
8k
Loss increase
Lterations
lierations
Sharpness
(a)
(b)
(c)
(d)
Figure 10: Loss geometry for WIDERESNET-28-44 on CIFAR-100.(a) We report the distribution of the loss increase for varying scaling values. (b) We show for one example how the basin formes as we increase the scaling. We use the normalized loss to plot all in one axis. (c) We show examples of samples exhibiting different basin widths. (d) We plot the sharpness measured the test data (s 1) and the basin width.
0.5 - 2,5
Nomulized Loss
50
Prequency
1,500
0.25
0.8
40
30
Basin widhh
1,000
0.4
20
500
0.2
10-
5
0-
0
10
15
20
10 15
20
25
200400600800
Loss increase
Trerations
Iterations
Relative Sharpness@0
(a)
(h)
(c)
(d)
Figure 11: Loss geometry for DENSENETI2I,(A) We report the distribution of the loss increase for varying scaling values. (b) We show for example how the basin formes the scaling. We use the normalized loss to plot all one axis. (c) We show examples of samples exhibiting different basin widths. (d) We plot the sharpness measured at the test data (s and the basin width.
increase
0.5 1 - 25
50
Normalized Loss
50
Trequency
11,500
0.25
0.8
40
Basin widhh
0.6
30
1,000
=
0.4
20
500
10
0.2
10
0
10
15
20
10
20 25
2k 4k 6k 8k
Loss increase
Iterations
leraatins
Relative Sharpness@00
(b)
(c)
(d)
Figure 12: Loss geometry for DENSENETI21 on (IFAR-100. We report the distribution of the loss increase for varying scaling values. (b) We show for one example how the basin formes as we increase the scaling. We use the normalized loss to plot all in one axis. (c) We show examples of samples exhibiting different basin widths. (d) We plot the sharpness measured at the test data (s = and the basin width.
from 100. We measure the robust test accuracy and average sharpness the clean examples and show the results Figure I3
As we can increasing dramatically improves robust test accuracy. The standard ResNet with s = achieves only 0% robust accuracy, while its scaled counterpart with s = 100 reaches 93%, which recovers the clean accuracy. Crucially, the relative sharpness measured at the penultimate layer strongly correlates with robust test accuracy across all scaling values. This direct correlation demonstrates that penultimate layer geometry indeed propagates the input space. Importantly,
27

---

Published as a conference paper at ICLR 2026
this does not mean that scaled networks adversarially robust to according Defintion Bnsteead, the flat geometry prevents first-order attacks from finding adversarial examples. More specifically, adversarial examples found at 1 transfer with 100% success other scales, indicating that the same vulnerabilities remain but are harder to reach. This highlights that analyzing adversarial robustness through sharpness
nuanced than it appears and requires carefid evaluation.
Relative sharpness
Robus Test Acc.
100
Robus Test Acc.
100
Robus Test Ace.
100-
80-
80-
80-
60-
60
60-
5-
40-
Clean
40-
Clean
40-
Clean
0
0-
20 40 80100
20
40
60
100
10
10
Scaling s
Scaling #
Relative Sharpness
Relative Sharpness
(a)
(b)
(c)
(d)
Figure 13: Unattackable networks. (a) report the relative sharpness of the scaled networks the test data. (b) Robust Test Accuracy with varying scaling s. (c) relative sharpness at the clean data
vs. Robust Test Accuracy. (d) transferability of the attacks created on 1.
Relative flainess
Relative flaness
800-
Relative flamess
Relative flamess
1.5k
40-
200
600-
600
20-
100
Relative
400-
400-
200
200
10
Attack iteration
Attack iteration
Attack iteration
Attack iteration
Attack iteration
Layer I
(b) Layer
(c) Layer
(d) Layer
Layer
Figure 14: We show the relative sharpness measure computed in the penultimate layer and in shallower layers - approximate the measure using Hutchinson trace estimation used in Petzka et al. (2021) on 500 images. We observe same phenomena as penultimate layer, which justifies that we focus only the penultimate layer for our theoretical and experimental analysis.
for WIDERESNETT28--4 Due to memory and runtime constraints,
Vieuna
LLama2
Guanaco
Relative sharpesss
Relative shurpness
5,000
4,500
5,000
3,500
4,000
0.5
4,000
3,000-
1
20 40 60 80
20
40
60
8D
20 40 60 8D
20 4u 60 8o
Attack iteration
Attack iteration
Attack iteration
Attack iteration
Figure 15: Top. We plot the relative sharpness and loss of the adversarial prompt for Viucna, LLama2 and Guanaco when attacked by the method of Zou et al. (2023). Bottom. We give per model example trajectories that first get sharper and then flatter again.
D.3 DETECTING ADVERSARIAL EXAMPLES
It is possible to detect adversarial examples using simple threshold on the relative sharpness mea- sure. We did not include a practical study of this since it would go beyond the scope this paper: Developing sound method requires more than fine-tuning the threshold. Moreover, this requires comparing the approach to a wide range of state-of of-the-art detection methods, which would be a paper of its own. Therefore, we provide preliminary results. We trained a decision stump on the sharpness of clean and adversar- ial samples CIFAR-10 for WIDERESNET-288 using 5-fold vols-vvaii,tion, which yields the
leave this interesting practical aspect for future work. Nonetheless,
28

---

Published as a conference paper at ICLR 2026
following accuracies: [0.92, 0.92, 0.93, 0.92, 0.92]. i..., adversarial examples can be detected with an average accuracy of 0.92 with little to no difference between the folds.
E RELATED WORK
Defenses and Guarantees
The most widely used method for improving robustness to adversarial
examples is adversarial training, which integrates adversarial perturbations into the training pro-
cess (Szegedy et al. 2014 (Shafahi al.]
2019
Kumari et al. [019 Perolat et al.] [2018] Shafahi
eet al. 2020; Cai et al. 2018 [Tramer
[ et al. 2018 Wu et al.] 2020 (0120/Carmo Carmon et (019). While this
approach enhances adversarial robustness, frequently leads to reduction in clean accuracy
[sipras et al. 2018: Rade & Moosavi-Dezfoolii 2021] Nakkiran 2019 Zhang et al. (019). This
trade-off remains controversial. prompting the question of whether is fundamentally impossible to simultaneously achieve high performance and robustness.
One hypothesis is that adversarial training is inherently more difficult than standard training, and that networks simply underperform when faced with this more challenging task (Schmidt et al. 2018 Conversely, Raghunathan et al. (2020) argue that deep neural networks are capable of fitting even random noise (Zhang et al. 7), suggesting that insufficient regularization, rather than task complexity, may be the root cause of this performance drop. Indeed, numerous works have explored strategies to improve generalization and thereby enhance the effectiveness of adversarial training.
These include incorporating additional data (Carmon et al. 2019]] Alayrac et al. 2019; Hendrycks et al. 2019; Gowal et al.) (021). using early stopping (Rice et al., 2020). and promoting flatness in the loss landscape (Wu et al. 2020 Stutz et al. 2021 Moosavi- Dezfooli et al. 2019 Xu et al., 2020
To better understand this controversy and nature of adversarial vulnerability. a line of research has proposed the in- and off-manifold hypothesis (Stutz et al. 2019 Shamir al.] [021] ZZhan al.]
2022 Haldar et al.] 2024) Melamed et al.) 2023). According to this perspective, adversarial exam- ples can be either on the data manifold or off it. Robust generalization is achievable primarily for in-manifold adversarial examples, while off-manifold ones tend to degrade clean accuracy during adversarial training. [ong al. (2018) support this view empirically, proposing defense mecha- nism that learns the data manifold and projects adversarial examples onto it, thereby restoring correct classification.
This perspective requires additional assumptions about the data distribution. In practice, we often have access to only a finite and relatively small tetasst--speciallyy when compared to the high di- mensionality of input spaces- which limits our ability to precisely model or leverage the underlying manifold structure.
From the other side, natural approach to controlling model behavior on unseen samples is through the Lipschitz constant: an input is close to a training sample, the model's prediction should not vary significantly. However, applying Lipschitz constraints directly to modern deep networks proves challenging. Global Lipschitz constants (Tsuzuku et al., 2018)) tend to be too coarse to be meaning- ful, while local I. Lipschitz constants (Hein & Andriushchenko 2017) 7 computationally intractable to estimate accurately. Moreover, Liang & Huang H2ang show that a large global Lipschitz constant does not necessarily imply large local constants, meaning that model may still be locally robust despite a high global value. Conversely, enforcing a small global Lipschitz constant can help control local behavior but often leads to degraded clean accuracy. Yang et al. (2020) suggest that improv- ing local Lipschitz constants alongside generalization could yield both robust and accurate models, though achieving this balance not straightforward in practice.
Efforts to explicitly control Lipschitz properties during training have inspired the development of certification-bant based robustness techniques. In these methods, models are trained to produce consis- tent predictions under random input perturbations, which yields certified guarantees of robustness on training samples (Cohen et al., 2019: Salman et al. 2019). Interestingly, Kanai et al. (202)) analyze robustness in both the input and parameter spaces, concluding that enforcing smoothness in the input space often results in a non- flat loss surface with respect to parameters, which in turn harms general- ization. This suggests that input-space robustness alone may not suffice to preserve clean accuracy, while parameter-s flatness alone may be insufficient for adversarial robustness. This tension ties into the broader understanding of generalization: robustness in parameter space- measuredee via loss
29

---

Published as a conference paper at ICLR 2026
surface flatness is known correlate with improved generalization (Hochreiter & Schmidhuber 1994 Liang et al. 2019: Tsuzuku et al. 2020; Foret et al.) 2020; Petzka et al. 2021). Accord- ingly, entorcing smoothness in both the input and parameter spaces lead to models that are both robust and generalize well, as demonstrated empirically by Wu et al. 2020), who also highlight the connection between weight flatness and the robust generalization gap. Taking another perspec- tive on measuring smoothness, Simon-Gabriel et al. (2019) show that adversarial examples tend to correspond to regions with higher input gradients, further linking gradient norm regularization to robustness. However, this line of research typically does not address whether such regularization helps against in m-nifolnf or off-manifold adversarial examples.
In our analysis, we connect flatness in parameter space, network Lipschitz properties, and adver- sarial robustness. We argue that while these defenses offer meaningful local guarantees, achieving global robustness necessitates additional assumptions about the structure and properties of the data manifold. Xuan et al. (002) came similar to our conclusions with respect to the effect of network confidence on adversarial robustness, but through adjusting temperature. They show empirically in preliminary experiments that high temperature cross-entropy loss promotes better adversarial robustness. Xu & Zhang (2024) derives a measure bounding robustness generalization gap using PAC-Bayes bounds. This measure surprisingly repeats the relative flatness (Petzka et al. 2021) that in our derivation, confirming that such theoretical connection can be proven from different perspectives.
F UNNORMALIZED PLOTS
RESNET-18
VGGI1
DENSENEr121
Relative sharpesss
60
150-
100-
20-
50-
0-
2
8
10
10
Attack iteration
Attack itcration
(b) Loss CIFAR-10
(a) Sharpness CIFAR-10
Relative sharpeess
3,000
80
2,000-
60
1,000
40
20
Attack iteration
Attack iteration
Sharpness CIFAR-100
(d) Loss CIFAR-100
Figure 16: We report the relative sharpness on the attack trajectory of attack for WIDERESNET- 28-4, RESNET- VGG11 and DENSENET121 on the test set of CIFAR-10 CIFAR-100. We observe that adversarial examples first reach a sharp region, but with strength of the attack increasing they are very flat region. We also display the standard deviation of the values on individual inputs.
G CODE FOR EXPERIMENTS
We use code from several resources, which we disclose here. First, the basis for training and attack-
ing the CNNs stems from (Sehwag et al.] 2020). We modified the code according to our needs. The code for 121 stems from the official PyTorch library. To attack and evaluate the LLMs, we use the official implementation of the attack (Zou et al. 2023). CIFAR-10 and CIFAR- 100 were also downloaded from PyTorch.
30

---

Published as a conference paper at ICLR 2026
H LLM USE
In this work, we used GPT-4o for both writing and coding support. On the writing side, assisted with editing and condensing text to improve clarity. For coding, GPT-5 was used for debugging, pro- viding autocomplete suggestions in VS Code, and generating code for LaTeX figures. Additionally, gpt-o3 was used verify steps in the proofs and try falsify certain steps.
31


Reviews:
[{"review_id": "1g21HSjBnV", "reviewer": "Reviewer_qEqm", "summary": "This paper challenges the theoretical hypothesis that flatness increases the robustness of neural networks and discovered that flatness implies local but not global adversarial robustness. Flatness tends to emerge in regions where the model is highly confident.", "questions": "1. Does the metric relative flatness possibly ignore the correlated curvature directions across layers?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The Uncanny Valley analysis helps explain why adversarial examples can appear deceptively safe, because they often lie in flat, vast, high-confidence region.\n2. The paper is generally well-written and well-presented.\n3. The relation between relative sharpness and adversarial robustness is clearly explained, and a precise robustness bound is given", "weaknesses": "1. A large part of the paper is used to justify that relative flatness at the penultimate layer is sufficient, which seems to have been stated in Petzka (2021).\n2. The entire analysis was built on the local flatness at the penultimate layer; did the authors rule out the effect of the geometry of the input space on the adversarial robustness?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6l3z2OB324", "reviewer": "Reviewer_HZEM", "summary": "This paper investigates the long-standing hypothesis that flat minima in the loss landscape imply increased adversarial robustness. Through a rigorous theoretical formulation, the authors show that flatness guarantees only local robustness rather than global robustness. Empirical evaluations across different architectures further support this finding, revealing that adversarial examples often reside in large, flat regions of the loss landscape.", "questions": "1. I think the first paragraph on page 1 could be divided into multiple shorter paragraphs to improve clarity. Also, in Figure 1 (mentioned on the first page), the notation $\\phi^{-1}$ is a bit confusing — it’s not immediately clear how the authors relate the feature space to the input space. It would be better to explain this mapping more intuitively in the introduction.\n\n2. On page 2, lines 90–93, the first and second listed contributions appear somewhat redundant. In the first contribution, you state that you theoretically establish the link between flatness and adversarial robustness, while in the second, you restate this point with more detail, specifying that the link is derived through the penultimate layer. I think these two points should be combined into a single, unified contribution for clarity.\n\n3. On page 4, in Definition 4 (Loss-change adversarial example), you define adversarial examples as those that increase the loss by more than $\\epsilon$. Then, in line 183, you state that “by using a conservative $\\epsilon > \\log(k)$ for cross-entropy loss, we can ensure a prediction flip.” However, in practice, when the number of classes $k$ is large, this threshold can correspond to very high loss values. Such cases may represent “strong” adversarial examples (in terms of loss magnitude). Moreover, the reverse implication is not guaranteed, when $l(f(x), y)$ is not close to zero, a prediction flip might occur with much smaller loss changes. Therefore, the relationship you establish between loss increase and prediction change may not hold universally. Could you provide more discussion or clarification on this limitation?\n\n4. On page 7, line 363, and page 8, line 378, there are two separate “Setup” paragraphs in Section 6 From Theory to Practice—one ending with a full stop and another continuing without. It would be clearer to merge them into a single, continuous setup description, as the current separation is somewhat confusing.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The authors rigorously extend Petzka et al. (2021)’s notion of relative flatness to the setting of adversarial robustness, providing formal derivations that are technically sound and mathematically non-trivial. The paper presents elegant formulations that establish a clear analytical link between adversarial robustness and the Hessian of the loss.", "weaknesses": "1. While overall well-written, the paper is dense and somewhat derivative-heavy. Key ideas, such as the geometric mapping between feature-space and input-space curvature, could be illustrated more clearly using diagrams or intuitive explanations. A more intuitive introduction or motivating example would help readers better grasp the high-level ideas before delving into the detailed derivations. In particular, the terms “relative flatness” and “relative sharpness” are used somewhat inconsistently, which may confuse readers.\n\n2. The empirical evaluation is somewhat limited. While the experiments illustrate the theoretical claims qualitatively, they rely on relatively weak PGD attacks (PGD-$l_2$ with $\\epsilon = 0.025$ in page 8) and lack comparisons with adversarially trained baselines (e.g., TRADES, AWP, or SAM-trained models). As a result, it remains unclear whether the observed relationships between sharpness and robustness persist under stronger or more realistic adversarial conditions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qYpAkHPEpM", "reviewer": "Reviewer_zHuA", "summary": "This paper re-assesses prior claims about the relationship between the flatness of the loss landscape and adversarial robustness. Consistent with previous claims the paper finds that flatness enhances robustness around particular training samples. However, in contrast to previous claims the current paper suggests that adversarial examples can also lie in flat regions. Hence, increasing flatness does not necessarily improve robustness.", "questions": "Given the lack of correspondence between classification accuracy and cross-entropy loss described above, which of the claims/results in the paper are still valid?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "The paper is well structured and generally clearly written.\n\nThe paper combines both theoretical and empirical research.", "weaknesses": "All the analysis in the paper is based on characterizing successful adversarial attacks through changes in loss rather than changes in predicted label (section 3.2). Specifically, it is claimed that an adversarial perturbation will increases the loss beyond a threshold. However, this is not true for cross-entropy loss. For example, consider a neural network that performs a 3-way classification task. If the true label of a sample is 0 and this network outputs logits [0.6, 0.1, 0.1], then the sample is classified correctly and the cross-entropy loss is 0.7944. If the sample is perturbed in such a way as to produce logits [0.6, 0.7, -5], then the predicted class label is wrong (i.e. the perturbation constitutes a successful adversarial attack), yet the loss decreases to 0.7462. The reverse is also the case: a large increase in loss does not necessarily indicate a successful attack. For example, if the same sample was perturbed so that the network produced logits [0.6, 0.5, 0.5], then the attack would be unsuccessful, but the loss would be increased to 1.0331.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fQxPQReG9u", "reviewer": "Reviewer_eTYw", "summary": "This paper provides a formal analysis of the relationship between flatness in the loss landscape and adversarial robustness in neural networks. While flat minima are often believed to enhance robustness, the authors demonstrate that this connection holds only locally, not globally. They derive a closed-form measure of relative flatness in the penultimate layer and use it to constrain input-space loss variation, enabling a theoretical assessment of network robustness. Empirical results across models and datasets support these findings, showing that adversarial examples often occupy flat regions where models are confidently incorrect.", "questions": "1. Given the similarity between the finding that adversarial examples often lie in large flat regions and the observation of a downward trend in the input loss landscape slope (as shown by IG in Fig. 1 of [1]), it would be valuable to explore whether the theoretical framework presented in this work can explain or align with that trend. Establishing such a connection could strengthen the theoretical grounding and unify observations across studies.\n\n2. Since adversarial training is known to significantly enhance adversarial robustness, it is important to examine how the proposed analysis interacts with or adapts to models trained adversarially. Specifically, how does relative flatness behave under adversarial training, and does the established relationship between flatness and local/global robustness still hold? Addressing this would clarify the scope and applicability of the theoretical insights.\n\n[1] Li, Lin, and Michael Spratling. \"Understanding and combating robust overfitting via input loss landscape analysis and regularization.\" Pattern recognition 136 (2023): 109229.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work presents a framework for analyzing adversarial robustness through relative flatness, a concept introduced in prior studies.\n2. While the conclusion that “flatness implies local but not global adversarial robustness” is not surprising, formally establishing this insight contributes meaningfully to the theoretical understanding of adversarial robustness.\n3. The finding that adversarial examples often lie in large, flat regions is intriguing.", "weaknesses": "1. The theoretical analysis appears to overlook the generalization from training to unseen test data. Even if the connection between relative flatness and adversarial robustness holds on the training data, it remains unclear how flatness measured on training examples translates to robustness on unseen test inputs.\n2. The paper lacks actionable insights for improving adversarial robustness, limiting its practical impact despite its theoretical contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)