Relative Flatness and Generalization
Authors: Henning Petzka
Michael Kamp
Linara Adilova
Cristian Sminchisescu
Mario Boley
Venue: NeurIPS 2021 Poster (2021)
Keywords: Flatness
generalization
interpolation

Abstract: Flatness of the loss curve is conjectured to be connected to the generalization ability of machine learning models, in particular neural networks. While it has been empirically observed that flatness measures consistently correlate strongly with generalization, it is still an open theoretical problem why and under which circumstances flatness is connected to generalization, in particular in light of reparameterizations that change certain flatness measures but leave generalization unchanged. We investigate the connection between flatness and generalization by relating it to the interpolation from representative data, deriving notions of representativeness, and feature robustness. The notions allow us to rigorously connect flatness and generalization and to identify conditions under which the connection holds. Moreover, they give rise to a novel, but natural relative flatness measure that correlates strongly with generalization, simplifies to ridge regression for ordinary least squares, and solves the reparameterization issue.

Relative Flatness and Generalization
Henning Petzka"
Michael Kamp"
Lund University, Sweden henning. petzka@math. tttimatthhhhtts..... se
CISPA Helmholtz Center for Information Security, Germany and Monash University, Australia
michael kamp@monash. edu
Linara Adilova Ruhr University Bochum, Germany and Fraumhofer IAIS
Cristian Sminchisescu Lund University, Sweden and Google Research, Switzerland
Mario Boley
Monash University, Australia
Abstract
Flatness of the loss curve conjectured to be connected to the generalization ability of machine learning models, in particular neural networks. While it has been empirically observed that flatness measures consistently correlate strongly with generalization, it is still an open theoretical problem why and under which circumstances flatness is connected to generalization, in particular in light of reparameterizations that change certain flatness measures but leave generalization unchanged. We investigate the connection between flatness and generalization by relating the interpolation from representative data, deriving notions of representativeness, and feature robustness. The notions allow us to rigorously connect flatness and generalization and identify conditions under which the connection holds. Moreover, they give rise to a novel, but natural relative flamess measure that correlates strongly with generalization, simplifies ridge regression for ordinary least squares, and solves the reparameterizztioo issue.
Introduction
Flatness of the loss curve has been identified as a potential predictor for the generalization abilities of machine learning models [6, 10, 11]. In particular for neural networks, it has been repeatedly observed that generalization performance correlates with measures of flatness, i.e., measures that quantify the change in loss under perturbations of the model parameters [4, 16, 21, 34, 39, 41, 44]. In fact, Jiang et al. [14] perform large-scale empirical study and find that flatness-based measures have higher correlation with generalization than alternatives like weight norms, margin,, and optimization-bans based measures, It is an open problem why and under which circumstances this correlation holds, in particular in the light of negative results on reparametrizations ReLU neural networks [5]: these reparameterizations change traditional measures of flatness, yet leave the model function and its generalization unchanged, making these measures unreliable. We present novel and rigorous approach to understanding the connection between flatness and generalization by relating it to the interpolation from representative samples. Using this theory we, for the first time, identify conditions under which flatness explains generalization. At the same time, we derive measure of relutive flatness that simplifies to ridge/Tikhonov regularization for ordinary least squares [36], and resolves
"qual contribution
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

iagut
hiddes layers
output layer
layar
I'mpprsentttionnee
v(z) = ((w))
l
R+
Eemp
Fiacness
M
Peature robustacce with constont Jaheia
w dwed
Figure| I: Decomposition of f = v o 6
Figure Overview: We theoretically connect a notion of representative data with notion of feature robustness and novel measure of flatness of the loss surface.
into feature extractor and model g for neural networks.
the reparametrization issue for ReLU networks [5] by appropriately taking the norm of parameters into account suggested by Neyshabur et al. [25].
Formally, we connect flatness of the loss surface to the generalization gap (f.S) E(f) Eemp (f. S) of a model f: X + y from a model class H. with respect to twice diffcrentiable loss function 8 x y + R+ and a finite sample set _ X V, where
E(f) Li ((f((z).)) and (f
Σ ((f(z).))
SS]
(RNJES
That is, Egen(J is the difference between the risk E(J) and empirical risk (Jmp(s: S) of on finite sample set S drawn iid. according to data distribution D on X' x y. To connect flatness to generalization, we start by decomposing the generalisation gap into two terms, a representativeness term that quantifies how well distribution D can be approximated using distributions with local support around sample points and a feature robustness term describing how small changes of feature values affect the model's loss. Here, feature value refers to the implicitly represented features by the model, i.e., we consider models that can be expressed as fx) = (W Φ(x)) g ((w)(z)) with feature extractor é and model v (which includes linear and kernel models, well most neural networks, see Fig. 1). With this decomposition, we measure the generalization ability of particular model by how well its interpolation between samples in feature space fits the underlying data distribution. We then connect feature robustness (a property of the feature space) to flatness (a property of the parameter space) using the following key identity: Multiplicative perturbations in feature space by arbitrary matrices A E Rm correspond to perturbations in parameter space, i.c..
vi(w, ((x) + Ad(x)) = ((w(((r) Ao(z))) = g(( (w + ()))(x)) = (w + ((x)) (1)
Using this key equation, we show that feature robustness is approximated by novel, but natural, loss Hessian-based relative flatness measure under the assumption that the distribution can be approximated by locally constant labels. Under this assumption and if the data is representative, then flatness is the main predictor of generalization (see Fig. for an illustration).
This offers an explanation for the correlation of flatness with generalization on many real-world data distributions for image classification [14, 21, 26), where the assumption of locally constant labels is reasonable (the definition of adversarial examples [35] even hinges on this assumption). This dependence on locally constant labels has not been uncovered by previous theoretical analysis [26, 37]. Moreover, we show that the resulting relative flatness measure is invariant to linear reparameterizztion and has a stronger correlation with generalization than other flatness measures [14, 21, 26). Other measures have been proposed that similarly achieve invariance under reparameterizations [21, 37], but the Fisher-Rao norm [21] is lacking a strong theoretical connection to generalization, our measure sustains more natural form than normalized sharpness [37] and for neural networks, it considers only a single layer, given by the decomposition of f (including the possibility choosing the input layer when idx). An extended comparison to related work is provided in Appdx. A.
The limitations of our analysis are as follows. We assume noise-free setting where for each x E X there is unique y y(x) y such that Pa (z) 1, and this assumption is also extended to

---

the feature space of the given model, i.e. we assume that o(z) o(x') implies y(z) y(r') for all x, x' X' and write y(æ) = (()())). Moreover, we assume that the marginal distribution Dx is described by a density function pr (z). that f(x)) 10(w, ((x)) = g( (w(x)) ) is local minimizer of the empirical risk on S, and that g. v/ & are twice differential. Quantifying the representativeness of a dataset precisely is challenging since the data distribution is unknown. Using results from density estimation, we derive a worst-case bound on representativeness for all data distributions that fulfill mild regularity assumptions in feature space ((X), i.e., a smooth density function Po(D) such dz and (=E@(x) Po(D) Po(D) (=)/||=[|** dz are well-defined and finite.
that S V2 (P)c))())]]= This yields a generalization bound incorporating flatness. In contrast to the common bounds of statistical learning theory, the bound depends on the feature dimension. The dimension-deeendenee a result of the interpolation approach (applying density estimation uniformly over all distributions that satisfy the mild regularity assumptions). The bound is consistent with the no-free-lunch theorem and the convergence rate derived by Belkin et al. [2] for a model based on interpolations. In practical settings, representativeness can be expected to be much smaller than the worst-case bound, which we demonstrate by a synthetic example in Sec. 6. Generally, it is bound that remains meaningful in the interpolation regime [1. 3, 24], where traditional measures of generalization based on the empirical risk and model class complexity are uninformative [22, 42].
Contribution. In summary, this paper rigorously connects flatness of the loss surface to generalization and shows that this connection requires feature representations such that labels are (approximately) locally constant, which is also validated in a synthetic experiment (Sec. 6). The empirical evaluation shows that this flatness and an approximation to representativeness can tightly bound the general- ization gap. Our contributions are: (i) the rigorous connection of flatness and generalization; (ii) novel notions of representativeness and feature robustness that capture the extent to which a model's interpolation between samples fits the data distribution; and (iii) novel flatness measure that is layer- and neuron-wise reparameterization invariant, reduces to ridge regression for ordinary least squares, and outperforms state-of -hethrrrrt flatness measures CIFAR10.
2 Representativeness
In this section, we formalize when a sample set S is representative for a data distribution D.
Partitioning the input space. We choose partition {V: l 1,..., [..). of X' such that each element of this partition V/2 contains exactly one of the samples r; from S. The distribution can then be described by set of densities pz(z) o; PPD(z) 1y, (2) with support contained in V; (where lv, (x) if E V; and otherwise) and with normalizing factor a; Jv, (z)ddr. Then the risk
ar Ez-p, [((f(x), y(x))] Since x; E V; for each i, we can change variables and consider density functions (E) = pz(2; + {) with support in neighborhood around the origin of X. The risk then decomposes as
decomposes as E()) = !s|
|9|
E()) - Σ ar Beway [l(f(1; VI, E))
Starting from this identity, we formalize approximation to the risk: In practical setting, the distribution Pp is unknown and hence, in the decomposition (2), we have unknown densities A; and unknown normalization factors a, We assume that each neighborhood contributes equally the loss, ie., we approximate each a; with iS Then, given a sample set S and an [S]-tuple A = (A)) of "local" probability density functions on X' with support supp(A) in a neighborhood around the origin 0x. we call the pair AS) c-representative for D with respect to a model f and loss & S,A)) I e, where
[s] Σ S Eg [[(f(2; + +), E))]
Eecp (e.,,)))))E(
(3)
If the partitions V; and the distributions X; are all chosen optimal so that the approximation a; exact and A; = A;. then ER S, A) 0 by (2). the support of each A; is decreased to the origin so that X do is Dirac delta function, then Exep S, (f, equals the
[ST
3

---

generalization gap. For density functions with an intermediate support, the generalization gap can be decomposed into representativeness and the expected deviation of the loss around the sample points:
[s|
Egen (f, =
-E-,, [e[[(x; + E) v(2; ((f()))(())))).)))))))))))))))))
S
The main idea of our approach to understand generalization is to use this equality and to control both representativeness and expected loss deviations for suitable -tuple of distributions A.
From input to feature space. An interesting aspect of e-representativeness. is that it can be considered in a feature space instead of the input space. For a model f = (w) o Φ) X + V we can apply our notion the feature space ((X) (see Fig. 1 for an illustration). This leads the notion of representativeness in feature space defined for an [S]-tuple A (A.) (Ss)iss of densities on d(X) by replacing Z; with cfr;) in (3), which we denote by E Rep S, A*). By measuring representativeness in feature space, this becomes notion of both data and feature representation. In particular, assumes that target output function y(((2)) also exists for the feature space. We can then decompose the generalization gap Egen (f) of = (vb Φ) into
[S]
E (J.,AA))))))
E
E +) + -(f((z))
S
1=1
The second term is determined by how the loss changes under small perturbations in the feature space for the samples in S. As before, for X₂ 6g the term in the bracket vanishes and (e S, A*) = Egen- But the decomposition becomes more interesting for distributions with support of nonzero measure around the origin. If the true distribution can be interpolated efficiently in feature space from the samples in S with suitable X so that E Rep SAA 0, then the term in the bracket approximately equals the generalization gap and the generalization gap can be estimated from local properties feature space around sample points.
(J,
Feature Robustness
Having decomposed the generalisation gap into representativness and second term of loss deviation, we now develop a novel notion feature robustness that is able to bound the second term for specific families of distributions A using key equation
(1). Our definition of feature robustness for a model f = (1b Φ) X' + y depends on a small number > 0, a sample set and a feature selection defined by matrix A E Rm of operator norm || < 1. With feature perturbations (z) = (I (-)A))(a (z) and
1 [S]
(f S,A)
IS (()))))))) vloa(=:))) 100---e((ee {(f()))
Σ
(4)
the definition of feature robustness is given as follows.
Definition 1. Let é: y x - R4 denote a loss function, and two positive (small) real numbers,
S C X x y finite sample set, and A E Ruxx matrix. A model f(x) (té +)(x) with x(X) C Rd is called ((6, S, S,A))e))featur robust, if (S, a.A) S S for all 0 < a S 6. More generally, for a probability distribution A on perturbation matrices in Th." we define
(t EA-A [ (5)
and call the model ((8, e) feature robust on average over A, if 8f s e 0 S 6.
Given a feature extractor Φ, feature robustness measures the performance t when feature values are perturbed (with constant feature extractor Φ). This local robustness at sample points differs from the robustness of Xu and Mannor [40] that requires a data-independent partitioning of the input space. The matrix A in feature robustness determines which feature values shall be perturbed. For

---

each sample, the perturbation is linear in the expression of the feature. Thereby, we only perturb features that are relevant for the output for a given sample and leave feature values unchanged that are not expressed. For & mapping into an intermediate layer of a neural network, traditionally, the activation values of neuron are considered as feature values, which corresponds to choice of A as projection matrix. However, it was shown by Szegedy et al. [35] that, for any other direction v E IR"", ||v|| = 1, the values (Ø(x), v) obtained from the projection Φ(x) onto v, can be likewise semantically interpreted as a feature. This motivates the consideration of general feature matrices A.
Distributions on feature matrices induce distributions on the feature space Feature robustness is defined in terms of feature matrices (suitable for an application of (1) to connect perturbations of features with perturbations of weights), while the approach exploiting representative data from Section 2 considers distributions on feature vectors, cf. (3). To connect feature robustness to the notion of e-representativeness. we specify for any distribution A on matrices A E I an -tpple AA (A:) of probability density functions X; on the feature space I with support containing the origin. Multiplication
feature matrix with a feature vector o(x;) defines feature selection A²(2;). and for each R" there is some feature matrix A with o(x;) + 2 Φ(z,) + AΦ(æ;) (unless (xx,) = 0). Our choice for distributions A on R" are therefore distributions that are induced via multiplication of feature vectors 6(x,) E R with matrices A R mmm sampled from a distribution on feature matrices A Formally, we assume that a Borel measure μad is defined by probability distribution A matrices mm We then define Borel measures μa on Ih" by (C(CC) = (A({A AΦ(z,) E C)) for Borel sets C C Rm. Then X is the probability density function defined by the Borel measure μt. As a result, we have for each that
EA-A [((((A(Z))) (()))) = EgnA, ((v()(z;)) + 2),v(((z;) + 2)]]
Feature robustness and generalization. With this construction and distribution AA the feature space induced by a distribution A on feature matrices, we have that
{(f) = Eemp(f. + E (
(5)
Here, can be any distribution on feature matrices, which can chosen suitably to control how well the corresponding mixture of local distributions approximates the true distribution. The third term then measures how robust the model is in expectation over feature changes for ~ A. In particular, if Kep (f, S, AA) ze O, then egen (f. Rs (f. A) and the generalization gap is determined by feature robustness. We end this section by illustrating how distributions on feature matrices induce natural distributions on the feature space. The example will serve in Sec. 5 to deduce bound on Rep (f, S,A) from kernel density estimation.
Example: Truncated isotropic normal distributions are induced by suitable distribution on feature matrices. We consider probability distributions KCs|ja(==)]l on feature vectors 2 R in the feature space defined by densities ka]lø(=))) (0, 2) with smooth -nvvarrann kernels, bounded support and bandwidth h:
1 || h
[1=;==1))
21
ky (n, 2) hm k
|<c |ch
(6)
with ||a>> = when ||= 2ill < h and O otherwise, and such that SrERm (=o, 2) dz 1 for all zo. An example for such kernel is truncated isotropic normal distribution with variance h I, (2;, =) = (2,,, (z)) The following result states that the densities in (6) can indeed be induced by distributions on feature matrices, which will enable us to connect feature robustness with c-representativeness.
Proposition 2. Let S (0(xz) |a; E S) be set of feature vectors in RT. With kn, defined as in (6), let >x(z) = kaljetz,n] (0,z) z) define an [S]-uupl Ag of densities. Then there exists distribution As on matrices in mmmxm of norm less than 8 such that for each = 1.....|.).
EANA, [((()(>))) v(())))))))))))))
Ee_a, ((((0(1))) + v))(((()))) E
The technical proof deferred to the appendix, but we describe the distribution As on matrices for later use: The desired distribution is defined on the set of matrices of the form rO for real number
5

---

and an orthogonal matrix O (i.e. = as product measure combining the (unique) The Haar
Haar measure on the set of orthogonal matrices ((m) with suitable distribution measure on (m)) induces the uniform measure on sphere of radius r via multiplication with vector length [17], and we choose measure on to match the radial change of the kernel kş-
Relative Flatness of the Loss Surface
Flatness is property of the parameter space quantifying the change in loss under small parameter perturbations, classically measured by the trace the loss Hessian Tr(H), where H is the matrix containing the partial second derivatives of the empirical risk with respect to all parameters of the model. In order to connect feature robustness (a property of the feature space) to flatness, we present how key equation (I) translates the empirical risk: For model f(x:, w) sb(w, ((x)) g( (ww(()) with parameters E Rdxm and g : Rd - ) a function on matrix product of parameters w and a feature representation X + Rm and any feature matrix Rmxm we have that
18)
Eempl wA,
f(v(w + ((;)),;
[8|
181
1
( w. ((x;) + Ab(x:)),, v.) - S
Subtracting Cemp(we (w, ((5)) = |s) E ((b(w, ((:)) yi). we can recognize feature robustmess (4) on the right side of this equality when labels are constant under perturbations of the features, iec. ((0A((z))
yt- In other words, flatness Eemp (w + v, Φ(S) Eemp (SS) describes the performance of a model function on perturbed feature vectors while holding labels constant. We proceed to introduce novel, but natural, loss Hessian-based flatness measure that approximates feature robustness, given that the underlying data distribution D satisfies the assumption of locally constant labels.
With W. (wa.)e E Rixm denoting the s-th row of the parameter matrix w, we let Ha (w, ((S) E Rm denote the Hessian matrix containing all partial second derivatives of the empirical risk (w, (SS) with respect to weights in rows w, and w.,, i.e.
H (w, (())) Eeem (FEmm(W,6(S) ((S))
(8)
ow,,00
Definition 3. For a model f(x, w) g(wó(z)) w E Rdx with twice differentiable function twice differentiable loss function and a sample relative flatness is defined by
d
(w)
(w., W.) Tr.H... ((S))
8,8/ml
where Tr denote the trace and (W,, W W. the scalar product of two vectors.
Properties of relative flatness (i) Relative flatness simplifies to ridge regression for linear models E E (X' Rd 9 id and id) and squared loss: To see this, note that for
fæz w)
any loss function f, the second derivatives with respect to the parameters w E Ie computes to and
82g Du,da, () Kyy... For E(j.v) |1 y/ the squared loss function, a2t/aga
8e
the Hessian is independent of the parameters In this case, KTr ||w| with constant
Exes (Tr(xx²) ), which is the well-known Tikhonov (ridge) regression penalty.
(ii) Invariance under reparameterization: We consider neural network functions
f(x) w'o x + +b') + b))...) 6
(10)
of neural network of L layers with nonlinear activation function o. By letting o' (xx) denote the composition of the first 1 layers, obtain decomposition f(r, w') gf w'')(x))) of the network. Using (9) obtain relative flatness measure (7,(w) for the chosen layer.

---

For well-defined Hessian of the loss function, we require the network function to be twice differ- entiable. With the usual adjustments (equations only hold almost everywhere parameter space), we can also consider neural networks with ReLU activation functions. In this case, Dinh et al. [5] noted that the network function and with the generalization performance- remains unchanged under linear reparameterization. i.e., multiplying layer 1 with a >> O and dividing layer A o by a, but common measures of the loss Hessian change. Our measure fixes this issue in relating flatness to generalization since the change of the loss Hessian is compensated by multiplication with the scalar products of weight matrices and is therefore invariant under layer-wise reparameterizations [cf. 26]. It is also invariant to neuron- wise reparaaeterrizations, i..., multiplying all incoming weights into neuron by a positive number a and dividing all outgoing weights by a [23], except for neuron- wise reparameterizations of the feature layer ø' Using simple preprocessing step (a neuron wise repa- rameterization with the variance over the sample), our proposed measure becomes independent of all neuron-wiss reparameterizations.
Theorem 4. Let denote the variance of the i-th coordinate of (xx) over samples E S and V = diog (o1,...,ome If the relative flatness measure Key, is applied to the representation
f(x) = of...o(ww'v (V--Iw/'o(.....
(4))...) V- +b)
then kyre is invariant under all neuron-wise (and layer-wise) reparameterizatioos
We now connect flatness with feature robustness: Relative flatness approximates feature robustness for a model at a local minimum of the empirical risk, when labels are approximately constant in neighborhoods of the training samples (ő(x), ((S) in feature space.
Theorem 5. Consider a model (xr w) g(wo(z)) as above, a loss function and a sample set S, and let Om C mm denote the set of orthogonal matrices. Let d be positive (small) real number and w = E Rdxm denote parameters af local minimum of the empirical risk on a sample set S. If the labels satisfy that y(oss(zz)) = v()(x:)) for all (Ii,, Us). E Sand all ||A||| S 1, then f(,,w) is ((ō, S, mm). e)-feature robust on average over Om = 2m
8"
(w) + (68))
Applying the theorem Eq. 5 implies that if the data is representative, i.e., Ehep (S.S, AA,)
for the distribution As of Prop. 2, then (f:,()(: W) S 2m (w) + ( The assumption on locally constant labels in Thm. 5 can be relaxed to approximately locally constant labels without un- raveling the theoretical connection between flatness and feature robustness. Appendix investigates consequences from even dropping the assumption of approximately locally constant labels.
8"
5 Flatness and Generalization
Combining the results from sections 2-4, we connect flatness to the generalization gap when the distribution can be represented by smooth probability densities on a feature space with approximately locally constant labels. By approximately locally constant labels we mean that, for small 6, the loss in (1r((x))) -leehhgbborooodddd around the feature vector of training sample x; is approximated (on average over all training samples) by the loss for constant label y(zx) on these neighborhoods, This and the following theorem connecting flatness and generalization made precise in Appendix D.4.
Theorem 6 (informal). Consider a model f(x,, w) = g(wá(x)) as above, loss function and sample set S. let m denote the dimension of the feature space defined by å and let ô be a positive (small) real number. Let w denote a local minimizer the empirical risk on sample set S. the distribution D has smooth density php on the feature space I." with approximately locally constant labels around the points x E S, then it holds with probability 1 - A over sample sets S that
(nt (1) + CL..). I.) C2lpE, 2m
Egen (f(,,), S |S|
VA
up to higher orders in | for constants C1, O2 that depend only on the distribution in feature space p% induced by b, the chosen [Spplpple As and the maximal loss L.
To prove Theorem we bound both e-representativeness and feature robustness in Eq. 5. For that, the main idea is that the family of distributions considered in Proposition 2 has three key properties: (i)
7

---

Conelnion becoun Tutroca and perio soa.-attttio
Geneniltziion Goneraiization Bound
0.3
0.1
-01
0.0
0.5
1.0
2.0
2.5
0.5
0.8
1.3
1.5
2.0
3.0
4.0
5.0
dog locally condiant labes
Class separation
Figure 3: The correlation between flatness and generalization increases with the degree of locally constant labels.
Figure 4: Approximation of representativeness via KDE together with relative flatness leads to tight generalization bound.
provides an explicit link between the distributions on feature matrices As used in feature robustness and the family of distributions As of e-representativeness (Proposition 2) (ii) it allows us to bound feature robustness using Thm. 5; and (iii) it is simple enough that it allows us to use standard results of kernel density estimation (KDE) to bound representativeness.
Our bound suffers from the curse of dimensionality, but for the chosen feature space instead of the (usually much larger) input space. The dependence on the dimension is a result of using KDE uniformly over all distributions satisfying mild regularity assumptions. In practice, for given distribution and sample set S, representativeness can be much smaller, which we showcase in a toy example in Sec. 6. In the so-called interpolation regime, where datasets with arbitrarily randomized labels can be fit by the model class, the obtained convergence rate is consistent with the no free lunch theorem and the convergence rate derived by Belkin et [2] for an interpolation technique using nearest neighbors.
combination of our approach with prior assumptions the hypotheses or the algorithm in accordance to statistical learning theory could potentially achieve faster convergence rate. Our herein presented theory is instead based solely interpolation and aims understand the role of flatness (a local property) in generalization: If the data is representative in feature layers and if the distribution can be approximated by locally constant labels in these layers, then flatness of the empirical risk surface approximates the generalization gap. Conversely, Equation 7 shows that flatness measures the performance under perturbed features only when labels are kept constant. As a result, we offer an explanation for the often observed correlation between flatness and generalization: Real-world data distributions for classification are benign in the sense that small perturbations in feature layers do not change the target class, i.e., they can be approximated by locally constant labels. (Note that the definition of adversarial examples hinges on this assumption of locally constant labels.) In that case, feature robustness is approximated by flatness of the loss surface. If the given data and its feature representation are further c-representative for small Rs O, then flatness becomes the main contributor to the generalization gap leading their noisy, but steady, correlation.
6 Empirical Validation
We empirically validate the assumptions and consequences of the theoretical results derived above 2 For that, we first show on a synthetic example that the empirical correlation between flatness and generalization decreases if labels are not locally constant, up to a point when they are not correlated anymore. We then show that the novel relative flatness measure correlates strongly with generalization, also in the presence of reparameterizations. Finally, we show in synthetic experiment that while representativeness cannot computed without knowing the true data distribution, it can in practice be approximated. This approximation -ltthuu technically not a bound anymore mnytore-tighlly bounds the generalization gap. Synthetic data distributions for binary classification are generated by sampling 4 Gaussian distributions in feature space (two for each class) with given distance between their means (class separation). We then sample a dataset in feature space 5 train a linear classifier
2 Code is available at https: http://www..aooooe./e/ea/aaaatta

---

s on the sample, randomly draw the weights of 4-yayer MLP o. and generate the input data as S (o *(S2) Sy). This yields dataset S and model f = Φ e such that ((S) has a given class separation. Details on the experiments are provided in Appdx. C.
Locally constant labels: To validate the necessity of locally constant labels, measure the correla- tion between the proposed relative flatness measure and the generalization gap for varying degrees of locally constant labels, as measured by the class separation on the synthetic datasets. For each chosen class separation, we sample 100 random datasets of size 500 on which we measure relative flatness and the generalization gap. Fig. 3 shows the average correlation for different degrees of locally constant labels, showing that the higher the degree, the more correlated flatness is with generalization. If labels are not locally constant, flatness does not correlate with generalization.
Approximating representative-
ness: While representativeness cannot be calculated without knowing the data distribution, it can be approximated from the training sample S by the error of a density estimation on that sample. For that, we use mul- tiple random splits of S into a training set Siain and Siea. train kernel density estima- tion on Stain and measure its er- ror on Stea Again, details can be found in Appx. C. The lower the class separation of the synthetic datasets, the harder the learning problem and the less represen- tative random sample will be. For each sample and its distribu- tion, we compute the generaliza- tion gap and the approximation to the generalization bound. The results in Fig. 4 show that the approximated generalization bound tightly bounds the generalization error (note that this approximation is technically not a bound anymore). Moreover, as expected, the bound decreases the easier the learning problems become.
FisnerRaa morm & Weight norm
10
20
30
40
50
60
90
7.5
0.77
7.0
po 0.29
0.81
6.5
8.0
set
5.5
5.0
Relative flatness
Pisher Rao norm Weight
PacBayye
5000
10000 Roalbva ristness Trace PacSayes
15000
20000
Figure 5: The generalization gap for various local minima corre- lates stronger with relative flatness than standard flatness, Fisher- Rao norm, PacBayes based measure and weights norm (points corresp. to local minima).
Relative flatness correlates with generalization: We validate the correlation relative flatness to the generalization gap in practice by measuring it for 110 different local catiimaa--hiieedd via different learning setups, such as initialization, learning rate, batch size, and optimization algorithm of LeNet5 [19] on CIFARIO [18]. We compare this correlation to the classical Hessian based flatness measures using the trace of the loss-Hessian, the Fisher-Rao norm [21], the PACBayes flatness measure that performed best in the extensive study of Jiang al. [14] and the L2-norm of the weights. The results in Fig. show that indeed relative flatness has higher correlation than all the competing measures. Of these measures, only the Fisher-Rao norm is reparameterization invariant but shows the weakest correlation in the experiment. In Appdx C we show how reparameterizations of the network significantly reduce the correlation for non-reparameetrrizaion invariant measures.
7 Discussion and Conclusion
Contributing to the trustworthiness of machine learning, this paper provides a rigorous connection between flatness and generalization. As be expected for local property, our association between flatness and generalization requires the samples and its representation in feature layers to be represen- tative for the target distribution. But our derivation uncovers a second, usually overlooked condition. Flatness of the loss surface measures the performance of model close to training points when labels are kept locally constant. data distribution violates this, then flatness cannot good indicator for generalization.

---

Whenever we consider feature representations other than the input features, the derivation of our results makes one strong assumption: the existence of a target output function (()(z)) on the feature space (( By moving assumptions on the distribution from the input space to the feature space, we achieve bound based on interpolation that depends on the dimension of the feature layer instead of the input space. Hence, we assume that the feature representation is reasonable and does not lose information that is necessary for predicting the output. To achieve faster convergence rates independent of any involved dimensions, future work could aim to combine our approach of interpolation with a prior-based approach of statistical learning theory.
Our measure of relative flatness may still be improved in future work. Better estimates for the generalization gap are possible by improving the representativeness of local distributions in two ways: The support shape of the local distributions can be improved and their volume- volume-parameter parameter d can optimally chosen. Both improvements will affect the derivation of the measure of relative flatness as an estimation of feature robustness for the corresponding distributions on feature matrices. Whereas different support shapes change the trace weighted average of the Hessian eigenvalues, the volume parameter can provide correcting scaling factor. Both approaches seem promising to us, as our relative measure from Definition 3 already outperforms the competing measures flatness in our empirical validation.
Acknowledgements
Cristian Sminchisescu was supported by the European Research Council Consolidator grant SEED, CNCS-UEFISCDI 1A--IIIPP-DD-PCEE0000----- 2020 grant DE- DE-ENIGMA (688835), and SSF.
DN--II-P-IDD-PCC--0011--001 the EU Horizon
Mario Boley was supported by the Australian Research Council (under DP210100045).
We would like to thank Julia Rosenzweig, Dorina Weichert, Jilles Vreeken, Thomas Gärtner, Asja Fischer, Tatjana Turova and Alexandru Aleman for the great discussions.
References
[1] Peter L Bartlett, Philip M Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. Proceedings of the National Academy of Sciences, 117(48):30033-30070 2020.
[2] Mikhail Belkin, Daniel Hsu, and Partha Mitra. Overfitting or perfect fitting? risk bounds for classification and regression rules that interpolate. In Advances in Neural Information Processing Systems, pages 2300-2311, 2018.
[3] Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical variinae trade-off. Proceedings of the National Academy of Sciences, (5499):55449158855555 2019.
[4] Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer T. Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. In Proceedings of the International Conference of Learning Representations, 2017.
[5] Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 1019-1028. JMLR. org, 2017.
[6] Gintare Karolina Dziugaite and Daniel M Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. AAAI, 2017.
[7] Gintare Karolina Dziugaite and Daniel M Roy. Data-dependent pac-bayes priors via differential privacy. In Advances Neural Information Processing Systems, pages 8430-8441, 2018
[8] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware mini- mization for efficiently improving generalization. In Proceedings of the International Confer- ence on Learning Representations, 2021.
10

---

[9] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 249-256. PMIR, 2010.
[10] Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. In Advances in Neural Information Processing Systems, pages 529-536, 1995.
[11] Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 9(1):1-42, 1997.
[12] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson, Averaging weights leads to wider optima and better generalization. In 34th Conference on Uncertainty in Artificial Intelligence, 2018.
[13] Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima sgd. arXiv preprint arXiv:171.004223 2017.
[14] Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. In Proceedings of the International Conference on Learning Representations, 2020.
[15] MC Jones, IJ McKay, and T-C Hu. Variable location and scale kernel density estimation. Annals of the Institute of Statistical Mathematics, 46(3):521-535, 1994.
[16] Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large- batch training for deep learning: Generalization gap and sharp minima. In Proceedings of the International Conference Learning Representationn, 2017.
[17] Steven G. Krantz and Harold R. Parks. Geometric integration theory. Springer Science and Business Media, 2008
[18] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[19] Yann LeCun and Corinna Cortes. MNIST handwritten digit database. Technical report, AT&T Labs, 2010.
[20] Yann LeCun, Bernhard E Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Handwritten digit recognition with back- propagation network. In Advances in Neural Information Processing Systems, pages 396-404. 1990.
[21] Tengyuan Liang. Tomaso Poggio, Alexander Rakhlin, and James Stokes. Fisher-rao metric, geometry, and complexity of neural networks. International Conference Artificial Intelligence and Statistics (AISTATS), 2019.
[22] Vaishnavh Nagarajan and Zico Kolter. Uniform convergence may be unable to explain generalization deep learning. In Advances in Neural Information Processing Systems, pages 11611-11622, 2019.
[23] Behnam Neyshabur, Ruslan Salakhutdinov, and Nathan Srebro. Path-sgd: Path-normalized optimization in deep neural networks. Advances in Neural Information Processing Systems, volume 28, 2015.
[24] Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. Workshop contribution at the International Conference Learning Representations, 2015.
[25] Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, pages 1376-1401, 2015.
[26] Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization deep learning. In Advances Neural Information Processing Systems, pages 5947-5956, 2017.

---

[27] Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha Sohl- Dickstein. Sensitivity and generalization neural networks: an empirical study. In Proceedings of the International Conference on Learning Representations, 2018
[28] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pages 8024-8035, 2019.
[29] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
[30] Henning Petzka, Linara Adilova, Michael Kamp, and Cristian Sminchisescu. A reparameterivaaion--nvaaa flatness measure for deep neural networks. In Workshop on Science meets Engineering of Deep Learning af NeurlPS, 2019.
[31] Akshay Rangamani, Nam H. Nguyen, Abhishek Kumar, Dzung T. Phan, Sang H. Chin, and Trac D. Tran. A scale invariant flatness measure for deep network minima. arXiv preprint arXiv: arXiv:/902.00 1902.02434, 2019.
[32] Levent Sagun, Utku Evci, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over pereppaaaetrizee neural networks. Workshop contribution at the International Conference on Learning Representations, 2019.
[33] Bernard W. Silverman. Density estimation for statistics and data analysis. Monographs on Statistics and Applied Probability. Chapman and Hall, 1986.
[34] Xu Sun, Zhiyuan Zhang, Xuancheng Ren, Ruixuan Luo, and Liangyou Li. Exploring the vulnerability of deep neural networks: A study of parameter corruption. arXiv preprint 20iv:20006.06620, 2020.
[35] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties neural networks. arXiv preprint 131.66312261999 2013.
[36] Andrey Nikolayevich Tikhonov, A. Goncharsky, V. V. Stepanov, and Anatolij Grigorevic Yagola. Numerical Methods for the Solution of Pll-Posed Problems. Mathematics and Its Applications. Springer Netherlands, 1995.
[37] Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Normalized flat minima: Exploring scale invariant definition of flat minima for neural networks using PAC-Bayesian analysis. In Proceedings of the 37th International Conference on Machine Learning, pages 9636-9647, 2020.
[38] Huan Wang, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. Identifying generaliza- tion properties in neural networks. arXiv preprint arXiv:1809.07402. 2018.
[39] Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust generalization. In Advances in Neural Information Processing Systems, volume 33, pages 2958 2988-296,,220 2020.
[40] Huan Xu and Shie Mannor. Robustness and generalization. Machine learning, 86(3):391-423, 2012.
[41] Zhewei Yao, Amir Gholami, Qi Lei, Kurt Keutzer, and Michael W Mahoney, Ressian-based analysis of large batch training and robustness to adversaries. In Advances in Neural Information Processing Systems, volume 32, 2019.
12

---

[42] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. Proceedings of the International Conference on Learning Representations, 2017.
[43] Chiyuan Zhang, Qianli Liao, Alexander Rakhlin, Brando Miranda, Noah Golowich, and Tomaso Poggio. Theory of deep learning iib: Optimization properties of sgd. arXiv preprint arXiv:1801.02254, 2018.
[44] Yaowei Zheng, Richong Zhang, and Yongyi Mao. Regularizing neural networks via adversarial model perturbation. arXiv preprint arXiv:2010.04925, 2020.
13


Reviews:
[{"review_id": "qnwyTGfgJ9", "reviewer": "Reviewer_5qca", "summary": "This paper describes a connection between flatness of minima and generalization in deep neural networks. The authors define a concept called feature robustness and show that it is related to flatness. This is derived through a straightforward observation that perturbations in feature space can be recast as perturbations of the model in parameter space. This allows the authors to define a flatness measure for minima in deep networks (this flatness measure is also invariant to rescalings of the layers in neural networks with positively homogenous activations). The authors combine their notion of feature robustness with epsilon representativeness of a function to connect flatness to generalization. They present empirical evaluations and comparison to other measures on CIFAR10.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is able to once again confirm the relationship between flatness and generalization in an empirical manner. The measure presented in this paper shows better correlation to generalization than other measures proposed in the literature. However the authors do not provide the details about which layer was used to divide the model into the $\\phi, \\psi$ decomposition. The authors also do not specify whether the correlation holds for every possible choice of $\\phi$ and $\\psi$ to decompose the model. These details will help us understand the flatness phenomenon better.\n\nThe key theorem relating generalization and flatness is Theorem 6 (following from equation 5) which decomposes the generalization gap for a function $f$ into the $\\epsilon$-representativeness of the feature mapping $\\phi (S)$ for the distribution $\\mathcal{D}$ and the feature robustness of the full model. The relationship between feature robustness and flatness follows directly from a Taylor expansion of the empirical loss, and the relationship between $\\epsilon$-representativeness and generalization follows by definition. Proposition 2 is helpful in showing a relationship between local truncated normal distributions and orthogonal matrices. However, can one show a result of the form - for every local distribution $\\lambda_i (\\xi_i)$ there exists a distribution on linear perturbations $A$ such that the equivalence between feature perturbations and sampling from the distribution holds in expectation? It seems to me that this is an important step in establishing the link between feature robustness and generalization. Moreover, are there specific functions $\\phi$ which make this possible, or does this hold for all feature mappings $\\phi$? The synthetic experiments in Figures 3 and 4 are encouraging but are still performed with deep networks. Do the authors expect this to hold for deep linear networks, for instance?\n\nExperiments on synthetic data showing the relationship between the feature robustness and $\\epsilon$-representativeness terms would be helpful. What is the contribution of each term to the generalization gap? Is there a tradeoff between them or are more robust feature maps $\\phi$ also more representative?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This is a theoretical paper that does not have immediate societal implications. The assumptions required for the analysis are outlined but a clearer discussion of the limitations of the conclusions is needed (this is only an upper bound on generalization, etc.)", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "20"}, "extra_text": {}}, {"review_id": "k6Aktbd7G9m", "reviewer": "Reviewer_zbyb", "summary": "The paper investigates the connection between loss-based flatness and generalization. The authors relate flatness to interpolation from representative data in the feature space, and use two notions to do that: 1) representativness and 2) feature robustness. They derive a new flatness metric -- a relative flatness -- and on a small set of empirical observations show that it correlates well with generalization and that it has some other desirable theoretical properties such as reparametrization invariance.", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I like the paper, it deals with and important problem, is pretty clearly written, and contains both theory and an empirical validation. Here are some of my detailed points, questions, and potential problems that could be addressed by the authors to make the paper better:\n\nTHE POSITIVES:\n1) the paper deals with an important question of the relationship between local loss Hessian based flatness and generalization. Flatness is a local property of the loss landscape, while generalization is a much higher level concept. A lot has been written on the connection, but as far as I can tell there hasn't been a satisfactory explanation for why the connection exists. This paper is a good step in that direction.\n2) The paper is pretty clearly written and I had an easy time following the argument.\n3) I appreciate that the authors ran an empirical validation of their theoretical results using a pretty large network and a non-trivial dataset (CIFAR-10) rather than just synthetic data.\n4) I like that the authors discuss the limitations of their approach, such as the assumption of constant labels.\n\nQUESTIONS AND WEAKER POINTS:\n1) Constant labels. I am a bit confused about the constant labels assumption. If I get it correctly, the assumption is that when you look at the partitioning of the input / feature space, as discussed from line 90 further, the cells are considered to have the same true label as the centroid of the cell? This seems pretty reasonable on the face of it, but I do wonder if small soft changes in the true label, e.g. from (1,0,0) to (1-e,e/2,e/2) as you move within the cell would do something to your theory? I think it is entirely plausible that the labels could be changing as judged by a human as a reference for the ground truth. Furthermore, how does the reference 35 use the constant labels? I didn't get that connection.\n\n2) The partitioning you are looking at could be e.g. a Voronoi partitioning (a minor point)? Do distances matter. Do shapes of the cells make any difference. Do they have to be convex? ​\n\n3) The verification in section 6 is only done on a single architecture and dataset, yet the validity of the measure is asserted generally. I think it would be better to do a broader set of experiments, including other data modalities and networks, given the breath of your theoretical claim. I am not suggesting you need to go to imagenet directly, but more diversity in architectures and dataset would be good to add.\n\nRELATED PAPERS?\nI remember seeing a proof relating the changes in weights to changes in features / logits in https://arxiv.org/abs/2002.03432 which might be relevant to you. Another paper that I got reminded of in your summary of the works on Tr(H) was https://arxiv.org/abs/1807.02581 .\n\nVERY  MINOR STUFF:\n1) I think  \\mathrm{Tr} for an operator looks better than Tr\n2) the colors in Figure 5 are hard to distinguish, could you use more different colors?\n\nUPDATE AFTER REPLY:\nI am happy with the detailed response and increase my score from 6 to 7. Thank you!\n\n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "1_0oRrXNQpc", "reviewer": "Reviewer_bRT2", "summary": "This paper develops a theoretical approach to tackling the relationship between flatness of the loss landscape around trained models and generalization. To accomplish this, the authors need to make certain assumptions about the data such as locally constant labels as well as its representativeness. The authors show that flatness with respect to the last layer parameters is related to feature robustness, and ultimately show that generalization performance can be bounded by their flatness quantity. The paper then presents a few experiments on synthetic datasets or small models to test the performance of the theory. They find that their measure performs similarly to others such as PacBayes and Trace.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Unless I misunderstood (which is definitely possible!) the theory only applies to models with a fixed feature embedding and where only the last layer parameters are trainable. If so, this limits the general applicability of the work and my excitement for it. It would be good for the authors to comment one way or another on this, perhaps in a limitations section which is missing. (Or not if I misunderstood.)\n\nWhat is the measure of correlation shown in Figure 5? Is it rank correlation? More details on the experimental setup would be appreciated.\n\nMy interpretation is that their measure works about as well as the other well-performing measures in the plot. Could the authors comment if the improvement should be considered significant?\n\nIn the relative flatness and trace measures in Figure 5, it looks like they both have a vertical “clump” of points that behaves very differently from the other points. Do the authors understand the origin of this clump? Is there a particular set of hyper parameters for which the measure does not seem to change? In some sense, this is worse performance to me than say PacBayes, which doesn’t seem to exhibit this.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed the limitations with respect to approximating representativeness, but a discussion of other limitations would be good. There is not a societal impacts section but I don’t know if that’s necessary for this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper provides a theoretical account of the relation between flatness of minima and generalization in deep neural networks. This is an important problem that attracted recent attention. The results presented here (Theorem 6) provide novel insight and are validated by experiments. The presentation is clear.
Decision: Accept (Poster)