Deep Learning is Singular, and That's Good
Authors: Daniel Murfet
Susan Wei
Mingming Gong
Hui Li
Jesse Gell-Redman
Thomas Quella
Venue:  (2021)
Keywords: deep learning theory
effective degrees of freedom
generalisation
posterior predictive distribution
real log canonical threshold
singular learning theory

Abstract: In singular models, the optimal set of parameters forms an analytic set with singularities and classical statistical inference cannot be applied to such models. This is significant for deep learning as neural networks are singular and thus ``dividing" by the determinant of the Hessian or employing the Laplace approximation are not appropriate. Despite its potential for addressing fundamental issues in deep learning, singular learning theory appears to have made little inroads into the developing canon of deep learning theory. Via a mix of theory and experiment, we present an invitation to singular learning theory as a vehicle for understanding deep learning and suggest important future work to make singular learning theory directly applicable to how deep learning is performed in practice. 

Under review as a conference paper at ICLR 2021
DEEP LEARNING IS SINGULAR, AND THAT'S GOOD
Anonymous authors Paper under couble-blind review
ABSTRACT
In singular models, the optimal set of parameters forms an analytic set with singu- larities and classical statistical inference cannot be applied such models. This is sigmificant for deep learning as neural networks are singular and thus "dividing" by the determinant of the Hessian or employing the Laplace approximation are not appropriate. Despite its potential for addressing fundamental issues in deep learning, singular learning theory appears to have made little inroads into the de- veloping canon of deep learning theory. Via a mix of theory and experiment, we present an invitation to singular learning theory as a vehicle for understanding deep learning and suggest important future work to make singular learning theory directly applicable to how deep learning is performed in practice.
INTRODUCTION
has been understood for close to twenty years that neural networks are singular statistical models (Amari et al.] 2003) Watanabe] (007). This means, in particular, that the set of network weights equivalent to the true model under the Kullback-Leibler divergence forms a real analytic variety which fails to be an analytic manifold due the presence of singularities. It has been shown by Sumio Watanabe that the geometry of these singularities controls quantities of interest in statistical learning theory, e.g., the generalisation error. Singular learning theory (Watanabe (209) is the study of singular models and requires very different tools from the study of regular statistical models. The breadth of knowledge demanded by singular learning theory Bayesian statistics, empirical processes and algebraic geometry is rewarded with profound and surprising results which reveal that singular models are different from regular models in practically important ways. To illustrate the relevance of singular learning theory to deep learning, each section of this paper illustrates key takeaway idea
The real log canonical threshold (RLCT) is the correct way to count the effective number of parameters in a deep neural network (DNN) (Section 4. To every (model, truth, prior) triplet is associated birational invariant known the real log canonical threshold. The RLCT can be understood in simple cases as half the number of normal directions to the set of true parameters. We will explain why this matters more than the curvature of those directions (as measured for example by eigenvalues of the Hessian) laying bare some of the confusion over "flat" minima.
For singular models, the Bayes predictive distribution is superior to MAP and MLE (Section 5). In regular statistical models, the 1) Bayes predictive distribution, 2) maximum a posteriori (MAP) estimator, and 3) maximum likelihood estimator (MLE) have asymptotically equivalent generalisa- tion error (as measured by the Kullback-Leibh illustrate in our experiments that even "being Bayesian" in just the final layers improves generali- sation over MAP. Our experiments further confirm that the Laplace approximation of the predictive distribution Smith & Le (2017); Zhang et al. (2018) is not only theoretically inappropriate but per- forms poorly.
divergence). This is not so in singular models. We
Simpler true distribution means lower RLCT (Section 6). In singular models the RLCT depends on the (model, truth, prior) triplet whereas in regular models it depends only the (model, prior) pair. The RLCT increases as the complexity of the true distribution relative to the supposed model increases. We verify this experimentally with simple family of ReL.U and SiLU networks.
'The code reproduce all experiments in the paper will he released Github. For now, see the zip file.

---

Under review as a conference paper at ICLR 2021
2
RELATED WORK
In classical learning theory, generalisation is explained by measures of capacity such as the l2 norm, Radamacher complexity, and VC dimension (Bousquet et al.) 2003). It has become clear however that these measures cannot capture the empirical success of DNNs (Zhang
2017 For instance, werrpaaameterised neural networks can easily fit random labels (Zhang et al. 2017, et al. 2018 Allen-Zhu et al. 2019b) indicating that complexity measures such as Rademacher complexity are very large. There is also a slate of work generalisation bounds in deep learning. Uniform con- vergence bounds (Neyshabur et al. 2015 Bartlett et al.) 2017; Neyshabur & Lil 2019) Arror et al. 2018) usually cannot provide non vacuous bounds. Data dett--deendentt bounds (Brutzkus et al.) 2018 Li & Liang 2018 Allen-Zhu et al.) (019a)) consider the "classifiabiltty" of the data distribution in generalisation analysis of neural networks. Algorithm-dependen bounds (Daniely 2017 Arora et al. 2019 Yehudaí & Shamir 2019 Cao & Gu 2019) consider the relation of Gaussian initial- Isation and the training dynamics of (stochastic) gradient descent to kernel methods (Jacot et al. 2018).
In contrast many of the aforementioned works, are interested in estimating the conditional distribution ((y(z). Specifically, we measure the generalisation error of some estimate án (ylz) in terms of the Kullback-Leibler divergence between q and Gn. see (8). The next section gives
crash course on singular learning theory. The rest of the paper illustrates the key ideas listed in the
introduction. Since we cover much ground in this short note, we will review other relevant work along the way, in particular literature on "flatness", the Laplace approximation in deep learning, etc.
SINGULAR LEARNING THEORY
To understand why classical measures of capacity fail to say anything meaningful about DNNs, it is important to distinguish between two different types of statistical models. Recall we are interested in estimating the true (and unknown) conditional distribution ((yla) with a class of models {p(vl æz w) w E W} where W C Rd is the parameter space. We say the model is identifiable if the mapping + p(y xz w) is one-to-one. Let ((x) be the distribution of æ. The Fisher information matrix associated with the model [p(y] ir, w) E W} is the matrix-valued function on W defined by
[log T(w)ij ʃ [log p(ul,, w)]]
dwy [log p(yl,, ()))(())))))))(()))))))
Ow:
if this integral is finite. Following the conventions in Watanabe 20099) have the following bifurcation of statistical models. A statistical model p(y) z. w) is called regular it is 1) identifiable and 2) has positive-definit Fisher information matrix. A statistical model called strictly singular if it is not regular.
Let (lw) be prior on the model parameters w. To every (model, truth, prior) triplet, we can associate the zeta function, ((z) K (KL) divergence between the model p x. w) and the true distribution g(ylz))
dw, & E C, where K(w) is the Kullback-Leibler
K(w) = // ((vl=) log ((Vl))
p(yl (r w) ((x) do dy.
For (model, truth, prior) triplet (p(ylæ, w). q(1)))))))))))))))()))))))))(q (p), let - be the maximum pole of the corre- sponding zeta function. We call À the real log canonical threshold (RLCT) (Watanabe) 2009) of the (model, truth, prior) triplet. The RLCT is the central quantity of singular learning theory.
By Watanabe 2009 Theorem 6.4) the RLCT equal to d/2 in regular statistical models and bounded above by d/2 in strictly singular models if realisabiliry holds: let
Wo (w W p(yl2 x, w) = [(1(>>)}
be the set true parameters, we say (tylar) is realisable by the model class if Wo is non-empty.
The condition of realisability is critical to standard results in singular learning theory. Modifications to the theory are needed in the case that ((y(z)) is not realisable, see the condition called relatively finite variance in Watanabe (2018).
Neural networks in singular learning theory. Let W C ddd be the space of weights of a neural network of some fixed architecture, and let f(x w) N x W a be the associated function.
2

---

Under review as a conference paper at ICLR 2021
We shall focus on the regression task and study the model
p(ulæ ar, w) (2 exp - f1x, w)ll²
but singular learning theory can also apply to classification, for instance. It is routine check (see Appendix A.T] that for feedforward ReI.U networks not only is the model strictly singular but the matrix [( (ww) is degenerate for all nontrivial weight vectors and the Hessian of K (w) degenerate
at every point of Wo.
RLCT plays an important role in model selection. One of the most accessible results in singular Bayesian information criterion (WBIC) Watanabe (2013), which we briefly review here for completeness. Let D, = {(T,, v)) be dataset of input-output pairs. Let T., (w) be the negative log likelihood
learning theory is the work related to the widely-
Ln(w) = - log pogp(y: (1, w)
and p(Dn)u)) = given by p(D,) Sw p(Dm| ((w)) dw and can be loosely interpreted as the evidence for the model. Between two models, we should prefer the one with higher model evidence. However, since the marginal likelihood is an intractable integral over the parameter space of the model, one needs to consider some approximation.
The marginal likelihood of model {p(ylæ, w) w W) is
The well-known Bayesian Information Criterion (BIC) derives from an asymptotic approximation of log lo (D,) using the Laplace approximation, leading to BIC = n.L,, (WMLE) + logn. Since want the marginal likelihood of the data for some given model to be high one should almost never adopt a DNN according to the BIC, since in such models d may be very large. However, this argument contains serious mathematical error: the Laplace approximation used to derive BIC only applies to regular statistical models, and DNNs are not regular. The correct criterion for both regular and strictly singular models was shown in Watanabe (2013) to be nI.,, (wo) + log n where wh E Wo and X is the RLCT. Since DNNs are highly singular À may be much smaller than d/2 (Section 6 it possible for DNNs to have high marginal likelihood consistent with their empirical success.
4 VOLUME DIMENSION, EFFECTIVE DEGREES OF FREEDOM, AND FLATNESS
Volume codimension. The easiest way to understand the RLCT is as volume codimension (Watan- abe [2009] Theorem 7.1). Suppose that W C Rd and Wo is nonempty, i.e., the true distribution is realisable. We consider a special case in which the KL divergence a neighborhood of every point vo Wo has an expression in local coordinates of the form
K(w) Σ C
(4)
where the coefficients Ca. > may depend on Un and d' may be strictly less than d. If the model is regular then this is true with d = d' and if it holds for d' < d then we say that the pair (p(y)lar, w), ()](x)) is minimally singular.
follows that the set Wo S W of true parameters is a regular submanifold of codimension d' (that is, Wo is manifold of dimension d - d' where W has dimension d). Under this hypothesis there are, near each true parameter Up E Wo. exactly d - d' directions in which to be varied without changing the model p(y ar, w) and d' directions in which varying the parameters does change the model. In this sense, there are d' effective parameters near Up.
This number of effective parameters can be computed by an integral. Consider the volume of the set of almost true parameters (tt,, vo) = JK( (e())dw where the integral is restricted to a small closed ball around Up. As long as the prior (w) is non-zero on Wo it does not affect the relevant features of the volume, so we may assume &p is constant on the region of integration in the first d' directions and normal in the remaining directions, so up to a constant depending only on d' we have
/
V (t, vo) x /CI Cd
(5)
3

---

Under review as a conference paper at ICLR 2021
and we can extract the exponent of t in this volume in the limit
log (V(at, vo)/V vo)}
= lim (-0
(6)
log(a)
for any > 0, + 1. We refer to the right hand side of (6) as the volume codimension vo-
The function K (wr has the special form 4 locally with d' - if the statistical model is regular (and realisable) and with d' in some singular models such reduced rank regression (Appendix A.2). While such a local form does not exist for a singular model generally (in particular for neural networks) nonetheless under natural conditions (Watanabe, 2009. Theorem 7.1) we have (t,, vo) ct^ + o(t²) where is a constant. We assume that in a sufficienlly small neighborhood of to the point RLCT vo (Watanabe 2009 Definition 2.7) is less than equal to the RLCT at every point in the neighborhood so that the multiplicity m = 1, see Section 7.6 of (Watanabe) (009) for relevant discussion. It follows that the limit on the right hand side of (6) exists and is equal to In particular À = d' 2 in the minimally singular case.
Note that for strictly singular models such as DNNs 2 may not be an integer. This may be discon- certing but the connection between the RLCT, generalisation error and volume dimension strongly suggests that 2λ nonetheless the only geometrically meaningful "count" of the effective number of parameters near Uop
RLCT and likelihood vs temperature. Again working with the model in Q consider the expec- tation over the posterior at temperature T as defined in (17) of the negative log likelihood (3)
E(T) = 1
1/ [nII (w)] B
luus S(xi, w)] + log(2x).
n.M
Note that when n is large (20) = M log(22) for any to E Wo so for T = 0 the posterior nMd
2
concentrates around the set Wo of true parameters and ETT) se 2 log(2r). Consider the increase AE = E(T + AT) E(T) corresponding to an increase in temperature AT' can be shown that AE = XAT where the reader should see (Watanabe 2013 Corollary 3) for precise statement. As the temperature increases, samples taken from the tempered posterior are more distant from Wo and the error E will increase. If À is smaller then for given increase in temperature the quantity E increases less: this is one way to understand intuitively why a model with smaller RLCT generalises better from the dataset D, the true distribution.
Flatness, It is folklore in the deep learning community that flatness of minima is related to gen- cralisation (Hinton & Van Camp [1993] [Hochreiter & Schmidhuber (997) and this claim has been revisited in recent years (Chaudhari et al. 2017 Smith & 2017 Jastrzebski et al. 2017 Zhang al., 2018). In regular models this can be justiñied using the lower order terms of the asymptotic expansion of the Bayes free energy (Balasubramanian 1997] $3.1) but the argument breaks down in strictly singular models, since for example the Laplace approximation of Zhang al. (2018) is invalid. The point can be understood via analysis of the version of the idea in (Hochreiter & Schmidhuber [997). Their measure of entropy compares the volume of the set of parameters with tolerable error to (our almost true parameters) standard volume
Vtoo vo) d/2
d = 2 log(tu) + Σ log
log
i=1
Hence in the case d d' the quantity - E. log(c.) ) measure of the entropy of the set of true parameters near Wo. point made for example in Zhang et al. (2018). However when d' < d this conception of entropy is inappropriate because the directions in which ((w) is flat near vo. which introduce the to dependence in (
5 GENERALISATION
The generalisation puzzle (Poggio et al.) 2018 is one of the central mysteries of deep learning. Theoretical investigations into the matter is an active area of research Neyshabur et al. (2017). Many of the recent proposals of capacity measures for neural networks
based on the eigenspectrum of the (degenerate) Hessian, e.g.. Thomas et al. (1.)(22) (2019); Maddox et al. (2020). But this is not appropriate for singular models, and hence for DNNs
4

---

Under review as a conference paper at ICLR 2021
Since are interested in learning the distribution, our notion of generalisation is slightly different, being measured by the KL divergence. Precise statements regarding the generalisation behavior in singular models can be made using singular learning theory. Let the network weights be denoted 0 rather than w for reasons that will become clear. Recall in the Bayesian paradigm, prediction proceeds via the so-called Bayes predictive distribution, p(y|2 D,) = p(ylor, commonly encountered in deep learning practice are the MAP and MLE point estimators. While in a regular statistical model, the three estimators 1) Bayes predictive distribution, 2) MAP, and 3) MLE have the same leading term in their asymptotic generalisation behavior, the same is not true in singular models. More precisely, let qn (y(2z) be some estimate of the true unknown conditional density g(yl a)) based on the dataset D... The generalisation ervor of the predictor dn (y(2r) is
d0. More
// ((y))) log ((vl(r)
G(n) = L x (ld.
((x) dy dz.
(8)
(n(u))))
To account for sampling variability, we will work with the average generalisation error, Gn)((), where En denotes expectation over the dataset D,,. By Watanabe (2009) Theorem 1.2 and Theorem 7.2), we have
E,, G,,) = \/n + o(1/n) if 9n is the Bayes predictive distribution,
where À is the RLCT corresponding to the triplet (p(y)æ: f), (tylx)) p(8)). In contrast, we should note that Zhang et al. (2018) and and [Smith & Le (2017) rely on the Laplace approximation to explain the generalisation of the Bayes predictive distribution though both works acknowledge the Laplace approximate is inappropriate. For completeness, a quick sketch of the derivation of (9) is provided in Appendix A.4 Now by (Watanabel 2009] Theorem 6.4) we have
(10) where C' (different for MAP and MIE) is the maximum of some Gaussian process. For regular mod- els, the MAP, MIE, and the Bayes predictive distribution have the same leading term for E,, G(n) since À = C' = d/2. However in singular models, C is generally greater than X, meaning we should prefer the Bayes predictive distribution for singular models.
G(n) = C/n n + o(1/n) if qn. is the MAP or MLE,
That the RLCT has such a simple relationship to the Bayesian generalisation error is remarkable. On the other hand, the practical implications of (99 are limited since the Bayes predictive distribu- tion is intractable. While approximations to the Bayesian predictive distribution, say via variational inference, might inherit a similar relationship between generalisation and the (variational) RLCT, serious theoretical developments will be required to rigorously establish this. The challenge comes from the fact that for approximate Bayesian predictive distributions, the free energy and generali- sation error may have different learning coefficients X. This was well documented in the case of neural network with one hidden layer (Nakajima & Watanabe 2007).
We set out to investigate whether certain very simple approximations of the Bayes predictive distri- bution can already demonstrate superiority over point estimators. Suppose the input-target relation- ship is modeled as in () but write 0 instead of w. We set q(z) (0, Σz). For now consider the realisable case, ((ylx) = p(y 0o) where On is drawn randomly according to the default initial- isation in Py PyTorch when model (2) is instantiated. We calculate E, G(n) using multiple datasets D,,
and a large testing set, see Appendix [A.5 for more details.
Since / is a hierarchical model, let's write it as fo(.) = hg((s);; w) with the dimension of w being relatively small. Let OMAP = (MMAP, WMAP) ) be the MAP estimate for 0 using batch gradient descent. The idea of our simple approximate Bayesian scheme is to freeze the network weights the MAP estimate for early layers and perform approximate Bayesian inference for the final layer e.g., freeze the parameters of g at UMAP and perform MCMC over w. Throughout the experiments,
g 3 + 3 is a feedforward ReLU block with each hidden layer having 5 hidden units and 3 > 3 either BAr or B RcLU(AA) where E 33xr, E R We set 3. We shall consider I or 5 hidden layers for g.
To approximate the Bayes predictive distribution, we perform either the Laplace approximation or the NUTS variant of HMC (Hoffman & Gelman] 2014) in the last two layers, i.e., performing inference over A, B in h(g(; UMAP); A, B). Note that MCMC is operating in a space of 18 di-
TTiis is similar in spirit to Kristiadi et al. (020) who claim that even "being Bayesian a little bit" fixes overconfidence. They approach this via the Laplace approximation for the final layer of ReLU network. It is also worth noting that Kristiadi et al. (220) do not attempt to formalise what it means to "fix overconfidenne", the precise statement should be terms of G(n).
5

---

Under review as a conference paper at ICLR 2021
Irikdken laperts) identiiy natwork
Siaada layerisi nig matwork iduniiy anliination
Savens NOME
0220
kast taoel 29 NONE
vithid
0.00
0.00
040
0322
610
400
505
1000
504
833
830
3000
bilden layerisi in netwo'k, I. SeLU activation retwork
hiddden layerist in networ ReUU octivation in in network
0042
00075
0.05
set lyger city rut mum..
0.033
lurt layer only lut Mear..
0000
0005
0013
0.00
03975
0002
1.003
<200 1200
0400 4.00
Figure 1: Realisable and batch gradient descent for MAP. Average generalisation errors G(n) displayed for various approximations of the Bayes predictive distribution. The results of the Laplace approximations are reported in the Appendix and not displayed here because they are higher than other approximation schemes by least an order of magnitude. Each subplot shows a different combination of hidden layers in (1 or 5) and activation function in /t (ReLU or identity). Note that the y-axis is not shared.
ErrorsE,,G(m)
Table 1: Companion to Figure 1
The learning coefficient is the slope of the linear fit 1/n versus E, G(2)) (no intercept since realisable). The R² value gives sense of the goodness-of-fit
(a) hidden layer(s) in g. identity activation in
(b) 5 hidden layer(s) in g. identity activation /
method
learning coefficient
R aquared
method
leamning coeffleiont
squared
last two layers (A,B MCMC
9, 709027
0.966124
last two bayers (A.D) MCMC
2828200
0,985161
last bayer only (B) MCMC
6.4 10380
088822
last layer only (B) MCMC
1.295544
0,982298
layers
Laplace
NaN
last layers (A.B) Laplace
NaN
last layer only (B) Laplace MAP
2154, 989,689266
001077
last layer only (B) Laplace
2008. 605589
0.801736
10.714216
0.91051
MAP
1.377733
0.963411
(c) hidden layer(s) in g. ReLU activation in
(d) 5 hidden layer(s) in g. ReLU activation in h
isethad
leaming coefficicut
squared
method
leamingg coefficiest
squared
Tast two layers (A. Bi MCMC
3,117187
0:973313
last two loy (A. B) MCMC
0.835993
0.957824
last layer only MCMC
3.152710
0.9801
layer (R) MCMC
4666277
0.920716
last two layars (A.B) Laplace last only (B) Liplace MAP
inf
NaN
last two layers (A. (A Laplace
inf
N&N
1120. 64806648288
0.742412
last layer only (Bi 1 Laplace
1416 1116.294258
0.80809
5,343311
0.972212
MAP
1.981483
0,889519
mensions in this case, which is small enough for us to expect MCMC to perform well. We also implemented the Laplace approximation and NUTS in the last layer only, i.e. performing infer- ence over B in highh (g(:; "MAP); AMAP); B). Further implementation details of these approximate Bayesian schemes are found in Appendix A.5
From the outset, we expect the Laplace approximation over w = (A, to be invalid since the model is singular. We do however expect the last-layer-only Laplace approximation over B to be sound. Next, we expect the MCMC approximation in either the last layer or last two layers to be superior the Laplace approximations and to the MAP. We further expect the last-two-layers MCMC to have better generalisation than the last-layer-only MCMC since the former is closer to the Bayes predictive distribution. In summary, we anticipate the following performance order for

---

Under review as a conference paper at ICLR 2021
these five approximate Bayesian schemes (from worst to best): mist-woo-laeer Laplace, last-layer- only Laplace, MAP, last-layer-only MCMC, last-two-layers MCMC.
The results displayed in Figure 1 are in line with stated expectations above, except for the sur- prise that the last-layer-only MCMC approximation is often superior to the last-two-layers MCMC approximation. This may arise from the fact that MCMC finds the singular setting in the last-two- layers more challenging. Figure [ the following factors: either 1 or layers in g. and 2) ReLU identity activation in h. Table D is a companion to Figure [ and tabulates for each approximation scheme the slope of 1/n ver- sus E, G(n), also known as the learning coefficient. The R² corresponding to the linear fit is also provided. In Appendix |A.5| we also show the corresponding results when 1) the data- date-gennrating mechanism and the assumed model do not satisfy the condition of realisability and/or 2) the MAP estimate is obtained via minibatch stochastic gradient descent instead of batch gradient descent.
we clarify the effect the network architecture by varying
6 SIMPLE FUNCTIONS AND COMPLEX SINGULARITIES
In singular models the RLCT may vary with the true distribution (in contrast to regular models) and in this section examine this phenomenon in simple example. As the true distribution becomes more complicated relative to the supposed model, the singularities of the analytic variety of true parameters should become simpler and hence the RLCT should increase (Watanabe) 2009) 37.6). Our experiments are inspired by (Watanabe) 2009 $7.2) where tanh(z) networks are considered and the true distribution (associated to the zero network) is held fixed while the number of hidden nodes is increased.
Consider the model p(yla, w) in (2) where f(z, c+ - q ReLLU((,, +b)) is two-layer
ReLU network with weight vector w (fw;) ) (b.) {q: 1 I: E R-#H+ and ui E R b, R. E R for 1 < i < H. We let W be some compact neighborhood of the origin.
Figure 2: Increasingly complicated true distributions qm (z. v) [11, 1| R.
Table 2: RLCT estimates for ReLU and SiLU networks. We observe the RLCT increasing as m increases, i.e., the true distribution becomes more "complicated" relative the supposed model.
Nonlimearity
RLCT
squared
3
ReLU
0.526301
0.027181
0.983850
ReLAl
0.590500
0024774
8.901341
SILU
0.339387
0.207669
0:988495
SILU
0.555630
0.021184
0.990971
Given integer m H we define network 8m. E W and qm (ylx) == plylar 8m) follows. Let S0(2) stand for rotation by 2n/ m/m. set = v (1, 0)" The components of sm are the vectors w; -i- w1 for 1 < i I m and w; = 0 for i > m, b, = -3 and = 1 for < i < m and b, q. 0 for i m, and finally decision boundaries lie within X 1|1, We let q(x) be the uniform distribution on X and define Jm (z, y) = gm gm (z) The functions f(r, sm.) are graphed in Figure 2 It is intuitively clear that the complexity of these true distributions increases with
O. The factor of ensures the relevant parts of the
We let ip be normal distribution N (0,5002) and estimate the RLCTs of the triples (p. 9m. s). We conducted the experiments with H 5, n 1000. For each (3, 4, 5), Table shows the

---

Under review as a conference paper at ICLR 2021
estimated RLCT. Algorithm 1 in Appendix A.3 details the estimation procedure which we base on [Watanabe 2013 Theorem 4). As predicted the RLCT increases with m verifying that in this case, the simpler true distributions give rise to more complex singularities.
Note that the dimension of W is d = 21 and so if the model were regular the RLCT would be 10.5. It can be shown that when dimension m. If such a model were minimally singular its RLCT would be {(44m + 1) m) = (3m + 1). In the case m 5 we observe an RLCT more than an order of magnitude less than the value 8 predicted by this formula. So the function K does not behave like quadratic form near Wo-
H the set of true parameters Wa S W is regular submanifold of
Strictly speaking it is incorrect to speak of the RLCT of ReLU network because the function K(w) is not necessarily analytic (Example A.44. However we observe empirically that the predicted linear relationship between [n, and 1/3 holds in our small ReLU networks (see the R² values in Table (2) and that the RLCT estimates are close to those for the two-layer SiLU network (Hendrycks & Gimpel 2016) which is analytic (the SiLU or sigmoid weighted linear unit is o (zz)
x(1 e -2) which approaches the ReLU as T - co. We use r = 100.0 in our experiments). The competitive performance of SiLU on standard benchmarks (Ramachandran et al. 2017) shows
that the non-analyticity of ReLU is probably not fundamental.
7 FUTURE DIRECTIONS
Deep neural networks are singular models, and that's good: the presence of singularities is neces- sary for neural networks with large numbers of parameters to have low generalisation error. Sin- gular learning theory clarifies how classical tools such as the Laplace approximation are not just inappropriate in deep learning narrow technical grounds: the failure of this approximation and the existence of interesting phenomena like the generalisation puzzle have a common cause, namely the existence of degenerate critical points of the KL function K K(w). Singular learning theory is promising foundation for a mathematical theory of deep learning. However, much remains to be done. The important open problems include:
SGD the posterior. A number of works (SimSekli 2017 Mandt et al. 2017 Smith et al. 2018) suggest that mini- batch SGD may be governed by SDES that have the posterior distribution as its stationary distribution and this may go towards understanding why SGD works well for DNNs.
RLCT estimation for large networks. Thcoretical RLCTs have been cataloged for small neural networks, albeit at significant effor() (Aoyagi (Aoyaii & Watanabe 22055]a). We believe RLCT estimation in these small networks should be standard benchmarks for any method that purports to approxi- mate the Bayesian posterior of a neural network. No theoretical RLCTs or estimation procedure are known for modern DNNs. Although MCMC provides the gold standard it does not scale to large networks. The intractability of RLCT estimation for DNNs is not necessarily an obstacle to reaping the insights offered by singular learning theory. For instance, used in the context of model selection, the exact value of the RLCT not as important as model selection consistency. We also demonstrated the utility of singular learning results such as ( and (10) which can be exploited even without knowledge of the exact value of the RLCT.
Real-world distributions are unrealisable. The existence of power laws in neural language model training (Hestness et al., 2017 Kaplan et al. (020) is one of the most remarkable experimental results in deep learning. These power laws may be sign of interesting new phenomena in singular learning theory when the true distribution is unrealisable.
REFERENCES
Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization overparameter- ized neural networks, going beyond two layers. In Advances in Neural Information Processing Systems, pp. 6155-6166, 2019a.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over- parameterization. In International Conference on Machine Learning, pp. 242-252, 2019b.
'Hironaka's 'ss resolution of singularities guarantees existence. However it is difficult to do the required blowup transformations in high dimensions obtain the standard form.
8

---

Under review as a conference paper at ICLR 2021
Shun-ichi Amari, Tomoko Ozeki, and Hyeyoung Park. Learning and inference in hierarchical mod- els with singularities. Systems and Computers in Japan, 34(7):34 42, 2003.
Miki Aoyagi and Sumio Watanabe. Resolution of Singularities and the Generalization Error with Bayesian Estimation for Layered Neural Network. In IEICE Trans., pp. 2112-2124, 2005a.
Miki Aoyagi and Sumio Watanabe. Stochastic complexities of reduced rank regression in Bayesian estimation. Neural Networks, 18(7):924 993, 2005b.
Sanjeev Arora, Ge, B Neyshabur, and Y Zhang. Stronger generalization bounds for deep nets via compression approach. In 35th International Conference on Machine Learning, ICML 2018, 2018.
Sanjeev Arora, Simon Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of op- timization and generalization for overparameterized two-layer neural networks. In International Conference on Machine Learning, pp. 322-332, 2019.
Vijay Balasubramanian. Statistical inference, Occam's razor and statistical mechanics on the space of probability distributions. Neural Computation, 9(2):3993363 1997.
Peter L. Bartlett, Dylan J Foster, and Matus Telgarsky. Spectrally-normalln normalized margin bounds for neural networks. In Advances in Neural Information Processing Systems, pp. 1240-6244 2017.
William M Boothby. An introduction differentiable manifolds and Riemannian geometry. Aca- demic press, 1986.
Olivier Bousquet, Stéphane Boucheron, and Gábor Lugosi. Introduction statistical learning the- ory. In Summer School on Machine Learning, pp. 169-207. Springer, 2003.
Alon Brutzkus, Amir Globerson, Eran Malach, and Shai Shalev- Shwaav-Shwartz. Sgd learns over- parameterized networks that provably generalize on linearly separable data. In International Conference Learning Representations, 2018.
Yuan Cao and Quanquan Gu. Generalization bounds of stochastic gradient descent for wide and deep neural networks. In Advances in Neural Information Processing Systems, pp. 10835-10845, 2019.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-SGD: Biasing gradi- ent descent into wide valleys. In International Conference on Learning Representations, 2017.
Amit Daniely. SGD learns the conjugate kernel class of the network. In Advances in Neural Infor- mation Processing Systems, pp. 2422-2430, 2017.
Simon S Du, Xiyu Zhai, Barnabas Poczos, and Aarti Singh. Gradient descent provably optimizes over-parameterized neural networks. In International Conference on Learning Representations, 2018.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (GELUs). arXiv preprint 1606v:1606.088 2016.
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun, Hassan Kianine- jad, Md. Mostofa Ali Patwary, Yang Yang. and Yanqi Zhou. Deep learning scaling is predictable, empirically, CoRR, abs/ 10tss/012200 2017.
Geoffrey E Hinton and Drew Van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings the sixth annual conference on Computational learning theory, pp. 5-13, 1993.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 9(1::1-42, 1997.
Matthew D Hoffman and Andrew Gelman. The To-U sampler: adaptively setting path lengths in hamiltonian monte carlo. J. Mach. Learn. Res., 15(1):1593-1625 2014.

---

Under review as a conference paper at ICLR 2021
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kemel: Convergence and gen- eralization in neural networks. In Advances in Neural Information Processing Systems, pp. 8571- 8580, 2018.
Stanislaw Jastrzebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in SGD, arXiv preprini 1rXiv:1711.04G22 2017.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020.
Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being Bayesian, even just a bit, fixes overconfidence in ReLU networks. arXiv preprint arXiv:2002.1011. 2020.
Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. In Advances in Neural Information Processing Systems, pp. 8157- 8166, 2018.
Wesley Maddox, Gregory Benton, and Andrew Gordon Wilson. Rethinking parameter counting in deep models: Effective dimensionality revisited. arXiv preprint arXiv:2003.02139 2020.
Stephan Mandt, Matthew Hoffman, and David Blei. Stochastic gradient descent as approximate Bayesian inference. The Journal of Machine Learning Research, 18(1):873349077, 2017.
Shinichi Nakajima and Sumio Watanabe. Variational Bayes Solution of Linear Neural Networks and Its Generalization Performance. Neural Computation, 19(4):1112-53, 2007.
Behnam Neyshabur and Zhiyuan Li. Towards understanding the role of over- aver-parametrizatioon in gen- eralization of neural networks. In International Conference on Learning Representations (ICLR), 2019.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control in neural networks. In Conference on Learning Theory, pp. 1376-1401, 2015.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring general- ization in deep learning. In Advances in neural information processing systems, pp. 5947-5956,
2017.
Jeffrey Pennington and Pratik Worah. The spectrum of the Fisher information matrix of a single- hidden-layer neural network. In Advances in Neural Information Processing Systems, pp. 5410- 5419, 2018
Mary Phuong and Christoph H. Lampert. Functional vs. parametric equivalence of ReLU networks. In International Conference on Learning Representations, 2020.
Tomaso A. Poggio. Kenji Kawaguchi, Qianli Liao, Brando Miranda, Lorenzo Rosasco, Xavier Boix, Jack Hidary, and Hrushikesh Mhaskar. Theory of deep learning III: explaining the non-overfitting puzzle. CoRR, abs/ 801 .00173, 2018.
Prajit Ramachandran, Barret Zoph, and Quoe V Le. Swish: self-gated activation function. arXiv preprint arXiv: 1710::/111005941 2017.
Levent Sagun, Léon Bottou, and Yann LeCun. Singularity of the Hessian in deep learning. CoRR, 0tss/6611.07477 2016.
Umut ŞimSckli. Fractional Langevin Monte Carlo: exploring Levy driven stochastic differential equations for Markov Chain Monte Carlo. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3200-3209, 2017.
Samuel L Smith and Quoc V Le. A Bayesian perspective on generalization and stochastic gradient descent. arXiv preprint arXiv: 1710.067100045/// 2017.
10

---

Under review as a conference paper at ICLR 2021
Samuel L Smith, Daniel Duckworth, Semon Rezchikov, Quoc V Le, and Jascha Sohl-Dickstein. Stochastic natural gradient descent draws posterior samples in function space. arXiv preprint arXiv:1806.09597, 2018.
Valentin Thomas, Fabian Pedregosa, Bart van Merrinboer, Pierre-Antoine Mangazol, Yoshua Ben- gio, and Nicolas Le Roux. Information matrices and generalization. arXiv:1906.0777 [cs, stat], 2019. arXiv: 1906.07774.
Sumio Watanabe. Almost All Learning Machines are Singular. In 2007 IEEE Sympositm on Foun- dations of Computational Intelligence, pp. 383-388, 2007.
Sumio Watanabe. Algebraic Geometry and Statistical Learning Theory. Cambridge University Press, USA, 2009.
Sumio Watanabe. A Widely Applicable Bayesian Information Criterion. Journal of Machine Learn- ing Research, 14:867-897, 2013.
Sumio Watanabe. Mathematical Theory of Bayesian Statistics. CRC Press, 2018.
Gilad Yehudai and Ohad Shamir. On the power and limitations of random features for understanding neural networks. In Advances in Neural Information Processing Systems, pp.
2019.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. In Proceedings of the 5th International Confer-
ence on Learning Representattinss 2017. arXiv: 1.11.03530.
Yao Zhang, Andrew M. Saxe, Madhu S. Advani, and Alpha A. Lee. Energy-entropy competition and the effectiveness of stochastic gradient descent machine learning. Molecular Physics, 116 (21-22):3214-3222 2018.
A APPENDIX
A.1 NEURAL NETWORKS ARE STRICTLY SINGULAR
Mayere neural networks are strictly singular (Watanabe) 2009 57.2). The degeneracy of the
Hessian in deep learning has certainly been acknowledged in e.g.. e.g..SSagun et al. (2016) which recog- nises the eigenspectrum concentrated around zero and Pennington & Worah (2018) which deliberately studies the Fisher information matrix of a single-hidden-layer, rather than multilayer, neural network.
We first explain how to think about neural network in the context of singular learning theory. A feedforward network of depth parametrises a function f N R the form
ʃ = Ar 0c-1 o Ac- ... a o A
where the A R - dh are affine functions and IR4 - Ti is coordinate-win some fixed nonlinearity a R + R. Let W be a compact subspace of RR containing the origin, where Rd is the space of sequences of affine functions (Ar) with coordinates denoted w....., Wa so that I may be viewed as a function ʃ : R-N x W - R.M. We define p(ylar, w:) as in 2 We assume the true distribution realisable, ((y(x)) = p(yla, wo) and that a distribution g(x) on R. is fixed with respect to which p(z, (p() on W we may apply singular learning theory to the triplet (p, g, sp).
ply xz) (z) and g(z, y) 4(y(z)((x). Given some prior
By straightforward calculations we obtain
K(w) 1
[lf(z, w) f(z, g())d
(1I)
K ((w)= Twidrey
/ Tnee
a (B f(z, w), ow, f(x, w) g(x)do
/ (5(x) (w f(x, wol, Twe f fxr w) )((x)dx
8²
(12)
I(w); 2 33M-3)/2 -2)/2 /
ou, (onn f(z, w); 2
ow; f(x, w) q(x)dr
(13)
11

---

Under review as a conference paper at ICLR 2021
where (-.-) is the dot product. We assume q(z) is such that these integrals exist.
will be convenient below to introduce another set of coordinates for W. Let w/y denote the weight from the kth neuron in the (1 - 1)th layer to the jth neuron in the /th layer and let b/ denote the bias of the jth neuron in the /th layer. Here S I с and the input is layer zero. Let ut and af denote the value of the jth neuron in the lth layer before and after activation, respectively. Let uf and a' denote the vectors with values u/ and ay respectively. Let d, denote the number of neurons in the Ith layer. Then
1
Σ
S + bj.
<<l<,, s S
with the convention that d = x is the input and wf - ys is the output.
In the case where a ReLU the partial derivatives
__1 do not exist all of RNN However Zwz
given E W we let D(w) denote the complement in R.V of the union over all hidden nodes of the associated decision boundary, that is
R. D(w)= U U
f 22((0) = 0}.
Isice Si
The partial derivative
Fwyy _ f exists the open subset {(x, w) D(w)) of of x W.
Lemma A.1. Suppose = ReLU and there are > layers. For any hidden neuron 1 S S di in layer I with 1 1S( l there is a differential equation
d1-1
d:t1
Σ
Wijh
Ow jk
b/
0b
Σ
1+1 ij
Ow
++1
}
-0
k=1
which holds on D(w) for any fixed W.
Proof. Without loss of generality assume M to simplify the notation. Let e, R denote A ReL.((.). Writing
unit vector and let H
(w)
85 3.iTT for a gradient vector
dz
0ʃ
0s
0ult1
0
Ou 11 -1 ((d)
05
41
Ow
1+1
Oui+I
ij
Tu
05 =
t1
0ʃ
0ulti
Ow
0ut+1
0s
dit
w
H(())e)) ak
= Σ
d1+1
Ou:
0s
[+1
w
1+1
0k
H(l!))
1=1
0f
0f 0ult1
0f
de+1
d
0f
Wij H(u)jes)
1+1
wij
1+1
H H(()))
06;
+T
Oul+1
Ou 1+1
The claim immediately follows.
Lemma A.2. Suppose a ReLU, > and that W has at least one weight or bias hidden node nonzero. Then the matrix I (w) is degenerate and if w E Wo then the Hessian of K at is also degenerate.
Proof. Let E W be given, and choose hidden node where at least one of the incident weights a (or bias) is nonzero. Then Lemma A.1 gives a nontrivial linear dependence relation E, A Dun, f = as functions on D(w). The rows of (w) satisfy the same linear dependence relation. At true parameter the second summand in 12) vanishes so by the same argument the Hessian is degenerate.
Remark A.3. Lemma A.2 implies that every true parameter for a nontrivial ReLU network is a de- generate critical point of K. Hence in the study of nontrivial ReLU networks it is never appropriate to divide by the determinant of the Hessian K at a true parameter, and in particular Laplace or saddle-poing approximations at true parameter are invalid.
12

---

Under review as a conference paper at ICLR 2021
The well-known positive scale invariance of ReLU networks (Phuong & Lampert 2020 is responsi- ble for the linear dependence of Lemma A.. in the precise sense that the given differential operator is the infinitesimal generator (Boothby 1986 gIV.3) of the scaling symmetry. However, this is only one source of degeneracy or singularity in ReLU networks. The degeneracy, as measured by the RLC.T, is much lower than would expect the basis of this symmetry alone (see Section (
Example A.4. In general the KL function (w) for ReLU networks is not analytic. For the minimal counterexample, let (ax) be uniform on [-N,N and zero outside and consider
((b) - ((>)( b) ReT.U())))
It is easy to check that up to a scalar factor
63 + b 0 5 N
(bb) 3 1s + 1
-N < <b00
so that K is C² but not C let alone analytic.
A.2 REDUCED RANK REGRESSION
For reduced rank regression, the model is
1 2. y
ply|x, w)
(2 exp
BAxl
where E R . N A an M H matrix and B an H x N matrix; the parameter w denotes the entries of and B, f.e. w = (A, B), and a > 0 a parameter which for the moment is irrelevant.
If the true distribution is realisable then there is we (Ao, Bu) such that (z/lz)) p(y|æ wo)- Without loss of generality assume q(x) is the uniform density. In this case the KL divergence from p(y|z, w) to ((y(a)) (z) is
K(w) = ((ul)) log (111()) drdy = |BA Bo (1 + E(w))
ply xr, w)
where the error E is smooth and E(w) 0 BA BoAuul 3) in any region where || [BA- Bo
C, so K ((w) is equivalent to | Bo We write K(w) = BA - Bo 2 for simplicity below.
Now assume that Ag is symmetric and that Bg is square, i.e. N = H. Then the zero locus of ((w) is explicitly given as follows
Wo = {(A, B) det B + and A = B AoAo-
It follows that Wo is globally graph over GL(H;R) R). Indeed, the set (B- B) with B E GT(H;; R) is exactly Wo. Thus Wo is smooth dim-dinnni submanifold of RH x
RHxM To prove that Wo is minimally singular in the sense of Section 4 it suffices to show that rank(D kk) 2 HM where D3, D denotes the Hessian, but as it no more difficult do so, we
find explicit local coordinates (u, v) near an arbitrary point (A, B) E Wo for which {u 0} = Wo
and K (u, v) a(u,v) u)|u in this neighborhood, where a is C function with a > c 0 for
some c. Write
A(v) = -B+
Then u, c + (A(u) + u, B + v) gives local coordinates on H2² x IR x near (A, B), and
K(u, v) =IB ( + - BB + u) BoAdl
|Bo Ao + (B + w)u Ao
=I + u)ul?
so for sufficiently small (and hence B + invertible) we can take alu, v) = BB + v)ul2/ 1112.
A.3 RLCT ESTIMATION
In this section we detail the estimation procedure for the RLCT used in Section 6 Let L,, (w) be the negative log likelihood as in (3). Define the data likelihood inverse temperature / > 0 to be
13

---

Under review as a conference paper at ICLR 2021
Algorithm 1 RLCT via Theorem 4 in Watanabe (2013)
Input: range of f's, set of training sets T each of size n, approximate samples {wi...., wa} from p" (2,) for each training set D,, and each
for training set D,, E 7 do for 8 in range of [3's do
Approximate 18 (w)) with 1 R E. (wir) where w1.... wn are approximate sam- ples from p^ (D,) end for Perform generalised least squares to fit X in (18), call result ((D,)
end for Output: 1 ED.err ((D,)
(Dalw) = I w) which can also be written
pd (D,, (w) = exp(-83nnnww)) The posterior distribution, at inverse temperature B, is defined as
(14)
pd (w) =T= [r,, ((w)
(D, (w)(ww))))))))))))))))))))) (D.)
(15)
Jw IT IP(: (rs w) p(w) where yp is the prior distribution on the network weights and / p² ( (Dnlw))(u) dw w
(D,))
(16)
is the marginal likelihood of the data at inverse temperature 3. Finally, denote the expectation of random variable R(w) with respect to the tempered posterior pd (w.(.)) as
[[[[]]]= /
R(w)p" (u)))) dw
(17)
In the main text, we drop the superscript in the quantities p(D,) rather than p² (D,)).
16
when 3 = 1, e.g.,
Assuming the conditions of Theorem 4 in Watanabe (0133 hold, we have
A + Op())
EC [nLn(w)) = nLn, (wo)
+
Un
(18)
B 23
where 30 is a positive constant and Un, a sequence of random variables satisfying U, = 0. In Algorithm I we describe an estimation procedure for the RLCT based on the asymptotic result in (18).
For the estimates in Table the posteriori distribution was approximated using the NUTS variant of Hamiltonian Monte Carlo (Hoffman & Gelman 2014) where the first 1000 steps were omitted and 20, 000 samples were collected. Fach A(D,,) estimate in Algorithm was performed by lin- car regression on the pairs {[1/B, E [nL,, {)))) where the five inverse temperatures 3 are centered on the inverse temperature 1/ 12g(20))
A.4 CONNECTION BETWEEN RLCT AND GENERALISATION
For completeness, sketch the derivation of 9 which gives the asymptotic expansion of the average generalisation error E, G()) of the Bayes prediction distribution in singular models. The exposition is an amalgamation of various works published by Sumio Watanabe, but is mostly based on the textbook (Watanabe) 2009).
To understand the connection between the RLCT and G(n), we first define the so-called Bayes free energy as
F(n) = - log.lopp)) (D,.)
whose expectation admits the following asymptotic expansion Watanabe 2009):
E,, (n) E, + A logn + ologn)
14

---

Under review as a conference paper at ICLR 2021
where S,, - related to the Bayes free energy as follows
Ei= l (z:) is the entropy. The expected Bayesian generalisation error is
(r) = EF + (F((m)
Then for the average generalisation error, we have
(,,) A/n + o(1/n).
(19)
Since models with more complex singularities have smaller RLCTs, this would suggest that the more singular a model is, the better its generalisation (assuming one uses the Bayesian predictive distribution for prediction). In this connection it is interesting to note that simpler (relative to the model) true distributions lead to more singular models (Section
A.5 DETAILS FOR GENERALISATION ERROR EXPERIMENTS
Simulated data The distribution of R is set to ((x) N(0, Jz). In the realisable case, y E R3 is drawn according ((y(z)) (z) = p(yl ar, 0o). In the nonrealisable setting, we set g(y(z:) xx exp(-1la) hwo (z)11//2,, where wig (Ao. Bo) is drawn according to the PyTorch model initialisation of h.
MAP training The MAP estimator is found via gradient descent using the mean-squared-error loss with either the full data set or minibatch set to 32. Training was set to 5000 epochs. No form of early stopping was employed.
Calculating the generalisation error Using held-out-test set Tw {(211, (()))))) we calculate the average generalisation error as
n2 log (0/l1(1))
Ry n
log án (v/lo())
(20)
Assume the held-out test set is large enough so that the difference between E, G(n) and 20 is neg- ligible. We will refer to them interchangeably as the average generalisation error. our experiments use n = 10, 000 and 30 draws of the dataset D, to estimate E.,.
Last layer(s) inference Without loss of generality, we discuss performing inference the w pa- rameters of h while freezing the parameters of at the MAP estimate. The steps easily extend to performing inference over the final layer only of f = h o Let ü; = DEMAP (x:). Define a new transformed dataset D,, {(T,, w)) We take the prior on w to be standard Gaussian. Define the posterior over w given D, as:
p(w)D,)) x p(D,)/))p(()) = mh exp( Ilye - h. (f.))12/2)((())
(21)
Define the following approximation to the Bayesian predictive distribution
ply pr D.)) = / plul,, (UMAP, w)) (D,)) dw.
Let wn be some approximate samples from p(w((x). Then we approximate plyla, Dn)) with
R Σ plylo, (UMAP, w,))
where a large number, set to 1000 our experiments. We consider the Laplace approximation and the NUTS variant of HMC for drawing samples from p(w[D,):
Laplace in the last layer(s) Recall MMAP (UMAP, WMAP) is the MAP estimate for fo trained with the data D,,. With the Laplace approximation, we draw wi,... wR from the Gaussian
NwMMAP Σ)
where Σ = (- log p(w (el(w/D,r (NAA )-1 is the inverse Hessiar of the negative log poste- rior evaluated at the MAP estimate of the mode.
MCMC in the last layer(s) We used the NUTS variant of HMC draw samples from (21) with the first 1000 samples discarded.. Our implementation used the pyro package in PyTorch.
'Following Kristiadi al. 2020 the code for the exact Hessian calculation is borrowed from https: t/g.thub.comm///
/bbp
15

---

Under review as a conference paper at ICLR 2021
hrldien laperto)
S.aadinn layerisi
Saaes NOME
0.12
kest teoilayers 29 NONE
0.00
8.23
1000
504
833
830
3000
Milden layerisi in netwo'k I. SeLU activation retwork
hidden layerist in neworr ReUU octivation in in network
0.119
ket lyger mum..
lurt layer only jut Meae..
9150
8.04
0125
2.23
0003
0.00
0109
120 2:0
C20000 4720
Figure 3: Realisable and minibatch gradient descent for MAP training.
Table 3: Companion to Figure
(a) hidden layer(s) in g. identity activation in
(b) 5 hidden layer(s) in g. identity activation in h
method
leamnnng enaflicient
R aquared
mothod
leaming coefflciost
squared
last two layers (A, B) MCMC
36.721 36.721594
0.992839
Jast two layers (A.D) MCMC
13,729278
0.921049
last luyer only (B) MCMC
20,678920
0.9836995
last Layer only (B) MCMC
9:170642
0.345613
layers (A.B) Laplace
NaN
last two layes (A.B) Laplace layer only Laplace
inf
NaN
layer only Laplace MAP
1768.655088
0388035
1943,793236
0.794669
NaN
MAP
14.123308
0.917302
(c) hidden layer(s) in g, ReLU activation
(d) hidden layer(s) g. ReLU activation h
methhed
loaming coefficient
R aquared
mothed
leaming cocfficiost
R squared
Last two layers (A,H) MCMC
22,
0.955400
last two layers (A,H) MCMC
4 4.652433
0.9222033
last layer only MCMC
10.675455
0.968584
(B) MCMC
3,533366
0,862125
last layers
Laplace
NaN
two layers (A.D) Laplace
last layer only (B) Luplace MAP
inf
NaN
last layer only (B) Laplace
10048852677
0.901399
35.647464
0.983284
MAP
6.256896
0,940437
16

---

Under review as a conference paper at ICLR 2021
.rildien leperto)
layelssi
D293
melieel
Saaes NCME
Fast teoltreers Da MON:
8.08
0.5
0.233
mm/
0.30
0.0
500
1000
893
3006
1ilden layerisi in netwo'k. I. SeLU activation retwork
hiddinn layerist in netwrr ReUU uctivation in in network
0:045
ket lyger biy mun..
0:033
turt layer only jut seae..
0.22
2.20
6835-
4200 1200
C200 470
Figure 4: Nonrealisable and full batch gradient descent for MAP training.
Table 4: Companion to Figure 4 The learning coefficient the slope of the linear fit 1/n versus
E, ((n) (with intercept since nonrealisable).
(a) hidden layer(s) in identity activation in
(b) 5 hidden layer(s) in identity activation in h
leamning coefficient
R squared
leaming coefficient
squared
layers (A,B) MCMC.
1086022
0.301991
last
(A.B) MUMC.
00806001
0.144260
layer only
last Layer only MCMC
0.121686
last two layers (A, B) Laplace
N&N
NaN
last two layers (A.B) Laplace
NaN
NaN
last layer only (B) Laplace MAP
30,692254 12,947999
0,029238
last Layer only (B) Laplace MAP
-33 817429
0,809044
0.90175
1,204743
0.242671
(c) I hidden layer(s) in g, ReLU activation in h
(d) 5 hidden layer(s) in ReLU activation in h
method
leurning cuefficient
squared
methud
leaming voeffisient squared
Last layers
layers
0,798055
last loyer only (B) MCMC
5,3846866
0,003113
last layer only (R) MCMC
1,141500
0.162585
last two layers (A,B) Laplace
NaN
NiN
last two layers (A.H) Laplace
NaN
NEN
last layer only Luplace
38.62910
0.099012
only (B) Laplace
-5.682602
0,000365
0.816794
1.6480733
0.284088
17

---

Under review as a conference paper at ICLR 2021
hilkkenn laperts)
hvadia layerisi
Saaes NOME
kast teoilayers 29 NONE
0.10
0.00
0.00
500
1000
504
893
839
3000
1ilden layerisi in netwo'k. I. SeLU activation retwork
5 hidddn layeris in neworr ReUU uctivation on in network
0.00
0.00
ket lyger mum..
turt layer only lut Meae..
0.50
0.33
8.03
0.05
0023
0.01
120 100-00-3
CW0.E 470
Figure 5: Nonrealisable and minibatch gradient descent for MAP training. Missing points on the MAP learning curve due to estimated probabilities too close O.
Table 5: Companion to Figure 5 The learning coefficient is the slope of the linear fit 1/n versus
E,, ((n) (with intercept since nonrealisable).
(a) hidden layer(s) in g, identity activation in
(b) 5 hidden layer(s) in g, identity activation in h
method
learning coefficient
R squared
methud
leuming voefficient
R squared
last two layers (A, MCMC
10000023
0.909991
last two layers (A.H) MCMC
0.808601
0.144260
last loyer only (B) MCMC
7,377871
0.978224
last layer only (R) MCMC
0.799114
0,127686
layers Laplace
NaN
last layers last layer only (B) Laplace MAP
Laplace
NaN
NNN
layer only Laplace MAP
33.8177299
12,947999
0.970173
1.204743
0.242671
(c) hidden layer(s) in g. ReLU activation in
(d) 5 hidden layer(s) g. ReLU activation in h
method
leamning cuefficient
R squared
method
leaming woeffisient
squared
Tast two layors (A.R) Ry MCMC
5.987187
0 (.548490
last two leyers (A, R) MCMC
O. 7949155
0.088305
layer only MCMC
5.3845386
0,501313
layer (B) MCMC
1.141580
0.162585
last layers
Laplace
NiN
two layers (A.B) Laplace
NaN
tast Inyer only (R) appaace MAP
38.629167
0/990122
last layer only (R) Laplaze
682662
0.000365
8.561722
01816794
MAP
1.6480773
0.264088
18

---

Under review as a conference paper at ICLR 2021
hrddiann laperto)
iduntit activation nutwoekk
holdan layerjsi natwerk iduntity csation rabwork
redth33 leyers
Leparce
300
239
606
400
500
623
C3e
3000
200
255
833
139
2000
layerisl in actevatin
ayenss in
network
50-
mithed lust lecs 14D1 lupkos
last bno levers la aapic lea Lapkes
lackeaee Ledias
620 C26
1000
200
pio
Figure Realisable and full batch gradient descent for MAP. average generalisation errors of Laplace approximations of the predictive distribution. The last-two-layers Laplace approximation results in numerical instabilities due to degenerate Hessian. Any missing points are due to estimated probabilities too close to O.
laparfs) in necnor idenlliy Givaionii nelbook
Shidkan layolsl mg melwwrk idutiiy attiaaion ralnock
bor not Uy splace ber oyer only 00 apal
lnd buen appiee lyper oil (i) vapkce
22
5.0
1.idden layerisi netwo'k, ReLU activatio retwork
Shidden layenst in networ ReLU activation nebwork
restisd
Lasiace
1200
Figure 7: Realisable and minibatch gradient descent for MAP training. Details are same as for Figure
19

---

Under review as a conference paper at ICLR 2021
1 hiddies loyorto) g necaor identiiy us sctivation in natwork
5 halian layelss) nig matwork, iduntiyy aci aciivaion in rabwork
laver Ladioce
laver thi Laiaace
3.5
layerlat BeLU acthvtion
Shidklen
ReUU netwark
reclis
4.53
Sy snpacce
2.53
2.5
102
Laghen
100
3:1h 1231
200
300
Figure 8: Nonrealisable and full batch gradient descent for MAP training. Details same for Figure
1 Irdddian laports) in mecnr identtiy us selivalioni in metlookk
5 Siibbankaeelal my malwark iluntiyy attivation in staoek
bur ayeer only do aplcs
Ind only (d) valice
HAA
4.-
activtion network
Shidden
ReUU network
method lost Lapitce
netiod last Lapiace
20
100
66. 5uh
1000
830
Figure 9: Nonrealisable and minibatch gradient descent for MAP training. Details are same for Figure
20


Reviews:
[{"review_id": "ioMbxZGoJrB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is a terse account of singularity of deep learning with a probabilistic view. Clearly written, gives an overview of the contributions and related work quickly and dives into the setup. The main byproduct of singularity is the inapplicability of classical methods. This is no news to many people in the field, yet I find the perspective provided in this work fresh and I think it has potential for further developments, although its current applicability is limited and it doesn’t say something that was not already known. Here are further comments:\n\n- The paper would benefit a lot from clearly laying out the concepts and definitions. Especially section 3 would benefit a lot from such clarity and would help a wider audience to follow the work. \n- The end of section 3 contains the key idea and would benefit from further clarity.\n- How does the measure compare with the other frameworks? What does it imply for existing models (examples)?\n- What is the relevance of the tasks and experiments at the end of section 5? How can one move from regression to high dimensional classification tasks?\n- In light of the under/over-parametrization debate, can the framework account for such a phase transition, or can the experiments reflect this?\n- What are the promises of this framework? What benefits would people in the community expect if they study this?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "abyT4sGG61Q", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies the connection of Deep Learning to Singular Learning Theory making the claim that the later could be a good foundation of the theory of Neural Networks as they are singular models. \n\n+++++++\n\nWhile I enjoyed the primer to singular learning theory, I found the paper's contribution marginal given the related work. The claim 'deep learning is singular' has already been claimed by Watanabe 07' in a much more general statement.  Moreover, each of the claims the authors made is unconvincing both from a theoretical and experimental perspectives. \nFor instance, the claim that RLCT governs the effective number of parameters. While the math looks sound to me, it has both has many strong assumptions on the model and lack of novelty given Watanbe 09'. To show that the calculation is meaningful, I would suggest showing the relationship to a real neural network and see how close the estimate to the real number of parameters (E.g,. in the random features (Rahimi & Recht 2008) model, the number of features is a good way to measure the effective number of parameters. Another way to estimate the real number of parameters is the location of the double descent peak (Belkin 18'). \n\nRegarding the empirical claims, the experiments are not convincing. In order to make a claim about deep learning, (say the bayes predictive error is superior to MAP or MLE) there should be either an extensive experimental demonstration of the claim (not two experiments) with proper ablation of when the claim fails (not to mention that the std of the experimental results puts all the claims to question). Same comments apply to the last section.\n\nTo conclude, my main comment is that while Deep Learning is singular (which is not a deep claim) to make the statement that the right way to study deep learning is by singularity theory, I would like to see either experiments or theory that give me insights about deep learning. This could be done by example by doing a theory on a toy model and showing that it holds for real deep learning applications (or at least for an array of synthetic distributions). I would like to qualify by saying that statistics is not my main field of study and I would be happy to receive clarifications if I misunderstood anything. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "XBy1AjYYhaA", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is more like a review of singular learning theory and its implication on deep learning. The authors point out that deep neural networks are singular models and ways to characterize generalization error for regular models cannot produce satisfactory results in this setting. Then the authors introduce the singular learning theory, which has been developed for decades. Then, a series of topics for deep learning, such as flatness and generalization, are studied within the framework singular learning theory, with a combination of theoretical analysis and numerical experiments. The paper is clearly written and well organized. \n\npro:\nThe authors point out that the study of deep learning should be put into the framework of singular learning theory. They verified this point from different aspects, where it is shown that results drawn from singular learning theory is better than those drawn from regular learning theory.\n\ncon:\nIt seems most results in the paper are illustration or clarification of existing results.", "extra_scores": {}, "extra_text": {}}, {"review_id": "CgWmojHjqXf", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the manifold of the weights in a neural network. The paper discusses the singular learning theory approach of Sumio Watanabe and argues for more exploration of this theory for understanding generalization performance of deep networks.\n\nMy opinion of this paper is lukewarm. There is a large amount of existing work on singular learning theory. I agree with the paper that the approach is a promising direction to understand generalization in deep learning. However, the intellectual contributions of this particular paper to the existing literature are difficult to ascertain and insufficient warrant publication.\n\nSome comments.\n1. I like the section on how flatness of the energy landscape connects with Real log canonical threshold (RLCT). The RLCT gives a much more refined treatment of different measures of local geometry of the energy landscape used in the literature (Hessian-based, local-entropy based, Bayes free energy etc.).\n2. The paper would benefit from Appendix A.1-4 to the main text. Some of these calculations have appeared elsewhere in the literature (see the work on K-FAC for the Fisher information) but the development in A.2-A.4 is interesting.\n\nSome questions that the authors could think about.\n1. The singular learning theory argues that the Bayes predictive distribution generalizes better MLE/MAP. How does this explain the fact that one sample from the posterior distribution also results in good generalization for deep networks?\n2. Singular Learning Theory is not new. While it is true that this approach has been overlooked in the recent work on deep learning, better experimental evidence is necessary in order to make a convincing case that it is a promising one. The results shown here are accurate computations using MCMC for small models. Can you also investigate whether one obtains non-vacuous generalization bounds for large deep networks using an estimate for the RLCT?\n3. Section 7: Is the true distribution of data really unrealizable? The reviewer is of the opinion that the fact that we can learn very good generative models for complex data indicates that it might be more viable to study the case when the true distribution is realizable for large-enough models.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to introduce ideas from singular theory to deep learning. All reviewers agree that the work is not yet ready for publication. The key issue seems to boil down to the fact that the paper does not propose nor verify any clearly motivated scientific hypothesis. Relatedly, the work includes many too broad or unscientific claims such as "To understand why classical measures of capacity fail to say anything meaningful about DNNs". Such statements should be given more precisely and with a proper citation.  

Based on this I have to recommend rejecting the paper. At the same time, I would like to thank the Authors for submitting the work for consideration to ICLR. I hope the feedback will be useful for improving the work.
Decision: Reject