Variational Inference via Resolution of Singularities
Authors: Susan Wei
Venue: ICLR 2022 Submitted (2022)
Keywords: singular learning theory
Bayesian neural networks
variational inference
normalizing flow

Abstract: Predicated on the premise that neural networks are best viewed as singular statistical models, we set out to propose a new variational approximation for Bayesian neural networks. The approximation relies on a central result from singular learning theory according to which the posterior distribution over the parameters of a singular model, following an algebraic-geometrical transformation known as a desingularization map, is asymptotically a mixture of standard forms. From here we proceed to demonstrate that a generalized gamma mean-field variational family, following desingularization, can recover the leading order term of the model evidence. Affine coupling layers are employed to learn the unknown desingularization map, effectively rendering the proposed methodology a normalizing flow with the generalized gamma as the source distribution.

Under review as a conference paper at ICLR 2022
VARIATIONAL INFERENCE VIA RESOLUTION OF SINGU- LARITIES
Anonymous authors Paper under double-bling review
ABSTRACT
Predicated on the premise that neural networks best viewed singular statisti- cal models, we set out to propose new variational approximation for Bayesian neural networks. The approximation relies on central result from singular learn- ing theory according to which the posterior distribution over the parameters of singular model, following an gesebraic-geometrical transformation known as a desingularization map, is asymptotically a mixture of standard forms. From here we proceed demonstrate that a generalized gamma mean-field variational family, following desingularization, can recover the leading order term of the model evi- dence. Affine coupling layers are employed to learn the unknown desingularization map, effectively rendering the proposed methodology normalizing flow with the generalized gamma as the source distribution.
INTRODUCTION
Singular statistical models are ubiquitous in modern machine learning. In contrast to their regular counterpart, singular models need not be identifiable nor possess positive-definite Fisher information matrix. These departures from classic regularity conditions have considerable implications for both theory and practice. Since neural networks are (strictly) singular models, accounting for such differences may be critical to some of the most pressing challenges deep learning theory including model selection and the generalization puzzle (Murfet et al. 2020).
In this work, singular learning theory (Watanabe 2009) is brought to bear on the challenge of performing inference in Bayesian neural networks (MacKay 1992: Neal, 1996). In particular, singular learning theory helps us understand the large-sample properties of the posterior distribution over neural network weights E IRd. By (Watanabe, 22018] Chapter 6), there exists a so-called desingularization map (also known as resolution map). 9 Rd Rdd, Rd g(( = w, such that when
is large, the posterior distribution in local weight set is proportional to exp(-ng? 2k1 22k, ... Sd 2 ... edd (b().
The kss and h's are natural numbers, and b(.) is a real-valued ( function.
We say the posterior distribution has been put into standard form if coordinates & have been found that allow the posterior to be locally written as in equation | Remarkably, it is possible under very general conditions write the posterior distribution over the parameters of singular model just such a mixture of standard forms. Based on equation [] we proceed to demonstrate that a certain mean-field variational family, following desingularization, can recover the leading order term of the log (normalized) evidence, up to constant terms that do not depend on sample size n or dimension d.
Let (x, 3) denote the input-target pair modeled jointly as p(æ, ylw) = ply|æ, w)p(2). Let us assume the parameter space W is a compact set in IR and Po y) = po(y(z))(z)) is the true Kata-generating mechanism. Throughout we suppose there exists Wp E W such that Po(y)æ:) = p(y 22, wa). In the parlance of singular learning theory, this condition is known as realizability. Let so(w) be compactly-supported prior. We shall refer to (p(:)) :), po(: ((()) as model-truth-prior triplet.
Define K K(w) to be the Kullback-Leebler divergence between the truth and the model, as a function of the model parameters wt
K K(w) (0)l(, Eop log Po(y)æ))
p(y w)

---

Under review as a conference paper at ICLR 2022
Following Watanabe (2009). we say model is regular if it is 1) identifiable ie., {w ((w) = 0} is a singleton, and 2) its Fisher information matrix [(w) is positive definite for arbitrary w E W. We call model strictly singular if it is not regular. The term singular will refer to either regular or strictly singular models.
Henceforth let p(y a w) be neural network model. We wish to approximate the intractable posterior distribution over neural network weights, p[w]D,) = 11=m p(::),."")ptt where D, = { M/) 1i
2(x)
a dataset of input-output pairs. The normalizing constant, Z(n) S p w) ( dw,,
is variously known as the (model) evidence and the marginal likelihood.
The asymptotic expansion of Zx (n),, variant of (n) defined in equation will prove crucial to justifying the proposed variational approximation. For strictly singular models, quantities such as (n) and (n) manifest as a singular integral, i.e., an integral of the form Jw e-"((w)pp()) ppww) dw where W C IRdd is compact semi-analytic subset, and f and behavior of singular integral critically depends on the zeros of
real analytic functions. The
VARIATIONAL INFERENCE FOR SINGULAR MODELS
Variational inference is an approximate inference technique which a family of densities Q, often called the variational family, is first posited and member of the variational family, some q" E Q, is then found via optimization according to some criterion that measures closeness the desired target density. To begin, let us write the posterior distribution p(w) w (D..) in two altemnae forms:
(n(w)
1 n
Σ
log (1
K (n(w)
log
p(yel w)
p(w]Da x e enbnnww ((w)
pDw)Dn) x nnka (w))
/ o-nLa(wrepuu dw ((n) Jw
Z(n) = / (nkk((u)pww))
On the left, the posterior is written in terms of the average negative log likelihood I., (w) and on the right terms of the average log likelihood ratio K2 (w),, which the empirical counterpart to K(w). Since (nx /nn) II,"" (o))) we refer to (n) as the normalized evidence.
Now, consider some general change of variables g: Rd -> TRdd, (g({) the KI. divergence between variational distribution g(E) and the desired posterior target,
In the new coordinate &,
p(((Dn) x (nnK((((t)) (((
is given by
(L(((E)IP((IPP))) +KA KLlG(Ellll( log ((n).
As long as the support of g is contained in the support of the posterior, we have (L(((E) 2 O and hence the following bound:
(g g) := (o(E), (L(((()))+((((()) & log ((n).
(2)
Equality in equation 2 is achieved if and only if 4(E) p(g)D.). It is easy to recognize that maximizing (g. g) is equivalent to maximizing the so-called evidence lower bound.
ELBO(g. FE Σ log p(us/ ((5)) - ( (og(((())))())))
(3)
since (g. = ELBO 9) nS, where S, - Σ log (v( is the empirical entropy. Indeed, just as (g. lower bound on the log (normalized) evidence log Z (n), so too is ELBO(q, g) a lower bound on the log (unnormalized) evidence log (m)).
To facilitate theoretical analysis, we will work with the deterministic counterparts to Vg g) and Z (n). respectively given by
V x (g. g) -FgnK(((EE) (KI,G()))))())))))))))))))))
2

---

Under review as a conference paper at ICLR 2022
and
2x(n) == J (=(k( dw.
(5)
It is plain to see that just (g.g) S log ((x), we have V (() < log Zx(n)) Our theoretical
results address the possibility that suP,eR V xx g) xo log (x( ) for some variational family Q and change-of-variablls, g.
By (Watanabe 2009 Theorem 6.7), for the leading order term in the asymptotic expansion of (n). we have
[x(n) = Cn CIog n) m)
(6)
The rational number À E [0, d/2] is an important quantity in singular learning theory known as the real log canonical threshold (RLCT) and the integer m 2 1 is its associated multiplicityy These two
quantities, to be defined in equation I1 are uniquely associated to a model-truth-pre triplet.
Equipped with equation 6 (Bhattacharya et al., 2020 Theorem 3.1) established that when g is
resolution map, the mean-field variational family 2(0.1) of equation 12 can achieve
sup , (9.9) 2 -olog n - constant independent of n.
(7)
(ER(o.11
(In Appendix C we prove this result under more general conditions than assumed in Theorem 3.1 of Bhattacharya et al. (2020).) Unfortunately, the bound in equation 7 provides little reassurance that ((0.1) is desirable variational family.
Contribution Following Lin (01 let us call in equation 6 the leading coefficient of (n).. We go beyond the analysis in Bhattacharya et al. (2020) by taking into account those terms in the leading coefficient that depend on the dimension d, call C(d). Our main result, Theorem 5. shows
that if g is a resolution map, then the same variational family [(0.1] of equation 12 can achieve
sup qEQio.1]
V (4 g) - logn log C(d).
Next, rather than presuming the resolution map theoretically tractable as in BBaattachara et al. (2020) we employ a normalizing flow to learn the unknown resolution map g at the same time learning the variational parameters in g. We are aided by the fact that a resolution map can attain the optimal value of V K (9. g) and therefore justifies learning the resolution map via optimization of (g g). Finally let us note that although the result in equation is stronger than that in equation 7 it does come at the cost additional assumptions as we will discuss at the end of Section [
Remark. When the model regular, we need not bother with singular learning theory and may obtain (n(n) & v(wo))
(2x)² H(w)) -d/2 via the Laplace approximation. The Laplace approximation, however, is egregiously inappropriate for singular models, in particular neural network models. in regular models, equation 6 is true generalization of the Laplace
Since d/2 and
approximation, holding for both regular and strictly singular models.
3 SINGULAR LEARNING THEORY
That the posterior distribution in singular models can be written, under quite general conditions, as mixture of standard forms predicated on the monomialization of K K(w). The following theorem from Watanabe (2009). adapted for notational consistency, gives precise conditions for the existence of the resolution map, an algebraic geometrical transformation such that K (w) can be written as a monomial. The result is itself based on Hironaka's resolution of singularities, celebrated result in modern algebraic geometry. To prepare, let W. = {wr E W K(w) S. for some small positive constant and W: R) be some real open set such that W. C (R) The theorem below will make use of the multi-index notation: for a given w = (W1,... Wa) E IRdd, define == the multi-index k = (kı,.... ka) with each kj a nonnegative integer.
: w ... where
Theorem 3.1 (Theorem 6.5 Watanabe (2009)). ). Suppose the model-truth-rriat triplet (p, Pos s) satisfies Fundamental Conditions I and Il with = in Watanabe (2009). We can find a real analytic manifold (R) and proper and real analytic map g M(R) W!R) such that
3

---

Under review as a conference paper at ICLR 2022
1. M g is covered by a finite set M UaMa where Ma [0. bl%
2. In each Ma.
(({(E)) = = 57 Saka....e 2ka
(9)
where Kg E N such that not all k3 are zero.
3. There exists C function b(E) such that
*(g(E)) (E) = bb) = ...
(10)
where h2 E N, (E)(
[ is the absolute value of the determinant of the Jacobian and b({)
>
<> Of & [0.
If we "plug in" equation 9 and equation 10 into the transformed posterior p({[Dr), we obtain the first
display of the paper, equation ! Theorem 6.5 of Watanabe (2099) holds for regular statistical models
well, e.g., by the transform = g{E) who + (ww))_22 we can put regular model-truth-prion triplet into the standard form. It is worth noting that neither the resolution map nor the multi-indices and h are unique for given triplet (p. po. po)
A crucial quantity that appears in singular learning theory is the real log canonical threshold (RLCT). Let (M,. ) be as in Theorem 3.1 and > 4, .j 1, 1,....d where h and k5 are the entries of the multi-indices h and & in local coordinate Ma., (For brevity, the dependence on a has been suppressed.) When ky 0. > is taken to be infinity. Uniquely associated to triplet (p, Pu, (p) are its real log canonical threshold (RLCT) and its multiplicity defined, respectively, as
2h,
min min >j. jEL.....
max #(i 23 = A}-
(11)
Let {a**) be the set of those local coordinates in which both the min and max in equation 11 are attained. Watanabe (2000) calls such set the essential coordinates.
Assuming the prior is proper, the RLCT of model-truthhprion triplet is at most d/2 / (Watanabe 2009
Theorem 7.2). When the model regular, the RLCT is exactly equal to d/2 and the multiplictyy m == (Watanabe) 2009 Remark 1.15). Murfet al. 2020) argues that (twice) the RLCT might be most natural count of parameters in singular models. In Appendix A we recall a simple toy example where the resolution map, the RLCT, and the multiplicity can be calculated explicitly.
Remark. In Section 6 will learn the resolution map using affine coupling layers. However as indicated by Theorem 3.1 the resolution map g(E) as well as the multi-indices and h in fact depend on a. Despite this, it is unclear if learning multiple resolution maps 9a would have any practical advantages since the RLCT is determined entirely by the essential local coordinates.
4 RELATED WORK
Bayesian learning in neural networks well precede the advent of modern deep learning MacKay (1992); Neal 1996). The resurgence of interest in Bayesian learning for deep neural networks, sometimes called Bayesian deep learning Wilson & Izmailov (000)), has been prompted by con- cerns of overconfidence and miscalibration. Since exact inference for Bayesian neural networks is intractable, all methods proceed by approximate inference. A major class of approximate inference techniques is based on scaling classic MCMC modern settings of large datasets and deep neural networks. Some scalable variants of MCMC suitable for deep neural networks include Welling & Teh (201T)/Chhen et al. (2014); Zhang et al. (2019). Another major approximate inference technique for Bayesian neural network is represented by variational inference which learns the target posterior via optimization. The various flavors of variational inference can be commonly characterized by two ingredients: 1) an approximating family, e.g., class of distributions over the neural network weights and 2) a criterion for measuring closeness of two distributions. The most commonly employed approximating family is undoubtedly the mean-field family of fully factorized Gaussian distributions. Many variational inference techniques share this approximating family even if they use different criterion measure closeness the target (Graves) [011] | Blundell et al. 2015] Miernandez-Lobatee (et al.) 2016 Li & Turner, 2016)) Khan et al. [0188] [un et al. 2019). The limitations of the mean-
Gaussian approximating family well known however MMacKy 1992.. The desire move beyond
4

---

Under review as a conference paper at ICLR 2022
mean-field Gaussian has motivated many recent methods to make use of more realistic covariance structures (hang et al. 2018 or more expressive approximating families, e.g., via normalizing flows (Louizos & Welling 2017). Monte Carlo dropout (Gal & Ghahramami| 2016) is another popular approximate inference technique for Bayesian neural networks which, despite first appearances, can in fact be viewed as variational inference. Another strain of work is based the idea of using stochastic gradient descent as a sampler for the underlying posterior distribution of interest. Works in this spirit include Mandt et al. (2018), Izmaiiov al. (2018) and Maddox al. (2019). Finally there are various approximate inference techniques for Bayesian neural networks that are not easily classifiable according to the distinctions above, e.g., temperature scaling (Guo et al., 2017) and deep ensembles (Lakshminarayanan et al. 2017).
THE GENERALIZED GAMMA MEAN-FIELD APPROXIMATION
Consider the mean-field variational family proposed in Bhattacbarya et al. (2020)
2(011) (1x.k. II k 3, (E)) : == (A1, (A1....., da) R ks (Ri,,... ka) 1R B= (3,.= (3,,...,Bal Bad)
(0, 20)
(5(5)) = A,,As.n (Es) x 22 - exp(- ) (0.11
(12)
where each univariate density 1j supported on (0, 1] is the density a truncated generalized gamma random variable. Let E .l....,d d} be the dimension that attains 7n- = Á where À is the RLCT of the underlying (unknown) model-
triplet. It was established in Bhattacharya et al. (2020) that if is a resolution map, then [ satisfies equation 7 in particular by setting À and k to their respective true values, and 3,- = n and all other 3y = 1.
In Theorem 5.1 below we will instead establish equation g In the proof show this can be accomplished by setting and to their true values as in Bhattacharya et al. (2020) though the optimal values of will be different. Finally, although assume below that 1, this is not necessary; as long as & d, we can set 3, n//mm for all j such that Xy X.
Theorem 5.1. Suppose the model-truth-priorr triplet such that Theorem 3.1 holld with [o. b] [o. 1). (({(E)) = = and ((o(E))lo'(E) = Let À denote the RLCT of the triplet and assume the multiplicity 1. Then sup V/r k g) -Àlog + log ((d) where O(d) is the term in the leading coefficient C that depends d.
Proof. If & [0, 1j and ( x ę² then necessarily b(E) IT 1/(h; + 1). Let %n = (h, + (2k()) Applying Corollary 5.9 in Lin 2011)) we obtain
(x(n) = / exp(-ne 2k de = On (logn)) -n- = C'
(13)
0.1j
where
T())
C (m- 1)! IT (2k,) II; - (2k 2)
(14)
If we denote by C(d) the terms in the leading coefficient that depend d, then have
d log C(d) = Σ log(2k)) - Σ log(2k,) j==+t1
Σ j=m+1
log Xy -
Σ
log(1 A//z).
1=1
j=m+1
Returning to our variational distribution, if the k; 's are well specified and we additionally set By- where j is the dimension that attains Xy- 5. we then have F. ((g(E)) Ay-- II G(),, B3),
5

---

Under review as a conference paper at ICLR 2022
where G(A, (3) is in B.1 Next, we have
KL(g(E)ll(((())))())) kLLa <>6(3))
Σ j=1
KL KLL
93
E
ci [(, + 11)
Egg log qj
h, Fg loge; log(h; + 1).
addition the Xy's are all well-specifted, then
d
KL(Gloo(((E)l)((() = Σ (B) - log B(k,, hy. B3) log(22;) + log 23}
where B(k, h, (3) is as Now let us make use of the fact that log B(k, (3) = -A log f and G(A, B) x 2/3 for large B. Since 3j- is large, we get
(x(9.9) - II G(A,, 3,) +
Σ (, log B(k,, hy. 9,)) log((k;) log 1og
j=1
-Rlog n i(1
II G()),
Σ
+ log B(, 13:
33)1
Σ
log(2kg)
log ()y))
j=m+1
-=m+1
33 are sufficiently large for j + j", we get
X Bj
(x x -Alog n log 1og(2k;)
Σ
log(},)+
+
Σ j=m+1
1-(1- log + 1(
II
j=1
j=m+1
(15)
Then there exist Bj.j # jr so that equation 15 matches - -Alog n + log C(d).
Theorem 5.1 critically assumes that b(E) x 1. We do not expect this to hold in reality. For example, even for the simple one- midden layer tanh network considered in Section [ it does not appear that Watanabe 2000). Currently, we are prevented from stating a more general version of Theorem 5.1 because there is no off-the-shelf derivation of the leading coefficient when the singular dt. We expect the generalization to be technically feasible but as its development requires advanced knowledge of algebraic geometry, it is beyond the scope of this paper and best left as separate investigation.
B(E) x
integral of the general form fo. exp(-ng2k)
In the next section, we proceed to learn the resolution map rather than presume it is known. This is an improvement over Bhattacharya al. (2020) which, due the difficulty of deriving resolution maps, was limited in its single experiment to the toy neural network (x) = b tanh (az) with weight = (a, b) e RR2 Again, though Theorem 5.1 is stated in terms of the truncated generalized gamma mean-field family, we do not believe the assumption b(g) x 1 is critical and proceed henceforth to work with the untruncated generalized gamma mean-field family, which we denote Q.
LEARNING TO DESINGULARIZE
When the resolution map g is known, the preceding results suggest to 1) apply the chaige-of-variables g(E) and 2) maximize ELBO(g, g) over the untruncated generalized gamma meaa-fieed family Q via c.g., stochastic gradient descent. However, theoretically, the resolution map g is notoriously difficult to derive. Computationally. the recursive blow-up procedure in algebraic geometry is entirely not scalable to high dimensions.
variables
We propose to learn the resolution map via a normalizing flow, which is commonly used to model complex distributions as the forward of simple source distribution through an invertible neural network G. An interesting direction of future work would be exploit properties of the resolution

---

Under review as a conference paper at ICLR 2022
map that may aid in the design of the normalizing flow architecture. For now, we simply make use of a common type of invertible architecture consisting of affine coupling layers. With r denoting a binary mask, so-called affine coupling layer acts as follow:
u, v E IR + (1 (1-17
r) r) o a + r @ (u @ exp(s(r G u)) + ((r o u)).
The binary mask r must alternate from one affine coupling layer to the next for otherwise there would be little expressive power in the resulting network. Let Gg be network consisting of alternating affine coupling layers, where denotes the collective parameters. Since the resolution map is to be learned, there is no loss in generality to assuming j 1. We shall need the gradient with respect to the variational parameters and the normalizing flow weights as part of employing stochastic gradient descent, i.e. V. " (ELBO(qx.4 A.B, Gg). Note that here abuse the notation slightly need to update 31 which should be set to the sample size n according to the proof of Theorem 5.1
do not
Although the source distribution in normalizing flow can have its own trainable parameters, it is common practice to adopt a parameter-less source distribution. In particular the source distribution is itself reparametrizable, then the learning of the associated parameters can be absorbed into the invertible transformation. The generalized gamma distribution is not easily reparametrizable for general values of X, k B. However for certain settings of X, we can in fact avail ourselves to the reparametrization trick, at least approximately, Let V be a gamma random variable with shape A and rate By. then v1/(2k;) interested in settings where x, is large, which case Vj is approximately Gaussian with mean \ and variance X//33. Letting T(c) == (FF'(G) (q) (((2k)))))),
(,,)) := Ej 2k3 exp(- (3,5- (o)))))) We will mostly be
FJ (ca) 1/(2aa),, where Fj (c) m
(A3 v BBg the reparametrization trick then leads to the objective function
Ea-((.n)
Σ log logplu (T)))) + (lllg((GG(((o))) + log (G'((())) -E, log q. (16)
Note that equation 16 is not the same as BLBO( G3) because we have made use of the Gaussian approximation for V, in the case of large Xy.
The entropy component of equation 16
E E log k., can be derived analytically, see Ap- pendix B.2 Next, the specific architecture of Go has rendered the log Jacobian term, log G²2() (-),, computationally tractable. The final piece to replace (o..) with an empirical average over M samples. In the experiments that follow we will consider either learning the variational parameters A, k, or fixing them at some initial value since other values can be learned through transformation that gets absorbed into G.
7 EXPERIMENTS
In this section, compare the effect of two source distributions for normalizing flow given by the affine coupling network Gg consisting of 2 pairs of alternating couplings with scaling and translation networks each consisting of 2 hidden layers with 16 hidden units, see Appendix for exact details of the architecture. We denote by n of_gaama Ag__ho__ flag the variational family that results from pushing forward the untruncated generalized gamma distribution with variational parameters initialized at Ao = (1, ..,....,))) kn = (ko..... ..)....,,,)) and Bo (n1, So...., ....,,,, Bo), and boolean flag indicating whether the variational parameters are subject updating. For the other approximation
resulting from pushing forward a Gaussian source distribution N N(uu, vo). we write analogously ff__aussiian 2o where po and Up are the fixed mean and variance. For each combination of An and 3o considered, we set 140 >a/ /o and vp = >a/
We employ the widely adopted variant of (minibatch) stochastic gradient descent known as Adam. The number of epochs and batch size were set to 2000 and n/10, respectively. Different constant learning rates were employed according to the parameter type: 1e-3 for affine coupling layer weights and k and le-1 for X and B.. Expectations in the objective function that are not analytically tractable are replaced with an average over M = 5 samples. At the end of training, we evaluate (g g) in nf nf_gamma without recourse the (approximate) reparametrization and by using the analytic expression for the entropy component of and 100 samples from q to approximate other components in Vi under expectation.
7

---

Under review as a conference paper at ICLR 2022
To date, there is very small collection of model-truthe ter--tttttttoottt prior triplets where the RLCT À and multiplicity mare known. We shall limit our experiments to two such triplets. This will allow us to compare the achieved (g G) following training to - -log n + (m - 1) log log n. Note that even in such triplets where the RLCT and multiplicity are known, the exact value of the leading coefficient C' is still usually unknown. Now, to compare nf_gamma and nf nf_gaussian to each other, we can simply see which achieves higher L (g. G) after training.
Looking ahead to downstream tasks, is natural ask whether the variational posterior predictive distribution, Pes(y]z, zr D,) = <p(y jr, Gg Gg(E))) inherits the desirable properties of the Bayes posterior predictive distribution, p(y|=; Dn) = (p(ulær, w)) The answer turns out to depend on the relationship between the variational real log canonical threshold Aus and the RLCT of the poodel-truthprit triplet. It may very well be that a variational family which is closer to the true posterior (in the KL sense) than another variational family may induce a worse approximation of the true Bayes posterior predictive distribution. We provide an in-depth account of this phenomenon through the lens of singular learning theory in Appendix E
Our first example concerns the tanh model-truth-prior triplet. Consider input x E R follow- ing the uniform distribution on [-1,1], and response variable y E I modeled as p(y|æ, w) V2= exp( (v f(x, )) where [=(()) = Exm bn tanh(a)æ) is a tanh network with H hid- den units and w is the collection of neural network weights {(an, b)) If the true distribution 1 (xpp-_2²) and the prior yp C function of w with
is given by po(u(x>) = ply|2, 0)
.2z
compact support, satisfying (0)) > 0. Aoyagi & Watanabe (0066) showed that À = H4-2++ and 4++2
m = 2 if 1² = H. and m 1 if 1 < H where is the maximum integer satisfying 1² S H. In contrast, were this a regular statistical model, we would have X = H. We consider two settings for (v] = p(y| we) in the experiment: 1) wo = 0 and 2) wo 5. Note for the latter, the corresponding RLCT and multiplicity are unknown.
Our next example is the reduced rank model-truth-prior triplet. Consider input x E M and response variable y Rº modeled as p(y [z, w) (2T) N/22 exp( SallyBB.zz [2] where {w = (A, B) e R- B E RRN.HH. ). This model is readily seen to be special case of neural network with hidden units H and identity activation function. We shall set M = H 3 and N = H. In the realizable case, i.e., po(y)22) = p(y|z, Ag, Bo), if the prior şo is a C function with compact support satisfying <p(A,. Bo) > 0, the RLCT was derived in in/Aoyagi & Watanabe (2005) for various values of N, M. H and r = rank (. Below we set Bo INXN and Ag = [ Hz JHxa). The rank r for B equals H. This then falls under Case (3) in Aoyyaii & Watanabe (2005) since N + H < M + leading to X = (NH - Hr + Mr)/2, m = 1. Note that were this a regular model, we would instead have x (M H + 1/2
Table provides a summary of the values of considered in each of the triplets and the corresponding RLCT and dimension. In the experiments, we tested various combinations of values of H, settings for We and priors. Throughout, the priors were designed to be mis-specified. The full results of these experiments can be found in Tables 2 Note that the comparison between different variational approximations have been made subject to the value of H, the prior, and the relationship A //0 1o0- We should also point out that not all runs reached convergence as can be seen from the count column in Tables 2 This seems largely to be a result of the constant learning rate we used for all training.
The first setting (Table 2 is the realizable tanh network with wo 0. In this case, the true RLCT is known as discussed above. As Table 2 demonstrates, there is no clearly discernible difference between nf_ ff_gamma and rf_gauss i an where the source distribution for both starts near zero. However when the source distribution has An/Be = og = 5. i.c. very far from Wo nf _f_aamaaa is more robust than _ff_gaussian. The second setting (Table 3 is analogous to the previous except the true generating mechanism is given by Wo 5. In this setting, the untruncated generalized gamma source distribution is seen to provide a better variational approximation than the Gaussian source distribution. This appears to hold quite independent of the initial values of X, k. B as well as the gradient flag. Interestingly, in contrast to the previous setting, there is a clear benefit to learning X, k. B.. The third setting (Table 4 is the realizable reduced rank regression. In this experiment, we do not see a discernible difference between the source distributions until we look at Ao/ /B = #o = 5, which is far from Wo (An, Bo). As in the first setting, nf gamma is more robust than n ff of__aussian
8

---

Under review as a conference paper at ICLR 2022
Finally, though we have cautioned against judging the quality of variational approximation according to the approximate posterior predictive distribution induced, it is nonetheless informative to visualize downstream uncertainty quantification. In Figure L we choose one line from the tanh Wo 0 experiment (Table 2 to further examine. For the two methods illustrated in in Figure [ their performance in terms of Vi is quite close indicated by Table 2 respective confidence bands are quite similar. Next, in Figure 2 we choose one line from the tanh wo 5 experiment (Table ! than of _e___ausssian inn. We see corresponding relationship between their respective posterior predictive distributions. Namely, the confidence band resulting from nf of_gauussia are far too conservative while that from nf namgamma is less so.
It is then comforting that the
further examine. This time, nf_ gamma achieves significantly better
appructrati Bosto prolatist pioltiiee
aprroii dle Sayza posterko ausiiie
(a) grss_gaussiannnm la___aannnc--__1
(b) garma 1atpa__1111111___111111
Figure 1: The model of interest is the tanh network with hidden units H 576. The data is generated
according to poly|æ, x w) plylæ, wo) where wi = O. The prior is taken to be (5.00, 100.00 (00.0/4). The predictive distributions resulting from different variational approximations trained on a dataset of size n 5000 are displayed.
approcmate Boyes posterior pred ctive
approdmaaee Bayas posterior sreditive
1200
330
362
band
(a) n nf_aaussiii an_5_5e-2
(b) n nf_gamaa 2aFF__gaaaa0___________
Figure 2: The model of interest the tanh network with hidden units H 576. The data is generated
according to Po(y)zz, zz w) = p(y]zz jz, wo) where wi = 5. The prior is taken to be N (0.0, 100.04a). The predictive distributions resulting from different variational approximations trained on a dataset of size n 5000 are displayed.
8 CONCLUSION
In this work we propose variational approximation for Bayesian neural networks by leveraging insights from singular learning theory. Namely, for large n, the posterior distribution over neural network weights is not Gaussian but rather can be put into a mixture of standard forms. From this, we demonstrate that the generalized gamma mean-field family, following desingularization. can in theory achieve the leading order term of the log normalized evidence. Because we choose to learn the desingularization map using affine coupling layers, the proposed work can be cast as a normalizing flow with an unconventional source distribution. Interestingly, for large values of the variational parameter X, the source distribution in each dimension is approximately N 1/26), which is reparametrizable in terms of the conventional source distribution N NO, 1). Though learning the variational parameters X, k., B in the souree distribution goes against conventional wisdom in normalizing flows, our experiments suggest some performance may be gained by learning the optimal untruncated generalized gamma source distribution at the same time learning the resolution map.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
Bayesian learning for neural networks is often touted as a panacea to the challenges of uncertainty quantification in deep learning. In this work, we propose variational approximation that performs well in terms of the ELBO achieved. However, we make no claim that our variational approximation superior in downstream uncertainty quantification. We explain extensively in the appendix that variational approximation which achieves higher ELBO may induce a worse approximate posterior predictive distribution than a variational approximation which achieves lower ELBO. According to singular learning theory, the performance of variational approximation in terms of the posterior predictive distribution it induces critically depends on the relationship between the variational RLCT and the underlying RLCT of the model-truth-prior triplet.
REPRODUCIBILITY STATEMENT
A zip file of the source code has been submitted as supplementary materials. For theoretical results, explanations of assumptions and proof of the claims are included the main text. No datasets are used; the experiments are solely based on simulated data.
REFERENCES
Miki Aoyagi and Sumio Watanabe. Stochastic complexities of reduced rank regression in Bayesian estimation. Neural Networks, 18(7):924-933, September 2005.
Miki Aoyagi and Sumio Watanabe. Resolution of Singularities and the Generalization Error with Bayesian Estimation for Layered Neural Network. IEICE Trans, pp. 21 11242124, 2006.
Anirban Bhattacharya, Debdeep Pati, and Sean Plummer. Evidence bounds in singular models: Probabilistic and variational perspectives. arXiv:2008. 04537 [math, stat], August 2020.
Charles Blundell, Julien Comebise, Koray Kavukcuoglu, and Daan Wierstra. Weight Uncertainty in Neural Network. International Conference on Machine Learning, pp. 1613-1622 PMLR, June 2015.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In Eric P. Xing and Tony Jebara (eds.), Proceedings of the 31st International Conference on Machine Learning, June 2014.
Andrew K. Foong, David R. Burt, Yingzhen Li, and Richard E. Turner. On the expressiveness of ap- proximate inference in bayesian neural networks. In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), ), Advances in Neural Information Pro- cessing Systems 33: Annual Conference Neural Information Processing Systems 2020, NeurlPS 2020, December 6- 12, 2020, virtual, 2020. URL
meurriiininninnnnnnno..o. wnwerrrps.... cc/ .html
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning. In Proceedings The 33rd International Conference on Machine Learning), 2016.
Alex Graves. Practical variational inference for neural networks. In J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Percira, and K. Q. Weinberger Advances in Neural Information Processing Systems, volume 24, pp.
Advances Curran Associates, Inc., 2011.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML'17, pp. 1321-1330. JoLR.org, 2017.
Naoki Hayashi. Variational Approximation Error in Bayesian Non-negative Matrix Factorization. vrXiv:1809.02966 [cs, math, stat]. February 2020.
Jonathan Heek. Well-Calibrated Bayesian Neural Networks. PhD thesis, University of Cambridge, 2018.
10

---

Under review as a conference paper at ICLR 2022
Jose Hernandez--obate Yingzhen Li, Mark Rowland, Thang Bui, Daniel Hernandez-Lobate and Richard Turner Black-box alpha divergence minimization. Maria Florina Balcan and Kilian Q. Weinberger (eds.), Proceedings of the 33rd International Conference Machine Learning, vol- ume 48 of Proceedings of Machine Learning Research, pp. 1511-1520, New York, New York, USA, June 2016. PMLR.
T. Hosino, K. Watanabe, and S. Watanabe. Stochastic complexity of variational Bayesian hidden Markov models. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2. pp. 1114-1119 vol. 2. 2005.
Pavel Izmailov, Dmitrii Podoprikhin, T. Garipov, Dmitry P. Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. Uncertainty in Artificial Intelligence (UAI), 2018.
Mohammad Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava. Fast and scalable bayesian deep learning by weight-perturbation in adam. In International Conference on Machine Learning, pp. 261 1-2620. PMLR, 2018.
Masahiro Kohjima and Sumio Watanabe. Phase Transition Structure of Variational Bayesian Non- negative Matrix Factorization. In Alessandra Lintas, Stefano Rovetta, Paul F.M.J. Verschure, and Alessandro E.P. Villa (eds.), Artificial Neural Networks and Machine Learning - ICANN 2017, Lecture Notes in Computer Science, pp. 146-154, Cham, 2017. Springer International Publishing.
Ranganath Krishnan and Omesh Tickoo. Improving model calibration with accuracy versus uncer- tainty optimization. arXiv:2012.07923 [cs], December 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles. In I. Guyon, U. V. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, 2017.
Yingzhen and Richard F. Turner. Rényi divergence variational inference. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, pp- 1081-1089, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819.
Shaowei Lin. Algebraic Methods for Evaluating Integrals in Bayesian Statistics. PhD thesis, University of California Berkeley, 2011.
Christos Louizos and Max Welling. Multiplicative Normalizing Flows for Variational Bayesian
Neural Networks. In International Conference on Machine Learning, 2017.
David J. C. MacKay. A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3):488 472, May 1992.
Wesley Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dAlché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, 2019.
Stephan Mandt, Matthew D. Hoffman, and David M. Blei. Stochastic Gradient Descent as Approxi- mate Bayesian Inference. arXiv: 1704: 04289 [es, stat], January 2018.
Daniel Murfet, Susan Wei, Mingming Gong, Hui Li, Jesse Gell- Gell-Reeman, and Thomas Quella. Deep Learning is Singular, and That's Good. arXiv:2010. 1550 [cx], October 2020.
Shinichi Nakajima and Sumio Watanabe. Variational Bayes Solution of Linear Neural Networks and Its Generalization Performance. Neural Computation, 194)::112255) 2007.
Radford M. Neal. Bayesian Learning for Neural Networks, volume 118 of Lecture Notes in Statistics. Springer New York, New York. NY, 1996.
Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable laplace approximation for neural networks. In 6th International Conference on Learning Representations, ICLR 2018 2018-Conference Track Proceedings, 2018
11

---

Under review as a conference paper at ICLR 2022
Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger Grosse. Functional Variational Bayesian Neural Networks. In International Conference on Learning Representations, 2019.
Kazuho Watanabe and Sumio Watanabe. Stochastic Complexities of Gaussian Mixtures Variational Bayesian Approximation. The Journal of Machine Learning Research, 7:625-644, December 2006.
Sumio Watanabe. Algebraic analysis for non-regular learning machines. In S. Solla, T. Leen, and K. Müller (eds.), Advances in Neural Information Processing Systems, volume 12. MIT Press, 2000. URL
neurips. cc/paaper/1999//leee pdf
Sumio Watanabe. Algebraic Geometry and Statistical Learning Theory. Cambridge University Press, USA, 2009.
Sumio Watanabe. Mathematical Theory of Bayesian Statistics. Chapman and Hall/CRC, 1st edition, 2018.
M. Welling and Y. W. Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference on Machine Learning, 2011.
Andrew Gordon Wilson and Pavel Izmailov. Bayesian Deep Learning and a Probabilistic Perspective of Generalization. arXiv:2002.0879) [cs, star), April 2020.
Guodong Zhang, Shengyang Sun, David Duvenaud, and Roger Grosse. Noisy Natural Gradient as Variational Inference. In International Conference on Machine Learning, pp. 5852-5861. PMLR, July 2018.
Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson. Cyclical Stochastic Gradient MCMC for Bayesian Deep Learning. In 1902. [Cs, Stat], 2019.
A TOY EXAMPLE OF RLCT CALCULATION
We recall Example 27 from Watanabe (2018) to illustrate the concepts of resolution map. RLCT and multiplicity for a simple model-truth truth-prior triplet. For univariate input æ E [0, 1] and univariate output E R. consider the model with parameter w (a, b) E [0, 1]2 given by
plz, ya w)
exp(
(y tann(bxx))
(17)
22n
Suppose the prior is uniform, i.c., ((w)= = and the truth given by po(x, 8) = p(x, y/0, 0). Then we can easily see that
K(w) b²a² Ko(w).
where
Ko(w) [fa(="((a(n
((tanb((bx) dx
2
The following desingularization map puts the triplet in standard form:
(o(w)
2
Furthermore have ((((E)) = hh where h (0, 0) and b(g) [g)(E))- Since (k1, k2) (1, and (h), h2)) = (0, 0) we have (Λ₁, A2) = (1/2, 1 1/2). Therefore for this particular model-truth-prior triplet, the RLCC is 1/2 with multiplicity 2.
12

---

Under review as a conference paper at ICLR 2022
B BASIC CALCULATIONS FOR THE GENERALIZED GAMMA DISTRIBUTION
B.1 TRUNCATED
We summarize basic calculations for the [0, 1] truncated generalized gamma distribution from Bhattacharya et al. (2020). Let gj be the univariate truncated generalized gamma density given in
cquation 12
The normalizing constant of qj given by B(),, ky, 85) where B(A, k. 3) T(A
2k
and x(a,z) = [(aj So t--le- dt the (regularized) lower incomplete gammaa function.
The quantity F = G()),88) where 8)
7(1,3)
B.2 UNTRUNCATED
Consider the univariate density
((E) x 5" exp(-3E2k)
supported on (0, 20) where h, k, > O. (h + 1)k(2k). Elementary calculations give
Egg2h =B- +()+))
[(2)
logo = 2h (A) log 3) where v is the digamma function.
A (vb(A) - log B) log Z where, Z = B (T(A)
2E Eg logq- =
is the normalizing constant
2k
q.
C GENERALIZATION OF (BHATTACHARYA ET AL., 2020, THEOREM 3.1)
Lemma C.1 helow states that any variational distribution kx. in the untruncated generalized gamma capable of achieving (x(g.g) > log + A where A is some constant that does not depend on n. Lemma C.I is straightforward extension of Theorem 3.1 of Bhattacharya et al. (2020) the [0, 1] truncated generalized gamma mean field variational family to the untruncated case. We also relax the condition Theorem 3.1 of Bhattacharya et al. (2020) that the variational parameters À and (and hence h) are well well-specified; here we only require that & be well-specified.
mean field Family Q with well-specified & where we additionally set 8,-
Lemma C.1. Suppose the model-truth -rrior---iorrrae triplet is such that Theorem 3.1 holds with K(g(E) ?2k and ((g(())) (E) th. Consider the untruncated generalized gamma mean-field family Q. we have sup Wx (x(g.) 2 -Alog n + A, where A is a constant that does not depend on n.
Proof. ky = ky for all j ......d.. we have
+ +2)
1- ky-
R
+ Xz-)
4 r
+
+
A 3333
EgnK(g(E)) = II B j=1
= =n
3,
II
I(A,)
r()g.)
r()))
3
13

---

Under review as a conference paper at ICLR 2022
Next, we have
(Ugg())lo(((()))(((
Bu [log dj h, log En] - Eg log b(E)
hi hi (1((2)) log B,) >, + 2, log Bg + log(2k)) log (A,))
Egg log b(E)
2k3
hi hi (>1))) - -h2 hi +1
log B3 + log(2k;) log [(2))
Eg log b(E)
2ky
2k,
h hi (}())) - X3 + RS log 3, + log(2k;) - log log r(2;)
= Σ
Egg log b(5)
2kj
h2 h (Aa)) - X3 + log(2kk) log r(2))
= Alogn n +
Σ i43"
2g
log 3r + Σ
Ey log b(E).
2
The last line follows from setting 3,- n where j [l...., d3 is such that X,- x. breaking ties arbitrarily. Finally, the desired incquality follows from the fact that Eg log b(E) is bounded below by some constant.
D EXPERIMENTS
Table 1: Summary of models considered.
H
RLCT
d
И
RLCT
d
576
12.0
1152
24
324.0
1224
1024
16.0
2048
32
560.0
2144
(a) tanh network
(b) Reduced rank
In the experiments, we train affine coupling network with two pairs of alternating couplings. The translation t is a feedforward (leaky) ReLU neural network with tanh output activation function. The scaling t another feedforward (leaky) ReLU neural network with identity output activation function. The models implemented are given below.
Translation netwodk
Sequential
201: Linear(in (in. Featuresedin, out features-16, bias-True)
[31: LeckyBelD (ongglllD(oggeieee..ee ive_sageee (001
teaturese16, bias-True) 131: (negatt lop.salppee0..11
binear(in (in. tetttreees-6,
161: (an_aarinnrtttteeee
out sat_rreccuees16, sias-True) JeekyBelUUnegett slope=C.011
Linear (ir_teeannaaaeeeeeeee Featuregell featuresedin, bias=True)
Scaling nework
Sequencial
(01: (inaarre out_foatures:16, bias-True)
Jasky@ciDinogetrrrs itspptepsss_____e___eee_p
[21: Linear (in_foatueea=16, fe__feacue,,,,, blas=Izua)
131: (exyRellU/opttiva_eeee 1101)
(11 (tcearrtn out ret_reeuree--1, stas-True)
LaakyBal0/jeqatiiio
(in_ar out sot_festureendnn bias-True)
[71: Tanh (1
14

---

Under review as a conference paper at ICLR 2022
E APPROXIMATE POSTERIOR PREDICTIVE DISTRIBUTION
Bayesian prediction proceeds by marginalization, i.e., averaging over all possible values of the model parameter. This results in the posterior predictive distributior
ply x. D..) == ply|æ, w w)p(w[D,) (D)) dw = (p(yl (z, w))((((P.)) (P.)
(18)
According to singular learning theory, Bayesian prediction via equation [8] is superior to MAP and MLE, in the sense that the expected generalization error of the posterior predictive distribution is smaller than that of the MAP or MLE. Specifically, let
G. (p. ())=)) ==
be the generalization error of predictive distribution (y)a))) According to Theorems 1.2 and 7.2 in Watanabe 2009), we have
EG, (G, P,)) - -/n o(l/n)
(19)
where the expectation is taken with respect D, and À is the RLCT of the model-truth- prior triplet. On the other hand, Theorem 6.4 of Watanabe (2009) shows that the expected generalization error of MLE (and similarly of MAP) is
( [r, wmte)) = C/n + o(1/n)
(20)
where C, the maximum of a Gaussian process, can be much larger than A. Such a distinction cannot be made in regular models in which the difference between the three estimators becomes negligible in the large n regime.
Now, consider the variational approximation to the posterior predictive distribution given by
Poo(ylz, D,.) = ()) ")) (w))
where q" is the optimal variational distribution in some variational family Q. It may be tempting to compare different variational approximations according to how well their respective variational posterior predictive distributions approximate the true Bayes posterior predictive distribution, e.g., Blundell et al. )015); Louizos & Welling (017)). This turns out to be a thorny issue, as documented various works on Bayesian neural networks Heek (2018); Krishnan & Tickoo (2020); Foong et al.) (2020). In particular, (e)(y| Dn)) does not necessarily inherit the desirable properties of the Bayes posterior predictive distribution p(y Lr, D,.. Here, we offer insights from singular learning theory to account for this.
Consider the (normalized) variational free energy
Fus(n)) E, (w) + KL(g(w))l<((w))
where is some variational distribution. the minimum (normalized) variational free energy (n) ming (n)( admits an asymptotic expansior") then it would be of the form
(n) = Xub log + (m.s 1) log log + Rn-
Note that An5 2 À necessarily. Now, if the generalization error of pos(ulæ, D,,) would admit an asymptotic expansion it would be of the form
(G,(pp D.)) Xot/n (oon))
'Though this may seem fundamentally distinct the maximum likelihood estimator (MLE) or maximum a posterior (MAP) solution commonly employed in training deep networks, both MLE and MAP may in fact be regarded impoverished estimates of equation 18 whereby p(w) (D.) is approximated with 5(w .)) a point mass at to
We should disclose that general conditions for such an asymptotic expansion the minimum variational free energy is still an open problem. The issue has so far been addressed on case by by-case basis, e.g., reduced rank regression Nakajima & Watanabel (2007), nonnegative matrix factorization Kohjima & Watanabel (2017):
Hayashi (2020) normal mixture model Watanabe & Watanabe (2006), hidden Markov model Hosino et
20055)
"Similar to the minimum variational free energy, expansion of F.G,, (pes(yla,,, D,)) has not been estab- lished in full generality at this point.
15

---

Under review as a conference paper at ICLR 2022
Importantly, Aub # Ass in general. Examples can be found where sometimes one is bigger, sometimes the other Nakajima & Watanabe (2007).
This development is to be contrasted with the Bayes free energy and the Bayes generalization error. Recall the normalized Bayes free energy, F, == log (n),, admits the following asymptotic expansion
F Alogn + (m 1) log log + Rn
(21)
The very same coefficient appears in the asymptotic expansion of the Bayes generalization error in equation 19 This means that minimizing the Bayes free energy is equivalent to minimizing the Bayes generalization error. In contrast, minimizing the variational free energy (equivalent to maximizing the ELBO) does not necessarily lead to lower generalization variational free energy (higher Anb) may have lower variational generalization error (lower 7oo).
since variational family with higher
16

---

Under review as a conference paper at ICLR 2022
Table The model of interest is the realizable tanh- with H hidden units and true weight wo O. The prior the model parameters is taken to be (ww) o For various combinations values and values for (u({²), (v)), we display the average and standard deviation of (g achieved after training a dataset of size 5000 over 10 Monte Carlo iterations.
count
mcan
std
(pp))
Ag/Ao = po
method
0.00
af_ gaussiaan 0_1
10.0
-12307.1 14
1153.17
nf_ nf_gumma__uu Laataaa__________1 10_1_100_________ False
10.0
-10894.83
836.64
nf nf_gamma, 10 1_ 100. True
10.0
-10768. 14
894.35
0.10
af gamma
1.0 False
10.0
-12476.67
877.12
nf gamma__ 10 (0_5,
True
10.0
11849.96
415.19
nf gwussiau.nnnmmm. 0.1 0.0001
10.0
-06044.6
528.57
nf gumgma
[_lse
32804.39
330.57
gf_gamma_ 100_ 1_100 True
10.0
11759.06
610.11
1.0)
1.00
nf gamma. 100 5_100 False
10.0
-14328.32
1215.64
nf__gumma_
1__10__Trru
10.0
-3309.331
1022.25
nf no__ssnsiann l_lc-2
10.0
11658.58
713.89
nf gi_gumma__. 300
100__FFllle False
10.0
-16913.87
2048.86
nf_gamma
5.00
af_gamma 25eaa_500____0 1,00_5_1100___0 Falsee
10.0
-1544.553
639.40
nf__amma___t 500 $_1000__110001 True
10.0
-15079.50
974.71
nf ga_sssnnian, Se-2
-20300.63
2274.93
576
0.00
nf__aassian__ 0__
10.0
4694.71
42.93
nf gammma 10 1_100 False
10.0
.919.577
85.38
af gammaa 1_1___00001111111111111111111111 True
10.0
-492..69
75.24
0.10
nf_gamma_ 10_5 10l_False
10.0
-6036.98
511.80
nf_ nf__gumma_ 10 100 True
9.0
-5999.62
116.19
nf gaussian 0.1. 0.001
$3.72
nI_ ni_gumma_ 100 TOU FalseFlsse False
10.0
54611.3
2:6.25
nf gamgamma, 100
True
10.0
5060.05
186.63
(5.0, 100.0)
1.00
af gamma_ 100_5_100. False
10.0
-7590.89
254.81
nf_gumma_ 100 100_5_1000____1__0____0001000110 10uurTrue
10.0
-6098.62
307 36
af gaussian 1e-2
10.0
8010.75
9394.81
af gamma 300
False
10.0
-7144,50
432.06
nf gamgmm 500 100 True
10.0
-684.81
199.32
5.00
af gamma 500 5_100 False
10.0
-8392.05
207
nf nf_gumma_5000 1asaa__50050010000000 1110___10_______
9.0
-8475.71
412.61
nf gaussian 5 5_5e-2
10.0
10974.67
4667.22
0.00
nf gaussian 0_1
-24543.22
693.44
gf_gumma_ Le____10__Fattta
10.0
-20607.2 23
1776.76
nf gammam 10 1_100 True
5.0
-19642.01
449.75
0.10
nf gfmgum $_100__Falee False
9.0
-25668.64
1783.73
nf gamma_ 10 10_5__ 100_ True
9.0
25693.887
2344.70
nf gaussian 0.1 0.001
10.0
-20014.07
505.24
nf gumgmmm
___00__Fasss False
10.0
-26573.95
1378,88
nf gu_ma_m__ 100_ 1_100 TTue
10.0
26255.66
2921.10
(5.0, 1.0)
1.00
nf gummaa 100 $_100__110F0000 False
9.0
-28239.06
1348.93
nf gamma__
100_TTuee
af goussian.. 1 1_le-2
10.0
-24311.66
2924.31
nI_ mi_gumma 300 100 1110100alllse False
10.0
-35135.85
10110.49
gf_mamaa____ 550 1_100_Trrum True
5.00
nf_gumma_5500 55005_100000 5_100__alsee Fulse
10.0
-31770.45
1765. 41
nf gammaa_ 500 100 5_100__True Truee
8.0
-31673.
2167.5
af gaussian 5 Se-2
10.0
-243192.
271227.44
1024
0.00
nf_ ga__saaiin 0_
10.0
-8396.27
62.13
nf_gamma, 10 1_100_1110_101111111 L_100__FFaat False
10.0
-705.24
77.06
af gamma, 1_100_00_____11111 True
10.0
0.10
nf nf_gamma_ 10 S. 100 False
9.0
-11616.24
1014.68
af gamma, 10. 0, 100 Trme
8.0
-10916.36
377.03
nf guussian. 0.1 1.0.001
10.0
-8425.56
71.42
nf_gumma_ T00 100, 1000_ False
10.0
T0916.45
479.14
(5.0. 100.0)
af gamma, 100
Trme
10.0
-9514.43
397.57
1.00
af gamma_ 1_1_1100_____1__1 1100__00____000011 False
9.0
-14374.43
1187.21
af gamma, 100 10005_000000000001 True True
8.0
-12994.13
765.11
nf gaussian Ie-2
10.0
-9408.79
480.26
ni gu__mamm__t 500
False
10.0
-2182585
9739.67
af gamma, 500 100 True
10.0
14
94773.50
5.00
nf gamma 500 500 1__10__Faart False
9.0
-12577.90
282.52
nf gf__maam_500 500 11000551000011111111111111101111 True
8.0
-16369.58
1287.13
nf gaussian_ 5_5c-2
10.0
1182386008
155290.52
17

---

Under review as a conference paper at ICLR 2022
Table The model of interest is the realizable tanh- with H hidden units and true weight wo 5. The prior the model parameters is taken to be (ww) o For various combinations values and values for (u({²), (v)), we display the average and standard deviation of (g. achieved after training a dataset of size 5000 over 10 Monte Carlo iterations.
count
mean
std
(pp))
Ag/Ao = method
0.00
af_ gaussiann 0_1
7.0
-481210,04
290152.33
nf_ nf_gumma__uu Li_taaa__________1 11____10__Faai False
6.0
-564637.97
185632.52
nf nf_gamma, 10 1_ 100. True
9.0
466874.65
229687 .03
0.10
af gamma nf gamma__ 10 (0_5_ Trme
1.0 False
10.0
-189065.78 73
110357.18
10.0
66869.47
18069.35
nf gwussiau.nnnmmm. 0.1 0.0001
9.0
-386620.24
270119.27
nf gumgmmm
___00__Faasss False
190311..0
142999
gf_gamma_ 100_ 1_100 True
10.0
-130158.69
76731.80
1.0)
1.00
nf gamma. 100 5_100 False
10.0
-18865.52
14299.86 86
nf__gumma_
5_100__ruuu
10.0
-53566.72
18364.18
nf no__ssnsiann I l_le-2
10.0
544933.96
194349.17
nf gi_gumma__. 300 100__FFFllee False
10.0
-89427.22
17721.7
nf_gamma
5.00
af_gamma 25eaa_5005__0 1100___110__FlF_
10.0
-33854,50
8232.85
nf_gamma__St 500 $_100__110000 True
10.0
-32275.02
7026.42
nf ga_sssnnian, Se-2
1008876.71
26177.59
576
0.00
nf__aussian__ 0__
8.0
407843.61
96924.16
nf gammma 10 1_100 False
5.0
-03729. 89
39713.42
af gammaa 1_1___000011111111 True
407507.22
172978.81
0.10
nf_gamma_ 1a_iaa______________1111111 10l_False
10.0
-154222.94
81217.70
nf_ mf__gumma_ 10 100 True
10.0
-64598.97
46518.60 60
nf gaussian 0.1. 0.001
313547.55
337434.22
nI_ ni_gumma_ 100 TOU _aiseFF False
10.0
-170076.39
81803:27
nf gamgam 100
True
10.0
-112674.00
72282.16
(0.0, 100.0)
1.00
af gamma 100_5_100 False
10.0
-37469.45
24653.37
nf_gumma_ 100 100_5_100____1 10000_uruee
10.0
47133.005
34424.37
af gaussian 1e-2
10.0
-23330.71
212480.52
af gamma 300
False
10.0
69257.33
21314.18
nf gammam 500 100 True
10.0
-8890.81
12706.57
5.00
af gamma 500 5_100 1.00_Foot False
10.0
-1709..24
6511.72
nf nf_gumma_5000 1a_aa__50000110000000 10_1__511__r___
10.0
-14548.94
4430.04
nf gaussian. 5 5_5e-2
10.0
-8386690
26349.19
0.00
nf gaussian 0_1
3.0
-85765.67
77889.00
nf_gumma_ 1_100__Faaeee
2.0
-924993.41 41
100733.15
gammam 10 1_100 True
7.0
-671229.30
160802.33
0.10
nf gummma $_100__Falas False
8.0
-558522.30
252813.58
nf gamma_ 10_ 100_ True
10.0
279065.90
225130.01
nf gaussian 0.1 0.001
6.0
452279.64
129732.34
nf gumgmmm
___00__Fasss False
10.0
-347688.5
61671.23
nf gu_ga_m__ 100_ 1_100 True
10.0
-213500.49
64295.75
1.0)
1.00
nf gummaa 100 5_100__11FF000 False
10.0
-183015.46
128751.22
nf gamma__
TTue
af goussian.. 1_le-2
6.0
8336952.29
71929 66
nI_ ml_gumma 300 100 1110100Pllsssss Palse
10.0
-196193.05
44549.27
gf_mamaa____ 550 1100_Truee
-179038.6
52969.12
5.00
nf_gumma_5500 111____11_____0
10.0
-66127.35
6789.22 25
nf gammaa_ 500 100 1_100__Truu Truee
10.0
-67 85
17901.20
af gaussian 5 Se-2
10.0
-216075.28
71401.02
1024
0.00
nf_ ga__saaiin 0_
3.0
-750263.23
140506.97
nf_gamma, 10 1_100_111___01111111 1100_Fatl False
3.0
-780374.21
154817.19
af gamma, 1_000_000____0011 True
7.0
-779285.19
156719.96
0.10
nf nf_gamma_ 10 S. 100 False
8.0
484597.1 15
24K508 84
af gamma, 10. 100 Trme
10.0
-277501
268802.22
nf guussian 0.1 1.0.00
5.0
-21535.06
238781.26
nf_gumma_ T00 1000_00at False
10.0
286336.87
64751.80 89
nf gamma, 100
Trme
10.0
48948.50
35427
(0.0. 100.0)
1.00
af gamma_ _10_1100_1_1111111 $100_0000__0000001 False
10.0
-163008.27
158054.21
af gamma, 100 10005_00000000000 True True
10.0
-101276.96
77255. .66
nf gaussian Ie-2
8.0
-82733.30
100242.24
gu__mamm__t 500
False
10.0
1T6TOTO.
35334,52
af gamma, 500 100 True
10.0
-140923.51
58184.19
5.00
nf gamma 500 500 1__100FFaaat False
10.0
45679..41
21039.66
nf gummaa__5500 500 10000_51000011111111111111101111 1e_1000Trrrr
10.0
49196.44
56363.79
nf gaussian_ 5_5c-2
10.0
-147644.89
32270.53
18

---

Under review as a conference paper at ICLR 2022
Table 4: The model of interest is the realizable reduced rank regression model with H hidden units and true weight Wi = (Ao. Bo) where Ag and By are as described in Section 7 The prior on the model parameters is taken to be (pw) N (u({)) ( (a)) For various combinations of values of Il and values for (p(s)), (p)), we display the average and standard deviation of / (g g) achieved after training dataset of size n 5000 over 10 Monte Carlo iterations.
count
std
H (p))
Ao/Bu =
method
0.00
of miLgassiian. 0_1
10.0
-87415.82
849.17
of_gamma 10 1_10__10_FF____1 Fulse
10.0
-19679.77
43.94
of g__mma_mm00, __100_ True
10.0
-19633.00 .08
59.80
0.10
of guggmmm 10_5. 10.55_1100 5_100 False
2.0
-20163.30
41.30
gammaa 10_5_
of gaussias 0.1 0.001
10.0
19356.22
148.83
Df gamma 100 100 False
10.0
19936.36
42.41
gi__ammu 100_ 1_100__Thr
119908.22
49.62
(5.0, 1.0)
1.00
of giL_gmmm__ooo Laeee__0000___ 1__00____Faaaa__
6.0
20538.223
98.10
nf garmaa 100 5. 100, True
202277.8
146.99
gaussian
10.0
19974.33
49.01
Dt ot_gamma__s 1atmmmaa_500_0000 [_100_FFFaal False
10.0
20362.09
124.35
nf m[_gamma. 5000
gamma 1_100_111TT___
10.0
20367 47
122.89
False
NaN
nf ga_gmmma___m 500 1110_5_1000'T01 True
1.0
21310.91
NaN
of gaussian 5_5e-2
10.0
-24732.96
1798.18
gaussian
10.0
-75338.44
767.59
nf_ g[gammm___ 10_ __0 False
10.0
-8049 29
24.39
0.10
of gamma 10, 1_ 1_100 100 True
10.0
-845.36
17.34
gaussian 0.1 0.001
10.0
-8012.99
13.94
of gamma 100 100 False
10.0
8179.44
6148
gamma. 100 1_10_110001111111111111111111111 True
10.0
-8110.22
44.60
(5.0, 100.0)
1.00
-8675.67
NaN
nf gammam 100 1_10__000____11111 100_TTuee
1.0
8367.80
NaN
of gaussiannn 1e-2
10.0
-8164.30
54.74
gu-mmmm 500 False
10.0
8472.94
107.69
5.00
of gammaa 500 1_100_ True
10.0
-8510.36
107 44
nf_ of_gaussiun 5_5e-2
10.0
-12594.11
1731.59
0.00
gaussian [1]
1010
152649.17
X34
of -i_gamma 10. 1100 False
10.0
334709.70
86.32
gf_gamma__ 10_ 1_10__0000000000
10.0
-34642.82
94.35
nf gaggasisnn ().1 0.001
10.0
-33988
181.45
nf [i_gamma__ 100 100 False False
10.0
-35191 26
82.93
(5.0, 1.0)
g__aamma
100100_110 True
:31127.52
of gagssian lc-2
10.0
33171.66
113.62
nf [i_gamma 500 1 False
10.0
-3597888 89.8
225.51
gamaam
1_100_Thuu
.55
nf gaussiin 5_5e-2
10.0
-56022 20
13175.17
0.00
nf gaugssiinnn 1I
1010
-13147
1093.66
af gamma 10_ L_lse_False
10.0
-14065.12
$2.64
0.10
of gamma 10_ 11000_ True
10.0
-14041.35
26.66
nf gaussian 0.1 0.001
10.0
-13990.93
31.67
(5.0, 100.0)
gamma,
False
10.0
-12293.75
1.00
nř' gamma. 100. 1 True
10.0
.14152.35
71.25
af gaussian le-2
10.0
-44272.72
99.10
gamma 500 1_00__000_000000 False
10.0
-14874.08
126.36
5.00
nf ___gmmm__500 150__5000111 1 11001100001100000 True
10.0
-14906.99 96
144.99
nf gaussian_nn 5_5e-2
10.0
333611.96
10019.58
19


Reviews:
[{"review_id": "cAdsjr1S0H", "reviewer": "Reviewer_kf3F", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is the first time I hear about the theory. I think the topic is very interesting. \n\nHowever, the paper has several issues. First, I feel that the theoretical improvements over previous methods are not significant. Most theoretical results are just minor improvements such as making assumptions less restricted. The main result in 5.1 is still similar to [Bhattacharya et al. 2020], which already has the conclusion that a mean-field distribution allows the estimation of lambda and m. \n\nSection 3 seems to be purely from previous work. \n\nSecond, the experiment is done on only two toy problems. The dataset has 5000 instances while the network has two hidden layers. Do you actually check whether the model is singular or not? I hope to see experiments based on realistic datasets. Can we also include other variational inference methods as the baseline when estimating the normalized evidence? \n\nThird, I don't know how significant it is to use normalizing flow to learn the resolution map? Is the analysis still valid for the complex flow model? In practice, the method has competitors from other variational inference methods. How practice is this method in estimating the normalized evidence. \n\nFourth, the writing of the paper can be greatly improved. A lot of my understanding depends on previous work by [Bhattacharya et al. 2020]. Here is a list of detailed issues that block my understanding. \n1. I think the paper should make clear the Laplacian approximation from the beginning and also the significance of estimating lambda and m. \n2. Equation 6, \"in the asymptotic expansion of ...\" what expansion? The paper may need to explain the expansion a little. \n3. Should section 3 go to the introduction since it is not the contribution of this paper? \n", "summary_of_the_paper": "The paper analyzes the approximation of the log-ratio (called normalized evidence in the paper) (the log-probability under the model minus the log-probability under the true model). Since the model is singular, the Laplacian approximation of the normalized evidence needs a new method. The mean-field distribution and a resolution map allow us to estimate two important numbers (lambda and m) in the approximation. \n\nThe paper has made several contributions to the study of this problem in several aspects. First, the paper improves previous results by relaxing the assumption. Second, the paper actually uses a normalizing flow to actually learn the transformation. \n\n\n", "summary_of_the_review": "The paper studies an interesting topic. However, there are several issues such as minor improvements over previous methods, practical value, and also writing problems. Overall, this is an interesting paper, but the quality is not very high. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "SVcBbaOO-L4", "reviewer": "Reviewer_jfja", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n- The authors make an effort to exploit recent progress in singular model theory\nfor developing practical variational methods\n- The summary of the related work is well presented, and appears to be comprehensive.\n- The authors are clear about the limitations of their approach.\n\nWeakness:\n- Theorem 5.1. does not seem to tell more than Theorem 3.1 in [Bhattacharya et al. (2020)].\nAm I missing here something?\n- The main contribution seems to be the effort to exploit the theoretic idea of a mean-field variational approximation with truncated Gamma distributions. However, the relationship between the theoretic idea and their experiments appears ad-hoc since in their experiments they do not use a mean-field approximation, but rather use a normalizing flow with a Gaussian source distribution that \ntries to approximate an untruncated Gamma distribution. Why is it an untruncated Gamma and not a truncated Gamma distribution?\nWhat is the relationship between the mean-field approximation and the normalizing flow?\n- Since the authors use a Gaussian approximation for the Gamma distribution, it would be interesting if the authors could show whether and when the approximation is justified in their setting. (It appears to be correct, since n is rather large)\n- The presentation of the experimental results needs improvement.\nThe experimental results in Table 2ff are essential, but can only be found in the appendix.\nThey should be summarized in the main paper.\n- Actually it is not clear to me why the Theorem 3.1 in [Bhattacharya et al. (2020)] suggests that \nthe ELBO found with a gamma mean-field approximation is better than a normal mean-field approximation.", "summary_of_the_paper": "The paper builds on the recent theoretic results of [Bhattacharya et al. (2020)],\nwhich shows that a mean-field variational approximation with carefully chosen approximation family\nleads to an ELBO $\\Psi$ which is sharp up to a constant C(d) which only depends on the dimensionality d of the parameter space.\nFor the proof, [Bhattacharya et al. (2020)] assumes that the approximation family is a Gamma distribution truncated to [0,1]. \nTherefore the authors of this paper conjecture that the Gamma distribution truncated to [0,1]\nis a good choice for the source distribution of a normalizing flow.\nTheir experimental results suggest that this choice leads a higher ELBO than using as source distribution a normal distribution.", "summary_of_the_review": "The idea of trying to use recent singular model theory to improve variational inference is interesting.\nHowever, this work, in its current form, leaves too many open questions about the relation between theory and their experiments.\nFurthermore, it is unclear what the take-home message is.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LBHn1YtOR_T", "reviewer": "Reviewer_32H4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The writing is generally clear. Although there are innovations, such as using affine coupling network to learn the desingularization map, but I am not quite sure about the contribution of this paper. Equation (8) is misleading because it is not exact equality but should be $\\asymp$, i.e., $\\sup_{q\\in\\mathcal{Q}_{(0,1]}}\\Psi_K(q,g)$ is bounded above and below by the right hand side up to constant multiples (see (15)). Therefore the lower bound of (8) is similar to the lower bound obtained by Bhattacharya et al. (2020) in (7). To improve upon their result, you need to show that $-\\log{C(d)}$ is smaller than their constant.\n\nI don't think recovering the leading order term of $\\log{\\bar{Z}_K(n)}$ guarantees that the mean field generalized gamma family is the optimal approximating family (as suggested in Pages 20-21 in Bhattacharya et al., 2020 and implicitly in this paper). There are many examples in the literature where dependent approximating families give clear improvement with respect to mean field (see \"Variational Inference with Normalizing Flows\" for example).\n \nHere are some other questions:\n1. Why do you work with $\\Psi_K(q,g)$ and $\\bar{Z}_K(n)$ in (4) and (5)? The results would be more significant if their stochastic counterparts $\\Psi(q,g)$ and $\\bar{Z}(n)$, which are the quantities of main interest, are considered instead.\n\n2. Why did you choose the generalized gamma in constructing the mean field approximation? It seems to me that other distributions with tails heavier than Gaussian, such as the $t$-or Cauchy distributions, would also work. \n\n#########################################################################################################\nTYPOS:\n1. In Section 2 the first display, there should be a $dw$ in $\\bar{Z}(n)$\n2. In Page 8 the last two paragraphs, write that Tables 1-4 are in Appendices D and E\n", "summary_of_the_paper": "Working with Bayesian neural networks, this paper proposed a variational algorithm to approximate posterior distribution of the network weights. To overcome model singularities, the authors used the idea of normalizing flow by transforming the weights through an affine coupling network, and subsequently worked on the desingularized parameter space. In addition, they derived an asymptotic expression for the ELBO, and compared the Gaussian and generalized gamma approximating families in the experiments.", "summary_of_the_review": "I have doubts about the contribution of this paper and its significance to the literature.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "E9x6qlkT2F", "reviewer": "Reviewer_PvQE", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Points marked with (*) are especially important, and should be addressed if the authors would like me to reconsider my score. \n\nStrengths:\n1) I enjoyed the discussing relating the ELBO score with the quality of the variational posterior predictive, and how good ELBO scores do not imply good posterior predictive distributions. Thank you for including this.\n\nWeaknesses:\n1) *As I understand, Eq. (1) shows for non-singular models, the posterior distribution can be expressed in a specific form i.e., a mixture of standard form distributions. However, this result only applies \"when n is large\". The discussion of what it means for \"n to be large\" is extremely limited. I expect this to depend on the network in question. Further, it is entirely unclear to me whether practical Bayesian Neural Networks are in a \"large n\" regime. This issue, at the very least, should be discussed within the paper. \n\n2) *The technical contribution of the paper, at least in terms of a practical inference method, seems highly limited in terms of novelty. The paper proposes using a normalising flow approximate posterior with a Gamma source. However, for expressive flows, the choice of source distribution can always be absorbed into the flow. And flows have regularly been used for posterior inference. The contribution here is thus highly limited.  \n3) The readability and presentation of the paper could be improved. For example, equations could be named when referred to within the text. Intuition could be added on why neural networks are singular models. The figure captions are hard to understand without referring back to the text extensively. Experiments could be named, much more of the results could be in the main paper, intuition behind theorem conditions could be provided. I think this is especially valuable as singular learning theory is not yet well known within the machine learning community. \n4) *The experimental evaluation of the paper is highly limited. I understand that simple networks are required to apply singular learning theory,  but the proposed method could be applied to much larger networks. The experiments within the related work, which is almost exclusively work within the Bayesian Deep Learning community, typically uses experiments with a much larger scale. Further, not all runs reached convergence, which undermines the claims made in the paper. \n5) *The technical results of the paper assume that b \\propto 1 (page 6), but the authors write that this condition is not expected to hold in reality, even for very simple experiments. This seems to be a major concern. This should, at the least, be mentioned when the theoretical results is discussed within the paper. The authors go on to write \"we do not believe this assumption is critical\", which as far as I can tell, is conjecture, and not convincingly discussed within the paper. \n6) Although the authors use a Gamma source distribution, in practice, they approximate this with a Gaussian distribution. This seems to further undermine the technical novelty of the paper. \n7) *The flow that is used within the paper is very shallow—it only has two coupling layers. It is well known that many layers of affine couplings to have expressive distributions. The authors claim \"though learning the variational parameters, k, in the source distribution goes against conventional wisdom in normalizing flows, our experiments suggest some performance may be gained by learning the optimal untruncated generalized gamma source distribution at the same time as learning the resolution map\". This claim is not well supported; the experimental results use particularly non-expressive flows, and as such, this claim only stands in this context. And within this context, I expect that learning parameters of the source distribution is increasing the expressivity of the flow. The conventional wisdom holds primarily for expressive flow distributions.  \n8) The authors write \"for large n, the posterior is not Gaussian, but a mixture of standard forms\". However, note that this is true only *post flow transformation. i.e., the distribution of the weights when transformed into a new coordinate system follows this form. However, in general, flows can express arbitrary distributions. This result seems to be somewhat oversold. \n\nClarifications:\n1) As far as I can tell, the theory assumes that the space of parameters is a compact set. However, in BNNs, we typically place prior support over the entire reals. This seems to be a theory-practice mismatch. Is this problematic?\n2) The asymptotic expansion of the normalised model evidence holds only for large values of n, right? I'm concerned about how this affects the validity of the theory. \n3) The authors write \"we are aided by the fact that a resolution map can attain the optimal value of Psi_K(q, g)\". How/where is this demonstrated? Please clarify.  \n4) Singular learning theory provides this resolution map, which in practice is learnt using a normalising flow. However, the flow is actually learning by maximising the ELBO objective, which is a lower bound on the likelihood. I don't understand why **in the context of singular learning theory** which maximising the likelihood (approximately) is a sensible target for the flow. I think it is a sensible target, when considered in the context of variational inference.", "summary_of_the_paper": "Inspired by asymptotic results from singular learning theory, the authors of the paper propose using a generalised gamma mean-field distribution with a normalising flow (that targets the \"desingularization map\") to perform variational inference. The authors additional build on prior work to derive a tighter bound on the log normalised evidence for variational inference using this approximate posterior combined with the \"correct\" desingularization map. \n\nNote: while I am familiar with the Bayesian Neural Network literature, I am not familiar with singular learning theory. This is reflected in my confidence score. As such, I was unable to verify the theory in the paper, and assess the significance of the theoretic results included in the paper.", "summary_of_the_review": "I have a number of concerns about the paper. In short, (i) limited technical novelty (ii) the theory does not seem to apply in practice (iii) the experimental evaluation is limited, and uses only small scale experiments (iv) room for improvement in terms of presentation e.g., many of the results are included only in the appendix. \n\nAs a result, I do not believe that this submission, in its current form, is of interest to the ICLR community. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a variational inference based on singular learning theory (SLT), where the resolution of singularity is learned by normalizing flow so that the latent distribution is factorized.

Pros:
- A unique idea to use SLT for variational inference.

Cons (only serious concerns):
- Goal is unclear.  The authors say that they propose variational inference based on SLT.  But apparently, they propose it not as an alternative to the state-of-the-art variational inference for neural networks (if so the experiments shown are far from the acceptable level).  The authors must clearly say for what purpose they propose a new method.  I would guess the proposed method is for analyzing singular models to compute their RLCT.  In that case, the authors should compare with existing methods for evaluating RLCT, e.g., MCMC based methods:

K. Nagata and S. Watanabe, "Exchange Monte Carlo Sampling From Bayesian Posterior for Singular Learning Machines," in IEEE Transactions on Neural Networks, vol. 19, no. 7, pp. 1253-1266, July 2008, doi: 10.1109/TNN.2008.2000202.

and discuss pros and cons of the proposed method.  For DNN, you should use the state-of-the-art MCMC sampling methods like

Wenzel, F., Roth, K., Veeling, B., Swiatkowski, J., Tran, L., Mandt, S., Snoek, J., Salimans, T., Jenatton, R. &amp; Nowozin, S.. (2020). How Good is the Bayes Posterior in Deep Neural Networks Really?. <i>Proceedings of the 37th International Conference on Machine Learning</i>, in <i>Proceedings of Machine Learning Research</i> 119:10248-10259 Available from https://proceedings.mlr.press/v119/wenzel20a.html.

as a baseline.  Approximating the posterior with normalizing flow can be another baseline.

- Large n issue.  SLT can be seen as a generalization of the asymptotic learning theory for the regular model, where the model complexity is represented by the parameter dimension d, and "asymptotic" means n >> d.  Watanabe revealed that the model complexity cannot be represented by d in singular models, and therefore the definition of "asymptotic" is not as clear as the regular case.  But it is known that typical neural networks are overparameterized and can achieve zero training error.  I have seen no work arguing that SLT holds in this regime.  If the authors insist that their method is applicable to deep neural networks, they should cite references where it would be proved that SLT holds in the overparameterized regime or prove it by themselves.

There are many more concerns including those pointed out by reviewers, and the paper is not ready for publication.
Decision: Reject