Linearised Implicit Variational Inference
Authors: Anshuk Uppal
Wouter Boomsma
Jes Frellsen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Implicit models
Variational Inference
Bayesian Deep Learning

Abstract: Bayesian neural networks (BNNs) are touted for robustness under data drift, resilience to overfitting and catastrophic forgetting whilst also producing actionable uncertainty estimates. In variational inference, these elegant properties are contingent on the expressivity of the variational approximation. Posteriors over parameters of large models are usually multimodal and highly correlated and hence cannot be well-approximated by simple, prescribed densities. We posit implicit variational distributions specified using differentiable generators are more flexible and propose a novel bound for training BNNs using such approximations (amortized neural samplers). The proposed bound uses an approximation of the variational distribution's entropy by locally linearising the generator. Unlike existing works, our method does not require a discriminator network and moves away from an unfavourable adversarial objective. Our formulation resembles normalizing flows but does not necessitate invertibility of the generator. Moreover, we use a differentiable numerical lower bound on the Jacobians of the generator, mitigating computational concerns. We report log-likelihoods on UCI datasets competitive with deep ensembles and test our method on out-of-distribution benchmarks.

Under review as a conference paper at ICLR 2023
LINEARISED IMPLICIT VARIATIONAL INFERENCE
Anonymous authors
Paper under double-blind review
ABSTRACT
Bayesian neural networks (BNNs) are touted for robustness under data drift, re- silience to overfitting and catastrophic forgetting whilst also producing actionable uncertainty estimates. In variational inference, these elegant properties contin- gent on the expressivity of the variational approximation. Posteriors over parame- ters of large models usually multimodal and highly correlated and hence cannot be well-approximmted by simple, prescribed densities. We posit implicit varia- tional distributions specified using differentiable generators more flexible and propose a novel bound for training BNNs using such approximations (amortized neural samplers). The proposed bound uses an approximation of the variational distribution's 's entropy by locally linearising the generator. Unlike existing works, our method does not require discriminator network and moves away from an unfavourable adversarial objective. Our formulation resembles normalizing flows but does not necessitate invertibility of the generator. Moreover, we use a dif- ferentiable numerical lower bound on the Jacobians of the generator, mitigating computational concerns. We report log- liceliooss on UCI datasets competitive with deep ensembles and test our method out-of- of-distributtin benchmarks.
INTRODUCTION
Deep neural networks considered state of the in numerous tasks in computer vision, speech and natural language processing. Scaling up neural architectures has led to outstanding performance on myriad of generative and discriminative tasks, albeit some fundamental flaws remain. Neural networks are usually trained by maximising likelihood resulting in a single best estimate of parameters which renders these models highly overconfident of their predictions, prone adversarial attacks and unusable in risk-averse domains. Furthermore, their usage remains restricted in sequential learning applications due to catastrophic forgetring (McCloskey Cohen [989) and Kata-scarce regimes due to overfitting. When deployed in the wild, deep networks do not output comprehensive measure of their uncertainty, prompting expert intervention.
The Bayesian paradigm provides solutions to a number of these issues. In summary, Bayesian neural networks specify a prior distribution over parameters p(0), and the neural network relates the parameters to the data D through likelihood p( p(O))) The goal is to infer conditional density over the parameters, called the posterior ((0 given by the Bayes' rule,
p(0 = ((()))(()) p(D(0))(0) p(D)
(1)
J ()
This conditional density provides a range of suitable parameters with a probability over them given by the dataset. After training, predictions from an ensemble of parameters (models) can then be combined, weighted by their posterior probability forming a Bayesian model average (BMA). The variance of these aggregated predictions informs the user/human about the model's confidence in particular prediction. Finding the normalization constant in eq. T is analytically intractable for large models, and hence there is clear focus approximate inference techniques. Various approaches have been proposed, including Markov chain Monte Carlo (MCMC, Neal] 1995), variational inference (VI, et al. 1996 Peterson
1987) and the Laplace approximation (Mackay 1991).
Variational inference is a strategy that converts the inference problem into an optimisation over a family of distributions (variational family), denoted hereafter by Q, indexed by variational parameters denoted by y. We optimise y using a lower bound on the marginal likeliood of the data log (DD) called the evidence lower bound (ELBO) Usually, are computationally limited to choosing simple

---

Under review as a conference paper at ICLR 2023
DE
MFVI
LIVI acc-jac
LIVI diff-Ib
HMC
MNF
predictve
COnd rnberve
mmas
Figure 1: Model confidence on toy regression. We compare both lower bounds for implicit variational inference presented (LIVI) in this work with standard uncertainty quantification approaches deep ensembles (DE), mean-field variational inference (MVI) Hamiltonian Monte Carlo (HMC) and multiplicative normalizing flows (MNF, Louizos Welling, 2017). We train using 25 dimensional noise input to MLP generator having a single hidden layer consisting of 25 weights modeling 105 neural network parameters. Here we plot the 5-95% percentile of the BMA represent model uncertainty. Notably both our approaches capture in-between uncertainties (Foong et al. 2019).
distribution families like an isotropic Gaussian distribution (Tanaaa 1998 Blundell et al. 2015)
The true posterior is much more complex and is approximated poorly using such approximations (Foong et al. 2019 2020). This issue is exacerbated in large models that contain many symmetries and correlations. Notably, there have been attempts to extend VI to more structured and expressive distributions (Saul & Jordan 1995; Bishop et al., 1997 Louizos & Welling, 2016) yet, capturing correlations between parameters with a flexible variational approximation remains the Achilles heel of these class of models.
We propose an approach based on implicit generative modelling where the distribution over vari- ables of interest is implicit and can only be sampled. This is in contrast to usual VI methods that use prescribed distributions with explicit parametrisation as the approximating density over latent variables (Diggle & Gratton 1984 Mohamed & Lakshminarayanan 2016). Although, this idea takes inspiration from GAN generators that try to recover the true data distribution, we do not require a discriminator network for training the generator and as result do not suffer from the complicacies introduced by an adversarial objective. As emphasised by Tran et al. (2017). is more natural way of capturing the generative process instead of forcing it to conform an assumed latent structure which could be misspecified.
Similar other works in implicit VI (Shi et al., 2018), we posit using general (non-invertible) stochastic transformations that can produce highly flexible implicit densities to model posteriors of neural networks. We believe that these approximations can better capture the intricacies of posterior landscape. Additionally, when trying to model complicated densities in high-dimensions, it is sensible learn a sampler instead of parameters of an expressive intractable approximation, especially these approximations do not admit one-line samplers (Devroye, 1996). For example, EBMs can be very flexible but are not easy to sample from (Song Kingmal 2021).
If we were to use fully correlated Gaussian to model the posterior of a neural network, we would need to optimize parameters quadratic in the number of weights of the network, (dim()))² to arrive at the optimum covariance matrix. In this work, we test our hypothesis of using under- parameterised generator to capture the important correlations in orders magnitude less parameters than that. At the same time, we hint the possibility that constrained generator will probably avoid modelling redundancies present in BNN posteriors like permutationally symmetric modes.
Succinetly, contributions presented follows:
We derive a novel lower bound for variational inference in Bayesian Neural Networks using implicit variational approximation avoiding unstable minmax objectives.
We augment this lower bound by reducing its compute requirement, as we substitute differentiable numerical lower bound for the entropy term comprising of Jacobians of neural networks.
We comprehensively empirically evaluate the capacity of this implicit variational approxima- tion and the quality of the posteriors inferred using different out of distribution benchmarks.
2

---

Under review as a conference paper at ICLR 2023
2 VARIATIONAL INFERENCE FOR BAYESIAN NEURAL NETWORKS
Consider the supervised learning setting, where we have training set D {(x,, y,) where X == [ are the covariates (inputs) and y = {Mi 1 are the labels (outputs). We consider
Bayesian regression classification model given by
pDD = = (2()))))(0) p(0) II p(u 2,, 0),
where the likelihood is parameterised by 0 E e Rm The objective function VI, called the ELBO is lower bound on the log marginal likelihood of the data log p(D), and the discrepancy between the two equal to the KL divergence between the approximate and true posterior given by
log P(D,0)
DRL (k).l log (D) -Bg-a, (t)
(3)
47 (7)
(())
where 44 is the variational approximation of the posterior with parameters 7 Г. Since the KI. divergence is non-negative, L is a lower bound on the evidence. This objective function can be written in terms of likelihood term and regularisation term
[(>) = (a [log (D
DRL.(4, (0))((())
S log P(D),
(4)
likelihood
regularisation term
where the likelihood term promotes the variational approximation model the data well and the regularisation term keeps the posterior close to the prior. Since the log-evidence, log p(D), does not depend on % minimising the the KI. divergence is equivalent to maximising the ELBO, i.e.,
arg min ( = arg max L(7).
(5)
2.1 IMPLICIT VARIATIONAL INFERENCE
In implicit VI (IVI), the variational distribution is only implicitly defined through its generative process over the parameters 0
=~g(=>, (2),
where the is fixed base distribution and 9- Rd + IRm is non-linear mapping and typically not diffeomorphism. For IVI, the likelihood term from eq. (4) and its gradients be estimated using Monte Carlo and the reparameterization trick. However, the regularisation term is more difficult as involves the entropy of q-,,
D (K)l (0)] (o) log (,) p(0) Hg(gg)) (0) [logp(0). p(0)]
Generally, the entropy of the generative process in eq. 6 is not available in an explicit form as the density of the process is not tractable. prevalant technique estimate the regularisation term uses density ratio estimators based on a GAN-like Iiseriminator(Sugiyan et al. 2012 Huszár (017).
and Geng et al. (202T) have given tractable and differentiable lower bound on this entropy.
Furthermore, when the dimensions of the base distribution d is smaller than m. the KI. divergence is not well defined. In the KL divergance, we integrate over the whole space but q, does not have full support over this space and exists on manifold embedded in the θ space. In the GAN literature this problem is called mode dropping and is caused by the inability of the generator to recover all modes of the true data distribution (Che al. 2020 Xu et al., 2018). To alleviate this, draw inspiration from works in the GAN literature (Che et al. 2020) and add m dimensional noise to the output the generator and redefine the variational approximation in the following section.
3

---

Under review as a conference paper at ICLR 2023
3 A DEEP LATENT VARIABLE MODEL AND ITS ENTROPY
As the variational distribution, we propose to use Gaussian deep latent variable model (DLVM) of real variable I E I and with real latent variable R with density
(0) (()=))(()) dz -x (e
(8)
We assume Gaussian base density and Gaussian output density, that is
((z) = (((00, La) <00 = (V(00) I
(9)
(10)
where g IRd + R. is the decoder//genertoor and 0² E R+ the fixed homoscedastic variance of the output density. In general, we do not have a closed form for (0) due the the integral in eq. (8) and the non-linear gy, but we note that KL divergence in eq. (7) is well defined for this variational distribution. Below we propose novel approximation of the differential entropy of this model.
This model can equivalently be viewed as variational autoencoder (VAE, Kingma & Welling 2014 Rezende et al., 2014) with a Gaussian prior and a Gaussian output density with constant constant homoscedastic variance and no encoder, or a implicit distribution from eq. (6) with added Gaussian noise. The latter is clearly seen from the generative process of by describe the generative process for The generative process of 6, which is
2 =~ ~~00 Ta) 0' = =))2));
(11)
0 0=0" 0' +m, n n~ N(0, o² Lm).
(12)
3.1 DIFFERENTIAL ENTROPY
We want to calculate the different entropy of the Gaussian DLVM given by
H [[) (0)) (lo (())..
(13)
We can general not compute this analytically since we do not have closed form of p(0). Since we can sample from p(0), we can approximate the expectation in eq. (33 using Monte Carlo sampling from p(z). However, since we do not have a closed form of p()), we still need an approximation of log (0). We could approximate p(0) using Monte Carlo sampling from p(z), but this approximation has high variance. Usually, the variance is reduced by learning and encoder and doing importance sampling. Here we derive an approximation without using encoder.
3.1.1 LINEARISATION OF THE GENERATOR
First we consider local linearisation of the generator, Assuming that the Jacobian of g exists, the first order Taylor polynomial of at za is given by
T (z) = g(zo) + Dg(zo) (z zu).
(14)
where D9( is the Jacobian of g evaluated in zo. This assumes that the Jacobian exists, i.c. the generator has at least one derivative. We can approximate g(z) by T1 (z) when z close to øg. We apply this approximation to (() from eq. (8) which gives us
g (())
Ez~g(=) (g =)] (
(15)
2 Eg (N - Dz(zo) (z 20). o² Immm = NN(0))(2)) = (0), (16)
where
m(00) = g(00) - Dg(20) zo
(17)
C(za) = Dg (20) Dg 200 + o² Im.
(18)
The result in eq. (16) can be obtained analytically by integrating over the latent variable, see e.g.
Tipping & Bishop (1999).
4

---

Under review as a conference paper at ICLR 2023
3.2 APPROXIMATION OF THE DIFFERENTIAL ENTROPY
We use the Gaussian approximation of (()) to approximate the entropy of the DLVM, that is
H [q((0]] = - -E=-0(t)) [ x - -()) =) [log [0)] = H [p()). (19)
Importantly, we do the linearisation of g around the latent value z that is used to sample each 0 in the expectation. We have that
log ==
P log 2T 2
log det ((zo)
1 (0 (l= ml(zo)); (20)
2
2
=0(0,20))
which means that our approximation of the entropy is
H[g(0)| = m
log 2T
2 g
loo det C(z)] +
(21)
2
As shown in appendix A.1 the last term can be written as
(z= =) h(0, z)] Bzng(z) tr (DD(())+ +
(Dg(z)
(22)
where we used the notation M² MMT for a matrix M. Now, if we let o² tend to zero, we find that
tr ((Dotz)) (Do(=)))"(((())) (D)())))})
lim Be-q(z) 01 (0)) -2
2 E (x))
(23)
[zTz| tr((a) d
E =>))(=)
(24)
2
2 2
Similar, we can also take the limit of the determinant term from eq. (TT. that
lim = Ez E==-(=)) [log det C(z)] a² +0 2
- EZzg((>) 2
[log det (D(())
(25)
and 25 gives us the final approximation. For small values of the output variance o², we can approximate the differential entropy of DLVM as
Combining eqs. 21
H (p)) ~ lim H[[(0)] = + 8²
m log 2nt + B [log det (Dg(z) Do(=)))
1
(26)
2
2
We can get a slightly more accurate approximation, by only applying the limit from eq. 23 and not the limit from eq. (25), which gives us
H Hp)(O) d
log 2T + Bemo(s) log det (Dg(z) + o²
1 2
(27)
LINEARISED IMPLICIT VARIATIONAL. INFERENCE (LIVI)
We propose a novel bound for IVI. As the variational distribution, we use the DLVM of eq. (8) which equivalent to adding noise to the implicit distribuion of eq. (6). Using the entropy approximation from eq. (26), we propose the approximate ELBO,
[(7) = Ee-a, (0) [log p 0)] + E (0) [log p (0)] + lim H[p(e))
(28)
o2-50
Eg (0) log p (D 0) + (0)) + Bz) [log det det (Dg(z) Dg(=)T) +c,
(29)
where $ + y log 2T. We reparameterise the above with the base variables 2, 17 to get
1
Σ(>) (2~g(2)))>0) logp(Dlg(2) n) logr(g(z) + n) + det (Dg(z) tc.
(30)
5

---

Under review as a conference paper at ICLR 2023
To avoid the calculation of the log-determinant term, we can follow Geng al. (2021 eq. 10) and lower-bound as
nn
log (z( = 2 _ log (2) 2 m log (1(),,
(31)
2
where Sm (2) (=)2 2 2 (1) are the singular values of the Jacobian Dg(z). This gives us lower
bound on [()) given by
E()) =Eo-,, (0) [log PD(00) log p(0) + Bz~a(=) [mlog s1 [x)] + E())
(32)
Again, by reparameterisation with 2, 77 we get
(()) = (2xg(()))rrr)))) [log (P)g(())(()) + n7) + log p()p((()))))))))))))))))(()))( + n)) + log 81 [1()] +
(33)
We denote E(r) the LIVI bound with accurate Jacobian and E(r) the LIVI bound with a diffcrentiable lower bounded on the determinant. Depending the amount of compute available, the two bounds provide a trade-off between the accuracy of uncertainties and the resources consumed. In both cases, the entropy maximisation promotes the generator to generate diverse weight samples which is in accordance with the principle behind Bayesian model averaging and supported by the performance of deep ensembles (Lakshminarayanaa al. 2017). We present connections with existing works in the literature in the following section highlighting similarities and divergences.
5 RELATED WORK
The usage of secondary network to generate parameters of primary network first appeared in the form of hypernetworks (Ha et al., 2017). Our approach is probabilistic and hence closer to Bayesian hypernetworks (Krueger al. 2017). Compared to approach, these models require invertibility of the generator and thereby avoid the complexities of estimating the entropy term. This corresponds using normalizing flow variational approximation. Training normalizing flow over large parameter spaces is computionally costly due to large Jacobian matrices, typically requiring particular focus on the design of the variational approximation to curb dimensionality of the flow. In particular. Louizos & Welling (017) use an expressive flow on multiplicative factors of weights in each layer and not on all weights jointly, Our bound uses a very similar change in volume formulation, log det det(Dg ( (z) D=(=)), for obtaining the log probability of samples under the variational density, but does not necessitate invertibility making it more general.
Subsequently, Shi et al. 2018) Tran et al. 2017): 7); Pawlowski et al. (2017) have successfully demonstrated implicit variational inference in BNNs using hypernetworks, (Shi et al. (2018); Tran et al. (017)) do not focus on the entropy term, but rather try to estimate the ratio of the variational approximation to the prior (regularisation-term) referred to as the prior-contrastive formulation by Huszáá (017]. (Tran et al. (2017) opt for training a discriminator network to maximally distinguish two distributions given only i.i.d. samples from each. This approach, though general, adds to the computational requirements and becomes more challenging in high dimensions (Sugiyama et al. 2012). ) To mitigate the overhead of training the discriminator for each update of the ELBO, many works limit the discriminator training single or few iterations. Furthermore, this approach entails an adversarial objective that are infamously unstable (Mescheder al.. 2017). Pawlowski al. (017)) treat all the weights as independent and find that a single discriminator network is inaccurate at estimating log ratios when compared to the analytical form of Bayes by backprop (Blundell al. 2015). and opt to use a kernel method that matches the analytical form more closely. (Shi et al. (2018) propose a novel way of estimating the ratio of the two densities using kernel regression in the space of parameters which obviates the need for a minmax objective. An obvious difficulty with kernel ridge regression in practice is its inaccuracy to estimate high-dimensional density ratios which is similar to using discriminators. This is especially the case given a limited number of samples from both the densities well as the RBF kerne.. While the RBF kernel still takes the same high- high-dimensional inputs and does not involve learning massive sets of parameters, its accuracy at larger scales is still doubtful. This work also proposes matrix multiplication neural network (MMNN) a novel generator architecture for generating large set of parameters. Pradier et al. 2018) are also motivated by the possibility of compressing the posterior in
a procedure called density ratio estimation (also

---

Under review as a conference paper at ICLR 2023
Table UCI regression datasets. We report RMSE and log- legliikeis on the test and average across three different seeds for each model to quantify the variance in the results.
Test RMSE L
Test LL
Dataset
LIVI
Deep Ensembles
KIVI
LIVI
Deep Ensembles
KIVI
2.461 0.131
3.28
1.00
2.798
0.173
-2.318
0.092
-2.41
0.25
-2.527
Concrete
5.150 +: 0.169
6.03
+ 0.58
4.702 + 0.116
-3.068
:: 0.120
3.06
#018
-3.054 +: 0.043
Energy
0,580 + 0.270
2.09
+ 0.29
0.167
+ 0.015
-1.38
0.22
-1.298 0.005
Kin8nm
0.081 0.000
0.00
0.0075=000010
1.180 0.011
1.20
0.02
Naval
0.001 + 0.000
0.000
0.000
0.00
+ 0.00
5.523 4 0.170
5.63
0.05
5.501 0.121
lower dimensional space and use an inference network with a generator. Their model differs from ours as they also consider the parameters of the generator/decoder to be stochastic. Moreover they require empirical weight samples to train which doubles the training steps. D-SIVI (Molchanov et al. 2019) and SIVI (Yin Zhou, 2018) use Monte Carlo (MC) averaging to approximate the entropy term. Both works the implicit formulation to only model the mixing coefficients and not all the weights the network. Our entropy term [ also has a similar form and can be MC approximated.
In the spirit of some recent works (Izmailov al. 2020 Daxberger al. 2021b a) that alternatively choose a lower dimensional representation to preclude costly, high-dimensional inference, our work can be seen allowing the approximate posterior in the form of the generator to choose which dimensions and parts of posterior are crucial and model them accordingly.
EXPERIMENTS
6.1
TOY DATA
In fig. T we compare inference with method against the gold standard for posterior inference on a simple toy dataset. After training, we also plot KDE-plot of the samples the generator outputs in appendix A.66 We infer from this plot that the generator is capable of representing non-trivial distributions as we can spot heavy tails and multiple modes.
6.2 UCI DATASETS
We perform experiments UCI regression datasets with the setups by Lakshminarayanan al. (2017) and Shi et al. (2018) using a BNN with one hidden layer MLP with 50 units on all of these datasets. We report the RMSE and log-likelihood on held data for our method. We use generator architectures that are either equally or less powerful than Shi et al. (2018) and do not assume independence across layers, i.c. using one MLP to generate all the weights of the BNN. All of the generator architectures are one hidden layered MI.P with a slightly varying number of units depending on the dataset. At this scale is feasible to estimate uncertainties using accurate Jacobians. We require far fewer number of samples (5-10) per iteration compared 100 used by KIVI to achieve very competitive results. We suspect they use high number of samples to curb the variance of the kernel estimator.
Our results are summarised in table [ variance in the dataset is assumed to be constant and we train observation noise parameter using type II maximum likelihood.
We train our method with homoscedastic assumption i.c. the
6.3 MNIST DATASET
Next we test our method on the MNIST dataset. While using the MMNN as the generator, we were able to achieve errors on the test set on par with KIVI for MLP with 400 and 800 hidden units. With the total number of parameters generated exceeding 400K even for 400 hidden units we chose to train the model only with the differentiable lower bound due to prohibitively high memory usage. For OOD testing we compare
method to last-layer laplace, deep ensembles and simple MAP estimate. We intentionally choose these methods to compare against as a mean-field approximation usually does not achieve good accuracy in-distribution data and has been shown, repeatedly, to suffer from
7

---

Under review as a conference paper at ICLR 2023
NLL with R-MNIST
ECE with R-MNIST
7.5
UVI
0.6
LIV
UVI
DE
LAP
MAP
2.5
0.0
50
100
150
50
100
150
Figure 2: Rotated MNIST benchmark. We compare a Bayesian neural network trained via our method with MAP, deep ensembles (DE) and last-layer laplace (LAP) where our best model achieves results competitive with deep ensembles and with post-hoc method like laplace.
many optimisation difficulties. On the other hand it is possible run HMC samplers at this scale, it is not preferable. Very few works the literature report results using full batch HMC (Izmailov et al.) 202T). Deep ensembles predict using neural networks that have converged onto different minima hence encompassing information from diverse modes of the posterior, and as such remains one of the best in terms of uncertainty estimation. As for benchmarks we choose two OOD benchmarks presented in Daxberger et al. (2021a). First we test the OOD AUROC and confidence of a LeNet5 BNN trained on MNÍST by using FMNIST, KMNIST and EMNIST. We used a MMNN architecture for generating over 40K parameters and trained using the differentiable numerical approximation with 3-6 samples depending on the architecture. We expand on few generator architectures here and leave the rest for appendix appendix A.2
The BNN trained with the implicit variational approximation, generator with 1225 dimension noise input and 2 matrix multiplication layers of 350 units each achieves accuracy of 99.07 1% + 0.02, and calibration error of 0.084 + 0.011 with nll -0.021 + 0.001 on the test set. The same model reports averaged OOD AUCROC of 97.15 + 0.17 with averaged confidence 68.53 = 0.24. According to results provided in Daxberger et al. [2021a] Table 1), our model does not outperform in terms of confidence values yet, we notice that the performance degrades very smoothly as encounters OOD data as opposed to models like Deep Ensembles and Laplace both of which fail relatively immediately and drastically in terms of confidence values. Our model does perform quite well on the averaged AUROC as well as on test set calibration error and log-likelihood.
To probe out of distribution performance further, we compare our method on the rotated MINST benchmark from Daxberger et al. (2021a). In this benchmark we plot the negative log-likelihoods and empirical calibration errors for different rotated MNIST images. In this benchmark task we plot results in fig. 2e-22for for three different architectures and our best (LIVI 3) remains the same architecture as above. Here too, we nearly match the performance of deep ensembles on these two metrics. The other two architectures, LIVI 1 has 1764 dimensional noise input and one matrix multiplication hidden layer with 350 units while LIVI has 900 dimensional noise input with 2 hidden matrix multiplication layers of 320 units each.
6.4 COMPARISON BETWEEN IMPLICIT VARIATIONAL. APPROXIMATIONS
Variational inference for BNNs relies heavily on the expressivity of the family of approximations chosen model the posterior. our case the architecture the generator represents the flexibility and overall modeling capability of the implicit variational density. We trained different architectures and noticed that generator architectures with more hidden layers perform better on in-distribution metrics like accuracy and -igliikl..ood. Additional hidden layers afford the generators the capacity to warp the input Gaussian noise into a suitable posterior distribution. On the other hand, the dimensionality of input noise becomes crucial for uncertainty quantification and OOD performance. We believe this is because the number of noise inputs are all the degrees of freedom available to the generator to model the parameters of the BNN. As such, the entropy of resulting posterior is
8

---

Under review as a conference paper at ICLR 2023
Empirical CDF of entropies over FMNIST
1.0
LIVI
DE
0.8
lap
0.6
0.4
0.0
1.0
1.5
Entropy of predictions
Figure 3: Empirical CDF plot. On the x-axis is the entropy of individual predictions and empirical CDF on the y-axis. As we count over the whole dataset, we would like to encounter more high entropy predictions closer to the value 2.30 the x-axis.
directly dependent on the this factor. Although this number cannot be increased without repercussions because the base distribution and the number samples affect the signal to noise ratio of our objective function eq. (29) and very large results in large gradient variance hindering covergence, requiring more samples during training or higher number of iterations to converge. In these experiments we also noticed that down scaling only the prior log probability has very positive effect the results. This is due to the fact that the prior term regularises the generator, forcing it to find minimas close to itself, a standard normal distribution. The scale of this prior log probability term is significant in the ELBO and gradients of this term are detrimental to the overall optimisation process. Unlike cold-posteriors/ Wenzel et 2020), we keep the gradients of the entropy term as is and only reduce regularisation by downscaling the prior.
As the last benchmark opt to ascertain the quality of our model's posterior and the implied predictive uncertainties by plotting the empirical CDF of predictive entropies across OOD im- Desssassmminaaaanann et 2017 Louizos Welling 2017) in fig. 3 Given a model trained on MNIST, the predictions on data point from different distribution should be given high entropy prediction like a uniform distribution. For this plot we first obtaín entropies of the output softmax distributions for all the models across data points and use an empirical CDF to represent how many of these predictive entropies are closer to a uniform distribution which has an entropy of 2.3. Ideally, we are looking lines closer to the right bottom corner, i.e. the number of low-entropy or highly confident predictions should be less. We compare our model to MAP, deep ensembles and last-layer Laplace and find that our model trained on MNIST is quite competitive in the quality of uncertainty estimates for this test over FMNIST dataset. For this plot we use the best generator architecture with a LeNet5 BNN which is called LIVI 3 in the tests above.
7 CONCLUSION
In this paper we present a novel method for implicit variational inference for Bayesian Neural Networks that circumvents the need for a discriminator network to estimate intractable density ratios. We find that modelling the posterior with a highly flexible approximation indeed does have benefits. Our methods, in the wide range of variational approximations, get closer to the performance of deep ensembles, a non-probabi method on in distribution and out distributions performance. Unlike conventional probabilistic methods we do not. One possible limitation of such hypernetworks can be generating massive parameter vectors for large neural networks. Works like Pawlowski et al. (2017): Shi et al. (2018) use different generator architectures to generate weights for each hidden layer in turn loose the information from modelling correlations across layers. Similarly this approach can be extended to multiple smaller generators the sacrifice of modelling correlations across layers.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Christopher Bishop, Neil Lawrence, Tommi Jaakkola, and Michael Jordan. Approximat- ing posterior distributions in belief networks using mixtures. In M. Jordan, M. Kearns, and S. Solla (eds.), Advances in Neural Information Processing Systems, volume 10. MIT Press, 1997. URL http::://ppocceeecngg..... wttppr/ppp..ccpppppe/////////
0O066110000Ea0111544477000000--2 pdf
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In Francis Bach and David Blei (eds.), Proceedings of the 32nd International Conference Machine Learning. volume 37 of Proceedings of Machine Learning Research, pp. 1613-1622, Lille, France, 07-09 Jul 2015. PMLR. URL https: hattps::/proceeeete http:///ccccec....////// mlr
Tong Che, Ruixiang ZHANG, Jascha Sohl-Dickstein, Hugo Larochelle, Liam Paull, Yuan Cao, and Yoshua Bengio. Your gan is secretly an energy-based model and you should use discriminator driven latent sampling. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12275-122877 Curran Associates, Inc., 2020.
Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux effortless bayesian deep learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021a. httpt:t////::::teeet
net wtt.:eewweeeeeeeueeeeeeeeeeeee htte///oommmnddooaaaaaa 4MMhn
Erik Daxberger, Eric Nalisnick, James U Allingham, Javier Antoran, and Jose Miguel Hernandez- Lobato. Bayesian deep learning via subnetwork inference. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2510-2521. PMLR, 18 24 Jul 2021b. URL www.ingssss.....m mlr www.mm.p.prsssvvee httpa/eeeeeeee//3///eeeeee/ee/// http///wwbbeeaaaaaaaeaa html
http:::/pprceedinngee.
Lue Devroye. Random variate generation in one line of code. In Proceedings Winter Simulation Conference, pp. 265-272, 1996. doi: 10. 10.1145/25662..26622)
Peter J.
Diggle and Richard J. Gratton. Monte carlo methods of inference for implicit statistical models. Journal of the Royal Statistical Sociery. Series B (Methodological), 46(2):193-22) 1984. ISSN 00359246.
www. jstor. org/ www.oooorgg//taallleoo http://tabeeee222ee
Andrew Foong, David Burt, Yingzhen Li, and Richard Turner. On the expressiveness of approx- imate inference in bayesian neural networks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 15897-15908. Curran Associates, Inc., 2020. URL.https:///rcceeeiii. neuriss.. cc/
Dape:/2020/Eilee/bddadae ort.aaacce...aao0001100.00006aae
Andrew YK Foong, Yingzhen Li, José Miguel Hernández-Lobato and Richard E Turner. 'in- between' uncertainty in bayesian neural networks, arXiv preprint arXiv:1906. 11537, 2019.
Vincent Fortuin. Priors in bayesian deep learning: A review. International Statistical Review, 1/insn. 12502. URL htt.p::////ln http:///nnineeebbbrrr .wlley
2022. doi: http:://ooiorr//00 com/do11 /b/////////// 10. 1111/ nsr 12502
Cong Geng, Jia Wang, Zhiyong Gao, Jes Frellsen, and Sø Hauberg. Bounds all around: training energy-based models with bidirectional bounds. In M. Ranzato, A. Beygelzimer. Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 19808-19821 Curran Asso- ciates, Inc 2021. URL htttt:://pooceeeeteee http:///ww...eeeeeo.c.cc/ppp///a hctpppppe////22///eeeee
neurips
David Ha, Andrew M. Dai, and Quoc Le. Hypemetworks. In International Conference Learning Representations, 2017. URL htt /tttpp:://penreeiiiiii hwtp://wwwwwwwwweeeeeeeeeeeee www./eee/ooooo www.//ouumumm... ? www.d-kkpAcell....
Ferenc Huszár. Variational inference using implicit distributions, 2017. URL
org (rgg/bbs/170 08235
10

---

Under review as a conference paper at ICLR 2023
Pavel Izmailov, Wesley J Maddox, Polina Kirichenko, Timur Garipov, Dmitry Vetrov, and An- drew Gordon Wilson. Subspace inference for bayesian deep learning. In Uncertainty in Artificial Intelligence, pp. 1169-1179. PMLR, 2020.
Pavel Izmailov, Sharad Vikram, Matthew Hoffman, and Andrew Gordon Gordon Wilson
What are bayesian neural network posteriors really like? In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pp. 4629-4640. PMLR, 18-24 Jul 2021. URL https:://rroeee nageeeddiigg..ee ngs www.nng........ m press/pres///ee 39/ zma lov21a la html
Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. In International Conference on Learning Representations, 2014.
David Krueger, Chin-Wei Huang, Riashat Islam, Ryan Tumner, Alexandre Lacoste, and Aaron 04759
Courville. Bayesian hypernetworks, 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NIPS, 2017.
Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix gaussian posteriors. In Maria Florina Balcan and Kilian Q. Weinberger (eds.). Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learming Research, pp. 1708 77161776, New York, New York, USA, 20 20-22 Jun 2016. PMLR. URL
mlr p.rssppressseeeee /tasss//4//// 11/4///oouzzsse htmll
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational bayesian neural networks. In International Conference on Machine Learning, pp. 2012-221112222 2227. PMLR, 2017.
David J.C. Mackay. Bayesian methods for adaptive models. PhD thesis, California Institute of Technology, 1991.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Gordon H. Bower (ed.), Psychology of Learning and Motivation, volume 24, pp. 109-165. Academic Press, 1989. doi: https://oi..or//1)
1016//80099-742210886666666
URL https: //wp:///wwwwwww/ htw.s////wwiieeeerrrrrrcooo SEEiccee/s1/S0079744108888
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. The numerics of gans. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. arXiv preprint 1XXi::661000388 2016.
Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih. Monte carlo gradient estimation in machine learning. JMLR, 21(132):1-62 2020.
Dmitry Molchanov, Valery Kharitonov, Artem Sobolev, and Dmitry Vetrov. Doubly semi-implicit variational inference. In Proceedings of the Twenty-Secome International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research. PMLR, 2019. URL 22255111222222222225677777883333
mlr prwss/ v8 htt.///hnllll....mmmmmmm
Radford M. Neal. Bayesian Learning for Neural Networks. University of
PhD
thesis.
Toronto, 1995.
URL https: www.sseeatiiioe icscholar
Kazuki Osawa, Siddharth Swaroop. Mohammad Emtiyaz Khan, Anirudh Jain, Runa Eschen- hagen, Richard E Turner, and Rio Yokota. Practical deep learning with bayesian princi- ples. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL / newrips.er.ppp.pp... http://ccsass.eeeeeeeoo.o//////a 0

---

Under review as a conference paper at ICLR 2023
Nick Pawlowski, Andrew Brock, Matthew CH Lee, Martin Rajchl, and Ben Glocker. Implicit weight uncertainty in neural networks. arXiv preprint arXiv:1111.01297, 2017.
K. B. Petersen and M. S. Pedersen. The matrix cookbook. Technical report, Technical University of Denmark, 11 2012. URL hhtt:://wwwtt compute. httppppt........kk/ppp////////// html Version 20121115.
Carsten Peterson. A mean field theory learning algorithm for neural networks. Complex systems, 1: 995-1019, 1987.
Melanie F. Pradier, Weiwei Pan, Jiayu Yao, Soumya Ghosh, and Finale Doshi-velez. Projected bnns: Avoiding weight-space pathologies by learning latent representations of neural network weights, 2018. URL/ttpp:://axxe
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, 2014.
Lawrence Saul and Michael Jordan. Exploiting tractable substructures in intractable networks. In D. Touretzky, M.C. Mozer, and M. Hasselmo (eds.), in Neural Information Processing Systems, volume 8. MIT Press, 1995. URL tts::tttttttpppreeeaiiai 1995/ fi eee11eeee/ee/1 f 8 9b8 22w.c22bbbb6622 100555800000 htt.a//e2.5511111111150181180a2.
wwwrips.......... CC/ cc/paper/ paper/ pddf
Lawrence K. Saul, Tommi Jaakkola, and Michael Jordan. Mean field theory for sigmoid belief networks. J. Artif. Int. Res., 4(1):61-76, 1996. ISSN 1076-9757.
Jiaxin Shi, Shengyang Sun, and Jun Zhu. Kernel implicit variational inference. In International Conference on Learning Representations, 2018. URL [Ru[htttss://peeeviiee wwtpeewwwwwwwweweeeeeeeeeeeeeeee /orrmm
10=r114e@W02
Yang Song and Diederik P Kingma. How to train your mergy-based models. arXiv preprint arXiv:2101.03228, 2021.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density Ratio Estimation in Machine Learning. Cambridge University Press, 2012. doi: 10.1017/CB09781 139C309977811933111
Toshiyuki Tanaka.
theory of mean field approximation. In M. Kearns, S. Solla, and D. Cohn (eds.), Advances in Neural Information Processing Systems, volume 11. MIT Press, 1998. URL htttps://proceeeinggg wwweiieerrppp... wttpppeee/9/////11. .df]
Michael E Tipping and Christopher M Bishop. Probabilistic principal component analysis. Journal of the Royal Statistical Sociery: Series B (Statistical Methodology), 61(3):611 (1-3))61--2221 1-622, 1999.
Dustin Tran, Rajesh Ranganath, and David Blei. Hierarchical implicit models and likelihood-free variational inference. Advances in Neural Information Processing Systems, 30, 2017.
Florian Wenzel, Kevin Roth, Bastiaan S. Veeling, Jakub, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the bayes posterior in deep neural networks really? Proceedings of the 37th International Conference on Machine Learning, ICML'20, 20. JMLR.org, 2020.
Qiantong Xu, Gao Huang, Yang Yuan, Chuan Guo, Yu Sun, Felix Wu, and Kilian Weinberger. empirical study on evaluation metrics of generative adversarial networks. arXiv preprint arXiv: /00.://80007755, 2018.
Mingzhang Yin and Mingyuan Zhou. Semi-implicit variational inference. In Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5660-5669, PMLR, 2018. URL http:hts///ppcccedinngg.. ml www.ppressveee... v00/ yin1 8b. html
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.1 DETAILS ON APPROXIMATION OF THE DIFFERENTIAL ENTROPY
In this section we derive eq. (2 To simplify the derivation, we will use the notation v² uTv for vectors and M = MTM for matrices.
Starting from the left hand side of eq. we have that
2 - ((0- (()= u(x))
E (000 x)]
E Bzma(=) E
(34)
Ex o(xx) ( ((0 (())))]
(35)
2
1
(t ( ((2)) -0(())))
(36)
2
The inner expectation simplifies to
E z)[(0 [x(2))²1 = E (0 - g(2) + Dg(z)z)) 2
(37)
= E (0)))
(0 g(2))² + (Dg(z) 2(0 -(2)) Dg(=)2
(38)
I' + (Dg(z) 2)²,
(39)
where that (0=))(0) o(),)) 1(0 [(2))و 0 and Eo g(z)) = o² I.. If
we plug in the result of eq. (39) into eq. (36). we obtain
(02lm. ur ((Dgle) + +z ) (
Ex Bo 2)] = E Exx)())
+ (Dg(z) ))
(40)
Note that eq. (40) could also be derived from eq. (34) using Petersen & Pedersen (2012 eq. 380) and some reordering the terms. Equations (37) to (39) also follows from Petersen & Pedersen (2012 eq. 325).
A.2 EXPERIMENT DETAILS
We use the MMNN architecture as presented in Shi al. (018 for generating weights of the MLP BNN that was trained on MNIST as well as LeNet BNN that was used for all the OOD benchmarks. For the MIP experiment to compare with KIVI we used one MM network that generated all the parameters of the network Following architectures were tried for LeNet5 generators:
Noise input 25x25, 2 MM hidden layers with 250 units, output layer size 350x 127.
Noise input 30x30, 2 MM hidden layers with 350 units, output layer same as above.
Noise input-35x33 2 MM hidden layers with 350 units, output layer same above.
Noise input 35x35, 3 MM hidden layers with 350 units, output layer same above.
Noise input 38x38,, 2 MM hidden layers with 350 units, output layer same as above.
Noise input 42x42, 2 MM hidden layers with 325 units, output layer same as above.
All the above architectures were trained without dataset augmentation and with a maximum of samples per minibatch. The last architecture required higher number of samples due to gradient noise which is proportional the dimensionality of input noise. This phenomenon has been widely observed in training high dimensional variational approximations (Osawa et al. 2019 Mohamed al. 2020). As all the architectures are trained for 100 epochs with the same learning rate, increasing gradient noise can significantly deter convergence when the input noise dimensions are increased.
A.3 PRIOR DOWNWEIGHTING
We choose to down scale the log prior probability in all the benchmark experiments. This term appears in the ELBO objective function and serves important purpose. When the prior is chosen by
13

---

Under review as a conference paper at ICLR 2023
Weight Samples
Figure 4: Zero-Centeree weights
Figure 5: Weights activated after down-weighting the prior
domain experts this term ensures that the approximate posterior inferred is not too different from the intelligently chosen prior and hence the log prior probability of samples coming from the variational approximation should be high when the ELBO being maximised. However, the choice of an appropriate prior is an active area of research in Bayesian deep (earning(Fortunn 2022) and this prior term and the regularisation effect is known to limit the variational approximation. Dll [g))) should be minimised prior is naively chosen to be a standard normal it forces most of the weights of the posterior to be zero-centered. This forces the model look for minimas that are very close 0 and has detrimental effect on the in-distribution performance. We use the plotting tool used by Shi et al.] (2018) to demonstrate this effect. The line-plot below has all of the weights of a neural network used solve a toy regression task on the x-axis and their respective magnitudes on the y-axis. We chose to sort the weights in order of their magnitude as the positions of weights are not very informative neural networks due to permutation invariance. In the first plot, most of the weights are zero-centred and are not very active, on the other hand the second plot shows what happens when we down weight the prior by just 0.I.
result of the optimisation of ELBO using gradient ascent and when the
A.4 DETAILS OF FIG.
In fig. I we compare both the objective functions presented in this work for training with implicit different methods for uncertainty quantification for neural networks. All models were trained for 10K iterations and had to learn observation noise present in the toy sinusoidal dataset. We deliberately removed a part the data to see the models tested were able to find in-between uncertainties. All methods were given the same sized hidden layered MLP with 7 and 10 units respectively. We trained 5 networks with different seeds for Deep Ensembles and average their predictions to make the plot. The variance of the predictions were then used for the confidence bands in blue. We also train the model with an observation noise parameter. For MFVI, we used KL down weighting to get it convergence and increase the weight in the end of training. For HMC
variational approximations
14

---

Under review as a conference paper at ICLR 2023
Kernel Density Estimate(single weight)
18
Figure 6: Multimodal densities over weights using KDE
sample 5000 samples using the library Pyro. We also tried to make multiplicative normalizing flows converge for this dataset, but with even 20K parameters and training for 15K iteration with a very long learning rate did help. We even tried KL down weighting reduce the effect of the prior in the initial iterations but that did not work either.
A.5 COMPUTATION GRAPH
Here we provide some details about how the combination of the joint generator-BNe BNNN model works. The Bayesian neural network classes for all types of architectures(feed forward, convolutions, etc.) require a generator in the ini ti function. As such, the generator networks reside inside the BNN and reparametrise it with simple this kind of implementation was the layers themselves. Py Torch provides different kinds of mutable layer implementations in r. module but these layers do expose their state i.e. their parameters in manner that allows changes on the fly during training. We reimplemented the layers allowing such resampling oceur with the generator. In the init function of the BNN, we generate one set of parameters with the generator, package it in ci ct that has the weight sample as well as a index know the number of weights used by a previous layer. This counter index is updated by each layer in their in t and sample 1 e e_pa rametere re function. As such, only the parameters of the generator are trainable, the parameters of the BNN are switchable and relay gradients to the generator via the likelihood or the entropy term.
function. The most important part of
A.6
KDE PLOOT
Figure 6 shows a KDE plot of weights randomly chosen from samples obtained from trained generator.
15


Reviews:
[{"review_id": "i9cZvpSLhAk", "reviewer": "Reviewer_9D7T", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Exact posteriors over parameters in BNNs are usually more complex than the prescribed-density approximations that are usually imposed in VI. The usage of implicit variational distributions allows for an increase in the flexibility of these methods. In this paper, the authors propose a novel bound for training BNNs in such schemes. This is achieved through the combination of a Gaussian deep-latent variable model and a Laplace approximation of the generator of the implicit distribution samples, which simplifies the estimation of the KL regularization term in the original ELBO objective function.", "summary_of_the_review": "The method seems interesting, and although strongly based on previous ideas, could prove to be an important contribution to the research community. The authors should consider comparing against some other state-of-the-art approaches and provide stronger experimental evidence for the properties of the proposed method. However, the proposal is promising and could be of use to other researchers. ", "strength_and_weaknesses": "# Strengths: \n\n* The constructed model is reasonable and simplifies terms of the objective function that are usually very hard to deal with. \n\n* The final objective function obtained seems an interesting step for implicit-distribution-based methods.\n\n* The paper is clearly written for the most part and can be followed easily.\n\n# Weaknesses:\n\n* The proposed system does not seem too different from previous proposals. As the authors mention, this is highly related to Bayesian hypernetworks and normalizing flows, and somewhat could be seen as a particular combination of both concepts. \n\n* Since the motivation behind the contribution is related to providing better uncertainty estimates for BNNs, I think the authors should provide a stronger experimental phase on which this is shown more extensively (e.g. adding comparisons against HMC in toy datasets and comparing with other methods that have shown high performance in this regard, s.a. [2]).\n\n* In several points of the article where previous literature on the topic is covered, I cannot help but notice that some important contributions are missing. For instance, [1] should be clearly mentioned here since it is highly related to the topic, and this applies both to the initial setup on page 2 as well as to the Related Work section. Moreover, both [1] and [2] could (and maybe should) be considered as benchmarks to compare against. Moreover, there has been an extensive ongoing research on the implicit approach applied to the function-space formulation of BNNs which is never mentioned. These methods have shown improved performance and several relevant properties that the regular weight-space formulation fails to reproduce. Some relevant examples here are [3,4,5,6], among others. In particular, these methods extend the formulation of Eq.(6) to implicit stochastic processes.\n\n* Scalability studies and a detailed comparison with other methods is not included anywhere. I strongly suggest the authors to provide some insights here.  \n\n## References:\n\n[1] Mescheder, Lars, Sebastian Nowozin, and Andreas Geiger. \"Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks.\" International Conference on Machine Learning. PMLR, 2017.\n\n[2] Santana, S. R., & Hernández-Lobato, D. (2022). Adversarial α-divergence minimization for Bayesian approximate inference. Neurocomputing, 471, 260-274.\n\n[3] Ma, C., Li, Y., and Hernández-Lobato, J. M. (2019). “Variational implicit processes”. In: International Conference on Machine Learning, pp. 4222–4233.\n\n[4] Sun, S., Zhang, G., Shi, J., and Grosse, R. (2019). “Functional variational Bayesian neural networks”. In: International Conference on Learning Representations.\n \n[5] Ma, C., & Hernández-Lobato, J. M. (2021). Functional variational inference based on stochastic process generators. Advances in Neural Information Processing Systems, 34, 21795-21807.\n\n[6] Rodrı́guez-Santana, S., Zaldivar, B., & Hernandez-Lobato, D. (2022, June). Function-space Inference with Sparse Implicit Processes. In International Conference on Machine Learning (pp. 18723-18740). PMLR.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Mq_tMCkEVb", "reviewer": "Reviewer_mGTw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper posits a new variational inference method based on implicit variational distributions. It develops a bound for estimating the entropy of the implicit distribution involved in the ELBO based on local linearization. The authors then use a differentiable numerical lower bound on the Jacobians of the generator to mitigate computational concerns. Experiments are conducted on UCI regression and MNIST classification. ", "summary_of_the_review": "Given the issues of approximations and limitations, I vote for rejecting this paper.", "strength_and_weaknesses": "# Strength\n- The paper develops a new differentiation bound on the entropy of implicit distributions. This may be useful for the probabilistic inference community. \n- The literature review is relatively thorough.\n\n# Weaknesses\n## Major issues\n- Regarding the ill-defined KL when d < m, the authors should discuss the quasi-KL measure [1] to make the paper more convincing.\n- My major concern about this paper is that there are too many approximations such that I am not convinced of the fidelity and reliability of the yielded bound. One approximation is about local linearization around the sample z. The approx. error here can be arbitrarily large? The second approximation lies in eq 23. You really use a $\\simga^2$ that approaches 0? But if doing so, you fall back to the ill-defined KL... The third approximation is eq 31, where Jacobians are replaced with their singular values. The approximation error here can be bounded but currently, the overall approximation error is unmeasurable. Thus, I question the reliability of this method and believe more theoretical analysis regarding the tightness of you bound of entropy is required.\n- The main technical novelty lies in the local linearization of the generator, which in my opinion, is limited. As said, more discussion or analyses on local linearization are needed.\n- The biggest limitation of this method is its poor scalability. It is two-fold. (1) The generator cannot trivially generate millions of parameters for modern NNs as it cannot have that wide output layer. (2) The singular values of Jacobian are expensive to estimate; even the Jacobians themselves cannot be easily estimated for modern NNs. As a result, the method cannot be applied to realistic datasets and models. Results on at least cifar-10 are appreciated.\n- Why the closely related KIVI is not included in the MNIST exps?\n\n## Minor issues\n- The writing is not good enough and there are typos. An example is the first paragraph of sec 3.1.\n- By inspecting figure1, I don't think LIVI is as good as HMC, DE, and even MNF. Though the authors highlight LIVI captures in-between uncertainty, but it seems that it is not good enough and at least worse than that of even MNF. Can the authors provide a quantitative estimation of the quality of the predictive distributions of these methods using something like the divergence from the ground-truth predictive distribution (provided by HMC in my opinion)? By the way, why isn't the closely related KIVI included here?\n\n[1] Variational Bayesian dropout: pitfalls and fixes", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JYAf4nRW8S2", "reviewer": "Reviewer_xXZT", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a variational inference method based on sampling model parameters from a generator neural network. It calculates the entropy of resulting implicit variational distribution by linearizing the network, resulting in a Gaussian entropy. Further, the paper proposes to bound the costly log determinant calculation of the variance by the maximum eigenvalue (via the maximum singular value of the Jacobian). It reports competitive performance on UCI regresson and MNIST uncertainty estimation benchmarks with deep ensembles, a kernel-based implicit VI method and a last-layer Laplace approximation.", "summary_of_the_review": "While in principle I like the approach that the paper takes -- I find it much more coherent and principled than prior works on implicit VI -- the evaluation is just not sufficient to recommend acceptance. If the authors want to take a primarily quantitative route, I'm afraid that some larger scale experiments, at least some ResNets on CIFAR, would be necessary to be convincing for the community. MNIST is simply not a meaningful quantitative benchmark anymore.\n\nPersonally, I would find some more qualitiative analyses on the approximations that the paper makes insightful. It's great that there are e.g. some histograms of samples that show the generator is multi-modal, but it is not too surprising that a neural network is capable of doing that. The main question is whether the proposed approach approximates the entropy decently well and what happens when the prior term is not down-weighted, so that we can attribute resulting performance gain to the method being an approximately Bayesian one rather than an arbitrary stochastic neural network.", "strength_and_weaknesses": "Strengths:\n* Generating parameters by a neural network is an appealing direction for variational as it allows for potentially complex, multi-modal variational distributions. This paper proposes a coherent and principled approach for this.\n* The technical exposition is extremely clear.\n\nWeaknesses:\n* The experimental evaluation is extremely small scale, not going beyond MNIST, which is not exactly informative.\n* There are unfortunately no ablation studies or relevant qualitative experiments. Given that the paper makes multiple approximations, I really would have wanted to see some more in-depth analysis of these choices rather than fairly tangential experiments on the architecture of the generator (I know this matter, but does not seem exactly relevant for the main text).\n* There is a lengthy discussion around down-weighing the prior term in the ELBO. I know it is common practice to temper the KL divergence in the variational objective and have used such tricks myself, however I find this somewhat contradicts the motivation of the paper to approximate the entropy term in the objective. Why bother approximating the entropy if we're not using the ELBO in a principled way anyway? \n\nMinor comments:\n* I don't understand why the variance on the parameters is taken to the limit of 0 rather than being treated as a variational parameter.\n* I'm not entirely sure whether this equivalence holds, but taking the outer product of the jacobians that the paper uses as equal to the Fisher, I wonder whether instead of using an iterative approach for calculating the singular value the structure of the Fisher could be exploited as e.g. in (Ritter et al., A scalable Laplace approximation for neural networks, ICLR 2018) for efficiently calculating eigenvalues/log determinant.\n* I would suggest adding HMC results to the UCI benchmark for reference.\n* FIg 3 should include reference values for the in-distribution test data, OOD entropies are meaningless in isolation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject