SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mu_ {\gamma} \left(\boldsymbol {z} ^ {\prime}\right) = g _ {\gamma} \left(\boldsymbol {z} ^ {\prime}\right) - \boldsymbol {J} _ {g} \left(\boldsymbol {z} ^ {\prime}\right) \boldsymbol {z} ^ {\prime}, \quad C \left(\boldsymbol {z} ^ {\prime}\right) = \boldsymbol {J} _ {g} \left(\boldsymbol {z} ^ {\prime}\right) \boldsymbol {J} _ {g} \left(\boldsymbol {z} ^ {\prime}\right) ^ {\intercal} + \sigma^ {2} \boldsymbol {I} _ {m}. \tag {A.3}
$$
Our approximation of the entropy is thus
$$
\tilde {H} [ \tilde {q} (\boldsymbol {\theta}) ] = \frac {m}{2} \log 2 \pi + \frac {1}{2} \mathbb {E} _ {\boldsymbol {z} \sim q (\boldsymbol {z})} [ \log \det  \boldsymbol {C} (\boldsymbol {z}) ] + \mathbb {E} _ {\boldsymbol {z} \sim q (\boldsymbol {z})} \mathbb {E} _ {\boldsymbol {\theta} \sim q _ {\gamma} (\boldsymbol {\theta} | \boldsymbol {z})} [ h (\boldsymbol {\theta}, \boldsymbol {z}) ]. \tag {A.4}
$$
We rewrite the last term of Eq. (A.4) by using Eq. (380) from Petersen et al. (2012) to solve the inner expectation obtaining
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {z} \sim q (\boldsymbol {z})} \mathbb {E} _ {\boldsymbol {\theta} \sim q _ {\gamma} (\boldsymbol {\theta} \mid \boldsymbol {z})} [ h (\boldsymbol {\theta}, \boldsymbol {z}) ] = \mathbb {E} _ {\boldsymbol {z} \sim q (\boldsymbol {z})} \mathbb {E} _ {\boldsymbol {\theta} \sim q _ {\gamma} (\boldsymbol {\theta} \mid \boldsymbol {z})} \left[ \frac {1}{2} \left(\boldsymbol {\theta} - \boldsymbol {\mu} (\boldsymbol {z})\right) ^ {\top} \boldsymbol {C} (\boldsymbol {z}) ^ {- 1} \left(\boldsymbol {\theta} - \boldsymbol {\mu} (\boldsymbol {z})\right) \right] (A.5) \\ = \frac {1}{2} \mathbb {E} _ {\boldsymbol {z} \sim q (\boldsymbol {z})} \left[ \left(g _ {\gamma} (\boldsymbol {z}) - \boldsymbol {\mu} (\boldsymbol {z})\right) ^ {\intercal} \boldsymbol {C} (\boldsymbol {z}) ^ {- 1} \left(g _ {\gamma} (\boldsymbol {z}) - \boldsymbol {\mu} (\boldsymbol {z})\right) + \operatorname {t r} \left(\boldsymbol {C} (\boldsymbol {z}) ^ {- 1} \sigma^ {2} \boldsymbol {I} _ {m}\right) \right] (A.6) \\ = \frac {1}{2} \mathbb {E} _ {\boldsymbol {z} \sim q (\boldsymbol {z})} \left[ \left(\boldsymbol {J} _ {g} (\boldsymbol {z}) \boldsymbol {z}\right) ^ {\top} \boldsymbol {C} (\boldsymbol {z}) ^ {- 1} \left(\boldsymbol {J} _ {g} (\boldsymbol {z}) \boldsymbol {z}\right) + \operatorname {t r} \left(\boldsymbol {C} (\boldsymbol {z}) ^ {- 1} \sigma^ {2} \boldsymbol {I} _ {m}\right) \right], (A.7) \\ \end{array}
$$
where, in Eq. (A.7), we substituted  $\pmb{\mu}(\pmb{z})$  by  $g_{\gamma}(\pmb{z}) - J_g(\pmb{z})\pmb{z}$ .
Now, we rewrite the two terms inside the expectation of Eq. (A.7) and apply the limit  $\sigma \to 0$ , following our small noise assumption in Section 3. For the first term inside the expectation, we get
$$
\begin{array}{l} \left(\boldsymbol {J} _ {g} (\boldsymbol {z}) \boldsymbol {z}\right) ^ {\intercal} \boldsymbol {C} (\boldsymbol {z}) ^ {- 1} \left(\boldsymbol {J} _ {g} (\boldsymbol {z}) \boldsymbol {z}\right) = \left(\boldsymbol {J} _ {g} (\boldsymbol {z}) \boldsymbol {z}\right) ^ {\intercal} \left(\boldsymbol {J} _ {g} (\boldsymbol {z}) \boldsymbol {J} _ {g} ^ {\intercal} (\boldsymbol {z}) + \sigma^ {2} \boldsymbol {I} _ {m}\right) ^ {- 1} \left(\boldsymbol {J} _ {g} (\boldsymbol {z}) \boldsymbol {z}\right) (A.8) \\ = \left(\boldsymbol {J} _ {g} (\boldsymbol {z}) \boldsymbol {z}\right) ^ {\intercal} \sigma^ {- 2} \left(\boldsymbol {J} _ {g} (\boldsymbol {z}) \boldsymbol {J} _ {g} ^ {\intercal} (\boldsymbol {z}) \sigma^ {- 2} + \boldsymbol {I} _ {m}\right) ^ {- 1} \boldsymbol {J} _ {g} (\boldsymbol {z}) \boldsymbol {z} (A.9) \\ = \sigma^ {2} \boldsymbol {z} ^ {\intercal} \left(\sigma^ {- 2} \boldsymbol {I} _ {d} \boldsymbol {J} _ {g} ^ {\intercal} (\boldsymbol {z}) \left(\boldsymbol {J} _ {g} (\boldsymbol {z}) \sigma^ {- 2} \boldsymbol {I} _ {d} \boldsymbol {J} _ {g} ^ {\intercal} (\boldsymbol {z}) + \boldsymbol {I} _ {m}\right) ^ {- 1} \boldsymbol {J} _ {g} (\boldsymbol {z}) \sigma^ {- 2} \boldsymbol {I} _ {d}\right) \boldsymbol {z} (A.10) \\ = \sigma^ {2} \boldsymbol {z} ^ {\intercal} \left(\sigma^ {- 2} \boldsymbol {I} _ {d} - \left(\sigma^ {2} \boldsymbol {I} _ {d} + \boldsymbol {J} _ {g} ^ {\intercal} (\boldsymbol {z}) \boldsymbol {J} _ {g} (\boldsymbol {z})\right) ^ {- 1}\right) \boldsymbol {z} (A.11) \\ = \boldsymbol {z} ^ {\intercal} \left(\boldsymbol {I} _ {d} - \sigma^ {2} \left(\sigma^ {2} \boldsymbol {I} _ {d} + \boldsymbol {J} _ {g} ^ {\intercal} (\boldsymbol {z}) \boldsymbol {J} _ {g} (\boldsymbol {z})\right) ^ {- 1}\right) \boldsymbol {z}, (A.12) \\ \end{array}
$$
where we obtained Eq. (A.11) using Eq. (159) by Petersen et al. (2012). Now, taking the limit of Eq. (A.12) as  $\sigma \to 0$ , we obtain
$$
\lim  _ {\sigma \rightarrow 0} \boldsymbol {z} ^ {\intercal} \left(\boldsymbol {I} _ {d} - \sigma^ {2} \left(\sigma^ {2} \boldsymbol {I} _ {d} + \boldsymbol {J} _ {g} ^ {\intercal} (\boldsymbol {z}) \boldsymbol {J} _ {g} (\boldsymbol {z})\right) ^ {- 1}\right) \boldsymbol {z} = \boldsymbol {z} ^ {\intercal} \boldsymbol {z}. \tag {A.13}
$$
For the second term in Eq. (A.7), we use some basic identities for matrix traces and inverse. Namely, a constant term inside a trace can be moved outside, and the trace of a matrix is the sum of its eigenvalues (Petersen et al., 2012, Eq. 12). Furthermore, eigenvalues are reciprocated when the corresponding matrix is inverted (Petersen et al., 2012, Eq. 287). We have that
$$
\begin{array}{l} \operatorname {t r} \left(\boldsymbol {C} (\boldsymbol {z}) ^ {- 1} \sigma^ {2} \boldsymbol {I} _ {m}\right) = \sigma^ {2} \operatorname {t r} \left(\left(\boldsymbol {J} _ {g} (\boldsymbol {z}) \boldsymbol {J} _ {g} ^ {\top} (\boldsymbol {z}) + \sigma^ {2} \boldsymbol {I} _ {m}\right) ^ {- 1}\right) (A.14) \\ = \sigma^ {2} \left[ \sum_ {i = 1} ^ {d} \frac {1}{s _ {i} ^ {2} + \sigma^ {2}} + \sum_ {d + 1} ^ {m} \frac {1}{\sigma^ {2}} \right] = \sum_ {i = 1} ^ {d} \frac {\sigma^ {2}}{s _ {i} ^ {2} + \sigma^ {2}} + \sum_ {d + 1} ^ {m} \frac {\sigma^ {2}}{\sigma^ {2}} (A.15) \\ = (m - d) + \sum_ {i = 1} ^ {d} \frac {\sigma^ {2}}{s _ {i} ^ {2} + \sigma^ {2}}. (A.16) \\ \end{array}
$$
In Eq. (A.15), we use the squared singular values of  $J_{g}(z)$  as per singular value decomposition (Petersen et al., 2012, Eq. 292). As before, we consider the limit of Eq. (A.16) as  $\sigma \to 0$  and obtain
$$
\lim  _ {\sigma \rightarrow 0} \operatorname {t r} \left(\boldsymbol {C} (\boldsymbol {z}) ^ {- 1} \sigma^ {2} \boldsymbol {I} _ {m}\right) = m - d. \tag {A.17}
$$
For small values of the output variance  $\sigma^2$ , we approximate the last term of  $\tilde{H}[\tilde{q}(\pmb{\theta})]$  in Eq. (A.4) by its limit as  $\sigma \to 0$ . Using the limits from Eqs. (A.13) and (A.17), we have
$$
\begin{array}{l} \mathbb {E} _ {\boldsymbol {z} \sim q (\boldsymbol {z})} \mathbb {E} _ {\boldsymbol {\theta} \sim q _ {\gamma} (\boldsymbol {\theta} \mid \boldsymbol {z})} [ h (\boldsymbol {\theta}, \boldsymbol {z}) ] \approx \lim  _ {\sigma \rightarrow 0} \mathbb {E} _ {\boldsymbol {z} \sim q (\boldsymbol {z})} \mathbb {E} _ {\boldsymbol {\theta} \sim q _ {\gamma} (\boldsymbol {\theta} \mid \boldsymbol {z})} [ h (\boldsymbol {\theta}, \boldsymbol {z}) ] (A.18) \\ = \frac {1}{2} \mathbb {E} _ {\boldsymbol {z} \sim q (\boldsymbol {z})} [ \boldsymbol {z} ^ {\intercal} \boldsymbol {z} + m - d ] (A.19) \\ = \frac {\mathbb {E} _ {\boldsymbol {z} \sim q (\boldsymbol {z})} [ \boldsymbol {z} ^ {\intercal} \boldsymbol {z} ] + m - d}{2} = \frac {d + m - d}{2} = \frac {m}{2}. (A.20) \\ \end{array}
$$
Applying the approximation in Eq. (A.20) to Eq. (A.4), gives us the final approximation
$$
\tilde {H} [ \tilde {q} (\boldsymbol {\theta}) ] \approx \frac {m}{2} + \frac {m}{2} \log 2 \pi + \frac {1}{2} \mathbb {E} _ {\boldsymbol {z} \sim q (\boldsymbol {z})} \left[ \log \det  \left(\boldsymbol {J} _ {g} (\boldsymbol {z}) \boldsymbol {J} _ {g} (\boldsymbol {z}) ^ {\intercal} + \sigma^ {2} \boldsymbol {I} _ {m}\right) \right], \tag {A.21}
$$
which is also stated in Eq. (14). It is important to note here that every Jacobian (w.r.t. input  $z$ ) matrix for generator map  $g_{\gamma}$  is a function of  $\gamma$ , the variational parameters, and hence needs to be differentiable. To obtain whole Jacobian matrices, we use the nnj wrapper from the StochMan library (Detlefsen et al., 2021), which gives us Jacobians in a forward pass rather than requiring a for loop over backward passes.
B Reparaterisation of the LIVI bounds
We note that it is trivial to reparameterise the LIVI bounds as the Gaussian DLVM is straightforward to reparameterise. Using the base variables  $z$ ,  $\eta$ , the generative process for a Gaussian DLVM, c.f. Appendix D, can be reparameterise as
$$
\boldsymbol {\theta} ^ {\prime} = g _ {\gamma} (\boldsymbol {z}), \quad \boldsymbol {z} \sim \mathcal {N} (0, \boldsymbol {I} _ {d}) \tag {B.1}
$$
$$
\boldsymbol {\theta} = \boldsymbol {\theta} ^ {\prime} + \boldsymbol {\eta}, \quad \boldsymbol {\eta} \sim \mathcal {N} (0, \sigma^ {2} \boldsymbol {I} _ {m}). \tag {B.2}
$$
The reparameterised LIVI with whole Jacobians, corresponding to Eq. (15), is
$$
\begin{array}{l} \mathcal {L} ^ {\prime} (\gamma) = \mathbb {E} _ {\boldsymbol {z} \sim q (\boldsymbol {z}), \boldsymbol {\eta} \sim q (\boldsymbol {\eta})} \bigg [ \log p (\mathcal {D} | g _ {\boldsymbol {\gamma}} (\boldsymbol {z}) + \boldsymbol {\eta}) + \log p (g _ {\boldsymbol {\gamma}} (\boldsymbol {z}) + \boldsymbol {\eta}) \\ \left. + \frac {1}{2} \log \det  \left(\boldsymbol {J} _ {g} (\boldsymbol {z}) \boldsymbol {J} _ {g} (\boldsymbol {z}) ^ {\intercal} + \sigma^ {2} \boldsymbol {I} _ {m}\right) \right] + c, \\ \end{array}
$$
and the reparameterised LIVI bound with a differentiable lower bound on the determinant, corresponding to Eq. (18), is
$$
\mathcal {L} ^ {\prime \prime} (\gamma) = \mathbb {E} _ {\boldsymbol {z} \sim q (\boldsymbol {z}), \boldsymbol {\eta} \sim q (\boldsymbol {\eta})} \left[ \log p \left(\mathcal {D} \mid g _ {\boldsymbol {\gamma}} (\boldsymbol {z}) + \boldsymbol {\eta}\right) + \log p \left(g _ {\boldsymbol {\gamma}} (\boldsymbol {z}) + \boldsymbol {\eta}\right) + \frac {d}{2} \log \left(s _ {1} ^ {2} (\boldsymbol {z}) + \sigma^ {2}\right) \right] + c. \tag {B.4}
$$
C Singular values explained
Here we write out the steps that lead from the entropy approximation in Eq. (15) to the efficient entropy approximation we actually use Eq. (16). The first step is to note that the determinant of a square matrix can be written as the product of its eigenvalues and then the log determinant by extension can be written as the sum of the log of these eigenvalues. Let  $s_1(z), \ldots, s_m(z)$  be the singular values of the Jacobian  $J_g(z)^\intercal$  (including zeros), using Petersen et al. (Eq. 286, 2012) we can write the log determinant term in Eq. (15) as
$$
\det  \left(\boldsymbol {J} _ {g} \boldsymbol {J} _ {g} ^ {\intercal}\right) = \prod_ {i = 1} ^ {m} s _ {i} ^ {2} (\boldsymbol {z}) \tag {C.1}
$$
Strictly speaking, the matrix  $J_{g}J_{g}^{\top}$  is not full rank and has only  $d$  non-zero singular values, and thus this determinant will be equal to 0. The additional noise added at the output of the generator resolves this issue as it appears in the  $C(z)$  in Eq. (A.1).
$$
\det  \left(\boldsymbol {J} _ {g} \boldsymbol {J} _ {g} ^ {\intercal} + \sigma^ {2} \boldsymbol {I} _ {m}\right) = \prod_ {i = 1} ^ {d} \left[ s _ {i} ^ {2} (\boldsymbol {z}) + \sigma^ {2} \right] \sigma^ {2 (m - d)} \tag {C.2}
$$
$$
\log \det  \left(\boldsymbol {J} _ {g} \boldsymbol {J} _ {g} ^ {\intercal} + \sigma^ {2} \boldsymbol {I} _ {m}\right) = \sum_ {i = 1} ^ {d} \log \left[ s _ {i} ^ {2} (\boldsymbol {z}) + \sigma^ {2} \right] + 2 (m - d) \log \sigma \tag {C.3}
$$
As we do not wish to calculate all the singular values to estimate Eq. (C.3) exactly, because this would be computationally expensive, we resort to using a lower bound in Eq. (17). The terms in the summation in Eq. (C.3) are positive, so estimating this with any one singular value should give us a lower bound.
Depending on what singular value we wish to compute, we can use the LOBPCG algorithm, which iteratively optimises the generalised Rayleigh quotient and converges in linear time. All details about the convergence results are available in Knyazev (2001). We also employ an efficient backward mode to get vector-jacobian products to work with the generalised Rayleigh quotient, and thus we never need to store large Jacobian matrices during the calculation of this singular value. The Rayleigh quotient is given as follows:
$$
\rho (\boldsymbol {v}) = \frac {\boldsymbol {v} ^ {\intercal} \boldsymbol {J} _ {g} ^ {\intercal} \boldsymbol {J} _ {g} \boldsymbol {v}}{\boldsymbol {v} ^ {\intercal} \boldsymbol {v}} \tag {C.4}
$$
D Novel generator architecture
Here we describe the novel generator architecture. A central property is that we drop dense connections deeper into the network. This resulted in an architecture that had fully connected matrix multiplication layers at the start and smaller individual networks deeper into the architecture. In other words, to decrease the number of parameters closer to the output of the network, dense connections are pruned away. The final output vector of parameters is generated via the concatenation of outputs from smaller networks, which can be thought of as producing weights for each layer in the BNN.
The architecture is pictorially represented in Fig. D.1. All computational blocks, as well as the input block in this network, are matrices, as we follow the standard recipe of the Matrix multiplication neural network (Shi et al., 2018). Starting from the left, we have a large noise vector represented in matrix format. We use the  $65 \times 65$ -dimensional input noise matrix for MNIST experiments and the  $130 \times 130$  input noise matrix for the CIFAR10 experiments. The noise is sampled from the standard normal distribution. Then we use a matrix multiplication layer to produce a higher dimensional output that is split and passed on to a series of disconnected individual matrix multiplication networks
Figure D.1: Our novel CMMNN architecture.
Table E.1: Dual Networks Setups. With this table we clarify the architectures we use for the primary (hypernetwork) and the secondary (BNN) network for training with all the different datasets used for experiments.
Toy Data	UCI Reg	MNIST	CIFAR10	CIFAR100
Hypernetwork	MLP	MLP	MMNN	CMMNN	CMMNN
BNN	MLP	MLP	LeNet	WideResNet(16,4)	WideResNet(28,10)
that produce weights and biases for individual layers of the WideResNet BNN. This way the task of capturing correlations within layers is taken care of by the individual sub-networks towards the end and capturing correlations across layers is taken care of by the homogeneous layer at the start of the network.
E.1 Metrics
For regression, we test generalisation by measuring negative log-likelihoods and mean squared error over the test set.
For the classification experiments, we measure the averaged OOD AUROC and confidence (maximum softmax value) (Daxberger et al., 2021a) on OOD datasets. The OOD AUROC and confidence are crucial metrics to understand the behaviour of the BNN on unseen data. A low confidence score indicates that the model is very uncertain about a classification, while a high AUROC score suggests that the model does not misclassify unseen data with high probability or confidence. Such misclassifications would increase the false positive rate, thereby lowering the AUROC. Therefore, these metrics allow us to measure the model's generalisation performance on OOD data hinting at the quality of the posterior. We further consider the negative log-likelihood (NLL) and expected calibration error (ECE) for some OOD classification experiments. These two metrics ensure that the implicit approximation is not placing mass in undesirable regions of the posterior and is learning the right variance in the function space. If the NLL is lower for OOD points, it signifies that the functional variance farther from the data increases, which is desirable. At the same time, a low ECE indicates that the model's predictive confidence falls with its accuracy as per expectations.
E.2 Experimental setup
Details of the architectures used for the generator/hypernetwork and the Bayesian neural network for all of the experiments in Section 6, can be found in Table E.1. For reference, our novel CMMNN architecture is implemented as CorreMMGenerator in our code.
E.3 Toy experiments
We use a 2D toy sinusoidal dataset for this example with 70 training data points. We intentionally leave a gap in the middle to visualise the epistemic uncertainty or the diversity of functions in this
region as this has been reported by Foong et al. (2019) to be tricky or unachievable for simpler variational approximations.
The BNN architecture is shared across all the methods and is a two-hidden-layered MLP with 7 and 10 units and ELU activations trained with losses unique to each of the methods. We also assume homoscedastic noise in the data that is estimated by all the methods using an extra parameter learnt via type II maximum likelihood. This is set in the data generating process at 0.3.
LIVI: Here, a generator network, which is a one-hidden layered MLP with an 80-dimensional noise input and ELU activations, produces a 105-dimensional output vector which is then used to reparameterise the layers (weights and biases) of a BNN for training.
MFVI: For mean-field VI, we take advantage of the bayesian-torch library (Krishnan et al., 2022). We construct the exact same architecture for the BNN using the reparameterisable layers provided by this library and we slowly warm up the KL term reaching 1 towards the end of training. This is done to help the method converge.
HMC: We sample 10000 samples after a warmup of 2000 samples using the same and neural network architecture. We use the NUTS sampler available in the Pyro (Bingham et al., 2018) package.
AVB: We also compare our method qualitatively against the conventional style of training implicit distributions which uses a density ratio estimating discriminator network. Using the same architecture as above for the BNN, we use a one-hidden layered discriminator network that we retrain within each training loop of the BNN to discriminate 20 samples from the prior and the posterior (generator) each. We run this inner training loop for 10 epochs. The algorithm here is exactly the same as Algorithm 1 in Mescheder et al. (2017a).
E.4 Details of UCI experiments
We use an MLP BNN with 50 units in one hidden layer. We report the RMSE and log-likelihood on held-out data for our method. We use generator architectures with far fewer parameters than Shi et al. (2018) and do not assume independence across layers, i.e., we use one MLP to generate all the weights of the BNN. All of the generator architectures are one hidden layered MLP with a slightly varying number of units depending on the dataset.
We train our method with a homoscedastic assumption, i.e., the variance in the dataset is assumed to be constant and we train an observation noise parameter using type-II maximum likelihood.
E.5 MNIST experiments
Of the wide-ranging approximate inference approaches we compare against, some focus on capturing multiple modes like deep ensembles, while others like MFVI capture information around a single mode sacrificing all correlations for computational feasibility. Similarly, the Laplace approximation, a popular post hoc method (MacKay, 1992), captures unimodal information but can be made much faster by considering a sub-stochastic Bayesian neural network (Daxberger et al., 2021a). Moreover, there is a range of approximations for estimating the hessian that again, trade-off correlations for computational feasibility. In all of these cases, a single MAP estimate remains the easiest baseline to implement and train but represents the minimum amount of information that can be obtained about the posterior surface.
For all methods, we use the same LeNet architecture and for (LLLA, DE & MAP) we train with equal weight decay of  $8 \times 10^{-4}$ . For MFVI, we use a standard normal prior.
LIVI For all our MNIST experiments we use an MMNN with  $65 \times 65$  input noise, one hidden layer of size  $250 \times 250$  and produce an output matrix of size  $350 \times 127$ , the elements of which are used as weights and biases of the LeNet BNN. For all experiments, we trained without dataset augmentation and with a maximum of 1 to 2 samples (from  $q_{\gamma}(\theta)$ ) per minibatch. This network generates close to 44000 parameters for the LeNet BNN while keeping its own parameters appreciably low compared to an MLP hypernetwork conserving computation required for the objective Eq. (18). The weights
and biases of the hypernetwork are the variational parameters that tune the output implicit distribution and here they number slightly over six hundred and ten thousand i.e. these are the total number of weights and biases in the hypernetwork.
LLLA The full Hessian for the last layer can be obtained very cheaply as the last layer Hessian is simply the Generalised Gauss Newton matrix since the network is linear in the last-layer weights (Immer et al., 2021). Hence, we strike a balance between expressivity and computational cost by obtaining the most accurate post-hoc covariance structure over a sub-stochastic network. We use a pretrained LeNet with a weight decay of 8e-4 for this method.
DE To represent deep ensembles we use five LeNet networks trained with different seeds and average over their predictions and use one of these networks to represent the MAP estimate. Each network is trained with a weight decay of 8e-4.
MFVI We use weights from a pre-trained LeNet for initialising the mean and retrain it for several more epochs using the ELBO and a standard normal prior to obtain an isotropic Gaussian posterior. Again this is done using bayesian-torch library and the MOPED functionality (Krishnan et al., 2020).
AVB We again compare our method to the existing adversarial training routine for implicit distributions which is represented by AVB (Algorithm 1, Mescheder et al., 2017a). For this method, we use an MLP discriminator with two hidden layers containing 2500 and 4500 neurons for classifying if a sample comes from the generator or the prior (regularisation term in Eq. (2)). We use 300 samples from the prior and the posterior (generator) for a full batch training of the discriminator for 5 epochs within each minibatch update of the main training loop.
KIVI To compare our method against Shi et al. (2018) we also use MLP BNNs as per their experimental setup and outperform on in-distribution test accuracy reported in Shi et al. (2018, Figure 2, left table). To test fairly against KIVI we use MLP BNN architectures only for this experiment as a LeNet would've been easily able to achieve better accuracy on the in-distribution test set. We find that with the same BNN and hypernetwork architectures we get better test accuracies on MNIST
Training To train LIVI, we use a Matrix Multiplication Neural Network (MMNN, Shi et al., 2018) as the hypernetwork architecture. For the LLLA, we use a complete ("full", Daxberger et al., 2021a) Hessian for the weights and biases in the last layer obtaining all correlations over a smaller parameter set to ensure feasibility. For deep ensembles, we average over five networks trained with weight decay. For one-to-one comparison with AVB we use the exact same generator and training hyperparameters as LIVI. To train LIVI with the aforementioned generator architecture it takes 12.3 GPU minutes on a Titan X (Pacal 12gb vram) GPU and consumes 980 megabytes of memory when training with a batch size of 256. On the other hand, to train a LeNet deep neural network with the same batch size on the same GPU it takes 3.07 GPU minutes and 700 megabytes of GPU memory.
E.6 CIFAR10 Experiments