SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
p (y \mid \boldsymbol {x}, \boldsymbol {w}) q (\boldsymbol {w} _ {c}) = \frac {\prod_ {\ell \in \mathcal {L} _ {p r e d} \wedge \mathcal {L} _ {p o s}} \phi_ {\ell} (y , \boldsymbol {w} _ {c}) [   [ y , \boldsymbol {w} _ {c} | = \ell ]   ]}{\mathrm {W M I} (\Delta , \Phi)}, w i t h (y, \boldsymbol {w} _ {c}) \vDash \Delta .
$$
By integrating over the collapsed set  $W_{c}$ , it further holds that
$$
\begin{array}{l} \int p (y \mid \boldsymbol {x}, \boldsymbol {w}) q (\boldsymbol {w} _ {c}) d \boldsymbol {w} _ {c} \\ = \frac {\int \prod_ {\ell \in \mathcal {L} _ {p r e d} \wedge \mathcal {L} _ {p o s}} \phi_ {\ell} (y , \boldsymbol {w} _ {c}) [   [ y , \boldsymbol {w} _ {c} \models \ell ]   ]   d \boldsymbol {w} _ {c}}{\mathrm {W M I} (\Delta , \Phi)}, w i t h (y, \boldsymbol {w} _ {c}) \vDash \Delta \\ = \frac {\operatorname {W M I} (\Delta \wedge (\mathbf {Y} = y) , \Phi)}{\operatorname {W M I} (\Delta , \Phi)} \\ \end{array}
$$
which proves the first equation.
Similarly, we have that
$$
\begin{array}{l} y p (y \mid \boldsymbol {x}, \boldsymbol {w}) q (\boldsymbol {w} _ {c}) \\ \propto \prod_ {\ell \in \mathcal {L} _ {p r e d}} y \phi_ {\ell} (y, \boldsymbol {w} _ {c}) ^ {[ y, \boldsymbol {w} _ {c} | = \ell ]} \prod_ {\ell \in \mathcal {L} _ {p o s}} \phi_ {\ell} (\boldsymbol {w} _ {c}) ^ {[ \boldsymbol {w} _ {c} | = \ell ]}, w i t h (y, \boldsymbol {w} _ {c}) \vDash \Delta \\ \end{array}
$$
By integrating over the collapsed set  $W_{c}$  and prediction  $y$ , it holds that
$$
\begin{array}{l} \int y p (y \mid \boldsymbol {x}, \boldsymbol {w}) q (\boldsymbol {w} _ {c}) d \boldsymbol {w} _ {c} d y \\ = \frac {\int y \prod_ {\ell \in \mathcal {L} _ {p r e d}} \phi_ {\ell} (y , \boldsymbol {w} _ {c}) ^ {[ y , \boldsymbol {w} _ {c} \models \ell ]} \prod_ {\ell \in \mathcal {L} _ {p o s}} \phi_ {\ell} (\boldsymbol {w} _ {c}) ^ {[ \boldsymbol {w} _ {c} \models \ell ]} d \boldsymbol {w} _ {c} d y}{\mathrm {W M I} (\Delta , \Phi)}, w i t h (y, \boldsymbol {w} _ {c}) \vDash \Delta \\ = \frac {\mathrm {W M I} (\Delta , \Phi^ {*})}{\mathrm {W M I} (\Delta , \Phi)} \\ \end{array}
$$
which finishes our proof.
Algorithm 1 CIBER
Input: input  $x$ , sampled weights  $\mathcal{W}$ , neural network model  $f_{w}$ , prediction ground truth  $y^{*}$
Output: predictions and likelihoods
1: Choose a partition  $(\mathbf{W}_s, \mathbf{W}_c)$  for network parameters
2: Derive approximate posterior  $q(\pmb{w}_c)$  from sampled weights  $\{\pmb{w}_c \mid \pmb{w} \in \mathcal{W}\}$  // cf. Section 4.2
3: Encode posterior  $q(\boldsymbol{w}_c)$  into WMI problem  $\mathcal{M}_{pos} = (\Delta_{pos},\Phi_{pos})$  // cf. Section 4.2
4:  $\mathcal{V}\gets \emptyset ,\mathcal{P}\gets \emptyset$  // Initialization
5: for sample  $\pmb{w}_s$  in  $\{\pmb{w}_s \mid \pmb{w} \in \mathcal{W}\}$  do
6: Encode neural network model  $f_{\mathrm{ReLU}}$  parameterized by  $(\boldsymbol{w}_s, \boldsymbol{W}_c)$  into an SMT formula  $\Delta_{f_w}$
7: Encode predictive  $p(Y \mid \boldsymbol{x}, \boldsymbol{w}_s, \boldsymbol{W}_c)$  into a WMI problem  $\mathcal{M}_{pred} = (\Delta_{pred}, \Phi_{pred})$
8: SMT formula  $\Delta \gets \Delta_{\mathrm{ReLU}} \wedge \Delta_{pos} \wedge \Delta_{pred}$
9: Weights  $\Phi \leftarrow \Phi_{pos} \cup \Phi_{pred}$
0: Weights  $\Phi^{*}\gets \Phi \cup \{\phi_{\ell}(Y) = Y$  with  $\ell = \mathrm{true}\}$
1: Add prediction  $y = \mathsf{WMI}(\Delta, \Phi^*) / \mathsf{WMI}(\Delta, \Phi)$  to prediction set  $\mathcal{V}$  // cf. Section 4.3
2: Add likelihood  $p = \mathsf{WMI}(\Delta \wedge (Y = y^{*}),\Phi) / \mathsf{WMI}(\Delta ,\Phi)$  to set  $\mathcal{P}$  // cf. Section 4.3
3: return  $y = \mathrm{MEAN}(\mathcal{V})$ ,  $p(y^{*} \mid x) = \mathrm{MEAN}(\mathcal{P})$
Table 5: Average test RMSE for the small UCI regression task.
BOSTON	CONCRETE	Yacht	NAVAL	ENERGY
CIBER (SECOND)	3.488 ± 1.123	4.880 ± 0.506	0.828 ± 0.241	0.000 ± 0.000	0.447 ± 0.081
CIBER (LAST)	3.478 ± 1.128	4.854 ± 0.503	0.752 ± 0.294	0.000 ± 0.000	0.447 ± 0.081
SWAG	3.517 ± 0.981	5.233 ± 0.417	0.973 ± 0.375	0.001 ± 0.000	1.594 ± 0.273
PCA+ESS (SI)	3.453 ± 0.953	5.194 ± 0.448	0.972 ± 0.375	0.001 ± 0.000	1.598 ± 0.274
PCA+VI (SI)	3.457 ± 0.951	5.142 ± 0.418	0.973 ± 0.375	0.001 ± 0.000	1.587 ± 0.272
SGD	3.504 ± 0.975	5.194 ± 0.446	0.973 ± 0.374	0.001 ± 0.000	1.602 ± 0.275
MCD	2.830 ± 0.170	4.930 ± 0.140	0.720 ± 0.050	0.000 ± 0.000	1.080 ± 0.030
VSD	2.640 ± 0.170	4.720 ± 0.110	0.690 ± 0.060	0.000 ± 0.000	0.470 ± 0.010
B Pseudo Code for CIBER
We summarize our proposed algorithm CIBER, Collapsed Inference Bayesian DEep LeaRning, for regression tasks, in Algorithm 1. For the classification task, the algorithm is basically the same except the encoding of the predictive of the distribution. Specifically, for a given class  $y$ , the predictive distribution  $p(y \mid x, w)$  can be encoded into a WMI problem as shown below:
$$
\begin{array}{r l} \Delta_ {p r e d} = & f _ {\boldsymbol {w}} (\boldsymbol {x}) \geq - d \quad \Phi_ {p r e d} = \left\{ \begin{array}{l l} \phi_ {\ell_ {1}} (\boldsymbol {W} _ {c}) & \text {w i t h} \ell_ {1} = (f _ {\boldsymbol {w}} (\boldsymbol {x}) \leq d) \\ \phi_ {\ell_ {2}} (\boldsymbol {W} _ {c}) = 1 & \text {w i t h} \ell_ {2} = (f _ {\boldsymbol {w}} (\boldsymbol {x}) > d) \end{array} \right\} \end{array}
$$
where  $\phi_{\ell_1}$  is a cubic polynomial that approximates the sigmoid function such that the posterior predictive distribution  $p(y\mid x)$  can be solved by WMI solvers by  $p(y\mid x) = \mathsf{WMI}(\Delta ,\Phi)$ . Further, the prediction of BMA for classification tasks is made by  $y^{*} = \arg \max_{y}p(y\mid x)$ .
C.1 Toy Regression in Figure 2
We evaluate the predictive distributions obtained by our CIBER and HMC respectively, in a toy dataset generated by sampling 10 input  $x$  uniformly distributed in the interval  $[-1, -0.5]$  and interval  $[0.5, 1]$ . For each input  $x$ , the corresponding target  $y$  is computed from a cubic polynomial with Gaussian noises. We apply to these data a Bayesian neural network which is a ReLU neural network with two hidden layers, where both parameter priors and likelihood are Gaussian distributions. We compare HMC and our CIBER in a few-sample setting which is common in most Bayesian deep learning applications, with 10 samples from the posterior distribution. An estimation generated by HMC with a sufficiently large number of samples of size 2, 000 is further presented as a ground truth.
Table 6: Average test RMSE for the large UCI regression task.
ELEVATORS	KEGGD	KEGGU	PROTEIN	SKILLCRAFT	POL
CIBER (SECOND)	0.088 ± 0.002	0.142 ± 0.074	0.115 ± 0.007	0.438 ± 0.009	0.251 ± 0.010	2.212 ± 0.230
CIBER (LAST)	0.088 ± 0.002	0.142 ± 0.072	0.118 ± 0.012	0.438 ± 0.009	0.251 ± 0.010	2.199 ± 0.182
SWAG	0.088 ± 0.001	0.129 ± 0.029	0.160 ± 0.043	0.415 ± 0.018	0.293 ± 0.015	3.110 ± 0.070
PCA+ESS (SI)	0.089 ± 0.002	0.129 ± 0.028	0.160 ± 0.043	0.425 ± 0.017	0.293 ± 0.015	3.755 ± 6.107
PCA+VI (SI)	0.088 ± 0.001	0.128 ± 0.028	0.160 ± 0.043	0.418 ± 0.021	0.293 ± 0.015	2.499 ± 0.684
SGD	0.103 ± 0.035	0.132 ± 0.017	0.186 ± 0.034	0.436 ± 0.011	0.288 ± 0.014	3.900 ± 6.003
NL	0.101 ± 0.002	0.134 ± 0.036	0.120 ± 0.003	0.447 ± 0.012	0.253 ± 0.011	4.380 ± 0.853
DKL	0.084 ± 0.020	0.100 ± 0.010	0.110 ± 0.000	0.460 ± 0.010	0.250 ± 0.000	6.617
ORTHVGP	0.095	0.120	0.117	0.461	—	4.300 ± 0.200
FF	0.089 ± 0.002	0.120 ± 0.000	0.120 ± 0.000	0.470 ± 0.010	0.250 ± 0.020	—
Table 7: Average test log likelihoods for image classification tasks on CIFAR-10 and CIFAR-100.
CIFAR-10	CIFAR-100
MODEL	VGG-16	PRERESNET-164	WIDERESNET	VGG-16	PRERESNET-164	WIDERESNET
CIBER	0.1927 ± 0.0029	0.1352 ± 0.0014	0.1913 ± 0.0029	0.9193 ± 0.0027	0.8144 ± 0.0065	0.7930 ± 0.0065
SWAG	0.2503 ± 0.0081	0.1459 ± 0.0013	0.1076 ± 0.0009	1.2785 ± 0.0031	1.0703 ± 0.4861	0.6719 ± 0.0035
SGD	0.3285 ± 0.0139	0.1814 ± 0.0025	0.1294 ± 0.0022	1.7308 ± 0.0137	0.9465 ± 0.0191	0.7958 ± 0.0089
SWA	0.2621 ± 0.0104	0.1450 ± 0.0042	0.1075 ± 0.0004	1.2780 ± 0.0051	0.7370 ± 0.0265	0.6684 ± 0.0034
SGLD	0.2001 ± 0.0059	0.1418 ± 0.0005	0.1289 ± 0.0009	0.9699 ± 0.0057	0.6981 ± 0.0052	0.6780 ± 0.0022
KFAC	0.2252 ± 0.0032	0.1471 ± 0.0012	0.1210 ± 0.0020	1.1915 ± 0.0199	0.7881 ± 0.0025	0.7692 ± 0.0092
The results are shown in Figure 2. Even with the same 10 samples drawn from the posterior distribution, since CIBER further approximates the 10 samples with a uniform distribution as  $q(\boldsymbol{w})$ , it yields a predictive distribution  $p(y \mid \boldsymbol{x})$  closer to the ground truth than HMC. The intuition behind is that using a uniform distribution instead of a few samples forms a better approximation to the true posterior since the uniform distribution in a collapsed sample represents uncountably many models.
C.2 Regression on Small and Large Datasets
Sampling from SGD Trajectories. During training, we use Gaussian log likelihood as the objective for obtaining smooth gradients and use early stopping to prevent over-fitting. At convergence, we start the sampling process by keeping running SGD and collecting the weights. At deployment time, we approximate the Gaussian predictive distribution with the triangular distributions.
Hyperparameters. The hyperparameters including learning rates and weight decay are tuned by performing a grid search to maximize the Gaussian log likelihood using a validation split.
Additional Results. Following the set-up of Izmailov et al. (2020), the test performance on small UCI datasets are averaged over 20 trials and the test performance of large UCI datasets are averaged over 10 trials. The test log likelihood results for small UCI datasets are unnormalized while those for large UCI datasets are normalized. Besides the log likelihood results as shown in Section 6.1, root-mean-squared-error (RMSE) results for small UCI datasets are further presented in Table 5 and RMSE results for large UCI datasets are presented in Table 6.
C.3 Image Classification
Sampling from SGD Trajectories. All the network models are trained for 300 epochs using SGD. We start the weight collection after epoch 160 with step size 5. We follow exactly the same hyperparameters as Maddox et al. (2019) including learning rates and weight decay parameters.
Additional Results. Following the set-up of Maddox et al. (2019), we run experiments with VGG-16, PreResNet-164 and WideResNet network models on both the image classification task and the transfer learning task. For the image classification task on CIFAR datasets, we present the log likelihood results in Table 7, the accuracy results in Table 8, and ECE results in Table 9. For the transfer learning task from dataset CIFAR-10 to dataset STL-10, we present the results for model VGG-16 and PreResNet-164 in Table 4 and the results for model WideResNet in Table 10.
Table 8: Average test accuracy for image classification tasks on CIFAR-10 and CIFAR-100.
CIFAR-10	CIFAR-100
MODEL	VGG-16	PRERESNET-164	WIDERESNET	VGG-16	PRERESNET-164	WIDERESNET
CIBER	93.64 ± 0.09	95.95 ± 0.06	95.63 ± 0.16	74.71 ± 0.18	79.23 ± 0.25	81.25 ± 0.35
SWAG	93.59 ± 0.14	96.09 ± 0.08	96.38 ± 0.08	73.85 ± 0.25	73.02 ± 10.30	82.27 ± 0.07
SGD	93.17 ± 0.14	95.49 ± 0.06	96.41 ± 0.10	73.15 ± 0.11	78.50 ± 0.32	80.76 ± 0.29
SWA	93.61 ± 0.11	96.09 ± 0.08	96.46 ± 0.04	74.30 ± 0.22	80.19 ± 0.52	82.40 ± 0.16
SGLD	93.55 ± 0.15	95.55 ± 0.04	95.89 ± 0.02	74.02 ± 0.30	80.09 ± 0.05	80.94 ± 0.17
KFAC	92.65 ± 0.20	95.49 ± 0.06	96.17 ± 0.00	72.38 ± 0.23	78.51 ± 0.05	80.94 ± 0.41
Table 9: Average test ECE for image classification tasks on CIFAR-10 and CIFAR-100.
CIFAR-10	CIFAR-100
MODEL	VGG-16	PRERESNET-164	WIDERESNET	VGG-16	PRERESNET-164	WIDERESNET
CIBER	0.0130 ± 0.0011	0.0250 ± 0.0005	0.0760 ± 0.0011	0.0168 ± 0.0025	0.1423 ± 0.0029	0.1650 ± 0.0046
SWAG	0.0391 ± 0.0020	0.0214 ± 0.0005	0.0096 ± 0.0006	0.1535 ± 0.0015	0.1031 ± 0.0471	0.0678 ± 0.0006
SGD	0.0483 ± 0.0022	0.0255 ± 0.0009	0.0166 ± 0.0007	0.1870 ± 0.0014	0.1012 ± 0.0009	0.0479 ± 0.0010
SWA	0.0408 ± 0.0019	0.0203 ± 0.0010	0.0087 ± 0.0002	0.1514 ± 0.0032	0.0700 ± 0.0056	0.0684 ± 0.0022
SGLD	0.0082 ± 0.0012	0.0251 ± 0.0012	0.0192 ± 0.0007	0.0424 ± 0.0029	0.0363 ± 0.0008	0.0296 ± 0.0008
KFAC	0.0094 ± 0.0005	0.0092 ± 0.0018	0.0060 ± 0.0003	0.0778 ± 0.0054	0.0158 ± 0.0014	0.0379 ± 0.0047
Table 10: Average test performance for image transfer learning tasks using WideResNet.
METRIC	NLL	ACC	ECE
CIBER	0.8259 ± 0.0148	75.02 ± 0.31	0.0336 ± 0.0009
SWAG	1.0142 ± 0.0032	76.96 ± 0.08	0.1303 ± 0.0008
SGD	1.1308 ± 0.0000	76.75 ± 0.00	0.1561 ± 0.0000
SWA	1.0047 ± 0.0000	77.50 ± 0.00	0.1413 ± 0.0000