SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For the choice of predictive distribution  $p(y \mid x, w)$ , we propose to use piecewise polynomial densities. Common predictive distributions can be approximated by polynomials up to arbitrary precision in theory by the Stone-Weierstrass theorem (De Branges, 1959). For regression, the de facto choice is Gaussian and we propose to use triangular distribution as the approximation, i.e.,  $p(y \mid x, w) = \frac{1}{r} - \frac{1}{r^2} |y - f_w(x)|$ , with domain  $|y - f_w(x)| \leq r$ , and  $r \coloneqq \alpha \sqrt{\sigma^2(x)}$  where the constant  $\alpha$  parameterizes the triangular distribution as described in Section 3. Here,  $\sigma^2(x)$  is the variance estimate, which can be a function of input  $x$  depending on whether the BNN is homoscedastic or heteroscedastic. Then  $p(y \mid x, w)$  can be encoded into WMI as:
$$
\Delta_ {p r e d} = \left\{ \begin{array}{l l} Y - f _ {\boldsymbol {w}} (\boldsymbol {x}) \leq r \\ Y - f _ {\boldsymbol {w}} (\boldsymbol {x}) \geq - r \end{array} \right. \Phi_ {p r e d} = \left\{ \begin{array}{l l} \phi_ {\ell_ {1}} (Y, \boldsymbol {W} _ {c}) = \frac {1}{r} - \frac {Y - f _ {\boldsymbol {w}} (\boldsymbol {x})}{r ^ {2}} w i t h \ell_ {1} = (Y > f _ {\boldsymbol {w}} (\boldsymbol {x})) \\ \phi_ {\ell_ {2}} (Y, \boldsymbol {W} _ {c}) = \frac {1}{r} - \frac {f _ {\boldsymbol {w}} (\boldsymbol {x}) - Y}{r ^ {2}} w i t h \ell_ {2} = (f _ {\boldsymbol {w}} (\boldsymbol {x}) > Y) \end{array} \right\}
$$
Similar piecewise polynomial approximations are adopted for classification tasks when the predictive distributions are induced by softmax functions. Those details are presented in the Appendix.
4.3 Exact Integration in Collapsed BMA
By encoding the collapsed BMA into WMI problems, we are ready to answer (Q2), i.e., how to perform exact computation of the integrals shown in Equation 2.
Proposition 7. Let the SMT formula  $\Delta = \Delta_{\mathrm{ReLU}} \wedge \Delta_{pos} \wedge \Delta_{pred}$ , and the set of weights  $\Phi = \Phi_{pos} \cup \Phi_{pred}$  as defined in Section 4.2. Let the set of weights  $\Phi^* = \Phi \cup \{\phi_\ell(Y) = Y \text{ with } \ell = true\}$ . The integrals in collapsed BMA (Equation 2) can be computed by WMI solvers as
$$
\begin{array}{l} \int p (y \mid \boldsymbol {x}, \boldsymbol {w}) q (\boldsymbol {w} _ {c}) d \boldsymbol {w} _ {c} = \mathsf {W M I} (\Delta \wedge (\boldsymbol {Y} = y), \Phi) / \mathsf {W M I} (\Delta , \Phi), a n d \\ \int y p (y \mid \boldsymbol {x}, \boldsymbol {w}) q (\boldsymbol {w} _ {c}) d \boldsymbol {w} _ {c} d y = \mathsf {W M I} (\Delta , \Phi^ {*}) / \mathsf {W M I} (\Delta , \Phi). \\ \end{array}
$$
With both questions (Q1) and (Q2) answered, we summarize our proposed algorithm CIBER in Algorithm 1 in the Appendix. To quantitatively analyze how close the approximation delivered by CIBER is to the ground-truth BMA, we consider the following experiments with closed-form BMA.
Figure 3: Posterior predictive distributions in Bayesian linear regression. The  $y$ -axis shows the absolute difference between an estimated predictive distribution  $p(y \mid x)$  and the ground-truth predictive distribution  $q(y \mid x)$ . Shaded regions are the  $95\%$  confidence interval. Best viewed in color.
Figure 4: KL divergence in Bayesian linear regression. The  $x$ -axis shows the number of samples the MC method uses for estimations, ranging from 50 to 150. The blue curve shows the MC method, and the green dashed curve shows CIBER using 50 samples.
Regression. We consider a Bayesian linear regression setting where exact sampling from the posterior distribution is available. Both the likelihood and the weight posterior are Gaussian such that the ground-truth posterior predictive distribution is Gaussian as well. With samples drawn from the weight posterior, CIBER approximates the samples with a uniform distribution as posterior  $p(\boldsymbol{w}|\mathcal{D})$  and further approximates the likelihood with a triangular distribution such that the integral  $p(y|\boldsymbol{x},\mathcal{D}) = \int p(y|\boldsymbol{x},\boldsymbol{w})p(\boldsymbol{w}|\mathcal{D})d\boldsymbol{w}$  can be computed exactly by WMI.
We first evaluate the posterior predictive distribution estimated by CIBER and Monte Carlo (MC) method, using the same five samples drawn from the weight posterior. Results averaged over 10 trials are shown in Figure 3 where the estimations by CIBER are much closer to the ground truth posterior predictive distribution than those by the MC method. Further, the averaged KL divergence between the ground truth and CIBER is 0.069 while the one for MC estimations is 0.130, again indicating that CIBER yields a better BMA approximation in the few-sample setting.
We further explore the question of how many samples the MC method needs to match the performance of CIBER. The performances of both approaches are evaluated using KL divergence between the ground-truth posterior distribution and the estimated one, averaged over 10 trials. The result is shown in Figure 4 where the dashed green line shows the performance of CIBER with 50 samples and the blue curve shows the performance of MC with an increasing number of samples. As expected, the MC method yields lower KL divergence as the number of samples increases; however, it takes more than 100 samples to match CIBER, indicating its low sample efficiency and that developing efficient and effective inference algorithms such as CIBER for estimating BMA is a meaningful question.
Classification. For analyzing classification performance, Kristiadi et al. (2022) propose to compute the integral  $I = \int \sigma(f_{*}) p_{\mathcal{N}}(f_{*}) df_{*}$  with  $\sigma$  being the sigmoid function and  $f_{*} = f(\boldsymbol{x}^{*}; \boldsymbol{w})$  that amounts to the posterior predictive distribution. We consider a simple case with  $f(\boldsymbol{x}; \boldsymbol{w}) = \boldsymbol{w} \cdot \boldsymbol{x}$  such that the ground-truth integral can be obtained. With a randomly chosen  $\boldsymbol{x}$ , the ground-truth integral is  $I = 0.823$ . The integral estimated by CIBER is  $I_{C} = 0.826$  while the MC estimate is  $I_{MC} = 0.732$ . That is, CIBER gives an estimate with a much lower error than the MC estimation error, indicating that CIBER is able to deliver high-quality approximations in classification tasks.
Bayesian Deep Learning. Bayesian inference over deep neural networks (MacKay, 1992) is proposed to fix the issue that deep learning models give poor uncertainty estimations and suffer from overconfidence (Nguyen et al., 2015; Hein et al., 2019; Meronen et al., 2023, 2021). Some methods use samples from SGD trajectories to approximate the implicit true posteriors similar to us: Izmailov et al. (2020) (SI) proposes to perform Bayesian inference in a subspace of the parameter space spanned by a few vectors derived from principal component analysis (PCA+ESS(SI)) or variational inference (PCA+VI(SI)); SWAG (Maddox et al., 2019) proposes to approximate the full parameter space using an approximate Gaussian posterior whose mean and covariance are from a partial SGD trajectory with a modified learning rate scheduler.
Some other approaches using approximate posteriors include MC Dropout (MCD) (Gal & Ghahramani, 2015, 2016) which is one of the Bayesian dropout methods and recently, one of its modifications called Variational Structured Dropout (VSD) (Nguyen et al., 2021) using variational inference is pro
Table 1: Average test log likelihood for the small UCI regression task.
BOSTON	CONCRETE	Yacht	NAVAL	ENERGY
CIBER (SECOND)	-2.471 ± 0.140	-2.975 ± 0.102	-0.678 ± 0.301	7.276 ± 0.532	-0.716 ± 0.211
CIBER (LAST)	-2.471 ± 0.140	-2.959 ± 0.109	-0.687 ± 0.301	7.482 ± 0.188	-0.716 ± 0.211
SWAG	-2.761 ± 0.132	-3.013 ± 0.086	-0.404 ± 0.418	6.708 ± 0.105	-1.679 ± 1.488
PCA+ESS (SI)	-2.719 ± 0.132	-3.007 ± 0.086	-0.225 ± 0.400	6.541 ± 0.095	-1.563 ± 1.243
PCA+VI (SI)	-2.716 ± 0.133	-2.994 ± 0.095	-0.396 ± 0.419	6.708 ± 0.105	-1.715 ± 1.588
SGD	-2.752 ± 0.132	-3.178 ± 0.198	-0.418 ± 0.426	6.567 ± 0.185	-1.736 ± 1.613
DVI	-2.410 ± 0.020	-3.060 ± 0.010	-0.470 ± 0.030	6.290 ± 0.040	-1.010 ± 0.060
DGP	-2.330 ± 0.060	-3.130 ± 0.030	-1.390 ± 0.140	3.600 ± 0.330	-1.320 ± 0.030
VI	-2.430 ± 0.030	-3.040 ± 0.020	-1.680 ± 0.040	5.870 ± 0.290	-2.380 ± 0.020
MCD	-2.400 ± 0.040	-2.970 ± 0.020	-1.380 ± 0.010	4.760 ± 0.010	-1.720 ± 0.010
VSD	-2.350 ± 0.050	-2.970 ± 0.020	-1.140 ± 0.020	4.830 ± 0.010	-1.060 ± 0.010
posed. Other state-of-the-art approximate BNN inference methods including deterministic variational inference (DVI) (Wu et al., 2019), deep Gaussian processes (DGP) (Bui et al., 2016) with Gaussian process layers and variational inference (VI) (Kingma & Welling, 2013). Closely related to DGP is the deep kernel process (Aitchison et al., 2021) that writes DGPs as deep Wishart processes.
WMI Solvers. WMI generalizes weighted model counting (WMC) Sang et al. (2005), a state-of-the-art inference approach in many discrete probabilistic models, from discrete to mixed discrete-continuous domains Belle et al. (2015a,b). Recent research on WMI includes its tractability (Zeng et al., 2020c, 2021; Abboud et al., 2020) and the advancements in WMI solver development. Existing exact WMI solvers for arbitrarily structured problems include DPLL-based search with numerical Belle et al. (2015a); Morettin et al. (2017, 2019) or symbolic integration de Salvo Braz et al. (2016) and compilation-based algorithms Kolb et al. (2018); Zuidberg Dos Martires et al. (2019); Derkinderen et al. (2020) that use extended algebraic decision diagrams (XADDs) (Sanner et al., 2012) as a compilation target which is a powerful tool for inference on mixed domains (Sanner & Abbasnejad, 2012; Zamani et al., 2012). Some exact WMI solvers aiming to improve efficiency for a certain class of models are proposed such as SMI (Zeng & Van den Broeck, 2019) and MP-WMI (Zeng et al., 2020a) which are greatly scalable for WMI problems that satisfy certain structural constraints. Approximate solvers are also proposed including sampling-based ones (Zuidberg Dos Martires et al., 2020) and relaxation-based ones (Zeng et al., 2020c,b). Recent WMI efforts converge in the pywmi library (Kolb et al., 2019). The SMT formulas considered in this work can be seen as distributional constraints on continuous domains. There is also plenty of work in neuro-symbolic AI exploring the integration of discrete constraints into neural networks models including the architectures (Ahmed et al., 2022b, 2023) and the loss (Xu et al., 2018; Ahmed et al., 2022c,a).
We conduct experimental evaluations of our proposed approach CIBER  ${}^{1}$  on regression and classification benchmarks and compare its performance on uncertainty estimation as well as prediction accuracy with a wide range of baseline methods. More experimental details are presented in the Appendix.
6.1 Regression on Small and Large UCI Datasets
We experiment on 5 small UCI datasets: boston, concrete, yacht, naval and energy. We follow the setup of Izmailov et al. (2020) and use a fully connected network with a single hidden layer and 50 units with ReLU activations. We further experiment on 6 large UCI datasets: elevators, keggdirected, keggundirected, pol, protein and skillcraft. We use a feedforward network with five hidden layers of sizes [1000, 1000, 500, 50, 2] and ReLU activations on all datasets except skillcraft. For skillcraft, a smaller architecture is adopted with four hidden layers of size [1000, 500, 50, 2]. All models have two outputs for the prediction and the heteroscedastic variance respectively.
$^{1}$ Code and experiments are available at https://github.com/UCLA-StarAI/CIBER.
Table 2: Average test log likelihood for the large UCI regression task.
ELEVATORS	KEGGD	KEGGU	PROTEIN	SKILLCRAFT	POL
CIBER (SECOND)	-0.378 ± 0.026	1.245 ± 0.090	1.125 ± 0.269	-0.720 ± 0.036	-1.003 ± 0.035	2.555 ± 0.115
CIBER (LAST)	-0.371 ± 0.023	1.178 ± 0.088	0.964 ± 0.231	-0.720 ± 0.036	-1.001 ± 0.032	2.506 ± 0.150
SWAG	-0.374 ± 0.021	1.080 ± 0.035	0.749 ± 0.029	-0.700 ± 0.051	-1.180 ± 0.033	1.533 ± 1.084
PCA+ESS (SI)	-0.351 ± 0.030	1.074 ± 0.034	0.752 ± 0.025	-0.734 ± 0.063	-1.181 ± 0.033	-0.185 ± 2.779
PCA+VI (SI)	-0.325 ± 0.019	1.085 ± 0.031	0.757 ± 0.028	-0.712 ± 0.057	-1.179 ± 0.033	1.764 ± 0.271
SGD	-0.538 ± 0.108	1.012 ± 0.154	0.602 ± 0.224	-0.854 ± 0.085	-1.162 ± 0.032	1.073 ± 0.858
ORTHVGP	-0.448	1.022	0.701	-0.914	—	0.159
NL	-0.698 ± 0.039	0.935 ± 0.265	0.670 ± 0.038	-0.884 ± 0.025	-1.002 ± 0.050	-2.840 ± 0.226
Table 3: Average test performance for image classification tasks on CIFAR-10 and CIFAR-100.
METRIC	NLL	ACC	ECE
DATASET	CIFAR-10	CIFAR-100	CIFAR-10	CIFAR-100	CIFAR-10	CIFAR-100
CIBER	0.1927 ± 0.0029	0.9193 ± 0.0027	93.64 ± 0.09	74.71 ± 0.18	0.0130 ± 0.0011	0.0168 ± 0.0025
SWAG	0.2503 ± 0.0081	1.2785 ± 0.0031	93.59 ± 0.14	73.85 ± 0.25	0.0391 ± 0.0020	0.1535 ± 0.0015
SGD	0.3285 ± 0.0139	1.7308 ± 0.0137	93.17 ± 0.14	73.15 ± 0.11	0.0483 ± 0.0022	0.1870 ± 0.0014
SWA	0.2621 ± 0.0104	1.2780 ± 0.0051	93.61 ± 0.11	74.30 ± 0.22	0.0408 ± 0.0019	0.1514 ± 0.0032
SGLD	0.2001 ± 0.0059	0.9699 ± 0.0057	93.55 ± 0.15	74.02 ± 0.30	0.0082 ± 0.0012	0.0424 ± 0.0029
KFAC	0.2252 ± 0.0032	1.1915 ± 0.0199	92.65 ± 0.20	72.38 ± 0.23	0.0094 ± 0.0005	0.0778 ± 0.0054
We run CIBER with two different ways of choosing the collapsed parameter set: CIBER (last) chooses all the weights at the last layer to be the collapsed set; CIBER (second) chooses three out of all the weights at the second-to-last layer to be the collapsed set. The heuristic we use for choosing the weights is to look into the sampled weights from SGD trajectories to see which ones have the greatest variance. The intuition is that a greater variance indicates that the weight is prone to have greater uncertainty and thus one might want to perform a more accurate inference over it.
Baselines. We compare CIBER to the state-of-the-art approximate BNN inference methods. We separate these methods into two categories: those sampling from SGD trajectories as approximate posteriors, which includes SWAG (Maddox et al., 2019), PCA+ESS (SI) and PCA+VI (SI) (Izmailov et al., 2020), vs. those who do not, which includes the SGD baseline, deterministic variational inference (DVI) (Wu et al., 2019), Deep Gaussian Processes (DGP) (Bui et al., 2016), variational inference (VI) (Kingma & Welling, 2013), MC Dropout (MCD) (Gal & Ghahramani, 2015, 2016), and variational structured dropout (VSD) (Nguyen et al., 2021). These methods achieved state-of-the-art performance on the small UCI datasets. We also compare to baselines Bayesian final layers (NL) (Riquelme et al., 2018), deep kernel learning (DKL) (Wilson et al., 2016), orthogonally decoupled variational GPs (OrthVGP) (Salimbeni et al., 2018) and Fastfood approximate kernels (FF) (Yang et al., 2015), which have achieved state-of-the-art performance on the large UCI datasets.
Results. We present the test log likelihoods for small UCI datasets in Table 1 and those for large UCI datasets in Table 2. In both tables, the first block summarizes SGD-trajectory sampling-based approaches and the second summarizes the rest. Underlined results are the best among all and bold results are the best among SGD-trajectory sampling-based approaches. From the results, our CIBER has substantially better performance than all others on three out of the five small UCI datasets four out of six large UCI datasets, with comparable performance on the rest, demonstrating that CIBER provides accurate uncertainty estimation. We also present the test rooted-mean-squared error results in Appendix, where CIBER outperforms all other SGD-trajectory sampling-based baselines on four out of five small UCI datasets and four out of six large UCI datasets; it outperforms all baselines on two small UCI datasets and one large UCI datasets and has comparable performance on the rest. This further illustrates that exact marginalization over conditional approximate posteriors enabled by WMI solvers achieves accurate estimation of the true BMA and boosts predictive performance.
6.2 Image Classification
CIFAR datasets. We experiment with two image datasets: CIFAR-10 and CIFAR-100 (Krizhevsky et al., 2009) and evaluate the test performance using three metrics: 1) negative log likelihood (NLL)
Table 4: Average test performance for image transfer learning tasks.
METRIC	NLL	ACC	ECE
MODEL	VGG-16	PRERESNET-164	VGG-16	PRERESNET-164	VGG-16	PRERESNET-164
CIBER	0.9869 ± 0.0102	0.9684 ± 0.0075	72.56 ± 0.23	75.70 ± 0.17	0.0925 ± 0.0028	0.0704 ± 0.0031
SWAG	1.3425 ± 0.0015	1.3842 ± 0.0122	72.30 ± 0.11	76.30 ± 0.06	0.1988 ± 0.0028	0.1668 ± 0.0006
SGD	1.6528 ± 0.0390	1.4790 ± 0.0000	72.42 ± 0.07	75.56 ± 0.00	0.2149 ± 0.0027	0.1758 ± 0.0000
SWA	1.3993 ± 0.0502	1.3552 ± 0.0000	71.92 ± 0.01	76.02 ± 0.00	0.2082 ± 0.0056	0.1739 ± 0.0000
that reflects the quality of both uncertainty estimation and prediction accuracy, 2) classification accuracy (ACC), and 3) expected calibration errors (ECE) (Naeini et al., 2015) that show the difference between predictive confidence and accuracy and should be close to zero for a well-calibrated approach.
We run CIBER by choosing the collapsed parameter set to be 10 weights and 100 weights at the last layer of the neural network models for CIFAR-10 and CIFAR-100 respectively. The weights are chosen using the same heuristic as the one for regression tasks, i.e., to choose the weights whose samples from the SGD trajectories have large variances. We compare CIBER with strong baselines including SWAG (Maddox et al., 2019) reproduced by their open-source implementation, standard SGD, SWA (Izmailov et al., 2018), SGLD (Welling & Teh, 2011) and KFAC (Ritter et al., 2018).
Transfer from CIFAR-10 to STL-10. We further consider a transfer learning task using the model trained on CIFAR-10 to be evaluated on dataset STL-10 (Coates et al., 2011). STL-10 shares nine out of ten classes with the CIFAR-10 dataset but has a different image distribution. It is a common benchmark in transfer learning to adapt models trained on CIFAR-10 to STL-10.
Results. We present the test classification performance on dataset CIFAR-10 and CIFAR-100 in Table 3 and that of transfer learning in Table 4. The neural network models used in the classification task are VGG-16 networks and the models used in the transfer learning task are VGG-16 and PreResNet-164. More results using different network architectures are presented in the Appendix. With the same number of samples as SWAG, CIBER outperforms SWAG and other baselines in most evaluations and delivers comparable performance otherwise, demonstrating the effectiveness of using collapsed samples in improving uncertainty estimation as well as classification performance.
7 Conclusions And Future Work
We reveal the connection between BMA, a way to perform Bayesian deep learning and WVC, which inspires us to approximate BMA using the framework of WMI. To further make this approximation scalable and flexible, we combine it with collapsed samples which gives our algorithm CIBER. CIBER compares favorably to Bayesian deep learning baselines on regression and classification tasks. A future direction would be to explore what other layers can be expressed as SMT formulas and thus amenable to SMT encoding. Also, the current WMI solvers are limited to polynomial weights, and thus the reduction to WMI problems is applicable to piecewise polynomial weights. This limitation might be alleviated in the future by the development of new WMI solvers that allow various weight function families.
We would like to thank Yuhang Fan and Kareem Ahmed for helpful discussions. We would also like to thank Wesley Maddox for answering queries on the implementation of baseline SWAG. This work was funded in part by the DARPA PTG Program under award HR00112220005, the DARPA ANSR program under award FA8750-23-2-0004, NSF grants #IIS-1943641, #IIS-1956441, #CCF-1837129, and a gift from RelationalAI. GVdB discloses a financial interest in RelationalAI. ZZ is supported by an Amazon Doctoral Student Fellowship.
Abboud, R., Ceylan, I. I., and Dimitrov, R. On the approximability of weighted model integration on DNF structures. arXiv preprint arXiv:2002.06726, 2020.
Xu, J., Zhang, Z., Friedman, T., Liang, Y., and Van den Broeck, G. A semantic loss function for deep learning with symbolic knowledge. In International conference on machine learning, pp. 5502-5511. PMLR, 2018.
Yang, Z., Wilson, A., Smola, A., and Song, L. A la carte-learning fast kernels. In Artificial Intelligence and Statistics, pp. 1098-1106. PMLR, 2015.
Zamani, Z., Sanner, S., and Fang, C. Symbolic dynamic programming for continuous state and action mdps. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 26, pp. 1839-1845, 2012.
Zeng, Z. and Van den Broeck, G. Efficient search-based weighted model integration. Proceedings of UAI, 2019.
Zeng, Z., Morettin, P., Yan, F., Vergari, A., and Broeck, G. V. d. Scaling up hybrid probabilistic inference with logical and arithmetic constraints via message passing. In Proceedings of the International Conference of Machine Learning (ICML), 2020a.
Zeng, Z., Morettin, P., Yan, F., Vergari, A., and Van den Broeck, G. Relax, compensate and then integrate. In Proceedings of the ECML-PKDD Workshop on Deep Continuous-Discrete Machine Learning (DeCoDeML), 2020b.
Zeng, Z., Morettin, P., Yan, F., Vergari, A., and Van den Broeck, G. Probabilistic inference with algebraic constraints: Theoretical limits and practical approximations. Advances in Neural Information Processing Systems, 33:11564-11575, 2020c.
Zeng, Z., Morettin, P., Yan, F., Passerini, A., Van den Broeck, G., et al. Is parameter learning via weighted model integration tractable? In The 4th Workshop on Tractable Probabilistic Modeling, 2021.
Zuidberg Dos Martires, P. M., Dries, A., and De Raedt, L. Exact and approximate weighted model integration with probability density functions using knowledge compilation. In Proceedings of the 30th Conference on Artificial Intelligence. AAAI Press, 2019.
Zuidberg Dos Martires, P. M. et al. Monte carlo anti-differentiation for approximate weighted model integration. In Proceedings of the Ninth International Workshop on Statistical Relational AI. International Workshop on Statistical Relational AI, 2020.
Proposition 7 Let the SMT formula  $\Delta = \Delta_{\mathrm{ReLU}} \wedge \Delta_{pos} \wedge \Delta_{pred}$ , and the set of weights  $\Phi = \Phi_{pos} \cup \Phi_{pred}$  as defined in Section 4.2. Let the set of weights  $\Phi^* = \Phi \cup \{\phi_\ell(Y) = Y \text{ with } \ell = true\}$ . The integrals in collapsed BMA (Equation 2) can be computed by WMI solvers as
$$
\int p (y \mid \boldsymbol {x}, \boldsymbol {w}) q (\boldsymbol {w} _ {c}) d \boldsymbol {w} _ {c} = \mathsf {W M I} (\Delta \wedge (\boldsymbol {Y} = y), \Phi) / \mathsf {W M I} (\Delta , \Phi), a n d
$$
$$
\int y p (y \mid \boldsymbol {x}, \boldsymbol {w}) q (\boldsymbol {w} _ {c}) d \boldsymbol {w} _ {c} d y = \mathsf {W M I} (\Delta , \Phi^ {*}) / \mathsf {W M I} (\Delta , \Phi).
$$
Proof. By construction, it holds that
$$
\begin{array}{l} p (y \mid \boldsymbol {x}, \boldsymbol {w}) \propto \mathsf {W M I} (\Delta_ {\text {p r e d}} \wedge (\boldsymbol {Y} = y), \Phi_ {\text {p r e d}}) \\ \propto \prod_ {\ell \in \mathcal {L} _ {p r e d}} \phi_ {\ell} (y, \boldsymbol {w} _ {c}) ^ {[ y, \boldsymbol {w} _ {c} | = \ell ]}, w i t h (y, \boldsymbol {w} _ {c}) \models \Delta_ {p r e d} \wedge \Delta_ {\text {R e L U}} \\ q (\boldsymbol {w} _ {c}) \propto \mathsf {W M I} (\Delta_ {p o s} \wedge (\boldsymbol {W} _ {c} = \boldsymbol {w} _ {c}), \Phi_ {p o s}) \\ \propto \prod_ {\ell \in \mathcal {L} _ {p o s}} \phi_ {\ell} (\boldsymbol {w} _ {c}) ^ {\llbracket \boldsymbol {w} _ {c} \models \ell \rrbracket}, w i t h \boldsymbol {w} _ {c} \vDash \Delta_ {p o s} \\ \end{array}
$$
Thus, we have that the likelihood weighted by the approximate posterior would be
$$
p (y \mid \boldsymbol {x}, \boldsymbol {w})   q (\boldsymbol {w} _ {c}) \propto \prod_ {\ell \in \mathcal {L} _ {p r e d} \wedge \mathcal {L} _ {p o s}} \phi_ {\ell} (y, \boldsymbol {w} _ {c}) ^ {\llbracket y, \boldsymbol {w} _ {c} \mid = \ell \rrbracket} w i t h (y, \boldsymbol {w} _ {c}) \models \Delta_ {\text {R e L U}} \wedge \Delta_ {p r e d} \wedge \Delta_ {p o s},
$$