Title: Approximate Message Passing for Bayesian Neural Networks
Venue: NeurIPS 2025
Accepted: no
Keywords: Bayesian Neural Networks

Message Passing

Uncertainty Quantification

Bayesian Inference
Abstract: Bayesian methods have the ability to consider model uncertainty within a single framework and provide a powerful tool for decision-making. Bayesian neural networks (BNNs) hold great potential for better uncertainty quantification and data efficiency, making them promising candidates for more trustworthy AI in critical applications, and as backbones in data-constrained settings such as real-world reinforcement learning.  However, current approaches often face limitations such as overconfidence, sensitivity to hyperparameters, and posterior collapse, highlighting the need for alternative approaches. In this paper, we introduce a novel method that leverages message passing (MP) to model the predictive posterior of BNNs as a factor graph. Unlike previous MP-based methods, our framework is the first to support convolutional neural networks (CNNs) while addressing the issue of double-counting training data, which has been a key source of overconfidence in prior work. Multiple open datasets are used to demonstrate the general applicability of the method and to illustrate its differences to existing inference methods.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a message passing algorithm for inference in BNNs. The approach represents the topology of the network as a factor graph. Using message passing is realiyed via belief propagation. The authors test their method on MNIST and Cifar 10\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- message passing approaches are an underdeveloped field in ML, especially for uncertainty-modeling so the authors raise a significant research question\n- The experiments on CIFAR-10 using a 890k parameter model hint at scalability potential of the method.\n\nWeaknesses:\n\nEvaluation: \n1. The benchmarks for evaluation (MNIST and CIFAR 10) are dated and make it very hard to asses the scalability and practicality of the proposed method. \n\n2. The technique is a Bayesian modeling approach, thus it should be compared to standard BNN approximations: for instance, ELBO, MC-Dropout, Deep ensembles. \n\n3. For a  Bayesian method should be evaluated via Log-likelihood, OOD detection. I don't fully understand the comparison to AdamW, which is an optimizer, and not a Bayesian method in the sense that it does not learn parametric or functional uncertainty. A BNN, an ensemble- or any architecture that utilizes backpropagation can be trained with Adam. For AdamW the baseline is thus a MAP approach. \n\n4. The most similar work to this is arguably probabilistic backpropagation (PBP) and stochastic expectation propagation (SEP). You say \"However, PBP treats the data as new examples in each consecutive epoch (double-counting), which makes it prone to overconfidence. Furthermore, EBP and PBP were both only deployed on small datasets and rely on gradients instead of pure message passing\".\n\nThe authors didn't show empirically that the double counting leads to inferior performance for uncertainty quantification. In literature one key problem of VI-based BNNs is underfitting and not overfitting, and PBP/SEP are more prone to underconfidence than overconfidence. Also regarding scalability this statement needs to be empirically verified that the proposed message passing approach is computationally more efficient.\n\nQUESTIONS:\nn/a\n\nLIMITATIONS:\nn/a", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
This paper explores a message passing framework for uncertainty quantification of BNN. It scales to CNN. The proposed method outperforms baselines on MNIST and CIFAR-10 datasets.

STRENGTHS AND WEAKNESSES:
The presentation needs improvement. Overall, it is hard to follow. It would be helpful to give an overview of the proposed method in the Introduction Section before describing contributions. Some terms, e.g., “double counting data” in line 97, are directly used without clear definition or explanation. In line 100, the authors mentioned “scale to CNNs”, without motivating it. What would be the benefits of scaling to CNN? 

In Section 3, the notations are used without definition. What is d, a, p? In Figure 1, what is z? It will be better to clearly define notations and the MLP, and then how Figure 1 is obtained. In line 116, are the latent variables manually defined?

Due to the poor clarity, the real contributions of the proposed approach are not clear. A better organization would greatly help highlight the novelty and contribution.

Experiments are insufficient. Table 2 shows the comparison to SOTA on CIFAR-10. The proposed approach only achieves the best performance with 3 measurements, out of 6. Plus, the improvement of Brier is marginal (0.009). The significance of the proposed method hence is unclear.

QUESTIONS:
Belief Propagation is also a variational approach with Bethe assumption [1].  The selection of factors affects the performance of belief propagation inference. How does the proposed approach connect or different from this?

[1] Generalized Belief Propagation, Jonathan S. Yedidia, William T. Freeman, Yair Weiss, NeurIPS 2000. 

Other questions about clarity are in Weakness section above.

LIMITATIONS:
Limitations are well discussed.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new approximate inference method for Bayesian Neural Networks (BNNs) using a message passing (MP) framework based on belief propagation. Unlike existing MP approaches that suffer from double-counting of training data and limited architectural support, the proposed method models the predictive posterior as a factor graph and approximates messages using Gaussian densities in natural parameter space.  Interestingly, the proposed approach supports convolutional networks. Training involves loopy belief propagation with a forward-backward message schedule, and prediction is made using the aggregated posterior approximation.\n\nIn their experiments, the authors illustrate their method on MNIST, CIFAR-10, and UCI tabular datasets, comparing it against SGD, AdamW, and IVON. The proposed approach shows better calibration and uncertainty quantification in low-data regimes. On MNIST, it significantly outperforms SGD in both accuracy and expected calibration error (ECE) when data is scarce. On CIFAR-10, the accuracy is lower than competitors, with also higher run times, but calibration is better. Finally note that the method is implemented in Julia with GPU support and is made available by the authors.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper presents a technically sound and carefully implemented message passing method that addresses two major shortcomings of prior MP methods for BNNs: support for CNNs and avoidance of double-counting. In particular, here the authors extend MP to architectures beyond MLPs, which is a notable advancement over prior work such as PBP. The Gaussian message approximation strategy is clearly detailed, and I confess I really like the summary table in Appendix F. The paper also offers insightful experimental evaluations with a focus on uncertainty calibration, which is in my opinion too neglected in many BNN comparisons, with convincing results in the small data regime. In addition, reproducibility is possible with the available code.\n\nHowever, I think (at least) two competitors are missing: a simple Laplace approximation and deep ensembles. The latter, in particular, was shown to be quite well-calibrated in some settings, and I would like to see how it performs with respect to the approximate MP approach proposed here. I would be willing to increase my score if such comparisons are added and show that the proposed method is competitive.\n\nOn a side note, I also have the following remarks:\n\n   * The exact same paragraph is repeated twice P7 and P8\n   * The acronym R-MP is introduced after its first use, and AP-MP is not introduced\n   * In the same vein, regression-based MP is applied to the MNIST classification task, so I am quite confused about the experimental setting\n   * Appendix C contains the description of a synthetic dataset, but I do not see any results on this case either in the paper nor in the Appendix (maybe I missed something)\n\nQUESTIONS:\n* In line with my comment above, how does the method perform compared to deep ensembles or Laplace approximations, which are simpler?\n* What would be the computational/memory overhead if we consider e.g. ResNets or Transformers?\n* How sensitive is the method to the choice of prior variance or moment-matching approximations? In particular, do we expect a stronger impact with deeper architectures?\n\nLIMITATIONS:\nLimitations are clearly discussed in the conclusion, as well as future work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors propose using a message passing framework to directly model the predictive posterior of a BNN. The authors argue that one of the core issues in such methods is that message passing for large datasets and networks is computationally intractable. To alleviate this, the authors present approximations for the messages of their message passing framework, and a number of computational and implementation tricks that enable more efficient training and prediction for these methods. The authors then present experiments utilising their methods to train MLPs and CNNs on MNIST and CIFAR10, whilst comparing to regular SGD training and IVON training.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The method appears to perform well, especially in terms of uncertainty quantification which is a key topic of interest for the wider community.\n- The authors provide more efficient implementations of their method, which is essential for the use of such methods for practitioners.\n- The paper is generally clear and straightforward to read.\n\nWeakness\n- The details of method are in my opinion difficult to track and follow (especially for someone not from the MP community). This weakness is the main reason for my clarity score.\n- The experiments are quite small scale. This weakness is part of the reason for my significance score, but it has negligible impact on it.\n- The authors only compare to a single other BNN method, which arguably does not represent the full field of BNN methods very well (and has been reported as difficult to train). This weakness is the main reason for my significance score.\n\nQUESTIONS:\nIn relation to the weaknesses above, I list my questions/suggested additions/changes here:\n1. I would suggest that the authors include a schematic or algorithm figure to more accurately describe their method, such that others may implement it more easily. Although it may be due to my limited knowledge of message passing, I would currently find it extremely difficult to implement the method myself.\n2. A more high level introduction of variational MP might also aid readers in understanding the proposed approximations for MP easier.\n3. Although not strictly necessary for newly proposed methods, I am curious on 1. how this method would scale to larger networks (i.e. does it work for larger networks) and 2. A practical discussion of how difficult these models are to run, such as hyperparameter tuning etc. that may have been required on a dataset/architecture level.\n4. I would strongly suggest that the authors include additional BNN baselines. Currently the authors only run against IVON, a recently proposed optimizer which has been reported by others to be difficult to use. My suggestions for additional methods would be: 1. A more traditional VI method (could be full mean field VI or variational bayesian last layers (VBLL)). 2. Perhaps a sampling-based method, e.g. SGLD or MALA 3.  Probabilistic backpropogation. In the current state of the work, it is difficult to gauge where the method lies within the wide landscape of BNN methods in terms of performance\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}