Title: Martingale Posterior Neural Networks for Fast Sequential Decision Making
Venue: NeurIPS 2025
Accepted: yes
Keywords: online learning

neural bandits

sequential decision making

Kalman filtering

frequentist

bayes

martingale posterior
Abstract: We introduce scalable algorithms for online learning of neural network parameters and Bayesian sequential decision making.
Unlike classical Bayesian neural networks,
which induce predictive uncertainty through a posterior over model parameters,
our methods adopt a predictive-first perspective based on martingale posteriors.
In particular, we work directly with the one-step-ahead posterior predictive, which we
parameterize with a neural network and update sequentially with incoming observations.
This decouples Bayesian decision-making from parameter-space inference:
we sample from the posterior predictive for decision making,
and update the parameters of the posterior predictive via fast, frequentist Kalman-filter-like
recursions. 
Our algorithms operate in a fully online, replay-free setting, providing principled uncertainty quantification without costly posterior sampling.
Empirically, they achieve competitive performance–speed trade-offs in non-stationary contextual bandits and Bayesian optimization,
offering 10–100 times faster inference than classical Thompson sampling while maintaining comparable or superior decision performance.


Peer reviews:
comments: SUMMARY:
This paper aims to improve the efficiency of online learning by applying the extended Kalman filtering to estimate the latent parameters. The proposed algorithm uses a recursive update of the best linear unbiased predictor and its associated error variance-covariance. Variants of the method that rely on full/low-rank approximation are considered and the tradeoff between accuracy and speed is demonstrated through a set of experiments.

STRENGTHS AND WEAKNESSES:
Strengths:
The paper introduces a novel approach for online learning based on linearization and low-rank approximations of error covariance matrices, with favorable computational efficiency.
It provides a comprehensive evaluation of the proposed methods, and the experimental results seem intuitive.
The proposed framework can be useful for solving contextual bandit problems, which have many real-world applications such as the recommender system.  

Weakness:
While the paper establishes certain theoretical guarantees (Proposition 4.1), it is a little hard to gauge the tightness of the bound and the result does not seem to be very informative of the actual performance of the algorithm. 
It looks like HiLoFi has a relatively high variance among the other methods in the recommender system task.

QUESTIONS:
1. Why do you focus on uncertainty in predictions rather than uncertainty in parameters? How does this approach compare to the traditional TS?
2. Can you provide an intuitive explanation of Figure 1, the in-between uncertainty induced by HiLoFi?
3. It might be helpful to plot the cumulative regret as well, which is a more standard metric of online learning, for the bandit MNIST problem.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- The paper proposes HiLoFi, which is a scalable method for maintaining uncertainty estimates of model parameters in online learning. The method uses a full-rank posterior over the final layer parameters with a low-rank approximation for hidden layers.\n- The method can be broken down into using EKF for iterative updates and uncertainty estimation (which doesn’t necessarily yield a proper distribution on the parameter space), and using a Bayesian approach to construct a valid posterior distribution for the target space.\n- Theoretical guarantees on low-rank approximation error are provided, and experiments are performed on several datasets.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- The paper is clearly written: the methodology is sound, and step-by-step derivations are given to the methods to make them understandable. \n\n- The algorithm lowers the computational burden of online learning—I can imagine it being used in situations where previous methods with GP would simply fail to fit, or BNNs that would take too long to train. \n\nWeaknesses\n\n- I would say that there is insufficient exploration of scalability to deeper or wider networks. Most tested architectures, 3-layer MLPs or shallow CNNs, do not reflect the scale encountered in harder tasks (e.g., protein engineering, molecular design).\n\n- Also, in many real-world online tasks, the bottleneck is not computation time but sample efficiency. The paper does not discuss this tradeoff explicitly (it is quite crucial for many online applications like robotics or experimental design).\n\nQUESTIONS:\n- In Figure 2, it seems that LoLoFi achieves better performance than HiLoFi. Do the authors have any hypothesis on why this happens?\n\n- Intuition: why only separates the last layer (in a 3-layer MLP, for example, it may be insufficient to evaluate the method's performance when more or fewer layers use a full rank error matrix)? Are there ablation experiments for a larger number of layers?\n\n- For proposition 4.1, are there empirical validations for this error bound? Can the rank d_h be dynamically tuned to trade off accuracy and compute?\n\nLIMITATIONS:\nThe ML model architecture choices are modest. This leaves questions about whether HiLoFi scales to models used in domains like protein design, large-scale NLP, or vision.\n\nAs mentioned in the weaknesses section, in many online learning scenarios, especially in scientific domains, samples are expensive but time is not the main constraint. It would be useful to see:\n\n- Whether HiLoFi (and its underlying TS algorithm) offers superior sample efficiency over other acquisition approaches.\n- How trade-offs between low-rank approximation fidelity and sample utility are managed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a hybrid frequentist–Bayesian approach for online learning and Bayesian inference. It uses Kalman filtering to recursively update neural network parameters and their associated uncertainty, and employs Bayesian predictive sampling for decision-making. The key idea is to avoid expensive posterior inference over network parameters by instead focusing on modeling predictive uncertainty directly. The method is fully online and single-pass, requiring no replay buffers or offline retraining. The authors evaluate their approach on three sequential decision-making tasks, spanning contextual bandits and Bayesian optimization, and demonstrate that it achieves a competitive trade-off between speed and accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe combination of frequentist Kalman-style updates with Bayesian predictive sampling in a fully online, replay-buffer-free neural-network framework is novel and well-motivated. This method is also scalable to applications including bandits, Bayesian optimization, and classification. The experiment evaluations are comprehensive.\n\n\nWeaknesses: \n\nThe paper has several issues that limit its clarity and broader accessibility. \n\nIt frequently assumes familiarity with key terminology (e.g., “replay buffer,” “in-different uncertainty”) without proper explanation. For instance, it mentions “replay-buffer approaches for Bayesian optimization” without specifying what they are (likely referring to GPs), nor does it clarify how replay buffers are costly in such methods. \n\nThe sentence “we focus on uncertainty in predictions rather than parameters” appears in the decision-making section without clearly explaining that this predictive uncertainty is used for downstream decisions (e.g., via Thompson Sampling), and why this shift of focus is beneficial in practice. \n\nThe connection between uncertainty quantification and decision-making needs to be made more explicit. Similarly, the purpose of Figure 2—contrasting full-feedback and partial-feedback settings—should be more clearly stated. \n\nAll experiment plots lack sufficient explanation. The term “performance” in Figure 5's caption is confusing as there is both \"time performance\" and \"rank performance\" in the right panel of Figure 5. The left panel of Figure 5 lacks error bars without justification or clarification of what uncertainty is being reported.\n\nQUESTIONS:\nWhat is the benefit of combining frequentist Kalman-style updates with Bayesian predictive sampling?\n\nWhy is the shift from uncertainty in parameters to uncertainty in predictions helpful for decision-making?\n\nWhy is Kalman filtering, especially the extended variant, a natural fit for online neural network training?\n\nWhy is replay buffer an issue in GP?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a class of scalable algorithms for sequential training of neural networks. The method sequentially computes best linear unbiased prediction (frequentist parameter estimate) and construct a pseudo posterior predictive using a Gaussian approximation with a block-diagonal error variance-covariance estimate and proceeds in an online fashion. The method is shown to achieve a good balance between speed and accuracy for a variety of online learning problems.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well written (although the notations are a bit too much) and the extensive experiments demonstrate promise to the proposed methodology. The only concern is novelty (since both sequential estimation by approximate posterior)\n\nQUESTIONS:\n1. If one is not interested in the uncertainty of the parameters, but rather of the predictions, why use a Bayesian model at all? For contextual bandits, example would be UCB-type algorithms. Since you are not constructing the full posterior, but approximating it as a normal by plug-in mean and variance frequentist estimates, what purpose does Bayesian inference serve in the work?\n2. I am a bit worried about the setup. Do you assume the parameters at each time $\\{\\theta_t\\}$ related in any way (as often done in hierarchical models for borrowing strength)? If not, what connects the observations at different time points? In your example of contextual bandit, the underlying $\\theta$ is assumed to be fixed (not depending on time), only the estimate $\\hat{\\theta}_t$ depends on time - since all previous observations (say from same arm) are connected through that parameter. Can the authors clarify the setup here. In the case you assume a linear dynamic evolution (as in Section 3.2), how sensitive is the algorithm to misspecification of this dynamics?\n3. For the bandit experiments, did you compare the results with a usual TS? Or maybe a high-dimensional version if the contexts are high-dimensional?\n4. Using a point estimate (for example MAP) and using it for sequential learning in Bayesian models, through the posterior predictive, has been proposed in the literature - for example (i) Sequential Algorithm for fast fitting of DPM models [Zhang, Nott, Yau, Jasra], (ii) Fast Bayesian Inference in DPM models [Wang, Dunson], (iii) Scalable nonparametric Bayesian learning for dynamic velocity fields [Chakraborty, Guha, Lei, Nguyen] - how does your work differ in essence from these. It might be good to include some of these references.\n5. Proposition 4.1 gives some idea - but can you elaborate on the role of the following $R_t, q_{\\ell,t}, q_{h,t}$ on the overall algorithm? It seems these are hyper parameters - a discussion on the tuning might be good.\n6. Can the method be used in probabilistic models where the observation noise is not additive? Or its variance unknown (related to point 5)\n7. Proposition 4.1 - even when no low-rank structure is utilized, there is no consistent estimation of $\\Sigma$ as $t\\to\\infty$ - can you comment on why this is true?\n8. Can you say something about the estimation quality of the BLUP, characterize $\\Vert \\theta_t - \\hat{\\theta}_t\\Vert$? Or at least the predictive accuracy?\n\nLIMITATIONS:\nSee the questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}