Bayesian Ensembling: Insights from Online Optimization and Empirical Bayes
Authors: Daniel Waxman
Fernando Llorente
Petar Djuric
Venue: Accepted by TMLR ()

Abstract: We revisit the classical problem of Bayesian ensembles and address the challenge of learning optimal combinations of Bayesian models in an online learning setting.  To this end, we reinterpret existing approaches such as Bayesian model averaging (BMA) and Bayesian stacking through a novel empirical Bayes lens, shedding new light on the limitations and pathologies of BMA. Further motivated by insights from online optimization, we propose Online Bayesian Stacking (OBS), a method that optimizes the log-score over predictive distributions to adaptively combine Bayesian models. A key contribution of our work is establishing a novel connection between OBS and portfolio selection, bridging Bayesian ensemble learning with a rich, well-studied theoretical framework that offers efficient algorithms and extensive regret analysis. We further clarify the relationship between OBS and online BMA, showing that they optimize related but distinct cost functions. Through theoretical analysis and empirical evaluation, we identify scenarios where OBS outperforms online BMA and provide principled methods and guidance on when practitioners should prefer one approach over the other.

Published in Transactions on Machine Learning Research (01/2026)
Bayesian Ensembling: Insights from Online Optimization and Empirical Bayes
Daniel Waxman Stony Brook University
dasiso
Fernando Llorente
fllorente &bal.
Brookbaven National Laboratory
Petar M. Djurié
petor. detto.djuuriessttnnsroooocm
Stony Brook University
Reviewed on OpenReview: https: // optp//veee.....w...omo. net/ forum? id= CCv BBn
Abstract
We revisit the classical problem of Bayesian ensembles and address the challenge of learning optimal combinations of Bayesian models in an online learning setting. To this end, we reinterpret existing approaches such as Bayesian model averaging (BMA) and Bayesian stacking through novel empirical Bayes lens, shodding new light on the limitations and pathologies of BMA. Further motivated by insights from online optimization,
propose Online Bayesian Stacking (OBS), a method that optimizes the log-score over predictive distributions to adaptively combine Bayesian models. A key contribution of our work is estab- lishing a novel connection between OBS and portfolio selection, bridging Bayesian ensemble learning with a rich, well-studied theoret framework that offers efficient algorithms and extensive regret analysis. We further clarify the relationship between OBS and online BMA, showing that they optimize related but distinct cost functions. Through theoretical analysis and empirical evaluation, we identify scenarios where OBS outperforms online BMA and provide principled methods and guidance when practitioners should prefer one approach over the other.
1 Introduction
Combining the opinions of multiple models is a pervasive problem in the statistical sciences, with many different names, approaches, and applications. In signal processing. for example, commonly encountered problem is one of sensor fusion, where information reported from several different sensors must be combined to obtain the best possible estimate (Khaleghi et al. 2013). In econometrics, the problem is often known forecast combination, spurred by the seminal work of Bates & Granger (1969). The Bayesian literature often refers to the problem as opinion pooling, whichreceived attention even in the early days of Bayesian statistics [tone 1961 DeGroot 1974 Genest & Zidek 1986). While many of the above approaches originated in the combination of point estimates, the combination of probability distributions has received significant attention since KKliaader et al. 2022).
In machine learning, this issue is commonly known as ensembling, with much recent interest. due to the development of diverse models, architectures, and training modalities (Dietterich) 2000). Many ensembling approaches exist, hinging on different assumptions and asymptotic guarantees. We will restrict our attention to Bayesian machine learning, where the goal is to combine the estimates of K different probabilistic models MI....,M MK- The classical approach in this setting, known as Bayesian model averaging (BMA), is to weigh the estimates of each model according to their marginal likelihood (Hoeting et al., 1999.. When the data were
* Work done Stony Brook University. Current address: Basis Research Institute <danSbasis.ai>

---

Published in Transactions on Machine Learning Research (01/2026)
generated by one of the models , My, this is the "correct" way to combine models from Bayesian perspective and is optimal, in the sense of choosing the correct model with probability 1 in the limit of infinite data (Yao et al. 2018).
More recently, Yao et al. (2018) critically examined BMA in a more typical setting where the data are not generated by any of the candidate models. They instead proposed Bayesion stacking, which optimizes a log-score criterion; asymptotically, Bayesian stacking corresponds to choosing the optimal convez combination of models, juxtaposed with the optimal singular model selected by BMA. Similar discussions and proposals have previously appeared in the econometrics community for forecasting models (Hall & Mitchell 2007 Geweke & Amisano 2012).
In this paper, we further study the online stacking problem, introducing a novel analysis via empirical Bayes before moving to a new analysis of Bayesian stacking in the online setting, We summarize our contributions as follows:
1. We show that Online Buyesian Slacking (OBS) corresponds exactly to the well-studied problem of (universal) online portfolio selection (OPS). This connection allows us to reinterpret Bayesian ensemble learning through the lens of online convex optimization and to leverage efficient, theoretically grounded algorithms (e.g., Exponentiated Gradient and the Online Newton Step).
We show theoretically how one can choose between OBS and online BMA with constant regret. Building on this connection, we discuss how regret bounds from the OPS literature can be applied to OBS.
3. We introducc a simple yet compelling argument via empirical Bayes to explain why BMA collapses and how Bayesian stacking aids in avoiding this common problem.
4. We perform an extensive empirical analysis of OBS using ensembles of state-of-the-art models, includ- ing Gaussian processes, variational Bayesian neural networks, and stochastic volatility forecasting models estimated with sequential Monte Carlo. We show that, all cases tested, OBS significantly outperforms online BMA (O-BMA) and dynamic model averaging (DMA), with additional computa- tional cost that is often negligible in comparison to online wraining/predicee of ML models. We also show that OBS can be particularly beneficial in non-stationary environments, often out purprrrmmiing any set of fixed weights.
The rest of this article is structured follows: in Section we review the problem of Bayesian ensembling, discussing the BMA and stacking approaches. In Section stacking corresponds to the well-known (universal, online) portfolio selection problem from online convex optimization, and discuss corresponding insights that show OBS's relation to O-BMA. In Section 4 we exploit this connection to derive a novel comparison of BMA and Bayesian stacking from an empirical Bayes perspective. Tn Section 5 we carry out a number of realistic experiments on synthesized and real data before finally concluding in Section
show how the online variant of Bayesian
2 Bayesian Ensembles: A Review of Existing Methods
In this section, we review the ideas of BMA and Bayesian stacking Yao et al. 2018 and discuss some related work.
2.1 Bayesian Model Averaging
Bayesian ensembling methods combine a set of Bayesian models for predictive inference. We will focus on methods that create linear mixtures of posterior predictive distributions: given models Mi...., each mapping x E X C Rd to a probability (density) over y E 3 C RT, trained on a dataset D, the task of Bayesian ensembling is thus to find a weight vector w in the (K- -idiinsiioa simlexx 8K which induces a posterior predictive distribution
p ens (y xx, D) = Σ wxpk(y x, ()
(1)
2

---

Published in Transactions on Machine Learning Research (01/2026)
where pb |x, D) is the posterior predictive distribution of model M₁ the dataset D.
The most common method for combining Bayesian models is BMA (Hoeting et al. 1999), which forms the weight for a linear mixture of predictive distributions by simulating posterior probability for model Mr given data D for weighting,
Pr(M& D) (D) Pr(Ms) (A) Pr(M)))
BMA 1
(2)
where ph (D) = p(D[Ma) the evidence of Ma, and Pr(M2) is the prior probability MM.
The appeal of BMA to a Bayesian is immediate: if Pr(M) has support over the data- aata-generating distributions, then the resulting mixture Equation (1) is straightforward application of Bayes' rule and is thus optimal in information-theorcti sense [Zelner 1988..
Much of our work focuses on the online setting, where BMA has several additional advantages. In this setting, we first obtain a new data point X++1, with which we must make prediction using the available data Dz- After making a prediction, the value M1+1 is revealed and the dataset Di+1 = DU (x.+1. this case, we can compute wrF BMA recursively using posterior predictive distributions, which allows us to apply exact BMA at time using the aurrently-availate information: denoting the weights at time Wr,ks with Wo, Pr(Ms), (2) becomes
updated. In
WL.PPs(U441 21)
BMA 1+1
(3)
En Wr,4P%(M++1 1,
Two main problems arise when using BMA: first, estimation via Equation 2 Equation requires access to predictive distributions, either through the marginal likelihood (i.e., the prior predictive) or the posterior predictive. This is often a surmounaabee problem, as marginal likelihoods are usually available in conjugate models, and BMA with approximated marginal likelihoods has also performed well empirically (Gómez-Rubio et al. 2020
Perhaps more importantly, BMA is only optimal in terms of predictive error in the so-called "M-closed" " setting, where the data were generated by one of the models M1...., MMx (Yao et al. 2018 Minka 2000 Indeod, in the limit of infinite data, BMA weights concentrate on the single model that most closoly resembles the data-generating process. We provide a novel empirical Bayes-based analysis of this fact in Section g This may result in arbitrarily poor posterior predictive accuracy of an ensemble using w BMA relative to some other set of weights w*. Additionally, O-BMA can numerically collapse to the "wrong" model, never recovering due numerical underflow (Waxman Djuric 2024
2.2 Bayesian Stacking
Bayesian stacking (Yao et al. 2018) (similarly explored by (Clyde & Iversen 2013 Le & Clarke 2017)
presents an alternative way to derive a weight vector w. In particular, Bayesian stacking finds the optimal weight vector w in the (K- -)-dimensional simplex gK by maximizing some score S(w, D') over separate dataset D':
aig arg max S(w, D'). wESK
(4)
Particular attention is given to the log-score; for the predictive dataset (i.e., holdout or validation set) D' {(x,, Wn the corresponding optimization problem is
w" arg max WESK
log
WkPa(Un Xn, D),
where Pk (yol Xn, D) is the predictive distribution model Mg. This can be recognized as finding the mixture of estimators with the highest predictive likelihood over D'. This, in turn, minimizes the KL divergence
We will typically = I,..., denote datasets that are not processed aequentially/online and l,..., T for their sequential/online counterparts.
3

---

Published in Transactions on Machine Learning Research (01/2026)
between the mixture q x) = =2 "apx(y x, D) and the predictive distribution, ply x), which represents the (unknown) true generating mechanism of (aa et al. 2018).
The optimization problem 5 is convex and provides nice asymptotic guarantees, but "wastes" data by requiring two separate datasets D and D'. Yao et al., (2018) address this by showing the score in (3) is well-approximated by the leave-one-out (T.00) predictive density p(yn | Xx: D-n), where D_n = D) {(xn: Mn)}- For many models, the LOO predictive density can be efficiently estimated with Pareto-smoothed importance sampling (Vehtari et al. 2024), resulting in an efficient optimization problem with single dataset D. The resulting mixture of estimators has empirically shown performance superior to that of other combination methods, leading to several recent applications.
2.3 Bayesian Stacking for Time Series & Online Bayesian Stacking
In the published discussion to Yao et al. (0018), Ferreira (0188) discusses Bayesian stacking for time series,
where forecasting densities naturally serve as predictive distributions, i.c., (5) becomes
T
K
w = arg max WESK
Σ
log
wapn (Me Xx, D1:--),,
(6)
(=
where the first t* data are devoted to (pre-)training each model, and Dia--1 denotes the first / 1- 1 data
points. This has notable similarities to pooling methods described by Hall & Mitchell (2007); Geweke & Amisang (2017), with the exception of the summation starting with index t = 1, reminiscent of the LOO approach. This method was applicd by Geweke & Amisano (2012), where optimal weights were computed quarterly. "Windowed" approaches for autocorrelated time series data, where only the last Tp points are considered, have also been deployed (Jore et al., 2010 Aastveit et al. 2014), but these are suboptimal in the more general case where data may be exchangcable conditioned on X2. Related to windowed approaches is dynamic model averaging, which is O-BMA with forgetting factors (Raltery et al. 2010). To the best of our knowledge, Bayesian stacking and its time series variants have not been applied in the online setting, where weights are estimated as new data become available. We will refer to approaches to the online problem as online Boyeston stacking (OBS).
As will see, OBS is special case of online convex optimization (OCO) (Hzan 2022 from experts" is well-studied. We show that the popular Hedge algorithm FFrend & Schapire 1997 generalizes BMA with learning rate, but optimizes a different loss function from OBS. Our proposed OBS is differentiated from existing approaches in several ways. First,, it directly emulates Bayesian stacking, of recent interest to the Bayesian community. Second, our connection to OCO yiclds an extremely efficient implementation, unlike methods based on data-driven portfolio selection (e.g., BBştüükk al. (2019)) that rely on particle filtering. Finally, our approach more general for online learning, whereas data-driven strategies rely on the autocorrelation of time series data.
where "learning
3 Insights from Online Convex Optimization
As Bayesian stacking is natively framed an optimization problem, it is natural to study OBS from the optimization perspective well. It turns out that our studies here are fruitful: by interpreting posterior predictive values as asset prices, OBS corresponds exactly to classical problem of portfolio selection. An OCO perspective additionally rediscovers a development of Vovk (2001), which shows that BMA is exactly the Hedge algorithm with a specific choice of learning rate. We additionally discuss a hybrid approach meant to detect whether the model class M-open or M-closed (or, more precisely, OBS mixtures will outperform expert selection).
3.1 The Portfolio Selection Problem
We will first review the basic ideas of online portfolio selection (OPS), but our discussion will necessarily be brief; the interested reader is referred to the recent survey of Li Hoi (2014), or the excellent lecture notes of Hazan ) Orabonal (2019).
4

---

Published in Transactions on Machine Learning Research (01/2026)
Problem Statement Consider investment manager oversecing K stocks, aiming to maximize total future wealth. They create the best constantly rebalonced portfolio (BCRP), i.e., at each time step they re-allocate current wealth such that wy is in stock k (classic portfolio selection assumes no transaction fees). Assets are allocated based on price-relative vectors Ft, where Te,k is stock k's relative value increase from day - to The goal is to maximize the (nultiplicative) wealth at time T, or more conveniently, the (additive) log-wealth:
T
II w re e.* log wealthy Σ log (w" WTT. rr).
wealthy =
In the online version of the portfolio selection problem, we seek to find we that minimizes the regret with respect to the optimal weights w",
T RT = Eoog log(w" w" T re)) - - log(w/ rt),
(8)
where the weights w, are determined after observing rt-1-
Although many approaches explicitly model returns as stochastic processes, others allow Tt,k to be chosen arbitrarily, and indeed by an adversary. Algorithms that achieve sublinear regret with respect to any non-negative sequence of returns Ti, are known universal (Orabona 2019 p. 144). We draw analogy between the price-relative vectors ry and the predictive likelihoods p(U x,, D.-1), which make the adversarial case of particular interest to us.
Algorithms The portfolio selection problem has an attractive structure from the optimization perspective: the log wealth is concave function defined over the simplex, a convex set. OPS therefore falls under the (online) convex optimization (OCO) unbrella (Hazan 2022; Orabona 2019 To keep with standard terminology, will introduce each method minimizing the convex loss function given by log wealthy, which is equivalent to maximizing the concave function log wealthr.
The simplest OCO algorithm is online (sub)gradient descent (OGD), a straight.orward extension of the classical gradient descent algorithm to a sequence of losses fı, each a function of some generie quantity belonging to a convex set XC. In our case, O, represents weights Wr, le is the negative log-return, - log w, re), and IC is the simplex g At each time instant, 0, is updated in the direction of the gradient Vl, and then projected back onto the convex set K. (e.g., (Hazan 2022
Algorithm 2.3)).
In many problems, it turns out to be beneficial to consider regularizers that account for the geometry of I.C explicitly, resulting in different update steps. This is an online form of the classical mirror descent (Nemirovsky 1983) When K. is the simplex g (as in OPS), one popular choice is entropic regularization (e.g.. Section 5.4.2)). The resulting algorithm is known as Exponentiated Gradients (EG) (Helmbold
et al
Hazan
2022
et al
1998
which uses the update step
Wi expe(pp-ny wi))
W
w O exp( 7/ V &ell
where O is the elementwise product, and n is learning rate, For the cost function f((w) log w.T,. rt, we have Vwlt= -re/w, w, re, and therefore,
W. exp((mr/ww] re)
W
(10)
Iw, O exp(1r;/w, rz)lla
The EG algorithm is conceptually simple, with relatively good regret bounds whenever the gradient is bounded. However, more modern algorithms, such as the online Newton step (ONS) (Hazan et al. 2007 can provide lighter bounds on the regret.
Of note for applications in non-stationary environments are OCO algorithms designed for such environments. In our experiments, for example, we will utilize discounted ONS (D-ONS) (Yuan & Lamperski 2020), an ONS variant that includes forgetting factor over the second-order information.
5

---

Published in Transactions on Machine Learning Research (01/2026)
Algorithm 1 Online Ensemble Update: OBS with EG or OBS with Soft-Bayes O-BMA.
Input: Data stream
models {M&] initial weights Wo E A learning rate 17.
for = 1 to T do
// Preediction Step
3:
Receive and compule each model's predictive densily p=(y) Xt, Dz-1) for k = I,..., K Form ensemble prediction via Equation [ Output prediction and receive true label y/-
4:
5:
// Update Step
Update model weights using OBS w/ EG 10 or OBS w/ Soft-Bayes or O-BMA
Update dataset: D, D U {(x,, 3t)}-
end for
Market Variability and Soft-Bayes In order to achieve optimal regret bounds with EG or ONS, we must assume that the ratio of the minimum return to maximum return at any time t t- called the market variabilily parameter- - is bounded. In this case, l is 1-exp-concave with bounded gradients, meaning that exp(-(,) w rr is a concave function (Hazan 2022 So long as this assumption holds, the ONS algorithm provides tighter regret bounds than EG.
Moving beyond the assumption of a bounded a the more recent Soft-Bayes proposes weight updates using learning rate 1) E (0, 1) as
(Oseau et al. 2017 Soft-Bayes
Wi+1 W. @
1 -1 (1-
Ft W.
(11)
Orseau et al. (2017) provide an interpretation of Bayts in terms of "slowing down" O-BMA, and the resulting algorithm provides -hht-oof-tho-art regret bounds without the assumption of bounded gradients. We remark that while Soft-Bayes is in some aspects similar to our work, their focus is developing algorithms and theoretical bounds for log-loss mixtures of experts. In the current work, we focus statistical insights, connecting this online log-loss problem to the recently popular Bayesian stacking, and showing empirically that OBS is viable for modern Bayesian machine learning.
3.2 Online Bayesian Stacking is Portfolio Selection
Our core insight in this section is that the utility of (6) becomes when the market gain Tı is defined by the predictive density (b( xx, D.-1). Indeed, our only requirement in a universal portfolio algorithm is that Ti, is nonzero, which is a rather mild and near-univesally satisfied assumption for the predictive distribution of a regression model. Furthermore, using a constant rebalanced portfolio in the regret ( is appropriate, as a constant rebalanced portfolio corresponds to the constant weights used in (offline) Bayesian stacking.
A point of nuance in applying OPS algorithms the existence of a market variability parameter a, requiring pre-determined maximum and minimum predictive densities. The maximum is clear for proper Bayesian models, and minimum may be assumed (e.g., via compact data spaces bounded model error). However, may still be very small, which can be problematic for EG/ONS regret analysis. Bounding regret using subgradient norms can produce tighter guarantees, since non-adversarial predictions from failing models (i.e., Wr.k R 0) are unlikely to suddenly dominate. If such outliers are a concern, Soft-Bayes also provides a-independent regret bounds.
We provide pseudocode for OBS and O-BMA in Algorithm ! The pseudocode underscores the similarity between OBS and O-BMA, replacing single update equation. Other algorithms for OPS may be used by changing Line 6.
3.3 Online Bayesian Model Averaging is the Hedge Algorithm with Learning Rate
Now that we have established an equivalence between OBS and the portfolio selection problem, one may wonder if similar connection holds for O-BMA. Indeed, it is classically known that O-BMA updates are
6

---

Published in Transactions on Machine Learning Research (01/2026)
specific choice of the Hedge algorithm (Freund & Schapire, 1997 which performs expert selection. While this connection is established (Vovk) 2001,, we rederive it as useful narrative tool to motivate O-BMA as a model selection algorithm, which we show clearly using Proposition 3.1
Close inspection reveals that BMA (+1, precisely recovered by EG when the learning rate is = and the loss
is
(((w)) = Σ W log Xx, Dr-1),
(12)
whose gradient is Ww, Vw,,e((w)) log (ye (x,, Dr-1). This suggests that O-BMA may act similarly to an OCO algorithm that minimizes regret with respect to different loss function,
L=
wp Pz x1, Dr-1).
(13)
This reveals key insight: OBS aims for the best post-hoc expert misture, while O-BMA targets the best single expert. The corresponding bound achieves constant regret; in fact, in the typical proof of this regret upper bound, we prove an equality regarding the regret. The following result is therefore known, but is presented a somewhat unorthodox way to allow us to show lower bounds on the regret.
Proposition 3.1. Let the regret of the BMA misture with respect to the best individual model be defined as Rr E. log Ps- (ye xx, Dz-1) E, log (Ex We.pPalle xt, Dt-1)), where hi is the model with the largest marginal likelihood. Then Rr related to an evidence gain in Ma-,
RT = Prg (Pr)/P((Ms>'
(14)
Thus, if Pr(Ma- (D) is bounded below, so is the regret. In particular, Proposition 3.1 applied the following theorem directly, which asserts O-BMA acts as an "optimizer" Equation (13).
Corollary 3.2. Let the regret of the BMA mirture with respect to the best individual model be defined as in Proposition 3.1 If the posterior probability of the optimal model My- exceeds its prior probability, i.e., Pr(Mk. (D) 2 Pr(Me-), then Ry bounded both above and below,
0 S Rr S - log Pr(Me=).
(15)
Thus, under typical scenarios (e.g,, uniform priors on M&), Equation solutions become strong online optimizers of Equation 15p.
15 becomes fairly light, and the
3.4 Regret Analysis
Connecting OBS with OPS makes regret bounds from the OCO literature available. Different choices of algorithms and assumptions on the predictive densitics P, affect the obtainable regret analysis (c.f. |aan Erven et all (202 Table 1)); however, without additional assumptions on p.. the most efficient algorithms typically obtain regret of order O(VT) (such as Soft-Bayes (Orseau et al. (017 If bound on the norm of the gradients may be assumed, ONS can provide regret on the order Ollog T') (with runtime scaling with K²). Variants of simple OCO algorithms may even provide regret with respect to time-varying optimal parameters, such as the D-ONS algorithm (Yuan & Lamperski 2020).
we provide an example that shows that, with mild additional assumptions, we can even recover regret bounds from O-BMA applications, albeit with potentially worse constants this is despite the fact that O-BMA cannot provide comparable bounds when regret is measured against the mixture loss.
We further discuss regret bounds Appendix A Appendix A.3
3.5
A Hybrid Approach
In the case where the M-closed scenario is plausible, but this fact unknown, OBS still achieves vanishing regret with respect to the singular best expert: this follows because the problem of determining optimal

---

Published in Transactions on Machine Learning Research (01/2026)
mixture of densities is harder than the corresponding expert selection problem, which is included special case.
Nevertheless, as determined in Proposition 3.1 the corresponding regret guarantees for O-BMA in the M-closed setting are extremely strong, achieving constant regret as an upper bound. When a model extremely close to the data generating process available, OBS thus potentially suboptimal.
One potential approach to ameliorate this is to consider hybrid method, where O-BMA and OBS mixtures are maintained and further averaged via secondary layer of O-BMA. That is, we consider two sets of weights, The corresponding predictive distribution
w wO-BMA and w along with a secondary set of weights E s?.
is thus
p juybrid(un (ye X1, D1.t-1== =m on Σ t
O-BMA
PR Xe: Dr-1)
+12
Σ
OBS wik
Pk Pk X,, Dr-1)
(16)
DO-BMA x 1)
OBB(i 1x Pt.
The predictive performance of this mixture achieves constant regret with respect to the best model, once again, while maintaining the expressiveness of the corresponding OBS solution. In particular, we may obtain the following bound:
Theorem 3.3. Let the regret of the hybrid minture in Equation (66] for the best individual model be defined as in Proposition 3.1| i.e., RT= =, loggog Pk- (ycl xx, D--1) log (ve xx: Dr-1), where ks is the model with the largest marginal likelihood. Further assume uniform prior weights over v and O-BMA, Then Ry may be bounded
Rr S log K + log
(17)
Proof. From the "outermost" perspective of ensembling (i.e., with v), we have by Proposition 3.1 regret bound with respect to any "outer" expert (i.e., the O-BMA or OBS mixtures):
Σ
Elogr
logp O-BMA (y: xt, Dr-1) Σ log Jogn"y hybrid (ye (x,xx,,))) Dr-1) S log 2.
(18)
Further, from Proposition
3.1 O-BMA achieves a bounded regret
Σ log pk- (ye xt, D,-1)
Σ log O-BMA (yal X,, Di-1) S log K.
(19)
Combining Equation
18 and Equation
19
completes the theorem.
We note by a symmetric argument that if the OBS mixture has a clearly stronger predictive likelihood, the "outer" level of O-BMA will select the OBS mixture as the best expert, similarly incurring a simple additive regret factor.
4 An Empirical Bayes Perspective
An empirical Bayes (EB) perspective (Robbins) 1964h elucidates BMA and Bayesian stacking properties, offering direct arguments for their limitations. Recall that in EB, hyperparameters ab are found by maximizing the marginal likelihood, rather than being marginalized out. We show that. O-BMA and OBS have subtly distinct EB justifications. While we focus notationally on the online setting, similar results hold for batch processing.
Let p(n)l)) be the prior predictive density of y1 (model parameters are integrated out), and p(usly MaslYY:4-1 11:-1111 1b) for i = .,...,t are posterior predictive densities. Then the prequential principle (Dawid 1984) studies a model through a predictive decomposition, choosing up via p(y)al<b) = II, p(yr V 1b). Further selecting db through the optimization problem
J = arg max
p CesplyyNNasa1 \1:-1.
(20)
is termed empirical Bayes or type-II maximum likelihood estimate.
8

---

Published in Transactions on Machine Learning Research (01/2026)
4.1 BMA is Empirical Bayes over an Indicator Variable
It is well-known in the literature that BMA collapses to the model with the highest marginal likelihood when the amount of data increases Yao et al. 2018 but explanations of this fact are not often presented. Using our insights that O-BMA is the Hedge algorithm, i.c., that it minimizes regret with the loss function (33 this fact straightforward to see: w/k should be 1 for the model with the highest marginal likelihood and
for the others. In this sense, we may interpret BMA as performing empirical Bayes over an indicator variable in the corresponding mixture distribution. To see this, we note that (13) can be written as
K
T wk log II
K
C
Px Xx. Dz-1)
Σ
wk logp:((() (Dr),
(21)
and that Σt=1 Wk log (PFPx(P)) (Dr) S maxy log Pæ (Dr). Hence, the optimal weight vector corresponds to wy for all i except for the model k* = arg max; Pk: (Dr) with the highest marginal likelihood, for which wy-
Note also that by Jensen's 's inequality, (21) is a lower bound to log(Ex tepp (Dr)) If we set wx = Pr(MMs,, can interpret BMA performing empirical Bayes
this is the log of the evidence of the BMA model, the discrete prior probabilities of the models.
4.2 OBS is Empirical Bayes Estimation over Mixture Weights
On the other hand, OBS can be seen performing empirical Bayes on the mixture weights themselves. To see this, we interpret w, and the posterior predictive density
p(uayYa--1 w) = Σ waPk (1-111
(22)
is the weighted mixture of the predictive densitics of the individual models. The objective function that is being maximized in stacking in Eq. (6) is
Σ log Σ WAPk (Alu(())
Σ log plulyy w) = log Pstack (Delw))
(23)
Hence, in stacking, we obtain the weights by maximizing the log-evidence of the 'stacking' model, log Petaace where the weights correspond to hyperparameters. Note the difference between this marginal likelihood and the one in BMA, where the marginal likelihood is itself mixture and each component is independent of
5
Experiments
Thus far, we have primarily focused on the theoretical properties of OBS. In this section, we provide empirical evidence that OBS can be beneficial. We consider four main scenarios: illustrative toy example (Section 5.1 variational Bayesian neural networks (Section 5.2), SMC-based stochastic volatility models
(Section 5.3 and dynamic Gaussian processes in non-stationary environments (Section 5.4 We provide details on our experimental setup, baselines, and code in Appendix
5.1 Subset Linear Regression
We revisit classical problem from Breiman 1996), also used by Yao et al. 15-dimensional Gaussian lincar model with weak predictors, as in Yao et al. 20018) Scction 4.2). We consider two scenarios with an ensemble of 15 regression models. In the "Open" setting, the ensemble consists of 15 univariate models (y zs 0gZz), none which match the true process, where performance depends on combining models. In the "Closed" setting, model & uses features ,,,..., ,,, making model 15 the true model. In both scenarios, hyperparameters were pre-trained using empirical Bayes on 1000 points, followed by online meployment of OBS or O-BMA for 5000 points. Further experimental details in Appendix C
(0188) generating i.idd. data from
9

---

Published in Transactions on Machine Learning Research (01/2026)
EG
-1.4
O-BMA SCRP EG
Average PLL
Average
-10
-1.8
-2.0
0-BMA
0
2000
4000
2000
4000
(a) Toy Example: Open (Section 5.1
(b) Toy Example: Closed (Section 5.1
Figure 1: The average prodictive dog-likelihood (higher better) in the toy example. 'EG" is exponentiated gradients, "ONS" the Online Newlon Step, "BCRP" the optimal constant rebalanced portfolio (offline baseline), and "O-BMA" " is O-BMA. Lines denote the median and shaded area represent the 10th to 90th percentiles over 10 trials. The first 100 samples suppressed for readability.
EG
BMA
ONS
BCRP
BMA
EG
ONS
BCRP
g 0.5
S 0.5
0.0 gllue
(a) Toy Example: Open
(b) Toy Example: Closed
Figure The final weights in the "open" and "closed" subset linear regression experiment.
As shown in Figures and 1b OBS variants (EG, ONS) dramatically outperform O-BMA the "open" " setting. In the "closed" " setting, OBS performs nearly as well the theoretically optimal O-BMA, with ONS slightly outperforming EG.
As detailed in the theoretical section, one reason we may expect O-BMA to perform worse than OBS is the weight collapse of BMA. We can empirically validate this property in our toy experiment, with the expectation that O-BMA collapses to single weight, and OBS retains a proper "mix" of models.
In Figure 2a we show the final weights in the "open" subset linear regression experiment for O-BMA, OBS, and the BCRP (i.c., the optimal pooling of Geweke & Amisano (2011), or Bayesian stacking from the prequential principle (Yao el al., 2.18)) ). In Figure 3a we show the evolution of the BMA and OBS weights new data arrive. Finally, in Figure 2b and Figure 3b we show the analogous final weights and the evolution of weights for the "closed" subset linear regression experiment.
These results show promising evidence of our approach: OBS converges to a set of weights very similar to the best retrospective weights within 5000 samples in both the "open" and "closed" variations. OBS with ONS seems to exhibit more the "collapsing" than OBS with EG, but these differences could be due to the hyperparameter choices, which were set the values, r = 10 for EG, and 8 0.8, n7 = 3 10 for ONS. We further observe the collapse of O-BMA, and that OBS can also "collapse" if appropriate.
5.2 Online Variational Inference
We now move to practical application in Bayesian machine learning, applying OBS to online Bayesian neural networks. We use the recently proposed Bayesian online natural gradient (BONG), which optimizes the expected log-likelihood with ouline mirror descent (Jones et al. 2024). Many different variants of BONG and related approaches are tested, unifying them under a common framework; we make online predictions on MNIST (LeCun al., 2010) using the best variant tested, "bong-dlr10-ef __nn. To create ensembles, we train five Bayesian feedforward neural networks with different initializations, We provide more details on the experimental setup in Appendix D
10

---

Published in Transactions on Machine Learning Research (01/2026)
BMA
EG
ONS
BMA
EG
ONS
1.0
0.10
0.2
0.5
0.0
0.0
0.0
0
2000
4000
2000
4000
2000
4000
2000
4000
2000 4000
2000
4000
(a) Toy Example: Open
(b) Toy Example: Closed
Figure 3: The evolution of the weight vector w, as a function in the "open" and "closed" subset. linear regression experiment. Results are shown for single trial due to the noisy nature of the plots. Dots on the right side of plot denote the final weights of the BCRP.
We visualize the resulting predictive log-likelihoods in Figure 4a example, we observe clear improvements in predictive performance using OBS instead of O-BMA and DMA. Furthermore, the specific OCO algorithm used does not seem to matter much, with Soft-Bayes, EG, and ONS all resulting in solutions similar to the BCRP, In Appendix D we additionally visualize the evolution and final weights, which show that BMA collapses single model, whereas OBS converges to the optimal, more balanced weights.
Although less dramatic than in the toy
5.3 Online Forecasting
We compare BMA and OBS variants for forecasting S&P data, which were used in an offline analysis by
Geweke Amisano 2011 2012), ensembling diverse GARCH models. Model parameters are estimated online using sequential Monte Carlo (Doucet et al.) 2001), where the typical "bank of filters" approach is equivalent to BMA. Eight models (2 per class, with priors sampled from uniform hyperpriors) were generated see Appendix for further details.
We visualize the resulting predictive log- bigliiklisoood in Figure we observe clear improvements in the predictive performance of OBS over O-BMA, with similar results obtained by Soft-Bayes, EG, and ONS. As this is real-world financial data, we expect some amount of non-stationarity: indeed, we find that DMA improves upon BMA, but remains inferior to the OBS methods, and that D-ONS performs the best. We show that BMA collapses once more in this example in Appendix F which can be seen in the evolution and final weights.
4b Again,
Table Median reward (pre- dictive log-likelhhood; higher is better) at the final time step in the GARCH experiment with 100 models.
Method
Median Reward
We additionally performed our online forecasting experiments with 25 models from each class, for a total of 100 models. Repeating across random seeds, we obtain the results in Table [ We used an EG learning 10 times larger than that used in previous experiments, guided by the logarithmic increase in the optimal theorctical learning rate the number of models increases. The results show that ONS and EG still outperform BMA, while Soft-Bayer performs worse. We posit that the issues with Soft-Bayes are due to the relatively short time horizon (on the order of 1000 time steps) relative to the number of models we are fundamentally trading off some fast adaptation for robustness when obtaining the gradient bound-free regret bounds of Boft-Bayes.
EG
3.46
BMA ONS
3.44
3.46
Soft-Bayes
3.31
5.4 Online Regression Non-Stationary Environments
A common issue in the ouline setting is non-stationarily, possibly due to covariate shift, or concept drift. To illustrate the effectiveness OBS in this setting, we apply OBS to the dynamic online ensemble of basis expansions (DOEBE) (Waxman & Djurić 2024), which uses O-BMA ensemble several online Gaussian process-based models. The DOEBE employs linear basis approximation GPs with random Fourier features Llázaro-Gredilae et al. 2010 Rahimi & Recht, 2007 and models non-stationary processes by imposing
11

---

Published in Transactions on Machine Learning Research (01/2026)
D-ONS Bayes
BCRP
Bayes
DMA
Average PLL
0-8MA
-1.5
Average
-0.9
-2.0
1500
1750 2000
750
1600 1250
2000
500
1000
(a) Bayesian Neural Network
(b) Forecasting with GARCH
(Section 5.2
(Section 5.3)
Figure 4: The average predictive log-likelihood (higher is better) in the MNIST and lorecasting experiments, respectively. The method descriptions follow those in Figure [ with the addition of Soft- Soft-Bayes. Lines denote the median and shaded area represents the 10th to 90th percentiles over 10 trials. The first 100 samples are suppressed for readability.
Table 2: The average predictive log- likeeiiddoood (median with 10th and 90th percentiles) in the non-stationary experiments; bolded values are best, and underlined values are second-best.
Method Type
Method
Elevators
SARCOS
Kuka #1
CaData
CPU Small
O-BMA
0.59
0.58
0.779:78
0.61
-0.84
0.83
0.33 34
Online Baselines
0.85
0.33
DMA
-0.59
0.58
g.
0.69
-0.81
-.800
0.33
0.81
Offline
BCRP
-0.57
0.57
0.80
0.66
68
- -0.81
1881
-0.81
0.35
0.37
0.58
-0.81
EG
-0.58
0.57
0.788:59
0.56
-0.81
-0.81
0.35
Online
0.58
0.81
Soft-Bayes
-0.58
0.80
0.70
0.81
-0.81
0.34 36
Bayesian
0.58
-0.1
ONS
-0.58
0.80
0.70
0.80
0.80
0.34
Stacking
0.59
0.81
D-ONS
-0.58 0.58
0.58
0.81 0.79
0.82
0.73
-0.80 0.80
0.80
0.350.36
random walk these linear parameters, using variance of.. This parameter is set to default value in Waxman & Djurie (2024), but it is found to be quite important for performance several real-world datasets.
We apply OBS instead of O-BMA to ensemble RFF-GPs with = 10- for k E 10, We use the same datasets as Waxman & Djuric
(0024) (excluding purely synthetic ones); a summary of datasets and experimental details in Appendix F Of note are SARCOS and Kuka #1, both of which are robotics datascts with covariate shift. For SARCOS and Kuka #1 only, we use the smoothed version of EG [Helmbold 10-3 and 10-2; herwisse, EG exhibits severe instabilities.
et all 1998)
with hyperparameterrs
We include numerical results in Table 2 and figures in Appendix We again find OBS to be beneficial. Though EG might be sensitive to outliers and occasionally performs poorly (particularly on SARCOS and Kuka #1), ONS and Soft-Bayes are rather robust and consistently outperform O-BMA. DMA again outperforms O-BMA, and D-ONS performs best on almost every dataset. Interesting future work includes analyzing adaptive regret non-stationary settings.
5.5 Additional Experiments
We include several additional experiments in the appendices. Namely, in Appendix G analysis on the learning rate in OCO algorithms. We come to the general conclusion that the learning rate does not significantly affect the results our experiments for a set a priori sensible values.
we perform sensitivity
Similarly, in Table we include ablation on the forgetting factor hyperparameter in DMA. We find that more aggressive forgetting can help in highly non-stationary environments, but that OBS methods (and, in particular, D-ONS) remain competitive, even without further hyperparameter tuning.
12

---

Published in Transactions on Machine Learning Research (01/2026)
6 Discussion & Conclusions
In this work, we critically study the idca of "Bayesian ensembles," " espocially in the online setting. We focused on two strategies for determining the weights in a linear ensemble: Bayesian model averaging (BMA) and Bayesian stacking (BS). In particular, we introduced and discussed OBS through the prequential principle. Through careful observation, we established a connection between the OBS problem and the OPS problem, allowing us to leverage the rich literature on online convex optimization. Below, we provide some practical guidance and limitations (Section 6.1 outlook (Section 6.2 and brief conclusions in Section 6.3
6.1 Practical Guidance
When to Prefer OBS. Our theory and experiments suggest that OBS is preferable in M-open scenarios, i.e., whenever the true data-generating model is not available, which we believe is typically the case in machine learning. When the problem is known to be M-closed, that. O-BMA is optimal, though OBS, using many different online optimization algorithms, still provides competitive performance. Under anticipated and extreme non-stationarity, DMA with small forgetting factors is usually a viable approach, but otherwise, OBS using OCO algorithms designed for dynamic environments should be preferred.
reiterate previously-known thcoretical results
Choosing Your OCO Algorithm. The theoretical and practical properties of OBS depend on the OCO algorithm used. Based on our experiments and analysis of the relevant theory, we provide the following recommendations: (1) In scenarios with anticipated non-stationarities, D-ONS should be preferred. (2) Tn more general scenarios where the anticipated non-stationarity is modest and gradual, guidance is somewhat more subjective. Based on our empirical experiments, we recommend ONS if the computation is not prohibitive, EG if the computation is extremely prohibitive and extreme non-stationarity is not a concern, and Soft-Bayes otherwise; however, we encourage practitioners to experiment in their particular domains.
6.2 Outlook
We view OBS as providing flexible framework, allowing for exciting rescarch along two different axes.
First, the OBS connection provides an impactful application for new techniques developed in OCO. The connection also motivates a set of potentially interesting assumptions to incorporate into future regret bound analyses. For example, some recent work in OCO has focused on so-called "data-dependent bounds," " which can provide more optimistic regret. bounds depending
the statistics of the incoming data (Tsai et al.j 2023 Putta & Agrawal 2025); understanding the statistics of predictive log-likeishoods in certain online Bayesian lcarning settings could thus be a natural place to develop new bounds. Online learning also serves as natural place to apply research on switching regret (Pasteris et al. 2024
Second, OBS is quite a general framework for ensembling Bayesian models. Though we considered a wide range of models and applications this paper (including variational Bayesian neural networks, GPs via basis expansion approximations, and stochastic volatility models via sequential Monte Carlo), we anticipate many more. For example, our sequential Monte Carlo experiment shows the benefits of OBS over the standard "bank of filters" approach to ensembling particle filters, suggesting novel applications in filtering theory.
It is also interesting to consider connections to other probabilistic machine learning settings, including non- Bayesian online deep learning methods (Valkanas et al. 2025), and applications of distributed or decentralized OCO Yan et al. 2012: Mateos-Nánen & Cortés 2014 to decentralized Bayesian inference methods that rely on O-BMA [Llorente et al. 2025).
6.3 Conclusions
Furthermore, we illustrate how O-BMA optimizes a different loss than OBS leading to a novel empirical Bayesian analysis, and show how O-BMA-based regret bounds can often be adapted to OBS. Finally, we empirically validate our theoretical claims with both synthetic and real datasets, showing that BMA collapses and that the proposed OBS algorithms deliver better performance.
13

---

Published in Transactions on Machine Learning Research (01/2026)
The connection established in this work between OBS and the well-studied problem of portfolio selection bridges optimization and statistics and suggests many interesting future directions. For example, in online or continual learning, one is often concerned with properties under regime changes; a corresponding future direction is to evaluate OBS with algorithms aimed at minimizing dynamic or adaptive regret (Hazan 2022 Chapter 10). Future statistical investigation ions include how to initialize ensembles well and whether the same rules of thumb apply as BMA.
Acknowledgments
This work was supported by the National Science Foundation under Award 2212506.
References
Knut Are Aastveit, Karsten R Gerdrup, Anne Sofie Jore, and Leif Anders Thorsrud. Nowcasting GDP real time: A density combination approach. Journal of Business & Economic Statistics, 32(1):48 68, 2014.
Amit Agarwal, Elad Hazan, Satyen Kale, and Robert E Schapire. Algorithms for portfolio management based on the Newton method. In Proceedings of the 23rd international Conference on Machine Learning, pp.
9- 9-16,22 2006.
Nalan Baştürk, Agnieszka Borowska, Stefano Grassi, Lennart Hoogerheide, and Herman K van Dijk. Forecast density combinations of dynamic models and data driven portfolio strategics. Journal of Econometrics, 210 (1):170 186, 2019.
John M Bates and Clive WJ Granger. The combination of forecasts. Journal of the Operational Research Sociely, 20(4):451 468, 1969.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milnee and Qiao Zhang. JAX: composable transformations of Nuthon+Nummly programs, 2018. URL http://github. com wwwm google/jax
Leo Breiman. Stacked regressions. Machine Learning, 24:49 64, 1996.
Merlise Clyde and Edwin S Iversen. Bayesian model averaging in the M-open framework. In Bayesion Theory and Applications. Oxford University Press, 01 2013. ISBN 9780199695607. doi: 10.1093/acpp
0o0/07880199995577.0000000
A Philip Dawid. Present position and potential developments: Some personal views statistical theory the prequential approach. Journal of the Royal Statistical Society: Series A (General), 147(2):278 290, 1984.
Morris И DeGroot. Reaching a consensus. Journal of the American Statistical association, 69(355):118 121, 1974.
Delve Developers. Delve datasets. https://www.cssttrrotoo.oe www.brrmmoo.eettedeeeeee/aa//a/e
1996.
Thomas G Dietterich. Ensemble methods in machine learning. In International Workshop on Multiple Classifier Systems, pp. 15. Springer, 2000.
Dongshong Ding, Jianjun Yuan, and Mihailo R Jovanović. Discounted online newton method for timo-varying time series prediction. In 2021 American Control Conference (ACC), pp. 1547 1552. IEEE, 2021.
A. Doucet, N. De Freitas, and N. Gordon. An introduction to sequential Monte Carlo methods. Sequential Monte Carlo Melhods in Practice, pp. 3 3-14, 2001.
R. Engle. GARCH 101: The use of /ARCH models in applied econometrics. Journal of Economic Perspeclives, 15(4):157- 168, 2001.
Marco A. R. Ferreira. Contributed discussion to "using stacking to average Bayesian predictive distributions". Bayesian Analysis, 13(3):986 987, 2018.
14

---

Published in Transactions on Machine Learning Research (01/2026)
Yoav Freund and Robert E Schapire. A decision-theoreti generalization of on-line learning and an application to boosting. Journal of Compuler and System Sciences, 55(1):119-139) 1997.
Christian Genest and James V Zidek. Combining probability distributions: A critique and an annotated bibliography. Statistical Science, 1(1):114 135, 1986.
John Geweke and Gianni Amisano. Optimal prediction pools. Journal of Econometrics, 164(11)130-141, 2011. ISSN 0304-4076. doi:
Annals Issue on Forecasting.
John Geweke and Gianni Amisano. Prediction with misspecified models. American Economic Review, 102(3): 482-486, 2012.
Virgilio Gómez-Rubio, Roger S Bivand, and Hávard Rue. Bayesian model averaging with the integrated nested Laplace approximation. Econometrics, 8(2):23, 2020.
Stephen G Hall and James Mitchell. Combining density forecasts. International Journal of Forecosting, 23 (1):1-13, 2007.
Elad Hazan. Introduction to Online Conver Optimization. MIT Press, 2022.
Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning, (9(2):169-199) 192, 2007.
David P Helmbold, Robert E Schapire, Yoram Singer, and Manfred K Warmuth. On-line portfolio selection using multiplicative updates. Mathematical Finance, 8(4):325 347, 1998.
Jennifer A Hoeting, David Madigan, Adrian E Raftery, and Chris T Volinsky. Bayesian model averaging: A tutorial (with comments by M. Clyde, David Draper and E. I. George, and a rejoinder by the authors). Statistical Science, 14(4):382-411 1999.
Matt Jones, Peter G. Chang, and Kevin Patrick Murphy. Bayesian online natural gradient (BONG). In The Thirtt-eightt Annual Conference on Newral Information Processing Systems, 2024. URL https:
//operreview.no
Anne Sofie Jore, James Mitchell, and Shaun P Vahey, Combining forecast densities from VARs with uncertain instabilitios. Journal of Applied Econometrics, 25(4):621-66 2010.
Sham M Kakade and Andrew Ng. Online bounds for Bayesian algorithms. Advances in Neural Information Processing Systems, 17, 2004.
Bahador Khaleghi, Alaa Khamis, Fakhreddine O Karray, and Saiedeh N Razavi. Multisensor data fusion: A review of the state-of-the-art. Information Fusion, 14(1):28 44, 2013.
Günther Koliander, Yousef El-Laham, Petar M Djurié, and Franz Hlawatsch. Fusion of probability density functions. Proceedings of the IEEE, 110(4):404-455 2022.
Miguel Mázaro-Gredille Joaquin Dandela Carl Edward Rasmussen, and Aníbal R Figueiras-Vine Vidal. Sparse spectrum Gaussian process regression. The Journal of Machine Learning Research, 11:1865-1881, 2010.
Tri Le and Bertrand Clarke. A Bayes interpretation of stacking for M-complete and M-open settings. Boyesian Analysis, 12(3):807-88) -29, 2017.
Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. ATT Labs [Onlinej. Available: http://yamm www.annn...nnnn..memm lecun. www.n..ow.eec.eeeoo nom/eráb, hondb//mmstt/// 2, 2010.
Bin Li and Steven CH Hoi. Online portfolio selection: A survey, ACM Computing Surveys (CSUR), 46(3): 1-36, 2014.
15

---

Published in Transactions on Machine Learning Research (01/2026)
Fernando Llorente, Daniel Waxman, and Petar M Djurié. Decentralized online ensembles of gaussian processes for multi-agent systems. In ICASSP 2025-2025 TEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1 5. IEEE, 2025.
Qin Lu, Georgios Karanikolas, and Georgios B Giannakis. Incremental ensemble Gaussian processes. JEEE Transactions on Paltern Analysis and Machine Intelligence, 45(2):1876 1893, 2022.
David Mlateos-únnez and Jorge Cortés. Distributed online convex optimization over jointly connected digraphs. IEEE Transactions on Network Science and Engineering, 1(1):23 37, 2014.
Franziska Meier, Philipp Hennig, and Stefan Schaal. Incremental local Gaussian regression. Advances in Neural Information Processing Systems, 27, 2014.
Thomas Minka. Bayesian model averaging is not model combination. Technical report, MIT Media Lab, 2000. URL https://tminka, .i gitgub.thhh... wwwpauu.io/pppprr/mmnn------- ienta-isnt-ae isnt-mc.pdt
AS Nemirovsky, DB Yudin, and ER Dawson. Problem Complerity and Method Efficiency in Optimization. John Wiley & Sons, Inc New York, 1983.
Objax Developers. Objax, 2020. URL
Francesco Orabona. A modern introduction to online learning. arXiv preprint arXi::1912.13913 2019.
Laurent Orseau, Tor Lattimore, and Shane Legg. Soft-Bayes: Prod for mixtures of experts with log-loss. In International Conference on Algorithmic Learning Theory, pp. 372-399. PMLR, 2017.
R Kelley Pace and Ronald Barry. Sparse spatial autoregressions. Statistics & Probability Letters, 33(3): 291-297, 1997.
Stephen Pasteris, Chris Hicks, Vasilios Mavroudis, and Mark Herbster. Online convex optimisation: The optimal switching regret for all segmentations simultaneously. Advances in Neural Information Processing Systems, 37:78278-78298, 2024.
Sudeep Raja Putta and Shipra Agrawal. Data dependent regret bounds for online portfolio selection with predicted returns. In 36th International Conference on Algorilluiic Learning Theory, 2025.
Adrian Raftery, Miroslav Kárný, and Pavel Ettler. Online prediction under model uncertainty via dynamic model averaging: Application to cold rolling mill. Technometrics, 52(1):52 66, 2010.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. Advances in Neural Information Processing Systems, 20, 2007.
Carl Edward Rasmussen and Christopher K. 1. Williams, Gaussian Processes for Machine Learning. The MIT Press, 11 2005. ISBN 9780262256834. doi: 10.7551 / 2000000..0...00....000.0........
Herbert Robbins. The empirical bayes approach to statistical decision problems. The Annals of Mathematical Statistics, 35(1):1-20, 1964.
Mervyn Stone. The opinion pool. The Annals of Mathematical Statistics, pp. 1339-1342, 1961.
Lus Torgo Regression datasets, 2024. URL http://///........oooo////g/gggs 6tm Accessed 2024-08-15.
dec.
Chung-En Tsai, Ying-Tine Ting Lin, and Yen-Huan Li. Data-dependent bounds for online portfolio selection without Lipschitzness and smoothness. Advances in Neural Information Processing Systems, 36:62764 2023.
Antonios Valkanas, Boris N. Oreshkin, and Mark Coates. MODL: Multilcarner online deep learning. In The 28th International Conference on Artificial Intelligence and Statistics, 2025. URL https: /ttps://openreeviw het:/forumniddslspiiXX.aa.
16

---

Published in Transactions on Machine Learning Research (01/2026)
Tim Van Erven, Dirk Van der Hoeven, Wojciech Kotlowski, and Wouter M Koolen. Open problem: Fast and optimal online portfolio selection. In Conference on Learning Theory, pp. 3864-3869 PMLR, 2020.
Steven Van Vaerenbergh, Miguel Lázaro- Lázaro-Gredilla, and Ignacio Santamaria. Kernel recursive least-squares tracker for time-varying regression. IEEE Transactions Neural Networks and Learning Systems, 23(8): 1313-1326, 2012.
Aki Vehtari, Daniel Simpson, Andrew Gelman, Yuling Yao, and Jonah Gabry, Pareto smoothed importance sampling. Journal of Machine Learning Research, 25(72):1-58, 2024. URL 725/19-556.htmn htmll
Mojmir Vinkler. Universal http::///tttaooooo.aaaraa////aaa 2024. Ac- cessed 2024-08-15.
Volodya Vovk. Competitive on-line statistics. Internalional Statislical Review, 69(2):213-2) 248, 2001.
Daniel Waxman and Petar M. Djurić. Dynamic online ensembles of basis expansions. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL http:///peenneeiieseee/rrrrrrrrr
Feng Yan, Shreyas Sundaram, SVN Vishwanathan, and Yuan Qi. Distributed autonomous online learning: Regrets and intrinsic srivacy-preserving properties. IEEE Transactions on Knowledge and Data Engineering, 25(1))22483 2493, 2012.
Yuling Yao, Aki Vehtari, Daniel Simpson, and Andrew Gelman. Using stacking to average Bayesian predictive distributions (with discussion). Bayesion Analysis, 13(3):917-100 2018.
Jianjun Yuan and Andrew Lamperski. Trading-off static and dynamic regret in online least-squares and beyond. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 6712-67177 2020.
Arnold Zellner. Optimal information processing and Bayes's theorem. The American Stalislician, 42(4): 278 280, 1988.
17

---

Published in Transactions on Machine Learning Research (01/2026)
A More Details on Regret Analysis
One of the main benefits of connecting OBS with OPS is the numerous regret bounds available in the literature. Stating regret bounds for applications of OBS in general not possible, as the available regret bounds depend on both the OCO algorithm and the class of models being ensembled. Below, we first state some general bounds reflecting the state of the art in OPS. We then provide an example of how these bounds may be applied, showing regret analysis for OBS applied to ensemble of approximate Gaussian processos.
A.1 Regret Bounds from Portfolio Selection
Different choices of algorithm and different assumptions the predictive densities P, affect the obtainable regret analysis (c.f. [ann Erven et al. (2020) Table 1)). However, what is generally possible in streaming algorithmms (i.e., those whose runtime is linear in T) without making further assumptions on the loss is regret proportional to (VVT).
By making further assumptions on the loss function and in particular, assuming that market variability parameter a exists we can obtain even sharper bounds, often for simple algorithms. For example, let Gp be a bound the p-norm of the gradients (which follows from the existence of bounded market variability parameter a). Then EG guarantees regret proportional to VT with an appropriate choice of learning rate
(Helmbold et al. 1998 Theorem 4.1). ONS provides even better logarithmic regret (Agarwal et al. 2006 Theorem 1).
In the absence a market variability parameter a, both EG and ONS may be modified with a "smoothing" " analysis, which artificially bounds the gradients, with the resulting FG regret bounds being proportional to 3//4 (Helmbold al.,
Theorem 4.2), and the resulting ONS regret bounds being proportional to provide regret bounds independent of G and without prior knowledge using time-varying learning rates, reducing dependence on T to VT.
1998.
Vol I (Agarwal et al. 2006
Theorem 1). Soft-Bayes
How to apply these bounds to achieve results for learning algorithms is an interesting question and depends on the learning algorithm and the type bound to prove. Below, we provide a simple example that shows OBS may preserve the asymptotic properties of existing regret bounds, even when regret is measured with respect to single expert rather than mixture.
A.2 Regret of Online BMA
We now prove Proposition 3.1
Proposition 3.1. Let the regret of the BMA misture with respect to the best individual model be defined as Elog log Ewerppeee xx, D.-1))
Rr Br==iog
Eog log pa- (vel xx, Dz-1)
(24)
where h. is the best model. Then Ry is related to an evidence gain in M
Pr(M (Dr) R = log Pr(Mi-
(25)
Proof. This follows quickly from the definition of the O-BMA weights Equation Denoting
e (BMA)
log (Ex (M xx, D.-1) and 1(k²) log Pk- (ye xx, Dz-1), note that
w (k)
exp( exp(-1( ((x)))
w expl (x)
Therefore, we arrive at telescoping term,
Σ (MMA)
(1)
w w
Clatt
(k.)
WT kk))
(k-)
exp
w
w
Taking the logarithm and plugging in the closed form values Equation
proves the proposition.
7
18

---

Published in Transactions on Machine Learning Research (01/2026)
A.3 Example: Online Ensembles of Basis Expansions
A recent example of O-BMA being applied in machine learning is in the ensemblling of (approxinate) Gaussian processes (GPs). In particular, Lu et al. (2022) proposes ensembling approximate GPs with O-BMA, with Waxman & Djuric (2024) generalizing the algorithm and bounds to more general linear basis expansions. They both prove regret bounds relative to any expert with fixed parameters, a setting in which BMA is the appropriate optimizer.
It will first be useful to state the following lemma:
Lemma A.1 (Kakade & Ng 2004), Theorem 2.2). Let E(:;3r) denote the negative log-likelihood, and assume it is C² with second derivatives bounded by E R. We concerned with the negative predictive log- bog-likelhhood l of Bayesion linear model with basis expansion o(.): Rd - Rp, using prior p(0) = N o F))
Let XI,..., XT be sequence of impuls such that [xx(x))] is bounded by for all t. Then we have the following bound between the cunulative log-loss of the Bayesian estimator and the log-loss for any fized value θ.:
T
10.12 F
Teo F
Σ l - (x(xx)
0.;vi)) <
log 1
20
In terms of the time horizon, the bound promised by Lemma A.1 is clearly Ollog T) in T. We now formally state and prove the theorem.
Theorem A.2. Let the negative log- lilkelihhhod ((::vr) be C² with second derivatives bounded by R. We then consider an online ensemble of basis functions (Waxman & Djurić, 2024) with basis expansions ((k); Rdx + Ft and priors p(0(k))= is bounded by 1.
= -o 0.0%
(k)? IF,)) for & E (1,.... K}. Purther, assume that ||
We will consider the log-loss of the ensemble at some pre-selected time T, denoted by, and its regret with respect to the performance of any single expert with a fixed parasnelrr O(k) Then (a) using O-BMA, the resulting regret Ollog T) in T; (b) if we further assume that the log-loss is bounded, then the resulting regret remains Olog T)) using OBS with ONS as the optimizer.
Proof. The result (a) is proved directly in Waxman & Djuric 20044 Theorem 1), which is adapted from Lu et al. (2022 Lemma 2). What remains to show is (b). To emphasize the similarity in proving the O-BMA and OBS results, we will present them side by side.
Let 1(x)) denote the log-loss of the kth expert at time The proof then proceeds in two steps: first, bounding the loss of the ensemble estimate to any individual expert, and then applying Lemma A.1 and combining the bounds.
Bounding Ensemble Losses to Experts We proceed by first bounding the ensemble loss to the loss of any individual expert. Beginning with O-BMA, and proceeding identically to Proposition 3.1 Lu et al. 2022
make the observation that
w exp(-(x) (s) exp(- (k))
(k)
(k)
w
Therefore, using initial weights (A) 1/K,
(k) w
(MA)
(k)
wo
1
exp
wi
(R)
Mwi
(k)
Thus, we can bound the regret of O-BMA with any individual expert as
T
Σ
(-BMA) - -1(00 < log M.
t=1
19

---

Published in Transactions on Machine Learning Research (01/2026)
For OBS, we apply the bound of Hazan et al. (007
Theorem 2), yielding
T
(BS (R)
(BS
Σ " (
Olog
t=1
where Big O notation absorbs constants related to M and the maximum gradient norm.
Bounding Expert Loss Now, we bound the cumulative loss Ei= (8) to the loss using the fixed parameter
@!*). This is achieved by applying Lemma A.1 to the kth expert, which results in
Σ
T
(k) - (x))
<
10.11
F& log
1
T Teo
(R)2
E OlogTT)
(1)
2
F5.
20
Combining the two bounds completes the proof.
B Experimental Setup
this appendix, we discuss our experimental setup, including the OCO algorithms and hyperparameters used throughout.
B.1 Experimental Setup
All experiments were conducted using Ubuntu 22.04 with an Intel i9-9900K CPU with 128 GB of RAM and
two NVIDIA Titan RTX GPUs. Code for our methods and experiments available online under an MIT
License. Implementations were Jax/Oni Jab/axaax (Bradbury
al. 2018
Objax Developers) 2020 based
modifying the codes of Waxman & Djuric (2024 and Jones et al.
2024 5
Optimization algorit
implementations were adapted from and tested against the Universal Portfolios library (Vinkler 2024
B.2 Baselines
Best Constantly Rebalanced Portfolio The BCRP is the so-called "static" solution to our stacking problem, equivalent the offline versions of Bayesian stacking presented in Hall & Mitchell (2007): Geweke & Amisand (2011). The resulting weights solution to Equation (6) i..,
w arg log WaPk (3h xx, 11.
(26)
WESK
k=1
In regression problems without concept drift, this forms a reasonable baseline against which to measure regret.
Online Bayesian Model Averaging Our main baseline is O-BMA, which we consider be the standard
online ensembling approach in Bayesian applications. The weights updated Equation 3 i.e.
Wr, (1e+1 X Dt)
BMA 1II
(27)
Σ. (P X D.)
MIT License
http:://ghhuu.coooooogt////////p Apache 2.0 License
MIT License
httee///hhlllaaoorrrrrrayaaaa/ne MIT License
htgp:///hhhhccc///a/////aaaaaaaa MTT License
20

---

Published in Transactions on Machine Learning Research (01/2026)
Dynamic Model Averaging In potentially non-stationary environments, also compare to dynamic model averaging (DMA) (Raftery et al. 2010), which is essentially O-BMA with forgetting factor (0, 1):
(7 D.)
DMA 8 1+1
(28)
>n w7 Pk (Je+1 Xx+1, D,)
While DMA loses some of the nice statistical properties of O-BMA, it can be more robust in non-stationary sccnarios, where the best expert may change over time. For the forgetting factor, we follow Raftery et al. (2010) and use a value of y = 0.99.
B.3 Portfolio Selection Algorithms
We include several OPS algorithms in our comparisons. We provide brief overviews of each and our default. set of hyperparameters, below.
Exponentiated Gradients The EG algorithm (Helmbold et al. 1998 for portfolio selection follows the updates in Equation 10 i.e.,
O exp(mr;/w re)
W [[, @ exp(1rı/w, r.)
(29)
where vy is a learning rate parameter. We choose a default value of 10-2 in our experiments.
Soft-Bayes The Soft- Boft-Bayes algorithm (Orseau 2017) provides several different potential updates, with regret guarantees independent of market variability parameter a. For our purposes, one useful formulation
is the "online" variant of Orscau et al. 2017 Section 6), which updates weights using time-van sime-varying lcarning rate V)
Wt+11 = Wi,k 11 M/ + Th Σn k We.Teek kb
M++1 +
(1-
1 7/t
wo,k-
(30)
The
Squaro-root regret is then guarantoed Orscan et al.
2017
Theorem 10) with the learning rate
7) log K
(31)
2KL
which we use in our experiments.
Online Newton Step In our experiments, we use the form of ONS specialized for the OPS problem in This algorithm requires paramcters n, and keops track quantitics A, and b,,
Agarwal et al. (2006).
defined as
A, =
2 log(w, rz) + Ik;
(32)
1=1
b
1+ v [log(w, rz)].
(33)
Weights are then obtained at each iteration by projecting onto the simplex,
W4+1 1 (6A,
(34)
where the projection operator 1 is defined as
I (v) arg min(w - v) AAT v).
(35)
Additionally, to help with very small market variability parameters, gradients may be artificially "smoothed"
w = (1 n/w + K
1.
(36)
We default values of 8 0.8 and B 17 10-2
21

---

Published in Transactions on Machine Learning Research (01/2026)
Discounted Online Newton Step The D-ONS algorithm (Yuan & Lamperski 2020 Ding al. 2021 | is
similar to the ONS algorithm (as in Hazan et al. information. We use the formulation of Ding et al. (2021), which the authors claim is more numerically stable
(007 ) but it uses forgetting factor for the second-order
than the discounting factor used in Yuan & Lamperski (020)) The D-ONS algorithm depends on parameters
fj and y E 10, and keeps track of a quantity
P+1 7 + + - (log(w,,, rr)].
(37)
Updates are performed as
1
Wr+1 1g
Wi
-
P. v [log(wr -ri)]
(38)
7
We default values of 17 1.0 and 7 0.99.
C Details and Weights for the Subset of Linear Regressors Experiment
In this appendix, provide more details on the experimental setup and results for the subset of the linear regression experiment in Section 5.1
C.1
Experimental Details
Data Generation Following Breiman points i.i.d according to:
1996 and Yao et al. 2018 Section 4.2), we generate 6000 data
x, N55 115, 11),
V/ xx ~ N(x, 0,1).
The ground truth parameters 0 E R are generated such that all 15 components individually weak predictors, and the signal-to-noise rat is 0.8; refer to Yao et al. 20118 Section 4.2) for the precise procedure used to generate 0.
Model Descriptions We consider two ensembles, each comprising K 15 Bayesian linear regression models:
"Open" setting: Model & (k = 1,..., 15) is a univariate regression altempting to learn the function = ORTK, using only the k-th component of x,.
"Closed" setting: Model (k = 1,..., 15) the first k components of xt, i.e., attempting to learn y = Σ5= 0,zØ. Model & = 15 uses all features and corresponds to the true data-generating family.
Standard conjugate priors were used for the parameters 0g (or vectors thereof) in each model.
Hyperparameter Specification For both scenarios, the first 1000 data points were used to set the hyperparameters of the base linear regression models (e.g., prior variances) via empirical Bayes (type-IT maximum likelihood on the marginal likelihood over these 1000 points). The subsequent 5000 points were processed online. This was accomplished by modifying the code of Waxman & Djurid (2024).
The OCO hyperparameters for OBS were set to reasonable default values and were not tuned further:
Exponentiated Gradients (EG): Learning rate = 10
Online Newton Step (ONS): Parameters 8 = 0.8, 17 10-2 2, B 3= 10-2
22

---

Published in Transactions on Machine Learning Research (01/2026)
BMA
EG
ONS
BCRP
Soft-Bayes
1.0
0.5
0.0
Figure 5: The final weights in the online variational inference experiment.
BMA
EG
ONS
Soft-Bayes
1.0
0.5
0.0
0
1000
0
2000
2000
0
2000
Figure 6: The evolution of the weight vector as a function of t in the online variational inference experiment. Results are shown for the last trial. Dots on the right side of plot denote the final weights of the BCRP.
Details and Weights for the Online Variational Inference Experiment
In this appendix, we provide more details on the experimental setup and results for the online variational inference experiment in Section 5.2
D.1 Experimental Details
As mentioned in the main text, use the bong-dlr10-ef lin model from Jones et al. (0024) which had the best performance among the variants tested in their experiments. The BONG performs variational inference using natural gradients, updating variational parameters ub, as
1441 t/ + F= Ba.-os, [log(ly xx, 0.)],
where Fe, is the Fisher information matrix and dv, the variational posterior. The bong-dlri0-ef lin variant uses variational family of Gaussians with low-rank covariance matrices (diagonal + rank 10 matrix) and approximates the predictive likelihood through lincarization using the empirical Fisher information matrix. We use the authors' implementation of BONG
Our experimental setup generally mirrors that of Jones et al. 2024), with the following exceptions: we use a leedforward neural network with layers of width 64 and 32. The prior mean sampled from the default flax initialization, and we form an ensemble over the prior variances of E {10-3 - 10 100 10'). Each "trial" corresponds to different random shuffling of the training data, with the first 2000 data points used for inference.
D.2 Weight Evolutions and Final Weights
Once again, we visualize the resulting weights Figure 5 and Figure G We find remarkable similarity between the final weights in all OCO algorithms tested and the BCRP solution, and that once again, BMA incorrectly collapses to single model.
httpe://gshaul.cc///eeergggggagp MIT License
23

---

Published in Transactions on Machine Learning Research (01/2026)
E Details and Weights for the Forecasting Experiment
In this appendix, we provide more details on the experimental sctup and results for the forecasting experiment in Section 533
E.1
Experimental Details
In the forecasting experiment, we use real data corresponding to the daily returns of the S&PP 500 index from 2015 through 2020, consisting of 1257 unique observations. In econometrics, such data are often modeled using Generalized Autorogressive Conditional Hoteroskedasticity (GARCH) approaches, which provide probabilistic predictions for time series. We refer the reader to Engle (0011 for a more detailed introduction to GARCH models.
Specification of Models In Geweke & Amisano (001 2012 the application of GARCH models stock index data is also used to test the ensembling of probabilistic models. In GARCH, the conditional variance of the process is modeled as an autoregrossive process depending on the lagged values of the conditional variance and the squared "innovations." For instance, the GARCH(1, 1) model with Gaussian innovations is given by:
Yt = Jec. ~ 10,,1 0f = ao + a + Bof.
(39)
(40)
where e, are the innovations and o, is the condit ional variance of the process. For brevity, we only discuss the GARCH(I,1) with Gaussian innovations. The variants differ in the assumed process for the conditional variance.
We assign truncated Gaussian prior densities to 0. = (oo, on Bz), and are interested in sequential estimation of the posterior distribution p(0,lgha)) Sequential Monte Carlo (SMC) algorithms are very well suited for this task Doucet et al. (2001). An SMC algorithm recursively computes particle approximation of the posterior,
p(Ollun:))
Pe ( Σ
5
0₂),
(41)
where 0 and p denote, respectively, the particles and weights. After the arrival of yt++ the weights are recomputed, and the particles are propagated to form the particle approximation of the posterior at + 1. Using this particle approximation, we can obtain an approximation of the posterior predictive density,
p(y++1l11)) Rs Σ 2e ply++ W.r. 022))
(42)
where off,1 only differs from 000) in the component (6) which is obtained by substituting the previous set of particles in Eq. (40).
We create ensembles using several different variants of the GARCH model, namely GARCHi (Student-t. innovations), GARCHNormal (Normal innovations), GJR-GARCHNornal (GJR-GARCH with Normal innovations), and EGARCHNormal (EGARCH with Normal innovations) Engle (2001)
Inferenee and Hyperparameter Specification With each variant, we sample two sets of hyperparameters and perform online posterior inference using an SMC algorithm with 1000 particles and Markov Chain Monte Carlo (MCMC) rejuvenation steps. At each iteration, the ensemble weights are used to evaluate,
8
Wk.t PP 222))..
RS
(43)
where Wk,r is the weight of model obtained at time using the approaches discussed in the paper. We ran 10 independent. simulations of this experiment, considering 10 different random seeds.
24

---

Published in Transactions on Machine Learning Research (01/2026)
E.2
Weight Evolutions and Final Weights
We visualize the resulting weights in Figure 7 and Figure
BMA
EG
ONS
BCRP
Soft-Bayes
1.0
0.5
0.0
Figure 7: The final weights in the lorecasting experiment.
BMA
EG
ONS
Soft-Bayes
1.0
0.5
0.0
1000
1000
1000
1000
Figure 8: The evolution of the weight vector w, as function of t in the forecasting experiment. Results are shown for the last trial. Dots on the right side of a plot denote the final weights of the BCRP.
F Details and Weights for the Non-Stationary Environment Experiment
In our experiment in non-stationary environments, we use the real or semi-real datasets in Waxman & Djurić (0044). This comprises the Elevators dataset (Torgo 2024), which originates from tuning elevators on an aircraft; the SARCOS dataset (Rasmussen & Williams 2005), which is simulated data corresponding to an inverse kinematics problem robotic arm; the Kuka #1 dataset. (Meier et al. 2014), which is real data from task similar to SARCOS; CaData (Pace & Barry 1997] which has housing data, and CPU Small (Delve Developers) 1996), which includes various performance propertics in a databaso of CPUs.
As mentioned in the main text, the models considered all belong to the dynamic online ensembles of basis expansions (DOEBE) family, which perform online GP regression using Kalman filtering and the random Fourier feature approximation Lázaro-Grediila, et al. (2010). Waxman Djurid (2024) find that adding a random walk to the model parameters is important in capturing "dynamic" (i.e., nonstationary) behavior when applying approximate GPs to several real-world datasets, which be related to "back-to-prior forgetting" (Van Vaerenbergh et al. 2012).
We create ensembles using an RFF GP with 100 features, trained on the first 1000 data points using the marginal likelihood. We then use different values of the random walk scale (aew E {10-4, 10-3, 10-1, 100)) and ensemble them using O-BMA and various OBS algorithms.
In the interest of space, we only report the weight evolutions for SARCOS, Kuka #1, and Elevators in Figure 10 The other dataset results are qualitatively similar to Elevators.
F.1 Cumulative Reward Plots
We show the cumulative reward plots (i.e., the average PL. function of 1) in Figure
25

---

Published in Transactions on Machine Learning Research (01/2026)
-0.6
D-ONS Bay-s
D-ONS BCRP
1.5
Average PLL
RMA
Average PLL
0.8-
ES
SSn-Bayes 0-BMA
Aver age PLL
1.0
-0.8
0.6
Soft-Bayes
0.90
ONS
-0.
MM
0.5
EG
5.0-10"
1.0-10" 15-10'
1.0-30"2.0-33 200000010"3000 10000000000000000000000000000000
5.0 10t 1.0-10"
1.5- 10 2.0-10*
(a) Elevators
(b) SARCOS
(c) Kuka #1
BCRP
2Ng coft-Bayes 0-8MA
Average PLLL
-0.6
-0.03
Average
-0.7
D-ONS RMA
0.25
32
Say-Rayes
0.1
5.0-107
3.0-30"
305 2.--00
2.0
8.0
(d) CaData
(e) CPU Small
Figure 9: The average predictive log- boglliklihhod (higher is better) in the non-stationary experiments. Lines denote the median and shaded area represents the 10th to 90th percentiles over 10 trials. The first 250 samples are suppressed for readability.
G Sensitivity to the Learning Rate
The learning rate marks an important hyperparameter in online convex optimization, with regret bounds often being quite sensitive to the learning rate. In this section, we investigate the practical effects of the learning rate parameter in EG and ONS.
For the experiment using the BONG (Section 5.2 & Appendix D) we performed additional experiments using differing hyperparameters in OCO algorithms. Specifically, we vary 7 in EG and 3 in ONS for values 10 and & E (0, 1, 2, 3, 4}. Results can be found in Table 3 We find that the results are not particularly sensitive to the learning rate, with all results except for EG with a high learning rale (10") and ONS with small learning rate (10- 4) outperforming O-BMA.
Results are similar in the forecasting example (Section 5.3 & Appendix E. where the results in Table 4
suggest largely comparable performance across learning rate values. The performance of EG degrades at very large or very small learning rates, but ONS seems fairly stable in this setting.
H Dynamic Model Averaging With Differing Forgetting Factors
In experiments in Section 5.4 used default "forgetting factor" for DMA, recommended in Raftery et al. (2010). This is similar to our other experiments, in which we leave OBS hyperparameters at reasonable defaults and do not otherwise tune them. Nevertheless, to further understand the impacts of this hyperparameter, we repeated our experiments with more aggressive and more conservative values of the forgetting factor. We report results in Table 5
Overall, find that DMA with extremely low forgetting factors performs well in highly nonstationary environments (Kuka #1, CaData), but it is still comparable to D-ONS, which has more robust guarantees regarding predictive performance. Moreover, this is without luning the lyperparameters of D-ONS. In other datasets tested, OBS methods are still superior to DMA (CPU Small, SARCOS, Elevators). We additionally remark that DMA has much similarity with a sliding window approach and loses many of the statistical properties of O-BMA; on the other hand, adaptive odaptive/dynamic dynamicc OCO algorithuss D-ONS still come with robust. theoretical guarantees.
26

---

Published in Transactions on Machine Learning Research (01/2026)
BMA
EG
ONS
Soft-Bayes
1.0
0.5
0.0
(a) Elevators
BMA
EG
ONS
Soft-
1.0
0.5
0.0
(b) SARCOS
BMA
EG
ONS
Soft-Bayes
1.0
0.5
0.0
(c) Kuka #1
The evolution the weight vector we as a function of t in the non-stationary environment experiment the Flevators, SARCOS, and Kuka #1 datasets. Results are shown for the first trial. Dots on the right side of a plot denote the final weights the BCRP.
Figure 10:
Table 3: Comparison of the average predictive log likelihood value (higher is better) for the BONG experiment (Section 5.2 while varying the learning rates of EG and ONS. Results are reported as the median + standard deviation across five trials.
Method
Learning Rate
Average PLL + Std. Dev.
EG (17)
10"
- 0.04
10-1
0..850 = 0.02
10-2
-0.850 = 0.02
10-3
-0.862 = 0.02
10
-0.865 = 0.02
ONS (8)
10
-0.878 0.03
10-1
-0.859 = 0.03
10
-0.860 = 0.02
10-3
-0.900 = 0.03
10-4
-0.948 + 0.03
Soft-Bayes
-0.847 + 0.02
O-BMA BCRP
-0.932 + 0.02
-0.843 + 0.02
27

---

Published in Transactions on Machine Learning Research (01/2026)
Table 4: Comparison of the average predictive log likelihood value (higher is better) for the GARCH experiment. (Section 5.3) while varying the learning rates of FG and ONS. Results are reported as the median + standard deviation across ten trials.
Method
Learning Rate
Average PLL Std. Dev.
EG
10"
2.776 = 0.23
10-1
3.188 + 0.13
10-2
3.249 = 0.13
10
2.978 0.14
10-4
2.858 0.17
ONS
10
3.256 + 0.12
10-1
3.330 = 0.13
10-2
3.300 = 0.13
10-3
3.229 = 0.10
10-4
3.139 0.12
Soft-Bayes
3.296 + 0.13
O-BMA BCRP
3.040 0.24
3.296 0.12
Table 5: Comparison of mothods on non-stationary datasets (c.f. Table 2 forgetting factors.
including DMA with additional
Method
Elevators
SARCOS
Kuka #1
CaData
CPU Small
O-BMA
-0.59
0.77
0.61
-0.84
0.33
DMA
-0.59
0.79
0.69
-0.81
0.33
DMA-0.9
-0.60
0.80
0.75
-0.80
0.33
DMA-0.95
-0.59
0.80
0.73
-0.80
0.33
EG
-0.58
0.78
0.56
-0.81
0.35
Soft-Bayes
-0.58
0.80
0.70
-0.81
0.34
BCRP
-0.57
0.80
0.66
-0.81
0.35
ONS
-0.58
0.80
0.70
-0.80
0.34
D-ONS
-0.58
0.81
0.73
-0.80
0.35
28


Reviews:
[{"review_id": "i1kSnA9LeM", "reviewer": "Reviewer_eozu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors examine the problem of Bayesian ensembles in an online learning setting, where the goal is to select an optimal combination of known Bayesian prediction models to approximate the true data generator. Known methods for Bayesian ensembles include Bayesian model averaging (BMA) and its well-studied online extension, as well as Bayesian stacking (BS) which have showed improved performance when the true data generator is not derived from one of the Bayesian options.\n\nThis submission's main contribution is a novel studying of BS's extension into the online learning model (OBS), as well as the delicate observation that OBS corresponds to the well-studied online convex optimization problem, online portfolio selection (OPS). This observation allows the authors to connect established regret bounds and analytical methods from online convex optimization to the problem of Bayesian ensembles. \n\nFrom this lens a novel theoretical analysis is also provided, explaining how BMA fails when faced with assumptions that BS can overcome. Empirical data is provided, illustrating the improved performance obtained by OBS over other ensembling methods under these assumptions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The submission makes mostly theoretical claims that is supported by proof and intuitions. I am especially fond of the extensive explanations given in the paper for the significance and novelty of their results - the introductory sections paint a clear picture of the prior works and background knowledge necessary for understanding the main contribution of this paper, that (in my opinion) is the connection between this machine learning problem of Bayesian ensembles and convex optimization.", "audience_yn": "Yes", "audience_explanation": "Establishing connections from machine learning problems to classical theoretical problems such as online convex optimization is a very intriguing direction for research and can easily inspire future explorations along this line of thought, as outlined by the author's concluding sections. From a personal perspective I think the field of machine learning would benefit from more theoretically-backed results such as the ones presented in this paper.", "requested_changes": "- Page 3, paragraph 2: \"In this case, **we** can compute...\" where the \"we\" is missing.", "broader_impact_concerns": "Impact statements are not necessary for this submission.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "PEm1YT0TK5", "reviewer": "Reviewer_8MRG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors study the ensembling, via linear combination, of bayesian supervised learning models in an online setting, i.e., where models are updated after each new labelled data point is ingested. The authors present Online Bayesian Stacking (OBS) , where the combination of models in the ensemble is chosen to optimize a log-predictive score. OBS is contrasted with O-BMA (online bayesian model averaging), in which individual models in the ensemble are weighted in proportion to the evidence that the model generated the data. A correspondence between OBS and online portfolio selection (OPS) from finance theory is demonstrated. O-BMA and OBS are also shown to be versions of empirical Bayes w.r.t. indicator variables and mixture weightes, respectively. Various algorithmic subtypes of OBS (exponentiated gradients, soft bayes, etc) are studies. Superior results for OBS as compared to O-BMA are obtained on a toy problem, an online version of MNIST, and S+P forecasting problem and non-stationary ensembling of gaussian processes. A rule of thumb is proposed where O-BMA may be preferable in the \"closed\" setting where the true data-generating model is among the models in the ensemble, but OBS performs better in the (much more common) \"open\" scenario where the true model is not in the ensemble.", "claims_and_evidence_yn": "Yes", "claims_explanation": "I think this paper may be able to get to the point where I could say 'yes' with some revisions, but I don't think it's there yet. \n\nThe authors mostly use single, default settings for hyperparameters in their experiments. I would like to see some comparisons where each method has undergone a hyperparameter optimization (or at least, would like to hear an explanation for why that's not feasible or appropriate). \n\nI would also like more detail about how to interpret and use the distinction between the 'open' and 'closed' setting when picking OBS vs. O-BMA. Common sense would dictate that there is actually a continuum here and I would presume that O-BMA could still be preferable if something very, very close to the ground truth model is available. E.g., if a model in the ensemble could achieve r-squared of 0.999, I would presume that O-BMA could still beat OBS. If that's not the case, I would like to hear why. It's hard to believe that anything even very slightly less that a perfectly closed setting means that OBS wins. I would also like to hear more about how to assess whether we are in the open or closed setting.   How does one distinguish between being in the open setting vs. just dealing with a problem with lots of inherent noise in the outcome variable? If the perfect model could only get r-squared 0.5 because of noise, how are you going to distinguish, in a practical way, between open and closed ?\n\n**Update after rebuttal**\n\nIn light of the open vs. closed discussions and the pointer to the hyperparameter experiments, I have changed my 'claims' answer to 'Yes'.", "audience_yn": "Yes", "audience_explanation": "I don't think the audience for this paper is very broad, but it's broad enough to merit TMLR's consideration.", "requested_changes": "As indicated above, I would like to see experiments with hyperparameter optimization and I would like much more elaboration on open vs. closed (whether or not it's a continuum where O-BMA still wins for nearly closed and also how to detect open vs. closed). \n\nSome suggested edits/typos:\n\n2nd paragraph page 3 “in this case, can compute” -> \"in this case, we can compute\"\n\nIn section 2.2, I would note that the second set is a 'validation set' or 'holdout set'  so that the terminology is more familiar to ML researchers. \n\nI'm not sure what 'contributed discussion' means in section 2.3. Maybe re-write that.\n\nOn page 5, in the 'simplest OCO' paragraph, I would not use the variable x, since it has a different meaning elsewhere in the paper. I would only refer to the w instead of explaining that x corresponds to w.", "broader_impact_concerns": "No concerns here.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "BtT8T1TTV8", "reviewer": "Reviewer_xoWH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies combining Bayesian predictive models in an online setting by\ncasting the stacking objective as a portfolio-selection problem. Building on\nonline convex optimization, it proposes Online Bayesian Stacking (OBS) with\npractical updates (e.g., Exponentiated Gradient, Soft-Bayes, Online Newton\nStep), provides an empirical Bayes interpretation explaining why BMA tends to\ncollapse to a single model, and leverages regret bounds from the portfolio\nliterature. Numerical experiments are carried out to show that OBS\noutperforms online BMA and dynamic model averaging.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The portfolio selection problem is a very general framework and it seems\nstraightforward to apply it to model averaging and selection problems.", "audience_yn": "Yes", "audience_explanation": "Obtaining the optimal weights for Bayesian model averaging is challenging, and\neffective methods are needed. This paper proposes to treat the problem as\nportfolio selection to leverage existing algorithms.", "requested_changes": "Putting model averaging weights optimization in the framework of\nportfolio selection seems straightforward. Please explain the novelty or\nchallenge of this approach.\n\nA key reason why the optimal weights are challenging to obtain is that they sum\nto one and are non-negative. This constraint makes the optimization problem hard\nto solve. But I don't see how the portfolio selection framework helps with\nthis. Please explain.\n\nWhy $w$ is $K-1$ dimensional simplex? Shouldn't it be $K$ dimensional simplex?\n\nThere is a large body of literature on determining the weights of model\naveraging, both in the Bayesian and frequentist settings. They are mostly\nignored by the authors. Please discuss the related literature and how the\nproposed method compares with them.\n\nProofread the paper for consistency in notations and terminologies. For example,\nit refers Proposition 3.1 as \"the following lemma\" and Corollary 3.2 as \"the\nfollowing theorem\".\n\nWhat do you mean by \"model parameters are integrated out\" in pgae 7 above (16)?\n\nIt is difficult to distinguish different methods in the figures on gray scale.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)