Input-Adaptive Bayesian Model Averaging

## Abstract

Abstract This paper studies prediction with multiple candidate models, where the goal is to combine their outputs. This task is especially challenging in heterogeneous settings, where different models may be better suited to different inputs.
We propose input adaptive Bayesian Model Averaging (IA-BMA), a Bayesian method that assigns model weights conditional on the input.
IA-BMA employs an input adaptive prior, and yields a posterior distribution that adapts to each prediction, which we estimate with amortized variational inference.
We derive formal guarantees for its performance, relative to any single predictor selected per input. We evaluate IABMA across regression and classification tasks, studying data from personalized cancer treatment, credit-card fraud detection, and UCI datasets. IA-BMA consistently delivers more accurate and better-calibrated predictions than both non-adaptive baselines and existing adaptive methods.

## 1 Introduction

Many applications require *adaptive predictions*. In personalized medicine, different patients respond differently to the same treatment (Mahajan et al., 2023); in fairness-sensitive domains, predictions need to adapt to subpopulations (Wang et al., 2019; Grother et al., 2019); and in fraud detection, behavioral data is often heteroskedastic and varies substantially across inputs (Varmedja et al., 2019).

When the data is complex, selecting a single model that performs well across all inputs is challenging. This motivates *model averaging* (MA), which produces an *ensemble* of models.
This idea dates back at least to the 1960s (see, e.g., (Clemen, 1989) for a historical perspective).

We denote data points by $x\in\mathcal{X}$, labels by $y\in\mathcal{Y}$, and the space of probability distributions on labels by $\mathcal{P}(\mathcal{Y})$. MA combines the predictive distributions of $m$ models
$\{f_{j}:\mathcal{X}\to\mathcal{P}(\mathcal{Y})\}_{j=1}^{m}$ into a weighted ensemble, $p_{\alpha}(y\mid x)\coloneqq\sum_{j=1}^{m}\alpha_{j}f_{j}(y\mid x)$, with weights $\alpha_{j}>0$ (often constrained to sum to one). MA accounts for the possibility that multiple models can provide plausible explanations of the data.

In classical MA, the same weights $\alpha_{1},\dots,\alpha_{m}$ are used for all inputs $x$. But in practice, different values of the input $x$ might call for different predictive models. This motivates *adaptive averaging*, where the weights $\alpha_{j}$ depend on $x$:

$$ $\alpha:\mathcal{X}\to\Delta^{m-1},\qquad x\mapsto\alpha(x)=(\alpha_{1}(x),\dots,\alpha_{m}(x)).$ (1) $$

The result is an adaptive weighted prediction,

$$ $p_{\alpha}(y\mid x)\coloneqq\sum_{j=1}^{m}\alpha_{j}(x)f_{j}(y\mid x).$ (2) $$

This model is also known as a *mixture of experts* (Jacobs et al., 1991; Jordan & Jacobs, 1994), where the adaptive weights $\alpha_{j}(x)$ are fit to maximize the predictive log likelihood of the data.

In this paper, we take a Bayesian perspective. We assume that the set of predictors $\mathcal{F}\coloneqq\{f_{1},\dots,f_{m}\}$ is fixed, and model the selection of a predictor as a random process. Our model constructs a random selector $g:\mathcal{X}\to\{e_{1},\dots,e_{m}\}$ where $\{e_{j}\}_{j=1}^{m}$ denote $m$ indicator vectors, i.e., $g(x)=e_{j}$ selects predictor $f_{j}$. Moreover, the prior on $g$ itself depends on the inputs $x$. Therefore, in our model, adaptivity arises not only from the variability of $g(x)$ across inputs, but also from allowing its prior to vary with $x$.

Under this model, MA is a natural consequence of the posterior predictive distribution. Consider a dataset $\mathcal{D}\coloneqq\{x_{i},y_{i}\}_{i=1}^{n}$. The posterior predictive distribution for a new input $x$ is

$$ $\displaystyle p(y\mid x,\mathcal{D})=\sum_{j=1}^{m}f_{j}(y\mid x)\,p(g(x)=e_{j}\mid x,\mathcal{D}),$ (3) $$

where $p(g(x)=e_{j}\mid x,\mathcal{D})$ is a *data dependent posterior* that incorporates both training inputs and labels. Eq. [3](https://arxiv.org/html/2510.22054v1#S1.E3) is an ensemble of candidate models, with weights $\alpha_{j}(x)$ equal to the posterior over $g$:

$$ $\alpha_{j}(x)=p(g(x)=e_{j}\mid x,\mathcal{D}).$ (4) $$

Unlike maximum likelihood approaches to MoE, this posterior captures the uncertainty over which predictor is most plausible for each input $x$.

Below, we first analyze the theoretical advantages of this adaptive Bayesian model averaging framework and derive finite-sample guarantees that compare its performance to that of any single predictor selected per input (Section [2.1](https://arxiv.org/html/2510.22054v1#S2.SS1)).
We then develop input adaptive Bayesian Model Averaging (IA-BMA), by
(i) constructing an input adaptive prior, following Slavutsky & Blei (2025), and
(ii) employing amortized variational inference to approximate the posterior (Section [3](https://arxiv.org/html/2510.22054v1#S3)).
We evaluate IA-BMA across regression and classification benchmarks (Section [4](https://arxiv.org/html/2510.22054v1#S4)), and show that IA-BMA achieves substantial gains in both accuracy and calibration compared to existing adaptive, and non-adaptive strategies.

### 1.1 Related Work

MA is regarded as the machine learning analogue of the “Condorcet’s jury” theorem (Mennis, 2006), leveraging the “wisdom of the crowd” to mitigate the inherent uncertainty in model selection.
Thus, MA is often used when there are alternative, potentially overlapping hypotheses and no clear justification for selecting a single preferred model. Applications include ecological research (Wintle et al., 2003; Thuiller, 2004; Richards, 2005; Dormann et al., 2008; Lauzeral et al., 2015; Zheng et al., 2024) and medicine (Jiang et al., 2021; Nanglia et al., 2022; Mahajan et al., 2023). More broadly, MA has been adopted in a wide range of machine learning tasks (e.g., Fernández-Delgado et al. (2014), Rokach (2010)).

As a form of model combination, MA is closely related to other ensemble techniques such as bagging (Breiman, 1996) and boosting (Freund, 1995). It is a variant of stacking procedure (Wolpert, 1992), in which outputs of base learners are combined to produce the final prediction.

MA has been shown to reduce prediction errors beyond those of the best individual component model (Dormann et al., 2018; Peng & Yang, 2022) and to mitigate overfitting (Dietterich et al., 2002; Polikar, 2006). In recent years, extensive surveys have reviewed MA (Kulkarni & Sinha, 2013; Woźniak et al., 2014; Gomes et al., 2017; González et al., 2020; Sagi & Rokach, 2018; Wu & Levinson, 2021), with some focusing specifically on decision trees (Rokach, 2016) or neural networks (Ganaie et al., 2022).

A Bayesian method for MA was introduced by Waterhouse et al. (1995), who place a prior directly on the averaging weights. In contrast, we reinterpret MA as a problem of random model selection, leading to *dynamic* model selection in which the choice of model adapts to the specific input.
Earlier work on dynamic model selection includes Cao et al. (1995); Giacinto & Roli (1999); Gunes et al. (2003); Didaci et al. (2005); Didaci & Giacinto (2004). However, these approaches focus on selecting a single model for each instance, rather than assigning instance-specific weights to average predictions across multiple models.

##### Input adaptive model averaging methods:

Few methods assign input-dependent weights. These date back to Mixture of Experts (MoE) (Jacobs et al., 1991), where a gating network maps the input $x$ to weights $\alpha_{j}(x)$, estimated by maxmizing the induced likelihood.

Rasmussen & Ghahramani (2001) extended this framework by using Gaussian Processes (GPs) as base models, providing nonparametric flexibility. They adopt a Bayesian perspective with a Dirichlet Process (DP) prior, yielding an infinite mixture. However, here weights and base models are learned jointly, and thus only a single family of base predictors is considered.

Although these methods often outperform standard model averaging, maximum-likelihood–based assignment tends to concentrate probability mass on the predictor that is most confident about the observed outcome $y$, frequently resulting in overconfident predictions (Freund & Schapire, 1997; Guo et al., 2017).
Several approaches proposed alternative strategies for weight assignment.

Woods et al. (1997) proposed a dynamic scheme based on local accuracy estimates. For a test input $x$, its neighborhood is identified (typically via $k$-nearest neighbors), and each classifier’s performance in this region is summarized as a local accuracy score. The classifier with the highest score is then selected to predict $x$.

Similarly, Chan & van der Schaar (2022) proposed an approach that assigns higher weight to models whose training domains better cover a test instance. Inputs are mapped into a learned low-dimensional space where models with similar predictions are closer together, and weights are set via kernel density estimation. Unlike Woods et al. (1997), where similarity is predefined, here it is learned from data. Motivated by Tenzer et al. (2022), the method assumes that models making random errors on an input are unlikely to agree.

Perhaps most relevant to our work is Bayesian hierarchical stacking (BHS) (Yao et al., 2022), which places priors on logit weights, and models them with hierarchical low-rank linear functions. The parameters are then estimated by maximizing the expected log predictive density.

Thus, prior work on adaptive model averaging has focused predominantly on methods targeting frequentist objectives, with relatively few Bayesian formulations.
In contrast to previous approaches, our model assumes a fully Bayesian setting in which the selector itself is random and, crucially, is defined locally relative to each input $x$. This yields an input-dependent prior $p(g\mid x)$ rather than a global prior $p(g)$. In turn, this prior induces an adaptive posterior that corresponds exactly to the Bayes-optimal weights, providing a principled approach for adaptive model averaging.

## 2 Probabilistic formulation of adaptive model averaging

We cast adaptive model averaging as a probabilistic model selection.
To reflect that some models may be better suited for different inputs, we assume a probabilistic model in which the *selection function* $g$ is treated as a random input-dependent variable.
For a training set $\mathcal{D}=\{(x_{i},y_{i})\}_{i=1}^{n}$, and a new input $x$, we assume the data generating process

$$ $\displaystyle x_{i},x\overset{\mathrm{iid}}{\sim}p(x),$ (5) $\displaystyle g\sim p(g\mid x,x_{1:n}),$ (6) $\displaystyle y_{i}\sim p(y_{i}\mid x_{i},g),\;y\sim p(y\mid x,g).$ (7) $$

We defer the precise specification of the adaptive prior $p(g\mid x,x_{1:n})$ to Section [3.1](https://arxiv.org/html/2510.22054v1#S3.SS1).

The predictive distribution for $y$ given a new input $x$ and the training data is then

$$ $p(y\mid x,\mathcal{D})=\int p(y\mid x,\mathcal{D},g)\,p(g\mid x,\mathcal{D})\,d\mu(g)=\int p(y\mid x,g)\,p(g\mid x,\mathcal{D})\,d\mu(g),$ (8) $$

where $p(g\mid x,\mathcal{D})$ is a posterior distribution on the space of functions(^1^11Formally, $p(g\mid x,\mathcal{D})$ is a density w.r.t some reference measure $\mu$ on a space of measurable functions $\mathcal{G}$.)
$\mathcal{G}\coloneqq\big\{g:\mathcal{X}\to\{e_{1},\dots,e_{m}\}\big\}$.

A draw from the posterior $g\sim p(g\mid x,\mathcal{D})$ induces a random index $J(x)$, defined by the relation $g(x)=e_{j(x)}$. Using this index, we can rewrite equation [8](https://arxiv.org/html/2510.22054v1#S2.E8) as

$$ $p(y\mid x,\mathcal{D})=\int p(y\mid x,g)\,p(g\mid x,\mathcal{D})\,d\mu(g)\\ =\sum_{j=1}^{m}f_{j}(y\mid x)\,p(J(x)=j\mid x,\mathcal{D}).$ (9) $$

A formal proof of this equality is outlined in Appendix [A.1](https://arxiv.org/html/2510.22054v1#A1.SS1).

Under our model, the predictive distribution is a mixture of the candidate predictions $f_{j}(y\mid x)$ weighted by the posterior probabilities $p(J(x)=j\mid x,\mathcal{D})$. In other words, the input adaptive weights $\alpha_{j}(x)$ *arise directly from the probabilistic formulation* itself, and *they are precisely the posterior probabilities* of each model being the generator at input $x$.

A central difficulty, of course, is that the true posterior is unknown.
In Section [3](https://arxiv.org/html/2510.22054v1#S3), we introduce a variational approximation to $p(J(x)=j\mid\mathcal{D}_{i},x)$ that preserves explicit dependence on both $x$ and $\mathcal{D}$.
Before presenting this approximation, we first analyze the performance guarantees that arise when the averaging weights are set to the true posterior probabilities $p(J(x)=j\mid x,\mathcal{D})$.

### 2.1 Likelihood guarantees

So far we have seen that the posterior probabilities $p(J(x)=j\mid x,\mathcal{D})$ arise naturally as input adaptive weights under our model. In particular, they are the Bayes-optimal weights, as they recover the true predictive distribution.

We now show that this choice also comes with performance guarantees: the posterior-weights predictor not only reflects the correct probabilistic formulation, but in expectation achieves likelihood performance competitive with any input-specific single-model selector. The next theorem formalizes this result (for proof see Appendix [A.2](https://arxiv.org/html/2510.22054v1#A1.SS2)).

###### Theorem 2.1 .

Denote $\mathcal{D}_{i}\coloneqq\{(x_{t},y_{t})\}_{t=1}^{i}$, and consider the posterior weights predictor $\hat{p}_{\alpha}^{(i)}$ assigning $\alpha_{j}(x;\mathcal{D}_{i})=p(J(x)=j\mid\mathcal{D}_{i},x)$ to the $j$-th predictor $f_{j}$.
Assume that $\mathbb{E}[|\log f_{j}(Y\mid X)|]<\infty$ for all $f_{j}\in\mathcal{F}$.
Then, for any measurable selector $j^{*}:\mathcal{X}\to\{1,\dots,m\}$ and any $n\geq 1$,

$$ $\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}\left[\log\hat{p}_{\alpha}^{(i)}(y_{i}\mid x_{i},\mathcal{D}_{i-1})\right]\geq\mathbb{E}\left[\log f_{j^{*}(x)}(y\mid x)\right]+\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}\left[\log\alpha^{(i)}_{j^{*}(x_{i})}(x_{i})\right],$ (10) $$

where the expectations are taken w.r.t the population distribution $(x_{i},y_{i})\sim p(x,y)$.

Thus, the posterior weights predictor can match any per-input selector (i.e., a rule that may pick a different $j$ for different $x$), up to a term depending on the gating weights assigned to the chosen model at each $x$.

Concretely, for the selector that picks the most probable model,
$j^{(i)}(x)\in\operatorname*{arg\,max}_{1\leq j\leq m}\alpha^{(i)}_{j}(x)$,
the penalty becomes $\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}\!\left[\log\max_{j}\alpha^{(i)}_{j}(x_{i})\right]$, which vanishes as the posterior sharpens, i.e., when
$\max_{j}\alpha^{(i)}_{j}(x_{i})\to 1$ (in probability or almost surely).

###### Theorem 2.1 .

## 3 IA-BMA: Input Adaptive Bayesian Model Averaging

Our *goal* is to develop a method for estimating this posterior distribution over models. By doing so, we obtain an averaging scheme that is consistent with both the training data $\mathcal{D}$ and the specific input $x$, thereby approximating the true predictive distribution that we ultimately aim to recover.

We begin by formulating the modeling assumptions for an adaptive prior that is conditioned jointly on the training covariates and a new input.
Building on this prior, we then develop a variational inference method to approximate the resulting posterior.

### 3.1 Adaptive prior

Based on the adaptive prior introduced in (Slavutsky & Blei, 2025), we posit a prior that encodes the plausibility of each model conditional on both the training covariate $x_{1:n}$ and a new input $x$ at which prediction is sought. This prior is defined through an energy-based formulation.

Specifically, for a predictor $f_{j}$ we consider the prior induced by the negative energy function

$$ $\displaystyle E(J=j;x_{1:n},x)\coloneqq\int\sum_{i=1}^{n}\log p(y|x_{i},f_{j})+\log p(y|x,f_{j})\;dy$ (11) $\displaystyle p(J=j|x_{1:n},x)\coloneqq\frac{1}{Z(f)}\exp\left(E(J=j;x_{1:n},x)\right),$ (12) $$

where the normalizing factor(^2^22This definition requires integrability of $\exp\left(E(J=j;x_{1:n},x)\right)$, and thus we assume that $\exp(E(J=j;x_{1:n},x))$ is integrable for each $j$.) is given by $Z(f)\coloneqq\sum_{j=1}^{m}\exp\left(E(J=j;x_{1:n},x)\right)$.

This prior allows beliefs about model plausibility to adapt to the new input $x$. Unlike a prior defined solely from the training data, which remains fixed across prediction points, our formulation updates the relative weight of each model once $x$ is observed. This makes the prior *input adaptive*, enabling model selection probabilities to shift dynamically with the prediction covariates. To build intuition, we next examine a simple analytical example.

##### A two-model Bernoulli example

Suppose $y\in\{0,1\}$, and consider two candidate logistic models

$$ $p(y=1\mid x,f_{j})=\sigma(\beta_{j}x),\qquad\sigma(u)\coloneqq\frac{1}{1+e^{-u}},$ (13) $$

with $j\in{1,2}$ and slopes $0<\beta_{2}<\beta_{1}$.
In this setting, the energy function is given by

$$ $\displaystyle E(J=j;x_{1:n},x)$ $\displaystyle=\sum_{i=1}^{n}\sum_{y\in\{0,1\}}\log p(y\mid x_{i},f_{j})+\sum_{y\in\{0,1\}}\log p(y\mid x,f_{j})$ (14) $\displaystyle=\underbrace{\,\sum_{i=1}^{n}\log\left(\sigma(\beta_{j}x_{i})\left[1-\sigma(\beta_{j}x_{i})\right]\right)\quad}_{\eqqcolon C_{j}}+\underbrace{\phantom{\sum_{i=1}^{n}}\log\left(\sigma(\beta_{j}x)\left[1-\sigma(\beta_{j}x)\right]\right)\,}_{\eqqcolon\ell_{j}(x)}$ (15) $$

and the adaptive prior is

$$ $p(J=j\mid x_{1:n},x)=\frac{\exp(C_{j}+\ell_{j}(x))}{\sum_{k=1}^{m}\exp(C_{k}+\ell_{k}(x))}$ (16) $$

Accordingly, the log-odds between the two models is

$$ $\log\frac{p(J=1\mid x_{1:n},x)}{p(J=2\mid x_{1:n},x)}=(C_{1}-C_{2})+\ell_{1}(x)-\ell_{2}(x).$ (17) $$

Thus, the log-odds depend both on the difference between training baselines $C_{1}-C_{2}$, and the change induced by conditioning also on the new input $x$ is $\delta_{x}\coloneqq\ell_{1}(x)-\ell_{2}(x)$.

Concretely, suppose the baseline difference is fixed at $C_{1}-C_{2}=\log 5\approx 1.61$, yielding $p(J=1\mid\mathcal{D})=\sigma(\log 5)\approx 0.83$.
Based solely on the training data, the prior thus strongly favors $f_{1}$. Now consider a new input $x=1$ with $\beta_{2}=1$. As $\beta_{1}$ increases, the discrepancy $\lvert\ell_{1}(1)-\ell_{2}(1)\rvert$ grows, and $\delta_{x=1}$ shifts the likelihood ratio toward $f_{2}$.
For example, when $\beta_{1}=3$, the prior shifts to a mild preference for $f_{1}$, at $\beta_{1}=5$ it flips to favor $f_{2}$, and by $\beta_{1}=9$ the preference for $f_{2}$ becomes very strong. These dynamics, along with additional parameter settings for coefficients and baseline differences, are shown in Figure [1](https://arxiv.org/html/2510.22054v1#S3.F1).

This analysis highlights the interplay between the baseline preference and the input-specific adjustment introduced by $x$. It shows that, in extreme cases, even strong baseline beliefs can be overturned by the adaptive correction at the queried input.

Figure: Figure 1: Illustration of the input adaptive prior. Each panel shows the posterior probabilities $p(J=j\mid\mathcal{D},x)$ as functions of $x$. *Top*: the baseline log-odds is fixed and $\beta_{1}$ varies; larger $\beta_{1}$ values increase the influence of $x$, producing stronger adaptive corrections. *Bottom*: $\beta_{1},\beta_{2}$ are fixed while the baseline log-odds $C_{1}-C_{2}$ varies; stronger baselines yield higher prior preference for $f_{1}$, but input-specific corrections can still substantially reshape the prior at certain $x$. The marked point ($x=1$) highlights how the adaptive prior shifts the relative model probabilities compared to the baseline.
Refer to caption: x1.png

##### Evaluation of the prior:

Evaluating the proposed prior requires computing an integral over the outcome space $\mathcal{Y}$, and thus depends on whether the outcome space is discrete or continuous.
When $\mathcal{Y}$ is *discrete* (e.g., in classification problems), the integral reduces to a finite sum over all possible outcome values. In this case, the evaluation is straightforward and can be computed exactly without approximation.
When $\mathcal{Y}$ is *continuous*, (e.g., in regression problems), the integral cannot typically be computed in closed form and may even diverge unless we restrict the domain of integration. Thus, to approximate the prior, as in (Slavutsky & Blei, 2025), we employ Monte-Carlo integration where we sample $K$ possible outcome values uniformly from a predefined integration range $[y_{\min},y_{\max}]$ and average the Normal log-likelihood (centered at the model’s prediction with unit variance) over the $K$ samples.

### 3.2 Amortized Variational Posterior

Equipped with the adaptive prior, we now turn to the estimation of the posterior $p(J=j\mid x_{1:n},y_{1:n},x)=p(J=j\mid\mathcal{D},x)$, which conditions not only on the covariates $x$ and $x_{1:n}$, but also on the training labels $y_{1:n}$.
This, in turn, will enable us to assign input adaptive weights for model averaging, bringing them closer to the ideal weights that recover the predictive distribution $p(y\mid x)$.

We do so by fitting variational distributions $q(f_{j};x)\approx p(J=j\mid\mathcal{D},x)$ parameterized as functions of the input $x$. This yields an *amortized posterior approximation*, which allows us to efficiently evaluate approximate posteriors at multiple inputs $x$.

In our case, in the context of a new input $x$, the true posterior distribution over predictors is Multinomial $p(J=j\mid\mathcal{D},x)=\rho_{j}(x)$ for $j\in\{1,\dots,m\}$, where each $\rho_{j}(x)>0$ and $\sum_{j=1}^{m}\rho_{j}(x)=1$.
Thus, we set the variational family to be the set of all multinomial distributions.

$$ $\mathcal{Q}_{x}\coloneqq\{q=(q(J=1;x),\dots,q(J=m;x)\in\Delta^{m-1}\}.$ (18) $$

For a given input $x$, our goal is to minimize the KL divergence

$$ $\displaystyle\min_{q\in\mathcal{Q}_{x}}D_{\mathrm{KL}}(q\|\,p)\coloneqq\sum_{j=1}^{m}q(J=j;x)\log\frac{q(J=j;x)}{p(J=j\mid\mathcal{D},x)}.$ (19) $$

Note that since the true posterior and the variational family share the same (categorical) form, the problem is well-specified: the KL depends only on estimating the probabilities $P(J=j;x)$.

### 3.3 Optimization

To minimize the KL divergence in Equation [19](https://arxiv.org/html/2510.22054v1#S3.E19), we optimize the evidence lower bound (ELBO) on the log-likelihood (Kingma & Welling, 2014; Rezende & Mohamed, 2015; Blei et al., 2017).
We parameterize the variational distribution with a neural network with weights $\theta$, producing $h_{\theta}(x)=(q_{\theta}(J=1;x),\dots,q_{\theta}(J=m;x))$, and optimize $\theta$ rather than the output directly.
Thus, our objective to fit the amortized posterior is

$$ $\displaystyle\mathcal{L}\left(\theta;x\right)$ $\displaystyle=\mathbb{E}_{q_{\theta}}\left[\log p(y\mid x,f_{j})\right]-D_{\mathrm{KL}}(q_{\theta}\|\,p(J|x_{1:n},x))$ (20) $\displaystyle=\sum_{j=1}^{m}\left[q_{\theta}(J=j;x)\log f_{j}(y\mid x)\right]-\sum_{j=1}^{m}q_{\theta}(J=j;x)\log\frac{q_{\theta}(J=j;x)}{p(J=j\mid x_{1:n},x)}.$ (21) $$

Note that the expected log-likelihood $\mathbb{E}_{q_{\theta}}\left[\log p(y\mid x,f_{j})\right]$ reduces to a weighted sum, so no sampling is required to evaluate our objective. The complete optimization procedure is summarized in Algorithm [1](https://arxiv.org/html/2510.22054v1#alg1).

Figure: Algorithm 1 IA-BMA: Amortized Posterior Learning (IABMA)

##### Weight assignment:

After training is complete (see Algorithm [1](https://arxiv.org/html/2510.22054v1#alg1)), with the estimate $\hat{\theta}$, for a new input $x$ we compute $(q_{\hat{\theta}}(J=1;x),\dots,q_{\hat{\theta}}(J=m;x))$ and assign $\alpha_{j}(x)=q_{\hat{\theta}}(J=j;x)$.
This yields a predicted value $\hat{p}_{\alpha}(y\mid x)=\sum_{j=1}^{m}\alpha_{j}(x)f_{j}(y\mid x)$.

## 4 Experiments

We evaluate IA-BMA via 7 experiments on classification and regression tasks, based on simulated data, 2 case-studies, and 4 UCI benchmark datasets.

We compare predictive distributions against (a) non-adaptive baselines: (i) best single predictor, (ii) uniform average over predictors, (iii) accuracy-weighted average, and (iv) classical Bayesian model averaging (BMA); and (b) adaptive methods (see Section [1.1](https://arxiv.org/html/2510.22054v1#S1.SS1)): (i) Mixture of Experts (MoE) (Jacobs et al., 1991), (ii) Dynamic Local Accuracy (DLA) (Woods et al., 1997), (iii) Synthetic Model Combination (SMC) (Rasmussen & Ghahramani, 2001), and (iv) Bayesian Hierarchical Stacking (BHS) (Yao et al., 2022).

In each experiment we train candidate predictors and fit all model averaging methods on the training set. We test the performance of the resulting predictive distributions: for regression tasks we report $R^{2}$ and root mean square error (RMSE); for classification tasks we report accuracy and expected calibration error (ECE).

Hyperparameters for our method and all baselines were tuned via binary search to maximize average performance (accuracy for classification, RMSE for regression) on a held-out repetition excluded from the analysis. The selected values and further implementation details are provided in Appendix [D](https://arxiv.org/html/2510.22054v1#A4), with additional data processing and predictor specifications in Section [C](https://arxiv.org/html/2510.22054v1#A3). Code to reproduce all results is included with the submission and will be released publicly upon acceptance.

### 4.1 Simulation

We start with evaluating IA-BMA on a simple two-dimensional binary task:
half of the observations were drawn from a Gaussian cluster centered at $(-1,0)$, and labels were assigned by a linear rule $y=\mathbbm{1}_{x_{1}+x_{2}>-1}$.
The remaining observations were sampled around $(1,0)$ on a ring; labels followed a circular rule $y=\mathbbm{1}_{r<1}$ with $r=\sqrt{x_{1}^{2}+x_{2}^{2}}$ measured from $(1,0)$.
We generated $n_{\text{train}}=1,000$ and $n_{\text{test}}=500$ examples and used only the 2-dimensional coordinates as features (region indicators were recorded for analysis but not used for training).

This example illustrates three subpopulations defined by their first dimension: (i) inputs that are perfectly separable by a linear boundary, (ii) inputs that are perfectly separable by a circular boundary, and (iii) inputs for which it is ambiguous whether they belong to the first or the second group.
Thus, and ideal weighting would put all weight on the best predictor for subpopulations (i) and (ii), and employ soft weighting at (iii).
All averaging methods operated on the same base classifiers: polynomial logistic regression (degrees 2 and 3), Linear Discriminant Analysis, and tailored “soft-circle” models that predict class probability via a logistic function of radial distance to a learned center.
Additional details are provided in section [C.1](https://arxiv.org/html/2510.22054v1#A3.SS1)

##### Results:

Figure [2](https://arxiv.org/html/2510.22054v1#S4.F2) shows that IA-BMA achieves highest accuracy and lowest ECE compared to all non-adaptive baselines, as well as all adaptive methods.

Figure: Figure 2: Simulation. Left: data (of one repetition). Results for accuracy (middle) and ECE (right) are reported for 10 repetitions. IA-BMA achieves highest accuracy and lowest ECE.
Refer to caption: x2.png

### 4.2 Case studies

#### 4.2.1 Personalized cancer drug‐response

An important example of heterogeneous data is personalized drug response prediction, where different models may perform better on different subpopulations. We evaluate IA-BMA on this task using the PRISM cancer drug response dataset.
The data consists of pairings of molecule-cell line RNA sequence features. For each drug–cell pair we form a continuous response $y$ so that larger values indicate greater sensitivity. We retain drugs with broad site coverage and construct inputs from the top variance genes.
All averaging methods operate over the same four base regressors—Ridge, Histogram-based Gradient Boosting Tree, XGBoost, and a Multilayer perceptron (MLP), each with pre-processing tailored to model class.
Additional details are provided in [C.2](https://arxiv.org/html/2510.22054v1#A3.SS2).

##### Results:

Figure [3](https://arxiv.org/html/2510.22054v1#S4.F3) shows that IA-BMA achieves higher $R^{2}$ and lower RMSE compared to all other methods. Further analysis is presented in Figures  [4](https://arxiv.org/html/2510.22054v1#A2.F4)–[7](https://arxiv.org/html/2510.22054v1#A2.F7) which display the weights assigned by each averaging method for randomly selected inputs. The results show that IA-BMA consistently favored the best (or nearly best) model, whereas other methods leaned toward other predictors, with MoE in particular overemphasizing MLP and XGB even when suboptimal.

#### 4.2.2 Credit-card fraud detection

Another domain characterized by heterogeneous data is fraud detection, where the rarity of fraudulent cases poses an additional challenge. We evaluate IA-BMA on this task using the IEEE-CIS Fraud Detection dataset.
The dataset consists from mixed Continuous (such as transaction amount) and high-cardinality categorical features (such as product category), and the target variable $y\in\{0,1\}$ indicated where a transaction was fraud.
All averaging methods operate over the same base classifiers: Logistic Regression with Lasso penalty, Histogram-based Gradient Boosting Tree, XGBoost (with class-imbalance weighting), and an MLP. Additional details are provided in Section [C.3](https://arxiv.org/html/2510.22054v1#A3.SS3).

##### Results:

Figure [3](https://arxiv.org/html/2510.22054v1#S4.F3) shows that IA-BMA achieves higher accuracy and lower expected-calibration error compared to all other methods.
Since in fraud prediction calibration matters within each bin, we analyzed per-bin confidence $\lvert p-0.5\rvert$, and found that IA-BMA achieves the lowest error in all high-confidence bins ($>0.25$). The corresponding analysis is shown in Figure [8](https://arxiv.org/html/2510.22054v1#A2.F8).

Figure: Figure 3: Experimental results for main case studies. Results are reported for 10 repetitions. IA-BMAachieves best results compared to all other averaging method on both case studies.
Refer to caption: x3.png

### 4.3 Experiments on UCI benchmark datasets

We evaluated IA-BMA on 4 UCI benchmark datasets spanning both classification and regression.
For *classification* we used spambase and credit-g;
for *regression* we used bike-sharing and california-housing.
All pre-processing information is detailed in Section [C.4](https://arxiv.org/html/2510.22054v1#A3.SS4)

Across all datasets we trained a common set of base learners. For classification: Multinomial Naive Bayes, $k$–NN ($k=3$), Random Forest, Extra Trees, and a linear SVM. For regression: Ridge ($\alpha{=}0.05$), Lasso ($\alpha{=}0.05$), $k$–NN ($k{=}3$, distance-weighted), Random Forest, and Extra-Trees. To encourage diversity, each model was trained on a subset of features (“feature bundles”). Full pre-processing steps, feature bundles, and model specifications are detailed in Section [C.4](https://arxiv.org/html/2510.22054v1#A3.SS4).

Results: Unlike complex heterogeneous datasets, UCI benchmarks are highly normalized and thus show less variability across methods. Even so, IA-BMA outperforms all competitors in all four experiments on at least one metric, and in two cases on both. Results are reported in Table [1](https://arxiv.org/html/2510.22054v1#A2.T1).

## 5 Conclusion

We introduced IA-BMA, a framework that casts model averaging as probabilistic model selection conditioned on the input. Within this formulation, the posterior distribution over models provides the natural, Bayes-optimal choice of input adaptive weights, thereby recovering the true predictive distribution. Our approach is grounded in an input-dependent prior on the selector function and implemented through amortized variational inference of the posterior.

We establish finite-sample bounds showing that the posterior-weights predictor achieves strong likelihood performance compared to any input-specific single-model selector.
Empirically, we evaluate IA-BMA across regression and classification tasks, including personalized cancer treatment response, credit-card fraud detection, and standard UCI benchmarks. We show that IA-BMA consistently outperforms both non-adaptive baselines and existing adaptive methods, delivering more accurate and better calibrated predictions.

## References

- Blei et al. (2017)
David M Blei, Alp Kucukelbir, and Jon McAuliffe.
Variational inference: A review for statisticians.
*Journal of the American Statistical Association*, 112(518):859–877, 2017.
- Breiman (1996)
Leo Breiman.
Bagging predictors.
*Machine Learning*, 24(2):123–140, 1996.
- Cao et al. (1995)
Jun Cao, Majid Ahmadi, and Malayappan Shridhar.
Recognition of handwritten numerals with multiple feature and multistage classifier.
*Pattern Recognition*, 28(2):153–160, 1995.
- Chan & van der Schaar (2022)
Alex Chan and Mihaela van der Schaar.
Synthetic model combination: An instance-wise approach to unsupervised ensemble learning.
*Advances in Neural Information Processing Systems*, 35:27797–27809, 2022.
- Clemen (1989)
Robert T Clemen.
Combining forecasts: A review and annotated bibliography.
*International journal of forecasting*, 5(4):559–583, 1989.
- Didaci & Giacinto (2004)
Luca Didaci and Giorgio Giacinto.
Dynamic classifier selection by adaptive k-nearest-neighbourhood rule.
In *Proceedings of the International Workshop on Multiple Classifier Systems*, pp. 174–183. Springer, 2004.
- Didaci et al. (2005)
Luca Didaci, Giorgio Giacinto, Fabio Roli, and Gian Luca Marcialis.
A study on the performances of dynamic classifier selection based on local accuracy estimation.
*Pattern Recognition*, 38(11):2188–2191, 2005.
- Dietterich et al. (2002)
Thomas G Dietterich et al.
Ensemble learning.
*The Handbook of Brain Theory and Neural Networks*, 2(1):110–125, 2002.
- Dormann et al. (2008)
Carsten F Dormann, Oliver Schweiger, Paul Arens, Isabel Augenstein, S. T. Aviron, Debra Bailey, Jacques Baudry, Regula Billeter, Rob Bugter, Roman Bukacek, et al.
Prediction uncertainty of environmental change effects on temperate european biodiversity.
*Ecology Letters*, 11(3):235–244, 2008.
- Dormann et al. (2018)
Carsten F Dormann, Justin M Calabrese, Gurutzeta Guillera-Arroita, Eleni Matechou, Volker Bahn, Kamil Bartoń, Colin M Beale, Simone Ciuti, Jane Elith, Katharina Gerstner, et al.
Model averaging in ecology: A review of bayesian, information-theoretic, and tactical approaches for predictive inference.
*Ecological Monographs*, 88(4):485–504, 2018.
- Fernández-Delgado et al. (2014)
Manuel Fernández-Delgado, Eva Cernadas, Senén Barro, and Dinani Amorim.
Do we need hundreds of classifiers to solve real world classification problems?
*Journal of Machine Learning Research*, 15(1):3133–3181, 2014.
- Freund (1995)
Yoav Freund.
Boosting a weak learning algorithm by majority.
*Information and Computation*, 121(2):256–285, 1995.
- Freund & Schapire (1997)
Yoav Freund and Robert E Schapire.
A decision-theoretic generalization of on-line learning and an application to boosting.
*Journal of computer and system sciences*, 55(1):119–139, 1997.
- Ganaie et al. (2022)
Mudasir A Ganaie, Minghui Hu, Ashwani Kumar Malik, Muhammad Tanveer, and Ponnuthurai N Suganthan.
Ensemble deep learning: A review.
*Engineering Applications of Artificial Intelligence*, 115:105151, 2022.
- Giacinto & Roli (1999)
Giorgio Giacinto and Fabio Roli.
Methods for dynamic classifier selection.
In *Proceedings of the 10th International Conference on Image Analysis and Processing*, pp. 659–664. IEEE, 1999.
- Gomes et al. (2017)
Heitor Murilo Gomes, Jean Paul Barddal, Fabrício Enembreck, and Albert Bifet.
A survey on ensemble learning for data stream classification.
*ACM Computing Surveys*, 50(2):1–36, 2017.
- González et al. (2020)
Sergio González, Salvador García, Javier Del Ser, Lior Rokach, and Francisco Herrera.
A practical tutorial on bagging and boosting based ensembles for machine learning: Algorithms, software tools, performance study, practical perspectives and opportunities.
*Information Fusion*, 64:205–237, 2020.
- Grother et al. (2019)
Patrick Grother, Mei Ngan, Kayee Hanaoka, et al.
Ongoing face recognition vendor test (frvt) part 3: Demographic effects.
*Nat. Inst. Stand. Technol., Gaithersburg, MA, USA, Rep. NISTIR*, 8280, 2019.
- Gunes et al. (2003)
Veyis Gunes, Michel Menard, Pierre Loonis, and Simon Petit-Renaud.
Combination, cooperation and selection of classifiers: A state of the art.
*International Journal of Pattern Recognition and Artificial Intelligence*, 17(08):1303–1324, 2003.
- Guo et al. (2017)
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger.
On calibration of modern neural networks.
In *International conference on machine learning*, pp. 1321–1330. PMLR, 2017.
- Jacobs et al. (1991)
Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton.
Adaptive mixtures of local experts.
*Neural Computation*, 3(1):79–87, 1991.
- Jiang et al. (2021)
Zhencun Jiang, Zhengxin Dong, Lingyang Wang, and Wenping Jiang.
Method for diagnosis of acute lymphoblastic leukemia based on vit-cnn ensemble model.
*Computational Intelligence and Neuroscience*, 2021(1):7529893, 2021.
- Jordan & Jacobs (1994)
Michael I Jordan and Robert A Jacobs.
Hierarchical mixtures of experts and the em algorithm.
*Neural computation*, 6(2):181–214, 1994.
- Kingma & Welling (2014)
Diederik P Kingma and Max Welling.
Auto-encoding variational bayes.
*International Conference on Learning Representations*, 2014.
- Kulkarni & Sinha (2013)
Vrushali Y Kulkarni and Pradeep K Sinha.
Random forest classifiers: A survey and future research directions.
*International Journal of Advanced Computer*, 36(1):1144–1153, 2013.
- Lauzeral et al. (2015)
Christine Lauzeral, Gaël Grenouillet, and Sébastien Brosse.
The iterative ensemble modelling approach increases the accuracy of fish distribution models.
*Ecography*, 38(2):213–220, 2015.
- Mahajan et al. (2023)
Palak Mahajan, Shahadat Uddin, Farshid Hajati, and Mohammad Ali Moni.
Ensemble learning for disease prediction: A review.
In *Healthcare*, volume 11, pp. 1808. MDPI, 2023.
- Mennis (2006)
Edmund A Mennis.
The wisdom of crowds: Why the many are smarter than the few and how collective wisdom shapes business, economies, societies, and nations.
*Business Economics*, 41(4):63–65, 2006.
- Nanglia et al. (2022)
S Nanglia, Muneer Ahmad, Fawad Ali Khan, and N. Z. Jhanjhi.
An enhanced predictive heterogeneous ensemble model for breast cancer prediction.
*Biomedical Signal Processing and Control*, 72:103279, 2022.
- Peng & Yang (2022)
Jingfu Peng and Yuhong Yang.
On improvability of model selection by model averaging.
*Journal of Econometrics*, 229(2):246–262, 2022.
- Polikar (2006)
Robi Polikar.
Ensemble based systems in decision making.
*IEEE Circuits and Systems Magazine*, 6(3):21–45, 2006.
- Rasmussen & Ghahramani (2001)
Carl Rasmussen and Zoubin Ghahramani.
Infinite mixtures of gaussian process experts.
*Advances in Neural Information Processing Systems*, 14, 2001.
- Rezende & Mohamed (2015)
Danilo Rezende and Shakir Mohamed.
Variational inference with normalizing flows.
In *Proceedings of the International Conference on Machine Learning*, pp. 1530–1538. PMLR, 2015.
- Richards (2005)
Shane A Richards.
Testing ecological theory using the information-theoretic approach: Examples and cautionary results.
*Ecology*, 86(10):2805–2814, 2005.
- Rokach (2010)
Lior Rokach.
*Pattern Classification Using Ensemble Methods*, volume 75.
World Scientific, 2010.
- Rokach (2016)
Lior Rokach.
Decision forest: Twenty years of research.
*Information Fusion*, 27:111–125, 2016.
- Sagi & Rokach (2018)
Omer Sagi and Lior Rokach.
Ensemble learning: A survey.
*Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery*, 8(4):e1249, 2018.
- Slavutsky & Blei (2025)
Yuli Slavutsky and David M Blei.
Quantifying uncertainty in the presence of distribution shifts.
*Advances in Neural Information Processing Systems*, 2025.
- Tenzer et al. (2022)
Yaniv Tenzer, Omer Dror, Boaz Nadler, Erhan Bilal, and Yuval Kluger.
Crowdsourcing regression: A spectral approach.
In *International Conference on Artificial Intelligence and Statistics*, pp. 5225–5242. PMLR, 2022.
- Thuiller (2004)
Wilfried Thuiller.
Patterns and uncertainties of species’ range shifts under climate change.
*Global Change Biology*, 10(12):2020–2027, 2004.
- Varmedja et al. (2019)
Dejan Varmedja, Mirjana Karanovic, Srdjan Sladojevic, Marko Arsenovic, and Andras Anderla.
Credit card fraud detection-machine learning methods.
In *2019 18th International Symposium Infoteh-Jahorina (Infoteh)*, pp. 1–5. IEEE, 2019.
- Wang et al. (2019)
Mei Wang, Weihong Deng, Jiani Hu, Xunqiang Tao, and Yaohai Huang.
Racial faces in the wild: Reducing racial bias by information maximization adaptation network.
In *Proceedings of the ieee/cvf international conference on computer vision*, pp. 692–702, 2019.
- Waterhouse et al. (1995)
Steve Waterhouse, David MacKay, and Anthony Robinson.
Bayesian methods for mixtures of experts.
*Advances in neural information processing systems*, 8, 1995.
- Wintle et al. (2003)
Brendan A Wintle, Michel A McCarthy, Chris T Volinsky, and Rodney P Kavanagh.
The use of bayesian model averaging to better represent uncertainty in ecological models.
*Conservation Biology*, 17(6):1579–1590, 2003.
- Wolpert (1992)
David H Wolpert.
Stacked generalization.
*Neural Networks*, 5(2):241–259, 1992.
- Woods et al. (1997)
Kevin Woods, W. Philip Kegelmeyer, and Kevin Bowyer.
Combination of multiple classifiers using local accuracy estimates.
*IEEE Transactions on Pattern Analysis and Machine Intelligence*, 19(4):405–410, 1997.
- Woźniak et al. (2014)
Michał Woźniak, Manuel Grana, and Emilio Corchado.
A survey of multiple classifier systems as hybrid systems.
*Information Fusion*, 16:3–17, 2014.
- Wu & Levinson (2021)
Hao Wu and David Levinson.
The ensemble approach to forecasting: A review and synthesis.
*Transportation Research Part C: Emerging Technologies*, 132:103357, 2021.
- Yao et al. (2022)
Yuling Yao, Gregor Pirš, Aki Vehtari, and Andrew Gelman.
Bayesian hierarchical stacking: Some models are (somewhere) useful.
*Bayesian Analysis*, 17(4):1043–1071, 2022.
- Zheng et al. (2024)
Yue Zheng, Jun Wei, Wenming Zhang, Yiping Zhang, Tuqiao Zhang, and Yongchao Zhou.
An ensemble model for accurate prediction of key water quality parameters in river based on deep learning methods.
*Journal of Environmental Management*, 366:121932, 2024.

## Appendix A Proofs

### A.1 Change of measure argument

Let $F:X_{2}\to X_{1}$ be a measurable function between two measure spaces $(X_{1},\mathcal{A}_{1},\eta)$ and $(X_{2},\mathcal{A}_{2},\nu)$. Let $g:X_{1}\to\mathbb{R}$ measurable function. Recall that the change of variables formula is given by

$$ $\displaystyle\int_{X_{1}}g\;dF_{\#}\eta$ $\displaystyle=\int_{X_{1}}(g\;\circ F)\;d\eta.$ (22) $$

where $F_{\#}\eta$ denotes the push-forward of $\eta$ through $F$.

Applying this to our setting, recall that a draw from the posterior
$g\sim p(g\mid x,\mathcal{D})$ induces a random index $j(x)$ defined by
the relation $g(x)=e_{j(x)}$.
Formally, the evaluation map

$$ $s_{x}:\mathcal{G}\to\{1,\dots,m\},\qquad s_{x}(g)=j(x),$ $$

pushes the posterior measure $p(g\mid x,\mathcal{D})$ forward onto a distribution over indices.
Using this push-forward, we can rewrite equation [8](https://arxiv.org/html/2510.22054v1#S2.E8) as

$$ $\displaystyle\int_{\mathcal{G}}p(y\mid x,g)\,d\,\mathbb{P}(g\mid x,\mathcal{D})$ $\displaystyle=\int_{\mathcal{G}}f_{s_{x}(g)}(y\mid x)\,d\,\mathbb{P}(g\mid x,\mathcal{D})$ (23) $\displaystyle=\int_{\{1,\dots,m\}}f_{j}(y\mid x)\,d\,(E_{x\#}\mathbb{P})(j\mid x,\mathcal{D})$ (24) $\displaystyle=\sum_{j=1}^{m}f_{j}(y\mid x)\,p(j\mid x,\mathcal{D}).$ (25) $$

### A.2 Proof of Theorem 2.1

###### Theorem .

Denote $\mathcal{D}_{i}\coloneqq\{(x_{t},y_{t})\}_{t=1}^{i}$, and consider the posterior weights predictor $\hat{p}_{\alpha}^{(i)}$ assigning $\alpha_{j}^{(i)}(x)\coloneqq p(J(x)=j\mid\mathcal{D}_{i},x)$ to the $j$-th predictor $f_{j}$.
Assume that $\mathbb{E}[|\log f_{j}(Y\mid X)|]<\infty$ for all $f_{j}\in\mathcal{F}$.
Then, for any measurable selector $j^{*}:\mathcal{X}\to\{1,\dots,m\}$ and any $n\geq 1$,

$$ $\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}\left[\log\hat{p}_{\alpha}^{(i)}(y_{i}\mid x_{i},\mathcal{D}_{i-1})\right]\geq\mathbb{E}\left[\log f_{j^{*}(x)}(y\mid x)\right]+\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}\left[\log\alpha^{(i)}_{j^{*}(x_{i})}(x_{i})\right],$ (26) $$

where the expectations are taken w.r.t the population distribution $(x,y)\sim p(x,y)$.

###### Proof.

Define the posterior-weights predictor

$$ $\displaystyle\hat{p}_{\alpha}^{(i)}(y\mid x,\mathcal{D}_{i-1})=\sum_{j=1}^{m}\alpha_{j}^{(i)}(x)f_{j}(y\mid x)$ (27) $$

For a fixed input $x_{i}$ and a fixed predictor $f_{k}$ we have that

$$ $\displaystyle\log\hat{p}_{\alpha}^{(i)}(y_{i}\mid x_{i},\mathcal{D}_{i-1})$ $\displaystyle=\log\left(\sum_{j=1}^{m}\alpha_{j}^{(i)}(x_{i})f_{j}(y_{i}\mid x_{i})\right)$ (28) $\displaystyle\geq\log\left(\alpha_{k}^{(i)}(x_{i})f_{k}(y_{i}\mid x_{i})\right)$ (29) $\displaystyle=\log f_{k}(y_{i}\mid x_{i})+\log\alpha_{k}^{(i)}(x_{i}).$ (30) $$

Taking $\mathbb{E}_{(x,y)\sim p(x,y)}\left[\cdot\mid x_{i},\mathcal{D}_{i-1}\right]$, since $f_{j^{*}(x)}(y_{i}\mid x_{i})$ is independent of $\mathcal{D}_{i-1}$,

$$ $\mathbb{E}\left[\log\hat{p}_{\alpha}^{(i)}(y_{i}\mid x_{i},\mathcal{D}_{i-1})\right]\geq\mathbb{E}\left[\log f_{k}(y_{i}\mid x_{i})\right]+\mathbb{E}\left[\log\alpha_{k}^{(i)}(x_{i})\mid\mathcal{D}_{i-1}\right].$ (31) $$

This holds for any $1\leq k\leq m$, hence for $k=j^{*}(x_{i})$,

$$ $\mathbb{E}\left[\log\hat{p}_{\alpha}^{(i)}(y_{i}\mid x_{i},\mathcal{D}_{i-1})\mid\mathcal{D}_{i-1}\right]\geq\mathbb{E}\left[\log f_{j^{*}(x)}(y_{i}\mid x_{i})\right]+\mathbb{E}\left[\log\alpha_{j^{*}(x_{i})}^{(i)}(x_{i})\mid\mathcal{D}_{i-1}\right].$ (32) $$

Taking $\mathbb{E}\left[\cdot\mid\mathcal{D}_{i-1}\right]$, by the law of total expectation,

$$ $\displaystyle\mathbb{E}\left[\log\hat{p}_{\alpha}^{(i)}(y_{i}\mid x_{i},\mathcal{D}_{i-1})\right]$ $\displaystyle\geq\mathbb{E}\left[\log f_{j^{*}(x_{i})}(y_{i}\mid x_{i})\right]+\mathbb{E}\left[\log\alpha_{j^{*}(x_{i})}^{(i)}(x_{i})\right].$ (33) $$

Averaging over $i$, we get

$$ $\displaystyle\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}\left[\log\hat{p}_{\alpha}^{(i)}(y_{i}\mid x_{i},\mathcal{D}_{i-1})\right]$ $\displaystyle\geq\mathbb{E}\left[\log f_{j^{*}(x_{i})}(y_{i}\mid x_{i})\right]+\frac{1}{n}\sum_{i=1}^{n}\mathbb{E}\left[\log\alpha_{j^{*}(x_{i})}^{(i)}(x_{i})\right].$ $$

∎

###### Theorem .

###### Proof.

## Appendix B Additional experimental results

In what follows we provide a deeper analysis of the performance of adaptive model averaging methods on the two case-studies.

### B.1 Cancer treatment response

To illustrate how different methods allocate weights, we sampled 16 cases as follows: for each classifier $f_{j}$, we randomly selected four examples from those where IA-BMA assigned the highest weight to $f_{j}$. Figures [4](https://arxiv.org/html/2510.22054v1#A2.F4)–[7](https://arxiv.org/html/2510.22054v1#A2.F7) display the weights assigned by each averaging method for Ridge, XGB, HGB, and MLP. For each case, we also report the RMSE achieved by the individual classifiers. This analysis shows that in all cases, IA-BMA places the largest weight on the model with either the lowest error or a near-tied second. By contrast, competing methods tend to favor other predictors. In particular, MoE consistently prioritizes MLP or XGB, even in instances where these models are locally suboptimal.

Figure: Figure 4: Cases where IA-BMA assigns the highest weight to Ridge.
Refer to caption: x4.png

Figure: Figure 5: Cases where IA-BMA assigns the highest weight to XGB.
Refer to caption: x5.png

Figure: Figure 6: Cases where IA-BMA assigns the highest weight to HGB.
Refer to caption: x6.png

Figure: Figure 7: Cases where IA-BMA assigns the highest weight to MLP.
Refer to caption: x7.png

### B.2 Credit card fraud

Credit card fraud prediction is a highly sensitive area, with risks of false alarms and misreporting, calibration is crucial not only overall but also within each bin. To this end, we analyzed the confidence measure $\lvert p-0.5\rvert$ where p is the estimated probability, which captures certainty for both positive and negative events, and compared the bin-wise errors across averaging methods. Figure [8](https://arxiv.org/html/2510.22054v1#A2.F8) shows that in all high-confidence bins (confidence $>0.25$), IA-BMA attains the lowest error, showing that most miscalculated predictions occur in low confidence instances.

Figure: Figure 8: Calibration across confidence bins in credit-card fraud prediction
Refer to caption: x8.png

### B.3 UCI benchmark datasets

Results are reported in Table [1](https://arxiv.org/html/2510.22054v1#A2.T1).

**Table 1: UCI benchmarks: mean (sd) across runs.**
| Dataset | Metric | Best<br>single | Uniform<br>Avg. | Freq<br>Avg. | BMA | MoE | DLA | SMC | BHS | IABMA |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Bike-sharing | R2 (↑) | 0.706<br>(0.022) | 0.752<br>(0.010) | 0.773<br>(0.012) | 0.774<br>(0.014) | 0.706<br>(0.022) | 0.781<br>(0.013) | 0.756<br>(0.010) | 0.752<br>(0.010) | 0.794<br>(0.010) |
| Bike-sharing | RMSE (↓) | 0.582<br>(0.033) | 0.491<br>(0.021) | 0.448<br>(0.020) | 0.447<br>(0.021) | 0.581<br>(0.033) | 0.446<br>(0.020) | 0.483<br>(0.020) | 0.491<br>(0.021) | 0.433<br>(0.018) |
| Cal.-housing | R2 (↑) | 0.772<br>(0.022) | 0.840<br>(0.018) | 0.840<br>(0.017) | 0.812<br>(0.017) | 0.778<br>(0.024) | 0.840<br>(0.018) | 0.817<br>(0.066) | 0.840<br>(0.018) | 0.844<br>(0.014) |
| Cal.-housing | RMSE (↓) | 0.036<br>(0.004) | 0.025<br>(0.003) | 0.025<br>(0.003) | 0.029<br>(0.003) | 0.035<br>(0.004) | 0.025<br>(0.003) | 0.029<br>(0.010) | 0.025<br>(0.003) | 0.024<br>(0.003) |
| Credit-g | Accuracy (↑) | 0.634<br>(0.036) | 0.676<br>(0.029) | 0.662<br>(0.038) | 0.648<br>(0.036) | 0.624<br>(0.036) | 0.668<br>(0.039) | 0.626<br>(0.046) | 0.682<br>(0.039) | 0.684<br>(0.047) |
| Credit-g | ECE (↓) | 0.260<br>(0.034) | 0.172<br>(0.022) | 0.169<br>(0.020) | 0.174<br>(0.023) | 0.296<br>(0.035) | 0.173<br>(0.025) | 0.222<br>(0.038) | 0.176<br>(0.025) | 0.175<br>(0.020) |
| Spambase | Accuracy (↑) | 0.699<br>(0.110) | 0.702<br>(0.094) | 0.738<br>(0.044) | 0.760<br>(0.024) | 0.760<br>(0.035) | 0.729<br>(0.052) | 0.757<br>(0.032) | 0.646<br>(0.132) | 0.764<br>(0.032) |
| Spambase | ECE (↓) | 0.114<br>(0.022) | 0.163<br>(0.049) | 0.148<br>(0.042) | 0.169<br>(0.018) | 0.095<br>(0.034) | 0.171<br>(0.051) | 0.222<br>(0.023) | 0.180<br>(0.061) | 0.146<br>(0.025) |

## Appendix C Experimental details

### C.1 Simulation

#### C.1.1 Data and processing

We generated a two-dimensional binary dataset with two subpopulations governed by different decision rules.
For the *linear* subpopulation, we drew $n_{\text{lin}}=n_{\text{train}}/2$ training points from a Gaussian cloud centered at $(-t,0)$ (with $t=1$),

$$ $X^{(\text{lin,train})}\sim\mathcal{N}\bigl(({-}t,0),\,0.1\,I_{2}\bigr),$ $$

and assigned labels by a linear rule $y=\mathbbm{1}\{x_{1}+x_{2}>-t\}$.
For the *circular* subpopulation, we drew $n_{\text{circ}}=n_{\text{train}}-n_{\text{lin}}$ points on a ring around $(t,0)$ by sampling
$\theta\sim\text{Unif}(0,2\pi)$ and $r=\sqrt{U}$ with $U\sim\text{Unif}(0,2)$, and set

$$ $X^{(\text{circ})}=(t,0)+\bigl(r\cos\theta,\,r\sin\theta\bigr),\qquad y=\mathbbm{1}\{r<1\}.$ $$

We used $n_{\text{train}}=1,000$ and $n_{\text{test}}=500$; the train/test splits were generated independently.

Only the two coordinates $(x_{1},x_{2})$ were provided as features.
A region indicator $z\in\{0\ (\text{linear}),\,1\ (\text{circular})\}$ was recorded for analysis but was not used during training.

#### C.1.2 Candidate predictors

All averaging methods were evaluated on the same 3 base classifiers:

- 1.
*Polynomial logistic regression (degrees 2 and 3).*
We fit logistic regression with polynomial features of degree $d\in\{2,3\}$ (no bias term in the expansion).
- 2.
*Linear Discriminant Analysis (LDA).*
A linear generative classifier fit on the raw coordinates, providing a single linear boundary.
- 3.
*Soft-circle classifiers (two instances).*
Each instance modeled the positive-class probability as a logistic function of radial distance to a fixed center,
$p_{\text{circle}}(y{=}1\mid x)\;=\;\sigma\bigl(\gamma\,(R-\|x-c\|)\bigr),\quad c=(0.8\,t,0),\;R=1.0,\;\gamma=5.0,$
yielding smooth circular decision regions around $(t,0)$.
We instantiated two copies of each model to allow the averaging procedure to allocate weight across similar experts.

### C.2 PRISM cancer experiment

#### C.2.1 Data and processing

We used the publicly available PRISM cancer drug response dataset.
The primary data(^3^33Repurposing_Public_23Q2_Extended_Primary_Data_Matrix.csv) was combined with an RNA-seq expression matrix(^4^44OmicsExpressionProteinCodingGenesTPMLogp1.csv), cell-line metadata(^5^55Cell_lines_annotations_20181226.txt), and tissue labels(^6^66Model.csv).
All files are available from [https://depmap.org/portal/data_page/](https://depmap.org/portal/data_page/).

The PRISM file reports drug–cell line responses with identifiers of the form ACH-#.
We normalized all identifiers to the canonical zero-padded format (ACH-XXXXXX).
Non-Continuous entries and all observations lacking a primary cancer site were excluded.
Responses correspond to log-fold changes (LFC), clipped to the range $[-6,6]$, and the prediction target was defined as $y=-v$, where $v$ is the clipped LFC.

We focused on the $40$ drugs with the greatest site-level heterogeneity.
Specifically, we computed the between-site variance of $y$ and retained compounds observed in at least $3$ distinct sites, with at least $5$ samples per site and at least $40$ samples overall.
A minimum per-site coverage threshold of $20$ samples was enforced.
To avoid domination by a few large tissues, we capped each site at $1.1\times s$, where $s$ is its sample size.
This yielded approximately $18{,}460$ drug–cell line pairs (slight variation across random splits), of which $80\%$ were used for training and $20\%$ for testing.

Gene expression features were restricted to the $100$ highest-variance genes.
Each gene was standardized to mean 0 and variance $1$ based on training statistics.
The final feature matrix consisted of standardized gene expression values and a categorical compound indicator.

The full processing code was submitted with this paper and will be released publicly upon acceptance.

#### C.2.2 Candidate predictors

All averaging methods were evaluated on averaging the same four regression models with reprocessing pipelines tailored per model:

- 1.
*Ridge regression ($\ell_{2}$ regularized linear model)*.
Gene features were imputed (median), standardized to zero mean and unit variance, and combined with a dense one-hot encoding of the compound identity.
- 2.
*Histogram-based Gradient Boosting regressor (HGB)*.
Tree-based model trained on raw gene values (median imputation only) together with a sparse one-hot encoding of the compound identity.
- 3.
*XGBoost regressor (XGB)*.
Gradient-boosted decision trees with squared-error objective, trained using the same pre-processing as HGB.
We used 400 estimators, learning rate $0.05$, maximum depth $8$, subsample ratio $0.9$, and column subsample ratio $0.8$, with $\ell_{1}$ and $\ell_{2}$ regularization.
- 4.
*Multi-layer perceptron (MLP)*.
A feed-forward neural network with hidden layers of size (128, 64), ReLU activations, learning rate $10^{-3}$, batch size 64, and early stopping based on a 10% validation split.
Inputs were preprocessed as for Ridge (dense, imputed, standardized gene features and dense one-hot drug encoding).

### C.3 IEEE-CIS fraud experiment

#### C.3.1 Data and processing

We used the IEEE-CIS credit-card fraud dataset, available at [https://www.kaggle.com/c/ieee-fraud-detection/data](https://www.kaggle.com/c/ieee-fraud-detection/data).

We removed rows with missing target (isFraud) and features with more than $50\%$ missing values.
To limit explosion in feature dimension, infrequent categories were grouped into a shared rare category.

In each repetition $80\%$ of the data was used for training and $20\%$ for testing. The training data was then reduced to obtained class balance, while in test data class imbalance was maintained. To reduced covariate shift in the train-test split we stratified jointly on (ProductCD, card4) crossed with per-row missingness bins and TransactionAmt quantile bins, with a fallback “RARE” bucket for very small strata.
This procedure yielded a stable empirical mix of products, card networks, and spending levels.
Specifically, to control the empirical mix of products, card networks, and spending levels we stratified jointly on (ProductCD, card4) crossed with per-row missingness bins and TransactionAmt quantile bins.

Continuous features were median-imputed and where appropriate, standardized to zero mean and unit variance. Categorical features were imputed to the most frequent level and one-hot encoded, with infrequent categories pooled into a rare-level. Class imbalance was addressed within each classifier as noted below.

#### C.3.2 Candidate predictors

All averaging methods were evaluated over the same following base classifiers.

- 1.
*Logistic Regression ($\ell_{1}$-penalized).*
We fit a penalized logistic model to the processed feature set, using an $\ell_{1}$ penalty with strength to encourage sparsity and robustness to correlated predictors. We used a saga solver, $\ell_{1}$ penalty with regularization strength of $0.05$, maximal number of iterations as 4000, and tolerance of $10^{-3}$.
- 2.
*XGBoost (XGB).*
We trained a gradient-boosted ensemble of shallow decision trees using histogram-based splits and early stopping. Depth, learning rate, and number of estimators were selected via a held-out validation set.
Hyper parameters were set as
maximal bin of 256, 300 estimators, maximal depth of 5, learning rate $0.1$,
row subsampling of 0.3, feature subsampling of 0.7, and $\ell_{2}$ penalty with strength 1.0.
- 3.
*Histogram-based Gradient Boosting (HGB).* We train boosted trees with a histogram grow policy, subsampling of observations and features, and $\ell_{2}$ regularization. Class imbalance was addressed via the standard positive-class weight $\frac{n_{\text{neq}}}{n_{\text{pos}}}$, estimated from the training examples. Hyperparameters (learning rate, depth, estimators, subsampling ratios) were fixed based on validation performance and kept constant across comparisons. Hyperparameters were set to maximal depth of 4, learning rate $0.07$, and $\ell_{2}$ regularization with strength $0.5$, and at most 350 iterations.
- 4.
*Multi-layer perceptron (MLP).*
We used a feed-forward network with two hidden layers of sizes 384, 192 and ReLU activations, trained with weight decay and early stopping on a validation split. Weight decay was set to $\alpha=3\cdot 10^{-3}$, batch size $512$, adaptive learning rate with initial value of $10^{-3}$,
early stopping with validation fraction $0.12$ and no change for 12 iterations, maximal number of iterations as $300$, and tolerance $10^{-4}$.

### C.4 UCI experiments

#### C.4.1 Data and processing

We evaluated IA-BMA on standard UCI tasks retrieved from OpenML.
We chose datasets with relatively large number of observations and features.
For *classification*, we used spambase (target: class) and credit-g (target: class).
For *regression*, we used bike-sharing (target: cnt) and california-housing (target: MedHouseVal).

We replaces common “unknown” tokens (e.g., ?, NA, NaN, unknown) with missing values, stripping whitespace on string columns in each dataset, and dropped features whose missing rate exceeded $40\%$.

We used an $80\%/20\%$ train-test split in each repetition.
For classification, we performed stratified sampling on the label to preserve class proportions in the test set, and then balanced only the training split by downsampling the majority class to the minority size.
For regression, we created an approximately balanced split by binning the continuous target into $12$ quantile bins and stratifying on those bins.
All pre-processing statistics (imputation, scaling, and one- hot vocabularies) were computed on the training partition and applied unchanged to the test data.

To encourage diversity among base models, we formed several heterogeneous, partially overlapping *feature bundles* and trained each model on a bundle tailored to its strengths.
Bundles were constructed from the training data as follows:

- •
B1: up to 3 Continuous features with highest absolute Pearson correlation with the target (continuous median-imputed for this computation).
- •
B2: up to 3 highest-variance Continuous features.
- •
B3: up to 3 categorical features with highest cardinality.
- •
B4: up to 5 remaining low-cardinality categorical variables.
- •
B5: all categorical features.
- •
B6: all Continuous features.
- •
B7: the union of B1 and B3.

Continuous features in non-tree models were median-imputed and standardized.
Categorical features were imputed to the most frequent level and one-hot encoded with a minimum frequency threshold of $10$ to pool rare levels; unknown categories at test time were ignored.

#### C.4.2 Candidate predictors

All averaging methods were evaluated on the same base models

## Appendix D Implementation details

In all our experiments the posterior network for IABMA and the gating network for MoE were implemented as feed-forward neural networks with hidden layers of size (64, 32, 16) and ReLU activations.
We used Adam optimizer for MoE and IABMA across all experiments.

Hyperparameters for our method and all baselines were tuned via binary search to maximize average performance (accuracy for classification, RMSE for regression) on a held-out repetition excluded from the analysis. The selected values and running times by experiment and method are reported below.

### D.1 Hyperparameters of ensemble methods

**Table 2: Hyperparameters of Mixture-of-Experts**
| Hyperparameter | Synthetic | PRISM (Cancer) | Fraud (IEEE-CIS) | UCI |
| --- | --- | --- | --- | --- |
| Learning rate | $10^{-3}$ | $10^{-3}$ | $10^{-3}$ | $10^{-3}$ |
| Batch size | 64 | 128 | 64 | 64 |
| Epochs | 10 | 20 | 10 | 10 |

**Table 3: Hyperparameters of Dynamic Local Accuracy (DLA).**
| Hyperparameter | Synthetic | PRISM (Cancer) | Fraud (IEEE-CIS) | UCI |
| --- | --- | --- | --- | --- |
| Neighborhood size $k$ | 50 | 50 | 50 | 50 |
| Temperature $T$ | 0.8 | 1.0 | 1.0 | 1.0 |
| Smoothing $\alpha$ | 1.0 | 1.0 | 1.0 | 1.0 |

**Table 4: Synthetic Mixture of Experts (SMC).**
| Hyperparameter | Synthetic | PRISM (Cancer) | Fraud (IEEE-CIS) | UCI |
| --- | --- | --- | --- | --- |
| Confident-cover threshold | 0.6 | 0.6 | 0.6 | 0.6 |
| Cover quantile (reg.) | – | 0.30 | – | 0.30 |
| Min coverage per model | 20 | 20 | 20 | 20 |
| Cov. reg. (reg. mix) | 0.9 (Gaussian scores) | 0.9 | 0.9 | 0.7 |

**Table 5: Bayesian Hierarchical Stacking (BHS).**
| Hyperparameter | Synthetic | PRISM (Cancer) | Fraud (IEEE-CIS) | UCI |
| --- | --- | --- | --- | --- |
| Temperature $T$ | 1.0 | 1.0 | 1.0 | 1.0 |
| Prior weight | 1.0 | 1.0 | 1.0 | 1.0 |
| Slab scale $s_{0}$ | 5.0 | 5.0 | 5.0 | 5.0 |
| Learning rate | $5\times 10^{-3}$ | $5\times 10^{-3}$ | $5\times 10^{-3}$ | $10^{-3}$ |
| Batch size | 64 | 128 | 64 | 64 |
| Epochs | 10 | 20 | 10 | 10 |

**Table 6: Anput Adaptive Bayesian Model Averaging (IA-BMA)**
| Hyperparameter | Synthetic | PRISM (Cancer) | Fraud (IEEE-CIS) |
| --- | --- | --- | --- |
| Learning rate | $10^{-3}$ | $10^{-3}$ | $10^{-3}$ |
| Batch size | 64 | 128 | 64 |
| Epochs | 10 | 30 | 10 |
| KL weight $\lambda_{\text{KL}}$ | 0.05 | 0.2 | 0.2 |

**Table 7: IABMA (PosteriorNet) hyperparameters per UCI dataset.**
| Hyperparameter | Spambase (clf) | Credit-g (clf) | Bike-sharing (reg) | Cal housing (reg) |
| --- | --- | --- | --- | --- |
| Learning rate | $5\times 10^{-3}$ | $5\times 10^{-3}$ | $1\times 10^{-3}$ | $1\times 10^{-3}$ |
| Batch size | 64 | 64 | 64 | 64 |
| Epochs | 10 | 10 | 10 | 10 |
| KL weight $\lambda_{\mathrm{KL}}$ | 0.1 | 0.1 | 0.8 | 3.0 |

### D.2 Runtimes

**Table 8: Method runtimes (seconds): mean (sd) across 10 repetitions.**
| Experiment | MoE | DLA | SMC | BHS | IABMA |
| --- | --- | --- | --- | --- | --- |
| Cancer | 147.359<br>(5.282) | 22.269<br>(0.454) | 0.072<br>(0.114) | 28.572<br>(1.167) | 252.985<br>(5.571) |
| Fraud | 439.502<br>(129.487) | 8.246<br>(1.719) | 688.473<br>(155.629) | 16.622<br>(3.139) | 461.312<br>(121.168) |
| Simulation | 5.664<br>(0.104) | 0.218<br>(0.008) | 0.079<br>(0.004) | 1.040<br>(0.079) | 5.889<br>(0.038) |
| Bike-Sharing | 25.080<br>(3.780) | 0.868<br>(0.179) | 0.007<br>(0.001) | 21.364<br>(1.063) | 29.663<br>(4.094) |
| Cal. housing | 8.510<br>(0.987) | 0.350<br>(0.041) | 0.006<br>(0.001) | 7.281<br>(0.324) | 9.815<br>(1.029) |
| Credit-g | 3.178<br>(0.049) | 0.439<br>(0.017) | 0.174<br>(0.007) | 1.184<br>(0.148) | 3.345<br>(0.048) |
| Spambase | 16.420<br>(0.287) | 0.642<br>(0.025) | 0.822<br>(0.159) | 1.781<br>(0.147) | 18.651<br>(5.122) |

## Peer Reviews

confidence: 1
contribution: 3
initial_rating: 6
presentation: 3
questions: None
soundness: 3
strengths: Cannot assess
summary: The paper studies a Bayesian way to combine different models.
weaknesses: cannot assess

---

confidence: 5
contribution: 2
initial_rating: 2
presentation: 2
questions: How does IABMA differ in substance from Mixture of Experts with a Bayesian treatment of gating? What new insights or properties does the input-dependent prior confer?

How sensitive is the model to the design of the energy-based prior and the range of integration for continuous outcomes?

What guarantees (if any) can be provided for the variational approximation quality or convergence of amortized inference?

Could the same adaptive weighting effect be achieved more simply with a discriminatively trained gating network, without the Bayesian formalism?

What is the computational complexity of evaluating Eq. (9) and optimizing the ELBO in large-scale settings?
soundness: 2
strengths: Conceptual coherence: The probabilistic formulation that derives adaptive weights from a Bayesian posterior is principled and internally consistent.

Empirical breadth: The experiments span both regression and classification, synthetic and real datasets, with comparisons to multiple baselines.
summary: This paper proposes a framework for combining multiple predictive models by assigning input-specific weights in a Bayesian manner. Unlike classical Bayesian Model Averaging (BMA), which uses global model weights, IABMA introduces an input-dependent prior over model-selection functions, leading to input-adaptive posterior weights. The posterior is approximated using amortized variational inference, yielding instance-specific model weightings. The authors derive a finite-sample likelihood guarantee showing that the proposed predictor performs competitively with the best per-input model selector.
weaknesses: 1.	Limited novelty: The proposed approach is largely a Bayesian reinterpretation of existing adaptive ensemble methods such as Mixture of Experts and Bayesian Hierarchical Stacking . The key innovation, introducing an input-dependent prior, is conceptually modest and primarily repackages known ideas in new notation.

2.	Superficial theoretical development: The likelihood guarantee is a straightforward adaptation of Jensen’s inequality, offering minimal insight into the behavior of the amortized inference procedure or its generalization properties. No convergence analysis, uncertainty quantification, or theoretical justification for the variational approximation is provided.

3.	Lack of methodological clarity: The construction of the input-adaptive prior is heuristic, based on an “energy” integral that lacks intuitive interpretation and appears computationally impractical for high-dimensional continuous outcomes. The method relies on ad-hoc Monte Carlo approximations, raising concerns about scalability and stability.

4.	Amortized inference design is under-specified: The paper treats the amortized posterior network as a black box, without ablation or sensitivity analysis on architecture, optimization, or overfitting. It is unclear whether performance gains come from the variational parameterization or the adaptive prior itself.

5.	Empirical evidence is weakly convincing: Improvements over baselines are small and inconsistent across datasets; Comparisons do not include recent or stronger baselines in adaptive ensembling (e.g., deep mixture-of-experts architectures); Some tasks (e.g., PRISM, fraud detection) lack details on train/test splits, data leakage control, and statistical significance of reported differences.

6.	Overstated claims: The paper claims “formal guarantees” and “Bayes-optimal adaptive weights,” but these rely on unverified approximations and assumptions. The results fall short of demonstrating real-world robustness or interpretability advantages.

7.	Expository issues: While the paper is long and dense, it lacks intuition; the heavy notation and abstract measure-theoretic framing (e.g., pushforward arguments) obscure rather than clarify the contribution.

---

confidence: 4
contribution: 2
initial_rating: 2
presentation: 1
questions: Could the authors please clarify how the proposed methods relate to existing literature on mixture of experts and/or input/covariate-dependent Bayesian mixture models? This seems necessary to properly understanding the paper’s contribution, and I didn’t find this sufficiently discussed in the paper.
soundness: 3
strengths: As mentioned, the authors study an important problem of how to best personalized combined models, that is, by taking an input-adaptive approach to model averaging. The approach seems reasonable, well-motivated, and empirical results appear okay. The theoretical analysis seems okay too, although I didn’t yet check it carefully due to the main weakness (see next section), for which is the main factor influencing my evaluation.
summary: The authors study the interesting and important question of how to adaptively combine multiple prediction models, where here “adaptively” refers to a weighted mixture of the models with weights that depend on the test-point input covariates. The goal is both to improve overall performance relative to an individual model, as well as to improve “personalization,” meaning that the best model is used for a given input. The main claimed contribution is to present a *Bayesian* approach to adaptive model averaging--to contrast their contribution from prior work, the authors claim that “Previous adaptive approaches (see Section 1.1) addressed the task of specifying the adaptive weights $\alpha_j(x)$ from a *frequentist* point of view[...]” (emphasis added). The authors claim a theoretical guarantee that compares the proposed approach to the performance of the individual models, and they present experiments on simulated data, as well as on cancer drug-response prediction and credit-card fraud detection.
weaknesses: **Originality/significance:** In my view, the main weakness of the paper is that it does not make clear if/how the proposed methods differ from or improve on prior approaches to input-adaptive model averaging, including Bayesian approaches. In particular, there are at least two large categories of methods that I think at least the related work, and probably also the experiments section, should compare against more thoroughly: (1) mixture-of-experts models and (2) dependent Bayesian mixture models (eg, dependent Dirichlet mixture models or even input-dependent Gaussian mixture models). 

For example, in the related work, the authors claim “Few methods assign input-dependent weights” before only providing one citation on mixture-of-experts, despite this area having a very large literature on data-dependent model averaging. It’s also not clear to me why MoE is not considered Bayesian. Eg, the authors could consult the following review papers:
- Masoudnia, S., & Ebrahimpour, R. (2014). Mixture of experts: a literature survey. Artificial Intelligence Review, 42(2), 275-293.
- Mu, S., & Lin, S. (2025). A comprehensive survey of mixture-of-experts: Algorithms, theory, and applications. arXiv preprint arXiv:2503.07137.
- Yuksel, S. E., Wilson, J. N., & Gader, P. D. (2012). Twenty years of mixture of experts. IEEE transactions on neural networks and learning systems, 23(8), 1177-1193.

Regarding dependent Bayesian mixture models, the authors could consult the following reviews, in particular with an eye to methods that are “covariate-dependent” mixture models:
- Barcella, W., De Iorio, M., & Baio, G. (2017). A comparative review of variable selection techniques for covariate dependent Dirichlet process mixture models. Canadian Journal of Statistics, 45(3), 254-273.
- Quintana, F. A., Müller, P., Jara, A., & MacEachern, S. N. (2022). The dependent Dirichlet process and related models. Statistical Science, 37(1), 24-41.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: Please see weaknesses section.
soundness: 3
strengths: One of the main strengths of the paper is that it is clearly written and structured. The authors lay out their argument in a straightforward manner, making the text and the core concepts of the proposed IABMA method relatively easy to follow.

The paper addresses an interesting and relevant problem in model averaging, particularly for heterogeneous data where input-specific models are needed. The proposal itself is presented in a "clean" and simple probabilistic formulation. The paragraphs are well written, the theory is well motivated and the flow of the paper is clear. The intuitive example in section 3.1 as well as the actual taking care of the problem (e.g. how to fit the variational distributions or how to optimize the KL divergence) are well formulated and explained.

Additionally, the authors provide motivation for their framework by connecting it to practical examples like personalized medicine and fraud detection. They also include some theoretical development to support their method, which helps to frame the potential benefits of the approach.
summary: This paper addresses the challenge/difficulty of combining multiple predictive models, which is especially difficult in heterogeneous settings where different models may be optimal for different inputs. Standard model averaging uses a single set of global weights, which performs poorly in these scenarios. To solve this, the authors propose Input-Adaptive Bayesian Model Averaging (i.e. the IABMA method), a Bayesian method that calculates specific weights for each model conditional on the input data $x$.

The IABMA method models the choice of the best model as a random selector function g that depends on the input x. It uses an input-adaptive prior and then calculates a posterior distribution over which the model is most plausible given the training data ($D$) and the specific input $x$. This posterior distribution, estimated using amortized variational inference, directly gives the optimal, input-specific weights for combining the models. The authors derive formal guarantees for the performance of IABMA and then evaluate it on regression and classification tasks, including popular UCI datasets such as personalized cancer treatment and fraud detection. The results demonstrate that IABMA consistently delivers competitive performance to existing non-adaptive  and adaptive methods.
weaknesses: While the core idea of the paper is clean and simple, this simplicity in my opinion necessitates a much stronger and elaborate set of experiments to fully justify its contribution. The initial motivation and theory are strong, but the paper ultimately falls short due to a weak and insufficiently comprehensive evaluation. This gap between the promising setup and the empirical evidence was a bit disappointing.

In my opinion, the existing experiments are not presented in a convincing manner. For instance, Figures 2 and 3 take a significant amount of space to present results that could be summarized more efficiently in a small table, especially since the paper has some space to spare. More importantly, it is not clear from the results that the authors have obtained if the performance differences between the proposed method and the baselines is statistically significant (it is not readable from the plots). The comparison to other adaptive competitors, such as Mixture of Experts for example, feels underdeveloped. In many cases, the results do not show a lcear advantage for IABMA over others, and the authors missed an opportunity to properly articulate why one would choose their method over existing, well-established alternatives.

Furthermore, the scope of the analysis is quite limited. The authors only consider a small set of candidate models (e.g., four regressors), which leaves several important questions unanswered. A more thorough evaluation could have investigated:
1.  How the method's performance scales with a larger number of candidate models.
2.  How the method performs when several predictors yield similar (or maybe redundant) performance
3.  The magnitude of the performance gain in a setting specifically designed to be highly heterogeneous, where input-adaptive averaging would be expected to clearly outperform input-agnostic methods.

Finally, the paper could be strengthened by doing some rewriting of its analyses and its context. The qualitative analysis in Appendix B.1, for example, is quite interesting and a portion of it should have been included in the main text to provide better insight and overview of the performance. Given the community's shift toward larger datasets, the paper would also benefit from at least one or two analyses on a larger-scale benchmark to demonstrate its relevance and scalability. Also, there is no discussion of computational cost; it is unclear if fitting the amortized variational posterior is more or less costly than maximizing the likelihood in methods like MoE, or how the methods compare in terms of speed.


Decision: Reject