Bayesian Ensemble for Sequential Decision-Making
Authors: Rui Liu
Enmin Zhao
Lu Wang
Yu Li
Ming Pang
Changping Peng
Zhangang Lin
Ching Law
Jingping Shao
Venue: ICLR 2026 Poster (2026)
Keywords: Ensemble Methods
Reinforcement Learning
Primary area: reinforcement learning
TL;DR: We propose Bayesian Ensemble (BE), a layer for ensembles that treats member selection as a bandit problem, learning a sampling distribution over members via Bayesian inference on rewards for efficient exploration.

Abstract: Ensemble learning is a practical family of methods for uncertainty modeling, particularly useful for sequential decision-making problems like recommendation systems and reinforcement learning tasks. The posterior on likelihood parameters is approximated by sampling an ensemble member from a predetermined index distribution, with the ensemble’s diversity reflecting the degree of uncertainty. In this paper, we propose Bayesian Ensemble (BE), a lightweight yet principled Bayesian layer atop existing ensembles. BE treats the selection of an ensemble member as a bandit problem in itself, dynamically updating a sampling distribution over members via Bayesian inference on observed rewards. This contrasts with prior works that rely on fixed, uniform sampling. We extend this framework to both bandit learning and reinforcement learning, introducing Bayesian Ensemble Bandit and Bayesian Ensemble Deep Q-Network for diverse decision-making problems. Extensive experiments on both synthetic and real-world environments demonstrate the effectiveness and efficiency of BE.

Published as a conference paper at ICLR 2026
BAYESIAN ENSEMBLE FOR SEQUENTIAL DECISION- MAKING
Rui Liu', Enmin Zhao' ; Lu Wang', Yu Li 2, Ming Pang', Changping Peng', Zhangang Lin', Ching Law', Jingping Shao' IDD.cm, China
PInstitute of Computing Technology. Chinese Academy of Sciences, China (liurui34, www.eemmiiaa.miinnnmmmm [haneenmin,.wanll224111 @jd.com, liyu ict.aacccn, (pangming8, pewgriigggngggggggingggg.. hiw.gggnnnnnnagggganiinnnggnnnno @jd.ccm
ABSTRACT
Ensemble learning is practical family of methods for uncertainty modeling, par- ticularly useful for sequential decision-making problems like recommendation systems and reinforcement learning tasks. The posterior on likelihood parame- ters approximated by sampling an ensemble member from a predetermined in- dex distribution, with the ensemble's 's diversity reflecting the degree of uncertainty. In this paper, we propose Bayesian Ensemble (BE), lightweight yet principled Bayesian layer atop existing ensembles. BE treats the selection of an ensemble member as a bandit problem in itself, dynamically updating a sampling distribu- tion over members via Bayesian inference observed rewards. This contrasts with prior works that rely on fixed, uniform sampling. We extend this framework to both bandit learning and reinforcement learning, introducing Bayesian Ensem- ble Bandit and Bayesian Ensemble Deep Q-Network for diverse decision-making problems. Extensive experiments on both synthetic and real- wol-world environments demonstrate the effectiveness and efficiency of BE.
INTRODUCTION
Sequential decision makinn under uncertainty lies at the core of recommendation systems and rein- forcement learning (RL) (Sutton Barto 1998), where agents must continually balance exploiting known high-reward actions against exploring uncertain options to maximize long-term returns. This ixploration-exploitaiii
trade-off's is most famously distilled in the multi-armed bandit (MAB) prob- lem. MAB is a foundational RL setting in which agent sequentially pulls of K arms, and each arm is associated with an unknown stochastic reward distribution (Slivkins) 2019: Lattimore & Szeeesvári 2020 Bouneffouf & Rish 2019 Bubeck & Cesa- Bianchi 2012). In order implement efficient exploration (select informative actions or visit different states), the agent must understand the "uncertainty" degree of its acquired knowledge. Therefore, uncertainty modeling often plays central role for these tasks.
Thompson sampling (TS) follows this uncertainty modeling paradigm. It maintains a posterior distribution of the parameters of the likelihood function for all possible arms' rewards (Thompson)
1933 Li al.) 22100) Chapelle & Li 2011: Kawale et al., 2015 Russo et al., 2018). This posterior distribution characterizes uncertainty of knowledge on arms. At each iteration, one sample of the likelihood parameters are drawn from the posterior distribution, and then the sample is utilized to select the "best" arm by calculating the reward expectation for each arm. After the actual reward is observed, the agent updates the posterior distribution.
A possible way to account for the complex relation of each arm's 's context and reward is to use neural networks for modeling the likelihood function. The key challenge lies in the fact that maintaining the exact posterior distribution is often computationally intractable under general circumstances, pecially in the neural network case (primarily owing to entailing exact Bayesian inference). A series
"Equal contribution. Corresponding authors.

---

Published as a conference paper at ICLR 2026
of works address this challenge by approximate Bayesian inference based variational inference, Markov chain Monte Carlo, gradient-based methods and ensemble- based methods (Lu & Roy 2017
Riquelme et al. 2018 Guo et al. 2020 Kveton al., 2019: Zhang al. 202T).
Among the methods discussed, ensemble learning is favored in practical applications due to its straightforward approach, versatility, and demonstrated effectiveness (Zhoul 2012). In ensemble learning, each iteration involves selecting a base model (an ensemble member) at random from a predetermined index distribution predict the reward for each option. These base models represent samples from the prior/posterior distribution of the likelihood parameters, with their variety reflect- ing the uncertainty in predictions. Updating the base models' parameters with the received rewards is tantamount to conducting an approximate Bayesian inference on these distributions. Ensemble- based techniques have been shown to offer substantial benefits for joint predictions and enhance per- formance in sequential decision-making tasks when compared to other uncertainty-modell agents. uncertainty (Snoek et al. 2019 Osband al. 2022).
Current ensemble-based Thompson Sampling methodologies primarily concentrate on refining the model parameters of ensemble members to better approximate the posterior distribution. However, these methods maintain the sampling index distribution (such a discrete uniform distribution) unchanged throughout the recision-making process. This approach does not fully account for the uncertainty inherent in the diversity of the ensemble members. In our setting, the performance of individual ensemble members similarly varies due to factors such as random initialization and the incorporation of prior functions. Therefore, we suggest an update to both the parameters of ensembble members and the index distribution governing them. The parameters are responsible for guiding the actions, while the index distribution is crucial in selecting the appropriate ensemble member. Essentially, we forge a direct nexus between the index distribution and the feedback rewards to achieve a more precise approximation of the posterior distribution.
Our contribution in this paper is two-pronged, Firstly, we propose Bayesian Ensemble (BE) a unified framework adapting Bayesian index distribution updating to both contextual bandits and rein- forcement learning, achieving state-of -fee-----te-rrttt art exploration efficiency through theoretically grounded variance reduction. This technique not only updates the parameters of ensemble members but also forges a direct link between the index and reward distributions by employing Bayesian inference to update the index distribution. BE's versatility allows it to enhance a broad spectrum of ensemble- based Thompson sampling methods and reinforcement learning methods. Secondly, we demonstrate the efficacy of BE in a variety of settings, including both synthetic and real- real-world environments, with a particular focus on recommendation systems within sequential decision-making problems. We conduct a thorough and meticulous comparison with cutting-edge deep Thompson sampling and ensemble reinforcement learning methods, providing comprehensive benchmark that will aid in advancing future research in this area.
RELATED WORK
There is a long history in the research of uncertainty estimation aiming for the decision-making problem, of which the key challenge is balancing exploration and exploitation. Much of the work, as the extension of vanilla Thompson sampling (Thompson) 1933] Russo et al., (2018), maintains an
approximate posterior distribution (used for posterior sampling) rather than making exact Bayesian inference. These methods include variational inference (Blundell et al. (0015), bootstrapping (Os- band et al. 2016), MCMC (Welling & Teh 2011; Ahn et al. 2012 Li al. 2016), dropout (Gal & Ghahraman 2016). ensemble sampling (Lu & Roy 2017) and so on.
Among these promising posterior sampling techniques, ensemble sampling (ensemble-basee TS) is preferable in practice owing to its simplicity and generality (Snock et al.) 2019). Variants of en- semble sampling based on neural networks has attracted considerable attention. To approximate posterior distribution, deep ensembles (Lakshminarayanan et al., (017)) yield predictive uncertainty estimates through multiple neural networks with random initialization. These ensemble members could further improve computing and storage efficiency through shared weights (Wen et al. 2020). Moreover, these ensemble- based approaches can be formalized as sampling an index from a fixed reference distribution and then mapping the index to an ensemble member. Deep ensembles (Lak- shmiiarryaaaa et al. 2017) and ensembles with prior functions (Osband et al. 2018)) sample neural networks from uniform distribution. The index of hypermodel (Dwaracherla al. 2020 is drawn
2

---

Published as a conference paper at ICLR 2026
pl++2)
p(9)
Update distribution by p())
2((+1)
x(
Envirunment
Bayesian Ensemblle
x(1)
x
fo(x,2(0)
base models
Reco Systems & RL lasks
Figure 1: The Bayesian Ensemble (BE) framework. The agent maintains a probability distribution p(t) for the index 2 E Z and bridges the gap between the index and reward distribution.
from a standard Gaussian distribution, and then is mapped to base model parameters with a neu- ral network. Epistemic Neural Networks (Osband et al.) 2023) maps distributional index to epinet which specially quantify uncertainty for the base network. Besides, Bayesian regret bounds of ensemble sampling for the linear bandit problem theoretically analyzed (Qin et al. 2022).
Ensemble++ (Li et al. 2025) proposes a novel shared-factor ensemble architecture with random linear combinations. Different from Ensemble++, method focuses the dynamic Bayesian updating of index distributions. It models member selection as an inner-bandit problem, enhancing uncertainty modeling accuracy by directly linking indices reward distributions.
Besides Thompson sampling, ensemble methods also play an important role in reinforcement learn- ing. After the proposal of Deep Q-Network (DQN) (Mnih al., 2015). Ensemble DQN (Anschell et al.) (217)) is introduced to reduce the variance of the standard DQN. Ensemble DQN uses the average weighted combination of multiple value functions to guide policy updates, making the re- inforcement learning process more stable. Inspired by Dropout, Random Ensemble DQN (Agarwal ict al. 2020) proposes random weighting of multiple value functions to address the distribution mis- match problem in offline reinforcement learning. Reset Deep Ensemble (Kim al. (023), through the periodic reset operation on multiple ensemble agents, demonstrates its effectiveness in sample efficiency and safety considerations. HyperDQN (Li et al. 2022) and HyperAgent (Li et al. 2024b) optimize a hypermodel that parameterizes the last layer of the Q-network for efficient exploration.
3 BAYESIAN ENSEMBLE FRAMEWORK
Sequential decision making involves optimizing long-term objectives through series of choices made under uncertainty. Multi-Armed Bandit (MAB) problems, the goal is to maximize cumu- lative rewards, with actions influencing only immediate outcomes and leaving subsequent environ- mental states unchanged. In contrast, Reinforcement Learning (RL. tasks require accounting for dynamic state transitions while optimizing for long-term cumulative rewards. Under uncertainty, the core learning process entails inferring the underlying objective: in MAB, by estimating un- known reward distributions; RL, by developing policies to guide long-term optimization. The epploration-exploiiation exploitation dilemma remains fundamental challenge for sequential decision decision-making making.
Ensemble-based methods have shown their effectiveness in characterizing uncertainty. The key dis- tinguishing feature of our proposed Bayesian Ensemble, illustrated in Figure []] lies in its dynamic updating of the index distribution for sampling base models via Bayesian inference unlike prior ensemble-based Thompson Sampling methods that maintain a fixed index distribution. This update directly links the index distribution to the feedback rewards through interacting with the environ- ment, leading more precise approximation of the posterior distribution, and thus enhance the performance for uncertainty modeling. Bayesian Ensemble advances beyond simple ensemble ag- gregation techniques by introducing learned, dynamic weighting mechanism for ensemble mem- bers. Within this framework, we elaborate on our approach as applied to sequential decision decision-making problems, particularly in bandit and reinforcement learning contexts.

---

Published as a conference paper at ICLR 2026
4 BAYESIAN ENSEMBLE BANDIT
We first introduce preliminary on the bandit problem setting. The problem is then addressed by our proposed Bayesian Ensemble Bandit (BEB) framework. Finally, we provide two examples on employing BEB to reinforce existing ensemble-based TS methods.
4.1 PRELIMINARY
We consider the multi-armed contextual bandit problem. Let X' denote the action space, and R denote the reward space. every iteration a set of actions 2'( X are provided the agent. Then, the agent selects action 2:(1) E 2*(1) and derives the reward y-(1) E R. Finally, (x:(0, ,()) is
utilized to update the agent policy. The objective is to maximize the cumulative rewards Ee r())
4.2 BEB FRAMEWORK
We focus on the finite reward space with N distinct rewards, denoted by R. {R; We utilize an ensemble of neural networks
E E x x
to model the probability distribution of the reward, where E X is the context vector of an action, 2 E is the index of a base model, E θ are the learnable parameters of the neural network ensemble, and A is the regular N 1 dimensional simplex. Each of the coordinates of A is the estimated probability corresponding to a possible reward value, i.e.,
Prfr- = R,) f(x; O), VI E [N].
(2)
At each iteration t, the agent maintains a probability distribution p(t) for the index 2 E Z and the parameters g(t) The index 21)) sampled from pd)) Then, the agent selects the action ((() E x(()
that maximizes the expected reward estimated by the ensemble:
(0)) arg max 2E
Σ
R4 f(1;2(0,,000)
(3)
and derives the reward ((), Finally, the instance (z:(*), ((1) is utilized to update the agent policy, i.e.. g(t+1) and ((++1), The overall algorithm Bayesian Ensemble Bandit (BEB) is shown in Algorithm|
Algorithm 1: Bayesian Ensemble Bandit (BEB)
Input: Initialization of learnable parameters ((0) of neural network ensemble, index
distribution (0)
for f = 0 to T - do
Receive 20(0;
Sample (() p(t);
(1)) + arg 20)) En R4
Apply (!) and observe (!);
Update ((++1) and p(t+1);
While maximizing cumulative rewards is the ultimate goal in multi-armed contextual bandit prob- lems, we often resort to minimizing surrogate loss function when learning model parameters, like in this case. This trade-off is primarily made due to computational efficiency, and aligns with existing ansemble-based methods. Formally, let f(.,-) R x N + R represent the loss function,
In fact, it is easy to see that the actual reward space could be R; only the space over which the predicted distribution required he finite.
4

---

Published as a conference paper at ICLR 2026
which is typically task-driven, and the binary cross-entropy loss is commonly used. Given train- ing data D S X' R. we leverage the Empirical Risk Minimization (ERM) principle to learn O. This involves minimizing the expected value of the empirical risk over probability distribution p governing the model index 2. Therefore, we estimate by solving the following optimization problem:
min BEB
Σ (2.))D
Exup (a;;,, 0))]
Learning the index distribution marks a critical departure from existing ensemble-base Thompson Sampling methods. We exploit the observation that the number of parameters in p (index distribu- tion) is typically much smaller compared to (ensemble parameters). This allows us to directly optimize p for long-term cumulative reward, eliminating the need for surrogate losses. Essentially, we bridge the gap between the index and reward distribution without relying on surrogate loss func- tions.
While specifics depend on the chosen structure, at each iteration, probabilistic techniques like exact or approximate Bayesian inference can be employed for updating p. Careful design of p based on the specific case is crucial for effective application.
4.3 UNIFYING BANDIT METIIODS
Many existing ensemble-basee Thompson Sampling methods can be captured as special cases of BEB. The crucial difference lies in their treatment of the index distribution p. Prior methods often fix throughout the decision-making process. BEB offers a powerful way to "supercharge" these methods by allowing p to be dynamic and adaptive through the learning process.
Furthermore, we can categorize ensemble- baased methods based on how they define the index conti- nuity. To illustrate this, we will showcase representative examples: discrete and continuous ensemble-basee Thompson Sampling method. Their enhancements when combined with BEB will be further evaluated in experiments.
ensembler e (Osband et al. 20118) is variation of deep ensembles [Lakshminarayanan et al. 2017) that incorporates randomized prior function into each member. These priors, implemented as randomly initialized neural networks with identical architecture, remain fixed throughout training and inference. They have proven effective for generating uncertainty estimates. Originally, the index distribution p in ensembles e was a discrete uniform distribution:
where K the number of ensemble members. Under the BEB framework, we enhance ensemble+ by transforming z to = = arg ma Wi, where the weights (w:) represent individual ensemble members and are drawn from Beta distributions:
Discrete
Beta(a,, 3.). This choice is motivated by the fact that the Beta distribution the conjugate prior the Bernoulli likelihood of the rewards, which enables efficient and exact Bayesian updates. The update rule follows Thompson sampling (Russo) et al. 2018). Through Bayes' theorem, if member i is selected at step and receives reward p(( E {0, T), the index distribution p can be updated with exact Bayesian inference: (o;, B.) (az, B,) +
(r(0) (())
hypermodel Dwwracherla et al. 2020) is a variation of deep ensembles (Lakshminarayanan et al., 2017) that treats ensemble members as specific parameter values of base model class. It em- ploys hypernetworks (Ha et al. (017) to map an index to a specific ensemble member. Originally, hypermodel uses the multivariate standard Gaussian distribution (z, NO, 1) for each i in the index vector) as its index distribution. As the hypermodel framework maps continuous, multi-dimensional index vector to model parameters, with multivariate Gaussian. This allows us to learn its mean and variance parameters via approxi- mate inference, extending the original method's use fixed standard Gaussian. Therefore, under the BEB framework, we instead employ Gaussian distributions with individual means μi and vari- ances of for each index component. This enables us to learn and adapt these parameters separately throughout the process. Similar to other BEB implementations, the update rule follows Thompson sampling (Russo et al., 2018). However, due to the complexity of the model, exact Bayesian infer- ence becomes impractical. To address this, hypermodel utilizes variational inference (Blundell
natural to model the index distribution
5

---

Published as a conference paper at ICLR 2026
et al. 2013) approximate the posterior distribution by updating the means and variances of the individual Gaussian components.
5 BAYESIAN ENSEMBLE REINFORCEMENT LEARNING
Besides bandit problems, Reinforcement Learning represents classic sequential decision mak- ing scenario. Prior research has investigated Ensemble-DQQ (Anschel et al. 2017) and Random Ensemble-DQN (Agarwal et al. 2020), demonstrating their capability to effectively reduce the vari- ance of Q-value estimates compared to standard DQN algorithms, which significantly stabilizes the learning process. However, we posit that the conventional practice of applying random or uniform weighting schemes across different evaluators fails to fully exploit their complementary capabilitiess and optimize collective performance. Similar to the idea of Self Imitation Learning (Oh et al. 2018),, propose that making better use of superior evaluators can help the agent obtain more cumulative rewards during the training process, thereby facilitating the agent to explore more effectively.
We consider the reinforcement learning (RL) framework formalized as Markov Decision Process (MDP), defined by the tuple .M- = (S, A, P.R.P. R, 7). = {Sy. $1....,$u) Sn. is the state space and A fag. 01 ................................ am } is the action space, denoting the set of states and actions available to the agent. P(s"|s, (. S A x + [0, 1] is the transition probability function. After transitioning from to s' via action a, the agent receives a reward r. Y [0, 1) is the discount factor that balances the importance of immediate and future rewards. At each time step i, the agent observes the current state s, 8, selects an action a, A according the probability of taking action a in state s. The environment then transitions to new state 5t-1 ~ P(-[st, az) and emits reward rs. The agent's objective is learn a policy и that maximizes the expected cumulative discounted reward: Ex 1Er= 7'r, so). To evaluate and improve policies, the action-value function Q (s, a) = E, 1Σn= / = og = a). These functions satisfy the Bellman equations. In particular, the optimal action-value function (s a) satisfies the Bellman optimality equation: (s,ssa a) = ((-ss())s..) [r maxx' (s' a The optimal policy x can be derived by acting greedily with respect Q"
policy π (a]s)), which represents
Inspired by the accelerated policy learning through mentiiperspeectvee value estimations in partially observable Markov decision processes (Li et al.) 2024a). we propose Bayesian Ensemble Deep Q- Network (BE-DQN) by integrating Bayesian principles into reinforcement learning training. BE- DQN maintains K independent Q-networks {Q(s, a; } in iteration i each Q-network parameterized by weights f/. These networks are trained jointly using the standard DQN objective, but with shared experience replay buffer.
Following the ensemble approach, we maintain a Beta distribution wy ~ Beta(an: 3x) for each k Q-network. At each iteration, BE-DQN samples probability (wi, "2,... wx) according to the Beta distribution of each Q-network, and selects the j-th Q-network with the maximum probability
for action selection. At the end of each iteration, the agent receives a reward (1) E {0. 1}. BE-DQN subsequently updates the j-th Beta distribution parameter through (ay, 3,) + (ay, 8,) + (r(9,
r()), where the update mechanism explicitly links reward feedhack with distribution adaptation.
In contrast to Ensemble DQN which estimates state action values ((s, a) through averaging outputs from multiple Q networks, BE-DQN employs a single Q-network sampling mechanism for action selection while utilizing weighted averaging across all Q-networks for objective function computa- tion. This update mechanism effectively mitigates the overestimation bias commonly induced by individual Q-functions (Van Hasselt et al., 2016 Anschel et al., 2017] Agarwal et al.) 2020), while maximizing exploitation and exploratory behavior (Ecoffet et al. 2021). Define the weight of the k-th model as P Σ wa,
w
the sum weight of all model is ΣK k= Pk 1. The BE-DQN target Q
network is defined as:
Veaa Eg(r + max Σ Pa P of
(5)
and the Target Approximation Error (TAE) is:
zikk Q1,, vaa.
(6)

---

Published as a conference paper at ICLR 2026
Algorithm 2: Bayesian Ensemble DQN (BE-DQN)
Initialize K Q-networks (s,, @h) with random weights 0 for & E (1,.... K} Initialize Experience Replay (ER) buffer B Initialize exploration procedure Explore(-)
for iteration i to N do Sample the probability {wi ..... w w from each distribution Beta(ar, Bx); Update weights - ... Pk 1)
Choose the index j with the max probability j = arg max; Pk 1-1 for time t= 1 to T do
Choose action using ((s, a: B.):
Vs.a = RB |r + max Σx=
al
for l: = 1 to do
0 arg ming Es[(Vs,a Q(+, a; 0;));
Explore(-), update B;
Update the j-th distribution (a,, t (ay, Bj) + (r(0, 100,111100 (r());
Output Q,1t
B.EDDNN DQN (ONE:ONNss a) = Ek=IPhQ( (s, (.))
We provide the algorithmic implementation in Algorithm 2
To demonstrate the effectiveness of Bayesian Ensemble DQN, we employ simple M states uni- directional MDP environment shown as Figure 2 Anschel et al. 2017). We analyze TAE Z is random processes with specific statistical characteristics. The expected value of TAE E[ is 0, the variance TAE is given by Var[Zi s,al o², and the covariance between D and Z3
vi vi j.ic K]. e [1. To focus exclusively on the TAEs, we eliminate the overestimation error by considering fixed policy for updating the target values. This ensures that any bias introduced by the choice of policy does not interfere with the analysis of TAES. By fixing the policy, we isolate the effects of temporal errors from other sources of uncertainty. Furthermore, we conveniently assume zero reward (r 0) everywhere in the environment. This assumption has no effect on the variance calculations because the reward does not contribute to the variance of the TAFs. In this environment, the Q value of DQN (Anschel et al. 2017 for
Cov[Z,, 24,,rl
i M
DON (s0, a) Zigaa + + ... + -M Zi-(M
and the variance is
M
Var Q
DQN
(s, a)]
?
2m 0
(8)
m=0
The variance of Ensemble DQN Anschel el al. 2017 is:
M K
E (50. a)]
?
2m
02
(9)
m=0
For the variance of the BE-DQN, we establish the following two theorems.
S+
St
Steur
Ste-1
Figure A M states unidirectional MDP environment with a zero reward any state.
7

---

Published as a conference paper at ICLR 2026
Theorem 1. The variance of BE-DQN V (su. a.] has upper bound matching the vari- ance of DQN, and with a lower bound matching the variance of Ensemble DQN under the same K, i,e, VarlQ, (.ED-DN (8o, a)] E [VarlQ, (so; a]], VarlQ,, (so, a)]l-
Theorem 2. The variance of each Vj E [K BE-DQN Q-value estimator VarlQ, (so, a, 0:)] has a upper bound matching the variance of DQN, and with a strict lower bound matching the variance of Ensemble DQN under the same K, i.e., VarlQ, (5o, a, 0,)] (VarlQ,-DeN (so, a)), VarlQ? (s0, a)]]-
See Appendix A. for detailed proofs. Theorem 1 [and 2 ensure that the variance of BE-DQN does not exceed that of standard DQN, thereby enhancing training stability and convergence - a critical property for reliable deep reinforcement learning in complex environments. Besides, com- pared with E-DQN, BE-DQN improves exploratory behavior by leveraging single Q-function for action selection at each step. This design encourages temporally coherent exploration while main- taining sufficient behavioral diversity across ensemble members, effectively balancing exploration with stable exploitation.
6 EXPERIMENTS
Our experiments span both Neural Testhed (Osband et al. 2022) and Mushroom (Schlimmer 1981 synthetic environment and real-world (Yahoo!R6B (Li et al. 201 T)) sequential Dociison-making en- vironments, echoing prior works (Osband et al.) 2022 Dwaracherla et al., 2020; Osband al. 2223 Song et al. 2021 Wu et al. 2022 Riquelme et al. 2018 8). In addition, we use MiniGrid (Chevalier- Boisvert et al., (023) as the testing environment for reinforcement learning. To demonstrate the effectiveness and efficiency of our proposed BEB and BE-DQN, we compare its regret, elapsed wall time, cumulative rewards and performances against strong baselines in these diverse environments. See the Appendix for detailed experiment settings.
6.1 RESULTS ON SYNTHETIC ENVIRONMENTS
In synthetic sequential decision making environments, the true reward distributions are given, which enables us to evaluate benchmark agents by calculating regrets, i.e., the cumulative reward differ- ence from the best agent. We validate our methods on Neural Testbed (Osband et al.) 2022) and Mushroom dataset (Schlimmer, 1981).
Neural Testbed (Osband et al. 2022) is an open-source library that synthesizes sequential decision- making problems by simulating an environment with data generated by neural network-based gen- erative models. Each environment corresponds to 2-layer fully-connected neural network with ReLU as the activation function, with independent Xavier initialization in each hidden layer, and with two nodes in the output layer, i.e., f X' - 2 the environment serves for the reward distribution. At each iteration f, the candidate arm set (+) of K arms are indepen- dently sampled from the standard Gaussian distribution, and the reward (()) of the selected arm (()) is sampled from f (*)) In our experiment, the size of the hidden layer of the generative model set 50, the number of actions K 20d, and the time horizon T 000. To some extent, the dimension of arms (actions) d represents the complexity of the problem. Therefore, we set the to show the superiority of our method
ground truth
dimension of arms (actions) d E (2, 10, 50} (i.e., X' = R over the state-of-the artt for uncertainty modeling - ensembles and hypermodel, in wide range of complexity degrees.
Different from Neural Testbed, the Mushroom dataset (Schlimmer 1981 is real-world classifica- tion dataset that contains 8,124 instances with 22 attributes and 2 classes: poisonous and edible. We synthesize the decision-making process by letting the agent decide whether to select given mush- room round by round following previous benchmarks (Riquelme et al.) 2018 Zhang et al. (021) The agent would receive reward 1 if edible mushroom is selected and reward 0 otherwise. We sample 20,000 data points in the experiment.
The cumulative regrets illustrated in Figure It can be obviously observed that our models (BEB-based) outperform their corresponding baselines significantly, across all environments with different settings on the dimension throughout the whole decision-making process (at every step).
8

---

Published as a conference paper at ICLR 2026
Iopermdel
Iyperated
230
60-
aveeblel!
1.5
70
40
20003
10000
13000
2000
3000
(a) Neural Testbed: d
(b) d = 10
typeradel
175
3ypermsequsun))
10
1.5
1.29
100
hyporneel
40
hyperodermatt))
conctbler
(193tMe-18BB)
20
3000
10000
15000
2000
5000
10000
15000
20000
cape
(c) d = 50
(d) Mushroom
Figure 3: Regrets (the smaller the better) on Neural Testhed and Mushroom with 90% confidence interval.
Specifically, at the end of the drcision-making process, for d 2, 10, 50, ensemble- (BEB) outperforms ensembler e+ by 37.0%. 12.8%, and 42.2%: hypermodel (BEB) outperforms hypermodel by 69.8%, 22.8%, and 30.3%. The results the Mushroom dataset similar (also shown in Figure (d)) (d)). BEB models win at every step, and at the end the decision-making pro- cess, BEB-based ensemble+ and hypermodel outperform their baselines by 8.7% and 4.8%, respectively. These results indicate BEB is able to boost the exploration efficiency for sequential decision-making problems in these purely simulated environments.
Compared with enemblet hypermodel, as the parameters of the index distribution are required to be updated additionally for BEB models, one may concern whether the performance gain deserves the extra time cost. Therefore, we demonstrate BEB's efficiency on Neural Testbed with different dimensions in Table [ For enemblet, owning to the conjugacy property, updating the index distribution is efficient via exact Bayesian inference. Thus in all cases, the clapsed time for enemble+ + (BEB) roughly equivalent to enemble+. In the case of hypermode since exact Bayesian inference is intractable, computational overhead the elapsed time for hype rmodel (BEB) increases by 20%. As expected, the additional overhead of BEB actually depends the updating rule for the index distribution, which is highly related to its based Thompson sampling method.
and
Table 1: Elapsed wall time(s) Neural Testbed
d
50
H- e-to-mbble ae
574.33
16.47
390.50
1.76
1165.07 + 14:00
end (BEB)
374.22
17.89
389.56
1.90
1162 + 38.65
model
41.83
1.76
43.25 + 1.68
00.6 1.18
51.05
1.64
55.75
1.26
$1.20 1.27
resort variational inference which incurs additional
6.2 RESULTS ON REAL-WORLLD ENVIRONMENT
Extending our analysis, we follow prior works (Gal & Ghahramani 2016 Song et al., 2021 Wu al. 2022) and conduct further experiments on the Yahoo!R6B dataset (Li et 2011). This dataset offers 28 million user visits to Yahoo!'s Today Module over 15 days, where each visit presents four key features: (1) User Information: A 36-dimensional binary vector capturing demographics like age and gender; (2) Displayed Article: The ID of the article presented to the user; (3) Candidate Articles: A set of possible articles from which the displayed article was randomly chosen: (4) Click Label: A binary indicator of whether the clicked the displayed article. We adopt the experimen-

---

Published as a conference paper at ICLR 2026
tal setup of [Song al. (2021): hyperparameter tuning day entries and performance evaluation on the remaining 14 days. As ground truth rewards are unavailable, we use the cumulative number of clicks (rewards) instead of regrets.
Due to the significantly higher computational demand of ensemblet compared to other methods, we focus solely on the hyperrodes in this experiment. Table 5 showcases the results on the Yahoo!R6B dataset. Notably, our method delivers a significant increase in click-through rate compared to both non-ensemble and ensemble- based methods. These compelling results demonstrate BEB's s superior exploration efficiency in a real-world sequential decision-making setting, further solidifying the effectiveness of our proposed approach.
Table 2: hoo!R6B
Cumulative rewards on Ya-
Namber of clicks
Relative (mp
Kanaues
25,413.1 821
-1X -58.15%
Groedy
38,423.3 #: 1,397.9
-21.6556
r-rredy
45,645.0 :: 1,40766
-15268
MC Mnopopn
45,007.2
705.0
-S.77%
Oratien TS
47,083.7
$49.0
-6.55%
hyge moste
40,676.8
1,358.7
Ryge moppomo (BULC))
300322.1
1,8812
6.3 RESULTS ON MINIGRID
To evaluate the efficacy of BE-DQN across diverse tasks, we conduct experiments in generated Box2D environments FourRooms, Empty-6x6, LavaGapS5, GoToDoor-5225 and NuliRoom-N2-S4 using the MiniGrid framework Sievvalerr-Biivedr et al.] 2023). As illustrated, agents with tri- angular avatars operate in discrete action spaces to achieve goal-oriented objectives. These tasks require dynamic interaction with environmental objects and navigation through procedurally gener- ated maze configurations that vary episodically.
We compare BE-DQN with four baselines: vanilla DQN (Mnih et al. 2015). Ensemble DQN (E- DQN) (Anschel et al. 2017). Random Ensemble DQN (RE-DQN) (Agarwal et al. 2020). and Un- biased Asymmetric Actor-Critic (UAAC) (Baisero & Amato 2022). UAAC is a statc-of-the-an (SOTA) method for addressing partially observable Markov decision processes. To ensure exper- imental rigor, we maintain consistent network architectures across all methods and report mean performance metrics with standard deviations per environment. The average score across all games is provided in Table 3 to demonstrate the overall performance for different methods. DQN ex- hibits limited robustness across most environments, primarily due to its reliance on single Q-value estimator. While past ensemble methods E-DQN and RE-DON partially mitigate this issue, their performance remains unstable owing to static weighting schemes that inadequately prioritize high- quality Q-estimators. BE-DQN, in contrast to RE-DQN, dynamically updates the distribution of each Q-value estimator and further exploits the individual strengths of estimators. Although the utilization of superior estimators promotes accelerated exploration and exploitation by the agent, the variance associated with it presents challenges for policy optimization. BE DQN effectively surmounts these limitations and stands out as a formidable competitor among the baseline meth- ods in reinforcement learning. Also, we compare Bayesian Ensemble DQN with UAAC, BE-DQN significantly outperformed UAAC, which further validates the effectiveness of BE-DQN.
Table 3: Performance of agents on MiniGrid after 1e5 frames training. We report the average rewards and standard error for each method. All results are averaged over 5 random seeds, with each seed tested for 100 episodes.
FourRooms
Empty-6x6
LavaGapS5-6x6
Go'ooDoo-55x5
Multiooon-N--84
vanilla DQN
0.004 = 0.008
0.026 2 0.0265
0.026
# 0.032
0.086 = 0.0543
0.002 1002=10 0.0040
Ensemble DQN
0.012 = 0.0117
0.162
0.1546
0.178
0.118000.1000
0.12 = 0.0358
0.012 0.0325
Random Ensemble DQN
0.01 :: 0.0063
0,186
1: 0.1540
0.12:.0.1330
0.128 1 0.0627
0.03 0.0200
Unbiused Asymmetric Actor Critie
0.036 4 0.0314
0.082
4: 0.1541
0.022 0.0075
0.106 4 0.0162
0.004 0.0049
Bayesian Ensemble DQN
0.04 0.028
0.248 0.1444
0.33 0.066
0.142 = 0,0407
0.11810.22
7 CONCLUSION
This paper proposes Bayesian Ensemble (BE), a unified framework that enhances ensemble-based methods for uncertainty modeling. By establishing direct Bayesian connection between the index distribution of ensemble members and the reward distribution, BE dynamically refines both the parameters of base models and the sampling strategy governing their selection. We instantiate this framework in two critical domains, bandit learning and reinforcement learning. Extensive empirical evaluation across diverse decision-making problems demonstrates BE's effectiveness and efficiency.
10

---

Published as a conference paper at ICLR 2026
8 REPRODUCIBILITY STATEMENT
To enhance the reproducibility of our work, we have made comprehensive efforts to document and share all relevant resources. The source code for our experiments is provided as supplementary material. Detailed experimental configurations, model hyperparameters, and procedural descriptions are thoroughly documented the Appendix. Furthermore, complete proof' for theoretical claims and theorems presented in the paper are included in the Appendix to ensure verifiability.
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference Machine Learning, pp. 104-1 14, 2020.
Sungjin Ahn, Anoop Korattikara Balan, and Max Welling. Bayesian posterior sampling via stochas- tic gradient fisher scoring. In International Conference on Machine Learning, pp. 1-8, 2012.
Horst Alzer. refinement the cauchy-schwarz inequality. Journal of mathematical analysis and applicatioon, 168(2):596-604, 1992.
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning. In International Conference on Machine Learning, pp. 176-185, 2017.
Andrea Baisero and Christopher Amato. Unbiased asymmetric reinforcement learning under partial observability. In International Conference on Autonomous Agents and Multiagent Systems, pp. 42-52, 2022.
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In International Conference Machine Learning, pp. 1613-1622, 2015.
Djallel Bouneffouf and Irina Rish. survey on practical applications of multi-armed and contextual bandits. CoRR, abs/1904. 10040, 2019.
Sébastien Bubeck and Nicolo Cesa-Bianchi. Regret analysis of stochastic and nonstochastic multi- armed bandit problems. Found. Trends Mach. Learn., 5(1):1-122, 2012.
Olivier Chapelle and Lihong Li. An empirical evaluation of Thompson sampling. In Advances in Neural Information Processing Systems, pp. 2249-2257, 2011.
Maxime Chevalier-Boisveed Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. In Advances in Neural Information Processing Systems, pp. 73383-73394. 2023.
Vikranth Dwaracherla, Xiuyuan Lu, Morteza Ibrahimi, Ian Osband, Zheng Wen, and Benjamin Van Roy. Hypermodels for exploration. International Conference on Learning Representattions, pp. 1-9, 2020.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O Stanley, and Jeff Clune, First return, then explore. Nature, 590(7847):580-58, 2021.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In International Conference on Machine Learning, volume 48, pp. 1050-1059, 2016.
Dalin Guo, Sofia Ira Ktena, Pranay Kumar Myana, Ferene Huszar, Wenzhe Shi. Alykhan Tejani, Michael Kneier, and Sourav Das. Deep Bayesian bandits: Exploring in online personalized rec- ommendations. In ACM Conference Recommender Systems, pp. 456-461, 2020.
David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. International Conference on Learning Representations, pp. 1-8, 2017.

---

Published as a conference paper at ICLR 2026
Jaya Kawale, Hung Bui, Branislav Kveton, Long Tran Thanh, and Sanjay Chawla. Efficient Thomp- son sampling for online matrix-factorizztion recommendation. In Advances Neural Information Processing Systems, pp. 1297-1305, 2015.
Woojun Kim, Yongjae Shin, Jongeui Park, and Youngchul Sung. Sample-efficient and safe deep reinforcement learning via reset deep ensemble agents. Advances Neural Information Pro- 2023.
cessing Systems, pp.
Branislav Kveton, Manzil Zaheer, Csaba Szepesvári, Lihong Li, Mohammad Ghavamzadeh, and Craig Boutilier. Randomized exploration in generalized linear bandits. In International Confer- ence on Artificial Intelligence and Statistics, pp. 2066-2076, 2019.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, pp. 1-8, 2017.
Tor Lattimore and Csaba Szepesvári. Bandit algorithms. Cambridge University Press, 2020.
Chunyuan Li, Changyou Chen, David Carlson, and Lawrence Carin. Preconditioned stochastic gra- dient langevin dynamics for deep neural networks. In AAAI Conference on Artificial Intelligence, pp. 1188-1794 1794, 2016.
Jinqiu Li, Enmin Zhao, Tong Wei, Junliang Xing, and Shiming Xiang. Dual critic reinforcement learning under partial observability. Advances in Neural Information Processing Systems, pp- 116676-116704, 2024a.
Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bande approach to personalized news article recommendation. In International Conference on World Wide Web, pp- 661-670, 2010.
Lihong Li, Wei Chu, John Langford, and Xuanhui Wang. Unbiased offline evaluation of contextual- bandit-based news article recommendation algorithms. In International Conference Web Search and Web Data Mining, pp. 297-306, 2011.
Yingru Li, Jiawei Xu, Lei Han, and Zhi Zhi-Quan Luo. Q-star meets scalable posterior sampling: Bridg- ing theory and practice via HyperAgent. In International Conference Machine Learning, pp- 29022-29062 2024b.
Yingru Li, Jiawei Xu, Baoxiang Wang, and Zhi- QuanQunn Luo. Scalable exploration via ensemblet+. In Advances in Neural Information Processing Systems, pp. 1-14, 2025.
Ziniu Li, Yingru Li, Yushun Zhang, Tong Zhang, and Zhi-Quan Luo. HyperDQN: A randomized exploration method for deep reinforcement learning. In International Conference on Learning Representations, pp. 1-13, 2022.
Xiuyuan Lu and Benjamin Van Roy. Ensemble sampling. In Advances in Neural Information Processing Systems, pp. 3258-3266, 2017.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver. Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):5295553 2015.
Junhyuk Oh, Yijie Guo, Satinder Singh, and Honglak Lee. Self-imitation learning. In International Conference on Machine Learning, pp. 3878-3887, 2018.
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in Neural Information Processing Systems, pp. 1-9, 2016.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. Advances in Neural Information Processing Systems, pp. 8626-8638, 2018.
Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Xiuyuan Lu, Morteza Ibrahimi, Dieterich Lawson, Botao Hao, Brendan O' O'Donoghue, and Benjamin Van Roy. The neural testbed: Evaluating joint predictions. Advances in Neural Information Processing Systems, 15:12254-12566 2022.
12

---

Published as a conference paper at ICLR 2026
Ian Osband, Zheng Wen, Seyed Mohammad Asghari, Vikranth Dwaracherla, Morteza Ibrahimi, Xi- uyuan Lu, and Benjamin Van Roy. Epistemic neural networks. In Advances Neural Information 2023.
Processing Systems, pp.
Chao Qin, Zheng Wen, Xiuyuan Lu, and Benjamin Van Roy. An analysis of ensemble sampling. In Advances in Neural Information Processing Systems, pp. 21602- 21614, 2022.
Carlos Riquelme, George Tucker, and Jasper Snoek. Deep Bayesian bandits showdown: An empiri- cal comparison of Bayesian deep networks for Thompson sampling. In International Conference on Learning Reprexentations, pp. 1-9, 2018.
Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. A tutorial on Thompson sampling. Found. Trends Mach. Learn., 11(1):1-96, 2018.
Jeff Schlimmer. Mushroom records drawn from the audubon society field guide to north american mushrooms. GH Lincoff (Pres), New York, 1981.
Aleksandrs Slivkins. Introduction to multi-armed bandits. Found. Trends Mach. Learn., 12(1-2): 1-286, 2019.
Jasper Snoek, Yaniv Ovadia, Emily Fertig, Balaji Lakshminarayanan, Sebastian Nowozin, D. Scul- ley, Joshua V. Dillon, Jie Ren, and Zachary Nado. Can you trust your model's uncertainty? eval- uating predictive uncertainty under dataset shift. In Advances in Neural Information Processing Systems, pp. 13969-139888 2019.
Yuhai Song, Lu Wang, Haoming Dang, Weiwei Zhou, Jing Guan, Xiwei Zhao, Changping Peng, Yongjun Bao, and Jingping Shao. Underestimation refinement: A general enhancement strat- egy for exploration in recommendation systems. In ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1818-1822, 2021.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 1998.
William R Thompson. On the likelihood that one unknown probability exceeds another view of the evidence of two samples. Biometrika, 25:285-299 2944, 1933.
Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q- learning. In AAAI conference artificial intelligence, pp. 2094-2100, 2016.
Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In International Conference on Machine Learning, pp. 681-688, 2011.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach efficient ensemble and lifelong learning. In International Conference on Learning Representations, pp- 1-13, 2020.
Kailun Wu, Weijie Bian, Zhangming Chan, Lejian Ren, Shiming Xiang, Shu-Guang Han, Hongbo Deng, and Bo Zheng. Adversarial gradient driven exploration for deep click-through rate predic- tion. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2050-2058, 2022.
Weitong Zhang, Dongruo Zhou, Lihong Li, and Quanquan Gu. Neural thompson sampling. In International Conference Learning Representations, pp. 1-12, 2021.
Zhi-Hua Zhou. Ensemble methods: foundations and algorithms. CRC press, 2012.
A APPENDIX
A.1 THEORETICAL PROOF
In this section, we provide the detail proof of Theorem 1 and Theorem
13

---

Published as a conference paper at ICLR 2026
Theorem 1. The variance of BE-DQN VariQu (su. a.] has upper bound matching the vari- ance of DQN, and with a lower bound matching the variance of Ensemble DQN under the same K, i,e, VarlQ, Rer (8o, a)] E [VarlQ, (so, a)l, VarlQ, 10 (so,
Proof. The Q value of BE-DQN for >> M is:
QEEDDNN
(80,
(s0, a)
Pk Zia Veo.a
K
K
Pk s0.0
7ik
+1
Σ
Ph: (s 1#
k=1
(10)
K
K
Pk Zike
+
Σ
Pk: Zz 1,8
Ph. 2 2 2
k=1
k=1
k=1
M-1
71-(M-1),
+
+ ?
Ple 1M
the variance is:
M-1 Σ
Var QuE DE.DON (so, a)l
Pk
2m o²
(11)
R=1
m=
The Upper bound of BE-DQN variance is:
Var Q BE DQN
(so, a)] = Σ
p Var (Q: DQN (so, (SNN(S0 a))
1=1
(12)
pk) 2 Var Var (so. a)]
Var Var DQN (so. a)}.
Based on Cauchy-Schwarz inequality (Alzer 1992 , the lower bound of BE-DQN variance is:
K
Var(Q, BE.DON
(gg a)]
2
pk Var Q DON
(so, a))
k=1
k=1
K
DON (so. a)l Pkc 2
Var[
(13)
K
Var (2 (su. a)]
K
VarlQ
(so. a)].
Theorem 2. The variance of each Vj E [L. BE-DQN Q-value estimator VarlQ RE-DON (8o, a, 0.) has a upper bound matching the variance of DQN, and with a strict lower bound matching the variance of Ensemble DQN under the same K, i,e., VarlQ (So. a, B?) D.D (so, a)), Var P.O (so, a)]]-
14

---

Published as a conference paper at ICLR 2026
Proof. The Q value of BE-DQN j-th estimator for i M is:
Q, D
(su. a, (.))
Zie + Veu.a
k
Zid
80,0
+
k=1
(14)
Pk 2
1
Pk: Z 20 2.
s0.0
k=1
km
K
Pk 7:- -1),
+ + 9
1
k=1
the variance is:
K
M-1
Var Var[Q, DON (5o, a, 0/)1 = 02, +
2m 0..
(15)
m=1
The Upper bound of BE-DQN j-th estimator variance is:
M 1
VarlQ BE-DON (so, a, 02)1 S 50
+
Pk
Σ
?
2m o2
1=1
m=
M-1
(16)
2mm
m=0
Var [
(so+ a)}.
The lower bound of BE-DQN j-th estimator variance is:
Var BF
BF-DON
(80, a, 0!)1
K
K
M 1
2m 02
K
K
k=1
1=1
372
K
M
2
2m 2
K
50
K
pk
7
dsm
km
(17)
2mm
m==0
Var[Q.DO
(so, a)]
K
VarlQ, (s,,
Our theoretical analysis derived under the assumption of zero-reward environment, where the variance of value estimates serves proxy for policy stability. This formulation allows iso- late and characterize the intrinsic uncertainty arising from bootstrapping and ensemble interactions, independent of reward noise.
Under the standard assumption that Var[Z = o the derivation also holds in non-zero environ- ments. Suppose after the final state 8M-1, the agents receive reward of 1. The value of BE-DQN
15

---

Published as a conference paper at ICLR 2026
for > M in this environment is:
K
Q BE-DON
(50, a) =
Σ
Pk 10:0 + v
K
Pk 7: 7ik s0.a
+
Ph (81, as a;
1=1
h=1
(18)
Pk sa.a
Zikk
+
Ph: - +
Ph Zz $2,0
K
+ y
Ph Zi-(M-11))
+ y:th
M 1MM-1g
U 1,07
1
Since VSM-iaa 1, the variance of BE-DQN remains:
K
M-1
Var Var(Q (so. a)] =
2
2m 0
(19)
8m
km
m==
Theorem 1 and 2 likewise hold. Because reward terms contribute only to the mean or bias, not the variance of the Q-update. Therefore, the bounds remain valid for any bounded reward, long as TAE variance is state-wise stationary. However, in certain environments the reward function is defined solely as function of the state. When the reward itself is stochastic (e.g., in multi- armed bandits where rewards follow Beta distribution), the 1 1/K lower bound no longer holds. Nevertheless, the variance of BE-DQN remains lower than that of standard DQN. To address this limitation, which falls beyond the scope of the present work, we intend to explicitly model and bound the variance of environmental rewards in future research, integrating into our theoretical framework.
A.2 SYNTHETIC AND REAL-WORLLD ENVIRONMENTS SETTINGS
The experiments are conducted on quad-core Intel Core i5 processor with 16GB of RAM. Each experiment is repeated 20 times with different random seeds.
We implement ensemblet based on deep ensembles with randomized prior functions (Osband et al. 2018). Each base model (ensemble member) is a 2-layer fully-connected neural network with ReLU as the activation function. The hidden layer size is tuned in {8, 16, 32, 50). Each prior func- tion (neural network) has the same configuration with the base model; its parameters initialized by LeCun normal initialization, and kept fixed during the training process. The ensemble size is tuned {10, 30, 100).
We consider a linear hypermodel over a 2-layer fully-connected neural network with ReLU as the activation function (Dwaracherla et al. 2020). The index distribution is standard Gaussian distri- bution; the index dimension is tuned in 66, 12. 18}. The hidden layer size is tuned in f8, 16, 32, 50). The additive prior also a linear hypermodel over an MLP with I hidden layer of 10 hidden units.
In the training process, the base model parameters and the index distribution (only for BEB) are up- dated successively every n steps. A replay buffer with capacity m maintained to provide training samples. We set n 1, m 10, 000 for Neural Testbed, n m 256 for Mushroom and n 80, 000, m = 80, 000 for Yahoo!R6B.
A.3 MINIGRID ENVIRONMENT SETTINGS
MiniGrid (Chevalier-Boiseel et al. 2023) is minimalist gridworld environment for reinforcement learning research, designed to study sample efficiency and generalization in partially observable Markov decision processes (POMDPs). It features discrete action space (e.g., move, tum left or right, pick), symbolic tile-based observations, and procedurally generated levels with varying complexity (e.g., doors, keys, obstacles).
For instance, as depicted Figure 4 (a), the agent in the FourRooms environment aims to reach the green goal square in the room. The agent must navigate in maze composed of four rooms
16

---

Published as a conference paper at ICLR 2026
442
(a) MiniGrid Exam-
(b) FourRooms
(c) Empty6x6
plc
1.5
(d) LavaGapS5
(e) GoToDoor5x5
(I) MultiRoomN2S4
Figure 4: (a) FourRooms in MiniGrid. The agent's objective (represented by red triangle) is to reach the target (a green box). (b-f) Performance of agents on MiniGrid after 1e5 frames of training. All results are averaged over 5 random seeds, with each seed tested for 100 episodes.
Table 4: Hyperparameters used in BE-DQN.
Hyperparameter
Value
Number of episodes
2, 000
Maximum steps per episode
50
Ensemble size
Discount factor
0.99
Learning rate
x 10-4
Batch size
32
Replay buffer size
5 x 104
Update frequency
Target network update frequency
500
Initial
0.1
Final e
0.02
e decay factor
0.995
interconnected by 4 gaps in the walls. To obtain a reward, the agent must reach the green goal square. Both the agent and the goal square are randomly placed in any of the four rooms. The agent receives 7 x 7 x 3 dimensional image of the environment as observation input. The three channels represent object ID, color ID, and state, respectively. The agent's position is fixed at the center of the bottom row in the observed image. The state encoding is consistent with the observation image, displaying all grids orientation. The third channel indicates that the agent is currently facing cast, south, west, north, respectively. is worth noting that we do not impose a step penalty in the environment. Instead, we assign the reward for the entire trajectory is set to if the agent achieves the goal within the allowed number steps, and 0 otherwise.
global coordinates at all times, without rotation according
the agent's
To evaluate the efficacy of BE-DQN across diverse tasks, conduct experiments in generated Box2D environments FourRooms. Empty-6x6, LavaGapS5, GoToDoor-5xt and MultiRoom-N2 and provide the learning curves for DQN (Mnih al. 2015), Ensemble DON (E-DQN) (Anschell
17

---

Published as a conference paper at ICLR 2026
FEDON
12:00M
16-oon
033
104 24
T6
(a) BabyAl
(b) Blackjack
Figure 5: Performance of agents on Baby Ti-o-To-Oii To-Obj) and Blackjack after 1e5 frames of training. All results are averaged over 5 random seeds, with each seed tested for 100 episodes.
al. 2017 Random Ensemble DQN (RE-DQN) (Agarwal et al. 2020). Unbiased Asymmetric
Actor- Critt-Criti (UAAC) (Baisero Amato 2022) and our BE-DQN in Figure (b-f). Additionally,
use sampling approach, sampling from weights {pf-1, ..... P }), named Weighted Ensemble
DQN (WE-DQN). The above comparison demonstrates the effectiveness of BE-DQN.
To further validate the effectiveness of our method, we conducted experiments with E-DQN, RE- DQN, and BE-DQN in both the BabyAl and Blackjack environments, thereby confirming the effi- cacy BE-DQN.
We run all experiments P40 GPU. The hyperparameters for training BE-DQN are summarized Table [ The ensemble size of DQN is set to 1, while the ensemble size of DQN, E-DQN, RE-DQN, and BE-DQN is 5. The remaining parameters and network architectures are consistent. The detailed information for Q-newwork structure is provided below.
single server with AMD EPYC 7763 64-Core Processor and NVIDIA
Conv2d: (in=3, out=32, kemel=3, stride=1, pad=0)
ReLU
Conv2d: (in=32, out=64, kernel=3, stride=1, pad=0)
ReLU
Conv2d: (in=64, out=64, kernel=2, stride=1, pad=()
ReLU
Flatten
FC: (in=input_din out=512)
ReLU
FC: (in=input.dim, out=1)
A.4 PERFORMANCE OF ENSEMBLE+ ON YAHOO!R6B SUBSET
Following the experimental setup outlined Appendix A.3, we the ensemble size to 30 and evaluated the models subset comprising the first million events of the Yahoo!R6B dataset. The results demonstrate that ensemblet (PEB) outperforms the ensembl e baseline by margin of 3%.
Table 5: Cumulative rewards the Yahoo!R6B subset (first 1M events)
Method
Number of clicks
Relative Imp.(%)
ensembles
185,4 = 37.65
-3.1% +
ensemblet (BEB)
2,255.5 = 94.46
18

---

Published as a conference paper at ICLR 2026
A.5 IMPACT OF ENSEMBLE SIZE
We investigated the impact of the number of ensemble members on ensemblet (BEB). All ex- periments were conducted in synthetic environment based the Neural Testbed, with an action dimension of 50 and 10 different random seeds.
For ensemble 1+ (BEB), evaluated ensemble sizes of 25, 50, 100. As illustrated in Table 6 observe a consistent improvement in performance as the number of ensemble members increases. Specifically, the relative regret reduction grows from 28.23% with 25 members 47.97% with 100 members. This trend suggests that the Bayesian Ensemble framework is particularly effective at scaling with larger models.
Table Relative Regret Reduction(%) of ensemble+(BEB) over ensemble+ across varying ensem- ble sizes
Ensemble Size
25
50
100
Regret Reduction
28.23%
33.21%
47.97%
A.6
SCALABILITY AND COMPUTATIONAL EFFICIENCY
The computational overhead of the hypermodel (BEB) is primarily attributed to the index updates derived from variational inference. However, distinct advantage of our proposed framework the inherent flexibility of this update mechanism, which allows for the modulation of update frequency achieve an optimal balance between performance and computational efficiency.
To investigate this trade-off, we conduct experiments by adjusting the update interval. The ex- periments are conducted in a synthetic environment based on the Neural Testbed, with an action dimension of 50 and 10 different random seeds. Specifically, we reduce the index update frequency to one-third (1 /3) of the original setting. As demonstrated Table this adjustment significantly lowers the computational cost. For instance, in the 36-dimensional setting, the elapsed wall time seconds to 77 seconds.
drops from approximately
Crucially, the results in Table & iindicate that this reduction in computational cost does not compro- mise the model's fundamental utility. While there a trade-off, the reduced-frequency model still achieves competitive relative regret reduction compared to the baseline hypermodel. This confirms that the BEB framework can maintain high effectiveness even under constrained computational bud- gets.
Table 7: Elapsed wall time(s) across different index dimensions.
Method
Index Dim =
Index Dim 12
Index Dim = 36
hypermodel
61.48 + 5.11
61.73 + 2.38
65.25 + 2.32
hypermodel (BEB) Reduced Freq.
70.19 3.97
73.59 5.09
76.99 + 3.98
hypermodel (BEB)
83.49 + 2.32
86.11 + 3.38
90.39 + 1.68
Table 8: Relative Regret Reduction (%) compared hypermodel across different index dimensions.
Method
Index Dim =
Index Dim = 12
Index Dim 36
hypermodel (BEB)
44.33%
43.01 %
16.47%
hypermodel (BEB) Reduced Freq.
37.98%
32.03%
4.37%
19


Reviews:
[{"review_id": "rdnzuYQ4LC", "reviewer": "Reviewer_uvj4", "summary": "This paper proposes **Bayesian Ensemble (BE)**, a framework designed to enhance existing ensemble-based methods for sequential decision-making. The core idea is to move beyond the standard approach of sampling ensemble members from a fixed, uniform distribution. Instead, BE treats the selection of an ensemble member as a bandit problem itself, dynamically updating a sampling distribution over the members using Bayesian inference based on observed rewards.\n\nThe authors instantiate this framework in two settings:\n1.  **Bayesian Ensemble Bandit (BEB):** This applies the BE layer to contextual bandit problems, showing how it can \"supercharge\" existing methods like `ensemble+` (using Beta distribution updates for a discrete index) and `hypermodel` (using variational inference on a Gaussian distribution for a continuous index).\n2.  **Bayesian Ensemble Deep Q-Network (BE-DQN):** This applies the framework to deep reinforcement learning. It maintains a distribution over an ensemble of K Q-networks and updates this distribution based on rewards, similarly to Thompson sampling.\n\nThe paper provides theoretical analysis for BE-DQN, proving that its Q-value variance is bounded between that of standard DQN and Ensemble DQN. Empirically, the authors demonstrate the effectiveness of their methods on synthetic benchmarks (Neural Testbed, Mushroom, MiniGrid) and a real-world dataset (Yahoo!R6B), showing improved regret and cumulative rewards over baselines.", "questions": "1.  **Missing Citations:** Could you please discuss the relationship between your proposed BE framework and the following highly relevant works:\n    * Li et al. (2025), \"Scalable Exploration via Ensemble++\"\n    * Li et al. (2024), \"Q-Star Meets Scalable Posterior Sampling... via HyperAgent\"\n    * Li et al. (2022), \"HyperDQN: A Randomized Exploration Method...\"\n    How does your work compare or contrast, especially regarding the `ensemble++`, `hypermodel`, and DQN enhancements?\n2.  **BE-DQN Design:** Can you provide more justification for the asymmetric design of BE-DQN, where action selection uses the `arg max` member while the target calculation uses a weighted average? Have you experimented with sampling the action-selection member $j$ from the distribution $p$ (i.e., $j \\sim p$), which would seem more aligned with Thompson Sampling?\n3.  **`ensemble+ (BEB)` Sampling:** Could you please clarify the *exact* sampling mechanism for the `ensemble+ (BEB)` variant? Is the member index $z$ at each step chosen by sampling $w_i \\sim Beta(\\alpha_i, \\beta_i)$ for all $i$ and then setting $z = arg~max_i w_i$? If so, how does this relate to the goal of Thompson sampling, and have you considered simply sampling $z$ from a categorical distribution whose parameters are derived from the Beta posteriors?\n4.  **Practicality of Variance Bounds:** Your theoretical analysis is based on a zero-reward environment. How do you expect the variance of BE-DQN to behave in practical environments with significant reward variance? Does this limitation on the theory affect the practical stability of the algorithm?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* **Novelty and Simplicity:** The core idea of treating ensemble member selection as a bandit problem and updating the selection distribution via Bayesian inference is simple, intuitive, and novel. It provides a principled way to prioritize more \"useful\" ensemble members over time.\n* **Generality:** The framework is shown to be flexible, adapting to both discrete index spaces (enhancing `ensemble+`) and continuous index spaces (enhancing `hypermodel`). Its application to both bandit and full RL problems demonstrates its broad utility.\n* **Strong Empirical Results:** The method shows consistent performance improvements over its corresponding baselines across multiple domains:\n    * **Bandits:** Lower regret in Neural Testbed and Mushroom environments.\n    * **Real-World Bandits:** Higher cumulative clicks on the large-scale Yahoo!R6B dataset.\n    * **Reinforcement Learning:** Stronger average performance across several MiniGrid environments compared to DQN, E-DQN, and RE-DQN.\n* **Theoretical Justification:** The paper provides a theoretical analysis for the RL variant (BE-DQN), proving that its Q-value variance is bounded, lying between the variance of Ensemble DQN (lower bound) and standard DQN (upper bound). This adds a layer of soundness to the method's stability claims.", "weaknesses": "1.  **Missing Related Work:** The related work section and subsequent discussions overlook some highly relevant and recent papers. Contextualizing the BE framework against these works is crucial:\n    * **Li et al. (2025, \"Scalable Exploration via Ensemble++\"):** This paper also focuses on scalable exploration using ensembles. The authors should discuss how their work differs from or relates to this version of Ensemble++.\n    * **Li et al. (2024, \"Q-Star Meets Scalable Posterior Sampling: Bridging Theory and Practice via HyperAgent\"):** Given that a key baseline and contribution of this paper is the enhancement of `hypermodel`, the authors must discuss \"HyperAgent,\" which also uses hypermodels for scalable posterior sampling in RL.\n    * **Li et al. (2022, \"HyperDQN: A Randomized Exploration Method for Deep Reinforcement Learning\"):** This paper also explores randomized exploration in deep RL, which is highly relevant to the BE-DQN contribution. The authors should position their method relative to this work.\n2.  **BE-DQN Design Justification:** The design of BE-DQN has an interesting asymmetry: it selects actions using the member with the *maximum probability* ($j=arg~max_{k}p_{k}^{i-1}$), but it computes the target value using a *weighted average* of all members ($\\sum_{k=1}^{K}p_{k}Q(...)$). The paper justifies this as mitigating overestimation, but this design is not fully ablated or motivated. It's unclear why a more standard Thompson Sampling approach (sampling an index $j \\sim p$ for action selection) was not used.\n3.  **Theoretical Limitations:** As noted, the variance analysis for BE-DQN is performed in a zero-reward MDP. The authors rightly state in the appendix that these bounds may not hold in environments with high reward stochasticity. This is a significant limitation that should be more prominently acknowledged in the main paper, as it bounds the applicability of the theoretical stability guarantees.\n4.  **Experimental Gaps:** The exclusion of the `ensemble+ (BEB)` method from the real-world Yahoo!R6B experiment due to computational cost is understandable but leaves a gap in the evaluation. This was one of the two main bandit instantiations, and its performance on a large-scale problem remains unknown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z68EiCKwfy", "reviewer": "Reviewer_sjuM", "summary": "This paper introduces a new approach to Bayesian ensembles in sequential decision making; the main contribution to this front is to explicitly model a distribution over member indices, i.e., explicitly treat model assignment as a bandit problem. This results in the \"Bayesian ensemble bandit (BEB)\" framework, which unifies and generalizes a few existing approaches for Thompson-based sampling in the literature. A version of deep Q-learning using Bayesian ensembles (BE-DQN) is similarly introduced. Both BEB and BE-DQN are evaluated on a set of synthetic and real-world datasets, which show modest improvements over baselines.", "questions": "To restate questions from weaknesses:\n\n1. How is a more general reward space handled?\n\n2. Do the Bayesian ensembles tend to collapse? How can this be alleviated, if so?\n\n3. How does ensemble+ perform on a smaller version of the Yahoo dataset?\n\n4. How does performance grow/stagnate as the number of ensemble members changes?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Optimizing over the index makes a lot of sense, and the proposed approach is both general and simple. In conjugate scenarios, it comes at essentially no computational overhead, and would intuitively improve results.\n- The empirical results overall paint a positive picture, of minimal computational overhead and marginal performance improvement. \n- Variance analysis of BE-DQN, even in somewhat restricted settings, is a nice sanity-check to suggest nice behavior.", "weaknesses": "- One concern is the reliance on {0, 1} rewards. This makes sense for updates, as the Bernoulli-Beta conjugacy is nice to exploit, but it remains unclear how implementation or performance would work in the more general case.\n\n- A well-known concern in Bayesian ensembles is collapse to a single model -- is this observed empirically, and is there any way to avoid this? \n\n- The experiments are in some ways lacking. For example, what is the effect of the number of members in an ensemble? How does ensemble+ perform on a subsample of the Yahoo dataset, if the entire dataset is too large? The BE-DQN results are also somewhat limited, being tested only on MiniGrid.\n\n- As a minor note, in several places \"enemble\" is written instead of \"ensemble\" (lines 406, 412, 414, 415).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gqjYJPADeD", "reviewer": "Reviewer_53ju", "summary": "This paper has proposed Bayesian Ensemble (BE), a modified framework of the existing framework of using ensemble for uncertainty modeling. The main novelty of Bayesian Ensemble is that **the index distribution of the ensemble is also updated**, in addition to the parameters of the ensemble model. Preliminary analysis results are presented in Section 5, and experiment results are demonstrated in Section 6.", "questions": "Please address the weaknesses listed above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The core idea of this paper, i.e. updating the index distribution of an ensemble in addition to its parameters, is both novel and natural.\n\n- The experiment results of this paper seem to be extensive.\n\n- Overall, this paper is well written and easy to follow. This paper has also done a good job of literature review.", "weaknesses": "- My main concern is that the necessity, importance and advantage of using \"Bayesian ensemble\", i.e. also updating the index distribution, need to be better explained and justified. In theory, even with a fixed index distribution, by appropriately updating the model parameters, we can still achieve a good approximation of the posterior. Thus, we need a better justification of Bayesian ensemble. I think the theoretical justification of this paper is relatively weak. In particular, the analyses (Theorem 1 and 2) are limited to a specific case. Also, the conclusions on variances are only vaguely connected to the \"exploratory behavior\". I recommend that the authors significantly improve the theoretical results of this paper: first, please analyze under a general setting rather than a specific case; second, please present the analysis results in term of regret bounds rather than the variance. One possible result is that the authors might prove a better regret bound of Bayesian ensemble compared to the standard ensemble with a fixed index distribution. This will significantly strengthen the paper.\n\n- The experiment results of this paper look interesting and strong, but it is unclear to me if this paper has done sufficient hyper-parameter sweep for the baseline agents. For instance, for ensemble+, how are the prior_scale, learning_rate, and $L_2$ weight decay tuned? Please explain. Similar questions for all other agents. \n\n- It seems that this paper has only considered Beta index distributions in the experiments, why? In addition to conjugacy, are there any other advantages of Beta distributions?\n\n- Minor comments:\n  - in equation 2, $r=r_i$ should be $r=R_i$\n  - Please make images in Figure 3 larger, they are hard to read\n  - If possible, please include Algorithm 1 and 2 in the main body of the paper to make it more self-contained", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8d2SaZfIJR", "reviewer": "Reviewer_uMJ8", "summary": "This paper proposes a new ensemble learning method for sequential decision making, including contextual bandits and reinforcement learning (RL), called Bayesian Ensemble (BE). BE updates the ensemble index distribution using Bayesian inference on observed rewards. The authors validate their BE approach using extensive experiments on synthetic and real-world datasets.", "questions": "It would be helpful for the authors to respond to the weaknesses pointed out above.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The contributions appear to be novel and technically sound.\n* A clear explanation of the BE approach is presented, for both the bandit and RL settings.\n* Bounds on the variance of the BE-DQN approach for RL provided in two theorems, both of which appear to be theoretically sound.\n* The experimental results on synthetic and real datasets, which include regret, wall-clock run time, and cumulative reward, are convincing.", "weaknesses": "* Table 1 in the paper shows that the BEB method incurs a computational overhead of approximately 20% for hypermodel, due to the use of variational inference. A more thorough study of the scalability of the BE approach should be provided, as well as some options for mitigating scalability issues.\n* The RL experiments are somewhat limited, since they only include experiments run using the MiniGrid environment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)