Planning With Uncertainty: Deep Exploration in Model-Based Reinforcement Learning
Authors: Yaniv Oren
Matthijs T. J. Spaan
Wendelin Boehmer
Venue: Submitted to ICLR 2023 (2023)
Keywords: Reinforcement learning
exploration
uncertainty
planning

Abstract: Deep model-based reinforcement learning has shown super-human performance in many challenging domains. Low sample efficiency and limited exploration remain however as leading obstacles in the field. In this paper, we demonstrate deep exploration in model-based RL by incorporating epistemic uncertainty into planning trees, circumventing the standard approach of propagating uncertainty through value learning. We evaluate this approach with the state of the art model-based RL algorithm MuZero, and extend its training process to stabilize learning from explicitly-exploratory decisions. Our results demonstrate that planning with uncertainty is able to achieve effective deep exploration with standard uncertainty estimation mechanisms, and with it significant gains in sample efficiency.

Under review as a conference paper at ICLR 2023
PLANNING WITH UNCERTAINTY: DEEP EXPLORATION IN MODEL-BASED REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Deep model-based Reinforcement Learning (RL) has shown super-human perfor- mance in many challenging domains. Low sample efficiency and limited explo- ration remain as leading obstacles in the field, however. In this paper, we demon- strate deep exploration in model-based RL by incorporating epistemic uncertainty into planning trees, circumventing the standard approach of propagating uncer- tainty through value learning. We evaluate this approach with the state of the art model-based RL algorithm MuZero, and extend its training process to stabi- lize learning from aeplicitly-xploratttee
trajectories. In our experiments planning with uncertainty is able to demonstrate deep exploration with standard uncertainty estimation mechanisms, and with it provide significant gains in sample efficiency in hard-exploration problems.
INTRODUCTION
In February 2022, state of the art performance in video compression on YouTube videos has been achieved with the algorithm MuZero (Mandhane al. 2022 Schrittwieser et al. 2020), setting new milestone in successful deployments of reinforcement learning (RL). MuZero is a deep model- based RL algorithm that learns an abstracted model of the environment through interactions, and uses it for planning. While able to achieve state of the art performance in extremely challenging do- mains, MuZero is limited by on-policy exploration that relies on random action selection. Effective, informed exploration is crucial in many problem settings (O' O'Donoghue et al., 2018), and can induce up to exponential gain in sample efficiency (Osband et al. 2016).
Standard approaches for exploration rely on estimates of epistemic uncertainty (uncertainty that is caused by a lack of information, (Hüllermeier & Waegeman, (02T) to drive exploration into under- unexplored areas of the state-action space (Bellemare et al. 2016 Sekar et 20000) or the learner's parameter space (Russo et al.) 2018)). These approaches often incorporate the uncertainty into value- learning non-stationary reward bonus (Oudeyer & Kaplan 2009) to directly approximate the total uncertainty in a value-like prediction propagated over future actions (0 (O'Donoghue et al.) 2018) achieve exploration that is deep. We refer deep exploration as exploration that is 1) directed over multiple consecutive time steps, as well as 2) farsighted with respect to information about future rewards, following the definition by (Osband et al. (016). Having access to deep exploration enables the agent to aim for (through farsightedness) and reach (through directness over multiple time steps) areas that are both attractive to explore as well as far (in number of transitions) from its initial or current state.
While propagating the uncertainty from future actions through the value function enables propaga- tion over far horizon, it also introduces several problems. First, the values become non-stationary with the uncertainty bonus, introducing potential instability into the value learning. Second, the horizon of the propagation is limited in the number of training steps: uncertainty in states that are far from the initial state will only propagate to the initial state after sufficiently many training steps, and not immediately. Third, the propagation speed is directly correlated with number of training steps, and as a result uncertainty from encountered high-uncertainty areas of the state space that are not trained often will not (or barely) propagate. To overcome these problems, this paper proposes to propagate uncertainty through the planning- tree of moodel-based RL instead of a netral-netwoor value function. This facilitates the decoupling of the propagation from the learning process by

---

Under review as a conference paper at ICLR 2023
propagating the uncertainty during online inference, and addresses the challenges originating from propagating the uncertainty through value learning.
To demonstrate that propagating uncertainty in the online planning-trect of model-based methods like MuZero can provide deep exploration, this paper's contribution is divided into three parts. First, propose framework for propagating epistemic uncertainty about the planning process itself through a planning tree, for example, in Monte Carlo Tree Search (MCTS, see Browne et al.) 2012 for an overview) with learned models. Second, we propose to harness planning with uncertainty to achieve deep exploration in the standard RL setting (in difference to prior approaches, Sekar et al., 2020 that required a pre-tirin phase to explore). by modifying the objective of an online planning phase with epistemic uncertainty. which we dub online planning to explore, or OP2E. Third, to stabilize learning from off-policy exploratory decisions in MuZero, we extend MuZero's training process by splitting training into exploration episodes and exploitation episodes, and generating the value and policy targets differently depending on the type of episode. We conduct experiments against hard-exploration tasks to evaluate the capacity of OP2E achieve deep exploration. In addition, we conduct an ablation study to evaluate the individual effects of the different extensions propose to training from explicitly exploratory trajectories. In experiments OP2E was able to significantly outperform vanilla MuZero in hard-exploration tasks, demonstrating deep exploration resulting in significant gains in sample efficiency. Our ablation study points out the potential value of discerning between positive and negative exploration trajectories and generating policy targets accordingly, as well as the resilience of n-step value targets to the presence of strongly off-policy trajectories.
This paper organized as follows: Section 2 provides relevant background for model-based RL, MuZero and epistemic uncertainty estimation in deep learning. Section 3 describes our contri- butions, starting with the framework for uncertainty propagation, followed by our approach for modifying planning with uncertainty to achieve deep exploration (OP2E) and finally the extensions proposed to stabilize learning from exploratory decisions. Section 4 evaluates OP2E against two hard-exploration tasks in comparison with vanilla MuZero and presents and ablation study of the different extensions to learning from exploratory decisions. Section S discusses related work, and Section concludes the paper and discusses future work.
2
BACKGROUND
2.1 REINFORCEMENT LEARNING
In reinforcement learning (RL), an agent Iearns behavior policy [[ass) through interactions with an environment, by observing states (or observations), executing actions and receiving rewards. The environment is represented with Markov Decision Process (MDP, Bellman 1957). or a partially- observable MDP (POMDP, Aström 1965). An MDP M is a tuple: M = (S, A, p, P, R),, where S set of states, A a set of actions, p a probability distribution over the state space specifying the probability of starting at each state & E S, R S x A + IR a bounded reward function, and P: SxAx5 > [0. 1] a transition function, where (ss' s, a) specifies the probability of transitioning from state s to state s' after executing action In a POMDP M' (S, A. p, P. S, O), the agent observes observations o E Ω. O S Axx + [0, 1] specifies the probability O a) of observing possible observation o.
In model based reinforcement learning (MBRL) the agent learns model of the environment through interactions, and uses it to optimize its policy, often through planning. In deep MBRL (DMBRL) the agent utilizes deep neural networks (DNNs) as function approximators. Many RL approaches rely on learning a state-action Q-value function Q*T a) R(s, a) "B(V"(s') or the corresponding state value function V*(s) = EiQ (s, a) la ~ T (-|s)], which represents the expected return from starting in state (and possibly action a) and then following policy
~ P(-ss, a)]
2.2 MONTE CARLO TREE SEARCH
MCTS is a planning algorithm that constructs a planning tree with the current state s, at its root, to estimate the objective: arg max, ('t, The algorithm repeatedly performs the following four steps: trajectory selection, expansion, simulation and backup. Starting from the root node no = St, the algorithm selects trajectory in the existing tree based on the averaged returns (14,
2

---

Under review as a conference paper at ICLR 2023
experienced in past trajectories selecting the action a in the same node nyg, and one of many search heuristics, such as an Upper Confidence Bound for Trees (UCT, Kocsis & Szepesvári 2006):
20og(2 20og(,,, (,,N((g)a')) ((vg.a)
O arg max g(nx, a) + 2Cpy
aAA
where N (n a) denotes how many times action a has been executed in node ni. The trajectory selection proceeds iteratively until it arrives at a Ieaf node, that has yet to be expanded in the tree. MCTS expands the node and estimates its initial value with rollout from a random policy. However, Alpha Go/Zero (Silver et al. 2016 2017 2018) uses value function v(nx) that is approximated by a neural network to estimate the leaf"s 's value and chooses potential actions a to expand first with another approximation of the MCTS policy m(a(m)). Last, MCTS propagates the return (discounted reward for visited nodes plus leaf's value) back along the planning trajectory, updating the averaged returns (M,, 0k), all the way up the root of the tree. By choosing different trajectory selection heuristics, MCTS can be used to optimize for different objectives. At the root of the tree, the value V (st) of current state is estimated based on the averaged returns experienced through every action a, and averaged over the actions:
((na.)) V (*((s) ~ EoEd Enee ((ng.e)) =: V
2.3 MUZERO
MuZero is MBRL algorithm that learns an abstracted model of the environment from interactions, and uses it for planning with MCTS. The model learned by MuZero consists of functions that are approximated with neural networks: the representation function &p = g(o)), the transition function 8841 = [&+ 0g), the expected-reward function fx r(áx, 0g), the value function ég v(x)) and policy function p(šx) ( (c)ss). In practice, the representation and dynamics are only used as πT
node-representations $5 = ng in the MCTS planning tree, and only need to represent the states of the environment far as they are useful for the MCTS. The variation of MCTS used by MuZero approximates the value of a leaf-node in the expansion step with a direct predictions from the value function v instead of Morte-Carlo simulations. MCTS is used in MuZero for two purposes: first, for inference of the best action to execute in the environment, and second to generate targets for the policy and value functions. MuZero is trained with on-policy value and policy targets. The policy targets are the relative visitations numbers to each child of the root node, and the value targets are n-step TD targets:
target Uz
Y tit + + -
f=0
Where i the policy that was executed the environment which was induced by the MCTS at every time step. With the Reanalyse variation of MuZero (Schrittwieser et al. 202I). the policy and value targets are constantly updated in the replay buffer based on new planning trees that are computed by Reanalyse, turning MuZero Reanalyse into (mostly) off-policy algorithm. The updated value targets are still n-step targets that use the sum of rewards Σ 7 observed in the environment following policy π. This prevents MuZero Reanalyse from being a completely off-policy algorithm and makes it challenging learn from explicily-eepporator trajectories that do not follow the current exploitation policy.
2.4 EPISTEMIC UNCERTAINTY ESTIMATION IN DEEP LEARNING
Defining, quantifying and estimating predictive epistemic uncertainty is an active field of machine learning research that encompasses many approaches and many methods (Hüllermeier & Waege- man Jain et al. 202T). In this work we assume epistemic uncertainty can be represented with probability theory, and estimated as the variance in probability distribution of predictions that are consistent with the observations Var(X (r)) = Vx [X jo.]. which is a standard approach for epistemic uncertainty quantification. This allows us investigate the propagation of uncertainty in predictions of a node n; in a planning tree with respect to observations and a trajectory of actions through the lens of variance propagation Vx, [X, lo., a which is well studied.
To directly estimate the epistemic variance in the local predictions of value and reward used by MuZero in planning use two methods: deep ensembles with prior functions (Osband et al.
3

---

Under review as a conference paper at ICLR 2023
counting. Ensembles scalable to arbitrarily large state spaces well 2018) and inexpensive for small ensemble sizes. The epistemic uncertainty estimation as computationally ensembles provide however is not necessarily reliable. As an additional method that is able to provide reliable estimates of epistemic uncertainty we use state visitation counting. This method estimates epistemic uncertainty directly based on the number of times each state state-action pair has been visited. While toy method, visitations counting enables us to evaluate the approach in the presence of both reliable and unreliable uncertainty estimators. Additional details regarding applying these methods to the functions used by MuZero can be found in appendix B
3 CONTRIBUTIONS
Our approach modifies MCTS to plan for an exploratory objective rather than an exploitatory ob- jective, i.e. to predict the best action that can be taken in the environment for gathering relevant information rather than for highest expected return. To apply this approach with MuZero as a use- case, we take three steps. First, we propose approximations for uncertainty propagated through planning tree, forward and back (Section 3.1]. Second, we propose to modify the UCT operator with the propagated uncertainty optimize action selection for exploration with so-called optimistic objective (Section 3.2). Third, we propose to extend the training process of MuZero by introducing a distinction between exploration and exploitation episodes and by modifying the target generation from exploratory trajectories. The target generation is extended to discern between exploration tra- jectories that resulted in returns that are higher than the agent's 's value approximations for the same initial state, and trajectories that did not. For exploration trajectories that resulted in improved re- turns, the exploratory decisions become the targets from the learning trajectory. Otherwise, the agent's s exploitatory approximations provide the target (Section
3.1 PROPAGATING UNCERTAINTY IN A PLANNING TREE
Selecting path no:r through decision tree is equivalent to choosing sequence of T ac- tions 40:7-1 that end up in a leaf node ny. In standard MCTS a deterministic model predicts the encountered rewards r in nodes nk, < k < T, and the estimated value er the leaf nr- These are used to update the n-step discounted return of each node ny on the selected path:
1/
Σ
Por
TK + 77K41
OSk<T,
VT =r=a0 uT
(3)
If the model that produced states, rewards and values cannot be fully trusted, ra and vr can be formulated as random variables in Markov chain that connected by random state-variables sg, O lo < T. These correspond to the nodes on the path, the corresponding states, or their representations, but not necessarily to the output &g of any MuZero transition model (Section
In exploration, paths that lead to under-explored states, or states where the model is not reliable yet, should be incentivized. In line with optimistic exploration literature, we incentivize choosing actions in the environment associated with paths in the planning tree that have uncertain returns vi in order to both improve the model as well as find high-reward interactions. For this we need to estimate the variance V. [un lot, -1] 1) of the return along selected path dn:r-1- 1. starting with observation or- To improve readability, we will omit in the following the condition on actions and observation, for example, (v)) Vy[Polos, al.
We will begin by deriving the mean and variance of the distribution of state-variables in the Markov chain for given sequence of actions dg.r-1- 1. Let us assume we are given differentiable transition function [(K,; (x)) E [S*+1 [82, ax] IR sl, which predicts the average next state, and differentiable uncertainty function E(sk, ag) = WaL+I[sk+1 as] E RlSix|s that yields the covariance matrix of the distribution. We will use an epistemic uncertainty measure to compute the latter, but in principle these functions could represent any distribution, containing any kind of uncertainty, epistemic otherwise. We assume that the mean Sy of the first state-variable su is given as an encoding function &n = E., [sp [o = g(oz), like in MuZero. The mean 5 of a later state-variable $ be approximated with 1st order Taylor expansion around the previous mean
4

---

Under review as a conference paper at ICLR 2023
8; = [sk]:
8k+1
= E Es [sk+1] = Es [B [8k+1 1k, asll =
Es, (skk a:)]
(4)
Es (fk. ax) + (sk - 5x) (e)[V V... (sf at))=
= f(5,, ax).
To approximate the covariance Ex+1 == Vorss [sk+1]. we need the law of total variance where for two random variables and y holds V/ [u] = E MV [=]= + Vv E [y|=]]]]]] (see appendix Afro proof in our notation), and again a Ist order Taylor approximation around the previous mean state Sy:
Σx+1
Veeti [sk+1]
B. Voeti [8*+1|8]]]
V. [B [sk+1 [k]]
(5)
J5 ax) Eg [5kkk, aN)
Note that here f(s,, ax)) E.g (skk ax)] = (Sk. 5x) V.f(s, 0x)ls=se =: (8x- 15k-8pp - Jj (Sk, oz) where J (32, ax) denotes the Jacobian matrix of function at state 8% and action dg. Using these state statistics, we can derive the means and variances of causally connected variables like rewards and values. We assume that the conditional reward distribution has mean r(s,, a4) := Er, [rx|st, agl and variance (sk, ax) ;= Vr, [Tk] Sk, ax). and that the conditional value distribution has mean ((s) = Evy [1r[sT] and variance (8(()) Voy [Ur[sT] Analogous above we can derive:
F& Er,,rx zo r(5k,a)),
Vr, [rx] (5 0x) + (5k. ai) Σn J, (5k, ax)
ür == Eoy [or] ((T))
Vo, [or] (5T) + Ju (5k) Eg J. (sk)
(7)
If assume that ræ and the n-step return Mh+1 from Equation 3 are independent,
compute
Eyg [vw] = Ereeeti [1x + 72++1] = [rx[ + Br |x+1] Exr [ur] = Euy [er] (8) Ve, [+]] = Vinve [rk + 70++1 = [r,[rx] + + Ve [P4+1],
(er Vo [1r] (9)
We can therefore compute the variance V/ lo. do:r-1] using one forward pass through the se- lected path computing all š and Σn, starting with the observation encoding 5o = g(oz), and one
backwards pass computing Ev,, [vx]] and V2, [v
As stated, the above derivation applies any type of uncertainty that is encapsulated in the distribu- tion that is propagated. For the purpose of driving exploration using uncertainty from sources that epistemic, we employ epistemic uncertainty estimators such as deep ensembles to estimate the (co-)variances Ess,, ag), of (sk: ax) (8T). To avoid modifying MuZero's 's planning further and allow for this approach to remain modular from the specific algorithm it is incorporated into, interpret MuZero's 's representation g. dynamics f. value v and reward r functions as outputting the conditional means 5g $g 5% == 8k, Dy = 0r, Fk f4. respectively.
3.2 PLANNING FOR EXPLORATION
The UCT operator of MCTS takes into account uncertainty sourced in the planning tree in the form of node visitation count (equation [], to drive exploration inside the planning tree. To drive exploration in the environment, we add the environmental epistemic uncertainty into the UCT in similar manner, the averaged standard deviation:
(10) Co 2 0 is a constant that can be tuned per task to encourage more or less exploration. The term of -ENi Nine 1.a) Veg [vt] sums the variances computed individually every backup step i through node n4. At each backup step i, with actions ai.- state means øt and covariances Σi the variance V, [n] is approximated based on equations and
a;-1 = arg max q()=-1.)) 2C, L. (a)n-11a)) +
o3 Co N( 1:a)
N(n--1-0
Vig [ (3 af) (s f of
J (s af) + + 2 [
(11)
VeL,, [ ] is computed iteratively backwards during the backup, starting from the last node be expanded T, where equation is used:
Vog [+]] As o2((5)) J. J. 1
(12)
Approaches to incorporating OP2E into other search heuristics such as PUCT (which is used by MuZero) can be found in appendix B.I Action selection in the environment can be done in the same manner as for exploitation (for example, sampling actions with respect to the visitation counts of each action the root of the tree, in the original MuZero), but based the exploratory tree.
5

---

Under review as a conference paper at ICLR 2023
3.3 LEARNING FROM EXPLORATORY DECISIONS
If the policy executed in the environment is exploratory, in particular with exploration that is consis- tent over multiple time-steps, the targets generated by an algorithms with on-policy learning prop- erties such as MuZero (see section 2.3) will reflect an exploratory policy and not the exploitation policy the agent is expected to learn. To alleviate this, we propose three main modifications to the training process which build off of each other: 1) separating training into alternating explo- ration/exploitaaiin episodes, 2) double planning and 3) max targets. The extensions we propose enable generating value targets that are off- off-policy both online as well as during Reanalyses. In addition, we maintain some targets that are on-policy but yield better observed returns. These ex- tensions are natural to combine with Reanalyse for further stabilizing of learning from exploratory trajectories.
Alternating episodes First, we propose to alternate between two types of training episodes: exploratory episodes that follow an exploration policy throughout the episode, and exploitatory episodes that follow reward-maximizing exploitation policy throughout the episode. This en- ables us to provide the agent with quality exploitation targets to evaluate and train the value and policy functions reliably, while also providing a large amount of exploratory samples, that explore the environment much more effectively and are more likely to encounter high-reward interactions earlier the agent's training. our experiments we used a ratio of one exploration episode per one exploitation episode, but other ratios as well as dynamically changing ratios are also natural to use. We leave an investigation into optimal exploration exploitation ratios to future work.
Double planning Second, we propose to compute two separate planning trees each time step in exploration episodes: an exploration tree following the heuristic in Equation and an exploita- tion tree following the regular UCT heuristic in Equation [ When generating value targets from exploration trajectories, the value bootstrap V2+n used in the target is the value approximated by the tree planning with exploitation policy Tpp instead of tree planning with an exploration policy π To:
Ut
target
r + y"v
(13)
1=0
This enables generating off-policy value-bootstraps vifa online (similar to Reanlyse's process off- line), but it's important note that the rewards Σ/ - " in the n-step value target are still the rewards observed in the environment following the exploration policy Zg. While this doubles the computation cost of planning (or halves the computation budget for each planning tree).. the results presented Danihelka et al. (202T) showed that with a modified version of MuZero the contribution of large planning budget reduces significantly, and with it, the expected consequence of halving it.
Max targets Third, when generating value targets from exploration episodes, we propose to eval- uate whether the exploratory interactions resulted in an improved return compared to the approxi- mation of the MCTS (a positive exploration trajectory, left) not (a negative exploration trajectory, right):
Σ + > vt
r + Vi S v
(14)
In positive exploration trajectories, the n-step exploratory value target vi nLstep
Σ 7 +
yf can be used. In negative exploration trajectories, a zero-step exploitatory target can be used instead sttepp = Fv. This is equivalent to generating the value targets with a max operator:
target max(u;; stop 1% step).
(15)
Similarly, policy targets for exploratory episodes can be generated based on a max operator in a similar manner, where the policy used as the target is the policy that has achieved the largest value:
target
arg max
(u)
peetep 0 (.step).
(16)

---

Under review as a conference paper at ICLR 2023
Results against the Slide environment
Results aga the Mountain Car environment
-120
OPZE ensemble
Episodic reward
Episodic reward
OP2E state visils
-140
Vanilla MuZero
OPZE ensemble OFZE state visits MuZero
60
-160
40
-180
20
10030 20000 2500C 30000
40000 50300 70000
Environmental steps
Environmentel steps
Figure 1: vanilla MuZero vs. OP2E with ensemble- uncertainty OP2E with state- visitation-counting uncertainty against both environments. The performance 10 independent seeds is averaged, and standard error of the mean (SEM) is presented in the shaded area. Both variants of OP2E are able to solve both tasks effectively while vanilla MuZero is not able to solve either of the tasks at all, average, under the training-stepp budget.
While this enables us to generate off off-policy targets from negative exploration trajectories. the pres- ence of O-step value targets may cause other effects that are detrimental to learning. We explore some of the effects of this series of modifications in an ablation study in the experiments section. In addition, we note that the max targets approach only applies environments with deterministic reward schemes.
EXPERIMENTS
To evaluate OP2E we compare the modified exploratory MuZero to vanilla MuZero against two hard-exploration tasks: Slide, toy task similar the Chain environment (Osband et al. 2016) to evaluate the extent to which deep exploration is achieved, as well as Mountain Car (Moore 1990) which is standard hard-exploration task used to evaluate advanced exploration approaches. In Slide, the environment consists a chain of 60 states, with the only positive reward at the rightmost end of the chain and rewards otherwise. We use non-Markovian reward of R(ss9, aright) Tmaz Telapsed, the difference between the maximum time steps allowed by the environment and the number of elapsed time steps by the agent. The timeout is set length of the environment times 3. The agent has access to three actions: left, right, stay. Left results in sliding back 10 positions, or otherwise to the starting position. Right results in proceeding to the right by position. Stay results in sliding back 5 positions. This results in an environment that is very adversarial to exploration that relies on random action selection. In both tasks two variants of OP2E are evaluated: one with deep ensembles with prior functions (OP2E ensemble) for scalable, realistic uncertainty estimation mechanism, and one with discretized state visitation counting (OP2E state visits) as an example of reliable but unrealistic epistemic uncertainty estimation. As proof of concept, and to avoid mod- ifying MuZero more than absolutely necessary, we replace the state-uncertan cntteunnctttttty covariance function Ess a) with 0, i.e. ignored uncertainty of the forward model. With the exception of the exploration coefficient C. (see appendix B35) no dedicated tuning of hyperparamers was conducted for any of the variations tasks used in the experiments. The results against both tasks are presented in Figure I OP2E is able to effectively solve both tasks, while vanilla MuZero is unable to solve the tasks at all under the training-steve steps budget provided.
4.1 ABLATIONS
An ablation study of the different modifications proposed for learning from exploratory trajectories in the Mountaincar environment are presented in Figure The state-visitation counting uncertainty estimator (exploratory state visits) was used all ablation experiments. In all ablations, the max targets agent agent with all modifications to training and target generation introduced in Section 1.3aapplied: max value targets (eq. [3). max policy targets (eq. [6). alternating between exploration and exploitation episodes and double planning. In every ablation study only modification is
7

---

Under review as a conference paper at ICLR 2023
Mountain Car, value targets ablation study
Mountain Car, policy targets ablation study
Episodic reward
80
Episodic reward
80
max targets
60
n-slep targets
zero-step in all
targets tree exploit palicy tangets
zero-step in exploration
20
20
tree explor. policy targets
10000
20000
30000
40000
50000
60000
0
10000
20000
30000
40000 50000
Environmental steps
Environmental steps
Mountain Car, alternating episodes ablation study
Mountain Car, double planning ablation study
Episodic reward
Episodic reward
75
alternating, max targets alternating, netep largets non-alternating, max targets non-alterating, n-step tagets
double planning. max targets no double planning, max targets
40
10000 20000 30000
40000 50000 60000
10000
20000
30000
40000 50000
60000
Environmental steps
Environmental steps
Figure 2: Target ablations against Mountain Car. The performance of 5 independent seeds is av- eraged, and standard error of the mean (SEM) is presented shaded area. A modified reward scheme is used for Mountain Car, appendix B.4 for details.
investigated and all other modifications are left active. For example, in the value targets ablation study (top left of Figure the value targets are varied, and in the policy targets ablation study (top right of Figure 2 the policy targets varied, but all other design decisions remain the same as in Figure I Value targets ablations compare between four variants of value targets: max targets, n-step targets, O-step targets in exploration episodes, and O-step targets in all (both exploratory and exploitatory) episodes. Policy larget ablations compare between three variants of policy targets: max targets, only-exploratory policy targets (policy targets in exploratory episodes are generated by the exploratory planning trees) and only-exploitatory policy targets (policy targets are generated from exploitatory planning trees all episodes). Alternating episodes ablations compare between four variants: max value targets with alternating episodes, max value targets without alternating episodes (all episodes are exploration episodes), n-step value targets with alternating episodes, n-step value targets without alternating episodes (all episodes exploration episodes). Finally, double planning ablations compares between an agent that uses double planning and an agent that does not (all episodes exploratory, all value bootstraps and policy targets exploratory).
In all experiments in the ablation study the max targets variant which uses all modifications proposed in section 3.3 dominates or matches the performance of all other variants in terms of final policy as well as learning stability.
While relying O-step value targets to turn MuZero into completely off-policy algorithm is clearly not conducive approach, Using max value targets demonstrates an ability to overcome this effect (Figure 2 top left). Generating policy targets based on exploitatory trees even in explo- ration episodes shows significant slowdown of learning (Figure top right). This implies that using Reanalyse to update policy targets from exploratory trajectories with exploitatory policy targets, without utilizing the max operator to discern between positive and negative exploration trajectories, can slow down learning significantly. We expect such a slowdown to occur as a result of diverting the agent away from high return exploratory trajectories that differ significantly from the agent's 's current exploitation policy. Alternating between exploration and exploitation episodes showed significant slowing of learning to reach the goal state, while showing slightly better final policy and learning stability (Figure[2) bottom left). We expect the delay in learning to be a result of alternating episodes with ratio of to and better ratios to be able provide the same benefit with less drawback.
8

---

Under review as a conference paper at ICLR 2023
5 RELATED WORK
Plan2Explore (P2E, Sekar et al. 2020) is a previous approach that harnesses planning with learned models and uncertainty for advanced exploration, although in different setting to the one tackled in this paper. P2E proposes pre-training of an exploration policy for a task, without access to a reward signal. The policy trained to choose actions that maximize the epistemic uncertainty associated with the transition model. This policy is regularly updated with newly collected samples, until the entire state space is sufficiently explored and the learned model therefore precise enough everywhere to plan well. P2E only considers state prediction uncertainty and propagates the uncertainty through planning as intrinsic reward as opposed to variance. In addition, P2E requires an extended pre- training phase and does not scale to large state-spaces, because the learned exploration policy is trained to explore the entire state space without trading off for more promising areas with respect to expected return. In contrast, our approach propagates the uncertainty explicitly, enables trade-off in real time between exploration and exploitation and is applicable to the standard online exploration setting in RL
Epistemic uncertainty has developed into a standard for driving exploration in RI. with methods such as Bootstrapped-DOO (Osband et al. (2016), the uncertainty Bellman-equation (UBE, O' D'Donoghue itt al. 2018) and intrinsic reward (IR, Oudeyer & Kaplan 2009). UBE approximates an analytically- motivated upper bound on the epistemic uncertainty associated with Q-values, in the form of variance. The variance is used to drive exploration using Thompson sampling. UBE has not been developed explicitly for model-based based approaches, and thus does not consider transition-baan certainty, nor does it take ad advantage of the ability of online planning to propagate information from future decisions into current decisions immediately. In IR the epistemic uncertainty propagates di- rectly through the learned Q-value function, by learning a modified function:
un-
Qin(st ax) = [[[r a.) + u(se, az) + yV (st+1)|st+1 ~ P(-|st, at)]
Where u($;, a) is an uncertainty-bong that is used to encourage the agent to explore less-visited transitions. Coupled with the limitations, strength of propagating the uncertainty through value learning compared to through planning is that the horizon of uncertainty propagation is not lim- ited to the horizon of the planning (which is logarithmic in the planning budget of a planning tree in the worst case, and linear in the best case). Instead, the horizon is limited by the number of learning steps, which constrains the propagation differently. For example, deep Q-networks (DQN, Mnih et al. 2015) make one gradient update-ster stel per environmental step, whereas planning always propagates immediately until the planning horizon. Moreover, these methods can be combined with our approach by using UBE replace uncertainty-in-leaf value prediction in MuZero. Propagat- ing newly discovered uncertainty until the planning horizon happens immediately with our method, whereas longer delays are propagated through the value function.
CONCLUSIONS
By propagating epistemic uncertainty in real time through a planning tree together with planning for an exploratory objective, OP2E demonstrates deep exploration while circumventing issues arising from propagating uncertainty through value learning. In addition, the extensions we propose to learning from sessiitently-xpllradttee dition to a more stable learning process, effects which we expect to grow stronger in the presence of environments with more complex reward schemes and a more varied range of near-optimal poli- cics. The framework developed for approximating the epistemic uncertainty as it propagates in a planning tree can provide backbone for additional approaches that take advantage of uncertainty estimation, such as planning for reliable exccution (by discouraging uncertain decisions) or reliable planning (by avoiding planning into trajectories where additional planning increases uncertainty). The propagation framework itself is essentially modular from both MuZero and MCTS, making an available choice for any approach that plans with learned model and planning trees. Our ap- proach strengthens the notion that incorporating epistemic uncertainty into different online inference mechanisms can provide effective exploration by harnessing the strengths of the mechanism make better exploratory decisions.
decisions show capacity to reach better final policies in ad-

---

Under review as a conference paper at ICLR 2023
REFERENCES
Karl Johan Aström. Optiall control of markov processes with incomplete state information Jour- nal of Mathematical Analysis and Applications, 10:174-205, 1965.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information pro- cessing systems, 29, 2016.
Richard Bellman. A markovian decision process. Journal of mathematics and mechanics, 6(5): 679-684, 1957.
Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and Al in games, 4(1):1-43 2012.
Ivo Danihelka, Arthur Guez, Julian Schrittwieser, and David Silver. Policy improvement by planning with gumbel. In International Conference on Learning Representations, 2021.
Joery de Vries, Ken
Voskuil,
and
Frank
Bryce.
Muzero.
GitHub
repository, 2021.
https://github http://tww....ooaaaaeee/////e// comm///aeeeee//uzzeerrreee
ee 7
Werner Duvaud. muzero-general.
GitHub repository, 2021.
https:
23a //
Jean-Bastien Grill, Florent Altché, Yunhao Tang. Thomas Hubert, Michal Valko, loannis Antonoglou, and Rémi Munos. Monte-carlo tree search as regularized policy optimization. In International Conference Machine Learning, pp. 3769-3778. PMLR, 2020.
Johanna Hansen.
bootstrap-dqn.
GitHub
repository,
2019.
dgn/tree/
Eyke Hüllermeier and Willem Waegeman. Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods. Machine Learning. 110(3):57-556, 2021.
Moksh Jain, Salem Lahlou, Hadi Nekoei, Victor Butoi, Paul Bertin, Jarrid Rector-Brooks, Maksym Korablyov, and Yoshua Bengio. Deup: Direct epistemic uncertainty prediction. arXiv preprint 10X.::2102.08501, 2021.
Levente Koesis and Csaba Szepesvári. Bandit based monte-carlo planning. In European conference on machine learming, pp. 282-293. Springer, 2006.
Amol Mandhane, Anton Zhernov, Maribeth Rauh, Chenjie Gu, Miaosen Wang, Flora Xue, Wendy Shang, Derek Pang. Rene Claus, Ching-Han Chiang, al. Muzero with self-competition for rate control in vp9 video compression. arXiv e-prints, pp. arXiv-2202, 2022.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning. nature, 51 518(7540):529 533, 2015.
Andrew William Moore. Efficient memory-basse learning for robot control. Technical report, Uni- versity of Cambridge, 1990.
Frank Nielsen. On the jensen-shannon symmetrization of distances relying on abstract means. En- Iropy, 21(5), 2019. ISSN 1399-4300. doi: 10.339//221050485. URI. htt.hhtt///// http///ww.mm/mmmmmmm com/ 1099 (099-9330/21/5/4440
Brendan O' D'Donoguue, Ian Osband, Remi Munos, and Volodymyr Mnih. The uncertainty bellman equation and exploration. In International Conference on Machine Learning, pp. 3836-3845, 2018.
10

---

Under review as a conference paper at ICLR 2023
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29:4026 4034, 2016.
Ian Osband, John Aslanides, and Albin Cassirer. Randomized prior functions for deep reinforcement learning. arXiv preprint 1806.333 2018.
Pierre-Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? typology of computa- tional approaches. Frontiers in neurorobotics, 1:6, 2009.
Daniel Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, Zheng Wen, al. A tutorial on thompson sampling. Foundations and Trends@ Machine Learning, 11(1):1-96, 2018.
Julian Schrittwieser, Toannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-669) 2020.
Julian Schrittwieser, Thomas Hubert, Amol Mandhane, Mohammadamin Barekatain, Ioannis Antonoglou, and David Silver. Online and offline reinforcement learning by planning with learned model. Advances in Neural Information Processing Systems, 34:27580-27591, 2021.
Ramanan Sekar, Oleh Rybkin, Kostas Danillidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via superviid world models. In International Conference on Machine Learning. pp. 8583 8592. PMLR, 2020.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484 489, 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-33) 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 1440 1144, 2018.


Reviews:
[{"review_id": "ss9tcjkqeT", "reviewer": "Reviewer_YegC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper identifies lackluster exploration as a standing problem in model-based deep RL algorithm involving tree search. Hence, the authors propose to estimate epistemic uncertainty in order to drive exploration.  Instead of directly integrating uncertainty bonuses in the value function, their method directly propagates uncertainty through the planning tree, thus decoupling uncertainty and value estimation. The paper brings forth three main contributions: (1) a propagation scheme for uncertainty through a planning tree, (2) a modification of the planning objective that includes epistemic uncertainty and encourages exploration and (3) a decoupled training process for exploratory and exploitative episodes, which is designed to stabilize training. The technique described in (1) enables efficient computation of propagated entropy by using Taylor approximations and independence assumptions between reward  at time step $k$ and n-step returns at time step $k+1$. (2) consists in adding an uncertainty bonus to the action selection criteria in the tree policy in order to encourage global exploration. Finally, the training process from (3) relies on alternating exploration and exploitation episodes. The former compute two planning trees in parallel: actions are selected from a tree with uncertainty bonus, while value bootstrap terms are obtained by a search procedure not involving uncertainty bonuses. Finally, policy and value utilize n-step exploratory targets only when greater than exploitation value estimates. The authors provide a brief experimental evaluation against vanilla MuZero, and an ablation study on toy environments (Slide, Mountain Car).", "summary_of_the_review": "The paper tackles an important problem by combining two interesting ideas (MBRL with tree search, and uncertainty-driven exploration) in a principled way. The quality of the paper is good, and the main contributions are clearly presented, but they are only validated on a limited number of toy problems. The lack of convincing experiments in challenging domains constitutes the main weakness of this paper, and I recommend rejection in its current state.", "strength_and_weaknesses": "Strengths:\n\n- The paper is well written, the motivation is clearly states and contributions are thoroughly explained.\n- While the training procedure introduces several modifications (alternating between exploration and exploitation, using exploitation value bootstraps and max targets), an appropriate ablation study is presented in Figure 2.\n\nWeaknesses:\n- The experimental evaluation of OP2E is limited to a selection of problems which is small (2 environments), and only includes toy problems. On the other hand, OP2E is presented as generally applicable in problems for which MuZero was designed. As a result, an evaluation on a larger number of challenging environments (e.g. standard exploration-hard games in ATARI), would be extremely helpful in improving this paper's soundness.\n- The paper argues that naively propagating uncertainty through the value function suffers from several problem (p. 1, last paragraph). While such arguments are well supported, they should also be validated empirically if possible.\n\nMinor comments:\n\n- Section 2.2: the objective of MCTS is conventionally to find $a = \\argmax Q^\\pi(s_t, a)$ rather than a policy maximizing each action-state value.\n- Section 2 is very detailed, and can be shortened significantly to create space for experimental results.\n- Eq. 4: is $\\mathbb{E}_{s_k} [\\mathbb{E}_{s_{k+1}}[s_k | s_k, a_k]]$ supposed to be $\\mathbb{E}_{s_k} [\\mathbb{E}_{s_{k+1}}[s_{k+1} | s_k, a_k]]$ instead?\n- Section 4: restricting the experimental evaluation to a deterministic world model significantly simplifies the uncertainty propagation mechanism. Although this is acknowledged, empirical results could benefit from an evaluation of stochastic forward models.\n- Section 5 could include relevant work on MuZero in stochastic settings [1].\n\nReferences:\n[1] Antonoglou et al. \"Planning in Stochastic Environments with a Learned Model\" in ICLR 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "VEnPKxUBjbJ", "reviewer": "Reviewer_6utc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new extension of the MuZero algorithm (called online planning to explore, or OP2E) that leverages estimates of uncertainty in the algorithm’s predictions of value and reward to generate exploratory behavior. This approach has the advantage of propagating the uncertainty estimates to the policy through the planning tree, rather than through a value function like in other methods. The paper claims that OP2E is better able to achieve deep exploration, than standard MuZero. The claims are supported by experimental results in two different domains.\n", "summary_of_the_review": "Even though the proposed algorithm appears to be novel and interesting, there are some flaws in the experimental evaluation. For this reason I suggest rejecting this paper.\n\nThe paper could be improved by expanding the experimental evaluation to include some additional environments that require deep exploration, and/or a larger more complex environment that requires deep exploration.\n", "strength_and_weaknesses": "The main strength of this paper is the extension to MCTS that propagates an estimate of uncertainty through the planning tree, and drives exploration in the environment. This idea is potentially very powerful if one has access to good measures of uncertainty.\n\nThe main weakness of this paper is the experimental evaluation of the claim that OP2E provides improved deep exploration over plain MuZero. The results provided in the two domains are not enough to convince me of this.\n\nThe Slide environment appears to be a genuinely hard exploration task, even though it is a tabular MDP. Results in this domain are good, and support the conclusions. The environment is described as giving a positive reward when the agent reaches the rightmost state. What is the magnitude of this reward? (This would help with my understanding of Figure 1.)\n\nI strongly disagree that Mountain Car is a “hard-exploration task used to evaluate advanced exploration approaches.” I think evaluating an exploration algorithm on Mountain Car does not provide much insight. Sarsa with a simple linear function approximation scheme and epsilon-greedy exploration can learn a good policy in less than 500 episodes (p. 246, Sutton & Barto, 2018). This is significantly less than ~20,000 episodes required to learn a good policy for OP2E. Mountain Car is at best a validation test to ensure that the components of an algorithm are working correctly, but not as evidence of advanced deep exploration.\n\nI suggest that the authors test their algorithm on a few more hard exploration problems. Ideally on mix of problems that are tabular and that require function approximation. Parisi et al. (2022) describe many such problems, like Deep Sea for example.\n\n\nReferences:\n\nSutton and Barto. (2018). Reinforcement learning: an Introduction. 2nd Ed. MIT press.\n\nParisi et al. (2022).  \"Long-Term Visitation Value for Deep Exploration in Sparse-Reward Reinforcement Learning\" Algorithms 15, no. 3: 81. (https://doi.org/10.3390/a15030081)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TXy4Psn58Fu", "reviewer": "Reviewer_u2v3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes planning with uncertainty, which incorporates epistemic uncertainty into planning trees for deep exploration in model-based RL. Built on top of the SOTA model-based RL algorithm MuZero, the proposed method has shown effectiveness of exploration with standard uncertainty estimation methods. ", "summary_of_the_review": "Overall, the proposed method is novel and interesting, but need to further strengthen the empirical evaluation.", "strength_and_weaknesses": "**Strength**\n\n+ This paper modifies Monte Carlo Tree Search (MCTS) for an exploratory objective, where the best actions are predicted for gathering relevant information rather than for highest expected return. This was done by first propagating uncertainty in a plannig tree, and then optimizing the action selection for exploration using a modified UCT operator with the propagated uncertainty.  \n\n+ Propagating uncertainty in planning tree is novel and interesting. \n\n**Weakness**\n\n+ Need to compare with more exploration baselines.\n\n+ Need to evaluate the proposed method on more diverse and challenging tasks. \n\nQuestions:\n\n+ Can the proposed method extend to other model-based RL methods? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "41w7S1iffH", "reviewer": "Reviewer_3Uzt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a modified version of MCTS that encourages exploration and incorporates uncertainty into the planning tree. The authors compare the performance of the proposed method against MuZero in two environments: Slide and Mountain Car.", "summary_of_the_review": "The reviewer highly appreciates this work as it is trying to implement the classic UCB type of exploration bonus to the powerful MuZero framework. But at the current moment, the reviewer is having a hard time accepting this paper based on empirical evidence of two toy environments. The reviewer understands that some of the environments in the MuZero paper require large computational resources, however, perhaps the author can still try the proposed method in some easy games in Atari that require a reasonable amount of computation to make the empirical results stronger and more convincing. The reviewer would like to change the rating once the author can provide more empirical evidence of the proposed method in more challenging tasks (say Montezuma in Atari).", "strength_and_weaknesses": "Strength:\n1. This paper is well-written and easy to follow.\n2. This proposed method adopts the classic UCB exploration bonus from RL theory, which is intuitive and the reviewer appreciates the efforts towards connecting RL theory and practice.\n3. The experimental results are strong, among all tested environments.\n\nWeakness:\n1. As an empirical paper, the reviewer thinks the experimental results are not enough to demonstrate the empirical benefits of the proposed method – the two tested environments are too toy which limits the contribution of this work.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject