OptionZero: Planning with Learned Options
Authors: Po-Wei Huang
Pei-Chiun Peng
Hung Guei
Ti-Rong Wu
Venue: ICLR 2025 Oral (2025)
Keywords: Option
Semi-MDP
MuZero
MCTS
Planning
Reinforcement Learning
Primary area: reinforcement learning
TL;DR: This paper presents OptionZero, a method that integrates options into the MuZero algorithm, which autonomously discovers options through self-play games and utilizes options during planning.

Abstract: Planning with options -- a sequence of primitive actions -- has been shown effective in reinforcement learning within complex environments. Previous studies have focused on planning with predefined options or learned options through expert demonstration data.
Inspired by MuZero, which learns superhuman heuristics without any human knowledge, we propose a novel approach, named *OptionZero*. OptionZero incorporates an *option network* into MuZero, providing autonomous discovery of options through self-play games. Furthermore, we modify the dynamics network to provide environment transitions when using options, allowing searching deeper under the same simulation constraints. Empirical experiments conducted in 26 Atari games demonstrate that OptionZero outperforms MuZero, achieving a 131.58% improvement in mean human-normalized score. Our behavior analysis shows that OptionZero not only learns options but also acquires strategic skills tailored to different game characteristics. Our findings show promising directions for discovering and using options in planning. Our code is available at https://rlg.iis.sinica.edu.tw/papers/optionzero.

Published as a conference paper at ICLR 2025
OPTIONZERO: PLANNING WITH LEARNED OPTIONS
Po-Wei Huang!?, Pei-Chiun Peng Hung Gueil, Ti-Rong Wu
"nnttttte of Information Science, Academia Sinica, Taiwan 'Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan
ABSTRACT
Planning with options sequence of primitive actions has been shown ef- fective in reinforcement learning within complex environments. Previous studies have focused planning with predefined options or learned options through ex- pert demonstration data. Inspired by MuZero, which learns superhuman heuristics without any human knowledge, we propose a novel approach, named OptionZero. OptionZero incorporates an option network into MuZero, providing autonomous discovery of options through self-play games. Furthermore, we modify the dy- namics network to provide environment transitions when using options, allowing searching deeper under the same simulation constraints. Empirical experiments conducted in 26 Atari games demonstrate that OptionZero outperforms MuZero, achieving a 131.58% improvement in mean human- human-normalizee normalized score. Our behav- ior analysis shows that OptionZero not only learns options but also acquires strate- gie skills tailored to different game characteristics. Our findings show promising directions for discovering and using options in planning. Our code is available https://rlg. htaps://rgg.is.siniiaa wttp:iicccccu..../pppp//pppoaaaa
INTRODUCTION
Reinforcement learning is decision-making process in which agent interacts with environments by selecting actions at each step to maximize long-term rewards. Actions, commonly referred to as primitive action, advance the state by one step each. While this granularity allows precise control at each time step, it can lead to inefficiencies in scenarios where predictable sequences of actions are beneficial. For example, in maze navigation task, it is more efficient choose a sequence of actions such as following straightforward path until reaching new junction rather than deciding an action at each time step. This approach reduces the frequency of decision-making and accelerates the learning process. To address these challenges, the concept of options (Sutton et al.) 1999) has emerged, providing a framework for executing temporally extended actions based on the current state. Options bridge single-step decision-male speeding up the learning process to handle complex scenarios but also simplifying the decision- making by reducing the frequency of choices agent must consider.
and strategic long-term planning, not only
Previous works have proposed adopting the concept of options by either predefining options or learning from expert demonstration data (Sharma al., 2016) Durugkar et al. 22166; de Waard al.) 2016 Gabor et al. 2019| Czechowski et al.] [021] Kujanpää et al. [023] 2024). However, the pre- specific environments, and expert data may
defined options often rely on a deep understanding not be available for every environment, making it difficult to generalize these methods to other envi- ronments. Moreover, when planning with options, previous methods require recurrently executing each action within the option to obtain the next states (de Waard et al., 2016) or verifying whether the subgoal can be reached through primitive actions (Czechowski et al., [02T]] Kujanpää et al.) 2023 2024). This increases the computational cost when executing longer options during planning, especially in scenarios where environment transitions are expensive.
Inspired by the success of MuZero (Schrittwisser et al. 2020), which employs a learned dynamics network to simulate the environment transitions during planning and achieves superhuman perfor- mance from scratch without requiring any human knowledge, this paper proposes novel approach,
Corresponding author; tirongwu iis siw.ca.ed...ww

---

Published as a conference paper at ICLR 2025
named OptionZero. We modify the MuZero algorithm by integrating an option network that pre- dicts the most likely option for each state. During training. OptionZero autonomously discovers options through self-play games and utilizes them during planning, eliminating the requirement for designing options in advance. Furthermore, OptionZero improves the dynamics network to eff- ciently simulate environment transitions across multiple states with options, significantly reducing the overhead for iterative examination of internal states.
We conduct experiments on Atari games, which are visually complex environments with relatively small frame differences between states, making them suitable for learning options. Our results show that using options with maximum lengths of and OptionZero achieved mean human- normalized scores of 1054.30% and 1025.56% respectively. In contrast, MuZero achieves a score of only 922.72%. In addition, we provide a comprehensive behavior analysis to examine the options learned and used during planning. Interestingly, the adoption of options varies across different games, align- ing with the unique characteristies of each game. This demonstrates that OptionZero effectively discovers options tailored to the specific game states and challenges of each environment. In con- clusion, our findings suggest that OptionZero not only discovers options without human knowledge but also maintains efficiency during planning. This makes OptionZero easily applicable to other applications, further extending the versatility of the MuZero algorithm.
2 RELATED WORKS
Numerous studies have explored the concepts of options in reinforcement learning. For example, de Waard et al. (2016) incorporated options from predefined option set into Monte Carlo tree search (MCTS) and extended it to focus exploration on higher valued options during planning. (Sharma et al. (2016) proposed using two policies for planning: one determines which primitive action to use, and the other determines how many times to repeat that action. [Drugkar et al. (2016) explored the effects of repetition and frequency by statistics in Atari games. /Vezhnevets et al. (2016) introduced method which learns options through end-to-end reinforcement learning. Lakshminnrayynaa et al. (2017) proposed a method that allows agents to dynamically adjust rates of repeated actions. [aconn eet al. (2017) derived an option-critic framework, which learns a policy over options and policy within options. The option policy not only determines how to select and execute an action within options but also learns when terminate the option. Kim et al. (2023) proposed to adaptively inte- grate multiple exploration strategies for options based on the option-critic framework. Riemer et al.)
(2000) introduced parameter-sharing approach for deep option learning. Young & Sutton| (2023) discovered options by learning the option policies and integrated them with a Monte Carlo search.
Jinnai al. (2019) formalized the problem of selecting the optimal option set, and produced an algorithm for discovering the suboptimal option set for planning. |Veeriah et al.) (2021) proposed a meta-gradient approach for discovering reusable, task-independent options. works have studied subgoals, which represent target state to achieve after several time steps, either segmented by predefined time step intervals or predicted dynamically by learned network. For example, [abor et al. (2019) used predefined subgoals for planning in MCTS. Czechowski et al.) (202T) introduced a Subgoal Search method to obtain fixed-length subgoals with a low-level policy that predicts primitive actions for reaching subgoals. Kujanpää et al. (2023) proposed Hierarchical Imitation Planning with Search (HIPS), which learns subgoals from expert demonstration data. Ku- janpää et al. 2024) extended HIPS to HIPS-e, adding a low-level (primitive action) search to the high-level (subgoal) search, guaranteeing that subgoals are reachable. In summary, these previous works either adopt predefined options, learn subgoals from expert data, or do not incorporate options in MCTS planning. Compared to these works, our goal is to automatically discover options without relying on predefined options or expert data and to use options during planning.
addition, several
MUZERO
MuZero (Schrittwieser et al. 2020) is based on the foundation of AlphaZero (Silvrr et al. 2018 distinguishing itself by learning environment transitions using neural networks. This allows MuZero to plan in advance without extra interaction with the environment, which is particularly advantageous in environments where such interactions are computationally expensive. Consequently. MuZero has achieved success in a wide range of domains (Schrittwieser et al. 2020; Danihelka et al.) 2022
Antonoglou et al. 202T Hubert al. 2021 Mandhane et al. 2022 Wang et al. 2023).
2

---

Published as a conference paper at ICLR 2025
For planning, MuZero adopts Monte Carlo tree search (MCTS) (Kocsis & Szepesvári 2006 Coulom 2007; Browne et al., 2012), integrating three distinct networks: representation, dynam- ics, and prediction. Specifically, for observation 2; at time step the search determines action a++1 using multiple simulations, each consisting of three phases: selection, expansion, and backup- The selection starts from the hidden state root node 8 selecting child nodes recursively until an
unexpanded leaf node s' is reached. For each non-leaf node s², the child node s*+1 (corresponding
action a +11) is selected according the highest PUCT (Rosin 2011 Silver al. 2017 score:
VL, (s b) 1 ((*h, abt1)
Q1sh (1sh.atti 1) + P(s 1
x Cpuct,
(I)
where Q(s*; a is the estimated Q-value, P(s*, al+1) is the prior probability, N(s a a1+1) is the visit counts, and Cpuct is constant for exploration. In the expansion phase, to expand the leaf node s', the dynamics network go is applied to perform the environmental transition: s',r" rd go (s'-11 aaa), where r¹ is the immediate reward. Note that when = O, the representation network he is used to initialize the root node: s ha(xz). Then, the prediction network fo is applied to evaluate its policy and value: p', v = fo()', where p' is used for P(s', a) and of is the estimated value for s'. The backup phase uses the obtained value of to update the statistics Q(s", a and N(s a a++ a
Nis 1 x Q(s", aa+1) + Gk-1 and Nis", (++)) == N(s a dtt) +1, (2)
Q(st, a ((tt))=
N(st,a ++
t where Gh+l - p +
the cumulative reward discounted by a factor 7.
During training, MuZero continuously performs self-play and optimization. The self-play process collects game trajectories, including It, Tr. a++1, W141, and o for all time steps. For each St, MCTS is conducted to produce the search policy Te. Then, an action d:+1 Tr is applied to the environment, obtaining an immediate reward u4+1 and moving forward to the next observation Ie-1. In addition, 24 is the n-step return. The optimization process updates the networks by sampling records from collected trajectories. For each sampled record n,, the process uses the networks to unroll it for K steps to obtain st with corresponding r vf, and rt for 0 k I K, where sf (z() and st r = -go(s)- (,44)) for k O. Then, all networks are jointly updated using a,44k)
K
K
K Er ++) + c|1011
L = Σ ( +
Σ Pl of)
(3)
where 1P the policy loss, 12 is value loss, 1" the reward loss, and c|101|2 is the L2 normalization.
OPTIONZERO
4.1 OPTION NETWORK
Options the generalization of actions to include temporally extended actions, which is applied interchangeably with primitive actions (Sutton et al.] 1999; |Bcoo et al., (017). In this context, options on Markov decision process (MDP) form special case of decision problem known semi-Markov decision process (SMDP). Given a state 8; at time step t and an option length L, we enumerate all possible options, denoted as 0t+1 = {at+1, a1+2) d+ J. by considering every sequence of primitive actions starting at st. When executing the option 0++1, we obtain a sequence of states and actions $t. 0c+1 S1+1, at42. S-1 at+L, St+L- Ideally, the probability of selecting each option can be calculated by multiplying the probabilities of each primitive action within the option, as illustrated in Figure Ta For example, when {a1, d2, d3, a1} for so is P(a1) x P(az) x P(as) X P(a,) 0.8* 0.4096, where P(a.) is the probability of selecting action a, A naive approach to obtaining the option probabilities involves using policy network to evaluate all possible states from & to St+L. However, this approach is computationally expensive, and the number of options grows exponentially as the option length L increases, making it infeasible generate all options.
4, the probability of option on =
In practice, since most options occur infrequently due to their lower probabilities, our primary in- terest lies in the dominant option. The dominant option, 0] {a1, 02 -.. a1), is defined such that
3

---

Published as a conference paper at ICLR 2025
0.2 g DB (aa)
2
R
stop
Stop
(6)10.5
0.75
bus (
(eg)
$0
0/2
0.64
0.36
50
612
0..6625
0.75
0.4375
0.25
(el) 75
o8 ing)
[eL 92.0 9c] [8.
0.512
0.488
of
[o)
(1..43
0.4219
0.5781
0
0.4096
0.5904
(a)
(b)
Figure I: An illustration of calculating option in a decision tree. Each node represents state with two possible actions, L and R, corresponding to the left and right transitions to the subsequent state. (a) The decision tree and probabilities for each option state s. (b) The procedure of determining the dominant option from the option network.
- > 0.5 P(a.) I 0.5, where II;-, P(a,) is the cumulative product of probabilities and 1 S L. For example, in Figure Ia the dominant option for so is 01 {aa, 02 Qz) because P(a)) x P((2) x P(aa) = 0.512 and P(an) x P(az) x P(aa) x P(as) = 0.4096, and the domi- nant option for &/ is o far, ]. This indicates that the length of the dominant option can vary, depending on how long the cumulative probabilities remain above the threshold of 0.5. In addition, this design ensures that there is only one dominant option for each state s, effectively preventing exponential growth in the number of possible options.
Next, we incorporate the option network into the prediction network in MuZero, denoted as Ω. p,v v = fs),, which predicts an additional option output, Ω, for predicting the dominant op- tion state s. Given the maximum option length L, the option network produces distributions, Ω = {ui, w2,,,., W,), which are used to derive the dominant option,
{aī ....... a7) where a. = arg max, wi (a). Each W; represents the conditional cumulative produet probability of select- ing sequence of actions from of to a;, i.e, wi (a,) = 11 P(a)). Furthermore, virtual action, called stop, is introduced to provide a termination condition. This stop action is the sum of proba- bilities for all actions except a", defined as w(stop)
1 w(a"). To derive the dominant option from Ω2 we progressively examine each w, from W1 to wL. selecting aj as of = arg max, Wi (a) until i L or af becomes stop action. We provide an example for obtaining the dominant options for state 8o and s/. as shown in Figure 1b This method allows for determining the dominant option any state without recurrently evaluating future states, reducing the computational costs.
4.2 PLANNING WITH DOMINANT OPTION MCTS
This subsection describes the modifications to MCTS implemented OptionZero to incorporate planning with the dominant option. For simplicity, we will use option to represent the dominant option in the rest of this paper. The planning generally follows the MuZero but with two modi- fications, including the network architecture and MCTS. For the network architecture, we add an additional option output to the prediction network, denoted as 2² pt, ot = fo(s*), where Q² p² and w are the option distribution, policy distribution, and value at state s respectively. Note that we use superscript s instead of subscript & in this subsection. This is because s represents the hidden state, obtained after unrolling & steps by the dynamics network from the initial hidden state 8 In contrast, 8ą denotes the actual observed state in the environment at time step k. As illustrated in the previous section, we can derive the option o from Ω* The dynamics network, denoted as sttl ++1A++ g*( Ak-1y is modified to predict the next hidden state sktl and the accumu- lated discounted reward 11,1, upon executing a composite action A at s The composite action, 1 can be either primitive action *+1 or an option o++ with the length 1. The accu- mulated discounted reward ,.++,,,+ is computed as E p where piùi represents the single immediate reward obtained by applying a' at state s² Note that the dynamics network supports unrolling the option directly, eliminating the need to recurrently evaluate each subsequent state from * to s
Next, we demonstrate the incorporation of options within MCTS. The search tree retains the struc- ture of the original MCTS but includes edges for options that can bypass multiple nodes directly, as shown in Figure 2] This adaptation integrates options subtly while preserving the internal actions within options, allowing the tree traverse states using either primitive action or option. Each
4

---

Published as a conference paper at ICLR 2025
Selection
Expansion
Backup
primitive selection
primitive expansion
option sélection
option expansion
Figure An illustration of each phase in MCTS in OptionZero.
edge within the tree associated with statistics (N(s, A), Q(s, A), P(s, A), R(s, A)), represent- ing its visit counts, estimated Q-value, prior probability, and reward, respectively. Moreover, for nodes that possess both a primitive edge and an option edge, the statistics of the primitive edge are designed to include those of the option edge. For example, if the tree traverses the node via the option edge, the visit counts for both the primitive and option edges are incremented. This ensures the statistics remain consistent with MuZero when only primitive edges considered within the tree. We illustrate the modifications made to each phase of MCTS in the following.
Selection. For any node s*, the selection of the next child node includes two stages: primitive selection and option selection. The primitive selection only considers primitive child nodes and remains consistent with MuZero by selecting the next action =+11 based on the PUCT score based on equation|] If the selected action *+1 matches the first action in option *+1 we then proceed with the option selection to determine whether to select this primitive action *+1 or option *++1, Option selection is similar to the primitive selection, using the PUCT score to compare both the primitive and option nodes. Since the option node is a successor node of the primitive node, the statistics for the primitive node need to be adjusted to exclude contributions from the option node in the option selection. We select either primitive or option nodes based on the higher PUCT score, which is calculated as follows:
QQ(s (h1))+ P( o0t1)
V E. N(a M]
if option node, 110 ( autt)+++++++++ P(st, attt) I+(N(s" (aatt) (eft)-N(66 N(6=" + X Opuct if primitive node.
I N(* .ot Cpuct
(4)
VE b))
The Pist,atti
max(0, P(st aa+1) P(s oc+l)) ensures that the prior The adjusted estimated Q-value, Q(s", aa is calculated as - -N...ott (0t1) Note that E, NNs* b) is the total visit counts for se-
mains non-negative
N(o* t 100 jattt (att)) Niak (u+4)-N((++ (x-1)
lecting a h+1 and ob+1, which is equivalent to N(* a111 because the statistics of the primitive node already include the statistics of option node. The selection process begins the root node o
until an unevaluated node s¹ is reached, as shown in Figure 2
Expansion. Assume the last two node in the selection path is s and s², where s is the parent node of s'. To expand node s we derive ,-+,, Ω2,, p², of using the dynamics and prediction network. The reward +l11 is from 8 s! and used to initialize the edge R(s" 1 edge of all primitive child nodes are initialized as (N and P(s dl+1) = p'. Then, if the length of option of+1 derived from f2' is larger than 1, we expand the internal nodes following the action within the option. The statistics of each edge are initialized as since these internal nodes are unevaluated. For the option node, the edge is initialized as (N(s), d++1)= R(s', +11) = Q1s +++ 0} and P(s", ,(s",of+1) =w!
+1.1 The R(s a Q(s' a(1))
The backup phase updates the visit counts and estimated Q-value from st back to s Con- sidering that s' may be accessed through various selection paths from s", all edges the possible paths from s to s must be updated. This ensures that both the visited count and estimated Q-value of all nodes remain consistent within the search, regardless of the selection path chosen. We first obtain the 1 - k-step estimate of the cumulative discounted reward as G = + + t where p k+1d is the discounted reward from s to s' and of is the value at state s'. Since not all edges have been evaluated, we calculate 1+,1 by using pid! -pakk where 1. and 1. represent discounted re- wards from the root node 8 to 8 and s², respectively. Then, we update the estimated Q-value of
Backup.
7t
5

---

Published as a conference paper at ICLR 2025
each edge, , 1) using a similar approach as introduced in equation 2 , Q 1 (s AAt2) x , + Gk+1
N(s A
+1
(5)
N(s - N(S A 1 +1,
During planning, the MCTS performs a fixed number of simulations, each including the above three phases. Upon the search completed, MCTS selects a child node from the root node s based on probabilities proportional to their visit counts and performs the composite action in the environment. Overall, the additional complexity introduced by OptionZero, including the costs for the option network and maintaining statistics for option edges, is negligible compared to the original MuZero.
4.3 TRAINING OPTIONZERO
We describe the optimization process using the self-play trajectory in OptionZero, shown in Figure 3 For better illustration, we utilize three additional symbols, including O, U. т, and f. Given state 8, at time step f, O, represents the i-th executed composite action starting from st, 24, is defined as the discounted reward obtained after executing O, denotes the action sequence length of O;, and + Σj= Tj is the accumulated length from O to O. For example, in Figure from the perspective of s₁, we can obtain O 0t-1 {a++1, a++2), }. O2 {aę-3). with corresponding discounted rewards Ui = u4+1 + 224+2. U22 = 44+3, action sequence lengths T1 = 2, T2 = 1, and accumulated lengths 9 2, 42 = 3. Then, the observed discounted reward 11, at 8, is calculated
as Ei= 7'u++++ aggregating the observed rewards provided by the environment with a discount factor 7. The nt-tepp return 2, calculated as U1 + 7 ... + ffr-11 + - 0rt.ry where ir = Note that ut+n is not always available, as St+n may be bypassed when options are executed. Consequently, we identify the smallest T such that Ł. 2 n. ensuring that the step count for the n-step return approximates n as closely as possible. In Figure 3 if = 5, since S is skipped by option, we then approximate the n-step return by using 6 as 2, = Ui + + ? + 7 US + + + + 7044
Next, we describe the training target for both the policy and option network. The search policy distribution ne calculated in the same manner as MuZero. For the option network, given an option length L at state St, examine its subsequent states to derive the training target, D, {or, dr+1, -t- Each di is a one-hot vector corresponding to the training target for wi. Specifically, for any state if the option network predicts option {a1 .g..... ..., } that exactly matches the composite action O = {ai. d2....... executed in the environment, then the option (a+11)(a:+1) for 0 < 1-1 1. Conversely, if o of O, then
learns the action sequence, i.e., Φ; the option learns to stop, i.e., Φ; = onehot stop). We iterate this process to set each é from 8 to S-+L-1- If Φe+z is set to learn stop, subsequent Φi+z should follow, i.e., Øe4j onehot( (nthot(stop) for i i where 0i+1 arg max, p:(a) determined according to the policy network. This method ensures that the option network eventually learns the cumulative probability of the dominant option, as described in subsection [.1] Figure 3 shows an example of setting the training target for the option network. If ot+2 o O2, then the option network learns {a++1, ax+2, stop). {stop, stop, stop]. and {ae+4, 0e+5; 14-6466). for 8e, 2,, and 8143, respectively.
1. Note that if the length of predicted option o; is zero, o, is defined as {a++1h,
0 = (er2)
02 (dc+s)
0g = (Dcra) U. = Mees Ve4a
0, (rer):
Ui = Me+1 + yuesa
Us Reea
U1 uess yutse
51
S1+2
St+3
St+1
Sc45
T,
Oco4
O
Pzez (nhhot(shop) mnchat (ctopp)
anchot(stsu)) umbbet((t.s))
onehotón:..) onwbut(0:.5))
anchotLssap)
(nhhot(chop)
0nehot(0:...)
Figure 3: An illustration of optimization in OptionZero. The notion is from the perspective of 8t-
During the optimization phase, the sampled state s; is trained with K unrolling steps, where each step can be either a primitive action option. This enables the dynamics network to learn the

---

Published as a conference paper at ICLR 2025
environment transitions that incorporate options. The loss is modified from equation B as follows:
Σ P ( P.)
Σ (2 of) Er()e Σ
1 ( 25) + c||0||, (6)
k=0
where fg O. Note that the option loss 1 includes L cross-entropy losses.
EXPERIMENT
5.1
OPTIONZERO IN GRIDWORLD
We first train OptionZero in GridWorld, a toy environment where the objective is to navigate an agent through a grid map with walls from a start position (S) to a goal (G) via the shortest possible route. The maximum option length is set to nine. Other training details are provided in Appendix A Figure shows the options learned by the option network four stages of training: 25%, 50%, 75%, and T00% completion. It can be observed that the learning behavior of OptionZero evolves distinctly across different stages. In the early stage (25%), the model mainly relies on primitive actions, identifying options only when approaching the goal. In the middle stages (50% and 75%), the model begins to establish longer options, progressively learning options with lengths from two up nine. In the final stage (100%), the model has learned the optimal shortest path using options. Notably, using only primitive actions, the optimal path requires an agent to take at least 30 actions. In contrast, OptionZero achieves this with just four options, accelerating the training process by approximately 7.5 times in this example. This substantial reduction highlights OptionZero's efficacy, especially in more complex environments. This experiment also shows that the option network can progressively learn and refine options during training, without requiring predefined options.
(a)
(b) 50%
(c) 75%
(d) 100%
Figure 4: Visualization of options learned by OptionZero at different stages of training in GridWorld.
5.2 OPTIONZERO IN ATARI GAMES
Next, we evaluate OptionZero on Atari games, which are commonly used for investigating options
(Sharma et al. 2016 de Waard et al. 2016 Durugkar et al. 2016 Bacon et al., 2017 Vezhnevets ett al.) 2016; Kim et al.) 2023 Lakshminaray visually complex environments and subtle frame differences between states, making training with primitive actions inefficient. We train three OptionZero models, denoted l,, ls. and fa, each configured with maximum option lengths L = 1, 3, and 6, respectively. Detailed experiment setups B The model serves as a baseline, identical to MuZero. where no options are used during training. In addition, we adopt the frameskip technique (Mnih et al. 2015). commonly used in training on Atari games, set to Namely, this results in frame difference between 24 states when executing an option of length 6. This requires OptionZero to strategically utilize options when necessary, rather than indiscriminately.
al. 2017 Riemer et al. (020) due to their
are provided in Appendix
Table I shows the results of 26 Atari games. Both E and éa outperform the baseline L in mean and median normal-er scores, with lg achieving the best performance at 1054.30% and 391.69%, representing improvements of 131 15% and 63.29% over f,. Overall, 20 out of 26 games perform better than é for fg. and 17 for lg- There are 12 games where scores consistently increase as the option length increases. For example, in up down, the scores rise by 63810.83 and 79503.9 from & to εg- Conversely, there are only four games where scores decrease as the option length increases. For example, in bank heist, scores drop by 156.54 and 511.26, respectively. We find that this decline is likely due to the difficulty of the dynamics network in learning environment transitions (Vries al. 2021 He et al. 2024 Guei et al. 2024) in games with more complex action spaces.

---

Published as a conference paper at ICLR 2025
As the option length increases, the number of possible option combinations grows. Although focus on learning the dominant option, the dynamics network still needs to learn across all dominant options. In games like bank heist, which offers wide range of strategic possibilities for different option combinations, the learning complexity for the dynamics network increases. Nevertheless, most of the games still improve when training with options, demonstrating that options enable more effective planning.
Table 1: Scores on 26 Atari games. Bold text in Eg and ls indicates scores that surpass 41
OptionZero
Game
Random
Human
81
Eg
Eg
alien
128.30
6,371.30
2,437.30
2,9000.07
3,748.73
amidar
11.79
1,540.43
780.26
820.77
862.17
assault
166.95
628.89
18,389.88
19,302.04
21,593.53
asterix
164.50
7,536.00
177.128.50
188,999.00
187,716.00
bank heist
21.70
644.50
1,097.63
950.13
906.53
battle zone
3,560.00
33,030.00
53,326.67 67
53,583,33
39.556.67
boxing
-1,46
9.61
97.71
95.09
96,00
breakout
1.77
27.86
371.30
375.58
364,11
chopper command
644.00
8,930.00
43,951.67
60,181.67
45,518.67
crazy climber
9,337.00
32,667.00
110,634.00
114,390.00
128,455.67
demon attack
208.25
3,442.85
103,823.1 17
117,270.57
109,092.33
freeway
0.17
25.61
29.46
31.06
31.45
frosthite
90.80
4,202.80
3,183.40
3,641.10
6,047.97
gopher
250.00
2,311.00
70,985.27
68.240.60
63,951.47
hero
1,580.30
25,839.40
13,568 20
19,073.18
19,919,22
jamesbond
33.50
368.50
8.155.50
13,276.67
8,571.17
kangaroo
100.00
2,739.00
8,929.67
12,294.00
13,951.33
krull
1,151.90
2,109.10
10.255.37
10,098.83
9,587.57
kung master
304.00
20,786.80
66,304.67
68,528.33
69,452.33
ms pacman
197.80
15,375.05
3,695.60
4,952.37
4,706.63
pong
-17.95
15.46
19.37
15.49
17.39
private eye
662.78
64,169.07
116.83
90.76
83.24
qbert
159.38
12,085.00
17,155.50
30,748.42
36,328.08
road runner
200.00
6,878.00
26,971.33
32,786.67
21,699.67
seaquest
215.50
40,425.80
3,592.53
5,606.663
6,754.50
up down
707.20
9,896.10
217,021.60
280,832.43
360,336.33
Mcan (%)
0.00
100.00%
922.72%
1054.30 %
1025.56%
Median (%)
0.00
100.00%
328.40%
391.69 %
329.77%
5.3 OPTION UTILIZATION AND BEHAVIOR ANALYSIS
This subsection analyzes how options are applied to better understand the planning process of Op- tionZero. Table 2 presents the average percentages of primitive actions (% a) and options (% o). and the distribution of different option lengths (% () across 26 games. In addition, columns "7", "% Rpt.", and "% NRpt." show the average action sequence length executed in games, the proportions of options that repeat single primitive action or involve more than action types, respectively. De- tailed statistics for each game are provided in Appendix(D From the table, we observe that primitive actions are generally the majority, accounting for over 60% in both la and fe. This is because Atari uses a frameskip of four, which means that each primitive action already spans across four states. The use of frameskip four is well-established previous research, and our experiments further cor- roborate these findings. However, there are still nearly 30% of states that can adopt options. When comparing the use of options, is notable that és applies options less frequently, with a percentage of 30.57% compared to 37.6 37.62% in f3. However, the average action sequence length for l (2.03) is longer than that of &g (1.69). This is because action sequences that involve taking two consecutive threc-step options in lz are merged into single six-step option in Lg. resulting in a lower usage rate of options in statistics.
summary, our findings reveal that OptionZero strategically learns to uti- lize options well as employ primitive actions at critical states instead of indiscriminately utilizing longer options.
Next, among the different option lengths used, observe that generally the longer option lengths are preferred. This suggests that if a state already has applicable options, it likely these options will be extended further, resulting in trend towards longer options. This behavior consistent with the
8

---

Published as a conference paper at ICLR 2025
Table 2: Proportions of options with different lengths and options with repeated primitive actions for &s and & in Atari games.
% a
%
%
% 3
%
%
% 6
% Rpt.
NRpt.
C3
62.38%
37.62% 30.57%
6.23%
31.39%
1.69
75.94% 74.12%
24.06% 25.88%
66
69.43%
8.55%
3.52%
1.86%
0.99%
15.64%
2.03
gradual increase in option lengths observed in gridworld as described in subsection 5.1 illustrating the capability of OptionZero effectively discover and extend longer options when beneficial.
Finally, we investigate the composition of primitive actions in options. From Table approximately 75% options consist of repeated primitive actions, similar to the findings in Sharma et (2016). For example, in freeway, a game where players control chickens across a traffic-filled highway from bottom top, the most commonly used options by OptionZero are sequences of repeated Up actions (U-U-U in l and U-U-U-U--U-Ü in la), guiding the chicken to advance upwards. In addition, OptionZero prefers repeated Noop actions, strategically pausing to let cars pass before proceeding. On the other hand, some games still require options composed of diverse combinations of primitive action. For example, in crazy climber, a game where players control the left and right side of the body to climb up to the top, OptionZero utilizes options consisting of non-repeated actions. These options often interleave Up and Down actions to coordinate the movements of the player's hands and feet, respectively. More interestingly, OptionZero also learns acquire options involving combination skills under specific circumstances. In hero, only 4.60% of options involve non-repeated actions. Although the chance is small, such options are crucial during the game. For example, as depicted in Figure 5 the agent executes a series of strategically combined options, including landing from the top, planting a bomb the corner destroy a wall, swiftly moving away to avoid injury from the blast, and then skillfully times its movement to the right while firing after the wall demolished. It is worth noting that there are total of 24 primitive actions executed from Figure 5a to Se but only four options are executed in practice, showing that using option provides effective planning. In conclusion, our results demonstrate that OptionZero is capable learning complex action sequences tailored to specific game dynamies, effectively discovering the required combinations whether the options are shorter, longer, or contain repeated actions. We have provided the top frequency of options used in each game in the Appendix D.5
LEVELE
2990
2990
(a) RF DO
DELLLL
L.L-F-RA- RE-RR--- RF-FF
(d) RF AF-RR--A AP-RFMM RF-RF
RF RF-RE RE-RR-- -P- KF-K
Figure 5: Sequence of game play from (a) to (e) for OptionZero in hero. The actions R, L, D, and F represent moving right, moving left, placing bombs, and firing, respectively.
5.4 OPTION UTILIZATION THE SEARCH
We further investigate the options used during planning. Table 3 lists the results for l,, la. and le. including the proportions of search trees that consist at least one option edge expanded in MCTS ("% in Tree"), the proportions of simulations that at least one option has been selected during search (*% in Sim."), the average tree depth, the median tree depth, and the maximum tree depth. Detailed statistics for each game provided in Appendix D.3], The results show that approximately 90% of search trees expand options, but only around 30% of search trees choose options during selection. Considering the nature of exploration in MCTS, it is reasonable that all simulations

---

Published as a conference paper at ICLR 2025
will incorporate options. Surprisingly, there are still certain game states for which the search process does not use options at all. Especially in hero, From Ey to fg. the proportion of search trees utilizing options decreases from 74.43% 54.39%. showing that there are numerous game states where options are not required. However, the performance remains consistent, suggesting that the planning could concentrate on applying options in certain states. Note that the less frequent use of options does not cause undesirable results; eventually, the search behaves similarly to that of MuZero.
Table 3: Proportions of options in search tree for lg and La in Atari games.
% in Tree
% in Sim.
Avg. tree depth
Median tree depth
Max tree depth
E 0.00%
0.00%
14.52
12.58
48.54
&g 91.43%
28.94%
20.74
18.23
121.46
87.48%
22.28%
24.92
19.35
197.58
Finally, we compare the tree depths of the MCTS process with and without options. It is naturally considered that applying options provides deeper tree, which helps in identifying longer future state sequences for better planning and avoiding pitfalls. From the statistics, the average search tree depths generally increase as the maximum option length increases, rising by 6.22 from & to &a and by 10.4 from & to lg. Interestingly, there are counterexamples where the average depth decreases, such hero. Although the average tree depth decreases in hero (22.30, 17.06, and 12.15 for f, Es, and (G), the performance is improved, as shown in Table T Furthermore, by comparing the median tree depth (19, 10, and 7) and maximum tree depth (50, 147, and 276) in hero, can be derived that the model learns to perform deep searches depending on whatever the state requires. Ultimately, whether to conduct a deeper or shallower search tree is learned by OptionZero automatically. For the maximum tree depth, the baseline L approaches the simulation budget of 50 nodes, meaning the search process may continuously exploit the same promising branch. When integrating with options, although the maximum depths increase, they do not always approach the simulation budgets of 150 and 300. The average numbers of maximum depths are 48.54, 121.46, and 192.27, equivalent to 97.08%. 80.97%, and 64.09% of the budgets, reflecting that the maximum depth is converging. This observation suggests that using an option length of 3 or is sufficient in Atari games.
6 DISCUSSION
This paper presents OptionZeero, a method that integrates options into the well-known MuZero al- gorithm. OptionZero autonomously discovers options through self- games and efficiently sim- ulates environment transitions across multiple states with options during planning, which not only eliminates the requirement for obtaining options in advance but also reduces the overhead for ex- amining consecutive states during planning. The empirical results on Atari games demonstrate a significant improvement of 131.58% in mean soiman-normalized scores, and the behavior analysis reveals that OptionZero effectively discovers options tailored to the specific challenges of each en- vironment. In conclusion, our findings suggest that OptionZero not only discovers options without human knowledge but also maintains efficiency during planning. This makes OptionZero easily applicable other applications, further extending the versatility of the MuZero algorithm.
As OptionZero builds upon MuZero, it can be easily applied to various environments. For example, when applied to two-player games, OptionZero is expected to discover optimal strategies for both players at specific states. In strategic games such as StarCraft, our approach can learn skillfully combined options, enhancing further explainability and facilitating human learning, as illustrated in subsection 5.3 OptionZero can also be integrated with Sampled MuZero (Hubert et al.) (02T) to support environments with complex action spaces, like robotic environments. Nevertheless, our experiments show that OptionZero does not improve performance in all games, especially in en- vironments with numerous option types or visually complex observations, the dynamics network might struggle to learn well. Future work could explore integrating OptionZero with other dynam- ics models, such as S4 (Gu et al. 2021) or Dreamer (Hafner et al., 2020). Finally, the current design of the option networks requires predefined maximum option length. Dynamically extending this maximum option length could be promising direction for future work. We hope our approach and findings provide promising directions in planning with reinforcement learning for future researchers.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
We do not foresee any ethical issues in this research work. All data are generated by our programs.
REPRODUCIBILITY STATEMENT
For reproducing the work, have provided the details of the proposed algorithm in Section and Appendix [ and the setup of training in Appendix B The source code, scripts for processing behavior analysis, and trained models are available https///lg.iisssmiia.eddd wtt.shtww......./pppppppppppoooo
ACKNOWLEDGEMENT
This research partially supported by the National Science and Technology Council (NSTC) of the Republic of China (Taiwan) under Grant Number NSTC 113-2211-E001-000-M13, NSTC 113- 2634-F-A49-004, and NSTC 113-2221-E-A49-127
REFERENCES
loannis Antonoglou, Julian Schritwieser,, Sherjil Ozair, Thomas K. Hubert, and David Silver. Planning in Stochastic Environments with Learned Model. In International Conference on Learning Representations, October 2021. URL httpt:://////. httt:s::/oppeeeeeeeee http:s////pperevee.... net.tweweee..eeee..... 2ot.rerriww..et//ooum...... forum?id
X6D9bAHhBQ1
Pierre-Lue Bacon, Jean Harb, and Doina Precup. The Option-Critic Architecture. Proceedings of the AAAI Conference on Artificial Intelligence, 31(1), February 2017. URL https://ojs.
aaai. azg/index.
Cameron B. Browne, Edward Powley, Daniel Whitehouse, Simon M. Lucas, Peter I. Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A Survey of Monte Carlo Tree Search Methods. IEEE Transactions Computational Intelli- gence and Al in Games, 4(1):1-43, March 2012. URL
/ org/
document / 61 45622
Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Rémi Coulom. Efficient Selectivity and Backup Operators Monte-Carlo Tree Search. In Com- puters and Games, Lecture Notes in Computer Science, pp. 72-83, Berlin, Heidelherg, 2007. Springer.
Konrad Czechowski, Tomasz Odrzygóżdž, Marek Zbysińskii, Micha'l Zawalski. Krzysztof Ole- jnik, Yuhuai Wu, Lukasz Kuciński, and Piotr Miloć. Subgoal Search For Complex Reasoning Tasks. In Advances in Neural Information Processing Systems, volume 34, pp. 624-638. Cur- ran Associates, Inc., 2021. /ttp://///ntppp//preeee.o...ooo.
www.://wwww.......ee.........e.e hwwp:/ss..ipi...c.pppppeeeeeee files/
Abst h/tp://0///51100066ec64262222222 AAstraa html
Ivo Danihelka, Arthur Guez, Julian Schrittwieser, and David Silver. Policy improvement by plan- ning with Gumbel. In International Conference on Learning Representations, April 2022. URL
net /tt.um/d--beeaaddeeeeeee
Maarten Waard, Diederik M. Roijers, and Sander C.J. Bakkes. Monte Carlo Tree Search with options for general video game playing. In 2016 IEEE Conference Computational Intelligence and Games (CIG), pp. 1 1-8, September 2016.
Ishan P. Durugkar, Clemens Rosenbaum, Stefan Dernbach, and Sridhar Mahadevan. Deep Re- inforcement Learning With Macro-Actions, June 2016. URL http:://arxii org/ abs/
606 04615
11

---

Published as a conference paper at ICLR 2025
Thomas Gabor, Jan Peter, Thomy Phan, Christian Meyer, and Claudia Linnhoff- Popiee.. Subgoal- Based Temporal Abstraction in Monte-Carlo Tree Search. In Proceedings of the Twenty-Eight International Joint Conference on Artificial Intelligence, pp. 5562-5568, Macao, China, August 2019. International Joint Conferences on Artificial Intelligence Organization. URL https:// www. www.aaa..rggprroceednnn.....
9/772
Albert Gu, Karan Goel, and Christopher Re. Efficiently Modeling Long Sequences with Structured State Spaces. In International Conference on Learning Representations, October 2021. URL
https:://ppereete httprr//wweeeeeeeeoooorrrrmiiee mett:/oorummiliuuLLPoo.
Hung Guei, Yan-Ru Ju, Wei-Yu Yu Chen, and Ti-Rong Wu. Interpreting the learned model muzero planning. arXiv preprint arXiv:241..0458) 2024.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to Control: Learn- ing Behaviors by Latent Imagination. In Eighth International Conference Learning Represen- tations, April 2020. URL httph::t//oott http://iccc.....tttua..laaaaaaa.
wwt.ccctvv.aaae//e. 20L__00000/// posterr S110TC4tDS.
html
Jinke He, Thomas M Moerland, Joery A de Vries, and Frans A Oliehoek. What model does muzero learn? In Ulle Endriss, Francisco S. Melo, Kerstin Bach, Alberto Bugarin-Diz, Jose M. Alonso- Moral, Senen Barro, and Fredrik Heintz (eds.), ECAI 2024 - 27th European Conference on Arti- ficial Intelligence, Including 1 13th Conference on Prestigious Applications of Intelligent Systems, PAIS 2024, Proceedings, Frontiers in Artificial Intelligence and Applications, pp. 1599-1606. IOS Press, 2024. doi: 10.3233/FAIA240666
Matteo Hessel, Ivo Danihelka, Fabio Viola, Arthur Guez, Simon Schmitt, Laurent Sifre, Theo- phane Weher, David Silver, and Hado Van Hasselt. Muesli: Combining Improvements in Policy Optimization. In Proceedings of the 38th International Conference Machine Learning, pp. 4214-4226. PMLR, July 2021.
Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Mohammadamin Barekatain, Simon Schmitt, and David Silver. Learning and Planning in Complex Action Spaces. In Proceedings of the 38th International Conference on Machine Learning, pp. 4476-4486 PMLR, July 2021.
URL httpps://proceeedngg.. httpp//ee/pppooooooooooeeeeeeaaa
html
Yuu Jinnai, David Abel, David Hershkowitz, Michael Littman, and George Konidaris. Finding Options that Minimize Planning Time. In Proceedings of the 36th International Conference on Machine Learning, pp. 1110-31229 PMLR, May 2019. URL pttps::/pprceeeeeeeeee.o .ml....gs.........mmm.m........m
press/v97/ ninaii 9a html
Woojun Kim, Jeonghye Kim, and Youngehul Sung. LESSON: Learning to Integrate Exploration Strategies for Reinforeement Learning via an Option Framework. In Proceedings of the 40th International Conference on Machine Learning, pp. 16619-16638. PMLR, July 2023.
Levente Kocsis and Csaba Szepesvári. Bandit Based Monte-Carlo Planning. In European Confer- ence on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, volume 2006, pp. 282 293, September 2006.
Kalle Kujanpää, Joni Pajarinen, and Alexander Ilin. Hierarchical Imitation Learning with Vector Quantized Models. In Proceedings of the 40th International Conference on Machine Learning, pp. 17896-17919 PMLR, July 2023. URL https: https::/prroceddlngg mlr kujanpaa23a.ht.o html
Kalle Kujanpää, Joni Pajarinen, and Alexander Ilin. Hybrid Search for Efficient Planning with Completeness Guarantees. Advances in Neural Information Processing Systems, 36, Febru- ary 2024. URL https:/ /twppaapa.sseeaa wwtaapersssipp......ee http://eeeeeeeeeeeeeeeeeeeeeeeee 16226aae0055boccef333388ff htt.a//ceeeeeaaaaoee---aaaaa---d html
Aravind Lakshminarayanan, Sahil Sharma, and Balaraman Ravindran. Dynamic Action Repetition for Deep Reinforcement Learning. Proceedings of the AAAI Conference on Artificial Intelligence, (1),, February 2017.
12

---

Published as a conference paper at ICLR 2025
Amol Mandhane, Anton Zhernov, Maribeth Rauh, Chenjie Gu, Miaosen Wang. Flora Xue, Wendy Shang, Derek Pang, Rene Claus, Ching-Han Chiang, Cheng Chen, Jingning Han, Angie Chen, Daniel J. Mankowitz, Jackson Broshear, Julian Schrittwieser, Thomas Hubert, Oriol Vinyals, and Timothy Mann. MuZero with Self-commetitionn for Rate Control in VP9 Video Compression, February 2022. httppttp://arxxiiv. http://wwww..r/a///////a/// http://bb/2222.6666
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Mare G. Bellemare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Joannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):529-53) February 2015. URL https: www. nature com/ articles/nature1e22
Matthew Riemer, Ignacio Cases, Clemens Rosenbaum, Miao Liu, and Gerald Tesauro. On the Role of Weight Sharing During Deep Option Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):511-5556, April 2020.
Christopher D. Rosin. Multi-armed bandits with episode context. Annals of Mathematics and Arti- ficial Intelligence, 61(3):203-233 230, March 2011. URL http://link.
IDC7/s10472-011-9233
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering Atari, Go, chess and shogi by planning with a learned model. Na- fure, 588(789)):604 609, December 2020. htLpttt:: /www nature. com/ homm/articles// s
Sahil Sharma, Aravind S. Lakshminarayanan, and Balaraman Ravindran. Learning to Repeat: Fine Grained Action Repetition for Deep Reinforcement Learning. In International Conference Learning Representations, November 2016. URL https:://peerrevee. wwtp:/wwwwwww...tteeeeeeeeee forum?
id=B1GoWV5eg
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of Go without human knowledge. Nature, 550(7666::354-3333 October 2017. URL https: wwww nwwuratuuee... www.cooo/caaaeeeee htt..////rrtcccce///tteeeeeeeeee
David Silver, Thomas Hubert, Julian Schrittwisser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. A general reinforcement learning algorithm that mas- ters chess, shogi, and Go through self-play. Science, 362(6419):1140-11 14441144, December 2018. URL https: www.. www.scccieiigg..e......oo..oo.ee wwwpor/////oooooooo//oo///o/oeot ww11////cinnccc aar6404
Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A frame- work for temporal abstraction in reinforcement learning. Artificial Intelligence, 111111111-2111
August 1999. URL htttps:////nt h/tpp::::/wwww.... www.sccieeeeee rect com wtt.c///cciinneerrtiieeeeeee
$000437029900052
Vivek Veeriah, Tom Zahavy, Mattco Hessel, Zhongwen Xu, Junhyuk Oh, lurii Kemaev, Hado P van Hasselt, David Silver, and Satinder Singh. Discovery of Options via Meta-Learned Subgoals. In Advances in Neural Information Processing Systems, volume 34, pp. 29861-29873. Curran Associates, Inc., 2021.
Alexander Vezhnevets, Volodymyr Mnih, Simon Osindero, Alex Graves, Oriol Vinyals, John Agapiou, and koray kavukcuoglu. Strategic Attentive Writer for Learning Macro-Actions In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL /tttps::/ e4492cbe (rt2bbeeebbbbbbtaaaaeeeeeaaaaaa Absseellteetttttttttt..t....
Joery De Vries, Ken Voskuil, Thomas M. Moerland, and Aske Plaat. Visualizing MuZero Models. In JCML 202/ Workshop on Unsupervised Reinforcement Learning, July 2021. URL hetps: //openrevien net. met/forum? t=tt:/fooommmmoodcckkkkppppppp
13

---

Published as a conference paper at ICLR 2025
Pengming Wang. Mikita Sazanovich, Berkin Ilbeyi. Phitchaya Mangpo Phothilimthana, Manish Purohit, Han Yang Tay. Ngān Vü, Miaosen Wang, Cosmin Paduraru, Edouard Leurent, Anton Zhermov, Po-Sen Huang, Julian Schrittwieser, Thomas Hubert, Robert Tung, Paula Kurylowicz, Kieran Milan, Oriol Vinyals, and Daniel J. Mankowitz. Optimizing Memory Mapping Using Deep Reinforcement Learning. October 2023. http:
abs 2ttpg//bb/20000004444
Ti-Rong Wu, Hung Guei, Pei-Chiun Peng, Po-Wei Huang, Ting Han Wei, Chung-Chin Chin Shih, and Yun-Jui Tsai. Minizero: Comparative analysis of alphazero and muzero on go, othello, and atari games. IEEE Transactions on Games, 17(1):125-137, 2025, doi: 10. 10.11009//.222233499990
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering Atari Games with Limited Data. In Advances in Neural Information Processing Systems, volume 34, pp- 25476-25488 Curran Associates, Inc., 2021. URL https: /ptps:: neuripss.......................s http:///00.0......aa.//./aaaaaa.
ht.ml
Kenny Young and Richard S. Sutton. Iterative Option Discovery for Planning, by Planning, Decem- ber 2023. URL http://aaxxv.oorg/aaaa.p 01569
14

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
In this section, we detail our OptionZero implementation, which is built upon a publicly available MuZero framework (Wu et 2025).
A.1
MCTS DETAILS
The MCTS implementation mainly follows that introduced in Section 4.2 with minor details de- scribed below.
Dirichlet noise To encourage exploration, in MuZero, Dirichlet noise is applied to the root node. Similarly, in OptionZero, since option can also be executed the environment, we apply Dirichlet noise to both primitive selection and option selection the root node.
Default estimated Q value For primitive selection, we follow the default estimated Q value for Atari games in the framework (Wu et al. 2025) that enhances exploration:
Qu(a1 (s(
(x.()) > 0
((s)
Nx(s) = 0,
where (s)ss = En >o Q2(s) = Eo ( (). and 1N(,,))> is the characteristic function that only considers primitive child nodes with non-zero visit counts.
For option selection, since the contributions of option child node are included in the statistics of its corresponding predecessor primitive child node, we use default estimated Q value that incorporates virtually losing outcome:
((s) a)) x ((s
N(s,a)+1
where (is, a) is the visit counts of the primitive child node, and Q a) is the mean value of the primitive child node.
A.2 MCTS COMPLEXITY
The complexity of the modified MCTS remains the same the original, with additional minor computational costs in introducing a new network head to predict and use the dominant option. Specifically, in the selection phase, the only added step is comparing the PUCT scores of option child nodes and primitive child nodes, as in equation 4 In the expansion phase, the option policy is evaluated along with policy p and value Since most network weights of the option policy head are shared with the rest of the network, the impact on runtime is negligible. While more nodes are initially expanded, each simulation evaluates only one node at time. In the backup phase, as the statistics of option edges can be easily derived from primitive edges, only the statistics of primitive edges are maintained in practice, eliminating the additional cost of updating all possible option edges.
A.3 GRIDWORLD ENVIRONMENT
The implementation also built upon the same framework (Wu et al. 2025), with a custom Grid- World environment added. The reward of the environment is defined follows: the initial total reward is 200 points, and for each action or option taken, one point is deducted from the reward.
A.4 NETWORK ARCHITECTURE
The network architecture follows a structure similar to MuZero. As discussed Section 4 the option network is incorporated into the prediction network. Specifically, besides the policy head, we add additional L 1 option heads for predicting Q = {uzz, W3 ..., .., initialized to predict the stop. Note that there is no need for extra prediction of w1, since we can directly get the first action of the dominant option from policy head by choosing aj = arg max, p(a). Additionally, the dynamics network is modified simulate the environment transitions of executing both primitive actions and
15

---

Published as a conference paper at ICLR 2025
options. By extending the original action input, the action sequence input to the dynamics network encoded into a fixed number of L planes for supporting options with different lengths, with the /th plane corresponding to the /th move inside an option. Note that when < L, the subsequent planes are set to zero, representing no moves.
Atari games We additionally adopt the state consistency et. 2021 Therefore, the SimSiam (Chen & He 2021) architecture is included to calculate the consistency loss.
GridWorld The network architecture generally follows the architecture tailored for Atari games. However, in the design of the representation network, removed the down-sampling mechanism, adopting a setup similar to MuZero for board games as in Wu et al. (2025).
B TRAINING OPTIONZERO
In this section, we describe the details for training OptionZero models used in the experiments. The experiments are conducted machines with 24 CPU cores and four NVIDIA GTX 1080 Ti GPUs. For the training configurations, we generally follow those MuZero, where the hyperparameters listed in Table
Table 4: Hyperparameters for training.
Parameter
Atari
Grid world
Optimizer
SGD
Optimizer: learning rate
0.1
Optimizer: momentum
0.9
Optimizer: weight decay
0.0001
Discount factor
0.997
Priority exponent (a)
1
Priority correction (3)
0.4
Bootstrap step (n-step return)
5
MCTS simulation
50
Softmax temperature
Frames skip
4
Frames stacked
Iteration
300
400
Training steps
60k
80k
Batch size
512
1024
# Blocks
2
Replay buffer size
IM frames 108k
8k games
Max frames per episode
Dirichlet noise ratio
0.25
0.3
Atari games For Atari games, each setting is trained for runs on each game, with each model taking approximately 22 hours complete. Since introduce additional head to predict option, the training time slightly increases as the max option length increases. For L,, the training time is approximately 21.89 hours. For &a and l.. the training times increase to around 22.28 hours and 22.95 hours, representing increases of 1.8% and 4.8%, respectively. The performance is measured based on the average score of the latest 100 completed games in each run during the training (Hessel) et al. 2021). The training curves are shown in Figure
GridWorld In this toy example, we aim to clearly show that the length the learned options can be extended the training time increases. For training, fixing the goal position and selecting random starting points. As for the evaluation, we fix the starting point and the goal shown in Figure 4 The evaluation also uses 50 MCTS simulations, and the original softmax function is replaced with max selection.
maximum option length L 9,
16

---

Published as a conference paper at ICLR 2025
alien
amiddr
assoult
astenxx
2736
23k
240c
5.00
Datk-
18k
2106
0.8k
13k
2.4k
0.5k
1.0k
0.3k-
96-
906
3.68
0.1k
306
60c
0.0k
6k
10k
33k
43c
12k
25k
40c
50c
B0k
T0k
23k
30c
50k Röl.
bank heist
battle
breakout
72k
105
1.14.
64k
70
450
75
3.8k
40k-
0.5k
32k-
160
0.1k
-25
80
5.0k
ak-
-30
33h
13k
50k
chopper command
crazy climber
demon attack
freeway
90k
140k-
30
105k
75k.
60k
100k:
50k
90k
20
45k
50k:
60k
30k
43k
15k
01
10K
23h
31h
40k
35k
10c
6K
10k
23
31N
43h
50c 60k
13k
20k
35k
40c
30k
80k
10c
20k
30c
40%
50K Bik
frastbite
gopher
hero
jamesbond
84.
sak-
20.06
76-
70k:
suk-
15.0k-
17.5k
17.5k
5k-
12.56
4k
40k-
10k-
10.0k
7.5k
24.
20k-
5.6k
5.0k
14:
0k
30k
kangaroo
kung master
ms pacman
28k-
book
241--
70k
4.8kc
10k
b0k-
36k-
5ok
64
12k-
30k
1.6k
24
15k
0.5k
0k
10k
23k
3tk
40k
53k
60x
0K
10k
23k
30k
45k
50c
60k
06
19k
20k
35k
40c.
50k,
60k
13k.
25k
30c.
40k
50K
65k
24
private eye
54k
qbort
40k
road runner
9.ck
12.
42k-
5.0k
34k
25k
50k-
s.ck
18k-
12k
13k
1.8
p.ck
50k
up
359k
6k.
300k
230k
38.
150k
50k
30c
10k
13k
20k
40c
Figure 6: Training curves 26 Atari games.
17

---

Published as a conference paper at ICLR 2025
C ABLATION STUDY FOR OPTIONZERO
For the ablation study, we train OptionZero with maximum option length = 3, but disable the execution of options in the environment, using them solely for MCTS planning, denoted n- (3. Since all composite actions are primitive actions, we define each as {ai+th where 0i+1 = arg max, Pr(a) and train the option network according to the policy network. According to the results shown in Table 5 n-Ê3 achieves mean numan-normalizz score of 1008.15 100,,15, which is 85.43% higher than the baseline f, indicating that OptionZero still enhances MCTS planning with options without executing them. Notably, in games that require precise step-by-step predictions, such as gopher, n-&3 outperforms fg. indicating that planning for every step remains crucial for certain games. However, in games that benefit from bypassing unimportant frames, such seaquest, the performance n-C3 is only comparable to baseline.
Table 5: Scores on 26 Atari games for the ablation study. Bold text in la and n-ły indicates scores that surpass l1
OptionZero
Game
Random
Human
E3
n-C3
alien
128.30
6,371.30
2,437.30
2,900.07
3,523.20
amidar assault
11.79
1,540.43
780.26
820.77
848.97
166.95
628.89
18,389.88
19,302.04
19,378.79
asterix
164.50
7,536.00
177,128.50
188,999.00
202,183.33
bank heist
21.70
644.50
1,097.63
950.13
1,081.10
battle
3,560.00
33,030.00
53,326.67
53,583333
65,660.00
boxing
-1.46
9.61
97.71
95.09
94.58
breakout
1.77
27.86
371.30
375.58
427.18
chopper command
644.00
8,930.00
43,951.67
60,181.67
79,340.33
crazy climber
9,337.00
32,667.00
110,634.00
114,390.00
122,865.67
demon attack
208.25
3,442.85
103,823.17
117,270.57
104,351.00
freeway
0.17
25.61
29.46
31.06
30.93
frostbite
90.80
4,202.80
3,183.40
3,641.10
3,923.63
gopher
250.00
2,311.00
70,985.27
68,240.60
73,338.67
hero
1,580.30
25.839.40
13,568.20
19,073.18
14,181.65
jameshond
33.50
368.50
8,155.50
13,276.67
7,172.17
kangaroo
100.00
2.739.00
8,929.67
12,294.00
11,175.33
krull
1,151.90
2,109.10
10,255.37
10,098.83
17,420.13
kung fu master
304.00
20,786.80
66,304.67
68,528.33
67,735.67
ms pacman
197.80
15,375.05
3,695.60
4,952.37
4,762.23
pong
-17.95
15.46
19.37
15.49
private eye
662.78
64,169.07
116.83
90.76
94.71
qhert
159.38
12,085.00
17,155.50
30,748.42
22,321.75
road runner
200.00
6,878.00
26,971.33
32,786.67
23,784.67
seaquest
215.50
40,425.80
3,592.53
5,606.63
3,378.60
up down
707.20
9,896. 10
217,021.60
280,832.43
238,409.40
Normalized Mean
0.00
100.00 %
922.72 %
1054.30 %
1008.15 %
Normalized Median
0.00
100.00 %
328.40 %
391.69 %
341.19 %
18

---

Published as a conference paper at ICLR 2025
D IN-DEPTH BEHAVIOR ANALYSIS
In this experiment section, we conduct detailed analysis for Ea and E 26 Atari games.
D.1 OPTIONS APPLIED IN GAMES
We present the statistics in all 26 Atari games conducted for the behavior analysis in Section Specifically, we provide the numbers of options types, option usages, proportions of options with repeated actions, and the average option lengths for l and & Table f and Table 7 respectively. The columns "# {a}" and "# {o}" show the numbers of available primitive actions and the numbers of the options recorded during evaluation, columns "% a" and "% o" show the proportions of actions and options applied during the game, columns "% Rpt." and "% NRpt." show the proportions of options with repeated primitive actions and options with more than one action type, and column "?" shows the average options length (including primitive action). We also provide the proportions of options with different lengths for both fg and & in Table
Table 6: Numbers of option types, option usages, proportions of options with repeated actions, and average option lengths for & 26 Atari games.
Game
{a]
# {o}
%
%
% Rpt.
% NRpt.
alien
18
185
67.66%
32.34%
94.54%
5.46%
1.61
amidar assault
10
187
65.51% %
34.49%
98.43%
1.57%
1.65
139
78. 18.84%
21.16%
57.20%
42.80%
1.30
asterix
163
92.82%
7.18%
90.74%
9.26%
1.10
bank heist
18
138
41.93%
58.07%
7.60%
92.40%
2.05
battle
18
217
88.89%
11.11%
95.33%
4.67%
1.18
boxing
18
240
39.53%
60.47%
50.54%
49.46%
2.12
breakout
64
79.00%
21.00%
84.03%
15.97%
1.32
chopper command
18
242
76.08%
23.92%
92.28%
7.72%
1.41
crazy climber
158
50.93%
49.07%
25.09%
74.91
1.94
demon attack
153
76.70%
23.30%
88. 74%
11.26%
1.38
freeway
30
39.20%
60.80%
95.04%
4.96%
2.19
frostbite
18
273
58.10%
41.90%
94.14%
5.86%
1.81
gopher
325
51.44%
48.56%
68.60%
31.40%
1.88
hero
18
346
81.72%
18.28%
95.40%
4.60%
1.34
jamesbond
18
376
51.34%
48.66%
67.24%
32.76%
1.90
kangaroo
18
230
29.84%
70.16%
70.54%
29.46%
2.36
krull
18
182
67.72%
32.28%
54.58%
45.42%
1.54
kung fu master
14
536
38.48%
61.52%
70.51%
29.49%
2.15
pacman
181
65.01%
34.99%
94.70%
5.30%
1.67
pong
24.57%
75.43%
76.98%
23.02%
private eye
18
233
93.60%
6.40%
78.74%
21.26%
1.08
qbert
105
50.04%
49.96%
97.84%
2.16%
1.97
road runner
18
144
85.95%
14.05%
65.92%
34.08%
1.23
seaquest
74.88%
25.12%
70.92%
29.08%
up down
116
52.06%
47.94%
88.89%
11.11%
1.91
Average
62.38%
37.62%
75.94%
24.06%
1.69
19

---

Published as conference paper at ICLR 2025
Table 7: Numbers of option types, option usages, proportions of options with repeated actions, and average option lengths for Eg in 26 Atari games.
Game
# {a}
# {o}
% a
%
% Rpt.
% NRpt.
alien
18
69.28%
30.72%
94.13%
5.87%
amidar
10
318
67.62%
32.38%
97. 19%
2.81%
2.22
assault
367
78.71%
21.29%
58.78%
41.22%
1.35
asterix
199
92.68%
7.32%
86.80%
13.20%
1.10
bank heist
18
48.17%
51.83%
11.88%
88.12%
2.28
battle zone
18
91.54%
8.46%
95.21%
4.79%
1.24
boxing
18
568
48.46%
51.54%
40.29%
59.71%
2.77
breakout
132
81.21%
18.79%
85.92%
14.08%
1.28
chopper command
18
351
82.75%
17.25%
87.56%
12.44%
1.40
crazy climber
724
60.69%
39.31%
15.15%
73.85%
demon attack
301
82.64%
17.36%
88.16%
1184%
1.34
freeway
118
45.38%
54.62%
90.66%
9.34%
3.45
frostbite
18
708
66.80%
33.20%
86.23%
13.77%
2.45
gopher
56.12%
43.88%
64.71%
35.29%
2.01
hero
18
89.58%
10.42%
85.05%
14.95%
1.39
jamesbond
18
735
66.08%
33.92%
86.88%
13.12%
2.30
kangaroo
18
718
40.00%
60.00%
64.44%
35.56%
3.43
krull
18
679
60.31% %
39.69%
45.89%
54.11%
2.07
kung master
14
1386
53.09%
46.91%
53.40%
46.60%
2.53
ms pacman
219
77.13%
22.87%
94.95%
5.05%
1.77
pong
741
36.82%
63.18%
61.09%
38.91%
3.71
private eye
18
488
97.04%
2.96%
77.05%
22.95%
1.06
qhert
450
62.79%
37.21%
92.84%
7.16%
2.52
road runner
18
225
96.19%
3.81 %
90.55%
9.45%
1.10
seaquest
18
621
82.41%
17.59%
76.38%
23.62%
1.38
down
226
71.65%
28.35%
84.88%
15.12%
1.84
Average
69.43%
30.57%
74.12%
25.88%
2.03
Table 8: Proportions of options with different lengths for la and in 26 Atari games.
Game
% % %
%
2 % % 4
%
% 6
alien
67. 07.66%
4.03%
28.31%
69.28%
4.24%
1.47%
0.97%
0.60%
23.44%
amidar
65.51%
3.52%
30.97%
67.62%
7.09%
2.36%
1.71%
0.60%
20.61%
assault
78.84%
11.55%
9.30%
78.71%
11.74%
6.61%
1.90%
0.63%
0.42%
asterix
92.82%
4.02%
3.17%
92.68%
5.99%
0.62%
0.29%
0.13%
0.28%
bank heist
41.93%
11.46%
46.62%
48.
20.37%
10.52%
6.66%
4.31%
9.96%
battle zone
88 89%
4.34%
6.77%
91.54%
3.16%
1.33%
0.53%
0.22%
3.23%
boxing
39.53%
8.56%
51.91% 5%
48.46%
13.74%
6.25%
2.73%
1.77%
27.05%
breakout
79.00%
9.96%
11.04%
81.21%
13.30%
3.81%
0.77%
0.29%
0.63%
chopper command
76.08%
7.09%
16.83%
82.75%
9.00%
2.16%
1.08%
1.32%
3.70%
crazy climber
50.93%
4.63%
44.44%
60.69%
6.37%
3.30%
1.44%
0.71%
27.49%
demon attack
76.70%
8.23%
15.07% %
82.64%
9.64%
3.46%
1.70%
0.69%
1.87%
freeway
39.20%
2.61%
58.19%
45,38%
4.93%
1.91%
0.98%
0.50%
46.29%
frostbite
58%
2.83%
39.08%
66.80%
3.29%
1.67%
0.92%
0.63%
26%70%
gopher
51.44%
9.24%
39.32%
56. 12%
22.73%
6.51%
3.45%
1.56%
9.64%
hero
81.72%
2.46%
15.82%
89.58%
2.57%
0.66%
0.24%
0.17%
6.78%
jamesbond
51.34%
7.10%
41.55%
66.08%
6.68%
3.14%
1.27%
0.99%
21.83%
kanguroo
29.84%
4.47%
65.68%
40.00%
8.47%
4.77%
3.47%
1.51%
41.78%
krull
67.72%
10.98%
21.30%
60.31%
12.44%
9,58%
5.16%
2.45%
10.06%
kung fu master
38.48%
7.74%
53.77%
53.09%
14.46%
5.03%
3.53%
1.60%
22.28%
pacman
65.01%
2.93%
32.07%
77.13%
7.24%
2.22%
0.86%
0.25%
12.31%
pong
24.57%
3.74%
71.69%
36.82%
6.98%
3.08%
3.04%
1.84%
48.24%
private eye
93.60%
4.67%
1.73%
97.04%
1.84%
0.45%
0.11%
0.07%
0.49%
qbert
50.04%
2.62%
47.34%
62.79%
4.75%
3.52%
1.62%
0.82%
26.50%
road runner
85.95%
5.41%
8.64%
96.19%
1.69%
0.49%
0.27%
0.13%
1.23%
seaquest
74.88%
12.27%
12.85%
82.41%
9.62%
2.79%
1.07%
0.66%
3.45%
up down
52.06%
5.16%
42.77%
71.65%
9.96%
3.85%
2.70%
1.33%
10.50%
Average
62.38%
6.23%
31.39%
69.43%
8.55%
3.52%
1.86%
0.99%
15.64%
20

---

Published as a conference paper at ICLR 2025
In addition, we observe other examples of using options in games, which are introduced as follows. The sample videos of 26 games are provided in the supplementary material, in which, for each frame, the video prints the applied move (can be either action option); additionally, if the planning suggests an option, it printed below the applied move.
kung fu master There are relatively more options with non-repeated actions since the player re- quires a combination of D and F, such as DR-DB DR-DRRE in &g to attack the opponents with an upper- cut Notably, the proportions options with non-repeated actions significantly increased in Lg. in which the player can even combine two uppercuts in an option, such as D------------DDE
ms pacman Many options contain only repeated actions, e.g.. UL-UL-UL Such a repetition is discovered since the game supports multi-direction control, allowing the agent to simply use the repeated UL to go through an L-shape passage without requiring options such as L-U-U.
seaquest The game also supports multi-direction control, allowing the agent to use URF-URF for moving with firing simultaneously
'L and R for turning left and right, D for squatting down, and F for attacking.
"U, D, I., and R for moving down, left, and right in the top view, respectivoly.
V and for moving up and right, F for firing.
21

---

Published as a conference paper at ICLR 2025
D.2 LEARNING PROCESS OF OPTIONS
In this experiment section, we show an example to illustrate how longer options are discovered during the training process, using example of options involving only in ms pacman, shown in Figure 7 In the 1st iteration, there are only primitive actions, where is the most frequently used (only primitive actions). Therefore, the agent begins using R with increased frequency, and the usage even exceeds 80% in the 3rd iteration. Mcanwhile, due to the high usage, the model starts learning options involving more R. The options R-R and R-R-R are therefore becoming the majority in the 4th and the 6th iteration, respectively. Eventually, the agent explores other actions, thus making the R-related options suddenly decrease after the 7th iteration. Note that in different training trials, the agent initially explores randomly, the first option learned does not consistently involve R. Depending on how the agent explores the primitive actions, options involving various combinations of actions are discovered different stages of the training process.
80%
R
R-R
R-R-R
60%
Proportion
40%
20%
0%
2
3
5
6
7
10
Iteration
Figure 7: Distribution of options R, R-R, and R-R-R at the beginning of learning ms pacman.
D.3 OPTIONS THE SEARCH TREE
In this experiment section, we present the detailed statistics in all 26 Atari games conducted for the behavior analysis in Section 5.4 Specifically, we provide the detailed proportions of options inside the search in Table 9 and the detailed tree depths in Table 10 In Table 9 the columns "% Env." and "% MCTS" represent the proportions of options applied suggested by the MCTS process. Note that due to softmax selection, the options suggested by the tree, "% MCTS", are not always applied, resulting in a lower "% Env.". Nevertheless, if the sug- gested options have higher visit counts, they are more likely to be applied, such as in crazy climber, freeway, and kung fu master, where options are well-trained for specific purposes, like climbing, crossing the road, and attacking with an uppercut, respectively. and therefore have higher probabili- ties of application. Also, the columns "% in Tree" and "% Sim." represent the proportions of the search tree that contains at least one option (in its 50 simulations) and the proportions of simulations whose selection path contains an option. In Table TO the columns "Avg" and "Max" represent the average and the maximum search tree depths; the columns "P25". "Pto". and "P7s" represent the 25th, 50th, and 75th percentiles. Based on the P30 and P75 depths, we observe that the search is not deep most cases, implying that the models learn perform deep searches only in certain states.
the environments and the options
22

---

Published as a conference paper at ICLR 2025
Table 9: Proportions of options in search tree for Eg and Eg in 26 Atari games.
Eg
fg
Game
Env.
% MCTS
in Tree
Env.
% MCTS
% in Tree
alien
32.34%
52.05%
99.81%
35.85%
30.72%
54.92%
96.80%
24.13%
amidar
34, 49%
52.26%
90.68%
31.95%
32.38%
48. 54%
90.17%
26.82%
assault
21.16%
33.99%
82.15%
11.93%
21.29%
35.45%
90.45%
11.95%
asterix
7.18%
15.02%
63.52%
7.11%
7.32%
17.19%
51.50%
4.99%
bank heist
58.07%
74.56%
99.45%
33.10%
51.83%
73.85%
97.71%
22.65%
battle zone
11.11%
30.41%
83.91%
14.91 %
8.46%
26.67%
80.12%
9.89%
boxing
60.47%
77.80%
99.09%
40.69%
51.54%
73.50%
98.70%
33.12%
breakout
21.00%
43.77%
90.74%
15.52%
18.79%
49.61%
90.37%
13.65%
chopper command
23 23.92%
41.11%
88 99%
21.96%
17.25%
33.43%
90.46%
15.90%
crazy climher
49.07%
67.94%
98.69%
39.96%
39.31%
59.56%
98.53%
37.09%
demon attack
23.30%
36.21%
93.16%
16.04%
17.36%
29.88%
82.75%
11.65%
freeway
60.80%
86.08%
99.11%
43.83%
54.62%
74.97%
98.75%
38.75%
frostbite
41.90%
65.81%
97.58%
41.18%
33.20%
61.49%
96.33%
30.52%
gopher
48.56%
63.83%
99.17%
25.33%
43.88%
65.21%
99.37%
20.31%
hero
18. 18.28%
39.68%
74.43%
20.07%
10.42%
26.64%
62.88%
11.60%
jamesbond
48.66%
67.76%
99.16%
39.10%
33.92%
52.62%
97.87%
27.33%
kangaroo
70.16%
86.86%
99.64%
55.49%
60.00%
78.64%
99.69%
44.32%
krull
32.28%
54.00%
99.04%
27.34%
39.69%
69.86%
99.79%
28.64%
kung master
61.52%
80.42%
99.62%
41.06%
46.91%
68
99. 80%
30.53%
pacman
34.99%
56.34%
96.90%
30.82%
22.87%
39.34%
95. 10%
21.84%
pong
75.43%
86.98%
99.46%
54.27%
63.18%
77.40%
99.48%
43.50%
private eye
6.40%
27.84%
67.01%
7.61%
2.96%
14.88%
44.25%
5.44%
qbert
49,96%
72.41%
99.16%%
38,83%
37.21%
$9.20%
33.9
29.35%
road runner
14.05%
28.43%
62.98%
12.36%
3.81%
8.70%
33.76%
3,99%
seaquest
25.12%
44.43%
95.88%
16.88%
17.59%
31.77%
92.72%
13.66%
up down
47.94%
66.05%
97.74%
29.12%
28.35%
44.52%
93.03%
17.53%
Average
37.62%
55.85%
91.43%
28.94%
30.57%
49.11%
87.48%
22.28%
Table 10: Tree depths for &1, Ez, and E in 26 Atari games.
E1
Es
fa
Game
Avg
Pas
Max
Avg
Max
Avg
Max
alicn
17.99
24.19
20
31
29.61
22
42
amidar
13.17
8
49
22.62
20
32
31.43
22
43
276
assault
9.86
7
9
12
45
8.81
6
11
78
9.30
7
11
127
usterix
10.39
6
7
11
49
7.83
5
7
9
100
7.15
5
6
8
102
bunk heist
13.77
13
15
49
17.10
14
17
20
88
16.22
12
15
19
114
battle zone
16.70
24
49
11.36
13
10.93
12
144
boxing
23
37
23
S4
breakout
11.86
7
43
10.64
13
90
9.49
11
83
chopper command
12.03
7
10
15
49
15.58
8
12
20
141
13.13
10
16
228
crazy climber
17.90
24
50
36.12
15
34
51
147
61.50
15
51
101
288
demon attack
9.52
8
11
49
11.17
10
13
135
9.88
12
freeway
48
35
44.50
60
frostbite
23.28
34
50
31.05
15
25
41
147
44.56
21
SS
294
gopher
12.98
10
13
15
46
17.01
12
16
21
114
15.16
11
14
18
145
hero
22,30
10
19
33
50
17.06
6
10
21
147
14.80
5
13
276
jamesbond
20.28
25.91
25
35
138
28.77
13
24
41
246
kangaroo
17.89
23
39.29
25
39
52
150
46.69
40
62
krull
10.29
7
9
48
15.24
13
19
64
21.65
17
22
204
kung fu master
15.83
12
15
19
49
26.10
19
26
32
84
27.27
16
26
36
138
ms pacman
14.87
8
12
20
49
21.51
10
17
30
135
23.58
15
31
169
pong
23.21
31
50.28
28
48
67
144
67.51
19
60
private eye
10.61
6
50
8.22
10
129
6.22
168
qbert
13.09
7
11
17
48
26.83
15
25
36
138
38.31
13
30
59
228
road runner
8.17
4
5
10
49
10.14
10
120
6.61
5
139
seaquest
10.94
8
10
12
49
12.34
8
10
13
114
11.43
7
9
13
115
up n down
10.40
49
17.42
22
150
13.91
13
18
Average
14.52 8.77
12.58
18.85 48.54
20.74
11.65
18.23
26.69
121.46
24.92
10.58
19.35
33.58
197.58
23

---

Published as a conference paper at ICLR 2025
Table 11: Prediction accuracy between options and environmental actions for Ez and le 26 Atari games.
En
Game
0-stop
1-step
2-stop
0-step
1-step
2-stop
3-step
4-step
5-stop
alien
73.43%
81.24%
66.76%
68.84%
75.00%
61.55%
56.89%
53.50%
51.49%
amidar
75.55%
81.32%
71.78%
78.02%
82.90%
62.28%
55.19%
50.08%
48,32%
assault
76.61%
83.70%
35.49%
76.55%
83.98%
37.41%
11.58%
3.94%
1.48%
asterix
64.41%
71.68%
29.46%
63.54%
69.28%
13.08%
7.10%
4.16%
2.59%
bank heist
91. 84%
93.82%
72.82%
90.59%
91.42%
55.11%
35.85%
23.09%
15.64%
battle zone
59.31
68. 10%
35.97%
53.03%
62.06%
33.77%
22.97%
18.79%
16.86%
boxing
87.66%
92.72%
76.38%
85.41%
89.93%
64.36%
51.47%
45.70%
41.32%
hreakout
65.61% %
72.96%
33.90%
58.44%
65.56%
19.29%
5.35%
2.77%
1.80%
chopper command
71.11%
76.93%
51.90%
69.16%
74.82%
33.97%
23.98%
19.02%
13.41%
crazy climber
84.24%
91.40%
77.09%
83 95%
89, 89.90%
67.90%
57.60%
53.24%
50.56%
demon attack
76.62%
84.59%
53.52%
73.07%
81.17%
35.98%
19.92%
12.17%
8,97%
freeway
76.15%
85.47%
83.13%
80.20%
88.91%
80.89%
77.34%
75.54%
74.57%
frosthite
73.53%
80.95%
75.86%
68.05%
75.05%
67.44%
62.74%
59.82%
$7.79%
gopher
85.18%
90.50%
71.23%
80.08%
86.74%
41.73%
28.23%
20.99%
17.57%
hero
60.77%
69.45%
56. 73%
60.65%
67 48%
47.38%
41.79%
39.61%
37.74%
jameshond
80.04%
85.72%
70.20%
12%
82.04%
63.76%
55.58%
52.27%
49.35%
kangaroo
88.71%
91.89%
84.32%
86.98%
90.48%
76.48%
68.02%
61.90%
59.13%
krull
73.75%
81.45%
49.56%
75.78%
82.41%
57.58%
35.83%
24.81%
19.37%
kung fu master
85.54%
90.36%
77. 14%
84.72%
89.95%
58.19%
47.24%
39.70%
36.20%
ms paeman
71.76%
78.05%
68.81%
70.02%
79.43%
49.92%
40.89%
37.39%
36.39%
pong
90.04%
94.63%
89.50%
86.81% %
92.52%
80.49%
74.85%
69.73%
66.69%
private eye
53.14%
61.30%
18.11%
47.93%
61.86%
21.19%
12.62%
9.67%
7.98%
ghert
76.17%
82.71%
79.03%
75.21%
83.02%
72.12%
63.94%
59.87%
57.88%
road
68.14%
71.35%
39.60 60%
65.75%
73.07%
34.80%
26.09%
21.45%
19.21%
seaquest
76.62%
82.36%
37.48%
19%
80.98%
35.47%
21.45%
15.91%
12.74%
up down
80.72%
88.11%
77.24%
77.96%
87.10%
56.07%
43.43%
34.44%
30.06%
Average
75.64%
82.03%
60.89%
73.50%
80.27%
51.09%
40.31
34.98%
32.12%
D.4 PREDICTION ACCURACY OF OPTIONS
In this experiment, we check how the options suggested by the MCTS process predict future actions. For example, if an option R-R-R R is suggested by the search at time l, we calculate the prediction ac- curacy using the recorded actions at, Gt+1, and 0c+2. The results for both E3 and la are shown in Table II As expected, the prediction accuracy decreases as the number of steps increases. Inter- estingly, y,, the accuracy of the I-step is higher than that of the O-step. We assume this phenomenon is caused by softmax selection for options with repeated primitive actions. For example, the agent may apply U than the suggested options R-R-R. When this happens, at step t + 1, there is likely a much higher probability of acting again prompt the original decision. On the other hand, the prediction accuracy of & is generally lower than those of la at the same step number. is hypoth- esized that increasing the maximum option length also increases the difficulty training prediction and dynamics networks, thereby lowering the accuracy. To summarize, this observation verifies that the learned options closely correspond to the probabilities of choosing primitive actions.
24

---

Published as a conference paper at ICLR 2025
D.5 TOP FREQUENTLY USED OPTIONS
Finally, we investigate the distribution of frequently used options for each game. The accumulated usages are summarized in Table 12 Additionally, extract the top frequently used options for both f3 and & in Table 13 and Table [4] Generally speaking, the distribution of frequently used options is similar between &3 and l.. where the distribution is highly unbalanced, with the top 25% options accounting for over 95% of the usage. The most extreme case occurs in freeway, where the top 1% (options with repeated U) takes a proportion of more than 80%. On the other hand, for crazy climber and ms pacman, the top 1 1 usages have both been significantly increased from la to &g. However, the increase in usage does not correlate to the performance, as lg outperforms observe ls in crazy climber but underperforms in pacman, shown in Table [ Notably, that from &g to lg. the numbers of discovered options only increase by factor of 2.43 on average, implying that there are not many options practical for the planning. This analysis demonstrates that a small proportion of frequently used options play important roles in gameplay, and the number of discovered options will not grow excessively when relaxing the limitation of maximum option length.
Table 12: Accumulated usages of the frequently used options for &a and le in 26 Atari games.
fg
Game
Top 1
Top 5%
Top 10%
Top 25%
Top 1%
Top 5%
Top 10%
Top 25%
alien
21.67%
74.01% %
89.35%
98.64%
53.56%
81.47%
90.76%
98. 98.15%
amidar
41. 11.82%
91.33%
98.26%
99.50%
49.67%
83.14%
94.95%
99.23%
assault
29.76%
70.26%
85.88%
97.12%
45.96%
81.53%
91.81%
98.31%
asterix
30.27%
75.00%
90.57%
97.38%
36.46%
77.98%
88.22%
97 97.86%
bank heist
22.46%
50.28%
71.80%
96.60%
20.68%
54.44%
70.90%
90.26%
battle zone
31.80%
74.53%
95.23%
99.31%
41.56%
78.69%
91.88%
98 58%58%
boxing
39.55%
59.41%
71.47%
89.90%
29.40%
59.22%
73.56%
90.43%
breakout
23.33%
61.10%
79.71%
95.22%
39.69%
75.29%
88.13%
97. 97.98%
chopper command
40.51%
90.41%
96.61%
99.33%
35.91%
75.72%
91.05%
99.05%
crazy climber
18.00%
52.19%
73.53%
95.66%
52.62%
82.73%
91.65%
98.26%
demon attack
36.88%
81.62%
91.80%
97.69%
46.19%
82.89%
92.90%
98. 68.65%
freeway
87.63%
91.25%
93.94%
97.71%
81.13%
89.75%
94.46%
98.51%
frostbite
58.73%
90.74%
96.43%
99.21
62.58%
86.18%
93.24%
98.26%
gopher
58.17%
77.73%
86.00%
96.81%
46.93%
83.74%
92.58%
98.51%
hero
57.14%
91.52%
96.54%
99.17%
52.72%
83.91%
91.80%
97.92%
jamesbond
51.74%
85.34%
93.23%
98.43%
55.25%
84.53%
93.46%
98.36%
kangaroo
43.29%
71.76%
81.93%
94.30%
46.48%
74.34%
85.05%
95.85%
krull
23.50%
56.37%
74.32%
93.80%
21.42%
54,80%
73.48%
92.43%
kung fu master
48.52%
79.10%
89.6 64%
97.27%
46.32%
75.97%
86.49%
96.42%
ms pacman
28.04%
88.11%
96.09%
99.22%
41.82%
75.41%
90.17%
98.02%
pong
59.22%
80.07%
87.07%
95.88%
49.13%
75.00%
86.40%
96.14%
private eye
37.06%
68.97%
84.19%
97.25%
33.63%
69.21%
81.91%
94.19%
qbert
53.07%
94.42%
97.80%
99.69%
67.30%
91.71%
96.82%
99.30%
road runner
28.77%
80.77%
90.97%
98.11%
42.47%
78.51%
89.48%
97.43%
seaquest
29.84%
67.32%
80.34%
94.60%
36.79%
73.06%
86.27%
96.69%
up n down
62.27%
85.91%
94.45%
98.97%
53.55%
86.24%
94.25%
99.14%
Average
40.89%
76.52%
87.97%
97.18%
45.74%
77.52%
88.53%
97.07%
25

---

Published as a conference paper at ICLR 2025
Table 13: The top frequently used options with their total share for &3 in 26 Atari games.
Game
Top
Top LF-LF-LF R-R-R
Top UL-UL-UL
%
alien
UR-UR-UR
10.01%
amidar
LF-LF-LF
DF-DF-DF
18.07%
assault
R-R
L-L
R-R-R
9.07%
asterix
U-U-U
D-D
UL-UL
2.79%
bank heist
L-R-L
L-UR-L
DL-UR-DL.
16.79%
battle
DF-DE-DF
DRE-DRE-DRE
LF-LF-LF
3.53%
boxing
R-R-R
DR-DR-DR
DRF-DRF-DRE
23.92%
breakout
N-N-N
N-N
L-L-L
10.52%
chopper command
DF-DF-DF
UF-UF-UF
ULF-ULF-ULF
9.69%
crazy climber
D-D-U
U-U-U
U-U-D
12.16%
demon attack
R-R-R
LALL
LF-LF-LF
10.77%
freeway
U-U-U
N-N-N
U-U
57.12%
frostbite
D-D-D
UR-UR-UR
DR-DR-DR
24.61 %
gopher
L-L-L
R-R-R
RF-RF-RF
26.55%
hero
R-R-R
DLF-DLF-DLF
LF-LF-LF
8.11%
jameshond
UR-UR-UR
L-DLF-L
L-L-L
21.31%
kangaroo
D-D-D
F-F
LF-LF-LF
30.37%
krull
DL-DL-DL
UR-UR-UR
DL-DL
10.17%
kung fu master
DR-DR-DR
DLF-DLF-DLF
ULF-ULF-ULF
20.89%
pacman
UL-UL-UT.
DI-DL-DI.
UR-UR-UR
14.52%
pong
LF-LF-LF
F-F
N-N-N
48.94%
private eye
URF-URF
LALL
L-L.
2.37%
qbert
D-D-D
R-R-R
L-L-L 36.60%
road runner
DLF-DLF-DLF
UL-UL-UL
DL-DL-DL
5.82%
seaquest
DF-DE-DF
DLF-DLF-DLF
DLF-DLF
6.12%
up n down
U-U-U
UF-UF-UF
DE-DE-DF
34.05%
Table 14: The top 3 frequently used options with their total share for lg in 26 Atari games.
Game
Top
Top 2
Top IFIILEIFIFII
%
alien
WR-DR-DR-DR-DR..
13.52%
amidar
LI--F--FFF
UT-UF--U-LT-UFAAA L-L
RI-RF RT-RF-R-RFF-FFR
13.30%
assault
R-R
U-L-L
8.35%
asterix
UL-UL
UR-UR
D-D
3.49%
bunk heist
DR-L
UR-L
R-DL
6.77%
battle zone
DF-DE-DF-DE-DER DR-DR-DR-DR
UL-UL-UL-UL-UT-U
U-U
2.43%
hoxing
N.N.N.N-....
breakout
F-F
N-N
L-L
9.75%
chopper command
D.F-DLF.DLP.DR-. D.D.----DDD
ensy climber
DL
UL-UL-DR DLU DLUU
U-U-DL-DL DL-DRU
14.20%
attack
LFLF
freeway
U.U.U.UUUU FFFFFF L-L DL DL-DI-DL-DL-DI-DDL R-R-----RR
U.U.
D-U-U.U-UUU
45.91%
frostbiie
DE-DE-DI-D-DF-DD R-R RE-RE-E- RE-REE
DRE-DRE-DRE DRR-DDR-DRE--DR-EEEE
14.63%
gopher
R-R-R-R-R-R
11.25%
jamesbond
LI-L--LL-L--L--LL
UR-UR-UR-UR-URAS
11.66%
Kangaroo
LLLLL
17.11%
knill
UR-UR UR-UR UR UR
DL
LILLLL
4.40%
kung
DR-DR-DLF-DR-DRR
LLLILL
DOF-UR--UR-UUREUUEL.
7.57%
ms pacman
UL-ULULA UI.
ULLL
DR-DR DR DR
9.56%
pong
RF-RF-RF-RE RF- RF-RR--F-R
R.R.RRRRRRR
LF-LF--F-LL--F--
24.58%
private eye
URFAUR FFFFFFF
ULFELF
DF-DE-DF.DE-DEE
0.78%
qbert
R
U-D-U.U-U..
21.94%
road runner
L-L
L-L-L-L-L-L
1.62%
seaquest
DLF-DLP
DRF-DRF
B.FDLF-DLF-DLL-DII
3.30%
up a dowa
U-U-U-U-U-U
DF-DF
15.18%
26


Reviews:
[{"review_id": "Na3Kc8R1Fu", "reviewer": "Reviewer_JCLU", "summary": "The work proposes a revamped approach to the well-known options framework, which allows agents to take temporally extended actions as well as myopic ones. By combining a network which learns options with MCTS MuZero (which models transition dynamics) the authors propose a method to utilise options alongside single actions in self-play games.", "questions": "Clerical: In section 5.2 the $l_{1}$ option setting is mentioned as a baseline, but Table 1 compared the options to something called $l_{0}$. Do $l_{0}$ and $l_{1}$ refer to the same baseline? If so, using consistent notation will help make the results section more readable.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The need for efficiency in decision making in RL is clear, as single-step actions are slow and computationally expensive (even more so in slow simulators). Thus, the problem addressed by OptionZero is clear and its existence is well-motivated. Additionally, since much prior work in options appear to be in manually defined and demonstration-based settings, the generalisability of OptionZero is a strong selling point.\n\nWithin fixed computational constraints, the idea of decreasing the frequency of queries to the network is a strong idea for the current state of RL. Also important is the notion of learning subroutines which the options network will identify as useful in different scenarios and not have to re-learn temporal relationships.\n\nThe flexibility to play options or primitive actions results in tailored reactions to scenarios, as an agent may need the fine-grained approach taken by traditional RL. The main results in Table 1 indicate the validity of the method, as using options provides a performance benefit more often than not, with longer option limits sometimes outperforming shorter ones.", "weaknesses": "It is unclear why options are outperformed by primitive actions in certain environments. The authors suggest that in environments with high combinatorial complexity, learning of the dynamics model may be difficult and thus options may simply produce more overhead than actual benefit. A more detailed analysis of these environments would be beneficial, for e.g. investigate whether there is a correlation between the stochastic branching factor of the environment and the performance of options.\n\nAdditionally, it seems that longer options may improve efficiency but not always increase performance when those options may be overextending in environments where more granular control is required. Have the authors considered implementing dynamic options lengths somehow? This may make the idea more viable, or at least a discussion on the complications of implementing that would be a good addition to the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MGxpYWJ1jL", "reviewer": "Reviewer_EWix", "summary": "This paper presents OptionZero, a novel approach that incorporates an option network into MuZero and allows the agent to learn temporary extended actions. The authors conducted empirical experiments and find OptionZero outperforms MuZero in teams of mean human-normalized scores on 26 Atari games.", "questions": "1. Why select these 26 Atari games instead of using the standard 57 Atari games?\n1. What is the hardware resource for conducting the experiments?\n1. How long does training a single Atari game with OptionZero take?\n1. How long does training a single Atari game with MuZero take?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written. The authors explain the use of options clearly with a toy example, demonstrating how options are used. The empirical results are also strong, achieving high mean normalized scores.", "weaknesses": "It's not clear the actual benefits options bring. In the intro, the paper claims options allow for \"searching deeper\", but the empirical analysis shows \"deeper search is likely but not necessary for improving performance\". While it's nice to have the option to do option, could the authors provide a more detailed analysis of options beyond a deeper search? \n\nThe paper could also benefit more from discussions of \n1) the trade-offs between increased complexity and performance gains\n2) how much tuning did the authors perform to make OptionZero work; were there failure cases/ideas during the development and how did the authors overcome them?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4VtKakuiEr", "reviewer": "Reviewer_S6xr", "summary": "The authors introduce OptionZero, an advanced extension of the MuZero algorithm that autonomously identifies temporally extended actions, known as options, without the need for predefined options or expert-provided demonstrations. By incorporating an option network, OptionZero enhances planning efficiency by decreasing both decision-making frequency and the computational load required for complex environments. Evaluated on a suite of Atari games, OptionZero demonstrates notable improvements in human-normalized scores compared to MuZero, accompanied by a detailed analysis of how options are utilized across varying game states and scenarios", "questions": "- How does the dynamics network handle complex action spaces, especially in games with highly varied option paths?\"\n\n- What are the specific computational costs of incorporating the option network, both in training and during MCTS simulations? Could the author discuss the associated overhead with the proposed method?\n\n- Can the model be applied to environments beyond games with less predictable state transitions, and how would option discovery be affected? I would suggest adding studies in robotic environments.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Novel idea for autonomous and adaptable option discovery. The proposed method's ability to autonomously discover and tailor options to diverse game dynamics removes the need for predefined actions, making it highly adaptable across different environments.\n\n- Convincing results for enhanced planning in RL.  By integrating an option network, OptionZero reduces decision frequency, enabling computational efficiency, particularly in visually complex tasks like Atari games.\n\n- Strong Performance Gains: On Atari benchmarks, OptionZero achieves a 131.58% improvement over MuZero, a significant improvement compared to previous SOTA papers.\n\n- Interesting ideas to adjust option lengths, balancing performance and training efficiency, particularly useful in tasks needing variable action sequences.", "weaknesses": "- Inconsistent Option Use Across Games: OptionZero's reliance on options appears to vary widely across Atari games. While longer options bring substantial gains in some games, they contribute less in others. This inconsistency suggests that the model’s option-based planning may struggle to generalize well across diverse, complex environments. The paper should discuss this limitation. \n\n- Challenges in Complex Action Spaces: In games with intricate action spaces, such as Bank Heist (Atari), OptionZero’s dynamic network encounters difficulty as option lengths increase, particularly with multi-step dependencies. This issue may restrict OptionZero’s application in environments where actions are highly combinatorial, relying instead on settings with more straightforward or predictable actions.\n\n- Reduced Prediction Accuracy for Longer Options: The model’s prediction accuracy tends to decrease as options become longer, affecting planning quality where extended strategies are essential. I would recommend adding an experiment to study this effect and discuss potential limitations of the proposed method. \n\n\n- Limited Application Beyond Games: Although the model shows promise in game environments, the paper does not investigate its potential beyond Atari-like settings. I would appreciate seeing results on other domains, maybe robotic such as Gymnasium-Robotics. Under the current evaluation, the method seems limited to game-based scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PWGy4d8enp", "reviewer": "Reviewer_jR7c", "summary": "This paper introduces the OptionZero framework, which incorporates options into MCTS and enables the automatic design of options through self-play, thereby avoiding cumbersome manual design. In Atari games, OptionZero achieves significant improvements compared to MuZero, achieving a 131.58% improvement in mean human-normalized score. It has shown promising directions for discovering and using options in planning.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper is well-written with a clear structure, making the research content easily comprehensible.\n\n2. By introducing the OptionZero framework and leveraging self-play for automatic option design, this study paves the way for new approaches. The novelty is good.\n\n3. The experimental results robustly support the effectiveness of the algorithm.", "weaknesses": "1. I am curious about whether the introduction of options has any impact on the theoretical optimality of MCTS.\n\n2. Is it possible to demonstrate the advancement of the algorithm more directly by solely utilizing the learned network for action selection, without relying on MCTS?\n\n3. What is the expected performance as the value of $l$ increases?\n\n4. What are the differences in the running wall-clock time required for OptionZero compared to MuZero?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)