ICLR 2025

Published as a conference paper at ICLR 2025
OPTIONZERO: PLANNING WITH LEARNED OPTIONS
Po-Wei Huang1,2, Pei-Chiun Peng1,2, Hung Guei1, Ti-Rong Wu1†
1Institute of Information Science, Academia Sinica, Taiwan
2Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan
ABSTRACT
Planning with options – a sequence of primitive actions – has been shown ef-
fective in reinforcement learning within complex environments. Previous studies
have focused on planning with predefined options or learned options through ex-
pert demonstration data. Inspired by MuZero, which learns superhuman heuristics
without any human knowledge, we propose a novel approach, named OptionZero.
OptionZero incorporates an option network into MuZero, providing autonomous
discovery of options through self-play games. Furthermore, we modify the dy-
namics network to provide environment transitions when using options, allowing
searching deeper under the same simulation constraints. Empirical experiments
conducted in 26 Atari games demonstrate that OptionZero outperforms MuZero,
achieving a 131.58% improvement in mean human-normalized score. Our behav-
ior analysis shows that OptionZero not only learns options but also acquires strate-
gic skills tailored to different game characteristics. Our findings show promising
directions for discovering and using options in planning. Our code is available at
https://rlg.iis.sinica.edu.tw/papers/optionzero.
1
INTRODUCTION
Reinforcement learning is a decision-making process in which an agent interacts with environments
by selecting actions at each step to maximize long-term rewards. Actions, commonly referred to as
primitive action, advance the state by one step each. While this granularity allows precise control
at each time step, it can lead to inefficiencies in scenarios where predictable sequences of actions
are beneficial. For example, in a maze navigation task, it is more efficient to choose a sequence
of actions – such as following a strai...